Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems
Authors: Jiayu Chen
Yuanxin Zhang
Yuanfan Xu
Huimin Ma
Huazhong Yang
Jiaming Song
Yu Wang
Yi Wu
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-agent reinforcement learning
curriculum learning
variational inference

Abstract: We introduce an automatic curriculum algorithm, Variational Automatic Curriculum Learning (VACL), for solving challenging goal-conditioned cooperative multi-agent reinforcement learning problems. We motivate our curriculum learning paradigm through a variational perspective, where the learning objective can be decomposed into two terms: task learning on the current curriculum, and curriculum update to a new task distribution. Local optimization over the second term suggests that the curriculum should gradually expand the training tasks from easy to hard. Our VACL algorithm implements this variational paradigm with two practical components, task expansion and entity curriculum, which produces a series of training tasks over both the task configurations as well as the number of entities in the task. Experiment results show that VACL solves a collection of sparse-reward problems with a large number of agents. Particularly, using a single desktop machine, VACL achieves 98% coverage rate with 100 agents in the simple-spread benchmark and reproduces the ramp-use behavior originally shown in OpenAI’s hide-and-seek project.

Variational Automatic Curriculum Learning for Sparse-Rewand Cooperative Multi-Agent Problems
Jiayu Chen 1f Yuanxin Zhang', Yuanfan Xu', Huimin Ma², Huazhong Yang', Jiaming Song", Yu Wang', Yi Wu 12: Tsinghua University, Shanghai Qi Zhi Institute, University Science and Technology Beijing, Stanford University, "j1a7681675355gmai1 com, "jxwuy1@gmiil.com
Abstract
We introduce a curriculum learning algorithm, Variational Automatic Curriculum Learning (VACL), for solving challenging goal-conditioned cooperative multi- agent reinforcement learning problems. We motivate our paradigm through a variational perspective, where the learning objective can be decomposed into two terms: task learning on the current task distribution, and curriculum update to a new task distribution. Local optimization over the second term suggests that the curriculum should gradually expand the training tasks from easy to hard. Our VACL algorithm implements this variational paradigm with two practical components, task expansion and entity progression, which produces training curricula over both the task configurations as well as the number of entities in the task. Experiment results show that VACL solves a collection of sparse-reward problems with a large number of agents. Particularly, using single desktop machine, VACL achieves 98% coverage rate with 100 agents in the simple-spread benchmark and reproduces the ramp-use behavior originally shown in OpenAl's hide-and-sede project. Our project website at htttts:://ssteeee www.le.ooooooooooooooooooooooooo cow wwttom/vvew//veeeo newreeewwwwaaaneeeeeeeeeee
Introduction
Building intelligent agents in complex multi-agent games remains a long-standing challenge in artificial intelligence
(23). Recently, it has been a trend to apply multi-agent reinforcement learning (MARL) to extremely challenging multi-agent games, such as Dota II [5]. StarCraft II
[29] and Hanabi [3]. Despite these successes, learning intelligent multi-agent policies in general still remains great RL challenge. Multi-agent games allow sophisticated interactions between agents and environment. Feasible solutions may require non-trivial intra-agent coordination, which leads to substantially
complex strategies than the single-agent setting. Moreover, the number of agents increases, the joint action space grows at an exponential rate, which results in an exponentially large policy search space. Thus, most existing MARL applications typically require shaped rewards, or assume simplified environment dynamics, or only handle a limit number of agents.
We tackle goal-conditioned cooperative MARL problems with sparse rewards through novel varia- tional inference perspective. Assuming each task can be parameterized by a continuous representation, introduce variational proposal distribution over the task space and then decomposing the overall training objective into two separate terms, i.c., a policy improvement term and a task proposal update term. By treating the proposal distribution updates the training curriculum, such variational objective naturally suggests a curriculum learning framework by alternating between curriculum update and MARL training. In addition, variables so that the variational curricula can be also applied over the number of agents and objects in the task, which leads generic and unified training paradigm for the cooperative MARL setting.
propose continuous relaxation technique for discrete
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We implement our variational training paradigm through a computationally efficient algorithm. Variational Automatic Curriculum Learning (VACL). VACL.onsiee two components, task expansion and entity progression, which generate series of effective training tasks in hierarchical manner. Intuitively, entity progression leverages the inductive bias that tasks with more entities are usually more difficult in MARL and, therefore, progressively increases the entity size in the environment. Task expansion assumes a fixed number of entities and is motivated by Stein variational gradient descent (SVGD) [77 to efficiently expand the task distribution towards the entire task space.
We provide thorough discussions on the differences between VACL and existing works (Sec. 3.4 and empirically validate VACL using single desktop machine in four physical multi- cooperative games, i.e., two in the multi-agent particle- world environment (MPE) [19] and two in the MuJoCo- based hide-and-seck environment (HnS) 2 (Sec. ED. In the MPE games, VACL learns significantly faster and better than all baselines and achieves over 98 % coverage rate with 100 agents in the
simple-spread testbed even using sparse rewards. In the HnS scenarios, VACL achieves over 90% success rates on both two games, including reproducing the ramp use behavior discovered in 2 and solving sparse-reward multi-agent Lock-and Return challenge where none of the baselines can produce a success rate above 15% using the same amount of training timesteps.
2
A Variational Perspective on Curriculum Learning
2.1
Preliminary
We formulate each goal-conditioned cooperative multi-agent task as multi-agent Markov decision process (MDP), M(n, 6) =< n, S, A, O,O,P, R, sg, Is. >, parameterized by discrete vari- able n (e.g., the number of agent= and continuous parameter E 4 (i.e., the initial state and goal; details in App. C.2). S is the state space. A is the shared action space for each agent. O is the observation space. O, = (s; denotes the observation for agent under state S. S denotes the initial state and Io denotes the goal. ((s') s, A) and R(s, A; go) are the transition probability and the goal-conditioned reward function given state and joint actions A = (a),..., an). y is the discounted factor.
We consider homogeneous agents and learn shared goal-conditioned policy To Te(a:los (ag]oo; 9) parameter- ized by for each agent i. The final objective is maximize the expected accumulative reward over the entire task space, i.c..
Σ E R(s A':'ssA'''0)) A':00) = EVnolVVn, Φ mo)),
J(O) E En,haat .st
where V (nn, o, Ta) denotes the value function for Th over task M (nz Φ). We remark that for problems with 0/1 sparse rewards, (nn, o, T&) can be also interpreted as the success rate of m
The main idea of curriculum learning is to construct a task sampler q(n, Φ) that always generates training tasks (n, Φ) that yield the largest learning progress for my to efficiently maximize ((0).
2.2 Stein Variational Inference over a Continuous Task Space
For simplicity, we ignore the discrete variable n and only focus on the continuous variable á in this
subsection. Herein, our objective is to maximize the expectation L = Ee--p(s) [V r)), where p()) is uniform distribution over feasible values of Φ. The subscript 0 of n is omitted for conciseness.
Given the objective L. motivate our automated curriculum learning procedure through the lens of variational inference. Let q()) denote the current distribution of training tasks, which will vary throughout learning: we can represent curriculum learning through an objective that optimizes g()).
'n can also be discrete vector where each dimension represents the number of a particular type of objects. We now simply assume n is a single integer denoting the agent number for conciseness. "This only rigorously holds when = 1. But let's assume it for simplicity. In practice, this statement approximately holds with close-to- y and relatively short horizon.

---

The following lower bound for the original objective L can be derived:
p(()
[p())
-1)
= Eeppplll m)] = E
((,
= Eddngg
x)
V ((O 8)
(2)
((())
{(0)
p(O) {(0)_
2 Bppa(())) [V(0, x]] + ())
V(0, x) log
(3)
Ls: policy update
E2: curriculum update
where the inequality due to x 1 loga, with equality achieved at p(ő) - q()) for all Φ.
Now, we have decomposed [( q) into a lower bound that contains two terms L1 and Lz. L1 is simply policy update objective under the current task curriculum q()) while the second term La can be interpreted as a curriculum update objective by maximizing (()) w.r.t the policy т. Therefore, the decomposed lower bound naturally yields a curriculum learning framework by performing an iterative learning procedure, i.e., alternating between (1) maximizing Li with æ (policy update) and (2) maximizing L2 with (()) (curriculum update). We remark that we only optimize w.r.t. Li instead of L + L2 for the purpose of stabilizing policy learning since can be very different from in the early stage of training. This also aligns with the curriculum learning paradigm. the following proposition, we show that if we can perfectly optimize the RL procedure for L under the curriculum q()), then the curriculum objective L2 will encourage (()) to converge p(ő).
Proposition 1. Assume that (0 E [0, and that there exists unique policy x* such that Vl, π²) = 1 for all Φ. For any q E P(Φ) \ {p} (i.e., any curriculum that is not the final one) and C2( q) < C2(x",
arg max, (I( (i.e., a policy that is optimal under the current curriculum),
Proof. First, have that (m .p) = 1 (definition of x² and L2("T", -p) 0. Suppose that 3m, q (T,
such that g # and ( ②) = 1; then it must be the case that whenever V(ò, T) # 1, (()) = 0, i.e.
VΦ supp(g), V( R) = 1. Then L2(0, T)
(gk((lpp) < O, which completes the proof. O
In other words, if our policy π already performs well under the current curriculum q(ő), then Le will encourage (()) become "closer" to p()), aligning with our eventual target of learning all tasks.
2.2.1 Stein Variational Gradient Descent for Curriculum Update
The most straightforward approach for optimizing L2 is to learn a generative model w.r.t. the current म, which can be particularly sample-inefficiet. Instead, consider updates to current curriculum via smooth updates, which is much more efficient than learning generative models.
Let represent the task distribution under the current curriculum (()) with set of particles Q= {o). To update each particle á in Q, we consider an incremental transform T(ő) = e + e where f()) is smooth function and is magnitude scalar. Our goal is then to derive suitable updates to Q that maximizes C2(q. т based on the transform T. Following 9 and (27, we introduce an assumption over the generalization ability of the task space.
Assumption 1. An optimal policy π that is trained over specific task o has some generalization ability to another goal o' close Φ. Specifically, if V T) c for any & E supp(q), then ()(( = for all ø' E B()), where B(Ø) is a small open ball around 6.
In the context of curriculum learning, this assumption suggests that for any policy that is trained on set of tasks, the performance (c.g., success rate) does not change if we only change the task parameters by an infinitesimal amount, In the following theorem, we derive the functional gradient of the curriculum update objective La in reproducing kernel Hilbert space (RKHS).
Theorem 1. Let = @ + =f() where ff is an element of some vector-valued RKHS of a positive definite kernel k(Ø, (') + x + TR, and q[q) the density of s) TT()) when 6 q, then
v L2( =) (0))
(4)
where (*())= B (k(d /V logp(o") + V k( ())
Proof is in App. A. While the above statement is very similar that used for Stein variational gradient descent [17]. the update for case also directly depends on the value V TT) and thus indirectly
3

---

depends on the current policy. Intuitively, if ( τ) ) is large, then ø' will contribute more to the curriculum update. whereas if ( .) = then o' will not contribute to the update.
Since we assume that p()) is uniform over the task space, Ver logp((") = O for the interior of +; and the task update becomes
(*)) = Bveel m)) Vor
(5)
For example, we take the RBF kernel k - =xpp-l) away from neighbouring points that have large k( o'), and the repelling force scales with (o',, T). If V ) O, i.e., ø' has not been solved at all, then there is no repelling force from o' to Intuitively, this process encourages the curriculum update to prioritize tasks that have not been fully solved yet, by exploring in the vicinity of existing tasks with high values.
o' 2 and then Eq. 5 will drive ø
2.3 Continuous Relaxation for Discrete Parameter
Now we switch to the discrete parameter in our task representation, which results in a non- continuous task space and can be difficult to directly combine with continuous optimization tech- niques. Therefore, we propose to utilize continuous variable alternative 2 to represent the discrete variable n so that the same variational inference technique for & can be applied here. Specifi- cally, assuming n denotes the number of agents, we represent n with a categorical distribution p(n;=)= Cal(zI, .2........ n parametrized by continuous vector [=1,...,,N, EN] assuming the maximum possible of agents is N. p(n; =) denotes the distribution which generates v agents with probability In this case, we can similarly introduce target distribution p(=) over the space of and derive the corresponding variational updates with (22, ω). Since we typically wish to learn a policy that can generalizes across different number of agents, such representation over multiple n's naturally satisfies this practical requirement. We also remark that this formulation can be directly extended to the problems with multiple discrete variables (e.g., number of both agents and objects) via a categorical distribution over all possible discrete combinations.
Variational Automatic Curriculum Learning
In this section, we implement the variational inference paradigm with computation-cfficient approxi- mations as our VACL algorithm in the context of sparse-reward cooperative MARL problems.
We will first present Task Expansion in Sec. which is the most important algorithmic component assuming a continuous task space. Then, Entity Progression, i.e., the component specialized for discrete task parameters, will be introduced in Sec. 3.2 Finally, the full algorithm and the literature
review will follow in Sec. 3.3 and Sec. 3.4
3.1 Task Expansion as Approximated Stein Variational Inference
Task Expansion a computation-effiiins
curriculum update method, which assumes a continuous task representation é and optimizes the variational lower bound L2 w.r.t. q(Ø). Directly optimizing L2
can be computational challenging. In the following content, we will present a few critical techniques for the efficient approximation computation.
3.1.1 An Ever-Expanding Particle Set
In the standard Stein variational inference framework, we typically maintain a fixed-size particle set Q and gradually update the distribution of these particles using Stein variational gradient descent. Since the policy is constantly changing throughout training, we need to repeatedly update the entire set of particles to match the corresponding proposal distribution under L2, which is computationally expensive. Note that in the cooperative MARL setting, assuming a fixed amount of agents and an ever-improving policy, if we initialize q()) to the proper sub-space of casiest tasks, (()) would monotonically expand towards the entire task space. Hence, we can simply maintain an ever- expanding particle set by repeatedly adding novel tasks with sufficiently high values (i.e., success rates) under the current policy without the need of modifying those existing particles in Q.
'If N is unbounded, we may consider parametrized discrete random processes such as the Chinese restaurant process. For all practical purposes here, this is unnecessary.

---

3.1.2 Value Quantization
Another issue for optimizing L2 is that whenever æ up- dates, V( π changes accordingly but evaluating the value function can be particularly expensive in MARL problems. As practical approximation, we roughly categorize all the tasks into 3 categories w.r.t. their val- ues, i.e., a solved subspace of highest values, Q {olV( x) mmax J. active subspace of moderate values, Qact = { I (ó, σ) S mmax): J. and the re- maining unsolved subspace. Here muin and max are quantization thresholds. Since the value function would monotonically increase as training proceeds, need to verify whether an active task becomes solved or whether an unsolved task becomes active, and maintain Quet and Q.n accordingly. Once a task becomes solved, future value estimation becomes longer necessary. In addition, the curriculum learning principle, when optimizing и under L1, we can also sample more training tasks from Qaet for more effective training.
V( re)
unsolved
solved
active set
task space
only
Figure 1: Task space partition.
3.1.3 Sampling-Basse Based Partiele Exploration
With an ever-expanding particle set Q, we need to explore novel task samples in the task space and add those tasks with sufficiently high values (i.e.. V 2 minn) to Q. The update equation for (()) in 5 naturally suggests diversity-basse task expansion scheme, namely exploring sufficiently novel tasks in the vicinity of Q. Due the difficulty in value estimation, we again follow the value quantization framework and simplify (5. as follows:
active set
exploration
tasks
(6) In the simplified (6. we simply ignore tasks with moderate values (i.e., active tasks in Qact) and only evaluate the expectation w.r.t. tasks with the highest values (i.e., solved tasks in Qoo1) as practical approxi- mation. Moreover, since Quo is ever-expanding, Eq.(6) also suggests that should explore novel tasks in the boundary region between Qsol and Qaet (Fig. [2. Hence, in practice, whenever an active task from Qaa becomes solved after policy update, select them seed tasks derive novel exploration task samples.
(-) x Ex'e (V Vork(o',)))
newly solved task
uniform noise / (-)-
We explore novel tasks in the boundary region between Q.n and Qact
Figure 2:
(6 suggests purely gradient-hased exploration, which may cause practical issues in cur- riculum learning since the feasible task parameterization space often highly constrained. Fig. 5(b) illustrates a maze scenario where agents need to navigate towards the landmarks and avoid the black obstacles. The task parameterization Φ includes the positions of agents and landmarks. The feasible subspace of ó is constrained by the obstacles while these feasibility constraints are only accessible as black box the curriculum learning setting. Therefore, directly computing constraint free gradient can easily lead to an exploration direction towards infeasible regions. Here we suggest simple sampling-basse enhancement for such highly-constrained problems, which promotes effective task-space exploration and can be viewed as zero-th order approximation of the projected gradient.
Pejection-Sampliing Exploration: We explore novel tasks deep by adding small uniform noise the gradient of seed tasks, namely &eep <- &oood +ee* ((@soo)) Unif(-6,,0) Then reject those infeasible tasks by querying the environment for configuration feasibility.
5

---

Algorithm 1: The VACL Algorithm
Input: 0, nog N, B, Bexpi // B: training batch size; Bexp exploration size Output: final policy Rg;
k < O, Q < (. Q GetEasy(no): // use easy tasks to initialize ((0)
repeat
2 2 mx+1 t- Inc( Oul t- 0, g < GetEasy
while not converge with ng do
// Policy Update
Myain + Samplee B x 2, Q*) + Sample B x (1 2), C*+1);
Train and evaluate Ta Mirain via MARL;
// Task Expansion
expand Qua (k, and Qai (k,++11 using the value estimates on Minin'
// generate novel exploration tasks
Maced < active tasks from Murinn that just become solved under ae
Mexp f- generate Bexp exploration task samples from seeding tasks in Mesdi
Add exploration tasks Mexp to (b,,++)),
// Entity Progression
Decay
k f k 1;
until n2 N:
3.2 Entity Progression for Massive Multi-Agent Learning
Directly applying the Stein variational update (Eq. (5) over the relaxation of the discrete variable results uniform expansion over the entire N-dimensional space of z, which can be computational challenging when N is particularly large. We leverage important inductive bias in cooperative MARL that problems with more agents are generally harder than fewer agents, which implies monotonic update scheme over the space by starting with Zng = 1, where no the smallest possible number of agents, and gradually increasing the probability mass of zµ for larger A values.
Entity Progression simplifies this procedure further by restricting to only have non-zero entries, ie., 2 + Σ = with K' > k, which suggests smooth and progressive transition from a simpler problem with i agents (i.e., 2 1) to harder problem with k' agents (i.c., 2g' 1). Note that can increase the agent number incrementally (i.e., k = + 1) or at an exponential rate (i.e., k' = 2k), the latter of which is typically more preferred in practice. We remark that entity progression can be directly applied to problems with multiple discrete parameters by viewing l as a discrete vector.
3.3 VACL: A Hierarchical Automatic Curriculum Learning Algorithm
Since the change of discrete variable may cause a drastic distribution shift of value functions, instead of using a joint continuous representation unifying both and o. VACL combines Entity Progression and Task Expansion in hierarchical manner: at the low level, task expansion maintains a separate curriculum (d) for each and performs particle-base based curriculum update under fixed n; at the high level, entity progression progressively increases to n by smoothly switching training distribution from 9dh (Ø) to q" (Ø). The full procedure is summarized in Algo.
3.4 Connection Existing Works
We focus goal-conditioned multi-agent cooperative problems and assume paramterized task space. This a common setting in the recent Automatic Curriculum Learning (ACL) literature 25) Some works also consider fixed set of tasks (20), which are typically referred to multi-task learning [33] The core idea of ACL is to train the policy using tasks with moderate difficulty, i.e., neither too hard
easy, which naturally emerges consequence of our variational paradigm.
popular class of ACI. methods learns a generative model, such as VAE 26 GAN 9 7 and
Gaussian mixture model [24] 12 22. over the task space with density concentrated tasks with

---

desired difficulties. These methods can be viewed as generic solution under our variational objective (Eq. B) by representing (()) as a generative model and directly optimizing L2 whenever the policy is updated. However, it can be extremely hample-inefficii to repeatedly learn g(() from scratch without leveraging the fact that (()) is ever-expanding in cooperative MARL. problems.
Another type of ACL methods is particle-based, which maintains set of solvable task samples and gradually expands the set towards the entire space. [10] [ assume fixed goal and generate tasks with starting states increasingly far away from the goal; M 21 consider a task space over simulator configurations and train from the easiest settings to the hardest for sim-to-reat adaptation. These methods are conceptually similar to the Task Expansion component in VACL for its particle- based nature and can be viewed as a random search procedure for optimizing C2 in a gradient- free manner. Our method provides principled interpretation of curriculum updates, and achieves superior performance in practice. Besides, 30 learn an individual policy paired with each task for discovering diverse behaviors while our work learns single goal-conditioned policy for the entire task space; 32 defines the active set as those parts of the task space which the value function exhibits the largest gradient w.r.t. the task variable and proposes to meta-learn (6))) to more effectively expand the curriculum distribution, which is complementary to our work. [4 [5] propose novel optimization methods based on the value function to minimize the KL-Divergence between g(Ø) and p(ő) while our work provides a gradient- praddeent-bssed exploration method to expand (()) to the entire task space.
propose population curriculum, which trains entity- invar-nivvariatt policies over training tasks with a growing number of agents. [8] adopt a similar curriculum learning method for Sokoban with an increasing number of boxes. These methods can be viewed as special case of our Entity Progression component by applying a hard transfer from 2 = to zgz = while our method smoothly switches between different number of entities and also yields better empirically performances.
[8] 31
Experiment
We consider four tasks over two environments. Simple-Spread and Push-Ball in the multi-agent particle-world environment (MPE) [T9], and Ramp-Use and Lock-and-
in the hide-and-seek environment (HnS) [2] Every experiment is repeated over 3 seeds and performed on desktop
machine with one 64-core CPU and one 2080-Ti GPU, which is used for forward action computation and training updates. More details can be found in Appendix.
4.1
The Multi-Agent Particle-Word Environment
The two tasks are illustrated in Fig. In Simple-Spreud, there are n agents and n landmarks and the agents need to occupy all the landmarks. The agents are penalized for collisions and only receive positive reward when all the landmarks are covered. In Push-Ball there are n agents, n balls and n landmarks. The agents need to physically push the balls to get every landmark covered. A success reward is given after all landmarks are covered. We scale n at an exponential rate following [8]
4.1.1 Main Result
We present most important results in this section and more analysis can be found in App. B. We
compare VACL with 5 baselines, including (1) multi-agent PPO (34] with uniform task sampling (Uniform), (2) population curriculum only (PC-Unif), (3) reverse curriculum generation (RCG) [0).
(4) automatic goal generation (GoalGAN) (9. which uses a GAN to generate training tasks, (5) adversarially motivated intrinsic goals (AMIGo) (6), which learns a teacher to generate increasingly challenging goals. We test all methods in Simple-Spread with n = 8 and Push-Ball with n 4. For PC- Un-ff and VACL, we start with ng in Simple-Spread and no
in Push-Ball and then switch to the desired agent number. Evaluation is always performed on the final N. Results are shown in Fig. 4 where VACL outperforms all baselines with clear margin. Simple-Spread, all baselines fail to solve the task: Uniform and PC-Unif fail to gain sufficient progress due to sparse rewards;
RCG quickly runs out of active tasks; GoalGAN and AMIGo takes an extremely long time to train a good neural goal-generator. In Push-Ball, Uniform and PC-Unif learn much slower, RCG quickly converges to a local minimum without further policy improvements, GoalGAN and AMIGo again consumes large number of samples to make training progress.
7

---

Simple-Sprean (n=0)
Push Ball (n~4)
0.8
0.5
coverage rdll
0.6
0.6
PC-Unif
RCG
0.4
GoalCAN
0.2
Timple-Spresse
Push-Ball
VACL
0.0
Agent
Landmark
Ball
2
0
2
3
tmestepstx100)
Uimestepaxx199
Figure 3: MPE tasks.
Figure 4: Comparison of VACL and baselines on MPE.
In addition to the baselines above, we also consider the set- ting of massive agents Simple-Spread and compare our VACL method with other coverage numbers in the existing literature. We consider two existing works. The atten- tional communication (ATOC) method [13), which adopts intra-agent communication channels and trains and n = 100 agents with dense rewards. ATOC is not open- open-sourced" We re-train VACL on the same environ- ment configuration (i.e., same agent/landmart size and room size) as ATOC for fair comparison with the num- ber reported in [3.. The other work is evolutionary population curriculum (EPC) [18, which reports the coverage rate on the standard Simple-Spread task with 24 agents. EPC adopts dense rewards and separate policies for each agent while VACL trains shared policy with sparse 0/1 rewards. The results are reported in Tab. | VACL achieves 98% coverage rate with n = 100 agents, which outperforms the highest number ever reported to the best of our knowledge.
Table I: The best coverage rate ever re- ported on Simple-Spread.
EPC
ATOC
VACI.
50
24
56.8%
/
97.6 %
50
92%
98.5 %
100
89%
98 %
4.1.2 A bllaioon Study
Choice of seed task: As implied by Eq. 5 and Eq. (6), VACL explores from the boundary between Qaet and Q by taking newly solved tasks as seeds to generate exploration tasks. A naive alternative is to simply take active tasks from Qast as seeds (Exp.Act.), which is similar to the exploration scheme in the RCG algorithm [10]. Note that exploration from active tasks may possibly produce unsolvable tasks, we consider an additional enhanced variant (Exp.Acr w. eval), which take active tasks as seeds but also perform value estimates (and training) those exploration tasks filter out unsolvable tasks before adding them to Qas-- Results of these variants are shown in Fig. 5(a) Using active tasks as seeds (Exp.Act.) leads to clear failure since the aggressive exploration brings too many unsolvable
tasks to Qxa- Although additional evaluation on exploration tasks (Exp.Act. w.eval) significantly stabilizes training, still yields much worse training progress than the standard VACL. This suggests that it is critical to explore from the vicinity of solved subspace.
SVGD-Principled Update: Our Stein variational paradigm suggests a gradient-based update rule to explore novel tasks Sec. 3.1.3 However, most existing particle-basee algorithms adopts a much simplified gradient-free -free alternative via uniform random noise, i.e., Deap Qseed + Unif( 8). To illustrate the effectiveness of our SVGD-pring WVGD-principled update rule, we consider a challenging Hard- Spread scenario in Fig. ((b) with 4 agents and 4 landmarks. Obstacles are in black and landmarks can only be placed in the right side. We initialize Qai with easy tasks where each landmark has an agent nearby and evaluate the performance of the learning policy those hardest tasks in Fig. (bb) We remark that due to the highly constrained task space, it becomes particularly challenging for the ACL method to gradually guide the agents to learn go through the narrow doors between obstacles. We evaluate the test performances of VACL and the simplified gradient-free version (Unif. noise) in Fig. 5(c) Our principled rule produces stable learning curve with fast convergence while the gradient- free version fails to solve the task.
Rejection Sampling: To investigate the effectiveness of cepection-samplngg based exploration (R.J.), we turn off rejection-samplnng (i.e., strictly following Eq (6) and conduct experiments on Hard-Spread (i.e., highly constrained task space) and Simple-Spread (i.e., less constrained task space)
in Tab. 2
We contacted the authors but did not get response.
8

---

Simple-Spread (n*4)
Hard-Spread (eval)
Hard-Spree Hard-SSpeed (n#4)
1.00
Unif noise
0.8
VACL
coverage rate
0.75
0.6
coverage
0.50
Exp.Act
0.25
0.2
Exp.Act
VACL
0.00
0.0
1
4
2
timesteps(x10")
timesteps( x110))
Choice of seed tasks
(b) The Hard-Spread scenario
(c) Exploration strategies
Simple-Spread (n=4)
Simplo-Spread (n=8)
Push-Ball (n=4)
1.0
0.8
coverage rate
coverage rate
0.9
coverage rate
0.6
0.2
0.7
0.4
0.0
1
3
0.0
2.0
timesteps(x100)
timesteps(x10")
timestepsi x100f)
400
1000
4000
8000
transfer
entity progresslon
(d) Sensitivity of particle number
(e) Effectiveness of Entity Progression
Figure 5: Ablation studies. (a) We evaluate different choices of seed tasks in Simple-Sreeae (b) We select the hardest tasks evaluate the performance of different exploration strategies in the Hard- Spread scenario, which has a highly constrained task space. (c) We compare our SVGD-principled exploration strategy with random exploration from seed tasks in Hard-Spread. (d) We perform experiments in Simple-Spread using different numbers of particles to represent Q. (e) We compare entity progression with direct entity transfer in standard Simple-Spread and Push-Ball.
piieetion-sampliig-babeee exploration clearly im- proves the final performance, particularly in Hard- Spread, where purely gradient-based expansion (i.e., (q. can hardly discover feasible novel tasks. It is worth mentioning that these feasibility constraints are shared across all the experiments, so all the baselines. including RCG, GoalGAN, and AMIGo, have access the feasibility check and are able to produce feasible training tasks.
Table 2: Ablation study on the rejection- sampling technique in Task Expansion on Simple-Spread and Hard-Spread with
coverage rate %
w. R.J.
w.o. R.J.
Simple-Spread Hard-Spreat
96.0 + 1.9
94.1 + 4.9
97.2 1.4
Sensitivity of Particle Number: We perform experiments with different numbers particles in SSpele with n = 4 and the results in Fig. Show that we do need sufficient particles to approximate q()) Insufficient particle size (e.g., 400) results in inaccurate estimation and poor performance. The results are robust when particle size is at least 1000. We choose 4000 particles to approximate (()) in all our experiments.
We compare entity progression with naive population curriculum, i.c., directly transfer the trained policy 7 to the next entity size ng (Iransfer). We train with the methods on Simple-Spread from no = 4 to = 8 and on Push-Ball from nn = to N N=44 4. The results are shown in Fig. Fig. In both two tasks, direct transfer yields significant performance drop, which we believe due to the drastic training distribution shift. For entity progression, due to its smooth transition between the continuous relaxation of 2ạ and z 1, produces much stable training curves. Entity progression also yields the much better final performances particularly on the more challenging Push-Ball scenario.
Entity Progression:

---

<1% 4.2 The Hide-and-Seek Environment
We consider two tasks in the hide-and-seck environment (HnS) as shown in Fig. 6 i.e., simplified hide-and- seek game called Ramp-Use Use and multi-agent and sparse- reward variant of the Bock-and-Retue
Unnlocked Box
task from the trans-
Seeker
Locked Box
fer task suite in 12]. Ramp-Use is a single-agent scenario for the ramp use strategy, with ramps, movable seeker and I fixed hider in the quadrant room. We need to train a seeker policy to use the ramp to get into the quadrant room for positive rewards. In Bock-and-Retunn crete parameter n becomes a 2-dimensional vector and each dimension represents the number of agents and boxes respective. Agents need to lock all the boxes and return to their birthplaces. Agents get reward when all boxes are locked and another success reward when the task is finished, i.e., all boxes locked and all agents back to the birthplaces.
Ramp
A
Agent
Wall
Hider
Birthplace
the dis-
Ramp-Use
Mrr--and-Retuun
Figure Tasks in the hide-and-seek en- vironment (HnS).
The results different methods are shown Tab. 3 train each algorithm on Ramp-Use for 25M timesteps and on Back-and-Return with (2, 2] for 180M timesteps. For Lock-and- Retuu with n = (4, 4), we train VACL for a total number 230M timesteps. We report the average success rate over seeds. VACL is the only algorithm able to solve both two tasks.
Table 3: Results of VACL and baselines in HnS tasks.
Uniform
RCG
GoalCAN
AMIGo
VACI
Ramp-Use
15.3% + 16.1%
76.7% + 2.1%
6.4% + 7.5%
40.8% + 29.2%
99.7%
0.5%
2)
5.0%
5.1%
<1%
29h
97.3%
0.1%
Lock-and-Retun
(4,4)
/
97.0%
1.6%
In Lock-and-Retunn start with the easy setting of no (2, 2) and proceed towards N = (4, 4). Note that n a 2-dimensional discrete vector in this task, there multiple paths transit from the continuous relaxation #(2.2) = 1 to #(1.4) 1. One is to double the number of agents first and keep the box number unchanged, and then
Lock-and-RRetum (n=(4, 4))
success rete
0.8
double the number of boxes, i.c., 2n = + 2(4.2) - ZN 1 (agent-first). The other first double the number of boxes and then increase the agent number, i.e., 2no = - 2(2,4) 1 > 2N 1 (box-first). Although both two paths are feasible transitions, they lead to different practical performances. The results are shown in Fig. 7 where we clearly observe that the strategy of increasing agent number first substantially outperforms the box-first strategy. Therefore, we generally suggest increase
0.8
box-first
agent-firat
0.0
0.5 timesteps/xx00)
Figure 7: Results of different entity switching strategies.
number of agents before objects in entity progression. We remark that since no baseline achieves a success rate above 15% even in the easy case of n = (2,2), we do not evaluate them in the hard setting.
Conclusion
In this paper, we propose a novel curriculum learning paradigm following the principle of Stein variational inference and develop an effective algorithm, Varitional Automatic Curriculum Learning (VACL), which solves collection of seware multi-agent cooperative problems. In this work, we consider cooperative tasks for the clear measurement of training progress. We also remark that VACL assumes a monotonic expansion of q(Φ), which only holds in the fully cooperative setting. There are also existing works that study ACL for competitive games [28 16 2, which we leave as future work. Our framework tackles an important research problem in the literature while experiments
10

---

are conducted on open-souree environments with student licences, so we believe our work does not produce any negative impact the society.
Acknowledgements
This work is supported by National Science Foundation of China (No. U20A20334, U19B2019, 61832007), National Key R&D Program of China 56.20019YFF00115005 Tsinghu EE Xilinx AI Research Fund and 2030 Innovation Megaprojects of China (Programme on New Generation 50000. We appreciate Jiantao Qiu and Xuefei Ning
Artificial Intelligence) Grant No. for their reviews of an early draft. We would like thank Chao Yu for her support and input during this project. Yi Wu would also thank Bowen Baker and Ingmar Kanitscheider since some of the very initial ideas are inspired by their early discussion at OpenAL. Finally, we thank all anonymous reviewers for their constructive comments.
References
[1] Ilge Akkaya, Marein Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's 's cube with robot hand. arXiv preprint arXiv:1910.07113. 2019.
[2] Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2020.
[3] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch. Mare Lanctot, Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
[4] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41-48,
2009.
[5] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 091.. 2019.
[6] Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua B. Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with (amiglo: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2021.
[7] Kuan Fang, Yuke Zhu, Silvio Savarese, and Fei-Fei Li. Adaptive procedural task generation for hard-exploration problems. In International Conference on Learning Representations, 2021.
[8] Dieqiao Feng, Carla Gomes, and Bart Selman. Solving hard ai planning instances using curriculum-driven deep reinforcement learning. In Proceedings of the Twenty-Nimhh Interna- lional Joint Conference on A Artificial Intelligence, IJCAI-20, 2020.
[9] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. International conference on machine learning, pages 1515-1528. PMLR, 2018.
[10] Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pages 482-495. PMLR, 2017.
[11) Boris Ivanovic, James Harrison, Apoorva Sharma, Mo Chen, and Marco Pavone. Barc: Back- ward reachability curriculum for robotic reinforcement learning. In 2019 International Confer- ence Robotics and Automation (ICRA), pages 15-21. IEBE, 2019.
[12] Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, and Chelsea Finn. Un- supervised curricula for visual meta-reiinforeement learning. In Advances in Neural Information Processing Systems, 2019.

---

[13] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent coopera- tion. In S. Bengio, H. Wallach. H. Larochelle, K. Grauman. N. Cesa-Biancht and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 7254-7264. Curran Associates, Inc., 2018
[14] Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforcement learning. In CoRL,, 2019.
[15] Pascal Klink, Carlo D' D'Ermo, Jan Peters, and Joni Pajarinen. Self-paced deep reinforcement learning. In NeurlPS, 2020.
[16] Joel Z Leibo, Edward Hughes, Mare Lanctot, and Thore Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00442, 2019.
[17] Qiang Liu and Dilin Wang. Stein variational gradient descent: a general purpose bayesian inference algorithm. In Proceedings of the 30th International Conference Neural Information Processing Systems, pages 2378-2386, 2016.
[18] Qian Long, Zihan Zhou, Abhinav Gupta, Fei Fang, Yi Wu, and Xiaolong Wang. Evolution- ary population curriculum for scaling multi-agent reinforcement learning. International Conference Leurning Representations, 2020.
[19] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compettive environments. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
[20] Tambet Matüisen, Avital Oliver, Taco Cohen, and John Schulman. reacher-student curriculum learning. IEEE transactions on neural networks and learning systems, 2019.
[21] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning. pages 1662-116.. PMLR, 2020.
[22] Steven Morad, Roberto Mecca, Rudra Poudel, Stephan Liwicki, and Roberto Cipolla. Embodied visual navigation with automatic curriculum learning in real environments. IEEE Robotics and Automation Letters, 2021.
[23] Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art. Autonomous agents and multi-agen systems, 11(3):377-434, 2005.
[24] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yve Oudeyer. Teacher algorithms for curriculum learning of deep RL in continuously parameterized enviromments. arXiv preprint 1w1.v:9910.07222 2019.
(25) Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Au- tomatic curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.0460 04664, 2020.
[26] Sebastien Racaniere, Andrew Lampinen, Adam Santoro, David Reichert, Vlad Firoiu, and Timothy Lillicrap. Automated curriculum generation through setter-solver interactions. In International Conference on Learning Representations, 2020.
[27] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. arXiv preprint arXiv:1906.0423 042799 2019.
[28] Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self- In International Conference Learning Representations, 2018.
[29] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782) 2019.
12

---

[30] Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley, Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, pages 9951. PMLR, 2020.
[31] Weixun Wang, Tiampei Yang, Yong Liu, Jianye Hao, Xiaotian Hao, Yujing Hu, Yingfeng Chen, Changjie Fan, and Yang Gao. From few to more: Large-scale dynamic multiagent curriculum learning, 2019.
[32] Jan Wöhlke, Felix Schmitt, and Herke van Hoof. A performance-basee start state curriculum framework for reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 1503-1511, 2020.
[33] Ruihan Yang. Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. In Advances Neural Information Processing Systems, 2020.
[34] Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, 2021.
13


Reviews:
[{"review_id": "mVfWW-Lirdt", "reviewer": "Reviewer_9Js4", "summary": "This paper proposed a variational automatic curriculum learning framework for solving goal-conditioned cooperative multiagent reinforcement learning problems. In this framework, the learning objective is decomposed into two parts: policy learning on current task distribution, and curriculum update to a new task distribution. The curricula are trained based on two components, including task configuration and the number of agents. The experimental results demonstrate the proposed method can achieve better performance than a few baselines with light weight computation resources. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In all, this paper studied an important method for solving cooperative multiple agent sequential decision-making tasks by using curricula learning. The authors tried to develop and analyze the solution method in a rigorous way. The authors also provide some good discussion on the connection to existing work. The method is tested over two environments and shown to outperform a few baselines. However, the motivation and some details related to the claimed contributions are not clearly explained, and the problems settings considered seems restrictive to homogenous agents.  \n\nSpecifically, it is a little unclear to me why Stein variational gradient descent is necessarily better than regular variational inference method. It might be better to provide some more background information. \n\nThere are some technical details not clearly explained. For example, In section 2.2.1, “to update the particles, we consider an incremental transform T(\\phi)=…”. Why it is necessary to introduce this transform? This seems to be one of the main ingredients for deriving the gradient update of task distribution. It might better if some design choices can be justified.\n\nWhat is the number of particles used for performing Stein variational inference? How sensitivity is results to it?\n\nAnother limitation is that the paper only considered homogenous agents and learn a shared goal-conditioned policy. It might worth discussing how the approach can be generalized to heterogenous agents’ cases, since many realworld problems involve collaboration among different types of agents.\n------ after rebuttal------\nI think the author should explicit demonstrate that the method can work on problems with heterogenous agents. Given that the rebuttal has addressed most of my concerns, I would like to increase my score to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author has provided some discussion regarding the limitation of their work. Additional limitations I see for this work are discussed in the Main Review.", "review_text": "", "extra_scores": {"ethical_concerns": "I don't any ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "Z-qmDpsG8-p", "reviewer": "Reviewer_spUJ", "summary": "The paper proposes a curriculum learning method for cooperative multi-agent reinforcement learning in sparse reward tasks. The authors derive a curriculum learning approach by forming a lower bound on the expected reward under a target distribution p(\\phi). This lower bound is maximized by alternatingly optimizing the policy under a variational distribution q(\\phi) and then adjusting the variational distribution based on the current performance of the agent. The authors investigate the theoretical behavior showing that the optimization of q(\\phi) minimizes a form of \"weighted\" KL-Divergence to p(\\phi). Further, they derive a gradient-based approach to optimize q(\\phi) and propose an approximate implementation of their approach that performs empirically well in different multi-agent RL benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \\\nThe paper presents novel and insightful content. I especially like the transition from the derived gradient in Eq (4) to the approximate implementation in Section 3.1. The proposed quantization nicely connects concepts such as success buffers to the lower bound on the agent performance. As such concepts are rarely connected to any theory, this is an exciting variation.\n\nThe experiments demonstrate good performance. In combination with the convincing theoretical motivation, this seems to be a promising approach for curriculum learning in (multi-agent) RL. Further, the ablations in the experimental section give a good impression of the inner workings of the method.\n\nWeaknesses: \\\nThe theoretical motivation for the curriculum does not seem to carry over to the discrete variables describing the number of agents. The introduced entity progression scheme seems detached from the theory. While I understand that the gradient-based operator may not be computationally advisable for large N, why did the authors not subsume the number of agents into the parameter \\phi. While this may not be in line with the continuity assumption about \\phi, it should be fairly straightforward to implement. And with a properly adjusted noise process for exploration, it could yield an implementation more similar to the theoretical motivation. Nonetheless, the overall method seems to work well.\n\nI see another weakness in the performance comparison in the experimental section. Apart from issues with clarity (see next section), the comparison methods did not use a curriculum over the number of agents. While I understand that there may be no related method ideally suited (because of the lack of a curriculum over both goal-state and agent number), I think one ablation should be added to the experiments: VACL without the curriculum over the number of agents. This would give an additional insight into the importance of the curriculum over agents and also yield a fairer comparison to the other curriculum learning methods like GoalGAN, RCG and AMIGo. \n\nIn the \"SVGD-Principled Update\" experiment, the (principled) gradient-based version from Eq (6) was not compared to the rejection sampling method used in Algorithm 1. Instead, only a version without the rejection step has been used. Why is there no such comparison in the hard-spread scenario? I think it would give a good intuition of whether the rejection sampling used in VACL performs similarly to the gradient-based method.\n\nClarity: \\\nThe paper is well written, and the concepts were understandable. However, some parts of the paper could be more concise, especially w.r.t. experimental details:\n - The rejection sampling makes use of an \"oracle\" that can identify infeasible regions. However, the infeasible regions are not specified in the experimental section except for hard-spread, where (as already mentioned) rejection sampling is not evaluated. The authors should clearly state whether additional knowledge was gained through the aforementionend oracle in the other environments. This would allow readers to judge whether VACL had an advantage over other methods like GoalGAN due to the access to the oracle.\n - Was GoalGAN also given access to the initial samples obtained by GetEasy(n)? I think it should be because it explicitly supports such seeding by initial tasks. I am curious because, in line 270, the authors mention that \"GoalGAN and AMIGo takes an extremely long time to train a good neural goal-generator.\"\n - In Algorithm 1: A reference to the implementation of the \"Sample\" function in the appendix would be good to guide the reader.\n - How were the continuous goals \\phi defined in the individual experiments? Unfortunately, I could not find any details on this. Especially in the hide-and-seek environment, this would be interesting. Further, what seed tasks are generated by the GetEasy function in the environments?\n - The discussion of the \"Choice of seed task\" in Section 4.1.2 seems to have an error. The text states that \"Exp.Act.w.eval\" works better than \"Exp.Act.\" while the plots indicate the opposite.\n\nI think the aforementioned points are another weakness of the paper, as some of those make the obtained results a bit less convincing.\n\nRelated Work: \\\nI have two suggestions for improvement for the related work:\n - The reference to the work by Wöhlke et al. (citation [30] in the main paper) seems to be presented a bit misleading in the related work section. Is it maybe because of a mix-up in the citations? While the authors say that \"[30] propose to meta-learn f^*(\\phi) to more effectively expand the curriculum distribution\", it instead seems that [30] use a different characterization of the \"active set\" as the authors proposed here. In [30], the active set is defined as those parts of the task space in which the value function exhibits the largest gradient w.r.t. the task variable.\n - There exists work [1,2] that proposes a similar curriculum scheme and links it to a probabilistic inference perspective. It also minimizes the KL-Divergence between q(\\phi) and p(\\phi). While I don't think that this existing work hinders the novelty and originality of the proposed approach due to the different methodology, I think it should be included in the related work section. In combination with the corrected description of reference [30], this would yield a better overview of methods based on value functions and/or probabilistic inference.\n\n[1] Klink, Pascal, et al. \"Self-Paced Deep Reinforcement Learning.\" Neural Information Processing Systems, 2020.\n\n[2] Klink, Pascal, et al. \"Self-paced Contextual Reinforcement Learning.\" Conference on Robot Learning, 2019.\n\n\nUPDATE:\n\nAfter reading the authors responses, I increased my score from 5 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitations and societal impact of their work in the conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "JT9tBH3kfGZ", "reviewer": "Reviewer_TiSD", "summary": "This paper introduces a curriculum learning method “VACL” to solve the multi-agents RL problems. The main idea of this paper is to train RL agents on an easy-to-hard task sequence based on an auto-generated curriculum.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of this paper is similar to some other curriculum RL methods, such as GoalGAN, setter and solver (https://arxiv.org/abs/1909.12892 ) and AMIGo. Due to the results of the experiments, this method achieves a great improvement on all previous work in a hide-and-seek environment. Meanwhile, VACL also achieves good performance on other tasks. \n \nOriginality: The main idea of this paper is similar to the GoalGAN and AMIGo. However, the curriculum generator is not as tricky as the mentioned methods to learn. The author uses a smooth function to represent the learning progress of a specific task instead, which achieves a higher sample efficiency. The essential part of this method is to use an incremental transform \\T to simplify q(\\phi). Due to the results, it’s obvious that this transform is available.\n \nClarity: I think this paper is written well. The expansion of the theoretical part is very detailed and logically fluent. The details of the experiments seem sufficient. \n \nHowever, here are some questions and suggestions for this paper: \n \nQuestions:\n1. In algorithm 1. \\geteasy(n_{0}). How to define easy tasks for agents? Are these easy tasks picked by humans? What is the difference of two \\geteasy used in \\geteasy(n_0) and \\geteasy(n_k+1)?\n2. Section 3.4 is not sufficient enough. Considering that the main idea of VACL is to sample an easy-to-hard curriculum for agents to learn, and also the evaluation of the difficulty of tasks is also based on the V. Why not add the method mentioned in section 4.1 to section 3.4? It causes me so puzzled when I first read section 4.1.\n3. Does the forgetting phenomenon occur in the experiments? If yes, how to solve it? I did not find any restrictions or guarantees if the agent forgets some tasks that have been resolved.\n4. For empirical, I think V cannot fully express the value of learning on the current task. Maybe you can consider another metric to move tasks into Q_{act}, such as \\delta V. Of course, this is just a possible optimization. \n5. How to ensure the upper and lower bounds of \"f\" used in eq.4?\n \nOverall, the paper offers a possible solution for evaluating the difficulty of tasks. \n\n=== Post Rebuttal ===\n\nThe paper studies an interesting problem. Its claims are well supported from both theoretical and experimental perspectives.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduces a new curriculum learning algorithm for cooperative multi-agent reinforcement learning. All reviewers appreciated the method's novelty and that the paper was well-written. All found the theoretical motivation convincing and insightful. There were some initial concerns about the experimental results raised by reviewer spUJ, but they found the author's response convincing and thus increased their score.
Decision: Accept (Poster)