It Takes Four to Tango: Multiagent Self Play for Automatic Curriculum Generation
Authors: Yuqing Du
Pieter Abbeel
Aditya Grover
Venue: ICLR 2022 Poster (2022)
Keywords: curriculum generation
unsupervised reinforcement learning
goal conditioned reinforcement learning
multi agent

Abstract: We are interested in training general-purpose reinforcement learning agents that can solve a wide variety of goals. Training such agents efficiently requires automatic generation of a goal curriculum. This is challenging as it requires (a) exploring goals of increasing difficulty, while ensuring that the agent (b) is exposed to a diverse set of goals in a sample efficient manner and (c) does not catastrophically forget previously solved goals. We propose Curriculum Self Play (CuSP), an automated goal generation framework that seeks to satisfy these desiderata by virtue of a multi-player game with 4 agents. We extend the asymmetric curricula learning in PAIRED (Dennis et al., 2020) to a symmetrized game that carefully balances cooperation and competition between two off-policy student learners and two regret-maximizing teachers. CuSP additionally introduces entropic goal coverage and accounts for the non-stationary nature of the students, allowing us to automatically induce a curriculum that balances progressive exploration with anti-catastrophic exploitation. We demonstrate that our method succeeds at generating an effective curricula of goals for a range of control tasks, outperforming other methods at zero-shot test-time generalization to novel out-of-distribution goals.

Published as a conference paper at ICLR 2022
IT TAKES FOUR TO TANGO: MULTIAGENT SELFPLAY FOR AUTOMATIC CURRICULUM GENERATION
Yuging Du
Pieter Abbeel
Aditya Grover UCLA adi tyageas.us.... www...lleee..
UC Berkeley
UC Berkeley
yuqi ng du@berkeley) leeeerreeeyyed.. edu
pabbee1 @berkel eyyeed
ABSTRACT
We are interested in training general-purpose reinforcement learning agents that can solve a wide variety of goals. Training such agents efficiently requires auto- matic generation of goal curriculum. This is challenging ploring goals of increasing difficulty, while ensuring that the agent (b) is exposed diverse set of goals in sample efficient manner and (c) does not catastroph- ically forget previously solved goals. We propose Curriculum Self Play (CuSP), automated goal generation framework that seeks to satisfy these desiderata by virtue of a multi-player game with 4 agents. We extend the asymmetric curricula learning in PAIRED (Dennis et al., 2020) to symmetrized game that carefully balances cooperation and competition between two off- off-policy student learners and two regret-maximizing teachers. CuSP additionally introduces entropic goal cov- erage and accounts for the non-stationary nature of the students, allowing us automatically induce curriculum that balances progressive exploration with anti- catastrophic exploitation. We demonstrate that our method succeeds at generating effective curricula of goals for a range of control tasks, outperforming other methods at zero-shot test-time generalization to novel ofi-t--diitiiuutiin goals.
it requires (a) ex-
INTRODUCTION
Reinforcement learning (RL) has seen great success in a range of applications, from game-playing (Silver al., 2016; Schrittwieser al., 2020) real world roboties tasks (OpenAl et al.) 2019). However, these accomplishments are often in the realm of mastering a single specific task. For many applications, we further desire agents who are capable of generalizing to variety of tasks. This can achieved via goal-conditioned RI, where an RI. agent is trained to solve many goals by specifying sequence of goals, i.e. a goal curriculum. Manually designing such curricula is tedious and moreover, does not scale due to being environment specific and heuristic-driven. Even when it is easy to engineer curriculum, learner's i current capabilities. For example, effective human tutors use an understanding of their pupil's current knowledge and learning trajectory to create curriculum.
would prefer curricula that can dynamically adapt to the
Our work builds on two broad desiderata for curriculum generation (Bengio al., 2009) that enable sample-efficiett learning of goal- conditindt policies: progressive exploration and anti catiicatastrophic exploitation. Progressive exploration refers curricula where the agent gradually learns solve tasks of increasing difficulty and diversity. Anti-catastrophic exploitation ensures that the agent periodically resolves previously attempted goals that could become harder to solve during the course of training a.k.a. catastrophic forgetting (French 1999. Kirkpatrick et al., 2017).
Many prior works have attempted incorporate aspects of these criteria for automatic curriculum generation. One approach is to parameterize a goal generator and pit against a discriminator that assesses goal difficulty (Florensa al.) 2018) or feasibility (Racaniere al. 2020 for the current agent. Racaniere et al.] (2020) also address desired qualities validity and diversity by crafting specific losses for these characteristics. However, these approaches rely on learning accurate dis- criminator for assessing goal difficulty, which can be challenging. Rather than employing discrim- inators in the standard adversarial minimax formulation, Dennis al. (2020) proposes an alternate minimax objective based on the agent's 's regret. The 'expert' used for computing the regret is another goal-conditioned policy whose utility is maximized by an environment generator.
Code available wathtps:://eo /ttttpp://gihubb. wwm. hgg////gig//////uuuuuus ngd/ccup

---

Published as a conference paper at ICLR 2022
propose g
Ra
RRB
GA
GB
R
R
GA
A
B
RA
A
RA
A
B
(a) Self Play
(b) Asymmetric Self Play
(c) Curriculum Self Play
Figure I: Interactions between learners A, B and teachers G., GB in Self Play variations. In (a), A plays against variations of itself in zero-sum game automatically generate curriculum. In (b), A acts as both an agent and a demonstrator by being rewarded for proposing hard goals to B. In (c), we separate the goal generation from the demonstrators and symmetrize the whole system.
An alternative method for automatic curriculum generation that has seen great success is self-play where an agent plays against other versions of itself (Silver et al. 2016 Bansal et al. 2018 Baker et al., 2019) (Fig. [.). Unfortunately, this method does not apply directly to complex tasks where we do not have the structure of a two player, zero-sum game. promising variation Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2018b), which involves minimax game between two similar agents: goal demonstrator, Alice, and the desired goal-conditioned agent, Bob (Fig. [p). Bob is asked to solve or reverse tasks demonstrated by Alice in the same environment. To motivate Alice to propose successively challenging goals, Alice rewarded whenever Bob fails. This method inherently captures an easy-to-hard curriculum, since any goal solved by Alice is achievable by Bob in principle. However, ASP tends to be sample inefficient as bottlenecked by Alice's 's ability to remember earlier goals and explore diverse goal space.
Our work aims bring together the strengths of goal generative methods and ASP while addressing the two desiderata. Our work focuses on the multi-goal setting with the curriculum generator con- trolling goals Portelaa et al. (2020). We propose Curriculun Self Play (CuSP), symmetric multi- agent setup for automatic curricula generation. CuSP optimizes for two off offlcollcy goal-condition students, Alice and Bob, and two regret-maxee maximmziaiii goal generators, GA and G (Fig. [Fi..Te). Draw- ing inspiration from the minimax regret objective in PAIRED (Dennis et al., 2020), we designate each goal generator being 'friendly' to one of the students (G'A for Alice, GB for Bob), and opti- mize each goal generator using the difference utility between the preferred student and the other student (i.e., GA is rewarded RA R where RA, RB are Alice's fice's and Bob's returns respectively when challenged with the same goal). This encourages proposing goals that are neither too easy nor too hard. With our symmetrized setup, Alice and Bob each have a 'friendly' goal generator that is inclined to propose more feasible goals without being too simple, and an 'unfriendly' goal gener- ator that is inclined to propose more challenging goals. As the tasks increase in difficulty overall, the former's cooperation helps in scaffolding learning while the latter's 's competition helps push the boundaries of the agent's capabilities.
We further address goal diversity and sample efficiency by leveraging the off policy entropy- regularized optimization of Soft Actor Critic (SAC) (Haarnoja et al. 2018) for training the goal generators. The goal generators' replay buffers track the regrets for all proposed goals. To account for non-stationary learners, we propose an update rule for the regrets in the goal generators' replay buffers based on the critic networks of Alice and Bob. In doing so, constantly keep track of the learners' current abilities and can propose increasingly challenging goals. At the same time, using the regret objective also places the current abilities in context. For example, if Bob has forgotten how to achieve a particular goal, the regret RA R increases and GA is more inclined to re-propose that goal. Thus the combination of using a replay buffer and the regret objective aids in addressing the twin goals of progressive exploration and catastrotttc exploitation.
Compared to prior methods for goal generation, we demonstrate that CuSP is able to satisfy the desiderata for good curricula. We show that this translates improved sample efficiency and gen- eralization to novel out-of-distribution goals across a range of robotics tasks, encompassing naviga- tion, manipulation, and locomotion. We highlight some emergent skills that arise via CuSP when generalizing to particularly challenging goals, and find that our method is able to handle cases when portions of the goal space infeasible.
2

---

Published as a conference paper at ICLR 2022
2 BACKGROUND
Problem Setup. We model the environment and the goal-conditioned learner using a Markov decision process. M (S, A, R. 7,7), where S is the state space, A is the action space, T Ax S - R is the environment transition function, R: G x S + R is the goal-conditioned reward function, G S S is the goal space, and y is discount factor. We consider both dense and sparse reward functions: r(s[g)= = -d(s, g). where d is a distance metric (e.g. L2 norm) between the state s and the goal g for the former, or r(slg) = [[((s, g) < E) for the latter. Our objective is to learn a policy π [ares St: g) that maximizes the success rate of reaching all goals g E G. To assess ability to generalize to out-of-distribution goals, we split the goal space into Gid and Good where the goal generators can only propose goals g E Gud. and we evaluate on sampled g Goodd
Self Play (SP). SP is an approach for automatic curriculum generation in competitive, zero-sum games. In this paradigm, an agent plays against past versions of itself in order to automatically generate a learning curriculum (Figure [b). That is, we consider Alice and Bob as two versions of the same underlying agent, and each agent's objective is the negative of the others', so Rt RB. However, this approach is only applicable for symmetric zero-sum games, and does not lend itself to learning curricula for goal-conditioned agents in other setups.
Asymmetric Self Play (ASP). ASP (Sukhbaatar et al., 2018b) extends self play to non-zero sum games by proposing an asymmetric parametrization of two competing agents. During each episode, Alice's objective is to propose and demonstrate a task for Bob complete (Figure [bb). In the goal- conditioned paradigm, we take Alice's 's final state s as the proposed goal g. Alice and Bob now have different objectives: Bob's return, R = Ee 7fr(sfllrr [1)), is simply based on reaching the proposed goal, while Alice's reward is based on Bob's inability to reach the goal, RA = 1 (d(s g) 2 c). While this approach lends itself well to more complex tasks such as robotic manipulation (OpenAT eet al. (021), it has poor sample efficiency. Since we rely on a learning agent, Alice, to propose and solve goals, the generated goals are limited by Alice's own ability to explore and learn challenging yet feasible goals. This can be particularly problematic towards the start of training, where we can waste many episodes proposing simple and redundant goals for Bob.
AUTOMATIC CURRICULUM GENERATION VIA CUSP
Our curriculum generation method consists of four main players: two goal-conditioned learning peers, Alice _ and Bob TB. and two goal generators, G and GB. T TA(a|s, g) and x xB(a|s, are both optimized to maximize the discounted sum of goal-conditioned rewards R(y) =e -'.''' 2'r(()), (g), across proposed goals g E Gid- Aside from any initial randomness in initializing TΛ and æB. the two policies are parameterized with the same architecture. Ga(glso, 2) and (n(gle so, 2) are parametrized as policies that output a goal g E Gid given an initial observation of the environment so and latent noise variable 2 ~ (0, ). The inclusion of such noise variable draws from generative models (e.g. GANs, VAEs) where such z is the input to the generator, and is also found in prior goal generative methods (Florensa et al. 2018). The goal generators are optimized to maximize the regret of their corresponding agent: A R (g)
R (g) and R G R (g) R
We refer to our overall curriculum generation approach as Curriculum Self Play (CuSP). We present an overview of CuSP in Algorithm Jand illustrate it in Figure D. The algorithm proceeds in rounds. In every round, both goal generators propose a goal, 9A from G and 98 from GB. To keep the symmetrization fair and hopefully scaffold learning initially, we first rollout the 'casier' goal for each respective agent (ic. 9A for Alice, 9B for Bob) and compute the corresponding environment returns and update the learners. Next, we rollout the 'harder' goal for each respective agent (ie. gB for Alice, 9A for Bob) and update again. At the end of the round, we update the goal generators.
Symmetrization. The regret objective Rd - RB helps G. propose goals that are challenging for Bob while still being feasible for Alice. However, this can be detrimental when Alice and Bob's performance diverge too quickly. To address this, we propose symmetrizing the multi-ageni system by introducing a corresponding goal generator GB whose regret objective is R RA, essentially pitting it in zero-sum game against G.. Having these two goal generators produces two types of goals for each agent: a) feasible yet not too simple goal from the 'friendly' goal generator (GA for Alice, GB for Bob), which can help scaffold learning quickly, and b) a challenging yet not too difficult goal from the 'unfriendly' goal generator (G for Bob, G for Alice), which can help push
3

---

Published as a conference paper at ICLR 2022
Algorithm 1: Curriculum Self Play (CuSP)
Initialize goal-conditioned learners TA, TB. goal generators GA, GB:
while not converged do
// generate goals
9a G 9b GB:
/1 evaluate easier goals first
Easy f = Easy +- ro
// update policies
nA + Irain.lernnernzz 1 Eusy); TH + train train.learnernnt Bayy)
11 evaluate harder goals second
R t (ol)) R t 9.)))
// update policies
T - train learner(TA: R R TB train learner(5B. RB
Hard):
// compute regrets and update generators
G + train.generator {[R Easy
R Hard)) 9A1, {
R Easy))
Hard [Rrd R R 9A} }
Gg train-generator (GB, E Rand)) 9B), {RR
end
(a) Adam
(b) PPO
(c) SAC
Figure 2: Performance of different optimization methods toy landscape that is primarily flat except for a single gaussian centered at (-0.1, 0.1) shown by the contours. Red shows most recently proposed points. SAC is able to find the peak while the other methods get stuck in the flat regions.
the boundaries of the agent's abilities. Both of these properties help encourage the generation of goals that are feasible yet progressively challenging.
Goal Generator Design. We motivate our goal generator design with our proposed desiderata: progressive exploration with anti-catastrophic exploitation. Although the goal generator is not an RL agent in the traditional sense only 'acts' a single timestep by proposing a goal and its "observation' only changes based on random noise variable optimizing such explore-exploit tradcoff effectively naturally inspires the use of an RL objective, and RL objectives have previously been used for one-step optimization (e.g., one-step Q-learning (Watkins & Dayan 1992)) )
Thus, train the goal generators we use an RL learning objective and parameterization inspired by single-time step variant of Soft-Actor Critic (SAC) (Haarnoja et al. 2018). SAC is an off policy algorithm that optimizes for trade-off between expected reward and policy entropy H(x) with
(a) SAC without regret updates at 2500 and 5000 steps. (b) SAC with regret updates at 2500 and 5000 steps.
Figure 3: SAC optimizing non-stationary toy landscape that primarily flat, but has gaussian peak starting at 1- 0.1) and moving diagonally to (0.1, -0.1). Red shows most recently proposed points. Without the regret updates, the optimizer lags and cannot find the optimum.
4

---

Published as a conference paper at ICLR 2022
coefficient a. In our setup, our goal generator only acts to propose goal per episode, each "trajectory" is only a single time step consisting the initial state (for which we concatenate the initial environment observation we get the corresponding "reward" for the goal generator policy by computing the regret between and on g. Thus the objective for each goal generator policy G' is given by
with latent noise variable =), the proposed goal action y. and
max g C
D (s.g) ta H(G)
exploitation
exploration
where the regret term addresses catastraatt exploitation by prioritizing goals where regret has increased, and the entropy term addresses progressive goal space exploration.
We equip the SAC goal generators with a replay buffer to store all past goals and their regrets. This helps address forgetting and improve the sample efficiency of goal generation compared to prior methods which optimize the generator as another agent using on-policy algorithms such as PPO (Schulman et al. 2017). We further take advantage of SAC's entropy regularization to progressively increase the diversity of the goals, with consideration of Alice and Bob's capabilities using the stored regrets. To empirically motivate the use of SAC, we compare SAC against PPO and Adam (Kingma & Ba 2015) in optimizing a synthetic landscape. One challenge of the regret objective is that the beginning of training, the landscape will mostly be flat when тA and TB are similarly we find that only SAC is proficient at optimizing a primarily flat landscape.
unskilled. In Fig. 2
Dynamic Regret Updates for Non-stationary Agents. However, using off policy RL algorithm for optimizing such a goal generator requires further modifications. For more typical agents, the stationary property of the euvironment. In our case this is no longer true Alice and Bob are continuously learning and part of the multi-agent environment, so the regrets for the same goal will also change over time. To address this issue, would like update the non-stationary regret values in the goal generator's replay buffer. However, it is expensive for Alice and Bob to replay all the goals in the environment. As a sample efficient trick, we leverage the observation that the critics Alice and Bob can provide a current estimate of their performance. Hence, after every round, we update the regrets of previous goals in the replay buffer using the difference between the corresponding value estimates from the critic networks of Alice and Bob. This allows us to keep updated estimate of feasibility based on Alice and Bob's 's current skills.
received rewards
To empirically validate that updating the replay buffer regrets helps in non-stationary landscape, we carry out investigations of SAC goal generator both with and without regret updates in Fig. 3 As hypothesized, updating the replay buffer allows the generator to find the moving optimum. The combination of the replay buffer and dynamic regret updates also strengthens our method by automatically prioritizing goals that have been forgotten. With the non-stationary regret updates, the goal generators always have an updated estimate of each learner's 's abilities. Thus Bob performs past goal, the regret increases for G and it is more likely to repropose it, addressing catastrophic forgetting for Bob. Likewise, symmetrization helps to address catastrophic forgetting for Alice. Without the corresponding GB. if Alice performs worse at goal, the regret A R decreases and A less likely to repropose unless Bob's performance decreases even further. This can potentially lead to neglecting to propose these challenging goals for Bob.
Summary. Inspired by Racaniere et al. (2020), we specify desired qualities of goal generator:
Progressive Feasibility the goals should be incrementally more difficult, accounting for the agent's 's current skill level and the fact that the agent is non-stationary learner. Progressive Diversity the goals should encourage increasing exploration of goals. Anti-Forgetting if previous goals are forgotten, they should be reproposed.
We summarize the contributions of each component of our method to desired qualities of feasibility, diversity, and anti-forgetting in Table [] We visualize generated goal distributions in Appendix A and ablate over each component of Table I Appendix BB5. to illustrate their importance.
4 EXPERIMENTS
Environments. We test our method across a suite of continuous control tasks adapted from the Deepmind Control Suite (Tassa al. 2020) and OpenAl Gym (Plappert al. 2018), using the
5

---

Published as a conference paper at ICLR 2022
Table 1: Contribution of each component of our method the desired qualities.
Progressive Feasibility
Progressive Diversity
Anti-Forgetting
Entropy Regularization
Regret Replay Buffer
Dynamic Regret Updates
Symmetrization
Point Mass Obstacle
Walker
Reach
Toss
Pick and Reach
G
Good
Figure 4: Illustration of the five environments, Gid showing training goal space and Good showing test goal space. The red points indicate what is used evaluate whether a goal was reached in each environment the point mass coordinates for Point Mass Obstacle, the torso center location for Walker, the ball location for Reach/Toss, and the block location for Pick and Reach.
MuJoCo simulator (Todorov et al. 2012 We analyze method across variety of task types: navigation, locomotion, and manipulation. Point Mass Obstacle navigation task with some walls. Walker is a locomotion task with goal torso position. Reach and Toss manipulation tasks with a goal ball position. Toss provides an interesting challenge as gravity plays a large role the robot must learn to aim precisely. Pick and Reach (OpenAl 2020) is manipulation task where success is measured by picking up block and moving to target position. Environments and goal spaces are visualized in Figure with details in Appendix B.2
Training Setup. For CuSP, we train the goal generators using the SAC implementation from Yarats & Kostrikov (2020) with their default architecture and byperparameters. Since symmetrization leads similar performance from Alice and Bob in expectation, we report the results from Bob only for consistency. We compare against a domain randomization Tobin et al.] (2017) baseline and two other baselines that minimax formulations for curriculum generation: the GoalGAN approach (Flo- rensa et al. 2018) and ASP+BC (OpenAl et al.) (021). We train all learner policies with the same architecture and SAC to isolate differences the goal generation method.
We aim to answer the following questions with our experiments:
Does the CuSP curricula induce agents that can generalize to novel goals from Good?
Does the CuSP curricula induce interesting emergent skills for solving harder goals?
Does the CuSP curricula handle cases where portions of the goal space are poorly specified (e.g. with impossible dimensions)?
4.1 TEST-TIME
GENERALIZATION
To assess whether our curriculum improves generalizability, we evaluate each method on its ability to reach randomly sampled out-of-distribution goals g Good- In Fig. 5 we see that across en- vironments, generally CuSP matches or outperforms all the baselines for task success. ASP+BC's sample inefficiency due to Alice's 's limitations leads to poor performance, while GoalGAN does well environments where the goal space more symmetrical (e.g. with navigation or the reaching tasks, where most points in the goal space are similarly difficult). That said, we note GoalGAN struggles more in environments where portions of the goal space more challenging (e.g. walk- ing to the right or aiming and tossing upwards). We hypothesize this due to the challenges of accurately learning the feasibility of different goals in asymmetric goal space. Lastly, find

---

Published as a conference paper at ICLR 2022
CUSP (Duns)
CeaGGNN
Domain Randomizaiion
Point Mass Obstacle
Walker
Reach
Toss
Pick and Reach
Success Rate
22
Rounds (x (x100)
Figure 5: Success rates across environments on random goals sampled from Good Results averaged across seeds for each method. Each round corresponds goal generation, then rollouts, then the respective goal generator updates. Generally we find our method matches or outperforms all baselines in out-of-distribution generalization, especially when tackling the harder locomotion task.
CuSP (Oun)
ASP+BC
....
GealGAN
Domain Randomization
Behind Obstacles
Walker Far Right
Toss Upwards
Success Rate
10-
1.-
10
0.6
0.6
0.6
0.4
10
30
20
40
50
0.0
30
70
30
40
53
0.0
10
70
50
Rounds (x 100)
Figure 6: Success rates for three more challenging goals in Point Mass Obstacle, Walker, and Ma- nipulator, and corresponding sample trajectories. faster than other methods, in Walker we learn to run quickly (albeit by using its torso) whereas other methods fail succeed all, and in Toss we learn to aim upwards accurately.
Point Mass we learn to navigate around walls
that the domain randomization baseline is particularly well suited for this OOD random evaluation.
The baseline is trained on good coverage of the in-distribution goal space, which is beneficial for
generalization to random goals in the navigation and manipulation tasks, but is less helpful for the more asymmetric goal spaces like Walker and Toss. For ablations on ASP, see Appendix [B5.22 We also ablate over each contribution of our method as detailed in Table I in Appendix B Specif- ically, we ablate the regret replay buffer, the dynamic regret updates, entropy regularization, and symmetrization. We find that only using the regret objective from PAIRED (Dennis et al. 2020) is insufficient, even with naively applying symmetrization, and that both entropy regularization and an updated replay buffer are crucial for high evaluation task success.
4.2 TASK-SPECIFE SKILLS
Next, we examine if CuSP can induce emergent skills in learners for tackling harder, environment specific goals. Below, we enlist representative examples in the more asymmetric environments where certain goals are clearly more challenging to solve. As seen in Fig. [ our method able to improve upon success rates on these goals.
Point Mass Obstacle Behind Obstacles: requires learning to move around the walls to the edge of the training space. Here, we learn to succeed much faster than the other methods. Walker - Far Right: requires learning to move forward quickly to reach a goal far out of the training space. Here, learn succeed the task where all other baselines fail, sug- gesting that our method produces a useful curriculum for learning challenging locomotion tasks while the others cannot, or require many more samples to do so.
Toss - Upwards: requires learning to toss the ball accurately upwards outside of the training space. Here, learn to succeed much faster than the other methods.
7

---

Published as a conference paper at ICLR 2022
Reach Ard
Reach Good
1.3
1.0
Success Rate
0.3
0.8-
Cuse Eaen
0.5
0.5
Donon Tondorienion
0.4
0.4
0.2
0.2
0.0
10
20
30
0.0
30
Rounds (x100)
Figure 7: Success rates for Reach with a third impossible dimension added to the specified goal space. We test on both in distribution and out of distribution randomly sampled goals in the 3D goal space, where the third dimension affected while the baselines generally performs worse with the higher dimensional space.
feasible in the environment. Generally we find CuSP is least
4.3 MISSPECIFIED GOAL DIMENSIONS
Lastly, we investigate if the goal generator methods (CuSP, GoalGAN, and DR) can handle the case where parts of the specified goal space is misspecified. To do this, we append extra dimension to the goal space, bounded between [-1, 1]. and evaluate performance on Gid and Good with randomly sampling the third dimension during evaluation. We use Reach
case study, and as seen in Fig. 7 agents trained using CuSP's s goal curricula are less affected by the increased goal space while the baselines plateau at lower success rate. In particular, the DR baseline is most adversely affected by the higher dimensionality. This suggests that while random sampling provides good coverage and is amenable to more simple goal spaces, seen above, the induced curricula degrades in quality with increasingly complex goal spaces. Note that we exclude ASP from these experiments as it does not use pre-specified goal space and only sets achieved states goals.
5 RELATED WORK
We summarize four main thrusts of work within automatic curriculum generation.
Modeling Goal Distributions. Many previous works have tackled the problem of generating "ap- propriately" difficult goals for an autocurricllum. In discrete environments, (Campero et al. (202I) propose an adversarial goal-generator that is rewarded when the learning agent achieves the goal with thresholded level of effort. However, they rely on heuristic time limit hyperparameter to de- termine 'suitable effort' (i.e. goal feasibility) and hyperparameters for the teacher reward, whereas our approach directly uses regrets from multiagent interactions to automatically gauge feasibility and provide a reward signal to the teacher. In continuous domains, IZhang et al. (2020) samples goals that maximize the uncertainty of the learner's Q-function, and Nair al. (2018) learn latent representation to sample 'imagined' goals from. GoalGAN (Florensa et al.) 2018) trains GAN to generate goals of intermediate difficulty for the agent, using discriminator to predict whether the generated goals would be appropriate by thresholding success rates. These methods use a heuristic measure of challenging goals, while we leverage the multi-agent setting automatically find in- teresting goals. Similarly to GoalGAN, the Setter-Solver paradigm (Racaniere et al. (2020) trains goal setter whose objectives discriminator for predicting goal feasibility. In contrast, our approach does not need to explicitly weigh the contributions of each loss or incorporate discriminators.
a balance of goal validity, feasibility, and coverage, well
Multi-Agent Curriculum Learning. The idea that interactions between multiple agents can lead to emergent curricula has been proposed in prior works (Leibo et al., 2019 Baker et al., 2019) and relates to the theory of mind (Rabinowitz al.) 2018 Grover et al. 2018a b) postulating human learning via social interactions. One parametrization of such interactions is through designating another agent(s) as a teacher who aims to select an appropriate curriculum for the student. For task- based curricula, Matüisnn et al. (2019) train a teacher policy for selecting sub-tasks for the student, and Graves al. (2017) Portelas al. 2019 treat task selection as a bandit problem for the teacher.
Rather than having an explicit teacher, Asymmetric Self-Play (ASP) (Sukhbaatar et al. 2018h) auto- matically generates curriculum for exploration by training a similarly parametrized agent, Alice, to
8

---

Published as a conference paper at ICLR 2022
demonstrate tasks that the learning agent, Bob, struggles to complete. OpenAl al. (2021) extend ASP to robotic manipulation tasks by incorporating an imitation learning loss for Bob to imitate Alice on challenging goals. As Alice is no longer setting the goals, we do not make use of be- haviour cloning in CuSP. While they demonstrate zero-shot generalization to diverse set of target tasks, relying on Alice to set the goals can also be incredibly sample inefficient, as we saw in our evaluations. [Sodhani & Pahuja (2018) propose a memory- memory-augmmntted version of ASP by providing the learners an external memory to speed up exploration, which is analogous to our method training with off- off-policy Iearners. (Sukhbaatar al. (20T8a) extends ASP to a hierarchical learning framework by learning subgoal representations. Competitive Experience Replay (Liu et al. 2018) generates an exploratory competitive curriculum through penalizing Alice if visiting states Bob has visited, but rewards Bob for visiting states found by Alice.
Environment Design. Another method for inducing curricula is through generating sequence of environments to learn in. Procedural level generation is a common approach for developing games, and can lead to increased generalizability (Justesen et al. 2018). POET (Wang et al. 2019) uses a population of adversaries to generate increasingly challenging environments for a population of learning agents. PAIRED (Dennis et al. (2200) proposes optimizing adversarial environment designer through minimax regret between two learning agents, inspiring the objective we use for our goal generators. However, this approach does not account for catastrophic forgetting and can suffer from sample inefficiencies from using an on-policy method and in cases where the regret landscape is flat. That said, environment design approaches can be used in complementary way with our approach, extending it beyond goal generation.
Skill Discovery. A closely related method for curriculum generation is through unsupervised skill discovery, where an agent explores an environment to automatically discover skills that can be com- posed to achieve a variety of tasks. Prior methods have used information theoretic objectives to encourage skill acquisitiin that covers a diverse set of behaviours (Eysenbach et al. 2018) or gener- ate a multi-task distribution (Gupta et al. 2020). For robotics tasks, (Hausman et al. 2018) propose method for learning latent skill space that can be composed and is transferable between tasks,
DADS (Sharma et al. 2019) proposes a combination of model- based and model-free RL that opti- mizes for discovering predictable skills that can be composed by planner, and Play-LMP (Lynch) ctt al., 2020) proposes using self-supervision across play data for skills discovery. Another method for curriculum generation is through using hindsight (Andrychowicz et al. [017] Li et al.) 2020) These works are complementary to ours as our method aims to induce a curriculum through sparse goal generation, which can be augmented with unsupervised skill discovery.
6 CONCLUSION
We propose a new method, CuSP, for automatic curriculum generation based on symmetric ex- tension of self play approaches for non zero-sum games. Our proposed method aims to address weaknesses of prior approaches by being designed to specifically consider a balance of progressive exploration and anti-catastrophic exploitation. To this end, we make use of two off-policy students and two off-policy regret-maximizing goal generators, each of which is more 'friendly' to one of the students. As result, we generate goals that are a) challenging yet feasible, b) diverse, and c) revisited mitigate anti-forgetting. We qualitatively highlight these desired qualities in goal genera- tion compared to prior methods, demonstrate quantitative gains at generalizing to out-of-distribution unseen goals, and find some emergent task-specific skills.
Limitations and Future Work. While our method shows promise for addressing key components of generating useful curriculum, there is still significant room for improvement. One limitation of our current approach is that it depends on specifying a goal space for the generator to act in, which was not necessary in the original ASP method. That said, specifying a goal space also allows us to imbue the system with human priors for what goals should or should not be valid, whereas ASP may need host-hoc goal filtering if Alice ends up in an undesirable state.
To further tackle automatic curriculum generation, future work should explore scenarios where goal spaces are not fully pre-specified, as well as extending to high dimensional goal spaces (e.g. images). Furthermore, another interesting avenue would be extrapolating CuSP larger number of student and teacher agents, which can potentially provide benefits through better goal space coverage and giving a more accurate estimate of feasibility.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
Thanks to Eugene Vinitsky for helpful discussions, and Hao Liu and Olivia Watkins for feedback on drafts. This work was supported by the Center for Human- Compa---l Artificial Intelligence and FAIR-BAIR collaboration between UC Berkeley and Meta.
ETHICS STATEMENT
Automatic curriculum generation has potential method for learning complex agent behaviours in an unsupervised way. Requiring less direct supervision (eg. through hand-tuned, shaped reward functions) can lead more robust AI systems for real-world applications. At the same time, it is important to consider potential impacts unsupervised reinforcement learning, as it may lead to umin- tended behaviours. By incorporating human engineering and priors through goal space specification in the goal generative methods, like CuSP, we can help restrict the set of goal spaces to more de- sirable (e.g. safe) behaviours. While our empirical studies have been done in simulated systems, careful and thorough analysis of learned behaviours should be done before deploying CuSP on real world systems.
REPRODUCIBILITY STATEMENT
Code is available at https: wittuuuuuuu..... www. hoom/yuqingd//uspp For experimental environment details, see Appendices B.1 B.2 For hyperparameters, see Appendix B.3 For ablations of CuSP
and ASP, see Appendices B.5.1 and B.5. respectively.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience play. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- neurips cc/papprr/217//ile. pdf
sociates, Inc., 2017. URL
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2019.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, llya Sutskever, and Igor Mordatch. Emergent com- plexity via multi-agent competition. In International Conference on Learming Representations, 2018. URL http:///pwneeeeeeeeeeeeee/eaaaac wttp:/orrrmmmdddyyy000xxxxxx
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with (amig)o: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2021. URL, htll/https:// httttss://ppereviiie n
net. forum wtt.ruummdd-TTBB... BEGGRRTR__MMMOXXXX
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised en- vironment design. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 13049-13061. Curran As- sociates, Inc., 2020. URL
neurips co pttpp/ppppp//....... http://ss..pppppre//////1/1111
Papee4499996666---a.ppp. Paper.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In International Conference on Learning Representa- fions, 2018.
10

---

Published as a conference paper at ICLR 2022
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Leurning, volume 80 of Proceedings of Machine Learning Research, pp. 1515-1528. PMLR, 10-15 Jul 2018.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135, 1999.
Irving L Glicksberg. A further generalization of the kakutani fixed point theorem, with application to nash equilibrium points. Proceedings of the American Mathematical Society, 3(1):170-174 1952.
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In international conference on machine learning, pp- 1311-1320. PMLR, 2017.
Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International conference on machine learning, pp. 1802-1811. PMLR, 2018a.
Aditya Grover, Maruan Al- Shedivat, Jayesh K Gupta, Yuri Burda, and Harrison Edwards. Evaluating generalization in multiagent systems using agent-interaction graphs. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1944-1946, 2018b.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-
learning for reinforcement learning, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In International Conference Learn- ing Representations, 2018 URL
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation, 2018
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //arxiv.cro abs 1412. 6980
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (14(13):52113556 2017.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms.
https://
github. com/ ccta://kkootiio///rrrroooaaaaa/ http://roo....--a---p-op---a--a. -gall 2018.
Joel Z. Leibo, Edward Hughes, Mare Lanctot, and Thore Graepel. Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research, 2019.
Alexander Li, Lerrel Pinto, and Pieter Abbeel. Generalized hindsight for reinforcement learning. Advances in Neural Information Processing Systems, 33, 2020.
Hao Liu, Alexander Trott, Richard Socher, and Caiming Xiong. Competitive experience replay. In International Conference Learning Representations, 2018

---

Published as a conference paper at ICLR 2022
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1113-1132. PMIR, 30 Oct-O1 Nov 2020. URL, http::/ppocceet/ ngs m wtt..mrl..rsss....mmmmm.mmmmmmle piress/vlle v100 ynch20a .htmmmmm0...mmmmm...mmmmmm
Tambet Matüisen, Avital Oliver, Taco Cohen, and John Schulman. srrcher-student curriculum learn- ing. IEEE transactions on neural networks and learning systems, 31(9):7732-377 2019.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, 31:9191-9200, 2018.
John Nash. Non-cooperative games. Annals of mathematiics, pp. 286-295, 1951.
OpenAL. Robogym. httppss//gghhh...cc///eeeea///pp 2020.
OpenAl, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin. Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng. Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019.
OpenAl OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D' Sa, Arthur Petron, Henrique P. d. O. Pinto, Alex Paino, Hyeonwoo Noh, Lilian Weng. Qiming Yuan, Casey Chu, and Wojeiech Zaremba. Asymmetric self-play for automatic goal discovery in robotic manipulation. 2021.
Matthias Plappert, Marcin Andrychowicz, Alex Ray. Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, and Wojeiech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research, 2018.
Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierr--vve Oudeyer. Automatic curriculum learning for deep rl: A short survey. arXiv preprint 0rXiv:2003.04664 2020.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Y Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments, 2019.
Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning, pp. 4218- 4227. PMLR, 2018.
Sebastien Racaniere, Andrew Lampinen, Adam Santoro, David Reichert, Vlad Firoiu, and Timothy Lillicrap. Automated curriculum generation through setter-solver interactions. In International Conference on Learning Representations, 2020,
net forum?
4d
Julian Schritwweeer, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, and et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839)6604 609, Dec 2020. ISSN 1476-4687.
doi: 10. 1088//415860000000000
URL http:/ dx dwi.oroooggoooodoooo
1038038/ 4 1 1586-166--0 020 03051-4
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics- aware unsupervised discovery of skills. In International Conference on Learning Representations, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieserr, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
12

---

Published as a conference paper at ICLR 2022
Shagun Sodhani and Vardaan Pahuja. Memory augmented self-play, 2018.
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and Rob Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning, 2018a.
Sainbayar Sukhbaatar, Zeming Lin, llya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fer- gus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Representations, 2018b.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm.control: Software and tasks for continuous control, 2020.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1 10.1109/RRO... http//ROS.2012...60 201.2.2322666000
Andrei Tychonoff. Über die topologische erweiterung räumen. Mathematische Annalen, 102 (1):544-561, 1930.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (poct): Endlessly generating increasingly complex and diverse learning environments and their solutions, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-292, 1992.
Stephen Willard. General topology. Courier Corporation, 2012.
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https://
github. cwm httpp//.....oolllaaaaaa///ttoooo
sac, 2020.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value dis- agreement. Advances Neural Information Processing Systems, 33, 2020.
13

---

Published as a conference paper at ICLR 2022
A DESIRED GOAL GENERATION QUALITIES
We begin by qualitatively examining the key properties of the curriculum of goals generated in Point Mass Obstacle. We see that qualitatively our method (Fig. [aa) proposes goals with more progressive exploration than either ASP (Fig. &cl (which seems to get stuck on the walls, limited by Alice's ability to explore) or GoalGAN (Fig. Bc) (which seems to get stuck more along the borders of the space, only prioritizing difficult goals). The DR generator (Fig. [bb) has the opposite result, having great coverage but without any consideration for feasibility or difficulty. Our method manages to balance the exploration with progressive feasibility, where we see CuSP proposing goals more towards the corners at first, then progressively moving towards the center and covering the space more thoroughly, with slightly more density in the top right where the maze is. We hypothesize there is more concentration in the corners initially when the goal generators are untrained and proposing goals towards the edges of the goal constraints.
61
00
-02
43:
62
-21
42 42 -12
(b)
Domain
Randomiza-
(a) CuSP
GoalGAN
(d) ASP+BC
tion
Figure 8: Progressive goal generation plots for Point Mass Obstacle after 500 rounds across all four methods. We see more progressive diversity from CuSP, whereas GoalGAN mostly stays on the outer border and ASP+BC gets stuck on the environment walls, limited by Alice's abilities.
B EXPERIMENTS
Compute. To train each method in each environment, we use I NVIDIA V100 GPU per seed in an internal cluster. This amounts to an approximate total of 400 GPUs for the final evaluations and hyperparameter sweeps. We additionally did extensive prototyping before scaling to the more complex environments, but do not have a reasonable estimate for how much compute was expended in that phase of the project.
the Point Mass environment
B. ToY EXPERIMENTS
For the toy evaluations in Figures and 3. we evaluate on the landscape
(-((2 ((x + 1)2 (y - 1))2) if(a + .1)² + (y .12 .01
regret(æz y)
-.01
otherwise
For Adam, we use the default configuration from PyTorch (Ir=0.001, As (0.9, 0.999), c=le-8, no weight decay). For PPO, we use the default configuration from (Kostrikov) 2018). We found our trends to hold robustly for fairly broad choice of Ir's and both with/without GAE (Schulman et al., 2018). the toy experiments, we carry out 5000 steps of goal proposal with one gradient update for every method at each step. For the non-stationary experiment in Figure 3] we diagonally perturb the center of the regret landscape from top left quadrant (initialized at (-0.1, 0l)) the bottom right quadrant at constant rate of 2e-4 every round. Hence, at the end of 2500 rounds, the center of the landscape is at the origin and at the end of 5000 rounds, the center of the landscape is (0.1, -0.1).
14

---

Published as a conference paper at ICLR 2022
B.2 ENVIRONMENT SPECIFICS
Table 2 describes the reward configurations we used our environments. For the manipulation environments, also incorporated HER (Andrychowiez et al., (2117). is the tolerance for success
(i.e. a goal g is successfully achieved at state the normed distance L2(s g) < c). We terminate the episode early if successful.
Env
Reward
Use HER (Andrychowicz et al. 2017)
e
Point Mass Maze
-L2
.05
Walker
L2
.1
Reach
L2
Toss
-2
.1
Pick and Reach
0 if at goal, -1 otherwise
Table 2: Environment Reward Configurations
We modify the default Point Mass environment by adding in walls to the top right corner and ini- tialize the agent in between the walls 10% of the time to help with exploration. We modify the manipulator environment by always initializing with the ball in the gripper and with the gripper in an upright position. We and an episode length of 100 steps for Reach, Toss, and Pick.
episode length of 1000 steps for Point Mass Obstacle and Walker
Specific definitions of in in-distribution and out-of-distribution goal spaces for each environment Table
Env Point Mass Maze
Gid
Good
-.25,.25 x .55 .25]
[-3. - 3, 3
Walker
--,, 10 x 2, 2]
[10, 20 x - 2, .2
Reach
25, 25 x 1.2,
|-44, .4] x [1,
Toss
1-5,55 x 1.3]
[-66, x 1.4
Pick and Reach
[.5,9] x [5
[4,1]] x (5.
Table 3: Goal spaces for each environment.
B.3 HYPERPARAMETERS
We train the learners and goal generators using the default SAC configuration and implementation from Yarats& & Kostrikov 2020), as listed in Table with separate networks for the actor and critic.
Parameter
Value
.99
Initial
.1
a LR
lc-4
Actor LR
le-4
Critic LR
le-4
Batch size
1024
Critic hidden dim
1024
Critic hidden depth
2
Actor hidden dim
1024
Actor hidden depth
2
Table 4: SAC Hyperparameters
To scale the output of the goal generators to the correct action space, we take the Tanh scaled output and rescale each dimension to the goal space. We use Appendix C of (Haarnoj et al.) 2018) to modify the log loss for SAC accordingly. To train the goal generator, we update after each round (i.e. per goal proposed) with 100 gradient updates. As the learners are updated once per time-step in the environment (100-1000 updates depending on the environment), increased the update frequency for the goal-genent goal-generator to keep pace with the learner updates since the goal generator operates on single time step per episode.
15

---

Published as a conference paper at ICLR 2022
B.4 BASELINES
We reimplement each baseline by comparing against any available public code and matching each paper's s recommended training hyperparameters as closely as possible within our compute limita- tions.
Off vs. On Policy Learners ASP+ al. 2021). Alice and Bob are trained using PPO. We also implemented this as baseline but found that the agents were unable to reach high success at the same rate as our SAC-trained agents. For a comparison that's more favorable the baselines, we report the success rates from training with the same agent configuration (ie. Bob trained with SAC) in the main paper to isolate learning differences the goal generation method.
B.5 ABLATIONS
B.5.1 CUSP ABLATIONS
SAC Motivation Without symmetrization, the regret-based goal generation objective the same as the proposed objective in PAIRED Dennis et al. (2020). However, our contribution extends be- yond the symmetrized objective as our proposed method reframes the goal generation process into a regulae-r agent with a memory buffer which motivated designing our method around SAC. PAIRED uses PPO optimize their environment generators, which we found was not sample efficient enough for our tasks (i.e. success rates were too low), as our toy experiment in Figure suggested.
Here we highlight a case study with the Toss task into why the entropy regularization and replay buffer components are important. Building up from just the regret objective in PAIRED, have in Figure
1. 9 1,a 0, replay buffer Regret objective only for single goal generator, in PAIRED.
2. 8 1, 0, replay buffer, symmetrized Regret objective with two symmetrized goal generators. Symmetrization seems to improve performance slightly, but is not sufficient alone.
3. 3 = a, = 0- Regret objective with replay buffer for the goal generator. Also improves performance slightly, although not much as symmetrization.
4. 3 = Regret objective with replay buffer and entropy regularization. This is par- ticularly helpful for the random OOD evaluation as the greater goal diversity improves multi-goal performance on the multi-goal evaluation task, expected.
5. 3 = 1 symmetrized - Regret objective with replay buffer and entropy regularization, and symmetrized goal generators.
6. T 300, 3 = .1, symmetrized (Full CuSP) Regret objective with a dynamically updated replay buffer and entropy regularization, and symmetrized goal generators.
16

---

Published as a conference paper at ICLR 2022
Exp manipulator toss, Task: random OOD
1.0
Replay
8=1.am Replay Symmetrized
0.8
f=1,2=0
B-1
0.6
B-1 Symmetrized
300, Symmetrized
0.4
0.2
0.0
10
20
30
40
50
Exp manipulator toss, Task: top mid OOD
1.0
Replay
$=1,a= Replay Symmetrized
0.8
6=1,a=0
B-1
0.6
#=1 Symmetrized
T- 300,0 p- Symmetrized
0.2
0.0
40
50
Figure 9: Full ablations of all components of CuSP from Table ! Success rates are averaged across 3 seeds.
Looking sample of the generated goals Figure 10 we find a stark qualitative improvement in goal diversity between the regret-only PAIRED condition and CuSP. In particular, in the baseline the generated goals converge only on the corners.
12
11
4330
3008
Figure 10: Goal plots after 2000 rounds for Toss. The left plot shows the goals generated in the 1,a 0, No Replay Buffer condition and the right plot shows the goals generated in the 300, 3 .1, Symmetrized condition from Figure Red indicates more recent goals.
Regret update and Symmetrization Ablations As we use the learners' critics for estimating the current regret for particular goal, we need to be careful about how we use the critic data since can be inaccurate, especially towards the beginning of training. We introduce two hyperparameters: one for specifying at which episode we begin regret updates T, and a weighting parameter 3 for how much to update the regrets by. We the update rule
regret 3 old regret (1 B) regret
We sweep across T 50, 100, 200 and 5, .9 select the best performing hyperparame- ters based on average performance on the training goal set, Gia- In Figure [T we ablate over the different components of our method: CuSP+SAC with entropy regularization (8 1,a 0).
17

---

Published as a conference paper at ICLR 2022
CuSP+SAC only (3 1). SOPPHSAA+Symmetiiziii (3 1), and CuSP+SAC+Regret Up- dates+Symmetrizatioo to investigate the effects of each component. most of the tasks, aside from Reach on random out of distribution goals, symmetrization improves performance. In particular, we see larger gains on tasks where 3 = performance is lower to begin with, such as Walker. Further gains from stale regret updates vary from being very small on tasks where 3 = 1 performance is already high (eg. for Point Mass Obstacles or Toss on random OOD goals), to larger improvements on the harder skill tasks (eg. Behind Obstacles or Walker Far Right goals). Interestingly, we also note that while entropy regularization is crucial for most of the environments, setting a 0 seems to be helpful for Walker.
Point Mass Obstacles
Reach
Exp point mass maze Taskc random
Exo má manipulator reach. Taskc random OOD
1.-
Success Rate
p-3
0.8
0.8
p-3 Symeeried
T-300= 35ymerrat
0.6
0.6
0.4
02
10
20
30
40
50
0.0
Toss
Walker
Exp manipulator toss. Task random 00D
Exp walker, Task: random
1.0
Success Rate
0.8
Symmentbe 300 2- Soymeraa
0.6
0.4
0.2
0.2
0.0
50
20
40
Figure 11: Success rates across environments on random goals sampled from Good ablating for SAC-based goal generators, symmetrization and stale regret updates. We generally find that having both components improve success rates across the tasks, although the gains vary depending how challenging the environment is to begin with. In particular, having both the regret updates and symmetrization helps most on the more challenging Walker tasks and the skill-specific tasks.
Multiagent Motivation. Here we investigate whether it is sufficient to formulate the CuSP game with only a single learner. Rather than using regret defined the difference in returns of two agents, we look at the difference in the returns of a single agent across two rollouts to identify if stochasticity in the policy alone is sufficient for providing the goal generator signal. As case study we look at the Toss task in Figure [2 where we find the success rate increases much more slowly with a single agent. We hypothesize that a challenge with only using single agent fro defining regret is that the regret landscape is much more flat - i.c. the regret is more likely to be very small and not provide much signal to the goal generators, as the stochasticity of a single agent policy is comparatively lesser than two separate agents. As a result, the goals are less likely to be a useful curriculum for the agent.
Eap matiuuller loss Task: randor OCD
Eap mamieulllr loss, Task: top, mid DCD
10
0.6
0.6
0.4
0.4
0
10
20
0
10
20
30
40
Figure 12: Comparison of CuSP with separately initialized and updated Alice and Bob, or a single Alice. Success rates are averaged across 3 seeds.
18

---

Published as a conference paper at ICLR 2022
B.5.2 ASP ABLATIONS
-22
-23
(a) ASP Sparse
(b) ASP Dense
(c) ASP Dense with SAC Bob
Figure 13: Progressive goal generation plots for the Point Mass environment after the first 1000 episodes, where each goal is the final position of Alice at the end of the episode and both agents are initialized at the center of the domain. We see that with the sparse version of ASP, the proposed goals converge on a single corner over time and gets stuck, whereas the dense implementation leads more diverse goals.
Point Mass
Point Mass Obstacles
1.0
ASF FPO Spans
1.0
ASF PPO Spans
Success Rate
0.6
0.6
0.4
0.4
10
20
30
Reach
Toss
ASP Sparte
ASP Sparse
Success Rate
0.6
0.6
0.0
0.0
Rounds (x100)
Figure 14: Success rates across environments where ASP achieved >> success rates on random goals sampled from Good across each of the ASP variants. For final results in the paper, we report ASP+BC (Dense, SAC).
In the original implementation, ASP+BC uses PPO to optimize both Alice and Bob. We find this very suboptimal for the environment setups in this work. For empirical validation, here we report results using 3 variations of ASP+BC:
ASP+BC (Sparse): Alice is rewarded +1 at the end of an episode of Bob does not succeed, receives reward otherwise. Both Alice and Bob optimized using PPO.
ASP+BC (Dense): Alice is rewarded negative Bob's reward each transition. Both Alice and Bob are optimized using PPO.
ASP+BC (Dense, SAC): Alice is rewarded negative Bob's reward at each transition. Alice is optimized with PPO while Bob is optimized with SAC.
19

---

define two Published as a conference paper at ICLR 2022
For all cases, use the behavioural cloning (BC) mechanism proposed in Section 3.2 of (OpenAl et al., 202T), where Bob is updated with the same clipped behavioural cloning loss on trajectories where it does not successfully achieve Alice's proposed goal.
We observe that with a sparse reward, Alice would tend to get stuck with proposing similar goals in a particular location. While Bob learns to succeed at that single goal, the lack of diversity causes Bob to perform poorly at generalization Good, as seen in the evaluations. To overcome this, we incorporated a dense reward implementation and swapped optimizing for Bob's policy using SAC. Our results presented in Section 4 use this stronger baseline for comparison against CuSP.
The results are shown in Figure 14
novel goals in
C THEORETICAL ANALYSIS
Our overall framework contains 4 players: two goal-conditioned agents, Alice TA and Bob and their corresponding goal generators, GA and Gg. Further, let II and 9 denote the space of policies and goal generative models respectively. At any given round, let ~ GA and 9B Gg denote the corresponding goals sampled from GA and Gn respectively. Accordingly, regrets for the sampled goals:
RR (AA, R(ga, TA) R(ga, ne [go, TB, R(gn, FB) ((gn,TA)
where R(g, TA) and R(g, TB) denote the empirical discounted sum of rewards for Alice and Bob respectively on goal g. Now, we define the objective function f below:
fI == R (9A, TB) G (gB. TB, TA).
(3)
The overall objective function for CuSP can be written as:
min max By Bgg. RA, = F(GA. GB.
(4)
To analyze the above objective, we define coordinating goal generator-solver teams (G, π) as a pair of goal generator G G and goal conditioned agent π II. Accordingly, we can interpret CuSP as a 2 player zero-sum game between two coordinating goal generator-solver teams (GA, TA) and (Gn, TB) defined via goal-conditioned agents A E IT (Alice) and л E II (Bob) along with their corresponding friendly goal generators GA E G and GB E G respectively.
Definition C.1. (CuSP game) The CuSP game a player zero-sum game between two coordinat- ing goal generator-solver teams (G RA) and (Gn, BB) defined via:
Alice A E IT and its friendly goal generator GA E and
Bob πy E II and its friendly goal generator GB G.
The game is zero-sum with payoff function given as F(GA, GB, for (G AA), and as FGG GB, A, TB) for (GB, TB)
Next, we state conditions under which Nash Equilibrium exists for coordinating goal generator- solver agents CuSP. The conditions depend highly on whether the game strategies (given by the product of goal generator space and policy space) are finite or continuous.
Proposition C.2. (Finite Games) Let (G E G x and (GB, πB) E G be two coordinating goal generator-solver agents for CuSP defined over finite goal space G and policy space II. Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.
Proof. Since G and IT are finite, we know that the product space G x IT is also finite. The result then follows directly from Nash's Theorem (Nash 1951).
20

---

Published as a conference paper at ICLR 2022
Proposition C.3. (Continuous Games) Let (AA, X II and (GB, TB) G II II be two ordinating goal generator-solver agents for CuSP defined over continuous goal space G and policy space II. Further, let G and II be nomempty compact meiric spaces and the payoff function F (Eg. be continuous. Then, there exists a mixed strategy Nash Equilibrium for the CuSP game.
Proof. Since G and II compact, we know by Tychonoff's theorem (Tychonoff 1930, Willard 2012) that the product space G x II is also compact. Combined with the fact that is assumed to be continuous, we can apply Glicksberg's Theorem (Glicksberg 1952) finish the proof.
Since we have established conditions for the existence of Nash equilibria for the CuSP game, we know from the minimax theorem that we can recover the Nash equilibirum solution by optimizing Eq. 4
Note 1: In practice, algorithms for optimizing Eq. 4 might not satisfy the necessary assumptions for the theoretical results. For example, we optimize the parameters of generators and policies specified as neural networks as opposed to optimizing in the space of functions directly. Further, we consider single-sample Monte Carlo estimate for the CuSP objective Eq. and optimize the goal generators and policy networks sequentially for fixed number of gradient updates in every round using RI. algorithms such as SAC. While these protocols are standard practice even related works, deriving theoretical guarantees in such scenarios is extremely challenging and an active area of thcoretical research with many open questions.
Note 2: If AA TB. then (g) = (B(g) for all so all corresponding regrets will also be zero and thus give no signal to the goal generators. The key point to note here that in practice, the goal generators have an exploration component that can avoid getting stuck, e.g., this can be achieved via e-greedy or via SAC which has exploration bonus (as done in CuSP).
Concretely, let TA = TB = Trand be randomly initialized agents. We claim that at initialization, (G rrand), (GB, rand) will not be in a Nash Equilibrium except in the case where a random policy is already able to achieve possible goals. To see why, suppose for the sake of contradiction that the agent teams (GA, Trand), (GB, Trand) are at Nash Equilibrium. That is, the teams have the joint best responses to each other. Let G be the set of all possible goals and G' C G be the set of goals that Trand can solve.
I' G, then A TB are already maximally competent and are therefore done learning. Note that this is also what we hope to converge to eventually during training, and the two policies matching can correspond to Nash Equilibrium in such case. At this point R(gA, A, TB) R BB AA) 0 for all _ so neither G Gn can increase their payoffs, and since TA; are parameterized identically, if either policy able increase its payoff on some g we can identically update the other policy such that π A = π until neither policy is able to improve its payoff further either.
If G C G. then for either team can pick any single goal g" G G' and learn policy T capable of achieving {9*} J G' using policy iteration (c.g., with greedy policy improvement assuming no forgetting of past goals). Such g" will eventually be proposed as long the goal generators have an exploratory component. This means there
strategy x¹ that acquires equal utility to Trand on G' and higher utility on g". Thus there exists strictly better strategy for either team (G Trand), (GB, Trand) by switching from Frand to x²., which contradicts that the players are already at a Nash Equilibrium to begin with.
21


Reviews:
[{"review_id": "wuvqxtpQJq9", "reviewer": "Reviewer_A4MW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main strengths\n\n* I found interesting the idea of multiple \"teachers\" and agents to account for catastrophic forgetting while exploring for new goals.\n* I really liked the idea of updating the regrets from the replay buffer using the new evaluations from the critic, as far as I know this is new and quite nice\n* Authors include quick and nice illustrations in Figures 1 and 2 illustrating the advantages of SAC and using updated regrets in the replay buffer\n* Multiple benchmarks used\n\nMain weaknesses\n\nMy bigger concerns with this work come from not well detailed parts/assumptions, the narrow comparison with AMIGO, the fact of highlighting the benefits of using a symmetric method with 4 agents but the lack of analysis of using larger numbers (e.g., 6 or 8) and leaving that part for future work. I detail all of these below:\n\n---\n* I didn't find a clear explanation of what is the action space of the goal-generation agents, authors comment that they need to be predefined and embeded with human priors of which goals are acceptable. This is an important limitation for which I found little discussion, an explanation of how these are defined in the experiments would be very beneficial.\n\n* How are critic and actor implemented, separated? do they share the same networks? since here the critic plays an important role updating the replay buffer it would be beneficial to have a closer look on this\n\n* In section 2 is mentioned that the goal distribution is separated into train and test but it is never clearly stated how this is done\n\n* In page 4, first paragraph you mention that for the goal generators you add a latent noise variable to the state, why?\n\n* In your method there is not really a \"main solver\" or agent, when reporting your results which one you pick? why?\n\n---\n* You cite the recent work from Campero et al. 2021 about AMIGO which also uses a goal generator agent and a goal achiever agent, it could be interpreted as a 2-agent version of what is presented here but in discrete spaces, I missed then a more in depth comparison with their work in benchmarks or , at least, theoretically.\n\n---\n* Since your work is not the first one exploring the idea of an agent looking for achievable yet challenging goals using a goal-generator agent and a reacher, the main contribution seems to be in the introduction of multiple of these and how they help each other, but then why stop in four? From the very beginning your work raises that question, thus is kind of disappointing that it is just left for future work.\n \n\n---\n\n### Post-discussion and updated version\n\nI believe that authors have done a good work in the discussion and answering my concerns. I would have liked to see an analysis of the effect and equilibrium with systems of larger number of agents, but the current work has more than sufficient analysis and contributions on its own as reflected in the additional ablations and discussion incorporated in the new version. I have updated my scores accordingly\n \n", "summary_of_the_paper": "This paper introduces and evaluates a new unsupervised reinforcement learning method for curriculum learning. It aims to training agents in continuous environments with increasingly difficult tasks with a 4-agent algorithm with two goal-agents or teachers that have their respective \"favourite\" student that they want to excel over the other teacher's favourite.", "summary_of_the_review": "The work introduces nice concepts and ideas that are not completely well supported nor fully explored. Thus although I like the  idea I am reluctant to give an accept at its current state\n\n### Post-discussion and updated version\nI think that the updated version supports much better author's statements and reflects better their contributions", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jBPtY5z32C", "reviewer": "Reviewer_JLa8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I found the paper written well and easy to understand. I thought the dynamic regret updates to the goal generating SAC buffer to be intuitive and appreciated the toy example they use to show it can better optimizing a non-stationary objective.\n\nThe main question for me that arose was: Where does the difference in Alice and Bob's performance come from? They both get to train on the same tasks and are the same algorithms. Is it from their different initializations? Is it from the fact that they first train on goals in different orders? Or is it simply that they are stochastic policies and when a task is somewhat hard one is likely to fail? If it is this last option, then I'd think you could come up with a similar and more compute efficient algorithm where a single generator is trying to pick goals to maximize the variance over returns of a single student. So in this setup you could have your single student just do 2 rollouts and use the difference in returns as the reward for the generator.\n\nIt feels like the H.E.R baseline should be included since you are using SAC to optimize your policies.\n\nWhy does the uniform goal distribution policy do so poorly on the reach task (Fig 7) with an extra goal dimension? I don't have a good intuitive sense for why CuSP should do better in this situation.", "summary_of_the_paper": "The authors propose a new curriculum generating algorithm for goal conditioned agents. In their setup, they have two students and two teachers; each teacher's job is to find a task their student can do better than the other student. They find that this symmetric curriculum self play (CuSP) outperforms asymmetric goal generating algorithms such as asymmetric self play and goal gan in a variety of goal based simulated robotics tasks.", "summary_of_the_review": "Overall I feel the paper is slightly above the acceptance threshold. It does a good job of explaining it's algorithmic choices with toy experiments and shows a reasonable bump over other curriculum methods on a series of goal conditioned tasks. However, I'm left with the question of why the method works at all and if there's a better formulation that achieves the same thing (see my first question in the main review). If this were clarified and some analysis included in the paper I think it would be a much stronger submission and would be a clear accept; in this case I would be more than happy to increase my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g66Jtjl170V", "reviewer": "Reviewer_GBqb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The idea is well-motivated and clearly presented\n2. Comprehensive experiments are performed to verify the effectiveness of the proposed method  \n\nWeakness:\n1. The curriculum generation is solely based on the concept of goals. The method would be more powerful if it can be used for non-goals based settings, such as those discussed in [1]. If would be better if the authors can provide some discussion about the general applicability of the proposed method.  \n\n2. The work is mostly based on empirical study, the paper could be strengthened if some theoretical analysis, such as sample/computation complexity, convergence etc. can be provided. Since this method needs to consider multiple agent learning, the computation overhead could be a concern.\n\nOther comments:\n1. For “Goal Generator Design”, It is unclear to me why a latent noise variable z need to be introduced.\n\n2. For the objective of goal generator policy, how is the exploitation term be constructed, especially under one-step optimization, does it include all the immediate rewards encountered in one episode?\n\n[1] Portelas et al., Automatic Curriculum Learning for Deep RL: A Short Survey.\n", "summary_of_the_paper": "This paper proposed an automated goal generation method, Curriculum selfplay (CuSP) which aims to generate goals of diversity and of increasing difficulty in sample efficient way, as well as to prevent catastrophically forgetting previously solved goals. The method is based on a 4-player game formulation, where cooperation and competition are balanced between two off-policy student learners and two regret-maximizing teachers, who plays the role of balancing progressive exploration and anti-catastrophic forgetting. The method is evaluated on a arrange of continuous control tasks and is shown to outperform a few baselines.", "summary_of_the_review": "In all, this paper is well written and provided a novel formulation for goal-conditional automatic curriculum learning. The paper could be strengthened if its use cases can be extended to more general curriculum learning settings (beyond goal-generation). It would also be better if more theoretical analysis, such as sample complexity/convergence and be provided. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6r1vbcTMK9H", "reviewer": "Reviewer_qGDC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors say this is a 2-player zero-sum game but (1) they do not solve this game i.e. find a Nash equilibrium. Self play does not converge to a Nash equilibrium. (2) there exist Nash equilibria of this game where Alice and Bob are identical in which case the two goal generators get zero reward no matter what goals they propose. (3) even if one solved a Nash of this game that wasn’t degenerate, it’s not clear to me why that would be a good curriculum. \nSo it seems that the method mainly relies on the training dynamics, not the static equilibrium properties of this setup. But even then, let’s say Alice and Bob start out as identical and the two goal proposers also start out as identical. Then shouldn’t Alice and Bob remain identical and the two goal proposers also remain identical because they all have the same updates? \nI don’t see how the given method satisfies the desiderata (progressive diversity, progressive feasibility and anti-forgetting). It seems like the first three rows are not a core part of the method but are used to heuristically improve the results. The last row refers to the actual newly proposed objective where you have symmetric goal setters and solvers. But as mentioned above, I don’t think this symmetric objective gives the desiderata like claimed.  \nWhy is SAC used to propose goals? Isn’t a goal a single action? \nThe results look decent, but do not perform much better than uniform (domain randomization) on most tasks. I would be interested to see how well PAIRED does on these tasks. \n", "summary_of_the_paper": "This paper proposes a modification to asymmetric self play where we have two goal generators and two solvers. Each goal generator is incentivized to propose goals that one solver can solve but the other cannot and each solver is incentivized to solve goals given to them. In experimental results the method seems to perform a bit better than domain randomization. \n", "summary_of_the_review": "I like this direction but I am worried that the main objective has problems that can prevent it from being an effective curriculum, even if they don't show up in these particular results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While one reviewer remained concerned about the possibility of convergence to bad equilibria and felt that the proposed method appears to be four minor changes from prior work (PAIRED), the authors demonstrate empirically that the proposed changes make a significant difference in their evaluation. Other reviewers were positive about this work and all others rated this work as an accept. Post rebuttal the most positive reviewer increased their score to an 8 and felt did a good job answering their concerns. They wanted to see an analysis of systems with larger numbers of agents, but felt that the current manuscript was more than sufficient to warrant acceptance, and fell into the category of a good paper with the additional ablations provided during the rebuttal.

The AC recommends accepting this paper.
Decision: Accept (Poster)