It Takes Four to Tango: Multiagent Self Play for Automatic Curriculum Generation

1 INTRODUCTION . Reinforcement learning ( RL ) has seen great success in a range of applications , from game-playing ( Silver et al. , 2016 ; Schrittwieser et al. , 2020 ) to real world robotics tasks ( OpenAI et al. , 2019 ) . However , these accomplishments are often in the realm of mastering a single specific task . For many applications , we further desire agents who are capable of generalizing to a variety of tasks . This can achieved via goal-conditioned RL , where an RL agent is trained to solve many goals by specifying a sequence of goals , i.e . a goal curriculum . Manually designing such curricula is tedious and moreover , does not scale due to being environment specific and heuristic-driven . Even when it is easy to engineer a curriculum , we would prefer curricula that can dynamically adapt to the learner ’ s current capabilities . For example , effective human tutors use an understanding of their pupil ’ s current knowledge and learning trajectory to create a curriculum . Our work builds on two broad desiderata for curriculum generation ( Bengio et al. , 2009 ) that enable sample-efficient learning of goal-conditioned policies : progressive exploration and anti-catastrophic exploitation . Progressive exploration refers to curricula where the agent gradually learns to solve tasks of increasing difficulty and diversity . Anti-catastrophic exploitation ensures that the agent periodically resolves previously attempted goals that could become harder to solve during the course of training a.k.a . catastrophic forgetting ( French , 1999 ; Kirkpatrick et al. , 2017 ) . Many prior works have attempted to incorporate aspects of these criteria for automatic curriculum generation . One approach is to parameterize a goal generator and pit it against a discriminator that assesses goal difficulty ( Florensa et al. , 2018 ) or feasibility ( Racaniere et al. , 2020 ) for the current agent . Racaniere et al . ( 2020 ) also address desired qualities of validity and diversity by crafting specific losses for these characteristics . However , these approaches rely on learning an accurate discriminator for assessing goal difficulty , which can be challenging . Rather than employing discriminators in the standard adversarial minimax formulation , Dennis et al . ( 2020 ) proposes an alternate minimax objective based on the agent ’ s regret . The ‘ expert ’ used for computing the regret is another goal-conditioned policy whose utility is maximized by an environment generator . Code available at https : //github.com/yuqingd/cusp . An alternative method for automatic curriculum generation that has seen great success is self-play , where an agent plays against other versions of itself ( Silver et al. , 2016 ; Bansal et al. , 2018 ; Baker et al. , 2019 ) ( Fig . 1a ) . Unfortunately , this method does not apply directly to complex tasks where we do not have the structure of a two player , zero-sum game . A promising variation is Asymmetric Self-Play ( ASP ) ( Sukhbaatar et al. , 2018b ) , which involves a minimax game between two similar agents : a goal demonstrator , Alice , and the desired goal-conditioned agent , Bob ( Fig . 1b ) . Bob is asked to solve or reverse tasks demonstrated by Alice in the same environment . To motivate Alice to propose successively challenging goals , Alice is rewarded whenever Bob fails . This method inherently captures an easy-to-hard curriculum , since any goal solved by Alice is achievable by Bob in principle . However , ASP tends to be sample inefficient as it is bottlenecked by Alice ’ s ability to remember earlier goals and explore a diverse goal space . Our work aims to bring together the strengths of goal generative methods and ASP while addressing the two desiderata . Our work focuses on the multi-goal setting with the curriculum generator controlling goals Portelas et al . ( 2020 ) . We propose Curriculum Self Play ( CuSP ) , a symmetric multiagent setup for automatic curricula generation . CuSP optimizes for two off-policy goal-conditioned students , Alice and Bob , and two regret-maximizing goal generators , GA and GB ( Fig . 1c ) . Drawing inspiration from the minimax regret objective in PAIRED ( Dennis et al. , 2020 ) , we designate each goal generator as being ‘ friendly ’ to one of the students ( GA for Alice , GB for Bob ) , and optimize each goal generator using the difference in utility between the preferred student and the other student ( i.e. , GA is rewarded RA − RB , where RA , RB are Alice ’ s and Bob ’ s returns respectively when challenged with the same goal ) . This encourages proposing goals that are neither too easy nor too hard . With our symmetrized setup , Alice and Bob each have a ‘ friendly ’ goal generator that is inclined to propose more feasible goals without being too simple , and an ‘ unfriendly ’ goal generator that is inclined to propose more challenging goals . As the tasks increase in difficulty overall , the former ’ s cooperation helps in scaffolding learning while the latter ’ s competition helps push the boundaries of the agent ’ s capabilities . We further address goal diversity and sample efficiency by leveraging the off policy entropyregularized optimization of Soft Actor Critic ( SAC ) ( Haarnoja et al. , 2018 ) for training the goal generators . The goal generators ’ replay buffers track the regrets for all proposed goals . To account for non-stationary learners , we propose an update rule for the regrets in the goal generators ’ replay buffers based on the critic networks of Alice and Bob . In doing so , we constantly keep track of the learners ’ current abilities and can propose increasingly challenging goals . At the same time , using the regret objective also places the current abilities in context . For example , if Bob has forgotten how to achieve a particular goal , the regret RA − RB increases and GA is more inclined to re-propose that goal . Thus the combination of using a replay buffer and the regret objective aids in addressing the twin goals of progressive exploration and anti-catastrophic exploitation . Compared to prior methods for goal generation , we demonstrate that CuSP is able to satisfy the desiderata for good curricula . We show that this translates to improved sample efficiency and generalization to novel out-of-distribution goals across a range of robotics tasks , encompassing navigation , manipulation , and locomotion . We highlight some emergent skills that arise via CuSP when generalizing to particularly challenging goals , and find that our method is able to handle cases when portions of the goal space are infeasible . 2 BACKGROUND . Problem Setup . We model the environment and the goal-conditioned learner using a Markov decision process , M = 〈S , A , T , R , G , γ〉 , where S is the state space , A is the action space , T : S ×A×S → R is the environment transition function , R : G ×S → R is the goal-conditioned reward function , G ⊆ S is the goal space , and γ is a discount factor . We consider both dense and sparse reward functions : r ( s|g ) = −d ( s , g ) , where d is a distance metric ( e.g . L2 norm ) between the state s and the goal g for the former , or r ( s|g ) = 1 { d ( s , g ) < } for the latter . Our objective is to learn a policy π ( at|st , g ) that maximizes the success rate of reaching all goals g ∈ G. To assess ability to generalize to out-of-distribution goals , we split the goal space into Gid and Good where the goal generators can only propose goals g ∈ Gid , and we evaluate on sampled g ∈ Good . Self Play ( SP ) . SP is an approach for automatic curriculum generation in competitive , zero-sum games . In this paradigm , an agent plays against past versions of itself in order to automatically generate a learning curriculum ( Figure 1a ) . That is , we consider Alice and Bob as two versions of the same underlying agent , and each agent ’ s objective is the negative of the others ’ , so RA = −RB . However , this approach is only applicable for symmetric zero-sum games , and does not lend itself to learning curricula for goal-conditioned agents in other setups . Asymmetric Self Play ( ASP ) . ASP ( Sukhbaatar et al. , 2018b ) extends self play to non-zero sum games by proposing an asymmetric parametrization of two competing agents . During each episode , Alice ’ s objective is to propose and demonstrate a task for Bob to complete ( Figure 1b ) . In the goalconditioned paradigm , we take Alice ’ s final state sAT as the proposed goal g. Alice and Bob now have different objectives : Bob ’ s return , RB = ∑ t γ tr ( sBt |g ) , is simply based on reaching the proposed goal , while Alice ’ s reward is based on Bob ’ s inability to reach the goal , RA = 1 ( d ( sBT , g ) ≥ ) . While this approach lends itself well to more complex tasks such as robotic manipulation ( OpenAI et al. , 2021 ) , it has poor sample efficiency . Since we rely on a learning agent , Alice , to propose and solve goals , the generated goals are limited by Alice ’ s own ability to explore and learn challenging yet feasible goals . This can be particularly problematic towards the start of training , where we can waste many episodes proposing simple and redundant goals for Bob . 3 AUTOMATIC CURRICULUM GENERATION VIA CUSP . Our curriculum generation method consists of four main players : two goal-conditioned learning peers , Alice πA and Bob πB , and two goal generators , GA and GB . πA ( a|s , g ) and πB ( a|s , g ) are both optimized to maximize the discounted sum of goal-conditioned rewards R ( g ) = ∑ t γ trt ( g ) , across proposed goals g ∈ Gid . Aside from any initial randomness in initializing πA and πB , the two policies are parameterized with the same architecture . GA ( g|s0 , z ) and GB ( g|s0 , z ) are parametrized as policies that output a goal g ∈ Gid given an initial observation of the environment s0 and a latent noise variable z ∼ N ( 0 , 1 ) . The inclusion of such a noise variable draws from generative models ( e.g . GANs , VAEs ) where such a z is the input to the generator , and is also found in prior goal generative methods ( Florensa et al. , 2018 ) . The goal generators are optimized to maximize the regret of their corresponding agent : RGA = RA ( g ) −RB ( g ) and RGB = −RGA = RB ( g ) −RA ( g ) . We refer to our overall curriculum generation approach as Curriculum Self Play ( CuSP ) . We present an overview of CuSP in Algorithm 1 and illustrate it in Figure 1c . The algorithm proceeds in rounds . In every round , both goal generators propose a goal , gA from GA and gB from GB . To keep the symmetrization fair and hopefully scaffold learning initially , we first rollout the ‘ easier ’ goal for each respective agent ( ie . gA for Alice , gB for Bob ) and compute the corresponding environment returns and update the learners . Next , we rollout the ‘ harder ’ goal for each respective agent ( ie . gB for Alice , gA for Bob ) and update again . At the end of the round , we update the goal generators . Symmetrization . The regret objective RA − RB helps GA propose goals that are challenging for Bob while still being feasible for Alice . However , this can be detrimental when Alice and Bob ’ s performance diverge too quickly . To address this , we propose symmetrizing the multi-agent system by introducing a corresponding goal generator GB whose regret objective is RB − RA , essentially pitting it in a zero-sum game against GA. Having these two goal generators produces two types of goals for each agent : a ) a feasible yet not too simple goal from the ‘ friendly ’ goal generator ( GA for Alice , GB for Bob ) , which can help scaffold learning quickly , and b ) a challenging yet not too difficult goal from the ‘ unfriendly ’ goal generator ( GA for Bob , GB for Alice ) , which can help push Algorithm 1 : Curriculum Self Play ( CuSP ) Initialize goal-conditioned learners πA , πB , goal generators GA , GB ; while not converged do // generate goals ga ∼ GA , gb ∼ GB ; // evaluate easier goals first RAEasy ← rollout ( πA , gA ) ; RBEasy ← rollout ( πB , gB ) ; // update policies πA ← train learner ( πA , RAEasy ) ; πB ← train learner ( πB , RBEasy ) ; // evaluate harder goals second RAHard ← rollout ( πA , gB ) ; RBHard ← rollout ( πB , gA ) ; // update policies πA ← train learner ( πA , RAHard ) ; πB ← train learner ( πB , RBHard ) ; // compute regrets and update generators GA ← train generator ( GA , { ( RAEasy −RBHard ) , gA } , { ( RAHard −RBEasy ) , gB } ) ; GB ← train generator ( GB , { ( RBEasy −RAHard ) , gB } , { ( RBHard −RAEasy ) , gA } ) ; end the boundaries of the agent ’ s abilities . Both of these properties help encourage the generation of goals that are feasible yet progressively challenging . Goal Generator Design . We motivate our goal generator design with our proposed desiderata : progressive exploration with anti-catastrophic exploitation . Although the goal generator is not an RL agent in the traditional sense – it only ‘ acts ’ in a single timestep by proposing a goal and its ‘ observation ’ only changes based on a random noise variable – optimizing such an explore-exploit tradeoff effectively naturally inspires the use of an RL objective , and RL objectives have previously been used for one-step optimization ( e.g. , one-step Q-learning ( Watkins & Dayan , 1992 ) ) . Thus , to train the goal generators we use an RL learning objective and parameterization inspired by a single-time step variant of Soft-Actor Critic ( SAC ) ( Haarnoja et al. , 2018 ) . SAC is an off policy algorithm that optimizes for a trade-off between expected reward and policy entropy H ( π ) with a coefficient α . In our setup , our goal generator only acts once to propose a goal per episode , so each ‘ trajectory ’ is only a single time step consisting of the initial state s ( for which we concatenate the initial environment observation s0 with a latent noise variable z ) , the proposed goal action g , and we get the corresponding “ reward ” for the goal generator policy by computing the regret between πA and πB on g. Thus the objective for each goal generator policy G is given by max G Eg∼G RG ( s , g ) ︸ ︷︷ ︸ exploitation +α H ( G ) ︸ ︷︷ ︸ exploration where the regret term addresses anti-catastrophic exploitation by prioritizing goals where regret has increased , and the entropy term addresses progressive goal space exploration . We equip the SAC goal generators with a replay buffer to store all past goals and their regrets . This helps address forgetting and improve the sample efficiency of goal generation compared to prior methods which optimize the generator as another agent using on-policy algorithms such as PPO ( Schulman et al. , 2017 ) . We further take advantage of SAC ’ s entropy regularization to progressively increase the diversity of the goals , with consideration of Alice and Bob ’ s capabilities using the stored regrets . To empirically motivate the use of SAC , we compare SAC against PPO and Adam ( Kingma & Ba , 2015 ) in optimizing a synthetic landscape . One challenge of the regret objective is that at the beginning of training , the landscape will mostly be flat when πA and πB are similarly unskilled . In Fig . 2 , we find that only SAC is proficient at optimizing a primarily flat landscape . Dynamic Regret Updates for Non-stationary Agents . However , using an off policy RL algorithm for optimizing such a goal generator requires further modifications . For more typical agents , the received rewards are a stationary property of the environment . In our case this is no longer true – Alice and Bob are continuously learning and part of the multi-agent environment , so the regrets for the same goal will also change over time . To address this issue , we would like to update the non-stationary regret values in the goal generator ’ s replay buffer . However , it is expensive for Alice and Bob to replay all the goals in the environment . As a sample efficient trick , we leverage the observation that the critics of Alice and Bob can provide a current estimate of their performance . Hence , after every round , we update the regrets of previous goals in the replay buffer using the difference between the corresponding value estimates from the critic networks of Alice and Bob . This allows us to keep an updated estimate of feasibility based on Alice and Bob ’ s current skills . To empirically validate that updating the replay buffer regrets helps in a non-stationary landscape , we carry out investigations of a SAC goal generator both with and without regret updates in Fig . 3 . As hypothesized , updating the replay buffer allows the generator to find the moving optimum . The combination of the replay buffer and dynamic regret updates also strengthens our method by automatically prioritizing goals that have been forgotten . With the non-stationary regret updates , the goal generators always have an updated estimate of each learner ’ s abilities . Thus if Bob performs worse on a past goal , the regret increases for GA and it is more likely to repropose it , addressing catastrophic forgetting for Bob . Likewise , symmetrization helps to address catastrophic forgetting for Alice . Without the corresponding GB , if Alice performs worse at a goal , the regret RA − RB decreases and GA is less likely to repropose it unless Bob ’ s performance decreases even further . This can potentially lead to neglecting to propose these challenging goals for Bob . Summary . Inspired by Racaniere et al . ( 2020 ) , we specify desired qualities of a goal generator : • Progressive Feasibility – the goals should be incrementally more difficult , accounting for the agent ’ s current skill level and the fact that the agent is a non-stationary learner . • Progressive Diversity – the goals should encourage increasing exploration of goals . • Anti-Forgetting – if previous goals are forgotten , they should be reproposed . We summarize the contributions of each component of our method to desired qualities of feasibility , diversity , and anti-forgetting in Table 1 . We visualize generated goal distributions in Appendix A and ablate over each component of Table 1 in Appendix B.5.1 to illustrate their importance .

Summary: This paper proposed an automated goal generation method, Curriculum selfplay (CuSP) which aims to generate goals of diversity and of increasing difficulty in sample efficient way, as well as to prevent catastrophically forgetting previously solved goals. The method is based on a 4-player game formulation, where cooperation and competition are balanced between two off-policy student learners and two regret-maximizing teachers, who plays the role of balancing progressive exploration and anti-catastrophic forgetting. The method is evaluated on a arrange of continuous control tasks and is shown to outperform a few baselines.