Automated curriculum generation through setter-solver interactions

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms use correlations between policies and environmental rewards to reinforce and improve agent performance . But such correlation-based learning may struggle in dynamic environments with constantly changing settings or goals , because policies that correlate with rewards in one episode may fail to correlate with rewards in a subsequent episode . Correlation-based learning may also struggle in sparsely rewarding environments since by definition there are fewer rewards , and hence fewer instances when policy-reward correlations can be measured and learned from . In the most problematic tasks , agents may fail to begin learning at all . While RL has been used to achieve expert-level performance in some sparsely rewarding games ( Silver et al. , 2016 ; OpenAI , 2018 ; Vinyals et al. , 2019 ) , success has often required carefully engineered curricula to bootstrap learning , such as learning from millions of expert games or hand-crafted shaping rewards . In some cases self-play between agents as they improve can serve as a powerful automatic curriculum for achieving expert or superhuman performance ( Silver et al. , 2018 ; Vinyals et al. , 2019 ) . But self-play is only possible in symmetric two-player games . Otherwise humans must hand-design a curriculum for the agents , which requires domain knowledge and is time-consuming , especially as tasks and environments grow in complexity . It would be preferable to have an algorithm that could automatically generate a curriculum for agents as they learn . ∗DeepMind and Stanford University Department of Psychology Several automatic-curriculum generating algorithms have been proposed , including some that help agents explore and learn about their environments ( e.g . Gregor et al. , 2016b ; Eysenbach et al. , 2018 ) , and some that attempt to gradually increase goal difficulty ( e.g . Florensa et al. , 2017 ) . Most of these approaches have been tested only on simple tasks in simple environments , and often assume that either the environment is fixed from one episode to the next or that the agent ’ s goal is fixed and unchanging . Ideally , curricula would apply to complex , varying environments and would support goal-conditioning to handle changing tasks . Surprise- or difficulty-based exploration may sometimes discover desired agent behaviors ( Gregor et al. , 2016b ; Burda et al. , 2018 ; Haber et al. , 2018 ) . This approach may not always be practical , though , since many difficult , but otherwise irrelevant tasks might “ distract ” exploration objectives . For example , training a self-driving car to successfully do flips might be challenging and novel , but it would not be particularly beneficial . Human curricula efficiently lead learners towards a desired competency , rather than along arbitrary dimensions of difficulty . Analogously , it would be useful for algorithms to leverage knowledge of the desired goal distribution to develop more targeted curricula . This paper take several steps toward automatic , targeted curriculum generation by proposing an algorithm for training a goal-conditioned agent in dynamic task settings with sparse rewards . The approach trains a “ setter ” model to generate goals for a “ solver ” agent by optimizing three setter objectives : discovering the subset of expressible goals that are valid , or achievable by an expert solver ( goal validity ) , encouraging exploration in the space of goals ( goal coverage ) , and maintaining goal feasibility given the agent ’ s current skill ( goal feasibility ) . We also propose an extension for targeting a distribution of desired tasks ( if one is known ) using a Wasserstein discriminator ( Arjovsky et al. , 2017 ) . We demonstrate our approach in a rich 3D environment and a grid-world wherein observation statistics and possible goals vary between episodes , and show that it substantially outperforms baselines , lesions and prior approaches . 2 RELATED WORK . Uniform sampling of sub-tasks Perhaps the simplest curriculum is training uniformly over subtasks of varying difficulty . For example , Agostinelli et al . ( 2019 ) trained a Rubik ’ s cube solver on problems sampled uniformly between 1 and K moves from the solved state . This curriculum leverages the fact that some sub-tasks can be solved before others , and that learning of these sub-tasks bootstraps learning of harder sub-tasks , and ultimately the task as a whole . However , in complex settings uniform training may not suffice , either because easier sub-tasks do not exist , they are still too hard to learn , or they do not help learning of harder sub-tasks . When uniform sampling is ineffective , hand-engineered curricula may work ( Elman , 1993 ; Bengio et al. , 2009 ; Zaremba & Sutskever , 2014 ; Graves et al. , 2016 ) . Their effectiveness has led to research on automated ways to derive curricula ( Graves et al. , 2017 ) . Here we outline a number of such approaches in the RL setting . Exploration Some work leverages exploration to encourage state diversity ( Gregor et al. , 2016b ) , state-transition surprise ( Burda et al. , 2018 ; Haber et al. , 2018 ) , or distinguishable skills ( Eysenbach et al. , 2018 ) . These exploration-based methods are usually validated in relatively simple , unchanging environments , and have not been tested as pre-training for goal-conditioned RL tasks . A few studies have considered varying environments ; e.g . Wang et al . ( 2019 ) considered evolving environments together with paired agents . However , because each agent is paired to a single environment , the method results in agents that are specialized to single , unchanging environments with fixed goals . Optimal task selection Other approaches include selecting tasks on which learning is progressing ( or regressing ) the fastest ( Baranes & Oudeyer , 2013 ) . However , it can be prohibitively expensive to determine goal regions and track progress within them , especially as task spaces grow larger and more complex . Some approaches work for a set of pre-specified tasks ( Narvekar & Stone , 2019 ) , but they require human effort to hand-select tasks from this set . Again , these approaches have also generally been demonstrated in simple , fixed environments . Agent-agent interactions Agent interactions can also generate effective curricula . For example , in symmetric two-player ( or two-team ) zero-sum games agents jointly improve and thus are forced to face stronger and stronger opponents . This natural curriculum may work on tasks where random play can achieve rewards with reasonable frequency ( Silver et al. , 2018 ) . But in other cases , handengineered auxiliary tasks may be used to avoid the difficult initial problem of learning from sparse rewards , such as imitation learning on data from experts ( Silver et al. , 2016 ; Vinyals et al. , 2019 ) . Or , dense shaping rewards may be needed ( OpenAI , 2018 ; Jaderberg et al. , 2019 ) . Furthermore , this type of curriculum has not been tested in goal-conditioned environments – while the environment might vary because of opponent play , or on a different map , the ultimate goal of winning is fixed . More fundamentally , while this type of curriculum works well for two-player zero-sum games , it is less clear how it can be used to train a single agent on a non-competitive , goal-conditioned task . Asymmetric agent-agent interactions , for example when one agent tries to repeat or undo another ’ s actions ( Sukhbaatar et al. , 2017 ) , can also be useful . However , this requires the desired task distribution to be close to the distribution generated by these reversing/repeating tasks . In goal-conditioned settings , guaranteeing this is likely as difficult as the original learning problem . Goal conditioning In the goal-conditioned setting , hindsight experience replay ( Andrychowicz et al. , 2017 ) has agents retrospectively imagine that they were trying to achieve the state they actually ended up in . While this is an active curriculum for starting learning , it does not necessarily encourage goal-space exploration , nor does it provide a framework for generating novel goals . Nair et al . ( 2018 ) used a generative model of state space to sample “ imagined ” goals , rewarding the agent based on similarity to the generative model ’ s latent space . Florensa et al . ( 2017 ) used a GAN to generate goals of intermediate difficulty for the agent , which resulted in goals that gradually expanded to fill the goal space . This work is closely related to part of our proposal , and we use it as an important benchmark . Critically , this approach has not been tested in environments which vary substantially from episode to episode , particularly ones where the valid goals change from episode to episode . This is an important distinction because training generative models with non-trivial conditioning can be challenging . In particular , while conditioning directly on an informative latent variable can work well , for example when trying to generate images from a given class ( Mirza & Osindero , 2014 ; Brock et al. , 2018 ) , even this problem is not completely solved ( Ravuri & Vinyals , 2019 ) . Adding the challenge of trying to discover latent variables with which to condition and performing even a simple manipulation of them makes things much more difficult ( Rezende & Viola , 2018 ) ( c.f . the difficulty of learning hierarchies of latent variables ( Sønderby et al. , 2016 ; Maaløe et al. , 2019 ) ) . This means that if the valid goals are not trivially observable from the environment , it may be difficult for the goal-setter to discover the goal structure via a generative loss alone . In section 4.2 , we demonstrate this particular failure mode , along with some successes . Summary A variety of automated curriculum generation approaches for RL have demonstrated some success , but the challenge of curriculum generation in the more complex settings remains open . This is because these approaches have not demonstrated success in tasks with the complexity reflective of difficult real-world tasks ; in particular , no approach can handle goal-conditioned tasks in dynamic environments , wherein the set of possible goals varies from one episode to the next , and the set of possible goals might be tiny compared to the set of expressible goals . 3 METHOD . Our model consists of three main components : A solver – the goal-conditioned agent we are training . A setter ( S ) – A generative model we are using to generate a curriculum of goals for the agent . A judge ( J ) – A discriminative model that predicts the feasibility of a goal for the agent at present . See appendix B for architectural details . See fig . 1 for training schematics ( see also Appendix B.2 ) . The solver agent trains on settergenerated goals using a distributed learning setup to compute policy gradients ( Espeholt et al. , 2018 ) . For setter training , three concepts are important : goal validity , goal feasibility and goal coverage . We say a goal is valid if there exists a solver agent policy which has a non-zero probability of achieving this goal . This concept is independent of the current policy of the solver . By contrast , feasibility captures whether the goal is achievable by the solver at present . Specifically , we say a goal has feasibility f ∈ [ 0 , 1 ] if the probability that the solver will achieve the goal is f . The set of feasible goals will therefore evolve as the solver learns . The judge is a learned model of feasibility , trained via supervised learning on the solver ’ s results . Finally , goal coverage indicates the variability ( entropy ) of the goals generated by the setter .

Summary: This paper proposes an autocurricula scheme to train a goal-conditional agent in a dynamic and sparse-rewarding environment. The main idea is to train a setter model to sample goals for next-step training, where the setter can make the decision either based on the training history or the environmental observation (conditional case). The paper proposes three criteria which leads to three types of loss to train the setter model, i.e., goal validity (the goal should be achievable by some existing policy), goal feasibility (how probable the current policy can achieve the goal), and goal coverage (the sampled goals by the setter need to cover all possible goals). A judge model is needed to output the feasibility of a given goal. So the autocurricula scheme contains the solver (agent), the setter, and the judge, each having its own combination of loss and they are trained together. Given a desired goal distribution, the paper proposes to additionally train a discriminator whose optimization objective is Wasserstein loss. In experiments, they evaluate the proposed method on three types of tasks in two environments, i.e., 3D color finding and grid world alchemy. The goals in the two environments are similar in that they all aim to achieve some color or color pairs. The difference lies in that the first one finds colors while the second pick up colors. Each environment can be changed between episodes by changing the colors of objects in the scenes. Experimental results show that different combinations of the three types of losses can bring improvements in some scenarios. Making setter and judge conditioned on environment observation can further improve the success rate. Given a desired distribution of goals, the learning becomes more efficient. The paper compares this method with Goal GAN as a baseline and outperforms it on the three tasks.