Automated curriculum generation through setter-solver interactions
Authors: Sebastien Racaniere
Andrew Lampinen
Adam Santoro
David Reichert
Vlad Firoiu
Timothy Lillicrap
Venue:  (2020)
Keywords: Deep Reinforcement Learning
Automatic Curriculum

Abstract: Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance.   But in dynamic or sparsely rewarding environments these correlations are often too small,  or rewarding events are too infrequent to make learning feasible. Human education instead relies on curricula –the breakdown of tasks into simpler, static challenges with dense rewards– to build up to complex behaviors.  While curricula are also useful for artificial agents, hand-crafting them is time consuming.  This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich,dynamic environments.  Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage to construct useful curricula.  We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent is tasked to achieve a single goal selected from a set of possible goals that varies between episodes, and identify challenges for future work.  Finally, we demonstrate the value of a novel technique that guides agents towards a desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex,  otherwise unlearnable goals,  and to our knowledge are the first to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary between episodes.

Published as a conference paper at ICLR 2020
AUTOMATED CURRICULA THROUGH SETTER-SOLVER INTERACTIONS
Sébastien Racanière, Andrew K. Lampinen"
Equal Contributions DeepMind sracaniere@googge.cco com, 1ampinen@@stanfor..cmm
Adam Santoro, David P. Reichert, Vlad Firoiu, Timothy P. Lillicrap DeepMind
(adamsantoro, rei chert, vl vladfi, countzeroo@googglle.coo
ABSTRACT
Reinforcement learning algorithms use correlations between policies and rewards to improve agent performance. But in dynamic sparsely rewarding environ- ments these correlations are often too small, or rewarding events are too infre- quent to make learning feasible. Human education instead relies curricula- the breakdown tasks into simpler, static challenges with dense rewards to build up complex behaviors. While curricula also useful for artificial agents, hand- crafting them time consuming. This has lead researchers to explore automatic curriculum generation. Here we explore automatic curriculum generation in rich, dynamic environments. Using a setter-solver paradigm we show the importance of considering goal validity, goal feasibility, and goal coverage construct use- ful curricula. We demonstrate the success of our approach in rich but sparsely rewarding 2D and 3D environments, where an agent tasked achieve a sin- gle goal selected from set of possible goals that varies between episodes, and identify challenges for future work. Finally, we demonstrate the value a novel technique that guides agents towards desired goal distribution. Altogether, these results represent a substantial step towards applying automatic task curricula to learn complex, otherwise unlearnable goals, and our knowledge the first to demonstrate automated curriculum generation for goal-conditioned agents environments where the possible goals vary between episodes.
INTRODUCTION
Reinforcement learning (RL) algorithms use correlations between policies and environmental re- wards to reinforce and improve agent performance. But such correlation-hased learning may strug- gle in dynamic environments with constantly changing settings or goals, because policies that cor- relate with rewards one episode may fail to correlate with rewards in a subsequent episode. Correlation-based learning may also struggle in sparsely rewarding environments since by defini- tion there are fewer rewards, and hence fewer instances when policy-reward correlations can be measured and learned from. In the most problematic tasks, agents may fail to begin learning at all.
While RL has been used to achieve expert-level performance some sparsely rewarding games (Sil- ver et al. 2016 OpenAl 2018 Vinyals al. 2019), success has often required carefully engincered curricula to bootstrap learning, such as learning from millions expert games or hand-crafted shap- ing rewards. In some cases self-play between agents they improve
powerful au- tomatic curriculum for achieving expert or superhuman performance (Silver et al.) 2018 Vinyals al. 2019). But self-play is only possible symmetric two-player games. Otherwise humans must hand-design curriculum for the agents, which requires domain knowledge and is cons-connuming, especially as tasks and environments grow in complexity. It would be preferable have an algorithm that could automatically generate a curriculum for agents as they learn.
DDeepMind and Stanford University Department of Psychology

---

Published as a conference paper at ICLR 2020
Several cesomaatic-crrieulume generating algorithms have been proposed, including some that help agents explore and learn about their environments (c.g. Gregor et al. 2016b; Eysenbach et al. 2018). and some that attempt to gradually increase goal difficulty (e.g. Florensa et 2017) Most of these approaches have been tested only on simple tasks in simple environments, and often assume that either the environment is fixed from one episode to the next that the agent's goal is fixed and unchanging, Idcally, curricula would apply to complex, varying environments and would support goal-conditioning handle changing tasks.
Surprise- or difficulty- -base exploration may sometimes discover desired agent behaviors (Gregor al.] 2016b; Burda et al., 2018; Haber et al.] 2018). This approach may not always be practical, though, since many difficult, but otherwise irrelevant tasks might "distract" exploration objectives. For example, training a self-driving car to successfully do flips might be challenging and novel, but it would not be particularly beneficial. Human curricula efficiently lead learners towards desired competency, rather than along arbitrary dimensions of difficulty. Analogously, it would be useful for algorithms to leverage knowledge of the desired goal distribution to develop more targeted curricula.
This paper take several steps toward automatic, targeted curriculum generation by proposing an al- gorithm for training goal-conditioned agent in dynamic task settings with sparse rewards. The approach trains a "setter" model to generate goals for a "solver" agent by optimizing three setter ob- jectives: discovering the subset of expressible goals that are valid, achievable by expert solver (goal validity), encouraging exploration in the space of goals (goal coverage), and maintaining goal feasibility given the agent's 8 current skill (goal feasibility). We also propose an extension for tar- geting a distribution of desired tasks (if one is known) using a Wasserstein discriminator (Arjovsky) et al. 2017). We demonstrate our approach in rich 3D environment and grid-world wherein observation statistics and possible goals vary between episodes, and show that it substantially out- performs baselines, lesions and prior approaches.
RELATED WORK
Uniform sampling of sub-tasks Perhaps the simplest curriculum is training uniformly over sub- tasks of varying difficulty. For example, Agostinelli et al. (2019) trained Rubik's 's cube solver on problems sampled uniformly between I and K moves from the solved state. This curriculum lever- ages the fact that some sub-tasks can be solved before others, and that learning of these sub-tasks bootstraps learning of harder sub-tasks, and ultimately the task as a whole. However, in complex set- tings uniform training may not suffice, either because easier sub-tasks do not exist, they are still too hard to learn, or they do not help learning of harder sub-tasks. When uniform sampling is ineffec- tive, hand-engincered curricula may work (Elman 1993 Bengio et al.. 2009 Zaremba & Sutskever, 2014 Graves et al.] (2166). Their effectiveness has led to research on automated ways to derive curricula (Graves et 2017). Here we outline a number of such approaches in the RL setting.
Exploration Some work leverages exploration to encourage state diversity (Gregor et al. 2016b), state-transition surprise (Burda et al. 2018 Haber al. 2018), distinguishable skills (Eysenbach al. 2018). These exploration- based methods are usually validated in relatively simple, unchanging environments, and have not been tested as pre-training for conl-conditioned RL tasks. A few studies have considered varying environments; c.g. Wang et al. 1019 considered evolving environments together with paired agents. However, because each agent is paired a single environment, the method results in agents that are specialized to single, unchanging environments with fixed goals.
Optimal task selection Other approaches include selecting tasks which learning is progressing (or regressing) the fastest (Baranes & Oudeyer 2013). However, it can be prohibitively expensive to determine goal regions and track progress within them, especially as task spaces grow larger and more complex. Some approaches work for a set of pre-specified tasks (Narvekar & Stone (019)) but they require human effort to hand-select tasks from this set. Again, these approaches have also generally been demonstrated in simple, fixed environments.
Agent-agent interactions Agent interactions can also generate effective curricula. For example, in symmetric two-player (or two-team) zero-sum games agents jointly improve and thus are forced to face stronger and stronger opponents. This natural curriculum may work on tasks where random play can achieve rewards with reasonable frequency (Silver et al., 2018). But in other cases, hand- engineered auxiliary tasks may be used to avoid the difficult initial problem of learning from sparse
2

---

Published as a conference paper at ICLR 2020
rewards, such imitation learning data from experts Silver et al., 2016: Vinyals et al. 2019) Or, dense shaping rewards may be needed (OpenAl 2018 Jaderberg et 2019). Furthermore, this type of curriculum has not been tested in gond-conditioned environments while the environment might vary because of opponent play, or on a different map, the ultimate goal of winning is fixed. More fundamentally, while this type of curriculum works well for two-player zero-sum games, it is less clear how it can be used to train single agent on a non-competitive, goal-conditioned task.
Asymmetric agent-agent interactions, for example when one agent tries to repeat or undo another's actions (Skkbbaatar et al. 2017), can also be useful. However, this requires the desired task distribu- tion be close to the distribution generated by these reversing/repeating tasks. In goal-conditioned settings, guaranteeing this is likely as difficult as the original learning problem.
Goal conditioning In the goal-conditioned setting, hindsight experience replay (Andrychowicz et al. 2017) has agents retrospectively imagine that they were trying to achieve the state they actually ended up in. While this an active curriculum for starting learning, does not necessarily encourage goal-space exploration, nor does it provide a framework for generating novel goals.
Nair al. (018 used generative model of state space to sample "imagined" goals, rewarding the agent based on similarity to the generative model's latent space. Florensa et al. (2017) used GAN to generate goals of intermediate difficulty for the agent, which resulted in goals that gradually expanded to fill the goal space. This work is closely related to part of our proposal, and we use it as an important benchmark. Critically, this approach has not been tested environments which vary substantially from episode to episode, particularly ones where the valid goals change from episode to episode. This is an important distinction because training generative models with non-triviat conditioning can be challenging. In particular, while conditioning directly on an informative latent variable can work well, for example when trying to generate images from a given class (Mirza & Osindero 2014 Brock et al. 2018), even this problem is not completely solved (Ravuri & Vinyals) 2019). Adding the challenge of trying to discover latent variables with which to condition and performing even a simple manipulation of them makes things much more difficult (Rezende & Viola, 2018) (c.f. the difficulty of learning hierarchies of latent variables (Sønderby et al. [016]] Maalóe al., 2.19)). This means that if the valid goals are not trivially observable from the environment, may be difficult for the goal-setter to discover the goal structure via a generative loss alone. In we demonstrate this particular failure mode, along with some successes.
section
Summary A variety of automated curriculum generation approaches for RL have demonstrated some success, but the challenge of curriculum generation in the more complex settings remains open. This is because these approaches have not demonstrated success in tasks with the complexity reflective of difficult real-world tasks; in particular, no approach can handle goal-conditioned tasks in dynamic environments, wherein the set of possible goals varies from one episode to the next, and the set of possible goals might be tiny compared to the set of expressible goals.
3 METHOD
Our model consists of three main components: A solver the goal-conditioned agent we are train- ing, A setter (S) - A generative model we are using to generate a curriculum of goals for the agent. A judge (J) - A discriminative model that predicts the feasibility of a goal for the agent at present. See appendix B for architectural details.
See fig. [ for training schematics (see also Appendix B.2). The solver agent trains on setter- generated goals using distributed learning setup to compute policy gradients (Espeholt et 2018). For setter training. three concepts are important: goal validity, goal feasibility and goal coverage. We say a goal is valid if there exists a solver agent policy which has non-zero probability of achiev- ing this goal. This concept independent of the current policy of the solver. By fonrast, captures whether the goal is achievable by the solver at present. Specifically, we say goal has feasi- bility f E [o. 1] if the probability that the solver will achieve the goal is f. The set of feasible goals will therefore evolve as the solver learns. The judge is learned model of feasibility, trained via supervised learning on the solver's 's results. Finally, goal coverage indicates the variability (entropy) of the goals generated by the setter.
3

---

Published as a conference paper at ICLR 2020
Distributed
Solver learner
Vvll.
Loo
Ldes.
Liaas.
training
Trajectory batching
params
LXE
Judge
goal
Environment
Solver
Judge
Setter
cond.
goal
rew.
goal
cond.
rew.
goal
cond.
Setter
Solver results
Solver results
Solver training.
(b) Judge training.
(c) Setter training.
Training schematic (see also appendix tioning observations of the environment (blue) when the environment varies across episodes. The desirability loss (red) can be used when the distribution of desired tasks is known priori.
Figure 1:
B.2 The setter and judge only receive condi-
3.1 REWARD AND LOSSES FOR THE SOLVER
Our solver is a goal conditioned RL agent. At the beginning of every episode receives goal sampled by the setter, and single reward Rg at the end of the episode. The reward Rg is 1 if the solver achieved the goal, or 0 if it did not after a fixed maximum amount of time. The solver could be trained by any RL algorithm. We chose adopt the training setup and losses from/Espehnn (2018). The solver consists of a policy π and a baseline function V which are traied using the V-trace policy gradient actor-critic algorithm with an entropy regularizer (see sseehhot et al. (2018) for details).
al.
3.2 LOss FOR THE JUDGE
The Judge is trained as a binary classifier to predict the reward, or 1. Given a goal g (see section 3.3). (g) are logits such that (g)()) p( Rg 1[g), where is the sigmoid function, Rg are returns obtained by the solver when trying to achieve those goals, and p(R, 1[g) is the probability assigned by the judge that the agent will have return of 1 when given goal We use cross-entropy loss with the input distribution defined by the setter, and labels are obtained by testing the solver these goals:
Laddee -o.. (emt [Rg log(a()(g)) (1 (log( (J(
3.3 LOSSES FOR THE SETTER
Our setter takes as input a desired goal fcasibility f (0, 1). In particular, we can sample a goal g S(z, f) for some sample from Gaussian prior (0, 1) and a desired feasibility f. or we can map backwards from a goal to latent = 5 (g. f), for which we can then compute the probability under the prior. Both directions are used training. With these features in mind, we define three losses for the setter that reflect the concepts of goal validity, feasibility, and coverage:
Validity: generative loss that increases the likelihood of the setter generating goals which the solver has achieved. This is analogous to the hindsight of Andrychowic et al. (2017), but from the setter perspective rather than the solver. Specifically:
Lval. = E, achieved by (aaerr,eeunif(0oo).fEeiii/ [- logg (5- + {,f))
Where is sampled from goals that the solver achieved, regardless of what was tasked with on that episode, small amount of noise to avoid overfitting and p(.) denotes the probability of
This common practice in generative models of images or discrete data. See e.g. Gregor al. 2016a)

---

Published as a conference paper at ICLR 2020
sampling that latent under fixed gaussian prior for the latent of S. This loss may not cover all valid goals, but it is a good estimate available without any other source of knowledge.
Feasibility: A loss that encourages the setter to choose goals which match the judge's feasibility estimates for the solver at present. Specifically:
Leass EsEN(O..)./Eumrr. (J) (())
This loss uniformly samples a desired feasibility f (to train the setter to provide goals at range of difficulties), then attempts to make the setter produce goals that the judge rates as matching that desired feasibility. Note although gradients pass through the judge, its parameters are not updated.
Coverage: A loss that encourages the setter to pick more diverse goals. This helps the setter to cover the space of possible goals, and to avoid collapse. Specifically:
cov
(S(,,
This loss maximises the average of the conditional entropy of the setter. Since the density of f is constant, adding a term (o)(p(J)) the above formula only changes the loss by a constant, and shows that our loss is equivalent to maximising the entropy of the joint distribution f), f).
The setter is trained to minimize the total loss Lsetter = Lval. + Liess. + Lovv Note that the sum Cress + Loov. can be interpreted as a KL-divergence between an energy model and the setter's distribution. Specifically, for a fixed feasibility f. define an energy function on the space of goals by Ey(g) = (J()) - -o- ( Let pr (g) = (9) / be the density of the distribution defined by this energy. where Z is normalizing constant. Then the sum of the feasibility and coverage losses is, up to constant, the average over f [0, 1] of the divergence K (p, (tpp:p/(SS(())) f))).
We also demonstrate two important extensions our framework which are critical in more compli- cated environments:
Variable environments and conditioned setters: While prior work has often focused on fixed en- vironments, such as the same maze each episode, we would like to train agents in variable worlds where the possible goals vary from one episode to the next. For this to be possible, our setter and judge must condition on an environmental observation. However, learning these conditional gener- ative models can be challenging if the valid goals are not trivially observable (see the related work section above). We demonstrate the success of our approach in these environments, and advantages with a conditioned setter and judge.
In complex task spaces, the goals we want agents accomplish will likely lie in small region within the space of all possible goals. Thus it may not be efficient to uni- formly expand difficulty. We propose additional loss for optimizing the setter towards desired goal distribution, when such a distribution is known. Specifically, we propose training a Wasserstein discriminator (Arjovsky et al. (017)) to diseriminate setter-generated goals from goals sampled from the desired goal distribution. The Wasserstein discriminator has the beneficial property that it can give useful gradients even when the distributions are non-overlapping, which is critical in this set- ting, since the easy goals the setter generates initially may not have any overlap with the target goal distribution. Specifically, the desirability discriminator loss is:
Desired goal distributions:
Lasse. Ededsided goal distribution [D(g)] - -ExeN(0.1, [D(S(z, f)]
and the setter trained with the loss:
Lies. Beee. (...EE,eN(..)),eetitt.. (EUniii(o.))
[D(S(2,
Where Boes. is hyperparameter. While targeting the desired distribution can be helpful, is usually not sufficient on its own the desired tasks may be infeasible at first, so the other setter losses are needed to develop a feasible curriculum. The desirability loss just tries to aim this curriculum in the right direction.
3.4 ENVIRONMENTS
We work in two environments, which are briefly described below (see appendix C for further details). In each, the solver receives goal as input during each episode, which must attempt to achieve.
5

---

Published as a conference paper at ICLR 2020
(a) 3D color-finding
(b) Recolored color-finding
(c) Grid-world alchemy.
Figure 2: Our environments. For the 2D grid-world task, the solver (white square) can pick-up
objects (bi-colored squares). The object it is currently carrying is displayed in the upper left.
3D color finding: semi-realistic 3D environment built in Unity (http://unity3d..cmm consisting of room containing colored objects and furniture (fig. 2a). The agent move and look around, and can pick up, manipulate, and drop objects. This results in a complex 46-dimensional action space. Objects and furniture The agent receives a color (or pair colors) as a goal, and rewarded if a patch (or two adjacent patches) in the center of its view contain average colors close to this goal. Both of these tasks sometimes require complex b For example, the agent might have to pick up an object of yellow color, move it object of blue color and look in between to obtain a green that was not otherwise present in the room. Our agents trained within our framework do indeed exhibit these behaviors. For our extensions, we also used version of this environment which the walls, ceiling, and floor of the room, as well as all objects, are procedurally recolored into one of two randomly chosen colors each episode (fig. (bb). This makes the achievable colors in each episode lie in a small subset of color space that overlaps little, if at all, with the achievable colors in other episodes.
randomly placed around the room the beginning of each episode.
grid world environment, containing variety of two-colored objects (fig. [2). The colors of the objects are randomly sampled each episode. The solver can move around the grid, and can walk over an object to pick it up. It cannot put down an object once it has picked it up. If it is already carrying another object, the two objects will systematically combine to make new object (specifically, the colors are combined by a component-win goal object as input, and is rewarded if it produces similar object. Because of the combinatorics of the possible object combinations, the irreversibility of picking an object up, and the difficulty of inferring the result of combining two objects, this environment is challenging for both the setter and the solver. Both have the challenging task of learning what is achievable in any particular episode, since each episode contains colors never seen before.
Grid-world alchemy:
max). The solver receives a
Evaluation: In each experiment we evaluate on fixed test distribution of tasks, regardless of what setter is used for training, in order have fair comparison between conditions. In both environments, the space of valid tasks (that could be done by an expert) occupies a small volume in the space of tasks expressible by the setter. the colour-finding tasks, we do not even know which goals are valid, because of color averaging, shadows, etc. We therefore test on the full set of expressible goals (most of which are invalid), but report performance as a % best observed scores.
4 EXPERIMENTS
4.1 COMPLEX ENVIRONMENTS REQUIRE ALL THREE LOSSES
See video in http:///wrrroooooomomomeeeeeeeee QS USuusesharin "To help with reproducibilityy we provide code for the networks used for the Setter: http://dw...ooo.coommmmmoe/////t UAORQ-wffWAAYysppsharnnn

---

Published as a conference paper at ICLR 2020
Reward 6% of best)
Reward (% of best)
100
Reward (% of best)
100
75
75
50
50
50
25
25
0-
0
0.1
0.3
0.5 Agent steps (billions)
1
Agent steps (billions)
Agent steps (billions)
Alchemy environment.
(b) Finding color.
(c) Finding color pairs (difficultt..
Figure 3: All three of our setter losses (V: validity, F: feasibility, C: coverage) are necessary (pink) in complex environments, but various subsets (green) suffice in simpler environments. (Curves plotted are rewards on random evaluation tasks averaged across three runs per condition + standard deviation across runs; performance is averaged over the last 1000 trials for bb, and 5000 for We plot performance as % of the best agent performance.)
100
Reward 1% best)
Reward (% optima!
75
Dracle sctter
50
Conditioned
50
Unconditioned Random
25
25
0-
0.2
Agent Stops [billions)
Agent steps (billions)
(a) Color-pair finding, recolored 3D.
(b) Grid-world alchemy.
Figure 4: In varying environments, setters that condition on environment observations outperform unconditioned setters, but unconditioned setters still much better than random goals. (Curves plotted are evaluation rewards averaged across three runs per condition + standard deviation across runs; performance is averaged over the last 15000 trials for (a). and 1000 for (b).)
4.1 Complex environments require all three losses
First, we demonstrate that necessary consider all goal validity, feasibility, and coverage in complex environments (fig. 3} In the alchemy environment the validity and coverage losses are necessary, while the feasibility loss not necessary, but does improve consistency (fig. [3a). In the 3D single-color-iinee finding task, various subsets of the losses suffice for learning the task (fig. 33). However, when the agent must find color pairs, fewer goals are possible and achieving a goal more often requires difficult manipulation of objects. Removing any of the setter losses results in substantially worse performance (fig. 3c). See Appendix B.3 for further analysis of the losses, and supplemental fig. 9 for visualization of the generated curriculum simple location-finding task.
4.2
ENVIRONMENTS THAT VARY REQUIRE OBSERVATION CONDITIONING
Environments that vary require observation conditioning
While much prior work automated curriculum generation focused on varying goals within fixed environment, we would like RI systems to perform well on varied tasks in varied environments. For this, they will need to experience a diversity of environments during training, creating the unique challenge of generating curricula that take into account both the current environment and the current abilities of the agent.
To address this implemented randomly colored version of our color-finding environment, and the grid-world alchemy task. In both, the set of possible goals changes each episode. We compare version of our algorithm in which the setter and judge condition before generating (or evaluating) a task to the basic unconditioned version used in the previous experiments, well random baseline (fig. 4 Solvers trained by the basic version of
environmental observation

---

Published as a conference paper at ICLR 2020
Cnntrge Targeted
O
Desired
Reward (desired, % best)
100
Rewardd of bes))
100
75
75
50
25
Targeted Untargeted Des.
0-
0.15
Agent stops (billions)
Agent (billions)
(a) Motivation.
(b) 3D color finding.
(c) Grid-world alchemy.
Figure 5: Targeting a desired goal distribution, when one is known. (Averages across three runs per condition + std. dev. across runs. Performance is averaged over the last 5000 and 1000 trials for (b) and respectively.)
model still outperform those trained with randomly generated goals. However, the version of our model which conditions on an observation results better solver performance. To the best of our knowledge, these are the first results demonstrating the success of any automated curriculum approach for goal-cond RL in a varying environment.
There are few points worth noting about our results in the alchemy environment. First, the uncon- ditioned setter had a tendency to not produce stable solver performance. Solver performance would generally degrade after reaching maximum, while the conditioned setter was able to more steadily maintain solver performance. This was observed across variety hyperparameter settings, and merits further investigation. Second, even our conditioned setters are not leading the agents to per- fect performance on this task.
However, in grid- world alchemy, the conditioned setter teaches the solver reach performance close to that of a solver trained by an oracle which samples from the true distribution possible tasks (fig. 4b). This suggests the limitation is not setter algorithm, but rather the limitations of the solver agent, for example, the fact that it lacks features like planning (Racanière et al. 2017) or relational inductive biases (Zambaldi et 2019) that have proven useful for similar tasks.
In more complex settings the setter may also need auxiliary supervision or stronger inductive bi- ases to overcome the challenges of learning conditional generative models. Indeed, we found that conditioning on a compressed representation (closer to the latent variables) in the recolored color- finding environment gave better results than conditioning on raw observations (see Fig. 10 in the Appendix). Furthermore, in more complex versions of the alchemy environment (for example, in- troducing more objects with more colors), even our conditioned setter algorithm could not learn to reliably generate feasible goals from raw observations. These results again highlight the challenges of learning conditional generative models when conditioning requires extracting latent variables and performing complex relational reasoning. This will be an important area for future work. Despite this caveat, the success of our setter-solver approach varied environments represents an important step towards generating curricula in environments closer to the richness of the real world.
4.3 TARGETING A DESIRED GOAL DISTRIBUTION MORE EFFICIENT
4.3 Targeting desired goal distribution is more efficient
In complex task environments discovering desired behaviors through difficulty- exploration may not be feasible. There may be many ways a task can be difficult, most of which are irrelevant to what we would ultimately like the agent to achieve. By targeting the desired goal distribution with desired-goal loss, the setter efficiently (fig. 5a). In reality, the path will not be perfectly direct, as the setter trades off feasibility, possibility, and coverage with targeting the desired tasks. However, it will generally be more efficient than untargeted setting, or training on only the desired tasks (if they are difficult).
push the solver toward mastering the desired tasks more
We first explore this in the 3D color-finding environment. We target distribution of pairs of 12 bright colors. These pairs are rarely achieved by a random policy, so discovering them is difficult without setter. Training on only the desired distribution thus results in learning. The untargeted
8

---

Published as a conference paper at ICLR 2020
Roward 1% of best)
100
Reward (% of best)
100
Reward (%6 of best)
100
75
75-
50
50
Ours
25
Goal GAN
0-
0
0.25
0.5
0.0
0.5
1.0
1
2
4
5
Agent steps (billions?
Agent steps (billions)
Agent steps (billlons)
(a) Location finding in 3D room.
(b) Color finding in 3D room.
(c) Color pair finding in 3D room.
Figure 6: Comparison to the Goal GAN FFlorensa et al. 2017). the closest approach to our own. (Curves plotted are averages across three runs per condition + standard deviation across runs; per-
formance is averaged over the last 1000 trials at each step for (b). and the last 5000 for (a.)
setter-solver setup does eventually learn these tasks. However, with targeting it discovers them much more rapidly (fig. 5b). and has a lasting advantage over the untargeted version (see supp. fig. 7
In the alchemy environment, the story is somewhat different (fig. 5cp. We chose the desired distri- bution to be the most difficult tasks in the environment, consisting of combining half the objects in the room. However, because the setter has the difficult challenge of learning the conditional genera- tive distribution (which built in the desired distribution), we find that learning from the desired distribution (if available) results in earlier learning. This is in contrast the 3D color finding vironment, where the desired distribution alone resulted in no learning. This again highlights the complexity of learning to generate goals when the valid goal distribution is conditional in complex, non-linear ways on the environment state. However, once the setter figures out the task structure, it is more easily able train the solver, and so it surpasses desired distribution training to reach asymptotic mastery sooner. Furthermore, the fact that the desired tasks are somewhat feasible early in learning means that the targeted setter has less of an advantage over the regular setter.
4.4
COMPARISON TO PRIOR WORK
4.4
Comparison prior work
We compare to the Goal GAN (Florensa et al., 2017). which is the closest to our approach. Our notion of goal feasibility is related to their binary partitioning of goals into those that are of interme- diate difficulty, and those that are not. However, our continuous notion of feasibility has advantages: it allows uniformly sampling feasibility, can be estimated from one run per goal, and may easier to learn. Furthermore, while their discriminator may implicitly encourage increasing coverage and identifying valid goals, training GANs can be difficult, and our explicit losses may be more effective.
We implemented asynchronous version of the algorithm outlined in Florensa al. (2017). which continuously trains the GAN and the agent, rather than iterating between training each. This allowed equalize computational resources between approaches, and apply their approach the same distributed solver agent we used, in order to have as fair a comparison as possible. See appendix for details. We first demonstrate that our implementation of their approach achieves similar per- Formance to our method on a simple (I, y) -ecation-finding task like that used in their paper, but implemented in our more complex 3D environment (fig. (6a), and learnt from pixels rather than state. However, we show that on our more complex color-finding tasks their approach is not as suc- cessful as ours (fig. cb|c and supp. fig. 8). Furthermore, maintaining and sampling from large memory buffer, and running the agents on each goal multiple times get label of whether it was intermediate difficulty were very costly, and their approach thus required more memory and wall- clock time than ours for an equivalent amount of agent steps. In addition, the instabilities introduced by the adversarial training resulted in less consistent results from their approach even on the simple location finding task.
Overall, our results suggest that our approach is more stable and is more effective on complex tasks. Furthermore, noted above, Florensa et al. did not attempt the challenge of curriculum generation in environments that vary (which is why we did not compare to their algorithm in the alchemy environment), while we have also demonstrated success in that setting.

---

Published as a conference paper at ICLR 2020
5 DISCUSSION
In this paper we outlined a strategy for automated curriculum generation for goal-conditioned RI. agents in complex environments. The curriculum is generated by training setter to propose goals for a solver agent. The setter is trained to choose goals based on their validity, feasibility and cov- erage, and we demonstrated that all three of these components are necessary in a complex environ- ment. Furthermore, we showed that this approach substantially outperforms a prior approach and baselines on complex tasks, including 3D environments with rich visual experiences, interactions with objects, and complex control (a nearly 50-dimensional action space). These results represent a substantial step towards automated curriculum generation in rich environments.
We also highlighted the necessity of employing curriculum generation in environments that vary from episode episode. To address this challenge, we demonstrated that by providing an environ- mental observation to the setter and judge, our algorithm can learn to generate reasonable curricula in variable environments. This approach outperformed a lesioned version without the environmen- tal observation and other baselines, and nearly reached the performance of an oracle curriculum based on the true task distribution (where available). To our knowledge, these are the first results to demonstrate automated curriculum generation for goal-conditioned agents in environments where the possible goals vary from episode episode. This is an important step towards developing auto- mated curricula in environments with complexity closer to the real world.
However, our work also highlights challenges when the environment varies. Learning a conditional generative model for the setter in combinatorially complex environments like our alchemy setting can be challenging. From only a generative loss, it is difficult for the model to learn how to extract the appropriate latent variables from an observation and manipulate them appropriately. Training setters in rich environments may require auxiliary information about the structure of the world, or breakthroughs in conditional generative modelling. This is an important direction for future work.
Finally, we pointed out the challenge of efficiently achieving competence on desired goals which distributed in small region of goal space. We demonstrated a loss that can help to solve this problem by targeting the setter toward the desired goal distribution.
Overall, we showed the success of setter-solver approach in rich environments, and extensions that allowed it to work in complex tasks with varying environments and guide the solver efficiently towards mastering desired goals. Although work remains to be done, we believe that the strategies we have outlined here will be a useful starting point for automatically devising curricula for RL agents in the increasingly complex tasks we desire them solve.
6 ACKNOWLEDGEMENT
We would like to thank the DeepMind Worlds Team for their help with the environments used in this paper.
REFERENCES
Forest Agostinelli. Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the rubiks cube with deep reinforcement learning and search. Nature Machine Intelligence, pp. 1. 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight Experience Replay. Ad- vances in Neural Information Processing Systems, (Nips), 2017. URL http::://axxvv.rrggg/.////
abs/1700.01499
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXiv:1701.07875 2017.
Wasserstein
gan.
arXiv preprint
Adrien Baranes and Pierre-Yvel Oudeyer. Active learning of inverse models with intrinsically mo- tivated goal exploration in robots. Robotics and Autonomous Systems, 61(1):49-73 2013.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, ACM, 2009.
10

---

Published as a conference paper at ICLR 2020
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1rXiv:1899.109% 2018.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity- learning. arXiv preprint arXiv: 04038, 2018.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 2016.
Jeffrey I. Elman. Learning and development in neural networks: The importance of starting small. Cognition, 48(1):71-99, 1993.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International Conference on Machine Learning. pp. 1406-1415, 2018.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv: 1XXXvv:8822000700 2018.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. arXiv preprint arXiv: 71177005.606 06366, 2017.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agmieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):471, 2016.
Alex Graves, Mare Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference Machine Learning-Volle Volune 70, pp. 1311- JMLR. org, 2017.
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. In Advances In Neural Information Processing Systems, pp. 3549-3557, 2016a.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 1rXiv: 1.07507, 201
Nick Haber, Damian Mrowca, Stephanie Wang, Li Fei-Fei, and Daniel L Yamins. Learning to play with intrinsiaallymmotvvat self-aware agents. Advances Neural Information Processing Systems, pp. 8388-8399, 2018.
Mark Harrower and Cynthia A Brewer. Colorbrewer. org: online tool for selecting colour schemes for maps. The Cartographic Journal, 40(1):27-37, 2003.
Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nico- las Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo. David Silver, Demis Hassabis, Ko- ray Kavukcuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, (64()443) 2019. ISSN 0036-8075. doi: 10.1126/science.aaaudee aaa26224 364/ 0664/643/888
URL
Lars Maaløe, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling, arXiv preprint arXiv:1902.02110 2019.
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
1rX1v:1411.1788 2014.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Vi- sual reinforcement learning with imagined goals. Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.

---

Published as a conference paper at ICLR 2020
Sanmit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning. In Pro- ceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 25-33. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
OpenAl. Openai five. https: httpa:://loo..peeeecee http://oooppaaa..oo/aeeeaaaaaaea www.:/ooaioooeoeceeeeoeeeo.
2018.
Sébastien Racanière, Théophane Weher, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. Imagination- -augmmetedd agents for deep reinforcement learning. In Advances neural information processing systems, pp. 5690-5701, 2017.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. arXiv preprint arXiv: 10887, 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming vaes. arXiv preprint
2018.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Joannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lilli- crap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Master- ing the game of Go with deep neural networks and tree search. Nature, 529(7587):484-499 2016. ISSN 14764687. doi: 1038//8/n 16961. URL http: /x. dei. wtt.:/ii..o////1110//1.00 nature 6961
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. Mastering Chess and Shogi by Self-Play with a General Rein- forcement Learning Algorithm. Science, 362:1 362:1140-11 1111140-11000 1144, 2018. URL http://axxiv.org
abs/1712.0182
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738-3746,
2016.
Rupesh Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. In Advances in neural information processing systems, pp. 2377-2385, 2015.
Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint arXiv: 1703.05700055 2017.
Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Woj- ciech M. Czamecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dal- ibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budden, Tom Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Toby Pohlen, Yuhuai Wu, Dani Yogatama, Julia Cohen, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Chris Apps, Koray Kavukcuoglu, Demis Hassabis, and David Silver. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. htttps:// com/blog/
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint 11t1::99000005..... 2019.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls. David Reichert, Timothy Lillicrap. Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep re- inforcement learning with relational inductive biases. In International Conference on Learning Representations, 2019.
Wojeiech Zaremba and Ilya Sutskever. Learning to execute. arXiv preprint arXiv:1410.10000 1410.461,, 2014.
12

---

Published as a conference paper at ICLR 2020
A
SUPPLEMENTAL FIGURES
Reward (random, % of best)
100
Reward (desired, % of best)
100
Targeted Untargeted Des, only
Targeted
Untargeted
75
75
50
50
25
25
Agent steps (billions)
Agent steps (billions)
(a) Random evaluation.
(b) Partially extended plot.
Figure 7: Supplemental plots for figure 5b Random evaluation is not impaired by the desirabil- ity loss, suggesting that in this setting is actually helping the setter to expand goals a useful direction. (b) Partial continuation of figures 5b showing the advantages persist. However, some of the experiment versions were killed by a server issue around billion steps, which is why the figure in the main text terminates there. This panel plots runs with slightly worse hyperparameter settings (weight Baes. of instead of 5), but that ran for longer.
Reward (training, %)
100
Reward (training, %
100
100
Ours
Reward (training,
75
Goal GAN
75
75
50
WW
50
25
0.25
0.5
0.0
0.5
1.0
2
4
5
Agent steps (billions)
Agent steps (billions)
Agent steps (billions)
(a) Location finding in 3D room.
(b) Color finding in 3D room.
(c) Color pair finding in 3D room.
Figure 8: Comparing scores agents obtain on the training levels proposed by our method and Goal GAN. In the location task (a) both methods generate levels of intermediate feasibility/dificully However, on more complex tasks (blc). while our method continues to generate tasks of intermediate feasibility (on average), the goal GAN tends overshoot and choose tasks that are too easy. This may be because the capacity of the discriminator is mostly used in determining which tasks are valid (since most tasks are impossible in these more complex environments), and so struggles to find valid but difficult tasks. By contrast, our judge is mainly responsible for determining feasibility, while the validity should mostly be handled by the setter losses. This frees up the capacity of the judge for accurately estimating feasibility. Furthermore, the fact that varying levels of feasibility will like give the judge more accurate data to train on. Finally, the adversarial training of Goal GAN results in noisier performance in the simpler task.
proposing levels at
13

---

Published as a conference paper at ICLR 2020
Figure 9: Visualizing the generated curriculum for the location-finde findiinnn- task which we used in section 4.4 We chose to visualize the curriculum for this task because it is in a lower-dimensional and easier- to-understand task space than the main tasks we considered. In this plot we show a heat map for the goal locations selected by the setter over the course of learning. The agent is placed in an L-shaped room. Goals are chosen by specifying two coordinates where the agent needs to go. We assume we have a rough idea of a bounding box for the coordinates of reachable locations in the room, and use those bounds to limit the goals sampled by the setter. The L-shaped room is actually placed in the bottom left part of this bounding box, and the setter needs to discover this. The first picture above shows how at random initialisation, the setter chooses locations which are not inside the room. Our validity loss will then help move these locations inside the room, shown in the second picture. After this, the setter gradually expands the set of sampled locations as the agent learns, until they fill the whole room, shown in the progression to the picture on the right.
B ARCHITECTURE & TRAINING DETAILS
Solver The solver consists of ResNet (3 sections, each with 2 residual blocks, with 16, 32, and 32 filters respectively, all of size 3 and stride 1) for vision and a goal input (processed through linear transformation)
core LSTM (256 hidden units). The output of the LSTM is fed into separate MLPs for policy (1 hidden layer, 64 hidden units) and value (1 hidden layer, 128 hidden units) outputs. The judge an MIPP (3 hidden layers, 64 units each).
Setter The setter consists of Real-Valued Non- Volume Preserving (RNVP) network (Dinh al., 2016), which has the useful property of providing access to exact log-likelihood of a sample, i.e. exact invertibility. The basis of this invertibility a clever trick used the RNVP design, where at each layer only half of the latent variables are updated, but their update depends multipiccatively on the other half of the latent variables. This allows for representing complex transformations, but because the conditioning latent variables are preserved in the output of the layer, it can be exactly inverted. By alternating between updating different subsets of the latent variables each layer, the variables can undergo complex evolution, with that evolution still remaining precisely invertible. For more details see sec. 3.3 and fig. of Dinh et al. (2016).
The setter has 3 blocks. Each block is a fully connected Highway Network (Srivastava et al. 2015) with 3 layers, and a hidden size of 32, 128, or 512, depending on the complexity of the generative task. Specifically, for the basic color-finding tasks (both single colors and pairs), the hidden size was 32, for the recolored color-finding tasks it was 128, and for the alchemy tasks it was 512. Non- linearities in all networks are leaky ReLUs, except when generating goals, where arctans used to get outputs between -1 and 1, which were then scaled appropriately for the goal domain.
Desirability discriminant As in Arjovsky al. 2017). we ensure that D Lipschitz by clipping the parameters of the discriminator D to the range -0.1, 0.1] before each setter training step.
B.1 OBSERVATION
SETTER & JUDGE
When the setter and judge are observation-condiiion agent ResNet, and the setter and judge share its weights (but do not share weights with the agent vision architecture). In the setter, the conditioning information, including the desired feasibility f and the output of the ResNet (where applicable) is concatenated with the part of the latent being transformed in each layer. In the judge, the output of the ResNet, if applicable, is concatenated with the goal before the goal is input to the judge. We found was useful to weight the vision information by fixed constants before inputting it allowed them to learn first to respond to the goals alone before trying to incorporate visual input. These constants were determined via hyperparameter sweep, and were 0.1 for the setter in all
their vision architecture is the same as the
the setter and the judge, likely because
14

---

Published as a conference paper at ICLR 2020
conditioned tasks, and 10-7 and 10 respectively for the judge in the alchemy tasks and recolored color-finding findiin tasks. (Despite the small weight on the visual input, the judge appears to still use it, the final solver performance is worse if the weight is reduced by another order of magnitude, or increased.)
B.2 TRAINING
The solver agents were trained using the framework of Espeholt et al. (2018), with the RMSProp optimizer, without momentum and learning rate of 2- 10 The setters were trained using Adam, with learning rates of 2 10-4 on the 3D tasks and 3 10 on the grid-world alchemy tasks.
The training setup described in fig. 1 at a high level. We provide now more details. We use distributed setup that is identical to Espeholt et al. (20T8), except for the addition of a learner to train the Judge and Setter. We therefore end up with three types of workers, that run asynchronously and communicate data to each other. Below, write in pscudo code what loops are running each type of worker. We have written those for the conditional setter case.
Algorithm 1: Solver-Actor loop
// The Solver-Actor collects data and send them to the learners;
Start environment;
Sample feasibility uniformly in [0, 1];
Get oissstoooserraation from environment and sample gool from Setter with given feasibility;
w while True do
trajectory= []:
for I...N do
Get observation from environment;
Choose action using the agent and observation;
Apply action to environment;
Get reward and discount;
trajectory += (observation, reward, discount, action); if this was the last step of the episode then
Send (firsl. observalion, gool, reward) to the Setter-Leerner;;
Restart environment to start new episode;
Sample feasibility uniformly in [0, 1]:
Get forst_observatio from environment and sample goal from Setter with given feasibility;
end
end
Send trajectory to the Solver-Learner; Request and apply updated weights for agent from Solver-Leamerr Request and apply updated weights for setter from Setter-Leamer;
end
Algorithm 2: Setter-Learner loop
// The Setter-Learner receives data from the Solver-Actor and use it to train the Judge and Setter; Initialise weights Setter and Judge:
while True do
Wait until we have received a batch of tuple (first corrsttobservationn gool, reward);
Use batch to train Judge with gradient update;
Do gradient update on setter using batch of
as conditioning:
end
15

---

Published as a conference paper at ICLR 2020
100
100
neward 1% best)
73-
75-
Roword of
Resicing 4x4
25
Resicing 4x4
Resnet processing
Resnet processing
0.5
1.0
2.
2.5
0.
0.5
10
2.0
5.5 4.0
steps (billions)
140
(billions) 100
(a) Finding color
(b) Finding colors
Figure 10: Learning a good latent representation from a conditioning observation can be difficult. For the recolored find colour task, the setter only needs to extract the two dominant colours in the room from the observation in order to choose valid goals. We can use this knowledge to hard-code better representation for the conditioning observation by simply resizing the observation image, instead of processing it with a deep ResNet. This representation will be closer to the true latent variables that determine the valid goals in the episode. When do this, the agent learns more rapidly, suggesting that this is indeed a better representation for conditioning the setter (and judge). However, choosing this representation requires using prior knowledge about the task structure. This knowledge might not always be available.
4
All Fi
Givinge city
35
7ON0 CADE 6011
(a) Effect of coverage loss
(b) Judge's 's loss.
(c) Feasibility vs. score.
(d) Validity loss.
Figure 11: Examining the indivudal losses. All curves are for the Color pair finding task. Note how curves (a) to (c) are only up to 100 million steps. This is still early training, but the losses are very stable after that point, as the setter and judge continually adapt the solver's improvements. Curve (d) shows data for the whole training as the loss slowly increases throughout. Purple curves averaged over nine runs. Green curves averaged over three runs. All curves show standard deviation.
FRNE BRG
4 km
Agent Enape
Figure 12: The setter learning to appropriately condition on the feasibility input to produce tasks of varying difficulty. This plot shows the average score (where the maximum score 1) of the solver on 1-color finding tasks generated by a setter for feasibility inputs of 0.25, 0.5 and 0.75. Each graph averaged over 3 setter/solver pairs. We only show the behaviour early in training, it is stable throughout. The setter is producing tasks close to the desired input feasibility, so that uniformly sampling feasibility causes it to produce tasks of varying difficulty.
16

---

Published as a conference paper at ICLR 2020
Algorithm 3: Solver-Learner loop
// The Solver-L is identical, and uses the same losses, as a learner from Espeholt al, (2018);
Initialie weights of Solver (agent);
while True do
Wait until we have received a batch of trajectories from Solver-Actor;
Do gradient update using the batch;
end
B.3 EXAMINING THE INDIVIDUAL LOSSES
Our experiments demonstrate that our setup leads to better agent performance. In this section, we examine the components of our system, evaluate whether they are performing as intended.
First, we look at coverage, which we measure using entropy. In the color pair finding task, the setter defines a distribution in the space [0, 1J6. The uniform distribution is maximum entropy distribution on this space, with an entropy of O. Entropy is particularly easy loss to maximise. Indeed, looking Fig. Taa we see it immediately climbs to maximal value, before slowly decreasing as the pressure from the two other losses builds up. In the same figure, we that removing the coverage loss leads to lower entropy early on, and larger collapse later. This means the setter will offer far less varied set of tasks for the solver train without the coverage loss. Finally, observe how using only the coverage loss leads to near maximum entropy being reached very quickly (the entropy then keeps slowly climbing during training). However, by ignoring validity and feasibility, most the tasks the setter proposes are impossible for an expert, or at least impossible for the agent at present. Thus the overall performance is worse.
While the feasibility loss is being optimised as soon as training start, this loss can only become useful once the judge has learnt how measure feasibility efficiently. Despite training on non-stationary distribution, we expect the supervised training of the judge to result in quicker convergence. We can indeed see in Fig. Ilb that after some very rapid oscillations, the judge's 's cross-entropy loss drops very quickly. Another indication that the judge is indeed training well, is that if we compare the average score of the agent on tasks generated by the setter to the average feasibility of those same tasks as measured by the judge, we see in Fig. [hc] two values match very quickly. Finally, we note that the setter is learning to produce tasks of approximately appropriate feasibility, as shown in fig. [2] The solver is performing slightly better than expected, presumably because of the lag in training the judge and solver, but the difference is not too great, suggesting that the lag is not too bad.
Finally, we see the validity loss in Fig. IId drops very quickly to a minimum value when training starts, and slowly increases throughout training. We see this good behavior since the hindsight provided by this loss is most useful early on when the setter needs to choose which are reasonable tasks to train even if it has not yet learnt much about the structure of tasks.
C ENVIRONMENT & TASK DETAILS
C.I COLOR FINDING
The room is L-shaped, which means that some objects may be hidden from some locations in the room. The objects present consist of pillow-like objects, bookshelf, chair, a few wall-mounted shelves and table. The furniture is placed randomly along the walls of the room at the beginning of each episode. The pillows randomly placed throughout the room, with some scattered on the floor, and some placed on shelves.
The actions for this environment consist of multiple speeds of moving forward, back, left, and right, rotating left and right, looking up and down, grabbing objects and moving with them, and manipulating held objects, rotating them along various This results in a total 45 actions. The agent has to choose 15 actions per second. Observations in the form of 72 x 96 RGB pixel images. Each pixel value was normalised in the range (0, 1).
moving them closer farther away.
17

---

Published as a conference paper at ICLR 2020
For single color finding, the agent received reward of if the color averaged over an 8 x 8 patch in the center the screen was within an Eg distance of c = 0.1 of the goal color E [0, 1 For color pair finding, the agent received reward of 1 if the color in an 8 x 8 patch left of the center of the screen was within e of the first goal color, and similar with a patch right of the center of the screen and the second goal color. That is, the agent needed to get both colors correct to receive any reward on the pair color finding task. If the agent received non-zero reward the episode would terminate, otherwise the episode would terminate with a reward of after 500 environment steps.
Desired distribution: Our desired distribution for the targeting experiments consisted of pairs of 12 colors: the primary colors and 3 secondary colors, and slightly more muted shades of these (all components moved by 0.3 towards the middle). We found ddes. = 5 to be optimal, though results in fig. 7b are from runs with Bies.
C.2 BRID-WORLDD ALCHEMY
The actions consist of movement in the four cardinal directions. The room is 9 x grid surrounded by an impassable wall of size I (for a total grid size of 11 x 11), with four objects randomly placed in it. Each object has two colors which are randomly sampled uniformly from [0, 1] only the red and blue components were sampled, with the green component fixed at zero. This makes the conditional generative problem slightly easier. The agent receives visual input in which every grid square is rendered as 2 x 2 in order to have the two-colored objects, i.e. it received visual input of size 22 22.
To avoid trivial solutions for the setter, it was necessary to avoid rewarding the solver (or training the setter's possibility loss) if the agent failed to pick up an object.
Our desired distribution for the targeting experiments consisted of the most difficult tasks in this world: combining half of the objects in the level. We found Bdes. = 1.5 to be optimal for this task.
Desired distribution:
D DETAILS OF COMPARISON TO FLORENSA ET AL. (2017)
In order to compare to the Goal GAN approach proposed by Florensa et al. (017)) it was necessary to make few changes.
wanted to make as fair comparison as possible, so we wanted to use their approach to train the same distributed solver agent we used. In order to do this, we had to modify their algorithm to run asynchronously. Specifically, instead of alternating between training the GAN and training the agent, we trained both simultancously in separate processes. Because of the asynchronous approach it was also difficult to have a single unified memory buffer; instead cach copy of the agent had its memory buffer which could hold up 10,000 goals, and added goals to it with a probability of 0.01 rather than performing an expensive comparison to the prior goals at each step. As in the original paper, we sampled 1/3 of the goals from the memory buffer, and 2/3 from the setter. Even with our modifications and the simpler MLP architecture (see below), their approach required more computation time than ours for the same number of agent steps.
As in our RNVP architecture, we use a latent noise vector sampled from a standard normal distri- bution of the same dimensionality as the goal outputs. We originally tried implementing their GAN with the same RNVP architecture we used for our setter (see above), but we had substantial issues with mode collapse, so we switched using an MLP architecture like was used in their original paper. We used 3 hidden layers with 64 units each in the location tasks, and 128 units each in the color tasks, and for the discriminator we used 3 hidden layers with 128 units in both tasks. The GAN was trained via the Adam optimizer with a learning rate of 5 10-4, All these hyperparameter were determined by a sweep.
E MISCELLANEOUS
Color palettes for plots were modified from Harrower & Brewer (2003).
18


Reviews:
[{"review_id": "rJgFyaPJ9S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the task of automatically inducing a curriculum for agents learning through reinforcement. Specifically, they use two agents — a setter agent that sets goals, and a solver agent that solves the goals provided by the setter.  While this has been explored before, the difficulty lies in training both agents simultaneously in a robust fashion. If the goals are too difficult, the solver will be unable to solve them and if they are too easy, the solver will be unable to improve. The authors propose a combination of different losses to help the setter balance its goal predictions — validity, feasibility and coverage. In addition, they train a judge model predict the reward that the solver agent would achieve on a goal proposed by the setter. Empirical results on two setups demonstrate the effectiveness of this approach in learning a good curriculum. \n\nPros:\n1. Clear writing, method is easy to understand. \n2. Novel objectives for a multi-agent training setup\n\nCons:\n1. Empirical results do not contain any baselines or prior work comparisons (only ablations of the proposed model)\n\n——\nUpdates:\nThanks to the authors for their response. I realize I was not very clear in my comment above. I wanted to point out that the authors could consider adding other (simpler) baselines such as Sukhbaatar et al. (2017) to make their empirical results more complete for the navigation tasks (even though these methods make certain assumptions, it would be interesting to see how much of an effect they have compared to the proposed method). If you are able to, I’m convinced the paper will be much stronger. Nevertheless, I think this is very interesting work!", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgk9s5TYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an autocurricula scheme to train a goal-conditional agent in a dynamic and sparse-rewarding environment. The main idea is to train a setter model to sample goals for next-step training, where the setter can make the decision either based on the training history or the environmental observation (conditional case). The paper proposes three criteria which leads to three types of loss to train the setter model, i.e., goal validity (the goal should be achievable by some existing policy), goal feasibility (how probable the current policy can achieve the goal), and goal coverage (the sampled goals by the setter need to cover all possible goals). A judge model is needed to output the feasibility of a given goal. So the autocurricula scheme contains the solver (agent), the setter, and the judge, each having its own combination of loss and they are trained together. Given a desired goal distribution, the paper proposes to additionally train a discriminator whose optimization objective is Wasserstein loss. In experiments, they evaluate the proposed method on three types of tasks in two environments, i.e., 3D color finding and grid world alchemy. The goals in the two environments are similar in that they all aim to achieve some color or color pairs. The difference lies in that the first one finds colors while the second pick up colors. Each environment can be changed between episodes by changing the colors of objects in the scenes. Experimental results show that different combinations of the three types of losses can bring improvements in some scenarios. Making setter and judge conditioned on environment observation can further improve the success rate. Given a desired distribution of goals, the learning becomes more efficient. The paper compares this method with Goal GAN as a baseline and outperforms it on the three tasks.\n\nThis paper is an early exploration of the effectiveness of autocurricula on goal-conditional tasks. The experiments to some extent verify the effectiveness of the proposed method. The high-level idea of setter-solver learning can be possibly generalized to other tasks. However, it is not convincing that the detailed techniques proposed in this paper can be easily generalized to more complicated environments and tasks. The writing of this paper is not clear and enough and the whole paper is difficult to understand. The experimental comparison is insufficient since only one baseline of curriculum (goal GAN) is compared (some of the methods mentioned in Section 2 can be easily applied though some of them were not specifically designed for goal-conditional tasks). \n\nDetailed comments:\n\n1) There are too many details and hyperparameters involved when applied to specific tasks as shown in this paper. Considering the two environments in this paper is easier than most game environments studied in other papers (many of them also uses some types of the curriculum), the experimental result is not very strong. The proposed scheme is possible to be too simple for a slightly more complicated environment or task. \n\n2) The success of training relies on interactions between the three to four types of losses, but they all have the same weights in the combined loss. Is there any special reason for not setting different weights for different losses?\n\n3) It is not clear how the inverse transform of the setter model (i.e., S^{-1} in L_val) is achieved.\n\n4) There are three random quantities involved in the loss terms, i.e., the noise \\xi applied to the sampled goals, the latent input z, the feasibility f. It is not clear whether the randomness on them will dominate the curriculum or not. \n\n5) The setter model takes z and f as inputs, where z is a vector but f is a scalar. Will it result in a setter model whose output does not change too much when changing f? \n\n6) How does the proposed method compare to simple goal selection by uncertainty, hardness, or curiosity? How does it compare to hindsight experience reply methods?\n\n7) It is helpful to visualize the generated curriculum, i.e., the trajectory of selected goals during training in the 2D/3D grid.\n\n8) Computing L_judge requires to test whether the agent can finally achieve the sampled goals. Is it too computational expensive during training? \n\n---------------------------\n\nUpdate after rebuttal:\n\nThanks for the detailed reply from the authors! They answer most of my questions. I think this paper has a moderate contribution and thus will keep my \"weakly accept\" rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1lArVYnKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studied the problem of reinforcement learning under the sparse or dynamic rewarding environment. \nThe authors propose a promising automated curricula generation scheme, which considers goal validity, goal \nfeasibility, and goal coverage to construct useful curricula for the underlying agents. Rewards and loss functions \nare proposed individually for the solver, judge, and setter. Empirical studies demonstrate the capability of the \nproposed model in generating task curricula across several complex goals. In general, I believe the studied \nproblem is interesting, and the proposed model is promising.  However, I am not familiar with the curricula \ngeneration in the reinforcement learning setting. All I can say is the approach is intuitively appealing, the text is \nwell written and easy to follow, even for an outsider. A minor concern is about the experiments. Most of the \nexperiments are presented in the metric of Reward (% of best). It would be helpful if the authors can conduct \nsome illustrative case studies (better in different scenarios) to show some specific tasks, the generated task-oriented \ncurriculum, and provide some intuitive discussion to show the readers why the generated curriculum is beneficial \nfor the agent. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors introduce a method to automatically generate a learning curriculum (of goals) in a sparse reward RL setting, examining several criteria for goal setting to induce a useful curriculum.  The reviewers agreed that this was an exciting research direction but also had concerns about baseline comparisons, clarity of some technical points, hyperparameter tuning (and the effect on the strength of empirical results), and computational tractability.  After discussion, the reviewers felt most of these points were sufficiently addressed.  Thus, I recommend acceptance at this time.
Decision: Accept (Poster)