Adaptive Procedural Task Generation for Hard-Exploration Problems
Authors: Kuan Fang
Yuke Zhu
Silvio Savarese
L. Fei-Fei
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
curriculum learning
procedural generation
task generation

Abstract: We introduce Adaptive Procedural Task Generation (APT-Gen), an approach to progressively generate a sequence of tasks as curricula to facilitate reinforcement learning in hard-exploration problems. At the heart of our approach, a task generator learns to create tasks from a parameterized task space via a black-box procedural generation module. To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the similarity to the target tasks. Through adversarial training, the task similarity is adaptively estimated by a task discriminator defined on the agent's experiences, allowing the generated tasks to approximate target tasks of unknown parameterization or outside of the predefined task space. Our experiments on the grid world and robotic manipulation task domains show that APT-Gen achieves substantially better performance than various existing baselines by generating suitable tasks of rich variations.

Published as a conference paper at ICLR 2021
ADAPTIVE PROCEDURAL TASK GENERATION FOR -ARD-EXPHI HARD-EXPLORATION PROBLEMS
Kuan Fang
Yuke Zhu
Stanford University
UT Austin & Nvidia
kuanfang@stanfoord.cce
yukezêcs wtw.us.eeeae....o edu
Silvio Savarese
Li Fei-Fei Stanford University fei feifeili@stanforr..cc
Stanford University
ssilvio@stanford.edu
ABSTRACT
We introduce Adaptive Procedural Task Generation (APT-Gen), an approach to progressively generate sequence of tasks as curricula to facilitate reinforcement learning in hard-exploration problems. At the heart of our approach, a task gener- ator Iearns to create tasks from a parameterized task space via a black-box proce- dural generation module. To enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balanc- ing the agent's performance in the generated tasks and the similarity the target tasks. Through adversarial training, the task similarity is adaptively estimated by task discriminator defined the agent's experiences, allowing the generated tasks to approximate target tasks of unknown parameterization or outside of the predefined task space. Our experiments on grid world and robotic manipulation task domains show that APT-Gen achieves substantially better performance than various existing baselines by generating suitable tasks of rich variations
INTRODUCTION
The effectiveness of reinforcement learning (RL) relies on the agent's ability to explore the task environment and collect informative experiences. Given tasks handcrafted with human expertise, RL algorithms have achieved significant progress on solving sequential decision making problems in various domains such as game playing (Badia et al. 22200 Mnih et al. (215)) and robotics (OpenA] et al. 2019 Duan et al., 2016). However, in many hard-exploration problems (Aytar et al. 2018]] Paine et al. 2020), such trial-and-error paradigms often suffer from sparse and deceptive rewards, stringent environment constraints, and large state and action spaces.
A plurality of exploration strategies has been developed encourage the state coverage by RL agent (Houthooft et al. 20166] Pathak et al. [0177: Burda et al. 2019 Conti et al.) (2018). Although successes are achieved in geal-nea tasks and games of small state spaces, harder tasks often require the agent to complete a series of sub-tasks without any positive feedback until the final mission is accomplished. Naively covering intermediate states be insufficient for the agent to connect the dots and discover the final solution. In complicated tasks, it could also be difficult to visit diverse states by directly exploring in the given environment (Mailard al. 2014).
In contrast, recent advances in curriculum learning (Bengio et al.) 2009 Graves et al. 2017) aim to
utilize similar but casier datasets or tasks facilitate training. Being applied to RL, these techniques select tasks from a predefined set (Matüsen et al., 2019) or a parameterized space of goals and scenes (Held et al. 2018 Portelas et al.) 2019 Racanière al., 2020) to accelerate the performance improvement on the target task or the entire task space. However, the flexibility of their curricula is often limited to task spaces using low-dimensional parameters, where the search for suitable task is relatively easy and the similarity between two tasks can be well defined.
Project page: https::/kuanfaatieeeeee wtw.raaaanggggggggaaeeeee github. io/apttggn/

---

Published as a conference paper at ICLR 2021
task
noise input
task parameter
roliouts in target task
((:: 0g)
Turget
task discriminator
larget task
policy [s;
task progress
procedural generation
P(z)
D(v:8,)
generated task
lask retum
generated task (w))
policy 9p)
E
update generator
Figure I: APT-Gen learns to create tasks via black-box procedural generation module. By jointly training the task generator, the task discriminator, and the policy, suitable tasks are progressively generated to expedite reinforcement learning in hard-exploration problems.
In this work, we combat this challenge by generating tasks of rich variations as curricula using pro- cedural content generation (PCG). Developed for automated creation of environments in physics simulations and video games (Summerville et al. 2018 Risi & Togelius 2019 Cobbe et al. 2020), PCG tools have paved the way for generating diverse tasks of configurable scene layouts, object types, constraints, and objectives. To take advantage PCG for automated curricula, the key chal- lenge is measure the learning progress in order adaptively generate suitable tasks for efficiently learning to solve the target task. In hard-exploration problems, this challenge is intensified since the performance improvement cannot always be directly observed on the target task until it is close to being solved. In addition, the progress in complex task space is hard to estimate when there does exist a well-defined measure of task difficulty or similarity. We cannot always expect the agent to thoroughly investigate the task space and learn to solve all tasks therein, especially when the target task has unknown parameterization and the task space has rich variations.
To this end, we introduce Adaptive Procedural Task Generation (APT-Gen), approach to progres- sively generate a sequence of tasks to expedite reinforcement learning hard-exploration problems. As shown in Figure [ APT-Gen uses task generator create tasks via black-box procedural generation module. Through the interplay between the task generator and the policy, tasks are con- tinuously generated to provide similar but easier scenarios for training the agent. In order to enable curriculum learning in the absence of a direct indicator of learning progress, we propose to train the task generator by balancing the agent's performance in the generated tasks and the task progress score which measures the similarity between the generated tasks and the target task. To encourage the generated tasks require similar agent's behaviors with the target task, task discriminator is adversarially trained to estimate the task progress by comparing the agent's experiences collected from both task sources. APT-Gen can thus be trained for target tasks of unknown parameterization or even outside of the task space defined by the procedural generation module, which expands the scope of its application. By jointly training the task generator, the task discriminator, and the pol- icy, APT-Gen is able to adaptively generate suitable tasks from highly configurable task spaces to facilitate the learning process for challenging target tasks.
Our experiments are conducted on various tasks in the grid world and robotic manipulation domains. Tasks generated in these domains are parameterized by 6x to 10x independent variables compared to those in prior work (Wang et al. 2019 2020 Portelas al. 2019). Each task can have different environment layouts, object types, object positions, constraints, and reward functions. challenging target tasks of sparse rewards and stringent constraints, APT-Gen substantially outperforms existing exploration and curriculum learning baselines by effectively generating new tasks during training.
RELATED WORK
Hard-Exploration Problems. Many RL algorithms aim to incentivize the agent visit more diverse and higher-reward states. Methods on intrinsic motivation augment the sparse and deceptive environment rewards with additional intrinsic reward that encourages curiosity (Pathak et al.
2

---

Published as a conference paper at ICLR 2021
2017 Burda al.) 2019 Raileanu & Rocktäschel 2020) and state novelty (Conti al. 2018 Eysenbach et al. 2019). Another family of exploration techniques is derived from an information-
theoretical perspective maximizing information gain of actions (Houthooft et al. 2016: Sun et al.) 2011]. When human demonstrations are available, they can be used to facilitate an RI. agent to Visit similar states and transitions illustrated in the demonstrations (Vecerik et al. 2017: Nair
iet al., 2018; Zhu et al. 2018). A combination of these techniques has been applied to solve hard- exploration problems video game domains (Aytar al., 2018: Ecoffet et al., 2019). However, these methods have focused on learning in relatively simple and fixed environments, and usually can be ineffective in tasks where explorations are thwarted by stringent environment constraints or naively covering states does not lead to the task success.
Curriculum Learning. Curriculum learning utilizes alternative datasets and tasks to accelerate
the learning process of challenging target tasks (Bengio al. 22099 Graves ct al. 2017). To apply curriculum learning to RL, several recent works learn to adaptively select a finite set of easy tasks
(Narvekar et al., 2017; Svetlik et al., 2017] Riedmiller et al.. 2018 Peng al.. 2018 Czarnecki
eet al. 2018 Matiisen et 2019 Narvekar & Stone| 2019 Lin et al.] (019) or auxiliary rewards (Jaerberg et al.] [017] SSenn et al. 2019) hand-designed by human maximize a progress signal
defined on the target task. Parameterized tasks have been used to form a curriculum through the
configuration of goals (Forestier et al. 2017 Held et al. 2018 Racanière al., 2020), environment 2020, Baker et al. |0200] Portelas et al. (2019), and reward functions (Gupta) al.] 2018 Jabri et al. 2019). OpenAl et al. (2019) and Mehha al. (2020) propose to actively adjust the hyperparameters in physical simulators to alleviate the domain shift. Most of these works are designed for task spaces parameterized by several discrete or continuous variables, where the task space can often be thoroughly explored and the similarity between two tasks could be well defined in the parameter space. In contrast, our approach is able to effectively generate tasks parameterized by combination of high-dimensional discrete variables and several continuous variables. While most of these works focus on parameterizing a single aspect of the task environment, our approach learns to generate new tasks of rich variations with configurable initial state probability. transition probability, and reward function. (Sukhbaatar et al. (2018b). Florensa et al.] (2017), and [Sukhbaatar ett al. (2018a) propose use adversarial agent set goals of growing difficulties by reversely traversing the state space from the goal. While this is related to the adversarial training framework in this paper in principle, we apply our framework beyond goal-reaching and reversible task domains.
layouts (Wöhlke et al.
Procedural Task Generation. and robotics (Fisher et al. 2012;
Procedural generation has been widely used in computer graphics
Izadinia al., [017]] Majerowicz et al.) 2013) Izatt & Tedrake)
2020 Schwarz & Behnke
2020). While an increasing number of task sets have been designed to
benchmark and empower reinforcement learning research (Kolve et al. 2017 Xia et al. 2018 18 (Savva
et al. 2019; Yu et al. 2019; James et al.) 2020). the design and implementation of each task often require nontrivial human expertise and heavy engineering. A few recent works utilize the random procedural generation of tasks (Cobbe et al. 2020; Fang et al. 2018 Raileanu & Rocktäschel] 2020) Silver & Chitnis 2020). However, their generation algorithms are handcrafted with limited config-
urable features. Evolution strategies (Wang et al.] 2019 2020), automated procedures (Justesen
et al. 2018), and learning-based methods (Gravina et al., 2019 Khalifa et al.] [2200] [Bottrager &
TTogelius] 2020) have been proposed to automatically discover diverse games and task environments
for training RL agents. Instead of covering the entire task space or discovering a diverse set of policies an open-ended manner, our approach aims to train the policy to solve the target tasks of interest by utilizing the generated tasks.
3 ADAPTIVE PROCEDURAL TASK GENERATION
We consider a reinforcement learning problem involving a target task that the policy learns to solve and a parameterized task space that we utilize to generate new tasks. In practice, the parameterized task space can be created by simulation program or a configurable procedure to set up the envi- ronment by a human or a robot in the real world. The target task can be an instance of an unknown parameter a task outside of the task space, long as there exist shared properties and transferable knowledge between the generated tasks and the target task. This follows the general paradigm of macher-studen curriculum learning (Matiisen et al. 2019 Portelas et al. 2019), while we allow the task space to be parameterized by either continuous or discrete dingh-dieesional variables and we do not assume the target task has known parameterization by these variables.
3

---

Published as a conference paper at ICLR 2021
We propose Adaptive Procedural Task Generation (APT-Gen), approach for progressively gen- erating tasks in highly configurable task spaces as curricula. To enable curriculum learning for hard-exploration problems, our key insight is that the learning progress can be jointly estimated by how well the policy can solve the current generated tasks and how similar the generated tasks are to the target task. Starting with set of tasks that the policy can easily learn to solve, our approach progressively adapts the generated tasks towards the target task while maintaining their feasibility to the policy. As shown in Figure [ our approach creates tasks via a black-box procedural generation module by jointly learning the task generator, the task discriminator, and the policy.
3.1 PROBLEM FORMULATION
We consider each task as Markov Decision Process (MDP) denoted by tuple M (S, A. p, P, 7) with state space S, action space A, initial state probability P. transition proba- bility P. reward function R. and discount factor y. The task space T defines a finite or infinite number of MDPs of similar designs and properties. We use a multi liutti-dimensional parameter space W to represent the inter-task variation of T. Given a task parameter w E W, task M (w) can be instantiated in the task space by predefined mapping M(.). While a generic task space can be composed of fully configurable MDPs, this work we assume that all tasks share the same S, A and y such that all policies share the same input and output dimensions. In this case, each (w) is defined by distinct set of p. P, R parameterized by The target task M is either an instance of unknown parameter TŌ E W or a task outside of 'T but shares the same S and A.
Our goal is to learn policy π to solve the target task M. During training, the curriculum is formed as a sequence of task parameters {w:)a with index i for constructing the corresponding sequence of generated tasks M (w:). The agent collects rollouts by unrolling in both M and M (wx). Each rollout is denoted as which is composed of sequence of state $t, action a and reward r, each time step t. In the generated tasks, the wi of the source task is recorded alongside with the т. Given fixed budget of total collected steps in both task domains the objective is to maximize the policy's expected return E. EE r'rs] in the target task M.
3.2 ADAPTIVE GENERATION FOR HARD BARDD-XPPLORTION PROBLEMS
The interplay between the policy π T s; 0p) and the task generator G(z; 0,) is formulated in teacher-studen paradigm (Matisen et al. 2019), where 8,.. 8g are learnable model parameters and is a noise input used in deep generative models (Goodfellow et al. 2014). In contrast to prior work (Held et al. 2018 Matüsen et al. 2019| Portelas et al. 2019 Racanière et al. 2020) which rely on evaluating performance improvements directly the target task or the entire task space, we propose to define the indicator of learning progress using the expected return EIE, r'r and a task progress 1) to enable curriculum learning in hard-exploration problems. The expected return measures the policy's ' performance in the generated tasks sampled by G. While the task progress is a continuous score which represents the generated tasks' similarity to the target task. The definition and learning process of will be detailed in Sec. B.3 When both the expected return and the task progress reach the maxima, the generated tasks are supposed to be indistinguishable from the target task and π is trained to be the optimal policy for the target task.
The training requires a careful balance between the task progress and the expected return. A highly configurable task space potentially contains large amount of tasks that are infeasible or of similar difficulties with the target task. the task distribution of G moves too fast towards the target task, the policy can quickly be overwhelmed by difficult tasks and lose track of what tasks can be effectively learned. On the contrary, sticking to the tasks that can be solved by the current policy will retard the learning progress and overfit the policy the easy scenarios. Our approach maximizes the task progress subject to a target minimum expected return d as a chosen hyperparameter. Then the training of the task generator amounts the optimization problem:
max ma 0g
subject to E. En'ri2
(1)
where w ~ G represents the generation process jointly determined by p(z) and G and E,~G.-11 is shorthand notation for E (B) [+]] to represent the expectation over distribution of
4

---

Published as a conference paper at ICLR 2021
Algorithm 1 Adaptive Procedural Task Generation (APT-Gen)
Require: target task M, parameterized task space M(),, prior probability p(z), learning rate a 1: Initialize parameters oz. 0g. 0d, 01, 02, B
2: Initialize replay buffers Dy and D Dtarget
3: while not converged do
4: Sample 2 ~ p(z) and create the generated task M (w) with w = G(z; 0g)
5: Collect rollout Ta in M (w) using π [a|s; 0p) and store w and Tn in D3
6:
Collect rollout Ttorget in M using (Ta) s; 8p) and store Ttarget in D Dearget
7:
Update 0g <- 0d a Vo, (f,,, D Dargett Dg)
8:
Update 0, + 01 a Vo, L. D. [VV (w; 01) D(Toi 02))
9:
Update 02 + 02 a Vo, Em, Teu.Ta~00 DD, [V22 w; 02) Err(r,)
Update 0g Og a Ve, Mr,C(( (6g 81 3)
11:
Update B as described in Sec. 3.2
Update Op using the RI. algorithm with sampled batches from Dg and Dtarget
13:
end while
rollouts. By re-writing this optimization problem as Lagrangian, we obtain:
max (Ewwal [n) + 3(E 2 -)).
(2)
where is the Lagrangian multiplier that balances the task feasibility and the task progress. B can have delayed effect on the optimization problem since и and 77 learned at the same time. We adopt an automated procedure (Schulman et al. 2017) to adjust 3 adaptively when B rrrr 8 exceeds a threshold.
Since the gradients cannot be directly backpropagated to the task generator, we follow the practice Konda & Tsitsikhss (2000); Matüien al. (2019) and use two value functions estimate the two expectation terms respectively. By taking the input task parameter w, the progress value function (w; 01) estimates the task progress v and the return value function (w 02) estimates the pected return (()). |E, 7'ra]. where 01 and 0g are learnable model parameters. The two value functions are trained to fit the two expectation terms over the distribution of t with respect to 0 and 02 using rollouts collected in the generated tasks with the policy л. The training of the task generator becomes learning 0g to maximize the task generator loss:
(0, 01, 02, 3) = EE>p(=)) [V
+ ([))((((;00))0))))) - 6)].
(3)
3.3 ADVERSARIAL TRAINING OF TASK PROGRESS
The goal of the task progress 1 is to guide the task generator G to generate tasks similar the target task. Since the difficulty level and the task similarity cannot be defined by an objective metric in many complex task domains, we argue that v) needs to jointly adapt with G and æ when the task distribution and the policy constantly evolve over the course of training. Ideally, should satisfy two requirements: First, when the maximum 17 is achieved convergence, a generated task Mww and the target task M should be indistinguishable from the perspective of the policy и. Second, since a small change in an ill-posed task parameter space can completely alter the required agent's behaviors to solve the task, 77 needs to provide smooth signal to adapt G in the task space.
To this end, we estimate using task discriminator (r; 02) defined on the agent's experiences in the task environment, where Ag the learnable model parameter. It takes as input and learns to estimate the probability of the task M being the target task M conditioned on the rollout induced by the policy π. The task progress v) of the task parameter w can be defined E.y, [T;;; 0a)]- In this way, D forms an adversarial training framework (Goodfellow al. 2014) against G and n, which jointly determine the likelihood of
The task discriminator is required to comprehensively compare the given task with the target task in APT-Gen. Unlike prior work which aims to discriminate policies (Ho & Ermon! (0166) and physics parameters (Mehta al. 2020), D computes the prediction score by taking the overall MDP defini- tion into account. Therefore, D is designed to separately encode the initial state s and each transi- tion St, Ct Ft, 8t+1) 1) of step / to discriminate the initial state probability p. the transition probability
5

---

Published as a conference paper at ICLR 2021
P and the reward function R respectively. The prediction is computed using pooling function across all encoded features. The implementation details of D are described in Appendix
To train the task generator, collect rollouts from generated tasks To and the target task as Ttarget, stored in two replay buffers D, and Dtarget respectively, The training of D(T; (T: con-
ducted by minimizing discriminator loss (Goodfellow et al. 2014) to classify the task sources of the collected rollouts:
(dd Dtargets Dg) = - Saarget Drarget [log ( 0a))] - Er log (og( 0a))l (4)
=
In principle, to learn a that produces the exact MDP definition of M, we would require M to be instance of the task space T and the training data be collected by arbitrary æ to fully investigate differences in the two task environments. However, this could be neither computationally practical necessary. Given that our goal is find optimal policy x² to solve the target task, only need M and M (w) to be indistinguishable from the perspective of the policy. In practice, the rollouts are collected using the updated (a]s, w; (2)) with apsilon-greedy exploration (Sutton & Barto 201T). One could also encourage explorations (Pathak et al. (017) in the policy learning to efficiently distinguish the two tasks, which we leave out of the scope of this work.
The pseudocode of the algorithm is outlined in Algorithm[] The training alternates among updates of the policy, the task discriminator, and the task generator in the same loop. New rollouts are continuously collected from both the target task and the generated tasks using the updated æ. In this work, we equally collect experiences from the two sources to train the policy, while a smarter strategy of choosing between task sources can be further investigated in future work.
EXPERIMENTS
The goal of our experimental evaluation is to answer the following questions: 1) Can APT-Gen facilitate reinforcement learning in hard-exploration problems? 2) What tasks be generated by APT-Gen for a target task during training? 3) Can APT-Gen be applied to target tasks outside of the task space predefined by the procedural generation module?
4.1 TASKS
The experiments conducted in con- figurable task spaces: Grid-World and Manipulation. The two task spaces are con- figured by 74 and 31 parameters consist of a combination of discrete and continuous variables. Each task space contains various tasks that share the same state and action spaces but different en- vironment layouts, object types, object positions, constraints, and reward functions. The tasks from these task spaces can have stringent penalties and constraints such as lava regions and pitfalls which make it hard for the agent succeed survive. In contrast to many handcrafted hard-exploration tasks in prior work (Salimans & Chen 20188) which provide sub-task rewards (c.g. finding a key and opening a door), our tasks only provide a sparse positive reward when the final task is accomplished, which introduces extra challenges for the agent. As shown Figure[2, we design three target tasks of different complexities in each task space for evaluation. Details of the task design and the task parameterization can be found in Appendix A
Grid-Worid-A
Grid-Worid-B
Grid-Worl--C
255
Manipulation-A
Manipulation-8
Manipulation-C
Figure 2: Target tasks in the two domains.
4.2 QUANTITATIVE RESULTS
We evaluate the performance of the agent in target tasks using different methods. All methods are trained with a fixed budget total steps collected by the agent. Appendix we provide implementation details, hyperparameters, training and evaluation protocols.

---

Published as a conference paper at ICLR 2021
Grid-Worid-A
Grid-Worid-B
Grid-Worid-C
average return
1.0
average retur
1.0
average return
1.0
0.2
0.2
0.2
-0.2
-0.2
-0.2
20
120
140
60
160
100
thousand steps
thousand steps
thousand steps
Manipulation-A
Manipulation-8
Manipulation-C
average return
1.5
1.5
1.5
JLP-GHN
KCM
1.0
1.0
1.0
GoolCAN
RM
average
Randam
R1P
-0.5
-0.5
-0.5
20
40
100
120
140
160
40
60
80
10C
140
200
80 100 12C
200
thousand steps
thousand steps
thousand steps
Figure 3: Quantitative results of the performance of the agent in the target tasks.
Baselines. We compare APT-Gen with various baselines, including 3 exploration strategies, cur- riculum methods, and a model-free RL baseline. ICM (Pathak et al.] 2017). RND (Burda et al.) 2019) and RIDE (Raileanu & Rocktäschel] 2020) adversarially learn intrinsic motivations to en- courage exploration in the state space. Random uniformly samples from the task parameter space to create tasks. ALP-GMM (Portelas et al.) (2119) and GoalGAN (Held et al., (0188) use Gaussian Mixture Models and GAN to sample tasks as curricula without mechanisms to estimate the task distance and handle complex task spaces. DQN (Mnih et al.) 2013 Hessel et al. (018)) directly applies Q-learning in the target task. To have a fair comparison, we use the same architecture for the corresponding components and search for the optimal hyperparameters for each method.
Comparative Analysis. In Figure 3 we present the agent's performance the target tasks by using different methods. The average return across 5 independent runs is reported and the shaded area indicates the standard deviation. To have fair comparisons, the x-axes of APT-Gen and curricu- lum learning baselines (Random, GoalGAN, ALP-GMM) indicate the total steps collected from the target and generated tasks, while x-axes of other baselines (DQN, ICM, RND, RIDE) indicate the steps collected from only the target task. In all scenarios, our approach achieves superior perfor- mance comparing with baseline methods. In Grid- -oldddoordd d tasks, APT-Gen successfully trains the agent find keys in separate locations and access different rooms in the right sequential order. In the Manipules in on tasks, APT-Gen enables the agent to solve the puzzle by moving around the obstacles the correct order without causing collisions. Especially, in Manipulat at on--C. our agent develops an effective strategy that first moves away from the target object away to yield the path for the obstacle to leave and then pushing it back towards the goal to complete the task.
Most baseline methods fail in hard tasks that require sequential problem solving over longer hori- zon, although some can achieve comparable results in easier scenarios (i.e. when there is only one room and the environment is mostly empty). ICM, RND, and RIDE demonstrate effective explo- rations when the environment relatively simple, but the agent is often thwarted by the penalties caused by constraints in the environment. Without any reward shaping for sub-tasks, naively reach- ing to the intermediate states (e.g. finding the keys) does not yield any immediate reward unless the goal is reached at the end of the same episode. Without mechanisms estimate the task similarity and to handle complex task spaces, curriculum learning baselines like ALP-GMM and GoalGAN fail to produce useful tasks that share similar challenges with the target tasks.
Out-of-Space Task. To demonstrate APT-Gen's performance in target tasks that are outside of the predefined task space, we train the model to solve different robotic manipulation task while still generating tasks from the task space defined in Sec. 4.1 The target task shares the same state and action spaces with the predefined task space, but the table has a different shape and a variety of static objects are placed on the table as environment constraints. As shown in Figure 4 APT-Gen efficiently learns to solve the oft-of-space task while baseline methods take much more steps or completely fail to learn. Qualitative results of out-of-space tasks will be discussed in Sec. 4.4
7

---

Published as conference paper at ICLR 2021
Manipulation-OOt--SSSaacec
15c
45k
2.
AUNGON
ATT-Cen (Dars)
Dch
ICH
Cour CAN
IND
Random
RCC
0.5
thousand steps
20k
40K
Mamputatio
Figure 4: Results on task that is out of the predefined task space.
Ablation Study
return
Model
Manipu -uulod--Spicc
1.0
w/ trog
0.
20
120 thousand steps
100
140 100
Figure 5: Results of ablation study in Manipulation-..
Figure 6: Progression of the generated tasks for various target
tasks in the two task domains and the out-of-space task.
4.3 ABLATION STUDY
We conduct ablation studies on the target task of Manipulatiinn and analyze the effect of the indicator of learning progress. As shown in Figure the performance degrades when using only either the expected return or the task progress the learning progress. The generation often adapts fast towards the target task when only counting on the task progress, although easier tasks can still emerge during the adaptation. When generating tasks only in response to the expected return, the policy is overwhelmed by easy tasks, which retards the learning progress of the target task. In these ablations, although the knowledge gained from such tasks sometimes could still benefit the training. they do not form the ideal curricula for solving the target task. As result, the performances achieved by these ablations are inferior to the performance of the full model of APT-Gen.
4.4 PROGRESSION GENERATED TASKS
We present qualitative results of the generated tasks in Figure 6 ated tasks and the target task (marked by green borderlines) with the number of collected en- vironment steps and the task name shown the upper right of the images. When learning for Grid-World-A, the task generator first creates easy tasks in which the goal (green tile) is close the starting position of the agent (red triangle) with few obstacles in between. Between 1.5k and 30k steps, the task generator gradually shifts the goal to the bottom right corner as the target task. At the same time, walls (grey tiles) are created form rooms enclosing the goal. At around 45k steps, the door is placed on the wall to lock the room and the key is placed in a further location in the labyrinth. The agent learns to grab the key and open the door in the target task after learning solve the generated task since the solutions now share a similar routine. In Man Manipule pullpplllt at on-C, gener- ated tasks start with a clear table surface and small distance between the target object (blue can) and the goal (cyan circle). As the agent learns to tackle such easy scenarios, green can is placed between as an obstacle while the goal grows larger to make sure the agent can still complete the task. At 60K steps, the environment further morphs towards the target task as more obstacles being added to the scene and the goal shrinking to the correct size.
Each row shows three gener-
In the out-of-space task, although the more complicated table and objects cannot be generated by the procedural generation module of limited capabilities, APT-Gen gradually leams to outline the scene of the target task by utilizing the available elements such cuboids and empty holes. By interacting with the environment and comparing experiences in both task sources, APT-Gen trains the policy to solve the out-of-space task by approximating the challenges in the target task.
8

---

Published as a conference paper at ICLR 2021
5 CONCLUSION
To expedite reinforcement learning in hard-exploration problems, we present Adaptive Procedural Task Generation (APT-Gen) to generate suitable tasks via black-box procedural generation modules as curricula. By jointly training the task generator, the task discriminator, and the policy, APT-Gen achieves superior performances to existing exploration and curriculum learning baselines in vari- ous target tasks in the grid world and robotic manipulation domains. By adversarially training the task discriminator to estimate the similarity between the target task and generated tasks, APT-Gen demonstrates be effective for target tasks of unknown parameterization and out of the predefined task spaces, which expands its potential use case. We hope this work could encourage more endeav- in utilizing procedural content generation for reinforcement learning.
Accnowledgement: We acknowledge the support of Toyota (118678-331-UDA))) cloud credits. We would like to thank Roberto Martín-Martin Austin Narcomey, Sriram Somasun- daram, Fei Xia, and Danfei Xu for feedback on an early draft of the paper.
and HAI-AWS
REFERENCES
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, 2018.
Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. arXiv preprint arXiv:2003. 2020.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In International Conference on Learning Representations, 2020.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In International Conference on Machine Learning, 2009.
Philip Bontrager and J. Togelius. Fully differentiable procedural content generation through gener- ative playing networks. ArXiv, abs/2002.05259, 2020.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. International Conference on Learning Representations, 2019.
Maxime Shevvlier-Boisverf Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https::/giitee www.///wwww..c///ccac..oo..o hww.://emmmmmmmmmmiiiiiii.a 2018.
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural gener- ation benchmark reinforcement learning. International Conference on Machine Learning, 2020.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via pop- ulation of novelty-seeking agents. In Advances in Neural Information Processing Systems, pp. 5027-5038, 2018.
Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http::/pybbllet.orrg 2016-2019.
Wojeiech Czarnecki, Siddhant M. Jayakumar, Max Jaderberg, Leonard Hasenclever, Yee Whye Teh, Nicolas Manfred Otto Heess, Simon Osindero, and Razvan Pascanu. Mix&match agent curricula for reinforcement learning. In International Conference on Machine Learning. 2018
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329-1338, 2016.

---

Published as a conference paper at ICLR 2021
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Go-explore: new approach for expllorxlooion problems. arXiv preprint arXiv:1901.1099) 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representations, 2019.
Kuan Fang, Yuke Zhu, Animesh Garg, Andrey Kuryenkov, Viraj Mehta, Fei-Fei, and Silvio Savarese. Learning trientreented grasping for tool manipulation from simulated self-suuprvision. In Robotics: Science and Systems, 2018.
Kuan Fang, Yuke Zhu, Animesh Garg, Silvio Savarese, and Li Fei-Fei. Dynamics learning with cascaded variational inference for multi-step manipulation. In Conference on Robot Learning, 2019.
Matthew Fisher, Daniel Ritchie, Manolis Savva, Thomas Funkhouser, and Pat Hanrahan. Example- based synthesis of 3d object arrangements. ACM Transactions on Graphics, (16):1111, 2012.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse cur- riculum generation for reinforcement learning. In Conference on Robot Learning, 2017.
Sébastien Forestier, Yoan Mollard, and Pierre-Yvel Oudeyer. Intrinsically motivated goal explo-
ration processes with automatic curriculum learning. ArXiv, abs/1708. 02190, 2017.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, 2014.
Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In International Conference on Machine Learning, 2017.
Daniele Gravina, A. Khalifa, Antonios Liapis, J. Togelius, and Georgios N. Yannakakis. Procedural content generation through quality diversity. In IEEE Conference on Games, 2019.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised meta-
learning for reinforcement learning. ArXiv, abs/1806.04640,, 2018
David Ha. Reinforcement learning for improving agent design. arXiv preprint arXiv: 181.v31810 2018.
David Held, Xinyang Geng, Carlos Florensa, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference on Machine Learning, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI Conference on Artificial Intelligence, 2018
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
Rein Houthooft, Xi Chen, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances Neural Information Pro- cessing Systems, pp. 1109-1117,201
Jialei Huang, Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas Guibas, and Hao Dong. Generative 3d part assembly via dynamic graph learning. In Advances in Neural Information Processing Systems, 2020a.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-ag control. In International Conference on Machine Learning, 2020b.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.053: 036700366, 2015.
10

---

Published as a conference paper at ICLR 2021
Hamid Izadinia, Qi Shan, and Steven M Seitz. Im2cad. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5143,,,4,, 2017.
Gregory Izatt and Russ Tedrake. Generative modeling of environments with scene grammars and variational inference. In International Conference on Robotics and Automation, pp. 6891-6897. IEEE, 2020.
Allan Jabri, Kyle Hsu, Ben Eysenbach, Abhishek Gupta, Alexei A. Efros, Sergey Levine, and Chelsea Finn. Unsupervised curricula for visual meta-reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
Max Jaderberg, Volodymyr Mnih, Wojciech Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference on Learning Representations, 2017.
Stephen W. James, Z.
Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5:3019- 3026, 2020.
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint arXiv: 10729, 2018.
A. Khalifa, Philip Bontrager, Sam Earle, and Togelius. Pegrl: Procedural content generation via reinforcement learning. ArXiv, abs/2001.09213. 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv::442.66990 2014.
James N Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barkkkkkaa Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences, 2017.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems, 2000.
Xingyu Lin, Harjatin Singh Baweja, George Kantor, and David Held. Adaptive auxiliary task weighting for reinforcement learning. In Advances Neural Information Processing Systems, 2019.
Odalric-Ambrym Maillard, Timothy A Mann, and Shie Mannor. How hard is my mdp?" the distribution-norm to the rescue". In Advances in Neural Information Processing Systems, 2014.
Lucas Majerowicz, Ariel Shamir, Alla Sheffer, and Holger H Hoos. Filling your shelves: Synthe- sizing diverse style-preserving artifact arrangements. IEEE Transactions on Visualization and Computer Graphics, 20(1 150-5515515 2013.
Tambet Matiisen. Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum leamn ing. In IEEE transactions on neural networks and learning systems, 2019.
Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher Joseph Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602. 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(754)))599-55533 2015.
11

---

Published as a conference paper at ICLR 2021
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In JEEE International Con- ference Robotics and Automation. IEEE, 2018.
Narvekar, J. Sinapov, and P. Stone. Autonomous task sequencing for customized curiicluum design in reinforcement learning. In International Joint Conference on Artificial Intelligence, 2017.
Sammit Narvekar and Peter Stone. Learning curriculum policies for reinforcement learning. In Pro- ceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jadwiga Tworek, Peter Welinder, Lilian Weng. Qi-Ming Yuan, Wojciech Zaremba, and Lefei Zhang. Solving rubik's cube with robot hand. ArXiv, abs/1910.07113, 2019.
Tom Le Paine, Caglar Gulcehre, Bobak Shahriari, Misha Denil, Matt Hoffman, Hubert Soyer, Richard Tanburn, Steven Kapturowski, Neil Rabinowitz, Duncan Williams, al. Making effi- cient use demonstrations to solve hard exploration problems. In International Conference on Learning Representations, 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference Machine Learning, 2017.
B. Peng, J. MacGlashan, R. Loftin, M. Littman, D. Roberts, and Matthew E. Taylor. Curriculum design for machine learners in sequential decision tasks. IEEE Transactions Emerging Topics in Computational Intelligence, 2:268-277, 2018.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep in continuously parameterized environments. In Conference on Robot Learning, 2019.
Sébastien Racanière, Andrew Kyle Lampinen, Adam Santoro, David P. Reichert, Vlad Firoiu, and Timothy P. Lillierap. Automated curriculum generation through setter-solver interactions. In International Conference on Learning Representations, 2020.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 2015.
Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for seneddurall-geneerat environments. In International Conference on Learning Representations, 2020.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. In International Conference on Machine Learning, 2018.
Sebastian Risi and Julian Togelius. Procedural content generation: From automatically generating game levels increasing generality in machine learning. arXiv preprint arXiv: 1:XXx:9911.1309 2019.
Tim Salimans and Richard Chen. Learning montezuma' 's revenge from single demonstration. arXiv preprint arXiv: 1888.23338) 2018.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun. Jitendra Malik, al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9339- 9347, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, 701117077777777
12

---

Published as a conference paper at ICLR 2021
Max Schwarz and Sven Behnke. Stillleben: Realistic scene synthesis for deep learning in robotics. In Interaational Conference on Robotics and Automation, pp. 10502-10508. IEEE, 2020.
William B Shen, Danfei Xu, Yuke Zhu, Leonidas J Guibas, Li Fei-Fei, and Silvio Savarese. Situa- tional fusion of visual representation for visual navigation. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 2881-2890, 2019.
Tom Silver and Rohan Chitnis. Pddlgym:
Gym environments from pddl problems. ArXiv,
abs/2002.06432,, 2020.
Sainbayar Sukhbaatar, Emily Denton, Arthur Szlam, and R. Fergus. Learning goal embeddings via self-play for hierarchical reinforcement learning. ArXiv, abs/ 1111118111.0.... 2018a.
Sainbayar Sukhbaatar, Ilya Kostrikov, Arthur Szlam, and Rob Fergus. Intrinsic motivation and automatic curricula via asymmetric self-play. In International Conference on Learning Represen- lations, 2018b.
Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. Procedural content generation via machine learning (pegml). IEEE Transactions on Games, 10(3):257-270. 2018.
Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In International Conference on Artificial General Intelli- gence. Springer, 2011.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. Cambridge, MA: MIT Press, 2011.
M. Svetlik, Matteo Leonetti, J. Sinapov, Rishi Shah, Nick Walker, and P. Stone. Automatic cur- riculum graph generation for reinforcement learning agents. In AAAI Conference on Artificial Intelligence, 2017.
Yonglong Tian, A. Luo, Xingyuan Sun, K. Ellis, W. Freeman, J. Tenenbaum, and Jiajun Wu. Learn- ing to infer and execute 3d shape programs. In International Conference on Learning Represen- fations, 2019.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.088177 2017.
Renmin Wang. J. Lehman, A. Rawal, Jiale Zhi, Yulun Li, J. Clune, and K. Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. ArXiv, abs/2003.08536 2020.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Poet: open-ended coevolution of en- vironments and their optimized solutions. In Genetic and Evolutionary Computation Conference, 2019.
Jan Wöhlke, Felix Schmitt, and Herke van Hoof. A performance-based start state curriculum frame- work for reinforcement learning. In Proceedings of the 19th International Conference on Au- tonomous Agents and MultiAgent Systems, pp. 1503-1511, 2020.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9068-9079, 2018.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, 2019.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunaasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, et al. Reinforcement and imitation learning for diverse visuomotor skills. In Robotics: Science and Systems, 2018.
13

---

Published as a conference paper at ICLR 2021
A ENVIRONMENT DETAILS
We describe the details of the parameterization, the state space, the action space, and the rewards of tasks from each task space. Examples of randomly generated tasks shown in Figure 7 Most random tasks are completely infeasible or trivially easy without a learned task generator.
Grid-Worid
Manipulation
Figure 7: Examples of randomly generated tasks from the two task domains.
A.1 GRID-WORLD.
The Grid-Word domain is based on the popular benchmark for RL research (Chevalier Boisvvrrt et al. 2018). In each task, the agent (red triangle) chooses discrete actions to reach the goal (green tile) by navigating around a grid surrounded by border walls and interacting with the objects therein. Diverse labyrinths the agent hits these obstacles it receives penalties or terminates the episode. Paired keys and doors may be placed in the labyrinth for the agent make of. To open a door, the agent needs to first find the key of the corresponding color. The agent perceives the whole map at each step but has no prior knowledge of the functionality and rewards of the different tile types and the goals. Each episode has finite horizon of 50 time steps. Although this task domain has relatively shorter horizon than some of the similar hard-exploration tasks (Salimans Chen 2018), no sub- task reward (e.g. finding a key and opening a door) is provided as in those tasks in prior work, which introduces extra challenges the agent.
be constructed with walls (gray tiles) and lava regions (orange tiles). When
Parameterization. The environment is 10 x 10 grid which consists of 8 x 8 configurable grid and the border walls surrounding the area. The environment parameterized by 74 independent variables including 8 x 8 array that represents the type of each tile in the grid and a 10-dimensional vector that represents the coordinates of the objects the environment (the goal. two doors, and two keys). The coordinates of the doors, the keys, and the goal in the Grid-World environment defined as continuous variables such that they can be smoothly adapted across space. The objects are placed to the closest tile the computed coordinates in the environment. To enable gradient descent in the task generation pipeline, the 8 x 8 array is converted to a 8 x 8 x 3 array where the last dimension represents the logits of the tile category. The objects can be initialized to one of the 10 10 locations. If the chosen location is not on the border walls or is already occupied by previous object, will be placed on an empty tile there. Otherwise, the object will not appear in the environment.
State and Action Spaces. The agent receives the state that includes the location of the agent as 2-dimensional vector, local view of the surrounding tiles as 7 x 7 array centered at the agent's lo- cation, and the relative positions of the objects (the goal, the doors, and the keys) as 10-dimensional vectors. If an object does not appear in the grid, the relative position will be set to (0, 0). Starting at the upper left corner of the grid, the agent chooses to move along one of the four directions by one tile at each time step. If the next tile along the chosen direction is empty, the agent will be moved
there. If a key is on the next tile, the agent will take the key and the corresponding door of the same color with the key will disappear.
14

---

Published as a conference paper at ICLR 2021
Rewards. If the agent reaches the goal, the agent will succeed with goal-reaching reward of 1. If the agent hits a wall, it will stay still and receive a penalty of 0.001. If the agent hits a lava region or a closed door, the episode will terminate with penalty of 0.5. In addition, time penalty of 0.001 is added to the return at each time step. If the agent is initialized on the goal, the episode will immediately terminate with penalty of-1.
A.2 MANIPULATION.
The Manipulat at io tasks involve a simulated robotic arm that interacts with multiple objects in configurable table-top environment. The task is adapted from from[Fang et al. (2019) and simulated by a real-time physies engine (Coumans & Bai, 2019-2019). The robotic arm has the same mesh and physical parameters of real-world Sawyer robot. The table is composed of configurable building blocks. Each block can be a flat surface, a pitfall, or a roadblock. I to 3 movable objects and a goal (cyan circle) are placed on the table at the beginning of each episode. The landscape of the table and the placement of the objects jointly form a puzzle in each task. The robot is asked to push the designated target object (blue can) to the goal using tool. To achieve the goal, the robot needs to move around objects that block the way and avoid pitfalls and roadblocks. Each episode has finite horizon of 15 steps and will terminate early if objects collide or fall off the table.
The table-top environment is formed by building blocks of a side length of 15 cm. The scene is parameterized by totally 31 independent variables including 6 x 4 array that represents the types of the building blocks, 6-dimensional vector that represents the initial position of the three objects, and a scalar that represents the range of the goal ranging from 10 cm to 50 em. Same in Grid-World the 6 array is converted to a 6 x 4 3 array where the last dimension represents the logits of the type of the building block. The objects are initially placed on the 90 em 60 em table surface. Since it is in general hard for GANs and Gaussian functions to directly capture the distribution of valid positions on complex table surfaces in presence of holes and obstacles, we instead adopt a structured generation process. The model first computes a x 24 array that represents which tile each object will be placed on. Continuous offsets ranging from -2 em to em are then added to the x, y coordinates of the tile. If an object is initialized to a non-flat tile or the same tile with a previous object, it will not show up on the table.
Parameterization.
State and Action Spaces. The agent receives the state that includes a continuous vector represent- ing the positions of the objects and the goal and an array representing the surrounding landscape of each object. The surrounding landscape is represented by 3 x 4 array where the first dimension corresponds to the object index and the second dimension corresponds to the height of the neighbor-
ing points. If the object does not appear on the table, its position will be set to (-1, -1). The goal scale is not provided to the agent as part of the state and requires the agent to figure it out through trial-and-error. The robot chooses from an action space of 12 discrete actions that represent which of the three objects to push and which of the four directions to push towards.
Rewards. The episode will terminate with a goal-reaching reward of 1, if the target object reaches the center of the goal which is circle of radius of r 10 em. Once the target object enters the goal region of a radius of r2, it will receive a progress reward d/(r2 - r)) where d is the moving distance towards the goal. The 72 controlled by the task parameter ranging from 10 cm 60 cm. If any object falls off the table, the episode will terminate with a penalty of -0,2. If objects collide with the roadblocks or other objects, the episode will terminate with a penalty of -0.1.
B IMPLEMENTATION DETAILS
B. NETWORK ARCHITECTURES
Neural networks are designed for the two task spaces respectively to tackle their different state spaces, action spaces, and task parameter spaces. The network architectures for each task space are shown in Figure and Figure P The design of the task discriminator is shown in Figure I0 These neural networks are implemented with fully-connected (FC) layers, convolutional (Conv) layers, av- erage pooling (Pool) layers, and flatten operations (Flatten). The names of different modalities of the states and the task parameters in each task domain indicated in the figures as they are sepa- rately processed or produced. All models are implemented in Tensorflow and the hyperparameters chosen through random search.
15

---

Published as a conference paper at ICLR 2021
W
H or 1f
FC
goal
kays
doors
grd
FC
Conv
FC
Fietten
Deconv
FC
FC
FC
Conv
FC
FC
FC
FC
FC
FC
Cony
gool
keys
doors
grd
goal
keys
doors
grid
Z
W
Q-Network
Task Generator
Value Network
Pn (sp)
( de
Concal
FC
Flatton
FC
FC
FC
Conv
FC
FC
FC
FC
goal
keys
doors
agent
agent
Initial State Encoder
Step Encoder
Figure 8: Network architectures for Grid World.
Os 4.)
" or BIE, 7r
goel scale
goal position
object position
table
FC
Conv
Deconv
Flattinn
Cony
Deconv
Conv
FC
FC
FC
FC
gool
objects
landscape
goal
goal postiion
object
table
FC
5
Q-Network
Task Generator
Value Network
C (s.)
Peep (sp ap S..))
Concal
Fieiten Conw
FC
goal
objects
landscape
objects
objects
5,,7
Initial State Encoder
Step Encoder
Figure Network architectures for Manipulation
16

---

Published as a conference paper at ICLR 2021
Pool
FC
FC
FC
0
P stp
Stop
51
(5 "r 52)
(5p ae Fe2
Task Discriminator
Figure 10: Network architecture of task discriminator.
Q-Network. The Q-network (Mnih et al. 2015) takes input as the current state and predicts the Q values for each discrete action. In both task spaces, we encode each vector in the state using a single fully-connected layer and encode the grid using two convolutional layers followed by fully- connected layer. The information of different modalities is merged at the end with a cully-connected layer at the end. The fully- fully-comnected layers for each modality are 64-dimensional and the final layer is 128-dimensional. Each convolutional layer has a 3 x 3 kernel of 16 channels and stride of 2. Each layer is followed by rectified linear unit (ReLU). Our model and the baseline methods the same Q-network in each task space.
The design of our task generators is inspired by generative adversarial networks (GANs) in other domains (Goodfellow et al.) 20014] Radford et al. 2015). It first encodes the input noise using 64-dimensional fully-connected layer and then separately produces each modality. The vectors are computed by cully-connected layers. We apply sigmoid function at the output layer of each continuous modality and scale the output by the range of each modality which defined by the task space. To compute the arrays (grid and table) of categorical values, we use stream consists of two deconvolution layers, convolutional layer, and softmax layer. The convolutional layers and the deconvolution layers both have 3 x 3 kernels of 16 channels and stride of 2. Following the practice of (Radford et al. 2015), batch normalization (Ioffe Szegedy 2015) used all layers of the task generators.
Task Generator:
Task Discriminator: As shown in Figure 10 the task discriminator separately encodes the initial state $1 using the initial state encoder dimt and each transition step (st, d,, rt, 8441) ) of time step t using the step encoder dstep- Fully-connected layers are used to predict a score for each encoding. Average pooling is applied the predicted scores the output layer. The architectures of dimit and @step for the two task spaces are shown in Figure R and Figure 9 The initial state encoders are similar the Q-networks with minor modifications dg, and 8t+1 and then merge the information using another
reduce the depth of the networks. The
step encoder separately encodes fully-comnected layer. the step encoder, we do encode the modalities in the states which do not change across time. The hyperparameters of these layers are the same as that in the -entworks.
Value Networks. The value networks that are used predict the task progress 7) and the expected return E. E. 7rrx] share the same architectures in each task space. The designs of the value networks are similar to the initial state encoders except that the inputs are the task parameter w instead of $1-
B.2 ADAPTIVE ADJUSTMENT OF THE KKT MULTIPLIER
We adopt an automated procedure (Schulman et al., 2017) adjust | balance the task progress and the expected return. The average expected return on the generated tasks is constantly evaluated in the replay buffer. When the average expected return exceeds a predefined threshold, /3 will be scaled accordingly. Otherwise, it will remain the same. Maximum and minimum values of 3 are chosen by hand to prevent the optimization to explode. Specifically, we use d = 0.5 with a tolerance of 0.1. If E 2(rr] < 0.4, / t- min(3 (3 2, 8); if EE Y'rl] > 0.6. 3 +- max(8/2 1/8). In practice, we find that the performance of the policy not sensitive to the choice of 8 between 0.3 and 0.7. An ablation study on the influence of 8 can be found in Sec.
17

---

Published as a conference paper at ICLR 2021
30k
45k
Grid Worldii
15k
30k
45k
Gind-World-B
Grid-Worid-C
Figure Progression of the generated tasks for target tasks Grid-World
B.3 TRAINING
Solver and Hyperparameters. For all experiments, we use the ADAM optimizer (Kingma & Ba 2014] with learning rate of 3 10-4, 81 0.9, B2 0.999 and the batch size of 128. Totally 10,000 environment steps collected to initialize the replay buffers. We only sample from the most recent 10,000 steps in each replay buffer to train the task discriminator to encourage it to estimate the return of the updated policy. After collecting each environment step. the policy is trained for 10 iterations and the other models are trained for iteration. The is updated every 500 environment steps according the average returns of the most recent 50 episodes. All hyperparameters chosen by random search.
Stability. One of the technical challenges is that the training could be unstable when the policy learns to solve a changing distribution of tasks, which a common issue in related problems such continual learning (Kirkpatrick al. 2017). This issue is intensified in the Grid World task space since the reward is more sparse. While fundamentally resolving this issue of reinforcement learning con-stationary task distribution is beyond the focus of our focus this paper, we propose two simple techniques to improve the stability of the RL agent during training. First, as the stability is often affected by overestimation the Q-values, we clip the next Q-values in the DQN algorithms by the maximum and minimum episodic return observed in the replay buffer. Second, we restore the policy network's s parameters the last round of evaluation. We found that these two modifications to the original DQN algorithm significantly improves the stability of the training in APT-Gen and baseline methods.
performance in the target task has reached above but drops since
Computation and Runtime. During each run, the method is trained on a single NVIDIA GeForce GTX1080 Ti GPU and 8 CPU cores with 32 GB memory. The overall data collection and training time of each run takes around 2 hours for Grid World and 30 hours for Robotic Manipulation.
B.4 EVALUATION
The evaluation is conducted after collecting every 1,000 environment steps. In all quantitative eval- uation in this work, each data point is evaluated for 50 episodes. The means and the error bars of the episode returns are computed across 5 different runs training the same method. To have fair comparisons, all methods are trained with fixed budget of total steps collected by the agent.
18

---

Published as a conference paper at ICLR 2021
C ADDITIONAL RESULTS ON TASK PROGRESSION
In Figure II and Figure 12 we present more examples of the generated tasks the two domains. Each row shows three generated tasks and the target task (marked by green borderlines) with the number of collected environment steps and the task name shown on the upper right of the images. As shown in the figure, the tasks generated by APT-Gen form curricula of ascending difficulties and similarity with the target task during training.
20k
40k
60k
Manipulation-A
Manipuiation-8
20k
40K
60k
Manipulation-C
Figure 12: Progression of the generated tasks for target tasks in Manipulation
D ABLATION ON MINIMUM EXPECTED RETURN
In this section, we conduct an additional ablative experiment demonstrate the influence of the minimum expected return 8. We train APT-Gen in Mani coniipulationse with different choices of 8 ranging from -0.1 1.1 for runs each. The training curves are shown in Figure 13 where we report the average returns achieved by the policy and the shaded area indicates the standard deviation across different runs. Using 0.3 < 8 S 0.7, the learned policies can achieve similar performances. Using 5 outside of this range, the feasibility of the generated tasks becomes less distinguishable by the threshold and the performance the policy decreases.
average return
1.5
= -0.1
1.0
5 0.3 = 0.5
0.5
6 0.7
0.0
5 0.9
1.1
-0.5
20
40
60
100
120
140
160
thousand steps
Figure 13: Ablation the minimum expected return.
19

---

Published as a conference paper at ICLR 2021
E DISCUSSION AND FUTURE WORK
While APT-Gen able to procedurally generate tasks as curricula for various target tasks, it does have limitations. These limitations will lead to several exciting future directions. First, without structured representation of the task, it is often hard for the task generator to produce task parameters that will result in useful and feasible tasks. Since our task generator directly uses convolutional layers to produce the category of each tile, it usually takes a large number of steps before it learns to properly form the walls of rooms and place the doors the right place. This can potentially be improved with recent advances of deep generative models using geometric structures (Tian et al.) 2019] Huuan al., (020a). Second, this work focuses on learning to generate tasks as curricula for single target task. Extending APT-Gen to generate tasks for multi-task learning and meta- learning would be promising future direction. Lastly, while the parameterized task spaces this work contain rich variations of the enviromment and the objects therein, the structure and properties of the robotic agent are fixed. It would be interesting to combine APT-Gen with recent works on morphology (Ha 2018 Huang al. 2020b) to adapt the agent the task.
20


Reviews:
[{"review_id": "txoGle0yoSJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper tackles the problem of facilitating RL agents' learning in sparse reward, hard-exploration problems. The authors approached this challenge by generating a curriculum of tasks needed to finish the originally assigned task. Though using other auxiliary tasks to assist RL training has been heatedly discussed, their method has its own highlights and novelty. Pros and Cons are listed as follows:\n\n[+] The overall framework is illustrated with clear explanations. The formulation of adversarial learning over the unrolling trajectories $\\tau$ is the key for connecting the task generator and discriminator in this case and is interesting. The implicit formulation of task generation provided the prerequisite for solving out-of-scope tasks (when only $\\mathcal{S}$ and $\\mathcal{A}$ are shared).\n\n[+] The experiments are well-designed with illustrative figures and analyses. Through the analysis, we see the advantage of having such curriculums do benefit agents' learning compared to previous methods. The ablative study also gave an intuitive explanation of the balance between having more difficult tasks in the curriculum and agents' performance on the original task.\n\n[-] Although the general idea of generating easier tasks to facilitate agents' learning in complex tasks is intuitive, how the authors  justify their method has the same property is not fully justified in my mind. As far as I'm understanding, the authors used a hyperparameter $\\delta$ to avoid making task generator $G$ moves too fast towards the target task, however, as was mentioned in Appendix B.2, the selection of the KKT multiplier is not critical, which causes some confusion on how the step-by-step difficulty enhancement was achieved. This result discussed is also having me question what if the task and reward scale is completely different, then would the effect of alternating $\\beta$ still be small?\n\n[-] The second problem is perhaps with the starting states in this training process. As the generator and discriminator was trained to adaptively fit the current RL agent's performance, the initiation of task generation should be important as a completely different setting of environment even with same state space and action space should be harmful for RL training in my mind(since there might be conflicts in $D_g$ and $D_{target}$ ). Then in this case, for example in a simpler scenario,  will this be a problem that affects RL training? Do we need to add constraints to the task generator at the beginning? This further leads to the question of whether the performance boost was provided by sampling on two similar environments (or even same as $G$ moves fast to the assigned task) and leading to more trajectories for RL training, or was the performance boost provided by the curriculum mentioned.\n\nBecause of the aforementioned questions, a borderline decision was made for now, please check the questions in the [-] to clear any misunderstandings.\n\n=====================================================================================================\n\nThe authors addressed my concerns in the discussion period and I therefore created my score to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "f2YdLXhLkxc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper proposes a framework to progressively generate a sequence of tasks as curricula for hard-exploration tasks. It demonstrates better performance on MiniGrid and robot manipulation tasks compared with exploration baselines and curriculum goal generation baselines.\n\nPros\n+ The paper is well-written and easy to follow. It provides enough implementation details.\n+ This paper tackles a hard problem that incorporates exploration, policy learning, and curriculum task generation. The adversarial training solution is clean. It proposes a new way to learn the similarity score by encoding the MDP.\n\nQuestions & Concerns:\n- The optimization problem on page 4.\n 1. it's not an equation, it's an optimization problem. There are many references in the paper using \"Eq. 1\", \"Eq. 2\" for the optimization problem.\n 2. There is nothing related to KKT condition for (2). KKT condition is some first-order necessary conditions for the saddle-point to be optimal. Form (2) in the paper doesn't use KKT. It's just a Lagrangian form.\n 3. $\\beta$ is usually called the Lagrangian multiplier rather than the KKT multiplier.\n 4. Since $\\beta$ is the dual variable, the optimization problem for $\\beta$ is $min_{\\beta} \\text{ }L$ with $\\beta$ lower bounded by *zero*. So that this problem is the upper bound of the primal problem. It shouldn't *explode* when you optimizing $\\beta$ since you can just choose $\\beta$ to be small enough to get close to the primal problem. \n 5. I understand the author may use some empirical way (an optimization problem with regularization) as they showed in their appendix, rather than primal/dual optimization. It's better to make this part rigorous.\n- Can you explain why $V_1$ and $V_2$ only take $w$ as input? Both $V_1$ and $V_2$ are some total expected returns (of rewards or similarity), so they are also related to the policy, not only the tasks. In the original deep RL framework, a value network should take states as input. In your setting, it should take the states, or trajectory (embedding) as input. It's not reasonable to have a value estimation that doesn't take the data samples as input. Do you assume $V_1$ and $V_2$ can always fit the returns generated by an arbitrary policy? \n- In the algorithm, do you have any inner loops to train the discriminator and the RL algorithm? By meaning \"while not converge\", do you mean the convergence for all the losses as well as the RL performance? Will they work in a coherent way?\n- In Fig 5, it seems APT-Gen without task progress can also work to some extent. This is confusing. Why the policy is not stuck on easy scenarios?\n- (Minor concern) POET is an important baseline but the results are not included. \n\nI'm willing to change my score if some concerns are addressed.\n\n========post rebuttal review=========\n\nAfter reading the response from the authors and the new version of this paper, I decided to increase my score to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "JJxO2GqY7hH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a procedurally content generation approach (APT-Gen) that generates a sequence of tasks for an agent to solve. These tasks are automatically generated in a way that helps an RL agent to learn hard-exploration problems. A main innovation of the approach is a task generator system that is both rewarded for generating tasks the agents can solve but also a sequence of tasks that are getting increasingly more similar to the target task.\n\nI find the paper interesting, the results convincing and don't have too much to critique. My main critique would be to add what the limits of the current approach are. What are the hard-exploration problems it fails on and how much can the target task be outside the predefined task space before performance degrades? A more systematic study of these limitations would further add to the paper and highlight potential future directions for improvements. \n\nMinor comments:\n- Based on how many independent runs are the results based on that are shown in Figure 3, 4, 5?  What do the shaded areas show? One standard deviation?\n- A related paper that does adaptively and procedurally generate levels, instead of tasks, is \"Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation \" Justesen et al., 2018", "extra_scores": {}, "extra_text": {}}, {"review_id": "GvRTz9_snNP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper \"Adaptive Procedural Task Generation for Hard-Exploration Problems\", the authors introduce a new method to automatically generate a curriculum for reinforcement learning algorithms for hard-exploration problems.\nThe approach uses a generative network to generate variants of the agent's environment with varying difficulty and then a discriminator to automatically access the resemblance of the generated environment with the target environment. The generating process then uses this resemblance metric and the observed task return to progressively drive the learning process toward environments that can be solved by the agent and that become closer and closer to the target environment. \n\nThe paper is well structure and clearly written. The illustrations, in particular, Figure 1 are particularly helping to understand the overall process of the algorithm. \n\n\nThere are, however, a few elements that can be improved in the paper. \n\nFirst, in several places, the paper is quite vague on the claim and differences with other algorithms. For instance, \"In contrast, we propose a general framework for highly configurable task spaces with parameters of much higher dimensions.\". What is the exact scale of the difference? I think the paper will be more convincing with precise examples and the exact numbers of the dimensions. \n\nRelated to this aspect, the task description provided in the paper is not sufficient to fully understand what are the degrees of freedom given to the generative network to generate different environments. In particular, what is changing between the different instances? This aspect is too central for the paper that it cannot be relegated to the appendix. \n\nAfter reading the appendix, we can then find the parameters of the environment are all discrete, with few possible values. This is a big deal actually because even if the number of parameters is larger in this paper, other papers referred here are using continuous parameter space, which is by definition infinitely larger. Thus the claims made in the paper need to be tuned down a little bit to underline this difference, and it could be important to discuss how the proposed approach to scale to continuous parameter spaces.  \n\nSimilarly, the justification of the baseline selection is not sufficient in the paper to understand the motivations of the authors and why these algorithms are used in this study, and also why some approaches are not included. For instance, the POET seems to follow a similar approach, yet working on continuous parameter spaces. According to the authors, the propose methods will work better because it is not trying to cover all the space of possible environment, but this remains to be demonstrated in the experimental section. \n\nOverall the method proposed in this paper shows some promising results, but the claims concerning the existing algorithm in the literature should be more clearly supported and demonstrated in the experimental section.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and very active participation in the author response period. The paper is well written [R3,R4], tackles a hard problem [R4] in a novel way [R4] with interesting and convincing results [R2]. R3 noted that an empirical comparison to POET would be appropriate. However, in my view the authors addressed these concerns in a satisfactory manner. It seems that R3 has not updated their assessment nor confirmed their current score based on the author response. I am therefore discounting the only review voting for rejection and am siding with R1, R2 and R4. Thus, I recommend acceptance.
Decision: Accept (Poster)