EAT-C: Environment-Adversarial sub-Task Curriculum for Efficient Reinforcement Learning
Authors: Shuang Ao
Tianyi Zhou
Jing Jiang
Guodong Long
Xuan Song
Chengqi Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
curriculum learning
sub-tasks
adversarial environment
path planning

Abstract: Reinforcement learning (RL)'s efficiency can drastically degrade on long-horizon tasks due to sparse rewards and the RL policy can be fragile to small changes in deployed environments. To improve RL's efficiency and generalization to varying environments, we study how to automatically generate a curriculum of tasks with coupled environments for RL. To this end, we train two curriculum policies together with RL: (1) a co-operative planning policy recursively decomposing a hard task into coarse-to-fine sub-task sequences as a tree; and (2) an adversarial policy modifying the environment (e.g., position/size of obstacles) in each sub-task. They are complementary in acquiring more informative feedback for RL: the planning policy provides dense reward of finishing easier sub-tasks while the environment policy modifies these sub-tasks to be adequately challenging and diverse so the RL agent can quickly adapt to different tasks/environments. On the other hand, they are trained using the RL agent's dense feedback on sub-tasks so the sub-task curriculum keeps adaptive to the agent's progress via this ``iterative mutual-boosting'' scheme. Moreover, the sub-task tree naturally enables an easy-to-hard curriculum for every policy: its top-down construction gradually increases sub-tasks the planning policy needs to generate, while the adversarial training between the environment policy and the RL policy follows a bottom-up traversal that starts from a dense sequence of easier sub-tasks allowing more frequent modifications to the environment. Therefore, jointly training the three policies leads to efficient RL guided by a curriculum progressively improving the sparse reward and generalization. We compare our method with popular RL/planning approaches targeting similar problems and the ones with environment generators or adversarial agents. Thorough experiments on diverse benchmark tasks demonstrate significant advantages of our method on improving RL's efficiency and generalization.  

Under review as a conference paper at ICLR 2022
EAVIRONMENT-ADVERSARAA, SUB-TASK CURRICU- LUM FOR EFFICIENT REINFORCEMENT LEARNING
ABSTRACT
Reinforcement learning (RL)'s efficiency can drastically degrade on long-horizon tasks due to sparse rewards and the learned policy can be fragile small changes in deployed environments. We address these two challenges by automatically gen- erating curriculum of tasks with coupled environments. To this end, we train two curriculum policies together with RL: (1) a co-operative planning policy recursively decomposing a hard task into coarse-to-fine sub-task sequences as a tree; and (2) an adversarial policy modifying the environment (e.g., position/size of obstacles) in each sub-task. They are complementary in acquiring more informative feedback for RL: the planning policy provides dense reward of finishing casier sub-tasks while the environment policy modifies these sub-tasks to adequately challenging and diverse the RL agent quickly adapt to different tasks/envirrnments. On the other hand, they are trained using the RL agent's dense feedback on sub-tasks the sub-task curriculum keeps adaptive to the agent's progress via this "iterative mutual-boosting" scheme. Moreover, the sub-task tree naturally enables an easy- to-hard curriculum for every policy: its top-down construction gradually increases sub-tasks the planning policy needs to generate, while the adversarial training between the environment policy and the RI. policy follows a bottom-up traversal that starts from a dense sequence of easier sub-tasks allowing more frequent mod- ifications to the environment. Therefore, jointly training the three policies leads to efficient RL guided by curriculum progressively improving the sparse reward and generalization. We compare our method with popular RL/planning approaches targeting similar problems and the ones with environment generators or adversarial agents. Thorough experiments on diverse benchmark tasks demonstrate significant advantages of our method improving RL's efficiency and generalization.
INTRODUCTION
Although RL achieves breakthrough success or even outperform humans a few challenging tasks (Lillicrap et al., 2016; Mnih al., 2015; Florensa et al.). it highly inefficient when targeting long-horizon tasks due to the sparse rewards collected via interactions. In addition, a policy trained specific complicated/simulate environment can be sensitive to small changes the deployed environment and thus generalizes poorly in practice. Hence, selecting generating more informative tasks and cnvironments interacting with an agent is an essential challenge on the path towards more efficient, robust, and versatile RI. In this paper, mainly study goal-conditioned RL (Kaelbling, 1993) whose policy is trained to adapt to any given goal/task: is challenging, but the resulted policy can be applied to multiple goals/tasks.
The sparse reward problem has motivated different reward shaping/relabeling methods and curiosity- driven exploration for RL, aiming modify the task reward be denser than states and actions. Hierarchical RL/planning (Elbanhawi & Simic, 2014; Nasiriany et al., 2019; Jurgenson et al., 2020; Pertsch al., 2020) decomposes complicated task/motion by searching a root-to-leaf path of sub-tasks on tree such that higher-level task invokes lower-level ones as its actions. However, building the tree requires hierarchical partition of the whole state-action space and prior knowledge to define the sub-tasks, which can be expensive unavailable. Moreover, the pre-defined sub-tasks can be either too easy or too challenging for the RL-agent in-training. Reward shaping (Laud, 2004) relies heuristics external guidance to augment the sparse task-reward with dense rewards for exploring uncertain actions, rarely-visited states, or intermediate goals. Hindsight experience replay (Andrychowicz et al., 2017) and its variants relabels the achieved states as pseudo-goal -oolll with nonzero rewards, but many of them might be redundant to provide informative feedback efficiently improving the policy. Hence, automatically modifying the sparse reward generating sub-tasks to

---

Under review as a conference paper at ICLR 2022
provide the most informative feedback maximizing the RL agent's learning progress is still an open problem, and resolving it can result in more efficient interactive learning.
Another primary challenge in RL is how to improve its robustness and generalization to small changes in the environments. A growing number of recent studies (Pinto et al., 2017; Vinitsky et al., 2020; Ferguson & Law, 2018) show that RL policy can be vulnerable to small perturbations in its inputs so training an adversarial agent to perturb the inputs or compete with the original agent may improve RL's robustness. This paper mainly focuses on improving RL policy's tolerance to the physical differences between the training environment and possible deployed environment, e.g., changes of the position and size of obstacles/objects. In practice, this is important for deploying an RL policy trained in simulator successfully to realistic environment. Moreover, when addressing the sparse reward problem, the engineered sub-tasks or relabeled goals might be redundant or too easy to provide effective feedback to RI. However, adversarially modifying their environments can effectively improve their utility and RL. policy's robust generalization against diverse perturbed environments. Although modifying environment models has been recently studied to assist RL (Co-Reyes et al., 2020; Gur et al., 2021; Wang et al., 2019), directly applying this strategy to long-horizon tasks might make them even more challenging and reward- reward-sparse, hence detrimental to RL's efficiency.
In this paper, we address the above two challenges by automatically gen- erating a curriculum of sub-tasks with adversarially perturbed environ- ments to train the RL agent, where the curriculum is adaptive to the agent's 's learning progress, To this end, we propose a marriage of sub- tasking and adversarial environments. The former decomposes a challeng- ing long-horizon task into easier sub- tasks offering dense rewards, and the latter modifies each sub-task to be sufficiently challenging yet ef- fective for improving RL policy's s tolerance to perturbed environments. As illustrated in Fig. 1, our ap- proach, "environment--ddversaee Task curriculum (EAT-C)", generates a tree-structured curriculum by (1) a path-plan aath-planning policy that recursively decomposes an assigned task (e.g.. a state-goal pair (s,g) .)) to sequences of sub-tasks (e.g.. navigating between consecutive sub-goals) with varying granularity (k = ....., 3); and (2) an environment policy that adversarially modifies each sub-task's 's environment. The training objective for the eath-planner is finding the most cost-effiiienttssortees path in each level kt while the objective for the environment policy is minimize the expected return of the RI. agent all sub-tasks. In FAT-C, training these two policies does not require external supervision or prior knowledge, we instead collect the time costs and rewards of the RL agent on previous sub-tasks to train them towards generating better curricula adaptive to RL progress, which then guide the training of the next episode.
Co operative Path Planner
Adversarial Environment Generator (EG)
Envtronment
e
Path Planner
[EG]
Auto generated Sub-task Curriculum
Top Do---own EASY 70 MARD
Bodiam-pp EASY TO HARD
Tral Fom Ploneer Gesmatr
R
E
compatte sach
Sab toil modty
crstnoraat
start/ gow stare
Goal-canditioned RL
unnirurmenc/ altared invironment
path planner predict sub goal transter datt only troining
RL Agent
sub-
transfar data
Main structure of EAT-C: The path-planner recursively generates sub-task tree for a given task (go.g) while the envi- ronment generator (EG) adversarially modifies the environment of each sub-task. RL agent is trained on bottom-up curriculum and its collected data are used to train the path-planne and EG.
Figure 1:
Together with the RL policy, the two curriculum policies can efficiently learn from each other by iterating the above noutual-boostinn scheme on the tree-structured curriculum of sub-tasks. A key advantage of the tree is that naturally enables an easy-to-hand curriculum train each of the three policies: (1) the top-down construction of the tree trains the path-planner to merely interpolate a few sub-gon gub-goals between (s, g) at first and then gradually increases their number for more fine-grained and challenging planning: (2) the adversarial training between the environment policy and the RL policy follows a bottom-up traversal of the tree, i.c., they start from learning many easier sub-tasks and more frequent perturbations between (s, g) respectively, which are easy for both policies, and then progressively evolve to handle more challenging cases. i.e., a few long-horizon sub-tasks allow less- frequent environment changes between (s.g). In experiments for discrete navigation and continuous
2

---

Under review as a conference paper at ICLR 2022
control tasks, we show that EAT-C is considerably efficient in learning to generalize to perturbed environments during finishing long-horizon tasks. Compared with recent RL methods equipped with hand-crafted curricula, environment generators, hierarchical policies, or path- wath-planning, EAT-C exhibits promising advantages on efficiency and generalization to different tasks and environments.
RELATED WORK
Goal-conditioned RL (Pong et al., 2018; Kaelbling, 1993) takes a goal as an additional input to its policy and aims to handle different goals/tasks by the same policy. However, it requires extensive exploration and costly training on various goals but still casily fails to reach distant goals in practice. Goal relabeling and reward shaping (Andrychowicz et al., 2017: Nasiriany et al., 2019) have been studied to mitigate these issues by modifying the task rewards to be dense but they introduce extra data bias and cannot control the utility of modified goals/rewards to the targeted ones. In order to provide tasks at the appropriate level of difficulty for the agent to learn, Held et al. (2018) and Racanière et al. (2019) train goal/task generator based the agent's performance on existing tasks. But it is usually non-trivial to determine and tune the appropriate difficulty level for each training stage. In EAT-C, an RL agent achieves dense rewards by finishing a sequence of easier sub-goals towards the final goal, while the adversarial EG modifies each sub-task to be sufficiently challenging and diverse from other sub-tasks. Such sub-task curricula lead to more efficient goal-conditioned RI.. Moreover, we do not need to carefully tune the difficulty levels of sub-tasks because the and EG together automatically generate the most informative tasks for each training stage of RL.
Planning algorithms (Sutton & Barto, 2018; Valle, 2006) are helpful to solving long-horizon tasks (Levine et al., 2011) by interpolating intermediate sub-goals. In navigation, planning usually refers to finding the shortest path between two nodes(states) on graph (La 2006; Dayan & Hinton, 1993) but a distance metric more accurate than Euclidean distance is usually unavailable. Moreover, it suffers from local minimal and performs poorly in narrow regions (Koren & Borenstein, 1991). Sequentially planning (Schmidhuber & Wahnsiedler, 1993; Zhang et al., 2020) sub-goals from the starting state to the goal state is inefficient in complex tasks, it needs search in large space. In RL, planning requires an environment model or learns a value function to improve the policy (Levine et al., 2011; Lau & Kuffner, 2005; Elbanhawi & Simie, 2014), which can be as challenging as model-free RL. Control-based methods require accurate models for both the robot and the environment (Howard & Kelly, 2007; Werling et al., 2012), as well as an accurate distance metric (Eysenbach et al., 2019), which can be rather daunting task. In EAT-C, we train a path-planner that "learns to plan" and to directly generate the shortest (in terms of time cost) path for the RI. agent. Hence, it requires neither expensive search procedures nor predefined distance metric. Moreover, it does not rely on learning a world/robot model or a value function. The easy-to-hard curriculum and dense feedback from the RL agent make it efficient to train. Furthermore, it does not need to plan for every step but only a few crucial waypoints.
Several recent works (Wang al., 2019; Portelas al., 2019; Gur et al., 2021) show that the efficiency and generalization of RL can be improved by generating/selecttnng different environments for training. Wang et al. (2019) studies method "POET" pair an adversarial environment generator with two agents to improve the generalization of the RL policy to novel environments. Portelas et al. (2019) proposes teacher algorithm to sample environments based on Gaussian mixture model capturing both the diversity of environments and the learning progress on them. This leads to a curriculum improving the efficiency of RL. However, the generated adversarial environments can be infeasible or too challenging for the RL agent and it is non-trivial to control their hardness. Moreover, modeling the distribution of environments requires collecting many samples from them and evaluating the RL policy on them, which can be costly and inaccurate especially for complicated environments. In EAT-C, we adversarially modify the environment for each easier sub-task, whose hardness is controlled by both the path planner and EG, so the modified sub-task is still feasible for the RL agent to learn. Moreover, a policy network for EG facilitates the modification of complicated environments and training it is efficient in our mutual boosting scheme with an easy-to-har curriculum.
Hierarchical planning (HP) methods (Nasiriany et al., 2019; Jurgenson et al., 2020; Pertsch et al., 2020) search for sequence of sub-goal -oaal on a tree to guide the exploration of an agent but building the hicrarchical partition of all possible sub-goals can be expensive. A conventional hierarchical planning algorithm (Kaelbling& & Lozano-Pérez, 201 1) also learns to predict sub-tasks of tree structure based
3

---

Under review as a conference paper at ICLR 2022
on set of pre-defined motion primitives. Hierarchical RL (HRL) for goal-reaching tasks has been recently studied in (Zhang et al., 2021; Nachum et al., 2018). Shu et al. (2018) trains the RL agent on humann designdd curriculum of tree-structured sub-goals. They learn sequence of primeval policies towards finishing complicated tasks, where the higher-level policies decompose a complex task into easier sub-tasks motion primitives that can be addressed by the lower-level policies or controllers. However, the hierarchy of sub-tasks need to be either determined by prior knowledge or discovered automatically, which is usually challenging due to the huge space of possible sub-tasks. In contrast, EAT-C trains one planning policy to decompose hard task into sub-tasks of multiple difficulty levels by only using the RL's time cost data. directly generates the sub-tasks and thus requires neither hierarchical partition of the task space nor predefined set of sub-tasks. Compared to HP and HRL, EAT--C (1) enables a mutual training between the path-planner and the RL agent; (2) has an adversarial EG to adjust each sub-task's environment to be more challenging: and (3) improves the training efficiency under the guidance of casy-to-hard curricula automatically generated for each stage.
3 PROBLEM FORMULATION
3.1 GOAL-CONDITIONE REINFORCEMENT LEARNING
Goal-conditioned RL or multi- goal RL learns a policy that can adapt to different goals. Given the state space S, the action space A, and the goal space G, a conlicondi policy is a mapping [alss, S g + .A that outputs an action a (or probabilities Pr(als:g) g) over actions a E A) given a state-goal pair (s, g). An RL agent starts from initial state s so and uses T[ass, g) to take sequence of actions and interact with the environment, which determines the agent's new state and reward after the action taking each step. The interaction with the enviromment is defined by a Markov decision process (MDP) {S, A, r. 73, where p(s'|s, a) & Pr(sc+1 = s s'|s; = S, at a) is the transition probability for the agent from state to s' after taking action a, r(s, alg) : S x A x G + R reward function, and y E [o, 1] is discount factor. For example, it is common to define r(s, alg) - 1 [d (s,g) g) S }), where
the indicator function and d(.,-) -) a distance metric, so the agent achieves reward of 1 if reaching an e-neighborhood of the goal g-
The learning objective of goal- goal-conditioned RL is to maximize its expected return over different tasks B5.g), i... max, E(so.))(EE [Ro)] where the return at step-t is defined as R = Σ "" ag aglg). This is usually used the objective for policy gradient methods. Define the action-value function alg) & E (Ress
8, 0g a, g). the optimal policy x¹ achieves the maximal Q(s, for
any feasible (s, a. g). Define the value function V(slg) - E[Re]s, 8, 9) Ea [Q(s, alg)] EAEA T T 8,, (){({) alg)) ) To reduce the variance of RG, Actor-critic methods additionally learns a model of V or Q as a "critic" the "actor" т. Their training objectives aim to minimize the Bellman residual, i.c., the difference between the two sides of Bellman equation:
a:lg) r(s,, (tl + + BEn- (Q (((++1())g)))) alg)]]- In experiments, to encourage exploration, we use soft actor-critic (SAC) (Haarnoja et al., 2018b).
(1)
3.2 GENERATING SUB-TASK CURRICULA FOR GOAL-CONDITIONEE RL
Training goal-conditional RL on long-horizon tasks can be highly inefficient since sparse rewards can only be achieved after taking a long sequence of actions, which cannot provide informative feedback to improve the policy. Moreover, the immature policy in earlier stages can casily fail and make the rewards even more sparse. In this paper, we train a path-planning policy Tp to recursively decompose long-horizon task into coarse-to- fine sequences of easier sub-tasks as a sub-task tree. Solving any of these sequence can accomplish the original task but the finer ones composed of more sub-tasks provide denser rewards to RL. Hence, the path-planner generates an easy-to-hard curriculum to train the RL agent, which starts from learning easier sub-tasks in finer sequences and gradually focus on more challenging sub-tasks in coarser ones. However, some sub-tasks can still be either too trivial for RL or redundant to other sub-tasks. Moreover, the RL agent is not trained to adapt to small changes in the environment. To address these two issues, we propose an environment generator (EG) n. that adversarially modifies the environment of each sub-task to be more challenging and informative for RL. Therefore, the path -lanner and the EG are complementary in producing an efficient curriculum of tasks and environments to train the RL agent.
4

---

Under review as a conference paper at ICLR 2022
3.2.1 CO-OPERATIVE PATH PLANNER: LEARNING EASY-TO-HARD SUB-TASKS
We extend "sub-goal tree (SGT)" (Jurgenson et al., 2020) to generate our curriculum of sub-tasks. Given long horizon task (5o9) with initial state so and final goal 9. recursively apply path planning policy 1. 9j) to interpolate sub-goals between and 9. In particular, given two sub-goals Si and gj- sampling from (p Ii 9j) yields sub-goal interpolated between yi and Øj- Hence, we can generate sub-goal tree 90:T for (5o.g) 9) by
Pr (go.T go = so, 9r = g) - Pr 90: & 90. 94)
Pr
(04.1l99990 Tp (g |so. (80.9). Sp
(2)
where T 2 with K being the depth of the tree. As shown in Fig. 1, layer-k of the sub-goal tree Porr interpolate sequence of 2 sub-goals - - (of 9 ... 22 between so and g. where o/ - 9T//22 in go.r, Vjj [ 1].
The goal of path planning is to generate cost-efficient sub-tasks for the RL agent, so we train Tp by minimizing the time cost (( of the sub-goal trajectory u on each layer-k, i.e.,
K Σ c(obie))
min Jap A E) 1mmp By
c(eb22))
-
c (gt sitz),
where c(gl g(41) represents the time-cost that the agent taken from g² to g4+1
3.2.2 ADVERSARIAL-EEVIROOOEEE
GENERATOR (EG) APPLIED TO EACH SUB-TASK
Given the next sub-task (o/ 14+1) in layer-k, EG policy मee adversarially modifies the environment E/_ of previous sub-task (g/- 1. ..) to be more challenging to the RL agent, i.e., sampling subtask-f's environment E Te (E) g) where sy A (Ei_1, gi gi+1) denotes the state of EG at subtask- As an adversary to the RL agent, the reward function for EG is defined as (s(" Et19) & -1 fr(s,, (g] = }} where (s,, a,) refer to the state-action of the RL agent at the end of subtask-t. Thereby, EG receives the minimum reward 1 when the RL agent successfully finishes subtask-1 and otherwise the reward is O. We can also define an MDP for EG, which mainly differs from the RL agent in that each step corresponds to a sub-task so EG is only allowed modify the environment at the begimning of each sub-task.
Similar goal-conditioned RL defined in Sec. 3.1, the learning objective of EG is to maximize its expected return over different tasks (so. g). i.e., max,, (so.g) [., (RG) where the return is
K defined R3 > k= Ei= V f Ef(g) with discount factor % E [0, 1]. By defining the corresponding value function Ve and action-value function Qe as in Sec. 3.1, we can apply any RL algorithm to train EG, e.g., use A2C (Mnih al., 2016) for the experiments.
4
SUB-TASK CURRICULUM
4.1 AUTO CURRICULUM GENERATION AND BUTUAL-BOOSTING
In EAT-C, we need to jointly train three policies, i.e., the path-planner up and EG policy ne that generate tree-structured curricula of sub-tasks, and the RL policy л to accomplish the targeted tasks. At the first glance, training three policies can be more difficult than training one RL policy and requires to collect more data via interactions. Moreover, it is challenging to directly train path-planner generating dense sub-goals and BG can also suffer from sparse rewards on long-horizon tasks. However, EAT-C allows the three policies help each other's training via mutual boosting mechanism, where each policy is progressively trained on curriculum of easy-to- hard sub-tasks using dense feedback from other policies on the sub-tasks. By iterating this mutual- caatuallboooting on sub-task curricula, EAT-C significantly improves the training efficiency of each policy and results in an RL agent with better generalization to unseen tasks and perturbed environments.
In each episode, we train the path-planner during its "top-down" construction of a sub-task tree: starts from learning to interpolate a few sub-goals between the given task (so, g) and gradually moves to more challenging cases generating dense sub-goals in bottom layers the tree. Since aims to generate the most cost-efficient path of sub- sub-goals for the RL agent, agent on those sub-tasks in the previous episode as training data, which provide dense rewards to accelerate the training of the path-planner.
the time cost of the RL
5

---

Under review as a conference paper at ICLR 2022
Given a constructed sub-task tree, we then train the RL policy and EG policy by "bottom-up" traversal of the sub-task tree, which naturally forms an easy-to-hard curriculum for each policy. Specifically, both policies firstly learn from dense rewards by finishing casier sub-tasks in bottom layers, where the RL agent only needs to reach a nearby sub-goal and the EG is allowed to frequently modify the environments between g). Due to the adversarial training between them, they not only learning from the environments but also from each other. As moving to the top layers, the RL agent receives less guidance from fewer sub-tasks, while the EG can only change the environment once in each tasks for more challenging tasks. As a result, the RL agent learns to adapt to different tasks and perturbed environments, while the EG learns to generate more powerful adversarial perturbations to train the RL agent in the next episode.
sub-task. Thereby, they need to improve their policies learned on easier
In EAT-C, for each long-horizon task, we iterate the above mutual-boosting training on new sub- task curricula re-generated by the updated path-planner and EG for multiple episodes. This is an imitation of human learning that repeatedly practicing the same complicated task in different ways (e.g., different sub-tasking and perturbed environments). The path-planning and adversarial modification of environments
complementary in constructing curriculum for more efficient RL: the former decomposes a hard task into easier sub-tasks while the latter modifies them to be sufficiently challenging and diverse so the RL agent can learn different skills with better generalization to unseen tasks or perturbed environments.
4.2 EAT-C ALGORITHM
Top-Down Planning of Sub-task Curriculum We provide the de-
Algorithm 1 EAT-C
tailed procedure of the top-down con-
1: Input: po. T, Tmax, c, n2
struction of the sub-task curriculum
2: Output: RL agent's policy æ,, planning policy vp. EG policy me
3: Initialize: n, Tp. Dp
and the update of path-plane planner Tp in Algorithm 2 (Appendix. C). For each layer-k from h
4:
while not converge do
5: Sample a task (80. with so ~ po and g E 9;
to k = K, FAT- C firstly updates the planning policy Tp in line 4 by an RL algorithm using the time cost data collected on sub- tasks up to layer-k from the previous episode's 's bottom-up up training (i.e., Al- gorithm 3, Appendix C), and then re- cursively generates the sub-tasks on layer-k for the current episode (line 5-8). Hence, the path-planner firstly learns to plan coarse trajectories of fewer sub-goals in top layers and gradually increases the sub-goals to form finer paths that can provide more guidance to RL in bottom layers. Since we always use the latest time cost data from Dp for training. the planning policy Tp keeps being optimized to generate cost-efficient sub-task trajectories for the latest RL policy
6:
for episode = 1, 1.2..... n do
7: Algorithm 2: top-down construction of sub-task tree go:r, train planning policy up based on Dy; Algorithm 3: bottom-up traversal of the sub-task tree gu.r,
8:
train RL policy и and EG policy we, collect Dp:
9:
end for end while
10:
Bottom-Up Curriculum for RL and EG Given a sub-task tree generated by Algorithm 2, EAT-C trains RL policy π and EG policy ne by following a bottom-up traversal of the tree. As shown in Algorithm 3, it starts from learning easier sub-tasks the bottom layer-K and gradually moves to top layer-0 (line 4-13), which recovers the original long-horizon task. In each layer, we reset the initial state of the RL agent (line 5) and apply n. and
sequence of 2 sub-tasks 00 towards the final goal g (line 6-11), and then we update the policies using the experiences collected on these sub-tasks (line 12). On each sub-task, EG adversarially perturbs the environment (line 7) and the RL agent is then applied to accomplish this modified sub-task (line 8). If the RL agent fails and ends at a state EAT-C recursively invoke the planning policy TP to add more sub-goals between and the sub-task's 's goal provide more detailed guidance to the RL agent until reaches the goal. This is described in line 9 and line 14-21.
EAT-C algorithm The complete procedure of EAT-C is introduced in Algorithm 1. Given long- horizon task (sp. g) (line 5), EAT-C iterates between the top-down and bottom-up procedures in Algorithm 2-3 for episodes (line 6-9) before moving to a new long-horizon task. Therefore, the planning policy Tp and EG policy ne are optimized to produce better curricula of sub-tasks to train the RL agent to complete task g) while the RL agent learns skills for solving different sub-tasks and generalizing to non-trivial changes of the environment.

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS
In this section, we evaluate EAT-C and compare it with broad range of RL methods two benchmarks, i.e., navigation and manipulation of a 6-point 2D robot to push an object to a goal state, and three compositional tasks in discrete space. In these experiments, efficiency on long-horizon tasks and generalization to environments with small changes. Moreover, we present an ablation study to evaluate the contribution of each part in EAT-C. In addition, we provide case studies to analyze the planned sub-task tree and the modified environment for each sub-task, which explains why EAT-C can improve RL in several aspects. Our experiments are conducted on two representative RL benchmarks and diverse set of baselines:
mainly focus on their
2D pusher (Yamada et al., 2020). As shown in Fig. 4, this is robot navigation and manipulation task in a continues space: a 2D robot with a 4-joint arm needs to firstly navigate to an object and then push it to goal location within an environment of multiple obstacles. We train EAT-C on 50 environments with the positions and sizes of obstacles randomly sampled from uniform distributions (more details in Appendix A). In each environment, we randomly sample 80 training tasks with different (so. Sobj. g). where 8o is the initial state, Sobj is the state of the object, and g is the goal state. Most evaluated policies in our experiments need 2 250 steps to finish each task so they are long-horizon tasks. The rewards are sparse since the agent only receives reward when near the object or when pushing the object and reaching the goal. In EAT-C, the path-planne generates sub-tasks between (50: Sobj) and (Sobj.g). .g). For every sub-task, EG can perturb the sizes and locations of at most three obstacles within pre-defined ranges. For the test, we randomly sample 30 new environments each having one randomly sampled task.
Discrete space tasks (Maxime Chevalier-Boisvert & Pal, 2018). We train and test RL policies on three types of compositional tasks depicted in fig 3, i.c., hunting, scavenging. and salad-making, as illustrated in Fig. 3.(a)-(c). The agent needs to take two or three key steps to finish each task and only get reward when finishing each key step. There are 250 environments used for training and 100 environments for test, each associated with one task. In EAT-C, the path-planner is applied to every two consecutive key steps. EG perturbs the environment by moving objects including tree and stones.
Baselines: We compare EAT-C with broad class of RL methods having related ideas to EAT-C: (1) ALP-GMM (Portelas et al., 2019) that generates a curriculum of diverse tasks with large progress for couulconditioned RL; (2) POET (Wang et al., 2019) with two auxiliary agents for generating curriculum of adversarial yet solvable environments to accelerate RL; (3) Ecological RL (Co-Reyes et al., 2020) that dynamically modifies the environment improve non-episodic (and thus long- horizon) RL without reset of the initial state; (4) A hierarchical RL method (Zhang et al., 2021); (5) Zhang et al. trains the RL agent by modelling goal proposal curriculum that samples goals the frontier of the set of goals that an agent is able to reach. All these baselines and EAT-C need to invoke their subroutine. For fair comparisons, we use Soft Actor Critic (SAC) (Haarnjja
an RL algorithm et al., 201 8a) in all evaluated methods for its stable and promising performance. All methods are not allowed to modify the environments during test since test environments are assumed be the realistic ones in which we deploy the RI. agents. We run different methods for the same number of interaction steps with the environment, i.e., 20 x 10 for 2D pusher and 1.25 x 10 for discrete space tasks. More details of experimental settings are given in Appendix A.
5.1
MAIN RESULTS
We report the performance of EAT-C and all baselines evaluated on the test tasks in Fig. 2(a) for 2D pusher and in Fig. 3(d)-(f) for the discrete space tasks. In all experiments, EAT-C outperforms all other baselines by a large margin on both the learning efficiency and the final generalization performance to test tasks in new environments. In most experiments, baselines adopting a curriculum of environments, i.c., ALP-GMM, POET, and Ecological RL, performs worse than EAT-C but better than the other baselines without changing environments for training. This indicates that building curriculum of training environments is essential to improving RL's generalization and robustness to small changes in the deployed environments. Moreover, among the three compositional tasks in Fig. 3, hunting and scavenging contain moving object, i.c., the deer and the predator, which require the RL agent to adapt to the changes of their locations. On these two tasks, EAT-C exhibits more advantages over other baselines than on the salad-making task, which does not contain any moving object. Therefore, EAT-C enables RL to efficiently learn to adapt to changes the environment.
7

---

Under review as a conference paper at ICLR 2022
100
By comparing the methods with en-
vironment changed, we notice that controlling the difficulty of the modi-
Suzees C0
600
fied environments is critical to earlier- stage learning efficiency. In particular, both EAT-C and POET trains another agent (e.g., the path-planner in EAT- C'and the antagonist agent in POET) to control the hardness of the train- ing tasks matching the capability of the RL agent, so their earlier-stage im- provement than others. In contrast, the environments selected in ALP-GMM might be too challenging and the ones modified by POET might be too easy, leading to poorer efficiency in earlier stages, Although these methods are designed train RL policies that can adapt to different environments or tasks, only EAT-C trains path-plane math-planner to decompose long- oong-horizon task into a curriculum of easy-to-hard sub-task sequences for training. The guidance of path-planner and its curriculum plays a critical role in outperforming these baselines.
20
0
10
15
6.0 04
0.8
12
16
Era Steps (Tfad]
Stape ("ati
64 nowk
00 Ablitton Studly
Figure 2: (a) report the success rate (mean averaged over 6 random seeds) of EAT-C and baselines test tasks in 2D Pusher environments. (b) Ablation study of EAT-C on 2D Pusher tasks.
5.2 ABLATION STUDY
EAT-C jointly trains a path-planner and an environment generator (EG) to produce a curriculum of sub-tasks to improve RL.. Hence, we conduct a thorough ablation study of how the performance changes if removing each component from EAT-C. In par- ticular, we compare the original EAT- C with three variants, i.e., EAT-C with path-planner removed, EAT-C with EG removed, and SAC (with both re- moved), on the 2D Pusher tasks. Since the last two variants are not trained on perturbed environments, for fair comparisons, we use the same envi- ronment for both training and test and only create new test tasks by sampling (so, Sobj. g). The test performance is reported in Fig. 2(b).
Hunting
Scavenging
Salad-Making
Step 1: ass
Step Evedu chasing
Step Collect carrot
Step 2: Hunt deer with axe
predators Stop 2: collect food
Step 2: Collect lettuce
Step 3: Eat food
Step 3: Make solad
Baiscvvesiie
10 10 (Aantin)
94 Dear SOGURT (UCAWARAn)
Figure 3: (a)-(c) illustrate the 2-3 key steps for completing each task. In Scavenging, the agent will have 2 points when it col- lects food each time. (d)-(f) report different methods' performance (mean over 10 random seeds) on multiple test tasks.
When the path-planner is removed from EAT-C, we no longer have any easy-to-hard curriculum of sub-tasks to train the RL agent or EG and they can only learn inefficiently from the original long-horizon tasks. The adversarial environments generated by EG make tasks for RL even harder and unsolvable. Hence, we can observe that only completed 50% of the test tasks within 1.6x 1 environment steps, compared to 90% of the original EAT-C. This indicates that the plan-planner and its generated sub-task tree are critical in creating an effective curriculum for RL.
When EG is removed from EAT-C, we still have the curriculum of sub-tasks to train the RL agent but some sub-tasks might be too trivial or redundant to provide informative feedback for improving the RL agent. This results in poorer efficiency in earlier-stages compared to the original EAT--C with an EG: it reaches 80% success rate after 0.84 10 interaction steps instead of 0.73 10t steps required by the original EAT-C. Although ETA-C without EG can eventually achieve comparable success rate at the end of training. During later stages, the success rate fluctuates unstably over time, while EAT-C with an EG performs more robustly due to the adversarial environments used for training. Moreover, EAT-C significantly improves SAC by a large margin via running SAC on an automatically generated curriculum of sub-tasks, which implies the importance of curriculum on improving RL's efficiency.
5.3 How DOES EAT-C WORK? AN EMPIRICAL STUDY
To better understand how the path planner and EG help RL in EAT-C, in Fig. 4, we visualize the task tree (with layer k E (0, 1, 2)) generated by the aath-planner and the EG's modifications to the environment in each sub-task at Episode and Episode 6 (episode was defined in Alg. 1) for a 2D
8

---

Under review as a conference paper at ICLR 2022
Dee stae
chraade before modited
ongealant 5o0g
apret
Jan bush A,P
JON Molt 2,
150 TAA 3,95%
200 Mow.ingngg
FuB Aab-raut 2,70
(0] Fp
(b) cptiode
Figure 4: Visualization of EAT-C. A 2D robot with j-joint arm starts from the initial state (pink), navigates the object (green) location, and then pushes the object to the goal state (black). The histograms (a) and (b) represents expected return EG taking action b,, and the costs of sub-tasks predicted by the path planner layer h = 2, respectively.
Pusher task (su, Sobj- (.b In the histograms, also report the expected return of EG and the time cost of the RL agent on each sub-task from the bottom layer for the two episodes.
The curriculum of sub-tasks generated by the path-planner. Each plot the tree describes sub-task, where the arrow highlights the sub-task and the yellow trajectory denotes the sequence of all sub-tasks of the layer. Comparing the sub-tasks in different layers, bottom layers (e.g.. k 2) provides more guidance and dense rewards to the RL agent while the sub-tasks in upper layers (c.g.. k = 1) are much harder. Comparing the same-layer sub-tasks generated in different episodes, the sub-tasks in Episode 3 do not take all obstacles into account, e.g., some sub-task sequences trespass obstacles and some sub-tasks are too close obstacles, because the planning policy not fully optimized yet to produce cost-efficient path for the RL agent. Hence, the time costs for the RL agent to accomplish these sub-taske can much higher than later episodes. Moreover, the time costs of some sub-tasks can be much higher than others and thus cannot provide dense rewards to assist RL. On the contrary, in Episode 6, the path-planner has learned to generate cost-efficient sub-tasks with similar hardness so the trajectories and sub-goals are distant from the obstacles and can provide dense rewards facilitating RL. Comparing the histograms of time costs for the two episodes, the RL agent is significantly improved by learning to complete the sub-tasks in the easy-to-hart (bottom-up) curriculum.
Adversarial modifications to obstacles in the environments: In each sub-task plot of Fig. 4, EG adversarially modifies some obstacles by changing their previous sizes and positions (depicted by the blue boxes) to make the sub-tasks sufficiently challenging and diverse. Similar to the path-planner, EG is improved over time: for example, in the sub-task "9o g1" layer-2, the RL agent needs to pass a corridor formed by three obstacles, while EG makes the corridor longer and narrower and thus more challenging for the agent to pass in Episode its modification in Episode is not ideal and even moves one obstacle away from the agent. The improvement of EG is also reflected by its increasing expected return shown in the two histograms. By modifying the environments to more difficult in sub-tasks, FG encourages the RI. agent to learn diverse skills in different sub-tasks. Hence, the sub-tasks are easy for the agent collect dense rewards but they are non-trivial and informative because of EG's modifications.
CONCLUSION
We propose a mutual learning and auto-curicclum framework "EAT-C" to improve the efficiency of RL on hong-horizon tasks as well its generalization and robustness to new environments. EAT-C trains planner to decompose hard task into coarse-to- carse-to-fine sequences of sub-tasks providing an easy-to-hard curriculum to train an RL agent, while an adversarial environment generator modifies these sub-tasks to be diverse and more informative to learn. The three policies are trained with data collected by each other. On two types of tasks, EAT-C outperforms a diverse set baselines, e.g., curriculum-basee RL, hierarchical RL, and planning based methods. It has the potential to be applied to more complicated tasks with dynamic environments or visual inputs such as games, which will be covered in our future works.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, P. Welinder, Bob McGrew, Joshua Tobin, P. Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In NIPS, 2017.
John D. Co-Reyes, Suvansh Sanjeev, G. Berseth, Abhishek Gupta, and Sergey Levine. Ecological reinforcement learning. ArXiv, abs/2006. 12477, 2020.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neural Information Processing Systems, volume 5. Morgan-
Kaufmann, 1993. URL https: /ttp::///////oeeeeee
www.eeiiiinnnnnnnnnneeeeeeeee.nm neurips ctss.cc/paae httpppape/1199/////// (112200666ec73344033888882330 Papeeee.p
Mohamed Elbanhawi and Milan Simic. Sampling-hased robot motion planning: review. IEEE Access, 2:56-77, 2014. doi: 10.1 D0.100//AACCSSS2214220000
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In NeurlPS, 2019.
M. Ferguson and K. Law. Learning robust and adaptive real-world continuous control using simulation and transfer learning. ArXiv, abs/ 802.04520, 2018.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforce- ment learning. ICLR (Poster).
Izzeddin Gur, Natasha Jaques, Kevin Malta, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Adversarial environment generation for learning to navigate the web. ArXiv, abs/2103.01991, 2021.
T. Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Schoon Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, and S. Levine. Soft actor-critic algorithms and applications. ArXiv, 2018a.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 201 8b.
David Hcld, Xinyang Geng, Carlos Florensa, and P. Abbeel. Automatic goal generation for reinforce- ment learning agents. ArXiv, abb/1105500666, 2018.
Thomas M. Howard and Alonzo Kelly. Optimal rough terrain trajectory generation for wheeled mobile robots. The International Journal of Robotics Research, 26:141 166, 2007.
T. Jurgenson, Orly Avner, Groshev, and Aviv Tamar. Sub-goal trees a framework for goal- reinforcement learning. ArXiv, abs/2002. 12361, 2020.
Leslie Pack Kaelbling. Learning to achieve goals. In PROC. OF IJCAI-93, pp. 1098.
Morgan Kaufmann, 1993.
Leslie Pack Kaelbling and Tomás Lozano-Pérez Hierarchical task and motion planning in the now. In 20/1 IEEE International Conference on Robotics and Automation, pp. 1477, 2011. doi: 10.1 10.1199/CAA..2115599000
Y. Koren and Borenstein. Potential field methods and their inherent limitations for mobile robot navigation. In Proceedings. 1991 IEEE International Conference on Robotics and Automation, pp. 1398-1404 vol.2, 1991. doi: 10. 10.1109//OOTT 1991..119999.111111.1
Manfred Lau and James J. Kuffner. Behavior planning for character animation. In Symposium on Computer Animation, pp. 271-280. ACM, 2005.
Adam Daniel Laud. Theory and Application of Reward Shaping Reinforcement Learning. PhD thesis, USA, 2004.
S. M. Valle. Planning Algorithms. Cambridge University Press, Cambridge, U.K., 2006.
10

---

Under review as a conference paper at ICLR 2022
Sergey Levine, Yongjoon Lee, Vladlen Koltun, and Zoran Popović. Space-time planning with parameterized locomotion controllers. 30(3), 2011.
T. Lillicrap, Jonathan J. Hunt, A. Pritzel, N. Heess, T. Erez, Yuval Tassa, D. Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, 00911199,029,,, 2016.
Lucas Willems Maxime Chevalier-Boisee and Suman Pal. Minimalistic gridworld environment for openai gym. 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(75400:529-533 February 2015.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In /CML, 2016.
Ofrr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In NeurIPS, 2018.
Soroush Nasiriany, Vitchyr H. Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. In NeurIPS, 2019.
Karl Pertsch, Olch Rybkin, Frederik Ebert, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. ArXiv, 0as/2006.132055 2020.
Lerrel Pinto, James Davidson, R. Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. JCML. 2017.
Vitchyr H. Pong, S. Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model-free deep for model-based control. ArXiv, abs/ 802.09081, 2018.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Y Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. In CoRL, 2019.
Sébastien Racanière, A. Lampinen, Adam Santoro, David P. Reichert, Vlad Firoiu, and T. Lillicrap. Automated curricula through setter-solver interactions. ArXiv, abs/1909. 12892, 2019.
Jürgen Schmidhuber and Reiner Wahnsiedler. Planning simple trajectories using neural subgoal generators. In Proceedings of the Second International Conference on From Animals to Animats 2: Simulation of Adaptive Behavior: Simulation of Adaptive Behavior, pp. 196-202, 1993. ISBN 0262631490.
Tianmin Shu, Caiming Xiong, and Richard Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning, ArXiv, abs/1712.07294, 2018.
Richard S. Sutton and Andrew Barto. Reinforcement Learning: An Introduction. Bradford
Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 10.1109/RR0S.2012.65580
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, P. Abbeel, and A. Bayen. Robust reinforcement learning using adversarial populations. ArXiv, abs/2008.0 2020.
Rui Wang, J. Lehman, J. Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. ArXiv, abs/1901.01753 2019.
11

---

Under review as a conference paper at ICLR 2022
Moritz Werling, Sören Kammel, Julius Ziegler, and Lutz Gröll. Optimal trajectories for time- criticclical street scenarios using discretized terminal manifolds. The International Journal of Robotics Research, 31:346 359, 2012.
Jun Yamada, Youngwoo Lee, Gautam Salhotra, Karl Pertsch, Max Pflueger, Gaurav S. Sukhatme, Joseph J. Lim, and Péter Englert. Motion planner augmented reinforcement learning for robot manipulation in obstructed environments. ArXiv, ab94200..940 2020.
Jesse Zhang, Haonan Yu, and Xu. Hierarchical reinforcement learning by discovering intrinsic options. ArXiv, abs/2101.06522 2021.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. ArXiv, abs/2006. 1485,, 2020.
Yunzhi Zhang, Pieter Abbeel, and Lerrel Pinto. Automatic curriculum learning through value disagreement. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), NeurIPS 2020. Curran Associates, Inc.
12

---

Under review as a conference paper at ICLR 2022
A DETAILS OF ENVIRONMENT IMPLEMENTATIONS
A.1 2D PUSHER
We simulate the environment of 2D pusher in Mujoco physics engine (Todorov et al., 2012). The positions of the robot, object and goal are defined as Prob: Pohj. and Pyout, respectively, and T is the maximum number of episodes (i.e., the horizon). An 2D-pusher agent with four joints can take actions of six dimensions, two for navigation and the rest four for arm control. The map size is 1 x 1 so both the x and y coordinates lie in (-0.5.0.5 The x and y coordinates of goals and objects are randomly sampled from uniform distributions of (003.33. -0.2) and ((01.1 0.1), respectively. The initial state of the agent is randomly sampled from uniform distribution of ((00.05 0.3). For obstacles, their initial coordinates and sizes are randomly drawn from an uniform distribution, as explained in the first row of Table. 1. We train 2D pusher using sparse reward: when the robot reaches vicinity of the object or the sub-goal state (lpprob Pobyll2 & 0.05) the agent will receive reward= 150/25 /2 where k is the layer of the sub-task tree where the sub-goal is located. Once the agent pushes the object to the goal state with ||Pgool Pooylll 122 & 0.05, the agent will receive a one-time reward= 150; otherwise there is no reward. By taking action, EG can change the size and the location of 0 ~ 3 obstacles near the agent. Assume that there are n obstacles in the environment, and we represent each obstacle-i by its location (1a, Me) (2D coordinates) and size (w;, ha) (width and height) as 02 = (I Vo, Ws, h₂). The action b, of EG is defined as
b, a AA;- (Az,, Ay, Au;, Ah;), Vi E [n]]
In order to provide feasible and smoothly changing environments RL along the sub-task trajectory in each layer, and to prevent the environment generator from being too powerful and overly adversarial, is important to restrict EG from changing the environment too much at one time for each sub-task. Hence, in the experiments, we constrain every dimension in an action of EG not to exceed some threshold, c.g., for 24, we apply
Ar, +- min max (Ax;, 82 Tmin J. Tmax
(5)
where &₂ E [0.1] and (Emin: Tmax) is the valid range of x-coordinate. The above environment parameterization can be easily extended other environments so EAT-C is general and principal scheme that can be adapted to different environments.
A.2 DISCRETE SPACE TASKS
The environment for the three tasks is an N x N grid. It is partially observed by the RL agent: the agent at each state obtains a local egocentric view of 5 x 5 grid around it, where the object in each cell of the grid is represented by a C-dimensional one-hot vector (there are C' possible types of objects). The agent can pick up and carry one object at a time. It can also combine two objects to construct new one by putting a carried object onto an existing object, e.g., it can combine wood with metal to make axe. The RL agent can take action such as moving in the cardinal directions, picking up an object, and dropping an object. In discrete space tasks, the environment generator (EG) can modify the environment by taking action to move an object/obstacle. In order to provide feasible and smoothly changing environments to RI. along the sub-task trajectory, and to prevent EG from being too powerful and overly adversarial, each action of EG can only move every object/obstacle to an adjacent cell around it.
A.3 ENVIRONMENT ENCODING OF BASELINES
The baselines in our experiments include two curriculum RL methods, i.c., ALP-GMM and POET, which both can control some training environments. For their fair comparisons to EAT-C,, we set their environment-dependen parameters exactly the same as the ones controlled by EG in EAT-C For 2DDpusherr as detailed in Table 1, the baselines control the same four environmenntdeeeeddent parameters as EAT-C. Each parameter is initialized by sampling from a uniform distribution and each mutation step can modify it by a small value it is within the valid range. For the discrete space tasks, ALP-GMM/POET can generate environments by sampling/mutatii parameter controlled by EAT-C. The valid range for location for each obstacle/object is (0, 1) and the mutation step size is 0.1.
parameters for mutation and generation of the
the location of each obstacle/object. which is the same
13

---

Under review as a conference paper at ICLR 2022
Table 1: Environment-depende parameters in baselines on 2D-pusher. Each baseline generates an environment of obstacles by uniformly sampling the four parameters defining each obstacle from the corresponding ranges. starts from the initial ranges below and can take mutation step one time to change the lower and upper bounds of each parameter's ss range, if the two bounds do not exceed their minimal and maximal values listed below. The two numbers in each tuple (..-) below corresponds to the lower and upper bound of the range.
Parameter
Obstacle
Obstacle
Obstacle width wy
Obstacle height hy
Type
X-coordinate it y y-coordinate y/
Initial Range
(-0.3, 0.3) (0.02,
(-0.3, 0.3)
(0, 0.1) (0.05,
(0, 0.1) (0.05, 0.05)
Mutation Step
(0.02, 0.02)
Minimal Value
(-0.5, -0.5)
(-0.5, 00.5)
(0,0)
(0, 0)
Maximal Value
(0.5, 0.5)
(0.5, 0.5)
(0.3, 0.3)
(0.3, 0.3)
B MODEL ARCHITECTURE AND HYPERPARAMETERS OF SAC (RL ALGORITHM USED IN ALL EXPERIMENTS)
We use the same neural network architecture (i.e., MLP) for the RL agent and the same RL algorithm (SAC) in all the experiments of all the methods.
Besides the reward of completing task/sub-task, it is common in MuJoCo and many other simulators to also issue a small instantancous reward after taking any action in order to encourage exploration. Moreover, different methods usually need to re-scale this exploration reward because they may need different levels of exploration. In our experiments, we tune the re-scaling factor for every method to get its best performance. Specifically, we chose 0.3 for CAT-C/ALP-GMMN POET and 8.0 for hierarchical RL/value disagreement/Ecologiee RL. An explanation of applying a smaller factor for the former three methods that they already have some strong exploration strategies and a larger factor might downweigh the task reward and thus results in performance degeneration.
Moreover, we use the same coefficient a of the entropy term in SAC's objective for all methods (they all use SAC as the RL algorithm). The coefficient a controls the degree of exploration and automatically tuned. A complete list of hyperparameters for SAC in 2D-pusher tasks is given in Table 2. They are exactly the same hyperparameters defined in in SAC paper (Haamnoa et al., 201 8a) and in Table I of their Appendix D except that we choose different values for them in 2D-pusher.
In the discrete space tasks, the environment is a 10 x 10 grid and the 5 x 5 partial observation (as mentioned in A.2) of the RI. agent can be represented as a x 5 x C one-hot tensor. We flatten this tensor to vector and process it by an MLP with three hidden-layers whose output dimensions are (64, 32), respectively. We apply another MLP with three layers of output dimensions (16, 16, 16) to process the inventory observation. The two MLPs' outputs are then concatenated and processed by an MLP with two hidden layers of output dimensions (16 action dimension) that outputs a probability distribution over all possible actions. We use ReLU as our nonlinear activation functions in all MLP models except their last layer, which uses a softmax function to compute the probability of taking each action. In EAT-C, the RL agent and EG share the same observations as well as the first two MLP models but they use different MLP models to output the actions. A complete list of hyperparameters of SAC in the discrete space tasks is given in Table. 3.
C PSEUDO-CODE OF EAT-C
In the training phase, we first randomly initialize Tp. and apply it to predict a sub-task tree using Euclidean distance as an initialization of the cost in line 4 of Algorithm 2 when no time cost data have ever been collected at the very beginning. Given the sub-task tree, we can train the RL agent by a bottom-up curriculum of the sub-tasks on the tree. In particular, we start from the bottom layer and train the RL agent to consecutively complete sequence of sub-tasks from the starting state to the goal state. As training proceed, the RL agent collects data of the time cost for completing feasible sub-tasks (g, g When the RL agent cannot complete the pre-assigned sub-task (gf gf+1) within time limit, we re-apply up to interpolate more sub- -obls between (o,, o by line 25 in Algorithm 3.
14

---

Under review as a conference paper at ICLR 2022
Table 2: SAC hyperparameters EAT-C (2D-pusher)
Parameter
Value
Optimizer
Adam
Learning rate
3.0 x 10
Discount factor (?)
0.99
Replay buffer size
1.0 x 106
Number of hidden layers for all networks
Number of hidden units for all networks Minibatch size
400
256
Nonlinearity
ReLU
Target smoothing coefficient (T)
5.0 x 10 10-3
Target update interval
Network update per environment step
Entropy target
dim(A)
Table 3: SAC hyperparameters in EAT-C (discrete space tasks)
Parameter
Value
Optimizer
Adam 5.0 10
Learning rate
Discount factor (7)
0.99
Replay buffer size
1.0 x 106
Number of hidden layers for all networks
Number of hidden units for all networks Minibatch size
256
256
Nonlinearity
ReLU
Target smoothing coefficient (r)
5.0 10-3
Target update interval
Network update per environment step Entropy target
dim(AA)
Algorithm 2 Top-Down Planning of Sub-task Curriculum
1: Input: (8o, 9). T, planning policy Fp and its training set Dp-
2: Output: tree structured sub-goals go.r,
3: for ki = 1, K do
4: Apply RL algorithm minimize Jap Eq. (3) computed time cost data up layer-k in Dn;
5: for 1,.... 1,.,,,,,,*1 do
2 6: Generate the sub-goal gi (p k-1 1,99 -
7: Add sf A-1 into the trajectory forr layer-k;
end
9:
end for
More technical details are given in Algorithm 3. In the test phase, apply up produce sub-task tree and only use the sub-task sequence in the bottom layer to guide the RL agent.
D ADDITIONAL EXPERIMENT RESULTS
D.1 LARGER VERSIONS OF PLOTS FOR MAIN RESULTS
Considering that the figures of experiment results in the main paper might be too small to read, we temporarily list the main results with more clear vision (Fig. 5, and Fig. 6).
15

---

Under review as a conference paper at ICLR 2022
Algorithm 3 More detailed Bottom-Up traversal in EAT-C
Input: RL policy и,, EG policy Te. sub-goal tree 90:7, Tmax:
2: Output: म, Dr
3: Initialize: Tp training set D, + 0. RL's replay buffer D t 0, EG's replay buffer D. < 0
4: for k ,,...,I 1,0 do
5: Reset RL agent's initial state to go:
6: for ( 1,2,3..... 2 do
7: EG adversarially modifies the environment E to E;;
8:
while S Tmax or &, & B(g c) do
9:
RL agent takes action ar [arls5, g²);
10:
RL agent moves to 8 ~ 7(87+11 (8,, a+) and receives reward r(8r, or lg
D t+ D+DDU (S:: Qr. (s7, a, (g) 8++1):
12:
end while
REACH(s, 9
14:
end for
for every gradient step do
16: Apply gradient steps in SAC: update Q, V, using samples drawn from D;
17: Apply gradient steps in A2C: update Qu. Te using samples drawn from De:
18: end for
19: end for
20: Procedure REACH(s, g):
21: if d(s,g)) g) & then
De De U (8,, by, (e F(1g),9(); 23: D2 + D, U (50, ST,
22:
T), 90 + Syi
else
25:
Re-apply Tf to predict temporary sub-goals Ii., for (oh-1.
26:
Add Addgin into the planned sub-goals trajectory 9
Re-apply agent start from 5p with the new sub-goal trajectory reach - and collect training data (Follow 8ine.8 to line.14);
REACH(s, g):
29:
end if
100
100
BAY-C ui
GRM POET
80
80
560
Hinurcvou RL. Vake Dioogremee
Success rate
60
Success rate
60
40
20
20
0
%
0
10
15
20
0.4
0.8
1.2
1.6
Env. Steps (*1e6)
Env. Steps (*1e6)
(a) Main
(b) Ablation Study
Figure 5: (a) report the success rate (sea averaged 6 random seeds) of EAT-C and baselines on test tasks in 2D Pusher environments. (b) Ablation study of EAT-C on 2D Pusher tasks.
16

---

Under review as a conference paper at ICLR 2022
Hunting
Scavenging
Salad-Making
Step 1: Pick up axe
Step 1: Evade chasing
Step 1: Collect carrot
Step 2: Hunt deer with
predators Step 2: collect food
Step 2: Collect lettuce
Step 3: Eat food
Step 3: Make salad
Hunting
(c) Salad-Making
MP oue
80
Aee Success fatee
Rvc. Sucosss Rate
80
Pyints
60
40
40
20
20
0
25
50
75
100
125
0
25
50
Ens Stean (*1=d)
Emn: Steps: (e1e]]
("1ed]
(d) Main (Hunting)
(e) Main result (Scavenging)
(Salad-Making)
Figure 6: (a)-(c) illustrate the 2-3 key steps for completing each task. In Scavenging, the agent will have 2 points when it collects food each time. (d)-(f) report different methods' performance Estd over 10 random seeds) on multiple test tasks.
D.2 ADDITIONAL ABLATION STUDY EVALUATING GENURALIZATION/ROOUSTTI ON NEW RANDOM ENVIRONMENTS
Following the suggestions of Reviewer Zhvq and o38w, we add an ablation experiment and report the results in Table 4. To evaluate the generalization and robustness, which are the advantages of EAT-C due to the adversarial environment generator (EG), we evaluate different methods on multiple new random environments during the test phase. This is different from the ablation study in Fig. 2(b), which evaluates all methods on the fixed training environment. The new results show large gap (80.24 42.23) between EAT-C and EAT-C (remove EG). This demonstrates that EG is important to improving the generalization and robustness of the RL policy. In the training environment (non-random) used in our original ablation study of Fig. 2(b), Hierarchical RL (HRL) does improve the performance of the default RL algorithm (SAC) on long-horizon tasks, i.e., 39.62 (SAC) 68.27 (HRL). However, in random and unseen environments, HRL generalizes much poorer than EAT-C
Table 4: Ablation Study Results
Test Setting
Multiple New Random Environments
Training Environment 92.04:16.49
EAT-C
80.24 = 12.25
EAT-C (remove EG)
12.23-10.34
85.47:19.12
EAT-C (remove Planner)
27.58 = 14.67
46.02+10.3
SAC
20.83 7.24
39.62 = 12.25
Hierarchical RL
22.04 10.44
68.27 + 6.99
Reviewer w38w and Zhvq raised the following concerns: (1) whether EG could make the environment
more reward sparse? ? (2) whether planner could always generate infeasible sub-goals? To answer these questions, in Fig 7, we report the average time-cost that the agent needs to complete each sub-task in layer-3 of the sub-task tree.
17

---

Under review as a conference paper at ICLR 2022
In earlier stages when Tp and the RL agent are not well trained, Tp may generate hard sub- tasks. However, after a little training on the sub-task curriculum, Tp is trained to generate minimum-cost path for the RL agent and the capability of the RL agent to finish the sub-tasks is also improved.
We apply EG to simple sub-tasks that are optimized to be simple in EAT-C (via optimizing the planner) for the RL agent. The goal of EG is to avoid learning similar and easy sub-tasks repeatedly, which cannot provide informative feedback to RL even if the reward is dense. On the other hand, set several restrictions to avoid over-adversarial environments, as mentioned in Appendix. A
10
ave 1me-cos!
Ave. time coxt (normalized)
2
250
500
750
1000
1250
Iterations (*10)
Figure 7: Average time-cost of the RL agent to complete sub-task from layer-3 of the sub-task tree. As the training proceed, time-cost that the agent needs to complete each sub-task decreases significantly, indicating that Jp does not propose infeasible goals, and EG does make the reward more sparse.
D.3 EXPERIMENTS ON 7DoF ROBOTIC ARM
Following the suggestions of Reviewer 7T33 and Zhvq, we add experiment of controlling 7DoF (degrees of freedom) robotic arm (i.e., the one used in (Jurgenson et al., 2020)) to evaluate how EAT-C performs complicated control task. We use MuJoCo as the simulator. In this experiment, the robotic arm learns to avoid obstacles and reach goal state (as shown in Fig. 8). We compare EAT-C with curriculum RL methods and SGT (Jurgenson al., 2020).
Both the training and test tasks have 5 obstacles with different and randomly sampled location and size parameters. The start-goal pair of each task are also randomly sampled. Both EAT-C and curriculum RL methods are able to modify exactly the same parameters defining the location and size of each obstacle.
We report the success rate of reaching the goal state without collision and the collision rate as the two metries to evaluate EAT-C and all the baselines. After training, we evaluate them on 100 new random tasks different from the training tasks. The results are reported in table 5 and EAT-C achieves the best performance on both the collision rate and success rate among all evaluated methods. This demonstrates the robustness and promising performance of EAT-C in more complicated and realistic tasks.
Methods EAT-C
Average Collision Rate
Success Rate
0.22 + 0.05
0.873 + 0.027
ALP-GMM
0.34 + 0.07
0.524 +'t 0.092
POET
0.42 + 0.07
0.544 + 0.084
SGT-PG
0.25 + 0.02
0.772 = 0.014
Table 5: Main results for the experiments on 7DoF robotic arm.
18

---

Under review as a conference paper at ICLR 2022
Figure 8: 7DoF Robotic Arm in training environment with randomly sampled obstacles (those cyan cubes).
E MORE RELATED WORK
Refer to the suggestions of reviewers, we summarize the main difference of EAT-C with the suggested reference in this section.
The sub-goal generation in (Pertsch et al., 2020) follows top-down and coarse-to-fine manner. However, they need search for each sub-goal in the tree from many possible candidates, which is expensive and requires a search tree (hierarchical partition of the whole sub-gon uub-goal space) much larger than our sub-goal tree (see Eq. 2). On the contrary, EAT-C learns a planning policy to directly generate su-bgoals and we do not need to build the search tree covering the whole sub-goal space. Another major difference is that they study a planning-only method while we study a mutual learning strategy between planning and RL to improve both planning and RL policies.
Zhang et al. (2020) trains high-level policy to find the shortest path of sub-goals adjacency space. However, the distance between any two points in the adjacency space is expected to reflect the time cost of the agent navigating between the two points in the environment, which can be very challenging
trained
infeasible to achieve in many tasks (If have such adjacency space, both planning and RI. can have dense feedback and simple supervised learning should work). In contrast, EAT-C trains planner directly generate min-cost path of sub-goals through an easy-to-hard curriculum (fewer sub-goals interpolated at first), which provides an casier and more efficient solution without requiring learning an adjacency space. Moreover, the data used train the planner in EAT-C are more informative than (Zhang et al., 2020) and cover multi caalti-gaaularity since they are collected from RL when completing the bottom-up sub-task curriculum.
In (Dayan & Hinton, 1993), the high-level managers set sequence of subgoals in the environment partitioned by Fuclidean distance, which does not consider the obstacles or the RL agent capabil- ity. Hence, there is no mutual training between high-level (planner) and low-level (controller) managers in (Dayan & Hinton, 1993). On the contrary, the planner EAT-C is jointly trained with the RL agent to produce min-cost path of sub-goals for RL, which results in more efficient curriculum of sub-tasks to train the RL agent.
Zhang et al. (2020) and Schmidhuber & Wahnsiedler (1993) plan sub-goals sequentially from the starting state to the goal state, which might be inefficient in complex tasks (requiring expensive search in large space) and cannot produce the easy-to-hard curricula on sub-goal tree as in EAT-C. In contrast, we train a planner to recursively produce coarse-to- sub-goal trajectories between the starting and goal states, which naturally provide easy-to-hard curriculum for every component.
19


Reviews:
[{"review_id": "ZWn-n2R9G6l", "reviewer": "Reviewer_o38w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The biggest strength of this paper is in its combination of two important and productive lines of research from HRL and environment design.  The combination of the two is relatively natural because they solve different aspects of sparse reward problem, and to the extent there are synergies between approaches, this is a productive direction of research.  \n\nHowever, it is not clear if what is achieved by this approach could be achieved by running existing HRL techniques in the context of adversarial environment design. The one thing that makes this papers approach distinct from this direct combination is that the adversarial design is done independently for every sub-goal.  However, it is not clear that this distinction is necessary for the approach to work.  In fact, the ablation study in Figure2 (b) leads me to believe that the adversarial environment design component is relatively unimportant, so I wouldn't be surprised if the somewhat marginal benefits could also be achieved by adversarial environment design separate from the hierarchy.  If this is the case, then it is difficult for me to see the benefits of combining the approaches, given the added complexity, added dependencies, and marginal benefits. \n\nIn terms of the benefits, the empirical results are also inconclusive.  One concern is that SAC by itself beats some of the baselines, which makes it seem like the baselines may not be configured properly.  In particular, POET has environment-dependent parameters which need to be tuned, so it is not obvious to me how this baseline works, or how it was ensured that these parameters were set correctly.  Even without these concerns, it seems that most of the results do not seem to be completely statistically significant.\n\nThe next major concern is that it seems that it would be easy for many of the components to fall into undesirable equilibrium.  For instance, the environment adversary could make unsolvable environments, and it seems like the action space of the environment adversary had to be restricted to avoid this.  Also if the agent does not have some capability, and the planner would not ask the agent to solve those goals since it is cooperative, and thus the agent would not gain experience for those goals, which could also halt progress.\n\nFinally, it is unclear to me if the hierarchy of subgoals being mutually trained with the RL agent is novel.  It appears similar to a few of the following and there should at least be a discussion of how these differ, if that is to be taken as a contribution:\n\"Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors\" Pertsch K and Rybkin O, et. al.\n\"Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning\" Zhang, T and Guo, S et.al \n\"Feudal reinforcement learning\" Dayan, P and Hinton, G et.al\n\"Planning simple trajectories using neural subgoal generators\" Schmidhuber, J and Wahnsiedler, R et.al\n\nMinor point: \n* The paper ends suddenly, I was expecting a conclusion section\n* I didn't realize the legends where different for Figure 2(a) and Figure 2(b), it would be less confusing if the reuse of colors could be avoided\n* The caption for Figure 2 blends into the text.\n* All references for Figure 3 are before all references to Figure 2, but they are displayed in the other order\n* I do not know what is meant by a \"mutual boosting scheme\", the term is used several times but does not seem to be defined.\n* The second to last paragraph of page 3 there is a typo: \"requires to collect\"\n* In the first paragraph of page 6, you first mention that the RL agent is trained on one rollout-per-layer rather than one rollout just for the leaves of the hierarchy.  This should be made clearer earlier, as it is a significantly different method with this change.\n* In the second paragraph of page 6 there is the claim that the sub-goal curricula is \"an imitation of human learning that repeated practice the same complicated task in different ways\".  I do not believe this claim, though I am not sure I understand it, but I do not think it is necessary to understand the method.\n* There is also a claim at the end of that paragraph that they adversary generates diversity.  I do not know why that would be true, it seems that the adversary could easily fixate on a particular type of modification.\n*The generic idea that environment design techniques make environments more reward sparse (for instance, at the end of the first paragraph of page 1), is true of fully-adversarial environment design techniques, but it is not true for techniques like:\n\"Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play\" Sukhbaatar, S et al.\n\"Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design\" Dennis, M, et al. \nboth of which are designed to densify the reward structure by designing hard but solvable levels, and would likely serve as better baselines.  In particular, the method used for the environment designer in this work appears to correspond to the \"maximin adversary\" in the second paper, which was used as a baseline.", "summary_of_the_paper": "This paper proposes a method for generating curriculum for goal-conditioned RL policies based on having one agent hierarchically generate sub-goals, and have the target agent train from rollouts on each level of this hierarchy, while a third agent adversarially modifies the environment to improve robustness and transfer.  The main contributions are introducing a mutual training regime between the path planer and the RL agent, and introducing adversarial environment design to training in each of the sub goals.", "summary_of_the_review": "I will be weakly recommending rejection since it is unclear to what extent the method works because of the integration of HRL and adversarial environment design, and to what extent the same results could be achieved through the simpler direct combination of the existing approaches in both fields.  In addition, it seems that the baselines could be made stronger, and I am not convinced by the empirical results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FxEEN_wBuHM", "reviewer": "Reviewer_Zhvq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, I enjoyed reading the paper. It is clearly written and does not try to oversell its contribution - the combination of a recent hierarchical goal-based RL approach and adversarial environment generation. My main questions/critiques revolve around the experimental section. I can imagine that some of them may have been already addressed in the appendix that has been mentioned in the main paper but seems to be missing in the current revision:\n\n* As I already mentioned, I think that the authors are in general objective in their assessments of the novelty of their method.\nOnly one statement on page 2 caught my eye: \"Unlike previous methods, training these two policies does not require external supervision or prior knowledge, we instead collect the time costs and rewards of the RL agent on previous sub-tasks to train them towards generating better curricula adaptive to RL progress, which will be used in the next episode.\" I think that there already exist hierarchical RL approaches that do not need additional feedback or priors in addition to the MDP specification, such as e.g. [1].\n* The authors compared their method to curriculum RL methods such as ALP-GMM. For these curriculum RL algorithms, an appropriately defined parameterization of the MDPs is crucial to obtain good learning performance. Unfortunately, details regarding this parameterization were not included in the main paper. For the curriculum RL methods, I would expect the parameterization to include both the parameters that the EG in EAT-C can control as well as the goal state to be reached. In this setting, to comparison is the most fair and consequently, the benefit of the particular structure of the proposed approach becomes the deciding factor in the performance difference. Could the authors clarify on the environment parameterization for the curriculum RL methods? If it differs from the aforementioned one, an additional investigation with this particular parameterization would clearly improve the paper.\n* Another point of confusion is the very poor performance of the hierarchical RL baseline, which is unexpected given the clear benefit that a reader would expected from hierarchical RL in long horizon tasks. Although I can imagine that this poor performance may be caused by the randomization over environment configurations, the authors should consider providing additional evidence for this hypothesis. This could be done by:\n\t* Evaluating the hierarchical RL method also in the non-randomized environment used for the ablations of EAT-C.\n\t* Evaluating the ablation of EAT-C without the EG in randomized environments.\n\t* Clarify details regarding the observability of the environment configuration (see next point).\n* Is the configuration of the environment hidden from the RL agent? Or is the agent aware of e.g. obstacle/object positions via observations? I am asking because compared to the experiments by Yamada et al in the 2D pusher task, much more samples are required to reach acceptable performance. So it seems that the generalization over environment configurations induces large challenges. Hence it would be important to carefully specify how the environment randomization is perceived by the agent.\n* A more \"realistic\" experiment could further improve the paper. Such an experiment could be e.g. the 7-DoF planning environment with obstacles done in the sub-goal trees paper by Jurgenson et al.\n* Could the EG in theory create infeasible tasks? Does this need to be prevented by the appropriate parameterization of the environment/EG actions?\n* In algorithm 2: Could the REACH procedure possibly run into an infinite recursion with a very poor planning policy $\\pi_p$ that always predicts infeasible goals? If so, could this be avoided with some naive pre-training? If this infinite recursion cannot happen, what prevents the possibly infinite recursion?\n\n[1] Kulkarni, Tejas D., et al. \"Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation.\" Advances in neural information processing systems 29 (2016): 3675-3683.", "summary_of_the_paper": "The paper proposes an RL method targeted at long-horizon tasks in environments with perturbed dynamics or configurations. By combining a recently proposed goal-based RL method with an adversarial approach to domain randomization, the authors obtain a method that allows them to learn robust policies in sparse long-horizon settings. The method is compared with different baselines form hierarchical- or curriculum RL, demonstrating promising performance.", "summary_of_the_review": "Despite its rather incremental nature, I like the overall paper. Mainly the questions w.r.t. experimental details and the smaller scale evaluation tasks prevent me from putting it above the acceptance threshold. If some of my issues/questions are addressed, I am happy to improve my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ATEfrqKKoBh", "reviewer": "Reviewer_7T33", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros:\n\n1. The authors address an important problem: a curriculum learning of challenging tasks under the sparse reward setting where current RL techniques are known to be inefficient. To me, the problem is real and practical.\n2. The paper includes comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed method. Thorough ablation studies are also performed.\n\n### Cons:\n\n1. Although the results look good on the two environments provided by the authors, providing additional experiments using more challenging benchmarks, such as 3D continuous control tasks, would strengthen the paper. For example, the 7DoF robotic arm problem from [1] seems quite relevant.\n2. The think the writing can be improved. First, certain details are missing (e.g. how exactly the environment generation was implemented). Second, certain details (e.g. Algorithm pseudocode, model architecture, etc) can be moved to the appendix to make space for a Conclusion and Future work Section, which is noticeably missing. Third,, the figures 3 and 4 include texts and line names that is very small, and difficult to read (for someone who printed the paper using standard size). Please fix these.\n\n[1] T. Jurgenson, Orly Avner, E. Groshev, and Aviv Tamar. Sub-goal trees - a framework for goal-based reinforcement learning. ArXiv, abs/2002.12361, 2020.", "summary_of_the_paper": "This work introduces a method called Environment-Adversarial Sub-Task Curriculum  (EAT-C) wjocj automatically generates a curricuulm of task-environment couples for efficient RL. To this end, EAT-C trains two policies, in addition to the main decision-making policy. One policiy recursively decomposes hard task to coarse-to-fine subtasks, while the other performs adversarial modifications of the environment in each of the subtask. The authors show empirically that EAT-C leads to efficient training of the policy and superior generalization capabilities compared to existing baselines.", "summary_of_the_review": "The authors address an important problem and provide comprehensive experimental details, including qualitative analysis, quantitative results and thorough ablations. However, further experiments on more challenging domains would strengthen the paper. Also, the writing can be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2zABEvJvLa0", "reviewer": "Reviewer_3c96", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "======Strengths======\n\n1. The main idea of this approach is easy to follow and seems intuitive. The approach also shows promising results in the experiments. Though evaluated in simple tasks, applications in harder environments and tasks might be plausible if the subgoal decomposing and the environment generation could be adapted to more difficult environments and tasks. \n\n2. The figures and the qualitative examples are very helpful for readers to understand the core idea as well as how the auto-curriculum really works in the experiments.\n\n======Weaknesses======\n\n1. While the main idea of the algorithm is clearly presented, implementation details are missing. E.g., I could not find information about how the environment generation was implemented (e.g., action space). A more detailed description and/or code would be very helpful.\n\n2. The performance of the algorithm may heavily depend on the tree structure generation approach. The current work adopts a specific kind of approach, which may not work for other kinds of tasks. It would be good to discuss the limit/generalizability of the overall algorithm.\n\n3. The conclusion section is missing. It would be good to summarize the main contributions and findings at the end of the paper.\n\nSome suggested references:\n\n[1] Kaelbling, L., & Lozano-Perez, T. (2011). Hierarchical task and motion planning in the now. In Ieee international conference on\nrobotics and automation. (Classic work on task and motion planning, a well-known type of hierarchical planning.)\n\n[2] Shu, T., Xiong, C., & Socher, R. (2018). Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In ICLR. (Human-designed curriculum for learning the composition of subgoals via a tree structure.)\n\n[3] Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., & Mordatch, I. (2019). Emergent tool use from multi-agent autocurricula. In ICLR. (Autocurricula emerged from adversarial multi-agent RL training.)\n", "summary_of_the_paper": "This paper introduces an auto-curriculum generation approach to train better low-level policies conditioned on the subgoals generated by a path-planning high-level policy. The core idea is to use a planning policy to decompose a task into subgoals via a tree structure (multi-level subgoal decomposition) and use an environment generation policy to adversarially make the environment harder for the subgoal policy to perform. Thanks to the tree structure, the adversarial training can follow a bottom-up process, i.e., in an easy-to-hard order. The evaluation on a 2D object using task and a 2D grid world environment shows the effectiveness of this approach.", "summary_of_the_review": "The proposed algorithm presents an interesting and novel way to adversarially generate a curriculum for training low-level RL policies. The algorithm seems general to some extent and performed well in two tasks. However, the writing needs improvement (a lack of details, missing conclusion, etc.). There should also be a discussion on the limit of the current algorithm, which relies on the performance of a specific plan-planning method that provides the true structure. I am willing to raise my score if these problems are addressed in the revision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I thank the authors for their submission and active participation in the discussions. This papers is borderline with three reviewers leaning towards acceptance [3c96,7T33,Zhvq] and one leaning towards rejection [o38w]. Reviewer o38w's main concerns are around the lack of details about how the baselines were tuned and missing training details (specifically the connectivity test used to reject candidate environments). During discussion both, reviewers Zhvq and 7T33, agree that the paper requires substantial restructuring/rewriting to properly address the reviewer's feedback which is currently mostly addressed in the appendix. Based on the discussion with reviewers, my assessment is that this paper is not ready for publication at this point and that it will benefit greatly from another iteration. I want to very strongly encourage the authors to further improve their paper based on the reviewer feedback.
Decision: Reject