Year: 2021

Abstract:
Reinforcement learning (RL)’s efficiency can drastically degrade on long-horizon tasks due to sparse rewards and the learned policy can be fragile to small changes in deployed environments. We address these two challenges by automatically generating a curriculum of tasks with coupled environments. To this end, we train two curriculum policies together with RL: (1) a co-operative planning policy recursively decomposing a hard task into coarse-to-fine sub-task sequences as a tree; and (2) an adversarial policy modifying the environment (e.g., position/size of obstacles) in each sub-task. They are complementary in acquiring more informative feedback for RL: the planning policy provides dense reward of finishing easier sub-tasks while the environment policy modifies these sub-tasks to be adequately challenging and diverse so the RL agent can quickly adapt to different tasks/environments. On the other hand, they are trained using the RL agent’s dense feedback on sub-tasks so the sub-task curriculum keeps adaptive to the agent’s progress via this “iterative mutual-boosting” scheme. Moreover, the sub-task tree naturally enables an easyto-hard curriculum for every policy: its top-down construction gradually increases sub-tasks the planning policy needs to generate, while the adversarial training between the environment policy and the RL policy follows a bottom-up traversal that starts from a dense sequence of easier sub-tasks allowing more frequent modifications to the environment. Therefore, jointly training the three policies leads to efficient RL guided by a curriculum progressively improving the sparse reward and generalization. We compare our method with popular RL/planning approaches targeting similar problems and the ones with environment generators or adversarial agents. Thorough experiments on diverse benchmark tasks demonstrate significant advantages of our method on improving RL’s efficiency and generalization.

text: Reinforcement learning (RL)’s efficiency can drastically degrade on long-horizon tasks due to sparse rewards and the learned policy can be fragile to small changes in deployed environments. We address these two challenges by automatically generating a curriculum of tasks with coupled environments. To this end, we train two curriculum policies together with RL: (1) a co-operative planning policy recursively decomposing a hard task into coarse-to-fine sub-task sequences as a tree; and (2) an adversarial policy modifying the environment (e.g., position/size of obstacles) in each sub-task. They are complementary in acquiring more informative feedback for RL: the planning policy provides dense reward of finishing easier sub-tasks while the environment policy modifies these sub-tasks to be adequately challenging and diverse so the RL agent can quickly adapt to different tasks/environments. On the other hand, they are trained using the RL agent’s dense feedback on sub-tasks so the sub-task curriculum keeps adaptive to the agent’s progress via this “iterative mutual-boosting” scheme. Moreover, the sub-task tree naturally enables an easyto-hard curriculum for every policy: its top-down construction gradually increases sub-tasks the planning policy needs to generate, while the adversarial training between the environment policy and the RL policy follows a bottom-up traversal that starts from a dense sequence of easier sub-tasks allowing more frequent modifications to the environment. Therefore, jointly training the three policies leads to efficient RL guided by a curriculum progressively improving the sparse reward and generalization. We compare our method with popular RL/planning approaches targeting similar problems and the ones with environment generators or adversarial agents. Thorough experiments on diverse benchmark tasks demonstrate significant advantages of our method on improving RL’s efficiency and generalization.

heading: 1 INTRODUCTION
text: Although RL achieves breakthrough success or even outperform humans on a few challenging tasks (Lillicrap et al., 2016; Mnih et al., 2015; Florensa et al.), it is highly inefficient when targeting long-horizon tasks due to the sparse rewards collected via interactions. In addition, a policy trained in a specific complicated/simulated environment can be sensitive to small changes in the deployed environment and thus generalizes poorly in practice. Hence, selecting or generating more informative tasks and environments interacting with an agent is an essential challenge on the path towards more efficient, robust, and versatile RL. In this paper, we mainly study goal-conditioned RL (Kaelbling, 1993) whose policy is trained to adapt to any given goal/task: it is challenging, but the resulted policy can be applied to multiple goals/tasks.
The sparse reward problem has motivated different reward shaping/relabeling methods and curiositydriven exploration for RL, aiming to modify the task reward to be denser than states and actions. Hierarchical RL/planning (Elbanhawi & Simic, 2014; Nasiriany et al., 2019; Jurgenson et al., 2020; Pertsch et al., 2020) decomposes a complicated task/motion by searching a root-to-leaf path of sub-tasks on a tree such that a higher-level task invokes lower-level ones as its actions. However, building the tree requires hierarchical partition of the whole state-action space and prior knowledge to define the sub-tasks, which can be expensive or unavailable. Moreover, the pre-defined sub-tasks can be either too easy or too challenging for the RL-agent in-training. Reward shaping (Laud, 2004) relies on heuristics or external guidance to augment the sparse task-reward with dense rewards for exploring uncertain actions, rarely-visited states, or intermediate goals. Hindsight experience replay (Andrychowicz et al., 2017) and its variants relabels the achieved states as pseudo-goals with nonzero rewards, but many of them might be redundant to provide informative feedback efficiently improving the policy. Hence, automatically modifying the sparse reward or generating sub-tasks to
provide the most informative feedback maximizing the RL agent’s learning progress is still an open problem, and resolving it can result in more efficient interactive learning.
Another primary challenge in RL is how to improve its robustness and generalization to small changes in the environments. A growing number of recent studies (Pinto et al., 2017; Vinitsky et al., 2020; Ferguson & Law, 2018) show that RL policy can be vulnerable to small perturbations in its inputs so training an adversarial agent to perturb the inputs or compete with the original agent may improve RL’s robustness. This paper mainly focuses on improving RL policy’s tolerance to the physical differences between the training environment and possible deployed environment, e.g., changes of the position and size of obstacles/objects. In practice, this is important for deploying an RL policy trained in a simulator successfully to a realistic environment. Moreover, when addressing the sparse reward problem, the engineered sub-tasks or relabeled goals might be redundant or too easy to provide effective feedback to RL. However, adversarially modifying their environments can effectively improve their utility and RL policy’s robust generalization against diverse perturbed environments. Although modifying environment models has been recently studied to assist RL (Co-Reyes et al., 2020; Gur et al., 2021; Wang et al., 2019), directly applying this strategy to long-horizon tasks might make them even more challenging and reward-sparse, hence detrimental to RL’s efficiency.
In this paper, we address the above two challenges by automatically generating a curriculum of sub-tasks with adversarially perturbed environments to train the RL agent, where the curriculum is adaptive to the agent’s learning progress. To this end, we propose a marriage of subtasking and adversarial environments. The former decomposes a challenging long-horizon task into easier subtasks offering dense rewards, and the latter modifies each sub-task to be sufficiently challenging yet effective for improving RL policy’s tolerance to perturbed environments. As illustrated in Fig. 1, our approach, “environment-adversarial subTask curriculum (EAT-C)”, generates a tree-structured curriculum by (1) a path-planning policy that recursively decomposes an assigned task (e.g., a state-goal pair (s, g)) to sequences of sub-tasks (e.g., navigating between
consecutive sub-goals) with varying granularity (k = 0, . . . , 3); and (2) an environment policy that adversarially modifies each sub-task’s environment. The training objective for the path-planner is finding the most cost-efficient/shortest path in each level k, while the objective for the environment policy is to minimize the expected return of the RL agent on all sub-tasks. In EAT-C, training these two policies does not require external supervision or prior knowledge, we instead collect the time costs and rewards of the RL agent on previous sub-tasks to train them towards generating better curricula adaptive to RL progress, which then guide the training of the next episode.
Together with the RL policy, the two curriculum policies can efficiently learn from each other by iterating the above mutual-boosting scheme on the tree-structured curriculum of sub-tasks. A key advantage of the tree is that it naturally enables an easy-to-hard curriculum to train each of the three policies: (1) the top-down construction of the tree trains the path-planner to merely interpolate a few sub-goals between (s, g) at first and then gradually increases their number for more fine-grained and challenging planning; (2) the adversarial training between the environment policy and the RL policy follows a bottom-up traversal of the tree, i.e., they start from learning many easier sub-tasks and more frequent perturbations between (s, g) respectively, which are easy for both policies, and then progressively evolve to handle more challenging cases. i.e., a few long-horizon sub-tasks allow lessfrequent environment changes between (s, g). In experiments for discrete navigation and continuous
control tasks, we show that EAT-C is considerably efficient in learning to generalize to perturbed environments during finishing long-horizon tasks. Compared with recent RL methods equipped with hand-crafted curricula, environment generators, hierarchical policies, or path-planning, EAT-C exhibits promising advantages on efficiency and generalization to different tasks and environments.

heading: 2 RELATED WORK
text: Goal-conditioned RL (Pong et al., 2018; Kaelbling, 1993) takes a goal as an additional input to its policy and aims to handle different goals/tasks by the same policy. However, it requires extensive exploration and costly training on various goals but it still easily fails to reach distant goals in practice. Goal relabeling and reward shaping (Andrychowicz et al., 2017; Nasiriany et al., 2019) have been studied to mitigate these issues by modifying the task rewards to be dense but they introduce extra data bias and cannot control the utility of modified goals/rewards to the targeted ones. In order to provide tasks at the appropriate level of difficulty for the agent to learn, Held et al. (2018) and Racanière et al. (2019) train a goal/task generator based on the agent’s performance on existing tasks. But it is usually non-trivial to determine and tune the appropriate difficulty level for each training stage. In EAT-C, an RL agent achieves dense rewards by finishing a sequence of easier sub-goals towards the final goal, while the adversarial EG modifies each sub-task to be sufficiently challenging and diverse from other sub-tasks. Such sub-task curricula lead to more efficient goal-conditioned RL. Moreover, we do not need to carefully tune the difficulty levels of sub-tasks because the path-planner and EG together automatically generate the most informative tasks for each training stage of RL.
Planning algorithms (Sutton & Barto, 2018; LaValle, 2006) are helpful to solving long-horizon tasks (Levine et al., 2011) by interpolating intermediate sub-goals. In navigation, planning usually refers to finding the shortest path between two nodes(states) on a graph (LaValle, 2006; Dayan & Hinton, 1993) but a distance metric more accurate than Euclidean distance is usually unavailable. Moreover, it suffers from local minimal and performs poorly in narrow regions (Koren & Borenstein, 1991). Sequentially planning (Schmidhuber & Wahnsiedler, 1993; Zhang et al., 2020) sub-goals from the starting state to the goal state is inefficient in complex tasks, as it needs to search in a large space. In RL, planning requires an environment model or learns a value function to improve the policy (Levine et al., 2011; Lau & Kuffner, 2005; Elbanhawi & Simic, 2014), which can be as challenging as model-free RL. Control-based methods require accurate models for both the robot and the environment (Howard & Kelly, 2007; Werling et al., 2012), as well as an accurate distance metric (Eysenbach et al., 2019), which can be a rather daunting task. In EAT-C, we train a path-planner that “learns to plan” and to directly generate the shortest (in terms of time cost) path for the RL agent. Hence, it requires neither expensive search procedures nor a predefined distance metric. Moreover, it does not rely on learning a world/robot model or a value function. The easy-to-hard curriculum and dense feedback from the RL agent make it efficient to train. Furthermore, it does not need to plan for every step but only a few crucial waypoints.
Several recent works (Wang et al., 2019; Portelas et al., 2019; Gur et al., 2021) show that the efficiency and generalization of RL can be improved by generating/selecting different environments for training. Wang et al. (2019) studies a method “POET” to pair an adversarial environment generator with two agents to improve the generalization of the RL policy to novel environments. Portelas et al. (2019) proposes a teacher algorithm to sample environments based on a Gaussian mixture model capturing both the diversity of environments and the learning progress on them. This leads to a curriculum improving the efficiency of RL. However, the generated adversarial environments can be infeasible or too challenging for the RL agent and it is non-trivial to control their hardness. Moreover, modeling the distribution of environments requires collecting many samples from them and evaluating the RL policy on them, which can be costly and inaccurate especially for complicated environments. In EAT-C, we adversarially modify the environment for each easier sub-task, whose hardness is controlled by both the path planner and EG, so the modified sub-task is still feasible for the RL agent to learn. Moreover, a policy network for EG facilitates the modification of complicated environments and training it is efficient in our mutual boosting scheme with an easy-to-hard curriculum.
Hierarchical planning (HP) methods (Nasiriany et al., 2019; Jurgenson et al., 2020; Pertsch et al., 2020) search for a sequence of sub-goal on a tree to guide the exploration of an agent but building the hierarchical partition of all possible sub-goals can be expensive. A conventional hierarchical planning algorithm (Kaelbling & Lozano-Pérez, 2011) also learns to predict sub-tasks of a tree structure based
on a set of pre-defined motion primitives. Hierarchical RL (HRL) for goal-reaching tasks has been recently studied in (Zhang et al., 2021; Nachum et al., 2018). Shu et al. (2018) trains the RL agent on a human-designed curriculum of tree-structured sub-goals. They learn a sequence of primeval policies towards finishing complicated tasks, where the higher-level policies decompose a complex task into easier sub-tasks or motion primitives that can be addressed by the lower-level policies or controllers. However, the hierarchy of sub-tasks need to be either determined by prior knowledge or discovered automatically, which is usually challenging due to the huge space of possible sub-tasks. In contrast, EAT-C trains one planning policy to decompose a hard task into sub-tasks of multiple difficulty levels by only using the RL’s time cost data. It directly generates the sub-tasks and thus requires neither hierarchical partition of the task space nor a predefined set of sub-tasks. Compared to HP and HRL, EAT-C (1) enables a mutual training between the path-planner and the RL agent; (2) has an adversarial EG to adjust each sub-task’s environment to be more challenging; and (3) improves the training efficiency under the guidance of easy-to-hard curricula automatically generated for each stage.

heading: 3 PROBLEM FORMULATION

heading: 3.1 GOAL-CONDITIONED REINFORCEMENT LEARNING
text: Goal-conditioned RL or multi-goal RL learns a policy that can adapt to different goals. Given the state space S , the action space A, and the goal space G, a goal-conditioned policy is a mapping π(a|s, g) : S×G 7→ A that outputs an action a (or probabilities Pr(a|s, g) over actions a ∈ A) given a state-goal pair (s, g). An RL agent starts from a initial state s = s0 and uses π(a|s, g) to take a sequence of actions and interact with the environment, which determines the agent’s new state and reward after the action taking in each step. The interaction with the environment is defined by a Markov decision process (MDP) {S,A,G, p, r, γ}, where p(s′|s, a) , Pr(st+1 = s′|st = s, at = a) is the transition probability for the agent from state s to s′ after taking action a, r(s, a|g) : S ×A×G 7→ R is a reward function, and γ ∈ [0, 1] is a discount factor. For example, it is common to define r(s, a|g) , 1 {d (s, g) ≤ }, where 1 is the indicator function and d(·, ·) is a distance metric, so the agent achieves reward of 1 if reaching an -neighborhood of the goal g.
The learning objective of goal-conditioned RL is to maximize its expected return over different tasks (s0, g), i.e., maxπ E(s0,g)[Eπ(R0)] where the return at step-t is defined asRt = ∑T i=t γ
i−tr(st, at|g). This is usually used as the objective for policy gradient methods. Define the action-value function Q(s, a|g) , E(Rt|st = s, at = a, g), the optimal policy π∗ achieves the maximal Q(s, a|g) for any feasible (s, a, g). Define the value function V (s|g) , E(Rt|st = s, g) = Ea∼π[Q(s, a|g)] =∑ a∈A π(a|s, g)Q(s, a|g). To reduce the variance of Rt, Actor-critic methods additionally learns a model of V or Q as a “critic” to the “actor” π. Their training objectives aim to minimize the Bellman residual, i.e., the difference between the two sides of Bellman equation:
Q(st, at|g) = r(st, at|g) + γEst+1∼p[Ea∼π[Q(st+1, a|g)]]. (1)
In experiments, to encourage exploration, we use soft actor-critic (SAC) (Haarnoja et al., 2018b).

heading: 3.2 GENERATING SUB-TASK CURRICULA FOR GOAL-CONDITIONED RL
text: Training goal-conditional RL on long-horizon tasks can be highly inefficient since sparse rewards can only be achieved after taking a long sequence of actions, which cannot provide informative feedback to improve the policy. Moreover, the immature policy in earlier stages can easily fail and make the rewards even more sparse. In this paper, we train a path-planning policy πp to recursively decompose a long-horizon task into coarse-to-fine sequences of easier sub-tasks as a sub-task tree. Solving any of these sequence can accomplish the original task but the finer ones composed of more sub-tasks provide denser rewards to RL. Hence, the path-planner generates an easy-to-hard curriculum to train the RL agent, which starts from learning easier sub-tasks in finer sequences and gradually focus on more challenging sub-tasks in coarser ones. However, some sub-tasks can still be either too trivial for RL or redundant to other sub-tasks. Moreover, the RL agent is not trained to adapt to small changes in the environment. To address these two issues, we propose an environment generator (EG) πe that adversarially modifies the environment of each sub-task to be more challenging and informative for RL. Therefore, the path-planner and the EG are complementary in producing an efficient curriculum of tasks and environments to train the RL agent.

heading: 3.2.1 CO-OPERATIVE PATH PLANNER: LEARNING EASY-TO-HARD SUB-TASKS
text: We extend “sub-goal tree (SGT)” (Jurgenson et al., 2020) to generate our curriculum of sub-tasks. Given an long horizon task (s0, g) with initial state s0 and final goal g, we recursively apply a path planning policy πp(g|gi, gj) to interpolate sub-goals between s0 and g. In particular, given two sub-goals gi and gj , sampling from πp(g|gi, gj) yields a sub-goal interpolated between gi and gj . Hence, we can generate a sub-goal tree g0:T for (s0, g) by
Pr πp (g0:T |g0 = s0, gT = g) , Pr πp ( g0:T2 |g0, gT 2 ) Pr πp ( gT 2 :T |gT 2 , gT ) πp ( gT 2 |s0, g ) , (2)
where T = 2K with K being the depth of the tree. As shown in Fig. 1, layer-k of the sub-goal tree g0:T interpolate a sequence of 2k − 1 sub-goals gk1:2k−1 , ( gk1 , g k 2 , · · · , gk2k−1 ) between s0 and g,
where gkj = gTj/2k in g0:T ,∀j ∈ [ 2k − 1 ] . The goal of path planning is to generate cost-efficient sub-tasks for the RL agent, so we train πp by minimizing the time cost c(gk0:2k) of the sub-goal trajectory gk0:2k on each layer-k, i.e.,
min πp Jπp , E(s0,g)Eg1:T−1∼πp [ K∑ k=1 c(gk0:2k) ] , c(gk0:2k) , 2k−1∑ t=0 c ( gkt , g k t+1 ) , (3)
where c(gkt , g k t+1) represents the time-cost that the agent taken from g k t to g k t+1.

heading: 3.2.2 ADVERSARIAL-ENVIRONMENT GENERATOR (EG) APPLIED TO EACH SUB-TASK
text: Given the next sub-task (gkt , g k t+1) in layer-k, EG policy πe adversarially modifies the environment Ekt−1 of previous sub-task (g k t−1, g k t ) to be more challenging to the RL agent, i.e., sampling subtask-t’s environment Ekt ∼ πe(E|skt , g) where skt , (Ekt−1, gkt , gkt+1) denotes the state of EG at subtaskt. As an adversary to the RL agent, the reward function for EG is defined as re(skt , E k t |g) , −1 {r(st, at|g) = 1} where (st, at) refer to the state-action of the RL agent at the end of subtask-t. Thereby, EG receives the minimum reward −1 when the RL agent successfully finishes subtask-t and otherwise the reward is 0. We can also define an MDP for EG, which mainly differs from the RL agent in that each step corresponds to a sub-task so EG is only allowed to modify the environment at the beginning of each sub-task.
Similar to goal-conditioned RL defined in Sec. 3.1, the learning objective of EG is to maximize its expected return over different tasks (s0, g), i.e., maxπe E(s0,g)[Eπe(Re0)] where the return is defined as Ret = ∑K k=1 ∑2k i=t γ i−t e re(s k t , E k t |g) with discount factor γe ∈ [0, 1]. By defining the corresponding value function Ve and action-value function Qe as in Sec. 3.1, we can apply any RL algorithm to train EG, e.g., we use A2C (Mnih et al., 2016) for the experiments.

heading: 4 ENVIRONMENT-ADVERSARIAL SUB-TASK CURRICULUM

heading: 4.1 AUTO CURRICULUM GENERATION AND MUTUAL-BOOSTING
text: In EAT-C, we need to jointly train three policies, i.e., the path-planner πp and EG policy πe that generate tree-structured curricula of sub-tasks, and the RL policy π to accomplish the targeted tasks. At the first glance, training three policies can be more difficult than training one RL policy and requires to collect more data via interactions. Moreover, it is challenging to directly train a path-planner generating dense sub-goals and EG can also suffer from sparse rewards on long-horizon tasks. However, EAT-C allows the three policies help each other’s training via a mutual boosting mechanism, where each policy is progressively trained on a curriculum of easy-to-hard sub-tasks using dense feedback from other policies on the sub-tasks. By iterating this mutual-boosting on sub-task curricula, EAT-C significantly improves the training efficiency of each policy and results in an RL agent with better generalization to unseen tasks and perturbed environments.
In each episode, we train the path-planner during its “top-down” construction of a sub-task tree: it starts from learning to interpolate a few sub-goals between the given task (s0, g) and gradually moves to more challenging cases of generating dense sub-goals in bottom layers of the tree. Since it aims to generate the most cost-efficient path of sub-goals for the RL agent, we use the time cost of the RL agent on those sub-tasks in the previous episode as training data, which provide dense rewards to accelerate the training of the path-planner.
Given a constructed sub-task tree, we then train the RL policy and EG policy by a “bottom-up” traversal of the sub-task tree, which naturally forms an easy-to-hard curriculum for each policy. Specifically, both policies firstly learn from dense rewards by finishing easier sub-tasks in bottom layers, where the RL agent only needs to reach a nearby sub-goal and the EG is allowed to frequently modify the environments between (s0, g). Due to the adversarial training between them, they are not only learning from the environments but also from each other. As moving to the top layers, the RL agent receives less guidance from fewer sub-tasks, while the EG can only change the environment once in each long-horizon sub-task. Thereby, they need to improve their policies learned on easier tasks for more challenging tasks. As a result, the RL agent learns to adapt to different tasks and perturbed environments, while the EG learns to generate more powerful adversarial perturbations to train the RL agent in the next episode.
In EAT-C, for each long-horizon task, we iterate the above mutual-boosting training on new subtask curricula re-generated by the updated path-planner and EG for multiple episodes. This is an imitation of human learning that repeatedly practicing the same complicated task in different ways (e.g., different sub-tasking and perturbed environments). The path-planning and adversarial modification of environments are complementary in constructing a curriculum for more efficient RL: the former decomposes a hard task into easier sub-tasks while the latter modifies them to be sufficiently challenging and diverse so the RL agent can learn different skills with better generalization to unseen tasks or perturbed environments.

heading: 4.2 EAT-C ALGORITHM
text: Algorithm 1 EAT-C 1: Input: p0, T , τmax, , n 2: Output: RL agent’s policy π, planning policy πp, EG policy πe 3: Initialize: π, πp, Dp 4: while not converge do 5: Sample a task (s0, g) with s0 ∼ p0 and g ∈ G; 6: for episode = 1, 2, . . . , n do 7: Algorithm 2: top-down construction of a sub-task tree g0:T ,
train planning policy πp based on Dp; 8: Algorithm 3: bottom-up traversal of the sub-task tree g0:T , train RL policy π and EG policy πe, collect Dp; 9: end for
10: end while Top-Down Planning of Sub-task Curriculum We provide the detailed procedure of the top-down construction of the sub-task curriculum and the update of path-planner πp in Algorithm 2 (Appendix. C). For each layer-k from k = 0 to k = K, EATC firstly updates the planning policy πp in line 4 by an RL algorithm using the time cost data collected on subtasks up to layer-k from the previous episode’s bottom-up training (i.e., Algorithm 3, Appendix C), and then recursively generates the sub-tasks on layer-k for the current episode (line 5-8). Hence, the path-planner firstly learns to plan coarse trajectories of fewer sub-goals in top layers and gradually increases the sub-goals to form finer paths that can provide more guidance to RL in bottom layers. Since we always use the latest time cost data from Dp for training, the planning policy πp keeps being optimized to generate cost-efficient sub-task trajectories for the latest RL policy π.
Bottom-Up Curriculum for RL and EG Given a sub-task tree generated by Algorithm 2, EAT-C trains RL policy π and EG policy πe by following a bottom-up traversal of the tree. As shown in Algorithm 3, it starts from learning easier sub-tasks on the bottom layer-K and gradually moves to top layer-0 (line 4-13), which recovers the original long-horizon task. In each layer, we reset the initial state of the RL agent (line 5) and apply πe and π to a sequence of 2k sub-tasks gk0:2k−1 towards the final goal g (line 6-11), and then we update the two policies using the experiences collected on these sub-tasks (line 12). On each sub-task, EG adversarially perturbs the environment (line 7) and the RL agent is then applied to accomplish this modified sub-task (line 8). If the RL agent fails and ends at a state s, EAT-C recursively invoke the planning policy πp to add more sub-goals between s and the sub-task’s goal to provide more detailed guidance to the RL agent until it reaches the goal. This is described in line 9 and line 14-21.
EAT-C algorithm The complete procedure of EAT-C is introduced in Algorithm 1. Given a longhorizon task (s0, g) (line 5), EAT-C iterates between the top-down and bottom-up procedures in Algorithm 2-3 for n episodes (line 6-9) before moving to a new long-horizon task. Therefore, the planning policy πp and EG policy πe are optimized to produce better curricula of sub-tasks to train the RL agent to complete task (s0, g) while the RL agent learns skills for solving different sub-tasks and generalizing to non-trivial changes of the environment.

heading: 5 EXPERIMENTS
text: In this section, we evaluate EAT-C and compare it with a broad range of RL methods on two benchmarks, i.e., navigation and manipulation of a 6-point 2D robot to push an object to a goal state, and three compositional tasks in a discrete space. In these experiments, we mainly focus on their efficiency on long-horizon tasks and generalization to environments with small changes. Moreover, we present an ablation study to evaluate the contribution of each part in EAT-C. In addition, we provide case studies to analyze the planned sub-task tree and the modified environment for each sub-task, which explains why EAT-C can improve RL in several aspects. Our experiments are conducted on two representative RL benchmarks and a diverse set of baselines:
2D pusher (Yamada et al., 2020). As shown in Fig. 4, this is a robot navigation and manipulation task in a continues space: a 2D robot with a 4-joint arm needs to firstly navigate to an object and then push it to a goal location within an environment of multiple obstacles. We train EAT-C on 50 environments with the positions and sizes of obstacles randomly sampled from uniform distributions (more details in Appendix A). In each environment, we randomly sample 80 training tasks with different (s0, sobj , g), where s0 is the initial state, sobj is the state of the object, and g is the goal state. Most evaluated policies in our experiments need ≥ 250 steps to finish each task so they are long-horizon tasks. The rewards are sparse since the agent only receives a reward when near the object or when pushing the object and reaching the goal. In EAT-C, the path-planner generates sub-tasks between (s0, sobj) and (sobj , g). For every sub-task, EG can perturb the sizes and locations of at most three obstacles within pre-defined ranges. For the test, we randomly sample 30 new environments each having one randomly sampled task.
Discrete space tasks (Maxime Chevalier-Boisvert & Pal, 2018). We train and test RL policies on three types of compositional tasks depicted in fig 3, i.e., hunting, scavenging, and salad-making, as illustrated in Fig. 3.(a)-(c). The agent needs to take two or three key steps to finish each task and only get reward when finishing each key step. There are 250 environments used for training and 100 environments for test, each associated with one task. In EAT-C, the path-planner is applied to every two consecutive key steps. EG perturbs the environment by moving objects including tree and stones.
Baselines: We compare EAT-C with a broad class of RL methods having related ideas to EAT-C: (1) ALP-GMM (Portelas et al., 2019) that generates a curriculum of diverse tasks with large progress for goal-conditioned RL; (2) POET (Wang et al., 2019) with two auxiliary agents for generating a curriculum of adversarial yet solvable environments to accelerate RL; (3) Ecological RL (Co-Reyes et al., 2020) that dynamically modifies the environment to improve non-episodic (and thus longhorizon) RL without reset of the initial state; (4) A hierarchical RL method (Zhang et al., 2021); (5) Zhang et al. trains the RL agent by modelling a goal proposal curriculum that samples goals at the frontier of the set of goals that an agent is able to reach. All these baselines and EAT-C need to invoke an RL algorithm as their subroutine. For fair comparisons, we use Soft Actor Critic (SAC) (Haarnoja et al., 2018a) in all evaluated methods for its stable and promising performance. All methods are not allowed to modify the environments during test since test environments are assumed to be the realistic ones in which we deploy the RL agents. We run different methods for the same number of interaction steps with the environment, i.e., 20× 106 for 2D pusher and 1.25× 106 for discrete space tasks. More details of experimental settings are given in Appendix A.

heading: 5.1 MAIN RESULTS
text: We report the performance of EAT-C and all baselines evaluated on the test tasks in Fig. 2(a) for 2D pusher and in Fig. 3(d)-(f) for the discrete space tasks. In all experiments, EAT-C outperforms all other baselines by a large margin on both the learning efficiency and the final generalization performance to test tasks in new environments. In most experiments, baselines adopting a curriculum of environments, i.e., ALP-GMM, POET, and Ecological RL, performs worse than EAT-C but better than the other baselines without changing environments for training. This indicates that building a curriculum of training environments is essential to improving RL’s generalization and robustness to small changes in the deployed environments. Moreover, among the three compositional tasks in Fig. 3, hunting and scavenging contain moving object, i.e., the deer and the predator, which require the RL agent to adapt to the changes of their locations. On these two tasks, EAT-C exhibits more advantages over other baselines than on the salad-making task, which does not contain any moving object. Therefore, EAT-C enables RL to efficiently learn to adapt to changes in the environment.
By comparing the methods with environment changed, we notice that controlling the difficulty of the modified environments is critical to earlierstage learning efficiency. In particular, both EAT-C and POET trains another agent (e.g., the path-planner in EATC and the antagonist agent in POET) to control the hardness of the training tasks matching the capability of the RL agent, so their earlier-stage im-
provement than others. In contrast, the environments selected in ALP-GMM might be too challenging and the ones modified by POET might be too easy, leading to poorer efficiency in earlier stages. Although these methods are designed to train RL policies that can adapt to different environments or tasks, only EAT-C trains a path-planner to decompose a long-horizon task into a curriculum of easy-to-hard sub-task sequences for training. The guidance of path-planner and its curriculum plays a critical role in outperforming these baselines.
5.2 ABLATION STUDY
1
2
3
1
2
1
2
3
Hun�ng Step 1: Pick up axe Step 2: Hunt deer with axe Step 3: Eat food
Scavenging Step 1: Evade chasing predators Step 2: collect food
Salad-Making Step 1: Collect carrot Step 2: Collect le�uce Step 3: Make salad
(a) Hun�ng (b) Scavenging (c) Salad-Making
(d) Main result (Hun�ng) (e) Main result (Scavenging) (f) Main result (Salad-Making)
When the path-planner is removed from EAT-C, we no longer have any easy-to-hard curriculum of sub-tasks to train the RL agent or EG and they can only learn inefficiently from the original long-horizon tasks. The adversarial environments generated by EG make tasks for RL even harder and unsolvable. Hence, we can observe that it only completed 50% of the test tasks within 1.6×106 environment steps, compared to 90% of the original EAT-C. This indicates that the plan-planner and its generated sub-task tree are critical in creating an effective curriculum for RL.
When EG is removed from EAT-C, we still have the curriculum of sub-tasks to train the RL agent but some sub-tasks might be too trivial or redundant to provide informative feedback for improving the RL agent. This results in poorer efficiency in earlier-stages compared to the original EAT-C with an EG: it reaches 80% success rate after 0.84×106 interaction steps instead of 0.73×106 steps required by the original EAT-C. Although ETA-C without EG can eventually achieve a comparable success rate at the end of training. During later stages, the success rate fluctuates unstably over time, while EAT-C with an EG performs more robustly due to the adversarial environments used for training. Moreover, EAT-C significantly improves SAC by a large margin via running SAC on an automatically generated curriculum of sub-tasks, which implies the importance of curriculum on improving RL’s efficiency.

heading: 5.3 HOW DOES EAT-C WORK? AN EMPIRICAL STUDY
text: To better understand how the path planner and EG help RL in EAT-C, in Fig. 4, we visualize the sub-task tree (with layer k ∈ {0, 1, 2}) generated by the path-planner and the EG’s modifications to the environment in each sub-task at Episode 3 and Episode 6 (episode was defined in Alg. 1) for a 2D
Pusher task (s0, sobj , g). In the histograms, we also report the expected return of EG and the time cost of the RL agent on each sub-task from the bottom layer k = 2 for the two episodes.
The curriculum of sub-tasks generated by the path-planner. Each plot on the tree describes a sub-task, where the arrow highlights the sub-task and the yellow trajectory denotes the sequence of all sub-tasks of the layer. Comparing the sub-tasks in different layers, bottom layers (e.g., k = 2) provides more guidance and dense rewards to the RL agent while the sub-tasks in upper layers (e.g., k = 1) are much harder. Comparing the same-layer sub-tasks generated in different episodes, the sub-tasks in Episode 3 do not take all obstacles into account, e.g., some sub-task sequences trespass obstacles and some sub-tasks are too close to obstacles, because the planning policy is not fully optimized yet to produce a cost-efficient path for the RL agent. Hence, the time costs for the RL agent to accomplish these sub-tasks can be much higher than later episodes. Moreover, the time costs of some sub-tasks can be much higher than others and thus cannot provide dense rewards to assist RL. On the contrary, in Episode 6, the path-planner has learned to generate cost-efficient sub-tasks with similar hardness so the trajectories and sub-goals are distant from the obstacles and can provide dense rewards facilitating RL. Comparing the histograms of time costs for the two episodes, the RL agent is significantly improved by learning to complete the sub-tasks in the easy-to-hard (bottom-up) curriculum.
Adversarial modifications to obstacles in the environments: In each sub-task plot of Fig. 4, EG adversarially modifies some obstacles by changing their previous sizes and positions (depicted by the blue boxes) to make the sub-tasks sufficiently challenging and diverse. Similar to the path-planner, EG is improved over time: for example, in the sub-task “g0 to g1” on layer-2, the RL agent needs to pass a corridor formed by three obstacles, while EG makes the corridor longer and narrower and thus more challenging for the agent to pass in Episode 6, its modification in Episode 3 is not ideal and even moves one obstacle away from the agent. The improvement of EG is also reflected by its increasing expected return shown in the two histograms. By modifying the environments to be more difficult in sub-tasks, EG encourages the RL agent to learn diverse skills in different sub-tasks. Hence, the sub-tasks are easy for the agent to collect dense rewards but they are non-trivial and informative because of EG’s modifications.

heading: 6 CONCLUSION
text: We propose a mutual learning and auto-curriculum framework “EAT-C” to improve the efficiency of RL on long-horizon tasks as well as its generalization and robustness to new environments. EAT-C trains a planner to decompose a hard task into coarse-to-fine sequences of sub-tasks providing an easy-to-hard curriculum to train an RL agent, while an adversarial environment generator modifies these sub-tasks to be diverse and more informative to learn. The three policies are trained with data collected by each other. On two types of tasks, EAT-C outperforms a diverse set of baselines, e.g., curriculum-based RL, hierarchical RL, and planning-based methods. It has the potential to be applied to more complicated tasks with dynamic environments or visual inputs such as games, which will be covered in our future works.

heading: A DETAILS OF ENVIRONMENT IMPLEMENTATIONS
text: A.1 2D PUSHER
We simulate the environment of 2D pusher in Mujoco physics engine (Todorov et al., 2012). The positions of the robot, object and goal are defined as prob, pobj , and pgoal, respectively, and T is the maximum number of episodes (i.e., the horizon). An 2D-pusher agent with four joints can take actions of six dimensions, two for navigation and the rest four for arm control. The map size is 1× 1 so both the x and y coordinates lie in (−0.5, 0.5). The x and y coordinates of goals and objects are randomly sampled from uniform distributions of U(−0.35,−0.2) and U(−0.15, 0.1), respectively. The initial state of the agent is randomly sampled from uniform distribution of U(−0.05, 0.3). For obstacles, their initial coordinates and sizes are randomly drawn from an uniform distribution, as explained in the first row of Table. 1. We train 2D pusher using sparse reward: when the robot reaches a vicinity of the object or the sub-goal state (||prob − pobj ||2 6 0.05) the agent will receive a reward= 150/2k, where k is the layer of the sub-task tree where the sub-goal is located. Once the agent pushes the object to the goal state with ||pgoal − pobj ||2 6 0.05, the agent will receive a one-time reward= 150; otherwise there is no reward. By taking an action, EG can change the size and the location of 0 ∼ 3 obstacles near the agent. Assume that there are n obstacles in the environment, and we represent each obstacle-i by its location (xi, yi) (2D coordinates) and size (wi, hi) (width and height) as θi = (xi, yi, wi, hi). The action bt of EG is defined as
bt , ∆θi = (∆xi,∆yi,∆wi,∆hi), ∀i ∈ [n]. (4) In order to provide feasible and smoothly changing environments to RL along the sub-task trajectory in each layer, and to prevent the environment generator from being too powerful and overly adversarial, it is important to restrict EG from changing the environment too much at one time for each sub-task. Hence, in the experiments, we constrain every dimension in an action of EG not to exceed some threshold, e.g., for xi, we apply
∆xi ← min{max{∆xi, βx · xmin}, βx · xmax}, (5) where βx ∈ [0, 1] and (xmin, xmax) is the valid range of x-coordinate. The above environment parameterization can be easily extended to other environments so EAT-C is a general and principal scheme that can be adapted to different environments.
A.2 DISCRETE SPACE TASKS
The environment for the three tasks is an N ×N grid. It is partially observed by the RL agent: the agent at each state obtains a local egocentric view of a 5 × 5 grid around it, where the object in each cell of the grid is represented by a C-dimensional one-hot vector (there are C possible types of objects). The agent can pick up and carry one object at a time. It can also combine two objects to construct a new one by putting a carried object onto an existing object, e.g., it can combine wood with metal to make an axe. The RL agent can take action such as moving in the cardinal directions, picking up an object, and dropping an object. In discrete space tasks, the environment generator (EG) can modify the environment by taking an action to move an object/obstacle. In order to provide feasible and smoothly changing environments to RL along the sub-task trajectory, and to prevent EG from being too powerful and overly adversarial, each action of EG can only move every object/obstacle to an adjacent cell around it.
A.3 ENVIRONMENT ENCODING OF BASELINES
The baselines in our experiments include two curriculum RL methods, i.e., ALP-GMM and POET, which both can control some environment-dependent parameters for mutation and generation of the training environments. For their fair comparisons to EAT-C, we set their environment-dependent parameters exactly the same as the ones controlled by EG in EAT-C. For 2D-pusher, as detailed in Table 1, the baselines control the same four environment-dependent parameters as EAT-C. Each parameter is initialized by sampling from a uniform distribution and each mutation step can modify it by a small value if it is within the valid range. For the discrete space tasks, ALP-GMM/POET can generate environments by sampling/mutating the location of each obstacle/object, which is the same parameter controlled by EAT-C. The valid range for location for each obstacle/object is (0, 1) and the mutation step size is 0.1.

heading: B MODEL ARCHITECTURE AND HYPERPARAMETERS OF SAC (RL ALGORITHM USED IN ALL EXPERIMENTS)
text: We use the same neural network architecture (i.e., an MLP) for the RL agent and the same RL algorithm (SAC) in all the experiments of all the methods.
Besides the reward of completing a task/sub-task, it is common in MuJoCo and many other simulators to also issue a small instantaneous reward after taking any action in order to encourage exploration. Moreover, different methods usually need to re-scale this exploration reward because they may need different levels of exploration. In our experiments, we tune the re-scaling factor for every method to get its best performance. Specifically, we chose 0.3 for EAT-C/ALP-GMM/ POET and 8.0 for hierarchical RL/value disagreement/Ecological RL. An explanation of applying a smaller factor for the former three methods is that they already have some strong exploration strategies and a larger factor might downweigh the task reward and thus results in performance degeneration.
Moreover, we use the same coefficient α of the entropy term in SAC’s objective for all methods (they all use SAC as the RL algorithm). The coefficient α controls the degree of exploration and is automatically tuned. A complete list of hyperparameters for SAC in 2D-pusher tasks is given in Table 2. They are exactly the same hyperparameters defined in in SAC paper (Haarnoja et al., 2018a) and in Table 1 of their Appendix D except that we choose different values for them in 2D-pusher.
In the discrete space tasks, the environment is a 10× 10 grid and the 5× 5 partial observation (as mentioned in A.2) of the RL agent can be represented as a 5× 5× C one-hot tensor. We flatten this tensor to a vector and process it by an MLP with three hidden-layers whose output dimensions are (64, 64, 32), respectively. We apply another MLP with three layers of output dimensions (16, 16, 16) to process the inventory observation. The two MLPs’ outputs are then concatenated and processed by an MLP with two hidden layers of output dimensions (16 , action_dimension) that outputs a probability distribution over all possible actions. We use ReLU as our nonlinear activation functions in all MLP models except their last layer, which uses a softmax function to compute the probability of taking each action. In EAT-C, the RL agent and EG share the same observations as well as the first two MLP models but they use different MLP models to output the actions. A complete list of hyperparameters of SAC in the discrete space tasks is given in Table. 3.

heading: C PSEUDO-CODE OF EAT-C
text: In the training phase, we first randomly initialize πp, and apply it to predict a sub-task tree using Euclidean distance as an initialization of the cost in line 4 of Algorithm 2 when no time cost data have ever been collected at the very beginning. Given the sub-task tree, we can train the RL agent by a bottom-up curriculum of the sub-tasks on the tree. In particular, we start from the bottom layer and train the RL agent to consecutively complete a sequence of sub-tasks from the starting state to the goal state. As training proceed, the RL agent collects data of the time cost for completing feasible sub-tasks (g, g′). When the RL agent cannot complete the pre-assigned sub-task (gki , g k i+1) within a time limit, we re-apply πp to interpolate more sub-goals between (gki , g k i+1) by line 25 in Algorithm 3.
Algorithm 2 Top-Down Planning of Sub-task Curriculum 1: Input: (s0, g), T , planning policy πp and its training set Dp. 2: Output: tree structured sub-goals g0:T , πp 3: for k = 1, 2, . . . ,K do 4: Apply an RL algorithm to minimize Jπp in Eq. (3) computed on time cost data up to layer-k in Dp; 5: for t = 1, . . . , 2k−1 do 6: Generate the sub-goal gkt ∼ πp(gkt |gk−1t−1 , g k−1 t );
7: Add gkt , g k−1 t into the trajectory g k 0:T on layer-k; 8: end for 9: end for
More technical details are given in Algorithm 3. In the test phase, we apply πp to produce a sub-task tree and only use the sub-task sequence in the bottom layer to guide the RL agent.

heading: D ADDITIONAL EXPERIMENT RESULTS
text: D.1 LARGER VERSIONS OF PLOTS FOR MAIN RESULTS
Considering that the figures of experiment results in the main paper might be too small to read, we temporarily list the main results with a more clear vision (Fig. 5, and Fig. 6).
Algorithm 3 More detailed Bottom-Up traversal in EAT-C 1: Input: RL policy π, EG policy πe, sub-goal tree g0:T , τmax, 2: Output: π, πe, Dp 3: Initialize: πp’s training set Dp ← ∅, RL’s replay buffer D ← ∅, EG’s replay buffer De ← ∅ 4: for k = K, . . . , 1, 0 do 5: Reset RL agent’s initial state to g0; 6: for t = 1, 2, 3, . . . , 2k do 7: EG adversarially modifies the environment Ekt−1 to E k t ;
8: while τ ≤ τmax or sτ /∈ B(gkt , ) do 9: RL agent takes action aτ ∼ π(aτ |sτ , gkt );
10: RL agent moves to sτ+1 ∼ p(sτ+1|sτ , aτ ) and receives reward r(sτ , aτ |gkt ); 11: D ← D ∪ (sτ , aτ , r(sτ , aτ |gkt ), sτ+1); 12: end while 13: REACH(s, gkt+1); 14: end for 15: for every gradient step do 16: Apply gradient steps in SAC: update Q, V, π using samples drawn from D; 17: Apply gradient steps in A2C: update Qπe , πe using samples drawn from De; 18: end for 19: end for 20: Procedure REACH(s, g): 21: if d(s, g) ≤ then 22: De ← De ∪ (sτ , bt, re(sτ , Ekt |g), gkt ); 23: Dp ← Dp ∪ (s0, sτ , τ), s0 ← sτ ; 24: else 25: Re-apply πip(g k t−1.g k t ) to predict temporary sub-goals g ′ 1:nfor (g k t−1, g k t ); 26: Add g′1:n into the planned sub-goals trajectory g k 0:2k−1 ; 27: Re-apply agent start from s0 with the new sub-goal trajectory to reach gk2k−1 and collect training data (Follow line.8 to line.14); 28: REACH(s, g); 29: end if
r
D.2 ADDITIONAL ABLATION STUDY EVALUATING GENERALIZATION/ROBUSTNESS ON NEW RANDOM ENVIRONMENTS
Following the suggestions of Reviewer Zhvq and o38w, we add an ablation experiment and report the results in Table 4. To evaluate the generalization and robustness, which are the advantages of EAT-C due to the adversarial environment generator (EG), we evaluate different methods on multiple new random environments during the test phase. This is different from the ablation study in Fig. 2(b), which evaluates all methods on the fixed training environment. The new results show a large gap (80.24 vs. 42.23) between EAT-C and EAT-C (remove EG). This demonstrates that EG is important to improving the generalization and robustness of the RL policy. In the training environment (non-random) used in our original ablation study of Fig. 2(b), Hierarchical RL (HRL) does improve the performance of the default RL algorithm (SAC) on long-horizon tasks, i.e., 39.62 (SAC) vs. 68.27 (HRL). However, in random and unseen environments, HRL generalizes much poorer than EAT-C.
Reviewer o38w and Zhvq raised the following concerns: (1) whether EG could make the environment more reward sparse? (2) whether planner could always generate infeasible sub-goals? To answer these questions, in Fig 7, we report the average time-cost that the agent needs to complete each sub-task in layer-3 of the sub-task tree.
• In earlier stages when πp and the RL agent are not well trained, πp may generate hard subtasks. However, after a little training on the sub-task curriculum, πp is trained to generate a minimum-cost path for the RL agent and the capability of the RL agent to finish the sub-tasks is also improved.
• We apply EG to simple sub-tasks that are optimized to be simple in EAT-C (via optimizing the planner) for the RL agent. The goal of EG is to avoid learning similar and easy sub-tasks repeatedly, which cannot provide informative feedback to RL even if the reward is dense. On the other hand, we set several restrictions to avoid over-adversarial environments, as mentioned in Appendix. A
D.3 EXPERIMENTS ON 7DOF ROBOTIC ARM
Following the suggestions of Reviewer 7T33 and Zhvq, we add an experiment of controlling a 7DoF (degrees of freedom) robotic arm (i.e., the one used in (Jurgenson et al., 2020)) to evaluate how EAT-C performs in a complicated control task. We use MuJoCo as the simulator. In this experiment, the robotic arm learns to avoid obstacles and reach a goal state (as shown in Fig. 8). We compare EAT-C with curriculum RL methods and SGT (Jurgenson et al., 2020).
Both the training and test tasks have 5 obstacles with different and randomly sampled location and size parameters. The start-goal pair of each task are also randomly sampled. Both EAT-C and curriculum RL methods are able to modify exactly the same parameters defining the location and size of each obstacle.
We report the success rate of reaching the goal state without collision and the collision rate as the two metrics to evaluate EAT-C and all the baselines. After training, we evaluate them on 100 new random tasks different from the training tasks. The results are reported in table 5 and EAT-C achieves the best performance on both the collision rate and success rate among all evaluated methods. This demonstrates the robustness and promising performance of EAT-C in more complicated and realistic tasks.

heading: E MORE RELATED WORK
text: Refer to the suggestions of reviewers, we summarize the main difference of EAT-C with the suggested reference in this section.
The sub-goal generation in (Pertsch et al., 2020) follows a top-down and coarse-to-fine manner. However, they need to search for each sub-goal in the tree from many possible candidates, which is expensive and requires a search tree (hierarchical partition of the whole sub-goal space) much larger than our sub-goal tree (see Eq. 2). On the contrary, EAT-C learns a planning policy to directly generate su-bgoals and we do not need to build the search tree covering the whole sub-goal space. Another major difference is that they study a planning-only method while we study a mutual learning strategy between planning and RL to improve both planning and RL policies.
Zhang et al. (2020) trains a high-level policy to find the shortest path of sub-goals in a trained adjacency space. However, the distance between any two points in the adjacency space is expected to reflect the time cost of the agent navigating between the two points in the environment, which can be very challenging or even infeasible to achieve in many tasks (If we have such an adjacency space, both planning and RL can have dense feedback and simple supervised learning should work). In contrast, EAT-C trains a planner to directly generate a min-cost path of sub-goals through an easy-to-hard curriculum (fewer sub-goals interpolated at first), which provides an easier and more efficient solution without requiring learning an adjacency space. Moreover, the data used to train the planner in EAT-C are more informative than (Zhang et al., 2020) and cover multi-granularity since they are collected from RL when completing the bottom-up sub-task curriculum.
In (Dayan & Hinton, 1993), the high-level managers set a sequence of subgoals in the environment partitioned by Euclidean distance, which does not consider the obstacles or the RL agent capability. Hence, there is no mutual training between high-level (planner) and low-level (controller) managers in (Dayan & Hinton, 1993). On the contrary, the planner in EAT-C is jointly trained with the RL agent to produce a min-cost path of sub-goals for RL, which results in a more efficient curriculum of sub-tasks to train the RL agent.
Zhang et al. (2020) and Schmidhuber & Wahnsiedler (1993) plan sub-goals sequentially from the starting state to the goal state, which might be inefficient in complex tasks (requiring expensive search in a large space) and cannot produce the easy-to-hard curricula on a sub-goal tree as in EAT-C. In contrast, we train a planner to recursively produce coarse-to-fine sub-goal trajectories between the starting and goal states, which naturally provide an easy-to-hard curriculum for every component.

Review summaries:
The paper proposes an RL method targeted at long-horizon tasks in environments with perturbed dynamics or configurations. By combining a recently proposed goal-based RL method with an adversarial approach to domain randomization, the authors obtain a method that allows them to learn robust policies in sparse long-horizon settings. The method is compared with different baselines form hierarchical- or curriculum RL, demonstrating promising performance.

This work introduces a method called Environment-Adversarial Sub-Task Curriculum  (EAT-C) wjocj automatically generates a curricuulm of task-environment couples for efficient RL. To this end, EAT-C trains two policies, in addition to the main decision-making policy. One policiy recursively decomposes hard task to coarse-to-fine subtasks, while the other performs adversarial modifications of the environment in each of the subtask. The authors show empirically that EAT-C leads to efficient training of the policy and superior generalization capabilities compared to existing baselines.

This paper introduces an auto-curriculum generation approach to train better low-level policies conditioned on the subgoals generated by a path-planning high-level policy. The core idea is to use a planning policy to decompose a task into subgoals via a tree structure (multi-level subgoal decomposition) and use an environment generation policy to adversarially make the environment harder for the subgoal policy to perform. Thanks to the tree structure, the adversarial training can follow a bottom-up process, i.e., in an easy-to-hard order. The evaluation on a 2D object using task and a 2D grid world environment shows the effectiveness of this approach.