EAT-C: Environment-Adversarial sub-Task Curriculum for Efficient Reinforcement Learning

Reinforcement learning ( RL ) ’ s efficiency can drastically degrade on long-horizon tasks due to sparse rewards and the learned policy can be fragile to small changes in deployed environments . We address these two challenges by automatically generating a curriculum of tasks with coupled environments . To this end , we train two curriculum policies together with RL : ( 1 ) a co-operative planning policy recursively decomposing a hard task into coarse-to-fine sub-task sequences as a tree ; and ( 2 ) an adversarial policy modifying the environment ( e.g. , position/size of obstacles ) in each sub-task . They are complementary in acquiring more informative feedback for RL : the planning policy provides dense reward of finishing easier sub-tasks while the environment policy modifies these sub-tasks to be adequately challenging and diverse so the RL agent can quickly adapt to different tasks/environments . On the other hand , they are trained using the RL agent ’ s dense feedback on sub-tasks so the sub-task curriculum keeps adaptive to the agent ’ s progress via this “ iterative mutual-boosting ” scheme . Moreover , the sub-task tree naturally enables an easyto-hard curriculum for every policy : its top-down construction gradually increases sub-tasks the planning policy needs to generate , while the adversarial training between the environment policy and the RL policy follows a bottom-up traversal that starts from a dense sequence of easier sub-tasks allowing more frequent modifications to the environment . Therefore , jointly training the three policies leads to efficient RL guided by a curriculum progressively improving the sparse reward and generalization . We compare our method with popular RL/planning approaches targeting similar problems and the ones with environment generators or adversarial agents . Thorough experiments on diverse benchmark tasks demonstrate significant advantages of our method on improving RL ’ s efficiency and generalization . 1 INTRODUCTION . Although RL achieves breakthrough success or even outperform humans on a few challenging tasks ( Lillicrap et al. , 2016 ; Mnih et al. , 2015 ; Florensa et al . ) , it is highly inefficient when targeting long-horizon tasks due to the sparse rewards collected via interactions . In addition , a policy trained in a specific complicated/simulated environment can be sensitive to small changes in the deployed environment and thus generalizes poorly in practice . Hence , selecting or generating more informative tasks and environments interacting with an agent is an essential challenge on the path towards more efficient , robust , and versatile RL . In this paper , we mainly study goal-conditioned RL ( Kaelbling , 1993 ) whose policy is trained to adapt to any given goal/task : it is challenging , but the resulted policy can be applied to multiple goals/tasks . The sparse reward problem has motivated different reward shaping/relabeling methods and curiositydriven exploration for RL , aiming to modify the task reward to be denser than states and actions . Hierarchical RL/planning ( Elbanhawi & Simic , 2014 ; Nasiriany et al. , 2019 ; Jurgenson et al. , 2020 ; Pertsch et al. , 2020 ) decomposes a complicated task/motion by searching a root-to-leaf path of sub-tasks on a tree such that a higher-level task invokes lower-level ones as its actions . However , building the tree requires hierarchical partition of the whole state-action space and prior knowledge to define the sub-tasks , which can be expensive or unavailable . Moreover , the pre-defined sub-tasks can be either too easy or too challenging for the RL-agent in-training . Reward shaping ( Laud , 2004 ) relies on heuristics or external guidance to augment the sparse task-reward with dense rewards for exploring uncertain actions , rarely-visited states , or intermediate goals . Hindsight experience replay ( Andrychowicz et al. , 2017 ) and its variants relabels the achieved states as pseudo-goals with nonzero rewards , but many of them might be redundant to provide informative feedback efficiently improving the policy . Hence , automatically modifying the sparse reward or generating sub-tasks to provide the most informative feedback maximizing the RL agent ’ s learning progress is still an open problem , and resolving it can result in more efficient interactive learning . Another primary challenge in RL is how to improve its robustness and generalization to small changes in the environments . A growing number of recent studies ( Pinto et al. , 2017 ; Vinitsky et al. , 2020 ; Ferguson & Law , 2018 ) show that RL policy can be vulnerable to small perturbations in its inputs so training an adversarial agent to perturb the inputs or compete with the original agent may improve RL ’ s robustness . This paper mainly focuses on improving RL policy ’ s tolerance to the physical differences between the training environment and possible deployed environment , e.g. , changes of the position and size of obstacles/objects . In practice , this is important for deploying an RL policy trained in a simulator successfully to a realistic environment . Moreover , when addressing the sparse reward problem , the engineered sub-tasks or relabeled goals might be redundant or too easy to provide effective feedback to RL . However , adversarially modifying their environments can effectively improve their utility and RL policy ’ s robust generalization against diverse perturbed environments . Although modifying environment models has been recently studied to assist RL ( Co-Reyes et al. , 2020 ; Gur et al. , 2021 ; Wang et al. , 2019 ) , directly applying this strategy to long-horizon tasks might make them even more challenging and reward-sparse , hence detrimental to RL ’ s efficiency . In this paper , we address the above two challenges by automatically generating a curriculum of sub-tasks with adversarially perturbed environments to train the RL agent , where the curriculum is adaptive to the agent ’ s learning progress . To this end , we propose a marriage of subtasking and adversarial environments . The former decomposes a challenging long-horizon task into easier subtasks offering dense rewards , and the latter modifies each sub-task to be sufficiently challenging yet effective for improving RL policy ’ s tolerance to perturbed environments . As illustrated in Fig . 1 , our approach , “ environment-adversarial subTask curriculum ( EAT-C ) ” , generates a tree-structured curriculum by ( 1 ) a path-planning policy that recursively decomposes an assigned task ( e.g. , a state-goal pair ( s , g ) ) to sequences of sub-tasks ( e.g. , navigating between consecutive sub-goals ) with varying granularity ( k = 0 , . . . , 3 ) ; and ( 2 ) an environment policy that adversarially modifies each sub-task ’ s environment . The training objective for the path-planner is finding the most cost-efficient/shortest path in each level k , while the objective for the environment policy is to minimize the expected return of the RL agent on all sub-tasks . In EAT-C , training these two policies does not require external supervision or prior knowledge , we instead collect the time costs and rewards of the RL agent on previous sub-tasks to train them towards generating better curricula adaptive to RL progress , which then guide the training of the next episode . Together with the RL policy , the two curriculum policies can efficiently learn from each other by iterating the above mutual-boosting scheme on the tree-structured curriculum of sub-tasks . A key advantage of the tree is that it naturally enables an easy-to-hard curriculum to train each of the three policies : ( 1 ) the top-down construction of the tree trains the path-planner to merely interpolate a few sub-goals between ( s , g ) at first and then gradually increases their number for more fine-grained and challenging planning ; ( 2 ) the adversarial training between the environment policy and the RL policy follows a bottom-up traversal of the tree , i.e. , they start from learning many easier sub-tasks and more frequent perturbations between ( s , g ) respectively , which are easy for both policies , and then progressively evolve to handle more challenging cases . i.e. , a few long-horizon sub-tasks allow lessfrequent environment changes between ( s , g ) . In experiments for discrete navigation and continuous control tasks , we show that EAT-C is considerably efficient in learning to generalize to perturbed environments during finishing long-horizon tasks . Compared with recent RL methods equipped with hand-crafted curricula , environment generators , hierarchical policies , or path-planning , EAT-C exhibits promising advantages on efficiency and generalization to different tasks and environments . 2 RELATED WORK . Goal-conditioned RL ( Pong et al. , 2018 ; Kaelbling , 1993 ) takes a goal as an additional input to its policy and aims to handle different goals/tasks by the same policy . However , it requires extensive exploration and costly training on various goals but it still easily fails to reach distant goals in practice . Goal relabeling and reward shaping ( Andrychowicz et al. , 2017 ; Nasiriany et al. , 2019 ) have been studied to mitigate these issues by modifying the task rewards to be dense but they introduce extra data bias and can not control the utility of modified goals/rewards to the targeted ones . In order to provide tasks at the appropriate level of difficulty for the agent to learn , Held et al . ( 2018 ) and Racanière et al . ( 2019 ) train a goal/task generator based on the agent ’ s performance on existing tasks . But it is usually non-trivial to determine and tune the appropriate difficulty level for each training stage . In EAT-C , an RL agent achieves dense rewards by finishing a sequence of easier sub-goals towards the final goal , while the adversarial EG modifies each sub-task to be sufficiently challenging and diverse from other sub-tasks . Such sub-task curricula lead to more efficient goal-conditioned RL . Moreover , we do not need to carefully tune the difficulty levels of sub-tasks because the path-planner and EG together automatically generate the most informative tasks for each training stage of RL . Planning algorithms ( Sutton & Barto , 2018 ; LaValle , 2006 ) are helpful to solving long-horizon tasks ( Levine et al. , 2011 ) by interpolating intermediate sub-goals . In navigation , planning usually refers to finding the shortest path between two nodes ( states ) on a graph ( LaValle , 2006 ; Dayan & Hinton , 1993 ) but a distance metric more accurate than Euclidean distance is usually unavailable . Moreover , it suffers from local minimal and performs poorly in narrow regions ( Koren & Borenstein , 1991 ) . Sequentially planning ( Schmidhuber & Wahnsiedler , 1993 ; Zhang et al. , 2020 ) sub-goals from the starting state to the goal state is inefficient in complex tasks , as it needs to search in a large space . In RL , planning requires an environment model or learns a value function to improve the policy ( Levine et al. , 2011 ; Lau & Kuffner , 2005 ; Elbanhawi & Simic , 2014 ) , which can be as challenging as model-free RL . Control-based methods require accurate models for both the robot and the environment ( Howard & Kelly , 2007 ; Werling et al. , 2012 ) , as well as an accurate distance metric ( Eysenbach et al. , 2019 ) , which can be a rather daunting task . In EAT-C , we train a path-planner that “ learns to plan ” and to directly generate the shortest ( in terms of time cost ) path for the RL agent . Hence , it requires neither expensive search procedures nor a predefined distance metric . Moreover , it does not rely on learning a world/robot model or a value function . The easy-to-hard curriculum and dense feedback from the RL agent make it efficient to train . Furthermore , it does not need to plan for every step but only a few crucial waypoints . Several recent works ( Wang et al. , 2019 ; Portelas et al. , 2019 ; Gur et al. , 2021 ) show that the efficiency and generalization of RL can be improved by generating/selecting different environments for training . Wang et al . ( 2019 ) studies a method “ POET ” to pair an adversarial environment generator with two agents to improve the generalization of the RL policy to novel environments . Portelas et al . ( 2019 ) proposes a teacher algorithm to sample environments based on a Gaussian mixture model capturing both the diversity of environments and the learning progress on them . This leads to a curriculum improving the efficiency of RL . However , the generated adversarial environments can be infeasible or too challenging for the RL agent and it is non-trivial to control their hardness . Moreover , modeling the distribution of environments requires collecting many samples from them and evaluating the RL policy on them , which can be costly and inaccurate especially for complicated environments . In EAT-C , we adversarially modify the environment for each easier sub-task , whose hardness is controlled by both the path planner and EG , so the modified sub-task is still feasible for the RL agent to learn . Moreover , a policy network for EG facilitates the modification of complicated environments and training it is efficient in our mutual boosting scheme with an easy-to-hard curriculum . Hierarchical planning ( HP ) methods ( Nasiriany et al. , 2019 ; Jurgenson et al. , 2020 ; Pertsch et al. , 2020 ) search for a sequence of sub-goal on a tree to guide the exploration of an agent but building the hierarchical partition of all possible sub-goals can be expensive . A conventional hierarchical planning algorithm ( Kaelbling & Lozano-Pérez , 2011 ) also learns to predict sub-tasks of a tree structure based on a set of pre-defined motion primitives . Hierarchical RL ( HRL ) for goal-reaching tasks has been recently studied in ( Zhang et al. , 2021 ; Nachum et al. , 2018 ) . Shu et al . ( 2018 ) trains the RL agent on a human-designed curriculum of tree-structured sub-goals . They learn a sequence of primeval policies towards finishing complicated tasks , where the higher-level policies decompose a complex task into easier sub-tasks or motion primitives that can be addressed by the lower-level policies or controllers . However , the hierarchy of sub-tasks need to be either determined by prior knowledge or discovered automatically , which is usually challenging due to the huge space of possible sub-tasks . In contrast , EAT-C trains one planning policy to decompose a hard task into sub-tasks of multiple difficulty levels by only using the RL ’ s time cost data . It directly generates the sub-tasks and thus requires neither hierarchical partition of the task space nor a predefined set of sub-tasks . Compared to HP and HRL , EAT-C ( 1 ) enables a mutual training between the path-planner and the RL agent ; ( 2 ) has an adversarial EG to adjust each sub-task ’ s environment to be more challenging ; and ( 3 ) improves the training efficiency under the guidance of easy-to-hard curricula automatically generated for each stage .

Summary: This paper introduces an auto-curriculum generation approach to train better low-level policies conditioned on the subgoals generated by a path-planning high-level policy. The core idea is to use a planning policy to decompose a task into subgoals via a tree structure (multi-level subgoal decomposition) and use an environment generation policy to adversarially make the environment harder for the subgoal policy to perform. Thanks to the tree structure, the adversarial training can follow a bottom-up process, i.e., in an easy-to-hard order. The evaluation on a 2D object using task and a 2D grid world environment shows the effectiveness of this approach.