Adaptive Procedural Task Generation for Hard-Exploration Problems

1 INTRODUCTION . The effectiveness of reinforcement learning ( RL ) relies on the agent ’ s ability to explore the task environment and collect informative experiences . Given tasks handcrafted with human expertise , RL algorithms have achieved significant progress on solving sequential decision making problems in various domains such as game playing ( Badia et al. , 2020 ; Mnih et al. , 2015 ) and robotics ( OpenAI et al. , 2019 ; Duan et al. , 2016 ) . However , in many hard-exploration problems ( Aytar et al. , 2018 ; Paine et al. , 2020 ) , such trial-and-error paradigms often suffer from sparse and deceptive rewards , stringent environment constraints , and large state and action spaces . A plurality of exploration strategies has been developed to encourage the state coverage by an RL agent ( Houthooft et al. , 2016 ; Pathak et al. , 2017 ; Burda et al. , 2019 ; Conti et al. , 2018 ) . Although successes are achieved in goal-reaching tasks and games of small state spaces , harder tasks often require the agent to complete a series of sub-tasks without any positive feedback until the final mission is accomplished . Naively covering intermediate states can be insufficient for the agent to connect the dots and discover the final solution . In complicated tasks , it could also be difficult to visit diverse states by directly exploring in the given environment ( Maillard et al. , 2014 ) . In contrast , recent advances in curriculum learning ( Bengio et al. , 2009 ; Graves et al. , 2017 ) aim to utilize similar but easier datasets or tasks to facilitate training . Being applied to RL , these techniques select tasks from a predefined set ( Matiisen et al. , 2019 ) or a parameterized space of goals and scenes ( Held et al. , 2018 ; Portelas et al. , 2019 ; Racanière et al. , 2020 ) to accelerate the performance improvement on the target task or the entire task space . However , the flexibility of their curricula is often limited to task spaces using low-dimensional parameters , where the search for a suitable task is relatively easy and the similarity between two tasks can be well defined . 1Project page : https : //kuanfang.github.io/apt-gen/ In this work , we combat this challenge by generating tasks of rich variations as curricula using procedural content generation ( PCG ) . Developed for automated creation of environments in physics simulations and video games ( Summerville et al. , 2018 ; Risi & Togelius , 2019 ; Cobbe et al. , 2020 ) , PCG tools have paved the way for generating diverse tasks of configurable scene layouts , object types , constraints , and objectives . To take advantage of PCG for automated curricula , the key challenge is to measure the learning progress in order to adaptively generate suitable tasks for efficiently learning to solve the target task . In hard-exploration problems , this challenge is intensified since the performance improvement can not always be directly observed on the target task until it is close to being solved . In addition , the progress in a complex task space is hard to estimate when there does not exist a well-defined measure of task difficulty or similarity . We can not always expect the agent to thoroughly investigate the task space and learn to solve all tasks therein , especially when the target task has unknown parameterization and the task space has rich variations . To this end , we introduce Adaptive Procedural Task Generation ( APT-Gen ) , an approach to progressively generate a sequence of tasks to expedite reinforcement learning in hard-exploration problems . As shown in Figure 1 , APT-Gen uses a task generator to create tasks via a black-box procedural generation module . Through the interplay between the task generator and the policy , tasks are continuously generated to provide similar but easier scenarios for training the agent . In order to enable curriculum learning in the absence of a direct indicator of learning progress , we propose to train the task generator by balancing the agent ’ s performance in the generated tasks and the task progress score which measures the similarity between the generated tasks and the target task . To encourage the generated tasks to require similar agent ’ s behaviors with the target task , a task discriminator is adversarially trained to estimate the task progress by comparing the agent ’ s experiences collected from both task sources . APT-Gen can thus be trained for target tasks of unknown parameterization or even outside of the task space defined by the procedural generation module , which expands the scope of its application . By jointly training the task generator , the task discriminator , and the policy , APT-Gen is able to adaptively generate suitable tasks from highly configurable task spaces to facilitate the learning process for challenging target tasks . Our experiments are conducted on various tasks in the grid world and robotic manipulation domains . Tasks generated in these domains are parameterized by 6× to 10× independent variables compared to those in prior work ( Wang et al. , 2019 ; 2020 ; Portelas et al. , 2019 ) . Each task can have different environment layouts , object types , object positions , constraints , and reward functions . In challenging target tasks of sparse rewards and stringent constraints , APT-Gen substantially outperforms existing exploration and curriculum learning baselines by effectively generating new tasks during training . 2 RELATED WORK . Hard-Exploration Problems . Many RL algorithms aim to incentivize the agent to visit more diverse and higher-reward states . Methods on intrinsic motivation augment the sparse and deceptive environment rewards with an additional intrinsic reward that encourages curiosity ( Pathak et al. , 2017 ; Burda et al. , 2019 ; Raileanu & Rocktäschel , 2020 ) and state novelty ( Conti et al. , 2018 ; Eysenbach et al. , 2019 ) . Another family of exploration techniques is derived from an informationtheoretical perspective as maximizing information gain of actions ( Houthooft et al. , 2016 ; Sun et al. , 2011 ) . When human demonstrations are available , they can be used to facilitate an RL agent to visit similar states and transitions as illustrated in the demonstrations ( Vecerik et al. , 2017 ; Nair et al. , 2018 ; Zhu et al. , 2018 ) . A combination of these techniques has been applied to solve hardexploration problems in video game domains ( Aytar et al. , 2018 ; Ecoffet et al. , 2019 ) . However , these methods have focused on learning in relatively simple and fixed environments , and usually can be ineffective in tasks where explorations are thwarted by stringent environment constraints or naively covering states does not lead to the task success . Curriculum Learning . Curriculum learning utilizes alternative datasets and tasks to accelerate the learning process of challenging target tasks ( Bengio et al. , 2009 ; Graves et al. , 2017 ) . To apply curriculum learning to RL , several recent works learn to adaptively select a finite set of easy tasks ( Narvekar et al. , 2017 ; Svetlik et al. , 2017 ; Riedmiller et al. , 2018 ; Peng et al. , 2018 ; Czarnecki et al. , 2018 ; Matiisen et al. , 2019 ; Narvekar & Stone , 2019 ; Lin et al. , 2019 ) or auxiliary rewards ( Jaderberg et al. , 2017 ; Shen et al. , 2019 ) hand-designed by human to maximize a progress signal defined on the target task . Parameterized tasks have been used to form a curriculum through the configuration of goals ( Forestier et al. , 2017 ; Held et al. , 2018 ; Racanière et al. , 2020 ) , environment layouts ( Wöhlke et al. , 2020 ; Baker et al. , 2020 ; Portelas et al. , 2019 ) , and reward functions ( Gupta et al. , 2018 ; Jabri et al. , 2019 ) . OpenAI et al . ( 2019 ) and Mehta et al . ( 2020 ) propose to actively adjust the hyperparameters in physical simulators to alleviate the domain shift . Most of these works are designed for task spaces parameterized by several discrete or continuous variables , where the task space can often be thoroughly explored and the similarity between two tasks could be well defined in the parameter space . In contrast , our approach is able to effectively generate tasks parameterized by a combination of high-dimensional discrete variables and several continuous variables . While most of these works focus on parameterizing a single aspect of the task environment , our approach learns to generate new tasks of rich variations with configurable initial state probability , transition probability , and reward function . Sukhbaatar et al . ( 2018b ) , Florensa et al . ( 2017 ) , and Sukhbaatar et al . ( 2018a ) propose to use an adversarial agent to set goals of growing difficulties by reversely traversing the state space from the goal . While this is related to the adversarial training framework in this paper in principle , we apply our framework beyond goal-reaching and reversible task domains . Procedural Task Generation . Procedural generation has been widely used in computer graphics and robotics ( Fisher et al. , 2012 ; Izadinia et al. , 2017 ; Majerowicz et al. , 2013 ; Izatt & Tedrake , 2020 ; Schwarz & Behnke , 2020 ) . While an increasing number of task sets have been designed to benchmark and empower reinforcement learning research ( Kolve et al. , 2017 ; Xia et al. , 2018 ; Savva et al. , 2019 ; Yu et al. , 2019 ; James et al. , 2020 ) , the design and implementation of each task often require nontrivial human expertise and heavy engineering . A few recent works utilize the random procedural generation of tasks ( Cobbe et al. , 2020 ; Fang et al. , 2018 ; Raileanu & Rocktäschel , 2020 ; Silver & Chitnis , 2020 ) . However , their generation algorithms are handcrafted with limited configurable features . Evolution strategies ( Wang et al. , 2019 ; 2020 ) , automated procedures ( Justesen et al. , 2018 ) , and learning-based methods ( Gravina et al. , 2019 ; Khalifa et al. , 2020 ; Bontrager & Togelius , 2020 ) have been proposed to automatically discover diverse games and task environments for training RL agents . Instead of covering the entire task space or discovering a diverse set of policies in an open-ended manner , our approach aims to train the policy to solve the target tasks of interest by utilizing the generated tasks . 3 ADAPTIVE PROCEDURAL TASK GENERATION . We consider a reinforcement learning problem involving a target task that the policy learns to solve and a parameterized task space that we utilize to generate new tasks . In practice , the parameterized task space can be created by a simulation program or a configurable procedure to set up the environment by a human or a robot in the real world . The target task can be an instance of an unknown parameter or a task outside of the task space , as long as there exist shared properties and transferable knowledge between the generated tasks and the target task . This follows the general paradigm of teacher-student curriculum learning ( Matiisen et al. , 2019 ; Portelas et al. , 2019 ) , while we allow the task space to be parameterized by either continuous or discrete high-dimensional variables and we do not assume the target task has a known parameterization by these variables . We propose Adaptive Procedural Task Generation ( APT-Gen ) , an approach for progressively generating tasks in highly configurable task spaces as curricula . To enable curriculum learning for hard-exploration problems , our key insight is that the learning progress can be jointly estimated by how well the policy can solve the current generated tasks and how similar the generated tasks are to the target task . Starting with a set of tasks that the policy can easily learn to solve , our approach progressively adapts the generated tasks towards the target task while maintaining their feasibility to the policy . As shown in Figure 1 , our approach creates tasks via a black-box procedural generation module by jointly learning the task generator , the task discriminator , and the policy .

Summary: This paper presents a procedurally content generation approach (APT-Gen) that generates a sequence of tasks for an agent to solve. These tasks are automatically generated in a way that helps an RL agent to learn hard-exploration problems. A main innovation of the approach is a task generator system that is both rewarded for generating tasks the agents can solve but also a sequence of tasks that are getting increasingly more similar to the target task.