Environment Generation for Zero-Shot Compositional Reinforcement Learning

1 Introduction . Consider purchasing an airline ticket , logging in to a website , or buying movie tickets . These tasks can be completed by mastering a small set of basic manipulation skills ( primitives ) , such as entering an appropriate text in a fill-in field , or selecting a date , and combining them in different ways to form complex , compositional tasks [ 42 ] . Humans can easily generalize between compositional tasks – purchase a ticket on an airline or fill out a form that they have not seen before , even when the task carries over several pages – but training autonomous agents to do this is far from straightforward . Yet , unlocking generalization across related tasks would pave the way towards autonomous agents that can automatically handle the details of completing wide variety of real-world user requests such as , “ Buy me a plane ticket to Los Angeles leaving on Friday ” . The complexity and diversity of real environments make this a formidable challenge , especially due to the exponentially exploding action space and sparse rewards . Generalizing across compositional tasks is challenging for several reasons , including the tractability of training [ 42 ] . Many compositional tasks require planning over an excessively long horizon while providing only sparse rewards , making it difficult for RL agents to learn . Presenting easy tasks 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . until the agent learns can be a solution , but it is not clear how to construct such a curriculum for complex compositional tasks [ 20 ] . Manually designing a pre-defined curriculum is tedious , and intractable . Domain randomization ( DR ) [ 41 ; 17 ] does not tailor the difficulty of the task to the ability of the agent . Automatically generating tasks tailored to the agent ’ s learning progress is a promising approach , but as we will show , current approaches ( e.g . [ 8 ] ) suffer from fundamental limitations in the context of generating compositional tasks . We present a new automatic curriculum generation algorithm , Compositional Design of Environments ( CoDE ) , which jointly trains a Generator and a population of Learner agents . The generator is trained with a novel multi-objective reward function . Its first objective is to maximize the regret between the agents in the population and the best-performing agent , stabilizing regret estimation and making it less vulnerable to becoming stuck in local minima . Second , the generator is trained to adjust the task difficulty to match the learners ’ proficiency using an explicit difficulty incentive designed for use with compositional tasks . In this way , the generator builds more challenging environments when the learners are performing well , and reduces the difficulty when the learners are struggling . We demonstrate that this difficulty incentive addresses degenerative edge cases present in prior work . By automatically searching for tasks on which the learners are performing poorly , the generator makes the learners more robust , enabling them to generalize to unseen tasks at test time . This ability is necessary for many real-world tasks , such as web navigation . Prior work on web navigation relied on collecting demonstration data from a series of training websites [ 36 ; 21 ] , or training a separate policy for every single website [ 17 ] . These techniques can not scale to a production-quality system , because they would require re-training the agents every time an airline company updated its site . In contrast , we show that CoDE produces robust agents with general form-filling and navigation skills that can be applied zero-shot to novel tasks at test time . Another challenge in learning compositional tasks is that the existing benchmarks do not match real-life complexities . For example , real websites are orders of magnitude more complex than existing benchmarks [ 36 ; 21 ] . To address this problem , we introduce two new benchmark tasks for compositional task generation , which we are releasing in open-source . The first provides a way to automatically generate Minigrid [ 6 ] navigation tasks which include subtasks such as locating a key to open a door . The second , generative MiniWoB ( gMiniWob ) , focuses on web navigation and manipulation ( form-filling ) tasks and enables a generator to construct increasingly complex formfilling websites , spanning multiple pages , out of common design primitives such as navigation bars , product carousels , item decks , and item carts ( Figure 1 ) . The evaluation environments in gMiniWoB are orders of magnitude more complex than MiniWoB , the prior web navigation benchmark [ 36 ] . This paper makes the following contributions . First , we formally introduce compositional tasks by drawing a connection to the Petri Nets graph formalism , and showing its relationship to POMDPs . We then analyze why prior techniques for automatically generating a curriculum of tasks are insufficient for compositional tasks , and propose a new algorithm , CoDE that addresses these weaknesses . We build two new automatic task generation environments spanning simple navigation tasks and web navigation , and release both in open-source . We demonstrate strong empirical results across both domains . In the context of web navigation , we show that CoDE generates a curriculum of increasingly challenging websites . Resulting agents successfully generalize to complex , unseen sites at test time , and without additional training solve wide range of form-filling tasks from flight purchases to login in . CoDE agents solve the most difficult tasks with ⇡ 90 % success rate , 4x improvement over the strongest baseline . Lastly , we demonstrate that the method scales up to real websites . The implementation of CoDE and gMiniWoB framework are available in open source at https : //github.com/google-research/google-research/tree/master/compositional_rl . 2 Related work . Compositional task representation : Compositional tasks [ 3 ] have been represented as a vector sum of primitive skills [ 9 ] , or sequence of subtasks [ 30 ] . The dependency graph representation [ 37 ; 38 ] considers a task as a graph of subtasks equipped with dependencies as logical expressions . We relax the definition of the compositional tasks to represent real-world challenges . First , we allow primitives that do not lead to task completion . Second , we allow for partially observed environments . Third , we formalize the task distribution with a family of Petri Nets [ 34 ] , which have also been used to describe robot tasks [ 10 ] , planning tasks [ 27 ] , and workflows [ 1 ] . We use PNs with color and hierarchy , structured similarly to the workflows , but allow dangling nodes . Beccuti et al . [ 4 ] map PNs to MDPs , while we propose mapping between a set of PNs to a set of POMDPs . Curriculum Generation : A variety of curriculum learning methods exist which do not generate whole environments . For example , Florensa et al . [ 13 ] train a Generative Adversarial Network ( GAN ) to generate a curriculum of goal images for a goal-conditioned policy to reach . Other methods focus on choosing which task out of a set of pre-defined tasks to present next [ 29 ; 45 ; 12 ] . Another line of research studies creating curricula where a target task is already given [ 40 ; 31 ] . Multi-agent training can also be effective for automatically generating a curriculum [ 19 ; 22 ; 16 ; 32 ; 39 ; 28 ] . Czarnecki et al . [ 7 ] takes an alternative approach and generates a curriculum over agents instead of tasks . Finally , Active Domain Randomization [ 25 ; 33 ] learns to set the parameters of an environment simulator in order to provide a curriculum of increasingly difficult environments . In contrast , we focus on sequentially constructing an entire environment or task , composed of multiple subtasks , without a target task is explicitly given . Environment Generation : POET [ 43 ; 44 ] generates terrains for a 2D walker with a population of environment-generating adversaries . Campero et al . [ 5 ] train a teacher to propose navigation tasks based on a length threshold . Most closely related to our work is PAIRED [ 8 ] , which generates environments that induce maximal regret between a pair of agents . However , PAIRED only demonstrates results on simple gridworlds , and does not expand to complex , compositional tasks . We analyze several deficiencies in PAIRED that inhibit its use for the compositional setting , including a degenerate case where it fails to work at all . Our proposed algorithm addresses these deficiencies . To the best of our knowledge , we are the first work to attempt to automatically generate compositional tasks , and the first to apply environment generation to the problem of web navigation . Web navigation benchmarks and tasks : MiniWoB [ 36 ] and MiniWoB++ [ 21 ] environments contribute a fixed set of manually curated toy websites , and train agents using expert demonstrations for each website . This can not scale effectively to cover the large variety of real-world websites , and can not adapt to changing websites . Further , these methods failed to solve complex tasks such as flight booking or social media interaction . Using a manually scheduled curriculum [ 17 ] improves upon these results , but does not adapt to the agent ’ s progress or generalize to unseen sites . This work differs in several ways . First , we introduce a new framework , gMiniWoB , that allows generating complex websites on-the-fly with tunable difficulty levels . Additionally , we do not rely on any expert demonstrations to augment sparse rewards . Most importantly , our web navigation agents generalize to unseen environments , an important requirement for real-world web navigation agents . 3 Compositional tasks problem definition . Compositional tasks are composed of subtasks , which in turn are composed of primitive skills . This type of problem can be formalized using a task dependency graph , in which dependencies between the subtasks are represented as edges ( see Figure 2 ) . We model tasks in which the agent must complete several subtasks in order to trigger a phase transition in the environment , such as locating a key and unlocking a door in order to enter a new room , or completely navigating and correctly manipulating one web page in order to transition to the next one . We assume that the navigation agent uses standard POMDP setup – a tuple ( S , A , T , O , r , ) , where s 2 S are states , o ⇠ O ( s ) are partial observations , a 2 A are actions , and 2 [ 0 , 1 ) is a discount factor . At each timestep t , the agent selects action at according to its policy ⇡ , receives reward r ( at , st ) , and the environment transitions according to T ( st+1|st , at ) . Agents seek to maximize their return , the discounted sum of rewards : R ( ⇡ ) = PT t=0 t rt ( at , st ) . We use Petri Nets ( PNs ) [ 14 ; 11 ] framework to formally define learnable compositional tasks . PNs are directional graphs consisting place and transition nodes , and easily model different dependencies such as sequential , concurrent , choice , etc . Places are system states ; transitions are points in the process that takes the system to the next state . Edges determine the dependencies between nodes . At execution time , PNs have tokens that propagate through the net between places and determine the system state . In PNs with colors ( CPN ) [ 14 ] tokens are assigned a value , and Hierarchical PNs [ 11 ] replace subnets with transitions . We define a primitive as a workflow ( a special type of PN ) with multiple places/transitions where there is no parallelism ( see Figure 2a ) . They can be combined in parallel or serial using another transition ( ex : Figure 2b ) . The set of colors , C , represents data semantics that agent needs to place in the environment to complete the task ( i.e . flight departure or return dates ) . In the form filling task , agent needs to complete all the relevant fields before proceeding to the submit button that presents the next page . To that end , we introduce a gate , a special transition state that completes a phase and moves the system to the next ( e.g . move between the pages in web navigation , or move between rooms in navigation tasks ) . Learnable compositional tasks , is a family of a directed acyclic PNs with colors induced with a set of primitives PC . Learnable tasks described above map compositional tasks to POMDP ( S , A , T , O ) that define RL navigation agents , and generator architecture which allows the generator to generate tasks only with the structure defined by PN . See Appendix A.1 for more details on PetriNets and their connection to POMDPs .

Summary: The paper addresses the problem that RL is bad at learning the compositional rules guiding compositional problems. It proposes a two pronged approach: first, it proposes a task (de)composition strategy that allows building up tasks (or POMDPs) compositionally. Second, it proposes a regret-driven curriculum strategy that is able to construct compositional tasks of increasing complexity without supervision, which, in turn, can be used to train a learner that is then able to pick up on the rules of compositionality guiding the domain.