Environment Generation for Zero-Shot Compositional Reinforcement Learning

1 Introduction . Consider purchasing an airline ticket , logging in to a website , or buying movie tickets . These tasks can be completed by mastering a small set of basic manipulation skills ( primitives ) , such as entering an appropriate text in a fill-in field , or selecting a date , and combining them in different ways to form complex , compositional tasks [ 42 ] . Humans can easily generalize between compositional tasks – purchase a ticket on an airline or fill out a form that they have not seen before , even when the task carries over several pages – but training autonomous agents to do this is far from straightforward . Yet , unlocking generalization across related tasks would pave the way towards autonomous agents that can automatically handle the details of completing wide variety of real-world user requests such as , “ Buy me a plane ticket to Los Angeles leaving on Friday ” . The complexity and diversity of real environments make this a formidable challenge , especially due to the exponentially exploding action space and sparse rewards . Generalizing across compositional tasks is challenging for several reasons , including the tractability of training [ 42 ] . Many compositional tasks require planning over an excessively long horizon while providing only sparse rewards , making it difficult for RL agents to learn . Presenting easy tasks 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . until the agent learns can be a solution , but it is not clear how to construct such a curriculum for complex compositional tasks [ 20 ] . Manually designing a pre-defined curriculum is tedious , and intractable . Domain randomization ( DR ) [ 41 ; 17 ] does not tailor the difficulty of the task to the ability of the agent . Automatically generating tasks tailored to the agent ’ s learning progress is a promising approach , but as we will show , current approaches ( e.g . [ 8 ] ) suffer from fundamental limitations in the context of generating compositional tasks . We present a new automatic curriculum generation algorithm , Compositional Design of Environments ( CoDE ) , which jointly trains a Generator and a population of Learner agents . The generator is trained with a novel multi-objective reward function . Its first objective is to maximize the regret between the agents in the population and the best-performing agent , stabilizing regret estimation and making it less vulnerable to becoming stuck in local minima . Second , the generator is trained to adjust the task difficulty to match the learners ’ proficiency using an explicit difficulty incentive designed for use with compositional tasks . In this way , the generator builds more challenging environments when the learners are performing well , and reduces the difficulty when the learners are struggling . We demonstrate that this difficulty incentive addresses degenerative edge cases present in prior work . By automatically searching for tasks on which the learners are performing poorly , the generator makes the learners more robust , enabling them to generalize to unseen tasks at test time . This ability is necessary for many real-world tasks , such as web navigation . Prior work on web navigation relied on collecting demonstration data from a series of training websites [ 36 ; 21 ] , or training a separate policy for every single website [ 17 ] . These techniques can not scale to a production-quality system , because they would require re-training the agents every time an airline company updated its site . In contrast , we show that CoDE produces robust agents with general form-filling and navigation skills that can be applied zero-shot to novel tasks at test time . Another challenge in learning compositional tasks is that the existing benchmarks do not match real-life complexities . For example , real websites are orders of magnitude more complex than existing benchmarks [ 36 ; 21 ] . To address this problem , we introduce two new benchmark tasks for compositional task generation , which we are releasing in open-source . The first provides a way to automatically generate Minigrid [ 6 ] navigation tasks which include subtasks such as locating a key to open a door . The second , generative MiniWoB ( gMiniWob ) , focuses on web navigation and manipulation ( form-filling ) tasks and enables a generator to construct increasingly complex formfilling websites , spanning multiple pages , out of common design primitives such as navigation bars , product carousels , item decks , and item carts ( Figure 1 ) . The evaluation environments in gMiniWoB are orders of magnitude more complex than MiniWoB , the prior web navigation benchmark [ 36 ] . This paper makes the following contributions . First , we formally introduce compositional tasks by drawing a connection to the Petri Nets graph formalism , and showing its relationship to POMDPs . We then analyze why prior techniques for automatically generating a curriculum of tasks are insufficient for compositional tasks , and propose a new algorithm , CoDE that addresses these weaknesses . We build two new automatic task generation environments spanning simple navigation tasks and web navigation , and release both in open-source . We demonstrate strong empirical results across both domains . In the context of web navigation , we show that CoDE generates a curriculum of increasingly challenging websites . Resulting agents successfully generalize to complex , unseen sites at test time , and without additional training solve wide range of form-filling tasks from flight purchases to login in . CoDE agents solve the most difficult tasks with ⇡ 90 % success rate , 4x improvement over the strongest baseline . Lastly , we demonstrate that the method scales up to real websites . The implementation of CoDE and gMiniWoB framework are available in open source at https : //github.com/google-research/google-research/tree/master/compositional_rl . 2 Related work . Compositional task representation : Compositional tasks [ 3 ] have been represented as a vector sum of primitive skills [ 9 ] , or sequence of subtasks [ 30 ] . The dependency graph representation [ 37 ; 38 ] considers a task as a graph of subtasks equipped with dependencies as logical expressions . We relax the definition of the compositional tasks to represent real-world challenges . First , we allow primitives that do not lead to task completion . Second , we allow for partially observed environments . Third , we formalize the task distribution with a family of Petri Nets [ 34 ] , which have also been used to describe robot tasks [ 10 ] , planning tasks [ 27 ] , and workflows [ 1 ] . We use PNs with color and hierarchy , structured similarly to the workflows , but allow dangling nodes . Beccuti et al . [ 4 ] map PNs to MDPs , while we propose mapping between a set of PNs to a set of POMDPs . Curriculum Generation : A variety of curriculum learning methods exist which do not generate whole environments . For example , Florensa et al . [ 13 ] train a Generative Adversarial Network ( GAN ) to generate a curriculum of goal images for a goal-conditioned policy to reach . Other methods focus on choosing which task out of a set of pre-defined tasks to present next [ 29 ; 45 ; 12 ] . Another line of research studies creating curricula where a target task is already given [ 40 ; 31 ] . Multi-agent training can also be effective for automatically generating a curriculum [ 19 ; 22 ; 16 ; 32 ; 39 ; 28 ] . Czarnecki et al . [ 7 ] takes an alternative approach and generates a curriculum over agents instead of tasks . Finally , Active Domain Randomization [ 25 ; 33 ] learns to set the parameters of an environment simulator in order to provide a curriculum of increasingly difficult environments . In contrast , we focus on sequentially constructing an entire environment or task , composed of multiple subtasks , without a target task is explicitly given . Environment Generation : POET [ 43 ; 44 ] generates terrains for a 2D walker with a population of environment-generating adversaries . Campero et al . [ 5 ] train a teacher to propose navigation tasks based on a length threshold . Most closely related to our work is PAIRED [ 8 ] , which generates environments that induce maximal regret between a pair of agents . However , PAIRED only demonstrates results on simple gridworlds , and does not expand to complex , compositional tasks . We analyze several deficiencies in PAIRED that inhibit its use for the compositional setting , including a degenerate case where it fails to work at all . Our proposed algorithm addresses these deficiencies . To the best of our knowledge , we are the first work to attempt to automatically generate compositional tasks , and the first to apply environment generation to the problem of web navigation . Web navigation benchmarks and tasks : MiniWoB [ 36 ] and MiniWoB++ [ 21 ] environments contribute a fixed set of manually curated toy websites , and train agents using expert demonstrations for each website . This can not scale effectively to cover the large variety of real-world websites , and can not adapt to changing websites . Further , these methods failed to solve complex tasks such as flight booking or social media interaction . Using a manually scheduled curriculum [ 17 ] improves upon these results , but does not adapt to the agent ’ s progress or generalize to unseen sites . This work differs in several ways . First , we introduce a new framework , gMiniWoB , that allows generating complex websites on-the-fly with tunable difficulty levels . Additionally , we do not rely on any expert demonstrations to augment sparse rewards . Most importantly , our web navigation agents generalize to unseen environments , an important requirement for real-world web navigation agents . 3 Compositional tasks problem definition . Compositional tasks are composed of subtasks , which in turn are composed of primitive skills . This type of problem can be formalized using a task dependency graph , in which dependencies between the subtasks are represented as edges ( see Figure 2 ) . We model tasks in which the agent must complete several subtasks in order to trigger a phase transition in the environment , such as locating a key and unlocking a door in order to enter a new room , or completely navigating and correctly manipulating one web page in order to transition to the next one . We assume that the navigation agent uses standard POMDP setup – a tuple ( S , A , T , O , r , ) , where s 2 S are states , o ⇠ O ( s ) are partial observations , a 2 A are actions , and 2 [ 0 , 1 ) is a discount factor . At each timestep t , the agent selects action at according to its policy ⇡ , receives reward r ( at , st ) , and the environment transitions according to T ( st+1|st , at ) . Agents seek to maximize their return , the discounted sum of rewards : R ( ⇡ ) = PT t=0 t rt ( at , st ) . We use Petri Nets ( PNs ) [ 14 ; 11 ] framework to formally define learnable compositional tasks . PNs are directional graphs consisting place and transition nodes , and easily model different dependencies such as sequential , concurrent , choice , etc . Places are system states ; transitions are points in the process that takes the system to the next state . Edges determine the dependencies between nodes . At execution time , PNs have tokens that propagate through the net between places and determine the system state . In PNs with colors ( CPN ) [ 14 ] tokens are assigned a value , and Hierarchical PNs [ 11 ] replace subnets with transitions . We define a primitive as a workflow ( a special type of PN ) with multiple places/transitions where there is no parallelism ( see Figure 2a ) . They can be combined in parallel or serial using another transition ( ex : Figure 2b ) . The set of colors , C , represents data semantics that agent needs to place in the environment to complete the task ( i.e . flight departure or return dates ) . In the form filling task , agent needs to complete all the relevant fields before proceeding to the submit button that presents the next page . To that end , we introduce a gate , a special transition state that completes a phase and moves the system to the next ( e.g . move between the pages in web navigation , or move between rooms in navigation tasks ) . Learnable compositional tasks , is a family of a directed acyclic PNs with colors induced with a set of primitives PC . Learnable tasks described above map compositional tasks to POMDP ( S , A , T , O ) that define RL navigation agents , and generator architecture which allows the generator to generate tasks only with the structure defined by PN . See Appendix A.1 for more details on PetriNets and their connection to POMDPs .

Summary: The paper presents a method, Compositional Design of Environments (CoDE), for automatically constructing a curriculum over a complex domain of environments. The algorithm is framed as a multi-agent game: a generator constructs an environment using a sequence of actions, and afterwards a population of agents attempts to solve the constructed environment. Agents are trained to maximize the extrinsic reward they receive when attempting to solve the constructed environments. The generator reward, the main contribution of this paper, has two components: the first is a population-based regret computed as the difference between the average reward of the best performing agent in the population and the average reward across all agents in the population. The second component is a "difficulty" reward that encourages the generator agent to only complexify its constructed environments so long as one agent in the population remains capable of solving the task to a sufficient degree (given by a success threshold). If all agents fail the environment by another degree (the failure threshold), then the generator receives a negative reward, encouraging it to simplify its environment distribution when necessary.