Adversarial Environment Generation for Learning  to Navigate the Web

1 INTRODUCTION . Autonomous web navigation agents that complete tedious , digital tasks , such a booking a flight or filling out forms , have a potential to significantly improve user experience and systems ’ accessibility . The agents could enable a user to issue requests such as , “ Buy me a plane ticket to Los Angeles leaving on Friday ” , and have the agent automatically handle the details of completing these tasks . However , the complexity and diversity of real-world websites make this a formidable challenge . General web navigation form-filling tasks such as these require an agent to navigate through a set of web pages , matching user ’ s information to the appropriate elements on a web page . This is a highly challenging decision-making problem for several reasons . First , the observation space is large , and partially-observable , consisting of a single web page in the flow of several web pages ( e.g . the payment information page is only one part of a shopping task ) . Web pages are represented using the Document Object Model ( DOM ) , a tree of web elements with hundreds of nodes . Second , actions are all possible combination of the web elements ( fill-in boxes , drop-downs , click on the buttons ) and their possible values . For example , the drop-down selection actions are only appropriate if there there is a drop-down menu present . Even if the agent is able to navigate the site to arrive at the correct page , and eventually select the correct element ( e.g . the ‘ departure ’ field for booking a flight ) , there are many possible values it can insert ( e.g . all user input ) . Therefore , the action space is discrete and prohibitively large , with only a valid set of actions changing with the context . Finally , the same task , such as booking a flight , results in a very different experience and workflow depending on the website . The agent must be able to adapt and operate in the new environment to complete the task . Therefore , the reinforcement learning ( RL ) agents should be capable of zero-shot generalization to new environments . Prior work made significant strides toward learning web navigation on a single website , yet the existing methods do not scale . Behavior cloning from expert demonstrations ( Shi et al. , 2017 ; Liu et al. , 2018 ) shows promising results , however , it requires a number of demonstrations for every single website . RL agent trained using synthetic demonstrations created with a generative model Gur et al . ( 2019 ) improves the performance . Yet , the method still requires training a separate policy for every single website requiring tens of thousands of interactions with every website . Lastly , the existing benchmarks ( Shi et al. , 2017 ; Liu et al. , 2018 ) have limited complexity . Their DOM trees are fixed and considerably smaller than real websites . We aim to train RL agents to solve web navigation form-filling tasks ; by correctly entering relevant information into unknown websites . Successful generalization to new websites requires training an agent on a large distribution of possible tasks and environments . The question is how to create a distribution that will not only cover most realistic tasks , but can be presented in a curriculum that is learnable by the agent . Manually designing a pre-defined curriculum of hand-built websites is tedious , and intractable . Another option would be to apply domain randomization ( DR ) ( as in e.g . Jakobi ( 1997 ) ; Sadeghi & Levine ( 2016 ) ; Tobin et al . ( 2017 ) ) to randomize parameters of websites , or automatically increase some parameter controlling the difficulty over time ( as in Gur et al . ( 2019 ) ) . However , all these approaches are likely to fail to cover important test cases , and can not tailor the difficulty of the parameter configuration to the current ability of the agent . Adversarial Environment Generation ( AEG ) trains a learning adversary to automatically generate a curriculum of training environments , enabling both increased complexity of training environments , and generalization to new , unforeseen test environments . However , if we naively train a minimax adversary—i.e . an adversary that seeks to minimize the performance of the learning agent—the adversary is motivated to create the hardest possible website , preventing learning . Instead , PAIRED ( Protagonist Antagonist Induced Regret Environment Design ) ( Dennis et al. , 2020 ) , trains the adversary to maximize the regret , estimated as a difference between two navigation agents ( protagonist and antagonist ) . While PAIRED shows exciting results , without an explicit feedback on how skillful antagonist is and mechanism to control the difficulty of the environment , the method is susceptible to local minima , and has hard time learning in the complex environments when the regret is zero . We present Flexible b-PAIRED , which builds on PAIRED framework , and jointly trains the adversarial RL agent ( adversary ) and a population of navigator agents . Flexible b-PAIRED adversary learns to present ” just-the-right-challenge ” to the navigation agents . We enable Flexible b-PAIRED adversary to tailor the environment difficulty to the ability of the best performing agent by introducing an explicit difficulty budgeting mechanism , and a novel multi-objective loss function . The budgeting mechanism gives the adversary the direct control of the difficulty of the generated environment . The adversary training simultaneously optimizes for an objective that ties in adversary difficulty budget with the navigator agent ’ s performance ( observed expected return ) , and the population-based regret similar to PAIRED . Lastly , to enable AEG web-design , we present a new benchmarking environment , gMiniWoB , and a web-design adversary architecture . gMiniWoB enables an adversary to construct websites of increasing complexity out of common design primitives such as navigation bars , product carousels , item decks , web forms , and item carts . The evaluation environments in gMiniWob are order of magnitude more complex than miniWob ( Shi et al. , 2017 ) . The adversary architecture is a LSTM-based decoder , seeded with a random seed . It first selects number of web pages . Then , at each step of an open loop , the adversary either emits a design element and its placement , or opts to skip an element and save design budget . The adversary ’ s used difficulty budget is a log-likelihood of joint probability of not adding design elements . This paper makes the following contributions : i ) A new benchmarking environment , gMiniWoB , which empowers the use of AEG for web navigation , by enabling the construction of websites out of compositional design primitives ; ii ) The Flexible b-PAIRED algorithm , which computes a more stable estimate of regret and directly incentivizes the adversary to tailor the complexity of the generated environment to the performance of the best-performing agent ; iii ) web navigation adversary decoder architecture , and iv ) empirical results demonstrating that Flexible b-PAIRED generates a curriculum of increasingly challenging websites , and produces agents that can successfully generalize to navigating complex , unseen sites at test time . Flexible b-PAIRED approach significantly outperforms prior work on minimax regret AEG ( Dennis et al. , 2020 ) , as well as a state-of-the-art approach for using RL to train web navigation agents ( Gur et al. , 2019 ) , resulting in agents that complete the most difficult tasks with more than 75 % success rate , 4x improvement over the strongest baseline . We are releasing gMiniWoB in open-source in the hopes of enabling further progress on this problem . We hope that this work will provide a meaningful way to make progress on the exceptionally challenging problem of learning to navigate the web , and will be of interest to the wider RL research community for auto-curriculum design in complex and compositional environments . 2 RELATED WORK . Web navigation benchmarks and tasks : Prior work on training agents to navigate the web introduced the MiniWoB ( Shi et al. , 2017 ) and MiniWoB++ ( Liu et al. , 2018 ) environments , a fixed set of manually curated toy websites , but relied on obtaining expert demonstrations for each website , which can not scale effectively to cover the large variety of real-world websites , and can not adapt to changing websites . Further , these methods failed to solve complex web navigation tasks such as flight booking or social media interaction ( Gur et al. , 2019 ) . Gur et al . ( 2019 ) take a step farther by training an RL agent to solve complex web navigation tasks using a scheduled curriculum . The curriculum linearly increases a parameter p , in which 1 − p controls the number of web elements that are solved by querying an oracle policy , which is obtained via expert data . This work differs in several ways . First , we introduce a new framework , gMiniWoB , that allows generating complex websites on-the-fly with tunable difficulty levels . Additionally , we do not rely on any expert demonstrations to augment sparse rewards . We use AEG to automatically learn to generate a curriculum of web navigation tasks that are tailored to the current skill level of the agent . Next , we make no assumption on the availability of any website while they assume websites are given a priori . Lastly , our web navigation agents generalize to unseen environments . Goal Generation : Florensa et al . ( 2018 ) trains a Generative Adversarial Network ( GAN ) for generating a curriculum of goals with fixed environment dynamics . A generator is trained to output new goals and the discriminator is trained to predict if the goal is achievable . The generator is bootstrapped from sample goals that the initial agent is able to reach in the environment . It is tested on simple navigation tasks with the same environments . In contrast , we train an adversary that generates a curriculum of environments , including goals , starting with an empty environment in which bootstrapping a generator network from sample episodes is not possible . We test on unseen environments with more complicated and high dimensional state and action spaces . Adversarial Environment Generation : Multi-agent training can be an effective method for automatically generating a curriculum of RL tasks ( e.g . Leibo et al . ( 2019 ) ; Matiisen et al . ( 2019 ) ; Graves et al . ( 2017 ) ; Portelas et al . ( 2020 ) ) . For example , Asymmetric Self Play ( ASP ) ( Sukhbaatar et al. , 2017 ) trains two agents , in which the second agent must learn to repeat the actions taken by the first , demonstrator agent . Both agents play in the same , fixed environment . In contrast , we use a third agent to learn to generate challenging new environments . POET ( Wang et al. , 2019 ; 2020 ) is Under review as a conference paper at ICLR 2021 DIV # text VAR * INPUT text=VAR * LABEL * DIV INPUT text= ” Username ” DIV # text “ First Name ” INPUT LABEL ( a ) An underspecified DOM tree template . The text box is always included , its text and label element are variables . DIV # text VAR * INPUT text=VAR * LABEL * DIV INPUT text= ” Username ” DIV # text “ First Name ” INPUT LABEL ( b ) A fully specified DOM primitive where a label is created and its text is assigned . DIV # text VAR * INPUT text=VAR * LABEL * DIV INPUT text= ” Username ” DIV # text “ First Name ” INPUT LABEL ( c ) A fully specified DOM primitive where only the inner text within the text box is assigned . Figure 2 : An example underspecified DOM tree template ( a ) and its instantiations ( b , c ) with different values . ( * ) indicates a variable ; either an element or one of its attributes . ( c ) is used in Page 1 and ( b ) is used in Page 2 in Figure 3. an AEG technique which uses a population of adversaries to generate the terrain a 2D walker agent must learn to navigate . To create a curriculum , POET requires generating many new environments , testing all agents within each one , and discarding environments based on a manually chosen a reward threshold , which wastes a significant amount of computation . Campero et al . ( 2020 ) use a teacher to propose navigation tasks ; the teacher ’ s reward is based on whether the agent takes more steps than a threshold , a hyperparameter that is linearly increased over the course of training . Most closely related to our work is PAIRED ( Dennis et al. , 2020 ) , which is an AEG method for training agents with minimal regret that works by constraining the environment-generating adversary using the performance of a second agent . However , PAIRED only demonstrated results on simple gridworld environments , and did not expand to the type of complex , high-dimensional state-action space required for web navigation . We improve on PAIRED using a more flexible estimate of the regret , as well as a budget mechanism , and show that this significantly improves performance . RL with Autoregressive Models : Keneshloo et al . ( 2020 ) outlines training sequence-to-sequence ( seq2seq ) models with RL algorithms . Previous models first pretrained a seq2seq model with ground-truth inputs and outputs and then finetuned with RL using different reward functions such as BLEU score . In this work , we propose a decoder-like autoregressive adversary model that is trained without any ground-truth data . The model is fed its own predictions from previous time steps and updated using a novel adversarial objective .

Summary: This paper improves upon existing approaches for learning to fill forms on the web automatically. The main idea is to train an adversary to generate a curriculum of environments to train an agent to learn to fill forms on the web. Training such an adversary can be challenging since the adversary may prove to be too strong for the main agent to learn anything from. Thus, the paper proposes few techniques to control or shape this adversary such that the main agent is able to learn quickly as compared to similar existing approach.