Learning to Navigate the Web

1 INTRODUCTION . We study the problem of training reinforcement learning agents to navigate the Web ( navigator agent ) by following certain instructions , such as book a flight ticket or interact with a social media web site , that require learning through large state and action spaces with sparse and delayed rewards . In a typical web environment , an agent might need to carefully navigate through a large number of web elements to follow highly dynamic instructions formulated from large vocabularies . For example , in the case of an instruction ” Book a flight from WTK to LON on 21-Oct-2016 ” , the agent needs to fill out the origin and destination drop downs with the correct airport codes , select a date , hit submit button , and select the cheapest flight among all the options . Note the difficulty of the task : The agent can fill-out the first three fields in any order . The options for selection are numerous , among all possible airport / date combination only one is correct . The form can only be submitted once all the three fields are filled in . At that point the web environment / web page changes , and flight selection becomes possible . Then the agent can select and book a flight . Reaching the true objective in these tasks through trial-and-error is cumbersome , and reinforcement learning with the sparse reward results in the majority of the episodes generating no signal at all . The problem is exacerbated when learning from large set of instructions where visiting each option could be infeasible . As an example , in the flight-booking environment the number of possible instructions / tasks can grow to more than 14 millions , with more than 1700 vocabulary words and approximately 100 web elements at each episode . A common remedy for these problems is guiding the exploration towards more valuable states by learning from human demonstrations and using pretrained word embeddings . Previous work ( Liu et al . ( 2018 ) ; Shi et al . ( 2017 ) ) has shown that the success rate of an agent on Web navigation tasks ( Miniwob ( Shi et al . ( 2017 ) ) ) can be improved via human demonstrations and pretrained word embeddings ; however , they indeed use separate demonstrations for each environment and as the complexity of an environment increases , these methods fail to generate any successful episode ( such as flight booking and social media interaction environments ) . But in environments with large state and action spaces , gathering the human demonstrations does not scale , as the training needs large number of human demonstrations for each environment . In this work , we present two methods for reinforcement learning in large state and action spaces with sparse rewards for the web navigation . First , when expert demonstrations or an instructionfollowing policy ( ORACLE ) are available , we develop curriculum-DQN , a curriculum learning that guides the exploration by starting with an easier instruction following task and gradually increasing the difficulty over a number of training steps . Curriculum-DQN decomposes an instruction into multiple sub-instructions and assigns the web navigation agent ( navigator ) with an easier task of solving only a subset of these sub-instructions ( Figure 1 ) . An expert instruction-following policy ( ORACLE ) places the agent and goal closer to each other . Second , when demonstrations and ORACLE policies are not available , we present a novel metalearning framework that trains a generative model for expert instruction-following demonstrations using an arbitrary web navigation policy without instructions . The key insight here is that we can treat an arbitrary navigation policy ( e. g. random policy ) as if it was an expert instruction-following policy for some hidden instruction . If we recover the underlying instruction , we can autonomously generate new expert demonstrations , and use them to improve the training of the navigator . Intuitively , generating an instruction from a policy is easier than following an instruction , as the navigator does not need to interact with a dynamic web page and take complicated actions . Motivated by these observations , we develop an instructor agent , a meta-trainer , that trains the navigator by generating new expert demonstrations . In addition to the two trainers , curriculum-DQN and instructor meta-trainer , the paper introduces two novel neural network architectures for encoding web navigation Q-value functions , QWeb and INET , combining self-attention , LSTMs , and shallow encoding . QWeb serves as Q-value function for the learned instruction-following policy , trained with either curriculum-DQN or instructor agent . The INET is Q-value function for the instructor agent . We test the performance of our approaches on a set of Miniwob and Miniwob++ tasks ( Liu et al . ( 2018 ) ) . We show that both approaches improve upon a strong baseline and outperform previous state-of-the-art . While we focus on the Web navigation , the methods presented here , automated curriculum generation with attention-equipped DQN , might be of interest to the larger task planning community working to solve goal-oriented tasks in large discrete state and action Markov Decision Processes . 2 RELATED WORK . Our work is closely related to previous works on training reinforcement learning policies for navigation tasks . Shi et al . ( 2017 ) and Liu et al . ( 2018 ) both work on web navigation tasks and aim to leverage human demonstrations to guide the exploration towards more valuable states where the fo- cus of our work is investigating the potential of curriculum learning approaches and building a novel framework for meta-training of our deep Q networks . Compared to our biLSTM encoder for encoding the Document Object Model ( DOM ) trees , a hierarchy of web elements in a web page , Liu et al . ( 2018 ) encodes them by extracting spatial and hierarchical features . Diagonal to using pretrained word embeddings as in ( Liu et al . ( 2018 ) ) , we used shallow encodings to enhance learning semantic relationships between DOM elements and instructions . Shah et al . ( 2018 ) also utilizes an attentionbased DQN for navigating in home environments with visual inputs . They jointly encode visual representation of the environment and natural language instruction using attention mechanisms and generate Q values for a set of atomic actions for navigating in a 3D environment . Modifying the reward function ( Ng et al . ( 1999 ) ; Abbeel & Ng ( 2004 ) ) is one practice that encourages the agent to get more dense reward signals using potentials which also motivated our augmented rewards for Web navigation tasks . Curriculum learning methods ( Bengio et al . ( 2009 ) ; Zaremba & Sutskever ( 2014 ) ; Graves et al . ( 2017 ) ) are studied to divide a complex task into multiple small subtasks that are easier to solve . Our curriculum learning is closely related to ( Florensa et al . ( 2017 ) ) where a robot is placed closer to a goal state in a 3D environment and the start state is updated based on robot ’ s performance . Meta-learning is used to exploit past experiences to continuously learn on new tasks ( Andrychowicz et al . ( 2016 ) ; Duan et al . ( 2016 ) ; Wang et al . ( 2016 ) ) . Frans et al . ( 2018 ) introduced a meta-learning approach where task-specific policies are trained in a multi-task setup and a set of primitives are shared between tasks . Our meta-learning framework differs from previous works where we generate instruction and goal pairs to set the environment and provide dense rewards for a low level navigator agent to effectively train . 3 SETUP . We train a RL agent using DQN ( Mnih et al . ( 2015 ) ) that learns a value function Q ( s , a ) which maps a state s to values over the possible set of actions a . At each time step , the agent observes a state st , takes an action at , and observes a new state st+1 and a reward rt = r ( st+1 , at ) . DQN aims to maximize the sum of discounted rewards ∑ t γ trt by rolling out episodes as suggested by Q ( s , a ) and accumulating the reward . We particularly focus on the case where the reward is sparse and only available at the end of an episode . More specifically , for only a small fraction of episodes that are successful , the reward is +1 ; in other cases it is −1 . This setup combined with large state and action spaces make it difficult to train a Q learning model that can successfully navigate in a Web environment . In this work , we further make the following assumption where we are given an instruction , I = [ F = ( K , V ) ] , as a list of fields F where each field is represented as a key-value pair ( K , V ) ( ex . { from : ” San Francisco ” , to : ” LA ” , date : ” 12/04/2018 ” } ) . At each time step , the state of the environment st consists of the instruction I and a representation of the web page as a tree Dt of DOM elements ( DOM tree ) . Each DOM element is represented as a list of named attributes such as tag , value , name , text , id , class . We also assume that the reward of the environment is computed by comparing the final state of an episode ( DN ) with the final goal state G ( I ) . Following Liu et al . ( 2018 ) , we constrain the action space to Click ( e ) and Type ( e , y ) actions where e is a leaf DOM element in the DOM tree and y is a value of a field from the instruction . Both of these composite actions are mostly identified by the DOM element ( e ) , e.g. , a text box is typed with a sequence whereas a date picker is clicked . This motivates us to represent composite actions using a hierarchy of atomic actions defined by the dependency graph in Figure 2 . Following this layout , we define our composite Q value function by modeling each node in this graph considering its dependencies : Q ( s , a ) = Q ( s , aD ) +Q ( s , aC |aD ) +Q ( s , aT |aD , [ aC == ” type ” ] ) ( 1 ) where a = ( aD , aC , aT ) is the composite action , aD denotes selecting a DOM element , aC |aD denotes a ” click ” or ” type ” action on the given DOM element , and aT |aD , [ aC == ” type ” ] denotes ” typing a sequence from instruction ” on the given DOM element . When executing the policy ( during exploration or during testing ) , the agent first picks a DOM element with the highest Q ( s , aD ) ; then decides to Type or Click on the chosen DOM element based on Q ( s , aC |aD ) ; and for a type action , the agent selects a value from the instruction using Q ( s , aT |aD , [ aC == ” type ” ] ) . 4 GUIDED Q LEARNING FOR WEB NAVIGATION . We now describe our proposed models for handling large state and action spaces with sparse rewards . We first describe our deep Q network , called QWeb , for generating Q values for a given observation ( st = ( I , Dt ) ) and for each atomic action aD , aC , aT . Next , we explain how we extend this network with shallow DOM and instruction encoding layers to mitigate the problem of learning a large number of input vocabulary . Finally , we delve into our reward augmentation and curriculum learning approaches to solve the aforementioned problems . Algorithm 1 Curriculum-DQN . Input : ORACLE policy Input : Sampling probability p and decay rate d Web Environment , ENVW for number of training steps do ( I , D , G ) ← Sample instruction , initial , and goal states from environment . //Warm-starting the episode . for each DOM element e in D do D̂ ← Get new state from ORACLE ( I , e ) with probability p. end for // Collect experience and train QWeb starting from the final state D̂ . Run Algorithm 2 ( QWeb , I , D̂ , ENVW ) Decrease p by a factor of d end for

Summary: This paper developed a curriculum learning method for training an RL agent to navigate a web. It is based on the idea of decomposing an instruction in to multiple sub-instructions, which is equivalent to decompose the original task into multiple easy to solve sub-tasks. The paper is well motivated and easily accessible. The problem tackled in this work is an interesting application of RL dealing with large action and state spaces. It also demonstrates superior performance over the state of the art methods on the same domains