Self-Educated Language Agent with Hindsight Experience Replay for Instruction Following

1 INTRODUCTION . Language has slowly evolved to communicate intents , to state objectives , or to describe complex situations ( Kirby et al. , 2015 ) . It conveys information compactly by relying on composition and highlighting salient facts . Such properties are essential when developing interactive agents in complex environments . As language may express a vast diversity of goals and situations , it could alleviate the training of interactive agents over heterogeneous and composite tasks thanks to its intrinsic structure ( Luketina et al. , 2019 ) . This property is all the more critical as classic Reinforcement Learning ( RL ) methods are facing generalization issues ( Cobbe et al. , 2018 ) , and learning hierarchical and structured policies remains an open-problem ( Barto & Mahadevan , 2003 ; Kulkarni et al. , 2016 ) . As recently advocated by Luketina et al . ( 2019 ) , language should thus be considered a first-class citizen to ease RL to improve on generalization and sample efficiency . Unfortunately , conditioning a policy on language also entails a supplementary difficulty as the agent needs to understand linguistic cues to alter its behavior . The agent thus needs to ground its language understanding by relating the words to its observations , actions , and rewards before being able to leverage the language structure ( Kiela et al. , 2016 ; Hermann et al. , 2017 ) . Once the linguistic symbols are grounded , the agent may then take advantage of language compositionality to condition its policy on new goals . It thus leads to the following questions : are we eventually making the reinforcement learning problem harder , or can we generate learning synergies between policy learning and language acquisition ? In this work , we use instruction following as a natural testbed to examine this question ( Tellex et al. , 2011 ; Chen & Mooney , 2011 ; Artzi & Zettlemoyer , 2013 ; Luketina et al. , 2019 ; Zang et al. , 2018 ; Hermann et al. , 2019 ; Chen et al. , 2019 ) . In this setting , the agent is given a text description of its goal ( e.g . ” pick the red ball ” ) and is rewarded when achieving it . The agent has thus to visually grounded the language , i.e. , linking and disentangling visual attributes ( shape , color ) from language description ( ” ball ” , ” red ” ) by using rewards to condition its policy toward task completion . On one side , the language compositionality allows for a high number of goals , and offers generalization opportunities ; but on the other side , it dramatically complexifies the policy search space . Besides , instruction following is a notoriously hard RL problem since it has a sparse reward signal . In practice , the navigation and language grounding problems are often circumvented by warm-starting the policy with labeled trajectories ( Zang et al. , 2018 ; Anderson et al. , 2018 ) . Although scalable , these approaches require numerous human demonstrations , whereas we here want to jointly learn the navigation policy and language understanding from scratch . In a seminal work , Hermann et al . ( 2017 ) successfully ground language instructions , but the authors used unsupervised losses and heavy curriculum to handle the sparse reward challenge . In this paper , we take advantage of language compositionality to tackle the lack of reward signals . To do so , we extend Hindsight Experience Replay ( HER ) to language goals ( Andrychowicz et al. , 2017 ) . HER originally deals with the sparse reward problems in spatial scenario ; it relabels unsuccessful trajectories into successful ones by redefining the policy goal a posteriori . As a result , HER creates additional episodes with positive rewards and a more diverse set of goals . Unfortunately , this approach can not be directly applied when dealing with linguistic goals . As HER requires a mapping between the agent trajectory and the goal to substitute , it requires expert supervision to describe failed episodes with words . Hence , this mapping should either be handcrafted with synthetic bots ( Chan et al. , 2018 ) , or be learned from human demonstrations , which would both limit HER generality . More generally , language adds a level of semantics , which allows generating textual objective that could not be encoded simple spatial observations as in regular HER , e.g. , ” fetch a ball that is not blue ” or ” pick any red object ” . In this work , we introduce Textual Hindsight Experience Replay ( THER ) , a training procedure where the agent jointly learns the language-goal mapping and the navigation policy by solely interacting with the environment illustrated in Figure 1 . THER leverages positive trajectories to learn a mapping function , and THER then tackles the sparse reward problem by relabeling language goals upon negative trajectories in a HER fashion . We evaluate our method on the BabyAI world ( Chevalier-Boisvert et al. , 2019 ) , showing a clear improvement over RL baselines while highlighting the robustness of THER to noise . 2 BACKGROUND AND NOTATION . In reinforcement learning , an agent interacts with the environment to maximize its cumulative reward ( Sutton & Barto , 2018 ) . At each time step t , the agent is in a state st ∈ S , where it selects an action at ∈ A according its policy π : S → A . It then receives a reward rt from the environment ’ s reward function r : S × A → R and moves to the next state st+1 with probability p ( st+1|st , at ) . The quality of the policy is assessed by the Q-function defined by Qπ ( s , a ) = Eπ [ ∑ t γ tr ( st , at ) |s0 = s , a0 = a ] for all ( s , a ) where γ ∈ [ 0 , 1 ] is the discount factor . We define the optimal Q-value as Q∗ ( s , a ) = maxπ Qπ ( s , a ) , from which the optimal policy π∗ is derived . We here use Deep Q-learning ( DQN ) to approximate the optimal Q-function with neu- ral networks and perform off-policy updates by sampling transitions ( st , at , rt , st+1 ) from a replay buffer ( Mnih et al. , 2015 ) . In this article , we augment our environment with a goal space G which defines a new reward function r : S × A × G → R and policy π : S × G → A by conditioning them on a goal descriptor g ∈ G. Similarly , the Q-function is also conditioned on the goal , and it is referred to as Universal Value Function Approximator ( UVFA ) ( Schaul et al. , 2015 ) . This approach allows learning holistic policies that generalize over goals in addition to states at the expense of complexifying the training process . In this paper , we explore how language can be used for structuring the goal space , and how language composition eases generalization over unseen scenarios in a UVFA setting . Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) is designed to increase the sample efficiency of off-policy RL algorithms such as DQN in the goal-conditioning setting . It reduces the sparse reward problem by taking advantage of failed trajectories , relabelling them with new goals . An expert then assigns the goal that was achieved by the agent when performing its trajectory , before updating the agent memory replay buffer with an additional positive trajectory . Formally , HER assumes the existence of a predicate f : S × G → { 0 , 1 } which encodes whether the agent in a state s satisfies the goal f ( s , g ) = 1 , and defines the reward function r ( st , a , g ) = f ( st+1 , g ) . At the beginning of an episode , a goal g is drawn from the space G of goals . At each time step t , the transition ( st , at , rt , st+1 , g ) is stored in the DQN replay buffer , and at the end of an unsuccessful episode , an expert provides an additional goal g′ that matches the trajectory . New transitions ( st , at , r′t , st+1 , g ′ ) are thus added to the replay buffer for each time step t , where r′ = r ( st , at , st+1 , g ′ ) . DQN update rule remains identical to ( Mnih et al. , 2015 ) , transitions are sampled from the replay buffer , and the network is updated using one step td-error minimization . HER assumes that a mapping m between states s and goals g is given . In the original paper ( Andrychowicz et al. , 2017 ) , this requirement is not restrictive as the goal space is a subset of the state space . Thus , the mapping m is straightforward since any state along the trajectory can be used as a substitution goal . In the general case , the goal space differs from the state space , and the mapping function is generally unknown . In the instruction following setting , there is no obvious mapping from visual states to linguistic cues . It thus requires expert intervention to provide a new language goal given the trajectory , which drastically reduces the interest of HER . Therefore , we here explore how to learn this mapping without any form of expert knowledge nor supervision . 3 TEXTUAL HINDSIGHT EXPERIENCE REPLAY . Textual Hindsight Experience Replay ( THER ) aims to learn a mapping from past experiences that relates a trajectory to a goal in order to apply HER , even when no expert are available . The mapping function relabels unsuccessful trajectories by predicting a substitute goal ĝ as an expert would do . The transitions are then appended to the replay buffer . This mapping learning is performed alongside agent policy training . Besides , we wish to discard any form of expert supervision to learn this mapping as it would reduce the practicability of the approach . Therefore , the core idea is to use environment signals to retrieve training mapping pairs . Instinctively , in the sparse reward setting , trajectories with positive rewards encode ground-truth mapping pairs , while trajectories with negative rewards are mismatched pairs . These cues are thus collected to train the mapping function for THER in a supervised fashion . We emphasize that such signals are inherent to the environment , and an external expert does not provide them . In the following , we only keep positive pairs in order to train a discriminative mapping model . Formally , THER is composed of a dataset D of 〈s , g〉 pairs , a replay buffer R and a parametrized mapping model mw . For each episode , a goal g is picked , and the agent generates transitions ( st , at , rt , st+1 , g ) that are appended to the replay buffer R. The Q-function parameters are updated with an off-policy algorithm by sampling minibatches from D. Upon episode termination , if the goal is achieved , i.e . f ( sT , g ) = 1 , the 〈sT , g〉 pair is appended to the dataset D. If the goal is not achieved , a substitute goal is sampled from the mapping model1 mw ( sT ) = ĝ′ and the additional transitions { ( st , at , rt , st+1 , ĝ′ ) } Tt=0 are added to the replay buffer . At regular intervals , the mapping 1The mapping model can be utilized with an accuracy criterion over a validation set to avoid random goal sampling . see algorithm 1 for more details modelmw is optimized to predict the goal g given the trajectory τ by sampling mini-batches fromD . Algorithm 1 summarizes our approach . Noticeably , THER can be extended to partially observable environments by replacing the predicate function f ( s , g ) by f ( τ , g ) , i.e. , the completion of a goal depends on the full trajectory rather than one state . Although we assess THER in the instruction following setting , the proposed procedure can be extended to any other goal modalities .

Summary: This paper proposes THER (textual hindsight experience replay), which extends the HER algorithm to the case when goals are represented as language. Whereas in HER the mapping from states to goals is the identity function, THER trains a separate modeling network which performs a mapping from states to goals represented by language. The policy (represented as a Q-function trained via DQN) takes in the goal (a command) as an additional argument as done in HER, which allows the agent to be commanded different tasks. The authors evaluate THER on the MiniGrid environment, where they demonstrate that THER greatly outperforms vanilla goal-conditioned DQN, even in the presence of significant label noise.