ACTRCE: Augmenting Experience via Teacher’s Advice

Sparse reward is one of the most challenging problems in reinforcement learning ( RL ) . Hindsight Experience Replay ( HER ) attempts to address this issue by converting a failure experience to a successful one by relabeling the goals . Despite its effectiveness , HER has limited applicability because it lacks a compact and universal goal representation . We present Augmenting experienCe via TeacheR ’ s adviCE ( ACTRCE ) , an efficient reinforcement learning technique that extends the HER framework using natural language as the goal representation . We first analyze the differences among goal representation , and show that ACTRCE can efficiently solve difficult reinforcement learning problems in challenging 3D navigation tasks , whereas HER with non-language goal representation failed to learn . We also show that with language goal representations , the agent can generalize to unseen instructions , and even generalize to instructions with unseen lexicons . We further demonstrate it is crucial to use hindsight advice to solve challenging tasks , but we also found that little amount of hindsight advice is sufficient for the learning to take off , showing the practical aspect of the method . 1 INTRODUCTION . Many impressive deep reinforcement learning ( deep RL ) applications rely on carefully-crafted reward functions to encourage the desired behavior . However , designing a good reward function is nontrivial ( Ng et al. , 1999 ) , and requires a significant engineering effort . For example , even for the seemingly simple task of stacking Lego blocks , Popov et al . ( 2017 ) needed 5 complicated reward terms with different importance weights . Moreover , handcrafted reward shaping ( Gullapalli & Barto , 1992 ) can lead to biased learning , which may cause the agent to learn unexpected and undesired behaviors ( Clark & Amodei , 2016 ) . One approach to avoid defining a complicated reward function is to use a sparse and binary reward function , i.e. , give only a positive or negative reward at the terminal state , depending on the success of the task . However , the sparse reward makes learning difficult . Hindsight Experience Replay ( HER ) ( Andrychowicz et al. , 2017 ) attempts to address this issue . The main idea of HER is to utilize failed experiences by substituting with a fake goal in order to convert them to successful experiences . For their algorithm to work , Andrychowicz et al . made the non-trivial assumption that for every state in the environment , there exists a goal which is achieved in that state . As the authors pointed out , this assumption can be trivially satisfied by choosing the goal space to be the state space . However , representing the goal using the enormous state space is very inefficient and contains much redundant information . For example , if we want to ask an agent to avoid collisions while driving where the state is the raw pixel value from the camera , then there can be many states ( i.e . frames ) that achieve this goal . It is redundant to represent the goal using each state . Therefore , we need a goal representation that is ( 1 ) expressive and flexible enough to satisfy the assumption in HER , while also being ( 2 ) compact and informative where similar goals are represented using similar features . Natural language representation of goals satisfies both requirements . First , language can flexibly describe goals across tasks and environments . Second , language representation is abstract , hence able to compress any redundant information in the states . Recall the previous driving example , for which we can simply describe “ avoid collisions '' to represent all states that satisfy this goal . Moreover , the compositional nature of language provides transferable features for generalizing across goals . In this paper , we combine the HER framework with natural language goal representation , and propose an efficient technique called Augmenting experienCe via TeacheR ’ s adviCE ( ACTRCE ; pronounced “ actress ” ) to a broad range of reinforcement learning problems . Our method works as follows . Whenever an agent finishes an episode , a teacher gives advice in natural language to the agent based on the episode . The agent takes the advice to form a new experience with a corresponding reward , alleviating the sparse reward problem . For example , a teacher can describe what the agent has achieved in the episode , and the agent can replace the original goal with the advice and a reward of 1 . We show many benefits brought by language goal representation when combining with hindsight advice . The agent can efficiently solve reinforcement learning problems in challenging 2D and 3D environments ; it can generalize to unseen instructions , and even generalize to instruction with unseen lexicons . We further demonstrate it is crucial to use hindsight advice to solve challenging tasks , but we also found that little amount of hindsight advice is sufficient for the learning to take off , showing the practical aspect of the method . We note that our work is also interesting from a language learning perspective . Learning to achieve goals described in natural language is part of a class of problem called language grounding ( Harnad , 1990 ) , which has recently received increasing interest , as grounding is believed to be necessary for more general understanding of natural language . Early attempts to ground language in a simulated physical world ( Winograd , 1972 ; Siskind , 1994 ) consisted of hard coded rules which could not scale beyond their original domain . Recent work has been using reinforcement learning techniques to address this problem ( Hermann et al. , 2017 ; Chaplot et al. , 2017b ) . Our work combines reinforcement learning and rich language advice , providing an efficient technique for language grounding . 2 BACKGROUND . 2.1 REINFORCEMENT LEARNING AND DEEP Q-NETWORKS . We first review the traditional reinforcement learning setting , where an agent interacts with an infinite-horizon , discounted Markov Decision Process ( S , A , γ , P , r ) . Here , S is the state space , A is the action space , γ the discount factor , P the transition model and r is a reward function r : S ×A → R. We consider a policy πθ ( a|s ) parameterized by θ . At time t , the agent chooses an action at ∈ A according to its policy πθ ( a|st ) where st ∈ S is the current state . The environment in turn produces a reward r ( st , at ) and transitions to the next state st+1 according to the transition probability P ( st+1|st , at ) . The goal of the agent is to maximize the expected γ-discounted cumulative return Eπ [ ∑∞ t=0 γ tr ( st , at ) ] with respect to the policy parameters θ . The action-value function is denoted as Qπ ( st , at ) = Eπ [ ∑∞ i=t γ i−tr ( si , ai ) |s = st , a = at ] , which is the expected discounted sum of rewards obtained by performing an action a at state s and following the policy π thereafter . We denote π∗ as the optimal policy such that Qπ ∗ ( s , a ) ≥ Qπ ( s , a ) for every s ∈ S , a ∈ A and policy π . The optimal Q-function Q∗ = Qπ ∗ satisfies the Bellman equation : Q∗ ( s , a ) = Es′∼p ( ·|s , a ) [ r ( s , a ) + γ max a′∈A Q∗ ( s′ , a′ ) ] ( 1 ) Q-learning ( Sutton & Barto , 2018 ) is an off-policy , model-free RL algorithm that is based on the Bellman equation . The algorithm uses semi-gradient descent ( Sutton & Barto , 2018 ) to minimize the squared Temporal Difference ( TD ) error : L = E ( Q ( st , at ) − yt ) 2 , where the yt = rt + γmaxa′∈AQ ( st+1 , a ′ ) is the TD target . Deep Q-Network ( Mnih et al. , 2013 ) builds on the Q-learning algorithm and uses a neural network to approximate Q∗ . It also uses a replay buffer as well as a target network to improve training stability . 2.2 GOAL ORIENTED REINFORCEMENT LEARNING . We review the goal-oriented reinforcement learning framework following Schaul et al . ( 2015 ) . We augment the previously defined infinite-horizon , discounted Markov Decision Process with a goal space G. A goal g is chosen from G and stays fixed for each episode . The goal induces a reward function rg : S ×A → R , that assigns reward to a state conditioned on the given goal . At time step t , the agent π ( at|st , g ) chooses an action conditioned on the current state st , as well as the current goal g. The agent ’ s objective is to maximize the expected discounted cumulative return given the goal , i.e. , Eπ [ ∑∞ t=0 γ tr ( st , at , g ) ] . 2.3 HINDSIGHT EXPERIENCE REPLAY ( HER ) . We follow Andrychowicz et al . ( 2017 ) and consider a specific family of goal conditioned reward functions , where the reward is either 0 or 1 , depending on the resulting state , rg ( st , at ) = fg ( st+1 ) , where fg : S → { 0 , 1 } . In many scenarios , the function fg acts as a predicate that determines whether the agent is successful or not according to g , and only assigns positive reward at the terminal state . Hence the reward function is very sparse and makes it difficult for an agent to learn . HER proposes a solution to this problem . The agent first collects an episode of experiences s0 , a0 , r1 , ... , sT−1 , aT−1 , rT , sT under the goal g , where rT = rg ( sT−1 , aT−1 ) is the terminal reward . If rT is zero , one can replace g with g′ ∈ G such that rg′ ( sT−1 , aT−1 ) = 1 . With an off-policy algorithm , one is able to learn from the goal transformed experience . In order to flexibly relabel with a desirable goal , the authors assume that there exists a representation map from the state space to the goal space T : S → G , so that for each s ∈ S , the corresponding goal representation satisfies the predicate fg ( T ( s ) ) = 1 . However , such mapping is not simple to construct . Although the author pointed out a trivial construction can be done by taking G = S and fg ( s ) = [ g = s ] , such goal representation is redundant and uninformative , and largely limits the applicability of the algorithm . Consider an example where the state is the first-person raw pixel observation in a 3-D environment , and the goal is to walk towards a particular object . There are many possible directions to approach the object from , which results in many different possible states that satisfy the goal . However , no subset of the raw pixel observation can abstractly represent the goal . 3 AUGMENTING EXPERIENCE VIA TEACHER ’ S ADVICE ( ACTRCE ) . For a goal oriented MDP , { S , G , A , γ , P , rg } , and a parameterized agent π ( at|st , g ) , our objective is to maximize the expected discounted cumulative return , i.e. , Eπ [ ∑∞ t=0 γ trg ( st , at ) ] for each g ∈ G. Following HER , we assume there exists one or more more goal representation mappings T : S → G , which describe what goal is satisfied at each state . For example , in the original HER paper , the mapping T is simply a subspace projection from the state . Such representation can work well for their robotic tasks , where subset of the state ( i.e. , the coordinates of objects and body parts ) can reasonably well represent the goal . However , in general , such mapping can not abstract meaningful information about the goal , and will cause redundancy . Language provides an abstract representation for a goal and hence reduces redundancy in representation . This motivates our proposal to use natural language for representing the goal space . Concretely , for each state s ∈ S , we define T as a teacher that gives an advice T ( s ) , a natural language description of the goal that is achieved in the state s. To implement such a teacher , one can ask a human to look at a state and describe the goal in words , or generate a goal description automatically , as part of a narration or accessibility feature in a game . Given T , we can convert a failure trajectory to a successful one by relabeling the original goal with the teacher advice . To illustrate , say the original goal for the episode is `` Go to the armor '' , but at this particular time step , the agent has reached the blue torch instead . The teacher then tells the agent that the goal that reflects the current state is `` Go to blue torch '' . The agent can hence take the advice and use it as a positive experience . Besides positive reward signals , we also find that negative experience is necessary for training . Hence , in addition to a teacher who describes the achieved goal with a positive reward , we also introduce a teacher who gives negative reward with a description of a goal that has not been achieved . We further consider the scenario where there is more than one goal that can be satisfied in a state , e.g. , Algorithm 1 Augmenting Experience via Teacher ’ s Advice ( ACTRCE ) Given : • an off-policy RL algorithm A , and replay buffer R. . e.g . DQN , DDPG , NAF , SDQN • A language goal space G and a desired goal space Gd . • A group of teachers { T } for episode = 1 , M do Sample a desirable goal description g ∈ Gd and an initial state s0 . for t = 0 , T − 1 do Sample an action at using the behavioral policy from A : at ← πb ( st||g ) . || denotes concatenation end for for every teacher T do Compute advice g′ = T ( sT ) . . Teacher ’ s advice in natural language for t = 0 , T − 1 do r = rg′ ( st , at ) Store transitions { ( st||g′ , at , r , st+1||g′ ) } in R . HER end for end for Perform training with A and a minibatch B from the replay buffer . end for the state that is described as “ Reach a blue torch '' may also be described as “ Reach the largest blue object '' . Each different description of the goal corresponds to a different teacher ( see more discussions and experiments on variations of teachers in Appendix E ) . Therefore , in general , there is a group of teachers of different kinds , each giving different advice . We then relabel the original goal with each advice and corresponding positive or negative reward , and augment the replay buffer with these trajectories . Algorithm 1 describes our proposed approach formally . Note that in the above formulation , we assume a MDP setting , and let the teacher give advice based solely on the terminal state g′ = T ( sT ) . By contrast , in the POMDP setting , we ask the teacher to give advice based on the history of states and actions during the episode , i.e. , g′ = T ( { s0 , a0 , . . . , sT } ) .

Summary: This paper considers the assumption implicit in hindsight experience replay (HER), namely that we have access to a mapping from states to goals. Rather than satisfying this requirement by defining goals as states, which involves great redundancy, the paper proposes a natural language goal representation. Concretely, for every state a teacher is used to provide a natural language description of the goal achieved in that state, which can be used to directly relabel the goal so the episode can be used as a positive experience.