DHER: Hindsight Experience Replay for Dynamic Goals

1 INTRODUCTION . Deep reinforcement learning has been shown an effective framework for solving a rich repertoire of complex control problems . In simulated domains , agents have been trained to perform a diverse array of challenging tasks ( Mnih et al. , 2015 ; Lillicrap et al. , 2015 ; Duan et al. , 2016 ) . In order to train such agents , it is often the case that one has to design a reward function that not only reflects the task at hand but also is carefully shaped ( Ng et al. , 1999 ) to guide the policy optimization . Unfortunately , many of the capabilities demonstrated by reward engineering are often limited to specific tasks . Moreover , it requires both RL expertise and domain-specific knowledge to reshape the reward functions . For situations where we do not know what admissible behavior may look like , for example , using LEGO bricks to build a desired architecture , it is difficult to apply reward engineering . Therefore , it is essential to develop algorithms which can learn from unshaped and usually sparse reward signals . Learning with sparse rewards is challenging , especially when a goal is dynamic . Dynamic goals are common in games and planning problems , often addressed using reward shaping or search ( Kaelbling , 1993 ; Mnih et al. , 2015 ; Di Rocco et al. , 2013 ) . However , the difficulty posed by a sparse reward is exacerbated by the complicated environment dynamics in robotics ( Andrychowicz et al. , 2017 ) . For instance , system dynamics around contacts are difficult to model and induce sensitivity in the system to small errors . Many robotic tasks also need executing multiple steps successfully over a long horizon , involve enormous search space , and require generalization to varying task instances . Policy gradient methods are breakthroughs in the challenging environments , such as PPO ( Heess et al. , 2017 ; Schulman et al. , 2017 ) , ACER ( Wang et al. , 2016 ) , TRPO ( Schulman et al. , 2015 ) and so on . They are used in environments , where an agent tries to reach a target , learns to walk , runs , and so on . Recently , sampling-efficient learning is introduced and demonstrates a significant increase in performance for off-policy actor-critic DQN ( Mnih et al. , 2015 ) and DDPG ( Lillicrap et al. , 2015 ) algorithms . Hindsight experience replay ( HER ) is very effective for improving the performance of off-policy RL algorithms in solving goal-based tasks with sparse rewards ( Andrychowicz et al. , 2017 ) . Similar to UVFA ( Schaul et al. , 2015a ) , it takes a goal state as part of input . However , it ∗Correspondence to : Meng Fang < mfang @ tencent.com > . assumes the goal is fixed . As a result , this assumption actually impedes the learning of RL agents in the environments of moving goals . In this paper , we address this challenge with a new method , Dynamic Hindsight Experience Replay ( DHER ) , for accomplishing tasks with moving goals . We follow the multi-goal setting in UVFA ( Schaul et al. , 2015a ) and HER ( Andrychowicz et al. , 2017 ) . It assumes that the goal being pursued does not influence the environment dynamics . We also need to have the knowledge of goal similarity . For example , in manipulation or grid world domains , we can use Euclidean distance between positions to measure the goal similarity . HER turns a failed episode to a success by composing a new task whose goal is achieved by that episode . Our idea allows an agent to learn from the failure one step further than HER : the agent not only sets a new goal but also hallucinates how to reach the original goal from the new one . Take playing frisbee for instance . When an agent jumps to catch the frisbee and yet misses it , the agent receives no positive feedback under the sparse reward setting . Using HER , the agent could set the end of its episode as the new goal — position of the frisbee ; with DHER , however , the agent finds a trajectory from its past experiences as the imagined path of the frisbee , and thereby extrapolates towards the original goal . In particular , we do the following for DHER . To finish the tasks with dynamic goals needs to explore experience and understand multiple goals . DHER uses replay buffers to allow the agent to learn from a couple of failures by assembling new ‘ experience ’ from different episodes . The proposed method retrieves memories to find the connection between the experience of different episodes . It largely improves the sample efficiency in dynamic goal task settings . More importantly , this strategy makes it possible to learn in the setting that both sparse rewards and dynamic goals exist . We evaluate our method along with the state-of-the-art baselines on new environments and manipulation tasks , which have sparse rewards and dynamic goals . Our results demonstrate that DHER is clearly better than others for these tasks . We also transfer policies trained in our simulation based on DHER to a physical robot and show that DHER can be applied to solving real-world robotics problems . We summarize our main contributions as follows : ( 1 ) We demonstrate that , both in simulation and real worlds , DHER succeeds in continuous control with a moving target . To our knowledge , this is the first empirical result on manipulation tasks that demonstrates model-free learning methods can tackle tasks of this complexity . ( 2 ) We show that with assembling new experience from two failures , the sample complexity can be reduced dramatically . We attribute this to global knowledge learning in a set of failed experience which breaks the constraint of local one-episode experience towards more robust strategies . ( 3 ) We design and implement a set of new environments and continuous tasks with dynamic goals , which would be of interest to researchers at the intersection of robotics and reinforcement learning.1 1Our code and environments are available at https : //github.com/mengf1/DHER . 2 RELATED WORK . Recent works in deep RL have shown impressive results in different domains , such as games ( Bellemare et al. , 2013 ; Silver et al. , 2016 ) , simulated control tasks ( Brockman et al. , 2016 ) and so on . There have been several proposed RL methods for playing Atari games , including DQN ( Mnih et al. , 2015 ) , UVFA ( Schaul et al. , 2015b ) and so on . UVFA trains a single neural network approximating multiple value functions for state and goal . For the continuous control , DDPG ( Lillicrap et al. , 2015 ) is a popular actor-critic algorithm that has shown impressive results in continuous control tasks . Dynamic goals appear in games and planning , often addressed by rewards shaping or search ( Kaelbling , 1993 ; Mnih et al. , 2015 ; Di Rocco et al. , 2013 ) . When the rewards are sparse , there is few work studying the dynamic goals to the best of our knowledge . Curriculum learning is also used for reinforcement learning scenarios . The idea is that solving easier problems first has advantages to learn more complex goals later and thus that learning can be optimized by presenting the problems in an optimal order , a curriculum ( Bengio et al. , 2009 ) . Narvekar et al . ( 2017 ) and Florensa et al . ( 2018 ) proposed methods to automatically produce subtasks or subgoals for a given target . Narvekar et al . ( 2017 ) produce subtasks according to predefined tasks of a given domain problem . Florensa et al . ( 2018 ) use a Generative Adversarial Network ( GAN ) to produce goals with different difficulties . Different from these methods , our approach produces a series of goals for an episode from failed experience . Experience replay is an important technique and introduced to break temporal correlations by mixing more and less recent experience for updating policies ( Lin , 1992 ) . It was demonstrated for its efficiency in DQN ( Mnih et al. , 2015 ) . Prioritized experience replay improves the speed of training by considering prioritizing transitions in the replay buffer ( Schaul et al. , 2015b ) . HER considers modifying experience in the replay buffer for continuous control ( Andrychowicz et al. , 2017 ) . By contrary , our approach assembles successful experience from two failures . Comparing with these methods , which do not consider dynamic goals , our approach uses a series of goals to assemble successful experience . As a result , our method is able to accomplish the tasks with sparse rewards and dynamic goals . 3 METHODOLOGY . We first review how HER works ( Andrychowicz et al. , 2017 ) , followed by details of the proposed DHER for dealing with dynamic goals . HER is a simple and effective method of manipulating the replay buffer used in off-policy RL algorithms that allows it to learn policies more efficiently from sparse rewards . It assumes the goal being pursued does not influence the environment dynamics . After experiencing an episode { s0 , s1 , · · · , sT } , every transition st → st+1 along with the goal for this episode is usually stored in the replay buffer . Some of the saved episodes fail to reach the goal , providing no positive feedback to the agent . However , with HER , the failed experience is modified and also stored in the replay buffer in the following manner . The idea is to replace the original goal with a state visited by the failed episode . As the reward function remains unchanged , this change of goals hints the agent how to achieve the new goal in the environment . HER assumes that the mechanism of reaching the new goal helps the agent learn for the original goal . 3.1 DYNAMIC GOALS . Dynamic goals are not static and change at every timestep . We follow the multi-goal setting of Andrychowicz et al . ( 2017 ) . The goals are part of the environment and do not influence the environment dynamics . We also assume that a dynamic goal gt ∈ G moves by following some law gt = g ( t ; γ ) , where γ parameterizes the law ( e.g. , acceleration in Newton ’ s law of motion ) , and yet its underlying moving law is unknown to the agent . Moreover , we need to have some basic knowledge of goals , i.e. , the measure of goal similarity on G. We assume that gt ∈ G corresponds to some predicate fgt : S → { 0 , 1 } and that the agent ’ s goal is to achieve any state s that satisfies fgt ( s ) = 1 . We use S = G and define fgt ( s ) : = [ s = gt ] , which can be considered as a measure of goal similarity between gt and s. The goals can also specify only some properties of the state . Take manipulation tasks for instance : G = R3 corresponds to the 3D Algorithm 1 Dynamic Hindsight Experience Replay with Experience Assembling Require : an off-policy RL algorithm A , replay buffer R , a reward function r 1 : Initialize A and replay buffer R 2 : for episode = 1 , 2 , · · · , M do 3 : Sample an initial goal g0 and an initial state s0 4 : for t = 0 , · · · , T − 1 do 5 : Sample an action at using the behavioral policy from A : 6 : at ← π ( st|gt ) 7 : Execute the action at and observe a new state st+1 and a new goal gt+1 8 : end for 9 : for t = 0 , · · · , T − 1 do 10 : rt : = r ( st , at , gt+1 ) 11 : Store the transitions ( st|gt , at , rt , st+1|gt+1 ) in R ( Standard experience replay ) 12 : end for 13 : Collect failed episodes to E 14 : for Ei ∈ E do 15 : Search another Ej ( i 6= j ) ∈ E where gaci , p = gdej , q 16 : if Ej 6= Ø then 17 : Clone a goal trajectory { g′0 , · · · , g′m } m=min { p , q } in which g′t = gdej , q−m+t from Ej 18 : for t = { 0 , · · · , m− 1 } do 19 : r′t : = r ( si , p−m+t , ai , p−m+t , g ′ t+1 ) 20 : Store the transition ( si , p−m+t|g′t , ai , p−m+t , r′t , si , p−m+t+1|g′t+1 ) in R ( DHER ) 21 : end for 22 : end if 23 : end for 24 : for t = 1 , · · · , N do 25 : Sample a minibatch B from the replay buffer R 26 : Optimize A using the minibatch B 27 : end for 28 : end for positions of an object sobj about which the observation could include additional properties of the object . For a more concrete example , consider pushing a block towards a moving target position . The success of a task is defined as f ( st , gt ) = 1condition ( ‖sobjt − gt‖ ≤ ) , where s obj t is the position of the object in the state st and denotes a tolerance by the environment . 1condition is an indicator function . The agent aims to achieve any state st that satisfies f ( st , gt ) = 1 . It receives a sparse reward rt : = r ( st , at , gt+1 ) = −1condition ( f ( st+1 , gt+1 ) 6= 1 ) upon making an action at . It is worth discussing the main difference between the implications of the dynamic goals and the static ones . Expressing a static goal in the following way , gstatict = g static , ∀t , highlights the key challenge of dealing with the dynamic goal . Namely , the agent has no access to the underlying law of the dynamic goal in our setting , whereas the law of being static is known to the agent in Andrychowicz et al . ( 2017 ) . In other words , the agent has no clue at all how to construct a new dynamic goal that is admissible by the environment .

Summary: This paper proposes a way of extending Hindsight Experience Replay (HER) to dynamic or moving goals. The proposed method (DHER) constructs new successful trajectories from pairs of failed trajectories where the goal accomplished at some point in the first trajectory happens to match the desired goal in the second trajectory. The method is demonstrated to work well in several simulated environments and some qualitative sim2real transfer results to a real robot are also provided.