Title: DHER: HINDSIGHT EXPERIENCE REPLAY FOR DYNAMIC GOALS

Abstract: Dealing with sparse rewards is one of the most important challenges in reinforcement learning (RL), especially when a goal is dynamic (e.g., to grasp a moving object). Hindsight experience replay (HER) has been shown an effective solution to handling sparse rewards with fixed goals. However, it does not account for dynamic goals in its vanilla form and, as a result, even degrades the performance of existing off-policy RL algorithms when the goal is changing over time.
In this paper, we present Dynamic Hindsight Experience Replay (DHER), a novel approach for tasks with dynamic goals in the presence of sparse rewards. DHER automatically assembles successful experiences from two relevant failures and can be used to enhance an arbitrary off-policy RL algorithm when the tasks' goals are dynamic. We evaluate DHER on tasks of robotic manipulation and moving object tracking, and transfer the polices from simulation to physical robots. Extensive comparison and ablation studies demonstrate the superiority of our approach, showing that DHER is a crucial ingredient to enable RL to solve tasks with dynamic goals in manipulation and grid world domains.

1. INTRODUCTION:

Deep reinforcement learning has been shown an effective framework for solving a rich repertoire of complex control problems. In simulated domains, agents have been trained to perform a diverse array of challenging tasks (Mnih et al., 2015; Lillicrap et al., 2015; Duan et al., 2016) . In order to train such agents, it is often the case that one has to design a reward function that not only reflects the task at hand but also is carefully shaped (Ng et al., 1999) to guide the policy optimization. Unfortunately, many of the capabilities demonstrated by reward engineering are often limited to specific tasks. Moreover, it requires both RL expertise and domain-specific knowledge to reshape the reward functions. For situations where we do not know what admissible behavior may look like, for example, using LEGO bricks to build a desired architecture, it is difficult to apply reward engineering. Therefore, it is essential to develop algorithms which can learn from unshaped and usually sparse reward signals.

Learning with sparse rewards is challenging, especially when a goal is dynamic. Dynamic goals are common in games and planning problems, often addressed using reward shaping or search (Kaelbling, 1993; Mnih et al., 2015; Di Rocco et al., 2013) . However, the difficulty posed by a sparse reward is exacerbated by the complicated environment dynamics in robotics (Andrychowicz et al., 2017) . For instance, system dynamics around contacts are difficult to model and induce sensitivity in the system to small errors. Many robotic tasks also need executing multiple steps successfully over a long horizon, involve enormous search space, and require generalization to varying task instances. Policy gradient methods are breakthroughs in the challenging environments, such as PPO (Heess et al., 2017; Schulman et al., 2017) , ACER (Wang et al., 2016) , TRPO (Schulman et al., 2015) and so on. They are used in environments, where an agent tries to reach a target, learns to walk, runs, and so on. Recently, sampling-efficient learning is introduced and demonstrates a significant increase in performance for off-policy actor-critic DQN (Mnih et al., 2015) and DDPG (Lillicrap et al., 2015) algorithms. Hindsight experience replay (HER) is very effective for improving the performance of off-policy RL algorithms in solving goal-based tasks with sparse rewards (Andrychowicz et al., 2017) . Similar to UVFA (Schaul et al., 2015a) , it takes a goal state as part of input. However, it assumes the goal is fixed. As a result, this assumption actually impedes the learning of RL agents in the environments of moving goals.

In this paper, we address this challenge with a new method, Dynamic Hindsight Experience Replay (DHER), for accomplishing tasks with moving goals. We follow the multi-goal setting in UVFA (Schaul et al., 2015a) and HER (Andrychowicz et al., 2017) . It assumes that the goal being pursued does not influence the environment dynamics. We also need to have the knowledge of goal similarity. For example, in manipulation or grid world domains, we can use Euclidean distance between positions to measure the goal similarity. HER turns a failed episode to a success by composing a new task whose goal is achieved by that episode. Our idea allows an agent to learn from the failure one step further than HER: the agent not only sets a new goal but also hallucinates how to reach the original goal from the new one. Take playing frisbee for instance. When an agent jumps to catch the frisbee and yet misses it, the agent receives no positive feedback under the sparse reward setting. Using HER, the agent could set the end of its episode as the new goal -position of the frisbee; with DHER, however, the agent finds a trajectory from its past experiences as the imagined path of the frisbee, and thereby extrapolates towards the original goal.

In particular, we do the following for DHER. To finish the tasks with dynamic goals needs to explore experience and understand multiple goals. DHER uses replay buffers to allow the agent to learn from a couple of failures by assembling new 'experience' from different episodes. The proposed method retrieves memories to find the connection between the experience of different episodes. It largely improves the sample efficiency in dynamic goal task settings. More importantly, this strategy makes it possible to learn in the setting that both sparse rewards and dynamic goals exist.

We evaluate our method along with the state-of-the-art baselines on new environments and manipulation tasks, which have sparse rewards and dynamic goals. Our results demonstrate that DHER is clearly better than others for these tasks. We also transfer policies trained in our simulation based on DHER to a physical robot and show that DHER can be applied to solving real-world robotics problems.

We summarize our main contributions as follows: (1) We demonstrate that, both in simulation and real worlds, DHER succeeds in continuous control with a moving target. To our knowledge, this is the first empirical result on manipulation tasks that demonstrates model-free learning methods can tackle tasks of this complexity.

(2) We show that with assembling new experience from two failures, the sample complexity can be reduced dramatically. We attribute this to global knowledge learning in a set of failed experience which breaks the constraint of local one-episode experience towards more robust strategies. (3) We design and implement a set of new environments and continuous tasks with dynamic goals, which would be of interest to researchers at the intersection of robotics and reinforcement learning.1 

2. RELATED WORK:

Recent works in deep RL have shown impressive results in different domains, such as games (Bellemare et al., 2013; Silver et al., 2016) , simulated control tasks (Brockman et al., 2016 ) and so on.

There have been several proposed RL methods for playing Atari games, including DQN (Mnih et al., 2015) , UVFA (Schaul et al., 2015b ) and so on. UVFA trains a single neural network approximating multiple value functions for state and goal. For the continuous control, DDPG (Lillicrap et al., 2015) is a popular actor-critic algorithm that has shown impressive results in continuous control tasks. Dynamic goals appear in games and planning, often addressed by rewards shaping or search (Kaelbling, 1993; Mnih et al., 2015; Di Rocco et al., 2013) . When the rewards are sparse, there is few work studying the dynamic goals to the best of our knowledge.

Curriculum learning is also used for reinforcement learning scenarios. The idea is that solving easier problems first has advantages to learn more complex goals later and thus that learning can be optimized by presenting the problems in an optimal order, a curriculum (Bengio et al., 2009) . Narvekar et al. (2017) and Florensa et al. (2018) proposed methods to automatically produce subtasks or subgoals for a given target. Narvekar et al. (2017) produce subtasks according to predefined tasks of a given domain problem. Florensa et al. (2018) use a Generative Adversarial Network (GAN) to produce goals with different difficulties. Different from these methods, our approach produces a series of goals for an episode from failed experience.

Experience replay is an important technique and introduced to break temporal correlations by mixing more and less recent experience for updating policies (Lin, 1992) . It was demonstrated for its efficiency in DQN (Mnih et al., 2015) . Prioritized experience replay improves the speed of training by considering prioritizing transitions in the replay buffer (Schaul et al., 2015b) . HER considers modifying experience in the replay buffer for continuous control (Andrychowicz et al., 2017) . By contrary, our approach assembles successful experience from two failures. Comparing with these methods, which do not consider dynamic goals, our approach uses a series of goals to assemble successful experience. As a result, our method is able to accomplish the tasks with sparse rewards and dynamic goals.

3. METHODOLOGY:

We first review how HER works (Andrychowicz et al., 2017) , followed by details of the proposed DHER for dealing with dynamic goals.

HER is a simple and effective method of manipulating the replay buffer used in off-policy RL algorithms that allows it to learn policies more efficiently from sparse rewards. It assumes the goal being pursued does not influence the environment dynamics. After experiencing an episode {s 0 , s 1 , • • • , s T }, every transition s t → s t+1 along with the goal for this episode is usually stored in the replay buffer. Some of the saved episodes fail to reach the goal, providing no positive feedback to the agent. However, with HER, the failed experience is modified and also stored in the replay buffer in the following manner. The idea is to replace the original goal with a state visited by the failed episode. As the reward function remains unchanged, this change of goals hints the agent how to achieve the new goal in the environment. HER assumes that the mechanism of reaching the new goal helps the agent learn for the original goal.

3.1. DYNAMIC GOALS:

Dynamic goals are not static and change at every timestep. We follow the multi-goal setting of Andrychowicz et al. (2017) . The goals are part of the environment and do not influence the environment dynamics. We also assume that a dynamic goal g t ∈ G moves by following some law g t = g(t; γ), where γ parameterizes the law (e.g., acceleration in Newton's law of motion), and yet its underlying moving law is unknown to the agent.

Moreover, we need to have some basic knowledge of goals, i.e., the measure of goal similarity on G.

We assume that g t ∈ G corresponds to some predicate f gt : S → {0, 1} and that the agent's goal is to achieve any state s that satisfies f gt (s) = 1. We use S = G and define f gt (s) := [s = g t ], which can be considered as a measure of goal similarity between g t and s. The goals can also specify only some properties of the state. Take manipulation tasks for instance: G = R 3 corresponds to the 3D Sample an initial goal g 0 and an initial state s 0 4:

for t = 0, • • • , T -1 do 5:

Sample an action a t using the behavioral policy from A:

6:

a t ← π(s t |g t ) 7:

Execute the action a t and observe a new state s t+1 and a new goal g t+1 8:

end for 9: Search another E j (i = j) ∈ E where g ac i,p = g de j,q 16:

for t = 0, • • • , T -1 do 10: r t := r(s t ,

if E j = Ø then 17:

Clone a goal trajectory {g 0 , • • • , g m } m=min{p,q} in which g t = g de j,q-m+t from E j 18: 

for t = {0, • • • , m -1}

for t = 1, • • • , N do 25:

Sample a minibatch B from the replay buffer R 26:

Optimize A using the minibatch B 27:

end for 28: end for positions of an object s obj about which the observation could include additional properties of the object. For a more concrete example, consider pushing a block towards a moving target position. The success of a task is defined as f (s t , g t ) = 1 condition ( s obj t -g t ≤ ), where s obj t is the position of the object in the state s t and denotes a tolerance by the environment. 1 condition is an indicator function. The agent aims to achieve any state s t that satisfies f (s t , g t ) = 1. It receives a sparse reward r t := r(s t , a t , g t+1 ) = -1 condition (f (s t+1 , g t+1 ) = 1) upon making an action a t .

It is worth discussing the main difference between the implications of the dynamic goals and the static ones. Expressing a static goal in the following way, g static t = g static , ∀t, highlights the key challenge of dealing with the dynamic goal. Namely, the agent has no access to the underlying law of the dynamic goal in our setting, whereas the law of being static is known to the agent in Andrychowicz et al. (2017) . In other words, the agent has no clue at all how to construct a new dynamic goal that is admissible by the environment.

3.2. DYNAMIC HINDSIGHT EXPERIENCE REPLAY:

At the first glance, we shall compose a new dynamic goal g dynamic = {s obj t0 , s obj t1 , • • • , s obj t T } from a failed episode s 0 , s 1 , • • • , s T in order to apply HER (Andrychowicz et al., 2017) to our problem setting. However, per the discussion above, this new dynamic goal g dynamic may be inadmissible by the environment, leading to no positive feedback to the agent at all. We tackle the challenge by drawing the following two observations. One is that many episodes fail in the replay buffer, implying that the agent can actually build a new goal upon more than one episodes. The other is that the more failed experience the agent has, the more possible for the agent to use the connection between achieved goals in an episode and desired goals in some other episode. Take the example of a moving object that the agent must reach, desired goals are the positions of the moving object and achieved goals are the positions of a gripper (controlled by the agent). There may exist some positions that both the object and the gripper have ever reached respectively.

After experiencing some episode s 0 , s 1 , • • • , s T , we store in the replay buffer every transition s t → s t+1 for this episode, defined as (s t , a t , r t , s t+1 ), where s t indicates a state s t at timestep t, and a t indicates an action and r t indicates a reward. Thus before t, there are a series of records {(s 0 , a 0 , r 0 , s 1 ) • • • , (s t , a t , r t , s t+1 )}. A state consists of three parts: observation o t , desired goal g de t and achieved goal g ac t , define as s t = o t , g ac t , g de t

, where normally g de t = g t and g ac t indicates goals that the agent has achieved.

We reuse the failed experience from the replay buffer with inverse simulation to create successful rewards for the agent, as shown in Algorithm 1 (lines 13-23). Our inverse simulation contains two main steps: First, given a failed episode, for its achieved goal trajectory, we try to find a desired goal trajectory from other episodes that could match it (line 15). Second, we assemble a new episode by matching the achieved goal trajectory of the given episode to the desired goal trajectory of the founded one (lines 17-21). Let g ac i,q indicate the achieved goal of the agent at timestep q in episode i and g de j,p indicate the desired goal at timestep p in episode j. Given a set of failed experience {E 0 , E 1 , E 2 , • • • }, we search and draw two failed episodes E i and E j (i = j), where ∃i, j, p, q, s.t. g ac i,p = g de j,q . If we find two such failed episodes, we combine the two experience by replacing the desired goals in E i by {g de j,t }, where j indicates E j and t ≤ min{p, q}. Based on this, we end up assembling a new experience E i based on E i with a new "imagined" goal trajectory {g de j,0 , • • • , g de j,t } where t ≤ min{p, q}. More details of the complete RL+DHER method are shown in Algorithm 1. Unlike HER, our DHER needs to search all failed experiences to compose a "imagined" goal trajectory. Hence, the efficiency of searching the memory is important. In our implementation, we use two hash tables to store the trajectories of achieved goals and desired goals, respectively.

4. EXPERIMENT:

We run extensive experiments to examine the proposed DHER for moving goals and compare it with some competing baselines. We first introduce the environments and tasks that we want to address, followed by the experimental results. Demo videos from our experiments are available at https://sites.google.com/view/dher. Figure 3 : Results on different environments.

4.1. ENVIRONMENTS:

Whereas grasping moving targets are fairly common in robotic applications, there are rarely existing environments featuring dynamic goals. We modify the robotic manipulation environments created by OpenAI (Brockman et al., 2016) for our experiments. As shown in Figure 2 , we assign certain rules to the goals so that they accordingly move in the environments while an agent is required to control the robotic arm's grippers to reach the goal that moves along a straight line (Dy-Reaching), to reach the goal that moves in a circle (Dy-Circling), or to push a block to the goal that moves along a straight line (Dy-Pushing). In addition, we also develop a new GREEDY SNAKE environment (Dy-Snake), in which the goal moves from one discrete cell to another. The greedy snake aims to reach the goal (and eat it). No matter what actions are taken by the agent, the underlying rule of changing the goals' positions remain the same. We assume that the law of the goal's motion is unknown to the agent. More details of the tasks are described in the appendix.

For the first three tasks, we follow the basic settings of OpenAI robotics environments (Brockman et al., 2016) . States are read from the MuJoCo physics engine. An observation consists of relative positions of the object and the target (grippers are blocked) Goals are positions in the 3D world coordinate system with a fixed tolerance (we use = 0.01 for the tolerance). Note the goals are able to move in our tasks. The velocity we use is v = 0.011. 1 epoch indicates 100 episodes. The start positions of the goals are randomly chosen. Rewards are binary and sparse: r(s t , a t , g t ) = -1 condition (|s obj t+1 -g t+1 | ≥ ) where s t+1 and g t+1 are respectively the state and dynamic goal after the execution of the action a t in the state s t . Two positions are overlapped if they are close within the tolerance . We use 3-dimensional actions, in which three dimensions correspond to the desired relative gripper position at the next timestep.

For the last task, it is very similar to the standard snake game. There is a 2D plane grid whose size is 30 × 40. A snake and a goal (food) both move in this grid. We use a 1 × 1 square as the body of the snake and do not allow the snake to grow any longer as it moves. The states of the system are represented by using the positions of the snake and food. Goals are the positions of the food. The snake has to move itself such that it resides in the same cell as the food at a certain timestep. The velocity of the goal is set to (0, 1). The start positions of the goal are set randomly in different episodes of the game, so are the start positions of the snake. Rewards are binary and sparse: r(s t , a t , g t+1 ) = -1 condition (|s snake t+1 -g t+1 | = 0) where s t+1 and g t+1 are the environment's state and goal after the agent executes action a t in the state s t . Observations are represented by the positions of the snake and food. We also add to the state the distance between the snake and food. Actions allowed in the game are the following: move up, move down, move left, and move right, for one cell per timestep.

4.2. BASELINES:

For the first three tasks which call for continuous control, we consider two competing baselines:2 • DDPG, which is a model-free RL algorithm for continuous control (Lillicrap et al., 2015) .

It learns a deterministic policy by using a stochastic counterpart to explore in the training.

• DDPG + HER, which improves the replay buffer of DDPG by the hindsight experience replay (Andrychowicz et al., 2017) .

• DDPG (dense), which employs the negative distance (-d) as dense rewards.

• DDPG (dense-2), which employs the negative distance (-d) as dense rewards if d ≥ . However, if d < (i.e., success) it uses (-d + 1.0) as rewards. 1.0 is a bonus.

For the last task of discrete control, we use the following baselines in the experiments: DQN (Mnih et al., 2015) and DQN + HER, which uses HER to enhance the replay in DQN.

• DQN, which is a powerful model-free RL algorithm for discrete action spaces (Mnih et al., 2015) .

• DQN + HER, which uses HER to enhance the replay in DQN.

• DQN (dense), which uses the negative distance (-d) as dense rewards.

• DQN (dense-2), which uses the negative distance (-d) as dense rewards. However, if d = 0.0 (i.e., success) it uses (-d + 1.0) as rewards instead. 1.0 is a bonus.

4.3. COMPARISON RESULTS ON THE ROBOTIC ENVIRONMENTS:

For the continuous control, we present three sets of comparison results in Figure 3 for the first three tasks, respectively. Consistently, the results show our DHER algorithm outperforms the others. The two baselines are not able to catch up even after we train them for thousands of iterations. Vanilla DDPG is slightly better than the version with HER. HER does not benefit DDPG in these tasks because the goals in HER are fixed, fundamentally misleading the agent in the attempt of solving the tasks with dynamic goals. The results of DDPG (dense) and DDPG (dense-2) suggest that even the dense rewards do not work well as they are agnostic to the task of interest. A good reward shaping may give rise to better performance by carefully tuning it for the task of dynamic goals.

Comparing Figure 3a and Figure 3b with Figure 3c , we find that DHER learns faster in Dy-Reaching and Dy-Circling than Dy-Pushing probably because Dy-Reaching and Dy-Circling are easier tasks than Dy-Pushing. In Dy-Pushing, all the algorithms take a fairly big amount of time to explore without receiving any positive feedback. However, the more failed experiences the agent encounters, the better change our algorithm is able to identify relevant episodes from them for assembling useful dynamic goals. As a result, DHER is able to pick up the momentum and learns faster and better than the baselines after a certain point. In Figure 3a , the performance decreases a little. The reason may be that as successful experience increases, some assembled experience is inconsistent with these successful experience.

4.3.1. COMPARISON USING DIFFERENT VELOCITIES OF GOALS:

To show the performance of our method on more complex tasks with different velocities, we study different methods in Dy-Reaching environment as shown in properties as the previous experiments. Overall the results show our method is much better than DDPG and HER. As a reminder, the threshold for calculating rewards we used is 0.01. In Figure 4 , the task becomes an easier task because of the slower velocity. It shows that our method quickly achieves to a good result around 5 epoch. Comparing with Figure 3a , it shows that the performance with v = 0.001 is better than the performance with v = 0.011 and get 15% improvements. However, it also shows the performance with v = 0.016 is worse than the performance with v = 0.011. Both DDPG and DDPG+HER failed when v = 0.016 and their performance is 0. This performance is consistent because the task becomes more difficult when the velocity increases.

4.4. COMPARISON ON THE DYNAMIC SNAKE ENVIRONMENT:

In the Dy-Snake environment, which is a discrete control environment, we present the results of the chasing food task in Figure 6 . The results show that the proposed algorithm works best. DQN is better than HER. HER fails for this task. That HER fails for this task shows just using achieved goals is not enough for the tasks with dynamic goals and can lead wrong direction. The results also show that around 800 episodes, the performance of DQN and DHER is close and DHER is slightly better than DQN. This is because the chasing food task is a simple task. The action space is very small and just 4 types of actions. After enough exploration, DQN also has competitive performance. However, at the beginning, DHER quickly achieves very good performance. It shows that assembling experience from two failures improves the performance very efficiently in this task. DQN (dense) and DQN (dense-2) help learn the policy at the early stage. However, in the long run, it does not lead to any particular benefits.

4.5. SIM TO REAL ADAPTATION:

We used policies for Dy-Circling task and a new Pouring task trained in our simulator3 to deploy them on a physical robot. As shown in Figure 8 , the policies were trained by using DHER and adapted to the real robot without any finetuning. However, the policy requires accurate localizations of the gripper and the goal. For Dy-Circling task, the robot's gripper was blocked. There were a toy turntable, whose speed is unknown, and a blue block on the turntable. We set the position 1cm above the block as the target position. The position of the block was predicted based on traditional contour shape analysis using camera images. For Pouring task, the robot gripped a can. A man held a cup and moved it. The cup was set as the target and with a green marker. We used the marker to estimate its position.

Our policies were transferred successfully for both tasks. With the accurate positions, we have 100% success rate for 5 trials. It was observed that the robot had learned to not only follow the current target but also step forward to the future target position. Demo videos about the experiments are available at https://sites.google.com/view/dher. We use Dy-Snake to demonstrate the experimental results for a special case that the law of the motion of the target (food) is known to agents. Because the motion of the food is very simple and controllable in Dy-Snake environment. We develop a direct extension of HER, called HER+, that modifies desired goals at every timestep based on the law of the motion of the food to create successful experience. More details of HER+ are described in the appendix.

We show the results in Figure 7 . DHER and HER+ are both better than DQN at the beginning. DHER is slightly better than HER+, which shows the efficiency of DHER is comparable in this simple task.

5. CONCLUSION:

We introduced a novel technique that assembles successful experience from a couple of failures. With this technique, our proposed algorithm called DHER ( Dynamic Hindsight Experience Replay) is able to address the tasks with sparse rewards and dynamic goals. Our technique can be combined with an arbitrary off-policy RL algorithm and we experimentally demonstrated that with DQN and DDPG. As far as we know, it is the first time that an agent is allowed to learn from assembled experience from two failures. is easy to obtain, i.e., the parameters of g(t; γ) are known. For example, it is able to calculate the velocity of an object by observing the trajectories of the object.

In this situation, we do not need to search failed experience. With the knowledge of the velocity of an object, it is straightforward that we can calculate any trajectory of desired goals. Thus for every failed experience, at any time step t, based on the achieved goal g ac t , we calculate new desired goals g de t = g(t) corresponding to the achieved goals in order to construct successful experience.

The details of HER+ are described in Algorithm 2. HER+ can be seen as a direct extension of HER. HER and HER+ can modify any failed experience to successful experience because they both assume the law of the motion of goals is known.

Algorithm 2 Hindsight Experience Replay Plus:

Require: an off-policy RL algorithm A, replay buffer R, a reward function r 1: Initialize A and replay buffer R 2: for episode = 1, 2, • • • , M do 3:

Sample an initial goal g 0 and an initial state s 0 4:

for t = 0, • • • , T -1 do 5:

Sample an action a t using the behavioral policy from A: 

:

Our code and environments are available at https://github.com/mengf1/DHER.

https://github.com/openai/baselines

We developed a new simulator according to our hardware. More details on our hardware setup are available in the appendix.

Figures and Tables:

Figure 1: The framework of DHER. DHER is a kind of experience replay method. It searches relevant failed experiences and then assembles them into successful experiences.

Figure 2: The proposed tasks with dynamic goals (red objects). Arrow indicates the movement of a goal. The first row indicates initial states. The second row indicates final states.

Figure 4: Low velocity: 0.001.

Figure 6: Snake with dynamic goals.

Figure 8: Adapting the policies trained based on DHER from our simulation to a real robotic arm.

Algorithm 1 Dynamic Hindsight Experience Replay with Experience AssemblingRequire: an off-policy RL algorithm A, replay buffer R, a reward function r 1: Initialize A and replay buffer R

Table content: <table/>

a t , g t+1 ) Store the transitions (s t |g t , a t , r t , s t+1 |g t+1 ) in R (Standard experience replay)

Table content: <table><tr><td>11:</td><td/></tr><tr><td>12:</td><td>end for</td></tr><tr><td>13:</td><td>Collect failed episodes to E</td></tr><tr><td>14:</td><td>for E i ∈ E do</td></tr><tr><td>15:</td><td/></tr></table>

do

Table content: <table><tr><td colspan="2">19: r 21: end for</td></tr><tr><td>22:</td><td>end if</td></tr><tr><td>23:</td><td>end for</td></tr><tr><td>24:</td><td/></tr></table>

Execute the action a t and observe a new state s t+1 and a new goal g t+1 Store the transition (s i,t |g t , a i,t , r t , s i,t+1 |g t+1 ) in R

Table content: <table><tr><td>8:</td><td>end for</td></tr><tr><td>9:</td><td/></tr><tr><td>18:</td><td>end for</td></tr><tr><td>19:</td><td>end for</td></tr><tr><td>20:</td><td>end for</td></tr><tr><td colspan="2">21: end for</td></tr></table>

Peer review (iclr2019):
 In this paper, the authors extend the HER framework to deal with dynamical goals, i.e. goals that change over time. In order to do so, they first need to learn a model of the dynamics of the goal, and then to select in the replay buffer experience reaching the expected value of the goal at the expected time. Empirical results are based on three (or four, see the appendix) experiments with a Mujoco UR10 simulated environment, and one experiment is successfully transfered to a real robot. Overall, the addressed problem is relevant (the question being how can you efficiently replay experience when the goal is dynamical?), the idea is original and the approach looks sound, but seems to suffer from a fundamental flaw (see below). Despite some merits, the paper mainly suffers from the fact that the implementation of the approach described above is not explained clearly at all. Among other things, after reading the paper twice, it is still unclear to me: - how the agent learns of the goal motion (what substrate for such learning, what architecture, how many repetitions of the goal trajectory, how accurate is the learned model...) - how the output of this model is taken as input to infer the desired values of the goal in the future: shall the agent address the goal at the next time step or later in time, how does it search in practice in its replay buffer, etc. These unclarities are partly due to unsufficient structuring of the "methodology" section of the paper, but also to unsufficient mastery of scientific english. At many points it is not easy to get what the authors mean, and the paper would definitely benefit from the help of an experienced scientific writer. Note that Figure 1 helps getting the overall idea, but another Figure showing an architecture diagram with the main model variables would help further. In Figures 3a and 5, we can see that performance decreases. The explanation of the authors just before 4.3.1 seem to imply that there is a fundamental flaw in the algorithm, as this may happen with any other experiment. This is an important weakness of the approach. To me, Section 4.5 about transfer to a real robot does not bring much, as the authors did nothing specific to favor this transfer. They just tried and it happens that it works, but I would like to see a discussion why it works, or that the authors show me with an ablation study that if they change something in their approach, it does not work any more. In Section 4.6, the fact that DHER can outperform HER+ is weird: how can a learn model do better that a model given by hand, unless that given model is wrong? This needs further investigation and discussion. In more details, a few further remarks: In related work, twice: you should not replace an accurate enumeration of papers with "and so on". p3: In contrary, => By contrast, which is the same to => same as compare the above with the static goals => please rephrase In Algorithm 1, line 26: this is not the algorithm A that you optimize, this is its critic network. line 15: you search for a trajectory that matches the desired goal. Do you take the first that matches? Do you take all that match, and select the "best" one? If yes, what is the criterion for being the best? p5: we find such two failed => two such failed that borrows from the Ej => please rephrase we assign certain rules to the goals so that they accordingly move => very unclear. What rules? Specified how? Please give a formal description. For defining the reward, you use s_{t+1} and g_{t+1}, why not s_t and g_t? p6: the same cell as the food at a certain time step. Which time step? How do you choose? The caption of Fig. 6 needs to be improved to be contratsed with Fig. 7. p8: the performance of DQN and DHER is closed => close? DHER quickly acheive(s) Because the law...environment. => This is not a sentence. Mentioning in the appendix a further experiment (dy-sliding) which is not described in the paper is of little use.