Title: LEARNING GOAL-CONDITIONED VALUE FUNCTIONS WITH ONE-STEP PATH REWARDS RATHER THAN GOAL-REWARDS Anonymous

Abstract: Multi-goal reinforcement learning (MGRL) addresses tasks where the desired goal state can change for every trial. State-of-the-art algorithms model these problems such that the reward formulation depends on the goals, to associate them with high reward. This dependence introduces additional goal reward resampling steps in algorithms like Hindsight Experience Replay (HER) that reuse trials in which the agent fails to reach the goal by recomputing rewards as if reached states were psuedo-desired goals. We propose a reformulation of goal-conditioned value functions for MGRL that yields a similar algorithm, while removing the dependence of reward functions on the goal. Our formulation thus obviates the requirement of reward-recomputation that is needed by HER and its extensions. We also extend a closely related algorithm, Floyd-Warshall Reinforcement Learning, from tabular domains to deep neural networks for use as a baseline. Our results are competitive with HER while substantially improving sampling efficiency in terms of reward computation.

1. INTRODUCTION:

Many tasks in robotics require the specification of a goal for every trial. For example, a robotic arm can be tasked to move an object to an arbitrary goal position on a table (Gu et al., 2017) ; a mobile robot can be tasked to navigate to an arbitrary goal landmark on a map (Zhu et al., 2017) . The adaptation of reinforcement learning to such goal-conditioned tasks where goal locations can change is called Multi-Goal Reinforcement Learning (MGRL) (Plappert et al., 2018) . State-of-theart MGRL algorithms (Andrychowicz et al., 2017; Pong et al., 2018) work by estimating goalconditioned value functions (GCVF) which are defined as expected cumulative rewards from start states with specified goals. GCVFs, in turn, are used to compute policies that determine the actions to take at every state.

To learn GCVFs, MGRL algorithms use goal-reward, defined as the relatively higher reward recieved on reaching the desired goal state. This makes the reward function dependent on the desired goal. For example, in the Fetch-Push task (Plappert et al., 2018) of moving a block to a given location on a table, every movement incurs a "-1" reward while reaching the desired goal returns a "0" goal-reward. This dependence introduces additional reward resampling steps in algorithms like Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) , where trials in which the agent failed to reach the goal are reused by recomputing rewards as if the reached states were pseudodesired goals. Due to the dependence of the reward function on the goal, the relabelling of every pseudo-goal requires an independent reward-recomputation step, which can be expensive.

In this paper, we demonstrate that goal-rewards are not needed to learn GCVFs. For the Fetch-Push example, the "0" goal-reward does not need to be achieved to learn its GCVF. Specifically, the agent continues to receive "-1" reward even when the block is in the given goal location. This reward formulation is atypical in conventional RL because high reward is used to specify the desired goal location. However, this goal-reward is not necessary in goal-conditioned RL because the goal is already specified at the start of every episode. We use this idea, to propose a goal-conditioned RL algorithm which learns to reach goals without goal-rewards. This is a counter-intuitive result which is important for understanding goal-conditioned RL. Let us consider another example to motivate the redundancy of goal-rewards. Consider a student who has moved to a new campus. To learn about the campus, the student explores it randomly with no specific goal in mind. The key intuition here is that the student is not incentivized to find specific goal locations (i.e. no goal-rewards) but is aware of the effort required to travel between points around the university. When tasked with finding a goal classroom, the student can chain together these path efforts to find the least-effort path to the classroom. Based on this intuition of least-effort paths, we redefine GCVFs to be the expected path-reward that is learned for all possible start-goal pairs. We introduce a one-step loss that assumes one-step paths to be the paths of maximum reward between pairs wherein the state and goal are adjacent. Under this interpretation, the Bellman equation chooses and chains together one-step paths to find longer maximum reward paths. Experimentally, we show how this simple reinterpretation, which does not use goal rewards, performs as well as HER while outperforming it in terms of reward computation.

We also extend a closely related algorithm, Floyd-Warshall Reinforcement Learning (FWRL) (Kaelbling, 1993) (also called Dynamic Goal Reinforcement learning) to use parametric function approximators instead of tabular functions. Similar to our re-definition of GCVFs, FWRL learns a goal-conditioned Floyd-Warshall function that represents path-rewards instead of future-rewards.

We translate FWRL's compositionality constraints in the space of GCVFs to introduce additional loss terms to the objective. However, these additional loss terms do not show improvement over the baseline. We conjecture that the compositionality constraints are already captured by other loss terms.

In summary, the contributions of this work are twofold. Firstly, we reinterpret goal-conditioned value functions as expected path-rewards and introduce one-step loss, thereby removing the dependency of GCVFs on goal-rewards and reward resampling. We showcase our algorithm's improved sample efficiency (in terms of reward computation). We thus extend algorithms like HER to domains where reward recomputation is expensive or infeasible. Secondly, we extend the tabular Floyd-Warshal Reinforcement Learning to use deep neural networks.

2. RELATED WORK:

Goal-conditioned tasks in reinforcement learning have been approached in two ways, depending upon whether the algorithm explicitly separates state and goal representations. The first approach is to use vanilla reinforcement learning algorithms that do not explicitly make this separation (Mirowski et al., 2016; Dosovitskiy & Koltun, 2016; Gupta et al., 2017; Parisotto & Salakhutdinov, 2017; Mirowski et al., 2018) . These algorithms depend upon neural network architectures to carry the burden of learning the separated representations.

The second approach makes this separation explicit via the use of goal-conditioned value functions (Foster & Dayan, 2002; Sutton et al., 2011) . Universal Value Function Appoximators (Schaul et al., 2015) propose a network architecture and a factorization technique that separately encodes states and goals, taking advantage of correlations in their representations. Temporal Difference Models combine model-free and model-based RL to gain advantages from both realms by defining and learning a horizon-dependent GCVF. All these works require the use of goal-dependent reward functions and define GCVFs as future-rewards instead of path-rewards, contrasting them from our contribution.

Unlike our approach, Andrychowicz et al. (2017) propose Hindsight Experience Replay, a technique for resampling state-goal pairs from failed experiences; which leads to faster learning in the presence of sparse rewards. In addition to depending on goal rewards, HER also requires the repeated recomputation of the reward function. In contrast, we show how removing goal-rewards removes the need for such recomputations. We utilize HER as a baseline in our work.

Kaelbling (1993) also use the structure of the space of GCVFs to learn. This work employs compositionality constraints in the space of these functions to accelerate learning in a tabular domain.

While their definition of GCVFs is similar to ours, the terminal condition is different. We describe this difference in Section 4. We also extend their tabular formulation to deep neural networks and evaluate it against the baselines.

3. BACKGROUND:

A reinforcement learning (RL) problem is formalized as a Markov Decision Process (MDP) (Sutton et al., 1998) . A MDP is defined by a five tuple (S, A, T, R, γ), that governs a sequence of stateaction-reward triples [(s 0 , a 0 , r 0 ), . . . , (s T , a T , r T )]. S is the state space, A is the action space, T (s, a) : S × A → S is the system dynamics, R(s, a) : S × A → R is the reward function and γ is the discount factor. In a typical RL problem the transition function T is not given but is known to be static. In RL, the objective is to find a policy π(s) : S → A that maximizes the expected cumulative reward over time, R t = ∞ k=t γ k-t r k , called the return. The discount factor, γ < 1, forces the return to be finite for infinite horizons. Reinforcement learning is typically formulated in single-goal contexts. More recently there has been interest in multi-goal problems (Andrychowicz et al., 2017; Pong et al., 2018; Plappert et al., 2018) , which is the focus of this work.

3.1. DEEP REINFORCEMENT LEARNING:

A number of reinforcement learning algorithms use parametric function approximators to estimate the return in the form of an action-value function, Q(s, a):

EQUATION

where T is the episode length. When the policy π is optimal, the Q-function satisfies the Bellman equation (Bellman, 1954) .

EQUATION

The Q * () function can be learned using Q-learning algorithm (Watkins & Dayan, 1992) . The optimal policy can be computed from Mnih et al. (2013) formulate a loss function based on the Bellman equation to approximate the optimal Q * -function using a deep neural network, Q m :

Q * greedily, π * (s t ) = arg max a∈A Q * (s t , a). In Deep Q-Networks (DQN),

EQUATION

where y t = R(s t , a t ) + max a γQ tgt (s t+1 , a; θ Qtgt ) , is the target and Q tgt is the target network (Mnih et al., 2015a) . The target network is a slower-changing copy of the main network that stabilizes learning. Mnih et al. (2015a) also employ replay buffers (Lin, 1993) that store transitions from episodes. During training, these transitions are sampled out of order to train the networks in an off-policy manner, avoiding correlation in the samples and thus leading to faster learning.

In this work, we use an extension of DQN to continuous action spaces called deep deterministic policy-gradients (DDPG) (Lillicrap et al., 2015) . DDPG approximates the policy with a policy network π tgt (s; θ π ) that replaces the max operator in y t . The target thus becomes y t = R(s t , a t ) + γQ tgt (s t+1 , π tgt (s t+1 ; θ π ); θ Qtgt ) and the loss function changes accordingly:

EQUATION

3.2. MULTI-GOAL REINFORCEMENT LEARNING:

Multi-Goal Reinforcement Learning (Plappert et al., 2018) focuses on problems where the desired goal state can change for every episode. State-of-the-art MGRL algorithms learn a goal-conditioned value function (GCVF), Q (s, a, g), which is defined similar to the Q-function (5), but with an additional dependence on the desired goal specification g ∈ G :

EQUATION

)

The structure of the goal specification, g ∈ G, can be arbitrary. For example, in a robotic arm experiment, possible goal specifications include the desired position of the end-effector and the desired joint angles of the robot. The states and achieved goals are assumed to be an observable part of the Goal-MDP to enable the agent to learn the correspondences between them, [(s 0 , a 0 , g 0 , r 0 ), . . . , (s T , a T , g T , r T )]. Consequently, this Goal-MDP is fully governed by the six tuple (S, A, G, T, R, γ). The reward, R(s, a, g) : S × A × G → R, and policy π(s, g) : S × G → A are also in turn conditioned on goal information.

Hindsight Experience Replay HER (Andrychowicz et al., 2017) builds upon this definition of GCVFs ( 5). The main insight of HER is that there is no valuable feedback from the environment when the agent does not reach the goal. This is further exacerbated when goals are sparse in the state-space. HER solves this problem by reusing these failed experiences for learning. It recomputes a reward for each reached state by relabeling them as pseudo-goals.

In our experiments, we employ HER's future strategy for pseudo-goal sampling. More specifically, two transitions from the same episode in the replay buffer for times t and t + f are sampled. The achieved goal g t+f is then assumed to be the pseudo-goal. The algorithm generates a new transition for the time step t with the reward re-computed as if g t+f was the desired goal, (s t , a t , s t+1 , R(s t , a t , g t+f )). HER uses this new transition as a sample.

4. PATH REWARD-BASED GCVFS:

In our definition of the GCVF, instead of making the reward function depend upon the goal, we count accumulated rewards over a path, path-rewards, only if the goal is reached. This makes the dependence on the goal explicit instead of implicit to the reward formulation. Mathematically,

EQUATION

)

where l is the time step when the agent reaches the goal. If the agent does not reach the goal, the GCVF is defined to be negative infinity. This first term (6a) is the expected cumulative reward over paths from a given start state to the goal. This imposes the constraint that cyclical paths in the state space must have negative cumulative reward for (6a) to yield finite values. For most practical physical problems, this constraints naturally holds if reward is taken to be some measure of negative energy expenditure. For example, in the robot arm experiment, moving the arm must expend energy (negative reward). Achieving a positive reward cycle would translate to generating infinite energy . In all our experiments with this formulation, we use a constant reward of -1 for all states, R P (s, a) = -1 ∀s, a.

For the cases when the agent does reach the goal at time step l (6a), the Bellman equation takes the following form:

EQUATION

)

Notice that terminal step in this equation is the step to reach the goal. This differs from Equation (3), where the terminal step is the step at which the episode ends. This formulation is equivalent to the end of episode occuring immediately when the goal is reached. This reformulation does not require goal-rewards, which in turn obviates the requirement for pseudo-goals and reward recomputation.

One-Step Loss To enable algorithms like HER to work under this reformulation we need to recognize when the goal is reached (7b). This recognition is usually done by the reception of high goal reward. Instead, we use (7b) as a one-step loss that serves this purpose which is one of our main contributions:

EQUATION

)

This loss is based on the assumption that one-step reward is the highest reward between adjacent start-goal states and allows us to estimate the one-step reward between them. Once learned, it serves as a proxy for the reward to the last step to the goal (7b). The Bellman equation ( 7), serves as a one-step rollout to combine rewards to find maximum reward paths to the goal.

One-step loss is different from the terminal step of Q-Learning because one-step loss is applicable to every transition unlike the terminal step. However, one-step loss can be thought of as Q-Learning where every transition is a one-step episode where the achieved goal is the pseudo goal.

One-step loss also different from the terminal condition in Kaelbling (1993) . Kaelbling (1993) defines Q P * (., ., .) similar to Eq (7) except the terminal condition is defined as Q P * (s t , a, g * ) = 0 when g t = g * . Under the stated assumptions, the two definitions are equivalent but one-step loss is advantageous as it can be applied to every transition unlike the Kaelbling (1993) terminal condition which can be only applied when g t = g * .

We modify an implementation of HER to include the step-loss term and disable goal rewards for our experiments. As in HER, we use the DDPG loss L ddpg while using the "future" goal sampling strategy described in the paper. The details of the resulting algorithm are shown as pseudo-code in Algorithm 1 in the Appendix.

4.1. DEEP FLOYD-WARSHALL REINFORCEMENT LEARNING:

The GCVF redefinition and one step-loss introduced in this paper are inspired by the tabular formulation of Floyd-Warshall Reinforcement Learning (FWRL) (Kaelbling, 1993) . We extend this algorithm for use with deep neural networks. Unfortunately, the algorithm itself does not show significant improvement over the baselines. However, the intuitions gained in its implementation led to the contributions of this paper.

The core contribution of FWRL is a compositionality constraint in the space of GCVFs. This constraint states that the optimal Q * value from any state s t to any goal g t+f is greater than or equal to the sum of optimal Q * values via any intermediate state-goal pair (s w , g w ):

EQUATION

We translate these constraints into loss terms and add them to the DDPG loss L ddpg and one-step loss L step . Taking cue from Mnih et al. (2015b) , we do not repeat the the main online network Q m in the loss term. We use a target network Q tgt and split the constraint into two loss terms. One loss term acts as a lower bound L lo and the other acts as an upper bound L up :

EQUATION

L up = ReLU[Q m (s t , a t , g w ) + Q tgt (s w , π t (s w , g t+f ; θ π ), g t+f ) -Q tgt (s t , a t , g t+f )] 2 . (11)

Note that the above terms differ only by choice of the target and main network.

FWRL Sampling:

We augment HER sampling to additionally get the intermediate state-goal pair (s w , g w ). Once a transition (s t , a t , r t , s t+1 ) and a future goal g t+f have been sampled from the same episode, we sample another intermediate state and goal pair (s w , g w ) such that t ≤ w ≤ t + f . For the Fetch tasks, we compare our method (red) against HER (blue) (Andrychowicz et al., 2016) and FWRL (green) (Kaelbling, 1993) on the distance-from-goal and success rate metrics. Both metrics are plotted against two progress measures: the number of training epochs and the number of reward computations. Except for the Fetch Slide task, we achieve comparable or better performance across the metrics and progress measures.

5. EXPERIMENTS:

We use the environments introduced in Plappert et al. (2018) for our experiments. Broadly the environments fall in two categories, Fetch and Hand tasks. Our results show that learning is possible across all environments without the requirement of goal-reward. More specifically, the learning happens even when reward given to our algorithm is agent is always "-1" as opposed to the HER formulation where a special goal-reward of "0" is needed for learning to happen.

The Fetch tasks involve a simulation of the Fetch robot's 7-DOF robotic arm. The four tasks are Reach, Push, Slide and PickAndPlace. In the Reach task the arm's end-effector is tasked to reach the a particular 3D coordinate. In the Push task a block on a table needs to be pushed to a given point on it. In the Slide task a puck must be slid to a desired location. In the PickAndPlace task a block on a table must be picked up and moved to a 3D coordinate.

The Hand tasks use a simulation of the Shadow's Dexterous Hand to manipulate objects of different shapes and sizes. These tasks are HandReach, HandManipulateBlockRotateXYZ, HandManipula-teEggFull and HandManipulatePenRotate. In HandReach the hand's fingertips need to reach a given configuration. In the HandManipulateBlockRotateXYZ, the hand needs to rotate a cubic block to a desired orientation. In HandManipulateEggFull, the hand repeats this orientation task with an egg, and in HandManipulatePenRotate, it does so with a pen.

Snapshots of all these tasks can be found in Figure 1 . Note that these tasks use joint angles, not visual input.

5.1. METRICS:

Similar to prior work, we evaluate all experiments on two metrics: the success rate and the average distance to the goal. The success rate is defined as the fraction of episodes in which the agent is able to reach the goal within a pre-defined threshold region. The metric distance of the goal is the euclidean distance between the achieved goal and the desired goal in meters. These metrics are plotted against a standard progress measure, the number of training epochs, showing comparable results of our method to the baselines.

To emphasize that our method does not require goal-reward and reward re-computation, we plot these metrics against another progress measure, the number of reward computations used during training. This includes both the episode rollouts and the reward recomputations during HER sampling.

5.2. HYPER-PARAMETERS CHOICES:

Unless specified, all our hyper-parameters are identical to the ones used in the HER implementation (Dhariwal et al., 2017) . We note two main changes to HER to make the comparison more fair. Firstly, we use a smaller distance-threshold. The environment used for HER and FWRL returns the goal-reward when the achieved goal is within this threshold of the desired goal. Because of the absence of goal-rewards, the distance-threshold information is not used by our method. We reduce the threshold to 1cm which is reduction by a factor of 5 compared to HER.

Secondly, we run all experiments on 6 cores each, while HER uses 19. The batch size used is a function of the number of cores and hence this parameter has a significant effect on learning.

To ensure fair comparison, all experiments are run with the same hyper-parameters and random seeds to ensure that variations in performance are purely due to differences between the algorithms.

5.3. RESULTS:

All our experimental results are described below, highlighting the strengths and weaknesses of our algorithm. Across all our experiments, the distance-to-the-goal metric achieves comparable performance to HER without requiring goal-rewards.

Fetch Tasks:

The experimental results for Fetch tasks are shown in Figure 2 . For the Fetch Reach and Push tasks, our method achieves comparable performance to the baselines across both metrics in terms of training epochs and outperforms them in terms of reward recomputations. Notably, the Fetch Pick and Place task trains in significantly fewer epochs. For the Fetch Slide task the opposite is true. We conjecture that Fetch Slide is more sensitive to the distance threshold information, which our method is unable to use.

Hand Tasks For the Hand tasks, the distance to the goal and the success rate show different trends.

We show the results in Figure 3 . When the distance metric is plotted against epochs, we get comparable performance for all tasks; when plotted against reward computations, we outperform all baselines on all tasks except Hand Reach. The baselines perform well enough on this task, leaving less scope for significant improvement. These trends do not hold for the success rate metric, on which our method consistently under-performs compared to the baselines across tasks. This is surprising, as all algorithms average equally on the distance-from-goal metric. We conjecture that this might be the result of high-distance failure cases of the baselines, i.e. when the baselines fail, they do so at larger distances from the goal. In contrast, we assume our method's success and failure cases are closer together.

6. ANALYSIS:

To gain a deeper understanding of the method we perform three additional experiments on different tasks. We ask the following questions: (a) How important is the step loss? (b) What happens when the goal-reward is also available to our method? (c) How sensitive is HER and our method to the distance-threshold?

How important is the step loss? We choose the Fetch-Push task for this experiment. We run our algorithm with no goal reward and without the step loss on this task. Results show that our algorithm fails to reach the goal when the step-loss is removed (Fig. 4a ) showing its necessity. For the hand tasks, we compare our method (red) against HER (blue) (Andrychowicz et al., 2016) and FWRL (green) (Kaelbling, 1993) for the distance-from-goal and success rate metrics. Furthermore, both metrics are plotted against two progress measures, the number of training epochs and the number of reward computations. Measured by distance from the goal, our method performs comparable to or better than the baselines for both progress measurements. For the success rate, our method underperforms against the baselines. 

8. APPENDIX:

Our algorithm 1 is different from HER Andrychowicz et al. (2016) because it contains additional step-loss term L step at line number 17 which allows the algorithm to learn even when the rewards received are independent of desired goal. Also in HER sampling (line 13), the algorithm recomputes the rewards because the goal is replaced with a pseudo-goal. Our algorithm does not need reward recomputation because the reward formulation does not depend on the goal and is not affected by choice of pseudo-goal. Our algorithm is also different from Floyd-Warshall Reinforcement learning because it does not contain L up and L lo terms and contains the additional L step .

Algorithm 1: Path-reward reinforcement learning 19 Update gradients for Qm and πm using loss L(. . . );

Result: Qm, πm

9. ABLATION ON LOSS AND GOAL REWARDS:

In Figure 6 and Figure 7 we show ablation on loss functions and goal rewards. In Figure 7 Our method is shown in blue with HER -Goal rewards + L step .

:



Figures and Tables:

Figure 1: Plappert et al. (2018) introduce challenging tasks on the Fetch robot and the Shadow Dextrous hand. We use these tasks for our experiments. Images are taken from the technical report.

Figure2: For the Fetch tasks, we compare our method (red) against HER (blue)(Andrychowicz et al., 2016) and FWRL (green)(Kaelbling, 1993) on the distance-from-goal and success rate metrics. Both metrics are plotted against two progress measures: the number of training epochs and the number of reward computations. Except for the Fetch Slide task, we achieve comparable or better performance across the metrics and progress measures.

Figure3: For the hand tasks, we compare our method (red) against HER (blue)(Andrychowicz et al., 2016) and FWRL (green)(Kaelbling, 1993) for the distance-from-goal and success rate metrics. Furthermore, both metrics are plotted against two progress measures, the number of training epochs and the number of reward computations. Measured by distance from the goal, our method performs comparable to or better than the baselines for both progress measurements. For the success rate, our method underperforms against the baselines.

Figure 4: (a) Effects of removing the step-loss from our methods. Results show that it is a critical component to learning in the absence of goal-rewards. (b) Adding goal-rewards to our algorithm that does have an effect further displaying how they are avoidable.

Figure 6: Ablation on loss functions for Fetch Push task. The Floyd-Warshall inspired loss functions L lo and L up do not help much. L step helps a little but only in conjunction with HER Andrychowicz et al. (2016).

Figure 7: Even when the Goal rewards are removed from HER Andrychowicz et al. (2016) training, the HER is able to learn only if the L step is added again. (HER-Goal Rewards+L step ) is our proposed method.

), . . . , (s b , g b , a b , s b+1 , g b+1 , r b ;g b+f b )] from M ; 14 L(. . . ) = 0 ; 15 for b ∈ 1to|B| do 16 (s b , g b , a b , s b+1 , g b+1 , r b , g b+f b ) = B[b] ; / * Step loss * / 17 L(. . . )+ = (Q m s b , a b , g b+1 -r b ) 2 ; / * DDPG loss * / 18 L(. . . )+ = (Q m s b , a b , g b+f b -r b -γQ tgt s b+1 , πtgt(s b+1 , g b+f b ; θπ ), g b+f b ) 2 ;

Initialize networks Q m s i , a i , g j ; θ Q and π(s i , sg ; θπ ) ; 2 Copy the main network to target networkQ tgt s i , a i , g j ; θ Q ← Q m s i , a i ,g j ; θ Q ; 3 Initialize replay memory M ; 4 for e ← 1 to E do 5 Take action a t ← -greedy(πm(s t , g; θπ )) ; 10 Observe s t+1 , g t+1 , r t ; 11 Store (s t , g t , a t , s t+1 , g t+1 , r t ; ge) in memory M [e] ; = [(s i , g i , a i , s i+1 , g i+1 , r i ; g i+f i

Table content: <table><tr><td/><td>/ * By default all states are unreachable</td><td>* /</td></tr><tr><td colspan="2">1 Sample ge ∈ G ;</td></tr><tr><td>6</td><td>Set t ← 0;</td></tr><tr><td>7</td><td>Observe state s t and achieved goal g t ;</td></tr><tr><td/><td>/ * Episode rollout</td><td>* /</td></tr><tr><td>8</td><td>for t ← 1 to T do</td></tr><tr><td>9</td><td/></tr><tr><td/><td>/ * Train</td><td>* /</td></tr><tr><td>12</td><td>for t ← 1 to T do</td></tr><tr><td>13</td><td>HER sample batch B</td></tr></table>

Peer review (iclr2019):
 This paper presents a reinterpretation of hindsight experience replay (HER) that avoids recomputing the reward function on resampled hindsight goals in favor of simply forcing the terminal state flag for goal-achieving transitions, referred to by the authors as a "step loss". The new proposal is evaluated on two goal-conditioned tasks from low-dimensional observations, and show modest improvements over HER and a function-approximation version of Floyd-Warshall RL, mostly as measured against the number of times the reward function needs to be recomputed. Pros: - minor improvement in computational cost - investigation of classical FWRL technique in context of deep RL Cons: - computational improvement seems very minor - sparse-reward implementations of HER already do essentially what this paper proposes Comments: The main contribution of the paper appears to be the addition of what the authors refer to as a "step loss", which in this case enforces the Q function to correctly incorporate the termination condition when goals are achieved. I.E. the discounted sum of future rewards for states that achieve termination should be exactly equal to the reward at that timestep. It*s not clear to me how this is fundamentally different than HER. One possible "sparse reward" implementation of HER involves no reward function recomputation at all, instead simply replacing the scalar reward and termination flag for resampled transitions with the indicator function for whether the transition achieves the resampled goal. Is this not essentially identical to the proposal in this paper? I would consider this a task-dependent implementation detail for an application of HER rather than a research contribution that deserves an entire paper. The authors claim the main advantage here is avoiding recomputation of the reward function for resampled goals. I do not find this particularly compelling, given that all of the evaluations are done in low-dimensional state space: reward recomputation here is just a low-dimensional euclidean distance computation followed by a simple threshold. In a world where we*re doing millions of forward and backward passes of large matrix multiplications, is this a savings that really requires investigation? It is somewhat telling that the results are compared primarily in terms of "# of reward function evaluations" rather than wall time. If the savings were significant, I expect a wall time comparison would be more compelling. Maybe the authors can come up with a situation in which reward recomputation is truly expensive and worth avoiding? All of the experiments in this paper use a somewhat unusual task setup where every timestep has a reward of -1. Have the authors considered other reward structures, such as the indicator function R=(1 if s==g else 0) or a distance-based dense reward? Would this proposal work in these cases? If not, how significant is a small change to HER if it can only work for one specific reward function? Conclusion: In my view, the main contribution is incremental at best, and potentially identical to many existing implementations of HER. The reconsideration of Floyd-Warshall RL in the context of deep neural networks is a refreshing idea and seems worth investigating, but I would need to see much more careful analysis before I could recommend this for publication.