LEARNING GOAL-CONDITIONED VALUE FUNCTIONS
Year: 2018

Abstract:
Multi-goal reinforcement learning (MGRL) addresses tasks where the desired goal state can change for every trial. State-of-the-art algorithms model these problems such that the reward formulation depends on the goals, to associate them with high reward. This dependence introduces additional goal reward resampling steps in algorithms like Hindsight Experience Replay (HER) that reuse trials in which the agent fails to reach the goal by recomputing rewards as if reached states were psuedo-desired goals. We propose a reformulation of goal-conditioned value functions for MGRL that yields a similar algorithm, while removing the dependence of reward functions on the goal. Our formulation thus obviates the requirement of reward-recomputation that is needed by HER and its extensions. We also extend a closely related algorithm, Floyd-Warshall Reinforcement Learning, from tabular domains to deep neural networks for use as a baseline. Our results are competitive with HER while substantially improving sampling efficiency in terms of reward computation.

heading: 1 INTRODUCTION
text: Many tasks in robotics require the specification of a goal for every trial. For example, a robotic arm can be tasked to move an object to an arbitrary goal position on a table (Gu et al., 2017); a mobile robot can be tasked to navigate to an arbitrary goal landmark on a map (Zhu et al., 2017). The adaptation of reinforcement learning to such goal-conditioned tasks where goal locations can change is called Multi-Goal Reinforcement Learning (MGRL) (Plappert et al., 2018). State-of-theart MGRL algorithms (Andrychowicz et al., 2017; Pong et al., 2018) work by estimating goalconditioned value functions (GCVF) which are defined as expected cumulative rewards from start states with specified goals. GCVFs, in turn, are used to compute policies that determine the actions to take at every state.
To learn GCVFs, MGRL algorithms use goal-reward, defined as the relatively higher reward recieved on reaching the desired goal state. This makes the reward function dependent on the desired goal. For example, in the Fetch-Push task (Plappert et al., 2018) of moving a block to a given location on a table, every movement incurs a “-1” reward while reaching the desired goal returns a “0” goal-reward. This dependence introduces additional reward resampling steps in algorithms like Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), where trials in which the agent failed to reach the goal are reused by recomputing rewards as if the reached states were pseudodesired goals. Due to the dependence of the reward function on the goal, the relabelling of every pseudo-goal requires an independent reward-recomputation step, which can be expensive.
In this paper, we demonstrate that goal-rewards are not needed to learn GCVFs. For the Fetch-Push example, the “0” goal-reward does not need to be achieved to learn its GCVF. Specifically, the agent continues to receive “-1” reward even when the block is in the given goal location. This reward formulation is atypical in conventional RL because high reward is used to specify the desired goal location. However, this goal-reward is not necessary in goal-conditioned RL because the goal is already specified at the start of every episode. We use this idea, to propose a goal-conditioned RL algorithm which learns to reach goals without goal-rewards. This is a counter-intuitive result which is important for understanding goal-conditioned RL.
Let us consider another example to motivate the redundancy of goal-rewards. Consider a student who has moved to a new campus. To learn about the campus, the student explores it randomly with no specific goal in mind. The key intuition here is that the student is not incentivized to find specific goal locations (i.e. no goal-rewards) but is aware of the effort required to travel between points around the university. When tasked with finding a goal classroom, the student can chain together these path efforts to find the least-effort path to the classroom. Based on this intuition of least-effort paths, we redefine GCVFs to be the expected path-reward that is learned for all possible start-goal pairs. We introduce a one-step loss that assumes one-step paths to be the paths of maximum reward between pairs wherein the state and goal are adjacent. Under this interpretation, the Bellman equation chooses and chains together one-step paths to find longer maximum reward paths. Experimentally, we show how this simple reinterpretation, which does not use goal rewards, performs as well as HER while outperforming it in terms of reward computation.
We also extend a closely related algorithm, Floyd-Warshall Reinforcement Learning (FWRL) (Kaelbling, 1993) (also called Dynamic Goal Reinforcement learning) to use parametric function approximators instead of tabular functions. Similar to our re-definition of GCVFs, FWRL learns a goal-conditioned Floyd-Warshall function that represents path-rewards instead of future-rewards. We translate FWRL’s compositionality constraints in the space of GCVFs to introduce additional loss terms to the objective. However, these additional loss terms do not show improvement over the baseline. We conjecture that the compositionality constraints are already captured by other loss terms.
In summary, the contributions of this work are twofold. Firstly, we reinterpret goal-conditioned value functions as expected path-rewards and introduce one-step loss, thereby removing the dependency of GCVFs on goal-rewards and reward resampling. We showcase our algorithm’s improved sample efficiency (in terms of reward computation). We thus extend algorithms like HER to domains where reward recomputation is expensive or infeasible. Secondly, we extend the tabular Floyd-Warshal Reinforcement Learning to use deep neural networks.

heading: 2 RELATED WORK
text: Goal-conditioned tasks in reinforcement learning have been approached in two ways, depending upon whether the algorithm explicitly separates state and goal representations. The first approach is to use vanilla reinforcement learning algorithms that do not explicitly make this separation (Mirowski et al., 2016; Dosovitskiy & Koltun, 2016; Gupta et al., 2017; Parisotto & Salakhutdinov, 2017; Mirowski et al., 2018). These algorithms depend upon neural network architectures to carry the burden of learning the separated representations.
The second approach makes this separation explicit via the use of goal-conditioned value functions (Foster & Dayan, 2002; Sutton et al., 2011). Universal Value Function Appoximators (Schaul et al., 2015) propose a network architecture and a factorization technique that separately encodes states
and goals, taking advantage of correlations in their representations. Temporal Difference Models combine model-free and model-based RL to gain advantages from both realms by defining and learning a horizon-dependent GCVF. All these works require the use of goal-dependent reward functions and define GCVFs as future-rewards instead of path-rewards, contrasting them from our contribution.
Unlike our approach, Andrychowicz et al. (2017) propose Hindsight Experience Replay, a technique for resampling state-goal pairs from failed experiences; which leads to faster learning in the presence of sparse rewards. In addition to depending on goal rewards, HER also requires the repeated recomputation of the reward function. In contrast, we show how removing goal-rewards removes the need for such recomputations. We utilize HER as a baseline in our work.
Kaelbling (1993) also use the structure of the space of GCVFs to learn. This work employs compositionality constraints in the space of these functions to accelerate learning in a tabular domain. While their definition of GCVFs is similar to ours, the terminal condition is different. We describe this difference in Section 4. We also extend their tabular formulation to deep neural networks and evaluate it against the baselines.

heading: 3 BACKGROUND
text: A reinforcement learning (RL) problem is formalized as a Markov Decision Process (MDP) (Sutton et al., 1998). A MDP is defined by a five tuple (S,A, T,R, γ), that governs a sequence of stateaction-reward triples [(s0, a0, r0), . . . , (sT , aT , rT )]. S is the state space, A is the action space, T (s, a) : S ×A → S is the system dynamics, R(s, a) : S ×A → R is the reward function and γ is the discount factor. In a typical RL problem the transition function T is not given but is known to be static. In RL, the objective is to find a policy π(s) : S → A that maximizes the expected cumulative reward over time, Rt = ∑∞ k=t γ
k−trk, called the return. The discount factor, γ < 1, forces the return to be finite for infinite horizons. Reinforcement learning is typically formulated in single-goal contexts. More recently there has been interest in multi-goal problems (Andrychowicz et al., 2017; Pong et al., 2018; Plappert et al., 2018), which is the focus of this work.

heading: 3.1 DEEP REINFORCEMENT LEARNING
text: A number of reinforcement learning algorithms use parametric function approximators to estimate the return in the form of an action-value function, Q(s, a):
Qπ(s, a) = Eπ [ T∑ k=t γk−tR(sk, ak) ∣∣∣∣∣st = s, at = a ] , (1)
where T is the episode length. When the policy π is optimal, the Q-function satisfies the Bellman equation (Bellman, 1954).
Q∗(st, at) = { R(st, at) + γmaxa∈AQ∗(st+1, a) if t < T R(sT , aT ) if t = T . (2)
TheQ∗() function can be learned using Q-learning algorithm (Watkins & Dayan, 1992). The optimal policy can be computed from Q∗ greedily, π∗(st) = argmaxa∈AQ∗(st, a). In Deep Q-Networks (DQN), Mnih et al. (2013) formulate a loss function based on the Bellman equation to approximate the optimal Q∗-function using a deep neural network, Qm:
L(θQm) = Eat∼π(st;θπm ) [ (Qm(st, at; θQm)− yt) 2 ] , (3)
where yt = R(st, at)+maxa γQtgt(st+1, a; θQtgt) , is the target andQtgt is the target network (Mnih et al., 2015a). The target network is a slower-changing copy of the main network that stabilizes learning. Mnih et al. (2015a) also employ replay buffers (Lin, 1993) that store transitions from episodes. During training, these transitions are sampled out of order to train the networks in an off-policy manner, avoiding correlation in the samples and thus leading to faster learning.
In this work, we use an extension of DQN to continuous action spaces called deep deterministic policy-gradients (DDPG) (Lillicrap et al., 2015). DDPG approximates the policy with a policy network πtgt(s; θπ) that replaces the max operator in yt. The target thus becomes yt =
R(st, at) + γQtgt(st+1, πtgt(st+1; θπ); θQtgt) and the loss function changes accordingly:
L(θQ, θπ) = Eat∼π(st;θπ)[(Qm(st, at; θQ)− yt) 2]. (4)

heading: 3.2 MULTI-GOAL REINFORCEMENT LEARNING
text: Multi-Goal Reinforcement Learning (Plappert et al., 2018) focuses on problems where the desired goal state can change for every episode. State-of-the-art MGRL algorithms learn a goal-conditioned value function (GCVF), Q (s, a, g), which is defined similar to the Q-function (5), but with an additional dependence on the desired goal specification g ∈ G :
Qπ (s, a, g) = Eπ [ T∑ k=t γk−tR(sk, ak, g) ∣∣∣∣∣st = s, at = a ] . (5)
The structure of the goal specification, g ∈ G, can be arbitrary. For example, in a robotic arm experiment, possible goal specifications include the desired position of the end-effector and the desired joint angles of the robot. The states and achieved goals are assumed to be an observable part of the Goal-MDP to enable the agent to learn the correspondences between them, [(s0, a0, g0, r0), . . . , (sT , aT , gT , rT )]. Consequently, this Goal-MDP is fully governed by the six tuple (S,A,G, T,R, γ). The reward, R(s, a, g) : S ×A×G → R, and policy π(s, g) : S ×G → A are also in turn conditioned on goal information.
Hindsight Experience Replay HER (Andrychowicz et al., 2017) builds upon this definition of GCVFs (5). The main insight of HER is that there is no valuable feedback from the environment when the agent does not reach the goal. This is further exacerbated when goals are sparse in the state-space. HER solves this problem by reusing these failed experiences for learning. It recomputes a reward for each reached state by relabeling them as pseudo-goals.
In our experiments, we employ HER’s future strategy for pseudo-goal sampling. More specifically, two transitions from the same episode in the replay buffer for times t and t + f are sampled. The achieved goal gt+f is then assumed to be the pseudo-goal. The algorithm generates a new transition for the time step t with the reward re-computed as if gt+f was the desired goal, (st, at, st+1, R(st, at, gt+f )). HER uses this new transition as a sample.

heading: 4 PATH REWARD-BASED GCVFS
text: In our definition of the GCVF, instead of making the reward function depend upon the goal, we count accumulated rewards over a path, path-rewards, only if the goal is reached. This makes the dependence on the goal explicit instead of implicit to the reward formulation. Mathematically,
QPπ (s, a, g ∗) =  Eπ [ l−1∑ k=t γk−tRP (sk, ak) ∣∣∣∣∣s, a, gl = g∗ ]
if ∃ l such that gl = g∗ (6a)
−∞ otherwise, (6b) where l is the time step when the agent reaches the goal. If the agent does not reach the goal, the GCVF is defined to be negative infinity. This first term (6a) is the expected cumulative reward over paths from a given start state to the goal. This imposes the constraint that cyclical paths in the state space must have negative cumulative reward for (6a) to yield finite values. For most practical physical problems, this constraints naturally holds if reward is taken to be some measure of negative energy expenditure. For example, in the robot arm experiment, moving the arm must expend energy (negative reward). Achieving a positive reward cycle would translate to generating infinite energy . In all our experiments with this formulation, we use a constant reward of -1 for all states,RP (s, a) = −1 ∀s, a. For the cases when the agent does reach the goal at time step l (6a), the Bellman equation takes the following form:
QP∗ (st, at, g ∗) =
{ RP (st, at) + γmax
a∈A QP∗ (st+1, a, g ∗) if t < l − 1 (7a)
RP (sl−1, al−1) if t = l − 1 . (7b)
Notice that terminal step in this equation is the step to reach the goal. This differs from Equation (3), where the terminal step is the step at which the episode ends. This formulation is equivalent to the end of episode occuring immediately when the goal is reached. This reformulation does not require goal-rewards, which in turn obviates the requirement for pseudo-goals and reward recomputation.
One-Step Loss To enable algorithms like HER to work under this reformulation we need to recognize when the goal is reached (7b). This recognition is usually done by the reception of high goal reward. Instead, we use (7b) as a one-step loss that serves this purpose which is one of our main contributions:
Lstep(θQ) = (QP∗ (sl−1, al−1, gl; θQ)−R(sl−1, al−1))2. (8)
This loss is based on the assumption that one-step reward is the highest reward between adjacent start-goal states and allows us to estimate the one-step reward between them. Once learned, it serves as a proxy for the reward to the last step to the goal (7b). The Bellman equation (7), serves as a one-step rollout to combine rewards to find maximum reward paths to the goal.
One-step loss is different from the terminal step of Q-Learning because one-step loss is applicable to every transition unlike the terminal step. However, one-step loss can be thought of as Q-Learning where every transition is a one-step episode where the achieved goal is the pseudo goal.
One-step loss also different from the terminal condition in Kaelbling (1993). Kaelbling (1993) defines QP∗ (., ., .) similar to Eq (7) except the terminal condition is defined as Q P ∗ (st, a, g∗) = 0 when gt = g∗. Under the stated assumptions, the two definitions are equivalent but one-step loss is advantageous as it can be applied to every transition unlike the Kaelbling (1993) terminal condition which can be only applied when gt = g∗.
We modify an implementation of HER to include the step-loss term and disable goal rewards for our experiments. As in HER, we use the DDPG loss Lddpg while using the “future” goal sampling strategy described in the paper. The details of the resulting algorithm are shown as pseudo-code in Algorithm 1 in the Appendix.

heading: 4.1 DEEP FLOYD-WARSHALL REINFORCEMENT LEARNING
text: The GCVF redefinition and one step-loss introduced in this paper are inspired by the tabular formulation of Floyd-Warshall Reinforcement Learning (FWRL) (Kaelbling, 1993). We extend this algorithm for use with deep neural networks. Unfortunately, the algorithm itself does not show significant improvement over the baselines. However, the intuitions gained in its implementation led to the contributions of this paper.
The core contribution of FWRL is a compositionality constraint in the space of GCVFs. This constraint states that the optimal Q∗ value from any state st to any goal gt+f is greater than or equal to the sum of optimal Q∗ values via any intermediate state-goal pair (sw, gw):
Q∗ (st, at, gw) +Q∗ (sw, π∗(sw, gt+f ; θπ), gt+f ) ≥ Q∗ (st, at, gt+f ) . (9)
We translate these constraints into loss terms and add them to the DDPG loss Lddpg and one-step loss Lstep. Taking cue from Mnih et al. (2015b), we do not repeat the the main online network Qm in the loss term. We use a target network Qtgt and split the constraint into two loss terms. One loss term acts as a lower bound Llo and the other acts as an upper bound Lup:
Llo = ReLU[Qtgt (st, at, gw) +Qtgt (sw, πt(sw, gt+f ; θπ), gt+f )−Qm (st, at, gt+f )]2 (10) Lup = ReLU[Qm (st, at, gw) +Qtgt (sw, πt(sw, gt+f ; θπ), gt+f )−Qtgt (st, at, gt+f )]2. (11)
Note that the above terms differ only by choice of the target and main network.
FWRL Sampling We augment HER sampling to additionally get the intermediate state-goal pair (sw, gw). Once a transition (st, at, rt, st+1) and a future goal gt+f have been sampled from the same episode, we sample another intermediate state and goal pair (sw, gw) such that t ≤ w ≤ t+ f .

heading: 5 EXPERIMENTS
text: We use the environments introduced in Plappert et al. (2018) for our experiments. Broadly the environments fall in two categories, Fetch and Hand tasks. Our results show that learning is possible across all environments without the requirement of goal-reward. More specifically, the learning happens even when reward given to our algorithm is agent is always “-1” as opposed to the HER formulation where a special goal-reward of “0” is needed for learning to happen.
The Fetch tasks involve a simulation of the Fetch robot’s 7-DOF robotic arm. The four tasks are Reach, Push, Slide and PickAndPlace. In the Reach task the arm’s end-effector is tasked to reach the a particular 3D coordinate. In the Push task a block on a table needs to be pushed to a given point on it. In the Slide task a puck must be slid to a desired location. In the PickAndPlace task a block on a table must be picked up and moved to a 3D coordinate.
The Hand tasks use a simulation of the Shadow’s Dexterous Hand to manipulate objects of different shapes and sizes. These tasks are HandReach, HandManipulateBlockRotateXYZ, HandManipulateEggFull and HandManipulatePenRotate. In HandReach the hand’s fingertips need to reach a given configuration. In the HandManipulateBlockRotateXYZ, the hand needs to rotate a cubic block to a desired orientation. In HandManipulateEggFull, the hand repeats this orientation task with an egg, and in HandManipulatePenRotate, it does so with a pen.
Snapshots of all these tasks can be found in Figure 1. Note that these tasks use joint angles, not visual input.

heading: 5.1 METRICS
text: Similar to prior work, we evaluate all experiments on two metrics: the success rate and the average distance to the goal. The success rate is defined as the fraction of episodes in which the agent is able to reach the goal within a pre-defined threshold region. The metric distance of the goal is the euclidean distance between the achieved goal and the desired goal in meters. These metrics are
plotted against a standard progress measure, the number of training epochs, showing comparable results of our method to the baselines.
To emphasize that our method does not require goal-reward and reward re-computation, we plot these metrics against another progress measure, the number of reward computations used during training. This includes both the episode rollouts and the reward recomputations during HER sampling.

heading: 5.2 HYPER-PARAMETERS CHOICES
text: Unless specified, all our hyper-parameters are identical to the ones used in the HER implementation (Dhariwal et al., 2017). We note two main changes to HER to make the comparison more fair. Firstly, we use a smaller distance-threshold. The environment used for HER and FWRL returns the goal-reward when the achieved goal is within this threshold of the desired goal. Because of the absence of goal-rewards, the distance-threshold information is not used by our method. We reduce the threshold to 1cm which is reduction by a factor of 5 compared to HER.
Secondly, we run all experiments on 6 cores each, while HER uses 19. The batch size used is a function of the number of cores and hence this parameter has a significant effect on learning.
To ensure fair comparison, all experiments are run with the same hyper-parameters and random seeds to ensure that variations in performance are purely due to differences between the algorithms.

heading: 5.3 RESULTS
text: All our experimental results are described below, highlighting the strengths and weaknesses of our algorithm. Across all our experiments, the distance-to-the-goal metric achieves comparable performance to HER without requiring goal-rewards.
Fetch Tasks The experimental results for Fetch tasks are shown in Figure 2. For the Fetch Reach and Push tasks, our method achieves comparable performance to the baselines across both metrics in terms of training epochs and outperforms them in terms of reward recomputations. Notably, the Fetch Pick and Place task trains in significantly fewer epochs. For the Fetch Slide task the opposite is true. We conjecture that Fetch Slide is more sensitive to the distance threshold information, which our method is unable to use.
Hand Tasks For the Hand tasks, the distance to the goal and the success rate show different trends. We show the results in Figure 3. When the distance metric is plotted against epochs, we get comparable performance for all tasks; when plotted against reward computations, we outperform all baselines on all tasks except Hand Reach. The baselines perform well enough on this task, leaving less scope for significant improvement. These trends do not hold for the success rate metric, on which our method consistently under-performs compared to the baselines across tasks. This is surprising, as all algorithms average equally on the distance-from-goal metric. We conjecture that this might be the result of high-distance failure cases of the baselines, i.e. when the baselines fail, they do so at larger distances from the goal. In contrast, we assume our method’s success and failure cases are closer together.

heading: 6 ANALYSIS
text: To gain a deeper understanding of the method we perform three additional experiments on different tasks. We ask the following questions: (a) How important is the step loss? (b) What happens when the goal-reward is also available to our method? (c) How sensitive is HER and our method to the distance-threshold?
How important is the step loss? We choose the Fetch-Push task for this experiment. We run our algorithm with no goal reward and without the step loss on this task. Results show that our algorithm fails to reach the goal when the step-loss is removed (Fig. 4a) showing its necessity.
What happens when the goal-reward is also available to our method? We run this experiment on the Fetch PickAndPlace task. We find that goal-rewards do not affect the performance of our algorithm further solidifying the avoidability of goal-reward (Fig 4b).
How sensitive is HER and our method to the distance-threshold? In the absence of goalrewards, our algorithm is not to able capture distance threshold information that decides whether the agent has reached the goal or not. This information is available to HER. To understand the sensitivity of our algorithm and HER on this parameter, we vary it over 0.05 (the original HER value), 0.01 and 0.001 meters (Fig. 5). Results show that for the success-rate metric, which is itself a function of this parameter, both algorithms are affected equally (Fig. 5a). For the distance-from-goal, only HER is affected (Fig. 5b). This fits our expectations as set up in section 5.2.

heading: 7 CONCLUSION
text: In this work we pose a reinterpretation of goal-conditioned value functions and show that under this paradigm learning is possible in the absence of goal reward. This is a surprising result that runs counter to intuitions that underlie most reinforcement learning algorithms. In future work, we will augment our method to incorporate the distance-threshold information to make the task easier to learn when the threshold is high. We hope that the experiments and results presented in this paper lead to a broader discussion about the assumptions actually required for learning multi-goal tasks.

heading: 8 APPENDIX
text: Our algorithm 1 is different from HER Andrychowicz et al. (2016) because it contains additional step-loss term Lstep at line number 17 which allows the algorithm to learn even when the rewards received are independent of desired goal. Also in HER sampling (line 13), the algorithm recomputes the rewards because the goal is replaced with a pseudo-goal. Our algorithm does not need reward recomputation because the reward formulation does not depend on the goal and is not affected by choice of pseudo-goal. Our algorithm is also different from Floyd-Warshall Reinforcement learning because it does not contain Lup and Llo terms and contains the additional Lstep.
Algorithm 1: Path-reward reinforcement learning /* By default all states are unreachable */
1 Initialize networksQm ( si, ai, gj ; θQ ) and π(si, sg ; θπ) ;
2 Copy the main network to target networkQtgt ( si, ai, gj ; θQ ) ← Qm ( si, ai, gj ; θQ ) ; 3 Initialize replay memoryM ; 4 for e← 1 toE do 5 Sample ge ∈ G ; 6 Set t← 0; 7 Observe state st and achieved goal gt ; /* Episode rollout */ 8 for t← 1 to T do 9 Take action at ← -greedy(πm(st, g; θπ)) ;
10 Observe st+1, gt+1, rt ; 11 Store (st, gt, at, st+1, gt+1, rt; ge) in memoryM [e] ;
/* Train */ 12 for t← 1 to T do 13 HER sample batchB = [(si, gi, ai, si+1, gi+1, ri; gi+fi ), . . . , (sb, gb, ab, sb+1, gb+1, rb; gb+fb )] fromM ; 14 L(. . . ) = 0 ; 15 for b ∈ 1to|B| do 16 (sb, gb, ab, sb+1, gb+1, rb, gb+fb ) = B[b] ;
/* Step loss */ 17 L(. . . )+ = (Qm ( sb, ab, gb+1 ) − rb)2 ;
/* DDPG loss */ 18 L(. . . )+ = (Qm ( sb, ab, gb+fb ) − rb − γQtgt ( sb+1, πtgt(sb+1, gb+fb ; θπ), gb+fb ) )2 ; 19 Update gradients forQm and πm using lossL(. . . );
Result:Qm, πm

heading: 9 ABLATION ON LOSS AND GOAL REWARDS
text: In Figure 6 and Figure 7 we show ablation on loss functions and goal rewards. In Figure 7 Our method is shown in blue with HER - Goal rewards + Lstep.

Review summaries:
This paper aims to improve on Hindsight Experience Replay by removing the need to compute rewards for reaching a goal. The idea is to frame goal-reaching as a shortest path problem where all rewards are -1 until the goal is reached, removing the need to compute rewards. While similar ideas were explored in a recent arxiv tech report, this paper claims to build on these ideas with new loss functions. The experimental results do not seem to be any better compared to baselines when measured in terms of data efficiency, but the proposed method requires fewer “reward computations”.