Understanding Hindsight Goal Relabeling Requires Rethinking Divergence Minimization
Authors: Lunjun Zhang
Bradly C. Stadie
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
multi-goal reinforcement learning
imitation learning

Abstract: Hindsight goal relabeling has become a foundational technique for multi-goal reinforcement learning (RL). The idea is quite simple: any arbitrary trajectory can be seen as an expert demonstration for reaching the trajectory's end state. Intuitively, this procedure trains a goal-conditioned policy to imitate a sub-optimal expert. However, this connection between imitation and hindsight relabeling is not well understood. Modern imitation learning algorithms are described in the language of divergence minimization, and yet it remains an open problem how to recast hindsight goal relabeling into that framework. In this work, we develop a unified objective for goal-reaching that explains such a connection, from which we can derive goal-conditioned supervised learning (GCSL) and the reward function in hindsight experience replay (HER) from first principles. Experimentally, we find that despite recent advances in goal-conditioned behaviour cloning (BC), multi-goal Q-learning can still outperform BC-like methods; moreover, a vanilla combination of both actually hurts model performance. Under our framework, we study when BC is expected to help, and empirically validate our findings. Our work further bridges goal-reaching and generative modeling, illustrating the nuances and new pathways of extending the success of generative models to RL.

Under review as a conference paper at ICLR 2023
UNDERSTANDING HINDSIGHT GOAL RELABELING REQUIRES RETHINKING DIVERGENCE MINIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Hindsight goal relabeling has become foundational technique for multi-goal reinforcement learning (RL). The idea is quite simple: any arbitrary trajectory can be seen as an expert demonstration for reaching the trajectory's end state, Intuitively, this procedure trains a goal-conditioned policy to imitate a sub-optimal expert. However, this connection between imitation and hindsight relabeling is not well understood. Modern imitation learning algorithms are described in the language of divergence minimization, and yet it remains an open problem how to recast hindsight goal relabeling into that framework. In this work, develop unified objective for goal-reaching that explains such a connection, from which we can derive goal-conditioned supervised learning (GCSL) and the reward function in hindsight experience replay (HER) from first principles. Experimentally. we find that despite recent advances in goal-conditioned behaviour cloning (BC), multi-goal Q-learning can still outperform BC-like methods; moreover, a vanilla combination of both actually hurts model performance. Under our framework, we study when BC is expected to help. and empirically validate our findings. Our work further bridges goal-reaching and generative modeling, illustrating the nuances and new pathways of extending the success of generative models to RL.
INTRODUCTION
Goal reaching is an essential aspect of intelligence in sequential decision making. Unlike the conventional formulation of reinforcement learning (RI.), which aims to encode all desired behaviors into a single scalar reward function that is amenable learning (Silver et al. (021), ). goal reaching formulates the problem of RL as applying sequence of actions to rearrange the environment into desired state (Batra et al., 2020). Goal-reaching is a highly flexible formulation. For instance, we
can design the goal-space to capture salient information about specific factors of variations that
care about (Plappert et al.
2018); we can use natural language instructions to define more abstract
goals (Lynch & Sermanet 2220 Ahn et al. 2022); we can encourage exploration by prioritizing
previously unseen goals (Pong et al. 2019 Warde-Farley et al.] 2018 8: Pitis et al.) 2020); and we
can even use self-supervised procedures naturally learn goal-reaching policies without reward
engineering (Pong et al. 2018 Nair et al. 2018b Zhang et al. 2021 OpenAl et al, 2021).
Reward is not enough. Usually, rewards are manually constructed, either through laborious reward engineering, or from task-specific optimal demonstrations, neither of which is scalable solution. How can RL agents learn useful behaviors from unlabeled reward-free trajectories, similar to how NLP models such as BERT (Dellin et al., 2018) and GPT-3 (Brown et al., (020) are able to learn language from unlabeled text corpus? Goal reaching
promising paradigm for unsupervised behavior acquisition, but it is unclear how to write down well-defined objective for goal-conditioned policies, unlike language models that just predict the next token. This paper aims to define such an objective that unifies many prior approaches and strengthens the foundation of goal-conditioned RL.
We start from the following observation: hindsight goal relabeling (Andrychowicz et al. 2017) can turn an arbitrary trajectory into sub-optimal expert demonstration. Thus, goal-conditioned RL might be doing a special kind of imitation learning. Currently, divergence minimization is the de facto way to describe imitation learning methods (Ghasemipour et al.) (2020), hindsight goal relabeling into the divergence minimization framework. On top of that, to tackle the sub-optine optimaliyyl of hindsight-relabeled trajectories, we should explicitly maximize the probability of
should be able recast

---

Under review as a conference paper at ICLR 2023
π
SE (ninds)) p ( s.a]]
Dy(p" plte" (pppplsaa))
Don't run off the cliff
Make the right destination of hitting gool the
arrive at
Stay the right stare-action visitetion distribution benchit hindsight
Max Entropy Regularization
probabilty
Figure 1: A unified objective for goal-reaching (see equation equation 14 for more details).
reaching the desired goal. Following these intuitions, we derive the reward function used in HER from first principles, as well as other behaviour cloning (BC) like methods such as GCSL (Ghosh et al. 2019) and Hindsight BC (HBC) (Ding et al. 2019).
Experimentally, we show that multi-gan goaa Q-learning based on HER-like rewards, when carefully tuned, still outperform goal-conditioned BC (such GCSL/ HBC). Moreover, vanilla com- bination multi-goal Q-learning and BC (HER + HBCC), supposedly combining the best of both worlds, in fact hurts performance. We utilize our unified framework analyze when a BC loss could help, and propose modified algorithm named Hindsight Divergence Minimization (HDM) that uses Q-learning to account for the worst while imitating the best. HDM avoids the pitfalls of HER + HBC and improves policy success rates on a variety of self-supervised goal-reaching environments. Additionally, our framework reveals largely unexplored design space for goal- meachin algorithms and potential paths of importing generative modeling techniques into multi-goal RI.
REINFORCEMENT LEARNING AND GENERATIVE MODELING
2.1 THE REINFORCEMENT LEARNING (RL) PROBLEM
We first review the basics of RL and generative modeling. A Markov Decision Process (MDP) is typically parameterized by (S, A, p¹ r): a state space S, an action space A, an initial state distribution (s), a dynamics function p(s' a) which defines the transition probability, and reward function r(s, a). A policy function defines a probability distribution S x A + R For an -henite- MDP. given the policy μ. and the state distribution at step (starting from pd at t = 0), the state distribution at step f + is given by:
(5')
pls' s, a)p(a s)p, (s)dsda
SxA
The stale visitation distribution sums over all timesteps via geometric distribution Geom (7):
pk(s) = 7) Σ 2 (s)
However, the trajectory sampling process does not happen in this discounted manner, so the discount factor (0, 1) is often absorbed into the cumulative return instead (Silver et al. (2014):
[ pa(s) / pla s)r(s, E I(n) -
Σ (r))))
o (sa) (sa))((all so)
p(s: (so. [(
From [and] 2
we also see that the future state distribution pt of policy μi, defined as geometrically discounted sum of state distribution at all future timesteps given current state and action, is given by
2

---

Under review as a conference paper at ICLR 2023
the following recursive relationship (Eysenbach et al. 2020b: Janner et al. 2020):
Pl (s+ ss = (1-7)p(s* s, + + SxA
p(s' s, a.a)) st)ppt (st ,'','' d''d''da' (4)
In multi-goal RI, an MDP is augmented with a goal space 9, and we learn a goal-conditioned policy S x G agent a reward of O when the goal is reached and -1 otherwise, and uses hindsight goal relabeling to
A + R.. Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) gives the
increase learning efficiency of goal-conditioned Q-learning by replacing the initial behavioral goals with achieved goals (future states within the same trajectory).
2.2 IMITATION LEARNING (IL) AS DIVERGENCE MINIMIZATION
We first review f-divergence between two probability distributions P and Q and its variational bound:
p(æ)
D,(P Q))P1Q)= - {x))f
dr N sup B.- - TET
(5)
((x)
where f is convex function such that f(1) = 0, f is the convex conjugate of f. and T is arbitrary
class of functions T' X - IR. This variational bound was originally derived in (Nguyen et al. (010)
and was popularized by GAN (Goodfellow al.] 2014 Nowozin et al. 2016) and subsequently by
imitation learning (Ho & Ermon 2016; Fu et al.] 2017 Finn et al. 2016 Ghasemipour et al. 2020).
The equality holds true under mild conditions (N Nguyen et al.) 2010). and the optimal 7 is given by
T (x) f(())))(())) The canonical formulation of imitation learning follows (Ho & Ermon
2016 Ghasemipour et al. 2020), where (s, a) from the expert:
min (p (8,a) a)
pf as) min max (x,))) [T(s, a)] - E (f
(6)
Because of the policy grudient theorem (Sutton et al. 1999), the policy i needs to optimize the cumulative return under its own trajectory distribution p"(s, a) with the reward being a() (T((s,))) Under this formulation, Sensen-Shammng
divergence leads to GAIL (Ho & Ermon 2016), reverse KL leads AIRL (Fu et al., 2017). Note that can in principle be any convex function (we can satisfy f(1) = 0 by simply adding a constant).
2.3
EHERGY-BASEDI MODELS (EBM)
An energy-based model (EBM) is defined by po(x:) exp(- exp(-Ea(x=) where Fgg RDD Z(0)
TK is
the energy function and (0) fx (x)(=Eo(=))dr is the partition function. The gradient of the log- log-likel log (o(x) w.r.t. 0 (known as contrastive divergence (Hinton 2002) can be expressed as
log po (x 00
(E(('))
(E)
Epp (u')
00
Sampling from Pa (x) can be difficult, as Langevin dynamics is often required (Welling & Teh 2011 Du & Mordatch, 2019: Grathwohl et al., 2019). Alternatively, EBMs be trained via Noise Contrastive Estimation (NCE) (Gutmann & Hyvärinen] 2012; Mnih & Kavukcuoglu 2013 Gao et al. 2020 Rhodes al. 2020), which assumes access to "noise" distribution pr. (x)x)) and learns energy functions through density ratio estimation. Let (z) = - -Eo(x) - -log Zo (Mnih & Teh 2012). NCE
learns classifier that distinguishes the data distribution p(z) from noise Pa (z), where noise samples
are l times more frequent (Mnih & Kavukcuoglu 2013 Mikolov et al. 2013):
p(D = x) (1)) (p)))((((())))))) (z) + l: Pm. = (ss - log Pm.( - log k)
(8)
To be more concise, we denote An (a., k) = so (x) log (x) - log k. Gradients of the weighted
binary cross entropy loss asymptotically approximate contrastive divergence (7) (Mnih & Teh 2012):
d
A-+00 alog logpo((x)
d0 Bm (ooe) (z, k)) + l- Ep.. (z) [log(1 ( ((Ao
(9)
20
Closely related to NCE the more recent InfoNCE loss Van den Oord et al. popularity in the contrastive learning setting (Chen al. 2020 He et al. in NCE as the batch size of negative samples in InfoNCE. In summary, NCE gives EBM a more tractable way to maximize data likelihood.
2018) which has gained
2020); we can interpret
3

---

Under review as a conference paper at ICLR 2023
3 GRAPHICAL MODELS FOR HINDSIGHT GOAL RELABELING
Consider the setting: given an environment ç (s', a, (0))p((0) (g)) where we generate dataset of trajectories D = { dg, $1, "..)) by sampling from the initial state distribution po(s). an unobserved actor policy puaa s). and the dynamics p(s' a). We aim train goal-conditioned policy (a s,g) from this arbitrary dataset with relabeled future states the goals. pt (g) is the behavioral goal distribution assumed to be given apriorí by the environment. To recast the problem of goal- roal-reaching as imitation learning equation 6 need where we define the target (expert) distribution and the policy distribution we want to match.
up f f-divergence minimization
1
π
π
9
5
E
Figure 2: Py (s, (s s, s,a) Figure 3: pt (s)pu(s) s,g)
Figure 4: pt (g) (s. a l) (used in HER HDM)
hindsight-relabeled distribution (used HBC/ GCSL)
The training signal comes from factorizing the joint distribution of state-action-gor differently. For the relabeled target distribution, assume unconditioned actor μl generating state-action distribution at first, with the goals coming from the future state distribution equation conditioned on the given state and action. For the condiiindt policy distribution, behavior goals given apriori, and the state-action distribution is generated conditioned on the behavioral goals. Thus, the target distribution (see Figure 2 for states, actions, and hindsight goals is:
(p a, s*) = ppp(s, a)p (s* s.a)
(10)
Note that pt is given by equation equation 4 and (s, a) similar to p a) equation equation 6 In the fashion of behavioral cloning (BC), if we do not care about matching the states, can write the joint distribution we are trying to match as (see Figure 3):
DCC a, g) p* (s) 8,9)
We can recover the objective of Hindsight Behavior Cloning (HBC) (Ding et al. 2019 Eysenbach
al. 2020a) and Goal-Conditionel Supervised Learning (GCSL) Ghosh al. (019) via minimizing
a KL KL-divergence:
min DRL. (p,(s, s*) Px (s, a, a..)) min E (p.(s.)) ( (s (s+]a))) log T(a 8,
(12)
In many cases, matching the states is more important than matching coneicoondiiooeed actions (Ross
al. 201T Ghasemipour al. 2020). The joint distribution for states, actions, behavioral goals for (see Figure is:
Pr (s. p² (g) ()
(13)
However, it important to recognize that even after adding this state-matching part, there is still missing component of the objective. Divergence minimization encourages the agent stay in the "right" state-action distribution given the benefit of hindsight, but we still need the policy to actually hit the goal (see Figure []. When we condition the policy on a goal, we should maximize the likelihood of seeing that goal the future state distribution (see Figure 5). Together with a maximum entropy regularization on the policy H(T) (Ho Ermon 2016; Schulman et al.) 2017), we propose the tollowing goal-reaching objective:
Figure 5: the future state distribution (s+ 1,, a) of goal-conditioned policy when starting from state visited by Js.
min Dj (p" (g) alg) Il p/ (s* a)))p((s.a)
SE
[log loogp [g]s, H(T))
(14)
ot (g) pa(e)
(a/$.0)
(a) f-divergence
(c) entropy
(b) goul likelihood term
4

---

Under review as a conference paper at ICLR 2023
Compared to equation 12 and equation 6 the f-divergence term equation 14(a] swaps the order between the expert and the policy. The coefficient 3 controls the importance of "hitting the goal". This incentive is already implicit in hindsight-relablles data, in many cases BC objective alone can learn to hit the goal as well (Ding et al. 2019; Ghosh et al. 2019. Lynch et al. 2019 Jang et al. (02T). However, hindsight-relabeled trajectories are often sub sub-optimal demonstrations: not every action is acting towards achieving the goal. Thus, maximizing the likelihood of achieving the goal is crucial, and it plays a key role in deriving the reward function of HER (Andrychowicz et al. 2017).
4 BRIDGING GOAL-REACHING AND GENERATIVE MODELING
In this section, we study how to optimize the proposed unifying objective 14 We show that the Q-function trained for goal-reaching Qe a, g) is implicitly doing generative modeling. Its temporal difference is modeling the density ratio between the hindsight-relabeled distribution and the goal- conditioned behavioral distribution. It also approximates an svergy-baset model for future-state distribution when marginalized over the goal distribution. In section 4.1 we demonstrate how goal- conditioned Q-learning does divergence minimization (equation a4))). In section 4.2 we show how Q-functions can define EBM for cumulative future states, which in turn allows the policy
to maximize goal likelihood (equation b+)). In section 4.3 we show that combining the derived
results yields the HER reward. In section 4.4 we study when a BC loss can help by analyzing the loss terms being left out by the HER reward.
4.1 DIVERGENCE MINIMIZATION WITH GOAI-CONDITIONED Q-LEARNING
This section decomposes equation 14 a). We start with the f-divergence bound from equations equation and equation G
()(pn(s,,gg) (u(s,a,s
max E
p(g)
[T(s, 9)1 -E E pa(x.a) p (t
(fT(T(s,,ss)) (15)
T
Ps (s.alg)
Now we negate to get r(s, a.g) (ss, a. 9). and the divergence minimization problem becomes:
max min E. pa(a.)) (=*100)))
(-r(s,,,, + E p(a) (r(s,ag9) pr(s.olg)
(16)
We can interpret r as a Gtyle (Ho & Ermon 2016) discriminator or reward. However, we
aim to derive discriminator- learning process that directly trains the Q-function corresponding to this reward Q0 a, g) = r(s. a.g) + 7 P"*Q(s, g) where p= is the transition operator:
P (TTQss, a, g) 1ppp (Q0 a', g.)].. Re-writing the equation equation 16 w.r.t Q:
min E pa(a.a) (- 7 pn' (*))(s, 0,8 ))] + E p(g)
[[Q. 7 7 p=(s,alg)
P (s,
(17)
p (s+)aa)
A similar change-of- saaenge-o-vvrrateee variable has been explored in the context of offline RL (Nachum et al. 2019a b)
and imitation learning (Kostrikov et al. 2019; Zhhu et al., 2020); we may call those methods the DICE (Nachum & Dai 2020) family. The major pain point of DICE-like methods is that they require samples from the initial state distribution p (s) (Garg et al. 2021). Here, the following lemma shows
that in the goal-conditioned case, we can use arbitrary offfine trajectories to evaluate the expected rewards under goal-conditioned online rollouts:
Lemma 4.1 (Online-to-olle to-offline transformation for goal reaching). Given a goal-conditioned policy Ta s. g), its corresponding Q-function QT a, g), and arbitrary state-action visitation distribution Pir a) of another policy p(a s), the expected temporal difference for online rollouts under is:
(s.ll)) [(Q" - 7. PP.PPF'' (P=Q")(s a, g)] = (sttaoo 1QT (s, ā, - .P= (sp
Using Lemma the objective in equation 16 becomes:
max min E (e (()sa))))
f (Q XP" st)s + -
- ((p"((ss,a)))) (18)
The function f is the convex conjugate of f in f-divergence. We can pick almost any convex function as f as long as ((*)))(1) = O. For instance, in the tradition of Q-learning, we can use:
f (x) (x + /2
(19)
5

---

Under review as a conference paper at ICLR 2023
where F and è constants. Its convex conjugate is f(z) (f)))) T, we can always pick a č that will ensure f(1) (1)) = 0, and ī does not affect the learning
Fx =2 For any
process. In summary, we have derived way minimize the first term (s, I| (r in equation 14directly using goal-conditioned Q-learning.
4.2 EBM FOR PREDICTING (AND CONTROLLING) THE FUTURE
In this section, we study how to maximize the goal likelihood term equation 14(b by defining an energy-hased model (EBM) to predict the future-state distribution for a policy, which the policy can turn optimize to control the future-state distribution. We start with the Bayes rule:
(+ (st (s, a |st)
P (s+ 80,0)
(20)
Ept (0) Px g)]
This relationship reflects that, if we define EBM for P s5) = ex, / (s+ where (s+)= exp a, st )dsda, then we can contrast the dynamics defined in equation with the marginal goal distribution defined in equation 13 In a slightly overloaded notation, we will now define a Q-function as Qo 9) g(s, a. g) log (g). For now, we shall assume that this
Q-function is separate from the one defined in equation 18 Rearranging equation 20 and setting
pt (g) to be (s+), we see the density ratio can be expressed as:
p (s+ (,)a) p++st)
Pr(s, a s+)
exp (xpQo(s,,,8))
(21)
(s,a bp- g)] Bpp(s) (exp Qo (s, g)g
Lemma 4.2 (Gradient of the noise-contrastive term in energy-based goal-reaching). Given the following definition for the logit of NCE-like binary classifier, with p= (g) p0):
Lo a, g, k) = Qo a, g) log E Qo a, g)l log k
(22)
The gradient of the negative NCE term in the density ratio estimation approaches zero as + oo:
d F pula (o)
k log (1 (1-
a, (.))))
k 0
dB
As for the positive classification term in NCE equation 9 we can make a similar argument that Ve log(1 + exp As(s, a, st ,k) k-100, (see appendix B.2). We have:
arg max (n.a) p- (o+)),a) (Qo [Qe(s, a,,ssss))) s log Ep (n) lexp Qo a, g)]]
(23)
To optimize the above objective on arbitrary behaviour data, we have to consider the fact that we do not have complete access the distribution of "positive samples" (s+ s, a), as sampling directly from this distribution requires on on-policy rollouts. Utilizing importance weights to learn from off- off-policy data while accounting for hindsight bias yields the following loss minimization instead:
Epp(s,a) { -11 - 7) Epp(' (s.a) 1Q0 a, s')] +Ept (g)p(s s.0)) kwn]w((s (s a, s' g) (o a, g)l
(24)
Learning single-step dynamics Q
Learning multi-step dynamics in
where wi(s, a, s') contrasts the goals that will likely be met by the agent beyond single step of dynamics p(s' a) in the future with other random goals that the agent has seen in the past:
exp Q(x a,
exp Q(s g)/lg)/11
w(s, a, 8',))
(25)
E Epp (m) ((p,
E Ept E. cxp Q (s', // A|]
Pushing down the likelihood the marginal
Pushing up the likelihood of the conditional
See Appendix B.3 for a full derivation.
4.3 DERIVING HER REWARDS
For condiconni Q-function Qu a, g). besides optimizing a Bellman residual as a regular Q(s, a) function would do, it has additional ofgree-of-freedone for the goal g which can be used to

---

Under review as a conference paper at ICLR 2023
define EBM that models the future- distribution. In such an EBM, the Q-function is used as the negative energy defined in equation 21 Combining the first term in the f f-divergence minimization part equation 18 and the single-step dynamics loss term in the EBM learning 24 gives us:
arg min E [a (e* (e.)))))))))))) [ (0ooo-""")(ss,,s
B (1- (1-7))0(ss,,,3 ( (26)
The crucial observation here is, given that f* a quadratic 19 and that there a stop gradient sign on px Q because it uses a target network, the loss above can be re- re-packaged into single quadratic loss of Bellman residuals for a specific reward thanks to the property of the dynamics defined in 4
arg min E (a(e (ea (a, Q
a(s.
st)) ((op"--Qo(ssa.st)))
(27)
2
T+
s' = st
r(s.a,s',st)
(28)
F,
s # st
have arrived at the reward function of HER THER (8, a, st). The remaining losses that this reward leaves out are the second term in f-divergence minimization part equation [8] and the second term in the EBM training
See Appendix B.4 for full derivation. Setting f= and B
equation 53 which we will analyze in the next section.
4.4 WHEN DOES BEHAVIOUR CLONING (BC) HELP?
We now proceed to answer the following question: when does BC help? Our idea is to identify the conditions for BC-like loss to emerge in the remaining losses unaccounted by the HER reward: a loss term in 18 about pushing Q-values of the current policy down, and the multi-step dynamics We may argue that, by not pushing the Q-values of the current policy down, the HER agent becomes more exploratory; by not pushing up the Q-values of the discounted future-state distribution beyond single step, the HER agent is encouraged reach goal sooner rather than later. Nevertheless, we shall see that combining those two remaining terms produces a BC-like term, which imitates the best actions based how much an action moves the agent closer to the goal.
term in EBM 54
We start with the following property of Boltzmann policies T(a 8.9) x exp @xpQ(s al. 2017, Schulman et al. 2017) (which we can apply thanks entropy regularization in
(Haarnoja
(a) (Q(s, = log Σ exp
H(T))
(29)
In equation 25 the Q-value of a particular action a is pushed up under the following condition:
exp expQ(x, a, (o) lexp (exp
a(xpp(s'a..g//1 g)/
< E, C(p 9 ()
(30)
((s,a
Both denominators on the left and right sides are averaged over all passible behavioral goals, so their values should be roughly the same. Moreover, we approximate the average-exponentee operation by using the max operation already computed in the backup operator (either exact (Mnih et al.) 2015) Van Hasselt al., 2016) or approximate (Lillicrap et al. 2015 Haarnoja et al., 2018)) and lower the threshold "hdm (compared to 7) accordingly. After those changes, the indicator function that decides whether the value of an action a should be pushed up becomes:
d a, s' g) 1 (exp Q(s a.
7dd
exp max Q(s d' 0)
(31)
Combining with the first term 29 we arrive at a BC-like loss 12 but with an indicator weighting:
Cram (Q) = E pa (o) (
( ((,
a, g) (Qo(s, log exp 99)]] (32)
Intuitively, this term imitates particular action when the value functions believe that this action can move the agent closer to the goal by at least - log Tnd steps. The idea of imitating the best actions is similar to self-imitation learning (SIL) (Oh et al.) 20188 Vinyals et al. 2019). but our algorithm|] operates in a (reward- free) goal-reaching setting, with the advantage function in SIL being replaced by the delta of reachability particular action can produce in getting closer to the goal.
7

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Hindsight Divergence Minimization
Given: Batch of data {(s, a, s',, st)}, where st is sampled via hindsight relabeling.
1: L MSE(Q( 8, a, s*) y maxe Q18', a', s),
7 a constant, default
2: [ [ -- 31 Thdm) ((s,a,s)
D Push up the values for reaching next states D Behaviour Cloning like loss
3: LBC = Q(s, 0, - log LA exp 0(s.,,87
4: f= 3. ( a. s*
max Q(s', a', s*) log (hdm) LBC
5: minimize L, update the target network Q
5 EXPERIMENTS
Figure 6: Intuitions about when HDM applies BC. From left to right: (a) the Maze environment with a goal the upper-left corner; (b) the Q-values learned through the converged policy. Lighter red means higher Q-value; (c) visualizing the actions (from the replay) that get imitated when conditioning on the goal and setting Tham = 0.95, with the background color reflecting how much an action moves the agent closcr to the goal based on the agent's 's own estimate; (d) Thim 0.85; (c) Thim 0.75. As we lower Thdm, the threshold - log "hum gets higher and fewer actions get imitated, with the remaining imitated actions more concentrated around the goal. HDM uses Q-learning to account for the worst while imitating the best during the goal-reaching process.
5.1 SSPERVUDER GOAL- REACHING SETUP
We consider the self-supervised goal-reaching setting for the environments described in GCSL (Ghosh et al. 2019). While the original HER (Andrychowicz et al. 2017) assumes that the agent has direct access the ground truth binary reward metric (which is used when relabeling is performed), we do not make such an assumption it can be unrealistic for real-world robot-learning (Lin et al. 2019). Instead, we simply use next-state relabeling to provide positive rewards. As seen Eq equation 28 positive reward is provided only when the relabeled hindsight goal is the immediate next state. The benefit of the next-state relabeling reward is that the training procedure is now completely self-supervised, and therefore provides a fair comparison to GCSL (Gosh et al. 2019.
Success Rate (%)
Four Rooms
Lunar Landar
Sawyer Push
Door Opening
Claw Manipulate
GCSL/ /
HBC HBC
78.27 #4.76
50.00 7.77
44.67 13.86
19.10 #5.97
16.80 #6.5
HER r = (0,1)
86.60 =4.22
39.30 6681
57.60 16.61
82.50 14.87
22.80 16.43
HER+ + SQL
88,50 #4.56
44.50
=9.6
57.20 16.3
84.70 +5.33
16.13 +8.43
HER r = (-1,0)
86.40 =5.11
50.80
4466
54.60 166.6
83.76 16.02
20.20 16.23
HER- HBC
82.90 6.24
35.33
4.55
52.63 18.05
76.44 1:.37
16.93 18.03
HDM (ours)
96.27 2.56
57.60
=7.21
66.00 15.13
88.60 44.63
27.89 6.46
Table 1:
Benchmark results of test-time success rates in self-supervised goal-reaching. We compare our method HDM with GCSL (Ghosh al. 2019). HER (Andrychowicz et al. (017) with two different types of rewards, SQL (Schulman et 2017) (Soft Q-Leaming) HER, and HER HBC. We find that HER can still outperforms GCSL/ / HBC, and a vanilla combination of HER HBC actually hurts performance. HDM selectively decides
on what to imitate and outperforms HER and HBC.
5.2 COMPARISONS AND ABLATIONS
As shown in Table 1. we compare HDM to the following baselines in terms of their goal-reaching abilities: GCSL GGosh et al. 2019) HBC (Ding et al. 2019), HER (Andrychowicz et al. 2017) with (0, 1) rewards, HER with Soft Q-Learning (SQL) (Shhulman et al. 2017). HER with (-1, 0)
rewards, and HER- + HBC. HER with (0, 1) reward gives the agent a reward of when the goal is reached and 0 otherwise; HER with (-1, reward gives reward of 0 when goal is reached and -1
otherwise. Those two types of rewards lead to different learning dynamics because the Q-function is
initialized to output values around 0. We also compare against HER SQL., because soft Q-learning
8

---

Under review as a conference paper at ICLR 2023
Normalized Gain over OCSL
Normalized Porformance Gain over GCSL
HOUIHOLLI HIR HER, HDM
Environments
Figure 7: Normalized performance gain over GCSL, calculated by normalizing the final performance difference between an algorithm and GCSL. We see that only HER with (-1.0) rewards and HDM consistently outperform GCSL, while HER + HBC performs worse than HER and sometimes GCSL as well.
known to improve the robustness of learning (Haamoja et al. 2018) HDM builds on top of HER with (-1, 0) rewards and adds BC-like loss with clipping condition equation 32 such that only the actions that move an agent closer to the goal get imitated (see Figure 66. Indeed, combining HER with HBC (which blindly imitates all actions in hindsight) produces worse results than not imitating at all and only resorting to value learning; HDM allows for better control over what to imitate.
The results in Table D and Figure show that HDM achieves the strongest performance on all environments, while reducing variances in success rates. Interestingly, there is no consensus best baseline algorithm, with all five algorithms achieving good results in some environments and subpar performance in others. In Figure 10 of the appendix, we ablate the additional dyper-parameters introduced by HDM: Nid equation 31 and the term in equation [4] Intuitively, Thdm controls the threshold that determines when an action is considered good enough for imitation, and , controls the trade-off between minimizing f f-divergence and maximizing future goal likelihood in equation [4]
The ablation shows that HDM outperforms HER and GCSL across a variety of Tadm and / values.
5.3 WHY DOES HINDSIGHT BC SOMETIMES FAIL?
Success Rate versus In Ag Change
In this section, aim to better understand whether the (under- whelming) performance hindsight BC had anything to do with our specific setting. We report an interesting metric that correlates with the performance of GCSL/ / HBC, which can be measured prior to training policy: initial ag (achieved-goal change ratio. We define the ag change ratio of π to be: the percentage of tra- jectories where the achieved goals initial states 8o are different from the achieved goals in final states 8T under æ, Most training signals are only created from ag changes, because they provide examples of how to rearrange an environment. We then define initial ag change ratio to be the ag change ratio of a random-acting policy To- Using notation from 28 it can be computed Egg- I-TER So, sT))-
In Charge
Figure 8: Success rate versus initial achieved-gon change ratio.
In Figure 8 we show the relationship between final success rates versus the initial ag (achieved-goe -goal) change ratio, across all environments. The performance of GCSL seems to be upper- bounded by a linear relationship between the two. This makes intuitive sense: a BC-style objective starts off cloning the initially random trajectories, so if initial ag change ratio is low, the policy would not learn to rearrange ag, compounding to low final performance. HER and HDM are able to surpass this upper ceiling likely because of the additional goal-likelihood term equation 14 besides imitation.
This finding suggests that in order to make goal-reaching easier, we should either modify the initial state distribution ((s) such that ag can be easily changed through random exploration (Florensa et al. 2017) (if the policy is training from scratch), or initialize BC from some high high-quality demonstrations
where ag does change (Ding et al. 2019 Nair al. 2018a Lynch et al. 2019).
CONCLUSION
This work presents unified goal-reaching objective that encompasses family of goal-conditioned RL. algorithms.. Our derivation illustrates the connection between hindsight goal relabeling, divergence minimization, and energy-based models. It reveals that there is largely unexplored design space:
we could potentially use other convex functions in f-divergence minimization (Nowozin et al. 2016 Ghasemipour al. 20220, and improve the optimization of the energy-based model (Du al. 2020 by going beyond NCE (Grathwohl et al., 2020). The primary limitation of our framework is that it does not account for exploration (Hafner et al.) 2020).. Eventually, would like to incorporate
empowerment (Klyubin et al. 2005] /Sekar et al. 2020 and input density exploration (Bellemare
et al. 2016 Pong et al.] 2019 into the framework.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. Conference on robot learning, pp. 1300-1313, PMLR, 2020.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01622 2022.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine. Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: challenge for embodied ai. arXiv preprint arXiv:2011.01973 2020.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901. 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems, pp.
2019.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.0800 2019.
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01331 2020.
Ben Eysenbach, Xinyang Geng, Sergey Levine, and Russ R Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. Advances in neural information processing systems, 33:14783-14795, 2020a.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning achieve goals via recursive classification. arXiv preprint wrXiv:2011.08900 2020b.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy- energy-based models. arXiv preprint arXiv:1611.03852. 2016.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning, pp. 482-495. PMLR, 2017.
Justin Fu, Katie Luo, and Sergey Levine, Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint arXiv:1710.112448 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation enror in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7518-7528, 2020.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34, 2021.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, pp. 1259-1277. PMLR, 2020.
Dibya Ghosh, Abhishek Gupta, Justin Fu, Ashwin Reddy, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals without reinforcement learning. ArXiv, abs/1912.06088, 2019.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Will Grathwohl, Kuan-Chieh Wang.
Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03265 2019.
Will Grathwohl, Kuan-Chieh Wang, förn-Henrik Jacobsen, David Duvenaud, and Richard Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference on Machine Learning, pp. 3732-3747. PMLR, 2020.
Michael U Gutmann and Aapo Hyvärinen. Noise-comtrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2), 2012.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLR, 2018
Danijar Hafner, Pedro A Ortega, Jimmy Ba, Thomas Parr, Karl Friston, and Nicolas Heess. Action and perception as divergence minimization. arXiv preprint arXiv:2009.01791, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IÊEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 2020.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1478):1771-1800, 2002.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural
information processing systems, 29, 2016.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. ArXiv, abs/2202.02005, 2021.
Michael Janner, Igor Mordatch, and Sergey Levine. Hamma--oodels Generative temporal difference learning for infinite-horizon prediction. Advances in Neural Information Processing Systems, 33: 1724-1735, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 2014.
11

---

Under review as a conference paper at ICLR 2023
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent-centric measure control. In 2005 ieee congress on evolutionary computation, volume 1, pp. 135. IEEE, 2005.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off off-policy distribution matching. arXiv preprint arXiv: 19X.:::99120000:: 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0299 52.709979,, 2015.
Xingyu Lin, Harjatin Singh Baweja, and David Held. Reinforcement learning without ground-truth state. arXiv preprint 0rXiv:1905.07880 2019.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07644 2020.
Corey Lynch, Mohi Khansari, Ted Xiao. Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play. In CoRI,, 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed repre- sentations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, 2013.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. Advances in neural information processing systems, 26, 2013.
Andriy Mnih and Yee Whye Teh. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the International Conference on Machine Learning, 2012.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-33 533,,33, 2015.
Ofir Nachum and Bo Dai. Reinforcement learning via rookhel-r duality. arXiv preprint trXiv:2001.018666 2020.
Ofrr Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019a.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans, Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074, 2019b.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on roboties and automation (ICRA), pp. 6292-6299. IEEE, 2018a.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018b.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11))5847-588) 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference Machine Learning, pp. 3878-3887. PMLR, 2018.
12

---

Under review as a conference paper at ICLR 2023
OpenAI OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, 1lge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D' Sa, Arthur Petron, Henrique P Pinto, et al. Asymmetric self- for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:21010048800 2021.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7750-7761. PMLR, 2020.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learn- ing: Challenging robotics environments and request for research. arXiv preprint arXiv: /802.0964, 2018.
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models; Model- free deep for model-based control. arXiv preprint arXiv: 00081, 2018
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering supf--upe reinforcement learning. arXiv preprint arXiv:/903. 930,,036,,, 2019.
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratin estimation. Advances in Neural Information Processing Systems, 33:4905-4916 2020.
Stephane Ross, Geoffrey Gordon, and Andrew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In AISTATS, 2011.
John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learming. arXiv preprint arXiv:1704.0m 1704.06440, 2017.
Ramanan Sekar, Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via supersuue world models. In International Conference on Machine Learning, pp. 8583-8599 PMLR, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. International conference on machine learning, pp. 387-395. PMLR, 2014.
David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299; 103535, 2021.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. arXiv-1807, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung. David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782) 2019.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint 18X.vv1811.1 11359, 2018
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pp. 681-688. Citeseer, 2011.
Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model as graph: Learning latent landmarks for planning. In International Conference on Machine Learning, pp. 1261 1-2620. PMLR, 2021.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. 2020.
Advances in Neural Information Processing Systems,
13

---

Under review as a conference paper at ICLR 2023
A NOTATIONS
o
Initial state distribution
p(s'1 s,a)
Environmental dynamics
(st 8,,a) Pa(s)
Discounted future state distribution under policy
State distribution (occupancy measure) visited by policy l
P"Q(s,a,g)
Transition operator in (goal-conditioned) Bellman backup. TD (s, a, Ppp(' s.0)x(a' (8',)) (Qo ('',,''',))) ,')
f
Convex function in /-divergence D.. Usually f(1) = 0.
f
Convex conjugate of function f
f())
Derivative of function
(s)
State-action visitation distribution the expert policy
(*)
State-action visitation distribution of the policy μ Note that ( a) (s))uaa s))
T(x)
Function used in the variational bound of f f-divergence, Also appears T(s, a) or [[s a, g) in the main text.
pt(gg
The behaviour goal distribution assumed be given apriori by the environment.
pt (s*)
The marginal hindsight goal distribution of given dataset replay, where pt (s+) (s.)(s.))pp] (s) s.)
*(a ss)
The goal-conditioned policy marginalized over the behavioral goal distribution (a s) = Ep (o) [T s,
Pa(s,a 1
The sttte-action visitation distribution of goal-conditioned policy æ when conditioned on the behavioral goal g
E(x)
Energy function of an EBM: po (2r exp( - Eg (x))
Z(0)
The partition function in an EBM Z(0) Jx exp( exp(-Eo(z))ddz:
(x))
Sigmoid function a (xx 1/(1 + exp(-z)).
(r())
Noise distribution Noise Contrastive Estimation (NCE)
k
The number of times noise samples are sampled more frequently than true data samples in NCE.
Ao
The logit of the positive sample classification loss in NCE.
a(5,
conditional EBM p= (s, a s+) exp s 3/ (s+).
Zx(
The partition function of a conditional EBM Pa (s, a st); (s(s*) ====)==sAAA exp g(s, a, s ddsdd.
H(T))
Entropy policy across (replay) states and goals H(x)) s.g)].
Ep(s)p (g) log (a
r(s.a)
The (learned) reward in traditional RL settings without goal- conditioning.
as
The (Ieared) reward in goal-conditioned divergence minimiza- tion.
a8, (HER a, st)
The (generalized) reward used in HER-style multi-goal RI.
The reward function used in HER. equals 0 when s' s and when s' + st. Also denoted as THER( s 8', +).
14

---

Under review as a conference paper at ICLR 2023
B PROOFS
B.1 MAIN LEMMAS
Lemma B.1 (Online-to-ofline transformation for goal reaching). Given a goal-conditioeed policy (a s.g), its corresponding Q-function QT a, 9), and arbitrary state-action visitation distribution
Pu(s, a) of another policy p(a 8), the expected temporal difference for online rollouts under is:
(p -7 -7.00---PP-00 -7,pp"")ss = (f - PP=
Proof of Lemma
4.1
E
(s,
Eple)or(ssalo) (0 (l
= (1 (1-
EE plo)e (+1g)
(s, - 71E pla')s.a) Q (5'','''g)
t=
a(ala.g)
(1
E
p(g) E
[Q"
E p= (elg)
107(s.a,0))
p(a)
(=(g) =(alsgg)
mal3.0)
= (1 -
1Q" []]]
= (1 - r) Σ
A p(g)
1Q (s, [s, - - + p(g)
1Q" a.g)
pa(e)
p (s)
<(als.g)
n(als,g)
0
(1 7) Σ
> M plo)) plg)e (s, 3 p() ("
<(a'ls'
= Eetalo (s. ā.g) - (ep d9
Lemma B.2 (Gradient of the coise-contrastive term in energy-based goal-reaching). Given the following definition for the logit of NCE-like binary classifier, with pt (g) pt()))
Lo a, 9, k) Qa (s, a, g) - log Ep- (3) lexp Qa 8, a, g)] log k:
(33)
The gradient of the negative NCE term in the density ratio estimation approaches zero as + co:
- E3 (p.
d
k log ( (1-o(Ao(s,a,g, k=209, 0
d
Proof of Lemma
B.2 is the sigmoid function, and Zo(s, ( Qo(s,
- (-ooooss, 1 exp (s, 1 1 expe(e(((ss log Zo(s, a))/k (34)
Plugging in the above into the loss and taking the gradient:
d
exp eex Qp(s,,,))) 9,
+1)]
E (a.a)
k. log
k ZZ
(35)
d8
p! (g)
k log (Hexp (Qo(s, log Zo(s,a) Zo( + +1)]
E ppu(saa)
(36)
do
pt())
d 1/k Q (8, + do (Qo(s,a.0) - log Zo (s,
exp (xpooss...0/zz(1..
= -pp((s.)) p² p'())
(37)
00
E (s.0a)
E Ept Qo d0 (o(s,, a, g) log Ept (g) lexp Qo (s, 31]]
exp
(38)
15

---

Under review as a conference paper at ICLR 2023
The first term inside the expectation of Pn (s(s,00 a):
B (g) d db Qo(s, a,
(39)
ppt (m) eexp (s 3)
The second term:
Rp exp Q0 a 8)]
d log Epp (pppt/laa loxp Qo(s,
(40)
Ep (7) exp (o s, a, 9] d0
d
1 Qo (8, a,g)
(41)
Ep (0) exp Co 8, a, 9)
d8
p (s) lexp Qo a, 9) d0 Qo(5,a,7)
(42)
(g) exp (s 0,7)
The two terms cancel out and yield a gradient of
Lemma B.3 apriori the behavioral goal distribution to be p" (g) == P a, s* sdddd. And assuming thar the state-action distribution of n marginalized over behavioral goals Ep' (g) Ipx (s, a g)] is the same as Pu(s,a) a), which means æ stays in the same state-action visitation distribution as μ.. Then: on trajectories generated by the point-wise mutual information (PMI) between state-action pair (8, a) and a future state g is given by the Q-function at convergence Q":
Q-functions estimate PMI on given trajectories). Given that we set
PMI((s.a).g) = Q (s, g) log Bot (a) lexp Q" (s, a, g)l log(f + (xPTQT (")(s,a,g))
Proof of Lemma B.3
We first start with the optimal I in the f-divergence bound equation 5 applied to the special case of the function f being a quadratic equation 19.
T =f f(((x)//(x))
(43)
(1(1) 1-r
(44)
Combining the two, get:
F -F
(45)
Now applying this identity to the f-divergence minimization problem in equation have set rss, a, 9) aT(s,,,9) in our derivation):
15
(note that we
Px (s (,9 p*())= g)
Pa s+) pp(s, a) s, - - -Q- - -p==="(ssss
(46)
F (AP* -"""""""""""
(47)
We now take the relationship in equation 21
(s (=(5,a) a)
exp Q7 a.,5
(48)
Ept
(2) Ept(w)
lexp (8,
Using this substitution, we arrive at:
p* (g)px(s,a)
(s, a, s*)
= F+ (TP" -0")
(49)
(s a)p (g 0) E Ept(() lexp Q (s, a, g)
Swapping the nominator and denominator and applying the assumption that pr a) (s(ssa):
Pa(s, a) (g s, E, (o) [exp Q* (s,
(50)
as (g) exp (s, a,
F (1PT00==
Taking the log on both sides, we get the following expression of PMT
pí (gl s,a)
log
a. E (x (s..))))) logg (yp= (1)0((((PPP(=00)xs_000))
p= (g)
16

---

Under review as a conference paper at ICLR 2023
B.2
NCE LOSSES
We now complete the derivation of the NCE losses in section 4.2 by illustrating that the positive classification loss of NCE reduces to an InfoNCE (Van den Oord et al. 2018 loss equation
equation 23 under our definition of the logit equation 22 More specifically, given that we already have Lemma B.2
22
we only need to show:
d log(1 + exp a k))
R-000,
We simply follow the definition of Ag equation equation 22 and take the gradient the above:
exp A,(s, k) d exp Do 00, k) do st
E p lexp a (e
1 + Soop(-Aoss,,,s--00 (Volo(s,a,st)
E Bpt Qo(s,
VoQo(s,a, Ept(o) exp Qo Voloo, Ep (g) cxp Qo a. a.g)
1 k (g) exp 20(s,a,9)
exp Qo st)
As k + we see that the gradient approaches 0 because the scalar on the left approaches 0.
Combining the above result about a part the positive classification loss in NCE with the Lemma in B.2 which deals with the negative classification loss in NCE, we can arrive at the combined NCE loss its limit is oo, as pointed out in equation of the main text:
arg max (p.aa)) E (st)s,)) ao(s,a,s*t -log E Ep (s a, g)] Q
As mentioned, this is roughly equivalent to the InfoNCE loss (Van den Oord et al. 2018), further validating our analysis so far.
B.3 EBM LOSSES
To optimize (p,. E (p [Q0(s, 8 - log (pt(g) jexp Qa 4,9]]] from an arbitrary dataset of behaviors Ps(s, a, st), we can easily that the problem lies in accessing p (s* s, a): we do not have complete access to the distribution of "positive samples", as sampling directly from p (s+ s, s,a) requires on-policy rollouts. But this can be resolved by using importance weights and Equation equation 4 rewrite E
(et (Qo (Q0
st]]
P (g ss',a') (o(s,a0)))))))
71 T PR (s.a) (8. s')] + pt (s.p
(o Pu(s,a)
E Pp (sa))(a))s))
(51)
p=(g)
Above, we have introduced a new notation म to address the following issue with p + [s, a): while is goal-conditioned policy, p# (s- [s, a) not conditioned problem was encountered (but ignored) C-Learning (Eysenbach et al.) 2020b), which simply assumed that the multi-goal policy was aposteriorí conditioned on the same future state that got sampled from this policy in the first place (a contradiction). To avoid this problem, we define ((a s) Ept [T 8, g)] and assume that p (s+|s. (ts, a) is sampled under a(a 8) beyond single step. We package all the EBM training losses into the following:
goal apriori. A similar
E (p. (- 7)
(Qos,,,
(52)
p 8 a')'(((a s') pt p+(g)((a' s',g) Qo((s ()0(s...0))
Bpa(s,)) log E,t lexp ao(s,a,g) -B ((e
(53)
=(a'l'',))
(g)
To decompose the importance weight, note that the policy is trying maximize the Q-values
under the entropy constraint in equation [4] resulting in Boltzmann policy (Haarnoja et al.
2017 Schulman al. 2017 Haarojja al. 2018): arg max, Ex(els B H(x))
17

---

Under review as a conference paper at ICLR 2023
exp a. /)/ exp ((s, 9. Minimizing equation 53 is equivalent to minimizing the follow-
ing loss, with 1 (.) being the stop-gratd hoopgrradeent sign:
exp (( 0.
E Ept exp exp Q(' (.6',,,7)//40 Q0((s, ( (54)
Σ (('
eu(s,a)
1
E (z) exp (s, 0,7)
(g)
p(slss,))
Proof of equuaion 54 Let Webm (s','', .) denote the importance weight:
W'ebm (s', d' p (g s' a')
#(a' |s)
(55)
p (g)
aaa s',
Firstly, because of the entropy constraint in equation Q-function: arg max, (aaalss [Q(s, a, g)) H(TT) exp (s, a, g)/ -/ exp ((s,a,))
14 we have Boltzmann policy defined on the
Taa 8,g) x exp (s, a,
(56)
Secondly, we note that the first term in importance ratio can be estimated from equation 21 Thus we have:
exp (')'
E (o) exp Q a',)) Σ. E p exp ((8' a.g)
Σx exp ((s',a,9)
Wehm (s',
(p) Q(s, .).g)
expQ (s', a'
(57)
Exp Q(. Σn E,' exp (ss a.g)
(58)
Taking the gradient equation equation 53 inside the expectation of (sans
Ep (g) (exp Q(s, Ve
ylEp (g) exp ( ()
(59)
Epp (m) exp (s, (,g)
Σx R.pt ( lexp ( a, g)
Dividing both the nominator and denominator of the right-hand side by 1/|AA,, putting the expectation pp back in, and utilizing the stop-gradient sign 1 (-), we arrive at the loss:
{
exp Q().a.g)
E. exp ((sp@(ss,,g)))) g)9// E, G Exp exp Q(s' (.9)/41441
10(s, (o(s,a.g)
E Pu((sa)) nt (g)
Ep (g) lexp a(s
p(a(s,,))
B.4 DERIVING HER REWARDS
The purpose of this section is to derive the reward function used in HER from the equation 26
arg min E (a.a)))p( (s)))) (s.0) [ (
B (1-7)Qo(s,a,s))
Q
Recall that we have defined pst
(s+ ss,a) in equation 4
+> l p(s' s,a)u(a' s')pt (st s a d''dd'
P (s+ s) (1 (ppsst s,)
SxA
And that have defined quadratic form of f in equation 19 (with and being constants):
f*(z) = (x /2)
Using the dynamics to expand the expectation and applying the choice of f being a quadratic, the loss becomes:
arg Bin pssatt 7) [2( (xp"Q- - -o)ss,,
+ E (aaaaallddssaaaaaaaaaaaaata (s | [ (Tp" Qo)( -1) 27
18

---

Under review as a conference paper at ICLR 2023
Assuming that there is stop gradient sign P*Q because of the target network, can rewrite the above as one single quadratic and see that the gradient of the above loss w.r.t Q is equivalent to the gradient of the following squared Bellman residual:
arg min R (olts, (s l..a) 2 (xP ]
where the reward function r(s, a, s's is:
7 + 3,
8' st
F,
s' + s*
C ABLATIONS AND SYPEERPARRMMTERS
In this section, include additional ablations and hyper- syper-parameters. We first describe the goal- reaching environments in more details in Figure 9 We use the following thresholds (for Fuclidean norms) for determining success: [0.08, 0.08, 0.05, 0.05, 0.1], which are tight thresholds based visualizations of the environments. We use the same network architecture, sampling and optimization schedules for all the methods, described in Table 2 As for "HDM. we set be 0.85 in Four Rooms and Lunar Lander, 0.5 in Sawyer Push and Claw Manipulate, and 0.4 for Door Opening. Ablation on this hyper-parameter can be found in Figure I0 For next state relabeling ratio, we set the default to be 0.2, increase it to 0.5 in Lunur Lander, and 0.6 in Sawyer Push and Door Opening. For the soft- Q-leanning (Schulman et al. 2017) HER baseline, we set the temperature parameter be 0.2, which we have tound to empirically perform well.
Figure 9: Goal-reaching environments from GCSL Ghosh al. 2019) that consider in this paper: reaching a goal location in Four Rooms, landing at a goal location in Lunar Lander, pushing a puck goal location in Sawyer Push, opening the door goal angle in Door Open (Nair al. 2018b). turning a valve to goal orientation in Claw Manipulate (Ahn et al. 2020)
Ablation HDM gamma Four Roams
Ablation HDM Sawyer Push
Ablation HDM Beta Four Rooms
Abletion HDM Beta Sawyer Push
Percoagge
HER GCSL
HDM
HDM Gomma
Ganona
HDM Bold
HDM Bats
Figure 10: Ablation studies HDM Gamma and Beta. HDM Gamma refers to hdm in equation 31 and HDM Beta refers to the term in equation 14 The orange line and the blue line denote HER and GCSL baseline performance. See Section 5.2 for further discussion of these results.
19

---

Under review as a conference paper at ICLR 2023
Table 2: Hyper-parameter:
Parameters
Value
Optimizer
Adam (Kingma & Ba
2014)
Number of hidden layers (all networks)
2
Number of hidden units per layer
[400, 300]
Fujimoto et al.
2018
Non-linearity
ReLUU
Polyak for target network
0.995
Target update interval
10
Ratio between env vs optimization steps
Initial random trajectories
200
Hindsight relabelling ratio
0.85
Update every # of steps in environment
50
Next state relabelling ratio
0.2
Learning rate
5.e-4
Batch size
256
20


Reviews:
[{"review_id": "uDy90mopcy", "reviewer": "Reviewer_4ceU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors present a unification of goal-conditioned behavior cloning/supervised learning and hindsight relabeling methods (e.g., HER) under the framework of divergence minimization. They show that HER is a special case of a divergence minimization when considered under a goal-conditioned Q-learning approach (when Q is treated as an EBM).\n\nTo this end, they propose a novel objective that aims to account for and balance the key elements required in goal-conditioned RL, and show promising results in a suite of standard tasks.\n\nThey further provide intuition about how excluding some of these elements (e.g., lack of goal-likelihood in the optimization) results in sub-optimal performance, and use this framework as a means to explain the poorer performance of pre-existing approaches.", "summary_of_the_review": "Overall, I believe this paper merits an acceptance. I think the work sheds interesting light on GCRL, particularly where approaches like HER fit in, and the empirical results seem to largely back up the intuitions presented. As a result, I believe the paper is of interest to the community, and is novel enough in its analysis to merit acceptance at a major conference.\n\nI have some issues with the presentation and experiments that I have listed in the Weaknesses section, and would be grateful to hear from the authors about addressing these.", "strength_and_weaknesses": "Strengths:\n* The unifying framework appears useful, and I found the unification of HER under a goal-conditioned RL framework interesting, and appears to provide insight into how these methods may work on a more theoretical level.\n* The authors provide empirical evidence that their approach HDM, which balances goal reaching and state-action matching, outperforms prior methods which don't explicitly manage this effectively.\n\nWeaknesses:\n* While the paper is easy to follow in sections, I found the overall narrative a bit unclear. For instance, Section 3 was quite hard to follow, as quantities and PGMs/distributions are introduced with not much intuition, for instance Eqs 10,11 and 12. Explaining the quantities in Eq 10 and 11, and why 12 gives HBC/GCSL would help here. Similarly, when the authors present their approach in Eq 14, the divergence term is flipped, but no explanation as to why this is done appears to be provided. Another case is when deriving the HER objective under their f-divergence formulation; they seem to arbitrarily combine the 1-step part of the EBM formulation with part of the f-divergence term, but don't give explanation why replacing the multi-step part of the EBM objective like this is fine? I also was a bit confused by the inclusion of 4.3 (HER derivation) and 4.4 (BC analysis) in section 4; it might make more sense to separate them (e.g., Section 4 being more focused on just the creation of the optimizable objective, and another section showing how their framework can unify these approaches).\n* I wasn't entirely convinced by some of the ablations. For instance, in the appendix the authors ablate values of Beta and Gamma; wouldn't we expect some extreme settings to recover the performance of HBC/GCSL? If not, why?\n* Nits:\n   * The term Bellman Residual is somewhat confusing, as it is overloaded with Bellman Residual learning [1]; perhaps using Bellman Error would be sufficient here, as I believe the authors explicitly state they put a stop gradient on the target values.\n   * How many seeds are used in the experiments?\n\n[1] Residual algorithms: Reinforcement learning with function approximation, Baird, ICML 1995", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i1gKmJ9R1g", "reviewer": "Reviewer_aKS5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyzes hindsight experience replay (HER) through divergence minimization with energy based models. With this analysis, it presents an approach that combines HER with behavioral cloning (BC) by only imitating actions in the dataset if they move the agent a certain amount towards the goal.\n\nThe paper concludes by comparing this adaptive HER + BC technique (termed hindsight divergence minimization, HDM) to HER, behavioral cloning and a more straightforward implementation of HER +BC.\nThe paper concludes with a hypothesis that BC-based techniques work better if the dataset used for learning has a higher proportion of achieved goals in its trajectories, and validates this hypothesis via experiments.", "summary_of_the_review": "The paper offers an interesting perspective on HER, and presents an approach to adaptively improve performance by incorporating BC into the objective if it is helping the agent learn. However, there might be related work that is not addressed in the paper, and might perhaps need to be compared to.", "strength_and_weaknesses": "### Strengths\n* This paper conducts a study of why hindsight replay helps when learning in goal-conditioned RL from the perspective of divergence minimization.\n* This study leads the paper to propose an adaptive algorithm that can take advantage of high quality demonstrations.\n* The perspective of goal-conditioned RL as a form of divergence minimization could be interesting.\n\n### Questions and Weaknesses:\n* The paper does not have a consolidated related works section. Are there other papers that aim to address goal-conditioned RL via a divergence minimization perspective? For example, it seems that the paper should compare to [1,2] in their exposition as well as their experiments.\n* While the rigor in section 4 is to be appreciated, it takes away from ease of understanding and clarity.\n* Section 5.3 was unclear on what _achieved-goal_ means. If that section essentially points out that GCSL works better in situations where the trajectories in the dataset get to the goal efficiently, then it is unclear why this is an important insight. BC with expert trajectories is more likely to work better. A better comparison would have been to also plot HER + HBC to show that these sorts of trajectories do not hard HER learning as much.\n* How do the results in Table 1 differ from the results in Figure 7?\n\n### References:\n[1] Ma, Y.J., Yan, J., Jayaraman, D. and Bastani, O., 2022. How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via $ f $-Advantage Regression. arXiv preprint arXiv:2206.03023.\n\n[2] Durugkar, I., Tec, M., Niekum, S. and Stone, P., 2021. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 34, pp.8622-8636.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cnklFpf-QE9", "reviewer": "Reviewer_CKyn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for goal-conditioned RL that combines three terms: (1) goal-conditioned behavioral cloning, (2) maximizing the likelihood of the goal, and (3) an exploration term. The paper proposes to use a different f-divergence for optimizing the first term, which ends up being done using a DICE-like TD update. Empirically, the proposed method outperforms baselines on a goal-reaching benchmark from prior work.", "summary_of_the_review": "My large number of concerns about the paper compel me to vote to reject it. I do think there might be some really interesting ideas in the TD-like method for minimizing the f-divergence. For example, might that allow a GCSL/HBC-style method to perform off-policy learning? ", "strength_and_weaknesses": "Strengths\n* The empirical results seem quite strong.\n\nWeaknesses\n* I am unsure about the motivation for the paper. The introduction seems to argue that there aren't any good ways of defining the goal-conditioned RL problem without reward engineering, but I'm not sure this is true (see below). The proposed objective is a combination of a \"goal likelihood term\" (which is the same as or quite similar to uncited prior work [5, 6, 8]) and an f-divergence term (from prior work, and cited). Most of the paper's contributions seem to be about analyzing the f-divergence term, but it is unclear why that term is needed at all: directly maximizing the goal likelihood term would yield the optimal goal reaching policy. Note that prior work has even considered maximizing the combination of the goal likelihood term and the f-divergence term, albeit with a different choice of f-divergence [9].\n* I am unsure about some of the claims in the introduction (see below)\n* I am unsure about certain parts of the math. It's possible they are correct and I was just confused by the notation (see details below).\n* Many parts of the paper were difficult to follow (see below).\n* The proposed method seems quite complex.\n\nDetailed comments\n* \"this connection between imitation and hindsight relabeling is not well understood\" -- I'm not sure this is true. There's been a fair bit of analysis of this in prior work (e.g., [1, 2, 3]).\n* \"reward function in hindsight experience replay\" -- Where is this shown? The HER paper uses a sparse reward on human-specified coordinates.\n* \"without reward engineering\" -- All four of these papers *do* require reward engineering, in the form of specifying a distance metric and (for some) specifying coordinates of interest. Consider referring to prior work that actually does learning without reward engineering [5, 6, 7].\n* \"it is unclear how to write down a well-defined objective for goal-conditioned policies\" -- I don't think this is true. See, e.g., [5, 6].\n* \"known as contrastive divergence\" -- I would recommend *not* referring to this as contrastive divergence, which typically refers to a specific approximate algorithm for fitting EBMs (instead of generating true samples from the model, it approximates the samples using a single gradient step).\n* \"NCE gives EBM a more tractable way to maximize data likelihood\" -- I didn't understand this. Would it be possible to elaborate?\n* Eq 10, Eq 11 -- I found these equations and the surrounding discussion confusing. Are these distributions supposed to be the same?\n* Eq 12 -- There seems to be a \"slight-of-hand\" here, where the future state is reinterpreted as the commanded goal. I'd recommend clarifying this by explicitly writing down the distributions for the relevant random variables ($g$, $s^+$).\n* Eq. 15 -- I'd recommend making sure the arguments to the f-divergence have the same arguments; currently, one has $g$ and one has $s^+$.\n* Eq 15 -- How does this relate to Eq 12? It seems like the direction of the KL has been silently reversed\n* Eq 15 -- Is this f-divergence being optimized? If so, w.r.t. what components?\n* Eq 16 -- How is $\\rho_\\pi(s, a, \\mid g)$ defined? Is this equal to $\\rho_{\\pi(a \\mid s)}(s, a \\mid s^+=g)$?\n* Lemma 4.1 -- What reward function is used for defining $Q^\\pi(s, a, g)$?\n* \"now define a Q-function as ... separate from the one defined in Eq. 18\" -- I found this confusing. I'd recommend using the standard convention of defining Q-function as expected discounted returns, and using a different variable for this.\n* Lemma 4.2 -- I found this hard to parse. I'd recommend adding scaffolding: what are the main aims of the result, what will this result be used to show?\n* Eq. 25: How is this similar to and different from [6]?\n* Eq 25 -- Can this be re-expressed without the Q-functions, directly in terms of the environment dynamics?\n* Eq. 28 -- I'm not sure $s' = s^+$ is well defined for continuous settings.\n* \"EBM training equation 53\" -- Broken reference?\n* Fig 8 -- I was unable to read the text in this figure.\n* Table 1 -- How are the numbers for GCSL computed? [10] reports much higher numbers for a GCSL-style method on the door task (This paper reports GCSL = 19% and the proposed method gets 87%, but [10] reports that a GCSL-style method gets 95.8%).\n* code, scripts folder -- Why are different random seeds used for different environments.\n* logger.py, L480 -- I'd recommend not including expletives in open source code.\n\n[1] https://proceedings.neurips.cc/paper/2019/file/3891b14b5d8cce2fdd8dcdb4ded28f6d-Paper.pdf\n\n[2] http://proceedings.mlr.press/v130/tang21b/tang21b.pdf\n\n[3] https://proceedings.neurips.cc/paper/2020/file/a97da629b098b75c294dffdc3e463904-Paper.pdf\n\n[4] https://arxiv.org/pdf/1912.06088.pdf\n\n[5] https://arxiv.org/pdf/2101.07123.pdf\n\n[6] https://arxiv.org/pdf/2011.08909.pdf\n\n[7] https://arxiv.org/pdf/1905.07866.pdf\n\n[8] https://arxiv.org/abs/2104.10190\n\n[9] https://arxiv.org/pdf/2206.07568.pdf\n\n[10] https://arxiv.org/pdf/2112.10751.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-tVhuGlk-K", "reviewer": "Reviewer_jwR4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper casts the problem of hindsight goal relabeling into a divergence minimization problem.\nThe authors propose a simple loss form to optimize and provide an algorithm to efficiently mix behavior cloning and RL in the goal relabelling procedure. \n\nTo do so, the authors first summarize the many mathematical tools they used in the paper (Rl, Divergence Minimization, EBM). They en combine those concepts to finally explore how to maximize this objective.  Their derivations also allow the author to see when BC may help improve the policy (as some trajectories may be sub-optimal).\n\nFinally, the authors perform a short empirical study over classic goal relabelling tasks and empirically show that their approach consistently outperforms other goal relabelling methods.", "summary_of_the_review": "I think this paper can be an exciting piece of work that completes the goal of relabelling the framework.\nHowever, the lack of clarity and some issues in the experimental results prevent me from being confident in my assessment. There are still many points that may remain obscure to me. Besides, the cluttered notation was really painful and prevented me from digging much into the mathematical details. Therefore, I am leaning to a borderline accept with little confidence\n\n", "strength_and_weaknesses": "The main strength of this paper is to formalize many algorithms into a single consistent framework and draw links between the methods. It also results in a clear objective and a practical algorithm. Unfortunately, I have difficulty grasping clear intuitions by following mathematical explanations. Too many mathematical references are interleaved in a few steps, and the notation sometimes feels too complex with too many nuances. For instance, Section 3 is a good example; there are many mental jumps to reach the final objective. On a single page, there are 4 figures; the authors refer to 4 different algorithms and introduce 5 equations with hard-to-parse notations $p_\\nu$ vs. $\\rho_\\nu$). I believe the authors wanted to be too comprehensive and highly rigorous, but it sometimes makes things too dense.\n\nThe final empirical results are also quite promising. However, I would recommend the authors provide more experimental details to be fully convincing (The appendix is a bit short on the matter.) First, I am not sure whether the number of seeds was provided. Then, training curves would be a nice add-on. Similarly, it is unclear whether the authors reproduced the results from other algorithms or whether they reported the scores (hence having potentially different training hyperparameters). Finally, I would have enjoyed more discussions on the hyperparameters (not only that they outperform classic methods on an extensive range). Idem testing the algorithm with different r and beta, for example, (-1,0) vs. (0, 1). What are the extreme case, how should they be set, etc.? I understand that the paper's primary focus is not on the empirical sections, but the paper could be greatly strengthened. The comparisons between the different baselines could be made more comprehensive. A simple table to highlight core elements (reward, clipping, etc.) would be helpful\n\nThen, I have a few other remarks:\n - In the algorithm, one step-q-learning seems like a significant constraint (l1), especially with a long horizon. Why not have a lambda/n-step return\n- How n-step return would behave with $\\gamma_{hdm}$ \n- Would it make sense to use DQfD[1] instead of pure BC? \n- There is no Bellman residual error here, it is only a TD error\n\n[1] Hester, Todd, et al. \"Deep q-learning from demonstrations.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject