Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
Authors: Vivek Myers
Bill Zheng
Benjamin Eysenbach
Sergey Levine
Venue: NeurIPS 2025 poster ()
Keywords: Goal-conditioned reinforcement learning
metric learning
Primary area: reinforcement_learning
TL;DR: We show how to learn representations of temporal distances that exploit quasimetric architectures in offline GCRL.

Abstract: Approaches for goal-conditioned reinforcement learning (GCRL) often use
learned state representations to extract goal-reaching policies. Two
frameworks for representation structure have yielded particularly
effective GCRL algorithms: (1) *contrastive representations*, in which
methods learn "successor features" with a contrastive objective that
performs inference over future outcomes, and (2) *temporal distances*,
which link the (quasimetric) distance in representation space to the
transit time from states to goals. We propose an approach that unifies
these two frameworks, using the structure of a quasimetric
representation space (triangle inequality) with the right additional
constraints to learn successor representations that enable optimal
goal-reaching. Unlike past work, our approach is able to exploit a
**quasimetric** distance parameterization to learn **optimal**
goal-reaching distances, even with **suboptimal** data and in
**stochastic** environments. This gives us the best of both worlds: we
retain the stability and long-horizon capabilities of Monte Carlo
contrastive RL methods, while getting the free stitching capabilities of
quasimetric network parameterizations. On existing offline GCRL
benchmarks, our representation learning objective improves performance
on stitching tasks where methods based on contrastive learning struggle,
and on noisy, high-dimensional environments where methods based on
quasimetric networks struggle.

Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
Vivek Myers
Bill Chunyuan Zheng
Benjamin Eysenbach"
Sergey Levine
UC Berkeley
Princeton University
Abstract
Approaches for goal-conditioned reinforcement learning (GCRL) often learned state representations to extract goal-reaching policies. Two frameworks for rep- resentation structure have yielded particularly effective GCRL algorithms: (1) contrastive representations, in which methods learn "successor features" " with a contrastive objective that performs inference over future outcomes, and (2) tem- poral distances, which link the (quasimetrie) distance in representation space to the transit time from states goals. We propose an approach that unifies these two frameworks, using the structure of quasimetric representation space (triangle inequality) with the right additional constraints to learn successor representations that enable optimal goal-reaching. Unlike past work, our approach is able to exploit quasimetric distance parameterization learn optimal goal-reaching distances, even with suboptimal data and in stochastic environments. This gives us the best of both worlds: we retain the stability and long-horizon capabilities Monte Carlo contrastive RL methods, while getting the free stitching capabilities of quasimetric network parameterizations. On existing offline GCRL benchmarks, our representation learning objective improves performance on stitching tasks where methods based on contrastive learning struggle, and on noisy, high-dimensionss environments where methods based quasimetric networks struggle.
Introduction
Learning temporal distances lies at the heart of many important problems in both control theory and reinforcement learning. In control theory, such distances form important Lyapunov functions [1] and control barrier functions [2], and are at the core of reachability analysis [3] and safety filtering [4] In Reinforcement Learning (RL), such distances are important not just for safe RL [5], but also for forming value functions tasks ranging from navigation [6] combinatorial reasoning [7] to
robotic manipulation [8, 9]. Ideally, these learned distances have two important properties: (i) they can encode paths that are shorter than those demonstrated in the data (i.c., stitching); and (ii) they can capture long-horizon distances with low variance.
Current methods for learning temporal distances typically only achieve one of these properties. Methods based on Q-learning [10, 11, 12] stitch trajectories with Temporal Difference (TD) updates find shortest paths, but often produce compounding errors that make it challenging to apply to long- horizon tasks [13]. Monte Carlo methods [14, 15] can directly learn goal-reaching value functions, which can connected to temporal distances [16], but their ability to find shortest paths remains limited. Methods based on learning quasimetric geometry [17, 18, 16], which impose a triangle inequality over distances as an architectural invariance, find shortest paths and don't 't require dynamic programming with compounding errors, but fail in stochastic settings and/or when learning from oolloo (suboptimal) data.
Website and code: https://tmmt website. gi tithub.io
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Learn temporal distance representations
Extract goal-reaching policy
temporal (T) invariance
((w)
a1)
(s)
vggg
((8 (2)
(
action (I)
Φ 3/
invariance
(0s')
Φ 03)
d (a))
d + d +
min, d -
time-condrastive MC. loss ((3=))) predicts 6()),))
triangle inequality with quasimetric architecture
T()) 02
(P-invariance)
Figure 1: (Left) TMD learns temporal distance da that satisfies the triangle inequality and action invariance. It does this by minimizing the distance between the learned distance and the distance between the successor features of the states and actions in the dataset. (Right) The learned distance is used to extract a goal-conditioned policy.
The aim of this paper is to build method for learning temporal distances that retains the long-horizon estimation capabilities of Monte Carlo methods but nonetheless able to compute shortest paths. We take an invariance perspective to do this. Temporal distances satisfy various invariance properties. Because they are value functions, they satisfy the Bellman equations. Prior work has also shown that they satisfy the triangle inequality, even in stochastic settings [18, 16]. The triangle inequality, also form of invariance [19], is powerful because it lets us architecturally winnow down the hypothesis space of temporal distances by only considering neural network architectures that satisfy the triangle inequality [17,20 20].
Importantly, the fact that temporal distances satisfy the triangle inequality holds for any temporal distance, including both optimal temporal distances and those learned by Monte Carlo methods. This raises important question: might there be an additional invariance that is satisfied by optimal temporal distances, but not those learned by Monte Carlo methods? Identifying such invariance properties that would enable to use Monte Carlo methods to architecturally winnow the hypothesis space, and then use this additional invariance property to identify optimal temporal distances within that space.
Our key contribution a method for learning optimal goal-reaching distances that combines the long- horizon. probabilistic inference of Monte Carlo temporal distances with the optimality and stitching capabilities of quasimetric architectures. We use a quasimetric architecture that imposes the triangle inequality as architectural constraint, combined with two additional invariance properties that apply transition level. When these invariances are enforced as constraints on features learned with Monte Carlo estimation, they impose structure roughly similar to the Bellman optimality equations across the space goal-conditioned (Kroenecker Delta) reward functions.
We translate these invariance properties into a practical method for Coal-Conditioned Reinforcement Learning (GCRL) that call Temporal Metric Distillation (TMD). To the best of our knowledge, TMD is the first GCRL method that uses quasimetric value parameterization to implicitly stitch behaviors, while also learning optimal policies stochastic settings with suboptimal data. On bench- well as visual observations, we demonstrate that our method
mark tasks of up to 21 -i-dimensions achieves results that considerably outperforms that similar baselines. Additional experiments reveal the importance of the enforced invariances and contrastive learning objective. Given the importance of long-horizon reasoning many potential applications of RI. today, we believe our work is useful for thinking about how to learn optimal temporal distances.

---

Stochastic Dynamics
Optimal Policies
Metric Architecture
((,
P( a,
suboptimal demos K
π
O eb(8)
optimal x²
dg S dg + do
prior work
CRL
GCIQL
CRL
GCIQL
CRL
GCIQL
QRL
CMD
QRL
CMD
QRL
CMD
this work
TMD (Ours)
TMD (Ours)
TMD (Ours)
Figure 2: TMD enables key capabilities over prior work: (Left) handling stochastic transition dynamies, (Center) learning optimal policies from offline data, and (Right) stitching behaviors as property of network architecture.
2 Related Work
Our method provides unifying framework connecting temporal distance learning to (optimal) offline GCRL. The resulting method gets the benefits of both, learning optimal goal-reaching policies from offline data with stochastic dynamics, and using a quasimetric architecture to optimally stitch together behaviors without the compounding errors of TD learning.
2.1 Temporal Distances
We build on prior approaches to learning temporal distances, which reflect the reachability of states [16]. Temporal distances are usually defined as the expected number of time steps to transit from one state to another [21, 22]. Recent work has provided probabilistic definitions that are also compatible with continuous state spaces and stochastic transition dynamics [16]. A key consideration when thinking about temporal distance is which policy they reflect: is this an estimate of the number of time steps under our current policy or under the optimal policy? We will use optimal temporal distance to mean the temporal distance under the optimal Disstance-miimizingg policy.
Algorithmically. this choice is often reflected in the algorithm one uses for learning temporal distances. Methods based on Q-learning typically estimate optimal temporal distances [11, 23, 24], and are often structurally similar to popular actor-critic methods. Some quasimetric methods also learn optimal temporal distances in deterministic MDPs by enforcing the triangle inequality as architectural constraint, effectively computing shortest paths a directed graph [17, 18]. Prior work has shown that temporal distance learning can be important for finding paths that are better than those demonstrated the data, and can enable significantly more data efficient learning [25] (akin to standard results in the theory of Q-learning [26]).
Methods based Monte Carlo learning typically operate by sampling pairs of states that occur nearby in time (though not necessarily temporally ;djjcce)y)adjaccett;;; distances are minimized for such positive pairs, and maximize for pairs of states that appear on different trajectories [27, 15]. These Monte Carlo methods often estimate the temporal distance corresponding to the policy that collected the data. Methods for goal- conditiood behavioral cloning [28, 29], though not directly estimating temporal distances, are effectively working with this same behavioral temporal distance [7]. Despite the fact that Monte Carlo methods do not estimate optimal temporal distances, they often outperform their Q-learning counterparts, suggesting that it is at least unclear whether the errors from learning the behavioral (rather than optimal) temporal distance are larger or smaller than those introduced by TD learning's 's compounding errors. Our work bridges these two notions of temporal distance,

---

providing a method that learns optimal temporal distances while reducing the reliance on TD learning to propagate values (and accumulate errors).
2.2 Offline Reinforcement Learning
Our investigation into temporal distances closely mirrors discussions in the offline RL literature about 2-step RL methods [30], which often use Monte Carlo value estimation, versus multi-step RL methods [31], which often use Q-learning value estimation. These 1-step RL methods avoid the compounding errors of Q-learning, yet are limited by their capacity to learn Q2 (s, a) rather than Qo However, their strong performance over the years [32, 15] suggests it is an open question whether the compounding errors of Q-learning outweigh the benefits of learning the behavioral value function, rather than the value function of the optimal policy.
3 Temporal Metric Distillation (TMD)
In this section, we formally define TMD in terms of the invariances it must enforce to recover optimal distances, and by extension, the optimal policy. In Section we will then show how these invariances can be converted into losses which can be optimized with a quasimetric architecture that enforces the triangle inequality.
3.1 Notation
We consider a controlled Markov process M with state space S, action space A. and dynamics P(s' 8, s,a). The agent interacts with the environment by selecting actions according policy (a s), i.e., a mapping from S to distributions over A. We further assume the state and action spaces are compact.
Policies π E are defined as distributions x(a s) for s E S, a E A. When applicable, for a fixed denote the state and action step random variables and a,, respectively. We will also use the shorthand
policy म,
5 - S for K Geom - -7).
We equip M with an additional notion of distances between states. At the most basic level, distance x S + R must be non negative and equal zero only when passed two identical states. We will denote the set of all distances D, defined as
D {d: S S + R: d(s,s) = 0, d(s. 2 0 for each s, E S}.
desirable property for distances to satisfy is the triangle inequality, which states that the distance between two states is greater than the sum of the distances between the states and a waypoint [18]. distance satisfying this property is known as a quasimetric. Formally, we construct
@ å {d E D d(s,g) S d(s, w) + d(w,g) g) for all 9,9, E S}.
(2)
we further restrict distances to be symmetric (d(x,, y) d(y, x)), we obtain the set of traditional metrics over S.
3.2 TMD Operators
TMD learns distance parameterization that made to satisfy two constraints: (i) the triangle inequality,
d(z,2) < d(x, 3) + d(y, 2) for any x, P. E S AUS,
(3)
and (ii) action invariance,
d(s, (s, a)) 0 for any E and A.
We will show that to ensure that we recover the optimal distance d [16] given the learned (backward NCE) contrastive critic distance, the missing additional constraint form of consistency over the environment dynamics with respect to the expected (exponentiated) distances. This constraint

---

resembles the "SARSA" -tyle Bellman consistency, which backs up values by averaging over dynamics learn on-policy values. So, what TMD doing with these additional constraints is weakening the form of Bellman consistency that is required to recover the optimal distance from the standard m Bellman operator to the weaker on-policy SARSA Bellman operator. TMD thus turns on-policy SARSA into an off-policy algorithm through the metric constraints.
We define this additional constraint as the fixed point of the following operator:
T(d)(z, 4) { log Ep a..) [e- d(dd''") wt] dd(x.v)) v)
log if x = (s,a) E A,
(5)
otherwise.
The triangle inequality Eq. (3) and action invariance Eq. (4) properties also be written in terms of operator fixed points:
(rd) min [(( v) + d(y. z) vES
(6)
0 [(d))s,
if (s.a) otherwise.
Id(
(7)
3.3 Properties of path relaxation
Path relaxation P [19] (Eq. 6) enforces invariance the triangle inequality, i.e., P(d) d if and only if d E Q. Theorem 1. Take d E D and consider the sequence
dn = Pd)
Then, dn converges uniformly to fixed point doo E Q.
In light of Theorem denote by P. limm- p the fixed point operator of P, and note that P. is in fact projection operator onto Q.
Proofs of Lemmas 5 to and Theorem 1 can be found in Appendix C.
3.4
The modified successor distance
The modified successor distance dip E can defined by [16]: 11.5
if æ y.
-logp" P(a =glsp plo*=glooms.00=0 for K
if x (s, a) E A,y gES
d (Ir, y)
P(a+=g 90=9)
(8)
- log B (T(((a)) [b((s,))))]] - log
if æ = s E S
ds (s.
log ((a g))
y = (g. a) Sxx.
The optimal successor distance d' can then be stated as
d (x, v) A min d (x.
(9)
#ell
This distance is useful since it lets recover optimal goal-reaching policies. For any 8, S, A. the distance is proportional to the optimal goal-reaching value function
((s, g) x. (s,
(10)
where (7 a) is defined the standard optimal Q-function for reaching goal [15]:
(s, A max E [s. -uT
P(ss = g so =s a, a)
(11)
sell
and
(s(
(3(5,
(12)
aEA
5

---

In fact, we can equivalently define dip ((s, ass 9) in terms of Q":
dsp ((s, a), g) = log V (g) log (s.
(13)
Similar to Myers et al. [16], we argue that contrastive learning can recover these distances, i.e.,
CTTT) daD
(14)
Then, through the operators in Section 3.2, we will extend this to the optimal distance d-g--
For convenience, we also define the set of realized successor distances
a (d E II).
(15)
Note that D does not necessarily contain the optimal distance daps as no single policy is generally optimal for reaching all goals.
Remark 2. The optimal successor distance d%p satisfies
dsp(s, (s, a)) - 0 for all s and a E A.
3.5 Convergence to the optimal successor distance
Applying the invariances Section 3.2 to the contrastive distance Eq. (14), the TMD algorithm can be defined symbolically
M(*) - TTo)
(16)
In other words, TMD computes the initial Ty distance C(x), and then enforces the invariance (architecturally or explicitly), expressed with the iterative application of T I followed by projection onto Q by P..
Theorem 3. The TMD algorithum converges pointwise to the optimal successor distance d for any policy n with full state and action coverage, i.e.,
lim n
dgp-
(17)
Our approach for proving Theorem 3 will be to analyze the convergence properties of (P, T I) over the space of "suboptimal" distances D4,, defined as
Dj a e d(x, y) 2 d'p (z, y) for all x, E S AUS).
(18)
Unfortunately, (P. o T o I) is not a contraction on D so we cannot directly apply the Banach fixed-point theorem would for the standard Bellman (optimality) operator. Instead, will show this operator induces a "more aggressive" form of tightening over D which will allow us to prove convergence dzp. We start by showing that d is fixed point of (P. T I) in Lemma 4. Lemma 4. The optimal successor distance drp is the unique fixed point of T o I on DJ.
Proofs are in Appendix B.
Implementing TMD
We show that the backward NCE contrastive learning algorithm can recover an initial estimate of dsp As justified by Theorem 3, we can then enforce the invariances to recover the optimal distance d-
The algorithm learns distance da parameterized by quasimetric neural network 0 such Metric Residual Network (MRN) [17]. By construction, this distance is quasimetric that invariant P. i.c., Pdg de.

---

4.1 Initializing the Distance with Contrastive Learning
Defining the critic
f(s, a(s.a.g) do -dosss the core contrastive objective is the backward NCE loss:
N Σ
LNCE (0, th; (8i, ar 2i)
log
(19)
Σラ= e (e3.0s)88))
i=1
which is enforced across batches of triplets {si, di, 8i-RJi=1 for k ~ Geom(1 -7) sampled from the dataset generated by policy zg-
The optimal solution to this objective is
((s,a.9) - log P(s = 9 5= -= --0==,,000 =
(20)
P(s+ = =g)
for some C(g) [33].
The parameterization [(, a,g) -d (ds ((0,))()))) .)) where de is quasimetriic-efforing parameteriza- tion (see [17, 18]) ensures that
C(g) P(s = = g) s P(s + = -g)
so the only valid quasimetric satisfying Eq. (20) is do dsp =e_ Optimality of C in Eq. (19) implies that the learned distance de C(TT) dso "
The additional invariance constraints and T can be directly enforced by regressing I|g. - Idg oo and ||gg T dglle co to zero. Theorem 3 guarantees that if we enforce those constraints and enforce invariance to P by using quasimetric architecture (e.g., MRN [17]), we can recover the optimal distance d=
In practice, we will directly enforce the constraints across the batches used our contrastive loss. We will use the MRN parameterization for dg for 0 = (v², Φ) on learned representations of states (16) and state-action pairs (6):
dg (8,9) A duUN (v()),,())) do (s, a)) dMRN (1)) a))
de ((x, a). A (d,, de a(s (s', (s',a') & dRRN (((s,
where
K
1
dRANN (x., y)
A
max max(0, F&M+ M)Mr+m) I...M
(21)
K
4.2 Action Invariance (Z)
Invariance to the I backup operator in Eq. (7) gives the following update across s, S A
do (1b(s), o(s, a)) + 0.
(22)
which can be directly enforced with the following loss across the batch:
LI (d, toi tsi ai s(..):
Σ duaN (v(ss),((s,
(23)
4.3 Temporal Invariance (T)
Invariance to the T backup operator in Eq. (5) corresponds to the following update performed with respect to Φ(s, a):
(a) +- EP((o)))) [c 1og d (
(24)
7

---

This update is enforeed by minimizing a divergence between the LHS and samples from the RHS expectation. Classic approaches for backups in deep RL include the l distance to the target (RHS) [34], or when values can be interpreted as probabilities, binary cross-entropy loss [35].
We the following Bregman divergence [36], which find empirically is more stable for learning the update in Eq. (24) (c.f. the Itakura-Saito distance [37] and Linex losses [38, 39J).
Dr(d, d') - exp(d - -d)) d.
(25)
We discuss this divergence and prove correctness in Appendix E. With the divergence, the T- invariance loss is:
N
CT (o, a.. =
ΣΣ
Dr Dr(dwas(d(s,, a.). (bg))) dRax (v)(s)) xbg,)) log 7)
(26)
We minimize this loss only with respect to é stopping the gradient through db. This avoids the moving target that classically necessitates learning separate target networks in RL [34].
4.4 The Overall Distance Learning Objective
We can express the overall critie loss as:
LTDD (0, di vb, B) = LxCr (0) v;; B) + c( (0, v; B) + LT (d) B))
(27)
for batch B ~ TT {8,, ai 8 9. )
We minimize Eq. (27) with respect to and 1h. where T is separate copy of the representation network & (stop-gradient). Here, Ç controls the weight of the contrastive loss and invariance constraints, and batches are sampled
{st, 0i, 8/s D: ) T3
for s/ the state following s,, and 9i the state K steps ahead of s; for K Geom - 7)) In theory, <- should be annealed between at the start of training (to extract the distance C(x)), toward 0 at the end of training to enforce invariance to (T e I), though in practice we find it suffices to keep constant in the environments tested.
In practice, we pick based on how much stitching and stochasticity we expect in the environment when Ç is large, we more aggressively try and improve on the initial distance C(TT) describing the dataset policy nT.
4.5 Policy Extraction
We finally extract the goal-conditioned policy g( S² + A with the learned distance de:
N
min 1. ma
Σ de ((s..,))))))) f-1=1
(28)
For conservatism [40], we augment Eq. (28) with a behavioral cloning loss against vav behavior- constrained deep deterministic policy gradient [41]. Using additional goals gi sampled from the same trajectory as s; in Eq. (28) could also be done through an extra tuned parameter (cf. Bortkiewicz et al. [42], Park al. [43]). Denoting these hyperparameters as À and respectively, the overall policy extraction objective is:
min E El Sny [A (T; d. v. (sr, on s/ [
(29)
C A
A)dMRN (p(s, asy)) (g))),, v19))) + Admax (6 an) (n),, + a a, l|
+1=1
where à. ((:
(30)
Prior offline methods use similar and À hyperparameters, which must be tuned per environ- ment [43].
8

---

Table 1: OGBench Evaluation
Methods
Dataset
TMD
CMD
CRL
QRL
GCBC
GCIQL
GCIVL
64.6
59.9(
7
27,3(100)
24.0(-00)
hunanoi dmaze, www_meddd_mmeeeeeee stttch
68
64.8 H3
+6.2(
290(+13)
(2.((
12,3(-00)
hunanoi www.dm laaiiis__arrrr_ttttiiee
23.0
9.3
4.0(2m))
3,5
(
0,5(201)
humanoidmazee wawaaaooodmeee_iatt____o.
(214)
5,0
0,2
0,5
0,20200
humanoidmaze.giie
(
0,2
L.((20))
1,7
29.3(4233)
1614199
(1.5(
25.2(
44.4
93
87 (1
29.0 +13)
12.1 (211)
12,3(m0a)
latmaze_large_navigats
(1,5(
84,1 (MM0)
82,76114
(4.6
24.0(=06)
(412)))))
15,7(=19)
uutnaz__aaree_titee
37,3(#23)
29.0
(
(,4(=1))
7,5(+07)
18,5(=03)
aatmaza..teleporr.eer..ooo
49.6(415)
02(100
19,5(200)
2,3(187)
2,4-299
(.3
32,00209)
uatmae__giant_ttite
2
2
0.4
0,0(-9))
0.0
0.0
scene_noiey
19,5(*1))
4.0
(2()0))
9.1()
1,2(=
25.9
26.4
(8,((1
36.0 36.0
31.7(199)
L4(1
31.8(219)
Lo(
14(299
aamual_atmazz__arrge sttttc
26.6(+2))
(1
(1,1(t1)
066+55)
23,6+
0.8++33
Gesual_antmzz__iaan_.ng cleuul_cube_trippe_ee
(0.1
(7.3
47,2(200)
0,1
0,4(=01)
0.1 0.1
10
17.7(107)
16.1
15.6(
8.6
16.2(297)
12,5(196)
(7.9(293)
standard error across six seeds. datasets contain separate tasks each. We record aggregate across 5 tasks.
bold best performance. Success rate (%) is presented
5 Experiments
In our experiments, we evaluate the performance of TMD on tasks from the OGBench benchmark [43]. aim to answer the following questions:
1. Do the invariance terms in Eq. (6) improve performance quantitatively in offline RL settings?
the contrastive loss in Eq. (19) necessary to facilitate learning these tasks?
What capabilities does TMD enable for compositional task learning?
5.1 Experimental Results
We evaluate TMD across evaluation tasks in OG- Bench for the environments and datasets listed in Table 1. The experiments use 6 seeds in all environments, and report the success rates ag- gregated across the 5 evaluation tasks (goals) provided with each environment. Of particular interest are the "teleport" and "stitch" en- vironments, which respectively test the ability to handle stochasticity and composition.
We compare against the Goal- CondiCondi Behavioral Cloning (GCBC), Conditiond Implicit Q-Learning (GCIQL), Goal- Conditioned Implicit Value Learning (GCIVL), Contrastive Reinforcement Learning (CRL), and Quasimetric Reinforcement Learning (QRL) algorithms, using the reference results provided by OGBench [43]. We implement and evaluate Contrastive Metric Distillation (CMD) [16], which also learns a quasimetric temporal distance, but does not enforce the constraint of T or I invariance and uses a separate critic architecture. GCBC uses imitation learning to learn policy that follows the given trajectories within dataset [44]. CRL [15] performs policy improvement by fitting a value function via contrastive learning. QRL [18] learns quasimetric value function to recover optimal
Predicted Distance to Goal
70
140
Figure 3: An example distance heatmap learned by TMD in pointmaze. Laainaze_arrge__ee targe_stitch. Darker colors indicate larger distances.

---

distances in deterministic settings. GCIQL and GCIVL use expectile regression to fit a value function [45].
TMD consistently outperforms QRL and CRL in the stitching environments. In the stochastic teleport environments, TMD outperforms both CRL and QRL. by considerable margin in pointmaze_ tttpa/e_eeepprttstttteee TMD outperforms CRL and QRL by over 3x. An example distance learned by TMD in antmaze_
visualized heatmap in Fig. 3.
5.2 Ablation Study
We perform an ablation study on the pointmaze teleport_stice environment to evaluate the impor- tance of the invariance terms and the contrastive initialization loss in TMD. We separately disable the contrastive, T invariance, and I invariance com- ponent during training and observe its effects. We also examine the em- pirical effects of stopping gradients when calculating Lr. We log the cor- responding success rate for each of the ablations in 4.
Ablations of the LMD loss
TMD (Ours) no Ly stopgrad
no LNCE loss
I loss
no T loss
0.1
0.2
0.3
Our ablation studies answer questions 2 and 3, which we demonstrate that by removing some of the invariances or removing the contrastive loss, the performance of TMD decreases to lev- els similar to CRL and QRI. Simi- larly,
Success Rate
Figure 4: We ablate the loss components of TMD in the tiiltoaaze_eleeprt_.ttttt environment.
the importance of keeping the contrastive objective, the performance of TMD degrades even more despite the presence other loss components. We also note the empirical performance of TMD is better when stop gradients on Ly. We provide further ablation details in Appendix D.2.
6
Discussion
In this work, we introduce Temporal Metric Distillation (TMD), an offline goal-conditioned rein- forcement learning method that learns representations which exploit the quasimetric structure of temporal distances. Our approach unifies quasimetric, semporal-differncce, and Monte Carlo learning approaches to GCRL by enforcing set of invariance properties the learned distance function. To the best of our knowledge, TMD is the first method that can exploit the quasimetric structure of temporal distances to learn optimal policies from offline data, even in stochastic settings (see Fig. 2). On a standard suite of offline GCRL benchmarks, TMD outperforms prior methods, in particular on long- horizon tasks that require stitching together trajectories across noisy dynamics and visual observations.
6.1 Limitations and Future Work
Future work could examine more principled ways to set the Ç parameter in our method, or if there are ways to more directly integrate the contrastive and invariance components of the loss function. Future work could also explore integrating the policy extraction objective more directly into the distance learning enable desirable properties (stitching through architecture, horizon generalization) at the level of the policy. While we used the MRN [17] architecture in our experiments, alternative architectures such as Interval Quasimetric Embedding (IQE) [20] that enforce triangle inequality could be more expressive. While the size of models studied in our experiments make them unlikely to pose any real-world risks, methods which implicitly enable long-horizon decision making could have unintended consequences poor interpretability. Future work should consider these implications.
10

---

Acknowledgements
We would like to thank Seohong Park, Qiyang (Colin) Li, Catherine Ji, Cameron Allen, and Kyle Stachowicz for relevant discussions and feedback on this work. This research was partly supported by AFOSR FA9550-22-1-0273 DARPA TIAMAT, and the DoD NDSEG fellowship.
References
[1] Eduardo D. Sontag. A 'Universal'' Construction of Artstein's Theorem Nonlinear Stabiliza- tion. Systems & Control Letters, 13(2):11-1223, 1989.
[2] Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control Barrier Functions: Theory and Applications. European Control Conference. pp. 3420-3431, 2019.
[3] Matthias Althoff. Reachability Analysis and Its Application to the Safety Assessment of Au- tonomous Cars. phdthesis, 2010.
[4] Kai-Chieh Hsu, Haimin Hu, and Jaime F Fisac. The Safety Filter: A Unified View of Safety- Critical Control in Autonomous Systems. Annual Review of Control, 7, 2023.
[5] Shangding Gu. Long Yang. Yali Du, Guang Chen, Florian Walter, Jun Wang, and Alois Knoll. A Review of Safe Reinforcement Learning: Methods, Theory and Applications. arXiv:2205. 10330, 2022.
[6] Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. LM-Nav: Robotic Navigation With Large Pre-Traine, Models of Language, Vision, and Action. Conference on Robot Learning, 2022.
[7] Raj Ghugare, Matthieu Geist, Glen Berseth, and Benjamin Eysenbach. Closing the Gap Between TD Learning and Supervised Learning a Generalisation Point of View. International Conference on Learning Representations, 2024.
[8] Yecheng Jason Ma. Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards Universal Visual Reward and Representation via Value-Implice Pre-Training. International Conference on Learning Representations, 2023.
[9] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A Universal Visual Representation for Robot Manipulation. Conference on Robot Learning, pp. 892-909, 2022.
[10] Long-Ji Ji Lin. Self- Self-Improvin Reactive Agents Based on Reinforcement Learning, Planning and Teaching. 1992.
[11] Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight Experience Replay. Neural Information Processing Systems, volume 30, 2017.
[12] Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal Difference Mod- els: Model-Free Deep RL for Model-Based Control. International Conference on Learning Representations, 2018.
[13] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing Off- Policy Q-Learning via Bootstrapping Error Reduction. Neural Information Processing Systems, 32, 2019.
[14] Alexey Dosovitskiy and Vladlen Koltun. Learning Act by Predicting the Future. International Conference on Learning Representations, 2017.
[15) Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive Learning as Cond-Condtti Reinforcement Learning. Neural Information Processing Systems, volume 35, pp. 35603-35620, 2022.
[16] Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learn- ing Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making, International Conference on Machine Learning, 2024.
[17] Bo Liu, Yihao Feng, Qiang Liu, and Peter Stone. Metric Residual Network for Sample Efficient Goal-Conditioned Reinforcement LareemmnntLLaannnn Learning. AAAI Conference on Artificial Intelligence, volume 37, pp. 8799-8800 8806, 2023.
11

---

[18] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal Goal- Reaching Reinforcement Learning via Quasimetric Learning. International Conference on Machine Learning, pp. 3641 1-3430,, 2023.
[19] Vivek Myers, Catherine Ji, and Benjamin Eysenbach. Horizon Generalization in Reinforcement Learning. International Conference on Leurning Representations, 2025.
[20] Tongzhou Wang and Phillip Isola. Improved Representation of Asymmetrical Distances With Interval Quasimetric Embeddings. NeurIPS 2022 NeurReps Workshop Proceedings Track, 2022.
[21] Junik Bae, Kwanyoung Park, and Youngwoon Lee. TLDR: Unsupervised Condiiiood RL via Temporal Distance-Awart Representations. Conference on Robot Learning, 2024.
[22] Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Model-Based Visual Planning With Self- supervisdd Functional Distances. International Conference on Learning Representations, 2021.
[23] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. International Conference on Machine Learning, 2018.
[24] Leslie Pack Kaelbling. Learning to Achieve Goals. International Joint Conference on Artificial Intelligence, volume 2. pp. 1094-1098, 1993.
[25] Chongyi Zheng. Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive Difference
Predictive Coding. International Conference Learning Representations, 2023.
[26] Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement Learning: Theory and Algorithms. CS Dept. 32:96, 2019.
[27] Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical Distance Learning for Semi- Semi-Supervised and Unsupervised Skill Discovery. International Conference on Learning Representattinss 2020.
[28] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to Reach Goals via Iterated Supervised Learning. International Conference Learning Representations, 2021.
[29] Vivek Myers, Andre He, Kuan Fang, Homer Walke, Phillipe Hansen Estruch, Ching-An Cheng, Mihai Jalobeanu, Andrey Kolobov, Anca Dragan, and Sergey Levine. Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control. Conference on Robot Learning, 2023.
[30] Benjamin Eysenbach, Matthieu Geist, Sergey Levine, and Ruslan Salakhutdinov. A Connection Between One-Step RL and Critic Regularization in Reinforcement Learning. Th International Conference on Machine Learning, pp. 9485-9988899 9507, 2023.
[31] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- Dimensional Continuous Control Using Generalized Advantage Estimation. arXiv:1506.02450
2018.
[32] Cassidy Laidlaw, Banghua Zhu, Stuart Russell, and Anca Dragan. The Effective Horizon Explains Deep RL Performance in Stochastic Environments. International Conference on Learning Representations, 2024.
[33] Zhuang Ma and Michael Collins. Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency. Empirical Methods in Natural Language Processing, pp. 3698-3707, 2018
[34] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, et al. Human-Leves Control Through Deep Rein- forcement Learning. Nature, volume 518, pp. 529-533, 2015.
[35] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Conference on Robot Learning, pp. 651-673, 2018
[36] I.. M. Bregman. The Relaxation Method of Finding the Common Point of Convex Sets and Its Application to the Solution of Problems in Convex Programming. USSR Computational Mathematics and Mathematical Physics, 7(3):200-217, 1967.
[37] Fumitada Itakura. Analysis Synthesis Telephony Based on the Maximum Likelihood Method. Reports of the 6th Int. Cong. Acoust, 1968.
12

---

[38]
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme Q-Learning MaxEnt
RL Without Entropy. International Conference on Learning Representations, 2023.
[39]
Ahmad Parsian and Snua Kirmani. Estimation Under LINEX Loss Function. Handbook of
Applied Econometrics and Statistical Inference, pp. 75-98. CRC Press, 2002.
[40] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning
for Offline Reinforcement Learning. Neural Information Processing Systems, volume 33, pp.
1179-1191, 2020.
[41] Scott Fujimoto and Shixiang Shane Gu. A Minimalist Approach to Offline Reinforcement
Learning. arXiv:2106.068800 2021.
[42] Michal Bortkiewicz, Wladek Palucki, Vivek Myers, Tadeusz Dziarmaga, Tomasz Arczewski, Lukasz Kucinski, and Benjamin Eysenbach. Accelerating Goal-Conditioned RL Algorithms and Research. International Conference Learning Representations, 2025.
[43] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Bench-
marking Offline Coal-Conditionee RL. International Conference on Learning Representations, 2025.
[44] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-Conditioned Imitation
Learning. Neural Information Processing Systems, volume 32, 2019.
[45] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning With Implicit
Q-Learning. arXiv:21 1.Xiv2210000169, 2021.
[46] Yim-Ming Wong and Kung-Fi Fu Ng. On a Theorem of Dini. Journal of the London Mathematical Society, 52-11(1):46-45 1975.
[47] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman -Milne, and Qiao Zhang. JAX: Composable Transformations of Python+NumPy Programs. 2018.
[48] Arindam Banerjee, Srujana Merugu, Inderjit Dhillon, and Joydeep Ghosh. Clustering With Bregman Divergences. SIAM International Conference on Data Mining, pp. 234-245, 2004.
A Code
Code and videos be found at agent structure follows the OGBench codebase [43]. The TMD agent is implemented in https: hatpa///ggnnttttteeeee htd.//md.pppp py.
github. io/. The evaluation and base
//ggituu..com wttp:/ooo//viieeiieeeeee http://vvveiieeeeeee/ee/ http:/dddddreeeeeeeeeeee///eeeee
B Analysis of TMD
This section provides the proofs of the results in Section 3.5. The main result is Theorem 3, which shows that enforcing the TMD constraints on learned quasimetric distance recovers the optimal distance d:p
Theorem 1. Take d E D and consider the sequence dn = (d).
Then, dn converges uniformly to fixed point doo E Q.
Proof. From Lemma 7, have that d (s, g) S dn(ss, g) for all 8, S. Thus, the sequence {d,) monotonically decreasing (and positive). By the monotone convergence theorem, the sequence converges pointwise to limit doc. Since compact, by Dini's theorem [46], the convergence is uniform, i.c., dp + doo under the - topology over D.
To see that dec is fixed point of P, note that if Pd d' + doc. we can construct disjoint neighborhoods of doc and N' d' (since L (D) is normed vector space and thus Hausdorff). By construction, the preimage P- (N') contains d'gc and is open by Lemma 6. Thus, another, smaller open neighborhood N N n p (N') of dxc. Now, since d, + doas there exists some l de, dx+1 N N. But then since de E p-'(NN), we have that dx+1 N'. This is contradiction as N and N' were disjoint by construction.
define
Thus, have that d is fixed point of P. That dx Q follows from Lemma
13

---

Theorem 3. The TMD algorithum converges pointwise to the optimal successor distance d for any policy π with full state and action coverage, i.e.
lim c o C(x) = =d n0000
(17)
Proof of Theorem 3. The initial distance C(T) dan 2 dgp for any policy л. So, C(x) E D Define the sequence of distances d, = (P. o [)((((>)) in Di. Note that P. and I are monotone decreasing. So, the restriction (d,, Ax is monotonically decreasing the domain X 8 A))SxAA) and thus converges pointwise on as n >> 0c.
Since T and P., are continuous operators (Lemma 6 and Eq. (5)), and T is fully-determing by the restriction to X, the sequence (P o d), = d converges pointwise on its full domain. The pointwise limit of d, is fixed point of (P. T o I), which must be the unique fixed point dip D by Lemma 4.
Lemma 4. The optimal successor distance dıp is the unique fixed point P. o T o I on Dt.
Proof of Lemma 4. For existence, we note
(P. I)d' (P. o (I
(Remark 2)
(P. o TT)d
(Bellman optimality of Q2))
P.d
(Lemma 5) (31)
d'nn
For uniqueness, need show that (P, T I) has no fixed points besides d'p in D Suppose there exists some d D such that (P, T o hdd== d. Then, we have for x E SUS x A. 8,9 S, and AA:
(P. o To Idd(r (( d( (g. d(x,g).
(32)
Denote by Q(s a) d((s,a).9) and let B be the goal-conditioned Bellman operator defined as
BQ(s,a) A Ep(e)) [1(s' =9} + +7Q(s',9))
(33)
At any fixed point d E DJ, we have
PP. ((),,)) = d((s. a).
(34)
This last expression implies that
Q(s, a) exp(-d((s.a).9))
T T)d((s, a),9)]
S (p(s'(a)) min expd((s (ss,('('') - log d'EA
a).
(35)
Since B is a contraction on the exponentiated distance space, and d((s,a).g) 2 do ((s, a)s,a).)) g).
Eq. (35) is only consistent with Q(s, a) = (s, This implies that
d((s, a). g) = din ((s, a).g).
(36)
We also know that at this fixed point, d(s, (s, a)) and thus from Eq. (36) have
d(s. g) (8.g). So, d = di, must be the unique fixed point of (P. T I).
(37)
14

---

C Path Relaxation and Quasimetric Distances
We provide short proofs of the claims in Section 3.2 Lemma 5, We have P(d) = d if and only if d Q
Proof. p( min dd( w) d(w,g) wES
s d(s, s) + d(s,g)
= d(s,g) g)
Lemma 6. The path relaxation operator P is continuous with respect the L topology over D.
Proof. Let d, d' D and > 0. We have
[P(d)(s.)) - (P((),))))(())ss
min [d(s, w) + d(w, 9)] - min (s, w) + d'(w, 9]]
wES
wES
munn|d(s, w) d(w. g) - d'(s, w) d' (w.g)]
wes
min d(s, w) d'(s, w)
+ min d(w, g) - d' (w g)
WES
wES
||d d' 1oo + l|d - d' 1o0
2 d' ooo
Thus, [|d - d' e/2, have P(d) P(d')
ll 20 <e
Lemma 7. For any 8,g E S and d E D we have that P(d)(s,g) s d(s,g).
Proof. Let Let E D and > O. We have
[P(d)(s.g) (P((),,))(((d((s.)))))
min [d(s, w) + d(w. 9)]
min (', w) + dww weS
min d(s, w) + d(w, g) - -d' - d' (ww,g))
wES
wes d(s, w) d' (s, w)) + + d(w, .) d'(w,g)l
< ||d - d' 1ox + ||d - 1000
2||d d' d'l
Thus, if I|d d' < 6/2,, we have ((d) P(d') |1 os <e.
Lemma S. We have P(d) d and only if Q
Proof. (=) Suppose P(d) = d. Then, for all 8,9, w E S we have
d(s.g) (s))(s.g) min [d(s, w) + d(w,g)) I d(s, w) d(w,9)
Thus, Q
(+) Suppose d E Q. Then, for all s, g S have
d(s.g) s min [d(s, w) d(w.g) 9)] (sd)(s wES
We also have (s))(a.g) S d(s,g) by Lemma 7. Thus, P(d) = d.
15

---

Table Hyperparameters for TMD
Hyperparameter
Value
batch size
256
learning rate
3-10-1
discount factor
0.995
invariance weight Ç
0.01 in medium locomotion environments, 0.1 otherwise
Table Network configuration for TMD.
Configuration
Value
latent dimension size
512
encoder MLP dimensions
(512, 512)
policy MLP dimensions
(512, 512)
layer norm in encoder MLPs visual encoder (visual- envs) MRN components
True impala-smi
T weighting on diagonal elements
(navigation, play) 0.5 (stitch, explore, noisy)
D Experimental Details
General hyperparameters are provided in Table 2.
We implemented TMD using JAX [47] within the OGBench [43] framework. OGBench requires per-environment hyperparameter a controlling the behavioral cloning weight to be tuned for each method based on the scale of its losses. We generally found TMD to work well with similar values to those used by CRL. We used the same values of a as CMD's ' implementation. For complete list of alpha values, please refer the code release of the paper.
To prevent gradients from overflowing, we clip the T invariance loss per component to be no more than 5. We also found using slightly smaller batch size of 256 compared to 512 be helpful for reducing memory usage.
D.1 Implementation Details
The network architecture for TMD is described in Table 3. The "MRN components" refers the number of ensemble terms K in Eq. (21). We found = components enabled stable learning and expressive distances. We weigh the off-diagonal element, corresponding to the product of the marginals (()) on a 0-1 scale compared to the diagonal elements, corresponding to the joint distribution p(s, g). A scale of 0 corresponds the off-diagonal elements weighing the same as the diagonal elements, and scale of means that only diagonal elements will matter for T -operattor
D.2 Ablations
The full ablation results for TMD in the pointmazeetellaprtt-siteh are presented in Table with success rates and standard errors.
Table 4: Ablation Success rate.
Ablation
Success Rate
None
29.3
No gradient stopping in Lr
(8.7
No contrastive loss
(.8(+
No I loss
13.3(#29)
No T loss
18.5(#2.1)
16

---

D.3 Computational Resources
Experiments were run using NVIDIA A6000 GPUs with 48GB memory, and CPU cores and GPU per experiment. Each state-based experiment took around 2 hours to run with these resources, and each pixel-based experiment took around hours.
Comparison of Bregman divergences for fitting distances
d'
2
Loss
0
3
6
d
(d,(d,d)) (e e 12
d'c (1 d'' log(1 c de de-d' d
Dr(d,d') exp(d d') -
Figure 5: Comparison of Bregman divergences for e- onto d in expectation. All losses are minimized at d = d', and share the property that they will be minimized in expectation when e- Ele Ele-d'. But only the Dr(d, d') loss has non-vanishing gradients d >> d' for large d'.
E
Bregman Divergence in T-invariance
Recall the divergence used in Eq. (25):
Dr(d,d') A exp(d dd)
(25)
This divergence is proportional to the Bregman divergence [36] for the function F(x) log(z). similar the Itakura-Saii divergence [37].
DF(E d' ,) F(e F(e -d
- -ea
= d' d + c (e (c- - -
= d' d expld - dd) -
Dr(d,d') + d' -
(38)
The minimizer of Eq. (25) satisfies
arg min Es (d, (d, d')] = - llog Eay
le- (e
(39)
d20
when d' is random "target" distance [48]. In other words, using Eq. (25) loss function regresses e-d onto the expected value of e (or onto the expected value of con as used in Eq. (26)).
The key advantage of this divergence when backing up temporal distances is that the gradients do not vanish when either d, d', or the difference between them is small or large. This property is not shared by more standard loss functions like the squared loss or binary cross-entropy loss when applied the probability space and the models (distances) are in log- log-probability space.
17

---

Algorithm 1: Temporal Metric Distillation (TMD)
1: input: dataset D, learning rate ?
2: initialize representations ④, v. policy
3: while training do
4: sample B (5t, Ce 9 D
5:
6:
(Ø, 1b) - (0, vb) 1/ L. v: 1)
b Eg. (27)
7: t v (0, 4. B)
> Eq. (28)
8:
return
Table 5: Ablation of -iivariiance loss in autmarz-tteleport-tittll
Loss
Success Rate
Dr (Ours)
(9.3
Dig
16. 16.1
DBCE
15.1 (cb19)
E.1 Empirical Comparison
In practice, we found it was important to use this divergence in TMD for stable learning (Table 5). Ths loss could also applied to other GCRL algorithms where learned value functions are probabilities but are predicted in log-space to improve gradients. Future work should explore this divergence in other GCRL algorithms improve training compared to the more commonly used squared loss or binary cross-entropy loss [35].
F Algorithm Pseudocode
Full pseudocode for TMD is provided in Algorithm 1. We provide the full TMD loss function in Eq. (27) and the policy extraction loss in Eq. (28) below for reference:
[TM vh; B) LNCE (Ø, B) CT (0, B) + LT (0.
(27)
C ( ar tb (sy, on s
Σ(
A)dMAN (6 d:s) (1g)))
W=
+ AdMARn (0(8: an)) ((g)) + allas ail? (28)
where âij = = 0j), batch B ~ = = (si, 9:)
The components of Eq. (27) are (see Section 4):
N
c(s
CNCE (0. v: (8t, a, 8/, 9
log
(19)
el (35)00)18)))
i=1
==
N
Cx (o.V::ss:. Ci 8 9
dMny (U(ss),((s:0)))
(23)
N Dr(duux(Osss) az), (g draN (bb())) ((g))) log
Cr (0. a, 91.91111)
Σ
(26)
Glossary
Q3 The behavioral Quunntion under policy πg. 4
18

---

(s(s, a) the optimal goal-conditioned Q-function for reaching goal g- 4, 5
(s) the optimal goal-conditioned value function for reaching goal g. 5
I1 All policies [a s) mapping states to distributions actions. 4
(T) the outcome of running CRL with policy म, equivalent to dep under suitable assumptions. 6
D the of all distances that upper bound the optimal successor distance dsp- 6, 14
D the set of all distances over states 8, 5, 13
Q the set of all quasimetrics over states S. 4,
A the action space. 4-8, 14, 19
I the action invariance operator defined in Eq. (7). 5-8, 14
P the path relaxation operator defined in Eq. (6). 5, 15
the state space. 4-8, 13-15, 19
T the backup operator defined in Eq. (5). 5-8, 14
P. the projection operator onto the set of quasimetrics Q, defined as the fixed point of path. 5, 6, 14
TT the behavior policy used to collect the offline dataset. 6-8, 18
the action at time step (random variable).
9: the state at time step (random variable).
s the state random future time step + K where K Geom(1 - 77).
D the set of all realized successor distances d% under policies æ E IT. 6
Dr The Bregman divergence defined in Eq. (25), analogous to the Linex loss [38, 39]. 8, 17, 18
CI The action invariance loss defined in Eq. (23). 8, 18
LT The -invarriacce loss defined in Eq. (26). 8, 18
CNcE The backward NCE loss defined in Eq. (19). 8, 18
M controlled Markov process with state space S, action space A, and dynamics P(s' s,
learned state-action representation network. 7, 8, 18
learned state representation network. 8,
weight of the invariance losses in the overall distance learning objective defined in Eq. (27), 8, 16,
daRN ensemble version of the MRN [17] quasimetric parameterization defined in Eq. (21). 7,
18
dep the optimal successor distance, defined Eq. (9). 5-7, 13, 14, 19
d the modified successor distance under policy defined in Eq. (8). 5, 19
dsp the successor distance [16]. 6, 7, 14, 19
action invariance the property that the distance between a state and a state-action pair with that state is zero, d(s, (8,a) a)) 0 for all E S,a E A. 4,
OGBench A benchmark for offline goal-conditioned reinforcement learning [43]. 9, 13, 16
quasimetric a distance satisfying the triangle inequality (see Eq. (2)). 1-4, 6, 7, 9, 10, 13, 19
19

---

Acronyms
CMD Contrastive Metric Distillation. 9, 16
CRL Contrastive Reinforcement Learning. 9, 10, 16, 19
GCBC Goal-Conditinned Behavioral Cloning. 9
GCIQL Goal-Conditioned Implicit Q-Learning. 9, 10
GCIVL Goal-Conditioned Implicit Value Learning. 9, 10
GCRL Goal-Conditioned Reinforcement Learning. 2, 10, 18
IQE Interval Quasimetric Embedding. 10
MLP Multi-Layer perceptron. 16
MRN Metric Residual Network. 6, 10, 16
QRL Quasimetric Reinforcement Learning. 9, 10
RL Reinforcement Learning. 2, 4, 8,
TD Temporal Difference. 1,3,4
TMD Temporal Metrie Distillation. 4246,,9 9, 10, 13, 14, 16, 18
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, we show with theoretical justification (Section 3.4) and experiments (Section 5) that our method outperforms the baselines and enables the capabilities discussed in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, limitations are discussed in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
20

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, full proofs are in Appendices B and C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, we provide an implementation of our method and experimental details in the code
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
21

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, code is referenced in Appendix A and datasets used from Park et al. [43] are openly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/nips. wttpccccpppppppeeeeececcccceccet http//uwieee//eee/eeee/eee/ /eeee///ooebemiiiia ssi aesionPooicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
22

---

Answer: [Yes]
Justification:
Yes, experimental details are discussed in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, report error bars across seeds in our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, provide compute resources in Appendix D.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
23

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuripp: http:///....pp..ii/cccccccoeeeee
Question:
Answer: [Yes]
Justification:
All components of the NeurlPS Code of Ethics are respected.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss both positive and negative societal impacts in Section 6.1.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The models and datasets used in this paper do not pose a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, credit the source of the datasets and baseline implementations used in our experiments [43].
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcode thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing human subjects were used in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
25

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing human subjects used in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLMs were used in this research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M policy (https: / htt.r/pps..ccoooeeeeeneee////22/ for what should or should not be described.
26


Reviews:
[{"review_id": "RMknRIUxSo", "reviewer": "Reviewer_cpYD", "summary": "The paper explores goal-conditioned reinforcement learning by constructing temporal distance representations.\nIt presents a method to learn these representations using contrastive learning, while ensuring action invariance, temporal invariance, and quasimetric parametrization.\nThese requirements are then translated into specific architecture choices or training objective losses.\nEmpirical validation is provided on a subset of tasks from OGBENCH, demonstrating that the proposed approach outperforms existing methods such as Contrastive RL and quasimetric RL", "questions": "see weakness above", "limitations": "yes", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- the paper studies a relevant topic related to representation learning for goal-conditioned RL.\n- the paper provides ablation studies to show the importance of each of their algorithmic choices\n\nWeakness:\n- I found the paper challenging to read and not self-contained. Section 3 introduces several concepts (path relaxation, backward NCE, exponentiated SARSA) without providing clear definitions, instead relying on citations from other papers. A more comprehensive explanation of these key notions would be beneficial for understanding the proposed approach.\n- The method presentation is difficult to follow due to an overwhelming list of operators and inconsistent notation. For instance, the meaning of $C(\\pi)$ is unclear.\n- The introduction claims that the method can learn optimal policies primarily through Monte-Carlo learning, avoiding the accumulation error associated with temporal difference learning. However, the proposed method actually employs a soft version of the Bellman optimality equation, which is a type of TD learning, contradicting the initial claim", "was_revised": true, "final_justification": "I find the paper quite difficult to follow, and the equations are poorly presented. There are frequent typos and unclear definitions of variables. Additionally, the proofs are quite vague. As a reader, I shouldn’t have to guess the authors’ intentions at each step. For example, in the proof of the fixed point of the operators in the appendix, I don’t understand why equation (34) holds.", "rebuttal": "Thank you for your thoughtful feedback. Your main concerns seem to be the presentation of the method and notation. We have revised the paper based on your suggestions, and clarify some points below. We have also expanded the experimental results to include more environments and another baseline. **Do these revisions address your concerns? Please let us know if there are any clarifications or quantitative results you would like to see.**\n\n> The introduction claims that the method can learn optimal policies primarily through Monte-Carlo learning, avoiding the accumulation error associated with temporal difference learning. However, the proposed method actually employs a soft version of the Bellman optimality equation, which is a type of TD learning, contradicting the initial claim\n\nWe assume you are referring to the $\\mathcal{T}$-invariance with this comment. Note that this update ( $e^{-d_{\\\\mathrm{MRN}}(\\\\phi(s, a), \\\\psi(g))} \\\\leftarrow \\\\mathbb{E}_{\\\\mathrm{P}\\\\left(s^{\\\\prime} \\\\mid s, a\\\\right)} e^{\\\\log \\\\gamma-d\\_{\\\\mathrm{MRN}}\\\\left(\\\\psi\\\\left(s^{\\\\prime}\\\\right), \\\\psi(g)\\\\right)}$ ) is merely averaging over the dynamics $s,a \\\\to s'$, but not handling the max over actions at the next state. Thus, it is more analogous to an IQL-style [1] update $Q(s,a) \\\\gets \\\\mathbb{E} [ r + \\\\gamma V(s') ]$ than a Bellman backup $Q(s,a) \\\\gets \\\\max\\_{a'} \\\\mathbb{E} [ r + \\\\gamma Q(s',a') ]$. \n\nWhen this update is combined with the quasimetric architecture and action invariance $\\\\mathcal{I}$, we show that it recovers optimal distances. So we have replaced TD updates with an update that averages over dynamics (unavoidable to handle stochastic dynamics) and additional invariances to recover optimal distances. Unlike TD methods, our approach doesn't need separate target networks, precisely because it avoids the accumulating TD errors.\n\nAlso note that our method learns optimal distances, not soft-optimal distances. The $\\\\mathcal{T}$ update is performed with exponentiated distances because distances are in log space, not because we are performing a softmax.\n\n> I found the paper challenging to read and not self-contained. Section 3 introduces several concepts (path relaxation, backward NCE, exponentiated SARSA) without providing clear definitions, instead relying on citations from other papers. A more comprehensive explanation of these key notions would be beneficial for understanding the proposed approach.\n\nWe have added a ''glossary'' section to the appendix that clearly defines all of these terms in context, with references to the original cited works. We have also revised the main text to more clearly introduce these terms.\n\n> The method presentation is difficult to follow due to an overwhelming list of operators and inconsistent notation. For instance, the meaning of $C(\\\\pi)$ is unclear.\n\nWe use $C(\\\\pi)$ to denote the initial distance / critic recovered by CRL [2]. The key insight is that the distance $C(\\\\pi)$ is an overestimate of the optimal temporal distance, and that applying the operators $\\\\mathcal{I}$ and $\\\\mathcal{T}$ (explicitly) and $\\\\mathcal{P}$ (implicitly through quasimetric architecture) correctly tightens this overestimate.\n\nWe will revise the text to clearly introduce and motivate these operators, and add them to the glossary section.\n\n**Additional Results**\n\nWe have added additional environments and the CMD baseline to our evaluation results in Table 1.\n\n**Table 1: OGBench Evaluation**\n\n| Dataset                          | TMD         | CMD         | CRL         | QRL         | GCBC        | GCIQL       | GCIVL       |\n|----------------------------------|-------------|-------------|-------------|-------------|-------------|-------------|-------------|\n| `humanoidmaze_medium_navigate`     | 64.6 (±1.1) | 61.1 (±1.6) | 59.9 (±1.6) | 21.4 (±2.9) | 7.6 (±0.6)  | 27.3 (±1.9) | 24.0 (±0.8) |\n| `humanoidmaze_medium_stitch`       | 68.5 (±1.6) | 64.8 (±3.7) | 36.2 (±0.9) | 18.0 (±0.8) | 29.0 (±1.7) | 31.2 (±1.0) | 12.3 (±0.5) |\n| `humanoidmaze_large_stitch`        | 23.0 (±1.5) | 9.3 (±0.7)  | 4.0 (±0.2)  | 3.5 (±0.2)  | 5.6 (±1.0)  | 0.5 (±0.1)  | 1.2 (±0.2)  |\n| `humanoidmaze_giant_navigate`      | 9.2 (±1.1)  | 6.7 (±0.9)  | 1.5 (±0.6)  | 0.7 (±0.4)  | 0.4 (±0.2)  | 1.1 (±0.4)  | 1.4 (±0.3)  |\n| `humanoidmaze_giant_stitch`        | 6.3 (±0.6)  | 4.0 (±0.6)  | 1.5 (±0.4)  | 0.6 (±0.4)  | 0.4 (±0.1)  | 1.0 (±0.1)  | 1.2 (±0.1)  |\n| `pointmaze_teleport_stitch`        | 29.3 (±2.2) | 15.7 (±2.9) | 4.1 (±1.1)  | 8.6 (±0.6)  | 31.5 (±2.3) | 25.2 (±1.0) | 44.4 (±0.7) |\n| `antmaze_medium_navigate`          | 93.6 (±0.6) | 97.4 (±0.5) | 99.4 (±0.5) | 99.5 (±0.6) | 94.8 (±0.4) | 93.8 (±0.3) | 94.7 (±0.4) |\n| `antmaze_large_navigate`           | 81.5 (±1.6) | 84.1 (±2.1) | 82.7 (±1.4) | 82.4 (±1.0) | 44.1 (±2.0) | 34.2 (±1.5) | 31.7 (±1.4) |\n| `antmaze_large_stitch`             | 37.3 (±2.7) | 29.0 (±2.0) | 10.8 (±0.4) | 6.3 (±0.4)  | 33.4 (±1.0) | 7.5 (±0.7)  | 6.7 (±0.4)  |\n| `antmaze_teleport_explore`         | 49.6 (±1.5) | 53.3 (±1.3) | 52.7 (±0.6) | 49.7 (±0.8) | 36.0 (±0.8) | 15.6 (±0.6) | 15.1 (±0.5) |\n| `antmaze_giant_stitch`             | 2.7 (±0.6)  | 2.0 (±0.5)  | 1.2 (±0.3)  | 1.3 (±0.2)  | 2.0 (±0.3)  | 1.4 (±0.2)  | 1.5 (±0.2)  |\n| `scene_noisy`                      | 19.6 (±1.0) | 4.0 (±0.5)  | 1.2 (±0.3)  | 0.8 (±0.1)  | 1.6 (±0.2)  | 25.9 (±0.6) | 26.4 (±1.1) |\n| `visual_antmaze_teleport_stitch`   | 38.5 (±1.5) | 36.0 (±1.2) | 31.7 (±3.2) | 29.1 (±0.5) | 31.8 (±1.5) | 10.4 (±0.6) | 12.4 (±0.5) |\n| `visual_antmaze_large_stitch`      | 26.6 (±1.2) | 25.1 (±1.3) | 24.6 (±1.2) | 25.6 (±0.9) | 23.6 (±0.8) | 11.5 (±0.6) | 10.7 (±0.5) |\n| `visual_antmaze_giant_navigate`    | 37.3 (±1.4) | 36.1 (±1.6) | 47.2 (±0.9) | 44.3 (±0.9) | 31.5 (±0.8) | 20.2 (±0.9) | 19.6 (±0.9) |\n| `visual_cube_triple_noisy`         | 17.7 (±0.7) | 16.1 (±1.7) | 15.6 (±0.6) | 16.4 (±0.8) | 16.2 (±0.2) | 12.5 (±0.6) | 17.9 (±0.5) |\n\n---\n\n[1] Kostrikov, I. et al., 2022. ''Offline Reinforcement Learning With Implicit Q-Learning.'' ICLR\n\n[2] Eysenbach, B. et al., 2022. ''Contrastive Learning as Goal-Conditioned Reinforcement Learning.'' NIPS"}, {"review_id": "BLQWbsN6wG", "reviewer": "Reviewer_gzRx", "summary": "This paper introduces Temporal Metric Distillation (TMD), a novel algorithm for offline goal-conditioned reinforcement learning. Instead of learning a traditional Q-value function, TMD learns a temporal distance function, which estimates the cost to travel between states. The core of the method lies in , contrastive learning initialization and action-temporal invariance. In general, this paper is backed by solid theoretical analysis and demonstrate strong performance across test tasks.", "questions": "1. The hyperparameter $\\zeta$ is important for TMD learning and is currently tuned heuristically. To what extend does parameter tuning affect the performance of TMD?  Could you provide some empirical results?\n\n2. The current method involves a two-stage process of first learning the distance function and then extracting a policy from it. What are the primary theoretical or practical challenges in creating a fully end-to-end model, potentially an actor-critic-like method?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper provides a formal proof that the TMD algorithm converges pointwise to the optimal successor distance under certain assumptions.\n2. TMD demonstrates superior empirical results.\n3. This paper successfully unifies Monte-Carlo learning, temporal difference learning and metric learning.\n\nWeaknesses:\n\n1. Hyperparameter tuning is required.\n2. Metric learning and policy extraction stages are seperated in TMD. Integration of the two stages would result in a more elegant algorithm pipeline.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. We will address some of your concerns below.\n\n> Hyperparameter tuning is required.\n\n> The hyperparameter $\\zeta$  is important for TMD learning and is currently tuned heuristically. To what extend does parameter tuning affect the performance of TMD? Could you provide some empirical results?\n\nThe $\\\\zeta$ hyperparameter can be eliminated in our method with a dual descent approach similar to that used in QRL [1] and CMD-2 [2]. The TMD algorithm minimizes the contrastive loss with additional invariances enforced by a loss weighted by $\\\\zeta$. Dual descent will increase the $\\\\zeta$ value until the constraints were satisfied. We will discuss this option in the revised method section. \n\nIn practice, we found it is often simpler to use a fixed value of $\\\\zeta$ to enforce the constraint (see table below for an example in `antmaze-large-stitch`). We will add this ablation as a plot to the paper.\n\n**$\\\\zeta$ ablation in `antmaze-large-stitch`**\n\n| $\\\\zeta$ | score         |\n| ------- | ------------- |\n| 0.01 | 0.24 ± 0.027  |\n| 0.05 | 0.34 ± 0.013  |\n| 0.1  | 0.37 ± 0.027  |\n\n> Metric learning and policy extraction stages are seperated in TMD. Integration of the two stages would result in a more elegant algorithm pipeline. \n\n> The current method involves a two-stage process of first learning the distance function and then extracting a policy from it. What are the primary theoretical or practical challenges in creating a fully end-to-end model, potentially an actor-critic-like method?\n\nWhile unsatisfying, it is often necessary to have a separate actor network in continuous domains due to the difficulty of maximizing the critic network with respect to the action space. Prior work [3,4,5,6] found this separate ''actor'' network amortizes the critic maximization and can even improve generalization with this extra distillation step. Future work could explore combining TMD with approaches like NAF [7], which use a critic parameterization that is amenable to analytic maximization over actions, though in practice such approaches often suffer from poor expressivity.\n\n---\n\n[1] Wang, T. et al., 2023. ''Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning.'' ICML\n\n[2] Myers, V. et al., 2024. ''Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making.'' ICML\n\n[3] Lillicrap, T. P. et al., 2016. ''Continuous Control With Deep Reinforcement Learning.'' ICLR\n\n[4] Fujimoto, S. and Hoof, H., 2018. ''Addressing Function Approximation Error in Actor-Critic Methods.'' ICML\n\n[5] Park, S. et al., 2024. ''Is Value Learning Really the Main Bottleneck in Offline RL?.'' arXiv:2406.09329\n\n[6] Eysenbach, B. et al., 2022. ''Contrastive Learning as Goal-Conditioned Reinforcement Learning.'' NIPS\n\n[7] Gu, S. et al., 2016. ''Continuous Deep Q-Learning With Model-Based Acceleration.'' arXiv:1603.00748"}, {"review_id": "6y3OunOVrN", "reviewer": "Reviewer_BR73", "summary": "I understand this work to be a follow-up of [1], which is in turn a follow-up of [2], which again can be seen as a follow-up of previous works such as [3] and [4], etc.\n\nThe offline setting prohibits direct access to the exploration policy.\n\n======\n\n\n[1] Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making\n\n[2] Distributional Distance Classifiers for Goal-Conditioned Reinforcement Learning\n\n[3] Search on the replay buffer: Bridging planning and RL\n\n[4] model-based visual planning with self-supervised functional distances", "questions": "How many seeds for ablation tests?\n\nI read through the Appendix in the supplementary ZIP. I didn't find any empirical performance metric on the experiments that are beyond success rates. For distance learning, maybe should pick an environment that can **demonstrate the convergence to true distance values**.\n\nAlso, one of this paper's highlight is the compilation of the losses which enforce the invariance. **Why not show curves of convergence on those soft constraints?** I get that this is partially shown in the ablation tests, but it does not really tell if the two infinity norms are indeed regressed to zero (which should correlate with higher performance?).\n\n\n====================\n\n\nPrevious work [2] argued that Monte Carlo temporal distances would cause trouble when being applied in stochastic environments as a basis for extracting policies. Also, the learned distances are not quasi-metric (do not obey triangle inequality). These are some of the motivations for this work. However, it seems that most of the concerns are based on two handcrafted stochastic MDPs that would give the MC temporal distances trouble, especially for two consecutive states connected by a stochastic transition. This however, does not necessarily apply to HER-based methods, where the goals and the next states are sourced independently, as used in for instance [5] and [6] by Bengio's group. Take the Figure 1 MDPs in [2] for example, if the update rule for MC-based temporal distances is Q(s, a_t, g) <- -1 + indicator(next state is goal) * Q(s_t+1, a_t+1, g) + indicator(next state is terminal & not g) * \\infty, the estimated distance would NOT be 1 for the distance between the consecutive states. Also, if the purpose of the auxiliary Q function is to estimate distances, there is little reason for a discount factor to be used at all. If my understanding about the temporal distances learned by [5] and [6] are correct, their more simple setup could be much more versatile for the online settings, where a policy can be accessed. **I suggest that these new developments should be discussed in the related works.**\n\n\n[5] Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learning\n\n[6] Rejecting Hallucinated State Targets during Planning", "limitations": "I find that the authors are not careful about the cited works. Just to give a few examples:\n\nLine 27 [10]: did the HER paper really use Q learning and stitch trajectories to find shortest paths?\n\nLine 29 [14] doesn't seem to be a representative method of MC based distance estimation. Please replace.\n\nThese may not be exhaustive, but I would wish that the authors do a full scan of the paper and correct any mistakes like these **and return to me the list of findings**. \n\n\nLine 114-116 didn't explain how s and (s, a) got into the input. Previously, only state-to-state distances were discussed but suddenly it shifted to the union of state space and state-action tuples. Btw, it would be less confusing to write (SxA)uS\n\nLine 266: should be figure 3. \n\n**I am willing to change my rating to accept if the authors properly address my concerns regarding the discussion of related works and improve on the writing of the paper.**", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "This paper is technically sound. However, the experiments are insufficiently convincing. There are some errors when discussing related works. The clarity and the flow of the paper need to be improved.", "was_revised": true, "final_justification": "I'm still concerned about the under-polished state of the submission, since I have no access to the revision. Changing the rating to borderline accept, but I am really on the fence. Given how divided the reviews are, sorry I cannot be the deciding vote!", "rebuttal": "Thank you for your thoughtful comments. Your primary concerns seem to pertain to writing and presentation. We have revised the manuscript based on your comments, and made substantial revisions to improve clarity of the writing and figures. **Do these changes address your concerns? Please let us know if there are any additional points of concern or revisions you would like.**\n\n> I read through the Appendix in the supplementary ZIP. I didn't find any empirical performance metric on the experiments that are beyond success rates. For distance learning, maybe should pick an environment that can demonstrate the convergence to true distance values.\n\nThank you for the suggestion. When visualized with a heatmap, the distances learned by TMD in `antmaze-large-stitch` are consistent with the maze structure. We will include this heatmap as a figure in our next revision.\n\n> Also, one of this paper's highlight is the compilation of the losses which enforce the invariance. Why not show curves of convergence on those soft constraints?\n\nGood idea, we will also include these plots. We cannot include images here, but during training we have verified that the individual loss components converge.\n\n> Previous work [2] argued that Monte Carlo temporal distances would cause trouble when being applied in stochastic environments as a basis for extracting policies. ... This however, does not necessarily apply to HER-based methods, where the goals and the next states are sourced independently, as used in for instance [5] and [6] by Bengio's group. \n\n> ... . If my understanding about the temporal distances learned by [5] and [6] are correct, their more simple setup could be much more versatile for the online settings, where a policy can be accessed. ... I suggest that these new developments should be discussed in the related works.\n\nThank you for noting these connections; we will add discussion of these papers to the related work. As you note, MC methods may be suitable for temporal distance estimation during online training. We focus on the setting of offline / off-policy learning, where TMD is able to provide the additional structure needed for optimality.\n\nAside: practical methods based on HER like [1] often introduce an optimistic bias in value functions in stochastic MDPs because the relabeled goals (viewed as a random variable) convey information about how the environment dynamics leading to them resolved. The classical notion of hindsight relabeling in [2] is theoretically correct only because it relabels with all goals. See Appendix I of [3] for more discussion. Distance learning methods like TMD avoid this issue.\n\n> How many seeds for ablation tests?\n\nEach experiment used six seeds. We have clarified this in the revised manuscript.\n\n> I find that the authors are not careful about the cited works. ... I am willing to change my rating to accept if the authors properly address my concerns regarding the discussion of related works and improve on the writing of the paper.\n\nWe have made the suggested revisions discussed below. Please let us know if there is anything else you would like us to revise.\n\n> Line 27 [10]: did the HER paper really use Q learning and stitch trajectories to find shortest paths?\n\nWe cited the HER paper here because it used DQN [4] and DDPG [5] on top of hindsight relabeling. As offline GCRL methods, these implicitly perform stitching over the dataset. We are happy to replace this reference with citations for DQN and DDPG as methods that stitch trajectories, and note that prior work has connected the idea of stitching to shortest paths [6,7,8]. Would this address your concern?\n\n> Line 29 [14] doesn't seem to be a representative method of MC based distance estimation. Please replace.\n\nWe cited Dosovitskiy & Koltun (2017) here because they use Monte Carlo estimation of goal-reaching values, which are proportional to distances (see Paragraph 3 of Section 2 in that paper). We will clarify that it is estimating MC values, and additionally cite CRL [9] and CMD [8]. \n\nThe updated sentence reads: ''Monte Carlo methods [10,9] can directly learn goal-reaching value functions, which can be connected to temporal distances [8], but their ability to find *shortest* paths remains limited.''\n\n> Line 114-116 didn't explain how s and (s, a) got into the input. Previously, only state-to-state distances were discussed but suddenly it shifted to the union of state space and state-action tuples. Btw, it would be less confusing to write (SxA)uS\n\nWe have modified the text to motivate the definition of distances over state-action pairs and states, with the $(S \\\\times A) \\\\cup S$ notation. Informally, d(s,g) and d((s,a),g) are analogous to V(s;g) and Q(s,a;g) in the standard GCRL formulation. To stitch trajectories, the distance must also be defined for goal-action pairs (see [8], Figure 4 for visual motivation). In the revised text, we note this after the definition in Eq. (7), and clearly state that we are extending the definition of the successor distance in [8] to support the larger $(S \\\\times A) \\\\cup S$ domain.\n\n> These may not be exhaustive, but I would wish that the authors do a full scan of the paper and correct any mistakes like these and return to me the list of findings. \n\nWe have substantially revised the manuscript to improve various aspects of writing and presentation. We cannot include the updated text and figures here, but we list some of the changes below.\n\n#### Corrections\n\n- Fixed references: lines 128, 135, 266\n\n- Fixed sign error in Eq. (23): $\\\\log \\\\gamma \\\\Rightarrow -\\\\log \\\\gamma$\n\n- Missing $-\\\\log\\\\gamma$ in Eq. (25), now reads $$\\\\mathcal{L}\\_{\\\\mathcal{T}}(\\\\phi, \\\\psi ;\\\\{s\\_i, a\\_i, s\\_i^{\\\\prime}, g\\_i\\\\}\\_{i=1}^N)=\\\\sum\\_{i=1}^N \\\\sum\\_{j=1}^N D\\_T(d\\_{\\\\mathrm{MRN}}(\\\\phi(s\\_i, a\\_i), \\\\psi(g\\_j)), d\\_{\\\\mathrm{MRN}}(\\\\psi(s\\_i^{\\\\prime}), \\\\psi(g\\_j))-\\\\log \\\\gamma)$$\n\n- Added $\\\\psi()$ around $g\\_j$ in Eq. (28)\n\n- Error bars in Figure 3 were standard deviation, not standard error. We have changed them to report standard error, and clarified in the caption. With this change it is clear that the ablation results are statistically significant.\n\n- Table 5 should read:\n\n| Loss              | Success Rate     |\n| ----------------- | ---------------- |\n| $D\\_T$ (Ours) | **29.3** (±2.2) |\n| $D\\_{\\\\ell\\_2}$ | 16.1 (±1.9)      |\n| $D\\_{\\\\rm BCE}$ | 15.1 (±1.9)      |\n\n#### General improvements\n\n- Expanded evaluations with 8 new environments and an additional baseline (CMD [8]). See the response to reviewer cpYD for the full results.\n\n- Updated Figure 1 to provide clearer intuition for how the invariances imposed by the method and architecture tighten the $C(\\\\pi)$ distance from CRL to yield the optimal distance.\n\n---\n\n[1] Andrychowicz, M. et al., 2017. ''Hindsight Experience Replay.'' NIPS\n\n[2] Kaelbling, L. P., 1993. ''Learning to Achieve Goals.'' IJCAI\n\n[3] Eysenbach, B. et al., 2021. ''C-Learning: Learning to Achieve Goals via Recursive Classification.'' ICLR\n\n[4] Mnih, V. et al., 2013. ''Playing Atari With Deep Reinforcement Learning.'' \n\n[5] Lillicrap, T. P. et al., 2016. ''Continuous Control With Deep Reinforcement Learning.'' ICLR\n\n[6] Ghugare, R. et al., 2024. ''Closing the Gap Between TD Learning and Supervised Learning—a Generalisation Point of View.'' ICLR\n\n[7] Wang, T. et al., 2023. ''Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning.'' ICML\n\n[8] Myers, V. et al., 2024. ''Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making.'' ICML\n\n[9] Eysenbach, B. et al., 2022. ''Contrastive Learning as Goal-Conditioned Reinforcement Learning.'' NIPS\n\n[10] Dosovitskiy, A. and Koltun, V., 2017. ''Learning to Act by Predicting the Future.'' ICLR"}, {"review_id": "BKqtTggX5l", "reviewer": "Reviewer_CFjL", "summary": "The paper proposes TMD, an offline GCRL method using temporal distance representations.\nTMD aims to recover an optimal temporal distance by enforcing three constraints: (1) triangle inequality (enforced by quasimetric), (2) action invariance, (3) SARSA-like distance constraint ($d((s,a), s’) = -\\log \\gamma$).\nThe paper includes a theoretical analysis showing that, under these constraints, TMD recovers an optimal successor distance.\nEmpirically, TMD demonstrates strong performance on OGBench, outperforming other offline GCRL methods in 6 out of 8 tasks.", "questions": "**Q1. Final formula for modified successor distance**\n\n* Could you clarify the motivation behind the final formula involving the pair y=(g,a)? \n* I couldn't locate a corresponding discussion or derivation in [1].\n\n**Q2. Visualization of learned Q-values**\n\n* It would be helpful to include a visualization of the learned Q-values or successor distances (similar to Figure 2 in [1]) to better understand the learned representations. \n\n**Q3. Evaluation in stochastic environments**\n* Given TMD’s design to handle stochastic transitions, could you provide results for environments with added transition noise or inherent stochasticity?\n\n**Clarifications & Minor typos**\n* (Eq 7) Does $K \\sim \\gamma$ mean $K \\sim Geom(1-\\gamma)$?\n* (Eq 24) If we are using Itakura-Saito distance, shouldn’t it be exp(d-d’) - (d-d’) - 1? Or is it because we are not using the gradient of d’?\n* (Eq 28) \\phi(s_i, \\pi(s_i)) -> \\phi(s_i, \\pi(s_i, g_j))\n* (L128) P\n* (L135) 11.5 \n* (L266) in 3 -> in Figure 3", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n**S1. Strong Experimental results**\n* TMD achieves state-of-the-art performance on OGBench, a challenging benchmark for offline GCRL, outperforming all other methods on 6 out of 8 tasks.\n\n**S2. Solid theoretical support**\n* The paper provides a theoretical justification that the three proposed constraints lead to an optimal successor distance metric (assuming full state-action coverage by the policy)\n* Also, the SARSA-like formulation of the method makes TMD easier to use.\n\n**S3. Ablation studies**\n* The ablation study clearly shows the contribution of each component of TMD.  \n* The ablation study also provides insights on details of TMD (e.g., utilizing Bregman divergence in the log-space, stop-gradient)\n\n&nbsp;\n\n**Weaknesses:**\n\n**W1. Slight mismatch in evaluation**\n* TMD is designed to support stochastic transitions, which is an important advantage over QRL.  \n* However, the OGBench environments used in evaluation is deterministic (please correct me if it’s wrong). \n* This raises questions about whether the stochasticity-aware aspects of TMD are being fully utilized.", "was_revised": true, "final_justification": "All of the questions were clarified, especially the question with the stochastic dynamics.\nWhile writing of the paper is concerning, I think it can be improved without requiring major changes (e.g., without introducing new concepts or sections) in the final manuscript.\n\nThus, I recommend accepting this paper.", "rebuttal": "Thank you for your thoughtful comments and feedback. We will address some of your concerns below. Please let us know if you have any additional questions.\n\n> TMD is designed to support stochastic transitions, which is an important advantage over QRL. However, the OGBench environments used in evaluation is deterministic (please correct me if it's wrong). \n\n> Q3. Evaluation in stochastic environments\n\nThe `teleport` environments feature portals that transport the agent to random locations. This makes them ideal for studying nontrivial stochastic dynamics. \n\n> Q1. Final formula for modified successor distance. Could you clarify the motivation behind the final formula involving the pair y=(g,a)? I couldn't locate a corresponding discussion or derivation in [1].\n\nUnlike the successor distance in CMD [1], we learn a distance over both states $s$ and state-action pairs $(s,a)$. This lets us reason about both ''Q'' values $d((s,a),g)$ and ''V'' values $d(s,g)$. Because the distance is a function over $((S \\\\times A) \\\\cup S) \\\\times ((S \\\\times A) \\\\cup S)$, we also need a sensible definition of the distance to a goal-action pair, $d(x, (g,a))$. This is what Eq. (7) provides—for a policy $\\\\pi$, the distance under $d^\\\\pi\\_{SD}$ from $x \\\\to (g,a)$ is the distance from $x \\\\to g$ plus $-\\\\log (a \\\\mid g)$. \n\nWe have revised the paper to clarify that we are extending the definition in [1] to handle this case.\n\n> Q2. Visualization of learned Q-values. It would be helpful to include a visualization of the learned Q-values or successor distances (similar to Figure 2 in [1]) to better understand the learned representations.\n\nWe have generated a heatmap of successor distances for the antmaze-large-stitch environment. We will include this as an additional figure in the revised paper.\n\n#### Clarifications & Minor typos\n\n> (Eq 7) Does $K \\\\sim \\\\gamma$  mean $K \\\\sim \\\\operatorname{Geom}(1-\\\\gamma)$?\n\nYes, we will fix this.\n\n> (Eq 24) If we are using Itakura-Saito distance, shouldn't it be exp(d-d')—(d-d')—1? Or is it because we are not using the gradient of d'?\n\nCorrect, we can drop that term since we are minimizing the divergence wrt d.\n\n> (Eq 28), (L128), (L135), (L266)\n\nThank you for noticing these. We will correct these errors in the revision.\n\n---\n\n[1] Myers, V. et al., 2024. ''Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making.'' ICML"}]

Author rebuttal:

Meta-review: This paper proposes Temporal Metric Distillation (TMD), an offline goal conditioned reinforcement learning method that learns temporal distance representations under quasimetric constraints. When combined with action invariance and quasimetric parametrization, the proposed framework recovers optimal goal reaching policies from offline RL data. The authors provide theoretical justification and convincing empirical results on OBBench compared against GCBC, GCIQL, GCIVL, CRL, QRL baselines.

Strengths:

The paper tackles an important problem in offline goal conditioned reinforcement learning addressing limitations of prior metric learning approaches in handling stochastic environment dynamics.

Authors provide solid theoretical grounding giving insights on conditions under which optimal successor distances are recovered.

Experiments show convincing empirical performance compared to based goal conditioned RL methods (table 1).

Weaknesses:

Some reviewers pointed out the method section is not easy to follow noting unclear notations, typos, and unpolished exposition of equations and that proofs are sometimes vague or inconsistent. One reviewer in particular emphasized that the proofs are not written formally and clearly and had to guess the authors' intent at each step of the proof.

Few details about the evaluation were not clearly stated in the submission. Details regarding performance in stochastic environments, convergence of individual loss terms, visualizations of learned distances, number of seeds used for the experiments were initially missing but the authors added the clarifications in rebuttal.

The author rebuttal provided more experiments (including CMD baseline), ablation analyses, and implementation details. These responses addressed the majority of reviewer concerns.

The decision is to accept. This paper makes a significant contribution to offline goal conditioned reinforcement learning. I strongly encourage the authors to incorporate the feedback in the camera ready version. Please make best efforts to improve the clarity of notation, formal and rigorous presentation of proofs as per reviewer `cpyd` (esp. eqn. 34), underlying rationale of the methods (esp. eqn 7), expand the related works section promised in rebuttal.
Decision: Accept (poster)