Scaling Goal-conditioned Reinforcement Learning with Multistep Quasimetric Distances
Authors: Bill Zheng
Vivek Myers
Benjamin Eysenbach
Sergey Levine
Venue: ICLR 2026 Poster (2026)
Keywords: Goal-conditioned reinforcement learning
quasimetrics
robotics
Primary area: reinforcement learning
TL;DR: Applying multistep value backup in a quasimetric distance architecture can yield surprisingly strong results.

Abstract: The problem of learning how to reach goals in an environment has been a long-
standing challenge in for AI researchers. Effective goal-conditioned reinforcement
learning (GCRL) methods promise to enable reaching distant goals without task-
specific rewards by stitching together past experiences of different complexity.
Mathematically, there is a duality between the notion of optimal goal-reaching
value functions (the likelihood of success at reaching a goal) and temporal dis-
tances (transit times states). Recent works have exploited this property by learning
quasimetric distance representations that stitch long-horizon behaviors using the in-
ductive bias of their architecture. These methods have shown promise in simulated
benchmarks, reducing value learning to a shortest-path problem. But quasimet-
ric, and more generally, goal-conditioned RL methods still struggle in complex
environments with stochasticity and high-dimensional (visual) observations. There
is a fundamental tension between the local dynamic programming (TD backups,
temporal distances) that enables optimal shortest-path reasoning in theory and the
statistical global MC updates (multistep returns, suboptimal in theory). We show
how these approaches can be integrated into a practical GCRL method that fits a
quasimetric distance using a multistep Monte-Carlo return. We show our method
outperforms existing GCRL methods on long-horizon simulated tasks with up to
4000 steps, even with visual observations. We also demonstrate that our method
can enable stitching in the real-world robotic manipulation domain (Bridge setup).
Our approach is the first end-to-end GCRL method that enables multistep stitching
in this real-world manipulation domain from an unlabeled offline dataset of visual
observations.

Published as a conference paper at ICLR 2026
MULTISTEP QUASIMETRIC LEARNING FOR SCALABLE GOAL-CONDITIONED REINFORCEMENT LEARNING
Bill Chunyuan Zheng!
Vivek Myers'
Benjamin Eysenbach"
Sergey Levine'
UC Berkeley
Princeton University
ABSTRACT
Learning how to reach goals in an environment is longstanding challenge in AI, yet reasoning over long horizons remains challenge for modern methods. The key question is how to estimate the temporal distance between pairs of observations. While temporal difference methods leverage local updates provide optimality guarantees, they often perform worse than Monte Carlo methods that perform global updates (e.g., with multi-step returns), which lack such guarantees. We show how these approaches can be integrated into practical offline GCRL method that fits quasimetric distance using a multistep Monte-Carlo return. We show our method outperforms existing offline GCRL methods on long-horizon simulated tasks with up to 4000 steps, even with visual observations. We also demonstrate that our method can enable stitching in the real-world robotic manipulation domain (Bridge setup). Our approach is the first end-to-end offline GCRL method that enables multistep stitching in this real-world manipulation domain from unlabeled offline dataset of visual observations and demonstrate robust horizon generalization.
INTRODUCTION
S
dl So s
dls $8)
So
d(sp 8)
Quasimetric Architecture: d(s0: 8) d(s s,)) + d(s sg)
Sample waypoints s, from future states
Figure 1: In this paper, we present Multistep Quasimetric Estimation (MQE). Unlike prior work in quasimetric distance learning that use single-step TD updates (Wang al., 2023) or contrastive bearning-based Monte-Carlo updates (Myers et al., 2024), MQE is the first work to incorporate multistep returns with real-world success.
It is natural for humans to inherent ideas of distances to represent task progress: GPS will tell you how far you are from the destination and a cookbook will tell you how long a recipe will take. Humans can solve tasks by taking the shortest route possible (stitching) and combining several learned tasks together in sequence in a new environment (combinatorial generalization). The AI problem of reaching goals presents a rich structure (formally, optimal substructure property) that can be exploited to decompose hard problems into easy problems, and reinforcement learning (RL) has been used address such problems. However, many past attempts leveraging this property in modern nigh-dimensional. stochastic settings still need much work. Current offline approaches tend separate TD learning (local updates) (Mnih al., 2013; Kostrikov al., 2022; Kumar al., 2020)
Website and code: httpa://nge-pan papeer.pppep...e .iterrggtttbb... io

---

Published as a conference paper at ICLR 2026
and MC learning (global value propagation) (Eysenbach al., 2022; Myers al., 2024); TD learning can theoretically recover the optimal Q function (Q"), while MC methods can only recover the behavior Q function (Qz)), but tend to work well in practice.
However, the effectiveness of both the temporal difference and Monte Carlo methods degrades, often from a combination of increasing the horizon length for TD methods (Park al., 2025b) or difficulties in finding the optimal temporal distance (Park et al., 2024a). These combined challenges present an intriguing opportunity find methods that can leverage the advantages of both approaches, where one can perform both local and global value propagation at once.
Our main contribution lies in Multistep Quasimetric Estimation (MQE), offline GCRL method that incorporates both multistep value learning and quasimetric architectures without needing explicit hierarchy. By leveraging such unique combinations of the benefits of TD and MC methods, MQE allows the learned policy to (i): display much stronger level of horizon gemeralization compared to previous methods, which allow us demonstrate the desired "stitching" behavior, (ii): provide stable method that can extract strong goal-reaching policies even from noisy data, and (iii): such stability in training allows it to be applied real-world robot learning problems without additional design choices. To our knowledge, MQE is the first method capable of taking advantage of multistep TD returns with global value propagation through quasimetric architectures. MQE achieves SoTA performance tasks that require complex control and timesteps respectively), and in real world robotic settings, MQE displays compositional generalization behaviors that not seen in previous RL algorithms.
reasoning (up to 21 DoF and 4000
2
RELATED WORK
Our work build upon previous works in offline RL and temporal distance learning.
Goal-conditioned Reinforcement Learning. Recent work has studied the stitching and horizon generalization capabilities of GCRL. Park et al. (2025b) show that while offline RL is easy to scale on short-horizon tasks, it is difficult to learn long-horizon tasks that require more complex reasoning within the agent, which can easily deviate from the optimal distance due to compounding TD) errors. Other findings have shown that combinatorial generalization and stitching do not necessarily need dynamic programming (Ghugare et al., 2024; Brandfonbrener et al., 2023; Garg et al., 2023), which gives promise to using simpler methods for learning stitchable policies.
Offline RL for Robotics. We demonstrate offline GCRL scaling to real-world robotic tasks. Although reinforcement learning has been used to obtain highly capable specialist policies in various embodiments (Luo et al., 2025; Ball et al., 2023; Seo et al., 2025; Smith et al., 2023), behavior cloning still remains the most capable method for training generalist policies (O (O'Neill et al., 2024). Efforts have been made to allow self-supervised and offline RL in robotics (Zheng et al., 2024), however, directly training a policy with offline RL remains difficult, as many researchers have instead used other ways to incorporate RL, such as rejection sampling (Nakamoto et al., 2025; Wagenmaker et al., 2025) or data set curation (Mark et al., 2024; Xu et al., 2024). We show that our method can use multistep quasimetric learning to obtain effective policies for real-world robotic tasks, outperforming non-GCRL methods on long-horizon manipulation.
Multistep RL. RL using multistep returns has been widely used in online RL and offline-to-online RL settings (Li et al., 2025; Tian et al., 2025). This is desirable because in on-policy settings, performing RL with multistep return is theoretically correct and yields better performance (Munos et al., 2016; Asis et al., 2018; Schulman et al., 2016). However, in an offline setting, the theory behind such correctness breaks down (Watkins, 1989), as the learning paradigm becomes an inherently off-polic manner. Methods that use the entire trajectory in a Monte Carlo manner and attempt to recover the Q function can also be seen as a multistep RL algorithm (Eysenbach et al., 2022). However, such approach can only recover the behavior Q function Q3 (Myers et al., 2024; Eysenbach et al., 2022).
Temporal Distance Learning. Our work is closely related by works that focus on successor representations (Dayan, 1993; Blier et al., 2021) and uses them to lear the probability of reaching goals. Previous works have shown that by using contrastive learning (Myers et al., 2024), one can
2

---

Published as a conference paper at ICLR 2026
recover a temporal distance suitable for goal-reaching. Other works have also shown that using contrastive learning as a way to parameterize distance learning may also recover behavior Q function (Eysenbach al., 2022). Previous works (Eysenbach et al., 2021; Myers et al., 2025c) have shown that is possible to learn an optimal goal-reaching policy and Q function in theory. Our work differs in that learn Bellman optimal Q function under the bias of behavioral dynamies, This tradeoff helps us achieve considerable empirical gains in long-horizon goal-reaching tasks, and shows compositionality in real-world robotic learning problems.
PRELIMINARIES
In this section, we define temporal distances and our learning objective.
Notation. We consider a controlled Markov process (CMP) M with state space S, action space A, transition dynamies P(s' a). and discount factor y. We consider goal-reaching policies Ta 8,9) S x S + A E II. We denote the behavior policy as In lieu of rewards, we optimize the maximum discounted likelihood of policy reaching the goal, in which we can represent the goal-conditioned Q function and the value function as:
0 (8, a) A Be Jur
P(s = 8,00 =a)
(s) Bos Juil
2" P(s = 50 =s)
(2)
Equivalently, define the optimal Q-function (7 a) - max (7 a). Previous work have shown that using contrastive leaning (Eysenbach et al., 2022; van den Oord et al., 2019) can recover the behavior distance, but not the optimal distance.
Similarly, prior work on MC learning (Myers et al., 2024; Eysenbach et al., 2022; 2021) has demonstrated that future states can be used goals. We use geometric distribution described in Eq. (3), This allows us to classify any future state in trajectory as goals, providing a robust way of learning goal-reaching policies.
s7 A S K Geom 1-)-
(3)
Quasimetric distance representations. Traditionally, offline RL algorithms use neural networks to represent the critic and value function Qg(s, a) and (s) (Kumar al., 2020; Kostrikov al., 2022).
Separately, other works have been using dot products Q(s, a. è (pss T (g)) (Eysenbach et al., 2022; Zheng et al., 2024) geometric norms [|ş(()) ( l for suitable values of (Eysenbach al., 2024; Park et al., 2024c). We use quasimetric architectures (Wang et al., 2023; Pitis al., 2020) to parameterize our value functions.
Formally, the space of quasimetric distances 2x over set X' is defined as the set of functions d: X X - R220 that satisfy the following properties for all x, E X (Cobzaş, 2013):
(i) Non-negativity: d(x; y) 2
Identity: d(ær x) = 0.
(iii Triangle Inequality: d(æ, 2) S d(x, y) + d(y, 2).
We will also the notation Dx to denote the more general class of distances that satisfy only (i) and (ii).
To enforce the quasimetric properties of the successor distances, we use the Metric Residual Network (MRN) architecture (Liu et al., 2023), which parameterizes space of quasimetrics in terms of a learned representation, shown in Eq. (4). MRN splits the representation into N equally sized components, and in each part, takes the sum of an asymmetric component (maximum of ReLU) and symmetric component (l2 norm) of the difference between the two embeddings and
3

---

Published as a conference paper at ICLR 2026
dARN (z, y)
max 1 ykM+m) + | 3kM++ml
(4)
m I...M
Given any function f 2- RNM Eq. (4) defines quasimetric distance on X. Liu et al. (2023) show that for sufficiently large M, this parameterization is universal. Thus, fitting a quasimetric distance function can be accomplished by fitting daRN (f (t)) for sufficiently expressive class of representations: f E @. We make use of this architecture with a learned representation over AUS to express our goal-conditioned Q function Qy a) and value function (g(ss as distances between states/state- action pairs and goals, as demonstrated in (Myers et al., 2025c).
4 MULTISTEP QUASIMETRIC ESTIMATION (MQE)
In this section, we develop MQE framework based on quasimetric architectures, which can be broken down into 3 parts: (1) multistep backup under quasimetric architecture, (2) imposing action invariance as form of value learning. (3) practical implementation details for extracting the goal-reaching policy. To the best of our knowledge, MQE is the first method to effectively combine multistep returns with a quasimetric distance parameterization, and achieves superior results to previous methods that use TD returns, contrastive learning for value estimation, hierarchical methods.
Definitions. We will define distance (quasi)metric states and state-action pairs that is proportional to the goal-conditioned Q and V functions (Myers et al., 2024): (,( (()
(s(s) Vú (g(g)e-d(s.))
(5)
We learn parameterized form of this distance with learned state and state-action (4()) Raa and Ø(s, a.) E NM representations and a quasimetric distance function (4):
d(ss,a), - dRRN d a), (g)), d(s,g) - duENN (v((), (g))-
(6)
4.1 MULTISTEP RETURNS WITH QUASIMETRIC ARCHITECTURE
The first design principle of MQE is to use multistep returns under a quasimetric architecture. To that end, start with the fitted onestep iteration that operates quasimetric distance representations, in which t denotes regressing from the LHS to the RHS:
cd(((a,))a) + -(sr-P(dlss)))---- e d(e'.))) a)).
This is similar to optimizing the critic in traditional RI. (Kostrikov et al., 2022; Lillicrap et al., 2019), the main difference being that we use a quasimetric architecture to represent the Q and value function. Due the quasimetric architecture of the network, the reward signal is embedded within the parameterized distance, as Vg (s) is defined to have a value of Vg(g) > when g. We denote this regression objective T.
Our insight for the T objective described Eq. (7) is that, instead of applying this invariance to only the future state s' ~ P(s' 8, a), we can extend this principle any state between the current state and goal. This transforms a onestep optimization into a on-policy multistep optimization procedure. "waypoint" between the current state
To do so, we first define the shorthand s, which refers and goal state. Empirically, we find that using combination of Bernoulli distribution and geometric distribution (described in Eq. (8)), capped at the index of the future state work the best.
k ~ minome - K) with probability 1 - p. with probabiiiyyy
2F + St for
(8)
We now optimize the same objective in Eq. (7), but across any such waypoint we sample. To account for the multistep nature of this objective, we modify Eq. (7) below to accommodate such changes.
d((e.a).)) "((o . dl.r"19)
4

---

Published as a conference paper at ICLR 2026
We denote this new objective T3, the future state of the sample is restricted by trajectories generated by Tg. This is similar to the R-step returns in prior work (Sutton, 1988; Munos et al., 2016; Li et al., 2025), although
do not sample future states with a fixed number of steps. In practice, can use any loss function to make the IHS equal to the RHS (in expectation) concerning Eq. (7) and form of Bregman divergence with LINEX losses (Parsian and Kirmani, 2002), as
Eq. (9). We it does not incur vanishing gradients when the two distances have become close in value (Banerjee al., 2004; Myers al., 2025c):
Dr(d, d') & exp(d d'
(10)
Using this loss, can concretely define both LTg and LT that can optimize the objectives of and TB.
N
N Dr((((s),)))) as), 9;)) - Kg log7). (1I)
LTg (0s di (8i, as, 8 .g. W,
These two losses, when applied to our quasimetric network, will allow us to propagate the value in either onestep or multistep manner.
relationship between p, multistep backup Ta, and onestep backup T. By changing we can adjust the distance (in expectation) from the waypoint from the current state, as = means that T3 equivalent to (as k 1). We demonstrate Section 5.3 why only using Cy is insufficient for learning good distance representation and goal-conditioned policy.
Remark:
4.2 LEARNING VALUE FUNCTIONS VIA ENFORCING ACTION INVARIANCE
With respect to quasimetric networks, (Myers et al., 2024) has demonstrated that if the training data is collected using Markovian policy, then the optimal critic should observe the property of Action Invariance. As a result, the optimal critic should observe the following property (Sutton and Barto, 2018):
(s) = max ()) aEA
(12)
This be satisfied, under construction, if d(ss, (8, a)) for each action (Myers et al., 2025c) (action invariance). Since our construction Q and value function do not observe this property, need enforce it using gradient descent, described in Eq. (13).
d( d(v(s). ((()))(ss))))) +
(13)
We remark that this is also similar to the value loss seen in other offline RL methods that employ both a value and Q function. namely IQL (Kostrikov al., 2022) and TMD (Myers et al., 2025c), and the desired outcome of Eq. (13) is similar to the value loss described in IQL when R 1. One common failure case of basic regressions, such as using L1 L2 regression for Eq. (13), is that the algorithm quickly converges to trivial solution in which s0ss, a) = (b(s) 0. which fails to learn any valuable distance measures. As a result, TMD requires hyperparameter Ç to regulate the magnitude of gradients of the invariance loss. To counteract this, we use a smoother formulation that allows more relaxed enforcement of Eq. (13) when the violation is low in magnitude. As a result, we can define LI as:
N
(s, (s,
Σ {-==1
(e ((dd -
(14)
Now the loss will scale with the magnitude of such deviation, which removes the need of hyperpa- rameter tuning for appropriate multiplier well stabilizing training dynamics.
5

---

Published as a conference paper at ICLR 2026
4.3 POLICY EXTRACTION
We extract the goal-conditioned policy T(8 S² - A using the learned distance using the behavior-regularized deep-deterministic policy gradient (DDPG BC) (Fujimoto and Gu, 2021):
N
E
Σ 1,=1
d((s,, ((,,, 9,) a log (a 9i)
(15)
Given that a smaller d measure correspond to higher Q value, as defined in Eq. (5), we can maximize the Q values by minimizing the distance produced by our quasimetric network. We tune the BC coefficient a per environment, We provide more hyperparameter details in Section D.
4.4 IMPLEMENTATION DETAILS & ALGORITHM
We concisely define our final learning objective in Algorithm 1. Unlike previous works hierarchical RL (Nachum al., 2018; Park et al., 2024b), randomly sample these waypoints, and we learn single critic Q that operates on S and A and single goal-reaching policy TT. As a result, our method does not contain any additional components and is simpler to implement than these hierarchical methods.
Algorithm I: Multistep Quasimetric Estimation
Require: Dataset D, Batch size B, training iteration T, Probability
1: Initialize quasimetric network Q with parameters (f, 1)), goal-reaching policy Tp
2: for = 1...T do
3: Sample (st, ag, 5/ S," 9: | ~D D
(8)
4: Update Q with multistep backup by minimizing LT, (1p, vi; (st, az sf" s ()
(9)
5: Update Q with action invariance constraints by minimizing Cr(p, di {s, (=)))
(14)
6: Update policy Th with DDPG+BC by minimizing C (Tpi (8i, as 9i )
(15)
7:
return TT
4.5 ANALYSIS
The key theoretical result show is that this algorithm is able to perform policy improvement top of the behavior policy Ty in tabular setting under standard assumptions. The tabular version of MQE can be viewed executing three steps:
1. Minimize Eqs. (11) and (14) under the data distribution from Tg:
To min [(,,a)>p" grup"""
Dr (d((s, 2pt e 19)) + (e--d(s((s,)) - 1)
dEDsxAus
(16)
Constrain the distance to be a quasimetric. Mathematically, this can expressed as projecting the distance into quasimetric space via path relaxation operator P Dx Qx (Myers et al., 2025a), defined as
P(d)(a, 2) - min [(x,, u) d(y,,))
(17)
NEX
Applying this to the learned distance, we construct = Pdd Extract the policy via Eq. (15):
min E (s.a) " [(((s, ((((s)(s.)))).))
(18)
This statement is formalized in Theorem below.
Theorem Suppose behavior policy ng induces full support over state action pairs (s, a) d in tabular setting. We fit distance by minimizing Eg. (16) and extract policy by minimizing Eq. (18).
Then the extracted policy satisfies V (s) Z *7 for all 8, g E S.
The proof is provided in Section C.

---

Published as a conference paper at ICLR 2026
scene-play
viausl-antmaze-glan
antmaze-colossal
puzzle-4x4
Figure 2: Tasks from various state and pixel-based environments for OGBench. Antmaze- 50% larger than any other mazes available on OGBench, and in stitch datasets, test the agent's ability generalize over horizon that up 1000% longer.
Aggregate OGBench Success Rates
Horizon Generalization AntMaze
Success Rate (%)
30
80
60
20
40
10
20
0
MQE
TMD
CMD
CRL
QRL
GCIQL
HIQL
B-SCCBBC
medium
large
giant
colossal
Maze Size
Method
MQE (Ours)
TMD
CRL
HIQL
(a) The overall success rate across a set of most challenging OGBench evaluation tasks
(b) Horizon generalization on dataset with training trajectories that are only 4m long. This means that colossal mazes have tasks that have horizons that 1000% longer than those training set. MQF also retains the hest horizon generalization skills compared to previous methods.
(90 tasks in total). MQE achieves by far the best performance these challenging tasks.
in the
Figure 3: Comparisons of MQE against prior methods on OGBench.
5 EXPERIMENTS
Our goal of experiments is to understand the benefits MQE brings when it comes to enabling a policy generalize compositionally (execute multiple tasks seen in training separately together) and in terms of horizon (generalize over a longer task when similar but shorter task was seen in training set). To that end, we pose the following questions:
1. How much does MQE improve the horizon generalization abilities of agents? What qualitative improvements does MQE bring in terms compositional generalization?
3. What are the design decisions to ensure the success of MQE?
Our experiments use challenging, Long-horizon problems in offline RL bench- marks well real-world settings. We use OGBench (Park et al., 2025a) for experiments simulated benchmarks and the BridgeData setup (Walke et al., 2024) for our real-world evaluation.
Experiment setup
5.1 SIMULATED EVALUATION ON OFFLINE GOAL-REACTIN TASKS
We evaluate MQE in both locomotion and manipulation in OGBench (Park et al., 2025a). For locomotion tasks, in addition to the three standard sized mazes, also designed a "colossal" -sized
7

---

Published as a conference paper at ICLR 2026
maze. This maze is 50% larger than that of the "giant" -sized mazes currently available on OGBench, and it requires as many as 4000 steps for an agent to traverse through the entire maze (see Fig. 2). We employ 13 state-based environments and 5 pixel-based environments (each pixel-based environment takes in a 64 x 64 x 3 observation) with 5 tasks each, bringing a total of 95 tasks to evaluate in our OGBench setup.
We compare against the following baselines: GCIQL (Kostrikov et al., 2022), CRL (Eysenbach et al., 2022), QRL (Wang et al., 2023), HIQL (Park et al., 2024b), nSAC+BC (Park al., 2025b; Haarnoja et al., 2018), CMD (Myers et al., 2024), and TMD (Myers et al., 2025c), These methods use either only TD learning (GCIQL, QRL, nSAC+BC), MC value estimation via contrastive learning (CRL, CMD, TMD), use horizon reduction techniques (nSAC+BC with value horizon, HIQL with policy horizon) or use quasimetric architecture for distance learning (QRL, CMD, TMD). We detail how these methods are implemented in Section D. By comparing against these methods, we can gain a better outlook on what advantage MQE has over other works that learn distances only locally, globally, or in a hierarchical manner.
Table 4 and Fig. show the performance of MQE across state- and pixel-based environments on OGBench. In general, MQE exhibits considerably better capabilities of extracting goal-reaching poli- cies, and in some instances (such as humanoidmaze--iee aniinnnnniittii exhibits a 10x improvement over the previous best methods, including HIQL and B-CA which does explicit policy horizon reduction. The only exception to this is vision-based manipulation, where HIQL performed better. We also compared the performance of MQE against TMD, CRL, and HIQL antmaze of various sizes, with the training set fixed to only 4 meters long via the usage of stitch datasets. We show that as the evaluation horizon becomes longer, MQE still exhibits strong horizon generalization, and it is the only method that still exhibits nonzero success rates in colossal mazes.
5.2 REAL-WORLD EVALUATION OF MQE
While OGBench environments focus on learn- ing long-horizon tasks using mixed quality data in a single environment, we can use real-world BridgeData tasks to evaluate a more sophis- ticated kind of compositionality. BridgeData tasks consist of individual object manipulation primitives (e.g. picking up banana and placing plate), and our evaluation tasks are signif- icantly longer (up to 4x), requiring the composi- tion of multiple tasks in the dataset (c.g. placing four different objects on the plate) (Walke et al., 2024). The dataset does not contain any exam- ple trajectories that compose multiple tasks in this way. Accomplishing this sort of temporal composition is an important goal in offline RL, because it allows "stitching" long behaviors out of shorter chunks.
Open drawer and place
Quadruple Pick and Place
deservado
goal state
goal state
Figure 4: We evaluate MQE multi-stage manip- ulation tasks on BridgeData. Below are examples of the starting observations and goal images.
We designed the tasks on BridgeData to test whether policies can compose multiple tasks at once without external guidance. Instead of instructing the policy to complete single pick and place (abbreviated as PnP), we evaluate policy's performance with PnP of up to 4 objects in sequence. To our knowledge, such a task in BridgelData has never been completed without the use of hierarchical policies or high-level planners. We also evaluate the policy on tasks requiring dependencies (i.e. the second task can only succeed when the first task succeeds), with the policy being tasked with opening drawer and then placing the item within the drawer all conditioned by one image of an opened drawer with an item inside. Only one previous work (Myers et al., 2025b) has shown consistent success when using an end-to-end policy, noting the challenges involved with this type of policy. Examples of the tasks are shown in Fig.
8

---

Published as a conference paper at ICLR 2026
Table 1: Ablation results
(a) Lr Ablation
(b) o." Sampling Ablation
Configuration
Success Rate (%)
Configuration
Success Rate (%)
With LT
26.5(#1.3)
k' Eq. (8)
26.5(#1.3)
Without Lı
7.9(#0.7)
k Geom X)
22. (2.1(=11) (#1.1)
Using expectile 0.7
11.3(=11)
W Unif[1, K]
18.9(#0.9)
Using expectile 0.9
8.8(+0.77)
W ~ Unif[1,50) 50]
17.8(#13)
= 50
1.7(#0.5)
=
0(+0.00
We use 6DoF, 5Hz, WidowX250 manipulator for our robot learning tasks and we train and deploy policy x(a s. g) conditioned on observations and goal images. We compare against the following methods: GCBC (Ding et al., 2020), GCIQL, and TRA (Myers et al., 2025b). We compare MQE against GCIQL, a commonly used offline RL method, and we compare MQE against both GCBC (Ding et al., 2020) and TRA (Myers et al., 2025b). TRA is designed for following both goal images and language instructions, but we only use goal images as the modality to test. We provide more details on policy training in Section E.
As in prior work that focused on long-horizon manipulation tasks (Black et al., 2024; Shi et al., 2025), we use task progress to measure the effectiveness of these policies due to the long-horizon nature of these tasks. We detail more on the experimental setup and how we assign progress in Section E.I.
Compositional Scaling in Bridge
Task Progress (%)
80
60
40
20
Fig. reports the overall task progress on single-stage tasks and 4 tasks that require composition- ality, and Table 6 reports the bi- nary success rate of each task. We provide further analysis of policy rollout in Section E. Here, ob- serve that while MQE helps with single-stage tasks (single PnP. open drawer) against GCBC, both TRA and GCIQL can still perform com- petitively. However, as the number of tasks needed to be performed in sequence increases, we see that MQE is able to retain a relatively high task progress, while both GCIQL and TRA's performance regressed.
0
1
2
3
# of Consecutive Tasks
MQE (Ours)
GCBC
TRA
GCIQL
Figure 5: Task progress on BridgeData tasks based on how many consecutive tasks the agent required to perform, plotted with both the mean and the standard error bars.
Taking a look at the two most difficult tasks, we have quadruple PnP and drawer open and place. These tasks are the most challenging since quadruple PnP required the agent to reason consecutive primitives together, and drawer open and place required the agent to complete the first task (open the drawer) before completing the second task (putting the mushroom in the drawer). Among all methods that we have tested, only MQE and TRA displayed positive success rate, as demonstrated in Table 6. We provide more details on policy rollouts in Section E.
5.3 ABLATION STUDIES
In this section, explore the design choices involved for MQE. To that end, investigate the heuristics needed for MQE. We use the humanoidmaze--iinn daaaoondddnmeigaattttttttooo environment and dataset, and explore the following design questions:

---

Published as a conference paper at ICLR 2026
How do different distributions for sampling the waypoint si" affect MQE's success?
the objective of action invariance I necessary in MQE?
How do À and p, the two hyperparameters that affect the geometric sampling of sy", change MQE's performance?
Should we use action invariance for value learning? Table la shows that when using the same set of hyperparameters, imposing action invariance as an explicit term helps to learn much better critic, compared to only using LT.- Additionally, loss, described in (Kostrikov et al., 2022). While expectile loss performed well as a value learning objective, it performed considerably worse than action invariance, which validates the theoretical results shown in Section C.
also implemented value learning via expectile
How should we sample waypoints? We first eval- uate the best distribution to sample future waypoints. Ta- ble 1b shows that when using geometric distribution, achieved much better performance than using a uniform distribution using fixed interval. We believe that since the goals are sampled via geometric distribution, match- ing the waypoint with another geometric distribution helps the network to generalize better between the two separate observations.
0.99
14.9
13.2
19.1
8.1
0.95
21.4
26.5
23.3
7.4
Distuni Waypoints
0.9
7.2
17.8
12.8
2.3
0.5
0.0
0.0
0.0
0.8
We then investigate how to combine the hyperparameters and p for best performance. Fig. provides an illustration of success rate over pairs of (p, A). The figure suggests that both hyperparameters need to be relatively high in value. This indicates that MQE needs: (1) waypoint far enough for the value to rapidly propagate and (2) being respected. We also see that if we increase the value of the sampling coefficient p. MQE cannot learn good policy for goal-reaching. This shows that T is nor sufficient to the agent did not learn good distance representation.
0.5
0.2
0.1
0.0
More Multistep Enforcement
Figure 6: Sucess rate of MQE on humanoidma iimanoidmmaz..om _aiitttnnnntttee aaan__sitcch stitch using 0.01, averaged over seeds.
high enough p to ensure that local consistencies
learn good distance for policy learning,
6 CONCLUSION
We introduced Multistep Quasimetric Estimation (MQE), novel method that combines the benefits of fast value propagation via multistep backup and the global constraint of quasimetric distances. MQE is able to solve challenging and long-horizon tasks in simulated benchmarks and on a real-world robotic manipulator.
Limitations and Future Work. While MQE achieves strong performance, we sample way- points based heuristics. This could incur more computation costs when finding the optimal way of sampling the waypoint for environments that are outside of our evaluation range. Future work can in- vestigate the theoretical connection between sampling waypoints and successor distances, investigate the effect of such policy learning on different policy classes such as aation-chunking policies, and apply the same method across methods beyond offline RL in scenarios such affline-to-online RL. or online RI.
Acdnowledgements We would like to thank Seohong Park, Qiyang Li, Michael Psenka, and Kwanyoung Park for helpful discussions. This material is based upon work supported by the National Science Foundation under Award No. 2441665 and ONR N00014-25-14 1-2000. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF and ONR.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Kristopher De Asis, J. Fernando Hernandez-Garcia G. Zacharias Holland, and Richard S. Sutton.
Multi-step Reinforcement Learning: Unifying Algorithm. arXiv:1703.0132) 2018.
Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient Online Reinforcement Learning With Offline Data. International Conference on Machine Learning. 2023.
Arindam Banerjee, Srujana Merugu, Inderjit Dhillon, and Joydeep Ghosh. Clustering With Bregman
Divergences. SIAM International Conference on Data Mining. pp. 234-245, 2004.
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, et al. To. A Aision-aaggaage-Acie Flow Model for General Robot Control.
arXiv:2410.2411 2024.
Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning Successor States and Goal- Doall-eeenddet Values: A Mathematical Viewpoint. 2XXx:2101.007220 2021.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? arXiv:2206.01079) 2023.
Stefan Cobzaş. Quasi-metric and Quasi-uniform Spaces, pp. 1-97. Springer Basel, Basel, 2013. Peter Dayan. Improving Generalisation for Temporal Difference Learning: The Successor Represen- tation. Neural Computation, (4):613-624, 1993.
Yiming Ding, Carlos Florensa, Mariano Phielipp, and Pieter Abbeel. Goal-conditioned Imitation
Learning. Set.:19900058338 2020.
Benjamin Eysenbach, Vivek Myers, Ruslan Salakhutdinov, and Sergey Levine. Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference. Neural Information Processing Systems, 2024.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-Learning: Learning to Achieve Goals via Recursive Classification. International Conference on Learning Representations, 2021.
Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive Learning as Coaa-Condiiioned Reinforcement Learning. Neural Information Processing Systems, volume 35, pp. 35603-35620, 2022.
Scott Fujimoto and Shixiang Shane Gu. A Minimalist Approach to Offline Reinforcement Learning.
Neural Information Processing Systems, 34:20132-20145 2021.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme Q-Learning: MaxEnt RL
without Entropy. arXiv:2301 002328,,.. 2023.
Raj Ghugare, Matthieu Geist, Glen Berseth, and Benjamin Eysenbach. Closing the Gap Between TD Learning and Supervised Learning a Generalisation Point of View, International Conference on Learning Representatiinns, 2024.
Tuomas Haamojja Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Max- imum Entropy Deep Reinforcement Learning With a Stochastic Actor. International Conference on Machine Learning. 2018.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning With Implicit Q-Learning, International Conference on Learning Representations, 2022.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. Neural Information Processing Systems, volume 33, pp. 1179-1191, 2020.
Qiyang Li, Zhiyuan Zhou, and Sergey Levine. Reinforcement Learning With Action Chunking. hrXiv:2507.07969 2025.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.
arXiv:1509.02971. 2019.
Bo Liu, Yihao Feng, Qiang Liu, and Peter Stone. Metric Residual Networks for Sample Efficient Goal-Cond Reinforcement Learning. arXiv:2208.08 133., 2023.
Jianlan Luo, Zheyuan Hu, Charles Xu. You Liang Tan, Jacob Berg, Archit Sharma, Stefan Schaal,
Chelsea Finn, Abhishek Gupta, and Sergey Levine SERL: A Software Suite for Sample-ffficies Robotic Reinforcement Learning. International Conference on Robotics and Automation, 2025.

---

Published as a conference paper at ICLR 2026
Max Sobol Mark, Tian Gao, Georgia Gabriela Sampaio, Mohan Kumar Srirama, Archit Sharma, Chelsea Finn, and Aviral Kumar. Policy Agnostic RL: Offline RL and Online RL Fine- Tuning of Any Class and Backbone. arXiv:2412.06685, 2024.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari With Deep Reinforcement Learning. 2013.
Learning arXiv:1312.5002
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G. Bellemare. Safe and Efficient
Off-Policy Reinforcement Learning. arXiv:1606.02647 2016.
Vivek Myers, Catherine Ji, and Benjamin Eysenbach. Horizon Generalization in Reinforcement Learning. International Conference on Learning Representations, 2025a.
Vivek Myers, Bill Chunyuan Zheng, Anca Dragan, Kuan Fang, and Sergey Levine. Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following. arXiv:2502.05454, 2025b.
Vivek Myers, Bill Chunyuan Zheng, Benjamin Eysenbach, and Sergey Levine. Offline Goal Con- ditioned Reinforcement Learning With Temporal Distance Representations. Neural Information Processing Systems, 2025c.
Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision- Making. International Conference
Machine Learning, 2024.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-Efficient Hierarchical Reinforce-
ment Learning. arXiv: 00118000089990 .8299. 2018.
Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering Your Generalists:
Improving Robotic Foundation Models via Value Guidance. arXiv:2410. 2025.
Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, et al. Open X-Embodiment: Robotic Learning Datasets and RT-X Models. International Conference on Robotics and Automation, 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking
Offline Conditond RI. International Conference on Learning Representations, 2025a.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is Value Learning Really the Main Bottleneck in Offline RI. AXXi:22406009299 2024a.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon Reduction Makes RL Scalable. arXiv:2506.0411 04168, 2025b.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. HIQL: Offline Goal- Conditioned RL With Latent States as Actions. Neural Information Processing Systems, 2024b.
Scohong Park, Tobias Kreiman, and Sergey Levine. Foundation Policies With Hilbert Representations. arXiv:2402.15567, 2024c.
Ahmad Parsian and SNUA Kirmani. Estimation under LINEX loss function. Handbook of applied econometrics and statistical inference, pp. 75-98. CRC Press, 2002.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FILM: Visual Reasoning With a General Conditioning Layer. AAAI Conference on Artificial Intelligence, 2017. Silviu Pitis, Harris Chan, Kiarash Jamali, and Jimmy Ba. An Inductive Bias for Distances: Neural Nets That Respect the Triangle Inequality. International Conference on Learning Representations, 2020.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- nigh-Dimensiooal Continuous Control Using Generalized Advantage Estimation. International Conference on Learning Representations, 2016.
Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control.
MtXiv:2505.22642 2025.
Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, et al. Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-LLnguagg-Action Models. arXiv:2502. 19417, 2025.
Laura Smith, Yunhao Cao, and Sergey Levine. Grow Your Limits: Continuous Improvement With Real-World RI. for Robotic Locomotion. arXiv:2310. 17634,,,.,,....11111111111111 2023.
12

---

Published as a conference paper at ICLR 2026
R. Sutton. Learning to predict by the method of temporal differences. Machine Learning, 3:3-9 9-, 1988.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Dong Tian, Ge Li, Hongyi Zhou, Onur Celik, and Gerhard Neumann. Chunking the Critie: A Transformer-Based Soft Actor-Critic With N-Step Returns. arXiv:2503.03660 2025.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning With Contrastive Predictive Coding. arXivv 00774, 2019.
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering Your Diffusion Policy With Latent Space Reinforcement Learning. arXiv:2506. 15799, 2025.
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Maansnn-Estruch et al. BridgeData V2: A Dataset for Robot Learning at Scale.
MrXiv:2308.12955 2024.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal Goal-Reaching Rein- forcement Learning via Quasimetric Learning. International Conference on Machine Learning, pp. 36411-36430, 2023.
Christopher Watkins. Learning From Delayed Rewards, phdthesis, 1989.
Charles Xu, Qiyang Jianlan Luo, and Sergey Levine. RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning. arXiv:2412.0988 2.99888, 2024.
Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine. Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching From Offline Data. International Conference on Learning Representations, 2024.
A WEBSITE AND CODE
We provide the website, alongside the full implementation of MQE, TMD, CMD, and the antmaze-colossal mazes on OGBench at the following URL: https: http:: paper github.io.
B LLM USAGE STATEMENT
We used large language models for proofreading and correcting grammatical errors, well as providing suggestions for formatting visualizations. Any ideas generated in the paper are all entirely created by the authors and LLMs are involved in that aspect.
C PROOF OF POLICY IMPROVEMENT
Here, we provide the full proof of Theorem 1.
Theorem 1. Suppose behavior policy ny induces full support over state action pairs (s,a) a) ~ d in a tabular setting. We distance by minimizing Eq. (16) and extract a policy by minimizing Eq. (18).
Then the extracted policy satisfies VT 2 V (s) for all s,g E S.
Proof of Theorem I.
We can express the algorithm described in Section 4.5 as the following two steps (combining the quasimetric projection and Eq. 18):
min "" (r((((s)a),))) B, (e eed(s"")))) + (e ((((s((sa)) 1)?]
(2) min "o (((((s,((s.))).0)
Define d to the modified successor distance induced by the behavior policy kg. as defined by Myers al. (2025c, Eq. (7)).
13

---

Published as a conference paper at ICLR 2026
The key is that the two terms in step (1) act different parts of d's domain. The LHS term acts S x AUS, while the RHS acts S x SUA U.A. The LHS regresses d towards d5p on 8 x AUS, while the RHS enforces action-invariance on S 8 A. The result that d < dgp while corresponding to the temporal distances of some family of goaleeeraetterized policies. It follows that since P is monotone decreasing, we have iPd < dso and this still corresponds to the temporal distances of valid goal-reaching policies as a consequence of the existence of a stationary optimal goal-reaching policy in MDPs/CMPs.
Step (2) therefore necessarily recovers a policy π such that d5p S dgp In other words, the extracted policy и satisfies V (s) Vggh (s) for all 9, g. S.
D OGBENCH EXPERIMENT DETAILS
This section provides additional experiment details for the OGBench experiments in Section 5.1.
D.1 BASELINE DETAILS
Here, we briefly describe the inner workings of each baseline OGBench.
Goal-conditioned Implicit Q-learning (GCIQL) uses expectile regression to learn a value function. Quasimetric RL (QRL) learns quasimetric distance using bootstrapping under quasimetric architecture, and constrains the distance with one-step cost in deterministic settings. Contrastive RL (CRL) uses binary cross entropy to regress the critic (defined dot product) towards goals that future states and repel those that are not. Contrastive Metric Distillation (CMD) uses InfoNCE loss (van den Oord al., 2019) recover Q3. Temporal Metric Distillation (TMD) (Myers al., 2025c) use contrastive learning to learn the behavior Q-function, and then tightens the bound to optimize towards Q". CMD and TMD enforce the quasimetric property of successor distance architecturally.
Additionally, we compare MQE against two horizon reduction methods, n-step Goal-Conditioned Soft Actor- Critic with Behavior Cloning (n-SAC+BC) and Hierarchical Implicit learning (HIQL), which explicitly reduce the policy and value horizon separately. n-SAC+BC is similar to SAC+BC, with stetep updates as described in Eq. (19) for a sampled batch B.
Cq E(s ma) (Bc))s/LCCEQQUUU ab. g) Σ 9) ((S0)ssstttSSSttttt ((sh+n:g).g))
(19)
Where Lacr(e, 31 A -y -ylog= x (1 y) log(1 x),, as it demonstrated superior performance in (Park et al., 2025b). HIQL trains the same value function GCIQL, but the agent extracts hierarchical policy using AWR-like objectives. All policies trained on OGBench are designed to return multimodal Gaussian distribution N ((;; 2), and during inference time, the neural network produces the distribution, and the policy samples from that as action.
D.2 TASK VISUALIZZTIOON
We provide the tasks used for antmaze-colossad environment on Fig. 7. The maze itself is 24 blocks in height and 18 blocks in width, which is 50% larger than the giant mazes in each dimension.
D.3 IMPLEMENTATION DETAILS & HYPERPARAMETERS
Table 2 details the common hyperparameters for all methods OGBench. Table shows the regularization hyperparameter that was found to be the best for performance.
To regulate Lz, impose hyperparameter act multiplier. This ensures that across both visual and state- state-based environment, Ln does not immediately overpower LT.- The final critic loss becomes:
14

---

Published as a conference paper at ICLR 2026
Task
Task 2
Task
Task 4
Task
Figure 7: Task visualizations from matmaze-colosee environment. The ant occupies the starting position, and it must reach the red dot to complete the task.
Table 2: Network configuration for MQE on OGBench.
Configuration
Value
batch size
256
latent dimension size
512
encoder MLP dimensions
(512, 512, 512) (512, 512, 512)
policy MLP dimensions layer norm in encoder MLPs visual encoder (visual- envs) MRN components Discount (7) Waypoint discount (X) Probability p for sampling next state of for waypoint
True
impala-small
8
0.995
0.95
0.2 (state-based), 0.1 (pixcl-based)
Table 3: BC coefficient for each environment
Environment
antmaze-navigate
0.1
antmaze-stitch
0.03
antmaze-explore
0.003
humanoidmaze
0.01
pointmaze
0.03
(cube, puzzle scene-play]
1.0
[visual-(cube,
Luusull-(uubeppuzzze puzzle, scene-play]
3.0
*-noisy
0.1
visual-antmaze
0.3
15

---

Published as a conference paper at ICLR 2026
Table 4: OGieenc Evaluation
Methods
Dataset
MQE
TMD
CMI
CRI.
QRI.
GCIQL
HIQI.
D-SAC+BC
pointaaze _gaasttgintteeeeeeeeeeeeeeeeeeee navigate
728(4244)
38.9
45,3(250)
27,4
68.8(224)
45,9
pointnaza, glanttiatttt atitch
59,6
& (
0.0
49.7(
0.0
antnaze, lat.ii___arr_arrp..oo
67,7
ay t92
ag(to
0.66000)
0.4
3,9
(12(+11))
antnaze, cataaaggnttttttttttae
2705
20
0.4
18 300
9.2(
22,3
(255
14.6
0.0
0.01
U.3 (
aninaze caaaaee_cooosessstiiis
27.6 (7.6
0,2(2
0,0
0.0(2200
0
huranaidnaze, saattaggatttttt_teeeea navigaae
(6.5(125))
9.
12,51
huzanoidmaze_pP gittmmmtsc_tatttttttte
26.5(113)
6,3
og(ten
(
1.5
3,3 7,3
1.7
cube_double_play
(08(211
13.
Latann)
0,4
40.2
(8.((122)
3,4t30g
puzzle_Aré_plas
18,(220
10.0/214
0,20mmg
0,4(2
25,70310
11,4
acane_play
768(240
21.9
38.2(200)
scene_noicy
(22))0)
25.2(*13)
19.35+22)
visual_acon Baauull_cee___a_________
saann_play
38 (
20,((225
49.91354)
visual_cube, ctt.il___t_tttrllleeeee
cect_ttrrpl___________ play
19.()90)
17.91133)
16,((11
16,3(-03
15.2(100)
21.0 (13)
21.2
142(211))
0,363
6.0(214)
(213)
21.6
(9.4
22.7
visual_cubee cube, triplrrrppp___ee noiay
17,7aa90
(697)
156teno
86182n
12,5
21.0(267)
vieuxl_puzzle, wwwple__x___pppppppee 4xd_play
17,9(219)
98(499
7,2
0,0(2100
16.2
60.1 (4-34)
10.34229)
visual www.antt_nnaaaaee atitch
26,9 (64(
14.5(329
22.3
0.0
0,2(201)
Overall
36.3
13,0(209
8,7
6,2(2)))
(.6()
11.
18,7
8.5(2
bold best performance. Success rate (%) is presented with standard error across eight seeds for state-based environments and four seeds for pixel-based environments. All datasets contain 5 separate tasks each. We record the aggregate across all 5 tasks.
1 (= = LTg (p,, sh {st, a, s ()E) + Cz(p, {s,, ( Co(p,10 (86, a, ot
(20)
We use LINEX losses (Garg et al., 2023; Parsian and Kirmani, 2002) to parameterize the Bregman divergence in TTy, and to prevent exploding gradients, set the
We use batch size of 256 for MQE, and we use NVIDIA A6000 GPUs for our experiments. For state-based environments, takes around 2.5 hours finish both training and evaluation. For pixel- based environments, it takes 6 hours to complete both training and evaluation. During policy training, we found out that instead of using larger batch size and randomly shuffle future states, as done by previous implementations (Park et al., 2025a), we achieved better performance by calculating the pairwise distance between each state and future state, at the cost of lower batch size but more comparisons between states.
D.4 POLICY EVALUATION
We maintain evaluation procedure similar to that of OGBench's. ss We evaluated each policy 50 times for the last training epochs (800k, 900k, 1M step for state- environments, 300k, 400k, 500k in pixel-based environments). For each of these evaluation epochs, use 8 seeds for every state-based environment and 4 seeds for every pixel-based environment.
D.5 FULL RESULT TABLE
We record the full success rate of all tasks in OGBench in Table 4.
D.6 VISUALIZATIONS
We provide visualizations on antmaze-large--xpll each s in the environment with Fig. 8. We calculate the distance in IQL using the learned value function (s) directly. For CRL, we use the following equation to calculate the distance to match the BCE formulation being used in it: d(s.g) = log a (yp(s, (b)(())), where 0 corresponds to zero vector with the same dimension as the action. Given that CMD only learns one representation sp(s, a), we parameterize the distance as dMRN(p((s, 0), (gg 0)). In TMD, we use d(s,, g) = d(b(ss) (()).
to show the learned distances d(s, g) for
16

---

Published as a conference paper at ICLR 2026
Out of all the heatmaps, distance, while CRL and QRL contain many visual artifacts. GCIQL also has the problem of value propagation, where only small region around the goal actually has a low distance measure. TMD has the best visualization out of all baselines, but still suffer from minor artifacts. In contrast, MQE recovers distance that has the desired structure, which makes it desirable for learning a good goal-reaching policy.
that both GCIQL and CMD carry features that similar L2
2H
2
TH
(a) GCIQL
(b) QRI.
(c) CRI.
2H5
2H.
(d) CMD
(c) TMD
(f) MQE (Ours)
d(o, *)
MAX
Figure 8: Comparisons of different learned distances. Brighter colors correspond to lower distance measure. The agent starts at the position the white star and the goal is set at the gold star. We compare especially the structure of the distance, we normalize the ddpg loss in policy learning.
E BRIDGEDATA EXPERIMENT DETAILS
We evaluated each policy with total of 15 trials each for each task. For each of the pick-and-place tasks, we assign points for each successful pick and place (i.e. move a desired object to the desired location). Therefore, a policy objects.
maximum of points for each PnP task that manipulates i
For open drawer and place, assign one point if the policy opens the drawer to the extent where the mushroom can be placed, but does not pull the drawer off the base completely, and assign another point if the policy is able put the mushroom in the drawer.
E. TRAINING CONFIGURATIONS
We use pretrained ResNet34 as the backbone of the policy and encoders. We not share the actor and critic encoder for both GCIQL and MQE. We co-train the actor and the critic for total of 500,000 total steps with batch size of 128, which takes around 60 hours when the model trained on 4 v4-8 TPUs. For TRA, produce the embedding of two separate encoders Φ, vb, and align each other using symmetric InfoNCE loss. During inference, we use FiLM (Perez et al., 2017) to embed the learned goal representation into the actor, staying consistent with the implementation from (Myers et al., 2025b). In addition, we also describe the common hyperparameters used for BridgeData setup Table 5.
17

---

Published as a conference paper at ICLR 2026
Table 5: Network configuration for MQE on BridgeData.
Configuration
Value
latent dimension size
256
encoder MLP dimensions
(256, 256, 256)
policy MLP dimensions
(256, 256, 256)
layer norm in encoder MLPs
True
MRN components
8
Discount (7)
0.98
Waypoint discount (A)
0.95
p 0.2
F.2 TASK SUCCESSES
Table 6 records the success rate of each task across all six tasks that evaluate, and Fig. 9 records the task progress for each task. Table 6, we only measure whether each task has been completed to its fullest. While it does give out stronger signal on whether task displays nonzero success rate, does not provide as much information on how the task was progressing overall.
MQE (Ours)
GCIQL
GCBC
TRA
0.8
Overall Task Progress
0.6
0.4
0.2
Single PnP
Double PnP
Triple PnP
Quadruple PnP
Open Drawer
Open Drawer and Place
Figure 9: Task progress on BridgeData tasks; plotted with both the mean and the standard error bars.
Table Binary success counts for each task.
Task
MQE (Ours)
GCBC
GCIQL
TRA
Single PnP
12/15
5/15
10/15
9/15
Double PnP
10/15
1/15
4/15
6/15
Triple PnP
4/15
0/15
0/15
1/15
Quadruple PnP
2/15
0/15
0/15
0/15
Open Drawer
12/15
10/15
5/15
11/15
Open Drawer Place
5/15
0/15
0/15
1/15
F POLICY ROLLOUT IN BRIDGEDATA
We provide the rollout of triple PnP in Fig. 10. We especially consider TRA because it also exhibits compositional generalization, yet there is no explicit policy improvement as compared to offline RL. methods such as MQE.
18

---

Published as a conference paper at ICLR 2026
Goal observation
MQE (Ours)
TRA
Poor policy extraction!
x
Figure 10: Inference from triple pup task. We note that due to poor policy extraction and generalization, TRA not able to complete the task.
19


Reviews:
[{"review_id": "wliiuaN3FT", "reviewer": "Reviewer_sYkc", "summary": "This paper introduces Multistep Quasimetric Estimation (MQE), an offline GCRL method that integrates multistep returns within a quasimetric network architecture.\nThe paper aims to reconcile the tension between the theoretical optimality of local TD updates and the superior horizon scaling of global Monte-Carlo methods.\nMQE updates values by regressing distances using geometrically sampled \"waypoints\" and enforcing action invariance.\nMQE is evaluated on complex stitching and noisy tasks from OGBench where it outperforms common baselines, including often outperforming the hierarchical HIQL.\nMoreover, the paper presents impressive compositional generalization in real-world robotic manipulation.", "questions": "* Is the quasimetric architecture essential for the success of the multistep backup, or could this backup strategy also improve standard value-based methods (e.g., IQL) without the explicit distance structure?\n\nNote: The paper title of the PDF does not match the title on OpenReview.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "The paper's main strength is the integration of multistep backups into the quasimetric learning framework.\nThis enables superior horizon generalization compared to prior methods, which the paper demonstrates extensively on challenging environments from OGBench requiring extremely long-horizon planning.\nThe success in the real-world BridgeData experiments is compelling.\nTo my knowledge, complex multi-stage compositionality (e.g., Quadruple Pick and Place) using a flat (non-hierarchical) policy architecture represents significant progress in scalable GCRL.", "weaknesses": "* The multistep backup (Eq. 9) is inherently biased towards the behavior policy​ when the step k′>1. While the authors mitigate this with 1-step consistency weighting and action invariance, the justification for how this combination robustly overcomes the strong bias of the multistep return is primarily empirical.\n* In my view, the paper is missing a comparison to recent work [1] that addresses the identical challenge of achieving long-horizon GCRL performance with a flat policy.\n\nThe presentation has several issues:\n* The related work section has several missing or ill-formatted citations.\n* The paper has several incomplete sentences, e.g., line 91.\n* Page 5 includes a lot of extra spacing.\n* Overall, I find the methods section (Section 4) hard to follow. It would be beneficial to add structure to the section and provide additional guidance to the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ecPystv5F5", "reviewer": "Reviewer_pzjw", "summary": "This paper addresses the challenge of scaling RL to real-world, long-horizon robotics-tasks via offline GCRL. The main contribution of the paper is a novel algorithm, Multistep Quasimetric Estimation (MQE). The main technical innovation is how to learn multistep returns under quasimetric architectures while preserving the ability to learn the optimal value function. The authors show that MQE obtains state-of-the-art results in challenging long-horizon GCRL tasks in OGBench and a real-world robotics task.", "questions": "- The waypoints are randomly sampled according to the geometric distribution. Are there any theoretical benefits of geom distr. over other distributions, esp those considered in App. E?\n    \n- What is the benefit of multi-step return in this setting? From another angle, what is the problem with single step returns?\n    \n- Intuitively, why does Lp violate optimality while Lt doesn't? Can the authors provide an illustrative example or proof?\n    \n- Expts:\n    \n    - Why does MQE do particularly poorly on cube_triple_noisy?\n        \n    - Why does MQE underperform baselines on visual tasks, especially compared to HIQL?\n        \n    - Why not compare against HIRL in Bridge Data?\n        \n- Is this method applicable to online settings?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- The paper addresses an fundamental challenge in RL - namely, how can we scale to real-world tasks? Personally, I think the approach of trying to induce compositional learning via offline GCRL, succ, and goal-stitching is very promising. The topic will definitely be interesting to the ICLR community.\n    \n- The proposed approach is empirically highly effective compared to baselines in state-based datasets and the real-world robotics task. Overall, the evaluation is thorough. There is a large-scale evaluation of many methods and tasks.", "weaknesses": "- **Writing:** the writing is the biggest weakness of this paper. Unfortunately, the authors focus on explaining technical details of the method to an audience of experts in offline RL / GCRL, while failing to communicate high level, key messages to a general audience (see below). I’m concerned that this will limit its interest/impact to the general ICLR audience.\n    \n    - The motivation and the problem being solved is not very coherent at the beginning of the paper. I found myself wondering for a while whether the authors were addressing GCRL, or offline RL. It took until I almost finished skim-reading the paper to confirm that the authors address offline GCRL. Looking back at the paper, I think this occurred because the abstract only addresses GCRL rather than offline GCRL, and the term “Offline GCRL” only appears in conjunction in 2 places in the manuscript.\n        \n    - What is the key insight of the paper? Why do multistep returns matter so much torwards improving performance on the long-horizon offline tasks?\n        \n    - Some minor typos in the citations:\n        \n        - Please make sure to use citep where appropriate.\n            \n        - See Lines 91, 96, 122, 399\n            \n    - Lack of clarity:\n        \n        - What is task prowess metric in Fig 3 and how does it differ from success rate?\n            \n- **Soundness of Method**: The proposed method relies on trick of sampling next states w/ higher prob according to a Bernoulli distribution, thus indicating the auxiliary loss in 4.2 is not sufficient to fix bias issues. Can the authors comment on this point? What is the relative contribution of this trick to the final performance of MQE vs the auxiliary loss?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FIm28XTbpj", "reviewer": "Reviewer_4Ky7", "summary": "The paper introduces Multistep Quasimetric Estimation (MQE), a novel goal-conditioned reinforcement learning (GCRL) method that unifies temporal-difference (TD) and Monte Carlo (MC) learning through a quasimetric distance representation. MQE leverages multistep returns under a quasimetric architecture to propagate value information efficiently across long horizons while maintaining theoretical consistency with optimal value functions. It further enforces action invariance and one-step consistency constraints to stabilize learning. Empirically, MQE achieves state-of-the-art performance on long-horizon offline GCRL benchmarks (up to 4000 steps) and demonstrates strong compositional generalization in real-world robotic manipulation tasks. Its key contribution is showing that multistep temporal consistency and quasimetric structures can be combined to enable scalable, end-to-end goal-reaching from raw, unlabeled offline data.", "questions": "1.Could the authors clarify the conditions under which the proposed multistep quasimetric estimation guarantees convergence or consistency? Specifically, how sensitive are these guarantees to function approximation errors or off-policy data distributions?\n\n2.The method integrates multistep temporal consistency into the quasimetric framework, but it remains unclear how much of the empirical gain arises from longer-horizon updates versus the quasimetric structure itself. Could the authors provide quantitative or qualitative evidence disentangling these effects (e.g., via controlled ablations or visualizations)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper makes an original contribution by unifying multistep value learning and quasimetric representations into a single, scalable framework for goal-conditioned reinforcement learning. The idea of integrating multistep temporal consistency with quasimetric architectures is both novel and technically elegant, addressing long-standing limitations in horizon generalization and stability. The paper is of high quality, with strong theoretical grounding, clear algorithmic exposition, and comprehensive experiments across both simulated and real-world robotic domains. Its clarity allows readers to follow complex ideas with precision, and its significance lies in demonstrating a practical path toward scalable, compositional goal-reaching in offline RL—bridging a crucial gap between theory and real-world applicability.", "weaknesses": "While the paper is strong overall, several aspects could be improved to strengthen its impact. The theoretical analysis, while elegant, could benefit from clearer discussion of its assumptions and limitations—particularly regarding stability and convergence in high-dimensional or stochastic environments. Finally, the presentation could be improved by providing more intuition and visualization of the learned quasimetric distances to help readers better grasp the geometric and representational properties driving the observed performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YgFKeGxbkJ", "reviewer": "Reviewer_m7gs", "summary": "This paper proposes a new method for offline GCRL. The authors introduce Multistep Quasimetric Estimation (MQE), which applies a multi-step Monte-Carlo return to a quasimetric distance-based method. They evaluate on OGBench and a real robot, where MQE can outperform all baselines.", "questions": "1. What happens if the hyperparameters are not tuned, or only tuned with a small number of trials? How much does the performance drop in that case?\n\n2. Can the authors provide theoretical support for using a geometric distribution to select the waypoint? Why is this distribution a reasonable choice compared with other alternatives?\n\n3. In line 386, the paper states that the task \"has never been completed without the use of hierarchical policies or high-level planners\". Then why is there no comparison against hierarchical policies? Also, can MQE be integrated into a hierarchical framework? If so, why not include such a comparison to better understand its advantages and limitations?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The theoretical part is clear and easy to understand. The method is simple, but it still brings a notable improvement in performance. This makes the idea accessible for other researchers and shows that even a straightforward change can lead to meaningful progress.\n\n2. The experiments provide strong comparison results. The authors choose many baselines, including some recent works. This helps confirm that the gains are not due to weak baselines and shows that the method remains strong under fair comparisons.\n\n3. The paper includes real-world testing. The method achieves a higher success rate in practice, and it is able to combine multiple motion segments into complete trajectories. This supports the claim that the approach can work outside simulation.\n\n4. There is enough hyperparameter study. The authors include an analysis that explains how different settings affect performance. This makes the method easier to apply and helps readers understand which parts matter most.\n\n5. The code had been released, showing its reproducibility", "weaknesses": "1. While MQE can indeed improve performance, the choice of hyperparameters and the selection of the best waypoint introduce heavy computational cost. This limits the method’s practical use, especially when scaling to more complex tasks or real-time applications.\n\n2. There are many citation errors in the paper, for example, at line 96, 91, 399, and 431. In addition, there are indexing mistakes and incorrect claims. For instance, in Algorithm 1, the loss at line 5 should refer to Eq. 11. Another example is at line 435, where the text claims that TRA-g reaches a positive success rate in the quadruple PnP task, but the table shows 0/15. These issues show that the writing and validation of statements are not strict enough.\n\n3. The value of $\\alpha$ used during policy extraction is a hyperparameter that must be tuned separately for different environments. This shows that the method needs heavy hyperparameter tuning to reach the reported performance, which is not practical. I suspect that much of the performance gain shown in the paper comes from tuning these hyperparameters instead of the strength of the method itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o7yNEB8bPV", "reviewer": "Reviewer_jD1e", "summary": "This paper focuses on offline goal-reaching reinforcement learning, and pinpoints two families of methods: on-policy algorithms based on Monte-Carlo returns (and in several cases contrastive learning), and off-policy algorithms which enforce the known invariances of goal-conditioned value function through quasi-metric network architectures. This works extends TMD (Myers 2025), which previously united this two frameworks, by considering multi-step returns. Interestingly, this induces significant gains in performance across standard goal-conditioned tasks from the OGBench suite. The authors further perform an evaluation on the BRIDGE hardware setup, which confirms the strong performance of the algorithm. These results are accompanied by relevant ablations on the multi-step loss.", "questions": "### Minor issues and questions\n- Line 12: remove \"in\"\n- There are several broken or wrongly formatted references through the paper\n- Line 122: this sentence is hard to follow: \"X have demonstrated that Y instead of Z\"\n- Equation 7: the notation is unclear, as $s$ and $a$ appear under the expectation, as well as on the left side. formulating the expectation over $s'$ alone would be cleaner in my opinion.\n- Equation 10: this choice seems to follow from TMD, which should be referred to in this case\n- Figure 1: the caption refers to a 10x larger horizon, is this an overstatement? What is the length of optimal trajectories in giant and colossal mazes?\n- What is the reasoning behind the selection of environments in Table 1? e.g. why is antmaze-large-explore evaluated instead of antmaze-large-navigate? The current selection appears somewhat arbitrary.\n- Two of the ablation studies (matching the first two questions in 5.3) were relegated to the Appendix. This should be noted in the text.\n- Table 3: how is the regularization parameter $\\alpha$ tuned in each environment/algorithm combination?\n\n### Conclusion\nDespite the impressive empirical results, I currently lean towards rejection. To the best of my understanding, this method is an n-step extension of TMD, which is a minor contribution but is not problematic per se. My main concern is that this strong connection between the algorithms is not directly highlighted in the paper, which as a result seems to over claim its contribution. In am happy to further discuss whether my understanding is correct during the rebuttal phase. Moreover, unlike TMD, this works lacks an analysis of the objective and its solutions.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "- Empirical results are very convincing, both in breadth (involving a large number of OGBench tasks as well as hardware experiments) and in outcomes (displaying large improvements over strong baselines).", "weaknesses": "- The proposed algorithm is strongly aligned with TMD, but in my opinion does not fully acknowledge this. To the best of my knowledge, each component in Section 4 (except for (11), which is a direct multi-step extension of the TD loss (12)) was already introduced in TMD. If I understand correctly, the final algorithm is simply TMD with a multi-step loss. If this is the case, it should be acknowledged in full.\n- This works claims to combine off-policy and on-policy learning, but does not comment on whether the final value estimates are recovering the optimal value, or the value of the behavioral policy. Can the authors provide a formal discussion of what the proposed objectives recover, and why it is motivated?\n- Related works are overall short (e.g. there is a single reference in the paragraph on GCRL), poorly formatted (several undefined references) and poorly written (the structure of the sentences in the last two paragraph is incorrect).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)