Transitive RL: Value Learning via Divide and Conquer
Authors: Seohong Park
Aditya Oberai
Pranav Atreya
Sergey Levine
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
goal-conditioned rl
offline rl
Primary area: reinforcement learning

Abstract: In this work, we present Transitive Reinforcement Learning (TRL), a new value learning algorithm based on a divide-and-conquer paradigm. TRL is designed for offline goal-conditioned reinforcement learning (GCRL) problems, where the aim is to find a policy that can reach any state from any other state in the smallest number of steps. TRL converts a triangle inequality structure present in GCRL into a practical divide-and-conquer value update rule. This has several advantages compared to alternative value learning paradigms. Compared to temporal difference (TD) methods, TRL suffers less from bias accumulation, as in principle it only requires $O(\log T)$ recursions (as opposed to $O(T)$ in TD learning) to handle a length-T trajectory. Unlike Monte Carlo methods, TRL suffers less from high variance as it performs dynamic programming. Experimentally, we show that TRL achieves the best performance in highly challenging, long-horizon benchmark tasks compared to previous offline GCRL algorithms.

Published as a conference paper at ICLR 2026
TRANSITIVE RL: VALUE LEARNING VIA DIVIDE AND CONQUER
Seohong Park*1 Aditya Oberai*1
Pranav Atreya'
Sergey Levine
'University of California, Berkeley (sechong, aoberai @berreleyy edu
ABSTRACT
In this work, we present Transitive Reinforcement Learning (TRL.,, a new value learning algorithm based on divide-and-conquee paradigm. TRL is designed for offline goal-conditioned reinforcement learning (GCRL) problems, where the aim is to find a policy that can reach any state from any other state in the smallest number of steps. TRL converts triangle inequality structure present in GCRL into practical divide-and-conquer value update rule. This has several advantages compared to alternative value learning paradigms. Compared temporal differ- ence (TD) methods, TRL. suffers less from bias accumulation, as in principle it only requires (log T) recursions (as opposed O(T)) in TD learning) han- dle a length-''' trajectory. Unlike Monte Carlo methods, TRL suffers less from high variance as it performs dynamic programming. Experimentally, we show that TRL achieves the best performance in highly challenging, long-horizon bench- mark tasks compared to previous offline GCRL algorithms.
Blog post: https: //ttpa:/San htt////seooopgg me/ hheeblog/rlie wolog//lwwtthhuttee ta-tttt---llee td-learning
Method
Horizon Decomposition
Recursions
1
TD
0-00
O(T)
T n
TD-n
00
OTT/n)
Divide & Conquer
T/2
T/2
00
O T)
Figure Transitive RL. Transitive RL is based on the divide-and-comquer paradigm, which can in theory reduce the number of Bellman recursions to O(log T)) in the best case, unlike TD-based methods.
INTRODUCTION
fundamental challenge in off-policy reinforcement learning (RL) is the curse of horizon (Liu et al., 2018). In off-policy RL, typically train value function using (some variant of) temporal differ- ence (TD) learning with Bellman backups, (s,,, a) + r(s, a) + y max,' Q(s', a'), and train pol- icy maximize the learned value function. The problem that, each Bellman update involves gression toward biased target value, and these biases accumulate over the horizon through recur- sion. This bias accumulation is one of the main obstacles hindering the scaling of off off-policy RL to complex, long-horizon tasks (Park et al., 2025b).
standard technique to mitigate this bias accumulation issue in practice is to mix Monte Carlo (MC) returns, either for short-horizon trajectory chunks (e.g., n-step returns) or even for full trajectories (e.g., purely MC-based approaches). While these approaches do mitigate bias accumulation, they now suffer from higher variance as the Monte Carlo horizon grows. Sometimes, it is possible to strike a balance in the bias-variane tradeoff using n-step returns with an appropriately tuned
Equul contribution.

---

Published as a conference paper at ICLR 2026
(Sutton & Barto, 2005; Park et al., 2025b). However, this does not fundamentalyy solve the curse of horizon, as it reduces the horizon length only by a constant factor (n). We also need to carefully tune for each task, and it requires optimality in length-n trajectory chunks.
In this work, we investigate whether we can find a different mechanism to mitigate the curse of horizon in important subset of RL problems: offline goal-conditioned RL (GCRL) (Kaelbling, 1993; Park et al., 2025a). Goal-conditioned RL aims to find a policy that can reach any state from any other state in the smallest number of steps. This is typically done by learning the optimal distance function d' (s, g) (which corresponds to the value function), defined as the minimal number of steps to reach g from s (see Section for the formal definition).
By its -oortest-path-iin nature, the optimal distance function d' satisfies the following triangle inequality in deterministic environments:
d' (s.9) S d' w) + d" (w.g))
(1)
for all states 8, w, g. This provides an additional divide-and-conquee structure that we can exploit. That is, we can compute the distance d' (s.g) by combining information about two "smaller" dis- tances, (s, w) and d² (w, g).
Our central hypothesis in this work is that, gate the curse of horizon more effectively than standard TD- or MC-based methods in offline goal- conditioned RL. Unlike TD-based methods, divide and conquer has the potential logarithmically reduce the number of biased recursions in principle, thanks to its recursive nature (Figure 1). Unlike MC-based methods, divide and conquer suffers less from the high variance of long-horizon MC turns, as it performs dynamic programming.
and-conquer value learning algorithm can miti-
Based on this idea, we develop new divide-and-conquer- rithm, Transitive RL (TRL), for long-horizon goal-reaching tasks. There are several scaling chal- lenges in implementing this idea in practice, such as value overestimation. We address these chal- lenges with newly introduced ideas, such as in-trajectory subgoals and distance re- re-weighting.
offline goal-conditioned RL algo-
Our main contributions are twofold. First, we propose TRL, a new scalable, practical value learning algorithm for offline goal-conditioned RL based on divide and conquer. Second, we empirically demonstrate TRL. often exhibits better performance than previous TD- and MC-based approaches on highly complex, long-horizon robotics tasks over 1000 environment steps. To the best of our knowledge, TRL is the first divide-and-conquer value learning method that scales to long-horizon robotic tasks beyond toy domains.
2 RELATED WORK
Offline RL and offline goal-conditioned RL. Much work has studied the offline reinforcement learning problem, which aims to find the best return-maiiniiin policy within the support of the offline data. Most works distinguish themselves from the way they mitigate value overestimation for out-of-distribution actions, usually either imposing conservatism constraints on the value func- tion (Kumar al., 2020), behavior regularizing the actor (Wu et al., 2019; Fujimoto & Gu, 2021; Tarasov et al., 2023; Gao et al., 2025; Park et al., 2025c), or implicitly performing Bellman updates without querying outtof-distribution actions (Kostrikov et al., 2022: Xu al., 2023; Garg et al., 2023). In offline goal-conditioned RL, whose aim is to train a multi-task policy that can reach any state, previous methods either extend offline RL algorithms to the goal-conditioned setting (Yang et al., 2022; Park et al., 2023) or leverage specific structures of the goal-conditioned RL problem with quasimetric learning (Wang et al., 2023), hierarchical policy extraction (Park al., 2023), or probabilistic interpretation (Eysenbach et al., 2021; 2022; Zheng et al., 2024). In this work, we pro- pose a new goal-conditioned RI. algorithm based on triangle inequality, as discussed in the next paragraph.
The triangle inequality in GCRL. Our main idea in this work is based on the triangle inequality in goal-conditioned RL (Equation (7)). The idea of using the triangle inequality dates back to the very first work in goal-conditioned RL by Kaelbling (1993). Since then, several prior works have leveraged this structure in three main ways. "Hard" approaches strietly impose the triangle inequal- ity via an architectural quasimetric constraint (Wang et al., 2023; Myers et al., 2024), "soft" ap- proaches use the triangle inequality as a value backup (Kaelbling, 1993; Dhiman et al., 2018 8; Jur-
2

---

Published as a conference paper at ICLR 2026
genson et al., 2020; Piekos al., 2023), and planning- approaches employ this inequality for planning (Eysenbach et al., 2019; Parascandolo et al., 2020; Jurgenson et al., 2020; Li et al., 2021). Among planning free ooolpy RL methods, while hard approaches have proven effective in prac- tice so far (e.g- QRI. (Wang et al., 2023)), soft approaches (to our knowledge) have not yet scaled beyond 2-D toy environments in the absence of additional planning (Kaelbling, 1993; Dhiman al., 2018; Jurgenson et al., 2020; Piekos et al., 2023).
Our method, TRL, is based on the soft triangle inequality; i.e. we use the triangle inequality for value backups. instead of hard quasimetric constraints. However, unlike previous soft approaches, we show that TRL scales to highly complex, long-horizan robotic tasks with 1B-sized datasets, even outperforming hard (i.e., quasimetric- approaches. This is made possible by our key techniques (in-sample maximization within in-sample subgoals; see Section 4.2), which prevent value overestimation when applying the triangle inequality in practice.
PRELIMINARIES
Problem setting. We consider controlled Markov process defined by a tuple M = AS),.
where S is the state space, A is the action space, and p(s' s, : S x A A(S) is the transition dynamics distribution. In the definition above, ((X) denotes the set of probability distributions on a space X' and placeholder variables are denoted in gray. We assume that we are given an unlaheled dataset D = {T")); ) consisting of length-T reward-free trajectories T = (so, 4o, $1...., ST) (in practice, different trajectories can have different lengths, but we assume they have the same length for the simplicity of discussion).
In this work, aim to develop better recipe for offline goal-conditioned RL. Offline GCRL aims to learn goal conditiood policy Ta s, s, S S S + A that maximizes the objective (T
Er (r (m.g) ΣT= 7'I(s; = 9)] for all s.g S purely from the dataset D (without additional environment interactions), where I(() denotes the 0-1 indicator function," 7 E (0, denotes the discount factor, and p² s. g) (ao 50. g)p($1 | 50, ao. ........ p(sT ST-1, aT-1) where so = Here, we note that states and goals live in the same space S, and the reward function is simply given as the 0-1 sparse reward function. We also consider variant of the above coal-conditioned RL objective defined by hitting times, where the agent enters an absorbing state upon reaching the goal (i.e., gets reward of most once) (Wang et al., 2023). We denote the corresponding hitting- time-based value function as (*(ss,) g). In this work, we mainly consider this hitting-time variant, as it has a close connection to temporal distances, as discussed in the next paragraph.
As commonly done by many previous works in goal-conditioned RL (Eysenbach et al., 2019; Jan- ner al., 2022: Ghosh et al., 2023; Park et al., 2023; Wang al., 2023; Park al., 2024b), we as- sume that the environment dynamics are deterministic, unless mentioned otherwise. In deterministic environments, we can define the notion of temporal distances. The temporal distance d' (s. g) from E S to E S is defined as the minimum number of steps to reach y from 8 (oo if it is not reach- able). From the definition, it is straightforward to see that V* V*(s, g) == max, V* g) = 7d (x.g) in the deterministic case.
3.1 PREVIOUS APPROACHES TO OFFLINE GCRL
An offline RL algorithm typically consists of two components: value estimation and policy extrac- tion. In this section, we review two representative paradigms for goal-conditioned value learning (MC and TD), and two standard techniques for policy extraction.
Monte Carlo (MC) value estimation. Monte Carlo value estimation is one of the simplest tech- niques for goal-conditioned value learning. It trains behavioral value function based on the dis- tances achieved in dataset trajectories. Among MC-based approaches, distance regression meth- ods (Tian et al., 2021; Shah et al., 2021) fit Q function Q(,,, S + E with the fol-
We assume that the state space is discrete for notational simplicity. This is only to simplify mathematical notation, and most discussions in this paper can be directly extended to continuous state spaces as well with appropriate measure- theoorett formulations (Blier et al., 2021).
3

---

Published as a conference paper at ICLR 2026
lowing loss:
T.MO (Q) = E TND. [(((s;, 8z. - - 0SiSiKT
where 7 is sampled uniformly from the dataset, and and j are sampled uniformly from the set {0, "...,. - 1}. While this objective is biased when either the dynamics or the data-collecting policy is stochastic (Akella et al., 2023), it is simple, stable, and often performs surprisingly well in practice (Tian et al., 2021; Shah al., 2021; Hejna et al., 2023).
Temporal difference (TD) value learning. Another class of methods employs temporal difference learning learn a goal-conditioned value function. As an example, goal-conditioned implicit Q- learning (GCIQL) (Kostrikov et al., 2022; Park et al., 2023) fits value functions with the following losses:
LIQL-V(V) = B. (V. Q(s, ((,,,)))) LIQL-9(Q) ' goD (Q(s. a. l1(s g) VV (s'.g))))
(3)
(4)
where goals are typically sampled with hindsight relabeling (Andrychowicz et al., 2017; Park et al., 2025a), e denotes the expectile loss with parameter к [0.5, 1), (x) |K I(x < 0)]2²
and Q denotes the target Q function (Mnih et al., 2013). The asymmetric expectile regression in Equation (3) approximates the max operator in the Bellman update (V (x,g) max.e. Q(+, a, g)) only with in-sample actions, avoiding querying the Q function with out-o--distribution actions. We note that when & 0.5, the above objective yields a behavioral (SARSA) goal-conditioned value function (Brandfonbrenee et al., 2021; Kostrikov et al., 2022; Park et al., 2025b).
Policy extraction. After learning a goal-conditioned value function, the next step train a policy to maximize the learned values. This procedure is called policy extraction, and we describe two standard techniques below.
Reparameterized gradients (Fujimoto & Gu, 2021) extract Gaussian policy by maximizing the following "DDPG+BC"" objective (Park et al., 2024a):
JDDPG4BO (T) =E a. [Q(s, a", .9) + a log s,g),, (=<=s()))
(5)
where a is reparameterized sample from the policy and the strength of the BC constraint. Intuitively, this maximizes the learned value function while not deviating too much from the dataset distribution to prevent out-of-distribution exploitation.
Rejection sampling (Ghasemipour et al., 2021; Chen et al., 2023; Hansen- Estruch et al., 2023) sim- ply defines a policy to be the arg max of behavioral action samples that maximize the value function:
arg max x (s, g) 01 aaa (als.g)
Q(s, at g).
where N denotes the number of samples, denotes equality in distribution, and T denotes goal- conditioned BC policy. Typically, the BC policy is modeled by an expressive generative model (e.g.. diffusion models (Sohl-Dickstein et al., 2015; 5; Ho et al., 2020) and flow matching (Lipman et al., 2024)) so that can effectively capture the potentially multi-modal distributions of the behavioral policy (Chen et al., 2023; Hansen-Estruch et al., 2023; Park et al., 2025b).
4 TRANSITIVE RL
As motivated in Section 1, our main aim is to develop practical divide-and-congee value learning algorithm for offline goal-conditioned RL. Offline goal-conditioned RL provides a natural shortest- path structure that is amenable to divide and conquer. Namely, in deterministic environments, the temporal distance function d' d' g) always satisfies the following triangle inequality (Kaelbling, 1993; Dhiman et al., 2018; Wang et al., 2023; Pickos al., 2023):
d' (s. < d" w) + d" (w.g)
While we mainly consider deterministic environments in this work, note that the GCRL, problem ex- hibits similar triangle-inquuality structure stochastic environments as well; Myers al. (2024).
4

---

Published as a conference paper at ICLR 2026
for all g E S. The equality holds when is on shortest path from g. Throughout this paper, we call w a subgoal.
Equivalently, from the relation (s, = (o.0), we can rewrite this in terms of the optimal value function (5,gg) as follows:
(*(*g)) V* (s,,
(8)
This motivates the following transitive Bellman update rule for a value function V(s,
if = 9. if (s. otherwise.
g( +
?'
(9)
max WES
(s, (w.,))
where & (the edge set) denotes the set of s') E S x S pairs such that $ s' and s' is reachable from by single action. If we initialize the value function with V(s.g) S for all 8, 9 E S, standard result in computer science (e.g., the Floyd- Warshall algorithm) implies that repeatedly applying this operator converges to the optimal value function (*, (Dhiman et al., 2018; Pickos et al., 2023).
We also have a variant the above update rule for an action-value function Q(+, a)g):
if 9.
((s,a.))
if g p(s, a) and + g. otherwise,
(10)
max wES
,
where we slightly abuse the notation denote p as the deterministic dynamics function.
Why do we want to use the triangle inequality? As described in Section 1, the main benefit of the triangle inequality is that we can perform divide and conquer instead of TD or MC value learning. Unlike TD learning, which requires O(T) Bellman recursions, divide and conquer can reduce the number of Bellman recursions to Olog T) in the best case (in theory, with optimally chosen sub- goals). Unlike MC learning, which suffers from high variance with long horizons, divide and con- quer performs dynamic programming (i.e., aggregates intermediate values) and thus mitigates the variance issue. Hence, we expect that the use of the triangle inequality potentially leads to scal- able value learning algorithm that scales better in long- horizoorizon tasks.
4.1 THE CHILLENNE
While the transitive Bellman update rule (Equation (9)) provides a natural way to perform divide and conquer for goal-conditioned value learning, directly implementing this idea in practice is not straightforward. The main problem is the maxwes operator in Equation (9). While we can simply iterate over the set of states to compute the maximum in the tabular case, doing so would lead to (po- tentially catastrophic) value overestimation under function approximation, because the max opera- over large number of subgoals can easily exploit any single subgoal's 's positively biased value estimation error. This issue is further exacerbated in the offline setting, where value overestimation poses a particularly severe challenge (Levine et al., 2020).
To address this challenge, previous works have mainly considered two practical solutions. Jurgenson al. (2020) use previous iteration of the value network (similar to target networks (Mnih al., 2013)) for the target value in Equation (9). Piekos et al. (2023) train separate generator network G(s, a, : S A + to approximate the maxwes operator, similar to DDPG (Lillicrap al., 2016). However,
will show in our experiments, these techniques are not sufficient to stabilize training, leading to zero success rates in most simulated robotic benchmark tasks. Indeed, most prior works that the transitive Bellman update without explicit planning have demonstrated their methods only on 2-D toy tasks, such as point mazes and discrete grid worlds (Kaelbling, 1993; Dhiman et al., 2018 Jurgenson al., 2020; Piekos et al., 2023).
5

---

Published as a conference paper at ICLR 2026
4.2 THE IDEA
Our key idea to handle the overestimation issue in the max operator is to perform in-sample maxi- mization using only in-trajectory subgoals. Specifically, we apply the following two modifications."
First, we replace the strict max operator in Equation (9) with soft expectile regression (Newey & Powell, 1987; Kostrikov et al., 2022). That is, we (conceptually) minimize the following loss:
E (V (ss, w)V (w. g)]},
(11)
where lg denotes the expectile loss described in Section 3, and V denotes the target network (Mnih et al., 2013). This expectile loss allows us to approximate the maximum in Equation (9) without having to iterate over the states explicitly.
Second, we only consider in-trajectory states as potential subgoals (we call them behavioral sub- goals). That is, for a dataset trajectory т (#o, ('2 ;1,...;877 8T), we update V(8, 83) only using V(s,,*))V V 83) with i S k S j, instead of considering arbitrary states as subgoals. While this decision might seem restrictive at first glance, we find that restricting to behavioral subgoals is cru- cial to make the transitive Bellman update work in practice (Section 5.3). Otherwise, the probability of an arbitrary state being a "valid" subgoal is low, and thus we would need much more aggressive maximization (i.e., higher expectile in Equation (11)), which leads to unstable training. Moreover, we experimentally find that behavioral subgoals can still be highly effective even when the dataset consists of uniformly random atomic motions. This is analogous to how behavioral value functions (i.e., one-step RL) are often sufficient in practice (Brandfonbrener et al., 2021; Eysenbach et al., 2022: Park al., 2025b) even on suboptimal datasets.
4.3 PRACTICAL RECIPE
Based on the high-level idea in Section 4.2, we now describe the full recipe of our method, which we call Transitive RL (TRL). TRL trains a goal-conditioned action-value function Q a, g) S x A S - IR based on the Q-based transitive Bellman operator (Equation (10)), and then extracts a goal-conditioned policy π (a 8,g) S x A(A).
4.3.1 VALUE LEARNING
TRL's value learning is based the high-level idea described in Section 4.2. In practice, additionally employ the following technique to further improve performance and stability.
Distance-based re-weighting. In transitive Bellman updates, the accuracy of the target value for a longer trajectory chunk (s; to 8,) depends on those of two shorter trajectory chunks (s; to 88 and 8ą. to 8,). Hence, it is particularly important to keep the shorter trajectory chunks' values accurate. To this end, we weigh the loss for each sample ss), in the batch by s(s;,s;) == 1/(1 + log, Q(s,, a,, 8,)),, where À is a hyperparameter. This adjusts the weight for each chunk to be (roughly) inversely proportional to its estimated distance, thus focusing more on shorter trajectory chunks. This bears similarity to classical dynamic programming where smaller subproblems are solved before larger ones.
With this technique, the final value loss for TRL is defined as follows:
Dr. (Q(s,, where D denotes loss function (e.g., squared regression, binary cross- crsss-enropp. etc.), and D, de- notes its expectile variant, (x, y) Ix-I(x > D(z 3). In our experiments, we use the binary cross-entropy (BCE) loss, following prior work in goal-conditioned RI. (Kalashnikov et al., 2018; Eysenbach al., 2022; Park et al., 2025b). Also, to handle the base cases, we replace Q(s,, 0, 8k) with k- if k i < 1 and Q(st, ag. 9j) with - if k < 1 in the above loss.
L (R) E,-D [w(s;, D,,
a,, sy), Q(5 a,, S(s, ak, sy))]]
(12)
4.3.2 POLICY EXTRACTION
After training a coal-conditioned value function, we extract policy to maximize the learned value. In TRL, we consider two policy extraction methods, reparameterized gradients and rejection sam-
In this section, for notational simplicity, we describe high-level idea based on the V version of the transitive Bellman operator (Equation (9)). In practice, use the Q version (Equation (10)).

---

Published as a conference paper at ICLR 2026
Digression: Does random midpoint sampling still lead to logarithmic Bellman recursions?
While we motivated divide-and-conques value learning from logarithmic dependency on the horizon in Section TRL does not always divide a trajectory into two equal-sized chunks. It instead samples random subgoal 8y between 8; and sj. One might wonder whether TRL's s random "sampling" variant still gives Ollog 7) recursions. The answer is still yes (in the ideal, tabular case), and we provide a proof in Section A.
Algorithm Transitive Reinforcement Learning (TRL)
> Value learning
Initialize value function Q(s, 0,), policy x(a s,9)
while not converged do
Sample (5o: ao, $1:..., ST)
Sample f. ~ Unif (10, 1,
13) such that i <
Sample k ~ Unif i + 1.....j-1111.....
Train Q by minimizing 1, (Q) (Equation (12))
Policy extraction (can be in parallel with above)
Extract ı using either reparameterized gradients (Equation (5)) or rejection sampling (Equation (6)
pling (Section 3). By default, we use reparameterized gradients, but we find rejection sampling to work better long- long-horizon puzzle tasks, where the behavioral policy is highly multi-modal.
provide pseudocode for TRL in Algorithm
5 EXPERIMENTS
In this section, we empirically answer several research questions about divide-and-conquu value learning and TRL. In Section 5.1, we compare TRL's divide-and-conquer update rule with previous TD- and MC-based update rules large-scale, long-horizon tasks. In Section 5.2, we compare TRL with previous offline GCRL methods on standard benchmark suite. In Section 5.3, we provide various ablation studies of TRL. We use four random seeds unless otherwise stated, and present 95% confidence intervals in plots and standard deviations in tables. In tables, we highlight numbers that are at or above 95% of best performance blue, as Park et al. (2025a).
5.1 Is DIVIDE AND CONQUER BETTER THAN TD AND MC ON LONG-HORIZON TASKS?
Our first goal is to see whether TRL's divide-and-conques value learning algorithm exhibits better scalability to long-horizon prob- lems than more standard approaches (TD, n-step TD, and MC) in practice. This was our main motivation for TRL (Section l).
Tasks. To empirically answer this question, employ three highly complex, long-horizon environments used in recent horizon scaling study by Park et al. (2025b): iittanoodmazee giintt, puzzle 4x45, and pu22le-4x6 (Figure 2).4 In humanoi dwwze- -giant the agent must control humanoid robot with 21 degrees of freedom to igate a large maze. In puzzle-4xB and puzzle-4x6, the agent must control robot arm press buttons solve the "Lights Out" puzzle. Each environment provides five tasks (i.e., five state- goal pairs) for evaluation. These tasks are highly challenging and have long horizons: for example, the test-time task horizon of humanoi dmaze setazze-ginn ant is 4000. For datasets, we employ the 1B- sized ones used by Park et al. (2025b).
humano idnazedddeeeeee idmaze-giatt
puzzle-4x5
puzzle-4x6
Figure Long-horizon tasks.
giant
We omit cube cube-octuple from Park et al. (2025b), none of the methods achieve non-trivial performance this task without hierarchical policies, which orthogonal the focus of this paper (i.e., value learning).

---

Published as a conference paper at ICLR 2026
Table 1: Results on large-scale, long-horizon tasks. TRL achieves performance on highly challeng- ing. long horizon benchmark tasks that require up to 3000 environment steps.
Fnnmant
TQI.
WGCSL
ETQI
QRI.
taski
G
52 AM
5x
0
13
D
2
64
71
bask?
2
humano dww.zeegeaaaee
task3
0.
16
68 140
1
2
01
1.11
16
44
baaks
2.
57
00
0.
1:2
2.12
78
94
tasks
overall
0aee
3
22
23 aM
3 22
2
2
3
79
79
taak
100
100
taak?
0.20
20
0 29
03
0 09
0
0 03
am
puzzle-4x5
task3
dn
0
0
0
D
0 #:
2
100
tasks
no
0
10
1
ma
0
0
0 63
2
8 k::
overall
20
1:
19
00
0.
taski
87
D
100
task2
0.
0.
k:
0
3 67
0
0.
2
46
86 a:3
puzzle-Ax6
task3
task4
0.0
0.
0.10
0.
0:
0.
0.
0.
1:1
tasks
0de
0.
0.an
0.
0.
0
0.
0.50
0.
0.
humano i dmaze-giant
puzzle-4x5
puzzle-4x6
Success Rate
100
50
50
20
TD-1
TD-6
TD-10
TD-25
TD-
TD-1
100
MC
TRL
TD-1
TD-6 5
TD-10
TD-25
TD-
TD-50
TD-100
MC
TRL
TD-1
TD-5
TD-10
TD-25
TD-50
TD-10
100
MC
TRL
Figure TRL matches the best, individually tuned TD-n baseline, without needing to set n.
Methods. In these environments, we mainly compare the divide-and-conquer value update rule of TRL with existing value learning paradigms.
TRL (ours), which employs divide-and-conquer value learning (Equation (12)).
TD and TD-n, which employ (n-step) temporal difference value learning (Section C.1).
MC, which employs Monte Carlo value learning (Equation (2)).
To conduct a controlled experiment that solely focuses on the value update rules, we fix the other hyperparameters, e.g., goal hindsight relabeling ratios and policy extraction methods, except for the BC coefficient a in DDPG+BC (Equation (5)), which we individually tune for each method and task. This makes the results fully compatible, enabling apples-to-appee comparison of different value learning paradigms.
Additionally, for reference, we also evaluate several standard offline GCRL algorithms on these long-horizon tasks. We consider four methods considered in the work by Park et al. (2025b), flow BC (FBC) (Chi et al., 2023), IQL (Kostrikov et al., 2022), CRL (Eysenbach et al., 2022), SAC+BC (Park et al., 2025b), WGCSL (Yang et al., 2022), and HIQL (Park et al., 2023), and three existing methods that employ the soft or hard triangle inequality, QRL (Wang et al., 2023), TDP (Kaelbling, 1993; Dhiman et al., 2018; Jurgenson et al., 2020), and COE (Piekos et al., 2023). Among them, IQL, SAC+BC, WGCSL, and HIQL are based on TD learning, and CRL is based on MC learning.
Results. We present the main comparison results in Table and the comparison results with TD- (1,5, 10, 25, 50, 100} in Figure 3. The results suggest that TRL achieves the best performance across all three environments, outperforming matching both TD- and MC-based approaches as well as other previous GCRL methods. Notably, TRL is the only triangle inequality- basedd method that achieves non-trivial performance these challenging tasks.
In particular, we highlight that 1-step TD learning struggles on long-horizon tasks. While TD-n MC value learning improve performance, Figure 3 indicates that it requires careful, task-
8

---

Published as a conference paper at ICLR 2026
dependent tuning of In contrast, TRL achieves the best performance without needing to set n, matching the performance of the best TD-n baseline individually tuned for each task. This highlights the benefits of our wride-and-conquee value learning framework.
5.2 How DOES TRL COMPARE TO PRIOR METHODS ON STANDARD BENCHMARKS?
Next, compare the performance of TRL with existing offline GCRL algorithms OG- Bench (Park et al., 2025a), a standard benchmark offline goal-conditioned RI. The main research question here is whether TRL is comparable existing algorithms regular, not necessarily long-horizon tasks.
Tasks. We consider 10 goal-reaching tasks from OGBench. They span across robotic maze igation ((point, ant, humanoid]maze)) ball control (antsoccer), robotic object manipulation (cube, scene) and puzzle solving (puzzle). We use the oraclerep variants of these tasks, as in Park et al. (2025b). For datasets, we use the standard navigate and play datasets consisting of task-agnostic trajectories that randomly navigate the maze or perform random tasks.
Methods. We consider five widely used offline GCRL methods in this section: BC, FBC, IVL (Kostrikov al., 2022; Park et al., 2023), IQL (Kostrikov al., 2022), CRL (Bysenbach al., 2022), and QRL (Wang et al., 2023). As in the previous section, additionally consider two algo- rithms that employ the soft triangle inequality, TDP (Kaelbling, 1993; Dhiman et al., 2018; Jurgen- son et al., 2020) and COE (Pickos et al., 2023).
OGBench Performance (10 Environments, 50 Tasks)
7
17
BC
FBC
IVL
IQL
CRL
QRL
TDP
COE
TRL
Figure 4: TRL achieves strong performance standard OGBench tasks. While TRL is specifically designed for short-horizon tasks, outperforms matches previous GCRL methods a standard benchmark.
Results. Figure 4 shows the aggregated performance of each method over 10 OGBench environ- ments and 50 evaluation tasks (see Table 2 for the full table). The results suggest that TRL achieves the best performance on average. While the gap between TRL and the second-bes, best method (IQL) is narrower than that in Section 5.1, this is expected some extent, since TRL has its strongest ad- vantages in long-horizon tasks. We also note that TRL is the only triangle inequality-base method that achieves strong performance these robotic tasks.
5.3 ABLATION STUDIES
We present ablation studies three components of TRL (expectile K, subgoal distributions, and distance-based re-weighting factor X) in this section.
Expectile K. TRL uses expectile regression approximate the max operator in Equation (9), We evaluate TRL with E {0.5, 0.7}. and present the results in Figure 5. The results show that while 0.5 (i.e., behavioral regression) works well in humanoidmaze- i and scene-play, x > 0.5 is crucial for achieving strong performance
Yemeaae glant
hrrmuse mndium
nonne play
100
502
320
Stepe
Figure 5: Ablation study the expectile
humanoi www.e-aiazeeeaaae The strong performance with = 0.5 on some tasks is akin to how one-step RL (i.e, behavioral value learning) is often enough to achieve solid performance in offline RL (Brandfonbrener al., 2021; Park et al., 2025b).

---

Published as a conference paper at ICLR 2026
Subgoal distributions. Another key feature of TRL is the use of behavioral (in-trajectory) sub- goals. In Figure 6, ablate this choice by com- paring behavioral subgoals ("in-traj"") with ran- dom subgoals ("random"). The results suggest that random subgoals substantially degrade per- formance, highlighting the importance of using behavioral subgoals.
sotne-play
pucale-tod
Performanes
100
Stege
Stage
Steps
Figure 6: Ablation study on subgoal distributions.
Distance-based re-weighting factor X. Fig- ure 7 shows an ablation study on TRL's distance- based re-weighting factor Note that a posi- tive À makes TRL focus more on shorter trajec- tory chunks. The results show that enabling re- weighting (i.e., À > 0) often leads to better per- formance.
cube-dveble
sosne-play
puzat
Perforance
502
120
Figure 7: Ablation study the weighting factor A.
WHAT'S NEXT?
In this work, we have shown that Transitive RL exhibits better horizon scalability than standard TD- MC-based value learning approaches in offline goal-conditioned RL. At a high level, this provides a piece affirmative evidence for the hypothesis we posed in Section 1: namely, a divide- and-conquer paradigm may potentially lead to ideal goal-conditioned value learning method that is free from the curse of horizon.
This is only the beginning of the journey. For example, it remains open question whether similar divide-and-conquer value learning technique could be applied to learn an unbiased value function in stochastic environments (which is limitation of TRL, as well as many other works leveraging the vanilla triangle inequality in GCRL). The successor temporal distance framework proposed by Myers et al. (2024) may provide clues to this question. Another open question whether TRL (or any divide-and-come awwaanndd-n---sslllleelea algorithm) can be extended to general reward- based RI. tasks, beyond goal-conditioned RI. We hope the ideas and techniques introduced this work help address these important open questions and facilitate future progress toward scalable value learning algorithms.
ACKNOWLEDGMENT
This work was partly supported by the Korea Foundation for Advanced Studies (KFAS), the NSF Graduate Research Fellowship, AFOSR F49550-22-1-0073,, and ONR N00014-22-1-2773. This re- search used the Savio computational cluster resource provided by the Berkeley Research Comput- ing program at UC Berkeley.
REPRODUCIBILITY STATEMENT
We provide the code at https: http://w....oomomrraraaaaaaaaaaa and describe the full experimental details in Section C.
REFERENCES
Ravi Tej Akella, Benjamin Eysenbach, Jeff Schneider, and Russ Salakhutdinov. Distributional dis- tance classifiers for goal-conditioned reinforcement learning. In ICML Workshop on New Fron- tiers in Learning, Control, and Dynamical Systems, 2023.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Neural Information Processing Systems (NeurIPS), 2017.
Léonard Blier, Corentin Tallec, and Yann Ollivier. Learning successor states and goal-dependent values: A mathematical viewpoint, ArXiv, abs/2101.07123, 2021.
10

---

Published as a conference paper at ICLR 2026
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman- and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:// cechub..om/jax-ml/..
David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. In Neural Information Processing Systems (NeurIPS), 2021.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In International Conference on Learning Repre- sentations (ICLR), 2023.
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robolics: Science and Systems (RSS), 2023.
Vikas Dhiman, Shurjo Banerjee, Jeffrey M Siskind, and Jason Corso. warsslw reinforce- ment learning: Learning from past experiences to reach new goals. ArXiv, http/8809.9999 2018.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridg- ing planning and reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2019.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. International Conference on Learning Representations (ICLR), 2021.
Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2022.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS). 2021.
Shen-Xiao Gao, Chenyang Wu, Mingjun Cao, Chenjun Xiao, Yang Yu, and Zongzhang Zhang- Behavior rebavioo-regulariizd diffusion policy optimization for offline reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), 2025.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations (ICLR), 2023.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected- q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning (ICML), 2021.
Dibya Ghosh, Chethan Bhateja, and Sergey Levine. Reinforcement learning from passive data via latent intentions. International Conference on Machine Learning (ICML), 2023.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit g-learning as an actor-critic method with diffusion policies. ArXiv, abs/2304.10073. 2023.
Joey Hejna, Jensen Gao, and Dorsa Sadigh. Distance weighted supervised learning for offline inter- action data. In International Conference on Machine Learning (ICML), 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). ArXiv, abs/ has/1606.084155 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems (NeurIPS), 2020.
Michael Janner. Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning (ICMI.), 2022.
11

---

Published as a conference paper at ICLR 2026
Tom Jurgenson, Or Avner, Edward Groshev, and Aviv Tamar. Sub-goal trees framework for goal- based reinforcement learning. International Conference on Machine Learning (ICML), 2020.
Leslie Pack Kaelbling. Learning to achieve goals. In International Joint Conference Artificial Intelligence (LJCAI), 1993.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishman, Vincent Vanhoucke, and Sergey Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning (CoRL), 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference on Learning Representations (ICLR), 2022.
Aviral Kumar, Aurick Zhou, G. Tucker, and Sergey Levine. Conservative q-learning for offline
reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv, abs/2005.01643, 2020.
Yunfei Li, Yilin Wu, Huazhe Xu, Xiaolong Wang, and Yi Wu. Solving compositional reinforcement learning problems via task reduction. In International Conference Learning Representations (ICLR), 2021.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. In International Conference Learning Representations (ICLR), 2016.
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code. ArXiv, abs/2412.062644 2024.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off-policy estimation. In Neural Information Processing Systems (NeurIPS), 2018.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach. Learning temporal distances: Contrastive successor features can provide a metric structure for decision- making. In International Conference on Machine Learning (ICML), 2024.
Whitney Newey and James Powell. Asymmetric least squares estimation and testing. Economet- rica, 55:819-847, 1987.
Giambattista Parascandolo, Lars Buesing, Josh Merel, Leonard Hasenclever, John Aslanides, Jes- sica B Hamrick, Nicolas Heess, Alexander Neitz, and Theophane Weber. Divide-and-conquer monte carlo tree search for goal-directed planning. ArXiv, abs/2004.1144 2020.
Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned with latent states as actions. In Neural Information Processing Systems (NeurIPS), 2023.
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline rl? In Neural Information Processing Systems (NeurIPS), 2024a.
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In International Conference on Machine Learning (ICMI), 2024b.
12

---

Published as a conference paper at ICLR 2026
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench:
Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICLR), 2025a.
Seohong Park, Kevin Frans, Deepinder Mann, Benjamin Eysenbach, Aviral Kumar, and Sergey Levine. Horizon reduction makes scalable. In Neural Information Processing Systems (NeurIPS), 2025b.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. In International Conference on Machine Learning (ICML), 2025c.
Piotr Piekos, Aditya Ramesh, Francesco Faccio, and Juergen Schmidhuber. Efficient value propa- gation with the compositional optimality equation. In NeurIPS Workshop on Condiionit Reinforcement Learning, 2023.
Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinchart, and Sergey Levine. Recon: Rapid exploration for open-world navigation with latent goal models. In Conference on Robot Learning (CoRL), 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing (ICML), 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transac- fions Neural Networks, 16:285-286, 2005.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin. and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2023.
Stephen Tian, Suraj Nair, Frederik Ebert, Sudeep Dasari, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Model-based visual planning with self-supervised functional distances. In Inter- national Conference on Learning Representations (ICLR), 2021.
Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang. Optimal goal-reaching rein- forcement learning via quasimetric learning. In International Conference on Machine Learning (ICML). 2023.
Yifan Wu, G. Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. ArXiv, abs/1911, 1111991111366 11361, 2019.
Haoran Xu, Li Jiang, Jianxiong Li. Zhuoran Yang, Zhaoran Wang. Victor Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. In International Conference on Learning Representations (ICLR), 2023.
Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang. Yali Du, Xiu Li, Lei Han, and Chongjic Zhang. Rethinking coal-condiiooee supervised learning and its connection offline rl. In Inter- national Conference on Learning Representations (ICLR), 2022.
Chongyi Zheng, Ruslan Salakhutdinov, and Benjamin Eysenbach. Contrastive difference predictive coding. In International Conference on Learning Representations (ICLR), 2024.
13

---

Published as a conference paper at ICLR 2026
A PROOFS
In this section, we show that TRL's random sampling variant of divide-and-conques value learning still gives logarithmic dependency on the horizon in the ideal, tabular case. Let ((n) be the expected number of maximum Bellman recursions to compute the value ((s,,8) 8j) for a length- trajectory chunk, (s;; 844 1.-.. ...... sjmitn) with the random sampling variant of transitive Bellman
updates. Note that B satisfies the following recursive equation:
B(1) = O,
(n) 1
max B(k),, B(n k)),
for >> 1.
Then, we have the following results: Lemma A.1. Let n 2 2 be an integer and ((n) - Ex=1 max(k; n - k). Then C(n) S 3n/4.
Proof. If n 2m (even),
C(n)
max(ki, n k)
k=1
(m + 2m + 1) + ..... (2m 1)))
2m
(m + 3m(m - 1))
2m 1
3m
2m
2m
m
3
If n 2m + 1 (odd),
C(n) n
max (k: n - k)
1
(2((m + 1) +... 2m))
2m
1
(m(3m 1))
2m
3m + 1
2
6m 3
4
3
Theorem A.2. B(n) S log n/ log(4/3).
Proof. Recall that we have the following recursive equation:
B(1)= = O,
B(n) 1
max (B(),, B( k)),
14

---

Published as a conference paper at ICLR 2026
for > 1.
First, note that B() I log 1/1og(4/3). For n 2 2, we proceed by induction, assuming that ((m) log // log( 4/3) for all < Then,
B(n)
(ax, B(n k))
11
max log(n k))
11+
11
log(4/3)
k=1
=1+ log(4/3)
log(max(k: n 10k-k) k))
k=1
1
11 log (4/3)
log
max(k: (k, n kc)
71
=1 log(4/3)
log ((n)
3
$ +1+ + log(4/3)
log
4 n
log
log(4/3)
where we use the inductive hypothesis in the first inequality, Jensen's 's inequality in the second, and Theorem A.I in the third.
Theorem A.2 shows that the number expected maximum Bellman recursions in the random sam- pling variant of transitive Bellman updates also has logarithmic dependency on the horizon length.
15

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RESULTS
We provide the full comparison results on the standard OGBench tasks (Section 5.2) in Table 2.
Table 2: Full results on standard OGBench tasks.
Enviremment
Task
BC
F3C
IVL
IQL
CAL
QRL
TDP
00B
TRL
tack1
53 124
98 12
84
92
23 111
31
40 1m
87
62..u
task2
pointraze waterazzellrrgeeaeiiiee httpr//eeerrrrrooorrrrrrrrrrrrrr
task4
23
78 **
21 am
86 sm
44 437
sm
tasks
30 us
BT
04 km
0.00
Dise
4B am
48
64 a:
34
13
19 am
Bisx
g0
task2
22
24 ax
44 ese
64 aoa
6B am
22
26 am
86
sutnaze large praclerep-vv
task3
82
91
96
69
80
task4
tack5
15
16
20
61, a.1
16
overall
22
22 a4
21 as
37
85
67
27
26 as
46
task1
10
31
91
a:1
4
76
15 1.
9.
hurano d he
task4
tack3
2
8
16
0.
6
2.77
30
9:
0
13.2
11
8.10
task5
12
68
20
14.17
everall
tack1
1.11
9:
10
49 2.77
3
0use
task2
Dio
Dise
1
0
Dise
0.
task3
1
12
tasks
21a
2.
2:
20 1at
0.16
2.42
0.
overall
2et
Set
2,te
task!
7
24
task3
R
14
62 an
87
49 an
4.22
89
task4
2et
11
34
69
3
2
48
overall
17
13 a:
task!
7
17
67
97
64
7
98
task2
6
16
96
tacki
20 16
92
27 u:7
tack5
1
14
82
89
86 1:
1.
1.
24
87
tack2
B4
D.
mmbertoobble-llg-orrad erag----
task3
75
86
39
30
task5
0:n
overall
69 s9
64 s4
36 AM
Dase
30
taski
80
99
71
19
48
32
scca-ppaay-oraclooep--
tackd
2ix
19 s6
43 130
24
13 m:
12
en
6.
2
3
76 117
task5
3
23
10
2.2
16:1
tacki
14
10:11
1a
99
3.13
1.
tack?
1:
1.
2.
99
0.00
1
2
89
task3
100
tasks
96
2
Dase
99
overall
2
98
2
2:10
99
taski
33
0:gn
0:s
puzzle puzzle-(xi fxl---lx--llyy--a-e ora.lerelee
task3
0:
6:
58 6st
1:34
D.se
38 am
task4
1:1
6
22
1:
0:s
0:an
34
overall
1.46
28
16

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL DETAILS
In this section, we describe the full details of our experiments. We provide the code and instructions at https: /tt///ggttbb.../// www.coom/oo//e htt///bbbar/ttr...
C.I METHODS
Here, we describe the previous offline GCRL algorithms considered in this work.
BC, FBC, IVL (Kostrikov et al., 2022; Park et al., 2023), IQL (Kostrikov et al., 2022), CRL (Ey- senbach et al., 2022), HIQL (Park et al., 2023), and QRL (Wang al., 2023). We employ the original implementations by Park et al. (2025a;b), and we refer to these works for full details.
WGCSL (Yang et al., 2022). We use our JAX (Bradbury et al., 2018) re-implementation of the original implementation of WGCSL. We gradually change the advantage percentile from 0 to 80, and use a discounted relabeling weight of 0.999 and a default advantage weight émin of 0.05.
TD and TD-n. The TD (which is equivalent to TD-1) and TD-n baselines used in Section 5.1 minimize the following IQL-like objective:
(Q) = ETED [D (Q(si, as 81), 7
(13)
(Q) ETWD [De (Q(si, ai (i) (siimm dntn, 8j)]]
(14)
TQT (Q) (Q),
(15)
where trajectories 7 are sampled from the same trajectory distribution used in TRL. When ++ > j. appropriately clip the value of n (this is omitted in the objective above for notational simplicity). Intuitively, the objective above can viewed as the closest TD-based variant (ablation) of TRL.
TDP (Kaelbling, 1993; Dhiman et al., 2018; Jurgenson et al., 2020). We refer to triangle- inequality dynamic programming (TDP) as GCRL algorithms that take a hard maximum over the entire state space to compute optimal subgoal (Kaelbling, 1993; Dhiman et al., 2018; Jurgenson et al., 2020). These algorithms were originally designed for tabular environments, so we consider sampling based variant of TDP for experiments. Specifically, to compute a hard maximum in our continuous-come control environments, we sample M subgoal candidates from the dataset, and take the maximum over these M states. Formally, we minimize the following loss:
(Q) Es,anD [D (Q(s, s))
(16)
(Q) = Ex,a,s'<D D (Q(s, a ),))),
(17)
(Q) = B.,a.g WD [D
(18)
(Q( = Bugag.W-D D Q ((s,a,)) g),
max Q(s,a, a, (n,,a,))QGe Qw. aw, 9)
(19)
TDP (Q)
+ (L))))
(20)
where g" is a randomly sampled goal from the dataset (note that this distribution is different from that of y. as is typically partially sampled with hindsight relabeling). P is tunable hyperparameter (a large number to approximate the distance between a random (s.g) pair (Jurgenson et al., 2020)),
and W ((w()) ai
(i) ) is set of M randomly sampled state-action pairs from the dataset.
COE (Piekos et al., 2023). COE is a triangle inequality- based GCRL algorithm that uses a separate generator network G(s, g, S x A S + S to predict the optimal subgoal. Since this algorithm was originally designed for online GCRL, we add an additional behavioral regularizer to constrain this generator network produce in-distribution states, as commonly done in offline RL (Wu al., 2019; Fujimoto & Gu, 2021; Tarasov et al., 2023). Formally, we minimize the following losses to train the value function and generator:
(Q) = E. D (Q(s,,,8)),,))
(21)
L (Q) = E s,a.grD,
[D (Q(s, a, g). Q(s a, w) (ww,g)
(22)
w=G(s.a.g)
I CCO (Q) 1 (Q),
(23)
COB (G) = E E. "D WD.
[-Q(s, a, w) Q(w, (w.9).9) -B1[w --'
(24)
==G(s,agg)
17

---

Published as a conference paper at ICLR 2026
where 9" is a randomly sampled goal from the dataset, B is hyperparameter that controls the strength of the regularizer. the above, we slightly abuse the notation by assuming that the goal- conditioned policy x is deterministic.
C.2 ORACLE DISTILLATION
In our experiments, we mainly employ the oraclerep variants, following Park et al. (2025b). In oraclerep environments, we are given an "oracle" representation of goals (p(g) + 2, where 2 is an oracle goal representation space), which typically corresponds to subset of the state dimen- sions. For example, in humano dmaze, a full (goal) state a 69-dimensional vector, but the corre- sponding oracle representation is 2-dimensional vector consisting only of the z-y coordinates. In these environments, we condition the policy and value functions on the oracle representation, not on the full goal state.
While we can easily incorporate this change in standard MC- and TD-based algorithms by simply pa- rameterizing the policy and value functions with the oracle representation (e.g., using TT s, (g))) instead of π (a s. s,g),, is not straightforward to use y in triangle inequality- methods, such QRL and TRL. because they assume that states and goals lie in the methods compatible with oracle representations, we apply a technique we call "oracle distillation." function, Q+ S A x Z R,
space. To make such
That is, we train additional oracle pppresentation-condiionn by distilling it from the original Q function, Q S x A x S + I, using the following loss:
T, distiil (Q+) g D (Q (s. p()))))) Q( (.))).
(25)
Here, the original Q function assumes that the state and goal spaces are the same and can therefore leverage the triangle inequality. After training Q", we extract an oracle eppresentation-condttoon policy from this distilled Q function.
C.3 IMPLEMENTATION DETAILS
Training and evaluation. In our experiments, we train all agents for 1M gradient steps. For eval- uation, use 15 episodes for each of the five evaluation goals provided by OGBench (Park al., 2025a). tables, we report performance averaged over the last three evaluation epochs (i.e., 800K, 900K, and 1M steps), following the protocol of OGBench. However, for the baselines from the work by Park et al. (2025b) (i.e,, FBC, IQI., CRI., SAC+BC, and HIQL in Table I), we use the evaluation result only at the 1M epoch, they were evaluated every 500K steps.
In Table 1, we take the performance of FBC, IQL, CRL, SAC+BC, and HIQL from the work by Park al. (2025b). In Table 2, we re-run all baselines (BC, FBC, IVL, IQL, CRL, and QRL) using the original OGBench hyperparameters (Park et al., 2025a) on the oraclerep tasks (except for the value goal hindsight relabeling ratios, which we describe below). these tables, we ensure apples- to-apples comparisons between TRL and other baselines by using the same default configurations (e.g., we use the same network size, discount factor, training steps, etc.)
Hyperparameters. We provide the full list of hyperparameters in Tables 3 to 5. For the experi- ments in Section 5.1, we mostly follow the hyperparameters by Park al. (2025b), and for the ex- periments in Section 5.2, we mostly follow the ones by Park et al. (2025a). In the hyperparameter tables, the tuple (ptur Pgeom Piraj Prama) denotes the hindsight goal relabeling ratios described in OGBench (Park et al., 2025a).
18

---

Published as a conference paper at ICLR 2026
Table Hyperparameters for long-horizon OGBench tasks (Table 1).
Hyperparameter
Value
Gradient steps
Optimizer
Adam (Kingma & Ba, 2015)
Leaminng
0.0003
Batch size
1024
MLP size
[1024, 1024, 1024, 1024]
Nonlinearity
GELU (Hendrycks Gimpel, 2016)
Target network update
0.005
Discount factor
0.999
Policy (p P Ploom Pioni Prana) ratio
Pleai-
(0,0, 1, (humanoidmaze). (0,0.5, 0.5) (puzzle)
Value (pen (Peuc goom: p
gar- quua) ratio (TRL. MC)
(,00,
Value (Paue Pogen (gga))pmma) ratio (CRL)
1,0,0
Value (Pear Pgeamm Pwrg- Prand) ratio (others)
(0.2,0.5, 0.0.3)
Policy extraction
Reparameterized gradients rejection sampling Table 5)
Table Hyperparameters standard OGBench tasks (Table 2).
Hyperparameter
Value
Gradient steps
10"
Optimizer
Adam (Kingma & Ba, 2015)
Learning
0.0003
Batch size
1024
MLP size
[512, 5112,1125112
Nonlincarity
GELU (Hendryeks Gimpel, 2016)
Target network update
0.005
Discount factor
0.99 (default), 0.995 (humamoidmaze)
Policy (p Pp Prom Peray pi
ganaa) ratio
(0,0,1,0)
Value (ne (Paun W
"uaj "gaja ratio (TRL., CRL)
(0, 0,
Value (p PeurmP Piraj.
Prand)Pron! Prond) ratio (others)
(0.2, 0.2,0.5000033
Policy extraction
Reparameterized gradients Table
Table 5: Task-specific hyperparameters. describe task-specific hyperparameters below (ar BC coeffi- cient for reparameterized gradients, N: sample count for rejection sampling, M: subgoal count, P: random goal distance, 3: goal regularization coefficient, <: expectile, X: distance- based re-weighting factor).
QRL
TDP
TDP
TDP
COE
COE
TD
KC
TRL [a,N) N)
TRL
TRL
Environment
Na
M
(a, N)
Na.
humano ddnaze- -gattt
0.001
(3, -]
8
1000
1
0.5
(0.3, -)
(0.3, -)
(0.1.-)
0.7
puzzle-4x5
3
(-,32
1000
3
10
(-, 32)
32)
(-,32)
puzzle-4x6
(-,32) 32)
(-,32) .2)
{-,38 32)
(-,32)
pointasze-large
0.0003
500
(10,
antmaze-large
0.003
(10.-
100
3
10
(0.7.
0.7
humanoidnaze-madii
0.001
[5,
500
0.1
3
(0.1.
0.7
humanooidnsse--arge
0.001
(5.
500
0.1
(0.1.
antsoccer-arena
0.003
(10.
0.3
(0.3.
cube-single
0.3
(5,
500
0.3
10
(1.
cube-double
0.3
(5,
8
500
0.3
10
(10,
0.7
scene
0.3
(1,
16
200
10
(1.
0.7
puzzle-3x3
(5,
10
(2.
puzzle-4x4
0.3
500
10
(2.
19


Reviews:
[{"review_id": "FQTGKKz9D7", "reviewer": "Reviewer_7UcZ", "summary": "This paper presents a novel goal-conditioned RL method with D&C updates. The discussion is restricted to discrete, deterministic RL problems, but I still find the work interesting.", "questions": "1. It is mentioned that “in practice, different trajectories can have different lengths.” I wonder about the technical details of handling trajectories of various lengths.\n2. Why is Eq. 7 a multiplication?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. The proposed idea is novel and interesting. I agree with the authors that this paper is a first step towards a promising direction.", "weaknesses": "1. The authors restrict the discussion to discrete, deterministic environments with trajectory data of equal lengths, although they claim that their proposal can be extended to continuous, stochastic environments and various-length trajectories. I suggest the authors actually do such extensions and present the extended version.\n2. The tasks used in the experiments are synthetic without clear real-life purposes.\n3. Appendix A is unnecessary. It's just homework-level math.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1mlm9lDTbc", "reviewer": "Reviewer_EYPx", "summary": "The paper introduces Transitive Reinforcement Learning (TRL), a new divide-and-conquer framework for value learning in offline goal-conditioned reinforcement learning (GCRL). The core idea is to exploit the triangle inequality of temporal distances to decompose long-horizon value estimation into shorter, composable subproblems. They replace the hard maximization over subgoals with soft expectile regression to mitigate overestimation bias and restrict subgoal selection to in-trajectory behavioral subgoals, which further stabilizes learning in the offline setting.", "questions": "1. The ablation study indicates that restricting subgoals to behavioral in-trajectory states is essential for achieving stable performance, whereas using random subgoals leads to a substantial degradation. This sensitivity suggests a strong dependence on subgoal quality and spatial distribution. Would the authors consider exploring an adaptive or learned subgoal sampling mechanism that dynamically selects informative intermediate goals during training?\n\n2. It would be valuable to understand how TRL performs when extended beyond goal-conditioned reinforcement learning to more general reward-based RL tasks. Do the authors expect the divide-and-conquer principle to remain effective in such settings, and have any preliminary experiments been conducted in that direction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. TRL’s key strength lies in its scalability to long-horizon tasks (up to 4000 steps), where it consistently outperforms or matches the best TD- and MC-based baselines. By reducing the Bellman recursion depth to logarithmic complexity, TRL fundamentally mitigates the bias accumulation problem that plagues TD methods over long trajectories.\n2. In contrast to TD-n approaches, TRL attains superior performance without the need for laborious, task-specific tuning of the horizon parameter 𝑛, offering a more robust and parameter-free alternative for long-horizon value learning.", "weaknesses": "1. The proposed approach fundamentally depends on deterministic environment dynamics for the triangle inequality assumption to hold. Extending the framework to learn unbiased value functions in stochastic environments remains an important and open avenue for future research.\n\n2. As shown in the ablation study, the results are highly sensitive to subgoal selection, which may introduce additional instability when applied to stochastic or noisy environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9scAvzxn5K", "reviewer": "Reviewer_KdC6", "summary": "This paper introduces a method called TRL for GCRL. The idea is to long horizon tasks by formulating them as GCRL which then allows \"dividing and conquering\" to stitch together shorter plan segments to avoid the small errors that normally stack up and ruin long-term plans.", "questions": "**Q1.** The authors state they take some baseline results from a prior paper (Table 1) but re-run others (Table 2). For the re-run baselines, how was fair tuning ensured? For example, the SGT-DP and COE baselines (Table 1) are also triangle-inequality methods that perform poorly - how can the authors be sure this is not due to a poor implementation or unfair tuning on their part?\n\n**Q2.** The oracle distillation seems important but it is only mentioned in Appendix D.2. Could you do an ablation of this?\n\n**Q3.** Using the BCE loss over the MSE loss seems unusual. Why was this chosen? Did you try MSE as well?\n\n**Q4.** The authors claim \"we experimentally find that behavioural subgoals can still be highly effective even when the dataset consists of uniformly random atomic motions\". Which experiments does this refer to? It would strengthen the paper to have results across different dataset types - \"random,\" \"medium,\" and \"expert\" - to support this claim.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The idea seems sensible and the results seem strong. It seems to make the idea of triangle inequality for value learning work, although I am not very familiar with current GCRL literature.", "weaknesses": "**W1.** Complicated algorithm with lots of moving parts: eta-quantile, M subgoals, expectable loss, reweighting, separate \"oracle distillation\" network, policy extraction. This is many more \"moving parts\" than standard algorithms like IQL or TD-n, which could make it difficult to tune and implement.\n\n**W2.** As far as I understand it relies on oracle representations (see Appendix D.2). This seems like a significant weakness.\n\n**W3.** Structure: The triangle inequality is referred to in the abstract, introduction, related work (where there’s a whole subsection on it) etc, but only properly introduced in the middle of page 4. The triangle inequality really needs to be described earlier - in or before the dedicated discussion paragraph in related work.\n\n**W4.** The paper tests on one suite of benchmarks. It would benefit from results on another set of benchmarks.\n\n**W5.** Limited to GCRL and deterministic environments.\n\nOther weaknesses: See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UJr6yASeWH", "reviewer": "Reviewer_6EwX", "summary": "This paper proposes a method called Transitive RL for offline goal-conditioned RL. It introduces a novel divide-and-conquer approach to learn a value function with reduced accumulated error. On the recent OGBench benchmark, it demonstrates superior performance compared to well-established baselines such as IQL and CRL.", "questions": "1. In Table 1, TRL significantly outperforms the selected baselines. However, I encourage the authors to include more recent offline GCRL methods, such as HIQL [1] and WGCSL [2].\n\n2. It would be preferable to combine Sections 5.1 and 5.2, as they largely overlap, and move the ablation studies to the main pages.\n\n3. Section 5.1 shows that TRL performs well on large-scale, long-horizon tasks. However, this evidence alone is insufficient to demonstrate that the performance gain arises from the divide-and-conquer value learning. The authors should design more detailed experiments to verify that the value function learned by the proposed method is indeed more accurate and less prone to overestimation, as stated in line 256.\n\n**References**\n\n[1] Park, S., Ghosh, D., Eysenbach, B., \\& Levine, S. (2023). HIQL: Offline Goal-Conditioned RL with Latent States as Actions. NeurIPS. \n\n[2] Yang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., … Zhang, C. (2022). Rethinking Goal-Conditioned Supervised Learning and Its Connection to Offline RL. ICLR.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. This paper provides an insightful connection between $V(s, g)$ and $\\max_w V(s, w) V(w, g)$. Accordingly, the authors derive a novel divide-and-conquer value function to replace the target in Eq. 10.\n\n2. The proposed algorithm is simple and achieves strong performance on recently proposed benchmarks.", "weaknesses": "The authors’ main hypothesis is that a divide-and-conquer algorithm *might* provide a path to overcoming the curse of horizon (line 55). I would argue that this is not a well-defined hypothesis. It would be more appropriate to state that a divide-and-conquer value learning algorithm can mitigate the curse of horizon in offline goal-conditioned tasks. Accordingly, **the paper’s framing should be revised to emphasise the offline goal-conditioned setting, rather than starting broadly from off-policy RL**.\n\nI find this submission interesting; however, the presentation and narrative could be improved. The paper lacks a smooth transition from the problem statement (and background) to the proposed idea, and then to the corresponding solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)