Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
Authors: Hongjoon Ahn
Heewoong Choi
Jisu Han
Taesup Moon
Venue: NeurIPS 2025 spotlight ()
Keywords: Reinforcement learning
Offline RL
Goal-conditioned RL
Hierarchical RL
Temporal Abstraction
Primary area: reinforcement_learning
TL;DR: We propose a novel value function learning scheme for hierarchical policy in offline GCRL

Abstract: Offline goal-conditioned reinforcement learning (GCRL) offers a practical learning paradigm in which goal-reaching policies are trained from abundant state–action trajectory datasets without additional environment interaction. However, offline GCRL still struggles with long-horizon tasks, even with recent advances that employ hierarchical policy structures, such as HIQL. Identifying the root cause of this challenge, we observe the following insight. Firstly, performance bottlenecks mainly stem from the high-level policy’s inability to generate appropriate subgoals. Secondly, when learning the high-level policy in the long-horizon regime, the sign of the advantage estimate frequently becomes incorrect. Thus, we argue that improving the value function to produce a clear advantage estimate for learning the high-level policy is essential. In this paper, we propose a simple yet effective solution: _**Option-aware Temporally Abstracted**_ value learning, dubbed **OTA**, which incorporates temporal abstraction into the temporal-difference learning process. By modifying the value update to be _option-aware_, our approach contracts the effective horizon length, enabling better advantage estimates even in long-horizon regimes. We experimentally show that the high-level policy learned using the OTA value function achieves strong performance on complex tasks from OGBench, a recently proposed offline GCRL benchmark, including maze navigation and visual robotic manipulation environments. Our code is available at https://github.com/ota-v/ota-v

Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
Hongjoon Ahn' Heewoong Choi Jisu Han
Taesup Moon 1.2.3
Department of Electrical and Computer Engineering (ECE), Seoul National University Interdisciplinary Program in Artificial Intelligence (IPAI), Scoul National University
ASRI INMC, Seoul National University
(hong0805, chw0501, ishodi, t.smoon @snu. ac kr
Abstract
Offline goal-conditioned reinforcement learning (GCRL) offers a practical learning paradigm in which goal-reaching policies are trained from abundant state-action trajectory datasets without additional environment interaction. However, offline GCRL still struggles with long-horinn mor-horizon tasks, even with recent advances that employ hierarchical policy structures, such as HIQL [33 Identifying the root cause of this challenge, we observe the following insight. Firstly, performance bottlenecks mainly stem from the high-level policy's inability to generate appropriate subgoals. Secondly, when learning the high-level policy in the long-horizon regime, the sign of the advantage estimate frequently becomes incorrect. Thus, we argue that improving the value function to produce clear advantage estimate for learning the high-level policy is essential. In this paper. we propose a simple yet effective solution: Option-aware Temporally Abstracted value learning, dubbed OTA, which incorporates temporal abstraction into the -inporal-difference learning process. By modifying the value update to option-aware, our approach contracts the effective horizon length, enabling better advantage estimates even in long-horizon regimes. We experimentally show that the high-level policy learned using the OTA value function achieves strong performance complex tasks from OGBench (32), a recently proposed offline GCRL benchmark, including maze navigation and visual robotic manipulation environments. Our code is available at httpps:/github.com.tt w/ota-v
Introduction
Offline goal-conditioned reinforcement learning (GCRL) has emerged as a practical framework for real-world applications by leveraging pre-collected datasets to train goal-reaching policies without requiring additional environment interaction 23 32. However, learning an accurate goal-conditioned naively training the value To mitigate the learning of erroneous policy, Hierarchical Implicit Q-Learning (HIQL) 33 one of the state-of-the-ar methods, adopts a simple hierarchical structure in which a high-level policy predicts subgoals, and low-level policy learns to execute actions toward those subgoals. Though a hierarchical policy is still learned from the noisy value function, both policies receive more reliable learning signals than when training flat, non-hierarchica policy. However, despite reasonable performance gains of hierarchical methods in some long-howe long-horizon environments, a recent challenging benchmark [32
value function in Long-horizon settings remains major challenge,
function often leads to noisy estimates and erroneous policies [3| 20
33).
Equual Contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

reveals that such hierarchical policy still cannot solve more complex tasks, such as long-horizon robotic locomotion or robotic manipulation.
To understand the failure in complex tasks more deeply. we raise the following question: Low-level policy vs. high-level policy: which is the bottleneck of HIQU? To answer this question, we analyze the hierarchical policy in failure cases by generating oracle subgoals for the low-level policy. Interestingly, we observe that the low-level policy achieves these subgoals with high accuracy, indicating that the failure stems from the inability of the high-level policy to generate appropriate subgoals. The limited performance primarily results from noisy value function, which fails to provide sufficiently informative learning signals for effectively training the high-level policy in long- hong-horizon scenarios.
Based on the phenomenon that the high-level policy eventually failed to extract meaningful learning signals from the value function, we identify the primary cause of these noisy signals as the order inconsistency of the learned value function in the long-horizon setting. Our analysis reveals that when the distance between the state and the goal exceeds a certain temporal horizon, the sign of the advantage estimate incorrect, causing erroneous regression weights for learning the high-level policy. Considering the issue with the value function, we argue that designing a value function that can produce clear advantage estimate for learning the high-level policy is necessary.
Motivated by the observation that the low-level policy performs remarkably well at reaching short- horizon subgoals, we propose simple yet effective value function learning scheme for high-level policy learning that reduces the horizon between the state and the goal. Specifically, we leverage the notion of option [45), semporally-extende course of action, by updating the value over sequences of primitive actions. This option-aware value learning substantially shortens the effective horizon compared to primitive action-aware value learning
(21), mitigating errors in long-horizon value estimation. We evaluate our approach on maze and robotic visual manipulation tasks from OGBench [32] and empirically show that using our value function enables the high-level policy achieve superior performance on long-horizon tasks.
In summary, contributions are threefold:
Through analysis of the failure cases of hierarchical policies, we identify that the failures stem from the inability of the high-level policy to generate appropriate subgoals. Further- more, we observe that the value function used for high-level policy learning has significant errors when the distance between the state and the goal is large.
To tackle this problem, we propose Option-aware Temporally Abstracted (OTA) value learning, which reduces the effective horizon compared to the conventional value learning objective (21]
Our experiments show that, even across long state-to-goal horizons, our value function achieves significantly lower errors, enabling the hierarchical policy to successfully solve complex maze and robotic manipulation tasks.
2
Related Work
GCRL. GCRL aims to train goal-conditioned policies to reach arbitrary goal states from given initial states, rather than optimizing for single, fixed task 42 25). Our work focuses specifically on offline GCRL [4, 26 52
43 32] in which goal-conditioned policies are learned entirely from pre-collected datasets without further environment interaction. Due the sparse rewards in goal-
33
reaching tasks, offline GCRL has relied on hindsight data relabeling [1] 40 [55) and more recently, imitation learning and value-based methods have been explored to better leverage suboptimal datasets 61 52 12). In these works, the value function is typically learned through temporal-differeme (TD) methods |1 34], or through alternative techniques such as statc-occupancy matching [26] 77) contrastive learning [27 9 24, and quasimetric learning [50]. However, whether the value functions can effectively generalize to long-horizon tasks remains an open question 32
Hierarchical RL. Achieving long- morghoorizon goals remains a fundamental challenge in GCRL 37
35
20 33
To address this, hierarchical RL methods have adopted either graph-based planning in
state space [8]
16 54
20
53 19 or waypoint-based subgoal generation 5
22
47
28
17
13
31
30
18
3
33
51. However, graph-based planning methods incur high computational overhead and
architectural complexity. Waypoint-basee approaches also face challenges in generating effective

---

subgoals in long-houe long-horizon settings, due to inaccurate value estimates when the state is far from the goal.
Option framework. To enhance the planning capabilities of an agent over long time horizons, one effective approach is to leverage temporal abstraction through the option framework, which involves learning sub-policies known as options [L] 45 41 (35]. In this framework, options serve as temporally extended actions that enable planning across multiple time scales. After establishing the theoretical connection between the option framework and semi-Markov decision processes [S],
46 and automatic option discovery [88 [S]. Our method is closely related to HIQL [33.. which trains high-level policy to generate subgoals. However, unlike HIQL, our approach leverages options defined in offline datasets to effectively reduce the planning horizon during value function training. As result, our high-level policy can generate subgoals over longer temporal horizons without relying on explicit option learning or option discovery.
research has progressed toward end-to-end option learning
[39
44 2
3 Preliminaries
Problem setting. Offline GCRL is defined over a Markov Decision Process (MDP), consisting of (S,A, ".,," p) in which S denotes the state space, AA the action space, 9 the goal space, r(s,g) the goal-conditioned reward function for state 8 E S and goal E g y the discount factor, po(-) the initial state distribution, and p(-|s, a) the environment transition dynamics for state E S and action a E A. We also denote V(s, [) as the goal-conditioned value function state s given goal g. We assume that the goal space is the same the state space (i.e., S = 9). An offline dataset 'D consists of trajectories T = (30, ao; 81, ST), each sampled from an unknown behavior policy jr. The objective is to learn an optimal goal-conditioned policy T[a|s.)) g) that maximizes the expected cumulative return J (T) = Eroep" (r).grp(g) o 7'r(s(: g)1, where p (r) (=( T p(8 |st, (t (8t, g). and p(g) is goal distribution.
Hierarchical Implicit Q-Learning (HIQL). In GCRL, accurately estimating the value function for
distant goals is the main challenge in solving complex long-horizon tasks [66 20 33). To address this
issue, HIQL [33] proposed hierarchical policy structure that utilizes value function learned with IQL 22 This hierarchical design enables the agent to produce effective actions even when value estimates for distant goals are noisy or unreliable. More specifically, HIQL trains a gond-condi state-value function V with the following loss:
((V) = E(e,e") a~p(s) [15 (r(s, 9) + 7V(s' V(s,
where the expectile loss is defined (u)
l < u² with 0.5, and V denotes the target V network F Following prior works 1 3
50 33 51). we adopt the sparse reward r(s,g) 1(s # g}. Under this reward, the optimal value (* g) corresponds to the discounted temporal distance, i.e., a discounted measure of the minimum number of environment steps required to reach the goal g from state s. HIQL separates policy extraction Cinraattioooiioooooo into two levels: a high-level policy r (S1+k (st.)) generates a k-step subgoal to guide progress toward the goal, while a low-level policy o' (uelse, $1++) produces primitive actions to reach the subgoal. Both policies are extracted using with the following objective:
advantage-wa wewigtago-weighted regression (AWR) [48 36
29
I B(e 9)D [exp hh AA A (S1,8884, 8144: 9)) log n ( (s)4))
(2)
J(T²) = B (1 /exp (g² (s,, $1+1, 8()))) log x (as St, s(++)] (3)
where 8hh and 3 are inverse temperature parameters, A k. g) = denotes the high-level policy advantage, and A A((s,, 8141, 8t+&) = (8 8141) V' 8(+4)
- (s(, 9)
denotes the low-level policy advantage. HIQL uses a single goal-conditioned value function V. which is shared between both T² and n" (i.e., Vh V = V). However, despite this design, HIQL still struggles with long-horizon, complex tasks, as shown in the recent offline GCRL benchmark. OGBench 32]
Note that since the inherent over-estimation problem of IQL, in this paper, assume that the environment dynamics is deterministic.
'Policy extraction refers to learning a policy from a learned value function, emphasizing the separation between value learning and policy learning.
3

---

AntMaze
HIQL
HIQL
Humanoi daaze
100
Success Rate (%)
80
60
20
0
medium explore
large explore
giant navigate
giant stitch
large navigate
giant navigate
giant stitch
stitch
Figure/ I: High-level policy is problematic. We evaluate HIQL by varying only the high-level policy while keeping the low-level policy fixed. The x-axis denotes different tasks under maze sizes and data types (See Section 6.1 for task details). Using learned high-level policy (HIQI,, ग = x o x x²), performance drops, whereas using the oracle high-level policy (HIQL. OS श = N' ग dracte) achieves high success rates, indicating the high-level policy is the main bottleneck.
4 Motivation
4.1 Low-Level Policy vs. High-Level Policy: Which is the Bottleneck of HIQL?
We investigate the failure cases of HIQL in long-horizon scenarios by identifying whether the main performance bottleneck is in the low-level policy or the high-level policy. To examine this, we fix the low-level policy n' and replace the high-level policy n² with an oracle policy T which always provides optimal subgoals reachable within a short horizon We refer to this variant as HIQU Os and pose the following hypothesis: if HIQI. os still fails in long-horizon tasks, then the low-level policy struggles to reach short-horizon subgoals. Conversely, if it achieves high success rate, the main problem lies in the high-level policy.
Figure shows the results of HIQL and HIQL os eight challenging maze navigation tasks from OGBench (32]. HIQL achieves less than 20% success rate on many tasks, indicating that HIQL significantly fails to solve the long-horizon tasks. In contrast, we note that HIQL O8 achieves a much higher success rate around 90%. These results indicate that, although the low-level policy generalizes well in short-horizon settings when provided with accurate subgoals, the primary failure of HIQL in long-horizon scenarios stems from inaccuracies in the high-level policy.
We identify two potential issues in Equation 2 that may underlie the failure of high-level policy learning: (1) an inadequate policy extraction scheme (i.e., the regression component in Equation (2)). and (2) an inaccurately learned value function (i.e., the advantage term in Equation (2). Since the same policy extraction scheme enables successful low-level policy learning, we do not consider to be the primary cause of failure. This suggests that the inaccurate value function used in the high-level policy advantage term may be the key contributor to the failure. In particular, as the distance between 8; and g increases, the value estimates become increasingly erroneous, leading to an imprecise evaluation of the high-level advantage. Although HIQL attempts to mitigate the noise in estimating the long-horizon value 4 through its hierarchical structure, the high-level advantage may still be substantially erroneous. In the following subsection, carefully analyze how such errors in estimating v adversely affect high-level policy learning.
4.2 Order Inconsistency of the Learned Value Function in the Long-Horizon Setting
Before analyzing the learned 4 in HIQL,, we first define order consistency of the value function.
Definition 4.1. (Order consistency) Assume that the environment is deterministic. Let 7* g) denote the optimal trajectory induced by the optimal policy xt( 8. the initial state so to the goal g, and let V be a learned value function. Given $i, 8j E with j > i, we say that V satisfies order consistency with respect to (s,, 8j g) if and only if V (s V (ss.g).
(S0, $1....,
'Specifically, A provides as a subgoal the center of an adjacent maze cell that lies on the shortest path from the current state to the goal.

---

300
d
50
200
-100-
100
100
100
150
500
500
d g) (s g))
d (s. g) (s 5+))
ds g) (5-g)
d g) (5 g))
(a) AntMaze-rrgge-xplorr
(b) AntMaze-glant-stitch
400
600-
-50
V
100
100
400
200
150
150
d*
2000
2000
2000
2000
(s 91 Is
g)
d dL 9) (s-9)
d 91 (s - g)
d'(s,g) g) (5-1)
(c) HumaaniiaMaae-llage--ttthhh
(d) HumanoidMaze-giinn-ttttt
Figure 2: Value order inconsistency in long-horizon settings. (Leff) We collect optimal trajectories from the initial state (0) to the goal (sk). (Middle) At each state along the trajectory. we compare the high-level value from HIQL (V*) and the optimal (V*). (Right) To better illustrate value order consistency, we convert the values into temporal distances: HIQL (d") and the optimal (d*).
Consider optimal trajectory r* { $1,.... ) generated by oracle policy. Along this trajectory, the optimal value function is increasing, such that (*(s). 9) > Thus, value order consistency refers to the alignment between the order induced by y and that induced by V We argue that achieving the order consistency between V(st, g) and V(s++k; g) is critical, as sign mismatches can invert the high-level advantage estimate A and hinder the learning of an appropriate high-level policy. With large values (e.g., 25 in AntMa ze 100 in Humano iMMze)), the sign mismatch of the advantage estimate can lead to significant performance degradation. When the advantage sign is incorrect, the magnitude of regression weights (which is the exponentiated advantage) is drastically increased or decreased, leading to improper subgoal regression for high-level policy. For example, if the range an ad advantage is 1,, 1] with 3h = 3. the regression weights vary from - xs 0.05 to e² xy 20, indicating that sign flip can significantly change the weight magnitude.
g) for all j > i.
To check whether the learned 7 of HIQL achieves the order consistency or not, we collected optimal trajectories for four different long-horizon tasks with specified goals using near-optimal policies, as illustrated in Figure 2 The trajectory lengths varied from 250 to 2000 steps. For each state 8, in the trajectory, we then visualize the learned value V"(*t, 9) alongside the optimal value function, computed as (s(.g) = - (1 - (1- (v..))) (1-111 in which d' (st, g) denotes the temporal distance between St and g. Since the value decays exponentially as the distance to the goal increases due to the discount factor 7, directly comparing relative values is visually challenging. Hence, transform V*(sgg) into estimated temporal distances using the following equation: (s. log (1 (1 (s,"
(8, / log 7. In this form, the criterion for value order consistency becomes d (st, g) > d (sj, g). where j
As shown in Figure we note that 4 closely matches V* when the state is near the goal (i.e., d' d'(x, (8.9) 2s 0). This alignment explains the strong performance of the low-level policy presented in Figure However, when the state-gou goal distance exceeds certain temporal horizon, the value order inconsistency frequently arises between V'h(st 9) and Vhh (st+k; g) due to the non-monotonicity of the learned /4 (6 This is due to the well-known fact that the learning target for the value in Equation (I) becomes noisier as the horizon becomes longer, and shows why the use of V becomes less effective high-level policy learning for long- long-horizon settings.
Motivated by the observation that 4 aligns well with /* and achieves order consistency in short- horizon settings, we propose simple yet effective solution based on temporal abstraction 45) This approach enables high-level value function learning to provide appropriate advantage estimates, even when d' (s, gg is large.
'hhe hyperparameter A in HIQL is chosen based on the characteristics of the environments and datasets. In = 25 for the AntMaze task and lo = 100 for the Humano dMaze task.
Figure
5

---

Option-aware Temporal Abstraction (OTA)
High-level advantage estimate A² along
r g) y (s 9) VOTA gs.
(o. v')
Sample
Offline dataset
So
Horizon contraction
Original TD learning
r(s.g) g) + +y 9) V (s.g)
State
Primitive Action @ Option
neg
pos
Figure 3: Option-aware temporal abstraction. (Left) OTA achieves temporal abstraction by computing the reward and target value from the state reached after executing the option (i.e. s.). (Right) By leveraging temporal abstraction, OTA provides clear high-level advantage estimates, particularly in long-horizon tasks.
5 Option-aware Temporally Abstracted (OTA) Value
In this section, we propose straightforward solution for learning Vh g) by leveraging options [45] to reduce the horizon length. An option can be regarded as temporally extended sequence of primitive actions that enable temporal abstraction. In our offline RL setting, an option starting from the state S; corresponds to sequence of n actions (a,, dt+1, ... at+x-1) extracted from trajectories in the offline dataset. By using temporally extended actions in planning, we reduce the effective horizon length, referring to the number of planning steps, to approximately d' (st, g)/n. Therefore, to ensure that the high-level value / is suitable for long-term planning, we modify the reward and target value Equation
TT to be option-aware.
Specifically, for a given abstraction factor and goal g. we define an option 2n.. (I, μ. Br where I S the initiation set, μ is the behavior policy used to collect the offline dataset D. and Sn.o is timeout-based termination condition that ends the option after steps or upon reaching Let s' si(2nng: st) denote the state resulting from executing 2n.g at state st, which is either 8t+n or g. For brevity, we denote s' s'(2n.g. g):.s s) as S Then, we reformulate the value learning objective in Equation (T) into an option-aware variant:
L(VOTA: n) (.(..
--oo (5 - (s. 9))]
where r(s9.g) - + g}{
We refer to V as the Option-aware Temporally Abstracted
(OTA) value function.
We argue that the high-level value function V would effectively address the value order incon- sistency. Using a 1-step target for value learning tends to be more sensitive to noise, especially in long-horizon tasks, whereas an option-aware target mitigates noise and empirically produces more order-consistent value estimates. The overall framework for learning V is illustrated in Figure 3
Connection to n-step TD learning. The target used in n-step TD learning and that in OTA value learning are closely related, as both primarily rely on the n-step forward value However, the key distinction between n-step TD and OTA lies in the choice of the discount factor 7, which controls how information decays during the TD update. Standard n-step TD learning typically uses the same y as in 1-step TD, causing the discount factor applied to the n-step target to decay exponentially with n. contrast, the discount factor in the OTA target is independent of n. This excessive decay in the standard n-step target hinders the order-consistent value learning, indicating that a direct extension from the n-step target to the OTA target is not straightforward. Instead, temporal abstraction through the option framework provides natural explanation for the insights presented in Section 4.2 As shown empirically in 6.5 with respect to value order consistency, standard n-step TD learning offers no advantage over 1-step TD learning.
We highlight the differences from Equation in Equation
'The TD target used in restep TD learning
1 (stti # g) V

---

GCBC
GCIVL
GCIQL
QRL
CRL
HIQL
OTA
100
AntMaze
HumanoidMaze
Success Rate (9%)
80
60
20
0
medium
large
glant
medium
large
giant
Robot manip
Figure 4: Evaluation on OGBench. We run 8 seeds for each dataset and use the performance
reported in OGBench for the baselines. For maze tasks, we report the average success rate grouped by maze size. For visual robotic manipulation, we report the average success rate across the four tasks.
Experiments
6.1 Experiment Setup
Tasks. We use OGBench 32)
recently proposed offline GCRL benchmark designed for realistic environments, long-horizon scenarios, and multi-goal evaluation. The Maze environment consists of long-horizon navigation tasks that evaluate whether the agent reach speeified goal position from given initial state. The Maze environments are categorized by agent type [PointMaze, AntMaze, and Humanoi dMaze), maze size (medium, um, large, and giant), and the type of trajectories in the dataset (mavigate, stitch, and explore). The Maze environments are well suited evaluating performance in long-horizon settings. For example, the Humano dazze- -iieee--iann -giant environment has a maximum episode length of 4000 steps.
The Visual-cube and Visual-scent environments focus visual robotic manipulation tasks. In Visual- the task involves manipulating and stacking cube blocks to reach a specified goal configuration. This environment is categorized by the number of cubes: s single, double, e, and triple. In contrast, Visual-ss sscene requires the agent to control everyday objects like windows, drawers, two-button locks in a specific sequence. Both visual environments use high-dimensional,, pixel-based observations with 64 x 64 x 3 RGB images. The robotic manipulation environments have shorter episode lengths (250 to 1000 steps) compared to the Maze environments. These robotic environments are a strong benchmark for evaluating the performance of an algorithm on high- dimensional visual inputs. A detailed description of the environments is provided in Appendix [..l
Baselines. For brevity, we will refer to the policy that utilizes the high-level policy learned with the OTA value OTA. We compare OTA against six representative offline GCRL methods included in OGBench. Goal-conditioned behavioral cloning (GCBC) is a simple behavior cloning method that directly imitates actions from the dataset conditioned on the goal. Goal-conditions
implicit V-learning (GCIVL) and goal-conditioned implicit Q-learning (GCIQL) [1] 33 estimate the goal- conditioned optimal value function using IQL-based expectile regression, and extract policies using AWR (36] and behavior-regularizzed deep deterministic policy gradient (DDPG+BC) [10] respectively, Quasimetric RL (QRL) [50] learns a value function that estimates the undiscounted temporal distance between state and goal via quasimetric learning and trains policy using DDPG+BC. Contrastive RL (CRL) g approximates the Q-function via contrastive learning between state-action pairs and future
states from the same trajectory, and trains the policy using DDPG+BC. HIQL [33 extends GCIVL with hierarchical policy, as detailed in Section
6.2 Evaluation on OGBench
We evaluate success rates on 14 datasets, including (AntMaze, HumanoidMaze ; (medium, large, giant)-( (navigate, stitch) and AntMaze- (medium, large) -explore. For both AntMaze and stuaanoidaaze we report the average success rate grouped by maze size. Additionally, for visual robotic manipulation, we evaluate the average performance across four tasks: Visual-Cube (single, double, triple) and Visual. wassuallSeeeee Scene. As shown in Figure most non-hierarchical baselines (i.e., GCBC, GCIVL, GCIQL, QRL, CRL) consistently fail on long-horizon tasks. While HIQL., a hierarchical policy, achieves up to 40% success on challenging tasks such AntMaze giant and Humano dl large, its performance drops significantly
7

---

VOTA
d
OTA
=0.6)
r(V") =0.67
Fly") =0.57
((VG
Bl -0.89
0.5
11.00
0.5
0.5
F0.88
0.5
0.5
0.5-
0.0
0.0
0.0
0.0
250
250
0
1000
1000
2000
2000
(s.
(s.g.)
d'
d'
d dls..]
d*(s.g)
(a) AntMaze-large-explon
(b) HumanooddMace-lrreennvvga
(c) HumamordaMezz--lrgeesttttt
1.0
((y)) -0.76 ST
-0.46
F(V)) -0.82 ~V =0.9
GeA
0.5
0.833
0.5
0.5
= 0.89
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0.0
500
500
0
1000
1000
2000
2000
ds
d'
d' g)
ds
d'(s.g)
(d) AntMaze-glant-stttcc
(e) Hummanoiddazz--glaannnnaavaa
(f) Humanodddaze-glan-ssttee
Figure 5: Value and temporal distance estimation. We visualize min-max normalized V Vo dh and dora and the order consistency ratios (VV)) and (VV))) across six different datasets.
in the most difficult setting, HumanoidMaze-giiann highlighting its limitations in long- mong-oorizon settings.
In contrast, we observe that OTA achieves a significant performance improvement over all baselines. Notably, as the maze size increases (i.e., from med um to l large to giant), the performance gap between OTA and other methods widens substantially. These results suggest that OTA performs effec- tive temporal abstraction and enhances high-level policy performance, even as task horizons become longer. Full benchmark results, including the PointMaze tasks, are provided in Appendix D
6.3 High-level Value Function Visualization
In Figure we compare the high-level value function / learned with HIQL and V learned with OTA across six challenging tasks. Using the visualization method from Figure [ we plot 4 and Van along optimal long-horizon trajectories r", together with the corresponding temporal distances dh and d The figure clearly shows that Vo exhibits a more monotonic increase than Vh, particularly when the distance between and g is large. To quantify this improvement, compute the order consistency ratio rf, which measures how reliably value estimates from (St, St+kk, .)) E r* produce directionally correct signals for high-level advantage estimation. Specifically, r"(V)
E 1(V(s g) > ((stg))}/T - + 1). where is fixed and 8t. $t+k E T Across all tasks, we observe that that (V()) > (V*),, indicating that OTA yields more order-consistent value estimates 3 Therefore, we confirm that OTA improves high-level value estimation in long-horizon tasks, leading to better high-level policy learning.
6.4 Effect of Varying Abstraction Factor 72
Learning the value function VoTA depends on the abstraction factor n, which determines the degree of temporal abstraction. Figure a--c)
a-c) illustrates how the value function changes as n is varied across 1, 2, 3, 5, 10, and 20 in Equation 4 while keeping the optimal trajectory and goal fixed for each dataset. As shown in Figure [] b,c), for long-horizon trajectories (i.e., those exceeding a length of 1500), the absolute scale of the value function increases with larger n. This trend arises since option termination condition introduces reward of -1 every n steps, which effectively compresses the value range as n increases.
Temporal abstraction not only changes the scale of the value function but also impacts the quality of the value estimation. Figure Figureb[a-c) shows that when d' (s. g) increases, which aligns with limitations commonly observed in standard HIQL. However, as n increases, the value function becomes more suitable for long-horizon tasks. To further evaluate the effect of temporal abstraction, we examine the order consistency ratio r", as shown in Figure O
the value function fails to learn as
We set &e 25 for AntMaze environment and A 100 for Humanoi idMaze environment.

---

AntMace large explore
20
HumánoidMizee herge-siitch
HumanoldMaze -un glant-naviggaae
1.0
-50
-50
Vêra
-100
-10
100
1000
df ds(s,g)
d' g) (b) HumanoidMaze darge-stitch
d d' o)
-arge-explore
(a) AntMaze
HumanoidMaze -giant-navigate
(d) consistency
Figure 6: Value estimation and order consistency. (a-c) Estimation of the value function VOTA with
varying abstraction factor n (d) Order consistency ratio r ("(OA across different values of
Table I: Average success rate and order consistency ratio. Simply using n-step TD learning increasing the discount factor in HIQL insufficient to achieve the perfor- mance improvements of OTA. Here, the dataset ALE refers to AntMaze-large-explle and HGS to
AGS AttMaze-giant-sticc. HLS to -umaroichaaee-argeeeeeee Humano -aaai-iiaaaannssstttte.
Success rates
Order consistency ratios e HIQL
Datasets
HIQL
OTA
OTA
I-step, y
n-step. 7
1-step, 7/"
1-step, 7
n-step.
1-step, ///
ALE
4 15
10
3 13
75 116
0.75 10.01
0.77
10.01
0.76
10.02
10.01
ACS
2 12
50 10
0
16
10.01
0.84
10.02
0.79
10.02
10.01
HLS
12
22
13
0.76 10.01
0.76 10.00
0.75
20.02
0.89
10.03
BGS
28
42
2
43
0.71 40.01
0.72 40.00
0.72 =0.01
20.01
which generally increases with However, beyond certain point, larger causes a drop in rV()), indicating that excessive temporal abstraction may lead loss of information.
6.5 Impact of n-Step TD and Increasing the Discount Factor 7
In the original HIQL, the high-level value function y is discounted by 7 at every step. contrast, the OTA value function Vo applies discounting only every steps. To investigate the source of the effectiveness of OTA, we modify the value learning approach of standard HIQL in two ways: (1) using n-step TD learning, and (2) increasing y. We evaluate both the success rate and the order consistency ratio rd across four datasets. In Table [ we set n = 15 for MntMaze and n = 20 for Humano i dMaze. To compute rf, collect trajectories per dataset and report the average consistency ratio (see Appendix B.2.3 for details of the collected trajectories).
The first variant uses the original y with n-step TD learning in HIQL, denoted as HIQL(n-step, 7). Table shows that this approach yields almost no improvement rf, and the success rates also show little gain except for Humanoi claaze aaaaae-aaaaaaeeeeee www.aaarrastiii targets still suffer from value function estimation errors when the discount factor remains unchanged.
These results indicate that n-step TD
The second variant keeps 1-step TD learning but modifies the discount factor to ///n, Under OTA training, the optimal value function becomes V*(*e, g) -(1 7 (sus)/") /) - 7). Therefore, to approximate this temporally abstracted optimal value function, a possible approach is to increase the discount factor to 1/n in Equation outperform standard HIQL in either success rate or r In contrast, OTA achieves significant gains in long-horizon tasks such as Humano dMaze-ga ant-stir ch. The experiments demonstrate that simply adjusting the discounting factor alone is insufficient, and the temporal abstraction is crucial for effective value learning in Long-horizon tasks.
[1 However, Table shows that simply increasing 7 fails to
Our analysis further suggests that n-step TD learning could potentially be improved by carefully adjusting y for each n. However, this would introduce additional complexity hyperparameter selection. In contrast, OTA fixes y regardless of n, which makes the approach much simpler.

---

6.6 Scalability Comparison of TD-Based OTA and QRL
Here, we demonstrate that OTA, which leverages a TD-based IQL loss, scales effectively with increasing state and action dimensionality. As discussed in Section 4.2] conventional TD methods rely on discount factor, which causes the advantage estimate to decay exponentially over long horizons. To avoid this issue, we explore alternative value learning approaches that do not depend on a discount factor.
In particular, we consider QRL, which learns undiscounted temporal distances between states through quasimetric learning (see Appendix C for more details). However, QRL relies on min-max optimization, which becomes computationally challenging in high- dimensional state spaces. As shown in Table 2 QRL achieves significantly lower success rates complex tasks such as Humano dMa ze and ví sual -cscen.. These results highlight the scalability and the practical advantages of our TD-based OTA, particularly in environments with high-dimensional state spaces.
Table Success rates for different high-level values.
Datasets
QRL
HIQL
OTA
gattaaree-iiaatt-naiieee
76 +2 65 #o
77 #4
12 13
12 24
92 10
Visual-cube-doublee
6
59
13
65
12
Visual-scene
12
#1
12
7
Conclusion
In this paper, we investigated the limitations of the hierarchical offline GCRL method HIQL,, particu- larly in long-horizon tasks. Our analysis revealed that the main performance bottleneck lies in the high-level policy, which suffers from inaccurate value estimates when the state-goal goal distance is large. To address this challenge, proposed OTA, method that incorporates temporal abstraction into IQL-based value learning through the concept of options. Experiments on challenging long-horizon goal-reaching tasks show that OTA enables high-level policies to achieve substantial performance improvements in long-term planning. The simplicity and effectiveness of OTA highlight its potential for advancing hong-horizon offline GCRL in real-world applications.
Acknowledgments
This work was supported in part by National Research Foundation of Korea (NRF) grant [No. 2021R1A222007884 No. (RS-2025-02263628) the Institute of Information communications Technology Planning & Evaluation (ITTP) grants 2021111221206 RS-2022-112201 13, RS-2022- I1220959, RS-2021-1122 1343], and the BK21 FOUR Education and Research Program for Future ICT Pioneers (Scoul National University), funded by the Korean government (MSIT).
10

---

References
[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[2] Pierre-Luc Bacon, Jean Harb, and Doina Precup. for the Advancement of Artificial Intelligence (AAAT), 2017.
The option-critic architecture. In Association
[3] Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. conliconditioned reinforcement learning with imagined subgoals. In International Conference Machine Learning (ICML), 2021.
[4] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jake Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In International Conference on Machine Learning (ICML), 2021.
[5] Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 1992.
[6] Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[7] Ishan Durugkar, Mauricio Tec, Scott Nickum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[8] Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[9] Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[10] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[11] Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations (ICLR), 2021.
[12] Xudong Gong, Feng Dawei, Kele Xu, Bo Ding, and Huaimin Wang. Goal-conditioned on on-policy reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[13] Nico Gürtler, Dieter Büchler, and Georg Martius. Hierarchical reinforcement learning with timed subgoals. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[14] Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas L Dean, and Craig Boutilier. Hierarchical solution of markov decision processes using macro-actions. In Conference on Uncertainty in Artificial Intelligence (UAI), 1998.
[15] Po-Wei Huang, Pei-Chiun Peng, Hung Guei, and Ti-Rong Wu. Optionzero: Planning with learned options. International Conference on Learning Representations (ICLR), 2025.
[16] Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[17] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language an abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.

---

[18] Tom Jurgenson, Or Avner, Edward Groshev. and Aviv Tamar. Sub-goal trees framework for goal-based reinforcement learning. In International conference on machine learning (ICML), 2020.
[19] Junsu Kim, Younggyo Seo, Sungsoo Ahn, Kyunghwan Son, and Jinwoo Shin. Imitating graph-based planning with goal-conditioned policies. In International Conference on Learning Representations (ICI.R), 2023.
[20] Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guided subgoal generation in hierarchi- cal reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[21] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming, In International Conference on Learning Representations (ICLR), 2022.
[22] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Sacedi, and Josh Tenenbaum. Hicrarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[23] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 2xXiv:2005.01642 2020.
[24] Grace Liu, Michael Tang, and Benjamin Eysenbach. A single goal is all you need: Skills and exploration emerge from contrastive r1 without rewards, demonstrations, or subgoals. In International Conference on Learning Representations (ICLR), 2025.
[25] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In International Joint Conference Artificial Intelligence (ICAAI, 2022.
[26] Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. In Advances in Neural Information Process- ing Systems (NeurIPS), 2022.
[27] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In International Conference on Learning Representations (ICLR), 2023.
[28] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[29] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09355 09359, 2020.
[30] Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long- horizon tasks via visual subgoal generation. In International Conference on Learning Representations (ICLR), 2020.
[31] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal- conditioned policies. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[32] Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl. In International Conference on Learning Representations (ICIR), 2025.
[33] Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned rl with latent states as actions. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[34] Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representa- tions. In International Conference on Machine Learning (ICML), 2024.
12

---

[35] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical rein- forcement learning: A comprehensive survey, ACM Computing Surveys (CSUR), 54(5):1-35, 2021.
[36] Xue Bin Peng, Aviral Kumar, Grace Zhang. and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00111 2019.
[37] Doina Precup. Temporal abstraction in reinforcement learning. University of Massachusetts Amherst, 2000.
[38] Rahul Ramesh, Manan Tomar, and Balaraman Ravindran. Successor options: An option discovery framework for reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), 2019.
[39] Jette Randlov. Learning macro-actions in reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 1998.
[40] Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[41] Matthew Riemer, Ignacio Cases, Clemens Rosenbaum, Miao Liu, and Gerald Tesauro. On the role of weight sharing during deep option learning. In Association for the Advancement of Artificial Intelligence (AAAI), 2020.
[42] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxi- mators. In International Conference Machine Learning (ICML), 2015.
[43] Harshit Sikchi, Rohan Chitnis, Ahmed Touati, Alborz Geramifard, Amy Zhang, and Scott Niekum. Score models for offline gonl-condd reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
[44] Martin Stolle and Doina Precup. Learning options in reinforcement learning. In Symposium on Abstraction, Reformulation, and Approximation (SARA), 2002.
[45] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1- 2):181-211, 1999.
[46] Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, et al. Strategic attentive writer for learning macro-actions. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
[47] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning (ICML), 2017.
[48] Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. In Advances in Neural Information Processing Systems (NeurIPS), 2018
[49] Tongzhou Wang and Phillip Isola. Improved representation of asymmetrical distances with interval quasimetric embeddings. In NeurlPS 2022 Workshop on Symmetry and Geometry in Neural Representations, 2022.
[50] Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang, Optimal goal-reaching reinforcement learning via quasimetric learning. International Conference on Machine Learning (ICML), 2023.
[51] Chengjie Wu, Hao Hu, Yiqin Yang. Ning Zhang, and Chongjie Zhang. Planning, fast and slow: online reinforcement learning with action-free offline data via multiscale planners. In International Conference Machine Learning (ICML), 2024.
[52] Rui Yang, Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjie Zhang. Rethinking goal-conditioned supervised learning and its connection to offline rl. In International Conference Learning Representations (ICLR), 2022.
13

---

[53] Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model as a graph: Learning latent landmarks for planning. In International Conference on Machine Learning (ICML), 2021.
[54] Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[55] Sirui Zheng, Chenjia Bai, Zhuoran Yang. and Zhaoran Wang. How does goal relabeling improve sample efficiency? In International Conference on Machine Learning (ICML), 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction section, we clearly explain our contribution devising an option-aware temporally abstracted value function learning scheme.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss on the limitations of OTA in Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not contain any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed descriptions of the environment setup and experimental hyperparameters in the Appendix, and the open-source code is released to ensure repro- ducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have released our code open-source to enable full reproducibility of the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.. cc/
public/ http://guudeee///eSuuuuuee ss son ollcc) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See NeurlPS code and data submission guidelines (https
/ //nips. httccc/pbbb. wtcc/pbblcc/gee wwt.//iieeeeeeee SCoeseubideeeeieee ss onPol icyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We show the details on the dataset, hyperparameters, training scheme, and baselines in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the error bars in the tables and bar plots in the Experiment section and Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We show the details on the compute resources (e.g., GPU) in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:p/ss//.uurpsss...co//bl//aa
Answer: [Yes]
Justification:
Fully conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is a foundational algorithmic study in offline goal-conditioned reinforcement learning. While the methods could potentially be used in future applications, there are no specific positive or negative impacts discuss.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used publicly available datasets and open-source code under appropriate licenses (c.g., CC-BY 4.0, MIT). All sources and licenses are properly cited in the paper.
Guidelines:
existing assets. The answer NA means that the paper does
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaperswi paperswithcooes com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have open- open-sourced our code and released the dataset used in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper doess not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We use LLM tool only for writing, editing, or formatting purposes.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : /nnurerrisps cc htcc/Coffeeences///c//oo 2025/ LLM) for what should should not be described.
21

---

Point
Humanoid
medium
large
giant
Ant
(a) Maze size
(b) Env type
Visual-cube-single
Visual-cube-double
Visual-cube-triple
Visual-scene
(c) Robotic visual manipulation
Figure 7: Dataset examples. For Maze environment, the task differ by (a) environment type (b) and dataset type. (c) -cube, the robot must manipulate the cube to the location specified by the blurred cube, which denotes the goal position.
A Limitations
Our method, OTA, has several following limitations. First, we introduce new hyperparameter, temporal abstraction factor n, reduce the effective horizon of the value function. Due to the additional hyperparameter, we should carefully select both k, the number of steps to reach subgoal, and Second, though carry out temporal abstraction obtain an order consistent value function for all state and goal pairs. Third, for the experiments on long-horizon tasks in which the trajectory length is than 1000, only use the maze dataset to evaluate our method.
the value function,
still cannot
B Experimental Details
B.1 Environments, Tasks, and Datasets
In this section, we provide detailed descriptions of each task, with dataset examples illustrated in Figure For more detailed description of the environment, see OGBench 32
Maze (Maze) a challenging long-horizon locomotion task, where the agent needs to reach the given goal position from the initial position. This environment is categorized into three different types of agent based on state and action dimension: 1) Pointmaze (PointMaze). which controls 2 degrees of freedom (DoF) point mass, 2) Antmaze (AntMaze), which controls a quadrupedal Ant with 8-DoF, and 3) Humanoidmaze (Humano (ammanoddMaze). dMaze), which controls 21- DoF Humanoid agent. Each maze environment is divided into medium, large, and giant based on maze size, from PointMaze-meet medium requiring a horizon length (i.e., maximum episode steps) of 1000, Humanoi dMaze giant requiring 4000. Each environment includes diverse aanasess-naviatt... st tithh and expl ore cre-collected via different dataset features:
navi gate: This dataset consists of trajectories collected as an agent, guided by a noisy expert policy, that attempted to reach randomly sampled goals.
22

---

Table Common hyperparameters for OTA. We refer to Appendix B.2.1 hyperparameter defini-
tion.
Hyperparameter
Value
Learning rate
3e-4
Optimizer
Adam
Minibatch size
1024 (Maze), 256 (Visual env)
Total gradient steps
1000000 (Maze), 500000 (Visual env)
MI.P dimensions
[512, GELU
Activation function
Target network smoothing coefficient
0.005
Discount factor
0.99 (default), 0.995 (Antmaze-giame HumanoicMazzi)
Image augmentation probability
0.5 (random crop)
Policy (peur: Phoj: Prona) ratio
(0.1.0) (default), (0,0.5, 0.5) (stitch), (0.0.1) (explore) (0.2, 0.3)
Value (pear Punjs Pmnna) ratio
stitch: This dataset contains shorter trajectories compared to those collected in the navigate setting. They designed to evaluate goal-stitching capabilities.
explore: This includes higher levels of action noise, resulting in lower-quality data, but with increased state coverage.
Visual-cube (Visual-cube) is challenging robotic visual manipulation task, where the agent must move and stack cube blocks to reach a specified goal configuration. The task includes three variants-singlee, double, and manipulated. The agent receives pixel-based images of the current observation and goal, each size 64 64 x 3, and outputs 5-DoF action vector. The task horizon ranges from 200 steps (s iigle) to 1000 steps (t (trip..) The agent is learned with noisy dataset, which is built from a suboptimal dataset with action noise, leading extremely low-quality data and longer effective horizons.
to the number of cubes that must be
Visual-scene (Visual souua-sceene scene) is also robotic visual manipulation task, where the agent needs to manipulate everyday objects window, drawer, two button meckswwhrre pressing a button toggles the lock status of the corresponding object (i.e., the drawer or the window). The agent receives pixel-based images of the current observation and goal, each of size 64 x 64 x 3, and outputs 5-DoF action vector. The task horizon range is 750, in that it involves unlocking object and manipulating the object. The agent is learned with noisy dataset, as mentioned above.
B.2
Implementation Details
B.2.1 Hyperparameters
use goal- sampling distribution for value and policy learning, following OGBench. Data sampling scheme is based on HER (I, taking three different goal-sampling distributions, definition is as follows:
implemented OTA on top of the official implementation of OGBench 332
Pu (glss s) is Dirac delta distribution centered at the current state (i.e., s),
Punj (g]s)) is the probability distribution over goals where each goal is uniformly sampled
from the future states within the same trajectory as the state 5,
(g]s) is the probability distribution that goal g is uniformly sampled from the entire
dataset D.
Task-specific hyperparameters are organized in Table 4 where hyperparameters are described in Equation (T) Equation (4).
23

---

Task category Type
OTA hyperparameters
Environment
Size
8h
g
k
Maze
navigate medium
0.5 3.0
25
large
3.0 3.0
25
5
qiant
3.0 3.0
20
5
PointMaze
stitch
medium um
1.0 3.0
20
large
1.0 3.0
20
10
giant
5.0 3.0
20
5
navigate medi um
1.0 3.0
25
5
large
1.0 3.0
25
5
giant
0.5 3.0
16
4
AntMaze
stitch
medium
0.5 3.0
25
5
large
1.0 3.0
25
giant
3.0 3.0
30
10
explore medium
3.0 3.0
25
5
large
3.0 3.0
20
15
navigate medium
0.5 3.0 100
20
large
0.5 3.0 100
giant
0.5 3.0 100
20
HumanoidMaze
stitch
medium
3.0 3.0 100
20
large
1.0 3.0 100
20
giant
0.5 3.0 100
20
Robotic visual manipulation
s single 1.0 3.0
20
4
Visual-cube noisy
double
3.0 3.0
20
4
triple 3.0 3.0
25
4
Vi sual -scene noisy
3.0 3.0 10
4
Table 4: Task specific hyperparameters for OTA. We refer to Appendix B.2.1 for each hyperparam- eter variable. Note that we individually tune the hyperparameters for each task.
24

---

Task category
Maximum episode length
Environment
Size
Maze
medium
1000
PointMaze
large
1000
giant
1000
medium
1000
AntMaze
large
1000
giant
1000
medium
2000
HumanoidMaze large
2000
giant
4000
Robotic visual manipulation
single double
200
Visual-cube
500
triple
1000
Visual-scene
750
Table 5: Maximum episode length of environments.
B.2.2 Training and Evaluation Details
In Maze environment, the model is trained for up to 1M gradient steps. We evaluate the model 800K, 900K, and 1M steps. At each evaluation point, measure the success rate using five fixed task goals provided by OGBench. Each goal is evaluated with 50 rollouts, resulting in 750 evaluation episodes per seed (i.e., 3 evaluation steps goals x 50 rollouts). We report the average success rate across these episodes and across 8 different random seeds. For Visual-cuke and Visual-- scene environments, the model is trained for 500K gradient steps. Evaluations conducted at 300K, 400K, and 500K steps using the same protocol: five fixed goals and 50 rollouts per goal. The maximum episode length of each environment is shown the Table 5 All results are averaged across 8 seeds. All experiments are conducted using NVIDIA RTX A5000 and A6000 GPUs.
25

---

path
path
path
path
path
UP
(a) AntMaze large environment path
path
path
path
path
APA
(b) AntMaze giant environment
Figure 8: Collected optimal trajectories for AntMaze environment.
We collect the optimal
trajectories from the initial state the goal
path
path
path
path
path
UH
LP
US
(a) HumanoidMaze large environment
path
path
path
path 4
path 5
(b) HumanoidMaze giant environment
Figure 9: Collected optimal trajectories for HumanoidMaze environment. We collect the optimal trajectories from the initial state (0) the goal
B.2.3 Collected Optimal Trajectories
To evaluate the order consistency of value for high-level advantage, we collect five optimal tra- jectories for each environment: AntMaze (large, giant) and Humano diMa ze- ( Large, giant). ). Each optimal trajectory is generated using the expert policy that was originally used during the offline dataset collection in OGBench.
The collected optimal trajectories for MntMaze and Humano dMa ze are shown Figures S and 9 respectively. Order consistency, as reported in Table is evaluated based on the five trajectories illustrated these figures and averaged over 8 random seeds. During value estimation, we apply moving average with appropriate temporal window size to smooth out short-term fluctuations and obtain stable value estimates. The optimal trajectories used for value visualizations in Figures 5 and 6 are as follows:
Trajectory selection for Figure
Figure 5 [): path 5
Figure b): path
Figure 5
c): path
Figure d): path
Figure c): path 5
26

---

Figure S D: path 1
Trajectory selection for Figure
Figure b(a): path
Figure 6 bb: path
Figure c(): path
C Quasimetric Reinforcement Learning (QRL)
QRL [50] is an goal-conditioned RL algorithm by utilizing the quasimetric structure for learning optimal value function V*, The quasimetrics are a generalization of metries in that they do require
symmetry. The optimal value function in QRL is an undiscounted temporal distance, V (s g) -d' (s.g), and the value function satisfies the triangular inequality, d' (s, s') + d' 9) 2 d' (s. g) for any s E S, and E G. To obtain the optimal value function using the quasimetric structure, the value function should have two properties: First, the value function should should have locally consistent value, d' (x, I -r. Second, the distance should be globally spread out, d" (8.g)
total cost of path connecting to g- To achieve those properties, QRI. optimizes the following objective to obtain the optimal value function:
AQE min max -B (logg) (8, ())))] + (n). A20
1OF (8, ()) - =²),
(5)
where ø is a monotonically increasing convex function, (-. is Interval Quasimetric Embeddings (IQE) [49] for the quasimetric model. the above objective, both the min and max operations should be applied simultaneously, which can induce unstable training. Using the value function, QRL
learns policy through optimizing the DDPG BC [10] like objective.
27

---

D Additional Results
D.1
Per-environment Results
We show the full per-environment results in Table 6 In this table, OTA outperforms the baselines in most cases.
Task category
Non-hierarchical
Hierarchical OTA
Environment
Type
Size
GCBC GCIVL GCIQL
QRL
CRL
HIQL
Maze
navigate medium
9
63
53
82
29
79
86
large
29 16
45 #5
34 #3
86
#9
39 #7
58 #5
85 #5
giant
1 42
0
0 +0
68
27
46
72
PointMaze
stitch
medium
23 +18
70 =14
21
80
+12
0
74
75 155
large
7 ::5
12 16
31 1:2
84
115
0 110
13 1:6
66 :18
giant
0 1:0
0
50
0 10
0 1:0
52 .17
navigate medium
29 1:4
72 118
71 1:4
88 13
95 11
96 11
96 .11
large
24
16
34
75
83
91
92
giant
0 1:0
0 :10
0 1:0
13
16 13
65 155
77 114
AntMaze
stitch
medium um
45 +11
44 #6
29 #6
59 =7
53 +6
94 #1
93 #1
large giant
3 13
18
11 12
67
84
0 +to
0 ::0
0 110
0 10
0 +00
2 #
37 :16
explore
medium
2 +
19 13
13
#1
3 12
37 10
94 .133
large
0 +:0
13
:0:
0 20
0 +
4 #
75 116
navigate medium
8 4:2
24 12
27 4:2
21 18
60 14
89 12
94 .111
large
1 1:
2
2
5
24 14
49 #4
83
giant
0 +00
0 =0
0 #0
=0
3 +2
12 +4
92 #11
numanoidMaze
stitch
medium
29 .15
12 12
12 13
18
36
1:2
88 1:2
88 .12
large
6 1:3
1
3 11
4 11
28 1:3
57 .13
giant
0 +00
0 =0
0
0
0 +0
3 12
79 1:33
Robotic visual manipulation
single
14
75
48
39
430
99
99
Viscal-cube noisy
double
5 411
17
22
6 12
6 13
59 13
65 .:2
triple
+1
9 14
+1
23 12
26
412
Visual-scene noisy
13 122
23 12
12 1:4
2 10
15
12
50 11
54
.12
Table 6: Performance comparison across various policy types and benchmarks. We shot average success rate 8 random seeds. Bold values indicate the best performance in each row. Baseline performances are the official results provided by OGBench.
28

---

D.2 Performance under Unified Hyperparameters
We report additional results where OTA is trained with the same hyperparameters HIQL, except for the temporal abstraction factor n. The experiments are conducted on complex maze environments.
As shown in Table 7 OTA consistently outperforms HIQL even under identical hyperparameter settings. This result indicates that incorporating temporal abstraction alone significantly enhances performance in long-horizon goal-conditioned tasks.
Task category Type
Hyperparameters 3
Methods
Environment
Size
n
k
3
HIQL
OTA
navigate large
25
3,0
3,0
58
8545
giant
25
3.0
3,0
46+9
72+6
PointMaze
stitch
large
25
3,0
3.0
13#6
46+7
giant
25
3.0
3.0
0.1
44
navigate large
25
3.0
3.0
91#2
91+1
giant
25
3.0
3.0
6511
70
AntMaze
stitch
large
25
3.0
3.0
67:5
7913
giant
25
3.0
3,0
2+2
29:5
explore medi um
25
3.0
3.0
37
9343
large
10
25
3.0
3.0
62:12 415
navigate large
20
100
3.0
3.0
4914
82.1
giant
20
100
3.0
3.0
1214
9111
HumanoidMaze stitch large
100
3.0
3.0
2813
4313
giant
20 100
3.0
3.0
312
61.3
Table 7: Performance under unified hyperparameters.
D.3 Performance on Visual Play Datasets
We evaluate the performance of OTA on visual play datasets, with results summarized in Table For fair comparison, we fix the hyperparameters 3 varying only 2.
, 38)) = (10, 3.0, 3.0) for both HIQL and OTA,
Task category
Methods HIQL. OTA
Environment
Data Env
play double
48 51.3
Visual-cube
play triple
21 28
Visual-scene play
50.5 5615
Table 8: Performance comparison on visual manipulation play dataset.
29


Reviews:
[{"review_id": "c9mQYHmm5V", "reviewer": "Reviewer_1VvB", "summary": "This paper focuses on the challenges of offline goal-conditional reinforcement learning (GCRL), pointing out that the failure of the hierarchical policy of HIQL is mainly due to the difficulty of generating suitable subgoals in the high-level policy, and the sign of the advantage signal frequently becomes incorrect. Thus, the paper proposes the Option aware Temporarily Abstracted (OTA) value learning method, which integrates temporal abstraction into temporal differential learning and improves the accuracy of advantage estimation. Experiments have shown that OTA performs well in complex tasks.", "questions": "1. Adjusting the temporal abstraction step of the value function in the high-level policy has improved the value estimation of long-horizon states, but does this have an impact on the value estimation of short-horizon states?\n2. Would it be better to use separate value functions with different step sizes for the high and low level policy?\n3. The temporal abstract step n and subgoal step k need to be adjusted in coordination. Is there any method that can more automatically determine the optimal combination of n and k instead of parameter tuning?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\\\nThis article first analyzes whether it is the bottleneck of the high-level policy or the low-level policy, then analyzes the reasons for the bottleneck of the high-level policy, and finally proposes methods. The whole process is gradual and easy to understand. The proposed method maintains simplicity. The experimental effect has been greatly improved.\\\nWeaknesses:\\\nIntroduced a new temporal abstraction step hyperparameter n, increasing the complexity of parameter tuning. The method improves the consistency of the value function, but it cannot guarantee complete orderliness and has certain limitations.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 1VvB for the positive comments on our analysis of the bottleneck in hierarchical policies and the simplicity of our method. We also appreciate your constructive feedback regarding the additional hyperparameter $n$ and the guarantee of complete orderliness, etc. Below, we address your concerns in detail.\n\n### Weakness 1: Additional hyperparameter $n$\n\nWe agree that introducing another hyperparameter $n$ may increase the complexity of parameter tuning. However, we would like to clarify that the search complexity of our method is not prohibitively large. Instead of performing an extensive hyperparameter sweep from scratch, you can follow the guidelines in Question 3 and attain strong performance within a much narrower search space. We hope this addresses your concern.\n\n### Weakness 2: A limitation in guaranteeing complete order consistency\n\nWe agree that our method does not fully guarantee perfect value ordering. Nevertheless, we would like to emphasize that our simple approach significantly reduces value‑ordering errors and, in turn, achieves much higher performance compared to the baselines.\n\n### Question 1: Impact of step $n$ on short-horizon value estimation\n\nThe objective of the high-level policy in long-horizon planning is to identify reachable subgoals that the low-level policy can achieve within a short-horizon. Accordingly, the high-level value function is only used for long-horizon planning. In contrast, training the low-level policy requires accurate value estimation within short horizons. \n\nThat is, short-horizon value estimation is utilized solely for training the low-level policy through the *low-level value function*, whereas the temporal abstraction step $n$ affects only the *high-level value function*. Therefore, adjusting the temporal abstraction step $n$ turns out not to have a direct impact on the value estimation of short-horizon states.\n\n### Question 2: Separate value functions with different step sizes for each policy\n\nWe apologize for the lack of clarity in the manuscript and appreciate the reviewer’s attention to this point.\n\nIndeed, we agree with the reviewer and already follow this design in our approach. Specifically, unlike HIQL which trains a single value function $V$ for both $V^h$ and $V^\\ell$, we train separate value functions for the high-level and low-level policies, each with a different step size -- i.e., the low-level value function, $V^\\ell$, is trained using a standard 1-step target, while the high-level value function, $V^h$, uses an $n$-step option-aware target. We will further clarify this point in the final version of the paper.\n\n\n### Question 3: Automatically selecting optimal $n$ and $k$\n\nWhile devising more automated strategies for selecting hyperparameters could be further explored, we have found that narrowing the search to a small, dataset‑ or environment‑specific range of plausible values works well in practice. For example, our OTA can perform effectively on short-horizon tasks such as AntMaze (≈ 1000 execution steps) with $(n,k) = (5,25)$, whereas it performs well on long-horizon tasks like HumanoidMaze (≈ 4000 execution steps) with $(n,k) = (20,100)$. This rough *linear scaling* trend of $(n,k)$ with respect to the execution steps implies that having prior knowledge about the specific execution steps of the dataset/environment can quickly narrow down the range of the relevant hyperparameters so that full exhaustive searches can be avoided."}, {"review_id": "U2N7QL5jlZ", "reviewer": "Reviewer_x6Va", "summary": "The paper investigates failure cases of HIQL (hierarchical implicit Q-learning) and proposes an improvement. The paper first provides a preliminary experiment that analyses failure cases of HIQL and concludes that these are probably caused by the high-level policy setting 'bad' goals, which in turn seems to come from imprecise V-value estimation. This problem is analyzed as being a problem mainly in long horizons. The proposed solution is to take a target value that instead of being based on 'regular' one-step bootstrapping, is based on effectively a n-step bootstrapping target. Experiments show that on various environments from OGBench and robot manipulation tasks, the proposed method outperforms non-hierarchical baselines as well as the original HIQL.", "questions": "Why are sign mismatches so crucial? To me it seems that going from 'just positive' to 'just negative' will change importance weight from slightly above 1 to slightly below one - might not be a big deal. Large errors that don't change the sign can make a bigger difference and more aversely affect the learned value function.", "limitations": "I think several limitations need to be more explicit: \n- bias caused by going from 1-step to multi-step returns where the expectile function will now encourage a optimistic estimate of the whole n-steps\n- assumption of deterministic dynamics.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Quality - Strenghts\n- The paper provides a clear and well-reasoned analysis of failure cases of HIQL\n- The empirical methodology over multiple environments is thorough and convincing. Results include error bounds over 8 random seeds. \n- Beyond raw performance, a visualization is provided to confirm the intuition of why this method might work better, and an ablation is performed over different values of the 'n' parameter. \n\nQuality - Weaknesses\n- Conceptually and theoretically, the new method could have been analyzed in more detail. In more detail: \n-- Limitations of the method are not mentioned. Some of these are inherited from IQL (sensitive to stochastic dynamics, which bias the expectile approximation of the 'max' operator over actions; recursive optimality), while some of them are increased by the method (the new target depends on (more) future states and actions, increasing the mentioned bias; the definition and analysis of \"order inconsistency\" seems incompatible with stochastic dynamics). To slightly expand on this point: the notion of \"optimal trajectory' is not well defined, as in general an optimal policy in a stochastic environment can induce a distribution of trajectories. Due to non-anticipated ('noisy') transitions, \"order consistency\" should not be expected. \n-- The paper strongly links the proposed approach to options, but the link to the original options in [46] is rather weak: in the original options paper, termination conditions are a Markov function (here: non-Markov), various options provide a high-level choice (here: only a single option), updates consider the aggretate discount gamma^n (here: n). In fact, I think the more useful analogy is to that of n-step returns.\n\nClarity - strengths\nThe paper is very clearly written in understandable language. Figures provide a clear illustration of the discussed ideas and contexts. \n\nClarity - weaknesses\nAs a rather minor weakness, I think the structure could be improved, as the introduction spends a lot of time detailing the proposed approach (better saved for the technical section) and is rather concise on the problem context, problem motivation, and solution need.\n \nSignificance\nThe empirical results are quite convincing, especially on the 'giant' versions of the mazes. On robot manipulation there is only a small (statistically likely non-significant) difference to HIQL, making it unclear how significant the result is beyond maps. \n\nOriginality\nThere is originality in the analysis of the problem, but the proposed solution has very limited originality, as this is essentially doing something very close to the well-known n-step returns. \n\nMinor comments: \nline 186: I'd rephrase this sentence. \"noisy\" is a bit ambiguous. I would typically interpret this as having a high variance, but the variance of the target doesn't depend on the horizon. Do you mean a bigger discrepancy between the target and the true value ('value error', if you will?).\nLine 194: Options are clasically not considered the same as macro-actions, as macro-actions are open-loop and options can be closed loop. Macro actions would also typically be fixed-lenths while options classically have a Markovian termination condition.", "was_revised": true, "final_justification": "The author's replies have partially addressed my concerns. I have thus updated my overall rating. A few questions and concerns remain as communicated in my reply to the authors' rebuttal.", "rebuttal": "We sincerely thank Reviewer x6Va for the positive comments on our analysis of the failure cases of HIQL and the strength of our experimental results. We also appreciate your constructive feedback regarding the limitations in stochastic dynamics and the relationship to n-step TD learning, etc. Below, we address your concerns in detail.\n### Weakness 1: Limitations inherited from IQL\nAs the reviewer has mentioned, our method and analyses were developed under the assumption of a **deterministic environment**. We should have stated this point more clearly, especially when introducing concepts such as \"order inconsistency\" and the notion of an \"optimal trajectory”, which only make sense in deterministic environments. We will revise the manuscript to clarify this assumption. While our method builds on IQL due to its simplicity and strong empirical performance, we do acknowledge that IQL has known limitations in stochastic environments.\n\nHowever, we note that this assumption is not unique to our work; several baseline methods [1,2,3] used in the offline GCRL benchmark also focus on deterministic environments, primarily due to the challenging nature of offline GCRL problems in more general stochastic environments. To that end, while our framework inherits certain limitations from IQL, we believe it still makes a significant contribution in deterministic environments by enabling objective comparisons with prior work under a consistent evaluation setting. Furthermore, we believe the core ideas of our approach (i.e., the option-aware target) could be extended to other offline RL algorithms that are more robust to stochastic environments [4,5], which we consider a promising direction for future work.\n\nAs the reviewer pointed out, expectile loss may lead to over-optimistic value estimates in stochastic dynamics, using the $n$-step forward states for updating the value _may_ exacerbate the bias. However, in deterministic dynamics, our experiments show that multi-step returns reduce bias and improve performance, balancing the bias-variance trade-off.\n### Weakness 2: Link to the original options\n- **Non-Markov termination condition**\n\nThank you for pointing out that our use of a fixed n-step timeout is a non-Markovian termination condition. We would like to clarify that the original options framework by Sutton et al. [6] is general enough to accommodate such cases through the use of semi-Markov options.\n\n- **Options as high-level choices**\n\nWhile options can represent high-level choices when learned explicitly, our setting differs: we leverage them implicitly from the dataset. The behavior policy $\\mu$ used during data collection consists of various high-level policies (e.g., grasping objects, moving items). Therefore, depending on which trajectories are sampled from the dataset, appropriate high-level choices are utilized, enabling temporal abstraction without explicit option learning.\n\n- **Aggregate discount $\\gamma^n$**\n\nSeveral works, including the original options paper [6], apply a discount factor of $\\gamma^n$ to rewards after $n$ steps. However, some studies [7,8,9] use option-level discounting when learning the value function. In our case, we adopt the option-level discounting scheme to enable temporal abstraction through an $n$-step target.\n### Weakness 3: Connection with $n$-step TD learning\nThe exact formula of TD target for $n$-step and OTA is as follows:\n- $n$-step target: $\\sum_{i=0}^{n-1} - \\gamma^i \\cdot 1(s_i \\neq g) + \\gamma^n V_\\theta (s_{t+n}, g)$\n- OTA target: $-1(s^{\\Omega}\\neq g) + \\gamma V_\\theta (s^{\\Omega}, g)$\n\nIn the above formula, since both targets mainly utilize n-step forward value, one _may_ think both formulas are quite similar. However, we want to clarify that extending the n-step target to the OTA target is not as straightforward. Rather, temporal abstraction through the option framework provides a natural explanation for the insights gained in our motivation experiments.\n\nThough the reward signal has a minor difference (i.e., the scale of rewards), the critical difference between $n$-step TD and OTA comes from the choice of $\\gamma$, which controls the information decay ratio as we perform the TD update. Since the standard $n$-step target typically uses the same $\\gamma$ as in 1-step target, the discount factor applied to the value function across the trajectory remains the same as in the 1-step TD. Or, equivalently, the discount factor applied to the value function in the n-step target decays exponentially with $n$ (e.g., $\\gamma^n \\approx 0.95$ for $(n,\\gamma)=(5,0.99)$ or $(n,\\gamma)=(10,0.995)$) while that in the OTA target is independent of $n$. \n\nTo demonstrate that the excessive decay of the discount factor in the standard $n$-step target hinders learning of the order-consistent value, we carried out additional experiments comparing the order consistency ratio ($r^c$) of value functions learned by 1-step TD, $n$-step TD, and OTA, all using the same discount factor $\\gamma$. We also used the same $n$ for $n$-step TD and OTA. In the table below, in contrast to OTA, we clearly observe that the $r^c$ for $n$-step TD has no improvement over 1-step TD since using the $n$-step TD target may still suffer from the estimation error of the value function due to the same level of discount factor.\n\n|env|$\\gamma$|$n$|1-step TD ($n$=1)|$n$-step TD|OTA|\n|-|-|-|-|-|-|\n|antmaze-large-explore|0.99|15|75±1|77±1|**97**±1|\n|antmaze-giant-stitch|0.99|10|91±1|84±2|**94**±1|\n|humanoidmaze-large-stitch|0.995|20|75±1|76±0.3|**89**±3|\n|humanoidmaze-giant-stitch|0.995|20|72±1|72±0.3|**94**±1|\n\nThe above results suggest that one may try to improve n-step TD by controlling $\\gamma$ appropriately for each $n$ so that the discount factor can be adjusted. However, it is clear that such an approach would introduce additional complexity in hyperparameter selection. In contrast, our OTA fixes $\\gamma$ regardless of $n$, which makes the scheme much simpler. \n\nIn the revised manuscript, we will clarify the connection between OTA and $n$-step TD learning. In particular, we will address the *clarity feedback* by summarizing some of the detailed explanations in the Introduction and adding a discussion of the similarity to n-step TD in Section 5.\n### Weakness 4: Small performance improvement in manipulation tasks\nWe analyze the order consistency ratio ($r^c$) of $V^h$ in HIQL on ``cube-single-play`` and ``cube-double-play``, both achieving a high ratio of 0.98. The reason for achieving high $r^c$ in HIQL is the horizon of both tasks is relatively small (less than 500) compared to complex maze tasks. We, therefore, conclude that in the case of short-horizon manipulation tasks, the core problem lies in the policy extraction scheme for high-level policy (i.e., AWR for $\\pi^h$), not in the value estimation. However, we believe that for more complex and long-horizon robot manipulation tasks in which 1-step TD cannot learn order-consistent value, OTA can improve the performance of HIQL.\n\n### Minor comment: Clarification of \"noisy\" and \"macro-action\"\n\nRegarding line 186, we agree that the term \"noisy\" can be ambiguous. Here, it refers to high estimation error between predicted and true values. As described in HIQL (see Figure 8 in HIQL), the variance of the value function increases with longer horizons, destabilizing value learning and resulting in larger value errors, as illustrated in our Figure 2. We will clarify this explanation in the related work section and revise the wording.\n\nFor line 194, we appreciate the distinction you pointed out between options and macro-actions. Since we define options along with the termination conditions from the offline dataset, they do not necessarily have fixed lengths. Therefore, the term \"options\" is more appropriate in our setting, and we will update the manuscript accordingly.\n \n### Question 1: Importance of sign matching\n\nFor high-level policy learning, the advantage is computed by $V^{h}(s_{t+k}, g) - V^{h}(s_t,g)$. With large $k$ values (e.g., 25 in ``antmaze``, 100 in ``humanoidmaze``), this results in large-magnitude advantages. If the sign of the advantage is incorrect, the magnitude of regression weights (which is the exponentiated advantage) is drastically increased or decreased, causing unintended improper subgoal regression for high-level policy. For example, an advantage range of $[-1, 1]$ with $\\beta^h=3$, the regression weights vary from $e^{-3}\\approx 0.05$ to $e^{3}\\approx 20$, which is significantly far from 'just negative' or 'just positive'. In our experiment, the magnitude of the advantage often exceeds 10, and we also use $\\beta^h>1$, making the impact of sign mismatch even more critical.\n\nWhile small-magnitude advantages may be less affected by sign errors, a low advantage magnitude implies that the objective becomes similar to behavior cloning, which does not give any useful information on learning proper subgoals. For this reason, we first scale $\\beta^h$ to ensure sufficient advantage magnitude, and then apply AWR.\n\nReference\n\n[1] Ghosh et al., “Learning to reach goals via iterated supervised learning”, ICLR 2021\n\n[2] Wang et al., “Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning”, ICML 2023\n\n[3] Park et al., “HIQL: Offline Goal-Conditioned RL with Latent States as Actions”, NeurIPS 2023\n\n[4] Villaflor et al., “Addressing optimism bias in sequence modeling for reinforcement learning”, ICML 2022\n\n[5] Liu et al., “A Tractable Inference Perspective of Offline RL”, NeurIPS 2024\n\n[6] Sutton et al., \"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.\", Artificial intelligence 1999\n\n[7] Klissarov et al., \"Flexible Option Learning\", NeurIPS 2021\n\n[8] Nachum et al., \"Data-Efficient Hierarchical Reinforcement Learning\", NeurIPS 2018\n\n[9] Chai et al., \"MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions\", ICLR 2025"}, {"review_id": "60VTJl6e4p", "reviewer": "Reviewer_qyqg", "summary": "The authors propose a new offline goal-conditioned RL (GCRL) algorithm called OTA. They begin by noting that the main bottleneck of HIQL (one of the state-of-the-art offline GCRL methods) is in the high-level policy. To address this, they propose training a separate high-level value function with temporally extended options (which are defined by subgoals), and extracting a high-level policy based on this high-level value function. They show that the resulting algorithm (OTA) substantially improves performance on many long-horizon navigation tasks in OGBench.", "questions": "I don't have additional questions other than the ones I asked in the weaknesses section.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "[Strengths]\n* The experiments in the motivation section (Section 4) are well designed. The authors convincingly argue that high-level policy learning is the bottleneck of HIQL, and OTA can resolve this issue effectively (Figure 5).\n* The proposed method is simple and effective.\n* The authors provide several analysis experiments (Sections 6.3-6.6), which are informative and helpful to understand the effect of the proposed method.\n* OTA achieves substantially better performance than the previous methods on several challenging tasks (e.g., `humanoidmaze-giant` and `antmaze-*-explore`).\n* The paper reads well in general.\n\n[Weaknesses]\n\nI don't see significant weaknesses of this work. Though I do have some concerns, mostly about empirical evaluation.\n* From Table 4, OTA seems to require careful tuning of four hyperparameters ($\\beta^h$, $\\beta^\\ell$, $k$, and $n$) for each setting. This potentially makes the empirical comparison unfair, given that the baselines use the same policy extraction hyperparameters for each dataset type (see Table 11 of the OGBench paper).\n* The authors only selectively report performance on manipulation tasks. For example, how does OTA perform on the standard `play` datasets?\n* OTA implicitly assumes that length-n trajectory chunks in the dataset are optimal (which is implicitly used in Eq. (4)). This is not always the case, especially when the dataset is highly suboptimal. The paper lacks a discussion about this point.\n\n[Minor issues]\n* L54: \"by updating the value over a sequence of primitive actions\" -- This sounds like the method does action chunking, which (as far as I understand) seems not to be the case.", "was_revised": true, "final_justification": "My initial (mostly minor) concerns have been resolved, and I'd like to maintain the original score of 5 (accept).", "rebuttal": "We sincerely thank Reviewer qyqg for the positive comments on the motivation of our paper and the analysis experiments. We also appreciate your constructive feedback regarding the empirical evaluation, including hyperparameters, manipulation tasks, etc. Below, we address your concerns in detail.\n\n### Weaknesses 1: Hyperparameter tuning\n\nThank you for the comments on tuning the hyperparameters of OTA. We agree that we searched the hyperparameters extensively to report our results. To show that OTA also outperforms HIQL with the same hyperparameters (e.g., $k, \\beta^h, \\beta^{\\ell}$) as HIQL, we also carry out additional experiments in complex maze tasks (i.e., large and giant mazes except for explore dataset). The table below shows the results with the same hyperparameters to HIQL except for $n$. In this table, the success rate of OTA is still much higher than HIQL, meaning that simply applying the temporal abstraction can improve the performance.\n\n\n| Environment     | Type     | Size     | $n$   | $k$   | $\\beta^h$  | $\\beta^{\\ell}$  | OTA | HIQL |\n|-----------------|----------|----------|-----|-----|-----|-----|-----------------|------------|\n| PointMaze       | navigate | large    | 5   | 25  | 3.0 | 3.0 | **85** ± 5 | 58 ± 5  | \n|                 |          | giant    | 5   | 25  | 3.0 | 3.0 | **72** ± 6 | 46 ± 9  |\n|                 | stitch   | large    | 5   | 25  | 3.0 | 3.0 | **46** ± 7 | 13 ± 6  |\n|                 |          | giant    | 5   | 25  | 3.0 | 3.0 | **44** ± 8 | 0 ± 0   |\n| AntMaze         | navigate | large    | 5   | 25  | 3.0 | 3.0 | **91** ± 1 | **91** ± 2  |\n|                 |          | giant    | 5   | 25  | 3.0 | 3.0 | **70** ± 2 | 65 ± 5  |\n|                 | stitch   | large    | 5   | 25  | 3.0 | 3.0 | **79** ± 3 | 67 ± 5  |\n|                 |          | giant    | 5   | 25  | 3.0 | 3.0 | **29** ± 5 | 2 ± 2   |\n|                 | explore  | medium   | 5   | 25  | 3.0 | 3.0 | **93** ± 3 | 37 ± 10 |\n|                 |          | large    | 10  | 25  | 3.0 | 3.0 | **62** ± 12 | 4 ± 5   |\n| HumanoidMaze    | navigate | large    | 20  | 100 | 3.0 | 3.0 | **82** ± 2 | 49 ± 4 |\n|                 |          | giant    | 20  | 100 | 3.0 | 3.0 | **91** ± 1 | 12 ± 4   |\n|                 | stitch   | large    | 20  | 100 | 3.0 | 3.0 | **43** ± 3 | 28 ± 3 |\n|                 |          | giant    | 20  | 100 | 3.0 | 3.0 | **61** ± 3 | 3 ± 2   |\n\n\n### Weaknesses 2: Performance on play datasets\n\nTo demonstrate the effectiveness of temporal abstraction between action sequences in the presence of noisy data, we conducted experiments using a noisy dataset. Additionally, we evaluated the performance of OTA on play datasets. The results are summarized in the table below. For a fair comparison, we fixed the hyperparameters $(k, \\beta^h, \\beta^{\\ell})$ for both OTA and HIQL and varied only $n=2$.\n\n| Environment     | Data      |  Env |  OTA | HIQL |\n|-----------------|-----------|------|------|------|\n| Visual-cube     | play      | double | **51** ± 3| 48 ± 4|\n|        |     | triple  |**28** ± 1|21 ± 5|\n| Visual-scene    |    play | -  | **56** ± 5  | 50 ± 5 |\n\n\n\n### Weaknesses 3: Assumptions on the data optimality\n\n\nFirst, we want to clarify that the update formula in Equation (4) does not assume data optimality. As in the definition of the option we used, we assume that the underlying policy is the behavior policy, which is used to collect the offline data, and the behavior policy is extremely suboptimal. Therefore, the option-aware value update in Equation (4) can be performed regardless of the data characteristics. In fact, we observed significant performance improvements on the extremely suboptimal datasets such as ``antmaze-medium-explore`` and ``antmaze-large-explore``.\n\n\n### Minor issue: Option description\n\nThank you for your comments on the notion of options we used. As the reviewer pointed out, we do not use action chunking, but the above explanation in the manuscript may induce misunderstanding. We will revise the explanation in the camera-ready version."}, {"review_id": "AzEgnxc9GL", "reviewer": "Reviewer_qrJ2", "summary": "This paper investigates the performance bottleneck of the prior method HIQL and concludes that inaccurate value estimation causes the high-level policy to suggest suboptimal subgoals. Further analysis reveals that this inaccuracy arises from the large temporal distance between the current state and the goal. To address this, the authors propose a novel, simple, and effective method called OTA, which estimates the value function over a sequence of actions, effectively transforming distant goals into nearer ones. Experiments show that this technique improves order consistency, meaning that states can be ranked by the value function in the same order as their temporal sequence, and outperforms the existing methods.", "questions": "1. In the abstract, the authors state that offline GCRL trains goal-reaching policies from unlabeled (reward-free) datasets. However, the datasets used in offline GCRL include binary rewards / reward function, that indicates whether the goal has been reached. Therefore, describing them as \"reward-free\" may be inaccurate and should be clarified.\n\n2. The term \"policy extraction\" is used multiple times (e.g., line 29 and line 149) without a prior explanation. It would improve clarity if the authors could briefly define what they mean by policy extraction in this context.\n\n3. The reward function defined in lines 121 and 207 as $r(s, g) = -1(s \\neq g)$ assumes that exact equality $s = g$ is achievable. This assumption is problematic in continuous state spaces, where reaching the exact goal is often impossible. A more realistic formulation would be $r(s, g) = -1(\\|s - g\\|^2 > \\epsilon)$, for some threshold $\\epsilon > 0$.\n\n4. In Figure 2, it appears that when the state $s$ is far from the goal $g$, the high-level value function $V^h$ tends to yield over-optimistic estimates. This behavior is noteworthy, and it would strengthen the paper if the authors could provide an explanation or hypothesis for why this phenomenon occurs.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. Section 4 is compelling and well-argued, offering a strong and intuitive motivation for the proposed method.\n\n2. The authors introduce a new concept, \"order consistency\", which is both interesting and potentially impactful for future research in goal-conditioned reinforcement learning.\n\n3. The experimental results are thorough and convincingly demonstrate the effectiveness of the proposed approach.\n\n**Weakness**\n\nThe paper does not provide sufficient theoretical justification for the OTA value function $V^h_{\\text{OTA}}$ as optimised in Eq. (4). Please try to address the following questions:\n\n(1) Can the value function be interpreted as the expected sum of rewards? If not, how to define this value function?\n\n(2) Does this formulation satisfy the Bellman recursion? It is the ground for optimisation following Eq. (4). \n\nI would like to increase my rating to Accept once the authors address the two questions.", "was_revised": true, "final_justification": "The authors have addressed my concerns and we had a nice discussion.", "rebuttal": "We sincerely thank Reviewer qrJ2 for the positive comments on the motivation of the proposed method and “order consistency” evaluation metric. We also appreciate your constructive feedback regarding theoretical justifications, terminology, reward assumptions, etc. Below, we address your concerns in detail.\n\n### Weakness 1: Theoretical justification for the OTA value\n\nWe thank the reviewer for these important questions.\n(1) The value function in our formulation can be interpreted as the expected sum of discounted rewards—i.e., the return—which aligns with the standard definition in reinforcement learning. \n(2)  Our approach builds on the options framework, where it has been thoroughly studied that the value function converges to the optimal value under Bellman recursion. For a more detailed theoretical proof, we kindly refer the reviewer to foundational work [1].\n\n### Question 1: Clarification of “reward-free dataset”\n\nWe agree that the term *\"reward-free\"* may be misleading without clarification. In our setting, the offline dataset used for GCRL training consists of trajectories in the form of $(s_0, a_0, s_1, a_1, \\dots)$, and does not include explicit reward signals. However, during training, we apply a sparse binary reward function using hindsight experience replay (HER) [2]. Thus, while the original dataset itself is unlabeled in terms of rewards, rewards are introduced during training via relabeling. To avoid confusion, we will replace the term \"reward-free dataset\" with \"reward-unlabeled dataset”.\n\n### Question 2: Clarification of “policy extraction”\n\nWe use the term *“policy extraction”* to refer to the process of deriving a policy from a learned value function, a usage consistent with prior works such as IQL [3], OGBench [4], and JaxGCRL [5]. This term emphasizes the decoupling between value learning and policy learning, particularly in algorithms like AWR, which IQL builds upon.\nSpecifically, we will make the following revisions:\n- Line 29 will be removed for clarity and conciseness.\n- In earlier sections (e.g., lines 49, 52, 60, ...), where we refer to general policy learning, we will consistently use *policy learning* instead of *policy extraction*.\n- Beginning with the related work section (e.g., line 123), where AWR is discussed explicitly, we will adopt the term *policy extraction* and provide a brief definition for clarity.\n        \n### Question 3: Reward function formulation\n\nWe agree that in continuous state spaces, a more appropriate reward formulation would be  $r(s,g)=-\\mathbf{1}(|s-g|^2>\\epsilon)$. However, in practice, this formulation introduces certain challenges: (1) it introduces an additional hyperparameter $\\epsilon$, (2) it increases the computational overhead in HER-style goal relabeling, especially for high-dimensional states. For these reasons, we chose to use a sparse binary reward of $-\\mathbf{1}(s\\neq g)$ in our experiments following other GCRL benchmarks [4,5]. We will clarify this design choice in the experimental section of the revised manuscript.\n    \n### Question 4: Over-optimistic value estimates in Figure 2\n\nThank you for your insightful observation regarding over-optimistic estimates of the high-level value function when $s$ is far from the goal $g$. We hypothesize that this behavior arises from the value learning mechanism in IQL, which employs expectile regression to estimate the upper expectile of TD targets. As the distance between $s$ and $g$ increases, the variance (noise) of the TD targets also tends to increase (Figure 8 in HIQL [6]). Consequently, the upper expectile can become significantly biased, leading to overestimated values for distant goals. That is, the estimation error tends to accumulate over longer horizons in IQL, resulting in increasingly over-optimistic value estimates. In contrast, our OTA value learning approach effectively mitigates this issue, even as the horizon increases. We will incorporate this explanation in the revised manuscript and supplement it with additional visualizations comparing OTA with the ground-truth value function to support the claim.\n\nReference\n\n[1] Sutton et al., \"Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.\" Artificial intelligence 1999\n\n[2] Andrychowicz et al., “Hindsight Experience Replay”, NeurIPS 2017\n\n[3] Kostrikov et al., “Offline Reinforcement Learning with Implicit Q-Learning”, ICLR 2022\n\n[4] Park et al., “Ogbench: Benchmarking Offline Goal-Conditioned RL”, ICLR 2025\n\n[5] Bortkiewicz et al., “Accelerating Goal-Conditioned RL Algorithms and Research”, ICLR 2025\n\n[6] Park et al., “HIQL: Offline Goal-Conditioned RL with Latent States as Actions”, NeurIPS 2023"}]

Author rebuttal:

Meta-review: The paper provides a clear analysis of failure cases of HIQL. It then provides a simple, well-grounded and effective technique that outperforms existing baselines on long-horizon tasks. The experiments are quite thorough and also provide several interesting insights, such as about the accuracy of the estimated value function. All reviewers have scores of at least 4 (minor accepts). The few minor weaknesses that were pointed out relate to some remaining limitations and some additional research questions that are still open. However, all reviewers agree that the contribution is significant.
Decision: Accept (spotlight)