Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Authors: Seungho Baek
taegeon park
Jongchan Park
Seungjun Oh
Yusung Kim
Venue: ICML 2025 poster (2025)
Keywords: Offline Hierarchical Reinforcement Learning
Offline Goal-Conditioned Reinforcement Learning
Graph-based Reinforcement Learning
Temporal Distance Representation Learning
Primary area: reinforcement_learning->batchoffline
TL;DR: We propose a graph-based offline HRL framework that significantly improves long-horizon reasoning and trajectory stitching.

Abstract: Existing offline hierarchical reinforcement learning methods rely on high-level policy learning to generate subgoal sequences. However, their efficiency degrades as task horizons increase, and they lack effective strategies for stitching useful state transitions across different trajectories. We propose Graph-Assisted Stitching (GAS), a novel framework that formulates subgoal selection as a graph search problem rather than learning an explicit high-level policy. By embedding states into a Temporal Distance Representation (TDR) space, GAS clusters semantically similar states from different trajectories into unified graph nodes, enabling efficient transition stitching. A shortest-path algorithm is then applied to select subgoal sequences within the graph, while a low-level policy learns to reach the subgoals. To improve graph quality, we introduce the Temporal Efficiency (TE) metric, which filters out noisy or inefficient transition states, significantly enhancing task performance. GAS outperforms prior offline HRL methods across locomotion, navigation, and manipulation tasks. Notably, in the most stitching-critical task, it achieves a score of 88.3, dramatically surpassing the previous state-of-the-art score of 1.0. Our source code is available at: https://github.com/qortmdgh4141/GAS.

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Seungho Baek Taegeon Park?
Jongchan Park 2 Seungjun Oh Yusung Kim 12
Abstract
tions where online data collection is costly, time-consuming, unsafe (c.g., robotics. healtheare. and autonomous driving) (Gulcehre et al., 2020; Jeong et al., 2024). By lever- aging offline datasets, agents can learn optimal behaviors while avoiding risks associated with real-world deployment. Building on this paradigm, offline goal-conditioned rein- forcement learning (GCRL) provides general framework for learning multi-task policy from diverse data. Neverthe- less, long-horizon and sparse-reward tasks remain a funda- mental challenge in offline GCRL (Ajay et al., 2021; ;; Li et al., 2024b). In such tasks, rewards are often provided only at the final state, making it difficult to establish a clear learning signal for intermediate decision-makine Without explicit guidance on how to decompose long-horizon goals into man- ageable subgoals, standard offline GCRL algorithms strug- gle to generalize and effectively utilize the available data, This limitation has motivated research into offline hierarchi- cal reinforcement learning (HRI.). HRL is well-established approach for tackling long-horizon and sparse-reward prob- lems by introducing a two-level decision detision-making frame- work (Vezhnevets et al., 2017: Nachum et al., 2018; 8;; Levy et al., 2019): A high-level policy generates subgoals that guide long-term planning. A low-level policy learns how achieve these subgoals using primitive actions. By breaking down complex tasks into lems, HRL can significantly improve learning efficiency.
Existing offline hierarchical reinforcement learn- ing methods rely on high-level policy learning to generate subgoal sequences. However, their ef- ficiency degrades as task horizons increase, and they lack effective strategies for stitching useful state transitions across different trajectories. We propose Graph- Assisted Stitching (GAS), novel framework that formulates subgoal selection as a graph search problem rather than learning an ex- plicit high-level policy. By embedding states into Temporal Distance Representation (TDR) space, GAS clusters semantically similar states from dif- ferent trajectories into unified graph nodes, en- abling efficient transition stitching. A shortest- path algorithm is then applied to select subgoal sequences within the graph, while a low-level policy learns to reach the subgoals. To improve graph quality, we introduce the Temporal Effi ciency (TE) metric, which filters out noisy in- efficient transition states, significantly enhancing task performance. GAS outperforms prior offline HRL methods across locomotion, navigation, and manipulation tasks. Notably, in the most stitching- critical task, achieves a score of 88.3, dramati- cally surpassing the previous state-of-the-ar score of 1.0. Our source code is available https: / //github... com/ http///tmmmmh1///......
learning prob-
Despite its potential, existing offline HRL methods exhibit major limitations primarily in high-level policy learning, leading to critical challenges: (1) Lack of Temporal Aware- ness in Subgoal Sampling: Most approaches sample sub- goal candidates by choosing states at fixed time intervals along a given trajectory (Park et al., 2023; Shin & Kim, 2023). However, this naive subgoal sampling strategy fails to capture the temporal dependencies between states, lead- ing to redundant or inefficient subgoals that hinder high- level policy learning. In offline datasets, where suboptimal behaviors and noisy actions are common, such heuristics may misinterpret temporal transitions, resulting in unstable high-level planning. (2) Challenges in Long- Horizon Rea- soning: As reported in recent studies, prior works achieve strong performance in medium-scale environments such as antmaze- but fail to scale to larger environments like antmaze- giant, where the task horizon is significantly longer (Park et al., 2025a; Sobal al., 2025), As task hori-
1. Introduction
Offline reinforcement learning (RL) has emerged promising approach to learning effective policies from pre- collected datasets without requiring active interaction with the environment (Lange et al., 2012; Kumar et al., 2019). This paradigm particularly valuable in real-world applica-
'Department of Computer Science and Engineering, Sungkyunkwan University, Suwon, Republic of Korea 'Department of Artificial Intelligence, Sungkyunkwan Uni- versity, Suwon, Republic of Korca. Correspondence to: Yusung Kim <yskim525 @ skku.edu>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Pre- Pre-iiinn Temporal Distance Representation
TD-aware Graph Construction
II
(s)
Dataset
(1) Temporal Effiiency F ittring
(2) TD aware Clustering
(3) Connectng Noces
3. Learning Low-level Policy
4. Task Planning and Execution
X
II
li
Dataset
8, a, , Pairr
8, hai
(s, a, 8', 8zub)
Tester
Figure 1: Overview of GAS. (1) A Temporal Distance Representation (TDR) is pretrained from an offline dataset. (2) A graph is constructed by selecting only high-TE states based on the Temporal Efficiency (TE) metric. (3) A TDDbbasdd subgoal-conditionel low-level policy is trained using all states. (4) The graph is utilized for task planning and subgoal selection, while action execution is performed by the low-level policy.
zons grow longer, sparse rewards cause the learning signal weaken, increasing noise and reducing the effectiveness of subgoal generation by high-level policy learning. (3) In-
The key contributions of this work are as follows:
We propose novel framework for offline HRL that lever- ages graph-based approach for subgoal selection to able efficient long-horizon reasoning and state transition stitching.
effective Trajectory Stitching: Offline datasets often con- sist of diverse trajectories collected attempting to achieve various goals (Park et al., 2025a;b). Effective offline HRI. should be able to stitch together meaningful state transitions across different trajectories. However, existing methods lack efforts in cross-goal stitching, where partial segments from different goal-oriented trajectories are reused form trajectory compositions that facilitate goal generalization.
We introduce TD-aware graph construction that groups semantically similar states while stitching meaningful state transitions across trajectories.
We introduce the TE metric, which filters out inefficient transition states to enhance graph quality and reduce con- struction overhead.
To address these limitations, we propose Graph-Assisted Stitching (GAS), novel framework that adopts a graph- based subgoal selection approach without the need for ex- plicit high-level policy learning for subgoal generation. GAS leverages pretrained Temporal Distance Representation (TDR) to estimate the minimum number of time steps quired for optimal policy to transition between states. States in the dataset are clustered at regular temporal dis- lance intervals, and the centroids of each cluster are treated as graph nodes. Edges are created between nodes if their temporal distance falls below predefined threshold. By con- structing this graph, each node groups semantically similar states, while edges enable stitching between states that were not directly connected in the dataset, significantly improving the effectiveness of transition stitching. Additionally, we in- troduce the Temporal Efficiency (TE) metric to enhance the quality of the graph. TE evaluates how close a state's tran- sition to the optimal temporal behavior. Only states with high TE values are selected for graph construction, reduc- ing computational overhead while preserving high-quality subgoal selection.
We evaluate GAS across diverse numeric and visual state environments, consistently outperforming baselines and achieving remarkable improvements soning and trajectory stitching tasks.
rea-
2. Related Work
2.1. Offline Goal-Conditioned RL
Offline goal-conditioned reinforcement learning (GCRL) aims to learn a multi-task policy from diverse data with- out further environment interaction (Ma et al., 2022; Yang et al., 2023; Cao al., 2024; Sikchi al., 2024: Wang et al., 2024), Previous methods have proposed various approaches for training goal-conditioned value functions. These include hindsight relabeling, which relabels achieved states goals for goal-conditioned learning (Andrychowicz et al., 2017); expectile regression for optimal value estimation (Kostrikov et al., 2022; Park et al., 2023); quasi-metric learning with dual objective (Wang et al., 2023); and contrastive repre-
2

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
sentation learning estimate returns and perform one-step policy improvement (Eysenbach et al., 2022; Zheng al., 2024). However, these approaches often struggle with long- horizon and sparse-rewand tasks, as rewards are typically provided only at the final state, making it difficult to estab- lish clear learning signal for intermediate secision-making.
2.4. Graph Construction in Online RL
Graph-based reinforcement learning (RL) constructs ex- plicit graphs, where nodes represent goal states and edges are formed based on reachability. Prior works leverage such graphs to address the hierarchy tradeoff (Lee et al., 2022) or to replace high-level policy learning with structured plan- ning (Eysenbach et al., 2019; Hoang et al., 2021). For graph construction, existing approaches commonly sample candi- date goals from the replay buffer and apply Farthest Point Sampling (FPS) (Eldar et al., 1994) select representa- tive nodes (Kim et al., 2021; Lee et al., 2022; Kim et al., 2023; Park et al., 2024a), while others apply a coarse-to- fine grid refinement strategy over manually defined goal spaces (Yoon et al., 2024). However, these methods are primarily designed for online RL to efficiently expand the graph, often measuring reachability through repeated online rollouts (Faust et al., 2018; Bagaria et al., 2020; Yoon et al., 2024), which not available in offline settings. FPS, widely used technique for node selection, is highly sensitive to data distribution and often produces suboptimal graph structures. In addition, many previous methods rely on man- ually specified low-dimensional goal spaces, whereas our method constructs a graph in latent space, enabling appli- cation to high-dimensional and visual state environments.
2.2. Offline Hierarchical RL
Offline hierarchical reinforcement learning (HRL) intro- duces a two-level decision massiinn framework, where a high-level policy generates subgoals and a low-level policy executes primitive actions to reach them. This structure en- ables temporal abstraction and decomposes complex tasks into more tractable sub-problems (Haarnoja et al., 201 8a; Nachum et al., 2019; Zhang et al., 2020). Prior methods have proposed various techniques, including skill discovery meth- ods to extract primitive skills for downstream learning (Ajay et al., 2021; Li et al., 2024b), and latent variable method for modeling the distribution of reachable subgoals (Shin & Kim, 2023). To extend offline HRL nigh-dimennional or visual state environments, recent methods adopt subgoal representation learning in a latent space. HIQI. (Park et al., 2023) propose a method where the high-level policy out- puts latent subgoal representations learned from the value function. Similarly, HHILP (Park et al., 2024c) produce sub- goals in a Temporal Distance Representation (TDR) space, enhancing the generalization ability of the policy. Despite these advances, existing methods often struggle with high- level policy learning due to lack of temporal awareness in subgoal sampling, weak learning signals in long-horizon tasks, and limited efforts in cross-goal stitching.
3. Preliminaries
3.1. Goal-Conditioned MDP
We consider continuous control tasks framed as a goal- conditioned Markov Decision Process (GC-MDP) (Kael- bling, 1993). Formally, each task is specified by the tuple (S, G. A.,,,, p.r, 7). where S represents the state space, and we assume the goal space G is identical to the state space (i.e., 9 S). The set A describes the action space, while p denotes the transition probability function p(s' [88 s).a).
2.3. Trajectory Stitching in Offline RL
Stitching refers to the process of combining partial segments from different trajectories to form new trajectories that were not explicitly present in the original dataset (Fu et al., 2020; Park et al., 2025a). Previous offline reinforcement learning (RL) algorithms have primarily relied on implicit stitching through Bellman backup mechanisms (Fujimoto et al., 2019; Wu et al., 2019; Kumar et al., 2020; Levine et al., 2020). More recent approaches aim to improve an agent's ss stitching ability by applying temporal goal augmentation techniques to supervised lcarning-basee based RL (Ghugare et al.. 2024), or by leveraging diffusion models to combine high-value trajectory segments from multiple trajectories (Li et al., 2024a; Kim et al., 2024). However, these approaches are typically tailored to single-task agents or -ow-dimensional goal spaces. As a result, they face structural limitations when extended to high-dimensional or visual state environments in multi-task settings. In contrast, our method performs explicit graph-based stitching and trains a multi-task agent by leveraging Temporal Distance Representations (TDR), which can scale to digh-dimensional state environments.
Let rr denote the reward at time step f, and y be the discount factor. The objective is to learn a policy π that maximizes the expected return:
T J ( (T) = Ex
Σ
7/r re
(I)
In goal-conditioned environments, especially those with sparse rewards, the reward function often takes the following form:
r(st, ar 9411.9) { if [[(((144)) -(0e)))))(((l1l <c (2) otherwise
where 2(-) denotes a projection function that extracts the goal-relevant components from the state or goal state. A reward of is provided when the Euclidean distance be- tween (((++ and 2(g) is below the threshold e, indicating proximity to the goal. Otherwise, the reward is set to
3

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
3.2. Hierarchical GC-MDPs
Specifically, define:
This section extends the discussion from goal-conditioned Markov Decision Process (GC-MDP) to hierarchical set- ting (Kaclbling, 1993; Sutton et al., 1999; Barto & Mahade- van, 2003). Specifically,, existing hierarchical approaches decompose the decision-making process into two levels: a high-level policy x², which is responsible for generating subgoals that guide the agent toward the final goal, and a low-level policy n², which aims to reach these subgoals rather than the final goal directly. To formally describe the hierarchical structure, one can consider two coupled MDPs:
V(8,g) = - (lb((s)) - +1ll2
(4)
By interpreting -Ilv(s) - (1))) as goal-conditioned value, optimize V via temporal difference objec- tive on offline data 'D with the expectile loss & (Newey & Powell, 1987). Accordingly, we define the TDR loss C as:
Be,, D 1 -1 1( ((((-11
+ g) +7 Vs -V
(5)
(x) = 1 1(x < <))=²²
(6)
High-Level MDP M = Gunat AA, pp, r", 7): The ac- tion space A corresponds to selecting subgoals Jsub E 9aah- The high-level reward function r" provides feedback on how effectively the chosen subgoal promotes progress toward the final goal 9inal E Ginal-
where V is a target value function, 7 is the discount factor, T the expectile coefficient, and 11ss # g} indicates that differs from the goal g. Through this procedure, the latent space H preserves the optimal temporal distance in S, thus maintaining glohal long-horizon relationships among states.
r)): The action space A' is identical to the original action space A, which consists of primitive actions a E A. The low-level policy x' (a 8, Isub) is trained to maximize r', which measures how effectively the subgoal Ssub is reached.
Low-Level MDP MM = 15,Guub: A', A',p','''
4. Graph-Assisted Stitching (GAS)
This study proposes the GAS framework to overcome the limitations of offline Hierarchical Reinforcement Learning (HRL). GAS eliminates the need for high-level policy learn- ing and instead adopts graph-based approach to enable efficient long-horizon reasoning and state transition stitch- ing. First, it constructs a graph using a clustering method that maintains consistent intervals in the Temporal Distance Representation (TDR) space, enabling stitching opportuni- ties across diverse trajectories through connections between nodes. Then, it utilizes shortest-path algorithm to sequen- tially select subgoals, while the low-level policy learns prim- itive actions to reach these subgoals. Finally, to address is- sues with low-quality graphs caused by suboptimal datasets, a Temporal Efficiency (TE) metric is introduced to not only enhance graph quality but also reduce construction overhead by filtering out ineffective states. (Figure 1)
Sampling transitions for training hierarchical policies is eru- cial. Typically, a transition for the high-level policy can be represented as (st. Stte: $g), where 8, is the current state, Sitc is the state observed after steps, and &g is the final goal ghnal sampled from a future state within the same trajec- tory. The high-level policy is trained to generate Si+c as the subgoal Jsub- For the low-level policy, a common transition tuple is (St, a, S1+1, St+c), where a is the action taken by the policy, 1 is the next state, and St++ is the subgoal Goub the low-level policy must achieve. In many existing meth- ods (Ajay et al., 2021; Park et al., 2023; Shin & Kim, 2023; Li et al., 2024b; Park et al., 2024c), the parameter c, which defines the subgoal horizon, is treated as a hyperparameter.
4.1. TD-Aware Graph Construction
3.3. Temporal Distance Representation Learning
To construct the graph, all states in the dataset are clustered in the Temporal Distance Representation (TDR) space at intervals of the target temporal distance HTD- Initially, the first state of the dataset is assigned as the center of the first cluster. For each subsequent state, we compute its temporal distance to the existing cluster centers and assign it to the nearest cluster if the distance is within HID- Otherwise, a new cluster is created with the state as its center. Once all states are clustered, the cluster centers are updated to the mean values of the states within each cluster. This approach effectively covers all states in the dataset while maintaining consistent spacing between clusters. The cluster centers are used as the nodes of the graph, and edges are added between nodes within HTD- The resulting graph facilitates natural stitching opportunities between states from different trajectories in the dataset.
Recent work proposed a method to learn a Temporal Dis- tance Representation (TDR) vp from offline datasets for re- inforcement learning (RL) (Park et al., 2024c). The central insight is to embed states into a latent space H. In this space, the Euclidean distance between any two points corresponds to the minimum number of steps required to transition from one state to another in the original state space S. (Figure 2)
Formally, we aim for:
d' ds -10)=1((e - -
(3)
where d' g) denotes the optimal temporal distance from s to g. To learn e : S + H. one may view this as a goal-conditioned value function problem (Kaelbling, 1993) and employ an existing offline RL algorithm such as IQL (Kostrikov et al., 2022; Park et al., 2023).
4

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
S
11
0
Temporal Efficiency
High Temporal Efficiency
Seached
Suptrall
A
Direction Vector
Trajectory
Space
(b) Space
Figure 2: Temporal Distance Representation (TDR). (left) Raw state space, where spatial proximity does not imply temporal closeness. (right) TDR space, where Euclidean distances correspond to the minimum number of steps re- quired to reach one state from another. This representation captures and preserves global temporal structure.
Figure 3: Temporal Efficiency (TE). (left) A transition with low TE, where the reached state diverges from the optimal direction. (right) A transition with high TE, where the reached state aligns well with the optimal direction. TE measures the directional alignment between the actual and optimal transition vectors over fixed temporal distance.
4.2. Temporal Efficiency Metric
4.3. Low-Level Agent Training
Using all states in the dataset for graph construction may not always be effective. In particular, when suboptimal trajecto- ries are included, low-quality states can negatively impact stitching ability or final task performance. To address this, we introduce the Temporal Efficiency (TE) metric to selec- tively choose states for graph construction. TE measures how efficiently state facilitates transitions by evaluating the similarity between states reached within the same trajec- tory over fixed temporal distance. (Figure 3)
We construct TD-aware dataset from an unlabeled (reward- free) dataset by applying pretrained Temporal Distance Representation (TDR). We discuss two key components of low-level agent training: value function learning (Sec- tion 4.3.1) and subgoal-conditiones policy learning (Sec- tion 4.3.2).
4.3.1. VALUE FUNCTION LEARNING
We utilize the TD-aware intrinsic reward proposed in prior work for value function learning (Park al., 2024c). The reward function is defined as:
Precisely, given a state Scur. we define the optimal future state Sapt as the state after temporal distance of HTD in the trajectory, and the actual reached state Sreached as the state observed HID steps ahead:
d , 8' Jaii)) (w('') (har)
(10)
F(8t, d) Smin ((xx)))((0))l2224
(7)
Given this reward, we optimize IQL. (Kostrikov et al., 2022; Park al., 2023) based critic objectives conditioned on har-
Supt = F(Scur: HTD), Srenched Sco
(8)
If the trajectory was collected with optimal behavior, v (Frenched) and (Sopp) are expected to be similar, whereas suboptimal trajectories often lead to larger discrepancies. We define the TE the cosine similarity between the direc- tional vectors, capturing the alignment between the optimal and actual transitions from the current state:
Lg ED
Eo (11th + VVV(s V .rair) - Q a, hanr))
(11)
Lv =E= (Q(s,, harr) V(s, Tuar))
(12)
where Q is the target Q-function, (2(-) the expectile loss (Eq. (6)), and Thair ~ Unif(Sd-1) is a unit vector sampled uniformly from the unit sphere. Hcre, gd-1 := { E Rd
fTE cos (so()) - (Scurr)), (Sreaccde)) (S(uu))) (9)
|e|| }. and d is the dimension of the Temporal Dis- tance Representation (TDR). This intrinsic reward provides dense learning signal by measuring the directional align- ment via the inner product with hair- thereby improving value prediction.
To ensure the quality of the constructed graph, we utilize only states with high TE values (c.g., fTE 2 0.9) for graph construction (Section 4). ensuring that only efficient tran- sitions incorporated. Through experiments, TE not only reduces computational overhead during the graph construc- tion but also enhances overall task performance by improv- ing the quality of generated nodes. The complete graph construction process, including the application of the TE metric, is shown in Appendix A.
4.3.2. LUBGOAL-CONDITIONN POLICY LEARNING
Previous offline hierarchical reinforcement learning (HRL) methods typically adopt step- step-based sampling (Ajay et al.,
5

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Algorithm Task Planning and Execution
Input: Graph Hgraph (V,E), Low-Level Policy Tlows TDR v Temporal Distance Threshold HTD. Goal Grial
// Precompute Shortest Distance from Nodes to Goal // Dijkstra (rniksstrabb(ggan (ghmma) Hgraph) dist,, E V}
AntMaaee
b) Scene
c) Kitchen
2AF
LF
Dists < Dijkstra( eb (ginal), Hgraph)
R
I/ Task Planning & Execution
Set current state: Sear = env. env.reset() Compute TDR embedding: heur = (
d)
Stithinng
Exploratory
while &cur F Ninall do
Figure 4: Environments and Dataset Types. (a), (b), and (c) show representative environments used in offline goal- conditioned benchmarks. (d), (e), and (f) illustrate three different dataset types in AntMaze (Gupta et al., 2020; Park et al., 2025a).
// Identify reachable nodes within HTD distance Vnear + {v V ||A øl12 S Hmp
I/ Select subgoal with shortest distance to goal Usubgoal = arg min (Dists(u) + || v1l2)
// Execute Low-Level Policy
2021; Shin & Kim, 2023; Li al., 2024b), where subgoal is selected fixed number of steps ahead (e.g.. steps) from the current state within the same trajectory (Park et al., 2023; 2024c). However, a fixed c-step horizon may not align with the temporal distance separation between graph nodes defined by HID, leading to inconsistency between training and task execution. To resolve this, we adopt a TD- aware subgoal sampling strategy that selects the state after fixed temporal distance HTD within the same trajectory. This corresponds to Suub F(tt, HTD), defined in Eq. (7), and ensures that the subgoal horizon used for policy learning aligns with the one used during task execution.
hdirr = dir(heur- Vsubgoal) // Eq. (13)
Ocar ~ Tow Tow (Scur, Rkiir)
Scur = env.step(aeu
heur (/(uu))
end while
4.4. Task Planning and Execution
GAS utilizes the constructed graph to support task plan- ning and subgoal selection during the task execution of the learned low-level policy. The framework does not require high-level policy learning for subgoal generation. At the beginning of each episode, Dijkstra's 's algorithm (Dijkstra, 1959) is used to precompute the shortest distances from all graph nodes to the final goal. Among the reachable nodes within temporal distance HTD. the agent selects the subgoal with the precomputed shortest distance. The low-level policy then predicts primitive actions to reach the selected subgoal. By removing the need for high-level policy learning, this method improves the overall efficiency and stability of the framework while achieving high performance in environ- ments requiring stitching ability. (Algorithm 1)
Once the subgoal selected, its representation also plays a critical role policy generalization. Previous studies have shown that leveraging normalized subgoal representations, rather than absolute ones, improve downstream task performance (Park et al., 2023; 2024c). To enhance general- ization, we represent subgoals as a relative direction vector, where hair = dir(w)(;) (ssub))):
(b - (5()) (vbb)) - (xx)l
dir(w(st), (ssub))
(13)
To train the low-level policy, we employ deep deterministic policy gradient and behavioral cloning (DDPG+BC) (Lilli- crap et al., 2016; 6; Fujimoto & Gu, 2021; Park al., 2024b) objective, which directly maximizes Q-values while con- straining the learned policy to stay close to the behavior pol- icy. The policy extraction objective, denoted as EDDPGHBC, is defined as:
5. Experiments
We evaluate GAS on OGBench (Park et al., 2025a) and D4RL (Fu et al., 2020), spanning diverse dataset types. We compare its performance against offline goal-conditioned and hierarchical baselines. For each dataset, we report the average normalized return across five test-time goals, except for kitchen, which uses a single fixed goal. Each goal is evaluated with 50 rollouts, and results are averaged over 4 random seeds. Bold numbers indicate results that are at least 95% of the best pest-performin method in each row. Details of the datasets and baselines are provided in Appendices C,
ED Qss 1² (St. harr) hair) + a log a(a st, Rair)] (14)
where 11t denotes a deterministic policy that outputs the mean action, a is a hyperparameter controlling the strength of the BC regularizer.
6

---

Graph-Assisted Stitching for Offline Hiirarchical Reinforcement Learning
Table 1: Evaluating GAS on state-based offline goal-conditioned benchmarks.
Dataset Type
Dataset
GCBC
GCIQL
QRL.
CRT.
HGCRC
HHILP
HIQI.
GAS (ours)
antmaze- wattte--miiinnn--iiiiee
33.1 56
74.6 448
81.9 #2
95.3 10
58.1 =5
96.3 04
95.3 1.
96,3 = 1.3
Locomotion
antmaze- large lagge-uvvigalt navigate
23,4 3.2
32,6 147
74,9 144
85,5
153
44.3 1.11
86.8 1.56
89.9 1222
93.2 1.0.
antmaze-gaan camnazz-giant-aan navigate
0.0 0.0
0.1 04
14.3 3.
15.0 457
7.2 a L.7
53.1 2.26
67.3 1555
77.6 1 29
aitmaze-medum--ee stisth
43.2 7:7
26.6 +
67.0 1106
$7.0 #7
65.9 57
96.0 1.2
92.0 28
98.1 =
Stitching
antmazo-large largo-ssthc
2.3 = 3.6
9.6 31
20.2 = 1.7
14.4 #59
10.7 = 5.8
34.1 # s.0
71.7 4:48
96.3 = 0.9
antmaze-gen seaaaaze-ginn-ttiie -iant-stithh
0.0 0.0
0.0 0:00
0.4 1.3
0.0 1:00
0.0 = 0.0
0.0 1:00
1.0 1 1.2
88.3 1: 3.6
antmaze bettizeeeediumm--eeee expllre
2.7 2.8
11.7 13
1,4 1.2
1,0 1.66
15.0 8.2
39.9 7.74
32.2 3.30
98.1 1. 0.4
Exploratory
nntmaze-lagee-xplond
0.0
0.6 05
0.3 1
0.0
0.0
2.4 19
2.9 2.3
94.2
acene-play
5.4 = 0.9
50.4 + 1.4
5.1 1.7
19.2 #.00
4.6 = L.3
43.4 #:5
40.0 9.6
73.6 = 8.0
Manipulation
kitchen-partial
69.5 =14.
55.6 +17.
61.9 #.8.
32.7 #11.7
71.1 = 622
66.7 9.90
73.1 #24
87.3 = 8.8
Table 2: Evaluating GAS on pixel-based offline goal-conditioned benchmarks.
Dataset Type
Dataset
GCIQL
QRT
CRL
HHIIP
HIQL
GAS (ours)
visual- -al--aatta-------aaaaaae
Aisual-antmazz-met
19.1 +
0.0 =.00
93.7 1.2
94.1 # 122
95.5 +
96.4 = 0.5
Locomotion
4.6 + 1.
0.0 =000
79.5 75
85.6 + 2.5
80.0 #
87.0 = 1.2
couual-antmzze-eee wiaaaizzeziiattttteee -aviggtee
1,5 a 0.8
0,2 a.
43.4 1.9
42.4 1: 1.9
34.1 1: 14.0
59.0 1:1.
visual-antmazz
4.2
0,0 0.00
68.0 a3
92.4 1.2
90.4 +:41
90.0 3.0
Stitching
Nauull-attmaze--arge-tto
0.2
0.1 2:08
14.7
33.8
38.5 4:57
75.2
0.0
0.2
0.0
3.6
0.9
55.8
Exploratory
0.0 + 0.0
2.33
0.0 0:00
0.0 + n.
0.9 +
65.9 :: 6.68
visull-nttazz--lrge-exppl
0.0 + 0.0
0.0 = 0.0
0.0 2:00
0.0 + 0.
0.0 0:00
15.1 1.68
Manipulation
sisual-scene-play
10.6
13.5 2.8
8.4 0.09
35.6 1
47.9 1:3.
54.4 1:62
5.1. Main Results
due to its contrastive learning objective (Eysenbach et al., 2022; Zheng et al., 2024). To improve performance in visual state environments, we plan to enhance GAS by incorporat- ing advanced representation learning techniques for visual feature extraction.
We first present the evaluation results numeric state vironments in Table 1. Overall, our GAS achieves the best performance across all tasks. Among the baselines, HIQL performs well on the relatively easier navigation dataset, How-
achieving a score of 67.3 in antmaze- saataaazegiiattnavaian ever, its performance dramatically drops 1.0 in antmaze- giant-stitch, revealing its difficulty in trajectory stitching. In contrast, GAS significantly improves performance in the same task, achieving 88.3 by explicitly performing trajec- tory stitching through TD-aware Graph Construction. Fur- thermore, while baseline models struggle with low-quality datasets, failing to learn meaningful policies, GAS maintains strong performance. For example, in caamazz-large-expllle GAS achieves 94.2, whereas HHIIPP and HIQL achieve only 2.4 and 2.9, respectively.
5.2. Ablation Study
5.2.1. TEMPORAL EFFICIENCY FILTERING
We investigate the impact of filtering inefficient transition states using the Temporal Efficiency (TE) metric and con- structing graph exclusively with high-quality states. Using all states for graph construction significantly increases the computational cost of clustering, particularly in large-scale datasets. Moreover, in datasets with many noisy transitions, the resulting graph may contain inefficient nodes that de- grade subgoal selection quality during task planning. Table 3 compares the performance of GAS with TE filtering to a variant that constructs the graph from all states without filter- ing, Experimental results show that constructing the graph from a small number of high-quality states, selected via the TE metric, not only reduces graph construction over- head but also improves task performance. The effect is par- ticularly pronounced in the antmaze- largaaze-arge--xplore. which contains many suboptimal trajectories. This indicates that GAS enables more refined subgoal selection by filtering out inefficient transition states from low-quality datasets.
Table 2 presents the performance results in visual state en- vironments. GAS outperforms the baselines in most cases. Notably, it is the only model that achieves meaningful per- formance challenging tasks such as saual-antmaze-giant stitch and visual-antmaze-lle wwwpez---lareeeplooree pared numeric state environments, the overall perfor- mance is lower, which can be attributed to a lack of high- dimensional representation learning for visual input. In con- trast, CRL shows improved performance in visual-an listal-antmaze- giant-navigate compared to its numeric counterpart, possibly
However, com-
7

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Table 3: The Temporal Efficiency (TE) metric improves task performance while reducing computational overhead. This table compares graphs constructed using all states and those built from states filtered by the TE metric. The TE-filtered States (%) column reports the percentage of states used for clustering, which significantly lowers graph construction overhead while leading to higher-quality nodes and improved task performance. As shown the # Nodes in Graph column, the number of nodes is consistently fewer than I % of the total dataset states across all tasks in this table.
TE- F'-Filtered States (%)
Nodes in Graph
Normalized Return
Dataset
# States in Dataset
All States
Ours
All States
Ours
States
Ours
A t
untmaze-giaaee swteeree-gaattttaiiaie navigute
IM
100
6
2092
978
63.4 3
77.6 29
+14,2
cetmaze-giant-stinee
1M
100
8
3490
1966
75.3
88.3 3/6
+13.0
antmaze-lange waannazz-lrge-expeeee exgllee
5M
2499
75,4 1:43
94.2 1.30
+18.8
scene-play
1M
100
6
2809
725
63.5 155
73.6 1.80
+10.1
60
100
TE-Filtered States (%)
Normalized Return
90
40
80
20
70
60
0.85
0.9
0.95
0.99
0.85
0.9
0.95
0.99
TE Threshold
TE Threshold
antmaze-giant-nawigate antmaze-giant-stiich
antmaze-large-explore
scene-play
Figure 5: Impact of the Temporal Efficiency (TE) threshold 0thresh on graph construction and task performance. (left) Percentage of states retained for clustering, relative to the total number of states the dataset. (right) Normalized return achieved under each @thresh. These results show that, with an appropriately chosen @thresh TE filtering significantly reduces graph construction overhead while preserving improving task performance.
5.2.2. TEMPORAL EFFICIENCY THRESHOLD
5.2.3. GRAPH NODE SELECTION METHOO
We further analyze the impact of the Temporal Efficiency (TE) threshold TE on both the proportion of retained states and task performance, as shown in Figure 5. For ex- ample, when the TF threshold is set to 0.99, approximately 2% to 8% of the states are retained, depending on the dataset. This clearly demonstrates that filtering low-efficiency states in advance not only reduces graph construction overhead but also helps preserve improve task performance. Note that the TE-Filtered States (%) column refers exclusively to the graph construction process. During low-level policy training, the entire dataset was utilized. Additionally, for all tested values of @thresh between 0.9 and 0.99, the num- ber of nodes remains below 1 of the total dataset states in all numeric state environments, which also helps reduce the computational cost during task planning. Additional statistics on the number of graph nodes for all datasets are provided in Appendix C.
In this experiment, compare different graph node se- lection methods, presented in Figure 6. Specifically, we evaluate two commonly used node selection algorithms: Farthest Point Sampling (FPS) (Eldar et al., 1994), widely adopted in prior graph-based reinforcement learning (RL) studies (Kim et al., 2021; Lee al., 2022; Kim et al., 2023; Park et al., 2024a), and K-Means++ (Arthur & Vassilvit- skii, 2007), and compare them to proposed method. Both baseline methods construct graphs in the Temporal Distance Representation (TDR) space using latent states. Additionally, to ensure a fair comparison, all methods use the same number graph nodes as GAS. Experimental results show that GAS consistently outperforms both FPS and K-Means++ across all tasks. This performance gain stems from enforcing uniform separation in temporal dis- tance between nodes, which enhances graph connectivity and improves subgoal selection during task planning.
8

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
100
Table 4: Comparison of subgoal sampling strategies dur- ing low-level policy training. Our subgoal sampling strat- egy aligns subgoal selection with the temporal distance HTD between graph nodes, maintaining temporal consisteney and improving task performance, partieularly for large HTD.
Normalized Return
80
70
Dataset
HyD
Random Direction Sampling
Step-based Sampling
Ours
antmase giant navigale antmase giant stitch
8
23,3 1:40
74,9 =s1
77.6 1:19
5.3 118
86.0 127
88.3 1.14
8
83.6 #
91.4 =28
94.2 +:0
50
scone-play
48
35.8 +35
17.6 =47
73.6 +:50
antmaze giant navigate
antmaze giant stitch
antmaze large
explore
Scene play
Average
37.0
67.5
83.4
FPS
K-mean+++
Ours
Table 5: Impact of the temporal distance threshold HTD on graph construction and task performance. Selecting an appropriate HTD per task can improve graph quality and overall task performance.
Figure 6: Comparison of graph node selection methods in the Temporal Distance Representation (TDR) space. Our method explicitly enforces uniform separation in temporal distance between nodes, leading to consistently better task performance.
Dataset
HID
Nodes in Graph
Normalized Return
4
4286
74,4 +
8 (ours)
978
77.6 #:
5.2.4. LOW-LEVEL SUBGOAL SAMPLING STRATEGY
untmaze-giant-oo navigate
12
431
70.8 +:15
16
268
64.1 4:22
In this experiment, we compare three subgoal sampling strategies used during low-level policy training, as shown in Table 4. The first is random direction sampling, where directions are sampled from uniform distribution. The sec- ond is step-based sampling, where subgoal is selected fixed number of steps ahead from the current state within the same trajectory. The last is our subgoal sampling strategy, which selects a subgoal located at a fixed temporal distance HTD within the same trajectory. Experimental results show that GAS consistently outperforms both baselines across all tasks. This performance gain stems from the alignment between the sampling horizon and the temporal distance between graph nodes, ensuring temporal consistency be- tween training and task execution. The effect becomes more pronounced in environments with a large HID- Note that these subgoal sampling strategies are used only during low- level policy training, while the value function is trained independently using randomly sampled directions.
4
12901
80.7
8 (ours)
1966
88.3
amtmaze-giant-stich
12
688
80.2 1:40
16
375
68.5 4:17
15143
98.2 1:13
8 (ours)
2499
94.2 #:3,0
amimazz-ange--eploree
12
1126
87.6
16
679
90.2 1.78
44
855
71.5 :: 10.2
48 (ours)
725
73.6 1.90
scene-play
52
600
70.7 1:08
56
478
72.5
6. Conclusion
We proposed Graph- Stitching (GAS), a novel of- fline hicrarchical reinforcement learning (HRL) framework that eliminates the need for explicit high-level policy learn- ing. Instead, GAS formulates subgoal selection as a graph- based planning problem, improving long-horizon reasoning and trajectory stitching. To achieve this, GAS constructs graph using Temporal Distance Representation (TDR) and clusters semantically similar states from different trajec- tory segments into unified graph nodes, enabling efficient transition stitching. To improve graph quality and reduce construction overhead, GAS filters out inefficient transitions by applying Temporal Efficiency (TE) metric. Through extensive experiments on OGBench and D4RL benchmarks, GAS consistently outperforms prior methods across diverse offline goal-come goal-conditioned tasks. In particular, GAS achieves a score of 88.3 in natmaze--iian-stine large-explore, dramatically surpassing the previous ot the-art scores of 1.0 and 2.9, respectively.
5.2.5. TEMPORAL DISTANCE THRESHOLD
The temporal distance threshold HTD serves as a key hy- perparameter that brings structural consistency to the GAS framework. is used across multiple components, includ- ing Temporal Efficiency (TE) filtering. TD-aware clustering, and edge connection during graph construction. In addition, HID is applied to subgoal sampling during low-level pol- icy training and subgoal selection during task planning. As shown in Table 5, we observe that the effect of HTD on per- formance varies across tasks. This indicates that selecting an appropriate HID per environment can improve graph quality and overall task performance.
and 94.2 in antmaze-

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Impact Statement
transformations of Python+NumPy programs, 2018, URL
https: https:://gittbbtcco/o com/ gom/ goomm//oooll//eeoeoooee j//jax.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Brockman, G., Cheung. V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym.
ArXiv, abs/1606.01 01540, 2016.
Cao, C., Yan, Z., Lu, R., Tan, J., and Wang. X. Offline goal- conditioned reinforcement learning for safety-critice tasks with recovery policy. In International Conference on Robotics and Automation (ICRA), 2024.
Acknowledgement
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Ko- rea government (MSIT) (No. RS-2024-00348376, High- Intelligence, High- Versatility, High-Adaptability Reinforce- ment Learning Methods for Real-World Composite Task Robots) and the Institute of Information Communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. RSS-2024-00438633 De- velopment of software reliability improvement technology through identification of abnormal open sources and auto- matic application of DevSecOps), (No. RS-2022-11221455 Self-directed Multi-Modal Intelligence for solving unknown Accel-
Dijkstra, E. W. A note two problems in connexion with graphs. Numerische Mathematik, 1(1):269-211 2712699-71,, 1959.
Eldar, Y., Lindenbaum, M., Porat, M., and Zeevi, Y. Y. The
farthest point strategy for progressive image sampling. In International Conference Pattern Recognition (ICPR), 1994.
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., Legg, S., and Kavukcuoglu, K. Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures. International Conference on Machine Learning (ICML), 2018.
open domain problems), (No. RS-2025-02218777 erated Insight Reasoning via Continual Learning) and (No. 111201,- Artificial Intelligence Graduate School Program).
Eysenbach, B., Salakhutdinov, R., and Levine, S. Search on the replay buffer: Bridging planning and reinforce- ment learning. In Neural Information Processing Systems (NeurIPS), 2019.
References
Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O. Opal: Offline primitive discovery for accelerating of- fline reinforcement learning. In International Conference on Learning Representations (ICLR), 2021.
Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. Contrastive learning as goal-conditioned reinforce- ment learning. In Neural Information Processing Systems (NeurIPS), 2022.
Andrychowiez, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In Neural Information Processing Systems (NeurIPS), 2017.
Faust, A., Oslund, K., Ramirez, O., et al. Prm-rl: Long- range robotic navigation tasks by combining reinforce- ment learning and sampling- planning. In Internu- tional Conference on Robotics and Automation (ICRA), 2018
Arthur, D. and Vassilvitskii, S. k-means++: The advantages of careful seeding. In Symposium on Discrete Algorithms (SODA), 2007.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv, abx/2004.07219, 2020.
Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. ArXiv; 0ts/16007006500 2016.
Fujimoto, S. and Gu, S. S. A minimalist approach to offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
Bagaria, A. al. Skill discovery for exploration and plan- ning using deep skill graphs. In International Conference on Machine Learning (ICML), 2020.
Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Interna- tional Conference on Machine Learning (ICML), 2019.
Barto, A. G. and Mahadevan, S. Recent advances in hier- archical reinforcement learning. Discrete event dynamic systems, 13:341-379 2003.
Ghosh, D.,, Gupta, A., Reddy, A., Fu, J., Devin, C., Eysen- bach, B., and Levine, S. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations (ICLR), 2021.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman- S., and Zhang, Q. JAX: Composable
10

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Ghugare, V., Lampert, C. H., Kipf, T., Anand, A., and Thomas, V. Closing the gap between td learning and supervised learning: A generalisation point of view. In International Conference on Learning Representations (ICLR), 2024.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Kostrikov, I., Yarats, D., and Fergus, R. Image augmentation is all you need: Regularizing deep reinforeement learning from pixels. In International Conference on Learning Representations (ICLR), 2021.
Gulcehre, C., Wang, Z., Novikov, A., Smith, V., Shao, Y., Szepesvari, C., Barreto, A., and Munos, R. RI unplugged: Benchmarks for offline reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2020.
Kostrikov, I,, Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. In International Conference on Learning Representations (ICLR), 2022.
Gupta, A., Kumar, V., Lynch, C., Levine, S., and Hausman, K. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning (CoRL), 2020.
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off offlpolliy q-learning via bootstrapping errrr reduction. In Neural Information Processing Systems (NeurIPS), 2019.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning (ICML), 201 2a.
Kumar, A., Zhou, A.., Tucker, G., and Levine, S. Conser- vative q-learning for offline reinforcement learning. In Neural Information Processing Systems (NewrIPS), 2020.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In International Conference on Machine Learning (ICML), 2018b.
Kumar, and Todorov, F. Mujoco haptix: virtual reality system for hand manipulation. In International Confer- ence Humanoid Robots (Humanoids), pp. 657-663, 2015.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). ArXiv, abs/1606.08415, 2016.
Lange, S., Gabel, T., and Riedmiller, M. Batch reinforee- ment learning. In Reinforcement Learning: State-of-the- Art. pp. 45-73. Springer. 2012.
Hoang, C., Sohn, S., Choi, J., Carvalho, W., and Lee, H. Successor feature landmarks for long-horizon goal- conditioned reinforcement learning. In Neural Informa- tion Processing Systems (NeurIPS), 2021.
Lee, S., Kim, J., Jang, I., and Kim, H. J. DHRL: A graph- based approach for long-horizon and sparse hierarchical reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2022.
Jeong, H., Kim, D., Kim, D. W., Back, S., Lee, H.-C., Kim, Y., and Ahn, H. J. Prediction of intraoperative hypoten- sion using deep learning models based on non-invasive monitoring devices. Journal of Clinical Monitoring and Computing, 1377-11357-1365 2024.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial. review, and perspectives on open problems. ArXiv, abs/2005.01643, 2020.
Levy, A., Konidaris, G., Platt, R., and Saenko, K. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations (ICLR), 2019.
Kaelbling, L. P. Learning to achieve goals. In International Joint Conference on Artificial Intelligence (IJCAI), 1993.
Li, G., Shan, Y., Zhu, Z., Long, T., and Zhang, W. Diffstitch: Boosting offline reinforcement learning with diffusion- based trajectory stitching. In International Conference on Machine Learning (ICML), 2024a.
Kim, J., Seo, Y., and Shin, J. Landmark-guided subgoal generation in hierarchical reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2021.
Kim, J., Sco, Y., Ahn, S., Son, K., and Shin, J. Imitating graph-based planning with goal-conditioned policies. In International Conference on Learning Representations (ICLR), 2023.
Li, Z., Nie, F., Sun, Q., Da, F., and Zhao, H. Boosting offline reinforcement learning for autonomous driving with hierarchical latent skills. In International Conference on Robotics and Automation (ICRA), 2024b.
Kim, S., Choi, Y., Matsunaga, D. E., and Kim, K.-E. Stitch- ing sabetrajjcctoies with conditional diffusion model for goal-conditioned offline rl. In Association for the Ad- vancement of Artificial Intelligence (AAAI), 2024.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous con- trol with deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2016.

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J., Levine, S., and Sermanet, P. Learning latent plans from play. In Conference Robot Learning (CoRL), 2019.
Sobal, V., Zhang, W., Cho, K., Balestriero, R., Rudner, T. G. J., and LeCun, Y. Learning from reward-free offline data: A case for planning with latent dynamics models. ArXiv, abs/2502. 14819, 2025.
Ma, Y. J., Yan, J., Jayaraman, D., and Bastani, O. How far i'll go: Offline goal-conditioned reinforcement learn- ing via f-advantage regression. In Neural Information Processing Systems (NeurIPS), 2022.
Sutton, R. S., Precup, D., and Singh, S. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. In International Conference on Machine Learning (ICML), 1999.
Nachum, O., Gu, S., Lee, H., and Levine, S. Data-efficien hierarchical reinforcement learning. In Neural Informa- fion Processing Systems (NeurIPS), 2018.
Todorov, E.., Frez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In International Confer- ence on Intelligent Robots and Systems (IROS), 2012.
Nachum, O., Gu, S. S., Lee, H., and Levine, S. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Rep- resentations (ICLR), 2019.
Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jader- berg, M., Silver, D., and Kavukcuoglu, K. Feudal net- works for hierarchical reinforcement learning. In Interna- tional Conference Machine Learning (ICML), 2017.
Newey, W. and Powell, J. L. Asymmetric least squares estimation and testing. Econometrica, 5594))819-847, 1987.
Wang, M., Yang. R., Chen, X., Sun, H., Fang, M., and Mon- tana, G. Goplan: Goal-conditioned offline reinforcement learning by planning with learned models. Transactions on Machine Learning Research (TMLR), 2024.
Park, J., Oh, S., and Kim, Y. Novelty- awwre graph traversal and expansion for hierarchical reinforcement learning. In Conference on Information and Knowledge Management (CIKM), 2024a.
Wang, T., Torralba, A., Isola, P., and Zhang, A. Optimal goal- reaching reinforcement learning via quasimetric learn- ing. In International Conference on Machine Learning (ICML), 2023.
Park, S., Ghosh, D., Eysenbach, B., and Levine, S. Hiql: Offline goal- conditiodeo rl with latent states as actions. In Neural Information Processing Systems (NeurIPS). 2023.
Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. ArXiv, abs/1911. 1136611, 2019.
Park, S., Frans, K., Levine, S., and Kumar, A. Is value learning really the main bottleneck in offline rl? In Neural Information Processing Systems (NeurIPS), 2024b.
Yang, R., Lin, Y., Ma. X., Hu, H., Zhang, C., and Zhang, T. What is essential for unseen goal generalization of offline gond-conditioned rl? In International Conference on Machine Learning (ICML), 2023.
Park, S., Kreiman, T., and Levine, S. Foundation policies with hilbert representations. In International Conference on Machine Learning (ICML), 2024c.
Yoon, Y., Lee, G., Ahn, S., and Ok, J. Breadth-first explo- ration on adaptive grid for reinforcement learning. In International Conference on Machine Learning (ICML), 2024.
Park, S., Frans, K., Eysenbach, B., and Levine, S. Ogbench: Benchmarking offline goal-conditioned rl. In Interna- tional Conference Learning Representations (ICLR), 2025a.
Zakka, K., Tassa, Y., and MuJoCo Menagerie Con- tributors. MuJoCo menagerie: A collection of high-quality simulation models for MuJoCo, 2022. URL https://github com/ nujocoomenageritn
Park, S., Li, Q., and Levine, S. Flow q-learning. In In- ternational Conference on Machine Learning (ICML), 2025b.
deepmind/
Zhang, T., Guo, S., Tan, T., Hu, X., and Chen, F. Generating adjjcceccy-constrae subgoals in hierarchical reinforce- ment learning. In Neural Information Processing Systems (NeurIPS), 2020.
Shin, W. and Kim, Y. Guide control: Offline hierarchical reinforcement learning using subgoal generation for long- horizon and sparse-reward tasks. In International Joint Conference on Artificial Intelligence (IJCAI), 2023.
Zheng, C., Eysenbach, B., Walke, H., Yin, P., Fang, K., Salakhutdinov, R., and Levine, S. Stabilizing contrastive rl: Techniques for robotic goal reaching from offline data. In International Conference on Learning Representations (ICLR), 2024.
Sikchi, H., Chitnis, R., Touati, A., Geramifard, A., Zhang, A., and Niekum, S. Score models for offline goal- conditioned reinforcement learning. In International Conference on Learning Representations (ICLR), 2024.
12

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
A. Graph Construction
Algorithm 2 TD-Aware Graph Construction
Input: Offline Dataset D, TDR e,, Temporal Distance Threshold HTD, TE Threshold @thressh
Output: Graph Hgraph (V)
// TE Filtering
Initialize TDR state set: H. +- (:: for each trajectory 5 E DD do
for each state Scur E do
heur (Seur)
Optimal state: Mopt = (F HTD)) Eq. (7)
Actual reached state: Preached 2) (Scur Hm))
OTF cos cos(/o heur, Mreached heur)
if QTE 2 0 then
H + H (hear}
end if
end for end for
// TD-awre Clustering
V + {h1}
C1 + (h1
for each state h2 E H., > do
Find the nearest center: h, = arg min ||h, hll2
if h4 - hell2 Hyp/2 then
Create new cluster: C; + (h,)
Insert a new cluster center node: V +- VU (h:}
else
Assign h; to existing cluster: C. + C. U (74) end if end for
Reset node set: V 0
for each existing cluster C. do
Update cluster center: lie [co Ei,ec. hi
Insert updated center node: V VU { end for
// Graph Edge Connection
Initialize edge set & +-0 0
for each pair of nodes (vi, 0;) E V do Compute distance: di = |1: 12
if d.j < HTD then E+ E (v,,,;))
end if end for
Computational complexity. The time complexity of TE (Temporal Efficiency) filtering is O(N). where N is the total number of states in the dataset. After filtering, we perform TD-aware clustering on M selected high- TE states, which incurs a time complexity of (M2). Finally, constructing the graph edges by computing pairwise distances between nodes requires (I))), where | denotes the number of graph nodes.
13

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
B. Additional Results
arttmaze-gian-naaiiaal
aattmaze-giantssitte.
antmazz-large-explor
scene-play
Normalized Reeurn
Normalizzed Return
Normalizee Return
Normalized
50
0.95
Expectile (TDR)
Expectile (TDR)
Expectile (TDR)
Exoectile
(TDR)
Figure 7: Ablation on TDR expectile. Performance across different values of the Temporal Distance Representation (TDR)
expectile. The figure shows that sufficiently high expectile values lead to robust performance across most environments.
antmazee-giantraaviitte
anisssze-gint--tttccc
arttmazeelarge-exxpore
scene-play
100-
1000
100-
Normalized Return
Normalizeed Return
Normal zed Return
75-
25
75
Normalized
50-
50-
50
50
25
25
16
16
32
16
32
43
54
Latent Dimension
Latent Dimension (TDR)
Latent Dimension (TDR)
Latent Dimension (TDR)
(TDR)
Figure 8: Ablation on TDR dimension. Performance across different latent dimensions of the Temporal Distance Represen- tation (TDR). Dimensions between 16 and generally lead to robust performance across tasks.
antmmaze-giant-aaviiate
antmaze-giant-sticch
antimaze-large-explore
scene-play
100
1.0
400
100
Normel zed Return
Normalizes Reed Return
Normal zed Return
Normelizes lzed
50-
50-
50
50
25
0.0
0.01
0.1
L.0
10.0
0.01
0.1
1.0
10.0
0.01
0.1
1.5
10.0
0.01
0.1
1.
10.0
Alpha
Alpha (DDPS+BCI
Alpha (DDPG+BCI
Alpha
IDDPG+BCI
(DDPG+BCI
Figure 9: Ablation on BC coefficient. Performance across different a values in deep deterministic policy gradient and behavioral cloning (DDPG+BC). The optimal a depends on dataset quality, and selecting within an appropriate range can improve task performance.
Ablation on TDR expectile. Figure 7 shows how varying the value of the Temporal Distance Representation (TDR) expectile affects performance across tasks. The expectile coefficient plays a key role in capturing temporal distance (i.e., the minimum number steps required to transition from one state to another in the original state space). We found that using a sufficiently high expectile value (e.g., 0.95 above) led to robust performance across most environments.
Ablation on TDR dimension. Figure 8 shows how varying the latent dimension of the TDR affects performance across tasks. We observe that dimensions between 16 and 48 generally lead to robust performance. Based on this analysis, we fix the latent dimension to 32 for all experiments to balance representational capacity and stability.
Ablation on BC coefficient. Figure 9 shows how varying the coefficient a in the deep deterministic policy gradient and behavioral cloning (DDPG+BC) objective affects performance across tasks. The coefficient controls the strength of the BC regularizer. As shown in the figure, the effect of a on performance varies depending on the dataset quality. For instance, in the antmaze-large. lxrgeeexoore environment, which contains many random movements and noisy actions, using a smaller a (i.e., assigning less weight to the BC term) leads better performance.
14

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
C. Environments and Datasets
ar traaaze modium inavig
antmaae medium (havigata)
antmaae large
antmaze glant jnavig
wantmaze giant (navigate)
tmaze-medium
w-ntmazeemmdim
antmare-large
wartmace (elich)
antmare-glart (attcc))
mintmaze-giant
mmazze-medium
(esplore)
antmare-large
(oxplorc)
ver cartmaaz-aarge (explors
scene
(partial)
(saa]
500
1000
1500
2000
2500
5000
10000
15000
20000
Graph Nodes
Graph Nodes
Figure 10: Number of graph nodes per dataset. (left): state-based environments, (right): pixel-based environments.
Table 6: Summary of offline datasets and environments.
Dataset
Dataset Type
# Transitions
Maximum Episode Length
State Dim
Action Dim
antmaze-medium navvigaee
Locomotion
1M
1000
antmaze-large e
Locomotion
IM
1000
29
8
nalmaze-ggint- niviataatattee
Locomotion
IM
2000
antmaze-medium-stiice
Stitching
1M
200
29
8
antmaze-large-sticch
Stitching
1M
200
29
8
caimaaze-gian-sttneee
Stitching
IM
200
8
antmazo-medium-exy explore
Exploratory
5M
500
29
8
antmaze-lrge-expplrre
Exploratory
SM
scene-play
Manipulation
1M
1000
40
5
Michen-partiaa
Manipulation
0.13M
526
Ssstall-nttaaz--mmdiuu-----
Locomotion
IM
1000
64x64x3
8
saaull-antmaz---ar--aaaaaa navigate
Locomotion
IM
1000
64 x3
8
Aisual-antmaze- sawpaaz--ggantnnvvigggeee navigate
Locomotion
1M
2000
64x64x3 x3
8
risual-antmaze antmaze-medium-see stitch
Stitching
1M
200
64x64x3
8
visual-antman www.aaaa-laaaaeellleeeea
Stitching
IM
200
64x64x3
8
visual-antmnn coad--anmazz--gine -iintasszeggttttttthhheeee
Stitching
IM
64x64x3
visual-ane aatal-antaee-mddiug mediuu-exxplor
Exploratory
5M
500
64x64x3
8
visual-antun natmare-large-explee
Exploratory
SM
500
64x64x3
8
soual-scene-play
Manipulation
IM
1000
64x x3
5
Graph node statistics. We report the number of graph nodes constructed for each environment, as shown in Figure 10. GAS constructs compact graph by performing T'Dawwrre clustering on high-TE states selected via Temporal Efficiency (TE) filtering. This substantial reduction the number nodes significantly improves overall efficiency in task planning and execution (Algorithm 1). For example, the antmaze-large-explor only 2,499 nodes are selected from 5 million transitions, corresponding to approximately 0.05% of all states the dataset. In the nodes selected, representing approximately 0.38% of all states.
18,830
Dataset statistics. Table summarizes the key statistics offline datasets and environments used in our experiments from OGBench (Park et al., 2025a) and D4RL (Fu et al., 2020).
15

---

Graph-Assisted Stitching for Offline Hiirarchical Reinforcement Learning
Environments
Antmaze / Visual-Antmare The agent controls a quadruped Ant robot (Brockman et al., 2016) with 8 degrees of freedom (DoF) to reach a designated goal location maze. This environment requires both high-level maze navigation and low-level locomotion skills. It includes three maze sizes: medium, large, and giant, where larger mazes require more long-horizon reasoning (Park al., 2025a). The state-based observation space consists of 29 dimensions, including the agent's current 2D coordinates (i.e., x-y position) and joint-related features. In the pixel-based environments, the agent receives only 64x64x3 3 RGB images rendered from a third- camera viewpoint, and the maze floor is color-coded to enable the agent to infer its location solely from visual inputs. All environments are simulated in the MuJoCo physics engine (Todorov et al., 2012).
Scene / Visual-Scene: The agent manipulates everyday objects (e.g.. cube blocks, a drawer, a window, and two button locks) using a 6-DoF UR5e robot arm (Zakka et al., 2022) to achieve specific configurations. This environment requires sequential reasoning and diverse manipulation skill composition (Park al., 2025a). For example, to place a cube the drawer, the agent must first press a button unlock it, then open the drawer before inserting the cube. This process requires the agent to make precise sequential decisions across diverse manipulation skills. The state-based observation space consists of 40 dimensions, including joint and object-related features. In the pixel-based environments, the agent receives 64x64x3 RGB images, with the robot arm rendered semi-transe transparent and colors adjusted to ensure full observability, without requiring high-resolution inputs or multiple camera views. All environments are based on the MuJoCo physics engine (Todorov et al., 2012).
The agent controls a 9-DoF Franka robotic arm (Gupta et al., 2020) to achieve four target subtasks, such as opening the microwave, moving the kettle, turning on the light switch, and sliding the cabinet door (Fu et al., 2020). Only small portion of the dataset contains successful trajectories that complete all four subtasks sequentially. The state-based observation space is 30-dimensional, including joint and object-related features.
Kitchen:
Dataset Types
Locomotion: Evaluates gonghhorizon reasoning ability. The agent must navigate toward a goal state over multiple steps, requiring long-term planning. The dataset (Park et al., 2025a) contains long- long-horizon goal-reaching trajectories collected by noisy expert policy (Haarnoja et al., 2018b) navigating to randomly sampled goals.
Stitching: Evaluates the agent's ability to stitch together partial trajectories by combining previously unconnected
segments into an optimal plan. The dataset (Park et al., 2025a) contains short goal-reaching trajectories collected by noisy expert policy (Haarnoja et al., 2018b) navigating to randomly sampled goals.
Exploratory: Evaluates the ability to learn from suboptimal datasets. The agent must utilize low-quality, high-coverage data to learn an effective policy. The dataset (Park et al., 2025a) is collected by commanding the noisy expert pol- icy (Haarnoja et al., 2018b) 8b) with random directions re-sampled every 10 steps under high action noise.
Evaluates sequential decision-maaing ability that requires reasoning about prior actions when planning future actions. The play-style dataset (Lynch et al., 2019) is collected using open-loop, non-Markovian scripted policies that randomly interact with diverse everyday objects (Park et al., 2025a). The partial dataset (Fu et al., 2020) is derived from set of unstructured and unsegmented human demonstrations, originally collected using the PUPPET MuJoCo VR system (Kumar& & Todorov, 2015; Gupta et al., 2020).
Manipulation:
Goal Specification for Evaluation We follow the goal specification protocol of OGBench (Park et al., 2025a), where each task provides five predefined state-goal pairs. For the kitchen-partin environment (Fu et al., 2020; Gupta et al.. 2020), we construct a single fixed goal state. Specifically, we extract the joint configuration from a dataset observation and combine it with the object configuration corresponding to the target goal state, as provided by the environment.
Evaluation Metric We use normalized return as the primary evaluation metric, where the maximum achievable reward is normalized to 100. The reward structure varies across environments. Antmaze provides a sparse reward of +1 only when the agent reaches the goal state. Scene assigns a reward of +1 only if all required subtasks are completed. Kitchen provides +1 for each successfully completed subtask, with a maximum cumulative reward of 4. All scores are linearly sealed to the range [0, 100) (Fu et al., 2020; Park et al., 2025a).
16

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
D. Implementation Details
Implementation Our implementations of GAS and seven baselines are based on JAX (Bradbury et al., 2018). We our experiments on an internal cluster consisting of RTX 3090 GPUs. We also test and tune different design choices and hyperparameters for both GAS and baselines, largely following prior work (Park et al., 2023; 2024c; 2025a). To ensure fair comparison, we use a unified set of model architectures and hyperparameter values across GAS and baselines (e.g., learning rate, discount factors, and batch sizes) whenever possible. Our implementations are publicly available at the following repository: https: : / thub. com/ gortmdgh4 41 /GAS.
Baselines We compare GAS against of four offline goal conditonie algorithms (GCBC, GCIQL, QRL, CRL) and three hierarchical algorithms (HGCBC, HIQL, HHILP). In goal-conditioned offline RL, we include GCBC, a simple goal- conditioned behavioral cloning method, and GCIQL (Kostrikov et al., 2022; Park et al., 2023), which approximates the optimal value function using expectile regression. We also evaluate QRL (Wang et al., 2023), which learns a quasimetric value function with a dual objective, and CRL (Eysenbach et al., 2022), which applies contrastive learning to estimate Monte Carlo value functions and perform one-step policy improvement. For hierarchical approaches, we consider HGCBC (Ghosh et al., 2021; Park al., 2023), an extension of GCBC (Gupta al., 2020; Ghosh et al., 2021) incorporating two-level policies, and HHILP, a hierarchical extension of HILP (Park et al., 2023; 2024c) that utilizes Temporal Distance Representation (TDR). Finally, we benchmark HIQI (Park et al., 2023), which extends GCIQL to a hierarchical structure and generates subgoals in latent space.
Temporal Distance Representation (TDR). We follow the goal relabeling strategy proposed in (Andrychowicz et al., 2017; Park et al., 2023; 2024c), with the exception that we exclude the case g. as our parameterization guarantees ss) = 0. Specifically, we sample either from a geometric distribution over future states within the same trajectory (with probability 0.625), or uniformly from the entire dataset (with probability 0.375). These values are inherited from the original hyperparameter settings (Park et al., 2024c), which ensure that = g is never sampled by assigning the probability mass to both of the two sampling schemes mentioned above.
Hyperparameters We provide a common list of hyperparameters in Table 7 and task-specific hyperparameters in Table 8. We apply layer normalization (Ba et al., 2016) to all MLP layers. For pixel-based environments, we adopt the Impala CNN (Espeholt et al., 2018) to process image inputs. While most components use 512-dimensional output features, we reduce the output dimension to 32 for the Temporal Distance Representation (TDR) to balance representational capacity and stability, as discussed in Appendix B. Following prior work (Park al., 2023; 2024e; 2025a), we do not share encoders across components. As result, in pixel-based environments, we use four separate CNN encoders for TDR, the Q-function, the value function, and the low-level policy. We also apply random crop augmentation (Kostrikov et al., 2021) with a probability of 0.5 to mitigate overfitting (Zheng et al., 2024).
Table 7: Common hyperparameters used across all datasets.
Hyperparameter
Value
Image representation architecture (pixel-based)
Impala CNN (Espeholt et al., 2018)
Image augmentation probability pixel-basede
0.5 (Kostrikov et al., 2021)
TDR MLP dimension
(512, 512, 512)
Value MLP dimensions
(512, 512, 512)
Policy MLP dimensions
(512, 512, 512) 32
TDR Latent dimension
TDR normalization
LayerNorm (Ba et al., 2016)
Value normalization
LayerNorm
Nonlinearity
GELU (Hendrycks & Gimpel, 2016)
Value expectile
0.7
Target network smoothing coefficient
0.005
Learning rate
0.0003
Optimizer
Adam (Kingma & Ba, 2015)
17

---

Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning
Table 8: Task-specific hyperparameters for each dataset.
Dataset
# Gradient Steps
Batch Size
Discount Factor
TDR Expectile
TTE
HTD
mntmare-mcdium-naigeee
1000000
1024
0.999
0.99
antmaze-large-navigal
1000000
1024
0.99
0.999
0.99
8
intmaze-giant-navigane
1000000
1024
0.995
0.999
0.99
antmaze- untmaze-medium: -eed-medd-iiiiii stitch
1000000
1024
0.99
0,999
0,99
8
antmaze-late large--strrg
1000000
1024
0.99
0.999
0,99
8
antmaze- siatizc--gian--sticii
1000000
1024
0.995
0,999
0,99
8
untmazze-medium-explarr
1000000
1024
0.99
0.999
0.99
8
0.01
antmaze-large-explore
1000000
1024
0.99
0.999
0.99
8
0.01
scene-play
1000000
1024
0.99
0.999
0.99
48
kitchen-partial
500000
1024
0.99
0.95
0.9
48
10
visual-antmazz-mmeddim -aaiammzzz--miiu---iiiieeeee
500000
256
0.95
8
lisual-anlmaze-laree navigate
500000
256
0.99
0.95
0.9
8
500000
256
0.995
0.95
visual-antmaze-mdddum medium-ticc stitch
500000
256
0.99
0.95
0.9
8
visual-antmazz-llrg--ttttt
500000
256
0.99
0.95
0.9
8
visual-antmaze-giaaa
ceatinaazz-ggan-ttiiiieee giant-stitch
500000
256
0.995
0.95
0.9
8
aat-aaattmaaedlle explore
500000
256
0.99
0.95
0.9
8
0.01
500000
256
0.99
0.95
0.9
8
0.01
visual-scene-paa
500000
256
0.99
0.95
0.9
24
E. Subgoal Visualizations
TRA
FLAS
PLD
Agent Subgoal Initial State Final Goal
Figure 11: Shortest path computed on the TD-aware graph from antmaze-giant-stitch The graph is constructed in latent space, but for visualization purposes, each node embedding is projected onto 2D plane using approximate 2D coordinates (i.e., x-y position).
18


Reviews:
[{"review_id": "KZ6bEhg2P4", "reviewer": "Reviewer_WQgj", "summary": "This paper replaces the high-level policy learning in traditional Hierarchical Reinforcement Learning (HRL) with a graph search problem. When constructing the high-level graph, GAS clusters states that are similar in the Temporal Distance representation space into a single graph node to achieve efficient trajectory stitching. Additionally, the paper introduces a Temporal Efficiency metric to improve the quality of the constructed graph. Subgoals are then selected through a shortest path algorithm within the graph. Experimental results demonstrate that GAS outperforms previous hierarchical reinforcement learning algorithms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for suggesting the additional relevant works: BEAG, HIGL, and PIG. While these methods are primarily designed for online learning and planning, we agree that they share conceptual relevance with our work in graph construction strategies, and we will discuss them in the revised manuscript.\n\n    [HIGL] Landmark-guided subgoal generation in hierarchical reinforcement learning, NeurIPS 2021.\n    [PIG] Imitating graph-based planning with goal-conditioned policies, ICLR 2023.\n    [BEAG] Breadth-First Exploration on Adaptive Grid for Reinforcement Learning, ICML 2024.\n\n* Both HIGL and PIG construct graphs using the Farthest Point Sampling (FPS) method. In our ablation study (Table 3), we compare our method with an FPS-based graph construction strategy under the same conditions: using the same pre-trained temporal-distance (TD) representation and the same state sampling based on temporal efficiency (TE), but replacing our clustering-based graph construction with FPS. \n\n   While FPS covers the latent space by iteratively selecting the farthest state from the set of already selected nodes, our TD-aware clustering constructs the graph by encouraging uniform separation in temporal distance between nodes. This difference enhances the temporal connectivity of the graph, which in turn leads to consistently stronger performance than FPS across all tasks, as shown in the table below.\n\n* BEAG constructs a graph by applying a coarse-to-fine grid refinement strategy over a goal space defined explicitly by low-dimensional coordinates (e.g., x and y). While this approach is effective in AntMaze-like environments, it has three major limitations. \n  - First, it heavily relies on human knowledge, since the user needs to manually define the goal space and set lower and upper bounds for each dimension.\n  - Second, it suffers from poor scalability in high-dimensional goal spaces. For example, tasks involving manipulation with multiple objects or visual observations lead to an exponential increase in the number of grid cells. \n  - Third, as shown in the table below, BEAG exhibits limited performance under offline-only training conditions. This is primarily because it determines reachability through repeated online rollouts, which are not available in the offline setting.\n\n* To assess BEAG fairly, we conducted additional experiments with extended online fine-tuning after offline training. We observed that BEAG reached performance levels close to, but still slightly below, GAS. In summary, while BEAG is effective in online and low-dimensional settings, it is less suitable for offline learning scenarios or high-dimensional domains.\n\n    ||||||||    \n    |:-|-:|:-:|:-:|:-:|:-:|:-:|\n    ||｜||**offline-only**||｜|**offline-to-online**||\n    |**Environment**|｜|**BEAG**|**FPS**|**GAS**|｜|**BEAG**|\n    |antmaze-giant-navigate |｜|24.9 ± 4.0|60.1 ± 4.3|**76.2** ± 0.6|｜|74.1 ± 4.9|\n    |antmaze-giant-stitch|｜|15.0 ± 2.4|70.6 ± 5.9|**85.0** ± 1.9|｜|71.3 ± 10.4|\n    |antmaze-large-explore|｜|20.2 ± 1.7|60.1 ± 6.1|**91.2** ± 2.8|｜|82.3 ± 4.3|\n    |scene-play|｜|❌|53.6 ± 4.4|**63.7** ± 3.2|｜|❌|\n    |kitchen-partial |｜|❌|77.0 ± 14.3|**87.3** ± 8.8|｜|❌|\n    |||||||| \n\nOther Comments Or Suggestions\n* We agree that the use of “%p” (percentage point) might be unclear or unconventional. To avoid confusion, we have revised the sentence in the abstract to the following: “GAS outperforms prior hierarchical RL methods across locomotion, navigation, and manipulation tasks, achieving up to 83.6% higher success rate in the most stitching-critical task.", "claims_and_evidence": "Yes, all the statements are essentially correct.", "methods_and_evaluation_criteria": "The datasets used for evaluation are highly comprehensive.", "theoretical_claims": "The paper does not include any theoretical derivations.", "experimental_designs_or_analyses": "The experiments lack baselines for graph-based goal-conditional reinforcement learning algorithms (Section 5.2), and the method of constructing the graph is not sufficiently compared with previous methods (Section 5.3.3).", "relation_to_broader_scientific_literature": "Please go ahead with the next question.", "essential_references_not_discussed": "Regarding the combination of graphs and goal-conditional RL, the paper only mentions DHRL and NGTE in the related works. However, there are other relevant works that also construct graphs to provide goals for low-level policies, such as:\n\n- Breadth-First Exploration on Adaptive Grid for Reinforcement Learning [ICML 2024],  \n- Landmark-guided subgoal generation in hierarchical reinforcement learning [NeurIPS 2021],  \n- Imitating graph-based planning with goal-conditioned policies [ICLR 2023].  \n\nAlthough these papers are not necessarily offline algorithms, they also build graphs to provide goals for low-level policies.", "other_strengths_and_weaknesses": "**Strengths:**\n\n- The overall writing logic of the paper is rigorous and coherent. The figures are simple and aesthetically pleasing, making the paper highly readable.\n\n**Weaknesses:**\n\n- The paper's survey of work combining graphs and goal-conditional RL is not comprehensive enough. In the experimental section, comparisons with other graph-based goal-conditional RL methods are missing. Although the previous papers may not be offline algorithms, the methods for constructing the upper-level graph can be directly transferred to offline RL. If the paper could strengthen the discussion, analysis, and comparison of relevant literature in this area, as well as add more graph-related baselines, I would be willing to increase the paper's score.", "other_comments_or_suggestions": "It is possible that there might be a typographical error in line 30 of the abstract, where it says “83.6%p.” It seems unusual and might need clarification or correction.", "questions_for_authors": "Please refer to weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qvUl14hABM", "reviewer": "Reviewer_g3Bk", "summary": "This paper presents a novel offline hierarchical RL method, which constructs a state graph in a learned temporal distance representation space, and selects subgoals from the graph rather than from a high-level policy. The constructed graph facilitates trajectory stitching, and improves the task performance given suboptimal demonstrations. Extensive evaluations are performed on both state- and pixel-based environments, the proposed method achieves strong performance across different tasks and data optimality settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s constructive feedback.\n\n1. While both our work and Ghugare et al. [1] aim to address trajectory stitching in the offline reinforcement learning setting, the two approaches differ significantly in structure and mechanism. Ghugare et al. [1] focus on improving generalization in flat, supervised learning-based goal-conditioned policies by applying temporal goal augmentation. Their method allows the agent to better handle unseen (state, goal) combinations by augmenting training with alternative intermediate goals.\n\n        [1] Ghugare et al. Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View. ICLR 2024.\n\n    In contrast, our method performs explicit graph-based stitching by leveraging temporal-distance (TD) representations and temporal efficiency filtering. This enables the agent to construct a TD-aware graph across trajectory segments and perform hierarchical subgoal planning to guide low-level policy learning across disconnected trajectories.\n\n    Moreover, our approach demonstrates strong performance in large-scale environments, such as AntMaze-Giant, and under challenging data conditions, e.g., datasets consisting mostly of short trajectories or low-quality demonstrations (random movements with noisy actions). These aspects highlight the robustness of our graph-based stitching strategy. We will include a discussion of [1] in the revised version of the paper.\n\n2.  As our method relies on a pre-trained Temporal Distance Representation (TDR), it is true that the quality of representation may degrade in regions of the state space that are not covered in the dataset. This limitation can affect both graph construction and final performance. However, we note that this is a general limitation shared by most offline RL and HRL methods, which inherently depend on the support of the offline dataset. \n\n    A key advantage of TDR is that it learns to preserve both distance and direction between states in the embedding space. This allows stitching to occur even between non-overlapping short trajectories, as long as the observed states are embedded close to each other. Moreover, in low-quality datasets, such as those composed of random movements with noisy actions, our method can still select temporally efficient states to construct an effective graph, leading to higher performance across diverse and challenging tasks. \n\n3. The expectile coefficient $\\tau$ plays a key role in capturing temporal distance between states. Higher $\\tau$ values emphasize short-step transitions, enabling efficient subgoal planning via shortest-path search on the TD-aware graph. The table below shows that sufficiently high $\\tau$ results in consistently robust performance across tasks.\n    ||||||\n    |:-|:-:|:-:|:-:|:-:|\n    |Dataset|$\\tau{=}0.9$|$\\tau{=}0.925$|$\\tau{=}0.95$ (ours)|$\\tau{=}0.975$|\n    |antmaze-giant-stitch|84.9 ± 2.8|82.5 ± 5.1|**85.0** ± 1.9|84.7 ± 0.9|\n    |scene-play|61.4 ± 1.4|60.8 ± 3.5|63.7 ± 3.2|**68.6** ± 2.0|\n    |||||||\n\n4. Performance according to the value of $H_\\text{TD}$:\n    As shown in the table below, we observe that the value of $H_\\text{TD}$ does have some impact on performance across different tasks. However, the method is generally not overly sensitive to the specific value, and robust performance can be achieved within a reasonable range of $H_\\text{TD}$. Of course, selecting an appropriate $H_\\text{TD}$ value per task can improve planning and graph construction quality. \n\n    |||||\n    |:-|:-|:-:|:-:|\n    |**Dataset**|**$H_\\text{TD}$**|**#Nodes in Graph**|**Normalized Return**|\n    |antmaze-giant-stitch|8|19769|79.8 ± 4.8|\n    ||12|15187|**86.5** ± 2.1| \n    ||16 (ours)|5501|85.0 ± 1.9|\n    ||20|2748|80.4 ± 2.2|\n    |||||\n    |scene-play|44|14428|60.4 ± 1.3|\n    ||48 (ours)|10208|**63.7** ± 3.2|\n    ||52|6993|57.6 ± 3.3|\n    |||||\n\n    ||||||||\n    |:-|:-:|:-:|:-:|:-:|:-:|:-:|\n    |**Dataset**|｜|**$H_\\text{TD}$**|**#States in Dataset**|**#Nodes in Graph**|｜|**#Nodes ${/}$ #States (%)**|\n    |antmaze-giant-navigate|｜|16|1M|4605|｜|0.46|\n    |antmaze-giant-stitch|｜|16|1M|5501|｜|0.55|\n    |antmaze-large-explore|｜|16|5M|5647|｜|0.11|\n    |scene-play|｜|48|1M|10208|｜|1.02|\n    |kitchen-partial|｜|48|136k|414|｜|0.30|\n    ||||||||\n\n5. Other Comments Or Suggestions:\n    We will revise the manuscript to include the missing citation [2], and ensure that all tables and figures are correctly cross-referenced throughout the text. As suggested, we will also include a graph visualization. The following anonymous link shows the shortest path computed on the graph from the antmaze-giant-stitch dataset:\n    * Anonymous link: https://imgur.com/a/I1C3yAW\n    * The TD-aware graph is constructed in a latent representation space, but for intuitive visualization, each node’s embedding vector is projected onto a 2D plane using approximate XY coordinates.\n\n    [2] Park, S., Kreiman, T., and Levine, S. Foundation Policies with Hilbert Representations. ICLR 2024.", "claims_and_evidence": "The benchmark evaluation and ablation studies support claims about the effectiveness of the overall approach and the independent design choices.", "methods_and_evaluation_criteria": "The proposed method is well-designed to solve the problem of interest. Constructing a structured graph in the temporal distance space from the dataset is intuitively helpful for trajectory stitching, especially from suboptimal demonstrations. The evaluation protocol and metrics also follow the common practice in the related work.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Both evaluation tasks and baselines are properly selected for the investigated research questions. The analyses on benchmark results and ablations are fairly thorough. For other comments, please see \"Other Strengths And Weaknesses\" below.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "One focus of this paper is solving trajectory stitching in the offline setting. I believe the algorithm proposed in [1] is related and should be discussed as well.\n\n[1] Ghugare et al. Closing the Gap between TD Learning and Supervised Learning - A Generalisation Point of View. ICLR 2024.", "other_strengths_and_weaknesses": "Strengths:\n- Explicitly depicting the temporal structure through a clustered graph is straightforward and effective in discovering subgoal candidates.\n- Leveraging the subgoal direction vector instead of the exact subgoal representation provides a more general form of subgoal conditioning, and is potentially helpful for better generalization.\n\nWeaknesses:\n- The foundation of the proposed methods is the quality of temporal distance representation, which should be affected by the state coverage of datasets and the choice of $\\tau$. More ablation should be conducted on how the hyperparameters of TD representation learning would impact task evaluation performance.\n- It’s essential to demonstrate how task performance will be affected by the value of $H_{TD}$, as it affects both graph construction and task planning", "other_comments_or_suggestions": "Please properly cite the prior work on line 261 – “Previous studies”. Several places, such as “Table 2” on line 355, and “Figure 5” on line 378, are not linked to the corresponding tables and figures. Additionally, it would also be helpful to illustrate a graph structure constructed from one (or several) evaluated datasets.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYFdT9hc6c", "reviewer": "Reviewer_Qgst", "summary": "This paper proposes a new graph-based method called GAS for offline GCRL, particularly for long-distance goal tasks. The main idea of GAS is to find subgoals for goal-conditioned RL tasks using shortest path planning in a state-space graph. GAS constructs this graph in a Temporal Distance representation space using the original states. In addition to representation learning, GAS reshapes the sparse terminal-only reward with Temporal Distance.\n\n## update after rebuttal\n\nSome of my concerns still remain. The paper uses many techniques to ensure success—not just TDR representation learning (which I still have doubts about despite the explanation in the rebuttal), but also a dense reward. However, many of the baselines don't use dense rewards, so I'm also worried about the fairness of the performance comparison.  I chose to keep my score unchanged.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful feedback and questions. Due to space limits, we focused on key reviewer concerns.\n\n1. ... value functions are noisy due to sparse reward signals. How does GAS ensure that representation learning is not negatively affected by this noise?\n\n        Temporal-distance representation (TDR) differs from value learning based on traditional temporal-difference learning. TDR is trained to preserve temporal relations, such as distance and direction between states in the embedding space. This enables more stable and noise-resilient representation learning, especially in sparse reward and long-horizon tasks. \n\n        While predictions over longer distances may still be affected by noise, GAS does not rely on separate high-level policy learning based on value estimates. Instead, it constructs a graph directly in the TDR space, and planning is performed using this graph structure. As a result, low-level policy training is performed within a temporally reliable distance range, effectively mitigating the impact of accumulation errors.\n\n2. ... How does GAS ensure the effectiveness of clustering? Additionally, how are edges between nodes constructed?\n\n        To address the challenge of clustering in high-dimensional spaces, GAS first trains a temporal-distance representation (TDR) that embeds states into a latent space where temporal proximity is preserved as Euclidean distance. This embedding significantly reduces the complexity of clustering by enabling accurate and geometry-aware clustering in the learned latent space, as opposed to directly clustering in the raw high-dimensional state space. We then select the centers of these clusters as graph nodes. \n\n        For edge injection, we again leverage the TDR. An edge is added between two nodes if their temporal distance is below a certain threshold. This approach ensures that edges represent actual reachability between subgoals, enabling reliable stitching and subgoal planning.\n\n3. ... Which of these contributes the most to performance improvement? In Table 5, GAS also uses larger batch sizes and wider network structures. Do these factors contribute to performance gains?\n\n        1) Among the components of GAS, we identify temporal-distance representation (TDR) learning as the most crucial contributor to performance improvement. TDR enables the selection of temporally efficient states, the effective clustering of the selected states, and meaningful edge injection between clusters. We note that in the case of training on low-quality datasets, which often include random movements and noisy actions, the benefit of temporal efficiency-based state selection becomes more prominent in Figure 5,\n        \n        2) Regarding the concern about batch size and network architecture, we clarify that GAS and all baseline methods were trained with exactly the same batch size and network width to ensure fair comparison.\n\n\n4. Since GAS is heavily algorithm- and experiment-focused, what is its computational complexity? Constructing a state graph is expected to have high computation and memory burden. How expensive is computing the shortest path on a large graph?\n\n        We clarify that temporal-distance representation (TDR) is trained as an additional pretraining process before graph construction. After TDR is learned, we perform:𝑂(𝑁) computation to measure temporal efficiency over all 𝑁 states in the dataset and select M (temporally) efficient ones, followed by 𝑂(M^2) clustering on the selected M states to construct the graph nodes. As shown in the table below, we use less than 1% of the total states in datasets as graph nodes in most environments, which significantly reduces both computation and memory usage.\n\n        For subgoal planning, we compute the shortest path using Dijkstra’s algorithm. Its time complexity is: 𝑂(𝑉^2) using array, or 𝑂((𝑉+𝐸)log𝑉) using heap-based implementations, where 𝑉 is the number of nodes and 𝐸 is the number of edges. Importantly, shortest path computation is only performed once per episode, given a new final goal.\n\n5. ... If the low-level policy fails to reach a subgoal, how is the next subgoal selected?\n\n        At every time step, we search for candidate nodes within a temporal distance of H_TD from the current state in the embedding space. Among these candidates, we can select the subgoal that results in the shortest estimated path to the final goal when passing through the candidate. The distances from all nodes to the goal are pre-computed at the beginning of each episode using Dijkstra’s algorithm. This strategy enables GAS to operate robustly and flexibly, even when the agent fails to reach a designated subgoal or deviates from the planned route.\n\n        Although the subgoal selection methods differ across baselines, we empirically found that subgoal selection at every step consistently led to their best performance. Therefore, we also adopted per-step subgoal selection for all baselins.", "claims_and_evidence": "GAS constructs a graph to help the agent perform stitching. However, it's a little confusing  how the stitching is done by the state graph. How exactly does this stitching process work? Typically, stitching is achieved through TD learning of the value function [1]. Does GAS's final stitching performance also benefit from the dense reward-based value function?\n\n[1] Levine, Sergey, et al. \"Offline reinforcement learning: Tutorial, review, and perspectives on open problems.\" arXiv preprint arXiv:2005.01643 (2020).", "methods_and_evaluation_criteria": "The paper combines representation learning with state graph construction to find effective subgoals along the shortest path in the state graph. To demonstrate the effectiveness of GAS, it provides extensive experimental results.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "* Some experimental details are not provided, including hyperparameter tuning for key parameters like $H_{TD}$ and the architecture of the representation function $\\psi(\\cdot)$. \n\n* Graph-based approaches are known for their computational inefficiency, especially with large state spaces and big offline datasets. However, GAS does not discuss the computational cost of the algorithm.", "relation_to_broader_scientific_literature": "The key contribution of GAS is construct the state graph with Temporal Distance which is computed by representation learning. This fall in the methods that try to reach distance goal by path planning to find landmarks/waypoints/subgoals in a graph. The paper also claim the constructed graph can help agent for stitching, as mentioned in **Claims And Evidence**. From the paper, it is unclear how exactly this is implemented. Additionally, the paper does not analyze whether this contribution is more significant compared to other graph-based methods.", "essential_references_not_discussed": "Many graph-based approaches for goal-conditioned tasks [2,3] are missing from the related work and baseline discussions.\n\n[2] Eysenbach, Ben, Russ R. Salakhutdinov, and Sergey Levine. \"Search on the replay buffer: Bridging planning and reinforcement learning.\" Advances in neural information processing systems 32 (2019).\n\n[3] Hoang, Christopher, et al. \"Successor feature landmarks for long-horizon goal-conditioned reinforcement learning.\" Advances in neural information processing systems 34 (2021): 26963-26975.", "other_strengths_and_weaknesses": "1. Some algorithmic details are unclear, such as how goals are selected during representation learning and how the value function is trained.  \n\n2. Graph-based approaches are known for their computational inefficiency, especially with large state spaces and big offline datasets. However, GAS does not discuss the computational cost of the algorithm.", "other_comments_or_suggestions": "Some notations are unclear. For example, what does $d$ represent in Equation 7? What does $z$ mean in Equation 9?", "questions_for_authors": "1. Representation learning in GAS relies on TD updates of the value function. In goal-conditioned RL (GCRL), especially in long-horizon tasks, value function are noisy due to sparse reward signals. How does GAS ensure that representation learning is not negatively affected by this noise?  \n\n2. When building the graph, states need to be clustered, with cluster centers serving as nodes and subgoals. Clustering in high-dimensional spaces is challenging. How does GAS ensure the effectiveness of clustering? Additionally, how are edges between nodes constructed?  \n\n3. GAS integrates multiple components, including representation learning, state graph construction, and reward reshaping. Which of these contributes the most to performance improvement? In Table 5, GAS also uses larger batch sizes and wider network structures. Do these factors contribute to performance gains?  \n\n4. Since GAS is heavily algorithm- and experiment-focused, what is its computational complexity? Constructing a state graph is expected to have high computation and memory burden. How expensive is computing the shortest path on a large graph?  \n\n5. Subgoals are provided iteratively to the low-level learning policy. If the low-level policy fails to reach a subgoal, how is the next subgoal selected?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oXVfdgEFFm", "reviewer": "Reviewer_kThB", "summary": "In this work, the authors propose an offline HRL framework that leverages graph to solve a long-horizon problems efficiently. To solve the challenges in long-horizon reasoning, the proposed method construct graph using temporal efficiency metric and clustering, leaving only high-quality subgoal. After then, the low-level policy can utilize a graph planning method over the constructed graph to achieve far apart final goals. As a result, the proposed method shows improved performance in several long-horizon manipulation and locomotion environments, indicating its improved capability in stitching and reasoning over long horizons.\n\n## Update After Rebuttal\n\nThe authors have addressed most of the questions I raised.\nWhile this work does differ from some prior studies in that it trains the low-level policy in an offline manner, I still believe that a major component of the algorithm—graph construction—got huge insights from previous works. Therefore, I find the novelty to be relatively limited and will maintain my original assessment.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s insightful feedback and questions. Due to space limits, we focused on key reviewer concerns.\n\n1. It does not clearly outline the differences from previous studies.\n\n        All existing graph-based methods, such as [HIGL], [L3P], [SoRB], and [DHRL], are designed for online learning settings, where exploration plays a central role. As a result, they have typically not been compared alongside recent offline RL/HRL benchmarks.    \n            \n       Except for [L3P], these methods construct graphs over low-dimensional goal spaces, such as (x, y) positions, making them inapplicable to high-dimensional or visual goal spaces. Additionally, these methods rely on Farthest Point Sampling (FPS) for graph construction. States are randomly sampled from the replay buffer. For the sampled states, nodes are selected iteratively by choosing the state that is farthest from the already selected nodes. This strategy is highly sensitive to the data distribution and quality, which can result in suboptimal graph structures.\n\n        In our Table 3, we compare GAS with a variant that uses FPS applied to TE-filtered latent states, after the same Temporal Distance Representation (TDR) pretraining. The results show that FPS performs better than traditional baselines when combined with our representation and filtering, but our proposed clustering and graph construction method still achieves the highest performance.\n\n2. Does the Temporal-Efficiency (TE) Metric offer clear benefits over prior research?\n\n        TE Metric enables selecting efficient transitions within the dataset and uses only these states for clustering and graph construction. As shown in Figure 5, applying TE filtering consistently improves performance across all environments. The benefit is especially pronounced in low-quality datasets.\n\n3. About K time-step sampling in Table 4.\n\n        In this experiment, we keep the graph construction process fixed while varying only the subgoal sampling method used for low-level policy training. Traditional HRL methods typically use K time-step sampling, where the subgoal is set to the state K steps ahead in the same trajectory. In contrast, our method samples the subgoal based on a temporal distance of K in the latent space. able 4 shows that this temporal-distance-based subgoal sampling consistently outperforms the conventional K time-step method in terms of task success rate.\n\n4. Expectile Loss in Eq (6)\n\n        Eq (6) is adapted from IQL (Kostrikov et al., ICLR 2022), which originally introduced the expectile loss formulation. A related discussion can be found in Point 3 of our response to Reviewer g3Bk.\n\n5. DDPG + BC setup\n\n        The hyperparameter α controls the trade-off between the Q-learning objective from DDPG and the BC loss. As shown in the table, the impact of α on performance varies depending on the quality of the dataset. For example, in antmaze-large-explore, which contains random movements and noisy actions, we found that lowering α (i.e., reducing the weight of the BC term) led to significantly better performance.\n\n    |||| \n    |:-|:-|:-:| \n    |**Environment**|****$\\alpha$****|**Normalized Return**|\n    |antmaze-giant-navigate|1.0 (ours)|**76.2** ± 0.6|\n    ||0.1|74.4 ± 13.0| \n    ||0.01|11.7 ± 7.4|\n    ||||\n    |antmaze-giant-stitch|1.0|77.8 ± 5.0|\n    ||0.1 (ours)|**85.0** ± 1.9|\n    ||0.01|58.2 ± 8.1|\n    ||||\n    |antmaze-large-explore|1.0|7.6 ± 2.3|\n    ||0.1| 80.2 ± 6.2|\n    ||0.01 (ours)|**91.2** ± 2.8|\n    ||||\n\n6. If no data exists in the middle of the space?\n\n        In all tasks, the dataset provided sufficient coverage in the regions necessary for task completion, and the graph remained connected enough to allow successful planning using Dijkstra's algorithm.\n\n        In extreme cases where the dataset is severely limited and the graph is not well constructed, this presents a fundamental challenge not only for GAS, but for any graph-based or even offline RL methods, as no method can infer reasonable actions in regions entirely unobserved in the dataset. To address this issue in practice, we need to perform additional online fine-tuning, which enables the agent to observe unseen regions and subsequently refine the graph as needed.\n\n7. If the subgoal is not successfully reached?\n\n        For GAS and all baseline methods, the agent checks the distance to the current subgoal at every step and automatically switches to the next subgoal once it is reached within a threshold. If the agent significantly deviates from the originally planned path during execution, we replan by selecting the nearest graph node to the current state in the latent space as a new subgoal. We then recompute the shortest path from that node to the final goal and resume subgoal progression accordingly. This type of dynamic subgoal replanning is a commonly used strategy in graph-based planning methods and allows GAS to operate robustly and flexibly even when execution deviates from the original plan.", "claims_and_evidence": "Compared to all baselines (single-level / hierarchical policies), the proposed method consistently demonstrates improved performance, showing relatively large gains particularly in stitching and exploratory tasks.\n\nHowever, while the TD-aware graph construction is indeed effective, it seems necessary to provide a more thorough explanation of related previous work. For example, the Farthest Point Sampling Algorithm used in [1] DHRL and the Greedy Latent Sparsification employed in [2] L3P also already use TD-aware node selection algorithms. It would be better to explain how the proposed method differs from these approaches and what advantages it offers.\n\nOne of the key contributions claimed is the graph construction based on the Temporal Efficiency Metric. In this aspect, does it offer clear benefits over prior research? Especially when looking at Table 4, the k time-step sampling already shows good performance (surpassing other baselines).", "methods_and_evaluation_criteria": "The proposed method and evaluation metric seems appropriate. However, there remains some concerns about whether the comparison is fair, given that all the baselines are either single-level RL/BC or HRL methods that do not use graphs.\n\nAdditionally, it would be good to visualize the constructed graph in order to include some qualitative results.", "theoretical_claims": "No proofs or theoretical claims", "experimental_designs_or_analyses": "Experiment design makes sense.", "relation_to_broader_scientific_literature": "This study presents an improved algorithm and performance in offline HRL compared to prior work (HIQL). However, it does not clearly outline the differences from previous studies [2-4]. Although some of these earlier studies used online HRL rather than offline HRL, and there are some differences in settings, a more thorough comparison and explanation of the differences are necessary, especially considering they also employ graph-based RL approaches.\n\n\nAn analysis of failure cases would be very helpful for future research. Is the issue due to problems in the graph construction? Or is it because the low-level policy fails to correctly follow the subgoals?", "essential_references_not_discussed": "There are more HRL papers that use graphs on offline/online data, but they seem to be missing here. In particular, since these works also proposed their own approaches to graph construction, it would be better to cite these papers [2-4] and compare how each of their graph construction methods differs from the one proposed in this study.", "other_strengths_and_weaknesses": "This study clearly demonstrates overall improved performance compared to the baselines. Moreover, it shows successful results on rather complex manipulation tasks, which sets it apart from previous graph-based RL methods.\n\n\nHowever, in some tasks, there may be bottlenecks that require fine-grained actions (such as passing through narrow gaps or performing precise movements). While the proposed graph construction method selects nodes based on TE and achieves efficient performance with a small number of nodes, it seems unlikely that it can effectively extract such critical points.", "other_comments_or_suggestions": "In Section 3.2, the definitions of the low-level MDP and high-level MDP are somewhat unclear. Do both levels share the same goal space? Is the low-level goal space identical to the high-level action space? Additionally, both transition probabilities are represented as p—are they actually the same?\n\n\nEquation 7 also feels a bit unclear. What does S_min refer to? And what exactly is happening inside the curly braces on the right-hand side? Is the search space in Equation 7 limited to that trajectory, or does it cover the entire dataset D?\n\nThe explanation of the clustering part is also somewhat lacking. While I was able to understand it through Figure 1 and the algorithm in the appendix, it’s inconvenient that this component—one of the main contributions—is not fully explained in the main text. How about creating a separate figure that breaks down the TE filtering and clustering steps in a step-by-step manner?\n\nLine 237 (left): \"the difference between these two values will be different\"—what exactly does \"difference\" mean here?", "questions_for_authors": "Is there a specific reason for using a loss function in the form of Eq (6)? Is this something inherited from HIQL? Would it not work well if you simply used an L2 norm?\n\n\nIn the DDPG + BC setup, how significant is the contribution of each component? How does performance change depending on the value of alpha? How was alpha tuned? Does using just BC alone result in poor performance?\n\n\nIn environments like Explore (as opposed to Navigation or Stitch), it seems likely there would be points where no data exists in the middle of the space. In such cases, if there’s no suitable point within the threshold, does it result in edges not being formed and the graph becoming disconnected? If there’s no solution found by Dijkstra’s algorithm, how is this handled as an exception?\n\n\nWhat happens if the subgoal is not successfully reached? After following a subgoal for a specific number of timesteps, does the agent simply move on to the next subgoal? (It seems there’s no description of this kind of exception handling in the algorithm.)\n\n\n[1] DHRL (referred in this paper) https://arxiv.org/pdf/2210.05150\n[2] World Model as a Graph: Learning Latent Landmarks for Planning [https://arxiv.org/pdf/2011.12491]\n[3] Search on the Replay Buffer: Bridging Planning and Reinforcement Learning [https://arxiv.org/abs/1906.05253]\n[4] Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning https://arxiv.org/pdf/2110.13625", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes GAS, a graph-based offline HRL framework that replaces high-level policy learning with shortest path planning in a state graph. The graph is built in a Temporal Distance representation space, clustering similar states to enable efficient trajectory stitching. A Temporal Efficiency metric ensures high-quality subgoal selection. GAS demonstrates strong performance on long-horizon tasks, outperforming prior HRL methods across diverse manipulation and locomotion benchmarks.
The reviewers expressed some concerns about comparisons to baselines (addressed in rebuttal), novelty and use of additional rewards. The authors should take these comments into consideration to clarify those elements in their paper.
Decision: Accept (poster)