RD-HRL: Generating Reliable Sub-Goals for Long-Horizon Sparse-Reward Tasks
Authors: Yixiang Shan
Haipeng Liu
Ting Long
Yi Chang
Venue: ICLR 2026 Poster (2026)
Keywords: Hierarchical Reinforcement Learning
Sub-goal
Key States
Choice Learning
Reinforcement Learning
Goal-conditioned Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We propose RD-HRL, which introduces a reliability-driven decision mechanism to address the issue of sub-optimal sub-goals caused by generalization errors in existing HRL methods.

Abstract: Long-horizon sparse-reward tasks, such as goal-conditioned or robot manipulation tasks, remain challenging in offline reinforcement learning due to the credit assignment problem. Hierarchical methods have been proposed to tackle this problem by introducing sub-goal planning guided by value functions, which in principle can shorten the effective planning horizon for both high-level and low-level planners, and thereby avoiding  the credit assignment problem. However, we demonstrate that the sub-goal selection mechanism is unreliable, as it relies on value functions suffering from generalization noise, which misguides value estimation and thus leads to sub-optimal sub-goals. In this work, to provide more reliable sub-goals, we novelly introduce a reliability-driven decision mechanism, and propose Reliability-Driven HRL (RD-HRL) as the solution. The reliability-driven decision mechanism provide decision-level targets for high-level policy, thereby providing noise-immune decision spaces for them, ensuring the reliability of sub-goals (which are termed as action-level targets in this paper). Comprehensive experimental results demonstrate that our approach RD-HRL outperforms baseline methods across multiple benchmarks, highlighting the competitive advantages of RD-HRL. Our code is anonymously available at \url{https://github.com/Looomo/RD-HRL-public}.

Published as a conference paper at ICLR 2026
RD-HRL: GENERATING RELIABLE SUB-GOALS FOR SPARSE-REWARD TASKS
Yixiang Shan
School Artificial Intelligence, International Center of Future Science
Jilin University China
(annyx20mmails jlu. edu. cn
Haipeng Liu, Ting Long
School of Artificial Intelligence
Jilin University
China
odu. liuhp22 jlu.edu.cn. longtingl edu.
Yi Chang
School of Artificial Intelligence, International Center of Future Science
Jilin University
Engineering Research Center of Knowledge-Driven Machine Intelligence Ministry of Education
China
yichang@ j lu.. edu. cn
ABSTRACT
Long-horizon sparse-reward tasks, such goal-conditioned robot manipula- tion tasks, remain challenging in offline reinforcement learning due to the credit assignment problem. Hierarchical methods have been proposed tackle this problem by introducing sub-goal planning guided by value functions, which in principle can shorten the effective planning horizon for both high-level and low- level planners, and thereby avoiding the credit assignment problem. However, we demonstrate that the sub-goal selection mechanism unreliable, as relies on value functions suffering from generalization noise, which misguides value esti- mation and thus leads to sub-optimal sub-goals. In this work, to provide more reliable sub-goals, novelly introduce reliab decision mechanism, and propose Reliability- -riven HRL (RD-HRL) as the solution. The reliability- driven decision mechanism provide decision-level targets for high-level policy, thereby providing noise-immune decision spaces for them, ensuring the reliabil- ity of sub-goals (which are termed action-level targets in this paper). Com- prehensive experimental results demonstrate that our approach RD-HRL outper- forms baseline methods across multiple benchmarks, highlighting the compet- itive advantages of RD-HRL. Our code is anonymously available at https: //qithub. wwwp://wwo/wo...ttee.......o..ot
public
INTRODUCTION
Reinforcement Learning (RL) Kaelbling et al. 1996 Wiering and Van Otterlo 2012 Li 2017
Sutton et al.) 1999) has achieved remarkable success in many tasks, but it still faces significant
challenges in long-horizon sparse-rewards situations (Lee al. 2022 Shin and Kim 2023; Wang
et al. 2020). In such tasks, the agent often receives meaningful feedback only upon reaching dis-
tant goals. which makes credit assignment and exploration particularly difficult (Zhou et al. 2020
Pignatelli et al. 2023). A common solution is Hierarchical Reinforcement Learning (HRL) Pateria
"Corresoponding Authors.

---

Published as a conference paper at ICLR 2026
g
Goal States
Current State
Sub-Optimal Trajectory
Bellman Backup
Transition Regions
Optimal Trajectory
Bellman Backup (Generalizod))
S
9
9
2
(c)
Figure I: The dataset contains two trajectories, TI and T2, as is shown in (a). Note that, T2 success- fully reached the goal point g. yet 72 contains a sub-optimal sub-trajectory; meanwhile, Ti fails to reach the goal, but provides an optimal sub-trajectory from & to 8 In (b), we show the optimal trajectory with blue dotted line, which contains the key transition { s + s }: in (c), we show that existing HRL methods tend to plan sub-optimal trajectories (the orange dotted line) and the attenuate generalized Bellman backup.
al. 2021), which decomposes complex tasks into hierarchy of tab-tasks to address the issue. Specifically, HRL is composed of high-level policy and low-level policy, where the high-level policy proposes intermediate sub-goals with a value function, while the low-level policy learns how to reach these sub-goals. By structuring the decision-making process in stages, HRL alleviates the learning difficulties faced by agents in long-horizon sparse-reward situations (Barto and Mahadevan, 2003 Nachum et al. 2018).
However, the value function which used to propose sub-goals is often subject to generalization errors in practice, leading to unreliable sub-goals. Consider simple scenario presented in Fig- [ (a), where the agent starts from state 8 and aims to reach the goal g. Intuitively, the optimal path should include $i+ is shown in the blue dotted line from Figure 1 (b). However, the value estimation of 5 relies on cross-trajectory Bellman backup (i.e., generalized Bellman backup from region
{s}. $ )), where the generalized signal is often attenuated and unreliable (Zhang et al. 2024). as is shown in Figure (c). This may leads to the underestimated value of the optimal sub-goal 8 s leading to the high-level policy to select suboptimal sub-goal x? instead, which eventually results in suboptimal trajectory, as is shown in the orange dotted line in Figure (c). For the sake of simplicity, we designate regions analogous to z, which facilitate the connection across different trajectories, transition regions.
Intuitively, if we can prevent the high-level policy from comparing sub-goal candidates whose value signals are unreliable, and restrict its decision space local regions that do not require such gener- alization, the impact of generalization error can be substantially reduced. Building on this insight, propose Reliability-Driven HRI. (RD-HRL., which augments HRI. with eeliability-driver decision mechanism. The reliability-driven decision mechanism selects decision-level targets for the high-level policy from transition regions Z {z}. thereby confining the high-level decision space to areas without generalization requirements and decoupling sub-goal selection from cross- trajectory value estimates. Note that for better presentation, we define the sub-goals generated by the reliability-driver mechanism for the high-level policy as decision-level target, and define the sub-gou sub-goals assigned by the high-level policy to the low-level policy as action-leve target. In this way, we decompose action-level target (i.e., sub-goals) planning into two reliable subproblems: (1) providing suitable decision-level targets through the reliability-driven decision mechanism, and (2) producing reliable action-level targets conditioned on decision-level targets.
Specifically,, the reliability-driven decision mechanism is composed of the Transition Region Ex- traction (TRE) module, the Target Identification (TI) module and the Target Evaluation (TE) mod- ule. The TRE module filters transition regions from the offline dataset, providing the candidates of decision-level target for the TI module; the TE module estimates the low-noise value of tran- sition regions for the TI module; the TI module selects proper transition regions as decision-level targets for the high-level policy with the help of the TE module. Combining the reliability-driven
2

---

Published as a conference paper at ICLR 2026
decision mechanism with the HRL framework, we propose RD-HRL for better decision- making in Long-horizon sparse-reward tasks. In summary, the contribution of our work can be summarized as:
We propose method Reliability-Driver HRL (RD-HRL), which novelly augments HRL with reliability-driven decision mechanism provide decision-level target from tran- sition regions for the high-level policy, thereby disentangling the high-level policy from generalization error, providing reliable action-level target (i.c., sub-goals).
We propose the Target Evaluation (TE) module for reliable decision-level target genera- tion, and theoretically proved that our proposed TE module can effectively reduce value noise in long-horizon sparse-reward scenarios. Experimental results further demonstrate the importance of the Target Evaluation (TE) module.
We conduct extensive experiments to validate the effectiveness of RD-HRL, along with in-depth ablation studies to verify the impact of its key designs.
PRELIMINARIES
2.1 OFFLINE GONL-CONDI REINFORCEMENT LEARNING
Offline goal-conditioned reinforcement learning (offline GCRL) is the most representative task in long-horizon sparse- reward scenarios, which can be formulated as a Markov Decision Process (MDP) (Sutton and Barto 2018) M {S, AS,A, A, R,g) with a dataset D, where S is the state space, the action space, P(8 1 [8r, ae) is the transition function, represents the discount fac- tor, 9 represents the goal, Rss,, at,g) g) is gondicondi reward function, which is commonly formulated as:
f1
||x(s;) g|12 S,
Tt = R(s ar.g)
otherwise, 0
where represents the reward received at time I, represents a given threshold, Φ(-) maps state from the state space to the goal space. Following Liu et al. (2022), we assume the goal space is identical to the state space, thus we omit 6(-) in this paper.
At each step f, the agent responds to the state of the environment 8; and goal g by action a, according to policy Te parameterized by 6, and gets instant reward rt. The interaction history is formulated trajectory T {(st, a, rt) 2 0}. which further consists D as D & {(st, Te, 8t+1) 2 0). Our goal is learning Ty to maximize the expected discounted accumulated reward without directly interacting with the environment, i...
To arg max ET
yrre
220
2.2 HIERARCHICAL REINFORCEMENT LEARNING
Hierarchical Reinforcement Learning (HRL) has been widely used for long-horizon sparse-reward tasks, which generally consists of high-level policy Tे (= and a low-level policy Th (t.(s; in which 9 represents the goal for the high-level policy, n/ generates action-level targets that are casier to achieve for x', and n' provides the action to be taken. Under the general setting, the task goal g is taken 9. Formally, n². (g") 9) and n/. (a.)st.g.) can be learned with advantage weighting regression (AWR) objectives (Qing et al. 2024 Wang et al. [023] Eysenbach
al. 2023 Park al. 2024a Osa al. 2019 Pateria 2021 Bai al. 2024):
Ca. Elexp(3 AM 81,9)) logmh, ((+.)))))))
(3)
Ca. Eleep(3 A' 8t, 8 s14 logT (arlsr, H1,,
(4)
where A represents the advantage signal, and A 8,, is approximated as V(8 g)
V(s,.) A' (ar, St, H1+ is approximated as V(s 8t++)++++--) ((s,, 81+H), Vs . g) is the value of 9t conditioned goal g. represents the subsumed temperature (Park et al.] 2024a) H is a
existing HRL. methods relies value functions affected by generalization noise select action-level targets, which makes the optimal action-level targets challenging (Park et al. 2024a Liu et al. 2022).
which also known as waysteps. As discussed Section
3

---

Published as a conference paper at ICLR 2026
Goal
Current State
High-level Policy
Planning Horizon
States
<
Transition Regions
Low-level Policy
Future Trajectory
T1 Module
Tl g)
(*r.
st
TI
( (*)
action a
a
(b)
(c)
Figure 2: Framework of RD-HRL. (a) Overview datasets, and transition regions filtered by the Transition Region Extraction (TRE) module. (b) The Transition Identification (TI) module provide g" as the decision-level target; (c) The high-level policy and low-level policy. During evaluating, the TI module predicts 21 the decision-level target gT shown in (b); then, g is provided to the high-level policy T² to generate the action-level target for the low-level policy n'.
METHODOLOGY
To improve the action-level target selection and enhance performance in long-horizon sparse-reward tasks, we propose a novel method called RD-HRL. Built based on HRI., RD-HRI. introduces reliability-driven decision mechanism, which is composed of an Transition Region Extraction (TRE) module, Target Identification (TI) module, and Target Evaluation (TE) module, as it is illustrated in Figure The TRE module extracts transition regions from offline trajectories, serving as the candi- date spaces where critical decision-level targets are likely to reside. TI selects a decision- target from these candidate regions, which acts as a reliable intermediate target to facilitate the high-level policy in generating action-level targets. TE evaluates and refines the selected decision-level targets, enabling the optimization of their selection and ensuring reliable guidance for the overall decision- making process. Finally, the high-level policy provides ceneralization-noisee noi-s--immmee action-level targets for the low-level policy to take actions. In the following, we will first give the details of TRE, TI, and TE of the reliability-driven decision mechanism. Subsequently, we will discuss how to incorporate the reliability -driven decision mechanism with HRL to enhance the decision-making long-horizon sparse-reward tasks. The pseudo code of RD-HRL can be found in Appendix B
3.1 COMPONENTS OF RELIABILITY-DRIV DECISION MECHANISM
In this section, deseribe the three essential novel components of our reliability eeiiabilityddriven decision mechanism: the Transition Region Extraction (TRE) module, the Target Identification (TI) module, and the Target Evaluation (TE) module.
The Transition Region Extraction (TRE) module responsible for extracting transition regions from the offline dataset, thereby supplying the TI module with the candidate set for decision-level target. Formally, transition region represents set of all states within block, where the block corresponds to the part where two or more trajectories are much closer to each other, for instance, in Figure (a). 21, 22 and in Figure 2 (a). We form the TRE module with simple yet effective strategy: first discretize the dataset into blocks, and then filter the transition regions from these blocks. Given dataset DD we first perform K-Means clustering (Sculley) 2010) over the states in D:
C= *=M-Means({2)) ~ N),
(5)
where N represents the number of clusters, |C| and each e E represents a cluster. For notational simplicity, we denote the cluster ID of state as Ca.: O S Cs. S N.
We then identify transition regions among these blocks. Intuitively, transition regions connect dif- ferent trajectories. Thus, we propose the Future Diversity Index (FDI) as a metric for quantifying the diversity of reachable futures from a region, thereby identifying transition regions. Formally, the FDI with respect to a cluster is defined as:
FDI(c) = He |st and 8, E T)] - T ~ D.
(6)
N
4

---

Published as a conference paper at ICLR 2026
The underlying reason is that, by connecting more trajectories, transition regions typically admit larger set of possible future directions. Note that we subtract in the numerator because, for any given cluster c, there must exist 8, s.t. Cze+: and must exist Sy s.t. Cap-1 # c. where 8, and sy denote two arbitrary states within cluster c, 8, * Sy, In other words, any cluster necessarily admits at least two possible future clusters: transitioning to other clusters, or remaining in the current one. Further, clusters with higher FDIs are considered as transition regions Z. In our paper, we consider any cluster e with (DI > as one of the transition regions. Formally, we have 2 {c FDI(c) > 0}. In this paper, we determine the optimal number of clusters using Within-Cluster Sum of Squares (WCSS) (Edwards and Cavalli-Sforza refer to Appendix|C details.
1965 Brusco and Steinley 2007 Duong et al. 2013), please
The Target Identification (TI) module is responsible for selecting E Z as the decision-level larget. Without loss of generality, we can model Tl as any arbitrary neural network parameterized by 0rz, represented as Tlør. (g [s.gs) where 8; is the state at time t, represents the overall task goal, and g" is the decision-leve target. Given the current state 8; and the overall task goal g. the TI module selects 2 E Z as the decision-level target g" for the high-level policy. By providing decision-level target for the high-level policy, the introduction of the TI module naturally restricts the decision space of the high-level policy to regions without generalization demands, thereby providing intermediate guidance for the high-level policy and addressing the generalization noise.
The Target Evaluation (TE) module is responsible for providing accurate evaluation of the decision-level targets for the TI module. Represented as TE (8, g). the TE module estimates values only for s 2, E Z, rather than for all E D. Our TE module performs updates only within the transition regions, abstracting the intermediate fine-grained RL steps into a single macro- step, thereby providing the temporal abstraction of value update. More importantly, by learning solely with respect to z ~ 2, our TE module naturally avoids generalization noise, providing reli- able learning guidance for the TT module.
3.2 RELIABLE DECISION MAKING WITH RD-HRL We now can couple the reliability-driven decision mechanism with HRI as RD-HRL for reliability- driven decision-making Given the current state 8, and the overall task goal g, the procedure of RD-HRL is given by:
g" TV Tlo,, (.),,.g. g" nb, (-1se, ((s.gg'') T)) a Té (-Is:.g"),
where g T' is the decision- target given by the TI module for high-level policy, g" is the action- level target given by the high-level policy for low-level policy, and a, is the action to be taken with respect to the current state 8, and task goal g.
Having established a basic understanding of the decision-making process in RD-HRL, we proceed to introduce the training objective of RD-HRL. Given transition regions 2 filtered by the TRE module, first learn the TE module with transition regions Z. Specifically, first denote the skeleton of T with Z as f = {..., Zis J.. Similar to previous work Park et al. (024a). our TE module can be optimized by:
CoTE EY.DD DI (st,,g) g) (re,,22 12 + t TEgie (st,
(8)
where 21, 22 E 81, 21, 8t, 22, OTE denotes the parameters of the target TE module, and Th is designed as:
11. 3s e 22), ||s: gll Se
.b (o. otherwise.
Note that since the transition region spans across trajectories, 8t, St, could be in different trajec- tories. For the sake of simplicity, we assume 8,, E T. there must 3 81g E 22, 8 E T and 81; 812- Therefore, d(st, : 8t,) ) can be approximated by i22 - 4, The advantages of the Trajectory Evaluation (TE) module arise from two aspects. First, states St, and St, may originate from different trajec- tories, which enables direct cross- trajettrayy propagation of value signals rather than generalization, there by overcoming generalization error. Second, by focusing exclusively on transition regions, the TE module requires only a limited number of value updates, thereby mitigating the accumulation of errors incurred in each update.
Given transition regions& 2 as well as the TE module, we then provide the learning procedure of the TI module. Similar to previous works (Park al. 2024a Osa et al. 2019 Pateria et al. 2021 Bai
5

---

Published as a conference paper at ICLR 2026
Table 1: The average normalized score of different HRL methods various environments, with + denoting the standard deviation. The mean and standard deviation are computed over 50 random seeds. We emphasize in bold scores within 3 percent of the maximum per task (2 M.X7* The results marked as "_" indicates that the authors of the corresponding work did not provide corresponding results, nor did they provide the corresponding code.
Dutasets
PlanDQ
MSSP
V-DTT
DTAMP
IID-A
HILL
IILP.PPa
HIQL
Diftuseriitte
RD HRL
antmaase medium diverse
93.0 =: 2,6
85.9 4: 22
52.6 1.4
28.7
58.7 = 8.1
43.52 7.6
49.2 5.1
86.8
4,5
$7.6 2.0
9.
antmase medium play
92.1:117
913 1.3
62.2 25
13
83.8 24
45.6 1.0
16.5 2: 10.4
$4.1
10.3
58.8 3
94,0-13
86.0 3.0
83.4 3.2
36,4
780+88
836=588
46. 12.7
64.5
10.2
88.2 5,3
75.2 3.5
91,3-4.3
antmaae- large play
65.3:163
86.5
11
1.6
80.01:3
807-6.1
4:0...8.
58.8 11.2
=611-2 7,5
68.4
2 6.5
95.3-2.1
7..0::4..
55.1 7.3
59.2.-311
92.266
212
59.2
12.7
51.9
17.4
68.3 + 2.5
61.1-6.3
play
71.5133
14.3
4.9
211,4
14.8
63.7
72.9-511
kitchen
75.0 + 7.1
36.9 + 33
46.0 1.6
63.41:88
73.3
6.79
5.
65.0 9.2
71.4 1.3
69.6-74
kitchen mixed
71.7
44.3
46.8
74.4114 1.4
4.3
$1.9 83
60.7
72.9-17
CALVIN
45.0 19.8
49.9 11.5
44.6
166:117
1211
145
4388 39.5
$2.1
6.88
al. 2024). the TI module can be optimized with an AWR-style objective, as follows:
LeTr = Z, (exp) A St, g)) log (82 g))l
(10)
where is transition region sampled from 2, 82 is state sampled from within the tra- jectory as 8t. AT represents the advantage function, which can be formulated as TESTE (8,,g) TEars (8..9) d(sr, 8x) represents the temporal distance between 8, and 8g. Different from learn- ing the TE module, for the TI module, we carefully select 8, E from the same trajectory as 8, to avoid decision-level uncertainty. Therefore, supposing 81 Ta, we have sy E Ts, s.t. Sy 8:; then, we have d(s s,) = t' - t. Note that we introduced a weight raised to the power of d(st, s,) Cog, as the distance between 8; and 8; (i.e.,
not always
With respect to the conventional HRL components, we first optimize the value function Vo, (s, g)- Specifically, with trajectory т being represented as T = {(s,, at, Ft, 8 8141) the value function V can be optimized by:
Lav E.NDlllVo (st, - (rr + VVo ($141, g))[la
(11)
where 7 is the discount factor, re 1 - - <e. By denotes the parameters of the target value function Park et al. (2024)); Florence al. (022)).
After learning the value function, following previous works (Qing et al. 2024 Wang et al. 2023
Eysenbach al., 2023 Park et al. 2024a| Osa et al. 2019; Pateria et al. 2021 Bai et al. 2024). the high-level policy me ($t+H St, 8.) and the low-level policy Te (a: 8,, S(+H) are learned with
Equation (3) and Equation (4p, respectively. Note that for the learning of mh (#++ 81, $2),
| replace with 8, E 2, E 2, as we are now conditioning on transition regions rather than g.
4 EXPERIMENT
In this section, conduct experiments to evaluate the performance of RD-HRL and analyze the impact of its key design. The computational resources and hyper-parameters details are available in Appendix F
4.1 EXPERIMENTAL SETUP
Benchmarks. We evaluate RD-HRL on long-horizon goal-conditioned tasks and robotic manipula- tion tasks, which serve as canonical benchmarks for long-horizon sparse-reward problems (Liu al. 2022). For long- -orizon goal-conditioned tasks, we adopt conventional antmaze- (medium, large) } [diverse, play] (Tooorov al. 2012: Brockman et al. (016)) as well untmaze-ultra uuttu-lldiiiiie (diverse, -ltra-(diverse, play) (Zhang et al.). which feature expanded and more complex maze layouts (requiring extended temporal reasoning with increased spatial challenges) as our benchmarks. For robotic manipulation tasks, adopt Kitchen (Gupta et al. 2020) and CALVIN (Mees et al. 2022)
benchmarks.
Baselines. We take methods that adopt states H-steps further as action-level targets, such as SOTA methods HIQL (Park et al.] 2024a), PlanDQ (Chen al., 2024a). MSCP (Wu et al.) (024), V- ADT (Ma et al.) 2023). DTAMP (Hong et al., (2023). ). HD-DA (Chen et al. 2024b). We also take the methods with median- based action-level target selection methods baselines, such as HILP (Park) (et al.) 2024b) and HILP- P-PPn (Park et al.) 2024b). Besides, we also take the method that uses three- layer hierarchical design, Diffuserl.in lite (Dong al. 2024 one of the baselines.

---

Published as a conference paper at ICLR 2026
Table 2: Ablation results of RD-HRL-TRE, RD-HRL-HP, RD-HRL-TE, RD-HRL-CU and RD-
HRL. We mark the best results of each task with bold.
Datasets
HIQL
RD-HRI-TRE
RD-HRL-HHP
RD-HRL-TE
RD-HRL-CU
RD-HRL.
antmaze-mediim--iiverr
86.8 1: 4.6
85.3 2.2
19.11119
90.711.9
92.813.1
94,6 2.5
mmtaaze-meddum-plag
84.1
66.717.3
86.813.
91.412.7
94,0
antmaze-large-divers
88.2 5.3
89.2 2.6
59.018.1
87994.22
89,411.3
91.314.3
antmaze-large-phee
86.1 + 7.5
84.8 4.9
54.5+7.3
85,8+13
90.244.3
95,3=2.1
antmaz--ullra--iverre
$2.9 + 17.4
59.8 74
27.8+33.3
35.3+4.4
68.1+2255
81.1=6.3
antmaze-ullra- 39.2 14.8
$2.9 9.2
32.6+9.2
57.8+4.9
66.0+-2.1
72.9-5.1
4.2 OVERALL RESULTS
The results shown in T Note that emphasize in bold scores within 3% of the maximum per task MAX),,*MAX, following previous work (Li et al. 2024). Overall, RD-HRL achieves top- 3% performance out of tasks.
With respect to the goal-conditioned tasks, RD-HRL achieves top-3% performance on all of the tasks, demonstrating the effectiveness of RD-HRL tasks with low-dimensional goal spaces. Es- pecially on untmaze-ultra-(lar diverse), which are more complex than mutmaze-(mdiine large)- (play, diverse]. RD-HRL outperforms our backbone method HIQL by 85.9% and 53.3%, demon- strating the superior performance of RD-HRL over conventional HRL methods on tasks with low- dimensional goal spaces.
For manipulation tasks, RD-HRL achieves the top-3% performance across all benchmarks except for the kitchen-partial task. We helieve that the poor performance on the kitchen-partial task arises because the dataset lacks trajectories of completion across subtasks, preventing our method from identifying transition regions. Notably, on CALVIN, our method outperforms HIQL by 57%. Given the high dimensionality of the manipulation tasks, these results verify RD-HRL's effectiveness in high-dimensional spaces.
To demonstrate the advantages of RD-HRL more intuitively, further visualize the decision- target g" provided by the TI module on antmaze- waanzzz-uttr--eee altrrediverse, please refer to Appendix E for details. We additionally evaluate RD-HRL against flat GCRL approaches, please refer to Appendix
G.2 for details.
4.3
ABLATION STUDY
In this subsection, conduct further experiments to investigate the key designs of RD-HRL. Specif- ically, we design the following variants for ablation studies:
RD-HRL-TRE: learning the TI module with state at + 2H rather than 2 ~ Z;
RD-HRL-HP. removing high-level planner x" and directly providing g to T';
RD-HRL-TE replacing TE with V;
RD-HRL-CU: removing cross- trajectrayy updating value from the TE module.
The results are summarized in Table 2 By analyzing the results in Table 2 we can easily conclude the following key findings:
(1) The benefits transition regions extracted by the TRE module extend beyond merely enabling larger H. Recall that we propose the Target Identification (TI) module, which provides transition regions extracted by the TRE module as decision-level target g for the high-level policy nh In this part, we conduct experiments to validate the effectiveness of learning decision-level target g" from the transition regions Z.
Specifically, further propose variant RD-HRL-TRE. which replaces Z as {8142H) }. and report the results in Table 2 As can be observed, (1) RD-HRL outperforms RD-HRL-TRE on all of the antmaze tasks, suggesting that the proposed transition region in method indeed enables reliable action-level target, leading to improved performance. (2) AD-HRL-TRE outperforms HIQL on most tasks, indicating that increasing H may enhance performance some extent.
(2) The Target Identification (TI) module goes beyond merely being a higher-level replacement of oي The Target Identification (TT) module delivers decision-level target to the high-level policy.

---

Published as a conference paper at ICLR 2026
One might curious whether TI merely stands higher-level replacement of the high-level policy, and whether it would instead be possible to pass such target directly to the low-level policy?
We believe the answers to the aforementioned questions negative. We answer these via the variant RD-HRL-HP, in which we remove the high-level policy, and directly provide the decision- level target to the low-level policy. The results summarized in Table 2 Comparison between RD-HRL-HP with RD-HRL reveals a significant performance drop. This is particularly evident in the antmaze-ultra- (diverse, play) environments, where performance declines by 65.2% and 55.3%, respectively. We attribute this to that the Target Identification (TI) module goes beyond being just higher-level replacement of x² The decision- target g² provided by the TI module may be unreachable for low-level policy; in such cases, the high-level policy is required to decompose g"
(3) The advantages of Trajectory Evaluation (TE) Module stem both form the direct cross- trajectory value update and the temporal abstraction. As an ablation, we first propose RD- HRL-TE by replacing the TE module with V in RD-HRL to examine its contribution. As is shown in Table 2 RD-HRL-TE exhibits a certain performance degradation, which is more pronounced in (ddv-errrrrrddvsseee complex environments such as antmaze-ullra- (diverse, play). This demonstrates that the TE module can handle long-horizon scenarios better than the original value function V. Moreover, despite the performance decline, we can see that RD-HRL-TE still achieves results comparable to or better than HIQL in 5 out of the 6 environments, further highlighting the advantages of the TI module introduced by RD-HRI.
We believe the advantages of the TE module stem from the two critical components: the deter- ministic cross-trae value update and temporal abstraction. To assess the contribution of these components, introduce variant termed RD-HRL-CU, in which the deterministic cross-trajectory value update is ablated while temporal abstraction is preserved. The corresponding results are re- ported in Table 2 As can be observed, on the one hand, RD-HRL-CU exhibits a certain degree of performance degradation compared with RD-HRL, with a notable drop of 16. 1% on antmaze- ultra-diverse. This directly demonstrates the advantage conferred by the directly cross-trajectory value update. On the other hand, RD--HRL-CU still consistently outperforms RD-HRL-TE across all datasets, providing empirical evidence that the retained temporal abstraction contributes substan- tially to its effectiveness. We attribute this to the fact that temporal abstraction reduces the frequency of value updates, thereby mitigating the cumulative errors incurred by stepwise updates. Please refer to Appendix D for theoretical proof.
4.4 FURTHER INVESTIGATIONS
4.4.1 COMPARISON BETWEEN HORIZON ENLARGEMENT AND RD-HRL
RD-HRL adopts the Target Identification (TI) module to provide decision-level target g" TT for high- level policy xh To some extent, this may be regarded as equivalent to extending H, since the decision-level target g²d typically more long-term than high-level targets g². Moreover, discus- sions in Section 4.3 indicate that increasing H may enhance performance. Thus, question arises: Is the improved performance due to the larger H?
To verify this conjecture, tested the performance of HIQL under different H and summarized the results alongside RD-HRL's performance in Figure 3 As can be observed, both on antmaze- ultra-play and natmaze-ultra-diveed
with the waysteps H increasing, the performance of HIQL improves, reaching its peak at H = 50. In other words, increasing H can improve the performance of HIQL at early stages. Yet, note that our RD-HRL still outperforms HIQL under H 50. After H = 50, the performance decreases as H gets larger. We attribute this phenomenon to the fact that H continues increase, the action-level targets g" of HIQL become increasingly difficult for 8; to achieve, consequently leading to performance degradation. As a brief recap, although increasing H does improve the performance, the TI module we introduced fundamentally what gives RD-HRL its competitive advantage.
4.4.2 EXTENDING TRANSITION REGIONS BEYOND EUCLIDEAN SETTINGS
While the results in Table 2 already demonstrate the feasibility of RD-HRL, specifically the TRE module, in Euclidean-spuen tasks, many robotic manipulation tasks involve non- non-Euclidean state spaces, which are harder to tackle. To verify the generality of method across these distinct
8

---

Published as a conference paper at ICLR 2026
Normalizud Score
Normalized Score
60
60
HIQL WaySteps=12
I[QL WaySteps=25
40
HIQL WaySteps=50
HIQL WayS:eps=75
20
20
I[QQ WaySteps=100
RD HRL
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Training Steps
Ivs
Training Steps
106
(a) antmaze-ultee caammaz-ultra-plee playa-play
(b) antmaze-ultra-divenn
Figure 3: On antmaze-ultraa(plat diverse) environments, RD-HRL's performance is compared with HIQL using different waysteps H. While the choice of waysteps H affects HIQL's performance, RD-HRL consistntty maintains its leading position.
geometrical settings, further compare RD-HRL and RD-HRL-TRE mixed, and CALVIN. The corresponding results are shown in Table
kitchen-partinn kitchen-
It can be observed that in non-Euclidean robotic manipulation tasks, directly us- ing $t42H decision-level target yields only limited performance gains over RD- HRL. This highlights the critical role of the transition region in such tasks. More- over, the performance degradation of RD- HRL-TRE on certain benchmarks (e.g.. CALVIN) further substantiates the neces- sity and effectiveness of the TRE module. Therefore, we conclude that extracting the transition region remains feasible and beneficial in non-Euclidean tasks.
Table 3: Comparison of RD-HRL and RD-HRL-TRE on manipulation tasks.
Datasets
RD-HRL- TRE
HIQL
RD-HRL
kitchen- kitchen-partial kitchen-mixed CALVIN
67.6 1
65.0 9.2
69.6 7.4
69.1 5.4
67.7 6.8
72.9 1.7
35.5 - 2.0
43.8 39.5
68.8 9.7
4.5 SUPPLEMENTARY EXPERIMENTS AND ANALYSIS
Beyond the aforementioned experiments, we performed supplementary studies, such as decision process visualizations and hyperparameter analyses, to validate the effectiveness of RD-HRL. Please refer Appendix for details.
RELATED WORKS
RD-HRL belongs to the narrow class of hierarchical reinforcement learning (HRL) methods. In broader sense, is also related to option- based RL methods.
The narrow class of HRL methods commonly designed two-level structure where a high- level planner selects sub-goals over a fixed horizon for low-level planners (Pateria al. 202 Barto and Mahadevan 2003 Botvinick] 2012). such PlanQD (Chen et al. 2024a), HIRO (Nachum ct al. al. 2018) and HD (Chen et al.). Such a commonly adopted design alleviates the credit assignment problem, while making sub-goal selection coarse and rigid. To address this, G-ADT Ma et al. (024) selects states with the highest attainable value rather than the expected value to promote exploration; DTAMP (Hong et al. 2023) generates the shortest sub-goal paths improve planning efficiency; RIS (Chane-Sant et al. 2021 and HILPs (Park et al. (2024b) choose intermediate states from value functions and Hilbert space (Young) 1988) representations, respectively. It worth noting that although some methods adopt fixed-level hierarchical designs with three or more layers (DiffuserLite (Dong et al., 2024) or HAC (Levy al. 2019) for example), all these methods still rely on noisy value functions for sub-goal selection, resulting in unreliable sub-goals.
Option-based methods exploit task semanties to decompose an overall task into semantically mean- ingful sub-tasks. For example, DDO (Fox et al.] (017) employs policy-gradient method under behaviour cloning to discover reusable segments (i.c., sub-tasks) from datasets, where each seg- ment can further contain finer-grained ones. Clustering-based approaches, such Evans and Simşek (2023), represent the dataset as graph and apply Louvain clustering (Blondel et al.) 2008) for discretisations, treating sub-clusters as low-level action candidates and higher-level clusters as high-level policies, thereby achieving layer-by-laeer decomposition without predefining the num- ber of options. Similarly, FraCOs (Cannon and Şimşek] 2025) identifies frequently recurring sub-

---

Published as a conference paper at ICLR 2026
sequences within successful trajectories callable options to build multi-level skill hierarchies, while (Hu et al., 2022) improves the efficiency of hierarchical discovery through causal graphs of environmental variables. Despite their ability to achieve semantical decompositions, option-based HRI methods often incur substantial training and planning costs, which motivates the development of low-cost HRI. approach with reliable sub-goals.
More generally, RD-HRL. can be viewed as related to trajectory-stitchin methods, while distin- guishing itself by providing reliable action-level targets through reliability-driven decision mech- anism. Please refer to Appendix H and for detailed discussion.
6 CONCLUSION AND DISCUSSION
In this paper, analyze the shortcomings of existing HRL methods when dealing with long-horizon sparse-reward tasks: they select action-level targets (i.e., sub-goals) based on conventional value functions that affected by generalization noise, which leads to sub-optimal trajectories. As the solution, we propose RD-HRL, which introduces the novel reliability-driven decision mechanism to select decision-level target from transition regions for high-level policy, thereby restricting the decision space of high-level policy to local regions without generalization requirements, yielding reliable action-level targets. Theoretical analysis and experimental results demonstrate the superior performance of our method RD-HRL.
Limitations and Future Directions. Although work has demonstrated promising performance on long-horizon sparse-reward tasks, the current learning process for transition regions Z remains relatively naive, relying exclusively on clustering. The decoupled execution of 2 extraction and policy learning inhibits flexible end-to-end joint optimization of regions and policies, thereby con- straining the method's overall performance, especially in nigh-ddmeesional observational tasks (e.g.. vision-based tasks). Developing an end-to-end method based on RD-HRL could further extend its applicability, representing crucial direction for future enhancements.
7 ETHICS STATEMENT
This research was conducted in accordance with established ethical standards for scientific work. Topies considered include, but are not limited to, involvement human subjects, dataset usage and release practices, potentially harmful insights, research methodologies and applications, con- flicts of interest and sponsorship, discrimination/bia/aaainne concerns, privacy and security issues, legal compliance, and research integrity (e.g., IRB approvals, documentation, and research ethics). Specifically, our study does not involve human subjects or personally identifiable information, and therefore Institutional Review Board (IRB) approval was required. All datasets used are publicly available and released under appropriate licenses. Potential risks, including fairesss, bias, privacy, and unintended harmful use of the findings, were carefully assessed, and steps were taken to min- imize such risks. We affirm that our work complies with research integrity guidelines, including accurate reporting, transparency, and reproducibility.
8 REPRODUCIBILITY STATEMENT
We have taken multiple steps to ensure the reproducibility of our results. The main text provides detailed descriptions of the model architecture and training procedure, while the appendix includes additional explanations of implementation details and hyper-parameters. All datasets used in our experiments are publicly available. Furthermore, we release the source code, configuration files, execution environment, and pre- pre-procsssing scripts in an anonymous repository, enabling researchers to faithfully reproduce our experiments.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grants No.62307020, Do.U2241229), the National Key R&D Program of China (Grant No.2023YFF09054000 the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (Grant W.YY22222XXXX0033, and the New Cornertoone Science Foundation through the XPL .ORERR PRIZE.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Abulikemu Abuduweili, Xingjian Li, Humphrey Shi, Cheng-Zhong Xu, and Dejing Dou. Adaptive consistency regularization for semi-supervised transfer learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6923-6932, 2021.
Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 9()):229,, 2020.
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training tn-the-willd device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461-12495, 2024.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning.
Discrete event dynamic systems, 13(4):341-77, 2003.
Vincent Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008 (10):P10008, 2008
Matthew Michael Botvinick. Hierarchical reinforcement learning and decision making. Current opinion neurobiology, 22(6):956-963) 2012.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540. 2016.
Michael Brusco and Douglas Steinley. A comparison of heuristic procedures for minimum within- cluster sums of squares partitioning. Psychometrika, 72(4):583-600, 2007.
Thomas Cannon and Özgür Şimşek. Accelerating task generalisation with multi-level skill hierar- chies. In The Thirteenth International Conference Learning Representations, 2025.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference on machine learning, pages 1430-1440. PMLR, 2021.
lan Char, Viraj Mehta, Adam Villaflor, John Dolan, and Jeff Schneider. Bats: Best action trajec- tory stitching. arXiv preprint 2rXiv:2204. 12026, 2022.
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. In The Twelfth International Conference on Learning Representations,
Chang Chen, Junycob Back, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Plandq: hierarchical plan orchestration via d-conductor and q-performer. In Proceedings of the 41st International Conference on Machine Learning, pages 6397-6412, 2024a.
Chang Chen, Deng Fei, Kawaguchi Kenji, Caglar Gulcehre, and Ahn Sungjin. Simple hierarchical planning with diffusion. In ICLR 2024, 2024b.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15004-15099 2021.
Dingsheng Deng. Dbscan clustering algorithm based on density. In 2020 7th international forum on electrical engineering and automation (IFEEA), pages 949-953. IEEE, 2020.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
Zibin Dong. Jianye Hao, Yifu Yuan, Fei Ni, Yitian Wang, Pengyi Li, and Yan Zheng. Diffuserlite: Towards real-time diffusion planning. Advances in Neural Information Processing Systems, 37: 12556-122583 2024.
11

---

Published as a conference paper at ICLR 2026
Khanh-Chuong Duong, Christel Vrain, et al. A filtering algorithm for constrained clustering with within-cluster sum of dissimilarities criterion. In 2013 IEEE 25th International Conference on Tools with Artificial Intelligence, pages 1060-1067. IEEE, 2013.
Anthony WF Edwards and Luigi Luca Cavalli-Sforza. A method for cluster analysis. Biometrics, pages 362-375, 1965.
Joshua B Evans and Özgür Şimşek. Creating multi-level skill hierarchics in reinforcement learning. Advances in Neural Information Processing Systems, 36:48472 48484, 2023.
Benjamin Eysenbach, Matthieu Geist, Sergey Levine, and Ruslan Salakhutdinov. A connection be- tween one-step rl and critic regularization in reinforcement learning. In International Conference on Machine Learning, pages 9485-9507. PMLR, 2023.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference robot learning, pages 158-168. PMLR, 2022.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv: hww.1703.08294 2017.
Lan-Zhe Guo and Yu-Feng Li. Class-imbalanced semi-supervised learning with adaptive threshold- ing. In International conference on machine learning, pages 8082-8094. PMLR, 2022.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning, pages 1025-1037. PMLR, 2020.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87-110, 2022.
Charles A Hepburn and Giovanni Montana. Model-based trajectory stitching for improved offline reinforcement learning. ar preprint arXiv:2211.11600 2022.
Mineui Hong, Minjae Kang, and Songhwai Oh. Diffused agnostic milestone planner. Advances in Neural Information Processing Systems, 36:387-405, 2023.
Xing Hu, Rui Zhang, Ke Tang, Jiaming Guo, Qi Yi, Ruizhi Chen. Zidong Du Ling Li, Qi Guo, Yunji Chen, et al. Causality--rivee hierarchical structure discovery for reinforcement learning. Advances in Neural Information Processing Systems, 35:20064-20076, 2022.
Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal artificial intelligence research, 4:237-285, 1996.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for Ilm reasoning through refined credit assignment. 2024.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Adaptive g-aid for conditional supervised learning in offline reinforcement learning. Advances in Neural Information Processing Systems,
2024.
Trupti M Kodinariya, Prashant Makwana, et al. Review on determining number of cluster in k-means clustering. International Journal, 1(6):90-95, 2013.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. Deep RL. Workshop NeurlPS 2021.
Seungjae Lec, Jigang Kim, Inkyu Jang, and H Jin Kim. Dhrl: A graph-based approach for long-horizon and sparse hierarchical reinforcement learning. Advances in Neural Information 2022.
Processing Systems,
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. In Proceedings of International Conference on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2026
Haoran Li. Yaocheng Zhang, Haowei Wen, Yuanheng Zhu, and Dongbin Zhao. Stabilizing diffusion model for robotic control with dynamic programming and transition feasibility, IEEE Trans. Artif. Intell., 5(9):45555)55555555 September 2024.
2024.
3387401
Yundong Li, Longxia Guo, and Yizheng Ge. Pseudo labels for unsupervised domain adaptation: A review. Electronics, 12(15):3325, 2023.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.0721 2017.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. cond-conni reinforcement learning: Prob- lems and solutions. arXiv preprint (iii:22201.08299 2022.
Yi Ma, Chenjun Xiao, Hebin Liang, and Jianye Hao. Rethinking decision transformer via hierarchi- cal reinforcement learning. arXiv preprint arXiv:231 02223111.00266 2023.
Yi Ma, Jianye Hao, Hebin Liang, and Chenjun Xiao. Rethinking decision transformer via hicrar- chical reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, pages 33730-33745 -3745,, 2024.
Andrzej Maćkiewicz and Waldemar Ratajczak. Principal components analysis (pea). Computers & Geosciences, 19(3):303-342, 1993.
Leland Mclnnes, John Healy, Steve Astels, et al. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(1 1):205, 2017.
Oier Mees, Lukas Hermann, Erick Rosete-Beas and Wolfram Burgard. Calvin: A benchmark for language-conditionee policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327-7335) 2022.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hicrarchical reinforcement learning. Advances in neural information processing systems, 31, 2018
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fermandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learn- ing robust visual features without supervision. Transactions on Machine Learning Research.
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Hierarchical reinforcement learning via addantagg--eiighed information maximization. arXiv preprint arXiv: 1901.01665....... 2019.
Scohong Park, Dibya Ghosh, Benjamin Eysenbach, and Sergey Levine. Hiql: Offline goal- conditioned with latent states as actions. Advances in Neural Information Processing Systems, 36, 2024a.
Scohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. In Proceedings of the 41st International Conference on Machine Learning, pages 39737-39761, 2024b.
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1-35, 2021.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1 11577-11568, 2021.
Eduardo Pignatelli, Johan Ferret, Matthicu Geist, Thomas Mesnard, Hado van Hasselt, Olivier Pietquin, and Laura Toni. A survey of temporal credit assignment deep reinforcement learning. arXiv preprint arXiv:2312.0107. 2023.
Yunpeng Qing, Shunyu Liu, Jingyuan Cong, Kaixuan Chen, Yihe Zhou, and Mingli Song. A2po: Towards effective offline reinforcement learning from an advantage-awes awarre perspective. Advances in Neural Information Processing Systems, 97-29064-29090, 2024.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An ancertainty-wwan pseudo-label selection framework for semi-supervised learning. arXiv preprint Stt::2101000322 2021.
13

---

Published as a conference paper at ICLR 2026
Grigory Sapunov. Deep learning with jax. 2023.
David Sculley. Web-scale k-means clustering. In Proceedings of the 19th international conference on World wide web, pages 1177-1178, 2010.
Congming Shi, Bingtao Wei, Shoulin Wei, Wen Wang, Hai Liu, and Jialei Liu. A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm. EURASIP journal on wireless communications and networking, 2021:1-16, 2021.
Wonchul Shin and Yusung Kim. Guide to control: offline hicrarchical reinforcement learning using subgoal generation for long-horizon and sparse-reward tasks. Proceedings of the Thirty-Second International Joint Conference Artificial Intelligence, pages 4217-4215 2023.
Avi Singh. Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning. arXiv preprint arXiv:2010. 14500, 2020.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning. Journal of Cognitive Neuroscience, 11(1):126-134) 1999.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
Siddarth Venkatraman, Shivesh Khaitan, Ravi Tej Akella, John Dolan, Jeff Schneider, and Glen Berseth. Reasoning with latent diffusion in offline reinforcement learning. In The Twelfth International Conference on Learning Representations.
Mianchu Wang, Rui Yang, Xi Chen, Hao Sun, Meng Fang, and Giovanni Montana. Goplan: Goal-conditioned offline reinforcement learning by planning with learned models. arXiv preprint arXiv:2310.20025, 2023.
Ruosong Wang. Simon S Du, Lin Yang, and Sham Kakade. Is long horizon rl more difficult than short horizon rI? Advances in Neural Information Processing Systems, 33:9075-9085, 2020.
Muning Wen, Ziyu Wan, Jun Wang, Weinan Zhang, and Ying Wen. Reinforcing Ilm agents via policy optimization with action decomposition. Advances in Neural Information Processing Systems, 37:103774-103805, 2024.
Marco A Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning, and optimization, 12(3):729, 2012.
Chengjie WU, Hao Hu, Yiqin Yang. Ning Zhang, and Chongjie Zhang. Planning, fast and slow: online reinforcement learning with oction-free offline data via multiscale planners. In Forty-first International Conference on Machine Learning. 2024.
Dong-Dong Wu, Deng- Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In International conference on machine learning, pages 24212-24225. PMLR, 2022.
Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in neural information processing systems, 36:18 36:18532- 1511881855500 2023.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in neural information processing systems, 35:4085- 4098, 2022.
Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, pages 38989-39007. PMLR, 2023.
Nicholas Young. An introduction Hilbert space. Cambridge university press, 1988.
14

---

Published as a conference paper at ICLR 2026
Ruiyang Zhang, Yang Liu, and Hao Sun. Physics-guide convolutional neural network (phyenn) for data-driven seismic response modeling. Engineering Structures, 215:110704, 2020.
Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefenstette, Yuandong Tian, et al. Efficient planning in compact latent action space. In The Eleventh International Conference on Learning Representations.
Ziqi Zhang, Jingzehua Xu, Jinxin Liu, Zifeng Zhuang, Donglin Wang, Miao Liu, and Shuai Zhang. Context-former: Stitching via latent conditioned sequence modeling. arXiv preprint
arXiv:2401.14435 2024.
Meng Zhou, Ziyu Liu, Pengwei Sui, Yixuan Li, and Yuk Ying Chung. Learning implicit credit assignment for cooperative multi-agent reinforcement learning. Advances in neural information processing systems, 33:1 11111883--111 11864, 2020.
Zhaoyi Zhou, Chuning Zhu, Runlong Zhou, Qiwen Cui, Abhishek Gupta, and Simon Shaolei Du. Free from bellman completeness: Trajectory stitching via model-basee return-conditioned super- vised learning. In The Twelfth International Conference on Learning Representations.
15

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We employ Large Language Models (LLMs) for grammar checking in our paper.
B PSEUDO CODE OF RD-HRL
Algorithm Training
1: Extract transition regions 2 with the TRE module.
2: Initialize Vo,, TE Tloy "0,, "0,-
3: while not converged do
4: # Learning of TI Té Tg..
5: Update O27 with Equation (10).
6:
Update 9, with Equation
7:
Update 0 with Equation
8:
# Learning of Voy TF
9:
Update By with Equation
10: Update OTE with Equation
end while
Algorithm Evaluating
Initialize environment env, obtain current state 8 and goal g-
2: done + False
3: while not done do
4: # Obtain decision-level target gf with Tler, (g
8t, g)
5: 9." +- Tl (- 8 (e.9), g).
6: # Obtain action-level target g" with rbe (g")s,.g"')
7:
g + n/i, (+|a,.g) 9. ".
8:
# Obtain action a with Tb (arlst, g*))
9:
+ Th. (+1s.,g²))
# Interact with environment env.
11:
state 81+1, reward, done env(a,)
12: end while
C DETERMINE OF NUMBER OF CLUSTERS N
Many studies have proposed pioneering methods for determining the optimal number of clusters in K-Means (Kodinariya al. 2013 Shi et al. 2021 Zhang et al.) (020). In this paper, determine the optimal number of clusters by computing the inflection point of the Within-Cluster Sum of
Squares (WCSS) (Edwards and Savalli-Sforza [9655 Brusco and Steinley 2007 Duong 2013)
of various numbers of clusters, in which WCSS is deñiedd as:
WCSS = Σ Σ 1
(12)
CEC.
where C' is the set of clusters and C, E C. WCSS calculates the sum of squared distances from all samples to their respective cluster centroids; a smaller WCSS value indicates that the samples are closer to their cluster centers, signifying better clustering performance. Consequently, the optimal cluster number can be conveniently selected by evaluating WCSS values across varying cluster numbers.
Specifically, for each dataset, we define its candidate set for the number clusters as {NN, N2, ..., J. where Ni < N2 < N,. For fixed dataset, N increases, the decrease WCSS is inevitable because each cluster becomes smaller, thereby reducing the distance between samples and their cluster centroids. However, this process is not linear. When the number of clus- ters N is smaller than the optimal number of clusters Ñ, WCSS decreases rapidly; whereas
16

---

Published as a conference paper at ICLR 2026
when N exceeds N, the rate of WCSS reduction slows down. Consequently, the optimal number of clusters N can be determined by detecting the point where the WCSS reduction rate exhibits notable transition. In our paper, we select N as N which has the maximum third-order derivative, as it represents where the WCSS trend exhibits significant change.
D THEORETICAL PROOF FOR THE ADVANTAGE OF THE TARGET EVALUATION MODULE
Proposition 1. The Target Evaluation module yields lower accumulated noise.
Proof of Proposition 7 Under long-horizon sparse-reward settings, the credit assignment issue tends to propagate noise in value function updates. Let us hypothesize that, for the ground-truth value V (sx), every update incorporates Gaussian noise scaled by B. Assuming the value function after -teepp updates (8:-N), we can now quantitatively analyze the noise introduced by N-step updates by caleulating the entropy of (s)-N) and (s²).. In other words, h h(V(s,-x) (s)(s))) quantifies the amount of noise introduced. For ease of presentation, we omit the condition and parameters 0 in this proof.
First of all, we need the analytical form of ((*-NN). For state 8t, we have:
V(s;) = -V + + + (V(441))
(13)
where e, is independent Gaussian noise, e N(0, 1). As and are fixed parameters, denote Y a,3. Then, we have (( for N N22 2:
(s -V(())-N+)++ + + (V -aVV + + 3V N
(14)
(15)
(aB +
T
(16)
(a.3 +
Bc- V(s:-N+2) + + +
(17)
(a. + (BB + 34-++1) ((8 + (aB (BG-N) T--
Pt-N
(18)
= =aa + (a + BEI + - ((8tNN+3)
(19)
(aB + N++ Tex+2 (e + BEc-N+1) TT-N+1
(20)
+
(21)
(22)
V(ss) (a.3 + N4*) k=0
(23)
N-1
+ Σ [hej 11 No N-j- (a. + BeI-
(24)
1=1
Pt-N
(25)
VI (aB + 3 (-N+)).
(26)
Note that considering the long-horizon sparse-reward settings, consider 0 for simplicity.
The Gaussian process with multiplicative noise leads computational complexity. For simplicity, since the logarithmic transformation log(-) is injective, which does not affect the differential entropy of the distribution, given Equation (26). we have:
log(V log((V
Σ log(a3 B. (x-N+k
(27)
= =ogg(V(s:)) + +N((N - 1)a3, (N 1):32).
(28)
Assuming V(s,) = N (N,,, o7), we have V(s,-N) = NN4 -1)a8,88,80 of (N - 1)33). Then have the differential entropy of log(V(s::) and log( (8)+N)) as:
17

---

Published as a conference paper at ICLR 2026
1 h(log(V(s))))
1 0
Flog(2Teo7 + log((e) 2
(29)
14
of (N 1),32
(SLx)(V( =
(NN-))
(pt + (N 1)a
(30)
With Equation 299 and Equation 30 we have the information of noise introduced by the plain
value function
= h(log(V - h(logg
(31)
Meanwhile, with our temporal abstracted value function TE, the steps required to update from TE(s;) to TE(S++N required N' < N steps, as the temporal abstracted value function TE is learned on the sketleon of trajectories. Although, similarly, we have the information of noise intro- duced by our temporal abstracted module TE as:
TE =
- h(log(V(ss)))
(32)
where (((o((T((++)))) is the differential entropy of logged updated temporal abstracted module TE. Then have:
VITE v
[n(log (T(()(((()+))))) -
[M(log(V((++ - (M(lo((V(s)))) (33)
(((og(TE(8(xx))) (((og(((8(xN)))
(34)
of (N' 1).32
2 logg(22e(00) + (N' - -1)88)) + log(pe + -
(14 + (N' 1)a
(35)
(00g(22ee00 + (N- 1),31) + log(m (N- (1)))
of + (N - 1),32
(m4 + (N 1))
(36)
= 2 1lgg(2ee(o) + (N' - 1))3)) 1
2 (log(2 + (N- -1)32))
(37)
+ 1oogmu + (N' - 1)a33) - -og(Hi + (N 1))
(38)
03 + 2 (Mt (N - 1)0;332
03
(39)
(Mt (N" 1)0,3y2
+ 1 (N -1)32
(N' - -1),32
(40)
(M (N 1))3333
(m (N' - -1)a3)22
As No < N, we have:
+
- + (N ((N--133)) < 0,
(41)
2
log(ue + (N' 1))B3) log(A + -N-1)aa8
- 1)a3) < O,
(42)
o²
of
(m + (N 1)a3)2 0.
(43)
(mg (N 1)a3)2
For the last factor, define
18

---

Published as a conference paper at ICLR 2026
N
S(N)
N 20.
(44)
(1r + (N - 1)a
Then we have:
af
(e + (N - 1)a.3)2 - 20,3(H4 + (N 1)) -
(45)
ON
(14 (N - 1)a3)4
(nt + (N 1)0,3) 20x - 1)
(46)
(Me + (N - 1)a:33
14 - (NN - 1)
(47)
(14 + (N 1)o33)3
Note that 7 = a,3 is set close to in RL, and under the sparse reward setting, we have S ue < 0f thus 2N
< 0, (NN ) is monotonically decreasing. Further, with N' N, we have:
1 (N - 1),32
322 (ee + (N 1)a33² (mu (N' 1)a332 (f(N) ((N-))(N))))))))))))))))) <
(N' - 1)
(48)
Considering Equation (40). Equation 411. Equation (42 Equation
43 and Equation (88 together,
we now have T/TE
'/V 0. conclusion, the Target Evaluation (TE) module yields lower noise.
E ANALYSIS OF DECISION-LEVEL AND ACTION-LEVEI TARGET PREDICTION
To better illustrate the Target Identification (TT) module in RD-HRL, taking untmaze-ultradiivers as an example, we show how the decision-level target steers the agent toward the final target in 6 and show how the high- and low-level policies accomplish the decision-level target in Figure 5 We have also visualised trajectories in the dataset, and the filtered transition regions 2 of antmaze- diverse in Figure 4 As be observed in Figure O6 the TI module generates decision- levell targets g from transition regions Z, guiding the high- and low-level policy towards the task goal. Diving into Figure [5]
observe that the agent achieves action-level target under the guidance of high-level policy; after the agent achieves the decision-level target, the TI module produces new decision-level targets.
F EXPERIMENTAL ENVIRONMENT AND HYPERPARAMETERS
RD-HRL is trained using Flax under JAX (Sapunov 2023 on an Ubuntu 22.04 LTS server, with 4 x NVIDIA A40 (Ampere architecture, 48GB VRAM each), 72-core processor (dual-socket Intel Xeon Platinum), and 503GB memory.
We design the TI module, TE module, value function, high-level and low-level policy as MLPs. The hyper-parameters are summarized in Table 4 please refer Table 4 for details.
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 PRELIMINARY ATTEMPT VISUAL SCENARIOS
RD-HRL has demonstrated significant advantages in goal-conditioned tasks and robotic manipula- tion tasks. Nevertheless, scaling RD-HRL to visual scenarios is promising idea.
We believe this can be solved in two ways: (1) As discussed in Section 6 we could try to de- sign the pipeline as an end-to-end process. In this scenario, filtering transition regions could be performed resk-relevant embeddings generated by jointly-leamed encoder. Or, more generally, (2) we could attempt to use a general large models to understand high-dimensional observations and generate universal embeddings, then perform clustering these embeddings. However, these
19

---

Published as a conference paper at ICLR 2026
Initial State
Action-lovel Target g²
Histryy Trijeclory
Current State
Decision level Target
g"d
Future Trajestory
Goal
G
G
*
Figure 4: Trajectories in Figure 5: After the low-level policy achieved the decision-level target, dataset (upper) and ex- the TI module produces decision-level targets by sampling from further tracted transition regions transition regions. (lower).
Table 4: Detail of paraaee-srameters We set the other hyperparameters to be consistent with those HIQL Park et al. (2024)).
Hyperparameter
Value
Value discounts
0.99
Goal dimensions
10 (kitchen, CALVIN), 29 (Antmaze)
Training steps
1000000
Batch size
1024
TI module dimensions
(256, 256)
TI module dimensions
(512, 512, 512)
Policy MLP dimensions
(256, 256)
Value MLP dimensions
(512, 512, 512)
Representation MLP dimensions
(512, 512)
Activation
GELU
Optimizer
Adam
Learning rate
0.0003
Target network decay rate
0.005
embeddings might be task-agnostic, and this approach suffers from the drawbacks of separate exe- cution discussed in Section 6 Therefore, we believe solution (1) is more promising direction. We will leave solution (1) as our future work.
Nevertheless, in this section, present preliminary attempt apply RD-HRL to challeng- ing visual scenarios. Specifically, we proposed the following variants and evaluated their perfor- mance visual environment, procgen-500 (Park et al. 2024a):
20

---

Published as a conference paper at ICLR 2026
Initial State
Goal
Curnent State
Actoo-loval Target
Target gf
Histryy Trajectory
Puture Trijectory
*
*
Figure 6: By generating decision-level targets, TI module coordinates the high-level and low-level planning processes.
RD-HRL*; Uses embeddings from DINOv2 (Oquab et al.) for discretization and filtering of trajectory-adjacent regions. DINOv2 pretrained Vision Transformer (ViT) (Han al. 2022 model. The results of RD-HRL" serve to validate the performance of RD-HRL in general embedding scenario.
RD-HRL Uses embeddings from HIQL for discretization and filtering of transition re- gions. As HRL, method, the encoder of HIQL produces task-relevant embedding; there- fore, the results of RD-HRL** preliminarily verify the performance of RD-HRL in task- relevant embedding scenario. Note that we use PCA (Maćkiewicz and Ratajezak 1993) to reduce the size of HIQL's embedding to match that of DINOv2's embedding before performing transition regions extraction.
RD-HRL: After flattening the raw RGB observations, we apply PCA to reduce their size to match that of DINOv2's embeddings before performing transition regions extraction.
The results are summarized Table 5
Note that Performance (train) refers to the performance on the environment levels used for training the policy, whereas Performance (test) refers to the performance on the environment levels used for testing the policy.
Table 5: Results of RD-HRL", RD-HRL and RD-HRL on visual task procgen-500.
Methods
RD-HRL
RD-HRL
RD-HRL ct
Performance (train)
3.5 2.2
11.5
5.1
+ 3.9
Performance (test)
0
2 1.7
7.5 + 4.3
It can be observed that both at the train level and the test level, when the transition region extraction is performed directly on the observation space, it is hard for RD-HRL to complete the tasks. However, when the transition region extraction is performed directly the general embeddings (RD-HRL"), RD-HRL achieves a certain success rate, indicating that general embeddings can help alleviate the limitations of RD-HRL in high-dimensional observation scenarios. Moreover, RD-HRI.** achieves
21

---

Published as a conference paper at ICLR 2026
Table 6: The average normalized score different GCRL methods on various environments, with + denoting the standard deviation. The mean and standard deviation are computed over 50 random seeds. We emphasize the best scores of each task in bold.
Datasets
GCBC
GC-POR
GC-IQL
RD-HRL
antmaze-mediim-livves
67.3+10.1
74.8 + 11.9
63.5 + 14.6
94.6+
antmaze-medium play
71.9+16.2
71.4
70.9 11.2
94.011.2
antmaze-large-diverse
20.24.9.1
49.0 17.2
50.7 -..7-181 18.1
91.3++4.3
antmaze-large-plyy
23.1+15.6
63.2+16.
56.5 + 14.4
95.3++211
uutmaze-ultra-diverss
14.4119.7
2
1.6:1550
81.1
antmaze-ultra-play
20.7449.7
31.0 19.4
29.8 + 13.6
72.94-5.1
kitchen-pe sacchen-partial kitchen-mixet mixed
38.5+11.8
18.4 - 14.3
39.2 + 13.5
69.617..4
46.7:20.1
27.9 = 17.9
51.3 = 12.8
1..9
CALVIN
17.3+
12.4 + 18.6
7.8 + 17.6
68.8++9.7
better results than RD-HRL* demonstrating that task-relevant embeddings can further address this limitation.
In summary, although this is only a rough initial validation, the results offer preliminary support for our future work aimed at enhancing RD-HRL's capability in high-dimensional settings, demonstrat- ing the feasibility of our future work.
G.2 COMPARISON OF RD-HRL WITH FLAT GCRL METHODS
We additionally evaluate RD-HRL against conventional GCRL approaches, including GCBC DDing al., 2019). GC-POR (Xu al. 2022). and GC-IQL (Kostrikov et al.] Park al. 2024a). The results are summarized Table 6 As can be observed, RD-HRL outperforms the baselines on all of the tasks, demonstrating the advantages of RD-HRL.
G.3 How ABOUT APPLYING THE TARGET EVALUATION (TE) MODULE ON HIGH-LEVEL. PLANNER, INSTEAD THE ADDITIONAL TI MODULE?
We introduce TI module trained with a Target Evaluation (TE) module that provides low-noise value estimates for decision-level target selection, yielding promising results. However, such design raises a natural question: Can the TE module be applied directly to the high-level planner in standard two-level HRL architecture, and obviate the TI module?
Before addressing this question further, we would like to clarify a crucial point: the module and TI module complementary components that operate synergistically, the TE module is fundamen- tally designed to estimate the value of transition regions, not for any states the dataset. However, the high-level planner requires ((s, g) for any S and any g G, which is capability that exceeds the scope of the TE module.
Nevertheless, conducted experiments and propose variant RD-HRL-TI which directly applies the TE module to the high-level policy hh Results are summarized in Table As can be observed, compared HIQL, directly applying the TE module z² brings only negligible performance improvement, which is far inferior to applying the TE module to the TI module. This suggests that combining the TE module with the TI module more reasonable design.
Table 7: Comparison among HIQL, RD-HRL and RD-HRL-TI.
Datasets
HIQL
RD-HRL-TI
RD-HRL
antmaze-ultra-play
39.2 + 14.8
40.0 8.9
72.9 4 5.1
autmaze-utra-diverree
52.9 + 17.4
54.7 4.2
81.1 1 6.3
22

---

Published as a conference paper at ICLR 2026
G.4 How SENSITIVE IS RD- ED-HRL TO THE NUMBER OF CLUSTERS?
To further explore the impact of cluster number N RD-HRL, evaluate RD-HRL with various values of .. The results are summarized Table As shown, the performance improved as the number of clusters increased, peaking at N=60. Further increasing the number of clusters led to performance decline. We believe this is because: (a) With fewer clusters, the divisions of clusters were too rough, preventing the TI module from accurately learning the accurate decision-level target. (b) As the number of clusters grew, their size decreased. This resulted in more clusters having high FDI, which obscured the advantage of true transition regions over other regions.
Table 8: Performance of RD-HRL with various N.
Datasets
= 10
= 20
N 40
N 60
N = 80
= 100
antmaze--llre
53.3 = 7.3
52.8 2.2
66.2 + 3,9
72.9 +
59.1 + 1.2
55.7 = 2.8
uutmazo-ultra-diverr
52.0 = 9.2
69.2 + 1.1
77.0 + 2.9
81.1 + 6.3
73.1 + 2.6
59.7 + 3.3
G.5 Is THE TRAINING OVERHEAD INCURRED BY THE TI MODULE AND THE TE MODULE JUSTIFIABLE?
To evaluate the acceptability of these additional computational costs introduced by the TI module and the TE module, we measured the training time consumption of HIQL and RD-HRL across different datasets, is summarised in Table It can be observed that the introduction of the TI module and the TE module indeed brings about decline in training efficiency; however, even for the slowest learning speed of 82 iterations per second with a batch size of 32 on antmaze-ultra-play, RD-HRL only requires 3.4 hours to learn 10 steps, meaning RD-HRL remains efficient.
In summary, although the introduction of the TI module and the TE module increases computational overhead, the overall execution time of RD-HRL remains within an acceptable range, thanks to the efficiency of the JAX framework.
Table 9: Comparison of training effiency of HIQL and RD-HRL.
Datasets
HIQL (iterations/s)
RD-HRL (iterations/s)
RD-HRL
(1M steps/h)
antmaze-ultra-play
257
82
3.4
untmaze-ultra-divesse
261
84
3.3
antmaze-large-play
249
87
3,2
antmaze-large-divenss
253
79
3.5
antmaze-medium play
255
88
3.1
antmaze-medium diverse
262
83
3,3
kitchen-mixed
311
129
2.1
kitchen-partial
315
132
2.1
CALVIN
326
125
2.2
G.6 INVESTIGATIONS OF ALTERNATIVES OF K-MEANS
To explore clustering methods beyond K-Means (Ahmed al. 2020 we experimented with the density-based method DBSCAN (Deng) 2020) and the graph-augmented method HDB- SCAN (Meinnes et al.) (017). and visualized their clustering results in Figure 7 As is shown, the clustering results of both DBSCAN and HDBSCAN turned out to be chaotic, making it impossi- ble for us to infer transition regions or conduct further experiments based on those cluster methods. We believe this is because that DBSCAN and HDBSCAN assume that clusters consist of density- connected regions, but trajectory data typically form sparse and non-uniform structures, making density-based assumptions unreliable.
23

---

Published as a conference paper at ICLR 2026
(a) DBSCAN
(b) HDBSCAN
Figure Cluster results of antmaze-ultra-divvrse with DBSCAN and HDBSCAN.
Table 10: Comparison of HGCBC and RD-HGCBC.
Dataset
HGCBC
RD-HGCBC
antmaze-ultra-dveen
39.4
42.0
antmaze-ultra-play
38.2
49.8
G.7 APPLICATION OF RD-HRL ON HGCBC
To verify whether the reliability-driver decision mechanism proposed in RD-HRL also benefit other methods, we incorporate the ediability-driver decision mechanism into HGCBC and propose RD-HGCBC. The results are summarized in Table I0, As shown. RD-HGCBC achieves clear im- provements on both antmaze-ultra-pay and animaze-ulraadivenss with a particularly notable gain of 30.4% on antmaze-ultra-play. This demonstrates the applicability of the RD framework other algorithms.
G.8 THE IMPACT THE FDI THRESHOLDING OF THE PERFORMANCE RD-HRL.
We have also evaluated how different FDI thresholding choices affect model performance, and summarize the results in Table I1 As can be observed, as the FDI threshold becomes more per- missive, the model's performance gradually degrades. The drop is particularly pronounced when the FDI x N decreases from to 2. This is because most non-transition regions have two future clusters; thus, when we relax the FDI threshold from 3 2, the model exposed to a large number of noisy transition regions, which leads to the observed performance degradation.
Table 11: The impact of the FDI thresholding.
FDI N +
2
antmaze-ultra-play
56,6
60.3
72.9
untmaze-ulltra-iive
62
64.3
81.1
24

---

Published as a conference paper at ICLR 2026
G.9 ANALYSIS OF TRANSITION REGIONS IN MANIPULATION TASKS
We further visualize the transition regions extracted from the kitchen-mixed environment in Figure to demonstrate the soundness of RD-HRL in manipulation tasks. As is shown, the frames in which the robot arm is positioned between the table and the cabinet are marked as transition regions. This is because the trajectories in kitchen-min mixed follow a consistent ordering: the agent typically manipu- lates the kettle and microwave on the table first, then moves on the light switch and burner bottom in the middle, and finally opens slides the cabinet doors at the top. Consequently, the intermediate frames, where the arm is located between the table and cabinet, serves as a state that connects tasks on the table (kettle and microwave) with tasks on the top (light switch, burner bottom or the cabinet doors). This illustrates that our method effectively identifies transition regions in manipu- lation tasks. Furthermore, the results in Table provide evidence of our method's ' performance manipulation benchmarks.
G.10 ANALYSIS OF SELECTED TRANSITION REGIONS AND DOWNSTREAM TASK SUCCESS
We computed the probability distribution over the transition regions selected by TI across 3,500 trajectories, and we also measured the normalized score associated with each selected transition region. The results are summarized in Figure 9 in which the the horizontal axis represents the IDs of the 18 extracted transition regions, labeled sequentially from 0 to 17; the vertical axis shows the probability of being selected after passing through given transition region and the average normalized score obtained after passing through that transition region, respectively.
can be observed that the transition regions chosen by the TI module correspond to high success rates, demonstrating that the TI module's selection of transition regions is positively correlated with downstream task success. In addition, transition region 14 provides a representative example of a low-return transition region. Its extremely low selection probability indicates that our method is capable of avoiding transition regions that yield low success rates for the current state. In other words, TI's selection of transition regions is closely related the success of downstream tasks.
G.11 IMPACT OF FDI ON TRANSITION REGIONS EXTRACTION
Theoretically, by connecting more trajectories, transition regions typically admit a larger set of pos- sible future directions. To further illustrate the role of FDI in identifying transition regions, we visualize the transition regions selected under different FDI values as is shown in Figure In sum- mary, using FDI regions.
(i.e., FDI >> 0) as the selection criterion yields more reasonable transition
G.12 EXTENDING RD-HRL TO ONLINE REINFORCEMENT LEARNING
We have also transferred the RD-HRL to HAC (Levy et al. 2019) and propose RD-HAC. The results summarized in Figure 11 As shown, in the early and middle stages (700k 1500k steps), RD-HAC exhibits significant advantage. (During the first 0-700k steps, the replay buffer still sparsely populated. As result, the RD mechanism cannot yet fully demonstrate its advantages.) As exploration progresses (after 1500k steps), the performance of RD-HAC and HAC becomes similar. However, RD-HAC achieves the best success rate of 91 % 1200k steps, whereas HAC reaches its best success rate of 89% at 1900k steps, demonstrating that the RD mechanism still shows its advantage in the online setting.
We believe this stems from RD's high-level understanding of the environment and its efficient use of samples. In the early and middle stages, the replay buffer collected by HAC during exploration is insufficient to support thorough understanding of the environment, leading to the possibility of suboptimal decisions. In contrast, in RD-HAC, by extracting transition regions, the introduced RD mechanism enables it to have comprehensive understanding of the environment even with a limited number of samples. As a result, RD-HAC outperforms HAC in the early and middle stages. However, as the samples collected by HAC during exploration become more abundant, the performance of HAC should improve faster, making the advantages brought by RD may no longer be pronounced.
25

---

Published as a conference paper at ICLR 2026
Figure 8: Example of transition regions in Sitchen-mixed
26

---

Published as a conference paper at ICLR 2026
80
Probability
Normalized Score
60
0.075
Probability
Normalized
0.050
20
0.000
3
7
10
11
12
13
14
15
16
17
Figure 9: Probability of transition regions and normalized scores achieved after each transition region antmaze-ulrra-divere.
aa) X 0
(b) N
(c) FDI -2
Figure 10: transition regions extracted with different FDI thresholds.
H THE STITCHING CAPABILITY OF RD-HRL
Prior work has suggested that an optimal value function can naturally achieve trajectory stitching. as it implicitly organizes the dataset into graph structure (Char et al. (022). However, this optimality assumption is often unrealistic, especially in long-horizon sparse-rewards tasks (Wen et al.) 2024] Kazemnejad al., 2024). Consequently, the cumulative noise and generalization noise flat value functions hinder existing HRL methods from performing reliable policy-level trajectory stitching.
Fortunately, by introducing both the TI module and the TE module, RD-HRL provides principled guarantee of trajectory stitching. On the one hand, the TI module constructs a decision domain for
RD-IIAC - IAC
0.25
Training Steps
400%
800k
1200k
1000k
2000
Figure 11: Comparison of HAC and RD-HAC on ant-fo.
27

---

Published as a conference paper at ICLR 2026
the high-level and low-level policies that is free from accumulated errors and generalization noise. Within the local decision domain partitioned by the decision-level targets, the conventional value function remains optimal for the high-level and low-level policies. On the other hand, the TE module provides the Tl module with a decision basis that is likewise robust against accumulated errors and generalization noise. Overall, the design of RD-HRL enables each policy to make decisions guided by (locally) optimal value functions, thereby ensuring reliable policy-level stitching.
policces.
I DISCUSSION OF UNRELIABLE SUB-GOALS
A large body of work can serve as indirect support for unreliable sub-goals. Take Figure I an ex- ample, unreliable sub-goals arise from unreliable value functions, which in turn stem from the value functions' reliance on generalization. This introduces indirect supervision signals for s in effect. For instance, in pseudo-labeling approaches (Pham al., 2021 Rizve et al., 2021; [L et al. (023), pseudo-labels are regarded as generalized supervisory signals, and numerous studies employ consis- tency regularization to reinforce the dominance of direct supervision in model learning (Abuduweili et al. 2021] Wu et al.. 2022]. or use thresholding of pseudo-labed to directly discard low-confidence
generalized signals (Guo and Li 2022). This implies that the signals from direct samples are more reliable than those from indirect samples (pseudo labels).
J DISCUSSION OF RD-HRL AND OTHER STITCHING METHODS
As is discussed in Appendix H our method is also analogous to trajectory stitching methods in broader sense. Here, we provide a detailed analysis of trajectory stitching methods and discuss in detail the differences and connections between RD-HRL and these methods.
To better leverage offline data and approach the dataset's optimal policy, many works focus on tra- jectory stitching. Some methods operate at the dataset level for data augmentation. For example, BATS (Char et al. (2022) trains a tabular MDP on collected data and uses learned dynamics models to generate short connecting trajectories, which are then used to train the policy. To overcome the lim- itations of Bellman completeness, MBRCSL (Zhou et al.) applies dynamic programming to stitch together segments from distinct trajectories without relying on Bellman completeness. Other ap- proaches focus on stitching subsequences to construct higher-quality trajectories, such as TS (Hep- burn and Montana, 2022). which searches for candidate next states that lead to higher returns. In addition, Venkatraman et al. enhances multi-modal data modeling by imposing batch constraints on out-of-distrribttio (OOD) data, combined with diffusion to achieve embedding- augmentation.
Other methods enable trajectory stitching at the policy level by introducing additional policy de- signs. Focusing on the stitching ability value or Q-functions, COG-RL (Singh et al.) 2020) em- ploys model-free dynamic programming based Q-learning stitch together different skills, while QCS (Kim et al. 2024) builds on this idea by introducing an assistance function as a regularization term improve Q O-guided stitching. With the rise of sequential decision decision-making approaches, sev- eral works have attempted trajectory stitching through the Decision Transformer (DT) (Chen et al,] 2021]. For example, EDT (Wu et al., 2023) enhances stitching by learning maximum achievable returns. ADT (Ma et al., (024) further combines a hierarchical learning framework, where higher levels provide value or goal prompts lower levels improve stitching. However, ADT still does not explicitly design for stitching, instead relying on the value function for implicit stitching. Simi- larly, QDT (Yamagata et (023) augments DT's stitching capability by leveraging Q-networks to relabel the return-to-ge to-un-to-go--o go (RTG).
Our method RD-HRL broadly falls into the policy-level stitching category. Similar to existing fixed- level HRL methods, the stitching ability of policy-level stitching also relies on the value function, which highly susceptible to the noise introduced by long-horizon sparse-reward setting, as is discussed in Section [ The noise immunity provided by the TI module and the TE module distin- guishes RD-HRL from these methods.
28


Reviews:
[{"review_id": "TsOdoxmB4o", "reviewer": "Reviewer_DegB", "summary": "The paper proposes RD-HRL, a hierarchical offline reinforcement learning method that aims to make sub-goal selection more reliable in long-horizon sparse-reward problems. It first clusters states to extract “transition regions” with a Future Diversity Index, then trains a Target Evaluation (TE) module that estimates values only over these regions and a Target Identification (TI) module that selects a decision-level target (a transition region) given the current state and goal. At evaluation time, TI provides a decision-level target to the high-level policy, which outputs an action-level sub-goal for the low-level policy to execute. Experiments on antmaze, kitchen, and CALVIN benchmarks report strong average performance and ablations analyze the contribution of each component.", "questions": "1. Could the authors clarify the intended role of $\\beta$ in Eq. (10)? Is it a temperature parameter (as in AWR) or related to the discount factor $\\gamma$? If the exponentiation form $\\beta^d(s_t, s_z)$ is intentional, what is the theoretical or empirical justification for raising $\\beta$ to the power of distance?\n2. What exactly does $g_{TI}$ represent at evaluation time - a region index, a representative state, or a learned embedding of the region? How is $\\pi_h$ conditioned on this variable in practice?\n3. There are mismatches between text, pseudocode, and tables (e.g., Algorithm 1 labels, Table 1 vs. Table 3). Could the authors reconcile these discrepancies and clarify whether they affect experimental results?\n4. Since the method relies on K-Means and Euclidean distances, how well does it scale to high-dimensional or non-Euclidean spaces? Have the authors tried graph-based or density-based alternatives, and if so, how do results compare?\n5. In Eq. (8), temporal distance $d(s_{t1}, s_{t2})$ is approximated across trajectories. How sensitive are TE’s estimates to this approximation, and does it introduce bias or variance in value learning?\n6. The paper includes several ablations, but could the authors include one specifically isolating the impact of the FDI thresholding and clustering scheme to show its independent contribution to performance?\n7. How frequently does TI select each transition region during evaluation, and does this selection correlate with TE’s estimated reliability or downstream task success?\n8. Why do the authors claim the robotic tasks in CALVIN benchmark are non-euclidean? Could they clarify what makes these non-euclidean?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Clear problem motivation: The paper articulates how value-function generalization noise can mislead sub-goal selection in hierarchical methods and uses a didactic example to illustrate the issue.\n- Architectural novelty: Introducing a layer that constrains high-level decisions to transition regions/bottleneck states and learning a temporally abstracted value function over these regions is an interesting design choice. The split between decision-level targets (TI output) and action-level targets (from the high-level policy) is conceptually clean.\n- Broad set of experiments: The method is evaluated on standard and challenging long-horizon benchmarks (antmaze including ultra variants, kitchen, CALVIN), and reports strong results relative to recent HRL baselines such as HIQL (Park et al., 2024a), PlanDQ (Chen et al., 2024a), DTAMP (Hong et al., 2023), and DiffuserLite (Dong et al., 2024), with helpful ablations (RD-HRL-TE, -TRE, -HP, -CU) and a horizon-size comparison.\n- Useful ablations and analysis: The ablations disentangle the effects of transition regions, temporal abstraction, and cross-trajectory updates; the horizon-size study clarifies that performance gains are not merely from increasing $H$ in a standard HRL pipeline.", "weaknesses": "Notation and parameter-role inconsistencies:\n  - Section 2.2 describes $\\beta$ as the discount factor even though $\\gamma$ is the discount factor throughout the paper. In AWR-style objectives $\\beta$ is typically a temperature; this should be corrected to avoid confusion with $\\gamma$.\n  - The TI objective (Eq. 10) uses $\\exp(\\beta^{d(s\\_t, s\\_z)} \\cdot A\\_{TI})$ together with text stating “raised to the power of $d(s\\_t, s\\_z)$.” This deviates from the standard AWR form $\\exp(\\beta \\cdot A)$ and from the high-/low-level objectives in Eq. 3 and 4. The paper should justify or correct this exponentiation vs. linear scaling.\n- Unclear writing. From a first read, it was very unclear whether $g\\_{TI}$ is a region identifier, a representative state from a region, or a learned embedding for the region. The paper would definitely benefit from a few additional iterations of editing/re-writing.\n- Cross-trajectory updates in Eq. (8) require a notion of temporal distance $d(s\\_{t1}, s\\_{t2})$ across trajectories. The approximation used can introduce bias or high variance when trajectories differ in speed/length. Some empirical sensitivity analysis or an uncertainty-aware alternative would strengthen the claim.\n- The reliance on K-Means clustering and a fixed FDI threshold may limit generality in high-dimensional or non-Euclidean state spaces. The procgen-500 pilot supports this concern: performance is poor when clustering raw pixels and improves only with external embeddings. This suggests that end-to-end learned representations or graph-based region discovery could be necessary for broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eha73TYK2p", "reviewer": "Reviewer_nnn4", "summary": "- This paper tackles the credit assignment problem in long-horizon, sparse-reward offline reinforcement learning (RL) tasks.\n- The authors argue that existing HRL methods are unreliable because the value functions used to select sub-goals suffer from generalization noise.\n- The paper proposes Reliability-Driven HRL (RD-HRL).\n- This method introduces a reliability-driven decision mechanism composed of three new components\n  1. Transition Region Extraction (TRE): K-Means clustering and a \"Future Diversity Index (FDI)\" to identify \"transition regions\"\n  2. Target Evaluation (TE): A value function trained only on states within these transition regions\n  3. Target Identification (TI): A policy that selects a long-term \"decision-level target\"\n- The authors claim this framework provides \"noise-immune\" guidance, leading to SOTA performance on benchmarks.", "questions": "- The central claim about solving \"generalization noise\" is confusing. Can the authors provide a precise explanation for how the TE module's \"cross-trajectory propagation\" is fundamentally different from the \"generalized Bellman backup\" it claims is the problem? Why is one reliable and the other not, when both operate on the same transition regions?\n- The paper claims SOTA on manipulation tasks (except one), but Table 1 shows this is false. Please correct this claim or explain the discrepancy\n- Can the authors provide a stronger theoretical or empirical justification for the FDI metric? As written, it seems arbitrary.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Despite some flawed analysis (see \"Weaknesses\"), the paper presents good results on some tasks. For example, Antmaze-Ultra benchmarks appear to be statistically significant and state-of-the-art\n- The paper introduces a new 3-level hierarchy (TI $\\rightarrow \\pi^h \\rightarrow \\pi^l$) to tackle the problem of sub-goal generation HRL\n- The paper provides an extensive set of ablations (e.g., RD-HRL-TRE, RD-HRL-HP, RD-HRL-TE) that attempt to validate the necessity of each new component", "weaknesses": "- The analysis is flawed and contains multiple errors that inflate performance\n  - The claim of achieving \"the best performance across all [manipulation] benchmarks except for the kitchen-mixed task\" is false. Table 1 clearly shows PlanDQ (75.0) and DTAMP (74.4) are superior on kitchen-partial and kitchen-mixed, respectively\n  - The claim of a \"57%\" outperformance over HIQL on CALVIN  is an incorrect calculation and a cherry-picked comparison. The 25-point delta (68.8 vs 43.8) is not 57%, and the more relevant comparison is to the second-place DiffuserLite (52.1)\n- The paper's method for claiming statistical wins is non-standard and not well-justified\n  - The 3% rule is justified by citing papers that use different rules (e.g., 5% rule in Chen et al and Li et al or CIs in Badrinath et al (see Table 1 row 1 -- halfcheetah))\n- The \"Future Diversity Index\" (FDI) metric is not clearly justified\n- There is a typo on line 144: \"To improve the the action-level...\"\n- In Table 3, the CIs for RD-HRL and HIQL on CALVIN clearly overlap", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W8WtvzDPjh", "reviewer": "Reviewer_5uMD", "summary": "This paper tackles the problem of unreliable sub-goal generation in Hierarchical RL(HRL) framework in long-horizon tasks. To address this issue, the author provides a reliability-driven HRL mechanism. This mechanism consists of three modules: Target Region Extraction (TRE) -- extracting target region from offline dataset, Target Identification (TI) -- selecting decision-level target guided by TE module, Target Evaluation (TE) -- evaluating target region candidates to help with TI module. This model, RD-HRL, achieved SOTA performance in several long-horizon RL tasks, including realistic datasets such as robotics tasks. Also, they provide an in-depth investigation of each component for their proposed mechanism.", "questions": "1. Is there any analysis or reference that explains the unreliable sub-goals problem? The paper cites ContextFormer, but the explanation of this issue is unclear there. If I'm wrong, please point me and reflect their exact analysis result and implications in the paper. If there are not enough analyses for this, it would be best to include an analysis in this paper.\n2. I believe the high-level intuition of this paper's suggestion as a solution makes sense and is generalizable over HRL. I wonder if this can be extended to other base models, not just on HIQL. e.g., HAC, HGCBC, etc., if the impl or idea is too specific to HIQL. I might consider this a narrower contribution than I expected.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Novelty and Generalizable Approach.** The issue of value functions being biased toward offline trajectories is a well-known problem in offline RL. This paper offers a general solution by identifying transition regions and sampling sub-goals within local regions that exclude transition areas. The authors also introduce a novel metric for extracting transition regions, which they identify as the primary source of the problem.\n2. **Performance.** The proposed model achieves state-of-the-art performance on challenging long-horizon RL tasks. This is a non-trivial contribution to the field.\n3. **In-depth Analysis.** The paper provides a thorough ablation study examining each component of the framework, so that the following researchers can investigate more easily.", "weaknesses": "1. **Unclear Definition of Unreliable Sub-goals.** The paper does not clearly explain what makes sub-goals \"unreliable.\" Adding intuitive examples would help. For instance, transition regions naturally contain diverse trajectories, which could affect value propagation and generalization. (If I understood correctly)\n2. **Limited Scope of Evaluation.** The approach is only tested with HIQL as the base model. While the hierarchical policy structure seems generally applicable, testing it on other hierarchical RL algorithms would strengthen the contribution. The choice of only IQL-based baselines seems limited, given the claimed generality of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ioMQJtOLFd", "reviewer": "Reviewer_XFsY", "summary": "This paper proposes Reliability-Driven HRL (RD-HRL) to address generalization noise in value functions when generating subgoals for hierarchical RL tasks. The method has three components: Transition Region Extraction (TRE), which identifies transition regions from the offline dataset; a Target Identification (TI) module, which selects targets from these regions; and a Target Evaluation (TE) module, which evaluates and refines the selected targets.", "questions": "1- This approach proposes defining subgoals for offline RL using a dataset of training trajectories. Do you think it would also work for online RL, where the agent must collect its own experience? Would the early subgoals be of sufficient quality?\n\n2- By the definition of Transition Region Extraction (TRE), do all transition regions consist of reliable and optimal subgoals, as illustrated in Figure 2(a)?\n\n3- In Eq. 8, how do you account for the current state? In addition to the quality of a subgoal with respect to the final goal, we should also consider which subgoal is closer to—and more easily achievable from—the current state.\n\n4- Are all modules trained jointly or sequentially? If sequentially, in what order?\n\n5- Can you provide examples of transition regions in manipulation tasks as well?\n\n6- Could you elaborate on the statement: \"states $s_{t1}$ and $s_{t2}$ may originate from different trajectories, which enables direct cross-trajectory propagation of value signals rather than generalization, thereby overcoming generalization error”?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- Well-written paper and easy to read.\n2- Comparison with recent baselines.\n3- Experiments on both navigation and manipulation tasks.", "weaknesses": "Please refer to the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)