Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning
Authors: Seungyul Han
Jaebak Hwang
Sanghyeon Lee
Jeongmo Kim
Venue: ICLR 2026 Poster (2026)
Keywords: goal-conditioned reinforcement learning
hierarchical reinforcement learning
sparse reward
long-horizon tasks
graph-based policy learning
subgoal planning
Primary area: reinforcement learning
TL;DR: We propose a new hierarchical RL framework (SSE) that, instead of relabeling subgoal failures as successes, treats them as terminal failures with zero reward, dramatically improving the high-level planner's reliability and efficiency.

Abstract: Long-horizon goal-conditioned tasks pose fundamental challenges for reinforcement learning (RL), particularly when goals are distant and rewards are sparse. While hierarchical and graph-based methods offer partial solutions, their reliance on conventional hindsight relabeling often fails to correct subgoal infeasibility, leading to inefficient high-level planning. To address this, we propose Strict Subgoal Execution (SSE), a graph-based hierarchical RL framework that integrates Frontier Experience Replay (FER) to separate unreachable from admissible subgoals and streamline high-level decision making. FER delineates the reachability frontier using failure and partial-success transitions, which identifies unreliable subgoals, increases subgoal reliability, and reduces unnecessary high-level decisions. Additionally, SSE employs a decoupled exploration policy to cover underexplored regions of the goal space and a path refinement that adjusts edge costs using observed low-level failures. Experimental results across diverse long-horizon benchmarks show that SSE consistently outperforms existing goal-conditioned and hierarchical RL methods in both efficiency and success rate.

Published as a conference paper at ICLR 2026
STRICT SUBGOAL EXECUTION: RELIABLE LONG-HORIZON PLANNING IN HIERARCHICAL REINFORCEMENT LEARNING
Jaebak Hwang, Sanghyeon Jeongmo Kim, Seungyul Han'
Graduate School of Artificial Intelligence
Ulsan National Institute of Science and Technology (UNIST)
Ulsan, South Korea 44919
(mi 1k, sanghyeon, jmkim22, syhan]
ABSTRACT
Long-horizon goal-conditioned tasks pose fundamental challenges for reinforce- ment learning (RI.), partieularly when goals are distant and rewards are sparse. While hierarchical and graph-based methods offer partial solutions, their reliance on conventional hindsight relabeling often fails to correct subgoal infeasibility, leading inefficient high-level planning. To address this, we propose Strict Sub- goal Execution (SSE), a graph-based hierarchical RL framework that integrates Frontier Experience Replay (FER) to separate unreachable from admissible sub- goals and streamline high-level decision making. FER delineates the reachability frontier using failure and partial-success transitions, which identifies unreliable subgoals, increases subgoal reliability, and reduces unnecessary high-level deci- sions. Additionally, SSE employs a decoupled exploration policy cover under- explored regions of the goal space and a path refinement that adjusts edge costs us- ing observed low-level failures. Experimental results across diverse long-howe morizon benchmarks show that SSE consistently outperforms existing goal coal-conditioned and hierarchical RL methods in both efficiency and success rate. Our code is avail- able at
INTRODUCTION
Recent advances in reinforcement learning (RL) have achieved impressive success across various domains (Mnih et al., 2013) Silver et al., 2016). Although prior works have enhanced sample ef- ficiency through advanced exploration (Burda al. 20188 Han & Sung. 202 Jo et al., (024) and experience replay mechanisms (Schaul et al. 2015b] Han & Sung 2017). they often struggle to adapt changing objectives without extensive retraining. Consequently, Goal--Cndittoned RL (GCRL) was introduced to explicitly condition the policy on a target goal, allowing agents to di- rectly pursue diverse states specified by the environment (Schaul et al., [2015a] Levy
2017]
Nasiriany et al.) 2019). However, in sparse-reward and long-horizon environments, goals are often distant, making exploration difficult and hindering effective learning due lack of intermediate guidance.
Hierarchical reinforcement learning (HRL) addresses long-horizon goal-conditioned tasks by de- composing control into high-level policy that proposes subgoals or skills and low-level policy that executes actions to reach them (Bacon al. 2 Vezhnevets et al. (017). While this decom- position can ease learning by introducing more attainable intermediate objectives, performance can degrade when proposed subgoals are not reliably reachable by the low-level policy. To mitigate this issue, graph-based HRI. methods construct a graph over the goal space where nodes represent states or regions and edges encode feasible transitions (Zhang et al. 2018 Nachum et al., [0188|] Huang ett al.) 2019: Eysenbach et al. 2019; Kim et al.) 2021 Zhang et al. 2021). This structure enables subgoal selection via shortest-path planning. Despite these advances, scalability remains challeng- ing when the final goal is far from the current state because high-level planning may require many
"Correspondence Scungyul Han

---

Published as a conference paper at ICLR 2026
decisions and lead to unstable training. To reduce reliance high-level policies, recent work has explored graph-based GCRL without explicit hierarchical structures (Lee et al.) [023] Yoon et al.)
2024). However, removing high-level reasoning reduce flexibility in environments with multiple goals or heterogeneous reward signals.
To overcome these limitations, propose Strict Subgoal Execution (SSE), graph-based HRL framework that improves long-horizon performance by explicitly characterizing the reachability boundary of candidate subgoals. SSE integrates Frontier Experience Replay (FER) to distinguish high-level outcomes into failure transitions and partial-success transitions, localizing where execu- tion breaks down and how far it progresses toward subgoal. This frontier-based separation enables the agent to identify unreliable subgoals, train policies to execute admissible subgoals more ac- curately, and reduce unnecessary high-level decisions. SSE also expands goal-space coverage by actively exploring undervisited graph regions and improves planning reliability by refining paths to avoid transitions with frequent failures. As a result, SSE solves a broader range of complex long- horizon tasks. Our contributions are summarized as follows:
1. Frontier Experience Replay (FER) for SSE:
We introduce FER, which delineates the reacha- bility frontier using two high-level samples: failure transitions and partial-succes transitions. By precisely localizing where attempts fail and how far progress extends, FER identifies unreliable subgoals, increases subgoal reliability and reduces unnecessary high-level decisions.
Decoupled Exploration for Goal Space Coverage: dedicated exploration policy is decoupled from the return-driven high-level policy to traverse underexplored regions of the goal space, improving coverage and sample efficiency.
Failure-Aware Path Refinement: To improve subgoal reliability, we adjust graph edge costs based on low-level failure statistics, encouraging path planning to avoid unstable transitions and strengthening subgoal execution.
Through extensive evaluation on diverse long-horizon goal-conditioned tasks, our method demon- strates higher success rates and better sample efficiency than prior GCRL and HRI. approaches, validating the effectiveness of the proposed SSE framework.
PRELIMINARIES
2.1 UNIVERSAL MDP, CONL-CONDT RL, AND GOAL RELABELING TECHNIQUES
We consider universal Markov decision process (UMDP) (S, G,, R, 7), where S, G. and A denote the state, goal, and action spaces, the transition dynamics, R the reward function, and E (0, 1] the discount factor Schaul al. (2015a). At time step t, the agent observes (st, (st,g), samples ag ~ T( (). and receives r, = R(s,, ae $141.9) with St+1 ~ P(-|s,, a.)) The goal of GCRL is to learn a goal-connitioned policy that maximizes the expected return Σn over horizon H. When S * G. GCRL typically assumes mapping the goal space. The goal g may be fixed or sampled per episode. In long-horizon goal-conditioned settings, learning is often inefficient due to sparse positive signals. Goal relabeling methods such as Hindsight Experience Replay (HER) (Andrychowicz et al.) (017)) improve sample efficiency by treating later achieved states as substitute goals. HER replaces the goal g with a future achieved goal ((se) for some 1' 2 ( and recomputes the reward:
S- G projecting states into
(s:; Qz, Tt, $1+1,9) g) + (St, ae, R(st, az $t41, S(+1,9)).
This converts unsuccessful attempts into useful signals while the original transitions unchanged.
2.2 HRL FRAMEWORKS AND GRAPH-BASED SUBGOAL PLANNING IN GCRL
In goal-conditioned settings, HRL addresses long-no challenges by decomposing control into a high-level policy T and a low-level policy x Bacon et al. 2017 y Vezhnevets et al. (017). In general, HRL frameworks in the GCRL domain typically assume that the goal space G is known to facilitate the high-level selection of feasible subgoals. Every le steps, ~²(- st. selects a subgoal 9r E G, which r-(- @t) attempts to reach using an auxiliary reward (Zhang et al. 2020;
Pateria et al., (202T: Hutsebaut- Sattsebaut-Buyssc et al.) 2022). However, when subgoals are too distant, the low-level policy may fail to reach them within the given horizon, and distance-based penalties can
2

---

Published as a conference paper at ICLR 2026
hinder learning under sparse rewards. To mitigate this, graph-based approaches construct a goal- space graph G = -V, E), where V is a set of landmark nodes and E contains edges weighted by the effort to transition between nodes. Landmarks are commonly chosen via farthest point sampling (FPS) (Kim et al. 2021 Lee et al. 2022) or placed on a predefined grid over the goal space (Yoon et al. 2024), assuming that the goal space is known to define the graph and the high-level poliey.
In general, the edge cost is defined as
02, d(v: + v2) == log, (1 + (1 (1- 700 (v,, 7'),,
where Q estimates the feasibility of reaching 82 from 81 under x², ddefined as the value function Q' of the low-level policy predefined estimator. In this paper, set modified version of edge cost and details on Q are provided in Appendix B.1 Given a subgoal á.. graph- based HRL computes the shortest path from s(ss) to Gr via Dijkstra's algorithm (Dijkstra 1959), yielding waypoints (WP1,..., WP,,) with wp; E V. The low-level policy sequentially targets these waypoints, switching to WPs+1 once wP; is reached, until Ør is attained. The high-level policy trained on transitions (St, 9. 91, 2μ=t ry s(tk) in B3p while n² is trained on (st, wP,, at, (,, in B'. Following prior work (Kim et 2021 Lee al. 2022), the low-level reward is rf 0 ())([(()))))))))))))) and -1 otherwise, where À > is a subgoal reachability threshold and ø is known. In contrast, some approaches omit high-level subgoals and directly train the low-level policy to reach the final goal g.
assuming intermediate subgoals are unnecessary (Lee et al. 2023 Yoon et al. 2024).
3 RELATED WORK
Coal-Conditioned RL and Hierarchical Approaches GCRL refers to RL settings where the agent is explicitly conditioned on a goal (Kaclbling 1993; Liu et al. 2022 Colas al., 2022). Modern GCRL typically employs Universal Value Function Approximators (UVFA) (Schaul et al., 2015a) to generalize across goals. A central challenge is solving long- long-horizon tasks with sparse rewards, where exploration is difficult. To address this, HRL introduces multi-level policies that de- compose complex tasks into temporally abstract subgoals (Vezhnevets et al. 2017 Nachum et al.] 2018b) or reusable skills (Eysenbach et al.) 2018; Lee et al. 2025). thereby reducing the effec- tive planning horizon. The effectiveness of this decomposition has been demonstrated in diverse settings (Barto & Mahadevan 2003] Kulkarni et al.) 2016 Nachum et al., 2011a 2019). Beyond hi- crarchical structures, other methods improve sample efficiency through structured exploration, such as prioritizing novel states (Warde-Farley al. 2018 Pong et al. 2019) or discovering useful goals via unsupervised learning (Mendonca et al. 2022]] Ecoffet et al. 2021 Chane-Sane et al. 2021).
Relabeling and Guidance Techniques in GCRL In sparse-reward GCRL, data relabeling is cen- tral for improving sample efficiency. HER converts failed trajectories into successes by replacing the intended goal with future achieved goal (Andrychowicz al., 2017). Since uniform hindsight-goal sampling can be suboptimal, subsequent work explores curriculum-basee relabeling (Fang et al., 2019), novelty- or priority-driven goal sampling (Zeng et al. (003), and hierarchical relabeling that better matches low-level behavior (Nachum et (018bb.) Orthogonal guidance approaches dis- courage infeasible subgoals, for example via learned adjacency constraints (Zhang et al. 2020) or adversarially generated goals that are challenging yet achievable (Levy et al. 2017).
Graph-based Approaches in GCRL Graph-based approaches have been introduced to GCRL to provide structured representation of the goal space, enabling planning over discrete landmarks and improving navigation in sparse-reward environments (Huang et al.] 2019 Eysenbach et al.] 2019 Kim al. 2).2]. Early work used graph structures to represent the state space and to guide exploration (Zhang et al. 2018 Nachum et al. 2018a) and this direction has since evolved through integration with latent modeling (Zhang et al. (02I) and policy-driven graph construction (Kim) ett al.) 2023). More recent advances focus either on aligning high-level decisions with low-level execution via graph-based planning (Lee et al., 2022) or on enhancing exploration with strategies such as frontier-based expansion (Hoang et al. 2021 Park et al.) 2024). curriculum-bases goal selection (Lee et al., 2023), and virtual subgoal generation for broader coverage (Yoon et al.) 2024). In related works, brontier-based exploration refers to exploring near boundary states. In our work, we also use the term frontier, but it denotes the boundary that identifies failure points and separates achievable from unachievable subgoals, which is distinct objective from contierrbasee exploration.
3

---

Published as a conference paper at ICLR 2026
4
METHODS
4.1
MOTIVATION: RETHINKING SUBGOAL EXECUTION IN GRAPH-BASED HRL
g16 g
g
0o
g
0
9
0
9 L4
Jo 10
9o
Dt & (5t2)
tv
Φ (5))
Distance between f and
& (50))
(5o)
(c)
Landmark nodes
State location
Subgoals
Final goal
Agent trajectory
Figure 1: Agent trajectories in goal space G. (a) Conventional HRL with HER relabels interme- diate states subgoals without enforcing exact subgoal completion, which lengthens high-level trajectories. (b) SSE with FER enforces exact subgoal completion, increasing subgoal reliability and reducing unnecessary high-level decisions, thereby improving learning efficiency. (c) After training, SSE reaches g with few high-level steps, here a single step in single-goal settings even from dis- tant starts. Agent locations ((st) G and is the i-th high-level step.
In GCRL, many methods improve goal- noal-reaching performance by adopting hindsight relabeling tech- niques such as HER (Andrychowicz et al. (017). which treat intermediate states as virtual goals to provide additional training signals. Recent graph-based HRL methods typically apply this idea to both the low level and the high level. While this benefits the low level, it introduces a critical issue at the high level. Fig. Πa) illustrates how graph-based HRL with HER operates: the high-level policy selects subgoal It given the current state st. and the low-level policy follows waypoint path to reach it. If ğ: is unreachable due to limited skill, obstacles, or excessive distance, the agent stops at an intermediate state, and such subgoals should be avoided. When HER is applied, every visited state along the failed trajectory is treated as if it were the intended subgoal. This causes the high-level policy repeatedly select ineffective subgoals and waste many decision steps. In addition, the resulting transitions can vary widely for the same subgoal, which hinders stable learning. As shown in Fig. [a), this produces unnecessarily long high-level trajectories, and even if the agent reaches the goal g. credit is spread over too many steps, preventing carlier decisions from being reinforced and often leading to failure on long-horizon tasks.
To address this issue, we propose the Striet Subgoal Execution (SSE) framework, which updates the high-level policy with positive returns only when the low level successfully reaches the assigned subgoal. We instantiate this principle with Frontier Experience Replay (FER). Unlike relabeling methods such as HER that synthesize additional successes, FER delineates the reachability frontier by recording two types of high-level samples, failure transitions and purtial-success transitions. By precisely localizing where attempts fail and how far progress extends, FER identifies unreliable subgoals, provides consistent training signals, and reduces unnecessary high-level decisions. As shown in Fig. [[b,,, this separation of success and failure keeps the resulting state ((*tr closely aligned with the selected subgoal 9t. producing consistent high-level transitions and eliminating wasteful actions. Consequently, as illustrated in Fig. [c), the resulting high-level policy solves tasks with far fewer decisions, often reaching the final goal g in single step in simple settings and handling multi-goal or long-horizon environments with only few well-chosen subgoal selections.
4.2 STRICT SUBGOAL EXECUTION WITH FRONTIER EXPERIENCE REPLAY
In this section, we describe the details of the SSE framework. We first define FER, the key com- ponent of SSE, which marks the reachability frontier by recording two high-level sample types in addition to standard successes: failure transitions that stop at the point of failure and partial-success transitions that record the last reliably reached waypoint. To formalize this, we basically follow the HRL setup from Section[]] At each time t, the policy selects a subgoal g. E 9 under the common as- sumption that the goal space G is known. A waypoint path (wP1..... wP,) on the graph G = (V, F))
4

---

Published as a conference paper at ICLR 2026
is then generated, and the low-level policy x/ follows this path until termination time t'. We regard the subgoal as reachable if [(((s)) - ğe < A; otherwise the attempt is treated as a failure. In the failure case, WPfinal denotes the last waypoint reached within tolerance before failure. Based this notion of reachability, we define FER as follows.
Definition 4.1 (Frontier Experience Replay) The high-level replay buffer B stores transitions as
(st: 9. 9t: Σ Tj. se) (success)
((+((s))) - -ell < X,
j=t (st, Gc. Sr) (stop-on-failure)
B
fl Dell 2 X,
tup 11
(st; 9, WPiinal:
Σ j=t
Tj. Stup) (partial success) ff occurs and WPfmmal exists.
Here, 8T the terminal state, Ej y is the cumulative reward collected until reaching the subgoal or waypoint, and twp the time step at which WPinna reached.
In FER, success transition records the full cumulative return up t' and continues the episode, stop-on-failure transition assigns zero return and sets the next state to 8T, which immediately truncates the episode so all future returns are forfeited and unreachable or unreliable subgoals are discouraged, and portial-success transition records WPtinal with its accumulated return, localiz- ing how far the attempl progressed and crediting only the reliably executed portion. Together, these signals delineate the reachability frontier, identify unreliable subgoals, and provide consistent high- level targets that reduce unnecessary high-level decisions. This setup separates reliable from unre- liable subgoals and suppresses failure-inducio exploratory coverage because the agent learns to avoid regions associated with failure. To counter- balance this effect, we introduce a decoupled high-level controller comprising high-level policy nh for exploitation trained on B. and complementary exploration policy Tx that promotes cov- erage. As in prior graph-based HRL, assume the goal space known for subgoal selection of both Th and n exp, and SSE introduces no additional assumptions. Detailed formulations follow.
actions, but it induce conservatism and reduce
High-level Policy z²: In prior work, subgoals are typically selected with Gaussian policies with small noise. Although adequate for those methods, this concentrates exploration near the current maximum subgoal. As noted above, we aim for a high-level policy that can target any point in the goal space, thereby broadening exploration. We therefore define an e-greedy nh as
Imaxx, t arg maxzeg ("(s,. 9. with probability 1-
Th St. g)
Grand Uniform(g)
with probability 6,
where Imax,t is the greedy subgoal, in practice generated by the actor network trained choose the maximum of Q": -rand is sampled uniformly from 9 to ensure persistent global exploration, and Q is trained off-policy using B%. This formulation allows T² to select diverse points in the goal space independently of the agent's current location, which crucial for broader exploration.
Exploration Policy " To promote exploration, TP targets low-density, underexplored regions of the goal space. To do this, we first identify novel regions Vnowel C G. for which we employ either grid-based estimator or model-based approach. In this paper, we consider both: the grid-based estimator is used for simplicity in 2D/3D goal spaces that are widely adopted in real-world settings (e.g., position goal spaces), whereas the model-based approach, although more complex, is adopted to facilitate extension to dighee-dimensional goal spaces. The exploration policy x exp is defined as:
9
with probability with probability 3:
n exp (ğe
gt.
Imax,t
(4)
Gnovel Uniform( V Vnovel) with probability 3.
where the novel region Vnovel is defined in one of two ways, depending on whether we use the grid- or the model-based learning scheme, that is, Vnovel = Vgridd novel or Vnovel Vmmddll The grid-based novel region is defined ygrid := C.iioooee, where C denotes the grid cell of size dg that partitions the goal space G, indexes the grid cells, and mnovel arg min, N(C)) is the
5

---

Published as a conference paper at ICLR 2026
N(C
(b)
Start posittio
Subgoal eased
Subgoal frax
Subgoai frass
* goal
[ cell c2'
Figure 2: Initial subgoals t 0 selected by x² and Tx with corresponding Ant agent trajec- tories at (a) early, (b) intermediate, and (c) final training stages in the U-maze task. The goal space (agent positions in the map) is partitioned into grid cells CG T²h selects between Imax and Irand to encourage broad coverage, while л T samples from Jnovel, Imax. and g to visit underexplored regions and the goal. Over time, unreachable areas excluded from subgoal candidates via SSE.
index of the cell with the smallest visitation count N (CT'). The model-based novel region is defined as Vimodel { arg max [lfg(v) fer (v)ll) where fc is the learner network with parameter <. and J.... is the initialized target network proposed by (Burda et al. 2018 to describe novel regions based prediction error. The learner fo is trained to minmiize the prediction error for node samples. Then, the novel subgoal Gnovel is uniformly sampled from novel. Both x² and TT operate under the same SSE mechanism, where episodes continue only upon successful subgoal completion and terminate on failure. To control exploration, we sample from TxP early in training and then gradually mix it with T using ratio (1 n). where ?) controls exploration strength. This balanced scheme preserves coverage, accelerates the discovery of reachable subgoals, and improves high-level generalization. Implementation details and parameters such as rj are described B.2
To illustrate the behavior of the proposed method, Fig. shows grid-wise visitation in the goal space and low-level trajectories toward various initial subgoals at three stages of training: (a) early (10K steps), (b) intermediate (150K steps), and (c) final (500K steps). The environment is a U-maze where a MuJoCo Todorov et al. 2012) Ant agent navigates to the final goal g located at the upper left of the map. In (a), most of the goal space remains unexplored, and initial subgoals selected by oxD (grid- based) and T², including the final goal, random subgoals, and novel regions, result in wide-ranging trajectories that promote broad exploration. As training progresses in (b), the agent expands its coverage across the map, and in (c), it consistently reaches the goal Notably, SSE enables the agent to reach any reachable subgoal in single high-level step, regardless of distance, supporting reliable execution and efficient long-range planning. This allows the agent to solve com- plex tasks using only a few high-level decisions.
4.3 MILLURE-AWAR PATH REFINEMENT
In the SSE framework, reliable subgoal execution is crucial as each subgoal must be reached within single high-level step. As described in Section graph-based methods use Dijkstra's 's algorithm to compute waypoint paths from ((x)) to 9. on a graph G = (V, E), with edge distances defined by in equation T However, this distance ignores failure cases like collisions getting stuck, causing agents to fail even on the shortest path. We observe these failure-prone regions significantly hinder subgoal success. To address this, introduce a failure-aware path refinement strategy that increases edge costs in unreliable regions, steering the planner toward safer alternatives. For each edge from node v1 node U2 we define the failure ratio ratiorall for the target node U2, and refine the edge by increasing its distance as the failure ratio increases as
H(m v2) d(vn - w2) x max (1, Grist rationa( Vel, 02 E V
(S)
where d is the original edge distance from equation T and Caist > 1 is a scaling factor. For consis- tency with exploration policy, the failure ratio for the target node is also defined in two ways, fol- lowing the grid- and model-based approaches: ratio
ratiogri or ratioнil ratio model The brid-based failure ratio is defined as ratio fail (8%) == Nr. I/N ). where 822 E C Nea (C is the number of failure episodes in the cell C The model-based failure ratio is defined
grid
grid

---

Published as a conference paper at ICLR 2026
model ratio (02) FG((2), where Fe is failure predictor network with parameters trained over node samples using a cross-entropy objective, assigning label to nodes from failed trajectories and label to nodes from successful trajectories. To ensure sufficient low-level policy competence, the awire path refinement is activated only after Acount successful visits. A higher failure ratio increases the adjusted edge distance, encouraging Dijkstra's algorithm to avoid unreliable regions.
As described above, we instantiate SSE with two variants that differ their exploration policy and failure-aware path refinement, namely grid-based approach and model-based approach. We refer to SSE with the grid-based variant as SSE (Grid) and SSE with the model-based variant as SSE (Model), and we empirically compare these two variants to analyze how the choice of approach affects learning performance.
Fig. 3 iillustrates the effect of the proposed grid- based path refinement in a bottleneck environ- ment. In (a), without refinement, the agent re- peatedly follows the shortest path through nar- row corridor near wall, often resulting in fail- ure. In (b), with refinement applied, increased edge costs in high-failure regions steer Dijk- stra's algorithm toward safer detours. When no alternatives exist (e.g., in the bottleneck), the
(b) Froquently failed region
agent still passes through, preserving reachabil- ity. This demonstrates that the refinement hances subgoal success while maintaining over- all reachability, supporting more stable execu- tion in complex tasks. In summary, the proposed SSE framework illustrated in Fig. 4 with a condensed version of the algorithm provided in Algorithm [ The full algorithm and implemen- tation details, including the graph construction and training losses for RI. training and model- based approach are provided in Appendix
Start position position Shortest path Agent trajectory
Figure 3: Comparison of agent trajectories (blue lines) in a map with a bottleneck: (a) without path refinement and (b) with the proposed path refinement. Green lines represent the shortest waypoint paths computed via Dijkstra's s algo- rithm, while red areas denote grid cells with high failure ratios, i.c., (atiora((Cr)) 0.05.
Algorithm 1 Strict Subgoal Execution (SSE)
Strict Subgoal Execution (SSE)
Failure-Aware Path Refinement
Initialize policies л π s n², and graph C for each iteration do for each episode do for each high-level selection step do
Exploration π exp
High-level
Se Dc
Sample subgoal ğ from x² and n
Plan waypoint path from 6(sc) to ğ: Roll out low-level policy x <'(s, wp,) Vi: if (ló(s;r) ăell < X (success) then Store the success transition and con- tinue
Success
Graph with à = (V, F)
S 9.. 1r
Stop on foilure
Low-level πt
(st 9. T.. 0,
else
Refined Waypoint Path ".p,.,
Store the failure and partial transitions, then terminate the episode
partial-success success
Portial
G. wpn. E/r Eije Tp. Stop)
end
end end
FER Bh
Low-level Replay
Train
B
Rollout
Count N(C ) and Naa((CT) Vm (Grid-based)
Update models fo and Fe (Model-based)
Update Q*, Q'. th n²' off-policy RL
Figure 4: The proposed SSE framework.
end
5
EXPERIMENTS
In this section, we evaluate our SSE framework on challenging long-horizon tasks, includ- ing 5 AntMaze environments (U-maze, m-maze, ze, AntMazeComplexx AntMazeRottlenedee

---

Published as a conference paper at ICLR 2026
(a) U-maze
(b) ir-maze
(c) AntMaze
(d) AntMaze Bottleneck
AntMaze DoubleBottlened
Complex
(1) AntkeyChest
(g) AntDouble
(h) ReacherWall
ReacherWall DoubleGoal
KeyChest
Figure 5: Considered long-horizon environments: AntMaze, 2 KeyChest, and Reacher tasks
and AutMaazeDouleeBttJeee leneck). These range from simple layouts (U maze) to complex structures with narrow corridors (antMazeBottleneeae
We also assess KeyChest tasks (Ant Ant (eDooubbeeeyhestt) where the agent must collect ing the final goal, even though the keys are not explicitly defined as goals. Additionally, we evaluate 2 Reacher tasks TreacherWa
keys before reach-
ReacherWa Doubb eGoa where 3D robot must navigate obstacles to reach one or two goals. KeyChest and DoubleGoal tasks require intermediate objectives, making them ideal for testing high-level planning. See Fig. 5 for visualizations and Appendix C for details. While the main experiments focus on fixed-goal settings, additional comparisons for random-goal setups included in Appendix D.I
5.1
PERFORMANCE COMPARISON
Success Rate
0.8
0.8
0.8
0.4
0.4
0.4
0.2
0.2
0.2M 0.4M 0.6M 0.8M
1.0M
%
1M
2M
3M
4M
SM
%
1M
2M
3M
4M
5M
Timestep
Timestep
Timestep
(a) U-maze
(b) m-maze
(c) AntMazeComppex
1.0
Success Rate
0.8
0.8
0.6
0.4
0.4
0.4
0.2
0.2
%
0.2M
0.4M 0.6M 0.8M
1.0M
0
0.5M
1.0M
1.5M
2.0M
%
1M
2M
3M
4M
5M
Timestep
Timestep
Timestep
AntMaazeBotteeeck
AnttMaazeDoubbeBstttennekk
AntKeyyhest
1.0
Success Rate
0.6
0.8
0.48
0.6
0.6
0.4
0.2
0.2
0.2
%
3M
4M
SM
0
0.1M 0.2M 0.3M 0.4M
0.5M
0
0.2M 0.4M
0.6M
0.SM 1.0M
Timestep
Timestep
Timestep
AntDoubleKeyChest
(h) ReacherWall
(i) ReacherWallDaubleeeee
SSE (Grid)
SSE (Madel)
BEAG
NGTE
DHRL
HRAC
PIG
HIRO HIGL
Figure Performance comparison on various long-horizon environments
We compare SSE with a range of hierarchical RL and recent graph-based methods. Specifically, we evaluate 2 HRL approaches: HIRO (Nachum al. 2018b), which improves sample efficiency via hindsight goal relabeling, and HRAC (Zhang et al. 2020). which penalizes subgoal selection based reachability. We also include 3 graph-based HRL methods: HIGL (Kim al.) (021), which applics intrinsic penalties over a graph. DHRL (Lee et al.] (022). which finds a path and gives waypoints to low-level as goal via graph planning, and NGTE (Park et al. 2024), which expands
8

---

Published as a conference paper at ICLR 2026
graphs using novelty better address fixed-goal settings. Additionally, consider 2 graph-based methods without explicit high-level policies: PIG (Kim et al. 2023), which integrates graph struc- tures into imitation learning to skip redundant subgoal actions, and BEAG (Yoon et al. 2024), which uses breadth exploration with imaginary landmarks for goal-reaching.
For our proposed framework, evaluate two variants: grid-based implementation SSE (Grid), which offers a simple implementation in practical 2D/3D goal spaces, and a learning-basse im- plementation SSE (Model), which is designed be scalable to higher-dimensional goal spaces, as described in Section 4 Both variants are evaluated with the best hyperparameter settings (Caist. dg. 7) from ablation studies, while all baselines use outhor-provided implementations. threshold À used to determine subgoal reachability is the same as the threshold used to define the low-level reward in Section [.2 We use the same values as the baseline methods to ensure fair and consistent comparison. Detailed descriptions of each algorithm, along with the hyperparameter configurations for our proposed method, are provided in Appendix
FER, the
Fig. 6 shows mean success rates over 5 seeds (solid lines) with standard deviations (shaded). From the results, both SSE variants consistently outperform other graph-based and conventional HRL methods across all benchmarks. Conventional HRL methods (DHRL, HIRO, PIG, HRAC, and HIGL) rely on random goal sampling for exploration but often struggle with fixed-goal tasks, while occasionally succeeding in random-goal setups provided in Appendix [..I] highlighting the increased challenge of fixed goals due to limited exploration opportunities. In relatively simple en- vironments such U-maze, m-maze, and AntMa zeComplex, baseline methods like BEAG, grid-based nodes, and NGTE demonstrate reasonable performance, but SSE typically converges more quickly. In bottleneck environments, SSE further excels by using failure-aware path refine- ment to avoid unstable regions as shown in Fig. 3 In more complex tasks like KeyChest and which require reaching intermediate objectives, baseline methods largely fail. Conventional HRL suffers from long high-level horizons, and goal-centric methods without high-level decision-making, such as BEAG, cannot reason about intermediate targets. In contrast, SSE mitigates these issues by strictly enforcing subgoal completion, reducing high-level decision steps. These results highlight the versatility, efficiency, and generalization of the proposed SSE framework. While both variants succeed in all tasks, SSE (Grid) converges faster due to lower overhead. In contrast, SSE (Model) is designed for scalability to high-dimensional spaces where discretization is infeasible, despite higher training complexity. In addition, extended evaluations Pnsh Ant Fall, and Ant 4Rooms in Appendix E.3 confirm that SSE consistently matches or exceeds baselines. Furthermore, computational analysis in Appendix D.2 demonstrates that SSE achieves lower complexity per iteration than baselines.
5.2 FURTHER ANALYSIS AND ABLATION STUDIES
NC!
(a)
(c)
Start position
Goal position
x
Subgoal Sree
Novel region
Chr
Keys
Grid C
Figure 7: Trajectory analysis for SSE subgoals Imax in AntDoub I eKeyChest at: (a) early stage, (b) reaches first key, (c) collects both keys, (d) reaches goal after collecting both keys (task success).
In the following analysis and ablation studies, we focus on the grid-based approach and refer to SSE (Grid) simply as SSE, as consistently achieves better performance and offers a richer set of hyperparameters for analysis and implementation. Fig. 7 presents trajectory analysis of the proposed SSE framework in the AntDoubleKeyChen progressively explores the map and collects both keys and the final goal. In the early stage (a) (t x 300K), the agent expands map coverage by sampling diverse subgoals, similar to simpler environments. As training progresses, the agent visits increasingly more regions, allowing it reach the first key within a single high-level step, as shown in (b) (t ss 1M). In more advanced stage (c) (t as 1.5M), it collects both keys in just high-level steps. Eventually, as shown in (d) (t as 3M),
environment, illustrating how the agent

---

Published as a conference paper at ICLR 2026
the agent completes the entire task, including both keys and the goal, in only three high-level steps. These results demonstrate that SSE allows the agent to reach any location in the map with a single high-level decision. As a result, it solve complex multi-goal tasks using minimal number of high-level steps, which highlights the effectiveness of SSE in long-horizon environments that require sequential decision-making making for the high-level policy. Notably, SSE solves this sequential task without an explicit curriculum. The reliability of its high-level policy, learned via FER, combined with an augmented state including key-possession flags, enables the agent autonomously discover the required sequence of sub- sub-objectives.
0.6
0.6
0.6
0.0
Pale
Success Rate
Success Rate
Success Rate
xf
Success
0.2
2.2
2.
0.2
TimesteD
4M
Tineste
19
2M 3M Timestep
Timestee
3M
4M
(a) Component evaluation
(b) Distance scaling Caias
(c) Grid size dg
(d) Exploration ratio y
Figure 8: Ablation study on Dnt environment
We conduct an ablation study on HeaDDubbeeeyyhher to evaluate the contribution of each com- ponent in SSE and to analyze the impact of key hyperparameters, as shown in Fig. For the compo- nent analysis, we consider five variants: (1) SSE with FPS, which replaces the grid-based landmarks with Farthest Point Sampling Kim et al. (02T): :): (2) SSE with HER, which replaces strict subgoal execution with conventional high-level policy that uses HER; (3) SSE without B% which dis- ables FER by using a standard replay buffer; (4) SSE without Tx,, which disables the exploration policy; and (5) SSE without path refinement (PR), which disables the failure-aware adjustment. All variants exhibit degraded performance. Notably, the settings using HER or disabling FER fail to learn entirely, emphasizing the importance of strict subgoal execution and the consistent high- level signals provided by FER. SSE with FPS eventually succeeds, though less efficiently than the grid version, while still outperforming FPSSbased baselines. This confirms that our framework re- mains effective regardless of the specific landmark selection strategy. Our hyperparameter analysis shows that setting Cdist = 5 achieves an optimal balance, though performance holds steady across a range of values. Similarly, performance also holds steady for grid resolution, though overly fine grids (e.g., dg = 1) can slow learning. For the exploration ratio 17, a value of 0.2 proves optimal. While hyperparameter tuning optimizes performance, SSE consistently outperforms all baselines. Additional analyses in other environments and sensitivity studies on the reachability threshold and the high-level discount factor 7 are provided in Appendix D further characterizing SSE.
LIMITATION
Our framework introduces new hyperparameters. such as the exploration ratio 1) and path refinement factor Caist, which require tuning. However, we find their effective ranges to be stable across diverse environments, and our ablation studies confirm that the framework maintains strong performance across a wide range of values, minimizing the overall tuning cost. While SSE also adds computa- tional steps, its principle of early termination subgoal failure yields significant efficiency gains. This prevents long, unproductive trajectories and results in faster computation time compared to other recent methods, as quantitatively analyzed in Appendix D.2.
7 CONCLUSION
In this paper, we proposed SSE, a graph-based HRL framework designed to improve reliability and efficiency in long-horizon, goal-conditioned tasks. By enforcing single-step subgoal reachability, SSE enables more direct and reliable high-level planning and significantly reduces decision hori- zons. The introduction of failure-aware path refinement and decoupled exploration policy further enhances subgoal reliability and map coverage. Extensive experiments demonstrate that SSE consis- tently outperforms existing HRL and graph- based methods across range of complex tasks. These results highlight the framework's 's effectiveness in enabling stable and generalizable behavior, mak- ing it promising approach for scalable hierarchical control in various long-horizon tasks.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported partly by the Institute of Information & Communications Technology Plan- ning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2022-11220405 Development of Core Technologies for Task-oriented Reinforcement Learning for Commercializa- tion of Autonomous Drones), (No. RS-2025 RS-2025-25442824, 25442824, AI Star-Fellowship Program (UNIST)), and (No. RS-2020-1120133 133,,,66,33,,,,,,, Artificial Intelligence Graduate School Support (UNIST)), and partly by National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-20 RS-2025-23523199 LLM--Bssed Multi-Agent Reinforcement Learning for End-to-End Large Autonomous Swarm Control).
ETHICS STATEMENT
This paper introduces foundational algorithm, SSE, designed to improve the long-horizon planning capabilities of reinforcement learning agents. All experiments were conducted in standard, simulated robotics environments. As such, our research does not involve human subjects, sensitive or person- ally identifiable data, nor does it directly address systems that interact with people. Therefore, issues of data privacy, dataset bias, and fairness are not directly applicable to this work.
REPRODUCIBILITY STATEMENT
We are committed to the reproducibility of our research. The complete source code for our proposed framework, SSF, and all experiments included anonymized zip file in the supplementary ma- terials. A detailed breakdown of the implementation, including network architectures and training procedures, can be found in Appendix B. All hyperparameters required reproduce our results are provided in Appendix C.3, with common settings listed in Table and environment-specfffe config- urations in Table 3. Furthermore, Appendix C details the full experimental setup, including descrip- tions of the baseline algorithms, specifications for all environments (Table 1), and the hardware and software configurations on which the experiments were conducted. We believe these resources will enable the reproduction of our findings.
REFERENCES
Marcin Andrychowicz, Filip Wolski. Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI conference artificial intelligence, volume 2017.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13:341-379, 2003.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810. 12..1...289444 2018.
Elliot Chane-Sane Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International conference on machine learning, pp. 1430-1440. PMLR, 2021.
Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre Oudeyer. Autotelic agents with intrin- sically motivated goal-conditioned reinforcement learning: a short survey. Journal of Artificial Intelligence Research, 74:1 1599 2022.
Edsger W Dijkstra. A note on two problems in connexion with graphs. Numerische mathematik, (1):269-271, 1959.
Adrien Ecoffet, Joost Huizinga. Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7844):580-588 2021.
11

---

Published as a conference paper at ICLR 2026
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search the replay buffer: Bridging planning and reinforcement learning. Advances in neural information processing systems, 32, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. arXiv preprini arX.v::802.0000 /80200607, 2018.
Meng Fang, Tianyi Zhou, Yali Du, Lei Han, and Zhengyou Zhang. Curriculun-guided hindsight experience replay. Advances neural information processing systems, 32, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Seungyul Han and Youngchul Sung. Amber: Adaptive multi-batch experience replay for continuous action control. arXiv preprint arXiv:1710.04423,, 2017.
Seungyul Han and Youngchul Sung. Diversity actor-critic: sample-efficient exploration. In International conference on machine learning. pp. 4018-4029. PMLR, 2021a
Sample-aware entropy regularization for
Seungyul Han and Youngchul Sung. A max-min entropy framework for reinforcement learning- Advances in Neural Information Processing Systems, 34:25732-25745, 2021b.
Christopher Hoang, Sungryull Sohn, Jongwook Choi, Wilka Carvalho, and Honglak Lee. Successor feature landmarks for long-horizon goal-conditioned reinforcement learning. Advances in neural information processing systems, 11-26963-26975. 2021.
Zhiao Huang. Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. Advances in Neural Information Processing Systems, 32, 2019.
Matthias Hutsebaut- Kevin Mets, and Steven Latré. Hierarchical reinforcement learning: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4(1):172- 221, 2022.
Yonghyeon Jo, Sunwoo Lee, Junghyuk Yeom, and Seungyul Han. Fox: Formation-aware explo- ration in multi-agent reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 12985-12994, 2024.
Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume pp. 1094-1098. Citeseer, 1993.
Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark-guided subgoal generation in hierarchical reinforcement learning. Advances in neural information processing systems, 34:28336-28349 2021.
Junsu Kim, Younggyo Seo, Sungsoo Ahn, Kyunghwan Son, and Jinwoo Shin. Imitating graph-based planning with condicondi policies. arXiv preprint arXiv:2303. 2023.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Sanghyeon Lee, Sangjun Bae, Yisak Park, and Seungyul Han. Prism: robust framework for skill- based reinforreen learning with noisy demonstrations, arXiv preprint arXi::2502.03752, 2025.
Seungjae Lee, Jigang Kim, Inkyu Jang, and H Jin Kim. Dhrl: graph-based approach for long- horizon and sparse hierarchical reinforcement learning. Advances in Neural Information Process- ing Systems, 35:13668-13366 13678, 2022.
12

---

Published as a conference paper at ICLR 2026
Seungjae Lee, Daesol Cho, Jonghae Park, and H Jin Kim. Cqm: Curriculum reinforcement learning with a quantized world model. Advances in Neural Information Processing Systems, 36:78824 78845, 2023.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv: 17/2.0094/ 2017.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.066 088299 2022.
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discover- ing and achieving goals via world models. Advances in Neural Information Processing Systems, 01124379-24399 2021.
Volodymyr Mnih, Koray Kavukcuoglu. David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312. 2013.
Ofrr Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv://100011255, 2018a.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances neural information processing systems, 31, 2018b.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv: 10618, 2019.
Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. Advances in neural information processing systems, 32, 2019.
Jongchan Park, Seungjun Oh, and Yusung Kim. Novelty-aware graph traversal and expansion for hierarchical reinforcement learning. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 1846-1855, 2024.
Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1-35, 2021.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering self-supervised reinforcement learning, arXiv preprint 10Xiv:9903.03600 2019.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015a.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint arXiv:1511.000 1.05955, 2015b.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittweeser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587)4444489, 2016.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International conference on machine learning, pp. 3540-3549. PMLR, 2017.
David Warde-Farles Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non- con-paraettric discriminative rewards. arXiv preprint arxiv: 1./ 1359, 2018
13

---

Published as a conference paper at ICLR 2026
Youngsik Yoon, Gangbok Lee, Sungsoo Ahn, and Jungseul Ok. Breadth- exploration adaptive grid for reinforcement learning. In Forty-first International Conference on Machine Learning, 2024.
Hongliang Zeng, Ping Zhang, Fang Li, Chubin Lin, and Junkang Zhou. Ahege: Adaptive hindsight experience replay with goal-amended curiosity module for robot control. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Amy Zhang, Sainbayar Sukhbaatar, Adam Lever, Arthur Szlam, and Rob Fergus. Composable plan- ning with attributes. In International Conference Machine Learning, pp. 5842-5851. Pmir, 2018.
Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model as a graph: Learning latent landmarks for planning. In International conference machine learning, pp. 12612-2220. PMLR, 2021.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. Advances in neural information processing systems, 33:21579-214 21590, 2020.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
We utilized a large language model (LLM) as an assistive tool during the preparation of this manuscript. The LLM's role was strictly limited to polishing the text, which includes improving clarity, conciseness, and correcting grammatical errors. The LLM was not used for research ideation. The authors have carefully reviewed and edited all content and take full responsibility for the scien- tific accuracy and integrity of this work. The LLM is not credited an author.
B IMPLEMENTATION DETAILS
This section presents additional implementation details of the proposed SSE framework. The graph construction method for the proposed SSE is detailed in Appendix(B.] while the training dynamics and implementation specifics of the framework are described in Appendix [..2 Each subsection highlights the design motivations and practical considerations for each module or mechanism.
B.1 GRAPH CONSTRUCTION OF SSE
In this section, we describe how existing methods define the graph = (V, E). which consists of a landmark node set V and the edge set E with edge distances d. We then explain how the proposed SSE framework constructs this graph.
Landmark Node Set V Construction
Existing HRL methods such as DHRL (Lee et al., 2022) and NGTE (Park et al.) 2024) construct the landmark node set V by selecting visited states during exploration. They employ the Farthest Point Sampling (FPS) algorithm to identify landmark nodes that are far apart from each other. This approach ensures that frequently visited regions well-represented,, as it builds V based actual agent trajectories. However,
limited to visited states, meaning unexplored or infrequently vis- ited areas cannot be selected as landmarks, slowing exploration in those regions. In contrast, BEAG YYonn et al. 2024) accelerates exploration by partitioning the goal space into a grid structure, cre- ating landmark nodes each grid intersection. This method allows for faster exploration by using virtual goal positions as landmarks, independent of visitation frequency. The structured grid layout enables more systematic and efficient exploration. To leverage this advantage and ensure struc- tured, reproducible setup, SSE adopts a grid-based landmark selection strategy inspired by BEAG. Given a 2D goal space of size x y and a grid size of dg. the landmark set V constructed as follows:
V = {(i j j. dg) E i 0,.
G
I - 1, 0,
v- -1.
y
(6)
dg
dg
For 3D goal space, the landmark node set is defined V {(i dg. dg. k dg) G
x 1, j ..... y - 1, h 0, ... وة - 1}. By constructing landmark nodes in 0,... dg this grid-based manner, SSE achieves faster and more structured exploration compared to visitation- based methods, ensuring efficient path planning and reliable subgoal execution.
dg
Definition of Edge Distance
Given the landmark node set V, the edge set E is defined the collection of distances d(u 02) between any two nodes v1, 02 E V. For previous graph-based RL methods, the edge distance d is computed as d(vi + 02) = log,, (1 (1- (v), 02, (2)7))) where 7 the low-level discount factor used for training the low-level policy n²., and Q is the value function estimating the traversal cost from to Uo described in Section Existing HRL methods like DHRL (Lee et al. 2022) and NGTE (Park et al. 2024) directly use the lew-level value function Q', which is trained with a step- based reward of to define the distance as the expected number of steps required for the lowelevel policy to navigate from en to U2. Although this method reflects actual navigation costs, it is sensitive instability during Q' training, resulting in fluctuating edge distances. In contrast, BEAG Yoon et al., 2024) measures the distance using the Euclidean norm dg = ||ا wall. providing a stable but less accurate representation of traversal costs. To leverage the strengths of both approaches, SSE defines the edge distance d(v, - U2) by combining Q' and dæ as follows:
d(u1 - U2)) = [log,r (1+ (1 10-711 (1->/1e(() (V1, U2, x')) + log, (1 (1 - -))e)(12))))) (7)
15

---

Published as a conference paper at ICLR 2026
This hybrid formulation allows SSE to benefit from the stability of Euclidean distances when Q' is not fully converged, while still capturing the true traversal cost as Q' improves. As a result, the edge set E is constructed E {d(v) + 02) v 02 E V). Here, d represents the edge distance before failure-aware path refinement is applied, ensuring both stability and adaptive accuracy in path estimation.
B.2 DETAILED Loss FUNCTIONS AND IMPLEMENTATION OF THE SSE FRAMEWORK
RI. training losses As described in Section the proposed SSE framework is an HRL structure that employs high-level policy ,² an exploration policy πoP,, and a low-level policy n'. The exploration policy does not require separate parameterization for high-level actions, whereas T² and n' are parameterized by 8h and 0', respectively, and are represented as Tgh and T/.. To evaluate these policies, SSE defines the parameterized high-level value function Q and the low-level value function 2a where t and vi are the respective parameters. As mentioned in Section 2 the high- level policy and value function are trained maximize external rewards, while the low-level policy and value function are optimized to reach designated waypoints incrementally. To facilitate this, the low-level policy receives the reward r/ at each time step, defined as follows:
rf L if [|0(s++1) - - < X, (agent reaches the current waypoint) otherwise,
(8)
where À is the reachability check threshold, wp; is the target waypoint at the current timestep t, and (wp..... wp,)) represent the shortest path from ((*)) to the subgoal Jt- The high-level and low-level policies, along with their value functions, FER B% and the low-level buffer B' through the TD3 algorithm (Fujimoto et al. 2018), a standard off-policy RL method. The value function losses for high-level and low-level policies are defined as follows:
trained using the transitions stored in the
(15") -E
(QM(se.)))
- +7 min Q (su Tos (se. (Bc.9))) )))))
min O'sy (S641, The ($141, (sc+1, (or (ol(es -rr (
27
Egg
(9)
112
where l' denotes the termination time of the low-level path execution, which is variable as the step concludes only upon success or failure. In the case of a partial success, twp (used to store transitions in FER) indicates the time step when the agent reached the last successful waypoint WPfinal- The high-level reward is then defined as r Ej= for a successful trajectory and r = O for a failed one. The terms Th and e are target network parameters updated via exponential moving average (EMA), and 7 and 7 are the discount factors for training the high-level and low-level policies, respectively. The actor losses for optimizing the policies defined as follows:
Cax -B [ (st, Ty. (5).9)) ((') EEg' Qeu (St. Tgg (St, wp:))] (10)
The parameters are optimized using the Adam optimizer (Kingma 2014) to minimize the respective loss functions. SSE distinguishes between the high-level discount factor 7d and the low-level dis- count factor n setting 7 0.99 as typical in RL, and 7² 0.4 to limit return propagation across high-level steps, encouraging shorter path optimization. In the initial stages of training, rewards are sparse. To address this, the FER B% is divided evenly, with half storing successful trajectories and the other half storing trajectories with zero reward. This design improves learning signals from success- ful experiences. The exploration ratio, which controls the balance between the exploration policy and the high-level policy, starts at 1:0 and decays by 0.05 per iteration until it reaches 1y:(1 (1(1))), as outlined in Section 4 This scheduling promotes exploration initially and shifts the focus high- level learning as training progresses. If the low-level agent fails to reach the high-level subgoal ğ., the trajectory is marked as failed. In cases where the agent becomes stuck, such as flipping over or hitting obstacles, its position may remain unchanged for long periods. To improve sample effi- ciency, if no movement is detected for 500 steps, the trajectory classified as failed and the episode is terminated. The complete SSE algorithm is provided in Algorithm
16

---

Published as a conference paper at ICLR 2026
Model training losses for SSE (Model) We now describe the implementation details of the pre- diction network fc. which computes the prediction error used to determine the novel region in the model-based approach, and the failure predictor network Fg,, which estimates the failure ratio of target nodes.
In the model-based variant of the exploration policy, estimate the novel region novel using Random Network Distillation (RND) (Burda et 2018), which was originally proposed in standard RL identify novel states based on prediction error. Following the RND formulation, target network [<qurx V + Ie and a predictor network fc - I& with parameters Ç. The predictor fc is trained on visited sample nodes stored in the buffer to regress the target output via the mean squared error objective
use fixed
LRND(() Deiniind [Ilf((o) - (v)1
(11)
As the predictor network is trained to minimize the prediction error on visited nodes, the prediction error becomes small in frequently explored regions and remains large for underexplored nodes. We
therefore use the squared prediction error [)(()) fa (v)]l as a novelty score, and nodes with larger error are treated targets, the model-base -ased variant is encouraged to visit underexplored regions of the goal space.
novel. By selecting nodes with high prediction error exploration
For failure- aware refinement, train failure predictor network Fg: V [0, 1] parameterized by S. which outputs the estimated failure probability of each node. The network is trained using node samples stored in buffer D.., where each node paired with binary label y {0, 1} indicating whether the trajectory containing v failed (y = 1) or succeeded (y = 0). We optimize E'g with the binary cross-entropy loss
(E) = - (E ly (U)) + +1 - y) log Fe("))]
(12)
Through this training, (v) approximates the empirical failure ratio node v, which we use the model-based failure estimate in our refinement scheme.
Algorithm Strict Subgoal Execution (SSE)
Input: Graph G (V, goal g, mapping function o, threshold A, exploration ratio 7) Initialize: Policies n², π exp, T², buffers B4, B¹, grid cells Cg (Grid) or Models fc, F' (Model)
for each iteration do for each episode do
Select the behavior policy: ग +- exp with probability y, otherwise π <- nh; for each high-level selection step do
Sample subgoal 9t x (*+.g)
Plan the waypoint path wP1.1 from (sx)) to gr using Dijkstra's algorithm over G with d for i = 1, n do
Roll out the low-level policy nf (8t, wp:) to reach the waypoint wP; Store the
transitions (82; WP,, az Te, 8641) into the low-level buffer B
Compute the reward sum: r Σ T3
Construct the FER:
[[0(s++1) - ge < then
Success: Store the success transition (s.g. g: Ge, Fsum; Sp.) into FER B
Failure: Store the stop-on-failure transition (s, 9. Gt. 0, ST) into FER B/: else
Store the partial success transition (s, g. WPfinal: Ejup, -1 Terminate the episode
rj. Stwp) into FER B
end
end
end
end
Update total visitation count (CC))) and failure count Ntai (CT) for all CT (Grid-based)
Update model parameters Ç. & to minimize LRND(() and LF(E) (E) (Model-based) Update 1h 8h using samples from FER B to minimize Lor (2i Cyp (8h))
Update 44.00 of using samples from B' to minimize Lo (w²), L (0')
end
17

---

Published as a conference paper at ICLR 2026
C EXPERIMENTAL SETUP
Our proposed framework is designed to be modular and general, enabling integration with wide range of baseline methods. For comparison, we employ the official codebases provided by the orig- inal authors for HIRO, HRAC, HIGL, DHRL, NGTE, and BEAG. All baselines are run using the hyperparameters specified in their respective publications, and conducted on an NVIDIA RTX 3090 GPU with an Intel Xeon Gold 6348 CPU (Ubuntu 20.04). Appendix [c.] provides descriptions of the baseline algorithms along with links to their official code repositories. Appendix C.2 provides the specifications for the nine environments shown Fig including their action and observation spaces, goal configurations, and episode horizons. The SSE-specific hyperparameter configurations for each environment are summarized in Appendix
C.1 DETAILS OTHER BASELINES
HIRO (Nachum et al. 2018b) introduces hierarchical architecture with relabeling of high-level transitions to account for changing low-level policies, thereby improving off off-policy sample efficiency and stability. Open-source code of HIRO is available
https://gihub.com//weer hww.bboooo/waaaaandiiinnoogo __too_pyttchhhhhh
HRAC Zhang et al.) (2200) adds a learned adjacency constraint to ensure subgoal feasibility. It penalizes high-level selections that attempt transitions deemed unreachable within limited hori- zon, thereby guiding the agent to learn feasible subgoal structures. Open-source code of HR is available at
HIGL (Kim et al. 2021) incorporates a coverage-driven and novelty-driven landmark selce- tion strategy. It performs graph-based planning via shortest paths and uses adjacency rewards to guide learning toward under-explored regions. Open-source code of HIGL available at https://iitub..cmm/juussuuiiiiip
DHRL (Lee al., 2022) constructs goal graph using Farthest Point Sampling and learns high-level behavior by planning over the graph. It emphasizes temporal abstraction and long- horizon planning via graph traversal and Q-learning. Open-source code of DHRL available official
BEAG Yoon et al. 2024) employs value- futte-function-driven imaginary landmarks to facilitate ex- ploration of unvisited areas. estimates landmark distances from a learned value function without relying solely on previously visited states, enabling efficient generalization. Open-source code of BEAG available at https//ggitubb.oommmmppsstte////
NGTE (Park et al., 2024) drives novelty-based exploration by identifying frontier nodes (outposts) and prioritizing expansion toward less-visited regions of the goal graph, en- couraging broad and diverse exploration. Open-source code of NGTE is available at https//ggthbb.com/laaeeeeccciiit
C.2 ENVIRONMENTAL DETAILS
We follow standard benchmarks and configurations widely adopted in prior hierarchical rein- forcement learning studies (Lee et al., 2022 Park et al., 2024 Yoon et al., [024), and introduce several environments designed evaluate high-level recision-making capabilities such as AntKeyChest or Ant BeDDDobleeeyClce Keehhs The environments used in our experiments are vi- sualized in Fig. and their characteristies described in Table
AntKeyyhest and IntDoubleeeeChees include key flags that are toggled from to when the agent successfully reaches the corresponding key location under predefined condition. www.heraallDoublleee....
contains two distinct goal positions, and two goal vectors are pro- vided along with corresponding success flags, which are set upon reaching each goal. The success threshold for determining whether a target is reached is 5 in the AntMaze environments and 0.25 in the Reacher environments.
C.3 HYPERPARAMETER SETUP
Table 2 summarizes the common hyperparameters used across all environments. These configura- tions are based on a combination of parameter tuning and default settings from baseline implemen- tations. In partieular, the buffer size, batch size, and network architecture follow the original baseline
18

---

Published as a conference paper at ICLR 2026
Table 1: Summary of environment specifications.
Environment
Spaces (Obs / Action)
Goal space
Reward
Start / Goal Po- sition
Episode
length
U-Maze
Obs: 29-Dof Action: 8-Dof
[-4,20]x
1 if goal reached
Start: (0,0)
600
[-4,20]
else 0
Goal: (0.16)
r-Maaze
Obs: 29-Dof
[-4,6]]x
if goal reached
Start: (8,0)
1000
Action: 8-Dof
[-4.36]
else 0
Goal: (24,0)
AntMazeComplex Obs: 29-Dof
[-4,2]]x
1 if goal reached
Start: (0.0)
2000
Action: 8-Dof
[-4,52]
else 0
Goal: (40,0)
AntMazeBottle- Obs: 29-Dof
[-4,20]x
if goal reached
Start: (0,0)
600
neck
Action: 8-Dof
[-4,20]
else 0
Goal: (0.16)
AntMazeDouble- Obs: 29-Dof
[-4.20]x
if goal reached
Start: (0,0)
1200
Bettleneck
Action: 8-Dof
[-4,66]
else 0
Goal: (16,32)
AntKeyChest
Obs: 30-Dof
[-4,66]x
if key reached
Start: (0,0)
2000
Action: 8-Dof
[-4,66]
if goal reached
Key: (0.16)
with key
Goal: (0,32)
AntDouble- KeyChest
Obs: 31-Dof
[-4,36]
key
Start: (0,0)
3000
Action: 8-Dof
[-4,36]
reached reached
Keyl: (16, 32)
key2
Key2: (16, 0)
Goal: (32,
with keyl
5 if goal reached
with two keys
ReacherWall
Obs: 17-Dof
[11,]|x
if goal reached
Start:
100
Action: 7-Dof
[1,1]]
(0.99, -0.19, 0)
[1,1]]
Goal: (0.6, 0.6, 0.1)
ReacherWall-
Obs: 22-Dof
[1,1]x
if goal reached if both goals reached
Start:
200
DoubleGoal
Action: 7-Dof
[1,1]x
(0.99, -..19,
[-1,1]
Goall:
(0.4, 0.4, -0.1)
Goal2:
(0.4, 0.8, 0.1)
(a) U-maze
(b) и-maze
(c) AntMaze Complex
(d) AntMaze Bottleneck
(e) AntMaze DoubleBottlenec
(f) AntKeyChest
(g) AntDouble KeyChest
(i) ReacherWall DoubleGoal
(h) ReacherWall
Figure 9: Considered long-horizon environments: 5 AntMaze, 2 KeyChest, and 2 Reacher tasks
code. Learning rates for each policy level, as well as the discount factors, were determined through structured parameter search.
19

---

Published as a conference paper at ICLR 2026
Table presents auvironment-specinn hyperparameters, including the minimum epsilon threshold Cmin- the path refinement scaling factor Cdist. the reachability check threshold X, and the grid resolu- tion dg. As mentioned in Section 2.2, the threshold is the standard, environment prreentdproviedd value used to check for low-level reachability (see equation 8), which is also used consistently across baselines. These values selected based targeted parameter searches conducted for each en- vironment.
In the case of LntDDuubleeyCChett a higher value of émin is required compared other en- vironments. This because the task involves discovering multiple intermediate objectives, which increases the need for broad exploration. Other parameters such as the exploration ratio 1 and the refinement scaling factor Cdist were also selected through environment-speiffe tuning, with further analyses reported in Appendix F.2
Table 2: Common hyperparameter settings used in SSE.
Hyperparameter
Value
Optimizer
Adam
Replay buffer size
2,500,000
Batch size
1024
ls High-level actor learning rate actoo 0.000005
High-level critic learning rate h Ceriti 0.00005
Low-level actor learning rate of "ecoo 0.0001
Low-level critic learning rate a eritic 0.001
High-level discount factor 7/h Low-level discount factor 7 Target update rate т
0.4
0.99
0.005
Table 3: Scenario-specine hyperparameters for SSE,
Scenario
Emin
Cdist
dg
AntMaze
U-maze
0.1
5.0
0.5
2
0.1
AntMazeBottleneck
0.1
5.0
0.5
0.1
T-maze
0.1
5.0
0.5
2
0.2
AntMazeComplex
0.1
5.0
0.5
hntharaeoobbleeBtttlleeeee
0.1
10.0
0.5
2
0.1
AntKeyChest
0.1
5.0
0.5
0.2
AntDoubleKeyChest
0.2
5.0
0.5
2
0.2
Reacher
ReacherWall
0.1
5.0
0.25
0.2
Reacher:allloouleeeall
0.1
5.0
0.25
0.2
D ADDITIONAL COMPARATIVE EXPERIMENTS
This section presents additional comparative experiments to further assess the generality and com- putational efficiency of the proposed SSE framework. We evaluate performance in random-godl training setup Appendix [D.I] where goals are sampled uniformly from the entire reachable state space. Appendix D.2 evaluates computational characteristics, including convergence speed and per- episode compute time.
D.1 PERFORMANCE COMPARISON UNDER RANDOM GOAL SETUPS
Figure I0 presents experimental results under a random goal setting where goals are sampled uni- formly from the entire valid state space during training. This setup contrasts with the fixed-goal
20

---

Published as a conference paper at ICLR 2026
scenarios discussed in the main text. The reward signal remains sparse and poses significant chal- lenge for goal discovery and policy optimization. Graph-base bash- methods such as BEAG, NGTE, and SSE maintain strong performance even under random goal sampling. These methods autonomously expand their subgoal graph during exploration and achieve success rates comparable to those in the fixed-goal setting. DHRL also benefits from the randomized goal distribution, partieularly in simpler environments like U-maze and AntMazeBott Leneck where the agent encounters train- ing signals more frequently. Conversely, methods such as HIRO, HIGL, and HRAC exhibit limited progress in most environments under random- -oaall conditions. This performance drop stems from their reliance on fixed-frege freeueryyyncy subgoal selection and the lack of structured exploration mecha- nisms suitable for sparse reward settings. In environments requiring multi-stage reasoning such as AntkeyChest, SSE the only method that consistently discovers the key and solves the full task. While NGTE is hierarchical and exploration-drive, it only occasionally learns to acquire the key and exhibits a very low overall success rate under this configuration.
1.0
1.0
3.0
Success Rate
Succcess Rate
0.6
0.6
0.6
0.5
0.2
02
0.2
0.2
%
0.
Timestep
Timestep
Timestep
Timestep
(4) U-maze
(b) AntMMazeBottteeeck
(4) ArttMazeCoopless
Antseeyhhhstt
5SE
BEAG
NGTE
DHRL
HRAC
PIG
HIRO
HIGL
Figure 10: Performance comparison in random goal setting
D.2 COMPUTATIONAL COMPLEXITY COMPARISON
Table 4 reports the average per-episode computation time and the number of episodes required to reach a 60% success rate. We compare SSE against NGTE and BEAG as they are the only baselines that succeed in at least of the considered fixed-goal settings. This makes them the most relevant references for evaluating performance and sample efficiency in sparse reward environments. SSE achieves faster convergence across tasks due to its early termination of episodes upon subgoal failure and a reduced number of transitions per episode. These features minimize unnecessary computation and accelerate learning.
This advantage is particularly evident in Long-horizun tasks such as AntKeyChest, these tasks, conventional HRI. methods consume many timesteps during failed attempts and require tended horizons for high-level planning. In shorter-horizon scenarios like AntMazeBott leneck, the computational benefits of SSE less significant since subgoal transitions and failure termina- tions occur less frequently. The operations introduced by SSE, including the evaluation of subgoal completion and uniform sampling, are lightweight and have linear time complexity. Consequently, SSE imposes minimal computational overhead while maintaining stable training dynamics.
Table 4: Average per-episode computation time (in seconds) and the number episodes required to reach 60% success rate.
Scenario
SSE
BEAG
NGTE
5,94s
6.46s 425.2 episodes
12.65s
AntMazeBottleneck
432 episodes
725.6 episodes
18. 13s
57.29s
119.07s
AntKeyChest
1658 episodes
fail
fail
E EXTENDED ANALYSES OF THE PROPOSED SSE FRAMEWORK
To further validate the design and generality of SSE, this section presents extended analyses across several dimensions. Appendix E. provides qualitative trajectory visualizations that illustrate how
21

---

Published as a conference paper at ICLR 2026
SSE dynamically adapts its subgoal planning time. Appendix E.2 presents ablation and sensi- tivity analyses to isolate the contribution of individual components and hyperparameters, including the path refinement scale Cdist, grid resolution dg, and exploration ratio 7/.
E. TRAJECTORY ANALYSIS IN OTHER ENVIRONMENTS
To gain deeper insight into the behavior of SSE, present trajectory analyses across representative environments.
Fig. 11 illustrates learning progression in AntMaazBotttleeee In the early stage (a), the agent has not yet discovered feasible subgoals, causing the high-level policy x² to select subgoals largely random. During this phase, the exploration policy promotes coverage by gradually expanding into novel regions. In the mid stage (b), the agent fails to traverse the narrow bottleneck, primarily because the low-level policy n' lacks sufficient experience that region and the associated failure statistics remain underrepresented. By the final stage (c), SSE successfully leverages failure-awe aware refinement and subgoal selection. This allows n/ navigate through the bottleneck and reach the goal reliably.
N(C
(a)
(c)
Start position
Goal position
Subgoal 9max
Novel region
Grid C
Figure 11: Trajectory analysis of SSE subgoals Imax in AntMazeBott across: (a) early
stage, (b) mid training, and (c) task success.
A similar trend is observed in most regions are unexplored, and subgoal selection remains uninformed. In the intermediate phase (b), the agent again struggles with the second bottleneck due to insufficient failure feedback. As training progresses (c), subgoals selected by T² become more effective. The updated path refine- ment guides the agent through safer routes, enabling successful traversal of both bottlenecks and completion of the long-horizon task.
shown in Fig. 12 Initially (a),
N(C
5
bb)
(c)
Start position
Goal position
Subgoal 6max
Novel region
Grid C:"
Figure 12:
Trajectory analysis of SSE subgoals πmax in AutharceDoub.eeottLeeeeeee across: (a) early stage, (b) mid training, and (c) task success.
22

---

Published as a conference paper at ICLR 2026
In Fig. 13 the agent in AntKeyChest begins by exploring the environment without clear objec- tive (a). Upon discovering the key (b), n increasingly selects subgoals leading to the key location. In the final stage (c), the agent demonstrates the ability sequentially reason over subtasks, first reaching the key and then navigating to the final goal with the required flag active, completing the task successfully.
N 5
(a)
(b)
Start position
Goal position
Subgoal 9max
Novel reglon
Cw
Keys
Grid C2
Figure 13: Trajectory analysis SSE subgoals Jmax ReyCest across: (a) early stage, (b) after reaching the key, and (c) reaching the goal with the key (task success).
E.2 ADDITIONAL ABLATION STUDIES
Component Evaluation
Fig. 14
presents the performance of SSE variants with key components ablated. We evaluate four variants: (1) SSE w/o By which replaces FER with a standard replay buffer, (2) SSE with HER which conventional fixed-step policy, (3) SSE w/o ex which disables decoupled exploration, and (4) SSE w/o PR which omits path refinement. Removing path refinement degrades performance long-horizon settings where mitigating unreliable transitions is crucial. Similarly, disabling the exploration policy consistently impairs goal-space coverage across all environments. Critically, SSE with HER and SSE w/o B fail in complex tasks like Ant DeeDDeubeeeeyeeessss despite achieving minor success in simpler maps. This failure demonstrates that conventional HER offers insufficient feasibility signals. Furthermore, confirms that FER vital for providing stop-on-fai failure feedback to teach the high-level policy to avoid unreliable subgoals. These components collectively ensure the decisive signals required for long-horizon planning.
1.0
1.0
1.0
0.8
0.8
D.8
Success Rate
Success Rate
Success Rate
0.6
0.6
0.6
0.4
0.2M 0.4M 0.6M 0.8M 1.0M Timestep
0
1M
2M
3M
4M
Timestep
Timestep
(a) AntMazeBottleneck
AntkeyChest
(c) AntDoublekeyyhest
SSE lours)
SSE w/o x3
SSE HER
SSE wro 7
SSE w/o PR
Figure 14: Component evaluation results in various maps
Effect of Path Refinement Scaling Factor Caist
Fig. 15 shows how the scaling factor Caist affects performance. This parameter controls the trade- off between path efficiency and safety. In simpler tasks like AntMazeBottlenee its impact minimal. In more complex, long-horizon tasks, its role becomes more pronounced. A moderate
23

---

Published as a conference paper at ICLR 2026
value, such as Cdist 5, provides effective balance, guiding the planner away from failure-pree prone regions without being overly conservative. In contrast, excessively high values (c.g. 10 or 20) can lead to inefficient detours, while value of may not sufficiently penalize risky paths. Overall, the performance remains high for values between 2 and 10, indicating a wide effective range.
1.0
1.0
1.0
Ga
G
0.8
Grae5
0.8
Capm
Gaue5
Success Rate
Caun20
Gae=10
Success Rate
Cay=10
Cau
GeuR29
0.6
Success
0.4
0.2
%
%
Timestep (a) AntMazeBottlenec
Timestep
Timestep
(b) AntkeyChest
(c) AntDoubleKeyyhest
Figure 15: Distance scail Cdist analysis in various maps
Effect of Grid Size dg
Fig. 16 illustratsss the effect grid resolution dg. This parameter balances the granularity of novelty detection with the stability of failure statistics. The analysis shows that performance is not highly sensitive to this parameter within the tested range. For instance, in AntMaze environments, there is may
little significant difference performance for dg values of 1, 2, and 4. coarse grid (dg merge distinct regions, while an overly fine grid (dg = 1) can make failure statistics less reliable. Thus, we find that a moderate resolution (dg for AntMaze) provides a suitable balance. The optimal choice is dependent on the environment's 's scale, with smaller, structured spaces like Reacher benefiting from a finer grid (dg = 1).
4,=2
Success Rate
Success Rate
0.6
Success
0.4
0.4
0.2
0.2M 0.4M 0.6M 0.8M 1.0M
%
1M
2M
4M
5M
%
1M
2M
3M
4M
5M
Timestep
Timestep
Timestep
(a) AntMazeBottleneck
(b) AntKeyChest
(c) AntDoubleKeyChest
Figure 16: Grid size dg analysis various maps
Effect of Exploration Ratio
Fig. shows how performance varies with the exploration ratio d, which controls the clas- sic exploration-exploitation trade-off. In simple environments like AntMazeBottlenede performance is consistent across a wide range of values. In long-horizon tasks like AntDoubleXeyCln
that require discovering intermediate objectives, the influence of 1/ natu- rally greater, but effective performance can be achieved with straightforward tuning. As shown in our experiments, setting the exploration ratio within small range of 0.1 to 0.2 is sufficient achieve strong performance across all tasks. This indicates that while the balance is important, extensive tuning of this hyperparameter required.
24

---

Published as a conference paper at ICLR 2026
1.0
1.0
1.0
0.8
0.8
0.8-
Success Rate
Success Rate
Rate
0.6
0.6
0.6-
0.4
0.4
0.4
0.2
0
0.2M 0.4M 0.6M 0.8M 1.0M
%
1M
1M
2M
3M
4M
5M
Timestep
Timestep
Timestep (e) AntDoubleKeyyhest
(a) AntMazeBottteneck
AntKeyChest
Figure 17: Exploration ratio 1) analysis in various maps
Effect of Reachability Check Threshold À
Fig. 18 illustrates the performance variation with respect to the reachability threshold X. We employ the standard environment
perroimeet--roviddedd provided value the default. Performance remains stable as long as X smaller than the final task goal threshold, ensuring sufficient precision for intermediate way- points. Conversely, extremely small values like 0.1 degrade performance in long-horizon tasks by foreing the low-level policy to achieve excessive precision. This hinders efficient exploration. These results confirm that the framework is robust to and that the standard environment value offers an optimal balance.
1.0
1.0
1.0
0-0.3
A-0.1
A-0.1
2-O.5
A-0.5
A-D.5
0.8
2-1
0.8
2-1
A-1
Success Rate
2-2
Success Rate
2=2
A+2
Success
0.2
0
0.2M
%
%
4M
5M
Timestep
Timestep
Timestep (c) AntDoublekeyyhest
(a) AntMazeBottteneck
(b) AntkeyChest
Figure 18: Distance threshold À analysis in various maps
Effeet of High-Level Gamma 7ht Fig. 19 confirms that 7 0.4 yields optimal performance. Theoretically, SSE enforces strict, single-step reachability, which inherently condenses the high- level decision horizon. In this regime, a standard high discount factor (e.g., 7"h = 0.99) fails to distinguish between immediate and delayed successes. Conversely, lower 9th sharpens credit signment, making the value function Q" highly sensitive to temporal inefficiencies. This sensitivity guides the policy prioritize direct, high-probability transitions, effectively delineating the reacha- bility frontier and accelerating convergence.
1.0
1.0
Success Rate
0.8
y'=0.
Success Rate
0.8
0.5
0.6
0.6
0.3
0.4
0.4
0.1
0.4M
0
2M
Timestep
Timestep AntkeyChest
Timestep
(a) AntMazeBottieneck
(c) AntDoubleeeyyhest
Figure 19: High-level gamma ntt analysis in various map
25

---

Published as a conference paper at ICLR 2026
E.3 COMPARISON ON ADDITIONAL ENVIRONMENTS
To assess generalizability, we conducted experiments on AntPush, AntFall, and 4 Since conventional baselines like HIRO failed to learn in these settings, foeus our comparison on the graph-based methods BEAG and NGTE. As shown in Fig. 20 standard graph-based algo- rithms struggle in tasks with complex contact dynamics invisible obstacles. Their rigid reliance on shortest-path planning often leads to failure in environments like AntPush and AntFall. In contrast, SSE demonstrates superior performance. We attribute this to the decoupled exploration policy Txp This mechanism allows the high-level policy x" to leverage diverse exploration data beyond the graph's shortest path, facilitating robust planning under complex physical constraints. Note that in AntPush, we froze graph edge updates after initial convergence to mitigate instability from object interactions. SSE maintained its performance advantage even under this stabilization protocol.
0.5
0.5
0.5
0.4
0.4
0.4
Success Rate
Success Rate
Success Rate
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0
0.2M 0.6M
0
0.4M
0.8M
0
1.0M
Timestep
Timestep
Timestep (c) Ant4Room
(a)) AntPush
(b) AntFall
Figure 20: Performance comparison on additional environments
E.4 MORE RANDOM SEEDS
To validate the statistical stability of our results, we supplemented our initial experiments (run with 5 seeds, a common practice in this field) by re-running the key complex tasks with 10 random seeds. We focused on tasks like AntKeyChest and AntDoubleKeyChest (Fig. 6, Fig. 8), where the standard deviation range of SSE with 5 seeds already showed clear separation from the baselines.
As shown Fig. 21 the experiment with 10 seeds produces a nearly identical performance curve, and no significant change in the mean success rate was observed compared to 5 seeds. More impor- tantly, we re-affirmed that the standard deviation bounds remained clearly separated from the base- line algorithms. This confirms that the superior performance of SSE was not achieved by chance with a low seed count and that our findings are reliable and statistically stable.
1.0
1.0
SSE: 10 sood
SSE 10 sead
5SE 10 seed
SSE
SSE -5 a002
258
seed
0.8
0.8
Success Rate
Success Rate
Success Rate
0
0.2M 0.6M 0.8M 1.0M
0
%
Timestep
Timestep
Timestep
(a) AntMazeBottieneck
(b) AntkeyChest
(c) AntDoubleKeyyhest
Figure 21: Performance comparison of SSE with seeds
26


Reviews:
[{"review_id": "3dlS5pVlrg", "reviewer": "Reviewer_cF88", "summary": "The paper mentioned that in long-horizon, sparse-reward tasks, high-level policies often choose subgoals that the low-level controller can’t reliably reach. When HER is applied at the high level, failures get relabeled. The authors propose Frontier Experience Replay (FER), which stores three kinds of high-level transitions—success, stop-on-failure (zero-return, early termination), and partial success to the last reliably reached waypoint. They introduce a decoupled exploration policy that prioritizes under-explored goal-space regions (simple grid-density estimator) alongside an ε-greedy high-level policy to improve coverage. The authors add failure-aware path refinement that inflates edge costs in high-failure regions of the goal graph, nudging Dijkstra planning away from unstable corridors. SSE substantially outperforms HRL (HIRO, HRAC) and graph-based methods (HIGL, DHRL, NGTE, PIG, BEAG) on success rate and often on learning speed", "questions": "This is a strong empirical paper with a simple, well-motivated idea that addresses a real failure mode in hierarchical goal-conditioned RL. Have you thought about how to extend to analyze theoretically with HER?\nHow often does path refinement genuinely alter the planned path (e.g., fraction of episodes where the refined path differs from shortest path)?\nhave you tried replacing grid-based novelty with learned density?if so, do the gains persist?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strong empirical results on a diverse suite, including tasks that require implicit sequencing\nAblation coverage is thoughtful: removing FER or replacing with HER largely breaks performance on harder tasks", "weaknesses": "The density estimator and failure statistics hinge on a grid. This is fine for 2D/3D but will be problematic in higher dimensions and for goals that include orientation or other factors.\nNo guarantees or formal properties regarding convergence or bias introduced by early termination/FER.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fQqxaMMtHK", "reviewer": "Reviewer_kJYV", "summary": "This paper has proposed a novel goal-conditioned hierarchical reinforcement learning approach, called SSE, to achieve reliable long-horizon planning. The proposed approach is evaluated in a set of simulated navigation tasks.", "questions": "1.\tDoes the proposed method assume that the goal space is low-dimensional? This may not be true for complex manipulation tasks.\n\n2.\tIs the proposed method applicable to real-world tasks? Only conducting experiments in simulators is not convincing enough.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "•  The paper addresses an important problem in hierarchical reinforcement learning — unreliable subgoal execution — which is crucial for long-horizon tasks.\n\n•  The introduction of Frontier Experience Replay (FER) is conceptually clear and provides a principled way to delineate reachable and unreachable subgoals, improving training stability.", "weaknesses": "•  The proposed framework assumes that the goal space is known and low-dimensional, which may not hold for complex real-world manipulation or visual tasks where the goal representation itself is high-dimensional and uncertain.\n\n•  The technical novelty is moderate — SSE combines known components (graph-based HRL, experience replay, path cost reweighting) rather than introducing fundamentally new learning principles.\n\n•  All experiments are conducted in simulators; there is no validation in real-world robotic systems, limiting the practical credibility of the claimed reliability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GlzO9LM2vz", "reviewer": "Reviewer_BjYj", "summary": "This paper introduces Strict Subgoal Execution (SSE), a hierarchical reinforcement learning framework designed for long-horizon, goal-conditioned tasks with sparse rewards. SSE incorporates Frontier Experience Replay (FER) to identify and filter out unreachable subgoals, improving the reliability and efficiency of high-level planning. It also uses a decoupled exploration policy to better explore the goal space and a path refinement mechanism to adjust edge costs based on low-level failures. Experiments on multiple benchmarks demonstrate that SSE achieves higher efficiency and success rates than existing goal-conditioned and hierarchical RL methods.", "questions": "Would it be possible to include comparative studies on a broader range of environments, using a larger number of random seeds as suggested?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript is clearly written, with figures that effectively elucidate the proposed methodology and equations that comprehensively convey its technical details.\n\n2. The experimental design is well structured, incorporating appropriate selections of baselines. The ablation study provides a thorough examination of the contribution of each component and offers a detailed analysis of the method’s sensitivity to hyperparameters.", "weaknesses": "1. To make a stronger case for the method's generality, the paper should include results from a broader set of environments. Specifically, in line with the existing HRL works, the authors may want to evaluate the method on Pusher, AntFall, AntGather and Ant4Rooms, in addition to the relatively simpler maze-based tasks. This would provide a better understanding of how well SSE adapts to various state spaces and task complexities. Including additional tasks would also help demonstrate whether the method can avoid unstable regions and enforce subgoal completion.\n\n2. The use of only five random seeds in the comparative study may be insufficient to establish statistical significance. Increasing the number of seeds would lead to more reliable and robust experimental conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0jX0L5JHi4", "reviewer": "Reviewer_dYta", "summary": "This paper proposed a novel method for hierarchical reinforcement learning, which leverages a frontier-based and failure-aware meta-policy replay buffer to ensure that goals that are reachable and at the frontier of experience are sampled. The method keeps track of this information with a graph-based encoding of the nodes in the environment. To ensure proper exploration, they leverage an exploration policy that is aware of all grid cells in 2D/3D space, and samples goals (at least partially) from these cells. \n\nThe main contribution are the novel replay buffer, and the method for having the graph be aware of what paths will fail, and having exploration be decoupled for goal space coverage.", "questions": "- how is $\\lambda$ set? I'm surprised there's no ablation on this.\n- the Strict Subgoal Execution (SSE) framework updates the high-level policy with positive returns only when the low level successfully reaches the assigned subgoal. This seems like the same high-level ideas as [1]. Could you contrast to this work? This seems like a graph-based version of that idea?\n- when comparing to methods like HIRO, what steps do you take to ensure that HIRO can use similar assumptions as this method, e.g. access to grid discretization scheme?\n- isn't having exploration be decoupled from goal space coverage a common strategy in this graph-based planning setting, e.g. [2], which you did not cite.\n\n[1] Self-Imitation Learning\n[2] Successor feature landmarks for long-horizon goal-conditioned reinforcement learning", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- A key strength of their method is that it learns transitions between goals, enabling more efficient planning (as implied by Figure 1). It appears that they are effectively learning a high-level policy.  \n- The method achieves strong training performance compared to baseline approaches.  \n- The evaluations demonstrate that each component of the algorithm plays an essential role in achieving good performance in the environment.  \n- The experiments clearly illustrate the benefits of the proposed components (e.g., Figures 2 and 3).  \n- The overall presentation is clear and well-explained.", "weaknesses": "- \"We assume the existence of a mapping φ such that φ(s) ∈ G, allowing the agent to infer goal progress from the current state.\"\n  This seems like a strong assumption. Is this mapping learned, or predefined? Clarifying this would help understand how generalizable the approach is.\n\n- The algorithm appears to involve substantial hand-crafting, which may limit its applicability to more complex or continuous environments. For instance, sampling from a grid-based estimator may not scale beyond grid-world settings.  \n  Similarly, assuming that subgoal reachability can be determined by $||\\phi\\left(s_{t^{\\prime}}\\right)-\\tilde{g}_t||<\\lambda$ is a strong and potentially unrealistic assumption.  Using this grid discretization for failure awareness also seems restrictive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)