Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals
Authors: Vivienne Huiling Wang
Tinghuai Wang
Joni Pajarinen
Venue: ICML 2025 poster (2025)
Keywords: Hierarchical Reinforcement Learning
Primary area: reinforcement_learning->deep_rl
TL;DR: GP-guided diffusion models generate more stable and efficient subgoals in off-policy HRL by combining the diffusion model's flexibility with the GP's uncertainty-aware feasibility, improving performance on complex tasks.

Abstract: Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of temporal abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy to generate effective subgoals. To address this issue, the high-level policy must capture a complex subgoal distribution while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a complex variety of subgoals while leveraging principled GP uncertainty quantification. Building on this framework, we develop a strategy that selects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks.

Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoals
Vivienne Huiling Wang
Tinghuai Wang" 2 Joni Pajarinen
Abstract
Li et al., 2021; Kim et al., 2021; Li et al., 2023; Wang et al., 2023a; 2024). Within this framework, a high-level policy decomposes the primary goal into a series of subgoals, ef- fectively directing the lower-level policy to achieve these subgoals. The efficacy of this approach hinges on the abil- ity to generate subgoals that are semantically coherent and achievable, providing a strong learning signal for the lower- level policies. The hierarchical structure not only enhances the learning process's efficiency but also considerably im- proves the policy's overall performance in solving complex tasks.
Hierarchical reinforcement learning (HRL) learns to make decisions on multiple levels of tempo- ral abstraction. A key challenge in HRL is that the low-level policy changes over time, making it difficult for the high-level policy generate effec- tive subgoals. To address this issue, the high-level policy must capture complex subgoal distribu- tion while also accounting for uncertainty in its estimates. We propose an approach that trains a conditional diffusion model regularized by a Gaussian Process (GP) prior to generate a com- plex variety of subgoals while leveraging prin- cipled GP uncertainty quantification. Building this framework, we develop strategy that se- lects subgoals from both the diffusion policy and GP's predictive mean. Our approach outperforms prior HRL methods in both sample efficiency and performance on challenging continuous control benchmarks.
The challenge with off-policy HRL lies in the simultancous training of both high-level and low-level policies, which can lead to a changing low-level policy that causes past experiences in reaching previously achievable subgoals to become invalid. This issue requires the high-level policy to swiftly adapt its strategy to generate subgoals that align with the constantly shifting low-level skills, Previous works like HIRO (Nachum al., 2018) and HAC (Levy al., 2019) have attempted to address this problem through relabeling strategy that utilizes hindsight experience replay (HER) (Andrychowicz et al., 2017a). This involves relabel- ing past experiences with high-level actions, i.e., subgoals, that maximize the probability of the past lower-level actions. Essentially, the subgoal that induced a low-level behavior in the past experience is relabeled so that it potentially induce the similar low-level behavior with the current low-level policy. However, the relabeling approach does not facil- itate efficient training of the high-level policy comply promptly with updates to the low-level policy. Some studies (Zhang et al., 2020; Kim et al., 2021) have proposed that the problem of inefficient training is aggravated by the size of the subgoal space, and attempted to constrain it. While such constraints can improve performance in specific set- tings, they often fail to scale effectively to more complex environments.
1. Introduction
In the domain of Hicrarchical Reinforcement Learning (HRL), the strategy to simplify complex decision-making processes involves structuring tasks into various levels of temporal and behavioral abstractions. This strategy excels particularly in environments where the challenges include long-term credit assignment and sparse rewards, making it a promising solution for long-horizon decision-making problems. Among the prevailing HRL paradigms, goal- conditioned HRL has been extensively explored in various studies (Dayan & Hinton, 1992; Schmidhuber & Wahn- siedler, 1993; Kulkarni et al., 2016; Vezhnevets et al., 2017; Nachum al., 2018; Levy et al., 2019; Zhang et al., 2020;
'Equal contribution 'epartment of Electrical Engineering and Automation, Aalto University, Finland "Huawei Helsinki Research Center, Finland. Correspondence to: Vivienne Huiling Wang evivienne.wang aallto.fi>
To address these challenges, this work introduces a condi- tional diffusion model-based approach for subgoal gener- ation (Sohl-Dickstein al., 2015; Song al., 2021; Ho et al., 2020). By directly modeling a highly expressive state- conditioned distribution of subgoals, our method offers a promising alternative to temporal difference learning, prov-
Proceedings of the ndndd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Hierarchical Reinforcement Learning with Uncertainty- Diffusional Subgoals
ing less susceptible to the destabilizing interactions inherent in the deadly triad (Sutton & Barto, 2018) and more adept at handling stochastic environments. While diffusion models excel at generating complex distributions, they often de- mand substantial training data and lack explicit uncertainty quantification. To mitigate these limitations and regularize learning, we incorporate a Gaussian Process (GP) prior as an explicit surrogate distribution for subgoals. The GP prior facilitates more efficient learning of the diffusion model based subgoal generation while simultaneously providing uncertainty quantification, informing the diffusion process about uncertain state regions.
set, P : PS x 8- + [0,1] is the state transition prob- ability function, R : S x A + R is the reward function, and 7 E [0, 1) signifies the discount factor. A stochastic policy π T(uls) maps any given state to a probability distri- bution over the action space, with the goal of maximizing the expected cumulative discounted reward Ex [2:t0 7're], where r' is the reward received at discrete time step L.
In a continuous control RL setting, modeled as a finite- horizon, goal-conditioned MDP M = A. R, 7), where 9 represents set of goals, we employ Hierarchical Reinforcement Learning (HRL) framework comprising two layers of policy akin to (Nachum et al., 2018). This frame- work includes a high-level policy TA (g (g]s) that generates high-level action, or subgoal, St ~ TT (- (x)) E 9, every k time steps when = O (mod k). Between these intervals, a predefined goal transition function It- = [ 8t-1, 8t) is applied when t * 0 (mod k). The high-level pol- iey influences the low-level policy through intrinsic re- wards for achieving these subgoals. Following prior work (Andrychowicz et al., 2017a; Nachum et al., 2018; Zhang et al., 2020; Kim al., 2021), we define the goal set 9 as a subset of the state space, i.e., 9 C S, and the goal transition function as f f9t-1, 1, 8x) = St-1 + 9t-1 St. The ob- jective of the high-level policy is to maximize the extrinsic reward as defined by r = Ei= Ra, t t= 0,1,
Building upon this foundation, we further develop sub- goal selection strategy that combines the GP's predictive mean with subgoals sampled from a diffusion model, lever- aging their complementary strengths. The GP's predictive mean, informed by state-subgoal pairs in the high-level re- play buffer, aligns subgoals with feasible trajectories and ensures consistency with the underlying structure of the state space. This is achieved through the GP's kernel, which captures correlations between states and subgoals based on patterns learned from the training data. To complement this precision, the diffusion model introduces adaptability by generating diverse subgoals from its learned distribution. Together, these approaches form a hybrid strategy: the GP offers structured, data-driven guidance towards reliable, low- uncertainty subgoals, while the diffusion model provides flexibility, enabling robust subgoal generation that balances reliability with versatility.
where R& is the environmental reward.
The low-level policy aims to maximize the intrinsic reward granted by the high-level policy. It accepts the high-level ac- tion or subgoal g as input, interacting with the environment by selecting an action a₂ ~ Te(119), 9t) AA each time step. An intrinsic reward function, rf || + gt 81+1 1|2. evaluates the performance in reaching the subgoal g.
Our main contributions are summarized as follows:
We introduce a conditional diffusion model for subgoal generation, directly modeling a conditiood dis- tribution and reducing susceptibility to instability from temporal difference learning.
This goal-conditioned HRI. framework facilitates early learning signals for the low-level policy even before profi- cient goal-reaching capability is developed, enabling concur- rent end-to-end training of both high and low-level policies. However, off-policy training within this HRL framework en- counters the non-stationarity problem as highlighted in Sec- tion 1. HIRO (Nachum et al., 2018) addresses this by rela- beling high-level transitions (8t, 9t: -i=t _
We employ Gaussian Process (GP) prior to regularize the diffusion model and explicitly quantify uncertainty, promoting more efficient learning and reliable subgoal generation.
R2, St4k) with alternate subgoal ğe to enhance the likelihood of the ob- served low-level action sequence under the current low-level policy, by maximizing ( 1atee111/::+41111- 0++++
We introduce a subgoal selection strategy that com- bines the GP's predictive mean, which aligns subgoals with feasible trajectories and ensures structural con- sistency, with the diffusion model's expressiveness, resulting in robust and adaptive subgoal generation.
The relabeled subgoals are generally considered to be sam- pled from a distribution that asymptotically approximates an optimal high-level policy within a stationary data distri- bution (Zhang et al., 2020; Wang et al., 2023a). This leads to the conjecture that learning a conditional distribution based on the relabeled subgoals inherently facilitates the achievement of stationarity in the high-level policy.
2. Preliminaries
Goal-conditioned HRI. In reinforcement learning (RL),
agent-ennviroment interactions are modeled as a Markov Decision Process (MDP) denoted by M = AS P, R, 7), where S represents the state space, A denotes the action
2

---

Hierarchical Reinforcement Learning with Uncertainty- Diffusional Subgoals
Diffusion Model Diffusion models (Sohl-Dickstein al., 2015; Song et al., 2021; Ho et al., 2020) have emerged powerful framework for generating complex data distribu- tions. These models pose the data-generating process as iterative denoising procedure po (x:-1 /xx)) This denoising is the reverse of a diffusion or forward process which maps a data example X ~ q(Xo) through a series of intermedi- ate variables X1.7 in T' steps with a pre-defined variance schedule 3,, according to
Starting with Gaussian noise, subgoals are then iteratively generated through series of reverse denoising steps by the noise prediction model parameterized by 0%
1 Vai
Bs a; Ear (g' ..1)) + VBe, g'
9
NOO, 1) if else -.
The learning objective of our subgoal generation model comprises three terms, i.e., diffusion objective Lam (On). GP based uncertainty Lgp (0n, App) and RL. objective Lapg (0n): ):
((x1:1(00)
II
g ((xx 1=1
T = argmin L0,) = Cam (0%) + of (0h, 8pp)
On.
q(x1|x(-1) == N v 1,
(2)
+ <nodpg (0n);
(4)
where v and 1) are hyperparameters.
The reverse process is constructed as (= =
((x; 0, P (x.). Parameters 0 are optimized by maximizing variational bound on the log likelihood of
We adopt the objective proposed by Ho et al. (2020) as the diffusion objective.
the reverse process LELBO = ( Dlo p
9(xi-7 Xo)
Cam (0n) = ((r
(5)
In this paper, employ two distinct notations of timesteps: one for the diffusion process and another for the reinforce- ment learning trajectory. Specifically, we denote the diffu- sion timesteps with superscripts E {1,...,N N). and the trajectory timesteps with subscripts E (1,...,T))
||E--0, -0- (Voig + vT vi- aie, 8, )1*]
(6)
where D/ is the high-level replay buffer, with the subgoals relabeled similarly to HIRO. Specifically, relabeling 9, in the high-level transition (st, 9:. Σith-1 R4 81+x) with g. aims to maximize the probability of the incurred low-level action sequence Ti( $tu+ Itx+&-1) 1), which is approximated by maximizing the log probability
3. Proposed Method
In this section, we present our method Hlerarhiical RL subgoal generation with DIffusion model (HIDI), which explicitly models the state-conditional distribution of sub- goals at the higher level. By combining with the temporal difference learning objective, the high-level policy promptly adapts to generating subgoals following a data distribution compatible with the current low-level policy.
log m (aa 11c4k-1,9n2-4
(+k-1
Σ
a - T (8 (8.. + const.
(7)
The purpose of the above diffusion objective is to align the high-level policy's behavior with the distribution of "opti- mal" relabeled subgoals, thereby mitigating non-stationarity in hierarchical models.
3.1. Diff'usiona Subgoals
We formulate the high-level policy as the reverse process of a conditional diffusion model as
While our method is versatile enough to be integrated with various actor-critic based HRL frameworks, we specifically utilize the TD3 algorithm (Fujimoto et al., 2018) at each hierarchical level, line with HIRO (Nachum et al., 2018), HRAC (Zhang et al., 2020) and HIGL (Kim et al., 2021). Ac- cordingly, the primary goal of the subgoal generator within this structured approach is optimize for the maximum expected return delineated by deterministie policy. This objective is formulated as:
N
II po. (g "g" s)), T. (g|s) = Po. po -N O)
with the end sample 9; being the generated sub- goal. Following Ho et al. (2020), a Gaussian distribu- tion N Po, (g', i), No, (g', s, i)) is used to model Pa, (gi "g", s), which is parameterized as a noise predic-
Cdpg(On) -EW [Qn(s
(8)
tion model with learnable mean
B
Given that g" (hereafter referred to as g without loss of gen- erality) is reparameterizee according to Eq. 3, the gradient of Ldpg with respect to the subgoal can be backpropagated through the entire diffusion process.
Ne, s, Vas g
Cop (,
VI Oi
and fixed covariance matrix Eo, (g'; 8,
(g' = B.
3

---

Hierarchical Reinforcement Learning with Uncertainty-Guide Diffusional Subgoals
3.2. Uncertainty Modeling with Gaussian Process Prior
goals that are consistent with the GP prior, particularly by fo- cusing learning on feasible regions supported by previously achieved transitions. This approach not only regularizes the diffusion model but also incorporates the uncertainty estimates provided by the GP, potentially leading to more robust and sample-efficient learning.
Whilst diffusion models possess sufficient expressivity to model the conditional distribution of subgoals, they face two significant challenges in the context of subgoal generation in HRL. Firstly, these models typically require a substantial amount of training data, i.e., relabeled subgoals, to achieve accurate and stable performance, which can be highly sam- ple inefficient. Secondly, standard diffusion models lack an inherent mechanism for quantifying uncertainty in their predictions, a crucial aspect for effective exploration and in HRL. To address these issues, we propose to model the state-conditional distribution of subgoals at the high level, harnessing& Gaussian Process (GP) prior explicit surrogate distribution for subgoals.
To formalize the effect of the GP regularization on the diffusion-bassee subgoal policy, we provide the following theorem and proposition. The GP loss introduces gradient signal that guides subgoal generation toward reliable regions identified by the GP posterior. Specifically:
robust
Theorem 3.1 (GP Regularization Guides Subgoal Align- ment). Lei g = l(e', s; 0s) be the subgoal generated by the diffusion model conditioned on state s and noise é'. Under mild regularity assumptions, the GP regularization term in the high-level objective encourages the learned distribu- tion p8,, (g]s) to align with the GP predictive mean (s(s) in regions of low predictive variance (s).
Given Dr, the relabeled high-level replay buffer, a zero- mean Gaussian process prior is placed on the underlying latent function, i.e., the high-level policy, g ~ To,, to be modeled. This results in a multivariate Gaussian distribution over any finite subset of latent variables:
We further quantify the mechanism through which this align- ment occurs:
p(g|s: , sp) = (glo,, KN + IOI,,
(9)
Proposition 3.2 (Gradient Weighting by GP Uncertainty). Let g f('', s; 0₂) be the subgoal generated by the dif- fusion model using the reparameterization trick. Then the gradient of the GP loss with respect to Oy satisfies:
where the covariance matrix K is constructed from a co-
variance function, or kernel, which expresses a prior no- tion of smoothness of the underlying function: [Kv] K K(is, 8,). Typically, the covariance function depends on small number of hyperparameters 0gn sp. which control these smoothness properties. Without loss of generality, we em- ploy the commonly used Radial Basis Function (RBF) ker- nel:
(5-16)10
(1.(8))
V on Lop Eyge'
V o
where (s((s and ds)s) denote the GP predictive mean and variance. This implies that the GP regularization applies stronger gradient pressure for parameter updates in state regions where the GP is more confident (i.e., d2((s is small).
D
2 )
K K(s,, = exp
8 Σ
(d)
20
d=1
0gp =7, Eo.
(10)
Here, D is the state space dimension, 72 is the variance parameter, f is the length scale parameter, and o2 is the noise variance. The hyperparameters 0g are learnable parameters of the GP model.
The detailed derivation and full proofs are provided in Ap- pendix A.1.3.
While the GP framework provides explicit uncertainty quan- tification, standard GP training requires ((N3) computa- tion to invert the covariance matrix, which is infeasible for large replay buffers (N). To improve efficiency, we adopt a sparse GP approach using M < N inducing states s (Snel- son & Ghahramani, 2005). These inducing states act as a summary of the full dataset Dj.
We leverage the GP prior to regularize and guide the opti- mization of the diffusion policy. Specifically, we incorporate the negative log marginal likelihood of the GP as an addi- tional loss term Cyp:
Lgp = -E Dr, To
|- logp(g's: s; On0 0gr)
Given a new state s., the sparse GP yields predictive distribution for the subgoal g.:
= Ban Dr, gmtt
28 (KN + 2'1
(11)
log KN + I'I log 2n
N
p(g. ls, Dr, 8) = Nig. (!,
(12)
where the predictive mean nad and variance of are given by:
By minimizing Eyp alongside the diffusion loss and RL objective, we encourage the diffusion policy to generate sub-
n1. = = K. (A
(13)

---

Hierarchical Reinforcement Learning with Uncertainty- Diffusional Subgoals
and
Theorem 3.3 (Single-Step Regret Bound for the Subgoal Se- lection). Let R (s) maxgeg R(s, g). Suppose there is baseline reward Rmin such that for all s E S, R(s. uu(s) Z Rmin- Under the assumption of a near-optimal difsussio policy (see Appendix A.2 for the full statement), the subgoal selection strategy defined above has a bounded single-step regret.
02 = K.. - (K - -QT) - o².
(14)
Here, k., Qu. KMN, A, KM. and K., depend on the kernel function evaluated at the inducing states s, the query state S., and the states in the replay buffer Dn. The inducing states and GP hyperparameters gp are learned by maxi- mizing the marginal likelihood. The detailed derivation of the sparse GP is provided in Appendix A.3. The learned in- ducing states implicitly define regions of interest, informing the subgoal generation process.
Furthermore, under standard policy improvement assump- tions (similar those in policy gradient methods (Sutton & Barto, 2018)), incorporating the GP mean provides theoretical guarantee for non-decreasing performance:
3.3. Inducing States Informed Subgoal Selection
Proposition 3,4 (Single-Step Policy Improvement). Let Qn g) be the high-level Q-value. Under certain condi- tions (see Appendix A.2 for the full statement), selecting the GP mean subgoal with a certain probability does not degrade performance single-step sense.
Building on the predictive distribution of the sparse GP model, we develop subgoal selection strategy that primar- ily relies on subgoals sampled from diffusion model, with the GP's predictive mean acting as a complementary regular- izer. At each high-level decision point, subgoals are selected using a hybrid mechanism that integrates the expressive ca- pabilities of the diffusion model with the structural guidance provided by the GP.
The detailed proofs for Theorem 3.3 and Proposition 3.4 are provided in Appendix A.2 and A.2, respectively.
Subgoals are predominantly sampled from the diffusion pol- icy Te, (g| s=), which generates diverse and adaptive sub- goals from its learned distribution. Complementing this, the GP's predictive mean, μ., (Eq. 13), derived from the sparse GP with inducing states and informed by state-subgoal pairs in the high-level replay buffer, leverages its kernel to cap- ture smoothness and correlations in the state space. This regularizes subgoal selection by encouraging consistency with the structure and dynamics observed in the training data, anchoring the selection process to meaningful patterns i..., low-uncertainty regions supported by experience, while mitigating over-reliance on the diffusion model's flexibility.
4. Related Work
HRL has been a significant field of study for addressing challenges such long-term credit assignment and sparse rewards. It typically involves a high-level policy that breaks down the overarching task into manageable subtasks, which are then addressed by a more specialized low-level policy (Dayan & Hinton, 1992; Schmidhuber & Wahnsiedler, 1993; Kulkarni et al., 2016; Vezhnevets et al., 2017; Nachum et al., 2018; Levy et al., 2019; Zhang et al., 2020; Li et al., 2021; Kim et al., 2021; Li et al., 2023; Wang et al., 2023a; 2024). The mechanism of this decomposition varies, with some approaches utilizing discrete values for option or skill se- lection (Bacon et al., 2017; Fox et al., 2017: Gregor et al., 2017; Konidaris & Barto, 2009; Eysenbach et al., 2019; Sharma al., 2020; Bagaria & Konidaris, 2019), and others adopting learned subgoal space (Vezhnevets et al., 2017; Li et al., 2021; 2023). Despite these variations, common challenge is the difficulty of leveraging advancements in the field of off-policy, model-free RI..
The selection strategy is formalized as:
Mr. g ~ Tea (g (s., with probability E.
with probability E,
g. =
(15)
To quantify the benefit of this subgoal selection, we consider the single macro-step reward R(s, g) for taking subgoal g E 9 state s, defined as:
Recent efforts to enhance HRL's learning efficiency through off- off-policy training have highlighted issues such as insta- bility and the inherent con-staiionaitty problem of HRL. For instance, (Nachum et al., 2018) introduces an off- policy method that relabels past experiences to mitigate non-stationary effeets on training. Techniques from hind- sight experience replay have been utilized to train multi- level policies concurently, penalizing high-level policies for unattainable subgoals (Andrychowicz et al., 2017a; Levy et al., 2019). To address the issue of large subgoal spaces, (Zhang et al., 2020) proposed constraining the high-level action space with an adjacency requirement. Wang et al.
R(s,g) = Tits
81 = 9t
where k is the number of lower-level steps that attempt to reach subgoal g.
Under a near-optimal diffusion policy assumption (inspired by similar assumptions in actor-critic analysis (Kakade & Langford, 2002)), we have the following guarantee for the
regret of our subgoal selection strategy:
5

---

Hierarchical Reinforcement Learning with Uncertainty- Diffusional Subgoals
(2020) improves stationarity by conditioning high-level de- cisions both the low-level policy representation and envi- ronmental states. Li et al. (2021) develops a slowness ob- jective for learning subgoal representation function. Kim et al. (2021) introduces a framework for training a high-level policy with a reduced action space guided by landmarks, Le.. promising states to explore. Adopting the deterministic subgoal representation of Li et al. (2021), Li et al. (2022) proposes an exploration strategy to enhance the high-level exploration via designing measures of novelty and potential for subgoals, albeit the strategy relies on on visit counts for subgoals in the constantly changing subgoal representation space.
Gaussian processes can encode flexible priors over func- tions, which are a probabilistic machine learning paradigm (Williams & Rasmussen, 2006). GPs have been adopted in various latent variable modeling tasks in RL. In Engel et al. (2003), the use of GPs for solving the RI. problem of value estimation is introduced. Then Kuss & Rasmussen (2003) uses GPs to model the the value function and system dynamics. Deisenroth et al. (2013) develops a GP-based transition model of a model-based learning system, which explicitly incorporates model uncertainty into long-term planning and controller learning to reduce the effects of model errors. Levine et al. (2011) 1) proposes an algorithm for inverse reinforcement learning that represents nonlinear reward functions with GPs, allowing the recovery of both a reward function and the hyperparameters of kernel func- tion that describes the structure of the reward. Wang et al. (2024) proposes a GP based method for learning probabilis- tic subgoal representations in HRL.
The broader topic of goal generation in RL has also been explored (Florensa et al., 2018; Nair et al., 2018; Ren et al., 2019; Campero et al., 2021; Wang et al., 2023a). GoalGAN (Florensa et al., 2018) employs a GAN to generate appropri- ately challenging tasks for policy training, however does not condition on observations and the sequential training of its GAN and policy. Nair et al. (2018) combines unsuper- vised representation learning with goal-conditioned policy training. Ren et al. (2019) proposes a method for generat- ing immediately achievable hindsight goals. Campero et al. (2021) introduces a framework wherein a teacher network proposes progressively challenging goals, rewarding the net- work based on the student's 's performance. SAGA (Wang et al., 2023a) introduces an adversarially guided framework for generating subgoals in goal- condicond HRL. How- ever, akin to the common challenges associated with GANs, SAGA may encounter issues such as stability and mode col- lapse due to its implicit modeling of subgoal distributions. In contrast, HIDI explicitly constructs subgoal distributions by progressively transforming noise into sample data, effec- tively capturing multimodal distributions and ensuring more stable training dynamics.
4.1. Environments
Our experimental evaluation encompasses a diverse set of long-horizon continuous control tasks facilitated by the Mu- JoCo simulator (Todorov et al., 2012), which are widely adopted in the HRL community. The environments selected for testing our framework, depicted in Figure 3, include Reacher, Pusher, Point Maze, Ant Maze (U-shape), Ant Maze (W-shape), Ant Fall, Ant FourRooms and variants with environmental stochasticity for Ant Maze (U-shape), Ant Fall, and Ant FourRooms by adding Gaussian noise with a standard deviation of a 0.05 to the (ær, y) positions of the ant robot at each step. Additionally, we adopt another variant labeled 'Image' for the large-scale Ant FourRooms environment. In this variant, observations are low-resolution images formed by zeroing out the (x, y) coordinates and appending a x 5 x 3 top-down view of the environment, as described in (Nachum al., 2019; Li et al., 2021). We evaluate HIDI and baselines under dense and sparse reward paradigms. Dense rewards are the negative L2 distance to the target, while sparse rewards are 0 below threshold and -1 otherwise. Maze tasks use 2D (z, v) goal space (Zhang et al., 2020; Kim et al., 2021), Reacher uses a 3D (z,, V, 2) goal space, and Pusher employs a 6D space including the object's 's position. Relative subgoals are used in Maze tasks, and absolute schemes in Reacher and Pusher."
Diffusion models have been introduced to offline RL do- main recently (Janner ct al., 2022; Wang et al., 2023b; Kang et al., 2024; Li et al., 2023; Chen et al., 2024). Janner et al. (2022) train an unconditional diffusion model to generate trajectories consisting of states and actions for offline RL. Approaches (Li et al., 2023; Chen et al., 2024) also extend diffusion model to offline HRL and generate trajectories at different levels. As a more related line of work, Diffsuion- QL (Wang et al., 2023b) introduces diffusion models into offline RL and demonstrated that diffusion models are su- perior at modeling complex action distributions. Kang et al. (2024) improve Diffsuion-QL to be compatible with max- imum likelihood-based RL algorithms. The success of of- fline RL methods leveraging diffusion policies ( (Wang et al., 2023b; Kang et al., 2024) motivates us to investigate the im- pact of using conditional diffusion model for the challenging subgoal generation in off policy HRL.
4.2. Analysis
We conduct experiments comparing with the following state- of- baseline methods: (1) HLPS (Wang et al., 2024): an HRL algorithm which proposes a Ga-based subgoal la- tent space; (2) SAGA (Wang et al., 2023a): an HRI, algo-
Detaills about the environments and parameter configurations are provided in the appendix.
6

---

Hierarchical Reinforcement Learning with Ancertainty-Guidde Diffusional Subgoals
Reacher (chnrse)
Pustver (ssarse)
Hix
H1LPS
5-405
11122
Stagpo
Tame Stagps
Stegpo
time Stegs
sparsel
baccen het
Mm
Way
Time Steps
Time Stepp
3xt
Tolrs tolere))
spp.. darmel
Stochustic Maa 11 danue)
Sucosss Rat8
Figure 1. Learning curves of our method and baselines, i.e., HLPS (Wang et al., 2024), SAGA (Wang et al., 2023a), HIGL (Kim et al., 2021), HRAC (Zhang al., 2020), and HIRO (Nachum al., 2018). Each curve and its shaded region represent the average success and 95% confidence interval respectively, averaged over 10 independent trials.
W l
Succes
4.4
HEEI HDH- HENA
HDN (M-3)
(4-3)
Time Stope
Time Staps
Time
Stass
145
(a)
(b)
(c)
(d)
Figure (a-b) Learning curves of various baselines: HIDI-A refers to HIDI without subgoal selection, HIDI-B refers to HIDI without subgoal selection and GP priors. (c) HIDI performance with varying diffusion steps. (d) HIDI performance with varying probabilities for subgoal selection.
rithm that introduces an adversarially guided framework for generating subgoals; (3) HIGL (Kim et al., 3021): an HRL. algorithm that trains a high-level policy with a reduced sub- goal space guided by landmarks; (4) HRAC (Zhang et al., 2020): an HRI. algorithm which introduces an adjacency network restrict the high-level action space k-step adjacent region of the current state; (5) HIRO (Nachum et al., 2018): an HRL algorithm that relabels the high-level actions based on hindsight experience (Andrychowicz et al., 2017b). Additionally, we present a theoretical analysis for HIDI in the Appendix A.2.
Can HIDI surpass state-of-the-art HRL methods in learning stability and asymptotic performance? Fig- ure illustrates the learning curves of HIDI in comparison with baseline methods across various tasks. HIDI consis- tently surpasses all baselines terms of learning stability, sample efficiency, and asymptotic performance. The advan- tage of the hierarchical diffusion policy approach is more pronounced in complex scenarios such as the Reacher and Pusher robotic arm tasks, and the Stochastic Ant Maze task, where the environmental stochasticity poses additional chal- lenges in generating reasonable subgoals. These results
7

---

Hierarchical Reinforcement Learning with Uncertainty- Diffusional Subgoals
highlight the benefits of employing generative models for subgoal generation, demonstrated by HIDI and SAGA. However, SAGA exhibits signs of instability in the more demanding robotic arm task Pusher, as well as considerably lower sample efficiency compared to HIDI in most tasks.
as surrogate distribution which potentially informs the diffusion process about uncertain areas. This high- lights the GP's role in focusing learning feasible regions consistent with past successful transitions.
Subgoal Selection: Our proposed subgoal selection strategy demonstrates clear benefits. Comparing HIDI to HIDI-A shows performance improvements of ap- proximately 7% and 8% on two challenging tasks, HIDI also achieves better sample efficiency compared to HIDI-A.
Is HIDI capable of generating reachable subgoals to address the issues commonly encountered in off-policy training within HRL? Figure 5 in the Appendix illus- trates generated subgoals and reached subgoals of HIDI and compared baselines in Ant Maze (W-shape, sparse) with the same starting location. This allows for an intuitive compari- son of subgoals generated by HIDI, SAGA, HIGL., HRAC, and HIRO. Notably, HIDI generates reasonable subgoals for the lower level to achieve, as demonstrated by the small divergence between the generated and reached subgoals, which provides stable learning signal for the low-level policy. In contrast, subgoals generated by HIRO are of- ten unachievable and fail to guide the agent to reach the final target; subgoals generated by HRAC frequently get stuck in local minima due to its local adjacency constraint; HIGL and SAGA show improvement in both constraining the subgoals locally while jumping out of the local opti- mum of subgoals, yet the high-level policy is not adequately compatible with the levelevel skills, i.e., the increasing gap between the generated subgoal and reached subgoal leads to inferior performance compared with HIDI. This is further confirmed by the measure of distance between the generated subgoal and reached subgoal in Table 1. Subgoals gener- ated by HLPS lie in a learned subgoal latent space and may not be quantitatively or qualitatively compared with other methods.
Diffusion Steps N: Fig. 2 (c) shows the learning curves of baselines using varying number of diffusion steps N used in HIDI. It empirically demonstrates that as N increases from 3 to 7, the high-level policy becomes more expressive and capable of learning the more com- plex data distribution of subgoals. Since N also serves trade-off between the expressiveness of subgoal modeling and computational complexity, we found that N = 5 is an efficient and effective setting for all the tasks during the experiment.
Subgoal Selection Probability controls the chance to perform subgoal selection. As shown in Fig. 2 (d), when e is large, e.g., 0.25, the trade-off between the expressiveness of subgoal modeling and uncertainty information might be affected, í.e., excessive subgoals sampled in uncertain regions may contribute to perfor- mance instability. When c is small, e.g., 0.05, the gain from subgoal selection would be decreased, and we set < 0.1 for all other results.
Scaling Factor ry: We investigate the impact of r in Eq. 4, which balances the diffusion objective and RL objective. As shown in Fig. 4 (Left), increasing " improves performance at early training steps (0 ~ 10%), while all three settings achieve similar performance at larger training steps (4.2 x 106 ~ 5 x 10%). We report all other results based on = 5 without loss of generality.
How do various design choices within HIDI impact its empirical performance and effectiveness? To under- stand the benefits of using diffusion policy for generating subgoals, we constructed several baselines. HIDI-A de- notes a baseline without performing the proposed subgoal selection strategy, HIDI-B is a baseline without adopting GP regularization and subgoal selection, while Baseline, adapted from a variant of the subgoal relabeling method i.e., HIGL, is HIDI without the diffusion model, GP regulariza- tion, and subgoal selection strategy. Fig. (a-b) illustrates the comparisons of various baselines and HIDI:
Scaling Factor v:: ve adjusts the influence of GP prior in learning the distribution of diffusional subgoals. As shown in Fig. 4 (Middle) in the appendix, increasing & gives stronger GP prior and may slightly affect the flex- ibility of diffusion model learning, while decreasing s renders the model approximate baseline HIDI-B. We set 1o = 10-3 for all other results.
Diffusional Subgoals: The performance improvement from Baseline to HIDI-B shows the benefit of adopting conditional diffusional model for subgoal generation, with a performance gain of ~ 15%%
5. Conclusion
Uncertainty Regularization: The sampling efficiency and performance improvement 155 and ~66 re- spectively from HIDI-B to HIDI-A indicates the advan- tage of employing the GP prior on subgoal generation
In this paper, we presented a conditional diffusion model- based framework for subgoal generation in hierarchical reinforcement learning. By directly modeling a state- conditioned subgoal distribution, the approach mitigates
8

---

Hierarchical Reinforcement Learning with Uncerlainty-Guided Diffusional Subgoals
instabilities arising from off-policy training and leverages Gaussian Process (GP) prior for explicit uncertainty quan- tification. We further proposed subgoal selection strategy that integrates the diffusion model's expressiveness with the GP's structural guidance. This ensures subgoals align with meaningful patterns in the training data while main- taining robustness, supported by theoretical guarantees on regret and policy improvement. Experimental results across challenging continuous control benchmarks highlight the efficacy of this integrated approach in terms of both sample efficiency and performance.
Chen, C., Deng, F,, Kawaguchi, K., Gülçchre, Ç., and Ahn, S. Simple hierarchical planning with diffusion. CoRR, abs/2401. 002444, 2024.
Dayan, P. and Hinton, G. E. Feudal reinforcement learning. In Advances in Neural Information Processing Systems, pp. 271-278, 1992.
Deisenroth, M. P., Fox, D., and Rasmussen, C. E. Gaussian processes for data-efficied learning in robotics and con- trol. IEEE transactions on pattern analysis and machine intelligence, 37(2):408-42) 2013.
Acknowledgements
Engel, Y., Mannor, S., and Meir, R. Bayes meets bellman:
The gaussian process approach to temporal difference learning. In International Conference on Machine Learn- ing. pp. 154 154-161, 2003.
We acknowledge CSC IT Center for Science, Finland, for awarding this project access to the LUMI supercom- puter, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through CSC. We acknowledge the computational resources provided by the Aalto Science-IT project. We acknowledge funding from Research Council of Finland (353198).
Eysenbach, B., Gupta, A., lharz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations (Poster), 2019.
Impact Statement
Florensa, C., Held, D., Geng, X., and Abbeel, P. Auto- matic goal generation for reinforcement learning agents. In International Conference Muchine Learning, vol- ume 80 of Proceedings of Machine Learning Research, pp. 1514-1523, PMLR, 2018
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fox, R., Krishnan, S., Stoica, I., and Goldberg, K. Multi- level discovery of deep options. CoRR, abs/1703 703.08294, 2017.
References
Andrychowicz, M., Crow, D., Ray, A., Schneider, J., Fong,
R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017a.
Fujimoto, S., van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pp. 1587- 1596, 2018
Gregor, K., Rezende, D. J., and Wierstra, D. Variational intrinsic control. In International Conference on Learning Representations (Workshop), 2017.
Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J.,, Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in neural information processing systems, 30, 2017b.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems, pp. 6840-6851, 2020.
Bacon, P.-L., Harb, J., and Precup, D. The option-critic architecture. In The AAAI Conference on Artificial Intel- ligence, volume 31, 2017.
Janner, M., Du, Y., Tenenbaum, J. B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9902-9915. PMLR, 2022.
Bagaria, A. and Konidaris, G. Option discovery using deep skill chaining. In International Conference on Learning Representations, 2019.
Campero, A., Raileanu, R., Küttler, H., Tenenbaum, J. B., Rocktäschel, T., and Grefenstette, E. Learning with amigo: Adversarially motivated intrinsic goals. In Inter- national Conference on Learning Representations, 2021.
Kakade, S. and Langford, Approximately optimal ap- proximate reinforcement learning. Proceedings of the Nineteenth International Conference on Machine Learn- ing, pp. 267-274, 2002.

---

Hierarchical Reinforcement Learning with Uncertainty-Guide Diffusional Subgoals
Kang, B., Ma, X., Du, C., Pang, T., and Yan, S. Efficient diffusion policies for offiine reinforcement learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Ren, Z., Dong, K., Zhou, Y., Liu, Q., and Peng, J. Explo- ration via hindsight goal generation. In Advances in Neu- ral Information Processing Systems, pp. 13464-13474 2019.
Kim, J., Seo, Y., and Shin, J. Landmark-guided subgoal generation in hierarchical reinforcement learning. Ad- vances in Neural Information Processing Systems, 34: 28336-28349, 2021.
Schmidhuber, J. and Wahnsiedler, R. Planning simple tra- jectories using neural subgoal generators. In From Ani- mals Animats 2: Proceedings of the Second Interna- tional Conference on Simulation of Adaptive Behavior, volume 2, pp. 196. MIT Press, 1993.
Konidaris, G. and Barto, A. Efficient skill learning using abstraction selection. In International Joint Conference on Artificial Intelligence, 2009.
Sharma, A., Gu, S., Levine, S., Kumar, V., and Hausman, K. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenen- baum, J. Hierarchical deep reinforcement learning: In- tegrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29: 3675-3683, 2016.
Snelson, E. and Ghahramani, Z. Sparse gaussian processes using pseudo-inputs. Advances in neural information processing systems, 18, 2005.
Kuss, M. and Rasmussen, C. Gaussian processes in re- inforcement learning. Advances in neural information processing systems, 16, 2003.
Sohl-Dickstein, J., Weiss, A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilib- rium thermodynamics. Praiiss..Prrrr Proceedings of the 32nd Interna- tional Conference on Machine Learning, pp. 2256-22666 2015.
Levine, S., Popovic, Z., and Koltun, V. Nonlinear inverse re- inforcement learning with gaussian processes. Advances in neural information processing systems, 24, 2011.
Song. Y., Sohl-Dickstein, J., Kingma, P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representattinss 2021.
Levy, A., Konidaris, G. D., Jr., P., and Saenko, Learn- ing multi-level hierarchies with hindsight. In Interna- tional Conference on Learning Representations, 2019.
Li, S., Zheng, L., Wang, J., and Zhang, Learning subgoal representations with slow dynamics. In International Conference Learning Representations, 2021.
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, 2018
Titsias, M. K. Variational learning of inducing variables in sparse gaussian processes. In Artificial intelligence and statistics, pp. 567-574. PMLR, 2009.
Li, S., Zhang, J., Wang, J., Yu, Y., and Zhang, C. Active hierarchical exploration with stable subgoal representa- tion learning. In International Conference Learning Representations, 2022.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ inter- national conference on intelligent robots and systems, pp. 5026-5033 IEEE, 2012.
Li, W., Wang, X., Jin, B., and Zha, H. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pp. 20035-20006 PMLR, 2023.
Vezhnevets, A. S., Osindero, S., Schaul, T., Heess, N., Jader- berg, M., Silver, D., and Kavukcuoglu, K. Feudal net- works for hierarchical reinforcement learning. In Interna- tional Conference on Machine Learning, pp. 3540-3549. PMLR, 2017.
Nachum, O., Gu, S., Lee, H., and Levine, S. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3307-3317, 2018.
Nachum, O., Gu, S., Lee, H., and Levine, S. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learning Rep- resentations, 2019.
Wang, R., Yu, R., An, B., and Rabinovich, Z. 12hrl: Interac- tive influence-batt hierarchical reinforcement learning. International Joint Conference on Artificial Intelli- gence, pp. 3131-3133 2020.
Nair, A., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. Visual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9209 9220, 2018.
Wang, V. H., Pajarinen, J., Wang, T., and Kämäräinen, J.- K. crate-conditioned adversarial subgoal generation. In Pruceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 10184-10191, 2023a.
10

---

Hierarchical Reinforcement Learning with Eiccrtaintt-Guuiee Diffusional Subgoals
Wang, V. H., Wang, T., Yang. W., Kämäräinen, J., and Pajarinen, J. Probabilistic subgoal representations for hierarchical reinforcement learning. In Forty-first Inter- national Conference Machine Learning, ICML 2024, Vienna, Austria, July 21 21- 2024, 2024.
Wang. Z.. Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023b. URL https: //openreview nett 1tt.:/oormmmm0-AA/PPP-------
Williams, C. K. and Rasmussen, E. Gaussian processes for machine learning, volume MIT press Cambridge, MA, 2006.
Zhang, I., Guo, S., Tan, T., Hu, X., and Chen, Gen- erating adfacency-coontraingg subgoals in hierarchical reinforcement learning. Advances in Neural Informa- tion Processing Systems, 2020.

---

Hierarchical Reinforcement Learning with Uncertainty-Guide Diffusional Subgoals
A. Appendix
A.1. Theoretical Analysis of Diffusion-Bei -asse Subgoal Generation and GP Regularization
In this section, we provide a theoretical analysis for: (i) the effectiveness of the diffusion model for subgoal generation in hierarchical reinforcement learning (HRL), and (ii) the role of Gaussian Process (GP) regularization in guiding the diffusion process to generate reachable and structured subgoals.
A.1.1. NOTATION AND PRELIMINARIES
Let:
s denote state and g E G C S denote a subgoal.
The high-level policy is parameterized by 02 and defined as reverse diffusion process:
To (g 8) Poh (go:N s) = (gx: 0, II pa.. (9t-1 g1. 8),
(16)
where 9o is the final generated subgoal and N is the number of diffusion steps.
The reverse diffusion step is defined
9i-1
9i Br VI- co, (9.,, + Bie, es ~(,11)
1
(17)
Va
with 0; 1 5; and ā;
IT; Oj.
The diffusion model is trained using the loss
Cam (0n)
[|e-cee co, (Vārg v1 āge, 8, })]
(18)
where D is the high-level replay buffer of relabeled subgoals.
A Gaussian Process (GP) prior is placed on the subgoal distribution conditioned on 8:
p(g 8; 0p)) Ng: 0, KN +011)
(19)
where KN is the covariance matrix defined via kernel function K(s, s'), and o² is the noise variance.
The GP regularization loss is defined as:
Cyp (On, 0pp) = ((.gg)>> |- pg 8; 0p)0)
(20)
The overall high-level training objective combines the diffusion loss, GP regularization (weighted by v)), and an RL objective Lesg (0%):
I(Or) = Lam (fxx) + sb (On, 0gp) + 17 Lapg (0%)
(21)
1.1. VALIDITY OF THE LEARNED SUBGOAL DISTRIBUTION VIA DIFFUSION
We first establish conditions under which the reverse diffusion process yields subgoal distribution close to the target distribution g(g s) (defined by the relabeled data in D
Theorem A.1 (Validity of the Learned Subgoal Distribution). Let q(gls) be the target distribution subgoals given state (implicitly defined by the relabeled data in Da.. Let Pe, (g]s) be the distribution generated by the reverse diffusion process parameterized by O,. Assume the noise prediction network co,, is trained by minimizing the simplified objective:
emNN(.,)) en-s (D.00)>>A [||e co.. (Vos go + v1 ace, 8, 1
Lam (0x) EE,2((1,N),
(22)
12

---

Hierarchical Reinforcement Learning with Uncertainty-Guidel Diffusional Subgoals
If the expected loss achieved by the optimal parameters 0f is bounded, E Cam (0%) S 82, then the KL divergence between
the target distribution and the learned distribution is bounded, D D s) por, (g|s)) S B(õ, N, (Bi)), where B is function that depends on the noise prediction error 6, the number of steps N, and the noise schedule {Br), satisfying lims- B()....) O. Consequently, by Pinsker's 's inequality, the total variation distance is also bounded:
|4(g| poz (ol S v PKI. (a(laa()))) (gls)) 2 V B(ā, N. (8.)
D
(23)
Proof. The proof relies on connecting the simplified objective Lam to the variational lower bound (ELBO) on the log- likelihood logpe, Po, (gols). The ELBO for the reverse process pe,, (gols) conditioned on s, marginalizing over 91:N, is:
Pea, (go.N(s)
log Po, (gols) 2 Relsnwlloo.)) log
(24)
((1: 190; 8)
=By logplon)
log po,, (0i-1 199,))9)))) 8)
Σ log g0 1g: 1,)
(25)
N
= l pe, (91,, - En
Σ
D go, 9o, 8) lpo, (gi- 1g 8))
Reconstruction Term
KL. terms
DKLlalgN go. s)(((()))
(26)
Prior Matching Term
where g(g5-1196, 9o: s) is the true posterior of the forward process, which is tractable and Gaussian: g(gi-119ts Go. s) (1. 90), 3,1) with (9 90) = / 1-o; + (0 and 3
19:
1-5,-1
I-a, 3z.
1-a;
The diffusion model Pan (gi- [9;,s) s) = (gi- 1: (g., s, 1,1)) uses a learned mean Ma, (gs: 8, i) Vi-a.E
[) VI-a c0, (gi, 8) and a fixed variance 3 (or sometimes 3,
Ho al. (2020) shows that minimizing the simplified objective Lam (0x) (Eq. 22) is equivalent to optimizing a weighted sum of the KI. terms in the ELBO (26), specifically minimizing the discrepancy between the predicted noise c0, and the actual noise used to generate g. Vā.go + /I ã.e. Let LV (0%) - (6g1)))))) [LBO for go]. Then:
DxL(a(pols)[la) (guls)) = (VL H (g(ools))
(27)
where H(g(go s)) the entropy of the true data distribution, which is constant w.r.t On. Minimizing Cam upper bounds the terms contributing to LVL... Specifically, the expected squared error in noise prediction relates the KL divergence terms:
Ego,e aall-- co,
co, (Vārgo + -I-aacc 8, 1) CgE [DKL
s) s)llpo, (9i-1 (9 s))] + const,
(28)
where C; (1-() depends on the variance o? = 3, or 3r used in pa.. (See Appendix B of (Ho et al., 2020) for details),
Summing over (with appropriate weighting, often simplified in practice as in Lam), if E[Lam (6;)1 I 82, it implies that the sum of the KL divergence terms in the ELBO expression (26) is bounded by some function (8,8, N, {,,.)..
LVL.(00) < Ex( log poz, (gol (i s)] + B' N, (B.)) + DKr(g(ow Igo. s)(lP()))
(29)
If we also assume the reconstruction term is well-behaved (or absorbed into the bound) and the prior matching term is small (as T - oo), minimizing Lam effectively minimizes an upper bound on LVLB. Thus. Dx (9(0)(9 ()) Po; (gols))
CVLB(07) H (g(guls)) is bounded by function B(5, N, {3.)) which incorporates B' and other terms, and vanishes as 0-
Finally, applying Pinsker's inequality, [| - po; (1l S { (gls)), yields the desired bound on the total variation distance.
13

---

Hierarchical Reinforcement Learning with Uncertainty-Guidel Diffusional Subgoals
A. 1.3. GP REGULARIZATION AND ITS INTERACTION WITH THE DIFFUSION MODEL
We now analyze how the GP regularization term influences the generated subgoal and guides the diffusion process.
Theorem A.2 (Guiding Effect GP Regularization), Let the overall loss function for the high-level policy parameters 8,.
be
L(Ow)) = Lam (On) + t Lap (On. Opp) + (O.).
The GP regularization term is defined using the reparameterization trick, where
g s(E'ss00
the subgoal generated by the reverse diffusion process from base noise é' ~ (O, and state condition s:
(0p (pp) = B'
[ logg f((' s; s; Dr. 902
(30)
where
PGP(g s;D,, N (=(ss)
the predictive distribution of the (sparse) GP conditioned the high-level data Dr, (see Eq. 39), assuming isotropic variance for simplicity. The gradient of this term with respect the diffusion model parameters O₁ is given by:
( (B-1u(()
(a(s))
Vo, F.
Vo,g
(31)
where g = f(', 0r). This gradient term encourages the parameters 0, to be updated during training such that the generated subgoals g tend move closer the GP predictive mean (s), with the influence being stronger in regions where the GP has low predictive variance (s)
Proof. We analyze the gradient of the GP loss term Lop with respect to the parameters 0₁ of the diffusion model, which defines the reverse process Pa, (g s).
The subgoal g generated by the diffusion model is given by
g f(e', s; 0n),
where e' I) and f(-) differentiable with respect to 0r. Using the reparameterizatiio trick, we write:
[ log f(e s; (%) s; Dn. 0gp Von (- (llogrrr (f(6', s; 0x))
Vo, Cgp Vo, Ean c'-N((.))
= -R ((,
s; Dr, 0 0g))
(V (- log Dn, O m)) Ve,g
g=f(e'.s.8))
For the Gaussian predictive distribution
PgP(g] s) = [Ng (s))
the negative liklliieehood is
1 |g n. (s)
log Pogp
log (og(2
20?
where D is the dimensionality of Differentiating with respect to g yields
(s(s)
V | log s)]
03(s)
14

---

Hierarchical Reinforcement Learning with Uincrtainty-Guuiee Diffusional Subgoals
Substituting this into our expression, obtain
(
($- o
Ve, Syp = PawDD..e
Veng
During optimization via gradient descent, the update for 0n is given by
0 + 0% a ((h)):
with a being the learning rate. The contribution from the GP loss term is -ord Vo, 0, Caps which effectively drives the parameters O, to adjust so that the generated subgoal g aligns more closely with the GP predictive mean fs, (s), particularly when (s) is small (indicating high confidence).
This completes the proof.
Remark A.3 (Connection KL Divergence). The GP regularization term Zyp can be related the KL divergence between the learned conditional distribution po., (g s) and the GP predictive distribution
PGr(g s) = -N(g (s),
Specifically,
Cgp (01, gp) = EENDD [Egppn (11s) 1- log Par (g ss]]
= EggDD [DKL (pon | par( $)) + H pos ( [s)]]
|)
where H-) denotes differential entropy. Thus, minimizing Cop reduces the KL divergence between the learned subgoal distribution and the GP predictive distribution, thereby encouraging the diffusion model to produce subgoals that are aligned with the GP's predictions.
A.1.4. DISCUSSION
Diffusson Model Validity. Theorem A.1 demonstrates that if the noise prediction error sufficiently small, the reverse diffusion process accurately approximates the target state-conditioned subgoal distribution. Hence, under reasonable conditions, the diffusion model is theoretically justified for subgoal generation in HRL.
GP Regularization Impact. Theorem A.2 shows that the GP regularization term actively pulls the generated subgoal towards the GP predictive mean. In regions where the GP is confident (i.e., Amin (KN + +o²/) is large), the generated subgoal tightly coupled with the observed, reachable subgoals. This guides the diffusion model, ensuring that even the learned distribution inherently non-stationary, the GP regularization mitigates extreme deviations by serving as a principled, uncertainty-awarc anchor.
A.2. Theoretical Analysis of Subgoal Selection
Assumption A.4 (Near-Optimal Diffusion Policy (for Single-Step)). We assume that after sufficient training and data coverage, for every s E S,
Eg-sT (1)5) [(ss g)l
max R(s.g') EG
for some small 5 > 0. This states that the diffusion-based subgoal distribution ne, nearly achieves the maximum possible single-step reward at each s.
Detailed Proof of Theorem 3.3. Let R2 (s) maxgeg R(s, g) be the optimal single-step reward. The regret of the subgoal selection strategy 7i at state is given by R" (s) Eg [R(s,g) g)].
15

---

Hierarchical Reinforcement Learning with Uncertainty-Guide Diffusional Subgoals
By the definition of the subgoal selection strategy (g's))) ((t) (1 me, (gls), the expected reward under this
policy is:
E Bg [R(s,g)
(((s.))
R(s.g) (s (())(s)(g) + (1 - 6) no, (g|s)] dg
Ri(s, g) ((s) + (1 5) /
R(s (g)))
E R(s, u + (1 - E) Eg g [R(s,))],
where we used the property of the Dirac delta function in the last step.
Given that Rss, (u())) 2 Rmin by assumption, and under Assumption A.4, have Egunro, [R(s,g)] R*)s) 6, lower bound the expected reward:
E Bg [R(s, g)] 2 E Rmin + s) 6).
Now, we can bound the single-step regret:
R* (s) Eg [R(s, g)l S R [E Rmann (1 5) (R* (s) -8]
R² (s) E Rmin
8) R)((ss +1 111-87 =)
eR" (s) ERmiu (1 =)0
=(R" (s) Rmin) + +1 - E)d.
Thus, the single-step regret of the subgoal selection strategy bounded by & RR (s) - Rmin + (1 E)
Detailed Proof of Proposition 3.4. This proof relies on the principle of policy improvement, fundamental con- cept in reinforcement learning (Sutton & Barto, 2018). The value function of policy is given by J(x)
Maoodo.g.-aa-bee [Ei=o >'r(]. where r the high-level reward. A single step of policy improvement involves changing the policy in a way that increases the value function.
Consider the expected Q-value under the subgoal selection strategy 7n at state s:
| (n(s,)) 8) [s (g) + (1 6) -T, (g|s)] dg
g
= = Qn (())s + +1 - Eg (-1s) Bg
(s,
Assume the high-level Q-function (s,s, g) is Lipschitz smooth in g with constant L:
(s. g1) (n g2)) < L||g: - gell. Yg1.80 E g.
(32)
If the GP's predictive mean uss) is close to subgoals g with high Q., then uus) will inherit high Q-values through smoothness. The GP is trained on state-subgoal pairs (s, g) from the replay buffer Br, containing high-reward transitions (by definition of R(s, g)). The GP kernel k(.,)) captures correlations in S x G. enabling generalization of high-Q subgoals to new states.
The predictive mean (s)s) minimizes the posterior expected squared error:
(u(s arg min Eg-p(gis. [llg' gl
(33)
g'
where p(gis, Bx) is the posterior subgoal distribution. If high Q subgoals in B, cluster around μus),, then:
Qn (())
E Ee-piela.B.)
(34)
16

---

Hierarchical Reinforcement Learning with Uncertainty- Diffusional Subgoals
This holds under unimodality concentration of high-Q subgoals in B
The diffusion policy To, generates subgoals through stochastic generative process. TD3 employs deterministic high-level actions with exploration noise. However, the diffusion model's sampling process inherently encourages diversity by gradually denoising from a Gaussian distribution. This results in subgoals distributed around high-reward regions in G, with a trade-off between exploitation (high-Qn. subgoals) and exploration (diverse subgoals).
Let g" = arg maxg Qn. g) denote the optimal subgoal. The diffusion policy Tas samples subgoals such that:
Eg (s, g)] s Qx(s, A,
where A > 0 quantifies the exploration penalty due to the diffusion process's s stochasticity. Meanwhile, the GP mean (ss) acts deterministic proxy for subgoals frequently visited in high-reward trajectories within B₁. If (u(s) approximates g' with error (i.e., (s)) g"ll < <), Lipschitz continuity of Qn implies:
(n(s) 2 Qn(s.g)) Le.
Thus, for Le A, we have:
Qn(s. 2 (n(gs"")) ) LE 2 Eguns [n((s.)) g)].
Therefore:
(11)) 2 E gg ( (4) (s, g)] +1 (1- -11-) E) gT (1))) (n..g)
=E BR (11) (1,
The value function can be expressed as J(r) = Egnds states under policy π.
B)]]. where da is the stationary distribution of
the induced state distribution dz, remains close to daa, which common assumption for a single-step policy improvement analysis as the change in policy is controlled by e, then:
( = Egnds, Egai (s,
Bandin
E ER (Qx(s,
Egndxon
E EgNao, (s.gg)
Thus, J(Tr) 2 (FTo) ) indicating that the subgoal selection strategy achieves value function least as good as the original policy Te, in single step.
A.3. Sparse Gaussian Process Derivation
As mentioned in the main text, standard Gaussian Process (GP) regression involves inverting an N N covariance matrix, where N is the number of data points in the high-level replay buffer D,. This has computational complexity ((V)). which is prohibitive for large datasets typical in RL. To address this, we employ a sparse GP approximation using inducing points (Snelson & Ghahramani, 2005).
We introduce a smaller set of M inducing states (Sm J where M < N. These inducing states are associated with corresponding "imaginary" target subgoals {g.. M m= The core idea is that the predictive distribution for any new state s, depends only on these M inducing points, which summarize the full dataset.
Let D = (5. g) denote the pseudo-dataset of inducing points. The joint distribution of the observed subgoals (from Dz) and the imaginary subgoals g under the GP prior is:
KNN + o01
KNM
(ls.)ss)) = N
0. KMN
KMM]
17

---

Hierarchical Reinforcement Learning with Uncertainty-Guider Diffusional Subgoals
where NNN, KNM, KMN N and KMM K M covariance matrices computed using the kernel function K between the states in D and the inducing states s. Specifically, [K [Ss.: 8m) and aK K ((Sm Sun' )
The sparse approximation assumes that the observed subgoals are conditionally independent given the imaginary subgoals The likelihood of single subgoal g given state and the inducing variables (s. g) is formulated based on the conditional distribution p(g|s, ss, S, g) = p(g|s, {f g g)df, where f are latent function values, A common approximation (used by Titsias (2009), related to FITC by Snelson & Ghahramani (2005)) leads to:
p(gss D,g) N (g K
MB. Kas kg K kg + o²),
(35)
where [kamm (s, 8m) and Ksn - ((s,8).
The overall likelihood for the set of observed subgoals g given corresponding states s and the inducing variables is approximated as:
p(gss,8,g) ~ II p(gn 8.g)
(36)
Ag K KN B
where A is diagonal matrix with [A] = ((,,,, S,)) k, K M1 kn, and k, is the vector of kernel evaluations between S..
and all inducing states s. Note that [a K K(,n, S.).
A standard Gaussian prior is placed on the imaginary subgoals g:
pg]s) 0, KM).
(37)
Using Bayes' rule on Eq. 36 and Eq. 37, the posterior distribution over the imaginary subgoals g given the observed data D = (s.g) and inducing states can be derived as:
Dr. 5) x p.g
= =Neg KM OT + d'1)-'g,
(38)
K Q Kv).
where Qu = KM + KMN + (2I)-'KNM Note the inversion now involves M x M matrices, making computation feasible (O(N or (MM)) depending on the exact method).
Finally, given a new state s., the predictive distribution of the corresponding subgoal g, is obtained by integrating the approximate likelihood (Eq. 35 applied to s.) against the posterior (Eq. 38):
p(g. Is, S), = - dgp(g. (s. 8, Dr.
(39)
Nig, (r,
where the predictive mean and variance o² are:
K o'N μ. = KMN
(40)
and
0. K.. - (K k. Q k, +
(41)
Here, k, is the vector k.]m = K(ss, 5m) and K.. = K(s,, S.).
The inducing states and the GP hyperparameters 0gp f%, f are typically optimized by maximizing the marginal likelihood (evidence) p(g]s,5 8), which can also be computed efficiently using the inducing points. This optimization encourages the inducing states to adaptively summarize the data distribution the state space.
18

---

Hierarchical Reinforcement Learning with Uncertainty- Diffusional Subgoals
Algorithm 1 HIDI
1: Input: Higher-level actor - lower-level actor Tb. critics Q", Q', goal transition h(-). higher-level frequency training episodes N.
2: for n = to do
3: Sample initial state so
4: ++0
5: repeat
6: if = 0 (mod kt) then
7: Sample subgoal via Eq. 15
8:
else
9:
Compute = 1 8t 1, 8c)
10:
end if
11:
Sample action aj ~ mo, (als8s St)
Execute at, observe 81+1 ~ (( 8c; a+)
Compute intrinsic reward r, R(r It> az)
14:
Store transition 11--1,99111 (tyy Tta 9t)
15:
Sample done signal
16:
t ++t+1
17:
until done is true
18:
if Training higher-level policy Tे) then
19:
Sample relabeled experience (st, Gt. 11 Update Tे. and GP hyperparameters via Eq. 4 Update critic Qh
20:
21:
22:
end if
23:
if Training low-level policy =/ then
24:
Sample experience (st, ae; It, Tt, 8t41) Update Té and critic Q'
25:
end if
end for
A.4. Implementation
We implement the two-layer hierarchical policy network following the architecture of the HRAC (Zhang et al., 2020), which uses TD3 (Fujimoto al., 2018) as the foundational RL algorithm for both the high and low levels.
Following the parameterization of Ho et al. (2020), the diffusion policies implemented an MLP-based conditional diffusion model, which is residual model, i.e., co(g", 8, and (a', 8, 9,9, i) respectively, where is the previous diffusion 3-layer MLP with 256 hidden units.
time step, is the state condition, and g the subgoal condition. is implemented The inputs to comprise the concatenated elements of either the low-level high-level action from the previous diffusion step, the current state, the sinusoidal positional embedding of time step i, and the current subgoal if it is high-level policy. We provide further implementation details used for our experiments in Table 3.
A.5. Algorithm
We provide Algorithm I to show the training procedure of HIDI.
A.6. Environments
Our experimental evaluation encompasses diverse set of long-horizon continuous control tasks facilitated by the MuloCo simulator (Todorov al., 2012), which are widely adopted in the HRL community. The environments selected for testing our framework include:
Reacher: This task entails utilizing robotic arm to reach specified target position with end-effector.
19

---

Hierarchical Reinforcement Learning with Uncertainty- Guided Diffusional Subgoals
(a) Reacher
(b) Pusher
(c) Point Maze
(d) Ant Fall
(c) Ant Maze (U)
(f) Ant Maze (W)
(g) Ant FourRooms
Figure Environments used our experiments.
Pusher: A robotic arm must push a puck-shaped object on a plane to designated goal position.
Point Maze: A simulation ball starts in the bottom left corner of a """-sapedd maze, aiming to reach the top left cornr.
Ant Maze (U-shape): A simulated ant starts in the bottom left of " """-shaped maze, targeting the top left corner.
Ant Maze (W-saape): A simulated ant starts at random position within a "3"-shaped maze, aiming for the middle left corner.
Ant Fall: This task introduces diree-dimensional navigation. The agent begins on platform elevated by four units, with the target positioned across a chasm that it cannot traverse unaided. To reach the target, the agent must push a block into the chasm and then ascend onto it before proceeding to the target location.
Ant FourRooms: In this task, the agent must navigate from one room to another to achieve an external goal. The environment features an expanded maze structure measuring 18 x 18.
Following (Zhang al., 2020; Kim et al., 2021), adopt variant with environmental stochasticity for Ant Maze (U-shape), Ant Fall, and Ant FourRooms by adding Gaussian noise with standard deviation of o 0.05 the (x, u) positions of the ant robot each step. Additionally, we adopt another variant labeled 'Image' for the large-scale Ant FourRooms environment. In this variant, observations are low-resolution images formed by zeroing out the (z, v) coordinates and appending 5 x 5 x 3 top-down view of the environment, as described in (Nachum et al., 2019; Li et al., 2021).
Variants:
We evaluate HIDI and all the basclines under two reward shaping paradigms: dense and sparse. In the dense setting, rewards are computed as the negative L2 distance from the current state to the target position within the goal space, whilst the sparse rewards are set to 0 for distances to the target below a certain threshold, otherwise -1. Maze tasks adopt a 2-dimensional goal space for the agent's (a,, y) position, adhering to existing works (Zhang et al., 2020; Kim et al., 2021). For Reacher, a 3-dimensional goal space utilized to represent the end-effector's (æ,, y, 2) position, while Pusher employs a 6-dimensional space, including the 3D position the object. Relative subgoal scheme applied in Maze tasks, with absolute scheme used for Reacher and Pusher.
20

---

Hierarchical Reinforcement Learning with Uncertainty-Guide Diffusional Subgoals
HIDI
SAGA
HIGL
HRAC
HIRO
0.82
0.95
1.57=#0.05
1.72==0.06
10,90 2.04
Table 1. The distance between generated subgoals and the reached subgoals, Le., the final state of k-step low-level roll-out averaged over 10 randomly seeded trials with standard error.
(W-shape,
(W-shape, sparse)
0.8
0.8
0.6
02
HIDI-
0.2
HIDIA (p- 10 *)
HIDI- In 5)
HIDIA (p- 10-1)
HIDI-B
HIDIA 10-5))
Trarren states
Steps
Steps
366
Figure 4. (Left) Impact of 1, which balances the diffusion objective and RL objective. (Middle) Impact of tb which adjusts the influence of GP prior in learning the distribution diffusional subgoals. (Right) Visualization of the learned inducing states (2D coordinates) compared with the complete training data.
A.7. Additional Results
Figure shows the generated subgoals and reached subgoals of HIDI and compared baselines in Ant Maze (W-shape, sparse) with the same starting location. HIDI generates reasonable subgoals for the lower level to accomplish, as evidenced by the minimal divergence between the generated and achieved subgoals which in turn provides stable learning signal for the low-level policy. In contrast, the subgoals generated by HIRO are unachievable and cannot guide the agent towards the final target. Subgoals from HRAC frequently get stuck in local minima due to its local adjacency constraint. While HIGL and SAGA demonstrate improvement in constraining subgoals locally while escaping local optima, the high-level policy is inadequately compatible with the low-level skills. Specifically, the increasing gap between the generated subgoal and reached subgoal leads to inferior performance compared to HIDI.
We qualitatively study the learned inducing states in Fig. 4 (Right), where the states of the complete set of training batch (100) and the final learned inducing states (16) are visualized. Note, only the z, y coordinates are selected from the state space both cases for visualization purpose. We can observe that with significantly smaller number of training data, the inducing points capture the gist of the complete training data by adapting cover more critical regions of the state space, e.g., the turning points in the Ant Maze (W-shape) task.
21

---

Hierarchical Reinforcement Learning with Uncertainty-Guide Diffusional Subgoals
Tasks
HIDI
HLPS
SAGA
HIGL
HRAC
HIRO
Reacher
0.95-0.00
0.54+0.14
0.80++0.05
0.89+0.01
0.74+0.110
0.76++0.02
Pusher
0.78 0.02
0.62:1012
0.20:10.12
0.3110.10
0.17:00.06
0.19:00.6
Point Maze
1.00
0.99110000
0.99110.00
0.71:10.20
0.9910.00
0.8910.110
Sparse
Ant Maze (U)
0.91=0.03
0.83++0022
0.82++0.02
0.52:#0.05
0.80=10.03
0.72++0.03
Ant Maze (W)
0.07=0.02
0.83++0.02
0.70++0.03
0.70++0.03
0.51+0.17
0.59++0.03
Stoch. Ant Maze (U)
0.91
0.80+
0.81+10011
0.75
0.73:10.02
0.61
Stoch. Ant Fall
0.84=0.02
0.78++0.04
0.48=00.05
0.52+0002
0.25+
0.28++0.03
Stoch. Ant FourRooms (Img.)
0.64 0.03
0.55++0.05
0.32++0.03
0.31+00.02
0.00++.00
0.00++0.00
Point Maze
1.00=0.00
1.00+10.00
0.94+0.00
0.98:10.22
0.99:00.00
0.81#0019
Dense
Ant Maze (U)
0.88=0.01
0.83+00011
0.80++0.04
0.83+0.07
0.76++0.04
0.75+00.07
Maze (W)
0.88
0.80++002
0.68++0.03
0.78+0.04
0.75++0.07
0.50++0.04
Stoch. Ant Maze (U)
0.02
0.88
0.87:10.03
0.82:10.03
0.70.10.04
0.80:10.03
Table Final performance of the policy obtained after 5M steps of training with sparse rewards, averaged over 10 randomly seeded trials with standard error.
(a) HIDI
(b) SAGA
(c) HIGH.
Goal
Subgoal
Reached
Subgoal
+ Start
+ End
(d) HRAC
(c) HIRO
Figure 5. Visualization of generated subgoals and reached subgoals of HIDI and compared baselines in Ant Maze (W-shape, sparse) with the same starting location.
22

---

Hierarchical Reinforcement Learning with Uncertainty- Guided Diffusional Subgoals
Module Parameter
Value
Number of hidden layers
Number of hidden units
256
Nonlinearity,
Mish
Diffusional Policy
Optimizer
Adam
Learning rate
10
Hyperparameter for RL objective
Number of diffusion steps N
GP loss weight
10
GP learning rate
3 x 10
Subgoal selection probability
0.1
Number of inducing states
16
Number of hidden layers
Number of hidden units per layer
300
Two- layer HRL, critic networks
Nonlinearity
ReLU
Optimizer
Adam
Learning rate, critic
10-3
Batch size, high level
100
Batch size, low level
128
Replay buffer size Random time steps
2 x 105
5 x 56
Subgoal frequency
10
Reward scaling, high level
0.1
Reward scaling, low level
1.0
Table 3. Network architecture and key hyperparameters of HIDI
23


Reviews:
[{"review_id": "FwabUdwk8t", "reviewer": "Reviewer_q2dx", "summary": "The paper introduces a novel framework for hierarchical reinforcement learning (HRL) that combines a conditional diffusion model for generating state-conditioned subgoals with a Gaussian Process prior for uncertainty quantification and regularization. This hybrid approach ensures robust and adaptive subgoal generation by balancing the GP's structural consistency with the diffusion model's flexibility. HIDI also incorporates a subgoal selection strategy that leverages both GP predictive means and diffusion samples to improve subgoal reachability and learning stability. Evaluated on challenging MuJoCo continuous control tasks, HIDI outperforms state-of-the-art HRL methods in terms of sample efficiency, stability, and success rates, particularly in sparse reward and stochastic environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments. \n\n**1. Comparison to Diffusion-based & Uncertainty-aware Baselines:**\n\nWe respectfully clarify that our evaluation already includes strong baselines covering an **uncertainty-aware approach, i.e., HLPS (Wang et al., 2024)**, and HIDI significantly outperforms HLPS on all tasks. Regarding diffusion-based RL, we emphasize that our experimental comparisons focused on state-of-the-art HRL methods, since our contribution lies in improving hierarchical RL decision making. **Directly comparing to methods like Diffuser or Diffusion-QL would be inapplicable: those methods address offline trajectory optimization or action-generation, whereas HIDI addresses online subgoal policy learning.** Nonetheless, we ensured our work is informed by those advances – for example, we cite Diffusion-QL’s success in modeling complex action distributions as a motivation. Rather than incorporate their entire pipelines (which would change the problem setting), we extracted the core insight (diffusion models can represent multi-modal distributions effectively) and applied it in a new context (subgoal generation). **By deliberately using a basic diffusion model, we demonstrated improvements on top of existing HRL algorithms without relying on specialized offline-training techniques.** We believe this choice actually strengthens our contribution: it shows that any future improvements in diffusion-based policy modeling can be combined with our HRL framework to yield even better results.\n\n**2. Benchmark Diversity and Alignment with Existing HRL Works:**\n\nWe acknowledge the reviewer’s point regarding the use of simulated MuJoCo environments. However, our choice is fully aligned with the standard practice in HRL research. **State-of-the-art HRL methods such as HIRO, HRAC, HIGL, SAGA, and HLPS have predominantly evaluated their methods on MuJoCo benchmarks, which provide a well-understood and controlled setting to measure progress in long-horizon, continuous control tasks.** Our selection of diverse tasks within the MuJoCo suite (including Reacher, Pusher, multiple Ant Maze variants, and a vision-based Ant FourRooms variant) was deliberate to **cover a wide range of challenges (e.g., sparse vs. dense rewards, stochasticity, and different state spaces)**.\n\nNonetheless, we agree that additional experiments on more diverse or real-world tasks (such as robotic manipulation on physical platforms or navigation in real environments) would further substantiate the generalizability of our approach. We are actively planning follow-up studies to evaluate HIDI in such settings.\n\n**3. Additional Theoretical Analysis:** \nWe have provided additional theoretical proofs of the HIDI framework (accessible via the anonymous link https://anonymous.4open.science/r/HIDI-32F6/icml2025___rebuttal.pdf), which addresses two critical aspects: (1) the validity of the diffusion model for state-conditioned subgoal generation, establishing conditions under which the reverse diffusion process accurately approximates the desired subgoal distribution and (2) the role of GP regularization in guiding the diffusion process by pulling generated subgoals toward the GP predictive mean, ensuring that subgoals are both achievable and structurally consistent. This comprehensive analysis reinforces our empirical results by providing formal guarantees for the effectiveness of our approach, and we invite the reviewer to refer to the additional theoretical analysis for full details.", "claims_and_evidence": "the claims made in the submission are largely supported by clear and convincing evidence, as the paper provides comprehensive experimental results and theoretical justifications", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem of hierarchical reinforcement learning (HRL), particularly in the context of long-horizon, continuous control tasks", "theoretical_claims": "The proofs for the theoretical claims are mathematically correct and logically consistent under the stated assumptions", "experimental_designs_or_analyses": "- While the selected baselines are strong, some recent diffusion-based or uncertainty-aware RL methods (if available) could have been included for a more comprehensive comparison. For example, methods that use diffusion models in other RL contexts might provide additional insights.\n- Although the chosen benchmarks are standard, they are limited to simulated MuJoCo environments. Additional experiments in more diverse or real-world tasks (e.g., robotic manipulation or navigation in real environments) would strengthen the generalizability of the results.", "relation_to_broader_scientific_literature": "The key contributions of the paper build on and extend several lines of research in hierarchical reinforcement learning (HRL), generative modeling, and uncertainty-aware learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SWh5juK94C", "reviewer": "Reviewer_6S8j", "summary": "This paper introduces Hierarchical Reinforcement Learning (HRL) using a conditional diffusion model combined with Gaussian Process (GP) regularization for sequential decision making. Specifically, the authors propose to train a diffusion policy at the high-level thus to generate subgoals that align with the low-level policy. Furthermore, to improve the data efficiency of diffusion policy, the authors introduced a Gaussian Process prior over subgoal states as a regularizer. During inference, a subgoal is first sampled from either the diffusion policy or the Gaussian prior, then the atomic action is sampled from the low-level goal-conditioned policy. The proposed method was evaluated on a set of continuous control task and showed better performance over baselines.\n\n## update after rebuttal\nThank the authors for the follow-up discussion. My concerns have been addressed. I'd like to raise my score from 1 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments. \n\n**1. Low-Level Policy Training & High-Level Adaptation:**\n\nAs correctly noted by the reviewer, our approach builds on the adapted HIRO framework, where the low-level policy trains on intrinsic rewards to achieve subgoals reliably, without direct extrinsic objective access. We focus on adapting the high-level policy because it faces non-stationarity – the set of feasible subgoals changes as the low-level improves. Adapting the high-level ensures generated subgoals align with current low-level capabilities while pursuing the task objective. Forcing low-level adaptation risks convergence to conservative local optima. Our approach (diffusion + GP regularization + selection) keeps the high-level agile, supported by ablation results (Fig. 2) showing improved sample efficiency and performance, and **theoretical analysis (Appendix A.1, updated proofs linked below)**.\n\n**2. Ablation Study Clarity (HIDI-A, HIDI-B):** \n\nThe reviewer's interpretation of the baselines is correct: HIDI-A denotes our framework without the subgoal selection strategy, i.e., the high-level always executes the diffusion model’s subgoal (with GP prior) without optionally selecting the GP’s predictive mean. HIDI-B is a variant with no GP regularization and no subgoal selection - the high-level uses only the diffusion policy for subgoals. \n\n**3. Clarifying the Role of the Gaussian Process (GP) Prior:**\n\nThe GP prior is a critical component of our framework, and its contribution is twofold: (1) The GP models the state-subgoal distribution from the replay buffer. The $L_{gp}$ loss term (Eq. 11) regularizes the diffusion model towards generating subgoals consistent with previously achieved transitions. This focuses learning on feasible regions, reducing wasted samples. The ~15-16% performance gain from HIDI-B (diffusion-only) to HIDI-A (diffusion+GP) in Fig. 2a-b quantifies this benefit. Table 1 further shows HIDI (with GP) generates more achievable subgoals (smaller generated vs. reached state distance) compared to variants without GP. We have strengthened our theoretical proof (accessible via the anonymous link https://anonymous.4open.science/r/HIDI-32F6/icml2025___rebuttal.pdf) by analyzing the role of GP regularization in guiding the diffusion process by pulling generated subgoals toward the GP predictive mean, ensuring that subgoals are both achievable and structurally consistent. (2) The GP predictive mean $\\mu_*$, reflecting low-uncertainty regions, guides exploration via our subgoal selection strategy (Eq. 18). With probability $\\epsilon$, selecting $\\mu_*$ steers the agent towards areas supported by experience, balancing exploration and exploitation. Removing the GP (HIDI-B) leads to less stable learning (Fig. 2).\n\nTo further substantiate our claims, we have conducted an additional ablation study in the challenging Ant Fall environment to specifically **evaluate the impact of GP alone (Baseline-GP) by replacing the diffusional policy with the original baseline policy**. From this comparison, we observe that **GP-alone approach can improve the baseline by ~16%**. This result highlights the significant performance benefit provided by the GP-driven regularization and uncertainty-aware subgoal selection, even independent of the diffusion policy.\n| Method |  HIDI (Proposed)  |  Baseline-GP  |  Baseline  |\n|:------------|:------------|:------------|:------------|\n|  Success Rate  |  0.84±0.02  | 0.67±0.02  |  0.51±0.03  |\n\n**4. Comparison to an Alternative High-Level Exploration Strategy:**\n\nFollowing the reviewer's suggestion, we have evaluated epsilon-exploration as an alternative strategy in the challenging Ant Fall environment: \n|  Subgoal Selection  |  HIDI (Proposed)  |  Epsilon-Greedy Exploration  |  HIDI-A  |\n|:------------|:------------|:------------|:------------|\n|  Success Rate  |  0.84±0.02  | 0.71±0.12  |  0.76±0.03  |\n\nFrom this comparison, we observe that epsilon-greedy exploration considerably degrades overall performance when subgoals are randomly generated with a probability of 0.1 (aligning with our subgoal selection scheme). This clearly demonstrates the benefit of our proposed subgoal selection strategy, which aligns subgoals with feasible trajectories, rather than relying on random subgoal sampling that is likely to yield infeasible subgoals.\n\n**5. Sparse Reward Setting Characterization:**\n\nWe utilize standard implementations of the MuJoCo environments commonly used in HRL research. The \"-1\" sparse reward setting aligns with the existing reward formulations employed in prior state-of-the-art HRL papers evaluating on these environments (e.g., HIGL, DHRL, LESSON, HESS). Changing this widely adopted setup would require re-running all baseline methods, which is infeasible given the current time constraints.", "claims_and_evidence": "Partially. The description of the ablation study is confusing. For details, please see the **Experimental Designs or Analyses** section.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No.", "experimental_designs_or_analyses": "1. The authors claim that to address the discrepancy between the high-level and low-level policies, _“This issue requires the high-level policy to swiftly adapt its strategy to generate subgoals that align with the constantly shifting low-level skills.”_ However, the manuscript does not specify how the low-level policy is trained. Given that this work is largely built on HIRO, I assume the low-level policy follows the same training paradigm—where it learns to achieve subgoals assigned by the high-level policy solely through intrinsic rewards, without direct knowledge of the task objective. If my understanding is correct, my question is: why do the authors focus on adapting the high-level policy to the low-level policy rather than the other way around? I could think of a local optimal where the high-level struggles to maximum the task target while only propose subgoals that are within the low-level policy's reach.\n2. I find it challenging to fully grasp the ablation study. Could the authors clarify the differences between HIDI-A and HIDI-B? According to the manuscript, _“HIDI-A denotes a baseline without performing the proposed subgoal selection strategy,”_ which I interpret as removing Equation (18). If that’s the case, how are subgoals selected during evaluation? Additionally, the manuscript states that _“HIDI-B is a baseline without adopting GP regularization and subgoal selection,”_ which I assume means that subgoals in HIDI-B are sampled directly from the conditional diffusion model. Does this imply that HIDI-A instead samples subgoals from the Gaussian process prior?\n3. Since the ablation study is somewhat unclear, I struggle to see how the Gaussian process prior contributes to sample efficiency and exploration. Could the authors provide further clarification or additional evidence?\n4. Furthermore, it would be helpful to compare HIDI against an alternative high-level exploration strategy, such as an $\\epsilon$-greedy approach, where the Gaussian process prior is replaced with randomly sampled subgoals.\n5. I am don't think the characterization of the ‘-1’ reward process in line 360 as being truly sparse. Could the authors present results on a task where rewards are genuinely sparse—i.e., zero reward everywhere except when the target is reached?\n\n\n====================================\nI'm happy to raise my score if my concerns are addressed.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RDwme6ghVc", "reviewer": "Reviewer_VpQ5", "summary": "This paper aims to mitigate the non-stationary effects caused by the simultaneous training of both high-level and low-level policies in HRL. The authors introduce a novel framework, HIDI, that leverages a conditional diffusion model regularized with a Gaussian Process (GP) prior to generate diverse and achievable subgoals. The authors validate this method on a broad set of sparse and dense reward control tasks and conduct ablation experiments to evaluate the impact of each component on the overall performance.\n\n### update after rebuttal\nThanks for the detailed response. My concerns have been addressed, therefore, I will increase my score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments. \n\n**1. Non-Stationarity Analysis:**\nWe clarify that HIDI does not claim to eliminate non-stationarity, an inherent challenge in HRL due to the evolving low-level policy. Instead, HIDI mainly addresses **mitigating its adverse effects**, in line with previous HRL methods such as HAC and HIRO.\n*   **Theoretical Analysis:** We have provided additional theoretical proofs of the HIDI framework (accessible via anonymous link - https://anonymous.4open.science/r/HIDI-32F6/icml2025___rebuttal.pdf) which detail: (1) the validity of using a diffusion model for state-conditioned subgoal generation, showing it can accurately approximate the target distribution, and (2) the role of GP regularization in guiding the diffusion process by aligning generated subgoals with the GP predictive mean, promoting achievable and structurally consistent subgoals. This analysis reinforces our empirical findings by providing formal justification for HIDI's effectiveness.\n*   **Adaptive Subgoal Generation:** Our approach employs a conditional diffusion model to generate a rich, state-conditioned subgoal distribution. Although this distribution is inherently non-stationary as the environment and low-level policy evolve, the diffusion model’s flexibility allows it to adapt quickly to new data. **By continually training on relabeled subgoals, the diffusion model adjusts to reflect the current capabilities of the low-level policy.** This results in subgoals that are more aligned with what the low-level can reliably achieve.\n*   **GP Regularization:** The incorporation of a GP prior further stabilizes the high-level policy by quantifying uncertainty. The GP provides structured guidance on regions of the state space that have been well explored, thereby encouraging the high-level policy to generate subgoals that are not only diverse but also practically achievable. In this way, **while the underlying distribution remains non-stationary, the GP acts as a corrective mechanism, anchoring the subgoal proposals to areas supported by historical data**.\n*   **Empirical Robustness:** Fig. 1 (smoother curves) and Table 1 (smaller generated vs. reached subgoal distance) show HIDI's improved alignment and robustness compared to baselines, indicating effective mitigation of non-stationarity issues.\n\n\n**2. Superior Sample Efficiency:**\nWe respectfully maintain that Fig. 1 supports our claim of superior sample efficiency, defined as achieving a performance level with fewer environment interactions (x-axis). HIDI consistently reaches high success rates faster than baselines.\n*   **Steeper Curves:** On challenging tasks (Reacher, Pusher, Ant Fall, Ant FourRooms), HIDI's learning curve rises more steeply, indicating faster convergence to high performance with fewer samples.\n*   **Performance at 3M Steps:** The table below shows HIDI achieves the highest IQM success rate at 3 million steps across nearly all tasks, demonstrating superior efficiency.\n\n**Table: Interquartile mean (IQM) of success rate with 95% stratified bootstrap confidence intervals, evaluated at 3 million steps:**\n| Task                          | HIDI             \t| HLPS             \t| SAGA             \t| HIGL             \t| HRAC             \t| HIRO             \t|\n|:------------------------------|:---------------------|:---------------------|:---------------------|:---------------------|:---------------------|:---------------------|\n| Reacher                       | 0.950 [0.945, 0.955] | 0.567 [0.432, 0.704] | 0.776 [0.734, 0.824] | 0.890 [0.886, 0.894] | 0.692 [0.602, 0.782] | 0.809 [0.790, 0.827] |\n| Pusher                        | 0.621 [0.604, 0.637] | 0.543 [0.431, 0.659] | 0.159 [0.056, 0.271] | 0.240 [0.148, 0.335] | 0.152 [0.103, 0.203] | 0.229 [0.178, 0.281] |\n| Ant Maze (W)                  | 0.899 [0.881, 0.917] | 0.722 [0.705, 0.738] | 0.462 [0.437, 0.489] | 0.622 [0.596, 0.651] | 0.463 [0.310, 0.623] | 0.449 [0.423, 0.476] |\n| Point Maze                    | 1.000 [1.000, 1.000] | 0.817 [0.743, 0.896] | 0.980 [0.975, 0.985] | 0.715 [0.565, 0.872] | 0.990 [0.986, 0.994] | 0.779 [0.754, 0.806] |\n| Ant Maze (U)                  | 0.908 [0.881, 0.936] | 0.890 [0.874, 0.907] | 0.820 [0.803, 0.838] | 0.590 [0.544, 0.638] | 0.821 [0.796, 0.846] | 0.779 [0.754, 0.806] |\n| Stoch. AM (U)                 | 0.910 [0.906, 0.914] | 0.857 [0.811, 0.904] | 0.840 [0.835, 0.845] | 0.591 [0.565, 0.619] | 0.741 [0.723, 0.759] | 0.750 [0.724, 0.777] |\n| Stoch.  Ant Fall          \t| 0.759 [0.742, 0.776] | 0.751 [0.713, 0.791] | 0.252 [0.209, 0.300] | 0.421 [0.404, 0.438] | 0.041 [0.016, 0.068] | 0.021 [0.000, 0.044] |\n| Stoch. Ant FourRooms (Img.) | 0.489 [0.464, 0.516] | 0.402 [0.354, 0.452] | 0.241 [0.214, 0.267] | 0.290 [0.273, 0.307] | 0.000 [0.000, 0.000] | 0.000 [0.000, 0.000] |", "claims_and_evidence": "Even when generating a state-conditioned distribution of subgoals, this distribution remains non-stationary. The authors lack a more detailed analysis to explain why HIDI is better equipped to address the non-stationary issue. Moreover, the authors claim that HIDI achieves better sample efficiency, but the experimental results in Figure 1 do not support this claim.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes. I check the selection of benchmarks and baselines, as well as the design of ablation studies.", "relation_to_broader_scientific_literature": "This work improves the training stability and convergence performance of HRL, which will facilitate the application of reinforcement learning in sparse reward tasks, such as long-horizon manipulation and navigation tasks.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "***Strength***\n1. The non-stationary issue is a critical problem in HRL.\n2. The use of a diffusion model to generate subgoals is both a reasonable and scalable approach, while incorporating a GP prior effectively reduces the sample requirements of the diffusion model.\n3. The experimental results demonstrate that HIDI significantly outperforms other baselines in terms of policy convergence and training stability.\n4. The authors conduct extensive ablation studies to analyze the contribution of each component.\n\n***Weaknesses***\n1. The authors do not provide a detailed analysis to explain why HIDI is better suited to address the non-stationary issue, as even when generating a state-conditioned distribution of subgoals, this distribution remains non-stationary.\n2. The authors claim that HIDI achieves superior sample efficiency; however, the experimental results in Figure 1 do not support this assertion.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VRtrkYdLJx", "reviewer": "Reviewer_tgC3", "summary": "This paper introduces an HRL framework that leverages a diffusion model for subgoal generation and GP regularization to enhance subgoal quality and reachability. The proposed method addresses the instability and inefficiency of traditional HRL by learning a state-conditioned subgoal distribution via a conditional diffusion model, while GP provides uncertainty estimates to guide reliable subgoal selection. A hybrid subgoal selection strategy balances exploration and exploitation by combining GP-predicted subgoal means with diffusion-sampled subgoals. Experimental results on MuJoCo robotic and navigation tasks demonstrate that HIDI achieves superior sample efficiency and final performance compared to existing HRL baselines (HIRO, HRAC, HIGL, SAGA).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the insightful comments. \n\n**1. Theoretical Analysis of Diffusion-Based Subgoal Generation and GP Regularization**\n\nWe appreciate the reviewer’s insightful comments on the theoretical justifications. In the anonymous link https://anonymous.4open.science/r/HIDI-32F6/icml2025___rebuttal.pdf, we have provided a formal, detailed proof addressing these concerns. Specifically, for the diffusion model, we rigorously demonstrate that, under a bounded noise prediction error, the reverse diffusion process converges to a valid state-conditioned subgoal distribution. Regarding the GP regularization, our analysis shows that the GP not only quantifies uncertainty but actively guides the diffusion process - pulling generated subgoals toward the GP predictive mean, thereby enforcing that the generated subgoals are both reachable and well-structured. This formal proof clarifies that the GP component does not merely filter outputs but plays a critical role in stabilizing subgoal generation in HRL. We invite the reviewer to refer to the additional theoretical analysis for details.\n\n**2. Subgoal Reachability Metric**\n\nWe thank the reviewer for this valuable suggestion. In our submission, we already **quantify reachability by reporting the distance between the generated subgoals and the reached subgoals (i.e., the final state after a k-step low-level rollout) in Table 1**. This metric, which is consistent with the state-of-the-art subgoal generation method **SAGA**, provides a clear quantitative measure of subgoal reachability. Additionally, Figure 5 offers a qualitative visualization of generated versus reached subgoals in the Ant Maze (W-shape, sparse) environment, using the same starting location for direct comparison across methods. We invite the reviewer to refer to Table 1 and Figure 5 in the manuscript for detailed quantitative and qualitative comparisons.\n\n**3. Ablation Study of GP Alone Impact**\n\nFollowing the reviewer’s suggestion, we conducted an ablation study in the challenging Ant Fall environment specifically evaluate the impact of GP alone (Baseline-GP) by replacing the diffusional policy with the original baseline policy. From this comparison, we observe that **GP-alone approach can improve the baseline by ~16%**. This result highlights the significant performance benefit provided by the GP-driven regularization and uncertainty-aware subgoal selection, even independent of the diffusion policy.\n| Method |  HIDI (Proposed)  |  Baseline-GP  |  Baseline  |\n|:------------|:------------|:------------|:------------|\n|  Success Rate  |  0.84±0.02  | 0.67±0.02  |  0.51±0.03  |\n\n**4. Failure Cases**\n\nIn our extensive experiments on the MuJoCo benchmarks, we have not encountered significant failure cases attributable to GP misguidance. In all tested environments, the GP consistently provided reliable uncertainty estimates that, when integrated with the diffusion model, improved subgoal reachability and overall performance. That said, we acknowledge that further investigation in even more challenging or real-world scenarios may reveal nuanced failure modes. We plan to explore these avenues in future work to better understand potential limitations. For now, our results indicate that within the tested settings, the GP component is robust and does not introduce harmful bias into subgoal generation or selection.\n\n**5. Quantification of Computational Overhead**\n\nWe quantified the overhead from the diffusion model and GP: HIDI's inference time is ~37% longer than baseline, using ~15% more GPU memory and ~12% more RAM.", "claims_and_evidence": "The paper mainly claims that incorporating diffusion-based subgoal generation and GP regularization improves the stability and efficiency of hierarchical reinforcement learning (HRL). I think it is generally well-supported:\n\n- Diffusion-based subgoal generation is validated through improved sample efficiency and final performance in MuJoCo experiments. The model learns a state-conditioned subgoal distribution, which reduces the instability often seen in HRL methods like HIRO.\n- GP-based regularization provides uncertainty estimation for subgoal selection, leading to better reliability and structured exploration, as evidenced by higher success rates and faster convergence in complex navigation tasks.\n\n**Potential Limitations:**\n\n- While GP improves stability, the paper does not provide a direct ablation study on the impact of GP alone. It would be valuable to isolate its effect from the diffusion model.", "methods_and_evaluation_criteria": "The proposed methodology is sound, as stated in the claim and evidence part.\n\nAlso, the evaluation methodology is appropriate, using: 1. MuJoCo benchmarks  2. HRL metrics (success rate, sample efficiency, asymptotic performance)3. Comparisons against strong baselines (HIRO, HRAC, HIGL, SAGA)\n\n\nHowever, diffusion models can be computationally intensive, and the paper does not quantify the additional overhead vs. standard HRL approaches although conducted thorough performance comparison with them.", "theoretical_claims": "The theoretical justifications rely primarily on:\n\n1. **Diffusion Model for Subgoal Generation**\n    \n    - The paper assumes that a diffusion model can effectively learn a state-conditioned subgoal distribution, which reduces instability in HRL.\n    - The reverse diffusion process is well-established in generative modeling, but its applicability to subgoal generation in HRL lacks theoretical guarantees. e.g., under what conditions does the learned subgoal distribution remain valid?\n2. **Gaussian Process Regularization**\n    \n    - The paper claims that GP improves subgoal reachability and structure by guiding selection based on uncertainty estimation.\n    - GP modeling is theoretically well-grounded, but the interaction between GP uncertainty and the diffusion model is not rigorously analyzed.i.e., does GP actually guide the diffusion process, or does it primarily serve as a filtering mechanism?", "experimental_designs_or_analyses": "The experimental design is thorough and sound, with clear comparisons to state-of-the-art HRL methods across multiple evaluation metrics.\n\nHowever, while GP improves subgoal reliability, an additional reachability metric (e.g., % of subgoals successfully achieved) would strengthen the argument as HRL models fail quite often when evaluated on more complex or real-world environments, adding such metric may help future work to understand the model's capability in a broader term.\nThe experiments do not explicitly remove GP to quantify its standalone impact.\nThe method performs well on MuJoCo, but an analysis of failure cases (e.g., when GP misguides subgoal selection) would provide valuable insights as it may help figure out whether the failure is a common one due to the complexity of the environment or that is from the model's own bias.", "relation_to_broader_scientific_literature": "This paper builds upon and extends prior work in Hierarchical Reinforcement Learning (HRL), diffusion models, and uncertainty-aware decision-making.\nUnlike HIRO, which relies on hindsight experience replay to correct subgoals, this paper generates subgoals dynamically using a diffusion model. Compared to HIGL (Zhang et al., 2021) and SAGA (Nasiriany et al., 2022), which also explore goal-conditioned RL, this paper adds GP for uncertainty estimation. Prior trajectory-based reinforcement learning (Janner et al., 2022) has incorporated diffusion models for planning, but this paper extends their use to subgoal selection in HRL, which is an important contribution.", "essential_references_not_discussed": "I think the authors did cover the necessary references in the paper.", "other_strengths_and_weaknesses": "As mentioned above.", "other_comments_or_suggestions": "I appreciate the authors' efforts to provide concrete proof of the model they proposed, no further comments.", "questions_for_authors": "I do not have further questions for the authors. I would be happy to increase my score if the authors answer my questions or successfully defend their claims.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new HRL framework, HIDI, which leverages a conditional diffusion model to generate state-conditioned subgoals. The authors propose training a high-level diffusion policy to generate subgoals compatible with a low-level policy and use a Gaussian Process (GP) prior over subgoal states to improve reachability and stability. During training, subgoals are sampled from either the diffusion model or the GP prior, and primitive actions are then generated by the low-level goal-conditioned policy. The authors evaluated their method on MuJoCo tasks and showed that it outperforms state-of-the-art HRL methods in terms of sample efficiency, training stability, and success rate.

All reviewers agreed that the key claims made in this work are well supported by both theoretical justifications and detailed experimental results. Two reviewers specifically noted that they closely examined the theoretical proofs and found them sound. Three reviewers highlighted that the experimental results demonstrate significant performance improvements over relevant baselines, particularly in terms of convergence and stability. They also made positive comments on the thorough ablation studies presented by the authors.

Some concerns were raised during the review process. One reviewer felt that more recent diffusion-based and uncertainty-aware RL methods could have been included as baselines. The authors clarified that comparisons with uncertainty-aware baselines were already included in the paper. They also argued that a direct comparison with diffusion-based RL methods is not feasible since those methods tackle a fundamentally different setting—offline trajectory optimization (or action generation) rather than online subgoal policy learning. This reviewer also raised concerns about the generalizability of the authors' conclusions, in particular because the experiments were limited to MuJoCo tasks. The authors argued that they deliberately selected tasks from the MuJoCo suite to cover a diverse range of challenges, including sparse and dense rewards, varying levels of stochasticity, and qualitatively different state spaces. The reviewer did not respond to the rebuttal.

Another reviewer was concerned about some of the authors’ claims, ablation results, and comparisons to alternative exploration strategies. The authors provided a detailed rebuttal, including new theoretical results and additional experiments. The reviewer acknowledged that most of the points they raised were addressed but remained concerned about potential convergence to suboptimal solutions. The authors clarified that, although HIDI's GP prior acts as an adaptive regularizer that promotes stability, its influence diminishes in uncertain regions, which allows exploration to be once again guided by the primary RL objective. The reviewer was satisfied with the rebuttal and raised their score to 3.

A third reviewer asked for clarification on how HIDI handles the non-stationarity of the state-conditioned subgoal distribution. The authors responded by noting that non-stationarity is a known challenge affecting all HRL methods, since lower-level policies are continuously updated. They then explained how HIDI mitigates its effects using strategies consistent with prior work. Importantly, the authors also introduced new proofs establishing conditions under which the reverse diffusion process accurately approximates the desired subgoal distribution, as well as proofs clarifying the role of GP regularization in guiding the diffusion process. Finally, the reviewer argued that the results in Figure 1 did not fully support the authors' claims about sample efficiency. The authors addressed this by presenting a detailed table summarizing learning curves, which confirmed faster convergence with fewer samples. The reviewer said that their concerns were addressed and increased their score to 3.

The fourth reviewer asked about the conditions under which the learned subgoal distribution remains valid. The authors provided new proofs showing that, under bounded noise prediction error, convergence to a valid state-conditioned subgoal distribution could be guaranteed. The reviewer suggested that quantifying additional metrics—such as the percentage of successfully reached subgoals—would help support the authors' performance claims. The authors clarified that these results were already captured in Table 1, which reports distances between generated subgoals and the final states reached after k-step rollouts. The reviewer said that their concerns were addressed and raised their score to 3.

Although the reviewers raised relevant concerns, the consensus is that this paper offers valuable theoretical and practical insights on how to automatically generate subgoals in HRL. The reviewers also agreed—especially after the rebuttal phase—that all key claims made in this work are adequately supported. Based on the reviewers’ feedback and the authors’ thorough rebuttal, I do not see any critical flaws, missing analyses or comparisons, or methodological issues. Finally, the constructive discussion phase, along with the reviewers’ acknowledgment of the paper’s contributions and their updated scores, gives me confidence that the ICML community will benefit from the insights presented in this work.
Decision: Accept (poster)