Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning
Authors: Xianghua Zeng
Hao Peng
Yicheng Pan
Angsheng Li
Guanlin Wu
Venue: NeurIPS 2025 poster ()
Keywords: offline reinforcement learning
diffusion models
structural information principles
Primary area: reinforcement_learning

Abstract: Diffusion-based generative methods have shown promising potential for modeling trajectories from offline reinforcement learning (RL) datasets, and hierarchical diffusion has been introduced to mitigate variance accumulation and computational challenges in long-horizon planning tasks. However, existing approaches typically assume a fixed two-layer diffusion hierarchy with a single predefined temporal scale, which limits adaptability to diverse downstream tasks and reduces flexibility in decision making. In this work, we propose SIHD, a novel Structural Information-based Hierarchical Diffusion framework for effective and stable offline policy learning in long-horizon environments with sparse rewards. Specifically, we analyze structural information embedded in offline trajectories to construct the diffusion hierarchy adaptively, enabling flexible trajectory modeling across multiple temporal scales. Rather than relying on reward predictions from localized sub-trajectories, we quantify the structural information gain of each state community and use it as a conditioning signal within the corresponding diffusion layer. To reduce overreliance on offline datasets, we introduce a structural entropy regularizer that encourages exploration of underrepresented states while avoiding extrapolation errors from distributional shifts. Extensive evaluations show that SIHD significantly outperforms state-of-the-art baselines in decision-making performance and demonstrates superior generalization across diverse scenarios.

Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning
Xianghua Zeng', Hao Peng', Yicheng Pan', Angsheng Li Guanlin Wu²
State Key Laboratory of Complex & Critical Software Environment, Beihang University, Beijing, China Zhongguancun Laboratory, Beijing. China
National University of Defense Technology. Changsha, China
(zengxianghua, penghao, yichengp, angsheng) (bbuga edu. cn, wuguanling in16@nudt. edu. cn
Abstract
Diffusion-based generative methods have shown promising potential for modeling trajectories from offline reinforcement learning (RI.) datasets, and hierarchical diffusion has been introduced to mitigate variance accumulation and computa- tional challenges in long-horizon planning tasks. However, existing approaches typically assume a fixed two-layer diffusion hierarchy with single predefined temporal scale, which limits adaptability to diverse downstream tasks and reduces flexibility in decision making. In this work, we propose SIHD, novel Structural Information-aen based Hierarchical Diffusion framework for effective and stable offline policy learning in long- long-horizon environments with sparse rewards. Specifically, we analyze structural information embedded in offline trajectories construct the diffusion hierarchy adaptively, enabling flexible trajectory modeling across multiple temporal scales. Rather than relying on reward predictions from local- ized sub-trajectories, we quantify the structural information gain of each state community and use it as a conditioning signal within the corresponding diffu- sion layer. To reduce overreliance entropy regularizer that encourages exploration of underrepresented states while avoiding extrapolation errors from distributional shifts. Extensive evaluations show that SIHD significantly outperforms state-of-the -o-thee-rrt baselines in decision decision-making performance and demonstrates superior generalization across diverse scenarios.
offline datasets, we introduce structural
Introduction
Offline reinforcement learning (also known as batch RL) Lange et al. 2012 trains policies using pre- collected data without further interaction with the environment Levine al. 2020 This paradigm particularly well-suited to high-stakes domains where online data collection is costly or infeasible, such healthcare [Tang et al., 2022). education [De Lima and Krohling 2021 and robotic control (iillaflor et al. 2022). Out-of
(OOD) states and suttion-uunderrppessenttt or absent in offline datasets- often cause temporal-diin -iiffreee-e learning methods suffer from severe extrapolation
errors [Kumar et al.) 2019 Fujimoto et al., 2019]. Additionally, suboptimal and diverse historical trajectories give rise to the mode-covering challenge [Chen et al.) 2023 2024b), where conventional unimodal policies fail to capture the multi- multi-modal nature of offline behavioral strategies.
To mitigate these challenges, recent approaches have incorporated diffusion models (Ho et al. 2020
with strong generative capacity to build expressive biffusion-based policies [Chen et al. 2023 2024h
Offline sequential dacision-making has been further reframed trajectory generation task, where diffusion models are conditioned on reward-related signals (e.g., returns or reward-to-go) to generate high-return sequences [Liang al. 2023 He et al. 2023 However, deploying these models in
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

a) offline trajectory
b) 2-layer rigid diffusion
c) multi-scale hierarchy d) regularized exploration
Figure/ 1: Illustrative example of navigation from the green start point to the red goal: (a) the offline
suboptimal trajectory; (b) the rigid two-layer diffusion hierarchy with a single predefined temporal scale; (c) the adaptive multi-scale hierarchical diffusion framework based structural information principles; (d) the enhanced state exploration guided by structural entropy regularization.
long-horizon tasks remains difficult due to the exponential increase in value estimate variance [Ren ct al. ]021] and the high computational cost of iterative denoising steps [Wang et al. [022].
To improve efficiency in long-horizon decision-making. hierarchical policies have been incorporated into diffusion- -based offline RL, enabling the decomposition of complex tasks into manageable subproblems guided by intermediate subgoals [Sacerdoti 1974 Pertsch et al.] (022). Early approaches (Ajay et al. 2022 He et al.) 2023] introduce manually predefined skills into diffusion models via one-hot encodings, which limit the scalability of the resulting policies. Subsequently, the HDMI framework [Li et al. 2023] advances this direction by learning reward-conditioned subgoals and generating goal-directed trajectories using hierarchical diffusion model. Building on this, the Hicrarchical Diffuser [Chen et al., 2024a] utilizes lightweight trajectory segmentation to streamline subgoal inference, further enhancing long-horizon planning performance.
Despite their success, existing methods typically rely on a single predefined temporal scale to segment offline trajectories (Figure I [)) and assume a fixed two-layer diffusion hierarchy composed of subgoal and action layers (Figure [bb).. Such rigid structures hinder adaptability to varying temporal patterns and task-specific complexities, limiting both decision-makin performance and flexibility. Recent work [Evans and Simşek 2023] has demonstrated the potential of multi-layer policy hierarchies, structured from state transition dynamics, to enhance generalization in compositional long-horizon tasks. This raises a central open challenge in offline RL; how can historical trajectories be systematically analyzed to construct diffusion hierarchy that is both generalizable and task-aware?
In this work, we propose novel Structural Information- Hierarchical Diffusion framework, called SIHD, for stable and effective offline policy learning. We begin by extracting structural information from a similarity-guider topological graph constructed over state elements, from which tree- nre-struc state communities are derived. Offline trajectories are adaptively segmented based on multi-scale diffusion hierarchy
community partitioning at each layer, enabling the construction (Figure (c).. We quantify the structural information gain of each state community and use it as conditional guidance between adjacent diffusion layers, thereby reducing reliance on reward predic- tion over locally receptive sub-trajectories. We further introduce structural entropy regularizer to promote exploration of underrepresented states in historical trajectories (Figure (d), mitigating over- reliance on offline datasets. To prevent extrapolation errors arising from distributional shifts between behavioral and learned policies, this exploration is constrained to the lowest-level communities in the hierarchy. Comparative evaluations the D4RL benchmark show that SIHD outperforms both non non-hierarchical and hierarchical state-of th-the-r baselines by up to 12.6% in decision- decision-making making perfor- mance and exhibits superior generalization in long-horizon sparse- reward tasks. Our contributions are summarized as follows:
We propose a novel hierarchical diffusion framework that leverages structural information from historical trajectories to enable adaptive modeling across multiple temporal scales.
We quantify the structural information gain of each state community and use it as conditional guidance for the corresponding diffuser over localized sub-trajectories.
We introduce a structural entropy regularizer that promotes exploration of underrepresented states within low-level communities, mitigating overreliance on offline datasets and reducing extrapolation errors from distributional shifts.
We conduct comprehensive evaluations on the D4RL benchmark, demonstrating that SIHD signifi- cantly outperforms state-of-the-an baselines in both decision-making performance and generalization on long- horizon,, sparse-reward tasks.

---

2 Related Work
2.1 Hierarchical Decision-Making
Hierarchical reinforcement learning (HRL. [Sacerdoti 1974 Pertsch et al. 2020] addresses long-
horizon decision-making tasks by decomposing them into layered subtasks, where higher-layer policies orchestrate lower-layer primitives to enable temporal abstraction. For example, Iris (Man- dlekar al. 2020 learns implicit hierarchical policies from offline robotic data, while HiGoC [Li et al. 2022 integrates hierarchical abstraction with goal-conditioned offline RL for multi-stage planning. Multi-level hierarchies [Evans and Şimşek 2023 extend HRL by systematically organizing skills at varying levels of granularity, enabling flexible adaptation to compositional tasks.
In offline settings, HRL mitigates challenges such distributional shift and sparse rewards through structured policy dccomposition. OPAL [Ajay et al. 2021] accelerates learning by discovering reusable primitives from offline datasets, while recent work [Villecroze et al. 2022 Rao et al. 2022 explores data-driven skill extraction without predefined task hierarchies. Despite these advances, offline HRL remains prone instability due to the deadly triad of function approximation, bootstrap- ping, and off-policy learning [Van Hasselt et al. 2018). further exacerbated by limited data coverage (Ma et al. 2022] and persistent reward sparsity Kumar et al., 2020).
These issues underscore the need for more stable and generalizable hierarchical frameworks for offline policy learning in mang-hrrion decision decision-making scenarios with limited interaction and feedback from the environment.
2.2 Diffusion-based Offline RL
Diffusion models [Ho et al. 2020] have emerged as powerful tool in offline reinforcement learning (RL), offering strong distribution-matchinn and -ennerecegnnnttoonn capabilities that help mitigate extrapolation errors [Kumar et al. 2019 Fujimoto et al., 2019] and poor mode coverage [Chen
al.) 2023] 2024b] commonly observed conventional methods. By synthesizing high-fidelity trajectories aligned with offline data, diffusion based methods [Janner et al., 2022] reformulate policy optimization as a generative modeling problem, enabling diverse behaviors while avoiding out-of-distribution actions.
Despite these strengths, diffusion- based offline RL faces challenges in long-horizon tasks, where vari- ance accumulation across extended trajectories [Ren et al. [202]] and high computational costs [Wang] al. 2022 limit scalability. Recent work has sought address these limitations by introducing hierarchical diffusion models (Li et al. 2023 Chen et al. 2024a]. which decompose decision-making into a two-stage process: high-layer goal planning followed by low-layer action generation. However, these methods typically rely on fixed temporal segmentation and manually defined two-layer hierar- chies, limiting their adaptability to tasks requiring dynamic horizon adjustments or reasoning over multiple temporal scales.
These limitations underscore the need for hierarchical diffusion frameworks that flexible, task- aware, and adaptively constructed from offline trajectories to enable robust long-horizon decision- making in offline RL.
2.3 Structural Information Principles
Structural information principles [Li and Pan 2016 Li 2024) quantify uncertainty in graph-based dynamics through structural entropy and hierarchical partitioning. Specifically, structural entropy measures the minimum number of bits required to encode the probability distribution of vertices reachable via single-step random walk on a graph. In its one-dimensional form, it corresponds to the Shannon entropy of the graph's 's degree distribution, providing an upper bound on dynamic uncertainty. Hierarchically grouping tightly connected vertices into communities reduces the encoding cost, yielding a tree-structured partitioning known as an encoding tree. Encoding trees derived from topological graphs have been effectively applied in graph learning (Wu et al.. 2022] network analysis Zeng et al. 2024], and decision- mmaking tasks [Zeg et al. 20.
In this work, we leverage structural information embedded in historical trajectories to construct an adaptive multi-scale hierarchical diffusion framework, enabling stable and effective policy learning in long- horizoor,zon, sparse-reward environments.
3

---

3
Preliminaries
In this section, we formalize the foundational concepts of offline reinforcement learning (RL), diffusion probabilistic models, and structural information principles. The primary notations used throughout the paper are summarized Appendix A. for reference.
3.1 Offline RL
Reinforcement learning is typically formalized as a Markov Decision Process (MDP) [Bellman defined by the tuple M =< S, A, P, R, >, where S denotes the state space, A the action space, P(s'ls, a) the transition function, R(s, a) the reward function, and yy the discount factor. The objective of parameterized policy (a(s)) is maximize the expected discounted return, 7 ( 4t)]. This return is estimated using state-action value function 2s(s, o), which learned via temporal-differeen
1957].
methods (Sutton 1988
In offline settings, the policy Ta(a|s) 5) is trained exclusively on static dataset Dx, {(s r.
collected by behavioral policy Tb(a (). The training objective of offline RL (Wu al., 2019 regularizes the learned policy towards the policy TT and is reformulated as follows:
max E Day (-))(()))) 7i
(s, nDxL (To(|ss][==
(1)
where 1) is the temperature coefficient that balances exploitation and behavioral regularization.
3.2 Diffusion Probabilistic Models
Diffusion probabilistic models [Ho et al. 2020 generative methods that progressively corrupt data samples with noise until they resemble a standard Gaussian distribution, and train a model to reverse this transformation. The forward process gradually adds Gaussian noise (O, I) to data samples no over K steps, according to predefined variance schedule
,8k:
q 20)
II (()) q( xR-1)
NZss vI B4 BxI).
(2)
To recover the original data, the reverse process is trained to approximate the reverse transitions q( xx) by predicting the added noise using neural network ca. starting from JK (),
K
Po(xo:k) = p(xx)
II (o(=--112H) A=
po( = 12 k). Eg (x), k)),
(3)
where μe and Σg denote the mean and covariance of the predicted Gaussian distribution.
Building on diffusion probabilistic models, the offline decision-making task framed as a goal-
conditional generative modeling problem [He et al. 2023 Li et al. 2023 by maximizing:
E [log pa (Toly (Tol)(TO))). To
Sa 81
... 8T
ag
01
or
where is historical trajectory sampled from D.y, and y(ro) is the conditional information (e.g.. target cumulative reward or constraint satisfaction).
3.3 Structural Information Principles
In structural information principles [i and Pan 2016), the encoding tree T of an undirected graph (V, satisfies the following properties: I) Each node a in uniquely corresponds to a subset
V. C V. 2) The root node À satisfies VA = V. 3) For each leaf node v in T, V. is a singleton {v} for some E V. 4) For every non-leaf node with la children (1, 0r,, the subsets (Va, V form a partitioning of Va. i.e., Va U Va, and Va, n Va, @ for + j.
The K-dimensional structural entropy & defined over all encoding trees T of height by S K. as:
H* (G)
(G) = min HT))), HT) = hrSk
Σ
Jо vol(A)
log vol(a) vol(a
(5)
a RET,

---

hierarchy consiruction
conditional diffusion
regularized exploration
Irajoctory planning
a
hicrarchical policy sampling
diffusion model training
Lo
cntropy rogularization
CBg.
hu g
Rdf
our proposed
conditional information
mulit-layer diffusion
lorward nossing
framework
process
T
Rul
To
community partitioning
Ts
Irajectory segmentation
multi-scale hicrarchy
trajectories
Figure 2:
The proposed SIHD framework comprising the hierarchy construction, conditional diffusion,
and regularized exploration modules.
where a' denotes the parent of a non-root node a, vol(a) = Eveva du is the sum of degrees (volume) of the vertex subset Va, and Ga is the total weight of edges that cross the cut between Va and its complement V Va.
The SIHD Framework
In this section, we present the detailed design of the proposed SIHD framework, whose overall architecture is illustrated in Figure Superscripts h and i denote the diffusion layer and the temporal index within the same layer, respectively, while subscripts y and sa indicate subgoal sequences and state-action subtrajectories.
The hierarchy construction module models the topological structure of offline states based on feature similarity and identifies hicrarchical communities by optimizing structural entropy to construct an adaptive multi-scale diffusion hierarchy (see Section 4.1 The conditional diffusion module feeds temporally segmented trajectories into corresponding layers of a shared diffusion model, integrating quantized rewards and structural signals to perform forward diffusion and reverse inference at multiple time scales (see Section (.2).. The regularized exploration module employs structural entropy as measure of generative diversity and adds a regularization loss that encourages exploration of underrepresented offline states during both training and inference (see Section 4.3)
4.1 Multi-Scale Diffusion Hierarchy
Instead of the rigid two-layer diffusion hierarchy operating at single scale (Li et al. 2023 Chen
et al. 2024a], we leverage feature similarity to identify structural relationships among offline states and derive a tree-structured partitioning of state communities construct the trajectory-adaptive multi-scale hierarchical diffusion framework.
Starting from historical trajectories Dr, (Figure 3 a)),, we extract all state elements S to form the vertex set S of observed states and establish weighted edges based on feature similarity (e.g.. cosine similarity) between each vertex and its top-k nearest neighbors, thereby forming the k-nearest- neighbor state graph G.. We determine the parameter k by selecting the value that maximizes the upper bound of dynamic uncertainty, H (G.). promote structural expressiveness for encoding trees over G.. We then apply the HCSE optimization algorithm Pan et al.) [2025) to derive the height- K. optimal encoding tree T that minimizes the K-dimensional structural entropy H HG.)).) The optimization procedure is detailed in Appendix A.2, As discussed in Section 3.3 the encoding tree
T," represents tree-structured partitioning of Go into communities (Figure (3)))), where each node a corresponds to state community V., S S at particular level of granularity, and the parent-child relationships reflect the hierarchical inclusion structure among communities.
Based on the community partitioning defined by T.", we perform adaptive hierarchical segmentation for each trajectory To in Dr,, 3 Specifically, for each layer h, we obtain the set of nodes Ur at height h from T We then segment the trajectory according to the community assignment defined by Uh. ensuring each resulting segment hh, is a temporally continuous sequence of states from the same community. Each trajectory segment Ta is formally defined as follows:
h,4
$ E the +1
the
9 = =8 Spi th2t
(6)
0 a that
a
5

---

4y
b96606606000006
offline trajectories
b) community partitioning
0) hierarchical segmentation
d) multi-layer subgoals
Figure Construction the multi-scale hicrarchical diffusion framework by minimizing the structural entropy of offline trajectories, deriving tree-structured community partitioning, segmenting trajectories hierarchically, and extracting multi-layer subgoal sequences.
where i represents the temporal index of the i-th segment at layer h, and the final state in Taa is designated as the subgoal g." The variable th, denotes the sequence length (i.e., temporal receptive field) the segment Th.i The complete segmentation procedure is detailed in Appendix A.4
We integrate the K-layer subgoals, extracted from the segmentation hierarchy (Figure d() into the control-as-in is-ifeerece framework [Levine] 2018] to define a hierarchical, multi-scale diffusion framework. At each layer h, we define a subgoal sequence Th of length i where for each subgoal glt the corresponding subgoal sequence Th. C Th)
at the next lower layer is defined recursively as:
h-1 -i 22
1
if 1,
if h =
The temporal scale th. of subgoal of is not predefined but inferred from the hierarchical partitioning in T and the reference trajectory Tp- For the base case where / = 1, the lower-layer subgoal sequence دنبر corresponds to the state-action segment paa
Following the control-as-inference paradigm, we introduce a binary optimality variable O indicating whether the trajectory segment 7,--,,4 is optimal with respect to the subgoal - The posterior probability of O is modeled using Boltzmann distribution the cumulative reward:
p(O, 199; C-1 1-1 x exp
Σ R ( 45 " 1 15
(8)
The following theorem enables the decomposition of the conditional generation problem for offline trajectories into a hierarchical diffusion process with dynamic temporal scales across multiple layers. The detailed proof is provided in Appendix B.1
Theorem 4.1. Given the K-layer subgoals for each offline trajectory To, constructed as described above, the conditional probability in Equation can be factorized follows:
K-11 II
p(Tolu(TO)) & p(T (ol
Ju ((TT)))(TT
P(T
(9)
A=1s=
where y(T exp (Etn R(se, ui) captures the cumulative reward over the full trajectory To v(7)..) imposes subgoal satisfaction constraints via Dirac delta function, ensuring exact compliance, and 13 denotes the length of the subgoal sequence Th the h-th layer.
4.2 Conditional Diffusion Model
At the top of the hierarchy, the diffusion model generates a subgoal sequence conditioned on the overall task reward. Subsequent layers generate subgoal sequences at intermediate levels of the hierarchy or state-action sequences
base level, with each generation process conditioned on the parent subgoal. At each of these layers, we compute the structural information gain within the associated state communities, integrating it into classifier-fit free guidance to enhance conditional sequence generation.
For the h-th diffuser, denote each target sequence Th. (a subgoal sequence if 1, state- action sequence if h 1) as the input o the diffusion process. We then iteratively construct

---

sequence of progressively noised samples _h. via the forward diffusion process:
"9
a a (T /1 T BxI), 1 Isis
(10)
To reduce training overload, we adopt a shared diffusion model E0,, for layer h. This model operates with classifier-free guidance [Liu al. 2023] to jointly predict the noise term (O I) and
estimate the reverse-step posterior distribution pe;, (T (lut) at each diffusion step k:
(T po. g.k-1 1 N - ) 8= co,
(T g.kt (1 (-))((TT))) + wU, k). (11)
where ,ee, and Σg, denote the mean vector and covariance matrix of the estimated noise The symbol 0 represents the absence of conditional input and replaces y(Tg)))) during unconditional generation with a guidance weight
As specified Theorem 4.2 the conditional input to the highest-level diffusion model, y(7 g. ). is defined as the exponential the cumulative reward over the full trajectory Tg. At lower layers, we use structural information- based values instead of cumulative rewards. For each sequence T we identify the tree node a at height state-action segment Thi belong to the community Va. We then define the conditional input y ((T5 as the structural information gain of node o:
the encoding tree T." such that states in the corresponding
v(Tg..) = H (G.;a) = 9a
log vol(a) vol(a-))
(12)
vol(A)
This gain term quantifies the additional information required to infer that single-step random state transition in G. occurs within the lower-level segment vaa given prior knowledge, provided by the subgoal g that the transition contained within the higher-level segment r To ensure consistency with the hierarchical subgoal constraints defined in Theorem 4.2 we replace the terminal state of each denoised trajectory fg.0 with its corresponding subgoal g."
4.3 Structural Entropy Regularizer
To mitigate overreliance on the limited state coverage of offline datasets, we introduce structural entropy-based exploration regularizer that encourages the hierarchical diffusion policy to explore un- derrepresented regions of the state space while limiting extrapolation errors resulting from deviations from the offline behavioral policy.
For each trajectory Tu. we extract the state transition (St, 8t+1) each timestep t and estimate the transition probability pe, (St41, 9t) using the base-layer diffusion model es, These estimated transition probabilities are used in place of feature similarity metrics to construct a new topology over the state set S, resulting in a complete weighted state graph Gz where each edge weight reflects diffusion-inferred transition likelihood. In 9% the weighted degree of each state S is defined as its diffusion- feffusiionbbaeed visitation probability Pe, (s), computed by summing the incoming transition probabilities. We define the structural entropy of the graph G/ under the encoding tree T as:
En,ey Ex,ev, po (8),8))
-log EeEV. Po, po, (8) E.EV, Po, (s)
Σ
(13)
Ever Po.
OET, YK
The following theorem provides variational lower bound on HT (g)) and establishes its theoretical connection to the Shannon entropy the state distribution Pa, over S. A detailed proof is provided in Appendix B.2
Theorem 4.2. For the encoding tree T² and the complete weighted state graph G'a the structural
eniropy HT² (G') admits the variational lower bound related to the Shannon entropy H(S):
H(S) Σ (m H((W) & HT: (G.) S H(())
(14)
K=1
where each layer-specific weight ny is defined as the maximum of height h in T. for the state graph G'
Ele Bo, vol(a)
8o, -a over all nodes a at
7

---

Algorithm The SIHD Planning Algorithm
1: Input: hierarchical diffusion probabilistic models {so, 1 k=1 the initial state so S, the planning
horizon H, the maximal cumulative reward Tmaa in nm.
2: Output: the action sequence {az)r=o H-1
3: Initialize the hierarchical subgoal sequence "g +- [80] for 2 2 K.
4: Initialize the state-action sequence Tp) [0, [so] and [1,:] [0] at the 1-th layer
5: for 0 H do
6: Sampling the starting noised sequence ig..t N((,
7:
li 6 -gg
8:
if T [-1] is satisfied then
9:
+2.1 (2, fco, Mi (rph.1)e 1h=1, mmaz)
10:
end
if
for
+ K do a + select the 1-layer node according to [-1]
12:
Calculate the conditional information (( via Equation 12
14:
E E (mo, Eo,) extract the mean vector and covariance matrix from é
EB, (Folle (1 (G,)) + w0, k)
15:
-1.1
end for
18:
[0::] t Tg [0, + ا1111
9.0 [0, 12.1 [1,:] t Tg. [1, 1]
1. 1.00
20:
end for
21:
Return the sequence T 1 [1, :] as the action sequence {a, [==)==
To promote more balanced coverage of the state space, maximize the Shannon entropy H(S) over the state distribution, thereby improving exploration of underrepresented states and reducing dependency historical trajectories. Conversely, preserve the ppcision-making hierarchy encoded in 7,, we minimize the structural entropy H(24;) (structural entropy of the community partition at each layer h), with appropriate weighting 7h- This regularization discourages large deviations from the behavioral policy Ta and mitigates extrapolation errors.
The regularized training objective for each diffusion model e0,, is defined as:
K-11 (r (1 w))(T(()) + w0, R)) - cl12 nT== H(S) Σ In H(U)))
[(0%) = E Σ
(T
(15)
where M an indicator function equal to if h = 1, and r is a weighting coefficient for the entropy regularization term. The training procedure of the SIHD framework summarized in Appendix A.5
4.4 The SIHD Planning
During inference, our SIHD planner proceeds as follows: We first initialize each hierarchy's subgoal buffers and the one-layer state-action sequence (lines 3 and in Algorithm IT. For each planning step, we (i) sample an initial noisy latent sequence from the diffuser prior (lines and in Algorithm D} (ii) invoke the subgoal proposer to revise the top-level goal if its terminal criterion is satisfied (lines 8-10 in Algorithm I and (iii) execute a reverse-diffusiee
rollout across latent timesteps, where at each diffuser step sample the next latent sequence (lines 11-17 in Algorithml I.. Finally, decode and update the one-layer state vate-action sequence (lines 18 and 19 in Algorithm iterations, we output the action trajectory (line 21 in Algorithm 1).
T. and after H
5 Experiment
In this section, we conduct comparative experiments on the D4RL benchmark (Fu al. 2020). cover- ing standardized offline and hong-horizon planning tasks, to evaluate decision-making performance

---

Table 1: Performance comparison between SIHD and baseline methods the Medium-Expers Medium, and Medium-Replaa datasets from the D4RL Gym-MuloCo benchmark tasks: "average
reward t standard deviation". Bold: the best performance, underline: the second performance.
Sym-MulaCts
HallCheetah
Happer
Walker2D
Tisks
Expert
Mecium
Replay
Experi
Medium
Replay
Expert
Medium
Roplay
CQL
91.6
11.0
45.5
106.1
58.5
95.0
105.8
72.5
77.2
IQL.
46.7
47.4
44.2
91.5
66.3
94.7
109.6
78.3
73.9
DT
36.6
107.6
67.6
82.7
108.1
MoReL
53.3
42.1
40.2
118.77
95.4
983.6
95.6
77.8
49.8
TT
95.0
46.9
41.9
110.0
61.1
91.5
101.9
79.0
Diffuser
0.3
42.8 0.3
0..
1033
1.3
74.3+1.
0..6:0044
106 = 0.2
70.6 +:
70.6 1.6
HDMI
92.1 1.4
28.0
113.5 219
7.6
99.6 2 1.5
107.9
= 1.2
79.9 2
81.7 2.1
HD
92.6 0.3
16.7
38.1
0.7
115.3
99.3+003
94.7 0.7
107.
81.0
81.1 2:2
STHD
94.4 + 0.5
48.7
+
1.1
47.0
+ 0.4
117.7 + 1.4
103.1 + 0.8
101.5 + 1.3
110.3 + 0.9
88.3 + 0.9
89.7 + 1.9
Av).
1992.1
0.7(1.5)
1.3333)
2.4211)
3.8(388
19(19)
0.7(0.6
4.316.1)
5.0(6.7)
Table 2: Performance comparison between SIHD and baseline methods on the U-Maze, Medium, and Large datasets from the Maze2D and AntMaze tasks: "average reward standard deviation", Bold: the best performance, underline: the second performance.
Mymoo
Single-ust ManeDD
Malt-task MazelD
AmiMase Medium
Tasks
U.Maze
Medium
Large
U.Maze
Medium
Large
U.Maze
Lange 17.5
IQL
47.4
3133
58.6
218
12.1
13.9
02.2
70:01
5.1 123.0 1: 6.4
41.2
Diffeser
113.9
121.
128.9 + L8
127.2 & 3.4
182.1 d: 5.8
76.0 7.6
31. 5.1
89.1=24
26
13.7
HiGaC
1.9 91.2
79.3
2.5
67331.
HDMI
120.1+225
121.8 1.6
128.6 + 2.9
131.3 + 1.8
131.6 1.9
135.4 4..4+22.
128. 3.6
3.0
2.5
140.2
165.5:0000
91.0
88
83.6
SIHD
+.9
148.5 2.6
161.7 1-1.7 3-4
157.0 = 0.6
156.8 + 1.7
169.4
96.5 2.8
92.2 5.0
89.4 - 4.2
Ab6. Avg.
18.2(12.6)
12.89.3)
5.91388
12.9:300
168((18))
3.9(2.4)
2.3(2.1)
3.M339
5.8(6.)
and generalization capabilities of the SIHD framework. The source code is publicly available via an anonymized link for peer review" All experiments are run with five random seeds. Additional analyses, including hyperparameter sensitivity and visualization of model behavior, are provided in Appendix
5.1 Offline Reinforcement Learning
We begin by evaluating the SIHD framework, configured with K. = 3 diffusion layers, on standardized Gym-MuJoCo tasks, characterized by dense rewards and short horizons. This setup allows to assess SIHD's offline decision- decision-making capabilities across datasets of varying quality. For comparison, we consider both non- non-bierarchical and hierarchical of-te-of-thh-art baselines. These include model-free methods such as CQL [umarr et al., 2020], IQL [Kostrikov et al. 2022]. and Decision Transformer
(DT) Chen et al. 2023 model-based methods such as MoRel. (Kidambi al. 2020 and Trajectory
Transformer (TT) Janner et al. 2021 and diffusion- based methods such as Diffuser [Janner et al.)
2022. HDMI Li et al. 2023), and Hierarchical Diffuser (HD) [Chen et al. 2024a
As shown in Table [ SIHD consistently achieves the highest average reward on offline datasets of varying quality for each task, demonstrating strong decision-man maaiinnnn performance and generalization ability. Although hierarchical diffusion baselines such as HDMI and HD perform competitively on most datasets, they show degraded performance on others (e.g., Medium Medium-Replay HalfCheetah and KodiummExpeers Walker2d), likely due to their simplistic two-layer hierarchies and fixed single temporal scales. These results underscore the importance of incorporating an additional diffusion layer and adaptively inferred temporal scales from offline trajectories to enhance hierarchical diffusion performance. On the high-quality Medium-Exx Expertt dataset, SIHD achieves an average performance improvement of 1.6% over the baselines. On the Medium and Medium- Replay datasets, which reflect average data quality, the performance advantage of SIHD is more pronounced, achieving performance improvements of 3.8% and 3.9%, respectively. These improvements demonstrate that SIHD alleviates dependence on high-quality offline datasets by introducing structural entropy-regularized exploration, thereby enabling more effective offline policy learning.
5.2 Long-Horizzo Planning
We next evaluate the long- decision-making performance of SIHD, configured with an increased number of diffusion layers (K. 4), on two sparse-reward navigation tasks: Maze2D

---

STIP
SIED DEI
STID UEB--o
STID-BR
Hopper 116
Walker2D
MinreDD
AmtMaze
110
95-
Madkin
Mediion
Mediun
1-Maxe
Malkin
Figure 4: Ablation study on hierarchical construction, conditional diffusion, and regularized explo- ration within the SIHD framework, evaluated in the D4RL benchmark.
and AntMaze. In these environments, the agent receives positive reward only upon successfully reaching a target location within an episode. The high-dimensional and noisy observations these environments further exacerbate the decision-making challenges. In addition to the baselines used for Gym-MuJoCo, we include leading methods-MMPI (Williams et al. 2016), IRIS [Mandlekar et al., 2020). and HiGoC (Li et al.] 2022 as
as additional baselines for Maze2D and AntMaze. Following prior studies (Janner et al. 2022 Chen et al.] [2024]] we introduce a multi-task variant of Maze2D in which the target location is randomized in each episode.
As shown in Table 2 SIHD achieves superior decision-making performance across all datasets, consistently attaining the highest task rewards among compared methods. Specifically, SIHD achieves average improvements in reward of 8.3%, 7.4%, and 4.4% in the single-task Maze2D, multi-task Maze2D, and AntMaze tasks, respectively. Compared to the results on Gym-MuloCo tasks, SIHD demonstrates more pronounced performance advantage in navigation tasks, attributable to its deeper diffusion hierarchy, which better supports the long-term decision-makinn of these environments. Furthermore, SIHD demonstrates more stable performance across varying data qualities. For example, while the second-best performing HD baseline suffers maximum performance drops of 27.4, 25.3, and 10.4 in the three task settings, SIHD limits these drops to just 17.1, 12.6, and 7.1, respectively. This robustness stems from the structural entropy regularizer, which encourages exploration of underrepresented states, reduces reliance on historical trajectories, and thereby enhances generalization ability.
demands
5.3 Ablation Study
To assess the contributions of key SIHD modules, we perform ablation studies on two Gym-MuloCo
tasks (Hopper and Walker2d) and two single-task navigation environments (Maze2D and AntMaze). Three model variants are constructed by selectively disabling core components: (1) replacing the multi-scale hierarchy (see Section 4.1) with a rigid two-layer single-scale diffuser (SIHD-DHH (2) removing the classifier-basel guidance (see Section 4.2) (SIHD-CG); and (3) removing the structural entropy regularizer (see Section 4.3) (SIHD-ER). As shown in Figure 4 the full SIHD framework consistently outperforms all three ablated variants, highlighting the importance of each component. Notably, SIHD-DH shows a larger performance drop than SIHD-CG and SIHD-ER, underscoring the critical role of the multi-scale diffusion hierarchy. This effect becomes more pronounced in long-hond horizon decision-man makiing tasks. Additional experiments on sensitivity analysis, computational efficiency, extended ablation studies, and qualitative comparison are provided in Appendix
Conclusion
This work proposes SIHD, a novel hicrarchical diffuuion framework that leverages structural infor- mation embedded in historical trajectories construct an adaptive multi-scale diffusion hierarchy and promote exploration of underrepresented states in offline datasets, thereby enabling effective policy learning. Extensive evaluations on the challenging D4RL benchmark demonstrate the superior -ocision-making performance and generalization capabilities of SIHD across diverse offline RL tasks. In future work, we aim to refine the hierarchical diffusion framework by further exploring how to more effectively represent and integrate subgoal constraints as conditional information. We also plan to extend the SIHD framework to other offline RI. environments and broader diffusion- based generative modeling domains.
10

---

Acknowledgments
The corresponding author is Yicheng Pan. This work is partly supported by National Key R&D Program of China through grant 202IYFB3500700, NSFC through grants 62322202, 62441612 and 62432006, Local Science and Technology Development Fund Hebei Province Guided by the Central Government of China through grants 24620102G and 254Z9902G, the Pioneer and Leading Goose R&D Program of Zhejiang through grant 2025C02044, National Key Laboratory under grant 241-HF-D07-01, Beijing Natural Science Foundation through grant L253021, Hebei Natural Science Foundation through grant F20242 and Major Science, Technology Special Projects of Yunnan Province through grants 202502AD080012 and 202502 CCF-DiDi GAIA Collaborative Research Fund through grant 202527, and partly supported by State Key Laboratory Complex Critical Software Environment through grant CCSE-2024ZX-20.
References
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2021.
Anurag Ajay. Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision maniing? NeurlPS 2022 Foundation Models for Decision Making Workshop. 2022.
Richard Bellman. A markovian decision process. Journal of Mathematics and Mechanics, pages 679-684, 1957.
Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. In The Twelfth International Conference Learning Representations, 2024a.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learn- ing via high-fidelity generative behavior modeling. In International Conference Learning Representations, 2023.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In International Conference on Learning Representations, 2024b.
Leandro M De Lima and Renato A Krohling. Discovering an aid policy to minimize student evasion using offline reinforcement learning. In 202/ International Joint Conference on Neural Networks
(IJCNN), pages 1-8. IEEE, 2021.
Joshua B Evans and Özgür Şimşek. Creating multi-level skill hierarchies in reinforcement learning. Advances in Neural Information Processing Systems, 36:48472-48484, 2023.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances in neural information processing systems. 36:64896 (114899664917,, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-66 6851, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902-9915. PMLR, 2022.
11

---

Rahul Kidambi, Aravind Rajeswaran, Pranecth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference Learning Representations, 2022.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.
Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 15:1179-1199 2020.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: ffute-of-the-are pages 45-73. Springer, 2012.
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review. arXiv preprint arXiv: 00999, 2018.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164 016443, 2020.
Angsheng Li. Science of Artificial Intelligence: Mathematical Principles of Intelligence (In Chinese). Science Press, Beijing, 2024.
Angsheng Li and Yicheng Pan. Structural information and dynamical complexity of networks. IEEE Transactions on Information Theory, 62:3290-3339, 2016
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Hierarchical planning through goal- conditioned offline reinforcement learning. IEEE Robotics and Automation Letters, 7(4): 102)6- 10223, 2022.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. International Conference on Machine Learning, pages (0035-20064. PMLR, 2023.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine PMLR, 2023.
Learning, pages
Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell. More control for free! image synthesis with semantic diffusion guidance. In Proceedings of the IEFF/CVF winter conference on applications of computer vision, pages 289-299, 2023.
Jason Yecheng Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. Offline goal-conditioned reinforcement learning via f-advantage regression. Advances in Neural Information Processing Systems, 35:310-323, 2022.
Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 4414- IEEE, 2020.
Yicheng Pan, Bingchen Fan, Pengyu Long, and Feng Zheng. An information-thherrticc perspective of hierarchical clustering on graphs. In UAI, volume 286 of Proceedings of Machine Learning Research, pages 3322-3345. PMLR, 2025.
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Shenghao Zhou, Dinesh Jayaraman, Chelsea Finn, and Sergey Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. Advances in Neural Information Processing Systems, 33:17321 1773321113333 2020.
12

---

Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, al. Learning transferable motor skills with hierarchical latent mixture policies. In International Conference on Learning Representations, 2022.
Tongzheng Ren, Jialian Li, Bo Dai, Simon S Du, and Sujay Sanghavi. Nearly horizon-free offline reinforcement learning. Advances in neural information processing systems. 34:15621-15634, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asiisedd interventionMMICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
Earl D Sacerdoti. Planning in hierarchy of abstraction spaces. Artificial intelligence, 5(2):115-135, 1974.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3: 9-44, 1988.
Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi- Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. Advances in neural information processing systems,
2022.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02644. 2018.
Adam R Villaflor, Zhe Huang, Swapnil Pande, John M Dolan, and Jeff Schneider. Addressing optimism bias in sequence modeling for reinforcement learning. In international conference on machine learning, pages
PMLR, 2022.
Valentin Villecroze, Harry Braviner, Panteha Naderian, Chris Maddison, and Gabriel Loaiza-Garn Ganem. Bayesian nonparametries for offline skill discovery. In International Conference on Machine Learning, pages 22284-222222 22299. PMLR, 2022.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In Deep Reinforcement Learning Workshop NeurIPS 2022, 2022.
Grady Williams, Paul Drews, Brian Goldfain, James Rehg. and Evangelos A Theodorou. Aggres- sive driving with model predictive path integral control. In 20/6 IEEE international conference on robotics and automation (ICRA), pages 1433-1440. IEEE, 2016.
Junran Wu, Xueyuan Chen, Ke Xu, and Shangzhe Li. Structural entropy guided graph hierarchical pooling. In ICML, pages 24017-24030. PMLR, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1rxiv:/911 I.] 1361, 2019.
Xianghua Zeng, Hao Peng, and Angsheng Li. Effective and stable role-based multi-agent collab- oration by structural information principles. Proceedings of the AAAI Conference Artificial Intelligence, (1nr)eeeee111110): (172:1177--11800, Jun. 2023a.
Xianghua Zeng, Hao Peng, Angsheng Li, Chunyang Liu, Lifang He, and Philip S Yu. Hierarchical state abstraction based on structural information principles. IJCAI, pages 4549 2023b.
Xianghua Zeng, Hao Peng, and Angsheng Li. Adversarial socialbots modeling based on structural information principles. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 392-400, 2024.
13

---

A Framework Details
A.1 Primary Notations
Table 3: Glossary of Notations.
Notation
Description
M
Markov Decision Process
S:A
State space; Action space
P;R
Transition function; Reward function
8,0,r
Single state, action, and reward
Q
Value network; Policy Network
T; D
Single trajectory; Offline dataset
N;I
Gaussian distribution; Diagonal matrix
Gaussian noise; Variance schedule
4E
Mean vector; Covariance matrix
P.P
Prior distribution; Posterior distribution
Conditional information: Single sugboal
9:7
Topology graph; Encoding tree
V.E
Vertex set; Edge
HHHH;;
Structural entropy; Shannon entropy
A
Root node; Leaf node
au
Single node; Node set at specific layer
A.2 Tree Optimization.
The HCSE algorithm is executed iteratively through stretch and compress operations on a-triangle subtrees, which are rooted at node whose children leaf nodes, illustrated in Figure 5 a). In the stretch stage, the a-triangle subtree is transformed into a binary subtree (Figure (J)) by inserting intermediate nodes that pair the children of a, ensuring that the binary branching constraint is satisfied. In the compress stage, the resulting binary subtree is compressed into subtree with exactly three layers of nodes (figureB(cc), thereby increasing the height of the subtree by one level.
stretck
compress
a) a-triangle subtree
b) binary subtree
compressed subtree
Figure 5: The stretch and compress optimization on each o-triangle subtree.
Algorithm 2 Jummmrizess the optimization procedure applied to each graph G. It begins by initializing
a one-layer encoding tree T for G, and then iteratively selects the set of h-layer nodes U4, that yield the largest structural entropy reduction AHT (G; This reduction is achieved by applying stretch and compress operations each a-triangle subtree, where a Un. The process continues until the desired tree height K. is reached.
Algorithm 2 The HCSE Optimization
Input: graph G, K +
2: Output: the optimal encoding tree T
3: Initialize the one-layer encoding tree T for G
4: while hT & K. do
5: Select the Ud that maximizes AHT (9;4n)
6: if AHT (G: = 0 then
7: Break
8: end if
9: for a E U4, do
10: Execute stretch and compress optimization on the a-triangle subtree
11: end for
12:
end while
13: Return the resulting encoding tree TT as T
14

---

A.3 Probability Calculation
To compute the joint probability pø, (st, S(41) for any adjacent pair of states, we proceed follows.
111 N(,0T), We then apply the We begin by sampling initial Gaussian noise trajectories "s.K reverse process described in Equation to iteratively denoise these trajectories, yielding n denoised trajectories 7 "=0" From each denoised trajectory, we extract the states at time steps and t + 1, constructing sample set (sf: + Finally, we apply two-dimensional Gaussian kernel density estimator approximate the joint distribution po, (St, 8t+1).
A.4 Trajectory Segmentation
Algorithm 3 Hierarchical Trajectory Segmentation
1: Input: the XC-layer encoding tree T each historical trajectory T0
2: Output: hierarchical trajectory segments {T ( M1=1 with s i
3: for h < IC to 1 do
4: Initialize the first segment T <- [[ [ai]] and the index variable i <-
5: Extract the node set Un the h-th layer
6: for < 2 to do
for 24, do
8:
if' St-1 E Va and E Va then
9:
Tg 00, : f su [0, + [t]]
10:
To [1
[1, + [a
11:
Break
else if 8t_
E Vo or St Va then
13:
4 i
Tea Break
+ [[s [ar]
15:
end if
17: end for
end for
19:
end for
Return the hierarchical segments (Thii)
hh=1
A.5 Model Training
Algorithm 4 The SIHD Training Algorithm
1: Input: the offline dataset Dr.,, the maximal tree height I.
2: Output: hierarchical diffusion probabilistic models {<o,, 1/
h=1
Construct the topology graph G, for offline states in DTTa
4: Derive the optimal encoding tree T,* via Algorithm
5: for each trajectory To Dr,, do
6: (Th,) Jh=1 hierarchically segment To via Algorithm
7: for <- K. to 1 do
8:
(Th)] extract the subgoal sequence from (T Ji=1
9:
for i t to 1/ do
10:
1 h,i t extract the lower-layer subgoal sequence for g +
11:
y(Th.)) + calculate the conditional information of nh,a 2h
12:
9.0 <- estimate the denoised sequence based on
13:
Optimize the h-th diffuser ea, by minimizing the training loss C(O)) in Equation 15
end for
15: end for
end for
15

---

Algorithm 5 Subgoal Updating Function Seu
1: Input: the layer parameter h, hierarchical diffusion probabilistic models (ee, D === hierarchical subgoal sequences {T) {= Jh= the maximal cumulative reward Tman in D
2: Output: the updated subgoal sequences "9
3: if h K. then
4: Sampling the starting noised sequence P.C.1 (O(
9
5: Update the noised sequence #,k..1 15.11 t K.
6: for + KKKu do
(F (fgg (1 mar + w0, k) ē+ E6x
7:
8:
(NOE, Mixe) f extract the mean vector and covariance matrix from é *x.11
9:
k ~N(wve Eox)
Return the subgoal sequence x. xx + 15 [1
10:
11:
end for
12:
else
13:
if [+++1.11-11 is satisfied then
14:
Tg 1,4111 + (h + 1, (com Ph (rgill } Tmax)
end
16:
Sampling the starting noised sequence +h..1 (0,
Update the noised sequence +h (h 4 T for + K to 1 do
18:
select the h-layer node according Ty [+1
20:
Calculate the conditional information ((*g,)) via Equation 12
h.1
21:
+ EB,, (Fghtt (1 - ())(F()))) + w0, k)
22:
(no, Eon) <- extract the mean vector and covariance matrix from è a. No Do,
sh.1
23:
24:
end for
25:
Update the subgoal sequence 4.1 <- Thi + 1g [b
26:
end if
16

---

A.6 Toy Example
To intuitively illustrate the hierarchical partitioning process, we present toy example which SIHD partitions a trajectory of 1000 timesteps into sub-trat tajeeeajesssies at different hierarchy levels h. In each sub-trajectory, the final state is treated that segment's subgoal.
Level h = K. (top, coarsest): (1, 2, 1000}-
Level h K. 1: ( 90). (91, 92, 200). (901,902, 1000).
Level h 2 (fine): 21, 30), (31, 32. ..... 47....... (968, 969, ................................ 1000}.
Level h = 1 (finest): {1, 2........ 8), (9, ...0............. 15).
(995, 996,
1000}.
A.7
Implementation Details
Our SIHD framework is built upon the officially released Diffuser codebase leveraging a hierarchical architecture which each diffuser processes trajectory segments corresponding state communities from the optimal encoding tree. To ensure consistent training across variable-length sequences, we pad subgoal and state-action segments to fixed sequence lengths for Gym-MuJoCo tasks and 16 for long-horizon navigation tasks by repeating terminal states. The diffusion backbone employs a Temporal U-Net [Ronneberger et al.) 2013 Ajay et al. 2022] with a Gaussian diffusion process, configured with multiscale temporal convolutions of 32 dimensions. Optimization is performed using the Adam optimizer with exponential moving average (EMA) decay model weights for stable updates. During training, diffusion models trained with a batch size of 32, while planning phases use a larger batch size of 644 to improve sample diversity during guided rollouts.
Horizon settings follow established benchmarks (Janner et al. 2022 H 32 for Gym-MuloCo; H E (120, 255, 390} for Maze2D (scaling with task complexity); and H E {225, 255, 450) for AntMaze environments. Classifier-free guidance is applied uniformly across tasks with a fixed weight of 0.1. The regularization coefficient 7j is selected from 0.01, 0.05, 0.1, 0.15.. 0.2 based on empirical analysis (see Appendix C This configuration ensures adaptability across both short-horizon locomotion and long-horizon navigation tasks while maintaining computational efficiency and stable performance.
A.8 Limitations
This work proposes the first multi-layer diffusion framework with adaptive temporal scales, lever- aging structural information embedded in historical trajectories. This design addresses the strict structure and overreliance on offline datasets observed existing hierarchical diffusion methods. On large-scale offline datasets, structural entropy-guidee state community partitioning introduces significant computational overhead. for each dataset and store the layer-wise community partitions avoiding redundant computation and training-timg -ime overhead. Subgoal constraints in the diffusion hierarchy are implemented using straightforward ending-state replacement strategy. We leave more comprehensive exploration of subgoal conditioning strategies to future work. Another important future direction is extending SIHD to more complex offline RL tasks and broader diffusion- based generative modeling domains.
mitigate this, we precompute the optimal encoding tree
dictionary- data structure,
17

---

B Theorem Proofs
B.1 Proof of Theorem 4.1
Proof. We begin by recalling the offline RL objective of modeling the posterior distribution over full trajectories To, conditioned on the optimality 1-55- L, captured via y(To):
p(roly(To)) = p(To O 1.--- 1) x p(To) p(O 145- 125-
to).
(16)
Using the decomposition of To into subgoal-conditionel segments layer K. 1, we write:
1
(TT) p(n)) II plg, -1
(17)
where each sub-trajectory =u Leveraging the posterior likelihood of optimal variable O from Equation 8, we have:
K,1 denotes the segment of To aimed at achieving the subgoal g. K-1
pk-1
p(Tolu(TO)) p(s))
II
p(g:
exp
Σ
R(s
ch 4k O is 25 xx+))
p(Toa x1T--11
96 1)
pK-1
Σ (1)
pp(0))
II
pigk-11)
p( 19,--1).
exp
(18)
We define the cumulative reward term for the top-level hierarchy as:
T
u(Tg.1)) = exp
Σ
R(s,, a.)
(19)
Thus:
(K-1
p(Toly(TO)) x p(T y(T, (.))
II p( K-1 1
(20)
Each term p(
1, o -1) is further recursively decomposed into lower-level subgoal trajectories:
K-
K- g 1 p
l
II
p(Tsa
p(
"> 10
K-2
1
(21)
Here, -[( enforces the consistency of the lower-level subgoals with the higher-level goal 9, k-1 using a Dirac delta function:
{
+00
if
X-2
= 9 K-1
y( u(T K-1,
x- 1
K-2 1-
(22)
otherwise.
Thus we can write:
p(T
9
KC
(23)
Plugging this into our earlier expansion:
p(TOlu(TO)) (.1) v(1 (.1)
II
II
p( K(TK-22
2-2,
g
x-2)..
(24)
By recursively applying this decomposition across all levels h H down 1, we obtain:
K-1 1
p(TOle(TO)) x P 1
pl (T(T
(25)
18

---

B.2 Proof of Theorem 4.2
Proof. Let a be an intermediate node (i.e., non-root and non-leaf) the encoding tree T and let {os)==1 denote its children. The structural entropy contribution of a and its children given by:
lo (();)) + Σ HTi (G);a))
Ja
vol(a)
ta
Jas
vol(a,)
log
log
vol(A)
vol(a-)
vol(A)
vol(a)
9a
-'- Jcu vol(A)
log vol(a)
ta
9o. log vol(a;) vol(A) vol(ar)
vol(a-)
(26)
The total structural entropy of G¹ under T can be expressed as: K-1
HT. = E
HT (G)a)) = Σ
Σ HT (G)
TET, aaT
aEl4
K-3
Σ
(g' + Σ (((:a)))))
+
Σ h=0 aEll
H (g:;
aEliic..
Σ NEllie
Ja
i= Ia log
vol(a) vol(A)
+
9o. log vol(as) vol(A)
+ Σ Σ HT
K--
(A)
vol(1)
1=0 oell,
K-1
Ja
E/= Jac
vol(a)
Ja
log vol(a)
log
+
vol(A)
vol(A)
vol(A)
vol(A)
aell
aEM
Σ po, (8) log (8)
K-1 Σ
Ja
D/e si-l
Jos
log vol(a)
vol(A)
vol(A)
aEll
H(S)
K-1
Σ
Ja
>/m Ja.
log
vol(a) vol(A)
vol(A)
1=1
asli
(27)
Since for each intermediate node a, the condition 9a -!- Ir, S 0 holds, the upper bound follows: HT (G.) - H(S).
(28)
The structural entropy can further be rewritten in terms contributions from each layer:
HTT H(S)
Ja
Eln Ia. log vol(1)
vol(a) vol(A)
h=1 aeul
E-1
E/e Ia:
9a vol(a)
aol
= =((S) +
Σ
log
vol(a)
vol(A) vol(A)
h=1
aELd
Ei=1 Ia: 9a vol(a)
vol(A) log vol(a)
2 ((S)) +
max oEldi
(29)
vol(a)
vol(A)
aeu,
H(S)
M
max aEul
Li=1 Sa. Ja vol(ar)
H(lth)
H(S)
[m H(U4)).
Therefore, the following lower bound holds:
K-1
Th (g.) 2 H(S)
Σ h=1
Im H(Um))
(30)
19

---

C Additional Experiments
C.1 Sensitivity Analysis
To investigate the effect of key parameters, such as the regularization weight vj and maximum tree height K. on SIHD performance, we conduct sensitivity analysis experiments on the HalfCheetah and AntMaze tasks. As shown in Figure [6 the decision-making performance of SIHD initially improves as parameter values increase, but plateaus beyond certain threshold and may decline with further increments. In the HalfCheetah task, smaller values of 17 for the Expert dataset yield better average rewards, as the offline trajectories are generated by a well-trained behavioral policy, requiring minimal additional regularization. Conversely, for the Medium dataset, larger values of lead to better performance due to the lower quality of the behavioral policy. In the AntMaze task, longer planning horizons increase the decision-making challenge, and using more diffusion hierarchy layers results in higher average rewards. However, excessively many diffusion layers shorten the length of individual subgoal sequences, degrading the quality of sequence modeling. Based on these findings, we selected K 4 as balanced and effective parameter setting.
HaiiChhetah Expert
HallCheetah Medium
AntMaaze Medium
AmtMaae Large
60.0-
48.5
50-
47.5
90
3.01
5.05
0.1
3.15
0.2
0.01
0.05
2.5
3.0
2.0
3.8
4.0
50
6.5
Regukriimtion Weight
Repulamatiin Weiahs
Thes Heggh
Maxinal Heggh:
Figure 6: Sensitivity analysis of the regularization weight 17 and maximum tree height K on the Expert and Medium datasets of the HalfCheetah task, and the Medium and Large datasets of the
AntMaze task.
C.2 Computational Efficiency
To evaluate computational efficiency and practicality, we compare the training and planning time of SIHD with baseline methods, including Diffuser and HD, across different Maze2D datasets. To ensure a fair comparison, we use the publicly available source code and hyperparameter configurations, and conduct all experiments on an Nvidia A800 GPU. The experimental results are reported in Table Despite the addition of additional diffusion hierarchy layers, SIHD maintains training and planning efficiency comparable to the 2-layer hierarchical diffusion (HD) baseline. Compared with the flat Diffuser baseline, SIHD achieves 82.5% and 54.8% reductions in training and planning time, respectively, while preserving the computational efficiency benefits of hierarchical diffusion.
Table Comparison of wall-clock time between STHD and baseline methods (Diffuser and HD) on the U-Maze, Medium, and Large datasets of the Maze2D task.
Training (8)
Planning (S)
Methods
U-Maze
Medium
Large
O-Maze
Medium
Large
Diffuser
11.7
48.3
42.2
0.8
4.7
4.9
HD
5.4
5.8
6.1
0.3
1.9
2.5
STHD
5.9
6.0
6.0
0.5
1.6
2.6
To further examine the effect of planning horizon on computational cost, we conduct additional exper- iments the Medium-Expert Hopper task with horizon settings of 32, 48, and 64, summarized in T While SIHD incurs a slightly higher computational overhead than the HD baseline in most cases, this additional cost remains modest and does not scale with longer horizons, demonstrating the practicality and scalability of the approach.
Table 5: Comparison of wall-clock time between SIHD and baseline methods (Diffuser and HD) on the Expert, Medium, and Replay datasets of the Hopper task under varying planning horizons.
Training
Planning (s)
Methods
H
32
H
48
H
=4
64
==
32
H
64
Difluser
6.1
11.2
0.8
1.5
HD
4.4
5.2
5.9
0.5
0.7
0.8
SIHD
4.6
5.5
6.0
0.6
0.7
0.9
20

---

C.3 Ablation Study
Indeed, additional diffusion layers introduce more capacity or parameters, which may influence the performance advantages of SIHD. To address concems about model capacity, we perform an additional ablation study on the single-task Maze2D benchmark, in which we reduce the parameter count of each diffusion layer so that the total number of parameters across all layers matches that of the two-layer HD model. As shown in Table even without increasing overall model capacity, increasing the number of hierarchical layers yields consistent, significant improvements in performance. This demonstrates that the benefit of SIHD arises from its hierarchical structure rather than simply from an increase in parameter count.
Table 6: Performance comparison between SIHD and baseline methods (HDMI and HD) with similar parameter counts on the Maze2D-U, Maze2D-Meddum, and Maze2D-LLagg datasets.
Methods
U-Maze
Medium
Large
HDMI
120.1 = 2.5
121.8 = 1.6
128.6 6 2.9
HD
128.4 = 3.6
135.6 = 3.0
155.8 2.5
SIHD
140.7 = 2.1 142.5 = 2.9
157.3 + 2.0
To assess the impact of community quality on performance, we introduce an ablation variant, SIHD- FT, which replaces structural entropy-based partitioning with fixed-interval temporal partitioning strategy. As shown in Table 7 SIHD-FT yields clear drop in average return compared to the full SIHD model. This demonstrates that minimizing structural entropy improves partitioning accuracy and plays a critical role in identifying key offline states and guiding the construction of effective hierarchical diffusion hierarchies.
Table 7: Performance comparison between SIHD and the ablation variant, SIHD-FT on the Maze2D- U, Maze21D-Medium, and Maze2D-LLarg datasets.
Methods
U-Maze
Medium
Large
STHD-FT
149.17 1.7
146.8
2.1
167.2 4.8
SIHD
157.0 0.6
156.8 # 1.7
169.4
21

---

C.4 Qualitative Comparison
First, we qualitatively analyze the subgoal sequences sampled by SIHD and the HDMI baseline in Maze2D navigation task, which requires navigating from green starting point to red goal point. For visualization purposes, we display the lowest-level subgoals extracted by SIHD, As shown in Figure 7 the subgoals generated by the single time-scale diffusion hierarchy in HDMI are largely fixed and may not correspond to task-relevant states. In contrast, the multi-scale trajectory segmentation and adaptive subgoal extraction in SIHD effectively identify key states critical for task completion, such as turning points along the navigation path, leading superior decision-makee performance.
Figure 7: Visualization of sampled subgoals comparing SIHD and the HDMI baseline in the Maze2I> navigation task from the starting state (green color) the goal state (red color).
Second, we visualize the offline trajectories in the Maze2D navigation task, together with trajectories sampled from SIHD and two classical baselines, Diffuser and HD. As shown in Figure g the offline dataset includes diverse but suboptimal paths with substantial repeated and invalid state explorations. The trajectory generated by Diffuser does not fully cover the offline dataset and displays relatively simple behavior patterns. The HD baseline achieves full coverage: however, repeated and invalid explorations persist and may even intensify due to overreliance on the offline dataset. In comparison, SIHD generates diverse trajectories while substantially redueing repeated exploration of invalid states. This improvement is attributed to the structural entropy regularizer, which mitigates overreliance on the offline dataset and enhances long-ho horizon decision-making performance.
Offline Trajectocies
Diffuser
Figure 8: Trajectory visualizations comparing SIHD with classical baselines, Diffuser and HD, the Maze2D navigation task from the starting state (green color) to the goal state (red color).
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly outlined the contributions and scope of our paper in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have outlined the limitations of our paper in the conclusion and the details are provided in our appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer: [Yes]
Justification: For each theoretical result, we have included the complete set assumptions and correct proof in appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
The source codes are accessible via an anonymous link in the experiment section, and specific experimental setups provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The main experimental results be replicated using the source codes accessible via an anonymous link provided in the evaluation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
Moolic//guides/Coeeeubbiis itw.oeeeSuumiissiooooooo.o for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppplccguua.eeaeaeccuuea
Ctteeee//ooSSSSSSSSS.Seaa www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided detailed experimental settings (hyperparameters and opti- mizer) in our evaluation and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In our evaluation section, we have conducted multiple trials with different random seeds, and the charts reflect the average performance and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided sufficient information on the computer resources in our appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://heeuria..cccccbbbb//itiia
Answer: [Yes]
Justification:
Our paper strictly maintains anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is focused fundamental research in reinforcement learning and is not specifically tied to any particular applications. Furthermore, there is no direct path to negative applications.
Guidelines:
26

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We appropriately cited the original papers for our experimental datasets and models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p///earrrppcc//oooeeeeeeee/ /c/////oeeeeeeccc//22////.. for what should or should not be described.
29


Reviews:
[{"review_id": "mfnxCTy7cM", "reviewer": "Reviewer_42zk", "summary": "This paper proposes SIHD, a hierarchical diffusion framework for offline reinforcement learning that adaptively builds multi-scale policy hierarchies based on structural information from offline trajectories. By quantifying information gain across state communities and incorporating it into conditional diffusion, SIHD enables flexible and effective long-horizon planning. A structural entropy regularizer further promotes exploration of underrepresented states while mitigating extrapolation errors. Experiments on D4RL benchmarks show that SIHD outperforms prior methods in both decision quality and generalization.", "questions": "1. How does the result benefit from the hierarchical diffusion structure compared with simple 2-layer diffusion structure? Can the authors show the difference with experiments like a case study?\n2. What is the intuiation of using such a encoding tree to establish the hierarchical partition?", "limitations": "The authors addressed the limitations while they mentioned this work doesn't have any potential negative societal impacts.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of this paper is novel and solid, improving from the 2-layer structure to the multi-layer structure is promising.\n2. The method proposed in this paper is interesting, especially about the threorical part. The partition algorithm is great, while the authors need to clarify the intuition of the partition.\n3. The empirical result is very strong, the authors show a impressive improvement compared with previous sota algorithms.\n\nWeakness:\n1. The writing of this paper needs to be polished. The idea and intuition of each algorithm component is unclear in the paper. Some of the notations is confusing, e.g. sec 4.1.\n2.(minor). The figure 2 is hard to read, it would be better if the authors could change the lines in the figure to make them easier to distinguish.\n3. Is the improvement of this work compared with previous work incremental?\n4. The experiments are limited to D4RL, the authors could add more experiments on different benchmarks like meta-world.", "was_revised": true, "final_justification": "My major concern was mainly related to the writings. Since the authors claim they will improve their writing and they have list the detailed updates, I tend to maintain my positive score.", "rebuttal": "Thank you very much for your valuable and insightful comments. They are extremely helpful in improving the quality of our paper. We have systematically and directly addressed each of the Weaknesses (W) and Questions (Q) you raised. Within the constraints of the rebuttal’s word limit and anonymity requirements, we have made every effort to provide thorough and detailed responses to clarify our contributions and resolve your concerns.\n\n$\\bullet$ W1: We have revised the paper to improve the clarity of writing and the presentation of algorithmic components. Specifically, we clarified the notation used throughout the paper: superscripts $h$ and $i$ denote the diffusion layer and temporal index within the same layer, respectively, while subscripts $g$ and $sa$ indicate subgoal sequences and state-action subtrajectories.\n\nIn response to the feedback on Figure 2, we have redrawn the figure to improve its readability. The three key modules are now visualized separately in distinct subfigures, allowing for a clearer depiction of the processing flow at each stage. Additionally, we expanded the introductory paragraph of Section 4 to briefly explain the roles of the three main modules:\n\nHierarchy Construction Module: captures the topological structure of offline states based on feature similarity and automatically uncovers hierarchical communities by optimizing structural entropy. These communities are then used to construct a flexible, multi-scale diffusion hierarchy.\n\nConditional Diffusion Module: inputs temporally segmented trajectories into corresponding layers of a shared diffusion model, integrating quantized rewards and structural signals to perform forward diffusion and reverse prediction at multiple time scales.\n\nRegularized Exploration Module: leverages structural entropy as a measure of generative diversity and introduces a regularization loss to encourage the exploration of underrepresented offline states during training and inference.\n\nThese changes are intended to make the overall pipeline and technical intuition easier to follow.\n\n$\\bullet$ W2: As outlined in the Introduction and Related Work sections, although prior studies [1, 2] have proposed hierarchical diffusion models, they are typically restricted to a fixed two-level structure with uniform temporal segmentation. These methods lack the flexibility to adapt across varying time scales and essentially operate as two-layer diffusion models. In contrast, our work addresses a core open challenge in the field—designing a truly general and flexible hierarchical diffusion framework. To the best of our knowledge, this is the first method to realize a multi-level diffusion hierarchy that adapts to diverse temporal abstractions, making our contribution a substantive advancement rather than an incremental improvement.\n\n$\\bullet$ W3: To ensure fair comparison with prior work, our initial experimental setup focused on benchmarks commonly adopted in hierarchical diffusion studies [1, 2], the majority of which are based on D4RL. Nevertheless, to address the reviewer’s concern regarding generality, we conducted additional experiments on the FrankaKitchen and NeoRL [3] benchmarks.\n\nIn the NeoRL benchmark, we also included MB-PPO [3], a strong model-based baseline known for its performance in this setting. The experimental results demonstrate that SIHD consistently achieves higher average returns across different offline tasks. These findings further substantiate the effectiveness and generality of our proposed framework in diverse decision-making environments.\n\n|FrankaKitchen|Partial|Mixed|\n|:-:|:-:|:-:|\n|Diffuser|$56.2 \\pm 5.4$|$50.0 \\pm 8.8$|\n|HD|$73.3 \\pm 1.4$|$71.7 \\pm 2.7$|\n|SIHD|$76.8 \\pm 1.7$|$74.0 \\pm 2.3$|\n\n|NeoRL|FinRL-L-99|FinRL-L-999|FinRL-M-99|FinRL-M-999|\n|:-:|:-:|:-:|:-:|:-:|\n|MB-PPO|$328$|$656$|$1213$|$698$|\n|HDMI|$415$|$733$|$1007$|$754$|\n|SIHD|$457$|$760$|$1294$|$772$|\n\n$\\bullet$ Q1: In Sections 5.3 (Ablation Study) and Appendix C.1 (Sensitivity Analysis), we have already evaluated simplified two-layer diffusion variants of our model—SIHD-DH (with a fixed time horizon) and SIHD-2 (with structure entropy but limited to two levels). The results, summarized in the table below, show that both variants perform noticeably worse than the full SIHD model. This confirms the quantitative advantage of a deeper hierarchical diffusion structure over simpler two-layer architectures.\n\n|AntMaze|U-Maze|Medium|Large|\n|:-:|:-:|:-:|:-:|\n|SIHD-DH|$93.1 \\pm 3.8$|$88.6 \\pm 5.4$|$85.2 \\pm 4.7$|\n|SIHD-2|$93.5 \\pm 3.0$|$89.8 \\pm 4.7$|$86.1 \\pm 4.5$|\n|SIHD|$96.5 \\pm 2.8$|$92.2 \\pm 5.0$|$89.4 \\pm 4.2$|\n\nAdditionally, in the Qualitative Comparison section, we extended Figure 7 to present visualizations of hierarchical subgoal generation on the Maze2D navigation task. Compared to HDMI, SIHD produces subgoals with more flexible and longer-range temporal transitions: low‑level diffuser generates fine-grained waypoints that navigate local obstacles, mid‑level diffuser proposes intermediate subgoals that bridge across small clusters of states—e.g., navigating between adjoining corridors, and high‑level diffuser identifies long‑range landmarks or critical ``turning points\" across the entire maze, allowing for flexible, multi‑scale planning. This illustrates how the hierarchical design enables more globally coherent planning behavior.\n\nDue to the limitations of the rebuttal format, we are unable to present the updated Figure 7 here, but we will include it in the camera-ready version after conference acceptance for improved clarity and illustration.\n\n$\\bullet$ Q2: The use of an encoding tree for hierarchical partitioning is grounded in the principle of structural information theory, which enables hierarchical decomposition of graph nodes without relying on any task-specific prior knowledge. This approach has been validated across various domains [4-6] for its effectiveness in capturing meaningful multi-level structure.\n\nImportantly, the encoding tree supports a well-defined metric—structural entropy—which quantitatively evaluates the quality of the hierarchical partitioning. Unlike standard Shannon entropy, structural entropy promotes not only diversity among nodes but also preservation of the hierarchical community structure. This advantage is formally established in Theorem 4.2 of our paper.\n\nBy leveraging the encoding tree and structural entropy, our framework enables a flexible and general multi-level trajectory diffusion process. Furthermore, the integration of structural entropy as a regularizer promotes diverse exploration under the hierarchical structure, which enhances both model performance and generalization.\n\nReferences:\n\n[1] Hierarchical diffusion for offline decision making. ICML 2023.\n\n[2] Simple hierarchical planning with diffusion. ICLR 2024.\n\n[3] NeoRL: A near real-world benchmark for offline reinforcement learning. NeurIPS 2022.\n\n[4] Robustness Evaluation of Graph-based News Detection Using Network Structural Information. SIGKDD 2025.\n\n[5] Effective Exploration Based on the Structural Information Principles. NeurIPS 2024.\n\n[6] Incremental measurement of structural entropy for dynamic graphs. AIJ 2024."}, {"review_id": "4oyjsoeSYH", "reviewer": "Reviewer_mARi", "summary": "The paper introduces SIHD, a novel hierarchical diffusion method for offline RL. The main idea is to build a multi-scale hierarchy by first partitioning the state space, and thus trajectories in the data, into tree-based structures, and then train diffusion models at different levels of this hierarchy to enable long-horizon inference. The method is tested on 3 domains from the D4RL benchmark and compares favorable to both non-hierarchical and hierarchical state-of-the-art baselines.", "questions": "Please address limitations above", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper considers a relevant and well-motivated problem (long-horizon generalization in offline RL)\n2. The paper tackles and important limitation of existing hierarchical methods (i.e., the hierarchy is mostly restricted to 2 layers), which limits their scalability to large horizons\n3. The proposed method performs well empirically in comparison to both non-hierarchical and hierarchical state-of-the-art baselines\n4. Experiments to ablate the main algorithmic design choices are provided, showing that each of them is indeed relevant\n5. Existing literature seems to be properly discussed (though I am not an expert of hierarchical methods for offline RL)\n\nWeaknesses:\n1. I found the paper very hard to read, especially Section 4 given the convoluted notation with many sub/superscripts. After reading it multiple times, I only got a high-level understanding of the proposed method, while I am not sure I fully grasp all the details to properly judge its significance. Both Figure 2 and 3 didn't help as they only list the main algorithmic components labeled by hard-to-read notation and without explaining their functionalities. I think a figure that more directly shows how a trajectory may be segmented into different hierarchies, which part goes into which diffuser, and what's the conditional information used for those would greatly help. \n2. Also, in was not clear what's the conditional information used at the different levels of the hierarchy. Equation 8 seems to imply that the subgoal g_i is used as conditioning, but then the right-hand side doesn't depend on g_i (unless that dependence is hidden in R?). Equation 9 seems to imply that the conditioning is the exponential of the reward for the last level of the hierarchy, while intermediate layers use subgoal information. Finally, Equation 12 seems to imply that intermediate layers actually use the structural information gain\n3. One limitation of the proposed method is that it needs to train one different diffusion model for each level of the hierarchy. While it is true this is in the attempt to overcome the limit of 2 layers in existing methods, this brings its own issues: (1) it may be computationally much more expensive to train and use at test-time, (2) it prevents apple-to-apple comparison with existing methods since the proposed one has more capacity/parameters. There is no discussion about this in the paper\n4. The motivation and benefits of the structural entropy-based exploration regularizer are not clear. What does it even mean to \"explore\" in offline RL? Why would we want to go out of the data distribution, when the majority of offline RL works actually try to stay within data support to prevent extrapolation errors?\n5. While the whole approach is motivated by enabling long-horizon predictions, the experiments are still done on relatively short-horizon tasks and with only 3-4 hierarchy layers. There is no scalability analysis (in terms of compute/data/etc) for these variables\n6. A more qualitative empirical evaluation showing, eg, what the diffusers learn at different hierarchies is missing\n7. It was not clear how the proposed method can be used at test time (eg for planning). I think this is relegated to an appendix, but it seems a very important aspect to have in the main paper", "was_revised": true, "final_justification": "Reasons why I have increased my score and confidence:\n- The authors have addressed all concerns about writing quality and updated the paper accordingly, which I think essential to reach the acceptance bar\n- The authors have provided additional experiments that I find quite valuable (especially the one about comparing algorithms at the same capacity)\n\nReasons not to increase further:\n- the main algorithmic limitations (eg scalability / complexity) are still there and seem intrinsic to the method. Experiments are quite small-scale to conclude these are not issues", "rebuttal": "Thank you very much for your valuable and insightful comments. They are extremely helpful in improving the quality of our paper. We have systematically and directly addressed each of the Weaknesses (W) and Questions (Q) you raised. Within the constraints of the rebuttal’s word limit and anonymity requirements, we have made every effort to provide thorough and detailed responses to clarify our contributions and resolve your concerns.\n\n$\\bullet$ W1: To improve clarity and readability, we have made comprehensive revisions to Section 4 of the paper. Specifically, we added clear explanations for the subscripts and superscripts used in our notation—where superscript $h$ and $i$ indicate the diffusion hierarchy level and temporal ordering within that level, and subscripts $g$ and $sa$ refer to subgoal sequences and state-action subtrajectories, respectively.\n\nIn response to your comments on Figures 2 and 3, we redesigned Figure 2 to visually separate the three core modules into distinct subfigures, each illustrating the technical workflow of one stage in a more intuitive manner. Additionally, we expanded Figure 3 to provide a more concrete, visual depiction of the full hierarchical diffusion process—from community partitioning, trajectory segmentation, and subgoal extraction, to conditional signal definition—based on offline trajectories.\n\n$\\bullet$ W2: In Equation 8, within the control-as-inference paradigm, the subgoal $g_i^{\\mathcal{K} - 1}$ is implicitly defined as the terminal state of the corresponding subtrajectory. Specifically, it is set to the cumulative length of preceding subtrajectories, i.e., $g_i^{\\mathcal{K} - 1} = \\sum_{j=1}^{i} l_{sa}^{\\mathcal{K} - 1,j}$. This dependence is not explicitly shown on the right-hand side of Equation 8 but is encoded through the structure of the generated subtrajectory $\\tau_g^{h,i}$.\n\nTo generalize this idea, Theorem 4.1 formalizes a unified formulation of hierarchical conditional diffusion, where the conditioning term $y(\\tau_g^{h,i})$ represents a flexible signal associated with the subgoal state. In prior approaches, this is often instantiated as the cumulative reward over the subtrajectory. However, such a reward-based signal is often insufficient—especially in long-horizon or sparse-reward settings—due to its limited informativeness and locality.\n\nTo address this, in our framework:\n\nAt the top layer, where the receptive field covers the full trajectory, we retain cumulative return-based conditioning to reflect global objectives.\n\nAt intermediate layers, where the model operates on shorter temporal windows with limited context, we introduce a structure-aware signal based on the expected hitting probability of subgoals via random walks on the offline state graph. This formulation, defined in Equation 12, captures meaningful topological relationships in the latent space and serves as a robust and informative form of guidance for lower-level diffusion.\n\nWe believe this hierarchical conditioning strategy is key to the improved performance and flexibility of our model, and we will clarify these connections more explicitly in the final version of the paper.\n\n$\\bullet$ W3: We acknowledge that introducing additional hierarchical levels may increase the computational burden during both training and inference. To address this, we included a detailed analysis in the original manuscript (Appendix C.2, Computational Efficiency), where we compare the runtime cost of SIHD with baseline methods such as Diffuser and HD. The results show that while SIHD does incur slightly higher training and inference time compared to HD, the overhead is modest and well-justified by the consistent performance gains. More importantly, SIHD retains a significant efficiency advantage over single-layer Diffuser models, reinforcing the practical feasibility and scalability of our approach.\n\nTo further address concerns about model capacity, we conducted an additional ablation on Single-task Maze2D in which we reduced the parameter count of each individual diffusion layer so that the total number of parameters across all layers matches that of the two-layer HD model. As summarized in the following table, even without increasing overall model capacity, increasing the number of hierarchical layers yields consistent and significant performance improvements. This demonstrates that the benefit of SIHD arises from its hierarchical structure rather than simply from added model size. We will include these discussions and results more explicitly in the final version of the paper.\n\n|Single-task Maze2D|U-Maze|Medium|Large|\n|:-:|:-:|:-:|:-:|\n|HDMI|$120.1 \\pm 2.5$|$121.8 \\pm 1.6$|$128.6 \\pm 2.9$|\n|HD|$128.4 \\pm 3.6$|$135.6 \\pm 3.0$|$155.8 \\pm 2.5$|\n|SIHD|$140.7 \\pm 2.1$|$142.5 \\pm 2.9$|$157.3 \\pm 2.0$|\n\n$\\bullet$ W4: In offline RL, it is indeed essential to avoid excessive extrapolation beyond the support of the dataset, as this can lead to unreliable value estimates. However, prior works [1, 2] have shown that controlled exploration—particularly toward underrepresented but plausible regions of the state-action space—can improve robustness to value estimation errors and enhance generalization, especially when guided by entropy-based regularization.\n\nThe challenge, as you noted, lies in balancing this exploration. Unconstrained deviation from the data distribution may introduce harmful out-of-distribution (OOD) behaviors. To address this, we introduce a structural entropy-based regularizer, which differs from conventional entropy maximization approaches in two key ways:\n\nStructural Awareness: Rather than encouraging exploration uniformly across the entire state space, our method promotes diversity within the hierarchical community structure inferred from the offline data. This ensures that the exploration remains aligned with the intrinsic topological structure of the data, as captured by the encoding tree.\n\nControlled Diversity: The structural entropy metric formalized in Theorem 4.2 not only promotes generative diversity but also penalizes excessive deviation from the established hierarchical state structure. This provides a principled way to balance exploration and support preservation, effectively mitigating extrapolation risks.\n\nOur ablation results further demonstrate that incorporating this regularizer improves performance compared to both no regularization and standard entropy-based exploration.\n\n$\\bullet$ W5: In our experimental evaluation, we use trajectory lengths up to 390 in Maze2D and 450 in AntMaze environments—substantially longer than typical short-horizon tasks—demonstrating SIHD’s capacity for long-range planning. To further investigate scalability, we conduct a detailed analysis in Appendix C.1 (Sensitivity Analysis), where we vary the number of diffusion layers from 2 to 6 across different scales of the AntMaze tasks.\n\nThe results indicate that increasing the number of hierarchy levels enables planning over longer time spans. However, we also observe that deeper hierarchies result in much shorter subtrajectory lengths per layer, which can degrade the generative performance of individual diffusion models. This trade-off highlights a key design consideration in balancing depth and subtrajectory complexity.\n\nMoreover, in Appendix C.2 (Computational Efficiency), we compare the training and inference time across tasks of varying scales. Despite introducing additional diffusion layers, SIHD maintains computational efficiency comparable to the two-layer HD baseline, due to effective parallelism and reduced per-layer sequence lengths.\n\n$\\bullet$ W6: We have expanded our Qualitative Comparison (Figure 7) to illustrate exactly what each diffusion level learns on the Maze2D navigation task: low‑level diffuser generates fine-grained waypoints that navigate local obstacles, mid‑level diffuser proposes intermediate subgoals that bridge across small clusters of states—e.g., navigating between adjoining corridors, and high‑level diffuser identifies long‑range landmarks or critical “turning points” across the entire maze, allowing for flexible, multi‑scale planning.\n\nBy visualizing these subgoals side‑by‑side (and contrasting them with those from the two‑layer HDMI baseline), readers can immediately see how additional hierarchy levels yield increasingly abstract and spatially dispersed goals. Due to rebuttal submission constraints, we cannot include the updated figure here, but it will appear in full in the camera‑ready version after conference acceptance.\n\n$\\bullet$ W7: In the current version of the paper, we provide a complete pseudocode implementation of the inference procedure in Appendix A.5. To improve accessibility and clarity, we have now moved a high-level summary of this procedure into the main text and added explanatory commentary to better convey the planning mechanism. \n\nAt inference, our SIHD planner proceeds as follows (see Algorithm4): We first initialize each hierarchy’s subgoal buffers and the 1‑layer state–action sequence (lines 3 and 4). Then for each planning step, we (i) sample an initial noisy latent sequence from the diffuser prior (lines 6 and 7), (ii) invoke the subgoal proposer to revise the top‑level goal if its terminal criterion is satisfied (lines 8-10), and (iii) execute a reverse‑diffusion rollout across latent timesteps—at each diffuser step to sample the next latent sequence (lines 11-17). Finally, we decode and integrate the new state and action back into the 1‑layer sequence (lines 18 and 19), and after $H$ iterations output the resulting action trajectory (line 21).\n\nReferences:\n\n[1] Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.\n\n[2] Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning. NeurIPS 2024."}, {"review_id": "wVitLq61Pr", "reviewer": "Reviewer_uo2i", "summary": "In this paper, a Structural Information-based Hierarchical Diffusion framework, namely SIHD is proposed, where structural information embedded in offline trajectories is utilized to construct the diffusion hierarchy. Furthermore, a structural entropy regularizer is presented to encourage exploration of underrepresented states. Extensive experiments were conducted on D4RL to show the effectiveness and generalization of the proposed method.", "questions": "Please refer to the Weaknesses part of the review.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe presented idea that using structural information existed in the historical trajectories for offline policy learning is novel and extensive experiments are conducted to show the effectiveness of the proposed framework. Moreover, theoretical guarantees are provided to enable the decomposition of the conditional generation problem for offline trajectories into a hierarchical diffusion process.\n\nWeaknesses:\nThe main concerns are the scalability and generalization of the proposed framework. Besides D4RL, it is unclear whether the required structure information for the deriving of the tree-structured partitioning of state communities exists in other offline datasets. Also, how can we guarantee the accuracy of the partitioning of the state communities?  Does it have a clear impact on the final performance of the presented method? As is shown in the experiments, the community partitioning is time-consuming and computation-expensive, it may be hardly to be extended to large-scale systems, experiments on some real-world applications may be necessary to verify the method’s scalability. Furthermore, only experimental results are provided to support the performance of the structural entropy regularizer while theoretical guarantees are lacked.", "was_revised": true, "final_justification": "The authors' response has addressed some of my concerns, and I'd like to keep the positive score of the paper. But it still seems the proposed framework is limited to some offline datasets that can be partitioned in a tree-structured manner, which may not be prevalent in reality.", "rebuttal": "Thank you very much for your valuable and insightful comments. They are extremely helpful in improving the quality of our paper. We have systematically and directly addressed each of the Weaknesses (W) and Questions (Q) you raised. Within the constraints of the rebuttal’s word limit and anonymity requirements, we have made every effort to provide thorough and detailed responses to clarify our contributions and resolve your concerns.\n\n$\\bullet$ W1: To ensure fair comparisons, our initial experiments focused on benchmarks commonly used in hierarchical diffusion research, most of which are based on D4RL. However, to address the reviewer’s concern regarding generalizability, we conducted additional evaluations on the FrankaKitchen and NeoRL [1] benchmarks.\n\nThe results demonstrate that SIHD consistently achieves higher average returns across these diverse offline tasks. This not only reinforces the performance and generality of our framework but also empirically validates the feasibility of deriving hierarchical structures based on structural information from offline states beyond D4RL. These findings suggest that the state-space structure necessary for tree-structured partitioning can be reliably inferred from a variety of offline datasets, highlighting the scalability and broad applicability of our approach.\n\n|FrankaKitchen|Partial|Mixed|\n|:-:|:-:|:-:|\n|Diffuser|$56.2 \\pm 5.4$|$50.0 \\pm 8.8$|\n|HD|$73.3 \\pm 1.4$|$71.7 \\pm 2.7$|\n|SIHD|$76.8 \\pm 1.7$|$74.0 \\pm 2.3$|\n\n|NeoRL|FinRL-L-99|FinRL-L-999|FinRL-M-99|FinRL-M-999|\n|:-:|:-:|:-:|:-:|:-:|\n|MB-PPO|$328$|$656$|$1213$|$698$|\n|HDMI|$415$|$733$|$1007$|$754$|\n|SIHD|$457$|$760$|$1294$|$772$|\n\n$\\bullet$ W2: In our method, the accuracy of state community partitioning is ensured through the minimization of structural entropy, which serves as an unsupervised optimization objective for identifying coherent hierarchical communities. This approach has been widely adopted in hierarchical decision-making [2] and intrinsic reward design [3] within reinforcement learning.\n\nTo further address your concern, we quantitatively evaluated the quality of the partitioned communities on the Maze2D task using two standard unsupervised clustering metrics: Silhouette score (S) and Calinski-Harabasz index (CH). The results show that our structural entropy-based partitioning achieves $S>0.8$ and $CH>170$, indicating strong intra-community cohesion and inter-community separation—both hallmarks of high-quality clustering.\n\nTo assess the impact of community quality on performance, we designed an ablation variant, SIHD-FT, which replaces structural entropy-based partitioning with fixed-interval temporal segmentation. Experimental results show a clear drop in average return for SIHD-FT compared to the full SIHD model. This confirms that minimizing structural entropy not only improves partitioning accuracy but also plays a critical role in identifying key offline states and guiding the construction of effective hierarchical diffusion structures.\n\n|Multi-task Maze2D|U-Maze|Medium|Large|\n|:-:|:-:|:-:|:-:|\n|SIHD-FT|$149.3 \\pm 1.7$|$146.8 \\pm 2.1$|$167.2 \\pm 4.8$|\n|SIHD|$157.0 \\pm 0.6$|$156.8 \\pm 1.7$|$169.4 \\pm 2.7$|\n\n$\\bullet$ W3: You are absolutely right that community partitioning introduces additional computational overhead, particularly in large-scale settings. To address this concern, we designed our system to perform state community partitioning entirely offline, storing the resulting structure as a dictionary. This avoids repeated computation during training and inference, ensuring that the hierarchical diffusion model remains practically deployable.\n\nIn Appendix C.2 (Computational Efficiency), we present a detailed comparison of training and inference time between SIHD and baselines such as Diffuser and HD across tasks of varying scale. Despite the inclusion of additional hierarchical layers, SIHD maintains comparable runtime efficiency to HD, thanks to parallelism and the reuse of precomputed structures.\n\n$\\bullet$ W4: While we provide strong empirical evidence demonstrating the benefits of the structural entropy regularizer, we also recognize the importance of theoretical support.\n\nTo that end, in addition to experimental validation, we include a formal theoretical analysis in Theorem 4.2, which establishes a lower bound formulation of Shannon entropy under a structural entropy framework. This result shows that our regularizer promotes exploration in a controlled manner: it encourages trajectory diversity while preserving the hierarchical community structure inferred from the offline data.\n\nUnlike conventional entropy maximization strategies [4] that risk inducing unsafe out-of-distribution actions, our formulation explicitly constrains exploration within topologically meaningful regions of the state space. This provides a theoretical foundation for the regularizer’s ability to balance generalization with robustness, mitigating extrapolation error in offline settings.\n\nReferences:\n\n[1] NeoRL: A near real-world benchmark for offline reinforcement learning. NeurIPS 2022.\n\n[2] Hierarchical State Abstraction Based on Structural Information Principles. IJCAI 2023\n\n[3] Effective Exploration Based on the Structural Information Principles. NeurIPS 2024.\n\n[4] Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning. NeurIPS 2024."}, {"review_id": "JAOC2EpgX7", "reviewer": "Reviewer_fnUf", "summary": "The authors investigate the classification of states in trajectories from the perspective of the undirected graph, and propose a multi-scale diffusion hierarchy model that simultaneously introduces a structural entropy regularizer to encourage exploration of underrepresented states while avoiding extrapolation errors caused by distributional shifts.", "questions": "1) The authors did not explain the meaning of $\\alpha^{-}$ in Section 3.3.\n2) In line 182, could the authors clarify how k is determined? Additionally, in line 184, since the HCSE optimization algorithm can be used to determine the optimal $\\mathcal{K}$, it would be helpful to understand why $\\mathcal{K}$ is directly specified in the experimental section.\n3) From equation (6), it seems that this partitioning will result in state-action sequences of varying lengths within the trajectory. When h = 1, could the authors clarify whether the corresponding $\\tau$ refers to a state-action pair or a sequence of state-action pairs? Additionally, since sequences of fixed length are required for training the diffusion model, how do the authors handle the issue of varying sequence lengths across different layers in the hierarchy?\n4) A simple example is helpful to help readers understand this paper:\n\nSuppose that a trajectory: [1, ..., 1000]\n\nh=1: [1, 2, 3, ..., 1000]\n\nh=2: [1,2], [3, 4, 5], [6, 7, 8, 9, 10], ...\n\nh=3: [1,2, 3, 4, 5], [6, 7, 8, 9, 10, 11, 12, 13], [14, ...]\n\n....\n\nThen, 1) whether the separation shown above is right? 2) What are the corresponding goals of each layer ?\n\n5) In Figure 3 (c), should k-1 be replaced with h? Additionally, $l_g^{K-1}$ appears in (c) but is not introduced prior to line 197. Would it be possible for the authors to provide clarification? And in line 202, the authors also do not introduce the meaning of $l_g^{h,i}$.\n6) Based on the authors' description, the paper utilizes classifier-free guidance. Could the authors clarify how the conditions of the diffusion model are pre-specified during inference?\n7) Could the authors clarify the meaning of \"shared\" mentioned in line 226? My understanding is that each layer represents a new diffusion model.\n8) In line 230, I suggest important references about class-free guidance, such as \n\n  [1] Classifier-Free Diffusion Guidance.\n\n  [2] More Control for Free! Image Synthesis with Semantic Diffusion Guidance.\n\n9) While I can understand that diffusion models are capable of modeling complex distributions of sequences, I am unclear about how the transition probability is computed. It would be helpful if the authors could provide a more detailed explanation.\n10) Would constructing an additional weighted state graph $\\mathcal{G}_s^\\prime$ incur significant time overhead? How is Shannon entropy calculated? I suggest that the authors provide a more detailed explanation to help readers better understand. Why do the authors only apply entropy regularization term on h=1?\n11) I notice that the paper does not provide a table of important hyperparameters. I suggest that the authors include the hyperparameter settings related to the experiments and methods.\n12) Could the authors clarify how much computational resource is required for state community partitioning on the D4RL dataset?\n13) In Table 4, intuitively, increasing the number of layers in the hierarchical diffusion model should directly increase the time for one generation. However, SIHD takes less time than HD. Why is this the case?\n14) I noticed that the authors mention online decision-making in Figure 2, but the experimental results do not include online RL. Therefore, I suggest that the authors either add relevant online RL experiments or revise Figure 2.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Pros: The theoretical framework is well-developed, the paper is well-organized, and the experiments are fairly comprehensive.\nCons: However, some parts of the paper are expressed ambiguously, making it difficult for readers to fully understand the intended meaning.", "was_revised": true, "final_justification": "The authors have addressed my concerns.", "rebuttal": "Thank you very much for your valuable and insightful comments. They are extremely helpful in improving the quality of our paper. We have systematically and directly addressed each of the Weaknesses (W) and Questions (Q) you raised. Within the constraints of the rebuttal’s word limit and anonymity requirements, we have made every effort to provide thorough and detailed responses to clarify our contributions and resolve your concerns.\n\n$\\bullet$ Q1: We have updated Section 3.3 to clarify our structural‑information notation by explicitly defining $\\alpha^-$ as the parent of any non‑root tree node $\\alpha$.\n\n$\\bullet$ Q2: For Line 182, we determine $k$ by sweeping over all plausible values in $[1, |\\mathcal{S}|]$, constructing for each state vertex a $k$‑nearest‑neighbor graph (keeping its $k$ most similar neighbors) and computing the resulting one‑dimensional structural entropy. We then select the $k$ that maximizes this entropy. A pseudocode description will be provided in Appendix A of the camera‑ready version after conference acceptance.\n\nAs for Line 184, $\\mathcal{K}$ denotes the maximum depth of both the coding tree and the diffusion hierarchy. While HCSE optimizes the partition structure under a fixed depth limit, it does not choose that limit itself. Accordingly, we specify $\\mathcal{K}$ upfront in our experiments, with detailed justification and a sensitivity analysis presented in Appendix C.1.\n\n$\\bullet$ Q3: Indeed, our trajectory segmentation is driven by the automatically discovered state communities rather than fixed time intervals, so the resulting sub‑trajectories naturally vary in length. To train the diffusion model—which requires inputs of uniform length—we pad each shorter subsequence with terminal states until it reaches the prescribed length (see lines 483–485 in Appendix A.6). Furthermore, when $h=1$, each trajectory segment corresponds to an entire sequence of state–action pairs (not just a single pair), as explained in the text.\n\n$\\bullet$ Q4: Here’s a toy example of how SIHD hierarchically partitions a trajectory of 1000 timesteps into sub‑trajectories at different levels $h$. In each sub‑trajectory, the final state is treated as that segment’s ``sub‑goal.\"\n\nLevel $h=\\mathcal{K}$ (top, coarsest): $[1, 2, \\dots, 1000]$.\n\nLevel $h=\\mathcal{K} - 1$: $[1, 2, \\dots, 90], [91, 92, \\dots, 200], \\dots, [901, 902, \\dots, 1000]$.\n\nLevel $h=2$ (fine): $[1, 2, \\dots, 30], \\dots, [78, 79, \\dots, 90], \\dots, [910, 941, \\dots, 950], \\dots, [968, 969, \\dots, 1000]$.\n\nLevel $h=1$ (finest): $[1, 2, \\dots, 8], [9, 10, \\dots, 15], \\dots, [988, 989, \\dots,994], [995, 996, \\dots, 1000]$.\n\n$\\bullet$ Q5: In Figure 3(c), $\\mathcal{K}-1$ ndeed represents an arbitrary intermediate layer and can be equivalently replaced with $h$. We have revised Figure 3 accordingly to improve clarity.\n\nAdditionally, we have updated the main text prior to line 197 to explicitly introduce the subtrajectory segmentation and subgoal extraction for the second-highest layer, as well as the parameter $\\tau_g^{h,i}$, which denotes the subgoal trajectory length. To further address your comment, we now clarify the notation $\\tau^\\mathcal{K}_g$ and $\\tau_g^{h,i}$ in the corresponding context to ensure they are well-defined before use.\n\nWe agree that including the trajectory partitioning example you suggested would enhance understanding. We will incorporate it into the appendix in the camera-ready version.\n\n$\\bullet$ Q6:\nDuring inference with the highest-level diffusion model, we employ classifier-free guidance by setting the reward signal—used as conditioning during training—to null. This allows the model to generate a sequence of subgoals without explicit reward input. For conditioning the next-level diffusion model, we select the next subgoal from this sequence and use it to compute the corresponding conditional signal.\n\nSpecifically, we locate the node in the optimal encoding tree—constructed from offline trajectories—that is closest to the subgoal state in Euclidean space. Based on this matched node (i.e., the state community), we compute the conditioning signal according to Equation 12. This process is repeated hierarchically for each layer until the full trajectory is generated.\n\n$\\bullet$ Q7: Here, ``shared\" refers to the fact that, for a given hierarchy level $h$, we train a single diffusion model whose parameters are shared across all sub‑goal sequences at that level. In other words, each layer $h$ indeed has its own diffusion network, but within that layer every sub‑trajectory’s generation uses the same model weights—i.e., the model is shared across sub‑goals at level $h$, rather than having a distinct network for each individual segment.\n\n$\\bullet$ Q8: Thank you for the suggestion. We have added citations to Classifier‑Free Diffusion Guidance [1] and More Control for Free! Image Synthesis with Semantic Diffusion Guidance [2] at line 230.\n\n$\\bullet$ Q9:  To compute the joint probability $p_{\\theta_1}(s_t, s_{t+1})$ between any adjacent state pair, we proceed as follows: we begin by sampling Gaussian noise sequences $\\tau_{g, K}^{1,1} \\sim \\mathcal{N}(0, \\mathcal{I})$. These represent initial noisy trajectories. We then apply the reverse process described in Equation 11 to iteratively denoise these trajectories, producing $n$ predicted sequences $\\tau_{g, 0}^{1,i}$ with $i=1, \\dots, n$. From each denoised sequence, we extract the states at time steps $t$ and $t+1$, forming a sample set $\\\\{s_t^i, s_{t+1}^i\\\\}$. Finally, we use a two-dimensional Gaussian kernel density estimator over this sample set to estimate the joint distribution $p_{\\theta_1}(s_t, s_{t+1})$.\n\nFor clarity and reproducibility, we will include the complete procedure for joint probability estimation as pseudocode in Appendix A, after conference acceptance.\n\n$\\bullet$ Q10: As explained previously, constructing the weighted state graph $\\mathcal{G}_s^\\prime$ requires estimating the joint probability between all adjacent state pairs in the offline trajectories, which is computationally expensive. To mitigate this overhead, we derive in Theorem 4.2 a lower bound of the structural entropy in the form of Shannon entropy. This formulation not only provides theoretical justification for the entropy regularization but also yields an efficient estimation strategy.\n\nSpecifically, we leverage the $n$ denoised sequences $\\\\{\\tau_{g, 0}^{1,i}\\\\}$ obtained from the base diffusion model $\\epsilon_{\\theta_1}$. The states $s_t^i$ from these sequences are hierarchically clustered according to their Euclidean distances to the layer-wise centroids of the encoding tree $\\mathcal{T}_s^*$, yielding community sets $\\\\{\\mathcal{U}_h\\\\}$ with $h=1,\\dots,\\mathcal{K}-1$. For each community set $\\mathcal{U}_h$, we approximate its Shannon entropy $\\mathcal{H}(\\mathcal{U}_h)$ using a $k$-nearest neighbors entropy estimator [3], which averages the Euclidean distances between each community and its $k$-th nearest neighbor, as detailed in Equation 15.\n\nSince the entire entropy computation is based solely on the lowest-level diffusion model $\\epsilon_{\\theta_1}$, the entropy regularization is applied only at $h=1$. This design choice ensures computational efficiency while still enforcing meaningful structure through the regularization. \n\n$\\bullet$ Q11: We have added a description of the key hyperparameter settings in Appendix A and will ensure that this information is included in the camera-ready version upon acceptance.\n\n$\\bullet$ Q12: All experiments in our work were conducted on five Linux servers, each equipped with an NVIDIA RTX A6000 GPU and an Intel i9-10980XE CPU running at 3.00 GHz. To reduce redundant computation, we performed state community partitioning offline and stored the resulting structures in a dictionary format. This design ensures that the community assignments do not need to be recomputed during training or inference, thereby maintaining the overall computational feasibility of training and deploying the hierarchical diffusion model on the D4RL dataset.\n\n$\\bullet$ Q13: Intuitively, increasing the number of layers in a hierarchical diffusion model would introduce additional time overhead. As shown in Tables 4 and 5, SIHD generally requires more training and inference time than the two-layer HD model in most scenarios. However, it is important to note that under a fixed time horizon, adding more hierarchical layers leads to shorter sequence lengths for each individual diffusion model. Furthermore, both training and inference are implemented with parallel computation across layers. As a result, the time overhead introduced by the hierarchical structure is significantly mitigated, and in some cases, SIHD even demonstrates slightly better time efficiency compared to HD.\n\n$\\bullet$ Q14: In Figure 2, the term ``online decision-making\" refers specifically to the use of the trained hierarchical diffusion model for online inference or trajectory planning, rather than interaction-based online reinforcement learning. To avoid any potential misunderstanding, we have revised Figure 2 accordingly to clarify this intent.\n\nReferences:\n\n[1] Classifier-Free Diffusion Guidance. NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications.\n\n[2] More Control for Free! Image Synthesis with Semantic Diffusion Guidance. IEEE/CVF 2023.\n\n[3] Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 2003."}]

Author rebuttal:

Meta-review: This work proposes SIHD, a hierarchical diffusion method for effective and stable offline policy learning in long-horizon environments with sparse rewards. It extracts structural information from a similarity-guided topological graph to build tree-structured communities. And then learns different diffusion processes (can be parameter-shared or a shared diffusion model) for each tree layer. The structural information gain of each state community is used as conditional guidance information. In this work, theoretical guarantees are provided for the decomposition of the conditional generation problem into a hierarchical diffusion process with dynamic temporal scales. Moreover, it uses a structural entropy regularizer to promote exploration of underrepresented states. Experimental results on D4RL, FrankaKitchen, NeoRL demonstrate that the proposed method is promising.

The strengths of this work are listed as follows.

1. Long-horizon tasks with sparse rewards in offline RL are challenging and well motivated.
2. Compared to typical two-layer diffusion processes, a tree-structure diffusion method is novel.
3. The experimental results justify the effectiveness of SIHD.

In the final version, please include the added experimental results and improve the writing of this work (especially the notations).
Decision: Accept (poster)