Simple Hierarchical Planning with Diffusion
Authors: Chang Chen
Fei Deng
Kenji Kawaguchi
Caglar Gulcehre
Sungjin Ahn
Venue: ICLR 2024 poster (2024)
Keywords: Hierarchical Offline RL
Hierarchical planning
Hierarchical Reinforcement Learning
Diffusion-Based Planning
Primary area: reinforcement learning

Abstract: Diffusion-based generative methods have proven effective in modeling trajectories with offline datasets. However, they often face computational challenges and can falter in generalization, especially in capturing temporal abstractions for long-horizon tasks. To overcome this, we introduce the Hierarchical Diffuser, a simple, fast, yet effective planning method combining the advantages of hierarchical and diffusion-based planning. Our model adopts a “jumpy” planning strategy at the high level, which allows it to have a larger receptive field but at a lower computational cost—a crucial factor for diffusion-based planning methods, as we have empirically verified. Additionally, the jumpy sub-goals guide our low-level planner, facilitating a fine-tuning stage and further improving our approach’s effectiveness. We conducted empirical evaluations on standard offline reinforcement learning benchmarks, demonstrating our method’s superior performance and efficiency in terms of training and planning speed compared to the non-hierarchical Diffuser as well as other hierarchical planning methods. Moreover, we explore our model’s generalization capability, particularly on how our method improves generalization capabilities on compositional out-of-distribution tasks.

Published as a conference paper at ICLR 2024
SIMPLE HIERARCHICAL PLANNING WITH DIFFUSION
Chang Chen', Fei Deng', Kenji Kawaguchi", Caglar Gulcehre Sungjin Abn
1 Rutgers University, 2 National University of Singapore, EEPLL
4 Google DeepMind, 5 KAIST
ABSTRACT
Diffusion-basee generative methods have proven effective in modeling trajectories with offline datasets. However, they often face computational challenges and can falter in generalization, especially in capturing temporal abstractions for long- horizon tasks. To overcome this, we introduce the Hierarchical Diffuser, a simple, fast, yet surprisingly effective planning method combining the advantages of hi- erarchical and diffusion- strategy at the higher level, which allows it to have a larger receptive field but at a lower computational cost-a a crucial factor for diffusion-basee planning methods, we have empirically verified. Additionally, the jumpy sub-goals guide low- level planner, facilitating a fine-tuning stage and further improving our approach's s effectiveness. We conducted empirical evaluations standard offline reinforce- ment learning benchmarks, demonstrating our method's superior performance and efficiency in terms of training and planning speed compared the non-hierarchical Diffuser as well as other hierarchical planning methods. Moreover, we explore our model's ' generalization capability, particularly on how our method improves generalization capabilities on compositional out-of offdistribution tasks.
planning. Our model adopts "jumpy" planning
INTRODUCTION
Planning has been successful in control tasks where the dynamics of the environment are known (Sut- ton & Barto 2018 Silver et al.) 2016). Through planning. the agent can simulate numerous action sequences and assess potential outcomes without interacting with the environment, which can be costly and risky. When the environment dynamics are unknown, a world model (Ha & Schmidhuber 2018 Hafner et al. 2018 2019) can be learned to approximate the true dynamics. Planning then takes place within the world model by generating future predictions based on actions. This type of model-based planning is considered more data-efficient than model- free methods and tends to transfer well to other tasks in the same environment (Moerland et al. 2023 Hamrick et al.. 2020).
For temporally extended tasks with sparse rewards, the planning horizon should be increased accord- ingly (Nachum et al.) 2019. Vezhnevets et al., 2017b] Hafner et al.) 2022). However, this may not be practical as it requires an exponentially larger number of samples of action sequences to cover all possible plans adequately. Gradient-based trajectory optimization addresses this issue but can encounter credit assignment problems. A promising solution is to use hierarchical planning (Singh) 1992, Pertsch et al.) (2020; Sacerdoti 1974| Knoblock] [190),, where a high-level plan selects subgoals that are several steps apart, and lowelevel plans determine actions move from one subgoal the next. Both the high-level plan and each of the low-level plans are shorter than the original flat plan, leading to more efficient sampling and gradient propagation.
Conventional model- based planning typically involves separate world models and planners. However, the learned reward model can be prone to hallucinations, making it easy for the planner to exploit it (Talvitie) 2014). Recently, Janner et al. (2022b) proposed Diffuser,, framework where a single diffusion probabilistic model Sohl-Diccsttiin et al. 20155] Ho et al. 2020 Song et al. 2021] is learned to serve both the world model and the planner. It generates the states and actions in the full plan in parallel through iterative refinement, thereby achieving better global coherence. Furthermore, also
"This work was partly done while C.G. was at Google DeepMind. C.G. is currently affiliated with EPFL. Correspondence sungj a ac.

---

Published as a conference paper at ICLR 2024
allows leveraging the guided sampling strategy Dhariwal & Nichol (2021) provide the flexibility of adapting to the objective of the downstream task at test time.
Despite such advantages of Diffuser, how to enable hierarchical planning in the ceffusion-basee approach remains clusive to benefit from both diffusion- -efuuidon-based and hierarchical planning simulta- neously. Lacking this ability, Diffuser is computationally expensive and sampling inefficient due to the current dense and flat planning scheme. Moreover, we empirically found that the planned trajectories produced by Diffuser have inadequate coverage of the dataset distribution. This deficiency is particularly detrimental to diffusion-based planning.
In this paper, propose the Hierarchical Diffuser, a simple framework that enables hierarchical planning using diffusion models. The proposed model consists of two diffusers: one for high-level subgoal generation and another for low-level subgoal achievement. To implement this framework, we first split each training trajectory into segments of equal length and consider the segment's 's split points as subgoals. We then train the two diffusers simultaneously. The high-level diffuser trained on the trajectories consisting of only subgoals, which allows for a "jumpy" subgoal planning strategy and a larger receptive field at a lower computational cost. This sparseness reduces the diffusion model's burden of learning and sampling from high-dimensional distributions of dense trajectories, making learning and sampling more efficient. The low-level diffuser is trained to model only the segments, making it the subgoal achiever and facilitating fine-tuning stage that further improves approach's effectiveness. At test time, the high-level diffuser plans the jumpy subgoals first, and then the low-level diffuser achieves each subgoal by planning actions.
The contributions of this work are as follows. First, we introduce a diffusion- based hierarchical planning framework for decision-making problems. Second, we demonstrate the effectiveness of our approach through superior performance compared to previous methods on standard offline-RL benchmarks, as well efficient training and planning speed. For example, our proposed method outperforms the baseline by 12.0% on Maze2D tasks and 9.2% on MuJoCo locomotion tasks. Furthermore, we empirically identify a key factor influencing the performance of duffusion-based planning methods, and showcase our method's enhanced generalization capabilities on compositional cattof-diitributioo tasks. Lastly, we provide a theoretical analysis of the generalization performance.
2
PRELIMINARIES
2.1
DIFFUSION PROBABILISTIC MODELS
Diffusion probabilistic models DSchl-Dickstein et al.) 2015 Ho et al. 2020 Song al. (021)) have achieved state-of the-art generation quality on various image generation tasks (Dhariwal & Nichol 2021 Rombach et al. 2022 Ramesh al. [2022] Saharia et al. 2022). They model the data generative process as M steps of iterative denoising, starting from a Gaussian noise XM 10 I):
M-1
(x(xu) p(xM)
po px xm+1)) dx m==
II
Here, X1:M5 are latent variables of the same dimensionality as the data Xo. and
px X = (x 4o (x)m))), I)
is commonly a Gaussian distribution with learnable mean and fixed covariance. The posterior of the latents is given by a predefined diffusion process that gradually adds Gaussian noise to the data:
q(mm
x,) (xmm Vām (1 amm)
where the predefined o.m. 0 as -> 20, making q(xMs xo) R IO, for sufficiently large M.
In practice, the learnable mean (x()) ) often parameterized as a linear combination of the latent Xm and the output of noise-prediction U-Net (xm)) (Ronneberger et al., 2015). The training objective is simply to make EØ (xm)) predict the noise e that was used to corrupt Xg into xm:
(())= = Exo.m. [lle - [x
(4)
where X.m. Xa + VI Ome, E NN(.
2

---

Published as a conference paper at ICLR 2024
2.2 DIFFUSER: PLANNING WITH DIFFUSION
Diffuser (Janner et al. 2022b is a pioneering model for learning a diffusion-based planner from offline trajectory data. It has shown superior long-horizon planning capability and test-time flexibility. The key idea is to format the trajectories of states and actions into a two-dimensional array, where each column consists of the state-action pair a single timestep:
S0
S1
ST
(5)
ag
a1
Diffuser then trains a diffusion probabilistic model px(x)) from offline dataset. After training, Po(x) is able to jointly generate plausible state and action trajectories through iterative denoising. Importantly, px(x) does not model the reward, and therefore is task-agnostic. To employ px(x) to do planning for a specific task, Diffuser trains a separate guidance function Jx(x), and samples the planned trajectories from perturbed distribution:
(x) o (x) exp (Ja(x)).
Typically, To (x) estimates the expected return of the trajectory, so that the planned trajectories will be biased toward those that plausible and also have high returns. In practice, (x)x is implemented as a regression network trained to predict the return R(x) of the original trajectory x given a noise-corrupted trajectory Xm input:
(()) Ex,,, [[] - Jo(xm)
where R(x) can be obtained from the offline dataset, Xm. Vān x + vI Om O(
Sampling from Po (x) is achieved similarly as classifier guidance (Dhariwal & Nichol 2021 Sohl
Dickstein al. 20155, where the gradient V Vx.. J is used guide the denoising process (Equation 2
by modifying the mean from fmmm to pmm:
Pm < p0(X++1); P' f 1m pm wa (xxm))))))
Here, is hyperparameter that controls the scaling of the gradient. To ensure that the planning trajectory starts from the current state s, Diffuser sets Su = s each X,, during the denoising process. After sampling full trajectory, Diffuser executes the first action in the environment, and replans at the next state s'. In simple environments where replanning is unnecessary, the planned action sequence can be directly executed.
HIERARCHICAL DIFFUSER
While Diffuser has demonstrated competence in long-horizon planning and test-time flexibility, have empirically observed that its planned trajectories inadequately cover the dataset distribution, potentially missing high-return trajectories. Besides, the dense and flat planning scheme of the standard Diffuser is computationally expensive, especially when the planning horizon is long. Our key observation is that hierarchical planning could be an effective way to address these issues. To achieve this, we propose Hierarchical Diffuser, a simple yet effective framework that enables hierarchical planning while maintaining the benefits of diffusion-based planning. As shown in Figure it consists of two Diffusers: one for high-level subgoal generation (Section 3.1 and the other for low-level subgoal achievement (Section 3.2).
3.1 SPARSE DIFFUSER FOR SUBGOAL GENERATION
To perform hierarchical planning, the high-level planner needs generate sequence of intermediate states (gi.-... gg) that serve as subgoals for the low-level planner to achieve. Here, H denotes the planning horizon. Instead of involving complicated procedures for finding high-quality subgoals (Li) et al. 2023) or skills (Rajeswar& et al. 2023 Laskin et al., (02T), we opt for a simple approach that repurposes Diffuser for subgoal generation with minimal modification. In essence, we define the subgoals to be every K-th states and model the distribution of subsampled trajectories:
SD
Sp
SK
SHK
g#
(9)
ag
ak
a
ag
ag
a
3

---

Published as a conference paper at ICLR 2024
Trajectories
Trajectory Level Plan
Diffuser
Dataset D
Diffuser
Subgoals
High-Level Level Plan
+
Subsample
High-Level Sparse Diffuser
Hierarchical Diffuser
Parallel Training
Parallel Low-Level Generation
Low-LLvvl
Segment
Sparse Diffuser
Sub- cub-trajecttoiis
Denoising
Figure 1: Test and train-time differences between Diffuser models. Hierarchical Diffuser (HD) is general hierarchical diffusion-basee basson planning framework. Unlike the Diffuser's s training process (A, left), the HD's training phase reorganizes the training trajectory into two components: a sub-goal trajectory and dense segments. These components are then utilized to train the high-level and low-level denoising networks in parallel (B, left). During the lesting phase, in contrast to Diffuser (A, right), HD initially generates high-level plan consisted of sub-goals, which is subsequently refined through the low-level planner (B, right).
We name the resulting model Sparse Diffuser (SD). While using every K-th states as subgoalas is simplifying assumption, it is widely adopted in hierarchical RL due to its practical effective-
ness (Zhang et al., 2023 Hafner al., 2022 Li et al. 2022, Mandlekar et al. 2020 Vezhnevets et al. 2017a). We will empirically show that, desipite this simplicity, our approach is effective and
efficient in practice, substantially outperforming HDMI (Li al. 2023), state-of-the-arr method
that adaptively selects subgoals.
The training procedure of Sparse Diffuser is almost the same as Diffuser. The only difference is that we need to provide the subsampled data xSD to the diffusion probabilistic model p0 (xSD) and the guidance function Jon (xSD). is important to note that, although the guidance function uses the subsampled data as input, is still trained to predict the return of the full trajectory. Therefore, its gradient Vew will direct toward a subgoal sequence that part of high-return trajectories. However, due to the missing states and actions, the return prediction may become less accurate than Diffuser. In all of our experiments, we found that even if this is the case, it does not adversely affect task performance when compared to Diffuser. Moreover, dense actions SD can improve return prediction and, in some environments, further improve task performance. We provide a detailed description in Section Section 3.4 and ablation study in Section 4.3
investigation suggests that including
is worth noting that Sparse Diffuser can itself serve standalone planner, without the need to involve any low-level planner. This is because Sparse Diffuser can generate the first action ag of the plan, which is sufficient if we replan each step. Interestingly. Sparse Diffuser already greatly outperforms Diffuser, mainly due to its increased receptive field (Section 4.3). While the receptive field of Diffuser can also increased, this comes with hurting generalization performance and efficiency due the increased model size (Appendix and
3.2 FROM SPARSE DIFFUSER TO HIERARCHICAL DIFFUSER
While Sparse Diffuser can be used as a standalone planner, it only models the environment dynamics at coarse level. This is beneficial for generating high-level plan of subgoals, but it is likely that some low-level details are not taken into consideration. Therefore, we use a low-level planner to further refine the high-level plan, carving out the optimal dense trajectories that go from subgoal to the next. This also allows us to avoid per-step replanning when not necessary. We call this two-level model Hierarchical Diffuser (HD).
4

---

Published as a conference paper at ICLR 2024
Low-level Planner. The low-level planner is simply implemented as Diffuser po(x trained on trajectory segments x()) between each pair of adjacent subgoals g, Sik and gi+1 ((++)kk
(
SiK a
SiK+1 a.k+1
S(++D)K
OSi<H.
(10)
((+1))K
We also train a low-level guidance function To (x( that predicts the return R(x(1) for each segment. The low-level Diffuser and guidance function are both shared across all trajectory segments, and they can be trained parallel with the high-level planner.
them to perform hierarchical planning as follows. Given a starting state go. we first use the high-level planner to generate subgoals g1:H. This can be achieved by sampling from the perturbed distribution Poan (xSD) x P0 (xSD) exp (Tos (xSD)), and then discarding the actions. Since the actions generated by the high-level planner are not used anywhere, in practice remove the actions from subsampled trajectories xSD when training the high-level planner. In other words, we redefine
Hierarchical Planning. After training the high-level and low-level planners,
SD
iso
SK
SHK]
loo gi
gH
(11)
Next, for each pair of adjacent subgoals gi and g++1, trajectory that connects them, by sampling from the distribution (x(x (x(())) exp (Ta (xx))) To ensure that the generated (1) indeed has gi and g.+1 its endpoints, we set SiK g: and ((++1)K gi+1 in each denoising step during sampling. Importantly, all low-level plans (x(") Ji=0 ( can be generated in parallel. In environments that require per-step replanning, we only need to sample x()) ~ Po (x))), then execute the first action an in the environment, and replan at the next state. We highlight the interaction between the high-level and low-level planners in Appendix B
the low-level planner generate dense
3.3 IMPROVING RETURN PREDICTION WITH DENSE ACTIONS
Sparse Diffuser with Dense Actions (SD-DA). The missing states and actions in the subsampled trajectories xSD might pose difficulties in accurately predicting returns in certain cases. Therefore, we investigate a potential model improvement that subsamples trajectories with sparse states and dense actions. The hypothesis is that the dense actions can implicitly provide information about what has occurred in the intermediate states, thereby facilitating return prediction. Mcanwhile, the sparse states preserve the model's ability to generate subgoals. We format the sparse states and dense actions into the following two-dimensional array structure:
So
SK
SHK
go
gH
ag aк
aи
ag ак a
a
X-SD SD
aj
ag+1
aØK+1
aj
aK+1
aK-1
a2K
((++))K-1
aa--
82K-1
(12)
where ахик in the last column are included for padding. Training proceeds similarly as Sparse Diffuser, where train diffusion model paso (XSD-DA) capture the distribution of SD-DA in the offline dataset and guidance function Jospaa SDDDA to predict the return of the full trajectory.
Hierarchical Diffuser with Dense Actions (HD-DA). This is obtained by replacing the high- level planner in Hierarchical Diffuser with SD-DA. The subgoals are generated by sampling from Posom (xtonx(SSDDii (DSDDAA) x Posoan (x SD-DDA exp (Josoau (xSD.DA)), and then discarding the actions.
3.4 THEORETIC ANALYSIS
Theorem I in Appendix demonstrates that the proposed method can improve the generalization capability of the baseline. Moreover, our analysis also sheds light the tradeoff's in the value of K and the kernel size. With a larger value of K, it is expected to have a better generalization gap for the diffusion process but more loss of state-action details to perform RL tasks. With larger kernel size, we expect a worse generalization gap for the diffusion process but a better receptive field to perform RL tasks. See Appendix H for more details.
5

---

Published as a conference paper at ICLR 2024
Table 1: Long-horizon Planning. HD combines the benefits of both hierarchical and diffusion paaseesoo-- planning. achieving the best performance across all tasks, results are averaged over 100 planning seeds.
Flat Learning Methods
Hlerarchical Learning Methods
Enviromment
MPP1
IQI.
Diffuser
IRIS
HiGaC
HDMI
HD (Ouns)
Maze2I>
U-Maze
33.2
47.4
113.943..1
120.1++2.55
128.4+33.6
Maze2D
Medium
10.2
34.9
121.5:12.7
121.811.
135. 6.3.0
Maze2D
5.1
58.6
128.0:6.4
128.6:22.9
155.842.
Single-task Average
16.2
17.0
119.5
123.5
139.
Mulu2D
U-Maze
11.2
21.8
128
18
131. 3:::1.8
141.1+12
Multi2D>
Medium
15.4
12.1
127
127-244344
1311 1.9
140.2+1.
Mula2D
Large
8.0
13.9
132.
14:.88
185. 12:2.5
165.5
Multi-tusk Average
21.5
129.4
132.8
AntMuze
U-Maze
62.2
76.0:17.6
8.44-244
91.211.
94.0 0::4499
Amtaze
Medium
31.9+5.11
64.8=26
79.3+2255
88.7 7-8.1
Antaaz
Large
47.5
0.0++0.0
43.7-1.3 1.3
87.34-3.1
4-5.8
AntMaze Average
30.9
36.0
66.0
79.3
88.8
4 EXPERIMENTS
In experiment section, we illustrate how and why the Hierarchical Diffuser (HD) improves Diffuser through hierarcheial planning. We start with our main results on the D4RL (Fu et al., 2020) benchmark. Subsequent sections provide an in-depth analysis, highlighting the benefits of larger receptive field (RF) for diffusion-based planners for offlin RL tasks. However, our compositional ditto--distribution (OOD) task reveals that, unlike HD, Diffuser struggles to augment its RF without compromising the generalization ability. Lastly, we report HD's s efficiency in accelerating both the trainig time and planning time compared with Diffuser. The performance of HD across different K values is detailed in the Appendix C For the sake of reproducibility, we provide implementation and hyper-parameter details in Appendix
4.1 LONG-HORIZON PLANNING
We first highlight the advantage of hierarchical planning on long-horizon tasks. Specifically, we evaluate on Maze2D and AntMaze (Fu et al. 2020), two sparse-reward navigation tasks that can take hundreds of steps to accomplish. The agent will receive reward of 1 when it reaches fixed goal, and no reward elsewhere, making challenging for even the best model-free algorithms (Janner jet al. 2022b). The AntMaze adds to the challenge by having higher-dimensions. state and action space. Following Diffuser (Janner et al., 2022b), we also evaluate multi-task flexibility on Multi2D, a variant of Maze2D that randomizes the goal for each episode.
Results. As shown in Table Hierarchical Diffuser (HD) significantly outperforms previous state of the art across all tasks. The flat learning methods MPPI (Williams et al. (2016), IQL. (Kostrikov et al. 2022), and Diffuser generally lag behind hierarchical learning methods, demonstrating the advantage of hierarchical planning. In addition, the failure of Diffuser AntMaze-Large indicates that Diffuser struggles to simultaneously handle long-horizon planning and sigh--imensional state and action space. Within hierarchical methods, HD outperforms the non-diffusion based IRIS (Mandlekar et al.) 2020) and HiGoC (Li et al. 2022), showing the benefit of planning with diffusion in the hierarchical setting. Compared with the diffusion-baal extraction procedures and more advanced model architectures, HD achieves >00% performance gain on Maze2D-Large and Multi2D-Larg despite its simplicity.
HDMI (Li et al. (023) that uses complex subgoal
4.2 OFFLINE REINFORCEMENT LEARNING
We further demonstrate that hierarchical planning generally improves offline reinforcement learning even with dense rewards and short horizons. We evaluate on Gym-MuJoCo and FrankaKitchen (Fu al. 2020), which emphasize the ability to learn from data of varying quality and to generalize to unseen states, respectively. We use HD-DA as outperforms HD in the dense reward setting. In addition Diffuser and HDMI, we compare to leading methods in each task domain, including model-free BCQ (Fujimoto et al. 2019), BEAR (Kumar et al., 2019). CQL (Kumar et al. 2020).
IQL. (Kostrikov et al. 2022). Decision Transformer (DT; Chen et al. 202T). model-based MoRel. Ki

---

Published as a conference paper at ICLR 2024
Table 2: Offline Reinforcement Learning. HD-DA achieves the best overall performance. Results averaged
over 5 planning seeds. Following Kostrikov (2022), we emphasize bold scores within 5% of maximum.
Gym Tasks
CQL
MOReL
Diffuser
HDMI
HD-DA (Ours)
Med-Expert
HaltChectah
55.2
91.6 6
86.7
86.8
95.0
53.3
88, 0..
92.1+11
92 02.
Med-Expert
Hopper
52.5
105-4
91.5
107.6
110.0
108.7
103.3+1.
113.5+0.9
115.3+1.1
Med-Expert
Walker2d
107.5 5
108.5 8
109. 6
108.1
101.9
95.6
106.9=0.2
107.9
107.1 0.1
Mediam
HalfChootah
42.6
41.0
47.3
12.6
46.9
12.1
42.8
48.0=0.9
16. 0.2
Mediam
Hopper
52.9
58.5
06.3
67.6
61.1
96
74.3 1.4
76.4+2.6
99 0-3
Medium
Walker2d
75.3
0.0
Med-Roye Med-Roplay
HalfChoctah
36.6
45.5
44.2
36,6
11.9
40.2
37.7=0.5
44.9=2.0
38, 0.77
Med-Replay
Hopper
18.1
95.0
94.7
82.7
93.6
93.6+0.4
99.6+1.5
94.7-+0.7
Med Replay
Walker2d
26.0
77.2
73.9
66.6
82.6
49.8
1.6
80.7 121
84.1.12.
Average
51.9
77.6
77.0
74.7
78.9
72.9
17.5
82.6
84.6
Kitchen Tasks
BC
BCQ
BEAR
CQL
IQL
Ry8-G
Difluser
HDMI
HD-DA (Ours)
Partial
FrankaKitchen
33.8
18.9
13.1
19.8
16.3
16.5
36.2
5.4
73.3+114
Mixed
FrankaKitchen
47.5
8.1
47.2
51.0
51.0
40.0
30.0 1 8.8
69.2
71.7:127
Average
40.7
13.5
30.2
50.4
48.7
43.3
53.1
72.5
Table 3: Ablation Model Variants. SD yields improvement Diffuser, and the incorporation of low-level refinement HD provides further enhance- ment in performance compared to SD.
Table 4: Guidance Function Learning. The included dense action helps learn guidance function, resulting in better RI. performance.
J
Performance
Dataset
Dataset
Diffuser
SD
HD
HD
HI-DA
HD
HI-DA
Gym MuJeCo
77.5
80.7
Hopper
101.7
88.8
93.1=3.1
94.7+007
Maze2D
119.5
133.4
139.9
Walker2d
106.1
133.0
77.2+3.3
84.14-2.2
Multi2D
HalfChectah
228.5
208.2
37.5=1.7
38 18..7
dambi et al. 2020), Trajectory Transformer (TT: Janner et al. 2021 Supervised Learning (RvS: Emmons al. 2022).
and Reinforcement Learning
Results. As shown in Table HD-DA achieves the best average performance, significantly outper- forming Diffuser while also surpassing the more complex HDMI Notably, HD-DA obtains >35% improvement on FrankaKitchen over Diffuser, demonstrating its superior generalization ability.
4.3 ANALYSIS
To obtain deeper understanding on HD improvements over Diffuser, we start our analysis with ablation studies on various model configurations. Insights from this analysis guide us investigate the impact of effective receptive field on RL performance, specifically for diffusion- based planners. Furthermore, introduce compositional compositional generalization capabilities. We also evaluate HD's performance on varied jumpy step K values test its robustness and adaptability.
(OOD) task to demonstrate HD's 's
SD already outperforms Diffuser. HD further improves SD via low-level refinement. This can be seen from Table where we report the performance of Diffuser, SD, and HD averaged over Maze2D, Multi2D, and Gym-MuJoCo tasks respectively. As mentioned in Section 3.1 here we use SD as standalone planner. the following, investigate potential reasons why SD outperforms Diffuser.
Large kernel size improves diffusion- planning for in-distribution tasks. A key difference between SD and Diffuser that the subsampling SD increases its effective receptive field. This leads us to hypothesize that a larger receptive field may be beneficial for modeling the data distribution, resulting better performance.
test this hypothesis, we experiment with different kernel sizes of Diffuser, and report the averaged performance on Maze2D, Multi2D, and Gym-MuJoCo in Figure We find that Diffuser's performance generally improves as the kernel size increases up to a certain threshold. (Critical drawbacks associated with increasing Diffuser's kernel sizes will be discussed in detail in the subsequent section.) Its best performance is comparable to SD, but remains inferior to HD. In Figure we further provide qualitative comparison of the model's coverage of the data distribution. We plot the actual executed trajectories when the agent follows the model-genene generated plans. Our results show that HD is able generate plans that cover all distinct paths between the
7

---

Published as a conference paper at ICLR 2024
start and goal state, exhibiting distribution closely aligned with the dataset. Diffuser has much worse coverage the data distribution, but can improved with a large kernel size.
HD
SD
Tirluser
HD
SD
Tiilaser
HD-DA
SD-DA
Diiffeer
Paformae -MaelD
140
Peffommne -MileD
150
Perfommnce Gym
130
140
120
130
10
Kemal Size
Kemel Size
Kenel Size
Figure 2: Impact of Kernel Size. Results of the impact of kernel size on performance of Diffuser in offline RI. indicates reasonably enlarging kernel size can improves performance.
Training Dataset
Diffuser
Diffuser - Large Kernel Size
HD
Figure 3: Coverage of Data Distribution. Empirically, we observed that Diffuser exhibits insufficient coverage of the dataset distribution. We illustrate this with an example fcaturing three distinct paths traversing from the start to the goal state. While Diffuser struggles to capture these divergent paths, both our method and Diffuser with increased receptive field successfully recover this distribution.
Large kernel size hurts out-of-distribution generalization. While increasing the kernel size appears to be a simple way to improve Diffuser, has many drawbacks such as higher memory consumption and slower training and planning. Most importantly, it introduces more model parameters, which adversely affect the model's generalization capability. We demonstrate this model produce novel plans between unseen pairs of start and goal states at test time, by stitching together segments of training trajectories. We report the task success rate in Table as well as the discrepancy between generated plans and optimal trajectories measured with cosine similarity and mean squared error (MSE). HD succeeds in all tasks, generating plans that are closest to the optimal trajectories, while Diffuser variants fail this task completely. Details can be found Appendix
task that requires the
Table 5: cut-Or-Distributti (OOD) Task Performance. Only Hierarchical Diffuser (HD) can solve the compositional OOD task and generate plans that most close to the optimal.
Metrics
Diffuser-KSt Diffuser-KS13
DiffuserKSS19
Diffuser-KS2
Successful
0.0%
0.0%
0.0%
0.0%
100.0%
Cosine Similarity
0.85
0.89
0.93
0.93
0.98
Deviation (MSE)
1269.9
1311.1
758.5
1023.2
198.2
Effect of Dense Actions. Though the dense actions generated from high-level planer are discarded the low-level refinement phase, we empirically find that including dense actions facilitates the learning of the guidance function. As shown Table validation loss of guidance fuction learned from HD-DA is lower than that of SD-SA, leading to better RL performance. We conduct the experiment on the Medium- Redium-Reppaay dataset where learning the value function is hard due to the mixed policies.
Efficiency Gains with Hierarchical Diffuser. A potential concern when introducing an additional round of sampling might be the increase planning time. However, the high-level plan, being K times shorter, and the parallel generation of low-level segments counteract this concern. In Table we observed a 10x speed up over Diffuser in medium and large maze settings with horizons beyond 250 time steps. Details of the time measurement in Appendix D
8

---

Published as a conference paper at ICLR 2024
Table 6: Wall-clock Time Comparison. Hierarchical Diffuser (HD) is more computationally efficient compared Diffuser during both training and testing stages.
Training
Planning
Environment
U-Maze
Med-Maze
L-Maze
MuloCo
U-Maze
Med-Maze
L-Maze
MuloCo
HD Diffuser
8.0
8,7
8.6
9.9
0.8
3.1
3.3
1.0
132.7
119.7
12.3
RELATED WORKS
Diffusion Models. Diffusion models have recently emerged as a new type of generative model that supports generating samples, computing likelihood, and flexible-model complexity control. In diffusion models, the generation process is formulated as an iterative denoising process [Sohl=
Dickstein et al. (2015); Ho et al. (2020). The diffusion process can also be guided to a desired direction such as to specific class by using either classifier-bascd guidance Nichol et al. (202I) or classifier-free guidance Ho & Salimans (2022). Recently, diffusion models have been adopted for agent learning. Janner et al. (2022b) have adopted it first and proposed the diffuser model which is the non-hierarchical version of our proposed model, while subsequent works by Ajay et al. (2022) : Lu
al. (2023) optimized the guidance sampling process. Other works have utilized diffusion models specifically for RL Wang et al. (202)); Chen et al. (2023), observation coenn-to-acceee to-action imitation modeling Pearce et al. (2022). and for allowing equivariance with respect to the product of the spatial symmetry group Brehmer et al. (2023). A noteworthy contribution in this field is the hicrarchical diffusion- planning method Li et al. (2023), which resonates closely with our work but distinguishes itself in the subgoal preprocessing. While it necessitates explicit graph searching, our high-level diffuser to discover subgoals automatically.
Hierarchical Planning. Hierarchical planning has been successfully employed using temporal generative models, commonly referred as world models Ha Schmidhuber (2018);
(20udh |Hffner et al. (2019). These models forecast future states or observations based historical states and actions. Recent years have seen the advent of hierarchical variations of these world models Chung et al. (2017): Kim al. (2019); : (axena al. (2021). Once trained, world model can be used to train separate policy with rollouts sampled from it Hafner et al. (2019); Deisenroth & Rasmussen (011 Ghugare et al. (2023); Buckman et al. Harnerr al. (2022), it can be leveraged for plan searching Schrittwieser et al. (2020); Wang & Ba (2020); Pertsch et al. (2020); Hu et al. (2023):
Zhhu et al. (2023). Our proposed method draws upon these principles, but also has connections to hierarchical skill- bkile-based planning such as latent skill planning Xie et al. (2020); Shi et al. (2022)
However, a crucial distinction of our approach lies in the concurrent generation of all timesteps of plan, unlike the aforementioned methods that require a sequential prediction of future states.
6 CONCLUSION
We introduce Hierarchical Diffuser, comprehensive hierarchical framework that leverages the strengths of both hierarchical reinforcement learning and diffusion- planning methods. Our approach, characterized by a larger receptive field at higher levels and fine-tuning stage at the lower levels, has the capacity to not only capture optimal behavior from the offline dataset, but also retain the flexibility needed for compositional out-o--distribution (OOD) tasks. Expanding our methodology to the visual domain, which boasts broader range of applications, constitutes another potential future direction.
Limitations Our Hierarchical Diffuser (HD) model has notable strengths but also presents some limitations. Foremost among these its dependency on the quality of the dataset. Being an offline method, the performance of HD is restriced by the coverage or quality of datasets. In situations where encounters unfamiliar trajectories, HD may struggle to produce optimal plans. Another restriction is the choice of fixed sub-goal intervals. This decision simplify the model's architecture but might fall short in handling a certain class of complex real-world scenarios. Furthermore, it introduces a task-dependent
Lastly, the efficacy of HD is tied to the accuracy of the learned value function. This relationship places limits on the magnitude of the jump steps K; excessively skipping states poses challenge to learn the value function.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work is supported by Brain Pool Plus Program (No. 2021H1D33A24031033645) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT. We would like to thank Michael Janner and Jindong Jiang for insightful discussions.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Johann Brehmer, Joey Bose, Pim De Haan, and Taco Cohen. EDGI: Equivariant diffusion for planning with embodied agents. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023. URL httpss//peeeeevvvw..eeeeo////dsd
Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, and Honglak Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. Advances in neural information processing systems, 31, 2018.
J. Carvalho, A.T. Le, M. Baierl, D. Koert, and J. Peters. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023. URL http:://peeeeevieeweeeeeeomnnddd
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. arXiv preprint arXiv:2 00000001344,. 2021.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. International Conference Learning Representations, 2017.
André Correia and Luis A Alexandre. Hierarchical decision transformer. In 2023 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pp. 1661-1666. IEEE, 2023.
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data- tata-efficient approach to policy search. In Proceedings of the 28th International Conference machine learning (ICMI-11), pp- 465-472 2011.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for
offline RL via supervised learning? In International Conference on Learning Representations, 2022. URL http:////weeeeeeeeeeeeeeteom/on/
Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, 2019.
Raj Ghugare, Homanga Bharadhwaj, Benjamin Eysenbach, Sergey Levine, and Russ Salakhutdinov. Simplifying model- RL: Learning representations, latent-space models, and policies with one objective. In The Eleventh International Conference on Learning Representations, 2023. URL
10

---

Published as a conference paper at ICLR 2024
David Ha and Jürgen Schmidhuber. World models. arXiv preprint arXiv:1803.10122 2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. arXiv preprint arXiv: 10211104551, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint /9X1v:992.001600 2019.
Danijar Hafner, Kuang- Huci Lee, lan Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. arXiv preprint arXiv:2206.08 00....0//444/ 2022.
Jessica B Hamrick, Abram L Friesen, Feryal Behbahani, Arthur Guez, Fabio Viola, Sims Witherspoon, Thomas Anthony, Lars Buesing, Petar Veličković, and Théophane Weber. On the role of planning in model-based deep reinforcement learning. arXiv preprint arXiv:2011.0422,, 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Edward S. Hu, Richard Chang, Oleh Rybkin, and Dinesh Jayaraman. Planning goals for explo- ration. In The Eleventh International Conference on Learning Representations, 2023. URL
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL http::///ooeeoovvvveeeeeoouu//la
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022a.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for ficxible behavior synthesis. arXiv preprint arXiv:2205.0999 2022b.
Kenji Kawaguchi, Zhun Deng, Kyle Luh, and Jiaoyang Huang. Robustness implies generalization via data-dependent generalization bounds. In International Conference on Machine Learning, pp- 10866-10894. PMLR, 2022.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International Conference on Machine Learning (ICML), 2023.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33: 21810-21823, 2020.
Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. ICML Workshop on Generative Modeling and Model-Based Reasoning for Robotics and A1, 2019.
Craig A Knoblock. Learning abstraction hierarchies for problem solving. In AAAI, pp. 923-928, 1990.
Iyya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with im- plicit Q-learning. In International Conference on Learning Representations, 2022. URL httpss//peeeevvveveee/eouuu//lss
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping enror reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 179-11999 2020.

---

Published as a conference paper at ICLR 2024
Yaqing Lai, Wufan Wang, Yunjie Yang, Jihong Zhu, and Minchi Kuang. Hindsight planner. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 690-699-69 1900698,400 2020.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:21 2021.
Jinning Li, Chen Tang, Masayoshi Tomizuka, and Wei Zhan. Hierarchical planning through goal- conditioned offline reinforcement learning. IEEE Robotics and Automation Letters, 7(4): 10223, 2022.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, 2023.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guider diffusion sampling in offline reinforcement learning. arXiv preprint arXiv:2304. 12824, 2023.
Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4414-4420 2020. doi: 10.1 0..11999JCR.499955220000099999
Thomas M Moerland, Joost Broekens, Aske Plaat, Catholijn M Jonker, al. Model-based rein- forcement learning: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-118, 2023.
Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv; 1909::999..112 2019.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:2 12. 2021.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, David Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. Deep Reinforcement Learning Workshop NeurlPS 2022, 2022. URL httpss///eeeeeeeeeeeeeee//mmnndp
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Shenghao Zhou, Dinesh Jayaraman, Chelsea Finn, and Sergey Levine. Long-horizon visual planning with goal-conditioned hierarchical predictors. Advances in Neural Information Processing Systems, 33:17321-17333 2020.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoe Le. Combined scaling for open- vocabblcyy image classification. arXiv preprint arXiv:2111.1008 2021. doi: (0-48500/RXXI22111.1000 URL httpss//ariv.org/aass2111...
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexandre Lacoste. Mastering the unsupervised reinforcement learning benchmark from pixels. In International Conference on Machine Learning, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint 27X1v:2204.06/25, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684- 10695, 2022.
O. Ronneberger, P. Fischer, and T. Brox. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister Intervention (MICCAI), volume 9351 of LNCS, pp. 234 241. Springer, 2015.
12

---

Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy net- works. In International Conference on Learning Representations, 2020. URL http:///eeeeeeeeeeeeeeeeoo///eee Published as a conference paper at ICLR 2024
Earl D Sacerdoti. Planning in hierarchy of abstraction spaces. Artificial intelligence, 5(2):115-135, 1974.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic tex--t--mmaee diffusion models with deep language understanding. arXiv preprint arXiv:2205. 11487, 2022.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders. Advances in Neural Information Processing Systems, 01222246-222577 2021.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-609 2020.
Lucy Xiaoyang Shi. Joseph J. Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. Conference on Robot Learning, 2022.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and search. nature, 529(7587):484-489, 2016.
Satinder P Singh. Reinforcement learning with a hierarchy of abstract models. In Proceedings of the National Conference on Artificial Intelligence, number 10, pp. 202. Citeseer, 1992.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, PMLR, 2015.
pp.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differen- tial equations. In International Conference on Learning Representations, 2021. URL
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Erik Talvitie. Model regularization for stable sample rollouts. In UAI, pp. 780-789, 2014.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal networks for hierarchical reinforcement learning. In International Conference on Machine Learning. 2017a
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 3540-3549, PMLR, 2017b.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.061 2022.
Grady Williams, Paul Drews, Brian Goldfain, James M Rebg. and Evangelos A Theodorou. Aggres- sive driving with model predictive path integral control. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1433 1440. IEEE, 2016.
Kevin Xie, Homanga Bharadhwaj. Danijar Hafner, Animesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer. arXiv preprint arXiv:2011.13897 2020.
Jingwei Zhang, Jost Tobias Springenberg, Arunkumar Byravan, Leonard Hasenclever, Abbas Ab- dolmaleki, Dushyant Rao, Nicolas Heess, and Martin Riedmiller. Leveraging jumpy models for planning and fast learning in robotic domains. arXiv preprint arXiv:2302. 2023.
Jinhua Zhu, Yue Wang. Lijun Wu, Tao Qin, Wengang Zhou, Tie- Yan Liu, and Houqiang Li. Making better decision by directly planning in continuous control. In The Eleventh International Conference Learning Representations, 2023. URL. htw.t/////poeeeeeeeeee////eeeeea
13

---

Published as a conference paper at ICLR 2024
This appendix provides detailed elaboration several aspects of our study. In Section outline our implementation procedure and the hyper-parameter settings used. Section B provides pseudocodes illustrating the processes of planning with Hierarchical Diffuser. We examine the robustness of HD with various values in Section The Out-of-distribution (OOD) visualizations and the corresponding experiment details outlined in Section Section Dection[]explainggg details of the wall clock measurement. Finally, starting from Section we present our theoretical proofs.
A IMPLEMENTATION DETAILS
In this section, describe the details of implementation and hyperparameters used during our experiments. For the Out-of-distribution experiment details, please check Section
We build our Hicrarchical Diffuser upon the officially released Diffuser code obtained from We list out the changes made below.
In our approach, the high-level and low-level planners are trained separately using segments randomly selected from the D4RL offline dataset.
For the high-level planner's training, we choose segments equivalent in length to the planning horizon, H. Within these segments, states every steps are selected. In the dense action variants, the intermediary action sequences between these states are then flattened concatenated with the corresponding jumpy states along the feature dimension. This approach trajectory representation is also employed in the training of the high-level reward predictor.
The sequence modeling at the low-level is the same as Diffuser except that using sequence length of K +
We set K = 15 for the long-horizon planning tasks, while for the Gym-MuJoCo, we use K = 4.
Aligning closely with the settings used by Diffuser, we employ planning horizon of H = 32 for the MuJoCo locomotion tasks. For the Maze2D tasks, we utilize varying planning horizons; H 120 for the Maze2D UMaze task, H 255 for the Medium Maze task, and H = 390 for the Large Maze task. For the AntMaze tasks, we set H 225 for the UMaze, H 255 for the Medium Maze, and H 450 for the Large Maze.
For the MuJoCo locomotion tasks, select the guidance scales from set of choices, (0.1, 0.010 0.001, 0.0001), during the planning phase.
B PLANNING WITH HIGH-LEVEL DIFFUSER
We highlight the high-level planning and low-level planning in Algorithm | and Algorithm 2 respectively. The complete process of planning with HD is detailed in Algorithm
B.1 PLANNING WITH HIGE-LEVEL DIFFUSER
The high-level module, Sparse Diffuser (SD), models the subsampled states and actions, enabling operate independently. We present the pseudocode of guided planning with the Sparse Diffuser in Algorithm
B.2
PLANNING WITH LOW-LEVEL DIFFUSER
Given subgoals sampled from the high-level diffuser, segments of low-level plans can be generated concurrently, illatrate generating one such segment as example Algorithm
B.3 HIERARCHICAL PLANNING
The comprehensive hierarchical planning involving both high-level and low-level planners is outlined in Algorithm 3 For the Maze2D tasks, employed open-loop approach, while for more challenging environments like AntMaze, Gym-MuloCo, and Franka Kitchen, a closed-loop strategy was adopted.
14

---

Published as a conference paper at ICLR 2024
Algorithm High-Level Planning
1: function S.MPLEHHGHLLEEEPLLANNCCCC
State Sparse Diffuser Mas guidance function
Toso guidance scale w, variance ) initialize plan xDD (N
2:
3: for m M
do
4:
j - M8 SD + wo VDDD Jor SD) SD
5:
Xm-1 ( (
6: Fix go in X mn to current state
7: end for
8: return High-level plan x
9:
end function
Algorithm 2 Low-Level Planning
1: function LAEELEL WWW.:ELLLOWLEEEPLLASSSSSSSS PLAN( (yi, 11+1), low-level diffuser pas low-level guid- ance function Jg. guidance scale w, variance om
2: Initialize all low-level plan x ~ I),
3: for m M
1 do
4: j + pa ( 11) wo Vxi, (x(
5:
x x ~ (2,, 2
6:
Fix So in x to gi: Fix SK in x to gi+1
7:
end for
8: return low-level plan x/
9:
end function
C ABLATION STUDY ON JUMPY STEPS K
In this section, report the detailed findings from ablation study concerning the impact of the parameter K in Hierarchical Diffuser. The results, which are detailed in Tables Sables/annc Jnnd correspond to Maze2l) tasks and MuloCo locomotion tasks, respectively. As increased K, initial enhancement in performance was observed. However, subsequent performance decline was noted with larger K values. This trend aligns with our initial hypothesis that a larger K introduces more skipped steps at the high-level planning stage, potentially resulting in the omission of information necessary for effective trajectory modeling, consequently leading performance degradation.
Table 7: Ablation K Maze2D. The model's 's performance increased with the value of K up until K 21. We report the mean and standard error over 100 random seeds.
Environment
K1 (Diffuser default)
HD-K7
HD-K15 (default)
HD-K21
Maze2D
U-Maze
113.9 +
127.0 + 1.5
128.4 + 3.6
121.0 + 2.1
Maze2D
Medium
121.5 2.7
132.5
1.3
135.6 + 3.0
130.3
2.4
Maze2D
Large
123.0 6.4
153.2
3.0
155.8 + 2.5
158.9
2.0
Sing-task Average
119.5
137.6
139.9
137.7
Multi2D
U-Maze
128.9 + 1.8
135.4 1.1
144.1 + 1.2
133.7
1.3
Multi2D
Medium
127.2 3.4
135.3
1.6
140.2 + 1.6
134.5
+ 1.4
Multi2D
Large
132.1 1. 5.8
160.2 1 1.9
165.5 1 0.6
159.3
3.0
Multi-task Average
129.4
143.7
149.9
142.5
D WALL CLOCK COMPARISON DETAILS
We evaluated the wall clock time by averaging the time taken per complete plan during testing and, for the training phase, the time needed for 100 updates. All models were measured using single NVIDIA RTX 8000 GPU to ensure consistency. We employ the released code and default settings for the Diffuser model. We select the Maze2D tasks and Hopper- Expert, representative for
15

---

Published as a conference paper at ICLR 2024
Algorithm 3 Hierarchical Planning
1: function WAMPLEEEHEBARCHCAAAAAAAAAII
diffuser Mams low-level diffuser μa. high- level guidance function Jos low-level guidance function Jg. high-level guidance scale WSD. low-level guidance scale w, high-level variance SD... low-level variance o
2:
Observe state s:
3: if do open-loop then
4:
Sample high-level plan x = SAMPLEHICHLEEEELLLAAN Mosn Jesn WSD: 0 DDDm 5: for i - 0.... H 1 parallel do Sample low-level plan x( ( SAMILELLOWLLVLLLLAN((O.... end for
6:
peo Jo. w, mm))
7:
8:
Form the full plan x with low-level plans x(() for = 0, for action in x do Execute a
9:
10:
11:
end for
12:
else
13:
while not done do
14:
Sample high-level plan x = SAMPLEHIGHLEVELLLANY Milso Jo. WSD, o²D.m)
// Sample only the first low-level segment Sample x (0) =
16:
LEA.LLLLLWLELLLLA((((O 91), p4s Jop w, o
17:
Execute the first of plan (0)
Observe state
19:
end while
20:
end
21:
end function
Table 8: Ablation on K MuJoCo Locomotion. The model's performance increased with value up until K 8. We report the mean and standard error over 5 random seeds.
Dataset
Environment
Kl (Diffuser default)
HD-K4 (default)
HD-K8
Medium-Esse Exprium-Expprtt.
HalfChcetah
88.9 0.3
92.5 + 0.3
91.5 + 0.3
Medium- Medium-Expert
Hopper
103.3 1.3
115.3
113.0
Medium- Expert
Walker2d
106.9 1 0.2
107.1 1 0.1
107.6 1 0.3
Medium
HalfChectah
42.8 + 0.3
46.7 0.2
45.9
Medium
Hopper
74.3 + 1.4
99.3 + 0.3
86.7
Medium
Walker2d
79.6 1: 0.6
84.0 0.6
84.2 0.5
Bedduum-Replan
HalfCheetah
37.7 0.5
38.1 0.7
39.5 1 0.4
Medium Replay
Hopper
93.6 + 0.4
94.7
91.3 1.3
Medium Replay
Walker2d
70.6 + 1.6
84.1 + 2.2
76.4
Average
77.5
84.6
81.8
the Gym-MuloCo tasks, from the D4RL. benchmark for our measurement purpose. On the Maze2I tasks, we set K 15, and for the Gym-MuJoCo tasks, set it to 4 as this is our default setting for RL tasks. The planning horizons of HD for each task, outlined in Table 9 are influenced by their need for divisibility by K, leading to slight deviations from the default values used by the Diffuser.
Table 9: Wall-clock time H value
Environment
Diffuser
Ours
Maze2d- Mazzee-Umaze
128
120
Maze2d-Medine
256
255
Maze2d-Lane
384
390
Hopper-Mede Medium-Exxerrt
32
32
16

---

Published as a conference paper at ICLR 2024
E COMPOSITIONAL OUT-OF-DISTRIBUTIN (OOD) EXPERIMENT DETAILS
While an increase in kernel size does indeed provide performance boost for the Diffuser model, this enlargement inevitably augments the model's capacity, which potentially increases the risk of overfitting. Therefore, Diffuser models may underperform on tasks demanding both a large receptive field and strong generalization abilities. To illustrate this, inspired by Janner et al. (2022a), designed compositional disti--diittiuttion (OOD) Maze2D task, as depicted in Figure 4 During training, the agent only exposed offline trajectories navigating diagonally. However, during testing, the agent required to traverse between novel start-goal pairs. We visualized the 32 plans generated by the models Figure 4 As presented in the figure, only the Hierarchical Diffuser can generate reasonable plans approximating the optimal solution. In contrast, all Diffuser variants either create plans that lead the agent crossing wall (i.e. Diffuser, Diffuser-KS133 and Diffuser- KS19) or produce plans that exceed the maximum step limit (i.e. Diffuser-13, 13, Diffuser-KS19 and Diffuser-KS25
To conduct this experiment, we generated a training dataset of 2 million transitions using the same Proportional-Derivaae (PD) controller as used for generating the Maze2D tasks. Given that an optimal path typically requires around 230 steps to transition from the starting point to the end goal, we set the plamning horizon for the Diffuser variants at 248, while for our proposed method, we set it at 255, to ensure divisibility by K 15. For the reinforcement learning task in the testing phase, the maximum steps allowed were set at 300. Throughout the training phase, we partitioned mitigate the risk of overfitting. To quantitatively
10% of the training dataset as validation measure the discrepancy between the generated plans and the optimal solution, we used Cosine Similarity and Mean Squared Error (MSE). Specifically, we crafted 10 optimal paths using the same controller and sampled 100 plans from each model for each testing task. To ensure that the optimal path length aligned with the planning horizon of each model, we modified the threshold distance used to terminate the controller once the agent reached the goal state. Subsequently, we computed the discrepancy between each plan and each optimal path. The mean of these results was reported in Table 5
Training Data
Test Time Planning
Diffuser
Diffuser K513
Diffuser
Diffuser KS25
Figure 4: Large Kernel Size Hurts the OOD Generalization. Increasing kernel size generally improves the offline RI. performance Diffuser model. However, when large receptive field and compositional out-of- distribution (OOD) generalization are both required, Diffuser models offer no simple solution. We demonstrate this with the sampled plans from both the standard Difuser and Difuser with varied kernel sizes (KS). None of them can come up with optimal plan by stiching training segments together. Conversely, our proposed Hierarchical Diffuser (HD) posseses both large receptive field and the flexibility needed of compositional OOD tasks.
F OOD GENERALIZATION IN MOTION PLANNING
We carried out an additional experiment focused on out-of-distrinn distriibittt (OOD) scenarios, assessing the model's capability navigate through unseen obstacles. Following the methodology of MPD)
17

---

Published as a conference paper at ICLR 2024
applied our HD model to the PointMass2D Dense task with OOD obstacles. To do this, we replaced the flat Diffuser planner used in MPD with our HD model, referred to as MP-HD. We used the official code of MPD for comparison.
(Carvalho et al.
2023
Our HD model achieved a noteworthy success rate of 81.0 = 38.8, surpassing MPD's performance of 75.0 + 43.3. We also visualized sample trajectories of HD from two randomly selected (start, goal) pairs in Figure These trajectories demonstrate that our model can effectively avoid collisions when faced with OOD test obstacles.
Figure 5: Sample Trajectories with Unseen Obstacles. HD generate multiple paths navigating from two randomly selected start and goal states. Obstacles in red were not present during training. We marked sub-goal states in eyan for clarity. We marked sub-goal states eyan for clarity. The trajectories begin at the green circle, with the target represented by purple circle.
G ADDITIONAL ABLATION STUDIES
G.1 TRANSFORMER BASED DIFFUSION
We compare our model (based on U-Net (CNN)) with connsfommerbbssee diffusion in this section. For this experiment, we use the hyperparameter setting the Decision Transformer (Chen al.. 2021) starting point for our investigation. The results, as shown in the Table 10 reveal that the Transsssfrrmer achieves similar performance to the HD-UNet in Maze2D tasks, though is slightly less effective in the Gym-MuloCo tasks. While the TID-Trasfformrr shows promise, would like to emphasize that our primary contribution is not the backbone architecture but the benefits of hierarchical structures.
G.2 SUB-GOAL SELECTION STRATEGIES
Hierarchical Diffuser (HD) select sub-goals with fixed time interval for simplicity. Here, we consider other choises:
Route Sampling (Lai et al. 2020) (RS): In line with HDMI, we also consider choosing waypoint with fixed length interval as sub-goals. Specifically, denote the distance moved after action a, as de. Then, the route length can be computed waypoints with fixed interval of S/k, where le is the number of sub-goals.
5 Σ11 & We pick the
Value Sampling (Correia & Alexandre 2023) (VS): Also inspired by HDMI, we also test the value sampling method, where the most valuable states are chosen as sub-goals. Specifically, the distance weighted accumulated reward is used to value each states after state Ws;;)= E4
Future Sampling (Andrychowicz et al. 2017) (FS):
Beyond RS and VS, we also explored hindsight heuristic method, randomly selecting future states sub-goals.
18

---

Published as a conference paper at ICLR 2024
Table 10: Ablation Study Backhone Architecture. Transfororre achieves comparable with HD-Uner on wide rang tasks. We report the mean and standard eroor over random seeds.
Task
HD-UNet
HD-Transformer
Maze2d-Large
128.4
3.6
127.9 3.2
Maze2d- Maze2d-Medium
135.6 + 3.0
136. 1 2.6
Maze2d-UMe
155.8 2.5
154.1 3.6
Mazeld Average
139.9
139.4
HalfCheetah
92.5 +: 0.3
88.4 = 0.6
MedExp-Hopper
115.3 + 1.1
103.9 = 5.9
Medexp-Walker2d
Walker2d
107.1 + 0.1
107.0 0.3
Modium-HalfChectall
46.7 + 0.2
45.3 +.3++00 0.5
Medium-Hopoe
99.3 + 0.3
94.0
Medium--alker22)
84.0 : 0.6
82.8 = 1.7
MedRep-Haae HalfCheetah
38.1 + 0.7 0.7
39.5 0.2
McdRep-Hopper
94.7 + 0.7
91.4 + 1.5
MedRep-Walker2d Walker2d
84.1 + 2.2
81.2 + 1.1
Gym Average
84.6
81.5
Notably, in RS and VS, certain states might never be chosen sub-goals, unlike FS and the fixed time interval sampling (TS) used in HD, which offers equal probability for each state to be selected as sub-goal.
Given the varying lengths of sub-tasks generated by these selection methods, integrating dense action at the high level was impractical. Hence, we focused our experiments on HD rather than HD-DA. At the low level, sub-trajectories were padded to a consistent length It's important to note that excluding dense action data at the high level may slightly hinder the learning of the value function, potentially leading to a marginal decrease in performance. The results, as presented in the Table [1] demonstrate that hierarchical framework is generally resilient across different sub-goal goal selection methods. While HD-VS and HD-RS exhibited somewhat lower performance, we hypothesize this may be due uneven sampling of valuable states, which could impact the planning guidance function's s effectiveness.
Table 11: Ablation Study on Sub-goal Selection. HD is generally resilient across different sub-goal selection methods, We report the mean and standard error over 5 random seeds.
Dataset
HD-DA
HD
HD-FS
HD-VSS
HD-RS
92.5 + 0.3
92.1 0.5
87.6 + 0.7
87.6 0.6
88.4 0.4
MedExx--opper
115.3 + 1.1
101.1 8.2
106.5 + 5.5
108.9 4.8
106.1 107.4 0.3
Walker2
0.1
107.4 0.3
107.0 + 0.1
107.4 0.2
46.7 46.7 0.2
66.2+00.2
43.9 0.4
43.2 43.2+0033 0.3
43.6 + 0.9
Medium-Hoppee Hopper
99.3 0.3
09.2
100.9 + 0.8
92.3 12.3-:0 4.2
95.8 + 1.3
Medium- Walker2d
84.0 + 0.6
0.8
83.1 + 1.0
0.9
82.9 + 1.1
MedRRe--Halfehcethh
38.1 0.7
37.5
39.7 + 0.3
3.7
38.1 0.8
MedRep-Hopper
94.7 + 0.7
93.4 3.1
90.9 + 1.7
91.3
92.6 + 1.2
MedRep- Walker2d
84.1 + 2.2
77.2 3.3
80.9 +
2.1 75.7
76.4 +
Average
84.6
82.1
82.3
80.8
81.3
H THEORETICAL ANALYSIS
In this section, show that the proposed method improve the generalization capability when compared to the bascline. Our analysis also sheds light on the tradeoffs in and the kernel size. Let
K (1, .1, T), f(x) TE [llc col co(vam vI Cm m]][ where 0 is arbitrary
normalization coefficient that can depend on K: e.g., 1/d where d the dimensionality
Given
the training trajectory data (0))) 1 the training loss is defined by (() n Σ ((x()) where
19

---

Published as a conference paper at ICLR 2024
(1) V Om x + II Omm,, and Xo ..... X are independent samples of trajectories. We have [()) - Ex [[(x))] Define Ô to be an output of the training process using (xo
(8)
(1)
(n)
( >i=1, and yp to be
the (unknown) value function under the optimal policy. Let @ be the set of 0 such that & θ and is independent of (x Ji-1. Denote the projection of the parameter space O onto the loss function by H f-r Tmm, Ille EØl co(vam vI ame,
(x
m)]l ame,m))l
0 E @} the conditional
Rademacher complexity by (H) = B(x)") 1 (suPhEN 1 - Ex((x0)
( xo E C.J. where
= = X : t arg maxxeE(M) (gg) where [g] g2 gHi is the first row of xo and nu Ei_1 1(x,8
(0) C.J. Define T {t [1]] nr 2 13 and Co dTc((1/v2) + /2) for some c such that c 2 Em,ell(c- - c/ (Xm. m)) for 1....,d d. where d is the dimension of e E R Here, both the loss values and Co scale linearly in d. Our theorem works for any > O, including
7 1/d, which normalizes the loss values and Co with respect to d. Thus, the conclusion of our theorem is invariant of the scale of the loss value.
Theorem 1. For any 8 > O, with probability at least 1 5,
T K
2n R,HH,(H)
[(0) < 2(0) + Coy/
K
(13)
n
n
ET
The proof is presented in Appendix [ The baseline is recovered by setting K = 1. Thus, Theorem demonstrates that the proposed method (i.e., the case of K > 1) can improve the generalization capability of the baseline (i.e., the case of K = 1). Moreover, while the upper bound on [(0) L() decreases as K increases, it is expected that we loose more details of states with larger value of K. Therefore, there is a tradeoff in K: i.c., with a larger value of K, we expect a better generalization for the diffusion process but more loss of state-action details perform RL tasks. On the other hand, the conditional Rademacher complexity term (H) in Theorem | tends to increase as the number of parameters increases. Thus, there is also tradeoff in the kernel size: i.e., with a larger kernel size, we expect a worse generalization for the diffusion process but a better receptive field to perform RL tasks. We provide the additional analysis on (H)) in Appendix !
ON THE CONDITIONAL RADEMACHER COMPLEXITY
In this section, we state that the term EXET RxH in Theorem is also smaller for the proposed method with K 2 2 when compared to the base model (i.e., with K 1) under the following assump- tions that typically hold in practice. We assume that we can express cø (x m) W Wg(V (Vg(Vm for some functions and some matrices W, V such that the parameters of g do not contain the entries of W and V, and that & contains 0 with W and V/ such that || S Çw and ||V] < ςv for some Çw and Çv. This assumption is satisfied in most neural networks used in practice as g is arbitrarily; e.g., we can set g = Eg. W = I and V = to have any arbitrary function (x m) W gVVg(((mm Xm, m. g(xm: m). We also assume that R.) does not increase when we increase ny. This is reasonable since R, (?)) = O ((2)) ) for many machine learning models, including
"
neural networks. Under this setting, the following proposition states that the term ΣLET 2n, (,()) of with the proposed method is also smaller than that of the base model:
Proposition 1. Let q 2 2 and denote by (F) and R2 (H) the conditional Rademacher complexities for K 1 (base case) and K 2 q (proposed method) respectively. Then, (. 2 (H) for any t e [1, ... T) such that is not skipped with K q.
The proof is presented in Appendix J
J PROOFS
J.1 PROOF OF THEOREM II
Proof. Let K E (1,..., T). Define [H] f1,..., H.. Define
((x) 7 F <g( Vo + v1 ame, m1]
20

---

Published as a conference paper at ICLR 2024
Then, have that [(() EE ) and C(Ō) Ex, [[(x)]] Here, f(x0 ((x)))) ).. ................................ ((x)")) not independent since 0 is trained with the trajectories data (x (3) >==1 which induces the dependence among {(x0 )))) ..... ((x(1)) To deal with this dependence, we recall that
go
g1
8H aиK
ag
ak
ai
aK+1
aиK+1
X DECK. R*,
X0
Lak-1
82K
a(#+1)K-11
where the baseline method is recovered by setting K = 1 (and hence H = T/K T). To utilize this structure, we define Cg by
go
81
gH
ag
аg
air
Cx
X
a1
aK+1
aиk+1
E k == arg max (g)), (E(H]
aa-1
a2K-1 -1
1(#+1)K-1
We first write the expected error as the sum of the conditional expected error:
Exo [E(x))] = Σ
Exnl (xxx)] E Cx] E C2).
Similarly,
1
{(x
(1)
Σ
Σ 1(x) Σ
1(x!)))
IA
1=1
KETE IEIe
KETC
IET
where Za ii [n]]
X
(
E Cx} and Lx (k E [H] [Z,
> 1}. Using these, we decompose the
difference into two terms:
Ex, [[(xo)]
1 Σ
{(x)
(1) = Σ
Exo (x(x))/xx E Cul Prixx E C₂)
[
(14)
1=1
[t 1
+
Σ
Exole(xo)1x00 E Ci]
e(x()))
(8)
k
1=1
Exo [x(xx))xx0 E Cal
(rr(xo E Cx))
[
Exx ((xo)( E Ckl
1
Σ
((x)))
[
kEle
TEIx
By following the proof of Lemma 5 of (Kawaguchi et al. 2023 and invoking Lemma of (Kawauuchi 6,
al. 2022). we have that for any 8 >> O. with probability at least
[
Ex, (x(xx)) /20 E Cul Prixx E C22
(15)
Exo1e(xx (((x)))xxx xo E Cal /PrrXxo E CC.)
(n(H/88)
C Σ / Prr(Xo E C.)
21m(H/8)
Here, note that for any hf, M) such that M > 0 and B 2 for all X, we have that P(f()) >
M) > P((()) > M) 2 (P()((X) 1(X)> BM + h(X)), where the probability is with respect
21

---

Published as a conference paper at ICLR 2024
to the randomness of X. Thus, by combining equation 14 and equation 15
have that for any
> 0, with probability at least 6, the following holds:
Exo [((x))]
2(x0
(4)
= Σ
1
[x
Ex (xx0)((x) Nu CRl
Σ ((xb"))
(16)
7
m2
Ixl eETx
kET
2 2lm(H/0)
c
Σ
CVPr(xx E C.)
We now bound the first term in the right-hand side of equation equation I6 Define
H (x T.mm |||e
solvoni v1 ame, [1]] E θ)
and
I Σ 1=1
R2(H) = E (x0
Be
sup NEH
X2 (2)
E EC.
with independent uniform random variables E1.... En taking values in (-1, 1). We invoke Lemma
4 of (Pham et al. 202I to obtain that for any 8 > O, with probability at least 6,
Ipl
Exo ((xx))(00
Cal
Σ e(x60)) iEIn.
(17)
71
[I
kETK
Im(H//)
I
(Rx(H) + V
n kElk
2|Ix|
2 I RH)H))
In(H/5)
I
+OV
n
2n
kE
RETR
2 2 (?))
H (H/5)
+ +CV
2n
RE
where the last line follows from the Sluchy-Schward inequality applicd on the term Evele V I2
as
I
VV
Σ
Σ KElK
Σ IISVH.
1=
72
KEIK
KE
kElK
On the other hand, by using Jensen's inequality,
H
1 VH
Pr(xo C2) s
Pr(xo E Cc²)=
A=1
which implics that
H
Pr(xo E EC2) S VH.
(18)
By combining equations equation 16 and equation 17 with union bound along with equation holds that any 5 > O. with probability at least 8,
Exc, [x(x))
Σ n
1
((x(1)
I (H)
H Hln(2H/8)
2ln(2H/0)
+ CV
+ Σ /Pr(xo E (a)
n
2n
n
kETc
I (H)
+ C (V2 v2)
(2(2HH/)
<
n
KETK
22

---

Published as a conference paper at ICLR 2024
Since H [T/K], this implies that any 8 >> O, with probability least 1 - 8,
Exo [8(x0]]
Σ e(x S Cop
T
In
2|2,
n2
n
BEZK
where Co C. C=2 + v2). This proves the first statement of this theorem.
J.2 PROOF OF PROPOSITION 1
Proof. For the second statement, let = 1 and we consider the effect of increasing K from one to an arbitrary value greater than one. Denote by (HH and (H) the conditional Rademacher complexities for K = 1 (base case) and K (after increasing K) respectively: i.e., want to show that R2 (H) 2 R, (H). Given the increasing value of K, let t {l...., T) such that s, is not skipped after increasing K. From the definition of H.
I. (1)
R.(H) =E
(x0 1
Eg
sup NEH
I
cd((x)))
x EC.
(19)
[T,
EE (x
)
B
sup 8E
[
& B,mm, llll- co (s(xx)) m) m]
xxo E e
where everything is defined for K = 1 and 5(x
/l 0 + vI - -1-6 amt. Here, we recall that Eg (xmm: m) W (Xm, m) for some function g and an output layer weight matrix W such that the parameters of does not contain the entries of the output layer weight matrix W. This implies that
es (x))) ) m) Wğm (Vx x where Zm (x) g(5(x), m) where {(x) Vama vI Om Ve, and that we can decompose θ W x with which 0 can decomposed into W E W. E. and A Ô. Using this,
I
R. (H() = E
(x0))
Eg
sup AEB
B.
Wam (Vx (1) )12)
X EC,
(20)
I
1=1
I.1 [Z,
= E (x
Ee
sup (W.V
1
E Em,elle, (Vxx 1)) X
(6)
C.
where W, is the j-th row of W. Recall that when we increase K, some states are skipped and accordingly d decreases. Let du be the d after K increased from one to some value greater than one: i.e., dg < d. Without loss of generality, let us arrange the order of the coordinates over j == 1,2.... do, do + +1. 1....,d so that j = do + 1, do + 2...., are removed after K increases.
Since @ contains f with W and V such that | >0 < (w and | < Çv for some Çw and Çv, the set W contains W such that W; 0 for j = do + do + 2....,d d. Define Wo such that Wo and (W
W Wo Wo where (
W. Notice that W {(W
[I(W, ( l < {w} } and Wo {WW, do
) |[ (W blee I (w Since we take supremum over W E W, setting W,- = 0 for j = do + 1, do + 2,..., attains a lower bound as
(H) (x)0) Eg
sup
1
E Em,elle, - Wjām (Vx) (2) 1')) x E C,
[Z,
[4,
E E Eg
y (x
(W.VV
sup
[
A
X
(8)
E EC,
i=1
[2,1
1
Em, Wj (V [ x EC
E (1))
Eg
sup
(W.V
23

---

Published as a conference paper at ICLR 2024
where A, م== Em,e Em,ellej Wigm [)
E 2i Em,,e(es))]] and the last line follows
from the fact that
Eg sup (W. j=do+1
Σ ((E,,el(3)) Eg
Σ j=do+1
[())
Σ
E ((s,)((s) =
j=do+1
Similarly, since O contains / with W and V such that 1/ S (w and |V <vv for some (w and (v, the set V contains V such that V3 0 for j = do + 1, do + 2....., d, where Vj is the j-th
row of V. Define Vo such that V = Vo x Vo where (V2) E Vo and (V:) (= E Vo Notice that V = {(V)) I1(V)))==1l>> I [v} and Vo = {(V))=== : (I( lll I (v). Since we take
supremum over E V, setting V = for j = do + 1, du 2...., d attains a lower bound as
da
(H) 2 E (xg
)z
1m
Eg
sup
[
( Em. [(eg - W5 (Vx) 1)) x E C.
|z,1
E (x))) iim
EE
sup
(.d. X EC
[z
E (0))
R&
sup
1
e, (do())) x EC,
[T,
E Eg
sup
1
Σ E.Bm,elle Wgm (V=) : ec
[Zg
2 (PL)
where (i(d) Em,e (G--Wy
Wssm (
(EL )o)x
(e))e,, #6
((z) )20 C, is the C. for zn with skipping states, and R2 is the conditional Rademacher complexity after increasing K 1. The last line follows from the same steps of equation and equation 20 applied for R, (H) and the fact that [Zel of R2 (H. is smaller than that of R2 (H) (due to the effect of removing the states), along with the assumption that R. (H) does not increase when we increase n4. This proves the second statement.
24


Reviews:
[{"review_id": "dLafEDl8RB", "reviewer": "Reviewer_zGcK", "summary": "This paper proposed a modified version of the diffuser called Hierarchical Diffuser.  The Hierarchical Diffuser incorporates a two-tiered approach: a high-level \"jumpy\" planning strategy that can quickly survey a broader scope of possibilities with reduced computational demands, and a low-level planner that refines these broader plans into specific, actionable steps. This hierarchy allows the method to operate more efficiently and effectively, with the higher level providing guidance that simplifies the task for the lower level. Empirical evaluations show that this method outperforms both traditional diffusion-based planners and other hierarchical approaches in speed and performance on standard offline reinforcement learning benchmarks.", "questions": "I don’t understand the meaning of SD-DA. It seems like only improves the performance of walker2d. I feel like predicting so detailed actions loses the benefit of high-level planning.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Using a hierarchical structure makes sense in long-horizon planning. \n- In replanning, a hierarchical structure is more efficient since it only needs to use low-level to \n- Show the relationship between kernel size and generalization.", "weaknesses": "- Current SOTA diffusers seem to have a better performance. For example [1] have a 167 score on large maze2d.\n- The Unet itself has a hierarchical structure. If an environment needs a hierarchical structure, a simple way is to increase the depth of the Unet. The authors might need to provide more results to show that they are better.\n- The improvement in Mujoco is not enough for me.\n- The paper said that they have evaluated generalization. However, the OOD task they test on is too simple. In other papers, harder OOD tasks are tested.  [1] add coins. [2] add obstacle. \n- For hierarchical structure, it is important to ensure it is smooth on the connection points. However, the paper didn’t talk about it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hqW3NuNsMj", "reviewer": "Reviewer_u9ra", "summary": "The paper proposes Hierarchical Diffuser (HD), a sub-goal based method for planning with diffusion models. In recent literature, Diffuser [1] was proposed to plan an optimal sequence of state and action between initial state and goal. Following diffuser, the authors design two diffusers. One diffuser plans for sub-goals between the start and the goal state. For each sub-goal segment, the second diffuser plans the optimal state-action sequence. Experiments are conducted on Maze, AntMaze, MuJoCo Gym and FrankKitchen benchmarks. The performance of HD is compared with relevant offline RL and hierarchical RL algorithms. Further, the experiments show improved receptibility, OOD generalization and less computation overhead as compared to diffuser. Relevant ablations are performed.", "questions": "See weakness above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Discretizing planning into sub-goals with diffusion models is shown to be advantageous as more diverse scenarios can be solved. Low-level diffusion planning becomes task-agnostic.\n\nEven the sparse diffuser or SD version is better than diffuser because of increased receptive field. The authors validate this by showing that increasing the kernel-size (hence the receptive field) of diffuser leads to better performance but weaker generalization.\n\nSD with dense actions leads to better fitting of the sparse objective, a typical bottleneck in hierarchical RL.\n\nHD takes lesser time than Diffuser due to shorter high-level sequence and parallelly solving low-level plans for all the segments.", "weaknesses": "The formulation has limited novelty. While a single diffuser is not sufficient for planning over long-horizons, the work introduces two diffusers: one sparse diffuser for planning sequence of sub-goals between initial state and goal, while a standard diffuser solves for individual sub-goal segments.\n\nGiven that there are also methods which perform state-only diffusion like Decision-diffuser [2], is it possible to perform relevant ablations to justify why states are not sufficient for having a good estimate of $J$? If you are following diffuser’s codebase, are you using the diffusion sampled actions or a separate controller for Maze2d tasks like diffuser?\n\nSolving for sub-goals independently without the knowledge about the final goal might lead to non-optimal behavior. How do you ensure that the sampled path for one segment does not overlap with the paths for the other segment? How many trajectories do you sample for individual segments and for the sub-goal sequence?\n\nAlso, what happens when actions do not lead to the exact sub-goal (unless you use inverse dynamics actions) for the maze case? Because there is no-feedback from the low-level diffuser to high-level diffuser, it might become challenging.\n\nOverall, I acknowledge that the presented method is a promising revision of Diffuser which performs better in all aspects. However, I believe that the contributions are not significant enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lCWHFqw4SW", "reviewer": "Reviewer_AgLN", "summary": "The authors of this paper present Hierarchical Diffuser, a simple yet effective hierarchical planning approach using diffusion models to improve the performance and efficiency of diffusion-based planning for long-horizon tasks. By utilizing a two-pronged structure, the Hierarchical Diffuser combines a high-level planner, the Sparse Diffuser, to generate sub-goals, and a low-level planner to refine the plan. This approach significantly outperforms non-hierarchical planning methods, while also enhancing computational efficiency in training and planning speed. Furthermore, the method displays improved generalization capabilities on out-of-distribution tasks compared to non-hierarchical Diffusers.", "questions": "- I have noticed that both Diffuser and Decision Diffuser employ “inpainting” techniques for trajectory generation in goal-conditioned tasks and the lower-level planner of HDMI. However, this paper utilizes return-guided sampling. What were the considerations behind the authors' choice between the two methods?\n- This paper proposes to get the sub-goals first with the planning-based subgoal extractor and then train HDMI with the sub-goals as supervision. Although this two-step process makes sense, the overall training process becomes ad hoc and less general. Is there any connection between the upper and lower layers of Hierarchical Diffuser's planner? Can it be trained end-to-end?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: This work creatively combines the strengths of hierarchical planning and diffusion models, achieving significant improvements in performance and computational efficiency compared to existing methods.\n- Quality: The contribution provides a thorough theoretical analysis of the method, backed by well-designed experiments and in-depth analyses. The authors emphasize the rationale behind the design decisions and the resultant benefits of their approach.\n- Clarity: The paper is well-organized, clearly written, and explains the method along with its advantages and limitations in a coherent and accessible way.", "weaknesses": "- Considering the existence of the HDMI algorithm, the contribution of this paper are limited. However, compared to HDMI, this paper has conducted extensive experiments on generalization and provided corresponding theoretical support. One suggestion I have is that the authors can emphasize the advantages of Hierarchical Diffuser from the perspective of generalization, and highlight the algorithm's generalization performance in the experimental design.\n- In the hierarchical framework, the quality of the generated high-level goals is crucial to the effectiveness of the algorithm. The authors emphasize that the Hierarchical Diffuser is simpler compared to existing hierarchical methods but have only designed one goal selection method. In addition to the fixed time interval heuristic method used in this paper, HDMI also mentions two heuristic methods based on spatial intervals and reward scale, which are equally simple and have low implementation costs. If the authors could take these heuristic methods into consideration, I believe it would have a positive impact on the completeness of the paper's content. Furthermore, the demonstration of the quality of the generated goals is essential, and there is a gap in this aspect in the current paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "plv5dmKSmT", "reviewer": "Reviewer_KFqH", "summary": "The paper proposes an hierarchical variant of Diffuser. The algorithm involves using a high-level planner that plans trajectories in a sparser manner and a low-level planner that adds detailed states to the trajectories produces by the high-level planner. Experiment result demonstrate that hierarchical planning produces better performance in long-horizon planning tasks and offline RL tasks. The authers also study the impact of kernel size in Diffuser and the impact of the jumpy steps. An interesting observation is that Diffuser with a small receptive field can not cover the data distribution well.", "questions": "1. According to Fig 3., Diffuser with a small kernel size covers one mode of the data distribution during actual execution. Is it the case that the standard Diffuser is able to \\textbf{cover} the full data distribution but just fails to achieve full data coverage during actual execution? If so, what might be the cause the such gap?\n2. Still concerning Fig 3., although HD seems to cover most of the data distribution, HD fails to cover two short segments/branches. Why does such pheomena occur?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The method is straightforward and demonstrates strong results in long-horizon planning tasks, especially the AntMaze.\n3. The study of relationship between receptive field of the diffusion model and the data coverage ratio is interesting and offers valuable insight for futher research on diffusion models for planning in more complex domains.", "weaknesses": "One minor weakness is that the baseline Decision Diffuser[1] is missing. Also the study presented in this paper focuses on using the CNN architecture as Diffuser does. A comparison between using CNN and transformer architecutre, such as Decision Diffuser, would be encouraged.\n\n[1] Ajay, Anurag, et al. \"Is conditional generative modeling all you need for decision-making?.\" ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)