DiffuserLite: Towards Real-time Diffusion Planning
Authors: Zibin Dong
Jianye HAO
Yifu Yuan
Fei Ni
Yitian Wang
Pengyi Li
YAN ZHENG
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion model; Reinforcement learning; Deep reinforcement learning;
Primary area: reinforcement_learning

Abstract: Diffusion planning has been recognized as an effective decision-making paradigm in various domains. The capability of generating high-quality long-horizon trajectories makes it a promising research direction. However, existing diffusion planning methods suffer from low decision-making frequencies due to the expensive iterative sampling cost. To alleviate this, we introduce DiffuserLite, a super fast and lightweight diffusion planning framework, which employs a planning refinement process (PRP) to generate coarse-to-fine-grained trajectories, significantly reducing the modeling of redundant information and leading to notable increases in decision-making frequency. Our experimental results demonstrate that DiffuserLite achieves a decision-making frequency of $122.2$Hz ($112.7$x faster than predominant frameworks) and reaches state-of-the-art performance on D4RL, Robomimic, and FinRL benchmarks. In addition, DiffuserLite can also serve as a flexible plugin to increase the decision-making frequency of other diffusion planning algorithms, providing a structural design reference for future works. More details and visualizations are available at https://diffuserlite.github.io/.

DiffuserLite: Towards Real-time Diffusion Planning
Zibin Dong! Jianye Hao"
Yifu Yuan' Fei Yitian Wang² Pengyi Li'
Yan Zheng'
CCollege of Intelligence and Computing, Tianjin University
UC San Diego Jacohs School of Engineering
MaloCe Score
Kitchen Seare
AstMilaze Soore
Deciiio Frequescy (Rl)
NN Nite (M)
Diffisert ltee
Diffosrrt ltee
DilluxerL. Lite
DiffuserLii Lite
50
Difhiser
Diflser
Diffuser
Diffoser
Difflser
76
DD
DD
DD
DD
DD
13Hz 0.48z 122HX
4.1M
59M
2.7M
Figure 1: Performance overview. We present DiffuserLLite, lightweight framework that utilizes progressive
refinement planning reduce redundant information generation achieves real-time diffuusio planning. DiffuseetLte significantly outperforms predominant frameworks, Diffuser and DD, regarding scores, inference time, and model size on three popular D4RL benchmarks. The decision-making frequency of Diffusert.iee achieves 122. 2zz which 112.7 times higher than predominant frameworks.
Abstract
Diffusion planning has been recognized as an effective decision-making paradigm in various domains. The capability of generating high-quality long-horizon ries makes it promising research direction. However, existing diffusion planning methods suffer from low peciiinn-making frequencies due to the expensive itera- tive sampling cost. To alleviate this, we introduce hiffuserlite, a super fast and lightweight diffusion planning framework, which employs a planning refinement process (PRP) to generate coarse-to coarse-to-fin--eee fine- fine-grained trajectories, significantly reducing the modeling of redundant information and leading to notable increases decision- making frequency. Our experimental results demonstrate that DiffuserLite achieves a decision-making frequency of 1 122.2Hz (112.7s faster than predominant frame- works) and reaches sate-offthe-art performance on D4RL, Robomimic, and FinRL benchmarks. addition, DiffuserLite can also serve as a flcxible plugin to increase the dicision-making frequency of other diffusion planning algorithms, providing structural design reference for future works. More details and visualizations are available project website.
horizon trajecto-
Introduction
Diffusion models (DMs) powerful generative models that demonstrate promising performance across various domains [52, 44, 29, 22]. Motivated by their remarkable capability in complex distribution modeling and conditional generation, researchers have developed a series of works applying diffusion models for decision-making tasks recent years [55]. DMs can play various roles in decision-making tasks, such acting as planners to make better decisions from long- term perspective [20, 1, 8, 36, 10], serving as policies to support complex multimodal-diitribeee modeling [38, 50, 6], and working data synthesizers to assist reinforcement learning (RL) training [33. 51, 16), etc. Among these roles, diffusion planning is the most widely applied paradigm [55]. Unlike auto-regressive planning in previous based RL approaches [14, 49, 15), diffusion planning avoids severe compounding errors by directly generating the entire trajectory rather than one-step transition [55]. Also, its powerful conditional generation capability allows planning the
"Contact me at zi biindong@outlook.. com
'Conespondence to: Jianye Hao (jianye. (jiante.haootiip hww.tho.......... ju.ottuu.edu... eed...d.......... cn), Zheng (yanzheng@tjja wuw.juu.euccm edu.cn)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

trajectory level without being limited to step shortsightedness. The diffusion planning paradigm has achieved state-of-the -hrtt (SOTA) performance in various offline RL tasks, including single-agent RL [27], multi-agent RI. [54], meta RI. [36], and more.
One key issue that diffusion planning faces is the expensive iterative sampling cost. As de- picted in fig. 1, the recision-making frequen- cies (number of actions inferred per second)
Total D-128
of two predominant diffusion planning frame-
Leval
works, Diffuser [20] and Decision Diffuser (DD)
Tatel 0-128
Xei
[1], are recorded as 1.3Hz and 0.4Hz, respec-
tively. Such a low decision frequency fails to
Level
Total 0-22
Kcy
meet the requirements of numerous real-world applications, e.g. real-time robot control [48] and game Al [39]. The low decision frequency is primarily attributed to modeling a denoising process for a long-horizon trajectory distribu- tion, which requires a heavy neural network backbone and multiple forward passes. A ques- tion may arise whether the generation of com- plete long-horizar successful planning. Experimental results in- dicate that it is not, for the detailed trajectory information in long-horizon segments is highly redundant. As shown in fig. 2, a motivation ex- ample in Antmaze, the disparities between plans increase as the horizon grows, leading to poor consistency between plans in consecutive steps. Besides, in practice, agents often struggle to reach the planned distant state. These facts argue that while it introduces redundant information in distinct parts. The details in closer parts are more crucial. Ignoring the modeling of these redundant parts in the diffusion planning process will significantly reduce the complexity of the trajectory distribution be fitted, making possible to build a fast and lightweight diffusion planning framework.
Lewd 2
Hertoee 9
Figure 2: Comparison of one-shot planning (top) and PRP (down) on Antmaze. The former directly gener- ates plans with temporal horizon of 129. The latter consists of three coarse to fine-grained levels with tempo- ral horizons of 0-128, 0-32, and 0-8, and temporal jumps of 32, 8, and 1. respectively. The visualization in the tig- ure illustrates the x-y coordinates of 100 plans. It shows that ohe-shot planning exhibits a significant amount of redundant information and large search space. In con- trast, PRP demonstrates better plan consistency and a smaller search space.
trajectories is necessary for
planning helps improve foresight,
Motivated by these insights, we propose build plan refinement process (PRP) speed up diffusion planning. First, we perform "rough" planning, where jumpy planning is executed, only considering the states intervals that are far apart and ignoring other individual states. Then, we refine a small portion of the plan, focusing on the steps closer to the current state. By doing so, we fill in the execution details between two states far apart, gradually refining the plan to the step level. This approach has three advantages: 1) It reduces the length of the sequences generated by the diffusion model, simplifying the complexity of the probability distribution reduces the search space of plans, making it casier for the planner to find well-performed trajectories. 3) Since only the first action of each step is executed, rough planning of steps further away causes no noticeable performance drop.
be fitted. 2) It significantly
Diffusion planning with PRP, which we call DiffuserLite, is simple, fast, and lightweight. Our experiments have demonstrated the effectiveness of PRP, significantly increasing decision-making frequency while achieving SOTA performance. Moreover, can be easily adapted to other existing diffusion planning methods. In summary, our contributions are as follows:
We introduce the plan refinement process (PRP) for coarse-to-fine fine-grained trajectory genera- tion, reducing the modeling of redundant information.
We introduce Diffuserl.ite .ite, lightweight diffusion planning framework, which significantly increases decision-making frequency by employing PRP.
DiffuserLite is simple and flexible plugin that can be casily combined with other diffusion planning algorithms.
Diffuserl.i Lite achieves super high decision- decision-making frequency (122.2Hz, 112.7x faster than predominant frameworks) and SOTA performance on multiple benchmarks in D4RL.

---

2 Preliminaries
f at) at state or 1] can be either sequence of states F2 = or or state-action pairs 24 = (0t, at), where T is the planning horizon. Each trajectory can be mapped to a property c. Diffusion planning aims to find trajectory that exhibits property closest to the target:
Problem Setup:
Consider system governed by discrete-time dynamics 0c+11
given an action a₁ A trajectory a = [To. ... -T--1]
xx = arg min d(C(()): Crarget)
where d is certain distance metrie, C is critic that maps trajectory to the property it exhibits, and Curget is the target property. The action to be executed a is then extracted from the selected trajectory (state-action sequences) or predicted by an inverse dynamic model a, h(o,, os+1) (state- only sequences). In the context of offlin RL, is a common choice to define the property as the corresponding cumulative reward C(x) = Ei= r(or, at) in previous works [20, 1].
Diffusion Models assume unknown trajectory distribution go (zx)) DMs define forward process { with S > 0. Starting with Tu, previous work [23] proved that one can obtain any æ, by solving the following stochastic differential equation (SDE):
dæ. [8))xads + g(s)dw., zo go(
doa d doga, Values of ds 20 de (SNR) /o strictly decreasing [23]. While this SDE transforms go into a noise distribution (xs) ~O, I). one
where w. is the standard Wiener process, and f(s) dloga, g²(s)
as. T. IR depend on the noise schedule but keep the ggnall-o-noise-raan
can reconstruct trajectories from the noise by solving the reverse process of eq. (2). Previous work [46] proved that solving its associated probability flow ODE can support faster sampling:
dæ. = f(s)a - (s)V -
1
(og (x.), æg (s(ss))
ds
in which score function V log 9a (x,) is the only unknown term and estimated by neural network e (x., s)/o, in practice. The parameter 0 is optimized by minimizing the following objective:
( = E (m) (ot) Ille s) <l
(4)
where E ~ ((E) = NO = a,æo + O.E. Various ODE solvers can be employed to solve Equation (3), such as the Euler solver [2], RK45 solver [9], DPM solver [32], etc.
Conditional Sampling helps to generate trajectories exhibiting certain properties a priori. There two main approaches: classifier glassifier-uuidanee guidance (CG) [7] and classifier -reeeffeeeeeeeeeee gredggui (CFG) [18]. CG requires an additional classifier log Po (c|#,, 8) to predict the log probability that a noisy trajectory æ, exhibits given property c. The gradients from this classifier are then used to guide the solver:
6o s, c) == c0 8) w o₂ V x. log P (c)x,,
(5)
CFG does not require an additional classifier but uses a conditional noise predictor to guide the solver.
( c) <x + (1 - w) Ee (x s)
(6)
Increasing the value of guidance strength w leads to more property-aligned generation, but decreases the legality of the generated trajectories [8].
3 Efficient Planning via Refinement
Diffuser [20] and DD [1] are two pioneering frameworks, upon which vast amount of diffusion planning works has been built [28, 26, 53]. Although the design details differ, they can be unified into one paradigm. In the inference step, multiple candidate trajectories are first conditionally sampled using the diffusion model. Then, critic is used to select the optimal one that exhibits the closest property to the target. Finally, the action to be executed is extracted. This paradigm relies on multiple forwarding complex neural networks, resulting in extremely low decision-making frequencies (typically 1-10Hz, or even less than 1Hz), severely hindering its real-worr weal-world deployment.
3

---

Current Shate
Level
Level
Level L-1
Foustion Noiie
Diffuution Danolding
Diffusion Deneising
bif Furunnn Dencising
Candidate Shotes
9o
R/o
Critic Selection
Crihic Selection
Gritic Selection
R2
Environtent
Inverse
Figure 3: Overview of DiffuserLite. Observing the current state os, level O of DiffuserL.it fixes of as op and generates multiple candidate trajectories. A critic is then used to select the optimal one. in which or, is then passed to the next level its terminal - 1. The plan refinement process continues iteratively until the last level with temporal jump IT.1 Finally, the action aa to be executed extracted using an inverse dynamic model a, = h(oo, 01)-
The fundamental reason is the requirement for highly complex neural networks to model the complex Long-horizon trajectory distributions [55]. Although some works have explored using advanced ODE solvers reduce the sampling steps around 5 [8], the time consumption of network forwarding still unacceptable. However, we notice that ignoring some redundant distant parts of the generated plan can be a possible solution to address the issue. As shown in fig. 2, the disparities between plans increase as the horizon grows, leading to poor consistency between the plans selected consecutive steps. Besides, agents often struggle reach the distant states given by plan practice. Both facts indicate that terms in distant parts of plan become increasingly redundant, whereas the closer parts crucial.
Regarding these findings, we aim to develop a progressive refinement planning (PRP) process. This process initially plans rough trajectory consisting of only key points spaced at equal intervals and then progressively refines the first interval by generative interpolating. It is worth noting that this is orthogonal to methods of identifying key points based task semantic information [27]. Specifically,
our proposed PRP consists of L planning levels. At each level / E (0, 1,... L - 1}. starting from the known first term Zo, DMs plan rough trajectories Fo:H;: with temporal horizon Hi and temporal jump 4. Then, the planned first key point is passed to the next level as its terminal:
Co:Hecht
[oo F1 T21, ... -H
Wo
[00 21 T2
, -1
F1. -1
By this design, only the first planned intervals are refined in the next level, and the other redundant details are all ignored, resulting coarse-to-fine generation process. The progressive refinement continues until the last level to extract action. To support conditional sampling for each level, define the property of a rough trajectory Do:Hel as the property expectation over the distribution of all its completed trajectories X (zo.H::1)):
C(za.Hes)) == Ee [C(x)]
(
PRP ensures that long-term planning maintains foresight while alleviating the burden of modeling redundant information. As a result, it greatly contributes to reducing model size and improving planning efficiency:
Simplifying the fitted distribution of DMs. The absence of redundant details in PRP allows for a significant reduction in the complexity of the fitted distribution at each level. This reduction in complexity enables to utilize lighter neural network backbone, shorter network input sequence lengths, and reduced number of denoising steps.
Reducing the plan-search space. Key points generated at former levels often sufficiently reflect the quality the entire trajectory, which allows the planner focus more on finding distant key points and planning actions for the immediate steps, reducing search space and complexity.

---

4 A Lite Architecture for Real-time Diffusion Planning
Employing PRP results in a new lightweight architecture for diffusion planning, which we refer to as Diffuserl.ite. DiffuserLite can reduce the complexity of the fit distribution and significantly increase the Recision-making frequency, achieving 122Hz on average for the need of real-time control. We present the architecture overview in fig. 3, provide pseudocode for both training and inference in algorithm and algorithm 2, and discuss detailed design choices in this section.
Diffusion model for each level: We train L diffusion models for all levels to generate state-only sequences. We employ DiT [40] as the noise predictor backbone, instead of the more commonly used UNet [43] due to the significantly reduced length of the generated sequences in each level (typically around 5). This eliminates the need for 1D convolution extract local temporal features. To adapt the DiT backbone for temporal generation, we make minimal structural adjustments following [8]. For conditional sampling, we utilize CFG instead of CG, as the slow gradient computation process of CG reduces the frequency of decision-maker making. During the training phase, at each gradient step, we sample a batch of Ho-length trajectories, slice each of them into L sujerraaecsoriis Ho:l evaluate their properties C (Zo:HH1)) as an estimation of (, for condition at each level, and then slice the sub-trajectories into evenly spaced training samples Z N;=fi for training the diffusion models. During the inference phase, as shown in fig. 3, diffusion models generate multiple candidate plans level by level from 0 to L - 1, and the optimal one is selected by the critic C.
The Critic C' in DiffuserLite plays two important roles: providing generation conditions during the diffusion training process and selecting the optimal plan from the candidates generated by the diffusion model during inference. In the context of Offline RL, both Diffuser and DD adopt the cumulative reward of trajectory as the condition:
Critic design:
H-1 r(o,, ac),
C(x) = Σ 1=0
where H the temporal horizon. This design allows rewards from the offline RL dataset to be utilized as ground-truth critic for acquiring generation conditions during training. During inference, an additional trained reward function is required to serve as the critic. The critic then helps select the plan that maximizes the cumulative reward, as depicted in the lower part of fig. 3. However, this design poses challenges in tasks with sparse rewards, as can confuse diffusion models when trajectories, especially for
distinguishing better-performe challenge, we introduce an option to use the sum of discounted rewards and the value of the last state as additional property design:
plans. To address this
H-2 2'r(on ar) + =
C(x)= Σ
(10)
where Voo)) = max " represents the optimal value function [47] and can be estimated by neural network through various offline RL methods. In the context of other domains, properties can be flexibly designed as needed, as long as the critic C can evaluate trajectories of variable lengths. It is worth noting that previous diffusion planning algorithms widely support this flexibility. In addition, it is even possible to skip the critic selection during inference, which is equivalent to using uniform critic, used in DD.
Action extraction: After obtaining the optimal trajectory from the last level through critic selection, we utilize an additional inverse dynamic model o2 h(o,, 0++1) to extract the action to be executed. This approach is suggested in [1].
Further speedup with rectified flow: DiffuserLite aims to achieve real-time diffusion planning to support its application real-world scenarios. Therefore, we introduce Rectified flow [30] for further increasing the decision-making frequency. Rectified flow, an ODE on the time interval [0, 1). causalizes the paths of linear interpolation between two distributions. If we define the two distributions as trajectory distribution and standard Gaussian, we can directly replace the Diffusion ODE with rectified flow to achieve the same functionality. The most significant difference that rectified flow learns straight-line flow and can continuously straighten the ODE through reflow. This straightness property allows for consistent and stable gradients throughout the flow, enabling the generation of trajectories with very few sampling steps (in our experiments, we found that one-step
5

---

Table 1: Time consumption per step and frequency in D4RL. All results obtained over 5 random seeds. DiffuserLite achieves an average decision frequency of about 122Hz on the R2 backbone and Hz averaging over three variations, which meets the requirements of real-time inference. Since the code for HDMI is not open-source, we make every effort to reproduce HDMI with the original settings to test its runtime cost and thus mark its results with underlines.
Environment
Metric
Diffuser
DD
HDMI
Diffuserian-D
DiffuserLie-RI
DiffuserLin--2
Runtime (s)
0.665
2.142
0.405
0.015
0.013
0.005
MuloCo
Prequency (Hz)
1.5
0.47
2.5
68.2
79.7
200.7
Kitchen
Runtime (s)
0.790
2.573
0,407
0.017
0.015
0.010
Prequency (Hz)
2.5
58.7
66.0
Runtime (s)
0.791
2.591
0.410
0.027
0.015
0.010
Antmaze
Frequency (Hz)
1.3
0.39
244
37.3
65.7
101.7
Average Runtime (S)
0.749
2,435
0.407
0.020
0.014
0.008
Average Frequency (Hz)
1.31
0.41
2.46
51.26
69.90
122.44
sampling sufficient to produce good results). We consider rectified flow optional backbone for cases that prioritize decision frequency. In appendix B, we offer comprehensive explanations of trajectory generation and training with rectified flow.
5 Experiments
We explored the performance of the DiffuserLite on various tasks on D4RL, Robomimic, and FinRL [12, 34, 41], and aimed to answer the following research questions (RQs): 1) To what extent can DiffuserLite reduce the runtime cost?
Taak Comporee
2) How is the performance of DiffuserLite on
235x fadeer
offline RL tasks? 3) Can DiffuserLite serve flexible plugin for other diffusion planning al- gorithms? 4) Can we summarize a list of simple and clear design choices for Diffuserl.ite?
300x faster
Watckcck
5.1 Experimental Setup
Difhuer
Diffused ze R3
Diffeeetue
DO
Diflowrt
Figure 4: Runtime and performance comparison in FrankaKitchen. The y-axis represents number of completed tasks (maximum of 4), and the x-axis repre- sents the required wall-clock time. Task success rates are presented in colored circles. All results averaged over 250 rollouts. Diffusert demonstrates significant advantages in both wallclock time and success rate.
Benchmarks: We evaluate the algorithm on various offline RL domains, including locomo- tion in Gym-MuJoCo [4], real-world manipu- lation in FrankaKitchen [13] and Robomimic [34], long-horizon navigation in Antmaze [12], and real-world stock trading in FinRL [41]. We train all models using publicly available datasets (see appendix A. for further details).
Our comparisons mainly include the basic imitation learning algorithm BC, existing offline RL methods CQL [25] and IQL [24], as well as two pioneering diffusion planning frameworks, Diffuser [20] and DD [1]. Additionally, we compare with a state-of-the-art algorithm, HDMI [27], which is improved based on DD. (Further details about each baseline and the sources of performance results for each baseline across the experiments are presented appendix A.2).
Baselines:
Backbones: As mentioned in Section 4, we implement three variations of DiffuserLite, based on different backbones: 1) diffusion model, 2) rectified flow, and 3) rectified flow with an additional reflow step. These variations will be used in subsequent experiments and indicated by the suffixes D, R1, and R2, respectively. All backbones utilize three levels, with total temporal horizon of 129 in MuloCo and Antmaze, and 49 in Kitchen. For more details about the hyperparameters selection for DiffuserLite, please refer appendix D.
Computing Power: All runtime results across our experiments are obtained on a server equipped with an Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz and an NVIDIA GeForce RTX3090.

---

Table 2: D4RL Performance. Results for DiffuserLite correspond to the mean and standard over random seeds. We detail sources for the performance of prior methods appendix A.2. Following Diffuser [20]. we emphasize in bold scores within 5 percent of the maximum per task (2 0.95 max).
Dataset
Environment
BC
CQL
IQL.
Diffucer
DD
HDMI
Diftusertn-D
Diffusertn-RRI R1
Diffusertn--R2
HaltCheesah
65-2
91.6
96.7
79.8
90.4
92.1
38.5+00.4
90.8 + 0.9
84.0 + 2.9
Medium--xpper
Hopper
52.5
105.
01.5
107
111.8
113.5
111.6 = 0.2
110.3 = 0.3
110.1 0.5
Walker2d
107.5
108.8
109.6
108.8
107.9
107.1 0.8
106.4
106.1
HallCheetah
42.6
44.0
47.4
44.2
49.1
48.0
46,9
48.6 + 0.7
45.3 :
Medium
Hopper
$2.9
58.5
06.3
58.5
79.3
76.4
100.9
1.1
99.5 0.7
96.8 :: 0.3
Walker2d
75.3
72.5
78.3
79.7
82.5
79.9
86.8 1: 0.6
85.1 +: 0.5
53.7 1 1.0
HallChhesah
36.6
45.5
44.2
42.2
30.3
44.9
41.6 0.4
42.9 + 0.4
30.6 + 0.4
Medium-Replay
Hopper
96.0
94.7
100.0
99.6
93.2
Walker2d
26.0
77.2
73.9
01.2
75.0
80.7
90.2
86.6
78.1
Average
51.9
77.6
77
78.3
81.8
82.8
86.0
85.1
81.9
Mixed Partial
Kitchen
51.5
52.4
51.0
50.0
65
69.2
73.6 1: O.T
71.9 + 1.4
64.8 :: 1.8
Kitchen
38.0
50.1
46.3
56.2
57
74.4 + 0.6
60.9 + 0.7
71.4 + 1.2
Average
51.3
48.7
74.0
Pla
Antmaze- Mediam
Anumaze- Large
0.0
20.8
60.8
65.8
12.0
0.0
0.0
0.0
72.0 1: 6.2
2:2
72.4 1: 2.3
88.0 2:2
88.8 3.2
69.4
6.5
Diverse
Animase Medium
0.8
67.3
73.8
0.0
0.0
92.4 3.2
89.2 :: 2.0
87.6 +: 2.0
Antmaae- Large
0.0
20.5
30.3
0.0
0.0
08.0 2.8
80.4 +: 5.1
70.2 +: 3.5
Average
43.6
58.0
0.0
77.6
52.5
Runtime per action (sccond)
0.749
2-435
0.0/20
0.014
0.808
Table 3: Robomimic Performance.
Table 4: FinRL Performance.
Dataset
BC
CQL.
BCQ
IRIS
Diffoseria-D
Dataset
BC
CQL.
MB-PPO
DD
HDMI
Diffowetza-D
Lin. PH
100.0
92.7
100.0
100.0
100.0
FiRRL-H-9H
270
444
757
782
801
736
Can-PeB
96.3
38.0
88.7
100.0
100.0
Fi-91-M-999
501
621
698
712
754
762
Squaro-H
78.7
5.3
60.0
78.7
81.8
Average
557
535
743
767
778
779
Average
91.3
45.3
79.6
929
5.2 Runtime Cost (RQ1)
The primary objective of DiffuserLite is to increase the decision-making frequency. Therefore, we first test the wall-clock runtime cost (time consumption for one action inference) of DiffuserLite under three different backbones, compared to Diffuser, DD, and HDMI, to determine the extent of the advantage gained. We present the test results in table 1 which shows that the runtime cost of DiffuserLite with D, RI, and R2 backbones is only 1.23%, 0.89%, and 0.51% % of the average runtime cost of Diffuser and DD, respectively. The remarkable improvement in dacision-making frequency does not harm its performance. As shown in fig. 4, compared to the average success rates of Diffuser (purple line) and DI) (grey line) on four FrankaKitchen sub-tasks, Diffuserl.it Lite improves them by [41.5%, 56.2%, 55.3%, 8.7 8.7%). respectively, while being 200-300 times faster. These improvements are attributed to ignoring redundant information in PRP, which reduces the complexity of the distribution that the backbone generative model needs to fit, allowing to employ a light neural network backbone and use fewer sampling steps to conduct perfect-enough planning. Its success in FrankaKitchen, a realistic robot manipulation scenario, also reflects its potential application in rorl-world settings.
5.3 Performance (RQ2)
DiffuserLLite Lite is then evaluated on various popular domains in D4RL, Robomimic, and FinRL, to test how well it can maintain the performance when significantly increasing the decision-makinn frequency. All results are presented in table 2, table 3, and table 4, and the detailed descriptions of the sources of all baseline results are listed in appendix A.2. Results in D4RL table show significant performance improvements across all benchmarks with high decision-makinn advantage is particularly pronounced in FrankaKitchen and Antmaze environments, indicating that the structure of DiffuserLite enables more accurate and efficient planning in morioooizon tasks, thus yiclding greater benefits. In the MuJoCo environments, more notable ad advantages are shown on sub-optimal datasets, i.c., "medium" and imndiuu-replay datasets. This sub-optimal advantage can be attributed to the PRP planning structure, which does not require one-shot generation of a consistent long trajectory, but explicitly demands stitching. This allows for better utilization of high-quality segments in low quality datasets, leading to improved performance. Results in Robomimic and
frequency. This
'Since the code for HDMI is not open- we make every effort to reproduce HDMI with the original settings to test its runtime cost and thus underline its results in table 1.
7

---

Table 5: Integrate with DiffuserLite as a plugin. We refer to AlignDiff with DiffuserLise plugin AlignDiff- Lite. A larger value of MAE Area indicates stronger alignment capability. AlignDiff-Lite greatly increases decision makinn frequency. while only experiencing a small performance drop.
Metric
GC
AlignDiff
AliguDiff-Lite
MAE Area
0.319 4 0.005
0.621 4 0.023 0.601 + 0.018 (3.2% J)
Prequency (Hz)
6.9
45.5 (580% 1)
Table 6: Performance of DiffuserLite with various PRP design choices. The left part shows comparison with 2/3/4 planning levels and the right part shows a comparison with 4 temporal horizon designs. Results correspond to the mean and standard error over 5 random seeds, the highest scores are emphasized in bold, and the default design choices used across other experiments are underlined.
Temporal horizon of each level [5,3,5,5]
Planning horizon=129
[9,17]
[5,5,9]
[3,5,17]
[5,5,9]
[9,5,5]
[17,5,3]
HalfChectah-me
75.6 +: 8.3
88.5 + 0.4
88.3 4: 0.5
85.6 0.6
88.5 0.4
89.0 + 1.7
Antmaze-Id
0.0 + 0.0
68.0 +: 2.8
69.3 + 3.4
34.7 1.77::0 4.1
68.0 = 2.8
67.3 + 3.4
34.0 +: 4.3
Planning horizon=19
[7.9]
[4,5.5]
[3,4.3.5]
[3.3.13]
[4.5,5]
[5,5.4]
[13,3,3]
Kitchen-p
72.8 + 0.5 74.4 + 0.6
72.3 4 1.9
66.7 1.7
74.4 + 0.6
74.2 4: 0.6
31.7 2.7
FinRL table are obtained by models trained on real-world datasets, and demonstrate that DiffuserLite continues to exhibit its superiority in these real-world tasks, achieving performance comparable to SOTA algorithms. This illustrates the potential application of DiffuserLite in real-world scenarios.
5.4 Flexible Plugin (RQ3)
To test the capability of DiffuserLite as a flexible plugin to support other diffusion planning algorithms, Aligndiff [8] is selected
algorithm backbone, and integrated with Diffuserl.it ite plugin, referred to as AlignDiff-Lite AlignDiff aims to customize the agent's behavior to align with human preferences and introduces MAE
non-reward-maximinn
metric to measure this alignment capability (refer to appendix A.3 for more details), where a larger value indicates a stronger alignment capability. The comparison of AlignDiff and AlignDiff-Lite is presented in table 5, showing that AlignDiff-Lite achieves a 560% improvement in decision-making frequency compared to AlignDiff, while only experiencing small performance drop of 3.2%. This result demonstrates the potential of DiffuserLite serving as a plugin to accelerate diffusion planning across various domains.
5.5 Ablations (RQ4)
How to choose the number of planning levels and the temporal horizons H,? We compared the performance of LiffuserLite with 2/3/4 planning levels and with four different temporal horizon designs, and reported the results the left and right part of table 6, respectively. The list in the first row represents the temporal horizon for each level, with larger values on the left indicating more planning at a coarser granularity, while larger values on the right indicate more planning at a finer granularity. For planning level design, results show performance drop with 2 planning levels, particularly in Antmaze, and a consistently excellent performance with 3 or 4 planning levels. This suggests that for longer-horizon tasks, it is advisable to design more planning levels. For temporal horizon design, results show a performance drop when the temporal horizon of one level becomes excessively long. This suggests the temporal horizon of each level is supposed to be similar and stay close. We summarized and presented a design choice list in appendix C.
Has the last level (short horizon) of Diffuserl ite already performed well in decision making? This is equivalent to the direct use the shorter planning horizon. If is true, key points generated by former levels may not have impact, making PRP meaningless. To address this question, conduct tests using a one-level DiffuserLite with the same temporal horizon as the last level of the default model, referred to as Lite w/ only last level. The results are presented in table 7, column 2. The notable performance drop demonstrates the importance of a oong-enough planning horizon.
Can decision-making be effectively accomplished without progressive refinement planning (PRP)? To address this question, we conduct tests using a one-level Diffuserl.ite with the same temporal horizon as the default model, referred to as Lite w/o PRP. This model only supports one-shot generation at the inference phase. We also test a "smaller" DD with the same network parameters and sampling steps as DiffuserLite, to verify whether can speed up DD by simply
8

---

Table 7: Ablation tests conducted to examine the effectiveness of PRP. Lite w/ only last level and Lite w/o PRP two ablated versions of DiffuserLite, while DD-small is version of DD that uses the same network parameters and sampling steps as DiffuserLiee. All results are obtained over seeds. The varying degrees of performance drop observed in each experiment highlight the importance of PRP.
Lite only last level
Enviromment
Oracle
PRP
smail
Hogper-me
111. 4 0.2
27.8 4: 10,2
96.6 1.0
67.9 4: 24.7
Hopper-m
100. +
19.4
+ 20.2
16.7+ 8.0
Hopper-mr
96.6 0.3
2.0 4: 0.2
02.5 + 31.3
1.0 = 0.4
Average
103.1
16.4 (84.1% 4)
75.2 (27.1% 43
28.5 (72.3%
Kitchen-m
73.6 0.7
48 2 1.4
26.9
0.0 0.0
Kitcheo-p
74.4 à 0.6
38.0 d: 3.1
23.9 d 0.5
1.8 4 0.8
Average
74.0
43.4 (41.3% 4)
25.4 (65.7% U1
(18.85% L)
reducing the parameters, referred to as DD-small. The results are presented in Table 7, column 3-4. The large standard deviation and the significant performance drop provide strong evidence for the limitations of one-shot generation planning. having difficulties in modeling the distribution of detailed long-horizon trajectories. However. DiffuserLite can maintain high performance with fast Socision-making frequency due to its lite architecture and simplified fitted distribution.
More ablations. We conducted further ablation studies model size, sampling steps, planning horizon, with or without value condition, and visual comparison to better elucidate DiffuserL.tt. Please refer appendix E for these sections.
Related Works
Diffusion models are a type of score-based generative model [46]. Two pioneering frameworks, Diffuser [20] and DD [1], were the first to attempt using diffusion models for trajectory generation and planning in decision-making tasks. Based on these two frameworks, diffusion planning has been continuously improved and applied various decision-making domains [36, 10, 54, 8, 17, 26, 21J. However, long-horizon estimation and prediction often suffer from potential exponentially increasing variance concerning the temporal horizon, called the "curse of horizon" [42]. To address this, HDMI [27] is the first algorithm that proposed a hierarchical decision framework to generate sub-goals at the upper level and reach goals at the lower level, achieving improvements long- Long-horizon tasks. However, HDMI is limited by cluster-based dataset pre-processing to obtain high-quality sub-goal data for upper training. Another concurrent work, HD-DA [5] introduces a similar hicrarchical structure and allows the high-level diffusion model to automatically discover sub-goals from the dataset, achieving better results. However, the motivation behind DiffuserLite is completely different, which aims to increase the decision makiing frequency of diffusion planning. Also, DiffuserLite allows for more hierarchy levels and refines only the first interval of the previous layer using PRP. Since HD-DA does not ignore redundant information, it fails to achieve a notable frequency increase. Compared to related works, Diffuserl.tt Lite has more clean plugin design and undoubtedly contributes to increasing decision bocision-making frequency and performance. We believe it can serve reference for the design of future diffusion planning frameworks.
7 Conclusion
In this paper, we introduce DiffuserLite, a super fast and lightweight diffusion planning framework that significantly increases pocision-making frequency by employing the plan refinement process (PRP). PRP enables coarse-to-fine fon--ieerr trajectory generation, reducing the modeling of redundant information. Experimental results various D4RL benchmarks demonstrate that DiffuserLite achieves super-high babision-making frequency of 22.2Hz (112.7x faster than previous mainstream frameworks) while maintaining SOTA performance. DiffuserLit Lite provides three generative model backbones to adapt to different requirements and can be flexibly integrated into other diffusion planning algorithms as plugin. However, DiffuserLitt Lite currently has limitations mainly caused by the classifier-free guidance (CFG). CFG sometimes requires adjusting the target condition, which becomes more cumbersome on the multi-level structure of DiffuserLite. future works, designing better guidance mechanisms, devising an optimal temporal jump adjustment, or integrating all levels one diffusion model to simplify the framework worth considering. DiffuserLite may also have some societal impacts, such as expediting the deployment of robotic products that could be utilized for military purposes.

---

8 Acknowledgements
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62422605, 92370132, 62106172), the National Key R&D Program of China (Grant No. 2022ZD01 (6402) and the Xiaomi Young Talents Program of Xiaomi Foundation.
References
[1] Anurag Ajay, Yilun Du, Abhi Gupta. Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference Learning Representations, ICLR, 2023.
[2] Kendall E. Atkinson. An Introduction to Numerical Analysis. John Wiley & Sons, 1989. URL http:// //wp: worldcat. iwwreatttrrr/ibiiiiiii org/1ssb./0471500222.
[3] Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint 1607.06450, ArXiv, 2016.
[4] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. In arXiv preprint 1606.01540, ArXiv, 2016.
[5] Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchi- cal planning with diffusion. In arXiv preprint #401.02644, ArXiv, 2024.
[6] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems, RSS, 2023.
[7] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. Advances in Neural Information Processing Systems, NIPS, 2021.
[8] Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yao Mu, Yan Zheng, Yujing Hu, Tangjie Lv. Changjie Fan, and Zhipeng Hu, Aligndiff: Aligning diverse human preferences via behavior- customisable diffusion model. In The Twelfth International Conference on Learning Represen- tations, ICLR, 2024.
[9] R. Dormand and P.J. Prince. A family of embedded runge-kutta formulae. Journal of Computational and Applied Mathematics, 1980.
[10] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai. Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Thirty-seventh Conference Neural Information Processing Systems, NIPS, 2023.
[11] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In International Conference Learning Representations, ICLR, 2022.
[12] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. In arXiv preprint 2004.07219, ArXiv, 2020.
[13] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon horizon tasks via imitation and reinforcement learning. In Proceedings of the Conference on Robot Learning, PMLR, 2020.
[14] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, ICML, 2019.
[15] Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. In International Conference on Machine Learning, ICML, 2022.
10

---

[16] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang. Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, NIPS, 2023.
[17] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is effective planner and data synthesizer for multi-task reinforcement learning. arXiv preprint PrXiv:2305.18453 2023.
[18] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, NIPS, 2021.
[19] Mineui Hong, Minjae Kang, and Songhwai Oh. Diffused task-agnostic milestone planner. arXiv preprint 2312.03395, ArXiv, 2023.
[20] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, ICML, 2022.
[21] Chiyu Jiang, Andre Cormman, Cheolho Park, Benjamin Sapp. Yin Zhou, Dragomir Anguelov, et al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9644-9653, 2023.
[22] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text- text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2023.
[23] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In Advances in Neural Information Processing Systems, NIPS, 2021.
[24] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learming. In The Tenth International Conference on Learning Representations, ICLR, 2022.
[25] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, NIPS, 2020.
[26] Kyowoon Lec, Scongun Kim, and Jaesik Choi. Refining diffusion planner for reliable behavior synthesis by automatic detection of infeasible plans. In Advances in Neural Information Processing Systems, NIPS, 2023.
[27] Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hicrarchical diffusion for offline decision making. In Proceedings of the 40th International Conference on Machine Learning. ICML, 2023.
[28] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo.. Adaptd- iffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning, ICML, 2023.
[29] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei. Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. AudioLDM: Text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, ICML, 2023.
[30] Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
[31] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR, 2019.
[32] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In Advances in Neural Information Processing Systems, NIPS, 2022.

---

[33] Cong Lu, Philip J. Ball, and Jack Parker-Holder, Synthetic experience replay. In Workshop on Reincarnating Reinforcement Learning at ICLR, 2023.
[34] Ajay Mandlekar, Danfei Xu, Josiah Wong. Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, CoRL, 2021.
[35] Diganta Misra. Mish: self regularized non-monotonic activation function. In 31 British Machine Vision Conference, BMVC, 2020.
[36] Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model as conditional planner for offline meta-rl. International Conference on Machine Learning, JCML, 2023.
[37] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models, 2021.
[38] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
[39] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
[40] William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 2212.09748, ArXiv, 2022.
[41] Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hul Chen, Zewen Li, Weinan Zhang, and Yang Yu. NeoRL: A near real-world benchmark for offline reinforcement learning. In sixth Conference on Neural Information Processing Systems Dutasets and Benchmarks Truck, NIPS, 2022.
[42] Tongzheng Ren, Jialian Li, Bo Dai, Simon S Du, and Sujay Sanghavi. Nearly horizon-free offline reinforcement learning, Advances in neural information processing systems, 34:15621-15634, 2021.
[43] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomedical image segmentation. arXiv preprint 04597, ArXiv, 2015.
[44] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-drives generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, 2023.
[45] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference Learning Representations, ICLR, 2021.
[46] Yang Song, Jascha Dobk-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, ICLR, 2021.
[47] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Trans. Neural Networks, 1998.
[48] Lei Tai, Jingwei Zhang, Ming Liu, Joschka Boedecker, and Wolfram Burgard. A survey of deep network solutions for learning control in robotics: From reinforcement to imitation. In arXiv preprint 16/2.07139, ArXiv, 2018.
[49] Thomas George Thuruthel, Egidio Falotico, Federico Renda, and Cecilia Laschi. Model-based reinforcement learning for closed-loop dynamic control of soft robotic manipulators. IEEE Transactions on Robotics, 2019.
12

---

[50] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
[51] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia. Scaling robot learning with semantically imagined experience. In Proceedings of Robotics: Science and Systems, RSS, 2023.
[52] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, ICCV, 2023.
[53] Siyuan Zhou, Yilun Du, Shun Zhang, Mengdi Xu, Yikang Shen, Wei Xiao, Yan Yeung, and Chuang Gan. Adaptive online replanning with diffusion models. In arXiv preprint 2310.09629, ArXiv, 2023.
[54] Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models. In arXiv preprint 2305.17330. ArXiv, 2023.
[55] Zhenghang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint 231 13012,3, ArXiv, 2023.
13

---

E
Figure 5: Part of selected benchmarks. From left to right, they HalfCheetah, Hopper, Walker2d, FrankaK-
itchen, Robomimic, and Antmaze.
A Details of Experimental Setup
A.1 Test Domains
Gym-MuJoCo [4] on D4RL consists of three popular offline RL locomotion tasks (Hopper, HalfChee- tah, Walker2d). These tasks require controlling three Mujoco robots to achieve maximum movement speed while minimizing energy consumption under stable conditions. D4RL provides three different quality levels of offline datasets: "medium" containing demonstrations of "medium" level perfor- mance. "medium-replay"" containing all recordings in the replay buffer observed during training until the policy reaches "medium" performance, and "medium-expp expert" which combines "medium" and "expert" level performance equally.
FrankaKitchen [13] requires controlling a realistic 9-DoF Franka robot kitchen environment to complete several common household tasks. In offline RL testing, algorithms are often evaluated on "partial" and "mixed" datasets. The former contains demonstrations that partially solve all tasks and some that do not, while the latter contains trajectories that completely solve the tasks. Therefore, these datasets place higher demands on the policy's 's "stitching" ability. During testing, the robot's s task pool includes four sub-tasks, and the evaluation score is based on the percentage of tasks completed.
Antmaze [12] requires controlling the 8-DoF "Ant" quadruped robot MuJoCo to complete maze navigation tasks. In the offline dataset, the robot only receives a reward upon reaching the endpoint, and the dataset contains many trajectory segments that do not lead to the endpoint, making it a difficult decision task with sparse rewards and long horizon. The success rate of reaching the endpoint is used as the evaluation score, and common model-free offline RL algorithms often struggle to achieve good performance.
Robomimic [34] requires learning policies to complete complex manipulation tasks from small number of human demonstrations. Due to the non-Markovian nature of human demonstrations and the demonstration quality variance, learning from human datasets is significantly more challenging than learning from machine- machine-generated generated datasets. In our experiments, we used dataset of PH (Proficient- Human) type, which consists of 200 demonstrations collected through teleoperation by one single experienced telcoperator.
FinRL [41] provides way build trading simulator that replicates the real stock market and supports backtesting with important market frictions such as transaction costs, market liquidity, and investor risk aversion, among other factors. In the FinRL environment, trading day for the stocks in the pool (30 stocks). The reward function is the difference in the total asset value between the end of the day and the day before. The environment may evolve itself as time elapses. In our experiments, the dataset has two suffixes, where "H" and "M" respectively indicate the quality of the dataset ("High" and "Medium"), and 999 indicates that the dataset includes 999 rollouts.
trade can be made per
A.2 Baselines
A.2.1 Runtime Testing
We run Diffuser 4 using the official repository from the original paper with default hyperpa- rameters.
14

---

We run DD $ using the official repository from the original paper with default hyperparame- ters.
Since the code for HDMI has not been released, made every effort to reproduce HDMI based on the Diffuserl.ite ite codebase, strictly adhering to every detail mentioned in the paper.
All runtime results are obtained on a server equipped with an Intel()) Xeon(R) Gold 6326 CPU @ 2.90GHz and an NVIDIA GeForce RTX3090.
A.2.2
Reward-maximizing
The performance of BC, CQL [25] and IQL [24] in table 2 is reported in the D4RL [12], Table 2:
The performance of Diffuser [20] in table 2, MuJoCo, is reported in [20], Table 1; The performance in Kitchen reported in [5]. Table 2: And the performance in Antmaze is obtained by the official repository from the original paper with default hyperparameters.
The performance of DD [1] in table 2, MuJoCo and Kitchen, is reported in [1]. Table I; And the performance in Antmaze reported in [19], Table
The performance of HDMI [27] in table 2 is reported in [27], Table
A.2.3
Behavior-customizing
The performance of GC (goal conditioned BC) [11] and AlignDifff [8] in table 5 is reported in [8], Table 2.
A.3 Details of AlignDiff
Why AlignDiff? Since DiffuserLite can be seen as integrating PRP into DD, the main experiments in the paper can be considered as results obtained when Diffuserl is used as a ceward-maximing algorithm plugin. Therefore, to evaluate the more general plugin capability of Diffuserl.tt Lite, non- -wwarddmaximizing algorithm backbone must be selected. AlignDiff, unlike the common setting in Offline RL, uses diffusion planning for behavior-customizne
which aligns with our requirements.
AlignDiff [8] is diffuusin planning algorithm used for cittaiir-customizing.. In our experiments, we integrate DiffuserLite as a plugin into AlignDiff, achieving a significant increase in decision-making frequency of approximately 560% with minimal performance loss (around 3.2 3.2%). This demonstrates the flexibility and effectiveness of DiffuserLite serving as a plugin. In this section, we provide detailed description of the problem setting for AlignDiff and the evaluation metric used to assess the algorithm's 's performance, aiming to help the understanding of the experimental content.
Problem setting: AlignDiff considers reward-free Markov Decision Process (MDP) denoted as M (S, A, P, ax. Here, S represents the set of states, A represents the set of actions, P S x A x S + [0, 1] is the transition function, and a {al. ou3 represents
of k predefined attributes $;-1), }. it assumes the existence of an attribute strength function that maps the trajectory to a relative strength e E [0, _] Each element of the vector indicates the relative strength of the corresponding attribute. value of 0 for go: implies the weakest manifestation of attribute a;, while value of 1 represents the strongest manifestation. Human preferences formulated as a pair of vectors (w m²), where u represents the target relative strengths, and m E 1 is binary mask indicating which attributes are of interest. The objective is to find policy = rTs) | m that minimizes the L1 norm [1 ço (E=(r')) m ||,,, where denotes the Hadamard product.
used to characterize the agent's 's behaviors. Given a state-only trajectory 7' {
vector (a2 (r') xa =
Area metric: To evaluate the algorithm's performance, the authors suggest that one can conduct multiple trials to collect the mean absolute error (MAE) between the evaluated and target relative strengths. For each trial, we need to sample an initial state sp. a target strengths eer and a mask m" " as conditions for the execution of each algorithm. Subsequently, the algorithm runs for T steps. resulting the exhibited relative strengths x evaluated by Ço. Then we can calculate the percentage of samples that fell below pre- pre-designed thresholds to create an MAE curve. The area enclosed by
15

---

the curve and the axes can be used to define an area metric. A larger metric value indicates better performance in matching. By integrating DiffuserLite, AlignDiff can maintain almost the same level of performance with only a 3.2% decrease. The overall performance is nearly twice as good as the goal-conditioned behavior clone (GC) while achieving a significant increase in decision frequency of 560%.
B Details of Rectified Flow
Similar to DMs, rectified flow [30] is also a probability flow-based generative model that learns transfer from go to gi through an ODE. In trajectory generation, we can define qi as the distribution of trajectories and qo as the standard Gaussian. The learned ODE can be represented as:
dæ = v( s)ds, initialized from 2o ~ 4o, such that æ1 ~ Q
(11)
where u R x [0, 1] + Rd a velocity field, learned by minimizing a simple mean square objective:
min E ( //'
ds -== v(x,, ds], with 2, (1 s) sx1
(12)
where y is any coupling of (qu- g1), and is parameterized deep neural network and eq. (12) is solved approximately with stochastic gradient methods. key property of rectified flow is its ability to learn straight flow, which means:
Straight flow: H, sx, (1-s)ar0 xo + su(x0, 0), Vs E [0,
(13)
A straight flow can achieve perfect results with fewer sample steps (even single step).
Reflow is iterative procedure to straighten the learned flow without modifying the marginal distributions, hence allowing faster sampling at inference time. Assume we have an ODE model da,, Us (x,, u,( s)ds with velocity field ey at the k-th iteration of the reflow procedure; denote by 21 = ODE un (#0) the x; we obtained at t = 1 when following the Un-ODE starting from 0. reflow step turns Ux into new vector field 1 that yields straighter ODEs while æ: ODE(vk++1 has the same distribution as æ ODE(vx](zo).
Uk+1 arg min gg L [1(0; 22) v( s) [dss;
with 21 ODE [ux (mo) and v, 22 + s) Eo, . (14)
where U is learned using the same rectified flow objective eq. (12), but with the linear interpolation of (mrg 21) pairs constructed from the previous ODE(vg]. For conditional sampling. rectified flow also supports blassffier-free guidance, in which we can train conditional velocity field v(z,, and apply CFG by:
nd (m,, s/c) wv s|c) + (1 - (1--w)ee w))((.,
(15)
where w is the guidance strength. In this way, we can directly replace diffusion models with rectified flow as the backbone Diffuserl.ite. In our experiments, we find that rectified flow achieves similar performance to the diffusion backbone when using the same number of sampling steps and neural network size. By further conducting reflow procedure, the planning of the model becomes more stable under the same number of sampling steps (as evidenced by decrease in the variance of experimental results). even possible to reduce the number of sampling steps to just one, resulting in only a small performance drop.
C PRP Design Insights
As the core planning framework of DiffuserLite, PRP design choices are crucial. Through experimen- tal results and discussions in section 5.5, some simple, stable, and effective PRP design insights can be summarized:
The planning horizon T supposed to be determined by the nature of the task. The sparser the rewards and the longer the episodes, the longer the planning horizon should be used.
16

---

The number of planning levels L is supposed to be designed based on the planning horizon, with longer horizon tasks requiring more planning levels. One can start from 3 or planning levels.
The temporal horizon of each level Hy is supposed to be similar and limited to within 10 to increase decision-making frequency.
Once the number of planning levels and temporal horizon are determined, the temporal jump is also determined.
Throughout our experiments, we consistently applied this set of design choices, achieving consistently excellent performance without the need for carefully adjusting parameters.
D Implementation Details
We introduce the implementation details of DiffuserLite in the section:
We utilize DiT [40] as the neural network backbone for all diffusion models and rectified flows, with an embedding dimension of 256, 8 attention heads, and DiT blocks. We progressively reduce the model size from DiT-S [40] until the current size setting, and there is still no significant performance drop. This suggests that future works could even explore further reduction of network parameters achieve faster decision speeds.
Across all the experiments, we employ DiffuserLite with 3 levels. In Kitchen, utilize planning horizon of 49 with temporal jumps for each level set to 16, 4, and 1, respectively. In MuJoCo and Antmaze, we planning horizon of 129 with temporal jumps of 32, and 1 for each level, respectively.
For diffusion models, we use cosine noise schedule [37] for a and 0, with diffusion steps T' = 1000. We employ DDIM [45] to sample trajectories. In MuJoCo and Kitchen, we use 3 sampling steps, while in Antmaze, we use 5 sampling steps.
For rectified flows, we use the Euler solver with 3 steps for all benchmarks. After one reflow procedure, we can further reduce it to 1 step for MuJoCo and 2 steps for Kitchen and Antmaze.
All models utilize the A optimizer [31] with a learning rate of 2e - 4 and weight decay of le 5. We perform 500K gradient updates with a batch size of 256. We do not employ the exponential moving average (EMA) model, as used in [20, 1]. We found that using the EMA model did not yield significant gains in our experiments. For reflow training, we first use the trained rectified flow to generate 2M dataset with 20 sampling steps. Then we use the same optimizer, but with a learning rate of 2e 5, to train the model for 200K gradient steps.
For conditional sampling, we tune the guidance strength within the range of [0, 1]. In general, a higher guidance strength leads to better performance but may result in unrealistic plans and instability. On the other hand, lower guidance strength provides more stability but may lead to a decrease in performance. In our implementation of DiffuserLite, we observe that the earlier levels are more closely related to makiin.. In contrast, the later levels only need to ensure reaching the key points provided by the previous levels. Therefore, we only apply conditional sampling to level O, while the other levels are not guided.
In MuJoCo, we only utilize the cumulative rewards as the generation condition. However, in Kitchen and Antmaze, we employ the discounted cumulative return and the value of the last generated state as the generation condition. The values are evaluated using a pre- trained IQL-value function [24]. We have found that this generation condition is beneficial for training models in reward- -wward-ppare environments, as it helps to prevent the model from becoming confused by suboptimal or poor trajectories.
The inverse dynamic model is implemented as a 3-layer MLP. The first two layers consist of a Linear layer followed by a Mish activation [35] and LayerNorm [3]. And, the final layer is followed by Tanh activation. This model utilizes the same optimizer as the diffusion models and is trained for 200K gradient steps.
17

---

Table 8: Performance of DiffuserLite under four different model sizes. All results are obtained over seeds. The results under default choice are underlined and the highest scores emphasized in bold.
Model sizes
Environment
0.68M
1.53M
2.7M
4.22M
HaltCheetah-me
75.4 5.1
86.9 = 0.6
58.5 + U.4
90.6 + 1.1
Runtime (8)
0.0153
0.0154
0.0155
0.0156
Antmaze-ld
0.3 1 0.5
60.7 2.5
68.0 2.8
73.0 4: 3.6
Runtime (8)
0.0262
0.0268
0.0270
0.0271
Kitchen-p Runtime
64.5 + 0.8
74.8 0.2
74.4 +
75.0 0.0
0.0163
0.0169
0.0170
0.0172
Table 9: Performance of DiffuserLLtt with different sampling steps. All results are obtained over seeds. The results under default choice are underlined and the highest scores are emphasized bold.
Sampling steps
Environment
10
HaliCheetah--me
74.4 1 15.7
88.5 1 0.4
89.0 1 0.7
89.2 + 1.3 0.048
Runtime (8)
0.0065
0.0155
0.026
Antmaze-Id
+ 0.9 0.0068
16.7 +
68.0 2.8
74.3 + 5.4 0.050
Runtime
0.0167
0.0270
Kitchen-p
5.7 +: 1.2
74.4 0.6
73.8 + 0.8
74.7 + 0.5
Runtime (s)
0.0068
0.0170
0.029
0.053
E Additional Experiment Results
E.1 Performance with Different Model Sizes
We compare the performance of Diffuserl.it ite using four different model sizes. Due to the use of DiT the backbone, fix the dimensions of each attention head, and the parameter design of the Transformer is as follows: [hidden_size size,, n headd,, depth] respectively set to [128, 4, 2] (0.68M), [192, 6, 2] (1.53M), [256. 8, 2] (2.7M), and [320. 10, 2] (4.22M). The results are presented in table which shows that slightly increasing the model size does not significantly decrease the inference speed, but it does improve the performance. This suggests that increasing the model parameter size can further enhance the performance of DiffuserLite.
E.2 Performance with Different Sampling Steps
We compare the performance of DiffuserLite using four different sampling steps. The results are presented in table which shows natural trade-off:: more sampling steps lead better performance but at the cost of slower decision-making speed. DiffuserLite strikes a balance between performance and speed by offering the choice of or 5 sampling steps. Researchers can flexibly adjust the sampling steps based on the specific requirements of their tasks.
E.3 Performance under Different Planning Horizon Choices
The planning horizon is an essential parameter that influences the performance of planning algorithms. To investigate the performances of DiffuserLite under different temporal horizon choices, we using three temporal horizon choices: 49, 129, and 257. The results are presented in table 10. In Hopper environment, longer planning horizon is required to avoid greedy and rapid jumps that may lead to falls. Consequently, the performance of DiffuserLite is slightly poorer under the 49 temporal horizon compared the other two choices (129 and 257), where no significant performance differences are observed. For Kitchen environment, the total length of the episode (280 time steps) the dataset poses a limitation. An excessively long planning horizon can confuse the model, as may struggle to determine the appropriate actions to take after completing all tasks. As result, the performance Diffuserl.it lit poor under the 257 temporal horizon, while no significant performance differences observed under the 49 and 129 temporal horizons.
18

---

2 1-0
L-32
(-64
(-96
Figure 6: Visual comparison between one-shot generated plans (upper) and PRP (lower) in Hopper. The ligure showcases 100 calfuuiion-geerrate plans starting from the same initial state. The one-shot generated plans directly generate all states from = 0 to t 96, while PRP utilizes three levels with temporal horizons of 97, 33, and 9, and temporal jumps of 32, 8, and 1, respectively. For case of observation, only Hopper motion every 4 steps is displayed in the figure.
Table 10: Performance Diffusert.äs under three different temporal horizon choices. results are obtained over 5 seeds and the results of default choice are emphasized in bold scores.
Temporal Horizon
Environment
48
128
256
Hopper-me
101.1 + 0.5
111.6 0.2
110.0 0.4
Hopper-m
96.6 10.9
100.9 1.1
98.9 1: 0.4
Hopper-mr
74.5 27.2
96.6 + 0.3
98.2 = 2.1
Average
9U.7
103.1
102.4
Kitchen-m
73.6 + 0.7
72.7 1.0
25.1 = 0.2
Kitchen-p
74.4 + 0.6
7.00
25.2 1 0.4
Average
74.0
73.9
25.2
E.4 Performance with without Value Condition
Diffuserl.te lite offers optional generation conditions, resulting in different critics, introduced in section To determine the suitability of these two approaches in different tasks, we conduct ablation experiments and present the results in table 11. We find that dense reward tasks, such as Hopper and HalfCheetah, the performance of both conditions is nearly identical. However, in sparse reward scenarios, such Kitchen and Antmaze, the of values demonstrates significant advantage. We present visual comparison in Antmaze in fig. 7, which shows 100 generated plans. With pure-rewards condition, it is difficult discern the endpoint location, as the planner often desires to stop at a certain point on the map. However, when using values, the planner indicates desire to move towards the endpoint. This suggests that sparse reward tasks are prone confusing the conditional generative model, leading to poor planning. The introduction of value-assisted guidance can address this issue.
E.5 Visual Comparison between One-shot Generated Plans and PRP
We visually compare the one-shot generation and PRP in Hopper, presenting the results in ap- pendix E.5. Regarding efficiency, PRP significantly reduces the length sequences that need be generated due to each level refining only the first jumpy interval of the previous level. This results noticeable increase in the forward speed of neural networks, especially for transformer- based backbone models like DiT [40]. Visually, we can more clearly perceive the difference in generated sequence lengths, and this advantage will further expand as the total planning horizon increases. In terms of quality, PRP narrows down the search space of planning, leading to better consistency among different plans, while the one-shot generated plans exhibit more significant divergence the far horizon.
19

---

Algorithm DiffuserLite Training
Input:
number of planning levels L, temporal horizon H4, temporal jump 1 and noise estimators cø, for each level E 1,...
1), dataset = {x) where æ the sequence of state-action pairs, critic C, diffusion steps T, condition mask probability p; while not done do sample batch of (x. ((a=)) from dataset D
for = 0 to L do
extract to H. and Hr from
C( - C( H..1)
I0, first state of e +- first state Ho. if l >> 0 then
sample 8; Uniform(T").
last state of f- last state Hi. end if
HigHH:: <- -.gDD:HH,,, + 0,EL
update 0, by minimizing
Eoe (Zü: 8, ((zo.HHt))) ell² with probability p else
ceo ( s)) e|l2
for end while
Algorithm DiffuserLite Inference
Input: number of planning levels L, temporal horizon H,, temporal jump Ji and noise estimators co, for each level {0, 1, L 1}. critic inverse dynamic h, diffusion steps T, current state o;;
for 1 = 0 L do sample e (0, I). first state of é1 +- or
if ( > 0 then last state of El last state of 20.Hi-1:1-1
end if
obtain To:H.: by solving diffusion ODE with DDIM solver end for
extract Ot, 0++1 from To:Hr.1::1
a, = hlo, 0c+1)
Table 11: Performance of DiffuserLite using conditions with or without values. Present only average performance varying-quality datasets, which are obtained over 5 seeds.
Condition
Hopper
HalfCheetah
Kitchen
Antmaze
Value
103.6
60.7
74.0
77.6
w/o Value
103.1
59.7
54.1
19.7
20

---

w/
w/oV
w/oV
Figure 7: Visual comparison of DiffuserLite using conditions with (upper) or without (lower) values. It displays 100 plans generated at the current state, where darker colors indicate closer proximity to the current state, and lighter colors indicate further. With the pure-rewards condition, can observe that the planned states lighter colors tend to cluster together at a certain point on the map, indicating the planner tries to stop at that non-endpoint. However, with the introduction of values, the planner can make correct long-term plans that lead to the endpoint.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See the abstract's last three sentences and the introduction's 's last paragraph in Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See the conclusion in Section 7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have the code and model checkpoints ready for release. Besides, in Appendix D, we provide sufficient implementation details for researchers to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and model checkpoints have been released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss. wttpcccpppccceeece.ccccccccccctt Seriides/Codeuuiie wwaibbiissioon www.issssoooolli cessinicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provided full details in Section 5.1, Appendix A.1 and Appendix A.2.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provided the mean and standard error several random seeds in the experimental results to demonstrate statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detailed the compute resources used for the experiments in Section 5.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /tthiicstttt/////eeeee/e httppr/://///us//eeeee htttp::/neuripsscc/pbbbbb htc//pulic/Ethicssuuiiiiiee
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics https: http:://neurippeeee http:/c/c..///ticcccctteeeeeeeee
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the conclusion in Section 7.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
25

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For all the datasets and algorithm baselines used in the paper, we have cited the original papers and provided the license, copyright information, and terms of use in the package in our code repository.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "KN9Kn8HlAU", "reviewer": "Reviewer_xeNN", "summary": "TLDR; Unlike traditional methods that generate the entire trajectory at once, this process gradually refines the plan at each stage, reducing computational costs and improving real-time performance.\n\nDiffuserLite is a lightweight diffusion planning framework designed to increase decision-making frequency in real-time applications. Traditional diffusion planning methods are limited by the high computational costs of modeling long-horizon trajectory distributions, leading to low-frequency decision-making. To overcome these limitations, DiffuserLite introduces a Progressive Refinement Process (PRP) that reduces redundant information and progressively generates fine-grained trajectories. As a result, DiffuserLite demonstrates superior performance compared to existing methods across various benchmarks such as D4RL, Robomimic, and FinRL. Additionally, DiffuserLite is designed as a flexible plugin that can be easily integrated with other diffusion planning algorithms, making it a valuable contribution to future research and practical applications.", "questions": "- What is the default model for Lite w/o PRP that you mentioned in section 5.5? You mentioned a one-level model, but the default design in Table 6 has three temporal horizons [5,5,9]. Is the entire planning horizon used as the temporal horizon? It would be helpful to clearly specify the actual temporal horizon used in the study.\n- In Figure 1 and Table 2, the scores of DD and Diffuser for the antmaze diverse dataset are recorded as 0. However, in the ICLR 2024 paper \"Reasoning with Latent Diffusion in Offline Reinforcement Learning,\" those models show higher scores for antmaze-diverse-v2. This discrepancy suggests that either a different dataset version was used or the hyperparameter settings need to be adjusted for a fair comparison.\n- Trade off between depth of searching and removing redundant information seems good idea. But is there a optimal point for this trade off? Or does it considered by task?\n- I think critic design is one the most important part in this paper, but there are lack of explanation. Were there any literature to solve this problem by modifying Critic parts?  Were there any other research that used the sum of discounted reward and discounted optimal value to solve the problems caused by sparse rewards?\n- According to Equation 10, Critic has the optimal value term. However, this approach is limited to offline methodology. Are there any alternatives or methods to overcome this limitation?\n- The paper explains that CFG is adopted instead of CG to prevent speed degradation. Given that CFG requires adjusting the target condition at each level in the multi-level structure, are there any methods to mitigate this limitation? Additionally, could you provide experimental results comparing the performance and speed of DiffuserLite using CG?\n- Table 2 shows that DiffuserLite-R1, which uses rectified flow, achieves the best performance across most metrics. However, the paper lacks a detailed analysis of these results. Could you explain the specific mechanisms through which rectified flow contributes to performance improvement?\"", "limitations": "- It is insufficient to prove the way to choose the number of planning levels and the temporal horizons. It would have been beneficial to compare the number of planning levels or temporal horizon with respect to the learning / inference frequency, analyzing the trade-offs when these factors are changed. Additionally, comparing whether it is better to choose the temporal horizon in a bottom-up or top-down manner according to the levels would have provided a more comprehensive analysis.\n- DiffuserLite may also have some societal impacts, such as expediting the deployment of robotic products, and it could potentially be utilized for military purposes.\" If they provide more examples that highlight the practical aspects of DiffuserLite, readers may more easily understand the potential of the paper.\n- For some baseline(DD and Diffuser), the results shown in Table 2 are too low to make the experimental results appear unfair. Refer to results from Table 1 of the existing study (arXiv:2309.06599) on diffusion-based models performing one-shot planning. For the same benchmarks (antmaze-diverse, kitchen-partial, kitchen-mixed), they showed better baseline performance. It is expected that even with the step-by-step planning method of this study, proper hyper-parameter tuning could achieve at least similar performance.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "A simple idea with a order of magnitude speed improvement.\n- The PRP process reduces time cost by starting with coarse goals and refines them into detailed goals.\n- It reduces unnecessary redundant planning and ensures faster inference.\n- With the speed improvement, DiffserLite shows reasonable performance.\n\nFlexibility (RQ3)\n- In the form of a plugin, it can be applied to other planning methods such as AlignDiff.", "weaknesses": "- The terms or concepts such as \"reflow\" lack adequate explanation. Although they are mentioned in the appendix, it is necessary to provide an explanation or a reference to this section in the main text. Even a basic explanation within the main part would be beneficial.\n- The abbreviation “PRP” is used to abbreviate multiple terms. On page 2, it refers to \"plan refinement process,\" while on page 4, it stands for \"progressive refined planning.\" Although these terms appear to be used interchangeably, it would be helpful to clarify this in the text.\n- It would be beneficial to have a theoretical justification for the experimental results. The concept is straightforward, but it would be helpful to explain why certain configurations of levels are better in the ablation study comparing the levels of DiffuserLite. Currently, the analysis seems too intuitive and empirical.\n- DiffuserLite adopts Conditional Filtered Guidance (CFG) instead of Conditional Guidance (CG) to prevent speed degradation. However, the adaptation of CFG requires adjusting the target condition at each level in the multi-level structure. It would be beneficial to compare the performance and speed of DiffuserLite using CG in addition to CFG.\n- Equation 10 in the paper shows that the optimal value function is added as a term in the Critic to solve zero Critic values issue in sparse reward settings for fine-grained planning. However, using this optimal value requires an offline methodology, which limits the approach. Methods studied in Online RL may replace optimal value. For example, it would be possible to consider studies on exploration strategies such as intrinsic motivation or curiosity-based reward, or value-based method enhancement studies such as double Q-learning or dueling DQN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions. I am deeply touched by the diligent review and appreciate your dedication. I summarize the issues and respond below to address your concerns:\n\n---\n- **Paper revision:** Following your suggestion, during revision, we will further introduce *Reflow* in main text, address PRP’s multiple abbreviation issue, update the D4RL score report for Diffuser/DD, and shift the design choices from Appendix to main text.\n- **About PRP configuration:** More PRP levels allow more redundant information discarded but may also accumulate errors across levels. We will provide theoretical justifications from this perspective during revision. However, due to numerical calculation and many estimations in PRP, theoretical justifications may not suggest an optimal configuration. As such, we conduct additional experiments to explore various design choices, as detailed in the following table. **This table is an expanded version of Table 6 in the paper.** You can see that the third column is the right part of Table 6. **The results align with the recommendations in Appendix C, emphasizing that 3 levels are a practical choice balancing performance and decision frequency. It is also important to balance the horizon length at each level**, avoiding overweighting at the top or bottom. Additional design choice recommendations can be found in Appendix C, and we will integrate these details into the main text based on your feedback for the revision.\n\n| Number of Levels | 2 | 3 | 4 | 5 |\n| --- | --- | --- | --- | --- |\n| Design Choice | [5,33]/[9,17]/[17,9]/[33,5] | [3,5,17]/[5,5,9]/[9,5,5]/[17,5,3] | [3,3,3,17]/[5,3,5,5]/[5,5,3,5]/[17,3,3,3] | [3,3,5,3,5] |\n| cheetah-me | 57.6/75.6/89.1/82.2 | 85.6/88.5/88.6/89.0 | 81.6/88.3/88.9/84.9 | 85.5 |\n| antmaze-ld | 0/0/69/15.3 | 34.7/68.0/67.3/34.0 | 34.3/68/70/60 | 26.7 |\n| Design Choice | [5,13]/[7,9]/[9,7]/[13,5] | [3,3,13]/[4,5,5]/[5,5,4]/[13,3,3] | [3,3,3,7]/[3,4,3,5]/[5,3,4,3]/[7,3,3,3] | [3,3,3,3,4] |\n| kitchen-p | 69/72.8/72/74.5 | 66.7/74.4/74.2/31.7 | 72.8/74.4/73.8/74 | 71.7 |\n| Average Cost | 0.0175s | 0.0209s | 0.0266s | 0.0321s |\n\n- **About CG and CFG:** We do not use CG primarily for two considerations: (a) CG requires NN gradients computation, which is very expensive. A comparison of decision frequency for DiffuserLite-D on MuJoCo, **27.2Hz (CG) vs. 68.2Hz (CFG)**, suggests that CFG should be preferred when high decision frequency is crucial. (b) Rectified flow cannot use CG for gudiance. Therefore, we solely employ CFG in Lite. Although CFG requires specifying a target condition for each level, various methods can address this: (a) using an additional NN to predict the max reachable return for the current state as the target or employing more sophisticated methods in [1], (b) we also observe that Level 0 has the most significant impact on decision performance, with most cases requiring only the tuning of target condition for Level 0.\n- **About critic design:** (Why is it helpful when sparse rewards?) Assuming $x_s$ as the planned trajectory and $\\tau$ as $x_s$ and its future trajectory, $R$ estimates the trajectory's discounted return. A critic with only reward terms optimizes $\\mathbb E_{q_0(x_0),q(\\epsilon),s,\\tau}\\left[\\Vert\\epsilon_\\theta(x_s,s,R(\\tau))-\\epsilon\\Vert^2\\right]$, whereas a critic with value terms optimizes $\\mathbb E_{q_0(x_0),q(\\epsilon),s}\\left[\\Vert\\epsilon_\\theta(x_s,s,\\mathbb E_\\tau[R(\\tau)])-\\epsilon\\Vert^2\\right]$. When rewards are sparse, a critic with value terms can receive more stable conditions during training, leading to better performance in sparse reward tasks. Additionally, **DiffuserLite does not restrict the use of any specific critic**; as demonstrated in AlignDiff-Lite, we utilized an attribute strength model [2] as a critic for preference aligning. **You can also deploy DiffuserLite in online tasks using curiosity-based rewards or value-based methods**.\n- In Section 5.5, *Lite w/ only last level* refers to one level with a horizon of 9, while *Lite w/o PRP* refers to one level with a horizon of 129. I apologize for any lack of clarity.\n- **Low Performance of Diffuser and DD in Antmaze:** In Antmaze, we run official code for Diffuser and referenced the report in [3] for DD. **We did not intentionally report low scores.** The higher scores in LDGC [4] may be due to (a) their use of an inpainting version of Diffuser as the original version yielded poor results (page 9, line 2), and (b) careful tuning of DD hyperparameters. To ensure a fair comparison, **we will update the score report during revision**. However, **Diffuser and DD still exhibit significantly lower scores than Lite.** Regarding Kitchen tasks, we have double-checked to confirm that both LDGC and our reports align with official reports.\n- **About DiffuserLite-R1:** We attribute its best performance to the straight flow property of Rectified Flow, which confers an advantage in very few sampling steps. Compared to Lite-D, we only change the generative backbone from diffusion model to rectified flow, without any other tricks.\n- **Practical Aspects:** Robomimic and FinRL provide real-world datasets with substantial differences compared to D4RL. The strong performance of Lite on these datasets underscores its potential for real-world applications.\n---\nI hope our response addresses your concerns. Please feel free to reach out if you need further clarification. Thank you once again for your insightful review.\n\n---\n[1] Yu, et al, \"Regularized Conditional Diffusion Model for Multi-Task Preference Alignment,\" in arXiv, 2404.04920, 2024.\n\n[2] Dong, et al, \"AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model,\" in The Twelfth International Conference on Learning Representations, 2023.\n\n[3] Hong, et al, \"Diffused Task-Agnostic Milestone Planner,\" in arXiv, 2312.03395, 2023.\n\n[4] Venkatraman, et al, \"Reasoning with Latent Diffusion in Offline Reinforcement Learning,\" in arXiv, 2309.06599, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vBgV8EgHYC", "reviewer": "Reviewer_XA4d", "summary": "This paper introduces DiffuserLite, a lightweight framework that utilizes progressive refinement planning to reduce redundant information generation and achieves real-time diffusion planning. \n\nKey contributions are:\n- Introduced the plan refinement process (PRP) for coarse-to-fine-grained trajectory generation, reducing the modeling of redundant information.\n- Introduced DiffuserLite, a lightweight diffusion planning framework, which significantly increases decision-making frequency by employing PRP.\n\nResults: \n- Achieved a decision-making frequency of 122.2Hz, 112.7x faster than existing frameworks and SOTA performance on multiple benchmarks, e.g. D4RL, Robomimic, and FinRL", "questions": "1. The paper discussed L = 2,3,4, what would the behavior be if L >=5? \n2. If the paper can share any any specific limitations or trade-offs the authors find during the development of this method, that could be helpful to discuss. It will guide people on how to use the method and strengthen the paper.", "limitations": "The authors addressed limitations in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: \nThis paper introduced a novel approach in diffusion planning: PRP, and leads to various benefits, resulting in SOTA in various benchmarks. \n\nQuality: \nResults are impressive. \nThe experiment design is rigorous with comprehensive evaluation across multiple benchmarks. \nThere are detailed information about how the experiments are set up and carried out. \n\nClarity: \nThe paper is well written. \nThe figures are in good quality and helps illustrate key points. \n\nSignificance: \nThe improvements and the flexibility of the method could potentially benefits real-time decision-making applications.", "weaknesses": "Details on Implementation and design choices: \nDiscussion in the main paper about design choices and hyperparameter selection could benefit this paper. More ablation study would be helpful to understand how this method can be generalized.  \n\nMany of the details are in appendix, it requires people to read though the appendix to fully understand the paper. \n\nSome of the tables can benefit from clearer presentation. e.g. table 6. While it does save some space, but it's really hard to read. \n\nComplexity and generalization: While the paper shows very impressive results on the benchmarks in the paper, the multi-level refinement process introduce quite some complexity. Additional theoretical analysis or more experimental results would help. It could be useful to provide data-driven recommendations on how to make design choices (e.g. for different levels) to make it more useful. \n\nFurther discussion on the scalability of this method to more complex, real-world scenarios would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful suggestions. I am deeply touched by your diligent review and appreciate your dedication. I summarize the questions you raised and respond below:\n\n---\n- **Implementation Details**: We have released the code at [diffuserlite/diffuserlite.github.io](https://github.com/diffuserlite/diffuserlite.github.io) to facilitate a better understanding of the implementation details. Furthermore, in response to your suggestion, we will incorporate more details on implementation and design choices into main text during revision to avoid the necessity of referring to Appendix for complete comprehension.\n\n- **Clearer Presentations**: We commit to double-checking all figures and tables during revision to ensure they are presented more clearly and address any potential ambiguity issues.\n\n- **Real-World Tests**: Robomimic and FinRL benchmarks provide real-world datasets that significantly differ from D4RL. The strong performance of DiffuserLite on these benchmarks demonstrates its potential for real-world applications. Besides, our released code allows for easy adjustments of model size and training datasets to evaluate scalability in more complex real-world scenarios. We will explore this in future work.\n\n- **PRP-Introduced Complexity**: While the methodology may appear complex, the implementation of our algorithm is not significantly more intricate than one-shot generation. After excluding components such as neural networks, generative backbones, and the reflow procedure, **the core algorithm comprises approximately 100 lines of code**, alleviating concerns regarding complexity.\n\n- **Design Choice Recommendations**: More PRP levels allow for more redundant information to be discarded but may also accumulate errors across levels. In the revision, we will provide theoretical justifications from this perspective. However, due to numerical calculation and many estimations in PRP, theoretical justifications may not suggest an optimal configuration. As such, we conduct additional experiments to explore various design choices, as detailed in the following table. **This table is an expanded version of Table 6 in the paper.** You can see that the third column is the right part of Table 6. **The results align with the recommendations in Appendix C, emphasizing that 3 levels are a practical choice balancing performance** and decision frequency, while too many levels can lead to error accumulation and reduced decision frequency, and too few levels may complicate trajectory distribution simplification, resulting in lower performance. **It is also important to balance the horizon length at each level**, avoiding overweighting at the top or bottom. Additional design choice recommendations can be found in Appendix C, and we will integrate these details into the main text based on your feedback for the revision.\n\n| Number of Levels | 2 | 3 | 4 | 5 |\n| --- | --- | --- | --- | --- |\n| Design Choice | [5,33]/[9,17]/[17,9]/[33,5] | [3,5,17]/[5,5,9]/[9,5,5]/[17,5,3] | [3,3,3,17]/[5,3,5,5]/[5,5,3,5]/[17,3,3,3] | [3,3,5,3,5] |\n| cheetah-me | 57.6/75.6/89.1/82.2 | 85.6/88.5/88.6/89.0 | 81.6/88.3/88.9/84.9 | 85.5 |\n| antmaze-ld | 0/0/69/15.3 | 34.7/68.0/67.3/34.0 | 34.3/68/70/60 | 26.7 |\n| Design Choice | [5,13]/[7,9]/[9,7]/[13,5] | [3,3,13]/[4,5,5]/[5,5,4]/[13,3,3] | [3,3,3,7]/[3,4,3,5]/[5,3,4,3]/[7,3,3,3] | [3,3,3,3,4] |\n| kitchen-p | 69/72.8/72/74.5 | 66.7/74.4/74.2/31.7 | 72.8/74.4/73.8/74 | 71.7 |\n| Average Cost | 0.0175s | 0.0209s | 0.0266s | 0.0321s |\n\n- **Performance with $L=5$**: To address your query, we include an experiment with $L=5$ in the table above (column 4). Increasing the number of levels beyond a certain point may lead to error accumulation and reduced decision frequency. Considering these factors, we still recommend using $L=3$.\n\n---\nI hope our response addresses your concerns. Please feel free to reach out if you need further clarification. Thank you once again for your insightful review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "puouMU1crz", "reviewer": "Reviewer_7Mwp", "summary": "The paper introduces a method to accelerate diffusion model-based planning called the Plan Refinement Process (PRP). This method divides the planning of the entire trajectory into several temporal spans, focusing on the most recent spans in each planning stage, thereby discarding redundant distant information. The paper also employs techniques such as critic design and rectified flow to ensure both the frequency and accuracy of planning. Experimental results show that the proposed DiffuserLite, which incorporates PRP, achieves a high decision-making frequency, which is significantly faster than predominant frameworks. This significantly enhances the real-time applicability of diffusion planning in various benchmarks. Additionally, DiffuserLite's flexible design allows it to serve as a plugin to improve other diffusion planning algorithm.", "questions": "In the Abstract, the website provided by the authors does not contain updated details and results. Please remember to update it.\nIn line 126, the authors mention that agents often struggle to reach distant states proposed by the planning process. Could you provide a deeper explanation for this? Otherwise, what is the significance of long-term planning?\nIn lines 176-178, why does this critic design make it difficult to distinguish better trajectories in short-term planning?\nWhat is the \"uniform critic\" mentioned in line 186? Could you further explain it?\nWhat are the specific metrics used in Tables 2, 3, and 4?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "One of the advantages of the paper is the combination of a coarse-to-fine architecture with diffusion model based planning. This method avoids redundant information in long-term planning and significantly improves planning efficiency. The authors also integrate classifier-free guidance (CFG), introduce a new critic, and use techniques such as rectified flow. As a result, the proposed DiffuserLite possesses the capability for fast and successful planning. Another advantage of the paper is the ability to flexibly apply the proposed method to other diffusion planning frameworks. This approach significantly increases the speed of planning inference with only a slight sacrifice in performance. The comprehensive comparisons in the paper demonstrate the potential of the proposed PRP in diffusion planning applications.", "weaknesses": "One of the primary innovations of the paper is the introduction of the Plan Refinement Process (PRP). However, this hierarchical approach is not entirely original, as similar concepts have been introduced in prior works such as HDMI and HD-DA, mentioned in the 6. Related Works section. Additionally, many of the other improvements, such as the use of rectified flow, rely on existing methods. As a result, the paper's novelty is somewhat limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. The concerns you have raised are insightful, and we aim to address them effectively in the following responses. To be concise, below, we use \"Lite\" to refer to \"DiffuserLite\".\n\n---\n**Weakness: Limited Novelty**\n\nWe aim to elucidate the differences between Lite and HDMI as well as HD-DA (It is actually a concurrent work and both it and Lite are published on arXiv in January 2024) from the perspectives of **Motivation**, **Technology**, and **Reproducibility**.\n\n**1. Motivation:** The motivation behind Lite lies in reducing redundant information during planning to increase decision frequency, whereas HDMI and HD-DA focus on generating higher-quality plans through \"subgoal planning\" and \"goal reaching\". The primary contribution of Lite lies in speed enhancement. Experimental results indicate that in terms of D4RL score, Lite outperforms HD-DA and HDMI, with a significantly higher increase in decision frequency **(Lite-R1: 52.2x $\\gg$ HDMI: 1.8x $\\approx$ HD-DA: 1.3x, compared to Diffuser)** [1,2]. We argue this disparity is due to the failure of the other two works to eliminate redundant information in planning, unlike Lite. *DiffuserLite addresses a distinct problem* and is not merely an improvement upon HDMI.\n\n**2. Technology:** While all three algorithms exhibit a hierarchical structure, Lite offers additional insights:\n- Lite demonstrates high-performance diffusion planning WITHOUT meticulous dataset handling or the inclusion of prior knowledge, unlike HDMI.\n- Lite proves that disregarding distant redundant information and using a significantly shorter planning horizon can still yield excellent results and greatly enhance speed compared to HDMI and HD-DA.\n- Lite conducts extensive ablation experiments discussing the impact of the number of hierarchical levels $L$ and interval at each level $I_l$ on performance, **summarizing best practices in hierarchical structure design**, a discussion lacking in HDMI and HD-DA (and they only use $L=2$).\n- Lite validates its algorithm on **real-world datasets** like Robomimic and FinRL, showcasing its practical applicability, which is not extensively covered in HDMI and HD-DA.\n- Lite boasts a simpler structure that can serve as a flexible plugin for other diffusion planners, as validated on AlignDiff, setting it apart from HDMI and HD-DA.\n\n**3. Reproducibility:** Currently, only DiffuserLite has released code on [Lite/Lite.github.io](https://github.com/Lite/Lite.github.io), making its results the most reliable.\n\n---\n**Questions**\n\n**Q1:** Thank you for your reminder. The website content is updated. : )\n\n**Q2:** Planning-based methods suffer from compounding errors, leading agents to deviate from the plan over time [3]. Hence, agents execute only a few steps based on the plan (typically 1 step in planning-based RL algorithms). Therefore, we believe that detailing every aspect of plan generation is unnecessary. We believe successful long-term planning requires: (a) Sparser details for more distant parts (as long as they can accurately reflect the plan's quality). (b) More details for nearer parts to ensure decision consistency with the plan.\n\n**Q3:** In lines 176-178, the described critic used in Diffuser/DD predicts the cumulative return within the horizon, leading to greedy plans, lacking of long-term judgment. For example, Hopper in Table 2 is a single-legged hopping robot, faster movement yields higher rewards. A common failure mode in Diffuser and DD is that the agent jumps abruptly and then falls, attributing this to the short-sightedness induced by greedy planning within the horizon. Critics with values can alleviate this issue by providing long-term judgment.\n\n**Q4:** In line 186, a 'uniform critic' refers to assigning the same judgment to all plans, assuming equal quality. DD generates one high-performance plan a priori for execution without a plan selection process. To align with our unified diffusion planning framework, DD can be viewed as utilizing a uniform critic for plan selection.\n\n**Q5:** The metric used in Table 2 is the normalized D4RL score, which is the episodic return normalized against expert performance (100 means expert-level [4]). The metric used in Table 3 is the success rate of manipulation tasks in Robomimic [5]. The metric used in Table 4 is the episodic return in the realistic stock market simulation [6]. Across all tables, higher values indicate better performance.\n\n---\n\nI hope our response addresses your concerns. Please feel free to reach out if you need further clarification. Thank you once again for your insightful review. Moreover, we have released the code at https://github.com/diffuserlite/diffuserlite.github.io to facilitate a better understanding of the implementation details. I hope it can help you : ) \n\n---\n### **References**\n[1] Chen, et al, \"Simple Hierarchical Planning with Diffusion,\" in The Twelfth International Conference on Learning Representations, ICLR, 2024.\n\n[2] Li, et al, \"Hierarchical Diffusion for Offline Decision Making,\" in Proceedings of the 40th International Conference on Machine Learning, ICML, 2023.\n\n[3] Xiao, et al, \"Learning to Combat Compounding-Error in Model-Based Reinforcement Learning,\" in arXiv,1912.11206, 2019.\n\n[4] Fu, et al, \"D4RL: Datasets for Deep Data-Driven Reinforcement Learning,\" in arXiv,2004.07219, 2021.\n\n[5] Mandlekar, et al, \"What Matters in Learning from Offline Human Demonstrations for Robot Manipulation,\" in Conference on Robot Learning (CoRL), 2021.\n\n[6] Liu, et al, \"FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance,\" in arXiv,2011.09607, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q4PS7hvA7o", "reviewer": "Reviewer_r5j2", "summary": "The paper introduces a lightweight framework employing a Plan Refinement Process (PRP) for generating trajectories from coarse to fine-grained levels. This approach reduces redundant information modeling, significantly enhancing planning efficiency. DiffuserLite achieves an impressive decision-making frequency of 122.2Hz, making it 112.7 times faster than existing frameworks and suitable for real-time applications. Additionally, it can be integrated as a flexible plugin to accelerate other diffusion planning algorithms. Experiments demonstrate state-of-the-art performance on multiple benchmarks, highlighting DiffuserLite's robustness, efficiency, and practical applicability across various domains.", "questions": "- The proposed  DiffuserLite is a classifier-based conditional diffusion model. How does it provide gradients?\n\n- There may exist some physical constraints for the generated trajectory, which are not differential and can not provide gradients. How does DiffuserLite work in that case?", "limitations": "The authors have adequately addressed the limitations.", "rating": 9, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- DiffuserLite achieves an impressive decision-making frequency of 122.2Hz, making it 112.7 times faster than existing frameworks, which is crucial for real-time applications.\n\n- The PRP generates trajectories from coarse to fine-grained levels, reducing redundant information modeling and significantly enhancing planning efficiency.\n\n- DiffuserLite can be integrated as a flexible plugin to accelerate other diffusion planning algorithms, demonstrating its adaptability and potential for widespread use across various domains.", "weaknesses": "The paper is well-written and easy-to-follow, I do not see obvious weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your effort in reviewing and acknowledging our work. Based on the questions you have raised, it seems you are particularly interested in the implementation details of DiffuserLite and its potential applications. We have already released the initial version of the code on [diffuserlite/diffuserlite.github.io](https://github.com/diffuserlite/diffuserlite.github.io/tree/main) to help you understand the algorithm better. And below is our response to your questions in the hope of addressing your concerns:\n\n---\n- **Q1:** Your question offers a different perspective on the planning procedure of DiffuserLite, particularly regarding treating the value critic as a classifier, which is indeed correct. This classifier plays two key roles in the planning framework: (a) guiding the Diffusion model to generate high-performance trajectories, a process that can be achieved either through gradient calculations (referred to as CG, **commonly utilizing automatic differentiation in DL software packages**) or by using the value as input to a neural network (referred to as CFG). Although mathematically equivalent, due to the high computational cost of gradient calculations that significantly reduce decision frequency, we have chosen to use CFG in DiffuserLite. Another reason for choosing CFG is that the Rectified flow backbone cannot use CG for guidance. Rectified flow is our alternative generative model backbone. To our knowledge, DiffuserLite is the first to validate its effectiveness in decision-making domains and leverage its reflow procedure further enhanced decision frequency. As Rectified flow cannot use CG, this is why we opted for CFG. To demonstrate the impact of using CG on decision frequency, **we tested DiffuserLite-D on MuJoCo benchmarks, resulting in an average decision frequency of 27.2Hz (CG) versus 68.2Hz (CFG), suggesting that CFG should be preferred when high decision frequency is crucial.**\n- **Q2:** This question is highly relevant as real-world applications often encounter non-differentiable physical constraints. As a diffusion planning algorithm, **DiffuserLite offers various ways to address this issue**: (a) Explicitly filtering out trajectories that do not meet physical constraints during planning: DiffuserLite generates multiple candidate trajectories at each decision step and evaluates their quality to select the optimal one. Similar to search methods, this procedure allows for the explicit filtering of candidates that do not adhere to constraints. (b) Incorporating additional guidance: While many physical constraints are non-differentiable, we can soften these constraints and approximate them using a neural network classifier to guide the generation of trajectories that best satisfy these constraints. Previous research has shown significant improvements in the physical plausibility of generated trajectories by utilizing gradient guidance from the environment dynamic model [1]. Given DiffuserLite's flexible, plugin-based framework, this method can be incorporated with DiffuserLite to ensure compliance with physical constraints. In conclusion, **we believe that a combined approach using (a) and (b) can lead to favorable performance in most real-world applications**. As demonstrated in the paper, Robomimic benchmark offers a real-world robot arm dataset with inherent physical constraints. With method (a), DiffuserLite has already achieved very good performance on this benchmark.\n\n---\nI hope our response addresses your concerns. Please feel free to reach out if you need further clarification. Thank you once again for your insightful review.\n\n---\n[1] Ni, et al, \"MetaDiffuser: diffusion model as conditional planner for offline meta-RL,\" in *Proceedings of the 40th International Conference on Machine Learning*, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a new diffusion-based planning scheme, which adopts a coarse-to-fine-grained generation procedure to overcome the high computational costs of modeling long-horizon trajectory distributions in the existing method. The proposed method, planning refinement process (PRP), can significantly speed up diffusion planning, while achieving reasonable performance. The strengths and weaknesses are summarized below:

**Strength:**

-	The coarse-to-fine-grained scheme introduced in this paper is interesting and intuitive.
-	The proposed method PRP has impressive speed improvement over the existing diffusion-based planning method.
-	The proposed method achieved reasonable performance on the D4RL benchmark.

**Weakness:**

-	Although the inference speed of this method can be greatly improved, it needs to train multiple diffusion models. This potentially leads to more costly training, as well as more storage requirements during the deployment phase.
-	As pointed out by one of the reviewers, the paper lacks of theoretical justification or guarantee for performance improvement. It sometimes can be tricky to choose the best configuration of planning levels and horizon length for different tasks.
-	The paper can benefit from further improving the clarity and removing ambiguity in the writing, as well as including several missing references as suggested by the reviewers. 

The major concerns of the reviewers have been addressed after the rebuttal and the paper received unanimously positive feedback from the reviewers. I hope the authors can further polish the paper in their final version based on the provided suggestions, and make it a stronger submission.
Decision: Accept (poster)