What Makes a Good Diffusion Planner for Decision Making?
Authors: Haofei Lu
Dongqi Han
Yifei Shen
Dongsheng Li
Venue: ICLR 2025 Spotlight (2025)
Keywords: Diffusion Models
Offline Reinforcement Learning
Decision Making
Planning
Primary area: reinforcement learning
TL;DR: A comprehensive empirical study about key elements underlying a good diffusion planner for deicsion making.

Abstract: Diffusion models have recently shown significant potential in solving decision-making problems, particularly in generating behavior plans -- also known as diffusion planning. While numerous studies have demonstrated the impressive performance of diffusion planning, the mechanisms behind the key components of a good diffusion planner remain unclear and the design choices are highly inconsistent in existing studies. In this work, we address this issue through systematic empirical experiments on diffusion planning in an offline reinforcement learning (RL) setting, providing practical insights into the essential components of diffusion planning. We trained and evaluated over 6,000 diffusion models, identifying the critical components such as guided sampling, network architecture, action generation and planning strategy. We revealed that some design choices opposite to the common practice in previous work in diffusion planning actually lead to better performance, e.g., unconditional sampling with selection can be better than guided sampling and Transformer outperforms U-Net as denoising network. Based on these insights, we suggest a simple yet strong diffusion planning baseline that achieves state-of-the-art results on standard offline RL benchmarks. Code: https://github.com/Josh00-Lu/DiffusionVeteran.

Published as a conference paper at ICLR 2025
WHAT MAKES A GOOD DIFFUSION PLANNER FOR DE- CISION MAKING?
Haofei Lu' Dongqi Han Yifei Shen? Dongsheng Li Tsinghua University Microsoott Research Asia luhf23@mails tesssssiinn tsinghua. edu. en (donggihan, yifei shen, Dongsheng. Lil emicrosoft.com
ABSTRACT
Diffusion models have recently shown significant potential in solving decision- making problems, particularly in generating behavior plans also known as diffu- sion planning. While numerous studies have demonstrated the impressive perfor- mance of diffusion planning, the mechanisms behind the key components of a good diffusion planner remain unclear and the design choices are highly inconsistent existing studies. In this work, we address this issue through systematic empirical experiments on diffusion planning in an offline reinforcement learning (RL) setting, providing practical insights into the essential components of diffusion planning. We trained and evaluated over 6,000 diffusion models, identifying the critical components such guided sampling, network architecture, action generation and planning strategy. We revealed that some design choices opposite to the common practice in previous work in diffusion planning actually lead better performance, e.g., unconditional sampling with selection can be better than guided sampling and Transformer outperforms U-Net denoising network. Based on these insights, we suggest a simple yet strong diffusion planning baseline that achieves state-of-the-an results on standard offline RL benchmarks.
INTRODUCTION
Decision making by learning from offline data has been fundamental approach in robotics and artificial intelligence (Bellman, 1957). It enables agents to acquire complex behaviors by observing and mimicking expert demonstrations, circumventing the need for explicit programming or exhaustive exploration. However, this paradigm faces significant challenges, particularly when dealing with long-horizon planning and high-dimensional action spaces. 'he complexity of modeling sequential dependencies and capturing the intricacies of action distributions makes it difficult to scale traditional methods (Deisenroth & Rasmussen, 2011) to more complex tasks (Parmas et al., 2018).
Recently, diffusion models have achieved remarkable success in image and video generation, demon- strating their ability to handle complex distribution and range dependencies (Ho al., 2020; Dhariwal & Nichol, 2021). Inspired by these works, several recent studies have applied diffusion models to planning sequential decisions, especially with continuous state and action spaces such as robotic manipulation tasks (Janner et al., 2022; Ajay et al., 2022; Lu et al., 2023; Li et al., 2023). The diffusion models are used to approximate the sequence of states and actions from current time step to future and by exploiting the diffusion models' conditional generation capacity such as diffusion guidance (Ho al., 2020; Ho & Salimans, 2021), the model can make plans (i.e. state trajectory) with desired properties such as reward maximization (i.e. offline reinforcement learning (Levine al., 2020)).
Despite achieving impressive performance across diverse array of tasks, there has been limited exploration into the fundamental components and mechanisms that constitute an effective diffusion planning model for decision making. Previous research exhibits lack of consistency and coherence in design choices. remains uncertain whether sub-optimal design choices might hinder the full
This work was done during the internship of Haofei Lu (luhf23 @ mails. maiissssinguuu.edu.omm at Microsoft. Research Asia. Correspondence Dongqi Han <donggihan microsoft.com?

---

Published as a conference paper at ICLR 2025
potential of diffusion models within decision dleisionn-making domains. Specifically, existing approaches have not adequately addressed essential facets such as the choice of diffusion guidance algorithm, network architecture, and whether the plan should contain states state-action pairs. This naturally raises the following fundamental question:
What makes good diffusion planner for decision making, especially offline RL?
We seek to answer the question by conducting a comprehensive empirical investigation into key design choices diffusion models for decision-making, in partieular for state-based roboties tasks. Our work contributes to the field of decision making and diffusion models in several aspects.
Comprehensive experiments: We conducted an extensive empirical study to explore what constitutes an effective diffusion planner. By training and evaluating over 6,000 models, we analyzed key components critical to decision making in diffusion planning, including guided sampling algorithms, network architectures, action generation methods, and planning strategies.
Insights and tips: We ran detailed experiments and data analysis to understand the role of each key component in constituting good diffusion planner. In particular, discovered that certain design choices, contrary to common practice in diffusion planning actually lead to better performance. Our work offers intuitive explanations and practical tips about the choices and provides insights about the strengths and limitations of diffusion planning.
A simple yet strong baseline: Building on the insights from our study, we suggest simple yet highly competitive baseline, named Diffissin Veteran (DV). This model achieves state-of- the-art performance in planning tasks in standard offline RL benchmarks.
BACKGROUND AND RELATED WORK
Offline Reinforcement Learning (Fujimoto et al., 2019; Levine et al., 2020; Fu al., 2020) is subfield of reinforcement learning (RL) where the agent learns from a fixed dataset of past experiences. This dataset typically consists of saat-aation-rewaadd-ee----s interactions with the environment. The challenge in offline RL is for the agent to derive an effective policy from this static dataset without further exploration interaction with the environment. Two major challenges arise in this context. First, the state and action spaces may be high-dimensional and involve long-range dependencies, making it difficult to model effectively (Levine et al., 2020). Second, the learned policy must be optimal, even though the behavior policy that generated the offline data may sub-optimal or different from the desired policy (Fujimoto et al., 2019).
tuples, which encapsulate the agent's
Recently, diffusion models have emerged as powerful framework for tasks such as image and video generation due to their ability to model complex distributions (Croitoru et al., 2023), which could mitigate the first problem. Morcover, diffusion guidance techniques (Ho et al., 2020; Ho & Salimans, 2021) allow the model to generate samples that adhere to the desired properties. The second challenge in offline RL, learning optimal policy, can be addressed by diffusion guidance techniques to produce behavior that maximizes rewards. Building on this insight, a growing body of research has explored the use of diffusion models to generate behavior trajectories, denoted as
Diffusion planning (Ajay et al., 2022; Janner et al., 2022; Liang et al., 2023; Dai et al., 2023; Yang al., 2023; Li et al., 2023; Yang al., 2023; Chen al., 2024; Dong al., 2024c) considers that at the time step t, a trajectory 7 consists of the current and subsequent I steps of state-action pairs or states:
St St+1
81+8-1 Gc+H-11
or [Sa
8111
S1HH1]]
(2.1)
ar
G,+1
There is a guidance function to model the reward, such as the immediate reward rt or the state value function v(x)) = D - " where 7 is discount factor (Sutton & Barto, 1998). In classifier guidance (CG) (Ho al., 2020), guidance network is learned simultaneously with the diffusion model, whose input is the generated trajectory and the output is accumulated rewards or value function. The gradient of the guidance network is used in the generation process of diffusion model to maximize the rewards. Examples of diffusion planning with CG are (Janner et al., 2022; Liang et al., 2023; Zhang et al., 2022). In classifier-fie free guidance (CFG) (Ho & Salimans, 2021),
2

---

Published as a conference paper at ICLR 2025
(b)
Noise Shape=[JV. .V1
$
Planning Strategy
Dense-step planning
planning steps N dimansion
[1
Network Backbonn
A
planningg
Training
Robot trajectory dataset (demonstratons)
Dencising
Transformer
U-Nec
Quided Sampling ot Aggorchmms
Trajectory Plan
59
Astion Generation
Shape=jH MP]
5g
Classitier guidance
Classitier-tran guiddane
Dofbuco model
Robat Ilastration
t+
Sampling with Selection
Future environment step the plan (ourrent actual step
action stsspp
Unconditional
Figure 1: Diffusion planning framework for decision making, (a) The generation of a sequence plan using the denoising process of diffusion model. A 3-joints robot is used illustrative example. (b) Important components and candidates in the framework. Each color corresponds to one component in the framework. star indicates the preferred choice in experiments.
takes the desired reward or value function an additional argument feed into the diffusion process. Instances are (Ajay et al., 2022; Li et al., 2023: Yang et al., 2023). However, despite some literature reviews such as Zhu et al. (2023), the field lack a systematical study to elucidate the design space of diffusion planning in offline RL with substantial experimental results.
Diffusion policy (Pearce al., 2023; Wang et al., 2023b; Hansen-Estruch al., 2023; Chen et al., 2023) is another kind of popular usage of diffusion model in decision making. The trajectory only includes at, without lookahead planning. The model is trained by combining the loss of imitation learning and model-free RI. as in classic offline RL methods (Kumar et al., 2020; Fujimoto & Gu, 2021). Diffusion policy methods hope to improve the performance of by leveraging the capacity of diffusion model to model complex distribution of actions (policy function). A recent study (Dong et al., 2024b) investigated the design space of diffusion policy, proposed that diffusion policies such DQL (Wang al., 2023b) can be computationally efficient and powerful candidate for decision-making tasks.
3
STUDY DESIGN
3.1 KEY COMPONENTS AND MECHANISMS OF DIFFUSION PLANNER
Recent pioneering work in diffusion planning (Janner et al., 2022; Ajay et al., 2022; Chen et al., 2024) has demonstrated the potential this approach in offline RL. However, the design choices in these studies vary significantly, and it remains unclear whether there is an optimal configuration for different domains. Our aim conduct a systematic analysis supported by comprehensive experimental results. To achieve this, we begin by listing key design components (excluding common deep learning hyperparameters such as learning rates) that have varied in previous studies. See Fig. 1(b) for overview.
Guided sampling algorithms: Classifier guidance (CG) (Ho et al., 2020), Classifier-free guidance (CFG) (Ho & Salimans, 2021), Monte Carlo sampling with selection (sample N unconditional trajectories and select the best, the criteria of which is given by a critic function learned simultaneously with diffusion model). Most previous diffusion planners used CG (Janner al., 2022; Wang et al., 2023a; Chen et al., 2024) or CFG (Ajay et al., 2022; Li et al., 2023; Yang et al., 2023) for offline RL.
Denoising network backbone: U-Net (Ronneberger et al., 2015); Transformer (Vaswani et al., 2017). U-Net was used in most previous diffusion planners for state- base- offline RL (Janner et al., 2022: Ajay al., 2022; Wang et al., 2023a; Li et al., 2023; Chen al., 2024)).

---

Published as a conference paper at ICLR 2025
Action Generation: Learn joint distribution of state and action and directly execute the generated action at the current step (used in, e.g. Janner et al. (2022); Liang et al. (2023)); Learn and use inverse dynamics to compute action from state plan (used in e.g., Ajay et al. (2022); Wang et al. (2023a)).
Planning strategy: Dense-step planning means the planned trajectory т (Eq. 2.1) corresponds to contiguous H steps in the environment (this is conventional setting in diffusion planning (Janner et al., 2022; Ajay et al., 2022; Lu et al., 2023)) ; Jump-step planning models H x m environment steps, where et al. (2024)).
N+ is the planning stride; Hierarchical planning (studied by Li al. (2023); Chen
Details of the implementation deferred to Appendix A and B.
3.2 EXPERIMENT PROCEDURE
Given the multitude of components involved, it challenging draw scientific conclusions directly from the collective results. Therefore, we structured our study using the following procedure: (1) Conduct comprehensive search the key components (Sect. 3.1) by combining grid search
and manual tuning to obtain the best results.
(2) Evaluate the effect of each component using the control variable method; that is, modify only
one component of the best model a time and compare it with the original.
(3) After identifying which components are important and understanding how they affect perfor-
mance, perform a deeper analysis to derive useful insights.
3.3 BENCHMARK
We conducted experiments on the D4RL, dataset (Fu et al., 2020), one of the most widely used benchmarks for offline RI. and imitation learning. The dataset covers a variety of task domains, including maze navigation, robot locomotion, robot arm manipulation, and vehicle driving, among others. For experiments, we selected three sets of behavior planning tasks that were most commonly studied in prior works in offline RL and diffusion planning (Janner et al., 2021; Ajay et al., 2022; Janner al., 2022; Liang al., 2023; Li et al., 2023; Lu et al., 2023; Chen et al., 2024). These tasks (Fig. 2) encompass both planming and control challenges, providing a comprehensive evaluation in various problem settings. The performance metric considered in this work is the standard RL objective: the average total rewards in an online testing episode.
dim(sA) 9: dim(8) - 60
dim(AA) = 2: dim(5)= =
Frank Kitchen
Maze2D
dim(AA) dim(() 29
AntMaze
Figure 2:
Rendering of the benchmarking tasks considered in this study, where dim(S) and (A) denote the dimension of the state and action spaces.
Maze2D involve navigating a 2D maze, requiring the agent to find an optimal path to a goal. These tasks are used to test planning capabilities in environments where spatial reasoning is critical.
AntMaze presents a navigation challenge with a simulated ant robot. The agent controls a multi- legged robot navigate through 2D maze, combining both locomotion and planning.
4

---

Published as a conference paper at ICLR 2025
Eav
Kitchen
Antmaze M.-div.
Maze2D
Calegory
Dataset
Mixed
Partial
avg.
L.-div.
L-play
M--play
arg.
Umaze
avg.
BC
47.5
33.8
40.7
0.0
0.0
0.0
0.0
0.0
30.3
3,8
13.0
Non-
BCQ
8.1
18.9
13.5
2.2
6.7
0.0
0.0
2.2
8.3
12.8
9.1
diffusion
CQL.
51.0
49.8
50.4
61.2
53.7
15.8
14.9
36.4
12.5
5.7
7.7
IQL
51.0
46,3
48.7
47.5
39.6
70.0
71.2
57.1
58.6
34,9
47.4
47.0
SIBC
45.4
81.3
74.0
DQL
62.6
60.5
61.6
56.6
46.4
78.6
76.6
64.6
Diffusion Policee
DQL*
82.6
87.3
140.6
159.8
IDQL
66.5
66.7
66.6
67.9
63.5
84.8
84.5
75.2
90.1
89.5
57.9
79.2
66.5
66.7
67.3
CEP
64.8
66.6
83.8
83.6
74.7
Dittuser
52.5
55.7
54.1
27.3
17.3
2.0
6.7
13.3
121.5
113.9
119.5
Diffusion Planners
AdpDfsr
51.8
55,5
53.7
8.7
5.3
6.0
12.0
8.0
167.9
129.9
135.1
144.3
DD
75.0
56.5
65.8
4.0
8.0
3.0
HD
71.7
73.3
72.5
83.6
88.7
128.4
135.6
155.8
139.9
DV (Ours)
73.6
94.0
83.8
80.0
76.4
87.4
89.0
83.2
203.6
150.7
136.6
163.6
Table 1: Normalized performance of various offline-RI. methods. Our results (DV) are averaged over 500 episode seeds. The results of other methods are obtained from literature. We omit the variance over seeds for simplicity; however, can be found in the detailed tables in Appendix D. The best average performance on each task set are marked in bold fonts. BC: vanilla imitation learning, BCQ: Fujimoto et al. (2019), CQL: Kumar et al. (2020), IQL: Kostrikov et al. (2021), SIBC: Chen et al. (2023), DQL: Wang et al. (2023b), IDQL: replicated by Dong et al. (2024b), CEP: Lu et al. (2023), Diffuser: Janner et al. (2022), AdptDfsr: Liang et al. (2023), DD: Ajay et al. (2022), HD: Chen et al. (2024).
et al. (2023), DQL* and IDQL*:
Franka Kitchen simulates a robot arm performing a variety of manipulation tasks in a kitchen environment achieve task goals across multiple stages.
4 EXPERIMENTAL RESULTS
We trained and evaluated over 6,000 diffusion models by sweeping the key components discussed in Sect. 3.1 and other hyper-parameters (See Appendix for details).
By summarizing the results from the experiments, we identified one kind of diffusion planning framework, called the Diffusion Veteran (DV). The pseudocode of DV can be found in Algorithm 1. As shown in Table I, DV outperforms all previous diffusion planning and diffusion policy methods. We hope DV will serve simple yet strong baseline for future research in diffusion planning.
Algorithm 1: Diffusion Veteran (DV) Simplified Pseudocode
Input: Planning horizon H, Dataset DD Discount factor 7, Candidate num N, Planning stride M. Initialize Diffusion Transformer Planner Diffusion Inverse dynamics Critic V
Calculate accumulated discounted returns R, Eand ~ h for every step t.
Function TRAINING:
3 Sample 8t,++M, (+HHH)) M,, 100-MM......((-1111-0 (H R2 from
Train planner 6a using 8; as condition and 8r, (H --JM as target output
Train Inverse dynamics ew using 8t, S as input, a, as target output
Train critic Ve using S1,44 (H as input, R₁ as target output
7 end
8
Function EXECUTION(S):
Randomly generate N plans using co. while fixing the first state during sampling
10
Select the best plan using critic V,
Use the inverse dynamics e, to generate action using and the next state in the best plan
end
With DV in place, we can then analyze the impact of each component in diffusion planning by looking into how each component influences its performance. Each of the following sub-sections will focus on one component that we have found to be crucial. In the end of this section, we will conclude our findings into practical tips.
5

---

Published as a conference paper at ICLR 2025
4.1
ACTION GENERATION
50%
50%
Maze2D-L
203.6
202.9
Antmaze-L-D
60.0
49.9
Maze2D-M
150
156.0
Antmaze-L-P
76.4
56.7
Maze2D-U
Antmaze-M--D
67.4
Antmaaz-MM-P
89.0
34.0
dim (A4)
Kitchen-NX 73.6
62.9
dim(xA)
Kitchen-P
94.0
53377
Separate
Joint
(.4) =
Figure 3: Comparison of performance between two action generation strategies. "Seperate" learns and inverse dynamics compute action from state plan. "Joint" means learning joint distribution of state and action and directly executing the generated action at the current step (see "action generatation" in Fig. I(b)). A straightforward conclusion drawn from the results is that "Separate" is better than "Joint" when tackling higher-dingg nigher-dimensional action spaces. The vertical dashed line indicates on-par performance.
The choice of action generation design (Sect. 3.1) remains a subject of ongoing debate within the field. On one side, the pioneering diffusion planner, Diffuser, along with subsequent studies (Janner et al., 2022; Liang et al., 2023; Chen et al., 2024), employs a diffusion model to generate the joint distribution action and state trajectories ("joint"). In contrast, studies by Ajay et al. (2022); Wang et al. (2023a); Du et al. (2024) have adopted inverse dynamics to generate actions based on planned states ("separate").
Our experimental findings favor the latter approach: Although both strategies perform comparably in simpler environments such as Maze2D, which lacks robotic control elements, the "separate" approach significantly outperforms the "joint" strategy in more complex settings like Kitchen and AntMaze, which feature robotic control and higher-dimensional action spaces.
This observed disparity may be attributed to the additional complexity introduced when modeling the joint distribution of sequential states and actions, compared to modeling only the states. This complexity becomes particularly pronounced in environments where state transitions involve more complex actions due to higher-dimensional action spaces.
We tested both diffusion models and vanilla MLP as the inverse dynamics, and found similar performance between them. We adhered to diffusion inverse dynamics (Appendix B.
4.2
PLANNING STRATEGY
Kitchen
Antimaze
Maze2D
90
80
250
20
240
120
Plrtter 23
Parner Stride
Panner Strde
Aetrage
Average
Avorage
Figure 4: Performance change of DV over planning stride. It reduces dense-step planning when Stride=1. The star indicates the choice of DV.
One crucial result we found is that jump-step planning (Sect. 3.1) is beneficial in almost cases, despite the fact that most previous work used dense-step planning. This is observed in DV (Fig. 4) and generally in diffusion planners (see Appendix D for extensive results).
An obvious benefit from jump-step planning is that with the planning steps, the model can look ahead farther. This may be crucial for planning tasks that require long-term credit assignment. The choice of stride should be related the actual clock-time interval between environment

---

Published as a conference paper at ICLR 2025
steps. Nonetheless, suggest to try jump-steps and sweep the stride. This observed phenomenon also implies that the diffusion planner should play the role of planning at a more abstract level or with longer timescale. Interestingly, this is consistent with the neuroscientific fact that the intrinsic timescale of the prefrontal cortex (higher-level planning) is longer than that of the motor cortex (low-level control) (Murray et al., 2014; Runyan al., 2017; Wang al., 2018). A recent study (Chen et al., 2024) demonstrated impressive planning performance (Table I, HD) using multi-timescale diffusion planning. Exploring the hierarchical paradigm diffusion planning could be an interesting future direction.
4.3
DENOISING NETWORK BACKBONE
(a)
(b)
Panning stride
Planning
Depeore 1ooe
11
stops
80.0
25 staps
ArtroetO Artnios Aetraaa MD Aetrize-MF
Avarage
Kitshen
Maze
25.9
353 235.2
Kitchen wecreen-parttte www.eeetiiiooo
150 S4s
Kichhn--artial-
8
steps
Kimwnn M.
Wishes
Agraae
NactD
MeadD
MastZD-J
Figure 5: Using Transformer as the backbone of denoising network. (a) Performance comparison between Transformer and U-Net. The Transformer outperforms U-Net in 8 out of 9 sub-tasks and in all main tasks. The amount of parameters in U-Net is comparable to that in Transformers. Note that the error bars Kitchen are too small visualize (See Table 10 for numerical results). (b) Visualization of attention weights of the first layer in the Transformer network during the denoising process. More plots can found Appendix
Most diffusion planners the D4RL dataset use 1-D U-Net for the denoising network. It is natural to question whether attention all you need (Vaswani et al., 2017) for diffusion planning. Thus, we examined the benefit of replacing U-Net with the Transformer architecture the backbone of the denoising model (Sect. 3.1) (see Appendix for details about network structures). The experimental results clearly support the utilization of Transformer (Fig. 5(a)) in diffusion planning, consistent with the latest trend in image and video generation (Peebles & Xie, 2023; OpenAI, 2024).
We conducted case study by looking into the attention weights of the trained Transformer in the Kitchen environment (Fig. 5(b)), which reflect the temporal credit assignment (i.e., to how many steps later should be paid attention the planning sequence). First, we that the model pays more attention to the long-range element in the trajectory compared to the short-range ones. It suggests that the long-term dependency is crucial in this task, which breaks the local inductive bias of convolutional neural networks such as U-Net. Second, an interesting finding is that the characteristic attention length consistent even with different planning stride (Sect. 4.2): 6 (attention step) x 4 (stride) Rs 25 (attention step) (stride), depicted in Fig. 5(b). It suggests that the Transformer finds the invariant correlations across the stride, contributing to the generalization performance.
More generally, found long-term attention existing in the Transformer in the other tasks well, although the attention patterns vary across different tasks. The attention patterns typically feature slashes, which attend to fixed number of steps prior, and vertical lines, which attend to key steps. We have included the attention weights visualization Appendix D. In-depth study will be needed to fully understand the role of long-term dependency and why Transformer is observed to outperform UNet in the future.
4.4 IMPACT OF NET SIZE
Since the experimental results are in favor Transformer, one may wonder whether a "scaling law" (Kaplan al., 2020) holds, in particular, whether performance scales up with model depth (Ye et al.,
7

---

Published as a conference paper at ICLR 2025
Kitchan
Expected netumn Nesuum--oou (harm))
Antmaze
Ma202D
16
250
20
18
120
Model
Model
-+B- Kitchen-
KitcensP
-e- Average
AntimaoeL-O
Netmaac-A5D
.+. Average
Mace2D
MaceODHJ
Average
Artimaze W0p
Mace23-M
Figure Performance change over depth of the Transformer network as diffusion planner. The star indicates the choice of DV.
2024). The results presented in Fig. 6 pass two clear messages: First, I-layer Transformer is not enough, except for the simplist sub-task (Maze2D-U)) Second, deeper model is not always better. This may be due to intrinsic difference between decision making and natural language processing and limitations of dataset size and quality, which requires further study to systematically address.
4.5
GUIDED SAMPLING ALGORITIMMS
(a)
(b)
Litcher-mbed-v0
Pmportioo
antmaze-LLdd
antmaze-L-pp
-1.00
-075 -0.50
-0.25
5.00
0.25
0.50
0.75
1.00
diversev2
d
Prgporion mm
0.04
kitchen-M
antmaze-M--p
1.00
0.75
-0.0
00.25
5.00
0.25
0.50
0.75
1.00
moze2d
maze2d-U
Regertenn
0.03
maze2d-L
0.02
maze2d-M
0.01
CFG
MCSS
1.00 -0.75 -0.50 -8.5 2.00
0.25
0.50
0.75
1.00
Figure 7: Analysis guided sampling algorithm. (a) Performance comparison among different guided sampling algorithms for reward maximization. (b) Histogram of the value (accumulated discounted return in the future (Zend 7 Trth normalized to [-1, 1J) of the data points in each environment. For AntMaze, the failed trajectories are omitted since their values are all O.
Another inconsistent design in previous work lies the choice of guided sampling algorithm (Sect. 3.1), which enables the diffusion planner to generate plans that perform better than the average level of the dataset. Fig. 7(a) visualizes the corresponding empirical results (normalized) in our model. We draw several conclusions from the results.
First, classifier guidance (CG) is comparable with classifier-free guidance (CFG), despite the fact that CFG generally considered better than CG in image synthesis (Ho & Salimans, 2021). A potential reason is that the target value of CFG may need to be adjusted over time since the total rewards an agent can obtain in the future may vary depending on the task stage, but we can only use fixed target value for CFG since there is no trivial solution.
Also, we observed that non-guidance can be better than guidance Monte Carlo sampling with selection (MCSS) performs overall the best, except for Franka Kitchen where MCSS lags slightly behind CFG. This an important finding since existing diffusion planners usually used CG or CFG (Chen et al., 2023; Wang al., 2023b)). To understand the potential underlying reasons, we plotted the value distribution of data each environment (Fig. 7()).. It can be seen that in Maze2D and AntMaze, there is substantial amount of optimal and near-optimal experiences, whereas in Kitchen most samples are sub-optimal (note that here the optimality is with respect to condition of diffusion model). This may explain why CFG performs better than MCSS in Kitchen. Thus can propose a hypothesis: No guidance (MCSS) can be better than guided generation (CG, CFG) if the dataset contains substantial portion of expert demonstration.
8

---

Published as a conference paper at ICLR 2025
4.6
COMPARISON DIFFUSION POLICY
Expected return (narmalized)
Non-difusion
Diflution
Maza2D
Difussion Planning
Expected return (noemelizee)
Kitchen
Polpy Pbeepin
80
150
60
100
40
50
20
8CD
COL
ICL.
Stac
DGL
IDCL
Diflueer
AdpDder
DV (Cuns)
HD
8C0
Cok
10L
SBBC
DOL
DaL
I00t
inot.
Diftwer
AdpDifar
by (Ouns
HD
Expected return (nemmelized)
AntMaze
80
Nonvdflusion
Diflusi
Dffiainn Plereong
Expecled return return (normalized)
90
MuJoCo locomation
Poloy
80
-0
20
60
8C0
Coa
SIBC
DC2
DOL.
I0GL
IDoL
Difluser
AdpDtsr
DV (Qurs)
8GQ
COL
121
SBBC
DGL
DaL
10ot.
ipat
Difuner
AdoDifer
HD
(Curs
Figure 8: Average performance of methods on different tasks. The horizontal dashed line indicates the best performance over all methods. DV (Diffusion planning) stands out in Kitchen, Maze2D, and AntMaze; while DQL (Diffusion policy) (Wang et al., 2023b) outperforms all diffusion planning methods in MuJoCo locomotion tasks. Refer to the caption of Table for method details.
Diffusion planning and diffusion policy represent two key approaches within diffusion braseed decision- making. After examining the core components of diffusion planners, turn comparison of diffusion planning and diffusion policy across different environments. The experimental results are illustrated in Fig. 8. We observed that diffusion planning outperforms diffusion policy in AntMaze, Kitchen, and Maze2D, whereas diffusion policy excels in MuJoCo locomotion tasks. The first three environments require precise goal achievement, such as positioning an object exactly, necessitating long-term plamning. This makes them well-suited diffusion planning, which generates entire trajectories in one step. Furthermore, these environments feature sparse reward structures, posing challenges for model-free RL algorithms typically used in diffusion policies (Wang et al., 2023b). In contrast, the objective in MuJoCo is simply to control agents to run faster, a task that is less related to lookahead planning and does not require intricate planning. RL loss functions help diffusion policy (Wang et al., 2023b) achieve better results in such scenarios.
4.7 VALIDATIONS ON ADROIT DATASET
To examine whether the conclusions drawn from our experiments can generalize to other tasks, we conducted experiments on the Adroit Hand dataset (Rajeswaran et al., 2018; Fu et al., 2020), which features notion-capturee human data applied realistic, high-degree-of-fee of-freedoof robotic hand, including both challenges from planning and control. It encompasses 8 challenging tasks highlighted the original paper, including as pen twirling, door opening, hammer use, and object relocation. We found that the results are consistent with our findings, supporting the generalizablity across tasks. The detailed results are deferred to Appendix C.
4.8 PRACTICAL TIPS TO TAKE HOME
Diffusion planning is most effective for tasks requiring long-term credit assignment, while diffusion policies better fit locomotion tasks that demand less long-term planning (Sect. 4.6)
Takeaway 1:
Takeaway 2: It is recommended to generate state plans with diffusion planners and inverse dynamics model to compute the corresponding actions (Sect. 4.1).
Takeaway 3: Implementing jump-step planning be highly beneficial; experimenting with different
planning strides is encouraged (Sect. 4.2).
Takeaway 4: It is worth trying Transformer the backbone of diffusion planner, especially in

---

Published as a conference paper at ICLR 2025
the tasks that require long-term lookahead planning (Sect. 4.3).
Takeaway 5: A single-layer Transformer is insufficient for effective planning (Sect. 4.4).
Takeaway 6: Larger models do not necessarily lead to better performance in diffusion planner for offline RL (Sect. 4.4).
Takeaway 7: Non-guidance approaches, such as Monte Carlo unconditional sampling with selection, can outperform classifier or classifier-free guidance when the dataset contains enough near-optimal trajectories (Sect. 4.5).
DISCUSSIONS
Synergy between diffusion planning and diffusion policy. A significant avenue for future research involves a deeper exploration of the distinctions between diffusion planning and diffusion policy. Drawing on Daniel Kahneman's 's seminal work Thinking, Fast, and Slow (Kahneman, 2011), 1), human cognitive processes are categorized into System I and System 2. Diffusion policies are analogous to System processes, as they operate rapidly and efficiently (Wang et al., 2023b), making them well-suited for tasks such as locomotion (Fig. 8) that do not require extensive deliberation or long- term planning. These policies manage routine decision making with the same efficiency as intuitive responses in human cognition. Conversely, diffusion planning mirrors System 2 thinking, charac- terized by its slower, more deliberate, and effortful nature. This approach is particularly effective for tasks that demand long-term credit assignment (Fig. 8), involving more computations to develop effective plans. In RL terminology, diffusion planning can be broadly classified as model-based, while diffusion policy aligns with model-free methodologies. Investigating the interplay between these two systems presents a compelling intersection for both machine learning and cognitive neuroscience (Gläscher al., 2010; Duan et al., 2016; Botvinick al., 2019). Studies from cognitive science indicate that the brain may use synergistic approach which arbitrates and selects the better system according to the current situation, and the preference may change over time (Lee et al., 2014; Han et al., 2024). We anticipate extensive future research focused on integrating the strengths of diffusion planning and diffusion policies to enable both efficient and effective decision-making AI.
Computational efficiency. Despite the effectiveness of diffusion planners, their computational cost substantial. Our work is orthogonal the optimization of computational cost (Dong et al., 2024a). Nonetheless, future work may consider new schemes such as the consistency model (Song et al., 2023) to improve computational efficiency.
Interpretability and safety. Our study focuses on single performance metric (total return), potentially overlooking qualitative aspects such as the interpretability and reliability of the diffusion planner. Future work may consider issues such as explainability (Puiutta & Veith, 2020) and safety (Xiao et al., 2023) of diffusion planning. Leveraging the experiences from computer vision domain will be worth investigating.
Sustainability. Our work required significant computational resources, particularly in terms of GPU energy consumption,
trained and evaluated thousands of models across diverse tasks. However, this investment in energy is not without purpose. We aim to provide a solid foundation for future research. Subsequent work can build upon our findings, reducing the need for extensive trial-and-error experimentation. In this way, our research contributes to energy efficiency in the long term, as researchers can reference our results and apply proven methods rather than duplicating essource-intensig exploratory efforts.
Open problems and future directions. In the current study, we have focused on standard Markov decision process problems (Bellman, 1957) using a popular offline RL benchmark (Fu et al.. 2020). The planning and control are based on joint states and coordinates. Numerous untouched problems exist, such as vision-based decision making (Du et al., 2024; Yang et al., 2024), goal-conditioned reinforcement learning (Liu et al., 2022: Wang et al., 2023a), partially observable environments (Schmidhuber, 1991), offline-to-online deployment (Matsushima et al., 2021), 1), and the scalability of diffusion plamning models (Kaplan et al., 2020). Future efforts are anticipated to fully address these limitations. However, even within the scope of the current work, we have found several interesting phenomena and tips that are counter to common practices. Our work should be considered as a new but solid starting point for behavior planning using decision models.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our results. To facilitate this, we include the source code of DV in the supplementary material, which is also avaliable at https:/ / github. . com/ LLuDDiffuu inweteraa.nnnnn Detailed descriptions of our experimental setup, including model architectures, training procedures, and hyperparameter settings, are provided in Appendix A and Appendix B. We have included comprchensive information the datasets used, along with any preprocessing steps, in Appendix B. For all key experiments, we have specified the evaluation protocols and metrics in Sect. 3 and provided extensive results in Appendix D. We have included the full list of hyperparameters and configurations in Appendix B.4.
ACKNOWLEDGMENT
This work is supported by Microsoft Research.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, 2022.
Richard Bellman. A Markovian decision process. Journal of Mathematics and Mechanics, pp- 679-684, 1957.
Matthew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 23(5):408 422, 2019.
Chang Chen, Fei Deng. Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401.02644 2024.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference Learning Representations, 2023.
Florinel-Alii Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9): 10869, 2023.
Yilun Dai, Mengjiao Yang. Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Picter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.00/ /1, 2023.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-basee and data-efficient approach to policy search. In Proceedings of the International Conference Machine Learning, pp. 465-472, 2011.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Zibin Dong, Jianye Hao, Yifu Yuan, Fei Ni, Yitian Wang, Pengyi Li, and Yan Zheng. Diffuserlite: Towards real-time diffusion planning. arXiv preprint arXiv:2401. 25444443434 2024a.
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yi Ma, Pengyi Li, and Yan Zheng. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. arXiv preprint arXiv:2406.09500 2024b.
Zibin Dong, Yifu Yuan, Jianye HAO, Fei Ni, Yao Mu, YAN ZHENG, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via Dehavion-customisable diffusion model. In The Twelfth International Conference Learning Representations, 2024c.
11

---

Published as a conference paper at ICLR 2025
Yilun Du, Sherry Yang, Bo Dai, Hanjum Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning, arXiv preprint arXiv:1611.02777 2016.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach offline reinforcement learning. Advances in neural information processing systems, 34:20132-200 20145, 2021.
Scott Fujimoto. David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pp. 1052-2062. PMLR, 2019.
Jan Gläscher, Nathaniel Daw, Peter Dayan, and John O' O'Doherty. States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4))585-555, 2010.
Dongqi Han, Kenji Doya, Dongsheng Li, and Jun Tani. Synergizing habits and goals with variational bayes. Nature Communications, 15(1):4461, 2024.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit g-learning actor-eritic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in Neural Information Processing Systems, volume 34, 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pp. 9902-9915. PMLR, 2022.
Daniel Kahneman. Thinking, fast and slow, macmillan, 2011.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0es 108361, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2x:14/2.69800 2014.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 179-11911 2020.
Sang Wan Lee, Shinsuke Shimojo. and John P O' O'Doherty, Neural computations underlying arbitration between model-based and model-free learning. Neuron, 81(3):687-699, 2014.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference Machine Learning, pp. 20035-20064. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiftuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877, 2023.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. coal-coonitioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08299, 2022.
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact mmegg-ggudded diffusion sampling in offline reinforcement learning. In International Conference Machine Learning, pp- 22825-22855. PMLR, 2023.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations, 2021.
John Murray, Alberto Bernacchia, David J Freedman, Ranulfo Romo, Jonathan D Wallis, Xinying Cai, Camillo Padoa-Schhe Aadoa-Schioppa, Tatiana Pasternak, Hyojung Sco, Daeyeol Lec, et al. A hierarchy of intrinsic timescales across primate cortex. Nature Neuroscience, 17(12):1661, 2014.
OpenAl. Sora. https: //ppps://///aaa////e com/ intei//orraaa.a..oooooooooa 2024.
Paavo Parmas, Carl Edward Rasmussen, Jan Peters, and Kenji Doya. PIPPS: Flexible model-based policy search robust to the curse of chaos. In International Conference on Machine Learning, pp. 4065 4074. PMLR, 2018.
Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205,, 2023.
Erika Puiutta and Eric MSP Veith. Explainable reinforcement learning: A survey, In International cross-domain conference for machine learning and knowledge extraction, pp. 77-95. Springer, 2020.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part IIl 18, pp. 234 241. Springer, 2015.
Caroline A Runyan, Eugenio Piasini, Stefano Panzeri, and Christopher D Harvey. Distinct timescales of population coding across cortex. Nature, 548(7665):92, 2017.
Jürgen Schmidhuber. Reinforcement learning in Markovian and non-Markovian environments. In Advances in Neural Information Processing Systems, pp. 500-506, 1991.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint #xXi:22010002002 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pp. 32211-32252 PMLR, 2023.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998 6008, 2017.
13

---

Published as a conference paper at ICLR 2025
Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a metu-reiiffrreemen learning system. Nature Neuroscience, 21(6):860, 2018.
Wei Wang, Dongqi Han, Xufang Luo, Yifei Shen, Charles Ling, Boyu Wang, and Dongsheng Li. Toward open-ended embodied tasks solving, In Second Agent Learning in Open-Endedness Workshop, 2023a.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023b.
Wei Xiao, Tsun-Hsuan Wang, Chuang Gan, and Daniela Rus. Safediffuser: Safe planning with diffusion probabilistic models. arXiv preprint 20X.:22306.00148, 2023.
Cheng-Fu Yang, Haoyang Xu, Te-Lin Wu, Xiaofeng Gao, Kai-Wei Chang, and Feng Gao. Planning as in-painting: A diffusion-based embodied task planning framework for environments under uncertainty. arXiv preprint arXiv:23/2.01097, 2023.
Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kael- bling, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. In The Twelfth International Conference on Learning Representations, 2024.
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of language models: Part 2.1, grade-school math and the hidden reasoning process. arXiv preprint 2rXiv:2407.20311 2024.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang. and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey, arXiv preprint 2:..023311.01223, 2023.
14

---

Published as a conference paper at ICLR 2025
A GUIDED SAMPLING ALGORITHMS
For decision making tasks, guided sampling algorithms are used to generate desired plans or actions. In this work, we compare three types of different guided sampling methods: classifier guidance (Dhari- wal & Nichol, 2021) (CG), classífier-free guidance (CFG) (Ho & Salimans, 2021), and Monte Carlo sampling from selections (MCSS).
Classifier guidance: Classifier guidance (CG) is introduced to guide the unconditional diffusion models qr (xx) to generate data over condition The conditioned score function is formulated as:
V log (x c) = V log gr (Zt) + V log Or (c/
where the second term is also know as noised classifier that predict the condition using noised data Tt. During sampling, the gradient of the classifier is then applied to the predicted noise eg (2r, l):
Eo (ate L, c) eo c () wa,, V a log qi (c/az)
where w is a weighting factor that controls the strength of the classifier guidance. For CG sampling, tuned wr in range (0.001, 10] each task.
Classifier-free guidance: To avoid training classifiers, classifier free guidance (CFG) is proposed. The main idea of CFG is train a diffusion model that can be used for both conditional noise predictor eg tg(2;, (2 i, e) and unconditional noise predictor eg (2x ():
c 1, e) = <,(m,, + w((e((2 4, co(
where co(2,, t) = ( (, Noise prediction of ( i, Ø) and ca (2 t, c) can be jointly learned by randomly discard conditioning with probability of Puncond. For decision making tasks, we can train diffusion models using condition of discounted returns, and using classifier-free guidance for better plan sampling. We can normalize the discounted return in the dataset for training. and use condition of 1 as target return for CFG sampling during inference (Ajay et al., 2022). However, experiments shows that fixing 1 as target may lead to unrealistic or unstable plans. Consequently, besides tuning the guidance strength w E [1.0, 6.0), we also tune for the best target return within range [0.5, 1.5) for each tasks to test CFG's best performance.
Monte Carlo sampling from selections: For Monte Carlo sampling from selections (MCSS), N selections are firstly sampled from an unconditional generative model as candidates. Then these candidates are evaluated with learned critic for the selection of the optimal one. One advantage for MCSS is that, it do not rely on any tpsk-specific hyperparameters for inference, such as the guidance strength w in CFG and CG, and target return in CFG. However, it needs to sample N - 1 more candidates during each decision making step.
B IMPLEMENTATION DETAILS
B.1 MODEL ARCHITECTURE
Planner: Our code is based on CleanDiffuser (Dong et al., 2024b). We examined U-Net (Ron- neberger et al., 2015) and Transformer (Vaswani et al., 2017) as the neural network backbone for all the diffusion planners. Specifically, we keep consistent with the implementation of U-NetID (Janner et al., 2022), with 5 kernel size, 2, 2) for channel multiplication, 32 base channels on MuJoCo, Kitchen and Maze2D, and 64 base channels on AntMaze. For Transformer, we use DiTID (Peebles & Xie, 2023; Dong et al., 2024c), with hidden dimension of 256, head dimension of 32, 2 DiT blocks on MuJoCo, Kitchen and Maze2D, and DiT blocks on AntMaze. All the planner diffusion models are trained with the Adam (Kingma & Ba, 2014) optimizer with learning rate of 3e 4, batch size of 128, for 1M gradient steps. All the diffusion models in this work are trained to predict the noise. However, for DD experiments Kitchen, the diffusion planner to predict the clean estimation, because it could achieve quite better performance.
Inverse dynamics: We used MLP-based diffusion model as the inverse dynamic model, whose input is the current state and the planned next-state; and the output is the action to execute. It is implemented with 3-layer MLP with additional 2-layer embedding layer and trained with 1M gradient steps. All the inverse dynamic models are trained with the Adam (Kingma & Ba. 2014) optimizer with learning rate of 3e- batch size of 128. We found that the using diffusion model
15

---

Published as a conference paper at ICLR 2025
inverse dynamics show similar performance with vanilla MLP inverse dynamics. The inverse dynamics trained on navigation tasks (Maze2D and AntMaze) are trained with policy centralization, where the original (st, St+1) + a, is modified with (0, 81+1 s,) de- We found this may improve the generalization ability of the inverse dynamics on navigation tasks.
Critic: We also implemented two types of critic models for guided sampling. The first type has the architecture of the D for the planner, with a linear output layer to produce the critic value. The second type is 2 blocks vanilla transformer, with hidden dimension of 256 as the value function, with a linear projection head on the first token output. We trained all the critic model using the Adam (Kingma & Ba, 2014) optimizer with learning rate of 3e 4. batch size of 128. The model will be trained with 200K gradient steps, if it is clean critic model Otherwise, it is trained for 1M gradient steps.
B.2 DIFFUSION SOLVER
We DDIM (Song et al., 2020) of temperature 1.0 for planner diffusion sampling, and DDPM with temperature of 0.5 for inverse dynamic action sampling. The sampling temperature is introduced to reduce sampling randomness (Ajay et al., 2022).
B.3 DATASET PRE-PROCESSING
Diffusion policy baselines (Chen et al., 2023; Wang et al., 2023b;
et al., 2023)
commonly learns policy, Q functions, and value functions using temporal difference manner, on standard transitions (8t, (et S1+1. rt). Admittedly, diffusion planners often require careful dataset pre- processing, including horizon padding, planning strides, return calculation, and truncation-termination -erriintionnn handling. An unsuccessful sequential dataset pre-processing may greatly reduce the planning ability of diffusion planners. Most planning tasks are usually sparse rewarded, and how optimality is defined, combined with different temporal credit assignment methods is also important. For MuJoCo and Kitchen, we use discount factor 7 = 0.997. For Maze2D and AntMaze, we use IQL-maze (Kostrikov al., 2021) reward shaping methods for temporal credit assignment in navigation planning tasks, where an -1 penalty is always applied to agent during every timestep. The plan trajectory is clipped 1000 steps on Maze2D. Refer to our code for more details.
B.4 FULL HYPER- PARAMETERS
We conducted several rounds of hyperparameter tuning, where each round conducted grid search on a subset of hyperparameters that we identified as most influential based on prior experiments and domain knowledge. We control this iterative process of selecting which hyperparameters to explore in each round, guided by preliminary results and insights. Table 2 displays the hyperparameters and default choices in our work.
'A clean critic model is only trained the original input data without noise, while noised critic model is trained using noised data using the diffusion model's noise schedule
16

---

Published as a conference paper at ICLR 2025
Table 2: Configuration Settings
Settings
Default
Choices
Guidance Type
MCSS
[MCSS, CG, CFG, None]
State-Action Generation
Separate
[Joint, Separate]
Advantage Weighting
True only on MuJoCo
[True, False]
Inverse Dynamic
Diffusion discount=0.997
[Diffusion, Regular]
Time Credit Assignment
(diccount-0.997, IQL-maze]
Planner Net. Backbone
Transformer
[Transformer, UNet]
UNet Channels Mult
(1, 2, 2, 2)
2, 2)
UNet Base Channels
32
116,32, 64] 256
Transformer Hidden
256
Transformer Block
[2, 8]
Planner Solver
DDIM
[DDIM, DDPM]
Planner Sampling Steps
20
20 1000000
Planner Training Steps
1000000
Planner Temperature
MCSS Candidates
50
[1, 20, 50] [4, 32, 40]
Planning Horizon
Planning Stride
[1. 4,5, 15, 25] MLP
Inverse Dynamics Net. Backbone
MLP
Inverse Dynamics Hidden
256
256 DDPM 10
Inverse Dynamics Solver
DDPM
Inverse Dynamics Sampling Steps
10
Inverse Dynamics Training Steps
1000000
1000000
Policy Temperature
0.5
0.5
17

---

Published as a conference paper at ICLR 2025
C RESULTS ON VALIDATION DATASET
In this section, we validate our insights and findings new of eight tasks, called Adroit Hand (Rajeswaran et al., 2018; Fu et al., 2020), to test the generalizability of our conclusions regarding diffusion planning derived from the main paper.
C.I EXPERIMENT SETUPS
As demonstrated in Fig. 9, there are four different types of challenging tasks the Adroit Hand environments. Each task consists of a dexterous hand attached to a free arm, which has around 30 actuated degrees freedom controlling and moving to complete different manipulation tasks, including opening the door, driving the nail, repositioning the pen orientation, and relocating the ball.
dim(A) 28; dim 39
Door
dim(AA) 26; dim(S) = 46
Hammer
dim(AA) = 24; dim(8) = 45
Pen
dim(AA) 30; dim(S) = 39
Relocate
Figure Rendering of the validation benchmarking tasks of Adroit Hand, where dim(S) and dim(AA (A) denote the dimension of the state and action spaces on each tasks.
Door The task consists of undoing the latch and swinging the door open. The latch has significant dry friction and bias torque that forces the door to remain closed. The agent leverages environmental interaction develop an understanding of the latch, no information about the latch is explicitly provided. The position of the door is randomized. The task is considered complete when the door touches the door stopper at the other end.
The task involves picking up hammer and driving nail into board. The nail position is randomized and has dry friction capable of absorbing up to 15N of force. The task is successful when the entire length of the nail is inside the board.
Hammer
Pen The task requires repositioning the blue pen to match the orientation of the green target. The base of the hand is fixed, and the target is randomized to cover all configurations. The task considered successful when the orientations match within specified tolerance.
18

---

Published as a conference paper at ICLR 2025
Relocate The task involves moving the blue ball to the green target. The positions of the ball and target are randomized over the entire workspace. The task considered successful when the object is within epsilon-bale ball of the target.
We conduct our experiments on two types of datasets: Cloned and Expert. The Cloned dataset consists of 50-50 split between demonstration data and 2,500 trajectories sampled from behaviorally cloned policy trained on these demonstrations. The demonstration data includes 25 human trajectories, which duplicated 100 times match the number of cloned trajectories. The Expert dataset comprises 5,000 trajectories sampled from an expert policy that successfully solves the task, as provided in the DAPG repository.
C.2 BASELINES
In order to better validate the performance of our diffusion veteran (DV), we also re-implement four representative baselines this of environments: (i) Diffuusion policies: DQL (Wang et al., 2023b) and IDQL (Hansen-Esttee et al., 2023); (ii) Diffusion planners: DD (Ajay et al., 2022) and Diffuser (Janner al., 2022).
C.3 EXPERIMENTAL RESULTS
In this section, we validate all our insights and findings of diffusion planning in the main paper from the same five perspectives: (1) Action Generation, (2) Planning Strategy, (3) Impact of Network Size, (4) Denoising Network Backbone, and (5) Guided Sampling Algorithms to better assess the generalizability of the main paper.
C.3.1 ACTION GENERATION
Table Results different action generation choices for diffusion planning on Adroit Hand. Data are Mean + Standard Error over 150 episode seeds.
Environment door
Dataset
Separate (DV)
Joint
cloned
1.5 0.0
15.2 + 0.4
door expert
104.7 0.5
104.7 0.2
hammer cloned
11.9 = 0.7
2.6 = 0.0
hammer expert
125.8 + 1.1
113.4 # 1.1
pen cloned
80.2 2.0
85.2 + 2.0
pen expert
122.2 # 1.8
112.7 1.8
relocate cloned
0.6 + 0.0
0.4 + 0.0
relocate expert
108.9 + 0.2
108.7 + 0.3
Average
69.5
67.9
The experimental results presented in Table 3 corroborate our previous findings regarding action generation strategies in diffusion planning. Specifically, generating state plans using diffusion planners and then computing the corresponding actions via an inverse dynamics model (the Separate (DV) approach) demonstrates superior or comparable performance involves generating actions directly. Averaged across all tasks and datasets, the Separate (DV) method achieves higher mean of 69.5 compared 67.9 17.9 for the Joint method. This overall performance gain underscores the effectiveness of decoupling state planning from action generation, allowing the diffusion model to focus modeling state distributions
the Joint approach, which
accurately.
19

---

Published as a conference paper at ICLR 2025
C.3.2 PLANNING STRATEGY
Table Results of different planning strategy choices on Adroit Hand. Data are Mean Error over 150 episode seeds.
Standard
Environment
Dataset
Stride
Stride (DV)
Stride 4
door
cloned
13.6 = 0.4
1.5 0.0
0.1 + 0.0
door expert
104.6 0.2
104.7 + 0.5
105.6 0.2
hammer cloned
3.9 + 0.2
11.9 + 0.7
3.2 + 0.8
hammer expert
112.5 # 1.2
125.8 + 1.1
125.9 + 1.5
pen cloned
81.6 + 2.0
80.2 2.0
pen expert
125.9 1.6
122.2 + 1.8
relocate cloned
0.1 + 0.0
0.6 0.0
0.0 + 0.0
relocate expert
108.0 + 0.3
108.9 + 0.2
1.9
Average
68.8
69.5
The results in Table 4 show that implementing jump-step planning strategies enhance the performance of diffusion planning. On average, planning with a stride of achieves a higher mean score compared to stride indicating the benefits of experimenting with different strides. Notably, the "pen" environment has maximum of 100 steps, which does not support planning with strides greater than 4; however, even within these constraints, stride planning demonstrates performance improvements. These findings suggest that increasing the planning stride can be beneficial for diffusion planning.
C.3.3 DENOISING NETWORK BACKBONE
Table 5: Results of different denoising network backbone choices on Adroit Hand. Data are Mean + Standard Error over 150 episode seeds.
#Model Parameters
2.64 M
3.96 M
15.80 M
63.11 M
Environment
Dataset Transformer (DV)
UNet
UNet
UNet
door cloned 1.5 + 0.0
-0.2 # 0.0
-0.2 + 0.0
1.2 + 0.4
door expert
104.7 + 0.5
-0.1 # 0.0
-0.0 + 0.0
103.7 + 0.6
hammer cloned
11.9 + 0.7
-0.2 + 0.0
-0.0 + 0.0
1.6 + 0.0
hammer expert
125.8 1.1
-0.1 0.0
-0.0 0.0
122.0 + 1.8
pen cloned
80.2 2.0
-0.7 0.2
-1.8 1.8+0... 0.3
73.4 + 5.1
pen expert
122.2 1.8
-1.3 0.1
-2.6 + 0.2
134.0 + 3.2
relocate cloned
0.6 + 0.0
-0.1 # 0.0
-0.1 + 0.0
0.0 + 0.0
relocate expert
108.9 0.2
-0.1 # 0.0
-0.1 + 0.0
106.5 + 0.9
Average
69.5
-0.4
-0.6
67.8
The results in Table show that when using a regular number of parameters, Transformers have clear advantage over UNet as the denoising backbone in diffusion planning. Specifically, the Transformer model achieves an average score of 69.5 with only 2.64 M parameters, while UNet needs significantly more parameters (up 63.11 M. around 25 times of the transformer) reach a similar performance level (average score of 67.8). This demonstrates that UNet requires multiple times the parameters match the efficiency of Transformers.
20

---

Published as a conference paper at ICLR 2025
C.3.4 IMPACT OF NETWORK SIZE
Table Performance change over depth of the Transformer network for diffusion planner on Adroit Hand. Data are Mean + Standard Error over 150 episode seeds.
#Model Parameters
1.46 M
2.64 M
3.82 M
Environment
Dataset
Depth
Depth (DV)
Depth 3
door
cloned
4.3 + 0.9
1.5 0.0
1.3 + 0.1
door expert
0.0 + 0.0
104.7 = 0.5
105.5 0.4
hammer cloned
17.0 + 2.4
11.9 + 0.7
1.0 = 0.0
hammer expert
76.1 4.9
125.8 # 1.1
126.0 1.4
pen cloned
4.3
80.2 2.0
75.7 5.4
pen expert
42.5 + 4.9
122.2 1.8
127.5 4.1
relocate cloned
0.7 + 0.1
0.6 0.0
0.0 + 0.0
relocate
expert
0.8 + 0.4
108.9 + 0.2
107.4 + 0.8
Average
23.3
69.5
68.1
The results presented Table 6 demonstrate that single-layer Transformer (Depth 1) is inadequate for effective planning, as evidenced by its significantly lower average score of 23.3 compared to deeper models. When the depth is increased to two layers (Depth 2), the performance improves markedly, achieving an average score of 69.5, However, further increasing the depth three layers (Depth 3) does not yield additional benefits; the average score slightly decreases to 68.1. These findings support our earlier observations: a single-layer Transformer insufficient for planning tasks, and simply enlarging the model does not guarantee better performance in diffusion planning for offline reinforcement learning.
C.3.5 GUIDED SAMPLING ALGORITHMS
The results Table indicate that non-guidance methods like Monte Carlo sampling with selection (MCSS) can outperform guidance-based approaches when the dataset contains sufficient near-optimal trajectories. MCSS achieves the highest average score of 69.5, surpassing classifier-free guidance (CFG) at 67.7, classifier guidance (CG) at 62.0, and unguided sampling (None) at 60.9.
Table 7: Results of different guided sampling algorithms on Adroit Hand. Data are Mean + Standard Error over 150 episode seeds.
Environment Dataset
MCSS (DV)
CFG
CG
None
door cloned
0.0
12.1 + 0.1
0.9 0.0
+ 0.1
door expert
104.7 + 0.5
103.5 + 0.4
104.3 + 0.1
103.7 4 0.2
hammer cloned
11.9 + 0.7
7.6 + 0.1
1.2 0.0
0.4 0.0
hammer expert
125.8 1.1
106.7 0.9
110.4 1.3
105.7 1.3
pen cloned
80.2 2.0
74.7 + 1.6
64.3 1.9
64.1 + 2.0
pen expert
122.2 + 1.8
128.2 1.4
107.9 + 1.9
105.8 + 1.8
relocate cloned
0.6 + 0.0
0.7 + 0.0
0.0 + 0.0
-0.0 + 0.0
relocate expert
108.9 + 0.2
108.2 0.5
107.0 + 0.3
106.8 + 0.3
Average
69.5
67.7
62.0
60.9
21

---

Published as a conference paper at ICLR 2025
C.3.6 COMPARISON WITH OTHER METHODS
Table Performance comparison with representative baselines on Adroit Hand. Data are Mean + Standard Error over 150 episode seeds.
Diffusion Policies
Diffusion Planners
Dataset
Environment
IDQL
DQL-TuneLR
DQL
Diffuser
DD
DV (Ours)
door cloned
0.6
-0.3 0.0
-0.1 + 0.0
0.1
15.4
1.0
door
expert
105.0 a 0.3
104.8 0.3
104.3 a 0.1
103.0 0.5
105.5 a 0.3
104.7 0.5
hammer cloned
0.5
0.2 0.0
0.1
1.2
1.6
11.9 0.7
hammer expert
127.6
128.3 0.1
55.9 5.2
103.
125.8
pen cloned
82.3 =:: 5.0
23.3 +: 4.0
28.3 + 4.3
61.7 à 5.0
72.0 + 4.2
80.2 2.0
pen expert
137.8 + 2.4
133.5 3.9
60.9 + 6.1
99.7 * 4.8
139.8 + 3.5
122.2 # 1.8
relocate cloned
= 0.0
+:
-0.0 +:
0.6 + 0.0
relocate expert
107.0 # 0.8
108.5 + 0.6
108.8 # 0.6
102.2 + 1.5
110.3 # 1.1
108.9 + 0.2
Average
71.0
62.3
44.8
58.9
71.2
69.5
Finally, we conduct a comprehensive comparison with our re-implemented baselines (Table 8). Notably, we find that using the default learning rate for DQL in this environment may lead to performance degradation. Therefore, perform learning rate search for DQL, select the optimal one, and denote it as DQL-TuneLR, where learning_ren rate (3e - 3, 3e 4, - 5]. For IDQL, use the officially recommended 256 candidates for high-density action value estimation. For DD, we conduct grid search over 35 possible configurations for each task, adjusting target_retus (0.4, 0.6, 0.8, 1.0, 1.2] and w_cff- (1.0, 1.5, 2.0, 2.5, 3.0. 3.5, 4.0) to guarantee its maximum performance. Our method requires only one-quarter of the candidates and no task-specific fine-tuning achieve comparable performance.
22

---

Published as a conference paper at ICLR 2025
D EXTENSIVE RESULTS
D.1 ATTENTION MAP ACROSS DIFFERENT TASKS
The following plots show some examples of the attention weights in different DDIM denoising steps in each task using Transformer backbones. We can see that a long-term dependency is generally existing the Transformer.
kitchen-partial-vo
Stride
Stride
DDIM
denoising
Step=0
Stap=5
Step=10
Step=15
Step=18
Stepa 19
step
Figure 10: Attention weights (averaged on multi-heads) of the first Transformer layer of DV on the Eitchen-Partia-vv dataset.
kitchen-miied-v0
Stride
Stride
DDIM denoising
Step=0
Step=5
Step=10
Step=15
Step=18
Step=19
step
Figure 11: Attention weights (averaged on multi multi-heads) of the first Transformer layer of DV on the Mitchen-Mixed-ee dataset.
23

---

Published as a conference paper at ICLR 2025
antmaze-large-diiereewwZ
Stride
Stride 25
DDIM denoleing
Step=0
Step=5
Step=10
Step=15 15
Stsp=18
Step=19
Figure 12: Attention weights (averaged on of the first Transformer layer of DV on the AntMaze-Large Lirge-Diverse-v2 dataset.
antmaze-large-playe
Stride
Stride 25
denoising step
Step=0
Step=5
Step-10
Step
Step-18
Step=19
Figure 13: Attention weights (averaged on multi-heads) of the first Transformer layer of on LaizzeLLage-Play--ae dataset.
antmaze-meddium-diivrrswzz
Stride
Stride 25
DDIM dencising
Step=0
Slep=5
Step=10
Step=16 15
Step= 18
Step=19
Figure 14: Attention weights (averaged multi-heads) of the first Transformer layer of DV on the dataset.
24

---

Published as a conference paper at ICLR 2025
antmaaz-meddiumpplayyyz
Stride 25
DDIM domaising step
Step=0
Slep=5
Step= 10
Step=15
Step= 18
Step= 19
Figure 15: Attention weights (averaged on multi-heads) of the first Transformer layer of DV on the AntMaze-Meddaa Medium-Play-v2 dataset.
maze2d-arrge-v
Stride
Stride
DDIM denoising
Step=0
Step=5
Step=10 10
Step=15
Step=18
Step=19
step
Figure 16: Attention weights (averaged on multi-heads) of the first Transformer layer of DV on the Maze2d-Large-vl dataset.
maze2d-medium--1
Stride
Stride = 25
DOIM dencising
Step#0
Step#5
Step#10
Step=15
Step=18
Step=19
Figure 17: Attention weights (averaged on multi-heads) of the first Transformer layer of DV on the Maze2D-Mediun-v/ dataset.
25

---

Published as a conference paper at ICLR 2025
maze2d-umaze-vf
Stride
Stride
dencleing step
Step=0
Step=5
Step=10
Step=15
Stap=18
Step=19
Figure 18: Attention weights (averaged on multi-heads) of the first Transformer layer of DV on the Maze2D-Umaze-vl datasct.
26

---

Published as a conference paper at ICLR 2025
D.2 EXTENSIVE RESULTS
Antmaze- Medium
Antmaze-Medt Medium
Antmaze- -ive-Lann Large
Antmaze- Large
Antmaze Medium
Antmaze-Me Medium
Antmaze-1 Large
Animaze-1 Hemazz-Laang Large
Maze2D
Maze2D
Maze2D
Partial
Mixed
Dataset
Maze2D
Maze2D
Maze2D
Partial
Mixed
Dataset
Average
Average
Average
Average
Average
Average
Environment
Environment
Umaze
Medium
Large
Play
Diverse
Play
Diverse
Kitchen
Kitchen
Umaze
Medium
Large
Play
Diverse
Play
Diverse
Kitchen
Kitchen
2.4
110.1 2.2
448.622.9
50.6 1.5
71.6 1.4
6..9 1.5
79.1 1.3
0.5
64.0 0.4
Stride
13
127.1 1.4
+
38.8 1.5
58.8 1.5
61.2 1.5
72.4 1.4
0.6
0.3
Stride
113.0
66.6
61.5
142,4
57.8
77.5
106.6
64.3 2.4
+.2
142.1 142.1 +2.
41.5
1.5 1.5
0.9
1.5
62.3
58.7 0.6
0.4
Stride 2/5
149.2
1.6.7
1.13
1.6
29.6
1.5
0.8
0..2
2.1
81.3
88.3 0.4
0.1
Stride 2/5
MCSS
107.4
2.4
2.1
13
43.7
6 1.3
8.8
20.6 4 1.1
59.9
0.5
61.3 0.4
Stride 4/15
163.6
136.6
150.7
203.6 1.4
79.4
1.9
1.6
0.9
2.0
83.8
9.3
0.1
Stride 4/15
1,4
1.0
81.0224
126.9 +22
174.1
1.4
59.0
1.4
1.3
Stride 8/25
1.2
1.9
1.5
1.6
1.6
2.0
8 1.8
Stride 8/25
127.3
71.3
160.9
83.2
1.5
58.8
4.4
60.8 2.3
+.0
59.6
1.5
14
58.615
1..1
64.5
0.54
63.5 6 0.4
Stride
22,8
2.4
1.7
18
57.1
5..9
0.66
6.88
54.4
84.3
0.4
72.7 0.2
Stride
2.3
45.2
2.1
57.0
2..0
1.68
5..0
0.4
Stride 2/5
114.1
2.2
21,6
0.7
0.1
0.4
0.1
0.3
0.1
Stride 2/5
48.7
41.4
59.8
80.5
25.8
88.6
None
CFG
41.5 122
42.6227
6..0
61.6211
0.2
11.6 1.4
0.3
Stride 4/15
2.4
2.1
2.3
0.4
3..0
0.3
6.4
9.6
0.
Stride 4/15
37.2
2.4
35.1
52.0
0.5
93.5
31.9
83.8
47.3 4.3
2..
2.2 7.0
2.0
Stride 8/25
58.7
153.4
4.
56.4 0.7 0.5
0.3
Stride 8/25
2.4
43.4
68.3
84.7
2.5
50.6
Table 9: The effect of guided sampling algorithms for DV, with different planning strides. In "Stride x/y", x is for Kitchen, and y is for Maze2D & AntMaze. Data are Mean + Standard Error over 500
episode seeds.
27

---

Published as a conference paper at ICLR 2025
Antmaze- Medium
Antmaze- Medium
Antmaze-1 Large
Antmaze- Large
Maze2D
Maze2D
Maze2D
Partial
Mixed
Dataset
Average
Average
Average
Environment
Umaze
Medium
Large
Play
Diverse
Play
Diverse
Kitchen
Kitchen
1.4
2.8.2
1.5
58.8 1.5
61.2
1
87.8 0.6
0.3
Stride
142.4
57.8
77.5
149,2
1.6
1.2.3
188.5
29.6
1.22
0.8
1..2
2.6
81.3
0.4
0.1
Stride 2/5
Transformer (DiTID)
136.6 1.3
150.7 1.0
203.6 1.4
13.
86.2
7..2
2.0
0.3
73.6
Stride 4/15
163.6
79.4
83.8
133.1 112
4.9
198.1
89.0 1.6
87.4
76.42
80.0 1.8
Stride 8/25
160.9
83.2
126.4 1.4
140.0 1.1
172.5 1.9
28.5
27.9
534415
53.7
1.5
1..6
Stride
146.3
40.9
12.0
120.0
138.7
168.7
66.2 1.4
1.5
59.9
2.6
3 0.5
Stride 2/5
142.5
57.5
1.5
32.0
U-Net (U-NetID
126.5 1.4
145.9 1.1
193.1 + 1.5
81.0 1.2
85.8
808 1.2
76.7
0.5
+.8
Stride 4/15
155.2
81.1
24.6
1214
146.6 1.0
189.7
81.9 1.2
812
72.7
68.0
Stride
152.6
75.9
8/25
Table 10: Changing denoising network backbone for DV, with different planning strides. In "Stride
x/y", x is for Kitchen, and is for Maze2D & AntMaze. Data are Mean + Standard Error over 500
episode seeds.
28

---

Published as a conference paper at ICLR 2025
Antmaze- Medium
Antmaze- Medium
Antmaze-1 Large
Antmaze- Large
Maze2D
Maze2D
Maze2D
Partial
Mixed
Dataset
Average
Average
Average
Environment
Umaze
Medium
Large
Play
Diverse
Play
Diverse
Kitchen
Kitchen
+
1.4
2.8.2
1.5
58.8 1.5
61.2
1
87.8
0.3
Stride
142.4
57.8
77.5
1.6
1.1.3
188.5
1.22
0.8
1..2
2.6
0.4
74.2
Stride 2/5
149,2
29.6
81.3
Separate
136.6 1.3
150.7 1.0
203.6 1.4
13.
86.2
7..2
2.0
0.3
73.6
Stride 4/15
163.6
79.4
83.8
133.1 1.2
198.1 4.9
89.0 1.6
87.4
2.0
80.0 1.8
Stride 8/25
160.9
83.2
124. 1.6
141.4
187.0 1.6
14
36.3
16.7
49.9
43.8 0.7
60.8 #04
Stride
151.1
44.2
52.3
139.8
143.7
202.6
2.7
2.4
0.4
0.6 0.6
53.7 0.8
62.9 0.4
Stride 2/5
162.0
1.6
58.3
Joinl
1.1
+.6
202.9
3.7
0.3
0.1
0.2
4.7
4.6
Stride 4/15
168.4
48.9
1.1
1.5
1.2
159
1.6
27.8
6.3+0.7
3.7
7.8
Stride 8/25
164.5
11.3
Table 11: The impact of action generation method for DV, with different planning strides. In "Stride x/y", x is for Kitchen, and is for Maze2D & AntMaze. Data are Mean + Standard Error over 500 episode seeds.
29

---

Published as a conference paper at ICLR 2025
Animare Medium
Antnazz-Medium
Antmare: large
Antmuze- Large
Medium- Replay
Medium-Biie Seauuu---ppeer Expert
Mediam Replay
Medium-Expss
Mediam- Moplay
Medium- Seatuu-Exppet Expert
Maze21
Maze20
Mon:2D
Pariall
Mixed
Medium
Medium
Medium
Datasel
Average
Average
Average
HaliChaetah
HalfC HallChootal bootah
HalfCheetth
Umma.e
Medium
Lange
Play
Divense
Diverse
Kitchen
Kirchea
Waller2d
Waller2d
Walker2d
Hopper
Hopper
Hopper
3.8
30.3
5
0.0
0.0
0.0
0.0
40.7
33.8
47.5
31.9
66
11.8
6.4
29.0
11.3
111.9
36.1
38.4
35.8
BC
11
9.1
12.8
6.2
2.2
0.0
0.0
2.2
13.5
18.9
8.1
55.1
15.0
57.5
54.5
1109
40.7
38.2
64.7
BCQ
None-diffusion Polises
5.7
12.5
14.9
15.8
61.2
51.0
79.2
26.7
1110
58.0
98.7
44.4
62.4
CQL
47,0
47.4
58.6
57.1
71.2
70.0
47.5
48.7
46.3
51.0
78.3
73.9
109.6
66.3
94.7
91.5
47.4
44.2
86.7
JOL
73.9 6.6
74,4
$1.
82.04.1
59.3
43.166
4.
454+16
77.925
63.155
002
+4.1411
86.2
1086=21
459422
37.1
92.6
74.0
76.6
78.6
366174
69
6266511
670009
870
9554
190
90.546
1013206
U11113
51.10S
078+03
96
64.6
10.8
61.
DQL
10.8
159,
140.6 1.0
152.0 0.8
186.8 88881117 1.7
80.5
87.
2.7
826.50
330
60.3
65. 111
10
868
00
965
1016200
1111204
525
47.
000
DQL+
Diftusion Policies
79.2
57.9
90
75.2
84.5
86.8
67.9
66.7
66.
825
86.1
1127
65.
1086
51.0
45
95.9
59,8
67.3
83.3
33.3450
48.7
M00
600
16.5
1-04
1106
110
10.
99.4
110.1
S1S2001
46.520.
913
IDOL*
7.77
11/4
4.
K3.6
K3.8
14.444
1007006
1107
98.0+2
25
34.1204
476414
935+03
CEP
9-
206
119.5
1139
121.5
123
13.3
6.7
6.7.53
20216
20
17.3
273
27.3
541
55.413
525+25
7966
06
7064
1.6
1069-02
14.314
936100
1083:13
428103
91.7400 05
889+03
144.3
135.1 1:88
129.9
9:46
167.9.50
8.0
4.0
6.0
3.3.34
87
53,7
40.4
$18+00
44126
84
94.7.4
1..7
10K
208
966622
922411
22
44.2
442006
89.6 89.008
Adaptiiliuee
Diffusion Planners
3.0
40444
4.0
0.0 00
65.
4.8
75.0+00
12.5144
130443
1088
29.3236
1010
1
4
39.441
39.5
90.6 1.3
DD
17
0.
1.399
155 2.5
335
128.4
28.4413 3.6
88.7
836
725
14
73.6
21
71.
34.
M1121
94.7-0
38.
561201
840-
99.5
467-
1672022100
10.1
1153111
92.500
HDD
136.6 1.3
|50.
203.6.1 1.4
890 1.
87.4
76.4
500 18
940
73.6
100
835
110,011
45.
9277
1636
83.3
166
818
0.1
81
0.
00
12
1D
Table 12: Normalized performance of various offline-Ru RL methods. Data are Mean + Standard Error (if available).
30


Reviews:
[{"review_id": "H7yvBKr3DR", "reviewer": "Reviewer_u6s9", "summary": "This paper analyses key components (guided sampling algorithms, network architectures, action generation methods, and planning strategies) critical to decision-making in diffusion planning.  The paper gives practical tips about the choices\nand provides insights into the strengths and limitations of diffusion planning. The experiments in the paper are very comprehensive.", "questions": "No", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The experiments in the paper are very comprehensive.", "weaknesses": "Although the experiments in the paper are rich, readers still want to see how the original innovation in theory can better apply diffusion models to decision-making tasks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ljFkHndlJ", "reviewer": "Reviewer_7fqj", "summary": "This paper presents an extensive experimental study aimed at understanding the factors that contribute to an effective diffusion planner for decision-making in offline reinforcement learning. The authors provide valuable insights into the role of various components within diffusion models. Building on these insights, they propose a straightforward yet robust diffusion planning approach that delivers state-of-the-art (SOTA) performance in standard offline RL benchmarks.", "questions": "No", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well-organized and easy to follow.\n2. The empirical analysis is comprehensive, providing solid support for the conclusions. \n3. Each conclusion is accompanied by decent explanations", "weaknesses": "While the paper provides strong evidence for the effectiveness of the proposed methods on the D4RL dataset, it is unclear how generalizable these findings are to other types of decision-making problems or datasets. More diverse datasets could strengthen the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ly06KywkUm", "reviewer": "Reviewer_ro9k", "summary": "The paper explores the design choices in diffusion model planning within offline reinforcement learning (RL). Through experiments on over 6,000 models, the paper systematically investigates key components of diffusion planning, including sampling algorithms, network architectures, action generation methods, and planning strategies. The study finds that some design choices, such as unconditional sampling outperforming guided sampling and Transformer outperforming U-Net, lead to better performance. Based on these insights, the paper proposes a simple yet strong baseline model called Diffusion Veteran (DV), which achieves state-of-the-art results on standard offline RL benchmarks.", "questions": "1.You mention that unconditional sampling outperforms guided sampling, which contrasts with results in typical image generation tasks. Could you elaborate on the underlying reasons behind this phenomenon?\n2.The paper primarily focuses on state-based tasks. Are there plans to extend the study to vision-based or goal-conditioned reinforcement learning tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.Comprehensive empirical study: The paper conducts a large-scale experimental study, using controlled variable methods to analyze the impact of each component on model performance, providing rich data support.\n2.Innovative insights: The study reveals design choices that contrast with common practices in diffusion planning, such as the advantages of unconditional sampling and the use of Transformer, offering new directions for future research.\n3.Simple yet effective baseline model: The proposed DV model is simple but performs exceptionally well, demonstrating high generalizability and effectiveness, laying a solid foundation for further research.\n4.Wide applicability: The DV model performs well in multiple tasks such as maze navigation and robot manipulation, demonstrating its adaptability and broad applicability.", "weaknesses": "1.Limited exploration of long-term dependencies: While the paper discusses the importance of handling long-term dependencies using Transformer, it does not delve deeply into how this is manifested across different tasks. The related discussion could be more robust.\n2.Potential typo in Equation 2.1: There seems to be a typo on the right-hand side of Equation 2.1, where S(t−1) appears, which might be incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5NpVpsOHpt", "reviewer": "Reviewer_fADJ", "summary": "In this paper, the authors investigated the design choices for diffusion model-based offline RL methods. \nThe design choices mainly focused on planning strategy, network architecture, guided sampling, and action generation (whether to generate both state and action directly, or generate only the state and estimate the action separately using an inverse dynamics model). \nThe tasks used in the study were Maze2D, AntMaze, and Franka kitchen (MuJoCo locomotion also used in section 4.6).", "questions": "Please refer to my previous comment on the weaknesses.\nIf either (1) validation results from tasks other than those used to investigate the design choices, or (2) validation results from 20-30 tasks were provided to support the insights on design choices, I would be inclined to recommend an Accept (assuming other reviewers do not point out any major weaknesses that I may have overlooked).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper investigates the design choices for diffusion model-based offline RL methods and identifies effective components. Although various (algorithm/implmentation) designs have been proposed in previous research on diffusion model-based offline RL methods, their effectiveness in a unified framework has not been sufficiently explored. \nIn general, the performance of reinforcement learning methods largely depends on design choices. \nTherefore, this paper, which provides insights into effective design choices, has value on the engineering front.", "weaknesses": "The number of tasks used to investigate the design choices is limited. This paper focuses on Maze2D (2 tasks), AntMaze (3 tasks), and Franka kitchen (4 tasks) (with MuJoCo locomotion tasks also included in section 4.6). However, for a paper investigating design choices, this is fewer than the number of tasks typically covered in papers accepted at ICLR (or conferences of a similar level). For instance, the paper [1] that investigated the implementation design of Offline + Online RL used 30 tasks in its study.\n\nMoreover, the paper does not verify the insights/findings on a different set of tasks (i.e., validation tasks) from those used in the design choice investigation. This leaves uncertainty about how generalizable the insights are (or whether they are simply overfitted to the tasks examined).\n\n\n[1] Ball, Philip J., et al. \"Efficient online reinforcement learning with offline data.\" International Conference on Machine Learning. PMLR, 2023.\n\n\nMinor comments:  \n\nLine 018: \n> We trained and evaluated over 6,000 diffusion models  \n\nI didn’t quite understand the breakdown of these 6,000 diffusion models. Were most of these models the ones trained and evaluated through the grid search mentioned in the step (1) in Section 3.2?  \n\nLine 174: \n> (1) Conduct a comprehensive search on the key components (Sect. 3.1) by combining grid search and manual tuning to obtain the best results.  \n\nWhat exactly does \"manual tuning\" refer to in this context?\n\nFigure 5.  \nIt seems that the Transformer score for Kitchen-M doesn’t have a confidence interval. \nAlso, I wasn’t clear on what the confidence intervals in the other parts of this figure represent (are they calculated based on 500 episode seeds?). \n\nTypoes:  \nline 101: Zhang et al., 2022) In  -> Zhang et al., 2022). In  \nline 158:  Chen et al., 2024)) -> Chen et al., 2024).  \nline 479: planning(Sect. 4.6) -> planning (Sect. 4.6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)