What Makes a Good Diffusion Planner for Decision Making?

Authors: Haofei Lu, Dongqi Han, Yifei Shen, Dongsheng Li
Primary area: reinforcement learning
Keywords: Diffusion Models, Offline Reinforcement Learning, Decision Making, Planning

Abstract:
Diffusion models have recently shown significant potential in solving decision-making problems, particularly in generating behavior plans -- also known as diffusion planning. While numerous studies have demonstrated the impressive performance of diffusion planning, the mechanisms behind the key components of a good diffusion planner remain unclear and the design choices are highly inconsistent in existing studies. In this work, we address this issue through systematic empirical experiments on diffusion planning in an offline reinforcement learning (RL) setting, providing practical insights into the essential components of diffusion planning. We trained and evaluated over 6,000 diffusion models, identifying the critical components such as guided sampling, network architecture, action generation and planning strategy. We revealed that some design choices opposite to the common practice in previous work in diffusion planning actually lead to better performance, e.g., unconditional sampling with selection can be better than guided sampling and Transformer outperforms U-Net as denoising network. Based on these insights, we suggest a simple yet strong diffusion planning baseline that achieves state-of-the-art results on standard offline RL benchmarks. Code: https://github.com/Josh00-Lu/DiffusionVeteran.

Decision: accept (spotlight)
Reviewer scores: 6, 10, 6, 8

Reviews and discussion:
<thread>
creation date:
1729499811534

thread_type:
official_review

<thread_object>
creator: reviewer_fadj

<content>
summary:
In this paper, the authors investigated the design choices for diffusion model-based offline RL methods. 
The design choices mainly focused on planning strategy, network architecture, guided sampling, and action generation (whether to generate both state and action directly, or generate only the state and estimate the action separately using an inverse dynamics model). 
The tasks used in the study were Maze2D, AntMaze, and Franka kitchen (MuJoCo locomotion also used in section 4.6).

strengths:
This paper investigates the design choices for diffusion model-based offline RL methods and identifies effective components. Although various (algorithm/implmentation) designs have been proposed in previous research on diffusion model-based offline RL methods, their effectiveness in a unified framework has not been sufficiently explored. 
In general, the performance of reinforcement learning methods largely depends on design choices. 
Therefore, this paper, which provides insights into effective design choices, has value on the engineering front.

weaknesses:
The number of tasks used to investigate the design choices is limited. This paper focuses on Maze2D (2 tasks), AntMaze (3 tasks), and Franka kitchen (4 tasks) (with MuJoCo locomotion tasks also included in section 4.6). However, for a paper investigating design choices, this is fewer than the number of tasks typically covered in papers accepted at ICLR (or conferences of a similar level). For instance, the paper [1] that investigated the implementation design of Offline + Online RL used 30 tasks in its study.

Moreover, the paper does not verify the insights/findings on a different set of tasks (i.e., validation tasks) from those used in the design choice investigation. This leaves uncertainty about how generalizable the insights are (or whether they are simply overfitted to the tasks examined).


[1] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." International Conference on Machine Learning. PMLR, 2023.


Minor comments:  

Line 018: 
> We trained and evaluated over 6,000 diffusion models  

I didn’t quite understand the breakdown of these 6,000 diffusion models. Were most of these models the ones trained and evaluated through the grid search mentioned in the step (1) in Section 3.2?  

Line 174: 
> (1) Conduct a comprehensive search on the key components (Sect. 3.1) by combining grid search and manual tuning to obtain the best results.  

What exactly does "manual tuning" refer to in this context?

Figure 5.  
It seems that the Transformer score for Kitchen-M doesn’t have a confidence interval. 
Also, I wasn’t clear on what the confidence intervals in the other parts of this figure represent (are they calculated based on 500 episode seeds?). 

Typoes:  
line 101: Zhang et al., 2022) In  -> Zhang et al., 2022). In  
line 158:  Chen et al., 2024)) -> Chen et al., 2024).  
line 479: planning(Sect. 4.6) -> planning (Sect. 4.6).

questions:
Please refer to my previous comment on the weaknesses.
If either (1) validation results from tasks other than those used to investigate the design choices, or (2) validation results from 20-30 tasks were provided to support the insights on design choices, I would be inclined to recommend an Accept (assuming other reviewers do not point out any major weaknesses that I may have overlooked).
</content>
</thread_object>

</thread>
<thread>
creation date:
1729698074292

thread_type:
official_review

<thread_object>
creator: reviewer_ro9k

<content>
summary:
The paper explores the design choices in diffusion model planning within offline reinforcement learning (RL). Through experiments on over 6,000 models, the paper systematically investigates key components of diffusion planning, including sampling algorithms, network architectures, action generation methods, and planning strategies. The study finds that some design choices, such as unconditional sampling outperforming guided sampling and Transformer outperforming U-Net, lead to better performance. Based on these insights, the paper proposes a simple yet strong baseline model called Diffusion Veteran (DV), which achieves state-of-the-art results on standard offline RL benchmarks.

strengths:
1.Comprehensive empirical study: The paper conducts a large-scale experimental study, using controlled variable methods to analyze the impact of each component on model performance, providing rich data support.
2.Innovative insights: The study reveals design choices that contrast with common practices in diffusion planning, such as the advantages of unconditional sampling and the use of Transformer, offering new directions for future research.
3.Simple yet effective baseline model: The proposed DV model is simple but performs exceptionally well, demonstrating high generalizability and effectiveness, laying a solid foundation for further research.
4.Wide applicability: The DV model performs well in multiple tasks such as maze navigation and robot manipulation, demonstrating its adaptability and broad applicability.

weaknesses:
1.Limited exploration of long-term dependencies: While the paper discusses the importance of handling long-term dependencies using Transformer, it does not delve deeply into how this is manifested across different tasks. The related discussion could be more robust.
2.Potential typo in Equation 2.1: There seems to be a typo on the right-hand side of Equation 2.1, where S(t−1) appears, which might be incorrect.

questions:
1.You mention that unconditional sampling outperforms guided sampling, which contrasts with results in typical image generation tasks. Could you elaborate on the underlying reasons behind this phenomenon?
2.The paper primarily focuses on state-based tasks. Are there plans to extend the study to vision-based or goal-conditioned reinforcement learning tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730099307998

thread_type:
official_review

<thread_object>
creator: reviewer_7fqj

<content>
summary:
This paper presents an extensive experimental study aimed at understanding the factors that contribute to an effective diffusion planner for decision-making in offline reinforcement learning. The authors provide valuable insights into the role of various components within diffusion models. Building on these insights, they propose a straightforward yet robust diffusion planning approach that delivers state-of-the-art (SOTA) performance in standard offline RL benchmarks.

strengths:
1. This paper is well-organized and easy to follow.
2. The empirical analysis is comprehensive, providing solid support for the conclusions. 
3. Each conclusion is accompanied by decent explanations

weaknesses:
While the paper provides strong evidence for the effectiveness of the proposed methods on the D4RL dataset, it is unclear how generalizable these findings are to other types of decision-making problems or datasets. More diverse datasets could strengthen the claims.

questions:
No
</content>
</thread_object>

</thread>
<thread>
creation date:
1730466700207

thread_type:
official_review

<thread_object>
creator: reviewer_u6s9

<content>
summary:
This paper analyses key components (guided sampling algorithms, network architectures, action generation methods, and planning strategies) critical to decision-making in diffusion planning.  The paper gives practical tips about the choices
and provides insights into the strengths and limitations of diffusion planning. The experiments in the paper are very comprehensive.

strengths:
The experiments in the paper are very comprehensive.

weaknesses:
Although the experiments in the paper are rich, readers still want to see how the original innovation in theory can better apply diffusion models to decision-making tasks

questions:
No
</content>
</thread_object>

</thread>
<thread>
creation date:
1732195166073

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We sincerely thank all the reviewers for reading our manuscript and providing insightful advice to improve it. Based on the constructive feedback, we have carefully revised the manuscript. The reviewers' comments have significantly contributed to enhancing the quality of the paper. In the revised manuscript, we have mainly made the following changes:

1. We included more tasks to strengthen our claims. Specifically, we conducted validation experiments on **eight new tasks from the Adroit Hand environment**. Our findings confirm that the conclusions are consistent with the new results (see Sect. 4.7 and Appendix C in the revised paper). We have also provided the corresponding source code in the supplementary material to ensure reproducibility.
2. We added further discussion about the presence of long-term dependency reflected by the attention weights in Transformers in Sect. 4.3 (and Appendix D.1).

The main changes in the manuscript are highlighted in blue.

By incorporating the reviewers' comments, the main contributions of our work are now more clearly substantiated with robust experimental results. We believe that this current work takes an intial step towards systematically understanding and applying diffusion models for decision making. We hope the extensive experiments presented in this paper will inspire future theoretical analyses and algorithm development. Please feel free to share any additional comments on the manuscript or the changes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733197080450

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Remarks

comment:
We greatly appreciate reviewers 7fqj, fADJ, ro9k, u6s9 as well as AC/SAC/PC for the dedication to the review process. As today is the final day of the rebuttal period, we would like to highlight the following points to further clarify the contributions of our work:

- While the original Diffuser paper (Janner et al., 2022) answered **whether** diffusion models can be used for planning, we addressed **how** to unleash the potential of diffusion planners. We have identified several design choices that, contrary to common practices in the literature, surprisingly enhance performance.

- Our proposed DV model achieved new SOTA results on three task sets (Kitchen, AntMaze, Maze2D) within the standard offline RL benchmark D4RL, thereby providing **a simple yet strong baseline** for future studies.
	
- Our empirical analysis dissects the components of diffusion planners. The insights gained are valuable in two ways: (1) they can **greatly reduce engineering efforts** in future experimental work, and (2) they provide **solid experimental evidence** to inform future theoretical research.

If you have **any remaining questions, please kindly let us know by the due**. Thank you once again for your valuable feedback throughout this process.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper has been well-received, with all reviewers praising its comprehensive empirical analysis, innovative insights, and the simplicity and effectiveness of the proposed DV model. The reviewers agree that the paper makes a significant contribution to the field of offline reinforcement learning by providing valuable insights into the design choices for diffusion models.

Strengths
-----------

- **Comprehensive empirical analysis:** The reviewers recognise the thoroughness of the experimental study, which uses controlled variables to analyze the impact of different components on model performance. This rigorous approach provides strong evidence for the authors' claims.

- **Innovative insights:** The paper challenges common practices in diffusion planning by demonstrating the advantages of unconditional sampling and the use of Transformer. These findings offer new directions for future research in the field.

- **Simple and effective model:** The proposed DV model is simple and effective, with strong performance across different tasks, including maze navigation and robot manipulation, suggesting high generalizability and effectiveness.

- **Clear presentation:** The paper is well-organized and easy to follow, with clear explanations for each conclusion.

Weaknesses
--------------
- **Limited task diversity:** A major concern raised by the reviewers is the limited number and type of tasks used to evaluate the proposed method. They suggest that including more diverse datasets and validation tasks would strengthen the paper's claims and demonstrate the generalizability of the findings.

- **Long-term dependencies:** While the paper discusses the importance of handling long-term dependencies, one reviewer feels that the discussion could be more in-depth, particularly regarding how this is manifested across different tasks.

- **Minor issues:** Reviewers pointed out a few minor issues like potential typos in equations and unclear explanations of confidence intervals in figures.

Based on the reviewers' feedback, I recommend the Authors consider including a discussion about long-term dependencies in the final version. For instance, providing a detailed analysis of how the model handles long-term dependencies in different tasks. Finally, I recommend addressing minor issues highlighted by Reviewers.