Diffusion-Based Planning for Autonomous Driving with Flexible Guidance

Authors: Yinan Zheng, Ruiming Liang, Kexin ZHENG, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu
Primary area: applications to robotics, autonomy, planning
Keywords: diffusion planning, autonomous driving

Abstract:
Achieving human-like driving behaviors in complex open-world environments is a critical challenge in autonomous driving. Contemporary learning-based planning approaches such as imitation learning methods often struggle to balance competing objectives and lack of safety assurance,due to limited adaptability and inadequacy in learning complex multi-modal behaviors commonly exhibited in human planning, not to mention their strong reliance on the fallback strategy with predefined rules. We propose a novel transformer-based Diffusion Planner for closed-loop planning, which can effectively model multi-modal driving behavior and ensure trajectory quality without any rule-based refinement. Our model supports joint modeling of both prediction and planning tasks under the same architecture, enabling cooperative behaviors between vehicles. Moreover, by learning the gradient of the trajectory score function and employing a flexible classifier guidance mechanism, Diffusion Planner effectively achieves safe and adaptable planning behaviors. Evaluations on the large-scale real-world autonomous planning benchmark nuPlan and our newly collected 200-hour delivery-vehicle driving dataset demonstrate that Diffusion Planner achieves state-of-the-art closed-loop performance with robust transferability in diverse driving styles.

Decision: accept (oral)
Reviewer scores: 6, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730312988810

thread_type:
official_review

<thread_object>
creator: reviewer_d65s

<content>
summary:
This paper proposes a diffusion-based motion planner for autonomous driving. The diffusion model is trained to generate joint trajectories of the ego and neighboring vehicles. Classifier guidance is introduced to regularize the safety, comfort, and feasibility of the ego vehicle's trajectory. Several practical implementation designs, such as data augmentation and normalization, are introduced to improve the model performance for closed-loop planning. Comprehensive experiments are conducted on the nuPlan benchmark.

strengths:
The proposed model is carefully designed and extensively tailored for practical application in autonomous driving. The experiments are comprehensive, with multiple baselines and detailed ablation studies. It achieves better performance than the current state-of-the-art methods on the nuPlan leaderboard. Detailed descriptions of the model implementation and experiment design are provided. This work provides useful lessons for the community on achieving good closed-loop planning performance with diffusion-based motion planners for practical autonomous driving applications.

weaknesses:
1. While I believe this work has good practical values for the autonomous driving community, diffusion models have been explored in several works for motion prediction, closed-loop simulation, and motion planning in autonomous driving. The authors attempted to differentiate their work from existing works by emphasizing that the diffusion model does not drive performance improvements in these cases but relies on rule-based refinement or LLM, and they claimed that they are the first to fully harness the potential of diffusion models to enhance closed-loop planning performance in autonomous driving. However, this statement is quite vague, and it is unclear what novel technical contributions the authors have made in this work. My current impression is that the proposed model combines existing techniques investigated in the literature on diffusion models and learning-based motion planning for autonomous driving (e.g., transformer architecture, classifier guidance with manually designed cost functions, data augmentation, etc.). To help the audience better appreciate their novelty and contributions, the authors may list their novel contributions at the end of the Introduction. 

2. Related to the first point, the authors should provide a more extensive review of the related literature on diffusion models for autonomous driving. The authors currently give a concise discussion in Sec. 3.2. It should be moved to the related work section and extended to be more comprehensive. In particular, the current review misses an essential line of research on diffusion models for closed-loop simulation (e.g., [1], [2], [3]). Also, some works have developed diffusion-based motion planners for nuPlan (e.g., [4], [5], and [6]; the authors cited [4][5], but they are not included as baselines). If possible, they should be included as baselines for comparison. 

[1] Zhong, Ziyuan, et al. "Guided conditional diffusion for controllable traffic simulation." ICRA 2023.
[2] Zhong, Ziyuan, et al. "Language-guided traffic simulation via scene-level diffusion." CoRL 2023. 
[3] Chang, Wei-Jer, et al. "Safe-sim: Safety-critical closed-loop traffic simulation with diffusion-controllable adversaries." ECCV 2024.
[4] Yang, Brian, et al. "Diffusion-es: Gradient-free planning with diffusion for autonomous driving and zero-shot instruction following." arXiv preprint arXiv:2402.06559 (2024).
[5] Hu, Yihan, et al. "Solving motion planning tasks with a scalable generative model." ECCV 2024. 
[6] Sun, Qiao, et al. "Large Trajectory Models are Scalable Motion Predictors and Planners." arXiv preprint arXiv:2310.19620 (2023).

questions:
1. In Table 1, the authors only reported the proposed diffusion planner with collision guidance. How does the model perform with other types of guidance, especially when multiple guidance costs are used? 

2. When multiple guidance terms are used, the guidance cost becomes a weighted sum of several cost terms. I wonder if the authors have taken special care to ensure that the scales of different cost terms and their gradients are well balanced. It would be great if the authors could discuss if the guidance performance is sensitive to energy function design and cost weights.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730544363742

thread_type:
official_review

<thread_object>
creator: reviewer_hoye

<content>
summary:
In this paper, a DiT-enabled framework is deployed to jointly tackle the motion planning task accompanying with predictions of surrounding agents for autonomous driving. Subsequently, a classifier-free gradient guidance are leveraged through a set of safety and comfortness costs during the diffusion step in guiding safe planning. Experimental results in Test14 demonstrate the effectiveness.

strengths:
1. A novel diffusion-based framework in solving the motion planning task. Intutive DiT-enabled framework for integrated prediction and planning with costs guidance. 

2. Strong planning results delivered against state-of-the-art baselines in nuPlan.

weaknesses:
1. Insufficient benchmark and metric comparison: 1) Additional results in other popular benchmark, such as Val14 and Test14-Hard are required to manifest the planning results under more diversed / challenging scenarios. 2) Other settings, such as closed loop reactive simulation, and open loop results are not verified.

2. Insufficient baseline comparison. Motion planning / trajectory simulation leveraging diffuision are not novel. Hence, the planning results using other diffusion strategies ([1] [2] for instance) seems needed.

Ref:

[1] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." The International Journal of Robotics Research (2023): 02783649241273668.

[2] Zhong, Ziyuan, et al. "Guided conditional diffusion for controllable traffic simulation." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

questions:
N/A.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612986461

thread_type:
official_review

<thread_object>
creator: reviewer_csii

<content>
summary:
This paper presents a novel approach for autonomous driving using a transformer-based Diffusion Planner. By leveraging diffusion models, the authors address key challenges in autonomous driving, such as modeling multi-modal driving behavior and ensuring trajectory quality without relying on rule-based refinement. The approach demonstrates state-of-the-art performance on the nuPlan benchmark and a newly collected dataset.

strengths:
The strengths of the paper are threefold.

1. The use of diffusion models in the autonomous driving planning task is novel. The authors effectively address the limitations of existing learning-based planning methods, such as handling multi-modal behaviors and out-of-distribution scenarios.

2. The paper provides a thorough explanation of the Diffusion Planner’s architecture and how it integrates prediction and planning tasks. The classifier guidance mechanism for adaptable planning behaviors is particularly well-explained.

3. The evaluations on the nuPlan benchmark and the delivery-vehicle dataset demonstrate impressive closed-loop performance, surpassing both learning-based and rule-based baselines. The method shows robust transferability, highlighting its potential for real-world applications.

weaknesses:
The overall quality of this paper is strong. One minor area for improvement is the quantity of simulation benchmarks used. The paper evaluates planner performance solely based on the Test14 random benchmark, which consists of approximately 280 closed-loop scenarios. It would be beneficial to include additional simulation benchmarks from nuPlan, such as the Val14 benchmark and the Test14 hard benchmark, to provide a more comprehensive demonstration of the advantages of the proposed method over the baselines.

questions:
The suggestion of improvement is stated in the *Weaknesses*. No further suggestions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706991738

thread_type:
official_review

<thread_object>
creator: reviewer_j7se

<content>
summary:
Current learning-based planning methods, such as imitation learning, often struggle with safety and adaptability, especially when dealing with the multi-modal behaviors typical of human drivers. To overcome these limitations, the authors introduce a novel transformer-based Diffusion Planner designed for closed-loop planning. This model captures multi-modal driving behaviors and ensures high-quality trajectories without relying on rule-based refinements. They integrate prediction and planning tasks within the same architecture to facilitate cooperative vehicle behaviors. Additionally, the Diffusion Planner enhances safety and adaptability by learning trajectory score gradients and utilizing a flexible classifier guidance mechanism.

strengths:
S1. Reduces complexity  issue by collectively considering the status of key participants in the driving scenario and jointly modeling the motion prediction and closed-loop planning tasks as a future trajectory generation task.

S2. Integrating closed-loop planning with a diffusion model is an effective approach, and the use of the architecture is clearly articulated.

weaknesses:
W1. Though PLUTO (hybrid method) performs better than Diffusion Planner in the NuPlan dataset, no comparison with PLUTO w or w/o refine is shown for the delivery-vehicle driving dataset.

W2. The paper would benefit from a more explicit and detailed statement of contributions, perhaps in a dedicated paragraph near the end of the introduction. This should clearly outline how the Diffusion Planner addresses each of the limitations mentioned and what specific novel aspects it introduces.

Minor Nitpicks

N1. A legend should be added to Appendix A, Figure 8
N2. Line 195: Conditions C could be mathematically defined

questions:
Same as weaknesses. Additional questions:

Q1. Line 466-477, “Due to space limitations, …”, what space limitations is the author mentioning?

Q2. Could the authors elaborate on W1 and explain why PLUTO (with and without refinement) was not included in the comparisons on the delivery-vehicle driving dataset, given its strong performance on NuPlan?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732032679539

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response 1/2

comment:
We thank all the reviewers for the effort engaged in the review phase and the constructive comments. 

**We have revised our paper (highlighted in blue text color). The modifications are summarized as follows.**
1. (For Reviewer J7se, Csii, hoYE, d65s). We add more extensive results on Val14 and Test14-hard benchmarks and compare them with existing diffusion-based planners in Appendix B.
2. (For Reviewer J7se, d65s). We add a detailed statement of contributions at the end of the introduction, Section 1.
3. (For Reviewer hoYE, d65s). We move the discussion of diffusion-based methods and additional traffic simulation-related work to the "Related Works", Section 2.
4. (For Reviewer J7se). We add additional explanations in Sections 5.1, 5.2, and Appendix A in response to the reviewer's request for further clarification.
5. (For Reviewer d65s). We add a case study on the use of combined collision and drivable guidance in Appendix F, along with a discussion on energy function design in Appendix C.3.

**Regarding the reviewers' common comments, we provide the following general responses**.
> **GR1. Performance on Val14 and Test14-hard in both non-reactive and reactive modes (without refine).**

We conduct additional experiments on Val14 and Test14-hard closed-loop non-reactive (NR) and reactive \(R\) results for all learning-based baselines as follows. **We have also included two recent works that use diffusion for planning [1] [2] ([1] borrows existing diffusion model with LLM, [2] released after our submission), with refinements or LLMs removed, to demonstrate the advantages of our model over existing diffusion-based methods**.

| Methods                         | Val14 (NR) | Val14 \(R\) | Test14-hard (NR) | Test14-hard \(R\) |
| ------------------------------- | ---------- | --------- | ---------------- | --------------- |
| PDM-Open*                       | 53.53      | 54.24     | 33.51            | 35.83           |
| UrbanDriver                     | 68.57      | 64.11     | 50.40            | 49.95           |
| GameFormer w/o refine.          | 13.32      | 8.69      | 7.08             | 6.69            |
| PlanTF                          | 84.72      | 76.95     | 69.70            | 61.61           |
| PLUTO w/o refine.*              | 88.89      | 78.11     | 70.03            | 59.74           |
| [1] Diffusion-es w/o LLM        | 50.00      | -         | -                | -               |
| [2] STR2-CPKS-800M w/o refine.* | 8.80       | -         | 10.99            | -               |
| Diffusion Planner (ours)        | **89.76**  | **82.56** | **75.67**        | **68.56**       |
*: Using pre-searched reference lines or additional proposals as model inputs provides prior knowledge.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes a novel transformer-based Diffusion Planner for closed-loop planning, which could resolve current bottlenecks in prior works. The structure is neat, without rule-based refinement and could effectively model the multi-modal driving behavior. Evaluations on large-scale benchmarks, both the nuPlan and the self-collected 200-hour (which is impressive in terms of public data), prove the efficacy of the proposed method.

This paper provides solid experiments with good motivation. The paper is well written. The proposed method is detailed with clear implementation on nuPlan for the closed-loop verification. Diffusion is a good methodology to be adopted for the autonomous driving community, with many potential applications for video generation, decision making, motion prediction. Many applications in the autonomous driving domain. Such a work would shed new light on the community, and deserve a broad attention to be presented at ICLR.

Please incorporate all the necessary comments raised by reviewers and supplement new (unfinished during rebuttal, if any) experiments to make the submission more convincing for the camera-ready.