DiffE2E: Rethinking End-to-End Driving with a Hybrid Diffusion-Regression-Classification Policy
Authors: Rui Zhao
Yuze Fan
Ziguo Chen
Fei Gao
Zhenhai Gao
Venue: NeurIPS 2025 poster ()
Keywords: Autonomous driving
diffusion model
end-to-end
transformer
Primary area: deep_learning
TL;DR: We propose an end-to-end autonomous driving framework based on a unified diffusion-regression-classification policy, achieving state-of-the-art performance on both the CARLA and NAVSIM benchmarks.

Abstract: End-to-end learning has emerged as a transformative paradigm for autonomous driving. However, the inherently multimodal nature of driving behaviors remains a fundamental challenge to robust deployment. We propose DiffE2E, a diffusion-based end-to-end autonomous driving framework. The architecture first performs multi-scale alignment of perception features from multiple sensors via a hierarchical bidirectional cross-attention mechanism. Subsequently, we design a hybrid diffusion-regression-classification decoder based on the Transformer architecture, adopting a collaborative training paradigm to seamlessly fuse the strengths of diffusion and explicit strategies. DiffE2E conducts structured modeling in the latent space: diffusion captures the multimodal distribution of future trajectories, while regression and classification act as explicit strategies to precisely model key control variables such as velocity, enhancing both the precision and controllability of the model. A global condition integration module further enables deep fusion of perception features with high-level goals, significantly improving the quality of trajectory generation. The subsequent cross-attention mechanism facilitates efficient interaction between integrated features and hybrid latent variables, promoting joint optimization of diffusion and explicit strategies for structured output generation and thereby yielding more robust control. Experimental results demonstrate that DiffE2E achieves state-of-the-art performance on both CARLA closed-loop benchmarks and NAVSIM evaluations. The proposed unified framework that integrates diffusion and explicit strategies provides a generalizable paradigm for hybrid action representation and shows substantial potential for extension to broader domains, including embodied intelligence.

DiffE2E: Rethinking End-to-End Driving with a Hybrid Difftssio-RerressnnCCCcsiiiittt Policy
Rui Zhao!
Yuze Fan'
Ziguo Chen Fei Gao Zhenhai Gao 1.2
'College of Automotive Engineering, Jilin University
National Key Laboratory Automotive Chassis Integration and Bionics, Jilin University https://infini inidrivee github. github.io///i io/DiffE2E/
Abstract
End-to-end learning has emerged as a transformative paradigm for autonomous driving. However, the inherently multimodal nature of driving behaviors remains fundamental challenge to robust deployment. We propose DiffE2E, a diffusion- based end-to-end autonomous driving framework. The architecture first performs multi-scale alignment of perception features from multiple sensors via a hierar- chical bidirectional cross-attention mechanism. Subsequently, design hybrid decoder based on the Transformer architecture, adopting collaborative training paradigm seamlessly fuse the strengths of diffusion and explicit strategies. DiffE2E conducts structured modeling in the latent space: diffusion captures the multimodal distribution of future trajectories, while regression and classification act explicit strategies to precisely model key control variables such as velocity, enhancing both the precision and controllability of the model. A global condition integration module further enables deep fusion of perception features with high-level goals, significantly improving the quality of tra- jectory generation. The subsequent cross-attention mechanism facilitates efficient interaction between integrated features and hybrid latent variables, promoting joint optimization of diffusion and explicit strategies for structured output generation and thereby yielding more robust control. Experimental results demonstrate that state-of-the-art art performance on both CARLA closed-lood bench- marks and NAVSIM evaluations. The proposed unified framework that integrates diffusion and explicit strategies provides generalizable paradigm for hybrid action representation and shows substantial potential for extension to broader domains, including embodied intelligence.
DifEEE achieves state-of
Introduction
End-to-end autonomous driving frameworks directly map sensor data to control commands, effectively avoiding error accumulation inherent in traditional modular pipelines and significantly enhancing both
decision-making efficiency and scenario adaptability 27 6 47
5 48
Most existing approaches
43), learning the mapping between environmental observations and vehicle actions from large-scale driving datasets. However, due to the inherently multimodal nature of driving behaviors, these methods are prone to producing suboptimal solutions 33 and often fail to generate the diverse trajectories consistent with real-world driving strategies.
adopt explicit policy architectures based on direct regression 42
9
53 23
62
The multimodal nature of driving decisions means multiple plausible actions may exist for the same scenario. Conventional regression- approaches tend to average over this diversity, often
"Contact me at rzhao@jlu. edu. I "ooresponndence to: Fei Gao (gaof ei 1232441233j edu. cn)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

@
Denoise
Trajectory Head
Replace
Diffusion Head
Denoise
Regresion
Condition
Noise
Noise
Planning Decoder
Planning Decoder
Diffusion Decoder
Multimodal Fused Feature
M Multimodal Feature
Condition
Perception E Encoder
Perception E Encoder
Perception Encoder
(a) Explicit Policy
(b) Explicit Policy with Diffusion Refinement
(c) Diffusion Policy
Comparison end-to-end training paradigms. Explicit Policy. Directly predicts trajectories through regression after sensor input processing. (b) Explicit Policy with Diffusion Refinement. Uses diffusion models to replace traditional explicit policy trajectory output heads. (c) Diffusion Policy, Uses diffusion models to directly generate trajectories based on perception encoder features.
Figure I:
resulting in suboptimal or even unsafe behaviors [12] This issue exacerbates safety risks and limits the reliability of regression-based policies in real-world autonomous driving systems. To address these limitations, recent research 6 3
31 41 has explored explicitly modeling multimodality via predefined discrete sets of trajectory candidates. However, such hard-coded schemes undermine the continuity of policy generation, reduce model adaptability, and ultimately degrade continuous decision making into a fixed mode selection problem, thereby weakening generalization capability.
Diffusion models have demonstrated strong capabilities in modeling complex multimodal distributions and generating high-qualit outputs, making them increasingly prominent in computer vision [18 44 45 [3..
This generation paradigm based on gradual denoising provides a new solution for addressing multimodal modeling and generalization problems in end-to-end autonomous driving. The successful application of diffusion models in robotic motion planning has already demonstrated their advantages in multimodal action sequence generation and long-term sequence prediction 7 57 24). However, applying diffusion models autonomous driving systems faces unique challenges: autonomous driving needs to simultaneously meet multiple stringent requirements in open road environments, dealing with the uncertainty of highly dynamic traffic participants while ensuring real-time response; generating feasible trajectories that conform to road topology while ensuring traffic efficiency.
Recent works have explored diffusion models autonomous driving planning (61) and end-to-end control [34 [4]. applying Denoising Diffusion Implicit Model (DDIM) [44), DPM-Solver (36), and Rectified Flow 35) for trajectory generation, revealing the strong potential of generative approaches. However, most integrate diffusion models only after planning decoders, using them to replace explicit policy heads (see Figure bb). This setup risks losing key perception features and constrains generation due to pre-pre pre-processed decoder outputs. While some methods (44 use trajectory anchors to enhance real- time performance, anchor-baa based designs can limit trajectory diversity. A more effective architecture is needed to fully harness the generative power of diffusion models.
To address the aforementioned challenges, introduce DiffE2F, an innovative end-to-end tonomous driving framework, as illustrated in Figure 2 DiffE2E first performs multi-scale alignment of LiDAR and image features via hierarchical bidirectional cross-attention mechanism, thereby enabling high-prees precision env environmental perception [9] 23). Building upon this, we design hybrid crriioonrrrgressioocccsssiiiiii
decoder based on the Transformer architecture and employ col- laborative training paradigm to seamlessly integrate the strengths of diffusion and explicit strategies. By combining diffusion models with explicit strategies, we structurally partition the latent space: on the one hand, diffusion models are leveraged to capture the distribution, diversity, and higher- order uncertainty of future trajectories; on the other hand, explicit strategies are utilized achieve fine-grained modeling of key control variables such as velocity. The cross-attention mechanism facilitates efficient interaction between integrated features and hybrid latent variables, supporting the joint optimization of diffusion and explicit strategies for structured output generation. DiffE2E

---

achieves state-of- offte-o-the-aae art performance on both the CARLA closed-loop benchmark and the NAVSIM evaluation, demonstrating enhanced safety and traffic efficiency in complex scenarios, as well as strong generalization capability.
In summary, contributions are follows:
We propose DiffE2E, an end-to-end autonomous driving framework that uses diffusion models for trajectory generation and the first to evaluate them in closed-loop tests within CARLA.
2. We propose hybrid deffusion-regressiin regressiooncsasssiccttiiooo decoder based the Transformer ar- chitecture, employing a collaborative training paradigm to achieve seamless integration of the advantages of diffusion and explicit strategies.
3. We conduct dual-platforn benchmark testing, achieving state-of-the-art performance across multiple benchmarks the CARLA simulator, and attaining PDMS of 92.7 in the non-reactive simulation NAVSIM, while maintaining higher real-time performance compared to other methods.
2 Preliminaries
Problem Definition:
This research focuses on end-to-end autonomous driving closed-loop control strategies based on diffusion models. The system directly takes multi-modal raw perception data as input, including front-view camera RGB images I # x/xxxxx Wx3, x3, LiDAR point clouds P7 E R RNxx x3
and vehicle state information S, E R&, The system predicts the ego vehicle's future trajectory 20. where the generative process is modeled as:
po(zo|C)
p(ær) Il po C)
(1)
201TT
In diffusion modeling, represents the intermediate variable at step t in the diffusion process, with the final predicted trajectory being ætg E RKK Kxxe, Here, each waypoint p. Rdd represents position information in the predicted trajectory. The condition information C is encoded from multi-modal sensor data through a cross-modal feature fusion module. Unlike traditional open-loop control, in closed-loop control, the trajectory decision at the current moment directly affects the perception input at the next moment, forming a dynamic feedback loop. This coupled relationship requires the model to have strong temporal consistency and robustness.
Denoising Diffusion Probabilistic Models (DDPM) are powerful generative models that capture complex multimodal distributions via two-phase process: forward diffusion gradually adds noise, while the reverse process reconstructs the data through iterative denoising. This framework naturally models the multimodality of driving behaviors. The forward process follows Markov chain that transforms data 2i into noise over T steps:
Diffusion Models:
((0,12,,)) Nxx;; V1
(2)
where (8.)7= controls noise levels. Through reparameterization, we can sample æ, directly:
F3 Va + V1 d.C. C~1(00
where a. = II 8:). While DDPM generates high-quality samples, its sequential process is computationally expensive. DDIM 44 addresses this with a non-Markovian process that accel- erates generation while maintaining quality, making diffusion models more practical for real-time autonomous driving applications.
3 Methodology
Overview: DiffE2E is an end-to-end autonomous driving framework, as illustrated in Figure 2 In the perception stage, it introduces a multimodal spatiotemporal fusion module, which employs hierarchical bidirectional cross-attention mechanism to align LiDAR and camera features and establish structured representation of the driving scene. During the decoding stage, a Transformer-basee

---

Convolation Thowker
Frami
Comvolatoon Drrowder
Fision Transformer
Focd
Blooder
Mali
LIDAR
LJDAR Peature
decessing
Eucoder
Dezader
LIBAR BEY
Poine
Diffusion-Regreee CalasionnnCasiiiaatiionn Decuder
putiesion Timetop
Terget Candition
Faviraement
DMTT Note Rig. Ctt Querr
Encoder
Coodiiie
Ditt. Head
Dett Ourput & Mog Cit.
62 Ag...
Embeoling Layer
Oft
Input Tokens
Layer LayerNara Narm
Mak-HHed
Mali: Hod
Netwods
Fealfumont
Stog Cir..
Hybrid Decoder
Norm
Narm
Beg- CM Head
Dift
Reg Clf. Output
Chitpu
Figure 2: Overall architecture DiffEZE. The main architecture consists of Transformer-base perception module and hybrid aefftssonneegressinnnalssiiiiiie exclusively used for the CARLA benchmark, while the black arrows (-+) represent the data flow shared between both the CARLA and NAVSIM benchmarks.
decoder. The blue arrows (-) indicate the data flow
hybrid cuffusionnrrerressii www.-rrresssnoanaassiiiiiiieeeee classificationn decoder is adopted, in conjunction with a collaborative training strategy that seamlessly integrates diffusion and explicit policy paradigms. The global condition integration module fuses scene representations with goal-oriented global context, while the cross-attention mechanism enables efficient interaction between integrated features and hybrid latent variables, thereby facilitating the joint generation of diffusion-based trajectories and explicit
outputs.
3.1 Multimodal Fusion Perception Module
The perception module aims to fuse multimodal sensor data to construct structured environmental representations. This paper adopts the TransFuser architecture as the basic perception backbone network [23]. with inputs including wide-angle front-view RGB images I, R H x3 and BEV representations p?d E "xx constructed from raw LiDAR point clouds P.D extracting initial features from both branches, they enter multi- cross- cross-fusion module composed of multiple Transformer layers, achieving deep alignment and information interaction between LiDAR and image features through cross-modal attention mechanisms. Finally, the module outputs high-dimensional fusion features C, global semantic representations, and image feature grids to support the fine-gr lneeggrined modeling requirements of downstream decision modules.
Nx3 After
3.2 Hybrid Diffusion-Reggee Diftusion-Reereeiion-Claaa Regression-Claaiiictttio Module
After the multimodal fusion perception module integrates information from diverse sensors, the DiffE2E framework introduces innovative architecture. By incorporating a Transformer-based hybrid sifftssioorrgressiioollaaiiiiiis
decoder and adopting a collaborative training paradigm, DiffE2E effectively combines the strengths of both diffusion-basee and explicit policy strategies, as illustrated in Figure 2 This section provides a comprehensive exposition the global condition integration, hybrid diffusion-regrest classsfiiaitssiiiiatiinn decoding process, and decoder output module.
Global Condition Integration: To enhance the influence of target points trajectory generation. they are used as global conditions (23] Original target points are first projected via linear layer fgoat into a shared high-dimensional space of dimension d, forming representations g Roxdd Meanwhile, diffusion timesteps are encoded into temporal embeddings temb E R to help the model adapt across denoising stages. Finally, perception features C & goal features, and timestep embeddings are fused, and combined with learnable positional encoding ECe, to form contextual representation for trajectory decoding, where le, lg. and l, denote the sequence lengths of

---

conditional features, goal representations, and timestep embeddings:
Č [Concat(C, g, temb) + Bros]] E Bl t&)
This global condition integration mechanism incorporates target point information and timestep embeddings into perception features, enhancing the model's awareness of navigation goals and enabling dynamic adjustment of feature representations during denoising for more precise trajectory generation.
Decoding: Let & and lu denote the sequence lengths of the noisy trajectory and explicit policy queries, respectively. The proposed hybrid decoder first encodes the noisy trajectory Ti into high-dimensional feature representations via linear projection layer Senc. These are then concatenated with the initial query vectors for explicit policy decoding, qo E &R& and augmented with learnable positional embeddings pos E (f++)))) This yields the decoder's initial input vector:
Hybrid
Zm [Concat(fen<(T))) (n) + Epcs] E (((+ LE (I...., T
At each time step t of the entire diffusion process, the input first passes through a multi-head self-attention layer process the internal feature relationships of Zm:
Ziid SelfAttnZmm),
Zmid (R(++())
(6)
Then, through cross-attention mechanism, Zmia interacts with the conditional features C, producing the final output:
Zout CrossAttn C,C), Zou E ((l+tt)xdd Č (llcttott)xxd
where Zoat denotes the feature vectors output by the decoder, and Ć represents the integrated global conditional features. Within the output representation Zouts the first la positions correspond the latent features for diffusion-based trajectory generation, while the remaining E, positions correspond the latent features for explicit policy outputs.
The decoder output module further processes the hybrid features Zeut E R((++())x realize joint decoding of both diffusion-based and explicit policy strategies. This module employs feature separation and task-specific decoding approach, decomposing the output features structurally in semantic space:
Decoder Output Module:
Zaitt = Zoui|: (x] E R&
(8)
Zup = Zomt (a la + ER
where Zair represents the nigh-dimensional latent representations for the diffusion trajectory, and Zuup contains structured feature information for explicit policy outputs.
3.3
Collaborative Training Strategy
Building upon the aforementioned hybrid oeffussonnnggressonnnlassiiiiiis propose a collaborative training paradigm that synergistically integrates diffusion-based generative modeling with explicit policy learning. The core objective of this strategy is combine the generative capabilities of diffusion models with the precision of explicit policies, achieving complementary advantages from both approaches.
decoder architecture,
Diffusion Loss Function: DiffE2E employs a trajectory reconstruction- based loss function for diffusion generation, which directly optimizes the model's ability to recover the original trajectory from noisy inputs. Using Mg to represent the entire model, the loss function is formulated as:
Lait E(-U(1,T)) X. (xs (xo) ||xx--Moxxx. Mo(xi, 4,)
Explicit Policy Loss Function: The explicit policy loss adopts a multi-task composite optimization strategy, enabling precise control of gradient flows and prioritization through task-specific weighting
5

---

coefficients. Specifically, Suup = E Eien A C.(v.. 0,), where f denotes the set of all explicit policy tasks, A; is the weight for task i, and (y,,, 0,) represents the objective for task i, with y; and y; denoting ground-truth labels and predictions, and 0, the corresponding network parameters.
For instance, in the velocity prediction task, we construct a multi-class classification model based on semantic stratification, covering four velocity states with clear physical meanings: brake, walking speed, low speed, and high speed. The accuracy of classification prediction is optimized via a weighted coss-py loss function:
fospeed (y, Pi; w)
wy Vo.i log(pn, + eps), s.t.
Pn.i = 1, 2 0
(10)
B
where B denotes the batch size, Vn.i E f0, indicates the ground-truth label (with one-hot encoding)
of the n-th sample under the i-th velocity class, Po.: E [0, 1 is the predicted normalized classification probability, w; > 0 is the class-balancing weight coefficient, and eps is a constant for numerical stability. Further details on explicit policy loss formulations are provided Appendix A
Experiments
This research is primarily evaluated using the CARLA simulator closed-loop benchmark [14 and the NAVSIM non-reactive simulation benchmark 12) CARLA offers diverse urban scenes and sensor emulation, with its closed-loop mechanism providing real-time feedback to assess decision quality over long horizons. NAVSIM, built on OpenScene [1] (a streamlined version of the nuPlan M dataset), provides 360 coverage via 8 cameras and 5 LiDARs, with 2Hz annotations of maps and object bounding boxes. Additional details are provided in Appendix B
Experiment Setup:
4.1 Experiment on CARLA
This study uses RegNetY-3.2GF 39 as the encoder for image and LiDAR inputs. To optimize computation and training efficiency, a two-stage strategy is adopted: the first trains the perception module with multi-task losses; the second trains the diffusion decoder conditioned on the frozen perception outputs. We adopt CARLA Longest6, CARLA Town05 Long, and CARLA Town05 Short as evaluation benchmarks | using the official Driving Score (DS), Route Completion (RC), and Infraction Score (IS) as metrics. Detailed implementation details and baseline descriptions are provided in Appendix B.1
Implementation Details:
As shown in Table I our proposed diffusion-based method, DiffE2E, demonstrates superior performance on the CARLA Longestó benchmark, consistently outperforming mainstream existing approaches. Across the three key evaluation metries, DiffE2E achieves a DS of 79 + 4. RC of 94 + and IS of 0.84 + 0.02, each exceeding or closely matching the best results reported to date. Notably, compared to the strong baseline TF++ (which also utilizes RegNerY-3.2GF as the backbone for perception and the same input modalities), DiffE2E delivers a 10-point improvement in DS (79 + 4 vs. 69 + 0), 0.12 increase in IS (0.84 + 0.02 vs. 0.72 + 0.01), and comparable stability in RC (94 + vs. 94 2), thoroughly validating the dual advantages of our model in both accuracy and safety. Furthermore, the standard deviation (+) of DiffE2E is similar to other methods, indicating robust and stable performance across multiple experimental runs. Traditional explicit-olliy--asse methods (e.g., LAV v2, TransFuser) generally exhibit lower DS and IS when facing complex scenarios, revealing their limitations in perceiving and reasoning about multimodal driving behaviors. In contrast, diffusion decoding significantly enhances the model's generalization and decision diversity capabilities. summary, DiffE2E achieves higher DS and IS, ensuring both high route completion rates and low infraction rates over long driving horizons, and demonstrates exceptional end-to-end closed-loop autonomous driving performance.
Main Results:
Visualization:
Figure 3 shows a comparison in typical right-turn scenario. Initially, both TF++ and DiffE2E plan similar paths by first merging right. When a vehicle appears, TF++ sticks to its preset path and collides, while DiffE2E adapts by temporarily going forward, then safely merging after the vehicle passes. This demonstrates DiffEEE's superior multimodal generation capability and real-time adaptability in dynamic traffic, effecctivlly avoiding collisions.

---

Table 1: Comparison on the CARLA Longestó benchmark. "C L" denotes use of camera and LiDAR as sensor inputs, "Pri." indicates the use of privileged information. The best and second best results are highlighted in bold and underline.
Method
Traj. Decoder
Img. Encoder
Input
DS
RC
IS f
Expert
81 + 3
90 +
0.91 + 0.04
WOR
ResNet-34
C
3
48
0.56 0.03
LAV vl 2
Explicit Policy
ResNet-18 17
C&L
33 1
70 1
0.51 1: 0.02
InterFuser 42
Explicit Policy
ResNet-50
17
C&L
47 = 6
74 + 1
0.63 + 0.07
TransFuser 9
Explicit Policy
RegNetY-3.2GF [39
C&L
47 - 6
93 + 1
0.50 + 0.06
TCP [53
Explicit Policy
ResNet-34 T7
C
48 +
0.65 0.04
ThinkTwice
Explicit Policy
ResNet-50 [17
C&L
51
64
0.80
LAV v2 22
Explicit Policy
ResNet-18 [17
C&L
58
83 1
0.68 0.02
Perception Plan'T (4)
Explicit Policy
RegNetY-3.2GF (39
C&L
58 + 5
88 +
0.65 + 0.06
DriveAdapter 25
Explicit Policy
ResNet-50 17
C&L
58 2
7
0.79 + 0.04
TF+
23
Explicit Policy
RegNetY-3.2GF 39
C&L
69 1 0
94 1 2
0.72 1 0.01
DiffE2 (Ours)
Diffusion Policy
RegNetY-3.2GF
[3
C&L
79 =
94 + 2
0.84 + 0.02
(4) Trans user++
(b) DIMEZZE
Visualization in CARLA Simulator. In both the LiDAR and scene visualizations, blue points represent the predicted trajectory, while red points in the LiDAR view denote the target waypoints.
Figure 3:
4.2 Experiment on NAVSIM
This study builds a model training framework based on NAVSIM's navtrain dataset. Unlike the CARLA setup, adopt VowNetV2-99 (30) the feature extraction backbone network in NAVSIM. The Predictive Driver Model Score (PDMS) is used as a comprehensive metric, combining key driving dimensions via weighted integration: No at-fault Collision (NC), Drivable Area Compliance (DAC),
Implementation Details:
(TTC), Comfort (C), and Ego Progress (EP). Detailed implementation details and baseline deseriptions can be found in Appendix B.2
Main Results: As shown in Table 2 DiffE2E achieves outstanding performance on the NAVSIM benchmark, attaining PDMS score of 92.7 surpasinnn Hydra-MDP++ (91.0), Hydra-MDP (91.0), GoalFlow (90.3), and DiffusionDrive (88.1). This result highlights the superiority of our hybrid diffusion-based end-to-end approach across multiple driving metrics. In terms of safety and compliance, DifIE2E particularly excels: it achieves a faultless collision rate of 99.9 (compared with Hydra-MDP++ at 98.6, MDP at 98.7, and GoalFlow at 98.4), and reaches the highest For the cime-to-collision drivable area compliance score of 98.6, on par with Hydra-MDP++ by 4.2 points. In terms of metric, DifEEE leads with score of 99.3, outperforming efficiency and comfort, DiffE2E achieves an ego progress of 85.3, just below Hydra-MDP++ (85.7) and comparable Hydra-MDP* (86.5). Additionally, its comfort score reaches 99.9, nearly matching the best best-performing methods, demonstrating its ability to generate smooth, human-like trajectories.
Unlike DiffusionDrrve and GoalFlow, which utilize diffusion as post-processing after the planning decoder, DiffE2E employs full diffusion decoding paradigm, enabling richer policy space modeling and greater expressive power. Furthermore, the hybrid design of diffusion and explicit policy
7

---

Table Comparison on Navtest benchmark. denotes using only the ego vehicle state as imput. * indicates that these two methods adopt diffusion models as post-planning decoders, whereas DiffE2E employs a diffusion model as the full decoder to directly generate trajectories. and denote the variants of DifEE2E with the image encoder replaced by ResNet-3 34 and with the sensor input restricted to images only, respectively.
denotes that the result obtained via sub-score ensembling three backbones [33)
Method
Truj. Decuder
Img. Enc.
Input
PDMS
NC1
DAC1
EP1
TTC1
Ct
Human
94.8
100
100
87.5
100
99.9
AD-MLP
58
Explicit Policy
S
65.6
93.0
77.3
62.8
83.6
100
VADv2 6
Explicit Policy
ResNet-34
17
C&L
80.9
97.2
89.1
76.0
91.6
100
UniAD
Explicit Policy
ResNet-34
83.4
97.8
91.9
78.8
92.9
100
LTF 9
Explicit Policy
ResNet-34
C
83.8
97.4
92.8
79
92.4
100
PARA-Drive
52
Explicit Policy
ResNet-34
84.0
92.4
79.3
93
99.8
TransFuser
Explicit Policy
ResNet-34
17
C&I
84.0
97.7
92.8
79.2
92.8
100
LAW 82
Explicit Policy
ResNet-34 17
C
84.6
96.4
95.4
81.7
88.7
99.9
DRAMA
Explicit Policy
ResNet-34
C&L
85.5
93.1
80.1
94.8
100
Hydra-MDDP
33
Explicit Policy
ResNet-34
30
C&I
86.5
98.3
96.0
78.7
94.6
100
DiffusionDDivv
34
Diffusion Policy"
ResNet-34
C&L
88.1
98.2
93.7
82.2
94.7
100
GoalFlow
54
Diflusion Policy
V2-99
C&I.
90.3
94.6
Hydra-MDP
33
Explicit Policy
VIT-L
15
V2-99
30
C&L
91.0
98.7
98.2
86.5
95.0
100
Hydru-MDP++
51
Explicit Policy
V2-99
E0
C
91.0
98.6
98.6
85.7
95.1
100
DiffEZE!
Diffusion Policy
ResNet-34
1+
C&L
89.8
99.2
96.8
83.6
96.7
100
DITEE2R:
Diffusion Policy
V2-99
30
90.9
99.7
97.1
84.2
98.2
99.9
DiMEZZ (Ours)
Diffusion Policy
V2-99
C&L
92.7
99.9
98.6
85.3
99.3
99.9
View
(b)
DiTTEZE
Figure 4: Visualization Navtest benchmark. Red trajectories denote the predicted paths of each method, while green trajectory corresponds to the ground truth.
(regression and classification) combines the strengths of both approaches: it captures the multimodal distribution of future trajectories while maintaining the accuracy and controllability of explicit policies. To further validate the impact of key design choices, we evaluate two variants: DiffE2Et (with ResNet-34 encoder, PDMS 89.8) and DiffE2E (using image input only. PDMS 90.9). The performance degradation of these variants underscores the importance of high-capacity visual encoders and multi-modal sensor fusion. Overall, DiffE2E achieves state-of-the-art or near-optimal results on all critical NAVSIM metrics, demonstrating robust, controllable, and efficient performance, and offering a new paradigm for end-to-end autonomous driving.
Visualization: To validate the generalization and superiority of DifiE2E, we selected two repre- sentative complex driving scenarios for comparative analysis (Figure 4. Green trajectories denote human references, and red ones indicate planned trajectories. In right-turn intersections, baseline methods often deviate cross boundaries, while DiffE2E accurately follows lane edges with smooth turns. In small intersection left-turns, DiffusionDrive misinterprets navigation intent and plans straight trajectory, TransFuser incorrectly chooses the right lane, while only DiffEEE accurately executes the left-turn instruction with a trajectory almost completely matching the reference. This demonstrates DiffE2E's accuracy and safety trajectory planning. More visualizations can be found in Appendix C

---

Table Ablation results of DifIE2E on CARLA Longestó benchmark.
POMS of Naviest Benchmark
$2,72330
Type
Method
DS
RC
18
52,72525
DinTEE
78.9
52. 22.73320
Input
w/o ego state
68.9
88.8
0.81
w/o command
69.6
93.8
0.77
52. 22.0055
Component
GRU
66.8
52.7000
Full Diffusion
70.1
91.1
0.76
Training Paradigm
Full Discrimination
70.3
83.5
0.84
12,76699
One-Stage Training
18.2
21.8
0.79
Two-Stage Training
$2,76900
25
Danois:: Sleps
Noise Prediction
20.1
27.2
0.72
Output
Trajectory Prediction
78,9
93.7
0.84
Figure 5: Ablation study of denoising steps on Navtest benchmark.
4.3
Ablation Studies
To systematically evaluate the contribution of each component in the DiffE2E framework, conducted ablation experiments on the CARLA Longestó benchmark (see Table 3). The results show that, the input level, removing the ego vehicle state or navigation command leads significant drop in driving score from 78.9 to 68.9 and 69.6, respectively, confirming the critical roles of ego state for precise planning and navigation command for intent understanding. At the architectural level, removing the GRU module reduces the score to 66.8, indicating its indispensability in modeling temporal dependencies in complex scenarios. Regarding training paradigm, using only the diffusion strategy (70.1) or the discriminative strategy (70.3) underperforms the hybrid design, demonstrating the effectiveness of combining both. Notably, single-stage training achieves only 18.2 points, 77% lower than two-stage training, and performs very poorly. This is because direct coupling of perception and planning in single-stage training makes it difficult for the model to simultancously ensure perception accuracy and planning rationality, causing significant conflicts between optimization objectives. This manifests as the vehicle hardly being able stay in the lane, with frequent yawing, deviation even collisions. For output types, when the diffusion model predicts noise, the score drops dramatically to 20.1 and the vehicle fails follow the lane, while directly predicting trajectories provides the best results. This suggests that, for high-precision driving tasks, modeling structured trajectory space clearly more effective. For further discussion on output mechanisms, please refer to Appendix D.I
Additionally, we conducted an ablation study on the number denoising steps in the diffusion model (see Figure 5). Because the CARLA euvironment is highly stochastic and the impact of denoising step count is relatively small, we adopt the more stable Navtest benehmark for analysis. Experimental results show that PDMS achieves the lowest score at one step (insufficient denoising), peaks at two steps, and then gradually declines, indicating that two steps are sufficient for denoising. This benefits from our cemporal-spatial decoupled hybrid modeling strategy and directly aligning the diffusion objective to trajectory error, allowing the model to focus more on learning the spatial distribution structure. Consequently, high-quality outputs can be produced with fewer iterations.
4.4 Real-time Performance Comparison of DiffE2E
To comprehensively evaluate the real-time per- formance of DifIE2E, we conducted inference latency comparisons on the NAVSIM bench- mark using a single NVIDIA RTX 3090 GPU, against existing diffusion- methods (Diffu- sionDrive and GoalFlow) and the non-diffusion method TransFuser. Results are presented in Ta-
Table 4: Comparison of inference latency and model parameters between different methods.
Method
Latency (ms)
Params
Img. Enc.
TransFuser 12
29.1
55M
ResNet-34
1k]
Difi
39.
60M
ResNet-34
12
GoalFlow [54
66.8
190M
V2-99
30
The experiments show that TransFuser, equipped with the lightweight ResNet-34 image encoder, achieves the lowest latency m29.1ms),
DifEEE (Ours)
42.8
105M
V2-99
30
followed by DiffusionDrivv (39.Ims), In contrast, DiffE2E and GoalFlow, both using the larger

---

VoVNetV2299 encoder, report latencies of 42.8ms and 66.8ms, respectively. Notably, inference la- tency exhibits clear positive correlation with model parameter scale, reflecting the inherent trade-off between model complexity and computational efficiency. Nevertheless, with an optimized diffusion sampling strategy, DiffE2E achieves real-time performance comparable to lightweight models while maintaining a relatively high parameter count (105M), fully validating its balanced optimization between computational efficiency and model expressiveness.
5 Related Works
End-to-end Autonomous Driving: End-to-end autonomous driving has advanced significantly in multimodal perception fusion and decision-making UniAD 21 constructs full-stack Transformer to coordinate to improve planning efficiency, VADv2 [6] models action-space distribution through a trajectory vocabulary library, SparseDrive 47 proposes sparse trajectory representation for efficient driving
tasks, VAD 27 designs vectorized scene representation
without BEV, and the Hydra-MDP series [3 31 designs a multi-teacher distillation framework to integrate rule-based systems with human driving knowledge. TransFuser [2] fuses camera and LiDAR features vía Transformer for intersection decisions; TCP (53) jointly trains trajectory and control predictions; InterFuser [2] introduces safety thinking maps for multi-vies multimodal fusion, and TF++ [23] enhances the decoder and proposes decoupled speed prediction. However, such volicy-based approaches often compress complex multimodal driving behaviors into a single deterministic output, which tends to yield averaged, less diverse suboptimal solutions in varied decision-making scenarios, making it difficult to fully capture the richness and uncertainty inherent in driving behaviors.
Diftusion Model in Transportation and Autonomous Driving: Diffusion models are profoundly transforming the transportation and autonomous driving fields with their excellent multimodal gener- ation capabilities. In transportation, Diffusion-ES [55] innovatively combines evolutionary strategies with diffusion models, achieving complex driving behavior generation without requiring differen- tiable reward functions, with its zero-shot performance significantly surpassing traditional methods in the nuPlan benchmark. VBD (22) uses game theory to guide adversarial scenario generation, enhancing simulation realism. MotionDiffuser (28) introduces a coremuuttiioninvariii architecture for constrained multi-agent trajectory sampling, ensuring interaction consistency. Diffusion Planner 61 leverages DPM-Solver (36) and classifier guidance for fast, safe, and personalized trajectory generation in closed-loop planning. However, most of these methods are based on perfect perception assumptions, ignoring the impact of state estimation errors caused by perception uncertainties in practical applications. In the field of end-to-end autonomous driving, although the application of diffusion models has achieved preliminary results, it still faces many challenges. DiffusionDrive (34] first introduced diffusion for end-to-end driving with an anchored strategy. HE-Drive [50] uses condi- tional DDPM [88 and vision-language models for scoring, producing human-likk like, spatiotemporally consistent trajectories at high computational cost. GoalFlow [54] addresses trajectory divergence via goal-driven flow matching and efficient one-step generation. These works demonstrate the enormous potential of diffusion models in the field of end-to-end autonomous driving.
Conclusion
This study presents innovative end-to-end autonomous driving framework, DiffE2E, which features a hybrid aeaaiionregressioonllssfiiiiiee introduces a collaborative training mechanism effectively integrate the strengths of diffusion models and explicit policies. We design a structured latent space modeling approach: the diffusion model captures the diversity and uncertainty of driving behaviors by modeling the future trajectory distribution, while the explicit policy precisely models key control variables such as ego- vehicle speed to enhance controllability and prediction accuracy. This hybrid paradigm not only increases the model's ss fault tolerance through multi- denoising generation but also leverages the structured generation mechanism and implicit ensemble reasoning to improve robustness against distribution shifts. In both CARLA closed-loop testing and NAVSIM non-interactive simulation, DiffE2E achieves otate-of-the-art performance, balancing traffic efficiency and safety, and demonstrates strong generalization potential.
decoder based on Transformer architecture and
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Natural Science Foundation of China (grant 52572475) and the Science and Technology Development Project of Jilin Province (grant 20250102130JC).
References
[1] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong. Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106. 100120000.1100,, 2021.
[2] Dian Chen and Philipp Krähenbühl. Learning from vehicles. In Proceedings of the IEEF/CVF Conference Compuler Vision and Pattern Recognition, pages 17222-17231, 2022.
[3] Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning, pages 66-75. PMLR, 2020.
[4] Dian Chen, Vladlen Koltun, and Philipp Krähenbühl. Learning to drive from world on rails. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15590-15599, 2021.
[5] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[6] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243 2024.
[7] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241273668. 2023.
[8] Kashyap Chitta, Aditya Prakash, and Andreas Geiger. Neat: Neural attention fields for end- to-end autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15793-15803, 2021.
[9] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer based sensor fusion for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45((1))12878-12895 2022.
[10] Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVE international conference on computer vision, pages 9329-9338, 2019.
[11] OpenScene Contributors. Openscene: The largest dp-to-date 3d occupancy prediction bench- mark in autonomous driving. https://gitthie com hwtp///peDDriieeb/bbbba Saapaaabbb/pnSeeeeee... 2023.
[12] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang. Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems, 37:28706-28719 2024.
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[14] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pages 1-16. PMLR, 2017.
[15] Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: visual representation for neon genesis. Image and Vision Computing. 149:105171, 2024.
[16] Kevin Frans, Danijar Hafner, Sergey Levine, and Pieter Abbeel. One step diffusion via shortcut models. arXiv preprint 4rXiv:2410.12557, 2024.

---

[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6853 2020.
[19] Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton. Model-based imitation learning for urban driving. Advances in Neural Information Processing Systems, 35:20703- 20716,, 2022.
[20] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference Computer Vision, pages 533-549. Springer, 2022.
[21] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17853-17862. 2023.
[22] Zhiyu Huang, Zixu Zhang, Ameya Vaidya, Yuxiao Chen, Chen Lv, and Jaime Fernández Fisac. Versatile behavior diffusion for generalized traffic agent simulation. arXiv preprint arXiv:2404.02524, 2024.
[23] Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hidden biases of end-to-end driving models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8240-8249 2023.
[24] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9915-9915. PMLR, 2022.
[25] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end au- tonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7953-7963, 2023.
[26] Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21983-21999 2023.
[27] Bo Jiang. Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang. and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8340-8350, 2023.
[28] Chiyu Jiang, Andre Cormman, Cheolho Park, Benjamin Sapp. Yin Zhou, Dragomir Anguelov, al. Motiondiffuser: Controllable multi-agent motion prediction using diffusion. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9644-9533, 2023.
[29] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Proc. NeurIPS, 2022.
[30] Youngwan Lee, Joong-won Hwang, Sangrok Lee, Yuseok Bae, and Jongyoul Park. An energy and gpu-computation efficient backbone network for real-time object detection. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition workshops, pages 0-0. 2019.
[31] Kailin Li, Zhenxin Li, Shiyi Lan, Yuan Xie, Zhizhong Zhang, Jiayi Liu, Zuxuan Wu, Zhiding Yu, and Jose M Alvarez. Hydra-mdp++: Advancing end-to-end driving via expert-guided hydra-distillation. arXiy preprint arXiv:2503. 1285000,.1222,. 2025.
12

---

[32] Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint (:Xiv:2406.0848) 2024.
[33] Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406. 06978, 2024.
[34] Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12037-12047, 2025.
[35] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 2022.
[36] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
[37] Chenlin Meng, Robin Rombach, Ruiqi Gao. Diederik Kingma, Stefano Ermon, Jonathan Ho. and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14297-14306 2023.
[38] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7077-7087, 2021.
[39] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick. Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020.
[40] Katrin Renz, Kashyap Chitta, Ommiel-Bogdan Mercea, A Sophia Koepke, Zeynep Akata, and Andreas Geiger. Plant: Explainable planning transformers via object-level representations. In Conference Robot Learning, pages 459-470. PMLR, 2023.
[41] Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8579- 8590, 2023.
[42] Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced au- tonomous driving using interpretable sensor fusion transformer. In Conference on Robot Learning, pages 737. PMLR, 2023.
[43] Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu. Reasonnet: End-to-end driving with temporal and global reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13723-13733, 2023.
[44] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.00250,, 2020.
[45] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 1tXiv:2011.1345550 2020.
[46] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[47] Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. In 2025 IEEE International Conference Robotics and Automation (ICRA), pages
IEEE, 2025.
[48] Ardi Tampuu, Tambet Matiisen, Maksym Semikin, Dmytro Fishman, and Naveed Muhammad. survey of end-to-end driving: Architectures and training methods. IEEE Transactions on Neural Networks and Learning Systems, 1364-134644 2020.
13

---

[49] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 6105-61 (10566114, 14. PMLR, 2019.
[50] Junming Wang, Xingyu Zhang, Zebin Xing. Songen Gu, Xiaoyang Guo, Yang Hu, Ziying Song, Qian Zhang, Xiaoxiao Long, and Wei Yin. He-drive: Human-like end-to-end driving with vision language models. arXiv preprint arXiv:2410.0505/1 2024.
[51] Wenhai Wang. Jiangwei Xie, Chuan Yang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint urXiv:2312.09245, 2023.
[52] Xinshuo Weng, Boris Ivanovie, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Par-
allelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449-15458 15458, 2024.
[53] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory- guiddd control prediction for end-to-end autonomous driving: A simple yet strong baseline. Advances in Neural Information Processing Systems, 35:61 19-6132, 2022.
[54] Zebin Xing, Xingyu Zhang, Yang Hu, Bo Jiang, Tong He, Qian Zhang, Xiaoxiao Long, and Wei Yin. Goalflow: Goal-driven flow matching for multimodal trajectories generation in end- to-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1602-1611 1611,,1,,200 2025.
[55] Brian Yang. Huangyuan Su. Nikolaos Gkanatsios, Tsung-Wer Wei Ke, Ayush Jain, Jeff Schneider, and Katerina Fragkiadaki. Diffusion-es: Gradient-free planning with diffusion for autonomous and iestruction-guided driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15342- -1353,, 2024.
[56) Chengran Yuan, Zhanqi Zhang, Jiawei Sun, Shuo Sun, Zefan Huang, Christina Dao Wen Lee, Dongen Li, Yuhang Han, Anthony Wong, Keng Peng Tee, al. Drama: An efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint arXiv:2408. 03601, 2024.
[57] Yanjie Ze, Gu Zhang, Kangning Zhang, Chenyuan Hu, Muhan Wang, and Huazhe Xu. 3d diffusion policy: Generalizable visuomotor policy learning via simple 3d representations. arXiv preprint 20l::2403.09544 2024.
[58] Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoging Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305. 10430, 2023.
[59] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pages 12104-12113, 2022.
[60] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. End-to-end urban driving by imitating reinforcement learning coach. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1-222-15233 2021.
[61] Yinan Zheng, Ruiming Liang, Kexin Zheng, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Weihao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, et al. Diffusion-based planning for autonomous driving with flexible guidance. arXiv preprint arXiv:2501.15564 2025.
[62] Julian Zimmerlin, Jens BeißBwenger, Bernhard Jacger, Andreas Geiger, and Kashyap Chitta, Hidden biases of end-to-end driving datasets. arXiv preprint urXiv:2412.09602 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the abstract and the introduction's 's last paragraph.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We are preparing release the code and model checkpoints. Besides, in Appendix we provide sufficient implementation details for researchers to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code and model checkpoints will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/pulee wttpcccppppp.cceeeeeee.ee..c.lct http://uuiiieeeeeeebbabieieeee hat////ooooeessseeeeeee oossiiccyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provided full details in Section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experiments randomly initialized multiple times to avoid the impact of random seeds on the robustness of the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detailed the compute resources used for the experiments in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See the conclusion in Section 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For all the datasets and algorithm baselines used in the paper, we have cited the original papers and provided the license, copyright information, and terms of use in the package in our code repository.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.paperssiit wwwpppppptsthhhoco thcode www.coo//aaaao datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented and the documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The paper does not involve LLMs as any important, original, or non- non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Details of Multi-task Loss Function Design
A.1 Loss Function Design
Compared to the deterministic mapping learning paradigm of explicit policies, diffusion models require denoising multiple noise levels, which introduces greater optimization complexity and computational overhead. To improve training efficiency and ensure stable model convergence in the CARLA simulation environment, we construct hierarchical two-stage training strategy: the first stage focuses on optimizing the parameters of the multi-modal perception module, while the second stage adopts an end-to-end joint training paradigm to optimize both the perception module and the diffusion decoder parameters.
In the first stage, we adopt multi multi-task learning framework and design task-specific objectives for core perception sub-tasks, including high-precision semantic segmentation, monocular depth estimation, and 3D object detection 23 An adaptive weighting mechanism is used to construct composite loss function, enabling efficient parameter learning for the perception module. Furthermore, this work proposes modular loss function system, with each component defined as follows:
Image Semantic Segmentation Loss Lie:: Supervises semantic segmentation predictions from the image perspective, using a class class-weighted cross cross-entropy loss function:
HxW
Lrem (y,
Σ Wc Σ
Vi. logloogi...... eps)
(11)
where C' the total number of semantic classes, H W is the image resolution, Vi.c {0. 1} indicates whether pixel i belongs to class e in the ground truth, Us.c [0,1] is the predicted probability, we > is the balancing weight for class c, and eps constant for numerical stability.
Bird's-Eye s-ye View Semantic Segmentation Loss Liev' For semantic predictions in BEV space, cross-entropy loss is calculated only within the camera-visible region Mealidd
Liev (Shev,, Shev; Myalia)
Sbev ()) (hev (2, eps) (12)
Myalid
(zas)
where Myalia represents the set of valid pixels, |Mratial is the number of valid pixels, Stev (I, y c) and Sber (z, y, c) are the ground truth label and predicted probability for class at BEV coordinate (x, y). respectively.
Depth Estimation Loss Ldepth: As an auxiliary task for the image branch, masked & loss used for depth supervision:
deepth (D, D.D: =
Σ
D(u, v) - D(u, v)
(13)
MMdoptil
where Maepth the set of valid depth pixels, D(,, v) and D(u, v) are the ground truth depth value and predicted depth value at pixel (u, v), respectively.
Object Detection Loss Laes: This loss function combines multiple subtasks to jointly optimize key attributes of objects, including position, size, orientation, velocity, and braking status, enhance overall detection accuracy and temporal consistency. It includes weighted combinations of several subtask loss functions. Each subloss function is defined as follows:
Center Heatmap Loss Lam: Uses Gaussian focal loss to optimize object center point predic- tion:
1 N Crm P. (P.P:a,))
(L [L- Pa log (x. (1- (' (Po.y) (o - Pa.)), otherwise
Σ
if P2
(14)
where N is the number of valid objects, Ω is the feature map space, Pr.y and Pr.y are the ground truth heatmap value and predicted heatmap value at position (z, y), respectively, and and y are focal loss modulation parameters.
22

---

Object Size Loss Lwa: Uses & loss to optimize object width and height prediction:
(w, W) N
W. W.il
(15)
where W, = [wy, h,]"" and W; = [ū, Jty" are the ground truth size and predicted size of the i-th object, respectively.
Center Point Offset Loss Lont: Also uses & loss to optimize fine-grained center point position:
N
Lon(O,
Σ 10; - O,1
(16)
N
where O; = [oz, oy.l and Ô; = [oz,i, og are the ground truth offset and predicted offset of the i-th object, respectively.
Yaw Classification Loss Lasi Uses cross-entropy loss to optimize yaw angle category prediction:
N log
exp(Yi..)) Σ exp(Yi.)
( Y. Y)
(17)
N
where is the ground truth yaw angle category of the i-th object, K is the total number of yaw angle categories, and Yi, is the predicted score of the i-th object belonging to the j-th category. Yaw Residual Loss Les: Uses smooth & loss to optimize yaw angle residual prediction:
N
Lrs RR R;
Σ
smooth (R, - RR;; 8)
(18)
N
2=1
where R2 and R. are the ground truth yaw angle residual and predicted yaw angle residual of the i-th object, respectively, and smooth;ı is the smooth & loss function. The smooth LI loss function is defined as:
[0.5-22/4 if | < [|2| 0.5 0.5 otherwise
, =
(19)
The above loss functions form the core components of the first-stage training for the perception module. By adopting a multi-task learning paradigm, the model can simultaneously reconstruct multiple scene feature representations, including semantic segmentation maps, depth maps, and bird's- see's view segmentation. This approach not only achieves efficient and comprehensive scene understanding but also lays a solid perceptual foundation for subsequent trajectory generation tasks, ensuring that downstream decision-making modules can perform precise reasoning based on high- quality scene representations.
Additionally, on the NAVSIM benchmark, we use agent detection as an auxiliary task to enhance the model's perception capabilities in complex driving scenarios. The agent detection task adopts a hierarchical loss structure, including two complementary subtasks: existence discrimination and geometric parameter regression:
Agent Existence Classification Loss: Through binary cross-entropy loss, the model's ability to determine the presence of agents the scene is optimized:
N
Lexialy, a N [a Un (og(h)) + (1 - a) - yn.) Ph log(1 (1- - pm)]]
Σ
(20)
where yn E {0, }} represents the ground truth label whether an agent exists in the æ-th sample, Pn [0, 1] is the predicted existence probability, and a E [0, 1] and 3 2 0 are focal loss modulation parameters used to adjust the gradient contribution ratio of easy and difficult samples, increasing the model's attention to difficult samples.
23

---

Table 5: Hyperparameters of DiffE2E on CARLA.
Benchmark
Type
Parameter
Value
Epochs (One-stage training)
30
Epochs (Two-stage training)
30
Loss weight: diffusion
1.0
Loss weights: semantic, BEV, depth, speed
1.0
Loss weights: center heatmap, size, offset, yaw
1.0
Non-diffusion
Prediction horizon (Timesteps)
10
Learning rate
3e-4
CARLA
Batch size (One-stage training)
256
Batch size (Two-stage training)
16
Weight decay
0.01
Noise schedule
Square Cosine Schedule
Noise coefficient
0.0001, 0.02
Diffusion
Number of forward diffusion steps
100
Number reverse denoising steps
Agent Geometry Parameter Regression Loss: For existing agents, an adaptively weighted smooth LI loss function is used to accurately regress their spatial position and morphological parameters:
M Σ Σ
Coox bb = M
P4 14-smoothh,1 (bm,, bm...0 0)
(21)
m- (E(..y..,hh0)
where M is the number samples with agents in the batch, by.i and 6m. represent the ground truth value and predicted value of the i-th parameter of the agent bounding box in the mi-th sample. respectively, μ [Miz (hy: pws 1hh [4] is the importance weight vector for each parameter, and is the threshold parameter for the smooth L1 loss. This hierarchical loss design can simultaneously optimize the classification accuracy and localization precision of agent detection, ensuring accurate regression of bounding box parameters while maintaining high recall rates.
B Details of Experimental Setup
B.1
CARLA Benchmarks
B.1.1
Implementation Details
This study is based on a dataset constructed using the MPC expert from the TF++ framework (23] comprising total of 750,000 frames of high-quality driving data. We employ the RegNetY-3.2GF network 39 as the encoder for both image and LiDAR inputs. To optimize computational resources and improve training efficiency, we design two-stage training strategy: the first stage focuses on the perception module, utilizing a multi-task loss function that includes semantic segmentation, depth estimation, object detection bounding boxes, and bird's s-eye view prediction. In the second stage, pre-trained perception module from stage one loaded and further trained, using its outputs as conditional information to train the diffusion decoder. The entire training is divided into two stages, each trained for 30 epochs, with an initial learning rate of 3e-4. Batch size is adapted for different stages-16 16 for the first stage and 256 for the second stage- to accelerate the convergence of the diffusion model. The controller used is consistent with that in TF++. The specific hyperparameter settings are shown in Table 5 All experiments are conducted on four NVIDIA 3090 GPUs.
In the CARLA autonomous driving evaluation system, we use three core metrics to measure model performance. First is Route Completion (RC), which quantifies the percentage of the predetermined route completed by the vehicle, calculated as RC = Drusial x 100%, where D is the actual Dons
distance traveled and Drotal is the total route length. Second is the Infraction Score (IS), used to evaluate driving safety by accumulating penalty coefficients for violations: IS = -I Pi. where p; the penalty coefficient for the i-th violation (such as pedestrian collision 0.50, vehicle collision 0.60, static object collision 0.65, running red lights 0.70, running stop signs 0.80, etc.), and N is the
24

---

total number of violations. Finally, the Driving Score (DS) serves comprehensive evaluation metric, combining completion and safety: DS RC IS. higher DS value indicates stronger ability to complete tasks while ensuring safety, making it key indicator for evaluating the overall performance of autonomous driving systems.
B.1.2 Benchmarks
Longestó Benchmark is high-difficulty, scalable evaluation benchmark designed to verify the performance of autonomous driving models in complex scenarios. Following [9]. we use Longestó as an alternative evaluation scheme for local ablation studies and multiple experiments. This benchmark selects the 6 longest routes from each town among the 76 training routes provided by CARLA, totaling 36 routes with average length of 1.5 kilometers (close to the official leaderboard's 1.7 kilometers). To increase evaluation difficulty, we set the highest traffic density, including numerous dynamic obstacles (vehicles, pedestrians), combined with 6 weather conditions (such as Cloudy, Wet, HardRain) and lighting conditions (such as Night, Dawn, Noon), while also including predefined adversarial scenarios (such obstacle avoidance, unprotected left turns, pedestrians suddenly crossing, etc.). Compared to benchmarks like NoCrash and NEAT routes with shorter routes and lower traffic density, Longestó more closely resembles the challenges of real scenarios, providing a more comprehensive validation of model robustness in complex urban environments (such lulti-lane intersections and dense traffic).
Town05 Benchmark is another important evaluation benchmark in CARLA, divided into Town05 Short and Town05 Long settings. Town05 Short includes 10 short routes (100-500 meters), each with 3 intersections: Town05 Long includes 10 long routes (1000-2000) meters), each with 10 intersections. These two settings are specifically designed to verify model driving capabilities in high-density dynamic traffic environments (including vehicles and pedestrians) and complex scenarios (such as unprotected intersection turns and randomly appearing pedestrians). Typically, models perform better on the Short benchmark, while performance decreases on the Long benchmark, reflecting the challenges of handling complex scenarios and long-term planning in long- long-distance driving, We use the same evaluation metrics as Longestó (DS, RC, IS) to measure model performance on the Town05 benchmark.
B.1.3 Baselines
WOR 4) model learning method based on the "world-on-rails" assumption, computing action value functions through offline dynamic programming to achieve driving policy distillation without actual interaction.
LAV 2 A full-vehicle learning framework that enhances training data diversity through viewpoint- independent BEV representation and trajectory prediction modules utilizing surrounding vehicle trajectories. The authors released two versions: LAV vl and LAV v2.
InterFuser 42 A safety-enhanced mudti-modal sensor fusion framework that integrates multi- view camera and LiDAR information through Transformers, outputting interpretable safety mental maps to constrain control actions.
TransFuser [9] A Transformer-baseee multi-modal fusion architecture that fuses image and LiDAR features through cross-view attention mechanisms, enabling navigation in complex long-distance scenarios.
TCP [53] A trajectory-control joint prediction framework combining GRU temporal modules and trajectory-guides attention mechanisms, integrating dual-branch outputs through scene-adaptive fusion strategies.
An interpretable planning Transformer based on privileged states, achieving global traffic element reasoning through object-centered representation and self-attention mechanisms, with attention weights visualizing decision bases.
PlanT 40
DriveAdapter 25 A perceptionn-planing decoupling framework that aligns student perception
features with teacher planning features through learnable adapters, enhancing policy safety through action-guided mask learning-
ThinkTwice (26): A cascaded decoding paradigm that achieves action iterative optimization through a three-stage "ooserreepreddcctrffiine""" mechanism, utilizing scene feature retrieval and future state prediction.
25

---

Table 6: Comparison on CARLA Town05 Long benchmark.
Method
Traj. Decoder
Img. Encoder
Input
DS
RC1 1
IS f
CILRS 10
Explicit Policy
ResNet-34 [[2]
C
7.8
10.3
0.75
LBC 3
Explicit Policy
ResNet-18., 17
12.3
31.9
0.66
Roach 60
Explicit Policy
ResNet-34 [1
41.6
96.4
0.43
ST-P3 200
Explicit Policy
EfficientNet-BA. [49
C
11.5
83.2
VAD 2
Explicit Policy
ResNet-50 [7
30.3
75.2
MILE [9
Explicit Policy
ResNet-18 [17
61.1
97.4
0.63
DriveMLM [51]
Explicit Policy
VIT-G [59]
C&L
76.1
98.1
0.78
VADv2
Explicit Policy
ResNet-50 [7
85.1
98.4
0.87
DiffE2E (Ours)
Diffusion Policy
RegNecY-3.2GF
C&L
90.8
100
0.91
Table 7: Comparison the CARLA TownO5 Short benchmark.
Method
Traj. Decoder
Img. Encoder
Input
DS
RC f
CILRS [10
Explicit Policy
ResNet-34 17
C
7.5
13.4
LBC
Explicit Policy
ResNeet118.34
17
C
31.0
55.0
TransFuser
Explicit Policy
RegNetY-3.2GF
39
C&L
54.5
78.4
ST-P3 20
Explicit Policy
EfficientNet-B4
49
C
55.1
86.7
NEAT g
Explicit Policy
ResNet-34 [17
C
58.7
77.3
Roach 660
Explicit Policy
ResNet-34 [17
C
65.3
88.2
WOR
ResNet-18.34 [17
C
64.8
87.5
VAD 27
Explicit Policy
ResNet-50 [17
C
64.3
87.3
VADv2
Explicit Policy
ResNet-50 [17
C
89.7
93.0
InterFuser
Explicit Policy
ResNet-50 17
C&L
95.2
DiffE2E (Ours)
Diffusion Policy
RegNetY-3.2GF
[3
C&L
95.2
99.7
T + 23::
An improved version of TransFuser that eliminates target point following bias through Transformer decoder. TF++ uses explicit uncertainty modeling with path and speed classification, while the TF++ WP variant retains waypoint prediction, achieving deceleration decisions through longitudinal averaging of continuous waypoints.
B.1.4 Other Results
In addition to CARLA Longest6, this research also rigorously evaluated the DiffE2E method on the CARLA Town05 Long and Town05 Short benchmarks, with detailed results shown in Table 6 and 7 On the Town05 Long benchmark, DiffE2E achieved a DS of 90.8 and an RC of 100, outperforming existing methods. On the Town05 Short benchmark, where environmental complexity is relatively lower, all algorithms generally showed improved performance, with the DiffE2E method still demonstrating excellent results, achieving DS of 95.2 and RC of 99.7. The outstanding performance across Town05 scenarios of varying complexity further validates the robustness and generalization capability of our method.
B.1.5 Analysis of failure cases
This section provides systematic visual analysis of the ablation study results from Section 4.3 with the aim professionally and intuitively comparing how different design choices affect the performance of the DiffE2E method. We focus on analyzing two representative types of failure cases: One-Stage Training and Noise Prediction.
As shown in Figure we systematically compare the performance of the diffusion model when using noise versus original trajectory as the output target the same ceersection-lef-tuuee scenario. According to the visualization results, when using noise as the output target, the predicted trajectory distribution tends be more concentrated during straight driving stage, this does not improve future trajectory prediction capability. After entering the intersection, the trajectory predicted based on noise deviates significantly, causing the vehiele fail make correct left turn along the intended path. In contrast, directly using the original trajectory as the prediction target allows the model to
26

---

(4) Noise
(b) Trajectory Prodiction
Figure 6: Visualization of output type ablation experiment.
(a) One-Stage Training
Two-Stage Training
Figure 7: Visualization of training stage ablation experiment.
better adhere to the left-turn route and complete the left turn successfully, demonstrating superior trajectory generation ability.
Figure systematically compares the performance of one-stage training and two-stage training strategies in the intersection-lefttuur scenario. The visualization shows that one-stage training leads to obvious deviations in the ego-vehicle trajectory, making it difficult for the vehicle to smoothly execute the left-turn maneuver. In contrast, with the two-stage training strategy, the generated ego-vehicle trajectory smoother and better aligned with the desired path, significantly improving decision accuracy and trajectory feasibility in complex intersection scenarios.
B.2 NAVSIM
B.2.1
Implementation Details
This research builds a model training framework based on NAVSIM's navtrain dataset. Unlike the image encoding architecture used in the CARLA simulation environment, we adopted VovNetV2-99 30 as the feature extraction backbone network in our NAVSIM experiments, choice inspired by the Hydra-MDP (33] method, which can more effectively capture visual features in complex driving scenarios. All experiments were conducted on four NVIDIA 3090 GPUs. The specific hyperparameter settings are shown in Table g
The navtrain dataset establishes highly complex driving environment evaluation system, char- acterized by carefully selected challenging scenarios with frequently changing driving intentions, deliberately excluding low-complexity driving situations such as stationary states and constant-speed cruising. This evaluation system quantitatively analyzes planning algorithm performance through combination of non-reactive simulation and closed-loop evaluation mechanisms, This study uses the
27

---

Table 8: Hyperparameters of DiffE2E on NAVSIM.
Benchmark
Type
Parameter
Value
Epochs
100
Batch size
64
Prediction horizon (Timesteps)
8
Learning rate
le-4
Ego progress weight
5.0
Non-diffusion
Time to collision weight
5.0
Comfort weight
2.0
NAVSIM
Loss weights: diffusion
10.0
Loss weights: agent class, agent box, BEV
10.0,100,10.0
Number bounding boxes
30
Noise schedule
Square Cosine Schedule
Noise coefficient
0.0001, 0.02
Diffusion
Number forward diffusion steps
100
Number reverse denoising steps
2
Predictive Driver Model Score (PDMS) comprehensive performance metric, which integrates performance across multiple key driving dimensions through weighted integration, including No at-fault Collision (NC), Drivable Area Compliance (DAC), Time-To To-Collision (TTC), Comfort (C),
and Ego Progress (EP). The detailed calculation methods for each metric are as follows:
The PDMS used in this study comprehensively evaluates autonomous driving policy performance by integrating sub-metrics across multiple key driving dimensions. Below is a detailed explanation of the definition and calculation method for each sub-metrie:
No at-fault Collision: Measures whether the ego vehicle has responsibility collisions with other traffic participants during simulation. The calculation adopts a ticred penalty mechanism: if responsibility collision occurs (such as active collision in dynamic scenarios), then scorenc = 0. resulting in PDMS of for the entire scenario; there is collision with static objects (such as stationary vehicles), then score = 0.5: if the ego vehicle is stationary or is rear-ended by vehicle from behind (non-responsibility collision), no penalty is counted, score 1.
Drivable Area Compliance: Evaluates whether the ego vehicle always remains within the drivable area. the ego vehicle deviates from the drivable area, then scoreDAC O, resulting in a PDMS of O; if compliant throughout, then scoreDAc =
Detects whether the minimum collision time between the ego vehicle and other vehicles is below the safety threshold. Default value scoreттe 1; if during the 4-second simulation, the predicted collision time between the ego vehicle and other vehicles is below the threshold (lypically 2 seconds), then scoreттc O.
Measures trajectory smoothness, based on thresholds for acceleration and jerk. Points are deducted if acceleration or jerk exceeds predefined thresholds (such as acceleration < 3.0 m/s? jerk < 5.0 m/s3). The final score is calculated by comparing the deviation of actual parameters from thresholds, normalized to the [0, 1] interval.
Comfort:
Evaluates the efficiency of the ego vehicle's progress along the predetermined path, comparing actual progress with theoretical maximum safe progress. The calculated score scoregp is the ratio of actual progress to theoretical maximum progress (limited between and 1). Theoretical maximum progress calculated by the PDM-Closed planner (based on search strategy for collision-fres trajectories) distance of the ego vehicle along the path centerline during simulation. If the theoretical maximum progress is less than meters, low negative progress values ignored.
Ego Progress:
safe upper limit; actual progress the longitudinal movement
PDMS integrates these sub-metrics through the following formula:
5 score + scorerre score
PDMS
II
scorem
x
(22)
5+5+2
me (NC.DAC)
weighted average
penalty term
28

---

The key logic of this metric design includes: (1) Safety first: NC and DAC serve as hard penalty terms, ensuring policy must meet basic safety requirements: (2) Multi-dimensional balance: coordinating driving efficiency, safety, and comfort through weight allocation (EP and TTC have the highest weights, followed by C): (3) Scenario adaptability: filtering simple scenarios (such as stationary or straight driving), focusing domination by trivial samples. This metric design aims to address the limitations traditional displacement error (ADE), providing simulation-driven multi-dimensional assessment that better reflects actual driving performance.
challenging conditions (such as turning, interaction), avoiding metric
B.2.2 Baselines
AD-MLP [58]: A lightweight multi-layer perceptron (MLP) network that uses only vehicle state as input, without a perception module, performing trajectory prediction only.
VADv2 o probability- -ased planning framework that constructs a vocabulary of 4096 candidate trajectories through trajectory space discretization.
UniAD 21 The first multi-task end-to-end framework that unifies perception, prediction, and planning tasks, using query mechanism achieve module collaboration.
LTF 99: A pure vision-based end-to-end driving framework that achieves implicit BEV representa-
tion through cross modal attention mechanisms using images, replacing LiDAR input.
PARA-Drive 52 A fully parallel architecture that simultaneously executes perception, prediction, and planning tasks, improving inference speed by 3 times.
TransFuser [9] A multi-modal end-to-end framework that fuses camera and LiDAR features, combining cross-modal attention and multi-scale feature interaction.
LAW [32] A latent world model that enhances spatiotemporal feature learning through future state prediction, jointly optimizing scene understanding and trajectory planning.
DRAMA 56 Mamba-based planner combining multi-scale convolutional encoders and state Dropout, using an efficient Mamba- TransffoTre decoder to generate long trajectory sequences.
Hydra-MDP 33 : A multi-teacher knowledge distillation framework that integrates human demon-
strations and rule-based experts (such traffic lights, lane keeping).
DiffusionDrrive 34 A truncated diffusion model for real-time sampling, reducing denoising steps
to 2 through anchor- anchor-guided noise initialization while maintaining multi molill coverage.
GoalFlow 54 A flow matching model with goal constraints, introducing feasible region scoring
mechanism and shadow trajectory refinement strategy.
Hydra-MDP++ 31: An enhanced version of Hydra-MDP, adopting a V2-99 30 image encoder and expanding metric supervision (such traffic lights, lane keeping, comfort).
C Visualization Comparison of Different Methods on the Navtest Benchmark
We conducted a systematic comparative analysis of the trajectory generation capabilities of DiffE2E, DiffusionDrive [34, and TransFuser [12] across diverse driving scenarios, with detailed results shown in Figure 8 and Figure
29

---

Front View
(a) DiffusionDrive
TransFuser
DiffE2E
Figure 8: Visualization Navtest benchmark.
30

---

ARME
AOMA
Front View
(a) DiffusionDrive
(b) TransFuser
(c) DiffE2E
Figure 9: Visualization on Navtest benchmark.
31

---

D Additional Studies
In this section, delve into two key research questions: RQI: Analysis of the impact mechanism of diffusion model output types, and RQ2: Generalization evaluation in out-of-distribution scenarios. Through systematic analysis of these questions, we aim to comprehensively reveal the technical advantages and potential limitations of the DiffE2E framework in end-to-end autonomous driving systems.
D.1 RQI: Impact Mechanism Analysis of Diffusion Model Output Types
Regarding model output parameterization, we conducted an in-depth study of two strategies for diffusion models: predicting noise versus directly predicting trajectories (original samples), as shown in Table 3 The results reveal significant disparity- -when the model outputs noise, the driving score drops to 20.1, 76% decrease compared to outputting trajectories, with vehicles barely able to follow lanes in the simulation environment. This substantial difference indicates that directly modeling the trajectory space is more effective than the noise space for tasks requiring high precision, such as autonomous driving.
We attribute this primarily to two factors: First, the trajectory space is inherently structured and semantically rich, with clear temporal correlations, spatial continuity, and a direct mapping driving task objectives (e.g., safety and efficiency). This allows the model to more effectively capture inherent patterns of driving behavior (such car following. lane changing, turning. and other typical driving paradigms). In contrast, the noise space lacks clear physical meaning and semantic structure, requiring the model to learn complex mapping relationships from abstract noise to specific trajectories. Second, in the noise prediction paradigm, there is a significant error accumulation effect during the
reverse generation process of diffusion models (x(") + x (TT-1) + + x (())). Small deviations in each denoising step are transmitted and amplified subsequent steps; as iterations progress, this cumulative error can cause the generated trajectory to deviate severely from the feasible driving space. This is particularly detrimental in tasks requiring high precision like autonomous driving.
D.2
RQ2: Generalization Evaluation in Out-of-Diste Distrrbbuttion Scenarios
DiftE2E not only performs excellently on the training set but also demonstrates outstanding gen- eralization ability in out-of-distribution test scenarios across both the CARLA simulator and the NAVSIM benchmark. Particularly in the complex traffic scenarios shown in Figure 3 DiffE2E exhibits highly intelligent contextual adaptation. For instance, when obstacle vehicle blocks the regular right-turn route in the right lane, the model dynamically adjusts its decision, temporarily planning safe straight path to avoid the vehicle ahead before completing the turn. This precise response and flexible adjustment to unseen traffic conditions fully demonstrate DiffE2E's robust generalization ability and decision-making intelligence when facing complex and variable driving environments.
This excellent generalization performance stems from the inherent advantages of the diffusion mechanism. During training. DiffE2E effectively builds robust mapping relationships from "non-ideal states" to target trajectories through systematic noise perturbation modeling and gradual denoising learning. This enables the model to handle various perception anomalies, rare behaviors, and unseen scenarios. More importantly, unlike traditional methods, DiffE2E models the complete trajectory probability distribution rather than a single deterministic solution. This distribution-bart approach naturally accommodates the inherent multi- multi-modality and uncertainty in autonomous driving tasks, laying a solid foundation for the model's deployment in complex real-world environments.
modeling
E Limitations & Future Work
In this section, we discuss the main limitations and directions for future research:
Sampling Algorithm. The current DDIM 44 sampling algorithm faces computational efficiency bottlenecks during trajectory generation. Although DiffEEE can generate trajectories with just 2 denoising steps, complex driving scenarios may require additional denoising steps ensure high-quality and diverse outputs. Additionally, the current generation process complex traffic scenarios could benefit from more explicit safety and reliability guidance.
32

---

Future work: We plan to explore more efficient sampling algorithms, such as DPM-Solver [36] and Elucidated Diffusion Models (EDM) 29 These methods can significantly reduce the number of sampling steps while maintaining generation quality through more precise ODE/SDE numerical approximations. We also aim to incorporate consistency models [46], distillation algorithms [77] or Shortcut Models [66] to further accelerate sampling efficiency.
Furthermore, will employ classifier-based energy -bargy-baseddde guidance techniques to constrain the sampling process. By integrating prior knowledge of traffic rules and collision avoidance, we expect enhance the safety and controllability of the generated trajectories. Combining efficient sampling algorithms with domain-specific guidance mechanisms is expected to significantly improve real- time performance while ensuring high trajectory quality.
33


Reviews:
[{"review_id": "Frx6eMMwjA", "reviewer": "Reviewer_yfNa", "summary": "This paper proposes DiffE2E, a novel end-to-end autonomous driving framework that integrates a hybrid diffusion-supervision decoder to jointly model multi-modal trajectory distributions and control-relevant semantics. Unlike previous works that either rely solely on regression-based policies or apply diffusion models only as post-processing modules, DiffE2E designs a Transformer-based hybrid decoder with structured latent modeling and collaborative training. It directly generates future trajectories via a denoising diffusion process, while simultaneously optimizing auxiliary supervision objectives (e.g., speed classification) to enhance controllability. A hierarchical multi-scale fusion module aligns LiDAR and camera features, and global goal conditioning is injected through cross-attention. Experimental results on CARLA and NAVSIM benchmarks demonstrate state-of-the-art performance, with ablation studies confirming the benefits of the hybrid training and decoder design.", "questions": "1. The above two weaknesses are also my primary concerns regarding the paper.\n\n2. In Table 3, the Full Diffusion and Full Discrimination variants achieve DS scores of 70.1 and 70.3, respectively, while the hybrid two-stage training achieves 82.9. This suggests that even when used as auxiliary losses, both components contribute significantly to performance. Could the authors provide further insight into the complementary nature of these losses and why their combination works so well?\n\n3. An important missing ablation: After training with the proposed two-stage setup, it would be helpful to separately evaluate the trajectories generated by the diffusion head and the supervision head at inference time. This would allow a more direct comparison of the strengths and weaknesses of each policy stream.\n\n4. There appears to be a typo in Figure 5: the PDMS score is marked as 181.5, which should likely be 81.5. Please correct this for clarity.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper introduces a unified Transformer-based decoder that jointly handles diffusion and supervised tasks via a structured latent space, which effectively combines the strengths of generative and discriminative paradigms.\n\n2. The method is comprehensively validated on both CARLA and NAVSIM with detailed metrics. DiffE2E outperforms prior methods including recent diffusion-based approaches (e.g., DiffusionDrive) in terms of planning metrics such as PDMS, TTC, and driving comfort, while maintaining a practical inference cost.\n\n\nWeaknesses\n1.  The use of diffusion policies in autonomous driving has already been extensively explored. While the proposed hybrid diffusion-supervision framework is a distinguishing aspect, the paper lacks a clear and in-depth explanation of why the hybridization is effective beyond empirical gains.\n\n2. The authors claim that diffusion policies offer improved robustness under data distribution shifts compared to supervised policies. However, the paper does not provide strong experimental evidence or analysis to substantiate this claim.", "was_revised": true, "final_justification": "My concerns have been addressed, so I raise my rating to Borderline Accept.", "rebuttal": "We sincerely appreciate the time and effort you have devoted to reviewing our work. Your thoughtful comments and constructive suggestions have greatly contributed to improving the clarity and rigor of our manuscript. Below, we provide detailed responses to each of your concerns.\n\n---\n\n**W1/Q1/Q2. Hybrid framework lacks justification beyond empirical gains / Further insight needed on why combining losses is so effective (Table 3).**\n\nThank you for your insightful feedback. In the revised manuscript, we have elaborated on the underlying rationale behind the hybrid diffusion-supervision framework and why it consistently outperforms its individual components. The effectiveness of this hybrid design stems from its ability to leverage the complementary strengths of diffusion models and supervised learning.\n\n* **Theoretical Perspective**\n\n  * **Complementarity Between Diffusion and Supervision**\n    Diffusion models excel at modeling complex, multimodal behavior distributions and capturing inherent uncertainties in driving tasks. However, they often require iterative sampling, which can limit their precision and efficiency. In contrast, supervised learning provides fast and deterministic predictions, but lacks diversity and uncertainty modeling. By jointly optimizing a diffusion-based generative objective and a supervised regression loss, our hybrid approach enhances both expressiveness and reliability, leading to improved accuracy and robustness. These theoretical advantages align well with our empirical findings on the CARLA and NAVSIM benchmarks.\n\n  * **Spatiotemporal Decoupling Strategy**\n    Our design further separates spatial and temporal prediction responsibilities: the diffusion model is used to generate spatial waypoints (paths), while the supervised model predicts velocity profiles. This decoupling improves coordination between path planning and motion control, strengthening closed-loop behavior. Compared to single-task or single-objective frameworks, our approach better captures the temporal diversity of driving behavior and maintains higher consistency across scenarios, enhancing both control stability and behavioral richness.\n\n* **Empirical Evidence**\n\n  As reported in Section 4.3, we conducted ablations comparing full diffusion (Full Diffusion) and full supervision (Full Discrimination). On the CARLA Longest6 benchmark, these configurations achieved Driving Scores of 70.1 and 70.3, respectively, while the proposed hybrid framework achieved 82.9—a relative improvement of over 15%. These results empirically validate the effectiveness of combining the two strategies.\n\n  Additionally, we observed that when relying solely on diffusion, the model struggled to maintain stable control during critical maneuvers, such as turning, due to imprecise velocity predictions. Conversely, fully supervised approaches like TransFuser lacked trajectory diversity and adaptability in dynamic environments. The hybrid output structure significantly improved closed-loop behavior, enabling more efficient, responsive, and robust driving in practice.\n\nWe have incorporated these theoretical and empirical analyses into the revised manuscript to better clarify the motivation and effectiveness of the hybrid approach. Thank you again for your helpful suggestions.\n\n\n**W2/Q1. Claims of robustness under distribution shift lack solid evidence.**\n\nThank you for highlighting the need for a clearer articulation of our model's objectives. We acknowledge that the original manuscript may have conveyed some ambiguity regarding the role of robustness in our design.\n\nTo clarify: the primary goal of adopting diffusion models in our work is to address **multimodal behavior prediction**, not robustness per se. The strong generative capacity of diffusion models allows them to effectively represent diverse and uncertain driving behaviors, which is essential for real-world end-to-end autonomous driving. That said, we also recognize that the **multi-step denoising nature** of diffusion models inherently improves fault tolerance, which incidentally contributes to robustness under distribution shifts.\n\n* **Theoretical Insights**\n\n  * **Multi-step Denoising as Implicit Correction:**\n    The iterative sampling mechanism of diffusion models allows progressive refinement of the trajectory. Even if the initial sample is noisy or biased, subsequent steps can adjust and correct for errors, offering stronger resilience to input disturbances.\n\n  * **Structural priors help mitigate distribution shifts:**\n    The sequential nature of generation imposes structural priors on the output space. Combined with stochastic noise sampling, this results in an implicit ensemble effect that mitigates the impact of observation noise and distributional variation.\n\n* **Experimental Support**\n\n  As shown in Figures 6 and 7, our model demonstrates greater stability and lower error rates across diverse driving scenarios. Compared to DiffusionDrive (which samples from a fixed distribution) and TransFuser (which lacks generative modeling), **DiffE2E** exhibits more consistent trajectory prediction, confirming its practical robustness.\n\nWe have revised the Introduction to clarify that robustness is an emergent benefit of the diffusion strategy, while the core contribution remains in enabling high-quality multimodal behavior modeling. The theoretical analysis and supporting results are further detailed in Appendix D.3.\n\n\n\n**Q3. Missing ablation: separate evaluation of diffusion vs. supervision heads at inference.**\n\nThank you for this important observation. We would like to clarify the architectural roles of the diffusion and supervision heads during inference.\n\nIn our method, diffusion and supervised representations are fused prior to decoding. Specifically, we concatenate the input queries corresponding to the diffusion and supervision tasks before passing them through a shared Transformer decoder. After decoding, we apply two separate output heads: the diffusion head is responsible for generating spatial waypoints (paths), while the supervision head produces velocity predictions. This spatiotemporal decoupling enables the model to combine the generative flexibility of diffusion with the precision of direct regression, producing trajectories that are both diverse and controllable.\n\nTo evaluate the contribution of each head independently, we conducted ablations using:\n\n* **Full Diffusion**, where both spatial waypoints (paths) and velocities are predicted via diffusion.\n* **Full Supervision**, where both outputs are generated using supervised regression.\n\nAs reported in Section 4.3, these two configurations yield Driving Scores of 70.1 and 70.3, respectively—substantially lower than the hybrid setup (82.9). This further supports the effectiveness of our design. We will update the manuscript to clarify this inference pipeline and the distinct roles of each prediction head.\n\n\n\n**Q4. Typo in Fig. 5: PDMS should be 81.5, not 181.5.**\n\nThank you for pointing this out, and we apologize for the confusion caused by the original figure.\n\nTo clarify: Figure 5 originally illustrated the **relative differences** in PDMS under various denoising steps. The baseline PDMS was 92.705, and we applied a vertical magnification factor of 10,000 to make the small variations visually discernible. The y-axis in that figure thus did not represent absolute PDMS values.\n\nHowever, we agree that this formatting may lead to misinterpretation. In the revised manuscript, we have updated Figure 5 to use the original PDMS value scale, clearly indicating the absolute scores without exaggeration. We have also revised the caption and corresponding discussion in Section 4.3 to ensure that the presentation is unambiguous and accessible.\n\n---\n\nWe hope that our responses have addressed your concerns comprehensively. Please do not hesitate to reach out should further clarification be required. Once again, thank you for your thoughtful and constructive review. Your feedback has significantly helped strengthen the quality and clarity of our work."}, {"review_id": "6SejDMVBLW", "reviewer": "Reviewer_diwU", "summary": "In this paper, the authors propose an end-to-end diffusion framework which uses diffusion as the primary paradigm for trajectory decoding, unlike previous works which have used diffusion as a refinement module. Specifically, their contribution is a hybrid diffusion and supervision decoder, which would replace the usual planning decoder in other deep learning based E2E driving approaches. In their hybrid decoder, both diffusion noise and supervision queries are processed through the same stream of transformer-based feature modeling before being processed by separate diffusion and supervision heads. Sensor fusion / encoding is handled by the same encoding architecture as TransFuser.", "questions": "- Per the point above, could authors comment on the performance of DiffE2E when using a ResNet-34 image encoder, then also DiffE2E when using camera data only? Since the image encoder is borrowed from previous works, it seems this component can be swapped and ablated. I would be curious to see how much of the performance gain can be attributable to image encoding. I also do not see this result in the ablation study (Table 3).\n- In the design of the hybrid decoder, the diffusion noise and supervision query are concatenated together and processed through the same stream. I am curious if the authors considered isolating feature modeling of diffusion noise and supervision queries in separate streams. \n- Equation 4: is g the output of f_goal, or is it the input to f_goal? Should f_goal --> g in Equation 4? The notation is confusing here.\n- The discussion on real-time performance considerations in the appendix was appreciated IMO, because it evaluates the feasibility of running diffusion-based decoders for end-to-end driving. Just a suggestion, but I would prefer to see Table 8 moved to the main body, especially since the latency was reasonable compared to explicit policies and given the model size.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The writing is very clear and Figure 1 is appreciated for delineating the difference between prior work with the proposed method. The method seems reasonably reproducible based on the writing.\n- Results show clear driving score improvement on CARLA benchmark. \n- Approach is an interesting application of hot paradigms (diffusion) to end-to-end driving use cases. \n- Supplementary videos were nice to visualize the closed loop results.\n\n**Weaknesses**\n- NAVSIM benchmark slightly outperforms other methods on most metrics, but uses both lidar sensor inputs and V2-99 image encoder. Table 2 is less convincing to me simply because there is not a result with DiffE2E benchmarked with ResNet-34 image encoder, and there is also not a result with DiffE2E with camera inputs only. If both of these results can outperform other methods, the results will be much more convincing for this benchmark. \n- Minor: Appendix D.1 and D.2 are swapped according to the description written under \"Additional Studies\".", "was_revised": true, "final_justification": "Authors have addressed my concerns and provided additional key results. With the changes provided, I am now more confident in the scoring of the submission. Confidence raised from 4 -> 5.", "rebuttal": "We sincerely appreciate the time and effort you have devoted to reviewing our work. Your comments have provided valuable guidance for improving the clarity, completeness, and rigor of our manuscript. Below, we present detailed responses to each of your questions and concerns.\n\n---\n\n**W1/Q1. Table 2 lacks fair comparison: no DiffE2E results with ResNet-34 or camera-only inputs.**\n\nThank you very much for this important observation. During the development of our framework, we indeed conducted experiments using ResNet-34 as the image encoder. However, since V2-99 was chosen as our primary baseline due to its superior performance, results with ResNet-34 were not included in the original submission. Furthermore, we appreciate your suggestion to evaluate the performance of DiffE2E under camera-only input, which we believe is valuable for assessing the robustness of our model under limited sensory input conditions.\n\nIn response, we have now incorporated additional experimental results based on two new configurations:\n\n* **DiffE2E-A**: The image encoder is replaced with ResNet-34, following the setup used in TransFuser, while all other settings remain unchanged.\n* **DiffE2E-B**: The LiDAR modality is removed, and instead, rear-view cameras (with the same resolution as the front-view images: 256×1024) are added to the input to enhance the field of view under camera-only settings.\n\nThe performance of these variants, along with the original DiffE2E configuration, is summarized below:\n\n| Method    | Img. Enc. | Input | NC   | DAC  | EP   | TTC  | C    | PDMS |\n| --------- | --------- | ----- | ---- | ---- | ---- | ---- | ---- | ---- |\n| DiffE2E-A | ResNet-34 | C\\&L  | 99.2 | 96.8 | 83.6 | 96.7 | 100  | 89.8 |\n| DiffE2E-B | V2-99     | C     | 99.7 | 97.1 | 84.2 | 98.2 | 99.9 | 90.9 |\n| DiffE2E   | V2-99     | C\\&L  | 99.9 | 98.6 | 85.3 | 99.3 | 99.9 | 92.7 |\n\nAs observed, **DiffE2E-A** achieves a PDMS of 89.8, indicating that while ResNet-34 is a viable backbone, its feature representation capacity is relatively limited. **DiffE2E-B**, which excludes LiDAR and relies solely on visual inputs, achieves a PDMS of 90.9. The original **DiffE2E**, leveraging both V2-99 and LiDAR, achieves the highest performance with a PDMS of 92.7. These results highlight the advantages of both using a more expressive image encoder and incorporating LiDAR for capturing 3D spatial cues essential to driving decisions.\n\nWe have included these new results and analyses in the revised manuscript to present a more complete and fair evaluation.\n\n**W2. Minor: Appendix D.1 and D.2 appear swapped per “Additional Studies” description.**\n\nThank you for your close reading. We have carefully reviewed the structure of the appendix and confirmed that the ordering of Sections D.1 and D.2 was indeed inconsistent with the narrative in the “Additional Studies” section. This has now been corrected in the revised version to ensure structural consistency and clarity for readers.\n\n**Q2. Hybrid decoder design: unclear if diffusion noise and supervision queries benefit from separate streams.**\n\nWe greatly appreciate your thoughtful and technical observation regarding the hybrid decoder design. In our current architecture, the diffusion noise queries and supervision-based queries are concatenated and jointly processed through a single shared Transformer decoder. This design choice was driven by the observation that both types of queries aim to model outputs within the same trajectory space. By allowing them to interact within a unified decoding stream, we enable deeper cross-signal integration and encourage the model to learn shared representations that are beneficial for both generative diversity and precise regression.\n\nTo further investigate this design choice, we also experimented with a dual-stream variant in which the diffusion and supervised branches are processed independently. While this theoretically avoids potential cross-task interference and preserves the modeling preferences of each branch, our preliminary results showed that such a design increased training complexity and hindered effective information fusion, ultimately offering no measurable performance gain. In fact, we observed a slight degradation in trajectory consistency and control stability.\n\nTherefore, we opted for the shared-stream strategy, which offers a better trade-off between architectural simplicity and modeling performance. In future work, we plan to explore more adaptive fusion strategies, such as confidence-based dynamic weighting or attention-gated mechanisms, to further improve the synergy between diffusion-based generation and supervised regression.\n\nThank you again for raising this critical design consideration, which has helped us better articulate the rationale and implications of our hybrid decoding approach.\n\n**Q3. Equation 4 notation is confusing.**\n\nThank you for pointing out the notational ambiguity in Equation (4). We have revised the expression by replacing $f_{\\text{goal}}(g)$ with $g$, which more accurately reflects the intended semantics. We also took this opportunity to audit related equations throughout the manuscript, clarifying the dimensions and meanings of key variables to improve overall clarity and notational consistency.\n\n**Q4. Latency results in Table 8 are insightful and deserve inclusion in the main paper.**\n\nWe completely agree with your assessment. Latency remains a major practical challenge for deploying diffusion models in real-time autonomous driving systems. In response, we have moved Table 8 and the accompanying latency analysis from the appendix to the main text, along with expanded discussion. The revised section now explicitly highlights how our method achieves a favorable balance between performance and real-time feasibility, demonstrating its potential for practical deployment relative to other approaches.\n\n---\n\nWe hope our responses have addressed your concerns comprehensively. Should you have any additional questions or require further clarification, we would be happy to provide it. Once again, thank you for your thoughtful, constructive, and detailed review. Your feedback has significantly improved the quality and clarity of our manuscript."}, {"review_id": "8cWdi2RKCx", "reviewer": "Reviewer_nP8L", "summary": "This paper proposed DiffE2E, a diffusion-based end-to-end autonomous driving framework. It combined diffusion models with supervised policies, effectively addressing the challenges of multi-modal driving behaviors and enhancing controllability and robustness at the same time. It enabled deep fusion of perception features with high-level targets, significantly improving the quality of trajectory generation. Through these approaches, it achieved state-of-the-art performance across multiple benchmarks in both CARLA closed-loop evaluations and NAVSIM benchmarks.", "questions": "1. The ablation studies regarding the ego state and the GRU module are not clearly explained, as neither is mentioned in the preceding context.\n2. It is recommended to include visualization examples for the ablation studies on the training paradigm. As this is one of the main contributions of the work, more in-depth discussions are suggested.\n3. The ablation study on the denoising steps yielded unexpected results. For instance, Diffusion-Planner[1] employs 10 steps, while DiffusionDrive uses 2 steps with an anchored Gaussian distribution. Further discussion on these findings would be beneficial.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is well-written and presents a wealth of experiments, including those conducted on CARLA and NAVSIM benchmarks, achieving impressive results.\nThis work focuses on end-to-end autonomous driving based on diffusion models, a promising research area. The proposed hybrid diffusion-supervised decoder demonstrates strong originality.\n\nWeaknesses:\nThe explanation and discussion of the supervised component in the hybrid diffusion-supervised decoder are insufficient. For instance, there is limited detail regarding the selected tasks and the reasons why these tasks enhance controllability and robustness, as well as improve closed-loop performance.", "was_revised": true, "final_justification": "Overall, I find this work innovative compared to prior diffusion-based approaches, proposing a novel integration method with supervised learning tasks with solid results. My questions have been thoroughly addressed, and I am accordingly raising my score to 5. However, there appears to be a discrepancy between the authors' rebuttal claim—that the diffusion branch generates waypoints while the supervised branch produces velocity curves—and the manuscript's description of the diffusion branch generating trajectories and the supervised branch outputting velocity classifications. I recommend aligning these descriptions for consistency.", "rebuttal": "We sincerely appreciate the time and effort you have devoted to reviewing our work. Your thoughtful comments and constructive suggestions have been instrumental in helping us improve the clarity, rigor, and overall quality of the manuscript. Below, we provide detailed responses to each of your concerns.\n\n---\n\n**W1. Insufficient explanation of the supervised component’s design and its role in improving controllability and robustness.**\n\nThank you for your valuable feedback. In the revised version, we have substantially expanded our explanation of the motivation, design rationale, and functional role of the supervised component in the hybrid diffusion-supervision framework. Relevant discussions have been incorporated into the related work, method, and experiment sections to clarify how the supervision branch enhances controllability, robustness, and closed-loop driving performance.\n\nIn particular, to address your comment regarding the insufficient discussion of how the supervised decoder contributes to performance gains, we have added the following clarifications:\n\n* **Motivation and Advantages of Supervised Learning Integration**\n  While diffusion models are well-suited for modeling complex, multimodal behavior distributions and trajectory uncertainty, their iterative sampling process can lead to instability in control execution—especially in closed-loop settings. This is particularly evident in velocity control, where small prediction errors can compound over time. In contrast, supervised learning offers efficient, deterministic mappings from perception features to control outputs, thereby improving controllability and real-time responsiveness. To leverage both paradigms, we design a hybrid decoder: the diffusion branch generates spatial waypoints, while the supervised branch predicts temporal velocity profiles. This combination enhances expressiveness, improves sampling efficiency, and stabilizes decision execution.\n\n* **Impact on Closed-Loop Driving Performance**\n  Our spatiotemporal decoupling strategy—diffusion for path planning and supervision for velocity prediction—substantially reduces the propagation of spatial errors to temporal control, resulting in improved closed-loop performance. In our CARLA experiments, we observe that relying solely on diffusion leads to velocity drift and tracking errors in complex maneuvers such as turns. Conversely, fully supervised models lack trajectory diversity and generalization capacity. The hybrid approach integrates the strengths of both: achieving more accurate trajectory generation and smoother execution. This design outperforms both single-branch baselines across all evaluation metrics, highlighting the supervisory component’s critical role in achieving robust and stable control.\n\nThese improvements and analyses have been systematically incorporated into the revised manuscript. Thank you again for your insightful suggestions.\n\n\n\n**Q1. The ego state and GRU ablations lack context and clear explanation.**\n\nThank you for highlighting the need for a more detailed explanation of the vehicle state and GRU modules in our ablation studies. In the original manuscript, these components were briefly described due to space constraints. In the revised version, we provide the following clarifications:\n\n* **Component Overview**\n  The ego state (e.g., speed, acceleration, yaw angle) provides essential information about the vehicle’s current motion status and dynamics, which supports more informed planning and control. Prior studies \\[1, 2] have demonstrated the importance of incorporating ego state into end-to-end driving models. In parallel, the GRU module serves to encode temporal dependencies in sequential observations, integrating historical context to improve prediction continuity and temporal consistency.\n\n* **Ablation on Ego State**\n  As shown in Table 3, removing ego state inputs significantly degraded performance: Driving Score (DS) dropped from 82.9 to 68.9, Route Completion (RC) fell from 96.2 to 88.8, and the Infraction Score (IS) decreased from 0.86 to 0.81. These results emphasize the vital role of ego state information in achieving high-quality planning and control.\n\n* **Ablation on GRU**\n  Similarly, removing the GRU module led to a DS of 66.8, RC of 75.5, and an IS increase to 0.88. The performance drop confirms that temporal modeling is essential for capturing motion patterns over time, and the absence of GRU leads to discontinuities and prediction inconsistencies.\n\nTogether, these ablations validate the necessity of both ego state encoding and temporal sequence modeling in our architecture. We have elaborated on these findings in the revised manuscript. Thank you once again for pointing this out.\n\n**Q2. Training paradigm ablations would benefit from visualization and deeper discussion.**\n\nThank you for this helpful suggestion. We fully agree that visualizing the ablation results would make the distinctions between training paradigms more intuitive and informative.\n\nAlthough we are unable to include figures in this rebuttal due to format constraints, we have added visualizations in the revised manuscript to illustrate the comparative outcomes of different training strategies. These include training curves, trajectory visualizations, and metric distributions. In addition, we have expanded the discussion in the experimental section to more thoroughly analyze the implications of one-stage vs. two-stage training, as well as full-diffusion vs. hybrid optimization. We believe these revisions significantly improve the clarity and interpretability of our results.\n\n**Q3. Denoising step results are unexpected and warrant further analysis.**\n\nThank you for raising this important point. In response, we provide the following clarifications regarding the observed behavior of our model under varying denoising steps:\n\n* **Spatiotemporal Decoupling Reduces Sampling Complexity**\n  Our model decouples spatial and temporal planning: spatial trajectories are generated via diffusion, while velocity control is handled by a supervised decoder. This separation allows the diffusion model to focus exclusively on path geometry, reducing its reliance on precise temporal modeling and enabling stable generation with fewer denoising steps.\n\n* **Task-Oriented Objective Enhances Efficiency**\n  Unlike traditional diffusion models that optimize for noise prediction, our model is trained to directly minimize the discrepancy between generated and ground-truth trajectories. This task-aligned objective avoids unnecessary high-frequency detail reconstruction, which further lowers the need for extensive sampling iterations.\n\n**Experimental Results:**\n  As shown in Figure 5, our model consistently achieves high and stable closed-loop performance (e.g., PDMS) across a wide range of denoising steps, with scores remaining tightly bounded between 92.705 and 92.724. This robustness further validates the superiority of our hybrid approach in both efficiency and reliability compared to conventional diffusion models. Notably, the original manuscript depicted relative PDMS changes with a 10,000× vertical scaling, using 92.705 as the baseline to highlight subtle variations across different denoising steps. In the revised version, we have reverted to using absolute PDMS scores on the y-axis to more intuitively illustrate the model’s stability under varying denoising steps.\n\nWe have added this discussion to the revised manuscript and clarified our task-specific adaptations to the diffusion process. Thank you again for your thoughtful and technical question.\n\n\n\\[1] Li, Z., Yu, Z., Lan, S., Li, J., Kautz, J., Lu, T., & Alvarez, J. M. (2024). *Is ego status all you need for open-loop end-to-end autonomous driving?* In CVPR (pp. 14864–14873).\n\n\\[2] Zhai, J. T., Feng, Z., Du, J., Mao, Y., Liu, J. J., Tan, Z., ... & Wang, J. (2023). *Rethinking the open-loop evaluation of end-to-end autonomous driving in nuScenes*. arXiv:2305.10430.\n\n---\n\nWe hope our responses have satisfactorily addressed your concerns. Please feel free to reach out if any further clarification is needed. Once again, we sincerely thank you for your thoughtful and constructive review, which has helped significantly improve the quality of our work."}, {"review_id": "nkEoDXKAG3", "reviewer": "Reviewer_vpqd", "summary": "Diffusion policies achieved great successes in the manipulation community. There are currently several concurrent efforts to integrate diffusion policies with end-to-end autonomous driving policies; this work is one of them.\nThe paper proposes to add a diffusion transformer head on top of the TF++ (and NavSim TF) architecture.\nSeveral implementation details are discussed and ablated.\nThe work sets itself apart from concurrent work by using more rigorous closed-loop evaluation and by achieving strong performance that advances the state-of-the-art on both the CARLA longest6 v1 and NavSim benchmark.", "questions": "Line 58, 59: “It first aligns LiDAR and image features via a hierarchical bidirectional cross-attention mechanism for accurate multiscale perception.”\nThe mechanism here seem to me to be the standard TransFuser sensor fusion module. If so it should be cited when described to remove ambiguity. If not, I would like the authors to go into more details on what the difference is.\n\nThere are a bunch of implementation details missing in the paper, such as which data augmentation strategies were used, which controller was used in CARLA, what GPS denoising strategy was used,  …\nI assume these details are the same as in TF++ / NavSim TF, but it would be better if the authors would explicitly state this.\n\nFrom the writing, it is a bit unclear which of the TF++ decoder heads are retained in this architecture. The paper claims that “the advantages of diffusion policy and supervised policy.” are combined. The loss function section describes the TF++ classification head. How is this output used / ensembled at inference? Is the path prediction GRU of TF++ still used?\n\n83 DS performance is around the same performance as the TF++ data annotator, which would mean DiffE2E reached expert-level performance on this benchmark. It might be worth pointing that out by including expert performance in the longest6 table.\n\nThere seems to be inconsistency in the paper on whether the perception encoder is frozen during the second stage of training. Line 493 claims it is not frozen, Line 572 claims it is.\nWhich one is correct?\n\nSome questions about Table 3:\n\nI find it surprising in Table 3 that the difference between one-stage training and two-stage training is so dramatic. TF++ also observed improvements from two-stage training, but it was by far not this dramatic (+ 64 DS). Do the authors have any intuition why adding the diffusion decoder induces such a dramatic change in the training paradigm? The batch size of 16 used in the first stage seems to be quite small, in particular if the learning rate is not adjusted. Why is such a small batch size used?\n\nWhy does noise prediction in Table 3 have a driving score > 0? How is the car able to drive if it predicts noise?\n\nCould the authors elaborate what full diffusion means precisely in Table 3? As I understand, Full Discrimination would be equivalent to TF++?\n\nWhat is the difference between the Two-stage Training and the Full Diffusion entry in Table 3?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The strength of the paper is that it develops a simple method that can be widely used in end-to-end driving architectures and advances the state-of-the-art performance across two challenging driving benchmarks.\nCompared to concurrent efforts for integrating diffusion policies in end-to-end driving architectures, this work distinguishes itself by using closed-loop evaluation and achieving better performance.\n\n\nI have many comments on the details of the paper:\n\nThe introduction talks about 2 problems in end-2-end driving: multi-modal futures and generalization issues when facing distribution shifts. While both are true, the proposed method only addresses multi-modality, so it is strange why the writing puts a focus on distribution shifts.\n\nLine 72. We propose DiffE2E, the first end-to-end autonomous driving framework that uses diffusion models to directly generate trajectories and validate them in closed-loop testing in the CARLA simulator.\nThis line, while technically correc,t is a bit misleading. \nThe first being claimed here is that the method is the first diffusion model to generate trajectories and is evaluated in CARLA. \nAs in the other works using diffusion were evaluated on other benchmarks.\nIt could be made more clear that the first is about CARLA not about diffusion.\n\nThe paper claims that most diffusion policies replace the head (GRU in TF++ [3]?) instead of both the decoder and head proposed  here (Transformer decoder + head) Figure 1b).\nIt seems to me that this claim is not correct, since the cited papers [1,2] seem to replace both with diffusion decoders.\n\nLike in many works, using NavSim Hydra-MDP is misrepresented in Table 2. The best Hydra-MDP version (Hydra-MDP-C) with 91 PDMS should be quoted, not the random hydra-mdp ablation (Hydra-MDP-V8192-W-EP) that is quoted in this work (and others).\n\nCARLA evaluation can be quite random, which is why it is common to average at least 3 evaluation seeds on longest6 v1 (and ideally 3 training seeds as well, although this is less common).\nThe paper doesn't report standard deviation and doesn’t mention number of seeds, so I assume the evaluation consists of a single seed (but the baseline numbers copied from TF++ paper [3] are 3 eval seeds).\nThe authors should evaluate their method with 3 evaluation seeds on longest6 and update Table 1 with mean and std.\n\nTable 1 incorrectly cites Perception PlanT as a privileged method. PlanT is a privileged method, but Perception PlanT is not (it uses the TransFuser backbone to predict its input, so Img. Encoder should be RegNetY-3.2GF).\n\nThe 1000x scaling factor in Figure 5 is a bit silly and makes the differences look bigger than they are.\nThe paper should use a 1x scaling. A more standard technique to “zoom” into the plot is to use a y-axis anchored at a non-zero value.\n\nThere is a bug in Figure 2. The box says CenterNet decoder at the depth prediction module and conv decoder at the bounding box prediction module. These two names should likely be swapped.\n\n[1] Bencheng Liao and Shaoyu Chen and Haoran Yin and Bo Jiang and Cheng Wang and Sixu Yan and  Xinbang Zhang and Xiangyu Li and Ying Zhang and Qian Zhang and Xinggang Wang: “DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving”, CVPR 2025\n\n[2] Yinan Zheng and Ruiming Liang and Kexin Zheng and Jinliang Zheng and Liyuan Mao and Jianxiong Li and Weihao Gu and Rui Ai and Shengbo Eben Li and Xianyuan Zhan and Jingjing Liu: “Diffusion-Based Planning for Autonomous Driving with Flexible Guidance”, 2025 ICLR\n\n[3] Bernhard Jaeger, Kashyap Chitta, Andreas Geiger: “Hidden Biases of End-to-End Driving Models”. 2023 ICCV", "was_revised": true, "final_justification": "I thank the authors for answering my questions. I have already rated the paper accept and will keep my rating after the rebuttal.", "rebuttal": "We sincerely appreciate the time and effort you have devoted to reviewing our work. Below, we provide detailed responses to your questions and concerns.\n\n---\n**W1. Concerns regarding distribution shifts.**\n\nThe core motivation for adopting a diffusion-based approach in this study is to address multi-modal behavior modeling in end-to-end autonomous driving. Although robustness to distribution shifts is not the primary focus, the iterative nature of diffusion models brings additional benefits. We have reorganized the introduction to highlight multi-modality as the central goal, with robustness treated as a secondary advantage. Furthermore, we provide both theoretical justifications and empirical evidence to support the robustness of diffusion models.\n\n* **Theoretical insights**\n\n  * **Multi-step denoising enhances error tolerance**:\n    The diffusion model refines predictions step by step, correcting early-stage errors and improving resilience to noisy inputs.\n\n  * **Structured generation improves robustness**:\n    The sampling process imposes structural priors over the trajectory and acts as implicit ensembling, reducing sensitivity to domain shifts.\n\n* **Empirical validation**\n\n  As shown in Figures 6 and 7, DiffE2E achieves higher trajectory stability and lower errors under dynamic conditions, outperforming both DiffusionDrive (anchored Gaussian prior) and TransFuser (purely supervised). This supports the dual advantage of our hybrid design in generating diverse yet robust trajectories.\n\n**W2, W4, W5, W6, W7, W8, Q1, Q2, Q4 — Revisions to details.**\n\nWe corrected all issues noted by the reviewer:\n\n* Line 72: The sentence has been revised to clarify that *DiffE2E* is the first to be evaluated in closed-loop testing within CARLA, not the first to use diffusion models.\n* Table 1: PlanT is now corrected; Expert Driving Score (DS) is added. After re-evaluation, results will be reported using mean ± standard deviation across multiple seeds to ensure statistical reliability.\n* Table 2: Hydra-MDP is updated to its best-performing variant (PDMS 91).\n* Figure 2: Decoder label errors are corrected.\n* Figure 5: The y-axis scale has been adjusted from 1000× to 1× for clarity.\n* Lines 58–59: Missing citations for TransFuser and TF++ have been added.\n* Appendix: Now includes implementation details on data augmentation, controller design, consistent with the setup used in TF++.\n\n**W3. *DiffusionDrive* and *DiffusionPlanner* also replace decoders.**\n\nOur initial understanding aligned with yours. However, upon reviewing the codebases of *DiffusionDrive*, *DiffusionPlanner*, and *GoalFlow* \\[1], we identified critical distinctions not fully detailed in the original papers.\n\n* **Conclusion:**\n  While *DiffusionPlanner* indeed uses a DiT-based decoder to directly predict multi-agent trajectories, both *DiffusionDrive* and *GoalFlow* apply diffusion models after their decoders. Specifically, these models first generate intermediate outputs through a decoder, which are then used as conditions for a separate diffusion module.\n\n* **Clarification:**\n  TF++ improves upon TransFuser by replacing the global average pooling layer with a Transformer decoder, enhancing spatial reasoning and goal-point fidelity. *DiffusionDrive* and *GoalFlow* retain this decoder and apply diffusion post hoc. In contrast, DiffE2E conditions the diffusion model directly on perception features, bypassing intermediate decoder outputs. This architectural decoupling is a core novelty of our approach.\n\n**Q3. TF++ decoder usage at inference remains unclear.**\nIn our architecture, auxiliary tasks such as depth estimation and semantic segmentation are used only during training to enhance scene understanding and improve perception features. These tasks decode from the perception module and are not used during inference.\n\n**Hybrid Diffusion and Supervised Strategy:**\n\nThe phrase refers to how we construct decoder queries: we fuse the noise input from the diffusion model with supervised task queries, using this combined representation as the query input. Specifically, the diffusion component is responsible for generating spatial waypoints (paths) to capture multimodal behaviors, while the supervised component predicts velocity profiles through direct regression. This decoupling allows each task to benefit from its most appropriate training paradigm.\n\n**GRU Usage:**\nWe retain the GRU from TF++ to enhance temporal consistency in path generation, which is critical for stable closed-loop behavior.\n\n**Q5. Perception encoder freezing is inconsistent across lines.**\nThe statement in Line 493 aimed to reflect findings from TF++, suggesting that joint optimization of the perception encoder and decoder may enhance closed-loop performance. However, after integrating the diffusion model, we found that its complex training dynamics made continued joint optimization detrimental to convergence.\n\nThus, DiffE2E adopts a staged training strategy: the perception encoder is frozen in the second stage to stabilize diffusion decoder training. We acknowledge the inconsistency in the original manuscript and will revise the description accordingly. An ablation study on encoder freezing will also be added to demonstrate its effect on performance.\n\n**Q6. Two-stage training gains in Table 3 seem unusually large.**\n\nWe indeed observed a substantial performance gap between single-stage and two-stage training, which we attribute to the optimization challenges introduced by the diffusion-based decoder. Unlike conventional supervised decoders, diffusion models must learn a sequential denoising process over multiple steps. This complexity makes joint training with the perception encoder from scratch prone to unstable gradients and poor convergence.\n\nIn single-stage training, the perception and diffusion modules must co-adapt simultaneously, often leading to suboptimal feature representations and degraded trajectory quality. In contrast, our two-stage approach first stabilizes the perception encoder via supervised learning, then freezes it while training the diffusion decoder independently. This separation enables more stable and effective optimization, resulting in the observed performance improvement.\n\nAs for the choice of a small batch size (16) in the first stage, it was motivated by the need for frequent parameter updates, which we found beneficial for learning high-quality features under limited memory constraints. Exploring larger batch sizes with appropriate tuning remains a promising direction for future improvement.\n\n**Q7. Surprising DS > 0 for noise prediction in Table 3.**\n\nIn our manuscript, *“noise prediction”* refers to the model’s estimation of the noise term $\\varepsilon_t$ at each diffusion timestep, which is distinct from the Gaussian noise injected during the forward corruption process. The reverse process in diffusion models generally adopts one of two common formulations:\n\n* **Noise Prediction (predicting $\\varepsilon_t$)**\n  The model predicts the added noise $\\varepsilon_t$ from the noisy input $\\mathbf{x}_t$. During inference, generation begins with pure noise $\\mathbf{x}_T$, and the model iteratively denoises it toward a final output. \n\n* **Target Prediction (predicting $\\mathbf{x}_0$)**\n  Alternatively, the model directly reconstructs the clean sample $\\mathbf{x}_0$ from the noisy input $\\mathbf{x}_t$. \n\nIn our ablation study, the noise prediction setting refers to predicting the noise term $\\boldsymbol{\\varepsilon}_t$ at each timestep, which is then used in the reverse diffusion process to reconstruct the full trajectory. However, this formulation prioritizes generative diversity over precision, often resulting in less accurate trajectory outputs. Consequently, it compromises control stability and safety in closed-loop evaluations, leading to inferior overall performance compared to directly predicting the final trajectory $\\mathbf{x}_0$, as shown in Table 3. By contrast, the target prediction formulation more closely aligns with trajectory-level supervision, yielding better spatial accuracy, improved control behavior, and higher DS. Therefore, we adopted the $\\mathbf{x}_0$-based training objective in the final version of our model.\n\n**Q8, Q9. Clarification of the ablation study setup.**\n\nDiffE2E adopts a two-stage hybrid diffusion and supervision paradigm: in the first stage, the perception module is trained with supervised objectives to learn stable feature representations; in the second stage, the perception encoder is frozen, and the diffusion decoder is trained independently to generate both trajectories and velocities. Accordingly, we conducted an ablation study to assess the impact of the first-stage training.\n\nIn the Full Diffusion setting, both trajectory and velocity are generated entirely through the diffusion process. All decoder outputs—including ego trajectory and speed—are sampled iteratively via denoising, resulting in a fully generative model. While this design increases output diversity, we observed unstable optimization and reduced accuracy in velocity prediction, ultimately degrading closed-loop performance.\n\nUnlike TF++, the Full Discrimination baseline fuses the target point with perception features before feeding them into the Transformer decoder, rather than injecting it directly into the GRU as in TF++.\n\nWe have clarified these distinctions in the revised manuscript and updated Table 3 to avoid confusion.\n\n**Paper Formatting Concerns**\n\nThank you very much for your valuable suggestion. In the revised version, we have thoroughly reviewed the manuscript to eliminate exaggerated or overly subjective language, ensuring that the overall tone remains scientific, objective, and in line with academic standards.\n\n[1] Xing et al., \"GoalFlow: Goal-driven flow matching for multimodal trajectories,\" CVPR 2025.\n\n---\n\nWe hope our responses have adequately addressed your concerns. Thank you again for your thoughtful and constructive review."}]

Author rebuttal:

Meta-review: The paper presents a framework for autonomous driving that integrates diffusion for trajectory decoding with additional loss functions to capture scene-relevant semantics. It received high review scores, with all reviewers acknowledging its strong empirical performance. However, the paper’s terminology is, in several places, inaccurate—for example, diffusion policies are also supervised. The AC requests that the authors revise the terminology, replacing “diffusion-supervision decoders” with “diffusion–regression–classification decoders” (all of which are supervised). The paper is accepted for publication, but the authors are expected to correct the terminology before final submission.
Decision: Accept (poster)