DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
Authors: ShuYao Shang
Yuntao Chen
Yuqi Wang
Yingyan Li
Zhaoxiang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: End-to-End Autonomous Driving
Direct Preference Optimization
Imitation Learning
Primary area: reinforcement_learning

Abstract: End-to-end autonomous driving has substantially progressed by directly predicting future trajectories from raw perception inputs, which bypasses traditional modular pipelines. However, mainstream methods trained via imitation learning suffer from critical safety limitations, as they fail to distinguish between trajectories that appear human-like but are potentially unsafe. Some recent approaches attempt to address this by regressing multiple rule-driven scores but decoupling supervision from policy optimization, resulting in suboptimal performance. To tackle these challenges, we propose DriveDPO, a Safety Direct Preference Optimization Policy Learning framework. First, we distill a unified policy distribution from human imitation similarity and rule-based safety scores for direct policy optimization. Further, we introduce an iterative Direct Preference Optimization stage formulated as trajectory-level preference alignment. Extensive experiments on the NAVSIM benchmark demonstrate that DriveDPO achieves a new state-of-the-art PDMS of 90.0. Furthermore, qualitative results across diverse challenging scenarios highlight DriveDPO’s ability to produce safer and more reliable driving behaviors.

DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
Shuyao Shang 2 Yuntao Chen Yuqi Wang" 1.2
Yingyan Li
Zhaoxiang Zhang 1.2
NLPR, Institute of Automation, Chinese Academy of Sciences,
University of Chinese Academy of Sciences MMiroMind
(shangshuyao2024, wangyuqi2020, watea000,,1yyii 1iyingyan2021, zhaoxiang.zhange zhang)@ia.ac.oom
cheayuntao0@@gmail.ccm
Abstract
End-to-end autonomous driving has substantially progressed by directly predicting future trajectories from raw perception inputs, which bypasses traditional modular pipelines. However, mainstream methods trained via imitation learning suffer from critical safety limitations, as they fail distinguish between trajectories that appear human-like but are potentially unsafe. Some recent approaches attempt to address this by regressing multiple rule-driven scores but decoupling supervision from policy optimization, resulting in suboptimal performance. To tackle these challenges, we propose DriveDPO, Safety Direct Preference Optimization Policy Learning framework. First, we distill unified policy distribution from human imitation similarity and rule-based safety scores for direct policy optimization. Further, introduce iterative Direct Preference Optimization stage formulated as trajectory-level preference alignment. Extensive experiments on the NAVSIM benchmark demonstrate that DriveDPO achieves new state-of-the-ar PDMS of 90.0. Furthermore, qualitative results across diverse challenging scenarios highlight DriveDPO's s ability to produce safer and more reliable driving behaviors.
Introduction
End-to-end autonomous driving has achieved remarkable progress in recent years. Unlike traditional modular pipelines, end-to-end methods directly predict future trajectories from sensor inputs, avoiding eгror accumulation across modules and simplifying the overall system design. Mainstream approaches [44 primarily rely on imitation learning. as shown Fig. [.. While effective in producing human-like behavior, imitation learning faces two critical safety issues. First, imitation learning minimizes the geometric distance between predicted and human trajectories. However, even slight deviations from human trajectories may lead to dangerous outcomes, as shown in Fig. (2a. Second, commonly used symmetric loss in imitation learning, such as mean squared error, penalizes deviations equally in both directions, while the safety impact can differ substantially across deviation directions, as shown in Fig. Pp. Consequently, policies trained solely through imitation often produce behaviors that appear reasonable but may be unsafe under actual driving conditions.
To address safety concerns, some recent methods 7 introduce rule-based teachers and adopt multi- target distillation to regress multiple rule-driven metrics as supervision signals, as shown in Fig. 22p. While such designs improve upon pure imitation learning regarding safety, they independently learn separate scoring functions for each anchor trajectory without directly optimizing the underlying policy distribution, ultimately leading to suboptimal driving performance.
"Co-first author.
CCoreponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Scor Model
Policy Model
Sup.
Humen Date
Sensor Inputs
Vorabulary
Anchor
Sup.
Human Data
Sensor Inputs
Vocabulary
A nchor
Score Model
Simmlator
(a) Imitation L Leanning Supervised by Human
(b) Score L Learning Supervised by Human and Rule
Chosen Traj.
Sensor Inputs
Anchor
Policy Modcl
Sup.
Human Data
Sensor Inputs
Anchor Vocabulary
Policy Model
Sup.
Rejected Traj.
Vocabulary
Simulater
Stage Unified Policy Distillation
Stage Safety DPO Fine-tune
(c) Ous: Policy Learning via Safety DPO
Figure/ 1: Comparison of different training paradigms for end-to-end autonomous driving, (a) The policy trained by imitation learning. [3] (b) The model trains multiple score heads using
human and rule-based supervision signals [5, (c) Our method first pretrains the policy using unified supervision signal that fuses human imitation similarity with rule-based safety scores, and then finetunes it via Safety DPO. Sup.: Supervision. Traj.: Trajectory.
Motivated by the safety issues of imitation learning and the indirect optimization limitation in score- based methods, we propose DriveDPO, Safety Direct Preference Optimization Policy Learning framework. First, to address the challenge that score-based methods optimize per-anchor scores instead of the overall policy distribution, we propose unified policy distillation approach that merges human imitation similarity and rule-based safety scores into a single supervisory signal for the policy model. Unlike score-based methods that construct separate score heads for each trajectory candidate, our method directly supervises the policy distribution over all anchors, enabling more coherent and end-to-ens policy optimization. However, directly combining imitation and safety supervision into a single training objective formulates a multi-objective optimization problem 10] To overcome this, we introduce the iterative Direct Preference Optimization framework [[]] and propose Safety DPO, which reformulates the supervision as trajectory-Ievar preference alignment task. It enhances the policy's responsiveness safety-oriented preferences through and targeted optimization. Through preference learning on trajectory pairs, Safety DPO promotes human-like and safe trajectories over those that human-like but unsafe, enabling more precise safety preference alignment in policy learning.
stable
We evaluate the proposed framework the NAVSIM benchmark 12 along with Bench2Drive
benchmark [13]. Under a unified setting using a ResNet-34 perception backbone [14], our method achieves new ofate-of-the-art PDMS of 90.0, surpassing the SOTA imitation based method by 1.9 PDMS and the SOTA score-based method by 2.0 PDMS. Further qualitative results also show that method substantially improves the safety of the learned policy in complex scenarios.
The contributions this paper can be summarized as follows: (1) We identify fundamental challenges in existing imitation learning methods and score- based methods: Pure imitation learning fails to distinguish between trajectories that appear human-like but are potentially unsafe, while score- based methods decouple score prediction from direct policy optimization, resulting in suboptimal performance. (2) To overcome these challenges, we propose DriveDPO, a Safety DPO Policy Learning framework that first distills unified policy distribution from human imitation and rule- based safety scores, followed by a DPO-based refinement stage for improved policy optimization. (3) We conduct comprehensive experiments the NAVSIM benchmark and achieve new state-of- the- art PDMS of 90.0, significantly advancing performance across multiple safety-critical metrics. By effectively suppressing unsafe behaviors, our method demonstrates great potential for safety-critical end-to-end autonomous driving applications.
Related Works
2.1 End-to-end Autonomous Driving
End-to-End Autonomous Driving 4 23 typically maps raw sensor inputs to driving actions, either in the form of trajectories lowelevel control commands, avoiding the cumulative errors and

---

Collision!
Collision!
Brake
Safe
Overtake
Follow-up Brake
(a) Tiny Deviation Risk
(b) Asymmetric Risk
Ego
genn
Traj.
Planmed TrR
Figure 2: Safety risks in imitation learning. (a) In overtaking scenarios, even minor deviations from the human trajectory can lead to lane departure and collisions. However, imitation learning assigns low penalties such trajectories due their high similarity to human demonstrations, introducing serious safety risks. (b) In emergency braking scenarios, deviations in different directions have different safety implications: planned trajectories that overtake the human trajectory may cause rear-end collisions, while those that lag behind tend to be safe. However, imitation losses like mean squared error penalize both equally, failing to reflect the asymmetry of driving risks. Traj.: Trajectory.
interface bottlenecks inherent in traditional modular pipelines. UniAD introduced modular end- to-end architecture that enables planning-oriented approach. VAD 22 further simplified UniAD's rasterized map representation into vectorized format. VADv2
3 introduced an anchor vocabulary, discretizing the continuous action space. SparseDrive 4 proposed an end-to-end method under the sparse paradigm. DiffusionDrive
incorporated diffusion policy for trajectory prediction. These methods rely on imitation learning, which leads to critical safety issues. Recent works like Hydra-MID MDP [5] introduced a nulti-teacher distillation framework that employs multiple score heads to regress imitation scores from human trajectories and rule-based metrics derived from simulator feedback. WOTE 7 employs BEV world model to predict future BEV states of trajectories and scores them. However, these score-based methods lack direct policy distribution optimization, as they only optimize per-anchor scores independently. In contrast, we propose directly optimizing the policy distribution toward safety-aligned behavior.
24
2.2 Reinforcement Learning Fine- Line-Tuning in Autonomous Driving
Reinforcement learning has gradually emerged an important paradigm for autonomous driving research. [25) demonstrated an on-vehicle deep RL system for lane following using monocular input and distance-based reward. [26] introduced implicit affordances to enable model-free RL in urban settings with traffic light and obstacle handling. CIRL (27 combined goal-conditioned RL and human demonstration improve success rates in CARLA. GRI (28) integrated expert data into off- policy RL for stable vision- based urban driving. Motivated by the success of reinforcement learning with human feedback (RLHF) (2) in large language models, an increasing number of studies have explored the paradigm of reinforcement learning finetuning in autonomous driving systems. DRIFT 34 proposed Reward Finetuning strategy for unsupervised LiDAR object detection, where a heuristically designed reward function acts as a proxy for human feedback. BC-SAC (35) introduced using a pre-trained imitation policy as the initial policy and finetunes in a simulated environment built from real driving data. Gen-Drive [36] proposed a behavior diffusion model that generates diverse candidate trajectories and is finetuned reinforcement learning to favor higher-reward outputs. AlphaDrive [18] adopts two-stage training strategy that combines supervised finetuning and reinforcement learning for planning- oriented reasoning. TrajHF [37], as concurrent work, introduced a framework that finetunes trajectory generator via reinforcement learning to produce trajectories more aligned with human driving style. However, TrajHF primarily focuses on driving style preference alignment without explicitly considering policy safety. In contrast, we introduce safety RI. finetuning into end-to-end autonomous driving, explicitly optimizing the policy to favor safer trajectories.
3

---

3 Preliminary
3.1 End to End Learning Task and Anchor Vocabulary
Given a raw sensor observation O, the model predicts trajectory points over the next T time steps. Each trajectory point is represented as (Iz,, yt, 0z), where = -,..., T. Traditional methods typically perform continuous trajectory regression by predicting each point in continuous space. In contrast, adopt the Anchor Vocabulary proposed in VADv2 []. which transforms the action space from continuous domain RT'x3 to a predefined discrete set of anchors V = {a' E 1 Ji=1. where the size of the Anchor Vocabulary is N, at denotes trajectory consisting of T' consecutive points, each with position and heading (I, 0). Under this formulation, the policy model Ty assigns a probability to each anchor in the vocabulary, forming discrete action distribution:
N Ex=1.
To (a:) = Pis i i= N,
(1)
The final predicted trajectory corresponds to the anchor with the highest probability:
arg max (T(a)) a.EV
(2)
3.2 Iterative DPO
Direct Preference Optimization (DPO) 38 is preference optimization framework introduced for reinforcement learning fine-tuning of large language models 33 The core idea of DPO is to optimize the policy directly based on preference pairs, encouraging it favor preferred outputs over less preferred ones. To mitigate the out-of-distributie which introduces an intermediate Reward Model to evaluate the quality of different outputs and has achieved notable success across various domains 44) Formally, given policy and a reward model rg. for each input æ the policy generates set of N candidate trajectories ) The reward model assigns scalar scores (a:) to each candidate. The sample with the highest score is selected as the chosen trajectory Owr and the one with the lowest score selected as the rejected trajectory a The DPO loss then encourages the policy to prefer an over a; relative to a fixed reference policy TRa
issues, iterative DPO [I] was proposed,
(aw) (Ra((a)) LDO - llog a [og Ta(am) - log
To To(a1)
(3)
where (.) is the sigmoid function, and / is temperature parameter.
4
Method
In this section, we propose a Safety DPO Policy Learning framework. First, we introduce the overall policy model architecture (Sec. 4.1.. which includes perception encoding, anchor tokenization, and cross-attention fusion for decision-making, To enable direct policy optimization that integrates human demonstrations and rule-based safety signals, we propose Unified Policy Distillation (Sec. 4.2) which integrates imitation similarity and rule-based score into a single supervision distribution. Finally, to alleviate the optimization challenges of multi-objective supervision and further improve policy safety, we present a Safety DPO method (Sec. 4.3), which fine-tunes the policy via iterative DPO.
4.1 Policy Model Architecture
The input to our model includes multi-view camera images, LiDAR point clouds, the current ego state, and navigation command. The model outputs probability distribution a predefined set of N discrete candidate trajectories, as shown in Fig.3p. We begin by constructing a discrete set of anchor trajectories define the output space of the policy network. Specifically, we apply k-means clustering on human driving trajectories from the NAVSIM Navtrain split [12 to obtain N representative anchor trajectories, denoted as: V {a:).N E NxTxxx a; E Txx Each anchor trajectory a, consists of T future steps, with each step represented by (Ix, Dt, 0z), L t= I,..T. These anchors are passed through an Anchor Tokenizer. which begins by applying NeRF-style Fourier positional encoding [49] to capture spatial structure:
r = (V) (s)n(2'mV), cos(2" ())2m//),..) TV),... ...
.,n sin(2'-1 --), (os) xV)) E R.N. NXTX6L
(4)

---

(c) Safety DPO
Policy Head
Anchor Tokenizer
111
Softmax
Linear
Construct Preference Pair
Sample
Linear
ReLU
DPO loss
ReLU
Linear
Chosen
Rejected
e
Linear
Pos Emb
KL loss
Policy Head
Scene Tokens
K, V
T
uniffed
Transformer Decoder
W1
PDMS(a;)
Softmax
Transfuser Backbone
W2
Anchor Tokens
Sim(a))
Encoder
Encoder
Anchor Tokenizer
Human Trajectory
Image
Lidar
Anchor 'ocabulary
Simulator
(a) Policy Model Architecture
(b) Unified Policy Distillation
Figure 3: Overall pipeline of the Safety DPO Policy Learning framework. (a) Given multi-view camera images and LiDAR point clouds, Transfuser backbone encodes the scene into a set of scene tokens. Meanwhile, a predefined Anchor Vocabulary is processed by an Anchor Tokenizer to produce anchor tokens. The anchor tokens attend to the scene tokens through Transformer decoder to capture context-aware representations, which are then passed the Policy Head to produce probability distribution (b) To provide a unified supervisory signal, we compute distribution Punified by combining human trajectory similarity and rule-based safety score. This distribution supervises the policy output via KI. divergence loss. (c) To further improve policy optimization, we introduce the iterative DPO framework. We sample K candidate trajectories from the predicted policy distribution Te, and construct a preference pair to fine-tune the policy via DPO loss.
where L is the positional encoding dimension. The encoded trajectories I' are then passed through a multi-layer perceptron (MLP) to produce the final anchor token representation:
Famchor MLP(T) NxTxd
where d' is the embedding dimension. For perception, we adopt Transfuser (50) as the backbone to fuse camera images and LiDAR point clouds. The model first encodes the image and LiDAR inputs separately and then fuses them through multiple Transformer [5T] modules obtain a unified scene representation. This results in set of Scene Tokens: Escene E R Mxd where M is the number of
scene tokens. In the final decision decision-making stage, inspired by 2 5 6 we employ a Cross-Attentiin Attention- based Transformer Decoder [5]] to align anchor tokens with the scene context. Each anchor token acts as query and attends over the scene tokens to obtain a context-enhanced representation:
Exneber = TransformerDecoode((( = Eanchors K Escene, V
E Egenne)
E Nxd
(6)
where Fanctur denotes the fused anchor representation enriched with scene semantics. Finally, the policy network applies an MLP to each context-enhancee anchor token Ennchor. followed by a softmax function obtain the final policy distribution over anchor trajectories:
To (a)) Softmax (MLP (MMMLP(Emmcte)))) Eandhor))
I,...,N N
4.2 Unified Policy Distillation
Score learning methods 7 typically construct independent score heads for each trajectory anchor, predicting per-anchor scores rather than directly optimizing the policy distribution. To address this
5

---

issue, we propose unified policy distillation approach that compresses all supervision signals into a single policy distribution target, enabling end-to-end consistency optimization of the policy output. Specifically, we combine human imitation similarity and a rule-based safety score to serve supervisory signal over the output probability distribution. For each candidate trajectory a; E TR TR3 in the anchor vocabulary V. we first define the imitation similarity between each candidate trajectory a, and the human reference trajectory á E ITxx as the negative Euclidean distance. To ensure comparability across different scenes, we apply a softmax function to obtain the normalized relative similarity Sim; across all anchors:
Sim(a; (a;) = Softmax (-Ila; - all2), i
To obtain a rule-based safety score for each anchor trajectory. we use the high-fidelity NAVSIM simulator [1] which can perform forward simulation for each trajectory and returns multiple rule- based indicators, including No At-Fault Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Time-to-Collision (TTC), and Comfort (C). These metrics are then aggregated into a single scalar score known as the PDM Score (PDMS), computed as: PDMS = NC x DAC x (5 x EP + x TTC +2 2 C)/12 We then can evaluate each anchor trajectory using the NAVSIM simulator to compute its corresponding PDMS through forward simulation:
PDMS = ForwardSimulation((..
i= 1,..,.. N
In constructing the unified supervision distribution for policy learning, we introduce the log transfor- mation to map the imitation similarity and the safety score from the range of [0. 1] to (-oo, 0J, which amplifies the differences when the values are small. As a result, an anchor has a low safety score, its corresponding value after the transformation will be significantly lower than that of a safe anchor, effectively distinguishing safe trajectories from unsafe ones. In contrast, coore-based methods regress scores for each anchor independently and cannot capture the sharp disparity between unsafe and safe trajectories. Moreover, the log transformation preserves the relative differences in imitation similarity. guiding the policy favor those that are safe and more aligned with human behavior. Finally, apply a softmax function to construct a soft-target distribution to introduce a competition mechanism among candidate anchors. The final unified supervision distribution Punified is:
(as) = Softmax (w) log (a)))) + w2 (Dg(((M()))))
i= i N
(10)
where wi and W2 are weighting coefficients. We train the policy model by minimizing the KL divergence between the predicted distribution Te and the unified supervision distribution Panifist:
Lunifed
KL (Punified
TT)
(11)
4.3 Safety DPO
With the pre-trained policy obtained via unified policy distillation, multi-objective optimization problem [8-10] arises due to the joint supervision from both imitation and safety signals. To mitigate this challenge. adopt the iterative DPO framework [I]] and propose Safety Direct Preference Optimization (Safety DPO), which further refines the policy by explicitly favoring trajectories that are both like and safe while suppressing those that appear human-like but risky. We begin by sampling K candidate trajectories from the current policy distribution To E IR... The naive approach selects the trajectory with the highest score in Punified as the chosen sample and the one with the lowest score as the rejected sample. However, this approach results in overly simplistic preference pairs, limiting the effectiveness of preference-based optimization. To address this, we continue to use the highest-scoring trajectory as the chosen sample a. but design two strategies for selecting the rejected sample at The first selection method is Imitation- Based Rejected Trajectory Selection, which identifies trajectories that are spatially close to the human reference but exhibit poor safety performance. Specifically, it selects the rejected trajectory that is closest to the human trajectory while having low PDMS:
a arg min ||a: - áll2 s.t. (DMS(a.) <T, == K
(12)
where r is predefined safety threshold and à denotes the human trajectory. The second selection method is Distance-Bases Rejected Trajectory Selection, which identifies unsafe candidates spatially close to the chosen trajectory. Specifically, it selects the rejected trajectory that has a low PDMS but is closest to the chosen trajectory aw:
al arg min | awll2 0;
s.t. PDMS(a:) <T, i= 1= K
(13)

---

Table Comparison of end-to-end driving methods on the NAVSIM. The best results are denoted by bold and the second best results are denoted by underline. C: Camera. L: LiDAR.
Method
Supervision
Input
NCT
DAC
EPT
TTC1
C7
PDMS
PDM-closed
S2
GT Perception
94.6
99.8
89.9
86.9
99.9
89.1
Human
100.0
100.0
87.5
100.0
99.9
Ego Status MI.P
Human
Ego State
93.0
77.3
62.8
83.6
100.0
65.6
VADv2
Human
C
97.9
91.7
77.6
92.9
100.0
83.0
UniAD
Human
C
97.8
91.9
78.8
92.9
100.0
83.4
Human
97.4
92.8
79.0
92.4
100.0
Transfuser (50
Human
C&L
97.7
92.8
79.2
92.8
100.0
84,0
PARA-Drive
53
Human
C
97.9
92.4
79.3
93.0
99.8
84.0
LAW !M
Human
95.4
81.7
88.7
DRAMA
0
Human
C&L
98.0
93.1
80.1
94.8
100.0
85.5
GoalFlow
55
Human
C&L
98.3
93.8
79.8
94.3
100.0
85.7
ARTEMIS 56
Human
C&L
98.3
95.1
81.4
94.3
DiffusionDrrive
Human
C&L
98.2
96.2
82.2
94.7
100.0
88.1
Hydra-MDP [5]
Human & Rule
C&L
98.3
96.0
78.7
94.6
100.0
86.5
Hydra-MDP++ 16
Human & Rule
C
97.6
96.0
80.4
93.1
100.0
86.6
WOTE
Human & Rule
C&L
98.4
96.6
81.7
94.5
99.9
88.0
Ours (w/o DPO)
Human & Rule
C&L
97.9
97.3
84.0
93.6
100.0
88.8
Ours (full)
Human & Rule
C&L
98.5
98.1
84.3
94.8
99.9
90.0
Table 2: Closed-loop results on Bench2Drive. The best results are in bold.
Method
Efficiencyt
Comfortnesst
Success Rate (%)f
Driving Score
AD-MLP
48.45
22.63
0.00
18.05
UniAD
129.21
43.58
16.36
45.81
VAD
157.94
46.01
15.00
42.35
TCP
76.54
18.08
30,00
59,90
Ours
166.80
26.79
30.62
62.02
Experiments show that both methods significantly improve the safety performance of the policy compared to naive preference construction, with the first method slightly outperforming the second. Therefore, our experiments adopt the first method as the default preference pair construction strategy. Finally, given constructed preference pair (aw. at), apply the standard DPO loss:
LDOO = - log (3(1)) (1og Ta(( - log
To To(a1)
(14)
(Re((aw) TREr(a))
where (-)) is the sigmoid function, is temperature parameter, and Teea the reference policy.
Experiments
5.1 Benchmark
NAVSIM NAVSIM [12 benchmark combines real-world sensor data with non-interactive sim- ulation mechanism, which is built upon OpenScene [57]. reprocessed version of the nuPlan dataset (58). For each frame, the NAVSIM dataset provides eight high-resolution camera im- ages and fused point cloud data sampled at 2 Hz. The model can take a history of 1.5 seconds as input and is tasked with predicting eight future waypoints over a 4-second horizon. The fi- nal standardized training set, Navtrain, contains approximately 103,000 samples, and the test set, Navtest, contains around 12,000 samples. NAVSIM introduces a simulation-based metric called the PDM Score (PDMS), which integrates No At-Fault Collision (NC), Drivable Area Compliance (DAC), Ego Progress (RP), Time-to- Coollssoooll (TTC), and Comfort (C) into a single scalar score: PDMS NC x DAC (5 x EP TTC + 2 x C)/12.

---

Bench2Drive Bench2Drive [33 is closed-loop evaluation benchmark for end-to-end autonomous driving. The official training set contains approximately 13,638 short clips, covering 44 categories of interactive scenarios, 23 weather conditions, 12 towns, and a full sensor suite. The evaluation set is organized into 220 short routes that assess various interaction capabilities under different towns and weather. The official closed-loop metrics are Driving Score (DS) and Success Rate (SR), with Driving Efficiency and Comfortness additionally reported; specifically, SR requires the vehicle to reach the destination within the time limit without traffic violations, while DS aggregates route completion and violation penalties into a weighted summary.
5.2 Implementation Details
For NAVSIM benchmark, our model is trained on the official NAVSIM [2] training set, and evaluated on the official test set Navtest. For Bench2Drive, our model trained on base subset and evaluated on the official 220 evaluation routes. We follow the same perception setup and ResNet-34 backbone used in Transfuser for a fair comparison. Specifically, we use a concatenated front-view image of size 1024 x 256 formed by three forward- cameras the visual input, fused with 64 x 64 BEV LiDAR feature map. In addition, the model receives a state vector consisting of the current vehicle speed, acceleration, and navigation information. The size of the anchor vocabulary is set to = 8192. We use fixed weights wi 0.1 and u12 = 1.0 for unified policy distillation. The number of frequency bands in the positional encoding is set to I. 10. The predefined safety threshold is set to 0.3. All experiments are conducted on 6 NVIDIA L20 GPUs, with a batch size of 16 per GPU. We use the Adam optimizer (59) with learning rate of le-4. The model is first trained for 30 epochs using unified policy distillation, followed by 10 epochs of fine-tuning with Safety DPO. We sample K 1024 trajectories from the policy distribution for each DPO iteration and set the 3 = 0.1. In DPO training, inspired by [I]] we introduce an explicit KL regularization term to suppress distributional drift during training. Finally, similar to [35 we continue applying the KL loss from unified policy distillation during the DPO fine-tuning stage as an auxiliary loss.
5.3 Comparison with SOTA Methods
We conduct comprehensive comparison of our method against representative end-to -o--end baselines on the NAVSIM Navtest split using ResNet-34 as the visual backbone, as shown in Table [ our method using only unified policy distillation without DPO fine-tuning (Ours w/o DPO) already achieves PDMS of 88.8, outperforming the current SOTA imitation learning method DiffusionDrive [24] 18 and the SOTA score learning method WOTE 7 (88.0). This demonstrates that our unified policy distillation method brings significant performance gains. After applying DPO fine-tuning, our model further improves safety-related metries: NC increases by 0.6, DAC improves by 0.8, and TTC achieves gain of 1.2. These results indicate clear improvement in the safety and reliability of our policy across diverse scenarios. Our method ultimately achieves a new state-of-the art PDMS of 90.0, outperforming the SOTA imitation learning method (24] by 1.9 and the SOTA score-basee based method (7) by 2.0. It also surpasses the PDM-closed method (52), which takes the privileged GT Perception. We also conduct closed-loop evaluation Bench2Drive (Table Our method outperforms representative baselines on Driving Score and Success Rate, demonstrating its effectiveness in closed-loop settings.
5.4 Ablation Study
Ablation on Unified Policy Distillation. Table 3 conducts the ablation study to evaluate the effectiveness of the proposed Unified Policy Distillation method. First, compared the Score Learning method (ID-2), Unified Policy Distillation (ID-1) significantly improves overall policy performance. Furthermore, to assess the importance of combining both imitation and rule-based signals, we conduct additional experiments using single-source supervision. Results show that the Imitation-only method (ID-3) lacks the ability to distinguish high-risk trajectories and performs poorly on critical safety metrics such as DAC, resulting in poor performance. On the other hand, the Rule-only method (ID-4), which ignores human intent, tends to produce aggressive and unreasonable behaviors, leading to suboptimal results in key safety metrics such as NC and TTC.
Ablation on DPO and Rejected Trajectory Selection. Table conducts the ablation study to verify the effectiveness of DPO and the proposed rejected trajectory selection method. Applying
8

---

Table Ablation on Unified Policy Distillation. Score indicates we regress scores independently for each trajectory anchor. Policy indicates we optimize the policy distribution over all anchors. Sup.: Supervision. The best results are denoted by bold.
ID
Score
Policy
Human Sup.
Rule Sup.
NCT
DACT
EP
TTC₁
PDMS
X
97.9
97.3
84.0
93.6
88.8
2
V
X
97.8
96.0
81.6
93.7
87.3
3
X
V
97.6
91.2
77.5
92.9
82.5
4
X
V
X
95.3
97.3
85.9
88.7
87.2
Table 4:
Ablation on DPO and Rejected Trajectory Selection. The best results are denoted by
bold.
Method
NC1
DAC
EPT
TTC1
C1
PDMS1
w/o DPO
97.9
97.3
84.0
93.6
100.0
88.8
vanilla Selection
98.4
97.5
83.5
94.6
100.0
89.3
Sestance-Bssel Selection
98.1
98.2
84.3
94.2
99.9
89.7
IillattionBBaeel Selection
98.5
98.1
84.3
94.8
99.9
90.0
Thansfusee
Hydra- MDP
Ours
Transfuser
Hydra-MPP
Ous
pop00
Ego
Agnt
Pedestrian Fan
Barrier
Lane
Wallwwyy
Cruewwyy
Park.Are Area Human Trajectory
Planned Trajectory
Figure 4: Qualitative comparison with baselines. We compare method against Transfuser (50 and Hydra-MDP (5). Both the Transfuser and Hydra-MDP suffer from collisions or off-road behaviors in complex road structures, fine- greigga turning, and challenging emergency braking tasks. In contrast, our method consistently generates safer trajectories.
the vanilla selection method, which uses the highest and lowest in Punified preference pair, already improves PDMS by 0.5, indicating that preference- based optimization can effectively suppress low-quality outputs and boost overall performance. Compared to the vanilla selection method, the Distance- Based strategy yields an additional 0.9 improvement in PDMS, and the Imitation-Bace strategy improves PDMS by 1.2. This indicates that these strategies further enhance the safety performance by more effectively identifying and suppressing human-like but risky trajectories.
5.5 Qualitative Comparison
Qualitative Comparison with Baselines. Figure presents qualitative comparison between our method and two baselines: Transfuser [50], representing imitation learning, and Hydra-MDP (5), representing score-based learning. While Transfuser and Hydra-MDP tend to generate unsafe behaviors such as lane departures or collisions complex scenarios, our method consistently maintains safe and compliant decisions under diverse conditions.

---

w/o DPO
DRO
w/ DPO
w/ DPO
"aadi Trajectocy
Agent
Walkway
Craswwa
Trajectory
Figure 5: Qualitative visualization of DPO-enhanced behaviors across diverse scenarios. The DPO-enhanced policy exhibits more cautious behavior such as early deceleration and smoother cornering, and shows improved responsiveness in challenging scenarios like sudden braking of lead vehicles and pedestrian crossings.
Qualitative Comparison between Policies with and without DPO. To intuitively demonstrate the impact of DPO fine-tuning policy behavior, conduct a qualitative analysis of several challenging scenarios as illustrated in Figure The DPO-enhanced policy favors safer and more conservative decisions in complex interaction scenarios, verifying that the DPO fine-tuning improves safety awareness across diverse conditions.
Limitations and Future Works
Despite the significant progress our method has made enhancing the safety of end-to-end driving policies, several limitations warrant further study. First, our approach relies on the PDMS metric as the core criterion for safety evaluation. Although PDMS integrates multiple dimensions, including collision avoidance, drivable area compliance, ego progress, time-to-collision, and comfort, it remains a predefined weighted composite metric. As such, cannot fully capture all potential risk factors in complex driving scenarios. Future work may explore more expressive and flexible trajectory evaluation metrics to build more comprehensive safety assessment mechanism. Second, our rule-based supervision depends high-fidelity simulators to provide rule-driven evaluation scores. While effective, the preferences derived from such simulation are inherently limited by the rules' design and the simulator's precision. Moreover, access such figh-fideityy simulators is scarce, constraining the scale and diversity of available data. Therefore, future research should explore preference optimization methods that do not rely ground-truth perception labels by automatically mining latent preference structures from historical trajectory data. Such efforts could facilitate the development of weakly-supervised even fully self-supervised safety alignment strategies.
7 Conclusion
In this paper, we identify key safety challenges in imitation learning for end-to-end autonomous driving, where models often produce trajectories that appear human-like but are potentially unsafe. In addition, we highlight the limitations of existing score-based methods, which decouple supervision from policy learning and fail to provide direct optimization of the policy distribution. To tackle these challenges, we propose DriveDPO, Safety DPO Policy Learning framework. By combining unified policy distillation with Safety DPO fine-tuning, DriveDPO enables direct and effective optimization of the policy distribution. Comprehensive experiments and qualitative comparisons on the NAV SIM dataset demonstrate that DriveDPO significantly improves safety and compliance, showing its potential for deployment in safety-critical driving systems.
10

---

Acknowledgements
This work was supported in part by the National Natural Science Foundation of China (No. 62320106010, No. U21B2042).
References
[1] Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853-17862, 2023.
[2] Bo Jiang, Shaoyu Chen, Qing Xu. Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340-8350, 2023.
[3] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243 2024.
[4] Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. arXiv preprint arXiv:2405. 19620, 2024.
[5] Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406. 06978, 2024.
[6] Kailin Li, Zhenxin Li, Shiyi Lan, Yuan Xie, Zhizhong Zhang, Jiayi Liu, Zuxuan Wu, Zhiding Yu, and Jose M Alvarez. Hydra-mdp++: Advancing end-to-end driving via expert-guided hydra-distillation. arXiv preprint arXiv:2503. 12820, 2025.
[7] Yingyan Li, Yuqi Wang, Yang Liu, Jiawei He, Lue Fan, and Zhaoxiang Zhang, End-to-end driving with online trajectory evaluation via bev world model. arXiv preprint wxX::2504400944 2025.
[8] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in neural information processing systems, volume 31, 2018
[9] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In Advances in neural information processing systems, volume 33, pages 5824-5836, 2020.
[10] Weiyu Chen, Xiaoyuan Zhang. Baijiong Lin, Xi Lin, Han Zhao, Qingfu Zhang, and James Kwok. tradient-based multi-objective deep learning: Algorithms, theories, applications, and beyond. arXiv preprint arXiv:2501.1094/ 2025.
[11] Xiong Wei, Hanze Dong, Chenlu Ye. Ziqi Wang, Han Zhong, Heng Ji. Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: bridging theory and practice for rlhf under kl-constraint. In Proceedings of the 4/st International Conference on Machine Learning. pages 547544, 2024.
[12] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang. Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmarking, In Advances in Neural Information Processing Systems, volume 37, pages 28706-28719, 2024.
[13] Xiaosong Jia, Zhenjie Yang. Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop loop end-to-end autonomous driving. Advances Neural Information Processing Systems, 37:819-8444 2024.

---

[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[15] Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In European Conference on Computer Vision, pages 41 414.33. Springer, 2020.
[16] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision, pages 533-549. Springer, 2022.
[17] Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprini W4Xiv:2406.0848// 2024.
[18] Bo Jiang. Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning. arXiv preprint arXiv:2503.07668, 2025.
[19] Chonghao Sima, Kashyap Chitta, Zhiding Yu, Shiyi Lan, Ping Luo, Andreas Geiger, Hongyang Li, and Jose M Alvarez. Centaur: Robust end-to-end autonomous driving with test-time training. arXiv preprint arXiv:2503. 1650, 2025.
[20] Yuntao Chen, Yuqi Wang, and Zhaoxiang Zhang. Drivinggpt: Unifying driving world modeling and planning with multi-modal autoregressive transformers. arXiv preprint arXiv:2412. 18607, 2024.
[21] Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified trans- former for scalable end-to-end autonomous driving, arXiv preprint arXiv:2503.07656 2025.
[22] Ziying Song, Caiyan Jia, Lin Liu, Hongyu Pan, Yongchang Zhang, Junming Wang, Xingyu Zhang, Shaoqing Xu, Lei Yang, and Yadan Luo. Don't shake the wheel: Mowentum-aware planning in end-to-end autonomous driving. arXiv preprint arXiv:2503.0312. 2025.
[23] Yongkang Li, Kaixin Xiong, Xiangyu Guo, Fang Li, Sixu Yan, Gangwei Xu, Lijun Zhou, Long Chen, Haiyang Sun, Bing Wang. al. Recogdrive: A reinforced cognitive framework for end-to-end autonomous driving. arXiv preprint (tx.iv::50000005522 2025.
[24] Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint arXiv:2411. 2024.
[25] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dicu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In 2019 international conference on robotics and automation (ICRA), pages 8248 8254. IEEE, 2019.
[26] Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-fre free reinforce- ment learning for urban driving using implicit affordances. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7153-7162, 2020.
[27] Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. Cirl: Controllable imitative rein- forcement learning for vision-based self-driving. In Proceedings of the European conference on computer vision (ECCV), pages 584-599, 2018.
[28] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, and Fabien Moutarde. Gri: General reinforced imitation and its application to vision-based autonomous driving. Robotics, 12(5))127, 2023.
[29] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in newral information processing systems, pages 27730-27744, 2022.
12

---

[30] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, volume 30, 2017.
[31] Timm Teubner, Christoph M Flath, Christof Weinhardt, Wil Van Der Aalst, and Oliver Hinz. Welcome to the era of chatgpt et al. the prospects of large language models. Business & Information Systems Engineering. 65(2):95-101 2023.
[32] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop. Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In Forty-first International Conference on Machine Learning, 2024.
[33] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[34] Katie Luo, Zhenzhen Liu, Xiangyu Chen, Yurong You, Sagie Benaim, Cheng Perng Phoo, Mark Campbell, Wen Sun, Bharath Hariharan, and Kilian Q Weinberger. Reward finetuning for faster and more accurate unsupervised object discovery. In Advances in Neural Information Processing Systems, volume 36, pages 12250-13266 2023.
[35] Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7553-7560. IEEE, 2023.
[36] Zhiyu Huang, Xinshuo Weng, Maximilian Igl, Yuxiao Chen, Yulong Cao, Boris Ivanovic, Marco Pavone, and Chen Lv. Gen-drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning. arXiv preprint 4rX.v:2410.05582, 2024.
[37] Derun Li, Jianwei Ren, Yue Wang, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Zhongpu Xia, Peng Jia, Xianpeng Lang, et al. Finetuning generative trajectory model with reinforcement learning from human feedback. arXiv preprint arXiv:2503. 2025.
[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728-53741, 2023.
[39] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447-4455. PMLR, 2024.
[40] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior ppo for 1lm alignment? comprehensive study. arXiv preprint vrXiv:2404.10719 2024.
[41] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-fres reward. In Advances in Neural Information Processing Systems, volume 37, pages 144198-124235 2024.
[42] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.08417 2024.
[43] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01 2024.
[44] Jie Liu, Zhanhui Zhou, Jiaheng Liu, Xingyuan Bu, Chao Yang, Han-Sen Zhong, and Wanli Ouyang. Iterative length-regularized direct preference optimization: case study on improving 7b language models to gpt-4 level. arXiv preprint arXiv:2406. 2024.
13

---

[45] Songjun Tu, Jiahao Lin, Xiangyu Tian, Qichao Zhang, Linjing Li, Yuqian Fu, Nan Xu, Wei He, Xiangyuan Lan, Dongmei Jiang, et al. Enhancing Ilm reasoning with iterative dpo: A comprehensive empirical investigation. arXiv preprint arXiv:2503. /2854, 2025.
[46] Gong Xudong, Feng Dawei, Kele Xu, Yuanzhao Zhai, Chengkang Yao, Weijia Wang, Bo Ding, and Huaimin Wang. Iterative regularized policy optimization with imperfect demonstrations. In Forty-first International Conference on Machine Learning. 2024.
[47] Daechul Ahn, Yura Choi, San Kim, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. Isr-dpo: Aligning large multimodal models for videos by iterative self-retrospective dpo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1728-1736, 2025.
[48] Geon Hoon-Hy Kim, Youngsoo Jang. Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, and Moontae Lee. Safedpo: A simple approach to direct preference optimization with enhanced safery. OpenReview, 2024. Preprint.
[49] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[50] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer- based sensor fusion for autonomous driving. IEEE transactions on pattern analysis and machine intelligence, 15(11))12878811 12888882885, 2022.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, volume 30, 2017.
[52] Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with mis- conceptions about leaming-based vehicle motion planning. In Conference on Robot Learning, pages 1268-1281. PMLR, 2023.
[53] Xinshuo Weng, Boris Ivanovic, Yan Wang. Yue Wang, and Marco Pavone. Para-drive: Par- allelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449-15458, 2024.
[54] Chengran Yuan, Zhanqi Zhang, Jiawei Sun, Shuo Sun, Zefan Huang, Christina Dao Wen Lee, Dongen Li, Yuhang Han, Anthony Wong, Keng Peng Tee, et al. Drama: An efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint arXiv:2408.0360/ 2024.
[55] Zebin Xing, Xingyu Zhang, Yang Hu, Bo Jiang, Tong He, Qian Zhang, Xiaoxiao Long, and Wei Yin. Goalflow: Goal-driven flow matching for multimodal trajectories generation in end-to-end autonomous driving. arXiv preprint 2rXiv:2503.05689 2025.
[56] Renju Feng, Ning Xi, Duanfeng Chu, Rukang Wang. Zejian Deng, Anzheng Wang. Liping Lu, Jinxiang Wang, and Yanjun Huang. Artemis: Autoregressive end-to-end trajectory planning with mixture of experts for autonomous driving. arXiv preprint arXiv:2504. 100-0000441990, 2025.
[57] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 815-824, 2023.
[58] Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, et al. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 629-636. IEEE, 2024.
[59] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we accurately claim the contributions made in policy learning for end-to-end autonomous driving.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the supplementary material, we provide a detailed analysis of the limitations of our proposed method.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In Sec. we provide a detailed description of the proposed method, and additional Implementation Details are included in the supplementary material to support reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code will open-sourced upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ptaip..cc/pbblic/guu.eee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed descriptions of the experimental settings in Sec. and in supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to computational expense, error bars are not formally reported; however, fix the random seed during all experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the computational resources used in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp://neurips. www..//cubb.eecceeeititetie ines?
Answer: [Yes]
Justification: Ethics.
The research presented in this paper fully conforms to the NeurlPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The proposed method has direct societal impact, but hope it can inspire further research toward safer end-to-end autonomous driving.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
The proposed method poses no such risks, so no specific safeguards necessary.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets used in this paper have been properly cited and their usage complies with the respective licenses and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
The core method in this paper does not involve LLMs as any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21


Reviews:
[{"review_id": "3Y9UyVcZ2R", "reviewer": "Reviewer_VJCc", "summary": "This paper proposes DriveDPO, a policy learning framework that brings the Direct Preference Optimization (DPO) method to end-to-end autonomous driving. To address the safety shortcomings of imitation learning (which can produce human-like but unsafe trajectories), the authors introduce a two-stage approach: (1) Unified Policy Distillation, which combines human imitation and rule-based safety scores to train an initial policy, and (2) Safety DPO, which fine-tunes this policy via pairwise preference learning on full trajectories. Experiments are conducted on the NAVSIM benchmark with a new metric (PDMS) and show gains over both imitation learning and rule-supervised baselines.", "questions": "1. How would the method perform if the rule-based safety scores (PDMS) were replaced by human preferences directly? Would a learned reward model generalize better?\n2. Are the selection strategies for preference pairs (especially imitation-based) robust across datasets? How sensitive are results to the threshold?\n3. Would the method handle out-of-distribution data, like in more complex or unseen scenarios?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Applying preference-based learning via DPO to autonomous driving is a logical and increasingly relevant extension of RLHF methods.\n2. The idea of adapting DPO to control tasks is practical, leveraging recent successes in language modeling for robotics/control.\n3. Performance on PDMS is solid, with gains over prior methods such as Hydra-MDP and WOTE, and qualitative results suggest better safety behavior.\n\nWeaknesses:\n1. The paper claims to distinguish between human-like but unsafe trajectories, yet does not provide empirical or qualitative evidence that the learned preferences actually capture safety-related behaviors. While the paper is motivated by this issue, the safety scoring function still heavily depends on rule-based metrics rather than explicitly learning to recognize subtle forms of human-like but risky behavior.\n2. DriveDPO is a relatively straightforward combination of known components—policy distillation with hand-crafted safety scores, and standard DPO. The technical contribution lies more in integration than innovation.", "was_revised": true, "final_justification": "The rebuttal properly addressed all my questions.", "rebuttal": "Thank you for your detailed and constructive reviews. We are glad that you found our method “is a logical and increasingly relevant extension of RLHF methods” and “performance on PDMS is solid”. We would like to address the Weaknesses (**W**) and Questions (**Q**) below.\n\n**[W1.1, Q1] The safety scoring function still heavily depends on rule-based metrics rather than explicitly learning to recognize subtle forms of human-like but risky behavior. Would a learned reward model generalize better?**\n\nThank you for the insightful suggestion. We trained a reward model to distinguish between *human-like and safe* trajectories versus *human-like but unsafe* ones, and used it for DPO training. As shown in the table below, we found that replacing rule-based metrics with a learned reward model is effective. This may be because it can explicitly learn to recognize human-like but risky behaviors in ambiguous or complex scenarios that cannot be well captured by handcrafted rules, thus offering better generalization. We will include this result analysis in the revised version.\n\n| Method               | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| -------------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| Baseline w/o DPO     | 97.9 | 97.3 | 84.0 | 93.6 | 100.0 | 88.8 |\n| Rule-based reward    | 98.4 | 97.5 | 83.5 | 94.6 | 100.0 | 89.3 |\n| Learned reward model | 98.2 | 98.2 | 83.7 | 94.3 | 100.0 | 89.6 |\n\n\n**[W1.2] The paper claims to distinguish between human-like but unsafe trajectories, yet does not provide empirical or qualitative evidence that the learned preferences actually capture safety-related behaviors.** \n\nTo analyze the overall model preference, we computed the proportion of *human-like but unsafe* and *human-like and safe* trajectories among the Top 100 predicted trajectories on the test set (see the table below). After DPO fine-tuning, the proportion of *human-like but unsafe* trajectories significantly decreased, while the proportion of *human-like and safe* trajectories notably increased, indicating that the learned preferences can effectively distinguish such safety-related behaviors. Additionally, we present qualitative results in Figure 5 of the main paper, which further demonstrate the model’s improved ability to capture safety-related behaviors after DPO fine-tuning.\n\n| Method  | Human-like but unsafe rate (%) | Human-like and safe rate (%) |\n| :-----: | :----------------------------: | :--------------------------: |\n| w/o DPO |             38.44              |            61.56             |\n| w/ DPO  |             12.81              |            87.19             |\n\n\n\n**[W2] DriveDPO is a relatively straightforward combination of known components. The technical contribution lies more in integration than innovation.**\n\nWe would like to respectfully clarify that our method goes beyond a simple integration of known components, but involves non-trivial innovations to introduce Safety DPO into end-to-end autonomous driving. First, instead of prior methods that independently predict multiple scores and implicitly derive a policy, we unify imitation similarity and rule-based safety into a single policy distribution that explicitly guides policy learning. Second, rather than naively selecting preference pairs based on safety scores,  we specifically address the human-like but unsafe issue in imitation learning by proposing the Imitation-Based Selection strategy, which focuses on more effective learning of safety-related preferences. The effectiveness of our method is consistently demonstrated across both the NAVSIM and Bench2Drive datasets, as well as under different vision backbones.\n\n\n\n**[Q1] How would the method perform if the rule-based safety scores (PDMS) were replaced by human preferences directly?** \n\nWe directly replaced the rule-based scores with human preferences and observed that, while the performance improved compared to the baseline, it was still inferior to our proposed method (as shown in the Table below). This may be because of the absence of safety-oriented human preference data at the trajectory level, which means that even if the model aligns with human preferences, the absence of rule-based safety constraints may still lead to unsafe driving behaviors. However, we believe that with access to safety-relevant and fine-grained human preference datasets, which are currently lacking in the driving community, the model’s performance could be further improved. This will be an important direction we plan to explore in future work.\n\n| Method                 | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| ---------------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| Baseline w/o DPO       | 97.9 | 97.3 | 84.0 | 93.6 | 100.0 | 88.8 |\n| Only human preferences | 98.0 | 97.9 | 84.1 | 93.7 | 100.0 | 89.3 |\n| Ours                   | 98.5 | 98.1 | 84.3 | 94.8 | 99.9  | 90.0 |\n\n\n\n**[Q2] Are the selection strategies for preference pairs (especially imitation-based) robust across datasets? How sensitive are results to the threshold?**\n\nAs shown in the tables below, we conducted closed-loop evaluation experiments on the Bench2Drive dataset [1] and tested the Multi-Ability success rate across various complex driving scenarios. Our method achieves improvements over representative baselines in key metrics such as Driving Score, Success Rate, and Mean Multi-Ability Success Rate, demonstrating its robustness across datasets. In particular, our method achieves the highest success rate on the Emergency Brake task, indicating that the learned policy is more responsive to potentially risky behaviors.\n\n| Method | Efficiency | Comfortness | Success Rate(%) | Driving Score |\n| ------ | ---------- | ----------- | --------------- | ------------- |\n| AD-MLP | 48.45      | 22.63       | 0.00            | 18.05         |\n| UniAD  | 129.21     | 43.58       | 16.36           | 45.81         |\n| VAD    | 157.94     | 46.01       | 15.00           | 42.35         |\n| TCP    | 76.54      | 18.08       | 30.00           | 59.90         |\n| Ours   | 166.80     | 26.79       | 30.62           | 62.02         |\n\n| Method | Merging | Overtaking | Emergency Brake | Give Way | Traffic Sign | Mean  |\n| ------ | ------- | ---------- | --------------- | -------- | ------------ | ----- |\n| AD-MLP | 0.00    | 0.00       | 0.00            | 0.00     | 4.35         | 0.87  |\n| UniAD  | 14.10   | 17.78      | 21.67           | 10.00    | 14.21        | 15.55 |\n| VAD    | 8.11    | 24.44      | 18.64           | 20.00    | 19.15        | 18.07 |\n| TCP    | 8.89    | 24.29      | 51.67           | 40.00    | 46.28        | 34.22 |\n| Ours   | 16.28   | 28.95      | 53.06           | 30.00    | 45.00        | 34.66 |\n\nWe also conducted ablation studies on preference pair selection strategies using the Bench2Drive dataset. As shown below, all proposed strategies lead to performance improvements in closed-loop evaluation, confirming the effectiveness and robustness of our design.\n\n| Method                              | Success Rate(%) | Driving Score |\n| ----------------------------------- | --------------- | ------------- |\n| Ours (vanilla Selection)            | 29.95           | 61.39         |\n| Ours (w/ Distance-Based Selection)  | 30.48           | 61.72         |\n| Ours (w/ Imitation-Based Selection) | 30.62           | 62.02         |\n\nWe further performed ablation on the threshold sensitivity of the selection strategy (as shown below). Results show that under all threshold values, the performance remain higher than the baseline, suggesting robustness to threshold variations. We finally adopted $\\tau = 0.3$ as the default setting based on overall performance.\n\n| threshold | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| --------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| w/o DPO   | 97.9 | 97.3 | 84.0 | 93.6 | 100.0 | 88.8 |\n| 0.1       | 98.3 | 97.8 | 83.5 | 94.4 | 99.9  | 89.4 |\n| 0.2       | 98.2 | 98.0 | 84.4 | 94.2 | 100.0 | 89.7 |\n| 0.3       | 98.5 | 98.1 | 84.3 | 94.8 | 99.9  | 90.0 |\n| 0.5       | 98.2 | 97.6 | 85.0 | 93.7 | 99.9  | 89.5 |\n\n\n\n**[Q3] Would the method handle out-of-distribution data, like in more complex or unseen scenarios?**\n\nAs shown in the table below, we directly tested the trained model on the NuPlan dataset [2], which includes complex scenarios such as Heavy Traffic and Unprotected Cross Turn, as well as out-of-distribution settings where the city is changed to Boston. The model still performs well across all metrics, demonstrating the robustness and generalization ability of our method on out-of-distribution data.\n\n| Scenarios in Nuplan    | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| ---------------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| Heavy Traffic          | 98.3 | 98.4 | 86.5 | 94.0 | 99.9  | 90.7 |\n| Unprotected Cross Turn | 98.7 | 97.9 | 82.4 | 95.5 | 99.9  | 89.5 |\n| In City Boston         | 97.5 | 97.0 | 83.2 | 92.0 | 100.0 | 87.6 |\n\n\n\n[1] Jia, Xiaosong, et al. \"Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving.\" NeurIPS 2024.\n\n[2] Caesar, Holger, et al. \"nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles.\" arXiv:2106.11810."}, {"review_id": "HmpyVMFTyI", "reviewer": "Reviewer_2wBa", "summary": "DriveDPO introduces a two-stage policy learning framework for end-to-end autonomous driving by using both imitation and rule-based safety signals and refining the policy using Safety DPO. \n\nIt achieves state-of-the-art performance on the NAVSIM benchmark, significantly improving safety-related metrics.", "questions": "- Are the constructed preferences always meaningful and safe? There could be some analysis on this.\n- Can DriveDPO function effectively without a high-fidelity simulator?", "limitations": "- Statistical Robustness (computational resources limitations pointed in the paper)\n- Fixed Perception Backbone: The policy model uses a fixed ResNet/Transfuser backbone", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths of the Approach\n1. Safety-Aligned Policy Learning: Combines imitation learning and rule-based safety via a unified distribution\n2. Direct Optimization of Policy Distribution: Unlike score-based approaches, DriveDPO optimizes the actual policy. Introduces Safety DPO\n3. Achieves a new state-of-the-art.\n\n\nWeaknesses of the Approach\n1. Dependency on Rule-Based Scores\n    - Requires a high-fidelity simulator (NAVSIM) to generate rule-based metrics (PDMS), which may not generalize well to all driving environments or be available in real-world settings. So this might not translate to training in real-world.\n    - Preference pairs are constructed using rule thresholds and heuristics. This may not reflect the full nuance of real-world human preferences or corner cases.\n2. Anchor Vocabulary Limits Output Flexibility: The discretization of trajectory space into anchors, while efficient, may limit expressiveness or fail to capture rare but valid driving maneuvers.\n3. No Online Learning or Interaction: The system is trained offline and evaluated in simulation; there’s no feedback loop with deployment environments.\n\nWeaknesses / Gaps in the Work\n1. No Exploration of Failure Cases: The paper doesn't systematically analyze where DriveDPO fails (e.g., rare edge cases, adversarial driving).\n2. No Ablation on DPO Parameters\n3. While many RLHF methods are cited, DriveDPO is not directly compared with methods using reward modeling + PPO (e.g., GenDrive, TrajHF) on the same benchmark.\n4. Simulation-only Evaluation: The policy is not validated in a closed-loop interactive simulator setting", "was_revised": true, "final_justification": "I believe the authors have addressed most of my and other reviewers' concerns. I also think the simplicity of the work is a strength of this paper. So I have raised my score.", "rebuttal": "Thank you for your detailed and constructive reviews. We are glad that you found that our method “significantly improving safety-related metrics”. We would like to address the Weaknesses (**W**), Questions (**Q**), and Limitations (**L**) below.\n\n**[W1.1, Q2] Requires a high-fidelity simulator to generate rule-based metrics. Can DriveDPO function effectively without a high-fidelity simulator?**\n\nWe understand your concerns. Actually, computing rule-based scores only relies on a few elements provided by most driving datasets: the ego vehicle’s states and trajectories, lane geometry, and the positions of dynamic objects. Rule-based scores can be derived from geometric relationships between trajectories and labeled information, so there is no dependency on a high-fidelity simulator. We also computed rule-based scores on the Bench2Drive dataset and successfully used them for training (see results in **W3**), demonstrating the potential applicability of our method to real-world training scenarios.\n\n**[W1.2] Preference pairs are constructed using rule thresholds and heuristics. This may not reflect the full nuance of real-world human preferences or corner cases.**\n\nThank you for the insightful suggestion. As shown in the table below, we employed a learned reward model to construct preference pairs for DPO training, which proved to be effective. This may be because it can learn human preferences in ambiguous or complex scenarios that cannot be well captured by rules, thus providing better generalization. We will include this experimental analysis in the revised version. We also believe that with access to large-scale and high-quality human preference datasets, which are currently lacking in the driving community, the model performance could be further improved. This will be an important direction we plan to explore in future work.\n\n| Method| NC| DAC  | EP| TTC  | Comf  | PDMS |\n| -------------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| w/o DPO| 97.9 | 97.3 | 84.0 | 93.6 | 100.0 | 88.8 |\n| Rule-based reward| 98.4 | 97.5 | 83.5 | 94.6 | 100.0 | 89.3 |\n| Learned reward model | 98.2 | 98.2 | 83.7 | 94.3 | 100.0 | 89.6 |\n\n**[W2] Anchor Vocabulary Limits Output Flexibility.**\n\nWe understand your concerns. In fact, anchor-based planners can flexibly compose complex driving behaviors by selecting different anchor trajectories at different time steps, thus enabling coverage of rare but valid maneuvers. We also conducted Multi-Ability evaluations on the Bench2Drive dataset (see **W3**), including tasks like Merging, Overtaking, and Give Way. The competitive results indicate that the anchor-based strategy does not limit the model’s ability to express complex driving behaviors.\n\n**[W3] The policy is not validated in a closed-loop interactive simulator setting. The system is trained offline and evaluated in simulation.**\n\nWe conducted a closed-loop evaluation on the Bench2Drive dataset [1] (shown in the table below). The results show that our method outperforms representative baselines in key metrics, including Driving Score, Success Rate, and Mean Multi-Ability Success Rate. In particular, our method achieves the best performance in the Emergency Brake scenario, indicating the policy’s sensitivity to potentially risky behaviors. We will include the closed-loop results and analysis in the revised version.\n\n| Method | Efficiency | Comfortness | Success Rate(%) | Driving Score |\n| ------ | ---------- | ----------- | --------------- | ------------- |\n| AD-MLP | 48.45| 22.63| 0.00| 18.05|\n| UniAD  | 129.21| 43.58| 16.36 | 45.81|\n| VAD    | 157.94| 46.01| 15.00 | 42.35|\n| TCP    | 76.54| 18.08 | 30.00  | 59.90|\n| Ours   | 166.80| 26.79| 30.62 | 62.02|\n\n| Method | Merging | Overtaking | Emergency Brake | Give Way | Traffic Sign | Mean  |\n| ------ | ------- | ---------- | --------------- | -------- | ------------ | ----- |\n| AD-MLP | 0.00| 0.00| 0.00| 0.00| 4.35| 0.87  |\n| UniAD  | 14.10| 17.78| 21.67| 10.00| 14.21| 15.55 |\n| VAD    | 8.11| 24.44| 18.64| 20.00| 19.15| 18.07 |\n| TCP    | 8.89| 24.29| 51.67| 40.00| 46.28| 34.22 |\n| Ours   | 16.28| 28.95| 53.06| 30.00| 45.00| 34.66 |\n\n**[W4] No Exploration of Failure Cases.**\n\nWe found that DriveDPO sometimes adopts more conservative driving strategies, which may reduce forward efficiency in specific scenarios. As shown in the table below, we computed the Ego Progress metric for non-collision cases and observed a decrease after DPO fine-tuning. This suggests that the policy makes a more conservative trade-off between safety and efficiency, which could result in suboptimal performance in interactions that require faster speeds. We will include the failure case analysis in the revised version.\n\n| Method| Ego Progress in Succeeded Case |\n| --------------- | ------------------------------ |\n| Baseline| 80.73|\n| w/ DPO Finetune | 80.19|\n\n**[W5] No Ablation on DPO Parameters.**\n\nThe ablation results are provided in Tables 1–3 of the supplementary material. We conducted systematic ablation studies and analysis on key DPO parameters, including the number of candidate trajectories $K$, the number of DPO fine-tuning epochs, and the starting epoch of DPO.\n\n**[W6] DriveDPO is not directly compared with methods using reward modeling + PPO (e.g., GenDrive, TrajHF) on the same benchmark.**\n\nWe thank you for the suggestion. However, GenDrive is not an end-to-end driving method and thus cannot be directly compared on the same benchmark. In addition, TrajHF relies on a private preference dataset, making it difficult to reproduce and compare fairly. To enable a fair comparison, we implemented a variant using reward modeling and PPO (see table below). While this approach shows improvements over the baseline, it still underperforms compared to our method. This may be due to the instability of PPO training, whereas our DriveDPO leverages high-quality preferences with more stable safety alignment. We also noticed a concurrent work, ReCogDrive [2], which adopts GRPO and is evaluated on the NAVSIM benchmark. Despite not using VLM as ReCogDrive, our proposed DriveDPO still outperforms ReCogDrive, indicating the advantages of our method.\n\n| Method| NC | DAC| EP| TTC|Comf| PDMS |\n| ----------------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| Baseline| 97.9 | 97.3 | 84.0 | 93.6 | 100.0 | 88.8 |\n| Reward modeling and PPO | 97.9 | 98.1 | 84.1 | 94.0 | 100.0 | 89.4 |\n| ReCogDrive (GRPO) | 98.2 | 97.8 | 83.5 | 95.2 | 99.8 | 89.6 |\n| Ours| 98.5 | 98.1 | 84.3 | 94.8 | 99.9  | 90.0 |\n\n\n**[Q1] Are the constructed preferences always meaningful and safe?**\n\nWe analyzed the safety and human-likeness of the constructed preference pairs. The table below reports the average safety score and Human Trajectory Distance for all sampled anchor trajectories, as well as for the chosen and rejected ones. We observe that the chosen trajectories have average safety scores close to 100, whereas the rejected ones have scores near 0. Both are also close to the human trajectory in terms of distance. This indicates that the constructed preference pairs effectively distinguish between *human-like but unsafe* and *human-like and safe* behaviors, demonstrating their reliability and validity.\n\n| Trajectory Set| Safety Score | Human Trajectory Distance |\n| ------------------------------- | ------------ | ------------------------- |\n| All sampled anchor trajectories | 26.92| 24.87|\n| Chosen trajectories| 93.34| 0.77|\n| Rejected trajectories| 8.68| 1.99|\n\n**[L1] Statistical Robustness**\n\nWe conducted five independent training runs of our model and report the mean and standard deviation of key metrics in the table below. The results show low variance across runs, indicating that our method exhibits strong statistical robustness.\n\n| Method   | NC| DAC          | EP           | TTC          | Comf         | PDMS         |\n| -------- | ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Baseline | 97.9         | 97.3         | 84.0         | 93.6         | 100.0        | 88.8         |\n| Ours     | 98.27 ± 0.17 | 98.13 ± 0.05 | 84.17 ± 0.34 | 94.43 ± 0.26 | 99.97 ± 0.05 | 89.77 ± 0.17 |\n\n**[L2] Fixed Perception Backbone**\n\nAs shown in the table below, we replaced the vision backbone of the policy model with more powerful architectures, including V2-99 [3] and ViT-L. The results show that our method consistently outperforms previous SOTA methods. Moreover, applying DPO fine-tuning further improves performance across all backbone configurations, demonstrating the generality and scalability of our approach. We will include these experimental results and analysis in the revised version.\n\n| Method| Backbone| NC| DAC  | EP| TTC | Comf | PDMS |\n| -------------- | ------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| GoalFlow| V2-99| 98.4 | 98.3 | 85.0 | 94.6 | 100.0 | 90.3 |\n| Hydra-MDP-B| V2-99| 98.4 | 97.8 | 86.5 | 93.9 | 100.0 | 90.3 |\n| Hydra-MDP-C| V2-99 & ViT-L | 98.7 | 98.2 | 86.5 | 95.0 | 100.0 | 91.0 |\n| Hydra-MDP++| V2-99| 98.6 | 98.6 | 85.7 | 95.1 | 100.0 | 91.0 |\n| Ours (w/o DPO) | V2-99| 98.6 | 98.7 | 84.7 | 95.0 | 99.9  | 90.5 |\n| Ours| V2-99| 98.9 | 99.1 | 85.2 | 95.9 | 100.0 | 91.4 |\n\n| Method|Backbone|NC| DAC| EP| TTC| Comf  | PDMS |\n| -------------- | -------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| Hydra-MDP-A| ViT-L| 98.4 | 97.7 | 85.0 | 94.5 | 100.0 | 89.9 |\n| Ours (w/o DPO) | ViT-L| 98.2 | 98.1 | 84.1 | 94.0 | 100.0 | 89.6 |\n| Ours| ViT-L| 98.5 | 99.0 | 84.0 | 95.0 | 100.0 | 90.3 |\n\n[1] Jia, Xiaosong, et al. \"Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving.\" NeurIPS 2024.\n\n[2] Li, Yongkang, et al. \"ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving.\" arXiv:2506.08052.\n\n[3] Lee, Youngwan, and Jongyoul Park. \"Centermask: Real-time anchor-free instance segmentation.\" CVPR 2020."}, {"review_id": "w5W6RJgwZC", "reviewer": "Reviewer_o9SK", "summary": "This paper proposes to fine-tune trajectory scoring based end-to-end driving models with direct preference optimization by selecting positive and negative trajectories. The goal is to filter out trajectories that are close to a human reference trajectory in euclidean distance but unsafe.\nAs positive trajectory the highest scoring trajectory is used. The negative trajectory either uses a trajectory with low distance to ground truth or low distance to the best trajectory, that still has low PDMS.\nDPO is used to finetune a basemodel which predicts a distribution over anchor trajectories.\nThe method is evaluated and ablated on the NavSim benchmark.", "questions": "What is the inference speed of the method, and how long did training with the 6 L20 GPUs take?\nThe introduction claims that indirect optimization leads to suboptimal driving performance. It is not clear to me why this should be the case.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper follows the standard formula of proposing a method, comparing it to baselines, and ablating its components.\nIt expands on the recent trend in end-to-end driving of optimizing the target metric (PDMS) via scoring a set of anchor trajectories. The paper proposes to use the DPO method from the RLHF community for this optimization. A novel way to first learn a distribution over trajectories and then select positive and negative samples for DPO fine-tuning is proposed. Since this is an active research direction, the work should be interesting for many researchers in the field.\n\nWeaknesses:\n\nThe abstract claims that DriveDPO achieves a state-of-the-art PDMS of 90.0. This is incorrect because the prior SOTA is Hydra-MDP-C [3] with 91.0 PDMS.\n\nFigure 1 a) cites Uni-AD and VAD. Both of these papers do not use Anchor Vocabularies as depicted in the figure.\nThis idea was developed by VADv2 (which is also cited). It would make more sense to only cite VADv2 here. \n\nNAVSIM [1] should be cited when discussing the downside of the L2 learning objective, as it has also made these points.\n\nL. 53: This sentence could be formulated clearer such that it is obvious to the reader that only methods using ResNet-34 as backbone are outperformed (and not other methods using bigger vision backbones).\n\nClaim (1) that this paper is the first to point out the problem of imitation learning  (L1/L2 loss) seems to be not correct. For example, this problem was already discussed in NAVSIM Figure 1 [1].\n\nL70: The introduction to end-to-end autonomous driving is somewhat inaccurate. It is claimed that these methods predict trajectories from raw sensor data. Predicting trajectories is recently quite popular in the end-to-end driving literature but does not characterize these techniques. Many methods [2, 4, 5, 6, 7, 8, 9, 10, 11, 12] also directly predict the control of the car, in particular earlier works from before 2021. \n\nSection 2.2 is missing earlier work on fine-tuning / training end-to-end architectures with reinforcement learning, such as [9, 10, 11, 12].\n\nL. 220 / 221: The paper incorrectly states that WOTE is the SOTA trajectory scoring method with 88.0 PDMS. This is incorrect because Hydra-MDP-C (CVPR 2024 Workshop) [3] is the prior work SOTA with 91.0 PDMS.\nThe second claim here is more nuanced. DiffusionDrive is claimed to be the SOTA imitation learning method.\nDiffusionDrive still technically scores trajectories, although with probabilities and not predicted metrics.\nSo there is an argument to be made that it can be categorized differently than methods like Hydra that score based on metrics, but the wording needs to be precise.\n\nL. 225: See above; this claim is not entirely correct.\n\nTable 1: Several methods are misrepresented in the Table 1 comparison:\n\n    • Hydra-MDP: The performance of the random ablation Hydra-MDP-V8192-W-EP is quoted as Hydra-MDP (86.5 PDMS). Instead, the best Hydra-MDP version, Hydra-MDP-C (91 PDMS), should be cited.\n    • WoTE: WoTE is quoted as achieving 88.0 PDMS, but the paper reports 87.1 PDMS.\n    • Hydra-MDP++: The best version of Hydra-MDP++ achieves 91.0 PDMS, but the weaker result with a smaller backbone and 86.6 PDMS is cited.\n    • GoalFlow: GoalFlow is quoted to achieve 85.7 PDMS (perhaps the random ablation Dim 256 Backbone resnet34 is referenced here), but the reported best performance of the method is 90.3 PDMS.\n\nIf the authors want to compare to methods using the same vision backbone (ResNet34) they can make the vision backbone a column and compare within the group (but also report the best performance of each method in another group / table).\n\n\nSo to summarize, I think the paper proposes an interesting idea that is of value to the community.\nHowever, I am concerned that the experimental evaluation may mislead the reader by overselling the achieved performance, via misrepresentation of baselines. I do not think it is necessary to make a state-of-the-art claim for publication, in particular because the performance difference is small, but I do think it is important to properly represent baselines and make correct and precise claims.\nI think the paper should be improved by making the claims more specific (e.g., outperforming methods that use the ResNet34 backbone), properly reporting the best performance of baselines, and more rigorously citing prior work.\nI would consider raising my score if these concerns are addressed.\n\nIf the authors want to make a claim around advancing the state-of-the-art on NavSim I suggest reporting the performance of the method with competitive vision backbones like VIT-L or V2-99 instead of / in addition to ResNet34.\n\nReferences:\n\n[1] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, Kashyap Chitta: “NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking”, NeurIPS 2024\n\n[2] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, Yu Qiao: “Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline”, NeurIPS 2022\n\n[3] Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, Yu-Gang Jiang, Jose M. Alvarez: “Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation”,  CVPR 2024 Workshop\n\n[4] Dean A. Pomerleau: “ALVINN: AN AUTONOMOUS LAND VEHICLE IN A NEURAL NETWORK”, NeurIPS 1988\n\n[5] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, Karol Zieba: “End to End Learning for Self-Driving Cars”,  ArXiv 2016\n\n[6] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, Alexey Dosovitskiy: “End-to-end Driving via Conditional Imitation Learning” ICRA 2018\n\n[7] Felipe Codevilla, Eder Santana, Antonio M. López, Adrien Gaidon: “Exploring the Limitations of Behavior Cloning for Autonomous Driving”, ICCV 2019\n\n[8]  Dian Chen, Brady Zhou, Vladlen Koltun, Philipp Krähenbühl: “Learning by Cheating.”, CoRL 2019\n\n[9] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, Amar Shah: “Learning to Drive in a Day.”, ICRA 2019\n\n[10] Marin Toromanoff, Emilie Wirbel, Fabien Moutarde: “End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances”, CVPR 2020\n\n[11] Xiaodan Liang, Tairui Wang, Luona Yang, Eric Xing: “CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving”, 2018\n\n[12] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, Fabien Moutarde: “GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving”, Robotics 2023", "was_revised": true, "final_justification": "My concerns have been adressed. I have updated my rating to accept.", "rebuttal": "Thank you for your detailed and constructive reviews. We are glad that you found that “the paper proposes an interesting idea that is of value to the community” and “the work should be interesting for many researchers in the field”. We would like to address the Weaknesses (**W**) and Questions (**Q**) below.\n\n**[W1] Reporting the performance of the method with competitive vision backbones in addition to ResNet34.** \n\nWe sincerely thank you for pointing out the issues regarding accurate citation of baseline performance and fair comparisons. We conducted additional experiments using stronger vision backbones V2-99 and ViT-L (see table below). Our method still outperforms existing SOTA approaches, including Hydra-MDP-A [1], Hydra-MDP-C [1], and Hydra-MDP++ [2]. In the revised version, we will revise our claims to clarify that DriveDPO achieves state-of-the-art performance when utilizing the same vision backbone, and we will explicitly categorize methods by backbone to ensure a fair comparison. We will also correct and include proper citations, as well as the best-reported performance of Hydra-MDP-C [1], Hydra-MDP++ [2], and GoalFlow [3].\n\n| Method         | Backbone      | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| -------------- | ------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| GoalFlow       | V2-99         | 98.4 | 98.3 | 85.0 | 94.6 | 100.0 | 90.3 |\n| Hydra-MDP-B    | V2-99         | 98.4 | 97.8 | 86.5 | 93.9 | 100.0 | 90.3 |\n| Hydra-MDP-C    | V2-99 & ViT-L | 98.7 | 98.2 | 86.5 | 95.0 | 100.0 | 91.0 |\n| Hydra-MDP++    | V2-99         | 98.6 | 98.6 | 85.7 | 95.1 | 100.0 | 91.0 |\n| Ours (w/o DPO) | V2-99         | 98.6 | 98.7 | 84.7 | 95.0 | 99.9  | 90.5 |\n| Ours           | V2-99         | 98.9 | 99.1 | 85.2 | 95.9 | 100.0 | 91.4 |\n\n| Method         | Backbone | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| -------------- | -------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| Hydra-MDP-A    | ViT-L    | 98.4 | 97.7 | 85.0 | 94.5 | 100.0 | 89.9 |\n| Ours (w/o DPO) | ViT-L    | 98.2 | 98.1 | 84.1 | 94.0 | 100.0 | 89.6 |\n| Ours           | ViT-L    | 98.5 | 99.0 | 84.0 | 95.0 | 100.0 | 90.3 |\n\n\n\n**[W2] Figure 1 a) cites Uni-AD and VAD. It would make more sense to only cite VADv2 here.**\n\nThank you for pointing this out. We will revise the caption to cite only VADv2, reflecting the correct origin of the idea.\n\n\n\n**[W3] NAVSIM should be cited when discussing the downside of the L2 learning objective. Claim that this paper is the first to point out the problem of imitation learning (L1/L2 loss) seems to be not correct. This problem was already discussed in NAVSIM Figure 1.**\n\nWe would like to clarify that NAVSIM [4] discusses the limitations of L1/L2-based evaluation metrics such as ADE, but does not discuss the issue of using L1/L2 objectives as training losses. That said, we acknowledge that the insight raised in NAVSIM aligns with the motivation of our work. Our contribution lies in being the first to extend this perspective to the training objective and demonstrate clear empirical benefits from doing so. We will include a citation and discussion of NAVSIM in the revised version.\n\n\n\n**[W4] L53: This sentence could be formulated clearer such that it is obvious to the reader that only methods using ResNet-34 as backbone are outperformed.**\n\nThank you for catching this ambiguity. We will revise the sentence to: “We outperform all prior methods that use ResNet-34 as the vision backbone.” \n\n\n\n**[W5] L70: The introduction to end-to-end autonomous driving is somewhat inaccurate. Many methods also directly predict the control of the car, in particular earlier works from before 2021.**\n\nWe appreciate this correction. We will revise this to: “End-to-end autonomous driving typically maps raw sensor inputs to driving actions, either in the form of trajectories or low-level control commands [5–14].”\n\n\n\n**[W6] Section 2.2 is missing earlier work on fine-tuning / training end-to-end architectures with reinforcement learning.**\n\nIn the revised version, we will add a paragraph in Section 2.2 to briefly review the earlier work on training end-to-end architectures with reinforcement learning: “Kendall et al. [11] demonstrated an on-vehicle deep RL system for lane following using monocular input and distance-based reward. Toromanoff et al. [12] introduced implicit affordances to enable model-free RL in urban settings with traffic light and obstacle handling. Liang et al. [13] proposed CIRL, combining goal-conditioned RL and human demonstration to improve success rates in CARLA. Chekroun et al. [14] developed General Reinforced Imitation, which integrates expert data into off-policy RL for stable vision-based urban driving.”\n\n\n\n**[W7] L. 220 / 221: The paper incorrectly states that WOTE is the SOTA trajectory scoring method with 88.0 PDMS. This is incorrect because Hydra-MDP-C is the prior work SOTA with 91.0 PDMS. DiffusionDrive is claimed to be the SOTA imitation learning method.** \n\nWe appreciate your pointing out the inaccuracies regarding prior work. To avoid confusion, we will restrict the comparison scope to methods using ResNet-34 and revise the original description to: “ Using ResNet-34 as the visual backbone, our method with only unified policy distillation achieves a PDMS of 88.8, outperforming the diffusion-based method DiffusionDrive, which scores trajectories through imitation probabilities, and the metric-based scoring method WOTE. ” \n\n\n\n**[W8] L. 225: See above; this claim is not entirely correct.**\n\nWe will clarify that our method achieves a new SOTA among methods using the ResNet-34 backbone by modifying the sentence to: “Our method ultimately achieves a PDMS of 90.0, establishing a new state-of-the-art among methods using ResNet-34 as the visual backbone.” As discussed in **W1**, we will further include comparisons and analysis with SOTA methods like Hydra-MDP-C using V2-99 and ViT-L as backbones in the revised version.\n\n\n\n**[W9] Table 1: Several methods are misrepresented in the Table 1 comparison**\n\nTo ensure fair comparison, we will group methods using the same visual backbone in Table 1 and clearly distinguish between the performance of methods based on different backbone architectures. We will include the best reported performances and citations for Hydra-MDP-C [1], Hydra-MDP++ [2], and GoalFlow [3] in the revised version. For WoTE [15], we will update its performance to 88.3 as reported in its most recent arXiv version.\n\n\n\n**[Q1] What is the inference speed of the method, and how long did training with the 6 L20 GPUs take?** \n\nThe table below presents the inference latency of different methods (some results are cited from Hydra-MDP++ [2]). Our method achieves faster inference speed under both ResNet-34 and V2-99 backbones. Additionally, training on 6 × L20 GPUs takes approximately 10 minutes per epoch, totaling around 7 hours for 40 epochs.\n\n| Method      | Backbone | Test GPU    | Latency (ms) |\n| ----------- | -------- | ----------- | ------------ |\n| Transfuser  | Resnet34 | NVIDIA V100 | 221.2        |\n| UniAD       | Resnet34 | NVIDIA A100 | 555.6        |\n| Hydra-MDP++ | Resnet34 | NVIDIA V100 | 206.2        |\n| Ours        | Resnet34 | NVIDIA L20  | 137.8        |\n| Hydra-MDP++ | V2-99    | NVIDIA V100 | 271.0        |\n| Ours        | V2-99    | NVIDIA L20  | 240.8        |\n\n\n\n**[Q2] The introduction claims that indirect optimization leads to suboptimal driving performance. It is not clear to me why this should be the case.**\n\nPrevious score-based methods regress multiple scores for each anchor using several metric heads in a multi-task learning manner. However, the objectives of different metrics may conflict with each other, leading to inconsistent or even conflicting gradient directions, which can adversely affect overall performance.\n\n\n\n[1] Li, Zhenxin, et al. \"Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation.\" arXiv:2406.06978.\n\n[2] Li, Kailin, et al. \"Hydra-mdp++: Advancing end-to-end driving via expert-guided hydra-distillation.\" arXiv:2503.12820.\n\n[3] Xing, Zebin, et al. \"Goalflow: Goal-driven flow matching for multimodal trajectories generation in end-to-end autonomous driving.\" CVPR 2025.\n\n[4] Dauner, Daniel, et al. \"Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking.\" NeurIPS 2024.\n\n[5] Penghao Wu, et al. \"Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline\" NeurIPS 2022.\n\n[6] Dean A. Pomerleau, et al. “ALVINN: AN AUTONOMOUS LAND VEHICLE IN A NEURAL NETWORK” NeurIPS 1988.\n\n[7] Bojarski, Mariusz, et al. \"End to end learning for self-driving cars.\" arXiv:1604.07316.\n\n[8] Felipe Codevilla, et al. \"End-to-end Driving via Conditional Imitation Learning\" ICRA 2018\n\n[9] Felipe Codevilla, et al. \"Exploring the Limitations of Behavior Cloning for Autonomous Driving\" ICCV 2019\n\n[10] Dian Chen, et al. \"Learning by Cheating.\" CoRL 2019\n\n[11] Alex Kendall, et al. \"Learning to Drive in a Day.\" ICRA 2019\n\n[12] Marin Toromanoff, et al. \"End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances\" CVPR 2020\n\n[13] Xiaodan Liang, et al. \"CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving\" 2018\n\n[14] Raphael Chekroun, et al. \"GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving\" Robotics 2023\n\n[15] Li, Yingyan, et al. \"End-to-end driving with online trajectory evaluation via bev world model.\" arXiv:2504.01941."}, {"review_id": "gvDRFIt4gi", "reviewer": "Reviewer_ndFx", "summary": "The paper introduces a new end-to-end autonomous driving method called DriveDPO. The method has two improvements. The first one is to regress score over all anchors with both human trajectory and rule supervision. The second one is using imitation-based DPO to select safety-aware trajectory pairs for perference learning.", "questions": "1. How does DriveDPO perform in highly interactive closed-loop environments with dynamic agents?\n2. Could the rejection sample selection process be learned instead of relying on heuristics?\n3. Why do more candidate trajectories or DPO fine-tuning epochs lead to poorer performance?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. This paper has clear motivation.\n2. The paper is well-writen and easy to follow.\n3. The ablation is extensive, clearly showing the contribution of each part.\n\nWeakness:\n1. The method is only evaluated on NAVSIM. The closed-loop performance is not explored. \n2. The novelty is limited. The unified policy distillation method to combine the rule-based score and imitation score by weighted sum is trival. The ablation does not compare with the seperate scoring method. \n3. The paper only use a one backbone model. Trying more backbone model will be better to demonstrate generality of its method.", "was_revised": true, "final_justification": "My concern about its closed-loop performance is addressed. But I still think the improvement of DPO is limited and the idea to add rule-based score with imitation-learning score is trival.", "rebuttal": "Thank you for your detailed and constructive reviews. We are glad that you found our paper “has clear motivation” and “is well-writen and easy to follow”. We would like to address the Weaknesses (**W**) and Questions (**Q**) below.\n\n**[W1, Q1] The method is only evaluated on NAVSIM. The closed-loop performance is not explored. How does DriveDPO perform in highly interactive closed-loop environments with dynamic agents?**\n\nThank you for pointing this out. We conducted closed-loop evaluation experiments on the Bench2Drive dataset [1] (see the table below). Our method outperforms representative baselines in key metrics such as Driving Score, Success Rate, and Mean Multi-Ability Success Rate, demonstrating its effectiveness in closed-loop settings. Notably, the Multi-Ability evaluation directly reflects performance in various highly interactive scenarios. In the particularly challenging Emergency Brake test, DriveDPO surpasses all baseline methods, indicating a strong ability to react to potential risk behaviors. DriveDPO also performs well in interactive tasks such as Overtaking and Give Way, further demonstrating its capability to generate strategies in complex environments with dynamic agents. We will include these closed-loop evaluation results and analysis in the revised version.\n\n| Method | Efficiency | Comfortness | Success Rate(%) | Driving Score |\n| ------ | ---------- | ----------- | --------------- | ------------- |\n| AD-MLP | 48.45      | 22.63       | 0.00            | 18.05         |\n| UniAD  | 129.21     | 43.58       | 16.36           | 45.81         |\n| VAD    | 157.94     | 46.01       | 15.00           | 42.35         |\n| TCP    | 76.54      | 18.08       | 30.00           | 59.90         |\n| Ours   | 166.80     | 26.79       | 30.62           | 62.02         |\n\n| Method | Merging | Overtaking | Emergency Brake | Give Way | Traffic Sign | Mean  |\n| ------ | ------- | ---------- | --------------- | -------- | ------------ | ----- |\n| AD-MLP | 0.00    | 0.00       | 0.00            | 0.00     | 4.35         | 0.87  |\n| UniAD  | 14.10   | 17.78      | 21.67           | 10.00    | 14.21        | 15.55 |\n| VAD    | 8.11    | 24.44      | 18.64           | 20.00    | 19.15        | 18.07 |\n| TCP    | 8.89    | 24.29      | 51.67           | 40.00    | 46.28        | 34.22 |\n| Ours   | 16.28   | 28.95      | 53.06           | 30.00    | 45.00        | 34.66 |\n\n\n\n**[W2.1] The novelty is limited. The unified policy distillation method to combine the rule-based score and imitation score by weighted sum is trival.**\n\nWe would like to respectfully clarify that introducing Safety DPO into end-to-end autonomous driving is a non-trivial contribution. First, while prior methods independently predict multiple scores for each anchor to derive a policy implicitly, our approach unifies human-likeness and rule-based safety into a single policy distribution, directly and explicitly supervising policy learning. In addition, to address the human-like but unsafe issue in imitation learning, we propose the Imitation-Based Selection strategy to construct preference pairs that better focus on effective safety preference learning. The effectiveness of our method is consistently demonstrated across both the NAVSIM and Bench2Drive datasets (as in **W1**), as well as under different vision backbones (as in **W3**).\n\n\n\n**[W2.2] The ablation does not compare with the seperate scoring method.**\n\nWe have included an ablation study in Table 2 of the main paper (see ID-1 and ID-2), which compares our Unified Policy Distillation with the separate scoring method. The results show that the unified approach significantly outperforms the separate one.\n\n\n**[W3] The paper only use a one backbone model. Trying more backbone model will be better to demonstrate generality of its method.**\n\nThank you for pointing this out. As shown in the table below, we replaced the vision backbone with more powerful architectures, including V2-99 [2] and ViT-L, and conducted experiments accordingly. The results demonstrate that our method consistently outperforms previous SOTA methods. Moreover, applying DPO fine-tuning further improves performance across all backbone configurations, proving the generality and scalability of our approach. We will include these experimental results and analysis in the revised version.\n\n| Method         | Backbone      | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| -------------- | ------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| GoalFlow       | V2-99         | 98.4 | 98.3 | 85.0 | 94.6 | 100.0 | 90.3 |\n| Hydra-MDP-B    | V2-99         | 98.4 | 97.8 | 86.5 | 93.9 | 100.0 | 90.3 |\n| Hydra-MDP-C    | V2-99 & ViT-L | 98.7 | 98.2 | 86.5 | 95.0 | 100.0 | 91.0 |\n| Hydra-MDP++    | V2-99         | 98.6 | 98.6 | 85.7 | 95.1 | 100.0 | 91.0 |\n| Ours (w/o DPO) | V2-99         | 98.6 | 98.7 | 84.7 | 95.0 | 99.9  | 90.5 |\n| Ours           | V2-99         | 98.9 | 99.1 | 85.2 | 95.9 | 100.0 | 91.4 |\n\n| Method         | Backbone | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| -------------- | -------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| Hydra-MDP-A    | ViT-L    | 98.4 | 97.7 | 85.0 | 94.5 | 100.0 | 89.9 |\n| Ours (w/o DPO) | ViT-L    | 98.2 | 98.1 | 84.1 | 94.0 | 100.0 | 89.6 |\n| Ours           | ViT-L    | 98.5 | 99.0 | 84.0 | 95.0 | 100.0 | 90.3 |\n\n\n\n**[Q2] Could the rejection sample selection process be learned instead of relying on heuristics?**\n\nThank you for the insightful suggestion. We trained a reward model distinguish between *human-like and safe* trajectories versus *human-like but unsafe* ones, and used it for DPO training (see the table below). Compared to relying on rule-based heuristics, using a learned reward model indeed leads to improved performance. This is likely because it can explicitly learn to recognize *human-like but risky* behaviors in ambiguous or complex scenarios that are difficult to capture with hand-crafted rules, thereby improving generalization. We will include this analysis in the revised version. Furthermore, we believe that with access to large-scale, high-quality human preference data in driving scenarios, the learned reward model could be further improved, which is a promising direction for our future work.\n\n| Method                | NC   | DAC  | EP   | TTC  | Comf  | PDMS |\n| --------------------- | ---- | ---- | ---- | ---- | ----- | ---- |\n| w/o DPO               | 97.9 | 97.3 | 84.0 | 93.6 | 100.0 | 88.8 |\n| Rule-based heuristics | 98.4 | 97.5 | 83.5 | 94.6 | 100.0 | 89.3 |\n| Learned reward model  | 98.2 | 98.2 | 83.7 | 94.3 | 100.0 | 89.6 |\n\n\n\n**[Q3] Why do more candidate trajectories or DPO fine-tuning epochs lead to poorer performance?**\n\nWhen the number of candidate trajectories K is overly increased, the selected rejected samples often have very low probabilities under the current policy distribution. As a result, they may contribute little to the policy optimization, ultimately degrading performance. As for the drop in performance after 20 epochs of DPO fine-tuning, this is potentially due to overfitting on the training set. We empirically set the number of candidate trajectories to 1024 and fine-tuned the policy with DPO for 10 epochs, which yielded the best performance in our experiments.\n\n\n\n[1] Jia, Xiaosong, et al. \"Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving.\" NeurIPS 2024.\n\n[2] Lee, Youngwan, and Jongyoul Park. \"Centermask: Real-time anchor-free instance segmentation.\" CVPR 2020."}]

Author rebuttal:

Meta-review: This paper proposes a two-stage policy learning framework called DriveDPO for end-to-end autonomous driving. The proposed framework uses both imitation learning and rule-based safety signals and refines the policy using DPO. The overall framework achieves good performance on the NAVSIM benchmark, with noticeable improvement on safety-related metrics.

The paper received generally positive feedback. The reviewers acknowledged its logical design and the performance of DriveDPO on the NAVSIM benchmark. There are also some concerns regarding this paper, specifically
- The novelty of the paper seems limited. It is built upon the Transfuser backbone, anchor vocabulary from VADv2, and the use of DPO and and rule-based score with imitation learning score is also trivial.
- The performance improvement does not seem to be very impressive. The proposed method uses additional rule-based information for policy learning, while many other baselines do not use such extra information, making the comparison not that fair for other baselines. Its performance suffers from a noticeable drop when removing rule-based signals (Table 2). Moreover, the improvement from adding DPO also seems pretty marginal (Table 1).

I encourage the authors to include the closed-loop evaluation results as well as some of the discussions during rebuttal in the final version of the paper.
Decision: Accept (poster)