Title: DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving
Venue: NeurIPS 2025
Accepted: yes
Keywords: End-to-End Autonomous Driving

Direct Preference Optimization

Imitation Learning
Abstract: End-to-end autonomous driving has substantially progressed by directly predicting future trajectories from raw perception inputs, which bypasses traditional modular pipelines. However, mainstream methods trained via imitation learning suffer from critical safety limitations, as they fail to distinguish between trajectories that appear human-like but are potentially unsafe. Some recent approaches attempt to address this by regressing multiple rule-driven scores but decoupling supervision from policy optimization, resulting in suboptimal performance. To tackle these challenges, we propose DriveDPO, a Safety Direct Preference Optimization Policy Learning framework. First, we distill a unified policy distribution from human imitation similarity and rule-based safety scores for direct policy optimization. Further, we introduce an iterative Direct Preference Optimization stage formulated as trajectory-level preference alignment. Extensive experiments on the NAVSIM benchmark demonstrate that DriveDPO achieves a new state-of-the-art PDMS of 90.0. Furthermore, qualitative results across diverse challenging scenarios highlight DriveDPO’s ability to produce safer and more reliable driving behaviors.


Peer reviews:
comments: SUMMARY:
The paper introduces a new end-to-end autonomous driving method called DriveDPO. The method has two improvements. The first one is to regress score over all anchors with both human trajectory and rule supervision. The second one is using imitation-based DPO to select safety-aware trajectory pairs for perference learning.

STRENGTHS AND WEAKNESSES:
Strengths:

1. This paper has clear motivation.
2. The paper is well-writen and easy to follow.
3. The ablation is extensive, clearly showing the contribution of each part.

Weakness:
1. The method is only evaluated on NAVSIM. The closed-loop performance is not explored. 
2. The novelty is limited. The unified policy distillation method to combine the rule-based score and imitation score by weighted sum is trival. The ablation does not compare with the seperate scoring method. 
3. The paper only use a one backbone model. Trying more backbone model will be better to demonstrate generality of its method.

QUESTIONS:
1. How does DriveDPO perform in highly interactive closed-loop environments with dynamic agents?
2. Could the rejection sample selection process be learned instead of relying on heuristics?
3. Why do more candidate trajectories or DPO fine-tuning epochs lead to poorer performance?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to fine-tune trajectory scoring based end-to-end driving models with direct preference optimization by selecting positive and negative trajectories. The goal is to filter out trajectories that are close to a human reference trajectory in euclidean distance but unsafe.\nAs positive trajectory the highest scoring trajectory is used. The negative trajectory either uses a trajectory with low distance to ground truth or low distance to the best trajectory, that still has low PDMS.\nDPO is used to finetune a basemodel which predicts a distribution over anchor trajectories.\nThe method is evaluated and ablated on the NavSim benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper follows the standard formula of proposing a method, comparing it to baselines, and ablating its components.\nIt expands on the recent trend in end-to-end driving of optimizing the target metric (PDMS) via scoring a set of anchor trajectories. The paper proposes to use the DPO method from the RLHF community for this optimization. A novel way to first learn a distribution over trajectories and then select positive and negative samples for DPO fine-tuning is proposed. Since this is an active research direction, the work should be interesting for many researchers in the field.\n\nWeaknesses:\n\nThe abstract claims that DriveDPO achieves a state-of-the-art PDMS of 90.0. This is incorrect because the prior SOTA is Hydra-MDP-C [3] with 91.0 PDMS.\n\nFigure 1 a) cites Uni-AD and VAD. Both of these papers do not use Anchor Vocabularies as depicted in the figure.\nThis idea was developed by VADv2 (which is also cited). It would make more sense to only cite VADv2 here. \n\nNAVSIM [1] should be cited when discussing the downside of the L2 learning objective, as it has also made these points.\n\nL. 53: This sentence could be formulated clearer such that it is obvious to the reader that only methods using ResNet-34 as backbone are outperformed (and not other methods using bigger vision backbones).\n\nClaim (1) that this paper is the first to point out the problem of imitation learning  (L1/L2 loss) seems to be not correct. For example, this problem was already discussed in NAVSIM Figure 1 [1].\n\nL70: The introduction to end-to-end autonomous driving is somewhat inaccurate. It is claimed that these methods predict trajectories from raw sensor data. Predicting trajectories is recently quite popular in the end-to-end driving literature but does not characterize these techniques. Many methods [2, 4, 5, 6, 7, 8, 9, 10, 11, 12] also directly predict the control of the car, in particular earlier works from before 2021. \n\nSection 2.2 is missing earlier work on fine-tuning / training end-to-end architectures with reinforcement learning, such as [9, 10, 11, 12].\n\nL. 220 / 221: The paper incorrectly states that WOTE is the SOTA trajectory scoring method with 88.0 PDMS. This is incorrect because Hydra-MDP-C (CVPR 2024 Workshop) [3] is the prior work SOTA with 91.0 PDMS.\nThe second claim here is more nuanced. DiffusionDrive is claimed to be the SOTA imitation learning method.\nDiffusionDrive still technically scores trajectories, although with probabilities and not predicted metrics.\nSo there is an argument to be made that it can be categorized differently than methods like Hydra that score based on metrics, but the wording needs to be precise.\n\nL. 225: See above; this claim is not entirely correct.\n\nTable 1: Several methods are misrepresented in the Table 1 comparison:\n\n    • Hydra-MDP: The performance of the random ablation Hydra-MDP-V8192-W-EP is quoted as Hydra-MDP (86.5 PDMS). Instead, the best Hydra-MDP version, Hydra-MDP-C (91 PDMS), should be cited.\n    • WoTE: WoTE is quoted as achieving 88.0 PDMS, but the paper reports 87.1 PDMS.\n    • Hydra-MDP++: The best version of Hydra-MDP++ achieves 91.0 PDMS, but the weaker result with a smaller backbone and 86.6 PDMS is cited.\n    • GoalFlow: GoalFlow is quoted to achieve 85.7 PDMS (perhaps the random ablation Dim 256 Backbone resnet34 is referenced here), but the reported best performance of the method is 90.3 PDMS.\n\nIf the authors want to compare to methods using the same vision backbone (ResNet34) they can make the vision backbone a column and compare within the group (but also report the best performance of each method in another group / table).\n\n\nSo to summarize, I think the paper proposes an interesting idea that is of value to the community.\nHowever, I am concerned that the experimental evaluation may mislead the reader by overselling the achieved performance, via misrepresentation of baselines. I do not think it is necessary to make a state-of-the-art claim for publication, in particular because the performance difference is small, but I do think it is important to properly represent baselines and make correct and precise claims.\nI think the paper should be improved by making the claims more specific (e.g., outperforming methods that use the ResNet34 backbone), properly reporting the best performance of baselines, and more rigorously citing prior work.\nI would consider raising my score if these concerns are addressed.\n\nIf the authors want to make a claim around advancing the state-of-the-art on NavSim I suggest reporting the performance of the method with competitive vision backbones like VIT-L or V2-99 instead of / in addition to ResNet34.\n\nReferences:\n\n[1] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, Kashyap Chitta: “NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking”, NeurIPS 2024\n\n[2] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, Yu Qiao: “Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline”, NeurIPS 2022\n\n[3] Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, Yu-Gang Jiang, Jose M. Alvarez: “Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation”,  CVPR 2024 Workshop\n\n[4] Dean A. Pomerleau: “ALVINN: AN AUTONOMOUS LAND VEHICLE IN A NEURAL NETWORK”, NeurIPS 1988\n\n[5] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, Karol Zieba: “End to End Learning for Self-Driving Cars”,  ArXiv 2016\n\n[6] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, Alexey Dosovitskiy: “End-to-end Driving via Conditional Imitation Learning” ICRA 2018\n\n[7] Felipe Codevilla, Eder Santana, Antonio M. López, Adrien Gaidon: “Exploring the Limitations of Behavior Cloning for Autonomous Driving”, ICCV 2019\n\n[8]  Dian Chen, Brady Zhou, Vladlen Koltun, Philipp Krähenbühl: “Learning by Cheating.”, CoRL 2019\n\n[9] Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, Amar Shah: “Learning to Drive in a Day.”, ICRA 2019\n\n[10] Marin Toromanoff, Emilie Wirbel, Fabien Moutarde: “End-to-End Model-Free Reinforcement Learning for Urban Driving using Implicit Affordances”, CVPR 2020\n\n[11] Xiaodan Liang, Tairui Wang, Luona Yang, Eric Xing: “CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving”, 2018\n\n[12] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, Fabien Moutarde: “GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving”, Robotics 2023\n\nQUESTIONS:\nWhat is the inference speed of the method, and how long did training with the 6 L20 GPUs take?\nThe introduction claims that indirect optimization leads to suboptimal driving performance. It is not clear to me why this should be the case.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDriveDPO introduces a two-stage policy learning framework for end-to-end autonomous driving by using both imitation and rule-based safety signals and refining the policy using Safety DPO. \n\nIt achieves state-of-the-art performance on the NAVSIM benchmark, significantly improving safety-related metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths of the Approach\n1. Safety-Aligned Policy Learning: Combines imitation learning and rule-based safety via a unified distribution\n2. Direct Optimization of Policy Distribution: Unlike score-based approaches, DriveDPO optimizes the actual policy. Introduces Safety DPO\n3. Achieves a new state-of-the-art.\n\n\nWeaknesses of the Approach\n1. Dependency on Rule-Based Scores\n    - Requires a high-fidelity simulator (NAVSIM) to generate rule-based metrics (PDMS), which may not generalize well to all driving environments or be available in real-world settings. So this might not translate to training in real-world.\n    - Preference pairs are constructed using rule thresholds and heuristics. This may not reflect the full nuance of real-world human preferences or corner cases.\n2. Anchor Vocabulary Limits Output Flexibility: The discretization of trajectory space into anchors, while efficient, may limit expressiveness or fail to capture rare but valid driving maneuvers.\n3. No Online Learning or Interaction: The system is trained offline and evaluated in simulation; there’s no feedback loop with deployment environments.\n\nWeaknesses / Gaps in the Work\n1. No Exploration of Failure Cases: The paper doesn't systematically analyze where DriveDPO fails (e.g., rare edge cases, adversarial driving).\n2. No Ablation on DPO Parameters\n3. While many RLHF methods are cited, DriveDPO is not directly compared with methods using reward modeling + PPO (e.g., GenDrive, TrajHF) on the same benchmark.\n4. Simulation-only Evaluation: The policy is not validated in a closed-loop interactive simulator setting\n\nQUESTIONS:\n- Are the constructed preferences always meaningful and safe? There could be some analysis on this.\n- Can DriveDPO function effectively without a high-fidelity simulator?\n\nLIMITATIONS:\n- Statistical Robustness (computational resources limitations pointed in the paper)\n- Fixed Perception Backbone: The policy model uses a fixed ResNet/Transfuser backbone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DriveDPO, a policy learning framework that brings the Direct Preference Optimization (DPO) method to end-to-end autonomous driving. To address the safety shortcomings of imitation learning (which can produce human-like but unsafe trajectories), the authors introduce a two-stage approach: (1) Unified Policy Distillation, which combines human imitation and rule-based safety scores to train an initial policy, and (2) Safety DPO, which fine-tunes this policy via pairwise preference learning on full trajectories. Experiments are conducted on the NAVSIM benchmark with a new metric (PDMS) and show gains over both imitation learning and rule-supervised baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Applying preference-based learning via DPO to autonomous driving is a logical and increasingly relevant extension of RLHF methods.\n2. The idea of adapting DPO to control tasks is practical, leveraging recent successes in language modeling for robotics/control.\n3. Performance on PDMS is solid, with gains over prior methods such as Hydra-MDP and WOTE, and qualitative results suggest better safety behavior.\n\nWeaknesses:\n1. The paper claims to distinguish between human-like but unsafe trajectories, yet does not provide empirical or qualitative evidence that the learned preferences actually capture safety-related behaviors. While the paper is motivated by this issue, the safety scoring function still heavily depends on rule-based metrics rather than explicitly learning to recognize subtle forms of human-like but risky behavior.\n2. DriveDPO is a relatively straightforward combination of known components—policy distillation with hand-crafted safety scores, and standard DPO. The technical contribution lies more in integration than innovation.\n\nQUESTIONS:\n1. How would the method perform if the rule-based safety scores (PDMS) were replaced by human preferences directly? Would a learned reward model generalize better?\n2. Are the selection strategies for preference pairs (especially imitation-based) robust across datasets? How sensitive are results to the threshold?\n3. Would the method handle out-of-distribution data, like in more complex or unseen scenarios?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}