$AutoDrive\text{-}P^3$: Unified Chain of Perception–Prediction–Planning Thought via Reinforcement Fine-Tuning
Authors: Yuqi Ye
Zijian Zhang
Junhong Lin
Shangkun Sun
Changhao Peng
Wei Gao
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous Driving; Vision-Language Models; Reinforcement Learning
Primary area: applications to robotics, autonomy, planning

Abstract: Vision-language models (VLMs) are increasingly being adopted for end-to-end autonomous driving systems due to their exceptional performance in handling long-tail scenarios. However, current VLM-based approaches suffer from two major limitations: 1) Some VLMs directly output planning results without chain-of-thought (CoT) reasoning, bypassing crucial perception and prediction stages which creates a significant domain gap and compromises decision-making capability; 2) Other VLMs can generate outputs for perception, prediction, and planning tasks but employ a fragmented decision-making approach where these modules operate seperately, leading to a significant lack of synergy that undermines true planning performance. To address these limitations, we propose ${AutoDrive\text{-}P^3}$, a novel framework that seamlessly integrates $\underline{\textbf{P}}$erception, $\underline{\textbf{P}}$rediction, and $\underline{\textbf{P}}$lanning through structured reasoning. We introduce the ${P^3\text{-}CoT}$ dataset to facilitate coherent reasoning and propose ${P^3\text{-}GRPO}$, a hierarchical reinforcement learning algorithm that provides progressive supervision across all three tasks. Specifically, ${AutoDrive\text{-}P^3}$ progressively generates CoT reasoning and answers for perception, prediction, and planning, where perception provides essential information for subsequent prediction and planning, while both perception and prediction collectively contribute to the final planning decisions, enabling safer and more interpretable autonomous driving. Additionally, to balance inference efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking. Extensive experiments on both open-loop (nuScenes) and closed-loop (NAVSIMv1/v2) benchmarks demonstrate that our approach achieves state-of-the-art performance in planning tasks. Code is available at https://github.com/haha-yuki-haha/AutoDrive-P3, https://openi.pcl.ac.cn/OpenAIDriving/AutoDrive-P3.

Published as a conference paper at ICLR 2026
UNIFIED CHAIN OF PERCIPTON-PPEDDCTOO--LAAIIN THOUGHT VIA REINFORCEMENT FINE- TUNING
AutoDrive-Pi:
AUTO
DRICAPE
Yuqi Ye Zijian Zhang" Junhong Lin', Shangkun Sun', Changhao Peng', Wei Gao School of Electronic and Computer Engineering, Peking University
ABSTRACT
Vision-language models (VLMs) are increasingly being adopted for end-to- end autonomous driving systems due to their exceptional performance han- dling long-tail scenarios. However, current VLM-based approaches suffer from two major limitations: 1) Some VLMSs directly output planning results with- out chain-of-thought (CoT) reasoning, bypassing crucial perception and predie- tion stages which creates significant domain gap and compromises decision- making capability; 2) Other VLMs can generate outputs for perception, pre- diction, and planning tasks but employ a fragmented decision-making approach where these modules operate seperately. leading to significant lack of syn- ergy that undermines true planning performance. To address these limita- tions, we propose AuloDrive-P's a novel framework that seamlessly integrates Perception, Prediction, and Planning through structured reasoning. We introduce the p3-Col dataset to facilitate coherent reasoning and propose Pš-GRPO, a hierarchical reinforcement learning algorithm that provides progressive supervi- sion across all three tasks. Specifically, AutoDrive- progressively generates CoT reasoning and answers for perception, prediction, and planning, where per- ception provides essential information for subsequent prediction and planning, while both perception and prediction collectively contribute to the final plan- ning decisions, enabling safer and more interpretable autonomous driving. Ad- ditionally, balance inference efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking. Extensive experiments on both open-loop (nuScenes) and closed-loop (NAVSIMv1/v2) benchmarks demonstrate that our approach achieves state-of-the-ane performance in plan- ning tasks. Code available at wttppppp....ppa.aiiiiiiiniaa///a
INTRODUCTION
Autonomous driving aims to predict trajectories that are both comfortable and collision-free by leveraging environmental and ego-vehicle information. Traditional approaches decouple the au- tonomous driving pipeline into three independent stages: perception (Li et al., 2024c] Liang et al.) 2022), prediction (Zhou et al. [023] [Shi al., 2024), and planning (Huang al. | [024b)] Liu al, 2025 However, these module design often leads to error accumulation, which significantly de- grades the final trajectory quality. Recent years have witnessed significant advancements in end-to- end training for autonomous systems (Hu al. 2023 2022 Jiang et al. 2023), shown in Fig. I[).. Nevertheless, these small-scale end-to-end models are constrained by limited dataset size and model capacity, resulting in lack of world knowledge and poor performance in long-tail scenarios.
To address long-tail scenarios, recent works (Tian et al. 2024 Wang et al. 2024 Zhoou et al.) 2025a by: Yuan al. 2025 Bai et al. 2026) introduce Vision-Languaal Models (VLMs) into tonomous driving. Leveraging large-scale pre-training, VLMs show strong adaptability to diverse scenarios. However, current VLM-based end-to-end systems face three key limitations: 1) Lack of
"Core contributor, Please appendix for individual contributions. Corresponding author.

---

Published as a conference paper at ICLR 2026
BEV Net
Perception
Prediction
Plonning
Images
Vision Language Model (VLM)
Images
Tnej
Ego
Ego Status
Ego Status
Status
Only
(o) Small-Scale End-to-fil End Model
(b) VLM Besed End-fo-End Model (Plorning-Only
Perceptions
Perception
Images
Vision Language Model
Images
Vision Language Model
(VLM)
(VLM)
Ego Status
Feeeback
Ege Status
Prediction
Planning
Prediction
Plarning
(c) Bosed Iad-to-End End Model (Separately)
(d) B.seB End-te-end Model (Synergistic, Ours)
Figure 1: The difference between AutoDrive-PP p8 and other paradigms. Our method combines an end-to-end training framework with a three-stage collaborative supervision form with VLM.
Chain-of-Thought (CoT) supervision: VI.M-based systems benefit from COT, but some VLMs di- rectly output trajectories (Fig. [(b),, limiting reasoning for decision-making. 2) Lack of multi-task synergy: Although most VLMs (Zhou et al. 2025a Wang et al. 2024) can answer perception, pre- diction, and planning queries
they treat these tasks seperately, resulting in poor synergy and weak planning. 3) Planning-only GRPO supervision: Existing Group Relative Policy Opti- mization (GRPO) applications optimize only planning metrics such as L.2 distance or closed-loop performance (Zhou et al., (025b)) Yuan al., (025). leaving perception and prediction without direct supervision. This yields superficial gains, limited interpretability, and unreliable planning.
We argue that these limitations lies in the failure to capture the staged CoT process across percep- tion, prediction, and planning. Autonomous driving fundamentally requires these three stages to work in synergy, where accurate perception enables reliable prediction, and both are indispensable foundations for robust planning. However, conventional approaches with planning-only optimiza- tion neglect this interdependence, treating perception and prediction as byproducts rather than core components. Accordingly, we reconsider the role of GRPO in autonomous driving. Rather than re- stricting supervision to the planning stage alone, GRPO should extended to explicitly encompass perception, prediction, and planning within a unified chain. Such formulation ensures synergistic interactions across three modules and promotes coherent reasoning throughout the entire pipeline.
To address above fundamental limitations, we propose novel threc-module supervised GRPO algo- rithm specifically designed for the AutoDrive-P framework, as illustrated in Fig, dd), which uni- fies Perception, Prediction, and Planning into cohesive architecture. The AutoDrive pd frame- work is capable of not only answering perception and prediction queries but also enhancing plan- ning performance through synergistic interactions among all three modules. During the Supervised Fine-Tuning (SFT) stage, we train the model using our proposed pá-CoT dataset, resulting in the AutoDrive- på base model. This model can generate responses following structured perception- prediction-llanning CoT format, thereby reducing the domain gap between VLMs and autonomous driving systems. Subsequently, inspired by the GRPO algorithm (Shao al. 2024 Guo al. 2025 Zhang et al. 2025). we propose P²-GRPO algorithm, which is a novel hierarchical and progres- sive optimization reinforcement fine-tuning (RFT) method that provides explicit supervision across perception, prediction, and planning modules. The p P²-GRPO algorithm not only improves the ac- curacy of perception and prediction but also significantly enhances the model's planning capability by ensuring coherent and context-aware decision-making.
We extensively evaluate AutoDrive-PP P3 using real-world datasets, including the closed-loop NAVSIMv 1/v2 (Dauner et al., 2024 Cao al. 2025) and the open-loop nuScenes (Caesar et al. 2020). Experimental results demonstrate that AutoDrive- achieves superior performance across various end-to-end autonomous driving benchmarks under both open-loop and closed-loop settings. More importantly, experimental results validate that our proposed poopepoopppp. P²-GRPO algorithm signifi- cantly enhances planning performance through its hierarchical and progressive supervision mecha- nism, which systematically improves perception and prediction capabilities and consequently leads more reliable and accurate planning decisions. Additionally, to balance inference efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking. The main contributions of this paper are summarized follows:
1. We present AutoDrive-Pá, end-to-end vision- vision-language driving framework that solves a key limitation of current VLMs by explicitly capturing the relationship between perception, prediction, and planning in autonomous driving.
2

---

Published as a conference paper at ICLR 2026
2. We introduce three-module supervised PR-GRPO algorithm that provides hierarchical and progressive optimization across perception, prediction, and planning tasks, signifi- cantly enhancing reasoning coherence and planning reliability by our proposed pa P3-CoT dataset. Additionally, to balance efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking.
3. We demonstrate that Auto Drice-Ps achieves state-of-the-ane performance multiple tonomous driving benchmarks, including both open-loop and closed-loop tests, underscor- ing the effectiveness and generality of our approach.
RELATED WORK
2.1 END-TO-END AUTONOMOUS DRIVING METHODS
Autonomous driving systems have transitioned from traditional modular designs-ffatturing decou- pled perception. prediction, and planning modules -toward end-to-end learning frameworks. Rep- resentative methods such as UniAD (Hu et al. 2023). ST-P3 (Hu et al. 2022) and VAD (Jiang al.) 2023) integrate these tasks into a single model trained jointly, improving planning performance. DiffusionDrive (Liao et al. 2025) integrates diffusion into trajectory planning, and WoTE (Li et al.) 2025) leverages BEV-based world model to predict future agent states, enabling online trajectory evaluation and selection. Though end-to-end autonomous driving methods make great progress, they still suffer from lack of world knowledge and poor performance in long-tail scenarios.
Due to the limited capacity of such compact models and their constrained semantic understanding of complex environments, recent efforts increasingly incorporate Vision Language Models (VLMs) into driving systems. Approaches including Drive (Tian et al.] 2024). EMMA (Hwang et al,, 2024) VLM-AD (Xu et al. 20224). OpenEMMA (Xing al. 2025 OmniDrive (Wang al.. 2024).
OpenDrive (Zhou et al. 2025a),, and Auto VLA (Zhou et al. 2025b) benefit from VLMs rich world knowledge and reasoning capabilities, demonstrating strong performance in driving scenar- ios. Nonetheless, while these methods are capable of answering QA-style queries about perception, prediction, and planning, they often address each task in fragmented manner rather than through unified modeling. This lack of integration prevents the planning module from fully leveraging per- ceptual and predictive features, ultimately limiting overall planning performance.
2.2 GROUP RELATIVE POLICY OPTIMIZATION
The Group Relative Policy Optimization (GRPO) algorithm (Shao et al. 2024 Guo et al. 2025).
introduced by DeepSeek, has demonstrated strong potential in enhancing the reasoning capabili- ties of Large Language Models (LLMs). With Vision-R1 (Huang et al. 2025) applying GRPO to Vision-Language Models (VLMs) and R1-VL (Zhang al. 2025) further adopting step-wise ward mechanisms, GRPO has proven effective in improving "LM-based reasoning. In the context of autonomous driving, several works, such AutoVLA (Zhou | (025b), Plan-R1 (Tang al.) 2025), AlphaDrive (Jiang et al. 2025), and AutoDrive-R2 (Yuan et al. (2025), have successfully incorporated GRPO enhance the performance of driving-oriented VLMs. While these methods achieve notable results, they primarily rely on supervised learning only on the final planning out- puts, without reinforcing perception and prediction modules through reward guidance. This narrow focus limits the synergistic effects between reasoning and low-level control, thus constraining the full potential of integrated planning capabilities.
3 PRELIMINARIES
VLM--ased End-to-end Autonomous Driving Problem Formulation. We model end-to-end au- tonomous driving as mapping inputs to trajectory Traj {(1, w))) where (2t, Mr) is the ego vehicle's position at time t. Given ego state E, sensor data S, and commands C,, the trajectory distribution is autoregressively factorized as:
T
P(Traj| E,S,C) = II P( v.) E, (2o, wo)...., (fc-11 1t-)
(1)
t=0
3

---

Published as a conference paper at ICLR 2026
Video
Ego Infos
are an expert driving assistant. As an expert driving assistant, analyze the driving video context and answer the perception, prediction and planning question
Ego status
Historical trajectory
Ego
command
Vision
Vision
Text Tokenizer
Encoder
AutoDrive-p3
(P' (P²
Perception token
Prediction token
Planning token
Perception-Coe CoT
Prediction-CoT
Planning-Co
qperapptinn shikkigs Lan's ssamies kay objects eult
Let ne andyys the prodiction recaceing step 140g narest stote Yign strught i fature
anclyse the plamnin resseing stop Trajectory Andyyii
1. There theee widds Trade ia 2. "Preet Tructh;
denations conirton amdll ""mppst dry Obyectr"
iegtectory
slghtt loterd
fomnymund The hounday bex coandiatte are (187, 126,
100) -/peroopnann
validee clred cr
novig stright age wchicee
-tndiiDDo 100, 1891 Affects the shiide, trajectrry aight
pprreptior.cveee jaunaatriiyy baoc (392, 126, Vieselys
Sart Adt: pS2, 126, 448, 1891, Vietrr, Vimagettt) cppetiiction
weee (0.01. Videh"
10 Se5
Pront ch
Figure 2: Overview of AutoDrive-P processes video and ego vehicle data through structured Perception- Prediction-Plamant Chain-of-Thought Thought (P8-CoT) reasoning, generating interpretable step-by-step rationale and structured outputs for perception, prediction, and planning.
This formulation integrates all inputs to predict future positions sequentially, capturing temporal dependencies in unified end-to-end framework.
Group Relative Policy Optimization (GRPO). GRPO improves learning stability by removing the dependence value function and optimizing group-level, sample-wist objective (Shao al. 2024]. For each question-answa annwerr pair (q, a), the behavior policy Te.. generates a group of re- sponses {o: [ The normalized advantage for the i-th response at step / is computed as:
ALe- R₂ mcan ( (R2
(2)
R, )
where R² is the reward of the i-th response. The GRPO objective integrates clipped surrogate loss with a KL penalty term:
G
[f) = (ataa))ro
(J. (J. -BKIL (Toll(
(3)
G
To(o)14) Tom (oslg)
To To(ail9) _olarl921 1-,, Te (o:1a)
J. min
Ai, clip
A,
(4)
By leveraging diverse responses sampled from the model itself, GRPO enhances the model's rea- soning capability through exposure to varied reasoning paths and solutions.
4 METHODOLOGY
In this section, we propose the AutoDrive-PP p3 framework, which integrates Perception, Prediction, and Planning for autonomous driving. Existing VLM-based datasets offer only fragmented QA pairs, unsuitable for GRPO training. To solve this, we create the PS-Col dataset with unified CoT sequences linking the three tasks. We then perform supervised fine-tuning for cold-start initialization to align VLMs with the autonomous driving domain and generate accurate P3 outputs. Finally, the P8-GRPO algorithm is introduced for post- post-training, providing hierarchical supervision and enabling collaborative optimization across modules to improve planning via iterative CoT reasoning.
4

---

Published as a conference paper at ICLR 2026
1.Data Sampling
Data Annotation
Perception Checking
Ego status
Historical trajectory
prediction
Ego command
Bounding Box
planning
T. Generation
Scene Understanding
Prediction Description
Planning Trajectory
AnnoTation
Bounding Box
Prediction
Planning
Prompt
Qwen2.5-VL--28
Action
Command:
Figure 3: The pipeline for constructing P8 dataset. We first sample data and annotations from existing datasets, then construct the labels of samples, focusing on key objects and using rule- based and manual filtering. Finally, with the help of advanced VLM, construct the CoT, focusing on the connection among perception, prediction and planning three stages.
4.1 P3-CoT DATASET
To cover all the thinking steps of human drivers and meet the need of VLMs, a high quality reason- ing dataset with key objects and detailed CoT annotations is strongly recommended. However, the following key challenges remain to be addressed: 1) lack of completed and comprehensive key ob- ject annotations, 2) requirements of unified chain of thought datasets for perception, prediction and pairs.
plannning, 3) a proper CoT format suitable for VLM training instead of question-and-answwe propose P3-Col' dataset, high-quality key objects' labels with CoT
To address these issues, designed for VLM GRPO post-training, as shown in Fig. 3 We first identify and annotate critical ob- jects in each key frame of the original dataset based on their potential impact on vehicle navigation, marking their bounding boxes as perception labels. We then derive prediction labels by projecting these critical objects' future trajectories. Finally, planning labels are obtained from the ego vehicle's planned trajectory. With these three-stage labels, we employ Qwen2.5-VL-77 to generate coherent CoT data that seamlessly connects all three stages, with manual verification to ensure the correctness and logical integrity of the synthesized reasoning chains. Employing this annotation pipeline, we organize a high-quality and comprehensive CoT dataset with key object an- notation and unified pa arthchitecture CoT format. P8-CoT includes 25303 frames from 850 scenes based on nuScenes, and 115434 frames from 1382 scenes based on NAVSIM. Additional description, statisties, and examples are attached in Appendix C
(Bai et al., 20255)
Furthermore, we highlight that the proposed P8_CoT dataset benefits the model at both the holistic and modular levels. From holistic perspective, the sequential reasoning process- from perception the model in developing coherent and strategic driving
to prediction, and then to behaviors. At the modular level, the specialized CoTs for perception, prediction, and planning respectively enhance the model's 's accuracy and reliability in executing each subtask. Comprehensive experiments in Section 3 validate these benefits across both levels.
4.2 SUPERVISED FINE-TUNING FOR COLD-START
To equip a VLM with autonomous-drivee knowledge and structured reasoning capabilities, we con- duct supervised fine-tuning (SFT) using the proposed p3-Col dataset. As illustrated in Fig. 2 the model processes multimodal inputs = [Zegoi Viidoo Femad; prommp and learns to generate structured outputs organized into perception, prediction, and planning modules. The target output follows unified format for each module:
y [pprocopioon Uprediction Vplanning]:
where
Vmnodute
[ Yanswer)
(5)
This approach enables the model to produce coherent reasoning traces followed by concrete answers, reasoning across all three autonomous
establishing foundational capability for
5

---

Published as a conference paper at ICLR 2026
SFT Phose
Rewards
AutiOrive-- Bone Medel
R
R.
S-meer())
Percaption Reword:
Frediction fd 0.35
fet
Collsiion
Pl-CoT
R₂
Group Advantage
Policy Model
Fercephion Reward: 0.67
Predichin Reward: 0.35
Harning Reward:
Short Progress
:
Question
Update
Revard: 0.86
Ferception
Rewardd 0.86
Prediction
Reward: 0.92
Wel Done!
Figure 4: The pipeline of P*-GRPO. We first cold start the base model using 1 -Co make up for the gap between VLM and autonomous driving and learn the CoT answer format. Next GRPO to find the best optimization path and update our model.
driving stages. The training objective minimizes the negative log- log-iikei of the target sequence:
T
CSFT
Σ
log P(ye (....)) 2),
where T' is the total length of the target sequence. After cold-start SFT, the VLM acquires essential driving capabilities and produces interpretable p3-CoT outputs that enhance both transparency and performance, forming a solid basis for subsequent reinforcement learning.
4.3 P3-GRPO ALGORITHM
Following the cold-start SFT phase, we further enhance the VLM'' reasoning capability across all three stages by applying the GRPO algorithm to the perception, prediction, and planning modules collectively, yielding the p²-GRPO algorithm, as shown Fig. Our approach employs a multi- component reward function to guide the policy model toward generating accurate, coherent, and well-structured outputs through coordinated reinforcement learning across these cognitive layers. The overall reward is computed as weighted sum of the following components:
R(g. a) Afomma
Riforma
Apere
Pyere + Apred Rered +
Aplan Rglans
where Aformat Aperes Apred, and Aplan are weighting coefficients for each reward term. This integrated reward structure explicitly encodes the causal relationship between modules: perception enables prediction, and together they provide the necessary foundation for effective planning. By simul- tancously optimizing together, our approach ensures that improvements in planning accuracy are grounded in corresponding enhancements in perceptual understanding and predictive capability.
Perception Reward (Rperc) measures object detection quality based on average IoU, precision (P), and recall (R), which encourages accurate and spatially precise perception, enabling reliable predic- tion and planning reasoning:
1.0.
if B = 0 and |Bpred 0.
Rpere
IoUng (0.5P +.55+00.55 0.5R), if B > 0 and [Bpood] O, 0.0,
otherwise.
Prediction Reward (Rpred) evaluates forecasting accuracy by combining behavior label correctness weighted by loU and detection quality, which links perceptual accuracy with semantic correctness to foster robust prediction:
(ERsnem loU,y T(s -==))) 85)
Ryred
x (IoUaeg +
(9)
EgSee loUlj
Planning Reward (Rptaa) quantifies trajectory quality via L2 distance:
2
Rplan
(10)
clip(L22, L2mm)

---

Published as a conference paper at ICLR 2026
Algorithm P'-GRPO: Prrepption-Predcctioon-o Group Relative Policy Optimization
Require: Policy model me with PS-CoT; dataset Apmd. Aplin; constraint 8; clip param é.
{Qn) reward weights Acemus Apere
Ensure: Optimized policy model my
iter NRL do Sample query ~D
Generate group responses {ar To(-112
for i
do
Parse o; into perception, prediction, planning components
Compute rewards: Ronnur Rpsc Ryees Rigan
Aggregate reward: RG Aromm Rioomax + Apenc Pipore + >xred Ri + Aplan R
for
Compute mean R xे E; R₁ and std on E.l. R)2
10:
for I M do
11:
Normalize advantage: A, R--
12:
for
13:
for
do
14:
mala. (Q) Compute ratio r; ]ar(,;[[]
15:
Surrogate objective: Jj min -
16:
for
17:
Compute policy loss: Lpolicy = - E; J., KL penalty: LxL. = 3. Teet
18:
Update To via gradient descent on Lpolicy + LKL
19:
end for
20:
return m
Table I: Performance comparison nuScenes Benchmark.
L2 (m) 1
Collision (%)
Method
VLM
Avg.
Avg.
Non- Non-Autoregressvve Methods
ST-P3 Hu et al.
2022
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0.71
VAD hiag et al. 2025
0.17
0.34
0.60
0.37
0.07
0.10
0.24
0.14
Ego-ML el 2024d
0.46
0.76
0.78
0.21
0.35
UniAD Hu
0.44
0.67
0.96
0.69
0.04
0.08
0.23
0.12
InsightDrive (Song
2025
0.23
0.41
0.68
0.44
0.09
0.10
0.27
0.15
Autoregressive Methods
GPT-Driver (Mao et
2023
0.20
0,40
0.70
0.44
0.04
0.12
0.36
GPT-3.5
DriveVLM (Tan el al.
2024
0.18
0.34
0.40
0.10
0.22
0.27
Owen2-V1.-7B
OpenEMMA Xing 2022
1.45
3.21
3.76
2.81
Qwen2-VL-7B
RDA-Driver [Huang
2124
0.17
0.37
0.69
0.40
0.01
0.05
0.26
0.10
LLaaVa-B
OmniDrive Wang al 2124
0.14
0.29
0.55
0.33
0.01
0.04
0.27
0.11
LLava-7B
OpenDrive VLA
Zhou al
2025a
0.14
0.30
0.55
0.33
0.02
0.07
0.22
0.10
Qwen2.5-VEs
L-B
AutoVLA Zhou el al. 20256
0.25
0.46
0.73
0.48
0.07
0.07
0.26
0.13
Owen2.5-VLan
AutoDrive- (Yann et [.l,]20 2025
0.35
0.49
0.62
0.49
Qwen2.5-VT.-33
AutoDrive-Pl (Ours-Detailod)
0.15
0.30
0.54
0.33
0.00
0.02
0.15
0.06
Qwen2,5-VL-3B
AutoDrive-Pi (Ours-Fast)
0.16
0.31
0.56
0.34
0.00
0.04
0.20
0.08
Qwen2,5-VL-3B
These rewards together form a coordinated learning signal that promotes synergy among perception, prediction, and planning, ultimately driving accurate and interpretable autonomous driving behavior. The complete algorithmic procedure is summarized in Algorithm [ Detailed formulations and
comprehensive analyses of each reward component provided in the Appendix D
5
EXPERIMENTS
5.1
BENCHMARKS
nuScenes (Caesar et al. 2020). The nuScenes dataset comprises 1,000 real-world driving sequences.
Following established evaluation protocols in related works (Hu et al.
2023 Diang al. 2023
Wang et al., 2024 Zhou et al. 2025a), we adopt two key metrics for planning performance:
displacement error and collision rate, using the same ST-P3 (Hu et al. 2022 metric settings.
NAVSIM (Dauner et al. 2024 Cao et al. 2025). To address the limited complexity of nuScenes, further validate approach using the NAVSIM benchmark. NAVSIMvI DDuuner et al. 2024)

---

Published as a conference paper at ICLR 2026
Table Performance comparison on NAVSIMv1 benchmark.
Method
Image
Lidar
NC1
DACT
EPT
TTC1
Comft
PDMS
Human
X
100.0
100.0
87.5
100.0
99.9
94.8
Constant Velocity
X
X
69.9
58.8
49.3
49.3
100.0
21.6
Ego Status MLP
X 93.0
77.3
62.8
83.6
100.0
65.6
VADv2 (Weng
2024
X
97.9
91.7
77.6
100.0
83.0
UniAD (uu et al.
X
97.8
91.9
78.8
92.9
100.0
83.4
LTT (Prakash of a. 2021
X
97.4
92.8
79.0
92.4
100.0
83.8
TransFuser (Prakash el a
2021
V
97.7
92.8
79.2
92.8
100.0
84.0
PARA-Drive (Weng et al.
2024
X
97.9
92.4
79.3
93.0
99.8
84.0
LAW (Li et al. al. 2024a
96.4
95.4
81.7
88.7
99.9
84,6
DRAMA Yuan
2024
98.0
93.1
80.1
94.8
100.0
85.5
Hydra-MDP
2024b
98.3
96.0
78.7
94.6
100.0
86.5
DiffusionDrive Lia0 al. 2025
98.2
82.2
100.0
88.1
WoTE (Li et al.
2025
98.5
96.8
81.9
94.9
99.9
88.3
AutoDrive-Pa p8 (Ours-Dees (ers-Detaille)
X
99.1
97.4
84.8
96.5
100.0
90.6
AutoDrive-Pa (Ours-Fast)
98.9
97.7
83.7
99.9
90.2
employs a simulation environment and uses the Predictive Driver Model Score (PDMS) for closed- loop evaluation. The PDMS is composite metric defined
5 EP + X TTC + Comf
PDMS = NC x DAC x
(11)
12
where the components include No Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Lime-to-Collistee (TTC), and Comfort (Comf). In addition, NAVSIMv2 (Cao al.) 2025)
offers a more comprehensive metric, named Extended Predictive Driver Model Score (EPDMS):
EPDMS NC x DAC DDC x TLC 55 55x EP + 5 TTC LK HC EC
16
(12)
where the components include Driving Direction Compliance (DDC), Traffic Light Compliance (TLC), Lane Keeping (LK), History Comfort (HC), Extended Comfort (EC) and other metries the same as PDMS. To reduce false positive penalties, NAVSIMv2 sets "human penalty filter" to true, disabling penalties when the human agent makes violation; otherwise, it is set false.
Table 3: Performance comparison on NAVSIMv2 benchmark.
EPDMS! False True
Method
NCT
DAC
DDCt
TLC
EP
TTC
LK
HC²
Human
100.0
100.0
100.0
90.3/
Ego Status MI.P
93.1
77.9
92.7
91.5
64.0
Transfuser (Pakassh al.
2021
96.9
89.9
97.8
99.7
87.1
95.4
92.7
98.3
87.2
76.7 / 84.0
HydraMDP ++ Li al 2024b
97.2
97.5
99.4
99.6
83.1
96.5
94.4
98.2
70.9
81.4 /-
DiffusionDrivve Laa at.
98.2
97.3
WoTE (Li al. 2025
98.5
96.8
98.8
99.8
86.1
97.9
95.5
98.3
82.9
84.2 87.7
AutoDrive-P p3 (Ours-Detailed)
99.1
97.4
99.2
99.8
88.0
98.7
96.3
98.3
85.5
86.2 / 89.9
AutoDrive-Pi (Ours-Fast)
98.9
97.6
98.9
99.8
86.8
98.5
95.4
98.3
80.6
85.2 88.7
5.2 IMPLEMENTATION DETAILS
nuScenes Benchmark. Model inputs consist of 3-second video clips composed of 6 frames, drawn solely from the front-view camera. Images are resized to resolution 448x252. The ego-state information provided is only ego speed. The model is trained for 10 epochs with batch size of 8.
NAVSIM Benchmark. Model input is constructed from 2-second video segments containing frames, combining the front, front-left, and front-right camera views, which is then resized to 672x168. The ego-state information includes the longitudinal and lateral velocity and acceleration components (i.e., Ua, Uys ags a.y) The model is trained for 10 epochs with batch size of 32.
Shared Settings We use Qwen2.5-VL-33 VL-3BB (Bai et al. 2025) as base model. All models are op- timized using the AdamW optimizer across 8 A100 GPUs. During training with Pa-GRPO,
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study on AutoDrive-Po on nuScenes Benchmark.
Planning (Collision, % )
Method
Perception f
Prediction f
1s
2s
3s
Avg.
UniAD (Hu et al.
2023
0.32
0.31
0.04
0.08
0.23
0.12
OmniDrive Wang et al.
2024
0,37
0.01
0.04
0.27
0.11
AutoDrive p3 (Only SFT)
0.33
0.23
0.01
0.08
0.40
0.17
PutoDrive-PS (SFT + Only Planning GRPO)
0.03
0.08
0.24
0.12
AutoDrive- (SFT PS-GRPO)
0.64
0.54
0.00
0.02
0.15
0.06
Table 5: Ablation study on different training setting on nuScenes benchmark.
Group
History
Sensor
L2 (m)
Collision (%)
Method
Size
Traj.
Type
1s
2s
3s
Avg.
1s
2s
3s
Avg.
Ablation I
4
Video
0.17
0.32
0.65
0.38
0.01
0.06
0.30
0.13
Ablation 2
8
x
Video
0.17
0.33
0.68
0.39
0.02
0.07
0.33
0.14
Ablation 3
8
Image
0.16
0.32
0.61
0.36
0.01
0.05
0.26
0.12
PS-GRPO
8
Video
0.15
0.30
0.54
0.33
0.00
0.02
0.15
0.06
Ps-CoT samples are generated for each scenario. The reward function incorporates multiple com-
ponents balanced by the following weights: Atormat Xpere. Apred. and Xylan in ratio of 1:2:2:5. Fol- lowing (Zhou et al., 2025b), we add PDMS to planning reward for NAVSIM benchmark. For each benchmark, we implement a tual-thinniing setup consisting of fast and detailed version, as shown in Fig. 5 The fast version is designed for efficiency; while it adheres to the P²-Col structure, it only yields the final answer from each module without reasoning. The detailed version, in contrast, provides the complete reasoning with answer for all modules.
5.3 COMPARISON WITH STATE-OF-THE-AN METHODS
As show in Table we compare AutoDrive--P* p3 with mainstream methods nuScenes dataset. We achieve the same level as SOTA methods 1.2 and overpass about 40% compared to SOTA methods at collision rare. In Table 2 and Table 3 AutoDrive-P" achieves the SOTA results with vision- only input on NAVSIMv 1/v2 benchmark, achieving 90.6 PDMS and 89.9 EPDMS. Specifically, our LLava-
method achieves comparable L2 scores with significantly smaller model (Qwen2.5-38) 7B used OmniDrive) and less training data (20k vs. 1000k samples used in OpenDrive while also attaining the best collision rate, demonstrating the superior efficiency and effectiveness of AuloDrive-P'.
5.4 ABLATION STUDY
Ablation Study AutoDrive-P3, We conduct ablation studies AutoDrive-PPi with nuScenes We compare three
to assess the effectiveness of joint RFT across perreption-pradiiioon--ppaiiniii settings-(1) Only SFT, (2) SFT Only Planning GRPO, and (3) SFT P8-GRPO-agaln end-to-end baselines: the small-scale UniAD and the VLM-based OmniDrive. As shown in Ta- ble Only SFT already achieves lower collision rate than basclines. Adding Planning GRPO further reduces the 3s collision rate, matching baseline performance. Crucially, P3-GRPO yields large improvements perception and prediction, surpassing all baselines and significantly boost- ing planning. These results demonstrate that P3- effectively captures the staged CoT across perception, prediction, and planning, leading to holistic gains autonomous driving.
two
Ablation Study on Training Settings. We conduct additional ablation studies on three training configurations: GRPO group size, historical trajectory usage, and sensor modality. As shown in Ta- 5 results demonstrate that: (1) increasing group size from 4 to 8 enhances performance through more diverse reasoning samples; (2) incorporating historical trajectories improves contextual under- standing; and (3) video sensors outperform image- based inputs by capturing temporal dynamics. Our full configuration achieves optimal results across all metrics.
Runtime and Dual thinking modes. We provide dual thinking modes' inference time compared to other methods, as shown Fig. [5] We employ vLLM 0.8.0 (Kwon et al. 2023) acceleration on an H100 GPU, achieving real-time performance (1 Hz).

---

Published as a conference paper at ICLR 2026
Perception
Prediction
Planning (Thinking Answer)
(Thinking Answer)
(Thinking-Answeer
Nlethed
Avg. 12 tee)
Ang, Colbaion (S)
RPS (Bko)
Ours- Detailed Thinking
UsAD 23' CVPR)
0.09
0.12
Perception
Prediction
Planning (Only Answer)
AuuNTA 25 NeutPS)
0.370.19 0.67008
(Only Answer)
(Onyy Answer)
Ous (Deilled) Rain
0,33 (0.34
05/10
Ours-Fost Thinking
Figure 5: Dual thinking modes and running time on nuScenes Benchmark.
(ART--FSS:
"aeeeeet Tboox 20)
wrthok percaption wttt///ooeeeeeeetttoooooooooo inpeeetiooiineeeee connarr ppecepption" Dfnte 2o
[359. 100, 448, 230). Vlabell: Vpedectrion Fwojit cronoeer perceptiacs
1366. 201, 1881, VTabell') Vear))) "dd" 1294, 128, 448,
octiont:: Viara jear 2dy
Voor""
230g.
jeant 2et"
jaentvEweet "paint 2dy":
1166. 132, 201, 1581, Vivture oecttasccttistt VSstraggen 3. wntt (obe [994, 128,
w pen' (0.0. 0.35) wortypote 24y: (0.0. a 05) V'anallt. 1n1 O
to 0.01 Vibatt:: 112 (...5....
440, 292). Viuoure_sction (aataaght"
(+2 (7, wittboont eay" Sy1) lat "pon__dy" (0.0, 1.07..
2000
1"iubel"" V2.5s1",, A.M.
10.0, 0.0), Viabelt" vo 5..5.33 Antappppiin a8t" j0.0. O 0.06), ("abeli" VI A.
ppent_as" [0.02. 2.6). Viadit 1'3 "rwll
AMD peint_etttt 10.0, 0.89), Viseel (11.S.1..witt post_ 24"" (0.0, 1,04).
Tubell" (2.0012. peint (0.0. 2.171 Mhedi" (2.5 wii
Ground Truth
ppeittady" (0.0. 3.5)
Ours-Detailes
Ground Truth
(Ours-Fast
Ours Detailed
OmniDDrie
Ours-Fast OmniDrive
VAD
OURS DETALLED
VAD
perception Let's cramine the image for key objects such as cers, buses, frucko, bicycles, pedestmtess and metercydlees First, natice pedestrion on pedestrias. The pedestrion
cuiss DE BÉTALLDD
thhkk percaption Let's examine the image for key sbjects each as cers, buses, fruudes bicycles, pedestrions ians and netercycles. First, sotice car the the postiioned tewards the cester- left of the coordinttes.
standing the sidewalk, siihtly to the right the center of the image Wieheen, I need identify the beanding box for pedestrion. firn bounding
appears Antohiest I see snother car the right side the road need to deternne its bounding box intee es wel. car pasitianed fowerds center-nigt the image... These bounding
defied by top- and betton-nght
cordinates
pedestrian's bounding bor appoxinately (358, 110). (n) The botton right camer of the pedestrian's branding ber apronimately of (448, 200), whlithese caardnates represent the pedestrion's 's beanding box image w Wiehhow let's
perceptionn Wo Ri (160. 192, 852, 197). Viabely's "oarl' Booo 281h (296, 124, 448, 252), ViabelV Yearl Tinll"" pewpepe
/hlick
genve Stox 201": (400. 130, 446, 2353. Vlabelt: A perceptions
think
Let's analyae the future octions for each key object lated.
Lat think The object in the scans pedestrizn lecated (400, 110, 448, 2353 The video contest indicates that the pedesttion is not the moing direction of ego vahilee stroight. padestmion does not effect the vahidee the future octice for pedestrion straggtffnwerd ar Vetrright- ww booo (400, 110, 445, 235).
(1.6art166 132, 202. 157 The current stotus is Vie streight V and the monigg direction
state alse stroght
car
ego
cormently gaing stroghtt
wehicle
Thenefare,
Addirionllyy
aa. (2-cor-1296 124, 448, 252js The current status is l'ge streight V and the future state alse V'oe stroght e This car is the moving direction of the ego but
Therefere, ween 2nh [168. 182, 202. 1177, Vffture action": Vatragerys, Woox, ady: (296. 124, 440. 252). V
"future sctiot" Vstrogen
ploming Oh, sec. The ago vchile''s careent speed 0.0 m/s, and the secends. The
natericaa teajectery shons that has been stationery
trojectery for the vehicle-tennl. "*Gurnnt Velicle Specde 0.0 w/s and the whhccle's future action \. "stragget Vinvdd "Ege Fature Action": og vehicle ge straight, and "etrajectoyy The whhile will ascelerate from 0.0 m/a positiv inrecciinn over the next isccends te Asaming Inear acceerationn the y-velues will inceesse over time.. wiiket's calculate the irqjectory pains trajectory reflects gradial
Lat's deternise the
age ege whhide start moving
movingAniato intato deterine the planing trajectory. need to coluucte the trajectory for the not seconds, with each point representing a 0.5-
secand intered The
starting from real, so secelerate over next
cordeates
Striight. oitie
i WTpenn__ 20V" (0.0. gonll boint ady": 10.0, 0,03), Vlebett" vo S.. Ank
V 041) boat ady: 00.0. 0.481.
"i 10.0. 0.00 0.09), Viabel vo 50 Saant Bath (0.0 0.16). Viabeli" VT. dr..
"hiie 2nt (0.0 2.53). Vioelt:: 1-2 Bs)" wittypo 20V: (0.0. 4.23).
5.)) WWll
19 Wiit post 24th 10.0. 0.433 Viedi 1 5.15 Woitrpornt 2dV" 10.0
1.033 Vi 122 o1l" boin say" (0.0. 2.09). Viabel" 22256. winy
"abbet" V3 "3.0"""
"poin ady":
Viabelt "aVW
(a)
(b)
Figure 6: Visualization. Our model, taking into account the scenarios comprehensively, makes efficient plans that are both reasonable and safe.
5.5 VISUALIZATTION
Fig. G(a) demonstrates our method's ability to accurately perceive pedestrian location and predict safe passage opportunities, avoiding the overly conservative decisions of comparison methods. In Fig. b)b), our approach successfully handles complex vehicle interactions by identifying key objects and their behaviors, producing trajectories that align with human driving habits.
6 CONCLUSION AND FUTURE WORK
In this work, we proposed AuloDrive- a novel VLM framework that establishes progressive connections between Perception, Prediction, and Planning. Our approach includes the pa Coo dataset with unified reasoning chains and labels, supervised fine-tuning for domain adaptation, and the P²-GRPO algorithm for hierarchical multi-task supervision. Experiments on NAVSIMv1/v2 and nuScenes show state-of-the-art performance, reducing collision rate by 40% on nuScenes. To balance inference efficiency with performance, we introduce dual thinking modes: detailed thinking and fast thinking. Although AutoDrive-P's achieves state-of-the-art performance, it faces limi- tations in hallucinatory phenomena during reasoning. Additionally, our reinforcement learning is conducted in offline simulators, lacking interaction with real-world environments. Future work will focus on mitigating hallucinations, reducing inference time, and deploying the system in closed-loop settings.
10

---

Published as a conference paper at ICLR 2026
7 ACKNOWLEDGMENTS
This work supported by National Science and Technology Major Project (2024ZZD0O Natural Science Foundation China (62271013). Guangdong Provincial Key Laboratory of Ul- tra High Definition Immersive Media Technology (2024B 1212010006), Guangdong Province Pearl River Talent Program (2021 QN020708), Guangdong Basic and Applied Basic Research Founda- tion 15100011155 Shenzhen Science and Technology Program (JCYJ20240813 3160022004, (Corresponding author: Wei Gao)
1 SYSPG2024121
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics and all authors have read and adhered to the Code of Ethics. In this study, no human subjects involved. The use of all datasets, including nuScenes
CCaesar et al. 2020) and NAVSIM (Dauner et al. 2024] Cao et al.) 2025), follows the relevant usage guidelines and public licenses, ensuring no violation privacy. We have been careful to avoid any biased or discriminatory results during our research process. No personally identifiable information used, and no privacy or security concerns will be raised due to our experiments. We are committed to maintaining transparency and integrity throughout the research process.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper reproducible. All code and datasets have been made publicly available in an anonymous repository to facilitate repli- cation and verification. The experimental setup, including training steps, model configurations, and hardware details, is described in detail in the paper. We have also provided a full description of AutoDrive-PP P3 to assist others in reproducing
experiments.
Additionally, the datasets used in our experiments are publicly available, ensuring consistent and reproducible evaluation results.
We believe these measures will enable other researchers to reproduce our work and further advance the field.
REFERENCES
Junjie Bai, Yu-Wei Chao, Qizhi Chen, Jinwei Gu, Moo Jin Kim, Zhaoshuo Li, Xuan Li, Tsung- Yi Lin, Ming- Yu Liu, Nic Ma. Kaichun Mo, Delin Qu, Shangkun Sun, Hongchi Xia, Fangyin Wei, and Xiaohui Zeng. Openpi Comet: Competition Solution For 2025 BEHAVIOR Challenge, January 2026.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392. 2025.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, et al. Pseudo-simulation for autonomous driving. arXiv preprint arXiv:2506.04211 2025.
Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li. Igor Gilitschenski, Boris Ivanovie, Marco Pavone, al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems, 37:28706-2877 2024.

---

Published as a conference paper at ICLR 2026
Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, and Xiaomeng Li. Holistic au- tonomous driving understanding by bird surd'seey-vviws injected multi-modal large models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13668- 13677, 2024.
Daya Guo. Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to- end vision-based autonomous driving via ppatial-temporal feature learning. In European Confer- ence on Computer Vision, pp. 533 549. Springer, 2022.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-orienter autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 77853-17862, 2023.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint 2wt.iv::50330677 2025.
Zhijian Huang, Tao Tang, Shaoxiang Chen, Sihao Lin, Zequn Jie, Lin Ma, Guangrun Wang, and Xi- aodan Liang. Making large language models better planners with reasoning- decciion alignment. In European Conference Computer Vision, pp. 73-90. Springer, 2024a.
Zhiyu Huang, Xinshuo Weng, Maximilian Igl, Yuxiao Chen, Yulong Cao. Boris Ivanovic, Marco Pavone, and Chen Lv. Gen-drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning. arXiv preprint 2rXiv:2410.05582. 2024b.
Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, et al. Emma: End-to-end multimodal model for au- tonomous driving. arXiv preprint arXiv:2410.23200 2024.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient au- tonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, pp. 8340-8350, 2023.
Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning. arXiv preprint arXiv:2503.07608, 2025.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint N1VV:2206608488. 2024a.
Yingyan Li, Yuqi Wang, Yang Liu, Jiawei He, Lue Fan, and Zhaoxiang Zhang. End-to-end driving with online trajectory evaluation via bev world model. arXiv preprint arXiv:2504.011 01941, 2025.
Zhenxin Li, Kailin Li, Shihao Wang. Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra- distillation. arXiv preprint arXiv:2406.06997, 2024b.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird's v-ew representation from lidar-camera camera via spatiotemporal transformers. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024c.
12

---

Published as a conference paper at ICLR 2026
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14864-14873, 2024d.
Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35:10421-10434 2022.
Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, al. Diffusiondrive: Truncated diffusion model for end- to-end autonomous driving. In Proceedings of the Computer Vision and Patlern Recognition Conference, pp. 12037-12047, 2025.
Haochen Liu, Zhiyu Huang, Wenhui Huang, Haohan Yang, Xiaoyu Mo, and Chen Lv. Hybrid- prediction integrated planning for autonomous driving. IEEE Trunsactions Pattern Analysis and Machine Intelligence, 2025.
Srikanth Malla, Chiho Choi, Isht Dwivedi, Joon Hee Choi, and Jiachen Li. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 1043-1052, 2023.
Jiageng Mao, Yuxi Qian, Junjie Ye. Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint 2rXiv:2310.014/33 2023.
Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chi- dananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Vi- sual question answering for autonomous driving. In European Conference on Computer Vision, pp. 252-269. Springer, 2024.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to- end autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7077-7087, 2021.
Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi- modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4542-4550, 2024.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint axXiv:240..03300, 2024.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mir+: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3955 39771 2024.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph vi- sual question answering. In European conference on computer vision, pp. 256-274 Springer, 2024.
Ruiqi Song, Xianda Guo, Hanghin Wu, Qinggong Wei, and Long Chen. Insightdrive: Insight scene representation for end-to-end autonomous driving. arXiv preprint arXiv:2503. 13047, 2025.
Shangkun Sun, Jiaming Liu, Huaxia Li, Guoqing Liu, Thomas H. Li, and Wei Gao. Streamflow: Streamlined multi-frame optical flow estimation for video sequences. Advances in neural infor- mation processing systems, 37:9205-999 22288 2024.
Shangkun Sun, Xiaoyu Liang, Songlin Fan, Wenxu Gao, and Wei Gao. Ve-bench: Subjective- aligned benchmark suite for text-driven video editing quality assessment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7105-7113, 2025a.
Shangkun Sun, Xiaoyu Liang, Bowen Qu, and Wei Gao. Content-Riaa AIGC Video Quality As- sessment via Intricate Text Alignment and Motion-Aware Consistency, February 2025b.
13

---

Published as a conference paper at ICLR 2026
Xiaolong Tang, Meina Kan, Shiguang Shan, and Xilin Chen. Plan-rl: Safe and feasible trajectory planning as language modeling. arXiv preprint arXiv:2505. 2025.
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevim: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402. / 22899, 2024.
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. CoRR, 2024.
Xinshuo Weng, Boris Ivanovic, Yan Wang. Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Dongming Wu, Wencheng Han, Yingfei Liu, Tiancai Wang, Cheng-zhong Xu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 8359 8367, 2025.
Shuo Xing, Chengyuan Qian, Yuping Wang. Hongyuan Hua, Kexin Tian, Yang Zhou, and Zhengzhong Tu. Openemma: Open-source multimodal model for end-to-end autonomous driv- ing. In Proceedings of the Winter Conference on Applications of Computer Vision, pp. 1001-1009, 2025.
Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint arXiv:2412. 14446. 2024.
Chengran Yuan, Zhanqi Zhang, Jiawei Sun, Shuo Sun, Zefan Huang, Christina Dao Wen Lee, Don- gen Li, Yuhang Han, Anthony Wong, Keng Peng Tee, et al. Drama: An efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint arXiv:2408.03601. 2024.
Zhenlong Yuan, Jing Tang, Jinguo Luo, Rui Chen, Chengxuan Qian, Lei Sun, Xiangxiang Chu, Yujun Cai, Dapeng Zhang, and Shuo Li. Autodrive-r Incentivizing reasoning and self-reflection capacity for vla model in autonomous driving. arXiv preprint arXiv:2509.01944 2025.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao. Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint arXiv:2503. 12937, 2025.
Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, and Alois C Knoll. Opendrivevla: To- wards end-to-end autonomous driving with large vision language action model. arXiv preprint arXiv:2503.23443. 2025a.
Zewei Zhou, Tianhui Cai, Seth Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. Au- tovla: A vision-language-actioo model for end-to-end autonomous driving with adaptive reason- ing and reinforcement fine-tuning. arXiv preprint arXiv:2506. 13757, 2025b.
Zikang Zhou, Zihao Wen, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Qenext: A next-generation framework for joint multi-agent trajectory prediction. arXiv preprint arXiv:2306. 10508, 2023.
14

---

Published as a conference paper at ICLR 2026
A A USE OF LARGE LANGUAGE MODELS
We acknowledge the use of Large Language Models to assist in the preparation of this manuscript. We utilize Google's is Gemini 2.5 Pro and DeepSeek-RR for writing assistance. The specific applica- tions are as follows:
Language and Readability: To improve the grammar, clarity, and overall readability of the manuscript through language polishing.
Format Checking:
Gemini 2.5 Pro and DeepSeek-RI are used to aid in improving gram- matical fluency, and enhancing the overall readability of the text.
We emphasize that all scientific claims, hypotheses, experimental designs, results, analyses, and final conclusions are meticulously formulated, reviewed, and verified by the human authors. The authors take full and final responsibility for entire content of this submission, accordance with the ICLR policy.
B CONTRIBUTIONS
The core authors of this paper are Yuqi Ye, Zijian Zhang and Wei Gao. Yuqi Ye and Zijian Zhang are co-first authors. Wei Gao is the corresponding author. Yuqi Ye is responsible for conceiving the overall framework, developing the training and inference code, conducting the experiments, and performing data processing. Zijian Zhang is responsible for metrics evaluation, data visualization, data labeling, and data checking. All the authors participate the discussion and writing of the paper.
C P3-CoT DATASET
In this section, we will provide detailed introduction to p3-Cof dataset, including motivation, data collection, data composition and distribution and comparisons between datasets.
C.I MOTIVATION
The ultimate goal of autonomous driving models is to drive like humans. This target places high demands on models, requiring them understand the driving environment and think like humans. Human drivers habitually identify and locate key objects that have a significant impact on driving decisions and actions during the driving, and subconsciously predict their future behaviors to help make the final driving routes. With deeper insight into autonomous driving models, the transforma- tion model architecture has undergone a shift from phased perception, prediction and planning to unified end-to-ena structure. Though end-to-end structures gradually become mainstream due the reduction of accumulative errors, segmented consideration perception, prediction and planning remains core design concept when designing man understanding the driving environment, paying too much attention to unimportant objects will instead reduce the concentration of human drivers and lead unsafe driving behaviors. In other words, focusing only on key objects is necessary for efficient and safe driving behaviors, which is the same for the models. Considering the high requirements of comprehensive scene understanding ability and complex logical decision decision-making capability in autonomous driving, the vision language show great potential. To cover all the thinking steps of human drivers and meet the need of VIMs, high quality reasoning dataset with key objects and detailed CoT annotations strongly recommended. This dataset should only focus on key ob- jects and ignore unnecessary objects identification and localization, and construct unified chain of thought (CoT) including perception, prediction and planning.
modules. Meanwhile, in the process of hu-
model (VLM) and reinforcement
However, the existing datasets can not meet such requirements. Some object grounding datasets CCesarr et al., 2020) have been proposed to meet the need for fine-tuning VLMs in autonomous driving perception tasks, but too many objects requiring grounding will introduce additional noises as mentioned before. Existing approaches (Ding et al. (0024) attempt to establish chain of thought datasets to supervised fine-tuning VLMs, and some of them recognize the special meaning of key
15

---

Published as a conference paper at ICLR 2026
car 16063
straight 8836
straight 228
Vehicle
stop
STop
Vehicle
Bike
P3-CoT Dataset
P3-CoT Dataset
truck 3097
bicycle 168
1451
36
Bike
motor- cycle 278
right 1018
rig
bus 1203
64
Human
others
left
Human
right 17
12
Pe Pedestrian
wheel-
2834
chair
straight 2436
stop 491
120
Figure 7: The distribution of categories on p (nuScenes).
Figure 8: The distribution of actions on ps (nuScenes).
objects. These efforts only provide general descriptions of driving conditions, while overlooking the necessity considering the driving process stages and the connection between different stages. DriveLM (Sima al. 2024) points out the importance of key objects and notices the significance of the connection among different thinking steps. But DriveLM lacks a clear definition of key objects and their impacts, and the annotations of Drivee.M-nuScenes frame selection and key object localization. Moreover, though Drivel.M uses graph to model the connection among the three stages of perception, prediction and planning, it formulates the data as question-ansee answer pairs which do not match the labels used in the CoT format and remains fixed content templates lacking sufficient flexibility and diversity. Therefore, the following key challenges remain to be addressed: 1) lack of completed and comprehensive key object annotations, 2) require- ments of unified chain of thought datasets for perception, prediction and plannning, 3) a proper CoT format suitable for VLM training instead of question-and-ansween pairs.
incomplete both in key
C.2 DATA COLLECTION
To address these issues, we propose PS-Col dataset, high-quality CoT dataset designed for VLM training and GRPO post-training. We first clarify significant concepts in our datasets. The key object of specific scene is defined those that human drivers will pay special attention in order to prevent potential dangerous events. This definition aims to simulate the attention of human driving, thereby aligning the model's focus with that human drivers. The concepts of stages our dataset are similar to end-to-end autonomous driving. The perception stage asks the model to localize targets in RGB images, but only key objects; the prediction stage utilizes the results from the perception stage to reason the future behaviors of key objects; the planning stage accepts the objects and cor- responding actions from the first two stages and obtains the final waypoints after comprehensive consideration. To construct pa dataset, first sample key frames of every scene in 2Hz following nuScenes settings, with 3 seconds history information as input and seconds future trajectory as ground truth, and all the bounding boxes of objects will be projected into the image view. Consid- ering that the distance between objects and ego car is an important factor in driving safety, we first filter out objects that are too far away and manually check the left objects to ensure high-quality key object annotations. It is worth noticing that we allow manual annotation of new objects that do exist in previous dataset. After getting the key object annotations, we use trajectories corre- sponding objects to determine future behaviors of objects in general directions like left or right and manually add the same labels according to videos if the object does not match any trajectory. And the way points in the future 3 seconds will be converted to the ego car coordinates as the ground truth. Given labels of the three stages and seconds history information, we use the advanced Owen2.5-VL-72B model synthesize chain-of-thought data and make appearing the previous stage can be used in the next stage to model the connections between different stages. The pipeline is shown in Fig. Employing this annotation pipeline, organize
that only the results
16

---

Published as a conference paper at ICLR 2026
Table 6: The number of key objects in each frame of P3-CoT. The numbers on the table head are the number of key objects in each frame. Average is calculated as total number of key objects divided by total number frames.
Dataset
1
2
3
>
Average
Pa-Col (nuScenes) 8187
9926
5121
1081
76
12
0.97
Po-CoT (NAVSIM)
24675
22915
20172
20794
20793
6085
2.08
Table 7: The distribution of prediction actions of P*CCoT (NAVSIM). The number of position of key objects in longitudinal and horizontal direction.
p8-CoT (NAVSIM)
Left
Front
Right
Far
7915
46909
4530
Middle
10601
33908
6558
Near
41435
31219
47815
Behind
9137
high-quality and comprehensive CoT dataset with key object annotation and unified perception- cnadiction-plannnngg arthchitecture in CoT format. p3-Cofl includes 19284 frames from 700 scenes in training set and 6019 frames from 150 scenes in test set based on nuScenes, and 103288 frames from 1192 scenes training set and 12146 frames from 136 scenes in test set based on NAVSIM. With enough amount of CoT data, PS-Cof dataset can support VLM training for both supervised fine-tuning and reinforcement post-training, and benefit from the connections among stages, VLM trained by the dateset can gain advantages both from staged thought process and lower accumulated error of end-to-end autonomous driving, which also brings additional explainability to the black box in autonomous driving.
C.3 DATA COMPOSITION
p3 P3-CoT is composed of training set of 19284 frames and validation of 5119 frames, attached with detailed annotations for perception, prediction and planning three stages with connection.
For perception stage, we have tallied up the number of key objects that occur in each frame in Table 6 and the distribution of future actions of objects in Fig. 8 and the distribution of categories in Fig. Considering the relatively simple road conditions of the nuScenes dataset, the key objects gathered from each frame should be sparse. The results listed in Table show that our pipeline for selecting key objects is reasonable and successfully reduces the additional and unnecessary objects in the scene. The distribution of categories presents the diversity of p3-Cof and that of actions is in line with the situation in reality.
For prediction stage and planning stage, annotate the future action of every object, including the ego vehicle and other vehicles, and give the detailed results in Table [ Table | and Fig. B For nuScenes, future actions of prediction stage and planning stage have similar distribution, but the action straight of planning stage is more than that of prediction stage due to requirements of
data collection. For NAVSIM, we keep the same command types as nuScenes and adapt prediction action types to NAVSIM. The actions of objects in NAVSIM focus on near range and front direction, showing the complexity NAVSIM in our settings.
C.4 COMPARISONS BETWEEN DATASETS
To highlight the advantages of P3-Col' dataset, we give comprehensive comparisons of our dataset and others in Table 9 We can tell from the table that P3-Col dataset annotates substantial number of frames and includes perception, prediction and planning all three stages. Unlike other datasets, Coo formulates the data in CoT format and maintains the close connection among stages, owning the special advantages to combine both staged interpretation and unified training process.
17

---

Published as a conference paper at ICLR 2026
Table The distribution of planning commands in P3-CoT. The number of planning commands of ego vehicle.
Planning Commands
Left
Right
Stop
Straight
P3 -o (nuScenes)
1299
1627
4919
16558
PS-CoT (NAVSIM)
27293
13524
1832
72785
Table 9: The comparisons of existing datasets of scale and structure. Source Dataset: Source of data sampled. Frames: the number of frames labeled by methods. Perception, Prediction, Planning: whether the dataset includes the information about perception, prediction and planning. Type: data organization format.
Source Dataset
Dataset
Frames
Perception
Prediction
Planning
Type
nuScenes- QA (Qian al.
2024
nuScenes
34149
QA
nuInstruct
(Ding al 2024
nuScenes
11850
V
QA
nuPrompt
Wu et al. 2025
nuScenes
34149
QA
Drivel M-MuSceness (Smma
2024
nuScenes
4871
QA
LingoQA
Marcu et al. 2024
LingoQA
28000
QA
DRAMA
Malla el 2023
DRAMA
17785
QA
(S-CoT(Ours)
nuScenes
24403
CoT+I ahel
NAVSIM
114348
CoT+L
C.5 DATA QUALITY ASSESSMENT
To ensure the high quality of the P3-CoT dataset, adopt manual assessment protocol through sampling inspection, following DriveLM (Sima et al.) (024), and other quality assessment (Sun cet al.. 2025al 2024 2025b). The evaluation is conducted at both holistic and modular levels. At the holistic level, each CoT label is manually inspected to ensure it strictly follows the prescribed reasoning structure- progressigg completely and sequentially from perception to prediction and then to planning, with all final module outputs present and correctly formatted. At the modular level, we perform a fine-grained manual check factual consistency and reasoning quality. The dataset is divided into 10 splits, each assigned to three independent annotators. They randomly sample 10% of their assigned split and meticulously examine the alignment between the generated CoT and the ground-truth labels for every module. This includes assessing the factual consistency of the reasoning steps, the fluency of the language, and the overall logical plausibility. In both levels of inspection, each reviewed CoT is labeled as "good" "poor". The accuracy for split is calculated based on the proportion of "good" samples. We set passing accuracy threshold of 90% for a split be considered qualified. Any split failing meet this standard is deemed imperfect, and the CoT data for that portion is regenerated and re-submitted for assessment. This iterative process continues until all splits pass the manual quality check by all annotators.
D REWARD SETTING
Perception Reward (Rpere): This reward measures the quality of key object detection. Let Epred and Bgt denote the sets of predicted and ground-truth bounding boxes, respectively. The reward is based on the average loU of matched boxes, precision (P), and recall (R)):
1.0,
Bul 0 and [pedd = 0,
Rpere
IoUne (0.5P + 0.5R), 0.0,
if | > 0 and [ > 0, otherwise.
(13)
The perception reward serves three crucial purposes: First, it ensures comprehensive scene under- standing by penalizing both false positives (detecting non-existent objects) and false negatives (miss- ing actual objects). Second, it maintains spatial accuracy through the loU metrie, guarantecing that
18

---

Published as a conference paper at ICLR 2026
detected objects are precisely localized. Third, it provides immediate and interpretable feedback to the model about its perceptual capabilities, enabling focused improvement in visual understanding.
This structured reward mechanism forces the model to develop sophisticated visual reasoning skills, including object recognition, spatial relationships understanding, and scene context interpretation. By explicitly rewarding accurate perception, we create solid foundation upon which prediction and planning modules can build reliable and safe driving strategies. The perception reward thus acts as crucial enabler for the entire cognitive pipeline, ensuring that subsequent decisions are based veridical representation of the driving environment.
Prediction Reward (Rprd): This component evaluates the model's ability accurately forecast the future behavior of detected objects, serving as the critical bridge between perceptual understanding and planning decisions. The reward function is carefully designed integrate both spatial localiza- tion accuracy and behavioral semantics:
EoasEM loUg I(8: - 85) (())=))))
Rprod
(IoUavg (0.5P + 0.5R)),
(14)
EiesEEM loU:;
where M represents the set of successfully matched prediction-ground truth pairs, and Il is the indicator function that returns when the predicted action label s; matches the ground truth 8y. and 0 otherwise.
The formula consists of two multiplicative components. The first term calculates weighted be- havior accuracy score, where each matched pair's label correctness is weighted by its IoU value. This design ensures that predictions with better spatial alignment contribute more significantly to the reward. The second term represents the fundamental detection quality, computed as the product of average IoU and the Fi-score (harmonic mean of precision and recall), which maintains the basic requirement of accurate object detection and tracking.
This reward design serves two crucial purposes. First, it explicitly encodes the dependency between accurate perception and reliable prediction - the model cannot achieve high prediction rewards with- out first establishing solid perceptual foundations. Second, it emphasizes that behavioral prediction quality is intrinsically tied to spatial accuracy; even correct action labels receive reduced rewards if the associated bounding boxes are poorly localized.
By designing the prediction reward in this manner, we force the model to develop comprehensive understanding of scene dynamics, where must not only identify objects correctly but also antic- ipate their future behaviors accurately. This approach ensures that the prediction module provides meaningful and reliable inputs to the planning system, enabling the generation of safe and efficient driving strategies that account for the predicted evolution of the traffic environment.
Planning Reward (Rplan): ): This component serves as the ultimate performance metric that evaluates the quality of the ego vehicle's 's planned trajectory, representing the final output of the entire reason- ing pipeline. The reward is calculated through an exponential transformation of the L2 distance between the predicted trajectory points and their ground-truth counterparts:
Rplan
(15)
+ (llip(22,0, L2
where L2 represents the mean Euclidean distance between corresponding points in the predicted and ground-trufe -tunt- trajectories across all future time horizons. Following ALA (Zhou al.) 20256), we add PDMS to planning reward for NAVSIM benchmark.
The planning reward serves as the ultimate validator of the entire p3 reasoning chain. While high rewards in perception and prediction are necessary prerequisites, they are insufficient without corre- sponding excellence in planning. This design explicitly teaches the model that accurate perception and reliable prediction are valuable precisely because they enable superior planning decisions. The planning reward thus creates a powerful end-to-end learning signal that backpropagates through all modules, encouraging the development of coordinated representations where each component works synergistically toward the final goal of generating safe, comfortable, and efficient driving trajecto- ries.
19

---

Published as a conference paper at ICLR 2026
All Roward
Per Perception Reward
Prediction Rewerd
Planninn Reward
mW
MMW
6.-0-
4.0
bral
1900 200
180 We Who
Figure 9: Training and testing rewards for perception, prediction, planning, and all reward. The results shows the consistent improvement in all rewards, which proves the tight inter-connection among three stages and the effectiveness of p3-CoT.
By placing the planning reward at the apex of our reward hierarchy, we ensure that the model opti- mizes not for intermediate metrics but for the ultimate objective of successful autonomous naviga- tion, while maintaining the interpretability and safety guarantees provided by the structured pd p3-CoT reasoning process.
These rewards with P²-GRPO algorithm ensure that improvements in planning performance are grounded in corresponding enhancements in perceptual understanding and predictive capability, creating a synergistic effect where each module's 's optimization contributes to the overall driving performance. The algorithm maintains the interpretability and safety guarantees provided by the structured P3-CoT reasoning process while achieving superior autonomous driving performance through multi-module reinforcement learning.
Reward Visualization. We visualize the training and testing rewards for perception, prediction, planning, and the all reward, as shown in Fig. 9 The results demonstrate that through training with Po_ dataset and P3-GRPO algorithm, the three modules exhibit mutual reinforcement, leading to consistent improvement in their respective rewards. This synergistic effect is particularly beneficial for planning performance, the enhanced perception and prediction capabilities provide more reliable inputs for trajectory generation. The progressive optimization across modules ensures coherent reasoning and decision-making, ultimately contributing to more robust autonomous driving performance.
we conduct additional ablation studies on three reward weight configurations. Results indicate that an unbalanced setting (c.g.. 1:1:1:7), which overemphasizes the planning reward, can hinder the optimization of perception and prediction modules. This imbalance ultimately leads to inferior overall planning performance, as accurate planning contingent upon reliable inputs from the preceding stages.
Ablation Study on Reward Weight. As shown in Table 10
Table 10: Weight Setting on nuScenes Benchmark.
Reward weight
Perception t Prediction
Planning (Avg. L.2) 4
1:2:2:5
0.64
0.54
0.33
1:1:1:7
0.63
0.53
0.34
E EXPERIMENTAL SETUP
We conduct experiments on two autonomous driving benchmarks: nuScenes and NAVSIM. The detailed experimental configurations are summarized in TT] For both datasets, we compare the Cold-Start baseline with our proposed P3-GRPO approach under consistent data settings. We also perform an ablation experiment by removing the KL divergence term from the training objective. As demonstrated in Fig. 10, the model without KL regularization suffers from significant performance degradation as training progresses, eventually leading to model collapse. This occurs because the absence of KL constraint allows the model to deviate excessively from the base policy, resulting in unstable optimization. Therefore, we recommend retaining the KL divergence term during training ensure the model maintains reasonable proximity to the base policy while improving performance.
20

---

Published as a conference paper at ICLR 2026
Table II: Experimental setup.
nuScenes
NAVSIM
Cold-Start
Pa-GRPO
Cold-Start
PR.GRPO
Data Setting
Video Shape
[6. 3, 252, 448]
2,2,, 448] 6
[4. 168, 672]
[4. 3, 168, 672]
History Traj
Future Traj
6
Ego Infos
Only V
Only V
Oz Bg,
Gy Vay Vy
Optimization
Epoch
5
2 8 AdamW 2e-5
10 256
Batch size
8 AdamW 2c-5
256 AdamW le-6
Optimizer
Adam W 1e-6
Learing Rate
GRPO Setting
Group Size
8 0.01
KL Weight
0.01
R. Divergence va Training Step
0.4
0.2
teatnsing 20ap
Figure 10: Ablation Study on KL Divergence.
F QUALITATIVE COMPARISON OF TRAJECTORY PLANNING
To further explain the advantages of our method, we provide an intuitive comparison of the results in this section. For nuScenes samples, the red points denote ground truth trajectory, the orange denotes trajectory with detail CoT, the pink denotes our trajectory with only CoT framework, the green denotes OmniDrive trajectory and the blue denotes VAD trajectory. For NAVSIM samples, the red points denote ground truth trajectory, the orange denotes trajectory with detail CoT, the pink denotes our trajectory with only CoT framework, the green denotes WoTE trajectory and the blue denotes DiffusionDrrive trajectory. Due to camera projection limitations, short and too deviated trajectories will not appear in the images, such as stop situations. The CoT and answers corresponding to the specific sample are shown on the right of the figure.
In Fig. II a the front view shows scene at night with waiting cars. Due to image distortion, the street lamps have shifted towards the green light a certain extent. The ground truth trajectory is to stop and wait, while the trajectories of comparison methods ignore the front car and move forward mistakenly. Our method first correctly observes the key objects, the closest two cars, and does not misunderstand the meaning of the lights, and then precisely gives the right future actions of the two. This result supports our planning decisions and our method finally takes the same actions as ground truth. This shows our method the powerful ability of scene understanding.
In Fig. b)), two trucks are parked by the roadside, and one of them blocks moving direction the ego car. The ground truth trajectory still moves forward and tends to return to the initial road, but it seems too close to the truck on the right. The comparison methods do not work well in this sample. Our method can still identify key objects before and provide good prediction answers. Though the trajectory is at a certain distance from the ground truth, we believe that our method takes into account the truck on the right correct its initial decision and avoid potential security issues.
21

---

Published as a conference paper at ICLR 2026
OURS-PAST.
OURS-FAST
pensgrlaenshttee ponoeptionocooom persegtionn jonbs Ad)
percertion Wer 2017 10, 76,
1307.
200) Viase"" Viee/Wii""
perception:
316, 049) 448, 204), VTute actionn Vatnagho//ele wifh' Sarveol tbox 201
20t Jomiehhiet 20 1D,
10.0 oz Viaaett aet) hate 201": (o 0.0). Vidan" 17, De
Tabalt" 2e13 ynlo ady: (0.0), o mg) Vidue"" on witt puhrt 201"
a AMt (0.0. 0). Vhahell 111 Ball.wo 201 (0.0 0.
1,49) Vaasdll 17d Sat1 wiit past AAV (0. 04, 2,66) Visell" 11 0617 with peant del wan
Gpeat ist": (0.0. 0.0). V'abel"" V12 "Tiil"
Ground Truth
Ground Truth Ours-Detailes JOurs- -Fast 'OniiDrive
JOurs-Fass -ast
OmniDrive VAD
OURS DETAILED:
OuRs DE T
everiie the image for bay dbjects auch as cors. bunes, be sighttine, and the road is vioble with come greanery the side viltires
perception Lat's comnbe Imoge for key objects nach care, bates, trucke bicycles 44bu292
the image need to detereise the coer disotte of each eack vuidd The first tack is on the arcund bounding coerdmates to be crezd (300, 100) (646, 180), tnkahlee. let's comple this information into
reed ideniiyy objects secadd
clerrly right halebaaet nead
image estreahe beseding cardeates side
image
Sele2 The
trak
Ueternine the coneiiattes these objett. de will cshrett the loueding baxes
beanding
bor comrdaates (1at. 124, 000. 240) wiid For secord car,
the
wil Bhos JaV 10, 64, 175, 250), Videll" Vlrnck' wiih perception
right sida, actimatt the banding hes coordhutes po09. 114, det, 240) hat
bbbox 2014: (313, 89, 425 182), Visben Virac "ragle
peocapicn jurletiett boot 201: (182. 126, 315 2001 VTabelV" V
then We have the muce the scass, The firet truck, -trudesto, 64, movig directin
" WWMM toa [810, 127, 446, 240). Vieeel" Vearl
crrrsss/D stattorry and stotiunary
ego vehilee second Since
beth taad stationery and there are to traffic lights the foture oction for beth tracle is to Vitte,
fist scor L25, 515, 250), currently ebgecte,
the
draction ago whicl sacind cae ccaro Sanee cars the atationary and there traffic lights the fonure oct lor for both cars to
200), she
peetegveee tho 201 (0, 179, 2007 MMunur oction): Vittpp woyhbbo 20%: plll. 425.
Morte junn bhoo 201 [182 126, a15, 2011
Let think The vahide's correst speed is 0.60 mn' and its future action go straggh The akttrkke kool tegjectary shows consiteer thiight patts dayet latend con
futur artint VonpyTialw
egs postton secaed intereds for the plamnng trajernery wi cololate the closs
and direciio WisTo
seconds buoee vehide moving
I sec. The ego vchici is carvently at 0.0 nv's, and the Hanorica trojectory
on the speed weiel. At 0.5a The weicc moves 0.60 ey's 0.5% 33 netura forward.
the statinery effect the velice. Since
ege vehide
velicle nove additional más 1.0. 0.80 openainnttll
sight
Adjustnng for lnteral
dinodly stonteary
ochir stop the ploned trajectery renses
the ancanddatte oppooieettly o2 Vad
moves
yelldatte the position of each 0 3-seooed interal for the rent seoress Sinee the
0o vet at do The whicle meees arditioed 0.66 2 Oo 1 neters Anjattnng deviction coandrate apprasinatlly
vehice ststeeary. and
will senah at leves''s caludate the
(0.0.
At De (0.0. 6.00e- At t Jo.0. o. at Gel (D.o o mpernin planingss
the s-coardaate
0.05 ted at oe The vahide moves addricaal Adpaiing
deviation
jontettee baiet t o.a Vasaery vi sel' wen
jonn be trojectory felowe
ppar aut (0.0 0.0 Vleueh 11, oil (wil boott 201": (0.0. 0.0). Vidae's 0. 0.00 Vieeld V2 SeV) pew. hel (0.0. 0.6). Vibelt 19.001's YOw"
Viasely vo 5.15 WWl pein An
AN (0.00 2 99). 122 Del weat de1" 10.00. 4 4.899 Vianel""
20V) lot pain 201 o0), Viaber" "e1ioh
(a)
(b)
Figure 11: Visualization examples of specific samples. Every example is equipped with visualiza- tion of results on the top and CoT answer on the bottom.
Except for the specific explanation of the samples, we also provide special cases with completed
questions and answers in Fig. 12 to Fig. 17 The first two are nuScenes samples and the last four are NAVSIM samples.
22

---

Published as a conference paper at ICLR 2026
OURS-FAST FAST:
tthink perceetionn //thnn perception
Ieonswer perception " ioontana 2d\": [260.
320, 220), Vlabel\": "ppedestrian"". In((()
"bbox_2d": [286, 130, 352, 184). \'labe\V": \'cal
nJ //rswerr perception>
rhinnk prediction /think prediction
konswer prediction [sontn( 2d)": [260,
112. 320, 220], Vffutrreaccto Lcctionn:: \"straight\ In)r()
straight straight.
"bbex 2d1": [286. 130, 352, 184), Vfuuuure action)
:: \"straight"/V/J">
</onswer. prenee_preciiiooo..
iethink planning </think planning?
kanower plamings" WoonWWWWiY(Woooon___... 2dV":
SINCS
0.0]. Vllbel\V: 1"0.5\1") (n1d((1 2p [0.0.
0.06]. VlabelV:: 1"1,0=1 Initf) "point 2dV":
"ooint_2dV:: [0.01.
0.33]. Vlabely:: 1 "s17, Inir(("poin point 2d)": [0.01.
0.93]. Vllbel\V: 1 "2,0s11). Unlr[V'poin 2d\": [0.02.
1.81]. VlabelV:: 1"2.5r1 Inint)"point 2dy": pointt_2d [0.03,
... GT ... Cwwur--Draalii ..-Ours-Fast .'-OmniDrive V'.AAA
2.89). Vlabel\": O=9/d:'
//answer planning
OURS-DETAILED:
<think perception>
Let's exomine the image for key objects such cars, buses, trucks, bicycles, pedestrions, and motarcycles. First, notice a pedestrion on left side of the image. need to determine the bounding box coordinates for this pedestrian, The pedestrion appears to be standing on the sidewalk, so the bounding box should cover the area from the top of the head to the the right side of the image. It smaller and further away. the bounding box for this ccr will smaller and positioned towards the center-right of imageNow. let's determine the coordinates for each object: Pedeestrinn:The pedestrian is on the left side of the image. The bounding box coordinates are approximately from (100, 120) to (200, 180).
bottom of the feet.Next. I see
Car: car is the right side the image, The bounding coordinates are approximately from (300, 150) (400,
These coordinates are based on visual estimation from the image.
Let's compile this information into the required format.
k/think perception>>
canswer perception>
["bbox_ 2d" [258. 112, 318, 225), "label": "pedestrion"). ("bbox_ 2d": [280, 132, 356, 183). "label": "car")
//nnswer perceptions
Kthink prediction
Let's analyze the future actions for each key object:
1. Pedestrian: The pedestrian is currently going straight and is not in the moving direction of the ego vehicle, The future state is affecting the ego vehicle, the future action remoins "straight."" " 2. Car: The car is currently going straight and in the moving direction of the ego vehicle. The future state is also to go straight.
also go stroight. Since pedestrian
Since the car is not affecting the ego vehicle, the future action remains "stroight."" "
Both objects predicted continue going stroight there k/think prediction>
indications of change their trajectory behavior.
kanswer prediction>
["bbox_2dd 2d" [258, 112, 318, 225), "future action": "stroight")
['bbox_ 24" [280, 132, 356, 189], "future_action"" "stroight")
//onswer grediction>
kthink planning>
Let's determine planning trajectory for the ego vehicle:
1. Current Vehicle Speed: The ego vehicle is 0.0 m/s, and the historical trajectory shows a slight forward movement.
2. Ego Future Action: The
vehicle is to go stroight.
Trajectory Colculation:
The ego vehicle will accelerate from 0.0 m/s to a positive y-direction over the next seconds.
The trajectory points are calculated based on the acceleration and the time intervals.
Trajectory Points:
At 0.5s: The vehicle moves slightly forward, [0.0, 0.02). At 1.0s: The vehicle moves further forward, [0.0, 0.17]. At 1.5s: The vehicle continues forward, [0.0, 0.44]. At 2.0s: The vehicle moves slightly to the right, [0.0, 1.13]. At 2.5s: The vehicle vehicle moves slightly the right, [0.01, 4,23]. The trajectory reflects a gradual acceleration and forward movement.
moves further forword, [0.01, 2.59]. At
planniin
kanswer planning>
"point_2c0 2d" [0.0, 0.02), "label": "0,5s"). ("point 2d": [0.0, 0.1), "label": "1. "1,0s"), ('point 2d": 0, o, 0.44), "labe!": 56"), "point_2df [0.0, 1.13], "label": "2.0s"). ("point_2d": 2d" [0.01, 2.59), "label": "2.55"), ("point_ 2d": [0.01, 4.23), "label": \3.0s/
planning>
Figure 12: Visualization nuScenes cases with completed questions and answers. AutoDrive-Pa successfully identifies and localizes the key objects, giving the correct actions. Based on these judgments, our method makes the efficiency planning decision in this sample.
23

---

Published as a conference paper at ICLR 2026
OURS-FAST:
<hhink perception- hatpiiinn//tiiiicpopttttoo
kenswer perception>
Veonbatwit(Vbbbo._.. [192. 129, 273, 1977.
V'ar\""lnJ'"" "M
</nsswer perception>
think www.eeeeedddnccoooo s/tiiinn>//tthkknoe prediction
predictions
Joonwnnntt(""bor 2d\": [192, 129. 273, 197). A
future action)": Vistrightt\VW/
stram gnt
</onswer prediction
</think planning>
planning:" json\n[[n\t(("ponnt 2d\": [0.00 [0.01,
1.1]. Vlabel\": 1"0,5s17, Iniif("poont 2dV": [0,02.
2.09), VlabelV": |"1 O01"),"",, pniinttpoint.... 2dV": [0.04,
2.971. VlabellV: V"1 "1.5s17], Int.t\ppointt........ 2dV": (0.06.
89]. Vlabell"; (22,s1"), inir(("poimt palntpoonn_t_lll 2d\":
4.76]. VlabellV: 122.5s1"), AMM(V/ointt_dV-" [0.1.
GT Ours- Detailed .--OursFastt .'-OmniDrive V.D
5.6]. Vlabely:: V33.s//Ww/'"
planning?
OUUSS DETAILED:
kchhin perception>
Let's exomine image key objects such as cars, buses, trucks, bicycles, pedestrions, motorcycles. The image is pixels wide and 252 pixels tall.
First,
the image. positioned slightly to the right of the center of the image.
car in
determine bounding coordinates. car oppears
Next, I estimate the bounding box for the car. The top-left corner of the car's bounding box seems to be around (200, 120)
pixels. bottom-right
(250, Now. I have the bounding box coordinates for the car. I will label this object as "car."
appears
pixels.
I don't see other key objects like buses, trucks, bicycles. pedestrians, matorcycles in this frame. Therefore, the only detected object </thhnn perception
the specified bounding coordinates.
crnsser perception
["bbox 2d" 129, 198], "label": "car")
k//nswer perception>
predictions
Let me think. The key object in the scene is car located at [194, 129, 272, 198). indicates that
currently going straight and Additionally, the car's 's future state is predicted to continue going stroight. traffic lights and the not affecting the ego vehicle, the prediction for the car's future action is
video
the moving direction the ego vehicle.
Since there
straightforward.
w/thin prediction>
konswer prediction>
['box 2d" 129, 198], "future_ actiion" "straight")
//nnswen prediction>
Khink planning?
Now, let's consider the planning for the ego vehicle.
The current vehicle speed is 2.18 m/s, and the ego future action is to stroight.
The historical trajectory shows consistent straight path with slight lateral deviations, indicating minor adjustments maintain
lone.
1. Ego Vehicle Speed and Action: The ego vehicle is moving at 16 m/s and plans to continue straight. The historical trajectory shows consistent forward motion with minor lateral adjustments.
2. Trajectory Calculation:
At 0.5s: The vehicle moves forward by (2.18 m/s 0.5 1.09 m). Adjusting for slight lateral movement, the x- x-coordinate is slightly negative.
1.0s: The vehicle moves an additional (2.18 m/s 1.0s 2.18 m).
At 1. 1.5s: The vehicle moves additional (2.18 m/s
1.5s 3.27 m).
2.0s: The vehicle
additional
m/s
2
4.36 m).
2.5s: The vehicle moves on additional (2.18 m/s 2.5s 5.45 m).
At 3. 3.0s: The vehicle moves additional (2.18 m/s 3.0s = 6.54 m).
Final Trajectory:
trajectory points calculated based the
slight lateral adjustments for lane-keeping.
k/think planning>
konswer planning?
["point_2d 2dt (0.01, 1.091, "label": "0..s"".(peint__a" 2d" 10.02, 2.16), "labe!: "1l.os""""poiit___" 2d":: 3.03], "lobe!":
"..5s",."'oint__ec [0.06, 3.7]. "labe!": "2oost)_"ppinn__2"" [0.08, 4.08), "label" \22. "22.bs\".."ppont__e [0.09, 4.43), "label": "3.0s")
planning?
Figure 13: Visualization nuScenes cases with completed questions and answers. AutoDrive-PP successfully recognizes the driving command and provides the best trajectory instead of conservative one compared with other method.
24

---

Published as a conference paper at ICLR 2026
OURS-PAST,
-hhnkk perception--//iie //thnkk perception
commerr perception jenW 2dV:: [17. 78, 153, 139]. Vlabel!": \"ehiclel Inlrti"bbbxbx.xe 20y": (316, 65, 367, 120]. Vlabell
"wehicle PaTen" //onswer peroeptiroonn tion
prediction
prediction jsen Donebro 2d\" [17. 78, 153, 139). "'future_posrton!" position!" Vffar, fromt Vnle(Vbbb 2dV": [316, 65, 367, 120].
Wuture_pesitioon" Vffar frontV/vajvn" //onswer predectiionnn
khhink plonning </lemk platrins
planning" jontnnWt "y_diinion (8,28, 0.05, o 02). Vlabett:: 10, s.1") wiff Sofffyyyyrr yuroiinn": [10 55, 0.15, 030, V'lbalVV: \
1 0. WA y rodion'": [15. 75, 0.32. 0.04]. VlabelV": 111 5.1. Intetix y. "o [20 65, 0,55, 0.05), \label' O22.0s17,.n Intef)
x y. rodinnt (25. 75, 0,78. 0 3,05). Viabeli: 1 (2.5 int y radion": (30 35. 1,01, 0.05). VlebelV" 1'3 13.0017.
rodian) (34 55.
1.22, 0.05). Vlobell!: 1'3.5a1)), ww ryrrolonn [38.25, 1.39. 0.04). Viebel!": V14.ii
</onower plorning
BOMS
Ours- Detallee
Ours-Past
WOTE
DiffusionDrive
Ground Truth
OUURS DETA
perception>
vanswer perception>
analyze the image step by step to identify key objects and their coordinates: Wins, *IImage Dimensions"*; The Vehicll iVn
(annee( 2 [13, 78. 139). Vi
image size is 672x168 picels.
OO the left side of the image, there a vehicle visible Coordinates: [13, 78, 152, 139]|n In Coordinates: [316, 53,
on the road. Based on itss position and sze, it appears to be a car.in In
(Veehcie"" lntef)
the center-right portion of the image, another vehicle is present, likely a truck or SUV An No pedestrians or bicycle are clearly visible in the image. An The background includes buildins, polm trees, and a Lous Vultton store sign. but these are not cottegrrized as key objects for the given question IntwFinal Ou joon (WW((Wb 2d\" [13, 78, 152. 139), Viabell": V'hehicl 1"), ln) "bbbbb 2d\ (316. 53, 369, 120). VlobelV": Wehidee/"nniin" "nlnneesooning The analysis confirms the presence of two vehicles in the image, with their respective cosrdnates accurately identified. No other key objects (pedestrions, bicycles) are detected. think perception>
"bbox 2dV: (316, 53, 369, 1201, Vlabelt:: l'vehicle "Tujj""
369, 120j "Other Objects**
</orawer perception
rthink predictions
predictions
**Initiol Observation":: The
in the Movement Analysis**:\n
vehicles: the left side the road anather vehicle the left center-right
vas Wnnnnnn adV [13, 75, 152, 139). position'" Vfaar frant Weok 2d1": [316. "future position": Vfaar, front Vront //nswerr
forword appears slightly lone. lnz. **Vehille
further ahead the
left of the ego vehicle's path, In positioned forther away
shead ego vehicle An3 "Fuuure Fesition "
moving
on the the prediction maintaining Its relative position
the moving forward cheod, the prediction suggests
sightly
left.
120].
ego vehicle.An moving forward along current path, remaining the center-right lone. wt. ""Concussen"" Based on current key objects
continue
positions and movement patterns, the future positions No significant changes direction speed are anticipated for either vehicle. </think prediction
consistent with their current trajectories.
iconsmer prediction=" joentnnntt(Vbbboe 2d\": [13. 78, 152, 139). Vfuture position!" Vferr front Wel(() Vel bboox__ 2dV: [316, 53, 369, 120), Postioeposition's Vffar, front VNeeji" predictions
tthink platning>
conswer planning
1. "EEgo Vehicle State"": Vn Currentt velecity (us): 10.85 m/s, vy: 21 m/s, ex: 0,20 m/s 100022 ay: O 27 m/st
u00b2. In Driving Command: 60 STRAIGHT. An Historical Trajectory: The ego vehicle has been moving forward
"x rodian'": [5.2 0.05.
with dight leftword you angle, indicating minor leftward rejeccooy over the past 1.5 seconds. AnZ. "Kay Object
0.02). V'lobel V'0.5st").
considerations""
**Vehicle the Left This vehicle predicted move forward and slightly to the left. The
wtnEV roddaan": [10 49,
vehicle must maintais safe distonce aveid collision Va "Vehicle Ahead"* This vehicle also predicted
0.18. 0. 03). Vilablly::
forword along its current path. The vehicle should continus streight to evaid patential collision. \ns.
"I dry") Wn radiont
"*Honmiig Points (Every 0.
forward with slight leftward angle. The slight leftward you angle. The vehicle adjustment needed An ang Oghn: The center-right
*0.55*h The vehicle
[15.68. 0.36, 0.04)
vehicle the left still to the left of the ego vehicle's path,
"tobelV: V1 (...55 Anitts
vehicle continues straight
"Kddanddi [20 0.56
adjustment
continues straight the 5e ego
maintains its trojectory with slight leftward angle. The adjutmentt
Vilaell::
the
**2 = chead remains center-right lone, so ego vehicle maintaies trojectory ego vehicle's path, so adjustment needed. righ lone, so no
ego vehicle's path,
ego
AnVEY's y radian
slight leftword yow ongle.
(25.18. 0.76. 0.04]
adjustment needed.
yow angle. The slight "*3.0""" The ego
"labll" 12. 52)5,)") "Kddannd" [29.68. 68, 0.94 04), Wiabel
vehicle on left is to the vehiccle continues straight with slight leftward you angle. The vehicle ahead remoins ege vehicle maintoins its trojestory with a slight leftwrdd you angle. The
"" " vehicle on the left is still to the left of the ego vehicle's 's path, so no adjustment is needed, In vehicle continues straight with slight leftword you angle. The vehicle shead remains the renter-right lene, so no adjustment needed. An4. "*Coccussion"" The ego vehicle can continue noving stroight with slight leftward yow angle,
edjustment needed.
"3 a. witt y. radion
Or**: The ego
(33.98, 1,1, o 04).
"label" 53y5 With
"x radian'' [30. 08, 1,26, o 04), Viabely"
os there are no inmediate collaion risks with the key objects, The trojectory is adjusted minmally to mointain a sefe distance from the vehicle on the left while continuing to follow the driving command of going straight.
t//tiink plaming:
</onswrr planning
Figure 14: Visualization NAVSIM cases with completed questions and answers. AutoDrive- P3 successfully predicts the future action of the truck and follows the forward vehicle in a safe distance.
25

---

Published as a conference paper at ICLR 2026
OURS-FASTI
irhhnk perceptionn pat.ppeccoonnn.c.cccccooooo penception
conswer perception." " WoonNneennteeeobobooooo 2dV: [42, 84, 215, 1671, V Vvrhicle wntof(/booo__dd"" (413, 85, 436, 1211, Vlabel\":
ppedestrian Taajiv" "/onsmer perception
krhink prediction /thik prediction
conswer prediction lon jontofinsf("bbbor 2d1": [42, 84, 215, 167].
Vhear,
201": [413, 85, 436, 121].
Muture positionn Vnear, right\"\nJinn </onswer. pww.cccccooooo
kthink plonnings W/hiik plaining:
votteee plannnng " "oonne " roddiaan" [2,51. 0.15, 0.12) Viobelly: VO vnt.
Ant rediont" (3.43, 0.59, 0.231, VlabelV: 11 On
VeV y-rdian" [8.65, 1.99. 0,313. Vlabelly: (11 wnWn'' vodian): (12.15, 2.45, 0.951 Vlabelly: 12,0 Vnninn) Inirf'' radiont:
115 85, 3,75. 0,381, Vlabell": V2,5
radienn" (19 55. 5,15, 0.39), Vlabely" V3, 133,0 An radiann" [23. 6.55. 0.99].
Tabelly V3.55V SnttttXX_y__dooooo [26.4, 7.88, 0.39], Vlobely': 144 "4.0s//vvii"
c/onswen.
F
Durs-Detailed
Ours-Fast
WOTE
DiffusionDrive
Ground Truth
OURSS DETABLED
perceprion=
corowen perceptions
Let's analyze the image step by step to identify and report the coordinates of key objects/welnl. "Vehicle
"oonnnittttttboo... 2d0
Detection"" \M
white SUV visible on the left side of the image, acoupylng significant portion of the frome. In
[39, 85, 215, 167],
The bounding box coordinates for this vehicle are opproximately [39. 85, 215, 167]. intn?. "*Pedestrion Detection"":An There is pedestrian walking
"labell" Veehicle Witeyy
the sidewalk near the rrigh of the image. in The
"bbox__ddV [413, 84, 438,
bounding box coordinates for this pedestrion are approximately (413, 84, 438, 119]. 1nin3. "*Other Objects** No bicycles or motorcycles are visibe in the image. in The background includes buildings, tress, and street signs, abjects. Vnliiinnl Output:hnn jontnn 2dV": [39. 85, 215, 167]. \
119). Vaabell \ "pedestrian"
but these are categorized
/onswer tion>
"label"" Veehicle))), (nboo("bbox 2dy": (413, 84, 438, 119). V'abbli": V'padestrian""ini) w//hh/ii perceetoo
prediction
predictions
"Initial Observation":; key objects image vehicle
the vehicle laz "*Future
20y
ego wehicle, Vehicle**: The prediction indicates that the
the right
[39. 215, 167).
remain right to collision ego vehicle, likely
side ego vehicle, suggesting ego 1n3.
position'": Vhner,
continue moving forword may slightly adjust its position
lefti"), Antef) "bbox.
pedestrian predicted move near the right cressing street walking along the sidewalk. 144 "No Key Objects Detected for Other "ositions""; Since no other
""uutur Pasition "
[413, 84, 438, 119],
"future position" Vhear. right\TWWJ'"
key objects are identified in the image, focus remains prediction
and pedestrian.
+/onswer prediction
cchink planning>
conswer planning>
"EEo Vehicle State""; The ego vehicle is currently moving forword with velocity of 58 m/s and on acceleration
"jontafinttt)
of 1.66 m/stw00b2. The driving command is to turn left. and the histerical trojectory shaws consistent forward
*x_y rodion!" [2.48, 0.1.
mation with slight leftward adjustment. \n2, **Plamiing Points (Every 0. Sn -0*t5*th; The ego vehicle moves
0. 0.06]. \'labelV: 5'
forward and slightly left, maintolning safe distance from the predicted future position of the vehicle on the left. The
nit[Vx radian'"" (5.25.
pedestrian on the right is still far enough to avaid collision. In "*1 "000; The vehicle continues its forward and
0.45, 0.18). V'label" \
word trojectery. further adjusting its path to maintain safe distance from the whiicle on the left. The paceestrion
"1.00 Dal") WAVIVVV radian
remains on the right, and no collision risk is present, in "1,, The ego vehicle's path is adjusted to ensure
[8. 18. 1.09, 0.271
safe clearance from the vehicle en the left. which also vehicle's is claser. The pedestrion the right still safe distance. An
forword, The pedestrian continues walk along the
"labell": V1. Wnln
sidewalk, maintakingg safe distance.
collision with the vehicle the left, which *22000t: refined the right, and collision
fine tuned avaid any potential
"x rodien (11.1,
0.321 Vlabbllv: V2.051",
sst The ego vehicle's path
safe passage present.
the vehicle
left, which
rodion':
directly
"3 m The vehicle
2.98. 0.351 Vlaben"
pedestrian on the right and leftword trojectory. maintaining sofe distance
whicce continues
The distance. In **3, 5yth: ego vehicle's path adjusted to ensure safe right,
-2.5e WWWV
11..8
0,371.
ahead. The pedestrian remoies cleorance from vehicle "r" ego vehicle completes forword and leftward trajectory. ensuring a safe distance
now
collision
"label" v3.0ss"" Wnifh
present
"x "od'nn" [19.6 5.2,
safe distance, Throuughut the plarning precess, the ego vehicle's trojectory is continuossly adjusted to maintain a safe distance from nWV' y radian (22.4
from vehicle
left. pedestrian on the right
no collision
present Ael
0.371. V'abelV (3,
the vehicle on the left, while the pedestrian on the right is consistently monsored to ensure no callision accurs. The 6,33, 0,371 Vaabel 4.0s
odjustments are minimal, primarlly invalvig slight leftward adjustments to maintoin a safe clearance from the vehicle on the left.
/onswer. planning>
w/think planning
Figure 15: Visualization NAVSIM cases with completed questions and answers. AutoDrive- successfully locates the key object on the left and take appropriate lane change action to move forward, while other methods provide a wrong trajectories turn right.
26

---

Published as a conference paper at ICLR 2026
OURS-PAST.
condwer perception Wonnv 20"" [0. 74, 175, 167]. Vlobel": Vivehile( Anif(Vbbox 2d\": [162. 88, 246, 162). Vilbell":
"wehicke)) (nWie Nbbo 2d)": [428. 76, 493, 122). Ylabel\": ""pekestrian"")// </onsheer per.ppptooooon tionn
cthink prediction
prediction Setevvnniiivbboe.ae 2dv": 74, 175, 1671. V'fotun_ppsstioon Viear, left1"), bnttrttboooo220" [162, 88, 246, 162).
ffuture_ p Vfa. left)"), bnbttt (455, 76, 493, 122). Vifituneppaitton"" Vhear, right/Wnnlv"" c/onswer
planing?
plamings
radian'" 0.11, 0.31, V'abell:: VO. vo.5511). intttt y yurdiann":
0.19], Vlabelt:: Ouy 0,25). V'lbbllY 122 vn y. rodion'
Xn rodion' : [8.51, 115.75 2,89, 0.27), VllbellV: 1 betttt"xy_rrddeere 119.79, 3.95, 0.281. ViabelV:: V.3.0s1 Antt (23.95, 5.09, 0.29).
0,23], Vlabelt": 5xV7, WAl redian)": 18.99,
lobel Y33.5s1),, intefl Y__ rodion""" (28.1, 6.25, 0.29], V'lbbll": 14,0s17Jviin
c/nnswer planning
Detsllet
Ours-Fast
WOTE
DiffusionDrive
Ground Truth
Let's analyze the image step by step to identify and report coardinetes key objects:Wninl **Vehicle
pirception>
"Joonin 2d1"
perception
Detection"1:1 (\n The image shows black vehicle on the left side frone. It appears be van SUV In
[0. 70. 175. 167]. Vllbell
Coordinates: [0, 70, 175, 167]In Another vehicle is visible further down the road, closer to the center. In
V'vehicle Wiif
Coordinates: [160, 87, 246, 162]in'n2, "*eeeestioon befection"=:\n Coordinates: [456, 77, 493, 121]Winn. "Otthe An
A pedestrion is walking on the sidewalk on the
"boox_ "boox_adV" [160, 87, 246,
right side of the frome. An
There are no bicycles or natorcycles visible the image. in The background includes buildings and traffic lights, but these are
162]. Vlabeli" Vieehille
"WrrYVbbbx 2dv; [456,
not categorized as key objects. AnVaFinal Ou joon jeenWWW 2di"; [0, 70, 175, 167]. Viebell.: Veehicee)
495, 121). Vlabel
"J, VeVr(Vbboxx 2d [160,, 87, 246, 162), Vlabell": ('veiicel"),, vetn((/bbx__22V 2d1": [456. 77, 493, 121), Vlabell":
ppedestrian
"peeestrion
/onsserr perceeption
/hink perception>
conower predictions
""Jsonnn "jsonin(nnit "bbox__dd 2
rhiik prediction>
[0. 70, 175, 1671. \ "future position'"! Vnear.
"Innttol Observation":: The objects in the image include black vehicle the left, another vehicle further aheed and to the left. and pedestrian the right side of the street. Vo?, "*Future Position of key Objects In
lefty "bbot 2dV"
black vehicle predicted
left its the left.An The pedestrion predicted The vehicle ahead and and the right.
[160, 87, 246, 162], \
left predicted to the curnent positions are likely to continue moving forword with slight leftward trojectory. pedestrion expected move along
"future posstion"": Vifar.
"*Resscning"" -M:
patha, the black vehicle and vehicle
left)) Wodoo 2dv"
[456, 77, 493, 1211, \
sidewalk right.
"future positian"" Vmiddle, right/TWW"" prediction
planning>
plarning
1. "EEg Vehicle State"*; The ego vehicle is currently moving forward with velocity of 4.72 m/s and on acceleration
of n/sw20022 The driving command
turn left. lnž. **Histerical Trajectory The historical trajectory shows
"x radian'": [2.4 0,08,
consistent forword motion with slight leftword adjustment over time, 1n3. **Future Plarning Points"":An
0.08). VllbellV: 5'0.5.17,
55055 The ego vehicle will continue noving forward while initiating left turn The pedestrian to the right, 50 no
n\r(V' y roriinn"" (5.15,
immediate collision risk exists. The black vehicle ahead also moving forward, maintaiing a safe distance. In
0.4. 0.171. Vilaell"
te] Or**: The ego vehicle continues its left turn, increasing the you engle. The pedestrian remains to the right, and
"1.0s'"), \nitt)x radians
the black vehicle ahead is still at a safe distance. The trajectory adjustment ensures no collision with the padestrian. in
":
0,98,
**1, "*","""" The ego vehicle further executees the left turn, with the yow angle increasing. The pedestrion is now further to the right, and the black vehicle ahead maintains a safe distance. The trojectory adjusted to ensure
"label V'1.5117
"x rodien" (11 25. 1.78. 0.25), Vilbelly
smooth turn witheut collision. in *22,0*", The vehicle continues the left turn, with the you angle appeaching
0.3 rodians. The padestrian is significantly the right. and the black vehicle ahead still
safe distance. The '2, Witt roddon
trajectory is fine-tused tuned to eveid potenttal collision. Vn *x2 The vehidle maintains the left tunn with
[14. 2868, D 2.26.
0.33 radions. The pedestrion safe distance. The trajectory adjusted
the you
further the right, and the black vehicle ahead still Ang 0r*0 The vehicle continues "labll" "2,5 Anten smooth and safe fum. considerable distance left with the yow angle 0.34 radions. pedestrion is trojectory fine-tused rodions. peckstrrin distonce. trojectory adjusted
y radion) [18. 1, 3.66,
right,
0.26]. Viobel 13.0s/7, A
black vebicle ahead still
(21.78.
""" Sy ego vehicle maintains
ahead you ongle
4.7. 0.261, Vilbell" \
significont distance turn. right.
"3.5 "3.5eV7 y redion
ensure
ego vehicle completes right, the black vehicle ahead
turn, with the you angle distance. The
The
" [25.6. 5.78, 0.26 \.
pedestrian is now af a consideaable distance trojectory is finolized to ensure a smooth and safe turn. Velne. *Callision "vaidance Throughout the planning precess, the ege vehicle's trojectory adjusted maintain a safe distance from the pedestrion and black vehicle ahead. The left furn is executed smoothly, with continuous monitoring of the pedestrian's 's position to ensure no callision occurs. In'n5. "Finnl "djustment*:; The final trojectory entures o safe and smooth left turn, avoiding any petential colliionns with the pedestrian and the black vehicle ahead. The trajectory is aptimized to maintain safe distance and exccotee the driving command effectively. //think planing?
"labelV: V4.0s%
/onnser plamning
Figure 16: Visualization NAVSIM cases with completed questions and answers. AutoDrive-P successfully locates key objects on the left and take a appropriate lane change action to move for- ward, while other methods drive into an illegal driving area.
27

---

Published as a conference paper at ICLR 2026
OURS-FAST,
xtthink perception- //tinkk perceptions
consser perception jenVe(Wi)H(Vbboo_aa 2dV": D. Vlabal\": (WMj"" " Tanswer__pereeghiin
ethink prediction
Teanswer prediction joeletWnliiivooexeea 2dV": "astunc_aattton" WWnT"" w/asswer__pedccceeee
(ehhnnk planning
planinng
plamnigs jonte S-l-y-reee r"dian!"
Vidball:: 5017) Anld" Ma redion": (3.15, Vlabaly: 1 O
".WW rodian [5.25,
Vlabell:: In Wntfii rediant" [7
Vlabely:: 122 02.0.1 Inif))
PLY Y ": [9 -1.02, -0.55) \'lebll) ("2.5e12, redion"; (12.1.
-0,55), V"label"::
rodionn""
113.98. -3.11 7017 Vaabbll V.3.51 Intt)" y_-reiinnn [15.12, -0.35 -1.021 Vlabell": V4.0:11WW"""
Ours Detooled
Curs-Fost
WOTE
DiffusionDrive
Ground Truth
OURS-DETATLED:
<think percsption
conswer perceptions
Let's analyze the image by identify key abjects-Intnit ""Image Overview"" The image wide-and with buildins, the road for any vehicles:
Wennlw 2d1
shot
The appears with
D. Vlabell DiJ </onswer percoptionn
immediate signs actiityy
types vehicles "Veehicle Detection""An
trucks.
road,
shoulders for
No vehicles siuble walking sidewalks bicycles
present Vein3, Pedestrion Checki stanning these areas. 1nin4. Bicycl//Moteeyccle Check: Look along motorcycles: None are present in the frome, Wnlnd. Fiina Assesomen"" After thorough exomination of the entire image, there are vehicles, pedestrions, or bicycles/matorgee totecycles detected viln6. "*Concllsion"" Since no key objects from the specified categories are visble the image, it concludes that such objects do not exist in this particuarr frome. In jtonkn[WnW 2d\": D. VlobelV:: [DM] "WieNo key objects detected the image. </trink perception>
sidewalks pedestrion
People
prediction
conmer predictions
*Iiitiol Observation""; The image is a wide, panoramic view of on urbon street intersection with modern buildings, a rood intersection, and some greenery. There are no visible key objects (e.g.. vehicles, pedestrions, other sigrificntt moving entities) in the current frame. Anin2. "Key Objects Andlysis* Vn- The provided prediction snswer indicates rhht there are no key objects detected in the image. This means that there
(oonin(ineel''bbbx 2dv
D. fuutuee position'": (D)
other vehicles, pedestrians, or , significant objects whose future positions need to be predicted. Vele3, "*Future Pesition Deterninstion Since no key conclude that there Therefore, the future position for other key objects not applicable.)
</onower prediction
objects are detected in the current frome, it is ressonsble could influence the vehicle's in'n, "*Conclusoo"* Given the absence of detectable key objects the image, it accurate to state that no future positions need obtained other key objects.
dynamic elements in the scane
////hiii predictionn
plannings
ronswer plannings
"EEg Stateh*; The ego
currently moving forword velocity 2.14 the x-direction
y-drrcction. driving command is acceleration historical trojectory right,
negligible y-velocity 0.04
m/s/w0002 in x-drrcction and 0.14 m/sw00022
"x_Y rodiann" (1.36, 0.0,
a consistent forward motion
-0,011, V'label"": 110.55
slight rightword adjustment VolnZ *PPanning *oints** points every 0.5 seconds for the next 4 seconds. Each point includes the ego vehicle's 's position and orientation maters
plarning trojectory is provided as a sequence of
08.0. -0.03, -0.041,
redians InVe3, "*Trojectory Analysss*:\ rightword turn, as indicated by the increasing negstive you angle. An **1 *1.0g*h The vehicle continues its forward motion with more pronounced rightward turn, aligning with the driving command In *1I 5 The fum becomes more
M0 5 The ego vehicle moves forward slightly while initiating
"labeli": 0w1""
"x_y rodiant": (5.06.
0.16. -0.12), Vlabell":
significant, and the vehicle is now clearly excculing a right turn, with a noficeable charge in direction. Aa ""
*1 b.Bay), watxxx rodiaa
The vehicle continues olong the planned trajectory. maintanning the rightward turn. with the yow angle further decreasing to reflect the ongoing turn. Ve- *22 5 The turn is nearing completion. and the vehicle is cligning more
17.15. -0.5, -0.25),
dligning
"label": v22 Owy Wil
clesely with the intended path, indicated by the decreasing you angle. In ong "*3.0ghh: The vehicle is almast fully turned.
"x radian': (9.2, -1.2
with the angle appreaching the final value, showing smooth transition to the new direction. An- **3 a The
-0.45), Viebel V2.
vehicle completes the turn, and the you ongle stubilizes, indicating the end of the furning maneuver. An vehicle continues clong the direction. with the angle maintoining the final orientation, An/v4, *CCollisinn Avoidance Consideration" Since key objects detected the image, there other vehicles pedestrians. The trojectory straightforword and does not require adjustments based other dynamic objects,
"0.0 The
Anni rediont"
-2.3. 63),
immediate risk of collision with
"laen V3 0.10
*x y t:: (12
" "Adjjstmennts the sesccon period. The increasing negative yow trojectory points consistent with the given driving trajectory are
Validation** The trajectory provided dligra with the driving
-388 Vlabel":
right smoothly
execution the "3.56 wilx rodiinn) and trajectory ensures smooth tronsition provided trojectory effectiely guides the ego
confirms the
safe and controlled "MFinal theck points vehicle's mation, considering the current speed acceleration. intended direction without abrupt changes. \nle7 "Conclusion" vehiile to execute smooth right turn over next seconds, each point reflecting gradual controlled change in direction. The trojectory eligns with the driving command and the wehicle's 's current dynamics, ensuring safe and efficient moneuver.
[14 -5 65, 02),
vehicle's current state,
expected range
"label" 14 Wwj"
</onswer planning
//think planning
Figure 17: Visualization NAVSIM cases with completed questions and answers. AutoDrive-P successfully recognizes the "Tum Right" command and provide the right planning trajectory, while other methods drive towards the building leading to collisions.
28

---

Published as a conference paper at ICLR 2026
G PROMPTS
In this section, we provide the completed and specific prompts used in the training/inference and prompts used to generate PS-CoT dataset. The prompt used in the training/inferecee is as follows.
You expert driving assistant.
expert driving assistant,
analyze the 3-second driving video context and answer the perception, prediction and planning question in the final frame. Output format is <think_perceptiinn </think_perceptionn-le answer answer_percoption> </answer hhin__preddciionn>nneeeeeeee pre.dicttoppeiiiiionnn </anewe:_prediiction think_planning: wtt.aiii.iaamingh..tassssseeeeer
</
</
answer_planningy:
Output the step-by-step Chain-of-Theught (CoT) reasoning process in think> </think> tags and final answer in <answer> </answer> tags, respectively.
Ego Future Action is Bigo_Future Future_Actionl Action]. You current vehicle speed is ( WWICCLR_SPRIN m/s, and the historical trajectory of the ego vehicle is CIISTORRIAALTTRAACCCOOOO
To sufficiently extract knowledge from Qwen2.5-VI.-72B, we ask Qwen2.5-V1.-722 to output the Chain-of- (CoT) step by step. Qwen2.5-VL-72B is also required to use the CoT of percep- tion tasks when it generates the CoT of prediction, and use the CoT of perception and prediction tasks when it generates the CoT of planning.
Perception CoT
PROMPT FROMPT_FORMAT """I will provide you with final frame image of video, an original question, and its answer related to the image. Your task is to answer it requires step-by-step Chain-of-Thought (CoT) reasoning with numerical or mathematical expressions where applicable The reasening process can include expressions like "let me think," *oh, see," other natural language thought expressions.
Input Format:
ieaiginal_question Original Answer: loriginal_answer
Original Question:
Output Format: <thinnkeste--b--stee reasoning process</think> <answer>easy to verify answer</answer>
QUESTION "Examine the final frame image of video for key objects and report the coordinates each detected object. Key object categories include: car, bus, truck, bicycle, pedestrian, motorcycle. The image size is 896x504."
ANSWER_FORMAT = "[lntt([\"bbxx_a
(bbox), \"label\"; A ("[aabl])
Frediction and Planning PROMPT_FORMAT """I will provide you with a final frame image of video, the key objects in this frame, an question, a video Context, vehicle speed, historical trajectory (last 3 seconds) and its prediction and planning answer. Your task is to answer it requires step-by-step Chain-of-Thought (CoZ) reasoning with numerical or mathematical expressions where applicable. The reasoring process can include expressions like "let ne think, "oh, I see," " or other natural language thought expressions.
Note that prediction and planning answers are the next 3-second future action for each object and ego vehicle planning trajectory. Video context is the 3-second context.
Input Format:
Key Objects: (key_objs)
Question: toriginal_quest.e
Video :ontext: trriginal_thinking
Current Vehicle Speed: (vehicle
(vehicle_speed) speed] m/s
Historical Trajectory (last 3 seconds, meters): (Historica Milsstrrcal_Teee T_aaaalaeeetorrr
Prediction Anawer: Anforigghallonswerrereetbaae
29

---

Published as a conference paper at ICLR 2026
Planning Answer: 'nloriginal_ http:aaa_aasseee__e_eeeeeeee
Output Format: cchnn_ppreddcllnnsttep-------eee prediction reasoning process</
think_prodiction>
fansser_predicciiomeessy to verify prediction ausser//assver_peedldtiooo <hane-_olamiingpsstt--rp------ planning reasoning process/think_plamiinn <amswer_plamingeassy to verify planning answer</answr__lamiinn
QUESTION = """Predict the future action for each object and give the the
ego venicle planning trajectory. Future action can be: stop, straight right, Teft. Planning trajectory (each point means 0.5s).
points the next seconds
Please use the format as [x, yl in meters, where x-axis is perpendicular, and y-axis is parallel to the direction If y 0, it If 0, means that the ego is to TURN RIGHT, and vice versa. Noto that current Vehicle Speed does affect the ego vehicle planning trajectory but you also should consider Historical Trajectory, Rey Objects' Predicion Answers, Ego Action and the Video Context. Uing numerical mathematical expressions where applicable.
are facing.
that the ego is to GO STRAIGHT, and vice yersa.
30


Reviews:
[{"review_id": "FvRqkeaM9K", "reviewer": "Reviewer_PCAX", "summary": "They introduce a chain of thought dataset that combines reasoning in Planning, Prediction and Perception. They aim to reduce the error Propagation gap of classical Systems, by conditioning Output on different reasoning Levels. They also contribute a unified GRPO Training Approach that aims to balance the learning contribution by each part of the System (Perception, Prediction, Planning).", "questions": "1. Could you Elaborate on why a key Limitation of VLM-based E2E Systems is \"Lack of Chain-of-Thought\"? \n2. Could you clarify wether you are using temporal Video data your VLM conditions on, or you are aggregating Frames/ Just conditioning on one frame? \n3. How did you validate the CoT Dataset gt?\n4. Can you Showcase that CoT reduces the error Propagation and the gap reduction really comes from using CoT and not from foundation model Training/ higher capacity of the model generally ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "They contribute a unification in VLM reasoning by conditioning Planning on Perception and Prediction. They additionally introduce the hierarchical GRPO Training Extension that is interesting and potentially/based on their results useful.", "weaknesses": "They justify their work with a broad Claim, that is not backed by any citation or Explanation: VLM-based AD NEEDS CoT. This cannot be stated as a simple fact. Additionally, based on the text and figure 2 (Backed by figure 5, 10, 11, 12, 13, 14) it does not seem to be the case, that the model really works with Videos. Perception CoT seems to be non-temporal, but based on a single frame. While this could be for visualization purposes and Explanation, an additional Statement or Explanation would be helpful. Otherwise it is not really understandable wether the Pipeline really conditions on temporal reasoning (which seems to be claimed) or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E7FXndRPzu", "reviewer": "Reviewer_NDzW", "summary": "This paper presents AutoDrive-P3, a unified end-to-end autonomous driving framework that integrates Perception, Prediction, and Planning through structured chain-of-thought reasoning. It introduces a new P3-CoT dataset and a hierarchical reinforcement learning algorithm (P3-GRPO) to ensure coherent reasoning across tasks.", "questions": "1. In Section 4.1, the data labeling pipeline is introduced, with more details in the Appendix. I am curious about what the prompt questions look like for the three tasks. In many related works, task-specific questions are carefully designed to extract knowledge from VLMs, what is the strategy here?\n\n2. How is the quality of the P^{3}-COT dataset ensured? Since the labels are generated by Qwen2.5-VL-72B, did the authors encounter imperfect labels during annotation? How was label quality evaluated? Were any quality-control methods such as cross-validation or human evaluation (e.g., questionnaires) applied?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to read, and the core idea is clearly presented.", "weaknesses": "1. As mentioned in the Introduction, both perception and prediction are important for planning, which I agree with. However, as described in the Method section, there is no direct connection from perception to planning, which seems inconsistent with Figure 1. I wonder how the information or features are transferred from perception to planning. How does the perception-CoT contribute to the planning-CoT?\n\n2. The authors should include a Related Works subsection to discuss prior work on VLM-based autonomous driving. There are many studies that use VLMs to enhance end-to-end autonomous driving across all three tasks. Moreover, different strategies exist for integrating VLMs into the pipeline. For example, [1] encodes the VLM into the pipeline and jointly fine-tunes it, while [2] uses the VLM to generate an auxiliary dataset to improve planning, a strategy that is similar to the proposed dataset generation approach.\n\n[1] J. Hwang et al., “EMMA: End-to-End Multimodal Model for Autonomous Driving,” TMLR, 2025.\n\n[2] Y. Xu et al., “VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision,” CoRL, 2025\n\n3. The results on nuScenes are close to those of OmniDrive and OpenDriveVLA in L2. Some discussion on possible reasons for this would be helpful, especially since this work emphasizes the benefits of jointly considering all three tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Ksned3H06", "reviewer": "Reviewer_DPeP", "summary": "This paper introduces AutoDrive-P³, a VLM framework for autonomous driving that unifies perception, prediction, and planning through structured Chain-of-Thought (CoT) reasoning. It proposes the P³-CoT dataset and P³-GRPO, a hierarchical reinforcement learning method to supervise all three stages, aiming for improved synergy and planning performance.", "questions": "1. The P³-CoT dataset was generated using Qwen2.5-VL-72B. How was the quality and correctness of the generated CoT reasoning verified beyond just checking the final labels? Could biases from the generating model affect the downstream training, especially on a different series of models?\n2. P³-GRPO applies hierarchical supervision. Could errors or suboptimal performance in the earlier perception and prediction reward stages negatively impact the final planning stage more than a planning-only reward might? How sensitive is the final planning performance to the accuracy of the perception/prediction rewards?\n3. The reward weights for P³-GRPO were set in a 1:2:2:5 ratio (format:perception:prediction:planning). Could you elaborate on the process for determining these weights? Have other weighting schemes been explored?\n4. The paper argues that fragmented VLM approaches lack synergy. Table 3 compares SFT-only, Planning-GRPO, and P³-GRPO. While P³-GRPO performs best, the Planning-GRPO version also shows improvement over SFT-only. Could you further discuss the specific benefits attributed solely to the hierarchical supervision (P³-GRPO) compared to just applying GRPO to planning?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's significance lies in its explicit attempt to unify perception, prediction, and planning within a VLM using CoT, addressing the limitations of fragmented or planning-only approaches. The quality is demonstrated through the creation of the P³-CoT dataset, designed to facilitate this unified reasoning , and the novel P³-GRPO algorithm, which extends reinforcement learning supervision beyond just planning. The clarity is good, effectively outlining the problem and the proposed solution. The strong results on nuScenes and NAVSIM suggest the approach improves planning reliability and interpretability.", "weaknesses": "The reliance on Qwen2.5-VL-72B to generate the P³-CoT dataset introduces potential biases or limitations inherent in the generating model. While manually verified, the quality of the generated CoT is crucial and may propagate errors. Additionally, the hierarchical nature of P³-GRPO, while intended to create synergy, could potentially lead to error accumulation if early stages (perception, prediction) are poorly optimized, impacting the final planning output. The paper acknowledges hallucination issues and the need for real-world testing, which are important next steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kkcUTcSATa", "reviewer": "Reviewer_K3hP", "summary": "This paper proposes AutoDrive, a unified vision-language-action (VLA) model for autonomous driving. The key idea is to train a single model that can understand visual inputs, interpret instructions, and output low-level driving actions. The authors introduce a new training method called P³-GRPO (Perception–Planning–Policy Generalized Reinforcement Preference Optimization), which combines supervision from perception, planning, and control tasks in a joint optimization framework.\n\nThe model is evaluated on standard driving benchmarks and shows strong performance across all metrics. The collision rate in particular is much lower than prior systems like DriveVLM and DriveGPT, and the authors demonstrate that the proposed P³-GRPO method is the main driver behind these improvements.", "questions": "1. What’s the actual runtime per frame or per driving decision? Without this, it’s hard to judge feasibility.\n\n2. Could the system be applied in a hybrid setup (like DriveVLM’s), where the large model runs only in low-frequency cases?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experimental results are really strong. The improvement in success rate and collision reduction is quite large compared to previous SOTA, which is impressive.\n\nThe P³-GRPO training framework seems like a solid contribution. It’s a clean idea that unifies different supervision signals under one preference optimization scheme, and it clearly helps the model learn better driving behaviors.\n\nThe paper is well written and easy to follow, with a clear structure and convincing ablations showing the effect of P³-GRPO.", "weaknesses": "The biggest missing piece is runtime analysis. There’s no mention of latency or inference speed anywhere in the paper. For a VLM-based driving model, this is a serious omission. These models are large and autoregressive, and it’s hard to tell whether the proposed system can actually run in real time.\n\nThere’s also no discussion of how the model could be deployed in practice. Prior work like DriveVLM already acknowledged this issue and proposed a hybrid setup: a lightweight driving system for high-frequency control and a heavy VLM that only runs in rare or complex cases. That kind of hybrid design makes practical sense, but this paper doesn’t touch on it at all.\n\nOverall, while the performance is strong, the work feels a bit detached from deployment realities. It would be much stronger if the authors at least analyzed compute requirements or discussed how AutoDrive could fit into a practical driving stack.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)