AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
Authors: Zewei Zhou
Tianhui Cai
Seth Z. Zhao
Yun Zhang
Zhiyu Huang
Bolei Zhou
Jiaqi Ma
Venue: NeurIPS 2025 poster ()
Keywords: End-to-end Autonomous Driving
Vision Language Action Model
Reinforcement Finetuning
Primary area: applications

Abstract: Recent advancements in Vision-Language-Action (VLA) models have shown promise for end-to-end autonomous driving by leveraging world knowledge and reasoning capabilities. However, current VLA models often struggle with physically infeasible action outputs, complex model structures, or unnecessarily long reasoning. In this paper, we propose AutoVLA, a novel VLA model that unifies reasoning and action generation within a single autoregressive generation model for end-to-end autonomous driving. AutoVLA performs semantic reasoning and trajectory planning directly from raw visual inputs and language instructions. We tokenize continuous trajectories into discrete, feasible actions, enabling direct integration into the language model. For training, we employ supervised fine-tuning to equip the model with dual thinking modes: fast thinking (trajectory-only) and slow thinking (enhanced with chain-of-thought reasoning). To further enhance planning performance and efficiency, we introduce a reinforcement fine-tuning method based on Group Relative Policy Optimization (GRPO), reducing unnecessary reasoning in straightforward scenarios. Extensive experiments across real-world and simulated datasets and benchmarks, including nuPlan, nuScenes, Waymo, and CARLA, demonstrate the competitive performance of AutoVLA in both open-loop and closed-loop settings. Qualitative results showcase the adaptive reasoning and accurate planning capabilities of AutoVLA in diverse scenarios.

VLA: A Vision-Language-Actiinn Model for End-to-End Autonomous Driving with A Adaptive Reasoning and Reinforcement Fine- Tine-Tuning
Zewei Zhou"
Tianhui Cai" Seth Z. Zhao Yun Zhang Zhiyu Huang Bolei Zhou Jiaqi Ma
University of California, Los Angeles
https://autorla, http://attolaa.ithubbbbb
Abstract
Recent advancements in Lasion-Language- Action (VLA) models have shown promise for end-to-end autonomous driving by leveraging world knowledge and reasoning capabilities. However, current VIAA models often struggle with physi- cally infeasible action outputs, complex model structures, or unnecessarily long reasoning. In this paper, propose Auto novel VLA model that unifies reasoning and action generation within a single autoregressive generation model for end-to-end autonomous driving. AutoVLA performs semantic reasoning and trajectory planning directly from raw visual inputs and language instructions. We tokenize continuous trajectories into discrete, feasible actions, enabling direct inte- gration into the language model. For training, we employ supervised tieettunig to equip the model with dual thinking modes: fast thinking (trajectory-only) and slow thinking (enhanced with chain-of-thouge reasoning). To further enhance planning performance and efficiency, we introduce reinforcement fine-tuning method based on Group Relative Policy Optimization (GRPO), reducing unnecessary reason- ing straightforward scenarios. Extensive experiments across real-world and simulated datasets and benchmarks, including nuPlan, nuScenes, Waymo, and CARLA, demonstrate the competitive performance of AutoVLA both open-loop and closed-loop settings. Qualitative results showcase the adaptive reasoning and accurate planning capabilities Aut in diverse scenarios.
Introduction
Autonomous driving systems typically adopt modular paradigm, decomposing the driving task into different sub-modules, such as perception [1-3], prediction [4-6], and planning [7-9]. While this design enables structured development, it may cause error accumulation and lack of joint optimization across modules, leading to suboptimal performance [10, 11]. End-to-end autonomous driving has gained prominence with unified model architecture that maps raw sensor inputs directly to final driving actions. These models are trained on human driving data, enhancing scalability and human-like behavior. Vision-basse approaches have garnered significant interest due to their affordability and case deployment [12-15].
However, conventional end-to-end methods [16-19] primarily focus on imitating expert trajectories, lacking essential world knowledge for understanding and reasoning about surrounding environments, particularly in long-tail or challenging scenarios. Recent advances in Vision-Languea
Models (VLMs) [20-22] have gained significant interest by introducing models capable of leveraging ex- tensive world knowledge and powerful reasoning. These models have shown strong potential in
Eqqual contribution. Email: (zeweizhou, tianhui] Corresponning author. Email: zhiyuhoucla.edi
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Camera streams
V-Vision Space
Planning
A-Action Space
Dual Thinking Mode Adaptation
Pinal Trilectory
Slow Thinking in Comples Sconarios
Fast Thinking
Simple Stantario
Change Left
Go Straiggt
Vislon Encoder
Action Codebook
Vision Token
Instruction
L-Language Space
Reasoning Token
Action Token
Tum Left Go Straigt...
@ Ego states
Text
1akeen Desucpptioo
Chain of Thought
Volocity| History Action...
Tokenizer
Auto Vision
Front
centre SLDW padda and conataction veficia bucoing the right
c
System Prompt
Language Action Model
Back Light cartic prese itait disaace bunandd
Task & Thinking Guidance
Text Token
Crlkout Droket Deseriptinn
rosldnng
gastuing ladyy cui aging sartic low esounds consirucionn soree
+ Supervised Fine-tuning
SFI
Reinforcement Fine- tuning
RFT
|II
velidd provissiitt
and beginning
GT Trajectory
Vislon Encoder
Reasoning Data
Reward Model
Vislon Encoder
nit, siru dee T. nurtinn vellout sitaraanii the lock
aesmmandd
GRPO
kotker and lark Plarmet lett appeart
LLM
LoRA
Figure 1: Auto is end-to-end autonomous driving framework based vision-language models that integrates world knowledge into the driving policy. It takes visual observations, vehicle states, and language instructions input and incorporates CoT reasoning and physical action tokenization to generate planning trajectories directly. The model is trained using supervised fine-tuning to jointly learn reasoning and action, and reinforcement fine-tuning is further applied to enable adaptive reasoning through fast and slow thinking modes, improving performance and efficiency.
improving adaptability and scalability across diverse driving scenarios [23-29]. Building upon VLMs, Lssion-LanuuageActtin (VLA) models extend this capability to action generation, enabling embodied agents, such as robots [30-32] and autonomous vehicles [33, 34], to produce feasible physical actions based on visual observations and language instructions.
Despite recent progress, existing VLA models face two critical limitations in autonomous driving, or complex structure for action generation. Some
as illustrated in Fig. 2. 1) Physically-infeasibe models generate textual actions or waypoints directly using VLMs [35-37], but these outputs can be physically infeasible and suffer from mode collapse. To address this, recent approaches introduce intermediate meta-actions [38-40] or latent action tokens [41-43], which are then processed by down- stream planners or decoders to produce physically feasible trajectories. However, the intermediate representations either break the end-to-end optimization paradigm or increase model complexity and training overhead. 2) Inflexible and inefficient reasoning across diverse scenarios. Most existing models [44, 45] employ a fixed reasoning strategy, lacking the ability to adaptively switch between direct action outputs for straightforward scenarios and chain-of- offthouught (CoT) reasoning for complex ones. Although DLi [46] introduces dual-process paradigm, it relies separate modules (i.e.. a VLM for slow reasoning and a conventional end-to-end model for fast responses), which results in complicated architecture, increased training overhead, and limited scalability [47].
To overcome these limitations, we propose Auto VLA, an end-to-end autonomous driving framework that directly integrates physical action tokens into pretrained VLM backbone, enabling direct learning of an autoregressive planning policy, as illustrated in Fig. 1. Our unified architecture seamlessly integrates reasoning and action generation, allowing adaptive switching between direct trajectory generation and CoT reasoning. In supervised fine-tuning (SFT), we leverage both trajectory- only data and CoT reasoning data to equip the model with dual-process. capabilities (fast and slow thinking). Furthermore, we propose reinforcement fine-tuning (RFT) [48], utilizing Group Relative Policy Optimization (GRPO) [49] with verifiable planning reward functions. This enables adaptive reasoning that balances planning accuracy and efficiency. The RFT method not only improves planning performance but also runtime efficiency by minimizing unnecessary reasoning.
We extensively evaluate VLA using real-world datasets, including nuPlan [50, 51], Waymo [52], nuScenes [53], and simulation datasets such as CARLA [54, 55]. Experimental results demonstrate that Auto VLA achieves superior performance across various end-to-end autonomous driving bench- marks under both open-loop and closed-loop tests. Empirical results validate that our RFT approach

---

(a) End-to-emt Model
Vision Space
Language Space
Space
(c) End-to-end Model
g Conventional End-to-oe Model
N Planning
Redundant Reasoning
Infoesible
Text
Planning
in Simple Scenarios
Trajectories
Images
Dual System +
Images
23 Vision Language Model
CoT Reasoning
23 Vision Language Model
Explanation
(b) Hybrid VLM End-to-end Model
(d) VLA as End-to-end Model
8
Conventional End to-end Model
N
Planning
Action Tokenization
N Planning
Images
High-lavel Decision t
Hiararchical
Images
X Vision Language
Reasoning
Language Model
System
Thinking Adaptation
Figure 2: Four paradigms of VLMS for end-to-end autonomous driving. Compared to other methods, our proposed VLA-bbssed paradigm enables direct trajectory planning and adaptive reasoning from visual inputs. By incorporating physical action tokenization into the language model, our model effectively integrates high-level scene reasoning and low-level trajectory planning.
markedly improves planning performance, enables adaptive fast and slow thinking capabilities, and reduces runtime by minimizing redundant reasoning. The main contributions of this paper summarized as follows:
We introduce AutoVI.A, an end-to-end autonomous driving framework leveraging a pretrained VLM backbone integrated with physical action tokens, enabling direct policy learning and semantic reasoning from raw visual observations and language instructions.
2. We propose an RL-based pest-traiiing method using GRPO to enable adaptive reasoning and further enhance the model's performance end-to-end driving tasks.
3. We demonstrate that VLA achieves superior performance across multiple autonomous driving benchmarks, including both open-loop and closed-loop testing.
2 Related Work
End-to-end Autonomous Driving. End-to-end autonomous driving approaches have made significant advances in recent years [10, 11, 56-64]. Methods such as UniAD [65] and VAD [66] explicitly integrate multiple driving tasks from perception to planning in a unified Transformer architecture, thereby enhancing planning performance. ParaDrive [67] discusses the necessary components within end-to-end driving architectures. Additionally, GenAD [68] and DiffusionDrive [69] adopt generative models to maintain trajectory continuity and produce multi-modal driving trajectories. However, integrating world knowledge into end-to-end driving systems remains challenging due to bottlenecks in semantic reasoning [34] and limited adaptability in complex environments [70].
VLA and VLM for Autonomous Driving, The gap between semantic reasoning and physical actions remains a critical challenge for VLA and VLM in end-to-end autonomous driving. Current research broadly follows three directions. The first directly formulates driving as a language-centric problem, utilizing VLMs for scenario understanding through caption generation [71-73] or question answering [74, 75). The second direction leverages VLA or VLM to produce high-level meta-actions or driving decisions [17, 38-40J, which are used to either supervise [12, 76-78] or guide [39, 79] traditional planners or end-to-end models. Although these approaches facilitate integration, they prevent full end-to-end optimization. Thus, a third direction directly integrates VLMs with action generation into VLA models, enabling the direct prediction of latent action tokens [34-66, 43] final driving trajectories [37, 44, 80-83]. However, simple trajectory decoders employed in these methods (e.g.. MLP [41, 84] or GRU [45]) may produce impractical trajectories and suffer from modal collapse. To address this issue, ORION [42] incorporates generative planners into VLM architectures, enhancing trajectory feasibility but increasing model complexity and computational demands. In work, integrate a physical action codebook for vehicle motion into a pretrained VLM to effectively bridge the semantic reasoning and physical action space.
Reinforcement Fine-tuning, RFT [48] has shown considerable promise enhancing the perfor- mance and adaptability of LLMs, as demonstrated in DeepSeck-RI [22]. In autonomous driving,
3

---

Knowladge Distillation
Large Vision
SweL22.5VVL-72
Reinforcement Fine tuning
III Group Relative Opiiizztion
Decision
Sample
Slow Thinking
<pis This complex scenarto rezuking udd lona reasoiing.. . The ego vetedo stoud watt
Fine-tuning
Trajectory
p
Reasoning
Reasoning Data
Tigestoo
poss before on
canswe--taion So-action, On-action Do-acdi on, torruction 2omead 20 3brrac 3irvacinn,, 3brstaraoar
Remoning
Action Tokan
Reasoning Format
Baace Acsizaai
Sample 21
Faot Thinking
Sditial Orlect lcontficationn
Qwen2.5-VL-3B
Trapedory
ahrughileweet
Intandion
Attanse
Finell Action Destiion
suren-aaction Scinaction
sacton B3--actinn, 42o-acior B8--lanwwce
System Prompt
Text Tokenizer
Vision Encoder
Sample 3
Slow Thinking
Role: You are advanced full seff sel-driving system.
nko This comples scenario requitng adcitional reasoning, The valllde whood aingomooinn
Tass: You will be provided with video observations from the ogo vericie's surrounding cameras, along with the vehicle's a current dynomic states, Your task is predict the mose spproprieee driving action for the neet awcords
vehiile should change
<answor cootonn ao--oaction 280 cact-aatioo asten Mtleaaatinn Tereactton 30 Stleractonn dilnaatiinn
FastiSlow Thinking: If nooessoyy, use step by stop ressooning (Cheiral--hoag
directl prodict the final driving
Diranwisn,
Multi-view Camera Streams
Instruction
Ego states
52: 0.00
$1:0 8:
Reward Model
Batory Comfort
Stwight
Acceteration
88:
progreex
Figure 3: Overview of the AutoVLA model and its training process. pretrained small VLM is used the backbone of AutoVLA. The model receives multi-view camera streams, system prompts, driving instructions, and vehicle status as input, and outputs textual reasoning and physical action tokens. In SFT, large VLM model with strong visual understanding capabilities is employed to collect reasoning data, which is used alongside trajectory data in SFT for training the Auto model. In RFT, we utilize GRPO train the model for improved alignment with verified reward functions, while enabling adaptive reasoning by penalizing excessive reasoning.
Gen-Drive [7] and TrajHF [85] employed the RFT to align the trajectory generation model with safety constraints and human driving preferences. RAD [86] combined 3D Gaussian splatting to generate scenarios and conduct closed-loop RL training. However, the application of RFT in end- to-end VLM/VLA-bast --aaedddd autonomous driving remains nascent. While previous methods, such as AlphaDrive [38], utilize GRPO instead of direct preference optimization (DPO) [87] to enhance planning performance and ensure training efficiency and stability, they are still limited to simplified settings involving only high-level meta-actions. In this work, we advance this direction by applying
RFT to optimize the end-to-end VLA framework in both scene reasoning and low-level planning, and we adopt GRPO to accelerate convergence and enhance training stability.
3
Auto VLA
The proposed Aut framework consists of two main components, as shown in Fig. 1. 1) VLM Backbone: It is capable of processing visual and textual input and generating corresponding tokens (reasoning and action), employing a unified autoregressive Transformer decoder. 2) Physical Action Token Generation: We extend the language model decoder to output physical action tokens that directly correspond vehicle movements. These tokens are designed to comply with physical constraints and can be reliably translated into physically feasible planning trajectories.
Training of AutoVLA is conducted in two stages, as illustrated in Fig. 3. 1) Supervised Fine- TineTTuning uses ground-truth trajectory data and distills high-quality reasoning data from a large-scale VLM. 2) Reinforcement Fine-Tuming uses task-specific reward functions to optimize planning performance while improving the running efficiency by minimizing unnecessary reasoning. The details of our model and training process are illustrated below.
3.1 Framework
Model Inputs. Auto takes as input multi-view, multi-frame camera data C from onboard cameras, high-level navigation instructions I, and ego vehicle states S, and performs scene reasoning

---

and trajectory planning. Specifically, we utilize three RGB cameras positioned at the front, front-left, and front-right sides of the vehicle. Each camera stream c' [ c/ c captures four sequential frames at a frequency of 2 Hz, including the current and three preceding frames, supplying temporal information for scene dynamics. Additionally, the model employs high-level navigation instructions I (e.g., Turn Left and Go Straight) to specify intended directions explicitly. The ego vehicle's s state encompasses current velocity, acceleration, and historical actions.
Base VLM Model. We adopt Qwen2.5-VL-3B [21] as the vision-language backbone of AutoVLA. Qwen2.5-VL is series of powerful multimodal large language models that possess strong visual understanding capabillties, and the open-source nature of the Qwen2.5- VL model facilitates task- specific fine-tumin tun-ttning. The 3B variant offers good trade-off between efficiency and performance. making it suitable for deployment in onboard devices.
Action Tokenization. To enable trajectory planning within the language model, we discretize continuous vehicle trajectories P E TRxxd into sequence of physical action tokens a = [a1, @r), where a E A. T' is the length of the tokenized predicted trajectory and each token is represented by short-term spatial position and heading movement (Ar, Ay, A0). This transforms the planning task into a next-token prediction problem, which can be conducted within the language model. We build our action codebook A {(1, 02 .... ox) } using a K-disk clustering method [88-90], which covers the majority of vehicle movement patterns. Finally. we obtain vehicle motion codebook that consists of K = 2048 discrete action tokens. Following [30, 91], these action tokens are incorporated into the VLM as additional tokens (i.e., <action_on 0>, <action_1>, ...). During inference. the model outputs a sequence of these action tokens, which are subsequently decoded into planning trajectory using the action codebook. More details about action tokenization and trajectory decoding are provided in the Supplementary Material.
Unified Reasoning and Action. AutoVLA unifies reasoning and action generation within a single autoregressive Transformer framework, enabling adaptive switching between fast and slow thinking depending on the driving scenario. In fast thinking mode, AutoVLA directly predicts physical action tokens without generating long CoT reasoning, enabling rapid responses in straightforward scenarios. In contrast, slow thinking mode involves structured CoT reasoning, where the model first analyzes the environment, identifies critical elements, and reasons through potential outcomes before deciding on the final driving action. To enable this dual thinking capability, VL. is trained with a mixture of direct action supervision and augmenttd data. We design system prompts and response formats to support both modes consistently.
3.2 Reasoning Data
Reasoning data provides high-quality CoT annotations that are essential for training VLMs with reasoning capabilities [42]. In driving tasks, reasoning involves understanding complex semantics and interactions in dynamic environments [92-95]. Despite its importance, the development of a high-quality, large-scale driving reasoning dataset remains a key challenge due to three major limitations: 1) limited scenario diversity and repetitive examples, 2) inadequate representation of critical perceptual cues, such as traffic signs and vehicle indicator signals, 3) low-quality reasoning process, such as repeatedly stopping at stop sign without justification.
To address these issues, we propose an automated reasoning annotation pipeline using the advanced Qwen2.5-VLL-72B model [21]. This pipeline enables automatic generation of high-accuracy reasoning annotations and supports knowledge distillation from a large capable model to a more compact target model. The pipeline generates structured reasoning annotations across four key components: detailed scene descriptions, identification of crucial objects, prediction of surrounding agents' intentions, and determination of appropriate driving actions. To regulate the reasoning outcomes, our approach incorporates ground-truth driving actions as hints, guiding the model to produce causal explanations that explicitly link driving decisions to scene context. This structured prompting method significantly reduces nonsensical outputs and minimizes the need for manual correction.
Employing this annotation pipeline, we compile a comprehensive reasoning dataset comprising approximately 45.6k CoT reasoning annotations for the nuPlan dataset and 7.2k annotations for the Waymo end-to-end driving dataset. Additionally, we reformat and integrate DriveLM [96], a VQA dataset built on nuScenes and CARLA simulation data, to augment our reasoning data. Additional details and illustrative examples provided in the Supplementary Material.
5

---

3.3 Supervised Fine-tuning
Supervised fine-tuning (SFT) is employed to train the model to generate both reasoning and action sequences. Given multi-frame camera images C. high-level navigation instruction I, and the ego vehicle state S, the model is trained to produce sequence of output tokens. The output sequence consists of language tokens = -...... [t.] for reasoning followed by action tokens (a1....,,ay dr]. To enable both fast and slow thinking during SFT, we curate training data with ground-truth assistant responses that either include only the final action tokens or combine CoT reasoning with the corresponding action tokens. In the fast-thinking mode, 1 is a fixed, short template indicating that reasoning is not needed. Conversely, in the slow-thinking mode, I begins with template that introduces the need for CoT reasoning. followed by a structured sequence of reasoning.
The first supervision signal is the standard causal language modeling objective, which minimizes the negative log- ligliikli of the target token sequence and facilitates the reasoning capabillity The other supervision signal focuses on the planning accuracy, and we introduce an auxiliary loss over action tokens [01,-.. "ai......0011 ,.T), which appear at positions x1++1 to FL+T the output sequence. Given an output sequence x (4..... Iz. ... ar), the loss functions are defined as:
1 LIM
N Σ
log Re,, S,I, Laction
Σ 1+11
logpo(r; Dei, C.I.S), (1)
T
where N + T, and pe denotes the model's predicted distribution parameterized by 0.
To jointly optimize reasoning and action generation, we combine the language modeling loss and the action loss into single SFT loss function. To address the imbalance between reasoning data and action-only data, and to encourage the model to learn from examples that include CoT reasoning, we apply per-sample weighting factor based on the presence of CoT in the ground truth. The overall loss for each training example is computed as follows:
C wi (LIM,i + )a Wi: Acot if CoT is present in GT otherwise
where X and Aco are hyperparameters that control the relative importance.
3.4 Reinforcement Fine-tuning
To further improve the performance of VLA and align it with driving requirements and task- specific rewards, introduce reinforcement beaning-based post-training method. This RFT stage enables the model to perform adaptive reasoning and optimize planning performance. We employ the GRPO algorithm [49], which stabilizes training and improves convergence efficiency. Moreover, the inherent multi modaiittodlltyy of planning, characterized by multiple feasible trajectories in the same scenario, naturally aligns with the group-based optimization framework of GRPO [38].
Given a scenario input query q, comprising sensor images, the ego vehicle's state, and driving instruction, we sample a set of G candidate outputs O = {or, (01,02... ......, oG) from the old policy Thas The current policy vo then optimized using the normalized group-relative advantage A,, by maximizing the following objective:
1
(f) B.g (O)n)
(T.
DDK (Teel
(3)
G
J. -mnn To(0:1g) Too (o:lg) Ai, clip
Ta(oilg) To Tos (o:1g)
1 + As A, r: mean (eaan(r))))
((r))_=1)
std((ry) 16
where and fotd denote the current and old policy parameters, T2 is the reward for sample o,, c and B hyperparameters controlling the clipping range and the weight of the KL divergence regularization term, and The is the reference policy from the SET stage.
The final reward function is defined as r r'Diving X,TCTT, where the X, denotes the balance weight. The term DDiving varies across benchmarks. For the nuPlan dataset, we employ the Predictive Driver Model Score (PDMS) [51] as the driving reward, which captures aspects such as safety, comfort,

---

Table 1: Testing Results on the NAVSIM (nuPlan) End-to-end Driving Benchmark
Method
PDMS 1
Collision
Arca f
Direction
Progress
TTC 1 Comfort t
Ego Status MLP
66.40
93.09
78.26
90.45
63.20
84.02
99.97
TransFuser [58]
83.88
97.78
92.63
97.97
78,88
92.89
99.98
DRAMA [56]
86.87
98.19
95.18
98.03
81.33
94.17
100.00
Myyrr--MDP [57]
91.26
99.07
98.29
95.79
85,20
96.56
100.00
Centaur [59]
92,10
99.23
98.72
96.77
85,96
97.17
99.97
TrajHF [85]
93.95
99.30
97.51
91.72
90.39
98.02
99.81
Aut (One-shot)
80.54
96.89
92.42
94,43
75.82
88.06
99.94
ALA (Post-RFT)
89.11
98.41
95.64
95,40
81.87
98.04
99.94
AutoVLA (Best-of-N)
92.12
99.14
97.08
95.51
87.55
97.12
99.98
travel efficiency, and other driving quality metrics. For the Waymo end-to-end driving dataset, due to the limited availability of Rater Feedback Score (RFS) annotations [52]. we the Average Displacement Error (ADE) as the driving reward. To discourage unnecessary long reasoning chains, incorporate CoT length penalty foor into the reward function. Additional implementation details are provided in the Supplementary Material.
Experiments
4.1 Experimental Setup
Datasets. We train the AutoVLA model using a diverse set of real-world and simulation datasets. The nuPlan (Open-Scene) dataset [50, 97] contains 120 hours of large-scale driving data with eight streams of camera data and object annotations. The Waymo end-to-end driving dataset (Waymo E2E) [52] comprises 4,021 20-second driving segments with eight streams of camera views and ego vehicle trajectories, especially focusing on challenging and long-tail scenarios, such as driving through construction areas risky situations. The nuScenes dataset [53] provides 1,000 urban driving scenes with six camera views. The CARLA-Garage dataset [55] provides over 500,000 frames of simulation camera data. In addition to the collected reasoning data, utilize the DriveI.M dataset [96] for nuScenes and CARLA datasets, by reformatting the VQA pairs to facilitate CoT reasoning.
Benchmarks. We evaluate AutoVLA on both open-loop and closed-loop benchmarks across real- world and simulated environments. Open-loop performance is assessed on two public benchmarks: the NAVSIM benchmark [51] from the nuPlan dataset and the nuScenes benchmark [65]. The NAVSIM benchmark employs PDMS to assess key aspects driving behavior, such as collision and ego progress. The nuScenes benchmark uses L2 distance and collision rate evaluation metrics. Additionally, we report our model's ' performance on the Waymo end-to-end driving benchmark using the RFS metric, which reflects judan-judged planning quality. Closed- performance is evaluated on the Beivh2Drive benchmark [54] in the CARLA simulator. Bench2Drive contains 44 interactive, closed-loop scenarios under varying locations and weather conditions, using metries such success rate, driving score, efficiency, and comfort.
Implementation Details. Each action token corresponds to 0.5 seconds of movement, and the planning horizon is set to seconds. Consequently, the model outputs 10 action tokens, from which 5-second trajectory can be decoded. For SFT, we use learning rate of 1 10-5 and the FSDP training strategy. The model is trained for 5 epochs using NVIDIA L40S GPUs. We use GPr batch size of and accumulate gradients over steps, resulting in an effective batch size of 32. The weighting parameters in the SFT loss function are set to X, = and Acot = 40. For RFT, we employ the LoRA adapter
[98] for parameter-efficient training. The learning rate for RFT is set to and the KI. regularization weight 3 is set to 0.04. We perform a single policy update at each step, allowing the use of simplified objective without the need for clipping tracking the old policy. The model is fine-tuned for 6, 000 steps, and the pest-performing checkpoint is selected for evaluation. Additional details are provided in the Supplementary Material.
10
4.2 Main Results
This section reports the main results of the Auto VLA model for various datasets and benchmarks, with additional results included in the Supplementary Material.
7

---

Defoee
Ground Truth Planning
Ground Trub Planning
8.004
Front
Back
(4)
POMS
Scene Deseriptions
ressoring Slow Thinking Before RFT
Fast Thinking After RFT
0.00
The acare opasara near builiing errannes The hast alex anowe a clear pash aread with inmeditite
(b)
0.90
Guttcct Otject Devemption:
Whits Car iF stede ahoad the ago veyy and appaare startonary or nlow mosing forword
This sTr wighttooonddddd
bollwar (rom. lantion
uoced partenn cot turningttti.
0..00
Reasoniing inneen
approscch
0.79
The ago sppoeriieeeee cummerty mouingat ca micdentee pared uf 205 avli, and to
pryesaautcaa The abzenooc adtos and cater
that with cherae staltonary.
sugaisiig Turn left with onteant speedd. Civce nead to ne anoundd cers the and crtteso
The FInK oupputt odtaon
0.79
Reot Aotion
0.6
26 <6.
Cassame The oupp sortor in cuesie, 378o-setinn 8985 crannaseo
Taaiiny Sky
Figure 5: Reinforcement fine-tuning results on the nuPlan dataset. (a) Comparison of PDMS and runtime before and after RFT; (b) Training reward curves for different GRPO group sample sizes; (c) Qualitative comparison of planning and reasoning performance before and after RFT.
Data Scaling Results. We train VLA on a mixture of the nuPlan and nuScenes datasets with varying training set sizes (10k, 50k, 100k, and the full 185k samples), with action-only supervision or with additional CoT reasoning supervision. The models are evaluated the respective standard test sets, and the results are shown in Fig. 4. We observe that increasing the amount of training data consistently improves planning performance on both datasets. In the nuPlan dataset, when using fewer than 50k train- ing samples, CoT reasoning does not outper- form action-only in terms of PDMS and Colli- sion Score. This is likely due to the increased difficulty of learning structured reasoning from limited data. However, as the training set size increases, models trained with CoT reasoning surpass those with action-only supervision, high- lighting the scalability advantages of reasoning- augmented learning. A similar trend is observed on the nuScenes dataset: as the training set size increases, models trained with CoT reason- ing consistently outperform those trained with action-only data in terms of L2 distance and collision rate.
nuPlan PDM Score
pu Plan No al-fauli Col. 100
18.44
70-
Neat-Back Cod.
clue
66.27
33
90-
Actin-iityy
Actoeventy Trantng
50-
16k
10ck
30t
100c
1
nuSocmes 1.2 (m)
nnSeenes Col. Rate (%)
Actiavaalyy Inisig Renseniy Taiain
Actinsedy Traning Reoning Triis
Oot
0.4
16k
100k
190%
Figure 4: Data scaling effect on planning per- formance for nuPlan and nuScenes datasets (log- scaled x-axis). Increasing the amount of training data consistently enhances planning performance.
The slow-thinking mode incurs a significantly higher runtime due to the generation of CoT reasoning compared to the fast-thinking mode, as shown in Table 2. To mitigate this overhead, we introduce RFT en- hance AutoVLA's adaptive thinking capability and avoid unnecessary reasoning in straightfor- ward scenarios. Specifically, we apply RFT to the full-data CoT reasoning model trained via SFT to enhance its planning performance. As shown in Fig. 5(a). RFT yields a 10.6% improvement in PDMS (on the NAVSIM testing set) and a 66.8% reduction in runtime (average over 500 testing scenarios). The reward curve in Fig. 5(b) illustrates the progressive improvement of the model's policy during RFT. Experiments with different GRPO group
RFT Performance.
Table 2: Runtime Analysis of Fast & Slow Think- ing Modes in VLA
Thinking Mode
Min. (s)
Max. (s)
Avg. (s)
Fast Thinking
0.997
1.116
1.072
Slow Thinking 7.607
13.706
10.518
8

---

Auticnoett was Phonanng
Actineetly Thinng
Make
(h)
wit cou yoluu le sperie messing por them
Collinn Pronneng
Post RFF
aronc oir gnr locie pinarit read foott aigh cratues conns arion siie calilae
Caratration Locandi
vank. inportan tioyraa scurra
7.557
vor shead.
RPS
7,406
1,47
7
ol th spntaci) jas bu you caulic an ricus wntt
7.29
lveer
foomt fongen aartly
Hase a her rpe
camp
Front
7
Figure 6: Performance comparison of AutoVLA with different training settings on the Waymo end-to-end driving dataset, along with example illustrating the model's reasoning capabilities.
sample sizes indicate that larger groups lead better performance by promoting broader exploration of training samples. As illustrated in Fig. 5(c), RFT also reduces unnecessary and slow reasoning in simple scenarios, driven by the CoT length penalty that encourages fast thinking for straightforward driving cases. A qualitative comparison shows that the SFT model produces suboptimal plans due to error accumulation in generation, whereas the RFT model (optimized via PDMS- reward) generates better trajectories.
nuPlan Benchmark Results. We evaluate AutoVLA against state-of-the- end-to-end driving models on the NAVSIM benchmark [51] and present results in Table 1. In best-of-N planning, we use an oracle scorer select the optimal trajectory from six generated candidates. After RFT. AutoVLA demonstrates significantly improved performance, aligning more closely with the NAVSIM reward signal. The best-of-N strategy further enhances performance, achieving the highest PDMS. Overall, Auto VLA achieves competitive results while demonstrating scalability across diverse datasets.
Waymo E2E Performance. We evaluate Auto the Waymo end-to-end driving dataset [52], which features long-tail and complex driving scenarios. The model's performance under various training settings on the test set is presented in Fig. 6. The results reveal that pretraining on a combination of nuPlan and nuScenes datasets significantly enhances performance, suggesting enhanced scene understanding through exposure to more diverse training data. Incorporating CoT reasoning in training further improves planning performance compared to action-only training. Post-training with RFT, using ADE as the reward function, achieves the best overall RFS metric. qualitative example in construction zone demonstrates the model's 's ability to reason about occlusions and generate effective detour plans.
CARLA Closed-loop Performance. We evaluate the closed-loop driving performance our AutoVLA model on the Bench2Drive benchmark [54] in the CARLA simulator. The model is trained using SFT with both trajectory-only and CoT data. During testing, the planning frequency is set to 2 Hz. The results, shown in Table 3, demonstrate that VLA outperforms existing end-to-end driving models in terms of overall driving score and success rate in the closed-loop test.
Table 3: Testing Results the Bench2Drive (CARLA) Closed-loop Driving Benchmark
Method
Driving Score
Success Rate (%) f Efficiency
Comfortness
AD-MLP [99]
18.05
0.00
48.45
22.63
UniAD-Base [65]
45,81
16.36
129.21
43.58
VAD [66]
42.35
15.00
157.94
46.01
TCP-traj [100]
59,90
30.00
76.54
18.08
DriveAdapter [101]
64.22
33.08
70.22
16.01
Orion [42]
77.74
54.62
151.48
17.38
AutoVLA
78.84
57.73
146.93
39.33
4.3
Ablation Studies
Text Waypoint Output. We use the same mixed training set from the nuPlan and nuScenes datasets to train a model that predicts waypoints waypoints. We evaluate its performance in open-loop planning setting using the standard test sets. The results, shown Table 5, indicate that our action tokenization and generation method significantly outperforms the text- based waypoint prediction approach. Additionally, due to the need to decode
text format, which are then converted into a trajectory of

---

Table Action Tokenization Accuracy with Different Codebook Sizes and Methods. (DCT: discrete cosine transform; MC: movement coverage; CU: codebook usage. Bold indicates best performance, underline the second-best, and tilde the third-best.)
Codebook Size
RT-1 (Action Bin) [102]
FAST (DCT) 1911
K-disk (Ours)
ADE(m) L
(DE(m) 4
ADE(m)
FDB(m)
ADE(m)
FDE(m)
MC
CU f
256
0.1440
0.2942
0.1708
0.2137
0.0687
0.1034
86.47%
100.0%
1024
0,1052
0,1883
0.0522
0.0588
0.0253
0.0282
97.41% %
100.0%
2048
0.1014
0.1775
0.0281
0.0309
0.0182
0.0203
99.42%
100.0% %
4096
0.1001
0.1739
0.0149
0.0161
0.0141
0.0155
100.0% %
91.46%
numerical values, the text-based method incurs a substantially higher computational cost in gen- erating the final trajectory. This shows the lim- itation of language models in handling precise numerical reasoning.
Table 5: Influence of Physical Action Tokenization
Metric
Text Waypoint
Physical Action
PDM Score f Avg. L2 (m) +
71.31
80.54
0.89
0.70
Action Tokenization Methods. We evaluate re- construction accuracy across different codebook sizes for several tokenization methods, including RT-1 [102], FAST [91], and our proposed K-disk lokenization, as shown in Table 4. For RT-1, ac- celeration and steering rate discretized using uniform action bins, and kinematic model is employed to reconstruct the trajectory. However, because only trajectory-level data is available and control actions must be indirectly inferred, this binning approach yields the highest recon- struction error. The FAST tokenization method is more sensitive to codebook size than our K-
Avg. Col. (%) 4
0.36
0.31
Runtime (s)
7.65
3.95
Table 6: Performance Comparison across Different Tokenization Methods on NAVSIM
Tokenization
PDMS f Collision
Progress
FAST (DCT) [91]
67.63
92.74
64.09
K didkk (Ours)
80.54
96.89
75.82
disk approach, achieving comparable reconstruction accuracy only at K 4096. In contrast, our proposed method consistently attains the highest reconstruction accuracy across all codebook sizes.
When the codebook size is small (K 256), reconstruction errors are significantly higher due to limited movement coverage, many movements cannot be adequately represented by the small set of codebook tokens. As the codebook size increases beyond K 2048, the improvements in tokenization accuracy and movement coverage become marginal, while codebook usage decreases as many tokens remain redundant and unused. Considering this trade-off, we select balance reconstruction accuracy, movement coverage, and efficient codebook usage.
2048 to
Furthermore, Table 6 reports planning performance on NAVSIM (nuPlan) for different tokenization methods. Our proposed method outperforms FAST. In FAST, the discrete cosine transform converts fixed-horizon planning trajectories into variable-length token sequences, complicating the selection of appropriate token lengths and making fixed-horizon planning difficult.
5 Conclusions
We propose AutoVLA, an end-to-end autonomous driving framework that unifies scene reasoning and action generation within single autoregressive model. We adopt SFT to enable the model to operate in both fast thinking (direct trajectory generation) and slow thinking (enhanced with long CoT reasoning) modes. In addition, we introduce RFT to enable adaptive reasoning by penalizing unnec- essary reasoning and aligning action generation with reward functions, improving both performance and efficiency. Experimental results demonstrate that Auto VLA achieves competitive performance on both open-loop and closed-loop planning benchmarks and exhibits strong reasoning capabilities.
Limitation and Future Work. Although our model with dual-process adaptation achieves near- real-time inference (1 Hz), it remains highly GPU-dependent, requiring significant memory and computing. Future work will focus on real-time applications, optimizing runtime efficiency, and reducing computation overhead (c.g., through model quantization) to enable real-time deployment.
10

---

Acknowledgements
This work was supported by the Federal Highway Administration Center of Excellence on New Mobility and Automated Vehicles, and by the National Science Foundation under Award No. 2346267, POSE: Phase DriveX: An Open Source Ecosystem for Automated Driving and Intelligent Transportation Research.
References
[1] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning sird's-ee-vieww representation from lidar-camera via spa- tiotemporal transformers. IEEE Transactions Puttern Analysis and Machine Intelligence, 2024.
[2] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 191-191. PMLR, 2022.
[3] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang. Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35:10421-10494 2022.
[4] Zikang Zhou, Zihao Wen, Jianping Wang, Yung- Hui Li, and Yu-Kai Huang. Qenext:
A next-generation framework for joint multi-agent trajectory prediction. arXiv preprint arXiv:2306. 10508, 2023.
[5] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion
prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3955 2024.
[6] Zhiyu Huang, Haochen Liu, and Chen Lv. Gameformer: thenetthcoretic modeling and learning of transformer- based interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3903-3913, 2023.
[7] Zhiyu Huang, Xinshuo Weng, Maximilian Igl, Yuxiao Chen, Yulong Cao, Boris Ivanovic, Marco Pavone, and Chen Lv. Gen-drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning. arXiv preprint 1:X55:2410.05582. 2024.
[8] Zhiyu Huang, Haochen Liu, Jingda Wu, and Chen Lv. Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving. IEEE transactions neural networks and learning systems, 2023.
[9] Haochen Liu, Zhiyu Huang, Wenhui Huang, Haohan Yang, Xiaoyu Mo, and Chen Lv. Hybrid- prediction integrated planning for autonomous driving. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
[10] Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu. Reasonnet: End-to-end driving with temporal and global reasoning. Proceedings of the IEEE/CVF conference computer vision and pultern recognition, pages 13723-13733. 2023.
[11] Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21983-21994, 2023.
[12] Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vip: Vision language planning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14760-14769, 2024.
11

---

[13] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pages 14749-14759, June 2024.
[14] Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St- p3: End-to-end vision-based autonomous driving via -eaporllemmporl feature learning, In European Conference on Computer Vision, pages 533-549. Springer, 2022.
[15] Shaoheng Fang, Zi Wang, Yiqi Zhong, Junhao Ge, and Siheng Chen. Tbp-former: Learn- ing temporal bird's-eye- s-eye- pyramid for joint perception and prediction in vision-centric autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1368 1368-1378 2023.
[16] Xingcheng Zhou, Mingyu Liu, Ekim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C Knoll. Vision language models in autonomous driving: A survey and outlook. IEEE Transactions on Intelligent Vehicles, 2024.
[17] Tianhui Cai, Yifan Liu, Zewei Zhou, Haoxuan Ma, Seth Z Zhao, Zhiwen Wu, and Jiaqi Ma. Driving with regulation: Interpretable decision-making for autonomous vehicles with retrieval-augmenner reasoning via 1lm. arXiv preprint arXiv:2410.04750 2024.
[18] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Putlern Analysis and Machine Intelligence, 2024.
[19] Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like human: Rethinking autonomous driving with large language models. In 2024 IEEE/CVE Winter Conference on Applications of Computer Vision Workshops ( (WACVW), pages 910-919.
IEEE, 2024.
[20] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Cameey, et al. Openai ol system card. arXiv preprint arXiv:2412.1677 2024.
[21] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint 1.Xi::2502.139255 2025.
[22] Daya Guo. Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepscek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025.
[23] Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision- langunlanguuag models self delf-driving corner cases. In 2025 IEEE/CVF Winter Conference Applications of Computer Vision (WACV). pages 7817-7826. IEEE, 2025.
[24] Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, and Marco Pavone. Driving everywhere with large language model policy adaptation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14948-14957, 2024.
[25] Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan- Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024.
[26] Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533, 2024.
[27] Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Mul- timodal language model for driving. European Conference on Computer Vision, pages 403-420. Springer, 2024.
12

---

[28] Katharina Winter, Mark Azer, and Fabian B Flohr. Bevdriver: Leveraging bev maps in llms for robust closed-loop driving. arXiv preprint arXiv:2503.03074, 2025.
[29] Kexin Tian, Jingrui Mao, Yunlong Zhang, Jiwan Jiang, Yang Zhou, and Zhengzhong Tu. suscenes-spatialgee A spatial understanding and reasoning benchmark for vision- vision-language models in autonomous driving. arXiv preprini arXiv:2504.03 164, 2025.
[30] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-actiin model. arXiv preprint arXiv:2406.0999 09246, 2024.
[31] Qingging Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma. Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language--ctiinn models. arXiv preprint arXiv:2503. 2025.
[32] Chia- Yu Hung. Qi Sun, Pengfei Hong, Amir Zadeh, Chuan Li, U-Xuan Tan, Navonil Majumder, and Soujanya Poria. Nora: A small open-sourced generalist vision language action model for embodied tasks, 2025. URL pettts://ariii orgg http//b///// 2504. 19854.
[33] Hidehisa Arai, Keita Miwa, Kento Sasaki, Kohei Watanabe, Yu Yamaguchi, Shunsuke Aoki, and Issei Yamamoto. Covla: Comprehensive ssioo-language-action dataset for autonomous driving. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1933-1943. IEEE, 2025.
[34] Xingcheng Zhou, Xuyuan Han, Feng Yang. Yunpu Ma, and Alois C Knoll. Opendrivevla: Towards end-to-end autonomous driving with large vision language action model. arXiv preprint arXiv:2503.2346. 2025.
[35] Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15120-15130, 2024.
[36] Katrin Renz, Long Chen, Elahe Arani, and Oleg Sinavski. Simlingo: Vision-only closed-loop autonomous driving with language-action alignment. arXiv preprint (rXiv:2503.09594, 2025.
[37] Jyh-Jing Hwang, Runsheng Xu. Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp. et al. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arxiv:2410.23262. 2024.
[38] Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unlcashing the power of vims in autonomous driving via reinforcement learning and reasoning. arXiv preprint arXiv:2503.07608 2025.
[39] Bo Jiang, Shaoyu Chen. Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large rision-language models and end-to- end autonomous driving. arXiv preprint arXiv:2410.22311. 2024.
[40] Wenhai Wang, Jiangwei Xie, Chuan Yang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemim: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245, 2023.
[41] Katrin Renz, Long Chen, Ana-Maria Marcu, Jan Hünermann, Benoit Hanotte, Alice Karnsund, Jamie Shotton, Elahe Arani, and Oleg Sinavski. Carllava: Vision language models for camera- only closed-loop driving. arXiv preprint arXiv:2406. 2024.
[42] Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. Orion: holistic end-to-end autonomous driving framework by vision -anguaee instructed action generation. arXiv preprint urXiv:2503.19555 2025.
[43] Waywe Research Team et al. Lingo-2: Driving with natural language, 2024.
13

---

[44] Shuo Xing, Chengyuan Qian, Yuping Wang, Hongyuan Hua, Kexin Tian, Yang Zhou, and Zhengzhong Tu. Openemma: Open-source multimodal model for end-to-end autonomous driving. In Proceedings of the Winter Conference on Applications of Computer Vision, pages 1001-1009, 2025.
[45] Tianqi Wang, Enze Xie, Ruihang Chu, Zhenguo Li, and Ping Luo. Drivecot: Integrating chain-of-thought reasoning with end-to-end driving. arXiv preprint 2tt.:2493..6996. 2024.
[46] Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large rision-language models. arXiv preprint arXiv:2402. 12289, 2024.
[47] Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reason- ing models can be effective without thinking. arXiv preprint arXiv:2504.09888 2025.
[48] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances neural information processing systems, 35:27730-27744, 2022.
[49] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0 24X0::2402.03300, 2024.
[50] Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, et al. Towards soaming-based planning: The nuplan benchmark for real-world autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 629-636. IEEE, 2024.
[51] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang. Zetong Yang. Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. Advances in Neural Informa- tion Processing Systems, 18780668779, 2024.
[52] Waymo Research. 2025 waymo open dataset challenge: Vision-based end-to-end driving. https://wayno com/ conn/ppe/caa hhalleenges/2255/22222 /nnges//2055/22-ddrriin./// 2025. Accessed: 2025- 04-25.
[53] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEFE/CVF conference on computer vision and pattern recognition, pages
2020.
[54] Xiaosong Jia, Zhenjie Yang. Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.0388 033877 2024.
[55] Bernhard Jacger, Kashyap Chitta, and Andreas Geiger. Hidden biases of end-to-end driving models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8240-828 (2000,824,,,00 October 2023.
[56] Chengran Yuan, Zhanqi Zhang, Jiawei Sun, Shuo Sun, Zefan Huang, Christina Dao Wen Lee, Dongen Li, Yuhang Han, Anthony Wong, Keng Peng Tee, et al. Drama: An efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint rrXiv:2408.03601 2024.
[57] Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint Ttle:24000069776 2024.
[58] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-basee sensor fusion for autonomous driving. IEEE transactions pattern analysis and machine intelligence, 1287811288888889918 2022.
14

---

[59] Chonghao Sima, Kashyap Chitta, Zhiding Yu, Shiyi Lan, Ping Luo, Andreas Geiger. Hongyang Li. and Jose M Alvarez. Centaur: Robust end-to-end autonomous driving with test-time training. arXiv preprint arXiv:2500. 1650, 2025.
[60] Zewei Zhou, Hao Xiang, Zhaoliang Zheng, Seth Z Zhao, Mingyue Lei, Yun Zhang, Tianhui Cai, Xinyi Liu, Johnson Liu, Maheswari Bajji, et al. V2XPnP: Yehicle-to-everything spatio- temporal fusion for multi-agent perception and prediction. arXiv preprint arXiv:2412.01812. 2024.
[61] Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. arXiv preprint arXiv:2405. 19620, 2024.
[62] Yiheng Li, Seth Z. Zhao, Chenfeng Xu, Chen Tang, Chenran Li, Mingyu Ding, Masayoshi Tomizuka, and Wei Zhan. Pre-training synthetic driving data for trajectory prediction. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5910-5917, 2024. doi: 10.100/1R0535522224..8000
[63] Wenzhao Zheng, Junjie Wu, Yao Zheng, Sicheng Zuo, Zixun Xie, Longchao Yang, Yong Pan, Zhihui Hao, Peng Jia, Xianpeng Lang, et al. Gaussianad: Gaussian-centric end-to-end autonomous driving. arXiv preprint arXiv:2412. 10371, 2024.
[64]
Yuping Wang, Shuo Xing, Cui Can, Renjie Li, Hongyuan Hua, Kexin Tian, Zhaobin Mo, Xiangho Gao, Keshu Wu, Sulong Zhou, Hengxu You, Juntong Peng. Junge Zhang, Zehao Wang. Rui Song, Mingxuan Yan, Walter Zimmer, Xingcheng Zhou, Peiran Li, Zhaohan Lu, Chia-Ju Chen, Yue Huang, Ryan A. Rossi, Lichao Sun, Hongkai Yu, Zhiwen Fan, Frank Hao Yang, Yuhao Kang, Ross Greer, Chenxi Liu, Eun Hak Lee, Xuan Di, Xinyue Ye, Liu Ren, Alois Knoll, Xiaopeng Li, Shuiwang Masayoshi Tomizuka, Marco Pavone, Tianbao Yang. Jing Du, Ming-Hsuan Hsuan Yang, Hua Wei, Ziran Wang, Yang Zhou, Jiachen Li, and Zhengzhong Tu. Generative ai for autonomous driving: Frontiers and opportunities, 2025. URL https: //arxiv. org/ abaaa/////// 2505. 08854.
[65] Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Gaaniing-orientee autonomous driving. In Proceedings of the IEEE/CVF conference on compuler vision and pattern recognition, pages 17853-17862, 2023.
[66] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajic Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pages 8340-8350, 2023.
[67] Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449-15458, 2024.
[68] Wenzhao Zheng, Ruiqi Song, Xianda Guo. Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. In European Conference on Computer Vision, pages 87-104. Springer, 2024.
[69] Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint 2rXiv:2411.1511 2024.
[70] Zhiqi Li, Zhiding Yu, Shiyi Lan. Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14864-14873, 2024.
[71] Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, and Jingjing Liu. Adapt: Action-aware driving caption transformer. 2023 IEEE International Conference Robotics and Automation (ICRA), pages 7554-7561, IEEE, 2023.
15

---

[72] SungYeon Park, MinJae Lee, JiHyuk Kang, Hahyeon Choi, Yoonah Park, Juhwan Cho. Adam Lee. and DongKyu Kim. Vlaad: Vision and language assistant for autonomous driving. In P Proceedings of the IEEF/CVE Winter Conference on Applications of Computer Vision, pages (800-987,20 2024.
[73] Kairui Ding, Boyuan Chen, Yuchen Su, Huan-ang Gao, Bu Jin, Chonghao Sima, Wuqiang Zhang, Xiaohui Li, Paul Barsch, Hongyang Li, et al. Hint-ad: Holistically aligned interpretabil- ity in end-to-end autonomous driving. arXiv preprint SiKi:220990660000 2024.
[74] Ana-Maria Mareu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driving. In European Conference on Computer Vision, pages 252-2699 Springer, 2024.
[75] Yiheng Li, Cunxin Fan, Chongjian Ge, Zhihao Zhao, Chenran Li, Chenfeng Xu, Huaxiu Yao, Masayoshi Tomizuka, Bolei Zhou, Chen Tang, Mingyu Ding, and Wei Zhan. Womd- reasoning: large-scale dataset for interaction reasoning in driving, 2025. URL https: // //arxivv.or org/ abbs/407. 22072000222 04281.
[76] Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through Vision-language model supervision. arXiv preprint arXiv:2412. 14446, 2024.
[77] Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M Patel, and Fatih Porikli. Distilling multi- modal large language models for autonomous driving. arXiv preprint arXiv:2501.09757, 2025.
[78] Pei Liu, Haipeng Liu, Haichao Liu, Xin Liu, Jinxin Ni, and Jun Ma. Vlm-e2e: Enhancing end-to-end autonomous driving with multimodal driver attention fusion. arXiv preprint arXiv:2502.1800 18042, 2025.
[79] Haicheng Liao, Hanlin Kong, Bonan Wang, Chengyue Wang, Wang Ye, Zhengbing He, Chengzhong Xu, and Zhenning Li. Cot-drive: Efficient motion forecasting for autonomous driving with Iims and chain-of-thought prompting. arXiv preprint arXiv:2503.07234, 2025.
[80] Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint 13.00:2310.01415, 2023.
[81] Songyan Zhang, Wenhui Huang, Zihui Gao, Hao Chen, and Chen Lv. Wisead: Knowledge augmented end-to-end autonomous driving with vision-language model. arXiv preprint arXiv:2412.09951, 2024.
[82] Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving, arXiv preprint arXhv:2407.00959 2024.
[83] Zhijie Qiao, Haowei Li, Zhong Cao, and Henry X. Liu. Lightemma: Lightweight end-to-end multimodal model for autonomous driving, 2025. URL battps://arxiii www.aaiwiv.......lloloooooooooot abs////s//555// 00284.
[84] Wenru Liu, Pei Liu, and Jun Ma. Dsdrive: Distilling large language model for lightweight end-to-end autonomous driving with unified reasoning and planning, 2025. URL https:
// //arxivv org/ a200 05360.
[85] Derun Li, Jianwei Ren, Yue Wang, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Zhongpu Xia, Peng Jia, Xianpeng Lang, et al. Finetuning generative trajectory model with reinforcement lcarning from human feedback. arXiv preprint arXiv:2503. 2025.
[86] Hao Gao, Shaoyu Chen, Bo Jiang, Bencheng Liao, Yiang Shi, Xiaoyang Guo, Yuechuan Pu, Haoran Yin, Xiangyu Li, Xinbang Zhang, et al. Rad: Training an end-to-end driving policy 2025.
via large-scale 3dgs-based reinforcement learning. arXiv preprint
16

---

[87] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. Advances in Neural Information Processing Systems, 36:53728-537777 2023.
[88] Zhejun Zhang, Peter Karkus, Maximilian Igl, Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed-loop supervised fine-tuning of tokenized traffic models. arXiv preprint arXiv:2412.05334, 2024.
[89] Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. arXiv preprint arXiv:2312.04533, 2023.
[90] Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: scalable multi-agent real-time motion generation via next-token prediction. Advances in Neural Information Processing Systems, 37:1 1044-114407,, 2024.
[91] Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747, 2025.
[92] Sung-Yeon Park, Can Cui, Yunsheng Ma. Ahmadreza Moradipari, Rohit Gupta, Kyungtae Han, and Ziran Wang. Nuplanqa: A large-scale dataset and benchmark for multi-view driving scene understanding in multi-modal large language models. arXiv preprint arXiv:2503. 12772, 2025.
[93] Dongming Wu, Wencheng Han, Yingfei Liu, Tiancai Wang, Cheng-zhong Xu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 2359-8367, 2025.
[94] Tianwen Qian, Jingjing Chen, Linhai Zhuo Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa:
A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4542 4550, 2024.
[95] Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. In European Conference on Computer Vision, pages 292-308. Springer, 2024.
[96] Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens BeiBwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision, pages 256-274. Springer, 2024.
[97] OpenScene Contributors. Openscene: The largest up-to-date 3d occupancy prediction bench- mark in autonomous driving. /ttps://githar www//oommmml/eeee teaaaeieeealbbb/eeeeeeeeee 2023.
[98] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 (2):3, 2022.
[99] Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao. Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305.104: 2023.
[100] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory- guided control prediction for end-to-end autonomous driving: A simple yet strong baseline. Advances in Neural Information Processing Systems, 35:6119-6133 2022.
[101] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7953-7963, 2023.
17

---

[102] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.0668 2022.
[103] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp. Charles Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9710-9719, 2021.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We do describe our contributions in the abstract and introduction of the paper: 1) a novel end-to-end autonomous driving framework leveraging a trained VLM backbone augmented with integrated physical action tokens, enabling direct policy learning and semantic reasoning from raw visual inputs. 2) an RL-based fine-tuning method using GRPO enable adaptive decision-making and further enhance Auto performance end-to-end driving tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the real-world implementation limitation of Auto and intro- duce our future work model quantization, as Section 5 shown.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
19

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper focuses on the experimental exploration of model architectures, and have no theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the necessary information for reproducibility is provided in Section 4 and the Supplementary Material, The model details also are discussed in the methodology the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
20

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release our code and dataset to facilitate community research.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
cublic/guides//// wttpess//ooeSuummsiioioiiooi
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips.ccooo wwtppblcc//acecccccccccccccc gewidee/Codeeiee
ississsss ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification; We provide all the implementation details and dataset settings of Auto VLA in both the SFT and RFT stages in the paper Section 4 and supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper reports results from single run per experiment, consistent with prior work on this task.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper provides all necessary details on computational resources, including GPU type (NVIDIA L40S), memory usage, and number of epochs, Section 4.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///nnrrrp.eeeee/eee wttpblic//thissciiieieoooooeeeee
Answer: [Yes]
Justification: The research presented in this paper adheres to the NeurIPS Code of Ethics, ensuring ethical standards data collection and analysis.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
22

---

Justification:
The paper includes discussion of potential societal impacts in the supplemen-
tary material.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no foreseeable risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original creators and asset owners are appropriately credited, and licenses and terms of use are explicitly stated and duly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release all the code, model weights, and datasets to facilitate research in the domain.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or human-subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing or humannsubject research.
Guidelines:
24

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We declare the usage of LLM/VLM the paper, and we integrate the physical action tokens into the pretrained VLM backbone to develop a novel end-to-end autonomous driving framework.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : httpss////nuurtie /ttpr/eeesp////// for what should or should be described.
/LLM)
25

---

We implement an action tokenizer based on the constructed action codebook. During training, continuous trajectories are discretized by mapping each 0.5-second segment to its nearest action token in the codebook A, resulting in a sequence of discrete tokens [a1, ..,..., ar). During inference, the language model autoregressively generates sequence of action tokens, each representing 0.5-second motion segment. These tokens are then converted back to their corresponding motion Auto VLA Supplementary Material
A
Action Tokenization
A.1 Action Codebook
To enable trajectory-level planning within a language model, we introduce physical action tokens, each representing a short-term feasible vehicle maneuver. These tokens are derived from clustering vehicle motion patterns in the Waymo Open Motion Dataset (WOMD) [103]. which contains extensive real-world vehicle trajectories. The resulting codebook, visualized in Fig. (a),), is used across experiments involving multiple real-world datasets, including nuPlan, nuScenes, and Waymo. Due to differences in vehicle dynamics compared to real-world data for simulation testing, we construct separate action codebook using the same clustering procedure on the CARLA-Garage dataset [55]. This produces another set 2048 action tokens for simulation testing, as shown in Fig. S1(b).
We begin by sampling short motion segments from the dataset to construct a discrete action codebook. Each segment represents 0.5 seconds of vehicle motion, characterized by its final-frame bounding-bol contour, computed based on the vehicle's s position, dimensions, and heading with respect to the current- frame coordinate. Then, we apply K-Disk clustering on these sampled segments, which iteratively selects diverse set of representative segments {m...... m&x j. such that no two segments are within a distance threshold 8 = 0.05 m, measured using average contour distance, For each selected segment Va, we extract its spatial displacement and heading change, denoted as (Ar, Ay, A0), and define it as the action token 0g. The resulting action codebook is A {u1,.... Oxk }. with K 2048, where each token encodes a distinct and physically feasible short-term vehicle behavior.
Real-world (WOMD)
0.2
.62
-450
-07
Figure S1: Visualization of action codebooks from (a) real-world dataset (WOMD) and (b) simulation dataset (CARLA). Grey arrows represent all 2048 tokens (position and heading), with 300 tokens randomly highlighted in blue for clarity.
A.2 Action Tokenizer
segments in the codebook and applied sequentially from the initial ego pose. Therefore, the model reconstructs a continuous trajectory the ego-centric coordinate frame by composing these local displacements and rotations.
B
Reasoning Data Collection
large-scale, high-quality reasoning dataset with chain-of-thought (CoT) annotations is essential for enabling robust reasoning capabilities in vision-language-ee
(VLA) models. In this paper, we introduce an automated reasoning annotation pipeline using the state-of-the-ar Qwen2.5-V1L-72B lasion-la model [21], as illustrated in Fig. S2. The pipeline significantly reduces reliance on
26

---

Scons Description:
The soun ts right and the sgo vaticle stopped an irarnaction The noet carerrs shows pie and retoo lyint directy ahead, rillating tuat velvdle must step.
Reasoning Format
Crttccl Ooject Description:
Stere Anddyds
Traffic Lights
Camera)
Ressoning
Orkket Object Montiootioo
Qwen2.5-VL-72B
Leastan: Directly shead the vahiide
ng
Desoripton:: The trafiic ligtts ars disaily nod, sirikiing clus stp.
Actio Dacabon
imporiaion The red light are crucial in they di canns the velidde inmredart aclicti stosping and woiling for the iight shange.
2 Pedestrian Crossings (Front Laft Front Right Cameeran]:
Text Tokenizer
Vision Encoder
Leceton: On atter adde the internatiion
System Prompt
Desorittonn pollsstions are siblle near the crossings.
importance Athhauh no pedeetrarn present, the crassing indicate potertal
Role: You udvanced full ucl-- faithing spstom
nequinn
Taakk You te prceiced with video vassevations from the ago vatrid's surmuunding
ong with
Multi-view Camera
Reasoning on Intert:
veiide must stop and was light turn green...
1. Red Traffle Lights: The primary foous is the truiflo light, which manccats that the
the optional olivig anlion
recoede
Driving Gemmandd Instruction: The criving coes suggoss "go given
use stppli loy-khs reasoning (Chaiin-of-The Thein-d- unive
aggtt
the beet shring action.
3 Vakidle The velidee alraady
the records,
vilccity accu lentionatinn zoro...
Instruction
Ego
Ground Longited
Best Driving Action: Stop
stepped,
Laft| Brrightt... ....
Velostyy Accelecation Hiitory
RC Example
Desoription:
and vohide need
vefii'e position light traffie carofully There consauotion vat and
Critical Object insoription:: The Whine trock turning right with postion lyge on (penwnl teig tawesss not binkelighh)) The conctruction validees wes costonary (trake ightt on) limis right
ngnt rignstum fol the Vivile brakk
sehind. Thus, the should change vollidle
in the
right
-Best Driving Action: change wight acceloution
Figure S2: Automated reasoning annotation pipeline for autonomous driving, illustrated with reasoning annotation example from the Waymo end-to-end driving dataset.
human annotations and facilitates effective knowledge distillation from more powerful, large-scale model to a more efficient, compact model.
B.1 Reasoning Annotation Pipeline
System Prompt. The system prompt specifies the model's role, task, expected CoT reasoning format, and examples of CoT reasoning. The definition of role and CoT reasoning format aligns with the required outputs of the AutoVLA model, which focuses structured CoT reasoning. We carefully design several representative reasoning examples to guide the model. Moreover, the reasoning process includes four main steps: 1) scene description and analysis, 2) critical object identification and description, 3) intention reasoning of the surrounding objects, and 4) decision-making and meta-action.
User Message. The user message includes the driving instructions, ego vehicle states, and multi-view camera streams. Notably, we introduce the ground-trui -ruth driving meta-action derived from the data as explicit hints in reasoning, guiding the model to produce causal explanations that directly associate decisions with the driving context. This structured prompting significantly reduces nonsensical outputs and minimizes manual revisions.
Reasoning Data Generation. We employ the most capable model in the Qwen-VI. series as the reasoning data annotation engine, leveraging its strong reasoning capacity and extensive world knowledge. The maximum length for newly generated reasoning textual tokens is set to 700. Additionally, we extract answers to relevant questions from the DriveLM VQA dataset [96], which focuses perception, prediction, and planning in the nuScenes and CARLA datasets, and then reformat them into our standardized reasoning format. These reformatted samples are then combined with the generated data to construct the final reasoning dataset.
Human Quality Check. We evaluate the quality of generated reasoning based on the accuracy and completeness of critical object identification, causal reasoning, and action decisions. The evaluation uses a binary scoring scheme: any error in these aspects results in a score of O; otherwise, the sample receives a score of 1. Human annotators assessed 3,000 randomly selected samples, yielding an overall accuracy of 88.8%, which demonstrates the high reliability of our proposed annotation pipeline. Erroncous samples were either corrected or discarded.
27

---

Halepuod OT Palue hipelry
Tijelay
Descriptiore
Reasoning
Scans Description:
Reasoning Data
reiidennia ana during dayy fimm with clear the right sidc. cre parked cars slorgiz bitth dks the street, and low ho can be seors The Yoon curreaa shows pain
The velidee
The scaee right, the appears be undergaing canal uctiin mairnonno work an indione by the SiP ESFT sign and nsange- puane--nndoo dowch abrised pois Theen are neveent wen cise ahoad, incauding a berge truck disectyy in front of ogo voticlo The road is woll by strouligite it, and the woather soarte duar No bullic lights are vistre
infersctiion with "STICI" sign
wrile the from right careraa showe "STOP" sagn and corricts shorss stragft road lsoding up intesscction moving
paltedd moal toy
scons
Critical Okiject Description:
Gritical Orjjett Description:
Large
STOP Bign.
alewdy mastonnrry sale
Looatol on tho right side of intersction, indiaating thut the ago warld must come comp siio bettore propeed mn. os regulatory
traks lights
This inpprtantt
velidd
complanno with motfoo ago
oking signta stonk
Sige:
skde roudwork.
sign
guiing
Visbti both videa the struat sicularty near the inte statioo
-What they aos cning They aes cotionery and morapyy parding aprice..
Orongo-annd White Striped Poles:
-Wy they ar importent The ego vehice must novigate amonnd these parkee comn
There placed slong side the road, marking boundiry
expeciiity when changing lanne surning
construction sone, They holp guido cihera stay the correct lane and anoid hazarda.
3. Clear Path to the Left:
Reasoning on Intane:
Oiseenedd through the front camerr..
Large
slowing sopped, cased felloing
lights ago
vaficde proceed.
KEEP Bign and Ohange White Stiped Polcs incicate the shoul) skay
Reasoring
Sinoe driving conmand go striigh, ago
de aimendy come dosse zoro and post behovior of decelonat ng ze1o
indicnned curmet velecity
reg
2. The drnvngg comerrddir strucs the ego ver sile to go left suggesting lo't turn or lono change is recessary.
3. ago veriids
and lo curennt veloctt rolutivcly low. Olion the presenoo the
moving forward with decdenation past asconda.
and salety.
indicaling
prowwed
Bost Driving Actions Move forward with decelaration
Glown shownco
wetli pedeeatane the wettile
This action aligna with the need to mainban a safe dhitarse from the truck ahead with dhe actunngg to the roed agga and ran senipiingg trough the construction zone. Decokenttnng ansames the ago unsquacted cbetodee appear.
coccutz quick ecodienaion other changing lancs to the lolk
Best Driving Change
quick asseberation
can me pood opperprnenl track diope complenely or Elery
camers, the need city proceed afler acceecatio enaume wilident now gption through the intentctionn while moirsiiing soflety
irarssctos The quick
Figure S3: Visualization of the reasoning data annotation on the Waymo end-to-end driving dataset.
B.2 Reasoning Annotation Visualization
Some CoT reasoning annotation examples from the Waymo end-to-end driving dataset are shown in Fig. S3, showcasing that our annotation pipeline can deal with situations where vehicle already stopped at stop sign can proceed, rather than remaining stopped indefinitely. The pipeline also accurately interprets construction--elated road control scenarios, enabling high-quality reasoning annotations. Examples on the nuPlan dataset are provided in Fig. S4, demonstrating the pipeline's capability to distinguish the functional relevance of stop signs and traffic lights across different lanes, rather than stopping at every detected stop sign.
C Details of Supervised Fine-tuning
We perform supervised fine-tuning (SFT) with the Fully Sharded Data Parallel (FSDP) strategy for efficient multi-GPU training. We enable mixed-precisie training using B6 for parameters,
28

---

Description:
Reasoning Data
Scere Desoription:
Reasoning Data
The scene dap buay urtan intersaciion with muliple lansa trattic, The buaticc lights shoud aam ret inciciinng ng that veliddes must stop. These are emance velidles vie bis the foor canern enc, inctultng withis van cirectty front the egn un ticae ans other cons. katthrr anoud. The right camena rin shtow and same cary moving along
The scane appears to an urten arse with mulitile lanes and varicus read fnstures The Fortt es shows dlear path anead with no iemrddane stocclo. The right wes indicatos qured rosd ending tawards coverdd
core parind
The
view
pedeettiinn
The
ridehtvlly strent hehlidd selicde The woother
cloar, and thore no volblic traffic lights affeccinn the curnert pachs.
Oritioal Object Desoription:
Oritical Object Descrigtione
1. Wiite Van (Frent Coreea Vaew::
Pedectioons near the sovorad area (Right View?:
octioon misecity in frort ago weidd
the
Stutes Studoonayy yy indicaled aod suflie iift..... Indlostor Lights lights.
Duscriptinn
pain ego
nontal pace and enoxing any invracibie interton
cross the road.
2. Truck (Right Camera View::
importance Paay steenton their movements as they sould polentbally step orr the read pn cspodia if the vohidde is surning right. Stop Sign Viewdy
Location in The edjpoent tane the right
Ststac Moving fowaad caoor Lights: No visble indicator liles
imoe mance: The trucica movement neads monioored to emeure thene sudidan
shange-
Desorprion
vistiie indioning that vehide must compleer
Views
skap before propoed
ocrioon On lutt
impprtanee Thisk in crui se dicrase the welide reot aotion ignoring the stop sags could kead to unsels driving cord oors and potorial accidenta.
-tultes Slulkonay. y. culor Lights Not appliaatio
iiaa fanco not in indioddyyyy voloornt to the ago vofidd's a ooooenr, roquiing ro coution.
Reasening
Recsosing on Indertt
The veriide cumennty instucted to tumn right The vont and right vioee show clear path for sumiing cut the prisence of peckotttana near the covered anas recuins caution The view shones slos sign which stggeesta that the vatede may need
Trurk:c Lighta: The rec bratic light indiaa that all with clso must stop and wast undil the light
famma prears
betore
ago
movinn ferwnrd,
driiing
comp sting the tams Howano: since the stop sign is furtne down the road and the vaticle apwed
-Ohhrr Viticiaa. The while van direcdt front ia stantmaryy and the truck the acjacent
The whicie cunnnily
accotecatio incirating
bow in moving lonwad wittnut any indiaation on et char ging beww..
tuat la proparing coucall the turn.
-gg velide Store The ego vehide in curmetty stadionary with owo locily and acceterstion, consinsen with requinmmnn to stop the red iige. Gaven ned tortte ago
Beat Driving Action: Turn right with a conatent speed. should proceed with
lgin
diadcty
etspped
right while maintaiing comstant speend This ensunes
age
green.
kmooth and conimated rumn in enylng nanm of the tn on the right adde This skos sign on the kiit la eircily rulovant to the caronr manosidd so the loou romaina sencuting the tum sohtly
Beet Briving Action: Stop
The ogo hilll aiould non rain statomary, schoring the nod buffiz light maintaningg light grewe,
procaed
lowaad.
Figure S4: Visualization of the reasoning data annotation on the nuPlan dataset.
System Prompt
User Message
RRle:: You are an Advanced Driver Assistance and Full Self- Driving System.
Input Description: The autonomous veticcle equipped tires and right, onabling comprehensive parception the surrunndin environment.
front,
-Task: You will recetve visual observations from the ego vehicle's cameras and dynamic information about the vehicle's current state. Your task to predict the optimal driing action for the next five seconds. First, carefully analyze the surronnding environment by cons dering trafic lights, the movements of other vehicles and pedestrans, lane markings. and any other relevant factors. If neceosary, use stsp-by by--bh-steep masoning (Chain-ffThcaup Othenwiso, iso, you directly predict the fnna driving action.
Multi-viww Camera Streams:
to arive the best drivig action.
-ront Camera: The first video presints the front the vahide, comprising four sequential frames sampled at 2 Hr.
Fast/SSlow Thinking: if necessary, use stop by-step reasoning [Chain of Thought) to ariiee the best driving action. Othewiise, you may directly predict the final crving action.
framnss)
Structure your reasoning as follows:
Front Left Camera: The second video presents
Scene Analysis: Describe traffic stuation, including relevant environmental cues such as traffic lights lane markings,
the front-left: view of vehicle, comprising
three criiical road users obstacies. specifying rolatve positions the oga vohidl.
Idantification Critical Objects: Identily
behaviors of surronncing vehicles or pedestrios..
sequeetiial fromes sampled Ht.
(frnn left camera frames]
Prediction Critical Object Behavior Predict the potential movements idemiiiid
Front Right Camera: thind video preserns the frant-riht view vihicle, comprising four aequantial frames sampled 2 Ho.
objects.
Intent Raasoning: Besed
observed anvironment and vahicie
the
inient ago wahicle.
S tfrnt right camera frames]
Final Action Decision: one fateral action and longitdiinal action: Lateral actions
exactly one': [move forward,
charge lane
The ourret velocity of the vehicle is 3.1 mUS, and the current acceleration is 1.4 mis?
-Ego States:
right. change right]
Longitudinal actions (chocen onel: (stop. decaleration
IInttruction: The driving instruction Keep Forward. Based on this information plan the action trajectory for the autonomous veriide over the next five seconds.
consiant spaid, quickk decelerwi decillration, quick accaliration, acosllration)
Present action dlearly reasoning
Figure S5: System Prompt and User Message of VLt.
gradients, and buffers to reduce memory usage and accelerate computation. Gradient checkpointing enabled to reduce GPU memory consumption. The learning rate warm-up is 500 steps and decays by 2% every 2,000 steps. The model is trained for 5 epochs, with gradient clipping applied at a maximum value of 1.0 to ensure training stability.
The model takes both system and user prompts as context inputs. As illustrated in Fig. S5, the system prompt defines the model's 's role, task, and the expected CoT reasoning format. The user message describes the multi-view camera observations, the ego vehicle's current state (i.e., speed and acceleration) and optionally historical states, and the high-level driving instruction. We use three camera views (front, front-left, and front-right), each providing four consecutive frames. Images in the video stream resized to maintain original aspect ratios but reduced to 28 x 28 x 128 pixels.
29

---

Algorithm 1 RFT for AutoVLA with GRPO
Input: Supervised fine-tuned policy TSFT, Action Codebook A, Group size G, Training step K,
Dataset D, Driving reward function "Driving Reasoning reward function TCoT, Balance weigh
Xy, KL regularization weight /
Output: Reinforcement fine-tuned policy TRPT
1: Initialize reference policy Tref < TSFT and current policy To +- WSPT
2: for training step 1 to K do
3: Sample scenario U from D
4: for sample i from to G' do
5: Sample imput query 9, final output o,, and per-token probability (o
6: To.aa (o:lg) +- To(0:)4)
7:
Calculate per-token probability (oil() of Traf under 0;
8:
Decode trajectory т A(o;)
9:
Calculate reward r; + "Driving (T, U) {.rCo((O;)
end for
11: Group average reward r, or f- mean(ri,.... ra), std(ri,...,
Group relative advantage A, f- (ra -))(o))
CRFT f -
(- (o (q) A + 5t (To (T
Tua (o,1g)
14:
Optimize xa according to LRFT
15:
end for
16: return Converged policy TRFT
D Details of Reinforcement Fine-tuning
Group Relative Policy Optimization (GRPO) employs group-based sampling to compute the advan- tage function, replacing conventional state-value estimators or critic models. This design accelerates training while aligning naturally with the inherent multimodality of planning, which requires evaluat- ing and selecting from set of candidate trajectories. The overall reinforcement fine-tuning (RFT) procedure is illustrated in Algorithm 1.
D.1 Kullback-Leiler (KL) Divergence
The KL divergence term DR. is incorporated in Eq. (3) regularize the current policy with respect to the reference policy:
Tea (o (o(())))
TTer(o.)))) no (ra(o:)))
DL ( (yet)
log
1,
(SI)
where denotes the parameters of the current policy Ta. q is the scenario input query, o, is the output of ith sample in the group, and rrr is the reference policy from the SFT stage. This regularization term penalizes large deviations from the reference policy, ensuring that policy updates remain within stable area. As result, the model retains useful knowledge acquired during SFT while improving driving behavior through reinforcement learning guided by validated reward functions.
D.2 Reward Function
The reward function provides the primary RFT training signal, guiding the optimization of policy updates. To further enhance comprehensive driving performance beyond imitating expert driving, we adopt hybrid reward design that integrates the primary driving reward I'Diving and CoT penalty Tcor to discourage lengthy reasoning. For the NAVSIM Benchmark, we employ Predictive Driver Model Score (PDMS) [51] as the primary reward. For the Waymo end-to-end dataset, due to the limited annotation of RFS, we employ the (normalized) Average Displacement Error (ADE) as the driving reward. The final reward function, balanced by weighting coefficient X,, is defined as:
Driviing
XxTCoT
(22)
30

---

In the nuPlan dataset, we adopt the PDMS score as the driving reward signal "Drivigg which serves as a comprehensive measure of driving quality. It is defined as follows:
Drriviing PDMS = NC DAC x 5TTC 2C + 5EP
(S3)
12
where the components include No at-fault Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Comfort (C), and Time-to-Collision (TTC). Detailed descriptions of each component can be found in [51]. Each sub-score is expressed as a percentage, and their weighted combination yields normalized score within the range [0, 1]. reflecting aspects of safety, comfort, and progress. For any scenario in which the planner fails (due to output errors), we assign a score of O.
For the Waymo dataset, we define the driving reward "Driving based on the ADE metric:
8 ADE
T Σ
Drriving
ADE
19. yell2,
(S4)
K
where ó denotes the maximum displacement error, and is a scaling factor to normalize the reward. The planning trajectory y is evaluated against the ground truth trajectory y, and ADE is computed as the average L.2 distances over time steps.
To penalize excessively long reasoning chains, we introduce CoT penalty term Tcor, defined via sigmoid function that normalizes the length of reasoning:
TcoT 1 (t--ttaa))
(S5)
where L denotes the length of the CoT reasoning, Leot is the tolerance threshold, and y is a scaling coefficient that controls the steepness of the penalty curve.
D.3 Implementation Details
We use the navtrain split of the nuPlan dataset for RFT for the NAVSIM benchmark and the validation split of the Waymo end-to-end driving dataset for the Waymo benchmark. The vision encoder of pretrained AutoVLA is frozen, and the model is fine-tuned using Low-Rank Adaptation (LoRA) to reduce training costs and memory consumption. Specifically, both the LoRA rank and alpha are set to 8, with dropout rate of 0.1. The pretrained SFT model is used as the reference policy during optimization. The hyperparameters y, Ltots and learning rate are set to 2 x 10-3 400, and 3 x 10-2, respectively. To ensure that the driving reward signal is dominant, the regularization weight is set to a relatively small value, X, 0.3. We also set 2, x 10 in the RFT of the Waymo dataset. Moreover, we configure the generation parameters with sampling temperature of 1.0, top-p 1.0, and top-k of 0.0 to encourage diverse and exploratory generation during GRPO sampling, which effectively covers a wider range of possible actions.
E
Experiment Details
E.1 Data Preprocessing
To enable mixed training across multiple driving datasets, we develop a unified data preprocessing pipeline that standardizes the format across all datasets. For each sample, we extract and standardize:
1) Ground truth trajectory coordinates and headings in the ego vehicle's 's coordinate frame at 2 Hz, 2) Image paths for multi-view camera image sequences consisting of 4 consecutive frames captured at Hz (providing 2 seconds of history), 3) CoT reasoning annotations, 4) Vehicle states including its current velocity and acceleration, 5) High-level driving instructions. The preprocessing pipeline handles dataset-specific differences in data format, sampling rates, and coordinate systems to create a consistent format. The size and distribution of the final formatted dataset are shown in Table S1.
nuPlan (NAVSIM). We randomly sample 45.6k scenarios from the nuPlan trainval split and generate reasoning data using our proposed automated annotation pipeline. The resulting reasoning samples, together with the remaining training data with only trajectory annotation, constitute our full training set of nuPlan. Following the NAVSIM benchmark, the navtest split is used as the test set.
31

---

Table S1: Training and Testing Data Size for Different Datasets
Dataset
Train Samples
Reasoning Samples
Test Samples
nuPlan (NAVSIM) [51]
166.3k
45.6k
12.1k
nuScenes [53]
19.0k
2.9k
5.6k
Waymo [52]
23.8k
7.2k
1.5k
CARLA [96]
274.5K
53.2k
nuScenes. We preprocess all samples from the training set. For samples included in the DriveL. dataset, we reformat the question-answer (QA) pairs to generate structured reasoning annotations following our four-step reasoning format. Samples not covered by DriveLM are also used for training but only with trajectory supervision. The validation set is used for lesting.
Waymo. The Waymo end-to-end driving dataset provides 2037 training and 479 validation segments, each containing 20-second video with driving logs for the entire duration. We sample reasoning data using a 4-second sliding window and extract trajectory-only data using a 2-second sliding window offset by 1 second from the reasoning samples. Due to noise in the position data, the estimated vehicle heading can exhibit abrupt fluctuations when the vehicle is stationary. To address this issue, we apply a motion threshold to detect stationary periods and smooth the heading accordingly. The test set comprises 1505 samples.
CARLA. The CARLA-Garage dataset is used to train the model for closed-loop evaluation. Since only front camera images are available, we use single-view inputs (with four consecutive frames) for CARLA training and testing. We sample data using sliding window with an offset of 0.5 seconds, and downsample the trajectories from 4 Hz to 2 Hz. For reasoning annotations, we leverage the DriveLM-CARIE dataset, which provides QA pairs similar to SorveLM-nuScenee and we reformat the QA pairs to generate samples with reasoning annotations.
E.2 Evaluation Metrics
nuPlan (NAVSIM). PDMS is the official benchmark metric in NAVSIM on the nuPlan dataset, with its formulation provided in Eq. (S3).
nuScenes. Following previous works, we adopt the L2 Distance measure the average displacement error between predicted and ground-truth trajectories, and use Collision Rate to assess the frequency of predicted trajectories overlapping with surrounding objects. We follow the UniAD protocol, reporting both metrics at 1, 2, and 3 seconds in the future, instead of averaging over the horizons.
Waymo. Following the Waymo end-to-end driving benchmark, Rater Feedback Score (RFS) is used to evaluate the driving performance. The Waymo dataset provides three human-annotated trajectories, each associated with scalar quality score in [0, 10). A planning trajectory is matched its closest reference trajectory, and a trust region is defined around it using lateral and longitudinal thresholds at fixed timesteps (3 and 5 s). These thresholds are scaled by a piecewise linear function of the rater's trajectory's 's initial speed. Predictions within the trust region inherit the matched rater's 'ss score, and those outside are penalized exponentially based on normalized deviation.tr
CARLA. We evaluate driving performance in closed-loop testing using four key metrics: Driving Score, Success Rate, Efficiency, and Comfortness following the Bench2Drive benchmark [51]. Driving Score reflects the overall route completion penalized by infraction severity. Success Rate measures the percentage of routes completed without major infractions within the time limit. Ef- ficiency quantifies how well the ego vehicle maintains reasonable speed relative to surrounding traffic, computed as the average speed ratio over multiple checkpoints. Confortness assesses the smoothness of the trajectory based on physical signals such as acceleration, jerk, and yaw rate, using expert-derived thresholds.
F.3
Model Training
Our primary base model is trained on the mixed training set of the nuPlan and nuScenes datasets. The ego state contains the vehicle's current velocity and acceleration. For the test on the Waymo end-to-end driving dataset, we further fine-tune this base model using the Waymo end-to-end driving
32

---

mtll
Fext Thinking
rest Thinkng
nost Thinking
shnightlonerr
can be made, <ihnns tanawen The fina oubut adionie
can be rade ethace ranswer The final cuoul acton
can maad <takce
racenam The fhe se
Carid Trre
Crcand Treb
This comptex ssonarto eeganing addition rascoringg
<hirko socepie consare requidig anditinal reasasing
<hhakk This ins soceples ccasaris requiing addili reesse
Scase Dourpptton
Scaee Descriptinn:
roake
arwniia hoot strs Thinsica tydio
pedertiont thef
tolees ho hou yie of ago
Foortor the age Thore book car to mooooo oh ago
ego These two cass
Cribiad Diject Docorption:
Olgict Desoripdis:
wash busk the foorrol nei wiaiee.. moungg couu the
Tran weits site the uet rgo whats sut groee igh
wit klekk BUY troet the ogt whicee reshtight
SuV
sulblonryy niy.......
pang sheegg
Deesaning as bokeet:
Reasceing on intere:
Bamnceing intere
white truck red truck
SUV: The object moving
rclic
itks traffc valiide should
the lime thad renains atd lnaay The velidle
shoul conince- Seco rolice dreen oijectis
ranain static tary. Secandl notica SUV.
(lat
stutadds
wiilie
aheidd....
Driving Action
Best Briving
Dust Driving Action
STOP:
STOP. ethare
Move forward with acculaaaion ethisie
cansee fraal ic
stw.aainnn <larwwer
raxpour
www.:/.00-aaaadd 2000 Aanssvrr
adton
eadias 1932 1al--addaa 1077 +taawar
Figure S6: Planning and reasoning results of ALA on the nuScenes dataset.
Table S2: Testing Results of Auto on the nuScenes Planning Benchmark
ST-P3 metrics
UniAD metrics
Method
1.2 (m) |
Collision
(%)
1.2 (m) 4
Collision (%)
1s
2s
3s
Avg.
1s
2s
3s
Avg.
1s
2s
Avg.
1s
2s 38
Avg.
ST-P3 [14]
1.33
2.11
2.90
2.11
0.23
0.62
1.27
0.71
VAD [66]
0.17
0.34
0.60
0.37
0.07
0.10
0.24
0.14
UniAD [65]
0.44
0.67
0.96
0.69
0.04
0.08
0.23
0.12
0.48 0.96 1.65 1.03 0.05
0.17 0.71 0.31
EMMA [37]
0.14
0.29
0.54
0.32
OpenEMMA [44]
1.45
3.21
3.76
2.11
Open VLA-3B [34]
0.14
0.30
0.55
0.33
0.02
0.07
0.22
0.10
0.19
0.58
1.24
0.67
0.02
0.18
0.70
AutoVIA (action only)
0.22
0.39
0.61
0.41
0.10
0.17
0.28
0.18
0.29
0.67
1.17
0.71
0.15
0.34
0.56 0.35
Aut (w/ CoT)
0.21
0.38
0.60
0.18
0.28
0.20
0.70
0.14
0.25
0.53 0.31
dataset. For the model trained Waymo, the ego vehicle's state encompasses current acceleration and 4-second history of vehicle positions and velocities. For closed-loop simulation testing, we train separate model using data preprocessed from the CARLA Garage dataset and DriReLM-CARRLA annotations. This model is trained with single-view inputs instead of the multi-view setup used in the primary model, and use a large resolution for the input images (with 28 28 x 384 pixels).
E,4 Inference
We employ a stochastic generation strategy using top-p and top-k sampling to generate reasoning and planning outputs. Higher sampling diversity (e.g., temperature= top-p=0.5, top-k=20) supports slow thinking modes, enabling the model to produce deeper and more elaborate reasoning chains. In contrast, more deterministic settings (e.g., temperature=0.1. 1, top-p=0.01, top-k=1) produce fast thinking, yielding consistent and direct responses.
33

---

Pight
Right
<thino This is snrightlorerrd sesnorio, and a direct decision can <thinke
Fest Thinking
<iivis> This is shrigglffowrrd sconario,
Faot Thinking
and a direct deciaton can made. <itiinis
<anewerr fivd caclios 177tocacton 1815
cancaees The Ennl caction 1001 (iC
Lett
Foose
Fuoot Riah
Front
Frone Right
Right
Slow Thinking
Slow Thinking
<thikt a complex acenarto requiring andtiinnl rasasing
<think ts complex scenarto requiting addiiional reasontig.
Descriptio::
chylin urban intereed with dleetl morked pedkettinn addevaiiie and visble prion sullo light, Avan biin front the edhicle and beo lanos shood with yellow tox markings. Two pecieiitane tne arw wallings the eddwwalk in the latt..
The vebide
interned
utbon ronement The froe camnes shown.
skip sign nuffo kintt and a "Nu Left Tuni" sign lult side of road... The hori-eghh camera snowe a careed road into parcing lor with stop aign and tump waninng wh the back camera showss trafli quound behind, siggesslin the ago vehicle recurdly began moving forward....
Peolestriars
Ootical Otyect Description:
Predictioo:
continue voiling whhoout creaaing read.
Traffic (Pront stewd:
importance: Must onsure do not suddenly ore the ossooaals during the lift tarn
Looated of the uppor left the intersoction, currently redi it onucial os-it ponoooss
2. Whits van (Prent View::
creaingg rafffi and the apo vetidaa entry imto the intersection.
Brosus Appeore be moving dlowly or perporing to proceed smsighe.
Block ahoad (5 (Front viewdy
Predictoos go straight turn right
dirccty sheed, proceeding forward the interscction ho ping
unnopectedly
Green traffic light (host viewa)
"Mo Ture" (Front viewt:
Stutua Groenin the ugo lane's disccton.
Located ou ow the stip this proniiiaa left tuma fun the wat satice''s curint lanc.
Predictirn (all sdose sate soncc fon last tum
Reasening
Inportanco: Co fimms of the ago voliddo has the gf leginn furn
Reasoning on intent:
Findly the ego velicle how beer stopped the paet 4 seconds, ind calling reaiiesss to coucale manouees
The tralfic iggh is pooil ones toward the auggesingg mighht on controllat.
turning toatic not frowed reswenet
grown light
agot
The ago viliule was stapped the past secor du and is moving only with low aooelo olion, calling starting
stop aign.
monel dlear
Finuly while fhre ars podostioons on the sidowalk, they ail ow no incent cross, and apposing volidko are vokkyy ling the intunootio...
-hiidyy thore inmesiate vehidiia bloking the kill sum pafts ane the etcnecttoona
Beat Driving Action: Move forward with acceteration.
Olene the rosd signs, veliile behowdyy and instuction to go fooward ago vehidee has atready slsp siip sign
move fonaard with acceieation the vahidde
uhoad. aligning with ago validde's inaruction <thinko
Best Driving Action: Turn with with acseleration
Tra erourso smooth manouwer respecting the sullic ight signal, allowing the ugo with tn vottlyy and wrldintly samo the turn while monttoinng surrunttngg wthinko
200-actton
ourput
Figure S7: Planning and reasoning results of AutoVLA on the nuPlan dataset.
F
Additional Results
F.1
nuScenes Results
We evaluate AutoVLA the nuScenes dataset following both ST-P3 [14] and UniAD [65] protocols, in comparison with state-of-the-ee end-to-end models. As shown in Table S2, our model demonstrated competitive performance in the nuScenes planning benchmark.
Fig. S6 shows AutoVLA' planning outputs. The model generates safe trajectories that closely align with ground-truth motions, accompanied by coherent and context-aware reasoning outputs. However, it can be observed that many nuScenes scenarios are relatively straightforward, often not requiring complex reasoning. This may explain the lack of performance gain in quantitative metrics when reasoning is introduced.
F.2 nuPlan Results
We present additional visualization results of our model on the nuPlan dataset in Fig. S7. In relatively simple scenarios such as curved roads and intersections, our model generates high-quality trajectories via fast thinking. In more complex scenes with numerous traffic regulations, it leverages slow thinking
34

---

From Right
Ground Planning
+
Ground
Slow
After RFT
Planing
edd aieento reasoning.
be unr with mio of paceesion pathwags and rosie The hust wos ahoon path
Front Left
butland wouther
Aling unitarr cakar and.
Critecl Oepect Deacription:
Viewge Parted driving
plong the oftne mociis the wetw
Thinking
Butlting arerar prraace (Freet Vinsf; lacaned directy hutrorr the vartills
Beene seriptionn
The score sppears urban anea packett pashwaage and roeds, showaa
remmariaate unlesu sumuure outs
bunsiigg While noti modis
criticll du sworn of cnniinngggg helpe madie colladione with foad objects
Cribical Uttject Disomptiore
Vebiclus on right sida pligh stauds Parted stong the side oftea read the dger sleas
Rezacring Indentt
The gg un curnently miniing slnway wethwwhhhhhhhh w slight decelecarion, antiioomeeiittttt
ou on entrrnnee france glas doors and pellare......
wncado clacetenazin
crende eenerreonnatti wataes, ctwirg commens and to right augaoting
The
pederssinne immineddinn
Reseontin inteen
accibly slleww tor enonts evnnesanr Howeore
vebicce stouli novigats cxrotall to ursan puskotr ara steut croos
the buildingg entrares. The ahooe -and vntirvataaddea wet runsb bucnongattt canstally, lett vets shous sidtwmbee de with and licing which ninwery rnsuore th curnent moneear win.
it properting
Turn right deoslavation. Glaen the unee shustics, in, the veriic whooud proceed within right sum wiile main riing controdld decallo enmure salitty...
Ture right with e deceteration Ovven nood to sanger anourd the verictosion on gpttard.... citteko
canoeer ourpur actinnl roctinn 2000o-caction 20000 silnswar
cansaeer The ouppor saton cuesion, 20300 90995. cransaneo
Figure S8: Qualitative comparison of planning and reasoning performance in complex scenarios, before and after RFT. Results indicate that RFT maintains reasoning capabilities in complex scenarios while enhancing planning performance.
Table S3: Waymo Vision- End-to-Ena Driving Challenge Leaderboard
Method Name
RFS (Overall)
ADE at 5s (Overall) ADE at 3s (Overall))
RFS (Spotlight)?
Poutine
7.9860
2.7419
1.2055
6.8929
HMVLM
7.7367
3,0715
1.3269
6.7269
UniPlan
7.6925
2,9864
1,3083
6,6544
DiffusionLTF
7,5919
2,9768
1.3605
6.5688
VLA
7.5566
2,9580
1.3507
6.9436
Swin- Wyjn-Trajcctory
7.5432
2,8135
1.2082
6.6791
waymo
7.5281
3,0182
1.3200
6,5953
ViT-Adapter-GRe GRU
7.4988
2.7024
1.1968
6.4543
Drive'Traj
7.4957
2,9556
1,3038
6.4101
open-Ilama
7.4288
3.2165
1,3140
6.2510
MTR-VP
7.3433
3,3485
1 1,4232
6.4023
WayPredict-XL
7.2922
3.2915
1,4385
6.3083
DriveTrraj
7.2787
3,4573
1.5346
6.2428
WayPredict
7.0641
3.5779
1.7242
5.8562
LightEMMA
6.5169
3.7395
1.7052
5.7103
NrstenResNet500
6.4719
3.9148
1.9446
5.7977
OpenEMMA
5.1575
12.4755
6.6842
4.7131
to produce better and cooplpiann planning results with CoT reasoning. As illustrated in Fig. S8, our VLA model preserves reasoning capabilities complex scenarios despite removing redundant reasoning after RFT. Qualitative comparisons demonstrate that RFT enhances planning performance in complex scenarios.
F.3
Waymo Results
To overcome the limited size of the Waymo end-to-end driving dataset, we augment the training data with samples from the nuPlan and nuScenes datasets. For convenience, we refer to this combined dataset as nuX. We use this nuX data to pretrain the model before fine-tuning it on the Waymo dataset. To further enhance planning performance, we apply RFT on the AutoVLA model after SFT. Given the limited number of RFS-labeled samples (only 480 the validation set), we adopt the average displacement error (ADE) as the primary reward signal.
35

---

"thie This in compres scenario requirnn additionaal ressoiing.
Siow Thinding
Thes
te stghee veriiee ieneling en ins fore nost lome murtings tnith hoot VitA eevara voniddes maina cing modderss distance cpel an dir along edge raad Truttic agg are gi allocing waraiiaaaaaaa
Critiea Diiped Dasoriation
up apeaacayy
mimittalt cu flow
-What they are dotigg Hoving fensand, Whet
Liwil continu moving fervard as the turfcl bn graan. -Wiy impa
collica stoms
Oyelition the Rood tnvs more
of swar, the eye out snleppe in aelos Whet they ane dhning liiling
Wheat will des Cond nus aingth sdga why impartane The ago with all mus ta inaac sare latara stance and an picset auci Jision
Boott ugau The oe murth ligh volcat shat with oa poosu and with coopping
Vellade
and
3. Syetiaion the Road The cyclit la idlnng silong the ed and pore iereed li rest tor with puth
coar saseets foowon oo cosertttinnn end......
Besn Drsing Actions
tnrfroouuturaaiinn vetterr 1280iration 1333 stameeere
tthie This complaxaceratio recating additionnal reasoring.
Slow Thanden
stridh and is axpppuaa he and reporn coen uoloor war these ore sower wahl los st soctet aad, scuclss perto to the litt ond right eites of the read, ves sectielyy
Critoal Onject Deveriptta::
Coutting iocrasdiiv contee citt fioervio woots anstiiggtatie andicaa
ciler nedtched und act scis ouui coppu doonga toan
surtic
pou an an way anttaay prevent crrcct passage fonsard
Reasoring inturt
Cartell tart The is gi- clurh clearb indcana that vetica should
Stud seary Valaciou Abead: presene cooped vaniciss and barass need warthr saarope ane not clessare
Driinn Oriing Actlone
Stop. ago veticee should semaln stadonery, to thettalt tertaffcco setup and we to the approprine signas clummnne proowed,
and vettic
matra
"arewam The fials actoraationii ii raction, sernact an, ii... stassseer
(b)
Cooeada eugen and appessied
ithi Tris complex peeplaccccenatiin recstting add tional reasceing. Scere Deacriprans:
Stow Thesting
Tharr
busties skong with desutt par partorsion wotlinger deoach de thes the ago biggee saiking shooe thar Rovend
Oratiaal Olipect Description
dinggliing clnoook apprasc
crosing the
Whht they desig Wilking gasit salowiik What aasy tham.
world lp
casss agrta cood. -Whh they
What iney deing un engie the road wanu they will dronncee coving i roud. (Wy aee iepustien The degle th the voricle, andrearaini raining salu disa you ssauntai avoid any patertat
Prenceing on Intent
5: "heoleg swning vieet th undinov on ins ownt the age with vewe tikke move in sotee ditiiiittee andeainn on fire theinng swilt eudio monumee ra could co averre er qutloo the dag beer
consta
the past nacoveaa
and geatriigh Hovover,
unioeen ploppissst our eirend the wase worns di at speid enluae suller
Boot Driwing Actioes
Move fanward with e decctarrtton This actir the age wehicle
varh cirtance from to protuatias and dog mantuuing
Dooclerriing ceveniiennl auchas the prooessoor sadion cnooiing ofen oodte the ownnn in ontor ta chilaaio
react
changes
ertton sectien E7rvection 125
pood pariiad
Oliaa the sumpees asenas requuiing addttined camaning
dlow Thinking
daytome with clsa via "terred amarted with Sevenal are viiiii, rocld ing white anc direcity hont the age withi iie and the partorl cara slong buth aldu cith
dertaal struet
cones, indicatin conatruction cona
Crttioal Olgnec Dnsccrption
White Trunko move chrecty troet oft egp veriiia, it aupparatt to most siciiiie ei shopping This vahide uchal as dictarae the floo the
Whail tie deing: Hoving slovle or sropping due construction oon szea. Whylt impertant: The age vehicde mu mai dhata and adjust
frutta Dessss Proatoo to eow oos chene mone cane mar costoatioe incensttation
Why they importarc ago
canitulyy to andld colia ois.
Whata Trank wnii rook au sionttyy iorto in the ago volic and spceessts ao slowing lo stupping ugu indi suil out speedi
2. Truft Gones The phaancc pfourt comes indicatera namowed lams, recuiing the age wet iole procere en uttuuaty and porartaal rnd rediaee apoed
ven sul lave bove risobaat spee the para i owan and canvirityy mone og veriitiody vas speen This siow ou ewns to right slgra rigliingg throug conerrct arns, Given theae obearattora, the ago vatiici ou contrnce deci and proceed cautioualy witile truck
mairzining
Mave fave and with seteeestten Thes sation was one teo vire di sullel dovipnes thro hurn the tn and are aingary noterzal hacands posed by the cores. viriino
aatir whale mnlaaai og ddde sccraias
raedeen The ria out aptinni dact wap-aa coin, Jic0_ <rasseer
Figure S9: Planning and reasoning results of ALt on the Waymo end-to-end driving dataset.
As of May 22, 2025, Aut achieves competitive performance on the Waymo End-to-Eii -ind Driving Challenge leaderboard, as shown in Table $3. The model ranks highly in both RFS Overall and ADE metrics and achieves the top score in the RFS Spotlight metric, which focuses on the most challenging scenarios. Qualitative results are shown in Fig. S9, where Auto demonstrates its ability to generate safe, context-aware trajectories in complex environments. The model effectively handles interactions, construction zones, and traffic regulations in diverse scenarios while providing coherent reasoning to justify its decisions.
36

---

Table S4: Ablation Study on the Waymo End-to-End Driving Test Set
Camera
Pretraining
Output
RFS (Overall) 1
ADE at 5s
Front Front
None
Action-only
6.938
3,595
None
CoT-enhanced
7.127
3,188
Multi
None
Action-only
7.239
3,243
Multi
None
CoT-enhanced
7.283
3.182
Multi
nuX
Action-only
7.406
3.116
Multi
nuX
CoT-enhanced
7.447
3.115
Post-RFT
7.557
2,958
Table S4 presents the ablation studies of the AutoVLA model in different training setups. The results indicate that multi-camera input consistently enhances driving performance. When trained solely on the Waymo end-to-end driving dataset, incorporating reasoning significantly improves perfor- mance compared to action-only setups. Moreover, pretraining on nuX data provides substantial performance boost, suggesting that such pretraining enhances the scene understanding of the model with more driving data. RFT can significantly improve planning performance by mitigating error accumulation in generation and aligning with the task-specific rewards.
F.4 CARLA Results
We evaluate the model after SFT in closed-loop testing within the CARLA simulator. The replanning frequency is set to 2 Hz, meaning the AutoVL.A model is queried every 0.5 seconds in simulation, and the planned trajectory is used to generate control commands. High-level driving instructions are derived from predefined route plan, while the vehicle's s current state (including speed and acceleration) obtained from the IMU and speedometer sensors. The model receives four RGB images from the front-camera sensor covering the past two seconds as visual input. Auto predicts a five-second trajectory, which is then used by a PID controller to compute the control actions (throttle, brake, and steering) that are applied the vehicle.
Closed-loop testing results in the CARLA simulator are shown in Fig. S10. Two representative scenarios are illustrated: (1) the ego vehicle equipped with AutoVLA successfully responds to cut-in vehicle, and (2) it executes smooth left turn. Additional closed-loop simulation results are available on the project website.
Irt3s
Figure $10: Closed-loop testing in the CARLA simulator. The two scenarios demonstrate VLt'' capability to (top) respond safely to cut-in vehicle and (bottom) execute smooth left turn.
G Broader Impacts
Autonomous driving a safety-critical system, further emphasized by the integration language guidance into the VLA model. This integration necessitates robust safeguards against adversarial attacks and proactive identification and filtering of unsafe human instructions. To mitigate potential hacking threats, it is essential to establish secure communication channel in the vehicle, comple- mented by a gated-release strategy for model updates rather than online continual reinforcement fine-tuning on individual vehicles.
37


Reviews:
[{"review_id": "6iV0zIQmI0", "reviewer": "Reviewer_5HFo", "summary": "This paper presents an end-to-end autoregressive model for autonomous driving. The proposed model is built upon a pre-trained VLM backbone that performs adaptive reasoning and predicts trajectories, where each trajectory is tokenized into a sequence of physically feasible tokens. The authors first adopt supervised fine-tuning to enable the model to perform both reasoning and planning tasks, and then apply reinforcement fine-tuning to train the model to avoid unnecessary reasoning. Experiments conducted on multiple datasets validate the proposed method.", "questions": "As noted in the weaknesses section, I would appreciate it if the authors could address the following questions:\n1. How do the authors justify the effectiveness of the proposed action tokenization method, given that it is only compared to a simple text action output?\n2. Why is RFT not used in closed-loop experiments? Could RFT improve performance on the nuScenes dataset when using CoT?\n3. Can the authors provide additional evidence to support the significance of the proposed framework", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and well-organized, and it is easy to follow and understand.\n2. The proposed VLA is a straightforward and technically sound extension of the VLM. The experiments are comprehensive and demonstrate that the method is promising in certain cases.\n3. The proposed adaptive reasoning is interesting and practical in terms of improving efficiency.\n\nWeaknesses:\n1. The ablation study on the text action output is not sufficient to demonstrate the effectiveness of the proposed action tokenization method. There are multiple advanced action tokenization schemes, such as per-dimension, per-timestep binning scheme [1] and [2], which are not compared.\n2. In reinforcement fine tuning, the authors formulate the planning problem as a VQA task, where the output actions are not executed and therefore do not affect subsequent observations. This formulation is misleading and potentially limits the model's closed-loop planning performance.It is also unclear why the authors did not use RFT in closed-loop experiments.\n3. The significance of the proposed framework is difficult to assess. Considering the model size and inference time, the model doesn't bring much improvements and sometimes performs worse in the benchmark evaluations. It would be helpful if the authors can provide the model sizes and FPS of all compared methods. Moreover, the proposed CoT training degrades performance in simple scenarios like nuScenes, which is quite counterintuitive. Regardless of efficiency, CoT should help the model to make better decisions, why does it hurt performance in this case? It is also unclear to me if the RFT can mitigate this issue.\n\n[1] Brohan et al. \"Rt-1: Robotics transformer for real-world control at scale.\" arXiv 2022. \n[2] Pertsch et al. \"Fast: Efficient action tokenization for vision-language-action models.\" RSS 2025.", "was_revised": true, "final_justification": "The work makes a certain contribution by presenting a straightforward extension of VLM for planning tasks, supported by extensive experiments. However, it still has limitations that cannot be easily addressed or overlooked, even though the authors have provided explanations. Therefore, I will keep my rating as borderline accept.", "rebuttal": "# 1. Action tokenization\nWe thank the reviewer for the comment regarding the evaluation of our action tokenization method. As suggested, we compare our tokenization approach with other advanced tokenization methods used in robotic control, including action bins in RT-1 [1] and the DCT-based method in FAST [2]. The table below summarizes the accuracy of each method. Specifically, we evaluate trajectory reconstruction accuracy (ADE/FDE) across 120K trajectories from the nuPlan dataset, using a consistent codebook size of 2048. \n\n[1] Brohan et al. \"RT-1: Robotics transformer for real-world control at scale.\" RSS 2023. \n\n[2] Pertsch et al. \"Fast: Efficient action tokenization for vision-language-action models.\" RSS 2025.\n\nIt is important to emphasize the structural differences between robotic manipulators (e.g., in RT-1 and FAST) and autonomous vehicles first. Robotic arms are typically fully actuated, holonomic systems capable of executing independent actions per dimension at each timestep. However, vehicles represent classical underactuated, nonholonomic systems with three degrees of freedom (i.e., position $x$, $y$, and $heading$) controlled indirectly via acceleration and steering rate. These systems are constrained by kinematics and cannot perform arbitrary motions. Therefore, our approach instead represents actions as short, feasible trajectory segments, encompassing position $x, y$, and $heading$, enabling the generation of smoother and physically feasible trajectories for vehicle systems.\n\nFor the RT-1 tokenization method, we discretize acceleration and steering rate using uniform action bins and apply a kinematic model to reconstruct the long-term trajectory. However, since we only have trajectory-level data and we have to indirectly derive control actions from it, this binning approach leads to the highest reconstruction error. In contrast, **our method achieves the lowest ADE and FDE, demonstrating its ability to represent the trajectory accurately**. FAST method performs comparably in terms of reconstruction accuracy, but it uses variable-length token sequences (typically 1 to 25 tokens per trajectory), which complicates the selection of appropriate token lengths and makes fixed-horizon trajectory prediction more challenging.\n\n| Tokenization      | ADE(m)↓          | FDE(m)↓        |\n|-------------------|-----------------|-----------------|\n| RT-1 (Action Bin)| 0.101           | 0.178           |\n| FAST              | 0.028           | 0.031           |\n| K-disk (Ours)     | **0.018**       | **0.020**       |\n\nWe also evaluate planning performance using the FAST tokenization method (without RFT) on the nuPlan (NAVSIM) data. The table below shows that **our method outperforms FAST in planning metrics, including PDMS, No At-fault Collision, and Ego Progress**. This is largely due to our tokens directly representing meaningful vehicle motions, whereas FAST tokens encode more abstract frequency components, which are harder to learn from limited data.\n\n| **Tokenization** | **PDMS↑**| **No At-Fault Collision↑**| **Progress↑**| \n|------------------|----------|---------------|--------------|\n| FAST             |    65.04      |   92.23           |  65.56          | \n| K-disk (Ours)    | **80.54**    | **96.89**         | **75.82**       |\n\n\n\n# 2. RFT in closed-loop experiments\nWe thank the reviewer for the insightful comment and agree that closed-loop RFT would provide a better assessment of long-term planning performance. However, closed-loop fine-tuning using existing open-source driving simulators (e.g., CARLA) is computationally prohibitive due to their limited simulation throughput and system bottlenecks. Furthermore,  the primary goal of our RFT approach is to improve trajectory planning accuracy and running efficiency, rather than to explicitly address robustness to distributional shifts. In future work, we plan to enhance the model architecture and develop more efficient simulation environments to enable closed-loop RFT and validate its capability.\n\n\n# 3. Benchmark evaluation and significance \nWe appreciate the reviewer’s concern regarding the significance of our proposed framework. We need to clarify that the primary goal of our work is to propose a general and scalable framework that can operate across a wide range of autonomous driving tasks, rather than optimizing for a single benchmark or dataset. Accordingly, **our evaluations are intentionally broad, covering diverse datasets such as nuScenes, nuPlan, Waymo, and CARLA, without relying on dataset-specific engineering or privileged modalities**. Our method consistently demonstrates competitive results across different datasets and benchmarks, which we argue is a meaningful indicator of its generalization ability.\n\nIt is important to note that many competing methods in the benchmarks are specialized and benefit from LiDAR sensors or privileged information (e.g., object-level annotations and map priors). In contrast, our approach is purely end-to-end, relying only on raw visual inputs and language and action supervision, which aligns with our goal of developing a more unified and scalable autonomous driving paradigm. \n\nMoreover, our results reveal clear data scaling trends: the performance of our model improves with increased data, suggesting that **its full potential has not yet been realized under the current limitations of available open-source datasets**. \n\n### Comparison of model sizes and FPS\nMany baseline methods do not report model parameters or FPS, making direct comparison difficult. We have included our model's size and runtime performance in the table below, and will add those results in the revised paper. We acknowledge that model size and inference speed are limitations. However, we note that the current system has not been optimized for efficiency, as our primary focus is to demonstrate the model's reasoning and planning capabilities. We also show that RFT enables adaptive thinking modes and improves performance and inference speed. \n\n| Methods       | Input               | Training                                                         | PDMS↑            | Model Size | FPS   | GPU          |\n|---------------|---------------------|------------------------------------------------------------------|-----------------|------------|-------|--------------|\n| TransFuser    | Cameras, LiDAR      | Object‑level Annotation, GT Trajectory, HD Map, Semantic Map     | 83.88           | 641.3 MB   | 6.67  | NVIDIA A100  |\n| DiffusionDrive| Cameras, LiDAR      | Object‑level Annotation, GT Trajectory, HD Map                   | 88.10           | –          | –     | –            |\n| Hydra‑MDP     | Cameras, LiDAR      | Object‑level Annotation, GT Trajectory, Scoring                  | 91.26           | –          | 2.80  | NVIDIA A100  |\n| Centaur       | Cameras, LiDAR      | Object‑level Annotation, GT Trajectory, Human Feedback           | 92.10           | –          | 3.58  | NVIDIA A100  |\n| TrajHF        | Cameras, LiDAR      | Object‑level Annotation, GT Trajectory, Human Feedback           | 93.95       | –          | –     | –            |\n| AutoVLA       | Cameras, Language   | GT Trajectory, CoT, Reward Feedback                              | 89.11           | 7.6 GB     | 0.77  | NVIDIA L40S  |\n\nNotably, our training method is also applicable to smaller VLMs, which can improve runtime performance in deployment. We also plan to explore LLM quantization in future work to enable near real-time inference even with slow reasoning.\n\n\n# 4. CoT training in nuScenes \nThank you for pointing out the performance degradation with CoT training. After careful examination, we found that this issue comes from the generation parameters used across different settings. For models trained with CoT, we adopted more diverse and random generation parameters (Temp=0.8, Top-p=0.5, Top-k=20) to encourage more reasoning outputs. In contrast, the action-only models employed more deterministic settings (Temp=0.01, Top-p=0.001, Top-k=1). However, this diversity-promoting generation parameter introduces increased randomness in inference, which negatively impacts trajectory accuracy metrics, such as L2 error on the nuScenes dataset. \n\nWe conducted an additional experiment to examine the influence of generation parameters. The results indicate that when CoT models are evaluated using the same generation parameters as the action-only models, the performance becomes comparable, with no significant difference. We will revise the paper accordingly to clarify this explanation and include the new experimental results.\n\n| Model       | Generation Parameter                | L2 Error(m)↓ | Collision Rate(%)↓ |\n|-------------|-------------------------------------|--------------|--------------------|\n| Action-only | Temp=0.01, Top-p=0.001, Top-k=1     | 0.70         | 0.31               |\n| Action-only | Temp=0.8, Top-p=0.5, Top-k=20       | 0.83         | 0.38               |\n| CoT         | Temp=0.8, Top-p=0.5, Top-k=20       | 0.85         | 0.35               |\n| CoT         | Temp=0.01, Top-p=0.001, Top-k=1     | 0.71         | 0.30               |\n\nRegarding the RFT in nuScenes, the core of RFT is verified reward functions. Unlike traditional displacement-based metrics (e.g., ADE), which merely enforce adherence to reference trajectories, the reward function necessitates comprehensive evaluation encompassing comfort, safety, and efficiency. The NAVSIM platform within nuPlan provides such a comprehensive evaluation platform with the PDMS metric. Conversely, nuScenes (designed for perception tasks) lacks a comparable planning evaluation framework, limiting the exploration of RFT. Additionally, nuPlan provides a larger and more diverse driving dataset (178K data), whereas nuScenes contains only 25K data from two similar urban locations as nuPlan (Boston and Las Vegas). Therefore, we argue that RFT's performance benefits can generalize from nuPlan to nuScenes dataset."}, {"review_id": "lEg04x2orT", "reviewer": "Reviewer_Tsqu", "summary": "The paper presents AutoVLA, an end-to-end vision–language–action (VLA) architecture for autonomous driving that (i) tokenises continuous vehicle motion into a learned “physical-action codebook” and feeds those tokens directly into a pretrained vision–language model (Qwen2.5-VL-3B) so that one autoregressive decoder jointly produces chain-of-thought (CoT) reasoning and a 5 s trajectory; (ii) trains with mixed supervision—trajectory-only (“fast thinking”) and trajectory + CoT (“slow thinking”); and (iii) applies reinforcement fine-tuning with Group-Relative Policy Optimisation (GRPO), using nuPlan PDMS and an explicit CoT-length penalty, to improve planning performance and suppress unnecessary reasoning.\nExperiments on nuPlan-NAVSIM, nuScenes, Waymo-E2E and CARLA Bench2Drive show competitive or SOTA open-loop and closed-loop metrics; ablations demonstrate gains from CoT, data scaling, action tokenisation and RFT.", "questions": "Please refer to the \"Weakness\"", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Contributes a 2048-entry motion codebook and integrates it as additional vocabulary inside the LLM, avoiding extra decoders (Sec. 3.1, Table 3). Produces physically feasible trajectories directly;  outperforming text-waypoint outputs in L2/collision and latency.\n2. Mixes reasoning-augmented and reasoning-free samples (Sec. 3.3) so the same model can adaptively emit CoT when needed (Fig. 6c). Ablation (Fig. 4) shows that with sufficient data (>50 k) CoT improves PDMS and collision metrics.\n3. Extends RFT beyond meta-actions (AlphaDrive) to full trajectory tokens (Sec. 3.4), achieving +10.6 % PDMS and −66 % runtime on NAVSIM (Fig. 6a). Provides evidence that penalising token length indeed triggers “fast thinking”.\n\n4. Four real/sim datasets;  both open-loop (nuPlan, nuScenes) and closed-loop (Bench2Drive). AutoVLA (Best-of-N) tops NAVSIM PDMS (92.1) and achieves best driving score & success rate on Bench2Drive (Table 2).\n\n Weaknesses\n1. The author only validates the effectiveness of AutoVLA in the autonomous driving scenario but does not mention the robotic scenario, which would more prominently highlight the importance of action tokenization mentioned in the paper.\n2. The size of the action codebook (K=2048) is empirically chosen, without conducting relevant ablation studies or verifying its dynamic feasibility.\n3. More meticulous and in-depth experiments and analyses are needed for the fast and slow systems. There are no relevant quantitative results in the main text, and the results in the appendix do not show significant advantages.", "was_revised": true, "final_justification": "The authors have provided analysis and explanations related to application scenarios, parameter experiments of the codebook, and ablation studies of the fast-slow system. However, they have not yet provided experiments in robotic scenarios, although this does not affect my judgment. I hope there could be some preliminary results in robotic scenarios to support the reliability of the method.", "rebuttal": "# 1. Robotic scenario\nWe thank the reviewer for raising this point. While our work focuses on autonomous driving, the underlying ideas of AutoVLA, are indeed inspired by and relevant to broader robotic scenarios. \n\nWe would like to clarify that the motivation for our action tokenization arises from the unique challenges posed by vehicle dynamics, which differ significantly from typical manipulation robots (e.g., RT-1 [1], FAST [2]). Most robotic platforms in those settings are holonomic and fully actuated, allowing for independent per-dimension action control at each timestep. In contrast, vehicles are underactuated, nonholonomic systems with kinematic constraints that prevent arbitrary motion, such as instantaneous lateral movements. \n\nTo address this, our action tokenization method encodes short, dynamically feasible trajectory segments in terms of position and heading, enabling smooth and physically consistent vehicle motion. Nevertheless, we agree that evaluating AutoVLA in general robotic scenarios would further highlight the broader utility of the proposed framework. Extending our framework to robot navigation represents a promising direction, and we plan to pursue this in future work.\n\n[1] Brohan et al. \"RT-1: Robotics transformer for real-world control at scale.\" RSS 2023. \n\n[2] Pertsch et al. \"Fast: Efficient action tokenization for vision-language-action models.\" RSS 2025.\n\n# 2. Action codebook size\nWe appreciate the reviewer’s question regarding codebook size. The codebook size $K=2048$ used in our AutoVLA framework was selected based on empirical evaluations that balance trajectory reconstruction accuracy with training stability. We conducted comparative evaluations of reconstruction accuracy using various codebook sizes across multiple tokenization methods, including RT-1, FAST, and our K-disk tokenization. The results demonstrate that our K-disk method consistently achieves the best reconstruction accuracy across different codebook sizes.\n\nWhen the codebook size is small ($K=256$), the reconstruction error is significantly higher due to limited expressiveness. As the codebook size increases beyond 1024, the improvements in accuracy become marginal, and the risk of redundant or overlapping motion patterns also increases. To strike a balance between accuracy and codebook efficiency, we selected $K=2048$ as a practical choice.\n \n| Tokenization | Codebook Size      | ADE(m)↓ | FDE(m)↓ |\n| ------------- | ----------------- | ------- | ------- |\n| RT-1 (Action Bin)    | 256           | 0.1440   | 0.2942   |\n| RT-1 (Action Bin)    | 1024          | 0.1052   | 0.1883   |\n| RT-1 (Action Bin)    | 2048          | 0.1014   | 0.1775   |\n| RT-1 (Action Bin)    | 4096          | 0.1001   | 0.1739   |\n| FAST (DCT)           | 256           | 0.1708   | 0.2137   |\n| FAST (DCT)           | 1024          | 0.0522   | 0.0588   |\n| FAST (DCT)           | 2048          | 0.0281   | 0.0309   |\n| FAST (DCT)           | 4096          | 0.0149   | 0.0161   |\n| K-disk (Ours)        | 256           | 0.0687   | 0.1034   |\n| K-disk (Ours)        | 1024          | 0.0253   | 0.0282   |\n| K-disk (Ours)        | 2048          | 0.0182   | 0.0203   |\n| K-disk (Ours)        | 4096          | 0.0141   | 0.0155   |\n\nWe also evaluated downstream performance using the PDMS metric on the nuPlan (NAVSIM) test data. Our results show that $K=2048$ yields the highest score among the testing configurations. While increasing the size from 1024 to 2048 leads to an improvement, expanding the size from 2048 to 4096 yields a marginal decrease. The result indicates that a codebook size of 2048 is sufficient to capture the majority of meaningful motion primitives, and that a larger codebook size introduces increased redundancy, making further expansion unnecessary. \n\nBased on analyses of trajectory reconstruction accuracy and driving performance across various $K$ values, we selected $K=2048$ as our codebook size.\n\n| Codebook Size | PDMS↑  |\n|---------------|-------|\n| 1024          | 79.47    |\n| **2048**      | **80.54** |\n|4096           | 80.39     |\n\nLastly, we emphasize that the discrete trajectory segments encoded in our codebook are derived from real-world driving data, inherently satisfying vehicle kinematic constraints. This ensures that the learned tokens are not only expressive but also physically feasible.\n\n\n# 3. Quantitative results for the fast-slow system\nWe thank the reviewer for highlighting the need for a more in-depth analysis of the fast-slow thinking system. We have conducted additional experiments and provided detailed quantitative results to support the effectiveness of our approach. The advantage of the fast-slow thinking system is its ability to maintain or improve planning performance while significantly reducing unnecessary slow reasoning, thereby enabling efficient inference. Our results in the main paper demonstrate that the RFT with the CoT length penalty training achieves better planning performance while substantially reducing average inference time.\n\nWe add results for RFT without the CoT length penalty, which shows slightly worse planning performance and inferior runtime efficiency due to excessively slow reasoning. This highlights the advantage of our training method for fast-slow systems: by incorporating a CoT length penalty, we guide the model to use slow reasoning only when necessary, achieving better overall running efficiency.\n\n| Model           | PDMS  | Avg. Runtime (s) |\n|-----------------|-------|-------------|\n|SFT               |          80.54          |   3.951  |\n| RFT (w/o CoT length penalty)   |  85.21  |   3.840  |\n| RFT (w/ CoT length penalty)    |  86.65  |   1.312  |\n\n\nWe further analyze the distribution of runtimes and mode usage across test samples. The results demonstrate that our proposed post-training method (RFT w/o CoT length penalty) significantly improves inference efficiency by reducing reliance on slow thinking while preserving reasoning capability when needed.\n\n| Method | Minimum (s) | Maximum (s) | Avg. Runtime (s) | Fast Thinking (%) | Slow Thinking (%) |\n|---------------|-------------|-------------|-------------|-------------|-------------|\n| SFT                              | 0.997       | 13.706       | 3.951       | 66.8       | 33.2       |\n| RFT (w/o CoT length penalty)     |  1.002      | 13.471      |  3.840      | 68.4        | 31.6       |\n| RFT (w/ CoT length penalty)      | 0.999       | 12.964       | 1.312       | 96.6       | 3.4       |\n\nWe also provide a breakdown of runtime by thinking mode across the testing samples. Fast thinking is significantly faster and more stable than slow thinking, while slow thinking, though more time-consuming, can be kept for complex scenarios.\n\n| Thinking Mode | Minimum (s) | Maximum (s) | Avg. Runtime (s) |\n|---------------|-------------|-------------|-------------|\n| Fast Thinking      | 0.997       | 1.116       | 1.072  |\n| Slow Thinking      | 7.607       | 13.706      | 10.518 |\n\nThese results demonstrate the advantage of our fast-slow thinking system. The ability to switch between fast and slow thinking enables AutoVLA to respond efficiently in straightforward scenarios while retaining the capacity for deeper reasoning when required."}, {"review_id": "vcRfnh3gH5", "reviewer": "Reviewer_xmFE", "summary": "The paper introduces AutoVLA, a novel Vision-Language-Action (VLA) framework for end-to-end autonomous driving. AutoVLA integrates reasoning and action generation within a single autoregressive model, supporting dual thinking modes: \"fast thinking\" for direct trajectory generation in straightforward scenarios and \"slow thinking\" with chain-of-thought (CoT) reasoning for complex situations. The training involves supervised finetuning (SFT) with trajectory and reasoning data, followed by reinforcement finetuning (RFT) to optimize planning performance and reduce unnecessary reasoning. The first highlight of the paper is its strong planning/driving performance on both real-world (nuPlan, nuScenes, Waymo) and simulated (CARLA) datasets. The second highlight is its ability to improve planning/driving performance when reasoning is involved, which is a capability not demonstrated by previous VLM-based frameworks.", "questions": "Overall, I think AutoVLA is a good paper. Please refer to the weakness section for questions. I only have two extra questions regarding the teacher model: how does the teacher model Qwen2.5-VL-72B perform on vehicles' ego status information? Are there any cases where the teacher model fails to make safe decision based on the textual velocity, acceleration, and historical actions?", "limitations": "Please refer to the weakness section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. AutoVLA unifies reasoning and action generation in a single model, addressing limitations of prior models that often produce infeasible actions or rely on complex intermediate representations.\n\n2. The model can dynamically switch between fast and slow thinking modes to balance efficiency and accuracy. This is a feature not explored in previous work.\n\n3. The paper evaluates AutoVLA across various benchmarks (nuPlan, nuScenes, Waymo, CARLA), demonstrating strong planning and driving performance in both open-loop and closed-loop settings. \n\n4. AutoVLA also achieves significantly stronger performance when incorporating reasoning in the language space, as shown in Figure 6 and Table 1. This suggests that the planning and language branches are closely related and complementary. This is a relationship that previous papers have not explored.\n\nWeaknesses:\n\n1. While the paper emphasizes the reasoning capabilities of AutoVLA, it does not quantitatively evaluate AutoVLA's performance on language-related benchmarks like DriveLM. Since DriveLM is also based on nuScenes and CARLA, including such results would further strengthen the paper and better validate the model’s reasoning ability.\n\n2. The previous problem raises a second question. The reinforcement finetuning (RFT) process penalizes long language outputs with the $r_\\text{CoT}$ term, which improves efficiency but may compromise the reasoning depth. A more systematic evaluation or additional case studies would help clarify the trade-offs involved.\n\n3. Since AutoVLA is trained on the Carla-Garage dataset for Table 2 according to Line 230 (which adopts PDMLite as the expert planner), it would be more appropriate to include comparisons with stronger baselines trained on PDMLite-gathered datasets (e.g., TF++ and SimLingo), rather than only comparing against weaker baselines trained on the Bench2Drive dataset.\n\n4. As the model employs a large vision encoder to process multiple camera views across several frames, this inevitably results in high computational cost and inference latency. A latency up to 1–2 seconds would make the model impractical for real-world deployment. It would also be helpful to report latency statistics in more detail (such as maximum, minimum, and average latencies for the two thinking modes) rather than only providing an average value for the entire dataset.", "was_revised": true, "final_justification": "The authors have provided detailed comparisons with other baselines and ablation studies addressing efficiency concerns, and I will therefore maintain my current rating of accept.", "rebuttal": "# 1. Evaluation on language-related benchmarks\nWe thank the reviewer for the suggestion of evaluating the model’s reasoning ability on language-related benchmarks. While we agree that this could offer additional insights, we would like to clarify that our model is not trained for VQA tasks typically assessed by DriveLM. Instead, we repurpose the DriveLM dataset to enable structured CoT reasoning in support of action planning. As a result, direct evaluation using the DriveLM benchmark is not well aligned with the functions of our model. \n\nMoreover, language reasoning performance in our setting is primarily determined by the underlying VLM and its training corpus. Thus, it does not serve as a key differentiator in our work. Our contribution lies in augmenting VLMs with planning capabilities for autonomous driving, rather than improving general language understanding.\n\n\n# 2. Evaluation of RFT on reasoning and planning\nWe thank the reviewer for raising this question regarding the trade-off between reasoning depth and efficiency in the RFT process. \n\nTo address the reviewer’s concern, we clarify that the CoT penalty during RFT is applied using a sigmoid-based normalization function, with a tolerance length $L_{tol}$ calibrated based on the average length of reasoning annotations in our training data. This design ensures that excessively long CoT outputs receive meaningful penalties, while preserving the flexibility for necessary CoT reasoning. The goal is to improve inference latency in time-sensitive planning tasks.\n\nTo evaluate the trade-off, we conducted an ablation study comparing the model’s performance with and without the CoT penalty. Due to the time limitation during the rebuttal period, we finished 1,500 training steps, and we compared the model with the same setting with a CoT length penalty. The results in the table below show that adding a CoT length penalty in RFT results in better planning performance and significantly reduces inference latency.\n\n| Model           | PDMS  | Avg. Runtime (s) |\n|-----------------|-------|-------------|\n| No CoT Penalty  | 85.21  |   3.840  |\n| CoT Penalty     | 86.65  |   1.312  |\n\nWe will include expanded results and analysis in the revised paper.\n\n\n# 3. Comparison with stronger methods on CARLA\nWe thank the reviewer for pointing out the need for more comparisons. We compare AutoVLA with methods trained on PDMLite-gathered datasets. We also carefully tuned the low-level controller, which we found to be a critical factor in tracking the planned trajectory. As a result, AutoVLA’s driving score improved from 78.84 to 80.12. Our updated results demonstrate that AutoVLA is comparable to these state-of-the-art models. \n\nSpecifically, while Transfuser++ employs LiDAR inputs and multimodal feature fusion, AutoVLA is a vision-only model. Similarly, SimLingo utilizes approximately 650k samples from different sources, whereas AutoVLA is trained with only 289k samples. Despite these differences, AutoVLA achieves comparable performance in closed-loop driving.\n\n| Method        | Driving Score | Success Rate | Efficiency | Comfortness |\n|---------------|---------------|--------------|------------|-------------|\n| TCP-traj      | 63.45         | 37.79        | 228.46     | 30.76       |\n| Transfuser++  | 84.21         | 67.27        | --         | --          |\n| SimLingo      | 85.07         | 67.27        | 259.23     | 33.67       |\n| AutoVLA       | 80.12         | 61.36        | 175.65     | 38.88       |\n\n\n# 4. Computational cost and inference latency\nWe acknowledge the concern regarding inference latency, which may be a limiting factor of our current system. However, we note that the current system has not been optimized for runtime efficiency, as our primary focus in this work is to demonstrate the model's reasoning and planning capabilities. We plan to apply model quantization and other techniques to improve inference speed, which have already been widely explored in LLM. We believe that with such runtime optimizations, the model can achieve near real-time performance even with slow reasoning.\n\nWe report more latency statistics over 1,500 test samples from the nuPlan dataset. The results show that the base SFT model relies more heavily on slow thinking, resulting in higher latency. Our proposed RFT method effectively reduces unnecessary slow reasoning while preserving its use in complex cases, leading to significantly improved runtime efficiency.\n\n| Method | Minimum (s) | Maximum (s) | Avg. Runtime (s) | Fast Thinking (%) | Slow Thinking (%) |\n|---------------|-------------|-------------|-------------|-------------|-------------|\n| SFT                          | 0.997       | 13.706       | 3.951       | 66.8       | 33.2       |\n| RFT (w/ CoT length penalty)  | 0.999       | 12.964       | 1.312       | 96.6       | 3.4       |\n\nAdditionally, we provide a breakdown of runtimes by thinking mode. The results show that fast thinking is significantly faster and more stable than slow thinking.\n\n| Thinking Mode | Minimum (s) | Maximum (s) | Avg. Runtime (s) |\n|---------------|-------------|-------------|-------------|\n| Fast Thinking      | 0.997       | 1.116       | 1.072  |\n| Slow Thinking      | 7.607       | 13.706      | 10.518 |\n\n# 5. Teacher model\nWe appreciate the reviewer’s suggestion to provide more details regarding our teacher model. Detailed descriptions of the teacher model and visualizations of the reasoning data annotations are included in the Supplementary Material. Our teacher model incorporates ego vehicle status (velocity, acceleration, historical actions) as language-based inputs. \n\nDue to Rebuttal time constraints, we randomly selected 60 Waymo scenarios to conduct an ablation study comparing inference accuracy with and without ego status input. Human verification revealed that excluding ego status decreased accuracy from 86.7% to 78.3%. Supplementary Material Figure S3 (scenario 1) exemplifies a situation where ego states are essential; specifically, the vehicle has been stopped at a stop sign, and it can accelerate and execute a left turn, rather than stop at the stop sign.\n\nFurthermore, our method integrates ground-truth driving actions as explicit hints to guide the generation of causal explanations. These ground-truth actions are generated by manually defined rules based on ground-truth trajectories. Consequently, the final decisions of our model can strictly adhere to ground-truth driving actions, achieving a human-verified accuracy of 93.6% across a randomly selected set of 3,862 Waymo scenarios. The error comes from the inherent limitation of the rule-based approach, particularly in handling complex multi-stage behaviors commonly encountered in challenging Waymo scenarios, such as accelerating from a stop sign and subsequently decelerating for pedestrians. Incorrect predictions identified during human verification are manually removed or revised to ensure accuracy."}, {"review_id": "1FGLavyK6o", "reviewer": "Reviewer_zoc9", "summary": "The authors propose a novel autoregressive framework for end-to-end automated driving, by using a VLA with tokenized and discretized actions. The actions were tokenized to remove the possibility of impossible actions. The authors' model works with a dual-thinking approach where slow-thinking produces a chain-of-though (CoT) as well as a set of action tokens for the next 5 seconds of planning. Meanwhile quick-thinking skips CoT and just returns a set of action tokens. The authors first train using a supervised fine-tuning approach in which they use both ground truth for planning as well as reasoning data for CoT, where reasoning data was created as distilled knowledge from a larger VLM. After SFT the authors propose an RFT approach using GRPO to enhance model efficacy and performance. The model was tested on multiple benchmarks and shows performant results.", "questions": "1. Please provide a more detailed explanation as to the exact procedure of tokenizing the action space. Why do you tokenize in such a way? Why do you use K-disk and not something else? Why do you use short-form why not a different approach etc.?\n2. Why is the method worse in some categories than other benchmark models? Please discuss and analyze the results, not only present them.", "limitations": "The limitations subsection only discusses the real-world feasibility of the method, while not touching up any other limitations of the model or discussing the results of experiments that were on average below the other methods in some cases by a larger margin.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "1. Quality - The paper is well written and all the claims the paper formulated are properly addressed and argumented. The methods that were used to test the model are appropriate and in line with traditional benchmarks. The paper does not provide a performance test, as it is only mentioned once, but no further analysis was done, only stating that it achieves near real-time performance using many GPUs. The results of experiments are just presented as is, without any real discussion other than putting into words what is said on the graphs. No deeper understanding of results, or discussing potential drawbacks of the method.\n2. Clarity - While the main methodological and experimental approaches are defined precisely, the main drawback I see here is that the actual process of discretizing and tokenizing the actions is not explained in detail as other parts which are less significant as this is one of the main contributions of the paper. Giving a better overview as to how it was discretized, why the specific use of short-term spatial positions, why K-disk clustering (and how it was set up) etc. would benefit the paper greatly.\n3. Significance - The results are promising but not as good as other simpler methods provided in benchmarks competing against the proposed model, in all but one category when compared to other models is this proposed model better, and that is when using best-of-N planning which other models don't have access to. Meanwhile the one-shot and post-RFT models are quite underperformant compared to others, averaging the lower half of metrics when ranked against provided models.\n4. Originality - The main insight from my perspective that the paper provides is that when tokenizing the action space there can be no nonsensical outputs. Which is more of a product of the method than an insight, since there are only feasible actions the model can only output feasible actions. Other parts of the model, pipeline and framework are not novel and do not provide new insights, but are rather logical results of steps taken in the paper. The paper does provide an overview of a framework though, but there is no real scientific insight gained here, more an engineering assessment of a framework, that in the end does not provide any inherent benefits over others performance-wise.", "was_revised": true, "final_justification": "Although the authors coud clarify several of ours and the other reviewers questions, the main issue of low novelty and questionable significance, as remarked also by another reviewer, remain. Thus I keep my rating.", "rebuttal": "# 1. Performance test and discussions\nWe thank the reviewer for highlighting the need for a more thorough analysis and discussion of the experimental results. We note that we have provided a more detailed analysis in the supplementary materials, and we provide additional analysis and discussions below.\n\nFirst, we would like to clarify that our near real-time inference results were obtained using a single GPU, not multiple GPUs as the reviewer noted. In the following sections, we include a detailed summary of model size, inference speed (in FPS), and hardware configurations, in comparison with other methods.\n\nFurther, following the reviewer’s suggestion, we conducted additional experiments to analyze runtime performance under various training settings. The results show that our proposed post-training method (RFT with CoT length penalty) can significantly improve inference efficiency by reducing slow thinking. These results show that RFT effectively shifts model behavior toward faster inference modes without sacrificing planning performance. \n\n| Method | Minimum (s) | Maximum (s) | Avg. Runtime (s) | Fast Thinking (%) | Slow Thinking (%) |\n|---------------|-------------|-------------|-------------|-------------|-------------|\n| SFT  | 0.997       | 13.706       | 3.951       | 66.8       | 33.2       |\n| RFT (w/o CoT length penalty)     |  1.002      | 13.471      |  3.840      | 68.4        | 31.6      |\n| RFT (w/ CoT length penalty)      | 0.999       | 12.964       | 1.312       | 96.6     | 3.4       |\n\nWe also break down runtimes by thinking mode. As shown in the table below, fast thinking yields much faster and more stable inference compared to slow thinking. Our training approach encourages the model to use fast thinking in simple scenarios and reserve slow reasoning only when necessary.\n\n| Thinking Mode | Min Runtime (s) | Max Runtime (s) | Avg. Runtime (s) |\n|---------------|-------------|-------------|-------------|\n| Fast Thinking      | 0.997       | 1.116       | 1.072       |\n| Slow Thinking      | 7.607       | 13.706      | 10.518 |\n\n\n# 2. Process of discretizing and tokenizing actions\nWe thank the reviewer for pointing out the need for greater clarity regarding our action tokenization process. We have provided more implementation details in the supplementary materials. To address the reviewer's comments, we include the following elaborations:\n1) Discretization Strategy: We want to emphasize the structural differences between robotic manipulators and autonomous vehicles first. Robot arms are fully actuated, holonomic systems capable of executing independent actions per dimension. However, vehicles represent classical underactuated, nonholonomic systems with three degrees of freedom (i.e., position $x$, $y$, and $heading$) controlled indirectly via acceleration and steering rate. Therefore, we discretize the action space by representing each action token as a short-term, kinematically feasible trajectory segment (0.5 seconds). This tokenized representation encodes motion primitives that are reachable under nonholonomic constraints, serving as building blocks for long-term trajectories.\n2) Use of Short-term Spatial Positions: Short-term trajectory segments provide a compact and expressive representation of immediate vehicle movement, which aligns well with the autoregressive generation of our model. This strategy reduces the complexity of long-horizon prediction and improves training efficiency.\n3) Why K-Disk Clustering: This method groups similar short-term state transitions into discrete tokens while preserving continuity and ensuring dense coverage of diverse motion patterns across three degrees of freedom (i.e., position $x$, $y$, and $heading$). \n\n[1] Brohan et al. \"RT-1: Robotics transformer for real-world control at scale.\" RSS 2023. \n\n[2] Pertsch et al. \"Fast: Efficient action tokenization for vision-language-action models.\" RSS 2025.\n\nWe compared our K-disk tokenization against RT-1 (action bins) and FAST (tokenizing trajectories based on discrete cosine transform) using a fixed codebook size of 2048. The results below report the accuracy of trajectory reconstruction based on ground-truth tokens. Our method achieves the lowest reconstruction error.\n\n| Tokenization      | ADE(m)↓          | FDE(m)↓        |\n|-------------------|-----------------|-----------------|\n| RT-1 (Action Bins)| 0.101           | 0.178           |\n| FAST              | 0.028           | 0.031           |\n| K-disk (Ours)     | **0.018**       | **0.020**       |\n\nTo assess downstream impact, we compare our model with the FAST tokenization method (without RFT) on the nuPlan (NAVSIM) test data. The table below shows that our method significantly outperforms FAST in planning metrics. This is largely due to our tokens directly representing physically meaningful vehicle motions, whereas FAST tokens encode more abstract frequency components, which are harder to learn from limited data.\n\n| **Tokenization** | **PDMS↑**| **No At-Fault Collision↑**| **Progress↑**| \n|------------------|----------|---------------|--------------|\n| FAST             |    65.04      |   92.23           |  65.56          | \n| Ours              | **80.54**    | **96.89**         | **75.82**       |\n\nThus, our discretization and tokenization strategy not only improves trajectory reconstruction accuracy but also translates into better planning performance.\n\n# 3. Analysis of benchmark results\nWe thank the reviewer for their feedback on the significance of our results. While we acknowledge that our model does not achieve absolute SOTA performance, we would like to clarify several key aspects of our framework that we believe underscore its contribution and broader significance.\n+ **Generality and Simplicity**. Unlike prior works that are often optimized for specific benchmarks and benefit from privileged modalities such as LiDAR, HD maps, and object-level annotations, our framework is designed to be **general, scalable, and end-to-end, relying only on camera inputs, language, and trajectory supervision**. We have listed the comparison of different methods in the nuPlan (NAVSIM) benchmark in the table below. Despite the absence of such privileged inputs, our model achieves competitive results in the benchmark without dataset-specific engineering. In addition, our model is competitive across multiple datasets and tasks (nuPlan, nuScenes, Waymo, and CARLA). We consider this cross-domain consistency a key contribution. Our results are also promising and suggest the viability of a unified formulation of planning and reasoning in autonomous systems.\n+ **Scalability with Data**. We observe clear data scaling trends in our experiments, indicating that model performance improves substantially with increased training data. Due to the **limited size of available open-source datasets, we believe that the full potential of our approach has not yet been realized**. \n\n**Best-of-N Analysis and RFT**. The best-of-N planning evaluation is intended to illustrate the performance ceiling achievable by leveraging the model’s inherent trajectory diversity. Although competing baselines do not include best-of-N evaluation, we emphasize that **our RFT method achieves performance close to this upper bound while operating in a single-shot manner**. \n\n| Methods        | Input             | Training                                                     | PDMS↑ |\n| -------------- | ----------------- | ------------------------------------------------------------ | ----- |\n| TransFuser     | Cameras, LiDAR    | Object‑level Annotation, GT Trajectory, HD Map, Semantic Map | 83.88 |\n| DiffusionDrive | Cameras, LiDAR    | Object‑level Annotation, GT Trajectory, HD Map             | 88.10 |\n| Hydra‑MDP      | Cameras, LiDAR    | Object‑level Annotation, GT Trajectory, Scoring            | 91.26 |\n| Centaur        | Cameras, LiDAR    | Object‑level Annotation, GT Trajectory, Human Feedback       | 92.10 |\n| TrajHF         | Cameras, LiDAR    | Object‑level Annotation, GT Trajectory, Human Feedback       | 93.95 |\n| AutoVLA        | Cameras, Language | GT Trajectory, CoT, Reward Feedback                          | 89.11 |\n\n\n# 4. Novelty\nWe appreciate the reviewer for the opportunity to clarify the contributions and insights presented in our work. We respectfully argue that the originality of our paper lies not only in these action tokenization methods but also in the following key contributions:\n1) **Unified Vision-Language-Action Planning Framework**: Our framework bridges vision, language, and action in a unified learning setup for autonomous driving tasks, where discrete, physically-plausible trajectory tokens form the basis for autoregressive prediction. Although individual components (e.g., trajectory learning or language reasoning) have been studied in isolation, their integration under the VLA paradigm is novel and offers new potential for end-to-end, generalizable planning.\n2) **Reinforcement Fine-Tuning for Trajectory Planning**: To the best of our knowledge, this is the first application of GRPO for trajectory-level policy refinement in autonomous driving with vision-language models. While RFT has been extensively explored in natural language generation, our work demonstrates that it can also be effectively applied to trajectory prediction and refinement, yielding measurable improvements in planning quality. This establishes a new direction for aligning planning policies with preference-based or task-driven feedback.\n3) **Fast-Slow Planning Paradigm**: We provide new insights into how SFT (mixture thinking-mode training) and RFT can be leveraged to enable an adaptive fast-slow planning paradigm for VLA models. Empirically, this strategy improves overall planning performance and significantly enhances runtime efficiency, as the fast-thinking mode can suppress language-based reasoning outputs in straightforward scenarios, but keep reasoning capabilities in challenging situations."}]

Author rebuttal:

Meta-review: This paper proposes AutoVLA, a vision-language-action framework for end-to-end autonomous driving that integrates trajectory planning and reasoning in a single autoregressive model. The authors introduce action tokenization, dual thinking modes (fast and slow), and reinforcement fine-tuning to improve efficiency and adaptability.

The final ratings are mixed, ranging from reject to accept (1 reject, 2 borderline accepts, 1 accept). Reviewer xmFE (accept) acknowledged the unified reasoning-planning design and found the results strong on several benchmarks. Reviewer Tsqu (borderline accept) appreciated the design of action tokenization and RFT, but requested further evaluation in broader robotic scenarios.  Reviewers 5HFo (borderline accept) and zoc9 (reject) remained concerned after the rebuttal, particularly regarding the novelty, practical impact, and inference efficiency of the system.

Overall, the AC finds this paper a good attempt to integrate large vision language models into end-to-end autonomous driving, and a well-engineered system with comprehensive experiments across multiple benchmarks. However, concerns remain regarding its practical significance - given the large model size, different from previous fast-slow systems with two separate models, the dual-thinking design still lacks real-time applicability. While some concerns remain, the paper still presents a well-executed contribution toward applying VLA in autonomous driving, and the AC recommends acceptance.
Decision: Accept (poster)