Title: AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: End-to-end Autonomous Driving

Vision Language Action Model

Reinforcement Finetuning
Abstract: Recent advancements in Vision-Language-Action (VLA) models have shown promise for end-to-end autonomous driving by leveraging world knowledge and reasoning capabilities. However, current VLA models often struggle with physically infeasible action outputs, complex model structures, or unnecessarily long reasoning. In this paper, we propose AutoVLA, a novel VLA model that unifies reasoning and action generation within a single autoregressive generation model for end-to-end autonomous driving. AutoVLA performs semantic reasoning and trajectory planning directly from raw visual inputs and language instructions. We tokenize continuous trajectories into discrete, feasible actions, enabling direct integration into the language model. For training, we employ supervised fine-tuning to equip the model with dual thinking modes: fast thinking (trajectory-only) and slow thinking (enhanced with chain-of-thought reasoning). To further enhance planning performance and efficiency, we introduce a reinforcement fine-tuning method based on Group Relative Policy Optimization (GRPO), reducing unnecessary reasoning in straightforward scenarios. Extensive experiments across real-world and simulated datasets and benchmarks, including nuPlan, nuScenes, Waymo, and CARLA, demonstrate the competitive performance of AutoVLA in both open-loop and closed-loop settings. Qualitative results showcase the adaptive reasoning and accurate planning capabilities of AutoVLA in diverse scenarios.

AutoVLA : A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning

Abstract

Recent advancements in Vision-Language-Action (VLA) models have shown promise for end-to-end autonomous driving by leveraging world knowledge and reasoning capabilities. However, current VLA models often struggle with physically infeasible action outputs, complex model structures, or unnecessarily long reasoning. In this paper, we propose AutoVLA , a novel VLA model that unifies reasoning and action generation within a single autoregressive generation model for end-to-end autonomous driving. AutoVLA performs semantic reasoning and trajectory planning directly from raw visual inputs and language instructions. We tokenize continuous trajectories into discrete, feasible actions, enabling direct integration into the language model. For training, we employ supervised fine-tuning to equip the model with dual thinking modes: fast thinking (trajectory-only) and slow thinking (enhanced with chain-of-thought reasoning). To further enhance planning performance and efficiency, we introduce a reinforcement fine-tuning method based on Group Relative Policy Optimization (GRPO), reducing unnecessary reasoning in straightforward scenarios. Extensive experiments across real-world and simulated datasets and benchmarks, including nuPlan, nuScenes, Waymo, and CARLA, demonstrate the competitive performance of AutoVLA in both open-loop and closed-loop settings. Qualitative results showcase the adaptive reasoning and accurate planning capabilities of AutoVLA in diverse scenarios.

1 Introduction

Autonomous driving systems typically adopt a modular paradigm, decomposing the driving task into different sub-modules, such as perception [1-3], prediction [4-6], and planning [7-9]. While this design enables structured development, it may cause error accumulation and a lack of joint optimization across modules, leading to suboptimal performance [10, 11]. End-to-end autonomous driving has gained prominence with a unified model architecture that maps raw sensor inputs directly to final driving actions. These models are trained on human driving data, enhancing scalability and human-like behavior. Vision-based approaches have garnered significant interest due to their affordability and ease of deployment [12-15].

However, conventional end-to-end methods [16-19] primarily focus on imitating expert trajectories, lacking essential world knowledge for understanding and reasoning about surrounding environments, particularly in long-tail or challenging scenarios. Recent advances in Vision-Language Models (VLMs) [20-22] have gained significant interest by introducing models capable of leveraging extensive world knowledge and powerful reasoning. These models have shown strong potential in

in Simple Scenarios

in Complex Scenarios

LLM

🔥

🔥

Supervised Fine-tuning

Reinforcement Fine-tuning

Reasoning

Data

GRPO

Dual Thinking Mode Adaptation

SFT

RFT

Reward

Model

GT

Trajectory

Turn Left | Go Straight …

Instruction

Camera streams

Vision

Encoder

Text

Tokenizer

…

…

Auto Vision

Language

Action Model

Velocity | History Action …

Ego states

Task & Thinking Guidance

System Prompt

V-Vision Space

L-Language Space

A-Action Space

Planning

Text Token

Vision Token

Vision Encoder

❄

❄

LoRA

🔥

LLM

…

Action Token

Action

Codebook

…

t = 0

t = 1

t = 2

t = 3

p=0.4

p=0.3

p=0.2

p=0.2

p=0.2

p=0.1

p=0.1

p=0.2

p=0.1

p=0.1

p=0.2

p=0.3

p=0.2

p=0.1

p=0.1

Final Trajectory

Not in Top-K

Top-K Token

Go Straight

The ego vehicle is approaching a construction zone during daylight …

-

Front:

A construction worker is in the center lane, holding a SLOW

paddle, and a construction vehicle is blocking the right lane …

-

Left:

The left lane is clear, with a few vehicles ahead .…

-

Back:

Light traAic is present at a distance behind the ego vehicle …

2. Critical Object Description

-

Construction Worker (center front): Holding a

SLOW sign

and

gesturing, likely managing traAic flow around a construction zone …

-

Forklift/Loader (front right): Stationary and blocking the right lane…

3. Reasoning on Intent

The ego vehicle was previously stopped and is now beginning to move

slowly (velocity ≈ 0.44 m/s, low acceleration). The driving command is

GO_STRAIGHT, but the direct path is blocked by both the construction

worker and the forklift. Meanwhile, the left lane appears clear and

viable, with enough space to safely merge.

4. Best Driving Action

: change lane to the left with an acceleration.

Chain of Thought

三

improving adaptability and scalability across diverse driving scenarios [23-29]. Building upon VLMs, Vision-Language-Action (VLA) models extend this capability to action generation, enabling embodied agents, such as robots [30-32] and autonomous vehicles [33, 34], to produce feasible physical actions based on visual observations and language instructions.

Despite recent progress, existing VLA models face two critical limitations in autonomous driving, as illustrated in Fig. 2. 1) Physically-infeasible or complex structure for action generation. Some models generate textual actions or waypoints directly using VLMs [35-37], but these outputs can be physically infeasible and suffer from mode collapse. To address this, recent approaches introduce intermediate meta-actions [38-40] or latent action tokens [41-43], which are then processed by downstream planners or decoders to produce physically feasible trajectories. However, the intermediate representations either break the end-to-end optimization paradigm or increase model complexity and training overhead. 2) Inflexible and inefficient reasoning across diverse scenarios. Most existing models [44, 45] employ a fixed reasoning strategy, lacking the ability to adaptively switch between direct action outputs for straightforward scenarios and chain-of-thought (CoT) reasoning for complex ones. Although DriveVLM [46] introduces a dual-process paradigm, it relies on separate modules (i.e., a VLM for slow reasoning and a conventional end-to-end model for fast responses), which results in a complicated architecture, increased training overhead, and limited scalability [47].

To overcome these limitations, we propose AutoVLA , an end-to-end autonomous driving framework that directly integrates physical action tokens into a pretrained VLM backbone, enabling direct learning of an autoregressive planning policy, as illustrated in Fig. 1. Our unified architecture seamlessly integrates reasoning and action generation, allowing adaptive switching between direct trajectory generation and CoT reasoning. In supervised fine-tuning (SFT), we leverage both trajectoryonly data and CoT reasoning data to equip the model with dual-process capabilities (fast and slow thinking). Furthermore, we propose reinforcement fine-tuning (RFT) [48], utilizing Group Relative Policy Optimization (GRPO) [49] with verifiable planning reward functions. This enables adaptive reasoning that balances planning accuracy and efficiency. The RFT method not only improves planning performance but also runtime efficiency by minimizing unnecessary reasoning.

We extensively evaluate AutoVLA using real-world datasets, including nuPlan [50, 51], Waymo [52], nuScenes [53], and simulation datasets such as CARLA [54, 55]. Experimental results demonstrate that AutoVLA achieves superior performance across various end-to-end autonomous driving benchmarks under both open-loop and closed-loop tests. Empirical results validate that our RFT approach

Redundant Reasoning

in Simple Scenarios

Conventional End-to-end Model

Vision Language Model

Planning

Explanation

Vision Language Model

Conventional End-to-end Model

High-level Decision

Hierarchical

System

Vision Language Model

Planning

Vision Language Model

Planning

CoT

Reasoning

Dual System

Action Tokenization

CoT

Reasoning

Infeasible

Trajectories

Thinking Mode Adaptation

(b) Hybrid VLM End-to-end Model

(a) Dual VLM End-to-end Model

(c) VLM as End-to-end Model

(d) VLA as End-to-end Model

Text

Images

Images

Images

Images

Images

Images

Images

Images

Images

Images

Images

Images

Action Space

Vision Space

Planning

markedly improves planning performance, enables adaptive fast and slow thinking capabilities, and reduces runtime by minimizing redundant reasoning. The main contributions of this paper are summarized as follows:

We introduce AutoVLA, an end-to-end autonomous driving framework leveraging a pretrained VLM backbone integrated with physical action tokens, enabling direct policy learning and semantic reasoning from raw visual observations and language instructions.

We propose an RL-based post-training method using GRPO to enable adaptive reasoning and further enhance the model's performance on end-to-end driving tasks.

We demonstrate that AutoVLA achieves superior performance across multiple autonomous driving benchmarks, including both open-loop and closed-loop testing.

2 Related Work

End-to-end Autonomous Driving. End-to-end autonomous driving approaches have made significant advances in recent years [10, 11, 56-64]. Methods such as UniAD [65] and VAD [66] explicitly integrate multiple driving tasks from perception to planning in a unified Transformer architecture, thereby enhancing planning performance. ParaDrive [67] discusses the necessary components within end-to-end driving architectures. Additionally, GenAD [68] and DiffusionDrive [69] adopt generative models to maintain trajectory continuity and produce multi-modal driving trajectories. However, integrating world knowledge into end-to-end driving systems remains challenging due to bottlenecks in semantic reasoning [34] and limited adaptability in complex environments [70].

VLAand VLM for Autonomous Driving. The gap between semantic reasoning and physical actions remains a critical challenge for VLA and VLM in end-to-end autonomous driving. Current research broadly follows three directions. The first directly formulates driving as a language-centric problem, utilizing VLMs for scenario understanding through caption generation [71-73] or question answering [74, 75]. The second direction leverages VLA or VLM to produce high-level meta-actions or driving decisions [17, 38-40], which are used to either supervise [12, 76-78] or guide [39, 79] traditional planners or end-to-end models. Although these approaches facilitate integration, they prevent full end-to-end optimization. Thus, a third direction directly integrates VLMs with action generation into VLA models, enabling the direct prediction of latent action tokens [34-36, 43] or final driving trajectories [37, 44, 80-83]. However, simple trajectory decoders employed in these methods (e.g., MLP [41, 84] or GRU [45]) may produce impractical trajectories and suffer from modal collapse. To address this issue, ORION [42] incorporates generative planners into VLM architectures, enhancing trajectory feasibility but increasing model complexity and computational demands. In our work, we integrate a physical action codebook for vehicle motion into a pretrained VLM to effectively bridge the semantic reasoning and physical action space.

Reinforcement Fine-tuning. RFT [48] has shown considerable promise in enhancing the performance and adaptability of LLMs, as demonstrated in DeepSeek-R1 [22]. In autonomous driving,

…

Ego states

Instruction

Turn Left | Go Straight | Turn Right

Velocity | Acceleration | History Action

…

…

…

Action Token

Text Tokenizer

System Prompt

- Role:

You are an advanced full self-driving system.

- Task:

You will be provided with video observations from

the ego vehicle's surrounding cameras, along with the

vehicle's current dynamic states. Your task is to predict

the most appropriate driving action for the next 5 seconds.

- Fast/Slow Thinking:

If necessary, use step-by-step

reasoning (Chain-of-Thought) ... Otherwise, you may

directly predict the final driving action.

…

Vision Encoder

Qwen2.5-VL-3B

- Scene Analysis

- Critical Object Identification

- Intention Reasoning

- Final Action Decision

Slow Thinking

Fast Thinking

<think>  This  is  a  complex  scenario  requiring

additional  reasoning…  The  ego  vehicle  should

wait  for  the  vehicle  on  the  right  to  pass  before

proceeding straight with caution.</think>

<answer><action_0><action_0><action_0><acti

on_18><action_22><action_26><action_35><ac

tion_32><action_42><action_38></answer>

Trajectory 1

<think> This is a complex scenario requiring

additional reasoning… The vehicle ahead is

slow-moving or possibly stopped. The ego

vehicle should change to the left lane.</think>

<answer><action_12><action_23><action_14><

action_89><action_17><action_33><action_86>

<action_75><action_67><action_83></answer>

<think> This is a straightforward scenario, and a

direct decision can be made...</think>

<answer><action_22><action_12><action_26><

action_34><action_31><action_33><action_68>

<action_53><action_42><action_58></answer>

Reward

S1

: 0.92

S2

: 0.00

S3

: 0.64

Group Relative

Policy Optimization

…

Multi-view

Supervised

Fine-tuning

Reasoning Data

GT Trajectory

Action

Reasoning

Reinforcement Fine-tuning

Sample 1 |

Sample 2 |

Slow Thinking

Sample 3 |

Trajectory 2

Trajectory 3

-

Safety & Comfort

-

Penalty for slow thinking

-

Travel progress …

Reward

Model

Qwen2.5-VL-72B

Knowledge

Distillation

Large Vision

GT Decision

Same

Inputs

1品

X

Gen-Drive [7] and TrajHF [85] employed the RFT to align the trajectory generation model with safety constraints and human driving preferences. RAD [86] combined 3D Gaussian splatting to generate scenarios and conduct closed-loop RL training. However, the application of RFT in endto-end VLM/VLA-based autonomous driving remains nascent. While previous methods, such as AlphaDrive [38], utilize GRPO instead of direct preference optimization (DPO) [87] to enhance planning performance and ensure training efficiency and stability, they are still limited to simplified settings involving only high-level meta-actions. In this work, we advance this direction by applying RFT to optimize the end-to-end VLA framework in both scene reasoning and low-level planning, and we adopt GRPO to accelerate convergence and enhance training stability.

3 AutoVLA

The proposed AutoVLA framework consists of two main components, as shown in Fig. 1. 1) VLM Backbone: It is capable of processing visual and textual input and generating corresponding tokens (reasoning and action), employing a unified autoregressive Transformer decoder. 2) Physical Action Token Generation: We extend the language model decoder to output physical action tokens that directly correspond to vehicle movements. These tokens are designed to comply with physical constraints and can be reliably translated into physically feasible planning trajectories.

Training of AutoVLA is conducted in two stages, as illustrated in Fig. 3. 1) Supervised Fine-Tuning uses ground-truth trajectory data and distills high-quality reasoning data from a large-scale VLM. 2) Reinforcement Fine-Tuning uses task-specific reward functions to optimize planning performance while improving the running efficiency by minimizing unnecessary reasoning. The details of our model and training process are illustrated below.

3.1 Framework

Model Inputs. AutoVLA takes as input multi-view, multi-frame camera data C from onboard cameras, high-level navigation instructions I , and ego vehicle states S , and performs scene reasoning

and trajectory planning. Specifically, we utilize three RGB cameras positioned at the front, front-left, and front-right sides of the vehicle. Each camera stream c i = [ c i t -3 , c i t -2 , c i t -1 , c i t ] captures four sequential frames at a frequency of 2 Hz, including the current and three preceding frames, supplying temporal information for scene dynamics. Additionally, the model employs high-level navigation instructions I (e.g., Turn Left and Go Straight) to specify intended directions explicitly. The ego vehicle's state S encompasses current velocity, acceleration, and historical actions.

Base VLM Model. We adopt Qwen2.5-VL-3B [21] as the vision-language backbone of AutoVLA. Qwen2.5-VL is a series of powerful multimodal large language models that possess strong visual understanding capabilities, and the open-source nature of the Qwen2.5-VL model facilitates taskspecific fine-tuning. The 3B variant offers a good trade-off between efficiency and performance, making it suitable for deployment in onboard devices.

Action Tokenization. To enable trajectory planning within the language model, we discretize continuous vehicle trajectories P ∈ R τ × d into a sequence of physical action tokens a = [ a 1 , . . . , a T ] , where a t ∈ A , T is the length of the tokenized predicted trajectory and each token is represented by short-term spatial position and heading movement (∆ x, ∆ y, ∆ θ ) . This transforms the planning task into a next-token prediction problem, which can be conducted within the language model. We build our action codebook A = { a 1 , a 2 , . . . , a K } using a K-disk clustering method [88-90], which covers the majority of vehicle movement patterns. Finally, we obtain a vehicle motion codebook that consists of K = 2048 discrete action tokens. Following [30, 91], these action tokens are incorporated into the VLM as additional tokens (i.e., <action_0> , <action_1> , . . . ). During inference, the model outputs a sequence of these action tokens, which are subsequently decoded into a planning trajectory using the action codebook. More details about action tokenization and trajectory decoding are provided in the Supplementary Material.

Unified Reasoning and Action. AutoVLA unifies reasoning and action generation within a single autoregressive Transformer framework, enabling adaptive switching between fast and slow thinking depending on the driving scenario. In fast thinking mode, AutoVLA directly predicts physical action tokens without generating long CoT reasoning, enabling rapid responses in straightforward scenarios. In contrast, slow thinking mode involves structured CoT reasoning, where the model first analyzes the environment, identifies critical elements, and reasons through potential outcomes before deciding on the final driving action. To enable this dual thinking capability, AutoVLA is trained with a mixture of direct action supervision and reasoning-augmented data. We design system prompts and response formats to support both modes consistently.

3.2 Reasoning Data

Reasoning data provides high-quality CoT annotations that are essential for training VLMs with reasoning capabilities [42]. In driving tasks, reasoning involves understanding complex semantics and interactions in dynamic environments [92-95]. Despite its importance, the development of a high-quality, large-scale driving reasoning dataset remains a key challenge due to three major limitations: 1) limited scenario diversity and repetitive examples, 2) inadequate representation of critical perceptual cues, such as traffic signs and vehicle indicator signals, 3) low-quality reasoning process, such as repeatedly stopping at a stop sign without justification.

To address these issues, we propose an automated reasoning annotation pipeline using the advanced Qwen2.5-VL-72B model [21]. This pipeline enables automatic generation of high-accuracy reasoning annotations and supports knowledge distillation from a large capable model to a more compact target model. The pipeline generates structured reasoning annotations across four key components: detailed scene descriptions, identification of crucial objects, prediction of surrounding agents' intentions, and determination of appropriate driving actions. To regulate the reasoning outcomes, our approach incorporates ground-truth driving actions as hints, guiding the model to produce causal explanations that explicitly link driving decisions to scene context. This structured prompting method significantly reduces nonsensical outputs and minimizes the need for manual correction.

Employing this annotation pipeline, we compile a comprehensive reasoning dataset comprising approximately 45.6k CoT reasoning annotations for the nuPlan dataset and 7.2k annotations for the Waymo end-to-end driving dataset. Additionally, we reformat and integrate DriveLM [96], a VQA dataset built on nuScenes and CARLA simulation data, to augment our reasoning data. Additional details and illustrative examples are provided in the Supplementary Material.

3.3 Supervised Fine-tuning

Supervised fine-tuning (SFT) is employed to train the model to generate both reasoning and action sequences. Given multi-frame camera images C , a high-level navigation instruction I , and the ego vehicle state S , the model is trained to produce a sequence of output tokens. The output sequence consists of language tokens l = [ l 1 , . . . , l L ] for reasoning followed by action tokens a = [ a 1 , . . . , a T ] . To enable both fast and slow thinking during SFT, we curate training data with ground-truth assistant responses that either include only the final action tokens or combine CoT reasoning with the corresponding action tokens. In the fast-thinking mode, l is a fixed, short template indicating that reasoning is not needed. Conversely, in the slow-thinking mode, l begins with a template that introduces the need for CoT reasoning, followed by a structured sequence of reasoning.

The first supervision signal is the standard causal language modeling objective, which minimizes the negative log-likelihood of the target token sequence and facilitates the reasoning capability. The other supervision signal focuses on the planning accuracy, and we introduce an auxiliary loss over action tokens a = [ a 1 , . . . , a T ] , which appear at positions x L +1 to x L + T in the output sequence. Given an output sequence x = [ l 1 , . . . , l L , a 1 , . . . , a T ] , the loss functions are defined as:

where N = L + T , and p θ denotes the model's predicted distribution parameterized by θ .

To jointly optimize reasoning and action generation, we combine the language modeling loss and the action loss into a single SFT loss function. To address the imbalance between reasoning data and action-only data, and to encourage the model to learn from examples that include CoT reasoning, we apply a per-sample weighting factor based on the presence of CoT in the ground truth. The overall loss for each training example is computed as follows:

where λ a and λ cot are hyperparameters that control the relative importance.

3.4 Reinforcement Fine-tuning

To further improve the performance of AutoVLA and align it with driving requirements and taskspecific rewards, we introduce a reinforcement learning-based post-training method. This RFT stage enables the model to perform adaptive reasoning and optimize planning performance. We employ the GRPO algorithm [49], which stabilizes training and improves convergence efficiency. Moreover, the inherent multi-modality of planning, characterized by multiple feasible trajectories in the same scenario, naturally aligns with the group-based optimization framework of GRPO [38].

Given a scenario input query q , comprising sensor images, the ego vehicle's state, and driving instruction, we sample a set of G candidate outputs O = { o 1 , o 2 , . . . , o G } from the old policy π θ old . The current policy π θ is then optimized using the normalized group-relative advantage A i , by maximizing the following objective:

where θ and θ old denote the current and old policy parameters, r i is the reward for sample o i , ϵ and β are hyperparameters controlling the clipping range and the weight of the KL divergence regularization term, and π ref is the reference policy from the SFT stage.

The final reward function is defined as r = r Driving -λ r r CoT, where the λ r denotes the balance weight. The term r Driving varies across benchmarks. For the nuPlan dataset, we employ the Predictive Driver Model Score (PDMS) [51] as the driving reward, which captures aspects such as safety, comfort,

travel efficiency, and other driving quality metrics. For the Waymo end-to-end driving dataset, due to the limited availability of Rater Feedback Score (RFS) annotations [52], we use the Average Displacement Error (ADE) as the driving reward. To discourage unnecessary long reasoning chains, we incorporate a CoT length penalty r CoT into the reward function. Additional implementation details are provided in the Supplementary Material.

4 Experiments

4.1 Experimental Setup

Datasets. We train the AutoVLA model using a diverse set of real-world and simulation datasets. The nuPlan (Open-Scene) dataset [50, 97] contains 120 hours of large-scale driving data with eight streams of camera data and object annotations. The Waymo end-to-end driving dataset (Waymo E2E) [52] comprises 4,021 20-second driving segments with eight streams of camera views and ego vehicle trajectories, especially focusing on challenging and long-tail scenarios, such as driving through construction areas or risky situations. The nuScenes dataset [53] provides 1,000 urban driving scenes with six camera views. The CARLA-Garage dataset [55] provides over 500,000 frames of simulation camera data. In addition to the collected reasoning data, we utilize the DriveLM dataset [96] for nuScenes and CARLA datasets, by reformatting the VQA pairs to facilitate CoT reasoning.

Benchmarks. We evaluate AutoVLA on both open-loop and closed-loop benchmarks across realworld and simulated environments. Open-loop performance is assessed on two public benchmarks: the NAVSIM benchmark [51] from the nuPlan dataset and the nuScenes benchmark [65]. The NAVSIM benchmark employs PDMS to assess key aspects of driving behavior, such as collision and ego progress. The nuScenes benchmark uses L2 distance and collision rate as evaluation metrics. Additionally, we report our model's performance on the Waymo end-to-end driving benchmark using the RFS metric, which reflects human-judged planning quality. Closed-loop performance is evaluated on the Bench2Drive benchmark [54] in the CARLA simulator. Bench2Drive contains 44 interactive, closed-loop scenarios under varying locations and weather conditions, using metrics such as success rate, driving score, efficiency, and comfort.

Implementation Details. Each action token corresponds to 0.5 seconds of movement, and the planning horizon is set to 5 seconds. Consequently, the model outputs 10 action tokens, from which a 5-second trajectory can be decoded. For SFT, we use a learning rate of 1 × 10 -5 and the FSDP training strategy. The model is trained for 5 epochs using 8 NVIDIA L40S GPUs. We use a per-GPU batch size of 1 and accumulate gradients over 4 steps, resulting in an effective batch size of 32. The weighting parameters in the SFT loss function are set to λ a = 1 and λ cot = 40 . For RFT, we employ the LoRA adapter [98] for parameter-efficient training. The learning rate for RFT is set to 3 × 10 -5 , and the KL regularization weight β is set to 0 . 04 . We perform a single policy update at each step, allowing the use of a simplified objective without the need for clipping or tracking the old policy. The model is fine-tuned for 6 , 000 steps, and the best-performing checkpoint is selected for evaluation. Additional details are provided in the Supplementary Material.

4.2 Main Results

This section reports the main results of the AutoVLA model for various datasets and benchmarks, with additional results included in the Supplementary Material.

Ground

Truth

Planning

<think> This is a complex scenario requiring additional reasoning

.

Scene Description:

The scene appears near a building entrance. The front view shows a clear path ahead with no immediate

obstacles... The weather seems clear and sunny, with no visible signs of rain or adverse conditions.

Critical Object Description:

1. White Car (Front view):

Just ahead of the ego vehicle, and appears stationary or slowly moving forward.

2. A Row of Cars(Left View):

Several cars are parked or waiting along the left side, with some space

between them. Paying attention to these cars is crucial to avoid any potential collisions while turning left...

Reasoning on Intent:

- The presence of the white car in the front and cars on the left necessitates a cautious approach to avoid

any collisions. The absence of active indicator lights suggests that the vehicles are stationary.

- The ego vehicle is currently moving at a moderate speed (4.979 m/s) and is instructed to turn left ...

- The vehicle has a significant lateral acceleration, suggesting that the vehicle is already turning left.

Best Driving Action:

Turn left with a constant speed.

Given the need to navigate around the cars on the left and …

</think>

<answer>

The final output action is: <action_520><action_920>…

</answer>

<think>

This is a

straightforward

scenario

, and a direct decision

can be made.

</think>

<answer>

The final output action is:

<action_520><action_920>…

</answer>

Fast Thinking After RFT

Slow Thinking Before RFT

Ground

Truth

Planning

Front

Back

Left

Right

(a)

(b)

(c)

/

Before RFT

6

After RFT

95

5

89.11

90

Runtime (s)

3.95s

3

80.54

80

2

1.31s

1

75

0

Runtime

PDMS

70

0.95

0.90

0.75

0.70

GRPO group size 4

GRPO group size 2

GRPO group size 8

0.65

0

1k

2k

3k4k5k

6k

Training Step

Data Scaling Results. We train AutoVLA on a mixture of the nuPlan and nuScenes datasets with varying training set sizes (10k, 50k, 100k, and the full 185k samples), with action-only supervision or with additional CoT reasoning supervision. The models are evaluated on the respective standard test sets, and the results are shown in Fig. 4. We observe that increasing the amount of training data consistently improves planning performance on both datasets. In the nuPlan dataset, when using fewer than 50k training samples, CoT reasoning does not outperform action-only in terms of PDMS and Collision Score. This is likely due to the increased difficulty of learning structured reasoning from limited data. However, as the training set size increases, models trained with CoT reasoning surpass those with action-only supervision, highlighting the scalability advantages of reasoningaugmented learning. A similar trend is observed on the nuScenes dataset: as the training set size increases, models trained with CoT reasoning consistently outperform those trained with action-only data in terms of L2 distance and collision rate.

RFT Performance. The slow-thinking mode incurs a significantly higher runtime due to the generation of CoT reasoning compared to the fast-thinking mode, as shown in Table 2. To mitigate this overhead, we introduce RFT to enhance AutoVLA's adaptive thinking capability and avoid unnecessary reasoning in straightforward scenarios. Specifically, we apply RFT to

10k

50k

100k

185k

40

50

60

70

80

PDM Score

44.06

61.38

73.62

80.54

51.17

65.19

71.79

74.97

nuPlan PDM Score

Action-only Training

Reasoning Training

10k

50k

100k

185k

80

85

90

95

100

No at-fault Col.

84.44

92.33

93.26

95.19

80.30

86.23

94.18

96.89

nuPlan /glyph1197o at-fault Col.

Action-only Training

Reasoning Training

10k

50k

100k

185k

0.7

0.8

0.9

1.0

1.1

1.2

L2

0.98

0.86

0.84

0.71

1.03

0.99

0.80

0.70

nuScenes L2 (m)

Action-only Training

Reasoning Training

10k

50k

100k

185k

0.3

0.4

0.5

0.6

0.7

Col. Rate

0.50

0.43

0.38

0.35

0.57

0.54

0.32

0.31

nuScenes Col. Rate (%)

Action-only Training

Reasoning Training

o

the full-data CoT reasoning model trained via SFT to enhance its planning performance. As shown in Fig. 5(a), RFT yields a 10.6% improvement in PDMS (on the NAVSIM testing set) and a 66.8% reduction in runtime (average over 500 testing scenarios). The reward curve in Fig. 5(b) illustrates the progressive improvement of the model's policy during RFT. Experiments with different GRPO group

(a)

Front

(b)

Ground Truth

Planning

<think> This is a complex scenario requiring additional reasoning

.

d

Scene Description:

The scene is a daytime urban environment with clear weather. The road is marked with double yellow lines, indicating a no-passing zone. There are

construction barriers and signs indicating road work ahead. Several vehicles are visible in the distance, traveling in both directions. The front left camera

shows a building and some greenery along the road. The front right camera captures a construction site with barriers and a few vehicles.

d

Critical Object Description:

1. Construction Barriers:

Located in the center of the road ahead, these barriers indicate ongoing road work. They are important as they may require the

vehicle to slow down and navigate around them.

2. Vehicles Ahead:

Positioned further down the road, these vehicles are traveling in the same direction as the ego vehicle. They are important as they may

slow down or stop due to the road work or other vehicles ahead.

Dd

d

Reasoning on Intent:

Firstly, notice the construction barriers. These barriers indicate that the road ahead may have reduced speed limits or require careful navigation. The ego

vehicle should approach these barriers with caution and potentially reduce its speed ……

- Given the current driving command instruction to "go straight," the ego vehicle should continue moving forward while being mindful of the road

conditions ahead. The presence of construction barriers and other vehicles suggests that the vehicle should slow down to navigate safely through the area.

d

Best Driving Action:

Move forward with a deceleration.

This  to safely approach the construction barriers, maintain a safe distance from other vehicles, and comply with any

road conditions indicated by the signs. ……

</think>

<answer>

The final output action is: <action_1457><action_1457>…

</answer>

sample sizes indicate that larger groups lead to better performance by promoting broader exploration of training samples. As illustrated in Fig. 5(c), RFT also reduces unnecessary and slow reasoning in simple scenarios, driven by the CoT length penalty that encourages fast thinking for straightforward driving cases. A qualitative comparison shows that the SFT model produces suboptimal plans due to error accumulation in generation, whereas the RFT model (optimized via PDMS-based reward) generates better trajectories.

nuPlan Benchmark Results. We evaluate AutoVLA against state-of-the-art end-to-end driving models on the NAVSIM benchmark [51] and present results in Table 1. In best-of-N planning, we use an oracle scorer to select the optimal trajectory from six generated candidates. After RFT, AutoVLA demonstrates significantly improved performance, aligning more closely with the NAVSIM reward signal. The best-of-N strategy further enhances performance, achieving the highest PDMS. Overall, AutoVLA achieves competitive results while demonstrating scalability across diverse datasets.

Waymo E2E Performance. We evaluate AutoVLA on the Waymo end-to-end driving dataset [52], which features long-tail and complex driving scenarios. The model's performance under various training settings on the test set is presented in Fig. 6. The results reveal that pretraining on a combination of nuPlan and nuScenes datasets significantly enhances performance, suggesting enhanced scene understanding through exposure to more diverse training data. Incorporating CoT reasoning in training further improves planning performance compared to action-only training. Post-training with RFT, using ADE as the reward function, achieves the best overall RFS metric. A qualitative example in a construction zone demonstrates the model's ability to reason about occlusions and generate effective detour plans.

CARLA Closed-loop Performance. We evaluate the closed-loop driving performance of our AutoVLA model on the Bench2Drive benchmark [54] in the CARLA simulator. The model is trained using SFT with both trajectory-only and CoT data. During testing, the planning frequency is set to 2 Hz. The results, shown in Table 3, demonstrate that AutoVLA outperforms existing end-to-end driving models in terms of overall driving score and success rate in the closed-loop test.

4.3 Ablation Studies

Text Waypoint Output. We use the same mixed training set from the nuPlan and nuScenes datasets to train a model that predicts waypoints in a text format, which are then converted into a trajectory of waypoints. We evaluate its performance in an open-loop planning setting using the standard test sets. The results, shown in Table 5, indicate that our action tokenization and generation method significantly outperforms the text-based waypoint prediction approach. Additionally, due to the need to decode

numerical values, the text-based method incurs a substantially higher computational cost in generating the final trajectory. This shows the limitation of language models in handling precise numerical reasoning.

Action Tokenization Methods. We evaluate reconstruction accuracy across different codebook sizes for several tokenization methods, including RT-1 [102], FAST [91], and our proposed K-disk tokenization, as shown in Table 4. For RT-1, acceleration and steering rate are discretized using uniform action bins, and a kinematic model is employed to reconstruct the trajectory. However, because only trajectory-level data is available and control actions must be indirectly inferred, this binning approach yields the highest reconstruction error. The FAST tokenization method is more sensitive to codebook size than our K-

disk approach, achieving comparable reconstruction accuracy only at K = 4096 . In contrast, our proposed method consistently attains the highest reconstruction accuracy across all codebook sizes.

When the codebook size is small ( K = 256 ), reconstruction errors are significantly higher due to limited movement coverage, as many movements cannot be adequately represented by the small set of codebook tokens. As the codebook size increases beyond K = 2048 , the improvements in tokenization accuracy and movement coverage become marginal, while codebook usage decreases as many tokens remain redundant and unused. Considering this trade-off, we select K = 2048 to balance reconstruction accuracy, movement coverage, and efficient codebook usage.

Furthermore, Table 6 reports planning performance on NAVSIM (nuPlan) for different tokenization methods. Our proposed method outperforms FAST. In FAST, the discrete cosine transform converts fixed-horizon planning trajectories into variable-length token sequences, complicating the selection of appropriate token lengths and making fixed-horizon planning difficult.

5 Conclusions

We propose AutoVLA, an end-to-end autonomous driving framework that unifies scene reasoning and action generation within a single autoregressive model. We adopt SFT to enable the model to operate in both fast thinking (direct trajectory generation) and slow thinking (enhanced with long CoT reasoning) modes. In addition, we introduce RFT to enable adaptive reasoning by penalizing unnecessary reasoning and aligning action generation with reward functions, improving both performance and efficiency. Experimental results demonstrate that AutoVLA achieves competitive performance on both open-loop and closed-loop planning benchmarks and exhibits strong reasoning capabilities.

Limitation and Future Work . Although our model with dual-process adaptation achieves nearreal-time inference (1 Hz), it remains highly GPU-dependent, requiring significant memory and computing. Future work will focus on real-time applications, optimizing runtime efficiency, and reducing computation overhead (e.g., through model quantization) to enable real-time deployment.

References

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird's-eye-view representation from lidar-camera via spatiotemporal transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning , pages 180-191. PMLR, 2022.

Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems , 35:10421-10434, 2022.

Zikang Zhou, Zihao Wen, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Qcnext: A next-generation framework for joint multi-agent trajectory prediction. arXiv preprint arXiv:2306.10508 , 2023.

Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. IEEE Transactions on Pattern Analysis and Machine Intelligence , 46(5):3955-3971, 2024.

Zhiyu Huang, Haochen Liu, and Chen Lv. Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3903-3913, 2023.

Zhiyu Huang, Xinshuo Weng, Maximilian Igl, Yuxiao Chen, Yulong Cao, Boris Ivanovic, Marco Pavone, and Chen Lv. Gen-drive: Enhancing diffusion generative driving policies with reward modeling and reinforcement learning fine-tuning. arXiv preprint arXiv:2410.05582 , 2024.

Zhiyu Huang, Haochen Liu, Jingda Wu, and Chen Lv. Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving. IEEE transactions on neural networks and learning systems , 2023.

Haochen Liu, Zhiyu Huang, Wenhui Huang, Haohan Yang, Xiaoyu Mo, and Chen Lv. Hybridprediction integrated planning for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2025.

Hao Shao, Letian Wang, Ruobing Chen, Steven L Waslander, Hongsheng Li, and Yu Liu. Reasonnet: End-to-end driving with temporal and global reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13723-13733, 2023.

Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21983-21994, 2023.

Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14760-14769, 2024.

Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14749-14759, June 2024.

Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. Stp3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision , pages 533-549. Springer, 2022.

Shaoheng Fang, Zi Wang, Yiqi Zhong, Junhao Ge, and Siheng Chen. Tbp-former: Learning temporal bird's-eye-view pyramid for joint perception and prediction in vision-centric autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1368-1378, 2023.

Xingcheng Zhou, Mingyu Liu, Ekim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C Knoll. Vision language models in autonomous driving: A survey and outlook. IEEE Transactions on Intelligent Vehicles , 2024.

Tianhui Cai, Yifan Liu, Zewei Zhou, Haoxuan Ma, Seth Z Zhao, Zhiwen Wu, and Jiaqi Ma. Driving with regulation: Interpretable decision-making for autonomous vehicles with retrieval-augmented reasoning via llm. arXiv preprint arXiv:2410.04759 , 2024.

Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human: Rethinking autonomous driving with large language models. In 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) , pages 910-919. IEEE, 2024.

Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025.

Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, et al. Automated evaluation of large vision-language models on self-driving corner cases. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 7817-7826. IEEE, 2025.

Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, and Marco Pavone. Driving everywhere with large language model policy adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14948-14957, 2024.

Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan-Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters , 2024.

Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533 , 2024.

Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In European Conference on Computer Vision , pages 403-420. Springer, 2024.

Katharina Winter, Mark Azer, and Fabian B Flohr. Bevdriver: Leveraging bev maps in llms for robust closed-loop driving. arXiv preprint arXiv:2503.03074 , 2025.

Kexin Tian, Jingrui Mao, Yunlong Zhang, Jiwan Jiang, Yang Zhou, and Zhengzhong Tu. Nuscenes-spatialqa: A spatial understanding and reasoning benchmark for vision-language models in autonomous driving. arXiv preprint arXiv:2504.03164 , 2025.

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 , 2024.

Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual chain-of-thought reasoning for vision-language-action models. arXiv preprint arXiv:2503.22020 , 2025.

Chia-Yu Hung, Qi Sun, Pengfei Hong, Amir Zadeh, Chuan Li, U-Xuan Tan, Navonil Majumder, and Soujanya Poria. Nora: A small open-sourced generalist vision language action model for embodied tasks, 2025. URL https://arxiv.org/abs/2504.19854 .

Hidehisa Arai, Keita Miwa, Kento Sasaki, Kohei Watanabe, Yu Yamaguchi, Shunsuke Aoki, and Issei Yamamoto. Covla: Comprehensive vision-language-action dataset for autonomous driving. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 1933-1943. IEEE, 2025.

Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, and Alois C Knoll. Opendrivevla: Towards end-to-end autonomous driving with large vision language action model. arXiv preprint arXiv:2503.23463 , 2025.

Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15120-15130, 2024.

Katrin Renz, Long Chen, Elahe Arani, and Oleg Sinavski. Simlingo: Vision-only closed-loop autonomous driving with language-action alignment. arXiv preprint arXiv:2503.09594 , 2025.

Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, et al. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262 , 2024.

Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning. arXiv preprint arXiv:2503.07608 , 2025.

Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large vision-language models and end-toend autonomous driving. arXiv preprint arXiv:2410.22313 , 2024.

Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245 , 2023.

Katrin Renz, Long Chen, Ana-Maria Marcu, Jan Hünermann, Benoit Hanotte, Alice Karnsund, Jamie Shotton, Elahe Arani, and Oleg Sinavski. Carllava: Vision language models for cameraonly closed-loop driving. arXiv preprint arXiv:2406.10165 , 2024.

Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. Orion: A holistic end-to-end autonomous driving framework by vision-language instructed action generation. arXiv preprint arXiv:2503.19755 , 2025.

Waywe Research Team et al. Lingo-2: Driving with natural language, 2024.

Shuo Xing, Chengyuan Qian, Yuping Wang, Hongyuan Hua, Kexin Tian, Yang Zhou, and Zhengzhong Tu. Openemma: Open-source multimodal model for end-to-end autonomous driving. In Proceedings of the Winter Conference on Applications of Computer Vision , pages 1001-1009, 2025.

Tianqi Wang, Enze Xie, Ruihang Chu, Zhenguo Li, and Ping Luo. Drivecot: Integrating chain-of-thought reasoning with end-to-end driving. arXiv preprint arXiv:2403.16996 , 2024.

Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289 , 2024.

Wenjie Ma, Jingxuan He, Charlie Snell, Tyler Griggs, Sewon Min, and Matei Zaharia. Reasoning models can be effective without thinking. arXiv preprint arXiv:2504.09858 , 2025.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024.

Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, et al. Towards learning-based planning: The nuplan benchmark for real-world autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA) , pages 629-636. IEEE, 2024.

Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems , 37:28706-28719, 2024.

Waymo Research. 2025 waymo open dataset challenge: Vision-based end-to-end driving. https://waymo.com/open/challenges/2025/e2e-driving/ , 2025. Accessed: 202504-25.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621-11631, 2020.

Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. arXiv preprint arXiv:2406.03877 , 2024.

Bernhard Jaeger, Kashyap Chitta, and Andreas Geiger. Hidden biases of end-to-end driving models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8240-8249, October 2023.

Chengran Yuan, Zhanqi Zhang, Jiawei Sun, Shuo Sun, Zefan Huang, Christina Dao Wen Lee, Dongen Li, Yuhang Han, Anthony Wong, Keng Peng Tee, et al. Drama: An efficient end-to-end motion planner for autonomous driving with mamba. arXiv preprint arXiv:2408.03601 , 2024.

Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation. arXiv preprint arXiv:2406.06978 , 2024.

Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE transactions on pattern analysis and machine intelligence , 45(11):12878-12895, 2022.

Chonghao Sima, Kashyap Chitta, Zhiding Yu, Shiyi Lan, Ping Luo, Andreas Geiger, Hongyang Li, and Jose M Alvarez. Centaur: Robust end-to-end autonomous driving with test-time training. arXiv preprint arXiv:2503.11650 , 2025.

Zewei Zhou, Hao Xiang, Zhaoliang Zheng, Seth Z Zhao, Mingyue Lei, Yun Zhang, Tianhui Cai, Xinyi Liu, Johnson Liu, Maheswari Bajji, et al. V2XPnP: Vehicle-to-everything spatiotemporal fusion for multi-agent perception and prediction. arXiv preprint arXiv:2412.01812 , 2024.

Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. arXiv preprint arXiv:2405.19620 , 2024.

Yiheng Li, Seth Z. Zhao, Chenfeng Xu, Chen Tang, Chenran Li, Mingyu Ding, Masayoshi Tomizuka, and Wei Zhan. Pre-training on synthetic driving data for trajectory prediction. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5910-5917, 2024. doi: 10.1109/IROS58592.2024.10802492.

Wenzhao Zheng, Junjie Wu, Yao Zheng, Sicheng Zuo, Zixun Xie, Longchao Yang, Yong Pan, Zhihui Hao, Peng Jia, Xianpeng Lang, et al. Gaussianad: Gaussian-centric end-to-end autonomous driving. arXiv preprint arXiv:2412.10371 , 2024.

Yuping Wang, Shuo Xing, Cui Can, Renjie Li, Hongyuan Hua, Kexin Tian, Zhaobin Mo, Xiangbo Gao, Keshu Wu, Sulong Zhou, Hengxu You, Juntong Peng, Junge Zhang, Zehao Wang, Rui Song, Mingxuan Yan, Walter Zimmer, Xingcheng Zhou, Peiran Li, Zhaohan Lu, Chia-Ju Chen, Yue Huang, Ryan A. Rossi, Lichao Sun, Hongkai Yu, Zhiwen Fan, Frank Hao Yang, Yuhao Kang, Ross Greer, Chenxi Liu, Eun Hak Lee, Xuan Di, Xinyue Ye, Liu Ren, Alois Knoll, Xiaopeng Li, Shuiwang Ji, Masayoshi Tomizuka, Marco Pavone, Tianbao Yang, Jing Du, Ming-Hsuan Yang, Hua Wei, Ziran Wang, Yang Zhou, Jiachen Li, and Zhengzhong Tu. Generative ai for autonomous driving: Frontiers and opportunities, 2025. URL https: //arxiv.org/abs/2505.08854 .

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 17853-17862, 2023.

Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8340-8350, 2023.

Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15449-15458, 2024.

Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. In European Conference on Computer Vision , pages 87-104. Springer, 2024.

Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. arXiv preprint arXiv:2411.15139 , 2024.

Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14864-14873, 2024.

Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, and Jingjing Liu. Adapt: Action-aware driving caption transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 7554-7561. IEEE, 2023.

SungYeon Park, MinJae Lee, JiHyuk Kang, Hahyeon Choi, Yoonah Park, Juhwan Cho, Adam Lee, and DongKyu Kim. Vlaad: Vision and language assistant for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 980-987, 2024.

Kairui Ding, Boyuan Chen, Yuchen Su, Huan-ang Gao, Bu Jin, Chonghao Sima, Wuqiang Zhang, Xiaohui Li, Paul Barsch, Hongyang Li, et al. Hint-ad: Holistically aligned interpretability in end-to-end autonomous driving. arXiv preprint arXiv:2409.06702 , 2024.

Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driving. In European Conference on Computer Vision , pages 252-269. Springer, 2024.

Yiheng Li, Cunxin Fan, Chongjian Ge, Zhihao Zhao, Chenran Li, Chenfeng Xu, Huaxiu Yao, Masayoshi Tomizuka, Bolei Zhou, Chen Tang, Mingyu Ding, and Wei Zhan. Womdreasoning: A large-scale dataset for interaction reasoning in driving, 2025. URL https: //arxiv.org/abs/2407.04281 .

Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint arXiv:2412.14446 , 2024.

Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Liu, Risheek Garrepalli, Vishal M Patel, and Fatih Porikli. Distilling multimodal large language models for autonomous driving. arXiv preprint arXiv:2501.09757 , 2025.

Pei Liu, Haipeng Liu, Haichao Liu, Xin Liu, Jinxin Ni, and Jun Ma. Vlm-e2e: Enhancing end-to-end autonomous driving with multimodal driver attention fusion. arXiv preprint arXiv:2502.18042 , 2025.

Haicheng Liao, Hanlin Kong, Bonan Wang, Chengyue Wang, Wang Ye, Zhengbing He, Chengzhong Xu, and Zhenning Li. Cot-drive: Efficient motion forecasting for autonomous driving with llms and chain-of-thought prompting. arXiv preprint arXiv:2503.07234 , 2025.

Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415 , 2023.

Songyan Zhang, Wenhui Huang, Zihui Gao, Hao Chen, and Chen Lv. Wisead: Knowledge augmented end-to-end autonomous driving with vision-language model. arXiv preprint arXiv:2412.09951 , 2024.

Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. arXiv preprint arXiv:2407.00959 , 2024.

Zhijie Qiao, Haowei Li, Zhong Cao, and Henry X. Liu. Lightemma: Lightweight end-to-end multimodal model for autonomous driving, 2025. URL https://arxiv.org/abs/2505. 00284 .

Wenru Liu, Pei Liu, and Jun Ma. Dsdrive: Distilling large language model for lightweight end-to-end autonomous driving with unified reasoning and planning, 2025. URL https: //arxiv.org/abs/2505.05360 .

Derun Li, Jianwei Ren, Yue Wang, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Zhongpu Xia, Peng Jia, Xianpeng Lang, et al. Finetuning generative trajectory model with reinforcement learning from human feedback. arXiv preprint arXiv:2503.10434 , 2025.

Hao Gao, Shaoyu Chen, Bo Jiang, Bencheng Liao, Yiang Shi, Xiaoyang Guo, Yuechuan Pu, Haoran Yin, Xiangyu Li, Xinbang Zhang, et al. Rad: Training an end-to-end driving policy via large-scale 3dgs-based reinforcement learning. arXiv preprint arXiv:2502.13144 , 2025.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728-53741, 2023.

Zhejun Zhang, Peter Karkus, Maximilian Igl, Wenhao Ding, Yuxiao Chen, Boris Ivanovic, and Marco Pavone. Closed-loop supervised fine-tuning of tokenized traffic models. arXiv preprint arXiv:2412.05334 , 2024.

Jonah Philion, Xue Bin Peng, and Sanja Fidler. Trajeglish: Traffic modeling as next-token prediction. arXiv preprint arXiv:2312.04535 , 2023.

Wei Wu, Xiaoxin Feng, Ziyan Gao, and Yuheng Kan. Smart: scalable multi-agent real-time motion generation via next-token prediction. Advances in Neural Information Processing Systems , 37:114048-114071, 2024.

Karl Pertsch, Kyle Stachowicz, Brian Ichter, Danny Driess, Suraj Nair, Quan Vuong, Oier Mees, Chelsea Finn, and Sergey Levine. Fast: Efficient action tokenization for vision-language-action models. arXiv preprint arXiv:2501.09747 , 2025.

Sung-Yeon Park, Can Cui, Yunsheng Ma, Ahmadreza Moradipari, Rohit Gupta, Kyungtae Han, and Ziran Wang. Nuplanqa: A large-scale dataset and benchmark for multi-view driving scene understanding in multi-modal large language models. arXiv preprint arXiv:2503.12772 , 2025.

Dongming Wu, Wencheng Han, Yingfei Liu, Tiancai Wang, Cheng-zhong Xu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 8359-8367, 2025.

Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4542-4550, 2024.

Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. In European Conference on Computer Vision , pages 292-308. Springer, 2024.

Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision , pages 256-274. Springer, 2024.

OpenScene Contributors. Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving. https://github.com/OpenDriveLab/OpenScene , 2023.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR , 1 (2):3, 2022.

Jiang-Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arXiv preprint arXiv:2305.10430 , 2023.

Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectoryguided control prediction for end-to-end autonomous driving: A simple yet strong baseline. Advances in Neural Information Processing Systems , 35:6119-6132, 2022.

Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7953-7963, 2023.

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 , 2022.

Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles R Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9710-9719, 2021.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We do describe our contributions in the abstract and introduction of the paper: 1) a novel end-to-end autonomous driving framework leveraging a trained VLM backbone augmented with integrated physical action tokens, enabling direct policy learning and semantic reasoning from raw visual inputs. 2) an RL-based fine-tuning method using GRPO to enable adaptive decision-making and further enhance AutoVLA's performance on end-to-end driving tasks.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the real-world implementation limitation of AutoVLA and introduce our future work on model quantization, as Section 5 shown.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper focuses on the experimental exploration of model architectures, and have no theoretical results in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All the necessary information for reproducibility is provided in Section 4 and the Supplementary Material. The model details also are discussed in the methodology of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release our code and dataset to facilitate community research.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all the implementation details and dataset settings of AutoVLA in both the SFT and RFT stages in the paper Section 4 and supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper reports results from a single run per experiment, consistent with prior work on this task.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides all necessary details on computational resources, including GPU type (NVIDIA L40S), memory usage, and number of epochs, in Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research presented in this paper adheres to the NeurIPS Code of Ethics, ensuring ethical standards in data collection and analysis.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper includes a discussion of potential societal impacts in the supplementary material.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no foreseeable risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The original creators and asset owners are appropriately credited, and licenses and terms of use are explicitly stated and duly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will release all the code, model weights, and datasets to facilitate research in the domain.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or human-subject research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or human-subject research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We declare the usage of LLM/VLM in the paper, and we integrate the physical action tokens into the pretrained VLM backbone to develop a novel end-to-end autonomous driving framework.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

AutoVLA Supplementary Material

A Action Tokenization

A.1 Action Codebook

To enable trajectory-level planning within a language model, we introduce physical action tokens , each representing a short-term feasible vehicle maneuver. These tokens are derived from clustering vehicle motion patterns in the Waymo Open Motion Dataset (WOMD) [103], which contains extensive real-world vehicle trajectories. The resulting codebook, visualized in Fig. S1(a), is used across experiments involving multiple real-world datasets, including nuPlan, nuScenes, and Waymo. Due to differences in vehicle dynamics compared to real-world data for simulation testing, we construct a separate action codebook using the same clustering procedure on the CARLA-Garage dataset [55]. This produces another set of 2048 action tokens for simulation testing, as shown in Fig. S1(b).

Webegin by sampling short motion segments from the dataset to construct a discrete action codebook . Each segment represents 0.5 seconds of vehicle motion, characterized by its final-frame bounding-box contour, computed based on the vehicle's position, dimensions, and heading with respect to the currentframe coordinate. Then, we apply K-Disk clustering on these sampled segments, which iteratively selects a diverse set of representative segments { m 1 , . . . , m K } , such that no two segments are within a distance threshold δ = 0 . 05 m, measured using average contour distance. For each selected segment m k , we extract its spatial displacement and heading change, denoted as (∆ x, ∆ y, ∆ θ ) , and define it as the action token a k . The resulting action codebook is A = { a 1 , . . . , a K } , with K = 2048 , where each token encodes a distinct and physically feasible short-term vehicle behavior.

(a)

(b)

Real-world (WOMD)

A.2 Action Tokenizer

We implement an action tokenizer based on the constructed action codebook. During training, continuous trajectories are discretized by mapping each 0.5-second segment to its nearest action token in the codebook A , resulting in a sequence of discrete tokens [ a 1 , a 2 , . . . , a T ] . During inference, the language model autoregressively generates a sequence of action tokens, each representing a 0.5-second motion segment. These tokens are then converted back to their corresponding motion segments in the codebook and applied sequentially from the initial ego pose. Therefore, the model reconstructs a continuous trajectory in the ego-centric coordinate frame by composing these local displacements and rotations.

B Reasoning Data Collection

A large-scale, high-quality reasoning dataset with chain-of-thought (CoT) annotations is essential for enabling robust reasoning capabilities in vision-language-action (VLA) models. In this paper, we introduce an automated reasoning annotation pipeline using the state-of-the-art Qwen2.5-VL-72B vision-language model [21], as illustrated in Fig. S2. The pipeline significantly reduces reliance on

…

Ego states

Instruction

Turn Left | Go Straight …

Velocity | Acceleration | History Action

…

…

Text Tokenizer

System Prompt

- Role:

You are an advanced full self-driving system.

- Task:

You will be provided with video observations from

the ego vehicle's surrounding cameras, along with the

vehicle's current dynamic states. Your task is to predict

the optimal driving action for the next five seconds and

use step-by-step reasoning (Chain-of-Thought) to arrive at

the best driving action.

…

Vision Encoder

Qwen2.5-VL-72B

- Scene Analysis

- Critical Object Identification

- Intention Reasoning

- Final Action Decision

…

Multi-view

Camera

Streams

Ground Truth Action

Lateral + Longitudinal Actions

- Scene Description:

The scene is at rainy night, and the vehicle need to identify the vehicle position light and traffic light carefully. There is a construction vehicle and a white truck in front of …

- Critical Object Description:

The White truck is turning right, with position light is on (general brightness, not brake light). The Construction vehicles are stationary (brake lights on).

- Reasoning on Intent:

Firstly, notice White truck, which will turn right on the right-turn lane. Secondly, notice Unclear Light in front. There are some lights in the night scene, and the ego vehicle does

not need to follow the unclear light as a traffic light. The ego vehicle is moving on the go straight lane in the past 4s, but the driving command is turn right now. Meanwhile, no vehicle is in the right

behind. Thus, the ego vehicle should change the lane to the right and follow the White truck.

- Best Driving Action:

change lane to right with an acceleration

Example

Scene Description:

The scene is at night, and the ego vehicle is stopped at an intersection. The front camera

shows multiple red traffic lights directly ahead, indicating that the vehicle must stop. ...

Critical Object Description:

1. Red Traffic Lights (Front Camera)

- Location: Directly ahead of the ego vehicle.

- Description: The traffic lights are clearly red, signaling all vehicles to stop.

- Importance: The red lights are crucial as they dictate the vehicle's immediate action-

stopping and waiting for the light to change.

2. Pedestrian Crossings (Front Left and Front Right Cameras):

- Location: On either side of the intersection.

- Description: No pedestrians are visible near the crossings.

- Importance: Although no pedestrians are present, the crossings indicate potential areas

where pedestrians may appear, requiring vigilance...

Reasoning on Intent:

1. Red Traffic Lights:

The primary focus is the red traffic lights, which mandate that the

vehicle must stop and wait until the light turns green…

2. Driving Command Instruction:

The driving command suggests 'go left,' but given the

red light, the vehicle cannot proceed in any direction until the light changes…

The vehicle has already been stopped for the past four seconds, and

its current velocity and acceleration are zero…

Best Driving Action: Stop.

Given the red traffic lights and the vehicle's current state of being stopped, the best action is

to remain stopped…

Reasoning Data

human annotations and facilitates effective knowledge distillation from a more powerful, large-scale model to a more efficient, compact model.

B.1 Reasoning Annotation Pipeline

System Prompt. The system prompt specifies the model's role, task, expected CoT reasoning format, and examples of CoT reasoning. The definition of role and CoT reasoning format aligns with the required outputs of the AutoVLA model, which focuses on structured CoT reasoning. We carefully design several representative reasoning examples to guide the model. Moreover, the reasoning process includes four main steps: 1) scene description and analysis, 2) critical object identification and description, 3) intention reasoning of the surrounding objects, and 4) decision-making and meta-action.

User Message. The user message includes the driving instructions, ego vehicle states, and multi-view camera streams. Notably, we introduce the ground-truth driving meta-action derived from the data as explicit hints in reasoning, guiding the model to produce causal explanations that directly associate decisions with the driving context. This structured prompting significantly reduces nonsensical outputs and minimizes manual revisions.

Reasoning Data Generation. We employ the most capable model in the Qwen-VL series as the reasoning data annotation engine, leveraging its strong reasoning capacity and extensive world knowledge. The maximum length for newly generated reasoning textual tokens is set to 700. Additionally, we extract answers to relevant questions from the DriveLM VQA dataset [96], which focuses on perception, prediction, and planning in the nuScenes and CARLA datasets, and then we reformat them into our standardized reasoning format. These reformatted samples are then combined with the generated data to construct the final reasoning dataset.

Human Quality Check. We evaluate the quality of generated reasoning based on the accuracy and completeness of critical object identification, causal reasoning, and action decisions. The evaluation uses a binary scoring scheme: any error in these aspects results in a score of 0; otherwise, the sample receives a score of 1. Human annotators assessed 3,000 randomly selected samples, yielding an overall accuracy of 88.8%, which demonstrates the high reliability of our proposed annotation pipeline. Erroneous samples were either corrected or discarded.

Scene Description:

The scene is at night, and the road appears to be undergoing construction or maintenance work, as indicated by the "KEEP LEFT" sign and orange-and-white striped poles. There are several vehicles ahead, including a large truck directly in front of the ego vehicle. The road is well-lit by streetlights, and the weather seems clear. No traffic lights are visible in the scene.

Critical Object Description:

1. Large Truck:

Located directly in front of the ego vehicle, the truck is moving slowly or is stationary. Its brake lights are on, indicating it may be slowing down or stopped. This is important because the ego vehicle must maintain a safe distance.

2. KEEP LEFT Sign:

Positioned on the right side of the road, this sign indicates that vehicles should stay in the left lane due to construction or roadwork. This is crucial for navigating the area safely and legally.

Orange-and-White Striped Poles:

These poles are placed along the right side of the road, marking the boundary of the construction zone. They help guide drivers to stay in the correct lane and avoid hazards.

Reasoning on Intent:

The Large Truck ahead is either slowing down or stopped, as indicated by its brake lights. The ego vehicle must decelerate to maintain a safe following distance and avoid a collision.

The KEEP LEFT Sign and Orange-and-White Striped Poles indicate that the ego vehicle should stay in the left lane. Since the driving command is to go straight, the ego vehicle should continue moving forward while adhering to these instructions.

The ego vehicle has been moving forward with a deceleration in the past 4 seconds, and its current velocity is relatively low. Given the presence of the truck ahead and the construction zone, it is reasonable to continue decelerating to ensure safety.

Best Driving Action: Move forward with a deceleration

This action aligns with the need to maintain a safe distance from the truck ahead while adhering to the road signs and navigating through the construction zone. Decelerating ensures the ego vehicle can respond appropriately if the truck stops completely or if any unexpected obstacles appear.

Reasoning Data

Scene Description:

The scene is a residential area during the daytime with clear weather. The ego vehicle is at an intersection with a "STOP" sign visible on the right side. There are parked cars along both sides of the street, and a few houses can be seen. The front left camera shows a clear path to the left, while the front right camera shows the "STOP" sign and a parked car nearby. The back camera shows a straight road leading up to the intersection. No pedestrians or moving vehicles are observed in the immediate vicinity.

Critical Object Description:

1. STOP Sign

:

Located on the right side of the intersection, indicating that the ego vehicle must come to a complete stop before proceeding.

What it is doing: The sign is static and serves as a regulatory control.

Why it is important: The ego vehicle must adhere to the stop sign to ensure safety and compliance with traffic rules.

2. Parked Cars :

Visible on both sides of the street, particularly near the intersection.

What they are doing: They are stationary and occupy parking spaces.

Why they are important: The ego vehicle must navigate around these parked cars,

especially when changing lanes or turning.

3. Clear Path to the Left :

Observed through the front left camera.

What it indicates: A potential route for the ego vehicle to take.

Why it is important: This path appears unobstructed, making it a viable option for the ego vehicle to proceed.

Reasoning on Intent:

The ego vehicle has already come to a stop as indicated by its current velocity being close to zero and the past behavior of decelerating to zero.

The driving command instructs the ego vehicle to go left, suggesting a left turn or lane change is necessary.

The front left camera shows a clear path to the left, indicating that it is safe to proceed in that direction.

Given the absence of any moving vehicles or pedestrians, the ego vehicle can safely execute a quick acceleration after changing lanes to the left.

Best Driving Action: Change lane to left with a quick acceleration.

This action aligns with the driving command to go left, the clear path observed in the front left camera, and the need to quickly proceed after stopping at the intersection. The quick acceleration ensures efficient navigation through the intersection while maintaining safety.

Reasoning Data

History and GT Future Trajectory

History Trajectory

History and GT Future Trajectory

History Trajectory

front_left

front

front_right

front_left

front

front_right

left

right

left

right

Future Trajectory

Future Trajectory

10

20

20

10

20

back_left

oack

back_right

back_left

back

back_right

B.2 Reasoning Annotation Visualization

Some CoT reasoning annotation examples from the Waymo end-to-end driving dataset are shown in Fig. S3, showcasing that our annotation pipeline can deal with situations where a vehicle already stopped at a stop sign can proceed, rather than remaining stopped indefinitely. The pipeline also accurately interprets construction-related road control scenarios, enabling high-quality reasoning annotations. Examples on the nuPlan dataset are provided in Fig. S4, demonstrating the pipeline's capability to distinguish the functional relevance of stop signs and traffic lights across different lanes, rather than stopping at every detected stop sign.

C Details of Supervised Fine-tuning

We perform supervised fine-tuning (SFT) with the Fully Sharded Data Parallel (FSDP) strategy for efficient multi-GPU training. We enable mixed-precision training using BFloat16 for parameters,

Scene Description:

The scene appears to be an urban area with multiple lanes and various road features. The

front view shows a clear path ahead with no immediate obstacles. The right view indicates

a curved road leading towards a covered area with some parked vehicles and pedestrians.

The left view shows a sidewalk with some greenery and a stop sign further down the road.

The rear view shows a relatively empty street behind the vehicle. The weather seems

clear, and there are no visible traffic lights affecting the current path.

Critical Object Description:

1. Pedestrians near the covered area (Right View):

- Location: On the right side of the road, near the covered area.

- Description: A few pedestrians are walking along the sidewalk near the covered area.

They appear to be moving at a normal pace and are not showing any immediate intention

to cross the road.

- Importance: Pay attention to their movements as they could potentially step onto the road

unexpectedly, especially if the vehicle is turning right.

2. Stop Sign (Left View):

- Location: Further down the left side of the road.

- Description: A stop sign is visible, indicating that the vehicle must come to a complete

stop before proceeding.

- Importance: This is crucial as it dictates the vehicle's next action. Ignoring the stop sign

could lead to unsafe driving conditions and potential accidents.

Reasoning on Intent:

- The ego vehicle is currently instructed to turn right. The front and right views show a clear

path for turning, but the presence of pedestrians near the covered area requires caution.

- The left view shows a stop sign, which suggests that the vehicle may need to stop before

completing the turn. However, since the stop sign is further down the road and the vehicle

is already turning right, it is likely not immediately relevant.

- The vehicle is currently moving at a moderate speed with a slight acceleration, indicating

that it is preparing to execute the turn.

Best Driving Action: Turn right with a constant speed

.

Given the clear path ahead and the absence of immediate obstacles, the ego vehicle

should proceed with turning right while maintaining a constant speed. This ensures a

smooth and controlled turn while staying aware of the pedestrians on the right side. The

stop sign on the left is not directly relevant to the current maneuver, so the focus remains

on executing the turn safely

Reasoning Data

Scene Description:

The scene depicts a busy urban intersection with multiple lanes of traffic. The traffic lights

ahead are red, indicating that vehicles must stop. There are several vehicles visible in the

front camera view, including a white van directly in front of the ego vehicle and other cars

further ahead. The right camera view shows a truck and some cars moving along the

adjacent lane. The left camera view captures a few cars and a pedestrian area under a

building. The rear camera view shows a white truck passing by.

Critical Object Description:

1. White Van (Front Camera View):

- Location: Directly in front of the ego vehicle.

- Status: Stationary, as indicated by the red traffic light. - Indicator Lights: No visible lights.

- Importance: This vehicle is directly in the path of the ego vehicle, and it is crucial to

maintain a safe distance while waiting for the traffic light to change.

2. Truck (Right Camera View):

- Location: In the adjacent lane to the right.

- Status: Moving forward. - Indicator Lights: No visible indicator lights.

- Importance: The truck's movement needs to be monitored to ensure there is no sudden

lane change into the ego vehicle's lane.

3. Cars (Left Camera View):

- Location: On the left side of the road.

- Status: Stationary.  - Indicator Lights: Not applicable.

- Importance: not immediately relevant to the ego vehicle's movement, requiring caution.

Reasoning on Intent:

- Traffic Lights: The red traffic light indicates that all vehicles must stop and wait until the light

turns green.

- Lane Markings: The ego vehicle is in the correct lane for moving forward, as per the driving

command 'keep forward.'

- Other Vehicles: The white van directly in front is stationary, and the truck in the adjacent

lane is moving forward without any indication of changing lanes.

- Ego Vehicle State: The ego vehicle is currently stationary with zero velocity and

acceleration, consistent with the requirement to stop at the red light. Given the red traffic

light and the stationary vehicle directly in front, the ego vehicle should remain stopped until

the light turns green.

Best Driving Action: Stop.

The ego vehicle should remain stationary, adhering to the red traffic light and maintaining a

safe distance from the vehicle in front. Once the light turns green, the vehicle can proceed

forward.

Reasoning Data

Front

Front Right

Front Left

Front

Front Right

Front Left

System Prompt

- Role:

You are an Advanced Driver Assistance and Full Self-Driving System.

- Task:

You will receive visual observations from the ego vehicle's cameras and dynamic information

about the vehicle's current state.  Your task is to predict the optimal driving action for the next five

seconds. First, carefully analyze the surrounding environment by considering traffic lights, the

movements of other vehicles and pedestrians, lane markings, and any other relevant factors. If

necessary, use step-by-step reasoning (Chain-of-Thought) to arrive at the best driving action.

Otherwise, you may directly predict the final driving action.

- Fast/Slow Thinking:

If necessary, use step-by-step reasoning (Chain-of-Thought) to arrive at the

best driving action. Otherwise, you may directly predict the final driving action.

Structure your reasoning as follows:

1.

Scene Analysis

: Describe the traffic situation, including relevant environmental cues such

as traffic lights, lane markings, and the behaviors of surrounding vehicles or pedestrians.

2.

Identification of Critical Objects

: Identify two to three critical road users or obstacles,

specifying their relative positions to the ego vehicle.

3.

Prediction of Critical Object Behavior

: Predict the potential movements of the identified

critical objects.

4.

: Based on the observed environment and current vehicle

state, reason about the desired intent of the ego vehicle.

5.

Final Action Decision

: Select one lateral action and one longitudinal action:

-

Lateral actions

(choose exactly one): [move forward, turn left, change lane to left, turn

right, change lane to right]

-

Longitudinal actions

(choose exactly one): [stop, deceleration to zero, maintain

constant speed, quick deceleration, deceleration, quick acceleration, acceleration]

Present the final action clearly after your reasoning steps.

- Input Description:

The autonomous vehicle is

equipped with three cameras mounted at the front, left,

and right, enabling a comprehensive perception of the

surrounding environment.

- Multi-view Camera Streams:

- Front Camera:

The first video presents the front

view of the vehicle, comprising four sequential

frames sampled at 2 Hz.

[front camera frames]

- Front Left Camera:

The second video presents

the front-left view of the vehicle, comprising four

sequential frames sampled at 2 Hz.

[front left camera frames]

- Front Right Camera:

The third video presents

the front-right view of the vehicle, comprising four

sequential frames sampled at 2 Hz.

[front right camera frames]

- Ego States:

The current velocity of the vehicle is

3.1

m/s

, and the current acceleration is

1.4 m/s²

.

- Instruction:

The driving instruction is

Keep Forward

.

Based on this information, plan the action trajectory for

the autonomous vehicle over the next five seconds.

User Message

gradients, and buffers to reduce memory usage and accelerate computation. Gradient checkpointing is enabled to reduce GPU memory consumption. The learning rate warm-up is 500 steps and decays by 2% every 2,000 steps. The model is trained for 5 epochs, with gradient clipping applied at a maximum value of 1.0 to ensure training stability.

The model takes both system and user prompts as context inputs. As illustrated in Fig. S5, the system prompt defines the model's role, task, and the expected CoT reasoning format. The user message describes the multi-view camera observations, the ego vehicle's current state (i.e., speed and acceleration) and optionally historical states, and the high-level driving instruction. We use three camera views (front, front-left, and front-right), each providing four consecutive frames. Images in the video stream are resized to maintain original aspect ratios but reduced to 28 × 28 × 128 pixels.

Algorithm 1 RFT for AutoVLA with GRPO

Input: Supervised fine-tuned policy π SFT , Action Codebook A , Group size G , Training step K ,

Dataset D , Driving reward function r Driving , Reasoning reward function r CoT , Balance weigh

λ , KL regularization weight β

r Output: Reinforcement fine-tuned policy π RFT 1: Initialize reference policy π ref ← π SFT and current policy π θ ← π SFT 2: for training step 1 to K do 3: Sample scenario U from D 4: for sample i from 1 to G do 5: Sample input query q , final output o i , and per-token probability π θ ( o i | q ) 6: π θ old ( o i | q ) ← π θ ( o i | q ) 7: Calculate per-token probability π ref ( o i | q ) of π ref under o i 8: Decode trajectory τ ←A ( o i ) 9: Calculate reward r i ← r Driving ( τ, U ) -λ r r CoT ( o i ) 10: end for 11: Group average reward ¯ r, σ r ← mean( r 1 , . . . , r G ) , std( r 1 , . . . , r G ) 12: Group relative advantage A i ← ( r i -¯ r ) / ( σ r ) 13: L RFT ←-∑ G i ( -π θ ( o i | q ) π θ old ( o i | q ) A i + β D KL ( π θ ( o i | q ) ∥ π ref ( o i | q )) ) 14: Optimize π θ according to L RFT 15: end for 16: return Converged policy π RFT

D Details of Reinforcement Fine-tuning

Group Relative Policy Optimization (GRPO) employs group-based sampling to compute the advantage function, replacing conventional state-value estimators or critic models. This design accelerates training while aligning naturally with the inherent multimodality of planning, which requires evaluating and selecting from a set of candidate trajectories. The overall reinforcement fine-tuning (RFT) procedure is illustrated in Algorithm 1.

D.1 Kullback-Leibler (KL) Divergence

The KL divergence term D KL is incorporated in Eq. (3) to regularize the current policy with respect to the reference policy:

where θ denotes the parameters of the current policy π θ , q is the scenario input query, o i is the output of i th sample in the group, and π ref is the reference policy from the SFT stage. This regularization term penalizes large deviations from the reference policy, ensuring that policy updates remain within a stable area. As a result, the model retains useful knowledge acquired during SFT while improving driving behavior through reinforcement learning guided by validated reward functions.

D.2 Reward Function

The reward function provides the primary RFT training signal, guiding the optimization of policy updates. To further enhance comprehensive driving performance beyond imitating expert driving, we adopt a hybrid reward design that integrates the primary driving reward r Driving and a CoT penalty r CoT to discourage lengthy reasoning. For the NAVSIM Benchmark, we employ Predictive Driver Model Score (PDMS) [51] as the primary reward. For the Waymo end-to-end dataset, due to the limited annotation of RFS, we employ the (normalized) Average Displacement Error (ADE) as the driving reward. The final reward function, balanced by a weighting coefficient λ r , is defined as:

In the nuPlan dataset, we adopt the PDMS score as the driving reward signal r Driving, which serves as a comprehensive measure of driving quality. It is defined as follows:

where the components include No at-fault Collision (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Comfort (C), and Time-to-Collision (TTC). Detailed descriptions of each component can be found in [51]. Each sub-score is expressed as a percentage, and their weighted combination yields a normalized score within the range [0 , 1] , reflecting aspects of safety, comfort, and progress. For any scenario in which the planner fails (due to output errors), we assign a score of 0 .

For the Waymo dataset, we define the driving reward r Driving based on the ADE metric:

where δ denotes the maximum displacement error, and κ is a scaling factor to normalize the reward. The planning trajectory ˆ y is evaluated against the ground truth trajectory y , and ADE is computed as the average L2 distances over T time steps.

To penalize excessively long reasoning chains, we introduce a CoT penalty term r CoT, defined via a sigmoid function that normalizes the length of reasoning:

where L denotes the length of the CoT reasoning, L tol is the tolerance threshold, and γ is a scaling coefficient that controls the steepness of the penalty curve.

D.3 Implementation Details

We use the navtrain split of the nuPlan dataset for RFT for the NAVSIM benchmark and the validation split of the Waymo end-to-end driving dataset for the Waymo benchmark. The vision encoder of pretrained AutoVLA is frozen, and the model is fine-tuned using Low-Rank Adaptation (LoRA) to reduce training costs and memory consumption. Specifically, both the LoRA rank and alpha are set to 8, with a dropout rate of 0.1. The pretrained SFT model is used as the reference policy during optimization. The hyperparameters γ , L tol , and learning rate are set to 2 × 10 -3 , 400 , and 3 × 10 -5 , respectively. To ensure that the driving reward signal is dominant, the regularization weight is set to a relatively small value, λ r = 0 . 3 . We also set δ = 2 , κ = 10 in the RFT of the Waymo dataset. Moreover, we configure the generation parameters with a sampling temperature of 1 . 0 , top-p of 1 . 0 , and top-k of 0 . 0 to encourage diverse and exploratory generation during GRPO sampling, which effectively covers a wider range of possible actions.

E Experiment Details

E.1 Data Preprocessing

To enable mixed training across multiple driving datasets, we develop a unified data preprocessing pipeline that standardizes the format across all datasets. For each sample, we extract and standardize: 1) Ground truth trajectory coordinates and headings in the ego vehicle's coordinate frame at 2 Hz, 2) Image paths for multi-view camera image sequences consisting of 4 consecutive frames captured at 2 Hz (providing 2 seconds of history), 3) CoT reasoning annotations, 4) Vehicle states including its current velocity and acceleration, 5) High-level driving instructions. The preprocessing pipeline handles dataset-specific differences in data format, sampling rates, and coordinate systems to create a consistent format. The size and distribution of the final formatted dataset are shown in Table S1.

nuPlan (NAVSIM). We randomly sample 45.6k scenarios from the nuPlan trainval split and generate reasoning data using our proposed automated annotation pipeline. The resulting reasoning samples, together with the remaining training data with only trajectory annotation, constitute our full training set of nuPlan. Following the NA VSIM benchmark, the navtest split is used as the test set.

nuScenes. We preprocess all samples from the training set. For samples included in the DriveLM dataset, we reformat the question-answer (QA) pairs to generate structured reasoning annotations following our four-step reasoning format. Samples not covered by DriveLM are also used for training but only with trajectory supervision. The validation set is used for testing.

Waymo. The Waymo end-to-end driving dataset provides 2037 training and 479 validation segments, each containing a 20-second video with driving logs for the entire duration. We sample reasoning data using a 4-second sliding window and extract trajectory-only data using a 2-second sliding window offset by 1 second from the reasoning samples. Due to noise in the position data, the estimated vehicle heading can exhibit abrupt fluctuations when the vehicle is stationary. To address this issue, we apply a motion threshold to detect stationary periods and smooth the heading accordingly. The test set comprises 1505 samples.

CARLA. The CARLA-Garage dataset is used to train the model for closed-loop evaluation. Since only front camera images are available, we use single-view inputs (with four consecutive frames) for CARLA training and testing. We sample data using a sliding window with an offset of 0.5 seconds, and downsample the trajectories from 4 Hz to 2 Hz. For reasoning annotations, we leverage the DriveLM-CARLA dataset, which provides QA pairs similar to DriveLM-nuScenes, and we reformat the QA pairs to generate samples with reasoning annotations.

E.2 Evaluation Metrics

nuPlan (NAVSIM). PDMS is the official benchmark metric in NA VSIM on the nuPlan dataset, with its formulation provided in Eq. (S3).

nuScenes. Following previous works, we adopt the L2 Distance to measure the average displacement error between predicted and ground-truth trajectories, and use Collision Rate to assess the frequency of predicted trajectories overlapping with surrounding objects. We follow the UniAD protocol, reporting both metrics at 1, 2, and 3 seconds in the future, instead of averaging over the horizons.

Waymo. Following the Waymo end-to-end driving benchmark, Rater Feedback Score (RFS) is used to evaluate the driving performance. The Waymo dataset provides three human-annotated trajectories, each associated with a scalar quality score in [0 , 10] . A planning trajectory is matched to its closest reference trajectory, and a trust region is defined around it using lateral and longitudinal thresholds at fixed timesteps (3 s and 5 s). These thresholds are scaled by a piecewise linear function of the rater's trajectory's initial speed. Predictions within the trust region inherit the matched rater's score, and those outside are penalized exponentially based on normalized deviation.tr

CARLA. We evaluate driving performance in closed-loop testing using four key metrics: Driving Score , Success Rate , Efficiency , and Comfortness following the Bench2Drive benchmark [51]. Driving Score reflects the overall route completion penalized by infraction severity. Success Rate measures the percentage of routes completed without major infractions within the time limit. Efficiency quantifies how well the ego vehicle maintains a reasonable speed relative to surrounding traffic, computed as the average speed ratio over multiple checkpoints. Comfortness assesses the smoothness of the trajectory based on physical signals such as acceleration, jerk, and yaw rate, using expert-derived thresholds.

E.3 Model Training

Our primary base model is trained on the mixed training set of the nuPlan and nuScenes datasets. The ego state contains the vehicle's current velocity and acceleration. For the test on the Waymo end-to-end driving dataset, we further fine-tune this base model using the Waymo end-to-end driving

<think> This is a straightforward scenario, and a direct decision

can be made. </think>

<answer>

The final output action is:

<action_1122><action_1456>…

</answer>

Ground Truth

Planning

Fast Thinking

Ground Truth

Planning

Ground Truth

Planning

<think> This is a complex scenario requiring additional reasoning

.

Scene Description:

There are many trucks and one trailer to the front left of the ego car.

There are two trucks and two trailers to the front right of the ego car.

Critical Object Description:

There is a

white truck

to the front left of the ego vehicle, and a

red and

white truck

to the front of the ego vehicle… The moving status of

the

red and white truck

is stationary. The moving status of

the white truck

is stationary…

Reasoning on Intent:

Firstly, notice the

red and white truck

. The

red and white truc

k is

obstructing the lane ahead and remains stationary. The ego vehicle

must yield and wait for the obstacle to clear…

Best Driving Action:

STOP. </think>

<answer>

The final output action is:

<action_617><action_404>…

</answer>

Slow Thinking

Ground Truth

Planning

Slow Thinking

<think> This is a straightforward scenario, and a direct decision

can be made. </think>

<answer>

The final output action is:

</answer>

Fast Thinking

<think> This is a straightforward scenario, and a direct decision

can be made. </think>

<answer>

The final output action is:

<action_452><action_459>…

</answer>

Fast Thinking

<think> This is a complex scenario requiring additional reasoning

.

Scene Description:

There is a white SUV in front of the ego car. There is a green traffic light

in front of the ego car. There is a black car to the front left of the ego

car...

Critical Object Description:

There is a

white SUV

to the front left of the ego vehicle and a green light

to the front of the ego vehicle …The moving status of the

White SUV

is

going straight…

Reasoning on Intent:

Firstly, notice the

White SUV

. The object is moving, so the ego vehicle

should continue... Secondly, notice the

Green light

. The object is a

traffic sign, so the ego vehicle should move ahead …

Best Driving Action:

Move forward with acceleration. </think>

<answer>

The final output action is:

<action_626><action_208>…

</answer>

Ground Truth

Planning

Slow Thinking

<think> This is a complex scenario requiring additional reasoning

.

Scene Description:

There are many cars and one pedestrian to the front left of the ego car.

There is one car in front of the ego car. There are two cars to the front

right of the ego car....

Critical Object Description:

There is a

black SUV

to the front of the ego vehicle, …, and a

red light

to front of the ego vehicle. The moving status of

the Black SUV

is

stationary…

Reasoning on Intent:

Firstly, notice

the Red light

. It is a traffic sign, so the ego vehicle should

remain stationary. Secondly, notice

Black SUV

…

Best Driving Action:

STOP. </think>

<answer>

The final output action is:

<action_1882><action_1067>…

</answer>

Ground Truth

Planning

dataset. For the model trained on Waymo, the ego vehicle's state encompasses current acceleration and a 4-second history of vehicle positions and velocities. For closed-loop simulation testing, we train a separate model using data preprocessed from the CARLA Garage dataset and DriveLM-CARLA annotations. This model is trained with single-view inputs instead of the multi-view setup used in the primary model, and we use a large resolution for the input images (with 28 × 28 × 384 pixels).

E.4 Inference

We employ a stochastic generation strategy using top-p and top-k sampling to generate reasoning and planning outputs. Higher sampling diversity (e.g., temperature= 1 . 0 , top-p= 0 . 5 , top-k= 20 ) supports slow thinking modes, enabling the model to produce deeper and more elaborate reasoning chains. In contrast, more deterministic settings (e.g., temperature= 0 . 1 , top-p= 0 . 01 , top-k= 1 ) produce fast thinking, yielding consistent and direct responses.

<think> This is a complex scenario requiring additional reasoning.

Scene Description:

The ego vehicle is at an intersection in an urban environment. The front camera shows a

stop sign, a red traffic light, and a 'No Left Turn' sign on the left side of the road… The

front-right camera shows a curved road into a parking lot with a stop sign and a bump

warning, while the back camera shows traffic queued behind, suggesting the ego vehicle

recently began moving forward…

Critical Object Description:

1. Traffic light (Front view):

Located at the upper-left of the intersection, currently red. It is crucial as it controls

crossing traffic and the ego vehicle's entry into the intersection.

2. Black car ahead (Front view):

Located directly ahead, it is proceeding forward into the intersection, helping validate that

vehicles are allowed to go straight.

3. 'No Left Turn' sign (Front view):

Located below the stop sign, this prohibits left turns from the ego vehicle's current lane.

Reasoning on Intent:

- The 'No Left Turn' sign and the stop sign indicate that vehicles must stop and cannot turn

left.

- The red traffic light is positioned toward the left, suggesting it might only control left-

turning traffic, not forward movement.

- The ego vehicle was stopped in the past 4 seconds and is now moving slowly with a low

acceleration, indicating it's starting from a stop sign.

Best Driving Action: Move forward with acceleration

.

Given the road signs, vehicle behavior, and instruction to go forward, ego vehicle has

already stop on the stop sign, and can move forward with acceleration to follow the vehicle

ahead, aligning with the ego vehicle's instruction.

</think>

<answer>

The final output action is: <action_28><action_1>…

</answer>

Slow Thinking

<think> This is a complex scenario requiring additional reasoning.

Scene Description:

The scene is a daytime urban intersection with clearly marked lanes, pedestrian sidewalks,

and visible green traffic lights. A van is in front of the vehicle, and two lanes ahead with

yellow box markings. Two pedestrians are walking on the sidewalk in the left...

Critical Object Description:

1. Pedestrians (Left View):

- Status: Walking on the sidewalk.

- Prediction: Will likely continue walking without crossing the road.

- Importance: Must ensure they do not suddenly enter the crosswalk during the left turn.

- Status: Appears to be moving slowly or preparing to proceed straight.

- Prediction: May go straight or turn right.

- Importance: Ego vehicle must monitor for potential conflict if the van turns unexpectedly.

3. Green traffic light (front view)

- Status: Green in the ego lane's direction.

- Prediction: Will allow safe execution of left turn.

- Importance: Confirms that the ego vehicle has the right-of-way to begin the turn.

Reasoning on Intent:

- Firstly, the ego vehicle has been stopped in the past 4 seconds, indicating readiness to

execute a maneuver.

- Secondly, the green traffic light ahead authorizes vehicles in the ego's direction to proceed,

including for left turns.

- Thirdly, there are no immediate vehicles blocking the left-turn path, and the intersection is

mostly clear.

- Finally, while there are pedestrians on the sidewalk, they show no intent to cross, and no

opposing vehicles are visibly entering the intersection…

Best Driving Action: Turn left with acceleration.

This ensures a smooth maneuver respecting the traffic light signal, allowing the ego vehicle

to safely and efficiently execute the left turn while monitoring surroundings.

</think>

<answer>

The final output action is: <action_842><action_1470>…

</answer>

Slow Thinking

Ground

Truth

Planning

Ground

Truth

Planning

Ground

Truth

Planning

Ground

Truth

Planning

Fast Thinking

Fast Thinking

<think> This is a straightforward scenario,

and a direct decision can be made. </think>

<answer>

The final output action is: <action_1726><action_191>…

</answer>

<think> This is a straightforward scenario,

and a direct decision can be made. </think>

<answer>

The final output action is: <action_1601><action_485>…

</answer>

Front

Front Right

Front Left

Front

Right

Left

Front

Front Right

Front Left

Right

Left

Front

Front Right

Front Left

Right

Left

Front

Front Right

Front Left

Right

Left

Ground

Truth

Planning

?

F Additional Results

F.1 nuScenes Results

We evaluate AutoVLA on the nuScenes dataset following both ST-P3 [14] and UniAD [65] protocols, in comparison with state-of-the-art end-to-end models. As shown in Table S2, our model demonstrated competitive performance in the nuScenes planning benchmark.

Fig. S6 shows AutoVLA's planning outputs. The model generates safe trajectories that closely align with ground-truth motions, accompanied by coherent and context-aware reasoning outputs. However, it can be observed that many nuScenes scenarios are relatively straightforward, often not requiring complex reasoning. This may explain the lack of performance gain in quantitative metrics when reasoning is introduced.

F.2 nuPlan Results

We present additional visualization results of our model on the nuPlan dataset in Fig. S7. In relatively simple scenarios such as curved roads and intersections, our model generates high-quality trajectories via fast thinking. In more complex scenes with numerous traffic regulations, it leverages slow thinking

Ground

Truth

Planning

Ground

Truth

Planning

Front

Back

Front Left

Front Right

<think> This is a complex scenario requiring additional reasoning

.

Scene Description:

The scene appears to be an urban area with a mix of pedestrian pathways and roads. The front view shows a

pathway leading towards a building entrance... The weather seems clear, and ...

Critical Object Description:

1. Vehicles on the right side (Right View):

Parked along the side of the road in the right view.

2. Building entrance and pathway (Front View):

Directly in front of the vehicle. The pathway leads towards

a building entrance. The entrance has glass doors and pillars ...

Reasoning on Intent:

The driving command instructs the vehicle to turn right. The front view shows a clear path leading towards

the building entrance. The right view shows a road with vehicles, which needs to be navigated carefully. The

left view shows a sidewalk with trees and a building, which is not directly relevant to the current maneuver.

The vehicle is moving at a low speed with a slight deceleration, suggesting it is preparing to make the turn.

Best Driving Action:

Turn right with a deceleration.

Given the need to navigate around the vehicles on the right and …

</think>

<answer>

The final output action is: <action_2022><action_2022>…

</answer>

Slow Thinking Before RFT

Slow Thinking After RFT

<think> This is a complex scenario requiring

additional reasoning

.

Scene Description:

The scene appears to be an urban area with a

mix of pedestrian pathways and roads. The

front view shows a pathway lined with bollards

and a building entrance... The weather seems

clear, and ...

Critical Object Description:

1. A Car (Right View):

Parked or slow driving

along the side of the road in the right view.

2. Building entrance (Front View):

Located directly in front of the vehicle, near the

bollards. The entrance is static and poses no immediate threat. The entrance will

remain static unless someone exits or enters the building. While not immediately

critical, being aware of the surroundings helps avoid collisions with fixed objects like

bollards.

Reasoning on Intent:

The ego vehicle is currently moving slowly with a slight deceleration, as indicated by the

velocity and acceleration values. The driving command is to turn right, suggesting the

vehicle intends to make a right turn soon. The absence of pedestrians in the immediate

vicinity allows for a smooth maneuver. However, the ego vehicle should navigate

carefully to ensure no pedestrians are about to cross.

Best Driving Action:

Turn right with a deceleration.

Given the current situation, the vehicle should proceed

with a right turn while maintaining a controlled deceleration to ensure safety …

</think>

<answer>

The final output action is: <action_2022><action_2022>…

</answer>

to produce better and regulation-compliant planning results with CoT reasoning. As illustrated in Fig. S8, our AutoVLA model preserves reasoning capabilities in complex scenarios despite removing redundant reasoning after RFT. Qualitative comparisons demonstrate that RFT enhances planning performance in complex scenarios.

F.3 Waymo Results

To overcome the limited size of the Waymo end-to-end driving dataset, we augment the training data with samples from the nuPlan and nuScenes datasets. For convenience, we refer to this combined dataset as nuX . We use this nuX data to pretrain the model before fine-tuning it on the Waymo dataset. To further enhance planning performance, we apply RFT on the AutoVLA model after SFT. Given the limited number of RFS-labeled samples (only 480 in the validation set), we adopt the average displacement error (ADE) as the primary reward signal.

From the test set, no ground truth is provided

Planning

<think> This is a complex scenario requiring additional reasoning

.

Scene Description:

The scene is a residential street during the daytime with clear visibility. The road is marked with orange traffic cones, indicating a construction zone ahead.

Several vehicles are visible, including a white truck directly in front of the ego vehicle and a few parked cars along both sides of the road.

Critical Object Description:

1. White Truck:

Located directly in front of the ego vehicle, it appears to be moving slowly or stopping. This vehicle is crucial as it dictates the flow of traffic in

the construction zone.

- What it is doing:

Moving slowly or stopping due to the construction zone.

- Why it is important:

The ego vehicle must maintain a safe distance and adjust

its speed accordingly.

2. Traffic Cones:

Positioned along the center of the road, these cones indicate a narrowed lane due to construction.

- What they are doing:

Marking the path for vehicles in the construction zone.

- Why they are important:

The ego vehicle must navigate around them

carefully to avoid collisions.

Reasoning on Intent:

1. White Truck:

The white truck is directly in front of the ego vehicle and appears to be slowing down or stopping. The ego vehicle should match its speed to

maintain a safe following distance.

2. Traffic Cones:

The presence of traffic cones indicates a narrowed lane, requiring the ego vehicle to proceed cautiously and potentially reduce speed

further if necessary.

The ego vehicle has been decelerating over the past 2 seconds and is currently moving at a relatively low speed. The driving command is to go straight, which

aligns with navigating through the construction zone. Given these observations, the ego vehicle should continue to decelerate and proceed cautiously while

maintaining a safe distance from the white truck and navigating around the traffic cones.

Best Driving Action:

Move forward with a deceleration.

This action ensures the ego vehicle safely navigates through the construction zone while maintaining a safe distance

from the white truck and avoiding any potential hazards posed by the traffic cones.

</think>

<answer>

The final output action is: <action_392><action_310>…

</answer>

<think> This is a complex scenario requiring additional reasoning.

Scene Description:

The scene is at night, and the road appears well-lit by streetlights. The ego vehicle is traveling on a multi-lane road with visible lane markings. In the front

view, several vehicles are ahead, maintaining a moderate distance. A cyclist is riding along the edge of the road. Traffic lights are green, allowing vehicles to

proceed…

Critical Object Description:

1. Vehicles Ahead:

Located directly in front of the ego vehicle, these vehicles appear to be moving steadily. They are maintaining a consistent distance from

each other, suggesting normal traffic flow.

- What they are doing:

Moving forward.

- What they will do:

Likely continue moving forward as the traffic light is green.

- Why important:

The ego vehicle

must maintain a safe following distance to avoid collisions.

2. Cyclist on the Road:

Positioned on the right side near the edge of the road, the cyclist is riding parallel to the ego vehicle.

- What they are doing:

Riding

along the edge of the road.

- What they will do:

Continue riding along the edge.

- Why important:

The ego vehicle must maintain a safe lateral distance to avoid any potential collision

with the cyclist...

Reasoning on Intent:

1. Traffic Lights:

The green traffic lights indicate that the ego vehicle can proceed without stopping.

2. Vehicles Ahead:

The vehicles ahead are moving steadily, suggesting that the ego vehicle can maintain its current speed and follow them.

3. Cyclist on the Road:

The cyclist is riding along the edge and poses no immediate threat to the ego vehicle's path.

Given the green traffic light, the steady movement of vehicles ahead, and the absence of any immediate obstacles or hazards, the ego vehicle should

continue moving forward at a constant speed. …

Best Driving Action:

Move forward with a constant speed. </think>

<answer>

The final output action is: <action_1386><action_1333>…

</answer>

Ground Truth

Planning

Slow Thinking

Ground Truth

Planning

Planning

<think> This is a complex scenario requiring additional reasoning.

Scene Description:

The scene is a residential area during the daytime with clear weather. The road is a two-lane street with no visible lane markings. There are houses, trees,

and bushes along both sides of the road. A pedestrian is walking on the sidewalk to the right of the ego vehicle. A dog is running ahead on the road. Several

parked cars are visible on the left side of the road…

Critical Object Description:

1. Pedestrian:

The pedestrian is walking along the sidewalk and appears to have no intention of crossing the road.

- What they are doing:

Walking on the sidewalk.

- What they will do:

Continue walking on the sidewalk without crossing the road.

- Why they are

important

: The pedestrian is near the ego vehicle's path, so it is crucial to maintain a safe distance and avoid any sudden movements that could startle

them.

2. Dog:

Located ahead of the pedestrian on the road. The dog is running freely on the road.

- What they are doing:

Running on the road.

- What they will do:

Continue running on the road.

- Why they are important:

The dog is in the path of the ego

vehicle, and maintaining a safe distance is essential to avoid any potential collision.

Reasoning on Intent:

1.

The pedestrian

has no intention of crossing the road. The ego vehicle should maintain a safe distance from the pedestrian while  moving forward.

2.

The dog

is running ahead of the pedestrian on the road. The ego vehicle should slow down to maintain a safe distance from the dog and avoid any sudden

movements that could cause it to swerve or collide with the dog

The ego vehicle has been moving forward with a constant speed in the past 2 seconds, and the current driving command is to go straight. However, due to

the presence of the pedestrian and the dog, the ego vehicle needs to adjust its speed to ensure safety.

Best Driving Action:

Move forward with a deceleration.

This action allows the ego vehicle to maintain a safe distance from the pedestrian and the dog while continuing to move

forward as instructed by the driving command. Decelerating ensures that the ego vehicle can react appropriately to any unexpected changes in the

environment, such as the pedestrian suddenly crossing the road or the dog swerving into the other lane.

</think>

<answer>

The final output action is: <action_87><action_125>…

</answer>

<think> This is a complex scenario requiring additional reasoning.

Scene Description:

The scene is at night, and the road appears to be undergoing construction or maintenance work. There are several vehicles stopped ahead, likely due to

the traffic control setup. A worker in high-visibility clothing is directing traffic, indicating a lane closure. The front left and front right cameras show clear

paths to the left and right sides of the road, respectively.

Critical Object Description:

1. Worker in High-Visibility Clothing:

Located in the center-right of the front view, the worker is actively directing traffic. This indicates that the left lane is

closed or restricted, and vehicles must stop or change lane.

2. Vehicles Ahead:

Several vehicles are stopped ahead, blocking the path. They appear stationary, likely due to the traffic control setup. The vehicles are

positioned in such a way that they prevent direct passage forward.

Reasoning on Intent:

1. Traffic Control Worker:

The worker in the high-visibility clothing clearly indicates that the vehicle should stop.

2. Stationary Vehicles Ahead:

The presence of stopped vehicles ahead reinforces the need to stop and wait for the appropriate signal or clearance before

proceeding. The ego vehicle has been stationary for the past 2 seconds, indicating compliance with the traffic control setup.

Best Driving Action:

Stop.

The ego vehicle should remain stationary, adhering to the traffic control setup and waiting for the appropriate signal or clearance to proceed. This

ensures safety for both the workers and the vehicles ahead, as well as maintaining smooth traffic flow.

</think>

<answer>

The final output action is: <action_0><action_0>…

</answer>

Ground Truth

Planning

Ground Truth

Planning

(a)

(c)

(b)

From the test set, no ground truth is provided

(d)

Slow Thinking

Slow Thinking

Slow Thinking

S

As of May 22, 2025, AutoVLA achieves competitive performance on the Waymo End-to-End Driving Challenge leaderboard, as shown in Table S3. The model ranks highly in both RFS Overall and ADE metrics and achieves the top score in the RFS Spotlight metric , which focuses on the most challenging scenarios. Qualitative results are shown in Fig. S9, where AutoVLA demonstrates its ability to generate safe, context-aware trajectories in complex environments. The model effectively handles interactions, construction zones, and traffic regulations in diverse scenarios while providing coherent reasoning to justify its decisions.

Table S4 presents the ablation studies of the AutoVLA model in different training setups. The results indicate that multi-camera input consistently enhances driving performance. When trained solely on the Waymo end-to-end driving dataset, incorporating reasoning significantly improves performance compared to action-only setups. Moreover, pretraining on nuX data provides a substantial performance boost, suggesting that such pretraining enhances the scene understanding of the model with more driving data. RFT can significantly improve planning performance by mitigating error accumulation in generation and aligning with the task-specific rewards.

F.4 CARLA Results

We evaluate the model after SFT in closed-loop testing within the CARLA simulator. The replanning frequency is set to 2 Hz, meaning the AutoVLA model is queried every 0.5 seconds in simulation, and the planned trajectory is used to generate control commands. High-level driving instructions are derived from a predefined route plan, while the vehicle's current state (including speed and acceleration) is obtained from the IMU and speedometer sensors. The model receives four RGB images from the front-camera sensor covering the past two seconds as visual input. AutoVLA predicts a five-second trajectory, which is then used by a PID controller to compute the control actions (throttle, brake, and steering) that are applied to the vehicle.

Closed-loop testing results in the CARLA simulator are shown in Fig. S10. Two representative scenarios are illustrated: (1) the ego vehicle equipped with AutoVLA successfully responds to a cut-in vehicle, and (2) it executes a smooth left turn. Additional closed-loop simulation results are available on the project website.

t=18s

t=22s

t=20s

t=7s

t=9s

t=13s

t=11s

G Broader Impacts

Autonomous driving is a safety-critical system, further emphasized by the integration of language guidance into the VLA model. This integration necessitates robust safeguards against adversarial attacks and proactive identification and filtering of unsafe human instructions. To mitigate potential hacking threats, it is essential to establish a secure communication channel in the vehicle, complemented by a gated-release strategy for model updates rather than online continual reinforcement fine-tuning on individual vehicles.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel autoregressive framework for end-to-end automated driving, by using a VLA with tokenized and discretized actions. The actions were tokenized to remove the possibility of impossible actions. The authors' model works with a dual-thinking approach where slow-thinking produces a chain-of-though (CoT) as well as a set of action tokens for the next 5 seconds of planning. Meanwhile quick-thinking skips CoT and just returns a set of action tokens. The authors first train using a supervised fine-tuning approach in which they use both ground truth for planning as well as reasoning data for CoT, where reasoning data was created as distilled knowledge from a larger VLM. After SFT the authors propose an RFT approach using GRPO to enhance model efficacy and performance. The model was tested on multiple benchmarks and shows performant results.\n\nSTRENGTHS AND WEAKNESSES:\n1. Quality - The paper is well written and all the claims the paper formulated are properly addressed and argumented. The methods that were used to test the model are appropriate and in line with traditional benchmarks. The paper does not provide a performance test, as it is only mentioned once, but no further analysis was done, only stating that it achieves near real-time performance using many GPUs. The results of experiments are just presented as is, without any real discussion other than putting into words what is said on the graphs. No deeper understanding of results, or discussing potential drawbacks of the method.\n2. Clarity - While the main methodological and experimental approaches are defined precisely, the main drawback I see here is that the actual process of discretizing and tokenizing the actions is not explained in detail as other parts which are less significant as this is one of the main contributions of the paper. Giving a better overview as to how it was discretized, why the specific use of short-term spatial positions, why K-disk clustering (and how it was set up) etc. would benefit the paper greatly.\n3. Significance - The results are promising but not as good as other simpler methods provided in benchmarks competing against the proposed model, in all but one category when compared to other models is this proposed model better, and that is when using best-of-N planning which other models don't have access to. Meanwhile the one-shot and post-RFT models are quite underperformant compared to others, averaging the lower half of metrics when ranked against provided models.\n4. Originality - The main insight from my perspective that the paper provides is that when tokenizing the action space there can be no nonsensical outputs. Which is more of a product of the method than an insight, since there are only feasible actions the model can only output feasible actions. Other parts of the model, pipeline and framework are not novel and do not provide new insights, but are rather logical results of steps taken in the paper. The paper does provide an overview of a framework though, but there is no real scientific insight gained here, more an engineering assessment of a framework, that in the end does not provide any inherent benefits over others performance-wise.\n\nQUESTIONS:\n1. Please provide a more detailed explanation as to the exact procedure of tokenizing the action space. Why do you tokenize in such a way? Why do you use K-disk and not something else? Why do you use short-form why not a different approach etc.?\n2. Why is the method worse in some categories than other benchmark models? Please discuss and analyze the results, not only present them.\n\nLIMITATIONS:\nThe limitations subsection only discusses the real-world feasibility of the method, while not touching up any other limitations of the model or discussing the results of experiments that were on average below the other methods in some cases by a larger margin.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces AutoVLA, a novel Vision-Language-Action (VLA) framework for end-to-end autonomous driving. AutoVLA integrates reasoning and action generation within a single autoregressive model, supporting dual thinking modes: \"fast thinking\" for direct trajectory generation in straightforward scenarios and \"slow thinking\" with chain-of-thought (CoT) reasoning for complex situations. The training involves supervised finetuning (SFT) with trajectory and reasoning data, followed by reinforcement finetuning (RFT) to optimize planning performance and reduce unnecessary reasoning. The first highlight of the paper is its strong planning/driving performance on both real-world (nuPlan, nuScenes, Waymo) and simulated (CARLA) datasets. The second highlight is its ability to improve planning/driving performance when reasoning is involved, which is a capability not demonstrated by previous VLM-based frameworks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. AutoVLA unifies reasoning and action generation in a single model, addressing limitations of prior models that often produce infeasible actions or rely on complex intermediate representations.\n\n2. The model can dynamically switch between fast and slow thinking modes to balance efficiency and accuracy. This is a feature not explored in previous work.\n\n3. The paper evaluates AutoVLA across various benchmarks (nuPlan, nuScenes, Waymo, CARLA), demonstrating strong planning and driving performance in both open-loop and closed-loop settings. \n\n4. AutoVLA also achieves significantly stronger performance when incorporating reasoning in the language space, as shown in Figure 6 and Table 1. This suggests that the planning and language branches are closely related and complementary. This is a relationship that previous papers have not explored.\n\nWeaknesses:\n\n1. While the paper emphasizes the reasoning capabilities of AutoVLA, it does not quantitatively evaluate AutoVLA's performance on language-related benchmarks like DriveLM. Since DriveLM is also based on nuScenes and CARLA, including such results would further strengthen the paper and better validate the model’s reasoning ability.\n\n2. The previous problem raises a second question. The reinforcement finetuning (RFT) process penalizes long language outputs with the $r_\\text{CoT}$ term, which improves efficiency but may compromise the reasoning depth. A more systematic evaluation or additional case studies would help clarify the trade-offs involved.\n\n3. Since AutoVLA is trained on the Carla-Garage dataset for Table 2 according to Line 230 (which adopts PDMLite as the expert planner), it would be more appropriate to include comparisons with stronger baselines trained on PDMLite-gathered datasets (e.g., TF++ and SimLingo), rather than only comparing against weaker baselines trained on the Bench2Drive dataset.\n\n4. As the model employs a large vision encoder to process multiple camera views across several frames, this inevitably results in high computational cost and inference latency. A latency up to 1–2 seconds would make the model impractical for real-world deployment. It would also be helpful to report latency statistics in more detail (such as maximum, minimum, and average latencies for the two thinking modes) rather than only providing an average value for the entire dataset.\n\nQUESTIONS:\nOverall, I think AutoVLA is a good paper. Please refer to the weakness section for questions. I only have two extra questions regarding the teacher model: how does the teacher model Qwen2.5-VL-72B perform on vehicles' ego status information? Are there any cases where the teacher model fails to make safe decision based on the textual velocity, acceleration, and historical actions?\n\nLIMITATIONS:\nPlease refer to the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents AutoVLA, an end-to-end vision–language–action (VLA) architecture for autonomous driving that (i) tokenises continuous vehicle motion into a learned “physical-action codebook” and feeds those tokens directly into a pretrained vision–language model (Qwen2.5-VL-3B) so that one autoregressive decoder jointly produces chain-of-thought (CoT) reasoning and a 5 s trajectory; (ii) trains with mixed supervision—trajectory-only (“fast thinking”) and trajectory + CoT (“slow thinking”); and (iii) applies reinforcement fine-tuning with Group-Relative Policy Optimisation (GRPO), using nuPlan PDMS and an explicit CoT-length penalty, to improve planning performance and suppress unnecessary reasoning.\nExperiments on nuPlan-NAVSIM, nuScenes, Waymo-E2E and CARLA Bench2Drive show competitive or SOTA open-loop and closed-loop metrics; ablations demonstrate gains from CoT, data scaling, action tokenisation and RFT.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Contributes a 2048-entry motion codebook and integrates it as additional vocabulary inside the LLM, avoiding extra decoders (Sec. 3.1, Table 3). Produces physically feasible trajectories directly;  outperforming text-waypoint outputs in L2/collision and latency.\n2. Mixes reasoning-augmented and reasoning-free samples (Sec. 3.3) so the same model can adaptively emit CoT when needed (Fig. 6c). Ablation (Fig. 4) shows that with sufficient data (>50 k) CoT improves PDMS and collision metrics.\n3. Extends RFT beyond meta-actions (AlphaDrive) to full trajectory tokens (Sec. 3.4), achieving +10.6 % PDMS and −66 % runtime on NAVSIM (Fig. 6a). Provides evidence that penalising token length indeed triggers “fast thinking”.\n\n4. Four real/sim datasets;  both open-loop (nuPlan, nuScenes) and closed-loop (Bench2Drive). AutoVLA (Best-of-N) tops NAVSIM PDMS (92.1) and achieves best driving score & success rate on Bench2Drive (Table 2).\n\n Weaknesses\n1. The author only validates the effectiveness of AutoVLA in the autonomous driving scenario but does not mention the robotic scenario, which would more prominently highlight the importance of action tokenization mentioned in the paper.\n2. The size of the action codebook (K=2048) is empirically chosen, without conducting relevant ablation studies or verifying its dynamic feasibility.\n3. More meticulous and in-depth experiments and analyses are needed for the fast and slow systems. There are no relevant quantitative results in the main text, and the results in the appendix do not show significant advantages.\n\nQUESTIONS:\nPlease refer to the \"Weakness\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an end-to-end autoregressive model for autonomous driving. The proposed model is built upon a pre-trained VLM backbone that performs adaptive reasoning and predicts trajectories, where each trajectory is tokenized into a sequence of physically feasible tokens. The authors first adopt supervised fine-tuning to enable the model to perform both reasoning and planning tasks, and then apply reinforcement fine-tuning to train the model to avoid unnecessary reasoning. Experiments conducted on multiple datasets validate the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written and well-organized, and it is easy to follow and understand.\n2. The proposed VLA is a straightforward and technically sound extension of the VLM. The experiments are comprehensive and demonstrate that the method is promising in certain cases.\n3. The proposed adaptive reasoning is interesting and practical in terms of improving efficiency.\n\nWeaknesses:\n1. The ablation study on the text action output is not sufficient to demonstrate the effectiveness of the proposed action tokenization method. There are multiple advanced action tokenization schemes, such as per-dimension, per-timestep binning scheme [1] and [2], which are not compared.\n2. In reinforcement fine tuning, the authors formulate the planning problem as a VQA task, where the output actions are not executed and therefore do not affect subsequent observations. This formulation is misleading and potentially limits the model's closed-loop planning performance.It is also unclear why the authors did not use RFT in closed-loop experiments.\n3. The significance of the proposed framework is difficult to assess. Considering the model size and inference time, the model doesn't bring much improvements and sometimes performs worse in the benchmark evaluations. It would be helpful if the authors can provide the model sizes and FPS of all compared methods. Moreover, the proposed CoT training degrades performance in simple scenarios like nuScenes, which is quite counterintuitive. Regardless of efficiency, CoT should help the model to make better decisions, why does it hurt performance in this case? It is also unclear to me if the RFT can mitigate this issue.\n\n[1] Brohan et al. \"Rt-1: Robotics transformer for real-world control at scale.\" arXiv 2022. \n[2] Pertsch et al. \"Fast: Efficient action tokenization for vision-language-action models.\" RSS 2025.\n\nQUESTIONS:\nAs noted in the weaknesses section, I would appreciate it if the authors could address the following questions:\n1. How do the authors justify the effectiveness of the proposed action tokenization method, given that it is only compared to a simple text action output?\n2. Why is RFT not used in closed-loop experiments? Could RFT improve performance on the nuScenes dataset when using CoT?\n3. Can the authors provide additional evidence to support the significance of the proposed framework\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-09"}