DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking
Authors: Weicheng Zheng
Xiaofei Mao
Nanfei Ye
Pengxiang Li
Kun Zhan
XianPeng Lang
Hang Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous Driving
Vision-Language Models
Hybrid Thinking
Visual Reasoning
Reinforcement Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The advent of Vision-Language Models (VLMs) has significantly advanced end-to-end autonomous driving, demonstrating powerful reasoning abilities for high-level behavior planning tasks. However, existing methods are often constrained by a passive perception paradigm, relying solely on text-based reasoning. This passivity restricts the model’s capacity to actively seek crucial visual evidence when faced with uncertainty. To address this, we introduce DriveAgent-R1, an autonomous driving agent capable of active perception for planning. In complex scenarios, DriveAgent-R1 proactively invokes tools to perform visual reasoning, firmly grounding its decisions in visual evidence, thereby enhancing both interpretability and reliability. Furthermore, we propose a hybrid thinking framework, inspired by human driver cognitive patterns, allowing the agent to adaptively switch between efficient text-only reasoning and robust tool-augmented visual reasoning based on scene complexity. This capability is cultivated through a three-stage progressive training strategy, featuring a core Cascaded Reinforcement Learning (Cascaded RL) phase. Extensive experiments on the Drive-Internal dataset, which is rich in long-tail scenarios, and the public nuScenes dataset show that, with only 3B parameters, DriveAgent-R1 achieves competitive performance comparable to top closed model systems such as GPT-5 and to human driving proficiency while remaining deployment-friendly, offering a proven path toward building more intelligent autonomous driving systems.

Published as a conference paper at ICLR 2026
DRIVEAGENTRRI:
ADVANCING VLM-BASED AUTONOMOUS DRIVING WITH ACTIVE PERCEPTION AND HYBRID THINKING
Weicheng Zheng 1.3*
Xiaofei M
Nanfei Ye²
Pengxiang Li
Kun Zhan
Xianpeng Lang²
Hang Zhao 1.41
'Shanghai Qi Zhi Institute LiAAto
TTongii University
*Tsinghua University
Project: http://tsinghuu--ar--lbb
ABSTRACT
The advent of Vision-Languuge Models (VLMs) has significantly advanced end- to-end autonomous driving. demonstrating powerful reasoning abilities for high- level behavior planning tasks. However, existing methods are often constrained by a passive perception paradigm, relying solely on text-based reasoning. This passivity restricts the model's 's capacity to actively seek crucial visual evidence when faced with uncertainty. To address this, we introduce DriveAgent-Rl, an autonomous driving agent capable of active perception for planning. In complex scenarios, DriveAgent-Ri proactively invokes tools to perform visual reasoning, firmly grounding its decisions in visual evidence, thereby enhancing both inter- pretability and reliability. Furthermore, we propose a hybrid thinking framework, inspired by human driver cognitive patterns, allowing the agent to adaptively switch between efficient text-only reasoning and robust tool-augmented visual reasoning based on scene complexity. This capability cultivated through three-stage progressive training strategy, featuring a core Cascaded Reinforcement Learning (Cascaded RL) phase. Extensive experiments is rich in long-tail scenarios, and the public nuScenes dataset show that, with only 3B parameters, DriveAgent-RI achieves competitive performance comparable to top closed model systems such as GPT-5 and to human driving proficiency while remaining aedloymmnt-frienddee offering proven path toward building more intelligent autonomous driving systems.
the Drive-Internal dataset, which
INTRODUCTION
The paradigm of end-to-end autonomous driving has been substantially advanced by the advent of Vision-Language Models (VLMs) (Tian al., 2024; Sima al., 2024; Xu et al., 2024; Hu al., 2023b; Jiang et al., 2023). By emulating human- like cognition, VLMs promise to unify perception, reasoning, and planning within a single, cohesive framework, holding the potential for superior generalization and deeper understanding of complex scenarios. In the context of planning, the task can be decomposed into two levels: low-level motion planning and high-level behavioral decision- making (Wang et al., 2025b; Tian et al., 2024). Compared to tasking VLMs with regressing continuous physical trajectories- a task for which they are not inherently optimized- a more promising direction to leverage their strengths in semantic understanding to predict high-level driving intentions (Wang et al., 2022; Paul et al., 2024). Accordingly, the core objective of this work is to endow agents with the capability for fine-grained, high-level behavioral planning.
To achieve this goal with greater transparency, interpretability, and robustness, we turn to the Multimodal Chain-of-Thought (M-Co'T) (Xu al., 2025a; Li et al., 2025a; OpenAl, 2025b; Cheng et al., 2025; Hu et al., 2024). The evolution of M-CoT has given rise to distinct reasoning paradigms: from early Text-based M-CoT (Xu al., 2025a), which reasons over textual descriptions of visual inputs, to the more sophisticated, interleaved Tool-based M-CoT (Hu et al., 2024), which actively
"These authors contributed equally to this work. 1 *Correppnndigg at: hangzhao @ isiniliaiihhaccuu

---

Published as a conference paper at ICLR 2026
Reasoning under M cool
Description The vehicle ahead partially on the road and a motorcycle is also present on the right side of the road. need check the vehicle ahead carefully.
Front View
('viww, indes": ""rom"" "bbox": [205, 190, 230, 2251. "description: : "the vchide shead")
["das__indaa"
The Rol Inspection image confirms the presence of ..., indicating minor collision.
Speed:
12.8 km/h
Reasoning Given the collision and speed, ... Due to the ongoing situation, stopping is prudent until further assessment is possible.
Navigation:
straight through the intersection
Prediction Decelerating twice ensures safety, and then stop to adhere to the traffic accident.
D"Decelerate, Straight", "Decelerate, Straight", "Stop, Straight", "Stop, Straight"]
Figure 1: An illustration of DriveAgent- RI's active perception capability. The agent proactively uses Rol Inspection to clarify an uncertain scene, discovering a minor collision between the vehicles ahead. This active perception corrects its initial assessment, leading safe plan to decelerate and then stop based on direct visual evidence. More visualization is shown in the Appendix A.10.
invokes tools to gather new visual evidence mid-reasoning, enabling a dynamic whine---il--seeing" process. While extensive research has sought to harness M-CoT for autonomous driving tasks (Tian al., 2024; Jiang al., 2025a; Li al., 2025c), these efforts have predominantly remained within the confines of Text-based M-CoT, a paradígm of passive perception. The passive perception struggles with two opposing yet related challenges: (1) cannot seek additional visual information to resolve ambiguity when the default view is insufficient; and (2) providing comprehensive multi-view data forces the model to process redundant visual inputs, increasing computational overhead and distracting it from critical cues. Human driving, by contrast, is an inherently active process of resolving uncertainty. Drivers will check blind spots or look again at a confusing traffic signal, demonstrating the ability to "actively perceive," which is the cornerstone of safe driving. Tool-based M-CoT provides a technical pathway to instantiate this more human-like driving intelligence, breaking the shackles of passive perception. However, leveraging this tool-based active perception for the core task of high-level behavioral planning remains a significant and unexplored research gap.
While this active perception paradigm is powerful, indiscriminately applying tool-based active perception to all driving scenes is computationally inefficient. Human drivers exhibit similar cognitive efficiency, relying on intuition for simple conditions and reserving deliberate, active exploration for complex or uncertain scenarios. Such an observation suggests more intelligent paradigm dynamically adapting its thinking mode: employing efficient Text-based M-CoT for routine cases while deploying the in-depth, Tool-based M-CoT only when the situation's 's complexity demands it. To our knowledge, such an adaptive mechanism that integrates these distinct M-CoT modalities for the autonomous driving planning task remains unexplored.
To bridge the identified gaps, we introduce DriveAgent-RR a novel autonomous driving agent whose design centered on two key innovations: active perception and hyybrid-thinking framework. First, it pioneers active perception for high-level behavioral planning. Leveraging Tool-based M-CoT, DriveAgentRR/ proactively seeks visual evidence resolve uncertainty, grounding its decisions in verifiable perception (See Fig. I). This is enabled by a multi-turn interactive framework centered around a specialized Vision Toolkit, which empowers the agent to acquire diverse visual information on an as-needed basis rather than passively processing redundant multi-view data (Qiao et al., 2025; Hu et al., 2023a; Sun et al., 2025). Second, we introduce a Hybrid- -hinnkin framework to an autonomous driving agent. It innovatively integrates efficient Text-basen M-CoT with in-depth Tool- based M-CoT, allowing the agent to adaptively switch its thinking mode based on scene complexity. We cultivate this capability through a three-stage progressive training strategy. Following an initial supervised fine-tuning (SFT) stage, the agent undergoes a novel Cascaded Reinforcement Learning (Cascaded RL) phase. In this core phase, our proposed algorithm first strengthens each thinking mode individually and then trains the agent to adaptively select the optimal one. On the Internal which features diverse set of long- tail scenarios, and public nuScenes datasets, our 3B DriveAgent-RR achieves performance competitive with top-tier models like GPT-5 (OpenAI, 2025a) and human drivers, while maintaining deployment-frient frienddyy
GRPO (MP-GRPO)
2

---

Published as a conference paper at ICLR 2026
efficiency. Crucially, ablation studies confirm this success is genuinely grounded in visual evidence rather than textual shortcuts, validating our active perception and hybrid-thinking framework as path toward safer, more interpretable autonomous driving.
Our main contributions are summarized as follows:
1. We pioneer active perception framework for high-level driving planning. The agent proactively invokes a vision toolkit to ground decisions in visual evidence, enhancing reasoning and reliability,
2. We introduce hybrid-thinkingg framework that adaptively balances efficient text-only reasoning with robust, tool-augmented visual analysis. This is enabled by novel three-stage progressive training strategy centered Cascaded RL.
DriveAgent-RR achieves both strong performance and deployment-friendle efficiency. With only 3B parameters, DriveAgent-RI proves competitive with top-tier models like GPT-5 and human drivers, while significantly reducing inference latency compared to passive approaches, establishing practical path toward scalable, intelligent autonomous driving.
DRIVEAGENT-RL
2.1 PROBLEM FORMULATION AND HYBRID THINKING
As illustrated Fig. 2. the agent's primary task is to generate a long-horizon driving intention sequence from multimodal inputs. Given initial visual context Lo (from front-view camera) and textual context To (vehicle speed and navigation), the agent outputs an 8-second meta-action sequence A (a1, 02 as, 04), with each action predicted at 2-second interval. Each meta-action a, (St, jr) comprises a velocity token 8, E V,=(Accelerate,, Keep Speed, Decelerate, Stop) and trajectory token jr E V==((Sraight Right Turn, Left Turn].
Text-based M-CoT:
T
A: R2 Ry R2.
<description>.. </description> <reasoning </reasoning- <prediction> </prediction>
Tool-based
Qwen2.5 LM Decoder
Vision Toolkit
To
Jo
Ti
T2
Retrieve View
Vision Encoder
Inspection
Depth Estimation
3D Object
Default
Front View
Front Right View
3D
Figure 2: The Thinki-T architecture of DriveAgent-RI. For simple scenarios (Top), the agent uses direct text- based reasoning (T₁ + A). For complex scenarios (Bottom), it iteratively interleaves thoughts (Tş) with tool calls Vision Toolkit, acquiring new visual evidence (Iş) refine its decision-making. The detailed visulization of this case is shown in Fig. 10, Appendix A.
To achieve this, DriveAgent-RL employs a hybrid-thinking mechanism. It first selects a reasoning <think_tooll token. Both paths adhere to
path by generating either <think_ unified CoT framework: description, reasoning, and prediction, where the description stage involves initial perception of the scene, the reasoning stage conducts logical analysis, and the prediction stage summarizes the analysis to determine the sequence.
Text-based M-CoT (Miex).. For common scenarios, DriveAgent-RI generates the <think <think_cext> text> token, relying entirely on its internal knowledge and the initial input (In, Tr) to make decisions through pure text-based reasoning.
3

---

Published as a conference paper at ICLR 2026
Tool-based M-CoT (Mool.. For complex uncertain scenarios where the initial visual context is insufficient, DriveAgent-RR Rl generates the <think sthink_tool> >> token to engage in active perception, proactively seeking visual evidence to resolve ambiguity and ground its decisions. This process updates its contextual history H& as follows:
H₂ = H + Th + Ix for l < K,
where (Tk: Th) = L.Mecoddr(HH. H and J VisionEncoderr (Fxecute (Tk)). The process begins with initial history Ho To @ Lo. where + denotes the concatenation of token embeddings. In each step k. DriveAgent-RI generates a textual thought Tp and potentially a tool call request To based on its current history Hg. If tool is called, it is executed to obtain new visual information, which is then encoded into an image embedding Iς and integrated into the history. The process terminates upon generating the final action sequence A or reaching the maximum interaction limit K.
Vision Toolkit Design. DriveAgent-RR is integrated with a powerful Vision Toolkit, endowing with the capability of active perception. These tools allow the model to explore the environment demand during its reasoning process to acquire critical information. The toolkit includes: (1) Retrieve View, which fetches clear images from any camera, including historical frames from a 5-second memory pool; (2) Rol Inspection, a "zoom-in" function that crops and magnifies a specified Region of Interest (Rol) from higher-resolution image to provide rich visual details; (3) Depth Estimation, which generates a depth map to provide crucial 3D spatial awareness; (4) 3D Object Detection, an vocablaayyl tool that identifies and localizes objects 3D space. Detailed descriptions and tool call formats are provided in Appendix A.2.
2.2 AUTONOMOUS DRIVING DOMAIN ALIGNMENT
Recent studies (Li et al., 2025c; Chen et al., 2025; Wang et al., 2025a) indicate that general VLMs exhibit shorteut tendency in driving planning: they rely on low-dimensional textual cues and neglect high-dimensional visual input. To first develop visual sensitivity and ensure decisions are grounded in evidence, inspired by Drive-R1 (Li et al., 2025c), we perform autonomous- domain alignment on the base model before high-level behavioral planning training. To this end, we build an autonomous-driving VQA dataset from images collected in real traffic, totaling 530K muestion-answer pairs. Questions in the dataset fall into four main categories: (i) scene description, (ii) recognition of traffic entities, (iii) localization of critical targets, and (iv) traffic commonsense and rules. Using this dataset, we fully fine-tune all parameters of Qwen2.5-V VL-3B (Bai et al., 2025) and obtain DriveAlign-3B, a model highly sensitive to visual evidence in driving scenes. DriveAlign-31 is used the unified initialization for the subsequent three-stage training, laying the visual-alignment foundation for DriveAgent-AR RR'ss active perception and hybrid thinking. See the Appendix A.3 for details of the VQA data construction.
2.3 PROGRESSIVE TRAINING VIA SET AND CASCADED RI.
To cultivate the hybrid-thinking capabilities of DriveAgent-R1, we devise progressive training strategy composed an initial Supervised Tine-Tuning (SFT) stage and a subsequent two-stage Cascaded RL phase shown in Fig. 3. This overall strategy follows "foundation building -> mode strengthening + intelligent selection" paradigm.
2.3.1 STAGE 1: DUAL-MODE SUPERVISED -UNN-T
We begin by cold-starting the model using SFT to endow it with a foundational understanding of the format and semantic boundaries of both thinking modes. To this end, we construct a high- quality dataset via a three-stage automated pipeline. The pipeline first partitions raw data into tool- unnecessary set Dieu and necl-secyssary set D Drext comprises simple scenarios where Qwen2.5- VL-3B already achieves high accuracy without tools, indicating no need for active perception. Conversely, Droo includes complex cases where the powerful S0wen2.5-VL-725 shows a performance gain only when using tools, establishing criterion for tool necessity. Next, B generates mode-specific CoT annotations, aligning with recent findings (Zhou et al., 2025) that demonstrate the efficacy of constrained prompted generation for high-quality annotation. To ensure data quality, these annotations undergo a rigorous validation and filtering process: a separate judge model scores the reasoning coherence, and samples falling below quality threshold are automatically
4

---

Published as a conference paper at ICLR 2026
Stage-1 Dual-Mode Supervised Fine- Tuning (DM-SFT)
Stage-2 Forced Contrastive Mode RL (FCM-RL)
Stage-3 Adaptive Mode Selection RL, (AMS-RL)
Widao Toditt
Caid Start
Pepplrre
Multimodal Input
GoT Data
w/ mode limitation
w/o
Deek
D
SFT
Response
Owen22.5 LM Decoder
Response
Responce
Response
Response
Vision
MP-GRPO
GRPO
Trainable
Frozen
Accuracy
Format
Tool Usage
Rleard
Reward
Reward
Figure 3: The progressive stage training strategy for DriveAgent-A RI. The process begins with (1) DM-SFT to establish a foundational understanding of both thinking modes. This is followed by core Cascaded RL phase, where (2) FCM-RL strengthens each mode independently, and (3) AMS-RL trains the agent to adaptively select the optimal mode.
regenerated. Finally, the data is passed through rule-based cleaning to create a balanced, high-quality dataset for SFT. A complete description of this pipeline is provided in Appendix A.4.
2.3.2 STAGE 2: FORCED CONTRASTIVE MODE REINFORCEMENT LEARNING
The first stage of our Cascaded RL phase aims deepen the agent's 's independent performance in each mode. We adopt Group Relative Policy Optimization (GRPO) (Guo et al., 2025) as our foundational algorithm, which has proven effective multimodal reasoning and autonomous driving (Jiang et al., 2025a; Liu et al., 2025; Su et al., 2025; Zheng et al., 2025). Unlike PPO, GRPO eliminates the need for a critic model by estimating the baseline from a group of outputs. Specifically, for each query q. GRPO samples group of outputs {01, 02, oG) from the old policy To.. and optimizes the policy ne by maximizing the following objective:
(G)PO(() Egp (o:)
(min (w;A,, clip(w,, = +) -DD
G
(o (g)
where wi meaaa [o.]g] is the importance sampling ratio, and 33 are hyper-parameters, and the advantage A, is computed by normalizing the rewards within the group. Building upon this standard formulation, we propose Mode-Partitionns GRPO (MP-GRPO) to prevent the agent from developing a bias against an initially weaker thinking mode. As illustrated in Fig. 3, the core mechanism of MP-GRPO is enforce balanced exploration across both thinking modes. For each input q from Dexplore we compel the agent to generate two mode-specific sets of G/2 responses by forcing the corresponding mode token (<think_text) or <think_tools) This creates full group of G responses:
O (ott))C/ G/2
Ji=1 (o G/2 Jj=1"
where each individual response, denoted by o, is sampled from the old policy Tew.. Crucially, all G responses are then aggregated into this unified set O(4) for reward normalization and advantage calculation. This design creates a multi-dimensional contrastive learning signal: it facilitates not only intra-mode comparison between different response trajectories but also moder-mode comparison. This multi-faceted contrast allows the agent to simultaneously enhance its specific capabilities in both modes and gain an initial understanding of the suitability of each mode for different scenarios, laying critical foundation for the final stage of adaptive selection.
Reward Design. The agent's learning is guided by an outcome-driven reward function that evaluates the entire reasoning trajectory. This reward is composite of two key components: accuracy reward (Rase) and a format consistency reward (Rmm), formulated as follows:
R Race
R
where the accuracy reward Race is a weighted Levenshtein distance against the ground-truth sequence, with ausition-action weights to counteract action imbalance (see Appendix A.5 for details). The
5

---

Published as a conference paper at ICLR 2026
format consistency reward Rim penalizes structural errors and incorrect mode usage, such invoking tools in the Mex mode.
2.3.3 STAGE 3: ADAPTIVE MODE SELECTION REINFORCEMENT LEARNING
The final stage of Cascaded RI. cultivates true hybrid thinking by training the agent to intelligently select the optimal mode. Using the native GRPO algorithm (Guo et al., 2025), the agent learns to generate the initial mode-selection token itself and commit to the corresponding reasoning path.
Reward Design. To guide this learning process, the reward function is augmented with conditional tool-usage reward, Riool, which encourages the effective and judicious use of tools only when the Mboe mode is chosen. The total reward is defined as:
R = Rase + Rim + I(mode = Musa)) Roool
where the definitions of Race and Rimt remain consistent with Stage 2. The core of Rooo contrastive mechanism that promotes impactful tool use. Within each group of G generated responses, we establish baseline by calculating the average accuracy of all text-only trajectories (Accea). A tool-based trajectory is rewarded only if its own accuracy (Race) surpasses this baseline by a margin, i.e., Rwol X (Race Accest margin). This design explicitly penalizes superftuous tool calls, incentivizing the agent to invoke active perception only when it provides clear performance advantage over pure text-based reasoning. detailed formulation is provided in the Appendix A.5.
EXPERIMENTS
3.1 SETUP
3.1.1 DATASET
Automated Label Generation for Meta-actions. We produce labels for meta-action sequences with rule-first, VLM-corrected pipeline. Starting from vehicle trajectory points, we use a 3-second sliding window to map each segment to a predefined vocabulary of speed and trajectory actions, employing thresholds iteratively optimized via dual VLM-human feedback. This yields four-step discrete meta-action action sequence for the next 8 seconds 2-second intervals. To correct small number of ambiguous or boundary cases, we then feed the resulting sequence and its BEV-map visualization into GPT-4.1 for refinement. Full rules and optimization details are in the Appendix A.7.
Training Data. All training data drawn from our Drive-Internal dataset, which comprises 35K video clips specifically targeting long-tail and complex scenarios. This dataset features over 40 distinct driving situations, including challenging events like road works and animal crossings. To leverage its rich diversity, one or two key frames are sampled from each clip, focusing on the moments surrounding critical events. For Stage (DM-SFT),
generate 4K high-quality CoT data using the pipeline from Sec. 2.3.1, splitting it evenly into the tool-necessary set Dioot and the cool-unnccessary set Diext (2K each). For the Cascaded RL phase, we construct an exploratory set Dexptore by sampling 15K instances from the data not assigned to Droot or Dteat, ensuring the sampling maintains a balanced distribution of actions at each timestep the sequence.
Test Data. We randomly sample 1.5K instances from Inter-llternal the in-distribution test set Drive-Internalges- To assess generalization, we also sample 2K instances from the nuScenes validation set as nuScenesuaa for cross-dttseet testing. All nuScenes samples taken from the middle of each clip to ensure sufficient history and future trajectories. Meta-action labels on nuScenes are generated by the same pipeline.
3.1.2 EVALUATION METRICS
Accuracy (Acc). We evaluate planning precision using First-Frame and Sequence Average Joint Accuracy. A prediction is considered correct only if the composite meta-action (both speed and trajectory) exactly matches the ground truth. To encourage safer behavior, we also employ Relaxed Speed Matching mechanism, awarding partial scores (0.5 and 0.2) for predictions that are one and two levels safer than the ground truth, respectively.

---

Published as a conference paper at ICLR 2026
Table 1: Results on the Drive-Internaleg and nuSceneshest All models are evaluated using one-shot prompting under two conditions: without tools and with access to the visual toolkit. To ensure consistent, structured reasoning format, closed- source models were queried via their official APIs in "no-thinking" mode. Parentheses show the absolute gain from using tools.
Drive-nttermllai
nuSeenesus
Model
First-Frame Joint Acc. (96
Seq. Avg. Joint Acc. (%)
First-Frame Joint Acc. (%)
Seq. Avg. Joint Ace, (9)
w/o Tools w/ Tools
w/o Tools
w/ Tools
w/o Tools
Tools
w/o Tools
w/ Tools
Human
49.59
49.29
50.48
48.24
Ow-n2.5-VL-38
23.64 (-0.42)
(-2.35)
(-2.7()
(-1.90)
Qwen2.5-Vi
27.58
24.01 (-3.57)
29.84
24.98 (-4, 86)
33.11
32.84 (-0.27)
27.60
28.82 (+1
Qwen2.5.VD VLL
32.76
32.97 (+0.21)
38.80
39.61 (+0.81
43.26
43.87 (+0.61)
39.13
40.47
Doubao-Secd-11
34.92
38.98 38.98
40.22
41.37(+1.15)
45.63
46,29 (+0.66)
41.91
43.10(+1.19)
Gemini-2.5-Flash
41.83
43.38 (5))
42.14
43.42 (+1.28))
45.46
46.60 14)
42.69
44.07 (+1.8))
GPT4.1
39.99
(8.38 19)
42.14
(+
46.84
48.25 (+1.41)
43.63
44.72(+1.09)
GPT-5
56.30
56.48 (+0.18)
47.19
47.97 (+0.78)
48.75
49.11 (+0.36)
44.85
45.14 (+0.29)
DriveAgent-R
45.27
51.34 (+-6077
43:29
45 (42.2()))13)
52.58
52,96
44.43
(+2.6))
Mode Selection Accuracy (MSA). To quantify DriveAgent-RI's adaptive mode selection, we introduce the Mode Selection Accuracy (MSA), inspired by Jiang et al. (2025b). It measures the alignment between the agent's ss adaptively chosen mode madaptive and the optimal mode m,. The optimal mode m is defined as whichever mode (Moooo MMex) achieves higher accuracy for given sample qj. MSA is then calculated as:
1 MSA
Σ amaatpiivve (9:) m²?)
3.1.3 IMPLEMENTATION DETAILS
All training conducted H20 GPUs For all multimodal inputs in our experiments, the maximum number of pixels per image is set to 259,20 200. Unless specified, only the front-view camera image serves as the default visual input. During the Cascaded RL phase, we the sampling temperature to 1.0, limit the maximum number of tool calls to 3, and group size to 4 per sample. For all evaluations, the sampling temperature is set to 0.7.
3.2
MAIN RESULTS
3.2.1
COMPARISON WITH SOTA VLMS AND HUMAN DRIVERS
To comprehensively evaluate DriveAgent-RI, we benehmark it against a suite of state-of-the-art closed-source VLMs, the open-source Qwen2.5-Vi series, and a human performance baseline on both the Drive-Internalest and nuSceneSiet- More details are shown in Appendix A.8. The main results summarized in Table 1, leading to two key observations.
DriveAgent-RI achieves performance comparable to GPT-5 and human drivers with only 3B parameters. Despite its 3B parameter size, DriveAgent-Rl RI shows remarkable performance and the most effective tool use, achieving a +6.0% accuracy gain on Drive- Internlneernaees.. Its performance is competitive with both GPT-5 and human drivers. Notably, the nuScenesies. DriveAgent-Rl surpasses GPT-5 in sequence accuracy (47. 10% vs. 45.14%) and approaches human-level proficiency.
Visual tool is double-edged sword. While VLMs like GPT-4.1 and Gemini-2.5-PPP improved with tool access, the Qwen2.5-VL-33 and -7B models showed consistent performance degradation. This indicates that effective tool use a non-trivial skill, requiring either high level of base model capability or targeted, specialized training to master.
3.2.2 PERFORMENCE ON DRIVEBENCH
To further validate DriveAgent-RR R1 against existing state-of-the-art -LMe-based driving agents, we conducted evaluation on DriveBench (Xie et al., 2025), benchmark designed for high-level driving tasks. We compare our model with DriveLM (Sima et al., 2024) and Dolphins (Ma et al., 2024).
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance Comparison on DriveBench. We compare DriveAgent-RI with representative VLM-based driving agents.
Method
Perception
Prediction
Planning
Behavior
DriveLM
16.85
44.33
68.71
42.78
Dolphins
32.66
18.81
DriveAgent-RI (Ours)
34.07
32.85
61.89
43.69
Table 3:
Open-loop planning performance on nuScenes validation set. We report the L2 Displace- ment Error (DE) and Collision Rate (CR) at different time horizons. Bold and underlined denote the best and second- bectnd-best results, respectively.
ADE (m) J.
Collision Rate (%)
Model
avg
avg
UniAD
0.44
0.67
0.96
0.69
0.04
0.08
0.23
0.12
VAD-Base
0.34
0.37
0.10
0.14
FMMA
0.14
0.29
0.54
0.32
DriveVLM-DD M-uual
0.15
0.29
0.48
0.31
0.05
0.08
0.17
0.1
OmniDrive
0.14
0.29
0.55
0.33
0.00
0.13
0.78
0.3
OpenDriveVLA (3B)
0.14
0.30
0.55
0.33
0.02
0,07
0.22
0.1
DriveAgent-RI (Ours)
0.24
0.28
0.01 0.12
0.14
DriveAgent-RI achieves superior scene understanding and decision-making. As shown in Table 2., DriveAgent-Ri achieves a Perception score of 34.07, doubling DriveLM's 16.85. This substantial margin confirms that proactive tool invocation captures critical visual details often missed by passive paradigms. Moreover, our top-ranking Behavior score (43.69) demonstrates that the DriveAgent-RI effectively translates this rich visual evidence into safe, correct decision-making
3.3 PERFORMENCE OF MOTION PLANING
To validate whether the high-level reasoning of DriveAgent-RI translates into effective low-level control, extended the architecture with lightweight MLP motion planning head to regress physical trajectories on the nuScenes dataset. The motion head acts as an "executor" for the frozen DriveAgent-R1. It uses simple MIP taking three inputs: meta-action sequence (one-hot), visual tokens, and ego-status, predicting 3s future waypoints at 2Hz
High-Level Reasoning Enables Precise and Safe Planning. On the nuScenes validation set, DriveAgent-RI achieves an average ADE of 0.28m, outperforming strong baselines like DLiveVLM- Dual (0.31m) while maintaining comparable collision rate. This confirms that the reasoning-driven meta-actions are the decisive factor enabling effective low-level control.
Table 4: Analysis of Foundational Capabili- ties. Overall scores on domain-specific and general VLM benchmarks. Numbers in blue denote the improvement over Qwen2.5-VL-3B Detail results are Appendix 1.9.1.
Table 5: Impact of Domain Alignment High- Level Behavioral Planning Task. We report se- quence accuracy on Drive-Internahest- The relative performance drop (%) upon removing images is shown in parentheses.
Seq. Joint Acc. (%)
DriveAligo-VOA...
Benchmarks
Model
image
wio image (rel drop)
Quen2.5.V1 Qwea2.5-VL-7B
72.9
64.4
Qwwn2,5-VL-3B
DriveAgent-Ke Varians
43.56
38.79 (-1.0%)
DriveAlign-38
DriveAgent-RI
45.42
(-
DriveAlign-31
84.6
66.4
3.4 ABLATION ON DOMAIN ALIGNMENT
To validate our domain alignment strategy, we analyze its impact on both the base model's capabilities and the downstream high-level behavioral planning task.
Domain Alignment Enhances Driving-Specific and General Capabilities. As shown in Table 4, domain alignment significantly boosts performance our driving-specife VQA benchmark (+11.7) and also improves scores across a suite general VLM benchmarks (+2.0). This confirms our strategy successfully injects domain knowledge while strengthening general capabilities.
8

---

Published as a conference paper at ICLR 2026
Domain Alignment Boosts Planning Performance and Mitigates Visual Neglect. As shown in Table 5, the agent built on our aligned model achieves higher accuracy (+1.86 Sequence Avg-Joint Acc). Crucially, its performance drops more significantly when images are removed (-15.8% vs. -1 -11.0% 1.0%), confirming its gains are grounded in visual evidence rather than textual shortcuts, thereby mitigating the visual neglect problem.
Table 6: Ablation on the progressive training strategy on the Drive-Internalesr We evaluate different combinations of our three training stages. To ensure fair comparison, also test variants where single RL stage is trained for two epochs, matching the total RL epochs of Rri
DmvrAgg-- 30 -EN0-FT
Triining Stage
Seg Aeg. Joint Arc. (1)
DMA-STT
FEM-RL
AMS-HIL
Must
Muo
Mukee
Figure 4: Progressive training gains on Drive-Internalust Accuracy in Madantive mode and MSA improve with each training stage.
Vattenn Only:
Tepech
.0.61
40:51
15.00
Vaiien (FCM)
Eppcc
64.30
44.64
56,54
Vunani-3 dell 52a
Tepech
4330
-1.30
44:9
57.32
Lepech
Ppoch
Varieei (wAMS5
44:11
51.51
Drtrekgemm-t (PCM-
Tepech
Lpsch
Tepech
44.97
45.42
68.32
3.5 ABLATION ON PROGRESSIVE TRAINING STRATEGY
To validate our progressive training strategy, we ablate different stage combinations in forced-text (Miex). forced-tool (Mool)), and adaptive-selection (Madaptiv)) modes on both the Drive-Internales and nuScenesest datasets to confirm the method's 's effectiveness (full nuScenesjet results are in the Appendix A.9.2). Our analysis yields two key findings.
Progressive Training Yields Stable Performance Gains. As illustrated in Fig. 4, our strategy achieves consistent performance improvements, resulting steady increase in both accuracy and MSA.
The Cascaded RL Strategy Surpasses Single-Strategy Stacking due to Functional Complementarity Table 6 reveals that our cascaded RI. strategy is synergistic, outperforming single-strategy stacking with the same total training epochs (Variant-3/5). This superiority stems from the complementary roles of its two stages: FCM-RL sharpens execution proficiency (achieving higher forced-mode accuracies of 44.30% vs. 43.78% in Mioal and 42.20% vs. 41.64% in Miext. while AMS-RL hones selection intelligence (yielding a higher MSA of 57.55% vs. 56.64%). Our cascaded approach successfully integrates these distinct strengths for superior overall performance.
Table 7: Ablation Active Passive Per- ception. We compare DriveAgent-RI with passive baselines on Drive-Internal Rela- tive drop (%) without images in parentheses.
Table 8: Performance and Efficiency Analysis. We compare inference latency and average out- put tokens for different perception and reasoning strategies.
Model
Seq. Aug, Joint Acc. (%)
Model/Mode
Latency (s)
Avg. Token
w/ image
w/o image (rel. drop)
Passive-SV
8.50
210.77
PassieeFV
42:70
30.64 (-7.2%)
Passive-SV SV
43.10
39.01 (-9.5%)
DriveAgent--R RI (Mooa)
7.91
314.45
DrivAgemmtR RI (Reuieve View Only)
44.39
38. 78 (-12.6%)
DriveAgent-ER -1 (Madaptiv)
6.74
265.57
DriveAgent KI (Full Toolkit)
38 (-15.8%)
3.6 ABLATION ON ACTIVE VERSUS PASSIVE PERCEPTION
To isolate the benefits of active perception, we compare DriveAgent-RI RI against two passive perception baselines: Passive-FV (front-view) and Passive-SV (surrounding views), of which were trained with the same SFT and RL pipeline for a fair comparison. For controlled comparison against Passive-SV, we evaluate a restricted variant, DriveAgent-RI (Retrieve View Only), which limited to using only the "Retrieve View" tool. As shown in Table 7, our findings twofold.
Active Selection is Superior to Passive Input. Active perception, as demonstrated by DriveAgent- R! (Retrieve View Only). shows dual advantage. First, it overeomes information deficits (outperforming Passive-FV by +1 +1.69 points). Second, mitigates the redundancy of multiple fixed views. Despite sharing identical information ceiling (the same pool of six views), DriveA outperforms

---

Published as a conference paper at ICLR 2026
the Passive-SV SV by +1.29 points. This confirms that actively retrieving task-relevant information on demand is more effective than passively processing a fixed set of inputs.
Active Perception Deepens Visual Reliance. A clear trend emerges: the more active and capable the agent's perception, the more its performance drops when visual inputs are removed. This confirms our framework successfully fosters genuine, vision-driven decision-makee model away from textual shortcuts. This heightened dependence on vision, however, also underscores the criticality of correctly interpreting the acquired evidence. In complex scenarios, misinterpreting new visual information can lead to errors, a challenge we explore in Appendix A.II.
process, steering the
3.7 EFFICIENCY ANALYSIS
To assess the computational efficiency of our framework, we evaluate inference latency and average output token length. Latency is measured on a single H20 GPU with a batch size of using the vLLM (K wwon et al., 2023) inference framework, averaged over 100 samples and repeated across 5 runs. The results, summarized in Table 8, highlight two key advantages of our design.
Active Perception is More Efficient than Passive Input. By selectively retrieving visual data only when needed, DriveAgent-R R/ avoids the computational overhead of processing the redundant images required by Passive-SV. This targeted approach reduces inference latency by over 20%.
Hybrid Thinking Balances Performance and Cost. The advantage of hybrid thinking clear when comparing the (Madative and Mianl modes. DriveAgemt-R under Mataptive reduces both inference latency (from 7.91s to 6.7 6.74s) and output tokens (from 314.45 to 265.57).
4 RELATED WORK
VLMs for Autonomous Driving. Recent advancements have seen Vision-Language Models (VLMs) applied to autonomous driving. unifying perception, reasoning, and planning (Sima et al., 2024; Tian et al., 2024; Mao et al., 2023). Research has primarily followed two paths: enhancing structured scene understanding (Sima et al., 2024; Qian et al., 2025; Nie et al., 2024; Shao et al., 2024) and translating reasoning into actionable behaviors, from high-level decisions to low-level motion planning (Jiang et al., 2025a; Li et al., 2025c; Tian et al., 2024). However, existing VLM-based planners predominantly operate under passive perception paradigm, reasoning over fixed set of inputs. DriveAgent-R breaks from this by introducing an active perception framework, where the agent proactively invokes visual toolkit to ground its decisions in visual evidence.
Multimodal Reasoning with Chain-of-Thoughtt Chain-of- Chain-of-Thought (CoT) has significantly im- proved the reasoning capabilities of LLMs (Wei et al., 2022: Chu al., 2023; Wang & Zhou, 2024) and has been extended to the multimodal domain (M-CoT) (Xu et al., 2025a; He et al., 2024; Wang al., 2024; Mondal al., 2024: Liu al., 2025). The evolution of M-CoT has progressed from Text-based methods, which reason over static textual descriptions (Xu et al., 2025a; Zhao et al., 2025;
Li al., 2025b), to more dynamic, interleaved approaches. Among these, Tool-based M-CoT is promising direction, enabling models to actively seek new information during reasoning (Qian et al., 2025; OpenAI, 2025b; Hu al., 2024; Su al., 2025; Zheng et al., 2025). Our work advances this frontier by introducing novel hybrid- paradigm that adaptively integrates efficient Text-based M-CoT with robust depth according to the scene's complexity.
M-CoT, allowing the agent to modulate its reasoning
CONCLUSION AND LIMITATIONS
We introduced DriveAgent-RU an autonomous driving agent that pioneers active perception and hybrid thinking for high-level behavioral planning. Our 3B model achieves performance competitive with top-tier systems like GPT-5 and human drivers while being deployment-frienddly. However, a key limitation is the agent's 's over-reliance on new visual evidence from its tools, as detailed in our failure case analysis (Appendix 1.1 Future work will directly address this challenge through two complementary directions: (1) strengthening the agent's foundational understanding of complex road topologies and traffic rules, and (2) cultivating a more nuanced reasoning process capable of resolving informational conflicts.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was sponsored by the Beijing Nova Program.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502. 1355002.19223 13923, 2025.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we on the right way for evaluating large vision-language models? In NeurIPS, 2024.
Xuesong Chen, Linjiang Huang, Tao Ma, Rongyao Fang, Shaoshuai Shi, and Hongsheng Li. Solve: Synergy of vissonn and end-to-end networks for autonomous driving. In CVPR, 2025.
Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang. Alex Jin- peng Wang. Zhi Chen, Wanxiang Che, et al. Visual thoughts: A unified perspective of understanding multimodal chain-of of-in-.f- arXiv preprint arXiv:2505. 2025.
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu. Navigate through enigmatic labyrinth survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint arXiv:2309. 0tte22399./542.... 15402, 2023.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. VImevalkit: An open-source toolkit for evaluating large modaityy models. In ACM MM, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. Multi-modal latent space learning for chain-of- thought reasoning in language models. In AAAI, 2024.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-orientee autonomous driving. CVPR, 2023a.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, al. Planning-oriented autonomous driving. In CVPR, 2023b.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. In NeurIPS, 2024.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajic Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In ICCV, 2023.
Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unleashing the power of vims in autonomous driving via reinforcement learning and reasoning, arXiv preprint arXiv:2503.07608, 2025a.
Lingjie Jiang, Xun Wu, Shaohan Huang, Qingxiu Dong, Zewen Chi, Li Dong, Xingxing Zhang, Tengchao Lv, Lei Cui, and Furu Wei. Think only when you need with large hybrid models, 2025b. URL https: /arxiv.. http://awaxvl/aaaaaaa//a
25.5. 14631.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016.

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In SOSP, 2023.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Dong, Ivan Vulié, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-o-ttooggge arXiv preprint wrXiv:2501.07444 2025a.
Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang. Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. Llava-st: A multimodal large language model for fine- nre-graaned spatial-temporal understanding, In CVPR, 2025b.
Yue Li, Meng Tian, Dechang Zhu, Jiangtong Zhu, Zhenyu Lin, Zhiwei Xiong, and Xinhai Zhao. Drive-rl: Bridging reasoning and planning in vlms for autonomous driving with reinforcement learning, 2025c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024a.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr large multimodal models. Science China Information Sciences, 2024b.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025.
Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving, 2024.
Jiageng Mao, Yuxi Qian, Junjie Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.014/3. 2023.
Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. Kam-cot: Knowledge augmented multimodal chain-of-thouu oftthughsss reasoning. AAAI, 2024.
Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. In ECCV, 2024.
OpenAl. GPT-5, Technical report, OpenAL, 2025a. Accessed: 2025-9-22.
OpenAL. Introducing 03 and 04 mini https: //tp///pp..a/////////e///e con/ www/indext index/ introducing-on -4 4000d//4-mii...... 2025b. Accessed: 2025-07-24.
Pranjal Paul, Anant Garg, Tushar Choudhary, Arun Kumar Singh, and K Madhava Krishna. Lego- drive: language-enhanced goal-oriented closed-looy end-to-end autonomous driving. In IROS, 2024.
Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Meng- meng Yang, Zheng Fu, Jinyu Miao, et al. Agentthink: A unified framework for tool- augmendd thain-of-thought reasoning in vision-language models for autonomous driving. arXiv preprint arXiv:2505. 2025.
Zhijie Qiao, Haowei Li, Zhong Cao, and Henry Liu. Lightemma: Lightweight end-to-end multimodal model for autonomous driving. arXiv preprint arXi::2505.00284 2025.
Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven L Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, 2024.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph vi- sual question answering. In ECCV, 2024.
12

---

Published as a conference paper at ICLR 2026
Alex Su, Haozhe Wang, Weiming Ren, Fangzhen Lin, and Wenhu Chen. Pixel reasoner: In- centivizing pixel-space reasoning with curiosity driven reinforcement learning, arXiv preprint arXiv:2505..1566, 2025.
Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang. Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. In ICRA, 2025.
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Zhiyong Zhao, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevim: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402. 12289, 2024.
Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. T-sciq: Teaching multimodal chain-of-thougg reasoning via large language model signals for science question answering. In AAAI, 2024.
Letian Wang, Yeping Hu, Liting Sun, Wei Zhan, Masayoshi Tomizuka, and Changliu Liu. Transferable and adaptable driving behavior prediction. arXiv preprint arXiv:220..0514p) 2022.
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M. Alvarez. OmniDrive: A holistic vision-language dataset for autonomous driving with counterfactual reasoning. In CVPR, 2025a.
Xuezhi Wang and Denny Zhou. Chain-of-thouge reasoning without prompting. NeurIPS, 2024.
Yuping Wang. Shuo Xing, Cui Can, Renjie Li, Hongyuan Hua, Kexin Tian, Zhaobin Mo, Xiangbo Gao, Keshu Wu, Sulong Zhou, al. Generative for autonomous driving: Frontiers and opportunities, arXiv preprint vrXiv:2505.08833 2025b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thouglt prompting clicits reasoning in large language models. NeurIPS, 2022.
Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vims ready for autonomous driving? an empirical study from the reliability, data and metric perspectives. In ICCV, 2025.
Guowei Xu, Peng Jin, Ziang Wu, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step. ICCV, 2025a.
Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M. Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vasion-language model supervision. CoRL, 2025b.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan- Yee K. Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. NeurlPS, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu. Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Hanxue Zhang, Haoran Jiang, Qingsong Yao. Yanan Sun, Renrui Zhang, Hao Zhao, Hongyang Li, Hongzi Zhu, and Zetong Yang. Detect anything 3d in the wild. In ICCV, 2025.
13

---

Published as a conference paper at ICLR 2026
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang. Qingsong Wen, Zhang Zhang, al. Mme-realworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that are difficult for humans? In ICLR, 2024.
Kesen Zhao, Beier Zhu, Qianru Sun, and Hanwang Zhang. Unsupervised visual chain-of-thought reasoning via preference optimization. arXiv preprini arXiv:2504, 18397, 2025.
Ziwei Zheng, Michael Yang. Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang. Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint 1442:2 2025.
Zewei Zhou, Tianhui Cai, Seth Z. Zhao, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. Autovla: A vision-languagg-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. In NeurlPS, 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 USAGE OF LLM
Large Language Models were utilized as a writing aid in the preparation of this manuscript. Their was strictly limited improving the clarity and readability of the text through sentence polishing and grammar correction. Additionally, we consulted LLMs for brainstorming potential names for concepts and components, such as Madaptive- DriveAlign- and Drive-Internal The core methodology, experimental results, and scientific insights presented in this paper are the original work of the authors.
A.2 VISION TOOLKIT: DETAILED DESCRIPTIONS AND CALL FORMATS
Retrieve View. This tool allows the agent to request the image from any specific viewpoint. Its design is motivated by human drivers who consciously check specific mirrors windows to get a clear view of the road conditions before critical maneuvers like lane changes or turn.. More importantly, this tool incorporates Historical Memory Pool, which caches images from all viewpoints over the past 5 seconds, sampled at IHz. This enables the model to intelligently look back at historical frames assess changes traffic light status or the movement trends of dynamic objects, thus avoiding the high computational cost and latency associated with processing redundant information from full video sequences.
<tool_cale
<tool_ <tool_name>Rettee name>>etrrivee View</tccl_np __m______ee_>_________ <params> ("frame_index", "-1s", "ivee_index"" "front_left") </apleeft"///aaaaaaaaaaaa </tool_call>
Listing 1: Call Format of Retrieve View
Rol Inspection. This tool empowers the agent with "zoom-in" capability for meticulous examina- tion of critical areas. Based on selected view, the agent can request to retrieve a higher-resolutio version of the image and then crop and magnify a specific Region of Interest (Rol) by providing the absolute pixel coordinates of its bounding box. This is not merely a passive utility; it is direct manifestation of the model's Inherent Localization Ability. This allows for ane-grained analysis and focused viewing of any region the agent deems important, which is vital for confirming details such as the status of distant traffic lights the text on traffic signs.
<tool <tool_call> call>>
<tool name>RoI Inspection</tocd name>
<params> ("view_index": "front_left" "bbox": [x_min, y_min, x_max, y_max], "description"; "the traffic 1ights"<</paran
</tool_call>
Listing 2: Call Format of Rol Inspection
Depth Estimation. Accurately perceiving the 3D spatial relationship between the ego-vehicle and other traffic participants (c.g., vehicles, pedestrians) is fundamental to safe driving. This tool leverages the state-of-the-ar monocular depth estimation algorithm, Depth Anything V2 (Yang et al., 2024), generate depth map for specified view. The resulting depth map provides rich geometric information, enabling the model to intuitively grasp the relative distances and spatial layout of objects, well the overall road structure, which is crucial for path planning and obstacle avoidance.
Sccol_calll <tool name>Depth Estimations/ted name> <params>| | "wveev inddx"nd "Front" 1</ortmm//armm""" Retool_calll
Listing 3: Call Format of Depth Estimation
15

---

Published as a conference paper at ICLR 2026
3D Object Detection. To comprehensively understand the dynamic environment, the agent must identify key objects and ascertain their precise 3D spatial information (position, size, and orientation). We integrate advanced open-vocabulary, single-monoulle 3D object detection tool that works with arbitrary camera views (Zhang et al., 2025). This tool not only detects a predefined set of common objects like cars, pedestrians, and traffic signs but its aopen-vocabllary nature also allows the model to dynamically specify new objects for detection based on the current scene's needs, demonstrating remarkable flexibility and adaptability. The output from this tool provides indispensable input for subsequent trajectory prediction and risk assessment.
<tool_call>
<tool_name>> Object Detecticns/tool _amo>>>mm>>>>> <params> prriew_index: "front", "object_text"; "barrier")</parant </tool_cal1>
Listing 4: Call Format of 3D Object Detection
A.3 DATASET CONSTRUCTION
A.3.1 DRIVEALIGN- VQA DATASET CONSTRUCTION
To build strong visual foundation for DriveAgent-RI, constructed the Drive AQA dataset, large-scale, high-quality collection of question- pairs specific to autonomous driving. The con- struction followed a systematic generation-ande and-vadddaiiinn pipeline, ensuring data diversity, relevance, and accuracy. The key stages are outlined below:
1. Image Collection and Diversification. We first collected million frames from real-world driving scenarios. To ensure a wide variety of scenes, we converted these images into CLIP embeddings, performed feature-based clustering, and then sampled a fixed number of images from each cluster. This process guarantees a diverse representation road types, weather conditions, and traffic densities.
2. QA Pair Generation. We employed the powerful Qwen2.5- VLLL2B model to generate question- answer (QA) pairs from the curated images. The generation process used mixed strategy: 70% of questions were derived from predefined templates covering key driving-related topics, while the remaining 30% were generated freely to capture more nuanced scene details (see the prompt below). The generation randomly utilized either single front-view image all six camera views to encourage robust, multi-perspective understanding.
Quality Validation and Filtering. To ensure the highest quality, we used Doubao-Seed 1.6 as an automated "judge" model to validate every generated QA pair. The judge model scored each pair on dimensions such as question relevance, answer correctness, and grounding in visual evidence. Only pairs that received perfect score were retained for the next stage.
4. Diversity Enhancement. In the final step, we utilized Doubao-Seed-LC. again to rewrite the validated questions. This step was crucial for increasing linguistic diversity and preventing the model from overfitting to specific phrasings in the templates. During this phase, we also filtered out overly simple or trivial QA pairs to maintain a high level of complexity in the final dataset.
This rigorous pipeline ultimately yielded the DriveAlign VQA dataset, comprising 530K high-quality question-answer pairs that effectively develop driving-specific visual sensitivity in our base model.
16

---

Published as a conference paper at ICLR 2026
Prompt for Free-Form VQA Generation
am preparing high-quality visual question answering (VQA) training data to pretrain a Qwen2.5-VL-3B vision-language model (VLM) for autonomous driving scene understanding. For each input front-view driving scene image provide, please strictly design four diverse and specific question-answa pairs based only on the visible content in the image. Your goal is to generate VQA data that focuses on the most crucial information for autonomous driving decision-making.
Do NOT create any questions or answers about objects, events, or details that are NOT present in the given image.
The questions should cover, as much as possible, the following critical aspects relevant to driving:
Lane lines and road boundaries
Presence, type, and status of traffic signs and lights
Positions and number of vehicles, pedestrians, or cyclists
Obstacles unusual road conditions
Road directions or upcoming maneuvers For each image:
Design 4 questions, each addressing different key element visible in the scene.
Provide an accurate and unambiguous answer to each question, strictly based on the image content.
Do not invent information. Only output what can be directly observed.
Present your output as follows:
Q1:
<question
A1: <answer 1>
Q2: <question
A2: <answer 2>
Q3: <question
A3: <answer 3>
Q4: <question 4>
A4: <answer 4>
Please begin when input the first image.
A.3.2 CONSTRUCTION PIPELINE OF DRIVE-INTERNAL DATASET
The Drive-Internal dataset was curated from massive database of real-world driving logs through a rigorous pipeline designed to capture high-value, complex driving situations. The construction process consists of three key stages:
1. Long-tail Instance Retrieval. We formulated a fine-gratd fine-grained taxonomy to address long-tail distributions, targeting corner cases such structurally abnormal vehicles, debris, and wildlife. We utilized a CLIP to perform open-vocabulary retrieval from the raw logs. Candidates were subsequently refined through human inspection to ensure high relevance to the target long-tail categories.
Critical Scenario Extraction. Complementing object-centric mining, we extracted challenging scenarios characterized by dynamic environmental interactions. By computing the variance in driving maneuvers, we isolated clips where the ego-vehicle undergoes significant state changes, thereby capturing moments that test the robustness of planning algorithms.
3. Recision-Centric Frame Sampling. To support planning tasks, keyframe selection is tailored to the scenario type. For maneuvers involving speed or directional changes, we select frames 0.5s-1.0s prior to the event. ensuring the model receives input within realistic reaction threshold. In steady-state driving scenarios, a frame maximizing contextual information chosen as the representative sample.
17

---

Published as a conference paper at ICLR 2026
A.4 DM-SFT COT DATA CONSTRUCTION PIPELINE
To provide high-quality data for the initial SFT phase, we developed an automated data construction pipeline structured as a three-stage workflow (see Fig. 5). This pipeline systematically processes raw datasets-which exclusively contain multimodal inputs and ground truth meta-actions- converts them into comprehensive reasoning datasets that effectively integrate both Macol and Mixt reasoning modalities.
Step-1: Mycassity Assesment for Tool Calling
Step-2: Mode Specits CoT Arnotattion
Based Data Deaniin
Reasoring
Ground Truth
Roasoming
Validaiion
usage control
High
Low
Date w/ CoT
Other
Judge
Formst standariiation
Rew Dataset
Ressoning wil Tasls
0.5
Reasoning
Wy Tocls
Scar Coll 10.3)
Content correction
Cold
Regonerate
Figure 5: The automated three-stage pipeline for SFT data construction. The process consists of: (1) Necessity Assessment, which partitions data into tool-necessary Duco and tool-unnecessary Diest sets; (2) Mode-Speae Specific CoT Annotation, where a teacher model's outputs are refined through dalidation-regeneratioo loop with judge model; and (3) Rule-Based Data Cleaning to produce the final dataset.
Step 1: Necessity Assessment for Tool Calling. This initial step partitions the raw dataset by discerning the necessity of tool use for each sample. Our two-model approach aims to establish clear boundaries for scenario complexity. We first employ a lightweight Qwen2.5-Ve VL--3B model to filter out simple scenarios where high accuracy (e.g., 0.9) is achievable in the Mix mode, forming the cool-unnecessary set Drex.. For the remaining samples, a powerful Qwen2.5-VL-72B oracle model determines the tool-necessary set Duol by identifying cases where the MMool mode yields significant performance gain over the Mie mode. This use of a highly capable oracle provides a reliable, conservative criterion for tool necessity, ensuring that we identify scenarios where external perception is genuinely beneficial, while leaving ambiguous cases for the exploratory set Desplore used in subsequent RL stages.
Step 2: Mode-Specific CoT Annotation and Validation. In this step, we generate mode- seee-specific CoT annotations using the powerful Qwen2.5-VL-72BB as the "teacher model." For each classified sample, the teacher model performs "inverse" reasoning from the ground-truth truth meta-actions to construct logical thought process in either the Muss or Miool mode. By anchoring the teacher model to the ground-truth action, we effectively steer the reasoning process. This approach corroborates recent observations (Zhou et al., 2025) regarding the utility of constrained or prompted generation in producing high-quality annotations. To ensure the quality and coherence of these annotations, we implement a rigorous validation and filtering process. A separate "judge model" scores the logical consistency of each generated CoT on scale s E [o, 1]. Samples with scores falling below predefined quality threshold T are automatically sent back to the teacher model for regeneration. This iterative refinement loop ensures that only high-fidelity reasoning paths are retained. To ensure balanced distribution of tool use within the Dioet set, a Dynamic Tool Preference Guidance mechanism is also employed during this process, preventing a preference tool bias in the annotated data.
Step 3: Rule-Based Data Cleaning. In the final step, the annotated data is subjected to a rule- based cleaning pipeline to ensure its fidelity and uniformity. This pipeline performs three primary functions: (1) Format Standardization, which validates the structural integrity of the CoT and the syntactical correctness of tool calls; (2) Content Correction, which rectifies content-level errors such annotation artifacts mismatches between the final prediction and the ground-truth label; and (3) Tool Usage Control, which normalizes tool usage by constraining call frequency and eliminating redundant illogical calls.
18

---

Published as a conference paper at ICLR 2026
After this pipeline, sample an equal number of high-quality instances from Dust and Diool to construct the final cold-start SFT dataset, providing a solid foundation for the model's hyyrid-thinking capabilities.
A.5 REWARD DESIGN DETAILS
Accuracy Reward (Rac)). The accuracy reward, Race is designed to evaluate the quality of the generated meta-action sequence by comparing it the ground-truth sequence. It is composite score that combines the accuracy of speed and trajectory predictions, with a higher weight on speed to emphasize safety. The formulation is follows:
Race = X Repeed + >j. Pirij
where >, 0.7 and x, 0.3 the weighting factors for the speed reward (Rapeed) and trajectory reward (Rmy), respectively.
Both Repeed and Ruaj are calculated based on weighted Levenshtein distance, which measures the similarity between the predicted sequence À = (âj...., âm) and the ground-truth sequence A (dl,...,ap)) The similarity score for a sequence is defined as:
Raeg [A A) = 1 D(A, A)
max(m, 72)
where D(A, A) is the weighted Levenshtein distance. The distance is computed recursively. Let ji be the distance between the first i elements of Å and the first j elements of A. Then:
D(i 1,j) Cael
D(I.)) min
D(i.j - 1) + Cin
D(i 1,j--1jj-0 - Cusb(ās, aj..)
The costs deletion (Caa) and insertion (Cins) are set to constant value of 0.6 to allow for more flexible alignment between sequences of potentially different lengths. The substitution cost, Csub- is defined as 1 - S(ā,, j). where S is position-aware match score:
S(ā, a, t) I(a a) Waat
Here, I(.) is the indicator function, which is the predicted action à matches the ground-truth action a, and 0 otherwise. Wa,t is the position-action weight for the ground-trute truth action a at timestep
This weighting mechanism is designed counteract the data imbalance. By up-wetthting rarer, more critical actions (e.g.. "Stop"", "Left Turn"), we encourage the model to learn a more balanced and robust fecision-making policy. The weights derived from the frequency fe,t of each action class c at each timestep in the training data. The calculation proceeds in three steps:
1. Raw Inverse Frequency Weight: A weight w/,, is calculated based the inverse frequency relative to the mean frequency f. that timestep, controlled by an exponent ? = 0.5:
(Lite) Jost +e
2. Clipping: The raw weights are clipped to a predefined range [Wminn Wmax] (e.g., [0.7, 1.3)) prevent extreme values from destabilizing training:
-", = clip(w'.t> Wmin Wmax)
3. Normalization: Finally, the clipped weights for each timestep are normalized to have mean of I, ensuring that the weighting scheme adjusts the relative importance of actions without altering the overall scale of the reward at each timestep:
We,t 1
Evev whr
IV
where V is the all possible actions for the given component (speed or trajectory).
This comprehensive reward design ensures that the agent is rewarded not only for correctness but also for learning handle critical, less-frequent driving maneuvers effectively.
19

---

Published as a conference paper at ICLR 2026
Tool Usage Reward (Rioot). In the final AMS-RL stage, the reward function is augmented with tool-usage component, Rool, to guide the agent in learning when to use tools. This reward is only applied to trajectories generated in the tool-use mode (Maal) and is designed penalize superfluous tool calls while rewarding effective ones. The core mechanism is a contrastive evaluation within each group of G generated responses for given input.
First, we establish performance baseline, Riext by calculating the average accuracy of all text- only trajectories within the group. If a group contains such text-only paths, this baseline cannot be established, and the Rioo reward is not applied (i.e., set to zero) for that group, as no contrastive comparison is possible. For groups where a baseline exists, the reward for each tool-assisted trajectory i is determined by its performance gain over this baseline, adjusted for penalty corresponding to the number of tool calls.
The tool usage reward, Rool, for trajectory i that uses N, tools is calculated as:
Ryool = (Race, Flest N₂ Cuol
Parameter Derivation: We set the per-call cost coefficient Ctoot based on "Minimum Effective Gain" principle rather than empirical tuning. We posit that active perception is justified only if corrects at least one meta-action in the 4-step sequence, corresponding to an accuracy gain of 0.25. Amortizing this gain
the median number of expected tool calls (2 calls, given range of 1-3), we derive Ctool = 0.25/2 = 0.125. This design explicitly penalizes superfluous tool calls: the accumulated cost N₂- Czao! outweighs the accuracy gain, the agent receives negative reward, incentivizing it to invoke tools only when they provide a clear performance advantage.
Finally, the reward is clipped to fixed range, for instance [-0.2, 0.2). to ensure training stability. This design explicitly incentivizes the agent to invoke active perception only when the accuracy improvement is significant enough to overcome the inherent cost of tool usage, fostering an intelligent and efficient decision-making process.
20

---

Published as a conference paper at ICLR 2026
A.6
TRAINING AND EVALUATION PROMPT TEMPLATE
Prompt Template for Model Training and Evaluation
<context>>
You are provided with front-view image of the car camera from the current frame. The navigation command is: fnaviga witiiiionnomanl... Your current speed is: {speed} km/h.
<task>
Your task analyze the driving scenario, construct natural, logical reasoning process and predict meta-actions. The following three stages are guidelines for structuring your thoughts:
1.
Briefly describe the key aspects of the driving environment relevant to the decision. (<4 sentences)
2. <reason ing>: Briefty explain logically how the perceived situation helps you to predict the meta-actions. (<4 sentences)
3. spredictions Based on reasoning, briefly explain how you arrived the meta-actions.
(<2 sentences)
Finally, output the meta-actions for the current frame and the next 3 frames (2s interval between frames).
Output ONE composite action for each frame, each consisting of one speed token from {"Stop", "Keep Speed", "Accelerate", "Decelerate") and one trajectory token from {"Left Turn", "Right Turn" "Straight Note that Left Turn and Right Turn include both major turns and minor adjustments like lane changes or small heading corrections.
<rule>
Your response should be concise, prefer short, direct sentences.
First check context suffices for prediction, then choose <think.with.tools s>> or <think.no.toolsx.
<tool usage>>
Under www.ii..wwhhhh...eee www.wwtwthtt.s......om you need to consider the following content:
You recommended to call the tools multiple times for better understanding (max calls).
Please call the tool meaningfully, as each call to the tool incurs additional costs.
Example:
<think_with with wttp:/iitttttttl(oott__
tcols> (or schink_no_toolle
<description>
(yeu can call tools here under <think_with__tolls>>> </descriptien>
<reasoning> igg
(you can call tools here under <think_ th th_toola>) </ </rassnning> <predictien>
... (you call tools here under <think with tools>)
</prediction>
</think__iit__toole>>>> (or s/think__n_ttools>>
<meta act aotso ('Sped,, Traj', 'Speed, Traj', 'Speed, Traj', 'Speed, Traj' </meta actions>
A.7 AUTOMATED META-ACTION LABELING PIPELINE
To generate high-quality training labels, developed robust automated pipeline that converts raw vehicle trajectory data into discrete meta-action sequences. Distinguishing itself from static rule-based direct VLM generation approaches, method employs Iterative Dual-Feedback
21

---

Published as a conference paper at ICLR 2026
Optimization strategy. This pipeline optimizes kinematic rule thresholds through cycle of VLM- based consistency checks and human verification, ensuring that the final labels are both physically plausible and semantically accurate before undergoing final refinement for complex cases.
A.7.1 RULE-BASED GENERATION
The core of our pipeline is rule-based labeling process that maps continuous trajectory data predefined vocabulary of speed and trajectory actions. This process is governed by sliding window mechanism.
Sliding Window Mechanism. We employ sliding window that spans a 3-second duration, encompassing 1 second of historical trajectory and 2 seconds of future trajectory relative to its center point. Starting from the current frame, the window slides forward four times in 2-second increments. This procedure generates four-step discrete meta-action sequence, representing the planned behavior for the next 8 seconds 2-second resolution.
Trajectory-to-Action Mapping. For each segment of the trajectory captured by the window, we derive a composite meta-action consisting of a longitudinal (speed) and a lateral (trajectory) component based on the following rules:
Longitudinal Action (Speed): The speed action is determined by the vehicle's average speed and acceleration within the window.
Stop: If the speed is below a threshold of 0.5 m./s//.
Accelerate / Decelerate: If the absolute acceleration exceeds a threshold of 0.3 m/s?
Keep Speed: If the absolute acceleration is within the +-0.3 m/s² range.
Lateral Action (Trajectory): The trajectory action is classified by calculating the overall angle between the initial and final direction vectors of the trajectory segment within the window.
-Straight: If the absolute angle of deviation is less than 15%. Left Turn / Right Turn: If the angle exceeds 15,, with the direction determined by the sign of the angle.
Physically Grounded Thresholds. The thresholds employed in our rules are physically grounded to ensure interpretability. Specifically, the speed threshold of 0.5 m/s effectively distinguishes intentional movement from sensor noise or station-keeping drift. Similarly, the acceleration threshold of 0.3 m/s² captures deliberate velocity changes while filtering out minor fluctuations typical of cruising. The heading deviation of 15" reliably separates straight driving from intentional lateral maneuvers. Crucially, these values were not arbitrarily chosen heuristics; rather, they are the converged results of an iterative Feedb-Fee optimization process, which effectively bridges kinematic rules with semantic understanding, as detailed in the following section.
A.7.2 ITERATIVE THRESHOLD OPTIMIZATION AND VLM-BASED REFINEMENT
While rule-based mapping provides a consistent baseline, determining the optimal boundaries for "intentional" driving actions can be ambiguous. To ensure high-quality labels and address the known limitations of VLMs in temporal grounding, we implemented "Rule-Bar Based Generation + VLM-as-Judge Human Verification" pipeline with a dual-feedback mechanism.
VLM-as-Judge with BEV Visualization. In this step, we leverage the advanced reasoning capabili- ties of GPT-4.1 to act proxy for human verification. We provide the model with bird's s-ess--y-vvie view (BEV) visualization of the ego-vehicle's future trajectory alongside the initial rule-based meta-action sequence. This strategy of providing explicit visual context effectively mitigates the temporal ground- ing weaknesses often observed in VLMs, challenge also highlighted and similarly addressed Xu et al. (2025b) through visual prompts. The model is then tasked with evaluating the sequence for logical consistency.
Dual-Feedback Iterative Tuning, We treated the kinematic thresholds (speed, acceleration. yaw) as parameters within a dual- feel-bec loop: 1) VLM Feedback: The VLM scored the consistency between the rule-genente generaedd labels and the BEV visualizations. 2) Human Feedback: We conducted
22

---

Published as a conference paper at ICLR 2026
random spot-checks to calculate the false-label rate. In each iteration, thresholds were adjusted to simultancously maximize the VLM's consistency alignment score and minimize the human observed error rate. The thresholds reported in the previous section represent the converged values from this optimization.
Final Refinement for Hard Cases. For the small fraction of remaining boundary cases where rule-based outputs remained ambiguous (low VLM confidence), we performed a final pass using GPT-4.1. By providing the model with the BEV visualization and the tentative rule-based sequence, we allowed it to refine the labels based on logical consistency and scene context. The prompt used for this final refinement detailed below.
VLM Refinement Prompt
System Prompt:
You are an expert driving meta-action evaluator corrector. You will be given bird's- eye-view (BEV) image where a blue polyline indicates the future ego trajectory. You will also receive 4-step rulc-based meta-action sequence, each step being a pair: [specd_action, trajectory _action]. Jactctoryacctio]]. Your job: (1) rate how consistent the sequence is with the visible trajectory shape and plausible driving behavior, (2) correct any unreasonable actions.
User Prompt:
Consider the image and the rule-based 4-step meta-action sequence.
Allowed speed actions: Stop, Keep Speed, Accelerate, Decelerate.
Allowed trajectory actions: Left Tum, Right Tum, Straight.
Return a strict JSON object with keys: score (0-10 integer), reason, and final www.m/eaaattteee actions
meta metaatioooctsons MUST be a list of EXACTLY strings, preserving the original speed action for each step.
Each string MUST be in the form <spedd>> <trajectory>> using ONLY the allowed labels.
Temporal semantics: the 4 steps cover (0-2s), [2-4s), [4-6s), [6-8s) after the current frame; the first step is the current 0-2s window.
Trajectory evidence: ONLY use the first 16 future trajectory points starting from the current ego position; ignore farther points.
Global trend tolerance: turning trend does NOT require all steps to be turning; sequences like 'Straight, Straight, Left Turn, Left Turn' are acceptable.
Stationary cases: if future points are nearly stationary (clustered at one location), DO NOT infer a turm; set the trajectory action 'Straight' for those windows.
Plausibility filter: only fix clearly unreasonable trajectory actions given the geometry and the 2s window (e.g., 'Stop, Left Turn' -> 'Stop, Straight').
Do not add any commentary. Output only the JSON.
A.8 HUMAN EVALUATION DETAILS
To establish a credible human performance benchmark, we recruited three licensed drivers, each possessing over two years of driving experience. The evaluation tasked these participants with the same high-level planning challenge assigned to the agent: predicting a sequence of four meta-actions for each driving scenario. To facilitate this, we developed dedicated evaluation interface, depicted Figure 6. For each sample, this interface presented the forward-facing camera view, current vehicle speed, and the navigation directive. Participants then selected the appropriate meta-actions for the subsequent 8-second horizon using a series of dropdown menus. The test sets from both Drive-Inters and nuScenes partitioned equally, with each participant evaluating unique third of the samples. The final "Human" baseline score reported in the main results is the average of the accuracies from these three evaluators.
23

---

Published as a conference paper at ICLR 2026
Human Evaluation
Indre
Spreet 2073700eass000
coco uncrp
eeoor serel, pnocredo Indermey sruil Uns cndegr
Frome
Fore
Fame 2
Fram 3
Spend
Spannd
Spend
Tra entory
Tralesoyy
Tragesary
Trigennyy
Figure 6: The user interface for human evaluation. For each scenario, evaluators were provided with the front-view camera image, navigation instruction, and the current vehicle speed. They used the dropdown menus to predict the 4-step meta-action sequence for the upcoming 8-second interval.
A.9 EXTENDED EXPERIMENTAL RESULTS
A.9.1 DETAILED RESULTS FOR DOMAIN ALIGNMENT
To provide comprehensive view of the benefits of our autonomous driving domain alignment strategy, this section presents detailed performance breakdown of our DriveAlign-3B model. Table 9 compares its performance against the base model, Swen2.5-VL-3B. and the larger Qwen2.5- VL-7B across our domain- speccfc DriveAlign- -VQues benchmark and a suite of eight general VLM benchmarks.
As referenced in the main paper, the results demonstrate that DriveAlign-31 not only achieves a significant improvement of +11.7 points in the domain-specific overall score but also enhances its general VLM capabilities. This dual improvement validates our approach of developing robust driving- specific priors without compromising, and in many cases strengthening. the model's foundational reasoning abilities.
Table 9: Detailed results of domain alignment on in-domain and general abilities. Scores on Drive VQAgea (upper) and 8 general VL.M benchmarks (lower). Numbers in blue/red are deltas vs. Qwen2.5-VIL-3B.
Task
Benchmark
DriveAlign-3B (ours)
Qwen2.5-VL-3B
Qwen2.5-VL-77
scene description recognition of traffic entities
87.8 (+5.5)
82.2
91.5 (+1144)
80.1
86.4
DriveAlign-QQA...
localization critical targets
85.1 (+12.9)
72.2
67.9
traffic commonsense and rules
74.0 (+17.1)
56.9
71.5
overall
84.6 (+11.7)
72.9
77.0
MMStar
56.7 (+2.9)
53.8
60.6
MM
60.6 (-0.6)
61.2
69.0
MMBench. EN (val)
80.4 (+1.1)
79,3
79.7
MMBench EN VI1 (val)
77.4 (+1.0)
76.4
80.3
General
MME--ealwee Realworld-lite
48.3 (+6.2)
42.1
44.6
OCRBench
81.4 (-1.2)
88.
MMMU (val)
46.0 (+4.4)
41.6
48.4
AI2D
80.1 (+1.9)
78.2
80.7
overall
66.4(+2.0)
64.4
68.9
Notes. For DriveAlign-VOs VQAue. each of the four topics contains 200 samples; the judge model is Doubao Seed 1.6. The general benchmarks suite includes MMStar (Chen et al.. 2024). MMVet (Yu al., 2024).
MMBench (Liu al., 2024a), MME-R Realworld-iit (Zhang al., 2024). OCRBench (Liu et al., 2024b), MMMU (Yue et al., 2024), and A12D (Kembhavi et al., 2016). The general evaluation was performed using VLMEvalKit (Duan et al., 2024) with the default configuration.
A.9.2 EXTENDED ABLATION RESULTS FOR PROGRESSIVE TRAINING STRATEGY
To further validate the robustness our progressive training strategy, we present the ablation study results the nuScenes dataset in Table 10. These results complement the main analysis conducted on
24

---

Published as a conference paper at ICLR 2026
Table 10: Ablation on the progressive training strategy on the nuScenes test set. We evaluate different combinations of our three training stages, To ensure fair comparison, we also test variants where a single RL stage (FCM-RL or AMS-RL) is trained for two epochs, matching the total RL epochs of our final model. Best results are in bold.
Faama Regizitt
$3.06
X
DWerlgg--5 38 -060-P7
FFMM Diwengeee $1
Traising Stage
Sequence Jeg-Aont (ec
Figure 7: Progressive training gains on nuScenes test set. Accuracy in the adaptive mode (Madaptve) and MSA
DM6-SFT
FCM-RL
AMS-RL
Maa
Mauex
1 Only)
39.17
39.53
42.00
(4
ggaad
spech
+2.92
40.58
42,99
56.49
Vasias- 3 (4PC34 s2)
2 specka
45.15
41.07
44.00
58.50
41.1
57.50
improve with each training stage.
Viaiasi 5 F1MMS (Z)
2
44.19
41.47
43.57
00.30
ArrreAgrre-RI (PCM-AMM)
Tepoch
Pepcch
Eepoch
the Drive-Internal dataset in Section 4.3.2. The performance trends observed nuScenes are highly consistent with those on Drive-Internal confirming that each stage of our training pipeline- from SFT to the cascaded FCM-RL and Ra--RLoonttrbbees to steady and significant improvement in the agent's 's capabilities. This cross-datase consistency underscores the general effectiveness and reliability of our proposed training methodology.
A.10 QUALITATIVE RESULTS
This section presents qualitative examples to illustrate the capabilities of DriveAgent-R/ in diverse driving scenarios. We begin with a simple case where the agent efficiently employs text-only reasoning (See Fig. 8 & Fig. 9). Subsequently, we showcase the agent's active perception: at busy intersection, it uses Rol Inspection and Retrieve View to check signals and pedestrian flow (See Fig. 10); in a low-visibility night scene, it leverages 3D Object Detection to uncover a hidden pedestrian (See Fig. 11); when approaching a barrier, the Depth Estimation tool provides the spatial awareness navigate safely and plan ahead (See Fig. 12). Finally, two additional examples highlight the crucial role of Rol Inspection in resolving ambiguity by assessing the risk of distant pedestrians and confirming the state of faraway traffic signal (See Fig. 13). Finally, to explicitly validate the superiority of our Hybrid Thinking framework, we present two comparative analyses against rigid "forced-mode mode" baselines. These cases demonstrate how the adaptive mode outperforms forced strategies by ensuring intent-driven tool usage in complex scenarios (See Fig. 14) and preventing information redundancy in straightforward scenes (See Fig. 15).
Front View
Reasoning under Me
Description The vehicle approaching tall booth with clear lane markings. speed is moderate, and the road ahead appears to straight. There are no immediate obstacles turns visible. The navigation command to proceed through the toll booth.
Reasoning Given the current speed and proximity to the tall boath, deceleration necessary safely navigate through. After passing booth, resuming normal speed logical. The straight trajectory aligns with the goal of proceeding through the toll area without deviation. No required this scenario.
Speed:
19.0 o km/h
Prediction Decelerating initially ensures passage through tall booth. Then, maintaining speed allows for smooth travel afterward.
Navigation:
Proceed through the toll booth
("Decelerate, Straight", "Decelerate, Straight", "Keep Speed, Straight", "Keep Straight")
Figure 8: In this routine driving scenario, the vehicle approaches a toll booth with clear lane markings and straightforward navigation. Recognizing the low complexity and lack of ambiguity, PriveAgent- RI employs its efficient Mixt mode. Through text-only reasoning, it formulates a safe and logical plan: decelerate to pass through the toll gate, then maintain speed to proceed, effectively handling a common situation without needing to invoke its vision toolkit.
25

---

Published as a conference paper at ICLR 2026
Front View
Reasoning under M text
Description The road straight with no visible turns or obstacles, Trees both sides, there of the intersection. The speed moderate at 40 km/h. The navigation command to continue straight.
the immediate vicinity
Reasoning Given the clear road ahead and lack of traffic,
maintaining current speed and trajectory logical. No turns stops are necessary the path remains straight. The moderate speed suggests steady driving without acceleration or deceleration.
Speed:
40.27 km/h
Prediction Based on this reasoning. the appropriate actions are to keep the current speed and stay straight frames.
Navigation:
Continue straight
[" Keep Speed, Straight", "Keep Speed, Straight", "Keep Speed, Straight", "Keep Speed, Straight")
Figure In this case, the road is straight and the view is open with no apparent obstacles, DriveAgent- RI engages its efficient text-only reasoning mode (Mex) due to the low scene complexity. By soiely analyzing the textual scene description and navigation command, accurately assesses the simple road conditions and formulates a plan to "Keep Speed, Continue Straight." This decision is made without invoking the vision toolkit, demonstrating the efficiency of the porridtiiinkng framework.
Reasoning under M
Description scene intersection multiple vehicles and pedestrians, There are motorcyclists and cars nearby, requiring careful navigation. I need check the traffic light status.
Front View
"trost. "bbor" intrsection))
"desoriptinn: "te bulfk
are seconds left, which need to wait cautiously for pedestrians to cross. will check front-right next.
light green
frams, indec" "ourrent" "view_ Jww...neee' "front_riht""
tront-right view captures motorcyclists and pedestrians crossing the Intersection, Indicating a busy area.
Front Right View
Speed:
3.07 km/h
Reasoning Given that the current speed low and road condition is busy, stopping necessary before proceeding with the After ensuring that there are pedestrians in front of the vehicle, accelerating with right turn aligns navigation command.
Navigation:
Turn right at the current intersection
Prediction The initial actions involve stopping and assessing the environment, After ensuring safety, accelerating with a right turn.
["Stop, Straight", "Stop, Straight", "Stop, Right Turn", "Accelerate, Right Turn"]
This case highlights the necessity of active perception when navigating challenging right turn at a busy intersection with heavy pedestrian and motorcycle traffic. The initial view provides insufficient detail ascertain the traffic light's status, creating uncertainty. To resolve this, DriveAgent-RI proactively engages its Meol mode. It first invokes the "Rol Inspection" tool to confirm the green light and then use the "Retrieve View" to get the front-right view to better assess the dense flow of crossing traffic. By grounding its reasoning in this actively acquired visual evidence, the agent formulates safe, context-awas plan: to stop and wait for pedestrians clear before executing the right turn.
Figure 10:
26

---

Published as a conference paper at ICLR 2026
Reasoning under M
Description The vehicle approaching bus the right side of the road. The bus appears to be stationary or moving very slowly. There are no immediate obstacles in the lane ahead.
View
Reasoning Given the proximity of the bus and the moderate speed, it's necessary to decelerate to ensure safety, need to confirm again if there any immediate hazards the road ahead.
('visw_ndoo'' "honn" "shject "bus")
object detection confirms presence of bus, At the same time, some padestrians are detected in
Speed:
immediate path. This supports maintsining a straight trajectory while decelerating.
26.79 km/h
The initial action should be to decelerate and continue straight. Once it's confirmed that there are no immediate hazards, keep current speed.
Prediction
Navigation:
Go straight on the current road
("Decelerate, Straight", "Decelerate, Straight", "Keep Speed, Straight", "Decelerate, Straight")
This case demonstrates the agent's ability uncover potential hazards in challenging low-visibility conditions. While navigating a narrow road at night, the initial perception identifies a nearby bus but fails to detect pedestrian in the immediate path due the darkness. By invoking the "3D Object Detection" tool, DriveAgent R/ performs active perception and successfully identifies the otherwise overlooked pedestrian. This critical, tool- -ro-ggrudded discovery prompts the agent to revise its initial assessment and decide to decelerate, prioritizing safety in response to the newly revealed risk.
Figure 11:
Reasoning under M
Description The vehidle approaching barrier gate the intersection. There are traffic cones on the left side, indicating controlled entry point.
Front View
Reasoning
speed the presence necessary to decelerate initially to ensure safety as the vehicle approaches gate. Once safely past gate, maintaining a steady speed while continuing straight aligns with the navigation command. comoonanenn peneeneeiin cnnoeeenee
cones,
("Veew Jnder") "Tron""
The depth map shows that the barrier gate and traffic
relatively close,
reinforcing the need to decelerate initially. road ahead appears clear the gate.
Speed: 5.20 km/h
The vehicle should decelerate initially, once past the gate, maintain steady speed and continue straight.
Prediction
Navigation:
Continue straightafter passing the barrier gate
D"Decelerate, "traleerte,SStrighhh" "Decelerate, Straight", "Keep Speed, Straight", "Accelerate, Straight")
In scenarios requiring precise distance judgment, such as passing through this barrier gate, the "Depth Estimation" tool proves invaluable. The generated depth map provides two critical insights: first confirms the gate's close proximity, reinforcing the decision to decelerate for safe passage. More importantly, also perceives the clear and open road beyond the barrier. This foresight into the upcoming environment allows DriveAgent-R* to formulate a confident, multi-stage plan: slow down navigate the immediate obstacle, and then accelerate once safely through.
Figure 12:
27

---

Published as a conference paper at ICLR 2026
Reasoning under tool
Front View
Description The scene shows straight urban road
immediate
tums, FedEx truck is parked on the left, and there pedestrians and vehicles further ahead. will use Rol, Inspection to zoom in on the pedestrian area for better visibility any potential hazards.
Veew Indes": "son", "Bbor" (3755 180, 460, 215). sees")
description
The Rol_ Ios_lneection image shows pedestrians
road. This indicates potential hazards that require caution.
Speed: 27.46 km/h
Reasoning Given the current speed and the presence of pedestrians near the road, it's prudent decelerate initially ensure safety, After confirming immediate obstacles, maintaining steady speed.
Navigation:
Prediction Decelerating first safety. Once safe, keeping steady speed allows for smooth progression along the intended path.
Continue straight.
("Decelerate, Straight", "Keep Speed, Straight", "Decelerate, Straight", "Keep Speed, Straight")
Figure 13: Even seemingly clear, straight road, DriveAgent-R/ exhibits proactive caution by investigating distant activity. While the initial view shows pedestrians far ahead, their exact proximity the road is uncertain. By deploying the "Rol Inspection" tool, the agent obtains a magnified view. revealing that the individuals are walking very close to the lane of travel. This specific, tool-acquired insight elevates the potential risk, prompting the agent to prudently decelerate to ensure safe passage.
28

---

Published as a conference paper at ICLR 2026
Key object:
The white car approaching from the intersection
Speed:
39 km/h
Navigation:
go straight
Text
Tool Mode
Adaptive Mode (Tool Mode)
<description>
<description>
<description>
The road straight with no visible turns or obstacles. Trees line both sides of the road, and there are pedestrians on the left. No immedlate need for acceleration or deceleration is apparent.
The road ahead is wet, There are trees and a few pedestrians
The road ahead clear There are trees... A motorcycle is on the left side ("view index": "front", "bbox": [300, 170, 350, 200), "description" "the intersection ahead")
wws. ("frame Index": "to", "view_ iid__indxxx "front_left")
<reasoning>
Given the clear path ahead, maintaining current speed appropriate. There are signs change direction. The navigation
indicating
suggests continuing straight, aligning with this observation. No
The front-left view shows motorcyclist the left side of the road <reasoning?
Inspection shows
oncoming entering the
external factors require altering speed
intersection ...ruuiree deceleration.
trajectory.
Given the wet road conditions and the presence of motorcyclist, deceleration might be necessary <prediction>
<reasoning>
<prediction>
the road may be narrowed it's
Based on the reasoning, the best course of action is to keep the current speed and stay straight.
necessary to decelerate and prepare for
possible lane change to the right
Considering deceleration is
<prediction>
necessary initially. After ensuring safety,
Based on the amalysis, deceleratinn
maintaining the current speed while
while making right turn to avoid the oncoming traffic...
continuing straight ...
actions>
('Keep Speed, Straight"
["Decelerate, Straight", "Decelerate, Straight", "Keep Speed, Straight", "Keep Speed, Straight"] </meta actions>
["Decelerate, Right Turn", "Decelerate, Straight", , "Decelerate, Straight", "Decelerate, Straight") </meta actions>
</meta actions>
X
ACC: 0/100
ACC: 25/100
ACC: 100 100/100
Figure 14: Comparison of reasoning modes a complex intersection scenario. The key object distant white approaching from the intersection (top). Left (Miear)): Relying solely the defauplt resolution, the model fails to detect the distant vehicle and incorrectly decides to maintain speed. Center (Maoat)) Although forced front-left view but fails to focus on the critical area. This illustrates that without the autonomous reasoning seek specific evidence, tool usage can be ineffective. Right (Madaptive): The model identifies uncertainty and autonomously switches to Tool Mode. It proactively uses Rol Inspection to zoom in on the intersection, successfully identifying the oncoming car and planning safe lane change. This demonstrates the superiority of active, intent-driven perception over passive tool invocation.
tools, the model lacks specific intent. It calls generic
29

---

Published as a conference paper at ICLR 2026
Key object:
The white car changing lanes to the right
Speed:
34 km/h
Navigation:
proceed straight
Text Mode
Tool Mode
Adaptive Mode (Text Mode)
<description>
<description>
<description>
The road is clear ahead Traffic is moderate white car in front, turning right...
The road is straight with multiple lanes, there are immediate
The road is clear ahead with no immediate obstacles. A white front has right turn signal, indicating it may be changing lanes...
obstacles A traffic sign is visible
<reasoning>
s
("frame Index": "t0",
The white car ahead is turning right,
"view index": "front_left Jeft")
<reasoning>
it's safe to decelerate slightly to avoid a
To ensure safety, need to decelerate slightly as there are lane changes
collision, ... T..The road remains straight for the next few seconds...
The front-left
view shows
Involving other vehicles. The traffic flow suggests maintaining steady speed ... so deceleration will gradual rather
<prediction>
similar road
Decelerating initially ensures navigation around turning vehicle. Keeping straight aligns with the clear road ahead...
ahead...
<reasoning>
than abrupt...
The road clear and straight,
<prediction>
maintaining steady speed prudent.
initial deceleration needed
<meta actions>
("view_inde index": "front", oobjec text": "traffic sign")
keeping the current speed while proceeding straight is appropriate.
["Decelerate, Straight", "Decelerate, Straight", "Decelerate, Straight", "Decelerate, Straight"] </meta actions>
<meta actions>
The detected
["Decelerate, Straight", "Decelerate, Straight", "Keep Speed, Straight", Decelerate, Straight") </meta actions>
sign and other
vehicles .eeep
current speed
<prediction>
Given the clear road ahead ... the logical keep current speed while driving straight...
actions>
["Keep Speed, Straight"
</meta actions>
ACC: 87.5/100
ACC: 25/100
ACC: 100/100
Figure 15: Comparison of reasoning modes in a clear, high-dynamic scenario. The key object white car cutting into the ego-lane (top). Center (Mroot): When forced to use tools in clear scenario, the model becomes distracted. It invokes 3D Object Detection for non-critical traffic sign, introducing irrelevant information that diverts attention from the immediate braking event, leading to dangerous "Keep Speed" prediction. Right (Madaptive): ): The model correctly recognizes that the initial visual context sufficient. It selects Text Mode, avoiding the noise and cognitive load of unnecessary tool calls. By focusing on the clear visual evidence of the cut-in, it correctly predicts the necessary deceleration. This highlights the importance of Hybrid Thinking in preventing "tool abuse" and information overload.
30

---

Published as a conference paper at ICLR 2026
A.11 ANALYSIS OF A FAILURE CASE IN COMPLEX INTERSECTIONS
To provide a transparent assessment of our agent's current limitations, this section analyzes a representative failure case in complex urban intersection (see Figure 16). In this scenario, the agent initially correctly identifies its traffic light as green. However, after proactively retrieving a side-view image to better assess the intersection, misitterprets red pedestrian signal as its own, causing it to override its correct initial judgment and crroneously plan to stop.
This failure highlights key limitation: the agent's over-reliance on new visual evidence from its tools, which stems from an insufficient grasp of complex road topologies and traffic rules. Future work will directly address this challenge through two complementary directions: (1) strengthening the agent's foundational understanding of complex road topologies and traffic rules through targeted domain alignment, and (2) cultivating a more nuanced reasoning process capable of resolving informational confficts. This will involve training the agent adversarial scenarios with ambiguous visual cues, teaching it to weigh evidence and resolve contradictions rather than uncritically accepting the most recent perception. By implementing these measures, aim to develop more robust agent that not only actively perceives its environment but also critically reasons about what it sees, leading to safer and more reliable decision-makee makinn.
Reasoning under ranl
Front View
Description The scene is nighttime urban intersection A pedestrian crossing visible with multiple pedestrians holding umbrellas, The traffic light ahead shows green indicating it's safe to proceed.
Trame Indee". "cument"'
"Viw Jndei "front_ight right))
front-right view shows ... The traffic light is red, indicating a to stop before proceeding.
Speed: 13.95 km/h
The pedestrian crossing is busy, and the traffic light is red Deceleration necessary ... decelerating further allow controlled right turn after stopping required by the traffic light's state.
Reasoning
Navigation: Turn right current intersection.
Prediction To safely navigate this scenario, will decelerate initially to manage the pedestrian activity and assess the traffic light the light remains red, will decelerate further before stopping.
Pred: ("Decelerate, Right Turn", "Decelerate, Right Turn", "Stop, Right Turn", "Decelerate, Right Turn"]
X
GT: ["Decelerate, Right Tur",. "Decelerate, Right Turn", "Keep Speed, Right Turn", "Keep Speed, Right Turn"]
Figure 16: A failure case demonstrating incorrect traffic light identification. The agent initially cor- rectly identifies the forward traffic light as green, but after retrieving a front-right view, it mistakenly focuses on the red pedestrian light, leading to an unnecessary "Stop" command in its final plan.
31

---

Published as a conference paper at ICLR 2026
A.12 DYNAMIC MODE SWITCHING ANALYSIS
In this section, we include two detailed figure sequences (Fig. 17 and Fig. 19) visualizing the agent's s ability to switch between Miezt and Mtoot based on evolving visual observations.
<description>
The road is wet, indicating rainy conditions. A stop sign visible on the right, and there's upcoming intersection The navigation command is to turn slightly right at the intersection and continue straight. //decription>
markings.
currently traveling km/h.
<reasoninng
The wet road requires cautious driving, especially near intersections. Decelerating before turning safety, right adjustment necessary align with the navigation command. After turning, accelerating straight logical to maintain flow. </reasoning>
t-4
<prodiction>
Decelerating first ensures safe entry into the intersection. Accelerating after the turn helps maintain along the intended path.
</prediction>
</think text>
<meta actions>
t-3
I'Decelerate, Straight", "Decelerate, Right Turn", "Accelerate, Straight", "Accelerate, Straight")
</meta actions>
DriveAgent-R1 Switches thinking mode from text mode to tool mode.
<think_ tool
<description>
t-2
The
on right cones current
speed low, indicating cautious driving. A slightly right turn is required at the upcoming intersection. Using Inspection help identify any potential hazards obstacles area where the truck located.
["view_ ("view_index" "front", 140, 230),
Cambridg
"description": "the truck on the right")
</decription>
t-1
<reasoning>
The Rol, Inspection confirms the presence of truck on the right, which may pose an obstacle safety
for left tur. Given the low speed and wet conditions, stopping necessary before proceeding with the turn. The navigation command indicates right turn, but due to the current low speed and potential hazards, stopping seems prudent bbefore continuing the turn. </ e/resonnngg
<predictions
Based on the analysis, stopping required the current frame and subsequent frames safely navigate the intersection.
</prediction>
</thinkk toob>
<mmtt actions>
['Stop, Right Turn", "Stop, Right Turn", "Stop, Right Turn", "Stop, Right Turn']
</meta actions
Figure 17: Dynamic Mode Switching from Mies: to Miool in response to emerging risk. This sequence illustrates the agent's ability to adapt its reasoning strategy during streaming inference. (Top 4): The agent operates in Text Mode. Observing the wet road conditions and the upcoming intersection, it relies on internal knowledge to formulate a plan to "Decelerate" and then "Turn Slightly Right." (Bottom t): As the vehicle approaches the intersection, the agent identifies truck on the right side as a potential conflict. immediately switches to Tool Mode and invokes Rol Inspection to confirm the target. Grounded by this specifie visual evidence, the agent updates its reasoning to prioritize caution, revising its decision to "Stop" to avoid potential collision hazards.
32

---

Published as a conference paper at ICLR 2026
<think,
edescriptions
left, they be moving using equipment. The road appears to be clear ahead with a yellow dashed line guiding the path. Using help distances accurately.
There people
("viow Index": "front")
The depth map shows that the people the left relatively close. ahead appears slightly further away, indicating immediate immediate obstruction. However, considering the potential movement of the individuals and their proximity to the vehicle, it's prudent to adjust speed and direction slightly. </description>
creasoning>
Given the depth estimation and the presence of people near the vehicle, slowing down necessary for safety... ... due to the proximity of the individuals and potential movement, minor adjustments like right turns might be needed initially... maintaining steady speed while continuing straight... </reasonnng>
t-3
<prediction>
Based on the reasoning, the initial action should involve slight deceleration and a right turn maintain safety, Once
achieved, keeping a steady speed trajectory would
appropriate.
</thinkk _ool>
t-2
emeta actions>
I'Decelerate, Right Turn", "Decelerate, Straight", "Keep Speed, Straight", "Keep Speed, Straight")
</meta actions?
DriveAgent-R1 Switches thinking mode
from tool mode to text mode.
<think text>
cdescriptions
t-1
The road is clear with no immediate obstacles or turns. Trees line both sides, and the lane markings visible. The speed moderate at 27.83 km/h. No other vehicles are in sight. </description>
<reasoning>
Given the clear road ahead and the navigation command to continue straight, maintaining current speed and trajectory logical. There are no indicating need for acceleration deceleration. The environment suggests steady driving conditions for the next few seconds. </reasoning>
sprediction>
Based reasoning, appropriate to keep the current speed stay straight.
t
This approach aligns with the straightforward nature of the road.
</prediction>
l</think, text>
kmeta actions>
["Keep Speed, Straight"" "Keep Speed, Straight"', "Keep Speed, Straighht", "Keep Speed, Straight")
</meta actions>
Figure 18: Dynamic Mode Switching from Myool to Meezr for efficiency, This sequence demon- strates the agent's 's capability to optimize inference cost by switching modes based on scene complexity, (Top- 4): The agent encounters pedestrians and equipment on the left. It activates Tool Mode and utilizes Depth Estimation to assess the spatial relationship ("the people on the left are relatively close"). To ensure safety, it plans a maneuver to "Decelerate" and "Turn Right" to increase the lateral distance from the hazard. (Bottom t): After passing the obstruction, the scene simplifies. The agent observes a clear road with no immediate threats and speed 27.83 km/h. It dynamically switches to Text Mode, using efficient internal reasoning to generate "Keep Speed, Straight" plan, validating that it can autonomously disengage tools when they are unnecessary.
A.13 COMPARISION WITH INOEELCCIONNGGIDD ATTENTION
In this section, we address the distinction between our tool-based active perception framework and Dastruction-uuided Attention (i.e., explicitly prompting the model via text to "focus on" specific regions). While both approaches aim to guide the model's 's focus, they differ fundamentally in their mechanisms for Information Gain and Generalization.
33

---

Published as a conference paper at ICLR 2026
Theoretical Distinction: Resolution Amplifier vs. Attention Compressed Features Standard VLMs encode imput images into sequence of visual tokens through a fixed-size vision encoder. This process involves downsampling, which inevitably leads to a loss of high-free figh-freeuency details necessary for resolving small or distant objects (e.g., traffic lights at a distance).
Instruction-Guider Attention (Passive) Providing a text instruction (e-g.. "Please pay close attention to the traffic lights in the center") operates strictly within the model's self-attention mechanism. guides the model to attend to specific visual tokens within the already compressed feature space. Crucially, if the physical visual evidence (e.g., the color of a distant light) was lost during the initial downsampling and encoding, the model cannot "see" it, regardless of how explicit the text instruction is.
Active Perception (Rol Inspection) In contrast, our Rol Inspection tool acts as resolution amplifier. accesses the lossless, raw image buffer. When the agent predicts bounding box, tool extracts this region from the high-resolution source and re-encodes it. This process introduces new, high-fidelity pixel information that was absent in the global view tokens, effectively bypassing the resolution bottleneck of the base encoder.
A.13.1 QUANTITATIVE ANALYSIS
To empirically validate this distinction, we conducted comparative study on the Drive-Internal test set. We compared DriveAgent-RI (equipped with active tools) against a baseline using Textual Attention Prompting. Since manually tailoring specific coordinates for every test case is infeasible and would require an oracle, we utilized a generic, ccene-agnostic instruction designed to cover common eritical objects: "Please pay close attention the vehicles and traffic lights in the center of the image."
Table 11: Comparison Active Perception vs. Instruction-Guudde Attention on Drive-Internal test Bold denotes the best performance.
Inference Setting
First-Frame Joint Acc.
Seq. Avg. Joint Acc.
Baseline (Mtent)
45.27
43,29
Instruction-Guicn Attention Active Perception (Rol)
46.72
43.83
48.81
44.65
The results in Table 11 demonstrate that while guiding the model's attention via text instructions offers a slight improvement, Active Rol Inspection yields a significantly larger gain (+3.54% First-Frame Acc and +1.36% Seq Acc). This confirms that guiding attention within low-resolution feature map insufficient for fine-graine ine-grained planning tasks; increasing the effective resolution is required.
34

---

Published as a conference paper at ICLR 2026
A. 13.2 QUALITATIVE ANALYSIS
In this section, we present representative failure case of Instruction-Guided Attention.
Speed: 31.5 km/h
Navigation: go straight
Prompt model with Instruction
Active Perception with Rol Rol_Inspection
"Please pay close attention to the vehicles and traffic lights in the center the image."
<think_text>
<think_tools tools>
<description>
<description>
The road is clear ahead with immediate obstacles.
The vehicle approaching intersection traffic light visible ahead. The road appears clear, and there are no
Traffic signals are not visible, but the presence of intersection suggests caution. building under construction is the left.
immediate obstacles or pedestrians, To better
understand the traffic light status and potential hazards, will the Inspection tool the traffic light.
</description>
<reasoning>
Given the clear path and moderate speed, deceleration might be necessary initially ensure safety approach any unseen hazards. After confirming the intersection safe,
("view Index": "front", "bbox": [298, 155, 320, 170), "description"
maintaining current speed while proceeding straight aligns with the navigation command.
"the traffic light in front")
The Rol Inspection tool confirms that the traffic light
</reasoning>
ahead is red.
This indicates that the vehicle should stop at the intersection.
<prediction>
Decelerating first ensures safety before resuming normal speed. This matches the logical sequence of actions for the
</description>
upcoming frames.
<reasoning> Given the red traffic .igte,.../rrassoning
</prediction>
<predictions <ppeeiiicionn>>
</think_texx text>
</think_toof
meta actions:
actions:
('Decelerate, Straight", "Keep Speed, Straight", "Decelerate, Straight", "Keep Speed, Straight")
["Decelerate, Straight", "Decelerate, Straight", "Stop, Straight", "Stop, Straight")
Figure 19: Comparison of Passive Prompting vs. Active Perception. Left: Despite being prompted
"pay close attention to the vehicles and traffic lights in the center of the image," the model fails to identify the traffic light status due to the limited resolution of global visual tokens, leading to risky "Keep Speed" decision. Right: DriveAgent-R1 actively invokes Rol Inspection, retrieving high- resolution crop that reveals the Red Light, leading to correct "Stop" decision. This demonstrates that active tools provide critical information gain that static prompts cannot achieve.
35


Reviews:
[{"review_id": "pG4ikFs3be", "reviewer": "Reviewer_XEzY", "summary": "This paper presents DriveAgent-R1, an autonomous driving agent with active perception for planning. It can dynamically switch between text-only reasoning and tool-augmented visual reasoning based on scene complexity, improving interpretability and reliability. Experiments show it achieves promising performance with only 3B parameters.", "questions": "1. How was the quality of the new dataset ensured? In related works, human evaluation or questionnaires are often used to validate annotations.\n\n\n2. Did the authors encounter imperfect labels during annotation? In my experience, VLMs often struggle with temporal grounding.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n\n\n2. The experimental results are promising", "weaknesses": "1. One concern is the missing of any closed-loop experiments. Since the entire pipeline (Stages 1–3) is trained on a specific dataset, once a scenario is classified as “not needing tools,” the model will no longer invoke them, regardless of the ego’s subsequent actions. I wonder if the model would make different decisions (e.g., switching between using and not using tools) if the ego took alternative maneuvers.\n\n2. I suggest that the authors include a discussion of recent studies on VLM-generated datasets for autonomous driving. For example, works[1][2] use VLMs to produce new annotations through predefined prompts or questions, which are closely related to this paper’s dataset generation strategy.\n\n[1] Y. Xu et al., “VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision,” CoRL, 2025.  \n[2] Z. Zhou et al., “AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning,” NeurIPS, 2025.\n\n3. Another concern relates to the use of different tools described in Section 2.1. These tools seem to offer no new information, they mainly act as passive “visual prompts,” such as cropping or highlighting key regions. How does this differ from simply providing the original image and prompting the model with instructions like “Please focus on the top-right region” or “Please focus on the grey van in the center”? It would be interesting to see a comparison of these approaches, as it might reduce the need for a complex tool-selection module.\n\n4. The paper uses only the front-view image in its pipeline. While this may be sufficient for relatively simple scenarios like those in nuScenes (which rarely include U-turns, reverse driving, or complex lane changes), it could limit generalization to more challenging cases. Without additional camera views, even human drivers would struggle with tasks such as merging or lane changing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b7yHCbOufu", "reviewer": "Reviewer_ZEQw", "summary": "This paper proposes DriveAgent-R1, a 3 B-parameter vision-language agent that introduces two main innovations for end-to-end autonomous-driving reasoning:\n(1) an active-perception framework, where the model invokes a Vision Toolkit (Retrieve View, RoI Inspection, Depth Estimation, 3 D Object Detection) to gather additional visual evidence when uncertainty arises; and (2) a hybrid-thinking mechanism that adaptively switches between text-only reasoning and tool-augmented multimodal reasoning.\nThe system is trained through a three-stage progressive pipeline – Dual-Mode SFT, Forced Contrastive Mode RL (FCM-RL), and Adaptive Mode Selection RL (AMS-RL) – using a Drive-Internal dataset (35 k clips) and nuScenes.\nExperiments show that DriveAgent-R1 achieves human- and GPT-5-level planning accuracy with lower latency.", "questions": "1. Could authors provide quantitative examples where hybrid thinking (adaptive mode) clearly outperforms either forced text or forced tool modes?\n\n2. How is scene complexity estimated before choosing \"think_text\" vs \"think_tool\"? Is it a learned token decision or a heuristic?\n\n3. What are the inference-time compute costs of the Vision Toolkit relative to full multi-view pipelines?\n\n4. How are erroneous or redundant tool calls penalized during RL training – is R_tool margin tuned empirically?\n\n5. Can the authors clarify what “human-driver accuracy ≈ 50 %” represents (expert annotation or real trajectories)?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Introduces a novel framework where the agent proactively gathers visual evidence via a Vision Toolkit, improving interpretability and robustness.\n\n2. Proposes an adaptive reasoning mechanism and a well-designed three-stage training pipeline that balances text and tool reasoning effectively.\n\n3. Achieves near GPT-5 and human-level performance with only 3B parameters, supported by thorough ablations and clear presentation.\n\n4. The paper is clearly written with reasonable experimental design, the overall results show significantly improvement. And the adaptive method for reasoning and calling tools is also novel.", "weaknesses": "1. **Possible over-claiming of novelty.** Several contemporaneous works already explore tool-augmented or hybrid multimodal CoT reasoning (e.g., AgentThink (Qian et al., 2025), DeepEyes (Zheng et al., 2025)). While DriveAgent-R1 applies this to high-level driving planning, the “first active-perception framework” claim should be softened.\n\n2. **Limited evaluation scope.**  Tests focus on two datasets (Drive-Internal and nuScenes); closed-loop or real-world driving performance is not analyzed. Comparisons to other active-sensing or attention-based perception approaches (e.g., DriveLM, LightEMMA) are missing.\n\n3. **Data and metric ambiguities.**  The construction of Drive-Internal (35 k clips) and the VQA dataset (530 k QA pairs) is only partly described; potential data leakage or bias is unexamined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "82Su8DRBtF", "reviewer": "Reviewer_YkZV", "summary": "The paper introduces DriveAgent-R1, an autonomous driving agent that incorporates active perception and a hybrid-thinking framework for high-level driving planning. In contrast to previous Vision-Language Models (VLMs), which are limited by passive perception, DriveAgent-R1 dynamically interacts with visual tools to resolve uncertainties in complex driving scenarios. DriveAgent-R1 achieves competitive performance with top models like GPT-5, human drivers, and other VLM-based systems, using only 3B parameters while maintaining deployment efficiency. It outperforms models using passive perception in scenarios requiring active visual inspection.", "questions": "1. The performance drops significantly without access to visual tools. How would DriveAgent-R1 handle sensor failures, such as loss of camera feed or occlusions? How does the agent perform when only partial visual inputs are available?\n2. While the results on the Drive-Internal and nuScenes datasets are promising, do you have plans to evaluate the model on other benchmarks or real-world driving scenarios to further validate its generalization capability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces active perception to autonomous driving planning, an important advancement beyond the passive perception typically used in VLM-based systems. The integration of tool-based M-CoT for real-time visual reasoning represents a novel approach in this field.\n2. The methodology and results are presented in a clear, structured manner, with helpful diagrams and an effective explanation of the agent's reasoning process. The use of visual tools (such as the Vision Toolkit) for active perception is well-described and easy to follow.\n3. The hybrid-thinking framework is well-designed, allowing the agent to switch between text-based reasoning and tool-augmented reasoning. The progressive training strategy is a strong contribution, effectively training the agent to adapt its thinking mode based on scene complexity.", "weaknesses": "1. There is insufficient statistical validation such as confidence intervals or multiple trials. The reported improvements (e.g., 6.07% gain) would be more convincing with more rigorous statistical analysis.\n2. While the Cascaded RL strategy is an interesting approach, the reward design and training stages could be more clearly explained, especially for readers unfamiliar with GRPO. Additionally, the computational cost of such a training strategy could be a concern for real-world deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HUwZBTY9nT", "reviewer": "Reviewer_LB8N", "summary": "This paper introduces DriveAgent-R1, a vision-language-reasoning framework designed to handle complex, long-tail driving scenarios through hybrid thinking. The core idea is to let the model actively retrieve task-relevant information from tools or external sources to enhance its reasoning about rare or unseen driving cases.\n\nThe paper also presents a new QA dataset focused on long-tail driving events, covering diverse edge cases such as unusual pedestrian behavior, obstructed lanes, or ambiguous traffic signals. The authors show that the model’s ability to retrieve external information makes a big difference: without it, performance drops notably (as seen in Table 5).", "questions": "Are there any plans to evaluate DriveAgent-R1 on driving metrics like DE or CR? Without that, it’s hard to tell whether the improvements in reasoning actually help with driving.\n\nCan the authors provide comparisons with domain-specific baselines (e.g., DriveVLM, DriveLM, DOLPHINS)? That would make the results much more meaningful.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes a new hybrid-thinking framework, which combines internal reasoning with external information retrieval. The experiments clearly show that the active retrieval step helps — when it’s removed, performance drops quite a bit (Table 5).\n\nThe authors also built a large QA dataset targeting long-tail driving cases. Once released, this dataset should be valuable to the research community, since long-tail reasoning data is still pretty rare for driving tasks.", "weaknesses": "The biggest issue here is the choice of baselines. The paper only compares against generalist LLMs like GPT, Gemini, and Qwen. These are not trained for driving at all, so this is really an out-of-domain test for them. DriveAgent-R1, on the other hand, is trained on driving-specific data, so it’s expected to do better. The comparison doesn’t tell us how much progress this paper actually makes within the driving domain.\nIn DriveVLM, for instance, the authors showed that a domain-specific model can outperform GPT by around 90% — so the marginal improvements here aren’t that surprising. For a fair evaluation, DriveAgent-R1 should be compared against specialized models like DriveLM, DOLPHINS, DriveVLM, or similar. Without that, it’s hard to judge the real impact of this model or dataset.\n\nThe evaluation focus is also a bit off. The stated goal is to improve driving and planning, but the experiments don’t report any driving metrics like Displacement Error (DE) or Collision Rate (CR). Other VLM-based systems such as DriveVLM, OpenDriveVLA, and OmniDrive all evaluate directly on these metrics to show actual improvements in control or planning.\nHere, DriveAgent-R1 is mostly tested on reasoning or QA accuracy, not on whether it leads to better driving behavior. That leaves a big gap — it’s unclear if this approach really translates into improved driving performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)