A‌ Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms
Authors: Kejin Yu
Yuhan Sun
Taiqiang Wu
Ruixu Zhang
Zhiqiang Lin
Yuxin Meng
Junjie Wang
Yujiu Yang
Venue: Accepted by TMLR ()

Abstract: The development of high-level autonomous driving (AD) is shifting from perception-centric limitations to a more fundamental bottleneck, namely, a deficit in robust and generalizable reasoning. Although current AD systems manage structured environments, they consistently falter in long-tail scenarios and complex social interactions that require human-like judgment. Meanwhile, the advent of large language and multimodal models (LLMs and MLLMs) presents a transformative opportunity to integrate a powerful cognitive engine into AD systems, moving beyond pattern matching toward genuine comprehension. However, a systematic framework to guide this integration is critically lacking. To bridge this gap, we provide a comprehensive review of this emerging field and argue that reasoning should be elevated from a modular component to the system's cognitive core. Specifically, we first propose a novel Cognitive Hierarchy to decompose the monolithic driving task according to its cognitive and interactive complexity. Building on this, we further derive and systematize seven core reasoning challenges, such as the responsiveness-reasoning trade-off and social-game reasoning. Furthermore, we conduct a dual-perspective review of the state-of-the-art, analyzing both system-centric approaches to architecting intelligent agents and evaluation-centric practices for their validation. Our analysis reveals a clear trend toward holistic and interpretable "glass-box'' agents. In conclusion, we identify a fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisecond-scale, safety-critical demands of vehicle control. For future work, a primary objective is to bridge the symbolic-to-physical gap by developing verifiable neuro-symbolic architectures, robust reasoning under uncertainty, and scalable models for implicit social negotiation.

Published in Transactions on Machine Learning Research (03/2026)
A Survey of Reasoning in Autonomous Driving Systems: Open Challenges and Emerging Paradigms
Kejin Yu 1x
ykj21 @mails tsinghhua edu.cn
Yuhan Sun 2*
512859020230stuu ecnu. edu.co
Taiqiang Wu
takiunu@connect. hku.
Ruixu Zhang'
rx-shang 25 @mailss tsinghua heattgghaaeuu..cr
Zhiqiang Lin
linzq 24 @maaiss. tsingguu...ngnua.... edu.cn
Yuxin Meng
meng- 25 @mails tsinghua. edu.cr cn
Junjie Wang !f
wangjun wawgiiinsss.ss.tsetseee edu. yang. yujiu Øsz. Lsinghuu edu.
Yujiu Yang'
Tsinghua Universily 2 East China Normal Universily
3 Thhe University of Hong Kong
Reviewed on OpenReview:
https: openreview. net/ permform?? id= XuQ7dc4bbn
Abstract
The development of high-level autonomous driving (AD) is shifting from perception-centric limitations to more fundamental bottleneck, namely, a deficit in robust and generalizable reasoning. Although current AD systems manage structured environments, they consis- lently falter in long-tail scenarios and complex social interactions that require human-like judgment. Meanwhile, the advent of large language and multimodal models (LLMs and MLLMs) presents a transformative opportunity to integrate a powerful cognitive engine into AD systems, moving beyond pattern matching toward genuine comprehension. How- ever, a systematic framework guide this integration is critically lacking. To bridge this gap, we provide comprehensive review of this emerging field and argue that reasoning should be elevated from modular component to the system's cognitive core. Specifically, first propose novel Cognitive Hierarchy to decompose the monolithic driving task cording to its cognitive and interactive complexity. Building on this, we further derive and systematize seven core reasoning challenges, such as the respossivenes--reeoonnng trade-off' and social-game reasoning. Furthermore, we conduct a pesseeeeree review of the state- of-the-art, analyzing both system-centric approaches evaluation-centric practices for their validation. Our analysis reveals a clear trend toward holistic and interpretable "glass-box" agents. In conclusion, identify fundamental and unresolved tension between the high-latency, deliberative nature of LLM-based reasoning and the millisccond-scale, safety-critical demands of vehicle control. For future work, primary objective to bridge the symbolic-to-plysicae gap by developing verifiable neuro- symbolic architectures, robust reasoning under uncertainty, and scalable models for implicit social negotiation.
architecting intelligent agents and
"Equal contributions. Kejin Yu and Yuhan Sun contributed this work research assistants Tsinghua University. "Corresponding authors.

---

Published in Transactions on Machine Learning Research (03/2026)
1 Introduction
"The eye sees only what the mind is prepored to comprehend."
Robertson Davies (1952)
Autonomous driving (AD) aims to build a transportation system that is saler, more efficient, and more accessible (Muhammad et 2020). A major bottleneck in autonomous driving systems is shifting from the physical limitations of perception and control systems a deficit in robust and generalizable reason- ing. This challenge manifests in scenarios requiring intricate situational understanding and commonsense, such as navigating temporary traflic control (Ghosh et al. 2024 compensating for percept ion system degradation (Matos et al. 2025 However, integrating reasoning capabilities into AD systems remains un- derexplored (Plebe et al. 2024). To bridge the gap, we provide a comprehensive view of the integration of large language and multimodal models (LLMs and MLLMs) as a cognitive engine to address these reasoning deficits in AD systems.
For AD systems, the central and remaining challenge is shifting from perception to reasoning, specifically for large-scale real-world deployments (Chen et al.) 2021). As reported by Waymo and Cruise, more funda- mental challenge for AD is the lack of advanced reasoning, such as "planning discrepancy" and "prediction discrepancy" " (Boggs et al. 2020). With the development of hardware, the bottleneck no longer lies in per- ceplual capability, but, in absence of an inlegrated reasoning framework (Chen et al. 2021 Mahmood & Szabolesi, 2025 Xu & Sankar 2024). Therefore, the current primary objective of AD is to develop a cohesive cognitive architecture that enables advanced rcasoning across these individual components.
Fortunately, L.MSs and MLLMs exhibit remarkable reasoning capabilities, providing a promising solution to the AD reasoning bottleneck. Through pre- traiiini on large-scale data, these LLMs and MLLMs exhibit powerful emergent reasoning abilitics (Webb) 2023 Yin et al. 2024) and can further leverage rich repository of commonsense knowledge for complex situational assessment (Zhang et, al. 2024a: Wei et al. 2024b). Although their internal mechanisms differ fundamentally from human cognition, they can effectively simulate deliberative thought processes and thus address the reasoning deficit in AD (Xu et al., 2025b)). As shown in Fig. [ the reasoning capacity enables AD systems to react appropriately in challenging scenarios. For instance, when ball rolls onto street, system with such reasoning can infer that unseen child may follow and thus prompt the vehicle to decelerate preemptively. Such a capacity for probabilistic, context- aware inference is crucial for navigaling the inherent unpredielability of real-workd environments, and its profound potential as new cognitive architecture motivates the systematic review presented in this survey.
To systematically analyze the contral role of reasoning in autonomous driving and establish a clear foundation for the subsequent discussion, this survey makes the following key contributions:
Articulation of the Core Reasoning Deficit in AD. We systematically position the integration of reasoning the central and unsolved challenge for next-generation autonomous systems. We move beyond broad AT-in-driving surveys to specifically articulate why LLM-based reasoning is essential for overcoming documented real-world failures (e.g., mishandling novel construction zones or misinterpreting human social cues), establishing a focused foundation for the review.
A Novel Cognitive Hierarchy for Driving. We propose a new conceptual framework to de- construct the monolithic driving task based on cognitive and interactive complexity. This hierar- chy comprises three distinct levels: (1) the Sensorimotor Level (vehicle-t--enviromment)) (2) the Egocentric Reasoning Level (vehiele-to-agentt) and (3) the Social-Cognitivy Level (vehicle- in-society). This framework provides principled methodology for analyzing the required reasoning capabilitics at each layer.
A New Taxonomy of Core Reasoning Challenges. Building on this cognitive hierarchy, we derive and systematize seven key challenges that impede the deployment of LLM- reasoning in AD. Our framework allows us to analyze how these challenges (c.g., Responsiveness-Rease Reassonin Tradcoff, the Social Game) manifest with different priorities each cognitive level, providing a structured problem space for the research community.
2

---

Published in Transactions on Machine Learning Research (03/2026)
F2
E2
>>>a>>
Current AD System
@
ast. procoeed hormaly
>>>>>>
Frrugg samaaly.
E1
>>oe>>
There & Age STARS
1D tiil
Abelroling to precede CRI
Theres Perveeideeoon
la 17233M
These valicc zas wara 1o
Current AD System
the la Bnudeo wull.
1%
fight beftl ertirig
Reasoning AD f
va ticl the major 1os..
all VETTRR per ger
Vitkilee Pooeeor or major reads.
Thee i veniia twingh lane
>>>>>>
Current AD System
Rude-koeat city wut.iounit mulo.
velride caniiic iigheeite
Reasoning in AD
The thonighh
3
This dlccuei
are right baatib
11 nere There saniing in
Danent speed
Hee cink comene
Figure 1: Motivation: why explicit reasoning matters in autonomous driving, The left panel summarizes seven recurring reasoning challenges in our taxonomy. The right panel presents three illustrative scenarios (E1 E3) that contrast a representative current AD system driven by rule-based heuristics with a reasoning- in-AD approach that integrates contextual signals, traffic rules, and multi-agent interaction cues via explicit inference (dashed boxes). The comparison highlights how brittle policies can yield unsafe or overly conser- vative actions (red), while structured reasoning supports context-appropriate decisions (green).
A Dual-Perspective Taxonomy and Analysis of the State-of-the-Art. We provide com- prehensive review of current research through a structured analysis that distinguishes between the intelligent agent and its validation. We explore the field from two complementary viewpoints: (1) system-centric approaches and (2) evaluation-centris practices. Our analysis reveals a clear trend toward holistic, interpretable "glass-box" agents but identifics a critical gap in methods for verifying their real-time safety and social compliance.
The detailed structure of this survey is presented as follows and summarized in Fig, 2 Sec. provides the preliminary concepts of rcasoning paradigms within LLMs and MLLMs. Sec. 3 introduces our new cogni- live hierarchy to deconstruct the autonomous driving lask. Building on this framework, Sec. details our taxonomy of the seven core reasoning challenges that must be addressed. We then conduct a comprehensive review of the state-of-the-ar from two perspectives: Sec. surveys system-centric approaches, analyzing cur- rent architectures and methodologies, while Sec. 6 reviews iiallaation-enttiie practices, covering the critical benchmarks and datasets. Finally, Sec. 7 concludes the survey by summarizing key findings and proposing promising directions for future research.
2 Preliminary: Reasoning in LLMs and MLLMs
Reasoning, cornerstone of human intelligence, refers to the systematic process of forming conclusions decisions from evidence and prior knowledge. It is integral to cognitive functions such as problem-solving, dccision-making, and critical analysis. In the domain of artificial intelligence, enhancing the reasoning capabilities of large language models (LLMs) and multimodal large language models (MLLMs) represents a
critical step in the pursuit of artificial general intelligence (AGI) (Huang & Chang) 2023 Huang & Zhang 2024 Han et al.) 2025 Yan et al.] 2025 Qu et al. 2025 Liu et al. 2025a; Zhang et al. 2024c). This advancement allows models to transition from pattern recognition and instruction execution more complex problem-solving. Specifically for MLLMSs, reasoning facilitates the coherent integration of information from
3

---

Published in Transactions on Machine Learning Research (03/2026)
Reasoning in LLMs and MLLMs 2
Reasoning as Cognitive Core of AD
Perception
Sensorimotor Level
33.
Control
Reactive Stratogy
Egocentric Reasoning Level
3.2
Cognitive Hierarchy for Reasoning in AD (5
Pleming-heod Stratooy
Reasoning, with Social Commonseree
Soclal-Cognitive Level
3.3
Reasoning with Traffic Regulation
Core Challenges 3.4
Ressoning about Agent Insentioon
Heterogeneous Signal Reasoning
Perception-Coguition Blas
12
Cogsttive Hisurily
Challenges
reasoning
Responsiveness- Reasonnin Tradeoff
Embodied Reasoning AD
Taxonomy of Reasoning
Challenges AD (
Decision Reality Aligument
Sensoninoorr
Egauentic Ressaiing
Socid oog
Tackling Long-tail Scenarios
(15
Vohicle
Regulatory Compliance (4.6
The Social Game (4..7
Goal & Determiniam
Bridging the Paradigm Cap 5.1
Topat Envionment
Reasoning-
Reasoning-EBhaoeed Enhaneed Perception
Reasoning- Seasoonigg-llfommed Informed Prediction
Architectures and
Methodologies
(55
System-Ceutric Approaches
5.2
Ressoning in Planning and Dbclelou-Makiing
Integrated and Eud-to- End Agents
Reasoning Supporting and Auxiliacy Tasks
Prevailing Trends
Discussion and Insight ($5.3
Coguitive Hierardly Design Principles
Fonudational Cognitive Abilities
Evaluation Practices
System-Level and Interactive Performance
Benchmarks and Datasots (%))
Evalutting Robnstoess and Tiness
Benchmark Dataset Landscape C6.2
Prevalling Trends
Discussion and Insight (66.)
Cognitive Hiezarchy Exalontion Principles
Verifible Neuro-Symholle Architoctures, Robust Rensoning Under Molti-Modid Uncertainty Dynamie Grounding External Regelatory Kuowledge, Generative and Adversurial Evaluation Sealable Models for Imppliiit Social Negotiation
Future Work (17
Figure 2: The outline of the survey on reasoning in autonomous driving systems.
diverse modalities, such as vision and text, to form unified understanding Wang et al. 2024d 2025c
Li
gt all, 2025b Shou et al. 2025 For specific reasoning paradigms of LLMs and MLLMs, please refer to
the Appendix A
A New Paradigm: Reasoning as the Cognitive Core of Autonomous Driving
Traditional AD systems often rely on modular pipeline that consists of perception, prediction, planning, and control (Huang & Chen 2020). While this architecture demonstrates success in structured environments, it suffers from fundamental limitations. These limitations include significant information loss between discrete modules (Zhou et al.) 2024). overreliance on predefined rules, and inherent brittleness when encountering uncertain or ambiguous scenarios.
We do not claim that perception and control have been fully solved. Instead, we emphasize that in complex and loig-tail scenarios, insufficient reasoning ability can amplify perception errors, rule conflicts, and inter-
4

---

Published in Transactions on Machine Learning Research (03/2026)
action uncertainty, thereby becoming one of the key constraints for reliable real-world deployment of AD systems.
This survey explores a new paradigm that moves beyond the traditional pipeline. We propose that reasoning should not be treated as another sequential module but instead should be elevated to the role of a cognitive core for the entire system (Xu et al. (2024b). The function of this core not to replace existing modules but rather to understand, coordinate, and empower them. This approach transforms the system from a rigid, linear process into integrated, intelligent agent capable of holistic scene comprehension and adaptive decision- To fully appreciate the motivation for this paradigm shift, contrast it with preceding frameworks.
Classical rule- cole//logi--bseed planning and decision systems Bhuyan et al. 2024 while strong in formal specification, can be brittle and difficult to scale to the dynamic, unstructured nature of real-world driving, partly due to the growing complexity of rules and exceptions in open environments (Bouchard et al. 2022). Causal inference models, though theoretically robust. in distinguishing correlation from causation, often face substantial practical challenges due to the immense complexity and the presence of unobserved confounding variables inherent in traffic scenarios, making comprehensive causal modeling difficult to operationalize scale (Zhang et al. 2020). Similarly, pre-LLM neuro-symbolic systems, which attempted to merge neural perception with symbolic logic, were hampered by significant. integration challenges, often requiring laborious manual design of interfaces and Nomain--seciite languages (Sun et al. 2020), while still facing the symbol grounding problem (Harnad 1990).
These limitations suggest the need for more scalable reasoning interface. In this context, view LLM- based reasoning not as "symbolic" reasoning in the classical sense but as meddatg,- learned, explicit deliberation that helps coordinate modules and handle rule conflicts and interactions in complex driving settings.
3 A Cognitive Hierarchy for Reasoning in Autonomous Driving
A monolithic view of "driving" is insufficient for enabling targeted intervention. To map the advanced rea- soning capabilities of a central Cognitive Core to concrete operational challenges, a granular deconstruction of the driving task itself is required. Drawing inspiration from the evolving complexity of the interaction between the vehicle, its environment, and human agents (Wilde) 19766 Wang et al., (022). we propose a new conceptual framework to hierarchically structure driving tasks. As shown in Fig. this framework comprises three distinct levels: (1) the Sensorimotor Level, (2) the Egocentric Reasoning Level, and (3) the Social-Cogiitive Level. This hierarchy provides principled methodology to deconstruct the monolithic concept of "driving" into layers of increasing cognitive and interactive complexity.
3.1 Sensorimotor Level
This level corresponds to the most fundamental operations of driving and representing atomic actions. These tasks typically involve direct mapping from perceptual input to executive output and require minimal complex decision-making. The core capabilities at this level are illustrated by the following representative examples:
Perception (Cognition Sensing). Perception refers to the identification of elements in the environment and the determination of their states. This category includes foundational modules for object detection and vehicle localization, analogous to human driver visually recognizing another vehicle traffic signal.
Control (Cognition & Actuation). Control pertains to the execution of direct physical com- mands. In an autonomous system, this involves carrying out basic vehicle commands for steering, acceleration, and braking, similar to a human driver responding to a simple directive to press a pedal or turn the steering wheel.
5

---

Published in Transactions on Machine Learning Research (03/2026)
Legend
I
L
P External
G Muliiscurie
+
Reactive Strategy
Reasoning with Social
Reasoning with Traffic Regulations
Reasoning about Agent Intentions
Perception
Control
Planning-based Strategy
Object
Steering
Lane Keeping
Commonsense
Detection
Acceleration
Local Path Planning
Automated Parking
Risk Prediction
Rule-Based Decision-Making and Path Planning
Interactive Merging
Vehicle
Proactive Safety
Multi-Agent
Localization
Braking
Obstacle Avoldance
Lane Change
Control
Intention Prediction
Sensorimotor Level
Egocentric Reasoning Level
Social-Cognitive Level
Figure 3: The proposed Cognitive Hierarchy for reasoning in autonomous driving. This framework de- constructs the monolithic "driving" task into three distinct levels of incrcasing cognitive and interactive complexity: (1) the Sensorimotor Level, (2) the Egocentric Reasoning Level, and (3) the Social-Cognitive Level.
3.2 Egocentric Reasoning Level
At this level, the system reasons from an egocentric perspective, focusing on its immediate interactions with other agents. This requires the integration of multiple internal functions to perform a closed-loop operation and involves reciprocal inference and strategic negotiation in multi-agent interaction. Task completion relies primarily the coordination between perception and control, allowing the system to react other agents based on observed data without deep understanding of complex social rules underlying intent. Reasoning at this level manifests in several key strategies, including but not limited to the following:
Reactive Strategy (Cognition & Perception & Actuation). A reactive strategy involves the system adjusting the state the vehicle direct response to real-time dynamic data from sensors. For instance, when the perception system detects that a preceding vehicle decelerating, it autonomously engages the brakes maintain safe following distance and lane centering. Another example occurs when the system identifies static obstacle the road and rapidly plans and executes a local avoidance maneuver, which requires precise coordination of steering and speed. Planning-based Strategy (Cognition & Multi-source Perception & Actuation). A planning-basee based strategy involves the system executing a complete sequence of complex actions to achieve a clear objective within a specific scenario. This process depends on a comprehensive model of the environment constructed from fused sensor data. For example, during automated parking, the system integrates information from cameras and radar understand the position and boundaries of a parking space. It then plans coherent series of steering, forward, and reverse movements to guide the vehicle into the location, governed primarily by geometric and kinematic principles.
3.3 Social-Cognitive Level
This level encompassos tasks that are necessary for achieving full autonomy. It explicitly requires the system emulate the advanced cognitive abilities of human driver by incorporating an understanding of social commonsense, traffic regulations, and the predicted intentions of other agents. This constitutes profound

---

Published in Transactions on Machine Learning Research (03/2026)
level of social reasoning, demanding that the system operate as socially-aware participant within the dynamic traffic environment. Key challenges at this level are illustrated by the following forms of reasoning:
Reasoning with Social Commonsense (Cognition & Perception & Actuation & Internal Knowledge). This capability involves leveraging an internal world model to infer unstated context and anticipate likely outcomes. For instance, upon perceiving a ball rolling into the roadway, the system infers a high probability that a child is following. Consequently, the system preemptively reduces the vehicle speed. This response is not simple perception-contoll reflex but anticipatory judgment derived from sophisticated world model.
Reasoning with Traffic Regulations (Cognition & Perception & Actuation & External Knowledge). This form of reasoning requires the system to query and integrate an external knowl- edge base of traffic laws to guide planning and decision decision-making making. For example, when navigating an intersection that lacks traffic signals, the system must retrieve and apply the relevant right-of-way regulation to determine the appropriate action. This involves applying formal rule from an external source to a dynamic, real-world context.
Reasoning about Agent Intentions (Cognition & Perception & Actuation & Agent Mod- cling) This capability involves modeling the behavior of other intelligent agents to navigate com- plex, interactive scenarios. To merge into dense traffic, for example, the system must predict whether an adjacent vehicle will yield by observing the dynamics of that vehicle, such as changes in speed and trajectory. Based on this prediction, the system determines whether to accelerate for the merge or to slow down and await a new opportunity. Such interactions often require game-theorete reasoning to model the reciprocal actions of intelligent agents.
3.4 From Cognitive Hierarchy to Core Challenges
Current AD systems, aligned with Society of Automotive Engineers (SAE) Levels 2 and 3 (Standard 2021
have established foundational capabilities at both the Sensorimotor and Egocentric Reasoning levels. While proficient in basic perception-to-action loops and rule-based planning, their reasoning at the Egocentric level often lacks the flexibility to manage novel or highly interactive scenarios (Badue et al.] 2021). The critical barrier to achieving higher automation (T.4 and beyond), however, remains the largely unaddressed Social-Connitive Level, which requires navigating complex, unwritten protocols of human social interaction in traffic.
The Survey Self-Driving Cars: A Survey (Badue et al. 2021 provides a comprehensive review of auton- omy systems, showing that docision-making remains a central challenge, with perception no longer the dominant obstacle. Similarly, A Survey of Deep Learning Techniques for Autonomous Driving (Grigorescu et al., 2020) discusses how deep learning methods have successfully improved perception, yet still struggle with complex planning and behavior arbitration. A broader survey of surveys (Chen et al. 2023 further highlights that Long-tail scenario reasoning and interaction still clude current architectures. Mcanwhile, Oc- cVLA (Liu et al. 2025b) augments perception with vision language reasoning to address semantic ambiguity in 3D scenes, implicitly acknowledging that perception alone is insufficient without higher-level inference.
ORION (Fu et al. 2025) incorporates explicit reasoning mechanisms into trajectory prediction, motivatod by the observation that failures often stem from incorrect logical interpretations of agent interactions rather than sensing errors. Similarly, Drive-R1 (Li et al. 2025a. enhances planning and decision-making through chain-of-thouulltgguiiee reinforcement learning, explicitly targeting reasoning deficiencies in long-horizon and safety-critical scenarios. The emergence of these surveys and methods across perception, prediction, and planning consistently reflects a shared premise: performance limitations in autonomous driving are in- creasingly dominated not by sensory accuracy, but by insufficient reasoning over complex, uncertain, and interactive environments.
Traditional methodologies, from intricate rule-based systems to purely data-driven models, are often too rigid to address the full speetrum of these higher-level reasoning challenges. Recent breakthroughs in LLMs and MLLMs, however, offer a more powerful and unified cognitive engine capable of enhancing Egocentric
I For such boundary tasks, can serve either as an Egocentric prediction enhancement, or be elevated Social- Cognitive approach in more complex negotiation scenarios.
7

---

Published in Transactions on Machine Learning Research (03/2026)
Egocentric Reasoning Level
Social Level
Challenge Heterogeneous Signal Reasoning
Challenge Perception- Cognition Bias
Challenge Responsiveness- Reasoning Tradeoff
Challenge Decision-Realty Alignment
Challenge Tackling Long-
Challenge 6 Regulatory Compliance
Challenge The Social
Game
181
Figure 4: The laxonomy of seven core reasoning challenges in aulonomous driving. These challenges are categorized by their corresponding cognitive level: C1-C4 at the Egocentric Reasoning level, while C5 C7 are at the Social-Cognitive level. Each numbered scenario illustrates a specific challenge analyzed in the text.
Reasoning while enabling Social-Cognitive capabilities (Hu et al. 2023). Their extensive world knowledge provides the context needed to anticipate agent intentions and understand implicit social norms. Further- more, their sophisticated mnstruction-folloiiii abilities facilitate more robust planning in complex interactive scenarios and promote interpretable vehicle behavior. Critically, the architecture of these models is highly amenable to reinforcement learning, allowing for continuous adaptation in both direct agent interactions and broader social contexts (Ouyang et al., 2022). Collectively, these attributes position advanced reasoning models as transformative technology for mastering the higher levels of the cognitive hierarchy.
However, adapting these general-purpose reasoning engines autonomous driving systems remains formidable challenge. A naive integration of such models is insufficient, necessitating structured approach to identify and categorize the key difficulties. To establish this foundation, the following section introduces comprehensive taxonomy of the reasoning challenges that must be addressed. This taxonomy serves as a framework to ically analyze the limitations of current approaches and to guide future research toward the development of autonomous vehicles.
4 A Taxonomy of Reasoning Challenges in AD Systems
Achieving higher levels of vehicle autonomy requires mastering the complex reasoning demands of both the Egocentric Reasoning (C1-C4) and Social-Cognitive (C5-C7) levels. To provide structured approach for tackling this multifaceted problem, this section introduces a taxonomy that deconstructs it into seven fundamental challenges. These challenges encompass a wide spectrum of functions that require higher-level cognition, from navigating direct agent interactions understanding implicit social contexts. Fach of these challenges is visually represented by the scenarios in Fig. and is subsoquently analyzed in detail.
4.1
C1 CCalllegee #1: Heterogeneous Signal Reasoning
An AD system must process and synthesize a multitude heterogeneous data streams, including cameras, LIDAR, radar, and HD maps, to build a coherent representation of its environment. For example, central reasoning engine must integrate these disparate signals, from raw sensor data 3D point clouds. The core challenge lies in fusing this diverse information to support complex high-level reasoning.
This challenge manifests in several key areas of current model limitations. For instance, language mod- cls natively struggle with non-textual, numerical data like LiDAR point clouds, which necessitate robust mechanisms for cross-modal aligument. Furthermore, many contemporary vision models are fundamentally 2D-cenrrc, creating a critical need for architectures that perform true 3D spatial reasoning across mul- tiple views. The computational burden of processing high-resolution vidco streams also requires efficient information compression techniques that preserve temporal coherence. Finally, to effectively leverage the reasoning capabilities inherent to large language models, shift is required from dense, unstructured feature
8

---

Published in Transactions on Machine Learning Research (03/2026)
representations to object-level tokens that align the perceptual representation with the symbolic reasoning process (Hu et al. 2023).
These advanced reasoning capabilities are essential for critical driving tasks. For example, to understand complex spatial relationships, such as localizing an object "behind and to the left of a blue van" or determining if an approaching vehicle is facing forward, the system must move beyond simple 2D object detection. It must fuse images from multiple viewpoints with 3D geometric data to reason about the pose, orientation, and relative positioning of objects within unified 3D space. Morcover, reasoning over high-rcsolution video streams is critical for tracking multiple objects and interpreting behavior over time, such identifying vehicle running red light. This requires the system to dynamically associate visual cues with specific actors while maintaining temporal consistency.
4.2 Challenge #2: Perception-Cognitie Bias
The reliability of an AD system is fundamentally challenged by uncertainty originating from both imperfect sensory imput and fallible cognitive models. Extrinsic factors, such as adverse weather and sensor mal- functions, degrade perceptual data, while intrinsic factors, like model "hallucinations, " introduce cognitive errors. Overcoming these challenges requires a robust reasoning mechanism capable cross-modal valida- tion, compensatory inference, and reality-checking to maintain a stable and accurate undorstanding of the environment.
Reasoning is essential for mitigating uncertainty from extrinsic sources. For example, adverse weather like rain and fog severely compromises optical sensors such as cameras and LiDAR. Simultancously, the signals from radar, while capable of penetrating such conditions, are difficult to interpret due to complex reflective properties (Zhang et al. 2023b Matos et al., 2024). A reasoning module must therefore dynamically weigh and fuse evidence from these disparate sources to form a coherent environmental representation. Similarly, when a primary sensor is occluded or fails, the system must employ compensatory reasoning. With sensor performance degraded, the system may infer the presence of concealed hazard from a lead vehicle's sudden braking and decelerate preemptively, even without direct visual confirmation.
The system must also employ reasoning to counteract intrinsic cognitive failures, most notably model hal- lucinations. An MLLM might generate non-existent. objects, such as a phantom traffic light on an open road, or omit critical, perceived objects like traffic cones from a descriptive inference. To address this, a reasoning layer must continuously perform reality-cheeking by cross-validating the outputs from the model against raw sensor data and map information. This process allows the system to filter fabricated entities, correct omissions, and align the final decision with the ground truth of the physical world.
4.3
C3 <Challlngee #3: Responsiveness Reassonin Tradeoff
AD systems face a fundamental trade-off between real-time responsiveness and the latency inherent to deep reasoning. In critical situations, system must react within milliscconds. However, achieving this speed is complicated by three primary factors. First, the complicated reasoning processes of large models (LLMs and MLLMs) are computationally intensive and time-consuming (Tian et al. 2025a). Second, reliance on remote foundational models for inference introduces significant network latency, which is often incompatible with the stringent real-time demands of driving (Krentsel et al. 2024 Tahir & Parasuraman, 2025). Third, the massive data streams from high-resolution, multi-modal sensors impose a. heavy computational load on
onboard hardware, further hindering the efficient exccution of sophisticated models (Wang et al. 2024c
Brown et al. 2023).
This tension between deliberative "slow thinking" and reactive "fast thinking" necessitates the development of dual-process architectures. Such a system must dynamically arbitrate between fast, reactive controller for immediate responses and a deep, deliberative engine for complex planning and prediction (Serban el al. 2018). This challenge is particularly evident scenarios that require the system to balance immediate action with strategic foresight. For example, a vehicle may need to execute an emergency maneuver to avoid a sudden obstacle, an action requiring the immediate activation of a reactive module. If this occurs during a complex, game-theoretic negotiation like a lane merge, the deliberative module must simultaneously evaluate
9

---

Published in Transactions on Machine Learning Research (03/2026)
the long-term consequences associated with its actions, demanding seamless coordination between the two processes. This trade-off also manifests acutely during high-speed merging maneuvers. When entering dense highway traffic, the system must identify a safe gap in a fleeting time window, a task that requires both rapid perception of vehicle dynamics and deep reasoning about the intentions of other drivers. The final decision must synthesize these inputs with an assessment of the stalus of the ego-vehicle, where slight delay risks a missed opportunity and a misjudgment in reasoning could lead to a collision.
4.4 Challenge #4: Decision-Reality Alignment
A core challenge for AD systems is grounding high-level semantic decisions in the physical reality of the vehicle and its environment. A significant gap often exists between the abstract reasoning of large models and the concrete requirements of motion planning, leading to inconsistencies between a conceptual decision and the final exccutable trajectory (Yao et al., (024)). For instance, chain-of-thought output like "change to the right lane to avoid the obstacle" may be conceptually sound but physically infeasible if it disregards the kinodynamic constraints of the vehicle (e.g., turning radius, tire adhesion) or the geometric constraints of the environment. This misalignment can produce plans that are unsafe or impossible to execute. Therefore, a reasoning mechanism is imporative to ensure that high-lovel strategic decisions are contimously validated against real-world physical laws and constraints.
The necessity of this reasoning is evident in common yet complicated scenarios. For example, when maneu- vering narrow road, high-lovel decision to "reverse to yield" is insufficient. The reasoning module must further assess the physical viability of this plan by evaluating the clearance behind the vehicle and ensuring the intended trajectory does not conflict with roadside obstacles, This challenge is amplified in dynamic and uncertain conditions, such as navigating sharp turn on a slippery surface. A high-level plan for a sharp turn may be invalidated by the physical reality of reduced tire adhesion. Reasoning must connect the semantic goal of "navigate the turn" " to the physical state of the environment, inferring a drastically reduced coefficient of friction for the road surface. This inference is then used to compute a safe vehicle speed, preventing loss of control that could occur if the decision were based solely on the posted speed limit.
4.5
cs Chhallenge #5: Tackling Long-tail Scenarios
The long-tail distribution of driving scenarios poses fundamental challenge to prevailing data-driven methodologics, which falter in novel or data-scarce situations. While current models perform well in common scenarios, their elficacy sharply degrades when encountering rare events like lemporary construction zones or extreme weather (Bogdoll et al., 2022 Tian et al.) (024b). This problem stems from two interconnected lim- itations: the infcasibility of comprchensive data collection and the inherent brittleness of pattern-matching mechanisms. Consequently, the key to handling long-tail events lies not in accumulating more data, but in equipping the system with robust reasoning capabilities to make sound decisions without direct experiential precedent.
The first limitation is data scarcity. Long-tail events are, by definition, rare, making the acquisition of sufficient real-world training data logistically impossible (Zhang et al. 2023a). While data synthesis offers a partial remedy, generating high-fidelity, physically realistic simulation data remains a formidable challenge, and it is impossible to enumerate all potential corner cases beforehand (Dosovitskiy et al.) 2017; Wang at all 202T). The existence of these "unknown unknowns" places a ceiling on any strategy that relies on exhanstive data coverage. Thercfore, an autonomous system must be able to generalize from underlying commonsense principles rather than merely interpolating from seen data. The second limitation is the operational mechanism of current models, which relies on pattern matching rather than logical inference. This approach is effective at generalizing within the training distribution but becomes brittle when confronted with out-of-distributton long-tail events @Grigorescu et al. 2020). Such scenarios often involve dynamic rule changes and conflicting information, which demand causal and logical deliberation that pattern matching cannot provide.
The critical role of reasoning is evident in how a system handles such scenarios. For instance, when a pedestrian suddenly emerges from occluded area, a system limited to pattern matching may fail without
10

---

Published in Transactions on Machine Learning Research (03/2026)
direct precedent. In contrast, reasoning-enablee system can proactively apply defensive driving principles, using commonsense knowledge that occlusions create blind spots or inferring that a wrong-way vehicle signals a downstream anomaly to reduce speed. This inferential capability is also essential for resolving rule conficcs. As an example, in the temporary construction zone, hand signals from a traffic officer instruct the vehicle to stop, contradicting a green traflic light. The challenge here is not perceptual but decision-Cheoretic.. The system must apply a known hierarchy of authority (e.g., human officer > traffic signal > map data) to deduce that standing regulations are temporarily superseded and execute the action corresponding to the highest-priority command.
4.6
c6 Challenge #6: Regulatory Compliance
An AD system must navigate a complicated and dynamic web of traffic regulations ensure safe and legal operation. This requirement presents significant reasoning challenge, which can be deconstructed into two primary dimensions: the complexity of interpretation and context-dependeenee The difficulty in interpreta- tion stems from regulatory landscape that includes not only thousands of legal statutes but also numerous local ordinances and unwritten driving conventions (Koopman & Wagner 2017 Lin 2015). A scenario in which a vehicle must reason about logal statutos, local customs, and rogional standards to determino proper bus-lane use highlights this complexily. Compounding this issue, traffic laws vary significantly across differ- ent jurisdictions, necessitating robust system capabilities for retrieving and interpreting applicable rules in any given context (Dixit ct al.) 2016). Furthermore, the application of these rules is highly dependent on the immediate situation. The system must continuously analyze the behavior of other road users, current road conditions, and environmental factors to select and prioritize relevant regulations.
Addressing this challenge demands a multi-step reasoning process: from scene perception, the system must retrieve relevant regulations, judge their applicability, and resolve any conflicts among them to yield a safe and lawful decision. Numerous driving scenarios illustrate this requirement. For instance, consider unprotected turns at intersections, which research indicates account for a substantial portion of traffic accidents (Li gt all, 2019). In these settings, the system must accurately infer the assignment of right-of-wan oayy and integrate this inference with predictions regarding the intent of other vehicles and a comprehensive risk assessment. Another critical scenario involves the system's 's response to emergency vehicles or temporary traffic controllers. The detection of such an event must trigger a specialized reasoning protocol to retrieve and execute overriding rules, such as yielding to an ambulance, even if this action requires temporarily violating standard traffic laws. This highlights the need for system-level, robust, dynamic prioritization among conflicting regulations.
4.7 CChallenge #7: The Social Game
In mixed-traffic environments, an autonomous vehicle must operate not meroly as a law-abiding agent but as a socially intelligent participant. This requires the system to reason about the implicit, nen-verbal communication of other road users to infer their intentions and anticipate their actions. Existing models, however, often treat human drivers as passive agents, failing to interpret the subtle modulations in speed and headway used to signal intent. This can lead to interactions that are overly conservative, inefficient, dangerously aggressive [Li et al. 2024a Poots
2024
Beyond inferring the intent of others, the system's 's own actions must be legible and its decisions transparent The absence of human-like cues can make the behavior of the vehicle umpredictable to pedestrians, creating unsafe and uncomfortable situations. Furthermore, the opaque nature of end-to-end models undermines the trust from passengers and regulators, as the rationale behind decisions is not accessible (Rezwana & Lownes 2024 Zhanguzzhnova et al. 2023). Therefore, a reasoning-enabled system must not only interpret social cues but also generate behavior that is socially compliant and readily understandable.
The application of social rcasoning is critical across three primary interaction domains. First, in vehicle-to- vehicle interactions like merging, the system must decode and respond to subtle cues. Human drivers use changes in speed and headway to implicitly signal the intent to yield or proceed, and the autonomous system must participate in this dynamic nogotiation (Naisch et al., 2025 Nozari et al. 2024). Second, in vehicle- to-pedestrian scenarios, such as the unsignalized crosswalk, the system must infer the intent to cross from posture and movement. It. must then project its own yielding intent through clear and considerate actions,
11

---

Published in Transactions on Machine Learning Research (03/2026)
such early and smooth deceleration, to establish shared understanding, Finally, for vehiclo-to-passenge interaction, the system must provide transparent explanations for its actions. The ability to articulate the rationale for a sudden maneuver in natural language linking sensor inputs to a high-level decision is crucial for building trust with the occupants of the vehicle.
Summary & Discussion. The preceding analysis of seven core challenges collectively underscores a necessary paradigm shift in the development of autonomous systems: a move from pure perception and control to sophisticated, human-like reasoning. To operate safely and effectively in complex world, an autonomous system must master spectrum of reasoning capabilities. The essential challenges that we have detailed are summarized as follows:
Egocentric Reasoning Level (several challenges remain unresolved):
Heterogeneous Signal Reasoning: Systems must fuse disparate data types build a coherent and unified world model as the prerequisite for all subsequent reasoning.
Perception-Cognitinn Bias: Reasoning is required to validate information and compensate for failures arising from environmental factors intrinsic model limitations, ensuring the world model is reliable.
The high latency inherent to large models must be recon- ciled with the millisecond-level reaction times required for driving, pointing toward hybrid architec- tures that balance deliberation and reaction.
Responsivenes--Reasoniii Tradeoff:
Decision-Reality Alignment: High-level semantic decisions must be continuously aligned with vehicle kinodynamic constraints and environmental physical laws to ensure that all plans are exe- cutable.
Cogial-Cognitive Level (Largely Unsolved):
Tackling Long-tail Scenarios: In novel edge cases where direct experience is absent, systems must rely on commonsense and social context to navigate situations that defy standard patterns.
Regulatory Compliance: Systems must interpret and adhere to the complex and dynamic set of formal societal rules embodied in traffic laws to ensure all actions are legally compliant.
The Social Game: The system must infer the implicit, informal rules of human interaction, interpreting the intent of other agents and engaging in legible, socially acceptable negotiation.
This comprehensive analysis reveals critical insight. While the initial set of challenges represents pro- found engincering hurdles in building and grounding a reliable agent, the ultimate bottleneck to achicving human-like aulonomy lies navigating the complexities of the social world. Addressing these latter chal- lenges requires fundamental move beyond pattern matching. It demands the deep, structured reasoning capabilities promised by large models. The pivotal open question for future research is therefore how to inte- grate these powerful reasoning engines into AD systems, guaranteeing the absolute reliability and real-time performance required for deployment in the physical world.
5
Sentrc-Centric Approaches: Architectures and Methodologies
5.1 Bridging the Paradigm Gap: From General Reasoning Situated Autonomy
A fundamental paradigm gap separates the reasoning required for AD from the general reasoning capabilities of LLMs and MLLMs. This disparity manifests across several critical dimensions, most notably in their core objectives, operational domains, and input modalities.
Goal & Determinism. The primary mandate of an AD system is to ensure safety and control within the physical world. This mandate requires reasoning framework that is highly deterministic and predictable. Given identical perceptual inpul, the decisions of the system must remain stable and consistent, as its logic is grounded in rigorous physical laws, traffic regulations, and control theory. Conversely, the core objective of an LLM or MLLM is to operate within the informational domain. The reasoning process inherently probabilistic and creative, prioritizing semantic coherence and plausibility over physical precision. As a result, LLM MLLM generate multiple distinct. yet valid responses to single prompt.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Input & Environment. The input to autonomous driving consists of real-time, high-dimensionall, and continuous physical world data from sensors (e.g., cameras, LiDAR, and millimeter wave radar). Real physical environments are full of noise and uncertainty and must be processed in real time. In contrast, the input to LLMs mainly comprises discrete symbolic textual data. LLMs operate within a relatively closed virtual world constructed by linguistic symbols. Even when external information is acquired through tool calling, their interaction mode remains structured.
In summary, reasoning in autonomous driving is deterministic, safety-critical process engincered for precise action in the physical world. The reasoning of LLMs and MLLMs, conversely, is probabilistic, generative process designed for semantic flexibility in the informational world. This fundamental paradigm gap ne- cessitates innovative approaches to adapt and ground the powerful general reasoning within the specialized context of autonomous systems. The following taxonomy provides a systematic review of these emerging system-centric innovations.
5.2 A Baskd Taxonomy within System-Centres Approaches
A siguificant body of research now focuses on bridging the paradigm gap by adapting foundation models to the specific demands of autonomous driving. These efforts universally leverage techniques such as Chain-of- Thought (Co'T) reasoning and structured representations to endow systems with more robust capabilities. To provide a clear narrative, we structure our taxonomy to reflect a logical progression from enhancing individual components to building holistic intelligent agents. As depicted in Fig. 5 the timeline presents the chronological release of each major method, illustrating the rapid pace at which this domain is evolving. Our taxonomy groups the research landscape into thematically ordered stages of rising complexity.
Our taxonomy commences with review of methods that enhance the foundational modules for environ- mental understanding (Fig. 0]: perception (Sec. 5.2.1 and prediction (Sec. 5.2.2 It then transitions to the core action-generation module responsible for creating safe and coherent behaviors: planning and decision- decision-making (Sec. 5.2.3). Subsequently, we explore holistic architectures that transcend the traditional modular pipeline by creating more integrated and end-to-end agents (Scc. 5.2.4). Our review culminates with an examination of supporting and auxiliary tasks that enable robust system development. and valida- tion (Sec. 5.2.5). Throughout this taxonomy, we systematically connect the discussed approaches to the core challenges (C1-CC CT) identified in Sec. 4 This genealogical view, complemented by comprehensive comparison table (sce Table | in Appendix B), serves as a quick reference for rescarchers to navigate the relationships and core attributes of existing hods.
5.2.1 Reasoning-Enhance Enhanced Perception
Recent research in perception primarily focuses on moving beyond simple object detection towards a deeper, contextual understanding of the driving scene. The goal is to enhance the visual comprehension, robustness, and structured represcntation capabilities of MLLMs.
Key works in this area exemplify several strategic directions. One direction involves architectural innovations to improve fine-grained perception. For instance, as shown in Fig. 6 (a), HiLM-D (Ding et al.) 2025a)
employs a two-stream framework to better interpret and perceive high-risk objects, addressing how standard
models often fail to capture critical multi-scale details (C1, C2). DMAD (Shen et al. 2025) proposes explicit. decoupling and restructuring of motion and semantic representations, and mitigates heterogencous signal interference via architecture-level reasoning, thereby enhancing autonomous driving perception (CI). Occ (Liu et al., 2025b) proposes novel occupancy- language framework, which achieves more critical fine-grained understanding of 3D spatial semantics and addresses the inference latency issue caused by the massive number of parameters in autonomous driving models based on 3D vision-language models (C1, C3). A second strategy focuses on data-centric contributions to enable robust evaluation. Reason2Drive Nie gt all, 2024) contributes a large-scale video-text dataset designed specifically for the structured assessment of perception and reasoning. This work directly addresses data scarcity and provides a new benchmark for verifying genuine scene comprehension (C1, C5). A third direction centers on creating an explicit link between visual evidence and language-based reasoning. ARGUS (Man al. 2025) utilizes bounding boxes
13

---

Published in Transactions on Machine Learning Research (03/2026)
Legand
Rassoning in Plarning and aeciiio-Maciing Arcfésscturss End Rexeoning Supporting Audilary Twsics
V.AAD
Ressoring--ttonn informeddddd Prediction
C1 Orniiiivv
CS Pousie
Jan.
C1 C4 CS X-river
G1
DMAD
HMVLM
Oct-DDcc Dec-
C1 C4 DrhweMM
C4 C5 Tel. Drive
2026
C5 Resson2Driree
Sul.-Sop.
TTink-Drive
C2 Hybrid Reesoning
Contrlibble Traffic Simutation
C4 DIL
Sewerit
Mar.
CI C4 C4-VLM-DD
Jul.-Sep Sep.
02
C3/OCCVLA
Jan- MMn.
2025
01 04 CS GoT-ALMATT
Orive lutmnon
DrVingg with Segulation
[d CS. Agentccnn
C4 Dognins
C5 07 WDMD Seasoning
Oct.-Dec.
DPT--rive
ORION
OrtwoOPT4
C4 C5 PARD OUT
2023
2024
Apt.-Jun.
G2 ARGUS
Optinizing for Safety
C7 Conacle Scone Seulzion (ChutSir)
Figure The chronological evolution of major methods in autonomous driving. This timeline highlights the rapid progression of the field and provides historical context for the thematic complexity laxonomy introduced in Sec. 5.2
explicit signals to guide the visual attention of the model, improving performance in tasks that require precise visual grounding (C1, C2).
5.2.2 Reasoning-Informe Prediction
In the context of AD, prediction involves forecasting the future behavior of other traffic participants. Current research in this area increasingly focuses on enhancing robustness in long-tail scenarios, augmenting data with domain knowledge, and ensuring that predictions align with safe decision-making.
Several distinct research strategies highlight this trend. One strategy focuses on improving predictive robust- ness situations with sparse or uncertain observations. For example, as shown in Fig. 6 (b), TRACE (Puthu- manaillam et all, 2025) improves behavior prediction by using Tree-of-Thoughh (ToT) and counterfactual criticism, which allows the model to generate and evaluate multiple hypothetical reasoning paths (C2, C4. C5). Another strategy involves injecting explicit domain knowledge into the models.
Li et al. 2024ch accomplishes this by generating millions of question-answer pairs based on traffic rules, thereby enriching the data resources for both prediction and decision-making (C5, C7). A third direction seeks to ensure consistency between prediction and subsequent planning.
5.2.3 Reasoning in Planning and Decision- Decision-Making
This category reviews works that focus specifically on the core action-generation module of the autonomous driving stack, which is responsible for planning the ego-vehicle's trajectory and making tactical decisions. Research in this domain has evolved significantly beyond traditional trajectory optimization. The central goal to create reasoning process that is not just optimal, but also compliant with external rules, coherent in its internal logic, and adaptive an open- world environment. We organize our review around these three central themes.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Perception
Eavironment Understanding Module
Prediction
End-To-Ead Agents
c Tmmakd: 25 Froees
Pemplie Quaria
lisse
Quatixn
TRACE:
VLM
Critic
Vioee
Visien Space
Moran Tort:
Il pedection wanting io king. makes
righ tan. Pleare reciaii coxicen.
Stitor wdate baodbock suppod... uove stright
Text
Lar Languey Model
Look
shoad, plaae with care
Vileod 2015 waile latillnck stapped...
Behavior hypotheses
Thi Token
da Ressening Space
move imaigt anar, pleas with car.
p0ues: pecemnta worring is welling. moko
Trace enhances the prediction capabiliy for the future behaviors of other enities through an iterative reasoning framework.
Seee Eecrri den
fright tem Moac reman custers.
[a] entiarices the cappabiltty capturing details across more scales,
ORION enhances the closed loop score.
Fnddy
Supporting & Auxiliary
Action-generaii Module
(e) TeLL-Drive enhances the robustness of the learned agent and strengthens its capability of transfariigg knowlediee
[c) Driving with Regulation enhances the complance of decisions.
Figure 6: Among various existing methods, we select and present five cases that enhance capabilities across different dimensions in AD system: (a) HiLM-D (Ding al.) 2025a) focuses
Geasoning-Enhancee Prediction; (c) Driving with Regulation (Cai et al. 2024) focuses on Reasoning in Planning and Decision-Making: (d) ORTON Fuu
Perception; (b) Trace (Puthumanaillam et al 2025 focuses on Reasoning-Infornn
et al. 2025) focuses on Holistic Architectures; (e) Tell-Drive (Xu et al. 2025a) focuses on Reasoning in
Supporting and Auxiliary Tasks.
Embedding External Constraints for Compliant Behavior. A primary research thrust is the explicit integration of external rules and safety norms into the reasoning framework. This marks paradigm shift from optimizing for pure performance to optimizing for compliance and social acceptability. These approaches ensure that an agent's 's behavior is grounded in the complex realities of the road. For instance, some methods directly tackle legal and safety verification. As shown in Fig. (c), Driving with Regulation (Cai et al., 2024)
integrates a regulation retrieval mechanism with CoT reasoning, enabling the system to generate decisions that are certifiably compliant with local traffic laws (C4, C6). Optimizing AD for Safety (Sun et al.) 2024) combines RLHF with L.Mss, using physical and physiological feedback to train agent that prioritizes human safety (C6, C7). Poutine (Rowe et al. 2025) utilizes sagaa-llanggggrrrjjctttooo pre-training and preference-based reinforcement learning to aligu the model's predictions with established driving preferences and safety regulations (C2, C5). Extending beyond hard constraints, other works focus on social compliance. Drive Like a Human (Fu et al. 2024) emphasizes that systems should exhibit human-like driving styles and be able to interact naturally with their enviromment, addressing the crucial challenges of social interaction and long-tail events (C5, C7).
Ensuring Internal Coherence through Structured Reasoning. Another body of research focuses on ensuring consistency between the model's high-level semantic reasoning and final low-level executable plan. This addresses the critical challenge of reasoning-action alignment. Chain-of-Thought has become a cornerstone technique for achieving this coherence by making the "thought process" explicit. Works like CoT-VLA (Zhao et al., 2025). Think-Driver (Zhang et al., 2024b), and PKRD-CoT (Luo et al. 2024) incorporate explicit rcasoning chains to generate transparent and justifiable driving instructions, directly improving interprelability and the alignment between reasoning and action (C1, C4, C5). To further ground this reasoning process, many methods utilize structured representations. For example, TOKEN (Tian et al,, 2024a) dccomposes scenes into object-level tokens, providing a scaffold that helps the LLM apply its common- sense knowledge long-tail planning scenarios (C1, C4, C5). HMVLM (Wang al. 2025a) employs a multi- stage CoT process that moves from scene understanding to decision-making to trajectory inference, ensuring
15

---

Published in Transactions on Machine Learning Research (03/2026)
logical flow (C4, C5). A variety of VLM-based architectures also contribute to this theme. For instance, VLM-AD Xu et. al. 2024a uses a VLM as teacher to provide rich reasoning labels, while DriveVLM (Tian et all 2024c and Dolphins (Ma et al. 2024a use hybrid systems and multimodal CoT to achieve human-
adaptability. Finally, reinforcement learning (RL) is increasingly used to fine-tune this alignment (C1, C3, C4, C5). VLM-RR. (Huang et al.,
2024 uses a pre-trained VLM to generate dense reward signals, while Li et al. 2025ah connects CoT reasoning to the RL policy to improve decision quality (C3, C4).
Drive-RI R1
Achieving Open-World Adaptability through Learning and Interaction. Furthermore, recent works are advancing planning and decision-making models from closed systems to open agents that can learn from experience and interact with external knowledge. This pushes the boundaries of adaptability, especially for handling novel siluations. DiLu (Wen al. 2024) pioneers this direction by enabling model to accumulate experience and correct past mistakes through internal memory and reflection mechanisms (C1, C4, C5). This transforms decision-making from scries of independent, one-time inferences into a dynamic process of continuous learning, Pushing this concept further, AgentThink (Qian et al., 2025 trains an agent to dynamically call external tools (e.g., calculators, search engines). This significantly enhances the depth and feexibility of its reasoning process, allowing it to solve complex problems that require external knowledge or precise computation, thereby evolving the agent from a "closed- world planner" to an "open-wort world problem solver" (C1, C4, C5).
5.2.4 Holistic Architectures: Integrated and End-to-End Agents
This category reviews works that move beyond enhancing individual modules to architecting more holistic intelligent agents. The primary motivation is to overcome the information bottlenecks and error propagation inherent in traditional sequential pipelines. Research in this area follows a trajectory that begins with the tight coupling of key modules, progresses toward fully unified end-to-end reasoning systems, and finally expands the scope of what a holistic agent can do.
Integrating Perception and Decision-Making. architectures that directly integrate perception and decision-making modules. These works reframe visual reasoning not as passive representation learning task but as an active, decision-oriented explanatory process. For instance, some research focuses on breaking through visual reasoning bottlenecks in complex scenarios. DriveLM (Sima et al., 2024 uses a graph-structured VQA datasct to handle complex scene interactions, while CoT-VLM4Tar (Ren el al. 2025) employs CoT reasoning to resolve abnormal traffic situations like phantom jams (C1, C4, C5). Other works improve the grounding of reasoning through structured multimodal inputs, Driving with LLMs (Chen et al.]
The first step toward holistic design involves creating
2024) achieves this by fusing object geometric vectors with language, while Hybrid Reasoning (Azaralza el al.] 2024) integrates mathematical and commonsense reasoning to improve robustness in adverse weather (C1, C2, C4). Collectively, these approaches aim to improve closed-loop performance and interpretability, as exemplified by X-Driver (Liu et al. 2025ch, which uses an autoregressive model to generate decision commands from vision-language inputs (C1, C4, C5).
The Evolution to End-to-Enu Reasoning Agents. The ultimate goal of this research direction is the development of fully end-to-end agents that map sensor inputs directly to control outputs. A critical evolution in this domain is the shift from opaque "black box" models to interpretable "glass box" systems. The foundation for this shift is often laid by new datasets, such as DriveCoT (Wang et al. 2024b), which provide end-to-end data with explicit CoT labels (C4, C5). Building on this, architectures like GPT-Driver (Mao gt all 2023 and DriveGPT4 (Xu et al. 2024b) have pioneered the glass-box concept by generating explicit natural language justifications alongside vehicle actions (C4, C5). These models must also handle diverse inputs. EMMA (Hwang et al. 2025), for instance, fuses non non-perecptual inputs like navigation instructions with visual data, though this highlights the trade-off between reasoning depth and real-time performance (C1, C3, C5). Perhaps the most critical technical challenge is ensuring consistency between high-level semantic reasoning and low-level numerical trajectories. As shown in Fig. 0 (d), ORION (Fu et al., (025) directly tackles this semantic-numert alignment gap with generative planner, improving loop scores and woal-world applicability (C3, C4).
16

---

Published in Transactions on Machine Learning Research (03/2026)
Expanding the Scope of Holistic Reasoning. Beyond single-agent architectures, cutting-edge research is expanding the scope of reasoning to handle more complex interactions and scenarios. One major fron- tier is shifting the focus from performance in average scenarios to robust reasoning in extreme conditions. OmniDrive (Wang et al. 2024a) exemplifies this by using a dataset built on counterfactual reasoning to evaluate decision-making in rare, long-tail events, pushing the field to address the most difficult cases (C1, C3, C5). Another frontier involves expanding from single-vehicle intelligence to collective multi-agent in- telligence. V2V-LLM (Chiu et al. 2025) introduces vehicle-to-vehicle question-answering framework that enables inter-vehicle perceptual fusion and cooperative planning. This represents critical step towards addressing complex social interaction challenges and realizing the potential of swarm intelligence on the road (CI, C5, C7).
5.2.5 Reasoning in Supporting and Auxiliary Tasks
Beyond the core driving pipcline, reasoning is also being applicd to crucial supporting tasks that form the de- velopment and validation ecosystem. These elforts are critical for creating robust. and scalable solutions. The research in this area can be broadly categorized into innovations in learning paradigms and the development of reasoning-driven tools for data gencration and analysis.
Learning Paradigms. Some research focuses on improving the training process itself, aiming to make learning more efficient and gencralizable. As shown in Fig. (c), TeLL-Drive (Xu et al.) 2025a) introduces a teacher-student framework where a powerful leacher LLM guides a more compact student deep reinforcement learning (DRL) policy, This approach improves the robustness of the learned agent and enhances its ability to transfer knowledge across different driving conditions. Such innovations address the core challenges of ceasoning-decision alignment (C4) and long-tail generalization (C5) by focusing on the scalability of the training paradigm itself, rather than just the model architecture.
Reasoning-Driven Simulation and Analysis. A major focus in auxiliary tasks is the generation and analysis of high-fidelity, diverse, and controllable scenarios, which are essential for safely testing agents in rare or dangerous situations. Several works leverage the generative capabilities of LLMs to create rich simulation environments. For instance, Editable Scene Simulation (ChatSim) (Wei et al., 2024c) allows developers to use natural language to edit and compose realistic 3D scenes for simulation training (C3, C5, C7). Similarly, Controllable Traffic Simulation (Liu et al. 2024c) uses hicrarchical CoT reasoning to generate complex and controllable traffic scenarios (C3, C5, C7). In a related direction, other tools use reasoning for offline data analysis. Crash Severity Analysis (Zhen et al. 2024) applies CoT reasoning to assess traffic accident reports, providing deeper, interpretable insights into event causation and severity (C4, C5). VLAAD-TW (Kondapally et al., (026)) introduces a spatiotemporal vision-language reasoning framework and dataset to enhance autonomous driving understanding under transitional weather, serving auxiliary reasoning support beyond core perceptiion/prddicee modules.
5.3 Discussion and Insights: Prevailing Trends
Prevailing Trends. This review of system-centric approaches reveals several dominant trends in the effort to integrate reasoning into autonomous systems:
Holistic Architectures: A clear trajectory exists away from optimizing isolated modules and toward designing holistic, reasoning-centric architectures. Techniques like CoT are applied across the entire stack to serve a cognitive backbone.
Internal Coherence: A strong focus is placed on ensuring internal coherence. CoT and structured representations are widely adopted to bridge the gap between high-level semantic decisions and lowelevel executable actions, a direct response to the Decision-Reality Alignment (C4) challenge. Interpretability: An architectural shift is underway from opaque "black-box" models to inter- pretable "glass-box, box" systems. These systems provide explicit justifications for their actions, ad- dressing demands for transparency in the Social Game (C7).
17

---

Published in Transactions on Machine Learning Research (03/2026)
Open-World Adaptability: An emerging trend targets open-world adaptability. Methods that incorporate memory (e.g,, Dit.)) or external tools (e.g., Agent signify a move from "closed- world planners" to "open-world problem solvers" capable of handling novel long-tail scenarios (C5). Compliance over Optimization: A significant paradigm shift involves prioritizing compliance over pure oplimization. growing body of work explicitly embeds external constraints, such as traffic laws (C6) and social norms (C7), directly into the decision-making framework.
Design Principles. Viewed through the cognitive hierarchy, recent methodological advances can be interpreted as strengthening differentiated reasoning layers. At the Ego- centric Level, works such as HiLM-D (Ding et al. 2025a), TOKEN (Tian et al., 2024a), HMVLM (Wang et ale 2025a), and ORION (Fu et al. 2025) enhance internal coherence and decision-reality alignment through object-centric representations, structured reasoning, multi-stage inference, and semantic-trajoetorry alignment. At the Social-Cognitive Level, approaches including Driving with Regulation (Cai el al. 2024), Optimizing AD for Safety (Sun et al. 2024), and Drive Like Human (Fu et al. 2024) incorporate rule grounding, risk awareness, and socially consistent interaction. Together, these developments shift the focus from isolated perception improvements toward structured reasoning across cognitive levels.
Cognitive Hierarchy-Inffrman
From the perspectives of training and system design, cognitive theory suggests three principles. First, hi- erarchical capability allocation: low-level sensing and control modules maintain determinism and physical verifiability, while high-level modules handle rule retrieval, intention modeling, and conflict arbitration, im- proving system stability and controllability. Second, risk-sensitive objective design: asymmetric penalties for high-risk errors align training objectives with socictal safety expectations rather than solcly trajectory accu- racy. Third, dual-process archilectures: combining fast reactive modules with trigger-based deep reasoning modules enables high-level inference only under uncertainty or rule conflicts, balancing real-time constraints and docision quality.
In conclusion, the integration of reasoning has injected a powerful new paradigm into autonomous driving research, opening promising avenues for solving long-standing challenges in long-tail scenarios and social interaction. However, the true viability of these system-centric innovations hinges on the ability to rigorously and reliably measure their performance. The development of such evaluation methods is therefore critical and parallel research frontier, which we explore in the following section.
6 Evaluation-Centicc Practices: Benchmarks and Datasets
Complementing the system-ccntricc innovations discussed previously, this section examines the parallel and equally critical frontier of evaluation. Progress in developing robust reasoning agents is intrinsically tied the methodologies used to validate their performance. In the domain of autonomous driving reasoning, the construction of benchmarks and datasets is not merely supporting activity; it constitutes critical rescarch area that delineates the boundaries for model learning and ensures rigorous scientific validation. This section provides a systematic review of these evaluation-centric practices, organized by the increasing complexity of the capabilitios they are designed to mcasure.
6.1 A Thematic Taxonomy of Evaluation Practices
Traditional evaluation metrics for AD, such as collision rates or trajectory error, are adept at quantifying physical performance but are insufficient to assess the cognitive capabilities of reasoning- basod agent. These metrics can confirm what system does, but often fail to reveal why it makes a certain decision or whether its underlying reasoning is sound. To address this evaluation gap, new generation of benchmarks and datasets is emerging that prioritizes the measurement of cognitive skills over purely physical outcomes.
This section organizes a review of these evaluation-centric practices. To provide historical context for this review, Fig. ? chronologically marks the release of each major benchmark, offering a visualization of the rapid evolution of this rescarch domain. Our taxonomy organizes the landscape into a thematic progression of increasing complexity. We begin with benchmarks designed to evaluate foundational cognitive abilities (Sec. 6.1.1.. We then examine flatforms that assess system-level and interactive performance in dynamic
18

---

Published in Transactions on Machine Learning Research (03/2026)
Legand
Social Comprehension and
WOMD
Scattl Comprehetcianant
Rasonale Gencration
Closed-Loop Validrrion
02
04
06
07
BenchZABVLM Barch
Evaluating Cenerttzatin the Lang Ve Veilyinng System Trusiwor thiness and Sulaty
D2 C5 CCB
COOA LM Berch
Juli-Sep.
Cauoel and Logizal Reazzetig
Nutrorgs
MMM
04 CriveLM-Sench
Jan.Mar.
D2 CS CB) DrixcBonch- Bonch
C1
C4 C5
Rearon/Erttee-nn denno
C2) SURDG-Bench
LingoA Banch
Apr--Jun,
Jull-Sep.
01
06 CrmiiDrivee
C4 CB C7
Rannt Tell
2025
C1 C2 Nesooneee Spariaa------------- Berch
ScSonnes
STSBanch Banch
NuSonnwe- Rench
Jan,
Drixskction
C| C2 CS 06 SeasTreNV
Oct--Dec
DrivMiLM
OrvingDoge
Apr-Jun.
2023
2024
04 08 LUMPIcL-uncc
Figure 7: The chronological evolution of benchmarks and datasets for autonomous driving reasoning, This timeline illustrates the rapid acceleration of evaluation-centric research and provides the historical context for the thematic taxonomy discussed in Sec. 6.1
closed-loop environments (Sec. 6.1.2 Finally, we review benchmarks that focus on robustness and trustwor- thiness, specifically by stress-testing agents in long-tail and safety-critical scenarios (Sec. 6.1.3 Throughout this taxonomy, systematically connect the discussed benchmarks and datasets to the core challenges (C1-C7) identified in Sec.
6.1.1 Evaluating Foundational Cognitive Abilities
The ability perform complex actions in the real world is predicated on set of foundational cognitive skills. We review benchmarks designed to measure these competencies, the essential precursors to intelligent driving behavior. These evaluations typically operate in an open-loop question-anssering format, focusing on what the agent understands about a static or prerecorded scene rather than how it acts within it.
Spatial Comprehension and Scene Understanding. This category of benchmarks evaluates an agent's 's ability to accurately perceive and model three-dimensionnal spatial relationships, a fundamental requirement for any physical agent. As a pioneering work, NuScencs- QA--enQttteeeee (Qian large-scale VQA benchmark for autonomous driving shown in Fig. has inspired family of more advanced evaluations. For instance, NuScenes-Mor MQA [Inoue al. 2024
2024) established the first
aa) (C1, C5). This foundation
scales the volume of questions (C1), while -asttenns-SSpatailAA---- -eBnchhh (Tian et al. 2025b) focuses specili- cally on complex spatial reasoning by generating answers directly from ground-truth LiDAR data, thereby mitigating the impact of perception model errors (C1, C2). To further refine this evaluation, SURDS- Bench (Guo et al., 2025 systematically assesses fine-grained spatial capabilities across distinct categories such as orientation and depth (C1, C2). This area is also complemented by other specialized benchmarks, including Road VQA [Tom et al.) 2023) for scene text comprehension (C1, C2, C5, C6) and STSBench- Bench (Fruhwirth-Reisine) et al. 2025 for evaluating spatiotemporal reasoning in complex multi-agent interactions (CI, C5, C7).
Causal and Logical Reasoning. Moving beyond spatial awareness ("what" and "where")), this class of benchmarks assesses an agent's ability to perform multi-step and logical inference to understand "how" and "why"" events unfold. The primary methodology here is evaluating explicit reasoning chains. For example, Drinch.M-Bench (Sima al. (202)) adopts a graph-structured VQA framework that spans the entire pipeline from object detection planning decisions (C1, C4, C6). Similarly, Reason2Drivo-Bort Bench (Nie al., 2024) provides over 600, 000 video-text, pairs that explicitly decompose the driving process into perception,
19

---

Published in Transactions on Machine Learning Research (03/2026)
(a) Leene-Level Spatial Reasoning
(b) Holistic Rcasoning & Planning
Ousniirriee
Btke
standing
Irat Lidi Arvalle
Skeakited Vrajorery
Seop Scene Gayph
Nign Conalike
hrovlee
Map blewes
Ansel Emnjenker
Itack figh1
Withc FiCE
back
HRpppe
pedeerien
M
Generatis
Inslicerre
Slandisa
Quemsoo Eempline Condmicccn
Wher the wiic aipe Are Uleraryy any SA22-40225 tod sto-ol (o
Scuoc Desriitien
Greara
Crastrtionaa
Destie realing
Coeateliinall Romsndng
Interactive behavior Modeling
(d) Safety-Critical Reliability Evaluation
DejsingDojo
Perception
Prediction
Drive Bench
MullS-agen
Open-wockl Knowlelge
Interarzion
Responee
World Moddl
Planning
Figure 8: Among various existing benchmarks, we highlight four widely used cases: (a) NuScenes- QA for scene-level spatial reasoning (Qian et al. 2024); (b) OmniDrive for holistic reasoning and planning (Wang)
gt al. 2025b;; (c) DrivingDojo for interactive behavior modeling (Wang et al. 2024e); (d) DriveBench for safety-critical reliability evaluation (Xie et al. 2025
prediction, and reasoning steps (C1, C4, C5). Other datasets, like DriveLMM-ol (Ishaq et al. 2025 (CI, C4, C6, C7) and question-answer pairs focused reasoning about interactions governed by traffic rules and human intentions. DrivingVQA (Corbière et al. 2025 specifically tests rule-based logical reasoning by deriving questions from official driving theory exams (C1, C2, C6, C7). More advanced forms of reasoning are also targeted. As shown in Fig. g (b), OmniDrive (Wang et al. 2025b) employs counterfactual reasoning to link plans with explanations (C1, C4, C5, C6), while Rank2Tell (Sachdeva et al. 2024) is designed for multimodal importance ranking and ratilnale generation (C1, C4, C6, C7). Collectively, these benchmarks Lest an agent's 's logical consistency and capacity for causal explanation.
(Li et al.) 2024b) (C1, C4, C5, C6, C7), provide millions of annotated
Social Comprehension and Rationale Generation. This final category of foundational benchmarks evaluates the highest level of cognitive skill: understanding the intentions of other agents and justifying the ego-vehicle's own actions. These capabilities are crucial for ensuring social compliance and building human trust (C7). LingoQA-Bench (Marcu et al. 2024) provides nearly 420, 000 action-justification VQA pairs, requiring a model not only to predict an action but also to explain the reason behind it (C4, C7). Similarly, DriveAction--encch (Hao et al. 2022) employs a tree-structured evaluation framework to assess whether a model's decisions are both human-like and logically sound (C1, C2, C5, C6, C7). These benchmarks collec- tively push models beyond being passive observers of the environment towards becoming active, explainable, and socially aware participants in traffic.
6.1.2 Evaluating System-Level and Interactive Performance
While foundational benchmarks are essential for assessing cognitive skills in isolation, the true viability of a driving agent can only be determined by evaluating its performance "in the loop," " where its actions have direct and immediate consequences. This section reviews the platforms and datasets designed for this stage of validation.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Closed-Loop System Validation. System-level evaluation ensures that agent's high-level reasoning translates into physically executable and effective low-level control. This directly tests Decision-Reality Alignment (C4). Benchmarks in this category provide comprehensive long-duration data that enable valida- tion of the entire system stack, from perception to action. An example is DriveMLM (Guo et al.) 2024). which offers a large-scale dataset of 280 hours of driving data with multi-modal inputs (CI, C4, C6). LaMPilot- Bench (Ma et al. 2024b) integrates large language models into autonomous driving systems by generating executable code as driving policies, evaluating language-driven autonomous driving agents (C4, C6). The explicit goal of such datasets is to provide the necessary resources to bridge the critical gap between abstract, language-based decisions and the continuous control signals required for safe driving.
Interactive and Critical Scenarios. Beyond ensuring internal coherence, robust evaluation must also assess how the agent performs when interacting with other dynamic agents, especially in safety-critical situations. The focus here shifts from general driving to the agent's 's ability to adapt and react under pressure. Bench2ADVT.M-Bene
(Zhang et al., 2025a) exemplifies this approach by offering a dedicated benchmark specifically designed for closed-loop testing in 220 curated "threat-critical" scenarios (C1, C2, C4, C5, C6, C7). This scenario-based methodology, often operationalized within interactive simulation platforms, is essential for rigorously assessing a model's performance and adaptability when safety is the line.
6.1.3 Evaluating Robustness and Trustworthiness
Building upon the assessment of system performance in standard interactive settings, this final stage of evaluation moves to the most demanding frontier: assessing an agent's behavior under stress, in the face of novelty, and against its own internal biases. These benchmarks are designed to move beyond average-case performance and probe the limits of system's 's reliability, which is a prerequisite for establishing public trust and ensuring safe deployment.
Evaluating Generalization to the Long Tail. A fundamental limitation of any data-driven model is its performance on events that are rare in its training distribution. We review benchmarks designed specifically to measure robustness against such long-tail scenarios, directly addressing core challenge for autonomous driving (C5). These datasets provide a controlled environment for testing a model's ability to generalize its reasoning to novel situations. For example, as shown in Fig. (c), DrivingDojo (Wang et al. (002e)) is an interactive dataset curated with thousands of video clips containing rare events like crossing animals and falling debris, as well as complex multi-agent interactions such as cut-ins (C2, C5, C6, C7). Similarly, CODA- LM-Bench (Chen et al. 2025) provides extensive annotations for corner cases involving abnormal pedestrian behavior and unique traffic signs (C2, C5, C6, C7). Other works, such as NuPrompt (Wu et al. 2025a) address this challenge by introducing maguage-prompter 3D object tracking, which tests and improves model's ability to generalize to unseen or rare attribute combinations (C1,C5).
Verifying System Trustworthiness and Safety. Beyond handling external novelty, reliable agent must also possess intrinsic trustworthiness. This category features benchmarks that move beyond conven- tional accuracy metrics to systematically evaluate system safety, reliability, and robustness against internal failures. This directly corresponds to the core challenges of CooepptionCognittinn Bias (C2) and Regulatory Compliance (C6). AutoTrust--Bena (Xing et al., 2025) introduces the first comprehensive benchmark for VLM trustworthiness in driving, establishing a systematic protocol that covers robustness to data corruption, fairness, privacy, and safety evaluation (C1, C2, C5, C6, C7). Complementing this, shown in Fig, S (d), CirreAction-Benc (Xie et al. 2025) focuses specifically testing the full pipeline under varicty of challenging conditions (C1, C2, C5, C6, C7). Furthermore, other datasets contribute to this theme by design. For example, DrivingVQA (Corbière et. al., 2025) (C1, C2, C6, C7) and DriveLMM-ol (Ishaq et al. 2025 (C1, C4, C6, C7) enhance safety by grounding decisions in explicit traffic rules, while to reason from ground-trute sensor data rather than relying on potentially fallible intermediate perception modules (C1, C2).
model reliability and corruption robustness by
(Tian et al.) (202b)) improves robustness by forcing models
21

---

Published in Transactions on Machine Learning Research (03/2026)
DrivaBench
mal- eoa de
OrniDrive
Coucle tanos
NuPrompt piorpra
DriveLM benchmark
Drivel of Oali
Reason2Drive (1,00d with degino ChiSl Crosa damn
COOM Do not aoe
NuScanes-OA
VLM Watorss: 2004)
DONEM Dasr
NuScenes- SpatialOA
CENERAA serongg
"sortal" Ieeawiin
NuScenes-MCA
Reeson2Drive ONCZ
LaMPilat- Bench amaation meddaard, anck dea CARLA
Capsioo
COrTE votice
CODA-LM Commeccare Ok ariing suggestions
Rank2Tell
co rotatons
Talk2Car
boens 16CC889R) commnndd coroo
WOMD-Reason Rea-oning
nuScenes
Waymo Open Motion
CARLA
CODA
AD Tasks
Figure 9: A genealogical chart of the autonomous driving benchmark ecosystem. We illustrate the derivative and inheritance relationships among key datasots, highlighting how foundational platforms (c.g., NuSconcs) have inspired a "family" specialized benchmarks designed to evaluate specific reasoning capabilities.
6.2 Comprehensive Benchmark Dataset Landscape
To synthesize the preceding categorical discussion and provide high-level, comparative view, this subsection presents resources that map the evaluation ecosystem. We have constructed a dataset gencalogy in Fig. g This chart reveals the derivative and inheritance relationships between different benchmarks, illustrating how foundational datasets like NuScenes have spawned an entire "family" of specialized evaluations. This genealogical view, complemented by comprehensive comparison table (see Table 2 in Appendix [), serves quick reference for researchers to navigate the relationships and core attributes of existing benchmarks.
6.3 Discussion and Insights: Prevailing Trends
Prevailing Trends. Our review of the evaluation--enttrcc landscape highlights several key trends that shape how the field measures progress:
From Physical Outcomes to Cognitive Processes: A clear shift is underway from metrics based on physical outcomes (e.g., collision rales, trajectory error) to evaluations targeting the cognitive process itself. Benchmarks increasingly focus on assessing the quality of rationale generation, the logical consistency of reasoning chains, and the understanding of causality, directly addressing the need for interpretability (C7).
From Static VQA to Dynamic Simulation: The evaluation paradigm evolving from static, open-loop, question-answeringg formats (e.g., NuScenes-QA) to dynamic, closed-loop, interactive simulations. This progression is essential for assessing system-level behavior and the critical Responsiveness- Reasnnssn--Raeinnn Tradeoff (C3).
Deliberate Curation of the Long Tail: There is a growing recognition that performance on "average" scenarios is insufficient. A dominant trend is the deliberate and resource-intensive cura- tion benchmarks (e.g., DrivingDojo, CODA-LM) specifically designed to test generalization and robustness rarc, safoty-critical, and Long-tail Scenarios (C5).
Emergence of Holistic Trustworthiness: Evaluation is moving beyond simple task accuracy. New benchmarks (e.g., AutoTrust-Bench) establish comprehensive protocols to measure holistic
22

---

Published in Transactions on Machine Learning Research (03/2026)
trustworthiness, including robustness to data corruption (C2), fairness, and adherence to safety and Regulatory Compliance (C6).
Cognitive Hierarchy-Inforeed Evaluation Principles. A similar stratification can be observed benchmark design. Egocentric-level datasets such as NuSccncs-QA QA (Qian et al. (024), DriveLLMBencch (Sima) et al. 2024), and Reason2Drive (Nie et al., 2024) emphasize spatial understanding, causal reasoning, and multi-ste decision consistency, targeting situational modeling capabilities. In contrast, OmniDrive (Wang et all 2025b), DrivingDojo (Wang et al.] 2024c), and Auto Autotrust-ennch (Xing et al.) 2025 focus on long- tail scenarios, counterfactual reasoning, safety robustness, and rule compliance, probing generalization and norm adherence in complex social contexts. These developments reflect a gradual shift from purely physical metrics toward layered cognitive capability assessment.
From a benchmark construction perspective, the cognitive hierarchy provides a capability-oriented evaluation principle: separating physical feasibilityy interaction logic, and social norm reasoning into distinct layers of measurement. Such stratified evaluation reduces ambiguity in mixed metrics and enables clearer mapping between performance gains and specific cognitive improvements.
In summary, the true viability of these system-centric innovations hinges on the ability to rigorously and reliably measure their performance. The development of such evaluation methods is therefore a critical and parallel research frontier, which will be essential for guiding the field toward truly intelligent. and deployable autonomous systems.
7 Conclusion and Future Work
This survey argues that the progression toward higher levels of vehicle autonomy is fundamentally a problem of reasoning. We introduce a taxonomy that deconstruets this problem into seven core challenges (C1- (C1-C7), spanning from Egocentric Reasoning (C1 C4) to the more complicated Social-Cognitive (C5 C7) domain. Our analysis posits that while egocentric challenges represent profound engineering hurdles, the ultimate bottleneck to achieving human-like operational capabilities lies in mastering the social-cognitive challenges.
Our review reveals a clear, accelerating response to this paradigm shift. In system-centric approaches (Sec. 5 research moves decisively from optimizing isolated modules toward holistic, reasoning-cente centric architectures. The shift toward interpretable "glass-box" agents underscores this trend. In parallel, evaluation-centre practices (Sec. ( are evolving. Focus shifting from physical outcomes to asscssing cognitive processes, rationale generation, and robust.esss in curated long-tail scenarios.
Despite progress in methodology and evaluation, our analysis concludes that fundamental tension remains unresolved. This tension exists between the powerful, deliberative, but high-latency symbolic reasoning offered by large models, and the millisecond-scale, physically-grounde,, and safety-critical demands of real- world vehicle control. Reconciling these two worlds (the abstract and the physical, the deliberative and the reactive) is the central, most pressing objective for future AD rescarch. Based on the critical gaps identified in our analysis, we outline the following primary directions for future research that are essential for bridging the gap from reasoning-capable models to road-ready AD agents.
Verifiable Neuro-Symbolic Architectures. The Desponsiveness-Reassiii Tradeoff (C3) and the Decision-Ree Reaaity Alignment (C4) gap represent the immodiate architectural barriers. Future work should move beyond simple dual--pocess concepts (i.e., "fast" " and "slow" thinking). A critical frontier is developing verifiable neuro-symbolic architectures. These systems must not only provide an arbitration mechanism to manage latency but also offer formal assurances that abstract symbolic plans (e.g., "yield to the merging vehicle") are reliably, safely grounded in the sub-symbolic, continuous control space of the vehicle.
Robust Reasoning Under Multi-Modal Uncertainty. A foundational challenge remains in how rea- soning engines handle the noisy, asynchronous, and often contradictory data from heterogencous signals (C1). While many models assume clean inputs, real-world operation demands robustness to the Perception- Cognition Bias (C2). Future research should develop architectures that can explicitly reason about data
23

---

Published in Transactions on Machine Learning Research (03/2026)
uncertainty, fuse conflicting evidence, and perform compensatory inference to maintain stable world model, especially during sensor degradation or failure in adverse conditions.
Dynamic Grounding in External Regulatory Knowledge. True Regulatory Compliance (C6) on a global scale requires more than pre-trained knowledge. Current models lack mechanism adapt to the varied legal frameworks of different jurisdictions. Future work should explore "open-world" systems that can dynamically query, retrieve, and interpret knowledge bases of formal tralfic laws. A key challenge will be grounding this symbolic, legal reasoning into the agent's real-time decision-making process.
Generative and Adversarial Evaluation. Tackling Long-tail Scenarios (C5) and the methodological limitations of evaluation highlight critical need for new verification and validation paradigms. Current benchmarks, while valuable, are finite and curated, primarily testing against. "known unknowns." Future research should invest in generative and adversarial evaluation frameworks. Such systems would leverage simulation and world models not just to test agents, but to automatically discover novel, systemic failure modes, thereby moving the field closer to addressing the challenge of "unknown unknowns."
Scalable Models for Implicit. Social Negotiation. Mastering the Social Game (C7) remains a major research frontier. Current approaches in both methods and evaluation are largely nascent and often limited to explicit Regulatory Compliance (C6). The next generation of research must move beyond legal compliance to model the fluid, implicit, and non-verbal social game of human interaction. This requires developing scalable models that can infer latent human intent, negotiate multi-agent interactions, and generate behavior that is not only safe but also socially legible and predictable to other road users.
Broader Impact and Societal Considerations
AD systems may have far-reaching societal impacts beyond technical performance. While this work focuses diagnostic and methodological analysis of reasoning capabilities in AD, we briefty situate these technical considerations within their broader societal context.
A primary concern relates to safety and accountability. In real-world deployments, failures of autonomous systems raise difficult questions regarding responsibility and liability, particularly when decision- makinn processes are opaque or hard to audit. Insufficient reasoning capabilities, especially in rare, ambiguous, or interactive scenarios, can exacerbate such concerns by making system bohavior less predictable and harder to justify post hoc. From this perspective, strengthening reasoning is not only technical objective, but also important for improving transparency, traceability, and auditability (e.g., via clearer decision rationales and system-level logging) in responsible deployment.
Environmental and systemic impacts also warrant consideration. Advances in autonomous driving may lower the barrier to private vehicle usage, potentially encouraging increased car ownership and traffic volume. Even with the adoption of electric vehicles, large-scale manufacturing and energy consumption carry non-negligible environmental costs. Optimizing autonomous driving systems without considering these broader effects may unintentionally reinforce car-centric transportation paradigms rather than more holistic mobility solutions.
Although this work does not directly address policy, ethics, environmental modeling, it highlights founda- tional technical limitation relevant to many of these societal concern.. Specifically, we argue that reasoning has become an increasingly critical bottleneck within the lightly coupled slack of perception, prediction, planning, and docision-maaing in complex and long-tail driving scenarios. Addressing this limitation is im- portant for enabling safer, more reliable, and more responsibly deployable autonomous systems.. We hope this perspective encourages future research that jointly considers technical robustness, evaluation practice, and broader societal implications.
Acknowledgments
This work was partially supported by the Tsinghua SIGS KA Cooperation Fund.
24

---

Published in Transactions on Machine Learning Research (03/2026)
References
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, loamnis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Ben- jamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roclofs, Anaïs White, Anders Andreassen, Tamara von Glchn, Lakshman Yagati, Mchran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Syg- nowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/ abs/2312.11880, 2023.
Mehdi Azarafza, Mojtaba Nayyeri, Charles Steinmetz, Steffen Staab, and Achim Rettberg. Hybrid reasoning based on large language models for autonomous car driving. In ICCMA, pp. 14-22. IEEE, 2024.
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In CVPR, pp. 19107 19117. IEEE, 2022.
Claudine Badue, Rånik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B. Cardoso, Avelino Forechi, Luan F. R. Jesus, Rodrigo Ferreira Berriel, Thiago M. Paixão, Filipe Wall Mutz, Lu- cas de Paula Veronese, Thiago Oliveira-Santos, and Alberto F. De Souza. Self-driving cars: A survey. Expert Syst. Appl., 165:113816, 2021.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, abs/2502. 2025.
Macicj Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. In AAAT, pp. 17682-17680 AAAT Press, 2024.
Bikram Pratim Bhuyan, Amar Ramdane-Cherl Ravi Tomar, and T. P. Singh. Neuro-symbolic artificial intelligence: a survey. Newral Comput. Appl., 36(21):12809-122) 2024.
Daniel Bogdoll, Maximilian Nitsche, and J. Marius Zöllner Anomaly detection in autonomous driving: A survey. In CVPR Workshops, pp. 4487 4498. IEEE, 2022.
Alexandra M Boggs, Ramin Arvin, and Asad J Khattak. Exploring the who, what, when, where, and why of automated vehicle disengagements. Accident Analysis & Prevention, 136:105406,, 2020.
Frédéric Bouchard, Sean Sedwards, and Krzysztof Czarnecki. A rule-based behaviour planner for autonomous driving, In RRRR volume 13752 of Lecture Notes in Compuler Science, pp. 263 279. Springer, 2022.
Nicholas E. Brown, Farhang Motallebi Araghi, Johan Fanas Rojas, Sherif Ayantayo, Richard T. Meyer, Zachary D. Asher, Ali Riza Ekti, Chich Ross Wang, Nicholas A. Goberville, and Ben Feinberg. Evaluation of autonomous vehicle sensing and compute load on chassis dynamometer. In ITSC, pp. 1989 1995. IEEE, 2023.
Tianhui Cai, Yifan Liu, Zewei Zhou, Haoxuan Ma, Scth Z. Zhao, Zhiwen Wu, and Jiaqi Ma. Driving with regulation: Interpretable decision-mal makinn for autonomous vehicles with aarieeal-auugmented reasoning via LLM. CoRR, abs/2410.04459, 2024.
Kai Chen, Yanze Li, Wenhua Zhang, Yanxin Liu, Pengxiang Li, Ruiyuan Gao, Lanqing Hong, Meng Tian, Xinhai Zhao, Zhenguo Li, Dit-Yan Yeung, Huchuan Lu, and Xu Jia. Automated evaluation of large vision-language models on self-driving corner cases. In WACV, pp. 7826-7826. IEFE, 2025.
25

---

Published in Transactions on Machine Learning Research (03/2026)
Long Chen, Yuchen Li, Chao Huang, Bai Li, Yang Xing, Daxin Tian, Li Li, Zhongxu Hu, Xiaoxiang Na, Zixuan Li, Siyu Teng, Chen Lv, Jinjun Wang, Dongpu Cao, Nanning Zheng, and Yue Wang. Mile- stones in autonomous driving and intelligent vehicles: Survey of surveys. IEEE Trans. Intell. Veh., 8(2): 1046-1056, 2023.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with Ilms: Fusing object-level vector modality for explainable au- tonomous driving, In ICRA, pp. 14093-14100 IEEE, 2024.
Qiping Chen, Yinfei Xie, Shifeng Guo, Jie Bai, and Qiang Shu. Sensing system of environmental perception technologies for driverless vehicle: A review of state of the art and challenges. Sensors and Actuators A: Physical, 319:112566, 2021.
Hsu-Kuang Chiu, Ryo Hachiuma, Chien-Yi Wang, Stephen F. Smith, Yu-Chiang Frank Wang, and Min-Hung Chen. V2V-LLM: vehicle-to-vehicke cooperative autonomous driving with multi-modal large language models. CoRR, abs/2202..0993 09980, 2025.
Charles Corbière, Simon Roburin, Syrielle Montariol, Antoine Bosselut, and Alexandre Alahi. DRIV- INGVQA: analyzing visual thain-tf reasoning of vision language models real-world scenarios with driving theory tests. CoRR, abs/2501.04673 2501.04671, 2025.
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inferencee Exploiting large language models for interpretable logical reasoning. In ICLR. OpenReview .net, 2023.
Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileam,, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verificattin reduces hallucination in large language models. In ACL (Findings), pp. Association for Computational Linguistics, 2024.
Xinpeng Ding, Jianhua Han, Hang Xu, Wei Zhang, and Xiaomeng Li. Hilm-d: Enhancing milms with multi-scale high-rosolution details for autonomous driving. Int. J. Comput. Vis., 133(8):5379-5395, 2025a.
Yifu Ding, Wentao Jiang, Shunyu Liu, Yongcheng Jing, Jinyang Guo, Yingjie Wang, Jing Zhang, Zengmao Wang, Ziwei Liu, Bo Du, Xianglong Liu, and Dacheng Tao. Dynamic parallel tree search for efficient LLM Association for Computational Linguistics, 2025b.
reasoning. In ACL (1), pp. 11233-11222
Vinayak Dixit, Sai Chand, and Divya Nair. Autonomous vehicles: Disengagements, accidents and reaction times. PLOS ONE, 11:e0168054, 12 2016. doi: p0.187// 0168054.
Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: an open urban driving simulator. In CoRL, volume 78 of Proceedings of Machine Learning Research, pp. 1 16. PMLR, 2017.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus GrefT, Andy Zeng,, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In JCML, volume 202 of Proceedings of Machine Learning Research, pp.
PMLR, 2023.
Christian Fruhwirth-
Dusan Malic, Wei Lin, David Schinagl, Samuel Schulter, and Horst Possegger. Stsbench: A spalio-temporal scenario benchmark for multi-modal large language models in autonomous driving. CoRR, abs/
2025.
Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human: Rethinking autonomous driving with large language models. In WACV (Workshops), pp. 910-919. IEFE, 2024.
Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. ORION: A holistic end-to-end autonomous driving framework by vision-language instructed action generation. CoRR, abs/2503.19955, 2025.
26

---

Published in Transactions on Machine Learning Research (03/2026)
Amurag Ghosh, Robert Tamburo, Shen Zheng, Juan R. Alvarez-Padilla, Hailiang Zhu, Michael Cardei, Nicholas Dunn, Christoph Mertz, and Srinivasa G. Narasimhan. Roadwork dataset: Learning to recognize, observe, analyze and drive through work zones. CoRR, abs/2406.07661. 2024.
Sorin Mihai Grigorescu, Bogdan Trasnea, Tiberiu T. Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. J. Field Robotics, 37(3):362-386) 2020.
Xianda Guo, Ruijun Zhang, Yiqun Duan, Yuhang He, Chenming Zhang, Shuai Liu, and Long Chen. Drivem- Il: A benchmark for spatial understanding with multimodal large language models in autonomous driv-
ing. CoRR, abs/2411.13112 2024.
Xianda Guo, Ruijun Zhang, Yiqum Duan, Yuhang He, Dujun Nie, Wenke Huang, Chemming Zhang, Shuai Liu, Hao Zhao, and Long Chen. Surds: Benchmarking spatial understanding and reasoning in driving scenarios with vision language models. In NewrIPS, 2025.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, and Pheng- Heng. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. CoRR, abs/ abs/2309.00615, 2023.
Longzhen Han, Awes Mubarak, Almas Baimagambetor, Nikolaos Polatidis, and Thar Baker. A survey of generative categories and techniques in multimodal large language models. CoRR,, abs/ abs/25506.101/ 2025.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Rea- soning with language model is planning with world model. In EMNLP, pp. 8154-8173. Association for Computational Linguistics, 2023.
Yuhan Hao, Zhengning Li, Lei Sun, Weilong Wang, Naixin Yi, Sheng Song, Caihong Qin, Mofan Zhou, Yifei Zhan, Peng Jia, and Xianpeng Lang. Driveaction: A benchmark for exploring human-like driving decisions in VLA models. CoRR, 0hs/2500.00060 2025.
Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42((-3))3355344 346, 1990.
Jiayi Hc, Hchai Lin, Qingyun Wang, Yi R. Fung, and Heng Ji. Self-correction is more than refinement: A learning framework for visual and language reasoning lasks. In ACL (Pindings), pp. 6405 6421. Association for Computational Linguistics, 2025.
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yihun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. In NewrIPS, 2023.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning- oriented autonomous driving. In CVPR, pp. 17853-17862. IEEE, 2023.
Jiaxing Huang and Jingyi Zhang. A survey evaluation of multimodal large language models. CoRR, Abs//2408155699 2024.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In ACL (Findings), pp. 1049 1065. Association for Computational Linguistios, 2023.
Ting Huang, Zeyu Zhang, and Hao Tang. 3d-r1: Enhancing reasoning in 3d vims for unified scene under- standing. CoRR, abs/2507.23417 2025.
Yu Huang and Yue Chen. Autonomous driving with deep learning: A survey of state-of-art technologies. CoRR, abs/2006.0609) 2020.
Zilin Huang, Zihao Sheng, Yansong Qu, Junwei You, and Sikai Chen. VLM-RL: unified vision language models and reinforcement learning framework for safe autonomous driving. CoRR, abs/2412.15544, 2024.
Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong Hc, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. EMMA: end-to-end multimodal model for autonomous driving. Trans. Mach. Learn. Res., 2025, 2025.
27

---

Published in Transactions on Machine Learning Research (03/2026)
Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Trpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahm, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Reutinghouse; Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do I can, not as say: Grounding language in robotic affordances. In CoRL, volume 205 of Proceedings of Machine Learning Research, pp. 287-318. PMLR, 2022.
Yuichi Inoue, Yuki Yada, Kotaro Tanahashi, and Yu Yamaguchi. Nuscenes-mqq mqq: Integrated evaluation of captions and QA for autonomous driving datasets using markup annotations. In WACV (Workshops), pp. 930-938. JEEE, 2024.
Ayesha Ishaq, Jean Lahoud, Ketan More, Omkar Thawakar, Ritesh Thawkar, Dinura Dissanayake, Noor Absan, Yuhao Li, Fahad Shahbaz Khan, Hisham Cholakkal, Ivan Laptev, Rao Muhammad Anwer, and Salman H. Khan. Drivelmm-ol: A step-by-step reasoning dataset and large multimodal model for driving scenario understanding. CoRR, abs/ abs/2503.10621, 2025.
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, and Jiawei Han. Graph chain-of-thoughtt. Augmenting large language models by reasoning on graphs. In ACL (Findings), pp. 163-184. Association for Computational Linguisties, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NewrIPS, 2022.
Madhavi Kondapally, K. Naveen Kumar, and C. Krishna Mohan. Eyes on the road, words in the changing skies: Vision-language assistance for autonomous driving in transitional weather. Trans. Mach. Learn. Res., 2026, 2026.
Philip Koopman and Michael Wagner. Autonomous vehicle safety: An interdisciplinary challenge. TEEE Intell. Transp. Syst. Mag., 9(1):90 96, 2017.
Alexander Krentsel, Peter Schafhalter, Joseph E. Gonzalez, Sylvia Ratnasamy, Scott Shenker, and lon Stoica. Managing bandwidth: The key to cloud-assisted autonomous driving. CoRR, abs/2410.16227, 2024.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. LISA: reasoning segmentation via large language model. In CVPR, pp. 9579 9589. IEEE, 2024.
Xiao Li, Kaiwen Liu, H. Eric Tseng, Anouck Girard, and Nlya V. Kolmanovsky. Interaction-awar decision- making for autonomous vehicles in forced merging scenario leveraging social psychology factors. In ACC, pp. 285 291. IEEE, 2024a.
Xiaofeng Li, Alexander Weber, Adrian Cottam, and Yao-Jan Wu. Impacts of changing from cermissive/pro lected left-turn to protected-only phasing: Case study in the city of tucson, arizona. Transporlation Research Record: Journal of the Transportation Research Board, 2673(4):616626, 2019.
Yiheng Li, Cunxin Fan, Chongjian Ge, Zhihao Zhao, Chenran Li, Chenfeng Xu, Huaxiu Yao, Masayoshi Tomizuka, Bolci Zhou, Chen Tang, et al. Womd-reasoning: A largo-scale datasct for interaction reasoning in driving. arXiv preprint ar Xiv: 04281, 2024b.
Yiheng Li, Chongjian Ge, Chenran Li, Chenfeng Xu, Masayoshi Tomizuka, Chen Tang, Mingyu Ding, and Wei Zhan. Womd-reasoning: A large-scale language dataset for interaction and driving intentions reasoning. CoRR, /////2247.044881. 2024c.
Yue Li, Meng Tian, Dechang Zhu, Jiangtong Zhu, Zhenyu Lin, Zhiwei Xiong, and Xinhai Zhao. Drive-rl: Bridging reasoning and planning in vims for autonomous driving with reinforcement learning. CoRR, abs/2506.18824, 2025a.
28

---

Published in Transactions on Machine Learning Research (03/2026)
Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, Shouzheng Huang, Xinping Zhao, Borui Jiang, Lanqing Hong. Longyue Wang, Zhuutaao Tian, Baoxing Huai, Wenhan Luo, Weihua Luo, Zheng Zhang, Baotian Hu, and Min Zhang. Perception, reason, think, and plan: A survey on large multimodal reasoning models. CoRR, abs/2505.04922, 0 2025b.
Patrick Lin. Why Ethics Mollers for Aulonomous Cars, pp. 69 85. Springer Berlin Heidelbery, Berlin, Heidelberg, 2015.
Hanmeng Liu, Zhizhang Fu, Mengru Ding, Ruoxi Ning, Chaoli Zhang, Xiaozhang Liu, and Yue Zhang, Logical reasoning in large language models: A survey. CoRR, abs/2502.09100, 2025a.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. In TCLR. OpenReview.nef, 2024a.
Ruixun Liu, Lingyu Kong, Derun Li, and Hang Zhao. Ocevla: Vision-languaage-ccionn model with implicit 3d occupancy supervision. CoRR, abs/2500 fbs/2509.05575. 2025b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. In ECCV (47), volume 15105 of Lecture Notes in Computer Science, pp. 38-55. Springer, 2024b.
Wei Liu, Jiyuan Zhang, Binxiong Zheng, Yufeng Hu, Yingzhan Lin, and Zengleng Zeng. X-driver: Explain- able autonomous driving with vision-language models. CoRR, abs/2505.05009, 2025c.
Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning- 2025d.
chain guided segmentation via cognitive reinforcement. CoRR, abs/2503.00G
Zhiyuan Liu, Leheng Li, Yuning Wang, Haotian Lin, Zhizhe Liu, Lei He, and Jianqiang Wang. Controllable traffic simulation through llm-guided hierarchical thain-of-thought reasoning. CoRR, abs/2409.15135, 2024c.
Xuewen Luo, Fan Ding, Yinsheng Song, Xiaofeng Zhang, and Junn Yong Loo. Pkrd-cot: A unified chain-of- thought prompting for multi-modal large language models in autonomous driving. CoRR, Abs/2242200002, 2024.
Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multimodal language model for driving. In ECCV (45), volume 15103 of Lecture Notes in Computer Science, pp. 403-420. Springer, 2024a.
Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupla, Kyungtae Han, Aniket Bera, et al. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15141-15151, 2024b.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisatiwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In NewrIPS, 2023.
Ali Mahmood and Róbert Szabolesi. A systematic review on risk management and enhancing reliability in autonomous vehicles. Machines, 13(8)6646, 2025.
Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang- Yan Gui, Jan Kautz, Yu-Xiong Wang, and Zhiding Yu. Argus: Vision-centric reasoning with grounded chain-of-thought. In CVPR, pp. Computer Vision Foundation IEEE, 2025.
Jiageng Mao, Yuxi Qian, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with GPT. CoRR, abs/2310.01415, 2023.
29

---

Published in Transactions on Machine Learning Research (03/2026)
Ana-Maria Marcu, Long Chen, Jan Hünermann, Alice Karnsund, Benoit Hanotte, Prajwal Chidananda, Saurabh Nair, Vijay Badrinarayanan, Alex Kendall, Jamie Shotton, et al. Lingoqa: Visual question answering for autonomous driving. In European Conference on Computer Vision, pp. 252 269. Springer, 2024.
Francisco Matos, Jorge Bernardino, João Durães, and João Carlos Cunha. A survey on sensor failures in autonomous vehicles: Challenges and solutions. Sensors, 24(16)55108, 2024.
Francisco Matos, João Durães, and João Cunha, Simulating the effects of sensor failures on autonomous vehicles for safely evaluation. In Informalics, volume 12, pp. 94. MDPI, 2025.
Khan Muhammad, Amin Ullah, Jaime Lloret, Javier Del Ser, and Victor Hugo C De Albuquerque. Deep learning for safe autonomous driving: Current challenges and future directions. IEEE Transactions Intelligent Transportation Systems, 22(7):4316-4336, 2020.
Mohammad Naiseh, Jediah R. Clark, Tugra Akarsu, Yaniv Hanoch, Mario Brito, Mike Wald, Thomas Webster, and Paurav Shukla. Trust, risk perception, and intention to use autonomous vehicles: an inter- disciplinary bibliometric review. Al Soc., 40(2):1091-1111, 2025.
Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. In European Conference Compater Vision, pp. 292 12--00.. Springer, 2024.
Sheida Nozari, Ali Krayani, Pablo Marin, Lucio Marcenaro, David Martín Gómez, and Carlo S. Regazzoni. Exploring action-oriented models via active inference for autonomous vehicles. EURASIP J. Adv. Signal Process., 2024(1):92, 2024.
Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. CoRR, abs/2112.000114 2021.
OpenAL. GPT-4 technical report. CoRR, abs/ /bs/2233308877, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NewrIPS, 2022.
Hanh Pham and Chris Ngo. Multimodal chain of continuous thought for latent-space reasoning in vision-language models. CoRR, abs/2508. 12587, 2025.
Alice Plebe, Henrik Svensson, Sara Mahmoud, and Mauro Da Lio. Human-inspired autonomous driving: A survey. Cogn. Syst. Res., 83:101169, 2024.
Jordan Poots. RACE-SM: reinforcement learning based autonomous control for social on-ramp merging, CoRR, abs/2403.03337 03359, 2024.
Gokul Puthumanaillam, Paulo Padrao, Jose Fuentes, Pranay Thangeda, William F. Schafer, Jae Hyuk Song, Karan Jagdale, Lconardo Bobadilla, and Melkior Ornik. TRACE: A self-improvin framework for robot behavior forecasting with vision-language models. CoRR, abs/25033 200/2553300761, 2025.
Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Mengmeng Yang, Zheng Fu, Jinyu Miao, Yining Shi, He Zhe Lim, Li Liu, Tianbao Zhou, Hongyi Wang, Huang Yu, Yifei Hu, Guang Tii, Guang Chen, Hao Ye, Lijun Sun, and Diange Yang. Agentthink: A unified framework for tool-augmented thain-of-though reasoning in dision-languee models for autonomous driving, CoRR, abs/2505.152955 2025.
Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pp. 45500, 2024.
30

---

Published in Transactions on Machine Learning Research (03/2026)
Xiaoye Qu, Yafu Li, Zhaochen Su, Weigao Sun, Jianhao Yan, Dongrui Liu, Ganqu Cui, Daizong Liu, Shuxian Liang; Junxian He, Peng Li, Wei Wei, Jing Shao, Chaochao Lu, Yue Zhang, Xian-Sheng Hua, Bowen Zhou, and Yu Cheng. A survey of efficient reasoning for large reasoning models: Language, multimodality, and beyond. CoRR, abs/ ftbs/2503.21614,, 2025.
Tianchi Ren, Haibo Hu, Jiacheng Zuo, Xinhong Chen, Jianping Wang, Chun Jason Xue, Jen-Ming Wu, and Nan Guan. Cot-vimátar: thain-of-thoughle guided vision-language models for traffic anomaly resolution. CoRR, abs/2503.01632 032,, 2025.
Saki Rezwana and Nicholas Lownes. Interactions and behaviors of pedestrians with autonomous vehicles: A synthesis. Future Transportation, 4(3):722-745, 2024.
Luke Rowe, Rodrigue de Schaetzen, Roger Girgis, Christopher Pal, and Liam Paull. Poutine: Vision- tanguage-tragectory pre-training and reinforcement learning post-training enable robust end-to-end au- tonomous driving. CoRR, abs/2506.1122 11234, 2025.
Enna Sachdeva, Nakul Agarwal, Suhas Chundi, Sean Roelofs, Jiachen Li, Mykel J. Kochenderfer, Chiho Choi, and Behzad Dariush. Rank2tell: A multimodal driving cataset for joint importance ranking and reasoning. In WACV, pp. 7498-7507. IEEE, 2024.
Alexandru Constantin Serban, Erik Poll, and Joost Visser. A standard driven software architecture for fully autonomous vehicles. In ICSA Companion, pp. 120-127. IEEE Computer Society, 2018
Yinzhe Shen, Ömer Sahin Tas, Kaiwen Wang, Royden Wagner, and Christoph Stiller. Divide and merge: Motion and semantic learning in end-to-end autonomous driving. Trans. Mach. Learn. Res., 2025, 2025.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: lan- guage agents with verbal reinforcement learning. In NeurIPS, 2023a.
Noah Shinn, Bock Labash, and Ashwin Gopinath. Reflexion: autonomous agent with dynamic memory and sell-reflection. ColR,, abs/ abs/2303.1 1206.11666, 2023b.
Yuntao Shou, Tao Meng, Wei Ai, and Keqin Li. Multimodal large language models meet multimodal emotion recognition and reasoning: survey. CoRR, abs//5509.24332 2025.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European conference on computer vision, pp. 256 274. Springer, 2024.
SAE Standard. J3016 202104; taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. SAE: Warrendale, PA, USA, 2021.
Jiankai Sun, Hao Sun, Tian Han, and Bolei Zhou. Neuro-symbolic program search for autonomous driving decision module design. In CoRL, volume 155 of Proceedings of Machine Learning Research, pp. 21-30. PMLR, 2020.
Yuan Sun, Navid Salami Pargoo, Peter J. Jin, and Jorge Ortiz. Optimizing autonomous driving for safety: A hunan-centric approach with llm-enhanced RLHF. ColR,, abs/ 240000400, 2024.
Nazish Tahir and Ramviyas Parasuraman. Edge computing and its application in robotics: A survey. CoRR, abs/2507.00053 2025.
Chunlin Tian, Xinpeng Qin, Kahou Tam, Li Li, Zijian Wang, Yuanzhe Zhao, Minglei Zhang, and Chengzhong Xu. CLONE: customizing 1lms for efficient latency-aware inference at the edge. Tn USENIX ATC, pp. 563-585. USENIX Association, 2025a.
Kexin Tian, Jingrui Mao, Yunlong Zhang, Jiwan Jiang, Yang Zhou, and Zhengzhong Tu. Nusconcs-spatialqu: A spatial understanding and reasoning benchmark for vision-language models in autonomous driving. CoRR, abs//504.03164 2025b.
31

---

Published in Transactions on Machine Learning Research (03/2026)
Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. In CoRL, volume 270 of Proceedings of Machine Learning Research, pp. 3656 3673, PMLR, 2024a.
Ran Tian, Boyi Li, Xinshuo Weng, Yuxiao Chen, Edward Schmerling, Yue Wang, Boris Ivanovic, and Marco Pavone. Tokenize the world into object-leve. level knowledge to address long-tai events in autonomous driving. In CoRL, volume 270 of Proceedings of Machine Learning Research, pp- 3656 3673. PMLR, 2024b.
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevim: The convergence of autonomous driving and large vision-language models. In CoRL, volume 270 of Proceedings of Machine Learning Research, pp. 4698-4726. PMLR, 2024c.
George Tom, Minesh Mathew, Sergi Garcia-Bordils, Dimosthenis Karatzas, and C. V. Jawahar. Reading between the lanes: Text videoqa on the road. In ICDAR (6), volume 14192 of Lecture Notes in Computer Science, pp. 137 154. Springer, 2023.
Daming Wang, Yuhao Song, Zijian He, Kangliang Chen, Xing Pan, Lu Deng, and Weihao Gu. HMVLM: mul- tistage reasoning-enhanced enhanced vision- laiiual model for long-tailes driving scenarios. CoRR, abs/2506. 2025a.
Jingkang Wang, Ava. Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In CVPR, pp. 9909-9918. Computer Vision Foundation IEEE, 2021.
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and José M. Álvarez. Omnidrive: A holistic Imm-agent framework for autonomous driving with 3d perception, 01533, 2024a.
reasoning and planning. CoRR,
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and José M. Álvarez. Omnidrive: A holistic vision-language datasel for aulonomous driving with counterfactual reasoning. In CVPR, pp. 22442 -22422. Computer Vision Foundation / IEEE, 2025b.
Tianqi Wang, Enze Xie, Ruihang Chu, Zhenguo Li, and Ping Luo. Drivecot: Integrating chain-of-thought reasoning with end-to-end driving. CoRR, abs/2403.1660 16996, 2024b.
Wenshuo Wang, Letian Wang, Chengyuan Zhang, Changliu Liu, Lijun Sun, al. Social interactions for autonomous driving: A review and perspectives. Foundations and Trendso in Robolács, 10(3-4):198 376, 2022.
Xu Wang, Mohammad Ali Maleki, Muhammad Waqar Azhar, and Pedro Trancoso. Moving forward: A review of autonomous driving software and hardware systems. CoRR, abs/2411.1022 2024c.
Xuezhí Wang, Jason Wei, Dale Schurrmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR. OpenReview.net, 2023.
Yaoting Wang. Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Mul- timodal chain-of-thought reasoning: A comprehensive survey. CoRR, abs/2503.12200 2025c.
Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multinodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. CoRR, abs/2401.06805 2024d.
Yuqi Wang, Ke Cheng, Jiawei He, Qitai Wang, Hengehen Dai, Yuntao Chen, Fei Xia, and Xiang Zhang. Drivingdojo dataset: Advancing interactive and knowledge-enrrched driving world model. In NewrIPS, 2024e.
32

---

Published in Transactions on Machine Learning Research (03/2026)
Taylor Webb. Large-scale ai language systems display emergent ability to reason by analogy, 2023.
Cong Wei, Yujic Zhong, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, and Yujiu Yang. Instructsog: Unilying instructed visual segmentation with multi-modal large language models. CoRR, abs/2412.14006 2024a.
Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Jie Hu, Dengjie Li, Zheng Zhao, and Yujiu Yang, Hyperseg: Hybrid segmentation assistant with fine-grained visual perceiver. In CVPR, pp. 8931 8941. Computer Vision Foundation / IEEE, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thoughi prompting elicits reasoning in large language models. In NeurIPS, 2022.
Xiao Wei, Haoran Chen, Hang Yu, Hao Fei, and Qian Liu. Guided knowledge generation with language models for commonsense reasoning. In EMNLP (Findings), pp. 1103 1136. Association for Computational Linguistics, 2024b.
Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative Ilm-agents. In CVPR, pp. 15077-15087. JEEE, 2024c.
Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to autonomous driving with large language models. In ICLR. OpenReview.ncee net, 2024.
Gerald JS Wilde. Social interaction patterns in driver behavior: An introductory review. Human factors, 18(5):477-492, 1976.
Dongming Wu, Wencheng Han, Yingfei Liu, Tiancai Wang, Cheng Zhong Xu, Xiangyu Zhang, and Jianbing Shen. Language prompt for autonomous driving. In AAAI, pp. 8359-8367 AAAT Press, 2025a.
Haoyuan Wu, Xueyi Chen, Rui Ming, Jilong Gao, Shoubo Hu, Zhuolun He, and Bei Yu. Totrl: Unlock LLM tree-of-thoughts reasoning potential through puzzles solving. CoRR, abs/2505.12711 12717, 2025b.
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. GSVA: generalized segmentation via multimodal large language models. In CVPR, pp. 3858 3869. IEEE, 2024.
Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives. CoRR, 00ee/2501.04000 2025.
Shuo Xing, Hongyuan Hua, Xiangbo Gao, Shenzhe Zhu, Renjie Li, Kexin Tian, Xiaopeng Li, Heng Huang, Tianbao Yang, Zhangyang Wang, Yang Zhou, Huaxiu Yao, and Zhengzhong Tu. Autotrust: Benchmarking trustworthiness in large vision language models for aulonomous driving. Trans. Mach. Learn. Res,, 2025, 2025.
Chengkai Xu, Jiaqi Liu, Shiyu Fang, Yiming Cui, Dong Chen, Peng Hang, and Jian Sun. Tell-drive: En- hancing autonomous driving with teacher 1mm-guided deep reinforcement learning. CoRR, abs/ 2200225000 2025a.
Cong Xu and Ravi Sankar. A comprehensive review of autonomous driving algorithms: Tackling adverse weather conditions, unpredictable traffic violations, blind spot. monitoring, and emergency maneuvers. Algorithms, 17(11):526, 2024.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models. CoRR, abs/2501 2500//60010066, 2025b.
33

---

Published in Transactions on Machine Learning Research (03/2026)
Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P. Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M. Wolff, and Xin Huang. VLM-AD: end-to-end autonomous driving through vision-language model super- vision. CoRRi abs/ 24424411144 2024a.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan- Yoe K. Wong, Zhenguo Li, and Heng- shuang Zhao. Drivegpt4: Interpretable end-to-end aulonomous driving via large language model. IEEE Robotics Autom. Lett. 911)):8186 8193, 2024b.
Yibo Yan, Jiamin Su, Jianxiang Hc, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges. In ACL (Findings), pp. 11798 11827. Association for Computational Linguistics, 2025.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung- Chinn Lin, Zicheng Liu, and Lijuan Wang. The dawn of imms: Preliminary explorations with gpt-4v(ision) CoRR, 0bss/2309.1/444 2023.
Ruoyu Yao, Yubin Wang, Haichao Liu, Rui Yang, Zengqi Peng, Tei Zhu, and Jun Ma. Calmm-drive: Confidence-aware autonomous driving with large multimodal model. CoRR, abs/2412.04209, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NewrIPS, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR. OpenReview.net, 2023b.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, (12)::wae4403, 2024.
Yiyao Yu, Yuxiang Zhang, Dongdong Zhang, Xiao Liang, Hengyuan Zhang, Xingxing Zhang, Mahmoud Khademi, Hany Hassan Awadalla, Junjic Wang. Yujiu Yang, and Furn Wei. Chain-of-reasoning: Towards unified hematicalll reasoning large language models via a multi-paradigın perspective. In ACL (1),
pp. 24914 24937. Association for Computational Linguisties, 2025.
Junzhe Zhang, Daniel Kumor, and Elias Barcinboim. Causal imitation lcarning with unobserved confounders. In NewrIPS, 2020.
Kumpeng Zhang, Shipu Wang, Ning Jia, Liang Zhao, Chunyang Han, and Li Li. Integrating visual large language model and reasoning chain for driver behavior analysis and risk assessment. Accident Analysis & Prevention, 198:107497, 2024a.
Qiming Zhang, Meixin Zhu, and Hao Frank Yang. Think-driver: From driving-scene understanding to decision-making with vision language models. In European Conference on Computer Vision Workshop, 2024b.
Tianyuan Zhang, Ting Jin, Lu Wang, Jiangfan Liu, Siyuan Liang, Mingehuan Zhang, Aishan Liu, and Xianglong Liu. Bench2advlm: A closed-loop benchmark for vision-language models in autonomous driving. CoRR, abs/2508. 0.0028, 2025a.
Xuanliang Zhang, Dingzirui Wang, Longxu Dou, Qingfu Zhu, and Wanxiang Che. A survey of table reasoning with large language models. Frontiers Comput. Sci., 19(9):199348, 2025b.
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, and Furn Wei. LLM as a mastermind: A survey of strategic reasoning with large language models. CoRR, Abs/2444.01120, 2024c.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEFF. Trans. Pallern Anal. Mach. Intell, 1579):10755-10111 2023a.
34

---

Published in Transactions on Machine Learning Research (03/2026)
Yuxiao Zhang, Alexander Carballo, Hanting Yang, and Kazuya Takeda. Perception and sensing for au- tonomous vehicles under adverse weather conditions: A survey. ISPRS Journal of Photogranmelrry and Remote Sensing, 196:146177, February 2023b. ISSN 0924-2716. doi: 2w.1010...bppssps:2222..222 URL
http://dx.doi...oooo http://dx.0o doi orgp:///oooo//./ooooo///// 1016/] iisrrsjrrs 2022. 12.0
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. Trans. Mach. Learn. Res., 2024, 2024d.
Symbat Zhanguzhinova, Emese Makó, Attila Borsos, Ágoston P&l Sándor, and Csaba Koren. Communication between autonomous vehicles and pedestrians: An experimental study using virtual reality. Sensors, 23 (3):1049, 2023.
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, Ankur Handa, Tsung- Yi Lin, Gordon Wetzstein, Ming-Y Yu Liu, and Donglai Xiang. Cot-vla: Visual thain-of-thougllt reasoning for vision-languago-action models. In CVPR, pp. Computer Vision Foundation TEEE, 2025.
Hao Zhen, Yucheng Shi, Yongcan Huang, Jidong Yang, and Ninghao Liu. Leveraging large language models with ch and prompt engineering for traffic crash severity analysis and inference. Comput., 13(9):232, 2024.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V. Lc, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In ICLR. OpenReview.nel, 2023.
Jian Zhou, Yulong Gao, Björn Olofsson, and Erik Frisk. Uncertainty-aware decision-making and planning for autonomous forced merging. In CDC, pp. 1095-1102 IEFE, 2024.
Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving math word problems via cooperative reasoning induced language models. In ACL (1), pp. 4471 1471-4485. Association for Computational Lingnistics, 2023.
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Ste- fan Welker, Ayzaan Wahid, Quan Vuong, Vincent Vanhoucke, Huong T. Tran, Radu Soricut, Anikait Singh, Jaspiar Singh, Pierre Sermanet, Pannag R. Sanketi, Grecia Salazar, Michael S. Ryoo, Krista Rey- mann, Kanishka Rao, Karl Pertsch, Igor Mordatch, Henryk Michalewski, Yao Lu, Sergey Levine, Lisa Lee, Tsang-We Wei Edward Lee, Isabel Leal, Yuheng Kuang, Dmitry Kalashnikov, Ryan Julian, Nikhil J. Joshi, Alex Irpan, Brian Ichter, Jasmine Hsu, Alexander Herzog, Karol Hausman, Keerthana Gopalakr- ishnan, Chuyuan Fu, Pete Florence, Chelsea Finn, Kumar Avinava Dubey, Danny Driess, Tianli Ding, Krzysztof Marcin Choromanski, Xi Chen, Yevgen Chebotar, Justice Carbajal, Noah Brown, Anthony Brohan, Montserrat Gonzalez Arenas, and Kehang Han. RT-2: vision-languago-action models transfer web knowledge to robotic control. In CoRL, volume 229 of Proceedings of Machine Learning Research, pp. 2163-2183. PMLR, 2023.
Appendix
A Reasoning Paradigms in LLMs and MLLMs
A.1 Reasoning Paradigms in LLMs
To address complicated challenges in natural language processing, such as answering long-tail questions and performing lask planning [Ichter et al. 2022 Yao el al. 2023b), reasoning mechanisms in LLMs decompose problems into a series of manageable, intermediate steps. This is fundamentally achieved by prompting the model generate an explicit thought process before arriving at conclusion (Kojima et al.) (022), a tochnique pionecred by methods like Chain-of Thouugh (CoT) (Wei et al. 2022). Building upon this principle, existing reasoning paradigms can be categorized based the architecture of the generated thought, process (Zhang)
35

---

Published in Transactions on Machine Learning Research (03/2026)
et a 2025b). J. This classification reflects an evolution in methodology designed to handle increasing tasks. These paradigms are broadly grouped into three categories: sequential reasoning, which constructs a single linear path (Zhou et al. 2023 Yao et al. 2023b]; structured reasoning, which explores multiple paths in Yao et al. 2023a): and iterative reasoning, which refines a solution through
parallel (Wang et al., 2023:
leedback loops (Madaan el a. 2023 Shinn et al. 2023b
Sequential Reasoning. Sequential reasoning represents the foundational paradigm, employing a linear, umidirectional process to generate single, continuous chain of thought (Yao et al., 2023b Creswell et al, 2023). This approach emulates human-like step-by-step problen-solving and serves as the basis for more advanced reasoning structures (Nye et al., 2021). The most direct implementation is the CoT method (Wei) gt all 2022), which instructs the model to produce a linear thought process, often activated through simple prompts (e.g., "Let's think step by step") (Kojima et al. 2022) or few-shot exemplars. Subsequent research has focused on enhancing the robustness and reliability of this lincar path. Stratogies include ensemble approaches such as Self-Consistency (Wang et al.) 2023) and CoRe (Zhu et al.) (202)), which select the most consistent answer from multiple reasoning chains, and self- correction methods such as Chain of Verification (CoV) (Dhuliawala et al. 2024), which introduce an explicit step to verify and refine the generated reasoning.
Structured Reasoning. To address complicated problems that require exploration beyond a single line of thought, structured reasoning transcends the limitations of linear approaches (Wei et al. 2022 Kojima et al.
2022 by concurrently exploring multiple reasoning paths (Wang ct al., 2023: Creswell ct al.
2023). This
paradigm organizes these paths within non-linear data structures, such as trees or graphs, ena blabiing& a model
to deliberately consider, evaluate, and integrate diverse lines of thought. (Shinn et al., 2023a Hao et al.
2023 An extension of the linear chain is the tree structure, implemented in the Tree of Thoughts (ToT) framework Yao et al. 2023a), which expands single thought chain into a tree of intermediate states (Ding at all 2025b Wu et al. 2025b). foresight module then evaluates each branch to systematically navigate the solution space. To facilitate more sophisticated reasoning, graph-based structures like Graph of Thoughts (GoT) (Besta el al.] 2024) advance beyond trees by allowing paths to not only branch but also merge (Jin et all, 2024). The capacity to synthesize information from different branches provides graph- reasoning with superior flexibility for problems that demand holistic analysis.
Iterative Reasoning. In contrast to the generative and exploratory nature of the preceding paradigms, iterative reasoning introduces a feedback loop for refinement and self-correction (Shinn et al.) 2023a He
et al. 2025 Yu et al., 2025). This approach targets higher solution accuracy and robustness Yao et al. 2023a), operating on a generate-andd-reine cycle where an initial solution undergoes progressive improvement. This paradigm can be categorized by the source of feedback. The fundamental form uses an internal loop, where model critiquos and revises its own output, scen in Self-Refine (Madaan et al. 2023). A more advanced form incorporates external feedback from tools like search engines, as exemplified by the ReAct framework Yao et al. 2023b), which interleaves thought, action, and observation. A third form leverages experiential feedback for long-term improvement. Methods like Chain of Hindsight (CoHl) (Liu et al. (024a) analyze past performance on a task to generate an improved reasoning path, which is then used to guide the model on similar future tasks, establishing a meta-level learning loop (Shinn el al. 2023a
A.2 Reasoning Paradigms in MLLMs
The paradigm of reasoning enables LLMs to perform complicated logical inference (Wei et al. 2022 For this intelligence to become applicable in the physical world, however, reasoning must transcend purely linguistic symbols and integrate with sensory modalities such as vision (Zitkovich et al.] (023). MLLMs are pivotal in achieving this integration. Applying reasoning within MLLMs extends the capabilities of these models beyond abstract textual processing, allowing them to perceive and understand the physical world directly (Yang et al. 2023). By transforming raw visual pixels into semantic concepts, MLLMs facilitate complex reasoning grounded in visual information /Zhang et. al., 2024d Pham & Ngo 2025).
In the course of the development of multimodal reasoning, numerous models have emerged, establishing new paradigms for applications. For example, in reasoning-base segmentation, models no longer depend on predefined semantic labels. Instead, these models can interpret and execute complex instructions, such
36

---

Published in Transactions on Machine Learning Research (03/2026)
"segment the fruit in the image with the highest vitamin C content," task that requires both functional
understanding and world knowledge (Tai et al. 2024 Xia et al.
2024 Liu et al.
2025d 2024b
Wei et al.
2025 2024a). Furthermore, reasoning capabilities have expanded from two-dimensional visual understanding
to dineesaimenioonal scene comprehension (Hong et al., 2023 In 3D multimodal reasoning, models integrate
data from multi-view images, depth maps, and point clouds (Guo el al.) 2023 These inpuls enable the models to infer spatial structures, manage occlusions, and reason about physical fcasibility (Azuma et al., 2022). Through the incorporation of explicit spatial representations and 3D encoders into language models, these systems can handle complex physical reasoning tasks, such as "determining whether a robotic
grasp an object without colliding with others" (Zhao et al., 2025 Huang et al. 2025
Driess et al. 2023
Zitkovich et al. 2023).
Currently, cutting-edge multimodal models, including the Qwen- VL series (Bai et al. 2025 Google Gem- ini (Anil et al. 2023), and GPT-5 (OpenAT 2023), significantly advance these capabilities. The reasoning mechanisms and cross-modal spatial modeling inherent to these models are instrumental in bridging the gap between 2D perception and 3D understanding (Hong al. 2023 This progress lays the foundation for in- telligent agents capable of interacting with the physical world and comprehending complex scenes (Zitkoiich gt all 2023 Such spatial reasoning abilitics also provide crucial technical support for domains like AD, where the real-time understanding of dynamic 3D environments is essential (Xu et al. 2024b).
B Methods and Benchmark Taxonomy
The two tables below correspond to Sec. 5 and Sec. respectively, providing a quick reference for researchers.
B.1
Methods Taxonomy
Table [ summarizes all methods in Sec. in chronological order, along with their classification in Sec. 5 the corresponding challenges in Fig. and the characteristies of cach method.
Table 1: comprehensive overview and comparison of key methods for autonomous driving reasoning. This table summarizes the primary attributes of each method, including its publication Date, Affiliation, core reasoning Challenges (C1-C7) it addresses, associated Keywords.
Date Methods
Affiliation
Challenges
Keywords
Reasoning in Planning and Decision-Making
Human-like driving; Social reasoning
Drive like a human
2023.07
C5, C7
Fu et al. 2024
Holistic Architectures: Integrated and End-to-End Agents
Object language fusion; Explainable planning
Driving with LLMSs
2023.10
C1, C4
Chen al.
2024
Reasoning in Planning and Decision-Making
Multimodal CoT;
Dolphins
2023.12
C1, C4, C5
Adaptive behavior
Ma al.
2024a
Holistie Architectures: Integrated and End-to-End Agents
Language justification; Glass-box driving
GPT-Driver
2023.12
C4, C5
Mao et al.
2023
Reasoning in Planning and Decision-Makinn
Explicit CoT; Interpretable planning
Think Driver
2024.01
C1, C4, C5
Continued on next page
37

---

Published in Transactions on Machine Learning Research (03/2026)
Date
Methods
Affiliation
Challenges
Keywords
Zhang et al.
2024b)
Reasoning in Planning and Decision-Making
Memory reasoning;
DiLu
2024.02
C1, C4, C5
Self-reflection
Wen et al.
2024
Holistic Architectures: Integrated and End-to-End Agents
End-to-end CoT; Reasoning supervision
DriveCoT
2024.03
C4, C5
Wang et al.
2024b
Optimizing AD for Safety
Reasoning in Planning and Decision-Makeen Making
RLHF safety; Human feedback
2024.06
C6, C7
(Snn al.
2024
Reasoning in Planning and Decision-Making
Hybrid VIM;; Reasoning planning
DriveVLM
2024.06
C3, C4
Tian al.
2024c
Editable Scene Simulation (ChatSim)
Reasoning in Supporting and Auxiliary Tasks
Language editing; Controllable simulation
2024.06
C3 C5, C7
Wei
2024c
Reasoning in Planning and Decision-Making
Object tokens; Structured reasoning
TOKEN
2024.07
C1, C4, C5
Tian al.
2024a
Reasoning- Enhanced Perception
Reasoning dataset; Perception benchmark
Reason2Drive
2024.07
C1, C5
Nic et al.
2024
Holistic Architectures: Integrated and End-to-End Agents
Symbolic neural fusion; Robust reasoning
Hybrid Reasoning
2024.08
C1, C2
Azarafza et al. 2024)
Crash Severity Analysis
Reasoning in Supporting and Auxiliary Tasks
Accident reasoning: Causal analysis
2024.09
C4, C5
Zhen et al.
2024
Holistic Architectures: Integrated and End-to-End Agents
Multimodal GPT: End-to-end reasoning
DriveGPT4
2024.11
C4, C5
Xu et al.
2024b
Reasoning in Planning and Docision-Making
Knowledge CoT;; Robust planning
PKRD-CoT
2024.12
C1, C4, C5
Luo et al.
2024
Reasoning in Planning and Decision- Decision-Making
VLM rewards; RL. alignment
VLM-RL
2024.12
C3, C4
Huang et al.
2024
Continued on next page
38

---

Published in Transactions on Machine Learning Research (03/2026)
Date
Methods
Affiliation
Challenges
Keywords
Holistic Architectures: Integrated and End-to-End Agents
Graph reasoning;
DriveLM
2025.01
C1, C4, C5
Integrated planning
Sima et al. 2024
Reasoning in Supporting and Auxiliary Tasks
Teacher staner:-ttdet;;
TeLL-Drive
2025.02
C4, C5
Reasoning distillation
Xu et al. 2025a
Tree-of- Tree-of-Thought,
Reasoning- Reasoning-Informeel Informed Prediction
TRACE
Counterfactual
2025.03
C2, C4, C5
prediction
Puthumanaillam et al. 2025
Holistic Architectures: Integrated and End-to-End Agents
Abnormal traffie; CoT reasoning
CoT-VLM4Tar
2025.03
C1, C4, C5
Ren et al.
2025
Driving with Regulation
Reasoning in Planning and Decision-Making
Regulation grounding; Compliant planning
2025.03
C4, C6
CCai et al.
2024
Reasoning- Enhanced Perception
Teasoning-EEnaaneel
Two-stream perception;
HiLM-D
2025.03
C1, C2
Risk object reasoning
Ding et al.
2025a
Reasoning- Informed Prediction
Reasoning-Informedd
Rule grounding: Knowledge supervision
WOMD- WOMD-Reesoninn Reasoning
2025.03
C5, C7
Li et al.
2024c
Holistie Architectures: Integrated and End-to-End Agents
Knowledge supervision;
ORION
2025.03
C3, C4
Generative planning
Fu
2025
Holistie Architectures: Integrated and End-to-Fnd Agents
VLA CoT; Reason-actio action alignment
CoT-VLA
2025.03
C3, C4
Zhao et al.
2025
Reasoning- Enhanced Perception
Reasoning-Enhanced
Box-guided reasoning:
ARGUS
2025.03
C1, C2
Visual grounding
Man al.
2025
Holistic Architectures: Integrated and End-to-End Agents
Vehicle communication; Cooperative reasoning
V2V-LLM
2025.04
C1, C5, C7
Chiu et al.
2025
Holistic Architectures: Integrated and End-to-Enn tnd Agents
Counterfactual evaluation; Long-tail reasoning
OmniDrive
2025.04
C1, C3, C5
Continued on next page
39

---

Published in Transactions on Machine Learning Research (03/2026)
Date
Methods
Affiliation
Challenges
Keywords
Wang et al.
2024a.
Holistic Architectures: Integrated and End-to-End Agents
Autoregrossive decisions; Closed-loop control
X-Driver
2025.06
C1, C4, C5
Liu et al.
2025c
Reasoning in Planning and Decision-Making
Hierarchical CoT; Multi-stage reasoning
HMVLM
2025.06
C3, C4
Wang et al.
2025a
Reasoning in Planning and Decision- Decision-Making
CoT-guuded RL; Policy refinement
Drive-R1
2025.06
C3, C4
Li et al.
2025a
Controllable Traffic Simulation Liu et al.
Reasoning in Supporting and Auxiliary Tasks
Scenario generation; Language control
2025.07
C3 C5, C7
2024c
Reasoning in Planning and Decision-Making
VLM teacher; Reasoning labels
VLM-AD
2025.08
C1, C4
Xu et al.
2024a
Sansoning-Enhaneee Perception
Occupancy reasoning:
OCCVLA
2025.09
C1, C3
3D semantics
Liu et al.
2025b
Holistic Architectures: Integrated and End-to-End Agents
Multimodal fusion; Multimodal fusion
EMMA
2025.09
C1, C3, C5
Hwang et al.
2025
Reasoning in Planning and Decision-Making
Tool calling; Open-world rcasoning
Agent
2025.09
C1, C4, C5
Qian et al.
2025
Reasoning in Planning and Decision-Me Decision-Making
Preference learning:
Poutine
2025.10
C2, C5
Safety alignment
Rowe et al.
2025
MotionSemantic
Reasoning-Enhanced Enhanced Perception
DMAD
Decoupling: Structurod
2025.11
C1
Perception Learning
Shen et al.
2025)
Spatiotemporal
Reasoning in Supporting and Auxiliary Tasks
VLAAD-TW
Reasoning: Selective
2026.01
C1, C5
Attention Fusion
Kondapally et al. 2026)
40

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: A comprehensive overview and comparison of key benchmarks for autonomous driving reasoning. This table summarizes the primary attributes of each benchmark, including its publication Date, core rea- soning Challenges (C1 C7) it addresses, associated Keywords, Data Scale, and community engagement metrics HGitHub Stars). Repository Links are also provided for reference.
Challenges
Keywords
Stars
Link
Ve
C1,C2,5,06
Driving VideoQAA, Soone Text; Road Sigus
8/222 videos 10,50K
Tom
2023
Gua ct
2014
01,04,06
Closed loop Driving modal Input; Deciston Aligumcct
30 somarios camas LIDAR
driving
179
Bank/Tell
Importance Ranking Language Explanatious; Ego-centric
116 clips
20026001
C1,44C66,7
Suchieva
2001
NnSoenoeMMQA
3D VQA; Markmup-AA; Spotial Rensoming
L46 QA 34k acmarioe came
2024.01
Inoue
2024
CL
2024.02
XuSovuoss A-Beech
C1.55
Milii-madal VQA; 3D Secue Graph
34K somes / 460K QA
210
2024
LoMPilot Beach
Language Model Programs Driving Instructios: Sode-ni-Policy
4.900 scenes (300 test)
2024.06 06
04,06
Mreach
Grapá VQA, End-to-ead Auronomonu Driving
2001 frames 22,008 QA pairs
20/24009
C1,C4C6.07
1.2%
Sinna es al
2004
LingoQA Bench
Artioas/rattiiicciion VQA; End to End Benchmark
28K scemarios 419.9K QA
2024.09
Mareu et al. 2024
Rrssem2Drriv--Be
Choin-basod Ressning: Cross Thiteots
600K vidoo-tect
2022409
Nie al
2024
Driving[oooo
Driving
2024.12
C2,05,08,C7
Winng all
2024g
Trist saletyy /sobustucis privacy Tairacss
10 driving somma, 18K QA poits
9024.12
Xing al
2025
01,02,05,06,07
DrivingVQA Coobière
VQA: Chain of Thougit
3,501 amitiple choice problems
2025.01
C1,02,06,C7
2020.02
MODA-1M-Bmmch
C2,05,06,07
Cornee cusre; LVLM enaluction
9, 9,768 acmariios over 63K ammotations
NaPromos
Langtage Promptt 3D Perception; Melti-Olijpet Trucking (MOT)
40. language prompts
2025.02
C1,55
149
es
2725
avg. 7.4 tracklets per prompt
MMall
Step-hy-Step Ressening
18k train 4k text QA
2025.00
01,CL066,77
Ishog et
2025
Xußrenes Spatia A-Benhh
Spetial Reascuing
3.53 QA
20255004
Taxs et al
2025h
2025
01,05,07
Spotio temporal Bensoning
MCQs
Fruhwitth: Reisinger al
2025
DriveAction:
QA
2022506
CLCC2,C..6.CC
Hao es
2025
Drniirrvve Wang
Connterfactual reasoning,
2025.00 05
C1,CLC5,06
488
2025b
9025.07
WOMMI-Rosiming
et 3024h
01,04,05,C8.0
Interaction Reasoning; Tralli Rukes
3M QA
Dem:2AADVMM Dench
Clased loop evaluation: Dual gestemn adapsation
220 threat critioal sectacios
3025 08
01,02,00
Zaang et
20155
DriveBenchs Bench
Reiasiilty; Corruption Robustaess
2001 framees 20,,088 QA
2025.10
C1,C2,6,.06
112
Xie al
2025
SURDS Bench
Spotial Reusoning: Spatial Understandng::
41,080 QA/ 9.250 test QA
Gua ct 2025
Driving VQA
B.2 Benchmark Taxonomy
Table summarizes all benchmarks in Sec. G in chronological order, including their dataset characteristics and the corresponding challenges in Sec. In addition, some GitHub and Hugging Face links are provided at the end for quick reference and navigation of the readers.
41


Reviews:
[{"review_id": "bClQPvnwpx", "reviewer": "Reviewer_1tK1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey examines LLM/MLLM-based approaches to autonomous driving across the full pipeline of perception, planning, and action, with an emphasis on driving reasoning in complex and long-tail scenarios. It introduces a structured taxonomy, framed as a cognitive hierarchy, and summarizes seven key reasoning challenges that limit current systems, mapping representative methods to each challenge. The paper also reviews evaluation practices and benchmarks for assessing reasoning and question-answering capabilities in autonomous driving research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "From a survey paper standpoint, this work is clearly organized and easy to follow. Its main contributions are presented in a structured way:\n\n* A cognitive framing of the autonomous driving problem (Section 3). *(I personally found this section a bit hard to digest; I discuss that in the next part.)*\n* A comprehensive taxonomy of challenges (Section 4).\n* A structured review of prior work (Section 5), followed by an open discussion of remaining gaps (Section 5.3).\n* A benchmark overview using the same category labels (Section 6), along with a discussion of what future benchmarks should capture (Section 6.3).\n\nOverall, these components make the survey notably comprehensive and support the authors’ central claims.", "audience_yn": "Yes", "audience_explanation": "Reasoning for autonomous driving has become a central topic to push toward safer, higher-level driving systems. This survey offers a clear entry point for the community by summarizing key challenges and organizing current approaches, helping readers both track existing progress and identify open questions highlighted throughout the paper.", "requested_changes": "Overall, I do not think the paper requires major revisions. However, one critical point stood out to me:\n\nWhile the writing is generally strong, I only fully engaged with the paper from Section 4 onward. I struggled to see how the proposed **cognitive hierarchy** in Section 3 connects to the rest of the survey. The hierarchy is presented as a key contribution, and the authors clearly put effort into defining it, but its practical role is not sufficiently grounded in the later sections. In particular, it is unclear how this framework helps structure the challenges, guide system design, or inform evaluation. As a result, readers may not fully understand why this hierarchy matters or how it advances reasoning in autonomous driving.\n\nI would recommend that the authors more explicitly tie the cognitive hierarchy to the subsequent taxonomy, survey of methods, and benchmarks: explaining what problems it helps clarify and how it should influence future research directions.", "broader_impact_concerns": "I did not identify any issues that need a specific impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BSAuoSRkpF", "reviewer": "Reviewer_3CCa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This long survey paper examines the state of Autonomous Driving (AD) research, particularly the AI reasoning facet of AD. It is not just a survey, but rather a position paper supported by a thorough survey analysis of the literature. Rather than viewing ML models in an AD system as simple classifiers/regressors, the interest here is in the use and development of Multimodal Large Language Models (MLLMs) that can perform explicit reasoning to support vehicle control or long-term planning. The authors propose a “cognitive hierarchy” that spans different levels of intervention for the AI models, from sensorimotor control to planning around other vehicles or humans to broader accounting for the social elements of safe driving (traffic regulations, implicit human intentions, etc.), Under this hierarchy, seven grand research challengers are identified across the three levels. The role and limitations of reasoning are then discussed for each of the challenges, leading to several proposals for promising directions of AD research, Last, the role of benchmarks is discussed with an emphasis on existing data/tools and their shortcomings.\n\nAs an outsider to AD, I found this to be a very nice contribution. Insiders would probably appreciate it even more.\n\nStrengths:\n- Structure: there is no simple listing of papers in this survey. They are all categorized within the proposed hierarchy with supporting illustrations and tables that situate them relative to one another. \n- A clear position: I appreciate the opinionated perspective on offer here. While I cannot vouch for how comprehensive the survey is, I can say that writing it required quite a bit of critical thinking and exploration.\n- Limitations and future work suggestions abound in this survey. It will be very helpful to junior researchers looking to advance the state of the art, or non-AD researchers looking to test AI reasoning methods on AD benchmarks.\n\nWeaknesses:\n- Length: the survey is a bit too long and at times repetitive. Some disciplined editing could shave off a few pages without affecting the content too much.\n- The writing could be improved in places; see Requested Changes for details.\n- There is no reflection on the broader societal impact of AD; see Broader Impact Concerns for details.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This survey is supported by some 120+ references which I believe counts as accurate evidence.", "audience_yn": "Yes", "audience_explanation": "Looking at the list of papers accepted to TMLR, I can see multiple that are relevant to AD. However, none seem to be cited in this survey. It might make sense to do so in order to “close the loop” between prior AD research in TMLR, and this survey (if accepted for publication). That being said, there are definitely TMLR readers who would be interested in this paper.", "requested_changes": "Issues with the writing:\n- p1, abstract: In the second sentence, you write about \"current systems\" and \"social interactions\". What kind of systems are you talking about? AD systems? If so, I am not sure what \"social\" means in that context.\n- p1, intro: \"The primary bottleneck is shifting...\": the bottleneck in what? In the advancement of AD research? In building better AD systems?\n- conclusion: LLMs/MLLMs are qualified as “symbolic”. Are they really? The typical view in AI is that symbolic methods are things like SAT, Constraint satisfaction, AI planning, heuristic search. I wonder whether you can use another high-level descriptor here, or explain your perspective.\n\nSmall typos and the like:\n- p1: \"i.e., a deficit...\" --> \"namely a deficit...\"\n- p1: \"for the large-scale real-world deployments\" --> \"for large-scale real-world deployment\"\n- p2: fig. 1: \"intersection wants to go straight\" --> \"intersection that wants to go straight\"; \"the vehicle need to slow down\" --> \"needs to slow down\"\n- p3: fig. 2: \"on the reasoning in...\" --> \"on reasoning in...\"\n- p4: \"increasingly sophisticated\" --> \"increasin\n- p4: CoT abbreviation unnecessarily defined twice\n- p5: “numerous models emerge” —> “numerous models have emerged”; “capabilities expand” —> “capabilities have expanded”\n- p6, fig. 3: “Objects Detection” —> “Object Detection”\n- p7: “This level represents the ultimate tasks in achieving full autonomy.” —> “This level encompasses tasks that are necessary to achieving full autonomy.”\n- p8: “socially aware participant” —> “socially-aware participant”\n- p9, fig. 4: The font is way too small; please increase it so that it is legible without much zooming in/squinting.\n- p9: “encompass the spectrum of higher-level cognition” —> “encompass a wide spectrum of functions that require higher-level cognition”\n- p9: “which necessitates robust” —> “which necessitate robust”\n- p10: “a lead vehicles sudden braking” —> “a lead vehicle’s sudden braking”\n- p11: “the context-dependency of application” —> “context-dependence”\n- p12: “the system need for a robust capability….” —> “the need for system-level, robust, dynamic prioritization among conflicting regulations.”\n- p13: “(e.g. cameras,…” —> “(e.g., cameras,…”\n- p14, fig. 5: unclear what the numbers/ranges such as “1-3, 7-9..” mean. Additionally, there should be an appendix table providing direct reference to each of the methods in this figure, similar to Table 1. Currently it is difficult to map them to the References.\n- p17: “which provides” —> “which provide”\n- p17: “pioneer the “glass-box”…” —> “have pioneered the “glass-box”…”", "broader_impact_concerns": "There are valid concerns regarding autonomous driving. For instance, it is difficult to allocate responsibility in the event of an accident. Better self-driving cars encourage the manufacturing of more cars, which, even if electric, has an environmental cost to it. Although the authors focus on a technical question, reflecting on the broader implications of developing “optimal” self-driving vehicles would be valuable. I recommend this short read for a discussion of some of these concerns: https://carleton.ca/news/story/self-driving-cars-transportation-woes/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JiWmbyH2nD", "reviewer": "Reviewer_Xqg8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper argues that reasoning is the primary bottleneck for the large-scale deployment of autonomous vehicles (AVs). To support this position, the authors propose a three-level cognitive hierarchy that decomposes driving tasks by increasing reasoning complexity: (1) Sensorimotor Level, (2) Egocentric Reasoning Level, and (3) Social-Cognitive Level. Based on this hierarchy, the paper introduces a taxonomy of seven challenges in autonomous driving that require reasoning. The authors then survey existing methods that incorporate reasoning, particularly those leveraging LLMs and MLLMs, within AV systems, and review benchmarks designed to evaluate reasoning capabilities. The paper concludes with a discussion of open challenges and future research directions.\n\nKey strengths:\n- Section 5 provides a thorough overview of current approaches to reasoning in AVs, and the summary of the prevailing trends was interesting.\n- The taxonomy of reasoning challenges in AVs is reasonable.\n- Table 1 is a useful and comprehensive overview of benchmarks, and I appreciated that the authors tied them back to the 7 proposed challenges.\n\nKey weaknesses:\n- The paper is overly long, unfocused, and conceptually confusing. The proposed cognitive hierarchy is confusing and internally inconsistent (see points below for more details). Existing surveys on related topics, such as MLLMs for autonomous driving (A Survey on Multimodal Large Language Models for Autonomous Driving, Cui et al. WACV Workshops, 2025) and Chain-of-thought for autonomous driving (Chain-of-Thought for Autonomous Driving: A Comprehensive Survey and Future Prospects, Cui et al. ArXiv 2025) are more concise, better scoped, and easier to digest.\n- Most of the figures (e.g., Figures 6, 7, 8, 9, 11, 13, 14) appear to be copied verbatim from prior papers, which is not acceptable.\n- Many references are inappropriate or misleading given the context (see points below for more details).", "claims_and_evidence_yn": "No", "claims_explanation": "The central claim that reasoning is the primary bottleneck in autonomous driving is not convincingly justified, and several issues throughout the paper weaken the argument.\n\nWhile it is reasonable to argue that reasoning plays an increasingly important role, the paper does not provide sufficient evidence to support the stronger claim that perception and control are no longer dominant bottlenecks. In practice, failures in autonomous driving systems often arise from tightly coupled limitations across perception, prediction, planning, and reasoning, rather than from reasoning alone.\n\nAdditional concerns:\n- The distinction between perception and reasoning is often unclear or misleading. Reasoning is crucial within perception itself (e.g., for handling rare or long-tail objects), yet the paper treats them as largely separable components.\n- The cognitive hierarchy is confusing and internally inconsistent. Many tasks assigned to the “Social-Cognitive Level” (e.g., reasoning about agent intentions) could reasonably be classified as egocentric reasoning. The terminology (e.g., “vehicle-to-environment,” “vehicle-to-agents,” “vehicle-in-society”) further confuses rather than clarifies these distinctions.\n- Figure 1 does not clearly motivate the need for reasoning.  It’s unclear what a “Current AD System” is and why it fails on these scenarios. For example, in E2 and E3, it seems to be reasoning (“The vehicle can turn left since it’s on the major road.”, “The gap is sufficient, so the vehicle can merge to the right.”). It's therefore unclear how to distinguish the \"Current AD System\" from the \"Reasoning in AD\" system.\n- Several citations are inappropriate or misleading, including references that are unrelated to reasoning in autonomous driving or are used in a confusing context (see requested changes). This is particularly problematic given that this is a survey paper.\n- Most of the figures (e.g., Figures 6, 7, 8, 9, 11, 13, 14) appear to be copied verbatim from prior papers, which is not acceptable.", "audience_yn": "Yes", "audience_explanation": "Reasoning is playing an increasingly important role in autonomous driving systems, and AV researchers will find certain parts of this paper useful and interesting, especially the taxonomy of reasoning challenges and the section on benchmarks for evaluating reasoning capabilities.", "requested_changes": "- Clarify and refine the central claim, avoiding the unsupported assertion that reasoning is the primary bottleneck.\n- The paper should have more original figures. The majority of figures in a survey paper should not be copied verbatim from other papers.\n- Substantially shorten the paper. For example, the background sections on generic LLM/MLLM reasoning paradigms can be significantly shortened or even removed. The open questions in Section 5.3 largely overlap with the reasoning challenges (eg, Reconciling Deliberation and Reaction Time is equivalent to the Responsiveness-Reasoning Tradeoff, the Symbolic-to-Physical Grounding Gap is equivalent to the Decision-Reality Alignment, etc) and are therefore redundant. Survey of current work on reasoning-enabled AV systems does not appear until page 14, reflecting that this survey is unnecessarily long.\n- Clarify the distinction between reasoning and perception: The paper treats perception and reasoning as largely distinct, despite reasoning being essential for perception tasks such as handling long-tail or ambiguous visual inputs.\n- Rework or simplify the cognitive hierarchy, ensuring that categories are consistent and intuitively defined. For example, I would argue that some of the tasks in the “Social-Cognitive Level” (Level 3) are forms of “Egocentric reasoning” (Level 2), such as reasoning about agent intentions. Moreover, the terms “Vehicle-to-environment”, “vehicle-to-agents”, and “vehicle-in-society” only add to the confusion. Shouldn’t “Reasoning about Agent Intentions” (Level 3 task) by definition be a “Vehicle-to-agents” task (Level 2)?\n- Revise or remove confusing figures, particularly Figure 1. The wording of Example E2 is quite confusing: “There is a vehicle at the intersection wants to go straight, while the vehicle wants to go straight as well.” And why are the challenges out of order in Figure 1? More importantly, clarify the difference between a \"Current AD System\" and a \"Reasoning in AD\" system.\n- Improve overall focus, prioritizing reasoning in autonomous driving rather than providing a broad and generic overview of LLM reasoning.\n- Correct citation issues, removing irrelevant references and clarifying misleading ones. For example, the first citation (Oviedo-Trespalacios et al., 2019) has nothing to do with autonomous driving. The citation following the statement about the importance of \"robust and generalizable reasoning\" (He & Lv, 2023, page 1) has nothing to do with reasoning. (Bojarski et al. 2016) is cited following a sentence about modular pipelines, despite being one of the seminal works on E2E driving. (Rowe et al. 2025) is incorrectly classified as a reasoning for prediction method when it is a reasoning for planning method. The authors need to evaluate their choice of citations more carefully, especially as this is a survey paper.", "broader_impact_concerns": "None.", "additional_comments": "None.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)