Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
Authors: Kemal Oksuz
Alexandru Buburuzan
Anthony Knittel
Yuhan Yao
Puneet K. Dokania
Venue: Accepted by TMLR ()

Abstract: The emergence of multi-modal foundation models has markedly transformed the technology for autonomous driving, shifting away from conventional and mostly hand-crafted design choices towards unified, foundation-model-based approaches,  capable of directly inferring motion trajectories from raw sensory inputs. This new class of methods can also incorporate natural language as an additional modality, with Vision-Language-Action (VLA) models serving as a representative example. In this review, we provide a comprehensive examination of such methods through a unifying taxonomy to critically evaluate their architectural design choices, methodological strengths, and their inherent capabilities and limitations. Our survey covers 37 recently proposed approaches that span the landscape of trajectory planning with foundation models. Furthermore, we assess these approaches with respect to the openness of their source code and datasets, offering valuable information to practitioners and researchers. We provide an accompanying webpage that catalogues the methods based on our taxonomy, available at: https://github.com/fiveai/FMs-for-driving-trajectories

Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challenges
Kemal Oksuz
hemal.oksue@boschh.com
Five AT Ltd., United Kingdom
Robert Bosch GmbH
Alexandru Buburuzan
olexandru. www.buuubuuuuuuubbbbeeeaeo Merrandww..ubmnnaann@Sooccc..ooo
Five AI Ltd., United Kingdom Robert Bosch GmbH
Anthony Knittel
aathong.kmiittll@boscccccm
Five Al Ltd., United Kingdom Robert Bosch GmbHI
Yuhan Yao Robert Bosch GmbH
yuhan. Gde. bosch. com
Puneet K. Dokania
dokania..unncee@Woschh
Five AT Ltd., United Kingdom
Robert Bosch GmblI
Reviewed on OpenReview: https: openreview. net/ forun? id- E2L5J2028k
Abstract
The emergence of multi-modal foundation models has markedly transformed the technology for autonomous driving, shifting away from conventional and mostly hand-crafted design choices lowards unified, foundation--oodel-beeee motion trajectories from raw sensory imputs. This new class of methods can also incorporate natural language as an additional modality, with Vision-Langungo-Actiin serving as a representative example. In this review, we provide a comprehensive examination of such methods through a unifying taxonomy to critically evaluate their architectural design choices, methodological strengths, and their inherent capabilities and limitations. Our survey covers 37 recently proposed approaches that span the landscape of trajectory planning with foundation ion models. Furthermore, we assess these approaches with respect to the openness of their source code and datascts, offering valuable information to practitioners and rescarchers. We provide an accompanying webpage that catalogues the methods based on our (axonomy, available at: https://github. cww/fiveai//W/. fariaii/-ss-ooo---oeeeeeeeee deia-ddiving--- sebvving-trajeetoriee.
approaches, capable of directly inferring
(VLA) models
Contents
Introduction
2
I.1 Scope and Contributions
3
I.2 Comparison with Previous Reviews
5
2 Notations and Background
7
A Hicrarchical Taxonomy
8
The authors from the ADAS Systems, Software & Services Business Unit of Robert. Bosch GmbH.

---

Published in Transactions on Machine Learning Research (03/2026)
3.1
Foundation Models Tailored for Trajectory Planning
8
3.2
Foundation Models Guiding Trajectory Planning
10
4
Foundation Models Tailored for Trajectory Planning
11
4.1
1
How to Fine-tune FM for Trajectory Planning
4.2
Models Focused Solely on Trajectory Planning
14
4.2.1
Models without CoT Reasoning
15
4.2.2
Text Output for CoT Reasoning
15
4.2.3
Initial orajectory Prediction for CoT Reasoning @
4.3
Models Providing Additional Capabilities
4.3.1
Models Providing Language Interaction Capability
................................
4.3.2
Models Providing Action Interaction Capability
................................
20
5
Foundation Models Guiding Trajectory Planning
21
5.1
Models Using Knowledge Distillation Only During Training
................................
5.2
Models Using Knowledge Transfer During Inference 3
................................
23
6
How Open Are the Dataset and Code of Existing Approaches?
25
Open Issues and Emerging Directions
28
8
Conclusive Remarks
31
A Further Details on the Openness of the Methods
46
1 Introduction
Foundation models (FMs) are large-scale models that leverage vast amounts of data to learn representations that can be effectively adapted to a variety of downstream tasks. Depending on the type of data they process, these models are generally referred to by different terms. FMs that operate only on language data, such as
BERT (Devlin et al. 2019 GPT-2 (Radford et al. 2019), Chat-GPT (OpenAI 2023) and Qwen (Bai et al.
2023a Yang al. 2024a Qwen Team
2025), are known as large language models (LLMs). Alternatively,
models that process both
language and visual data are considered vision language models (VLMs), with
examples including CLIP
Radford et al. 2021 Flamingo (Alayrac et al. [022]. LLaVA (Liu et al. 2023
2024a b). GPT-4o (OpenAl 2024), Intern-V. VL CChn et al. 2024d Gao et al.
2024c
Zhu et al. 2025 and
Gemini (Gemini Team 2025). As opposed to LLMs and VLMs, which generally output language data, a class
of FMs are designed to generate images from language inputs (Saharia ct al.
2022
Pecbles & Xie 2023). or
from both language and vision inputs (Bruce el. al. 2024
Genie Team 2024
2025
Meta Chameleon Team
2025 NVIDIA 2025).
These models generally serve as backbone upon which more specialised models for various different domains are built using fine-tuning; autonomous driving (AD) is no exception. The scope of this work is to investigate the usefulness of FMs for AD. However, before delving into the AD-specific approaches, a natural question to ask would be: "Can off-the-shelf FMs, specifically VLMs, undersland driving scenarios without being explicitly trained or fine-tuned for them?", ? To answer this, we prompt GPT-4o (OpenAl 2024) with three complex driving scenes and related questions, and show the interaction in Fig. 1 The first two examples include rare driving scenes, where GPT-4o's replies are highly accurate and insightful. For example, in the first case, it not only understands the scene but also provides useful instructions on why and how one should drive cautiously in the given scenario. Similarly, in the last example, the model captures the misleading prompt and provides a reasonable explanation that promotes driving safely. Therefore, due to the scale of the architecture, training procedure, and the vast. amount of training data involved, these models have learned representations that can already understand driving scenes surprisingly well, making them a highly promising precursor to build AD-specific solutions- key driver the paradigm shift currently observed in the AD industry. Nevertheless, deploy FMs efficiently and reliably on edge devices for AD, several aspects of customising such models (e.g., chain-of-thought inference cost, scale, accessibility to open weights,
2

---

Published in Transactions on Machine Learning Research (03/2026)
For
TURN
LFT
30
there anything unusual this scene?
How can turn right while am driving on this lane?
Is it safe to drive on the road?
The image shows a road with a sign indicuting that cows may be present for the next mile. There are several cows crossing the road, which suggosts that you should exercise caution while driving. It is important to drive slouly and be prepored to stop if necessary to avoid hitting any animals. The speed limit is also reduced to 30 mph likely for safety reasons.
Yes, there is something unusual in this scene, The image shows a truck that has overteraed on the road. This is not a normal occurrence and indicates that there may have been an accident. The overturned truck is blocking part of the rood, which could cause traffic delays and pose Aazard to other vehicles. Drivers should exercise cantion and possibly seek an alternate route to avoid the obstruction.
The image shows a lane with markings indicuting that vehicles in this lane must turn left. If you need to turn right, gou should not be in this lone. Instead, you should move to a lane that allows right turns, if available. Always follow the road markings and signs to ensure safe and legal driving.
Figure 1: GPT-44'ss response to driving-related prompts on three different scenarios.
iiimiinng/iffrenceee efficiency, obtaining suitable data for fine-tuning, etc.) form integral parts of the ongoing research and design choices in this area. Through this review, we present holistic perspective these aspects, highlighting progress, limitations, and open research directions.
1.1 Scope and Contributions
FMs can be used in several ways for AD. Some methods leverage syuthetic data for training (Chi et al.
2025
Yang et al. 2025ab and evaluation (Ljungbergh et al.] 2024 Yan et al. 2025: Cao et al. 2025a for which the
models generating Al data, such as AD world models (Hu al. 2023a Wang et al.) 2024b Gao et al..
2024b
Wen et al.
2024b Zhao al.) 2025a c). can be helpful. A number of methods enhance scene understanding
and reasoning capabilities of an FM for AD using visual question answering (VQA) tasks Yang et al.
2024by
Ding et al. 2024b Ma et al.]
2024a Nie et al. 2024 Lu et al. 2025 Qian et al. 2025a Jiang et al.
2025c
Some methods also use an FM to improve specific capability of the AD model such as perception (Pan
et al.
2024b Xinpeng et al.
2025), prediction (Zheng et al.] 2024a Zhou et al., 2025a) or control (Wang
ct al.
2023b Sha et al. 2025p. Within this context,, a group of methods leverage FMs to yield textual actions,
commonly by classifying the scene into one of the predefined meta-actions, such as "go straight" and "slow
down"
[Chen et al.
2023a Fu et. al. 2024 Wang al.) 2024c Wen al.) 2024a Jiang et. al.] 2025b Ma
et al.
2024b Jin et al. 2024 Zhou et al.
2024b
Li et al.
2024a Wang et al. 2025d). Last but not least,
some models benefit from an FM to directly operate the vehicle, typically through trajectory planning Pan
lel al.]
2024a Xu et al. 2025b Tian et. al.
2024 Fu et al.] 2025a Wang el al., 2025b Hwang el al.) 2025
Renz et al. 2025). While each of the above capabilities is essential for developing a robust and accurate AD
model, trajectory planning is, arguably, the most critical task for driving, where the others serve as auxiliary functions for this primary goal. Consequently, considering the profound effect of FMs on this crucial task, in
this paper, we primarily focus how trajectory planning models for AD benefit from FMs!
Although the trajectory planning, scope also includes the models predicting outputs in different forms such as control signals. One example of this DriveGPT4 (Xu et 2024 which predicts target speed and turning angle of the autonomous vehicle.
3

---

Published in Transactions on Machine Learning Research (03/2026)
FM
Trejectone
(Nr, Nil my Ne YJll
(a) Trajectory planning directly from the sensor data
Critical Objects that Need Attention:
Red traffic light indieating that vehicles must stop.
Vehicles the infersection: monitoring their movements is important.
Melo-Actions:
PM
Speed: STOP (due to the rod traffie lights),
Steering STRAIGHT (continue stzaight the light green)
Plan the trajectory.
Trojectory [Xx, Vi)
(Xa, v.3
2
(b) Trajectory planning using CoT reasoning critical objects and meta-action
In the current scene, several important objects should be cousidered for futare ressoning and driving decisions:
Trufiic Lights: The rnd truftic light indieate that vehinles must stop and soast until the lights turn green.
Vehioles: There are multiple vehicles, including sifeer car in forrrroudd and other cars the background. Their positions and movements need to be monitored.
FM
pedostrions walling the sidewalk: the right. Their sufety and movementss should considered. Road Signs: The 'No Turn on Rod' sign on the right side of the image
Pedestrions:
What are the important objocts in the current soeme?
crucial understanding traffic rules
intersection.
Lane Markings indicate the direction of travel
Lone houndarics are essential for maintaining proper lane discipline.
(c) Trajectory planning and language Interaction
PM
Truiectory for changing lane to left: (Nx, Nal. (N Yall
2
Direct lane the left.
0
(d) Trajectory planning via action Interaction
Trajectory Planning Model
whhon intersection with traffic lights, The traffic lights red, FM
Somee Descriptton: image
indicating that vehicles should stop. There are several cars visible, including one abber car foreground that appears to be waiting the intersection.
Trajectorg (N Vi).
driving system and this is the front camera view. Provide description of the and the critical objects. Finally; provide meta-action for speed from see of ((ACCELERAT) TE, KEEP, DECELERA TE, STOP)) and one for steezing from (STRAIGHT,; LEFT. RIGHT!).
You
Critical Objects that Nend Attertion:
(x,, Vell 2
Red traflic lights: indicating that vehicles must stop.
Vehicles at the intersection: monitoring their
impertant.
Meta- Actions:
Speed: STOP (due to the red traffic lights) Steering: STRAIGHT (contimue straight light surns green)
(e) Knowledge transfer from an FM to guide trajectory planning
Figure 2: Different ways of how FMs are helping trajectory planning. While (a-d) provide examples FMs tailored for trajectory planning, (e) is an example for an FM guiding trajectory planning. The input image is from nuScenes (Caesar et al., 2020), the question is from -riveee-- (Sima et al., 2024), the user instruction is from the SimLingo dataset (Renz et al. 2025).
There are, in fact, various ways FMs can be leveraged for enhanced trajectory planning, as illustrated in Fig. 2 A common approach involves adapting existing FMs through minimal architectural modifications followed by fine-tuning them on task-specific dataset. This fino- tuning can be as simple as training the model to output trajectory directly from the sensor data (Mao et al. 2023 Zhang et al. 2024 Yuan et al.
4

---

Published in Transactions on Machine Learning Research (03/2026)
2024 Xu et al. 2025c Xie et al. 2025 Zhou et al. 2025b ( Fig. 2 2())) Additionally, some models use chain-of-thought (CoT) reasoning, a technique that helps L.Mss break down problem into multiple steps for enhanced reasoning. Fig. bbb) illustrates its common usage, in which the FM first leverages CoT in the form of critical objects and meta-actions, before producing the final trajectory. The fact that VLMs are equipped with a linguistic modality has also motivated several approaches that leverage this capability to enable language and/or action interactions with trajectory planning models (Xu et al., 2024 Sima et al. 2024 Hwang et al.) 2025 Shao al. 2024a, Renz et al. 2025). As illustrated in Fig, ((c), language interaction capability can offer reasoning behind model's behaviour, giving reassurance to users. Differently, the action interaction capability facilitates driving assistance by executing driving commands of the user (refer to Fig. 2()), similar to existing vision language action model (VLA) models (Brohan et al.) 2023 Kim et al. 2024 p. Alternatively,
there are approaches where FMs guiide an existing trajectory planning system by Transferring their kedgl
during training and/or inference (Pan et al.
2024a
Tian et al. 2024 Jiang et al.
2024
Wang et al.
2024a
Liu ct al.)
2025: Jiang et al.
2025a Guo ct a.
2025
Qian et al. 2025b Xu ct al.) 2025b Hegde ct al. 2025
Chen el. al. 2025 Han et. al. 2025 J. Fig. 2
illustrates this set of approaches with an example, where a
VLM (GPT-44 40 OpeeAA| 2024 in this case) outputs description of the scene, critical objects that the ego
should pay attention to, and a meta-action.
Although, as briefly indicated above, numerous studies benefit from FMs for trajectory planning in autonomous driving, the field still lacks a holistic understanding of the progress achieved. The sheer variety of existing heterogeneous approaches has made increasingly difficult to tell what truly distinguishes one method from another, since the underlying architectures, datasets, and training procedures vary widely and have great. impact the performance. In this work, we aim to bring structure to this fragmented landscape by systematically analysing and comparing current techniques, highlighting the architectural design choices and capabilities that influence their effectiveness, and providing unified perspective to guide further progress in applying foundation models to trajectory planning for autonomous driving. Our primary contributions, therefore, are:
1. We introduce a hierarchical taxonomy of the methods that employ FMs for trajectory planning in autonomous driving and systematically analyse 37 existing methods based on this taxonomy. Our sludy is a comprehensive e.fort to organise, interprel, and unify this repidly evolving field, where a variety of heterogenous approaches has emerged without clear benchmarking and understanding of their distinctions. We believe that our study provides a structured foundation to help the field in making methodical progress.
2.
In addition to providing practical guidance on how to tailor and to fine-tune an FM for trajectory planning and stralegies for curating datasets for different use cases, we also assess these approuches in terms of how open their code and data are, in order to give practitioners and researchers useful pointers for their reprodueibility and reuse. We also outline key future challenges and open research questions from multiple perspectives, including efficiency, robustness, evaluation benchmarks and sim-to-real transfer.
Given that pioneering works in this domain first emerged as preprints in late 2023 (e.g., GPT-Driver (Mao) ct al. 2023). DriveGPT4 (Xu et al.] 2024)), ) this review encompasses approximately two years of rescarch progress through October 2025 in leveraging FMs for aulonomous trajectory planning, Within this period, we select 37 methods published in peer-reviewed venues to provide a clear conceptual analysis and well-structured organisation. We also include a limited number of representative preprints to incorporate emergent research. For example, V2X-VLM YYou et al. 2024) represents a specialised FM for trajectory planning uniquely utilising additional infrastructure images, while FASTONAD++ + et al. 2025b) employs a FM selectively to guide trajectory planning only when the AD model has high uncertainty. We emphasise that our objective is not to compare or rank these methods, but rather to organise and analyse them conceptually.
1.2 Comparison with Previous Reviews
Considering that there are several reviews or surveys focusing on AD, we would like to mention key differences between ours and the existing ones. One particular set of these reviews focuses on AD, usually to present.
5

---

Published in Transactions on Machine Learning Research (03/2026)
Bounding buxes
Ageut coorditates
traj
Vision Encoder
Vision
Perception
Explicit laterface
Prediction
Expplicit Iuterface
Planning
Features
(a) Modular AD models
ede.
Vision Features
Fostures,
egg. Prediction Features, Ego Query
REV
Map Pootures
Vision Encoder
Lateat
Perception
Lateas Interface
Prediction
Latent Interface
Planning
O tral
Interface
(b) E2E-AD models
Legend
Vision Encoder
Vision Pearures
Vision Adapter
O test
Optional
Large
Language Model
Vision token
Text
Oual
Text token
PPon the trojectory'
Tokenizer
(c) Foundation Models for AD
Figure 3: Trajectory planning methods. (a) Modular approaches explicit interfaces between different modules. (b) End-to-End (E2E-AD) models replace explicit interfaces with latent ones, allowing all modules to be jointly differentiable.
(c) FM-based methods that follow the typical VLM pipeline. The text output of the VLM can also be used, e.g, CoT reasoning. In this illustration, we simplify the pipelines to provide a high-level overview of how these models work, yet they can include different number components and more complex connections across the modules. Input images are taken from nuScenes (Cacsar al. 2020
and discuss the state-of-the-art at the time they were published Yurtsever et al. 2020 Badue et al. 2021
Janai et al.] 2021 Tampuu et al.] 2022 Coelho & Oliveira 2022 Li et al. 2023b Zhao et al. 2024 Chen
et al.
2024a).
Some of these surveys have narrower scope such as focusing on end-to-end (E2L) trainable
models (Tmppuu et al. 2022 Coclho Oliveira 2022 Chen et al. 2024a), models using reinforcement
learning Kiran al.j 2022), imitation learning-bean based approaches (Le Mero al. 2022), or existing AD
datasets [Liu et al.] (224cc). Alternatively, rather than trajectory planning, some reviews focus on a single
auxiliary task for AD such perception (Wang et al. 2025a), occupancy prediction (Xu et al. 2025a) motion planning (Teng et al. 2023). While these are useful for their respective purposes, they do not pay special attention to how FMs can be used within the context of trajectory planning. As result, our review is complementary to this set of reviews.
Due to the common adoption of FMs in several fields, including AD, a few review papers consider how these models can help AD (Gao et al. 2024a Yang al.] (2224c Zhou et al. 2024a 2025b). Although they consider the use of FMs for AD, their scope is broader than ours. Specifically, these studies aim to cover how FMs can be useful from various perspectives, including perception, data generation, scene understanding, quite limited, whercas we focus exclusively on trajectory planning, allowing more comprchensive discussion. As one similar survey to ours, Jiang el. al. (2025d) discuss different architectural paradigms for the models built on VLMs and their chronological progress. In addition, we include knowledge transfer approaches using FMs our scope, introduce hicrarchical taxonomy over this broader set of models, delve deeper into the design choices for adapting VLM for trajectory planning and elaborate on the openness of the methods that can be useful for researchers and practitioners.
al.] 2025a Cui al.)
well trajectory planning. Consequently, the discussion trajectory planning is

---

Published in Transactions on Machine Learning Research (03/2026)
2 Notations and Background
Notations. We denote the set of observations by X, where X typically includes subset of (i) sensor readings generally from cameras, lidar, or radar, (ii) the state of the ego vehicle such as its velocity, acceleration and steering, and (iii) an indicator of the route, which can be in the form of GPS target point or high level command such as fgo straight, turn left, turn right). We use Ouraj E RExP to represent the output trajectory, such that l is the prediction horizon and p is the dimension of the output at each time step, commonly p = 2 for bird's eye view (BEV) coordinates. Considering that the models typically aim to maximise the likelihood of the data, Otraj can be assumed to be a sample from the predictive distribution of the model, Otraj ~ p(Orraj] X), where Otra) denotes the corresponding random variable. Furthermore, we use Onittraj to denote initially-predicteed trajectory to account for methods predicting
initial trajectory before refining it to obtain Otraj. T and Otext denote the input text to the FM and the output text from it, where each of them is a sequence of text tokens, i.e., T = {T*} and Otext = (Otext) u with n and m being their soquence lengths.
FMs typically generate the next token of text output conditioning on the previous tokens, known as the autoregressive (AR) generation or next token prediction. In the context of VLMs, which predict a categorical distribution over the text tokens, we formulate this as O ~ p(0 XX T. (0 (O(ex)) 1}=1) with Otext being the random variable for the i-th token. For clarity, when we refer to the entire text output, we use Otext ~ p(Otext X T). Additionally, in the case of knowledge transfer (refer to Fig. [()),, the set of observations provided to the FM can be different from that provided to the trajectory planning model for AD. For example, while the AD model may receive multi-view images, the corresponding FM may receive only the front image, potentially with additional privileged information, o.g., VLM-AD (Xu et al. 2025b). As a result, while discussing the approaches with knowledge transfer, we denote the observations provided to the FM as XFM, and corresponding latent representations as Z.
Trajectory planning for AD. The trajectory planning task for AD is typically considered as a non- deterministic environment where the world is partially observed through the sensors. Reinforcement Learning (Kazemkhani et al., 2025 and Tree-Search (Huang et al.] 2024h methods typically formulate this environment as either a Fully- or Partiialy-OOserveed Markov Decision Process (Russell et al. 1995), , and explicitly define numeric rewards and penalties to be optimisod, while considering the distribution of expected rewards to form a control policy. In contrast, Imitation Learning delegates planning strategy to teacher, where the objective of such planner is to predict how a teacher model would act, and use this prediction as a plan for operating the vehicle. Essentially, given set of observations X from the sensors, the trajectory planning task involves determining a trajectory for the ego vehicle to follow (Otrat). And, to operate vehicle, Otraj is typically propagated to a controller such as a petoottional-ntgraaa-a- Derivative (PID) controller (Bennett 2001 to obtain control signals, such as for the accelerator and steering.
Model architectures for trajectory planning can be divided into three main groups, as shown in Fig. 3 Modular approaches, shown in Fig. (a), use explicit interfaces, where each module independently aims
solve subproblem such perception (Zhu et al. 2021 Zeng et al. 2022 Liu ct al.
2022
Wang et al.
2023a
Oksuz et al. 2023 Yavuz et al. 2024 Li et al. 2025b). prediction Shi et al.] 2022
2023
Zhou et al.
2023b
Cheng et al.
2023
Prutsch et al. 2024h or planning iGao et al. 2020 Chen et al.)
2024b
This architectrre
does not allow gradient flow during training, i.e., it is not E2E-trainable, and each module is optimised for its
own objective. E2E-AD models in Fig. (b) replace explicit interfaces by latent representations from the
previous module, to optimise all modules jointly for trajectory planning (Hu et al., 2023b Jiang et al. 2023
2026
Li et al. 20224c Weng et al.] 2024 Zheng et al.] 2024b Liao et al.] 2025 Zhang al.) 2025 Song
et al.
2025). FM-based models are recent approaches that are built directly on FMs to benefit from their
vast
world knowledge Xu et al.] 2024
Zhang et al.
2024 Wang et al. 2025b Renz et al.
2025
Hwang et al.
2025
Fu ct al. 2025a;
Xic ct al. 2025
Chen ct al. 2025). Fig. D(c) illustrates a typical VLM
pipeline (Bai
el al.
2023b Li el al.
2023a Liu el al. 2023 2024b), in which a vision encoder (He et. al.
2016
Dosovilskiy
et al.
2021
Liu et al.
2021 outputs vision features, which are then projected onto the LLM embedding
space using a vision adapter, e.g., a linear layer (Liu et al. (023). Finally, an LLM processes vision and text.
tokens to yield the final trajectory. As more recent approaches, existing E2E-AD and FM--baed methods are predominantly trained using Imitation Learning.
7

---

Published in Transactions on Machine Learning Research (03/2026)
FM-based methods for AD
FMs tailored for trajectory planning
FMs guiding trajectory planning
Models focused solely on traj planning (Sec. 4.2)
Models providing additional capabilities (Sec. 4.3)
Knowledge distillation only during training (Sec. 5.1)
Knowledge transfer
during inference (Sec. 5.2)
ONo CoT
Language interaction
CarL.LaVA (Rons, et al., 2024;
Driv (Nu et nl., 2024)
VLP (Pan al., 20244)
VLM-E2E (Tán ot al., 2025)
Driv=GPT4-v22 (Xu et al., 2025c)
DriveLLM-gent (Sima al., 2024)
VLM-AD (Xu et al., 20255)
DME-Driver (Han et al., 2025)
V2X-VLM (You et al., 2024)
Emma (Huing ot al,, 2026)
DiMA (Hogdo et al., 2025)
Senne-F2E (Jiang es al. 2024)
as CoT
OpeaDeiveVLL. LA (2hon
2026)
Solve-E2E (Chen et al., 2025)
DatVLA (Jiang al., 2025a]
GPT-Driver (Mao et nl., 2023)
DIMA-MILMM (Hegdle ot al. 2025)
DriveVTM--uul (Tion ot al., 2024)
Drive
2024)
Omni-QQ (Wang al,, 2025b)
Solve-22E (Chen al., 2025)
VL. (Zhou et ul., 2025b)
Omni-T. (Wang et ol., 2025b)
D-MA-Dual (Hegrlo ot al. 2023)
RAG-Driver (Yuan
2024)
Orion (Pu nl., 2025a)
IE.-rive (Wang
2024a)
S4-Driver (Xie et al., 2025)
Action interaction
VITT-Auto (Goo et al., 2025)
Trajectory as
DriveMLM
al., 2025a)
FASIONAD (Qion
2025b)
Agent-dziver (Mao et al., 2021)
LMDriive (Shao et al., 202
AsyucDriver (Chen et al., 2021c)
(Zhnng al., 2024)
Language & action interaction
BLL (Chen et al., 2025)
Simlingo (Remz et al., 2025)
Figure 4: Taxonomy of trajectory planning methods utilising or getting help from FMs.
3 A Hierarchical Taxonomy
We now present a taxonomy to systematically study trajectory planning approaches utilising FMs. Broadly speaking, there are two main categories: FMs tailored for trajectory planning, and FMs guiding trajectory planning. However, these main categories do involve various subcategories, eventually forming a hierarchical taxonomy shown in Fig. Specifically, cach of the eight Icaf nodes in the taxonomy tree corresponds to the formulations in Fig. 5 In what follows, we discuss them in detail.
3.1 Foundation Models Tailored for Trajectory Planning
The primary characteristic of the methods in this group is that they, partly or fully, utilise existing pre-trained FMs by tailoring and fine-tuning them to trajectory planning for AD. Therefore, effectively, they build FMs that are directly used for AD use cases. Considering the auto-regressive generation of FMs and the convention that the trajectory Otraj is commonly predicted after the text output Otext; these approaches can be formulated as:
Otext ~ p(Orext X T) ((X, T), and Otruj ~ p(Orraj X, T, Otext) f(X, T, Otext),
(1)
where f(.) the fine-tuned FM. By design, these methods have the advantage of directly exploiting the vast world knowledge of FMs and tailor it towards autonomous driving applications via fine-tuning, a significant advantage over traditional methods for trajectory planning. Additionally, the choice of FM also allows these models to have new capabilities. For example, in the case of language-based FMs, capabilities such as language- and action-based interactions with users are possible along with the original task of trajectory planning- during inference. Further details regarding the design choices and categorisation of these methods are provided in Sec.
We identify 22 current approaches from the literature falling under this broader category. Below, we divide them further into two subgroups depending the features they have.
Models focused solcly on trajectory prediction. These models either have no text prompt T = 0, or a fixed one. In the latter case, T simply aims to trigger the model to yield the desired output, e.g., the fixed prompt could be "plan the trajectory". Since there is no variability in the language, OtraJ primarily relies on input obscrvations X. Methods using CoT reasoning solely to provide better trajectories during inference also fall this category. Similarly, these mothods either use a fized single prompt or a set of multiple sequential fired prompts in a pre-defined order exploit reasoning abilities of FMs. However, the choice of the nature of
8

---

Published in Transactions on Machine Learning Research (03/2026)
X
X
X
O Intttral
p p( (X)
Foundation Model
Foundation Model
Foundation Model
Oust
p 7 XX]
Oust p ( IX Omittra)
Ouai ((
Ouaj
p (1rrg IX Opei))
Ore p Oreen Ouinns)) )
(a) No CoT
(b) CoT
(c) Trajectory
T
Foundation Model
Foundation Model
Foundation Model
Opee
p(O IX.
Ozest
p p | T)
Ouai ((
Oual ((uuujXX.T)
Owal p(Oraa T)"
(d) Language interaction capability
(e) Action interaction capability 5
(1) Language & action interaction capabilities
Trajectory Planning Model
Ouii p(Orru(X.Z)
@ for the methods using FMs for knowledge
Legend
distillatiinn only during training ?
Optional
T
for the methods using FMs for knowledge
transfer during inference
Owz can additionally condition on Owee or Omatroj
Xpm
Foundation Model
Red highlights the main characterietio of a group.
(g) FMs guiding trajectory planning
Figure 5: Formulations of subcategories in our taxonomy. (a f) FMs tailored for trajectory planning. Specifically, (a-c) are focused solely trajectory plamning. These methods either do not have text prompt T or have fixed one, hence T is optional. (c) shows case where O Orraj optionally conditions on the text output Otext- Similarly, Oinittraj can also benefit from a text output as CoT, which is omitted for clarity. (d- f) are the subcategories providing additional capabilities. In (d-f), Otraj is shown without CoT for clarity and highlights that Otraj can also be obtained using different forms CoT in (b) or (c). (g) FMs guiding trajectory planning via knowledge distillation.
these prompts plays crucial role in understanding different factors inftuencing the decision making of these models, therefore, further divide this category into three subcategories depending how, if, CoT is used.
No CoT, hence Otext = 0 and O p(O X) (see Fig. 5 a)). Fig. 2 a) sets an example
for this category. As an example, given front view image, speed of the ego and GPS target points,
CarLLaVA (Renz et al. 2024 only outputs the trajectory.
Text outputs for CoT, hence Otext ~ (O X) and Otraj ~ p(Orraj [X, Otext) (see Fig. (()) Fig. (b) can be considered as an example of this category. One example of this subcategory is DriveVLM (Tian et al., 2024), where text output including scene description (e.g., weather, road type, critical objects), influence of the critical objects ego and planning decision are generated before the model outputs the trajectory.
Initial trajectory prediction Oinittraj for CoT, hence predicting Otraj p(Otraj) X, Otext: Oinittraj) (refer Fig. c)). Such models first yield Oinittraj and then refine it further to Otraj- These models can still leverage text output as well for CoT reasoning while generating either Onnittra or Otraj- To illusraate, FeD (Zhang et al.) 2024) aims improve the initially inttiall--peddited trajectory based on the textual feedback it generates, e.g., if O could result. in collision.

---

Published in Transactions on Machine Learning Research (03/2026)
Models providing additional capabilities. In addition to trajectory planning, these models exploit the language understanding of FMs to build additional features to cater for language- and/or action-based interactions. Below we further divide them into three subcategories.
Language interaction capabilities via Otext ~ p(Orext X, T) (see Fig. d)). These language interaction capabilitiess primarily involve question-answer or description tasks, generally for the purpose of informing users about the surrounding or the potential action taken by the model, as also illustrated in Fig. Cc). These capabilities also serve as tool to understand the decision-making process of these models-allowingg allowing practitioners to provide explanations towards model's actions, debug model's outcomes, or to potentially reassure users about its behaviour. As an example, Orion (Fu et al. 2025a) can respond to different questions such as "Could you describe the overall environment and objects captured in the images provided?" and "Has the traffic light influenced the driving strategy of the ego vehicle in the previous frames?" Accordingly, the set of the text, inputs T that such models can respond is richer than the models providing text output as CoT, in which T typically does not exist or it can be limited to single prompt, e-g.; "Whal is my fulure trajectory?" " Xie et al. 2025). The trajectory planning Otraj in these models can follow one of the alternatives in Fig. 5 (-c) depending on if CoT reasoning is used.
6 Action interaction capabilities where text prompt T may directly result in modifications to the planned trajectory by the model (see Fig. (()))- For example, an input that prompts a model to follow instructions would be "turn left from the next intersection" and this prompt may result in replanning of the predicted trajectory. Such instruction following prompts indicate the model to behave in particular way and are normally diverse in nature. Few examples from LMDrive (Shao let al. 2024ap include "Feel free to start driving.", "Depart at the second erit on the roundabout." ", and "Execule a right manewver, prepare for highway eril". " These are much more diverse compared to the navigational instructions (or route indicators) which consist of pre-defined set of basic commands such as turn right or go straight, therefore, we do not consider models that can only interact with a navigator in this category. Note, in addition to instructions that directly impact actions, other forms of instructions can surely have indirect impact. For example, an input "watch oul for crossing pedestrians" " that provides a warning would be considered as notice instruction prompt, while "crash into the vehicle front" " would be considered as a mislcading instruction prompt where the model is expected to avoid following this prompt and follow a safe trajectory.
Some models can also have both action and language interaction capabilities, as illustrated in Fig. 5 f). SimLingo [Renz et al. 2025) is the only example falling under this subcategory among the methods we consider in this paper.
3.2 Foundation Models Guiding Trajectory Planning
Inspired by the well-known knowledge distillation work Hinton et al. 2014), these models do not build FM for driving use cases, rather they mostly use off-the-shelf FMs to help improve their existing trajectory planning models for AD. From broader perspective, the methods falling under this category can be formulated as,
OtraJ ~ p( Ours X, Z) f(X,
where Z is the transferred knowledge from the FM to the corresponding AD model f(-,-). Note, in this case, f((:-) is either a modular approach (Chnn et al. 2024ch (see Fig. 3 (a)), where the FM can be used at any level
to improve trajectory planning, or an E2E-AD model (see Fig. 3 b)) (uu et al. 2023b Jiang et al. 2023).
We identify 15 methods in this category and further split them into two subcategories depending on whether Z is needed during inference or not, as illustrated in Fig. [g).
? Knowledge distillation only during training. These approaches employ an FM for knowledge distillation during training the AD model. As an example, this can be achieved by prompting a VLM with a text input and sensor data to obtain a structured output such as a meta-action, which can be distilled into the AD model by appending a meta-action prediction module to it, similar VLM-AD (Xu et al.) 2025bb. In such a case the FM is not needed for inference, effectively corresponding to Z 0 in Eq. (2) Hence, the prediction of the model is conditioned only on the observations X during inference, i.e., Otraj p( ((Otraj|X) X))
10

---

Published in Transactions on Machine Learning Research (03/2026)
This offers the advantage of maintaining the inference efficiency of the AD model as the FM is not needed for inference.
@ Knowledge transfer during inference. These methods utilise an FM not only during training, but also during inference with the intention to leverage the knowledge of FMs more effectively. This corresponds to * in Eq. ( where is usually taken as a scene description (Liu et al., (025), typically involving perception knowledge such as the objects in the scene, or planning decision, which can include a trajectory (Tian et al.] 2024 Chen et al.) 2025) a meta-action (Jiang et al., 2024) from the FM. In cither of the cases, Z is typically used either as an internal representation of the FM, or directly the output of the FM. In the latter, the FM output can also be a plain text, in which case an additional text encoder is typically employed for encoding the text before propagating to the AD model. Nevertheless, due to the use of FM at inference, this group of approaches typically requires additional compute for inference compared to the methods mentioned in the category above.
4 Foundation Models Tailored for Trajectory Planning
We now delve deeper into the design and development of FMs tailored for trajectory planning for AD. We first elaborate on the important ingredients one must pay attention to before even beginning to tune an FM for trajectory planning in Sec. 4.1 and then discuss different approaches for cach of the subcategorics in detail in Sec. and Sec.
4.1 How to Fine-tune an FM for Trajectory Planning
Data Curation. The structure of the data to be curated while adapting an FM for trajectory planning depends primarily on the desired use case of the model, as illustrated in Fig. Ba).
Driving Dalasel. The dataset for trajectory planning would simply comprise the pairs of observations and the trajectory, i.e., (X,Otraj) Otraj)- As aforementioned, the observations (X) typically include a subset of the sensor data, the ego status, and an indicator of the route. An example of this form of dataset nuScenes (Caesar et al.] 2020). including multi-view camera, radar and lidar data as sensor input, and future ego positions the trajectory target.
Driving Dataset with Co Reasoning. For complex situations where reasoning is crucial, one might be interested in exploiting the CoT reasoning ability of FMs for enhanced understanding of the scene. To enable that, each tuple in the driving dataset is extended by Otext (the text to enforce CoT reasoning) leading to the datasct consisting of tuples (X, Otext, Otraj)- An example datasct with lextual descriptions that can be used as CoT reasoning is BDD-X (Kim et al., 2018 When CoT is performed using an initial trajectory Oinittraj: dataset might not be curated and stored in advance, and instead it is generated by the model. These driving datasets could also include typically fixed imput prompt T, such the example in Fig. (bb) with T = "Plan the trajectory".
Language Interaction Capability. Although this resulting driving dataset can be used for trajectory planning, additional tuples are needed to account for the language or action interaction capabilities, depending on the use case. Specifically, for the language interaction capability, the dataset is extended with new tuples, (X, Otext) where T, Otext is the question-anne -nnswer pair grounding on X. An example dataset with language interaction is the Chat-B2D dataset used by Orion (Fu et al. 2025a). ). Action Interaction Capability. If the model is expected to generate a trajectory, while answering the question, then such tuples include Otraj, i.e., (X, T. Otext: Otraj); in which case the answer Otext could serve as CoT reasoning. As for the action interaction capability, the model is expected to consider the user instruction for trajectory planning. Accordingly, the data tuples typically follow (X, T, Otraj), such that. T is the user instruction and Otraj is the trajectory corresponding Lo T. An example dataset with language and action interaction is the SimLingo dataset (Renz et al., 2025). Again, optionally, Otext can be included for CoT reasoning during instruction following. We further discuss the different choices of the CoT reasoning in See. 4.2 and the datasets designed to include additional capabilities in Sec. 4.3
11

---

Published in Transactions on Machine Learning Research (03/2026)
Driving Detaset
Driving Datzeet with CoT Extend ouch buple with Oseet (T,X, text, Otraj)
Lauguage Interaction Capability Add nen txples by cunnting (T, Otext
Action Interaction Capability Adit new tupliss by cunsting (T, Orra))
(T, X, trai)
(T.X. X. text Otrai)
(T. O text Oray)
Trniectorn [St Vih.
CoT Reasoming
Trajectony
Question
Ansver
Instrre Bon Apply
Trojectory
Vih
Is ego velicle at. jusetion?
velicle
stop
(Bu sull
Drive more slowly
on turning lane appraaching junation.
and sttog now..
[[ Fih.
Stotus
to below the
(Na
speed
(a) Data curation for different use-cases
Vision Encodes
Exspinst
LLM
Eucodles
Vision
ML. oe Linear
LIM
Viston Encuder
Q.Former
(Laff) A Eank inderfire patrestaally boved on template
(Middle)
Profection of cisiom festuree wiik loger
(Right) Projection of aision features wilh Query-furrue
(b) Common vision adapters
Text
Text Tolomiser
Tokentzer
Text Tobenixer
with Action
Tobens
LLM
Tokamisee weth Action Tokens
Opai
Owaj
Ouri
Plaming Head
Tokeniace
(Lefft) Trajectory as teat, obtuined by A.R generution
(Middle) Trajeckony ns artinn tnkeng, obtaised by AR generation
(Right) Trajectory as set of nambbrrs. obtained single pars
(c) Different approaches to represent and generate the trajectory
Legend
Viston Eacoder
Viston
Viston Faeeder
Vision Adapter
Vieion tokm
Not trained
Adapter
Action tokom
LoRA Eme-tuning
L.M
L.M
fine-tuning
Text
Text Tokmiisen
Tokeuizet
embeddling
tuning
Goey (6) Impliis optional.
(Left) Vision language aligmment
(Right) Fine taaittiingg the model
(d) An example two-stage training approach
Figure The steps fine-tune an FM for trajectory planning. (a) Data curation. (b) For the model design, either an off-the-shelf VLM can be used or a vision encoder can be combined with an LLM using one of the depicted vision adapters. (c) Trajectory representation. Vision encoder and adapter omitted for clarity. (d) Among various model training strategies, this is example two-stage training approach Liu et al. 2023). usually adopted when off-the-shell VLM is not used.
based on
Model Design. While designing the model architecture, approaches (Renz al. 2025 Zhou al.
2025b use off-the-shelf VLMs, providing the advantage of initialising all parameters jointly from a state trained on large dataset. Differently, another group of approaches (Xu et al.] 2024 Chen et al. 2025 Fu et al. 2025a) combines a preferred vision encoder, such as ViT (Dosovitskiy et al. 2021 with a preferred vision adapter. Both approaches follow the typical VLM architecture (see Fig. (()). where the vision adapter connects the vision encoder and the LLM. Additionally, the latter approach combine preferred vision encoder and LLM, requires designing a vision adapter. As shown in Fig. 6 b), in general, there three different choices of the vision adapter:
LLM using a randomly-initialien
12

---

Published in Transactions on Machine Learning Research (03/2026)
The vision adapter can be tert interface where the vision encoder directly outputs perception and/or prediction information such as the locations of the objects and map elements in the scene.
This information is then propagated to the LLM as text, in which case the model is not trained end-to-end as in Mao et al. (2023).
lineur layer or an mulli-layer perceptron (MLP) can project all vision lokens from the vision encoder onto the LLM input space (Luu et al. 2023) as used by Renz et al. (2025).
Different from a linear layer or MLP block, Queryformer LL et al., 2023a) relies on cross-attention
between set of typically randomly sandomly-initialised latent representations and the tokens from the vision
encoder to selectively project the most useful set of representations. This approach is taken by Omni-Q (Wang et al. 2025b).
Additionally, including custom modules can impose certain inductive biases absent in off tff-thh-shelf FMs. For example, as off-the-shelf VLMs are generally not pretrained on 3D perception tasks or using videos, some approaches design custom modules (Fu et al., 2025a Wang et al. 2025b Xie et al. 2025 Chen et al., 2025
to enforce the model to consider these clues absent in the VLMs.
Trajectory Representation. Another crucial aspect in tailoring an FM for AD is how to represent the trajectory, as this can consequently require modifications in the model design as well. Fig. ((c) illustrates common trajectory representations, which we elaborate on below:
Trajectory as text, obtained by the standard AR teat generation of the LLM. For example, consider a model that generates a single character in each pass and assume that 1.54, 0.21 is 2D point of the trajectory. As result, this 2D point is generated sequentially as '1', 9, '5', '4" 51, '0,, 9, '2', '1' in an AR manner, as illustrated in Fig. (Left). Consequently, while this approach generally does not require any modification to the tokenizer model architecture, it requires multiple passes through the LLM to yield single point of the trajectory, increasing the inference time.
Trajectory as action tokens. Instead of generating a single 2D point in multiple passes, one can discretise the action space and represent these discrete actions in the vocabulary of the LLM by the action tokens. One example is to discret 2D BEV space using a grid and use the points on the grid as the action tokens ([Sima et al. 20244] Brohan et al., 2023). These action tokens can be included in the tokenizer by mapping the rarcly-used tokens in the vocabulary to each of these 2D points (Brohan) et al.] 2023). Zhoo el al. (2025b) also follow a similar approach by building vocabulary for the actions with 2048 actions determined by clustering the actions in terms of the relative 2D position and the heading angle in the next 0.5 seconds. In these examples, the trajectory consists of multiple points, and hence multiple AR passes within the LLM are required to obtain a trajectory, as shown in Fig. 6 (c)(Middle) As an alternative, similar the planning vocabulary in VADv2 (Jiang et al.j 2026), an action token can represent an entire trajectory where the trajectory can be obtained in a single pass, however, this might be more prone to errors due to long-horize planning.
Trajectory set of numbers, generally obtained in single pass by using a planning head. Alterna- tively, the trajectory can be obtained by an additional planning head that utilises the represontations learned by the LLM. There are two main variants of this approach. As shown in Fig. 6 c) (Right), some methods (Renz et al. 2024 Zhang et al.] 2024 Renz et al., 2025) use learned embeddings that attend to the observations (and a text input, if exists) before being decoded as a trajectory by the planning head. On the other hand, another group of approaches (Fu et al. 2025a Xu et al. 2025c) does not introduce learned embeddings, and instead the planning head relies on the representat ions of the LLM in the last layer. In any case, this approach requires designing planning head to be appended to the LLM. MLP (Renz et al.] 2025) and generative models, such as a variational autoencoder (Fu et al. 2025a or diffusion model [Laoo et al. 2025), have been explored in the literature for this purpose.
Model Training. Fine-tuning the FM is typically carried out in a single step or in multiple steps, where each step generally targets a specific subset of the modules. For example, similar to LLa VA-syle training (Liu) et al. 2023), ; as illustrated in Fig. 0 d)), one can first train the vision adapter by freezing the vision encoder and LLM, especially when a custom vision adapter is introduced with randomly-initialised parameters. After training the adapter, all model parameters or subset of them can be fine-tuned for the target task. To provide an overview of how the existing methods in the literature tun- their models, we introduce the
13

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1:
Design choices of FMs tailored for AD (refer to Fig. 4 Unless mentioned otherwise in the corresponding paper, we assume that each mothod uses full parameter mno-tuning of a module, generates the trajectory in the form of text, and does not have route indicator as an input. Custom VLM implies that the model does not employ an off-the-shelf VLM but combines vision encoder with an LLM itself, in which case we specify the details of the vision encoder and LLM, e.g., SigLIP-ViT-G as in the case of S4-Driver. Please refer to Sec. for the details of the training symbols. Model Design and Model Training
Obacevat Soms
Model
Sumsors
Route Indioster
VLM
Vision Eausddrr
Vision Achapier
LLM
Parsan
Toajectory Represent foom
Gar
Roox es all
X011
Foui cumera
MaVAA-Nist
VIT-L
Limear
Thyy Lian
2-50
Numbhrrs
Xa al.
20250
Frout cumeers
Custem
SKeLIP VIT-L
Libear
Quea
A
2-133
Numbers
2004
Infe-
Ploecnce-2
DuVir X
Linear
BART
2020
Text
GPT-Driss
Mao
300
Custom
Braod on USIAD
Tict
Char- GPTR3
14177B
Tixi
Quen-Vl
Chatom VIT A
MLP
Queelll
Text
Zamn et
10%5
Freot cxmarne
Q.n22.5-VI.
Chaton VIT A
MLP
Queu2.5
>-3
Astion tokon
Youn st al.
2011
Fout, cameraa
LLAV
V
Linear
LLaMA2 X
0-713
Text
Xie al. 2725
JRY cameras
Paclia
SigLI--VIT-C * +Costoon
Liueer
UL.2 A
Text
Blao ct 2024
Ageat- Driver
JRY cameras
Custom
UnIAD *
Test
Chat-CPTs A
>11TD 2
Text
P00
Zhnng 9/4
Front
LLAVA
VIT-I G
Linear A
>-7
Numbers
300
Custem
EVA-02-1 1
Q Q.Ptttnce
LLAVA--LG-IM
>7B
Tixi
DrixOPT
Faut
Custem
Ville--VI-L. *
Valle-Curinn
Lluna2
>-B
Tixt
Sinss
Fout camnena
BLIP-2
VIT-L O
Q Pormer A
Plan Io 6
2-7B
Action tolun
Pinos
Hongg 6 al
0125
38y cameras
Genatal
Unimnow
Unlmown A
Unknown
1.50
Text
15
Zhon et A
2026
30y camrras
Cuetem
Reciemeer
Quan 2.5-heirurtt
24.58-77
ARt teet
16
Hogle
21/5
300 camrra
Custem
Bevlormer
Q Portnee A
2-7B
Tixt
Wang
20256
302
Custem
FVA-03-1
MLP x
>-7B
Text
Wong
cameras
Custom
EVA-02-1 1
Q Pietnee
LLAVU-1--IMM
>7B
Text
Druss
303" cameras
Chstem
EVA-02-1 1
Q Portuee
Vieunz-1.5
>7B
Numbers
Cll e al
Orix
360' costartiie
lidar
Chetem
VIT *
CD MG-M *
Q Portnee
Liana A
Text
Mloo al.
360' CHLLACTIO, Jollr
Cuetom
ReeNet 3 PotitPilles *
Q Pormne
MoVA-nt.--111 *
>-B
Numbers
Stadrngo
Tunx es al:
Fourt comtra
Misi-IntewWV
InsenViT
MLP
Quva2 A
1B
Numbers
following symbols that represent the change of the parameters in a module across training (spanning all stages in the case of multi-stage training):
indicates keeping modules completely frozen during training.
represents low-rank adaptation (LoRA)-style fine-tuning of module (u et al. 2022); ; which assumes that the adaptation needed for new tasks can be captured by low-rank updates. This approach is efficient but limits the capacity of the model for adaptation.
implies the standard fine-tuning of all parameters, utilising the full capacity of the module for fine-tuning, though it requires more resources than using LoRA.
represents training a module by initialising it either randomly, e.g., once a custom module is introduced, or from a model pretrained only on domain-specific data, e.g., a vision encoder trained only on nuScenes [Caesar et al. 2020).
Using these symbols, Tab. presents the main design choices of 22 methods within the scope of this section.
4.2 Models Focused Solely on Trajectory Planning
Following the taxonomy provided in Fig. we now discuss different subcategories. Tab. 2 provides an example approach for each of these subcategories.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Example models solely focused on trajectory planning. For such models, T either or fixed, and CoT reasoning includes Text or an mitially-planned trajectory Inittraj for self-correction.
Model Type
Example Model
Observations (X)
System Prompt (T) TTxxt Output (Ouxa)
Prons image: speed of ego; GPS target point
Oun [X
CarLaaVA
Multiview imagee; speod, acceleration
Oreg ~ pr N. Owet)
S4-Driver
past trujectory egu; high level
What my uutiure trajec- Meta-decision,, eig, keep spred then tory?
do decelesating
commund
Please conluate the pro- Collision with vehioles or podestriane,
Orag pio (X. Oroer. Oruitsaa)
FeD
Frons image: speed of ego; GPS target point and high level command
dieted future locations of traffic light violatious, devintion from the expers planned zouse
ego velicle
4.2.1 Models without CoT Reasoning
As one of the earlier approaches that uses a VLM for trajectory planning, CarLLa (Renz et al. 2024 is built on VA-NN-NeXt (Liu ct al.) 2024b), which combines ViT (Dosovitskiy et al. 2021 vision encoder with Tiny-Llama (Touvron et al. 2023a) as the LLM. Given the front-view image, current ego speed and two GPS target points as a route indicator, the model predicts (i) 20 path waypoints that are equidistant points at meter apart, thus independent of the time, to control the steering and (ii) 10 speed waypoints, which are points at equal time intervals, specifically 0.25 seconds apart, to control the accelerator and the brake. These outputs are obtained in one-pass using ML.Ps as the planning head, following the design in Fig. [6] (Right).. Note, unlike other methods we explore, the Tiny-Llama in CarLLa VA is trained from scratch. In 2024, this model, trained on approximately 3 million images, a competition based on driving in a closed-loop simulator. Given three camera views as front, front-left and front-right, DriveGPT4-v2 (Xu et al.] (005cc), as a similar model, combines SigLIP-VIT (Zhai et al.) 2023b) with Qwen (Bai et al.) 2023a) to predict the target speed and turning angle controls directly. addition to that, the model supervised to predict a trajectory and equidistant route waypoints, similar the path waypoints of CarLLaVA. Different from CarLLa an expert model provided with privileged information, such as the objects in the scene and potential hazard information, is trained to augment. the training dataset of DriveGPT4-v2. This augmentation is shown to have a notable effect on driving performance. While these two approaches do not have a system prompt T, V2X-VLM (You et al., 2024 uses a fixed T = "Please predict the ego vehicle positions over next 45 timestamps." ". V2X-VLM, is built on Florence-2 (Xiao et al.) 2023) (using DaViT image encoder (Ding et al.) 2022) with BART language model (Lewis et al. (020)), , and utilises DATR-V2S V2X dataset (Yu al. 2022 to focus specifically on levernging an external camera from the infrastructure in the environment.
the CARLA 2.0 challenge (Dosovitskiy et al. 2017))
Summary of trade-offs:: Models without CoT Reasoning 0
As these approaches are solely designed for driving, the dataset curation is relatively simple, where each data example typically consists of (X, Otraj) pairs. Moreover, due to the absence of CoT reasoning, these models demonstrate superior computational efficiency particularly when trajectory generation can be accomplished through a single forward pass. Nevertheless, these advantages entail certain trade-offs: the omission of CoT reasoning may limit driving performance, and the lack of language generation capabilities diminishes their explainability.
4.2.2 Text Output for CoT Reasoning
CoT reasoning is typically prompting strategy of the FMs that help them to break down the problem into multiple steps during inference. This is commonly achieved by prepending a step-by-step solution to a similar problem (Wei el al. 2022 Yao et al. 2023 Sel et al. 2024) to the input prompt, as a demonstration on how to approach and solve the problem. Furthermore, simply appending "Let's think step by step " to the prompt is also found to be quite effective, also known as the zero-shot CoT (Kojima et al. 2022). These enhanced ways of prompting strategies facilitate FMs to yield its output step- by-sep, improving their performance in certain tasks such arithmetic reasoning.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Similarly, FMs tailored for trajectory planning leverage CoT reasoning for improving the driving performance, however, in slightly nuanced manner. In particular, these approaches explicitly fine-lune the FMs to yield the output step-by-step, instead of modifying the imput prompt T during inference time. Besides, the prompt T, in this case, is typically fixed, c.g., "What is my future trajectory?" (sce S4-Driver in Tab. 2 and can polentially involve multiple steps. As a result of the fine-tuning, lext output Otext includes a predefined set of explanations that are expected to improve trajectory planning, i.c., Otraj- The scope of Otext varies significantly across methods, ranging from only a meta-action a comprehensive description of the scene and agents' behaviours (please refer to Fig. 2(b) for more comprehensive example).
Methods using a more comprehensive Otext as the CoT reasoning generally follow the common pipeline of the modular AD models (see Fig. ((a),, hence generating the CoT reasoning in the order of perception and prediction information followed by planning decision, e.g. meta- action and trajectory. As a pioneering approach following this sequence, GPT-Driver (Mao et al.] 2023) fine-tunes Chat-GPT3.5 (OpenAT] 2023 to yield notable objects in the scene, along with their potential impact on ego, and a meta-action as CoT reasoning before yielding the proposed trajectory. As an earlier method, GPT-Driver adopts a leat interface as the vision adapter (refer to Fig. 6 (Leff)) hence it is not trainaaae. Specifically, the perception and prediction information is extracted by the corresponding modules of UniAD (Hu et al. 2023b), and propagated to Chat-GPT3.5 as text. DriveVLM (Tian et al. 2024) follows similar sequence in terms of CoT reasoning, where scene description and analysis are followed by meta-action prediction, and finally trajectory planning. To preserve the capabilities that the VLM gained before fine-tuning, Qwen-VL. (Bai ct al.] 2023b: is fine-tuned by combining AD datasets with the LLaVA dataset, which includes examples from different domains, referred to as co-tuming. Similar to the previous methods, Auto- (Zhou et al., 2025b) also relies comprehensive Otext for CoT reasoning, including a scene doscription, identification of the crucial objects, the intentions of the surrounding agents, and ideal driving actions. Differently, it follows an adaptive CoTT mechanism where the model refers to CoT only for complex scenarios, considering its inefficient nature. To train such a model, a dataset is constructed as a combination of action-only scenes and reasoning-augmenntee scenes. The model is first trained using supervised fine-tuning, followed by reinforcement learning, specifically GRPO (Shao et. al. 2024b), where the reward function promotes driving-relatee measures such as safety and comfort while discouraging CoT reasoning if deemed unnecessary.
There are also methods that generate a brief description or explanation as CoT reasoning. One example is RAG-Driver (Yuan et al. 2024), which only predicts an explanation of the action that the ego is executing, such as "The car moves forward then comes to a stop because traffic has stopped in front", known as action explanation and justification in BDD-X dataset (Kim et al. 2018). Relying on this as CoT reasoning, the model outputs speed and turning angle of the ego. As the model is named after, it leverages relrieval- augmented generation (RAG), which retrieves relevant information from external sources to enhance the model's prediction. Accordingly, in the AD domain, given a test sample, the two most similar samples in terms of cosine similarity retrieved from the training data and fed to the LLM. As the retrieved data have known control signals, they are expected to improve driving performance for the test sample. Alternatively, S4-Driver (Xic et al., 2025) show that it is even useful to predict only one of the four pre-determined meta-actions (i.e., (keep stationary, keep speed, accelerate and decelerate)) via CoT reasoning before planning the trajectory. The model is built on PaLI-3 VLM (Chen et al. 2023b),, which is based on the UL2 LLM (Ty ct al. 2022), ; and equipped with custom modules to facilitate learning 3D scene ropresentations.
The inference time and accuracy trade-off of CoT reasoning. One key impact of CoT reasoning that needs to be taken into consideration is its effect inference time. This is because Otext: as the CoT reasoning, is typically generated in an AR manner, requiring multiple passes through the LLM. This is espocially important for the applications requiring fast inference speed, such as trajectory planning. To provide an example, we test the throughput of SimLingo, a relatively lightweight model with 1B parameters that processes only the front image, with and without CoT reasoning. Specifically for SimLingo, T = "Predict the waypoints." " prompt yields the trajectory without CoT reasoning, while using T = "What should the ego do next?" triggers a relatively lightweight CoT reasoning with 2 or 3 short sentences about the what the action should be and why. For cach of these prompts, we propagate 200 images to SimLingo five times using its official implementation. We observe that the CoT reasoning increases the inference time of SimLingo by 4.5x, from 3.6fps to 0.8sps on an Nvidia A4500 GPU, which can be a significant bottleneck for practical deployment. In this case, the
16

---

Published in Transactions on Machine Learning Research (03/2026)
authors report relatively small performance gain of using CoT reasoning, where the driving score increased from 84.41 to 85.07 and success rate improved from 64.84 to 67.27 on Bench2Drive benchmark (Jia et al., 2024). As a result, the compute requirement of the designed CoT reasoning as well as the performance gain it contributes to should be taken into considcration while using it in practice.
Summary of trade-offs: Models using Text Output for CoT @
While text-based CoT reasoning offers the advantage of grounding trajectories in the model's rationale, thereby enhancing explainability and potentially driving performance, these benefits come with substantial Irade-ofls. Foremost, the additional requirement of annolating training examples with text output Otext increases the complexity of data curation. Furthermore, as previously established, CoT reasoning can drastically increase inference latency, thereby imposing severe constraints on the practical deployment of such models in real-world applications.
4.2.3 Initial Trajectory Prediction for CoT Reasoning
Different from using only Otext for the CoT reasoning, few approaches leverage it to assess or refine a trajectory that the model initially predicted as Oinittraj- Such methods can also leverage Otext to improve Onittrraj or Otraj further. As an earlier approach, Agent-driver (Mao et al. 2024) essentially extends GPT-Drrier (Mao et al. 2023) (Sec. 4.2.2] with a multi-step CoT reasoning involving notable objects (with their potential effects) and mela-action (Otext), and an initial trajectory Oinittraj- predicted occupancy map from the perception module is also used to check whether Onnittraj resulted in collision. If so, Oinittraj is corrected as Otraj otherwise it is accepted as the final trajectory Otraj. Following GPT-Driver, it is built on GPT 3.5 and employs a lext interface as the vision adapter, hence it is not E2E-trainable. Alternatively, FeD (Zhang et al., 2024) first predicts a trajectory Oinittraj ~ p(Oeraj] (X) with the initial system prompt "Predict ten future locations in 2.5 seconds", ", This is then followed by refining Oinittraj using the feedback of the same model as the CoT reasoning including potential collisions, traffic light violations deviations from the route or expert behaviour, as detailed in Tab. 2 Finally, the trajectory is refined, i.e. O traj p(Ouraj X, Otext, Oinittras) by conditioning the feedback Otext and the initial trajectory Oinittraj- As result, unlike Agent-driver, relying on frozen occupancy prediction model, the feedback Omittraj is learned in an E2E manner during training, and provided as an output such that Otraj considers it. FeD also employs an expert model with privileged information, similar to DriveGPT4-v2, to augment. the dataset with the demoustrations from the expert (refer to Sec. 4.2.1] but differently by using feature distillation from the expert. Alternatively, Solve- VLM (Chen al. 2025) is designed to have the final trajectory always in two-steps using Trajectory CoT. Specifically, coarse trajectory, Onittraj, is predicted among a sct of predetermined trajectories, and then, Onittrajj is refined to be more precise Otraj- Unlike FeD, the model does not explicitly make explanations, e.g., about potential collisions or traffic violations. Solve- VLM uses a custom VLM by combining EVA-02 (Fang et al. 2024 with a LLaVA-based LLM via their proposed SQ-Former to incorporate 3D representations into the vision fcatures before passing them to the LLM. As Solve (Chen et al.) 2025) also transfers knowledge to another AD model, we will further explore it in Sec. 5 while discussing methods leveraging knowledge transfer.
Summary of trade-offs: Models using Initial Trajectory Prediction for CoT ?
These approaches offer the advantage of iterative trajectory refinement, thereby potentially enhancing driving performance. By utilising exclusively the initial trajectory prediction for CoT reasoning (i.e., without incorporating additional text-based CoT), these methods can be adopted with computational efficiency, particularly when trajectories are generated through a single forward pass. Given that these models perform self-refinement of their outputs, the data curation does not require an additional overhead; however, the model design must incorporate appropriate mechanisms to facilitate this refinement process. A notable drawback, though, is that without text generation capabilities, these models lack direct explainability and cannot interact with the user.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Table 3: Characteristics of the models providing language interaction capability. If a model is trained in multiple settings such as ArieeLM-Agent or Emma, then we include only one of them for clarity.
Model
Tralnting Dataswt
Training Dataset Size
Overview Langunge Annotations
Moin Ausotalox for Language
Auguncaattion of QucAk with FM
Examsple Queetion
Model Training
Language Fiallattion
Step I. Vasto-lungiagn
Driseipyg of
Quece Additional sue culanced with ollc jnetfrection, some descripe and Char-6iP7 for the Quoektrne Bnn-X cueron
BDD
10K BOOK QkAa Actins dossijtion, ucisnn Haman Eer BDD-X,
Is thess eas the vidii adupter
aligunnct by only lasinia
CIDE.
BLBC,
Chat-SPPT QuiAs
ago voliode"
Step 2. Midfnestunigg be ROUGE, CP T-Soee
nott datswet
the cuntets dataen
tim.
conlinig both
2mn-A dittects
Mri Sun 2I24
OC froet view Imagr with 3306 QUA pains
dourpttue
Vidoo clip lavel
Imaageloeel pemepptton,
Buman
Quettone
the vebble motice
avord thidd Finsttrinn seing trals whes the viludd ing
What object should
SPICE and CIPT-Souee plannng preficcion ViAAs, and chooilaation ax- Inlavior
prodiction, plansing, be- bastour and motion Vigat
ged ling to and poseible locatio?
VQJA
Custom Tatxor: MUK
Step I. Pwerahiing on
Euroo Blwangee
Cintom
SD abject deteetius, dii- Using perteptiion su- Weyro Open Mo Weysro Open Mition aod grapi caiiaaion, astalkuus inlete lo Dettect: hours Modiage csliuaiion
Dutioct botco diising
the abeud trm- ponarly Moxleed
augooseale cralon diiiect Takk uxtrics sudi Step Fan-tuning us Preciion Recall cuve Lag Wesmno Upen Mation object detection
Dellaoe:
hunute
moCaptim:: Dvecription of
the socur, objects and jou textial (ikk
Step 1.
Valan langunlan
anDlptiion LoMM "nCopption: Farth
aliganeet
uilaatiion BLEU and
Oon Aboa et al 2050
usCappion
u uxx narios in 700 sioro dije
miStrnes dacaset with (sSoomnsQQ: Soore carses A Alagter and GPTT
-4 lorm detoing soe-
custesas, counting suery tha canduaioos lo subecrosil) U2A Note homure
standing Qlib inclodin anSorres Qli Parap-
soves
qpottiee and any Are thene any cans the focze righe of the stuyged lass?
VQJAa
Step &, Fias-toning for tae- kir ards qpacieu lose.
Step 9.
Plnestuning for BRRTT-Sore
suBers QA: Accunnsy
obyst, query-ttatie
compateen-typ guostmm un-: Homan
nnX Only sackers
Step 1. Fine-trnin for tra- XETROU and ROUGE
tin perclction
mxX: CIDEi BLRE.
meX. Action descrptton cad justiicaiion
yattryy plannnn
Drvvdd Mourr meSortas daraest with semprien, Innage-leed Iimann and Llluradd No adkitional ang tended with the -4 looms drving cxplins, prodecton, plan NIB for the culeasion unising imngia surioe (00) vižao dips nine, belavioo and motio
Vidoo-liy lord
de
What ore the Step "haining only nomerrtt th Eafi model
DAMA-MG.1M Bogle al 2022
murestion for the texaion
agrnta the back Step Flanstuing E2E Qualtatton eniizatton right the ego car? anddd and joinb
VQA
muSoensn dacaart Secue desriptins, atlen conneecfattaal muscet- I10 vileo clips tiit ing the comupl qua- iu /~4 hous driving tisu), deciskn aukkag plassing, general
If docals acoike. Step 1. Vistn langnage CTDD) lengsrge eni alc madh kif abgamere rainn 30 pencep- tottien Average Preet- bun, what cotid iiu leoks ontaegueennd Step Huestuing soning
Onrui-9/2 Wang
Jtc OnnilDrive
GPT4
and Average Beaal
comtriss)
countecfacton tea-
Step
Vaico langung:
Secue: discription, Io-
alganon using 2D
BID Base dolunct with laciius dastigh of erile Basdd in CARIA A "It sorm driting soe- ital chjects, mera- artune sutice and 2.13d Qlcka and sction srascuing of shunlatur stalic und to 950 vidro clyee
Hon has the cureii iian lesks spred changed com- Step 2. pazed to the porroraa algimmett without VOJA frnvee
Orewn I' 4t al plate
Clas- BID
Languageactum Quiitative eniiatioon
the ego, roall escrtial QucaVVL-72B listorral Infommation
Step Flueturiin the
medel with VQJA distrett
Rese
esyle
sunduted DroeLM bustorr and motlon VQAAe
Insure prsliction.
perceptions Based stankiio
bo
CARLA Quellious and Au tlece
taiiin laiis SPICE GPT DrisoLM delenet
4.3
Models Providing Additional Capabilities
We now discuss models that provide language or action capabilitics in addition to trajectory planning
4.3.1
Models Providing Language Interaction Capability
Clearly, the ability to interact via language is a remarkable feature enabled by the language interface of LLMs. Tab. 3 provides an overview of approaches using language interaction, including characteristics regarding their training datasets, training approaches, and language evaluations. Below, we summarise our observations regarding their training dalasets and evaluations. Language Interaction Capability Datasets. Models with language interaction capability generally require pairs of VQAs for training, which are not part of standard driving datasots such as nuScones (Cacsar et al. 2020). Hence, after creating the necessary question templates, there are three main approaches to annotate these datasets, depending on whether the dataset is real-world or simulated:
For simulation datascts, usually curated using CARLA (Dosovitskiy et al. 2017 predefined answer templates are populated by the simulation state including agents' velocities and positions, the weather, junctions, and traffic lights. This approach is used in DriveLM--ARTA (Dosovitskiy et al. 2017) and the SimLingo dataset [Renz et al. 2025), and is a relatively easy approach resulting in accurate language annotations.
2Since SimLingo the only approach providing both capabilities @ we present its language capabilities Tab. and discuss SimLingo in detail in Sec. 4.3.2 instead reserving a section for it.
18

---

Published in Transactions on Machine Learning Research (03/2026)
For real-world datasets, the desired details of the scene are usually not available. As result, some approaches rely on human anmolations such as Drivee-M-nuSonnes (Sima et al. 2024). Though this results in accurate annotations, it requires significant resources to annotate a large dataset.
Alternatively, FMs such as GPT-4 (OpenAl 2024) are frequently used to create language labels. Specifically, the question templates are provided to the FM along with a system prompt and sensor data to obtaín the answers. This approach is used to annotate both simulation datascts (Fu et al.) 2025a) and real-world datasets (Hegge et al., 2025). As this is an automated approach, is efficient, though manual inspection might be necessary to ensure the quality of the annotations.
After constructing the VQA dataset, several methods augment questions or answers using an LLM to increase language variability. Furthermore, the size of the training dataset varies significantly across methods. For instance, while DriveGPT4 is trained on 56K VQAs, Orion uses 2.11M VQAs for training. Additionally, the VQAs generally focus on perception, prediction and planning as subtasks that allow a model to achieve good driving performance. DriveLM (Sima et al. 2024) and OmniDrive (Wang et al. 2025b) are proposed as VQA benchmarks. The DriveLM dataset includes VQA pairs for these lasks, such "whaal object should the ego vehicle notice first when the ego vehicle is getting to the next possible location?". Alternatively, OmniDrive stands out with counterfactual VQAs, such as "If I decide to accelerate and make a left turn, what could be the consequences?", where the counterfactual questions are designed using templates relying on the meta-actions such as accelerate and make a left turn in this example. Consequently, the answers are obtained by a combination of a based checklist and GPT-4.
Evaluating Language Interaction Capability, For evaluating the quality of the generated text, the methods commonly employ measures from natural language processing literature, comparing machine- generated text with reference text. These measures include BLEU (Papineni et al. 2002), ROUGE (Lin) )004) METEOR (Banerjee & Lavie 2005) as well as CIDEr (Vedantam et al. 2015) and SPICE (Anderson et al. 2016), which are specifically introduced for image captioning. Furthermore, model- evaluation measures are also utilised. For example, GPT-Score (Fu et al. 2023). used by Sima ct al. (2024), is obtainod by prompting Chat-GPT with the question, the reference answer, and the machine-generater answer, and machine-generated (2026) use BERT-Score (Zhang* et al., 2020) to compare the similarity of the machine-generale text with the reference in the BERT embedding space (Devlin et al. 2019).
asking for numerical score about the accuracy of the machine generated answer. Alternatively, [hou et al.
A Discussion of Existing Approaches. Here, we highlight the key aspects of the trajectory planning approaches providing language capability. Please refer to Tab. and Tab. 3 for the details on their model design and language interaction capability, respectively. DriveGPT4 (Xu et al.] 2024) aims to retain the capabilities of the LLM while training a trajectory planning model. To achieve this, they found useful to keep non-AD related VQAs in addition to 56K driving-related VQAs during training. As the model is based custom VLM obtained by combining video encoder {Luo et al. 2025) with Llama2 (Tourron lt al.) 2023b,, the training of the model is handled in two stages, vision-language alignment and model fine-tuning, following Fig. (Right)) Alternatively, DriveLM-Agennt (Sima et al. 2024), the proposed baseline of the DriveLM benchmark, fine-tunes BLIP-2 (Li et al. 2023a a VLM based on the Flan-T5 LLM CChung et al.) 2024), on the VQAs of the benchmark in a single stage. Emma (Hwang et al.) 2025) based on Gemini VLM (Gemini Team 2025), uses a large dataset of 203K hours of driving scenes for the pretruining of the model. This pretrained model is then fine-tuned on specific domains for adaptation, such on NuScenes dataset. (Caesar al., 2020). Instead of the conversation lype questions and answers, the language capabilities of Emma seems to be limited to the questions regarding predetermined set of perception tasks including object dctection, road graph estimation, and road blockage detoction. Different from the existing approaches, OpenDriveVLA (Zhou et al. 2026) and DiMA-MLLM (Hegde et al. 2025 rely on BevFormer [Li et al. 2025b) as the vision encoder to effectively extract a 3D representation of the
scene to address the limitation of the FM-based vision encoder such as CLIP (Radford et al.) 2021 Zhai et al., 2023b) or EVA (Fang et al. 2024). However, this can also cause potential disadvantage in comparison such models, considering that Former is not pretrained ct al., 2025b) share the same architecture, where EVA-02-L (Fang et al., 2024) vision encoder is combined with LAVV--1.5 LLM {Liu et al. 2024a), except their vision adapters. Specifically, Omni-L relies on a linear layer following LLaVA (Liu et al. 2023) (see Fig. (Midddl))) while Omni-Q employs Q-Former
a large dataset. Omni-Q and Omni-L (Wang
19

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: Characteristics of the models providing action interaction capability.
Model
Training Dataset (with semsors and size)
Exampple Notice Instructions
Example Action Instructions
Mechanisan to Avoid Misleading Instructions
Model Training
Instruction foooooolooowwogg Evaluntion Messures
I'm rumning short on time. Ix
it poséille for to utilisn
emeegeacy lams hyposs
280 hours of driving
veliccio shrad?
DrM
Ca al. 2.2h6
and right) and lidar
marios CARLA. Sinsorec
cameraa (Eroor, roar, left
x
you change to the lett lane?
using the training Quaitative enluation datneett
Fine-tume
sur.
are obetacles
Can you switch to differrnt lane to hypaos?
Foel free to start deiving.
Please watch for the pedietrions up ahead.
3M
Find your way out nt the fixat exit on the roundsboou, plenes.
Step Train vision
Driving performance
lected in CARLA at 10fps
Be mindfai the velicle crowing
racoder with percep. procided setiie and no- Lie instructios. LungAnta brechmar's proposed this purppoo soune pa- pec.
simated
LMDriie
(-85 houns of driving data).
light
tion tasles using the
Shap
2024n
Semsors:
(front.
your left.
At the forthooming
froat
lidaz scusor
rouz, and right)
Please alert the un-
turn. Just head for the left
Step 2. modeel teained and-to-euad
vicinity ahoud.
evem road surface in the
laze. Maiatain your course
this
Geuily linluss
Simlingo
3.IM front image (-2115 houns
Hit the veliile Ford Crown.
Fine-tume the VLM Accuracy of the model for using training type
de
lected in CARLA as sipe
Diirett one bow to the left.
dataset
ston
supervised by 3D perception tasks (refer to Fig. o (Right)) Their analyses suggest that using a linear layer is more beneficial than Q-Former in terms of both language capabilities and open-loop driving performance. Finally, Orion (Fu et al. 2025a combines EVA-02-L with Vicuna-1.3 (Vicuna Team 2023) using a Q- Q-Former variant. The proposed Q-Former, coined as QT-Former, also considers the temporal aspect of the observations through memory bank improve the driving performance. Furthermore, Orion presents notable analyses using different architectures planming heads (refer to Fig. ([ight))) Their analyses conclude that the planning head based on variational autoencoder perform better than using MLP diffusion model.
Summary of trade-offs: Models Providing Language Interaction Capability A
This group of approaches offers the distinct advantage of answering user questions, primarily serving to enhance model explainability and provide reassurance regarding the system's decision-makinn processes (i.c., trajectory). Nevertheless, similar to text-basod CoT reasoning, the generation of text output through an AR mechanism can substantially increase inference time, thereby constraining the deployment, of this functionality. Furthermore, datasel curation for these approaches presents greater complexity compared to methods only using text-based CoT. This is because developing language interaction capabilities typically requires cach observation annotated with multiple question-answer pairs, in contrast to the singular text output per training example requisite for text- based CoT reasoning.
4.3.2 Models Providing Action Interaction Capability
We now review models that can plan a trajectory by considering instruction from the user. We present an overview of these methods in terms of training dataset, training approach and evaluation in Tab.
Action Interaction Capability Datascts. All three approaches in this category (refer to Fig. are trained and lested on synlhetic dolasels collected using CARLA simulator (Dosovitskiy et al. 2017 This is likely because annotating data with pairs instructions and actions is relatively easier for synthetic data as the simulator state includes comprehensive information about the scene. Furthermore, unlike other approaches discussed in this paper, both DriveMLM and LMDrive rely on both lidar and camera. Additionally, LMDrive stands out with the functionality to consider notice instructions such watch the tunnel coming up. Shao
20

---

Published in Transactions on Machine Learning Research (03/2026)
et al. (2024a) introduce specific benchmark called LaugAuto-Notice to mcasure the performance of the model to respond to such notice instructions, demonstrating that LMDrive effectively leverages notice information to improve driving performance. Another crucial capability that such models are expected to have is a mechanism to avoid misleading instructions, as such instructions can result in unsafe consequences. Ideally, upon capluring a misleading instruction, the model should reject and follow safe trajectory. Both LMDrive and SimLingo incorporate this crucial capability into their models.
Evaluating Action Interaction Capability. Unlike language interaction, the evaluation measures for action interaction are not yet well-establsheed. In addition to the qualitative evaluation, which is limited in terms of the number of examples and tends to have selection bias, two different approaches are adopted to evaluate action interaction. Shao et al. (2024a) design a benchmark in which the model is instructed solely by the user, similar to an extended version of the navigational commands, where the standard CARLA performance measures are reported. As an alternative, Renz et al. (2025) directly measure the percentage of the trajectories that is following the given instruction, namely, the accuracy of the model to follow instructions.
A Discussion of Existing Approaches. Tab. provides overview of the approaches with action interaction capability. DriveMLM (Cui et al. 2025a combines EVA (Fang al.) 2023) and GD-MAE (Yang ct al. 2023) as vision and lidar encoders, respectively, with Llama LLM (Touvron et al. 2023a) through Q-Former modules. The model is then supervised to output four predefined commands (i.e., (keep, accelerate, decelerate, stop}) for controlling the accelerator, and five steering actions (i.e, [follow, left change, right change, left borrow, right borrow)). Similarly, LMDrive (Shao et al., 2024a] processes inputs from multiple cameras and lidar using a multimodal vision encoder including ResNet (He et al. 2016) and PointPillars (Lang et al. 2019). This encoder is initially pretrained using object detection, traffic light status classífication, and
trajectory planning before it is combined with Llama (Touvron et al.) 2023a) where the model is supervised for trajectory planning. Unlike other approaches, the LLM is kept frozen during the training. To determine if the given user instruction is completed, the model predicts an additional flag, making LMDrive the only approach with this functionality. Finally, SimLingo (Renz ct al. 2025) is built on Mini-InterrVL VLM Gao let al., 2024c) as an extension to CarLLaVA (Sec. [.2). Different from other approaches, SimLingo has @ both language and action interaction capabilities, as well as the option to use CoT reasoning (please refer to our discussion inference time-accuracy trade-off of CoT reasoning in Sec. 4.2.2 for further details). The model is trained in single stage dataset including training examples for these functionalities. The instruction following capability, coined as action dreaming, aims to align the predicted trajectory with the natural language instructions. Consequently, it is shown that action dreaming helps improving the driving performance more than the pure language tasks, i.c., VQA and CoT reasoning, as shown in the paper. SimLingo also pay altention to the resampling of the dataset via carefully creating data buckets for predefined driving characteristics and then assign different sampling ratios to each bucket.
Summary of trade-offs: Models Providing Action Interaction Capability S
The approaches with only action interaction capability offer the distinct advantage of trajectory planning conditioned on user instructions, serving as a potentially valuable additional feature. Since this functionality itself docs not require generating text output, such models can be deployed efficiently in real-world systems. However, the dataset curation protocols for developing these models remain insufficiently investigated for real-world systems. Essentially, the model requires training with a substantial volume of text input (user instructions) paired with corresponding output trajectories. Crucially, the text input should exhibit sufficient variability to enable robust responses to diverse user instructions. Akin to other subgroups lacking text generation, these models could inherently suffer from a deficiency in explainability especially when CoT reasoning is not employed either.
5 Foundation Models Guiding Trajectory Planning
An alternative way of utilising FMs for trajectory planning is to use an existing approach for AD, such as a modular or an E2E one (see Fig. aab))) and transfer the knowledge of chosen FM into either only during training or during both training and inference. A key distinction among these approaches come from their choice of using FM during inference which, of course, results in increased computational cost and
21

---

Published in Transactions on Machine Learning Research (03/2026)
T as the set questions
(Top) VLP
T as the description of an agent/iung /waa clement/ /scctic
Text Encoder
VLM
Vision Encoder
Vision beatures
Pereeption
Lavent Interface
Predictton
Lateui Interface
Planing
Adapter
Distilation Loss
(Center) Generic E2E-AD Model
(Right) VLM-AD
T
LLM
O
Aux Tasks
0
Reenack: Black
(Bottom) DiMA (red arrows) and Solve-E2E (purple artows)
T.0 and Ong refor to mirhools.
Figure 7: Overview of the approaches that uses an FM only during training for knowledge distillation. (Center) generic E2E-AD model (please refer to Sec. 2 for the details.) (Top) VLP prompts the model with the agent/map elements/sen /scene descriptions to align the representations of CLIP text encoder with those of the E2E-AD model. The arrows from the text encoder represent the representations. (Right) VLM-AD prompts VLM with the privileged information (the red line the input image) and few questions to align the representations of the E2E-AD model. (Bottom) DiMA and Solve jointly trains E2E-AD model and the FM, where the LLM makes predictions by conditioning on the representations of the E2E-AD model. Please refer to Sec. 5.1 for further details.
memory requirements. Considering that, based on our taxonomy (refer to Fig, 4 we discuss the models using knowledge distillation only during training in Sec. 5.1 and those requiring an FM also for inference in
Sec. 5.2
5.1 Models Using Knowledge Distillation Only During Training 7
As this group of approaches carries out knowledge distillation only during training, they do not need the FM for inference. Fig. 7 provides an overview of the existing approaches within this category. Among the first approaches, VLP Pan et al. 2024a) aims to align the representations of an E2E-AD model with an off-the-shelf CLIP [Radford et al. 2021). To obtain CLIP representations, only the text encoder of the CLIP is used. Specifically, the text encoder is prompted by (i) the descriptions of cach agent and map elements such as their semantic class and location, and (ii) a planning prompt,, such as "the self-driving cur is driving in an urban area. It is going straight. Its planned trajectory into the E2E-AD model in two levels as shown in Fig. Top), using contrastive learning. While the agent and map element representations of the planning prompt is incorporated into its ego query as the input to the planning module. Similar to VLP, VLM-AD (Xu et al. 2025bb keeps the FM frozen but instead prompts a VLM, GPT-4o (OpenAl, 2024), with (i) the front image including the privileged information of the future trajectory of the ego shown by the red line on the image in Fig. Rightt, and (ii) six different questions to obtain a meta-action and reasoning from the VLM. These responscs are then structured using onc-hot encoding and a CLIP text encoder, and an adapter layer is appended to the E2E-AD model to predict this structured output. Finally, cross-entropy loss is used for knowledge distillation.
Otraj" 7. Then, these representations are distilled
distilled into vision features of the E2E-AD model, the representation
Different from the previous approaches, DiMA (Hegde et al. 2025) and Solve (Chen et al. 2025) train E2E-AD and LLM jointly for knowledge distillation. Specifically, as shown in Fig. 7 (Bltoo)) both approaches propagate representations of the E2E-AD model to the LLM of LLaVA-v1.5 (Liu et al. 2024a), such that the LLM predicts the trajectory and a text output by conditioning on these representations. While Solve propagates only the representations after perception, DiMA aims for more comprehensive distillation. As a result, in DiMA, (i) BEV features, agent/map representations, and the ego query are all propagated to the
22

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5: Design choices of the models that transfer knowledge from an FM during inference. * Frozen, Standard fine-tuning, ④ LoRA fine-tuning. Broadly speaking, the last three columns correspond to the feature volumes (latent interfaces in blue) in Fig. (b) For P++ the used FM type for the main experiments is not clear in the paper, hence it is marked as not available (N/A).
Model
Sensor Toput (Krxs)
Used FM
Transforred Knowledge from FM
Inference (2)
How Encode
X. Trazaforrre Inla
Viston
Prediction Iaput
Planning Input
X
Scane doscription
encodder
eloecription, driowr's gome, and logte
TLAVA
comeras
VIT-L Vicuno v1.5
oction.
Leamuoble eubeedding loyer
lnng
27/5
307 CHIIEETN
VIT- Vinume-l1.
Mittosction
Oue-but smaoling
Tian <t al 2024
Pront camera
Quenn VL
Trajectory
None
/
Soloe 10bx-E25-Ammm
Clen al. 10/25
302 cattarna Seatures
LLAVA-L--IEE SILLM
Trajasury
None
/
DiHAA-Dse!
Lngdl C 2026
J03 camera Seatures
LLM planning feotures in the last layer None
/
Wang JIlAI
XIF-Ordee
303 comeras
Liamad
Weights of worring funition to selict the most sutibll trajectory among mud- None
VOP--Asdo al. 2025
Foont
Quenz-VL
VIM Senturee of the detected objects, action. trajectory propocals
None
Feisteno preiettemined objects (eg, obstsele))
Rinary encuding for object hoarsslle inbooblngg meto: octioos
RasionA
traffic lights,
Jian
DviseDriser
Voctosiand encoding Llunnd the scatn
Ego states, occupancy odjo- cant scane, the state the trallic light, lane change and velecity decisions
Leaemable vaabaskding baver
Jaan st 20550
LLM (red arrows from LLM to the blue interfaces in Fig. 7 Bottomm),, and (ii) a distillation loss is introduced to align the representations of the LLM and the planning module of the E2E-AD (red arrow from the LLM the planning module in Fig. BBttoom)). Furthermore, DiMA includes auxiliary tasks such as masked BEV token reconstruction for effective representation learning during training. Both approaches fine-tune the LLM using LoRA (Hu et al. 2022) for the trajectory planning task.
Summary of trade-offs: Models Using Knowledge Distillation Only During Training ?
These approaches are unique in this review by their elimination of FM dependency during inference. Consequently, the resulting AD models typically have fewer parameters and exhibit a higher inference rate, thereby facilitating the deployment of computationally efficient systems. However, this indepen- dence from FMs excludes access to natural language interaction capabilities. Specifically, such models cannol process user questions instructions, and constrained in terms of model explainability. Moreover, the complexity of the dataset curation process varies depending whether the FM is jointly trained and the specific type of knowledge being distilled, which often necessitates additional input-output pair annotations.
5.2 Models Using Knowledge Transfer During Inference
Unlike the methods in the previous section, the approaches we discuss here are the ones employing FMs during both training and inference for more effective knowledge transfer. Tab. 5 provides the main characteristics of these approaches, including the type of knowledge transferred from the FM and how this knowledge is integrated into the AD model. In the following, we elaborate on these approaches based on the type of knowledge each transfers to the AD model, which is usually a (i) scene description involving perception or prediction features, (ii) a planning decision such as a meta-action or trajectory, or (iii) both.
Methods that Transfer Scene Description. These methods aim to transfer scene descriptions, such "n black van driving in the ego lane, away from the ego car", taken from VLM-F2E (Liu et al. 2025
23

---

Published in Transactions on Machine Learning Research (03/2026)
Specifically, Liu et al. (0255) prompt an off-the-shelf BLIP-2 Li et al. 2023ap with front image of driving scene to obtain such a description. As this description is essentially a text, CLIP text encoder (Radford et al. 2021) converts it into a text representation. Then, the representation is mapped to shifting and scaling factors using MLPs to update the BEV features of the E2E-AD model, in this case UniAD (Hu et al., 2023b). DME-Driver (Han et al., 2025) follows a similar approach, including knowledge of the scene description as well as the driver's 's gaze and the driver's logic on the taken action, such as the presence of the pedestrians for slowing down. Particularly, given the front view of the driving scene, LLaVA (Liu et al., 2023] is fine-tuned to produce the text output, which is converted into embeddings using BERT (Devlin et al.) 2019). Finally, BEV features of the F22-AD model attend to these embeddings through cross-attention before being fed into the occupancy prediction and planning modules of UniAD (Hu et al. 2023b). Though these methods show that such a knowledge transfer is useful, they do not guide the E2E-AD model erplicilly with a planning decision such as a meta-action or a trajectory, which we discuss next.
Methods that Transfer Planning Decisions. These methods transfer the planning decisions of the FM to the AD model, usually in the form of meta-action trajectory. For example, Semna-E21 (Jiang et al. 2024) transfers the meta-action of the VLM by fine-tuning it specifically for this task. The model is fine-tuned in multiple stages to progressively specialise the VLM for planning: In driving fine-tuning, the VLM is supervised with VQAs in driving scenarios, followed by planning mne-tuning for meta-action classification. After the VL. is trained, it is frozen and the meta- of the VLM is converted into on embedding using a learnable layer. Then, in order to benefit from the knowledge of the VLM, the planning query in the AD model, VADv2 (Jiang et al. 2026), attends to this meta-action embedding of the VLM. DifVVLA (Jiang et al. 2025a) also relies on Senna-VI.N VLM but with two main differences. First, instead of a learnable embedding layer, a one-hot encoding of the mcta-action is passed to the planning module. Second, the VADv2 planner is replaced by diffassio planner (Liao et al. 2025), which generates the trajectory by conditioning on this meta-action encoding as well as the BEV features, map and object queries sequentially. The approach is trained and evaluated using the NAVSIM v2 benchmark (Cao et al. 2025b).
Differently, some approaches, including DriveVLM, Solve and DIMA, fine-tune their VLMs for the trajectory planning task. This manifests itself as an additional advantage of combining the planned trajectories from the AD model and the VLM for potentially improving the driving performance. To begin with, DriveVLM- Dual (Tian et al.) 2024) uses the trajectory of the VLM as the imput query of the planning module of the E2F-AD model, such that the planning module refines it further. This approach is incorporated into UniAD (Hu et al.) 2023b), VAD (Jiang et al. (023) and AD-MLP (Zhai et al. 2023a). Similarly, Solve- 12E-Async (Chen et al. 2025) uses the trajectory of the LLM as an additional query of the planning module in the E2E-AD model, yet asynchronously. That is, to account for the longer inference time of the VLM, (i) the prediction horizon of the VLM is designed to be longer than that of the E2E-AD model, and (ii) the E2E-AD model uses the last predicted trajectory of the VLM as the additional planning query. Alternatively, DiMA-Dual (Hegde et al. 2025) transfers the representation yielded by VLM for trajectory planning. Specifically, max-pooling is used on the last layer features of the planning representations obtained in the VLM and E2E-AD models, and then the E2E-AD model predicts the trajectory from the resulting pooled features. Finally, as a different approach, HE-Drive (Wang et al. 2024a) fine-tunes Llama3 (Llama Team, Meta Al 2024) to output the weights of a function scoring candidate trajectories. Specifically, Wang ct al. (2024a) use a diffusion-basse planner to sample multiple candidate trajectorics, each of which is scorod factors such as collision risk, larget speed compliance and comfort. For example, if there is a stopped vehicle in front of ego and the ego needs to slow down, the weight of the target. speed compliance is increased by the VLM, thereby helping to select the best trajectory.
considering various driving-relate
Methods that Transfer Scene Description and Planning Decision. Some approaches transfer both a scene description and a planning decision. VDT-Auto (Guo et al., 2025) fine-tunes Qwen2-VT. (Bai et al. 2023b) on driving-related VQAs to output object descriptions, as well as a meta-action and trajectory. The corresponding representations in the VLM are then transferred to the AD model by freezing the VLM. Specifically, these embeddings are used as inputs to diffusion-lused planner (Liao et al. 2025). which refines the trajectories conditioned on these embeddings and the vision features. Alternatively, FasionAD++ (Qian et al. 2025b) guides the VLM to output (i) planning state including binary variables indicating the existence of, e.g., traffic lights, obstacles, intersections; and (ii) a meta-action. The planning state updates BEV
24

---

Published in Transactions on Machine Learning Research (03/2026)
features, agent, map and ego queries through an adapter layer, while the meta-action is incorporated into the ego features only. As a result, F provides knowledge transfer at multiple levels into the E2E-AD model. Additionally, ++sionAD++ takes the inefficiency of the VLM into account by referring to the VLM only when the uncertainty of the AD model increases beyond a certain threshold. Unlike other methods that. we discuss this section, AsyncDriver (Chen el al.) 2024ch transfers LLM knowledge to the planning module of modular approach (sec Fig. B())) Specifically, it fine-tunes an LLM predict useful information for AD by appending it with an assistance aligrement module. Conditioned the output of the LLM, this module predicts relevant perception features such traffic light states, occupancy of the adjacent lane, well as features affecting the trajectory planning such as lane change and velocity decisions. The latent encoding of the LLM, used as the input to the assistance alignment module, is propagated to the modular planner via a feature adapler. To align the LLM and the modular planner, they are trained jointly on the nuPlan dataset (Caesar et al. 2022). During inference, the latent encoding from the LLM is passod asynchronously to the modular planner improve the planning decision, maintaining the previous high-level instruction features during intervals. This asynchronous connection between the separate real-time and LLM-based planners provides balance between quality and inference speed for responses.
Summary of trade-offs: Models Using Knowledge Transfer During Inference
This category of approaches encompasses systems that integrate a FM with an AD model. The inference latency of such coupled architectures relies upon specific design parameters. For instance, contimuous reliance on the FM by the AD model can substantially increase inference overhead, whereas selective and conditional FM invocation can enhance average computational efficiency. Additionally, the inference time is influenced by whether the transferred knowledge is obtained following AR steps within the FM. Regarding explainability, the integrated FM component. might generale natural language explanations; however, ensuring semantic aligument between the AD model's decision decision-making processes and the FM's textual outputs presents considerable challenges. Similar to approaches employing knowledge distillation, the complexity of dataset curation depends upon whether the FM undergoes joint training and the specific nature of the information being transferred, necessitating the collection of additional text input-output pairs.
6 How Open Are Data and Code of the Existing Approaches?
(1) Traliing
(3) Evaiuntio
(1) Depitymmnnt
Code
Toining Code
Driving Evaluation
Cidie
Evalustion Code
Language/ Action
Infixencc Code
Codo
prblllely
cial tox tocluding ressarch
commrendtlly
licoeee
Model Weights
Deiving
Language/ Capabilities
Driving Evalsation
Languspe/
publiely aclosed in
Publicly available comsscrciilly
Troining
Tata
available licomee
Dataset
Training
Tatnest
Weights
uaage
conenneeetclll fox ressanch
(a) The code and data for each stage of development
(b) The levels of openness for each asset
Figure 8: Representation of different stages of model development, and the levels of openness.
Open models and datasets play vital role in the advancement of research and acceleration of practical deployment. They enable rescarchers to quickly build upon existing work to improve the state-of-the-art while allowing practitioners to build high-performing real-world systems without the exlensive time needed to reproduce the existing work. It also fosters user trust as transparency reveals strengths and weaknesses of approaches. Morcover, open-source resources lower oconomic barriers, promoting adaptation in low-income regions. Considering the importance of openness, we present the level of openness for all the 37 approaches for trajectory planning considered in this work.
25

---

Published in Transactions on Machine Learning Research (03/2026)
Table 6: Openness characteristics. While every effort has been made to ensure the accuracy of this table, we cannot guarantee its completeness or correctness. The information is provided for general reference only and should not be interpreted as legal advice. If in doubt, please confirm with the corresponding authors. The table reflects the status as of 08 October 2025. Entries marked as N/A indicate that the asset is "not applicable" to the method. We further outline the conventions we used while classifying the openness of the approaches in Appendix A which also clarifies the symbols * and ** We label the first method of each taxonomy group with its corresponding icon.
Tuaising
Puslrsaion
Deploymeck
Neebor
Code
Model Weighte Protmited F51s
Driving Twvining Dotant
Language/ Autkan Thiining Tetlow
Betsing Evalucian Clude
Driving Dalsnnt
Actiien Evaluriaa
Loupssp Aackis Dationt
Modee Wi Wright.
C) Apedkiaa
D?
A/A
8/A
N7A
C3 Apucled
CL
Us Quee 3.B
D. Cultes
K/A
C1
CARLA Langest
X/A
8/A
C1
D1
BAIB-YEX
DAII--YXX
Ci
Komeer
DL GPP-33
D2
K/A
EI
8/A
K/A
C1
D1
X61
CL
Quee VL.
Sivee
8/A
C1
X/A
8/A
C1
D1
Jatn
ne Greed.-V
8/A
2/A
37A
C1
01
Apecketi1
Viere +1
De
BESS
D2
K/A
A
8/A
8/4
C5
D1
WORD
K/A
8/6
C3
GPP/35
DI
D2
8/A
C3
8/A
X1A
C5
01
Reag al 5000
C1
CARIA
8/A
C1
LAV
X/A
8/A
C1
11
K/A
N/A
M/A
01
50
101
C1
Do
n2 BEOX
Cratees en lemee
No keow
C1
Certen an Hovee
DJ
No Home
C1
D1
2631
BUI-22
A
01
Evens
D1
Di
DI
D1
C1
Quea2
w as lavess
A
A
radeinee--g Q4
Apuiled
D1
De
G2
D2
De Oritee
Worgee
Coddess
Elall v1.
ot
Coler
LEAVA
Ouivves
Owlen
Chnoe
Di
Apalsto
12
D1
DS
DS
Apui
css
cs
Dor
el
(o LLMMATT
Caicas
Clatses
CARLA Towals
Cuitecs
Di
C1
DI
Apeckiuu
De
B3
CAKLA
ds
Dar
100
G3
DS
D2
Cs A
Agel
To
D3
ULi
M/A
C1
X/A
C1
D1
DI
D2
8/A
CARLA
DS
X/
3/A
D1
C1
LLaVA v2.5
I'm
N/A
C1
NA
8/A
07
01
be
D3
K/4
8/4
C1
Ls BUP?
Sinee
D2
N/A
C1
CARLA Tim
XA
3/A
C1
DL
LiaVA
Apaluta
U3
122
8/A
D2
2/A
3/A
Apallad
D2**
Ting d al pe.0
C1
LLaVA v2.5
Ii
n> NAY 12
N/A
NAVILL
N/
87A
C7
11
Do
VI.
K/A4
C1
ELAV .15
D2
N/A
C1
X/A
3/A
C1
01
Eagle
LLaVA v1.
C1
D2
8/A
D2
2/A
3/A
C1
01
Dae Al 1013
C1
Do
D3
K/A
C1
8/A
8/4
C1
D1
Unsosialee
D2
Ap=1.00
U1
2
D2
K/A
Aaxidnn
saPias
12
X/A
N/A
D
C3
Dg
26

---

Published in Transactions on Machine Learning Research (03/2026)
0%
10%
90%
30%
9095
50%
0093
709
80%
L005E
Training Code
Model Weight of Pretrolned FMs
Driving Training Dataset
28
Longuage/ Action Capabilities Training Dataset
Driving Evaluation Code
Driving Evalusatio Datzent
Langauge/ Action Rvaluation Code
Langsagee/
Actionn Action Evalustion Daiawut
Inference Code
28
Model Architecture and Weights
31
D1/C1
D8/Cs
Figure 9: Distribution of openness levels across data and code assets for the 37 surveyed methods. Numerical labels on the bars represent the count of approaches in each openness category per asset.
Following Eiras et al. 2024 consider training, evaluation and deployment different stages of building model, and accordingly score the assets, which is "data" and "code". For asset, higher score implies higher openness level. Since we study trajectory planning methods that leverage FMss, the underlying assets for our use cases require additional elements, as shown in Fig. B(a). Specifically, we include pretrained FM weights in the training stage as an additional data asset. We also consider driving and language/ /action datasets separately, as language annotations are generally derivod from existing driving datasots, and multiple language annotations can exist for the same driving dataset, e.g., muCaption (Yang et al., 2025b), nuX (Ding et al. 2024a), nuScenes-QA (Qian et al. 2024) and DriveLMMnuSeenes (Sima et al. (022).. For the levels of openness, we score each asset between and 3 to provide targeted information for research and commercial purposes (refer to Fig. g b)), defined as:
1 implies an asset, i.e., code or data, is not publicly released or is released with a license that restricts
both non-commercial and commercial usage, including research.
2 represents that the asset is publicly available and can be used for commercalllaal purposes, including research, but not for commercial use. One example license for this is Creative Commons 4.0 International.
CltributioonNonCommmeeeo
3 is for assets that are publicly available and can be used for both research and commercial purposes, e.g., those with Apache License v2.0 or MIT License.
In Tab. D we classify the code and data of all the approaches using these levels of openness. We observe that there is no approach with all assels being available for both research and commercial purposes. Only five of the approaches (i.e., Omni-Q, Orion, LMDrive, SimLingo and AgentDriver) released all of their assets openly with some assets limited for commercial usage. Four of these approaches are FMs tailored for trajectory planning, and one of them is an FM guiding trajectory planning of modular AD model. Consequently, open-source implementation is available for FM guiding trajectory planning for an E2F-AD model (rows 23-36).
To demonstrate and summarise per-asset openness status, we provide the distribution of the openness levels of each asset in Fig. g The figure indicates that the model weights of pretrained FMs are commonly open for rescarch (31 out of models), cnabling the models to be developed. On the other hand, this doesn't 't necessarily translate into their derivatives being open. Specifically, the model archilecture and weights is the most restricted asset with only models providing them openly for research and commercial usage (D2 or D3). Similarly, as another crucial aspect of reproducibility, only 10 models openly share the training code (C2 or C3). This discrepancy creates a significant bottleneck: without model parameters and implementation details of the training pipelines, it is difficult for the community to build upon the previous research. Ultimately, this presents a challenge for reproduction and reuse.
We also observe a clear distinction between the openness of synthetic and real- datasets. Synthetic datasets, generated typically via the CARLA simulator (Dosovitskiy et al. 2017), often fully-open (D3),
27

---

Published in Transactions on Machine Learning Research (03/2026)
allowing for unrestricted use in both research and commercial projects. In contrast, the real-world datasets used by the surveyed methods, including nuScenes Caesar et al. (2020) and BDD-X Kim et al. (2018) are commonly restricted to non-commercial use cases only (D2). This gap creates friction when translating research into industrial practice and may introduce inconsistencics in standardised evaluation across academic and commercial domains.
7 Open Issues and Emerging Directions
In this section, we identify and discuss the open issues within our scope, while also considering emerging directions in the broader research landscape.
Deploying these models can be challenging due to the high inference cost, especially when CoT reasoning is used. Except for few models using knowledge distillation only during training (Sec. 5.1 all other approaches in our scope require an FM for inference. Consequently, mainly due to the large number of parameters and AR generation of the output, their latency is commonly longer than what is suitable for AD. This is usually accepted as 10-30fps to provide effective responsiveness to the changing environment and to match the latency sensor data (camera, lidar) (Caesar et al. 2020 Sun et al., 2020 Ettinger et al. 2021). For example, Orion (Fu ct al. 2025a has more than 7B parameters (see Tab. 1 and an inference rate of 0.8fps on an Nvidia A800 GPU, even without CoT {Fu et al.) 2025b). Additionally, referring to our previous experiment in Sec. 4.2.2 the throughput of SinLingo (Renz et al. 2025); a relatively smaller model than its counterparts, using only the front view, without and with CoT reasoning, is 3.6fps and 0.8fps, respectively, on an Nvidia A4500 GPU respectively. While, efficient processing has been recently considered in number of models, such as the use of separate slow, reasoning-based FM systems alongside a fast decision system for rapid response times (Qian et al. 2025b; Zhou et al.) 2025b Wang et al.] 2025c), fast control cycles have generally not been a priority for current models. Therefore, more efficient approaches are necessary to be able deploy such models in the real-world.
The image shows a road with cows grazing on the side. It is important to note that driving on this road is not safe for cows. The cows are crossing the road, which is clear and dangerous situation. To ensure the safety of cows and other animals, it is crucial to avoid driving on the road where cows are present, Instead, it is recommended to fet the cows pass on the road or to find safer area to drive.
For mile
Mini In
No, the current route not too busy.
Is it safe to drive the road?
SimLingo
Figure 10: An example showing vision-language collopse in the VLM of SimLingo after fine-tuning. We first. prompt Mini-InternVI, the VLM that SimLingo is initialised from, and then SimLingo, i.c., Mini-InternVI. after fine-tuning, with the image and text shown in the figure.
Fine-tuned VLMs become less capable of interpreting the world lvision-language collapse). The superiority of FMs tailored for trajectory planning in benchmarks demonstrates that they effectively leverage transfer learning via fine-tuning the pretrained models. However, this fine-tuning procedure can result in the loss of capabilities of the VLM that could potentially be more helpful for trajectory planning. This crippling effect might be emerging due to the negative effect of fine-tuning, studied concept forgetting (Mukhoti ct al. 2024) which is an extension of the traditional catastrophic forgetting (Kirkpatrick et al. 2017) for fine-tuned FMs. We show this effect in Fig. 10 where we prompt the VLM used in SimLingo with the same image and text inputs before and after fine-tuning. For comparison purposes, we use one the inputs we employed for prompting GPT-4o Fig. [ where GPT-4o provides a comprehensive accurate response. Although it is not as comprehensive as GPT-4o, the response of Mini- Mini-InternVL, a relatively small VLM that SimLingo is initialised from, is also quite accurate. Specifically, it captures that the cows are crossing the
28

---

Published in Transactions on Machine Learning Research (03/2026)
AAA
What are the traffic rutes that you pay attention?
Then, could you please park between the green motoregele and black car the left?
g
The traffic light is green indiearing that vehicles can proceed through the intersection. Stay within the designated lones and avoid crossing solid lines. Note the presence of parked vehicles and ensure
Sure, parking now [Trajectories are planned and executed until parking is completed]
block dricewags or entrances. Adhere to the speed limits for the area.
It looks like a historical town. Which town is it? Is there anything worthwhile visic?
I am done with my visit. Please start drizing and continue to my hotel in Oxford.
It appears that you are Woodstock, a histories] town Oxfordshime. It is known for its historical huildings and proximity to Blenheim Palace, UNESCO World Heritage Site.
Okay, I amn dricing there (Trajectories planned and executed for reaching the target destination.]
want to park here and see around for some time, la parking allowed here?
I getting late for my meeting, Please overtake the ahead and drine faster.
Yes, parking appears io be allowed Aere. There ure vehicles parked along the side of the road, including a greem motoreycle and black There are no visible signs indicating that parking prohibited.
Sure, will do my best without violating traffic riles. Trajectories are planned and executed by adjusting the speed accordingly.]
Figure 11: Example VQAs and user instructions that the current models lack.
road and suggests either letting the cows pass or finding safer area drive. On the other hand, after fine-tuning, SimLingo provides self-contrudictory response, mentioning that it is not safe to drive followed by an explanation that the route is not busy, implying safe route. Furthermore, SimLingo does not provide any insight about the enviroument, such as the presence of the cows. Further research and thorough analysis is essential to ensure preservation of necessary capabilities of VLMs after fine-tuning on AD-specific datasets.
Towards Agentic VLAs. Existing approaches with language interaction capability generally focus on interpreting the current scene to provide explainability to the user. There however potential to offer further use cases if the question-answar pairs are tailored accordingly. For example, the user can ask non-driving- related questions such as the historical details of landmark in town that the ego driver passes through, Alternatively, the model could help people with visual impairments understand their
illustrated in Fig. surroundings, such as why the traffic is stopping. One limitation of current models with action interaction capability, is that they are limited to instructions that can be executed in a short time horizon. DLiveMLM and SimLingo consider instructions to change speed or lane, which can be completed within the horizon of a single trajectory output. As a result, the model does not need to remember the instruction over mullippee calls, or check if it is completed, which impractical. Though the instruction set used in LMDrive is larger with 56 different types of instructions, it still doesn't require the model to break down a human-like complex instruction into steps in the example "please start driving and continue to my hotel in Ozfort", 7, illustrated in Fig. More sophisticated planning models in the robotics literature (Huang et al. 2022 Belcak et al.
2025 can help leverage similar capabilities in real-world for AD models.
Challenges of ensuring safety of action interaction capability in a real-world setup. Providing action-interaction capability allows substantial flexibility of commands that can be provided by a user. However, due to the open domain of user instructions, there are substantial challenges in ensuring that the vehicle operates safety, regardless of the commands given. Existing methods do include training of action
29

---

Published in Transactions on Machine Learning Research (03/2026)
safety in the dataset, described in Section domain remains open issue. Furthermore, all models investigated in this work with action interaction capability are trained and tested using synthetic setups using CARLA simulator. As domain shift results in a
3.1 however ensuring safety with an unrestricted command
significant drop in performance (Luo et al. 2018 Zhou et al. 2023a Oksuz et al., 2023: Kuzucu ct al. 2024), deploying existing models without addressing sim-to-real gap is impractical. Therefore, designing meth to address sim-to-real gap and curating real-world training datasots is necessary to be able develop, and eventually deploy, models with action interaction capability.
Further analyses necessary to identify what is significant for improving driving performance. There are major differences in model and experimental design choices across methods, which makes it difficult. to understand why a method makes a difference. For example, Orion is trained on the B2D-Base dataset (Jia et al.) 2024) with around hours of video clips, combines an off-the-shelf vision encoder with a custom Q-Former with memory blocks for videos, trains the model three stages, has more than 7B parameters, uses a variational autoencoder as trajectory planning head, and employs comprchensive CoT including scene description, scene analysis, action reasoning, and history review. This models achieves 77.74 driving score the closed-loop B2D benchmark, significantly outperforming E2E-AD approaches such VAD with 42.35 driving score (Jia et al. 2024). On the other hand, SimLingo employs Mini-InternVI, (1B paramcters) as an off-the-shelf VLM, fine-tunes it on more than 200 hours of video in a single stage relying on a data resampling approach coined as bucketing, uses a different set of language annotations and CoT, and finally achieves a driving score exceeding 85, These major differences between models, training approaches as well as training datasets make difficult to understand why model makes a difference. As a result, a deeper understanding of how these factors affect the model is necessary.
Lack of standardised benchmarks and metrics for FMM-bsed trajectory planning. Building upon the aforementioned challenges, the establishment of standardised benchmarks and metrics is essential to facilitate rigorous and equitable comparison across different approaches, and to systematically identify the koy factors contributing to performance improvements (c.g., training data charactcristics, architectural design choices, model scale). Such benchmarks and evaluation metrics should address multiple dimensions of model performance, including:
Driving performance under resource-comstruinng settings. FM-basod approaches for AD exhibit considerable variation in model size. For instance, SimLingo comprises 1B parameters (Renz el al. 2025), while Orion contains approximately 7B parameters (Fu et al. 2025a). Consequently, inference latency varies substantially across methods, further influenced by design decisions such as CoT usage. Therefore, evaluating driving performance under controlled resource constraints (e.g., bounded inference time) becomes crucial for assessing the practical viability of FM- methods in real-world deployment scenarios.
Language interaction and action interaction capabilities. Beyond core driving competencies, certain FMs designed for trajectory planning incorporate additional capabilities for language and action interactions. To comprehonsively evaluate these capabilities, standardised benchmarks encompassing both real-world and simulated data are necessary.
Reasoning of the models. Furthermore, analogous to the common benchmarks used for evaluating FMs, standard benchmarks for evaluating the reasoning capabilities of FMs tailored for trajectory planning are needed. Drawing from the broader FM literature, the ARC-AGI benchmark (Chollet et al. 2025 exemplifies this approach by designing visual reasoning tasks that humans solve intultively while FMs struggle. Similar domain-specific benchmarks should be developed for AD to assess reasoning abilities when processing complex linguistic inputs, as well as multimodal language-visse vision grounding capabilities. Developing such benchmarks presents substantial challenges and warrants dedicated attention from the research community.
Improving driving and evaluation with World Models. Training current FM-based methods for AD typically relies on recorded driving demonstrations from an expert, which can be either derived from another model with access to privileged information. Consequently, model performance is fundamentally bounded by the capabilities of the expert, and the availability of training data remains inherently limited. Furthermore, existing closed-loop simulators exhibit significant constraints: they are either characterised by low fidelity, as exemplified by CARLA (Dosoviskkiy et al. 2017). or impose substantial
30

---

Published in Transactions on Machine Learning Research (03/2026)
computational demands, as in the case of novel viewpoint syuthesis approaches (Kerbl et al. 2023 In contrast, world models aim to learn the underlying dynamics of the autonomous vehicle's operating environment, thereby enabling the generation of plausible future states. Therefore, world models can be leveraged to synthesise novel training data (Ren et al. 2025 Zhao et al.) 2025b) as well as to facilitate evaluation procodures that emphasise rare and safely-critical edge cases, thus addressing long-tail distributional challenges. Additionally, world models can also be integrated with driving models, allowing world dynamics to influence driving decisions (Wang et al. 2024d), for example by rolling out possible futures using the world model and selecting the most suitable trajectory for driving.
8 Conclusive Remarks
In this paper, we provided comprehensive review of trajectory planning methods that utilise an FM. To offer complete and coherent perspective, we introduced taxonomy of these methods based on how an FM is employed. Using this taxonomy, we discussed the corresponding approaches separately in detailed and comparative manner, providing a unified yet critical point of view. We also investigated the openness of the approaches to assist practitioners and researchers in selecting suitable models. Finally, we identified open issues that are critical for developing practical models with the desired functionalities. With this review, the community can better understand the current state of the field and the directions to pursue for developing more capable solutions for AD leveraging FMs.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millicah, Maleolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic propositional image caption evaluation. In European Conference on Computer Vision (ECCV), 2016.
Claudine Badue, Rånik Guidolini, Raphael Vivacqua Carneiro, Pedro Azevedo, Vinicius B. Cardoso, Avelino Forechi, Luan Jesus, Rodrigo Berriel, Thiago M. Paixão, Filipe Mutz, Lucas de Paula Veronese, Thiago Oliveira-Santos, and Alberto F. De Souza. Self-driving cars: A survey. Expert Systems with Applications, 165:113816, 2021. URL
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023a. URL httts:://///tt org/ abs /bs/ 2309 16609
Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Pong Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023b. URL http::://xxxvv.orr///// .t0./..../...226666....
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005.
Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. Small language models are the future of agentic ai, 2025. URL https //arxiv.org/ate 250ea. 02153
31

---

Published in Transactions on Machine Learning Research (03/2026)
Stuart Bennett. The past of pid controllers. Annual Reviews in Control, 25:43-55 53, 2001. ISSN 1367-5788.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysatof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wa Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-languago-action models transfer web knowledge to robotic control, 2023. URL https://arxisn https/////xxvv.rr/abbbaa./ 2307 15818
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanic Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative interactive environments, 2024. URL https://arxiv. org/ / abs abs/ 15391
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. muscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Holger Cacsar, Juraj Kabzan, Kok Scang Tan, Whyc Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sanmny Omari. Nuplan: closed-loop ml-based planning benchmark for autonomous vehicles, 2022. URL https://arxiv.. http::////xxvooo//aabbba abs/ 2206 11810
Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aicllo, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Pseudo-simulation for autonomous driving. Tn Conference on Robot Learning (CoRL., 2025a.
Wei Cao, Marcel Hallgarten, Tianyu Li, Daniel Dauner, Xunjiang Gu, Caojun Wang, Yakov Miron, Marco Aiello, Hongyang Li, Igor Gilitschenski, et al. Pseudo-simulation for autonomous driving, arXiv preprint arXiv:2506. 04211, 2025b.
Dian Chen and Philipp Krähenbühl. Learning from all vehicles. Tn Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17222-17231, 2022.
Li Chen, Penghao Wu, Kashyap Chita,, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-lo-end autonomous driving: Challenges and frontiers. IEEE Tronsactions on Pottern Analysis and Machine Intelligence, 46(12):10164 10183, 2024a.
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi Wang, Tianyu Liu, and Baobao Chang. Towards end-to-end embodied decision making via multi-modal large language model: Explorations with gpt4-vision and beyond. In NewrIPS Workshop at Foundation Models for Decision Making, 2023a.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with 1lms: Fusing object-level vector modality for explainable autonomous driving. In IEEE International Conference on Robotics and Automation (ICRA), pp. 14093- 14100, 2024b.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Scbastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, Daniel Salz, Xi Xiong, Daniel Vlasic, Filip Pavetic, Keran Rong, Tianli Yu, Daniel Keysers, Xiaohua Zhai, and Radu Soricut. Pali-3 vision language models: Smaller, faster, stronger, 2023b. URL https://taatt //aaxiv/..ii////// atggabss//10100 099999999900999
32

---

Published in Transactions on Machine Learning Research (03/2026)
Xuesong Chen, Linjiang Huang, Tao Ma, Rongyao Fang, Shaoshuai Shi, and Hongsheng Li. Solve: Synergy of language-vision and end-to-end netsorks for autonomous driving. In Compuler Vision and Pattern Recognition Conference (CVPR), 2025.
Yuan Chen, Zi-han Ding, Ziqin Wang, Yan Wang, Lijun Zhang, and Si Liu. Asynchronous large language model enhanced planner for autonomous driving. In European Conference on Computer Vision (ECCV), 2024c.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linnustic tasks. In IEEE/CVF Conference Computer Vision and Pattern Recognation (CVPR), 2024d.
Jie Cheng, Xiaodong Mci, and Ming Liu. Mor: Self-supervisod pro-training for motion forccasting with masked autoencoders. JEEE/CVP International Conference on Computer Vision (ICCV), 2023.
Haohan Chi, Huan ang Gao, Ziming Liu, Jianing Liu, Chenyu Liu, Jinwei Li, Kaisen Yang, Yangcheng Yu, Zeda Wang, Wenyi Li, Leichen Wang, Xingtao Hu, Hao Sun, Hang Zhao, and Hao Zhao. Impromptu vla: Open weights and open data for driving sssoo-llnguuge--cttion models. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE trunsactions on pattern analysis and machine intelligence, 45(11):12878 12895, 2022.
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Are prize 2024: 2025. URL https://aaxiii http///awww...rraa/////aaaa///// 2412. 04604
Technical report,
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fiecunne language models. Journal of Machine Learning Research, 25(70):1 53, 2024.
Daniel Coclho and Miguel Olivcira. A review of end-to-end autonomous driving in urban environments. JEEE Access, 10135, 2022.
Erfei Cui, Wenhai Wang, Zhiqi Li, Jiangwei Xie, Haoming Zou, Hanming Deng, Gen Luo, Lewei Lu, Xizhou Zhu, and Jifeng Dai. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. Visual Intelligence, 3(1), November 2025a. URL http:://xx doi org//0
1007//84227-025-0008---
Yixin Cui, Haotian Lin, Shuo Yang, Yixiao Wang, Yanjun Huang, and Hong Chen. Chain-of-thought for autonomous driving: comprehensive survey and future prospects, 2025b. URL https://arxiv.orr org/ abs/2505. 20223
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
Kairui Ding, Boyuan Chen, Yuchen Su, Huan ang Gao, Bu Jin, Chonghao Sima, Wuqiang Zhang, Xiaohui Li, Paul Barsch, Hongyang Li, and Hao Zhao. Hint-ad: Holistically aligned interpretability in end-to-end autonomous driving. In Conference on Robot Learning (CoRL), 2024a.
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davit: Dual attention vision transformers. In European Conference on Computer Vision, 2022.
Xinpeng Ding, Jianhua Han, Hang Xu, Xiaodan Liang, Wei Zhang, and Xiaomeng Li. Holistic autonomous driving understanding by bird's eiy'seeyo-view injected multi-modal large models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CrioionnCCCC (CVPR), 2024b.
33

---

Published in Transactions on Machine Learning Research (03/2026)
Alexey Dosovitskly, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 1-16. PMLR, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL
Francisco Eiras, Aleksandar Petrov, Bertie Vidgen, Christian Schröder de Witt, Fabio Pizzati, Katherine Elkins, Supratik Mukhopadhyay, Adel Bibi, Botos Csaba, Fabro Steibel, Fazl Barez, Genevieve Smith, Gianluca Guadagni, Jon Chun, Jordi Cabot, Joseph Marvin Imperial, Juan A. Nolazco-Flores, Lori Landay, Matthew Thomas Jackson, Paul Röttger, Philip H. S. Torr, Trevor Darrell, Yong Suk Lee, and Jakob N. Foerster. Position: Near to mid-term risks and opportunities of open-source generative ai. In International Conference on Machine Learning (ICML), 2024.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, and Dragomir Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset, 2021. URL
brgg/bs//// 2104
10135
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Compuling, 149:105171, September 2024. ISSN 0262-8856. doi: 10.1016/j imavi..2024.
URL http: doi. 1016/0 .ww..i...2....11...... 105171
Daocheng Fu, Xin Li, Licheng Wen, Min Dou, Pinlong Cai, Botian Shi, and Yu Qiao. Drive like a human:
Rethinking autonomous driving with large language models. In IEEE/CVF Winter Conference on Applications of Compuler Vision Workshops CoccchopssAAACCCA (WACWW), 2024.
Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. Orion: A holistic end-to-end autonomous driving framework by vision-language instructed action generation. In IEEE/CVP International Conference on Computer Vision (ICCV), 2025a.
Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, and Xiang Bai. Orion official implementation. httpss///githul com/ 2025b. (Last accessed: 04 September 2025).
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166 2023.
Haoxiang Gao, Zhongruo Wang, Yaqian Li, Kaiwen Long, Ming Yang, and Yiging Shen. A survey for foundation models in autonomous driving, 2024a. URL ltttps///axxiv w4tp://20.r/aabb/22.....2
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In TEEE/CVP Conference on Computer Vision and Pottern Recognition (CVPR), 2020.
Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. In A Aduances in Neural Information Processing Systems (NeurIPS), 2024b.
34

---

Published in Transactions on Machine Learning Research (03/2026)
Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internyl: flexible-transfer pocket multi-modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):1 17, 2024c.
Gemini Team. Gemini: A family of highly capable multimodal models, 2025. URL https://axxiv.org/
abs/2312. 11805
Genie Team. Genie 2: A large-scale foundation world model. http:///eepmiinn.ggggl////ssssee genie-2-a-large hug.:a/ggeaaaaarrrouuoaaaaaaaaa-
2024. Last Accessed: 2025-09-07.
Genie Team. Genie 3: A new frontier for world models.
2025. Last Accessed: 2025-09-07.
Ziang Guo, Konstantin Gubernatorov, Selamawit Asfaw, Zakhar Yagudin, and Dzmitry Teetseukko.. Vdt-auto: End-to-end autonomous driving with vim-guided diffusion transformers, 2025. URL https://arxiv.orz
abs/2502. 20108
Wencheng Han, Dongqian Guo, Cheng Cheng-Zhong Xu, and Jianbing Shen. Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving. In AAAI Conference on Artificial Intelligence, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), 2016.
Deepti Hegde, Rajeev Yasarla, Hong Cai, Shizhong Han, Apratim Bhattacharyya, Shweta Mahajan, Litian Lüu, Risheek Garrepalli, Vishal M. Patel, and Fatih Porikli. Distilling multi-modal large language models for autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
GeofTrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In Deep Learning Workshop in Advances in Neural Information Processing Systems (NeurIPS), 2014.
Anthony Hu, Lloyd Russell, Hudson Yoo, Zak Murez, George Fedoscev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-l: A generative world model for autonomous driving, 2023a. URL https: htt.///orrg/bbb/20.117000
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Planning-oriented autonomous driving. In IEEE/CVF Conference Computer Vision and Pattern Recognàtion (CVPR), 2023b.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extract ing actionable knowledge for embodied agents. In International Conference on Machine Learning (ICML), 2022.
Zhiyu Huang, Chen Tang, Chen Lv, Masayoshi Tomizuka, and Wei Zhan. Learning online belief prediction for efficient pomdp planning in autonomous driving, 2024. URL httpss://axxv..or abs/aba////// 15315
Jyb-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo, Dragomir Anguelov, and Mingxing Tan. EMMA: End-to-end multimodal model for autonomous driving. Submitted to Transactions on Machine Learning Research, 2025. URL httpa///ppppeeeeeeeeeeeeerrrrrna
Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. Computer vision for autonomous vehicles: Problems, datasets and state of the art, 2021. URL httpps///aaaiii...rr///////////
35

---

Published in Transactions on Machine Learning Research (03/2026)
Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. In NeurIPS 2024 Dalasets and Benchmarks Track, 2024.
Anqing Jiang, Yu Gao, Zhigang Sun, Yiru Wang, Jiju Wang, Jinghao Chai, Qian Cao, Yuweng Heng, Hao Jiang, Yunda Dong, Zongzheng Zhang, Xianda Guo, Hao Sun, and Hao Zhao. Diffvla: Vision-language guided diffusion planning for autonomous driving, 2025a. URL http::://aaxii..rrrabbbbba 19388
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. IREE/CVP International Conference on Computer Vision (ICCV), 2023.
Bo Jiang, Shaoyu Chen, Bencheng Liao, Xingyu Zhang, Wei Yin, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Senna: Bridging large vision-language models and end-to-end autonomous driving, 2024. URL. https: /tapxi://axxi....o http://wwwww..ooaaabaa//// 2410 22313
Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning, 2025b. URL https
//axxiv.or
07608
Bo Jiang, Shaoyu Chen, Hao Gao, Bencheng Liao, Qian Zhang, Wenyu Liu, and Xinggang Wang. VADv2: End-to-end autonomous driving via probabilistic planning. In International Conference on Learning Representations (ICLR), 2026.
Hao Jiang, Chuan Hu, Yukang Shi, Yuan Hc, Ke Wang, Xi Zhang, and Zhipeng Zhang. Structured labeling enables faster vision-language models for end-to-end autonomous driving, 2025c. URL https: //arxiv. warg/abs//aaa 05442
Sicong Jiang, Zilin Huang, Kangan Qian, Ziang Luo, Tianze Zhu, Yang Zhong, Yihong Tang, Menglin Kong, Yunlong Wang, Siwen Jiao, Hao Ye, Zihao Sheng, Xin Zhao, Tuopu Wen, Zheng Fu, Sikai Chen, Kun Jiang, Diange Yang, Seongjin Choi, and Lijun Sun. A survey on lasion-languug--acctin models for autonomous driving, 2025d. URL https: //axxiv www.xii..og/abbb. 2a..b////5662444
Ye Jin, Ruoxuan Yang, Zhijie Yi, Xiaoxi Shen, Huiling Peng, Xiaoan Liu, Jingli Qin, Jiayang Li, Jintao Xie, Peizhong Gao, Guyue Zhou, and Jiangtao Gong. Surrealdriver: Designing Ilm-powered generative driver agent framework based on human drivers' driving-thinking data. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024.
Saman Kazemkhani, Aarav Pandya, Daphne Cornelisse, Brennan Shacklett, and Eugene Vinitsky. Gpudrive: Data-driven, multi-agent driving simulation at million fps. In International Conference on Learning Representations (ICLR), 2025.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Thans. Graph., 42(4):139 1, 2023.
Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata. Textual explanations for self-driving vehicles. European Conference on Computer Vision (ECCV), 2018.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael
Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source
vision-language-action model, 2024. URL hattps://arsie org/ abs/ abs//406.00m 09246
B Ravi Kiran, Ibrahim Sobh, Victor Talpacrt, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-4999 4926, 2022.
36

---

Published in Transactions on Machine Learning Research (03/2026)
James Kirkpatrick, Razvan Pascamu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agniesska Grabska-Barwinsse Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-355 3526, 2017.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shol reasoners. In Advances in Neural Informalion Processing Systems (NeurIPS), 2022.
Selim Kuzucu, Kemal Oksuz, Jonathan Sadeghi, and Puneet K. Dokania. On calibration of object detectors: Pitfalls, evaluation and baselines. In European Conference on Compuler Vision (ECCV), 2024.
Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast. encoders for object detection from point clouds. In Computer Vision and Pattern Recognition (CVPR), 2019.
Luc Le Mero, Dewei Yi, Mehrdad Dianati, and Alexandros Mouzakitis. A survey on imitation learning techniques for end-to-end autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, 23(9):14128-1414) 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequencc pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th annual meeting of the association for computational linguistics, pp. 7871-7880, 2020.
Boyi Li, Yue Wang, Jiageng Mao, Boris Ivanovic, Sushant Veer, Karen Leung, and Marco Pavone. Driving everywhere with large language model policy adaplation. In IEEE/CVI /VV Conference Computer Vision and Pattern Recognition (CVPR), 2024a.
Jing Li, Jingyuan Li, Guo Yang, Lie Yang, Haozhuang Chi, and Lichao Yang. Applications of large language models and multimodal large models in autonomous driving: A comprehensive review. Drones, 9(4), 2025a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), 2023a.
Xin Li, Yeqi Bai, Pinlong Cai, Licheng Wen, Daocheng Fu, Bo Zhang, Xuemeng Yang, Xinyu Cai, Tao Ma, Jianfei Guo, Xing Gao, Min Dou, Yikang Li, Botian Shi, Yong Liu, Liang He, and Yu Qiao. Towards knowledge-driven autonomous driving, 2023b. URL https::/ org/ absaabas///// 2tt.//231..11311
Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-target hydra-distillation, 2024b.
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M. Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? Tn IERE/CVE Conference on Compuler Vision and Pottern Recognition (CVPR), 2024c.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Beyformer: Learning bird's-cyo-viert reprosentation from lidar-camera via spatiotemporal transformers. IEEE Truns. Pallern Anal. Mach. Intell. (TPAMI), 47(3):2020 2036, 2025b. ISSN 0162-8828. doi:
TP/M/ 2024.355554 URL https: otg/10. 1119///1..11//PPPIII111
Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, and Xinggang Wang. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving, 2025.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summarics. In Text Summarization Branches Out, pp. 74-81. Association for Computational Linguistics (ACL), 2004. URL https://aclanthollg org 1013/
37

---

Published in Transactions on Machine Learning Research (03/2026)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In TEEE/CVF Conference on Computer Vision and Pattern Recognition. (CVPR), pp. 26296-26338 June 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL http::///laav--ugggttbbb. io/ blog/2024-01330-11aaa next//
Mingyu Liu, Ekim Yurtsever, Jonathan Fossaert, Xingcheng Zhou, Walter Zimmer, Yuning Cui, Bare Luka Zagar, and Alois C. Knoll. survey on autonomous driving datasets: Statistics, annotation quality, and a future outlook, 2024c. URL https://aaxxii http:///awxvvvo.o/bbbbee/ ht////1./..2/..////// 01454
Pei Liu, Haipeng Liu, Haichao Liu, Xin Liu, Jinxin Ni, and Jun Ma. Vlm-e2e: Enhancing end-to-end autonomous driving with multimodal driver attention fusion, 2025. URL https://axx...or//ass////
18042
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embodding transformation for multi-view 3d object detection. In European Conference on Compuler Vision (ECCV), 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
William Ljungbergh, Adam Tonderski, Joakim Johnander, Holger Caesar, Kalle Åström, Michael Felsberg, and Christoffer Petersson. Neuroncap: Photorealistic closed-loop safety testing for autonomous driving, 2024.
Llama Team, Meta AI. The llama 3 herd of models, 2024. URL
21783
Yuhang Lu, Yichen Yao, Jiadong Tu, Jiangnan Shao, Yuexin Ma, and Xinge Zhu. Can Ivims obtain a driver's license? benchmark towards reliable agi for autonomous driving. In Proceedings of the AAAI Conference Artificial Intelligence, 2025.
Ruipu Luo, Ziwang Zhao, Min Yang, Zheming Yang, Minghui Qiu, Tao Wang, Zhongyu Wei, Yanhao Wang, and Cen Chen. Valley: Video assistant with large language model enhanced ability, 2025. URL
https://arxiv.. wrgg/ba////// /2006. 07207
Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang. Taking a closer look at domain shift: Category- level adversaries for semantics consistent domain adaptation. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, and Chaowei Xiao. Dolphins: Multinodal language model for driving. In European Conference on Computer Vision (ECCV), 2024a.
Yunsheng Ma, Can Cui, Xu Cao, Wenqian Ye, Peiran Liu, Juanwu Lu, Amr Abdelraouf, Rohit Gupta, Kyungtae Han, Aniket Bera, James M. Rehg, and Ziran Wang. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In IEEE/CVF Conference on Computer Vision and Pollern Recognition (CVPR), 2024b.
Jiageng Mao, Yuxi Qian, Junjic Ye, Hang Zhao, and Yue Wang. Gpt-driver: Learning to drive with gpt. In NewrIPS Workshop on Foundalion Models for Decision Making, 2023.
Jiageng Mao, Junjic Ye, Yuxi Qian, Marco Pavone, and Yue Wang. A language agent for autonomous driving. In Conference on Language Modeling (CoLM), 2024.
Meta Chameleon Team. Chamelcon: Mixed-modal carly-fusion foundation models, 2025. URL https ;
7arxivv. cetrzii.org//bbae
38

---

Published in Transactions on Machine Learning Research (03/2026)
Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puncet K Dokania. Fine-tuning can cripple your foundation model; preserving features may be the solution. In TMLRR, 2024.
Ming Nie, Renyuan Peng, Chunwei Wang, Xinyue Cai, Jianhua Han, Hang Xu, and Li Zhang. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. In European Conference on Computer Vision (ECCV); 2024.
NVIDIA. Cosmos world foundation model platform for physical ai, 2025. URL https://arxiv.org/aaa 2501. 03575)
Kemal Oksuz, Tom Joy, and Puneet K. Dokania. Towards building self-aware object detectors via reliable uncertainty quantification and calibration. In IEEE/CVE Conference Computer Vision and Pollern Recognition (CVPR), 2023.
OpenAl, Chatgpt: Optimizing language models for dialogue, 2023. Last Accessed: 2025-08-01, https o/chat.openai.com/
OpenAl. Gpt-4 technical report, 2024. URL http:://aaai...orrraa////.......
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vip: Vision language planning for autonomous driving. In IEEE/CVF Conference Compuler Vision and Pallern Recognilion. (CVPR), 2024a.
Chenbin Pan, Burhaneddin Yaman, Senem Velipasalar, and Liu Ren. Clip-bevformer: Enhancing multi-view image-based bev detector with ground truth flow. Tn Proceedings of the TEFE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 15216 15225, June 2024b.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation machine translation. In Proceedings of the Annual Meeting of the Association for Computer Linguistics (ACL), 2002.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7077-7057 2021.
Alexander Prutsch, Horst Bischof, and Horst Possegger. Efficient Motion Prediction: A Lightweight & Accurate Trajectory Prediction Model With Fast Training and Inference Speed. In International Conference on Intelligent Robots and Systems (TROS), 2024.
Kangan Qian, Sicong Jiang, Yang Zhong, Ziang Luo, Zilin Huang, Tianze Zhu, Kun Jiang, Mengmeng Yang, Zheng Fu, Jinyu Miao, Yining Shi, He Zhe Lim, Li Liu, Tianbao Zhou, Huang Yu, Yifei Hu, Guang Li, Guang Chen, Hao Ye, Lijun Sun, and Diange Yang. Agentthink: A unified framework for reasoning in vision- language models for autonomous driving, 2025a. URL tool-augmented wwwoorrggaa. abs/ 2500/ 15298
https://arxiv,
Kangan Qian, Ziang Luo, Sicong Jiang, Zilin Huang, Jinyu Miao, Zhikun Ma, Tianze Zhu, Jiayin Li, Yangfan He, Zheng Fu, Yining Shi, Boyue Wang, Hezhe Lin, Ziyu Chen, Jiangbo Yu, Xinyu Jiao, Mengmeng Yang, Kun Jiang, and Diange Yang. Fasionad+++ Integrating high-level instruction and information bottleneck in fat-slow fusion systems for enhanced safety in autonomous driving with adaptive feedback, 2025b. URL
https://arxii https:://axxvv.oor/abbssss www./bsg////eeeeeeeooeeee 2500. 2511208660000002222
Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: multi-modal visual question answering benchmark for autonomous driving scenario. In AAAI Conference on Artificial Intelligence, 2024.
Qwen Team. Qwen2.5 technical report, 2025. URL httpss///ariiv.oo////////2..1111
39

---

Published in Transactions on Machine Learning Research (03/2026)
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Xuanchi Ren, Yifan Lu, Tianshi Cao, Ruiyuan Gao, Shengyu Huang, Amirmojtaba Sabour, Tianchang Shen, Tobias Pfaff, Jay Zhangjie Wu, Runjian Chen, et al. Cosmos-drive-drrenn:: Scalable synthetic driving data generation with world foundation models. arXiv preprint arXiv:2506.0900 2025.
Katrin Renz, Long Chen, Ana-Maria Marcu, Jan Hünermann, Benoit Hanotte, Alice Karnsund, Jamie Shotton, Flahe Arani, and Oleg Sinavski. Carllava: Vision language models for camera-only closed-loop driving, 2024. URL httph:://a///te http:///.wrr/bbb///..ooooo 10165
Katrin Renz, Long Chen, Elahe Arani, and Oleg Sinavski. Simlingo: Vision-only closed-loop autonomous driving with language-action alignment. In IEEE/CVF Conference Computer Vision and Pallern Recognition (CVPR), 2025.
Stuart Russell, Peter Norvig, and Artificial Intelligence. A modern approach. Artificial Intelligence. Prentice- Hall, Egnlewood ClifJs, 25(27):79 80, 1995.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Lit, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Raphael Gontijo-Lppes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohamnad Norouzi. Photorealistic text-lo-image diffusson models with deep language understanding. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Ruoxi Jia, and Ming Jin. Algorithm of thoughts: enhancing exploration of ideas in large language models. In International Conference Machine Learwing (ICML), 2024.
Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. Languagempe: Large language models as decision makers for autonomous driving, 2025. URL https://arxii: wtg//br/////aa aataaa//////.//// 2210//231.....2222
Hao Shao, Yuxuan Hu, Letian Wang, Steven L. Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models, 2024a.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024b. URL https: arg/abs/2402..oommm 2212222222.0322
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems (NewrIPS), 2022.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying, 2023.
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Jens Beißwenger, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision (ECCV), 2024.
Ziying Song, Caiyan Jia, Lin Liu, Hongyu Pan, Yongchang Zhang, Junming Wang, Xingyu Zhang, Shaoqing Xu, Lei Yang, and Yadan Luo. Don't shake the wheel: Momentum-awaae planning in end-to-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025.
40

---

Published in Transactions on Machine Learning Research (03/2026)
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguclov. Scalability in perception for autonomous driving: Waymo open dataset. In IEEE/CV /CF Conference on Computer Vision and Pallern Recognition (CVPR), 2020.
Ardi Tampuu, Tambet Matiisen, Maksym Semikin, Dmytro Fishman, and Naveed Muhammad. A survey of end-to-end driving: Architectures and training methods. TEEE Transactions Neuma Networks and Learning Systems, 33(4):1364 1384, April 2022. ISSN 2162-2388. doi: http://dx.dow doi http///d.ddi..or///.../oo 1109/TNNLS 2020 3043505
URL
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, et al. U12: Unifying language learning paradigms. arXie preprint arXiv:2205.052 05131,, 2022.
Siyu Teng, Xuemin Hu, Peng Deng, Bai Li, Yuchen Li, Yunfeng Ai, Dongsheng Yang, Lingxi Li, Zhe Xuanyuan, Fenghua Zhu, and Long Chen. Motion planning for autonomous driving: The state of the art and future perspectives. IEEE Transactions on Intelligent Vehicles, 366):3692-3711 2023.
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Yang Wang, Zhiyong Zhao, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision- lailuag models. Tn Conference on Robot Learning (CoRL), 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Mane Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaune Lample. Llama: Open and efficient foundation language models, 2023a. URL
https://arxia: http///axivvv.. http:///aaxii..rg//bb/////////// htt///abb2222..112 2etp://222111197711
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Tnan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Rman Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Tliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodrigues, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https:///rrrs org/ abs/2307 09288
Ramakrishua Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: bonsensus-based image description evaluation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Vicuna Team. Vicuna 1.5: An open-source chatbot impressing gpt-4 with 90% chatgpt quality. https : //vicuna. org. 2023. Last Accessod: 2025-08-01.
Hai Wang, Jiayi Li, and Haoran Dong. A review of rision-based multi-task perception research methods for autonomous vehicles. Sensors, 25(8), 2025a.
Jumming Wang, Xingyu Zhang, Zebin Xing, Songen Gu, Xiaoyang Guo, Yang Hu, Ziying Song, Qian Zhang, Xiacxiao Long, and Wei Yin. He-drive: Human- like end-to-end driving with vision language models, 2024a. URL https://arxxi: www.aiiiwww.eiaeeeeeeeeeeoooeree
orw.abs.////oo.....oooo........r abs/ 050051
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In IEEE/CVF International Conference on Computer Vision (ICCV), 2023a.
41

---

Published in Transactions on Machine Learning Research (03/2026)
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M. Alvarez. Omnidrive: A holistic vision-language dataset for autonomous driving with counterfactual reasoning. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2025b.
Shiyi Wang, Yuxuan Zhu, Zhiheng Li, Yutong Wang, Li Li, and Zhengbing He. Chatgpt as your vehicle co-pilot: An imitial attempt. IEEE Transactions on Intelligent Vehicles, 8(12):4706-4721, 2023b.
Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drivedreamer: Towards weal-worldd world models for autonomous driving. In European Conference Computer Vision (ECCV), 2024b.
Yan Wang, Wenjie Luo, Junjie Bai, Yulong Cao, Tong Che, Ke Chen, Yuxiao Chen, Jenna Diamond, Yifan Ding, Wenhao Ding, et al. Alpamayo-rl: Bridging reasoning and action prediction for generalizable aulonomous driving in the long lail. Xiviv preprint arXin:251 0tt:2511.0008, 2025c.
Yixuan Wang, Ruochen Jiao, Sinong Simon Zhan, Chengtian Lang, Chao Huang, Zhaoran Wang, Zhuoran Yang, and Qi Zhu. Empowering autonomous driving with large language models: A safety perspective. In International Conference on Learning Represenlations (ICLR) Workshop on LLM Agents, 2024c.
Yujin Wang, Quanfeng Liu, Zhengxin Jiang, Tianyi Wang, Junfeng Jiao, Hongqing Chu, Bingzhao Gao, and Hong Chen. Rad: Retrieval aabieeee--ggeeeted decision beeisinn-making of meta-actions with vision- Laigua- models in autonomous driving, 2025d.
Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVE /CFF Conference on Computer Vision and Pattern Recognation, pp. 14749 14749-14759, 2024d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Licheng Wen, Daocheng Fu, Xin Li, Xinyu Cai, Tao Ma, Pinlong Cai, Min Dou, Botian Shi, Liang He, and Yu Qiao. Dilu: A knowledge-driven approach to autonomous driving with large language models. In International Conference on Learning Representations (ICLR), 2024a.
Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang. Panacca: Panoramic and controllable video generation for autonomous driving. In IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), 2024b.
Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Bin Xiao, Haiping WU, Weijian Xu, Xiyang Dai, Hondong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing unified representation for variely of vision tasks. In /BE//CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Yichen Xie, Runsheng Xu, Tong He, Jyh-Jing Hwang, Katie Luo, Jingwei Ji, Hubert Lin, Letian Chen, Yiren Lu, Zhaoqi Leng, Dragomir Anguclov, and Mingxing Tan. S4-driver: Scalable self-supervised driving multimodal large language model with spatio-temporal visual representation. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025.
Ding Xinpeng, Han Jinahua, Xu Hang, Hang Xu, Zhang Wei, and Li Xiaomeng. Hilm-d: Enhancing milms with multi-scale high-resolution details for autonomous driving. International Journal of Computer Vision (IJCV), 2025.
Huaiyuan Xu, Junliang Chen, Shiyu Meng, Yi Wang, and Lap-Pui Chan. A survey on occupancy perception for autonomous driving: The information fusion perspective. Information Fusion, 114:102671, 2025a.
42

---

Published in Transactions on Machine Learning Research (03/2026)
Yi Xu, Yuxin Hu, Zaiwei Zhang, Gregory P. Meyer, Siva Karthik Mustikovela, Siddhartha Srinivasa, Eric M. Wolff, and Xin Huang. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. In Conference on Robot Learning (CoRL), 2025b.
Zhenhua Xu, Yujia Zhang, Euze Xie, Zhen Zhao, Yong Guo, Kwan- Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters, 2024.
Zhenhua Xu, Yan Bai, Yujia Zhang, Zhuoling Li, Fei Xia, Kwan-Yee K. Wong, Jianqiang Wang, and Hengshuang Zhao Drivegpt4-v2: Harnessing large language model capabilities for enhanced closed-loop autonomous driving. In Computer Vision and Pottern Recognition Conference (CVPR), 2025c.
Tianyi Yan, Dongming Wu, Wencheng Han, Junpeng Jiang, Xia Zhou, Kun Zhan, Cheng-zhong Xu, and Jianbing Shen. Drivingsphere: Building a high-fidelity 4d world for closed-loop simulation. In Proceedings of the Computer Vision and Pottern Recognition Conference (CVPR), 2025.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024a. URL https: http///xxvvvvvrr/ abs/ 2abb/2407 2407 10671
Honghui Yang, Tong He, Jiaheng Liu, Hua Chen, Boxi Wu, Binbin Lin, Xiaofei He, and Wanli Ouyang. Gd-mae: Generative decoder for mae pre-training on lidar point clouds. In CVPR, 2023.
Kairui Yang, Zihao Guo, Gengjie Lin, Haotian Dong, Zhao Huang, Yipeng Wu, Die Zuo, Jibin Peng, Ziyuan Zhong, Xin WANG, Qing Guo, Xiaosong Jia, Junchi Yan, and Di Lin. Trajectory-LLM: A language-based data generator for trajectory prediction in autonomous driving. In International Conference on Learning Representations (ICLR), 2025a. URL https://openrevies net/forummid--Uppxlrrsse
Senqiao Yang, Jiaming Liu, Renrui Zhang, Mingjie Pan, Ziyu Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Hongsheng Li, Yandong Guo, and Shanghang Zhang. Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. In AAAI Conference on Artificial Intelligence, 2025b.
Yi Yang, Qingwen Zhang, Ci Li, Daniel Simöes Marta, Nazre Batool, and John Folkesson. Human-centric autonomous systems with Iims for user command reasoning. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, 2024b.
Zhenjie Yang, Xiaosong Jia, Hongyang Li, and Junchi Yan. LIm4drive: A survey of large language models for autonomous driving, 2024c. URI. //atps://arxiv...
20t1..//331.0000 01043
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Feyza Yavuz, Baris Can Cam, Adnan Harun Dogan, Kemal Oksuz, Emre Akbas, and Sinan Kalkan. Bucketed ranking-ban based losses for efficient training of object detectors. In European Conference on Computer Vision (ECCV), 2024.
Junwei You, Haotian Shi, Zhuoyu Jiang, Zilin Huang, Rui Gan, Keshu Wu, Xi Cheng, Xiaopeng Li, and Bin Ran. V2x-vim: End-to-end v2x cooperative autonomous driving through large vasion-language models, 2024. URL. http:////aaav..//////////////// 09251
43

---

Published in Transactions on Machine Learning Research (03/2026)
Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 21361 21370, 2022.
Jianhao Yuan, Shuyang Sun, Daniel Omciza, Bo Zhao, Paul Newman, Lars Kunze, and Matthew Gadd. Rag-driver: Generalisable driving explanations with retrieval-auee -atriveal-ungmeneee in-context learning in multi-modal large language model. In Robolics: Science and Systems, 2024.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE Access, 8:58443-560 58469,8844444 2020.
Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-object tracking with transformer. In Europeon Conference on Computer Vision (ECCV), 2022.
Jiang- Tian Zhai, Ze Feng, Jihao Du, Yonggiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation of end-to-end autonomous driving in nuscenes. arviv preprint ar 0rxi:22305.100 10430, 2023a.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, In IEEE/CVF International Conference Computer Vision (ICCV), 2023b.
Bozhou Zhang, Nan Song, Xin Jin, and Li Zhang. Bridging past and future: End-to-end autonomous driving with historical prediction and planning. In Proceedings of the Computer Vision and Pallern Recognition Conference (CVPR), 2025.
Jimuyang Zhang, Zanming Huang, Arijit Ray, and Eshed Ohn-Bar. Feedback-guided autonomous driving. In IEFE/CVF Conference on Computer Vision and Pattern Recogrittion (CVPR), 2024.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations (ICLR), 2020.
Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang, Yida Wang, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, and Xingang Wang. Drivedreamer4d: World models are effective data machines for 4d driving scene representation. In Proceedings of the Computer Vision and Pollern Recognition Conference (CVPR), pp. 12015 12026, June 2025a.
Guosheng Zhao, Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Xueyang Zhang, Yida Wang, Guan Huang, Xinze Chen, Boyuan Wang, Youyi Zhang, Wenjun Mei, and Xingang Wang. Drivedreamer4d: World models are effective data machines for 4d driving scene representation. Tn Proceedings of the IEFE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 12015 12026, June 2025b.
Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang. Drivedrcamer-2: Lim-enhanced world models for diverse driving video generation. In AAAI Conference Artificial Intelligence, 2025c.
Jingyuan Zhao, Wenyi Zhao, Bo Deng, Zhenghong Wang, Feng Zhang, Wenxiang Zheng, Wanke Cao, Jinrui Nan, Yubo Lian, and Andrew F. Burke. Autonomous driving system: A comprehensive survey. Expert Systems with Applications, 242:122836, 2024. ISSN 0957-4174. doi: https://doi.org// http:///do..ogg10011////// 10.111//...sssa..2222 122836.
Jilai Zheng, Pin Tang, Zhongdao Wang, Guoqing Wang, Xiangxuan Ren, Bailan Feng, and Chao Ma. Veon: Vocabulary-enhanced occupancy prediction. In European Conference Computer Vision (ECCV), 2024a.
Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-ent autonomous driving. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Giil Varol (eds.), European Conference Computer Vision (ECCV), 2024b.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. IEFF Transactions on Pattern Analysis and Machine Intellggnce, 45(4):4396 -4416, 2023a.
44

---

Published in Transactions on Machine Learning Research (03/2026)
Xiaoyu Zhou, Jingqi Wang, Yongtao Wang, Yufei Wei, Nan Dong, and Ming-Hsuan Yang. Autooce: Automatic open-ended semantic occupancy annotation via vision-language guided gaussian splatting. In TEEE/CVE International Conference on Computer Vision (ICCV), 2025a.
Xingcheng Zhou, Mingyu Liu, Fkim Yurtsever, Bare Luka Zagar, Walter Zimmer, Hu Cao, and Alois C. Knoll. Vision language models in autonomous driving: A survey and outlook, 2024a. URL https
2t3p:/abb/2231..44411 14414.
Xingcheng Zhou, Xuyuan Han, Feng Yang, Yunpu Ma, and Alois C. Knoll. Opendrivevla: Towards end-to-end autonomous driving with large vision language action model, 2026.
Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understanding of driving scenarios. In European Conference on Computer Vision (ECCV), 2024b.
Zewei Zhou, Tianhui Cai, Seth Z. Zhaa, Yun Zhang, Zhiyu Huang, Bolei Zhou, and Jiaqi Ma. Autovla: A jision-languuage-cction model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine-tuning. In Aduances in Newral Information Processing Systems (NeurIPS), 2025b.
Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023b.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Xuehui Wang, Yue Cao, Yangzhou Liu, Xingguang Wei, Hongjie Zhang, Haomin Wang, Weiye Xu, Hao Li, Jiahao Wang, Nianchen Deng, Songze Li, Yinan He, Tan Jiang, Jiapeng Luo, Yi Wang, Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Han Lv, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Gc, Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-lime recipes for open-source multimodal models, 2025. URL https: http///axvvv../// http://aarrri.rrrbbba///aaaaa/// absp//aa//220..000
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. International Conference on Learning Representations (ICLR), 2021.
45

---

Published in Transactions on Machine Learning Research (03/2026)
APPENDIX
A Further Details on the Openness of the Methods
We classify and discuss the openness of the asscts of each method in Sec. Here, we include the conventions, which also clarify few outlier cases in the openness classification in Tab. 6
If evaluation of an approach is performed on multiple datasets, the dataset with the least restrictive license considered for the classifying "Driving Evaluation Dataset" and "Language/Action" Evalua- tion Datasct" asscts. The name of the dataset in question is noted an explanation. This selection chosen reflect when at least part of the results can be reproduced by the community.
On the other hand, if training is performed on multiple datasets jointly, the dataset with the most restrictive license is considered for classifying "Trajectory Planning Training Dataset" and "Language/ Action Capabilities Training Dataset' assets. The name of the dataset in question is noted as an explanation. This convention is chosen as all datasets within mixture are needed to Drive-VLM different dataset mixtures, with results reported independently. In such case, the score of the least restrictive mixture of datasets is used for the corresponding cell entry.
reproduce the results. However, few methods, e.g. Drive VLM (Tian et al., 2024], train the models
To provide further clarity on the basis of our classification, we include the license name for the code asscts, and the dataset name for the data asscts (except model weights).
For Omni-Q (Wagg et al. 2025b), Orion (Fu et al. 2025a and AsyncDriver (Chen et al. 2024c). the license of the released model weights is in conflict with pretrained FMs weights. We classily these entries based on the released model license and warn the reader that additional restrictions may apply (noted as *).
For LMDrive (Shao et al. 2024a) and Senna-E2E (Jiang et al., 2024), there are two contradicting licenses for the model weights, the same huggingface space. We classify the entry based on the more restrictive license which also aligns with the openness score of the pretrained FMs weights (noted **).
no license is specified, default copyright. law applies, which is highly restrictive.
If the model combines vision encoder and LLM-instead of using an off-the-shelf VLM-we consider the most restrictive license for the pretrained FMs weights classification. Note, any components trained from scratch are ignored (e.g. LLM in CarLLaVA Renz et al. (2024)
Closed-loop evaluation benchmarks like CARLA Town05 (Prakash et al. 2021), CARLA Longest (Chitta et al., 2022), Bench2Drive (Jia et al. 2024 and LAV (Chen & Krähenbühl] 2022 are considered within the "Driving Evaluation Dataset asset as they provide route splits or auxiliary scenario configuration files.
46


Reviews:
[{"review_id": "Y9Jm0DnquL", "reviewer": "Reviewer_mqh2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This survey provides a structured overview of how foundation models are being applied to autonomous driving trajectory planning. \n\n1. It proposes a clear taxonomy that separates approaches where foundation models are tailored for trajectory planning versus those where foundation models guide planning, with finer subcategories (e.g., language/trajectory chain-of-thought, interaction, distillation); \n2. It compiles and reviews a broad set of recent methods with comparative tables/figures;\n3. It offers practical guidance on adapting foundation models to planning, including data curation, trajectory representations, and fine-tuning strategies; \n4. It introduces an openness/reproducibility assessment framework (with explicit scoring levels) to characterize the availability of code/data/weights and highlight reproducibility gaps, alongside a discussion of open challenges for deployable FM-based planners.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I find the survey’s key claims convincing:\n\n1. The authors lay out an explicit taxonomy (e.g., foundation models tailored for trajectory planning vs guiding planning) and then support it with a broad, organized coverage of recent work summarized through structured discussion and tables/figures. \n\n2. Their reproducibility/openness conclusions are particularly solid because they use a clearly defined scoring rubric (with concrete levels and guidance for edge cases such as licensing), so the assessment is transparent and checkable rather than anecdotal. \n\n3. They also strike the right tone in how they interpret results: the survey explicitly flags confounders that make cross-paper performance comparisons tricky, which reduces the risk of over-claiming and makes the takeaways feel more trustworthy.", "audience_yn": "Yes", "audience_explanation": "Many readers would likely find this survey interesting because it makes a fast-moving and fragmented area much easier to navigate.\n\nIt offers a clear taxonomy of how foundation models are used for autonomous driving trajectory planning, backs it with a curated and structured coverage of recent methods (with tables/figures), and includes practical guidance on adaptation choices like data curation, trajectory representations, and fine-tuning.   \n\nIt’s also especially useful for researchers and practitioners who care about reproducibility and deployment, since it proposes an explicit openness scoring framework and discusses real-world constraints such as inference cost/latency (including CoT overhead) versus real-time needs.", "requested_changes": "While the survey is already strong, I would encourage the authors to (i) explicitly describe the literature collection/inclusion methodology, and (ii) add lightweight comparability and taxonomy-assignment annotations in the main tables (e.g., evaluation setting, dataset/metric flags, and brief rationale for categorization), which would further improve transparency and prevent readers from over-interpreting cross-paper comparisons.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "f2E0zFiLXs", "reviewer": "Reviewer_K1tz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary\n\nThe authors provide a survey of recent approaches that use multi-modal foundation models (FMs) for trajectory planning in autonomous driving. The main contribution is a hierarchical taxonomy that separates (i) FMs tailored for trajectory planning (fine-tuned / adapted to directly output trajectories) and (ii) FMs guiding trajectory planning (where FM knowledge is transferred to a planner, e.g., via distillation), and connects these groups to minimal mathematical formulations (Sec. 3; Fig. 4–5; Eq. (1)–(2)). The paper claims to systematically analyze 37 methods under this taxonomy, and provides practical discussion on fine-tuning and data curation (Sec. 4.1). In addition, the authors include an 'openness' assessment (code/data/weights) with a structured scoring scheme (Sec. 6; Fig. 8; Tab. 6; Appendix A), and summarize open challenges such as deployment latency constraints, sim-to-real gaps, vision-language collapse after fine-tuning, and evaluation/benchmarking limitations (Sec. 7). Figure 1 reads as a motivating qualitative example of VLM-style scene understanding.\n\n# Strengths and Weaknesses\n\n## Strengths\n\n* S1: The taxonomy is comprehensive and well structured. The two-level split (tailored vs guiding, and then further sub-branches) is intuitive and captures meaningful differences in how FMs are used in planning.\n* S2: I like that the taxonomy is not only descriptive, but also linked to basic formal formulations (Sec. 3.1; Eq. (1)–(2); Fig. 5). The notation (observations, trajectories, text prompts/outputs) helps clarify the conditioning structure and makes the categorization more rigorous.\n* S3: The technical overview is detailed and organized in a way that is useful for practitioners (Sec. 4–5; Tables 1–6). In particular, the discussion covers (although briefly) relevant dimensions such as data curation, model adaptation choices (e.g., vision adapters), trajectory representation alternatives, and training strategies (Sec. 4.1; Fig. 6).\n* S4: The openness assessment is a clear contribution (Sec. 6; Tab. 6; Fig. 8; Appendix A). The asset-level scoring is explicit and useful for reproducibility, and it highlights that 'openness' is currently a bottleneck in this area.\n* S5: The open-issues section is concrete and grounded in examples (Sec. 7). For instance, the paper discusses inference-time costs of reasoning-style generation (e.g., the reported CoT latency drop from 3.6 fps to 0.8 fps in one example) and provides an intuitive example.\n\n## Weaknesses\n\n* W1 (Diagram inconsistency): Figure 5 has a visual inconsistency with the mathematical formulation (Fig. 5; Sec. 3.1; Eq. (1)). For example, in Fig. 5(b) (Text as CoT), the formula indicates that the generated text output should condition the trajectory generation, but the diagram does not clearly show this dependency (missing/unclear input arrow / two-stage flow). Since Fig. 5 is central to the taxonomy, this affects clarity and technical correctness of the manuscript.\n* W2 (taxonomy presentation too abstract): Sections 3.1 and 3.2 introduce the main taxonomy branches mostly at an abstract level (Sec. 3.1–3.2). Adding 1–2 representative method examples (with references) directly in these sections would improve readability and make the categorization easier to follow for readers not already familiar with the literature.\n* W3 (Data curation without references): The 'Data curation' discussion is currently too abstract for a section positioned as practical guidance (Sec. 4.1). The paper would benefit from concrete dataset examples (with references).\n* W4 (search or selection protocol missing): The claim of 'systematically analyzing 37 methods' is not fully traceable without an explicit selection protocol (Sec. 1–2). A short paragraph describing search sources/venues, time window, and inclusion/exclusion criteria would make the survey stronger and easier to update.\n* W5 (missing discussion in taxonomy groups): While the paper categorizes methods well, it lacks a structured 'pros/cons/implications' synthesis per taxonomy group (Sec. 4–5 vs. Sec. 7). Right now, the open-issues section is cross-cutting, but the reader still has to infer 'when should I use which leaf node and why'. A per-leaf-node summary (strengths, limitations, compute/latency implications, data needs, typical failure modes) would significantly increase the value of the taxonomy.\n* W6 (missing depth in asset availability): The openness analysis is good, but it could benefit from a clearer high-level synthesis of available datasets/assets and their characteristics (Sec. 6; Tab. 6).\n* W7 (missing references): Some datasets which are mentioned in the manuscript are not properly cited (e.g. DAIR-V2X with \"DAIR-V2X: A Large-Scale Dataset for Vehicle-Infrastructure Cooperative 3D Object Detection\")", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes. The claims are supported by the provided manuscript, although it still needs some corrections before this manuscript can be published.", "audience_yn": "Yes", "audience_explanation": "Yes. TMLR readers interested in foundation models, planning, and autonomy would benefit from (i) a structured taxonomy that organizes a rapidly growing literature, (ii) the practical consolidation of design dimensions (training strategies, trajectory representations, interaction modalities), and (iii) the asset-level openness audit that directly impacts reproducibility and reuse (Sec. 3–6). The open-issues section is also relevant to both researchers and practitioners because it frames constraints that matter for deployment and evaluation (Sec. 7).", "requested_changes": "# Requested changes\n\n## Critical\n\n* C1: Fix Fig. 5 consistency: revise the diagrams (especially Fig. 5(b)) to match the conditioning / AR information flow implied by each respective formulation.\n* C2: Add a brief survey methodology / selection protocol supporting the 'systematic analysis of 37 methods' framing (Sec. 1–2).\n* C3: Add a per-taxonomy-group 'pros/cons/implications' synthesis (ideally a per-leaf-node summary table: strengths/limitations/risks/when-to-use, plus intuition on compute/latency and data needs).\n* C4: Verify that all the mentioned datasets are properly cited (e.g. DAIR-V2X)\n\n## Minor\n\n* M1: Add representative method examples/citations directly in Sec. 3.1 and Sec. 3.2 when defining the taxonomy branches.\n* M2: Strengthen Sec. 4.1 data curation with concrete dataset examples.\n* M3: Improve Tab. 6 readability and summarize key assets in the main text.", "broader_impact_concerns": "No major broader impact concerns are raised by the survey itself, but given that autonomous driving is safety critical, a brief statement on dataset governance (privacy/licensing) and safety validation context would improve completeness and set appropriate expectations for deployment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zFdW5eXxuT", "reviewer": "Reviewer_t9if", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary of Contributions:\n\nThis paper provides a survey of foundation-model-based trajectory planning for autonomous driving. Its main contribution is a hierarchical taxonomy that organises existing methods into two broad classes: (i) foundation models tailored for trajectory planning, which are fine-tuned to directly predict trajectories, and (ii) foundation models guiding trajectory planning, which use FMs for knowledge distillation or auxiliary guidance within existing AD pipelines. Using this taxonomy, the paper systematically reviews 37 recent approaches, discussing their architectural choices, use of chain-of-thought reasoning, language/action interaction capabilities, and training–inference trade-offs. It also offers practical guidance on fine-tuning, data curation, and trajectory representations, evaluates code/data openness, and highlights key open challenges for future research.\n\nKey Strengths:\n\n* Well-structured taxonomy that brings order to a fragmented literature.\n* Useful discussion of practical design trade-offs and reproducibility.\n\nKey Weaknesses:\n\n* Lacks quantitative benchmarking or direct empirical comparisons.\n* Some discussions of challenges could be more concrete or actionable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims of the paper are about taxonomy, categorisation, and synthesis of existing work, rather than new empirical results. These claims are supported by clear definitions, consistent use of the proposed taxonomy, and accurate citations to a broad set of representative methods. The paper presents its arguments clearly, avoids overclaiming, and appropriately frames limitations and open challenges, which is consistent with TMLR’s evaluation criteria for survey submissions.", "audience_yn": "Yes", "audience_explanation": "The paper would be of interest to at least a subset of TMLR’s audience, particularly researchers working on foundation models, autonomous driving, multimodal learning, and decision-making systems. By providing a structured taxonomy, critical synthesis of recent methods, and a focused discussion on design trade-offs and open challenges, the paper offers value both as a reference for newcomers to the area and as a conceptual framework for researchers actively developing FM-based trajectory planning methods.", "requested_changes": "1. The paper should more explicitly state that it does not aim to compare or rank methods empirically, but rather to organise and analyse them conceptually. While this is largely implicit, making it explicit early (e.g., in the Introduction or Scope subsection) would help avoid misinterpretation of the paper’s goals.\n\n2. In a few cases, the boundaries between subcategories (e.g., CoT-based planning vs. language-interaction capabilities) could be clarified further to ensure that categorisation decisions are fully unambiguous and consistently motivated across all surveyed methods.\n\n3. A compact table summarising key trade-offs across categories (e.g., inference cost, reliance on language supervision, deployment feasibility) would improve accessibility and practical usefulness.\n\n4. While open challenges are discussed, the paper could more explicitly highlight the lack of standardised benchmarks and metrics for FM-based trajectory planning, and how this complicates fair comparison across approaches.\n\n5. The paper could benefit from a short forward-looking subsection identifying emerging directions (e.g., world models, instruction-following safety, efficiency-aware FM integration) to further guide future research.\n\n6. Some sections are dense due to the breadth of coverage; light tightening of language or additional signposting could further improve readability, especially for readers less familiar with autonomous driving pipelines.", "broader_impact_concerns": "I do not identify major unaddressed broader impact concerns specific to this work. The submission is a survey paper that analyzes and organizes existing research on foundation models for trajectory planning, and it does not introduce new algorithms, datasets, or deployment claims that would raise immediate ethical risks.\nThat said, the topic area—autonomous driving—does have inherent societal and safety implications, including concerns around robustness, accountability, bias, and safe deployment. These issues are already widely recognised in the field and are partially reflected in the paper’s discussion of open challenges (e.g., robustness, evaluation, and sim-to-real transfer). If desired, the paper could briefly acknowledge these broader implications in a dedicated paragraph, but the absence of a detailed Broader Impact Statement does not appear to be a significant omission given the survey nature of the work.\nOverall, no additional Broader Impact requirements are strictly necessary for this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)