EMMA: End-to-End Multimodal Model for Autonomous Driving
Authors: Jyh-Jing Hwang
Runsheng Xu
Hubert Lin
Wei-Chih Hung
Jingwei Ji
Kristy Choi
Di Huang
Tong He
Paul Covington
Benjamin Sapp
Yin Zhou
James Guo
Dragomir Anguelov
Mingxing Tan
Venue: Accepted by TMLR ()

Abstract: We introduce EMMA, an End-to-end Multimodal Model for Autonomous driving. Built on a multimodal large language model foundation, EMMA directly maps raw camera sensor data into various driving-specific outputs, including planner trajectories, perception objects, and road graph elements. EMMA maximizes the utility of world knowledge from the pre-trained large language models, by representing all non-sensor inputs (e.g. navigation instructions and ego vehicle status) and outputs (e.g. trajectories and 3D locations) as natural language text. This approach allows EMMA to jointly process various driving tasks in a unified language space, and generate the outputs for each task using task-specific prompts. Empirically, we demonstrate EMMA’s effectiveness by achieving state-of-the-art performance in motion planning on nuScenes as well as competitive results on an in-house large-scale benchmark. EMMA also yields competitive results for camera-primary 3D object detection on the Waymo Open Dataset (WOD). We show that co-training EMMA with planner trajectories, object detection, and road graph tasks yields improvements across all three domains, highlighting EMMA’s potential as a generalist model for autonomous driving applications. We hope that our results will inspire research to further evolve the state of the art in autonomous driving model architectures.

Published in Transactions on Machine Learning Research (07/2025)
End-to-End Multimodal Model for Autonomous Driving
EMMA:
Jyh-Jing Hwang" 1, Runsheng Xu', Hubert Lin', Wei-Chih Hung', Jingwei Ji, Kristy Choi Di Huang, Tong He, Paul Covington, Benjamin Sapp, Yin Zhou, James Guo Dragomir Anguelov, Mingxing Tan
Waymo LLC
Reviewed OpenReview: https: https://openrevieew openreview. net/ ptt/formm? id= km3t55mmUU
Abstract
We introduce EMMA, an End-to-end Multimodal Model for Autonomous driving. Built upon multi-modal large language model foundation like Gemini, EMMA directly maps raw camera sensor data into various driving-specific outputs, including planner trajectories, perception objects, and road graph elements. EMMA maximizes the utility of world knowledge from the pre-trained large language models, by representing all non-sensor inputs (e.g. navigation instructions and ego vehicle status) and outputs (e.g. trajectories and 3D locations) as natural language text. This approach allows EMMA to jointly process various driving tasks in unified language space, and generate the outputs for each task using lask-specific prompls. Empirically, demonstrate EMMA's effectiveness by achieving state-of-the-art performance in motion planning on nuScenes as well as competitive results on the Waymo Open Motion Dataset (WOMD). EMMA also yields competitive results for camera-primary 3D object detection on the Waymo Open Dataset (WOD). We show that co-training FMMA with planner trajectories, object detection, and road graph tasks yields improvements across all three domains, highlighting EMMA's potential as generalist model for autonomous driving applications. We hope that results will inspire research to further evolve the state of the art in autonomous driving model architectures.
1 Introduction
Autonomous driving technology has made significant progress in recent years. To make autonomous vehicles a ubiquitous form of transportation, they must navigate increasingly complex real-world scenarios that require understanding rich scene context well as sophisticated reasoning and decision-making,
Historically, autonomous driving systems employed a modular approach, consisting of specialized components
for perception (Yurtsever et al., 2020 Li et al.) 2022b;
Lang et al., 2019; Sun et al. 2022 Hwang et al.
2022,, mapping (Li et al.] 2022a
Tancik et al.] 2022),
prediction (Nayakanti et al. 2023 Shi et al. 2024)),
and planning TTeng et al. 2023 Lioutas et al. 2022). While this design lends itseel to easier debugging
and optimization of individual modules, it poses scalability challenges due the limited module commuication. In particular, the expert-designed interfaces between modules, such the perception and behavior modules, may struggle to adapt. to novel environments because they are often pre-defined based on
targeted scenarios (Bansal et al. 2019 Jiang et al. 2023 Nayakanti et al. 2023 Seff et al. (023). End-to-end
autonomous driving systems (Hu et al. 2023 Zhai et al. 2023 Li et al.
2024 have recently emerged as a
potential solution, directly learning to generate driving actions from sensor data. This approach eliminates
the need for symbolic interfaces between modules and allows for joint optimization of driving objectives from
"Equal contributions: Equal contributions.
1 Conttctt emails: Mingxing Tan <tammmggggiigkkwwymooooo... Jyh-Jing Hwang <jyhh@hurymo.com>>

---

Published in Transactions on Machine Learning Research (07/2025)
raw sensor imputs. However, these systems are often specialized for specific driving tasks and trained on limited datasets, hindering their ability to generalize to rare or novel scenarios.
Multimodal Large Language Models (MLLMs) (Gemini Team Google 2023 Achiam et al. 2023) offer a promising new paradigm for AI in autonomous driving that may help to address such challenges. This is because ML,LMs, as generalist foundation models, excel in two key areas: (1) they are trained on vast, internet-scale datasets that provide rich 'world knowledge" beyond what is contained in common driving logs, and (2) they demonstrate superior reasoning capabilities through techniques such as chain-of-thought reasoning (Wei et al. 2022 Zhang et al.) 2023b) that are not available in specialized driving systems. While recent efforts (Chen et al. [2024b] Tian et al., 2024) have explored integrating and augmenting the capabilities of existing driving systems with MLLMs, we propose to develop an autonomous driving system in which the MLLM is a first class citizen.
We introduce the End-to-End Multimodal Model for Autonomous Driving (EMMA), built on top of a multimodal large language model, such as Gemini Gemini Team Google 2023) or PaLI (Chen et al., 2024d) without additional specialized components. Figure shows the overview of the EMMA framework. EMMA accepts camera images and plain text for other non-vision inputs such as high-level driving commands and historical context. By recasting driving tasks as visual question answering (VQA) problems, EMMA leverages Gemini's pre-trained capabilities and extensive world knowledge. After FMMA is fine-tuned with driving logs from all tasks using task-specific prompts (see Figure 2 for more examples), it generates various driving outputs such as future trajectories for motion planning, perception objects, road graph elements, and scene semantics. Our experiments showcase EMMA's strong performance on several planning and perception benchmarks despite this simple design. Additionally, we find that FMMA can produce interpretable, human-readable outputs for many perception tasks such as road graph estimation, and is able to function as generalist model that is both scalable and robust for autonomous driving. Notably, as used here and throughout the paper, the EMMA generalist model refers to machine learning model that has been trained and fine-tuned on a large volume of driving data to perform a wide range of driving tasks in the autonomous driving domain.
We summarize our key findings below:
1. EMMA exhibits strong performance in end-to-end motion planning, achieving state-of-the-art performance on public benchmarks muScencs (Cacsar et al. 2020) and competitive results on the Waymo Open Motion Dataset (WOMD) (Chen et al. 2024a). We also show that we can further improve motion planning quality with more internal training data and chain-of-thought reasoning.
2. EMMA demonstrates competitive results for various perception lasks including 3D object detection, road graph estimation, and scene understanding. On the camera-primary Waymo Open Dataset (WOD) (Hung et al., 2024), EMMA achioves better precision and recall for 3D object detection than state-of-the-art methods.
3. We demonstrate that EMMA can function as a generalist model in the autonomous driving domain, which jointly generates the outputs for multiple driving related tasks. In particular, EMMA matches or even surpasses the performance of individually trained models when is co-trained with motion planning, object detection, and road graph tasks.
4. Finally, we show EMMA's capacity to reason and make decisions in complex, long-tail driving scenarios.
In the remainder of this paper, Scction 2 describes the detailed method of EMMA for end-to-end motion planning and generalist tasks in autonomous driving. In Section 3 we present experimental results of EMMA on public and internal datasets. Finally, we discuss related works in Sections 4
Despite these promising results, EMMA is not without its limitations. We discuss the limitations in-depth in the Appendix Section A.5 In particular, it faces challenges for real-world deployment due to: (1) limitations in 3D spatial reasoning due to its inability to fuse camera inputs with LiDAR or radar, (2) the need for realistic and computationally expensive sensor simulation to power its closed-loop evaluation, and (3) the increased computational requirements relative to conventional models. We plan to better understand and address such challenges in future work.
2

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Router
Driving rationale
obgocte:
High-level command tum turn right, straight....
Behowior description
Chain of thoughht reasoning
The cyclst curner soop ng the intersccionn Their antio cated erooyy indiccttes they cross
you potert cousing colibion
evesive action. ng the divertion,
The wetein cornentiyy shoed
trglechoin ruc pertic it will continue sira
Context:
historicel ego status anything describable text
Mete driving decision:
Planning: ego future waypoints
EMMA: End-to to-End Multimodal Model
built on top of Gemini
vidan
Other capabilities
Spatial reasoning
Road comprehension
Scene understanding
Figure 1: EMMA overview diagram. It takes 3 inputs (left): 1) a high-lovel command from the router, 2) historical status of the ego vehicle, and 3) surround-view camera videos. The model then predicts ego future trajectories (right) for motion planning that will be transformed into vehicle driving control signals. Further, we can ask the model to explain its rationale (top) before predicting trajectories, which enhances both the performance and explainability of the model through chain-of-thought reasoning. Notably, we incorporate visual grounding into the rationale so that the model also predicts the accurate 3D/BEY location for critical objects. In addition end-to-end planning, we highlight three additional perception capabilities of model (bottom).
2
Method
While we will show EMMA can be compatible with various MLLMs such as Gemini (Gemini Team Google 2023) and PaLI (Chen et al. 2024d) in our experiments, this section will focus on our main EMMA based on Gemini. We leverage the auto-regressive Gemini models that are trained to process interleaved textual and visual inputs to produce text outpuls:
O = D= V)
(1)
where G is the Gemini model, O represents natural language outputs, T represents natural language prompts, and V denotes images or videos. The language output O = 02,,0, 0n) is generated via next-token prediction, i.c., the output probability can be reprosented as P(O V) = -I[= P(oiloei, T, V) for n output tokens. Our goal is to adapt G for aulonomous driving applications, thereby harnessing the world knowledge obtained during its pre-training phase.
As shown in Figure sensor data are represented as stitched images videos V: all router commands, driving context, and task-specific prompts are represented in language prompts T; and all output tasks are presented as language outputs O. A challenge is that many of the inputs and outputs need to capture 3D world coordinates, such as waypoint BEV (Bird's Eye View) locations (x,z) y) for motion planning and the location and size of 3D boxes. We consider two representations: The first direct text conversion to floating-point numbers, expressed as Teoordinates ((.(.,,)) A text({(z; (zi, 9;))), where the specified decimal precision depends on the distance unit and decimal points. RT-2 (Brohan et al. 2023) exemplifies this approach in robotic control. The second approach uses special tokens to represent each location action, formulated T Tevordinates = {(z,, x tokenize (f (z;, 9;))), with resolution determined by the learned or manually defined discretization scheme. MotionLM (Self et al.] (202)) leverages this method for motion forecasting. We note that the two approaches have their respective strengths and weaknesses. We opt for the text representation such that all tasks can share the same unified language representation space and they can maximally reuse the knowledge from the pre-trained weights, even though the text representation may produce more tokens than specialized tokenization.
map autonomous driving tasks into our Gemini-based EMMA formulation. All
3

---

Published in Transactions on Machine Learning Research (07 (225))
2.1 End-to-Eme Motion Planning
EMMA employs unified, end-to-end trained model to generate future trajectories for aulonomous vehicles directly from sensor data. These generated trajectories are then transformed into vehicle-specific control actions such as acceleration and turning for autonomous vehicles. EMMA's end-to-end approach aims to emulate human driving behavior, focusing on two critical aspects: (1) first, the use navigation systems (e.g. Google Maps) for route planning and intent determination, and (2) second, the utilization of past actions ensure smooth, consistent driving over time.
Our model incorporates three key imputs to align with these human driving behaviors:
1. Surround-view camera videos (V): Provides comprchensive environment information.
2. High-level intent command iTintent:: Derived from the router, includes directives such as "go straight", "burn left." "turn right", elc.
3. Set of historical ego status (Tego): Represented as a set of waypoint coordinates in Bird's Eye View (BEV) space, Tego = {(r,, 3r) for Ta timestamps. All waypoint coordinates are represented as plain text without specialized tokens. This can also be extended to include higher-order ego status such as velocity and acceleration.
The model generates future trajectories for motion planning, represented as of future trajectory waypoints for the ego vehicle the same BEV space: Otrajectory {(2,, ) for future Ty timestamps, where all output waypoints are also represnted as plain text. Putting everything together, the complete formulation is expressed as:
O trujectory = iuTintent Tego: V).
(2)
We then fine-tune Gemini with this formulation for end-to-end planner trajectory generation, as illustrated in Figure [. We highlight 3 characteristics of this formulation:
1. Self-supervised: the only required supervision is the future locations of the ego vehicle. No dedicated human labels are needed.
Camera-only: the only sensor input required is surround-vier cameras.
3. HD map free: no HD map is needed beyond the high-level routing information from navigation system such as Google Maps.
While we are not the first to adopt this general formulation (Li al. 2024 conducted a thorough investigation, partieularly examining the impact of including the historical ego status our contribution lies in adapting this formulation specifically for MLLMs for autonomous driving. Our self-supervised approach exists alongside other notable methods that explore reconstruction via spatio-temporal scene decomposition (Yang ct al. 2023). world modeling (Zhang et al.) 2023a). or joint motion prediction (Wagner et al. 2024). Beyond this self sell-supervised foundation, the following sections explore enhancements for EMMA by incorporating reasoning and developing generalist setups with human or auto-labels.
2.2 Planning with Chain-of- Thought Reasoning
Chain-of-thouglt prompting Wei et al. 2022 is powerful tool in MLLMs that enhances reasoning capabilities and improves explainability. In EMMA, we incorporate chain-of-thought reasoning into end-to-end planner trajectory generation by asking the model to articulate its decision rationale Orationale while predicting the final future trajectory waypoints Otrajectory.
We structure the driving rationale hierarchically, progressing from 4 types of coarse-to-fi to-finn--geeeooo fine-grained information:
R1 - Scene description broadly describes the driving scenarios, including weather, day time, traffic sations and road conditions. We provide concrete example for prompting Gemini. Example prompt: Assume you are an autonomous vehiole, and the images come from your frond cameras. Can you describe the current scenario terms of weather, time of the day, road environment, lane options, and your ego lane position?
4

---

Published in Transactions on Machine Learning Research (07 (225))
Task Prompts and shown)
Camera input (Video))
Answers
Decoded Visualization
my future driving actions?
waypoinns x_ 2
Detected objects in & 1. theta vehicte).
Octect everything 3D?
The lancsl can drive towards are: Ik. lnnd x y. valkdl.
Estimate the drivable roadgraph?
EMMA: End-to-Enw Multimodal Model
ahoad temporanly blocked?
No, the road ahead is clear
busit top Gemiin
Imore task prompta)
[more answers]
Figure 2: Illustration of EMMA Generalist. Starting with a task prompt (left), EMMA generates corresponding textual prediction (middle right), which then be decoded into target output format, visualized and overlaid with the input image (right). EMMA Generalist is highly versatile, capable of performing a wide range of driving-related tasks, such as end-to-end motion planning, object detection, road graph estimation, and scene understanding Q&A. In the answer lext, italicized words represent placeholders that will be dynamically substituted with actual values during task execution.
Example answer: The weather is clear and sunny, and daytime. The road is four-lone undivided street with crosswalk in the middle. There are cars porked on both sides of the street.
R2 Critical objects are the on-road agents that can potentially influence the driving behavior of the ego vehicle, and we require the model to identify their precise 3D/BEV coordinates. For instance: pedestrion at [9.01, 9.22], vehicle at [11.58, 0.35].
R3 Behavior description of critical objects describes the current status and intent for the identified critical objects. We provide a concrete example for prompting Gemini. Example prompt: Assess the potential risks posed by the [focused_
with red bounding box. Summarize any immediiate concerns that need addressing maintain safety, paying close attention to how the objects may affect your route. Example answer: The pedestrian is curvently standing on the sidewalk, looking toward the road, and maybe preparing to cross the street. The vehicle is currently ahoud of me, moving in the direction, and its fulure trajectory suggests will continue straighd.
R4 Meta driving decision includes 12 categories of high-level driving decisions, summarizing the driving plan given the previous observations. An example would be should keep my current low speed.
We highlight that the driving rationale captions are generated using an automated tool without any additional human labels, ensuring scalability of the data generation pipeline. Specifically, we leverage off-the-shelf perception and prediction expert models to identify critical agents, and then use Gemini models with carefully designed visual and text prompts to generate scene and agent behavior descriptions. Meta driving decisions are computed using heuristic algorithm that analyzes the ego vchicle's ground-truth trajectory.
During both training and inference, the model predicts all four components of the driving rationale before predicting the future waypoints, i.e.,
(0 (Orationales Otrajectory) G intent; T Togos V).
Where Orstionale denotes an ordered text output of (R1, R2, R3, R4) for driving rationale. Empirically, we observe that the prediction order of Orationale and O Orajectory does not result a significant difference quality after model convergence. This suggests that we can predict O Ousjectory first and apply early stopping during inference for time-critical applications.
5

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
2.3 EMMA Generalist
While end-to-end motion planning is the ultimate core task, a comprehensive autonomous driving system requires additional capabilities. Specifically, it must perceive the 3D world and recognize surrounding objects, the road graph and the traffic conditions. To achieve this goal, we formulate EMMA a generalist model capable of handling multiple driving tasks through training mixtures.
Our vision-language framework represents all non-sensor inputs and outputs as plain text, providing the flexibility necessary to incorporate many other driving tasks. We employ instruction-tuning a well-cstablished approach in LLMs, to jointly train all tasks together with task-specific prompts included in the inputs T of Eq. [ We organize these tasks into three primary categorics: spatial reasoning. road graph estimation, and scene understanding.
Spatial reasoning is the ability to understand, reason, and draw conclusions about objects and their relationships in space. This enables an autonomous driving system to interpret and interact with its surrounding environment for safe navigation.
Our primary focus for spatial reasoning is 3D object detection. We follow Pix2Seq (Chen et al. 2022a and formulate the output 3D bounding boxcs as O boxes set(text(z, y, y, w, h, 0, cls)} where (xt, y. 2) are the center location in the vehiele frame, I, w, are the length, width, and height of the box, 0 is the heading angle, and cls is the class label in text. We convert a 7D box to text by writing floating-point numbers with two decimal places, separated by spaces between each dimension.
We then represent the detection tasks using fixed prompt Tdetect 3D, such as "detect every object in 3D", as follows:
Oboxes G 3D, V).
While Oboxes is an unordered set of boxes, the predictions from an auto-regressive language model are always ordered. We find that sorting the 3D bounding boxes by depth improves detection quality, unlike the findings in Pix2Seq (Chen et al. 2022a).
Road graph estimation focuses on identifying critical road elements for safe driving, including semantic elements (e.g., lane markings, signs) and physical properties (e.g., lane curvature). The collection of these road elements forms road graph. For example, lane segments are represented by (a) nodes, where the lanes encounter an intersection, merge, or split and (b) edges wet these nodes following the direction of traffic. The full road-graph composed of many such polyline segments.
While edges within each polyline are directional, each polyline does not necessarily have unique order relative to the other elements. This bears similarity to object detection (e.g., (Carion et al. 2020 Chen
et al. 2022ah), where each box is defined by ordered attributes (top-left corner, bottom-right corner), but a rolative ordering between boxes does not necessarily exist. There are several existing works that model
Qiao et al, polyline graphs with Transformers (Yuan et al. 2024 Liao et al. 2024a b 2023 Ding et al.
2023
2023 Liu et al. 2023 Li et al., 2022a), sharing similarities with language models.
Our general modeling formulation in EMMA is as follows:
O Poodgraph = G (oTTsstiate__roadeeee V).
(5)
where Oroudgraph is text-cncoded road graph represented as waypoints, Testimate roadgraph is prompt asking the model to predict the roadgrah, and V denotes the surrounding images.
We focus specifically on predicting drivable lanes, i.., the lanes that the ego vehicle can drive towards in the scene. These are neighboring lanes in the same traffic direction and lanes branching out from the current ego lane. To construct Ornadgraph, we (a) convert lanes into sets of ordered waypoints and (b) transform these sets of waypoints into text. It is beneficial to use sample-ordered waypoints to represent both traffic direction and curvature. Just like detoction, we also find that ordering lanes by approximate distance improves the prediction quality. An example of our polyline text encoding is: "xx1,y1 and... and yn,yn); where "x,y" are floating point waypoints with precision to 2 decimal places, ";" separates polyline instances.
6

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Scene understanding tasks test the model's understanding of the whole scene context, which can be relevant. for driving. For example, roads can be temporarily obstrucied due to construction, emergency situations, or other events. Detecting these blockages in timely manner and safely navigating around them is essential for ensuring the smooth and safe operation of autonomous vehicles; however, multiple cues in the scene are required to determine if there is a blockage or not. We focus on how our model performs on this temporary blockage detection task, using the following formulation:
O blocckae = (T temporary_ blockage
T Traad Froad_user users V);
(6)
where O bloockuge is the model output signaling potential obstruction, V denotes the surrounding images, Troad_use denotes the all the objects on the road ahead, Teemporar__biookep is the text prompt "is the road ahead temporarily blocked?".
2.4 Generalist Training
Our unified vision-language formulation enables the simultaneous training of multiple tasks with a single model, allowing for task-specific predictions at inference time through simple variations of the task prompt Trask- This training procedure is both straightforward and flexible.
For each task, we construct a dataset Dtask containing [Daskl training examples. During each training iteration, we randomly sample batch from the available datasots, with the probability of selecting an example from specific dataset proportional to the dataset size: i.e., |Daskl// Σ, [Dr). |D.). set the total number of training iterations to x Σ, [Del, ensuring that the training ratio among tasks is governed by the relative dataset sizes. The optimal training ratio is influenced by several factors, including task complexity, inter-task correlations, and the degree of transferability across tasks.
To train for epochs,
Our experimental results demonstrate that the generalist model, trained across multiple tasks, consistently outperforms each specialist model that is trained on a single task. This highlights the advantage of the generalist approach: enhanced knowledge transfer, improved generalization, and increased efficiency.
3 Experiments
Our experiments are primarily based on Gemini 1.0 Nano-1 (Gemini Team Google 2023), with additional results provided for a variant of EMMA based on PaLl (Chen et al. 2024d]. We first sunmarize the main datasets used for various experiments in Section 3.1 And then we present the results of end-to-end planner trajectory generation on two public datasets in Section 3.2 Next, we conduct experiments on our internal datasets, studying the impact of chain-of thouugh and data scaling in Scction 3.3 Scction 3.4 focuses on 3D object detection experiments. Our co-training results for the generalist. model are summarized in Section 3.5 Finally, showcase visual results that highlight EMMA's capabilities in challenging, long-tail scenarios in Scction 3.6
3.1 Summary of Datasets
Before diving into the experimental details on how we validate EMMA, we summarize the main datasets in this section. Overall, we leverage three public datasets, nuScenes (Caesar et al., 2020), Waymo Open Motion Dataset (WOMD) (Chen al.] 2024a) and Waymo Open Dataset (WOD) (Sun al., 2020). We also constructed three large-scale internal datasets for end-to-end motion planning, 3D detection and road graph estimation. We summarize the dataset sizes in Table [
The nuScenes dataset (Caesar et al. 2020) offers a comprehensive autonomous vehicle sensor suite for evaluation. It. consists of 1,000 scenes, each spanning 20 seconds, and includes information from 6 cameras that collectively provide 360-degree coverage in the field of view.
The WOMD dataset comprises 103k real-world urban and suburban driving scenarios, each lasting 20 seconds. These scenarios further segmented into 1.1M examples, each representing 9-second window:
7

---

Published in Transactions on Machine Learning Research (07/2025)
Dataset. Name
Total Hours of Driving
Number of Training Examples
nuSeenes CCassrr et al.
2020
6
18,686
WOMD
Chen al.
2024a
572
487,061
Internal Motion Planning Dataset
203, 117 (355x)
24,374,046 (50x)
WOD
Sun ct al.
2020
6
158,081
Internal Detection Dataset
6250
11, 11,765,140
Internal Roadgraph Dalaset
64,135 135
8,304,671
Table 1: Summary of main training datasct scales. This table details the scales of the three public datasets (nuScenes, WOMD, WOD) and three large-scale internal datasets leveraged for studying data scaling and generalist properties.
second is used as imput context, and the remaining 8 seconds serve as the prediction target. The dataset includes detailed map features such as traffic signal states and lane characteristics, along with agent states such as position, velocily, acceleration, and bounding boxes.
We build large-scale internal motion planning dataset, boasting over 24 million real-world driving scenarios, cach 30 seconds long. This makes it roughly 355 times larger than WOMD or any other publicly available driving dataset. To elficiently leverage this massive scale, we sample just one frame per scenario, yielding 24 million diverse training examples. This approach maximizes dataset diversity while maintaining computational efficiency.
We also construct two separate, large-scale internal datasets for detection and road graph tasks, comprising 12 million and 8 million examples, respectively. For the detection dataset, we prioritized scenarios with diverse objects, sampling one example every seconds from these scenarios. The road graph dataset, on the other hand, focuses on diverse scenarios and geo-locations, so we sample one example every 30 seconds, aligning with our motion-planning dataset.
Lastly, also validate the camera-based 3D object detection task on the public WOD benchmark (un et al. 2020 Hung et al. 2024). This benchmark offers 1150 20-second scenes, each providing meticulously synchronized and calibrated high-quality LiDAR, camera, and 3D box data from a variety of urban and suburban environments.
3.2 Lillto-End Motion Planning on Public Datasets
We conduct the end-to-end planner trajectory generation experiments on two public datasets, WOMD CCee et al. 2024a) and the nuScenes dataset [Caesar et al. 2020). EMMA is trained with the simplest end-to-end planner trajectory generation formulation as in Equation 2 unless specified otherwise. That is, given camera images, ego vehicle history, and driving intent, the model is asked to predict the future ego waypoints for a certain time horizon.
3.2.1 Driving on the Waymo Open Motion Dataset (WOMD)
For fair comparisons, we align our scttings with WOMD. Additionally, we reproduce and adapt internally enhanced versions of the state-of-the-art motion prediction models, MotionLM (Self al.] 20233) and Wayformer (Nayakanti et al. 2023). serving with high-level command intent as an additional input.
planner baselines. These baseline models are augmented
As shown in Table g our model outperforms the MotiouLM baseline when we train on the same dataset, with Gemini pre-trained weights. When pre-trained with our mega-scale internal dataset (denoted as EMMA+), our model outperforms both MotionLM and Wayformer. The full EFM+ (w/ CoT) surpasses the previous state-of the-art models significantly by 13.5% the 5s prediction horizon. We also apply the EMMA method to open-sourced MLLM, PaLI-X (Chen al. 2024d), denoted as EMMAt (PaL.). We show that EMMA
8

---

Published in Transactions on Machine Learning Research (07/2025)
Method
L2 (m)
1s
L2 (m)
3s
L2 (m) 5s
MotionLM
Self et al. 2023
0.045
0.266
0.696
Wayformer' Nayakanti al.
2023
0.046
0.252
0.628
EMMAT (based on PaLI)
0.034
0.274
0.797
EMMA +f (based on PaLI)
0.031
0.239
0.680
EMMA
0.032
0.248
0.681
EMMA (w/ CoT)
0.030
0.241
0.664
EMMA+
0.030
0.225
0.610
E (w/C CoT)
0.027
0.203
0.543
Table 2: End-to-end motion planning experiments internal planning benchmark. CoT denotes equipping with chain-of-thought reasoning (Eq. B FMMA+ achieves the best quality across different prediction time horizons. EMMA and EMMA+ denotes using PaLI-X (Chen et al. 2024d) as our basc model, while the default EMMA and EMMA+ use Gemini as the base model. "Enhanced, reproduced baselines.
Driving Quality vs Number of Sampled Trajectories on WOMD
average distance error (5DE@5s)
0.724
0.72
0.70
0.88
0.66
0.64
0.621
0.62
0.60
12
number of sampled trajectories
Figure Ablation study on the number of sampled trajectories. As more trajectories are sampled, the quality measured by ADE@5s also improves, but the benefits diminish after 12+ samples.
can generalize well across different MLLMs with a large amount of training data, yielding better quality (at Is and 3s) than previous state-of-the-arl baselines.
We note the differences in inputs between MotionLM and EMMA: MotionLM takes inputs of agent location history, agent interactions, the road graph, and traffic light states. These agent boxcs are producod by specialized off- board perception models that look at both past and future observations and are trained with large amount of carefully curated human labels, the road graph is manually generated using full run segments, and all inputs heavily use LiDAR data with superior depth estimation. In stark contrast, EMMA only takes camera images and ego vehicle history as input, without the need of any labels or additional models (besides leveraging the Gemini pre-trained weights).
During inference, sampling final trajectory from multiple candidates plays critical role in the final performance. Both MotionLM and Wayformer generate 192 candidate trajectories, which are subsequently aggregated into 6 clusters using k-means clustering, resulting in 6 representative trajectories to be selected the final output according to their probabilitics. For fairness, Top-K decoding stralegy, lo = 24. We then compute the pairwise L2 distance between all trajectories and select the one with the lowest average L2 distance as the final predicted trajectory, which can be viewed as the "median" trajectory among all the predictions. We investigate the impact of the number of sampled trajectories on ADE, as illustrated in Figure 3 The results highlight that sampling from multiple trajectories leads to notable improvement. in ADE, however, with dininishing return.
also sample multiple trajectorics using a

---

Published in Transactions on Machine Learning Research (07/2025)
Method
self-supervised!
L2 (m) 1s
L.2 (m)
2s
L2 (m)
3s
Avg L2 (m)
UniAD (Hu et al..
2023
0.42
0.64
0.91
0.66
DriveVLM Tian et al.
2024
AX
0.18
0.34
0.68
0.40
AX
VAD (Jiang et al. 2023
0.17
0.34
0.60
0.37
OmniDrive Wang et al.
2024a
0.14
0.29
0.55
0.33
Ego-MLP* (Zaai et al.
2023
0.15
0.32
0.59
0.35
BEV-Planner (Li et al.
2024
0.16
0.32
0.57
0.35
EMMA (random init)
0.15
0.33
0.63
0.37
EMMA
0.14
0.29
0.54
0.32
EMMA+
0.13
0.27
0.48
0.29
Table 3: End-to-end motion planning experiments on muScenes (Caesar et al. 2020
EMMA (random init)
denotes models are randomly initialized; EMMA denotes models are initialized from Gemini; EMMA+ denotes models that are pre-trained on our mega-scale internal data. EMMA achieves stato-of-the-art performance the nuScenes planming benchmark, out performmnn the supervised (with perception and human labels) prior art by 6.4% and sele-supervised (no extra labels) prior art by 17.1%. "Ego-MLP results are taken from reproduced version in BEV-Planne
3.2.2 Driving on the nuScenes Dataset
follow the standard protocol of nuScenes for planning evaluation: predict the next 3 seconds of future driving actions based on 2 seconds of historical data. We measure the planning quality with L2 errors at 1-, 2- and 3-second time horizons, aligning with established baseline methods, in particular BEV-Planner Li et al. 2024).
In our experiments,
We train and evaluate EMMA with the simplest end-to-end planner trajectory generation formulation as in Equation 2 (self-uupervised, without Shain-of-though reasoning nor generalist training). As shown in Table sell-supervised FMMA achieves state-of-the-art results in planning on nuScenes, out performiig all previous supervised (with intermediate perception labels and/or /or human labels) and self-supervised (no extra labels) methods. Under the same self-supervised setup, EMMA outperforms BEV-Planner (Li et al. 2024 by 17.1% in average L2 metric; even compared to OmniDrive (Wang el al. 2024a that heavily uses intermediate perception human labels, our self-supervised EMMA improves the average L2 metric by 12.1%.
Unlike in WOMD, we note that sampling multiple trajectories did yicld significant improvements. We hypothesize that this is due to nuScenes' shorter prediction time horizon (3s) in simpler driving scenarios. Thus, report only top-1 predictions for results.
3.3 End-to-End Motion Planning with Chain-of- Thought Reasoning on Internal Dataset
In this section, present our studies of end-to-end planning with chain-of-thought on our internal dataset. This dataset contains 24 millions of scenarios, orders of magnitude larger than any publicly available autonomous driving dataset. The model takes in 2 seconds history to predict the driving actions for 5 seconds into the future.
Table 4 presents the results of our experiments on chain-of-thought reasoning applied end-to-end planning, By adopting the thain-of-thougle formulation (Equation 3 achieve notable 6.7% improvement over the standard end-to-end planning approach detailed in Equation
? We also conduct ablation study to analyze the contributions of different rationale components. Our findings reveal that both driving meta-decision and critical object identification significantly enhance performance, contributing improvements of 3.0% and 1.5%, respectively. When these components are combined, the gains are even more substantial. Conversely, while scene description has neutral impact on driving performance, it enhances the model's explainability. These results demonstrate that chain-of- of-thought reasoning can meaningfully improve driving performance, particularly when its components carefully selected and integrated.
10

---

Published in Transactions on Machine Learning Research (07/2025)
Scene description
Critical object
Meta decision
Behavior description
Relative improvements
over baseline e2e planning
X
X
+ 0.0%
X
X
X
+ 1.5%
X
X
X
3.0%
X
/
X
+ 5.7%
X
+ 6.7%
Table 4: Ablation study on chain-of-thought reasoning components. It improves end-to-end planning quality by up to 6.7% by combining all elements. In particular, driving meta-decision and critical objects contribute the improvements of 3.0% and 1.5%, respectively. The details of each component described in Section 2.2
End-to-End Motion Planning Quality Scales with Training Data
scenarios
52
12% scerarios
52.05
30% scenarios
eval perplexity
50
100%
50.13
47.56
46.79
46.07
45.26
44.69
10
1020
10t1
training flops
Figure EMMA data scaling experiments on our mega-scale internal dataset. Each curve represents the eval perplexity for end-to-end motion planning training proceeds with more steps. The x-axis is training compute, measured by floating-point operations (FLOPs) in log scale. The same EMMA model is trained on four sizes of datasets that are sampled with different percentages from 3% to 100% (denoted by different colors). In general, FMMA tends to achieve better quality until overfitting when given more training compute, but it overfits quickly on smaller datasets. We observe the driving quality has not saturated when using the full large-scale dataset.
We also perform series of data scaling experiments for end-to-end planning, the results of which are illustrated in Figure 4 As we train the model on larger training set, we observe lower eval perplexities before overfitting. Our findings indicate that the driving quality of EMMA has not yet plateaned, even with the current mega-scale dataset.
3.4 3D Object Detection
We validate our 3D object detection performance on the 3D camera-primary detection benchmark from the Waymo Open Dataset (Sun al., 2020 using the Longitudinal Error Tolerant (LET) matching (Hung et al. 2024). We evaluate two versions: EMMA and EMMA+, similar carlier sections, where EMMA+ is pre-trained on the 3D detection lask using our internal dataset. The quantitative results reported on the official test set and summarized in Figure
Our findings show that after pre-training, EMMA+ achieves competitive performance on the benchmark. Since our model produces set of delected boxes without individual confidence scores, we compare the precision/recall instead of LET-3D-AP, which calculated based on the precision/reen /recall curve. We also compare the commonly used F1-score, where EMMA's F1-score computed using the single procision/rocal and other models' Fi-scores are calculated by picking the maximal Fi-score on the curve (often called F1-max).
11

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Vehicle
Pedestrian
10
10
0.8
0.8
precision
precision
*
Ft: 0.85
0.62
Ft: 0.61 EMMA
F1: 0.25 EMMA
F1-max: 0,82 BEVFormer
F1-max: 0.70 BEVFormer
F1-max: 0.82 MV-FCOS3D++
F1-max: 0.63 MV-FCOS3D+.
0.0
0.0
0.8
Figure 5: Camera-primary 3D object detection experiments on WOD (unn et al. 2020 using the standard
LET matching (Hung al. 2024). EMMA+ achieves competitive performance on the delection benchmark both precision/recal and Fl-score metrics. Compared state-of-the-art methods, it achieves 16.3% relative improvements in vehicle precision the samee recall 5.5% recall improvement the same precision.
Figure 5 shows the performance comparison. In generally, EMMA- demonstrates substantial improvements over stale-of-the-arl methods such as BEVFormer (Li et al. 2022b), achieving a 16.3% relative increase in vehicle precision at the same recall, and a 5.5% recall improvement at the same precision. EMMA+ also achieve better F1-score than prior arts. Performance on the pedestrian class is also comparable to that of MV-FCOS3D+++ (Wang al. 2021). Additionally, we provide performance breakdown across different ranges, highlighting that our model performs especially well in the near range. Our results underscore that with sufficient data and large enough model, multimodal approach can surpass specialized expert models in 3D detection quality.
3.4.1 Road Graph Estimation
Road graph estimation is complex task that predicts group of unordered polylines, each of which is represented sequence of waypoints. We measure the quality of road graph prediction with two metrics: (1) lane-level precision and recall, where we define a true positive match between a predicted lane polyline and a groundtruth lane polyline if and only if their Chamfer distance is within meter; and (2) pixel-level precision and recall, where polylines are rasterized into a BEV grid with meter resolution we then treat the BEV grid as a image and compute precision and recall based on per-pixel matching.
As discussed in Section 2.3 this task involves several design choices. One is about the representation of road graph polylines, where choice is to define the start and end points of each lane, with intermediate points added as needed to accurately capture the road's curvature. Another critical design choice is the construction of target label sequences used for model training. Drawing inspiration from Pix2Seq (Chen et al.) 2022ab in the context of object detection, one effective design choice is to pad the targets and apply random shufiling. This technique helps the model handle unordered outputs and prevents premature termination during training.
Figure presents our ablation studies on various design choices. Starting from our best designs, we systematically ablate each of the following configurations and assess the resulting quality degradation. We then summarize the key insights from our analysis.
Polyline representation: dynamic sampling is better than fixed sampling. A simple polyline representation is to sample a fixed number of sparse control points per lane, e.g. two end points plus fixed number of intermediate points capture curvature. However, we find better approach is dynamically adjust the number of points per polyline according the curvature and length of the lane. By keeping consistent waypoint density rather than a consistent number of waypoints, achieve representation
12

---

Published in Transactions on Machine Learning Research (07/2025)
that more accurately captures the lane structure intricacies, yielding around 40% to 90% difference in the metrics as shown in Figure 6 By adapting the waypoint density to the road geometry, particularly in areas with sharper curves or varying lane lengths, we achieve a flexible representation that more accurately captures the lane structure intricacies
Polyline representation: ego-origin aligned sample intervals are better than naively aligned sample intervals. The road graph typically stored and accessed in global coordinate frame, meaning lane origins and extensions are independent of the ego vehicle position. To improve accuracy, it is essential to adjust lane point samples to start from the ego vehicle coordinate frame origin. Specifically, sampling polyline points relative to the AV position (ego-origin) avoids arbitrary offsets that can arisc from directly transforming points sampled in the global coordinate frame into the ego coordinate frame. This prevents a 25% to 60% drop in prediction quality.
Target sequence construction: shuffled ordering is better than arbitrary ordering. We organize polyline largets into bins based on their endpoint distance from the ego vehicle, providing rough global ordering. For instance, we categorize lanes into nearby lanes and those further away that serve as connecting lanes. During training, we dynamically shuffle the polylines within each distance bin to enhance the model robustness and coverage. This dynamic shuffling within each bin improves the model's ability to generalize across different lane configurations, leading to more accurate predictions.
Target sequence construction: padding is better than non-padding. Similar to Pix2Seq (Chen et al. 2022a), we find that padding targets to prevent early termination is highly effective. In addition padding the total number of polyline targets, also pad the number of points within each polyline. We use "invalid"" tokens to represent padded points within polylines. Each polyline is also explicitly tagged with final "valid" or "invalid" token to denole whether it contains any nonpadded points. This approach ensures consistent imput sizes, which helps maintain the integrity of the model during training and reduces the risk of premature truncation, leading to more reliable and accurate predictions.
Target sequence construction: adding punctuation and other semantically redundant token improves quality. In the target sequence construction, we notice that it is beneficial to use language- like structures and punctuation to group targets (e.g., " "xx,y and x,y); Additionally, explicitly including semantically redundant tokens such as marking padded targets "invalid" " instead of relying on implicit omissions of "valid" markers improves performance. This approach, incorporating punctuation and redundancy, results in a boost of up to 10% in lane-level metrics. We attribute this improvement to the language-related pre- training of Gemini. By leveraging similar structured expressions, Gemini can be more easily adapted to other tasks.
instead of "xy xy;...")
3.4.2 Scene Understanding
Figure 7 summarizes our studies on the scene understanding task for temporary blockage delection. Our study is based on our internal datasets specifically curated for these scenarios. For this study, we establish our basclines by showing a picture to human and asking them to judge whether a lane is temporarily blocked. They can answer 'yes', 'no', or 'unsure'. Our baseline will treat all 'unsure' answers as incorrect, baseline=filtering will filter out all examples with 'unsure' answers. In contrast, our model is fine-tuned to predict 'yes' 'no' for all examples. As shown in the figure, our naive model that is directly fine-tuned for only this task achieves better performance than the basclinc comparison, but underperforms on the Aaseline+fillering comparison. To boost the model performance, our first allempt is to co-train this task with road graph estimation, but the naive mixture doesn't improve performance. Our second attempt is to first pre-train the model on road graph estimation, and then fine-tune on these two tasks. Results show when the pre-training is long enough, the quality is boosted, showcasing the model's ability to integrate multiple tasks for enhanced performance.
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Ablation Study on Roadgraph Estimation
decrease in metric (%6
lene-level
lane-level recall
pixel-iavel recall
100
dynamic fixed sampling
ego-arigin natvely aligried
shuftied arbitrary order
padding + padding
semanttc punctuation
Figure Ablation study on road graph estimation. To evaluate the influence of different components in our road graph estimation model, we ablate each configuration and measure the corresponding impact quality. Dynamic sampling (leftmost.) of road graph polylines based on lane curvature and length proves to be the most significant factor, leading to a substantial 70% to 90% change in lane-level precision and recall. In contrast, aligning the model with a languago-like representation, i.c., semantic punctuation (rightmost), has a more modest effect, contributing to only <10% change in precision and recall of any metric.
Temporary Blockage Detection
82.0
80.0
75.0
65.0
baselne
80.0
57.9
direct fine-tuning
naive mixture
mix short pretraining
mix long pretraining
Figure 7: Scene understanding experiments. direct fine-tuning denotes solely using the temporal blockage data during fine-tuning; naive mixture denotes co-training this scene task with road graph estimation; mix short pretraining denotes pre-training on road graph esitmation first, and then fine-tune on the mixture of both tasks; mix long pretraining denotes a longer pre-training before fine-tuning. The naive fine-tuning is already close strong human baseline, but long nog-ppretaining with training mixture can further boost the quality.
3.5 Generalist
We explore the development of the EMMA Generalist by co-training on multiple tasks and analyzing their synergies, as summarized in Table 5 For this study, we focus on three core tasks: end-to-end planning, 3D object detection, and road graph estimation.
Co-training all three tasks yields significant improvements, with the generalist model outperforming the single-task models by up to 5.5%. We attribute these results to the complementary nature of the tasks. For example, road graph estimation becomes easier when the model can accuralely identify the locations of vehicles. Similarly, driving quality is closely tied to understanding agent interactions, a skill enhanced by 3D object detection. Paring the mixture down to only two tasks still yiclds improvements, with certain combinations leading greater gains than others. For instance, detection performance improves most when co-trained with driving, and road graph estimation similarly benefits most when paired with driving. This
14

---

Published in Transactions on Machine Learning Research (07/2025)
Relative improvement over single task
e2e planning
3D detection
road graph
e2e planning
detection
road graph
+1.6% (=1.0%)
+2.4% ( (0.8%)
+1.1% (+2.8%)
+5.6% (=1.1%)
- 1..% (+2.9%)
+3.5% (=0.9%
/
1..1% (12.8%)
+5.5% (=1.1%)
+2.4% (40.8%)
Table 5: Generalist co-training experiments. (+++ indicates standard deviation. By co-training on multiple tasks, EMMA gains a broader understanding of driving scenes, enabling to handle various tasks at inference time, while enhancing individual task performance. Notably, cortain task pairings yield greater benefits than others, suggesting these tasks are complementary. Co-training all three lasks together yields the best quality.
suggests the driving task plays a prominent and influential role, serving as key contributor to overall performance improvements.
These findings suggest that pursuing generalist model is a promising direction for future research, with the potential for deeper insights into task synergies and performance optimization.
3.6 Visualizations
We group visual examples by scenario type: Examples (a)-(d) showcase how EMMA safely interacts with rare, unscen objects animals the road. Examples (c)-(f) feature EMMA navigating through construction areas. Examples (g)-()) showcase EMMA following traffic rules at interscctions with traffic lights or traffic controllers. Examples (k)-(1) highlight EMMA respecting vulnerable road users like motorcyclists.
Given these examples, we demonstrate the following capabilities of EMMA:
Generalizability: Adapts well to diverse real-world driving scenarios across different enviromments and attends objects beyond fine-tuning categories, such as squirrels.
Predictive driving:
Proactively adjusts to the hehavior other road users for safe and smooth driving.
Obstacle avoidance: Consistently adjusts trajectories to avoid obstacles, debris and blocked lanes.
Adaptive behavior: Safely handles complex situations like yielding, construction zones, and following traffic control signals.
Accurate 3D detection: Effectively identifies and tracks road agents, including vehicles, cyclists, motorcy- clists, and pedestrians.
Reliable road graph estimation: Accurately captures road layouts and integrates them into safe trajectory planning.
To conclude, these scenarios highlight EMMA's capability to operate safely and efficiently in variety of challenging and diverse driving scenarios and environments.
4
Related Works
End-to-end autonomous driving research enjoys rich history and has evolved significantly since ALVINN (Pomerleau) 1988 employed shallow neural networks to predict control signals. The field benefited from further deep learning advancements: c.g. DAVE-2 (Bojarski et al. 2016) and ChauffeurNet (Bansal et al. 2019) leveraged deeper neural architectures and incorporated sophisticated perception and motion planning modules respectively. Recent research has expanded to include multimodal inputs (Codevilla)
ct al. 2018 Prakassh al. 2021), multi-task learning (Chitta et al. 2022 Wu et al. 2022). reinforcement
learning (Chekroun et al. 2023 Chen et al. 2021 Kendall et al. 2019 Liang al.
2018
Toromanoff et al.
2020), and distillation (Chen et. al.] 2020
Zhang & Ohn-Bar
2021
Zhang et. al.
2021 Unified planning
15

---

Published in Transactions on Machine Learning Research (07/202) 2025)
(a) A garbage bag appears the freeway, our predicted trajectory suggests to nudge slightly the right avoid it.
(b) ladder appears on the freeway, and our predicted trajectory suggests switch to the to bypass it appropriately.
(c) we encounter a small squirrel on the road and our predicted trajectory instinctively slows down to avoid the animal. Note EMMA wasn't explicitly trained to detect squirrels.
(d) white dog appears in our lane, and our model predicts to slow down and yield. Our model also accurately detects surrounding vehicles, including those in adjacent lanes and the parking lot.
Figure EMMA prediction visualization. Each row contains a scenario with our model's predictions: end-to-end planning (left), 3D object detection (middle), and road graph estimation (right).
16

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
(e) As construction zone blocks the left lanes, our predicted trajectory suggests passing through on the right, while the road graph estimation correctly identifies the blocked area.
(f) Our lane is blocked by construction cones, so our predicted trajectory suggests to move into the left lane, even though it's in the opposite direction. EMMA captured the blockage and performed detour.
(g) traffic controller signals proceed through the intersection, and predicted trajectory aligns with the instruction.
(h) Our predicted trajectory suggests stop as we approach intersection with a yellow light, demonstrating cautious and safe behavior.
Figure 9: EMMA prediction visualization. Each row contains a scenario with our model's predictions: end-to-end planning trajectory (left), 3D object detection (middle), and road graph estimation (right).
17

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
(i) While crossing an intersection, our predicted trajectory nudges slightly to the left due to nearby cars and bicyclist partially occupying our lane.
(i) Our model predicts driving trajectory to patiently wait at red light (left). The model also accurately predicts surrounding 3D objects (middle) and road graph with lane centers (right).
(k) A fleet of fast-moving motorcyclists pass The predicted trajectory suggests pausing to allow them pass safely. Notably, motorcyclists accurately identified by model (middle).
(1) motorhike moving on narrow lane night, and yields right. Our predicted trajectory adjusts, guiding us pass safely by nudging slightly to the left.
Figure 10: EMMA prediction visualization. Each row contains scenario with our model's predictions: end-to-end planning trajectory (left), 3D object detection (middle), and road graph estimation (right).
18

---

Published in Transactions on Machine Learning Research (07/2025)
frameworks such as VAD (Jiang et al., 2023 Chen et al. 2024c). UniAD (Hu et al. 2023 PARA-Drive (Weng et al. 2024), and GenAD (Zheng et al. 2024 integrated planning with conventional modules in open-loop environments. More studies have been proposed to examine the robustness, safety, and transferability from synthetic environments to real-world domains. However, recent findings from AD-MLP (Zhai et al.) 2023 and BEV-Planner (Li et al. 2024) revealed that these methods could potentially overfit to ego status despile their good performance on benchmarks. Our work revisits the simplicity of earlier end-to-end models such as ALVINN and DAVE-2, enhancing them with powerful MLLMs.
Vision language models for autonomous driving have gained increasing interest, focusing achieving explainable driving behavior and generalizability through end-to-end learning frameworks. DriveGPT4 (Xu et al. 2024) and SiDrriee/had et al. (2024) utilize LLMs to explain vehicle actions and predict control signals in an iterative Q&& format. Drive Anywhere (Wang et al. 2024c) introduces
feature extraction
from MLLMs for text-based driving decision queries, while OmniDrive (Wang et al., 2024a) features a 3D vision-language model design for reasoning and planning. Other approaches use MLLMs in graph-based VQA contexts ((Sima et. al., 2024)), integrate LLMs in a BEV-based planner ((Pan et al.
2024h, or apply
chain-of-thoughtt reasoning (Tian et al. 2024 Wang et al. 2024b Bhattacharyya et al.
2023 to tackle
multiple driving-related tasks. Modular architectures such as LLM-Drive (Chen et al.
2024b leverage
LLMs with object-level vector inputs for planning, In contrast, our work studies end-to-end fine-tuning of a state-of-the art MLLM for driving tasks, employing a generalist approach that emphasizes open-world driving capabilities.
Multimodal large language models (MLLM) extend LLMs Vaswani et al.
2017
Devlin 2019 Raffel
et
al.
2020
Gemini Team Google, 2023
Reid al., 2024
Chowdhery et al.
2023:
Anil et al.) 2023 Rad ford
et al.
2018 2019 Brown et al. 2020 Achiam et al. 2023
Touvron et al. 2023a bj Dubey et al. 2024 to
multiple modalities, leveraging their generalizability, reasoning capabilities, and contextual understanding.
Early explorations (Donahue et al., 2015 Vinyals et al. 2015
Chen et al.) (022aa) focused on specific
vision-language problems or open-set object detoction Liu ct al.
(0024b); Zarcian et al. (2021
Gu et al.
(2022), while recent research has scaled up both trask diversity and model sizes for improved ceneralizability
and few-shot capabilities (Cho et al.]
2021 Chen et al.
2022b Wang et al.)
2022 Lu
2022 Alayrac
ct al.
2022 Yu ct al. 2022 Chen al.
2023 2024d
Wang et al. 2024d
Peng et al. 2024
Huang et al.
2023
Lu et al. 2024 Notable examples include Flamingo (Alayrac et al.
2022). 70B model (Hoffmann 2022 which achieved state-of-the-art quality for multiple few-shot vision benchmarks, and CoCa (Yu
let al. et al.
2022 2.1B parameter model which demonstrated state-of-the-ar performance on zero-shot transfer and various downstream tasks including ImageNet classification. PaLI (Chen et al. 2023 2024d), at 55B parameters, achieves better performance across multiple vision and language tasks by scaling both the vision and language model components jointly. These carly works demonstrate the strong performance and generalizability of MLLMs. Recent trends have seen the integration of native multi-modal inputs in LLMs, such Gemini (Gemini Team Google 2023 Reid al.) 2024). GPT-4o, and Llama3-v (Dubey et al. 2024 Liu et al., 2024a). Notably, researchers also apply MLLMs to robotic navigation (Zhang et al. 2024 Sun
ct al. 2024) and manipulation (Brohan et al., 2023: Alakuijala et al. 2024 Wang et al.) 2023). Our work
explores the application of these promising new models for generalist end-lo-end autonomous driving.
5 Conclusion
In this paper, we present EMMA, a Gemini-powered end-to-end multimodal model for autonomous driving. It treats Gemini as a first class citizen and recasts autonomous driving tasks as vision question answering problems to fit the paradigm of MLLMs, aiming at maximizing the utility of Gemini's world knowledge and its reasoning capability equipped with chain-of-thoughh tools. Unlike historical cascaded systems with specialized components, EMMA directly maps raw camera sensor data into various driving-speciff outputs, including planning trajectories, perception objects, and road graph elements. All task oulputs are represented as plain text and thus can be jointly processed in a unified language space through task-specific prompts. Empirical results show that EMMA achieves stato-of-thee-rt or competitive results on multiple public and internal benchmarks and tasks, including end-to-end planning, canera-primat 3D object detection, road graph estimation, and scene understanding. We also demonstrate that single co-trained EMMA can
19

---

Published in Transactions on Machine Learning Research (07/2025)
predict multiple tasks, while matching even super-passing the performance of individually trained models, highlighting its potential as generalist model for autonomous driving.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ngee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint AXK.:2303.0877, 2023.
Minttu Alakuijala, Reginald McLean, Isaac Woungang, Nariman Farsad, Samuel Kaski, Pekka Marttinen, and Kai Yuan. Video-language critic: Transferable reward functions for language-condingg conditoone robotics. Transactions on Machine Learning Research, 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NewrIPS, 2022.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXie preprint arXiv:2305. 10400, 2023.
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. RSS, 2019.
Apratim Bhattacharyya, Sunny Panchal, Mingu Lee, Reza Pourreza, Pulkit Madan, and Roland Memisevie. Look, remember and reason: Grounded reasoning in videos with language models. In ICRA, 2023.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for self-driving cars. arXiv preprint or XXXx:1604.0730 2016.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Kraysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-languagg-action models transfer web knowledge to robotic control. In CoRL, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Tn NewrIPS, 2020.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, and Fabicn Moutarde. Gri: General reinforced imitation and its application to vision-based autonomous driving. Robolics, 2023.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In CoRL, 2020.
Dian Chen, Vladlen Koltun, and Philipp Krähenbühl. Learning to drive from a world on rails. In ICCV, 2021.
Kan Chen, Runzhou Ge, Hang Qiu, Rami Ai-Rfou, Charles R Qi, Xuanyu Zhou, Zoey Yang, Scott Ettinger, Pei Sun, Zhaoqi Leng, et al. Womd-lidar: Raw sensor dataset benchmark for motion forecasting. In ICRA, 2024a.
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with 1lms: Fusing object-level vector modality for explainable autonomous driving. In ICRA, 2024b.
20

---

Published in Transactions on Machine Learning Research (07 (225))
Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402. 13243, 2024c.
Ting Chen, Saurabh Saxena, Lala Li, David J Flect, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. In ICLR, 2022a.
Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yr Yi Lin, David J Fleet, and Geoffrey E Hinton. A unified sequence interface for vision tasks. In NeurIPS, 2022b.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A Jointly-Scaled Multilingual Language- Image Model. In TCLR, 2023.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Scbastian Goodman, Xiao Wang, Yi Tay, et al. Pali-x: On scaling up multilingual vision and language model. In CVPR, 2024d.
Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-basse sensor fusion for autonomous driving. PAMI, 2022.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unilying vision-and-languaage lasks via text generation. In ICML, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. JMLR, 2023.
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In ICRA, 2018.
Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking, NewrIPS, 2024.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding, In NAACL, 2019.
Wenjie Ding, Limeng Qiao, Xi Qiu, and Chi Zhang. Pivotnet: Vectorized pivot learning for end-to-end hd map construction. In ICCV, 2023.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kale Saenko, and Trevor Darrell. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2015.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishck Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint 47X.v:2407.91783, 2024.
Gemini Team Google. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:2312.11800 2023.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLRR 2022.
21

---

Published in Transactions on Machine Learning Research (07/2025)
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. In NewrIPS, 2022.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, 2023.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. In NewrIPS, 2023.
Wei-Chih Hung, Vincent Casser, Henrik Kretzschmar, Jyh-Jing Hwang, and Dragomir Anguelov. Let-3d-ap: Longitudinal error tolerant 3d average precision for camera-only 3d detection. In ICRA, 2024.
Jyh-Jing Hwang, Henrik Kretzschmar, Joshua Manela, Scan Rafferty, Nicholas Armstrong-Cret Tiffany Chen, and Dragomir Anguelov. Cramnet: Camera-radar fusion with ray-constrained cross-attention for robust 3d object detection. In ECCV, 2022.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In ICCV, 2023.
Alex Kendall, JefTrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day. In ICRA, 2019.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast. encoders for object dctection from point clouds. In CVPR, 2019.
Isabel Leal, Krzysztof Choromanski, Deepali Jain, Avinava Dubey, Jake Varley, Michael Ryoo, Yao Lu, Frederick Liu, Vikas Sindhwani, Quan Vuong, et al. Sara-rt: Scaling up robotics transformers with self-adaptive robust atlention. In ICRA, 2024.
Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In ICRA, 2022a.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's sec's--yy-view representation from multi-camera. images via spatiotemporal transformers. In ECCV, 2022b.
Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M Alvarez. Is ego status all you need for open-loop end-to-end autonomous driving? In CVPR, 2024.
Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. Cirl: Controllable imitative reinforcement learning for vision-based self-driving. In ECCV, 2018
Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construction. In ICLR, 2023.
Bencheng Liao, Shaoyu Chen, Bo Jiang, Tianheng Cheng, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Lane graph as path: Contimuity-presevvii path-wise modeling for online lane graph construction. In ECCV, 2024a.
Bencheng Liao, Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Maptrv2: An end-to-end framework for online vectorized hdd map construction. IJC,, 2024b.
Vasilcios Lioutas, Adam Scibior, and Frank Wood. Titrated: Learned human driving behavior without infractions via amortized inference. Transactions on Machine Learning Research, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NewrIPS, 2024a.
22

---

Published in Transactions on Machine Learning Research (07/2025)
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024b.
Yicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang, and Hang Zhao. Vectormapnet: End-to-end vectorized hd map learning, In ICML, 2023.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In ICLR, 2022.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In CVPR, 2024.
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. In ICRA, 2023.
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning for autonomous driving. In CVPR, 2024.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. Tn ICLR, 2024.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In NewrIPS, 1988.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In CVPR, 2021.
Limeng Qiao, Wenjie Ding, Xi Qiu, and Chi Zhang. End-to-end vectorized hd-map construction with piecewise bezier curve. In CVPR, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Lya Sutskever, et al. Improving language understanding by generative pre-training- OpenAT blog, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Nyy Sutskever, el al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicra,, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint or orXiv:2403.055.0 0555330 2024.
Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In ICCV, 2023.
Hao Shao, Yuxuan Hu, Letian Wang, Guanglu Song, Steven I. Waslander, Yu Liu, and Hongsheng Li. Lmdrive: Closed-loop end-to-end driving with large language models. In CVPR, 2024.
Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Mtr++: Multi-agent motion prediction with symmetric scene modeling and guided intention querying. PAMI, 2024.
Choughao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongvang Li. Drivelm: Driving with graph visual question answering. In ECCV, 2024.
Pci Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV, 2022.
Xingpeng Sun, Haoming Meng, Souradip Chakraborty, Amrit Singh Bedi, and Aniket Bera. Beyond text: Utilizing vocal cues to improve decision making in Ilms for robot navigation tasks. Transactions on Machine Learning Research, 2024.
Matthew Tancik, Vincent. Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022.
Siyu Teng, Xuemin Hu, Peng Deng, Bai Li, Yuchen Li, Yunfeng Ai, Dongsheng Yang, Lingxi Li, Zhe Xuanyuan, Fenghua Zhu, et al. Motion planning for autonomous driving: The state of the art and future perspectives. T-IV, 2023.
Xiaoyu Tian, Junru Gu, Bailin Li, Yicheng Liu, Chenxu Hu, Yang Wang, Kun Zhan, Peng Jia, Xianpeng Lang, and Hang Zhao. Drivevlm: The convergence of autonomous driving and large vision-language models. In CoRL, 2024.
Marin Toromanoff, Emilic Wirbel, and Fabicn Moutarde. End-to-end model-frec reinforcement lcarning for urban driving using implicit affordances. In CVPR, 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient. foundation language models. arXiv preprint Xiv:20332 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arxi preprint or (a0::207709288 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Atlention is all you need. In NewrIPS, 2017.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.
Royden Wagner, Omer Sahin Tas, Marvin Klemp, and Carlos Fernandez. Jointmotion: Joint self-supervision for joint motion prediction. arXiv preprint Xiv:2440 05489, 2024.
Mianchu Wang, Rui Yang, Xi Chen, Hao Sun, Meng Fang, and Giovanni Montana. Goplan: Goal-conditioned offline reinforcement learning by planning with learned models. Transactions on Machine Learning Research, 2023.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple learning framework. In ICML, 2022.
Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint Kxi:2245500153 2024a.
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Feos3d: Fully convolutional one-stage monocular 3d object detection. In ICCV, 2021.
Tianqi Wang, Enze Xie, Ruihang Chu, Zhenguo Li, and Ping Luo. Drivecot: Integrating thtougt reasoning with end-to-end driving. arXiv preprint arXix:2408. 16996, 2024b.
Tsun-Hsuan Wang, Alaa Maalouf, Wei Xiao, Yutong Ban, Alexander Amini, Guy Rosman, Sertac Karaman, and Daniela Rus. Drive anywhere: Generalizable end-to-end autonomous driving with multi-modal foundation models. Tn ICRA, 2024c.
24

---

Published in Transactions on Machine Learning Research (07/2025)
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NewrIPS, 2024d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, ct al. Chain-of-thought prompting clicits reasoning in large language models. In NeurIPS, 2022.
Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In CVPR, 2024.
Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guider control prediction for end-to-end autonomous driving: A simple yet strong baseline. In NeurIPS, 2022.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kwan- Yee K Wong, Zhenguo Li, and Hengshuang Zhao. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. RA-L, 2024.
Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arViv preprint ar #xXx:2311.00077 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca:
Contrastive captioners are image-text foundation models. TMLR, 2022.
Tianyuan Yuan, Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao Streammapnet: Streaming mapping network for vectorized online hd map construction. In WACV, 2024.
Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of autonomous driving: Common practices and emerging technologies. IEEE access, 2020.
Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, 2021.
Jiang- Tian Zhai, Ze Feng, Jinhao Du, Yongqiang Mao, Jiang-Jiang Liu, Zichang Tan, Yifu Zhang, Xiaoqing Ye, and Jingdong Wang. Rethinking the open-loop evaluation end-to-end autonomous driving in nuscenes. Xiviv preprint 2rXie:2305.10430, 2023.
Jimuyang Zhang and Eshed Dar-Bar. Learning by watching, In CVPR, 2021.
Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. arXiv preprint ar 201102711.010017, 2023a.
Yue Zhang, Ziqiao Ma, Jialu Li, Yanyuan Qiao, Zun Wang, Joyce Chai, Qi Wu, Mohit Bansal, and Parisa Kordjamshidi. Vision-anddlanguuaee navigation today and tomorrow: A survey in the era of foundation models. Transactions on Machine Learning Research, 2024.
Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In ICCV, 2021.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. Thansactions on Machine Learning Research, 2023b.
Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chemming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. In ECCV, 2024.
25

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
A
Appendix
We organize this appendix section as follows:
We summarize all categories in the meta decision in the chain-of- chain-of-thought reasoning formulation in Table G
2. We supply details on the 3D object detection metrics.
3. To provide insights for future development, we present three distinct failure examples in Figure
4. To facilitate reproduction, we provide an example of the concrete prompts used for EMMA Generalist and their corresponding model-predicted answers in Table 7
5. Despite the promising results, we acknowledge the limitations of our work and propose directions for future research in Section A.5
A.1 Meta Decision in Otain-of-Thhouhh Reasoning
We describe the chain-of-thought reasoning in Section 2.2 in the main paper. One important component is the meta decision, where we first partition the decision into 12 categories with heuristics and transform them into natural languages.
We summarize the categories in Table G We use speed 3 different future timestamps, i.e., at future 0, 1, and 3 seconds, as the decision points for different categories. If we are able to identify the cause of speed changes, e.g,, due traflic signs critical objects, then also append to the description. We plan to explore more fine-grate fine-grained meta decisions and reasoning in future work.
A.2 Distance Breakdowns of 3D Detection Metrics
We plot the distance breakdowns of the camera-primary 3D object detection experiments in Figure 11 We observe that the performance gap between EMMA+ and the baseline models diminishes the distance to objects increases. We attribute this phenomenon to the potentially lower resolution of the camera input used by EMMA compared to that of the baseline models.
Speed at Os stationary
Speed at 1s
Speed at 3s
Meta Decision Description
stationary
stationary
"Stay stationary."
stationary
moving
"Start moving soon."
stationary
stationary
moving
"Stay stationary for now, then start moving soon."
moving
constant
constant
Keep speed."
moving
constant
increase
"Keep speed, then accelerate."
moving
constant
decrease
"Keep speed, then brake."
moving
increase
increase
"Acellrrae.""
moving
increase
constant
"Acellratee then keep high speed."
moving
increase
decrease
"Aceleratee then brake."
moving decrease
decrease
"Brake."
moving decrease moving
constant "Brake, then keep low speed." "Brake, then accelerate."
decrcase
increase
Table 6: Summary of 12 categories of the meta decision in chain-of-thoughl reasoning. We use speed at different future timestamps, i.e., at future 0, 1, and 3 seconds, as the decision points. If we are able to identify the cause of speed changes, e.g., due to traffic signs or critical objects, then we also append to the description.
26

---

Published in Transactions on Machine Learning Research (07/202) /025)
Vehicle Range (0, 30)
Pedestrian Range (0. 30)
precision
0.-
precision
06
*
0.4
EMMA
FI: 0:45 EMMA
Fi-max: 0,90 BEVFormer
F1-max 0.80 BEVFormer
Fi-mex: 0.91 MV-FCOS3D-
F1-maxc 0.76 MV-FCOS3D++
0.0
02
0.4
04
G8
0.0
02
0.4
04
08
16
recall
recall
Vehicle Range (30, 50)
Podestrian Range (30, 50)
10
0.8
0.
precision
06
0.
0.89 EMMA+
0.62 EMMA-
Ft: 0.59 EMMA
FI: 0.20 EMMA
F1-max: 0.82 BEVFormer
Fl-max: 0.71 BEVFormer
F1-max: 0.81 MV-FCOSID
F1-max: 0.60 MV-FCOS3D++
11.3
recall
recall
Vehicle Range +inf)
Pedestrian Range +inf)
18
10
0.8
precision
Ft: 0.73 EMMA-
F1: 0.35 EMMA- Fl: 0.08 EMMA
F1: 0.39 EMMA
0.2
F1-maxc BEVFormer
F1-max: 0.53 BEVFormer
MV-FCO53D--
MV-FCOS3D++
015
0.
recall
Figure 11: Distance breakdowns the camera-primary 3D object detection experiments on WOD Sun
et al. 2020 using the standard LET matching (Hung et al., 2024). We observe that the performance gap between EMMA+ and the bascline models diminishes as the distance to objects incrcases. We attribute this phenomenon to the potentially lower resolution of the camera input used by EMMA compared to that of the baseline models.
A.3 Failure Examples
While Section 3.6 showcases numerous successful predictions from EMMA across trajectory, detection, and road graph tasks, equally important analyze its limitations. To provide insights for future development, present three distinct failure scenarios in Figure 12 each illustrating different type of error.
27

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
EMMA's predicted trajectory exhibits suboptimal clearance from the motorbike. While the prediction is validated when the motorbike later tracks further right, the ideal path would have included greater initial lateral offset.
EMMA fails detect distant oncoming vehicle, though correctly identified subsequent frame. This one-frame detection delay suboptimal, as the ego vehicle plans to nudging to the left. Earlier detection would have led to a safer and more controlled trajectory.
While EMMA correctly identifies the immediate drivable lanes, it fails to detect an upcoming lane merge further down the road. This oversight may be attributed to the degraded painted arrow.
Figure 12: Failure examples of EMMA prediction. Each row contains a scenario where EMMA's predictions were partially flawed: end-to-end planning trajectory (left), 3D object detection (middle), and road graph estimation (right).
A.4 Concrete Prompts and Answers
As demonstrated in Scction 2.3 and Figure 2 EMMA is capable of generating dosired outputs based on varying prompts. To facilitate reproducibility, Table provides the specific prompts used and their correspouding model-predicted answers.
28

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Image
Prompt and Answer
Prompt: Assume I am driving at the coordinate 0.0, 0.0. The high-level behavior attention
is: go straight. The past trajectory under vehicle coordinate is: .4.09, 0.01 and -3.94, 0.01 and
-3.71, 0.01 and -3.42, 0.01 and -3.05, 0.00 and -2.60, 0.00 and -2.07, -0.00 and -1.45, -0.00 and
-0.76, -0.00 and 0.00, 0.00. The past ego velocity under vehicle coordinate la: 0.91, -0.01 and
1.29, -0.01 and 1.66, -0.01 and 2.04,
and 2.45, -0.02 and 2.88, -0.02 and 3.26, -0.02
3.65, -0.01 and 3.98, -0.00. The past ego acceleration under vehicle coordinate is: 0.32. -0.00
and 0.34, -0.00 and 0.37, -0.00 and 0.38, -0.00 and 0.40, -0.00 0.43, 0.00 and 0.38,
and 0.39, 0.01 and 0.34, 0.01. What my future trajectory in next seconds under vehicle coordinate?
Answer: 0.83, 0.01 and 1.72, 0.01 and 2.67. 0.02 and 3.67, 0.04 and 4.73, 0.05 and 5.84, 0.07 and 6.95, 0.09 8.08, and 9.23, 0.13 and 10.41, and 11.62, 0.17
12.87, 0.19 and 14.15, 0.22 and 15.47, 0.24 and 16.80, 0.26 and 18.15, 0.27 and 19.52, 0.28 and 20.92, 0.28 and 22.36, 0.28 and 23.83, 0.28 and 25.34, 0.27 and 26.89, 0.26 and 28.49, 0.25 and
30.14, 0.23 and 31.83, 0.22
What are the 3D bounding boxes for objects in the following list: vehicle, pedestrian, and eyclist? Describe the bounding hoxes in the format, of [renter_xx x, center y, bottom x, width, length, beight, heading, class), where heading is in radian, class in text, and other dimensions are meter?
Prompt:
Answer: -12.91 -9.23 -0.21 12.99 3.21 3.45 -2.25 vehicle and 21.81 .4.15 -0.15
2.10 1.65 -2.18 vehicle and 25.61 -1.11 -0.12 1.89
0.86 1.86 -0.23 eyelist and 39.12 3.25 -0.13
4.39 2.13 1.79 3.14 vehicle and 45.51 3.18 -0.12 4.65 2.10 1.54 3.14 vehicle and 43.81 21.51
-0.01 4.71 2.19 1.51 -2.16 vehicle and 40.01 20.11 0.09 4.92
2.39 2.02 -2.17 vehicle and 44.81
3.16 -0.11 4.51 2.10 1.55 3.14 vehicle and 19.11 13.39 0.02 1.11 0.83 1.68 -0.05 pedestrian and
3.02 0.05
2:23 1.92 -3.14 vehicle and 51.11 15.41
1.12
0.85 1.78
pedestrian
and 62.51 3.35
0.49
4.51
2.15 1.85 -3.14 vehicle and 72.11 2.99
0.92
4.45 2.13 1.74 -3.13 vehicle
and 79.21 3.41
2.14
1.72 -3.13 vehiele and 85.51 3.35 1.45
4.36 2.08 1.60 -3.13 vehicle
and 85.11 11.11 1.41 4.49 2.15 1.76 3.11 vehicle and 92.21 3.51 1.69 4.45 2.18 1.90 -3.13 vehicle
Prompt: My current position is 1.28, -0.00. Where are driveable lanes in frout of me? Describe the lanes series of points relative 0.00, 0.00?
Answer: (0.00, -0.11 and 10.00, -0.11 and 20.00, -0.11 and 30.00, -0.11 and 40.00, 0.11 and 51.11, -0.11 and invalid and invalid and invalid and invalid) valid (51.11, -0.11 and 61.10, -0.06 and 71.09,
and 81.08,
invalid 91.07, 0.09 and invalid and invalid
and invalid and invalid) valid: (0.00. -0.12 and 10.00. 19.19 and 15.11, -0.22 and 21.99, 0.62 and 27.72, 2.16 and 33.11, 4.65 and 38.02, 8.00 and 42.35, 12.05 and invalid and invalid) valid (0.00, -0.12 and 10.00, 10.13 and 17.11, -0.14 and 23.01. 0.58 and 28.76, 2.09 and 34.16, 4.56
and 39.07, 7.91 and 43.43, 11.95 and invalid and invalid) valid (0.00, -0.12 and 10.00, -0.13
and 16.11, -0.14 and 22.01, 0.35 and 27.76, 2.05 and 33.14, 4.55 and 38.00 38.00, 7,98 and 42.29, 12.09 and invalid aud invalid) valid: (0.00, -3.31 and 10.00. -3.25 and 20.00, -3.19 and 30.00.
-3.13 and 40.00, -3.07 and 49.35, -3.01 and invalid and invalid and invalid and invalid) valid
(invalid and invalid and invalid and invalid and invalid and invalid and invalid and invalid and invalid and invalid) invalid... ...
Table 7: The specific prompts used and their corresponding model-predicted answers. Numerical values are color-coded in blue and predicted separators in red for better visualization.
29

---

Published in Transactions on Machine Learning Research (07/2025)
A.5 Limitations, Risks, and Mitigations
In the main paper, we demonstrate state-of-the-art end-to-end motion planning on the nuScenes planning benchmark. We also achieve competitive performance for camera-primary 3D detection on WOD. Furthermore, our generalist setup improves the quality across multiple tasks through joint training. Despite these promising results, we acknowledge the limitations of our work and propose directions for building on this foundation and addressing such challenges in future research.
Memory and video capability: Currently, our model processes only a limited number of frames (up to 4), restricting its ability to capture the long-term dependencies essential for driving tasks. Effective driving requires not just real-time decision-making but also reasoning over extended time horizons, relying long-term memory to anticipate and respond to evolving scenarios. Enhancing the model's ability to perform long-term reasoning is promising area for future research. This could potentially be achieved by integrating memory modules or extending its capability to process longer video sequences efficiently, enabling more comprehensive temporal understanding.
Extension to LiDAR and radar input: Our approach heavily relies on pre-trained MLLMss, which typically do not incorporate LiDAR or Radar inputs. Expanding our model to integrate these 3D sensing modalities presents two key challenges: 1) There is a significant imbalance between the volume of available camera and 3D sensing data, resulting in less generalizable 3D sensing encoders camera-based counterparts. 2) The development of 3D sensing encoders has not yet reached the scale and sophistication of camera-based encoders. A potential solution to address these challenges is to pre-train a large-scale 3D sensing encoder using data carefully aligned with camera inputs. This approach may foster better modas-mydality synergy and substantially improve the generalization capabilities of the 3D sensing encoder.
compared to their
Verification of the predicted driving signals: Our model can directly predict driving signals without. relying on intermediate outputs, such as object detection or road graph estimation. This approach introduces challenges for both real-time and post-hoc verification. We have demonstrated that our generalist model can jointly predict additional human readable outputs such as objects and road graph elements, and the driving decision can be further explained with chain-of-thought driving rationale. However, there is no guarantee that these puts will be always consistent despite the empirical observations that they are often indeed consistent. Improving driving rationale is one of our future research directions.
Sensor simulation for closed-loop evaluation: To accurately assess end-to-end autonomous driving system in closed-loop environment, comprehensive sensor simulation solution is necessary. However, the computational cost of sensor simulation is often much higher than that of behavior simulators. This significant cost burden can hinder thorough testing and verification of an end-to-end models; however, the field of efficient sensor simulation methods continues to rapidly evolve, with the promisc of significantly miligating this burden.
More targeted and accurate open-loop evaluation: While existing open-loop evaluation offers low computational cost, its results are sometimes unreliable. The popular nuScenes (Cacsar et al. 2020) benchmark, for instance, exhibits well-documented limitations. Key metrics like collision rate are sensitive to hyperparameter choices, such as the BEV grid resolution (Weng et al. 2024 Zhai et al. 2023). Furthermore, many of its scenarios lack planning diversity and can be trivially solved by extrapolating historical trajectories. Initiatives like NAVSIM (Dunner et al. 2024) are beginning to address these vulnerabilities, but this highlights a critical need for future work on developing more challenging and trustworthy open-loop evaluation frameworks.
Challenges of onboard deployment: Autonomous driving demands real-time decision-making, which poses significant challenge when deploying large models due to their increased inference latency. This creates a need for optimizing the model or distilling it into a more compact form suitable for deployment, all while maintaining performance and safety standards.
While the main paper is focused on establishing the EMMA architecture, the framework is amenable to various well-established optimization techniques. To illustrate this adaptability, we have experimented with a latency-optimized EMMA configuration. By incorporating strategies such as SARA-RT (Leal et al. 2024).
30

---

Published in Transactions on Machine Learning Research (07 (07/2025)
employing shorter action sequences, streamlining by removing explicit reasoning chains, etc., this variant achieves an inference speed of 3 FPS, a 67% speedup compared to UniAD's 1.8 FPS. This preliminary result underscores that the core EMMA architecture possesses the flexibility to be effectively adapted into specialized, low-latency variants. Achieving a delicate balance between model size, efficiency, and quality is crucial for the successful real-world deployment of autonomous driving systems, and represents a key area for future rescarch.
31


Reviews:
[{"review_id": "iBlYYsAZK3", "reviewer": "Reviewer_DKk7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nS1) The authors make a pioneering attempt to unify all driving tasks as a language processing formulation. The proposed framework is simple yet powerful, achieving state-of-the-art performance.\n\nS2) Sufficient details are provided to help the audience understand the behavior of the model.\n\nS3) The authors also demonstrate the benefits of chain-of-thought reasoning and co-training, which will inspire future research in developing a generalist driving agent.\n\nS4) The presentation is clear and easy to follow.\n\n**Weaknesses**\n\nW1) My biggest concern is the latency of this framework. The paper does not compare previous methods in terms of inference speed. Autoregressively predicting chain-of-thought may exacerbate this issue, which contradicts real-world applications.\n\nW2) The proposed method is developed based on a private Gemini model using in-house datasets, making the community hard to follow.\n\nW3) From my understanding, the chain-of-thought component requires labels from other models. Could you clarify why EMMA can still be considered a self-supervised method in this case?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces EMMA, an end-to-end multimodal model for autonomous driving built on top of Gemini. Instead of using modular designs, it directly maps raw observations to desired scene understanding and planning outputs in the form of natural language. Through a hierarchical reasoning process, the resulting model achieves state-of-the-art planning results on both nuScenes and an in-house large-scale benchmark. The paper also explores the benefits of co-training with other perception tasks, showing the potential of this unified framework.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "C1) It would be beneficial to report and compare the inference speed.\n\nC2) Evaluating planning performance on nuScenes is known to be unstable [1,2]. Testing on more recent benchmarks like NAVSIM would be more convincing.\n\n---\n\n[1] Rethinking the Open-Loop Evaluation of End-to-End Autonomous Driving in nuScenes\n\n[2] Is Ego Status All You Need for Open-Loop End-to-End Autonomous Driving?", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WhC5Pr8klt", "reviewer": "Reviewer_HGfn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* This paper presents a thorough set of experiments across multiple autonomous driving tasks and provides clear, well-structured comparisons with existing methods. The results are systematically reported, and the manuscript is written in a clear and accessible manner, making it easy to follow the design and understand the full picture. \n* It presents a unified end-to-end model that reframes multiple autonomous driving tasks-including motion planning, 3D object detection, and road graph estimation-as vision-language problems using a MLLM foundation, enabling joint processing and co-training across tasks for improved efficiency and performance.\n* The model leverages the broad world knowledge and generalization capabilities of MLLMs, allowing it to better handle rare and complex driving scenarios that are underrepresented in standard autonomous driving datasets; as training data and foundation models scale, EMMA’s performance and robustness continue to improve, especially in long-tail situations.\n* It achieves SOTA results in end-to-end motion planning on public benchmarks such as nuScenes, and demonstrates competitive or superior performance in 3D object detection and road graph estimation compared to existing methods.\n* The integration of a hierarchical CoT reasoning module requires EMMA to generate structured rationales-including scene descriptions, identification and behavior of critical objects, and meta driving decisions-prior to trajectory prediction, which not only boosts planning quality but also enhances the explainability of the model’s decisions.\n ---\n\n## Weaknesses\n* The impact of this work could be significantly increased by open-sourcing the model, dataset, or benchmark, enabling broader access and fostering further research and development within the community.\n* Limited number of image frames restricts its ability to capture long-term dependencies and hinders reasoning about evolving or complex scenarios that require extended temporal context.\n* Although EMMA can jointly predict driving signals and human-readable outputs such as object lists and road graphs, there is no formal guarantee of outputs consistency, posing challenges for both real-time and post-hoc verification of the model’s decision-making-a critical issue for autonomous driving safety.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "EMMA introduces a unified, end-to-end multimodal model for autonomous driving built on a large multimodal language model foundation, which directly maps raw camera data and text inputs to key driving outputs such as planner trajectories, 3D object detections, and road graph elements. By representing all non-sensor inputs and outputs as natural language, EMMA enables joint processing of diverse driving tasks in a single language space, supporting flexible prompting and co-training across tasks. The model achieves cutting-edge results in end-to-end motion planning on public benchmarks, as well as competitive performance in 3D object detection and road graph estimation. EMMA’s generalist design allows a single model to match or surpass the performance of individually trained models on multiple autonomous driving tasks, demonstrating strong scalability and robustness. The integration of a CoT reasoning module further enhances both planning performance and model explainability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Open-source the EMMA model, its dataset, or its benchmark to enable broader access for the research community, facilitate further research and development, and enhance the overall impact of the work.\n* Extend EMMA’s temporal reasoning capabilities by enabling longer video sequence processing, larger context window, or integrating memory modules.\n* Investigate methods to reduce computational overhead for real-time deployment.\n* Develop formal mechanisms to verify consistency across predicted outputs and enhance safety assurances.", "broader_impact_concerns": "* The model’s ability to generate detailed scene descriptions (via chain-of-thought rationales) raises privacy risks if deployed in real-time systems, particularly when combined with surveillance-capable sensors.\n* Accountability Gaps. As a generalist model producing multiple outputs (trajectories, object detections, road graphs), EMMA complicates liability attribution in accidents. Existing legal frameworks struggle to assign responsibility between developers, manufacturers, and users when AI systems fail. \n* Widespread adoption of EMMA-based systems could disrupt transportation-sector jobs, disproportionately affecting drivers in logistics and ride-hailing services.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fqu9laqEYY", "reviewer": "Reviewer_3oH1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nEMMA stands out for its unified multimodal architecture, combining motion planning, object detection, and road graph estimation into a single vision-language model, which eliminates the need for separate perception modules and enhances generalization across various tasks. Another strength is its incorporation of chain-of-thought reasoning, where the model generates structured natural language explanations for its decisions, leading to a significant 6.7% improvement in planning accuracy. Finally, EMMA demonstrates state-of-the-art performance using only monocular camera input, outperforming many models that rely on LiDAR or HD maps, showcasing the effectiveness of pretraining large multimodal models for autonomous driving tasks.\n\n**Weakness**:\n\n1. Can authors provide more concrete details about the internal dataset used for pretraining and evaluation? The paper does not specify the scale, the types of long-tail scenarios included, or how diverse driving behaviors are captured. Since EMMA+ benefits significantly from this internal data, understanding how it was constructed—especially regarding navigation prompt design and scenario diversity—is crucial for evaluating the model’s planning results on Nuscenes dataset.\n\n2. Can authors clarify the training configurations used for each experimental result in Section 3? While Section 2 clearly outlines the objective functions for different tasks, it remains unclear how the results in different tables were obtained. For example, in Table 1, the planning results appear to come from training EMMA only on planning data, whereas in Section 3.3, EMMA+ is pre-trained on the 3D detection task. Although the paper emphasizes EMMA as a unified model, many experiments seem to be conducted with task-specific training, which makes me confused.\n\n3.  Can authors further clarify whether the end-to-end planning tasks in Section 3.4 include the reasoning component introduced earlier in the paper? I think that Section 3.4 is arguably the most important part of the experimental analysis, as it directly evaluates the effect of each task on the performance of the unified driving model. While the paper provides a clear breakdown of how different tasks contribute to planning, detection, and mapping, it remains unclear whether chain-of-thought reasoning is integrated during multitask training. One of the key advantages of using a vision-language model for autonomous driving lies in its explainability and open-vocabulary reasoning capabilities. However, these strengths are not sufficiently discussed or evaluated. Although Table 2 shows that reasoning yields some improvements in planning metrics, this gain is measured in an open-loop setting, where the practical value of such improvement is limited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The EMMA paper introduces a novel end-to-end autonomous driving model that unifies motion planning, 3D object detection, and road graph estimation within a single large multimodal language model framework. By representing both inputs (multi-camera images) and outputs (trajectories, objects, maps) in natural language, EMMA leverages the reasoning and generalization capabilities of a pre-trained foundation model (Gemini) without relying on LiDAR, HD maps, or intermediate perception modules. It further incorporates chain-of-thought (CoT) reasoning to explicitly describe the decision-making process behind trajectory predictions, leading to measurable improvements in planning accuracy. Through multitask co-training and large-scale vision-language pretraining, EMMA achieves state-of-the-art performance on nuScenes and Waymo benchmarks using only camera data, while offering enhanced interpretability and flexibility across tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Authors should include **collision rate** metrics in Table 1. Compared to L2 errors, collision statistics provide a more intuitive and safety-relevant measure of planning quality.\n\n2. The paper would benefit from including **failure cases** to illustrate the model’s current limitations. While several successful qualitative examples are shown, presenting scenarios where EMMA fails—such as inaccurate trajectory predictions, misaligned reasoning outputs, or poor handling of rare cases—would provide a more balanced evaluation and help clarify the boundaries of the model’s generalization capabilities.\n\n3. The paper should include a discussion on potential **latency** issues associated with LLM-based planners, as planning speed is a critical constraint in real-time autonomous driving systems. Given EMMA is built on a large multimodal language model, inference latency may limit its deployment in high-speed or latency-sensitive scenarios. \n\n4. The paper would be improved by including **concrete prompt and output examples** for each of the tasks, such as planning, object detection, and road graph estimation.\n\n5. By the way,  do the authors plan to make public the internal dataset you use in the future? I think this is far more significant than training a model", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eKjWXclq9C", "reviewer": "Reviewer_C53W", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Generating a driving rationale improves the interpretability of e2e planning models beyond auxiliary perception outputs.\n- The proposed hierarchical structure of the driving rationale is intuitive and well adapted to the different levels of behavior generation in autonomous driving.\n- The authors perform experiments on nuScenes, Waymo Open Motion, Waymo Open Dataset and a very large internal dataset. The results suggest that their generalist model is able to outperform specialized models on e2e motion planning and object detection.\n- Notably, the simplest self-supervised version of EMMA (cf. section 3.1) is able to outperform related methods that require more supervision during training.\n- They ablate their design choices, leading to interesting insights (e.g., that adding punctuation to sequences improves results).\n\nWeaknesses:\n- The description of how the text-based driving rationale is generated lacks details. For example, what are the carefully designed visual and text prompts, or what heuristic algorithm is used to generate meta driving decisions?\n- The road graph estimation (section 3.3.1) is not compared to other methods. A comparison with related work on map perception (Liao et al., 2023) or the map perception outputs of e2e models like UniAD would strengthen the paper.\n- The main results in table 1 highlight the model versions, which are trained in a self-supervised manner. Therefore, the related work section should include a paragraph on recent self-supervised learning methods for autonomous driving (e.g., for scene reconstruction (Yang et al., 2024), motion forecasting (Wagner et al., 2024), and world models (Zhang et al., 2024)).\n\nLiao et al., \"MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction\" ICLR 2023\n\nYang et al. \"EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision\" ICLR 2024\n\nWagner et al. \"JointMotion: Joint Self-Supervision for Joint Motion Prediction\" CoRL 2024\n\nZhang et al. \"Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion\" ICLR 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors built an e2e autonomous driving model using multimodal large language models.\nThe proposed model takes video data and high-level commands as input and generates trajectories of future waypoints. \nThe model can be trained in a self-supervised manner (only ego-vehicle trajectories as targets) or as a generalist model that also performs auxiliary perception tasks and generates a text-based driving rationale to improve its interpretability.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)