Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving
Authors: Thomas Tian
Boyi Li
Xinshuo Weng
Yuxiao Chen
Edward Schmerling
Yue Wang
Boris Ivanovic
Marco Pavone
Venue: CoRL 2024 (2024)
Keywords: Multi-modal LLM
Autonomous Driving
Representation Alignment
TL;DR: We propose TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization of LLM’s reasoning capabilities to enhance autonomous vehicle planning in long-tail scenarios.

Abstract: The autonomous driving industry is increasingly adopting end-to-end learning from sensory inputs to minimize human biases in system design. Traditional end-to-end driving models, however, suffer from long-tail events due to rare or unseen inputs within their training distributions. To address this, we propose TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization of LLM’s reasoning capabilities to enhance autonomous vehicle planning in long-tail scenarios. TOKEN effectively alleviates data scarcity and inefficient tokenization by producing condensed and semantically enriched representations of the scene. Our results demonstrate that TOKEN excels in grounding, reasoning, and planning capabilities, outperforming existing frameworks with a 27% reduction in trajectory L2 error and a 39% decrease in collision rates in long-tail scenarios. Additionally, our work highlights the importance of representation alignment and structured reasoning in sparking the common-sense reasoning capabilities of MM-LLMs for effective planning.

Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving
Ran Tian 12 Boyi Li Xinshuo Weng'
Yuxiao Chen' Edward Schmerling!
Yue Wang
Boris Iyanovic Marco Pavonel.4
IVVIDI UC Berkeley University of Southern California *Stanford University
TOKEN: A Multi-Modal LLM with Object-Centric Tokenization for Autonomous Driving
Hierarchical Planning
Q Gieen für input Jyfommstion, pleese...?
<Multi-view view video, HD-map, Symbolic representation>
Scome-oentric Tokenization
Object-entric
Tokenization
@: mitseess wade kep ferard slon Byr mat. fesitiet Place pian 3ornndd wator ylas. A: onlian stjcts goansfug.... tnierction yioming.... fave detiie
Large Language Model (LLaMA)
gevrallen...
A:
Scene Understanding
Reasoning
Q: sutonsenaas pay etbertive stject ienated
Q: Thr cotonanss owbride mede to kep fessmi along toss....eehhidd Flate plan it acondd watoos pire.
Wayy??
nurging adiidd's
Voliee 12.1)
A: Thme ispernt sbyete
(R6, The shrii remaix nougly stutic en....
rehstr stoul
(&66
line
The enster
Figure I: TOKEN is novel Multi-Modas Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization of LLM''s reasoning capabilities to enhance autonomous vehicle planning in long-tail scenarios.
Abstract: The autonomous driving industry is increasingly adopting end-to-end learning from sensory inputs minimize human biases in system design. Tradi- tional end-to-end driving models, however, suffer from long-tail events due to rare or unseen inputs within their training distributions. To address this, we propose TOKEN, a novel Multi-Modal Large Language Model (MM-LLM) that tokenizes the world into object-level knowledge, enabling better utilization LLM's rea- soning capabilities to enhance autonomous vehicle planning long-tail scenarios. TOKEN effectively alleviates data scarcity and inefficient tokenization by lever- aging traditional end-to-end driving model to produce condensed and semanti- cally enriched representations of the scene, which optimized for LLM planning compatibility through deliberate representation and reasoning alignment training stages. Our results demonstrate that TOKEN excels in grounding, reasoning, and planning capabilities, outperforming existing frameworks with a 27% reduction in trajectory I.2 error and 39% decrease in collision rates in long-tail scenar- ios. Additionally, our work highlights the importance of representation alignment and structured reasoning in sparking the common-sense reasoning capabilities of MM-LLMs for effective planning. More details the project website.
Keywords: Multi-modal LLM,, Autonomous Driving, Representation Alignment
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
The autonomous driving industry is increasingly pursuing end-to-end learning from sensory inputs to reduce human inductive bias in system design [1, 2]. Despite the remarkable progress, end-to-end models inherently suffer from severe performance degradation in long-tail scenarios. For example, state-of-the-ar end-to-end autonomous driving planners often fail to navigate temporary construc- tion sites and react too aggressively to jaywalkers; even simple rule-based planners can significantly outperform high- cagh-capacity end-to-ene models in these long-tail scenarios [3]. This motivates recent efforts to fine-tune Large Language Models (LLMs) into autonomous vehicle planners [4, 5, 6], aim- ing to leverage the benefits of both high-capacity models and the common-senee reasoning abilities that emerge from world- knoll-enowwedge training.
LLM-based planners, in their simplest form, depend on textual scene descriptions prompts, mak- ing their performance highly reliant on the quality and detail of these descriptions. Detailed prompts require extensive engineering and generate many tokens for the LLM to process. Conversely, our evaluations show that simple, heuristic prompts do not tap into the common-sense reasoning abil- ities of LLMs due to insufficient scene understanding. As a result, Multi-Modal Large Language Models (MM-LLMs),, which naturally integrate various data modalities beyond text, are emerging as promising foundations for developing autonomy stacks in autonomous vehicles.
The predominant approach is to leverage pre-trained encoders (typically pre-trained using visual- text alignment) to extract features from the sensory inputs, followed by a querying transformer that uses latent queries to tokenize the features into dense latent tokens and feed them to the LLMs [7, 8, 9, 10, 11]. Training an effective scene tokenizer (encoder and querying transformer) often re- quires a significant amount of question-answer pairs (QAs) (for example, Flamingo used more than one billion QAs reach satisfactory performance [12]). However, current MM-LLM datasets for autonomous driving typically contain fewer than one million QAs [7, 13]. Consequently, without careful model and training scheme designs, these models often exhibit poor performance in rea- soning and planning tasks due to a lack of scene understanding and grounding capability. The key challenge is to enable the scene tokenizer to extract informative and structured information that can unlock the common-senss reasoning ability of the LLM in a low-data regime.
We propose TOKEN (Fig. 1), novel MM-LLM framework that utilizes object-centric tokenization to tokenize the world into few object-level tokens to enhance the planning ability of autonomous vehicles, especially in long-tail scenarios. Our key insight is that object-level latent tokens, with each token representing a relevant object in the scene, are much more informative and easier for the LLM to interpret compared to unstructured dense tokens. TOKEN not only produces a condensed and semantically informed representation of the scene but also enables us to use a state- of-the-art end-to-end driving model as the pre-trained scene tokenizer, effectively alleviating both the data scareity and inefficient tokenization challenges present in current MM-LLM frameworks.
In Sec. 5.1, we compare TOKEN to alternative MM-LLM frameworks and demonstrate its supe- rior grounding, reasoning, and planning capabilities in a low-data regime. In Sec. 5.2, we compare TOKEN to the state-of-the end-to-end (SOTA) autonomous driving planner [2] and showcase its strong performance long-tail scenarios, including navigating around construction sites, executing 3-point turns, resuming motion after full stop, and overtaking parked cars through the oncoming lane. In Sec. 5.3, compare TOKEN to the SOTA LLM- based planner [5] and demonstrate its su- periority in long-tail scenarios. We further conduct an ablation study to highlight the importance of proper representation alignment and structured reasoning process alignment in effectively evoking the common-sense reasoning ability of the LLM backbone for planning. To the best of our knowl- edge, we are the first to conduct an in-depth analysis to demonstrate the promising potential and necessity of such alignment in effectively leveraging MM-LLM mitigate long-tail challenges.
2 Related Work
End-to-End Driving & Long-tail Event Mitigation. Traditional end-to-end autonomous driving models inherently suffer from performance degradation in long-tat tail scenarios [1, 2]. To mitigate this issue, previous works focused on detecting such situations online [14, 15, 16, 17] and switching

---

the planner to a model--ased planner [18, 19] to ensure safety, In contrast, we leverage pre- trained end-to-end driving model to tokenize the scene and share features with a LLM,, leveraging its common-sense reasoning ability to enhance planning performance in long-tail scenarios.
LLM-Based Motion Planning. In light of LLMs' outstanding understanding and generalization abilities, previous works have attempted fine-tune LLMs into motion planners [4, 5, 6, 3]. How- ever, LLM-based planners are highly dependent on the quality and resolution of the scene descrip- tions. While a more comprehensive and fine-grained description can help the LLM better parse and understand the scene, it also makes the L.M inefficient to train and run inference. Additionally, designing templates to textualize scenes requires extensive prompt engineering. To alleviate the model from text-based scene description, recent works exploits object-centric vectors using parsed symbolic information [20]. However, relying parsed symbolic information making the system less robust inductive bias and vulnerable salient information from the scene context [21]. In contrast, we directly uses raw sensory inputs to enhance the LLM's scene understanding ability.
Multi-Modal Large Language Models. MM-LLMs have been increasingly integrated into the au- tonomous driving stack (c.g., 3D detection [22], driving co-pilot [9]. driving scene summarization [23].) ). Previous works typically fine-tune existing MM-LLMs (e.g., [24, 25, 26]) using driving- related QAs The existing MM-LLMS are mostly optimized for visual understanding. Consequently, autonomous driving MS fine-tuned using these models [7, 8, 9, 10, 11] often lack the ground- ing, 3D understanding, and behavior reasoning abilities. To mitigate this issue, recent works attempt to integrate detection head into the querying transformer (the latent queries used for token extrac- tion additionally interact with the detection queries to guide the tokens to capture 3D information) [27] or leverage BEV encoder pre-trained in driving tasks to extract features for the querying transformer [28]. However, the tokens in both works remain unstructured and entangled (i.e., one object's ş information could be distributed across multiple tokens), leading ineffective tokenization. Our work differs from existing ones in two key ways: first, we object-centric tokenization and pre-trained end-to-end driving model to tokenize the scene into structured and disentangled tokens; second, we analyze the importance of representation alignment and structured reasoning process alignment to effectively evoke the common-senss reasoning ability of the MM-LLM for planning. More related work on object-centric representation in robotics are discussed in App. A
3 TOKEN Framework
We propose novel MM-LLM framework, TOKEN, tailored for autonomous driving. It consists of three modules: scene tokenizer that tokenizes the sensory inputs into object-level tokens, an adapter that aligns the object token's 's embedding space with the text embedding space, and an LLM.
Object-Centric Scene Tokenization. Existing MM-LLMs' scene tokenizers typically leverage pre- trained vision encoders (e-g., CLIP [7, 8, 9, 10, 11)) to extract features from the sensory inputs, followed by a querying transformer [25] that uses learnable queries tokenize the features into dense latent tokens and feed them to the LLM. Training an effective scene tokenizer poses sig- nificant challenge. On one hand, the scale of existing question-answent autonomous driving is far from enough. On the other hand, relying on latent queries to extract to- kens in a low-data regime often results in unstructured and redundant tokens that are difficult for the LLM to interpret and process efficiently, which leads to poor performance in reasoning and planning tasks due to a lack of scene understanding and grounding capability (Sec. 5.1).
MM-LLM datasets for
Driving is highly structured and object-oriented task the ego vehicle's behavior is largely con- strained by the traffic agents and map elements around it. Instead of training the driving scene tokenizer from scratch through the driving scene, we obtain object-centric tokens from existing end-to-end autonomous driving stacks trained on tasks such as detection, tracking and segmentation and are thus already optimized to encode rich spatial, temporal, and semantic information directly associated with relevant objects in the scene. This approach allows us to leverage state-of-the-art multi-modal end-to-end driving models as our scene tokenizer, effectively addressing both the data scarcity and inefficient tokeniza- tion challenges present in current MM-LLM frameworks. We illustrate our general framework in
answering tasks and using unstructured tokens to encode
3

---

Extract tokens using pre-traimed object-eetti querying transformers
<Multi-view video, HD-map, Symbulic representation>
Mapping
tipa! infermation, piesse...?
Object-cmnriic Tokenization
Transformer- driving model
Detection
Tokenization
Prediction
Adapter
draving Italaing Male catrectiny
Large Language Model (LLaMA)
LoRA d
Figure 2: TOKEN obtains object-centric tokens from existing end-to-end autonomous driving stacks and uses a condensed and semantically innfore representation to encode the scene.
Fig. 2 and introduce the specific design choices our scene tokenizer in Sec. 4.1. In addition to the object-level tokens, unstructured scene-level latent tokens learned from scratch can be optionally included compensate for missing information, such as weather conditions.
Adapter. While the object-level tokens generated by our scene tokenizer already encode rich in- formation, it is crucial to align the latent token embedding space with the text embedding space in order for the LLM to understand and extract information. We perform a wide range of QA tasks to train the adapter to align the tokens, paving the road for the subsequent behavior planning task.
Experimental Setup
4.1 Design Choice the Scene- Tokenizer
In this work, leverage the transformer-based end-to-end driving model, PARA-Drive [2], as our scene tokenizer to extract object-level tokens. PARA-Dris Drive is parallelized modular autonomous vehicle stack that encompasses a diverse set of modules for the co-training of bird's-eye view (BEV) features from multi-view video input. Each module is a querying transformer that uses latent queries to attend to the BEV features and decode the corresponding task output. We pre- train PARA-Drive
Track quopie
Muep quetring
BEY fustons
object-centric and scene-centric tasks, including mapping, object tracking, occupancy prediction, and motion predic- tion, as shown in Fig. 3. In object-centric tasks, each query produces a latent token that encodes information about a specific scene object.
Cocspency querving
Zeedickes quariing taauliirer
G
Crie atardon
Figure 3; End-to-Ene driv- ing model (PARA-Drive) as the scene-tokenizer.
Specifically. the track query Qinck is trained produce a token Ftrack RIx256 that encodes object i's 3D bounding box and mantic category, the motion query O is trained to produce token z E R2x256 that encodes object i's potential dynamic behavior, and the map query Dimpp is trained to produce token *mup E IR. Ix266 that encodes the map element j's geometry and semantics (e.g., crossing area) information. We concatenate the track token and motion token to constitute non-map object tokens: Pigent = Trrck + motionn and directly use the map token æhup as the map element token. Although we don't use latent tokens from the occupancy prediction module, we still include this module during training to fairly compare with PARA-Drive. We follow the same training procedure in [2] to train scene tokenizer. non-map element tokens share one MLP adapter and map element tokens share another one.
4.2 Dataset Construction
To train and evaluate TOKEN, we construct a dataset based on the NuScenes dataset [29], including visual question-answering (QA) pairs that span the full stack of autonomous driving development. We illustrate a few examples of these QAs in Fig. 1; more can be found App. B.
Perception. We build perception QAs based on the DriveLM dataset [7], covering object semantics and dynamic behavior identification. Additionally, create object-lane association QAs to enhance the model's understanding of map elements and objects' topological relationships to the ego vehicle.

---

the past four front view Behavior Reasoning, Behavior reasoning QAs include two types of questions: 1) object level behavior analysis questions where we ask the model to reason about whether an object is critical (i.e., is likely to influence the ego vehicle's planning) and provide the corresponding reason; 2) scene-level critical object grounding questions where we ask the model to predict the locations of critical objects in the ego vehicle's 's local frame. Behavior reasoning QAs aim to enable the model to understand context-dependent critical objects, thereby connecting the object tokens with the LLM to simplify the scene for downstream planning.
Route-conditioned Hierarchical Planning. Different from previous works that use the relative position of the ground-truth ego trajectory to define high-level commands ("keep forward" and "turn right/right",, we re-labeled the NuScenes dataset to use road-level navigation signals as high-level commands, including: "keep forward along the current road," "prepare turn right/left at the next intersection," " "turn right/left at the intersection," "left/right U-turn," "" and "left/right 3- 3-point turn." We utilize chain-of-thoo -fthoughh reasoning to align the model's ' planning process and guide it to progressively generate the driving plans in three steps. First, the model identifies the critical objects in the current driving scene, including their categories and 2D) locations in the ego frame. Next, it proposes the desired behavior mode, detailing interaction plans with the critical objects (e.g., overtake) and lane- level decisions (e.g.. left lane change). Finally, it generates 3-second motion plan (6 waypoints).
4.3 Training Strategy
We use LLaMA-2-7B [30] our LLM backbone. We keep the scene tokenizer frozen and use a Low-Rank Adaptation (LoRA) module [31] to fine-tune the LLM. We note that the tokenizer is trained on the NuScenes dataset, ensuring that both the tokenizer pre-training and the MM-LLM training share the same source driving training data. We train TOKEN in three stages: pre-training, reasoning fine-tuning, and planning fine- tonettuning..... More details are in App. C.
Experimental Results
5.1 On the Value of Object-Cennie Centric Tokenization
Experiment Design. We compare TOKEN against existing MM-LLMs to demonstrate its effec- tiveness and efficiency in scene understanding, grounding, and planning. Our experimental setup focuses on the impact of 1) different tokenization schemes and 2) pre-training on driving data.
Baseline. We compare TOKEN against (1) VILA-1.5 [32], which uses trainable VIT to tokenize frames, (2) Video-LLaMA [26]. which leverages a pre-trained and frozen ViT to extract visual features and a querying transformer to produce visual tokens from the extracted features, and (3) BEV-TOKEN, a variant of TOKEN that directly uses dense bird's-eye view (BEV) features from the same pre-trained PARA-Drive instead of sparse object-level tokens, similar to most MM-LLMS (e.g., -LMMo. We follow the same training recipe for all models and use the same LLaMA- 2-7B the LLM backbone. For Video-LLaMA and VILA-1.5, frames to render a 2-second video input for each QA.
Metrics. We use classification accuracy to measure the model's ability to 1) classify the object at a location in the ego frame, 2) identify the lane in which the object is located, and 3) answer other categorical questions in the DriveLM dataset [7]. To evaluate the model's 's ability to localize and reason about critical objects, we use precision and recall to measure its grounding ability (we use Hungarian matching to match the predictions with the ground truth), and use accuracy measure its ability to identify whether an object is critical given the object's 's center location in the ego frame. We consider three variants of trajectory L2 error to evaluate the predicted motions from different perspectives in various scenarios: the overall, turning, and progress errors, which are calculated from the original L2 distance, heading difference, and cennttudinal-eeghhee L2 distance between the prediction and GT We use the average collision rate over the entire horizon to measure the safety of a motion plan. More details about our evaluation protocol can be found in App. D.
We report the errors at 1s. 2s. and 3s. the averaged error over these time steps (denoted as Ave and the averaged error over the entire horizon (denoted as Ave
5

---

Scone understanding
Critical object grounding
Collision (8)
Method
Obj. class.
object usso, acc.
Acc.
Precision
Recall
Import
Averus
Avtu
Aveu1
Video-LLaMA
0.28
0.38
0.22
0.58
0.27
1.72
3,81
2.39
VILA-1,
0.37
0.22
042
0.19
0.16
0.55
0.28
1.56
4.41
2,09
1.66
1.98
BEV-TOKEN
0.68
0.64
0.61
0.58
0.62
0.76
0.39
1.01
2.02
1.14
0.96
0.39
TOKEN
0.92
0.68
0.76
0.87
0,76
0.92
0.26
0.71
1.47
0,81
0.68
0.15
PARA-Drive
NA
NA
NA
NA
NA
NA
0.23
0.68
1:50
0.80
0.66
0.19
Table I: Quantitative evaluation of the scene understanding, critical object grounding, and
planning tasks. TOKEN significantly outperforms baseline VLMs due to its use of driving-task
pre-trained features and object-centric tokenization. Bold numbers denote the best results in each
column, and the numbers shaded in green indicate significant improvements. We also show the PARA-Drive's 's planning performance as reference (shaded in grey).
(m) 1
Heading 12 (rad)..
Loc, weighted
12
Uoltroon
Long-tiil
Mathod
1<
2+
34
1s
34
Ave.
Antil
point tnn
PARA-Dme
0.50
1.38
2.70
1.55
1.2M
0.40
0.83
1.25
1.45
11900
0.75
2.30
4.19
2.56
1.96
5,35
TOKEN
0.39
1.29
2,60
1.43
1.18
0.21
0.35
0.71
0.42
0.36
0.68
2.15
4.35
2.30
1.98
4,00
TOKEN
0,20
0.73
1.77
0.90
0.73
0.35
0.41
0.37
0.35
0.33
0.37
1.28
2,93
1.53
1.24
0.00
Recaue motion
PARA-Drise
0.79
1.30
0.35
0.11
0.35
1.97
Goo full cop
TOKEN
0.70
1.58
0.65
0.09
0.24
0.19
2.66
1.38
Ovenakke
PARA-Drise TUKEN
1.94
0.35
0.05
0.11
0.10
0.74
0.04
0.09
1.58
Conoruction run:
PARA-Driee
0.38
0.95
1.65
0.99
0.34
0.05
0.08
0.11
0.08
0.07
0.69
1.61
2.84
1.71
1.47
6.70
TUKEN
0.36
0.63
1,38
0.73
0.60
0.03
0.04
0.86
0.04
0.04
0.47
1,03
2,08
1.19
1.09
2,32
Table 2: TOKEN significantly outperforms PARA-Drive in long-tail scenarios.
Results. We present the quantitative evaluation of each model's scene understanding, object ground- ing, and planning abilities in Tab. 1 (only traj. 1.2 results are shown; full planning results are in App. E). Video-LLaMA and VILA-1.5 perform noticeably worse across all metrics, particularly in critical object grounding. This is because their encoders are optimized for visual-text alignment, and the limited driving QAs do not sufficiently develop their 3D understanding ability. Notably, Video-LLaMA and BEV-TOKEN share the same tokenization scheme and only differ in visual fea- tures, highlighting the value of visual features pre-trained with driving-related tasks. The benefits of pre-training features using embodied-relatee tasks are also observed in MM-LLM for robot manipu- lations [33]. Interestingly, TOKEN significantly outperforms BEV-TOKEN in all tasks, even though they tokenize the same visual features. This demonstrates the benefits of object-centric tokenization.
Takeaway 1: Object-centric tokenization with pre-trained driving-aware features enables better grounding, reasoning, and planning performance in low-data regimes.
5.2 Generalization in Long-tail Driving Scenarios
We evaluate the planning performance of TOKEN against PARA-Drive in long-tail events. TOKEN uses the PARA-Drive as the scene-tokemizer, thus they share the same visual encoder and scene information, and only differ in the planner structure.
Long-tail Events Construction. We manually inspected the NuScenes dataset and identified the following long-tail scenarios for evaluation, each representing less than 1% of the training data: 1) executing 3-point turns; 2) resuming motion after a full stop; 3) overtaking parked cars through the oncoming lane; and 4) navigating around construction sites. More details can be found in App.
Quantitative Result. In Tab. 2, we summarize the quantitative evaluation for each long-tail scenario. TOKEN significantly outperforms PARA-Drive in terms of the quality and safety of predicted plans. Specifically, TOKEN predicts more accurate turning maneuvers in 3-point turns (60% reduction in heading L2 Aveau). more effective motions after yielding (28% reduction in longitudinal weighted 1.2 Avean). and safer plans when navigating around blocking vehicles and construction sites (100% and 67% collision rate reductions in the overtake and construction zone scenarios, respectively).
Qualitative Examples. In our qualitative analysis, TOKEN consistently outperforms PARA-Drive During 3-point turn (Fig. 4), TOKEN accurately generates the turming maneuver while PARA- Drive struggles. In Fig. 5, after yielding to pedestrians, TOKEN predicts a forward motion, whereas PARA-Drive remains stationary. In the opposite direction overtake scenario (Fig. 6). TOKEN pre- diets motions that avoid collisions and attempt to return to its lane post-overtake, unlike PARA- Drive, which predicts unsafe motions. Finally, TOKEN generates plans that effectively steer around the construction zone, while PARA-Drive predicts motions that result in collisions or deviations from the lane (Fig. 7). More detailed analysis can be found in App. G.

---

1-2s
1-68
[-8s
100s
P'RR--Drive
TOKEN TOKEN" GT
Prompt: The autonomous vehicle needs to make left 3-point turn. What objects are important for the autonomous vehicle's planning? What are these objects, their (xy) locations? how should the autonomous vehicle interact with them? Please plan autonomous vehicle's future trajectory using waypoints, every seconds.
TOKEN:
There is no important object for the autonomous vehicle's planning The autonomous vehicle should make sharp left turn, and its 3s future trajectory is... [o...tomitte]]]
Figure 4: Planning visualization in the 3-point U-turn scenario (zero-shot performance). TOKEN* denotes a model variant trained with additional synthetic data augmentation (few-shot performance).
PARA-Drive
TOKEN
0.0
0.5
1.0
1:5
2.5
3.0
Time [s]
The autonomous vehicle needs tutn left. What objects are important the autonomous vehicle's plarxing?? What are these objects, their (x. y Jocalions, and how should the autonomous vehidle interact with them? Please (omitted
Promut:
TOKEN:
There are important objects: pedestrian 5.44,
5,4), bus at (-4-, 16.3) It should continue drive, accelerate and alightly left. [ommite]
Figure 5: Planning performance visualization. The red dots in the left plot represent the identified critical objects. The middle plot visualizes the predicted motions. The right plot shows the speed plan inferred from the predicted motions. Note that TOKEN enables the ego vehicle resume motion after a full stop.
Since 3-point turn is a zero-shot generalization scenario, we evaluate TOKEN's few-shot learning ability by generating synthetic 3-point turn motions and augmenting the training dataset with 40 samples (0.12% of the training data). The resulting model TOKEN' predicted motions (dark gray lines in Fig. 4) show qualitative improvement and better alignment with a 3-point turn. This example demonstrates the data efficiency of adapting LLM-bbsed planners to new concepts.
Takeaway 2: By leveraging the LLM's common-sense reasoning capabilitiess TOKEN predicts more accurate plans in long-tail scenarios in which traditional end-to-end planning methods struggle.
5.3 On the Value of Alignment
We compare TOKEN with the SOTA LLM-based planner Agent-Driver [5]. Agent-Driver queries text-based scene information using various tools and then fine-tunes GPT-3.5 into motion planner. Our evaluation (Tab. 3) shows that TOKEN and Agent- Deiverrriver have similar overall performance, but TOKEN significantly outperforms agent-driver in long-tail scenarios with much smaller model and less privileged information (details in App. H). We hypothesize that evoking common-sene reason- ing in M-pooweed planner requires proper alignment rather than just larger L.M backbone. Motivated by this finding, we further conduct detailed ablation study to shed more light on this.
Tra L. (m))
Heading 1.2 (rad) 1
Lon, weightal Inj 1.2 (m))
Callisian (96
Split
Methed
Is
2x
3s
Averys.
Ave.
1s
2x
36
Avtyys:
Avea
1s
2x
Averas.
Ave.
Aveal
Vil
Agent-drivar
0.23
0.69
1.52
6.11
0.67
0.03
B.06
0.11
0.05
0106
0.43
1.27
2.88
1.49
1.23
6.13
TOKEN
0.26
0.71
1.47
081
0.68
0.02
0.64
0.86
0.04
0.83
0.50
1.32
273
1,52
1.27
0.15
Long-tail 4bil
Agent-Driver
0.38
1.31
293
1.54
1.36
0.07
0.21
0.35
0.1
0.38
0.64
2.36
4.78
2.56
2.11
2.67
TOKEN
0.26
0.81
1.77
0.85
0.78
0.05
0.00
0.18
0.11
0.89
0.50
1.47
309
1.69
1.40
0.35
Table 3: Quantitative comparison with an LLM-based planner Agent-Driver.
Alignnesi
(m))
Heading (rad))
Lon.
Colliina
Ave
V51
0.54
0.85
1.67
[196
0.50
0.05
0.08
0108
0.07
0.08
0.66
1.n2
3.14
1.81
1.52
0.22
0.28
0.75
1.55
0.86
0.72
0.15
0.155
01135
0105
0.04
0.5
1.33
2.73
1.51
1.26
1119
0.76
0.07
8.02
0.04
0.06
0.04
1.27
0.18
1.74
0.30
0.99
3.09
1.13
0.94
0.11
0.17
0.21
0.16
0.15
0.60
1.78
3.69
2/02
1.68
0.52
0.56
1.10
2.26
124
1.05
0.14
0.19
022
0.15
0.17
0.64
1.93
398
2.1%
1.82
0.47
0.26
831
1.77
1198
0.78
808
818
0118
611
0.89
0.90
1.07
300
1.69
1.40
1.38
Table 4: Value of alignment. Both the pre-training representation alignment and structured reason-
ing alignment are necessary to leverage the LLM's common sense reasoning ability.
7

---

PARA-Drive
TOKEN
6|-]
The autonomous vehucle needs to keep lorward along the road. What objects are important for the autonemous vehicles planning? What are these objects, their locations, and how should the autonomoos vehicle internct with them? Pleose [omiited)
Frompt:
TOKEN;
There Is
The autonomous vahidl should proceed with caution and prepare evertake the at (-19, 8.3) vehicle's left shauld the adjacent right lane
object
driving decelerate and slightly right,... [ommittd]
Figure 6: Planning performance visualization. The right plot visualizes the predicted motion plans two consecutive time steps. TOKEN enables the ego vehicle to decelerate when approaching the obstacle and drive back to the original lane after overtaking.
PARA-Drive
Prompt autonomous vehicle needs keep forward along road, objects are important the autonomous vehicle's planning? these objects, their (s. y) locations, and how should the autoromoes vehicle internct with them? Please plan.. Jomitted]
TOKEN:
There are important objects: traffic come at (22, 13
at 11.7;, traffic cone (1.7, 1688, traffic cone at (16, 1988) These
cones are blocking the ego vehicle's lane. The autonomous vehide should bypass the traffic cone at (22, 13:4) from autoromous vehide's right side, bypass the traffic cone at (2.1, 11.7) from the autonamous vehicless right side, bypass the traific come at (17, 16.3) from the autonoenous vehicle's right side, bypass the traffis
(1., 19.8) from the autenomous vehicle's right side [omitted]
Figure 7: Planning performance visualization. The red rectangle denotes the identified critical traffic cones. The right plot visualizes the predicted motions at two constitutive time steps. TOKEN instructs the ego vehicle steer and bypass the traffic cones from the ego vehicle's right side.
Ablation Study. We test three variants of TOKEN to investigate the value of alignment: 1) no rep- resentation alignment nor reasoning alignment treating TOKEN as a traditional end-to-end model and training to directly predict the ego vehicle's motion plan: 2) with representation alignment but no reasoning alignment; 3) without representation alignment but with reasoning alignment.
Results. We summarize the quantitative evaluation results over the evaluation set and all long-tail scenarios Tab. 4. Our evaluation shows that without the pre-training stage, the performance vari- ation is not significant in both the validation and long-tail splits. This indicates the necessity of aligning the sensory token's embedding space with the text embedding space. Interestingly, with representation aligmment pre-training, additional reasoning process alignment does not significantly benefit the entire validation set but significantly improves the accuracy and safety of predicted mo- tion plans in the long-tail split. This suggests that proper reasoning process alignment is essential to unlock the common- sense reasoning ability encoded in the LLM backbone. Qualitative comparisons can be found in the App. I.
Takeaway 3: Representation and structured reasoning process alignments are important for evoking the common-sense reasoning ability of the I.IM backbone for planning.
We include additional results in App. to demonstrate the performance improvement brought by the additional HD-map modality, the impact of interaction mode (as opposed to simple behavior mode) in the reasoning alignment, and further highlight the few-shot learning ability of TOKEN.
Limitation & Future Work
A strength and limitation of TOKEN is using a pre-trained and frozen end-to-end driving model as the scene tokenizer. This allows for extracting informative tokens and controlled experiments. However, TOKEN's 'ss performance tightly coupled with the quality of the pretrained tokenizer. We show failure case in App. K in which the critical object is not detected by the tracking querying transformer in PARA-Drive Consequently, TOKEN is unable to understand the scene and generates incorrect behavior. Further work will focus on co-training PARA-Drive to leverage the knowledge within the LLM to improve the scene tokenizer. Another limitation is outputting the trajectory plan as text, with each digit as a separate token. This increases computational expense and complicates motion generation, leading to misalignment between predicted behavior and the motion plan (e.g.. TOKEN predicts the correct behavior but incorrect motion in Fig. 6). Future work will focus on quantizing motions into discrete I.LM-understandable tokens [34] using dedicated trajectory decoder [35]. More discussions about inference speed and closed-loop evaluations are in App. L.
8

---

References
[1] Y. Hu, J. Yang, L.. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, et al. Planning-orienterr autonomous driving. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Patten Recognition, pages 17853-17862, 2023.
[2] X. Weng, B. Ivanovic, Y. Wang, Y. Wang, and M. Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[3] M. Hallgarten, J. Zapata, M. Stoll, K. Renz, and A. Zell. Can vehicle motion planning gener- alize to realistic long-tail scenarios? arXiv preprint arXiv:2404.07555 2024.
[4] J. Mao, Y. Qian, H. Zhao, and Y. Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415 2023.
[5] J. Mao, J. Ye, Y. Qian, M. Pavone, and Y. Wang. A language agent for autonomous driving. arXiv preprint arXiv:2311.10811 2023.
[6] B. Li, Y. Wang, J. Mao, B. Ivanovic, S. Veer, K. Leung, and M. Pavone. Driving everywhere with large language model policy adaptation. 2024.
[7] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, P. Luo, A. Geiger, and H. Li. Drivelm: Driving with graph visual question answering. arXiv preprint urXiv:2312.141500 2023.
[8] W. Wang. Xie, C. Hu, Zou, Fan, W. Tong, Wen, S. Wu, H. Deng, Z. Li, et al. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245, 2023.
[9] Y. Ma, Y. Cao, J. Sun, M. Pavone, and C. Xiao. Dolphins: Multimodal language model for driving, arXiv preprint arXiv:2312.00433 2023.
[10] X. Ding, J. Han, H. Xu, X. Liang W. Zhang, and X. Li. Holistic autonomous driv- ing understanding by bird's seeee-ieww injected multi-modal large models. arXiv preprint arXiv:2401.00988 2024.
[11] X. Tian, J. Gu, B. Li, Y. Liu, C. Hu, Y. Wang, Zhan, P. Jia, X. Lang, and Zhao. Drivevim: The convergence of autonomous driving and large vision-language models. arXiv preprint arXhv:2402.12289, 2024.
[12] J.-B. Alayrac J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Mil- lican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 11-23716-237366 2022.
[13] Qian, Chen, L. Zhuo Jiao, and Y.-G. Jiang. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 4552-4550, 2024.
[14] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[15] Q. M. Rahman, P. Corke, and F. Dayoub. Run-time monitoring of machine learning for robotic perception: A survey of emerging trends. IEEE Access, 9:20067-20075. 2021.
[16] R. Sinha, A. Sharma, S. Banerjee, T. Lew, Luo, M. Richards, Sun, E. Schmerling, and M. Pavone. A system-level view out-of-distribution data in robotics. arXiv preprint arXiv:2212.14020, 2022.

---

[17] A. Elhafsi, R. Sinha, C. Agia, E. Schmerling, I. A. Nesnas, and M. Pavone. Semantic anomaly detection with large language models. Autonomous Robots,
2023.
[18] L. Sun, X. Jia, and A. D. Dragan, On complementing end-to-end human behavior predictors with planning. arXiv preprint arXiv:2103.05661, 2021.
[19] R. Tian, L. Sun, A. Bajesy, M. Tomizuka, and A. D. Dragan. Safety assurances for human- robot interaction via confidence-awan game-theoretic human models. 2022 International Conference Robotics and Automation (ICRA), pages 1229-11223, IEEE, 2022.
[20] L. Chen, O. Sinavski, Hünermann, A. Karnsund, A. Willmott, D. Birch, D. Maund, and J. Shotton. Driving with Ilms: Fusing object-level vector modality for explainable autonomous driving. arXiv preprint arXiv:2310.01993 2023.
[21] N. Mu, J. Ji, Z. Yang, N. Harada, H. Tang, K. Chen, C. R. Qi. R. Ge, K. Goel, Z. Yang. et al. Most: Multitmodality scene tokenization for motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2024.
[22] J. H. Cho, B. Ivanovic, Y. Cao, E. Schmerling, Y. Wang. X. Weng. B. Li, You, P. Krähen- bühl Y. Wang, et al. Language-image models with understanding. arXiv preprint arXiv:2455.03685, 2024.
[23] B. Li, L. Zhu, R. Tian, S. Tan, Y. Chen, Y. Lu, Cui, S. Veer, M. Ehrlich, J. Philion, al. Wolf: Captioning everything with world summarization framework. arXiv preprint arXiv:240.11800, 2024.
[24] J. Bai, S. Bai, S. Yang, S. Wang. S. Tan, P. Wang. J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023.
[25] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[26] H. Zhang, Li, and L. Bing. Video-llama: An instruction-tunel audio-visual language model for video understanding. arXiv preprint arXiv:2306.02885. 2023.
[27] Wang, Z. Yu, Jiang, S. Lan, M. Shi, N. Chang, Kautz, Y. Li, and M. Alvarez. Om- nidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533. 2024.
[28] H. Shao, Y. Hu, L. Wang, G. Song, S. L. Waslander, Y. Liu, and H. Li. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[29] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[30] H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, N. Bashlykov, Batra, Bhargava, Bhosale, al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[31] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu. Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.08 09685, 2021.
10

---

[32] J. Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07333, 2023.
[33] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
[34] A. Seff, B. Cera, D. Chen, M. Ng. A. Zhou, N. Nayakanti, K. S. Refaat, R. AI-Rfou, and B. Sapp. Motionlm: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8579-8590, 2023.
[35] Y. Chen, Tonkens, and M. Pavone. Categorical traffic transformer: Interpretable and diverse behavior prediction with tokenized latent. arXiv preprint arXiv:2311.18307, 2023.
[36] A. Brohan, N. Brown, Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, Finn, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023.
[37] M. S. Sajjadi, D. Duckworth, A. Mahendran, S. Van Steenkiste, F. Pavetic, M. Lucic, L. J. Guibas, K. Greff, and T. Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512-9524, 2022.

---

Supplementary Materials
A Extended Related Work
Object-Centric Representation. Existing MM-LLMS often utilize Vision Transformers (ViTs) to tokenize visual inputs into latent tokens. These tokens, resembling static grid over the inputs, are unstructured and pose challenges for LLMs to interpret in embodied reasoning tasks [33]. To address this, prior work [20] employs vectors of symbolic representations generated by a state estimation module to encode scene information, which are then used as tokens for the L.M. Our work closely related [33] and [36], which use object scene representation transformers [37] extract object-level latent tokens. However, unlike [33] and [36], our paper focuses on the autonomous driving domain and an end-to-end driving model to tokenize the scene.
B Dataset Construction
B.1 Examples of the QAs
In Fig. 8, we illustrate examples of QAs used in training TOKEN.
Scene Understanding
Reasoning
Hierarchical Planning
Q Whet lene is the refiide (1.8 2233
Shoil oebri
slong
drbiie aisiiie die sutimanng noiile
(-32,
There & iuporiant objecss oar at (86, 3.91, The misormmu whitle aloud girld to the car 966, 39. should rrmadt ruubly statie, und.
A: Wahick at (21., 1211 is merging inte ego
onlerr'
autonsmoas vhlide's
Q: What the otserzed status of the dijas at
O: Sunid antormmns whitle pay stimttee
O: The sutonmmas volidd made no kuep flrnard sting the neal (nmittel)
63.8, 6.01 Al Settweryy
to the obser bocied at (3., & 217 Whyt A: Ths dbjict b painerias with velccit 6-16, 0.21 nts This, it is crossing in fount of the auhmomoous whitls. The ammaer be Ye..
A: Thoe me isppriast elycci trofiic coe /2. 13,4 taffle ooe at (2.1, (1.7. 1686, troffi
C: Wilee dhe tppe the wljert at 68886
11.71, toffie
(1.6 B5A) The tryffi Marking the ariiie's lawe The antommn coisii dusid bypass the
A: Pedestrion
taffle ooe 12... 1340 men the mitonessss retice''s right ade
Figure 8: Examples of perception, reasoning, and planning QAs.
B.2 Road-Level Navigation Signal
Previous works on VLM/LLM for autonomous vehicle planning often prompt the model with high-level command based the relative position of the ground-truth ego trajectory, including "keep forward" and "turn right/leftt" However, these high-level commands are not only unrealistic but also simplify the planning problem by removing the need for behavior planning. Therefore, we re-labeled the NuScenes dataset to use road-level navigation signals as high-level commands, including "keep forward along the current road," "prepare to turn right/left at the next intersection," "turn right/left at the intersection," " "leftrightt U-turn," and "left/right 3-point turn."
B.3 Interaction Mode Labeling
We use a combination of heuristics and manual labeling to annotate the interactions between the ego vehicle and the other traffic agents. We first use two types of categorical modes to describe the lane-relationship between a traffic agent and the ego vehicle (agent-ego lane mode) and the relative motion between traffic participant and the ego vehicle (homotopy Agent-ego lane mode at time step t encodes the topology relationship between the ego's 's' current lane and the traffic agent's
12

---

lane, including: LEFT, RIGHT, AHEAD, BEHIND, and NOTON, where NOTON describes that the traffic agent is not on any derivable lanes the scene (e.g., a parked vehicle in a parking lot). To compute the agent-ego lane mode for each traffic agent, we follow [35] to first identify the lane on which each agent is located and then leverage the lane topology map to annotate the agent-ego lane mode. We project the agent's center to the lane polyline and use its relative position in the local Frenet frame to determine its lane association. Homotopies describe the relative motion between pair of agents shown in the video, including: [S, CW, CCWJ (static, clockwise, counterclockwise) Combining agent-ego lane mode, homotopy, agent ground truth state information, and scene con- text information (e.g., cgo is located near an intersection) together, we can leverage heuristics to annotate the interaction. For example, within a 3-second horizon, a static object's agent-ego lane mode changes from AHEAD, to LEFT, to BEHIND, while the ego vehicle performs RIGHT-LANE- CHANGE, KEEP-LANE, then SEFT-LANE-CHANCEE
indicating the ego vehicle overtakes that ob- ject from the ego vehicle's left side. Finally, we use human labelers to verify and correct interaction labels in the following categories: 1) bypass blocking traffic cones to navigate around a construction zone; 2) yield to pedestrians; 3) yield to vehicles; 4) overtake traffic agents via straddling the lane dividers; 5) overtake traffic agents via lane-change.
C Training Details
During pre-training ("representation alignment"), we disable LoRA and only train the adapter to enhance embedding space alignment between the scene and text tokens. We use only perception QAs (150k QAs) to train the adapter for 5 epochs with learning rate of 5e-4. reasoning fine- tuning ("reasoning alignment"), we train the adapter and LoRA together using the reasoning and planning QAs for 10 epochs with a learning rate of le-4. Finally, train the adapter and LoRA together using just the planning QAs for another 10 epochs to maximize the model's performance on planning, maintaining the learning rate at le-4.
D Evaluation Protocol
In this section, we provide a detailed description about our evaluation protocol. In Section 5.1 the main text, we introduce the three variants of trajectory L2 error (the overall, turning, and progress errors) and the collision rate used to evaluate the predicted motion plans. As noted in [2], different evaluation protocols used to compute these metries can lead significant metric variations. We use the same evaluation protocol described in [2] with one exception: exclude samples where any future motion is missing near the end of sequence (the frame masking strategy described in [2]). Including these partially invalid samples would significantly lower the L2 errors, as the L2 errors of these invalid frames are set to zero.
E Additional Result: On the Value of Object-Centrie Tokenization
In Tab. 5, we present the full quantitative evaluation of each model's performance in the planning task. We observe that TOKEN significantly outperforms all baselines across all planning metrics.
Traj 1.2 (m)
Heading L2 (rad) J
weighted traj (m)
Collision (96).
Method
2s
3s
Averzs
Avezi
1s
2x
Aveuzi
Aveal
1s
3s
Aveizi
Aveus
Avoul
Video-LLaMA
0.27
1.72
6.34
3.01
2.39
0.06
0.14
0.20
0.13
0.13
0.56
3.36
9.20
4.36
3.52
2.64
V.L
0.28
1.56
4,41
2.09
1.66
0.05
0.11
0.19
0.12
0.10
0.29
1.92
6.47
2.89
2.24
1.98
TOKEN
0.39
1.01
2,02
1.18
0.95
0.00
0.05
0.06
0.05
0.05
0.75
1.79
3.55
2.03
1.71
0.39
0.36
0.04
2.7
Table Planning performance evaluation. TOKEN significantly outperforms baseline VLMs due to its use of triving-tast pre-trained features and object-centric tokenization.
F Long-tail Events Construction
We manually inspected the NuScenes dataset and identified the following long-tail scenarios for evaluation, each representing less than 1 1% of the training data: 1) executing 3-point turns; 2) resum-
13

---

ing motion after full stop; 3) overtaking parked cars through the oncoming lane; and 4) navigating around construction sites. We note that our design choice of extracting these long-tail driving sce- narios is statistically valid with respect to the accessible dataset. As most open-sourced datasets down-sample the meta dataset and only provide selected driving segments, they are inevitably al- tered and deviate from the actual daily driving distribution. Nevertheless, the above scenarios we selected are still representative and challenge the traditional end-to-end driving
3-point turns, which has one scene (scene-0778, frame 6-30) in the evaluation split and O scenes in the training distribution. We extracted 25 key frames from the scene in which the ego vehicle is performing the 3-point U-turn for evaluation to remove the noise from other nominal behaviors in the scene.
Executing 3-
Resuming motion after a full-stop. which includes 14 scenes in the training split and 8 scenes in the evaluation split. We extract key frames from each scene where the ground truth (GT) motion plan captures the acceleration behavior after the full stop. This results in 70 key frames in the training split (0.28% of the total training samples) and 40 key frames in the evaluation split. The scenes and key frames in the training split are: scene-0208 (frame 25-29), scene- (frame 21-25), scene-0067 (frame 24-28), scene-0159 (frame 4-8), scene-0185 (frame 26-30), scene-0262 (frame 8-12), scene-0862 (frame 18-22), scene-0025 (frame 6-10) scene-0072 (frame 24-28), scene-0157 (frame 12-16), scene-0234 (frame 4-8), scene-0423 (frame 6-10), scene-0192 (frame 14-18), and scene-0657 (frame 12-16). The scenes and key frames in the evaluation split are: scene-0921 (frame 21-25), scene-0925 (frame 19-23), scene-0968 (frame 7-11), scene-0552 (13-17), scene-0911 (frame 24-28), scene-0221 (frame 11-15), scene- scene-1064 (frame 21-25), and scene-0331 (frame 8-12).
Overtaking parked cars through the oncoming lane, which includes 14 scenes in the training split and scenes in the evaluation split. We extracted key frames from each scene where the ground truth motion plan captures the ego vehicle steering into the adjacent lane to overtake blocking object and then returning to its original lane. This results in 248 key frames in the training split (0.9% of the total training samples) and 102 key frames in the evaluation split. The scenes and key frames in the training split are: scene-0001 (frame 12-39), scene-0011 (frame 1-39), scene-0023 (frame 1-8), scene-0034 (frame 23-39), scene-0318 (frame 10-30), 0079 (frame 14-26), scene-0408 (frame 12-30), scene-0411 (frame 4-20), scene-0422 (frame 18-39), scene-0865 (frame 24-39), scene- 11mee-115 (frame 18-30), scene- 1065 (frame 24-35), scene-0200 (frame 20-39), and scene-0752 (frame 10-28). The scenes and key frames in the evaluation split are: scene-0038 (frame 4-33), scene-0271 (frame 3-11), scene-0969 (frame 14-33), scene-0329 (frame 3-33), and scene- 1065 (frame 24-35)
Navigating around construction sites. This common and challenging scenario requires the au- tonomous vehicle to actively change lanes to bypass a construction zone. Although there are two scenes (scene-0980 and scene-0535) in the training split, none exist in the evaluation split. There- fore, we moved one training scene (scene-0980, frames 16-30) to the evaluation split. We selected 15 key frames that capture the ego vehicle decelerating and steering into the adjacent lane to bypass the traffic cones.
G Detailed Qualitative Result
In this section, we provide an in-depth analysis of the qualitative results shown in Sec. 5.2.
Executing a 3-point turn. During 3-point turn, vehicle makes sharp left turn, backs up. and then makes another left turn to complete the maneuver. In Fig. 4, we compare the motion plans from TOKEN and PARA-Drive. Despite receiving a "3-point turn" command, PARA-Drive predicts straight movements at = 2s and t = 4s, likely due to the absence of such examples in its training set. contrast, TOKEN understands the command and generates the correct turning behavior. When approaching the curb to stop and back up, both PARA-Drive and TOKEN predict forward motions at 8s, likely due to the lack of 3-point turn examples in the dataset. At t 10s, when the vehicle has enough clearance, both models predict left-turn motions, with TOKEN's prediction more closely aligning with the ground truth.
14

---

Overtaking parked cars through the oncoming lane. Fig. 6 shows an example of qualitative comparison between the motion plan from TOKEN and PARA-Drive at two constitutive time steps. At t = 58, PARA-Drive predicts a motion that collides with the blocking vehicle, while TOKEN instructs the ego vehicle to decelerate to avoid the object. Interestingly, although TOKEN correctly predicts in language that the ego vehicle should decelerate and steer to the right, the motion plan only reflects the deceleration behavior. We hypothesize that this is caused by insufficient data that helps the LLM associate low-level motion with mid-level behavior. When the ego vehicle straddles the lane divider and prepares to overtake, TOKEN instructs the ego vehicle to drive back to its original lane after overtaking, while PARA-Dris predicts forward motion that straddles the lane divider.
Navigating around construction sites. Fig. 7 shows an example of qualitative comparison between the motion plan from TOKEN and PARA-Drive two constitutive time steps. At / 8s, TOKEN instructs the ego vehicle steer and bypass the traffic cones from the ego vehicle's 's right side, while PARA-Drive predicts motion that collides with the blocking traffic cones At & 12s, TOKEN instructs the ego vehicle to steer to the right to keep forward along the current lane, while PARA- Drive predicts a motion that deviates from the lane center.
H
Comparison with the SOTA LLM-based Planner Agent-Driver
We compare TOKEN with the SOTA LLM-based planner Agent-Driver [5]. They have the following differences:
Scene representation. Agent-Driver queries text-based scene information using various tools (e.g.. object detection, mapping, etc.) and uses the queried text-based information as an input prompt to instruct the LLM to plan the ego vehicle's motion. TOKEN tokenizes the scene into a few object- level tokens. This makes TOKEN more efficient in terms of information density per-token (e.g., TOKEN uses one token encode an object's s semantic, geometry, and dynamic information while the same information is tokenized into around 60 text tokens in Egent-Driver),
LLM-bbckbone. Agent-Driver fine-tunes the entire GPT3.5 while TOKEN uses LORA (with a rank of 64) to fine-tune LLaMA2.
Chain-of- Thouughh Reasoning, Both Agent-Driver and TOKEN utilize chain-of-thought reason- ing to align the model's planning process. However, Agent-Driver uses coarse reasoning process that instructs the L.M to directly generate the ego vehicle's 's discretized steering and acceleration command description based on the scene description, as opposed to TOKEN's more structured rea- soning process that instructs the model to reason about the semantically meaningful effect of the objects (c.g.. blocking the ego vehicle's path) and interaction plans (c.g., overtake).
We use the Drint-Ds predictions from the official repository and show the overall quantitative evaluation in Tab. 3 of the main text. In Tab. 6, we show the detailed quantitative evaluation of each long-tail scenario. In addition, since Agent-Driver includes the ego-state information the prompt, we train a variant TOKEN (denoted as TOKEN+) that also uses the ego-state information as input for planning (current velocity and current acceleration) and show the quantitative evaluation of TOKEN+ in Tab. 6. We that TOKEN and Agent-Driver have similar overall performance, but TOKEN significantly outperforms Agent-Driver in long-tail scenarios. Furthermore, when using ego-state information as input, TOKEN significantly outperforms Agent-Driver in all splits.
I On the Value of Alignment Qualitative Results
In Fig, 9, we show few qualitative comparisons between TOKEN and variant of TOKEN trained with representation alignment but without reasoning alignment. We can see that the predicted motion plans are more aligned the GT motion with reasoning process alignment.
15

---

Tog 1.A. (m)
Heading 1:2
1mn.
waightod
try 1.2 (m))
Collisiu (9)
Spli
Maleed
1n
2+
3x
Antiss
2x
Sc
Avcuss
Assu
3+
Antuas
Avc
Ave.a
val
Agmi-Dhrvve
0:23
0.68
1:50
0:50
0.96
0.79
0.385
0.90
0.85
0.50
0.67
1.26
2.75
1.48
1.21
0.13
TIKEN
0:36
0.70
1.46
0:51
0.68
0.13
0.18
[121
0.17
0.18
0.50
1:30
2.72
1.51
1.26
0.15
TOKE-
0.17
0.52
1.21
01.64
0.52
0.00
0.16
IL19
0.15
0.37
0.33
1.00
2.30
1.21
1,00
0.13
polnt sun
Agcut Deiver
0.38
1.31
2-93
1.54
1.26
0.20
0.74
1.23
0.72
0.63
0.54
2.26
4.78
2.56
2.11
5.67
MIKE
1:70
2,40
1.4%
1.18
0.21
2.15
TOKEN
0.30
0.73
1.77
0.90
0.73
0.87
0.22
0.38
0.26
0.22
0.57
1.28
2.93
1.53
1.24
1.46
Resune motion
2.51
full stop
1.58
2.66
1.13
TOKLN
0,43
1.37
0.59
0.46
0.12
0.78
2,30
0.84
Agent-Diivzz
2000
0.88
1.46
1.45
0.77
TOKEN
1,63
0.90
0.74
0.04
0.07
0.53
1.36
1.31
0.19
TOKEN
1.04
0.83
0.90
Table 6: Quantitative comparison with an LLM-based planner Agent-Driver: TOKEN signifi-
cantly outperforms Agent-Driver in long-tail scenarios. TOKEN+ denotes a variant of TOKEN that uses ego-state input, similar to Agent-
J Additional Results
J.1 TOKEN with HD-map Information
In the main text, we use multi-view video as the sensory input. In this section, we include HD-map as an additional input to evaluate the performance of TOKEN. We utilize the CTT encoder described in [35] to fuse each traffic agent's past state history with each lane's ground truth center line and produce a traffic agent token as an additional token for each traffic object. We use TOKEN + to denote the variant of TOKEN with HD-map information and show its quantitative evaluation in Tab. 7. We can see that the additional map information and the past state history significantly improve the planning performance in both evaluation split and long-tail scenarios.
L2
(m)
Heading (zd)
weighted ual L2 (m) 1
Collision J
Spli
Metiud
3x
Avcuus
Assa
3+
Astists
Avea
Avouss
Anca:
Avcai
TOKEN
0.26
0.82
1.32
1.52
0.15
TOKEN
0.35
0.02
0.86
0.83
0.92
2:11
8J08
Long
0.26
0.81
0.05
0.11
0.09
1.47
TOKEN
0.30
0.71
1.37
0.83
RJ17
0.85
1.18
1.43
0.31
Table 7: Quantitative performance of TOKEN +mup, variant of TOKEN with HD-map information.
J.2
Ablation on the Effect of Structured Reasoning Process Alignment.
One of the unique features of TOKEN is that reasons about semantically meaningful interactions with identified critical objects (e.g., bypassing the blocking traffic cones). We hypothesize that this structured reasoning process supervision enhances the model's planning performance by encour- aging the model to understand the interactions between the ego vehicle and other traffic objects, aligning more closely with how an expert reasons in the real world. To ablate the effect of structured reasoning process alignment, we removed it from the planning QAs' answer labels and used a sim- ilar chain-of-thought reasoning and task planning method as in Agent-Driver (i.e., instructing the LLM to generate the steering and acceleration command description first, followed by the motion plan) to TOKEN (denoted as TOKEN -inteacct). ). We show the evaluation result in Tab. 8. We can see that the structured reasoning process alignment improves the planning performance in both the evaluation set and the long-tail scenarios.
L2
Heading L22 (rad)
Lon, weighted
(m)
Collision (5))
Split
Method
1s
Arta:
Ave.aa
J0ea
3s
Ave cs.
Ave.
Ale.1
TOKEN TOKEN
0.52
1.58
1,31
0.71
0.50
1.52
1.27
Long-tiil
TOKEN TOKEN
1.01
0.09
0.1
0.58
2.00
1.65
0.59
0181
0.65
IL10
6.11
0.50
1.09
1.40
0.36
Table 8: Quantitative performance of TOKEN
inttercct, a variant of TOKEN that uses similar
tfain-of-thought reasoning as Agent-Driver does.
16

---

TOKEN [no reasoning alignment]
TOKEN
GT
Figure 9: Qualitative comparison between TOKEN and variant of TOKEN trained with represen- tation alignment but without reasoning alignment. With reasoning process alignment, TOKEN is able to instruct the ego vehicle resume motion after full-stop (top figure) and safely overtake static obstacles (middle and bottom figures).
J.3 Few-Shot Learning.
To stress test TOKEN's few shot learning ability, we further remove 50% long-tail scenes from the training split and re-train PARA-drive and TOKEN and compare their performance. In Tab. 9, show the quantitative evaluation result in long-tail scenarios. We see that TOKEN only degrades slightly as opposed to PARA-Drive's significant performance degradation. For example, Traj L2 Ave of PARA-Drive degraded by 24% while TOKEN only experiences 9% degradation with 50% long-tail scenes removed. The results indicate the superior few-shot learning ability of TOKEN.
12a mn
Healen 12 mnd)
Los
weiikea te 12 inl Abcus
Collaon (N)
Metool
Arcuas
Arca
Jotai
Arca
Long tadl Und Ineectad ratenu
PARA-Drive
8.56
0.06
2.16
1.15
691
0.09
0.20
0.36
6.21
6.88
0447
1,11
3.51
1.96
1.5
0.51
TOKIN
0331
8.8
838
6.18
0.99
3,47
1,00
PARA-19w
1:12
1.50
8m0
0.20
0.16
196
(val) (va legetail muining settcc scsanedd
TORES
0.28
0.26
133
8.9
R38
8366
8.15
0.24
6.5
6.3
8.56
166
3.29
194
1.82
041
Table 9: Planning performance of TOKEN and PARA- with 50% long-tail tail training scenes
removed.
17

---

Missed object
TOKEN
GI
autonomous vehicle try plan safe efficient motion. aulonomous velicle neds right at next interscction. What objects are important for the autociomuus vehicle's planning? What are these objects, their (s3) localions? and how to interact with them? Please plan the autonomeus vahicie's 3s future trajectory using waypoints, one every 0.5 scconds.
Prompt: You
TOKEN:
Them no imporiant object for the aulonomous vehicle's planning, The autonomous vehicle should keep forcard, and its 3-second future trajectocy is.. jomilted]
Figure 10: Failure mode example: The critical object (an oncoming motorcycle annotated by the red rectangle) is not detected by TOKEN's scene tokenizer, resulting in a dangerous motion plan.
K
Failure Mode
One limitation of TOKEN is using a pre-trained and frozen PARA-Drive model as the scene tok- enizer, which makes the TOKEN's performance tightly coupled with the quality of the pretrained tokenizer. In Fig. 10, we illustrate a failure mode where the critical object (the motorcycle) is not detected by the tracking querying transformer in PARA-Drive. Consequently, TOKEN assumes the road is clear proceed and fails to generate a motion that yields the oncoming motorcycle. Further work will focus co-training PARA-Drive leverage the knowledge within the LLM to improve the scene tokenizer.
L Extended Limitation & Future Work
TOKEN currently is not validated in closed-loop evaluations. We think that open-loop metrics are still valuable, and enable us to efficiently validate the model in a controlled environment free from the noises brought by other components in closed-loop evaluations (c.g., agents reactivity, motion optimization, and control). To mitigate the bias induced by general open-loop metrics, we specifically evaluate our planning metrics using key frames that reflect the critical behavior/decision changes. We will investigate TOKEN's reasoning and planning performance with closed-loop simu- lation in our future work. Last but not least, one of the limitations of deploying LLM-based systems onboard is their slow inference speed. In our current setup, TOKEN takes approximately 1.3 sec- onds to generate mid-level behavior and around 1.8 seconds to generate the motion plan without any speed optimization using an 1100 GPU. One practical deployment solution is to use TOKEN solely as runtime decision-level monitor: instead predicting the full motion plan, could predict route- conditioned mid-level behavior (c.g.. bypassing construction zone) much slower frequency to inform the traditional motion planner. Although there are many ongoing efforts aimed at improving LLM inference speed through quantization and caching, the large model size still makes it infeasible for onboard deployment. One direction we are particularly interested in is distillation. In our future work, we are very excited about distilling the 7B TOKEN model into a much smaller, lightweight model that can retain the benefits of common-sense reasoning in long-tail events while being more suitable for onboard deployment.
18


Reviews:
[{"review_id": "44jJ3Wrda9", "reviewer": "Reviewer_Ybio", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "One significant challenge of driving with LLMs is tokenizing scenes. Describing scenes with structured or natural language may cause loss or inaccuracy of information. The authors argue that embedding objects/scenes with an object-centric transformer and then aligning the modalities can better tokenize the scenes, thus providing better performance in downstream planning tasks in long-tail scenarios.", "summary_of_recommendation": "While the paper presents an innovative approach to scene tokenization for LLM-based autonomous driving, several key limitations undermine its overall impact. Further work is needed to address concerns about dataset construction, evaluation methodology, scalability, and real-world applicability before this method can be considered truly promising for autonomous driving systems.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "See review", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dQQ6RZgvbw", "reviewer": "Reviewer_vv3a", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, the pre-trained end-to-end backbone is used as a scene tokenizer for the MM-LLM planning task. The representation and structured reasoning process alignments are important for evoking the common-sense reasoning ability of the LLM backbone for planning. The open-loop performances demonstrate the superiority of Token in long-tail scenarios.", "summary_of_recommendation": "The overall text flows smoothly and is easy to understand. However, the novelty and the closed-loop performances should be further considered.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 2, "questions_for_rebuttal": "I hope to see closed-loop results; please refer to the \"Weaknesses\" section for more details.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TdWGu4L67d", "reviewer": "Reviewer_gVTX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes object-centric tokenization (TOKEN) to extract informative and structured information to address long-tail scenarios for the planning ability of autonomous vehicles. These long-tail scenarios include executing 3-point turns, resuming motion after a full stop, overtaking parked cars through the oncoming lane, and navigating construction sites.", "summary_of_recommendation": "The paper shows a good potential to address long-tail scenarios for autonomous driving, but some revisions are needed to fix the concerns.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Elaboration on the longitudinal-weighted L2 distance for a 3-point turn, as shown in Table 2, and the PARA-Drive’s planning performance (shaded in grey in Table 1).\n\nWhy there is some fragmentation in the predictions of TOKEN and PARA-Drive?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper "Tokenize the World into Object-level Knowledge to Address Long-tail Events in Autonomous Driving" introduces TOKEN, a Multi-Modal Large Language Model (MM-LLM) designed to improve autonomous vehicle planning in long-tail scenarios, or unusual and infrequent events, by tokenizing the world into object-level knowledge.   

Strengths:
- Insight that object level tokenization needs less data is useful
- Aligns text level representations like Llama with object level representations
- Long tail scenario is a data hungry regime, achieves few shot generalization

Weaknesses:
- Performance highly depends on pretrained tokenizer, and the latter needs lots of data. Inference latency might be slower given larger models?
- Trajectory end to end performance not better than PARA-driver. This is a major weakness and is not well explained. 
- No experiments on hardware

Reviewer Ybio initially gave a "Weak Accept" rating, questioning the construction of the long-tail dataset, the evaluation prompt generation process, and the scalability of the object-centric tokenization approach. The authors responded by providing additional details about their dataset construction methodology, explaining that it was statistically valid given the available data. They also clarified that evaluation prompts were manually generated to ensure neutrality. Regarding scalability, they acknowledged the current limit of 64 object tokens but suggested potential improvements like pre-filtering object tokens. 

Reviewer vv3a initially recommended a "Weak Reject," raising concerns about the novelty of the approach, the lack of closed-loop evaluation, and the need for analysis of inference time and deployment possibilities. The authors addressed these concerns by highlighting the differences between their work and LMDriver, clarifying the training data used for the tokenizer, and discussing potential solutions for inference latency and onboard deployment. They also acknowledged the limitations of open-loop evaluation but emphasized its value for efficient model validation. The reviewer did not update their "Weak Reject" recommendation. 

Reviewer gVTX provided a "Weak Accept" rating but pointed out the model's dependence on the pre-trained tokenizer, its limited performance in certain scenarios, and the lack of elaboration on PARA-driver's superior performance in some cases. The authors responded by discussing the possibility of co-training the tokenizer in future work, explaining the nuanced evaluation of motion plans using different L2 error variants, and addressing the comparison with PARA-driver in their general statement. They also committed to releasing their dataset and code. 

Overall I recommend weak accept.
Decision: Accept