WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving
Authors: Yiheng Li
Cunxin Fan
Chongjian GE
Seth Z. Zhao
Chenran Li
Chenfeng Xu
Huaxiu Yao
Masayoshi Tomizuka
Bolei Zhou
Chen Tang
Mingyu Ding
Wei Zhan
Venue: ICML 2025 poster (2025)
Keywords: Language Q&A Dataset
Autonomous Driving
Interaction Reasoning
Multi-modal Learning
Primary area: applications->robotics
TL;DR: We introduce WOMD-Reasoning, the largest language dataset focusing on traffic rule-induced interactions in driving.

Abstract: Language models uncover unprecedented abilities in analyzing driving scenarios, owing to their limitless knowledge accumulated from text-based pre-training. Naturally, they should particularly excel in analyzing rule-based interactions, such as those triggered by traffic laws, which are well documented in texts. However, such interaction analysis remains underexplored due to the lack of dedicated language datasets that address it. Therefore, we propose Waymo Open Motion Dataset-Reasoning (WOMD-Reasoning), a comprehensive large-scale Q&As dataset built on WOMD focusing on describing and reasoning traffic rule-induced interactions in driving scenarios. WOMD-Reasoning also presents by far the largest multi-modal Q&A dataset, with 3 million Q&As on real-world driving scenarios, covering a wide range of driving topics from map descriptions and motion status descriptions to narratives and analyses of agents' interactions, behaviors, and intentions. To showcase the applications of WOMD-Reasoning, we design Motion-LLaVA, a motion-language model fine-tuned on WOMD-Reasoning. Quantitative and qualitative evaluations are performed on WOMD-Reasoning dataset as well as the outputs of Motion-LLaVA, supporting the data quality and wide applications of WOMD-Reasoning, in interaction predictions, traffic rule compliance plannings, etc. The dataset and its vision modal extension are available on https://waymo.com/open/download/. The codes & prompts to build it are available on https://github.com/yhli123/WOMD-Reasoning.

WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving
Yiheng Li
Cunxin Fan
Chongjian Ge Seth Z. Zhao? Chenran Li' Chenfeng Xu'
Huaxiu Yao"
Masayoshi Tomizuka
Bolei Zhou? 2 Chen Tang 14f Mingyu Ding 13t
Wei Zhan'
Abstract
1. Introduction
Language- guided models improve explainability, controlla- bility, and performance of autonomous driving tasks (Tou- vron et al., 2023; Brown et al., 2020; Chen et al., 2023; Fu et al., 2023; Malla et al., 2023; Xu et al., 2024a; Li et al., 2024a; Arai et al., 2024; Park et al., 2024; Li et al., 2024b; Cui et al., 2024; Ma et al., 2024; Yuan et al., 2024; Greer Trivedi, 2024; Ding et al., 2024a; Han et al., 2024; Wang et al., 2024b), owing their enormous text-based knowl- edge acquired through the pre-training. To help language models elevate abilities on driving-related tasks, various language-based datasets (Sima et al., 2023; Qian et al., 2024;
Language models uncover unprecedented abili- ties in analyzing driving scenarios, owing to their limitless knowledge accumulated from text-based pre-training. Naturally, they should particularly excel analyzing rule-based interactions, such as those triggered by traffic laws, which are well documented in texts. However, such interac- tion analysis remains underexplored due to the lack of dedicated language datasets that address it. Therefore, we propose Waymo Open Motion Dataset-Reasoning WOMMDReesooiigg. a com- prehensive large-scale Q&As dataset built on WOMD focusing on describing and reasoning traffic rule-induced interactions in driving sce- narios. WOMID-Reasnnnng also presents by far the largest multi-modal Q&A dataset, with 3 mil- lion Q&As on real-world driving scenarios, cov- ering a wide range of driving topics from map descriptions and motion status descriptions to nar- ratives and analyses of agents' interactions, be- haviors, and intentions. To showease the applica- tions of WOMDD-Reasoning we design Motion- LLaVA, a motion-language model fine-tuned on WOMD Reasonina.n Quantitative and qualitative evaluations are performed on WOMD-Reasoning dataset as well as the outputs of Motion-LLaVA, supporting the data quality and wide applica- tions of WOMD-Reasoning in interaction pre- dictions, traffic rule compliance plannings, etc. The dataset and its vision modal extension are available on
Kim et al., 2018; Sachdeva et al., 2024a; Malla et al., 2023; Inoue et al., 2024; Zhou et al., 2024; Tian et al., 2024b;
Arai et al., 2024; Park et al., 2024; Li et al., 2024b; Ma et al., 2024) have been proposed to fine-tune these L.LMs. However, due to the complexity and variations summariz- ing traffic rule-based interactions in real driving scenarios, existing datasets often focus interactions based on spatial proximity. For example, in BDD-X (Kim et al., 2018), inter- actions are often limited to the spatial blocking, like 'The car moves back into the left lane because the school bus in front of it is stopping.': while in another example, DriveL.M (Sima al., 2023) attributes interaction to 'keep safe dis- lance' without further detailed explanations. Likewise, most existing works only cover interactions induced by proximity, as shown in our amalysis in Table 1. While reasoning about many non- non-proximate interactions, like those caused by traf- fic rules, are vital for the safety-critical decision-make in autonomous driving systems (Wang et al., 2022; Zhang et al., 2022; Zhao et al., 2024; Ding Zhao, 2023: Roelofs et al., 2022), these interactions are rarely included in previ- ous datasets, leading to suboptimal performance of LLMs fine-tuned on these datasets. Additionally, most datasets in driving are not large enough to support multi-modas fine- tuning across diverse tasks such as scene description, pre- diction, and planning.
The codes & prompts to build it are available on https//github..om/yylll 2ttm/yyhli13//MMDDD WWWOO--eesooiii
"Equal contribution 'UC Berkeley FUCLA 'UNC-Chapel Hill AUT Austin. Correspondence to: Mingyu Ding <md @ cs. cs.unc.eduz, Chen Tang <chen.tang @ austin.utcxas... .ddu
For the sake of interaction descriptions and reasoning, this work introduces WOMD-RRasoning dataset, large-scale multi-modal dataset centered on language Q&As based on WOMD (Ettinger et al., 2021). To incorporate interactions induced by traffic rules and human intentions, we build an automated data curation pipeline by prompting ChatGPT
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

WOMMD-Rcsooning A Large-Seale Dataset for Interaction Reasoning in Driving
Agent Other Agent
Future Travctory IX for Future Destination)
Traffic Light For Lane (Green/
Intersectin of Interest
Speed Bumps Crosswalss
(a)
(b)
Interaction
Intention
Interaction
Intention
interactions expected between the ego agent and surrounding NO they approach intersection?
What the anticipated intention of the ego agent in the near future?
will interaction the ego agent and surrounding agent #1 unfold??
ago agent's plan immediate future?
Q
Surrounding agent AD will yleld to ego egent sign and not moving, while the ego agent approaching the intersection without stop sign.
The ego agent intends to continue towards the intersection speed. It will not need yield to surrounding agent #O who at stop sign and will yisld the ego agent...
Surrounding agent #1 will yleld to the ego agent because is moving ego agent the right way with a green arrow traffic light.
The ego agent intends to complete its tumn and intersection has the right of way with a green arrow traffic light...
stop
consiant
Figure 1. Examples of traffic rule-induced interactions in hetween the ego agent and agent #0, attributing it correctly to the stop signs. (b) shows the traffic light-controlled yielding interaction between the ego agent and agent #1. The front-view visualization is created by MetaDrive simulator (Li al., 2022)
dataset. (a) captures the traffic rule-induced interaction
(Brown et al., 2020; OpenAl et al., 2024) with a set of well- designed prompts. Together with a rule-based translator to create auto-
driving language models.
Our contributions can be summarized follows:
convert initial motion data into language, mated pipeline to generate the language dataset and achieve an approximate accuracy rate of 90%. WOMID-Reasoning dataset covers interactions induced by traffic rules and hu- man intentions with 409k Q&As, like those shown in Fig- ure As indicated in Table 1, Reasoning contains the highest number of total Q& and interaction-speciff As, supporting wide range of applications. including scene description, prediction and planning. Figure 2 fur- ther demonstrates its uniqueness of interaction descriptions in WOMD-Reasoning with vocabulary statistics. In addi- tion to comprehensive and in-depth interactions, WOMD- Reasoning is also the largest language dataset for real-world driving data, to the best of our knowledge and by its publica- tion. Additionally, to facilitate multi-modal driving-related tasks like training end-to-end driving models, in addition to the LiDAR (Chen et al., 2024) info provided by Waymo, we render both the corresponding bird's 's eye view and front view videos of covered driving scenarios by the MetaDrive simulator (Li et al., 2022), which are attached to the dataset.
We propose WOMD-Reasoning the largest multi- modal dataset with 3 million Q&A pairs centered on interaction reasoning in driving. It provides extensive insights into critical but previously overlooked inter- actions induced by traffic rules and human intentions. Quantitative and qualitative evaluations verify its wide coverage in interaction descriptions and reasoning, as well as its accuracy.
Fine-tuned with WOMD-Reasoning our Motion- LLaVA is capable of providing detailed and insight- ful interaction predictions on various driving scenar- ios, supporting the effectiveness of WOMD-Reasoning. Extensive evaluations with various metrics validate the wide coverage and high quality of the answers of Motion-LLaVA fine-tuned on WOMD-Reasoning, ranging from driving scenario descriptions to traffic rule-compliant planning.
To showease the applications of WOMD-Reasonn Reasoning, we fine- tune our Motion-LLaVA, which is built upon LLaVA (Liu et al., 2024), sign and a set of effective training and inference strate- gies ablated, Motion-LLaVA achieves high accuracy in both driving scenario understanding and interaction analysis, which unequivocally advocate the effectiveness of WOMD- Reasoning in boosting the interaction prediction abilities of
To optimize the performance of Motion-LLaVA, a motion- model designed for WOMD- Reasoning with robust capabilities, we benchmark task performance across various configurations, including input modalities, reasoning techniques, and network architectures.
WOMD-Reasoning
With careful de-
2

---

WOMD- A Large-Seale Datasct for Interaction Reasoning in Driving
Table Comparison between WoaD-Reasoning Dataset and Previous weal-world Language Datasets for Driving, WOMD-
Reasoning covers more comprehensive kinds of interactions. with a significantly higher size supporting wide range of applications.
Statisties
Interactions
Applications
Dutnse
Data Source
Total Scemes
Total QutAs
Inseraction Q&Ass
Distunce induced
Traffic Rule-indueed
Human Intention-induuee
Scene Descriptios
Metion Prodiction
Planning
Motinn
muScenes-QAAa QAiQQu et al,, 2024)
CaSoenessCleiarccoo et al. 20201
34k
460k
X
BDD-X(Kin et sl., 20183
BDDXX et al,, 2017)
26k
26k
26k
x
x
x
Deivel M/Sina al,, 2023)
muSoenes( Caooar el., 20200
5k:
44%k
1119
x
Kmnk2TellSSacddere
2124a)
Rani2TelliSSchdeea
20124a3
116
>116
>116
/
/
DRAMA(Malla ot al,, 2023)
DRAMA/Malla et al., 2023)
18k
1022
<18k
X
X
(ilnommot(Dnng et al., 2024h)
mn miSornes(Caossr et sl,, 2020)
850
91k
<46k
K
X
7
nutiaptiont Yang al, 2025)
auSoenes(Cassar al., 20220)
<ik
420k
X
et et al. 20244
muSeenes(Casaar 20200)
850
300,
WOMD-Roswening
WOMD (Piringor et at, 2021)
63k
2,9408
40%
2. Related Work
sis to human labelers. To minimize human labor, first employ manual rules and then utilize ChatGPPT- (OpenAl et al., 2024) to build our dataset automatically. The resulting dataset, HOMDDReasoningg is rich in interaction details, particularly those stemming from traffic rules and human intentions. The designed automated data-curation pipeline enables the dataset to be significantly larger. We also create high-quality human-verified subset. Preliminary human evaluation provides positive feedback with an approximate accuracy of 90%.
Application of Language Modal in Autonomous Driving. In recent years, language has been widely explored in au- tonomous driving, incorporating language into implicit features could enhance the explainability (Yang et al., 2023; Roh et al., 2020; Sriram et al., 2019; Chen et al., 2019; Mao et al., 2023a; Zhang et al., 2024). For example, Kuo et al. (Kuo et al., 2022) employ language features in the trajectory prediction model, ensuring both explainability and consis- tency between language and predictions. Similarly, Zhong et al. (Zhong et al., 2023) use language input and LLMs to generate conditions in diffusson models, thereby control- ling the generation of driving scenarios. Recently, general driving agents have been developed (Xu et al., 2024a; Chen et al., 2023; Sima et al., 2023; Mao et al., 2023b), aim- ing to integrate all functions pereeption, prediction, and planning- into a single language agent. However, we ob- serve that their performance in interaction analysis is not entirely satisfying, especially in non-proximal interactions like those induced by traffic rules. This shortcoming is largely attributed to the lack of interaction analysis in lan- guage datasets specific to driving scenarios.
Fine-tuning LLM for Driving Tasks. Recent years have witnessed a surge of research on leveraging LLMs and vision-language models (VLMs) for driving applications (Seff al., 2023; Chen al., 2023; Jin al., 2023; Xu et al., 2024b; Sima et al., 2023; Shao et al., 2023; Mao et al., 2023a; Tian et al., 2024a). However, handling het- erogeneous inputs such as vectorized motion representation remains challenging due to their domain gaps to most ex- isting VLMs and LLMs. Notable prior work, such as (Mao et al., 2023a), designed natural language prompt structures to accommodate numerical motion data, while another ap- proach (Chen et al., 2023) proposed training LLMs with custom-built motion vector encoders.
Language Datasets in Autonomous Driving. Language datasets for driving scenarios have recently been developed to support related LLM-based work. Due to the controlla- bility of simulations, several iimulation-based datasets have been created (Chen et al., 2023; Sima et al., 2023). However, simulations often fail to capture real-world interactions. To address this, recent studies (Kim et al., 2018; Sachdeva et al., 2024a; Malla et al., 2023; Wang et al., 2024a) incorporate human labeling of woal-world driving scenarios to ensure the inclusion of genuine interactions. Nevertheless, obtaining high-quality human labels is labor-intensive and costly, lim- iting the size and coverage of these datasets. Moreover, in- teractions driven by traffic rules and human intentions have often been overlooked. To streamline the labeling process, some studies (Deruyttere et al., 2019; Nie et al., 2023; Qian et al., 2024; Sima et al., 2023) have generated based labels, although these typically cover only basic language elements like scene descriptions, leaving interaction analy-
Our Motion-LLa VA approach is designed for benchmarking the performance of WOMD-Reasoning Inspired by the design of (Chen et al., 2023), Motion-LLaVA augmenting existing LLM architectures with tailored components. We employ motion prediction model as a motion vector en- coder, utilizing its prior knowledge streamline the training pipeline. To enhance reasoning accuracy, we adopt a Chain- of- of-Thought (CoT) (Wei et al., 2022) approach, allowing the model to base its reasoning on factual grounding, mitigating potential hallucinations. Based on WOMD-Reasoning our Motion-LLaa demonstrates the practicality of applying VLMs real-world vectorized motion data to accurately answer diverse driving-related questions correctly.
3. Method
In this part, we first introduce how to build the WOMD- Reasoning dataset. To verify the effectiveness of WOMD-
3

---

WOMD-Reasoning: A Large-Seale Datasct for Interaction Reasoning in Driving
Reasoning, we introduce Motion-LLaVA, a multi-modal fine-tuning approach, to build motion language models upon WOMD-Reasoning. The Motion-LLaVA method as well as the fine-tuning process are also included here.
first prompt Motion-LLaVA answer the Factual ques- tions, which include questions in map environment, ego, and other agents' motion status. Then, we aggregate these answers and feed them into Motion-LLat VA as the contexts for answering Reasoning questions in interactions and in- tentions analysis. Examples of these contexts can be found in the Appendix A.8. We ablate in Part A. that such CoT strategy can effectively improve the quality of answering reasoning questions.
3.1. Building WOMMD-Resooning Dataset
Manually constructing autonomous driving datasets with fine-grained natural-language labels regarding the interac- tions among road participants would involve intense hu- man labor, which is one of the main reasons why previous datasets have insufficient interaction analysis and a very lim- ited overall size (Sachdeva al., 2024b). To reduce human labor while obtaining reasonably useful data on vehicle inter- actions, we propose a fully automatic data-curation pipeline to label the WOMD dataset with language: We first develop a rule-based program to interpret the motion dataset, which contains trajectories and the HD map, into language; and then build a set of prompts to utilize the reasoning abilities of ChatGPT-4 (OpenAI al., 2024) to generate interaction analysis and reasoning, and organize the results into the target Q&& format. Details of each step are attached in the Appendix A. Codes for interpreting the motion dataset as well as the full prompts for ChatGPT-4 are available in supplemental materials.
Specifically, we take LLaVA-v1.5577 (Liu et al., 2024) as the pre-trained VLM. Since MultiPath++ (Varadarajan et al., 2021) did not release their codes nor checkpoints, we take the implementations by (Konev, 2022). Our multi-modal fine-tuning takes GPU days (x= 1 day on 2xNVIDIA A6000 GPUs) to train epoch on the entire training set of WOMD- Rea-oniisnnn.. For quantitative evaluations, we ran- domly select x 1,000 QA pairs from the validation set of WOMDD-Reasoning.
To benchmark Motion LLaVA model, we utilize a suite of standard language metrics, including BLEU-4 (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), CIDEr (Vedantam al., 2015), SPICE (Anderson al., 2016), and ROUGE-L (Lin, 2004), along with GPT Score (Radford et al., 2018) to assess the quality of the generated content. Details on the prompts used to calculate the GPT Score can be found in Appendix A.5.
Our analysis performed on Microsoft Azure ChatGPT-4- Turbo API, which costs around 12,750 USD. The details of WOMD-Reasoning will be provided in Part 4.
4. WOMD-Reasoning Dataset Specifications
3.2. Fine-tuning Multi-modal Model on WOMDD-Raaooning
In this section, we provide detailed information about WOMD-Reasoning, We first present the organization and statistics of the dataset, then provide quality analysis based on case studies and preliminary human evaluations. Fi- nally, we compare our dataset to previous ones to show the strengths of WOMD-Reasoning in diverse interactions, huge size, and wide potential applications.
To showcase the application of WOMD- dataset, we use our Motion-LLaVA approach to fine-tune upon LLaVA (Liu et al., 2024). LLaVA is chosen to accom- modate motion data and text prompts input at the same time, with the vision encoder adapted to a motion data encoder. Inspired by (Shao et al., 2023), we utilize encoders from motion prediction models MultiPath++ (Varadarajan et al., 2021) for encoding motion data. Ablations in Table 15 prove the usefulness of this encoder. Some detailed fine- tuning strategies are in Appendix A. Details on how we process agent IDs in Motion-LLaVA are listed in Appendix A.3. The overall Motion-LLaVa VA pipeline is illustrated in Figure 4.
4.1. Dataset Statistics
Table 2 provides the statistics of Q&As in WOMD- Reasoning dataset, as well as in every Q&A class. Our Q&As can be classified into two parts: The Factual con- tents and the Reasoning contents, where the first part serves as the context for the second part. The factual contents contain three topics: Map Environment, which includes the existence and category of intersections, existence and places of stop signs and crosswalks, and count of lanes, etc. Ego Agent's Motion Status and Other (Surrounding) Agents' Motion status, which describe each involved agent's speed, acceleration, direction, related traffic light and sign, and relative position to the intersection center as well as to the ego agent. The factual descriptions alone are also good for training language scene describers.
During training, we unfreeze all components including the motion vector encoder, and train on all Q&A pairs in simultaneously to avoid the potential catastrophic forgetting. To prevent Motion-LLaVA from forgetting its inherent LLM's knowledge, we mix up a few questions irrelevant to the current driving scenario into the training process (Zhai et al., 2023). During inference, we adopt Chain-of- Chain-of-Thought (CoT) Reasoning strategy (Wei et al., 2022) to reduce the hallucination. Specifically, we
4

---

WOMD- A Large-Seale Dataset for Interaction Reasoning in Driving
Table 2. Sizes RoMDDReasoning dataset.
120.000
Interactions
Map
Eggo
Other
Interactions
Insentions
Total
Scenes
Intention
Agent
Agents
120.000
Training
1888
268k
1,635k
287k
52k
2,40
52k
Number of Decurrences
Validasion
41k
58k
341k
5%
510k
11k
00.000
Tocal
2298
326k
1,976
346k
63k
2,400
Per Scene Ave.
3.58
5.09
30.8
5.41
0.98
45.95
00.000
Table Human Evaluations of WYOMD-Reasonin Dataset.
40,000
Q&A Class
All
Interactions
Intentions
20.000
Accuracy
91 .999
91.03%
87.50%
Paitic
Aed
Oreitabe
Follw
Stop
Our key part, the Reasoning contents, first comes with Q& on Interactions, which includes a summary of inter- actions between each surrounding agent and the ego agent and the reasoning behind such interactions. Then by sum- ming up all interaction information, offer the Intentions, which give the predicted intention of the ego agent, consid- ering its responses to all the interactions. The interaction and intention reasoning parts provide an unprecedented tool for training language-basse prediction and planning agents.
Vacabulary
Figure 2. Selected vocabulary statistics in WOMD- Reasonnns.ning. We stat vocabularies strongly related to traffic rule-induced and human intention-inducee interactions in WOMD MeaD-nnassnnng, illus- trating that it contains abundant such interaction descriptions and reasoning.
human intention-induced interactions.
WOMD ReaDonnea contains traffic rule-induced inter- action information. Firstly, in Figure (a), we show a traffic scenario of an intersection controlled by the stop sign. In this case, the agent #0 is quite far away from the ego agent at the starting (current) moment, thus their interac- tions would rarely be considered in previous datasets, as they mainly cover distance-induced interactions. However, the two agents do have a significant interaction: Controlled by the stop signs, the agent #0 has to yield to the ego agent, even though the ego agent is still far from the intersection. This case vividly proves the remarkable importance of the traffic rule-induced interactions we cover.
As shown in Table 1, WOMD-Reasonin contains 346k unique traffic rule-induced and intention-induced interaction Q&As, as well as 63k comprehensive intention predictions of the ego agent containing responses to the interactions, which rarely exist in previous datasets. We will show these features through examples and statistics in the following section. Furthermore, we also claim that to the best of our knowledge, WOMMD-Reasoning is the largest real world language data set for driving, since our WOMD-Reesoning provides roughly 3 million Q&A pairs for 63k scenes in WOMD, which are 6- 113 times greater than those of other datasets we compare to.
Another example is a traffic fight-controlld intersection shown in Figure/ 1 (b). Similar to the previous one, agent #0 is initially not close to the ego agent. However, thanks to our analysis of traffic rules, our language dataset effectively captures the interaction that the agent #1 would yield the ego agent who is turning left and having right-of-wan due to arrow green light. This further proves that traffic rules are essential in determining the interactions.
Compared with existing data sets, as shown in Table 1, our WOMD-Reasoning is the largest real-world dataset in terms of total scenes, total Q& and the total interac- tions included. Also, WOMD-Reasoning supports Q&As in scene descriptions, prediction, and planning. Together with the simulated video of each covered scenario, it is suit- able for training VLMs to perform nearly all major tasks in autonomous driving. Diving even deeper into the inter- action reasoning these datasets, we summarize that most interactions covered in previous datasets are caused by very near distance. However, as we show in Sec. 4.3, many initially
WOMD- ReaDonnng contains human intention-induced interaction information. Beside traffic rules, our dataset also covers human intention-inducee interactions. We pro- vide an additional example show this, as well as demon- strate whole picture of each section of the dataset. In Figure 3, WOMD-Reesome provides fruitful information on the human intention-induced interaction overtaking, based on pattern recognition abilities built in the GPT. We also show 1-2 Q&As for each category of the dataset for completeness.
interactions happen when the two counterparts far away, but they establish interactions by following the traffic rules or by human intentions. Our WOMD-Reasoning contains a wealth of this interaction information, as shown in Figure 2, the vocabulary statistics of the interaction and intention part of our dataset.
4.2. Interaction Contents
4.3. Dataset Quality Evaluation by Human
To testify to the quality of WOMD MoMD-Reasoning, we show examples from it containing rich traffic rule-induced and
To grab a sense of the accuracy of execute a human evaluation process. Consent was obtained
we
5

---

WOMD-Rcasonin A Large-Seale Datasct for Interaction Reasoning in Driving
Agent
Road Environment
Can you describe the number of lanes available to the ego agent?
There are three lanes on the ego agent's side.
Other Agent
Future Trajectory
Future Destination)
Ego Agent Status
Could you specify lane position of ego agent?
The ego agent is on the first lane from the left.
Other Agent Status
What is the facing direction of surrounding agent #0 relative ego agent?
Surrounding agent #O is heading in the same direction ego agent.
What kind of interaction be expected between the ego agent and surrounding aignnt
Surrounding agent #O will overtake the ego agent as it is accelerating and the ego agent is decelerating.
Interactions Intention
Figure 3. demonstration of Q&As in each part of WOMD- dataset. We show Q&As in all categories regarding the scenario while demonstrating language analysis of overtaking, a human intention-induced interaction in WOMD- dataset.
Answers to the Questions
prediction tasks.
Large Language Model
5. Dataset Application and Evaluation with Motion-LLaVA
Motion Data Embeddings
Language Embeddings
To demonstrate a real-world application of WOMD- Reasoning, we design Motion-LLaVA, a multi-modal model taking motion data as input, and fine-tune it on WOMD- Reasoning. We show that the model gains significant abili- ties from WOMD-Reasome
Projector
Motion Vector Encoder
Vectorized Molion Data
Questions
predicting interactions, es- pecially those caused by traffic rules, and in planning right- of way compliant route for the ego. Furthermore, its abil- ities extend to answer various questions related to driving across factual and reasoning categories, which is supported by both qualitative showcases and quantitative evaluations on language metrics. These highlight the broad applications of Reasonnas for the autonomous driving industry.
Figure 4. The Motion-LLaVA pipeline fine-tuning multi- modal model with WOMD- Reasonnng Motion data go through pre-trained motion vector encoders from Multipath++ (Varadara- jan et al., 2021) and a projector layer to serve together with the questions in WOMD-Reasoning as the inputs. The answers in WOMD-Reasoning serve as the supervision.
from all the people interviewed. We ask interviewers eval- uate whether the correct answer to the question is included in the answer provided in the dataset. We invite 4 people to judge 1,610 Q& in total. The results are shown in Table 3. We conclude that WOMD-Reasoning achieves a satis- fying accuracy, especially considering it is automatically generated. Further human-based data cleaning is ongoing.
5.1. Interaction Prediction
Since WOMD-Reasonine is proposed to focus on the inter- action analysis, one of the key abilities of Motion-L.La fine-tuned on it is to predict interactions, especially the traffic rule-induced interactions. We present a few inter- action prediction examples answered by LLaVA in Figure 5. These results confirm that by fine-tuning on WOMD-Reasoning Motion-LLaVe unleashes the poten- tial of language models for predicting traffic rule-related interactions in unseen real-world scenarios.
4.4. Vision Extension with Simulations
To support applications requiring visual information, we provide an API extend WOMD--Reasoning scenario de- scriptions to the vision domain, compensating for the ab- sence of raw camera data in the original WOMD dataset. We achieve this by utilizing an open-souree traffic scenario simulation platform, namely ScenarioNet (Li et al., 2023), and employing the MetaDrive (Li et al., 2022) simulator to replay WOMD ReaD-nnnn scenarios in both BEV layout and ego-view images through 3D rendering. This approach enables visual inputs to support the application of WOMD- Reasoning for future research in the vision and language domain. For video generation, we produce a 90-frame 10 Hz video aligned with Waymo's 's trajectory data, including 10 frames of historical footage and 80 frames of future tra- jectory. Users may choose to use the historical portion for
5.2. Traffic Rule Compliant Planning
Based on thorough interaction analysis, traffic rule com- pliant planning can naturally be achieved by summariz- ing interactions, as we do in producing ego agents' inten- tions in WOMD-Reasoning. Motion-LLaVA also perfectly learss this ability. In Figure 6, we demonstrate that Motion- L. fine-tuned on WOMDD-Resonnin can offer traffic rule-compliant future trajectory plans, which clearly extends the applications of WOMD-Reasoning into planning.
6

---

WOMD-Reasoning A Large-Scale Dataset for Interaction Reasoning in Driving
Ego Agent
Future Trajectory (X for Future Destination
Intersection of Interest
Other Agent
(Green/ Red)
Crosswalks
STOP
What interactions are likely occur belween the ego agent and surrounding agent
What kind of interaction can be expected between the ego agent and surrounding agent #3 the intersection?
How ego agent surrounding
agent #2 interact in the intersection?
Surrounding agent #0 will yield to the ego agent because they are both approaching intersection, but surrounding agent is at stop sign and the ego agent is already in the Intersection.
Surrounding agent #3 will yield to the ego agent because it is decelerating approaching stop sign the ego agent accelerating and exiting the Intersection.
Surrounding agent #2 will yield the ego agent because moving the ego agent has green arrow traffic light.
Figure Interaction predictions made by Motion-LLaVa on various WOMD scenarios.
Ego Agent
Future Trajectory (X for Future Destination)
Traffic Lane Green Red)
Intersection of Interest
Speed Bumps Crosswalks
Other Agent
MOL
What ego agent's intended action in the immediate future?
What ego agent planning shortly?
ego agent's plan in the immediate future??
The ego agent intends to continue exiling the intersection. proceed straight as the traffic light green and there are immediate interactions that require to yleld stop. Surrounding agent will lead the ego agent, and surrounding agents #o and will yleld to the ego agent to their positions and traffic light status...
The ego agent intends complete left tur and exit the intersection. will continue to accelerate as exits intersection, and does respond to any of the surrounding agents as they are either not moving, ylelding due traffic lights, not in conflicting path.
The ego agent intends continue exiting intersection.
proceed straight, and surrounding agent #0 will yield to due to the stop sign. Therefore, the ego agent continue need to yield to surrounding agent #o.
to
without
Figure 6. Traffic rule-compliant planning made by Motion-LLaVA on various WOMD scenarios.

---

WOMD-Rccsoninn A Large-Scale Dataset for Interaction Reasoning in Driving
Ego Agent
Future Trajectory IX for Future Destination)
Traffic Lights For the Lane (Green / Red]
Intersection of Intarest
Other Agent
Road Environment
Can you describe the type of intersection
The intersection is a 3 way intersection.
present in the current driving scenario?
Ego Agent Status
What is the ego agent' current motion status?
The ego agent accelerating.
Other Agent Status
Where is surrounding agent #5 in relation to the ego agent?
Surrounding agent #5 is 18 meters behind the ego agent.
S Surrounding agent will follow the ego agent as they are both heading towards the intersection in the same direction, with surrounding agent #5 behind the ego agent.
Interactions & Intention
What kind of interaction will occur between the ego agent and surrounding agent #5?
Figure 7. An example set of answers to diverse driving-related questions generated by Motion-LLa VA on WOMD scenario. We see that Motion-LIaa is capable of answering all topics questions covered by WOMD- Reasonngg accurately.
5.3. Answering Various Driving-related Questions
metrics and question types, highlighting the advantages of utilizing the motion data as input. We hypothesize that this performance gap is due to the information loss that occurs when converting motion data into BEV plot with scale bar. We also visualize and compare the answers of Motion- LLaVA and these baselines on the same driving scene in Appendix A.6.
We further present a set of Q&A examples answered by Motion-LLaVA covering each category of WOMD- Reasoming in one specific driving scenario, shown in Figure We observe that Motion-LLaVA model capable of pro- viding accurate narratives and analysis diverse driving- related contents thanks to the wide coverage of WOMD- Reasoning.
In addition to language metrics, we also provide objective metrics to the answers to questions regarding Facts. We report the root median squared error (Median Error), and accuracy within 1.0-meter tolerance (Acc1.0) o) in Table 5, following similar metrics used in (Jin et al., 2023). Re- sults confirm that Motion-LLaVe is capable of providing numerically accurate answers to factual questions.
5.4. Validation of Motion-LLaVA
To validate the effectiveness of Motion-LLaVa and the fine- tuning process, we present evaluation metrics of all gen- erated answers, as well as those in the two categories, i.e. the Facts which include the road environment and agents' motion status, and the Reasonings which cover analysis on interactions and intentions. Benchmark results are shown in Table 4. We first verify the effectiveness of these language metrics with shuffled ground-truths, which attach shuffled true answers to questions randomly. Results confirm that low scores are granted for this model, indicating that simi- lar structure with wrong answer content would not be able to cheat our metrics. Then we benchmark the off-the-shelf LLaVA (Liu et al., 2024), where BEV plots are used as the vision input for the driving scenario. This non-fin non-fine-tuned version earns very low score in all metrics, confirming that LLaVA itself cannot answer driving-relate questions cor- rectly. Then, we perform fine-tuning on WOMD-Reasoning Reasoning. We compare two models with different modalities as motion input: LLaVA using the BEV plot of the driving scene as the vision input (Liu et al., 2024), and Motion-LLaVA which uses motion data as input. We observe that both fine-tuned multi-modal models significantly outperform previous base- lines across all metrics and question types, underscoring the for addressing
Table 5. Accuracy of Motion- Motion-LLLVA answers to factual questions.
Model
Acc (f).
Median Error (L)
Fine-tuned LLaVA (Liu et al. 2024)
10.9%
5.0
Motion-LLaVA (Ours)
56.2%
1.0
6. Dataset Effectiveness in Vehicle Trajectory Prediction
In this section, we evaluate the effectiveness of WOMD- Reasoning in a real-world driving task vehicle trajectory prediction. Our experiments indicate that with our WOMD- Reasoning dataset involved, the metries of vehicle trajectory predictions are significantly improved.
6.1. Language-assisted Trajectory Prediction Experiment Setup
We adapt the Multipath++ (Varadarajan et al., 2021) trajec- tory prediction model to accommodate the language input. The adaption mainly contains two steps:
value of fine-tuning on WOMDD-Resonnnn driving-related queries. Compared to the fine-tuned LLa VA, Motion-LLaa -LaVA achieves superior performance across all
Language Acquisition and Encoding The language we use comes from Motion-LLLaVA's output. We run Motion-
8

---

WOMD- A Large-Seale Dataset for Interaction Reasoning in Driving
Table Benchmarks of fine-tuned multi-modal models with language metrics.
Questions
Model
ROUGE (t)
BLEU (t)
METEOR (1)
CIDEr (t)
SPICE (1)
GPT Score (t)
Shuffled Ground-truths
0.311
0.263
0.375
1.48
Non-Fine-tunede LLaVA (Liu et al., 2024)
0.512
0.211
0.275
1.36
0.455
2.31
Fine-tuned LLaVA (Liu 2024)
0.779
0.581
0.439
5,51
0.735
6.88
Motion-LLaVA (Ours)
0.792
0.616
0.449
5.69
0.744
7.02
Facts
Shuffled Ground-truths
0.528
0.349
0.273
2.26
0.401
2.06
Only
Non-Fine-tuned LLaVA (Liu et al. 2024)
0.561
0.322
0.326
1.50
0.483
2.46
Fine-tuned a.AV (Liu al,, 2024)
0.821
0.702
0,495
6.05
0.783
6.91
Motion-LLaVA (Ours)
0.840
0.736
0,516
6.35
0.794
7.09
Reasonings Only
Shuffled Ground-truths
0.479
0.326
0.281
1.81
0.462
1.87
Non-Hinnetuned aVV (l et al,, 2024)
0.262
0.097
0,201
0.166
0.283
0.91
Fine-tuned LLaVA (Liu et al., 2024)
0.596
0.452
0.357
2.26
0.557
6.14
Motion-LLaVV, (Ours)
0.614
0.474
0.366
2.52
0.571
6.76
LLaVA on the cases covered by in the training and validation set of WOMD. For each sce- nario included, Motion-L_Laa. would provide a comprehen- sive set of Q&As, where we pick the interaction Q&As for this experiment. Each of these Q&As As includes the interac- tion info between a specific agent and the ego agent. We pick the answer parts of these Q&As to use. These answers then fed into T5 encoder (Raffel et al., 2023) followed by few MLP layers to be encoded. In the end, for each scenario included, we have set of language embeddings, each contains info on the interaction between the ego agent and one specific agent.
dataset
tory prediction models.
7. Conclusion
We first build WOMD-Reasoning a language-centered multi-modal dataset upon WOMD, focusing on analyz- ing and reasoning interactions. Specifically, our WOMD- Reasoning emphasizes interactions induced by traffic rules and human intentions, such interactions governed by traf- fie lights and signs, and interactions like overtaking and fol- lowing, which are both rarely covered in previous datasets. Besides, WOMD-Reesonne
provides million Q&A pairs, which makes it the largest real-world language dataset in autonomous driving to the best of our knowledge, cover- ing a range of topics from scene descriptions, prediction, to planning. To demonstrate the applications of WOMD- Reasoning, we propose Motion-LLaVA, which fine-tunes multi-modal model to accurately answer questions covering both driving scenario descriptions and interaction reason- ings. Motion-LLaVa realizes the function of interaction prediction and traffic rule-compliant planning, taking advan- tages of the WOMD- Reasoningg and the text-based knowl- edge intrinsically built language models. Motion-LLat proves the effectiveness of WOMDD-Reasnning in assisting real-world driving through supporting various language- involved downstream tasks.
Introducing Languages into Multipath++ Our next step is to introduce these encoded languages into each agent's 's fea- ture. For Multipath++, use MPA (Konev, 2022), an open- source Multipath++ implementation, our code base. Our language introduction module is inserted right after their "Other Agents History Encoder", and before their "MCG (multi-context gating) encoder". Our module is a cross- attention block, which takes each agent's history encoding as queries, letting them cross-attend to their corresponding language embeddings containing info about the interaction between the ego agent and that specific agent. In this way, the interaction information in Motion-LLaVA is introduced into the features of each agent.
6.2. Enhancing Trajectory Prediction with Motion-LLaVa Outputs
A Acknowledgements
This work was supported part by Berkeley DeepDrive, NSF Grant. 2235013, NSF Grant. 2235012, and the Mi- crosoft Accelerate Foundation Models Research Initiative.
Table Benefits of Motion-LLaVA Outputs for Prediction Task.
Model
min FDs
M
Multipath++ (Konev, 2022)
1.27
12.59
Motion-L.LVA Outputs
1.18
11.69
Impact Statement
A
-7.35%
-7.10%
Our work is of broad interest to the natural language pro- cessing (NLP) and autonomous driving communities. The proposed approach has no new ethical or social issues on its own, except those inherited from NLP or autonomous driving.
As shown in Table 6, we observe significant metric im- provements with language involved, which strongly sup- ports WOMD- Reasonnnasss ability to help downstream tasks like predictions. can also help to understand the driving behaviors, and to improve explainability of vehicle trajec-
9

---

WOMD-Reasoningg A Large-Seale Dataset for Interaction Reasoning in Driving
References
Cui, C., Ma, Y., Cao, X., Ye, W., and Wang, Z. Drive as you speak: Enabling human-like interaction with large language models in autonomous vehicles. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) Workshops, pp. 902-909, January 2024.
Anderson, P., He, X., Batra, D., Parikh, D., Lee, M., Ehsan, A., and Zitmick, C. L. Spice: Semantic propositional image caption evaluation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 382-398, 2016.
Deruyttere, T., Vandenhende, S., Grujicie, D., Van Gool, L., and Moens, M.-F. Talk2car: Taking control of your self-driving car. arXiv preprint arXiv:1909. 100388,99..088, 2019.
Arai, H., Miwa, K., Sasaki, K., Yamaguchi, Y., Watanabe, K., Aoki, S., and Yamamoto, I. Covla: Comprehensive sasion-lnguaae-acction dataset for autonomous driving. arXiv preprint arXiv:2408. 10845,, 2024.
Ding, X., Han, J., Xu, H., Liang, X., Zhang, W., and Li, X. Holistic autonomous driving understanding by bird's- eye-view injected multi-modal large models. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13668- 13677, June 2024a.
Bai, S., Chen, K., Liu, X., Wang. J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al. Qwen2.5-vl technical report. arXiv preprint mxiv:2552.13992, 2025.
Banerjee, S. and Lavie, A. Metcor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, 2005.
Ding, X., Han, J., Xu, H., Liang, X., Zhang, W., and Li, X. Holistic autonomous driving understanding by bird hard's-eye- view injected multi-modal large models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13668-13677, 2024b.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neclakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Ding, Z. and Zhao, H. Incorporating driving knowledge in deep learning based vehicle trajectory prediction: A survey. IEEE Transactions on Intelligent Vehicles, 8(8): 3996-4015, 2023, doi: 10. 12.1100//VV.222333666
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving, 2020.
Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Prad- han, S., Chai, Y., Sapp. B., Qi, C., Zhou, Y., Yang, Z., Chouard, A., Sun, P., Ngiam, J., Vasudevan, V., Mc- Cauley, A., Shlens, J., and Anguelov, D. Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset, 2021.
Chen, H., Suhr, A., Misra, D., Snavely, N., and Artzi, Y. Touchdown: Natural language navigation and spatial rea- soning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 12538-12547, 2019.
Fu, C., Lin, H., Wang, X., Zhang, Y.-F,, Shen, Y., Liu, X., Cao, H., Long, Z., Gao, H., Li, K., et al. Vita-1.5: To- wards gpt-4o level real-time vision and speech interaction. arXiv preprint (0tiv::5010019955 2025.
Chen, K., Ge, R., Qiu, H., AI-Rfou, R., Qi, C. R., Zhou, X., Yang, Z., Ettinger, S., Sun, P., Leng, Z., Baniodeh, M.. Bogun, I., Wang, W., Tan, M., and Anguelov, D. Womd-lidar. lidar: Raw sensor dataset benchmark for motion /aattps://arxx. wwwp:wiivoor/oooooooooooo abs/
Fu, D., Li, X., Wen, L., Dou, M., Cai, P., Shi, B., and Qiao, Y. Drive like human: Rethinking autonomous driving with large language models, 2023.
forecasting, 2024.
Gai, X., Zhou, C., Liu, J., Feng, Y., Wu, J., and Liu, Z. Medthink: Explaining medical visual question answering via multimodal decision decision-making rationale, 2024. URL https: / /ttp//axxvv//.//////////////.atv org/ abs / 2404 12372.
2304. 03834.
Chen, I.., Sinavski, O., Hünermann, J., Karnsund, A., Will- mott, A. J., Birch, D., Maund, D., and Shotton, J. Driving with Ims: Fusing object-level vector modality for ex- plainable autonomous driving, 2023.
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., Li, H., and Qiao, Y.
Llama-adapter v2: Parameter-efffcient visual instruction model. arXiv preprint arXiv:2304.15014 2023.
Chu, Z. Chen, J. Chen, Q. Yu, W. He, T., Wang, H., Peng. W., Liu, M., Qin, B., and Liu, T. A survey of chain of thought reasoning: Advances, frontiers and future. arXiv preprint X0v9 15402, 2023.
Greer, R. and Trivedi, M. Towards explainable, safe au- tonomous driving with language embeddings for novelty
10

---

WOMMD-Rasooinng A Large-Seale Datasct for Interaction Reasoning in Driving
identification and active learning: Framework and experi- mental analysis with real-world data sets. arXiv preprint W0Xi::2402.07320, 2024.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp. 74-81, 2004.
Han, W., Guo, D., Xu, C.-Z., and Shen, J. Dme-driver: Integrating human decision logic and 3d scene perception in autonomous driving. arXiv preprint 2:Xiv:2401003644 2024.
Liu, H., Li, C., Wu, Q., and Lec, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Ma, Y., Cui, C., Cao, X., Ye, W., Liu, P., Lu, J., Abdelraouf, A., Gupta, R., Han, K., Bera, A., Rehg, J. M., and Wang, Z. Lampilot: An open benchmark dataset for autonomous driving with language model programs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tem Recognition (CVPR), pp. 15141-15151, June 2024.
Inoue, Y. Yada, Y., Tanahashi, K., and Yamaguchi, Y. Nuscenes-man mqqa: Integrated evaluation of captions and qa for autonomous driving datasets using markup anno- tations. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision (WACV) Work- shops, pp. 930-938, January 2024.
Malla, S., Choi, C., Dwivedi, I., Choi, J. H., and Li, J. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1043-1052, 2023.
Jin, B., Liu, X., Zheng, Y., Li, P., Zhao, H., Zhang, T., Zheng, Y., Zhou, G., and Liu, J. Adapt: Action- aware driving caption transformer, 2023. URL https: //a //axxiv.. org/ abs/ 220.. 00673.
Mao, J., Qian, Y., Ye, J., Zhao, H., and Wang, Y. Gpt- driver: Learning to drive with gpt. 2023a. URL https:
Jin, B., Zheng, Y., Li, P., Li, W., Zheng, Y., Hu, S., Liu, X., Zhu, J., Yan, Z., Sun, H., al. Tod3cap: Towards 3d dense captioning in outdoor scenes. In European Conference on Computer Vision, pp. 367-384. Springer, 2024.
/ //arxiv. hrw. /2000/2311..mm 01415.
Mao, J., Ye, J., Qian, Y., Pavone, M., and Wang, Y. A language agent for autonomous driving, 2023b.
Nic, M., Peng, R., Wang, C., Cai, X., Han, J., Xu, H., and Zhang, L. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving. arXiv preprint arXiv:2312.03666 2023.
Kim, J., Rohrbach, A., Darrell, T., Canny, J., and Akata, Z. Textual explanations for self-driving vehicles, 2018.
Konev, S. Mpa: Multipath++ based architecture for motion prediction, 2022. URL hRtttt:///axxx orgg abs/ 2206. 10041.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji. S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M.. Belgum, J., Bello, I., Berdine, J., Bernadeut-Shaping G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R. Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Cur- rier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J.. Gu, S. S., Guo, Y., Hallacy, C.,, Han, J., Harris, J., He. Y., Heaton, M., Heidecke, J., Hesse. C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun H., Kaftan, T., Lukasz Kaiser, Kamali, A., Kanitscheider, L., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J.,
Kuo, Y.I..,, Huang, X., Barbu, A., McGill, S. G., Katz, B., Leonard, J. J., and Rosman, G. Trajectory prediction with linguistic representations, 2022.
Li, B., Wang, Y., Mao, J., Ivanovic, B., Veer, S., Leung, K., and Pavone, M. Driving everywhere with large language model policy adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14948 14958,,197,,.0 2024a.
Li, Q., Peng, Z., Feng, L., Zhang, Q., Xue, Z., and Zhou, B. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Li, Q., Peng, Z., Feng, L., Liu, Z., Duan, C., Mo, W., and Zhou, B. Scenarionet: Open- source platform for large- scale traffic scenario simulation and modeling. Advances in Neural Information Processing Systems, 2023.
Li, Y., Li, L., Wu, Z., Bing, Z., Xuanyuan, Z., Knoll, A. C., and Chen, L. Unstrprompt: Large language model prompt for driving in unstructured scenarios. IEEE Journal of Radio Frequency Identification, 2024b.

---

WOMD-Reasoninn A Large-Scale Dataset for Interaction Reasoning in Driving
Knight, M., Kokotajlo, D., Eukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosie, K., Krueger, G., Kuo, V., Lampe, M., Lan, L., Lee, T., Leike, J., Leung, J., Levy, D.,
training. OpenAl Blog, 2018. URL htthttts::/oppeaii com/ ccwp:///eaarrrrh//aannggeeeeappe
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M.. Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-lo-text transformer, 2023.
Li, C. M., Lim, R., Lin. M., Lin, S., Litwin, M., Lopez, T.,
Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y. Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P. McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Míshkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, Noh, H., Ouyang, L., Oeeefe C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P. Michael, Pokorny, Pokrass, M., Pong V. H., Powell, T., Power, A., Power, B., Prochl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt,, H., Schnurr, D., Schulman, J., Sel- sam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J.. Slama, K., Sohl, I., Sokolowsky, B., Song. Y., Stau- dacher, N., Such, F. P. Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, F.,, Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. F.C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weimmann, C., Welihinda, A., Welinder, P., Weng. J., Weng, L.., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S.. Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024.
Roclofs, R., Sun, L., Caine, B., Refaat, K. S., Sapp, B., Ettinger, S., and Chai, W. Causalagents: robustness benchmark for motion forecasting using causal relation- ships, 2022.
Roh, J., Paxton, C.,, Pronobis, A., Farhadi, A., and Fox, D. Conditional driving from natural language instructions. In Conference on Robot Learning, pp. 540-551. PMLR, 2020.
Sachdeva, E., Agarwal, N., Chundi, S., Roelofs, S., Li, J., Kochenderfer, M., Choi, C., and Dariush, B. Rank2tell: multimodal driving dataset for joint importance ranking and reasoning. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision (WACV), pp. 7513-7522, January 2024a.
Sachdeva, E., Agarwal, N., Chundi, S., Roelofs, S., Li, J., Kochenderfer, M., Choi, C., and Dariush, B. Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 7513-7522, January 2024b.
Seff, A., Cera, B., Chen, D., Ng. M., Zhou, A., Nayakanti, N., Refaat, K. S., Al-Rfou, R., and Sapp, B. Mo- tionlm: Multi-agent motion forecasting as language modeling, 2023. URL / /bs/ 2309 2309.16534. 16534.
Shao, H., Hu, Y., Wang, L., Waslander, L., Liu, Y., and Li, H. Lmdrive: Closed-loop end-to-end driving with large language models, 2023. URL https: /tt///axvvvoo/ orgg/ abs 22312.11222 07488.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. Bleu: method for automatic evaluation of machine transla- tion. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Luo, P., Geiger, A., and Li, H. Drivelm: Driving with graph visual question answering, 2023.
Park, S., Lee, M., Kang, J., Choi, H., Park, Y., Cho, J., Lee, A., and Kim, D. Vlaad: Vision and language assistant for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 980-987, 2024.
Sriram, N., Maniar, T., Kalyanasundaram, J., Gandhi, V., Bhowmick, B., and Krishna, K. M. Talk to the vehi- cle: Language conditioned autonomous navigation of self driving cars. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 5284-5290. IEEE, 2019.
Qian, T., Chen, J., Zhuo, I.., Jiao, Y., and Jiang, Y.GG. Nuscenes-qa: A multi-modal visual question answering benchmark for autonomous driving scenario, 2024.
Tian, R., Li, B., Weng, X., Chen, Y., Schmerling, E., Wang. Y., Ivanovic, B., and Pavone, M. Tokenize the world into object-level knowledge to address long-tail events in autonomous driving. arXiv preprint arXiv:2407.0095: 2024a.
Radford, A., Narasimhan, K., Salimans, T., and Sutskever, Improving language understanding by generative pre-
12

---

WOMD ROMDDReasoning: A Large-Seale Dataset for Interaction Reasoning in Driving
Tian, X., Gu, J., Li, B., Liu, Y., Wang. Y., Zhao, Z., Zhan, K., Jia, P., Lang. X., and Zhao, H. Drivevim: The conver- gence of autonomous driving and large vision-languee models. arXiv preprint arXiv:2402.12289, 2024b.
Xu, Z.,, Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.-Y. K., Li, Z-, and Zhao, H. Drivegpt4: Inter- pretable end-to-end autonomous driving via large lan- guage model, 2024b. URL https: httpp:///axxxv.org/// abs/2310. 01422.
Tonmoy, S. M. T. I., Zaman, S. M. M., Jain, V., Rani, A., Rawte, V., Chadha, A., and Das, A. A comprehensive survey of hallucination mitigation techniques in large language models, 2024. URL httph:ttp////..... /ttp://axxvvvoo org/ 0////240..011111
Yang. S., Liu, J., Zhang, R., Pan, M., Guo, Z., Li, X., Chen, Z., Gao, P., Li, H., Guo, Y., et al. Lidar-llm: Exploring the potential of large language models for 3d lidar un- derstanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 9247-9255, 2025.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13977. 2023.
Yang, Z., Jia, X., Li, H., and Yan, J. LIm4drive: A survey of large language models for autonomous driving. arXiv e-prints, pp. arXiv-2311, 2023.
Varadarajan, B., Hefny, A., Srivastava, A., Refaat, K. S., Nayakanti, N., Cornman, A., Chen, K., Douillard, B., Lam, C. P., Anguelov, D., and Sapp. Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction, 2021.
Yuan, J., Sun, S., Omeiza, D., Zhao, B., Newman, P., Kunze, L., and Gadd, M. Rag-driver: Generalisable driving explanations with retrieval-augmente in multi-modal large language model. arXiv preprint arXiv:2402. 10828, 2024.
in-context learning
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus- based image description evaluation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3566-4575. 2015.
Zhai Y., Tong, S., Li, X., Cai, M., Qu, Q., Lee, Y. J., and Ma, Y. Investigating the catastrophic forgetting in multimodal large language models, 2023, URL https: / //arxiv.org//// abs 220. 10313.
Wang, S., Yu, Z., Jiang, X., Lan, S., Shi, M., Chang, N., Kautz, J., Li, Y., and Alvarez, M. Omnidrive: A holis- tic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533 2024a.
Zhang, S., Huang, W., Gao, Z., Chen, H., and Lv, C. Wisead: Knowledge augmented end-to-end autonomous driving with dision-language model. arXiv preprint arXiv:4412.0995/ 2024.
Wang. T.-H. Maalouf, A., Xiao, W., Ban, Y., Amini, A., Rosman, G., Karaman, S., and Rus, D. Drive any- where: Generalizable end-to-end autonomous driving with multi-modal foundation models. In 2024 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 6687-6694, 2024b. doi: 10.1109//CRA71177.2222 10611590
Zhang, Y., Zhang, S., Liang, 7., Li, H. L.,, Wu, H., and Liu, Q. Dynamical driving interactions between human and mentalizing designedd autonomous vehicle. In 2022 IEEE International Conference on Development and Learning (ICDL), pp. 178-183. IEEE, 2022.
Zhao, X., Sun, J., and Wang, M. Measuring sociality in driving interaction. IEEE Transactions on Intelligent Transportation Systems, 2024.
Wang, W. Wang, L., Zhang, C., Liu, C., Sun, L., et al. Social interactions for autonomous driving: A review and perspectives. Foundations and Trends@ in Robotics, 10 (3-4):198-376, 2022.
Zhong, Z., Rempe, D., Chen, Y., Ivanovic, B., Cao, Y., Xu, D., Pavone, M., and Ray, B. Language-guin gugdde traffic simulation via scene-level diffusion. In Tan, J., Toussaint, M., and Darvish, K. (eds.), Proceedings of The 7th Con- ference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pp. 144-177, PMLR, 06- 09 Nov 2023. URL https: /ptppceeeeings...e/ee .mlr. press v22. zhong23on .tt.mml...m.m..mm.mmmm.
Wei, J., Wang, X., Schuurmans, D.,, Bosma, M., Xia, F., Chi, E.,, Le, Q. V., Zhou, D., et al. Mhain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Xu, H., Gao, Y., Yu, F., and Darrell, T. End-to-end learning of driving models from large-scale video datasets, 2017.
Zhou, Y., Huang, L., Bu, Q., Zeng, J., Li, T., Qiu, H., Zhu, H., Guo, M., Qiao, Y., and Li, H. Embod- ied understanding of driving scenarios. arXiv preprint WrXiv:2433004593. 2024.
Xu, Z., Zhang, Y., Xie, E., Zhao. Z., Guo, Y., Wong, K.- Y. K., Li, Z., and Zhao, H. Drivegpt4: Interpretable end-to-end autonomous driving via large language model, 2024a.
13

---

WOMD- A Large-Seale Datasct for Interaction Reasoning in Driving
A. Appendix
In this appendix, we first show the building process of WOMD-Reasoning in detail in A.I. Specifically, we show the construction of the interpreted language of the motion data with an example (A.1.1), and we revisit the detailed methods utilizing GPT to build the language dataset as well as providing the full set of prompts (A.1.2), Some examples of the vision provided by simulations are presented in A.2.
modal of WOMD-Reesonin
Then, for Motion-I.J VA, whose language output enhances trajectory prediction as shown in Section 6, we provide further implementation details. Specifically, in A.3, we describe how agent information from driving scenarios is incorporated into the input of Motion-LLaVA. The training hyperparameters used for the downstream trajectory prediction module are presented in A.4. The prompt used to obtain the GPT Score for evaluation is provided in A.5. We further provide a qualitative comparison in A.6 of the benchmarked multi-modal models listed in Table 4, and introduce in A.7 additional multi-modal models not included in the table. Examples of aggregated context (A.8) and factual description context (A.9) used in Chain-of- Thouuht interaction reasoning are also provided for reference. An ablation study on the fine-tuning strategy used in Motion-LLLaA is included in A0 along with an analysis of the data efficiency of (OMMD-Ressoning in helping Motion-LLaVA learn driving-related reasoning in A.
A.1. Details in Building WOMD-Reasoning
In this appendix, we show the details of how we build WOMD- dataset, including detailed explanations on each step we take, the example of the translated motion scenarios, and the GPT prompts. The full version of codes and prompts used herein can be found in the supplemental materials.
A. AUTOMATIC TRANSLATION OF DRIVING SCENARIOS INTO LANGUAGE DESCRIPTIONS
ChatGPT cannot effectively process raw data from motion datasets, as it is mainly trained on textual information rather than motion data. Therefore, we first translate the motion information into textual descriptions by a rule-based program. Accurate motion translation is critical to the quality of the dataset, as it serves as the only input observable to GPT. Ensuring thorough and well-structured translations is essential for GPT to perform accurate interaction analysis and reasoning.
To ensure enough information for analyzing interactions involving the ego agent, we translate the motion data into an ego-centric description of the traffic scenario. Specifically, before describing the ego agent status, we first narrate the map environment around the ego agent, including information about related intersections, lanes, stop signs, crosswalks, speed bumps, etc. The above information provides a comprehensive spatial guidance that facilitates describing the motion of either ego or other agents accurately. Then, we describe the status of the ego agent in the scenario, including its motion status (e.g.. the velocity, acceleration, etc.) and the positional status (e.g., the lane it occupies and its position related to the intersection center). Furthermore, the traffic light and signal information related to it are summarized in the description. Besides the ego agent, the depiction of other agents, including other vehicles, bicycles, and pedestrians, is also crucial for GPT to analyze the interactions. Therefore, we then describe every other agent in the scenario in the same way as we describe the ego agent. We additionally include the positional relation between the other agent and the ego one to support the identification of every agent's 's position.
Utilizing the methods above, can translate motion datasets into languages. Due to the limitations of computational resources of GPT, we only choose to translate those scenarios that have objects _ac_s_______________________ee__ cratttss____te_se__ee_eeeeeeeeee iii_interss labels in WOMD,
which indicates confirmed significant interactions happening in the scenario. We build two subsets of WOMD- with the same setting: the training set is built on the training set of WOMD, while the validation set is built on the interactive validation set of WOMD. In total, we translate 63k scenarios into language. An example of the translation can be found in Table
A.1.2. GPT-BASED INTERACTION ANALYSIS
With the language translation of WOMD input, we further design a set of prompts to utilize ChatGPT-4 (OpenAI et al., 2024) to build the Q&& dataset, taking advantage of its abilities in analyzing interactions. The prompts come with 4 main parts. (1) System Prompt, which lets the GPT know its role and the format of its input, i.e. the language translation. (2) Responsibility Prompt, which describes the responsibilities of the GPT, including the questions it should ask and answer, as well as the output format it should use. This will be talked about in detail in the next paragraph. (3) Rules Prompt, which contains global rules to guide the output and the analysis. (4) In-context Prompt, which provides a few numan-written
14

---

WOMD- RMMID-Reasoning A Large-Scale Dataset for Interaction Reasoning in Driving
Deseriptions on map environment:
The ego agent is heading towards intersection. The intersection center is 18.0 meters in front of the ego agent, and is 17.0 meters on the left of the ego agent. The intersection is 4 way intersection.
Deseriptions on ego agent: The ego agent is on the lane from the right, out of lanes. Its current speed is m/s. is accelerating. Traflic Light for the ego agent is red. The ego agent is approaching a crosswalk 3 meters ahead. Descriptions on surrounding (other) agents:
Surrounding agent # 0 vehicle. It is 23 meters in front of the ego agent, and is 17 meters on the left of the ego agent. It is heading right the ego agent. current speed 6 m/s. It accelcrating. is in the intersection. It is 29 meters away from the ego agent. It is approaching a crosswalk 3 meters ahead.
Surrounding agent # 2 is a vehicle. It is 11 meters on the right of the ego agent, and is meters behind the ego agent. It is heading left of the ego agent. It is not moving. It is on the same side of the intersection the ego agent. It 34 meters away from the intersection center.
Descriptions on agents after seconds:
The following is the description of the ego and surrounding agents after 3.0 seconds:
Surrounding agent # 0 will be meters front of the ego agent, and will he 5 meters the left of the ego agent. It will be heading right of the ego agent. Its speed will be m/s. will be departing from the interscction. Looking from the agent's current angle, it will be on the same side of the intersection. will be 18 meters away from the intersection center.
The ego agent will be 13 meters in front of the current place, and will be 2 meters on the right of the current place. It will be heading in the same direction the current moment. Its speed will be 3 m/s. It will be departing from the intersection. Looking from the agent's current angle, it will be the same side of the intersection. will be 20 meters away from the intersection center.
Surrounding agent 2 will be 13 meters on the right of the ego agent, and will be meters behind the ego agent, will be heading left of the ego agent. It will not be moving. Looking from the agent's current angle, it will be on the same side of the intersection. will be 34 meters away from the intersection center.
Table 7. An example of the language translation of motion data.
examples of input-output pairs to perform in-context learning. An example of each part of the prompts can be found in Table 8, 9, 10, and 11 respectively.
The questions in the responsibility part of the prompt decide what would be present the generated language Q&A dataset. To provide a comprehensive language dataset, and to present interaction analysis as accurately as possible, we organize the questions into chain to enable well-designed Chain-of-Thought (Wei et al., 2022; Chu al., 2023): First, ask questions about the map environment, the ego and other agents' motion. These questions would let the GPT provide Q&As useful for fine-tuning scene description agents while enhancing the GPT's understanding of the input information at the same time. Then, we ask questions about the interactions that occur in the observable future period. To include traffic rule-induced and intention-induca interactions, we ask the GPT to think about the interaction between each pair of agents by answering a sequence of questions: Q1: Are the two agents vehicles, and are they close to each other with traffic rules or signals governing their movements? If the answer is yes, we request that GPT analyze their yielding relations according to the traffic rules. Otherwise, we ask Q2: Are the two agents vehicle and pedestrian respectively with intention conflicts? If so, the vehicle generally should yield to the pedestrian for safety. If both questions do not help find the interaction relations, we come to Q3: Does the scene show patterns of certain intention-induced interaction? We provide descriptions of few common interactions like overtaking, following, etc. GPT can choose the best fit from these patterns to try to cover more interaction types. Based on this pipeline, we can provide interaction analysis with rich information on traffic rule-induced and human intention-inducee interactions. After that, we can finally wrap up the interactions between the ego and all other agents to provide the intention of the ego agent.
A.2. Examples of Simulated Vision Modality
Figure provides some whole examples of the simulated scenarios generated by MetaDrive (Li et al., 2022) simulator using ScenarioNet (Li et al., 2023) traffic scenario simulation platform.
15

---

WOMD- R A Large-Scale Dataset for Interaction Reasoning in Driving
System prompt for GPT:
You are an Al assistant to analyze real-world driving scenes. You provided with the detailed information (formated as: [start of the input] the detailed information [end of the input]) of the real-world driving scenario, which primarily includes information about the ego vehicle (denoted as 'Ego Agent'), ) other vehicles, eyclists pedestrians (denoted as 'Surrounding agent #u'), intersections and road lanes. Note that imput includes the current situation as well as the situation in a future moment for your reference. Unless are told to guess. only provide information agents other things not mentioned in the input does NOT exist, and you should NOT guess what would happen if they exist.
are certain according to the input. The input is complete. Any
Table 8. System prompt for GPT.
t=
t=
t=8s
t=
t=55
t=
Figure More visual examples of WOMD-Reesoning through simulations.
A.3. Processing Agent IDs for Motion-LLaVA
Careful processing of agent IDs is essential for enabling Motion-LLaVA to accurately distinguish between different agents. In our approach, these IDs are used purely as textual identifiers within the prompts, allowing the LLM to differentiate agents through natural language understanding. The prompt format is:
Ego agent: <motien>'n Agent #0: <motion>>l An... An Now, please answer: [Question]
where <motion> denotes the encoded motion data from each agent's perspective. The IDs assigned by range: #0-#99 for vehicles, #100-#199 for bicycles, and #200 #299 for pedestrians. We remap WOMD agent IDs these local ranges to prevent downstream models from overfitting to specific agents' behaviors. Importantly, these agent IDs do not encode any additional information beyond their role as identifiers.
A.4. Hyperparameters for Multipath++
In the experiment using languages from Motion-LLaVA outputs to assist vehicle trajectory predictions, we use batch size of 128 and learning rate of le - for Multipath++. As language is not available for every agent, we use "No capt i ons ava i lable e for this agent. for those agents without language inputs. The 15 -r__ xxl model is used as the language encoder.
A.5. Prompts for evaluating GPT Score
We employ the OpenAl GPT-4-turbo model (OpenAl et al., 2024) to evaluate the performance of all models benchmarked on the WOMD- MoaD-Rncsoning.... The GPT Score prompt presented in Table 12. Notably, we design our GPT Score prompt using a similar template the one from (Chen al., 2023).
A.6. Qualitative Comparison of Benchmarked Multi-modal Models on Answering Questions in WOMD-Reasoning
In this appendix, we present examples of answers to questions in WOMD- MoMD-Reasoning generated by the benchmarked multi-modal models listed Table The selected scenario corresponds to Figure 7, which already showeases answers from
16

---

WOMD- A Large-Seale Dataset for Interaction Reasoning in Driving
Motion-LLaVA, Additionally, we provide results for both the Non-Fine-tuned (i.e., off-the-shelf) and Fine-tuned LLaVA models, both using BEV plots vision input.
For the Non-Fine-tuned LLaVA, we perform prompt engineering to make the answers coherent. However, the low GPT scores indicate that it remains challenging for LLaVA to directly comprehend the scenario:
Q: Can you describe the type of intersection present in the current driving scenario?
A: The type of intersection present in the current driving scenario is a four-way intersection.
GPT Score: 0. Incorrectly identifies the type of intersection.
Q: What is the ego agent's current motion status?
A: The ego agent is currently stopped at the intersection.
GPT Score: 0. Incorrectly describes the motion status of the ego agent.
Q: Where is surrounding agent #5 in relation to the ego agent?
A: Surrounding agent #5 is located to the right of the ego agent.
GPT Score: 0. Incorrect location surrounding agent #5 in relation to the ego agent.
Q: What kind of interaction will occur between the ego agent and surrounding agent #5?
A: The ego agent will pass by the surrounding agent #5 at the intersection. GPT Score: 0. Incorrect prediction of the interaction between the ego agent and surrounding agent #5.
Quantitatively, the Non-Fine-tuned LLaVA achieves a remarkably low GPT Score of 2.31 across all questions in the validation set, demonstrating difficulty in understanding the driving scenarios based solely on the prior knowledge of the Vision-Languaee Model (VLM). For comparison, we evaluate the performance of the Fine-tuned LLaVA with vision input under the same scenario. The fine-tuned model achieves much more reasonable results, illustrated below:
Q: Can you describe the type of intersection present in the current driving scenario? A: The intersection is way intersection.
GPT Score: 10. The student's answer matches the correct answer exactly.
Q: What is the ego agent's current motion status?
A: The ego agent is accelerating.
GPT Score: 10. The student's answer matches the correct answer exactly.
Q: Where is surrounding agent #5 in relation to the ego agent?
A: It is 14 meters the right and 10 meters in front of the ego agent.
GPT Score: 0. The student's answer incorrectly describes the position of surrounding agent #5,
Q: What kind of interaction will occur between the ego agent and surrounding agent #5?
A: Surrounding agent #5 will follow the ego agent they are both heading towards the intersection and are on the same side.
GPT Score: 5. The student's answer is partially correct but does not specify that agent #5 is behind the ego agent.
The results indicate that fine-tuning significantly improves LLaVA's ability to interpret driving scenarios. This improvement can be attributed to the inclusion of our WOMD-Reasoning dataset. However, despite these enhancements, notable under- performance persists in questions related to Other Agent Status and Interaction & Intention, especially when compared to the results in Figure 7, which are generated using Motion-LLaVa We attribute this limitation to potential information loss during the conversion of vectorized motion data into BEV plot with a scalar bar.
A.7. Additional Baseline Language Models on WOMD-Reasoning
Our main results Table 4 primarily evaluate different variants of LLaVA (Liu et al., 2024) to demonstrate the effectiveness of WOMD-Reasoning, For clarity, we explain the categories of questions assessed. The questions are grouped into two main types: factual questions and interaction reasoning questions.
Factual questions:
17

---

This context corresponds to the WOMD- A Large-Seale Dataset for Interaction Reasoning in Driving
Environment--elated::
Questions about the scenario environment, such the existence and type of intersections, the number of lanes the ego vehicle's 's side, and the presence location of crosswalks and stop signs.
Ego agent-related: Questions regarding the ego vehicle's properties, including speed, motion status, lane position, and direction.
Surrounding agents-related:
Questions about other agents in the scenario, covering their type, speed, motion status, and position relative to the ego vehicle and the intersection.
Interaction reasoning questions:
Questions about the interactions that will occur between each surrounding agent and the ego agent.
Questions about the overall intention of the ego agent in the given scenario.
We further evaluate additional language models (LMs) to confirm the effectiveness of WOMD-Reasomee Specifically, we assess the answer quality of several baseline models, including LLaMA-Adapter v2.1 (Gao et al., 2023), VITA-1.5 (Fu et al., 2025), and Qwen2.5-Ve. VL (Bai et al., 2025) (all 7B versions), both before and after fine-tuning on WOMD BoMD-Reasoning. The results are summarized in Table 13.
We observe that without fine-tuning, all models can hardly answer driving-related questions, supporting the motivation of building WOMD-Reasoninn benefit from the fine-tuning. Besides, we find that fine-tuned our Motion-LLa works better than fine-tuned LLaVA or LLaMA-adapter, proving its well-designed structure in utilizing WOMD-Reasoning information.
We then fine-tune LLaMA-Adapter on WOMD-Reasoninn which also significantly
A.8. Example Aggregated Context for Chain-of-Thoughh Reasoning
We present an example of an aggregated context, derived from the answers generated by Motion-LLaVA for factual questions, which is used by Motion-LLaVA to infer answers to reasoning-based questions. CoT configuration in Table 14:
The ego agent turning left. Its current speed m/s. is accelerating.
Surrounding agent #7 is vehicle. It 26 meters on the left of the ego agent and 8 meters in front. is heading right of the ego agent. Its current speed is m/s. It is decelerating.
The blue and orange sections contain answers to questions appearing in WOMD-Reesoniin: blue addresses the Road Environment and Ego Agent Status, while orange covers Other Agent Status.
To generate this context, we prompt the Motion-LLat VA model with relevant questions and then concatenate the answers to create a complete context.
A.9. Example Factual Description Context for Chain-of-Thought Reasoning
We provide an example of a factual description context, generated using our motion data translation program as part of constructing WOMD-RResoning as described Appendix 1.1.1. This context used by Motion-L.LaVe to infer answers to reasoning-based questions and corresponds to the CoT w/ Facts Contexts configuration in Table 14. Notably, these contexts exclude future information compared to the translated motion data, thereby avoiding any information leakage:
The ego agent is in intersection. The intersection center is 6.0 meters in front of the ego agent, and is 0.0 meters on the left of the ego agent. The intersection is a 4 way intersection. It is turning left. It is exiting the intersection. current speed m/s. is accelerating. Traffic Light for the ego agent is green. The ego agent approaching crosswalk meters ahead.
Surrounding agent #7 is vehicle. is meters on the left of the ego agent, and is meters in front of the ego agent. It is heading right of the ego agent. Its current speed is 4 m/s. is decelerating. It is heading towards the intersection. It is on the left of the intersection. Traffic Light for it is red 14.5 meters ahead. It is 26 meters away from the intersection center. It is approaching a crosswalk 4 meters ahead.
Similar to the aggregated context in Appendix A.8, the blue and orange sections contain answers to questions appearing in WOMDDRResoninn: blue addresses the Road Environment and Ego Agent Status, while orange covers Other Agent Status.
18

---

WOMDD-easoning: A Large-Seale Dataset for Interaction Reasoning in Driving
A.10. Ablations on Motion-I.LaVA
Chain-of-Thoughh Inference. To address the hallucinated response problem (Gai et al., 2024; Tonmoy et al., 2024; Sima al., 2023) in reasoning interactions, use Chain-of-Thought (CoT) approach to provide Motion-LLaVA with structured context for more accurate reasoning. Table 14 compares model performance with and without CoT on interaction and intention analysis. We see that with CoT, the model performs better in GPT Scores. We further show that if our motion data translation program A.I.I is used during inference to provide contexts, i.e., with factual description context (an example can be found in Appendix A.9) and CoT. the performance can be significantly improved, proving the potential of the CoT inference strategy.
Influence of Pre-training for the Motion Vector Encoder. To establish a pre-training strategy for the motion vector encoder, examine factual question answering across Motion-LLa VA with four pre-training configurations: no pre-training, autoencoder-style pre-training, motion-caption pre-training (Chen et al., 2023), and pre-training with motion prediction task. All configurations shared the same encoder structure. In the motion-caption pre-training, we pre-train all but LLM weights in Motion- LL,,,, by penalizing errors in captioning the motion. For autoencoder-style pre-training and pre-training with a motion prediction task, we retain only the encoder after pre-training, to function as the motion vector encoder for Motion-LLaVA.
The performances of models with these pre-training settings are presented in Table 15. Generally, pre-training motion vector encoders improve the response quality. Specifically, pre-training with motion prediction tasks outperforms the other two approaches. We attribute this to the close alignment between WOMD-Reasoning 's question categories and the information needed in motion prediction tasks. As result, pre-training with the motion prediction task achieves the highest GPT Score and overall accuracy, especially in numerical precision reflected by Acc1.0-
Motion-LLa VA Fine-tuning Strategy. The original LLaVA (Liu et al., 2024) training pipeline consists of two stages: a projector alignment stage (also known as pre-training) align vision and language features, followed by fine-tuning stage to enhance instruction-followinn capabilities. Throughout this process, the vision encoder remains frozen. Since Motion-LLaVA builds on the architecture of the original LLaVA (Liu et al., 2024), it seems reasonable to adopt a similar training approach for the WOMMD-Ressonie task.
However, based on extensive experiments with fine-tuning strategies in Table 16, we find that incorporating projector aligmment stage or keeping the motion vector encoder frozen throughout, which are practices employed in LLaVA (Liu et al., 2024), resulted in degraded performance on WOMDD-Ressnning. Consequently, our Motion-L_LaV. adopts single-stage fine-tuning approach, where the motion vector encoder is unfrozen.
A.11. WOMD-Reasoning Dataset Efficiency Analysis
To assess the data efficiency of WOMDDReasoniig. we conduct scaling studies to analyze how model performance improves as the used dataset's size increases, from 0.5M IM and up 3M Q&A pairs (full dataset).
data involved, the fine-tuned model first gains the ability answer factual questions, which hits a good GPT score with only 0.5M data. As the dataset grows, the model gradually learns to answer interaction reasoning questions. These justify the size of WOMD- as well as its data efficiency in helping models to answer bierarchical driving-related questions.
In Table 17, find that with more WOMD-Reasonin
19

---

WOMD-Rccsoninn A Large-Scale Dataset for Interaction Reasoning in Driving
Responsibility Prompt GPT:
You are responsible for the following responsibilities:
<First>, print "[Env QA]" Generate Q& A on all the scemario description inputs about the scenario environment. The Q& must be the following format: "[QI[Question" content] [ Answer content]" The Q&A should cover ALI. information about the CURRENT moment in the input description. No information in the provided future moment should be used. Your questions should include all the questions the following categories except answer not mentioned the input: (1) About the intersection: its existence? Its type? (2) About the lanes: How many lanes on ego's side? (3) About stop signs: its existence? where is it? which direction is it for?
<Second>, print "[Ego QA]". Generate Q&A A on scenario description inputs about the ego agent. The Q&A must be in the following format: "[Q][Question content] [A][wrr content]" The Q&A should cover ALL information about the CURRENT moment in the input description. No information in the provided future moment should be used. Your questions should include all the questions in the following categories: (1) About its motion: its speed? its motion status? (2) About its position: its position? its lane? (3) About direction: its facing direction? turning direction?
<Third>, print "[Sur QAI" and generate Q&A on all the scenario description inputs about surrounding agents. The Q&A must be the following format: "[QI[Question content] [A] Answer content]" The Q&& should cover ALL information about the CURRENT moment the input description. No information in the provided future moment should be used. For each surrounding agent, your questions should include all the questions in the following categories except the answer is not mentioned in the input: (1) About itself: its type? (2) About its motion: its speed? its motion status? (3) About its position: its position to the ego agent? its position the intersection? its lane?
<Fourth>, start new paragraph and print "[Int QA]". For each surrounding agent, start new paragraph and provide one Q&A for the interactions between it and the ego agent. The Q&A must be in the following format: "[Q][Question content] [A][Answer content]" Here [Question content] should have the meaning "What interactions will happen between surrounding agent #n and the ego agent?" without any additional information like "considering what" or "given what", but you must use diverse ways to put it. (i.e. ask in different ways for different surrounding agents) When providing [Answer content]. you should use the descriptions for both the current moment and the future moment, and analyze based on them. Pay attention to the special situations where you should consider that there is interaction:
<Fifth>, for the ego agent, start new paragraph, print "[Intention]", and provide one Q&A for the intention of the ego agent, The Q&A must be in the following format: "[Q][Question content] [Answer content]" Here [Question content] should have the meaning "What will be the intention of the ego agent?", but you must diverse ways put it. When providing [Answer content], and think about following questions in the time period from now to the provided future moment based given description: (1) What is the ego agent's 's intention if no other agents exist? (2) If applicable, what actions does the ego agent take to respond to traffic lights, stop signs, crosswalks, speed bumps and traffic rules? If some traffic controls are not applicable, do not output that. (3) In each interaction involving the ego agent, what actions do the ego and surrounding agent take respond to each other? If one agent does have interaction with the ego agent, exclude in your answer.
Remember to use diverse ways to put the [Answer content]. Do not include any numbers regarding speeds positions in your [Answer content].
Table 9, Responsibility prompt for GPT.
20

---

HOMDD-Rasooiing A Large-Scale Datasct for Interaction Reasoning in Driving
Global rules prompt for GPT:
Here some global rules that your need follow:
Please provide summary and analysis the driving scenario according to your responsibillites.
2. Never use bullet points. Be concise and compact.
3. Give natural language, no parenthesis.
4. Give the results in the order of the responsibilitics. Always start new paragraph for each responsibility, but one responsibility can take several paragraphs. Do not combine multiple responsibilities together. Finish one responsibility before starting another.
5. not show the intersection number, number or ego agent's number. The surrounding agents' numbers need to be shown.
6. Output in full. Never omit anything regardless of any reason. You are allowed to give very long outputs.
7. Remember that all the agents and traffic controls in the scene have been provided, so if any agents or traffic controls mentioned in the
they do not exist. You don't need consider them give any output about them. Therefore, do not output words like "not mentioned" because this inaccurate.
8. The traffic light info is
current moment only. The future traffic light info unknown.
9. Any action caused by traffic controls like lights, stop signs or crosswalks may also constitute yielding interactions. Traffic controls are only indication of right-of-way, but any actions that are caused by the traffic controls are still considered interactions. For example, car is waiting for the red light stop sign while someone other cross the intersection with green light, the car still considered yiclding the other
When analyzing future, talk about the period between the current future moment provided, not the two moments.
Table Rule prompt for GPT.
21

---

WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving
Context prompt for GPT:
Here one given example that you can refer to to help you better understand you responsibility <Fourth> <Fifth>; For the input case of:
[start of input] The ego agent is in intersection. The intersection center is 10.0 meters in front of the ego agent, and is 1.0 meters the left the ego agent. The intersection is 4-way intersection.
exiting the intersection. 's current speed 6 m/s. accelerating. The ego agent is approaching stop sign meters ahead. There are 4 stop signs in the intersection. The ego agent is approaching crosswalk meters ahead, Surrounding agent # is vehicle. It 4 meters the left of the ego agent, and is 2 meters behind the ego agent. It heading the opposite direction the ego agent. Its current speed is m/s. is accelerating. It is departing from the intersection. is on the same side of intersection the ego agent. is 12 meters away from intersection center.
is going straight.
[end of input]
you may give the following analysis:
[Int QA]
[Q] What kind of interaction can be expected between the ego agent and surrounding agent #0 given their positions and directions?
[A] Surrounding agent #0 will have interaction with the ego agent their intended paths have conflicts.
[End Int QA]
[Intention]
[Q] What will the ego agent aim do in the upcoming moments? [A] The ego agent intends continue exiting the intersection. There a stop sign on its has already stopped and started afterwards. Surrounding agent #1 will yield to it so the ego agent does not need to respond. Surrounding agent #3 will follow it but the ego agent does not need to respond as well. Therefore, the ego agent will continue in the intersection and finish its left turn.
[End Intention]
Here is another given example that can refer to to help you better understand you responsibility <Fourth> and <Fifth>: [start of input]
The ego agent is heading towards intersection. The intersection center is 28.0 meters in front of the ego agent, and 3.0 meters on left the ego agent. The intersection is way intersection.
The ego agent is on the 2 lane from the left, out 3 lanes. It current speed is m/s. is decelerating. Traffic Light for the ego agent is red 9.6 meters ahead. The ego agent is approaching a crosswalk meters ahead.
Surrounding agent # 0 vehicle. It 29 meters in front of the ego agent, and is 4 meters the right of the ego agent. is heading left of the ego agent. current speed 6 m/s. is accelerating. It in the intersection. is 29 meters away from the ego agent. The ego agent at a speed bump. ... [end of input)
you may give the following analysis:
[Int QA]
[Q] What kind of interaction can be expected between the ego agent and surrounding agent #0?
[A] The agent will yield to surrounding agent #0 because they are both approaching the intersection and the traffic light red for the ego agent, indicating that the ego agent must stop.
[End Int QA]
[Intention]
[Q] What will the ego agent aim to do in the upcoming moments? [A] The ego agent intends enter the intersection, but it has stop the red traffic light and wait for it change green before proceeding through
intersection. will also yield to surrounding agents #0 who is already intersection. Therefore, the ego agent will wait until the lights turn green and the surrounding agent #0 passes before entering the intersection. [End Intention]
Table 11. In-context prompt for GPT.
22

---

WOMD-Reasoning A Large-Scale Dataset for Interaction Reasoning in Driving
would like to request your feedback on the performance an assistant in response to the user question displayed above. The user asks the question on observing a driving scenario. The correct answer included for your reference. ### Scoring rules:
Your scores for each answer should be between (worst) and 10 (best).
the answer mostly incorrect includes important relevant information, such identifying correct agents or actions, give a score less than 3.
Give intermediate scores for partially correct answers or numbers are close. Only give a score of flawless answers.
Think critically carefully. importantly, check the answer factual correctness, reward partial credit where applicable.
#GG
Grading process each the xxx questions, provide one line assessment the student's answer the format:
<4 n. <short one sentence assessment>. Score: <score 0-10>..
A few examples of what a good assessment might look like:
Acknowledged question unrelated to driving and attempted to answer. Score: 10.
Doesn't answer question has all information necessary. Score:
Incorrectly stated there vehicles around even though there is one. Score:
4. Unable answer question given information available. Score: 10.
5. Give 12 m/s or m or percentage for the correct answer of 13 (within 10% deviation). Score: 9.
Give 15 m/s or m or percentage for the correct answer of 13 (within 20% deviation). Score: 7.
Give 18 m/s or m or percentage for correct answer of (within 40% deviation). Score: 5.
Give 2 m/s or percentage for the correct of 13. Score: 0.
Table 12, Prompt for GPT Score.
Table 13, Additional language model baselines evaluated on WOMD-Reasoning We report the answer quality of several vanilla and FrOMD-Reasoning -in-ooniiin--i--uneeed fine-tuned models, including LLaMA-Adapter, ViTA, and Qwen2.5-VL (all 7B versions), to further validate the effectiveness of WOMD-Reasoning
Model
Fine-tuned WOMD-Rease WOMDD-Rasooiig
ROUGE (f)
BLEU (f)
METEOR
CIDEr (f)
SPICE (1)
GPT Score (1)
LLaVA
0.512
0.211
0.275
1.36
0.455
2.31
LLaMA-Adaptory2) v2.1
0.413
0.174
0.235
0.91
0.372
1.62
VITA-1.5
0.278
0.121
0.190
0.40
0.227
1.77
Qwen2.5-VL
0.384
0.156
0.247
0.62
0.379
2.36
LLaVA
0.779
0.581
0.439
5.51
0.735
6.88
LLaMA-Adapter
0.722
0.470
0.375
4.72
0.691
5.14
Motion-LLaVA (Ours)
0.792
0.616
0.449
5.69
0.744
Table 14. Effeets of CoT in answering reasoning- related questions.
Reasoning
ROUGE (t)
BLEU
METEOR
CIDEr
SPICE
Score
Inference
Method
(?)
(1)
(7)
(f)
Direct
0,616
0.474
0.365
2.54
0.578
6.59
CoT
0.614
0.474
0.366
2.52
CoT w/
Facts Contexts
23

---

WOMD-Rccsoninn A Large-Scale Dataset for Interaction Reasoning in Driving
Table 15. Benefits of using pre-trained motion vector encoder for fine-tuning multi-modal models WOMD- for factual
Q&As.
Motion Vector Encoder Pre-training
ROUGE (1)
BLEU (1)
METEOR (1)
CIDER (1)
SPICE (1)
Aceyo (T)
Median Error())
GPT Score (*)
None Motion Caption
0.814
0.684
0.774
30.0%
2,00
6.79
0.824
0.711
0.498
6.05
0.775
42.6%
2.24
6.97
AutoEncoder
0.837
0.727
0.516
6.35
0,803
47.3%
1.41
7.07
Motion Prediction (Ours)
0.840
0.736
0,516
6.35
0.794
56.2%
1,00
7.09
Table 16. Refinement of Motion-LLaVA training strategy.
Multi-modal Fine-tuning
Projector Alignment
Encoder Freezing
ROUGE (1)
BLEU
METEOR
CIDEr
SPICE
Ace1o (1)
Median (
GPT Score
(t)
(t)
(1)
(1)
(1)
Strategy
LLaVA
0,821
0.678
0.484
5.82
0.771
2.3%
12.00
6.17
Projector Alignment
x
/
0.821
0.697
0.492
5.96
0.782
2.3%
8.06
6.49
Encoder Freezing (Ours)
0.840
0.736
0.516
6.35
0.794
56.2%
1.00
7.09
Table Performance Motion-LLLVA with different WOMD-Rcasoning dataset sizes on factual interaction reasoning questions. Results confirm the data efficiency of
in improving driving-related reasoning.
Factual Questions
Interaction Reasoning Questions
Dataset Size
ROUGE
BLEU
METEOR
CIDEr
SPICE
GPT
ROUGE
BLEU METEOR
CIDEr
SPICE
GPT
0.5M
0,806
0.683
0.477
5.77
0.760
6.69
0.562
0.410
0.339
1.94
0.513
5.67
IM
0.818
0.701
0.491
5.90
0.773
6.74
0.589
0.447
0.354
0.547
6.36
3M
0.840
0.736
0,516
6.35
0.794
7.09
0.614
0.474
0.366
2.52
0.571
6.76
24


Reviews:
[{"review_id": "AalPTLtiO9", "reviewer": "Reviewer_BaWk", "summary": "This paper introduces the Waymo Open Motion Dataset-Reasoning (WOMD-Reasoning), a comprehensive question-and-answer dataset designed to articulate and assess the interactions prompted by traffic rules within driving scenarios. To demonstrate the utility of WOMD-Reasoning, the paper proposes Motion-LLaVA, a motion language model specifically fine-tuned using this dataset. The quantitative and qualitative evaluations confirms the dataset's quality and its usefulness in autonomous driving.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments and insightful suggestions, and we address your comments and concerns below.\n\n> Q1. More LM baselines.\n\nWe are grateful for the suggestion. While our main goal for fine-tuning Motion-LLaVA is to show the effectiveness of WOMD-Reasoning, we agree that testing and fine-tuning more baselines would make our claims more reliable. Therefore, we test the quality of answers provided by a few vanilla and WOMD-Reasoning fine-tuned baselines, including LLaMA-Adapter, VITA and Qwen, all using 7B version. Results are shown below:\n\n| Model | Fine-tuned on WOMD-Reasoning | ROUGE (↑) | BLEU (↑) | METEOR (↑) | CIDEr (↑) | SPICE (↑) | GPT Score (↑) |\n|-|-|-|-|-|-|-|-|\n| LLaVA | ❌ | 0.512 | 0.211 | 0.275 | 1.36 | 0.455 | 2.31 |\n| LLaMA-Adapter v2.1 | ❌ | 0.413 | 0.174 | 0.235 | 0.91 | 0.372 | 1.62 |\n| VITA-1.5 | ❌ | 0.278 | 0.121 | 0.190 | 0.40 | 0.227 | 1.77 |\n| Qwen2.5-VL | ❌ | 0.384 | 0.156 | 0.247 | 0.62 | 0.379 | 2.36 | \n| LLaVA | ✅ | 0.779 | 0.581 | 0.439 | 5.51 | 0.735 | 6.88 |\n| LLaMA-Adapter v2.1 | ✅ | 0.722 | 0.470 | 0.375 | 4.72   | 0.691 | 5.14 |\n| Motion-LLaVA (Ours) | ✅ | **0.792** | **0.616** | **0.449** | **5.69** | **0.744** | **7.02** |\n\nWe observe that w/o fine-tuning, all models can hardly answer driving-related questions, supporting the motivation of building WOMD-Reasoning. We then fine-tune LLaMA-Adapter on WOMD-Reasoning, which also significantly benefit from the fine-tuning. Due to limited resources, fine-tuning other baselines are still ongoing. Besides, we find that fine-tuned Motion-LLaVA works better than fine-tuned LLaVA or LLaMA-adapter, proving its well-designed structure in utilizing WOMD-Reasoning info.\n\n> Q2. Downstream Applications\n\nWe thank the reviewer for this suggestion. To justify WOMD-Reasoning's ability in downstream tasks, we perform a trajectory prediction experiment to see the influence of using language outputs of Motion-LLaVA fine-tuned on WOMD-Reasoning. Multipath++ is used as the trajectory prediction baseline. As the interaction part is the most significant part in our dataset, we introduce the interaction part of language outputs into Multipath++, by cross-attending the T5 embedding of them with corresponding agents' features. Each experiment is run with 3 seeds for reliability. Averaged results are shown below,\n\n| Model | minFDE_6 (↓) | MR_6 (↓) |\n|-|-|-|\n| Multipath++ | 1.27 | 12.59% |\n| Multipath++ w/ Language | **1.18** | **11.69%** |\n| relative &Delta; | -7.35% | -7.10% |\n\nWe observe a significant boost from language outputs, which strongly supports WOMD-Reasoning's ability to help downstream tasks like predictions. It can also help understanding the explainability of driving behaviors.\n\n> Q3. More comparisons.\n\nWe thank the reviewers for these important papers to compare to! We will cite and compare to them, and we list a brief comparison here in our Table 1's form:\n\n| Dataset | Data Source | Total Scenes | Total Q&As | Interaction Q&As | Distance-induced | Traffic Rule-induced | Human Intention-induced | Scene Descriptions | Motion Prediction | Motion Planning |\n|-|-|-|-|-|-|-|-|-|-|-|\n| OmniDrive | nuScenes | <1k | N/A | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |\n| NuInstruct | nuScenes | 850   | 91k | <46k | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |\n| nuCaption | nuScenes | <1k | 420k | ~140k | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ |\n| Rank2Tell | Rank2Tell | 116 | N/A | N/A | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |\n| Tod3Cap | nuScenes | 850 | ~2,300k   | 0 | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ |\n| Ours | WOMD | **63k** | **2,940k** | **409k** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |\n\nThe stats further support that WOMD-Reasoning is the largest language Q&A dataset for driving by its release, with uniqueness in analyzing traffic rule-induced interactions.\n\n> Q4. The method part applies a motion encoder to replace the image encoder in LLaVA. Is there any adaptation?\n\nWe appreciate the question on the design of Motion-LLaVA. Yes, we thoroughly adapt LLaVA to better encode motion info for answering driving-related questions, which are discussed in Appendix A.7. Besides introducing a motion encoder from Multipath++ to replace the vision encoder in LLaVA, we have also made following adaptations:\n1) We use the encoder of a *pre-trained* Multipath++ as motion encoder. Motion prediction pre-training ensures the quality of the feature extracted from motion data, which is ablated in our Table 13.\n2) We remove the projector alignment stage in LLaVA, and keep the motion encoder unfrozen in fine-tuning, to help the model better encode info for answering driving-related questions. The benefit of this is ablated in Table 14.\n3) In Motion-LLaVA, we design a prompt for feeding motion embeddings from the Multipath++ encoder into LLM. The format is: `Ego agent: <motion>\\nAgent #0: <motion>\\n...\\nNow, please answer: {Question}`. Here, `<motion>` represents the encoded motion data from each agent's ego-centric viewpoint. The Agent IDs are randomly assigned and serve only to differentiate agents in downstream Q&As.", "claims_and_evidence": "* This paper proposes a large-scale dataset with 3 million Q&A pairs centered on interaction reasoning in driving, which is valuable for the community if released.", "methods_and_evaluation_criteria": "* This paper introduces Motion-LLaVA that provides interaction prediction for driving scenarios. The model utilizes a motion prediction model as a motion  encoder, followed by a LLaVA that generates language outputs.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "* There are no comparison with baselines other than LLaVA in the manuscript. Some multi-modal VLMs could also achieve this, like Qwen[1], VITA[2], or llama-adapter[3]. As a dataset paper, the reviewer believes a through study of the baselines should be conducted.\n\n[1] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., ... & Zhu, T. (2023). Qwen technical report. arXiv preprint arXiv:2309.16609.\n\n[2] Fu, C., Lin, H., Wang, X., Zhang, Y. F., Shen, Y., Liu, X., ... & He, R. (2025). Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957.\n\n[3] Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., ... & Qiao, Y. (2023). Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010.", "relation_to_broader_scientific_literature": "* As a language dataset designed for driving setting, it is essential to demonstrate its impact on downstream autonomous driving tasks like perception, motion prediction or trajectory prediction planning, and that's most of the existing research focusing on, like DriveLM[9] or Hint-ad[10]. Thus I recommend the authors conduct some experiments for the application of the dataset.\n\n[9] Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., ... & Li, H. (2024, September). Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision (pp. 256-274). Cham: Springer Nature Switzerland.\n\n[10] Ding, K., Chen, B., Su, Y., Gao, H. A., Jin, B., Sima, C., ... & Zhao, H. (2024). Hint-ad: Holistically aligned interpretability in end-to-end autonomous driving. arXiv preprint arXiv:2409.06702.", "essential_references_not_discussed": "* Table. 1 lists previous real-world language datasets for driving. However, some datasets like OmniDrive[4], NuInstruct[5], nuCaption[6], rank2tell[7], Tod3Cap[8] are not included.\n\n[4] Wang, S., Yu, Z., Jiang, X., Lan, S., Shi, M., Chang, N., ... & Alvarez, J. M. (2024). Omnidrive: A holistic llm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.01533.\n\n[5] Ding, X., Han, J., Xu, H., Liang, X., Zhang, W., & Li, X. (2024). Holistic autonomous driving understanding by bird's-eye-view injected multi-modal large models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13668-13677).\n\n[6] Yang, S., Liu, J., Zhang, R., Pan, M., Guo, Z., Li, X., ... & Zhang, S. (2023). Lidar-llm: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint arXiv:2312.14074.\n\n[7] Sachdeva, E., Agarwal, N., Chundi, S., Roelofs, S., Li, J., Kochenderfer, M., ... & Dariush, B. (2024). Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 7513-7522).\n\n[8] Jin, B., Zheng, Y., Li, P., Li, W., Zheng, Y., Hu, S., ... & Zhao, H. (2024, September). Tod3cap: Towards 3d dense captioning in outdoor scenes. In European Conference on Computer Vision (pp. 367-384). Cham: Springer Nature Switzerland.", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "* The method part primarily applies a motion encoder to replace the image encoder in LLaVA. Is there any adaptation specifically designed for the setting?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bL33f6EM2D", "reviewer": "Reviewer_TMNS", "summary": "The paper introduces WOMD-Reasoning, a large-scale dataset designed for interaction reasoning in autonomous driving, built upon WOMD. The dataset addresses a critical gap in understanding traffic rule-induced and human intention-induced interactions, which are often overlooked in existing driving datasets that primarily focus on proximity-based interactions. WOMD-Reasoning contains 3 million Q&A pairs spanning scene descriptions, motion predictions, and planning tasks for autonomous driving. To validate its effectiveness, the authors develop Motion-LLaVA, a motion-language model fine-tuned on WOMD-Reasoning, which demonstrates improved performance in interaction prediction and traffic rule-compliant planning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments and insightful suggestions. Please see our responses below:\n\n> Q1. Dataset accuracy: Eval criteria, diversity and annotation reliability.\n\nWe thank the reviewer for questions on validations, please see our responses below:\n1) **Evaluation Criteria**: We perform 3 evals on WOMD-Reasoning and Motion-LLaVA fine-tuned on it:\n    - For human assessment of WOMD-Reasoning, the criteria is detailed in Q7.\n    - For evaluating the outputs of Motion-LLaVA, we use language metrics (ROUGE, BLEU, METEOR, CIDEr, and SPICE) along with a GPT Score which assesses semantic understanding beyond mere text similarity shown in Table 11.\n    - For evaluating WOMD-Reasoning in real-world applications, we add a trajectory prediction task, whose prediction metrics are minFDE_6 and MR_6.\n2) **Distributional Diversity** WOMD-Reasoning's scenario diversity is backed by that of WOMD, a large-scale highly-interactive dataset. Also our paper's Figure 2 shows that among 63k scenarios covered in our dataset, there are ~120k `yields`, ~74k `lights`, ~74k `stops`, showing our dataset’s extensive coverage of traffic rule-induced interactions beyond just near-end events.\n3) **Human Annotation Reliability** To justify the size of our human eval, we scale it up from ~1.6k to ~2.2k Q&As, and the correct rate remains stable (91.99% -> 91.80%), justifying the current eval scale.\n4) **More Validations** We are tuning our data generation pipeline and incorporating WOMD's recent upgrades to offer new versions of the dataset with even better accuracy soon. We have also completed several human verification demos. We are proceeding to the human labeling.\n\n> Q2. Quantitative tests in real-world driving.\n\nWe appreciate the questions on testing on the real world. First we address that testing Motion-LLaVA results on WOMD's val set can reflect real-world interactions, as WOMD is a real-world dataset. To provide more quantitative evals, we introduce language outputs of Motion-LLaVA into a trajectory prediction model Multipath++, seeing ~7% improvements on minFDE_6 and MR_6 in real-world WOMD cases. Details are included in our answer to reviewer BaWk's Q2.\n\n> Q3. Dataset's reliability and biases.\n\nWe appreciate the question on reliability. We analyze the eval criteria, distribution diversity and human eval in Q1. Besides, in our response to reviewer BaWk's Q2, we show that WOMD-Reasoning helps a trajectory prediction model to enjoy ~7% improvements, which supports its overall reliability.\n\n> Q4. Error analysis.\n\nWe appreciate the suggestions on error analysis. In our evaluations, we find that 1) Some scenarios are too complicated to describe in text. 2) Some errors still come from automatic translation program despite efforts. 3) We also see occasional error by the LM due to imprecise attention (e.g., confusing “following” when a car is behind but not in the same lane). Therefore, we are continuously updating and human-verifying the dataset to further improve its quality.\n\n> Q5. Evidence on traffic rule-compliance planning.\n\nWe appreciate the suggestions. As the first step, we add a trajectory prediction task in our answer to reviewer BaWk's Q2, which proves overall effectiveness of our dataset on real-world tasks. We will follow the suggestions to quantify traffic rule-compliance metrics to give more specific evidence.\n\n> Q6. Literatures Comparison.\n\nWe thank the reviewers for these important references. We discuss them below and will cite them in our next version.\n- **DriveLM**(ECCV 2024) We have cited and compared to it in Table 1 and in the introduction (line 28, right column). We will update its version in our next version.\n- **WiseAD**(ArXiv 2024) We see that WiseAD seems to be a driving model work rather than a new dataset. Their Tab. 1 lists the datasets they use for training. We hope our dataset will help encode traffic rule knowledge in models like WiseAD.\n\n> Q7. Criteria for evaluating Q&A pairs, and for reasoning.\n\nWe appreciate questions on the eval criteria of Q&A pairs. Generally, we ask evaluators to judge whether the correct answer is included in the dataset's answers. Practically:\nFor **facts**, the answer must be totally correct. (e.g. if number of lanes is asked, that number must be correct.)\nFor **interaction reasoning**, the correct interaction or intention keyword must be included. (e.g., if human sees Agent #0 is required by rules to yield to Agent #1, then keywords `Agent #0 yields to Agent #1` must exist)\nThese standards help us to maximize the eval accuracy while minimizing the human labor. We are performing human verifications to further boost data quality.\n\n> Q8. Motion-LLaVA v.s. non-language-based models.\n\nWe thank the reviewer for the comparison request. However, instead of comparing, the outputs of Motion-LLaVA can be used as an addition to help real-world trajectory prediction models to perform better, like the ~7% improvement we observe (see our response to reviewer BaWk's Q2).", "claims_and_evidence": "The submission makes strong claims about WOMD-Reasoning as the largest multi-modal dataset for interaction reasoning in autonomous driving, with well-supported evidence from dataset comparisons, language model benchmarks, and fine-tuning evaluations of Motion-LLaVA. \n\nHowever, some claims require stronger validation, particularly the high dataset accuracy. More details on evaluation criteria, distributional diversity, and human annotation reliability. \n\nThe claim that WOMD-Reasoning enables end-to-end traffic rule-compliant planning is not fully substantiated, as planning evaluations are mostly qualitative rather than tested in real-world driving or simulations. Additionally, while the dataset was automatically generated, its reliability and biases are not thoroughly analyzed. \n\nTo strengthen the paper, the authors should provide error analysis, real-world validation, and detailed evaluation metrics for reasoning quality and planning effectiveness.", "methods_and_evaluation_criteria": "Proposed WOMD-reasoning generally make sense for the problem of interaction reasoning in autonomous driving, but there are some points could be improved:\n\n1) Lack of real world/close loop evalutation: Using closed-loop evaluation, where the model's predictions influence simulated vehicle behavior, would better demonstrate its practical impact.\n\n2) The claim that WOMD-Reasoning enhances traffic rule-compliant planning is mostly based on qualitative results. Metrics such as collision avoidance rate, rule violation rate, or trajectory efficiency in a planning framework could provide stronger evidence.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "While the overall experimental design is appreciated, some issues remain. \n\nThere is a lack of systematic error analysis for the generated Q&As, particularly regarding potential (hallucinations, inconsistencies, and ambiguous cases) introduced by the automated pipeline. Without an in-depth breakdown of failure modes, it is unclear how reliable the dataset is for training and evaluating reasoning-based autonomous driving models. Motion-LLaVA’s failure cases are not analyzed, making it difficult to understand when and why the model makes incorrect predictions. A detailed error analysis of both the dataset and model outputs would provide stronger evidence of robustness and highlight areas for further improvement. Also, more transparent human evaluation is needed to ensure the reliability of the dataset and model outputs", "relation_to_broader_scientific_literature": "The dataset and model enable better reasoning capabilities for language-driven autonomous driving systems, facilitating safer and more interpretable decision-making.", "essential_references_not_discussed": "There are some literatures that lack detailed  comparsion and discussion to outline the novelty:\n\n[1] Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., ... & Li, H. (2024, September). Drivelm: Driving with graph visual question answering. In European Conference on Computer Vision (pp. 256-274). Cham: Springer Nature Switzerland.\n\n[2] Zhang, S., Huang, W., Gao, Z., Chen, H., & Lv, C. (2024). WiseAD: Knowledge Augmented End-to-End Autonomous Driving with Vision-Language Model. arXiv preprint arXiv:2412.09951.", "other_strengths_and_weaknesses": "Refer to above.", "other_comments_or_suggestions": "Refer to above.", "questions_for_authors": "1. How do you ensure that the dataset accurately reflects real-world traffic interactions, especially for complex or rare cases?\n\n2. What criteria were used to determine the correctness of Q&A pairs, and how was reasoning correctness evaluated?\n\n3. How does Motion-LLaVA compare against non-language-based trajectory prediction or planning models in real-world performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5JVRVrj6Qc", "reviewer": "Reviewer_LoCq", "summary": "The paper introduces WOMD-Reasoning, a large-scale, multi-modal dataset for reasoning about interactions in autonomous driving, focusing on traffic rule-induced and human intention-induced interactions—areas underrepresented in existing datasets. It also presents Motion-LLaVA, a multi-modal model fine-tuned on this dataset, demonstrating improved interaction prediction and traffic rule-compliant planning. Evaluations show strong performance gains over prior baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful feedback. Please see our response below.\n\n> Q1. Details on the BEV input used for LLaVA & Evaluations on real-world scenarios.\n\nWe appreciate the reviewer's suggestions:\n1) **For the BEV input**, it comes from plotting WOMD data. Road elements are plotted with line segments, while trajectories of agents are plotted with arrow-shaped boxes to indicate position and orientation. Trajectories are labeled with their Agent IDs. Additionally, an XY scale bar is added to help to interpret the position and the velocity.\n2) **For real-world evaluations**, we first address that WOMD is a large real-world dataset. Therefore, testing our model on WOMD's cases can reflect the real world situations. To provide more evals on real-world tasks, we test introducing language outputs of Motion-LLaVA into trajectory prediction model Multipath++, seeing ~7% improvements on minFDE_6 and MR_6 in real-world WOMD cases. Details can be found in our answer to reviewer BaWk's Q2.\n\n> Q2. Generalization to other datasets or cases w/o motion inputs.\n\nWe appreciate the question on generalizations. \n- **For other motion datasets**, our translation program convert motion data into raw language suitable for feeding LLMs to produce data in our dataset's format. With minor modifications in the translation program's IO part, it can be used for any motion dataset. Our translation program is included in the supplementary materials and will be open-sourced.\n- **For non-vectorized motion inputs**, we believe that the well-developed perception models can convert non-vectorized inputs into vectorized motion data. Furthermore, with our simulated visual modal, users can directly train VLMs with WOMD-Reasoning to take vision inputs.\n\n> Q3. BEV preparations and limitations\n\nWe thank the reviewers for suggesting the comparison between BEV and motion vectors. The BEV preparations are detailed in Q1. BEV and motion vectors are equivalent representations of the same scenario. However, BEV images convey the coordinates and velocity info more implicitly, which poses challenges to extract these info. Therefore, our Motion-LLaVA choses motion data as inputs.\n\n> Q4. Simplifying the visuals.\n\nWe appreciate the suggestions on simplifying the visuals to improve readability. We will fix this in the next version.\n\n> Q5. Data efficiency.\n\nWe thank reviewer's suggestion on showing the data efficiency of WOMD-Reasoning by fine-tuning Motion-LLaVA with different data sizes. The results of the suggested experiments are listed below:\n\nFor the **factual** questions:\n| Dataset Size | ROUGE (↑) | BLEU (↑) | METEOR (↑) | CIDEr (↑) | SPICE (↑) | GPT Score (↑) |\n|-|-|-|-|-|-|-|\n| 0.5M | 0.806 | 0.683 | 0.477 | 5.77 | 0.760 | 6.69 |\n| 1M | 0.818 | 0.701 | 0.491 | 5.90 | 0.773 | 6.74 |\n| 3M | 0.840 | 0.736 | 0.516  | 6.35 | 0.794 | 7.09 |\n\nFor the **interaction reasoning** questions:\n| Dataset Size | ROUGE (↑) | BLEU (↑) | METEOR (↑) | CIDEr (↑) | SPICE (↑) | GPT Score (↑) |\n|-|-|-|-|-|-|-|\n| 0.5M | 0.562 | 0.410 | 0.339 | 1.94 | 0.513 | 5.67 |\n| 1M | 0.589 | 0.447 | 0.354 | 2.23 | 0.547 | 6.36 |\n| 3M | 0.614 | 0.474 | 0.366 | 2.52 | 0.571 | 6.76 |\n\nWe find that with more WOMD-Reasoning data involved, the fine-tuned model first gains the ability to answer factual questions, which hits a good GPT score with only 0.5M data. As the dataset grows, the model gradually learns to answer interaction reasoning questions. These justify the size of WOMD-Reasoning, as well as its data efficiency in helping models to answer hierarchical driving-related questions.\n\n> Q6. Comparisons with different backbone models.\n\nWe thank the reviewer for suggesting on using more LM baselines. We provide this comparison in our response to reviewer BaWk's Q1. Results show that w/o fine-tuning on WOMD-Reasoning, all baselines can hardly answer driving-related questions, supporting our dataset's motivation. We then fine-tune LLaVA and LLaMA-Adapter on WOMD-Reasoning, which both significantly benefit from the fine-tuning. Finally, we find that fine-tuned Motion-LLaVA works better than other fine-tuned models, supporting its superiority.\n\n> Q7. Processing of Agent IDs when fed into the network.\n\nWe appreciate the question on agent IDs. These IDs are neither categorical inputs nor embedded. Instead, they are used as textual identifiers within the prompt, enabling the LLM to distinguish agents through natural language understanding. The prompt format is: `Ego agent: <motion>\\nAgent #0: <motion>\\n...\\nNow, please answer: {Question}`, where  `<motion>` represents the encoded motion data from each agent's own viewpoint.\n\nThese IDs only serve as range allocations, i.e. #0-#99 indicate vehicles, #100-#199 indicates bicycles and #200-#299 indicates pedestrians. We transform WOMD agent IDs to these local IDs to avoid downstream models overfitting to specific agent's behavior. Besides this, specific agent IDs do not carry any additional information.", "claims_and_evidence": "The claims are well-supported.", "methods_and_evaluation_criteria": "The methods are appropriate:\n\n1. Automated dataset generation with ChatGPT-4 and rule-based systems is clearly described.\n2. Evaluations use standard metrics (BLEU, ROUGE, CIDEr, etc.) and human validation.", "theoretical_claims": "No formal theoretical claims or proofs.", "experimental_designs_or_analyses": "The experimental design is sound. It would be improved by providing more details on the BEV input used for LLaVA and considering evaluations on real-world scenarios.", "relation_to_broader_scientific_literature": "The paper builds on prior datasets like BDD-X, DriveLM, and DRAMA, addressing their limitations in interaction reasoning. It advances the field by covering traffic rule-based and intention-driven interactions at scale.", "essential_references_not_discussed": "The paper covers the most relevant work.", "other_strengths_and_weaknesses": "Strengths:\n\n1. WOMD-Reasoning is the largest language-based dataset for autonomous driving interaction reasoning to date. It covers underexplored areas such as traffic rule-induced and human intention-induced interactions. Besides, despite being largely automatically generated, the dataset achieves around 90% accuracy in human evaluations, showing strong reliability for research purposes. The authors also provide the related code in the supplementary materials.\n2. The provision of simulated BEV and ego-view videos adds versatility to the dataset, supporting vision-language research and training for autonomous systems.\n3. The model fine-tuned on WOMD-Reasoning shows significant improvements in interaction prediction and traffic rule-compliant planning, key tasks for autonomous driving.\n4. The application of CoT strategies reduces hallucination and improves reasoning quality, a practical enhancement for language-based models.\n\n\nWeakness:\n1. The paper focuses on WOMD data. It’s unclear how well Motion-LLaVA generalizes to other datasets or real-world cases without vectorized motion inputs.\n2. The BEV inputs used in baseline comparisons with LLaVA are not well-detailed. Clarifying their preparation and limitations—especially in comparison to motion vectors—would be helpful.", "other_comments_or_suggestions": "1. Some figures are cluttered, with overlapping numbers and diagrams that reduce readability. Simplifying the visuals would improve clarity and accessibility.\n2. Consider running scaling studies to analyze how model performance evolves with different dataset sizes (e.g., 0.5M → 1M → 3M Q&A pairs). This would provide insights into the data efficiency of WOMD-Reasoning.\n3. Include more comparisons with different backbone models (e.g., other LLMs or vision-language models) to better understand the advantages of Motion-LLaVA’s architecture and components.", "questions_for_authors": "1. Clarify how agent IDs or numbers (e.g., surrounding agent #0, #1) are processed when fed into the network. Are they treated as categorical inputs, embedded, or pre-processed in another way?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w9jrPBOkRT", "reviewer": "Reviewer_Qnzh", "summary": "The authors provided the new Q&A Reasoning dataset built on top of the famous Waymo Open Motion Dataset (WOMD-Reasoning) with the help of ChatGPT-4 and MetaDrive simulator for visualization purposes, and checked the baseline performance of Motion-LLaVA on top of it.\n\n## update after rebuttal\nAuthors did a slight downstream testing and increased a number of Human-evaluated QA pairs, so I increased the overall score from WR to WA", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful and constructive feedback. We address your comments and concerns below.\n\n> Q1. Human Assessment Scale.\n\nWe are grateful for the reviewer's suggestion on providing stronger evidence of validations. To explore the validity of our assessment, we further enlarge our assessment to ~2.2k Q&As. With that being done, the total correct rate comes from 91.99% in the paper to 91.80%, which does not fluctuate much. This suggests that our sampling for assessment is valid and effective. Besides, we hope to confirm that our dataset improvement and human verifications are continuously ongoing. We will update the dataset and the validity when newer data become available.\n\n> Q2. Real-world risks of imperfect (intention) languages.\n\nWe thank the reviewers for raising this question. We hope to address that, since language cannot thoroughly describe exact positions and velocities etc., language models are more commonly being used as an auxillary driving model to provide explainability and introduce high-level knowledge (like traffic rules in our case) or explainability. Therefore, its failing in infrequent cases would generally not cause accidents due to its auxillary nature. Furthermore, our intentions part serve as the summary of interactions between ego agent and all other agents, which is a very comprehensive task than other classes, therefore we believe that 87.5% is satisfying for an auto-labeled dataset. To further improve our dataset's quality, we will also update our dataset with human verifications to further ensure its reliability to 99+%.\n\n> Q3. Downstream impacts on real-world Autonomous Driving tasks.\n\nWe highly appreciate this question. To justify WOMD-Reasoning dataset's ability in downstream tasks, we perform a trajectory prediction task using outputs of our Motion-LLaVA model fine-tuned upon WOMD-Reasoning.  Multipath++ is used as the trajectory prediction baseline model. As the interaction part is the most significant part of our dataset, we introduce the interaction part of language output into Multipath++, by cross-attending the T5 embedding of them with corresponding agents' features. Each experiment is runned with 3 seeds to ensure reliability. The results are shown in the following table\n\n| Model | minFDE_6 | MR_6 |\n|---|---|---|\n| Multipath++ | 1.27 | 12.59% |\n| Multipath++ with Interaction Language | **1.18** | **11.69%** |\n| relative &Delta; | -7.35% | -7.10% |\n\nWe observe a significant performance enhancement using language outputs from motion-LLaVA fine-tuned upon WOMD-Reasoning. This strongly supports WOMD-Reasoning's ability to fine-tune LMs to help downstream tasks like predictions or explainabilities of driving behaviors. We will add final results of these experiments into our manuscript in our next version.\n\n> Q4. More quantitative analysis.\n\nWe thank the reviewer for suggesting more downstream useful evaluations to prove WOMD-Reasoning's abilities. In Q3 we provide evidence proving the effectiveness of WOMD-Reasoning in trajectory prediction tasks. Since WOMD is a highly-interactive real-world driving dataset, prediction evaluations on WOMD offer evidences of the usefulness of WOMD-Reasoning on real-world applications.\n\n> Q5. Validation of Motion-LLaVA - \"facts\" measured. \n\nWe thank the reviewers for requesting details on the definition of facts in Motion-LLaVA evaluations. Our Facts questions can be categorized into 3 types:\n1) **Environment-related questions** focus on the scenario’s surroundings, covering details such as the presence and type of intersections, the number of lanes on the ego vehicle’s side, and the existence and location of crosswalks or stop signs. \n2) **Ego agent-related questions** assess the ego vehicle's characteristics, including its speed, motion status, lane position, and directional attributes. \n3) **Surrounding agents-related questions** focus on other agents in the scenario, covering their type, speed, motion status, and position relative to the ego vehicle and intersection.\n\nWe hope this clarifications help, and we will include this in our next version.\n\n> Q6. Was Motion-LLaVA fine-tuned with motion prediction task or not?\n\nWe are grateful for the question. While Motion-LLaVA is not fine-tuned with motion prediction task, we additionally use Motion-LLaVA outputs to illustrate the effectiveness of WOMD-Reasoning data in motion prediction tasks, the table is shown in Q3. We will add final results of these experiments into our manuscript in our next version.\n\n> Q7. Typo in Line 149\n\nWe thank the reviewer for pointing this typo out. We will fix this in our next version.", "claims_and_evidence": "Claims:\n1. Providing of WOMD-Reasoning, the largest multi-modal dataset with 3 million Q&A pairs focused on interaction reasoning in driving.\n2. Fine-tuning on WOMD-Reasoning, Motion-LLaVA provides detailed and insightful interaction predictions on various driving scenarios\n3. Ablation study on Motion-LLaVA design\n\nEvidence:\n1. To be shared but there is no doubt that it is done, also refer to the Tables 1 and 2.\n2. and 3. Using LLaVA + Multipath++ encoders (Figure 4), authors finetuned the solution to get the results shown in Tables 4 and 5 underlining the correct approach / architecture used; some ablations are shown in Tables 13 and 14 (Appendix)", "methods_and_evaluation_criteria": "* One of the main drawbacks is the validity of the automatically generated by ChatGPT data - especially about intentions. Authors tried to assess the quality of auto labels by human evaluation in Section 4.3 - \"4 people to judge 1610 Q&A\". This is quite small in comparison to the size of the dataset (3 millions of Q&A), less than 1%.\n* Moreover, the results of human assessment - e.g. 87.5% for Intentions - is quite low, especially for Planning, where each % is prone to be converted to some disastrous accident on the road\n* There is no information provided on whether the incorporation of WOMD-Reasoning into any Autonomous Driving solution actually helps with any of real Autonomous Driving metrics (safety / comfort / progress), or even improves just the prediction power of common methods.", "theoretical_claims": "No theory involved", "experimental_designs_or_analyses": "* Quantitative analysis concentrates mostly on Language-related metrics - and very sparsely on Acc / Median Error. That's far from being useful\n* In Section 5.4 \"Validation of Motion-LLaVA\" there is no details on what type of \"facts\" were measured (map elements, other agents states, ego state). Moreover, was Motion-LLaVA fine-tuned with motion prediction task or not?", "relation_to_broader_scientific_literature": "Two main directions: Autonomous Driving and Multi-modality Large Language Models. The paper is to combine these areas on the task of reasoning for AD.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Main items were emphasized in \"Methods And Evaluation Criteria\" and \"Experimental Designs Or Analyses\" sections", "other_comments_or_suggestions": "Typo:\n* Line 149: \"some studies x\"", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces the Waymo Open Motion Dataset (WOMD) of traffic rule-induced interactions when driving, with 3 million Q&A text annotations. The dataset is used to fine-tune a Motion-LLaVa model for predicting interactions in driving scenarios.

Reviewer BaWk, Qnzh , LoCq and TMNS praised the dataset contribution. Reviewer LoCq praised the addition of different view videos and of the fine-tuned WOMD-Reasoning model. Reviewer Qnzh praised the ablations.

Reviewers BaWk and TMNS suggested to include several references, baselines and downstream applications (which the authors did in the rebuttal). Reviewer TMNS also suggested more systematic error analysis. Reviewer LoCq requested some clarifications. Reviewer Qnzh had multiple questions on which they engaged with the authors.

Given the the average score of 3.5 I recommend acceptance.
Decision: Accept (poster)