TUMTraf VideoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
Authors: Xingcheng Zhou
Konstantinos Larintzakis
Hao Guo
Walter Zimmer
Mingyu Liu
Hu Cao
Jiajie Zhang
Venkatnarayanan Lakshminarasimhan
Leah Strand
Alois Knoll
Venue: ICML 2025 poster (2025)
Keywords: Traffic Scene Understanding
Video Understanding
Vision Language Model
Spatial Temporal Reasoning
Autonomous Driving
Primary area: applications->computer_vision
TL;DR: We introduce a large video-language dataset for unified complex roadside traffic scenarios understanding.

Abstract: We present TUMTraf VideoQA, a novel dataset and benchmark designed for spatio-temporal video understanding in complex roadside traffic scenarios. The dataset comprises 1,000 videos, featuring 85,000 multiple-choice QA pairs, 2,300 object captioning, and 5,700 object grounding annotations, encompassing diverse real-world conditions such as adverse weather and traffic anomalies. By incorporating tuple-based spatio-temporal object expressions, TUMTraf VideoQA unifies three essential tasks—multiple-choice video question answering, referred object captioning, and spatio-temporal object grounding—within a cohesive evaluation framework. We further introduce the TraffiX-Qwen baseline model, enhanced with visual token sampling strategies, providing valuable insights into the challenges of fine-grained spatio-temporal reasoning. Extensive experiments demonstrate the dataset’s complexity, highlight the limitations of existing models, and position TUMTraf VideoQA as a robust foundation for advancing research in intelligent transportation systems. The dataset and benchmark are publicly available to facilitate further exploration.

TUMTraf VideoQA: Dataset and Benchmark for Unified Opatio-Temporaa Video Understanding in Traffic Scenes
Xingcheng Zhou'
Konstantinos Larintzakis
Hao Guo
Walter Zimmer'
Mingyu Liu" Hu Cao Alois C. Knoll
Jiajie Zhang'
Venkatnarayanan Lakshminarasinhan
Leah Strand
Weather
Spatio-Temporal Grounding
Q: How is the weather in the video?
Q: Which category la ((e5, 0.O, 0.75, 0:54)/?
Q: Could you identity ((04, 0.68, 0.21, 0.88]] traffic tootage and give the standardized apatictemporal localization for both the first and last frames is
weather the video is clear.
[(05, O.O, 0.75, pedestiing
Counting
A: [t04, 0.0, 0.13, 0.25), (04, 1.0, 0.34, 0.46]]
Q: How many motoroycles are in the footuge?
Q: What is the motion status of [[e12, 0.O, 0.83, 0.34]/? 0.34)) moving.
Zero
Referred Object Captioning
Existence
3D Positioning
Q: What are the key details of [(:00, 1.0, 0.81, 0,308/? 1.0, lower section. has large windows and distintive design that emphasizes Its urban utilty.
bus teaturing root, white blue
any
does [1s23, 0333, 0,46, 0.500) (1:20, 0.3, 0.381/2
A: Yes.
A: [[G20), 0.33, 0.46, 0.50]] appears the front of ||:20, 0.30, 0.3, 0.36%)
Figure 1: TUMTraf VideoQA introduces comprehensive benchmark for video-level traffic scene understanding. Our baseline model, TraffiX-Qwen, is capable of solving multiple tasks, including video QA, spatio-temporal grounding, and referred object captioning, within unified model. In our approach, the spatio-temporal location of objects is represented as tuples (c, fn, x, v), where serves as a unique object identifier fn denotes the normalized frame timestamp, and (z, y) denote the center of the object in the image, normalized with respect to the image dimensions.
Abstract
We further introduce the TrafliX-Qwen baseline model, enhanced with visual token sampling strategies, providing valuable insights into the challenges of fine-grained spatio-temporal rea- soning. Extensive experiments demonstrate the dataset's complexity, highlight the limitations of existing models, and position TUMTraf VideoQA as robust foundation for advancing research in intelligent transportation systems. The dataset and benchmark are publicly available to facilitate further exploration.
We present TUMTraf VideoQA, a novel dataset and benchmark designed for spatio-temporal video understanding in complex roadside traffic scenarios. The dataset comprises 1,000 videos. featuring 85,000 multiple-choice QA pairs, 2,300 object captioning, and 5,700 object grounding annotations, encompassing diverse real-world conditions such as adverse weather and traf- fie anomalies. By incorporating tuple-based spatio-temporal object expressions, TUMTraf VideoQA unifies three essential tasks-multiple- choice video question answering, referred object captioning, and spatio-temppral object ground- withi a cohesive evaluation framework.
1. Introduction
With the advancement of intelligent roadside infrastruc- ture and Large Language Models (LLMs) (Grattafiori et al., 2024), leveraging language to achieve more generalized and interpretable understanding of traffic scenes becomes increasingly important. This involves accurately capturing the relationships among traffic participants, generating de- scriptive captions their appearances, and analyzing their spatio-temporal positions and interactions (Zhang al.,
'School of Computation, Information and Technology, Technical University of Munich, Germany. Correspondence to: Xingcheng Zhou <xingcheng www.tng..houeeeeo dum.de>, Hu Cao <hu.cao@@ dum.de>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TUMTraf VideoQA: Dataset and Benchmark for Unified
Video Understanding in Traffic Scenes
2024; Zhou & Knoll, 2024). Traditional models for traf- fie scene understanding are typically designed for specific tasks, such object recognition, object association, and traffic flow analysis. Although these methods have achieved notable success within isolated domains, they often face sig- nificant challenges in scalability, generalization to diverse traffic conditions, and real-world deployment. The emer- gence and rapid development of large foundation models (Liu et al., 2023; Zhou et al., 2024a) present new opportu- nities to address these challenges. These models offer the potential to overcome traditional limitations by leveraging their ability to generalize across multiple tasks, integrate multimodal information, and adapt to complex, dynamic traffic scenarios in a more flexible and unified manner.
BACK LEIT
FRONT LOST
PUpTT
(a) Objects with the prompt: A white truck that is stationary in the same direction. (Wu et al., 2023b)
(b) Frame- object expres- sion using numerical coordi- nates (Sima et al., 2024).
(c) Object referring in (Zhang et al., 2020) with prompt: What is beneath the adult.
(d) Location of the green bus
Previous studies have primarily advanced traffic scene un- derstanding through image-based crestion-answering tasks in driving environments (Sima et al., 2024; Zhou et al., 2024b; Qian et al., 2024b). However, image-level Vision- Language Models (VLMs) are inherently limited in their ability to capture the temporal dynamics crucial for compre- hending complex traffic events. In contrast, intricate traffic scenarios often require multi-frame video analysis for accu- rate real-world understanding. Besides, despite the growing number of vision- language datasets developed for driving scenarios, significant gap persists in the exploration of multimodal datasets specifically designed for the roadside traffic domain. In particular, video-based datasets captured from a third-party perspective and tailored to traffic scene understanding remain notably underexplored.
[[ccI.0.0.0550..44) in the vidco. (Ours)
Figure 2: Different methods for describing objects in images and videos using language expressions. We adopt a tuple- based spatio-temporal object representation for the unique object reference, as shown in (d).
periments with various efficient visual token sampling strategies, we offer valuable insights and outline poten- tial future research directions.
2. Related Work
2.1. Vision-Language Datasets in Traflic Scenes
To bridge the gap in this domain, we propose TUMTraf VideoQA, a video language dataset designed to benchmark the model understanding capabilities in roadside traffic sce- narios. The dataset encompasses video question-answeringg object captioning, and spatio-temporal grounding tasks, cap- turing key elements crucial for understanding real-world traffic scenes. An illustrative example from the dataset is shown in Figure 1. The main contributions of this work can be outlined as follows:
A growing number of open-source datasets have been re- leased to facilitate autonomous driving and intelligent trans- portation systems, such BDD100K (Yu al., 2020), Waymo Open Dataset (Sun al., 2020), and Ego4D (Grau- man et al., 2022). In recent years, the rapid advance- ments in LLMs drive significant efforts to integrate lan- guage for the development of vision-language foundation models in this domain. As summarized in Table 1, several pioneering datasets have been introduced for traffic scenar- ios, particularly focusing on vehicle-cene centtri environments (Liu et al., 2024b). NuScenes-QA (Qian et al., 2024b) pro- vides a question-answeringg benchmark tailored for driving scenes. Meanwhile, DRAMA (Malla et al., 2023) is de- signed for video-level open-ended tasks aimed at evaluating driving instructions and assessing the importance of objects within their environments. Besides, referring to specific traffic participants through natural known as referred object grounding and tracking-is cru- cial task in traffic scene understanding. Some works (Wu et al., 2023a;b) extend the KITTI (Geiger et al., 2013) and nuScenes (Caesar et al., 2020) datasets, by associating natu- ral language descriptions with specific vehicles and pedestri- ans. This facilitates fine- ane-grained identification and tracking of traffic participants, allowing for precise object localiza-
We present TUMTraf VideoQA, a comprehensive video-language dataset designed for complex traf- fic video understanding. The dataset captures a di- verse range of real-world scenarios, including extreme weather conditions and critical corner cases such as traffic accidents.
We propose a novel benchmark that evaluates model performance across three key tasks, including video question answering, referred object captioning, and spatio-temporal grounding, facilitating fine-grained reasoning in traffic scenarios.
We establish the LauffX--wwen baseline and provide detailed results and analyses. Through extensive ex-
2

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified
Video Understanding in Traffic Seenes
Table I: Summary and comparison of visual-language datasets in the traffic domain for question answering, video grounding, and referred multi-obeect tracking. The table's upper section presents QA tasks, while the lower section covers grounding and referring tasks. We introduce the first roadside video understanding dataset and unify these tasks in one benchmark.
Datuset
Venue
Tasks
QA Gen.
Videos/cennes
+AsCCaptions
# Grounding
Domain
DRAMA (Malla al, 2023)
23
Video QA
Marual
18k
102k
Driving
LingoQA (Marcu et al., 2024)
ECCV'24
Video QA
Marual
28%
419k
Driving
NuSeenes-QA (Qian et 2024b)
AAAI'24
Image QA
Template
460k
Driving
DriveLM (Sima et al, 2024)
BCCV
Image QA
Temp. Man.
188k
4.2M
Driving
City (Sun et al., 2024)
ACM MM'
Scene QA
Temp. Man.
193
450k
City
HC-STVG (Ting et al., 2022)
ACM 22
Video Grounding
Marual
5.6k
5.6k
General
DVD-ST (Ji al. 2024)
Video Grondiing
Marual
2.7k
5.7k
General
UCA (Yuan et al., 20124
C2
Video Ground, & Cap.
Manual
1.8k
23.5k
Surveillance
Refer-KITTI (Wa al., 2023a)
CV
Refemed-MOT Refemed-MOTT
Marual
818
Driving
NuPrompt (Wi al., 20236)
25
I.M
850
35k
Driving
TUMTraf VideoQA (Ours)
ICML/25
Video QA, ST Grounding
Temp.
I.IM
1k
87.3k
5.7k
Roadside
tion based on language descriptions in complex driving en- vironments. In the traffic domain, OATS (Agarwal & Chen, 2023) introduces structured representation based on or- dered atomic activities for fine-grained scenario understand- ing, Action-slot (Kung et al., 2024) further leverages slot attention to recognize multiple activities, achieving strong results on both synthetic and real-worls datasets. However, most existing efforts primarily focus on driving scenarios and are typically constrained to individual tasks such as question answering, video grounding, referred multi- object tracking. A significant research gap also remains in the availability of large-scale datasets designed specifically for roadside surveillance scenarios. Our work aims to bridge this gap by providing a comprchensive dataset tailored for multiple tasks in roadside traffic understanding within a unified framework.
methods focused on improving cross-modality tuning and alignment (Bi et al., 2025a; Gao et al., 2023; Bi et al., 2025b). Despite these advances, achieving structured and precise fine -ne-grained feature representations in video understanding remains an open challenge.
2.3. Language-Bans Basedd Object Referring
Referring objects in visual data, such as images and videos, is typically achieved by associating them with predefined definitions or language descriptions. Figure 2 illustrates four commonly used methods for representing objects through language expressions. The inherent ambiguity of natural language, coupled with the modality gap between visual and linguistic representations, presents significant challenges. Object representation in tasks such as object referring often necessitates careful dataset curation to ensure that linguistic expressions uniquely or collectively correspond to specific objects in videos. For example, some datasets include only scenarios with uniquely identifiable objects (Tang et al., 2022), while others contain expressions that jointly refer to multiple objects (Ji et al., 2024). However, in complex real- world applications such as autonomous driving, tex- tual descriptions alone are often insufficient to uniquely specify an object. To address this challenge, DriveLM (Sima et al., 2024) introduces a structured tuple represen- tation, <c, CAM, x. y>, where e denotes the object iden- tifier. CAM specifies the camera, and Ez,y> represents the 2D center coordinates within the camera's coordinate system. Alternatively, ELM (Zhou et al., 2024b) simpli- fies the problem by converting temporal video tasks into frame-level questions, using a tuple <c, xr y> to identify objects within individual frames without temporal depen- dencies. Despite the advancements, formulating a unified, precise, and unique language representation for objects in video remains an open challenge.
2.2. Fine-Grained Video Understanding
Fine-grained video understanding centers on the precise analysis of intricate video content, targeting tasks that de- mand nuanced reasoning across spatial and temporal dimen- sions. Some representative tasks include spatio-temporal grounding (Zhang et al., 2020; Tang et al., 2022), mapping specific objects or events to precise locations and times within a video based on a given query; video object refer- ring (Ding et al., 2023; Wu et al., 2023a;b), which involves tracking objects through space and time given text prompts: video temporal grounding (Lin et al., 2023; Huang et al.,, 2024), identifying specific moments intervals in video that align with a provided textual query. These tasks re- quire high-precised, nuanced multimodal alignment, and the ability to capture subtle temporal and spatial dynamics, making it particularly challenging. With the progress in visual LLMs. recent works enhance capabilities of video understanding (Tang et al., 2023) to facilitate comprehen- sion across both abstract and fine-grained levels, some ap- proaches introduced enhanced video representations via celf-supervised learning (Qian et al., 2024a), while several
In this work, we design a spatio-temporal object representa- tion in videos with a four-element tuple format (c, fn. x, v),
3

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified
Video Understanding in Traffic Scenes
Video Selection
Metadata Curation
QA Generation
Q: What's the movement state of ((c4, 0.75, 0.21, 0.36|/7 [fot, 0.75, 0.21. 0.38% sppeers be moving.
Video Pool
Trajectories
Detection
Weather Database
Weather History
Are there any bicycles
traffic fnotage?
Object Captions
Camera Extrinsics
Road Blocks
Accident Information
Automatic Selection
Manual Selection
Entract
Rule- suilebbsee QA genration
Language
&
Video Motadata
V
Weather Road Type
Accidents Road Blocks Special Vehicles
hd
Quality Check
QA Enhancement
Time of Day
Extract
Adjunt
Adjust
Final Video Pool
Open Voc. Trackers
Bin
Final QA Pool
Off-the-shelf Tools
Mult. Trackers
Adjust
Quality Check
Figure 3: The workflow of the semi-automati annotation pipeline for TUMTraf VideoQA generation, integrating external database, leveraging various off-the-shelf tools and LLMs, with human quality checks ensuring accuracy.
where e denotes a unique object identifier, fn indicates the normalized frame timestamp, and (xt, y) corresponds to the object's s normalized spatial coordinates within the frame. The same object is consistently assigned the identifier throughout the video, while its spatial position changes time. This formulation enables precise tracking and referencing of objects across both spatial and temporal di- mensions, facilitating robust language- based interaction in dynamic environments. Besides, it provides a standardized interface for fine-grained video understanding, enabling more detailed and structured analysis.
characteristics of real-world traffic environments.
Metadata Curation. The video metadata includes envi- ronmental conditions, object positions, trajectories, appear- ances, traffic flows, and more, serving as the basis for gen- erating high-quality annotations. External data sources in- clude historical weather records, traffic accident reports, and camera calibration details. To ensure precise time-specifer weather and traffic information, we align video timestamps with these records using GPT-4o and Text-embedding- large (OpenAl et al., 2024). For visual metadata, we utilize state-of-the-ait object detectors and trackers (Wang et al., 2024; Zhao al., 2024), along with open- open-vocabulary detec- tors (Yan et al., 2023; Wu et al., 2024), to generate bounding box and trajectory data. We then transform 2D information into camera- based pscudo-3D locations using camera cal- ibration matrices, facilitating the generation of questions related to object motion and relative spatial positioning. To capture object appearance details, we utilize large VI.Ms (OpenAI et al., 2024; Liu et al., 2024a), which automatically generate textual descriptions for cropped object bounding boxes. A manual quality assurance step is conducted to thoroughly evaluate the accuracy and completeness of the metadata. Any identified deficiencies trigger necessary ad- justments and a reprocessing cycle to ensure data quality and integrity before progressing to the next stage.
3. TUMTraf VideoQA Dataset
3.1. Dataset Creation
Our data generation process comprises three primary stages: Video Selection, Metadata Curation, and QA Pair Genera- tion, as shown in Figure 3. To ensure high-quality, diverse, and balanced annotations, we introduce a semi-automatic labeling pipeline that combines automated processes with human verification for enhanced accuracy and consistency.
Video Selection. The video data in TUMTraf VideoQA are collected from multiple roadside infrastructure points over data collection period spanning more than two years. The dataset encompasses diverse perspectives, covering var- ious urban, suburban, and highway scenarios. It includes broad range of video content, capturing various distinct traffic scenarios, such as traffic accidents, rescue operations, congestion, roadblocks, and uncommon vehicle occurrences. Furthermore, the dataset encompasses a variety of environ- mental conditions, including sunny, rainy, cloudy, snowy, and foggy weather, along with technical challenges scenar- ios such as obstructed camera lenses and vibrations. The video segments are carefully selected to include a diverse range of traffic participants-incluee vehicles, pedestri- ans, and obstacles-eee cattrring the complexity and dynamic
QA Generation & Filtering. To ensure balance between question diversity and accuracy, adopt hybrid approach that combines template-basee and LLM-driven generation strategies. Approximately 15 question templates are manu- ally designed for each question type and further expanded using --ggerreeeteedd variations. These templates are pop- ulated with relevant objects and metadata generate initial QA pairs using GPT-4o-mini. The LLM is then prompted to refine the generated content by rephrasing either the ques- tion alone both the question and its corresponding answer, depending on the context. Once QA pairs are generated for

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
each question type, a selective quality evaluation is con- ducted to assess their accuracy and relevance. This iterative process involves refining question templates, adjusting off- thelf tools, and discarding QA pairs that do not meet the predefined quality standards. The validated QA pairs are then integrated into the TUMTraf VideoQA dataset, en- suring high-quality and diverse annotations.
temporal extent and spatial positions within the video.
We adopt three evaluation metries to assess the performance of this task, i.e., Temporal error E Spatial error E, and Sratio-Tempords eπror Ess. Temporal error E,, and Spa- tial error & use the L.1 loss, which measures the abso- lute temporal differences Afn and the spatial displacement As = (IAzz Ay)ll2 The apatio-Tempone L2 loss and captures deviations across both spatial and tem- poral dimensions. For each metrie, both the start and end frames are considered, with the formulations as follows:
error Ear adopts
3.2. Tasks and Metrics
TUMTraf VidcoQA benchmark comprises three core tasks to thoroughly evaluate model performance in traffic scenes: Multi-Choice Question Answering (MQA), Video Referred Object Captioning (V-ROC), and Spatio-Tempora Object Grounding (ST-OG). QA pairs related to weather and traffic accidents are included for training and future research but are not considered in the benchmark evaluation.
A + AS" E
As'+As' E.
(1)
2
2
1 (I(AS,, AF, A + I( AE", Ib)))
Ext
(2)
2
3.3. Dataset Statistics
Multi-Choice Question Answering. The MQA task as- sesses the model's capabilities across five key dimensions: Positioning, identifying the relative 3D spatial location of objects; Counting, determining the number of oceurrences of a particular object or class across the video; Motion, analyzing the movement status of objects; Class, catego- rizing objects based on their type or attributes; Existence, querying whether a specific object or category is present in the video. Following (Qian et al., 2024b), each dimension is further divided into easy and hard levels, depending on whether the question requires single-hop or multi-hop rea- soning. We show the template of easy and hard questions in Sec C.2. We use Top-1 accuracy as the evaluation metric and report the mean accuracy across all question types.
Question Count
3000
2000
1000
20
40
60
80 100 120
Number of Words
(a) Distribution of question word num counts across question type.
(b) Class distribution of Multi-Choice QA.
250
600
Answer Count
700
500
150
400
300
100
200
50
100
Video Referred Object Captioning The task evaluates the model's s capability to describe the appearance of specified object in natural language. It aims to generate detailed and accurate summaries that effectively capture the object's 's key visual attributes. Unlike the image-based referred object captioning task (Sima et al., 2024; Zhou et al., 2024b), we query an object based on its spatial and temporal location within a video, which adds a significant level of complexity. In this task, we adopt common NLG metrics (Sai et al., 2022), including BLEU, CIDEr, ROUGE, METEOR, and SPICE, to measure the quality of descriptions.
20
30
40
50
60
0
20
40
60
80
100
Number of Words
Time Span (seconds)
(c) Distribution word counts in Video Re- ferred Object Captioning.
answer
(d) Temporal window lengths
in Ppatio-Tempora Ground- ing.
Figure 4: Statistical distributions of the TUMTraf VideoQA dataset, including word counts in questions and answers, distribution of question types, and temporal window lengths for object grounding.
TUMTraf VideoQA dataset consists of 1,000 videos, 85,000 multi-choice QA pairs, 5,700 spatio-temporal grounding prompts, and 2,300 referred object captioning. Video du- rations range from 10 seconds to 2 minutes. We split the videos into training and validation sets with a ratio of 7:3, ensuring that videos in the validation set do not overlap with those in the training set. Generated QA pairs inherit the split of their associated videos, forming distinct videos and annotations for training and validation. Figure 4 provides an overview of the dataset's 's statistical distributions, includ- ing question complexity, question-type distribution, answer lengths, and the temporal window distribution of queried
Spatio-Tempors Object Grounding, Accurately identi- fying the ppatio-temporal positions of a specified object is crucial in traffic scenarios. Unlike traditional video ground- ing (Tang et al., 2022) or referred multi-object tracking tasks (Wu et al., 2023b), which primarily focus on locating objects within individual frames across the video, ST-OG simpli- fies the process by providing start and end frames along with corresponding spatial coordinates in standardized tu- ple format: [(c, ff x', y²), (c, f" x". "."]. This task serves to assess a model's performance in effectively associating objects across frames while accurately determining their
5

---

TUMTraf VideoQA: Dataset and Benchmark for Unified Opatio-Temporal Video Understanding in Traffic Scenes
objects in the spatio-temporal grounding task. Figure 5 shows video statistics of the TUMTraf VideoQA dataset, highlighting its diversity across scene types, traffic con- ditions, and object categories. We categorize scenes into three types: highways (rural), urban intersections (city), and country roads (rural/urban). The traffic volume trends, track durations, and category distributions across different times and locations collectively indicate that the dataset captures realistic traffic dynamics and reflects patterns consistent with real-world distributions. These statistics underscore the broad coverage and its suitability for training and eval- uating fine- fine-grained video understanding models in diverse traffic scenarios. Further details and statistics are available in Sec. A.1.
Ropenee Xe
Large Language Model (Qwen2)) f+
Spedill Fruiag
Test Dokeniaer
Ao Spriid Pooling
fitk LA 1A LRR
Mali-Ro Token Saxplar &.
dus Frims
Paoererr
Ai
Lim Fou
f.
Lasacble
Figure 6: Overview of the TraffiX-Qwen baseline model. Yellow and orange colors represent the combination of multi- resolution visual tokens from different visual strategies, while blue indicates textual tokens.
Welcce Rare
tasks within a unified framework. The architecture of the TUMTraf VideoQA baseline, as illustrated in Figure 6, con- sists of four core components: visual encoder fu. cross- modality projector Jv, token sampler S., and large language model fa. following (Li et al., 2024).
10:33
(a) Traffic distribution throughout the year by time of day and
scene type.
Visual Encoder.
The video is uniformly divided into 100 segments, including the first and last frames, resulting in total of N = 101 frames. Given the sampled video input X E BRNxxxxxx, we adopt SigLIP (Zhai et al., 2023), a Transformer-bused model pre-trained on large-scale language-image datasets, as the visual encoder. Each frame is processed at resolution of 384 x 384, and the video is en- coded into sequence of visual features Z, [uI,-.-, uN). where e; (X,) E TxC, containing T spatial tokens of dimension C.
15000
11000
12000
(b) Track duration of different road users across scene types.
Token Sampling Strategy. We leverage a simple yet ef- fective frame-level multi-resolution sampling strategy to enhance feature representation. We evaluate four primary sampling strategies: spatial pooling, multi-resolution spa- tial pooling, multi-resolution token pruning, and multi- resolution temporal pooling. The output Z, from the last layer of SigLIP is denoted as Znigh, which is reduced to I'' tokens after down-sampling. We define the set of high- resolution frames as keyframes, denoted by KC(-). Addition- ally, a learnable token is appended to the end of each frame to explicitly differentiate them. The number of tokens used in various strategies is presented in Table 2.
bayyk
(c) Average hourly traffic volume by scene type and object class.
Figure TUMTraf VideoQA dataset video distribution over scene types, traffic conditions, and categories.
4. TraffiX-Qwen Baseline
This method applies spatial pooling to each feature map Znigh. resulting in a down-sampled repre- sentation Z = fpool (Ziggh with N x T' tokens, as shown in Eq. 3. We use the notation [-N to represent the operation of sequentially concatenating the processed feature maps.
Spatial Pooling:
4.1. Model Architecture
We introduce CreffiX-Qwen a baseline model for the TUM- Traf VideoQA dataset that effectively addresses all three
6

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Tempoolll Video Understanding in Traffic Seenes
Table 2: Comparison of visual token numbers across differ- ent token sampling strategies. We keep the high resolution 27x27 and the low resolution at 14x14.
of Qwen-2, namely 0.5B and 7B, to establish baselines of different scales. The answer generation process in our TraftiX-Qwen baseline model is formulated as:
Method
Number of Visual Tokens
Max Tokens
Spotial Pooling
X + N
19,897
T P(X, Sc((o), = II Po.w Su(o)), Xx) (7)
MultiRes Spatial-Pooling
T+ - x
20,430
MultiRes Token-Pruning
T+ -1)xXXXX-
18,574
MultiRes Temporal-
KxT -NV (N -) +
20,963
4.2. Baseline Training
(z.)
[Z Zeeum N
(3)
Our baseline model undergoes two-stage training process consisting of video-language alignment and visual instruc- tion fine-tuning, to enhance its understanding of traffic sce- narios and reasoning capabilities for long videos. Both stages are trained with NVIDIA A GPUs.
m=
MultiRes Spatial Pooling: Compared the naive spatial
pooling, this strategy selects the first frame the keyframe K (1), and is retained at its original resolution Zriga It is formulated in Eq. 4.
Video-Language Alignment. This step aims to align video representations with language embeddings, ensuring that the LLM can effectively interpret the visual features. We freeze both the visual encoder and the LLM, and train only the projector layer. To facilitate the training, we initialize the parameters of the 2-layer MLP from the VA--VA- model, which has been pre-aligned with large-scale cross- modality datasets, including 3.2M single-image and 1.6M One Vision caage-cattion pairs. In this stage, we further train the projector on raw TUMTraf VideoQA data, utilizing endede captioning pairs without transforming it into the multiple-choice QA for epoch.
(Z.) /Zngar Zuam;
Zisam
n=
(4)
MultiRes Token Pruning: Similar to MultiRes Spa- tial Pooling, the first frame is designated as the keyframe. Token-wise cosine similarity is then computed between the keyframe and each subsequent frame, while visual tokens with lowest similarity are selectively retained based on pre- defined ratio r, formulated as Zprumed f (Znigh) shown in Eq. 5. To ensure visual token efficiency comparable to spatial pooling, is set to 0.25. A similar strategy is also applied in autonomous driving scenarios (Ma et al., 2024).
Visual Instruction Fine-Tuning. Building upon the robust representations established during the alignment stage, we further fine-tune our baseline model on the training set of TUMTraf VideoQA. The multi-choice QA pairs are refor- matted into the instruction-folloiii format to prompt the model to generate the corresponding answers. During this stage, we freeze the vision encoder and projector layers and finetune the Qwen-2 model with fall-parameter fine-tuning to adapt its reasoning and contextual understanding to the traffic environment. The model is fine-tuned for 1 epoch.
S. (Z.) KZagh Zeem: (ppraed Zrem) n=2]
MultiRes Temporal Pooling: In this strategy, the keyframe set is adaptively queried by input questions KC()) = 2(xg). Based on the temporal regions of inter- est derived from the question, K keyframes are selected, which are preserved with high-resolution representations migh- Meanwhile, the remaining frames undergo spatial pooling, resulting in Zio as expressed in Eq. 6. Typically, K < 2, and for general questions without specific temporal focus, the first frame is set as the default keyframe.
Table 4: Training and inference cost of TraffiX-Qwen
Ver.
#Vision
#Proj.
#LLM
Inf/QA
#Trainable
Train Hour
Sa(Zo) = = Zremm W if E K((),, if & K)())
0.5B
397.8M
1.M
493.8M
~1.6s
495 6M
28h
7B
397.8M
17.0M
7612.6M
3.88
7629.6M
36h
where 2ej Zinght
(6)
Computational Cost. Table 4 summarizes the training and inference details of the TraffiX- -ween models. We show the number of parameters of the vision backbone, the projector, and the LLMs. Inference speed is reported as the average time per question, using autoregressive decoding on a single A100 GPU, without acceleration strategies such as quantization or caching.
Large Language Model. We adopt Q (Yang al., 2024) as the pre-trained LLM in our TraffiX-Qwen baseline. Qwen-2 demonstrates strong capabilities in in-context learn- ing and instruction following, supporting context lengths of up to 32k tokens. This allows for the processing of complex and forg-form inputs effectively. We utilize two versions
7

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
Table 3: Evaluation of Open-source models and TraffiX-Qwas baseline on the Multi-Choice QA track of the TUMTraf VideoQA Dataset, where represents easy, single-hop questions, and H denotes hard, multi-hop questions.
Positioning
Counting
Motion
Class
Existence E
Models
Category
Overall
I
E
H
F
H
H
Open-Source Source Models
LLAVA-OneVision (Li al., 2024)
0.5B
42.10
25. 26
27.62
30:45
54.87
37.04
57.06
39.57
85,29
58.35
45.82
7B
46.92
22.03
69.42
54.85
61.14
60.48
51.92
56.50
77.08
63.25
56.36
Qwen2-VV VL (Bai al., 2023)
2B
36.73
26.05
38.10
39.78
56.46
35.19
32.10
38.49
68.87
67.32
43,91
7B
36.03
24.35
66.91
49.11
61.65
38. 10
44.83
40.20
54.00
73.03
48.82
Video (Cheng et al., 2024)
2.0-7B8FF
42.54
18.14
44.13
37.56
59.37
35.87
39.05
44.07
44.56
65.56
43.09
2.0-7B-16F
42.41
10.47
55.98
41.94
53.80
52.26
44.16
47.75
66.93
64.82
48.05
TraffX- Trefix--wen Baseline
Spatial Pooling
75.54
68.47
82
83,92
79.95
59.73
93.06
85.37
78.84
Baseline-O.5B (Ours)
MultiRes Inatial-Poolingg
76.36
69.32
86.10
75.86
83.73
79.59
80.57
61.70
92.73
85.37
79.07
MultiRes Token-Pruning Pruning
76.61
73.40
86,33
76. 88
83,48
78.60
80.01
60.43
93.34
85 27
79.44
MultiRes Semporal-Pooling
75.85
74.07
85.65
76.92
84.05
80.64
80,26
62.21
93.06
85.55
79.83
Spatial Pooling
76.99
76.14
87.07
76.81
86.58
82.07
82,72
64.11
93.62
85.27
81.14
Baseline-76 7B (Ours)
MultiRes Spotial-Pooling
78.89
76.99
87.07
77.49
88.29
81.82
83,52
65.95
93.01
85.51
81.85
Multiees Tokco-Pruning
76.93
77.24
87.41
77.76
86.46
80.64
82,66
65.00
93.84
85.48
81.34
MuliiRes Temporal Pooling
78.57
77.24
87.53
78.22
87.09
82,68
83.33
65.76
93.78
85,34
81.95
5. Experiments
VideoLLaMA2 (Cheng et al., 2024) on our Multi-Choice QA task. The results indicate that increasing model size significantly enhances their performance in zero-shot video QA scenarios, with improvements from 5% to 10%. Notably, VideoLLaMA2 benefits from incorporating more frames, leading to notable boost in accuracy. Among the three models with 7B parameters, Qwen2-VL. and VideoLLaMA2 achieve comparable overall performance, whereas LLaVA-One Vision outperforms both, achieving the highest accuracy. Furthermore, all models struggle with positioning questions, highlighting their limitations in spatial reasoning.
Extensive experiments conducted the TUMTraf VideoQA dataset. We evaluate SOTA open-source VLMs in zero-shot setting to assess their spatio-temporal reasoning abilities, analyze the dataset's characteristics, and examine the impact of different visual sampling strategies on perfor- mance. During inference, the temperature is set to zero to ensure deterministic outputs and enhance consistency.
5.1. Quantitative Results in Multi-Choice QA
Table 3 presents the quantitative results in this task, offering several key insights, which are summarized follows.
Difficulty of Question Types. The accuracy across differ- ent question types reveals consistent trends of difficulty for both open-source VLMs and our baseline models. Among the evaluated question types, existence questions are the least challenging, achieving the highest accuracy. This is followed by counting and motion questions, which necessitate the extraction and reasoning of information across multiple video frames. In contrast, positioning questions, which require deeper understanding of 3D spatial relationships, emerge as the most challenging- Moreover, the accuracy of multi-hop questions is generally lower compared to single-hop questions, reflecting the increased complexity of complex reasoning tasks that demand the capture of more fine-grained details and intricate reasoning processes.
Effect of Token Sampling Strategy. Experimental results from the 0.5B and 7B baseline models demonstrate that multi-resolution strategies can enhance model performance to some extent, with MultiRes Temporal Pooling yielding the most significant gains. Notably, the MultiRes strategy can greatly improve positioning tasks that rely on spatial recognition, while having minimal impact on existence and counting tasks. Moreover, MultiRes Token Pruning effec- tively enhances positioning and counting accuracy but may inadvertently discard critical visual tokens, leading to lim- ited or adverse effects on motion and existence tasks. While MultiRes Temporal Pooling enhances fine-graine ane-grained reason- ing, it has little impact on easy recognition tasks like exis- tence. Although multi-resolltion methods provide richer multi-granularity visual representations, the overall perfor- mance improvements remain moderate.
Open-Source Model Performance. We evaluate the per- formance of three open-source models: LLaVA-One Vision (Li et al., 2024), Qwen2-VL (Bai et al., 2023), and
5.2. Results in Spatio-Tema Temporal Grounding
The quantitative results for the Spatio-Temport Grounding task, presented in Table 5, underscore the complexity of the
8

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified
Video Understanding in Traffic Scenes
Table 5: Evaluation of Spatio-Temporal Errors Across Open-
Table 6: Performance of Open-Source models and TraffiX- Qwen on Referred Object Captioning.
Source models and TraffiX- -wenn Baseline.
Model
Temporal EL
Spatial E1
ST EL
Model
Bleu 4
ROUGE CIDR
METEOR
SPICE
Open-Source Models
Open-Source Models
LLAVA-Oue Vis-ooo (0.5B)
0.48
10.16
0.0102
LLAVA-One (0.5B)
0.7285
0.7212
0,8415
LLAVA AVA--nneissoon (7B)
5.77
14,09
0.1326
(7B)
0.7680
0.7750
0,8142
Qwen2 VL (2B)
8.72
17.93
0.2086
Qwen2-VL (2B)
0.7729
0.7793
0.8127
(uve2--VL.(B))
10.47
20.14
0.4119
Qwen2-VL (7B)
0,7615
0.7647
0,8032
Videol LM2 (7B-8F)
6.25
19.94
0.2391
VideoLLaMA2 (7B-8F)
0.6225
0.6360
0.6896
VideoLLaMA2 (7B-16F)
18.69
0.2111
Vid2 (7B-16F)
0.7218
0.7383
0.7895
TrafX-Qween Baseline 0.58
TrufiX-Qwert Baseline
Spatial Pooling
34.99
50.44
2.5195
35.24
46.35
MultiRes Spatial-Pooling
50.26
2,4306
45.75
0.5B-Spatial Spotial-PPoolingg
0.1220
0,1892
0.2600
MuliiRes Token Pruning
35.07
50.79
2.5730
35,30
46.48
0.5B- M.58-MultiRes-SS MultiiRes-SattialPPoonn
0.1211
0,1894
0.2607
MultiRes Temporal-
$1.00
2.5464
47.17
0.58-Man M---MMiiiRe---- MoleeRes-Token-- MaiennPnnning
0.1230
0,1934
0,2650
Trafti--Qwer Baseline 7B
0.5B-Multice MMlMMMMMMRRTTTTT- MultiRes-Temporal Temporal-Poolin
0.1228
0,1912
0.2629
Spatial-Pooling
36.74
52.04
2,5613
36.42
47.32
7B- Spatial-Poollngg
0.1083
0.1737
0.2382
MuliiRes Spotial-Pooling
53.26
2.6113
49.16
7B- MrltMMult-sssseeee Spatssa--aaaaaaanaa Pooling
0.1 133
0.1822
0.2493
MuliiRes Token Pruning
37,83
52.31
2.6162
36.56
47,80
7B- Me--MMutRRsTTToeee Token-Pruning
0,1152
0.1748
0.2454
Temporal-
52.58
2.4236
48.53
7B Met-MuliRRe--te Santiies-TTempral
0.1166
0.1790
0.2496
6. Conclusions and Future Works
In this work, we introduce TUMTraf VideoQA, a novel benchmark that aims at advancing spatio-temporal video understanding in complex real-world traffic scenarios. The dataset provides large-scale collection of high-quality videos and annotations specifically curated for roadside surveillance, covering three fundamental tasks: multi-choice video QA, spatio-temporal grounding, and referred object captioning within a unified evaluation framework. Extensive evaluations using SOTA vision language models, along with the introduction of the Trafti -ween baseline model, estab- lish a strong foundation for future research and development. We believe further optimization, such as quantization, and pruning, is promising direction for improving its deploy- ment efficiency in traffic monitoring. TUMTraf VideoQA serves as a comprehensive benchmark to facilitate further advancements in traffic video analysis and contribute to the development of next next-generation traffic foundation models.
task. Findings across temporal, spatial, and spatiotemporal errors exhibit a general consistency, revealing that without fine-tuning, open-source VLMs struggle to understand the task and cannot accurately regress the corresponding tu- ples, leading unreliable temporal and spatial localization. For the fine-tuned TrafliX-Qwen baseline models, multi- resolution strategies appear to diminish spatial and temporal grounding performance, in contrast to their effectiveness in Multi-Choice QA and Referred Object Captioning tasks. This suggests that while multi-resolution techniques en- hance frame-based object recognition by providing finer visual details, dynamically adjusting frame-level resolution can introduce ambiguity in inter-frame representations, ad- versely affecting temporal grounding and, consequently, spatial localization capabilities across the video.
Impact Statement
5.3. Results in Referred Object Captioning
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
As shown in Table 6, Qwen2- VL (7B) surpasses all other open-source models by a considerable margin, demonstrat- ing its strong performance on referred object captioning task. For baseline models, both the 0.5B and 7B variants exhibit performance improvements across various metries when enhanced with multi-resolution strategies. Moreover, the 7B models consistently outperform their smaller counterparts in both open-source and fine-tuned baseline settings. The im- pact of the visual token sampling strategy, however, varies with model size. MultiRes Temporal Pooling yields the most significant gains for the 0.5B model, whereas MultiRes Spa- tial Pooling proves most effective for the 7B models.
Acknowledgment
This research supported by the project AUTOtech.agi (Grant No.011S22088L) and the Federal Ministry of Educa- tion and Research of Germany (BMBF).
This work was also supported by the project VIDETEC-2 (Grant No.01F2232E) and the Federal Ministry for Digital and Transport of Germany (BMDV).

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified
Video Understanding in Traffic Seenes
References
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., Martin, M., Nagarajan, T., Radosavovic, I., Ramakrish- nan, S. K., Ryan, F., Sharma, J., Wray, M., Xu, M., Xu, E. Z., Zhao, C.,, Bansal, S., Batra, D., Cartillier, V., Crane, S., Do, T., Doulaty, M., Erapalli, A., Feichtenhofer, C., Fragomeni, A., Fu, Q. Gebreselasie, A., González, C., Hillis, J., Huang. X., Huang, Y. Jia, W., Khoo, W., Koláī, J., Kottur, S., Kumar, A., Landini, F., Li, C., Li, Y., Li, Z., Mangalam, K., Modhugu, R., Munro, J., Murrell, T., Nishiyasu, T., Price, W., Puentes, P. R., Ramazanova, M., Sari, L., Somasundaram, K., Southerland, A., Sugano, Y., Tao, R., Vo, M., Wang. Y., Wu, X., Yagi, T., Zhao, Z., Zhu, Y., Arbeláez, P., Crandall, D., Damen, D., Farinella, G. M., Fuegen, C., Ghanem, B., Ithapu, V. K., Jawahar, C. V., Joo, H., Kitani, K., Li, H., Newcombe, R., Oliva, A., Park, H. S., Rehg, J. M., Sato, Y.. Shi, J., Shou, M. Z., Torralba, A., Torresani, I.., Yan, M., and Malik, J. Ego4d: Around the world in 3,000 hours of egocentric video. In 2022 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 18973-18990, 2022. doi: 10. 11110//CVR52688.2222.00000
Agarwal, N. and Chen, Y.-T. Ordered atomic activity for fine-graine fine-grained interactive traffic scenario understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8624-8636, October 2023.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J. Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Bi, J., Wang. Y., Chen, H., Xiao, X., Hecker, A., Tresp, V., and Ma. Y. Llava steering: Visual instruction tun- ing with 500x fewer parameters through modality linear representatioon-seerrigg 2025a. https:/ //attps:://riiiy arg/abs/ /2422 12359.
Bi, J., Wang, Y., Yan, D., Xiao, X., Hecker, A., Tresp, V., and Ma, Y. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection, 2025b. URL https: // /arxiv..vvv org/ abs/ /5022. 12119.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: multimodal dataset for autonomous driv- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Huang, B., Wang, X., Chen, H., Song, Z., and Zhu, W. Vtimellm: Empower Ilm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14271-14280, 2024.
Ji, W., Liu, X., Sun, Y., Deng, J., Qin, Y., Nuwanna, A., Qiu, M., Wei, L., and Zimmermann, R. Described video detection, 2024. URL https: //arxiv. www.ooog///////eeeee/e.eeeeee//e 2407 006600100..
Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., and Bing, L. Videollama 2: Advancing apatial-temporaa modeling and audio understanding in video-Ilms, 2024. URL https: //arxiv. org/ 2406. 07476.
Kung, C.-H., Lu, S.-W., Tsai, Y.-H,, and Chen, Y.-T. Action- slot: Visual action- centric representations for atomic activity recognition in traffic scenes. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 18451-18461, June 2024.
Ding, H., Liu, C., He, S., Jiang, X., and Loy, C. C. Mevis: A large-scale benchmark for video segmentation with mo- tion expressions, 2023. URL httpp:://aaattitee abs/ 2308 08544.
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., Li, H., and Qiao, Y. Llama-adapter v2: Parameter-efffcient visual instruction model. arXiv preprint arXiv:2304. 1..3304.55010 15010, 2023.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer, 2024. URL https: / //arxiv. org.ii.orrr/aeee abs//////////// 2400. 03326.
Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. Vision meets robotics: The KITTI dataset. The International Journal of Robotics Research, (2(11):23111223 August URL https:
Lin, K. Q., Zhang, P., Chen, J., Pramanick, S., Gao, D., Wang, A. J., Yan, R., and Shou, M. Z. Univtg: Towards unified video-language temporal grounding. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2782-2792, 2023. doi: 10.1100//C0510000. 2023.00262.
2013. doi: 10. 10.11177022783444114499999 // //doi. . wwgg
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., and Anirudh Goyal, e. a. The llama 3 herd of models, 2024. URL https: //a rxiv, www..oorg//ooooooo....w.. orgg abs/ abs/2407 2407 21783.
Liu, H., Li, C., Wu, Q., and Lee, Y.J. Visual instruction tun- ing. 2023. htthstp::/ /ttpaaaww....a...aaaa.//......at abs///b//aae 2304. 08485.
10

---

TUMTraf VidcoQA: Datasct and Benchmark for Unified
Video Understanding in Traffic Scenes
Liu, H., Li, C., Li, Y., Li, B., Zhang. Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, oct, and world knowledge, January 2024a. URL https://llava: 1024-01-30-1lav--ae
Computer Vision ECCV 2024: 18th European Con- ference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part LII, pp. 256-274, Berlin, Heidelberg, 2024. Springer- ISBN 978- 978-3-031-7229423 doi: 10. 10.1007/978-33-01--7999----00000 URL. https: //oii. ozg/10.1007/9799 001- 020-0007777770-----------------0 0_11
Liu, M., Yurtsever, E., Fossaert, J., Zhou, X., Zimmer, W., Cui, Y., Zagar, B. L., and Knoll, A. C. A survey on au- tonomous driving datasets: Statistics, annotation quality, and future outlook. IEEE Transactions on Intelligent Ve- hicles, pp. 1-29, 2024b. doi: 00.1109/TIV.22243334473.
Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Pat- naik, V. Tsui, P. Guo, J., Zhou, Y., Chai, Y., Caine, B., Vasudevan, V., Han, W., Ngiam, J., Zhao, H., Tim- ofeev, A., Ettinger, S., Krivokon, M., Gao, A., Joshi, A., Zhao, S., Cheng, S., Zhang, Y., Shlens, J., Chen, Z., and Anguelov, D. Scalability in perception for au- tonomous driving: Waymo open dataset, 2020. URL https:/ http://axivvvv..////v///.//...a. org/ hts//bbs///00 191..1911... 04838.
Ma, Y., Abdelraouf, A., Gupta, R., Wang, Z., and Han, K. Video token sparsification for efficient multimodal Ilms in autonomous driving, 2024. URL https: ptttts://araaa //arxiv. org/ hbs//bb//a/ abw/aass//9 2409. 11182.
Sun, P., Song, Y., Liu, X., Yang, X., Wang, Q., Li, T., Yang, Y., and Chu, X. 3d question answering for city scene understanding. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, pp. 2156-2165, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706686.. doi: 10. 1145/3664647.366 1022. 3681022.
Malla, S., Choi, C., Dwivedi, I., Choi, J. H., and Li, Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEF/CVF winter conference applications of computer vision, pp. 1043-1052, 2023.
Marcu, A.-M., Chen, L.,, Hünermann, J., Karnsund, A., Hanotte, B., Chidananda, P., Nair, S., Badrinarayanan, V., Kendall, A., Shotton, J., Arani, E., and Sinavski, O. Lingoqa: Visual question answering for autonomous driv- ing. In Computer Vision - ECCV 2024: 18th European Conference, Milan, Italy, September 99-October 2024, Proceedings, Part LXXVII, pp. 252-269, Berlin, Heidel- berg. 2024. Springer- ISBN 978-3-031-7297999 doi: 10. 10.110079978833001--999-0--- doi. 0tt.://r/.....10//////----111110 5_15.
URL https:///ddoii.. /dgg
10.1145/3666
Tang, Y., Bi, J., Xu, S., Song, L., Liang, S., Wang. T. Zhang, D., An, J., Lin, J., Zhu, R., Vosoughi, A., Huang, C., Zhang, Z., Zheng, F., Zhang, J., Luo, P., Luo. J., and Xu, C. Video understanding with large language models:
A survey. CoRR, abs/2312. 2023. URL https: / //doi. aooi.org//0... 48550/8555550
URL https://
1743217432.
Tang, Z., Liao, Y., Liu, S., Li, G., Jin, X., Jiang, H., Yu, Q. and Xu, D. Human-centric spatio-temporal video grounding with visual transformers. IEEE Transactions on Circuits and Systems for Video Technology, 32(12): 8238 8249, 2022. doi: 10.1109/TCSVT.222133338555
OpenAl, Achiam, J., Adler, S., Agarwal, S., and Lama Ah- mad, e. Gpt-4 technical report, 2024. URL https: w/arxiv.org//aaee..... abtp//b//222 2303. 08774.
Qian, R., Lin, W., See, J., and Li, D. Controllable aug- mentations for video representation learning. Visual In-
Wang, A., Chen, H., Liu, L., CHEN, K., Lin, Z., Han, J., and Ding, G. YOLOv10: Real-time end-to-end object detec- tion. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: / / openreview. net/ wwc.mmiidtaaat 3Nybb11.
telligence, 2, 2024a. doi: 10. 10.10077/44266-023000033 URL https: www.annn...rrri.eeeee.. com/ article/ 000344-
10. 1007 /
Qian, T., Chen, J., Zhuo, L., Jiao, Y., and Jiang, Y.GG. Nuscenes-qu: multi-modal visual question answering benchmark for autonomous driving scenario. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 38. pp. 1542-4550 2024b.
Wu, D., Han, W., Wang, T., Dong, X., Zhang, X., and Shen, J. Referring multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14633-14642, 2023a.
Sai, A. B., Mohankumar, A. K., and Khapra, M. M. A survey of evaluation metrics used for nlg systems. ACM Comput. Surv., 55(2), January 2022. ISSN 0360-0300. doi: 10. 15/4348566 httsttss://doi.. 1145/3485766
Wu, D., Han, W., Wang, T., Liu, Y., Zhang, X., and Shen, J. Language prompt for autonomous driving, arXiv preprint arXiv:2309.0433 2023b.
Wu, J., Jiang, Y., Liu, Q. Yuan, Z., Bai, X., and Bai, S. General object foundation model for images and videos at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3783-3795, June 2024.
Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., BeiBwenger. J., Luo, P., Geiger, A., and Li, H. Driv- elm: Driving with graph visual question answering. In

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Tempores Video Understanding in Traffic Scenes
Yan, B., Jiang, Y., Wu, J., Wang, D., Yuan, Z.,, Luo, P., and Lu, H. Universal instance perception as object discovery and retrieval. In CVPR, 2023.
Zhou, X., Liu, M., Yurtsever, E., Zagar, B. L., Zimmer, W., Cao, H., and Knoll, A. C. Vision language models in autonomous driving: A survey and outlook. IEEE Transactions on Intelligent Vehicles, pp. 1-20, 2024a.
Yang, A., Yang, B., Hui, B., Zheng. B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang. F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J. Tu, J., Zhang, J., Ma, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Ge, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z., and Fan, Z. Qwen2 technical report. arXiv preprint ar (rXi:2440711067/ 2024.
doi: 10. 00.1009/TT12222444421113
Zhou, Y., Huang. L.,, Bu, Q. Zeng, J., Li, T., Qiu, H., Zhu, H., Guo, M., Qiao, Y., and Li, H. Embod- ied understanding of driving scenarios. arXiv preprint arXiv:2403.0443 2024b.
Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., and Darrell, T. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2633-2642, 2020. doi: 10.1109/ CVPR42600.2020.00277
Yuan, T., Zhang, X., Liu, K., Liu, B., Chen, C., Jin, J., and Jiao, Z. Towards surveillance video-and-language understanding: New dataset, baselines, and challenges. In 2024 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 22052-22061, 2024. doi: 10. 101109CCVP822233322200000000
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In 2023 IEEE/CVF International Conference on Computer Vi-
sion (ICCV), pp. 11941-11952, 2023. doi: 10.1109) ICCV51070.2023.01100
Zhang, D., Zheng, H., Yue, W., and Wang, X. Advancing its applications with Ilms: A survey on traffic management, transportation safety, and autonomous driving. In Hu, M., Cornelis, C., Zhang, Y., Lingras, P., Skezak, D., and Yao, (eds.), Rough Sets, pp. 295-309 Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-65668.
Zhang, Z., Zhao, Z., Zhao, Y., Wang, Q.. Liu, H., and Gao, L. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In CVPR, 2020.
Zhao, Y., Lv, W. Xu, S., Wei, J., Wang, G., Dang, Q., Liu, Y., and Chen, J. Detrs beat yolos on real-time object detec- tion. In 2024 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16965-16974, 2024. doi: 10. 00.100//VVP55273..22220 2022000035.....
Zhou, X. and Knoll, A. C. Gpt-4v as traffic assistant: An in-depth look at vision language model on complex traf- fic events, 2024. URL https: / /arxiv...../ org/ abs/ 2402. 02205.
12

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Opatio-Tempora Video Understanding in Traffic Scenes
A. TUMTraf Vi Dataset
A.1. Dataset Statistics
December
400
November
October
dia-daay
300
2D
September
August
700
Month
July-
June
100
10
Hour
April
0
January
Februery
March
April
Moy
june
July
August
September
October
Nevember
December
February
January
10
15
Highway
Intersection Type
Country Road
Time Day
Month
(a) Temporal Distribution of Vidco (b)
Weather-Baset Distribution of (c) Scene Distribution Across Different Perspec-
Weather Conditions Over the Years. Videos.
tives.
Figure 7: Dataset distribution of video recordings by time, weather conditions, and perspectives.
The video selection process is meticulously designed ensure comprehensive coverage of diverse daytime periods, weather conditions, road types, etc. The distribution of the video statistics in the TUMTraf VideoQA dataset is illustrated in Figure 7. Figure 7a provides overview of the distribution of videos by hour of the day and month, with weather conditions represented through color coding. The majority of traffic footage was captured between 5:00 AM and 8:00 PM, with fewer recordings available during hours with limited natural light. Figure 7b illustrates the distribution of videos by weather conditions for each month. The dataset predominantly includes videos recorded between February and May, a period characterized by a wide variety of weather scenarios, thereby enhancing the dataset's 's representativeness. Figure 7c depicts the distribution of video recordings by hour of the day for each camera type and camera. The three camera nutegories--urveilli cameras positioned on highways, intersections, and country roads-are are represented proportionately. ensuring video coverage across these categories from dawn to nighttime.
Yesa
are
Answer Count
16000
Eskoecce So Grounding
No
12000
Couttey
8000
Desciiter poter Class
D video
One
4000
B
five
Zero
than cars
and
10
50 60 Number of Words
20
30
40
(a) Word Cloud Visualization of Multi-(b) Burst Figure of Questions in Multi-(c) Length Distribution of Different Ques- tion Types.
Choice QA.
Choice QA.
Figure 8: Distributions of video recordings across time, weather conditions, and camera types in the dataset.
In addition to video statistics, Figure 8 illustrates the distribution and characteristies of annotations in the TUMTraf VideoQA dataset. Figures 8a depict word clouds for answers across all three tasks, highlighting common terms and their frequencies. Figure 8b presents sunburst chart that visualizes the distribution of question formats, revealing that most questions begin with "How," "What," and "Can". Figure 8c shows the distribution of answer lengths, indicating that the majority of answers consist of fewer than 10 words, with only a small number exceeding 19 words.
A.2. Spatial Question Curation
Comprehending spatial relationships in 3D space is a critical challenge in traffic scene analysis. In our semi-automatic annotation pipeline, we calculate spatial locations by projecting 2D coordinates into 3D space under the planar assumption, leveraging historical camera intrinsic and extrinsic matrices. Specifically, from a third-party roadside perspective, we formulate spatial reasoning questions by treating each object as an ego-centric reference and formulate the questions that reveal its 3D positional relationships with surrounding traffic participants.
13

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified
Video Understanding in Traffic Scenes
front Leftt
front right
Readlade Sensorrs
front
if 15t < 0 15
front left
15 < < 75
left
right
left
75 8 S 105
front right
if 75" < S -150
relative position
right
if 105 < S -75
back left
if 105"
<0 165
back
back
back right
165 < < - -155
back
else.
beck
(8)
Figure 9: Illustration of the eight spatial regions used to categorize the relative positions of objects in traffic scenes. In this example, the orange car is located to the front right of the black car.
We focus objects that remain in motion throughout the video. The motion direction of each object is computed based on the difference between its 3D coordinates in consecutive frames. To determine the relative position between two objects, we measure the angle f between the motion direction of the moving object and the vector connecting it to another object. Subsequently, the relative position of the second object with respect to the moving object is classified according to the angular criteria defined in Eq. 8. We then divide the spatial relationship into eight distinct regions: front, front left, left, front right, right, back left, back right, and back. Figure 9 illustrates the angular division used classify the relative position of objects in TUMTraf VideoQA dataset.
B. Benchmark Analysis
B.1. Impact of Frame Number on TralliX-Qwer Performance
Table 7: Impact of the number of frames on the performance of the TraffiX-Qwen baseline model on the validation set. We report results using spatial pooling as the sampling strategy.
Object Captioning METFOR
Spatlo-Temporal Grounding
Multi-Cholce QA
Models
BLFUJ
SPICE
Temp. Fill
Spa Fqj
SP Fq.
Positioning
Counting
Motion
Class
Existence
Overall
Tramix-QQeen-.58
without
31.78
32.94
39.72
0.1332
0.1979
0,2739
70.18
59.98
80.45
56.74
70.62
67.59
33.31
34.10
42.89
0.1205
0.1913
0.2601
71.15
76.92
82.56
67.13
84.01
76.35
11
34.58
34.85
45.12
0.1220
0.1888
0.2594
71.92
76.89
82.94
70.54
88.84
78.97
72.00
Dill.
+3.21
+2.30
+6.63
0.0112
400087
-0.1399
+1.82
+20.58
+2.14
+13.10
+18.59
+11.25
TrafiX-Qeen-7R
without
31.80
34.66
40.74
0.1332
0,1905
0.2710
73.32
63.93
81.44
58.65
77.72
71.01
33.06
35.16
44.33
0.1094
0.1791
0,2418
76.50
78.23
83.36
69.12
84,78
78.20
76.50
101
36.74
36.42
47.32
0.1083
0.1737
0.2382
76.56
81.94
84.33
73.42
89.44
81.14
Diff.
+4.4
+1.76
+6.58
-0.0249
-0.0168
-0.0328
+3.24
+18.01
+2.89
+14.77
+11.72
+10.13
To assess the extent to which baseline model learns from visual tokens and how much it attempts to fabricate answers, we conduct a series of ablation studies. We investigate the impact of the number of frames on TUMTraf VideoQA performance, detailed in Table 7. Additionally, we include extreme where visual information is provided the model, and the train baseline model was prompted to answer questions directly.
The experimental results reveal intriguing phenomena in both the 0.5B and 7B models. First, when no visual input is provided, and the model relies solely on the question to generate answers, the baseline model could still reach relatively high performance across all three tasks. This demonstrates the model's 's inherent reasoning capabilities are probably derived from the question alone and highlights that, in domain -ppecffcc datasets such as traffic scenarios, the model appears to learn
14

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
and exploit underlying text-based patterns and biases present in the data, which may contribute its ability to fabricate seemingly accurate responses without actual visual grounding.
Besides, introducing visual input is found to be crucial for correctly solving TUMTraf VideoQA tasks. Across all three tasks, the results consistently show that increasing the number of input frames will improve model performance. Notably, the improvements are most pronounced when moving from no visual input to frame and from 1 frame to 11 frames. However, the performance gains became less significant when increasing the input from 11 frames to 101 frames. This diminishing improvement may be attributed to the inherent difficulty of LLMs in effectively extracting visual context from a large number of tokens. For the 0.5B baseline model, the performance with 11 frames is nearly equivalent to that with 101 frames, reflecting its relatively limited in-context learning capabilities. Therefore, effectively representing video data and addressing the hallucination problem of VLMs in such domain-specifie scenarios are critical directions for future research.
Furthermore, the increase in the number of frames has varying impacts on different task types, with substantial differences observed. This variation also indirectly reflects how much the model learns from visual input and how much affects the reasoning process. For Multi-Choice QA tasks, the gains for positioning and motion categories are the smallest, ranging from only 1.82% 3.24%. indicates that the model still struggles to extract answers from visual information effectively based on the current model architecture. In contrast, for counting, class, and existence tasks, the performance improvements exceed 10%, which suggests that VLMs effectively extract features and answer questions in these cases.
B.2. Performance Evaluation under Consistent Frame Settings
Table 8: Performance of various models under unified 101-frame input setting. Qwen2-VL uses 96 due to architectural constraints. We report results across three sub-tasks: object captioning, spatio-temporal grounding, and multi-choice QA.
Object Captioning
Grounding Spa.El
Multi-Choice QA
Model
Size
Overall
BLEU 4
ROUGE L
CIDEr
Temp.E)
STEE
Positioning
Counting
Motion
Class
Existence
LLaVA-One Vision
0,5B
0.0
0.9
0.0
1.00
1.00
1.00
25.6
25.1
11.9
12.4
0.6
15.1
7B
0.0
0.5
0.0
1.00
1.00
1.00
26.4
24.1
13.6
13.0
0.8
15.6
Qwen2-VL
2B
3.8
15.1
0.18
0.65
0.68
0.73
31.2
38.4
58.2
55.4
74.5
51.5
7B
5.0
14.9
0.14
0.70
0.70
0.76
31.8
55.3
54.3
50.0
75.8
53.4
Video LLaMA2
7B
0.0
7.1
0.0
1.00
1.00
1.00
27.6
27.5
26.3
18.7
52.2
30.4
Trafli--wwer
0.5B
35.0
50.4
2.52
0.12
0.19
0.26
72.0
80.6
82.6
69.8
89.2
78.8
7B
36.7
52.0
2.56
0.11
0.17
0.24
76.6
81.9
84.3
73.4
89.4
81.1
To further clarify whether TraffiX- -eexx---eenss 's performance advantage is due to more frame input, we conduct additional experiments using a unified 101-frame setting for all evaluated models. As open-source video LLMs models often adopt model-specific frame sampling strategies that are tightly coupled with architecture and training paradigms, many existing models (e.g., LLaVA-One Vision, Video-LLaMA2) are not specifically designed handle long video sequences.
As shown in Table 8, the results reveal several key observations. LLaVA-One and Video-LLaMMA exhibit notably poor performance when evaluated with 101-frame inputs, indicating their limited capability in modeling long-rang temporal dependencies. In contrast, Qwen2-Vi VL demonstrates improved performance with increased frame input, suggesting that its architecture can leverage additional temporal information to enhance reasoning. Most notably, TraffiX- consistently achieves superior results across all metrics, even under the same input frame conditions. These results show that TraffiX- Qwen achieves robust fine-grained video understanding not merely through longer inputs, but through effective temporal reasoning and structured object representation.
B.3. Visualization of Multi-Choice QA Results
Figure 10a presents a radar chart depicting the performance of open-source models on the Multi-Choice QA task. The results indicate substantial variability in zero-shot performance across different question types, with each model exhibiting strengths in specific categories. Notably, tasks requiring positioning skills, such as 3D scene understanding, pose significant challenges for all models, suggesting that this question type demands advanced spatial reasoning capabilities, which remain a limitation for current LLMs.
Figure 10b illustrates the performance of TUMTraf VideoQA fine-tuned baseline models. Fine-tuning leads to a notable improvement in overall performance, particularly for the 7B parameter model, which consistently outperforms the lightweight
15

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spapio-Temporal Video Understanding in Traffic Scenes
0.5B model across multiple dimensions. However, the performance gap is not overwhelmingly large, indicating that lightweight models retain considerable practical value and effectively handle the majority of tasks.
Class- E (57.06)
Motton-E (61.65)
Cless-E
Motion-E (08.29)
Existence-E (85-29)
Counting-E E (98.42)
Existence- - (23.44)
Counting-f (87.53)
Positloning-
Positioning [78.89]
55
Posttioning. (77.44)
(26.05)
Counting-H H (54.85)
Existence-H H- (73.03)
Counting-Ht (78.22)
Existence-
48) Motion-H
(56.50) Class-H
(82.88) Motion-H
(65.5
class-H
(a) Performance radar chart of the open-source models the TUMTraf VideoQA Multi-Choice QA task.
(b) Performance radar chart of the TraffiX-Qweet baseline on the TUMTraf VideoQA Aulii-Choic QA task.
Figure 10: Results visualization for the open-source models and Traffi-- baseline models the Multi-Choice QA.
16

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Tempores Video Understanding in Traffic Scenes
B.4. Example of MultiRes Token Pruning
We present several examples of multi-resolution similarity- -based token pruning techniques applied to video data from our dataset. As shown in Figure 11, while this approach maintains high resolution to certain extent, its lack of semantic- aware selection capabilities may result in the loss of task-critical information in certain scenarios. Specifically, it mainly preserves visual tokens for moving vehicles and dynamic objects, such swaying trees, while pruning stationary vehicles as background information due to their lack of motion. It shows its effectiveness in separating dynamic objects from static backgrounds but also highlights the need for improvement handling the rest of the important traffic participants.
PITESAANAR
Illustration of cosine similarity based token pruning, with dark-colored patches representing discarded tokens and preserved ones highlighted. We demonstrate the three samples on highways, country roads, and intersections separately.
Figure 11:
B.5. System Prompt
We craft dedicated system prompt for our experiments with the TUMTraf VideoQA dataset. Figure presents the prompt used the experiments. The prompt is adopted across both open-source models and fine-tuned Fraffi-Qwen baseline ensure fair and consistent evaluation across different models.
System Prompt:
You are an Al assistant specializing in the analysis of traffic scenes from surveillance footage. Each object's 's position at a specific moment in the video is represented as a tuple: (c, nf, x, y), where c is the unique identifier for the object, nf is the normalized timestamp of the frame (a float between and 1), and x and y are the normalized coordinates (also between 0 and 1) of the object's position within that frame. Provide precise and informed responses the questions.
Figure 12: The system prompt used in the experiments of TUMMTraffi-Videoo/// dataset.
17

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
B.6. Qualitative Evaluations of Spatio-Temporal Object Grounding
Figures 13 through 16 illustrate several qualitative examples of spatio-temporal object grounding, highlighting the challenges and limitations of the task. Figure 13 presents example where the referred object fire truck parked at the roadside, visible throughout the entire video from start to finish. The baseline 0.5B model demonstrates satisfactory temporal localization but exhibits some inaccuracies in spatial localization. In contrast, the baseline 7B model achieves more accurate spatial localization but only identifies the temporal range from 0.2s to 2.95s.
Q: Can you locate [[c12, 0.03, 0.05, 0.26)] in the traffic video and provide the standardized spatiotemporal localization for its first and last visible frames?
FRRDEAAEE Tome CD
1:A0 S01E Troe
DAO A0 Tomn et
[[c12, 0.01, 0.05, 0.26), (c12, 1.0, 0.05, 0.26)]
Gre306800000 Fraw
2-2000 J0 Tork 2a06
Ground Truth
Baseline-
Baseline-7B
Figure 13:
Spatio-Temporte Object Grounding: A fire truck parked at the roadside.
Figure 14 depicts white car moving along country road, appearing in the video from 10. 10s until the end. The baseline model predictions indicate that the 0.5B model provides a relatively accurate estimate of the initial position, whereas the 7B model exhibits a greater deviation in its ending location.
Q: Can you track [(c14, 0.75, 0.55, video and return the standardized spatiotemporal localization for the first and final frames where it appears?
0.27))
GT: [(c14, 0.51, 0.26), (c14, 0.97, 0.65, 0.36)]
Question
Ground Truth
Basellne-0.58
Basellne-7B
Figure 14: Temppral Object Grounding: A white car moving along a country road.
18

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
Figure 15 presents the grounding result of white sedan in nighttime scene. Due the object's considerable distance in the reference frame, it appears quite small and makes feature extraction challenging. Additionally, due its extended temporal span, the model struggles with cross-frame object association. As result, both the 0.5B and 7B models fail to accurately capture its end position, instead predicting minimal spatial displacement. This highlights the difficulty of grounding objects with large temporal windows, where precise localization over time remains significant challenge.
Q: Could you track [lc6, 0.61, 0.75, 0.02]] and provide the standardized spatiotemporal localization for both the initial and tinal trames its appearance?
DE2366,7100 Frow
OCC SHOCBOO Pore 1311
READEEANACE frow 1230
GT: [(c6, 0.59, 0.77, 0.01), (c6, 0.99, 0.36, 0.44)]
14.3 SU00O
GA wyp 22-00000 2640
GA KOUAEOOO
(AS Kipo hore 14.00
Question
Ground Truth
Baselino-0.5B
Baselino-7B
Figure 15:
Object Grounding: A white sedan in nighttime scene.
In Figure 16, we show an example of temporal grounding for motorcycle at the intersection. Compared to big cars, the grounding of vulnerable traffic participants much more challenging. Both the 0.5B and 7B baseline models fail effectively localize the motorcycle in either the temporal spatial domain, highlighting the difficulty of the task for smaller and less distinct objects.
you find [[c18, 0.86, 0.85, 0.43]] in the traffic video and provide the standardized spatlotemppral localization both the and last frames appears in?
GT: [(c18, 0.83, 0.87, 0.36), (c18, 0.91, 0.75, 0.7))
Question
Ground Truth
Basellne-0.5B
Baseline-7B
Figure 16: Temporal Object Grounding: motorcycle moving through an intersection.
19

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
B.7. Qualitative Evaluations of Referred Object Captioning
In this section, we present several examples from the referred object captioning task. The left side of each image shows the object to be described, while the right side includes the task description, the corresponding ground truth, and the responses generated by the 0.5B and 7B Traffix-Qwen bascline models. We prompt the model with the question using list of two tuples that indicate its Spatio-temporal position at two specified timestamps. The experimental results, evaluated using multiple NLG metrics, reveal that the 7B model achieves higher accuracy in deseribing the appearance details of target objects. However, despite its smaller parameter size, the 0.5B baseline model is also capable of generating satisfactory descriptions, demonstrating its potential practicality in resource-constrainee scenarios.
Figure 17 presents sample to describe occluded white van. Both the 0.5B and 7B models from the TraffiX- -wwei baseline accurately identify the vehicle boxy-shaped white van. However, the 0.5B model introduces extra hallucinations and incorrectly describes the van as having Volkswagen logo, which is not present in the image. Both the 0.5B and 7B models achieve relatively high metric scores, with the 7B model performing better, particularly in BLEU-4 and SPICE.
(+7,0.01 (G7.0.01.0.2..0333 0.2..0.33) Frome 0.30 sec
Q: What are the main characteristics of [(c7, 0.01, 0.22, 0.33), (c7, 0.21, 0.1, 0.89)]?
GT: [(c7, 0.01, 0.22, 0.33), (c7, 0.21, 0.1, 0.89)] white van with a boxy shape. features distinctive front grille with logo, and has high roof design, which suggests ample cargo space. The vehicle appears to have large headlights and a simple, utilitarian design.
Baseline-O.55 0.50:: (c7, 0.01, 0.22, 0.33), (c7, 0.21, 0.1, 0.89)] is a white van with a boxy shape and a distinctive front grille featuring the Volkswagen logo. has large windows and a high roof, providing ample cargo space.
(s7.0.21,0.1,0088) Frame 6.30
BLEU-4: 0.41 ROUGE-L: 0.70 METEOR: 0.47 SPICE: 0.72
Baseline-7B: [(c7, 0.01, 0.22, 0.33), (c7, 0.21, 0.1, 0.89)] is white van with boxy shape. It features a distinctive front grille and has headlights that are visible. The vehicle appears to have high roof, providing ample cargo space.
BLEU-4: 0.58 ROUGE-L: 0.72 METEOR: 0.53 SPICE: 0.81
Figure 17: Referred Object Captioning Example: A partially occluded white van with boxy shape.
Figure 18 illustrates scenario to describe a dark-colored sedan based on two perspectives captured at different timestamps in the video. The ground truth description from ChatGPT-40 accurately specifies the color as dark purple, while the TraffiX-Qwen baseline, with both the 0.5B and 7B version, classify the vehicle color as black, a visually similar designation. Regarding vehicle type, the 0.5B model identifies it as a hatchback, whereas the 7B model recognizes it as an SUV. Moreover, the 7B model detects distinctive alloy wheels, aligning with the description in ground truth. The quantitative evaluation across four metrics indicates that the 7B model slightly outperforms the 0.5B model, with the most significant improvement observed in the SPICE metric.
Figure 19 presents case where the question refers to bus with distinctive green roof. In the Traffix- fraffx-Qwen baseline, the 0.5B model incorrectly describes it as a white van with a boxy shape, whereas the 7B model accurately identifies it as a bus with green and white colors and provides a corresponding detailed description. It shows that the 7B model achieves better performance than the 0.5B model for this sample. However, in terms of NLG metrics, both descriptions receive the same ROUGE-L score, which not a reasonable reflection of their accuracy differences. Among the four reported metrics, SPICE captures the quality of descriptions more effectively. To address such limitations, some studies have introduced LLMs-based evaluation metrics for assessing model performance, which will be explored part of our future work.
20

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temports Video Understanding in Traffic Seenes
0: 42.68) Fromé 0:10 sea
Q: What are the main characteristics of [(c1, 0.01, 0.42, 0.87), (c1, 0.97, 0.58, 0.17)]?
GT: [(c1, 0.01, 0.42, 0.87), (c1, 0.97, 0.58, 0.17)) is a dark purple coupe with a sleek design and distinctive rounded roofline. It features sporty alloy wheels and modern rear light design.
Baseline- 0.50:: [(c1, 0.01, 0.42, 0.87), (c1, 0.97, 0.58, 0.17)] is a black hatchback vehicle with a sleek design. It features distinctive rear lights and modern tailgate. The vehicle has a compact size, making it suitable for urban environments.
KeT
14.45
BLEU-4: 0.33 ROUGE-L: 0.61 METEOR: 0.38 SPICE: 0.38
Baseline-7B: [(c1, 0.01, 0.42, 0.87), (c1, 0.97, 0.58, 0.17)) is a black SUV with a sleek design. It features distinctive alloy wheels and has modern aesthetic, characterized by smooth lines and spacious appearance.
BLEU-4: 0.39 ROUGE-L: 0.69 METEOR: 0.42 SPICE: 0.47
Figure 18: Referred Object Captioning Example: A Mark-purple-colore sedan from two perspectives.
G4.00001
1,0.35)
Frome
Q: What are the main features of [(c4, 0.01, 0.21, 0.35),
(c4, 0.98, 0.29, 0.41)] captured in the traffic footage?
GT: [(c4, 0.01, 0.21, 0.35), (c4, 0.98, 0.29, 0.41)] is a green bus with distinctive design featuring multiple roof vents. has streamlined shape typical of public transport vehicles.
Baseline-0.5B: [(04, 0.01, 0.21, 0.35), (c4, 0.98, 0.29, 0.41)] is white with a boxy shape and distinctive front grille featuring the Volkswagen logo. It has large windows and high roof, providing ample cargo space.
(c4,0.58
3,0.41) Frame
BLEU-4: 0.29 ROUGE-L: 0.58 METEOR: 0.32 SPICE: 0.34
Baseline-7B: [(04, 0.01, 0.21, 0.35), (c4, 0.98, 0.29, 0.41)] is bus featuring green and white color scheme with blue accents. It has a distinctive design with prominent green roof and blue accents along the lower part of the vehicle.
BLEU-4: 0.30 ROUGE-L: 0.58 METEOR: 0.34 SPICE: 0.48
Figure 19: Referred Object Captioning Example: bus with a distinctive green roof.
21

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spapoo-Temporal Video Understanding in Traffic Scenes
C. Dataset Examples
C.1. Sample Videos
The TUMTraf VidcoQA dataset encompasses a diverse and highly engaging collection traffic scenarios, capturing a wide range of complex real-world traffic situations and weather conditions. These scenarios cover various traffic dynamics and environmental factors, making the dataset suitable for evaluating models across different conditions. We showcase several representative scene types to illustrate the diversity and characteristics of our dataset more intuitively.
(a) Accident
Rescue
(c) Traffic Jam
(d) Fog
(e) Snow
(f) Rain
g Dawn Dusk
22

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
The depicted scenarios include but are not limited to: Traffic Accidents 20a, demonstrating various types and severities of collisions; Rescue Operations 20b, capturing emergency vehicle actions under special circumstances; Traffic Jams 20c, reflecting congestion during peak hours unexpected events; and scenes under diverse weather conditions, such Fog 20d, Snow 20e, and Rain 20f, showeasing the dataset's adaptability to complex environments. Additionally, the dataset includes scenarios with unique lighting conditions, such as Dawn and Dusk 20g, simulating traffic dynamics low-light settings.
C.2. Question Templates
In this section, we provide some representative examples of question templates for each task. Figures 21 through 25 show templates for the five categories in the Multi-Choice QA task. Figure 26 provides templates for the Spatio-Temporal Object Grounding task and Figure 27 presents templates for the Referred Object Captioning task.
Question Templ ate Examples for
Easy: enay:
Q: What do you to the {relative-position) of {object-id] at normmalized_framee of the video duration?"
O: What exists to the {relative.position) side of cobject_id} at (normalized, frame} of the video duration?"
Q: What is present to the frelative.
( of {object.id} at [normalized_framal of the video duration?"
Q: What can be observed to the (relative position] of {object id} at (normalized frame) of the video duration?"
Q: " What is on the drelative.positiinn side relative to {object.id] i at [normalized_frant
the video duration?"
: " What can be found the frelative position) of {object id) (normalized frame} of the video duration?"
Q: What
you observe the drelative_positions side of {object_id) at porrmalized_rrang } of the video
duration?"
Q At /normalized frame) of the video duration, what is visible to the (relative ) of {object.id]) id}
Q: At commalized_frame of the video duration, what can be seen to the (relative-position) of {object.id]??
Question Template Examples for Positioning Hard:
Q: "How is {object id 2} positioned with respect to {object id 1} at (normalized frame) of the video duration?"
Q: "Can you specify the location of {object.id relative to {object.id.1} at (normalized_franl of the video duration?"
Q "What is the relative location of (object_id_2) to {object_id_1) at (normalized_frant) of the video duration?"
Q "Can you describe where {object id 2} is in relation to {object id 1} at [normalized frame} of the video duration?"
Q: WWere idd located relative to {dbject.id_1} at (normalized _ramme) of the video duration?"
: "Can you describe the relative position of {object_id_2) {ohject_id_1) (normalized_framl of the video duration?"
: Where (object id 2} be found relative to (object id 1) (normalized frame) of the video duration?"
Can you tell the relative location of (object.id.2) compared to {object.id_1) at (normalized_framm of the video duration?"
Figure 21: Example Positioning question templates. {object.id}. {objectidd_1) and {object.dd_2) represent the objects be- ing inquired about, (normalized frame) is placeholder for specific moment the video duration, and (relative sellaatvveppsition) represents the relative position.
23

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes
Question Template Examples for Counting -an-tyyyg-as
Q: How many {class name pl] are captured in the video?"
Q: " How many {class_name.pp are visible in the traffic footage?"
Q: " How many {class nclass_name nam.-aammmp...e-- pl] can you detect the video?"
Q: " Howw many {class [class_name name pl} are observable in the video?"
Q: HHow many {class_name-p)
can be seen in the video?"
Q: Hoow many instances of {class_name ppl} are there in the video?"
Q: 'How many (class_name.ph) does the video show?"
Question Template Examples for Counting Hard:
Q: "Can you identify how many (class.name.pl are always (motion_status] in the video?"
Q: "In the traffic footage, how many {class_nnd _name_pp are amotion_statuus for the whole duration?"
Q: " "How many {class name pl} are present to the {relative } side of {object id] at (normalized frame) of the video duration?"
At (normalized frame) of the video duration, how many {class pp] exist to the (relative.position of [object_id]?"
Q: "How many {class name name j are visible to the (relative.positeee side of {object at (normalized frame) (faamm) the video duration?"
Figure 22: Example Counting question templates. "class_name.ppp } is placeholder for the plural form of the object class be- ing inquired about, fobject id} is placeholder for the representation of the object being inquired about, [normalized frame) is placeholder for specific moment in the video duration, {relative [relative_position represents the relative position, and mootion (motion_status] is placeholder for the motion status.
Question Temple ate Examples for
Q: What is the moving status (object id]?"
Q: " Can you report the motion status of (object.id]?"
O What's the movement state of (object.id]?"
Q: What's the movement status of (object id)?"
Q What's the activity status of (object_id]?"
: How would you describe the motion status of (object i
Q What is your description of {object.id}ss 's motion status?"
Q: How would you define the movement status of idbject.id??"" 2"
Q: " "Can you outine the motion status of (object.id)?""
Question Template Examples for Mot Motion-Haad:
Q: "Is the motion status of {object.id.1) equal to that of 2
Q: Are (object id }} and (object id 2} in the same motion state?"
Q: "Is (object_id__]" motion status equivalent to {object_in sidettddd_2] "2)
Q: "Are the motion statuses {object_id id_1 and {object id } the same?"
Q: " Are the motion states of [object_id_1 and (object_id_2) the same?"
Q: "Do (object id and (object id 2} have matching motion statuses?"
Q: "Is {object.id_I)) motion status identical to (object.id.2) "d""""" "s"
O: "Do id and {ddject.id_2) share the same motion status?"
Figure 23: Example Motion question templates. (object.id}, (object.id_I) and {object.id 2d represent the objects being inquired about.
24

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Temports Video Understanding in Traffic Scenes
Question Template Examples for Class-Easy:
Q: Which class does (object id] belong to?"
Q: " What category {object_id) classified under?"
Q: " What is the classification of {object id)?"
Q: " How is {object_id] categorized?"
Q: What class label can be given to {object.id)?"
Q: What is the specific class of [objectt.d]?""
Q: "How is {object } classified?"
Q: What is the category classification of {d]
Q: What type object is {ddj
Question Template Examples for Class-Haree
:: Are the fobject id and {object.id iigject_id_/ id 2) of the same type?"
Q: "s the class of {object_id_1} the same as the class of (object_id_2)?"
: "Do {object id.1} and {object id 2 belong to the same category?"
Q: Are {object-id_1] and (object_id_2) from the same category?"
Q: "Are the classes {object.id_1) and {object.id _d_ identical?"
Q: Doo the classes of (ohject_id_1) and (object_id_2) match?"
Q: " Does (object_id_1) belong the same category as {object.id.2)."
Q: "Do {object id } and {object id id.2 2) share the same class?"
Q: "Is {object.id_1) in the same class category as idd))ct.id.2)?""
Figure 24: Example Class question templates. {object id), (object id 1}, and {object id 2) represent the objects being inquired about.
Question Template Examples for Existence-Easy:
Q: "Are there any (class_name.pp) visible in the video?"
Q: " Are any {class name pl) present the traffic footage?"
Q: " Do you spot any {class_named -ttassnnameppll } in the video?"
Q: Are there any instances of fclass name pl} visible in the traffic footage?"
Q: Do any _class_name_pl) make appearance in the video?"
Q: Can you spot any (class_name name pl} within the traffic video?"
Q: "Can you confirm the presence of (class _ame } in the video?"
Question Template Examples for Existence-Hardi
Q: " Aree there any {class_nane sas_nnam_ppp that are innotionsstatus for the whole video?"
Q: "Do any (class name pl} appear to be (motion status) for the whole duration of the traffic footage?"
: " Can you spot any (class_name.pl) to the {relative.position)) of {object.id} at (normalized_frang) of the video
duration?"
Q: "Can any (class_name.p) be seen to the (relative_position of {object_id) at (normalized_franc of the video duration?"
: Are there (class name_pll pp} visible to the (relative.position) side of {objectidd} at (normalized_framl the video duration?"
Figure 25: Example Existence question templates. {class_nanne _pm is a placeholder for the plural form of the object class be- ing inquired about, (object is placeholder for the representation of the object being inquired about, [normalized_ placeholder for a specific moment in the video duration, "relative_position represents the relative position, and [motion_statie is a placeholder for the motion status.
25

---

TUMTraf VidcoQA: Dataset and Benchmark for Unified Spatio-Tempont Video Understanding in Traffic Seenes
Question Template Examples for Spatio Peatoo-Tempora Object Grounding:
Q: "Can you track {object.id} in the traffic video and submit the standardized spatiotemporal localization for the
first and final frames where it appears?"
Q: :Can you trace {object_id} in the traffic footage and provide the standardized spatiotemporal localization for both the first and last frames of its presence?"
Q: "Can you find {object.id] in the traffic video and provide the standardized spatiotemporal localization for both the first and last frames it appears in?"
Q: "Can you identify {object_id} in the traffic video and provide the standardized spatiotemporal localization for its first and last visible frames? The output should consist of two tuples formatted as (id, nf, x, y), where is the object's unique identifier,
the normalized frame number detection, and x and y are the normalized coordinates of the bounding box center in each frame."
Q: "Can you locate (object_id) in the traffic video and provide the standardized spatiotemporal localization for its first and last visible frames? The output should be list containing two tuples, with each tuple structured as (id, nf, x, y). In this format, id denotes the unique identifier of the object, nf represents the normalized frame number in which the object is detected, and x and are the normalized coordinates of the object's bounding box center within the respective frame."
Figure 26: Example Temporal Object Grounding question templates, {object id} is placeholder for the representa- tion of the object being inquired about.
Question Template Examples for Referred Object Captioning:
Q What are the main features of (object captured in the traffic footage?"
Q: What are the key details of {object id)?"
Q: " What are main visual characteristics (object.id]?"
Q: " What are the main characteristics of (object id}??
Q: What the distinguishing features of {object_id)?"
Figure 27: Example Referred Object Captioning question templates. [object.id] is placeholder for the representation of the object being inquired about.
26


Reviews:
[{"review_id": "S3QU10DNuk", "reviewer": "Reviewer_9vbN", "summary": "This paper provides a comprehensive dataset tailored for multiple tasks in traffic scenarios. It includes QA such as predicting the weather, counting objects, providing motion status, spatio-temporal grounding, and more. It consists of 1,000 videos, with 85,000 QA pairs, 2,300 object captioning, and 5,700 object grounding annotations. The benchmark tasks include video QA, object captioning, and grounding. The accompanied baseline model is based on Qwen-2 plus multiple important visual modules such as multi-resolution token sampling and temporal/spatial/token pooling. The paper provides the results from both the baseline model as well as other open-sourced multimodal LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your positive feedback and insightful questions. We provide detailed responses to your concerns below.\n\nQ1: Would the authors provide a brief discussion on how this dataset, especially the QA part, will remain challenging for upcoming new multimodal LLMs given the high baseline results?\n\nA1: \nDuring our literature review, we also observed that some VQA benchmarks in vertical application domains, such as earth observation, autonomous driving, and smart cities, tend to achieve relatively high baseline accuracy (generally > 60\\%) after fine-tuning (e.g., EarthVQA[1], nuScenes-QA [2], City-3DQA [3]). Our dataset shows a similar trend, indicating that large VLMs can more easily adapt to constrained, domain-specific tasks. However, we believe that the high baseline accuracy does not indicate that the task is simple or lacking in meaningful challenges. Our benchmark involves multiple complex tasks that remain crucial for developing future multimodal LLMs, especially those for traffic-related applications. Below, we provide a detailed discussion:\n\n1.\tOur benchmark unifies three core tasks, i.e., multi-choice QA, referred object captioning, and spatiotemporal object grounding through tuple-based spatiotemporal object expressions for traffic scene understanding. While the QA task demonstrates relatively high accuracy, the other two tasks still exhibit substantial room for improvement. Importantly, as discussed in the paper, techniques that enhance QA performance, such as visual token strategies, do not consistently benefit the other tasks. This reveals that optimizing for QA alone is insufficient, and highlights the need for holistic model designs that jointly address all tasks.\n\n2.\tEven within the multi-choice QA task, simply scaling model size yields diminishing returns. Although the 7B model achieves higher accuracy, the improvement over smaller models (e.g., 0.5B) is marginal. This suggests that simply increasing LLM size or replacing it with newer variants alone is unlikely to bring large improvement in this domain. Together, these findings emphasize the importance of domain-specific model design and training strategies that consider multiple tasks tailored to traffic-centric understanding.\n\n3.\tIn real-world traffic applications, where computational resources are often limited, developing lightweight models becomes a key priority. Rather than focusing solely on improving benchmark scores, we encourage future work to explore trade-offs between performance and efficacy with lightweight LLMs and techniques (i.e., pruning, quantization, and knowledge distillation).  Our benchmark and baseline models provide practical references to guide and evaluate such efforts toward building efficient, real-time multimodal systems.\n\nIn summary, we believe our dataset and benchmark remain highly important for the development and evaluation of upcoming multimodal LLMs, particularly those tailored to traffic domains. It presents practical challenges, encourages efficient model development, and supports a holistic evaluation across multiple multimodal tasks.\n\n[1] Junjue Wang,  et al., “EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answering”\n\n[2] Tianwen Qian, et al., “NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario”\n\n[3] Penglei Sun, et al., “3D Question Answering for City Scene Understanding”", "claims_and_evidence": "The paper’s data analysis, model comparisons, and ablation studies show how existing open-sourced models perform relatively poorly with fine-grained spatio-temporal tasks, and the authors’ enhanced baseline with multiple techniques still shows the dataset’s complexity.", "methods_and_evaluation_criteria": "Yes. The tasks—multi-choice QA, spatio-temporal grounding, and referred object captioning—directly address real roadside surveillance needs like identifying moving vehicles, counting objects, and localizing them over time. The chosen metrics (QA accuracy, spatio-temporal error, and NLG measures on captions) are standard and align with these tasks.", "theoretical_claims": "No theoretical claims are provided in this paper.", "experimental_designs_or_analyses": "One problem with this dataset is that the baseline models already score relatively high on the QA task already (e.g., 81.95% for overall QA accuracy in Table 3). Would the authors provide a brief discussion on how this dataset, especially the QA part, will remain challenging for upcoming new multimodal LLMs?", "relation_to_broader_scientific_literature": "This dataset is a great comprehensive video dataset that is positioned as a challenging benchmark for advancing research in intelligent transportation systems.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "No assets are provided in the submission. The reviewer recommends the authors to release the codebase as well as the dataset upon acceptance.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See Section Experimental Designs Or Analyses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O2SX6Ja5Xl", "reviewer": "Reviewer_QbCi", "summary": "This paper proposes a new traffic VQA dataset captured from the roadside. The paper proposes three tasks based on the dataset, including Multi-Choice Question Answering, Video Referred Object Captioning, and Spatio-Temporal Object Grounding. The author further proposes a new, unified method to tackle all three tasks. The established benchmark offers a comprehensive analysis and demonstrates the strong performance of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your positive feedback and valuable comments. Below, we provide point-by-point responses to the concerns raised. \n\nQ1: What are the differences between the proposed multi-choice QA and existing VQA work, aside from the camera view?\n\nA1: Thank you for the thoughtful question. Beyond the camera perspective, our work differs from previous VQA works in several key aspects:\n\n1.\tThe object reference expression in existing video QA works cannot be directly applied to the domain of spatiotemporal traffic scene understanding. We also argue it in reply to **Reviewer z4TA, A3-2**. This is due to the inherent ambiguity of natural language, the modality gap between visual and linguistic representations, and the need for accurate and unique object referring in traffic scenarios. Purely language-based descriptions often fail to achieve precise cross-modal object association. Hence, our work introduces a structured tuple-based object representation, enabling accurate and interpretable fine-grained analysis of traffic videos.\n\n2.\tOur benchmark is specifically designed for unified intelligent traffic monitoring and scene understanding, which is currently missing in existing works. We aim to bridge this gap by providing a dataset and benchmark tailored to traffic scene analysis needs. Our dataset reflects real-world traffic distributions, and we include additional figures to support this point in response to **Reviewer mL1S, A3**:  https://imgur.com/a/eoZXJEb\n\n3.\tWe unify three previously disjoint tasks in this domain, i.e., multiple-choice video QA, referred object captioning, and spatiotemporal object grounding, into a single benchmark. This positions our work beyond traditional VQA, offering a more comprehensive and domain-relevant challenge.\n\nQ2: Why investigate different token sampling strategies when they show similar performance across tasks?\n\nA2: Thank you for raising this question. In traffic scenes, particularly in the context of intelligent traffic monitoring and scene understanding, a key domain-specific characteristic is that the visual background tends to remain relatively static within a video.\n\nFor existing VLMs, efficient visual representation is critical to overall performance. Therefore, exploring different visual token sampling strategies that remove redundant information while preserving key content is particularly important in this domain. Our study investigates whether tailored visual token strategies can leverage this static background nature to improve efficiency and performance. This motivation is also consistent with some prior works for roadside visual-centric tasks [3][4], where techniques are applied to suppress background redundancy and enhance downstream task performance.\n\nQ3: Why emphasize “traffic anomalies” in the abstract and contributions without dedicated benchmark analysis?\n\nA3: Our intention was to reflect real-world distribution, where traffic anomalies (e.g., accidents) are rare but critical cases. While our benchmark does not focus solely on anomaly analysis, we still provide high-level annotations indicating whether each video contains anomalies. This supports valuable future research focused on anomaly detection and emphasizes the diversity and realism of our dataset. We will revise the content to clarify this.\n\nQ4: Define the difficulty levels of QA questions (e.g., single-hop vs. multi-hop) in the supplementary material.\n\nA4: Thank you for highlighting this point. We currently include a subset of QA templates in Supplementary Material C.2. To improve clarity, we will introduce an additional subsection that clearly defines and categorizes the difficulty levels of QA questions. \n\nQ5: Clarify why there are multiple bars for a class of questions in Figure 4(a).\n\nA5: In Figure 4(a), we plot the distribution of question word counts for each question task/type. Since questions within the same task type can vary significantly in length and structure, we chose to visualize this distribution to better reflect the linguistic diversity of the dataset, rather than simply reporting the total number of questions per type. We will revise the caption to make this clearer.\n\n\nQ6: L55 - The statement justification for the importance of QA in traffic scenes is unclear.\n\nA6: Thank you for pointing out the unclear phrasing. We will revise this part to emphasize the motivation and importance of introducing LLM/VLM for the intelligent traffic scene understanding domain and the development of AV in our revised version. \n\n\nQ7: Missing highly relevant work on traffic event recognition [1,2] in the Related Work section.\n\nA7: We appreciate your suggestion. We will include the missing references and expand the section to properly acknowledge prior work on traffic event recognition.\n\n\n[3] Zekun Zhang et al., Object Detection With Self-Supervised Scene Adaptation, CVPR 2023.\n\n[4] Shuo Wang et al., The 8th AI City Challenge, CVPR-W 2024.", "claims_and_evidence": "L55 “they often face significant challenges in scalability, generalization to diverse\ntraffic conditions, and real-world deployment.” Unclear why? \n\nAlthough the authors attempt to explain this with the following sentence, it is still unclear why the recent large models can address the mentioned challenges, especially the scalability and generalization to diverse traffic conditions and real-world deployment. In other words, the statements justifying the importance of QA in traffic scenes are very vague and not convincing. Note that I am not against the importance of the topic but feel the statement could be improved.", "methods_and_evaluation_criteria": "The performance of the method is strong and evaluated fairly.", "theoretical_claims": "Yes, I have checked the correctness of all proofs or equations in the main paper.", "experimental_designs_or_analyses": "The experimental design is very thorough and comprehensive.", "relation_to_broader_scientific_literature": "The proposed traffic VQA dataset can offer new opportunities for studying more fine-grained traffic understanding topics and facilitate the development of AV.", "essential_references_not_discussed": "Missing highly relevant discussion of work proposing a description for traffic event recognition [1,2] in Related work - Fine-Grained Video Understanding or Language-Based Object Referring.\n\n[1] Agarwal and Chen, Ordered Atomic Activity for Fine-grained Interactive Traffic Scenario Understanding, ICCV 2023\n\n[2] Kung et al., Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes, CVPR 2024", "other_strengths_and_weaknesses": "**strengths:**\n\n- the paper is well-written and easy to follow.\n\n**Weakness:**\n\n- It is unclear what is the difference between the proposed Multi-Choice Question Answering and existing VQA work, besides the camera view.\n- Unclear why the authors want to investigate different strategies of token sampling as they all perform very similarly in terms of performance across different tasks.\n- Unclear why the authors highlight “traffic anomalies” (L40 in the abstract) and “critical corner cases such as traffic accidents” (L96 in the contributions). This would mislead readers to expect to see benchmark results analyzing the performance under these anomaly scenarios.", "other_comments_or_suggestions": "- It would be great to have a detailed definition of the difficulty level of questions, i.e., single-hop or multi-hop, in the supplementary material.\n- Unclear why there are multiple bars for a class of questions in Figure 4 (a)", "questions_for_authors": "Please see the comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ld8sFrX2wQ", "reviewer": "Reviewer_z4TA", "summary": "The paper presents a comprehensive video-language dataset designed for complex traffic video understanding, named TrafficX-VideoQA. Meanwhile, a benchmark is provided, including multiple-choice video question answering, referred object captioning, and spatiotemporal object grounding tasks. Experimental results demonstrate that TrafficX-VideoQA is complex.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you very much for your positive feedback and valuable reviews. Our responses to your comments are detailed below:\n\nQ1. There is a need to incorporate the recent work [1] for a more comprehensive analysis.\n\nA1. Thank you for pointing out this important related work. We have discussed a detailed comparison with it in our response to Q3 below, and we will also include this reference in our final manuscript to improve the completeness and introduce our work more clearly.\n\nQ2. Why only focus on multiple-choice video QA, referred object captioning, and spatiotemporal object grounding?\n\nA2. Thank you for the insightful suggestion. We fully agree that video understanding covers many valuable tasks beyond our current focus, such as action recognition, event localization, video captioning, and anomaly detection. Following the survey [2], we broadly categorize video understanding tasks based on the granularity of required information into abstract, temporal, and spatiotemporal levels.\n\nIn traffic scenarios, high-level events are typically composed of interactions among traffic participants. As such, fine-grained spatiotemporal understanding and reasoning serve as a crucial foundation for interpreting complex traffic events.  Based on this, our benchmark focuses first on spatiotemporal understanding through three core tasks in traffic scene comprehension: multiple-choice video QA, spatiotemporal object grounding, and referred object captioning. We sincerely appreciate your suggestion and agree that using VLMs for higher-level semantic tasks such as traffic event recognition is a promising direction. We will add this to the Future Work section of our paper.\n\nQ3. Compared to [1], what are the strengths of the paper?\n\nA3. Thank you for raising this important question. We have reviewed this work and highlighted the following strengths of our dataset and benchmark compared to the work [1] as follows:\n\n 1. **Data Source and Data Novelty**.\n\nRef [1] is built upon existing datasets, providing additional textual caption annotations without contributing new video data or organizing raw footage. Moreover, most of the videos used in [1] are sourced from the internet. In contrast, our dataset is collected from real-world traffic scenes, ranging over 2 years using multiple intelligent infrastructure systems installed from over 20 diverse perspectives. From over 1,000 hours of raw video data, we curate our benchmark through a semi-automatic data annotation process. Compared to internet videos, our data more accurately reflects the distribution and characteristics of real-world traffic scenarios, offering higher data novelty and practical value. We include figures to support this point as suggested by Reviewer mL1S, A3: https://imgur.com/a/eoZXJEb\n\n2. **Inherent Limitation for Fine-Grained Spatiotemporal Tasks**.\n\nRef [1] focuses on video understanding tasks that require abstract semantic information, such as high-level event descriptions. However, such object reference is not well-suited for fine-grained spatiotemporal tasks in traffic scenes. This limitation arises from several factors: the inherent ambiguity of descriptive natural language expressions, the modality gap between visual and linguistic representations, and the high visual similarity of traffic participants in surveillance footage.\n\nThis limitation is also clearly illustrated in Ref [1], Figure 1 (top video). The sentence query states: “A black car drove past the corner, and a gray car followed closely behind it.” However, the video scene contains multiple black cars and multiple gray cars, making the reference inherently ambiguous. Such ambiguity prevents the benchmark from supporting more fine-grained VideoQA tasks, particularly those requiring precise object grounding and temporal reasoning.\n\n3. **Unified Benchmark with Structured Object Representation**.\n\nOur benchmark unifies three fine-grained tasks, i.e., multi-choice QA, referred object captioning, and spatiotemporal object grounding, via a tuple-based spatiotemporal object expression. This design enables benchmarks on more complex tasks, such as relative spatial reasoning, which are critical in the traffic domain but missing from [1].\n\n4. **Unified Model for VideoQA Benchmark**.\n\nIn [1], each proposed task is evaluated separately using existing models like CoCap or UEDVC. While valuable, these approaches are separate for each task and do not support unified understanding. In contrast, we propose TraffiX-Qwen, a unified VLM-based model trained with multi-task learning across all three tasks. This contributes a solid baseline toward generalizable traffic visual foundation models.\n\n[1] Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges. CVPR 2024\n\n[2] Video Understanding with Large Language Models: A Survey. arXiv:2312.17432, 2024.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "I have checked proofs for theoretical claims. For example, the paper states that \"most existing efforts primarily focus on driving scenarios and are typically constrained to individual tasks such as question answering, video grounding, or referred multi-object tracking\". From Table 1, the data are supported.", "experimental_designs_or_analyses": "I have checked the experimental designs and analyses. A comprehensive analysis was conducted for three key tasks in the paper.", "relation_to_broader_scientific_literature": "The paper facilitates further advancements in traffic video analysis and contributes to the development of next-generation traffic foundation models.", "essential_references_not_discussed": "There is a need to incorporate the recent works[1] for more comprehensive analysis.\n[1] Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges. CVPR 2024", "other_strengths_and_weaknesses": "The writing and the organization are good. And the proposed dataset and benchmark is valuable.", "other_comments_or_suggestions": "Why only focus on multiple-choice video question answering, referred object captioning, and spatiotemporal object grounding tasks? Wouldn't it be better to cover all the tasks of existing video understanding?", "questions_for_authors": "1. Why only focus on multiple-choice video question answering, referred object captioning, and spatiotemporal object grounding tasks? Wouldn't it be better to cover all the tasks of existing video understanding?\n\n2. Compared to the reference [1], what are the strengths of the paper?\n[1] Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges. CVPR 2024", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IZsuNuCXc0", "reviewer": "Reviewer_mL1S", "summary": "The paper introduces TraffiX-VideoQA, a benchmark for evaluating spatio-temporal video understanding in traffic scenes. It provides 1,000 videos, 85,000 QA pairs, 2,300 object descriptions, and 5,700 grounding annotations, covering diverse traffic conditions. The authors propose TraffiX-Qwen, a baseline model leveraging multi-resolution visual token sampling to improve temporal reasoning. Experiments compare LLaVA-OneVision, Qwen2-VL, VideoLLaMA2, showing that TraffiX-Qwen achieves superior performance, especially in multi-choice VideoQA. The paper contributes a new dataset, benchmarks existing models, and introduces an improved VideoQA method for real-world traffic scenarios.\n\n## update after rebuttal\nThe author addressed most of the issues mentioned in my initial review, so I am maintaining my original positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe appreciate your valuable feedbacks and address your concerns as follows.\n\nQ1: Clarification on whether TraffiX-Qwen’s performance gain stems from more input frames.\n\nA1: As open-source VideoQA models often adopt model-specific frame sampling strategies, which are tightly coupled with architecture and training, some models (e.g., LLaVA-OneVision, Video-LLaMA2) are not designed to handle longer frame sequences. Hence, we used different frame settings that were in line with their original paper and codebase. We also fully agree that comparisons under a unified frame setting would strengthen the results. Therefore, we conduct extra experiments with 101 frame inputs for the models used in our paper. (Qwen2-VL, whose model structure imposes constraints on input frame numbers, is adapted to 96 frames.)\n\n\n|**Model**|**Size**|**BLEU_4**|**ROUGE_L**|**CIDEr**|**Temp.E↓**|**Spa.E↓**|**ST.E↓**|**Pos.**|**Count.**|**Motion**|**Class**|**Exist.**|**Overall**|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|LLAVA-OneVision|0.5B|0.0|0.9|0.0|1.0|1.0|1.0|25.6|25.1|11.9|12.4|0.6|15.1|\n||7B|0.0|0.5|0.0|1.0|1.0|1.0|26.4|24.1|13.6|13.0|0.8|15.6|\n|Qwen2-VL|2B|3.8|15.1|0.18|0.65|0.68|0.73|31.2|38.4|58.2|55.4|74.5|51.5|\n||7B|5.0|14.9|0.14|0.70|0.70|0.76|31.8|55.3|54.3|50.0|75.8|53.4|\n|VideoLLAMA2|7B|0.0|7.1|0.0|1.0|1.0|1.0|27.6|27.5|26.3|18.7|52.2|30.4|\n|**TraffiX-Qwen**|0.5B|35.0|50.4|2.52|0.12|0.19|0.26|72.0|80.6|82.6|69.8|89.2|78.8|\n||7B|36.7|52.0|2.56|0.11|0.17|0.24|76.6|81.9|84.3|73.4|89.4|81.1|\n\nResults show that: 1. LLaVA-OneVision and Video-LLaMA2 perform far worse with 101 frames, likely due to the lack of long-range temporal modeling. 2.\tQwen2-VL shows improved performance with more frame inputs, suggesting that additional temporal information can indeed be beneficial. This also highlights Qwen2-VL’s stronger temporal reasoning capability.\n\nImportantly, even under the same frame input setting, TraffiX-Qwen still consistently outperforms all baselines, confirming that the performance gain stems from multiple aspects, not just access to more temporal information. These results will be added to the revised paper.\n\nQ2: The role of YOLOv8, ByteTrack, etc in the pipeline.\n\nA2: We use 2D detectors and trackers only during the TraffiX-VideoQA dataset construction to generate meta-information aligned with human annotations, improving quality and consistency. They are not used in TraffiX-Qwen’s training or inference. Our baseline is a fully end-to-end VLM that processes raw videos and generates answers directly, without external modules. This design simplifies the pipeline and promotes research toward integrated video understanding in the traffic domain.\n\nQ3: Statistics supporting the claim of dataset diversity.\n\nA3: We add 3 figures illustrating dataset distribution at: https://imgur.com/a/eoZXJEb.\n\nThe figures statistically support the dataset’s diversity. We group scenes in our dataset into 3 key types: highways (rural), urban intersections (city), and country roads (rural/urban). They show that the dataset aligns well with real-world traffic distributions. The figures will be added in the revised version.\n\nQ4: Missing information on model efficiency and computational cost.\n\nA4: We provide the training and inference details in the paper:\n\n|**Version**|**#Vision**|**#Projector**|**#LLM**|**Inf Speed/QA**|**#Trainable**|**Train Hour**|\n|-|-|-|-|-|-|-|\n|**0.5B**|397.8M|1.8M|493.8M|~1.6s|495.6M|28h|\n|**7B**|397.8M|17.0M|7612.6M|~3.8s|7629.6M|36h|\n\nInference time is measured as the average to process a 10s–1min video with autoregressive decoding on 1*A100 GPU (no acceleration). We believe further optimization (e.g., quantization, pruning, distillation) is a promising direction for improving deployment efficiency in traffic monitoring. We will include this table and discussion in the revised version.\n\n\nResponse to other suggestions:\n\n1. We have included an initial discussion regarding how sampling strategy impacts spatiotemporal reasoning in Sec 5. We will expand this with new experimental results and discussions. \n2. Qualitative results for ST-OG and V-ROC have been added to Appendix B.5, B.6, covering both successful and failure cases. For the MC-QA task, we will provide extra qualitative visualizations and analyses.\n3. Initial analysis of frame count effects on TraffiX-Qwen is provided in Appendix B.1. We will expand this with an extra discussion of how frame sampling impacts temporal reasoning capabilities.\n4. We agree that including extra AD benchmarks (e.g., Ego4D etc.) brings valuable context and will add them.\n5. We will add more details on model complexity, and potential acceleration techniques to support practical deployment.\n6. We agree that using fixed camera views may introduce distributional bias in traffic patterns and will include a discussion on this in limitation.\n7. We will improve Fig. 6. refine the definition of spatiotemporal grounding, and emphasize the role of human verification during annotation process.", "claims_and_evidence": "The paper makes several strong claims, particularly regarding the effectiveness of TraffiX-Qwen and the value of the TraffiX-VideoQA dataset. The dataset is well-constructed, covering diverse traffic conditions, and the model comparison with existing vision-language methods is comprehensive. However, some claims lack sufficient evidence or require further clarification:\n1) TraffiX-Qwen processes more frames per video than competing models, which could naturally give it an advantage. It would be more convincing if additional comparisons were made with the same number of frames across all methods.\n2) The paper does not introduce new evaluation metrics, even though it discusses performance differences in detail. If the authors claim methodological novelty in evaluation, this should be clarified.\n3) While the paper uses SOTA models (YOLOv8, DETR, ByteTrack), it does not analyze their individual contributions. A proper ablation study would strengthen the claim that these choices are optimal for this dataset.\n4) The paper asserts that the dataset captures various real-world conditions, but lacks quantitative statistics (e.g., distribution of intersection types). Providing these details would improve the credibility of this claim.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally well-suited for the task. TraffiX-Qwen’s multi-resolution visual token sampling is a reasonable approach to improve spatio-temporal reasoning while managing computational cost. The TraffiX-VideoQA dataset is well-structured and relevant for evaluating VideoQA in traffic scenes. However, there are some aspects that need further clarification:\n1) TraffiX-Qwen sees more frames than competing models, making it difficult to isolate the method’s true impact. A fairer comparison with a fixed number of frames across all models would strengthen the results.\n2) While the dataset covers various traffic scenarios, it lacks quantitative details (e.g., the proportion of different road types or intersection structures). Providing such statistics would enhance its credibility.\n3) The paper adopts standard metrics but does not introduce new ones. This is fine, but given the complexity of the task, a discussion on potential evaluation limitations (e.g., how frame count affects results) would be valuable.", "theoretical_claims": "The paper does not contain theoretical claims or formal proofs, as its contributions focus on dataset construction, model development, and empirical evaluation. The proposed multi-resolution visual token sampling is validated experimentally rather than theoretically. While this is reasonable for the scope of the work, a theoretical discussion on how the sampling strategy impacts spatio-temporal reasoning could strengthen the justification.", "experimental_designs_or_analyses": "The experimental setup is mostly well-structured, and the benchmarking of multiple vision-language models provides useful insights. However, there are some concerns regarding experimental fairness and completeness:\n1) TraffiX-Qwen processes more frames per video than other models, which could give it a natural advantage. A comparison where all models use the same number of frames would make the results more conclusive.\n2) The impact of the multi-resolution visual token sampling strategy is not explicitly studied. Ablation studies would help clarify these factors.\n3) The paper presents numerical comparisons but lacks discussions on where and why models fail. A qualitative analysis of failure cases would provide valuable insights.", "relation_to_broader_scientific_literature": "The paper provides a solid discussion of prior work, particularly in the areas of VideoQA datasets (NuScenes-QA, DRAMA, DriveLM) and vision-language models (LLaVA, VideoLLaMA2, Qwen2-VL). This establishes a clear foundation for its contributions.", "essential_references_not_discussed": "The paper provides a solid literature review, but some essential references are missing. For example, the dataset comparison could be expanded to include related benchmarks from autonomous driving and traffic analysis (e.g., Ego4D, BDD100K, Waymo Open Dataset). While these are not VideoQA datasets, they are relevant for understanding real-world traffic dynamics.", "other_strengths_and_weaknesses": "Beyond the points discussed earlier, there are additional strengths and weaknesses worth mentioning:\nStrengths:\n\n● The paper is well-structured and written clearly, making it easy to follow, even for researchers unfamiliar with VideoQA.\n\n● The VideoQA task in traffic scenarios is logically structured, making the dataset suitable for future extensions, such as integrating multimodal sensor data (e.g., LiDAR).\n\n● Given the focus on traffic scenarios, this dataset has strong potential for real-world applications in intelligent transportation systems and automated video monitoring.\n\nWeaknesses:\n\n● The proposed TraffiX-Qwen model, while effective, appears computationally expensive due to the multi-resolution visual token sampling strategy. The paper does not discuss training or inference efficiency, which may affect deployment feasibility.\n\n● The dataset is sourced from fixed camera locations, which may introduce a bias toward certain traffic patterns (e.g., urban over rural areas). A discussion of dataset biases and their impact on generalization would be valuable.", "other_comments_or_suggestions": "Here are some additional comments and suggestions for improving the paper:\n1) Some figures, particularly Fig 6 (dataset distribution visualization), could be clearer. Improving resolution or providing more readable labels would help.\n2) The paper does not specify the training hyperparameters, batch size, or computational resources required for TraffiX-Qwen. Adding this information would improve reproducibility.\n3) The definition of spatio-temporal grounding could be made clearer, particularly in terms of how annotations were generated and whether human verification was involved.\n4) Some important dataset details (e.g., intersection type distributions) that are currently missing might be better suited for the appendix.", "questions_for_authors": "1) TraffiX-Qwen processes more frames per video than competing models. Could you provide additional results where all models use the same number of frames? This would help determine whether the performance improvement comes from the model design or simply from having access to more temporal information.\n2) The paper uses YOLOv8, DETR, and ByteTrack for object detection and tracking, but does not analyze their individual contributions. Have you conducted ablation studies to evaluate how each of these components affects the final performance?\n3) The paper mentions that TraffiX-VideoQA covers diverse traffic environments, but does not provide specific statistics on intersection types, traffic conditions, or vehicle categories. Could you share these statistics to support the claim of dataset diversity?\n4) The paper does not discuss the computational cost of TraffiX-Qwen. What are the training time, required GPU resources, and inference speed? This information would help assess the feasibility of deploying the model in real-world settings.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work propses a new dataset for evaluating traffic scenarios with evaluation based on question answering spanning tasks like: multi-choice vqa, referred object captioning, and object grounding/tracking. This work also evaluates open VLMs like LLava-OneVision, VideoLlama2, Qwen2-VL, and adapt Qwen-2 with pre-training (on raw data) and "instruction" tuning on QA pairs from the training set of the proposed dataset.

The proposed dataset presents a challenging benchmark to existing VLMs. The reviewers found dataset contribution and evaluation as important for transportation systems, especially for fine-grained evaluation. Hence, I'm inclined to recommend this work further.
Decision: Accept (poster)