TAU-106K: A New Dataset for Comprehensive Understanding of Traffic Accident
Authors: Yixuan Zhou
Long Bai
Sijia Cai
Bing Deng
Xing Xu
Heng Tao Shen
Venue: ICLR 2025 Poster (2025)
Keywords: Anomaly Detection
Traffic Accident Understanding
Multimodal Large Language Models
Instructional Dataset
Primary area: datasets and benchmarks

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in general visual understanding tasks. However, their potential for high-level, fine-grained comprehension, such as anomaly understanding, remains unexplored. Focusing on traffic accidents, a critical and practical scenario within anomaly understanding, we investigate the advanced capabilities of MLLMs and propose TABot, a multimodal MLLM specialized for accident-related tasks. To facilitate this, we first construct TAU-106K, a large-scale multimodal dataset containing 106K traffic accident videos and images collected from academic benchmarks and public platforms. The dataset is meticulously annotated through a video-to-image annotation pipeline to ensure comprehensive and high-quality labels. Building upon TAU-106K, we train TABot using a two-step approach designed to integrate multi-granularity tasks, including accident recognition, spatial-temporal grounding, and an auxiliary description task to enhance the model's understanding of accident elements. Extensive experiments demonstrate TABot's superior performance in traffic accident understanding, highlighting not only its capabilities in high-level anomaly comprehension but also the robustness of the TAU-106K benchmark. Our code and data will be available at https://github.com/cool-xuan/TABot.

Published as a conference paper at ICLR 2025
TAU-106K: A NEW DATASET FOR COMPREHENSIVE UNDERSTANDING OF TRAFFIC ACCIDENT
Yixuan Zhou Long Bai
Sijia Cai
Bing Deng'
Xing Xu
Heng Tao Shen?
Alibaba Cloud Computing
Tongji University (Tongji)
{yi xrixunnzhou.coo zyx, bailong interxuxingehotmall com, shenhengtao@hotmail..o
wwooongbiiiiaooooo bai, seephen.cs]. dengbing. (aaenngddd@aaaabbaaaa -ineiaa--ii-ieeeeee com,
ABSTRACT
Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in general visual understanding tasks. However, their potential for high-level, fine-graing aneeggained comprehension, such as anomaly understanding, re- mains unexplored. Focusing on traffic accidents, critical and practical sce- nario within anomaly understanding, we investigate the advanced capabilities MLLMs and propose TABot, multimodal MLLM specialized for accident- related tasks. To facilitate this, we first construct TAU-106K, large-scale multi- modal dataset containing 106K traffic accident videos and images collected from academic benchmarks and public platforms. The dataset is meticulously anno- tated through a video-to-image annotation pipeline to ensure comprehensive and high-quality labels. Building upon TAU-106K, we train TABot using a two-step approach designed to integrate multi quuti-graaullrrity tasks, including accident recog- nition, temporal grounding, and an auxiliary description task to enhance the model's understanding of accident elements. Extensive experiments demon- strate TABot's superior performance in traffic accident understanding, highlight- ing not only its capabilities in high-level anomaly comprehension but also the robustness of the TAUU-166, benchmark. Our code and data will be available https:/ /ttp:/gggttu.t.t// com cool xuan/TABot
INTRODUCTION
Traffic Accident Detection (TAD) has always been a crucial and practical task in public safety and transportation management. The development of advanced technologies, such as computer vision and deep learning, has enabled the automation of TAD, providing real-time accident alerts and anal- ysis. Despite significant research on traffic accident detection (Shah et al. 2018 Zhu et al.) 2019) Kim et al., 2019, Haresh et al., 2020) Chahremannezhad et al. 2022), existing TAD methods often rely conventional visual feature extraction techniques that lack high-level comprehension and multimodal alignment capabilities in interpreting intricate spatial-temporal dynamics.
Recent advancements in aligning large language models (LLMs) visual-language datasets have driven remarkable progress in multi-modal understanding tasks (Achiam et al., (023).. With exten- sive pre-training and instruction tuning, multimodal LLMs (MLLMs) have demonstrated superior performance in spatially visual understanding and logical reasoning for the general-purpose com- prehension (Li et al. 2023 Zhu et al. 2023) Liu et al.. 2024). However, due to the inherent gap between general and specific tasks, MLLMs trained on general domains often underperform in spe- cialized fields due to limited domain-specific data. For instance, illustrated in Figure T the most advanced MLLMs, such as GPT-4o (Achiam et al., (023), Qwen2-VL (Yang et al.) 2024), and Gemini-1.5.. (Team et al. 2024), may misinterpret the visual content of traffic accidents where
" "qqual contribution. Work done during an internship Alibaba Cloud Computing.
I Project lead.
1 "Correppnniigg authors.

---

Published as a conference paper at ICLR 2025
Question: Do you notice any traffic accident scenarios in this picture? Please respond with a "Yes' or 'No". Following that, describe the image in one sentence.
GPT-4o: No, the image shows busy city intersection with
vehicles motorcycles waiting traffic light.
Qwen2-VL: No. The image shows busy urban street with cars, scooters, and pedestrians, but there is no visible accident.
Gemini-1.5-PP No. The image shows sunny city street scene with several motorbikes and cars intersection under a highway overpass. BottChth (Ours): Yes. Because a black car brake was not on time, collided with white coat electric bicycle rider.
Figure One example to illustrate the limitations of general MLLM in understanding traffic accidents. In the scenario where vehicle collision occurs due to sudden lane change by the leading vehicle, GPT-4o, Qwen2- LL, and Gemini-1.5-Pro fail to detect this issue.
vehicle collision occurs, leading to inaccurate responses. We argue that the failure of general MLLMs to understand traffic accidents stems from the following two main reasons: (i) Traffic ac- cident detection requires MLLMs to grasp ambiguous concepts like anomaly and accident, which are context-dependent and defined by human criteria. However, existing MLLMs are trained on general- purpose data focusing on fundamental semantics, lacking the specialized understanding of such high-level semantics. (ii) The visual representations of accident occurrences differ significantly from general scenes, necessitating realigning these visual representations with the semantic under- standing towards traffic accidents. Both of these limitations highlight the need for infrastructure that includes seeicfnt-specffic annotations and specialized MLLMs understand traffic accidents.
To pioneer an MLLM specialized in traffic accident comprehension, we first created TAU-106K, large-scale multimodal traffic accident dataset containing 106K videos and images with detailed accident-oriented annotations. In particular, we aggregate academic benchmarks and crawl traffic accident videos from public platforms, building a diverse and high-quality visual foundation. To ensure annotation quality and efficiency. we design video-to-image annotation pipeline, resulting in comprehensive annotations that are manually crafted by human labors. The annotations cover accident recognition, description, temporal localization, and spatial grounding at both the image and video levels, providing detailed and structured information for MLLMs to understand traffic accidents.
Using TAU-106K, we reorganize the annotations into instructional data to unlock MLLMs potential in traffic accident understanding and introduce TABot, end-to-end MLLM specialized for traffic accident comprehension across both image and video modalities. We adopt a two-step training ap- proach: functional tuning engage multi-granularity accident detection capabilities activation, and instruction tuning to enhance contextual comprehension and instruction following capabilities. In particular, during functional tuning, we propose two training strategies to serve temporal localiza- tion, the most crucial task in traffic accident understanding: (i) Negative Segment Referring (NSR), which utilizes contrastive learning to heighten the model's sensitivity to accident boundaries, and (ii) Video Spatial Aligmment (VSA), which inserts spatial information into the training of video tasks, serving as fine-grained complement temporal localization. Followed by the functional tuning, we further generate multi-turn dialogues using an automated paradigm (Liu et al. 2024) and per- form instruction tuning to enhance the dataset's 's utility and capabilities of MLLMs for human-lite chatting and traffic accident understanding.
2 RELATED WORK
Multimodal Large Language Models. Extensive research has focused on enabling LLMs to pro- cess visual information, typically by adding an adapter between pre-trained visual models and LLMs to align features from different modalities (Li al. 2023 Zhu al. 2023 Liu et al.) (2024). Some advanced multimodal LLMs, such as Qwen2-VL VL YYng et al. 2024). ), unified image and video un- derstanding into single model, but still struggle with more fine-grained tasks. On the side of image
2

---

Published as a conference paper at ICLR 2025
modal, object grounding has been key focus, with series of works (Chen al. 2023b Bai al.
2023
Peng et al. 2023 Chen et al. 2023e] You et al. (023) standardizing grounding coordinates to
text format and achieving robust grounding capabilities. On the other hand, videos,
com- plex form of visual data, introduce greater challenges in aligning with video content (Maaz et al,
2023
Lin al. 2023 Chen et al. 2023a Zhang | Qian et al. 2024 He et al., 2024|Cheng 2024 Xu et al. 2024 et al. 2024]Chen et al. 2023d] 2024p. VTimeLLM (Huang et al. and TimeChat (Ren al. 2024) address temporal localization by proposing time-aware atten- tion mechanisms. GroundingGPP (Li et al. (024) unified fine-grained capabilities across image and video for comprehensive multimodal understanding. Despite these advancements, previous works focus on general-purposs understanding, remaining largely unexplored in some specific scenarios, such as traffic accident understanding.
2023
et al.
2024
Traffic Accident Detection and Understanding. Traditional Traffic Accident Detection (TAD) methods are classified into single-stage (Hasan et al., 2016) and two-stage paradigms (Yao et al,) 2019 Fang et al.) 2022b). Single-stage approaches often rely on frame-to-frame errors but under- perform in forecasting non-ego accidents and are sensitive to dynamic backgrounds (Hasan et al.] 2016). Two-stage methods extract visual features, such bounding boxes and optical flow, and ap- ply TAD models to predict anomalies (Fang et al., 2022b). However, these methods depend heavily the quality of feature extraction. Recent advances have integrated textual information into TAD, with TTHF (Liang et al. 2024) introducing text-driven attention mechanisms for anomaly detection in videos, and SUTD- TrafficQA (Xu et al. 2021) modeling fundamental question-ant -anweernn and reasoning tasks for traffic accident scenes. On the MLLM front, empirical studies (Cao et al. (023) have validated GPT-4(V)'s effective recall and description capabilities for traffic accident images. To this end, the potential of MILMs in accident understanding remains unexplored, particularly in ppatial-temporal grounding and reasoning over traffic accident videos.
Data Collection
Accident
Normal
Unrelated
Key Accident Frame at 0.46 Timestamp
Accident Category Label: MV&nMV
Accident Category Label MV&nMV
Accident Temporal Label: (0.41, 0,53]
Accident objects Label:
Accident Description Current vehicle is driving crossroads, At the moment 0.41, since electric bike with people did follow traffic rules and crossed the road abruptly, the current vehicle collided with electric bike with people at the moment 0.53, the accident concludes with the electric bike was hit the ground.
electric bike with two people
[0.18, 0.54, 0.52, 1.00]
Accident Description The current
vehicle collided with the electric bike with two people.
Video-level Annotations
Image-level Annotations
Figure The data collection and annotation pipeline for building TAU-106K.
3 TAU-106K FOR VIDEO-IMAGE TRAFFIC ACCIDENT UNDERSTANDING
To advance the development MILMs for traffic accident analysis, we introduce TAU-106K a comprehensive dataset integrating video and image data for traffic accident understanding, manually labeled with multi-granularity annotations through a video-to-image annotation pipeline (Figure)
3.1 VIDEO-BASEE DATA COLLECTION AND ANNOTATION
Video Data Collection and Preprocessing. While traffic accident understanding a critical public safety task and has been extensively studied, the available open-source benchmarks limited in both scale and diversity, often featuring low-resolution video data. To address this, we aggregate established traffic accident benchmarks such TAD (Xu al. 2022), DoTA (Yao et al. 2022
3

---

Published as a conference paper at ICLR 2025
and CCD (Bao al. 2020), selecting high-quality video clips the data foundation for further annotation. We further expand the dataset by crawling road surveillance and dashcam footage from platforms like YouTube and BiliBili,, capturing diverse real-world traffic conditions. Despite the abundance of traffic accident videos on the Internet, they are often unstructured and lack annota- tions. For the crawled raw videos, first crop them into individual clips using the scene change detection toolkit PySceneDetect, and then manually filter out irrelevant or low-quality videos. Consequently, we obtain collection of 51.5K traffic-focused video clips sourced from academic benchmarks and social media platforms, as illustrated in the upper part of Figure
Video-based Accident Annotations. All existing benchmarks for traffic accident understanding lack comprehensive annotations, especially in terms of accident descriptions, which are crucial for enabling MILLMs to understand accident events in detail. To bridge this gap, we annotate from scratch or supplement existing annotations in three key aspects:
Accident Category: accident occurrence and detailed accident types. Each clip is reviewed to de- termine if an accident is present, labeled either Accident or Normal. For Accident clips, we further categorize the accident type into five subcategories: single motor vehicle (SMV) accident, multiple motor vehicle (MMV) accident, multiple non-motor vehicle (MnMV) accident, motor vehicle and non-motor vehicle (MV&nMVV accident, and vehicle and pedestrian (V&P) accident.
2. Accident Duration: the specific time points of the accident occurrence. Annotators precisely identify the start and end timestamps of the accident within each clip, yielding the time points fend)) In particular, the start time tstart should be the exact frame when the accident event begins, such as the moment of collision, while the end time fnd is marked when the event concludes (e.g., stopping). Both timestamps are normalized within the clip duration ensure consistency.
3. Accident Description: a detailed textual description of the nature of the accident, which is absent in all existing traffic accident benchmarks while being substantial for MILMs to understand the accident event in detail. To ensure consistency and precision, we design a structured annotation template, guiding annotators to provide detailed and structured descriptions of the accident events.
The description template for Accident is structured to depict the Traffic Scenario (urban, highway, etc.), Accident Content including the objects involved the accident (vehicles, pedestrians, etc.) and the nature of the accident (collision, scrape, etc.). and Aftermath, ensuring comprehensive and structured annotations. The labeled start and end timestamps are also incorporated into the descrip- tion to provide temporal context for the accident event. Beyond the accident event itself, annotators are also encouraged infer the Potential Causes, such as traffic rule violations or improper driv- ing behaviors. The detailed template is also dependent on the Footage Source, either Dashcam or Surveillance camera. For intuitive understanding, we decompose the example shown in Figure into the structured format in the gray block.
[Footage Source: Current vehicle is driving on] [Traffic Scenario: crossroads]. At the moment of 0.41, [Potential Causes: since an electric bike with two people did not follow traffic rules and crossed the road abruptly]. [Accident Content: the current vehicle collided with the electric bike with two people], the moment of 0.53, [Aftermath: the accident concludes with that the electric bike was hit to the ground.]
In practice, these three annotation tasks are performed simultaneously, with multiple rounds of re- view and correction to ensure quality and consistency. This integrated approach ensures coherence in annotations, reflecting the interconnected nature of these tasks.
3.2 IMAGE-BASED DATA COLLECTION AND ANNOTATION
molti-granuulaiity while lacking spatial details, which crucial for MLLMs to understand fine-grained visual features. To address this, we further derive images from the video clips and label them with detailed spatial information, whose detailed annotation pipeline is illustrated in the bottom right part of Figure? 2
The above video annotations
Image Data Collection and Selection. Besides few image-only accident datasets (c.g., Task- Fix (uan et al. (02I)). most of the image data in our TAU-106K is sampled from the video clips. Guided by the temporal localization annotations in the video clips, first extract candidate frames
4

---

Published as a conference paper at ICLR 2025
9.8%
44.1%
SrOKS
IPC A'ceree 9.7%
Hodi
drining
plectric
1.3% NAV Jectire
Countr dara
D
Fider
2.2%
Turning ended setttnttteesctsssttoo
My ALL.e
32.9%
(a) Data source distribution.
(b) Accident type distribution.
(c)
Word cloud of description.
Figure 3: Data source distribution, accident type distribution, and word cloud of accident descrip-
tions in TAU-106K dataset.
by uniformly sampling frames within the labeled accident duration. These frames are then evaluated by annotators to select keyframes that best represent the accident events, based on the Accident De- scription in the video annotations. Notably, the time points of the selected keyframes are preserved to keep the temporal connection between the video and image data, which also enables our video spatial alignment strategy in model training. In addition to accident-related frames, we randomly sample accident-free frames to maintain data balance and prevent model bias.
Image Annotations Derived from Video Annotations. For the images sourced from existing benchmarks, we adopt the available annotations and extend them to our multi-granularity anno- tations. On the other hand, for the images derived from video data, we inherit the accident-related annotations from the video clips, including the Accident Category and Accident Description, where the latter extracted from the content of the accident part in the video-based accident description to maintain annotation consistency and reduce workload. In particular, labels for the involved objects are derived directly from the accident descriptions, ensuring that the annotated objects are those explicitly mentioned. For instance, given the accident description as "A blue car collides with pedestrian in white clothes", the corresponding objects will be labeled as blue car and pedestrian in white clothes, respectively. This instance-specifie labeling helps MILMs focus on the objects di- rectly involved in the accident, minimizing distractions from irrelevant objects of the same category that may appear in the scene.
Table 1: The comparison of TAU-106K with other accident-specifie general- pureooe benchmarks. CLS': Accident Categories, 'TL': Temporal Annotation, 'Bbox': Object Grounding Annotation,
'CAP': Caption Annotation, and 'QA': Question-Answer Pairs.
Dolaset
Years
Domain
# Viders
Annolations
Avg. Words
Avg. Duration
Dashcum (Chun et al. 2017
2016
Trafiic
3,000
TL.
5.0 seconds
A3D Yao et 2019
2019
Traffic
1.500
TL:
8.5 seconds
CCD
2010
Traffic
1,500
5.0 seconds
TAD Ly el at 20241
2021
Traffic
500
CAP: TL;
35.8 seconds
DADA (Fing a 2021
2021
Trafhe
200
CAP: TL, Driver Attention
scconds
SLTDD TraliiQA ef
2021
2021
Traffic
10,080
QA pairs
13.6 seconds
DeTA (Yo et 2044
2022
Traffic
4,677
CAP: TL; Bbox
15.6 seconds
CAP [Ting eraie
Traffic
11,727
CAP: TL,; Fixed Form CAP
6.3
6.2 sesonds
TAD-IO6K (Ours)
2024
Traffic
$1,544
CAP: TL; Bbox; Free-Form CAP
32
10.3 seconds
Charads-SIA (Gao
Daily
9,348
Froo-Form CAP
soconds
DiDeMo (Anné Hommrnks ei ar
2017
2017
Open
10,464
TI;; Free-Form CAP
7.5
30 seconds
ActivityNal Caploos Krismna 2017
2017
Open
19,209
TL.: Frre-Fomm CAP
13.5
180 seconds
3.3 DATA STATISTICS AND ANALYSIS
TAU-106K comprises 106K multimodal data instances, including 51.5K video clips and 54.8K im- ages, all with high-quality annotations. The majority of the video clips and images are in 720p res- olution and are sourced from both open-source benchmarks and social media platforms, as shown in Figure ((a) Among the TAU-106K, 56% of instances are labeled as Accident and 44% as Nor- mal, with detailed category distribution shown in Figure The balanced distribution of accident- related and accident-free instances ensures that the model is trained robustly, avoiding biases towards accident occurrences. The average video duration of processed and filtered clips is 10.3 seconds, with annotated accidents lasting an average of 3 seconds (approximately 25% of the video clip). As for the image data, 45K accident-involved objects are grounded, with an average of 1.6 bounding boxes per image and an average bounding box area covering 7.9% of the image. Our accident de-
5

---

Published as a conference paper at ICLR 2025
Video Accident
Accident Temporal Localization
Accident Recognition Description
Accident Spatial Grounding
Recognition Description
Yes.
The Heaponedio cemakaed oir entestomp of (865, 9.80
(RD).
Buno The wours (beb. 9.88, 9,8 mno Mue truck [666, 0.56 nieer (088, 008.08800000000000 0.8
thue louck su movol and hil on dlecttic bloysle rider in bass
lruck cul duy with sussson rimma on elecato blyycl
Large Language Model
Video Adapter
Image Adapler
T Tokenizer
Q-former
Encoder
Visual Encoder
Teak-spacifc Questions
Intceo) (RD) oodidinn
you, ploase sodidinn
Drnagal (HUI) bullaas en accadentt in the image?? If you place desurtov the accekent and
(Ndcol
happoned.
(mmaga) [ooa) Pleas provide the region schane the acddannt lousted and the iscation
Figure 4: The model architecture and functional capabilities of the TABot.
scriptions are detailed and diverse, covering broad range of traffic scenarios, accident types, and objects involved, as shown in the word cloud of accident descriptions Figure
We provide more comprehensive comparison between TAU-106K and other datasets, focusing on key features such as size, domain, annotation types, and the characteristics of the textual captions, depicted in Table I According the comparison, our TAU-106K is the largest dataset in terms of the number of videos and the diversity of annotations, supporting a wide range tasks. In particular, benefiting from manual annotation process that is labor-intensive yet worthy, the labeled free-form accident captions in TAU-106K are much more diverse and detailed than other datasets, achieving a largest average length of 32.1 words per caption. This makes TAU-I06K valuable resource for training and evaluating accident-aware models in traffic video understanding.
TABOT: CHATBOT FOR TRAFFIC ACCIDENT UNDERSTANDING
We introduce TABot, a multimodal ane-grained MLLM developed by leveraging instructional data constructed from the TAU-106K dataset. TABot is compatible with both video and image modalities, enabling it to perform fine ime-grained understanding and reasoning tasks in traffic accident scenarios. The proposed TABot integrates suite of traffic accident-related tasks, depicted in Figure
4.1 MODEL OVERVIEW
We advance the TABot upon GroundingGPT (Li et al. 2024), a model known for its strong perfor- mance in fine-gr grained image and video understanding. By fine-tuning this general-purpose purpose MLLM on our annotated TAU-106K dataset, we enhance its capabilities for traffic accident comprehen- sion several functional tasks including Accident Recognition, Accident Description Genera- tion, Accident Temporal Localization, and Accident Spatial Grounding, as illustrated in Figure 4 It is notable that we normalize the responses to the temporal localization and spatial grounding tasks to the video duration and image size, respectively, to ensure consistency and facilitate model training. The normalized responses are denoted as {1starts bend) for temporal localization and Emin Vmin Tmaz : Umaz] for spatial grounding, enclosed in specific tokens to indicate the temporal boundaries and spatial regions.
Following previous works (Li et al. 2024) we adopt a two-stage fine-tuning approach: Firstly, dur- ing the functional tuning stage, BABot is jointly fine-tuned on both image and video data, focusing on the four key tasks mentioned above. We generate structured single-round conversations for each task to facilitate the model's understanding of traffic accidents from different perspectives. Two additional training strategies are proposed to further improve performance in temporal localization: Negative Segment Referring (NSR) and Video Spatial Alignment (VSA), which promote the per- formance from the perspective of contrastive learning and spatial understanding, respectively. NSR samples accident- free segments before the occurrence of an accident and trains the model by re- ferring the sample segments containing accidents, serving negative data highlight the

---

Published as a conference paper at ICLR 2025
perception of accident occurrences. On the other hand, benefiting from the unified video-to laleo-to-image annotation pipeline, VSA involves the spatial grounding annotations into the video training process, complementing spatial information from images into the temporal localization task. As for the im- plementation details, we extend the answer to the temporal localization task to include the spatial grounding annotations. For example, the response the temporal localization task '(0.30, 0.45)' may be further extended with "Ar the timestamp 0.38, an accident occurs at the region of [0.21, 0.35, 0.87, 0.57J." This alignment improves TABot's fine-grained spatial understanding of acci- dents in video contexts. Additionally, to ensure the model's flexibility in handling multiple tasks, task-specific flag tokens (Accident Recognition Description [RD], Temporal Localization [TL], and Spatial Grounding (SG]) are inserted at the start of each query to guide TABot's responses.
With the above functional tuning. TABot is endowed with the capabilities to perform coarse- and fine-grained traffic accident understanding tasks. To further advance the TABot's comprehensive understanding and conversational skills, we utilize the textual captions of the video clips and images as the abstracts to prompt the powerful LLMs (Achiam et al., 2023), to conclude the above func- tional tasks and generate additional accident-oriented dialogue, such as the causes accidents or prevention suggestions. The generated model. Through such instruction tuning, TABot is upgraded to a chat version (TABot-Chat) with enhanced instruction-folloiing capabilities and more comprehensive understanding of traffic cidents. In particular, the task flag tokens are maintained the instruction tuning stage to guide the model's responses specific tasks, ensuring the functional capabilities of the model towards multiple eecident-orientee tasks.
dialogue set is then used to instruct the TABot
5 EXPERIMENTS
We set GroundingGPT-77 (Li et al. 2024). a pre-trained general-purpose MLLM with temporal and spatial grounding capabilities, as the baseline model for our TABot. The detailed experimental settings of the two-step approach are described as follows:
Functional Tuning. We train LLM and both visual adapters of the GroundingGPT model through our TAU-106K dataset for epochs using 8 x H800 GPUs The initial learning rate is set to 2e-5 with batch size of 32, requiring about 20 hours to complete.
Instruction Tuning, We extend training with the instruction-tuning dataset generated by LLaMA- 70B (Dubey al., 2024), leading to our TABot-Chat model. To avoid catastrophic forgetting, combine the single-round and multi-round dialogue conversations during this stage, further training the model for I epoch on H800 GPUs for about 9 hours with the learning rate and batch size unchanged.
Evaluation Metrics. For evaluation purposes, the TAU-106K dataset was split into training and testing sets in 9:1 ratio, ensuring the same distribution of normal/accident instances and scene continuity across both. We evaluate the TABot on four functional tasks for both image and video data. The evaluation metrics are as follows:
1) Accident Recognition. Recall, Precision, and F1 scores are used assess the model's accuracy in distinguishing accidents from normal scenes in both image-level and video-level contexts.
2) Accident Description. BLEU-1 score, Rouge-I. Fl score, and BERT score are employed to measure the model's ability to generate coherent and accurate accident descriptions. We further leverage GPT-4o to estimate the quality of the generated descriptions, referred to as GPT-4 score.
3) Accident Temporal Localization. We reported the Intersection over Union (IoU) between pre- dicted and true temporal intervals, along with Average Precision (AP@30 30, 50, AP@70)
4) Accident Spatial Grounding. We evaluate the model's grounding through reporting detection metrics: mean Intersection over Union (mloU) and Average Precision (AP@30, 30, AP @ 50, AP@ 70).
' performance on accident region and object
5.1 VIDEO-LEVEL TASKS
In this subsection, we present the results on video-level tasks of our proposed models, including TABot, TABot-Chat, and their comparison with several existing methods: Video-ILLaVA (Lin al.
7

---

Published as a conference paper at ICLR 2025
2023
TimeChat (Ren et al., 2024). VTimeLLM (Huang et al. 2024). GroundingGPT (Li et al. Qwen2-VI VL (Wang et al. 2024), and Semini--15-PPe (Reid et al. 2024).
2024
Table 2: Experimental results on video accident recognition traffic scenes. "@A A" and " "@N" represent the class-wise results accidents and normal scenes.
Video Accident Recognition
Methods
Acc
Recw
PrewA
F@FA
Rec
PrewNN
FT Nf
Video-I.LaV (Lin al.
2023
50.20
99.70
50.10
66.69
0.70
70.00
1.39
TimeChat |Ren al. 2024
54.65
91.80
52.67
66.93
17.50
68.09
27.84
VTimeLLM (Huang el al.
2024
50.00
100.00
50.00
66.67
0.00
0.00
0.00
GPoundingGiT (Li
2024
50.00
100,00
50.00
66.67
0.00
0.00
0.00
Qwen2-VL VL (Wang et al.
2024
72.65
53.46
87.23
66.29
92.08
66.16
77.00
Gemini-1 1.ii-115-PPP P.5-Pro (Reid et al. 2024
69.61
61.82
74.18
67.44
77.70
66.25
71,52
Bo (Ours)
81.00
78.65
85.10
81,75
83.77
76,90
80.19
TABot-Chat (Ours)
82.05
79.70
86.00
82.73
84.80
78.10
81.31
As presented in Table 2 most previous models struggle to recognize traffic accidents, with aceu- racies ranging from 50% to 54.65%. VTimeLLM and our baseline Grounding tend to classify all videos as abnormal, indicating several false positives. Although Qwen2-VL and Geminii-155PPP show some improvement, they tend to classify the videos as normal, exhibiting a lack of accident perception. In contrast, our TABot, trained on our TAU-106K dataset, demonstrates a significant improvement, reaching an accuracy of 80.95% and outperforming all prior methods. Further instruc- tion tuning with multi-round dialogue data, our TABot-Chat variant further presents an accuracy of 82.05% and improved overall performance for both accident and normal scenarios.
Table 3: Experimental results video accident description and accident temporal localization.
Video Accident Description
Accident Temporal Localization
Methods
BLEU
Rouge
BERT
GPT-4
AP6:30
APe550
AP
mloU
Video-I.LVa. (Lin al.
2023
22.20
24.81
60.72
26.17
TimeChat (Ren el all
6044
7.12
18.16
58.77
12.67
23.00
7.90
2.50
18.07
VTimeI.LMM (Huang ei al.
2024
25.25
23.32
60.84
18.62
GroundingGPT (La al. 2024
9.77
16.43
55.70
14.00
4.60
2.40
0.90
3,79
Qwen2 VL. Wang el al. 2024
15.38
23.64
61.61
39.80
32.91
15.76
5.42
20.75
Gemini-1.5- 1.5 [Reid ct al. 2024
12.83
19.57
60.79
23.66
13.87
5.14
1.64
9.31
TABot (Ours)
54.59
57.94
82.31
55.60
39.44
20.12
9.80
25.93
TABot-Chat (Ours)
55.70
58.32
83.78
55.73
37.90
20.70
7.80
25.33
For the tasks of video accident description and temporal localization, the performance of our mod- els is detailed in Table 3 TABot excels in generating accurate and contextually relevant accident descriptions, achieving the highest BERT and GPT-4 scores, indicating high semantic alignment with human judgments. As for the most challenging task of accident temporal localization, previous models struggled to pinpoint the occurrence of accidents, and only Owen2-VI. demonstrated cer- tain capability in fine-grained localization within videos. Our TABot also significantly surpasses all temporal localization
existing methods in this fine-gra lineggriieed task, establishing a new state-of-the-a performance. After instruction tuning, the TABot-Chat variant shows improved description capabil- ities as we expected, with a slight decrease in temporal localization performance. This decrease can be attributed to the model's 's enhanced conversational abilities, which may lead to slight decline in the model's focus on temporal localization.
5.2 IMAGE-LEVEL TASKS
In addition the video-level tasks, we also evaluate our proposed models on image-level tasks. The experimental results are presented in Tables 4 and where we compare our models against several
state-of-the-are methods (7hu al., 2023 Li et al. 2024
Bai
al. 2023 Wang et al. 2024 Reid
al. 2024 Achiam et al. 2023).
Table 4 presents the results of the image accident recognition. Our TABot outperforms all methods in image accident recognition, demonstrating the quality of our dataset and the effectiveness of our training strategies. As for accident description, the superior performance our models is evident, validating that model excels in generating accurate and contextually relevant descriptions of accidents. TABot-Chat, following instruction tuning, attains excellent values of 77.26 and 55.73 for BERT and GPT-4 scores, indicating high semantic alignment with human judgments.
8

---

Published as a conference paper at ICLR 2025
Table Experimental results on image accident recognition and description traffic scenes.
Methods
Image Accident
Recognition Rec
Image Accident Descrittion
Acc
Rec@F
Pre
FT
Pre@N
FT
BLEU
Rouge
BERT
GPT.4
MinGPTTa Zhu al. 2023
64.05
15.57
68.89
72.08
45.13
54.06
49.54
9.63
11.56
45.84
11.67
GroundingGiP at. 2024
63.75
79.15
67.45
72.84
39.25
54.20
45.53
7.22
7.81
45.00
21.08
Owen VL. Max
Bai et 2023
69.95
87.87
70.48
78.22
41.45
68.23
51.57
4.59
4.27
43.08
28.46
Owenn-VL Wang
2024
58.35
40.07
83.53
54,16
87.44
47.84
61.84
23.31
24.53
66.12
32.01
Gemini-1 1.5-Tro (Redd 2024
80.99
0.00
0.00
0.00
80,99
1.00
89.50
16,28
21.53
64.44
24.54
GPT-4a (Achian ef al. 2025
63.65
45.44
90.73
60.55
92.62
51.62
66.30
4.78
5.18
43.05
35.71
TAA
90.75
90.31
92.30
88.42
48.62
43.31
75.20
55.12
TABot-Chat (Ours)
90.50
94,90
89.33
92.03
84,48
92.36
88.24
50.28
45.67
77.26
55.73
Table 5: Experimental results accident region and object grounding in traffic images.
Accident Region Grounding
Accident Object Grounding
Methods
AP@: 30
AP@ 50
A 70
mlot.
AP@33
AP@:50
AP@ 70
mloU
MiniGPT4 (Zhu el al. 2023
50.57
34.85
24.67
39.36
70.33
56.65
33.24
49.72
GroundingGPPT er at. 2024]
26.55
14.25
7.82
62.23
49.06
27.34
43.75
Owen VT. VI. Max (Bai et 2024
43.73
26.47
12.79
30.72
59.97
45.27
28.25
43,00
Qwen2-VL VL Wang cr 2024
60.21
47.52
29.70
43.02
71.66
57.48
35.66
50.38
Gemini- 1.5-Pro [Roid et 2024
37.20
17.42
37.85
34.99
20.09
31.98
TABot (Ours)
80.05
70.03
45.52
57.83
78.05
65.86
39.88
54.95
TABot-Chat (Ours)
80,29
69.87
44.95
57.63
77.64
65.41
39.68
54.78
Table showcases the results for region- and object-level grounding. Our TABot significantly out- performs the baselines in terms of AP and mloU for both accident regions and objects. Similar to phenomenon observed in video tasks, instruction tuning slightly perturbs the model's grounding performance, but the overall performance remains competitive. The essential fine- grained ground- ing performance gap compared to previous methods further highlights the necessity of collecting traffic accident data and training models this specific domain.
5.3 ABLATION STUDY
The Effectiveness of Joint Training. To evaluate the impact of joint training on image and video data, additionally train the TABot using single modality (TABot-single trained on image or video data only) and compare the results with our joint training model (TABot). According to the main results in Tables 6 the joint training model outperforms the single modality models most tasks, especially for image-level tasks. The improvement in accident recognition and description tasks is more pronounced than in spatial grounding tasks, indicating that the primary benefit of video data is the scale-up in the amount of training data, which is particularly effective for tasks requiring richer contextual information. On the other hand, incorporating image data into video tasks leads to minor performance drop, suggesting that the model's focus video data may have slightly compromised its performance on image tasks.
The Effectiveness of VSA and NSR. Benefiting from unified video-image annotation pipeline, VSA can explicitly incorporate spatial grounding annotations at specific time frames into the training of video temporal localization. As shown in Table 7 our VSA strategy leads to consistent improve- ment in the model's temporal localization capabilities, demonstrating its effectiveness in involving spatial information NSR, it improves the model's 's overall performance across both image and video tasks by enhancing its capacity to differentiate accident events from normal content, as indicated in Table 7 However, there is a marginal decline in spatial grounding performance, and we attribute this to the model's focus on temporal localization, which may have led to a slight trade-off in spatial understanding. This drawback is compensated when the NSR is combined with the VSA, as all tasks achieve their best performance, demonstrating the complementary nature of these two strategies.
complementary signal enhance the model's temporal perception. As for
Table 6: Ablation study of separate (TABot--ingle) or joint (TABot) training on image and video data. "AG", "OG" & "TI." denote the AP@50 of Accident region Grounding, accident Object Grounding, and Temporal Localization.
Image Understanding
Video Understanding
Model
Acc
BERT
GPT-4
AG
0G
Acc
BERT
GPT-4
TL
TiBgle-ingge TABot
77.95
74.16
48.22
68.97
64.70
80.95
82.62
$4.63
20.28
90.75
75.20
55.12
70.03
65.86
81.00
82.31
55.60
20.12

---

Published as a conference paper at ICLR 2025
Table Ablation study on the additional training strategies.
TABot
Image Understanding
Video Understanding
NSR
Acc
BERT
GPT-4
AG
OG
Acc
BERT
GPT-4
TI.
88.45
75.09
54.28
68.57
64.06
80.50
82.08
55.23
19.30
V
88.00
74.73
53.82
70.20
64.21
81.90
81.72
54.78
18.90
88.60
74.83
53.91
70.36
64.55
80.80
82.26
55.53
19.92
90.75
75.20
55.12
70.03
65.86
81.00
82.31
55.60
20.12
Table 8: Ablation study the training strategy of the instructing tuning.
IABot-Chat
Image
Understanding
Video Understanding
Mixed Data
Task Flag
Acc
BERT
GPT-4
AG
Acc
BERT
GPT-4
TI.
84.55
75.44
50.18
68.71
64.52
79.50
82.43
53.32
5.10
85.50
75.59
52.83
69.14
64.76
79.35
82.14
54.51
13.30
x
88.30
76.56
52.04
69.22
64.11
80.20
83.10
55.40
18.90
90.45
77.20
55.73
69.46
64.96
81.25
83.51
55.73
19.50
Training Strategies for Chat Version. In the TABot-Char model, we observe that directly perform- ing instruction tuning without additional recipes significantly degrades the model's performance in functional tasks, for example, the accuracy for image accident recognition decreased to 84.55%. To maintain
improve the functional performance, took some data-centric approaches: (1) mix the datasets used for Functional Tuning and Instruction Tuning. (2) introduce task flags to spec- ify the target response for the model in multi-task framework. As presented in Table based on our training data paradigm, we successfully improve the conversational performance of TABot-Chat while maintaining excellent functional results.
Table 9: The ablation study of reasoning captions the temporal localization task.
Model
AP@
AP@50
AP@70
mloU
TABot
39.44
20.12
9.80
25.93
Reason Caption
34.20
16.90
6.60
21.67
Effectiveness of Reasoning Description. In the application of MLLMs to traffic accident under- standing, the most critical task is to achieve precise temporal localization of accidents in videos. The labeled reason caption our TAU-106K dataset is a portent the content of the accident, making accident detection and localization more trackable. Here evaluate the effectiveness of the reason- ing caption in the temporal localization task by conducting an ablation study as shown in Table 9 The results show that the removal of reasoning captions leads to significant performance drop in the temporal localization task, validating our claim that reasoning captions serve as valuable cues for accident understanding. Our future work will focus on developing more reasoning tasks based on the reasoning captions in TAU-106K to achieve accident forecasting and causality analysis tasks.
DISCUSSION AND CONCLUSION
To advance the exploration of MLLMs for traffic accident understanding, we introduced video- image-text joint dataset TAU-106K, which includes 51.5K video clips and 54.8K images, with high-quality annotations covering coarse- and fine-grained accident-oriented information. Upon our comprehensive dataset, we proposed TABot, a unified MI.LM that is compatible with video and im- age data and can handle various traffic accident understanding tasks including accident recognition, description, temporal localization, and spatial grounding. Our method and dataset lay the founda- tion for MLLM to infer and understand fine-grained representations of traffic accident scenarios. Our publicly available data and code will facilitate further research MLLM for traffic accidents. Future work will include more detailed grounding and addressing the hallucination problem.
ACKNOWLEDGMENTS
This work was partially supported by the National Natural Science Foundation China under Grants No. 62476201 and the New Cornerstone Science Foundation through the XPLORER PRIZE.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXivv2303.08777 2023.
Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pp. 5803-5812, 2017.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint ar 2111:2338./2966 2023.
Wentao Bao, Qi Yu, and Yu Kong. Uncertainty-bant traffic accident anticipation with spatio- temporal relational learning. In Proceedings the 28th ACM International Conference Mul- timedia, pp. 2682-2690, 2020.
Yunkang Cao, Xiaohao Xu, Chen Sun, Xiaonan Huang, and Weiming Shen. Towards generic anomaly detection and understanding: Large-scale visual- ssual-linguistic model (gpt-4v) takes the lead. arXiv preprint 2rXi:2231..02782, 2023.
Fu-Hsiang Chan, Yu- Ting Chen, Yu Xiang, and Min Sun. Anticipating accidents in dasheam videos. In Computer Vision-ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part IV 13, pp. 136-153. Springer, 2017.
Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang. Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305.1322 2023a.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra. Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023b.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306. /5/95, 2023c.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qing- long Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. tervvl: Scaling up vision foundation models and aligning for generic visual- visual-linguistic tasks. arXiv preprint ar 142338 2023d.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.1682) 2024.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint Xiv:22406007476 2024.
Abhimanyu Dubey, Abhinav Jaubri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint ar arXiv:2407.21700 2024.
Jianwu Fang, Dingxin Yan, Jiahuan Qiao, Jianru Xue, and Hongkai Yu. Dada: Driver attention prediction in driving accident scenarios. IEEE transactions intelligent transportation systems, 23(6):4959-499 4971, 2021.
Jianwu Fang, Lei-Lei Li, Kuan Yang. Zhedong Zheng, Jianru Xue. and Tat-Seng Chua. Cog- nitive accident prediction in driving scenes: A multimodality benchmark. arXiv preprint 1rXiv:2212.09388.. 2022a.

---

Published as a conference paper at ICLR 2025
Jianwu Fang, Jiahuan Qiao, Jie Bai, Hongkai Yu, and Jianru Xue. Traffic accident detection via self-supervised consistency learning in driving scenarios. IEEE Transactions on Intelligent Trans- portation Systems, 23(7):9001-911, 2022b.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, pp. 5267-5275, 2017.
Hadi Ghahremannezhad, Hang Shi, and Chengjun Liu. Real-time accident detection in traffic surveillance using deep learning. In 2022 IEEE international conference on imaging systems and techniques (IST). pp. 1-6. IEEE, 2022.
Sanjay Haresh, Sateesh Kumar, M Zeeshan Zia, and Quoc-Huy Tran. Towards anomaly detection in dashcam videos. In 2020 JEEE Intelligent Vehicles Symposium (IV), pp. 1407-1414. IEEE, 2020.
Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K Roy-Chowdhury, and Larry S Davis. Learning temporal regularity in video sequences. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 733-742, 2016.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmenter large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13504-13514, 2024.
Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower Il to grasp video moments. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp.
2024.
Caitlienne Diane C Juan, Jaira Rose A Bat-og, Kimberly K Wan, and Macario O Cordel II. Investi- gating visual attention-bases traffic accident detection model. Philippine Journal of Science, 150 (2), 2021.
Hoon Kim, Kangwook Lee, Gyeongjo Hwang, and Changho Suh. Crash to not crash: Learn to identify dangerous vehicles using simulator. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 978-985, 2019.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp. 706-715, 2017.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-197422 PMLR, 2023.
Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, et al. Groundinggpt: Language enhanced multi-modal grounding model. arXiv preprint arXiv:2401. 0660011111,, 2024.
Ronggin Liang, Yuanman Li, Jiantao Zhou, and Xia Li. Text-driven traffic anomaly detection with temporal high-frequency modeling in driving videos. IEEE Transactions on Circuits and Systems for Video Technology. 2024.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
Hui Lv, Chuanwei Zhou, Zhen Cui, Chumyan Xu, Yong Li, and Jian Yang. Localizing anomalies from weakly-labeled videos. IEEE transactions image processing, 30:4505-44 4515, 2021.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.0m 055244 2023.
12

---

Published as a conference paper at ICLR 2025
Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint TrX1::2306.14824, 2023.
Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting Zhuang, and Siliang Tang. Momentor: Advancing video large language model with fine-grained temporal reasoning. arXiv preprint arXiv:2402. 1435, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Ankit Parag Shah, Jean- Bapstite Lamare, Tuan Nguyen-Anh, and Alexander Hauptmann. Cadp: A novel dataset for cctv traffic camera based accident analysis. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1-9. IEEE, 2018.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint Siti::2003005538 2024.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing dision-languae model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9878-9888, 2021.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong-Yos You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-fre free baseline for video large language models. arXiv preprint arXiv:2407.15841, 2024.
Yajun Xu, Chuwen Huang, Yibing Nan, and Shiguo Lian. Tad: A large-scale benchmark for traffic accidents detection from video surveillance. arXiv preprint arXiv:2209, 12386, 2022.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Yu Yao, Mingze Xu, Yuchen Wang, David J Crandall, and Ella M Atkins. Unsupervised traffic ac- cident detection in first-person videos. In 2019 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 273-280. IEEE, 2019.
Yu Yao, Xizi Wang, Mingze Xu, Zelin Pu, Yuchen Wang, Ella Atkins, and David Crandall. Dota: Unsupervised detection of traffic anomaly in driving videos. IEEE transactions on paltern anal- ysis and machine intelligence, 45(1):444 4599, 2022.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du Bowen Zhang. Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere any granularity. 2023.
arXiv preprint arXiv:2310.0777
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction- tunee audio-visual language model for video understanding. arXiv preprint arXiv:2306.000 02855, 2023.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee. Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chumyuan Li. Llava-next: A strong zero video understanding model, April 2024, URL https:///laa alva vl. www.wl.ttti..... wightu..........g.
next
13

---

Published as a conference paper at ICLR 2025
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing usion-languase understanding with advanced large language models. arXiv preprint 200.22334..0092, 2023.
Rixing Zhu, Jianwu Fang, Hongke Xu, and Jianru Xue. Progressive temporal-pptiial-seaanii anal- ysis of driving anomaly detection and recounting, Sensors, 19(23)50098, 2019.
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL DETAILS ON DATA COLLECTION AND STATISTICS
A.1 DATA COLLECTION AND ANNOTATION
Data Source. We collect 106K media data, including 51,544 video clips and 54,767 images sourced directly or extracted from the video clips. In detail, as shown in Table TO our dataset comprises 13,536 video clips and 34,968 images from open-source traffic benchmarks, and 38,008 video clips and 19,799 images from social media platforms.
Table 10: The detailed distribution of our TAU-106K dataset from different data sources. Data Source
Video
# IImage
DoTA (Yao et al., 2022)
4,672
9,502
CCD (Bao et al., 2020)
4,464
6,010
DADA (Fang et al., 2021)
1,923
3,799
DashCam (Chan et al.. 2017)
1,727
2,348
TAD-1 (Lv et al., 2021)
352
628
TAD-henchmark (Xu al., 2022)
208
546
Drive-Anomaly (Zhu al., 2019)
105
207
RetroTrucks (Haresh et al., 2020)
56
348
TrafficS (Ghahremannezhad et al., 2022)
29
58
SUTD-TrafficQA (Xu et al., 2021)
9,674
CADP (Shah et al., 2018)
914
TaskFix (Juan et al., 2021)
713
YouTubeCrashh (Kim al., 2019)
221
Youtube (Social Media Platform)
29,364
12,345
BiliBili (Social Media Platform)
7,577
6,476
TikTok (Social Media Platform)
1,067
978
Data Annotation Process. We collaborate with a professional data annotation team to conduct the annotation of all traffic video and image data. The overall annotation process divided into two parts: video-based annotation and image-based annotation. Each part is separated into anno- tation and verification phases carried out by different annotators. The annotators in the verification phases are tasked with verifying the quality of each data item as "satisfactory" or "unsatisfactory". Unsatisfactory items were sent back to the annotation pipeline for refinement.
Annotation Phase
Steph video
Quality block duldeleess
All the aeriiuant diicing solid
Video Annotation
dhoing Bee wee daphiio Tir comur dhobee
colos cac sui lig and weth Iolorrt
Stppi-ide video
video
dee wurd
Sepp' Coffrrmsadeectoon
Siight
Quality Guidieen
congin and
and
and aod raceiig
Image Anoottton
To Bunnding fet sad accamely ti
Stepl: dlject bas condstee desziptian
Fao te shaer tir Bett shoe
Figure 5: The annotation process and quality control of the TAU-106K dataset.
We use an internal annotation tool to enable interactive use with our annotators and a diagram of the annotation protocol used in data engine, which is illustrated in Figure 5 Specifically, annotator in the video annotation phase adopts Steps 1, 2 and 3 to provide the timestamps, semantic tags and detailed description for accident video, and another annotator B focuses on quality verification.
15

---

Published as a conference paper at ICLR 2025
After the video annotation, the fine video data is sent to the image annotation phase, annotator C utilizes Steps and to perform keyframe selection and accident-related object annotation, and then annotator D conducts the image-level quality verification. We employed team of 50 experienced annotators and all of them followed the same annotation guidelines presented in both video and image verification phases. According four different annotators to uphold high standard for annotation. Moreover, benefiting from our proposed vidco-to-image annotation pipeline, the image annotators and verifiers double-check the annotations from the video phase, ensuring the consistency and accuracy of the annotations across different modalities, which is also label-efficient and cost-effective. We present some annotation samples in Figure
annotation workflow, each data item involved least
Motor Vehicle and Motor Vehicle (MV&mnMV)
0.40
0.52
vehicle
bicycle wearing black light, collided driving normally, causing knocket light.
Multiple non-Motor Vehicle (MnMV) Accident
A vehicle was driving on city street when two electric bicycle riders collided because one, dressed dark clothing, failed to brake in time. The other rider, wearing white helmet. was also struck, and both fell to the ground.
Multiple (MMV) Accident
0.45
0.65
The current vehicle traveling the road at the intersection, A black car did not slow doown and yield while making and collided with silver car that was going straight.
Single Motor Vehicle (SMV) Accident
current driving highway. car after tumning right
Vahicle and Padestrian (V&IP) Accident
0.35
0.71
A silver gray car hit padestrian crossing the street. The car stoppad, and the pecisstrian to the ground and then stood up
Figure 6: The video annotation examples of different types of accidents.
A.2
DATA STATISTICS
Data Categories. Our dataset covers various categories of traffic scenarios, objects, and accidents:
Traffic Scenarios: urban streets (49%), intersections (19%), country roads (17%), highways
(12%), and other traffic scenes (3%).
Objects: (58%), trucks (12%), electric bikes (11%), pedestrians (5%), vans (3%), bicycles
(3%), buses (2%), guardrails (2%), motorcycles (2%), and other objects (2%).
Accident Categories: multi-motor-vvehidee accidents (59%), motor-vehicle & non-motor-vehicle accidents (18%), single-motor-vehicl accidents (17%), vehicle & pedestrian accidents (4%), and multi-non-mootr--ehicch. accidents (2%).
Video Duration Distribution. As shown Figure (a), the video duration distribution of the accident-related video clips is visualized. The video clips collected from previous benchmarks or cropped from the raw crawled videos are relatively short, with the majority of the video clips lasting less than 20 seconds. Rarely, some video clips exceed 50 seconds, with the longest video clip lasting 12 minutes. For better visual presentation, we restrict the x-axis to seconds, which covers the majority of the video clips.
16

---

Published as a conference paper at ICLR 2025
4384
Dusaton benees))
Mirma trngo Inpwees.
(a) Video Danation Dismbbttio
(b) Temponal Interviis Distibation
Figure 7: (a) Video duration distribution of the accident-related video clips; (b) Temporal intervals distribution of the accident events in the video clips. "Frequency" refers to the number of occur- rences of videos with certain length in the TAU-106K dataset.
Temporal Localization Distribution. To statistically examine the temporal distribution of accident events the video clips, we analyze and visualize the annotated temporal intervals of these events, as illustrated in Figure (b). The temporal intervals corresponding to accident events in existing benchmarks tend to exhibit a relatively concentrated distribution, with the majority of accidents oc- curring towards the later portions of the video clips. In contrast, the temporal intervals of accident events in the newly collected video clips display a more dispersed pattern, resembling a Gaussian- like distribution. This broader temporal spread results in more balanced distribution of accident intervals across the video clips. Such distribution helps to mitigate potential biases toward spe- cific temporal segments, enabling MILMss more effectively and authentically learn the temporal dynamics and characteristics of accident events. By ensuring a diverse temporal representation, the proposed dataset enhances the robustness and generalizability of temporal localization models.
B ADDITIONAL DETAILS ON MODEL TUNING DATA
B. FUNCTIONAL TUNING DATA TEMPLATES
To avoid the trained model only responding to the specific instruction in the training data, we pre- defined set of questions for each task facilitate the model be activated facing diverse queries.
Single-turn: Temporal Localization. Besides the diverse question set, we provide a set of answer templates to prompt the model to generate human-like responses in the temporal localization task:
question_terpee
"Do you know the exact t times the traffic accident kioked off and wrapped up?", Limes the traffic accident in video?", "Any idea about the start and and time of that traffic accident we saw?",
"Can you give
"Show me when the traffic accidens geta going and when it's all over?",
"What is the start and end time of the traffic accident in the video?",
"Could you specify the timing of traffic accident's a onset and cenclusion?", "Please specify the precise timing of the traffic accident
and concluaton.", "At what timeatampa doea the traffic accident commence and finish?", duration of the traffic sccideat from beginning end?", "Whon is the traffic accidens initlated and terminated in the toozage?"
"Can you delineste
answer_templase
"Retween 11." ".format, "In the time period format,,, "During span "It happens in format,
formatt,,
"At (1.", ".omma,, "Exactly at (f.".format,
"Through 11.". 11.".format, "Within wlndow formmtt "In the { mark, ", format, "Azound ().
randon, choice (guestion_tempiates)) <video>
GPT:
zandom.co cho ice janswer_templater (annotation[ (*aceident- -eegentt
17

---

Published as a conference paper at ICLR 2025
Additionally, introduced in the main text, also prompt the model to describe the tent within the particular temporal segments: random samopedd normal segments or labeled accident segments. The pseudo code of temporal referring question question-answer answer pair generation is presented:
quest gonstton_teee ton_tepaaatee
"Whot'a happened during [) in the video?" .oomaat, "What'a the fnot dent in the period of 11?".tormat,
"Maybe something wrong happened duzing (3 in the provided .ileott,format, the period
"What's traffic situation
"Is the treffic flow captured by the video normel during (1?* .oommtt,, "Dose the sest dent happen doring traffic disruptions "Is there any indication of an abnormal treffie event during ()2", ".?"mfrrmat, mishaps the time "Aze there signs of vehicular distress or accidents within ()?*, format.
the =ideo?".format"
"Doea the video record
accidents around ()2",
"Could you identiff
(12",
Jser: randon. chot sguestion_teet atess) (annotation) ("aocident_segmentae <vicee>>
GPT: annotation ("ccccaaedddddddsssssiiaa
Single-turn: Accident Spatial Grounding. There are two spatial grounding tasks during our train- ing process: accident-involved object grounding and accident region grounding. As for the accident- involved object grounding task, the pseudo code of generating conversations is presented as follows:
question_templatee
"where is the I1 involved in the accident?" format, "Where is the It involved in the accident in the image?" "gzovide coordinates
format, forrmat, accident in the image?" .oomat,
() involved "Can you point out the (I involved in the scaident in the image and provide the coardinates of its location?", "eoacion"...omat,,
"Beip to locate the {) involved in the accident in the image and give its coordinates please." .ormmt,
"In the given image, could you find and tell the ecordinates of the () involved in the accidenz?" .ormat, location of
"Guide "oteiiiaae....oiii.e ".fomnt,
accident within image by providing "I'd Like to know the exact coordinates of the () involved in the accident in the photo.". format, "would you kindly provide the ecordinates of the () involved in the accident located in the pi piature?" "ore?".fonnee format,
involved
its coordinates."
"can you find the () involved in the accident in the image and give me the coordinates of located?",
where
for object in annotation| ('acciddeeis objects":; User: random, choice (queaticn (object [*label"] <image>
GPT: object "bbox"
We define the accident region the union of all the bounding boxes of the involved objects. The question-answe pair generation for the accident region grounding task is formulated as follows:
aoc1 Deentrregion UNTON jobject ['bbox"] for object in arnotation ("acoldent_ohjeotta")
question_templates -
the accicent in the image?", "where the accident?*,
"Whore ia the accidens in the image?",
"Pzovide the coordinates
"Can you point out the accident in the inage and provide the cocrdinates its location?",
"Help me to locste the accident in the inage and give me its coordinates, please.",
you find "Guide me to the location of the accident within the image providing its coordinates.",
the given image,
tell
coordinates the accident?*,
"I'd like
scasdent "Rould you kindly provide the coord nates of the accident located in the picture?",
exact coordinates
photo.",
"Can you find the accident the inage and give me the coordinates of where it is located?
Jser: randon, choice (questior_temppLate)) <image>
GPT: accident_reeee
18

---

Published as a conference paper at ICLR 2025
Single-turn: Accident Recognition and Description. Here is the pseudo code of generating versations for the task of accident recognition and description:
"Is there traffic accident in the given video clip?",
"Does this video capture a traffic accident?",
*Is traffic accidant cocurring any point in this vidao?",
"an you detect any traffic colliatonn In this video?", a road
accident visible in this video clip?", "Throughout this video, 1a there an incident involving traffic accident?",
there
"Do you observe a car accident happening
the sequence of this video?", "Does this video document any vehicular collisions crsshes?", "can you point out if there's a traffic-related accident depicted in this video?",
there any part this video that shows traffic mishap colliaton?",
"Rachh this video and confirm if a traffic accident takes place at any moment."
User: zandon. choice (question cvideo>
GPT:
Yes (for Accident) No (for Normal) annotation anncaatool/Patibb---dccripppee
B.2 INSTRUCTION TUNING DATA GENERATION
To generate instruction tuning data for traffic accident understanding, we design a set of instructions to guide the general-purpose multimodal language model, such as I.LaMA-70B (Dubey et al., 2024) in our work, to generate multi-turn conversations. Trained on the generated multiple rounds of conversations, our TABot expected to be endowed with a more comprehensive understanding of traffic accidents and equipped with the capability to provide more contextually relevant responses. The statistics the generated conversation pairs based on our dataset are summarized in Table [I
Specifically. follow the ln-context-leamming (ICL) paradigm of LLaVA (Liu al., 2024) and adapt it to our traffic accident understanding scenario. The detailed ICL prompting instruction for Ilama3 is illustrated in Table 12 In particular, the caption of the video is provided for the model to imagine the visual content of the video, which should be accident-oriented in our work. Therefore, organize the caption in structured way: Sentence 1 describes the video source; Sentence describes the accident-related content, in other words, the labeled accident description in the TAU- 106K dataset; Sentence complements the accident event with more detailed information, such as the temporal information and the objects involved in the accident.
Besides these complete sentences, we also list the annotated temporal segments and the bounding boxes of the sxcident-involvee objects in the video clips, leading the model to refer to the specific content when generating responses to the user queries. An example the structured caption and the generated multi-turn conversations are presented in Table 13
Table 11: Generated conversation pairs based our TAU-106K.
Task
Size
Response formatting
Detection & Description (Image)
55K
Detect and describe the accident
Detection & Description (Video)
52K
Detect and describe the accident
Temporal Localization
28K
(teturs, tend)
Temporal Referring
54K
Describe the accident
Accident Grounding
28K
[x0, y0, x1, yl]
Object Grounding
45K
[xO, y0, y1]
Complex Conprehension
70K
Multi-rount conversation
All
332K
19

---

Published as a conference paper at ICLR 2025
messages = "role: :"system", "content": Y"o are an Al visual assistant, and are seeing single video. What you see is provided with few sentences, describing the same video you looking at. Answer all questions you seeing the video. The video mainly focuses on traffic situation.
In particular, if there is an accident, the timesteps of the accident are provided in the format of (start.time, end_time with normalized time values.
an accident, specific object locations involved in the accident are given, along with detailed coordinates. The accident region is the area where the accident occurred, presented as the union of all the bounding boxes of the involved objects. These coordinates are in the form of bounding boxes, represented [xl, yl, x2, y2] with floating numbers ranging from 0 I. These values correspond to the top left x, top left y, bottom right x, and bottom right y.
addition,
Design a conversation between you and person asking about this photo. The answers should be that a visual assistant is seeing the vidco and answering the question. Ask diverse questions and give corresponding answers.
Include questions asking about the visual content of the video, including the object types, counting the objects, object actions, object locations, relative positions between objects, etc. Only include questions that have definite answers: (1) one can see the content in the video that the question asks about and can answer confidently: (2) one can determine confidently from the vidco that it is not the video. Do not ask any question that cannot be answered confidently.
Also include complex questions that are relevant to the content in the video, example, asking about background knowledge of the objects in the video, asking discuss accidents happening in the video, reasoning about the possible causes of the accident, analyzing the traffic rules that have been violated, etc. Again, do not ask about uncertain details. Provide detailed answers when answering complex questions. For example, give detailed examples reasoning steps make the content more convincing and well-organized. You can include multiple paragraphs necessary.") }
for sample in fewshot.samples.
messaaes append (f"role": "user", "content "; sample 'context" }}) messages. append (f (["role;; "trpll""""ssssstatii "assistant", "content" sample 'respense' }}
messages.. append ( ("rol""""""""""" "user", "content": 'nn join (query)))
For each query. we illustrate the prompt construction process for Llama3 to collect query [ 'response' from query 'context' J, using few-shot in-context-learinng, where examples are from fewshot. warehhttssmples.. each example including input sample ('context'' and output s sample 'response' ]. Note that messages is the final prompt. The bold text indicates the tailored instruction for the accident-oriente
Table 12:
conversation generation. In this illustra- tion, we provide the instruction used to generate the conversation response, please see also see its in-context-leaming example in Table 13 for details.
20

---

Published as a conference paper at ICLR 2025
Captions
This video is sourced from a traffic surveillance camera.
Surveillance cameras captured the road surface of a rural road, at 0.41 normalized timestamp, truck collided with an electric rider drove straight ahead, 0.53 normalized timestamp, the owner of electric bike fell the ground.
At 0.450 normalized timestamp. man in red coat riding an electric bike located at [0.324, 0,301, 0.488, 0.683) and a straight freight car located at [0.472, 0.101, 0.810, 0.849] were involved the accident.
accident timesteps: {0.41, 0.53]
accident objects:
man in a red coat riding an electric bike: [0.324, 0.301, 0.488, 0.683] straight freight car: [0.472. 1010 0.810, 0.849] accident.region. [0.324, 0010 0.810, 0.849]
Conversation
Question:
Can you describe the types of vehicles involved accident captured in video?
Answer:
The accident involved freight truck and an electric bike. The freight truck was moving straight ahcad rural road when it collided with the electric bike.
Question:
At what point in the video does the accident start and end?
Answer:
The accident starts the normalized timestamp of 0.41 and ends 0.53.
Question:
Where exactly did the accident occur in relation to the video frame?
Answer: The accident occurred within the coordinates [0.324, 0.101, 0.810, 0.849] in the video frame. This region represents unified area of both involved objects, the electric bike and the freight truck, during the accident.
Question:
How did the rider of the electric bike respond to the collision?
Answer:
After the collision, the rider of the electric bike fell to the ground. This incident happened around the 0.53 normalized timestamp in the video.
Question:
What is the appearance of the person involved in the accident?
Answer:
The person involved in the accident, specifically the rider of the electric bike, wearing red coat. This detail is captured around the 0.450 normalized timestamp the video when describing frame.
their location
==
Question:
Can you analyze the possible of the accident shown the video?
Answer:
Based on the video, one possible cause of the accident could be the relative positions and move- ments of the freight truck and the electric bike. The truck was moving straight ahcad on a rural road, suggesting that the visibility the reaction time could have been compromised for either the truck driver the electric bike rider. Without specific video evidence of traffic signals signs, might that perhaps there was failure to yield right way or issue with speed control. How- ever, definitive causes would require further investigation into the traffic conditions, driver behavior, and environmental factors the time of the accident.
Table 13: One example used in in-context-leaming to construct visual conversation data.
21

---

Published as a conference paper at ICLR 2025
C SUPPLEMENTARY ABLATION EXPERIMENTS
Fine-tuning Other Baselines. We extend experiments to include recent MILMs,, Video- LLaMA-2 (Cheng et al., 2024) and Qwen2-VL (Yang et al., 2024), fine-tuned on our proposed TAU- 106K dataset evaluate its quality and significance. The results, shown in Table 14] demonstrate significant performance improvements across all tasks after fine-tuning, validating the effectiveness of our dataset and training recipes. After fine-tuning, the performance gap between Video-LLaMA- 2 and GroundingGPP is reduced, highlighting the importance of fine-tuning target datasets for accident understanding tasks. Our ToB model based on GroundingGPT, still outperforms fine- tuned Video-LLaMA-2, particularly in image understanding tasks. As for the SOTA model Qwen2- VL, the pre-trained model already achieves competitive performance, and the fine-tuning on TAU- 106K further boosts the performance, reaching the highest performance in all tasks. These findings emphasize the necessity fine-tumin toning on domain- specific datasets, demonstrating the effectiveness and quality of our comprehensive TAU-106K dataset.
Table 14:
The results of fine- tine-tuning other baselines on TAU-10KK.
Video Understanding
Image Understanding
CLS (Acc)
TL (AP@50)
CAP (BERT)
CLS (Acc)
AG (AP@50)
CAP (BERT)
GroundingGPPT
50.00
2.40
55.70
63.75
14.25
45.00
TABot
81.00
20.12
82.31
90.75
70.03
75.20
Video-LLaMA-2
64.00
2.10
62.20
63.30
31.57
63.21
TABot
79.90
19.30
83.36
77.80
57.25
73.71
Qwen2-VL.
72.65
15.76
61.61
58.35
47.52
66.
TABot
82.65
22.50
83.09
92.00
77.61
76.32
Experiment Results of 7:3 Split. Since the amount of our TAU-106K dataset is large enough, 1/10 of the data (5K videos and 5K images) is sufficient for testing. However, we have conducted addi- tional experiments with a 7:3 train/test split verify the model's generalization ability to overfitting, as reported in Table 15 The model's performance remains consistent across different tasks with a slight decrease in accuracy, demonstrating its robustness to different train/test splits.
Table 15:
The results of fine-tuning TAU-106K with the 7:3 split, Video Understanding
Image Understanding
CLS (Acc)
TL (AP@50)
CAP (BERT)
CLS (Acc)
AG (AP@ 50)
CAP (BERT)
TABot (9:1)
81.00
20.12
82.31 81.57
90.75
70.03
75.20
TABot (7:3)
79.95
19.08
88.97
68.29
74.55
Data Imbalance in Fine-grained Categories. The class imbalance issue is inevitable in data col- lection and our data distribution also fairly reflects real-world situations. The primary purpose of this paper is to facilitate the development of MLLM on large-scale traffic datasets and learn models that closely resemble real-world conditions. Besides, we provide Table 16 to show the accuracy of different types of traffic accidents in both video/image accident recognition tasks. While categories like MnMV show slightly lower performance due to the limited amount of training data, the model's performances across different accident types are generally comparable. Overall, the model achieves consistent and satisfactory results across the normal class and the five accident classes.
Table 16: Accident proportion and class-wise accuracy for video and image accident recognition.
Accident
Proportion TAU- 106k
Accuracy for Video
Accuracy for Image Accident Recognition
Calegory
Accident Recognition
Normal
44.1 %
78.80
92.23
SMV
9.7%
80.56
79.75
MMV
32.9%
84.39
87.40
MV&nMV
9.8%
92.02
82.87
V&P
2.2%
94,29
84.78
MnMV
1.3%
77.78
78.26
22

---

Published as a conference paper at ICLR 2025
Qualitative Visualization and Comparison. We provide visualizations and comparisons the functional image and video understanding tasks Figure g and I1 Besides, the multi-turn
dialogue samples on traffic accident videos presented in Figure and Figure 13
Can you confirm the occurrence of vehicle accident this picture? Please respond with a "Yes' 'No'. Following that, describe the image in one sentence.
Question [RD]:
Ground Truth: Yes. An electric bicycle rider wearing black
collided with red car making right due faulty brakes.
GPT-4o: Yes, the image shows traffic accident involving red car that appears to have collided with another vehicle a busy intersection.
Qwen2-VI. VL: No. The image shows busy street with multiple cars and pedestrian crossing the road, but there visible accident.
Gemini-1.55 P.o-Pro No. red car turning right at an intersection a city. TABot-Chas (Ours): Yes. Because a clad electric bicycle rider missed the brakes, collided with red car turning right.
Question [SG]: Where is the traffic accident? The answer should be given in normalized
Tmiin Uminn Iman, Ymar format.
Ground Truth: [0.310, 0.343, 0.881, 0.935]
Qwen2-VL: [0.31, 0.35, 0.94, 0.97]
Gemini-1.5-Pro: [0.38, 0.24, 0.98, 0.94]
TABot-Chat (Ours): [0.321, 0.319, 0.878, 0.937]
Figure 8: One qualitative example of the functional image understanding tasks.
Question [RD]:
Does this video document any vehicular collisions or crashes? Please respond with 'Yes" or 'No'. Following that, describe the vidco in one sentence.
Ground Truth: Yes. The surveillance camera showed black colliding with blue-clad electric rider at city intersection, knocking the rider to ground.
Qwen2-VL: VL: Yes. A motorcycle is hit by car and the rider falls to the ground.
Gemini-1.5-Pro: Yes. A cyclist is hit by car at crosswalk.
TABot-Chat (Ours): Yes. Surveillance cameras captured the road surface of city street, black through the intersection, it collided with electric bicycle rider dressed in blue, the cyclist was knocked to the ground.
Show me when the traffic accident gets going and when it's all over. The answered timestamp should be formatted in normalized manner, using (start.time, end time).
Question [TL]:
Ground Truth: {0.55, 0.86}
Qwen2-VL: {0.50, 0.80)
Gemini-1.5-Pro: {0.80, 0.88). TABot-Chat (Ours): (0.56, 0.84)
Figure 9: One qualitative example of the functional video understanding tasks.
23

---

Published as a conference paper at ICLR 2025
Question [RD]: Are there any indications of traffic accident in this image? Please respond with "Yes" 'No'. Following that, describe the image in one sentence.
Ground Truth: Yes. blue car did not pay attention to the road conditions while making a left turn, which caused the current vehicle to take evasive action, leading to collision between the blue car and the current vehicle. Subsequently, both crashed into the left road guardrail.
GPT-4o: Yes, the image shows car colliding with barrier while another vehicle in the fore- ground.
Qwen2-VL: No. The image shows car driving on road with other vehicles turning left and traffic signs visible.
Gemini-1.5-Pro: No. The image shows driving highway, approaching bridge.
TABot-Chat (Ours): Yes. Because blue car not notice the condition of the road when turning left, causing the current vehicle to avoid it, the current vehicle collided with the blue car, then the blue car and the current vehicle collided with fence the left side of the road.
Question [SG]: In given image, could you me coordinates blue car recorded crash incidents? The answer should given in normalized [mmins Umin: Amaz Ymaz format.
Ground Truth: [0.392, 0.275, 0.998, 0.956]
Qwen2-VL: [0.38, 0.38. 1.0, 0.66]
Gemini-1.5-Pro: [0.36, 0.28, 0.88, 0.82]
TABot-Chat (Ours): [0.396, 0.283, 1.000, 0.939]
Figure 10: One qualitative example of the functional image understanding tasks.
Question [RD]: Do you observe car accident happening in the sequence this video? Please respond with "Yes' or 'No'. Following that, describe the video in one sentence.
Ground Truth: Yes. A vehicle was driving city street when two electric bicycle riders collided because one, dressed in dark clothing, failed brake in time. The other rider, wearing white helmer, was also struck,
both fell to ground.
Qwen2-VI. VL: Yes. A motorcycle falls down on the road in the video. Gemini-1..5Prre Yes. A dashcam video shows a motorcyclist falling from their vehicle after col- liding with another vehiele city street.
TABot-Chat (Ours): Yes. The vehicle was traveling on streets the city, because an electric bicycle rider dark clothes did not brake in time, it collided with a white helmeted electric bicycle rider, both riders fell to the ground.
Can you delineate the duration of the traffic accident from beginning end? The answered timestamp should formatted a normalized manner, using [start,time, end mime).
Question [TL]:
Ground Truth: {0.39, 0.75)
Qwen2-VL: {0.25, 0.75)
Gemini-1.5-Pro: {0.80, 0.95) TABot-Chat (Ours): (0.39, 0.72]
Figure 11: One qualitative example of the functional video understanding tasks.
24

---

Published as a conference paper at ICLR 2025
User: How many vehicles were involved in the accident captured by the traffic surveillance cameraa TABot-Chat: There were vehicles involved in the accident: silver bread truck and white car. Additionally, the white car also crashed into sign on the side of road.
User: What is the duration of the accident as seen in the video? TABot-Chat: The accident starts the normalized timestamp of 0.12 and ends at 1.0, indicating relatively long duration.
User: Can you describe the sequence of events in the accident?
Bot---hhat: The accident unfolded in two stages. Initially, at 0.218 normalized timestamp, the white car collided with the silver bread truck the intersection. Later, 0.609 normalized times- tamp, the same white car crashed
sign on side the road.
User: Where did the accident occur in relation to the video frame?
TABot-Chat: The accident occurred within the region bounded hy the coordinates [0.317, 0.000, 0.941, 0.279] in the video frame. This region encompasses both collision scenes, including the intersection where the white car silver bread truck and the area where white car crashed into the sign.
User: What be inferred about the white car's behavior during the accident? TABot-Chat: Based on the video, appears that the white car was involved two separate col- lisions. The first collision occurred when it the silver bread truck, and the second collision happened when it crashed into the sign on the side of the road. This sequence of events suggests that the white car may have lost control failed to stop after the initial collision, leading to the subsequent crash into the sign.
User: Can you provide any insights into the possible causes the accident?
The video evidence suggests that the white car may have been traveling at an exces- sive speed failed obey traffic rules, leading the initial collision with the silver bread truck. The subsequent crash into the sign could be attributed the driver's loss of control failure react appropriately after initial collision. However, without additional information, such as speed of the vehicles, road conditions, or driver behavior, a definitive cause cannot be determined solely from the footage.
TABot-Chat:
Figure 12: One qualitative example of the multi-turn dialogue on the traffic accident video.
25

---

Published as a conference paper at ICLR 2025
What type of vehicles were involved in the accident captured by the traffic surveillance camera?
User:
TABot-Chat: accident involved black car and blue tricycle. The black car was moving on the country road when collided with the blue tricycle, which was turning left.
What was the cause of the accident according the video? TABot-Chat: The accident occurred because the black car failed to brake on time, resulting in collision with the blue tricycle.
User:
User: At what point in the video does the accident start and end?
TABot-Chat: The accident starts the normalized timestamp of 0.30 and ends 0.89.
User: Where exactly did the accident occur relation to the video frame?
TABot-Chat: The accident occurred within the coordinates [0.490. 0.223, 0.582, 0,410] in the video frame. This region represents the unified area of both involved objects, the blue tricycle and the black car, during the accident.
What happened to the tricycle and the cyclist as result of the accident? TABot-Chat: The tricycle overturned, and both the tricycle and the cyclist were hit from the road surface.
User:
User: Can you analyze the possible consequences of the accident shown in the video? TABot-Chat: Based on the video, the consequences of the accident could be severe. The tricycle overturning and the cyclist being hit from the road surface suggest potential injuries to the cyclist. The accident could also have caused damage both vehicles involved. Additionally, the accident may have caused disruption to traffic flow on the country road. thorough investigation would be necessary to determine the full extent of the consequences.
User: What traffic rule might have been violated by the black car driver? Tottttthhat: The black driver might have violated the rule maintaining safe following distance or failing to yield the blue tricycle, which was turning left. The driver's failure to brake on time suggests possible failure to exercise due care and caution while driving.
Figure 13: One qualitative example of the multi-turn dialogue on the traffic accident video.
26


Reviews:
[{"review_id": "o5fVucGVYQ", "reviewer": "Reviewer_bWpe", "summary": "This paper is focused on traffic accident analysis. The authors first introduced the TAU-106K dataset which contains both video and image modalities, and a wide range of fine-grained annotations including both spatial and temporal information. Then trained and evaluated a MLLM model, TABot. The training process has two stages, functional tuning and instruction tuning. The functional tuning involves multiple LLM training tasks to improve model performance in spatial and temporal understanding. The instruction tuning uses multi-round dialogue data generated based on captions by GPT-4o to learn the cause of accidents or prevention suggestions, which leads to the TABot-Chat model. TABot and TABot-Chat show superior performance in comprehensive understanding of traffic accident scenarios.", "questions": "- TAU-106K is used as both the training set (=90% of the data) of the proposed TABot model and the testing set (=10% of the data) which is unbalanced. It is better to give more train/test splits tests, such as 7:3 for checking the model is not only benefit from the overfitting learning.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Accident data as the rare cases are hard to collect, TAU-106K dataset provides 56% accident data and 44% normal which is a good balance overall. It has both video and image modalities with well annotated information from accident category, temporal location and description which is good for the comprehensive analysis of the accident videos from the ego car view.\n\n- The majority of TAU-106K data is high quality as 720p, as image and video both are provided, it enables image-wise spatial annotation and training with that spatial information. There are also evaluation and ablation studies of (i) image-only model vs normal model, (ii) image task and video task, which is good insight for the community.\n\n- The authors have annotators to precisely identify start and end timestamps of accidents within videos. This metadata is important in traffic scene analysis because it can distinguish the accident frames from non-accident frames, and also helps the Negative Segment Referring (NSR) task.\n\n- The traffic accident description includes the aftermath, to help comprehend traffic accidents, e.g. potential causes of the accident.\n\n-  Proposed method TABot follows the mainstream manner of training multi-modal LLM with a Q-former architecture. But it is good enough to test the different downstream tasks.\n\n- Evaluation is done by comparison of the existing methods on the proposed dataset showing the challenge on the current accident localization.", "weaknesses": "- Comparison method for OSS model, which is not trained or fine-tuned on the same  train dataset that is hard to convince the the proposal method advantages. \n\n- Accident data is not balanced for the fine-grained categories, e.g., MMV accidents take the major distribution. It is a concern for the training bias, this needs to be considered in the model design to avoid the data distribution affection for the inference performance. While, accidents and normal still balanced good.\n\n- The reason behind the conclusion regarding the “coverage of the collected traffic data is good” needs to be supported more. As the paper focuses on traffic accident analysis and creating traffic accident dataset, it would be great to share more insights/data/analysis regarding this (e.g., list common traffic types/objects/scenarios and show that the dataset has covered them).\n\n- Lack of comparison with the existing dataset regarding traffic accidents. It should have a single section and related work to describe it, currently the it is mixed in the related works, also lack of the comparison with some literatures such as : \n\nWTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YDOCvRGps2", "reviewer": "Reviewer_oM9T", "summary": "The authors introduce a novel dataset, TAU106k, designed to facilitate the training of multimodal Large Language Models (MLLMs) for a more comprehensive understanding of accidents. This dataset supports tasks such as accident recognition, temporal accident localization, and spatial-temporal object grounding. Additionally, the authors propose a method called TABot and compare its performance with various MLLMs on the TAU106k dataset.", "questions": "1. Please clarify the motivation behind this work (Weakness #1).\n2. Please address the copyright issues (Weakness #3).\n3. Please explain how consistency in annotation is maintained (Weakness #4).\n4. Please provide a rationale for the absence of a reasoning task in the experiments (Weakness #5).\n5. Please discuss the limited performance improvement of the proposed algorithm (Weakness #6).", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Traffic accident understanding is a critical task in the development of automated driving systems.** To support the training of multimodal Large Language Models (MLLMs) for traffic accident understanding, the authors have collected and annotated a large-scale dataset, TAU106k. This dataset includes structured annotations for accident recognition, temporal accident localization, and spatial-temporal object grounding. Overall, TAU106k comprises 52K video clips and 54K images.\n\n2. **Training Grounding GPT-7B on this dataset yields substantial performance improvements across multiple tasks,** demonstrating that TAU106k effectively supports domain-specific applications.", "weaknesses": "**While the paper demonstrates significant effort in creating the dataset, I have the following concerns:**\n\n1.\t**Lack of clear motivation:** In the introduction, the authors do not clearly articulate the motivation for applying MLLMs to traffic accident understanding, especially since they begin by discussing “traffic accident detection.” Although MLLMs are attracting significant attention in the community, the value of using MLLMs, which may not necessarily operate in real-time, is unclear in this context. Justifying the need for this approach would strengthen the paper. In the rebuttal, I would like to hear the authors' feedback on this concern.\n\n2.\t**Limited comparison with existing datasets:** The authors are encouraged to provide a more comprehensive comparison between TAU106k and other existing datasets to highlight its unique contributions. For example, please demonstrate key features such as size, diversity of scenarios, types of annotations, or specific tasks supported.\n\n3.\t**Potential copyright issues with the dataset:** In line 191, the authors mention that the dataset is compiled from various sources, including academic datasets, YouTube, and BiliBili. While they have promised to release the dataset after acceptance, I am concerned that some of the collected data, particularly from platforms like YouTube, may involve copyright issues. If this is the case, the dataset and model release may not be feasible as promised, which would weaken the manuscript’s contribution. Please provide more details on your data collection process, including any steps taken to address copyright concerns, such as obtaining permissions or using only publicly licensed content\n\n4.\t**Annotator consistency and accuracy:** In lines 233-234, the authors state that their annotation strategy ensures consistency and coherence. However, it is unclear how many annotators are assigned to each item and how they ensure labeling accuracy if only one annotator is involved. Please provide the details on the annotation process, such as the number of annotators per item, any inter-annotator agreement measures used, and quality control procedures implemented.\n\n5.\t**Lack of a reasoning task:** In line 219, the authors note that “annotators are encouraged to infer the potential causes of the accident,” but this statement is vague. While the dataset could enable reasoning tasks for traffic accident understanding, the authors do not provide a detailed description or corresponding experiments. This omission detracts from the dataset’s uniqueness.\n\n6.\t**Limited improvement from the proposed algorithm:** In Tables 5 and 6, the proposed components show only incremental improvements over the base MLLM (GroundingGPT-7B). If the authors conducted cross-validation or multiple runs, these improvements might not be statistically significant, which could diminish the contribution of the proposed method.\n\nComments:\n\n1. In L191, what is the method used from scene change detection?\n2. L214, there is a typo (”Due”).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7WKbuLHVKH", "reviewer": "Reviewer_1r9r", "summary": "This paper proposes a large-scale multimodal dataset, TAU-106K, aimed at addressing the limitations of multimodal large language models (MLLMs) in high-level and fine-grained anomaly understanding, specifically for traffic accident scenarios. The TAU-106K dataset comprises images and videos sampled from existing datasets and retrieved from online sources, along with annotated text data. The manuscript also proposed a two-stage fine-tuning approach named TABot to integrate multi-level accident understanding tasks, including accident recognition, spatiotemporal grounding, and an additional accident description task to further enhance the model’s understanding of traffic accidents. While TAU-106K shows potential as a valuable resource, the paper would benefit from a clearer elaboration on the unique challenges of the traffic accident domain and additional experimental comparisons.", "questions": "1. How does this dataset compare with existing related datasets? Can specific data be provided? Are there any additional benefits beyond language descriptions? What sets it apart from traditional VQA tasks?\n2. Could the annotation process from video to image be more detailed?  And how is multi-turn dialogue accuracy ensured? What steps are taken to ensure annotation reliability, efficiency, and cost-effectiveness?\n3. Anomalous events typically show a long-tail distribution. Is this characteristic present in the dataset? How are the resulting distributions in the training and test sets structured?\n4. Instruction tuning is classic and common. What is the two-step fine-tuning approach that differs from previous methods? How does Video Spatial Alignment differ specifically from GroundingGPT? Besides input data differences, what specific innovations are presented? Negative Segment Referring is vaguely explained—does it refer to non-accident samples, and how does it differ from standard practices? Hybrid data and task markers are often seen in model fine-tuning; what is their connection to traffic accident detection?\n5. The experimental section and method descriptions aren’t well aligned. For instance, what are the distinctions between Negative Segment Referring, multi-turn dialogues, TABot (Video), TABot (Ours), and TABot-Chat (Ours)? Does Ours use both video and image training data? How is multi-turn dialogue in -Chat specifically designed? More clarity is needed.\n6. What is the relationship between the four tasks: accident recognition, accident description, temporal localization, and spatial grounding? Could these tasks validate each other to improve model performance? What is Image Understanding in the experiment section, and how does it relate to the other four tasks?\n7. The authors achieved remarkable results with GroundingGPT. How is the performance of the proposed method based on other models?\n8. For some metrics, GroundingGPT achieves 100% accuracy, while for others, it scores 0%. What causes this drastic disparity, and could it be related to experiment settings or metric choices?\n9. In the case study, how do prediction results from different methods compare to ground-truth annotations? Additionally, some details need review. For instance, the conclusion mentions an image count of 55k, which seems inconsistent with earlier descriptions.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper focuses on the important field of traffic accident scene understanding, addressing the current limitations of large models in high-level and fine-grained semantic comprehension. It constructs large-scale multimodal models, with a particular emphasis on anomalies, which are critical focal points in the field of autonomous driving. The introduction of this dataset is expected to further promote advancements in this area.\nThe proposed dataset is designed for various tasks related to traffic accident scenes, including accident recognition, accident description, temporal localization, and spatial grounding. It also features multi-granularity annotations for the collected images and videos, effectively aiding the model in further learning and understanding of traffic accidents.", "weaknesses": "The presentation of the dataset requires further optimization, as the focus on traffic accidents carries significant practical implications and research value. However, the paper lacks clarity regarding the challenges associated with understanding traffic accidents, and the characteristics of such incidents need to be more precisely articulated.\nWhile the two-step fine-tuning approach is widely recognized, the distinctions between the proposed method and established general methods are not clearly delineated. This creates a disconnect between the methodology and the traffic accident focus of the dataset. Furthermore, the method’s design does not sufficiently address specific tasks, necessitating clearer articulation of its innovations and contributions.\nIn the experimental section, there is a need for more comprehensive validation of the challenge of dataset and the effectiveness of method, particularly concerning experimental setup and evaluation metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UtdVtCX7Cj", "reviewer": "Reviewer_Vxed", "summary": "This paper proposes a new dataset, TAU-106K, to help MLLMs gain a comprehensive understanding of traffic accidents. In addition, the paper also proposes a two-stage fine-tuning mechanism to use the proposed dataset. The experimental results basically prove the effectiveness of the proposed method and data.", "questions": "The author can think more about the proposed data and benchmarks:\n\n* In addition to the defined traffic accidents, there are actually more conner cases, which are often the most difficult to learn in the open world. How to use the proposed method to let MLLMs learn this knowledge?\n\n* Is there greater potential for the generalization of data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The dataset proposed in this paper covers a wide range of knowledge about traffic accidents, which has greatly promoted the development of MLLMs in this specific field. The diversity and details of the proposed dataset are meaningful and meet the expectations of MLLMs to learn specific knowledge.", "weaknesses": "This paper provides a meaningful dataset and benchmark, but I still have some concerns:\n\n* Although the experimental results basically prove the superiority (table 1, 2, 3, 4), the highlighted information is confusing: Is the performance improvement due to the rich knowledge of the data or the fine-tuning strategy? Both aspects should have an effect, so there should be more comparisons and ablations. For example, you can use your proposed dataset to fine-tune other baseline models.\n\n* Accident Recognition uses the Yes and No responses of the model to make binary classifications. In a random situation, are the probabilities of the model responding Yes and No balanced? If not, will this cause the evaluation to be disturbed by some irrelevant prior knowledge? You can start from reporting the base rates of Yes/No responses.\n\n* There are some spelling and grammatical errors that the author would like to note (for example: line 192 'hey', line 238 'is', line 375 'a the', ...).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)