TAU-106K: A New Dataset for Comprehensive Understanding of Traffic Accident

Authors: Yixuan Zhou, Long Bai, Sijia Cai, Bing Deng, Xing Xu, Heng Tao Shen
Primary area: datasets and benchmarks
Keywords: Anomaly Detection, Traffic Accident Understanding, Multimodal Large Language Models, Instructional Dataset

Abstract:
Multimodal Large Language Models (MLLMs) have demonstrated impressive performance in general visual understanding tasks. However, their potential for high-level, fine-grained comprehension, such as anomaly understanding, remains unexplored. Focusing on traffic accidents, a critical and practical scenario within anomaly understanding, we investigate the advanced capabilities of MLLMs and propose TABot, a multimodal MLLM specialized for accident-related tasks. To facilitate this, we first construct TAU-106K, a large-scale multimodal dataset containing 106K traffic accident videos and images collected from academic benchmarks and public platforms. The dataset is meticulously annotated through a video-to-image annotation pipeline to ensure comprehensive and high-quality labels. Building upon TAU-106K, we train TABot using a two-step approach designed to integrate multi-granularity tasks, including accident recognition, spatial-temporal grounding, and an auxiliary description task to enhance the model's understanding of accident elements. Extensive experiments demonstrate TABot's superior performance in traffic accident understanding, highlighting not only its capabilities in high-level anomaly comprehension but also the robustness of the TAU-106K benchmark. Our code and data will be available at https://github.com/cool-xuan/TABot.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730132101650

thread_type:
official_review

<thread_object>
creator: reviewer_vxed

<content>
summary:
This paper proposes a new dataset, TAU-106K, to help MLLMs gain a comprehensive understanding of traffic accidents. In addition, the paper also proposes a two-stage fine-tuning mechanism to use the proposed dataset. The experimental results basically prove the effectiveness of the proposed method and data.

strengths:
The dataset proposed in this paper covers a wide range of knowledge about traffic accidents, which has greatly promoted the development of MLLMs in this specific field. The diversity and details of the proposed dataset are meaningful and meet the expectations of MLLMs to learn specific knowledge.

weaknesses:
This paper provides a meaningful dataset and benchmark, but I still have some concerns:

* Although the experimental results basically prove the superiority (table 1, 2, 3, 4), the highlighted information is confusing: Is the performance improvement due to the rich knowledge of the data or the fine-tuning strategy? Both aspects should have an effect, so there should be more comparisons and ablations. For example, you can use your proposed dataset to fine-tune other baseline models.

* Accident Recognition uses the Yes and No responses of the model to make binary classifications. In a random situation, are the probabilities of the model responding Yes and No balanced? If not, will this cause the evaluation to be disturbed by some irrelevant prior knowledge? You can start from reporting the base rates of Yes/No responses.

* There are some spelling and grammatical errors that the author would like to note (for example: line 192 'hey', line 238 'is', line 375 'a the', ...).

questions:
The author can think more about the proposed data and benchmarks:

* In addition to the defined traffic accidents, there are actually more conner cases, which are often the most difficult to learn in the open world. How to use the proposed method to let MLLMs learn this knowledge?

* Is there greater potential for the generalization of data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730366046293

thread_type:
official_review

<thread_object>
creator: reviewer_1r9r

<content>
summary:
This paper proposes a large-scale multimodal dataset, TAU-106K, aimed at addressing the limitations of multimodal large language models (MLLMs) in high-level and fine-grained anomaly understanding, specifically for traffic accident scenarios. The TAU-106K dataset comprises images and videos sampled from existing datasets and retrieved from online sources, along with annotated text data. The manuscript also proposed a two-stage fine-tuning approach named TABot to integrate multi-level accident understanding tasks, including accident recognition, spatiotemporal grounding, and an additional accident description task to further enhance the model’s understanding of traffic accidents. While TAU-106K shows potential as a valuable resource, the paper would benefit from a clearer elaboration on the unique challenges of the traffic accident domain and additional experimental comparisons.

strengths:
This paper focuses on the important field of traffic accident scene understanding, addressing the current limitations of large models in high-level and fine-grained semantic comprehension. It constructs large-scale multimodal models, with a particular emphasis on anomalies, which are critical focal points in the field of autonomous driving. The introduction of this dataset is expected to further promote advancements in this area.
The proposed dataset is designed for various tasks related to traffic accident scenes, including accident recognition, accident description, temporal localization, and spatial grounding. It also features multi-granularity annotations for the collected images and videos, effectively aiding the model in further learning and understanding of traffic accidents.

weaknesses:
The presentation of the dataset requires further optimization, as the focus on traffic accidents carries significant practical implications and research value. However, the paper lacks clarity regarding the challenges associated with understanding traffic accidents, and the characteristics of such incidents need to be more precisely articulated.
While the two-step fine-tuning approach is widely recognized, the distinctions between the proposed method and established general methods are not clearly delineated. This creates a disconnect between the methodology and the traffic accident focus of the dataset. Furthermore, the method’s design does not sufficiently address specific tasks, necessitating clearer articulation of its innovations and contributions.
In the experimental section, there is a need for more comprehensive validation of the challenge of dataset and the effectiveness of method, particularly concerning experimental setup and evaluation metrics.

questions:
1. How does this dataset compare with existing related datasets? Can specific data be provided? Are there any additional benefits beyond language descriptions? What sets it apart from traditional VQA tasks?
2. Could the annotation process from video to image be more detailed?  And how is multi-turn dialogue accuracy ensured? What steps are taken to ensure annotation reliability, efficiency, and cost-effectiveness?
3. Anomalous events typically show a long-tail distribution. Is this characteristic present in the dataset? How are the resulting distributions in the training and test sets structured?
4. Instruction tuning is classic and common. What is the two-step fine-tuning approach that differs from previous methods? How does Video Spatial Alignment differ specifically from GroundingGPT? Besides input data differences, what specific innovations are presented? Negative Segment Referring is vaguely explained—does it refer to non-accident samples, and how does it differ from standard practices? Hybrid data and task markers are often seen in model fine-tuning; what is their connection to traffic accident detection?
5. The experimental section and method descriptions aren’t well aligned. For instance, what are the distinctions between Negative Segment Referring, multi-turn dialogues, TABot (Video), TABot (Ours), and TABot-Chat (Ours)? Does Ours use both video and image training data? How is multi-turn dialogue in -Chat specifically designed? More clarity is needed.
6. What is the relationship between the four tasks: accident recognition, accident description, temporal localization, and spatial grounding? Could these tasks validate each other to improve model performance? What is Image Understanding in the experiment section, and how does it relate to the other four tasks?
7. The authors achieved remarkable results with GroundingGPT. How is the performance of the proposed method based on other models?
8. For some metrics, GroundingGPT achieves 100% accuracy, while for others, it scores 0%. What causes this drastic disparity, and could it be related to experiment settings or metric choices?
9. In the case study, how do prediction results from different methods compare to ground-truth annotations? Additionally, some details need review. For instance, the conclusion mentions an image count of 55k, which seems inconsistent with earlier descriptions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570546136

thread_type:
official_review

<thread_object>
creator: reviewer_om9t

<content>
summary:
The authors introduce a novel dataset, TAU106k, designed to facilitate the training of multimodal Large Language Models (MLLMs) for a more comprehensive understanding of accidents. This dataset supports tasks such as accident recognition, temporal accident localization, and spatial-temporal object grounding. Additionally, the authors propose a method called TABot and compare its performance with various MLLMs on the TAU106k dataset.

strengths:
1. **Traffic accident understanding is a critical task in the development of automated driving systems.** To support the training of multimodal Large Language Models (MLLMs) for traffic accident understanding, the authors have collected and annotated a large-scale dataset, TAU106k. This dataset includes structured annotations for accident recognition, temporal accident localization, and spatial-temporal object grounding. Overall, TAU106k comprises 52K video clips and 54K images.

2. **Training Grounding GPT-7B on this dataset yields substantial performance improvements across multiple tasks,** demonstrating that TAU106k effectively supports domain-specific applications.

weaknesses:
**While the paper demonstrates significant effort in creating the dataset, I have the following concerns:**

1.	**Lack of clear motivation:** In the introduction, the authors do not clearly articulate the motivation for applying MLLMs to traffic accident understanding, especially since they begin by discussing “traffic accident detection.” Although MLLMs are attracting significant attention in the community, the value of using MLLMs, which may not necessarily operate in real-time, is unclear in this context. Justifying the need for this approach would strengthen the paper. In the rebuttal, I would like to hear the authors' feedback on this concern.

2.	**Limited comparison with existing datasets:** The authors are encouraged to provide a more comprehensive comparison between TAU106k and other existing datasets to highlight its unique contributions. For example, please demonstrate key features such as size, diversity of scenarios, types of annotations, or specific tasks supported.

3.	**Potential copyright issues with the dataset:** In line 191, the authors mention that the dataset is compiled from various sources, including academic datasets, YouTube, and BiliBili. While they have promised to release the dataset after acceptance, I am concerned that some of the collected data, particularly from platforms like YouTube, may involve copyright issues. If this is the case, the dataset and model release may not be feasible as promised, which would weaken the manuscript’s contribution. Please provide more details on your data collection process, including any steps taken to address copyright concerns, such as obtaining permissions or using only publicly licensed content

4.	**Annotator consistency and accuracy:** In lines 233-234, the authors state that their annotation strategy ensures consistency and coherence. However, it is unclear how many annotators are assigned to each item and how they ensure labeling accuracy if only one annotator is involved. Please provide the details on the annotation process, such as the number of annotators per item, any inter-annotator agreement measures used, and quality control procedures implemented.

5.	**Lack of a reasoning task:** In line 219, the authors note that “annotators are encouraged to infer the potential causes of the accident,” but this statement is vague. While the dataset could enable reasoning tasks for traffic accident understanding, the authors do not provide a detailed description or corresponding experiments. This omission detracts from the dataset’s uniqueness.

6.	**Limited improvement from the proposed algorithm:** In Tables 5 and 6, the proposed components show only incremental improvements over the base MLLM (GroundingGPT-7B). If the authors conducted cross-validation or multiple runs, these improvements might not be statistically significant, which could diminish the contribution of the proposed method.

Comments:

1. In L191, what is the method used from scene change detection?
2. L214, there is a typo (”Due”).

questions:
1. Please clarify the motivation behind this work (Weakness #1).
2. Please address the copyright issues (Weakness #3).
3. Please explain how consistency in annotation is maintained (Weakness #4).
4. Please provide a rationale for the absence of a reasoning task in the experiments (Weakness #5).
5. Please discuss the limited performance improvement of the proposed algorithm (Weakness #6).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718822675

thread_type:
official_review

<thread_object>
creator: reviewer_bwpe

<content>
summary:
This paper is focused on traffic accident analysis. The authors first introduced the TAU-106K dataset which contains both video and image modalities, and a wide range of fine-grained annotations including both spatial and temporal information. Then trained and evaluated a MLLM model, TABot. The training process has two stages, functional tuning and instruction tuning. The functional tuning involves multiple LLM training tasks to improve model performance in spatial and temporal understanding. The instruction tuning uses multi-round dialogue data generated based on captions by GPT-4o to learn the cause of accidents or prevention suggestions, which leads to the TABot-Chat model. TABot and TABot-Chat show superior performance in comprehensive understanding of traffic accident scenarios.

strengths:
- Accident data as the rare cases are hard to collect, TAU-106K dataset provides 56% accident data and 44% normal which is a good balance overall. It has both video and image modalities with well annotated information from accident category, temporal location and description which is good for the comprehensive analysis of the accident videos from the ego car view.

- The majority of TAU-106K data is high quality as 720p, as image and video both are provided, it enables image-wise spatial annotation and training with that spatial information. There are also evaluation and ablation studies of (i) image-only model vs normal model, (ii) image task and video task, which is good insight for the community.

- The authors have annotators to precisely identify start and end timestamps of accidents within videos. This metadata is important in traffic scene analysis because it can distinguish the accident frames from non-accident frames, and also helps the Negative Segment Referring (NSR) task.

- The traffic accident description includes the aftermath, to help comprehend traffic accidents, e.g. potential causes of the accident.

-  Proposed method TABot follows the mainstream manner of training multi-modal LLM with a Q-former architecture. But it is good enough to test the different downstream tasks.

- Evaluation is done by comparison of the existing methods on the proposed dataset showing the challenge on the current accident localization.

weaknesses:
- Comparison method for OSS model, which is not trained or fine-tuned on the same  train dataset that is hard to convince the the proposal method advantages. 

- Accident data is not balanced for the fine-grained categories, e.g., MMV accidents take the major distribution. It is a concern for the training bias, this needs to be considered in the model design to avoid the data distribution affection for the inference performance. While, accidents and normal still balanced good.

- The reason behind the conclusion regarding the “coverage of the collected traffic data is good” needs to be supported more. As the paper focuses on traffic accident analysis and creating traffic accident dataset, it would be great to share more insights/data/analysis regarding this (e.g., list common traffic types/objects/scenarios and show that the dataset has covered them).

- Lack of comparison with the existing dataset regarding traffic accidents. It should have a single section and related work to describe it, currently the it is mixed in the related works, also lack of the comparison with some literatures such as : 

WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding

questions:
- TAU-106K is used as both the training set (=90% of the data) of the proposed TABot model and the testing set (=10% of the data) which is unbalanced. It is better to give more train/test splits tests, such as 7:3 for checking the model is not only benefit from the overfitting learning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732442373581

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Response to Reviewers

comment:
We sincerely thank all reviewers for acknowledging the significance and novelty of our work.
We are greatly encouraged by the reviewers' recognition of the following aspects:

1. The significance and comprehension of the proposed TAU-106K dataset for traffic accident understanding.
2. The effectiveness of the proposed training recipes and the improvements they bring to the baseline model.
3. The publication of the dataset and the code to facilitate future research in the field.

We have carefully considered all reviewers' comments and suggestions and summarized some common concerns about our dataset and methodology as follows:

**1. [Comparison with Existing Datasets]:** We provide a more comprehensive comparison between TAU-106K and other datasets, focusing on key features such as size, domain, annotation types, and the characteristics of the textual captions. Through the comparison, the unique advantages of TAU-106K in terms of the number of videos, the variety of annotations, and the diversity and flexibility of the free-form accident captions are highlighted.

**2. [Annotation Process and Quality Control]:** We provide a detailed description and illustration of the annotation process and quality control mechanism, including the annotation tool, the annotation protocol, and the verification phases. We also present the workflow of the video-to-image annotation pipeline, which ensures the consistency and accuracy of the annotations across different modalities.

**3. [Copyright Issues of Dataset]:** We clarify that the data source of TAU-106K is from open-source academic benchmarks or public platforms. To avoid copyright and privacy issues, we only provide video URLs and annotation data to the corresponding data sources. 

**4. [Fine-tuning on Other Baselines]:** We extend our experiments to include additional MLLMs fine-tuned on our proposed TAU-106K dataset, to highlight the significance and quality of our dataset.

Besides these common concerns, we have also one-by-one addressed each reviewer's specific comments and suggestions in detail.

According to the reviewers' feedback, we have revised the manuscript and upload the **revision** to the platform, and extended the **supplementary material** to provide more detailed information about the dataset and the methodology. 
We believe that the revised version has significantly improved the quality and clarity of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
After discussion, this submission received 4 positive scores . The major concerns about `comparison with other datasets`, `annotation process and quality control`, `copyright issue` and `fine-tuning on other baselines` were comprehensively solved. After reading the paper, the review comments and the rebuttal, the AC thinks the remaining issue is to include all the revised content to the camera-ready version and correct typographical errors.