When, Where, and What? A Benchmark for Accident Anticipation and Localization with Large Language Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 14:51:50

Abstract: As autonomous driving systems increasingly become part of daily transportation, the ability to accurately anticipate and mitigate potential traffic accidents is paramount. Traditional accident anticipation models primarily utilizing dashcam videos are adept at predicting when an accident may occur but fall short in localizing the incident and identifying involved entities. Addressing this gap, this study introduces a novel framework that integrates Large Language Models (LLMs) to enhance predictive capabilities across multiple dimensions—what, when, and where accidents might occur. We develop an innovative chain-based attention mechanism that dynamically adjusts to prioritize high-risk elements within complex driving scenes. This mechanism is complemented by a three-stage model that processes outputs from smaller models into detailed multimodal inputs for LLMs, thus enabling a more nuanced understanding of traffic dynamics. Empirical validation on the DAD, CCD, and A3D datasets demonstrates superior performance in Average Precision (AP) and Mean Time-To-Accident (mTTA), establishing new benchmarks for accident prediction technology. Our approach not only advances the technological framework for autonomous driving safety but also enhances human-AI interaction, making the predictive insights generated by autonomous systems more intuitive and actionable.

Main paper: 
 related work: 
 As autonomous driving technology is gradually integrated into everyday use, ensuring its safety has become paramount. The ability of deep learning models to automatically detect or even predict accidents in advance could significantly increase confidence in autonomous driving systems. In this context, the concept of accident anticipation task was introduced in 2016 by Chan et al. [6] , which builds on the foundation of accident detection to enable early prediction of accidents.Addressing the complexities of traffic accident recognition, numerous studies [1, 15, 18, 19, 22, 29, 30, 32, 35, 39, 42, 47, 50, 52, 54, 55] have integrated Convolutional Neural Networks (CNNs) with sequence processing networks like Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) cells, Gated Recurrent Units (GRUs), and Graph Convolutional Networks (GCNs). This synergy enables the extraction of intricate motion patterns and temporal features from video data, facilitating the identification of potential accident precursors. Yao et al. [49] and Takimoto et al. [49] exemplify this by merging CNNs with RNNs and GRUs to analyze temporal scene dynamics and predict accidents. Basso et al. [3] introduce a CNN-based architecture for detailed vehicle behavior analysis, while Thakare et al. [41] suggest a convolutional autoencoder for feature extraction with reduced computational load, though it struggles with capturing extensive spatial patterns. Other enhancements include the adoption of attention mechanisms [2, [22] [23] [24] 43] and Transformers like UniFormerv2 [25] , VideoSwin [31] , and MVITv2 [11] , which excel in processing visual data and understanding dynamic traffic interactions through self-attention mechanisms.However, existing frameworks for accident anticipation and object detection often operate independently. While these models can predict accidents in advance, they fall short in identifying the participants involved in these accidents and thus lack the ability to implement appropriate actions in response to them autonomously. To address this gap, this paper extends the accident anticipation task to include the accident localization task, which predicts the occurrence of accidents in videos in advance and accurately identifies the individuals involved in the accidents.In addition, with the rapid development of large-scale language models, more and more autonomous driving models are using multimodal large-scale models for tasks such as voice-guided driving and trajectory prediction. For example, LMDrive [38] , UNIAD [17] , CAVG [26] , and DiLu [48] , and DriveMLM [46] use multimodal sensor data, such as point clouds, combined with natural language instructions to guide vehicle navigation. GPT-Driver [33] turns trajectory planning into a language modeling task and fine-tunes GPT-3.5 accordingly. TrafficGPT [53] integrates ChatGPT with a traffic foundation model and trains on multimodal data inputs to provide comprehensive support for various traffic-related tasks. However, most existing works rely on complex multimodal inputs, which limits the range of usable datasets and complicates the creation of new datasets. In our model, we process outputs from smaller models, such as the probability of accidents occurring and information about the participants in the accidents, and use them as inputs to LLaVa-NEXT [27] , thereby improving the understanding and analysis of traffic accident scenarios by the LLMs. 
problem formulation: 
 This study extends the conventional scope of accident anticipation by incorporating the task of accident localization. Our objective is to devise a model that is capable of: (1) predicting the likelihood of a traffic accident occurring, (2) providing timely accident warnings if an accident is imminent, and (3) localizing the reference objects (traffic agents) involved in the accident. Given a 𝑇 -frames dashcam video, the model is tasked with calculating a probability score 𝑠 𝑡 for each frame 𝑡 ∈ [1,𝑇 ], indicating the potential of an accident at that moment. An accident is predicted to occur at time step 𝑡 if the probability score 𝑠 𝑡 first surpasses a predefined threshold 𝑠 𝜃 . We define the Time-to-Accident (TTA) as Δ𝑡 = 𝜏 -𝑡 𝜃 , where 𝑡 𝜃 is the time step when the score exceeds 𝑠 𝜃 , and 𝜏 represents the actual time step of the accident occurrence.To localize the objects involved in accidents, we approach the task as a mapping problem: the model is required to predict the probability scores 𝑠 1:𝑁 𝑡 for 𝑁 objects in each frame 𝑡, aiming to pinpoint the specific objects within the video that are involved in the accident. An object, denoted as the 𝑖th object, is considered to be involved in an accident if 𝑠 𝑖 𝑡 > 0.5; otherwise, it is not involved. 
proposed model: 
 Our model framework is meticulously crafted to not only anticipate accidents but also to identify objects that could precipitate such incidents, providing timely linguistic warnings for passengers. We frame the proposed model into three stages: Feature Extraction and Fusion, Accident Anticipation and Location, and Verbal Accident Alerts, as shown in Figure 2 .In the first stage, the input dashcam video is first encoded by the MobileNetv2 [37] in the feature extractor, followed by the dual vision attention mechanism, producing a set of vision-aware features 𝑂 • 𝑉 , corresponding to each frame of the video. Concurrently, the raw dashcam video is also fed into the object detector to identify the object vectors 𝑉 𝐵 = {𝑉 1 𝐵 , 𝑉 2 𝐵 , . . . , 𝑉 𝑇 𝐵 } of the reference objectors via the pre-trained detector Cascade R-CNN [4] . Each vector at frame 𝑡, represented as 𝑉 𝑡 𝐵 = {𝐵 𝑡 1 , 𝐵 𝑡 2 , . . . , 𝐵 𝑇 𝑁 }, indicates the bounding boxes of 𝑁 detected objects. Next, these object vectors are refined through feature extractor and a dynamic object attention mechanism to extract precise object-aware features 𝑂 • 𝐵 . Dual Vision Attention. This component is responsible for accepting the vision-aware features 𝑂 • 𝑉 . In contrast to traditional methods such as MaskFormer [8] , DETR [5] , and MDETR [21] , which require extensive token numbers of images for self-attention and incur significant computational overhead, we introduce a dual vision attention mechanism inspired by DANet [12, 13] . As depicted in Figure . 3, it employs a "hindsight fusion" strategy. This strategy judiciously allocates attention to vision features 𝑂 𝑉 extracted by the feature extractor through a two-pronged method: channel attention and position attention. Specifically, the vision features 𝑂 𝑉 is converted to query 𝑄 𝑃 , key 𝐾 𝑃 , and value 𝑉 𝑃 representations via distinct convolutional layers. These representations are then utilized to generate the attention maps in the position attention, which can be represented as follows:EQUATION)where 𝛾 is a trainable coefficient and 𝜙 softmax denotes the softmax activation function. Furthermore, the channel attention mechanism is distinctively designed to bypass the convolutional layer embeddings typically used in position attention, favoring a direct attention approach instead. Formally,EQUATION)where 𝛽 is also a learnable coefficient. The computed channel attention maps 𝑊 𝐶 along with the position attention maps 𝑊 𝑃 are subsequently integrated to form the refined vision-aware features 𝑂 • 𝑉 = 𝐹 𝑃 ⊕ 𝐹 𝐶 . To enhance computational efficiency, we utilize down-sampling and up-sampling in conjunction with the dual vision attention mechanism. This condenses feature dimensions into a more computationally friendly latent space. This approach improves feature representation by addressing both channel and position-specific nuances while minimizing computational demands through strategic dimensional adjustments. Dynamic Object Attention. The dynamic object attention mechanism is innovatively designed to dynamically adjust attention weights across various objects within multi-agent traffic scenes, effectively enabling the model to prioritize high-risk entities. Traditional attention mechanisms typically necessitate updating the attention matrix via gradient descent and backpropagation after processing a batch through the model. These approaches render the Drawing inspiration from the capsule networks [36] , we pioneer a novel chain-based attention strategy, termed dynamic diffuse attention. This mechanism fine-tunes the granularity of the feature matrix across various iterations rather than in a batch-centric manner. As illustrated in Figure . 4, the dynamic diffuse attention begins with the application of a weight matrix 𝑊 to effectuate a dimensional transformation on the object features 𝑉 𝐵 across the 𝑛 𝑡ℎ iteration, 𝑛 ∈ [1, 𝑛], resulting in the enhanced object features, denoted as the F 𝐵 = 𝑊 × 𝑉 𝐵 . Then, the embedding object features are embedded undergoes the following operation:EQUATIONwhere 𝑊(𝑛) 𝐵 is a learnable weight matrix with the same shape as V 𝐵 , and 𝜙 dropout and • represent the application of softmax function and element-wise multiplication, respectively.We also update the object feature representation by integrating dynamic diffuse noise. The embedded object features 𝐻 𝐵 are converted via the squash activation function and then modulated by𝜙 softmax (𝑊 (𝑛)𝐵 ), to which we add a level of diffuse noise D (𝑛) to compute the update Δ𝑊 (𝑛) 𝐵 for the weight matrix 𝑊 (𝑛) 𝐵 :EQUATIONThis equation underpins the correlation between two matrices through element-wise multiplication, which improves the weighting of features with higher correlation. The noise D (𝑛) is intricately designed as a Markov chain 𝑝 (𝐷 (𝑛) |𝐷 (𝑛-1) ), allowing the noise from the previous iteration D (𝑛-1) to inform the noise in the current iteration D (𝑛) , following the principles outlined in [16] . This stochastic approach aims to mitigate overfitting and convergence problems by progressively refining the noise through iterations:EQUATIONwhere 𝜖 denotes random Gaussian noise, introducing a measured degree of unpredictability and variance into the model. In addition, ) , and 𝛼 (𝑛) is obtained as the 𝑛th value in a sequence generated through linear interpolation between 0.1/𝑁 and 20/𝑁 over 𝑁 iterations. Finally, we update the weights using Δ𝑊 𝐵 :𝑊𝛼 (𝑛) = 𝛼 (0) 𝛼 (1) • • • 𝛼 (𝑛(𝑛+1) 𝐵 = Δ𝑊 (𝑛)𝐵 +𝑊 (𝑛) . The output of the 𝑁 th iteration𝑊 𝑁 𝐵 is the object-aware feature 𝑂 • 𝐵 . Notably, down-sampling and upsampling operations are also applied in dynamic diffuse attention to reduce computational costs and further transform the feature dimensions into a latent space.Next, we utilize a tri-layer Multilayer Perceptron (MLP) to adeptly amalgamate the vision-aware 𝑂 • 𝑉 and object-aware 𝑂 • 𝐵 features. This integration facilitates the generation of cross-modal features 𝑂 𝐶 , which serve as the input for the subsequent stage. Formally,EQUATIONwhere the 𝜙 MLP is the MLP, while ∥ denotes matrix concatenation. In the second stage, we seamlessly introduce two novel modules for the task of accident anticipation and location. Accident Anticipation Module. This module is architected to estimate in real-time the probability scores 𝑆 = {𝑆 1 , 𝑆 2 , . . . , 𝑆 𝑇 } for each frame of the input video. This estimation serves to identify the likelihood of an accident occurring, thereby facilitating the earliest possible detection and providing a critical lead time for preventive action. To achieve this, we employ GRUs and MLPs to refine the cross-modal features 𝑂 • 𝐶 synthesized during the first stage of our framework. Subsequently, we implement a series of three convolution-deconvolution operations across varying receptive fields. This approach ensures the assimilation of temporal dependency over diverse scales, culminating in a nuanced and precise prediction of accident probability for any given frame of the video. Accident Localization Module. In this module, we utilize a sophisticated attention mechanism in conjunction with a GRU to compute the probability values of accident occurrence for each detected object. To ensure coherent reasoning, we harmonise the vision-aware 𝑂 • 𝑉 , object-aware 𝑂 • 𝐵 , and cross-modal 𝑂 • 𝐶 features by projecting them onto the same semantic space through linear projection and L2 normalisation. This projection yields the query 𝑄 • , key 𝐾 • and value 𝑉 • representations, formalised as follows:𝑄 • = 𝑊 • 𝑄 𝜙 MLP 𝑂 • 𝑉 , 𝐾 • = 𝑊 • 𝐾 𝜙 MLP 𝑂 • 𝐵 , 𝑉 • = 𝑊 • 𝑉 𝜙 MLP 𝑂 • 𝐶 (7) where 𝑊 • 𝑄 ,𝑊 • 𝐾 ,𝑊 •𝑉 represent learnable matrices tailored for linear projection. The transformed query 𝑄 • , key 𝐾 • , and value 𝑉 • are then fed into the attention block, articulated as follows:EQUATIONwhere 𝑑 𝑘 denotes the dimension of the transformed vectors. The attention-derived matrix 𝐹 𝑐 is further refined by a GRU. This GRU uses scatter and gather operations to efficiently parallelize the acquisition of contextual information alongside the learning of spatiotemporal interdependencies between agents. This innovative approach enhances the model's ability to capture and analyse the complex dynamics present in multi-agent traffic scenes. Subsequently, a softmax function calculates likelihood scores for each detected object. This crucial step allows the identification of the top-𝑘 objects that have the highest association with potential accidents. By prioritizing these objects, our model focuses on the most critical elements within the traffic scenes and significantly enhances its practical utility by providing actionable insights for accident prevention.Recent studies [14, 26] have highlighted the importance of natural language commands in improving passenger experience and acceptance of autonomous vehicles (AVs). Therefore, beyond the critical functions of accident anticipation and localization, our model framework endeavors to enhance human-AI interaction by providing verbal accident alerts to passengers.This stage is intricately designed to deliver precise and timely traffic accident warnings, utilizing the latest Large Language Model (LLM), LLaVa-NEXT [27, 28] . It processes dashcam video footage, coupled with accident localization data and structured prompts [44] as inputs. These prompts encompass exhaustive scene semantic annotations derived from the second stage-such as probability scores and Time-to-Accident (TTA) derived from stage two outputs-to guide the model to fully understand complex semantic scenes.To prepare the input for the Mistral-7B model [20] , we use CLIP [34] and Vision Transformer (ViT) [10] for initial object recognition and image tokenization within the video. This process identifies key entities such as traffic signs, vehicles, and pedestrians, thereby enriching the visual cues available to the model. At the same time, the input prompts are tokenized into sequences using the Bidirectional Transformer (BERT) model's WordPieces tokenizer [9] , and then also integrated into the Mistral-7B model. Finally, the Mistral model synthesizes this multimodal information and generates dialogues that articulate the expected timing of the accident and the specific accident-involved traffic agents. To the best of our knowledge, we are the first to leverage the linguistic prowess of the LLMs to produce accident alert dialogues. This innovation fills a crucial gap in the realm of safe driving and human-machine interaction, marking a significant step forward in the integration of linguistic capabilities into autonomous driving technologies. 
training: 
 Our training loss function consists of three main components: the score loss 𝐿 𝑆 for predicting the probability scores of all frames in dashcam videos, the anticipation loss 𝐿 𝐴 for predicting whether accidents occur in dashcam videos, and the localization loss 𝐿 𝑀 for locating vehicles involved in accidents.The score loss 𝐿 𝑆 is calculated using the ground-truth accident time 𝜏 and the probability scores 𝑠 𝑛,𝑡 at time step 𝑡. Specifically, given the positive videos (i.e., videos with accidents), we set the probability scores 𝑠 𝑝,𝑡 of each frame to be close to 1, while for negative videos (i.e., videos without accidents), we set these scores 𝑠 𝑛,𝑡 to approach 0. To account for the increasing relevance of frames closer to the accident time, we introduce a weighting coefficient 𝑒 -max( 𝜏 -𝑡 𝜆 ,0) that penalizes probability scores closer to the accident time 𝜏, where 𝜆 is a decay factor set to 20. For positive and negative videos, the labels L 𝑝 𝑆 and L 𝑛 𝑆 are set to 1 and 0, respectively, resulting in the following formulation for 𝐿 𝑆 :EQUATIONFurthermore, the anticipation loss 𝐿 𝐴 can be defined as follows:EQUATIONwhere 𝑙 𝑎 is the output of the accident anticipation module, and 𝑉 is the number of dashcam videos. We assign a label L 𝑝 𝐴 = 1, indicating a positive instance. Conversely, for videos devoid of accidents, we denote these as negative instances with a label L 𝑛 𝐴 = 0. In addition, the localization loss 𝐿 𝑀 is specifically designed to instruct the model in discerning whether each detected object within the video plays a role in an accident. For every object 𝑛 ∈ [1, 𝑁 ] that appears in the video, we define labels for objects positively associated with an accident (𝐿 𝑝 𝑀 = 1) and those negatively associated (𝐿 𝑛 𝑀 = 0). Consequently, the localization loss, 𝐿 𝑀 , is formulated as follows:EQUATIONHere, 𝑙 𝑡,𝑛 𝑚 represents the predictive output for the 𝑛 𝑡ℎ object at frame 𝑡, with 𝑇 signifying the total frame count. This loss function enhances the model's capability in accurately determining the involvement of each detected object in potential accident scenarios across all frames, thereby optimizing the accuracy of accident localization.During the first training phase, the final loss function 𝐿 is the sum of score loss 𝐿 𝑆 and anticipation loss 𝐿 𝐴 , i.e., 𝐿 = 𝐿 𝑆 + 𝜂𝐿 𝐴 , where 𝜂 is a constant coefficient. In the second training phase, the loss function 𝐿 consists only of 𝐿 𝑀 . This structured approach allows for a nuanced and effective model training strategy that addresses the complexities of traffic accident detection and localization in dashcam video. 
experiment 6.1 datasets: 
 DAD. The Dashcam Accident Dataset (DAD) [6] compiles a collection of 620 dashcam recordings from six prominent cities in Taiwan, each lasting 5 seconds and captured at a rate of 20 frames per second. From these recordings, 1750 video segments were extracted, including 620 accident segments and 1130 non-crash segments. For the segments with accidents, the collision time was set to the 90th frame. Among the three datasets discussed, the DAD dataset is the only one that includes annotations for object detection bounding boxes, object IDs, object categories, and labels indicating the occurrence of accidents. This unique composition makes the DAD dataset particularly suitable for tasks related to the localization of objects involved in accidents. The segmentation of the dataset for model training and evaluation purposes allocates 70% of the data to the training set, which is further divided into 455 accident and 829 non-accident segments, while the test set contains 165 accident and 301 non-accident segments.CCD. The Car Crash Dataset (CCD) [1] , is an extensive collection of 4500 video recordings annotated with different environmental conditions (day/night, different weather conditions such as snow, rain, or clear sky), the involvement of bicycles and pedestrians, and detailed explanations of the causes of the accidents. Each video, which captures 5 seconds of footage at a playback rate of 10 frames per second, marks accidents in positive cases at the 40th frame. This dataset is strategically divided into training (80%) and test (20%) sets, maintaining a balance of one positive to two negative videos in both segments. A3D. The AnAn Accident Detection (A3D) dataset [49] , contains 1500 dashcam video clips from different East Asian urban environments, representing a range of weather conditions and times of day. These clips are each 5 seconds long, with a frame rate of 20 frames per second achieved through down-sampling. In the dataset, accidents within positive video segments are identified at the 80th frame. The split of the data for development purposes is set at 80% training and 20% testing.In the area of traffic anticipation and localization tasks, three primary evaluation metrics are used: Average Precision (AP), Mean Time-To-Accident (mTTA), and Accident Object Localization Accuracy (AOLA). Average Precision (AP). AP serves as a measure to evaluate the model's ability to accurately detect the occurrence of traffic accidents within videos, especially in scenarios where there is an imbalance between positive and negative samples. In binary classification tasks, assuming that 𝑇 𝑃, 𝐹 𝑃, and 𝐹 𝑁 represent the number of true positives, false positives, and false negatives, respectively, we can calculate the model's recall 𝑅 = 𝑇 𝑃 𝑇 𝑃 +𝐹 𝑁 and precision 𝑃 = 𝑇 𝑃 𝑇 𝑃 +𝐹 𝑃 . Recall indicates the proportion of positive instances that are correctly predicted, while precision reflects the proportion of positive predictions that are actually positive. A precision-recall curve is plotted from these values, and AP is defined as the area enclosed by this curve and the coordinate axes. In practice, the area under the curve is approximated by discrete summation:EQUATIONMean Time-To-Accident (mTTA). mTTA quantifies the ability of the model to predict in advance the occurrence of an accident among the positive samples. If an accident occurs at frame 𝜏, TTA is defined as Δ𝑡 = 𝜏 -𝑡 𝜃 , where 𝑡 𝜃 satisfies 𝑠 𝑡 ≥ 𝑠 𝜃 for 𝑡 ≥ 𝑡 𝜃 and 𝑠 𝑡 < 𝑠 𝜃 for 𝑡 < 𝑡 𝜃 , where 𝑠 𝜃 represents the threshold for the accident probability score. Across all possible thresholds 𝑠 𝜃 ∈ [0, 1], mTTA is the average of all TTAs, i.e., 𝑚𝑇𝑇 𝐴 = 1 𝑛 𝑠 𝜃 𝑇𝑇 𝐴. Accident Object Localization Accuracy (AOLA). AOLA assesses the accuracy of the model in predicting the occurrence of accidents for all detected objects. For a total of 𝑁 𝑉 videos, each containing 𝑓 frames and 𝑁 𝑂 objects per frame, AOLA is defined as follows: where 𝑛 𝑜 denotes the number of correctly predicted objects per frame.𝐴𝑂𝐿𝐴 = 𝑁 𝑉 𝑖=1 𝑓 𝑗=1 𝑛 𝑜 𝑁 𝑉 𝑖=1 𝑓 𝑗=1 𝑁 𝑂 (13)In this study, Pytorch is used for the implementation, and training and testing are performed on an A40 48G GPU. For the pre-trained model, we use MobileNetv2, from which 1280 feature dimensions are extracted. For the model hyperparameters, we set the number of dynamic routing iterations within the Dynamic Object Attention mechanism to 8, with a maximum of 19 objects detected per frame.For the loss function parameters, we set a decay coefficient 𝜆 = 20 and a loss function ratio coefficient 𝜂 = 10. For the training parameters, we set the model learning rate to 1 × 10 -4 , with a batch size of 16. We use ReduceLROnPlateau as the learning rate scheduler to ensure that each model is trained for at least 10 epochs. See Appendix for more implementation details.We conduct extensive experiments on the DAD, CCD, and A3D datasets. Our model demonstrates superior performance in both AP and mTTA metrics, as detailed in Table 1 . Notably, on the DAD dataset, our model achieved a remarkable 14.6% improvement in AP and a 16.4% increase in mTTA compared to the second-performing model. While enhancements on the CCD and A3D datasets were more modest, this can be attributed to the already near-optimal performance of competing models on these datasets. Additionally, as indicated in Table 2 , our model secured the top scores across both AP and mTTA metrics. Our analysis revealed that, unlike competing models which faced challenges in optimizing the trade-off between AP and mTTA, our model adeptly maintains this balance throughout the training process. Table 5 illustrates that while other models, such as DSTA, peaked in AP at the 20th epoch before experiencing a rapid decline, our model reaches peak performance by the 2nd epoch and maintains a minimal decline in performance thereafter, highlighting its rapid convergence and resilience to overfitting. Furthermore, our model undergoes rigorous multi-class accuracy (AOLA) testing on the DAD dataset, achieving an accuracy rate of nearly 90%. This test involves classifying each video frame into one of 19 possible object categories, demonstrating the model's accuracy in recognizing and classifying a wide range of objects in complex traffic scenes. Achieving such a high accuracy rate, especially in a multi-class setting, underscores the effectiveness and adaptability of our model and sets a new benchmark in accident anticipation and localization for autonomous driving systems.Ablation Studies of Different Components. Table 3 presents our ablation study for four key components: dual vision attention, dynamic object attention, accident anticipation module, and accident localisation module, highlighting their indispensability within our model framework. Model A, lacking dual vision attention, shows decreases in AP, mTTA and AOLA metrics, highlighting the importance of incorporating learnable attention weights in global image processing. Model B, devoid of dynamic object attention, experiences a significant decrease in all three metrics due to the absence of key object features, further highlighting the importance of computing fine-grained correlations between detected objects to focus the model on accident-relevant traffic agents for more accurate anticipation. Furthermore, Model C, which omits the output of probability scores and focuses solely on binary accident prediction, maintains its AP score but experiences reduced performance in mTTA. Finally, Model D, which excludes the accident localization module, results in a significant decrease in the AOLA metric and a decrease in both AP and mTTA scores. This indicates that the prediction of accident-involved traffic agents not only improves the model's accuracy (AP), but also its timeliness (mTTA). In summary, the results of these ablation studies confirm the effectiveness of each model component. Together, these components synergistically perform the tasks of accident anticipation and localisation with improved accuracy and timeliness. See Appendix for more details on ablation studies. Ablation Studies of Dynamic Object Attention. This study introduces the dynamic object attention mechanism that leverages noise generated by a Markov chain of diffusion model. Through multiple iterations, this mechanism progressively learns the correlations between different detected entities and iteratively updates their feature representations. To validate the importance of multilayer iterations and the efficacy of incorporating diffusion noise, we conduct a series of ablation experiments. As illustrated in Table 4 , Experiments 1-5 demonstrate that the model achieves optimal Average Precision (AP) and ALOA metrics when the number of iterations, Num-iteration, is set to 6. An increase or decrease in the number of iterations respectively leads to overfitting or underfitting. Furthermore, the duration of the process does not significantly increase with additional iterations, making Num-iteration=6 the optimal choice. Experiments 6-11 investigate the impact of varying the number of test iterations while maintaining the same number Figure 6 shows the temporal variation of the output probability scores indicating the likelihood of an accident. As shown in Figure 6 (a), our model successfully identifies vehicles involved in accidents (indicated by red bounding boxes) and outputs probability scores close to 1 after the accident. Prior to the accident, the model's predicted probability scores exceed the threshold early, suggesting that the model can detect changes in the target agents' behavior in the image and infer the increasing likelihood of an accident under continuing conditions, thus assigning higher probability scores in anticipation. Conversely, as shown in Figure 6 (b), the model's predictions do not exceed the threshold, indicating that no accident has occurred in the video.Table 5 : Ablation studies of the dynamic object attention on noise. "None" indicates no noise is applied, while "Same" indicates using identical Gaussian noise for each iteration loop. "Different" indicates using different Gaussian noise for different iteration loops. "Linear" denotes using a simple linear relationship between the Gaussian noise across loops. "Markov chain" describes the method used in this study.  
conclusion: 
 In this study, we extend accident anticipation to accident localization by using LLMs for detailed scene analysis, enabling precise accident warnings about what, when, and where of potential incidents, thus significantly improving driving safety. We also present a novel three-stage model tailored to the task of traffic anticipation and localization. It introduces a novel attention mechanism that dynamically refines feature representations, prioritizing high-risk objects in traffic scenes. Moreover, we are the first to apply the LLMs to generate verbal accident alerts in accident anticipation, significantly enhancing human-AI interaction. Our proposed model showcases superior performance on key metrics in real-world datasets such as DAD, CCD, and A3D, setting a new benchmark in this field.  
: 
 complex driving scenarios, where not only the timing but also the location and nature of potential incidents are critical.Traditional models often treat visual perception and decisionmaking as separate entities, limiting the use of rich sensory data for proactive driving adjustments. Furthermore, these models typically do not account for the dynamic nature of driving environments, failing to adapt to real-time changes and the complex interactions between various traffic participants. This static approach limits their effectiveness in the unpredictable and varied conditions typical of real-world driving. Moreover, the outputs from these models are often not translated into clear, actionable insights, reducing their practical applicability and hindering their potential to enhance safety in autonomous driving technologies.To address these gaps, our research introduces a comprehensive framework that leverages Large Language Models (LLMs) to enhance the predictive capabilities of autonomous driving systems. By integrating cutting-edge linguistic and cognitive technologies, our approach not only predicts potential incidents more accurately but also improves the interaction between human operators and AIdriven systems, providing a richer, more intuitive user experience.Our key contributions are:1) We have expanded the traditional scope of Accident Anticipation (What and When) to include the localization of objects involved in potential accidents (Where), a task we refer to as Accident Localization. For the first time, we utilize LLMs to analyze complex scene semantics, offering precise and timely accident alerts to passengers. Our system predicts whether an accident will occur (What), when it might happen (When), and where it would occur (Where), thereby filling a crucial gap in accident prevention and enhancing the safety of autonomous driving.2) We introduce a novel chain-based attention mechanism that iteratively refines feature representations through a dynamic routing mechanism enhanced by Markov chain noise models. This process allows our system to dynamically adjust attention weights across various objects within multi-agent traffic scenes, prioritizing those with higher risk levels. This attention mechanism is part of a three-stage model that preprocesses outputs from smaller models to generate multimodal inputs (image and text) for large models, guiding these large models to provide more accurate and detailed scene descriptions.3) Our model has undergone rigorous testing on benchmark datasets such as DAD, CCD, and A3D, where it has demonstrated superior performance in key metrics like Average Precision (AP) and Mean Time-To-Accident (mTTA). The results not only surpass existing methodologies but also mark a significant advancement in accident prediction technology, setting new standards for the field. 
 
 Appendix:  


Decision: The paper received four positive reviews out of four. I am therefore recommending to accept the paper.

Reviews: review: paper_summary: paper_summary: This paper addresses the critical need for enhanced accident anticipation in autonomous driving systems. Traditional models predict the timing of accidents but often fail to localize incidents and identify involved entities. To overcome these limitations, the authors propose a novel framework that integrates Large Language Models (LLMs) to improve the predictive capabilities regarding what, when, and where accidents might occur. The framework demonstrates superior performance on benchmark datasets (DAD, CCD, and A3D) in terms of Average Precision (AP) and Mean Time-To-Accident (mTTA), setting new standards in accident prediction technology.
strength_weakness: strength_weakness: 1. The integration of LLMs for accident prediction and localization represents a holistic approach to enhancing autonomous driving safety, addressing both temporal and spatial aspects of accident anticipation.
2. The chain-based attention mechanism and the dynamic object attention mechanism show significant advancements in prioritizing high-risk elements and refining feature representations.
3.  Rigorous testing on multiple benchmark datasets demonstrates the effectiveness and robustness of the proposed framework, establishing new performance benchmarks.
limitations: limitations: 1. The figures in your paper are a bit blurry, such as Figure 3 and Figure 4. Please consider replacing them with clearer ones.
2. The methods section does not provide enough detail on key aspects. For instance, more detailed information about the prompt generation after stage 2  would be beneficial.
3. Certain critical terms such as "channel attention" and "position attention" are not adequately defined, which can hinder the reader's understanding of the paper's contributions and methodologies. And the motivation of this **Dual Vision Attention** module is not clear. The authors need to highlight its necessity. Besides, the visualization of the proposed attention mechanism would be beneficial.
4. The definition of the mathematical symbols that appear in the article is not clear, such as the shape of matrix $O_{V}^o$, $O_{B}^o$ , which makes it difficult for the reader to understand the details of each module.
paper_summary: paper_summary: This paper focuses on predicting accidents in vehicle driving. It proposes novel attention designs, an accident localization module and uses LLM  for human text generation. First, videos are sent to the encoder for feature extraction. At the same time, another parallel branch is used for object detection. Then, video features and object instance features are concatenated and sent to the accident anticipation/localization modules. The output prompts are sent to the LLM. Based on the accident analysis prompts and object detection results, the MLLM can output texts for accident analysis results.
strength_weakness: strength_weakness: This paper is clear and easy to follow. The motivation is clear and convincing, the methodology is novel with sufficient details, and the experiments are solid. The application of LLM here is reasonable and can be seen as a good application of MLLM. The reviewer is satisfied with this paper.
limitations: limitations: I am not an expert in this specific topic, so I do not have too many concerns about this paper. But there are still some minor issues:

1. The authors can provide the efficiency comparison of methods. Considering this is for vehicle driving purpose, an online system is preferred.

2. The authors can cite more papers on MLLM for autonomous driving in the related work section.
[1] Xu, Zhenhua, et al. "Drivegpt4: Interpretable end-to-end autonomous driving via large language model." arXiv preprint arXiv:2310.01412 (2023).
[2] Ma, Yingzi, et al. "Dolphins: Multimodal language model for driving." arXiv preprint arXiv:2312.00438 (2023).
paper_summary: paper_summary: This work tackles the research problem of Accident Anticipation and Localization in driving scenes. Unlike previous works focusing on accident anticipation (When and What), this work expands the task with the localization of potential accidents (Where). Precisely, the proposed method performs respectively image feature extraction, object detection, accident anticipation, and accident localization. The outputs of them are used to form the prompt that is used to query a finetuned Large Language Model. The LLM summarizes and decides if there's a potential accident and where it could happen.
The proposed method achieves state-of-the-art performance in several benchmarks such as DAD, CCD and A3D, more importantly it extends the accident anticipation task to accident localization.
strength_weakness: strength_weakness: - The extension of accident anticipation to more fine-grained accident anticipation and localization 
- Leveraging state-of-the-art LLM (Mistral-7B) and VLM (CLIP) to provide more precise and interactive accident anticipation for driving.
- The careful design and detailed ablation validate the effectiveness of the proposed method
- The state-of-the-art performance in several benchmarks such as DAD, CCD and A3D
limitations: limitations: - The use of language model is not the first time in driving scenarios, works such as [1, 2]. A discussion with similar applications of LLMs could be provided.
- I have concerns about the inference speed of the proposed complex pipeline since it is critical for anticipating accidents on time.
-  The proposed pipeline uses an off-the-shelf object detector to locate the objects and the rest of the modules seem dependent on the object detection quality. A discussion on the possible miss detection of objects and its impact on the LLM decision is interesting to have. 
[1] Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
[2] DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Model
paper_summary: paper_summary: This work proposes an approach for accident anticipation in autonomous driving systems. It introduces using LLMs to enhance the model's capabilities in predicting when, where, and when the accident would occur. The proposed approach anticipates accidents and also improves the interaction between the driver and the driving system by providing a better user experience. The proposed approach consists of a novel chain-based attention mechanism to refine feature representations, which enables the model to focus on different objects that could be involved in the accident. The model consists of three stages: Feature extraction and Fusion, Accident Anticipation and Localization, and Verbal alerts to the driver.  For evaluation of the model, datasets DAD, CCD, and A3D are used and the results show that the proposed model has improved over the state-of-the-art on the metrics Average Precision (AP) and Mean Time-To-Accident (mTTA).
strength_weakness: strength_weakness: --> The paper is well-written and well-organized which makes it easy to read.
--> Experimental evaluation on multiple datasets with improvements over the current SOTA is a definite positive of the paper.
--> Also ablation experiments show the contribution of the individual components and help understand the design decisions. 
--> Figures showing the architecture of the different modules help in an easy understanding of the proposed approach.
limitations: limitations: --> Fig. 1 caption, should it not be "yellow and red.." instead of "red and yellow.."?
--> It is not clear how the outputs from the Accident Anticipation Module and Accident Localization Module are converted to text prompts that could be passed as input to the LLava-Next text model (Mistral-7B)
--> My understanding is that here the use of LLava-Next is only to generate a textual response (dialogue) so that it can be relayed to the driver via some text-to-speech model. It seems to be an overkill to use a heavy LLava-Next model for this task. Also, there is no evaluation of the dialogues generated by the model. 
--> The results shown in the paper only evaluate the accident anticipation but not the localization output. How is localization contributing to improving the anticipation results? 
--> Results in Table 1 on CCD for the model DSTA are wrongly reported. Results from the DSTA paper [1], show a score of 4.87 and not 3.87.