CRASH: Crash Recognition and Anticipation System Harnessing with Context-Aware and Temporal Focus Attentions

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 04:31:20

Abstract: Accurately and promptly predicting accidents among surrounding traffic agents from camera footage is crucial for the safety of autonomous vehicles (AVs). This task presents substantial challenges stemming from the unpredictable nature of traffic accidents, their long-tail distribution, the intricacies of traffic scene dynamics, and the inherently constrained field of vision of onboard cameras. To address these challenges, this study introduces a novel accident anticipation framework for AVs, termed CRASH. It seamlessly integrates five components: object detector, feature extractor, object-aware module, context-aware module, and multi-layer fusion. Specifically, we develop the object-aware module to prioritize high-risk objects in complex and ambiguous environments by calculating the spatial-temporal relationships between traffic agents. In parallel, the context-aware is also devised to extend global visual information from the temporal to the frequency domain using the Fast Fourier Transform (FFT) and capture fine-grained visual features of potential objects and broader context cues within traffic scenes. To capture a wider range of visual cues, we further propose a multi-layer fusion that dynamically computes the temporal dependencies between different scenes and iteratively updates the correlations between different visual features for accurate and timely accident prediction.
Evaluated on real-world datasets—Dashcam Accident Dataset (DAD), Car Crash Dataset (CCD), and AnAn Accident Detection (A3D) datasets—our model surpasses existing top baselines in critical evaluation metrics like Average Precision (AP) and mean Time-To-Accident (mTTA). Importantly, its robustness and adaptability are particularly evident in challenging driving scenarios with missing or limited training data, demonstrating significant potential for application in real-world autonomous driving systems.

Main paper: 
 introduction: 
 The introduction of Advanced Driver Assistance Systems (ADAS) and Autonomous Vehicles (AVs) marks a significant leap forward in our quest for safer roads [1, 13, 19] . By aiming to predict and prevent traffic accidents before they happen, these technologies are at the forefront of transforming our transportation landscape. This capability is crucial, enabling vehicles to make decisions that avoid collisions and protect passengers [6] .Despite the progress we have made, the road to reliable accident anticipation is filled with hurdles. Traffic, by nature, is chaotic and full of surprises. From a sudden stop in the flow to a pedestrian stepping out unexpectedly, the variables are endless. This complexity is compounded when you consider the diversity of how accidents can occur, the subtle yet vital visual cues that can get lost among everyday traffic elements, and the unpredictable behavior of other road users. The current solutions are inadequate in several ways when they come to addressing these issues:Firstly, existing methods are predominantly object-centric, relying on the detection of traffic agents within bounding boxes. They often overlook crucial environmental elements-such as lane markings, pedestrian paths, and traffic signs-that are not captured by rigid bounding box constraints, thereby failing to leverage a broader spectrum of visual information and contextual cues.Secondly, there exists a propensity within numerous models to accord equal significance to all detectable entities within a traffic scene, an approach that might neglect the layered semantic interrelations that subsist among different entities. Such an approach risks overlooking essential insights that could significantly enhance the precision of accident anticipation Thirdly, observational constraints intrinsic to real-world driving scenarios introduce substantial impediments. These encompass limitations of sensory apparatuses and environmental contingencies such as obstructions, adverse meteorological conditions, or traffic congestion. The majority of prevailing models, calibrated and assessed under conditions of optimal observational integrity, exhibit pronounced performance diminution when confronted with data-deficient scenarios. This discordance underscores a critical lacuna within contemporary research, necessitating models that manifest robust performance under suboptimal observational conditions.In response to these articulated challenges, the present study introduces "CRASH, " an avant-garde accident anticipation framework that meticulously integrates global contextual information with profound spatio-temporal interactions. This initiative is spearheaded by the introduction of a novel Object Focus Attention (OFA) mechanism within the object-aware module, which adeptly refines and accentuates key local features, extrapolating the essential spatialtemporal dynamics pivotal for accident prediction. Moreover, we pioneer a context-aware module that harnesses Fast Fourier Transform (FFT) along with our innovatively devised Context-aware Attention Blocks (CAB). This ensemble endeavors to distill nuanced global visual information, thereby amplifying the model's contextual comprehension and broadening the ambit of visual cues amenable for predictive analysis. To sum up, our contributions are threefold:(1) We present a novel context-aware module that extends global interactions into the frequency domain using FFT and introduces context-aware attention blocks to compute fine-grained correlations between nuanced spatial and appearance changes in different objections. Enhanced by the proposed multi-layer fusion, this framework dynamically prioritizes risks in various regions, enriching visual cues for accident anticipation.(2) To realistically simulate the variability and randomness of missing data that is commonly encountered in real-world driving, we augment the renowned DAA, A3D, and CCD datasets with scenarios featuring missing data. This innovation expands the research scope for accident detection models and provides comprehensive benchmarks for evaluating model performance.(3) In benchmark tests conducted on the enhanced DAD [5] , A3D [39] , and CCD [2] datasets, CRASH demonstrates superior performance over state-of-the-art (SOTA) baselines across key metrics, such as Average Precision (AP) and mean Time-To-Accident (mTTA). This showcases its remarkable accuracy and applicability across a variety of challenging scenarios, including those with 10%-50% data-missing and limited 50%-75% training set scenes. 
related work: 
 The task of predicting traffic accidents requires models capable of making timely and accurate predictions based on dashboard video before accidents occur. This task is made complex by the inherent variability of traffic scenes and the unpredictable movements of road users.The surge in deep learning applications within computer vision has catalyzed the exploration of advanced models for accident anticipation. To tackle these challenges, recent studies have leveraged various deep learning approaches, including Convolutional Neural Networks (CNNs) [5, 10, 15, 21, 25] , sequential networks [11, 31, 36, 37, 40? , 41] like Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) units, and Gated Recurrent Units (GRUs) to distill essential visual features from traffic scenes. Moreover, Graph Neural Networks (GNNs) [18, 23, 33, 34, 39] and transformer-based models [12, 35] have been investigated for their potential to capture the complex spatial and temporal dynamics in traffic scenes. In addition, generative models [3, 38] such as Generative Adversarial Networks (GANs), Variational Auto Encoders (VAEs), and Diffusion models are also employed in this field. For instance, Corcoran et al. [7] presented a dynamic-attention recurrent CNN to analyze both spatial and temporal features in traffic scenes. Similarly, Bao et al. [2] utilized an uncertainty-aware graph to model spatial relationships and predict traffic accidents, while Liu et al. [22] focused on pedestrian intent prediction through spatio-temporal analysis.As research in traffic accident prediction deepens, a prominent challenge emerges: how to effectively manage and interpret the vast amount of information processed by models when dealing with complex traffic scenes. In this context, the incorporation of attention mechanisms [17, 18] marks a significant advance in the field, enhancing the ability of models to process complex interactions and maintain temporal coherence, thereby improving prediction accuracy and interpretability. In particular, the efforts of Karim et al. [17] and Song et al. [29] have integrated spatial and temporal attention to prioritise relevant segments and regions in driving scenes. Thakare et al. [32] proposed a convolutional autoencoder approach for efficient feature extraction and classification, addressing computational efficiency. Additionally, the interpretability of models has gained prominence in research. Monjuru et al. [26] introduced an explainable artificial intelligence (XAI) strategy, embedding the Grad-CAM (Gradient-weighted Class Activation Mapping) attention mechanism within the GRUs to produce semantic feature maps. Despite these advances, most studies focus on interactions between dynamic objects, overlooking crucial scene elements such as traffic lights, pedestrian crossings, and sidewalks. Furthermore, they typically rely on surface-level visual features that are close to the accidents, failing to adequately capture potential accident precursors in global scenes. Our work aims to fill this gap by integrating key scene elements and multi-layered features into our proposed model, thereby enriching the visual scope of accident detection. This integration allows for the capture of a wider range of semantic information, significantly improving the model's ability to anticipate traffic accidents with improved accuracy and timeliness. 
methodology 3.1 problem formulation: 
 The primary objective of this study is twofold: (1) to predict the probability of a traffic accident occurring, and (2) if an accident does occur, to predict it as early as possible. Similar to the previous work [2] , taking the 𝑇 frames of the dashboard video stream 𝑉 = {𝑉 1 , 𝑉 2 , . . . , 𝑉 𝑇 } as input, the goal is to estimate the probability 𝑃 = {𝑝 1 , 𝑝 2 , . . . , 𝑝 𝑇 } of an accident in each frame. If an accident occurs at time 𝑡 ∈ [1,𝑇 ], we define the Time-to-Accident (TTA) as Δ𝑡 = 𝜏 -𝑡 𝑜 , where 𝜏 is the ground-truth accident time, and 𝑡 𝑜 is the earliest frame in which the probability score 𝑝 𝑡 exceeds a predetermined threshold 𝑝 𝑜 . Consequently, a video is classified as containing an accident (positive) if 𝑝 𝑡 ≥ 𝑝 𝑜 for any 𝑡 ≥ 𝑡 𝑜 and as not containing an accident (negative) if 𝜏 = 0. Our proposed model aims to enhance the precision of accident detection and maximise the TTA, enabling the earliest possible anticipation of accidents.The overall pipeline of CRASH is shown in Fig. 1 . It consists of five critical components: object detector, feature extractor, object-aware module, context-aware module, and multi-layer fusion. Initially, the object detector and feature extractor produce the object 𝐹 𝑜 and context 𝐹 𝑐 vectors for the raw input videos 𝑉 . Next, the objectaware module is used to progressively update the spatial-temporal representation of the object vectors, producing the object-aware vectors F𝑜 . In parallel, the context vectors 𝐹 𝑐 are fed into the contextaware modules for global semantic feature extraction, resulting in the context-aware vecotors F𝑐 . Finally, the multi-layer fusion iteratively fuses and mulls over the output from the feature extractor and these modules to identify and predict potential incidents that could lead to accidents, generating the probability 𝑃 for each frame of the input videos.Object Detector. Given 𝑇 -frames dashboard video, a Cascade R-CNN [4] is employed to detect the top-𝑛 dynamic objects with the highest recognition scores within the video stream, such as vehicles, motorcycles, and pedestrians. Then, we utilize the VGG-16 [28] to embed these selected 𝑛 objects into 2D object vectors 𝐹 𝑜 ∈ R 𝑛×𝑑 , where 𝑑 is the embedding dimension.Feature Extractor. The feature extractor is primarily responsible for extracting the semantic feature from the whole video 𝑉 . The VGG-16 and Multilayer Perception (MLP) are used in this extractor to generate the context vectors 𝐹 𝑐 ∈ R 𝑑 .Object-aware Module. Accidents usually occur due to specific interactions between dynamic traffic agents, which are marked by decreasing spatial distance or irregular trajectories. Therefore, it is necessary to analyze each object's position and past movements, integrating data across both time and space. In this module, we leverage the object vectors 𝐹 𝑜 and the weighted dual-layer hidden states H𝑡-1 = { H 1 𝑡 -1 , H 2 𝑡 -1 } ∈ R 𝑛×𝑑 encodedby the two-layer GRU and our proposed TFA attention mechanism in the multi-layer fusion to focus on the traffic agents most likely to cause accidents.Specifically, we propose a query-centric Object Focus Attention (OFA) mechanism that maps dual-layer hidden states HEQUATION𝑡 at time step 𝑡 -1, each assigned unique linear projection weights. This process facilitates the calculation of spatial-temporal relationships between object vectors 𝐹 𝑜 and their associated contextual and semantic features within the hidden states 𝐻 𝑡 -1 . At time step 𝑡, this process can be defined as follows:           𝑄 1 𝑡 = 𝑊 1 𝑄 (𝜙 MLP ( H 1 𝑡 -1 )) 𝑄 2 𝑡 = 𝑊 2 𝑄 (𝜙 MLP ( H 2 𝑡 -1 )) 𝐾 𝑡 = 𝑊 𝐾 (𝜙 MLP (𝐹 𝑜 )) 𝑉 𝑡 = 𝑊 𝑉 (𝜙 MLP (𝐹 𝑜 )) (1)where𝑊 1 𝑄 ∈ R 𝑑 ×𝑑 , 𝑊 2 𝑄 ∈ R 𝑑 ×𝑑 , 𝑊 𝐾 ∈ R 𝑑 ×𝑑, and 𝑊 𝑉 ∈ R 𝑑 ×𝑑 are all learnable weights. 𝜙 MLP denotes the MLP. Furthermore, We make matrix product on 𝐾 𝑡 and 𝑉 𝑡 and use the generated similarity query vectors 𝑄 1 𝑡 , 𝑄 2 𝑡 to weight the object vectors 𝐹 𝑜 , producing the enhanced object-aware vectors F𝑜 . Mathematically,EQUATIONwhere 𝑊 𝛼 and 𝑊 𝛽 are both the linear projection weights for the query vectors. Moreover, 𝜙 Softmax represents the Softmax activation function, and 𝑑 𝑘 is the projection channel dimension.Context-aware Module. In addition to establishing spatiotemporal relationships between dynamic objects, modelling the visual context of detected objects -such as lane markings, sidewalks, and traffic signs -is crucial for distinguishing potential accident causes from others. To the end, we present a context-aware module that uses global context vectors to capture a wider range of visual cues and contextual features. This module not only identifies focal points within the input videos but also recognizes broader contextual relationships within the entire visual scene, going beyond the limitations of bounding boxes.Departing from traditional methods that emphasize temporal features, this module focuses on spectral features. Inspired by the spectral and hierarchical transformers [14, 27] , we first use the 1D convolutional layer to expand the number of channels to 𝑐 for the context vectors 𝐹 𝑐 . Thereafter, FFT is applied to transform context vectors into the Fourier domain, transitioning from temporal to spectral space. We then employ a parametrically learnable Spectral Gating Unit (SGU) alongside innovative context-aware attention blocks. This structure assigns weights to each frequency, enhancing the detection of subtle edge and contour details within the global visual scene. Formally,EQUATIONHere, 𝜙 FFT represents the Fast Fourier Transform (FFT) function, while 𝜙 conv1D denotes a one-dimensional convolutional layer. Furthermore, 𝑊 𝑓 ∈ R 𝑐 ×𝑤×ℎ is a learnable weight matrix produced by the SGU. The spectral features, denoted as 𝑆 𝑐 ∈ R 𝑐 ×ℎ×𝑤 , correspond to the context vectors 𝐹 𝑐 , where 𝑐 is the number of channels and ℎ and 𝑤 are the height and width of the feature map, respectively. Importantly, since the spectral features 𝑆 𝑐 are complex numbers rather than real numbers, they cannot be directly subjected to gradient calculation and backpropagation. To address this, the Context-aware Attention Block and Inverse Fast Fourier Transform (IFFT) are introduced to further enhance the spectral features and transform the spectral space back to physical space. Furthermore, the features are processed by a MLP to eventually generate the vision-conditioned context-aware vectors F𝑐 , which improve the training stability of our model. It can be represented as follows:EQUATION) where 𝜙 IFFT denotes the IFFT function, while ⊕ and ⊙ signify the concatenation operation and element-wise multiplication, respectively. Correspondingly, 𝜙 AvgPool and 𝜙 MaxPool are the average and maximum pooling layers, respectively. In addition, the contextaware vectors F𝑣 ∈ R 𝑑 , with embedding dimension 𝑑 = ℎ × 𝑤.Multi-layer Fusion. Accidents can occur unexpectedly at any moment in traffic scenes, and typically occupy a small proportion of the entire video stream, exhibiting a long-tail distribution. Most existing methods tend to focus on anomalies within key frames of the video stream. They directly feed the top-layer frame features into a linear layer to predict the probability of an accident. However, certain frames that are close to anomalous moments, despite lacking direct abnormal phenomena, often contain enriched contextual information that is crucial for assessing the likelihood of an accident. To fully exploit the semantic and contextual features embedded in every frame of the input video, we introduce the Temporal Focus Attention (TFA) mechanism. Comprising 𝑘 attention layers, each with two attention blocks as depicted in Fig. 1 , this mechanism systematically integrates representations from diverse frame before preceding the prediction of accident probabilities. The core idea is to expand the model's recognition scope and reference range to all visual information in the video stream, focusing dynamically on the embedded features at each moment to improve the model's ability to identify key frames in input video.From a technical perspective, the context 𝐹 𝑐 , object-aware F𝑜 , and context-aware F𝑐 vectors are fused and encoded by a dual-layer GRU, which can be expressed as follows:EQUATIONwhere ∥ signifies the vector concatenation, while ℎ 𝑖 𝑡 represents the hidden state of the 𝑖-th layer GRU at time step 𝑡, and 𝑂 𝑖 𝑡 is the GRU's final output for the 𝑡-th frame video, subsequently input into a MLP to generate the accident probability score 𝑝 𝑡 .In the TFA layer, each block inputs the hidden states produced by the dual-layer GRU from the past 𝑀 frames, denoted as 𝐻 𝑡 = {𝐻 1 𝑡 , 𝐻where W 𝑖 𝑄 , W 𝑖 𝐾 , W 𝑖 𝑉 ∈ R 𝑑 ×𝑑 are learnable matrices for the linear projection. The 𝑗-th attention head head 𝑖 𝑗 and the output 𝑅 𝑖 𝑦 from 𝑦-th RTA layer's attention block is computed as:EQUATION)where 𝑚 is the total number of the attention head. Inspired by ResNet [16] , the TFA block integrates Gated Linear Units (GLUs) [8] for optimizing the output. This ensures effective backpropagation of larger gradients to the initial layers, facilitating these layers to learn as rapidly as the top layer. Mathematically,EQUATION)where R𝑖 𝑦 ∈ R 𝑚×𝑑 is the enhanced output of 𝑦-the TFA layer for 𝑖-th attention block, and 𝜙 GLUs is the GLUs function.Finally, the outputs of 𝑖-th attention blocks R𝑖 1 , R𝑖 2 , . . . , R𝑖 𝑘 across 𝑘 attention layers are dynamically aggregated using distinct learnable weights to obtain the final hidden state H𝑡 . This process is formalized as follows:EQUATION)where 𝛾 𝑖 1 , 𝛾 𝑖 2 , and 𝛾 𝑖 𝑘 , 𝑖 ∈ [1, 2] are the learnable parameters. The final hidden states of the TFA layer can be defined mathematically as 𝐻 𝑡 = 𝜙 AvgPool ( R1 weighted ) ⊕ 𝜙 AvgPool ( R2 weighted ), which are fed into the Object Focus Attention (OFA) in the object-aware module for feature fusion.We incorporate a multi-task learning paradigm into our training loss, which can be bifurcated into two components: (1) anticipation loss L 𝑎 and (2) enhancement loss L 𝑒 .The anticipation loss L 𝑎 is computed based on the discrepancy between the model-predicted accident probabilities 𝑝 𝑡 at time step 𝑡 and the ground-truth accident timing 𝜏. To better align with the task of real-world traffic accident anticipation, we refine the traditional cross-entropy loss function in anticipation loss by integrating a penalty term 𝑒 - 1 2 max( 𝜏 -𝑡 𝑓 ,0) into the positive loss component. This adjustment applies increasing loss values to video stream that are closer to the moment of an accident, encouraging the model to predict accidents earlier. The anticipation loss L 𝑎 is expressed as follows:EQUATION)where 𝐵 is the batch size, 𝑙 𝑣 represents the binary label of accident occurrence within each video (1 for an accident, 0 for none), while 𝑇 is the total number of frames per video, and 𝑓 is the frames per second (fps) of the video. Furthermore, we introduce an innovative enhancement loss, L 𝑒 , to mitigate the significant error accumulation in the initial stages of the GRU within the multi-layer fusion. Specifically, position encoding is integrated into all hidden states produced by the secondlayer GRU, and a classical multi-head self-attention mechanism is employed to extract relevant semantic information from these hidden states, resulting in the hidden state maps 𝑝 𝑒 : where 𝜙 𝑀𝐻𝐴 and 𝜙 𝑃𝐸 denote the multi-head attention mechanism and position encoding mechanism, respectively. Next, we compute the enhancement loss L 𝑒 using the hidden state maps 𝑝 𝑒 and the ground-truth accident timing 𝜏. Formally,EQUATIONEQUATIONEventually, the final loss is then calculated as the sum of anticipation loss L 𝑎 and enhancement loss L 𝑒 , adjusted for homoscedastic uncertainty through Gaussian probability:EQUATIONwhere 𝜇 1 and 𝜇 2 are manually-set hyperparameters, while 𝜌 1 and 𝜌 2 represent uncertainty coefficients, initially set to 1. Overall, the multi-task training loss is meticulously designed to provide a dynamic balance between anticipating accidents and enhancing model sensitivity to critical features, thus taking into account more accident-related factors. 
experiment 4.1 experiment setup: 
 We evaluate the efficacy of our model using three esteemed datasets: Dashcam Accident Dataset (DAD), Car Crash Dataset (CCD), and AnAn Accident Detection (A3D) datasets. These datasets, referred to complete datasets, provide a unique perspective on traffic accidents in various scenes. More details can be found in the Appendix.Recognizing a gap in research concerning data omissions in this field, the experiment setup is intentionally designed to simulate the variability and randomness of missing data encountered in real-world scenarios. Specifically, we propose three specialized versions of each primary dataset, referred to as missing datasets: DAD-missing, A3D-missing, and CCD-missing. These datasets are meticulously crafted to realistically mimic the variability and randomness of data omissions encountered in real-world settings. They include emulated missing observation rates of 10%, 20%, and 50%, as well as a fixed pattern of missing one or two frames every five frames (1/2 in 5 frames). These scenarios cover a broad spectrum of potential data loss situations, from minimal to severe. A stochastic mechanism is used to determine which observations are missing, avoiding the introduction of bias and more accurately reflecting the unpredictability inherent in real-world data collection. To evaluate the adaptability and effectiveness of our model, we conduct training on reduced versions of the datasets, specifically 50% and 75% subsets. We then evaluate the performance of our proposed model on both complete and missing datasets. These evaluations aim to gauge the model's adaptability to unfamiliar data and its proficiency in handling data omissions, providing a comprehensive evaluation of the robustness of our proposed model.This study evaluates model performance by considering both the accuracy (Average Precision) and timeliness (Time-to-Accident) of model predictions.Accuracy. Accident detection accuracy of the model is quantified by recall (R), which is defined as the ratio of correctly identified Table 1 : Comparison of models seeking balance between mTTA and AP on the complete datasets. Bold and underlined values represent the best and second-best performance in each category. Instances where values are not available are marked with a dash ("-").Model DAD [5] CCD [2] A3D [39] AP(%)↑ mTTA(s)↑ AP(%)↑ mTTA(s)↑ AP(%)↑ mTTA(s)↑ DSA [5] 48 Timeliness. The Time-to-Accident (TTA) is the metric used to evaluate the model's predictive timeliness. It measures the interval between the model's initial accident prediction (once the risk level surpasses a pre-set threshold) and the actual occurrence of the accident. A greater TTA indicates that the model can foresee accidents well in advance, providing drivers with more response time. The Mean Time-to-Accident (mTTA) calculates the average TTA values across various thresholds. Under strict recall rate conditions, we also evaluate the model's early warning effectiveness at a recall of 80%, referred to as TTA@R80.The proposed model is implemented using PyTorch and trained on an NVIDIA A40 (48GB) GPU over 80 epochs with a consistent batch size of 10. We use the Adam optimiser, initialising the learning rate at 1 × 10 -4 uniformly across all datasets. The object detector is configured to detect up to 19 candidate objects, and the embedding dimension for VGG-16 is set to 4096, and the hidden state dimension of the GRU is fixed at 512. In addition, the ReduceLROnPlateau strategy is used to schedule the learning rate, which adjusts the rate in response to the model's performance across epochs. See Appendix for more implementation details.Compare with SOTA Baselines on Complete Datasets. Table 1 illustrates that our model exhibits SOTA performance across all metrics on the DAD, A3D, and CCD datasets for considering the trade-off between timeliness (mTTA) and accuracy (AP) of accident anticipation. Specifically, on the CCD and A3D datasets, our model's AP and mTTA metrics have already reached or exceeded all baselines. On the DAD dataset, which covers a wide range of Table 2 : Comparison of models for the evaluation metrics on missing datasets. @R80 refers to the TTA@R80, which represents the value of mTTA at a recall of 80%. Bold and underlined values represent the best and second-best performance.Model drop-10% drop-20% drop-50% 1 in 5 2 in 5 AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ DAD [5] Ustring [2] 53 Frames AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ DAD [5] Training Set AP(%)↑ mTTA(s)↑ @R80(s)↑ AP(%)↑ mTTA(s)↑ @R80(s)↑ DAD [5] Ustring [2] 51.15 urban scenarios and traffic complexities, our model achieves an optimal AP of 65.30%, surpassing the second-ranked GSC model by 8.11%. Additionally, it maintained a competitive mTTA of 3.05 seconds. These results demonstrate our model's superior capability to navigate through complex and variable traffic scenes, including different levels of congestion, urban roads, and traffic conditions. Furthermore, Table 4 presents a detailed comparison of our model against the top baselines on the DAD dataset, highlighting the model's superior performance. Our model achieves the highest AP value and the corresponding highest mTTA value within the 5-second accident detection horizon. This indicates an average lead time before an accident of 1.91 seconds, which is 14.37% higher than the second-place DSA, providing more time to take preventive measures. Notably, comparing mTTA values without considering AP is not practically significant in real-world driving scenarios. Therefore, our experiments exclude such comparisons. Performance Comparison on Missing Datasets. Table 2 demonstrates the robustness of our model in handling missing observations. Our model significantly outperforms all other baselines when tested on datasets with 10%, 20% and 50% randomly missing datasets. On the A3D-missing, CCD-missing, and DDA-missing datasets, our model outperforms the leading models with an average improvement of at least 10.59% in AP and 6.69% in mTTA. With 10% of the data missing, our model outperforms all SOTA baselines tested on complete data, demonstrating significantly higher values in both AP and mTTA metrics-evidencing its superior predictive capability.As expected, the performance of the model is directly influenced by the amount of input data available. The more input frames that are omitted, the greater the impact on predictive performance. However, even in datasets with significant data omission (Drop-50%), our model's performance remained superior to other baselines and competitive with models tested on complete data. Furthermore, in datasets with continuous data loss (1 in 5 and 2 in 5 frames), our model's metrics were still better than most state-of-the-art (SOTA) baselines, demonstrating its robustness and broad applicability in real-world driving scenarios. Performance Comparison on Limited Training Sets. To demonstrate the scalability and efficiency of our model, we train it and some open-source baselines on a reduced portion of the training sets (50% and 75%) and evaluate them on both complete and missing datasets. Our model significantly outperforms all other baselines, as detailed in Table 3 , despite severe performance drops observed in top models like GSC and Ustring. Remarkably, our model still stands out with significantly higher AP and mTTA values across the board in missing data scenarios, even when trained on substantially less data, as shown in Table 5 . This finding emphasizes the model's capacity to minimize training data requirements, showcasing its adaptability in situations characterized by data loss and fragmentation errors common in the perception process. Ablation Study for Core Components. Table 6 reports the ablation results of five critical components in CRASH: Object Focus Attention, Context-aware Attention Block, Fast Fourier Transform, Temporal Focus Attention, and enhancement loss L 𝑒 .Evaluation across the DAD, A3D, and CCD datasets shows that models lacking any of these components exhibit reduced performance, as evidenced by significant decreases in AP, mTTA, and TTA@80% metrics compared to the holistic model. In particular, the integration of CAB and FFT emerges as a significant performance enhancer, underscoring their indispensable role in the contextaware module to improve the model's ability to capture contextual cues and thus improve prediction accuracy. Importantly, OFA significantly improves model performance by detecting critical interactions between traffic entities, which are essential for accurate accident prediction. It identifies potential anomalous intentions and movement trends of traffic agents within scenes, enabling the model to focus on the high-risk objectors. In addition, the inclusion of the enhancement loss L 𝑒 refines the hidden states of the model, while TFA enhances the semantic features within these states. This focus on key hidden states for anticipatory analysis, combined with the sophisticated information processing of other components, greatly improves the model's prediction reliability.Case Study for TFA Mechanism. To further demonstrate the effectiveness of the proposed TFA mechanism, we visualize the distribution of attention weights across eight attention layers within two distinct TFA blocks. As illustrated in Fig. 2 , the higher attention layers, particularly layers 4 to 8, receive a greater proportion of attention weights. This observation suggests that the influence on these upper layers increases with the number of attention layers, likely due to enhanced vector interactions. Contrary to initial expectations, the top layer does not dominate in terms of attention weights. Instead, the mid-upper layers (4-7) receive heightened attention, suggesting they may contain critical semantic features for accident prediction. This deviates markedly from traditional techniques that typically rely on the top layer's representations for accident prediction, which may overlook critical semantic features inherent in other layers.In light of these findings, we introduced the TFA mechanism to allocate weights dynamically across different attention layers. This allocation is meticulously calibrated based on the continuous evolution of hidden states within the video sequence, ensuring that each layer contributes optimally based on its informational content. Subsequently, the multi-layer fusion iteratively fuses and updates these outputs from these diversified TFA attention layers using the assigned weights, providing the object-aware module with a wide range of visual cues. In summary, the TFA mechanism promotes a more nuanced understanding and exploitation of the hierarchical features within the network, enabling the detection of subtle cues that are critical for early and accurate accident anticipation. Fig. 3 illustrates the accident anticipation capabilities of our model in challenging real-world driving scenarios. CRASH demonstrates a consistent ability to accurately identify impending accidents across a wide range of environmental conditions and to issue timely warnings at least 3 seconds in advance of potential incidents (TTA>3) in complete datasets, as shown in Fig. 3 (a-b ). Remarkably, even in scenarios featuring by data missing, as highlighted in Fig. 3 (c), our model calculates the likelihood of an accident in real-time with remarkable accuracy. In addition, Fig. 3 (d ) reveals that our model remains capable of predicting accidents at least 1.45 seconds in advance (TTA>1.45) in scenarios with up to 20% missing data despite being trained on only 50% of the training set. These qualitative results highlight the exceptional robustness of the model and its potential to tackle corner-case traffic scenarios. 
conclusion: 
 This study introduces a novel accident anticipation framework CRASH for autonomous driving, which integrates five key components: object detector, feature extractor, object-aware module, context-aware module, and multi-layer fusion. These components work in concert to analyze and interpret various features, capture the fine-grained correlations between different traffic agents, and account for the inherent uncertainty and long-tail effects in the accident anticipation. Rigorous evaluations conducted on the DAD, A3D, and CCD demonstrate the robustness and adaptability of CRASH, demonstrating its superior performance even in scenarios with data constraints and missing data. In addition, we introduce the enhancing versions of these datasets-DAD-missing, A3D-missing, and CCD-missing-to simulate the variability and randomness of real-world data omissions, further refining accident anticipation methodologies in data-missing scenes. 
 
 Appendix:  


Decision: This paper proposes a method to anticipate future crashes integrating information from object detectors. While some reviewers raised concerns the rebuttal is very thorough even adding additional ablation studies and clarifying novelty. If this modification can be added in the final version the paper can be considered solid.

Reviews: review: paper_summary: paper_summary: This paper introduces CRASH that is a framework for autonomous vehicle (AV) accident anticipation. This framework deals with unpredictable traffic, uncommon accident occurrences, complex scene dynamics, and restricted camera views etc. It combines five components: object detection, feature extraction, an object-aware module to prioritize high-risk objects, a context-aware module to capture broader scene information using the Fast Fourier Transform (FFT), and a multi-layer fusion module that dynamically analyzes temporal connections between scenes and visual features. The authors show that CRASH performs better than existing models on real-world datasets, achieving high average precision (AP) and mean time-to-accident (mTTA) measurements. The experiments are conducted with real-world dataset, including Dashcam Accident Dataset (DAD), Car Crash Dataset (CCD), and AnAn Accident Detection (A3D) datasets.
strength_weakness: strength_weakness: The paper is well structured with a clear presentation of the research on accident prediction. 

The experiments are conducted with three datasets. 

The proposed methods showcase good results on different datasets with complete or missing data.
limitations: limitations: The five-component architecture (object detection, feature extraction, object-aware module, context-aware module, multi-layer fusion) raises questions about its necessity for accident classification and time-to-accident (TTA) prediction. A deeper understanding of the relationships between these modules and their individual contributions to the overall performance is crucial. The current ablation study only analyzes the impact of removing individual components, but a more thorough examination is needed to fully grasp their interplay within the framework.

The whole model has VGG and C-RCNN for object detection. Exploring other object detection models could potentially improve performance or offer a more efficient solution.

The paper mentions using VGG16 backbones for both object detection and feature extraction.  It's unclear whether these backbones are shared or separate.  Furthermore, the model differentiates object-aware and context-aware features based on shape, but a more detailed analysis of their feature distributions and how they differ is missing. For exmaple, visualization of these features would be helpful in understanding their characteristics.

The proposed object-aware module requires further explanation to differentiate it from a standard self-attention module.  Specifically, how does the attention mechanism focus on objects relevant to accident anticipation?

Similar to the object-aware module, a comparison between the proposed context attention block and a general self-attention block is necessary. Understanding the structural differences would be beneficial.

The paper mentions that the Fast Fourier Transform (IFFT) and the context attention block improve the training stability of the model. However, the reviewer suggests that this claim needs further explanation and justification.

Table 1 shows good performance on complete datasets, but the significant increase in complexity compared to previous work doesn't seem to be fully justified by the marginal performance improvement.

Minor: Figure 1 lacks the input to the context-aware module, hindering a complete understanding of its functionality.
paper_summary: paper_summary: The article introduces a novel framework called CRASH, designed for predicting traffic accidents around autonomous vehicles (AVs) using camera footage, to enhance AV safety. The main challenges include the unpredictable nature of traffic accidents, their long-tail distribution, the complexity of traffic scene dynamics, and the limited field of vision of onboard cameras.

The CRASH framework integrates five components: an object detector, a feature extractor, an object-aware module, a context-aware module, and multi-layer fusion. The object-aware module prioritizes high-risk objects by calculating the spatiotemporal relationships between traffic agents. The context-aware module extends visual information from the temporal to the frequency domain using Fast Fourier Transform (FFT), capturing detailed visual features and broader context cues. The multi-layer fusion dynamically computes temporal dependencies between different scenes, iteratively updating correlations between visual features for accurate and timely accident prediction.

Evaluated on real-world datasets (DAD, CCD, and A3D), the CRASH model surpasses existing top baselines in key evaluation metrics such as Average Precision (AP) and mean Time-To-Accident (mTTA). Its robustness and adaptability are particularly evident in challenging driving scenarios with missing or limited training data, demonstrating significant potential for application in real-world autonomous driving systems.
strength_weakness: strength_weakness: 1. The collaboration in this paper adheres to academic standards.
2. A considerable number of comparative experiments were conducted, and some results show noticeable improvements.
3. The figures are clear and aesthetically pleasing.
limitations: limitations: 1. This paper only considers a single camera as input, which does not align with the multimodal theme of MM24.
2. Current mainstream autonomous vehicles are equipped with multi-view cameras and LiDAR covering 360 degrees, and LiDAR is crucial for perception. However, this paper only explores the use of a single camera. Additionally, for collision prediction, it is essential to consider displacement and velocity information obtained from IMU and GNSS, as vehicles are not always stationary or moving in a straight line at a constant speed. Therefore, the problem chosen in this paper lacks sufficient practical value.
3. The lack of code makes it difficult for reviewers to replicate the results. Additionally, there is no plan to release the code, which reduces the contribution to the community.
4. The motivation is unclear, and there is no solid theoretical basis to justify the necessity and rationality of the proposed method. It seems more like finding a target after shooting an arrow.
5. There are some grammatical issues, such as a question mark in the citation on line 152.
paper_summary: paper_summary: The paper presents CRASH, an innovative framework designed to enhance the collision prediction capabilities of autonomous vehicles (AVs), a critical advancement for improving safety. Addressing the inherent unpredictability and complexity of traffic accidents, the CRASH system integrates a suite of components including an object detector, feature extractor, object-aware module, context-aware module, and multi-layer fusion. Notably, the object-aware module prioritizes high-risk objects through spatial-temporal analysis, while the context-aware module broadens visual information using FFT. The multi-layer fusion dynamically assesses temporal dependencies for precise predictions. Tested on real-world datasets, CRASH outperforms current benchmarks in Average Precision (AP) and mean Time-To-Accident (mTTA), showcasing robustness and adaptability, particularly in scenarios with limited training data. This work significantly contributes to the field by pushing forward the performance boundaries of collision prediction for AVs.
strength_weakness: strength_weakness: Innovation: The paper introduces an innovative accident prediction framework termed "CRASH," which enhances the accuracy of accident anticipation by integrating global contextual information with intricate spatio-temporal interactions.

Theoretical Approach: The authors have developed a novel Object Focus Attention (OFA) mechanism and a context-aware module that leverages the Fast Fourier Transform (FFT) and Context-aware Attention Blocks (CAB). These innovations allow for the computation of fine-grained correlations between nuanced spatial and appearance changes in different objects, contributing to a deeper understanding of the dynamics preceding an accident.

Experimental Design and Data Augmentation: The paper's experimental design is robust, with comprehensive benchmark tests conducted on the DAD, A3D, and CCD datasets. The authors have demonstrated the superior performance of the CRASH framework over state-of-the-art (SOTA) baselines across key metrics. A particular strength of this work is the innovative approach to data augmentation, simulating the variability and randomness of missing data encountered in real-world scenarios. This provides a more realistic evaluation of the model's performance.

Overall, the manuscript is well-structured, with a clear presentation of ideas and findings. The logical progression of the paper is easy to follow.
limitations: limitations: The paper contributes to the field of autonomous vehicle safety with the CRASH framework, which integrates several novel components for accident anticipation. However, there are a few aspects that require further consideration:

Computational Efficiency: The integration of FFT and multi-layer fusion in the CRASH framework is an innovative approach; however, it may come at the cost of significant computational overhead. This could result in substantial processing delays, which is a critical concern in the context of real-time collision prediction for autonomous vehicles. If the computational demands lead to latency, the system might fail to predict collisions in time. It is suggested that the paper include an analysis of the computational efficiency.

Object Detection Accuracy: A potential limitation is the framework's dependence on object detection, where errors could significantly affect collision predictions. The paper would benefit from a sensitivity analysis on how object detection inaccuracies impact the system's reliability.
paper_summary: paper_summary: The paper presents CRASH, a novel accident anticipation framework designed for autonomous vehicles (AVs). It aims to predict accidents from camera footage by integrating an object detector, feature extractor, object-aware module, context-aware module, and a multi-layer fusion approach. The object-aware module focuses on high-risk objects, while the context-aware module uses Fast Fourier Transform (FFT) to analyze traffic scenes. The multi-layer fusion dynamically computes temporal dependencies and updates correlations between visual features. The framework has been evaluated on three real-world datasets and has shown robust performance, especially in challenging scenarios with limited training data.
strength_weakness: strength_weakness: 1. Clarity of Writing: The paper is well-written with a clear and logical flow, making it easy to understand the proposed system and its components.
2. Visualization: The figures and tables are clearly designed, enhancing the readability and comprehension of the framework and results.
3. Dataset Validation: The authors have validated their model on multiple datasets, which speaks to the robustness and generalizability of the CRASH framework.
limitations: limitations: 1. Innovation: The paper presents a framework that shares significant similarities with prior work, such as DSTA. The authors could improve the novelty by providing a more detailed motivation and a comparative analysis that highlights the unique contributions and advantages of CRASH.
2. Framework Description: The abstract lacks a clear motivation for the study. A more compelling rationale for the development of CRASH over existing methods would strengthen the paper.
Figure Captioning: The caption for Figure 1 could be more descriptive to better explain the framework's architecture without needing to refer back to the text.
3. Ablation Studies: The analysis of ablation studies is not as thorough as it could be. While Table 6 contains valuable data, the discussion in the text (743-784) lacks detailed description and quantitative analysis to fully understand the impact of each component.
4. Detail Issues: There are minor issues such as a citation format error on line 152 and inconsistency in the use of abbreviations for "Table" and "Fig.", which should be corrected for publication standards.
Given the strengths and limitations outlined, my recommendation is borderline reject. The paper presents a solid framework with good empirical results but needs to address the issues of novelty and detailed analysis to be considered for acceptance. The final score may be adjusted based on the authors' rebuttal and their ability to address these concerns.