When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Authors: Dong Xiao
Guangyao Chen
Peixi Peng
Yangru Huang
Yifan Zhao
Yongxing Dai
Yonghong Tian
Venue: ICML 2025 spotlightposter (2025)
Keywords: Anomaly Detection
Multimodal Network
Autonomous Driving
Primary area: applications->computer_vision
TL;DR: We propose an asynchronous hybrid anomaly detection network that leverages event data to accurately detect anomalies during driving with extremely low response time.

Abstract: Anomaly detection is essential for the safety and reliability of autonomous driving systems. Current methods often focus on detection accuracy but neglect response time, which is critical in time-sensitive driving scenarios. In this paper, we introduce real-time anomaly detection for autonomous driving, prioritizing both minimal response time and high accuracy. We propose a novel multimodal asynchronous hybrid network that combines event streams from event cameras with image data from RGB cameras. Our network utilizes the high temporal resolution of event cameras through an asynchronous Graph Neural Network and integrates it with spatial features extracted by a CNN from RGB images. This combination effectively captures both the temporal dynamics and spatial details of the driving environment, enabling swift and precise anomaly detection. Extensive experiments on benchmark datasets show that our approach outperforms existing methods in both accuracy and response time, achieving millisecond-level real-time performance.

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Dong Xiao 12
Guangyao Chen I Peixi Peng Yangru Huang'
Yifan Zhao Yongxing Dai
Yonghong Tian 1236
Abstract
Anomaly detection is essential for the safety and reliability of autonomous driving systems. Cur- rent methods often foeus on detection accuracy but neglect response time, which is critical in time-sensitive driving scenarios. In this paper, we introduce real-time anomaly detection for au- tonomous driving, prioritizing both minimal re- sponse time and high accuracy. We propose a novel multimodal asynchronous hybrid network that combines event streams from event cameras with image data from RGB cameras. Our net- work utilizes the high temporal resolution of event cameras through asynchronous Graph Neural Network and integrates it with spatial features extracted by CNN from RGB images. This com- bination effectively captures both the temporal dynamics and spatial details of the driving en- viromment, enabling swift and precise anomaly detection. Extensive experiments benchmark datasets show that approach outperforms ex- isting methods in both accuracy and response time, achieving millisecond-level real-time per- formance. The code is available at https: //github... wwwppku-XD/ XD/DUXXXDEEeetADD
Rrsi
Auonly Swe
Anomaly Detection Model
Thrakeld
Teaference
detection
Response Time
Figure I. For real-time anomaly detection with an emphasis on response time, the overall response time primarily influenced by the model's inference duration and the time taken to identify anomalies.
revolutionize the transportation industry by enhancing road safety, reducing traffic congestion, and improving fuel effi- ciency (Huang et al., 2018; Caesar et al., 2020). The core of autonomous vehicles lies in their ability to perceive and interpret complex and dynamic driving environments accu- rately and efficiently. Among the various perception tasks, anomaly detection plays a pivotal role in ensuring the safety and reliability of autonomous systems. Anomalies, such as unexpected obstacles, erratic behaviors of other road users, sudden changes in the environment, can pose significant risks if not promptly identified and addressed (Fang et al., 2024). For example, a pedestrian darting onto the road from behind a parked vehicle or sudden road obstruction re- quires the autonomous vehicle to react within milliseconds to prevent potential accidents (Gehrig & Scaramuzza, 2024).
1. Introduction
Autonomous driving technology has been at the forefront of research and development in recent years, promising to
'Equal contribution 'National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking Uni- versity DDepartment of Software and Microelectronies, Peking Computer Engineering,
University 'School of Electronic Peking University "State Key Laboratory of Virtual Reality Tech- nology and Systems, SCSE, Beihang University 'Baidu Inc "Peg Cheng Laboratory. Correspondence to: Guangyao Chen <gy.chen @ poku.edu.cn. Peixi Peng <pxpeng @ pku.cdd..cn>. Yonghong Tian <yhtian @ cittaa@@ku.cdducom pku.edu.cn>.
Anomaly detection vital component in ensuring the safety of autonomous driving systems. Despite substantial progress, current methods often prioritize detection accuracy over an equally crucial factor: response time (Cui et al., 2023; Zeng et al., 2023). Many state-of-the-an solutions rely on increasingly sophisticated deep neural networks, which can ineur large inference latencies (Yao al., 2022:
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Karim et al., 2023). In a domain where delay of even few hundred milliseconds can dictate the difference between safe braking and collision (Tian et al., 2024), these lengthy detection times significantly compromise the very safety guarantees that autonomous vehicles are designed to deliver (Wang et al., 2023).
tection accuracy but also substantially lowers response time, fulfilling the stringent safety requirements of real-world au- tonomous driving environments (Karim et al., 2023; Yao et al., 2022).
Our main contributions are summarized as follows:
To overcome this limitation, we focus on the task of real- time anomaly detection in autonomous driving, specifically targeting the detection of sudden hazardous anomalies from the ego-vehicle perspective, while explicitly incorporating response time into the performance evaluation metrics. In addition to detection accuracy, this perspective emphasizes minimizing the total response time- encompassing both inference latency and the delay between anomaly occurrence and detection (Tian et al., 2024). By centering on this time- critical requirement, our approach seeks to bridge the gap between high-accuracy detection and the urgent demands of real-world driving scenarios, where prompt decision- making is paramount for safety.
We formalize the task of real-time anomaly detection, emphasizing the pivotal role of rapid response safety- critical autonomous driving scenarios. In doing so, we underscore how existing methods overlook this time- sensitive aspect, compromising overall system safety,
We propose a novel network architecture that syner- gistically fuses event stream data with RGB images to strike an optimal balance between minimal inference latency and high detection accuracy. By capitalizing on the asynchronous, fine-grained temporal informa- tion from event cameras and the rich spatial features of conventional images, our model delivers reliable performance even under challenging conditions.
To address the challenge of real-time anomaly detection, we introduce a multimodal asynchronous hybrid net- work that reduces inference latency and detection delays while preserving high accuracy. Our approach strategically combines event cameras and conventional RGB cameras to exploit their complementary advantages. Event cam- eras capture brightness changes at microsecond resolution, providing sparse yet highly informative data for dynamic scenes (Gallego et al., 2020). At the same time, RGB cam- eras deliver rich spatial context, albeit with higher latency and susceptibility to motion blur (Zhou & Jiang, 2024). By fusing the asynchronous, sparse event streams with con- tinuous image data, we enable the model to capture both fine-grained spatial details and rapid temporal cues.
We conduct extensive experiments on multiple bench- mark datasets, demonstrating that our approach not only surpasses state-of-the-art baselines in detection accuracy but also significantly reduces response time. These findings validate the practical effectiveness of our method, reinforcing its suitability for real-world autonomous driving applications.
2. Related Works
Ego- Ego-View Traffic Accident Detection (TAD). TAD aims to identify accidents within specific time frames and regions using two primary approaches: frame-level and object-level methods. Frame-level methods extract features from video frames and classify them to detect accidents (Vijay et al., 2022; Zhou et al., 2022). For example, You and Han (You & Han, 2020) developed a traffic dataset and utilized a 3D-CNN for accident localization. Reconstruction based techniques (Chong & Tay, 2017: Zhao et al., 2017: Gong et al., 2019) identify anomalies by comparing reconstructed frames with actual ones. Due to the limited availability of real accident data, synthetic datasets and domain adapta- tion methods are often employed to enhance performance (Batanina et al., 2019; Tamagusko et al., 2022). Object-level methods focus on the consistency of object movements over time by using detectors and trackers to generate trajecto- ries, thereby reducing the impact of dynamic backgrounds. These methods analyze trajectories (Santhosh al., 2021; Chakraborty et al., 2018) or evaluate the consistency of object positions (Le et al., 2020; Taccari et al., 2018; Hu et al., 2021a) to detect accidents. Yao et al. (Yao et al., 2022; 2019) proposed an unsupervised approach that pre- diets future object positions, flagging significant deviations
In particular, we employ an asynchronous Graph Neural Net- work (GNN) to process the event stream data, harnessing the inherent sparsity and asynchronous nature of event cam- eras (Li et al., 2021). Meanwhile, a CNN- -xNed ResNet tracts high-level spatial features from the RGB images (He et al., 2016). Subsequently, by leveraging a GRU module to jointly learn the spatio-temporal relationships of object- level and frame-level event streams and RGB features, our model can anticipate anomaly trends in advance, achieving both accurate and rapid anomaly detection. Notably, we the first to introduce the unique characteristics of event streams as critical features for road traffic anomaly detec- tion, and we fully exploit their high temporal resolution and asynchronous nature through an asynchronous network architecture. This approach is specifically designed for real- time operation, enabling crucial millisecond-level response times in autonomous driving environments. Extensive ex- periments on multiple benchmark datasets demonstrate that our approach not only outperforms existing methods in de-
2

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
as potential accidents. Additionally, object-level strategies model interactions among objects to detect abrupt contex-
time, identifying unexpected objects or behaviors that may pose risks.
tual changes that may indicate accidents (Fang et al., 2022; Yamamoto et al., 2022; Roy et al., 2020; Vijay et al., 2022). MOVAD (Rossi et al., 2024) achieves efficient online de- tection of traffic anomalies based solely on dashcam videos by combining a Video Swin Transformer and an LSTM module, and it is the first to introduce the concept of online traffic anomaly detection.
We define an anomaly indicator function A, as:
A [[2, is anomalous),
(I)
where I[-) is the indicator function, returning 1 if the condi- tion is true and 0 otherwise.
A detection model f()) assigns an anomaly score 8t f(x)). An anomaly is detected when the score exceeds threshold 0:
Ego-View Traffic Accident Anticipation (TAA). TAA focuses on predicting potential collisions by identifying unusual behaviors in advance, providing critical time for safe decision decision-making making. It primarily involves predicting ob- ject trajectories to assess accident likelihood through de- tection, tracking, and prediction workflows (Haris et al., 2021; Thakur et al., 2024). In complex environments like highways, methods such as SVMs and HMMs analyze ve- hicle trajectories to forecast accidents (Xiong et al., 2017: -utoerrez-Osorio & Pedraza, 2020). To handle frequent obstructions in dashcam footage, recent approaches employ Dynamic Spatial Attention (DSA) (Chan et al., 2017) and RNNs to capture complex spatial and temporal relation- ships (Karim et al., 2022). Beyond trajectory prediction, TAA also involves identifying high-risk areas (Karim et al., 2023; Shimomura et al., 2024) and modeling driver attention (Chen et al., 2023). Risk localization techniques integrate agent representations with regional interactions to highlight areas with high accident probabilities (Zeng et al., 2017). The DRAMA dataset (Malla et al., 2023), which combines visual and textual data, enhances prediction accuracy by pro- viding detailed descriptions of potential accident scenarios. Driver attention models utilize gaze direction to emphasize possible dangers, focusing on risky areas (Bao et al., 2021). Additionally, attention maps improve TAA interpretabil- ity by highlighting high-risk zones, supporting integrated approaches that combine trajectory prediction, risk localiza- tion, and driver attention modeling (Monjurul Karim al., 2021).
A. = I[s, > 0).
Response Time. Response time R is a critical metric, com- prising the detection delay and the model's inference time:
R ATdetection + T inference
where dedtecttoo Tdetection T Toceurencee is the delay be- tween the anomaly occurrence and its detection. Tinfrrence is the time taken by the model to process the input and produce a result. The goal of Real-Time Anomaly Detection is to minimize R while ensuring high detection accuracy by:
Minimizing Inference Time (Tinterence:: Developing efficient algorithms that process data rapidly to reduce computational delays.
Reducing Detection Delay ( Enhancing the model's ability to promptly identify anomalies im- mediately after they occur.
4. Method
To achieve real-time anomaly detection with minimal infer- ence time and detection delays, we propose a multimodal asynchronous hybrid network that integrates sparse event streams with RGB image data. Our framework processes RGB images using a ResNet to extract appearance features and captures event data through an asynchronous GNN with spline convolution. The image features shared unidi- rectionally with the GNN, enabling the GNN enhance event feature representation without reciprocal communi- cation. This design significantly improves performance, especially in scenarios with sparse events, such as static or slow-moving conditions.
3. Real-Time Anomaly Detection
Real-Time Anomaly Detection in autonomous driving is essential for ensuring safety by swiftly identifying and re- sponding to unexpected objects and behaviors in the driving environment. This task demands a system that operates with minimal latency, capable of detecting dynamic changes such as pedestrians suddenly crossing the road or vehicles appearing abruptly. The primary challenge is to achieve high precision in anomaly recognition while maintaining response times at the millisecond level.
The features from both modalities are fused and passed to detection head, which generates bbondigb boxes. These object-level features are further refined using global graph that incorporates bounding box priors, enhancing spa- tial anomaly detection. To capture temporal dependencies, we employ a Gated Recurrent Unit (GRU) that processes the video sequences. An attention mechanism assigns higher weights to potentially anomalous objects, ensuring focused
Problem Formulation. Let (x denote sequence of sensor observations, where X, E RD represents the sensor data at time t. The objective is to detect anomalies in real-
3

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
and efficient analysis. The combination of spatial, temporal, and attention-enhanced features enables accurate and swift anomaly detection.
efficient aggregation of neighbor information, accelerat- ing computation through lookup tables during deployment. Temporal consistency is maintained by aggregating nodes into a voxel grid and applying directional voxel pooling, which preserves the temporal order of events.
In Sec. 1.1, we describe the network backbone, highlighting the unidirectional integration of image and event features and the role of asynchronous GNNs in feature representation. Sec. 4.2 elaborates on how spatial and temporal features are fused for anomaly detection. Figure 2 presents the overall architecture of our proposed framework.
Feature Fusion. To integrate the extracted features from both modalities, we fuse the CNN and GNN outputs by aug- menting each GNN node feature fo with the corresponding CNN feature gr (f;)) sampled at the node's location:
f = [f.. 91 (2:)]-
(7)
4.1. Multimodal Asynchronous Hybrid Network
This fusion enhances the model's s ability to leverage spatial information from images alongside the temporal dynam- ies captured by event streams, improving the detection of anomalies in diverse driving scenarios.
To achieve real-time anomaly detection, we propose Mul- timodal Asynchronous Hybrid Network that efficiently inte- grates RGB images and event streams. Our network consists of two parallel branches: a CNN for processing image data and a GNN for handling event streams. This dual-branch architecture enables rapid and accurate feature extraction from both modalities.
The Multimodal Asynchronous Hybrid Network is opti- mized for real-time performance by utilizing asynchronous processing and efficient feature fusion. This design ensures minimal lateney and high accuracy in detecting anomalies, making well-suited for the stringent requirements of au- tonomous driving systems.
Image Feature Extraction. The CNN branch, denoted as Fj. processes input images I E R x3 to extract rich spatial features. Utilizing ResNet architecture, F, generates detection outputs Dj and intermediate feature maps G = ( V at various layers. These intermediate features are reused in the GNN branch to enhance computa- tional efficiency.
4.2. Anomaly Detection Network
To enable real-time anomaly detection, our Anomaly Detec- tion Network efficiently extracts and processes object-level features from both event streams and RGB images, captur- ing spatial and temporal dynamics with minimal latency.
Asynchronous Event Graph Construction. Event streams
E = {e+- (z,, t, pt)}. where z; (u,, 31)) denotes pixel coordinates, i; is the timestamp, and pt E 1- repre- sents polarity, are captured by event cameras when lumi- nance changes exceed a threshold C:
Object Feature Extraction. We utilize an asynchronous GNN to extract features from event data overlapping with detected bounding boxes. For each object i at time 4, the GNN generates an event-based feature ot,
| > C.
(4)
oti AsyncGNN (E,,s; 0GNN),
(8)
These events are modeled as nodes in a graph G (V,E) with normalized spatial coordinates d; ( - x ) and
where E., represents the event points within the bounding box of object i, and faNN are the GNN parameters.
scaled timestamps 4, = Bl., Edges are formed based on
spatial and temporal proximity within radius R, and edge features are defined as:
Concurrently, features from the RGB image are extracted using a CNN, denoted as Jt.i- We concatenate the GNN and CNN features to form comprehensive feature vector:
1
ei 2 (MJayy - 2y, +
(5)
2
Pt. 9t..:
(9)
where ay and jay are the normalized spatial coordinates of nodes and j. Each node connects to up to 16 neighbors to maintain computational efficiency.
which is then reduced in dimensionality via a fully con- nected layer:
fa, = ((pr.; 00).
(10)
Event Feature Extraction. We employ Deep Asyn- chronous Graph Neural Network (DAGT) (Gehrig & Scara-
Relational Learning. To model tem- poral dependencies and interactions between objects, we employ Gated Recurrent Units (GRUs). For each object i, the bounding box features by, and the fused features fe, are processed as follows:
muzza, 2024) to process the event graph using residual graph convolutional layers with spline convolutions:
f = Woli+ + Σ Wegg)ss
(6)
JEN(()
h = hh, 01),
(11)
where We and W(es)) are learnable weights, and (i) de- notes the neighbors of node i. Spline convolutions enable
h = (,,, 02).
(12)
4

---

Datasets. We employ two datasets, ROL (Karim al., 2023) and DoTA (Yao et al., 2022), both annotated with de- tailed temporal, spatial, and categorical information, making them highly suitable for traffic anomaly detection research. The ROL dataset provides comprehensive annotations for each video clip, which encompass object descriptions, acci- (14) dent details, and scene contexts. Temporal annotations pin- point the initial appearance of risky traffic agent and the on- set of an accident, offering insights into the dynamics of risk When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Events
Gruph
dkikkl
Invertion
Look-up-Tali Table Sprire Corv.
Clyect-level
Fext. Sempling
Albention modale
Ibbox
Fusion
GRU
DCant
[4) Multimmdie Asyrchrencus byfrrd Netwerk
Anomelyy
Concet
Figure 2. Overview of the proposed multimodal asynchronous hybrid network. (a) The framework integrates RGB images and event streams as inputs. Appearance features are extracted from RGB images using a ResNet architecture, and event features are derived from streams through an asynchronous graph neural network (GNN) utilizing spline convolution. These features then fused and processed by detection head to generate object bounding boxes. (b) At the object level, features are refined through a global graph, leveraging bounding box priors, and temporal dependencies are captured using gated recurrent units (GRU), attention mechanism dynamically assigns weights to detected objects, enhancing the focus and accuracy in anomaly detection by emphasizing anomalous objects.
Here, 81 and 0g the parameters for the GRUs handling bounding box and fused features, respectively.
5. Experiments
5.1. Experimental Setup
Attention Mechanism. To prioritize significant objects, we apply an attention mechanism to the GRU outputs. The attention weights for bounding box and fused features are computed as:
0b2* softmax (tanh (HI,,u)).
(13)
0f, softmax (tanh (H (H)),
(15)
H = Hyaay
(16) development and collision occurrence. Spatially, the dataset includes bounding boxes for each traffic agent, which are ini- tially detected using YOLOv5, subsequently tracked across frames with DeepSort (Veeramani et al., 2018), and finally refined by human annotators to ensure high accuracy. Cate- gorical annotations in the dataset include traffic agent types and scene contexts, enhancing the dataset's utility for in- depth traffic behavior analysis and research. DoTA stands as the first openly available dataset tailored for traffic video anomaly detection, featuring robust temporal, spatial, and categorical annotations. It comprises over 4,600 video clips, collected under a variety of regional, weather, and light- ing conditions, with each video documenting one specific anomaly. Temporal annotations DoTA detail the start, duration, and conclusion of anomalies, while spatial annota- tions provide bounding boxes coupled with unique tracking IDs for each object involved in the anomalies. Currently, our DoTA dataset, ROL dataset, and all existing real-world inon-synthetie first-person autonomous driving anomaly detection datasets lack the event modality. To address this, we utilized the v2e (Hu et al., 2021b) conversion technique to generate and supplement event modality data. This al- lows us to simulate the continuous event streams that would
where Ho, and F. are the hidden states for bounding box and fused features, and wh, wy are learnable parameters.
Risk Score Prediction. The attention attention-weighhee weighted features are concatenated to form unified representation for each object:
hn. = (ho,e,s hyssl,
(17)
which is then passed through fully connected layer and softmax activation to compute the riskiness score $t.s:
$t,i = softmax (o (be.s:0))
(18)
Here, fg are the parameters of the final classification layer. This network architecture ensures that anomalies are de- tected accurately and promptly by integrating spatial fea- tures from RGB images with temporal dynamics from event streams, all processed through efficient asynchronous and recurrent mechanisms tailored for real-time performance.
5

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
be captured by event cameras in real-world scenarios. It is important note that v2e is solely used for generating supplementary event data and serves no other purpose in our work.
calculates the area under the Precision-Recall curve, offering intuitive measure for imbalanced datasets by emphasizing both precision and recall.
Mean Average Precision (mAP). mAP (Lin et al., 2014) evaluates detection performance across varying loU thresh- olds (from 0.5 to 0.95 in increments of 0.05), thus capturing a more nuanced perspective on overall detection robustness.
Implementation Details. Our proposed multimodal anomaly detection model is implemented in PyTorch. For object detection, RGB frames are resized to 224 224 pixels and processed using a ResNet50 backbone for feature extrac- tion. We adopt the YOLOX framework for bounding box detection, optimizing with IoU loss, class loss, and regres- sion loss. For anomaly detection, features from the tracked objects are fed into a GRU network to capture temporal dependencies, complemented by an attention mechanism that focuses on potential anomalies. Asynchronous event data are handled using an asynchronous GNN layer, which models interactions at the frame level with bounding box priors aiding spatial-temporal analysis. The integration of spatial and temporal features from the ResNet backbone facilitates comprehensive anomaly detection.
Mean Time-to-Accident (mTTA), mTTA (Fang al., 2023) measures the average earliest time at which risky agent's score St,5 surpasses threshold 8, reflecting the model's capability to foresee accidents before they occur.
Frame-Level AUC (AUC-Frame). AUC-Frame evaluates the model's ability to detect risky frames within a video. It is defined as the area under the ROC curve at the frame level:
AUC-Frame = L. (PR(() ((FPR(())
(19)
To address cross-modalin training schedule ambiguity, both modalities are trained with clearly defined parameters. The RGB component is trained for 30 epochs using a batch size of 64, and the dataset contains 1,920 images per epoch, re- sulting in a total of 57,600 data passes during training. For the event-basee component, derived from the v2e conver- sion of the image-based dataset, we employ a batch size of 32 and a dataset size of 1,920 samples per epoch, with training spanning 1 150,000 iterations. This is equivalent to approximately 2,500 passes over the event dataset, ensuring comprehensive learning of the event-derived features.
where TPR(t) and FPR(t) denote the true positive rate and false positive rate at threshold f, respectively.
Mean Response (mResponse). To capture not just if but also how quickly anomalies are detected at various sensi- tivity levels, we introduce mResponse. Unlike a single- threshold evaluation, mResponse measures the average de- tection delay across multiple thresholds, offering a more holistic view of real-time performance. Formally, it is de- fined as:
1 Σ n1 3=1
mResponse-
Response,,
(20)
Training employs the Adam optimizer for the GRU-attention module with learning rate of 0.001, and the Adam W op- timizer for the GNN-ResNet combination with a learning rate of X 10-4, A ReduceLROnPlateen scheduler adjusts learning rates to optimize training stability. Class weights in the ROL dataset are set to 0.27 for the negative class and for the positive class to balance the model's 's response. These detailed training schedules and parameters ensure effective integration of detection and attention mechanisms for both spatial and temporal anomaly identification.
where n is the number of thresholds and Response, is the detection delay at the j-th threshold. By aggregating re- sponse times across varying operational sensitivities, mRe- sponse provides a more robust measure of how promptly the model flags anomalies, making it particularly suitable for real-world, safety-critical scenarios.
5.3. Result Analysis
5.2. Evaluation Metrics
We evaluated the effectiveness of our proposed model (OURS) against several established methods for anomaly de- tection on the ROL dataset. Performance was assessed using key metrics including Area Under the Curve (AUC), Aver- age Precision (AP). Frame-Level AUC (AUC-Frame), and mean Time-to-Aceident (mTTA). The compared models in- clude ConvAE (Hasan et al., 2016), ConVLSTMAE (Chong & Tay, 2017), AnoPred (Liu et al., 201 8), FOL (Yao et al., 2019) (with variants FOL-IoU, FOL-Mask, FOL-STD, and FOL-Ensemble (Yao et al., 2022)), MAMTCF (Liang et al., 2023), AM-Net (Karim et al., 2023), STEE (Zhou et al., 2022), and TTHF (Liang et al., 2024).
To comprehensively evaluate our real-time anomaly detec- tion model, we employ a set of metrics designed to measure both predictive accuracy and timeliness:
Area Under the Curve (AUC). AUC (Hanley & McNeil, 1982) quantifies how well the model distinguishes between risky and non-risky agents by computing the area under the ROC curve, thereby providing single metric that balances true positive rate (TPR) and false positive rate (FPR).
Average Precision (AP). AP (Everingham et al., 2010)
6

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Table I. Comparison of the proposed model with existing methods on the ROL. and DoTA test datasets
METHOD
AUC(%) yt
AP(% yt
AUC-FRAME( %
(ST
FPSt
MRESPONSE(SM)
DATASETS
ROL
DoTA
ROL
DOTA
ROL
DoTA
ROL
DOTA
ALL
ROL
DOTA
CONVAE(HASAN ET Al.,, 2016)
0.759
0.779
0.493
0.521
0.608
0.663
1.64
1.75
82
2.44
2.31
CONWLSTMAE(CHONG & TAY, 2017)
0.713
0.501
0.479
0.626
0.594
0.595
1.47
1.82
65
2.67
2.58
ANOPRED(LIU AL., 2018)
0.773
0.790
0.517
0.541
0.610
0.675
1.74
67
2.39
2.26
FOI--IOU( Al., 2022)
0.817
0.830
0.539
0.563
0.660
0.730
1.95
2.09
56
2.14
1.79
FOL-MASK YAO ET Al.,, 2022)
0.826
0.846
0.546
0.569
0.674
0.725
1.98
1.99
44
1.98
2.05
FOL-STD(YAO ET AL., 2022)
0,837
0.852
0,550
0.573
0.679
0.714
1.94
2.10
41
2.01
2.03
FOL. ROL--NNSMMBLER E[ ET AL.,, 2022)
0.849
0.866
0.563
0.577
0.698
0.744
2.05
2.35
MAMTCF(LLIANG ET AL., 2023)
0.841
0.862
0.568
0.581
0.701
0.766
2.01
2.11
98
1.88
1.81
AM-NET(KARIM ET AL., 2023)
0.855
0.874
0.576
0.586
0.707
0.765
2.18
2.24
61
1.96
1.88
STFE ET AL.. 2022)
0.862
0.881
0.579
0.602
0.728
0.793
2.23
2.34
77
2.04
1.99
TTHRRLIANG ET AL., 2024)
0.871
0.891
0.585
0.618
0.733
0.847
2.35
2.41
2.58
OURS
0.879
0.896
0.570
0.623
0.736
0.823
2.80
2.78
579
1.17
1.21
As summarized in Table 1, OURS achieves leading perfor- mance in both AUC and AP metrics, demonstrating superior capability in distinguishing risky agents and balancing pre- cision with recall. While the TTHF method outperforms OURS in the AUC-Frame metric, OURS significantly sur- passes all other models, including TTHF, in terms re- sponse time. This superior response time is attributed to OURS's reliance asynchronous Graph Neural Networks (GNNs) and event stream integration, which enable in- ference speed approaching 600 FPS. Additionally, OURS achieves an exceptionally low mean response time (mRe- sponse), highlighting its promptness in detecting anomalies. This low latency is a result the model's high inference speed and efficient anomaly detection mechanisms, as fur- ther evidenced by the favorable mTTA values. In contrast, the TTHF method, which integrates text information fusion, exhibits slower response times despite higher detection per- formance. Overall, the comparative analysis underscores that OURS not only sets a new benchmark in AUC and frame-level anomaly detection but also significantly en- hances early risk localization capabilities. These results establish OURS as the leading model real-time anomaly detection, combining high accuracy with exceptional timeli- ness, as detailed in Table 1.
diverse driving scenarios. Meanwhile, event features cap- ture asynchronous and dynamic motion cues, effectively representing the relative movement between objects and the autonomous vehicle. This enables the model to rapidly and reliably detect anomalies, particularly in challenging condi- tions such as extreme lighting or at night, further strength- ening the robustness of the system.
GRU Module for Temporal Dynamics. The GRU module is critical for capturing and leveraging temporal information. Integrating GRUs increases AUC from 0.805 to 0.817 and AP from 0.479 to 0.508, indicating a more accurate classifi- cation of anomalies. Moreover, m'TTA improves from 1.44 to 1.98 seconds, demonstrating the GRU's effectiveness in accumulating temporal features and enabling early detec- tion essential feature for real-time anomaly detection.
A Attention Module. Although secondary to GRUs in mod- eling temporal dependencies, the Attention module con- siderably boosts the model's sensitivity to anomalies by concentrating on salient regions. This targeted approach proves especially beneficial in complex or cluttered envi- ronments, where focusing on relevant features is critical for accurate anomaly detection.
BBox and Object Modules for Precise Localization. The BBox module refines the model's localization capabili- ties, delivering more precise bounding box information and thereby improving AUC and AP metrics. Building on this, the Object module leverages these bounding boxes to further enhance detection accuracy by extracting detailed object- level features. This enhancement is particularly valuable in crowded or occluded scenes, where precise object recogni- tion is challenging.
5.4. Ablation Studies
We conducted ablation experiments to investigate the im- pact of various modules model performance, focusing on metrics including AUC, AP, AUC-Frame, mTTA, and mAP. Table 2 provides summary of the results on the ROL dataset.
RGB + Event. Incorporating both RGB and event data sig- nificantly enhances the model's overall performance. RGB features offer rich visual information that aids in distinguish- ing between object categories such as vehicles and pedestri- ans, while also complementing event stream data to improve object detection accuracy, as reflected by increased mAP in
Together, the multimodal integration (RGB + Event), GRU, Attention, BBox, and Object modules comprehensively el- evate the model's performance across all metries. While RGB images increase mAP through richer spatial details, the GRU and Attention modules substantially enhance temporal
7

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Table Ablation study results for different model configurations ROL. dataset, showing the effects of different components on AUC, AP, AUC-Frame, mTTA, and mAP.
RGB
EVENT
GRU
ATTENTION
BBox
OBJECT
TWO-STAGE
AUC(S)
AP(S)T
AUC-FRAME(SS)
MTTA(S)1
MAP(S)T
0.805
0.648
41,66
Y
0.817
0.508
0.668
1.98
43.59
0.819
0.498
0.637
1.32
43.77
0.691
0.813
0.507
0.688
1.73
43,57
0.839
0.531
0.703
2.11
43.29
0.845
0.539
0.694
1.96
42.94
0.868
0.561
0.726
2.51
43.82
0.736
detection accuracy and sensitivity. Simultaneously, BBox and Object modules refine localization and object recog- nition, culminating in robust, high-performing anomaly detection framework.
trade-off exists between accuracy and real-time require- ments.
5.5. Qualitative Evaluation
Our multimodal asynchronous hybrid network is designed in modular fashion, allowing the network depth (i.e., number of ResBlocks and look-up-table Spline convolution layers) to be increased for more complex data. Experiments show that increasing the number of layers slightly improves detec- tion accuracy, with a modest increase inference latency. See Table 3.
Our model leverages multiple feature types: RGB features encode appearance, bounding box features represent object position and movement, event stream features capture rapid and abnormal motion, and object-level features describe local details. The object-level attention mechanism assigns scores to detected objects, highlighting those most relevant to anomaly detection.
As illustrated in Figure 4, when vehicle suddenly cuts in, its attention score increases as it approaches, indicating its growing importance as a potential anomaly. Visualizations of these attention scores across frames demonstrate how the model dynamically focuses on critical objects, providing insight into which features most vital for identifying anomalies. This enhanced interpretability aids in under- standing the model's decision-making process and improves its credibility and deployment safety.
Table 3. Performance and latency trade-off with different network depths on ROL.
LAYERS
AUC
AP
AUC-F
MTTA
FPS
MRESPONSE
0.879
0.570
0.736
2.80
579
1.17
0.885
0.574
0.739
2.89
312
1,31
0.892
0.577
0.740
2.93
166
1.56
These results indicate that our model is highly scalable, with only minor latency trade-offs for improved detection accuracy in increasingly complex environments.
Figure presents three challenging scenarios from the ROL dataset that highlight our model's effectiveness in detect- ing traffic anomalies critical for autonomous vehicle safety, These examples involve small objects and complex abnor- mal movements. Each example includes three rows. The top row shows Ground Truth, where normal objects appear in white and anomalies in red. The middle row displays our model's 's predictions, highlighting anomalies with scores above 0.5 in red. The bottom row compares frame-level anomaly scores to the Ground Truth, offering a straightfor- ward visual measure of model's 's accuracy.
To further enhance global feature modeling, replaced the original CNN backbone with ViT and Swin Transformer. Transformer architectures can capture long-range dependen- cies and improve detection accuracy. However, the self- attention mechanism has ((22) complexity, introducing additional inference latency. Experimental results are shown in Table 4.
Table 4, Performance comparison of different backbone architec- tures on ROL.
Lane Merger Scenario. In Figure 3(a), vehicle abruptly merges into the autonomous vehicle's 's lane. At frame 20, the anomaly score is 0.33, signifying the early onset of abnormal behavior. By frame 30, the score rises to 0.58, accurately highlighting the vehicle's unusual trajectory and intensified event flow. Here, the GRU module proves essen- tial by aggregating temporal cues, boosting the detection sensitivity as the threat unfolds.
MODEL
AUC
AP
AUC-F
MTTA
FPS
MRESPONSE
OURS(CNN)
0.879
0.570
0.736
2.80
579
1.17
CIN-VVITB
0.886
0.581
0.745
2.87
213
1.51
These results show that Transformer backbones can improve detection accuracy when latency carefully managed, but
8

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
(a) As the side vehicle initiates an abrupt lane change, a series of anomalous events is triggered, resulting in a steady rise in the anomaly score.
(b) vehicle abruptly enters the autonomous vehicle's 's field of view with an interoepting trajectory, causing sharp spike in the anomaly score.
Figure 3. In various traffic anomaly scenarios, the actions of hazardous vehicles significantly threaten the autonomous vehicle, causing elevated anomaly scores.
Frane
fully emerge in the scene. This capability significantly low- ers detection latency, which can be crucial for avoiding potential collisions.
Atiention Map
0.21
Frame
Inter-Frame
Frame
Figure 4. When a vehicle suddenly in, its attention score in- creases as it approaches, indicating its growing importance as potential anomaly.
Figure 5. In scenarios where high-speed objects suddenly emerge, continuous stream of events helps the model perform inter-frame anomaly detection, allowing for carlier and timely anomaly detection.
Oncoming Collision Risk. Figure 3(b) depicts oncom- ing vehicle veering toward the autonomous vehicle, posing a high collision risk. The model's anomaly score quickly escalates, driven by erratic movement and close proximity, which are detected by leveraging bounding box data to as- sess relative positions. This underscores the importance of precise spatial cues for robust anomaly detection.
6. Conclusion
We focus on the task real-time anomaly detection in autonomous driving, underscoring the need minimize response time without compromising detection accuracy. To address this challenge, proposed multimodal asyn- chronous hybrid network that combines event streams from event cameras with RGB image data. By employing an asynchronous GNN for high-temporal-resolttte event data and a CNN for rich spatial features, our framework captures both the temporal dynamics and spatial details of driving en- vironments. Extensive experiments on benchmark datasets demonstrate that our approach significantly surpasses isting methods in detection accuracy and response time, achieving dilliseccon-leeved responsiveness. This work lays a foundation for future research in time-critical perception tasks and advances safe, reliable deployment of autonomous vehicles.
Early Detection Advantage. Comparisons in Figure 3 indicate that our method achieves earlier anomaly detection than AM-Net (Karim et al., 2023), thereby reducing latency in critical situations. These examples confirm that our model not only covers a wide spectrum of anomalies but also reacts with temporal precision essential for real-world autonomous driving applications.
Frrae- Anomaly Detection. Figure 5 illustrates a sud- den pedestrian appearance across two consecutive frames. Harnessing the asynchronous event stream properties allows the model to detect anomalies between frames, effectively anticipating the presence of fast-moving objects before they

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Impact Statement
fic incident detection from cameras: A semi-supervised learning approach. In 2018 2 International Confer- ence on Intelligent Transportation Systems (ITSC), pp. 1840-1845. IEEE, 2018.
This paper introduces a multimodal asynchronous hybrid network for real-time anomaly detection in autonomous driving. By integrating sparse event stream data with con- ventional image-basets based inputs, the method reduces detec- tion delays, enhances response times, and improves overall safety and reliability in dynamic driving environments. Its broader impact lies in potentially reducing traffic accidents and fatalities, but it also raises ethical concerns, including job displacement, potential system failures, and privacy is- sues-particularly in urban settings where both public and private data may be collected. Ensuring responsible deploy- ment requires rigorous testing, robust safety measures, and appropriate regulatory frameworks that uphold transparency, fairness, and accountability. Future work should address these challenges and any unintended social consequences, enabling autonomous driving technology to realize its trans- formative potential safely and ethically.
Chan, F.-H., Chen, Y.-T., Xiang, Y., and Sun, M. Antici- pating accidents in dashcam videos. In Computer Vision- ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part IV 13, pp. 136-153. Springer, 2017.
Chen, Y., Nan, Z., and Xiang, T. Fbinet: Feedback loop network for driver attention prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13371-13380, 2023.
Chong, Y. S. and Tay, Y. H. Abnormal event detection in videos using spatiotemporal autoencoder. In Advances Neural Networks-ISNN 2017: 14th International Sym- posium, ISNN 2017, Sapporo, Hakodate, and Muroran, Hokkaido, Japan, June 21-26. 2017, Proceedings, Part Il 14, pp. 189-196. Springer, 2017.
Acknowledgements
Cui, A., Casas, S., Wong, K., Suo, S., and Urtasun, R. Gorela: Go relative for cowppont-iinvarian motion fore- casting. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7801-7807. IEEE, 2023.
The authors gratefully acknowledge financial support from the National Natural Science Foundation of China (Grant Nos. 62332002, 62027804, 61825101, and 62402015), the China Postdoctoral Science Foundation (Grant No. 2024M750100), and the Postdoctoral Fellowship Program of the China Postdoctoral Science Foundation (Grant No. GZB20230024). Computational resources were kindly pro- vided by Pengcheng Cloudbrain.
Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303 303-338,14 2010.
References
Fang, J., Qiao, J., Bai, J.. Yu, H., and Xue, J. Traffic acci- dent detection via self-supervised consistency learning in driving scenarios. IEEE Transactions on Intelligent Transportution Systems, 23(7):9601-9614 2022.
Bao, W., Yu, Q., and Kong, Y. Drive: Deep reinforced acci- dent anticipation with visual explanation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 7619-7628, 2021.
Fang, J., Qiao, J., Xue, J., and Li, Z. Vision-bane traf- fic accident detection and anticipation: A survey. IEEE Transactions on Circuits and Systems for Video Technol- ogy, 2023.
Batanina, E., Bekkouch, E. I., Youssry, Y., Khan, A., Khattak, A. M., and Bortnikov, M. Domain adaptation for car accident detection in videos. In 2019 ninth inter- national conference on image processing theory, tools and applications (IPTA), pp. 1 1-6. IEEE, 2019.
Fang. J., Li, I..I., Zhou, J., Xiao, J., Yu, H., Lv, C., Xue, J., and Chua, T.-S. Abductive ego-view accident video understanding for safe driving perception. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 22030-22048 2024.
Batzner, K., Heckler, L.,, and König, R. Efficientad: Accu- rate visual anomaly detection at millisecond-level laten- cies. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 128-138, 2024.
Gallego, G., Delbrück, T., Orchard, G., Bartolozzi, C., Taba, B., Censi, A., Leutnnggerr S., Davison, A. J., Conradt, J., Daniilidis, K., et al. Event-based vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154-180, 2020.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11221-11631, 2020.
Gehrig, D. and Scaramuzza, D. Low-latency automotive vision with event cameras. Nature, 629(8014):1034 1040, 2024.
Chakraborty, P., Sharma, A., and Hegde, C. Freeway traf-
10

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Gehrig, M., Aarents, W., Gehrig, D., and Scaramuzza, D. Dsec: A stereo event camera dataset for driving scenarios. IEEE Robotics and Automation Letters, 6(3):4947-4954, 2021.
Karim, M. M., Yin, Z., and Qin, R. An iitention-guident mul- tistream feature fusion network for early localization of risky traffic agents in driving videos. IEEE Transactions on Intelligent Vehicles, 9():1192-1808) 1803, 2023.
Gong, D., Liu, L., Le, V., Saha, B., Mansour, M. R., Venkatesh, S., and Hengel, A. v. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In Proceedings of the IEEE/CVF international conference computer vi- sion, pp. 1705-1714, 2019.
Le, T.-N., Ono, S., Sugimoto, A., and Kawasaki, H. Atten- tion r-enn for accident detection. In 2020 IEEE intelligent vehicles symposium (IV), pp. 313-320, IEEE, 2020.
Li, Y., Zhou, H., Yang, B., Zhang, Y., Cui, Z., Bao, H., and Zhang, G. Graph-based based asynchronous event process- ing for rapid object recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 934-943, 2021.
Gutierrez-Osorio, C. and Pedraza, C. Modern data sources and techniques for analysis and forecast of road acci- dents: A review. Journal of traffic and transportation engineering (English edition), 7(4)):32- 446, 2020.
Liang, R., Li, Y., Yi, Y., Zhou, J., and Li, X. A memory- augmented multi-task collaborative framework for un- supervised traffic accident detection in driving videos. arXiv preprint arXiv:2307. 14575, 2023.
Hanley, J. A. and McNeil,
B. J. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29-36, 1982.
Liang, R., Li, Y., Zhou, J., and Li, X. Text-driven traffic anomaly detection with temporal high-frequened model- ing in driving videos. IEEE Transactions Circuits and Systems for Video Technology, 2024.
Haris, M., Moin, M. A., Rehman, F. A., and Farhan, M. Ve- hicle crash prediction using vision. In 202/ 55th Annual Conference Information Sciences and Systems (CISS), pp. 1-5. IEEE, 2021.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Visi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Hasan, M., Choi, J., Neumann, J., Roy-Chooee Shyy-hhhu A. K., and Davis, L. S. Learning temporal regularity video sequences. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 733-742, 2016.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 770-778, 2016.
Liu, W., Luo, W., Lian, D., and Gao, S. Future frame prediction for anomaly detection-a a new baseline. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6536-6545 2018.
Hu, H., Wang. Q.. Cheng, M., and Gao, Z. Cost-sensitive semi-supervised deep learning to assess driving risk by application of naturalistic vehicle trajectories. Expert Systems with Applications, 178: 15041, 2021a.
Malla, S., Choi, C., Dwivedi, I., Choi, J. H., and Li, J. Drama: Joint risk localization and captioning in driving. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 1043-1052, 2023.
Hu, Y., Liu, S.-C., and Delbruck, v2e: From video frames to realistic dvs events. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1312-1321, 2021b.
Monjurul Karim, M., Li, Y., and Qin, R. Towards explain- able artificial intelligence (xai) for early anticipation of traffic accidents. arXiv e-prints, pp. arXiv arXiv-2109 2108, 2021.
Ristea, N.-C., Croitoru, F.-A., Ionescu, R. T., Popescu, M., Khan, F. S., Shah, M., et al. Self-distilled masked auto- encoders are efficient video anomaly detectors. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Patlern Recognition, pp. 15984-15995, 2024.
Huang, X., Cheng, X., Geng, Q. Cao, B., Zhou, D., Wang, P., Lin, Y., and Yang, R. The apolloscape dataset for au- tonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 954-960, 2018.
Rossi, L., Bernuzzi, V., Fontanini, T., Bertozzi, M., and Prati, A. Memory-augmented online video anomaly IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6590-6599 IEFF, 2024.
Karim, M. M., Li, Y., Qin, R., and Yin, Z. A dynamic spatial-temporal attention network for early anticipation of traffic accidents. IEEE Transactions on Intelligett Transportation Systems, (000)9999990000 2022.
detection. In ICASSP 2024-2022 2024

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Roy, D., Ishizaka, T., Mohan, C. K., and Fukuda, A. Detec- tion of collision-pront pariionn--ooneeee vehicle behavior intersections using siamese interaction Istm. IEEE transactions in- telligent transportation systems, 23(4):3137-3147, 2020.
Yamamoto, S., Kurashima, T., and Toda, H. Classifying near-miss traffic incidents through video, sensor, and object features. IEICE TRANSACTIONS on Information and Systems, 105(2):377-386, 2022.
Santhosh, K. K., Dogra, D. P., Roy, P. P., and Mitra, A. Ve- hicular trajectory classification and traffic anomaly detec- tion in videos using a hybrid cnn-vae architecture. IEEE Transactions on Intelligent Transportation Systems, 23 18):11891-11902 2021.
Yao, Y., Xu, M., Wang. Y., Crandall, J., and Atkins, E. M. Unsupervised traffic accident detection in first- person videos. In 2019 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), PP. 273-280. IEEE, 2019.
Shimomura, K., Hirakawa, T. Yamashita, T., and Fujiyoshi, H. Potential risk localization via weak labeling out of blind spot. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4504- 4513, 2024.
Yao, Y., Wang, X., Xu, M., Pu, Z., Wang, Y., Atkins, E., and Crandall, D. J. Dota: Unsupervised detection of traffic anomaly in driving videos. IEEE transactions on pattern analysis and machine intelligence, 45(1):444-45, 2022.
You, T. and Han, B. Traffic accident benchmark for causal- ity recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16, pp. 540-556. Springer, 2020.
Taccari, L., Sambo, F., Bravi, L., Salti, S., Sarti, L., Si- moncini, M., and Lori, A. Classification of crash and near-crash events from dashcam videos and telematics, In 2018 st International Conference intelligent trans- portation systems (ITSC), pp. 2460-2465. IEEE, 2018.
Zeng, K.-H., Chou, S.-.,, Chan, F.-H., Carlos Niebles, J., and Sun, M. Agent-centrie risk assessment: Accident anticipation and risky region localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2222-2230 2017.
Tamagusko, T., Correia, M. G., Huynh, M. A., and Ferreira, A. Deep learning applied to road accident detection with transfer learning and synthetic images. Transportation research procedia, 64:90-97, 2022.
Thakur, N., Gouripeddi, P., and Li, B. Graph (graph): A nested graph- based framework for early accident antici- pation. In Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, pp. 7533-7541, 2024.
Zeng. W., Liao, R., Urtasun, R., and Liang, M. Systems and methods for actor motion forecasting within a surround- ing environment of an autonomous vehicle, November 2 2023. US Patent App. 18/346,518.
Zhao, Y., Deng, B., Shen, C., Liu, Y., Lu, H., and Hua, X.-S. Spatio-temporal autoencoder for video anomaly detection. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1933-1941, 2017.
Tian, B., Gao, H.-a., Cui, L., Zheng. Y.. Luo, L., Wang, B., Zhi, R., Zhou, G., and Zhao, H. Latency-aware road anomaly segmentation in videos: A photorealistic dataset and new metrics. arXiv preprint arXiv:2401. 004442 2024.
Zhou, B. and Jiang, J. Deep event-hased object detec- tion in autonomous driving: A survey, arXiv preprint MxXiv:200500399. 2024.
Veeramani, B., Raymond, J. W., and Chanda, Deepsort: deep convolutional networks for sorting haploid maize seeds. BMC bioinformatics. 19:1-9, 2018.
Zhou, Z., Dong, X., Li, Z., Yu, K., Ding, C.,, and Yang, Y. Spatio-temporal feature encoding for traffic accident detection in vanet environment. IEEE Transactions on Intelligent Transportation Systems, 23(10):19772-19781 2022.
Vijay, T. K., Dogra, D. P., Choi, H., Nam, G., and Kim, I.-J. Detection of road accidents using synthetically generated puuti-pprrpeectiee accident videos. IEEE Transactions on Intelligent Transportation Systems, 2422):196 2022.
Wang, X., Su, T., Da, F., and Yang, X. Prophnet: Efficient agent-centric motion forecasting with anchor-informed proposals. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21995- 22003, 2023.
Xiong, X., Chen, L., and Liang, J. A new framework of vehicle collision prediction by combining svm and hmm. IEEE Transactions on Intelligent Transportation Systems, 1903);69--100,22 2017.
12

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
A. Ablation Studies
Table Ablation study results for different model configurations on ROL dataset, showing the effects of different components on AUC, AP, AUC-Framd mTTA, and mAP.
RGB
Event
GRU
Attention
Bbox
Object
Two-stage
AUC(%)
AP(
AUC-FFrame(S)) mTTA(s)
mAP(())
0,805
0.479
0.648
1.44
41.66
0.817
0,508
0.668
1.98
43.59
r
/
0.819
0.498
0.657
1.52
43.77
/
0.823
0.518
0.691
2.06
35.76
0.813
0.507
0.688
1.73
43.57
v
0.839
0,531
0.703
2.11
43.29
/
0.845
0.539
0.694
1.96
42.94
/
4
0.868
0.561
0.726
2.51
43.82
/
/
0.879
0.570
0.736
2,80
45.15
We conduct a comprehensive ablation study presented in Table 5, with the analysis detailed as follows:
RGB and Event Modalities. Both RGB and Event modalities are crucial for the model's performance. RGB features capture the static appearance of objects, while Event features extract dynamic motion and anomaly- related characteristics. As illustrated in Figure 6, objects exhibiting abrupt and irregular motions generate significant number of anomalous events. In the fourth and seventh groups of the ablation study, the exclusion of RGB and Event inputs, respectively, led to performance degradation. Specifically, omitting RGB features caused a substantial decline in anomaly detection accuracy and mean Average Precision (mAP) for object detection, as object detection heavily relies on appearance information. Similarly, removing Event inputs reduced anomaly detection accuracy and the Time to Anomaly (TTA) metric, since the Event stream provides essential inter-frame information that enables the model to detect anomalies more promptly. These results highlight the complementary nature of the two modalities: RGB features provide rich static appearance information, while Event streams capture dynamic motion information, thereby enhancing the sensitivity of anomaly detection.
Two-Stage Single-Stage Architecture. The eighth and ninth experiments compare the model's performance using two-stage versus a single-stage architecture. In the two-stage architecture, Event and RGB features are first utilized for object detection, and the resulting detections serve as priors for anomaly detection. In contrast, the single-stage architecture directly leverages fused Event and RGB features for both tasks simultancously. The superior performance of the single-stage model can be attributed to its shared feature extraction module, which allows anomaly detection to benefit directly from features optimized for object detection, thereby enhancing sensitivity to anomalous behaviors.
Bounding Box (BBox) Priors. The fifth and sixth experiments investigate the impact of using BBox priors for anomaly detection. BBox priors provide critical information about the relative positions of objects concerning the autonomous vehicle, which is particularly valuable for detecting anomalies. As shown in Figure 8(b), irregular BBox movements often serve as early indicators of anomalies. For instance, when a vehicle abruptly merges into the lane of the autonomous vehicle, this irregularity can be anticipated from the temporal movement of the BBox. The absence of BBox priors led to decreased anomaly detection accuracy. Unlike the DoTA (Yao et al., 2022) framework, which bases anomaly detection on predicting future object trajectories and focuses on irregular BBox movements, our method employs a Gated Recurrent Unit (GRU) capture the temporal sequence of BBox movements, leveraging this information for anomaly detection. The inclusion of BBox priors enhances anomaly detection by providing essential positional information, and their absence negatively impacts performance, underscoring their significance.
GRU and Attention Modules. The second and third experiments assess the importance of the GRU and attention modules within the anomaly detection framework. Compared to the ninth experiment, removing either the GRU or the attention module resulted in a significant decrease in anomaly detection accuracy. The GRU is pivotal for modeling temporal information, as most anomalies in the ROL and DoTA datasets develop over time. Anomalies are often characterized by gradually accumulating abnormal features rather than sudden appearances. For example, depicted in Figure 9(c), merging vehicle requires approximately 20 frames to transition into the lane of the autonomous vehicle. During this period, the GRU incrementally aggregates anomaly features, resulting in a continuously increasing anomaly score. This demonstrates the GRU's s role in modeling the temporal accumulation of anomaly features.
While the GRU captures temporal anomaly accumulation, the attention module focuses on spatial anomalies at each time step. assigns higher attention weights to anomalous objects the object level, enabling the model to prioritize these anomalies
13

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
4.3
Framne 140
Fraiie I750
Score: 0.54
P30
Frame 1E0
350
Anomaly Score
0.8
Bsodiee- otsenic
0.6
0.4
0.2
0.0
Frame
Figure 6. In scenarios where high-speed objects suddenly emerge, continuous stream of events helps the model perform framm anomaly detection, allowing earlier more timely anomaly detection.
and assign them higher anomaly scores. Together, the GRU and attention modules provide complementary strengths: the GRU tracks temporal patterns, and the attention module enhances the spatial prioritization of anomalous objects. Ablation studies show that incorporating the attention module significantly improves anomaly detection performance, confirming its critical role enhancing detection accuracy. Figure 6 illustrates the interaction between the GRU and attention modules. The GRU facilitates faster accumulation of anomaly scores, allowing for earlier detection, while the attention module enables more accurate anomaly judgments. In contrast, baseline anomaly score curve without GRU and attention exhibits greater fluctuations.
Overall, the ablation analysis underscores the importance of multimodal inputs (RGB and Event), BBox priors, and temporal- spatial mechanisms (GRU and attention) for effective anomaly detection. These findings validate that each component plays a vital role in enhancing the model's performance, particularly in dynamic environments where both temporal and spatial anomaly features are essential.
Table Ablation study results for different two-stage model configurations ROL dataset, showing the effects of different components on AUC, AP, AUC-Frame, mTTA.
No.
RGB
BBox
Event(asyn.)
Event(sync.)
Flow
GRU
Attention
AUC(%)
AP(%))
AUC-Framc($))
mTTA(s)
1
0.855
0.559
0.702
2.18
2
0.840
0.544
0.698
2.25
3
0.851
0.542
0.705
2.50
0.559
0.713
2.48
5
V
0.868
0.561
0.726
2.51
B. Two-Stage versus Single-Stage Networks
The primary difference between two-stage and single-stage networks lies in their operational flow. Two-stage networks initially perform object detection and subsequently conduct anomaly detection without reusing the features extracted during
14

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
the object detection phase. Table 6 summarizes the impact of various components the performance of the two-stage model.
Experiment 1. In the first experiment, only bounding box (BBox) features and optical flow were utilized as input modalities, with Gated Recurrent Unit (GRU) module responsible for temporal feature extraction. This configuration achieved high anomaly detection score but exhibited a lower mean Time to Anomaly (mTTA) due to the synchronous nature optical flow. These results indicate that while optical flow effectively captures motion-related features, its performance is constrained by its reliance on synchronized data.
Experiments 2 and 3. In the second and third experiments, optical flow was replaced with synchronous and asynchronous event streams, respectively. Synchronous event streams were generated by aggregating events within fixed time windows into event frames, which were then processed similarly to optical flow. However, this synchronous configuration resulted in the lowest Area Under the Curve (AUC) and mTTA, as it negated the inherent asynchronous properties of event streams. Conversely, asynchronous event streams, processed using an asynchronous Graph Neural Network (GNN), significantly outperformed synchronous streams by leveraging motion-related information essential for anomaly detection. The asynchronous approach maintained the temporal granularity of raw events, which is critical for identifying subtle anomalies.
The fourth experiment introduced an attention mechanism to focus on critical features, leading to further improvements in detection performance. The attention module enhanced the model's ability to identify anomalies by selectively weighting more relevant object-level features. This underscores the importance of prioritizing specific regions and features, especially in complex anomaly detection scenarios.
Experiment 4.
Experiment 5. In the fifth experiment, RGB features were incorporated alongside BBox and asynchronous event streams, resulting in the highest overall performance improvement. Although RGB features had limited direct impact on anomaly detection, they indirectly enhanced performance by improving object detection accuracy. This demonstrates that RGB features, while not directly influential in anomaly detection, play a supportive role by providing more accurate object localization and detection in the initial stage.
Overall, these findings highlight the effectiivnness of asynchronous event streams and attention mechanisms in enhancing two-stage networks. Additionally, they acknowledge the indirect benefits of RGB features through improved object detection, thereby contributing more robust anomaly detection performance.
Scoret
Frame #14
Frame #14
Frame #15
Figure 7. Examples from the Rush-Out dataset demonstrate the effectiveness of our approach. At a tunnel exit under intense backlighting. vehicle suddenly emerges from the tunnel edge. Leveraging the advantages of event data, our method enables rapid and accurate anomaly detection in such challenging scenarios.
C. New Datasets
The ROL (Karim et al., 2023) and DoTA (Yao al., 2022) datasets encompass diverse range of traffic anomalies, including scenarios both with and without the presence of autonomous vehicles. To specifically evaluate the low-latency capabilities of our model, we identified and selected scenarios that best highlight this aspect. These scenarios are characterized by the sudden appearance of pedestrians or vehicles, posing significant risks to autonomous vehicles. Such abrupt occurrences typically result from the occlusion by large objects or the autonomous vehicle's 's narrow field of view, causing fast-moving entities to unexpectedly enter the vehicle's 'ss sight and create hazardous situations. In these contexts, the low response latency of our method particularly advantageous.
15

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Table 7. Comparison of the proposed model with existing methods on the Rush-Out test dataset
Method
AUC(%)
^
AUC-Framm %))
(T)
mResponse(s))
AB et al., 2016)
0.789
0.502
0.612
1.03
1.88
ConLLSTMMAE(Chog & Tay, 2017)
0.738
0,490
0.603
1.05
2.05
AnoPred(LLiu et al., 2018)
0.790
0.521
0.616
1.14
1.92
FOL-IoU(Yao et al., 2022)
0.823
0.543
0.666
1.30
1.61
FOL-Mask(Yao et al., 2022)
0.835
0.553
0.681
1.35
1.54
FOL-STD(Yao al., 2022)
0.845
0.561
0.689
1.32
1.48
FOL--nnemmlee Yao al., 2022)
0.870
0.572
0.698
1.45
1.62
AM-Net(Karim al., 2023)
0.879
0.579
0.710
1.48
1.55
OURs
0.887
0.592
0.755
1.71
1.29
To facilitate this evaluation, curated the Rush-out' dataset by extracting instances of dangerous driving scenarios involving sudden outbursts of objects from the ROL and DoTA datasets. The Rush-out dataset comprises 1,084 videos, each recorded at 20 frames per second (fps) with resolution of 1280 x 720. Table 7 presents performance comparison on the Rush-out dataset using the same baseline methods referenced in Table 5. Our proposed method surpasses the state-of-the-art (SOTA) by 0.23 seconds in the mean Time to Anomaly (mTTA) metric and achieves a 0e--ood improvement in the mean Response Time (mResponse) metric.
The Rush-out dataset predominantly features high-speed scenes where even single frame can result in significant displacement of anomalous vehicle, thereby increasing the associated risk. Despite the heightened complexity of these scenarios, our method consistently demonstrates superior performance in anomaly detection metrics.
Figure 7 and Figure 8 illustrate the results obtained on the Rush-out dataset. Specifically, Figure 8 depicts a scenario where a vehicle abruptly emerges from the edge of a tunnel exit under strong backlighting conditions. The intense lighting induces significant visual blurring, rendering traditional detection methods less effective for fast-moving objects. Our approach leverages event streams, which are particularly robust under extreme lighting conditions, highlighting the unique advantage of event cameras in complementing conventional RGB imagery. By utilizing this modality, our method reliably detects rapidly emerging vehicles across consecutive frames, enabling early anomaly detection and timely alerts in such challenging environments.
D. Qualitative Evaluation
Our model prioritizes rapid inference speed and minimal anomaly detection latency, which are crucial scenarios where fast-moving objects suddenly appear. Figure 8 presents typical instances pedestrians or vehicles emerging unexpectedly, By leveraging continuous event streams between consecutive video frames, our model performs inter-frame anomaly detection, thereby effectively reducing detection latency.
Figure 8(a) illustrates case where boy suddenly runs out from behind a truck. In this scenario, the boy enters the field of view mid-road due to a blind spot. Although the boy is not detected in the first frame, the continuous event stream between the first and second frames enables the model to detect the boy in advance, assigning high anomaly score and significantly reducing detection latency.
Similarly, Figure 8(b) and Figure 8(c) depict vehicles abruptly emerging from the side. Due their high speed, the model utilizes event streams between frames for inter -nter-frame anomaly detection, allowing for earlier identification of anomalies and further reduction in detection latency.
Figure 9 showcases additional results of real-time traffic anomaly detection, where most cases involve abnormal vehicles or pedestrians obstructing the autonomous vehicle's path, leading to traffic anomalies.
In Figure (a), turning vehicle increasingly approaches the autonomous vehicle, resulting in elevated anomaly scores. The relative position between the abnormal vehicle and the autonomous vehicle is key factor in this detection.
16

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
HALL
(a) The boy suddenly rushed out from behind the truck on the left.
Curkee
IS. Destee
Chedee
Neure:
Scorraade
Stor
Scores
(b) A pickup truck suddenly rushed out from the right side of the field of vision.
9
FOUR
zortal
cope: 0.86
Dita chang Ching
lis Ourrorre
(c) A pickup truck suddenly rushed from the right side the field vision.
Figure 8. The anomaly detection results of suddenly rushing of the scene, the objects moving at high speed in consecutive frames can use the characteristics of event streams to help detect abnormalities hetween two frames.
Figure 9(b) shows cyclist suddenly changing lanes, creating collision risk with the autonomous vehicle. The cyclist's irregular motion generates high anomaly scores, these irregular movements produce numerous anomalous event streams that are central to anomaly detection.
Finally, Figure 9() depicts scenario where normally moving vehicle on the right abruptly changes lanes, posing collision risk with the autonomous vehicle. The vehicle's irregular lane change, combined with its decreasing distance from the autonomous vehicle, the primary factor for anomaly detection.
E. Failure Cases and Limitations
Our system's performance relies heavily the accuracy of the object detector. As shown in Figure 10, in Frame 25 and Frame 30, blurry images and small objects led to complete detection failure, which in turn caused the anomaly detection framework to fail. Although an object was detected in Frame 35 when it was very close to the ego vehicle, the attention score was only 0.56 due to missing previous detections, preventing the GRU from building a robust temporal feature set. These cases highlight that our approach is limited by the reliability of the object detector, and improvements in detection accuracy are essential.
17

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Frame #1
Frame #20
Frame #25
Frame #30
Framé #1S
Frame #20
Frame #25
Frame #30
(a) The vehicle ahead on the left makes a turn, progressively reducing its distance to the autonomous vehicle, posing significant risk.
OUT CHASPISIIE
GAT
SAR OPAS
CRAS
Frame #64
Frame #69
Frame #7
Frame #79
arees nae die Seasnss
evco
CRAS
GRACHES
Frame #64
ince yaer
Frame #69
sihus
Frame #74
Frame #79
(b) A eyclist on the left changes lanes unexpectedly, creating high risk of collision with the autonomous vehicle, which ultimately results crash.
CAPCCAGIES
CRUCHES
Frame H20
Frame 430
Frame /40
Frame #45
Eas ISA
lh Ganonaaltts T6C
cab GRAGI
Scarve (199
Stare: 8.45
Frame #20
Frame #30
Frame #40
Frame #45
(c) A vehicle on the right, initially driving normally, suddenly changes lanes, decreasing its distance to the autonomous vehicle and leading to high anomaly risk.
Figure 9. Additional traffic anomaly scenarios for real-time anomaly detection primarily involve abnormal vehicles obstructing the autonomous vehicle's path, posing significant risks its operation.
18

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
Frame 25
Frame 30
Frame 35
Frame
Attention Map
28h0t 3011
Value None
Value None
Value 0.56
Figure 10. Our approach limited the reliability of the object detector, and improvements in detection accuracy essential.
F. Sensitivity Analysis
Our model remains robust across different settings. At the standard 0.5 IOU and 0.5 confidence thresholds, the model achieves an AUC of 0.879. As shown in Figure 11, increasing these thresholds sharply reduces AUC by filtering out more detection boxes, while lowering them causes gradual decline false positives are effectively down-weighted. Overall, the AUC ranges from 0.65 0.879, confirming the stability of our configuration.
Impact of Confidence and loU Thresholds on AUC
0.715
0.742
0.721
0.012
044
0.751
0.742
Threshold
0.5
0.768
0.842
0.879
0.852
0.771
AUC
0.75
06
0.692
0.721
0.735
0,082
0.653
Figure 11. Sensitivity analysis of the model's hyperparameters IOU threshold and target detection confidence threshold,
19

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
G. Real-Time Anomaly Detection Methods
We compared our approach with several recent real-time video anomaly detection methods, including AED-MAE (Ristea al., 2024), EfficientAD (Batzner et al., 2024), and MOVAD (Rossi et al., 2024). AED-MAE is state-of-the of-the-art method for surveillance videos that emphasizes detection speed, but relies on static background modeling, making it less suitable for dynamic traffic environments. EfficientAD is designed for industrial anomaly detection and also operates at the frame level. MOVAD is the first online real-time anomaly detection method specifically for autonomous driving. Our approach achieves the best balance between accuracy and speed, meeting the unique requirements of the traffic domain. Results on ROL and DoTA are shown in Table 8.
Table Comparison with recent real-time anomaly detection methods on ROL and DoTA.
METHOD
TYPE
AUC-FRAME( )t
MTTA(S))
MRESPONSE(S)
FPST
DATASETS
ROL
DOTA
ROL
DOTA
ROL
DOTA
EFFICIENTAD AED-MAE
FRAME
0.519
0,549
0.89
0.97
3.65
2.68
557
FRAME
0.571
0.652
1.01
1.35
3.36
2.79
1655
MOVAD
FRAME
0.719
0.821
2.47
2.55
2.61
1.33
158
OURS
OBJECT
0.736
0.823
2.80
2.78
2.35
1.21
579
H. Synthetic Event Data and Validation
Currently, mainstream traffic anomaly detection datasets do not contain real event (DVS) modality. To address this, we employ the V2E (Video to Event) method to convert conventional video data into event streams, thereby supplementing the event modality input. V2E effectively simulates core DVS sensor characteristics, including Gaussian threshold distribution, temporal noise, leak events, and intensity-dependee 101, where V2E- ge-eretee data raised ResNet34 accuracy on real DVS from 81.69% to 83.36%, and up 87.85% 85% after fine-tuning (compared to 36.74% using real data only).
bandwidth. Its fidelity has been validated datasets such N-Caltech
Although the DSEC (Gehrig et al., 2021) dataset contains real event data in autonomous driving scenarios, it only covers normal driving and lacks anomaly events, making it unsuitable for direct use in anomaly detection tasks. To further validate effectiveness of V2E-generated data, we generated a simulated DSEC+V2F dataset and conducted experiments on ROL and DoTA. The results show that using DSEC+V2E versus DSEC alone leads minimal changes in anomaly detection performance (AUC, AP, mTTA, etc.), confirming that the V2E method can effectively supplement the event modality and maintain model robustness. See Table 9.
Table 9. Performance comparison of DSEC and DSEC+V2E the ROL and DoTA test
METRICS
AUC(%))
AUC-FRAME % f
MTTA(S)1
MRESPONSE(S)L
DATASET
ROL
DOTA
ROL
DOTA
ROL
DOTA
ROL
DOTA
DSEC
0.841
0.857
0.697
0.794
2.24
2.18
1.66
1.79
D2E
0.846
0.862
0.712
0.808
2.46
2.37
1.45
1.61
In summary, despite inherent differences between synthetic and real event data, our systematic experiments demonstrate the effectiveness of V2E and the robustness of our detection model, laying the groundwork for future real event-based anomaly detection datasets.
I. Deployment Feasibility and Hardware Adaptation
Our method can be deployed end-to-end mainstream autonomous driving chips. For example, Orin chips, the system can process approximately 560k events per second under normal load, and up to 10M events/second at peak, with a total computational cost of about 87.32 TFLOPs and 42W power consumption, which is within the chip's capability. Although two cameras (RGB Event) are used, hardware synchronization limits timing errors to 78 microseconds, and the event camera itself introduces only about 6ms delay, meeting real-time requirements. For resource-constraing chips, quantization and other optimizations can further improve deployment efficiency.
20

---

When Every Millisecond Counts: Real-Time Anomaly Detection via the Multimodal Asynchronous Hybrid Network
J. Computational A Analysis
Our method requires only 8.732 MFLOPs per event and uses 23.5 GB of video memory. Under typical conditions ROL dataset, with an average event of 560k events per second, this overhead manageable. high-speed driving scenarios-where the event rate can rise 1-10M events/s-the worst-case computational load is approximately 87.32
TFLOPs.
Current autonomous driving chips such as Atlan, Thor, and Orin can fully deploy our algorithm on-board, while chips like Xavier and Parker, although sufficient for normal driving, might face challenges under extreme conditions. The specifications of several representative chips are summarized Table 10.
Table 10. Representative autonomous driving chip specifications.
CHIP
ARCHITECTURE
COMPUTE POWER
PARKER
2eDEVRER
TOPS
XAVIER
4XARM CORTEX A578
30 TOPS
ORIN
NVIDIA CUSTOM CARMEL
254 TOPS
ATLAN
ARM6412 ARM CORTEX-A78
1000 TOPS
THOR
AE (HERCULES) ARM NEOVERSE v2 2000 TFLOPS FP8
K. Extreme scenarios
We collected data on severe weather and low-light scenes from the ROL and DoTA datasets, and performed comparative experiments with the two best-perfoming baseline methods. Benefiting from the advantages of event cameras in extreme lighting scenarios, our method outperforms other methods by large margin in these scenarios. See Table 11.
Table 11. Performance comparison in adverse weather or low-light scenarios.
METHOD
AUC(%)
AP(% )
AUC-FRAME( % f
MTTA(S)f
MRESPONSE(S)L
STFE
0.631
0.397
0.582
1.26
3.12
TTHF
0.652
0.416
0.594
1.31
2.98
OURS
0.719
0.442
0.612
1.47
2.35
21


Reviews:
[{"review_id": "WYN2eLrTIm", "reviewer": "Reviewer_rGL6", "summary": "This paper focuses on real-time anomaly detection tasks in autonomous driving, aiming to balance detection accuracy and response time. The core algorithm is a multimodal asynchronous hybrid network that integrates event streams from event cameras with RGB camera image data. Process event streams through asynchronous graph neural networks (GNNs) and utilize their high temporal resolution; Extracting spatial features from RGB images using Convolutional Neural Networks (CNN). The combination of the two can capture spatiotemporal information of the driving environment, achieving fast and accurate anomaly detection.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your motivating review and concrete suggestions. Detailed responses to your questions are listed as follows.\n\n> **Weakness**: V2E Transformation vs. Real Event Stream\n\n(1)The current traffic anomaly detection dataset does not have a real Event mode. Our subsequent work is to use event cameras to collect this data. \n(2)V2E effectively mimics key DVS sensor traits (Gaussian Threshold Distribution, Temporal Noise, Leak Events, Intensity-dependent Bandwidth). For example, on N-Caltech 101, using V2E data raised ResNet34 accuracy on real DVS from 81.69% to 83.36%, reaching 87.85% with fine-tuning (vs. 86.74% with real data alone). \n(3)DSEC[1] is an autonomous driving dataset and contains real Event modalities, but since it is a completely normal driving dataset, it cannot be used for our anomaly detection. In order to verify the validity of the data generated by V2E, We extended our evaluation using the DSEC dataset by generating a simulated dataset, **DSEC+V2E**. Training on DSEC for normal samples and testing on ROL and DoTA revealed only minor variations in anomaly detection accuracy, confirming the robustness of our model when incorporating V2E-generated event data. While there are the inherent differences between synthesized and real event data, our results clearly demonstrate that the V2E method effectively supplements event modality and maintains the robustness of the detection model. For more V2E issues, please refer to the Q1 in Reviewer WT2W.\n\n|Dataset|DSEC (mAP)|ROL(AUC)|DoTA (AUC)|ROL (AUC-F)|DoTA (AUC-F)|ROL (mTTA)|DoTA (mTTA)|ROL (mR)|DoTA(mR)|\n|---|---|---|---|---|---|---|---|---|---|\n|DSEC|41.9|0.841|0.857|0.697|0.794|2.24|2.18|1.66|1.79|\n|DSEC+V2E|44.2|0.846|0.862|0.712|0.808|2.46|2.37|1.45|1.61|\n\n\n> **Q1**: Model Scalability and Real-Time Performance\n\n(1)The ROL and DoTA datasets already include highly complex scenarios—extreme weather, nighttime conditions, intense lighting, and diverse camera perspectives. Despite these challenges, our model maintains excellent real-time performance and accuracy.    (2)Our multimodal asynchronous hybrid network is designed modularly, allowing us to increase its depth (i.e., number of layers) to handle even more complex data. While deeper networks offer slight improvements in accuracy, they also introduce a small increase in processing delay. \n(3)Our experiments on the ROL dataset demonstrate the trade-off between accuracy and latency. In summary, our model’s modular architecture enables it to effectively adapt to increasingly complex environments with only a minor trade-off between improved accuracy and additional computational delay. (Layers contains a ResBlock and a look-up-table Spline convolution)\n\n|Layers|AUC(%)|AP(%)|AUC-Frame(%)|mTTA(s)|FPS|mResponse(s)|\n|---|---|---|---|---|---|---|\n|4|0.879|0.570|0.736|2.80|579|1.17|\n|5|0.885|0.574|0.739|2.89|312|1.31|\n|6|0.892|0.577|0.740|2.93|166|1.56|\n\n\n> **Q2**: Transformer-based Architectures\n\nReplacing the CNN backbone with a ViT or Swin Transformer enables global feature modeling and boosts detection accuracy by capturing long-range dependencies. However, Transformers add latency; for example, ViT’s self-attention has O(N²) complexity. Our experiments confirm that Transformer-based architectures improve accuracy when latency is managed.\n\n|Model|AUC (%)|AP (%)|AUC-Frame (%)|mTTA (s)|FPS|mResponse (s)|\n|---|---|---|---|---|---|---|\n|Ours(CNN)|0.879|0.570|0.736|2.80|579|1.17|\n|CNN→Swin|0.881|0.576|0.739|2.85|278|1.44|\n|CNN→ViT-B|0.886|0.581|0.745|2.87|213|1.51|\n\n> **Q3**: Real-world Deployment\n\nAfter calculation, our model can be fully deployed on most autonomous driving chips, and a small number of autonomous driving chips can be deployed through quantization.\n(1)Computing resource limitations: For instance, under normal conditions the system handles 560k events per second [2], and even at peak loads (up to 10M events/second), the overhead (~87.32 TFLOPs, 42W) is manageable on chips like Orin. \n(2)Although we need two camera devices, we can ignore the transmission delay issue. Hardware synchronization limits timing errors to ~78 microseconds, while event camera delays are around 6 ms. \nFor more detailed deployment information, please refer to the response to Experimental 2 in Reviewer WT2W.\n\n> **Q4**: Different Vehicle Types or Camera Configurations\n\nOur training data was collected from a diverse range of vehicles, including both sedans and trucks. Because our method is designed around an object-level approach that centers on the target detector rather than the camera or vehicle type, it naturally generalizes across different vehicle configurations. Actually, we used 20fps RGB camera data. However, the system can easily benefit from higher frame rate cameras in real-world deployments, potentially further enhancing detection performance without compromising the model's scalability.\n\n**References**\n\n[1] DSEC: a stereo event camera dataset for driving scenarios  \n\n[2] Prophesee Evaluation Kit - 2 HD", "claims_and_evidence": "The claims in the submitted materials are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria proposed in this paper are of great significance for the problem and application of real-time anomaly detection in autonomous driving.", "theoretical_claims": "This paper does not involve theoretical proof.", "experimental_designs_or_analyses": "The experimental design is rigorous, and multiple advanced methods are selected as baselines for comparative experiments, tested on different datasets to ensure the reliability of the results. The ablation experiment analyzes the impact of different modules on performance by gradually removing them, which helps to understand the role of each part of the model.", "relation_to_broader_scientific_literature": "In the field of autonomous driving anomaly detection, existing methods mainly focus on detection accuracy and ignore response time. In this paper, the response time is innovatively incorporated into the performance index. The proposed multimode asynchronous hybrid network is an improvement on the traditional method that only relies on a single mode or complex neural network, which can ensure the accuracy while greatly improving the response speed.", "essential_references_not_discussed": "There are no significant missing key references.", "other_strengths_and_weaknesses": "Strength:\n1. This paper has strong innovation in methodology, with novel multimodal fusion and asynchronous network design.\n2. This paper conducts thorough experiments, verifies multiple datasets and indicators, and conducts in-depth analysis of the model through ablation experiments.\n3. This paper incorporates response time as a core performance indicator, effectively addressing the shortcomings of existing methods in time sensitive scenarios. The research results have strong practical application value.\n\nWeakness:\nThe ROL and DoTA datasets used lack event modality. Although V2E transformation technology is used to generate supplements, there are differences between the generated data and the real event stream, which may not fully reflect the characteristics of the real scene.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The model adopts a multimodal asynchronous hybrid network with a complex structure. As the amount of input data increases or the complexity of the scene increases, the computational complexity of the model may significantly increase. Can the model be extended to adapt to more complex scenarios while ensuring real-time performance?\n\nCould Transformer-based architectures further improve detection accuracy? For example, Transformer with linear attention.\n\nWhat are the limitations of real-world deployment?\n\nHow does the model scale to different vehicle types or camera configurations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vNlx89JRXh", "reviewer": "Reviewer_WT2W", "summary": "This paper introduces a multimodal asynchronous hybrid network designed for real-time anomaly detection in autonomous driving. The main contribution lies in combining high-temporal-resolution event stream data captured via event cameras processed asynchronously using GNN and spatial information extracted from RGB images using CNNs. This integration leverages the strengths of both modalities-temporal responsiveness from event streams and spatial detail from RGB data-enabling high-accuracy anomaly detection with exceptionally low response times (millisecond-level). Extensive experimentation on benchmark datasets (ROL and DoTA) demonstrates superior performance in accuracy and significantly reduced latency compared to existing methods, validating the practical efficacy of the approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful feedback and practical recommendations. Detailed responses to your questions are listed as follows.\n\n> **Methods And Evaluation Criteria 3**: Two-stage Contribution\n\n(1)Our two-stage model first performs object detection to generate bounding boxes, providing precise localization and object-level features for subsequent anomaly detection. In complex traffic scenarios—such as multi-vehicle interactions or crowded pedestrian areas—this initial detection stage enhances object recognition accuracy. (2)For example, when a vehicle changes lanes suddenly, the bounding box prior helps quickly lock onto the anomalous object, reducing false detections. (3)Moreover, in fast-moving or low-light scenarios where RGB images suffer from motion blur or poor illumination, asynchronous event streams capture fine luminance changes to compensate. This dual-modality support boosts detection robustness, as shown in our experiments. (4)In contrast, while a single-stage model can meet millisecond-level response requirements by sharing features and reducing redundant computation (achieving a higher FPS), it may sacrifice some accuracy. Thus, the two-stage model trades off increased latency for improved detection accuracy, making it well-suited for scenarios that demand high reliability, such as highway cruising.\n\n> **Experimental 2**: Computational Overhead\n\n(1)Our method requires only 8.732 MFLOPs per event, uses 23.5 GB of video memory. Under typical conditions in the ROL dataset, with an average event rate of 560k EV/s, this overhead is manageable. In high-speed driving scenarios—where the event rate can rise to 1–10M EV/s—the worst-case computational load is approximately 87.32 TFLOPs. (2)Current autonomous driving chips such as Atlan, Thor, and Orin can fully deploy our algorithm on-board, while chips like Xaiver and Parker, although sufficient for normal driving, might face challenges under extreme conditions. The specifications of some representative chips are as follows:\n\n|Chip|Architecture|Compute Power\n|-|-|-|\n|Parker|2×Denver|1 TOPS|\n|Xaiver|4×ARM Cortex A578|30TOPS|\n|Orin|Nvidia custom Carmel|254 TOPS|\n|Atlan|ARM6412 × Arm Cortex‑A78|1000 TOPS|\n|Thor|AE (Hercules) ARM Neoverse v2|2000 TFLOPS @ FP8|\n\n\n> **References**: Related Work\n\nWe acknowledge the importance of recent advances in multimodal and asynchronous processing outside the driving anomaly detection domain and we will cite these two articles in our paper. (1)For instance, [1] proposes a Recurrent Asynchronous Multimodal Network (RAM Net) for monocular depth estimation by combining event cameras and RGB cameras. This approach leverages ConvGRU units to support asynchronous inputs, preserving the high temporal resolution of event data while fusing rich spatial information, which enhances prediction accuracy in dynamic scenes. (2)Similarly, [2] introduces an Asynchronous Event Graph Neural Network (AEGNN) that efficiently processes sparse event data by performing local asynchronous updates within a spatiotemporal graph, significantly reducing computational overhead. \n\n> **Q1**: synthetic event data generation affect \n\n(1)The current traffic anomaly detection dataset does not have a real Event mode. Our subsequent work is to use event cameras to collect this data.(2)V2E addresses this issue by considering various types of noise. **First**, it introduces temporal noise by utilizing Poisson noise to generate physically consistent noise events. **Second**, it simulates shot noise, which arises from photon statistics under low-light conditions in event cameras. **Third**, it accounts for threshold mismatch, where event trigger thresholds vary across different pixels in real event cameras, by modeling this phenomenon with a Gaussian distribution. For experimental verification of V2E, please refer to the response to Weakness in Reviewer rGL6.\n\n> **Q2**: Explainability of Abnormal Events\n\n(1)RGB features correspond to appearance features, BBox features correspond to position and movement features, event stream features correspond to features of rapid and abnormal movement of objects, and object-level features are local detailed features. (2)Our model's object-level attention mechanism assigns scores to detected objects, highlighting those more relevant to anomaly detection. As shown in [Figure R2-1](https://anonymous.4open.science/r/RTAD-3B3C/R2-1.png), when a vehicle suddenly cuts in, its attention score increases as it approaches, indicating its growing importance as a potential anomaly. Visualizations of these attention scores across frames illustrate how the model dynamically focuses on critical objects, providing insights into the features it considers vital for identifying anomalies. This enhanced interpretability aids in understanding the model's decision-making process, thereby improving its credibility and deployment security.", "claims_and_evidence": "- The claims regarding the model's capability to achieve high accuracy and minimal response times through multimodal asynchronous data integration are clearly supported by extensive experiments and ablation studies.\n- The empirical evidence presented, particularly in Table 1 and Table 2, convincingly supports the model's superiority in accuracy (AUC, AP) and response time metrics compared to existing state-of-the-art methods.\n- The paper employs a v2e method to convert traditional video into event stream data. This conversion may not fully capture the noise characteristics and the specific data format of real event cameras, potentially affecting the model's generalization performance.", "methods_and_evaluation_criteria": "- The proposed integration of RGB images with asynchronous event data via an asynchronous GNN and CNN is novel, effectively utilizing the complementary strengths of both modalities.\n- The evaluation metrics employed (AUC, AP, AUC-Frame, mTTA, mResponse) are comprehensive and well-suited to reflect both accuracy and real-time performance.\n- The paper might benefit from additional insights or a more detailed discussion of why two-stage significantly differ in contribution across scenarios, which was evident in the results.", "theoretical_claims": "- The theoretical claim regarding the efficiency and latency benefits provided by asynchronous event streams and the sharpness-aware features of the proposed network is well-supported by experimental validation.\n- Since the v2e conversion is theoretically prone to errors, it would be beneficial to analyze how these errors affect the overall results.", "experimental_designs_or_analyses": "- Experimental designs are thorough, carefully constructed, and address key practical and theoretical considerations relevant to real-time anomaly detection.\n- Although the method shows an advantage in inference speed, a more detailed analysis of computational overhead (e.g., FLOPs, GPU/CPU runtime) is recommended. Additionally, evaluating its feasibility on low-power devices would provide further insights into its practical deployment.", "relation_to_broader_scientific_literature": "The paper effectively contributes to broader fields, particularly autonomous driving and safety-critical real-time applications, by providing a novel, easily adaptable model that emphasizes response time.", "essential_references_not_discussed": "While comprehensive, authors might also benefit from discussing recent advances in multimodal and asynchronous processing models outside the driving anomaly detection domain.\n[1] Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction\n[2] AEGNN: Asynchronous Event-based Graph Neural Networks", "other_strengths_and_weaknesses": "The clarity of the paper is excellent, and its originality stems from a meaningful integration of event-based cameras with conventional CNN processing, addressing a practical need for low latency in autonomous systems.", "other_comments_or_suggestions": "The paper does not discuss in depth the explainability of abnormal events. For example, which features are the most critical? Is it possible to visually analyze how the model determines anomalies? This is crucial to improving credibility and deployment security.", "questions_for_authors": "- Could you elaborate on how errors or artifacts introduced by synthetic event data generation affect anomaly detection accuracy in real-world applications?\n- Have you explored the interpretability of the model, specifically regarding visual explanations or attention maps, and how anomalies are identified?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DlPlkguJm", "reviewer": "Reviewer_M9QQ", "summary": "The paper presents a real-time anomaly detection framework for autonomous driving through a novel multimodal asynchronous hybrid network. The method integrates high-temporal resolution event data from event cameras with spatially rich RGB images processed by a CNN, combined with an asynchronous GNN. Temporal dependencies are captured via a GRU enhanced with an attention mechanism, which enables rapid and accurate anomaly detection. Experiments on the ROL and DoTA datasets show that the proposed approach outperforms current methods in both detection accuracy (e.g., improvements in AUC, AP) and responsiveness (nearly millisecond-level inference at approximately 600 FPS).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Claims 3**: V2E Transformation\n\nPlease refer to the response to Weakness in Reviewer rGL6 and Q1 in Reviewer WT2W.\n\n> **Exp 3**: Inference Speed\n\nWe evaluated inference speed on different platforms: RTX3090, RTX4090, and A100-80G. The RTX4090 achieved the fastest speed at 603 FPS, followed by the A100-80G at 579 FPS and the RTX3090 at 517 FPS. These modest differences demonstrate that our approach consistently delivers real-time performance across various hardware configurations.\n\n> **Exp 4**: Ablation Studies\n\nOur ablation studies show that both the GRU and attention modules are critical. The GRU captures long-term dependencies to improve temporal feature aggregation and stability, raising the mTTA from 1.44 to 1.98 seconds. Meanwhile, the attention module focuses on key regions—especially when fusing RGB images with event stream data—ensuring the model prioritizes the most informative cues. Together, these modules enable our full model to achieve an AUC of 0.879, demonstrating their complementary benefits for anomaly detection.\n\n|RGB|Event|GRU|Attention|AUC|AP|AUC-Frame|mTTA|mAP|\n|-|-|-|-|-|-|-|-|-|\n|√|√||√|0.819|0.498|0.657|1.52|43.77|\n|√|√|√||0.817|0.508|0.668|1.98|43.59|\n|√|√|||0.805|0.479|0.648|1.44|41.66|\n|√|√|√|√|0.879|0.570|0.736|2.80|45.15|\n\n> **Exp 5**: Event Processing\n\nWe compared our asynchronous GNN with two lightweight methods: [AsyNet](https://arxiv.org/abs/2003.09148), which uses sparse convolutions, and [AEGNN](https://arxiv.org/abs/2203.17149), which employs asynchronous GNNs. Our experiments show that our approach achieves better detection performance with far lower computational overhead. Specifically, AsyNet uses 367 MFLOPs per event, AEGNN uses 10.98 MFLOPs, and our method only uses 8.732 MFLOPs per event. This efficiency and accuracy highlight the benefits of our approach for processing event-based data.\n\n> **Ref**: Related Work\n\nIn the revised manuscript, we will discuss recent advances [1,2] in asynchronous processing and multimodal fusion that address real-time constraints. For example, recent video-question answering work uses non-parametric probes like QUAG to decouple and evaluate both intra- and inter-modal interactions. \n\n> **Weaknesses 1**: Adverse Weather or Low-light Scenarios\n\nI collected data on severe weather or low-light scenes on the ROL and DoTA datasets, and performed comparative experiments with the two best performing methods in the comparative experiments. Benefiting from the advantages of event cameras in extreme lighting scenarios, we surpass the performance of other methods by a large margin in this scenario.\n\n|Method|AUC|AP|AUC-Frame|mTTA|mResponse|\n|---|---|---|---|---|---|\n|STFE|0.631|0.397|0.582|1.26|3.12|\n|TTHF|0.652|0.416|0.594|1.31|2.98|\n|OURs|0.719|0.442|0.612|1.47|2.35|\n\n> **Weaknesses 2**: Failure Cases\n\n[Figure R3-1](https://anonymous.4open.science/r/RTAD-3B3C/R3-1.png) shows that our system's performance relies heavily on the object detector's accuracy. In Frame25 and Frame30, blurry images and small objects led to a complete detection failure, which in turn caused the anomaly detection framework to fail. Although an object was detected in Frame35 when it was very close to the ego vehicle, the attention score was only 0.56 due to missing previous detections, preventing the GRU from building a robust temporal feature set. These cases highlight that our approach is limited by the reliability of the object detector, and improvements in detection accuracy are essential.\n\n> **Suggestions 1**: Visualizations\n\nThe [Figure R3-2](https://anonymous.4open.science/r/RTAD-3B3C/R3-2.png) shows that our target-level attention module assigns varying scores to bounding boxes. For instance, a vehicle cutting in starts with a low score (0.21 in Frame 20) that rises as it approaches (0.45 in Frame 25 and 0.71 in Frame 30). This indicates that our attention mechanism, combined with event stream features, effectively emphasizes fast-moving objects, leading to higher anomaly scores.\n\n> **Q1**: Asynchronous GNN \n\nOur asynchronous GNN module requires only 8.732 MFLOPs per event—far lower than synchronous methods (74559 for Events+YOLOv3, 6984 for RED, and 27659 for ASTM-Net). This efficiency is achieved by processing events asynchronously, eliminating the heavy cost of frame-based operations.\n\n> **Q2**: Sensitivity Analyses\n\nOur [Figure R3-3](https://anonymous.4open.science/r/RTAD-3B3C/R3-3.png) shows that the model remains robust across different settings. At the standard 0.5 IOU and 0.5 confidence thresholds, the model achieves an AUC of 0.879. As shown in [Figure R3-4](https://anonymous.4open.science/r/RTAD-3B3C/R3-4.png) and [Figure R3-5](https://anonymous.4open.science/r/RTAD-3B3C/R3-5.png), Increasing these thresholds sharply reduces AUC by filtering out more detection boxes, while lowering them causes a gradual decline as false positives are effectively down-weighted. Overall, the AUC ranges from 0.65 to 0.879, confirming the stability of our configuration.", "claims_and_evidence": "Yes\n1. The authors claim substantial improvements in detection accuracy and inference speed.\n2. These claims are substantiated by comprehensive experiments, including comparative studies and ablation experiments that report key metrics such as AUC, AP, frame-level AUC, and mean Time-to-Accident (mTTA).\n3. The evaluation is limited to the ROL and DoTA datasets, which may not fully represent complex real-world scenarios (e.g., extreme weather, low-light conditions).\n4. Additional experiments on diverse conditions would strengthen the evidence regarding the method’s generalization capabilities.", "methods_and_evaluation_criteria": "Yes\n1. The integration of event and RGB data is well-justified, effectively capturing both dynamic and static scene information.\n2. The choice of an asynchronous GNN, along with GRU-based temporal modeling and attention mechanisms, is well aligned with the goal of reducing inference latency.\n3. It would be beneficial to include comparisons with Transformer-based models or other lightweight asynchronous techniques to better position the advantages of the proposed asynchronous GNN.\n4. A deeper discussion on the trade-offs between various event processing methods could provide further clarity on the design choices.", "theoretical_claims": "1. The paper primarily focuses on algorithmic innovation and empirical validation rather than formal theoretical proofs, which is appropriate given its applied focus.\n2. The intuitive arguments supporting asynchronous processing and multimodal fusion are well supported by experimental results.\n3. Given that the v2e conversion process can introduce errors, an analysis of how such errors might propagate through the model and impact overall performance would add further rigor to the study.", "experimental_designs_or_analyses": "1. The experimental setup is thorough, with extensive comparisons to state-of-the-art baselines and detailed ablation studies that isolate the contributions of individual components (e.g., GRU, attention, BBox modules).\n2. The analyses convincingly demonstrate the model’s improvements in performance metrics.\n3. The paper would benefit from additional experiments testing inference speed across diverse computing platforms.\n4. Module-level ablation studies, particularly regarding the impact on detection performance when components like GRU or attention are removed, could help clarify the contributions of each module.\n5. A comparative analysis of the computational overhead of the asynchronous GNN versus other lightweight event processing methods is also recommended.", "relation_to_broader_scientific_literature": "The contributions are well-situated within the existing literature on anomaly detection, event-based vision, and sensor fusion in autonomous driving. The paper builds upon established methods such as ResNet, YOLOX, and asynchronous GNNs, while addressing the critical issue of latency. It extends prior work by demonstrating that multimodal fusion can simultaneously achieve high detection accuracy and rapid response, a balance that is crucial in safety-critical applications.", "essential_references_not_discussed": "While the paper cites many relevant works, it could benefit from discussing additional recent advances in asynchronous processing and multimodal fusion techniques, especially those addressing real-time constraints in other domains. Including references to cutting-edge approaches in temporal modeling or sensor fusion from the latest conferences could provide a more comprehensive context.\n\n[1] Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion\n[2] InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions", "other_strengths_and_weaknesses": "1. Limited discussion on the performance under diverse real-world conditions such as adverse weather or low-light scenarios.\n2. The paper could include a more detailed theoretical analysis or discussion of potential failure cases and limitations.", "other_comments_or_suggestions": "More qualitative comparisons, such as side-by-side visualizations of detection outputs in challenging scenarios, would further enhance the clarity of the presentation.", "questions_for_authors": "1. Can you provide more details on the computational overhead of the asynchronous GNN module compared to synchronous processing methods?\n2. Have you conducted sensitivity analyses on the hyperparameters related to the fusion strategy and detection thresholds? How stable is the model’s performance across different settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Y1BU859Oa", "reviewer": "Reviewer_piFo", "summary": "This paper presents a multimodal asynchronous hybrid network for real-time anomaly detection in autonomous driving scenarios by combining event-camera streams with RGB-camera data. The method uses an asynchronous Graph Neural Network for event-stream processing and a CNN for spatial feature extraction from RGB images. This approach effectively captures spatiotemporal dynamics in driving environments, enabling fast and accurate anomaly detection.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your inspiring review and actionable suggestions. Below, you will find our detailed responses to your questions.\n\n> **Q1**: Real-time Anomaly Detection Methods\n\nWe did omit some real-time anomaly detection papers, but these methods are not applicable to the traffic domain. For example, AED-MAE (CVPR 2024)[1] is a state-of-the-art anomaly detection method for surveillance videos that focuses on detection speed. However, it incorporates static background information into its approach and operates at the frame level. In autonomous driving, interference from dynamic backgrounds would significantly degrade its performance. Similarly, industrial anomaly detection methods such as EfficientAD(WACV 2024) [2] are also highly efficient but operate at the frame level, making them unsuitable for the complex environments of autonomous driving. MOVAD(ICASSP 2024)[3] is the first method to propose online real-time anomaly detection specifically for autonomous driving. We compare our approach with these three methods, and our method achieves the best balance between accuracy and speed, specifically designed to meet the unique requirements of autonomous driving. A key innovation of our work is its focus on achieving robust performance in dynamic environments. We tested these three models on ROL and DoTA and the results are shown in the following table.\n\n|Method|Type|AUC-Frame(%)|AUC-Frame(%)|mTTA(s)|mTTA(s)|mResponse(s)|mResponse(s)|FPS|\n|---|---|---|---|---|---|---|---|---|\n|Datasets|-|ROL|DoTA|ROL|DoTA|ROL|DoTA|-|\n|EfficientAD|Frame|0.519|0.549|0.89|0.97|3.65|2.68|557|\n|AED-MAE|Frame|0.571|0.652|1.01|1.35|3.36|2.79|1655|\n|MOVAD|Frame|0.719|0.821|2.47|2.55|2.61|1.33|158|\n|OURs|Object|0.736|0.823|2.80|2.78|2.35|1.21|579|\n\n\n> **Q2**: Inadequate literature review\n\nIn fact, our comparison methods already include SOTA in traffic anomaly detection, such as MAMTCF (ArXiv 2023), AM-NET (IEEE TIV 2023), and TTHF (IEEE TCSVT 2024). In our revised manuscript, we will expand Table 1 to incorporate additional recent methods, including three state-of-the-art real-time anomaly detection approaches—EfficientAD, AED-MAE and MOVAD. We will clearly articulate the distinctions between existing video-frame-based real-time anomaly detection methods and our object-centric framework, which is specifically designed to address the challenges inherent in autonomous driving scenarios. Additionally, we acknowledge that our original claim that \"most existing methods focus on accuracy but ignore speed\" may have been too broad. In the revision, we will refine this statement to \"most existing object-centric methods focus on accuracy but ignore speed\" to more accurately reflect the current state of the art.\n\n> **Q3**: Methodological Novelty\n\nOur method is the first to utilize both event streams and RGB data for traffic anomaly detection, making it a novel contribution. The key innovation of our approach lies in leveraging the unique characteristics of event streams as critical features for road traffic anomaly detection.  Through graph-based modeling in asynchronous GNNs, we perform neighborhood aggregation only on active event nodes, combined with a lookup table acceleration mechanism in Spline convolution, enabling efficient feature extraction. Moreover, the continuity of event streams allows the model to perform anomaly detection between RGB frames (Figure 4), enabling early anomaly detection, which is another key innovation of this work. Finally, by incorporating global and object-level graph modeling along with a dynamic attention mechanism, the model leverages a GRU module to capture the temporal dependencies of these features, focusing on anomalous motion patterns of specific objects, thereby achieving accurate anomaly identification.\n\n\n> **Q4**: Figure 1\n\nFigure 1 illustrates our anomaly detection process for autonomous driving. When a leading vehicle exhibits abnormal behavior, the model's anomaly score gradually increases until it surpasses the anomaly score threshold. Therefore, our primary focus is on whether the anomaly score can quickly reach the threshold when detecting an anomalous object.This process involves two key time components: \\(T_{\\text{inference}}\\), which represents the model's processing latency, and \n\\(\\Delta T_{\\text{detection}}\\), the additional time required to recognize and confirm the anomaly—essentially, the time it takes for the anomaly score to reach the threshold. The sum of these components defines the overall response time, which reflects both the inference speed of our model and the latency in anomaly detection (i.e., the delay in quickly identifying an object at risk of being anomalous).\n\n\n**References**\n\n[1] Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors, CVPR 2024.\n\n[2] EfficientAD: Accurate Visual Anomaly Detection at Millisecond-Level Latencies, WACV 2024.\n\n[3] Memory-augmented Online Video Anomaly Detection, ICASSP 2024.", "claims_and_evidence": "The paper claims that existing methods focus on improving detection accuracy while neglecting inference speed, but it doesn't fully consider the existing real-time anomaly detection methods, resulting in an insufficient literature review. Some sentences are poorly constructed and show obvious translation marks, affecting clarity. Figure 1 lacks explanation, and the method's innovation is questionable as the used GNN and ResNet are common in anomaly detection. The multimodal fusion is just simple sampling and merging without in - depth modeling of complex relationships between modalities.", "methods_and_evaluation_criteria": "The proposed multimodal asynchronous hybrid network is logically sound. By employing lightweight networks and straightforward fusion strategies, it enhances the real-time performance of anomaly detection.", "theoretical_claims": "The paper does not explicitly propose theoretical claims requiring validation or proofs. Its primary contribution lies in introducing a novel multimodal network architecture for real-time anomaly detection, validated experimentally for effectiveness and superiority.", "experimental_designs_or_analyses": "The experimental design is generally reasonable, with extensive evaluations on multiple benchmark datasets demonstrating the method’s advantages in accuracy and response time. However, the selection of baseline methods in Table 1 is problematic: most compared methods are outdated and not real-time approaches. Including state-of-the-art real-time methods would strengthen the results’ credibility.", "relation_to_broader_scientific_literature": "This work focuses on real-time anomaly detection and proposes a multimodal asynchronous hybrid network, which offers some innovation in autonomous driving research. However, the paper overlooks existing methods specifically targeting low-latency anomaly detection, weakening the persuasiveness of its contributions and results.", "essential_references_not_discussed": "The paper fails to discuss key works dedicated to real-time anomaly detection, such as algorithms or architectures optimized for low-latency performance.", "other_strengths_and_weaknesses": "Other Strengths: Clear visualizations highlight the method’s advantages, and the experimental design and analysis are thorough.\nOther Weaknesses: Inadequate literature review, unclear phrasing in sections, limited methodological novelty, and suboptimal baseline selection. Additionally, the claim that \"most existing methods focus on accuracy but ignore speed\" is questionable, as numerous real-time approaches exist in the field.", "other_comments_or_suggestions": "See \"Other Strengths and Weaknesses\" above.", "questions_for_authors": "My main concerns are outlined in \"Other Strengths and Weaknesses.\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper makes a significant and practically relevant contribution to real-time anomaly detection in autonomous driving. It addresses both research and deployment challenges, offering a system that is fast, accurate, interpretable, and ready for edge deployment. While not all components are architecturally novel, their combination and adaptation for event-based asynchronous fusion is both innovative and timely. The thorough rebuttal and strong experimental rigor elevate the paper to a clear accept.
Decision: Accept (spotlight poster)