# Innovative Approaches to Traffic Anomaly Detection and Classification Using AI
## 2 Algorithms and Methods
### Multimodal Large Language Model (MLLM)  
Large language models (LLMs) have gained importance for their versatility, leading to the development of multimodal large language models (MLLMs) capable of processing diverse data types, including images, videos, and text. MLLMs are being applied to traffic anomaly detection, where they analyze video sequences, interpret contextual information, and identify events such as dangerous driving behaviors or accidents. By integrating MLLMs into traffic surveillance systems, road safety can be enhanced, and traffic management optimized. A novel approach to video anomaly detection is introduced in [54], using predefined text descriptions to represent both normal and abnormal situations. Unlike traditional methods that rely on labeled video data, this framework incorporates domain knowledge and large language models (LLMs), such as ChatGPT (chat.openai.com, accessed on 20 March 2023), to generate comprehensive text descriptions of events. The overall inference process is illustrated in Figure 18.  
The detection process utilizes contrastive language-image pre-training (CLIP) [55], a vision-language model that calculates cosine similarity between image features and predefined text features. To adapt CLIP for this task, a conditional text similarity measure with trainable parameters is introduced. These parameters are optimized using triplet loss and regularization loss, removing the need for labeled data. Triplet loss maximizes the difference between normal and abnormal text descriptions by identifying the most similar and dissimilar examples, while regularization loss ensures pseudo-normal frames have low anomaly scores. Experiments on the ShanghaiTech and UCFcrime datasets demonstrate superior performance compared to unsupervised methods and results comparable to weakly supervised models. Additionally, the method is efficient and resilient to noisy text descriptions. Figure 17: The online frame-level VAD architecture. f[t] is the frame at time t, x is the output of the reducer, NF is the number of frames in input to the VST, s[t] is the anomaly classification score of the frame f[t]. From [52]. AccidentBlip2 [56] is a MLLM designed for traffic accident detection using only vision. Building on the Blip2 architecture, AccidentBlip2 integrates key innovations to optimize its performance in this domain. A key component of AccidentBlip2 is the Motion Oformer module, which facilitates temporal inferences from multi-view images. Using a vision Transformer (ViT), the module extracts image features and fuses them in the Oformer, where cross-frame attention ensures efficient temporal information transfer. The model extends beyond single-vehicle setups by incorporating a multi-vehicle system, addressing blind spots and enhancing accident detection across the environment. Each vehicle's neural network detects accidents involving itself or other vehicles. Multi-view features extracted by a pre-trained ViT-14g model are processed by the Motion Oformer for autoregressive inference, as shown in Figure 19.  
Evaluations on the DeepAccident [57] dataset show that AccidentBlip2 outperforms benchmarks like Video-LLaMA and V2 XFormer. It achieves 66.5% accuracy in single-vehicle scenarios and 72.2% accuracy in multi-vehicle setups for detecting environmental accidents. In addition, the results highlighted that by using the Motion Oformer module and multi-vehicle perception, AccidentBlip2 demonstrates superior capability in analyzing multi-view temporal data, making it a robust solution for accident detection in complex traffic environments. ViTA [58] is an algorithm designed to accelerate video-to-text conversion in retrieval-augmented generation (RAG) frameworks, addressing latency challenges in processing large video volumes. By leveraging vision-language models (VLMs), ViTA optimizes the relationship between output tokens and processing time to enhance efficiency without sacrificing accuracy.