# Innovative Approaches to Traffic Anomaly Detection and Classification Using AI
## 2 Algorithms and Methods
### Multimodal Large Language Model (MLLM)
The algorithm employs a hybrid approach, combining lightweight and heavyweight VLMs to balance speed and detail. In the first stage, a lightweight VLM (e.g., BLIP) generates a quick, basic description of the video content. This description serves as a prompt for a heavyweight VLM (e.g., InternLM-XComposer2), which extracts additional details while limiting token generation, as illustrated in Figure 20.  
Figure 18: Overall inference process of the proposed anomaly detector using text descriptions. From [54]. This two-stage strategy reduces latency by minimizing the output token count required from the heavyweight VLM. Experimental results on real-world datasets like StreetAware [59] and Tokyo MODI [60] show that ViTA achieves up to a 43% reduction in latency compared to systems relying only on heavyweight VLMs. Importantly, the algorithm maintains high query accuracy, ensuring information extraction without quality loss. An innovative framework utilizing multimodal large language models (MLLMs) is proposed in [61] for real-time detection of critical road safety events. This approach integrates the logical and visual reasoning capabilities of models like Gemini-Pro-Vision 1.5 to analyze driving video sequences and extract key safety information, addressing limitations of traditional methods reliant on complex models and large datasets. The framework employs a multi-stage question-and-answer (QA) process to guide MLLMs in identifying and classifying entities within video frames. Detected hazards are categorized using three dimensions: \"What\" (entity classification), \"Which\" (specific feature identification), and \"Where\" (location and distance of hazards). This structured methodology ensures the comprehensive assessment and rapid detection of safety-critical events. Figure 19: Architecture of MotionBlip2. From [56]. Figure 20: Overview of the workflow of our video-to-text conversion algorithm. Initially, a lightweight VLM is employed to generate a text description of a video clip. Subsequently, a heavyweight VLM is employed, utilizing the initial description as a prompt to generate additional information while limiting the output token count. From [58]. Experiments conducted on the DRAMA [62] dataset, which focuses on driving hazard detection using natural language queries, demonstrated that few-shot learning achieved a 79% accuracy, outperforming other methods. The framework also excelled in tasks like scene classification, vehicle direction identification, and risk agent categorization, significantly surpassing previous visual reasoning models. The integration of multimodal data and dynamic context optimization further enhanced prediction reliability in driving scenarios. The authors of [56] reviewed the application of MLLMs and vision large models (VLMs) in object detection for transportation systems, highlighting their potential to enhance safety, efficiency, and reliability. By integrating text, images, videos, and sensor data, MLLMs offer a comprehensive understanding of transportation environments, enabling context-aware object detection and few-shot learning to reduce reliance on large annotated datasets. The study categorizes MLLM-based applications into three main tasks: perception and understanding, navigation and planning, and decision making and control. Real-world case studies demonstrate the versatility of MLLMs through examples such as road safety attribute extraction, safety-critical event detection, and visual reasoning in thermal imaging. These applications underscore the effectiveness of MLLMs in diverse transportation scenarios. Despite their strong potential, MLLMs face notable limitations in traffic anomaly detection. They often struggle with complex visual scenes, such as occlusions or overlapping objects, and exhibit weaknesses in recognizing fine details for accurate anomaly detection. A significant issue is the known phenomenon \"hallucination\", where the model generates outputs not grounded in the input data, leading to false detections.