# Urban Road Anomaly Monitoring Using Vision-Language Models for Enhanced Safety Management
## 2 Methods
### Training and Optimization
#### 2.3.1 URA VLMs  
Based on the InternVL selected as relatively suitable for real-time monitoring of road anomalies, we proceeded with the URA VLMs training and optimization using prompt engineering, a retrieval-augmented generation (RAG) database, and chain-of-thought (COT) techniques. This phase focused on enhancing the model's capability to accurately classify the road and interpret and describe flood-related data; the key feature of RAG is presented in Appendix A Table A1. The inference establishes a comprehensive judgment framework consisting of the following steps: (1) First, identify road anomalies, which refer to the occurrence of seven types of road anomalies. (2) For flooding cases, proceed to the reference detection VLM, which identifies a reference object within the image that can be utilized to describe depth. The RAG provides parameters for the reference object to ascertain depth in relation to the flooding situation. (3) These parameters are then combined to generate prompts. Next, the VLM evaluates the depth of water accumulation and determines the severity based on this depth. (4) After concluding the types of road anomalies and assessing the waterlogging depth, merge the results to generate the final safety level for the dashboard (Figure 3). \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Model** & **Main Features** & **Suitability for Real-Time Monitoring** & **Suitability for Urban Road Anomaly Classifications** & **Suitability for Urban Road Waterlogging Monitoring** \\ \hline  & & & Medium-Low Moderate suitability due to its additional segmentation & High Highly suitable for detailed anomaly detection and pixel-level & Low It requires integration with other models or methods for depth estimation or event detection. \\ \hline  & & & Medium-Low & Low \\
4 & Mask R-CNN (Object detection model) & Capable of instance segmentation with pixel-level accuracy [51]. & Moderate suitability due to its additional segmentation overhead, which slows down processing. & High highly suitable for detailed anomaly detection and pixel-level segmentation. & It requires integration with other models or methods for depth estimation. \\ \hline  & & & Highly suitable due to its balance between speed and accuracy. & High & Medium-Low \\
5 & Efficient Det (Object detection model) & Balances accuracy and speed using a compound scaling method [52]. & Highly suitable due to its balance between speed and accuracy. & Suitable for detecting and classifies road anomalies. & It requires integration with other models or machine learning methods for depth estimation. \\ \hline  & & & Medium-High & Medium-High & Medium-High \\
6 & InternVL (Vision–Language Model) & Integrated sampling for contextual and vision monitoring [53]. & Medium-High & High & The model can be applied to this \\ \hline \hline \end{tabular}
\end{table} 
Table 5: _Cont._We constructed three specialized expert models--Road Anomalies Detect VLM, Waterlogging Reference Object Detect VLM, and Waterlogging Depth Describe VLM--for specific tasks using a COT design that employs multi-step prompts. In this framework, each step requires the URA-VLMs to perform distinct tasks. This approach not only enhances the accuracy of road anomaly detection but also contributes to the efficient allocation of resources for real-time monitoring. The purpose of this research method is to decompose an end-to-end (E2E) task into multiple steps, thereby improving both accuracy and efficiency. Our model aims to classify types of road anomalies and further assess the depth of flooding to categorize the severity of road abnormalities. By decomposing the task into several subtasks, we significantly enhance accuracy. To optimize resource utilization, the first step involves classifying road anomalies and determining the presence of a waterlogging issue, as such problems are often absent. This strategy effectively reduces the number of VLM calls in subsequent steps, alleviating server load.