SurveillanceVQA-589K: A Benchmark for Comprehensive Surveillance Video-Language Understanding with Large Models

## Abstract

Abstract Understanding surveillance video content remains a critical yet underexplored challenge in vision–language research, particularly due to its real-world complexity, irregular event dynamics, and safety-critical implications. In this work, we introduce SurveillanceVQA-589K, the largest open-ended video question answering (VQA) benchmark tailored to the surveillance domain. The dataset comprises 589,380 QA pairs spanning 12 cognitively diverse question types, including temporal reasoning, causal inference, spatial understanding, and anomaly interpretation, across both normal and abnormal video scenarios. To construct the benchmark at scale, we design a hybrid annotation pipeline that combines temporally aligned human-written captions with Large Vision-Language Model (LVLM) assisted QA generation using prompt-based techniques. We also propose a multi-dimensional evaluation protocol to assess contextual, temporal, and causal comprehension. We evaluate eight LVLMs under this framework, revealing significant performance gaps, especially in causal and anomaly-related tasks, underscoring the limitations of current models in real-world surveillance contexts. Our benchmark provides a practical and comprehensive resource for advancing video-language understanding in safety-critical applications such as intelligent monitoring, incident analysis, and autonomous decision-making. The dataset and code are publicly available at: https://huggingface.co/datasets/fei213/SurveillanceVQA-589K .

## 1 Introduction

Surveillance videos have become pivotal data sources for smart cities [19]. In contrast to publicly available or social media videos, surveillance footage varies significantly in acquisition methods, content attributes, and application purposes [51, 44]. They encompass diverse spatiotemporal conditions [39, 42], spanning day-night cycles, varied weather, and heterogeneous environments like streets [28, 36], shopping centers [3], and transportation hubs [29], resulting in high data heterogeneity. Abnormal events captured are often sudden, low-frequency, and diverse, posing challenges for perception and modeling [32, 31].

Current computer vision tasks for surveillance videos primarily focus on the detection and classification of abnormal events, often relying on predefined event types and handcrafted features [41, 61, 12, 2, 57, 48]. While such goal-oriented approaches can be effective in specific scenarios, they typically lack deeper semantic modeling of event progression, behavioral motivations, and environmental context [55]. This limits the potential of surveillance videos in intelligent urban governance, behavior prediction, and multimodal reasoning [40].

To address these limitations, our previous work, UCA [56, 55], has proposed a multimodal understanding framework for surveillance videos. It incorporates fine-grained language annotations and temporal markers, covering tasks such as moment localization, caption generation, and dense captioning. This framework extends the boundaries of traditional surveillance video analysis by enhancing its semantic expressiveness. However, UCA primarily focuses on descriptive tasks and lacks interactive question answering (QA) mechanisms, which makes it less aligned with recent trends in complex semantic understanding and reasoning within multimodal systems [20].

To further enhance the semantic reasoning capabilities of models in the surveillance domain, we introduce QA tasks to enable interactive and cognitively rich understanding as shown in Figure [1](https://arxiv.org/html/2505.12589v1#S0.F1). Unlike descriptive tasks, QA tasks allow models to perform logical reasoning, causal inference, and complex semantic analysis on video content. To achieve this, we construct SurveillanceVQA-589K, a large-scale QA dataset specifically designed for surveillance videos. The dataset consists of four surveillance video datasets as video sources and contains approximately 589,000 question-answer pairs, including 12 QA types covering both normal and abnormal video content, such as factual summarization, behavior/spatial-temporal analysis, causal reasoning, anomaly detection, etc. This design aims to elevate video understanding to a higher cognitive level.

We benchmark 8 Large Vision-Language Models (LVLMs) on SurveillanceVQA-589K, including variants like VideoLLaMa3 [58], LLaVA series [60, 59, 22], Qwen2.5-VL series [4], and InternVL series [9], ranging from lightweight 0.5B to general-purpose 7B models. Despite their success in open-domain tasks, current LVLMs demonstrate clear limitations in surveillance scenarios. Performance on complex tasks, such as causal inference and abnormal event analysis, remains poor, with most models scoring below the midpoint. Furthermore, while fine-tuning improves general understanding, it fails to significantly enhance structured reasoning tasks like anomaly detection and classification.

Our main contributions are as follows.

- •
We construct SurveillanceVQA-589K, the largest surveillance video QA dataset to date, containing 589,380 QA pairs across 12 task types and 18 abnormal event categories, enabling comprehensive semantic understanding under both normal and abnormal conditions.
- •
We introduce a scalable hybrid annotation pipeline, combining human-aligned captions with LVLM-generated content, and propose a multi-dimensional evaluation framework tailored to surveillance video reasoning tasks, including contextual, temporal, and causal metrics.
- •
Our benchmark evaluates eight open-source LVLMs (0.5B–7B) and reveals systematic weaknesses in causal inference and anomaly understanding, offering a practical testbed for real-world applications such as intelligent security and video anomaly response systems.

## 2 Related Works

### 2.1 Surveillance Video Analysis Benchmark

In recent years, there has been a growing interest in the academic community toward understanding the content of surveillance videos, which has led to the development of several benchmark datasets to support research in this domain. Early datasets predominantly focused on anomaly detection in surveillance scenarios, including UCSD Ped1 and Ped2 [26], the Avenue dataset [34], the Subway dataset [1], the ShanghaiTech Campus dataset [35], UCF-Crime [43], MEVA [10], NWPU Campus dataset [5], and MSAD [62]. While these datasets have laid a strong foundation for surveillance video analysis, the majority of these datasets still lack accompanying textual annotations, rendering them inadequate for comprehensive multimodal vision-language research.
Notably, UCA [56] distinguished itself from prior surveillance datasets through its rich linguistic annotations. However, the current version of UCA includes only approximately 20,000 manually labeled descriptions and lacks an interactive, question-answering (QA) based evaluation framework. Such a framework is increasingly recognized as a critical tool for assessing high-level reasoning, anomaly understanding, and semantic generalization in modern multimodal systems.

To address this gap, we propose the construction of a novel QA-driven benchmark for surveillance video understanding. This benchmark is designed to enable interactive evaluation and foster deeper semantic reasoning over real-world surveillance video content.

### 2.2 Video-language Understanding Benchmark

With the emergence of LVLMs [24, 11, 30], traditional benchmarks have become increasingly inadequate in capturing the full range of model capabilities. Recent benchmarks aim to address this by incorporating diverse tasks [46, 27], multi-level granularity [45], and scalable evaluation protocols [7]. Benchmarks such as OwlEval, MME [14], SEED-Bench [23], MM-Vet [54], and MMBench [33], MVBench [25], Vision-R1 [18], and EMMA [17] cover a wide spectrum of tasks, from image captioning and reasoning to fact verification. These works propose multi-dimensional metrics—including linguistic consistency, semantic alignment, and visual grounding—to characterize model behavior more comprehensively. In the video domain, models must handle temporal dynamics and evolving semantics [47, 8]. Earlier benchmarks such as TVQA [21] and Next-QA [49] used multiple-choice formats to assess temporal localization and event comprehension. More recent work, such as VideoChatGPT [37], introduces multi-turn QA grounded in video input, emphasizing coherence and contextual consistency over extended interactions. FunQA [50] pushes reasoning further by evaluating a model’s ability to identify unexpected or humorous events, highlighting challenges in modeling incongruity and causal anomalies in temporal sequences. Svbench [53] proposes temporal multi-turn QA chains, which are specifically for streaming video understanding.

Our work extends these efforts by addressing the unique challenges of surveillance video analysis. Unlike open-domain or entertainment-based datasets, SurveillanceVQA-589K emphasizes real-world diverse abnormal events, spatiotemporal analysis, complex reasoning, etc.

## 3 SurveillanceVQA-589K

The SurveillanceVQA-589K dataset includes 31,548 video clips with textual annotations, 27,966 clips labeled as normal and 3,585 as anomalous, resulting in a total of 589,380 QA pairs. The following contents show the procedure of QA pairs generation, illustrated in Figure [2](https://arxiv.org/html/2505.12589v1#S3.F2).

Figure: Figure 2: Our overall framework, including QA generation and evaluation.
Refer to caption: extracted/6450145/framework.png

### 3.1 Video Annotation Generation

The examples of human-labeled, LVLM-labeled, integrated annotations are shown in Appendix [A](https://arxiv.org/html/2505.12589v1#A1).

#### 3.1.1 Human-labeled Annotation

Following the annotation protocol established in UCA [56], we extended manual annotation efforts to the MSAD, MEVA, and NWPU surveillance datasets. This process involved generating event-level captions that included both precise timestamps and detailed event descriptions.

To ensure annotation accuracy and consistency, we provided annotators with clear guidelines and comprehensive training, following the protocol established in UCA [56]. The annotation was carried out by a team of technically proficient annotators who were fairly compensated according to local wage standards. The process was supervised by AI researchers who regularly reviewed and validated the outputs to ensure both quality and ethical compliance. The entire annotation phase spanned approximately one month and resulted in a high-quality corpus. We then integrated this newly annotated data with existing annotations from UCA, completing the manual annotation collection with a total of 31,548 sentence-level annotations accompanied by precise timestamps.

#### 3.1.2 LVLM-labeled Annotation

Subsequently, using the timestamp information obtained during the manual annotation phase, we employed the video processing toolkit MoviePy to automatically segment the original videos and extract the corresponding short clips. We then utilized the powerful multimodal model LLaVA-Video-7B-Qwen2 [60] to perform in-depth analysis on each segmented clip, generating detailed descriptions. As a result, we obtained 31,548 segment-level annotations produced by the LVLM.

#### 3.1.3 Integrated Human-LVLM Annotations

Figure: Figure 3: Procedure of integrating annotations.
Refer to caption: extracted/6450145/prompts.png

In this stage, we performed a deep integration of the high-quality information obtained from manual annotations with the rich segment-level descriptions generated by the large multimodal model LLaVA-Video. The objective of this integration was to combine the precision of human annotations with the diversity and depth of model-generated descriptions. The guiding principle for prompt design was to preserve the semantic integrity of human annotations while enriching them with complementary information from the model output. To facilitate this process, we introduced Qwen-Turbo [52], an advanced language model, to leverage its natural language processing capabilities for comprehensive analysis, redundancy elimination, and content optimization. Specifically, Qwen-Turbo was tasked with identifying and resolving redundant or inconsistent expressions, while enhancing semantic richness and logical coherence. This resulted in more fluent, structured, and contextually aligned event-level descriptions for each video clip.

Through this integration, we obtained 31,548 refined textual annotations. The procedure for integrating human and LVLM annotations, along with the generation prompt, is illustrated in Figure [3](https://arxiv.org/html/2505.12589v1#S3.F3).

### 3.2 Automatic QA Generation

After obtaining the clip-level descriptions, we proceeded to generate QA pairs.

#### 3.2.1 QA Category Definition

Specifically, we employed Qwen-Max [52] to analyze the annotation data and classify each video segment as either normal or abnormal.

After obtaining two distinct sets of video clips (normal vs. abnormal) along with their corresponding textual annotations, we designed category-specific prompts to guide the generation of QA pairs. For normal clips, the prompts were crafted to elicit a comprehensive understanding of video content, focusing on global scene descriptions, temporal sequencing, spatial detail extraction, and behavioral inference. These prompts support general video comprehension and open-ended QA tasks. In contrast, for abnormal clips, the prompts emphasized event detection, anomaly type classification, subject identification, detailed incident descriptions, and causal reasoning. This design is tailored to the specific requirements of surveillance anomaly detection and incident-level semantic analysis.

This balanced approach ensures both efficiency and semantic richness across diverse surveillance contexts. The detailed prompts are summarized in Table [7](https://arxiv.org/html/2505.12589v1#A2.T7) of the Appendix [B](https://arxiv.org/html/2505.12589v1#A2).

#### 3.2.2 QA Generation

Based on the customized prompts, we employed Qwen-Max [52] to generate QA pairs for each video clip, following different generation strategies for normal and abnormal categories. For normal QA tasks, six QA types (Summary, Generic, Temporal, Short Temporal, Spatial, and Reasoning QA) were defined, with three QA pairs generated per type. For abnormal QA tasks, the other six QA types (Detection, Classification, Temporal, Description, Cause, and Result QA) were applied, but with one QA pair generated per type to emphasize critical semantic cues. Notably, all of the video clips (including normal and abnormal clips) are suitable for the normal QA tasks with their corresponding normal QA pairs, while only abnormal clips are suitable for the abnormal QA tasks with particularly defined abnormal QA pairs.

The defined QA categories are illustrated in Figure [1(b)](https://arxiv.org/html/2505.12589v1#S0.F1.sf2), and detailed descriptions of each QA task are provided in the Appendix [B](https://arxiv.org/html/2505.12589v1#A2).

### 3.3 Data Statistics

#### 3.3.1 Overall statistics

**Table 1: Overall data statistics of SurveillanceVQA-589K.**
| Dataset | Number of<br>Videos | Total Video<br>Duration | Number of Text<br>Annotations | Number of Segments | QA<br>Pairs |  |
| --- | --- | --- | --- | --- | --- | --- |
| Normal | Abnormal |  |  |  |  |  |
| MSAD | 201 | 4.23h | 1783 | 1417 | 366 | 34290 |
| MEVA | 720 | 16.76h | 2057 | 2044 | 13 | 37104 |
| NWPU | 255 | 16.29h | 4166 | 4121 | 45 | 75258 |
| UCA | 1854 | 121.9h | 23542 | 20380 | 3162 | 442728 |
| Total | 3030 | 159.18h | 31548 | 27962 | 3586 | 589380 |

Table [1](https://arxiv.org/html/2505.12589v1#S3.T1) presents statistical information for the SurveillanceVQA-589K (containing MSAD, MEVA, NWPU, and UCA), including the number of videos, total duration, textual annotations, segmented clips (categorized as normal and abnormal), and the total number of QA pairs. More details are presented in Appendix [C](https://arxiv.org/html/2505.12589v1#A3).

#### 3.3.2 Categorization Statistics

In terms of dataset partitioning, we split the dataset at the clip level rather than by entire videos, using an 8/2 ratio for the training and testing sets. Due to the difficulty of annotating every second of footage, the total duration of the training and testing sets is slightly shorter than that of the original raw video collection. Table [2](https://arxiv.org/html/2505.12589v1#S3.T2) presents the distribution of QA pairs for normal and abnormal events across the training and testing sets. This table reports the number of QA pairs generated for both event types and illustrates how their distribution varies between subsets for each dataset. Here, the normal category refers to non-anomalous clips, while the abnormal category corresponds to anomalous clips, which span 18 distinct abnormal event classes as detailed in the Appendix [C](https://arxiv.org/html/2505.12589v1#A3).

**Table 2: Distribution of QA pairs for normal and abnormal events across the training and testing sets**
|  | MEVA | MSAD | NWPU | UCA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Abnormal | Normal | Abnormal | Normal | Abnormal | Normal | Abnormal | Normal |
| Test | 72 | 7362 | 1776 | 5112 | 240 | 14850 | 15192 | 73368 |
| Train | 240 | 29430 | 7008 | 20394 | 840 | 59328 | 60696 | 293472 |

#### 3.3.3 Comparisons with existing datasets

These comparisons, as shown in Table [3](https://arxiv.org/html/2505.12589v1#S3.T3), reveal a consistent trend: early video QA datasets are limited in scale, typically containing 200 to 900 videos and 2,000 to 4,000 QA pairs, and primarily focus on normal events with fixed question formats. Moreover, most existing datasets rely exclusively on either human annotations or LLM-generated content, with few employing a hybrid approach that leverages both. UCVL [6] serves as an intermediate example by exploring abnormal event QA, yet it still falls short in terms of scale, diversity, and task coverage compared to our SurveillanceVQA-589K dataset.

**Table 3: Overall comparisons with existing datasets.**
| Aspect | MVBench [25] | Video-MME [15] | MMB-Video [13] | UCVL [6] | SurveillanceVQA-589K |
| --- | --- | --- | --- | --- | --- |
| Videos | 200 | 900 | 600 | 1699 | 3030 |
| QA Pairs | 4000 | 2700 | 2000 | 16990 | 589380 |
| Content | Normal | Normal | Normal | Anomaly | Normal&Anomaly |
| QA Forms | MCQ | MCQ | Open-ended | MCQ&Open-ended | Open-ended |
| Generation | LLM | Human | Human | LLM | LLM&Human |
| Evaluation | Matching | Matching | LLM | Matching&LLM | LLM |

## 4 Experiments on SurveillanceVQA-589K

We provide the evaluations, baselines, settings, and experimental results in this section. More detailed experimental results and explanations have been given in the Appendix [D](https://arxiv.org/html/2505.12589v1#A4).

### 4.1 Evaluation Design

We follow VideoGPT+ [38] to design our evaluation criteria. During the evaluation phase, we adopt LLM-based evaluation strategy, using an open API with GLM-4-Flash [16]. This approach ensures that the evaluation process not only considers the semantic consistency of the answers but also leverages the interpretive and reasoning capabilities of advanced language models.

This evaluation framework provides a comprehensive assessment of model-generated answers across four key dimensions. Contextual Integration (CI) measures whether the answer accurately reflects the factual content of the video, avoiding errors or misinterpretations. Detail Orientation (DO) assesses the inclusion of specific and complete key elements. Contextual Understanding (CU) evaluates the alignment of the answer with the overall narrative and emotional tone of the video. Temporal Understanding (TU) focuses on the correctness of event sequences and time-related logic. Each dimension is rated on a 0–5 integer scale, with 5 indicating full accuracy and relevance, and 0 indicating a completely incorrect response. To calculate Average Score (Avg), individual scores are normalized by multiplying each by 0.25 and summing the results. This scoring scheme enables fine-grained, quantitative evaluation of model performance across multiple facets of video understanding.

### 4.2 Baselines and Settings

We evaluate 8 open-source video-language models, including the VideoLLaMA3 [58], InternVL2.5 [9], LLavA-OV-Qwen2 [22]), LLaVA-Video-Qwen2 [60], and Qwen2.5-VL-Instruct series [4], with parameter sizes ranging from 0.5B to 7B. Each model performs inference on one question at a time to prevent information leakage between questions. We select Qwen2.5-VL-Instruct-3B for LoRA fine-tuning, which offers a good balance between parameter size and performance, and conduct the fine-tuning for one epoch on our training set. Evaluation is conducted on our test set, and all experiments are carried out on an NVIDIA RTX 4090 GPU. Table [4](https://arxiv.org/html/2505.12589v1#S4.T4) shows the characteristics of these evaluated open-source LVLMs.

**Table 4: Overview of Evaluated Open-Source Video Models**
| Model Name | Key Features |
| --- | --- |
| Video-LLaMA3-2B/7B [58] | Uses any-resolution vision tokenization and differential frame pruner to reduce |
|  | information loss and computation cost. Trained on high-quality video data. |
| InternVL2.5-2B [9] | Based on the ViT-MLP-LLM framework. Incorporates dynamic high-resolution |
|  | representations, Progressive scaling Strategy, and Chain-of-Thought (CoT) reasoning. |
| LLavA-OV-Qwen2-0.5B/7B [22]) | Multimodal model capable of understanding single images, multiple images, |
|  | and videos. Supports cross-modal transfer learning. |
| LLaVA-Video-7B-Qwen2 [60] | Trained only on text-image data with AnyRes technology. Fine-tuned |
|  | on LLaVA-Video-178K for enhanced video instruction understanding. |
| Qwen2.5-VL-3B-Instruct [4] | Strong instruction-following capabilities across text, image, |
|  | and video. Improved cross-modal alignment and QA generation. |

### 4.3 Results on SurveillanceVQA-589K

#### 4.3.1 Analysis of overall LVLMs performance

Table [5](https://arxiv.org/html/2505.12589v1#S4.T5) presents the evaluation results of various LVLMs across five key dimensions: CI, DO, CU, TU, along with Avg. LLaVA-Video-7B-Qwen2 achieved the highest overall performance, with an average score of 2.72. This strong performance can be attributed to its integration of the LLaVA and Qwen2 architectures and its use of the AnyRes technique, which enhances image-to-video reasoning. This combination enables the model to maintain high cross-modal consistency and accuracy, especially in capturing the temporal flow and nuanced context of video events. In contrast, InternVL2.5-2B demonstrated the lowest overall average, with notably weak scores in CI and DO.

**Table 5: Model performance averaged on different QA tasks across five evaluation dimensions. ^† represents our finetuned LVLMs.**
| Method | CI | DO | CU | TU | Avg |
| --- | --- | --- | --- | --- | --- |
| LLavA-OV-Qwen2-0.5B | 2.89 | 2.62 | 2.89 | 2.64 | 2.76 |
| InternVL2.5-2B | 1.77 | 1.72 | 1.97 | 1.72 | 1.79 |
| VideoLLaMA3-2B | 2.82 | 2.55 | 2.83 | 2.58 | 2.69 |
| Qwen2.5-VL-3B-Instruct | 2.70 | 2.54 | 2.72 | 2.45 | 2.60 |
| LLaVA-NeXT-Video-7B | 2.78 | 2.62 | 2.80 | 2.50 | 2.68 |
| LLavA-OV-Qwen2-7B | 3.15 | 2.85 | 3.12 | 2.89 | 3.00 |
| LLaVA-Video-7B-Qwen2 | 3.17 | 2.85 | 3.15 | 2.92 | 3.02 |
| VideoLLaMA3-7B | 2.93 | 2.67 | 2.93 | 2.70 | 2.80 |
| $\text{Qwen2.5-VL-3B-Instruct}^{\dagger}$ | 2.83 | 2.71 | 2.83 | 2.58 | 2.74 |

While its surface-level integration of context and detail may lag, it retains potential in complex reasoning scenarios due to a Chain-of-Thought (CoT) reasoning mechanism. Interestingly, the LLavA-OV-Qwen2-0.5B model, with just 0.5B parameters, achieved a competitive average score of 2.76. This demonstrates that smaller-scale models can still deliver strong video comprehension, thanks to efficient modality transfer learning between visual and textual inputs. Its performance affirms that parameter size is not the sole determinant of effectiveness in multimodal video tasks. The Qwen2.5-VL-3B-Instruct model also showed robust performance. Designed for multimodal instruction-following, this model emphasizes cross-modal alignment and optimized question-answer generation, equipping it to handle a variety of video, image, and text tasks with precision. Its balanced size and architecture strike an effective trade-off between computational efficiency and capability in understanding context. Finally, Video-LLaMA3-2B/7B incorporates advanced features like Any-resolution Vision Tokenization and the Differential Frame Pruner, which improve video representation quality and processing efficiency across varied resolutions and temporal segments.

#### 4.3.2 Analysis of LVLMs performance across normal QA tasks

Table [6](https://arxiv.org/html/2505.12589v1#S4.T6) presents the performance of several LVLMs on a range of QA tasks, including Summary, Generic, Temporal, Short Temporal, Spatial, and Reasoning QA. The evaluation is conducted on both normal video clips (blue) and abnormal video clips (green), with performance metrics reported for each task. Among the models, LLaVA-OV-Qwen2-7B demonstrates the highest overall performance, indicating robust capabilities in handling spatial and reasoning-based questions. In contrast, InternVL2.5-2B exhibits the lowest performance across most tasks, with a Summary QA score of 0.56/0.37, suggesting limitations in processing video-based QA tasks effectively.

A notable trend is the higher performance on normal video clips compared to abnormal ones across all models and tasks. For instance, VideoLLLaMA3-7B scores 3.01/2.70 in Generic QA for normal versus abnormal clips, respectively, highlighting the challenge posed by abnormal video contexts.

#### 4.3.3 Analysis of LVLMs performance across abnormal QA tasks

Table [6](https://arxiv.org/html/2505.12589v1#S4.T6) presents the performance of various vision-language models on abnormal QA tasks, including Detection QA, Classification QA, Subject QA, Description QA, Cause QA, and Result QA, evaluated exclusively on abnormal video clips with all values reported in Brown. In QA tasks involving abnormal video scenarios, LVLMs demonstrate notable performance variations and task-specific capabilities. Overall, the LLaVA-OV-Qwen2 series models excel in Detection tasks, with the 0.5B variant achieving top scores on CI and CU metrics. For Classification tasks, both LLaVA-Video-7B and LLaVA-OV-Qwen2-7B showcase superior performance across different evaluation criteria. The LLaVA-Video-7B model consistently demonstrates advantages in Subject and Description type questions. Notably, all models perform substantially weaker on higher-order reasoning tasks such as Cause and Result inference, with scores generally falling below the midpoint threshold. LLaVA-NeXT-7B performs relatively better in causal reasoning while LLaVA-Video-7B shows slight advantages in result inference. These patterns indicate that current vision-language models still face significant challenges in understanding causal relationships and reasoning about complex video content, particularly in abnormal video scenarios. Additionally, model scale appears to influence performance, with larger 7B-parameter models generally outperforming smaller variants across most abnormal video QA tasks, though distinct specializations emerge across different task categories.

Overall, the abnormal QA scores in Table [6](https://arxiv.org/html/2505.12589v1#S4.T6) are much lower than normal QA scores in Table [6](https://arxiv.org/html/2505.12589v1#S4.T6), indicating more difficulties in abnormal QA interacting and reasoning.

**Table 6: Performance of different vision-language models across QA tasks. Blue: normal QA tasks on normal video clips, Green: normal QA tasks on abnormal video clips, Brown: abnormal QA tasks on abnormal video clips. ^† represents our finetuned LVLMs.**
| Model | Summary | Generic | Temporal | Short Temporal | Spatial | Reasoning | Detection | Classification | Subject | Description | Cause | Result |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLavA-OV-Qwen2-0.5B | 2.76/2.43 | 2.78/2.52 | 2.62/2.43 | 2.60/2.45 | 3.10/3.12 | 2.93/2.69 | 2.94 | 2.47 | 2.69 | 2.43 | 1.67 | 1.60 |
| InternVL2.5-2B | 0.56/0.37 | 2.20/1.93 | 1.89/1.68 | 2.08/1.95 | 1.92/1.89 | 2.41/2.26 | 1.88 | 1.12 | 1.46 | 0.55 | 0.64 | 0.74 |
| VideoLLaMA3-2B | 2.49/2.00 | 2.84/2.55 | 2.73/2.49 | 2.61/2.44 | 2.97/2.99 | 2.89/2.66 | 1.87 | 2.03 | 2.46 | 1.89 | 1.37 | 1.18 |
| Qwen2.5-VL-3B-Instruct | 2.20/1.49 | 2.66/2.17 | 2.66/2.26 | 2.69/2.31 | 2.86/2.75 | 3.01/2.70 | 1.85 | 2.19 | 2.24 | 1.74 | 1.32 | 1.13 |
| LLaVA-NeXT-7B | 2.17/1.72 | 2.93/2.60 | 2.56/2.28 | 2.66/2.48 | 2.95/2.95 | 3.06/2.81 | 2.32 | 2.56 | 2.59 | 2.11 | 1.96 | 1.65 |
| LLavA-OV-Qwen2-7B | 3.10/2.79 | 3.08/2.86 | 2.95/2.81 | 2.79/2.66 | 3.31/3.34 | 3.08/2.89 | 2.53 | 2.60 | 2.68 | 2.54 | 1.46 | 1.55 |
| LLaVA-Video-7B | 3.05/2.85 | 2.99/2.80 | 2.85/2.77 | 2.78/2.71 | 3.47/3.51 | 3.25/3.05 | 1.92 | 2.59 | 2.76 | 2.76 | 1.52 | 1.81 |
| VideoLLaMA3-7B | 2.68/2.29 | 3.01/2.70 | 2.81/2.55 | 2.65/2.45 | 3.06/3.06 | 2.99/2.78 | 2.03 | 1.53 | 2.66 | 2.01 | 1.75 | 1.27 |
| $\text{Qwen2.5-VL-3B-Instruct}^{\dagger}$ | 2.62/1.99 | 2.76/2.36 | 2.84/2.48 | 2.93/2.59 | 2.89/2.89 | 3.06/2.85 | 1.58 | 2.13 | 2.56 | 2.06 | 1.77 | 1.22 |

### 4.4 Comprehensive analysis

#### 4.4.1 Analysis across different LVLMs and used techniques.

Smaller models, such as LLavA-OV-Qwen2-0.5B, excel in CU and DO but struggle with more complex tasks like TU. These models perform well in simpler scenarios but face limitations in long-duration tracking and anomaly detection. Medium models, like Qwen2.5-VL-3B-Instruct, provide a good balance between performance and efficiency, particularly in handling temporal sequences and contextual integration. Larger models, such as LLaVA-Video-7B-Qwen2 and Video-LLaMA3-7B, excel in temporal reasoning and anomaly detection, benefiting from advanced architectures and video-specific fine-tuning.

The experimental results indicate that fine-tuning yields modest performance gains. Fine-tuning models on specific monitoring video data significantly improves their performance in particular environments, helping them better understand context and identify anomalous events. However, fundamentally improving the model’s capability to align visual and textual information remains the key to achieving substantial progress. The strengths of these models can largely be attributed to their design choices. Techniques like Differential Frame Pruning and Progressive Scaling help the models handle long video sequences more efficiently while retaining important temporal details. Techniques like Any-Resolution Vision Tokenization, are capable of processing low-resolution video efficiently, suitable for monitoring applications where video quality may vary.

#### 4.4.2 Analysis across QA types and video contexts.

The performance of vision-language models varies significantly across different QA types and video contexts. Normal video QA tasks, such as Summary, Generic, Spatial, and Reasoning QA, generally achieve higher scores across all models. For instance, models like LLaVA-OV-Qwen2-7B and LLaVA-Video-7B consistently score above 3.0 on Spatial and Generic QA, indicating strong capabilities in visual description and object localization in well-structured scenes. In contrast, abnormal video QA tasks, especially Cause and Result QA, remain highly challenging. Across all models, scores for Cause QA fall below 2.0, with InternVL2.5-2B as low as 0.64, and the best-performing model (LLaVA-NeXT-7B) only reaching 1.96. This suggests a widespread limitation in causal reasoning, especially under chaotic or low-frequency events like violence or accidents.

Moreover, models generally perform better on normal video clips than on abnormal ones, even for the same QA types. This indicates that scene stability significantly influences model comprehension.

Analysis of finetuned models.

After LoRA fine-tuning, $\text{Qwen2.5-VL-3B-Instruct}^{\dagger}$ achieved an average score of 2.74, compared to 2.60 from the original model, a relative improvement of approximately 5.4%. Notable gains were observed in CI (from 2.70 to 2.83) and DO (from 2.54 to 2.71), indicating enhanced ability in understanding scene context and capturing key details. While fine-tuning improved the model’s performance on general understanding tasks, it has limited impact on anomaly-specific QA, particularly for Detection and Classification questions. This highlights a limitation of current fine-tuning approaches: they do not adequately enhance a model’s ability to produce structured, domain-specific expressions required for precise anomaly identification and classification.

## 5 Ethical Considerations

In this study, we strictly adhere to the ethical guidelines published by NeurIPS, addressing key ethical considerations throughout our data construction, model evaluation, and sharing processes. The video datasets used (such as MEVA, MSAD, NWPU, and UCF) are all publicly available and intended for research purposes, with no personally identifiable information involved, ensuring the legality and compliance of data sources. To mitigate potential biases, we incorporated diverse video scenarios and carefully crafted prompts during the QA generation and evaluation stages, aiming to reduce cultural or contextual bias inherent in language models or the original datasets. Moreover, we explicitly oppose any unauthorized use of our dataset or methods (e.g., for abusive surveillance or discriminatory profiling), and we deliberately avoided including high-risk content such as facial recognition. During data annotation, we employed a collaborative pipeline combining human annotators and large language models (e.g., Qwen-Max), with humans responsible for event segmentation and verification, and fair compensation provided in accordance with local wage standards. We plan to release our dataset and code under a research-friendly license with clear ethical usage guidelines, promoting responsible practices in multimodal research.

## 6 Conclusion

In this study, we introduce SurveillanceVQA-589K, the largest open-ended video QA benchmark tailored specifically to real-world surveillance scenarios. The dataset contains 589,380 QA pairs spanning 12 cognitively diverse task types across both normal and abnormal surveillance video contexts. We propose a hybrid annotation pipeline that combines human-aligned captions with LVLM-assisted QA generation, enabling high-quality, scalable annotation. We benchmark eight state-of-the-art open-source LVLMs ranging from 0.5B to 7B parameters. Our experiments reveal that while these models demonstrate promising performance on general understanding tasks (e.g., Summary, Spatial, and Qeneric QA on normal videos), they struggle significantly with complex semantic reasoning, particularly in anomaly-specific tasks such as causal inference and result prediction indicating a clear performance bottleneck in high-level temporal and logical reasoning. We also examine the impact of fine-tuning through LoRA on a 3B model. While fine-tuning yields moderate gains on general tasks, it provides limited improvement in structured anomaly detection and classification, highlighting that current parameter-efficient tuning approaches are insufficient for enabling domain-specific, structured expression. Overall, this work provides a comprehensive testbed for evaluating multimodal models in realistic surveillance settings. Our findings underscore the pressing need for LVLMs to develop stronger causal reasoning, temporal modeling, and structured response generation capabilities.

## References

- Adam et al. [2008]
Amit Adam, Ehud Rivlin, Ilan Shimshoni, and Daviv Reinitz.
Robust real-time unusual event detection using multiple
fixed-location monitors.
*IEEE Transactions on Pattern Analysis and Machine
Intelligence*, 30(3):555–560, 2008.
- Al-Lahham et al. [2024]
Anas Al-Lahham, Muhammad Zaigham Zaheer, Nurbek Tastan, and Karthik Nandakumar.
Collaborative learning of anomalies with privacy (clap) for
unsupervised video anomaly detection: A new baseline.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 12416–12425, 2024.
- Arroyo et al. [2015]
Roberto Arroyo, J Javier Yebes, Luis M Bergasa, Iván G Daza, and Javier
Almazán.
Expert video-surveillance system for real-time detection of
suspicious behaviors in shopping malls.
*Expert systems with Applications*, 42(21):7991–8005, 2015.
- Bai et al. [2025]
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai
Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun
Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng
Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang,
Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Cao and Others [2023]
Yuxiang Cao and Others.
A new benchmark dataset for semi-supervised video anomaly detection
and anticipation in complex scenes.
*Pattern Recognition*, 139:109433, 2023.
- Chen et al. [2025]
Haoran Chen, Dongyi Yi, Moyan Cao, Chensen Huang, Guibo Zhu, and Jinqiao Wang.
A benchmark for crime surveillance video analysis with large models.
*ArXiv*, abs/2502.09325, 2025.
- Chen et al. [2024a]
Liang Chen, Yichi Zhang, Shuhuai Ren, Haozhe Zhao, Zefan Cai, Yuchi Wang, Peiyi
Wang, Xiangdi Meng, Tianyu Liu, and Baobao Chang.
Pca-bench: Evaluating multimodal large language models in
perception-cognition-action chain.
*ArXiv*, abs/2402.15527, 2024a.
- Chen et al. [2024b]
Lin Chen, Xilin Wei, Jinsong Li, Xiao wen Dong, Pan Zhang, Yuhang Zang, Zehui
Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng
Zhao, and Jiaqi Wang.
Sharegpt4video: Improving video understanding and generation with
better captions.
*ArXiv*, abs/2406.04325, 2024b.
- Chen et al. [2024c]
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo
Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al.
Expanding performance boundaries of open-source multimodal models
with model, data, and test-time scaling.
*arXiv preprint arXiv:2412.05271*, 2024c.
- Corona et al. [2021]
Kellie Corona, Katie Osterdahl, Roderic Collins, and Anthony Hoogs.
Meva: A large-scale multiview, multimodal video dataset for activity
detection.
In *Proceedings of the IEEE/CVF Winter Conference on
Applications of Computer Vision (WACV)*, pages 1060–1068, January 2021.
- Cui et al. [2023]
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai
Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, Tianren Gao, Erlong Li, Kun
Tang, Zhipeng Cao, Tongxi Zhou, Ao Liu, Xinrui Yan, Shuqi Mei, Jianguo Cao,
Ziran Wang, and Chao Zheng.
A survey on multimodal large language models for autonomous driving.
*2024 IEEE/CVF Winter Conference on Applications of Computer
Vision Workshops (WACVW)*, pages 958–979, 2023.
- Doshi and Yilmaz [2020]
Keval Doshi and Yasin Yilmaz.
Any-shot sequential anomaly detection in surveillance videos.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition workshops*, pages 934–935, 2020.
- Fang et al. [2024]
Xinyu Fang, Kangrui Mao, Haodong Duan, Xiangyu Zhao, Yining Li, Dahua Lin, and
Kai Chen.
Mmbench-video: A long-form multi-shot benchmark for holistic video
understanding.
*ArXiv*, abs/2406.14515, 2024.
- Fu et al. [2023]
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin,
Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong
Ji.
Mme: A comprehensive evaluation benchmark for multimodal large
language models.
*ArXiv*, abs/2306.13394, 2023.
- Fu et al. [2024]
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan
Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li,
Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong
Ji, and Xing Sun.
Video-mme: The first-ever comprehensive evaluation benchmark of
multi-modal llms in video analysis.
*ArXiv*, abs/2405.21075, 2024.
- GLM et al. [2024]
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas,
Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun,
Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei
Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai
Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam
Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv,
Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An,
Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan
Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang.
Chatglm: A family of large language models from glm-130b to glm-4 all
tools, 2024.
- Hao et al. [2025]
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan
Wang, and Yu Cheng.
Can mllms reason in multimodality? emma: An enhanced multimodal
reasoning benchmark.
*ArXiv*, abs/2501.05444, 2025.
- Huang et al. [2025]
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaoshen Cao, Zheyu Ye, Fei Zhao, Zhe Xu,
Yao Hu, and Shaohui Lin.
Vision-r1: Incentivizing reasoning capability in multimodal large
language models.
*ArXiv*, abs/2503.06749, 2025.
- Kashef et al. [2021]
Mohamad Kashef, Anna Visvizi, and Orlando Troisi.
Smart city as a smart service system: Human-computer interaction and
smart city surveillance systems.
*Computers in Human Behavior*, 124:106923, 2021.
- Kim et al. [2025]
Byeong Su Kim, Jieun Kim, Deokwoo Lee, and Beakcheol Jang.
Visual question answering: A survey of methods, datasets, evaluation,
and challenges.
*ACM Computing Surveys*, 57(10):1–35, 2025.
- Lei et al. [2018]
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L. Berg.
Tvqa: Localized, compositional video question answering.
In *Conference on Empirical Methods in Natural Language
Processing*, 2018.
- Li et al. [2024a]
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen
Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al.
Llava-onevision: Easy visual task transfer.
*arXiv preprint arXiv:2408.03326*, 2024a.
- Li et al. [2024b]
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying
Shan.
Seed-bench: Benchmarking multimodal large language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 13299–13308, 2024b.
- Li and Lu [2024]
Jian Li and Weiheng Lu.
A survey on benchmarks of multimodal large language models.
*ArXiv*, abs/2408.08632, 2024.
- Li et al. [2024c]
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan
Xu, Guo Chen, Ping Luo, et al.
Mvbench: A comprehensive multi-modal video understanding benchmark.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 22195–22206, 2024c.
- Li et al. [2013]
Weixin Li, Vijay Mahadevan, and Nuno Vasconcelos.
Anomaly detection and localization in crowded scenes.
*IEEE transactions on pattern analysis and machine
intelligence*, 36(1):18–32, 2013.
- Li et al. [2025]
Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, and Guangyao Shi.
A survey of state of the art large vision language models: Alignment,
benchmark, evaluations and challenges.
2025.
- Liang et al. [2023]
Xiucheng Liang, Tianhong Zhao, and Filip Biljecki.
Revealing spatio-temporal evolution of urban visual environments with
street view imagery.
*Landscape and Urban Planning*, 237:104802, 2023.
- Ling et al. [2017]
Chih Wei Ling, Anwitaman Datta, and Jun Xu.
A case for distributed multilevel storage infrastructure for visual
surveillance in intelligent transportation networks.
*IEEE Internet Computing*, 22(1):42–51,
2017.
- Liu et al. [2023a]
Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao.
Mm-safetybench: A benchmark for safety evaluation of multimodal large
language models.
In *European Conference on Computer Vision*, 2023a.
- Liu et al. [2023b]
Y. Liu, Dingkang Yang, Yan Wang, Jing Liu, and Liang Song.
Generalized video anomaly event detection: Systematic taxonomy and
comparison of deep models.
*ACM Computing Surveys*, 56:1 – 38,
2023b.
URL [https://api.semanticscholar.org/CorpusID:256808510](https://api.semanticscholar.org/CorpusID:256808510).
- Liu et al. [2024a]
Yang Liu, Dingkang Yang, Yan Wang, Jing Liu, Jun Liu, Azzedine Boukerche, Peng
Sun, and Liang Song.
Generalized video anomaly event detection: Systematic taxonomy and
comparison of deep models.
*ACM Computing Surveys*, 56(7):1–38,
2024a.
- Liu et al. [2024b]
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike
Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.
Mmbench: Is your multi-modal model an all-around player?
In *European conference on computer vision*, pages 216–233.
Springer, 2024b.
- Lu et al. [2013]
Cewu Lu, Jianping Shi, and Jiaya Jia.
Abnormal event detection at 150 fps in matlab.
In *Proceedings of the IEEE international conference on computer
vision*, pages 2720–2727, 2013.
- Luo et al. [2017]
Weixin Luo, Wen Liu, and Shenghua Gao.
A revisit of sparse coding based anomaly detection in stacked rnn
framework.
In *Proceedings of the IEEE international conference on computer
vision*, pages 341–349, 2017.
- Ma et al. [2019]
Meiyi Ma, Sarah M Preum, Mohsin Y Ahmed, William Tärneberg, Abdeltawab
Hendawi, and John A Stankovic.
Data sets, modeling, and decision making in smart cities: A survey.
*ACM Transactions on Cyber-Physical Systems*, 4(2):1–28, 2019.
- Maaz et al. [2023]
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.
Video-chatgpt: Towards detailed video understanding via large vision
and language models.
*arXiv preprint arXiv:2306.05424*, 2023.
- Maaz et al. [2024]
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan.
Videogpt+: Integrating image and video encoders for enhanced video
understanding.
*arXiv preprint arXiv:2406.09418*, 2024.
- Nawaratne et al. [2020]
Rashmika Nawaratne, Damminda Alahakoon, Daswin de Silva, and Xinghuo Yu.
Spatiotemporal anomaly detection using deep learning for real-time
video surveillance.
*IEEE Transactions on Industrial Informatics*, 16:393–402, 2020.
- Pathirannahalage et al. [2024]
Iroshan Pathirannahalage, Vidura Jayasooriya, Jagath Samarabandu, and Akila
Subasinghe.
A comprehensive analysis of real-time video anomaly detection methods
for human and vehicular movement.
*Multimedia Tools and Applications*, pages 1–46, 2024.
- Pawar and Attar [2019]
Karishma Pawar and Vahida Attar.
Deep learning approaches for video-based anomalous activity
detection.
*World Wide Web*, 22(2):571–601, 2019.
- Sreenu and Durai [2019]
G. Sreenu and M. A. Saleem Durai.
Intelligent video surveillance: a review through deep learning
techniques for crowd analysis.
*Journal of Big Data*, 6, 2019.
- Sultani et al. [2018]
Waqas Sultani, Chen Chen, and Mubarak Shah.
Real-world anomaly detection in surveillance videos.
In *Proceedings of the IEEE conference on computer vision and
pattern recognition*, pages 6479–6488, 2018.
- Tsakanikas and Dagiuklas [2018]
Vassilios Tsakanikas and Tasos Dagiuklas.
Video surveillance systems-current status and future trends.
*Computers & Electrical Engineering*, 70:736–753,
2018.
- Wang et al. [2024a]
Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang,
Akshay Uttama Nambi, Tanuja Ganu, and Hao Wang.
Multimodal needle in a haystack: Benchmarking long-context capability
of multimodal large language models.
*ArXiv*, abs/2406.11230, 2024a.
- Wang et al. [2024b]
Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu,
Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang.
Exploring the reasoning abilities of multimodal large language models
(mllms): A comprehensive survey on emerging trends in multimodal reasoning.
*ArXiv*, abs/2401.06805, 2024b.
- Weng et al. [2024]
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang.
Longvlm: Efficient long video understanding via large language
models.
In *European Conference on Computer Vision*, 2024.
- Wu et al. [2024]
Peng Wu, Xuerong Zhou, Guansong Pang, Yujia Sun, Jing Liu, Peng Wang, and
Yanning Zhang.
Open-vocabulary video anomaly detection.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 18297–18307, 2024.
- Xiao et al. [2021]
Junbin Xiao, Xindi Shang, Angela Yao, and Tat seng Chua.
Next-qa: Next phase of question-answering to explaining temporal
actions.
*2021 IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR)*, pages 9772–9781, 2021.
- Xie et al. [2024]
Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel,
Jingkang Yang, and Ziwei Liu.
Funqa: Towards surprising video comprehension.
In *European Conference on Computer Vision*, pages 39–57.
Springer, 2024.
- Xu et al. [2016]
Zheng Xu, Chuanping Hu, and Lin Mei.
Video structured description technology based intelligence analysis
of surveillance videos for public security applications.
*Multimedia Tools and Applications*, 75:12155–12172,
2016.
- Yang et al. [2024]
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu,
Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang,
Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang
Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue,
Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang
Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu,
Zeyu Cui, Zhenru Zhang, and Zihan Qiu.
Qwen2.5 technical report.
*arXiv preprint arXiv:2412.15115*, 2024.
- Yang et al. [2025]
Zhenyu Yang, Yuhang Hu, Zemin Du, Dizhan Xue, Shengsheng Qian, Jiahong Wu, Fan
Yang, Weiming Dong, and Changsheng Xu.
Svbench: A benchmark with temporal multi-turn dialogues for streaming
video understanding.
*ArXiv*, abs/2502.10810, 2025.
- Yu et al. [2023]
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu,
Xinchao Wang, and Lijuan Wang.
Mm-vet: Evaluating large multimodal models for integrated
capabilities.
*arXiv preprint arXiv:2308.02490*, 2023.
- Yuan et al. [2024a]
Tongtong Yuan, Xuange Zhang, Bo Liu, Kun Liu, Jian Jin, and Zhenzhen Jiao.
Surveillance video-and-language understanding: from small to large
multimodal models.
*IEEE Transactions on Circuits and Systems for Video
Technology*, 2024a.
- Yuan et al. [2024b]
Tongtong Yuan, Xuange Zhang, Kun Liu, Bo Liu, Chen Chen, Jian Jin, and Zhenzhen
Jiao.
Towards surveillance video-and-language understanding: New dataset
baselines and challenges.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 22052–22061, 2024b.
- Zanella et al. [2024]
Luca Zanella, Willi Menapace, Massimiliano Mancini, Yiming Wang, and Elisa
Ricci.
Harnessing large language models for training-free video anomaly
detection.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition*, pages 18527–18536, 2024.
- Zhang et al. [2025]
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen,
Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al.
Videollama 3: Frontier multimodal foundation models for image and
video understanding.
*arXiv preprint arXiv:2501.13106*, 2025.
- Zhang et al. [2024a]
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi
Feng, Ziwei Liu, and Chunyuan Li.
Llava-next: A strong zero-shot video understanding model, April
2024a.
URL
[https://llava-vl.github.io/blog/2024-04-30-llava-next-video/](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/).
- Zhang et al. [2024b]
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li.
Video instruction tuning with synthetic data, 2024b.
URL [https://arxiv.org/abs/2410.02713](https://arxiv.org/abs/2410.02713).
- Zhou et al. [2019]
Joey Tianyi Zhou, Jiawei Du, Hongyuan Zhu, Xi Peng, Yong Liu, and Rick
Siow Mong Goh.
Anomalynet: An anomaly detection network for video surveillance.
*IEEE Transactions on Information Forensics and Security*,
14(10):2537–2550, 2019.
- Zhu et al. [2024]
Liyun Zhu, Lei Wang, Arjun Raj, Tom Gedeon, and Chen Chen.
Advancing video anomaly detection: A concise review and a new
dataset.
In *The Thirty-eighth Conference on Neural Information
Processing Systems Datasets and Benchmarks Track*, 2024.

## Appendix A Annotation Data Examples

## Appendix B Data Generation Details

**Table 7: Designed Prompts to guide question-answer pair generation.**
| Type | Task Description | Key Instructions | Example Questions |
| --- | --- | --- | --- |
| Anomaly QA |  |  |  |
| Anomaly Detection | Determine if an event related to violence, crime, or danger occurs. | Answer with ’Yes’ or ’No’ based on the presence of violence, crime, or danger. | Does this video contain any potentially violent or criminal activities? |
| Anomaly Classification | Identify and classify any detected dangerous event using predefined categories. | Classify the anomaly into categories like Abuse, Assault, etc., or return ’None’. | What type of abnormal event is present in the video? |
| Anomaly Subject | Identify the primary subject involved in the abnormal event. | List key subjects involved in the anomaly, or return ’None’ if no anomaly is detected. | Who is the main person involved in the unusual event? |
| Anomaly Description | Provide a detailed description of the detected anomaly, including setting and actions. | Describe the event, environment, and actions in detail, or return ’None’. | What is happening in the detected abnormal event? |
| Anomaly Cause | Logically infer the root cause of the detected abnormal event. | Analyze environmental factors and interactions to explain the cause, or return ’None’. | What led to the unusual event occurring in the video? |
| Anomaly Result | Infer and describe the outcome of the detected abnormal event. | Describe the consequence, situation evolution, and impacts, or return ’None’. | What happens as a result of the abnormal event? |
| Normal QA |  |  |  |
| Summary QA | Generate questions to extract a detailed description of the entire video content. | Generate three questions targeting the full sequence, with answers integrating all details. | Can you describe the entire video in detail from start to finish? |
| Generic QA | Generate questions focusing on significant aspects like appearance and motion. | Generate three questions on different aspects (appearance, motion, reasoning), with detailed answers. | Describe the entire process the person goes through from start to finish. |
| Temporal QA | Generate questions focusing on the sequence and timing of events. | Generate three questions using time references (beginning, middle, end), with answers based on the caption. | When does the main character start the primary task, and what leads up to it? |
| Short Temporal QA | Generate concise questions focusing on specific temporal events in the video. | Generate three questions on temporal aspects using approximate time references, with answers based on the caption. | When does event x happen in the video? |
| Spatial QA | Generate questions focusing on spatial details like colors and outfits. | Generate three questions on spatial aspects (colors, attire, location), with detailed answers. | What is the color of the woman’s shirt? |
| Reasoning QA | Generate questions focusing on actions, objects, and reasoning behind events. | Generate three questions on actions, objects, and reasoning, with concise answers including context. | Why did the player throw the ball? |

For this normal case, the generated JSON file includes the following QA categories:

- •
summary_qa_pairs: Cover the full narrative of the video, summarizing the entire sequence of events.
- •
generic_qa_pairs: Focus on essential visual and behavioral information, including appearance, actions, trajectories, and inferred intentions.
- •
temporal_qa_pairs: Address the order and timing of events, using general time references (e.g., beginning, middle, end).
- •
spatial_qa_pairs: Explore spatial details such as clothing colors, physical positions, and scene layout.
- •
reasoning_qa_pairs: Emphasize causal and inferential questions related to actions, locations, and motivations ("what," "where," "why").
- •
short_temporal_qa_pairs: Provide concise questions about specific moments or transitions within the video.

For abnormal video segments, six QA categories are defined, with one QA pair per category to ensure a concise yet focused representation of key anomalous elements. Taking the MSAD video "MSAD_anomaly_videos_Shooting_169.mp4" as an example, which depicts a shooting incident in a street setting, the scene involves multiple vehicles (a silver SUV, white SUV, and motorcycle) and several individuals (including men and women dressed in black). The video captures a violent exchange involving a firearm threat and return fire.

The corresponding JSON file for this abnormal case includes the following QA categories and content:

- •
get_anomaly_detection: Determines whether the video contains violent, criminal, or dangerous events.
- •
get_anomaly_classification_prompt: Classifies the anomaly type—in this case, as a shooting.
- •
get_anomaly_subject: Identifies the key individuals involved in the anomaly—two men on a motorcycle (initiators with firearms) and a woman who returns fire.
- •
get_anomaly_description: Provides a detailed account of the shooting (focused between 35.1–41.0 seconds), describing the environment (street), appearances (black clothing), and actions (threatening, shooting, fleeing).
- •
get_anomaly_cause: Infers the likely cause of the anomaly—here, the armed threat initiated by the motorcyclists.
- •
get_anomaly_result: Analyzes the consequence or outcome of the anomalous event.

This streamlined QA structure for abnormal segments allows the dataset to efficiently highlight critical details necessary for real-time anomaly detection and situational understanding, contrasting with the more comprehensive QA setup used for normal videos.

## Appendix C More Data statistics

Figure [4](https://arxiv.org/html/2505.12589v1#A3.F4) illustrates the distribution of event durations on the training/test sets in this dataset. The majority of events are concentrated between 5 and 20 seconds, with a particularly high concentration in the 5-10 seconds and 10-15 seconds intervals. Notably, normal events in the training set dominate the distribution, making up the largest proportion of the dataset.

Figure: Figure 4: Distribution of event durations on the training/test sets
Refer to caption: extracted/6450145/Duration_Distribution.png

Regarding video event categorization (as shown in Table [8](https://arxiv.org/html/2505.12589v1#A3.T8)), in addition to the Normal category representing non-anomalous cases, we establish a classification system comprising 18 distinct abnormal categories for the Abnormal class. These include: Abuse, Arrest, Assault, Burglary, Explosion, Fighting, Fire, Object Falling, People Falling, Pursuit, Robbery, Shooting, Shoplifting, Stealing, Traffic Accident, Threat, Vandalism, and Water Incident—covering a wide range of incidents from violent behaviors (e.g., Assault, Fighting) to environmental hazards (e.g., Fire, Water Incident).

Notably, we have observed that some abnormal categories generated by Qwen-Max exhibit redundancy or lack general applicability. To improve the accuracy of statistical analysis and standardize the taxonomy, we perform category consolidation and refinement. For instance, “Chasing” and “Chase” are unified under Pursuit due to semantic equivalence, while overly broad categories such as “Emergency Situation” and “Weapon Present” are excluded from statistical summaries due to their ambiguity in defining concrete abnormal behaviors. It is important to emphasize that this refinement is applied only during the statistical analysis phase, and the original Qwen-Max outputs remain unaltered.

**Table 8: Statistics on abnormal categories and numbers**
| Category | MEVA | MSAD | NWPU | UCA |
| --- | --- | --- | --- | --- |
| normal | 2044 | 1417 | 4121 | 20384 |
| Abuse | 0 | 1 | 0 | 138 |
| Arrest | 0 | 2 | 0 | 143 |
| Assault | 0 | 38 | 4 | 451 |
| Burglary | 0 | 7 | 4 | 204 |
| Explosion | 0 | 15 | 0 | 73 |
| Fighting | 1 | 34 | 6 | 400 |
| Fire | 0 | 52 | 0 | 217 |
| Object Falling | 1 | 55 | 1 | 101 |
| People Falling | 3 | 109 | 9 | 570 |
| Pursuit | 0 | 1 | 1 | 19 |
| Robbery | 1 | 63 | 7 | 575 |
| Shooting | 0 | 15 | 0 | 56 |
| Shoplifting | 0 | 0 | 0 | 26 |
| Stealing | 2 | 5 | 13 | 250 |
| Traffic Accident | 4 | 52 | 2 | 441 |
| Threat | 0 | 1 | 0 | 7 |
| Vandalism | 2 | 25 | 2 | 255 |
| Water Incident | 0 | 6 | 0 | 6 |

## Appendix D More Experiment Results

More results with five metrics including CI, DO, CU, TU, AVG, across normal QA tasks normal video clips
have been shown in Table [9](https://arxiv.org/html/2505.12589v1#A4.T9).

**Table 9: Performance of different vision-language models across normal QA tasks on normal video clips. All values are shown in blue, with the highest value in each row bolded. ^† represents our finetuned LVLMs.**
| Task | Metric | LLavA-OV- | InternVL | VideoLLa | Qwen2.5-VL- | LLaVA- | LLavA-OV- | LLaVA- | VideoLLa | Qwen2.5-VL- |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2-0.5B | 2.5-2B | MA3-2B | 3B-Instruct | Next-7B | Qwen2-7B-ov | Video-7B | MA3-7B | 3B-Instruct^† |  |  |
| SummaryQA | CI | 2.95 | 0.45 | 2.65 | 2.29 | 2.32 | 3.34 | 3.30 | 2.82 | 2.77 |
| DO | 2.63 | 0.48 | 2.35 | 2.12 | 2.08 | 2.92 | 2.78 | 2.62 | 2.53 |  |
| CU | 2.88 | 0.75 | 2.64 | 2.36 | 2.34 | 3.21 | 3.22 | 2.79 | 2.73 |  |
| TU | 2.61 | 0.55 | 2.33 | 2.03 | 1.93 | 2.91 | 2.91 | 2.51 | 2.45 |  |
| Avg. | 2.76 | 0.56 | 2.49 | 2.20 | 2.17 | 3.10 | 3.05 | 2.68 | 2.62 |  |
| Generic QA | CI | 2.97 | 2.27 | 3.02 | 2.83 | 3.11 | 3.27 | 3.19 | 3.20 | 2.92 |
| DO | 2.60 | 2.13 | 2.70 | 2.56 | 2.86 | 2.91 | 2.79 | 2.87 | 2.67 |  |
| CU | 2.90 | 2.32 | 2.94 | 2.77 | 3.03 | 3.17 | 3.10 | 3.11 | 2.85 |  |
| TU | 2.65 | 2.06 | 2.70 | 2.50 | 2.73 | 2.94 | 2.88 | 2.87 | 2.58 |  |
| Avg. | 2.78 | 2.20 | 2.84 | 2.66 | 2.93 | 3.08 | 2.99 | 3.01 | 2.76 |  |
| Temporal QA | CI | 2.73 | 1.83 | 2.84 | 2.77 | 2.68 | 3.08 | 2.98 | 2.91 | 2.97 |
| DO | 2.52 | 1.90 | 2.66 | 2.61 | 2.50 | 2.85 | 2.75 | 2.70 | 2.80 |  |
| CU | 2.73 | 2.03 | 2.83 | 2.75 | 2.67 | 3.03 | 2.95 | 2.89 | 2.92 |  |
| TU | 2.48 | 1.80 | 2.61 | 2.51 | 2.38 | 2.84 | 2.74 | 2.72 | 2.68 |  |
| Avg. | 2.62 | 1.89 | 2.73 | 2.66 | 2.56 | 2.95 | 2.85 | 2.81 | 2.84 |  |
| Short Temporal | CI | 2.71 | 2.04 | 2.73 | 2.79 | 2.70 | 2.92 | 2.90 | 2.76 | 3.04 |
| DO | 2.48 | 2.00 | 2.45 | 2.66 | 2.67 | 2.63 | 2.65 | 2.46 | 2.93 |  |
| CU | 2.76 | 2.27 | 2.74 | 2.80 | 2.78 | 2.92 | 2.94 | 2.79 | 3.00 |  |
| TU | 2.46 | 2.00 | 2.51 | 2.52 | 2.50 | 2.68 | 2.65 | 2.56 | 2.75 |  |
| Avg. | 2.60 | 2.08 | 2.61 | 2.69 | 2.66 | 2.79 | 2.78 | 2.65 | 2.93 |  |
| Spatial QA | CI | 3.22 | 1.90 | 3.07 | 2.94 | 3.03 | 3.41 | 3.60 | 3.16 | 2.97 |
| DO | 2.96 | 1.83 | 2.80 | 2.78 | 2.88 | 3.17 | 3.30 | 2.91 | 2.85 |  |
| CU | 3.23 | 2.10 | 3.10 | 2.97 | 3.06 | 3.41 | 3.57 | 3.18 | 2.98 |  |
| TU | 3.02 | 1.85 | 2.89 | 2.74 | 2.83 | 3.23 | 3.40 | 2.98 | 2.76 |  |
| Avg. | 3.10 | 1.92 | 2.97 | 2.86 | 2.95 | 3.31 | 3.47 | 3.06 | 2.89 |  |
| Reasoning QA | CI | 3.05 | 2.45 | 3.02 | 3.11 | 3.19 | 3.22 | 3.39 | 3.13 | 3.14 |
| DO | 2.75 | 2.29 | 2.69 | 2.94 | 2.97 | 2.87 | 3.05 | 2.80 | 3.06 |  |
| CU | 3.07 | 2.57 | 3.05 | 3.13 | 3.19 | 3.23 | 3.38 | 3.15 | 3.14 |  |
| TU | 2.84 | 2.32 | 2.81 | 2.85 | 2.90 | 3.00 | 3.18 | 2.91 | 2.90 |  |
| Avg. | 2.93 | 2.41 | 2.89 | 3.01 | 3.06 | 3.08 | 3.25 | 2.99 | 3.06 |  |

More results with five metrics including CI, DO, CU, TU, AVG, across normal QA tasks on abnormal video clips
have been shown in Table [10](https://arxiv.org/html/2505.12589v1#A4.T10).

**Table 10: Performance of different vision-language models across normal QA tasks on abnormal video clips. All values are shown in green, with the highest value in each row bolded. ^† represents our finetuned LVLMs.**
| Task | Metric | LLavA-OV- | InternVL | VideoLLa | Qwen2.5-VL- | LLaVA- | LLavA-OV- | LLaVA- | VideoLLa | Qwen2.5-VL- |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2-0.5B | 2.5-2B | MA3-2B | 3B-Instruct | Next-7B | Qwen2-7B-ov | Video-7B | MA3-7B | 3B-Instruct^† |  |  |
| Summary QA | CI | 2.56 | 0.22 | 2.06 | 1.46 | 1.78 | 2.97 | 3.03 | 2.35 | 2.05 |
| DO | 2.35 | 0.26 | 1.92 | 1.46 | 1.68 | 2.69 | 2.63 | 2.28 | 1.96 |  |
| CU | 2.56 | 0.60 | 2.17 | 1.68 | 1.90 | 2.90 | 3.02 | 2.43 | 2.14 |  |
| TU | 2.24 | 0.40 | 1.84 | 1.38 | 1.52 | 2.61 | 2.72 | 2.11 | 1.82 |  |
| Avg. | 2.43 | 0.37 | 2.00 | 1.49 | 1.72 | 2.79 | 2.85 | 2.29 | 1.99 |  |
| Generic QA | CI | 2.66 | 1.95 | 2.68 | 2.23 | 2.72 | 3.01 | 2.96 | 2.83 | 2.43 |
| DO | 2.41 | 1.90 | 2.48 | 2.15 | 2.58 | 2.74 | 2.63 | 2.63 | 2.35 |  |
| CU | 2.65 | 2.08 | 2.67 | 2.29 | 2.71 | 2.97 | 2.94 | 2.81 | 2.47 |  |
| TU | 2.37 | 1.80 | 2.39 | 2.01 | 2.39 | 2.71 | 2.68 | 2.53 | 2.18 |  |
| Avg. | 2.52 | 1.93 | 2.55 | 2.17 | 2.60 | 2.86 | 2.80 | 2.70 | 2.36 |  |
| Temporal QA | CI | 2.50 | 1.57 | 2.53 | 2.26 | 2.31 | 2.89 | 2.86 | 2.61 | 2.51 |
| DO | 2.38 | 1.69 | 2.47 | 2.27 | 2.29 | 2.74 | 2.68 | 2.50 | 2.52 |  |
| CU | 2.57 | 1.84 | 2.59 | 2.37 | 2.40 | 2.90 | 2.88 | 2.65 | 2.57 |  |
| TU | 2.29 | 1.60 | 2.34 | 2.12 | 2.10 | 2.69 | 2.65 | 2.44 | 2.33 |  |
| Avg. | 2.43 | 1.68 | 2.49 | 2.26 | 2.28 | 2.81 | 2.77 | 2.55 | 2.48 |  |
| Short Temporal | CI | 2.51 | 1.89 | 2.52 | 2.33 | 2.48 | 2.75 | 2.77 | 2.53 | 2.61 |
| DO | 2.36 | 1.89 | 2.30 | 2.32 | 2.51 | 2.52 | 2.60 | 2.30 | 2.65 |  |
| CU | 2.61 | 2.15 | 2.60 | 2.44 | 2.63 | 2.82 | 2.87 | 2.61 | 2.70 |  |
| TU | 2.31 | 1.88 | 2.35 | 2.15 | 2.30 | 2.53 | 2.60 | 2.37 | 2.40 |  |
| Avg. | 2.45 | 1.95 | 2.44 | 2.31 | 2.48 | 2.66 | 2.71 | 2.45 | 2.59 |  |
| Spatial QA | CI | 3.23 | 1.85 | 3.09 | 2.83 | 3.03 | 3.45 | 3.65 | 3.17 | 3.00 |
| DO | 2.98 | 1.79 | 2.83 | 2.68 | 2.89 | 3.20 | 3.33 | 2.91 | 2.87 |  |
| CU | 3.23 | 2.09 | 3.11 | 2.86 | 3.06 | 3.43 | 3.62 | 3.19 | 2.97 |  |
| TU | 3.04 | 1.82 | 2.92 | 2.64 | 2.83 | 3.27 | 3.45 | 2.99 | 2.76 |  |
| Avg. | 3.12 | 1.89 | 2.99 | 2.75 | 2.95 | 3.34 | 3.51 | 3.06 | 2.89 |  |
| Reasoning QA | CI | 2.79 | 2.26 | 2.74 | 2.75 | 2.92 | 3.00 | 3.15 | 2.86 | 2.87 |
| DO | 2.53 | 2.16 | 2.48 | 2.65 | 2.72 | 2.70 | 2.87 | 2.60 | 2.86 |  |
| CU | 2.83 | 2.43 | 2.84 | 2.84 | 2.95 | 3.04 | 3.21 | 2.95 | 3.00 |  |
| TU | 2.60 | 2.17 | 2.57 | 2.57 | 2.64 | 2.81 | 2.98 | 2.70 | 2.72 |  |
| Avg. | 2.69 | 2.26 | 2.66 | 2.70 | 2.81 | 2.89 | 3.05 | 2.78 | 2.85 |  |

More results with five metrics including CI, DO, CU, TU, AVG, across Normal QA tasks on abnormal video clips
have been shown in Table [11](https://arxiv.org/html/2505.12589v1#A4.T11).

**Table 11: Performance of different vision-language models across abnormal QA tasks on abnormal video clips. All values are shown in brown, with the highest value in each row bolded. ^† represents our finetuned LVLMs.**
| Task | Metric | LLavA-OV- | InternVL | VideoLLa | Qwen2.5-VL- | LLaVA- | LLavA-OV- | LLaVA- | VideoLLa | Qwen2.5-VL- |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Qwen2-0.5B | 2.5-2B | MA3-2B | 3B-Instruct | Next-7B | Qwen2-7B-ov | Video-7B | MA3-7B | 3B-Instruct^† |  |  |
| Detection QA | CI | 3.10 | 1.97 | 1.90 | 1.92 | 2.39 | 2.54 | 1.95 | 2.07 | 1.57 |
| DO | 2.69 | 1.71 | 1.81 | 1.93 | 2.51 | 2.50 | 1.85 | 1.97 | 1.75 |  |
| CU | 3.11 | 2.00 | 1.94 | 1.88 | 2.29 | 2.54 | 1.96 | 2.11 | 1.54 |  |
| TU | 2.86 | 1.83 | 1.81 | 1.69 | 2.09 | 2.52 | 1.90 | 1.95 | 1.45 |  |
| Avg. | 2.94 | 1.88 | 1.87 | 1.85 | 2.32 | 2.53 | 1.92 | 2.03 | 1.58 |  |
| Classification QA | CI | 2.36 | 0.86 | 1.88 | 2.00 | 2.47 | 2.52 | 2.47 | 1.37 | 1.90 |
| DO | 2.53 | 1.22 | 2.07 | 2.40 | 2.74 | 2.61 | 2.72 | 1.49 | 2.42 |  |
| CU | 2.55 | 1.36 | 2.22 | 2.30 | 2.63 | 2.77 | 2.71 | 1.84 | 2,18 |  |
| TU | 2.43 | 1.05 | 1.94 | 2.05 | 2.41 | 2.48 | 2.45 | 1.41 | 2.01 |  |
| Avg. | 2.47 | 1.12 | 2.03 | 2.19 | 2.56 | 2.60 | 2.59 | 1.53 | 2.13 |  |
| Subject QA | CI | 2.76 | 1.30 | 2.49 | 2.24 | 2.69 | 2.75 | 2.83 | 2.71 | 2.55 |
| DO | 2.52 | 1.46 | 2.32 | 2.23 | 2.56 | 2.51 | 2.59 | 2.53 | 2.69 |  |
| CU | 2.87 | 1.74 | 2.64 | 2.43 | 2.75 | 2.86 | 2.93 | 2.83 | 2.66 |  |
| TU | 2.62 | 1.36 | 2.38 | 2.07 | 2.36 | 2.62 | 2.71 | 2.55 | 2.35 |  |
| Avg. | 2.69 | 1.46 | 2.46 | 2.24 | 2.59 | 2.68 | 2.76 | 2.66 | 2.56 |  |
| Description QA | CI | 2.56 | 0.35 | 1.98 | 1.76 | 2.21 | 2.71 | 2.92 | 2.05 | 2.11 |
| DO | 2.28 | 0.46 | 1.73 | 1.68 | 2.06 | 2.40 | 2.58 | 1.90 | 2.03 |  |
| CU | 2.61 | 0.81 | 2.10 | 1.88 | 2.27 | 2.67 | 2.92 | 2.23 | 2.17 |  |
| TU | 2.27 | 0.55 | 1.76 | 1.64 | 1.92 | 2.39 | 2.62 | 1.88 | 1.91 |  |
| Avg. | 2.43 | 0.55 | 1.89 | 1.74 | 2.11 | 2.54 | 2.76 | 2.01 | 2.06 |  |
| Cause QA | CI | 1.69 | 0.46 | 1.31 | 1.29 | 2.03 | 1.51 | 1.56 | 1.69 | 1.64 |
| DO | 1.49 | 0.55 | 1.23 | 1.28 | 1.87 | 1.16 | 1.24 | 1.71 | 1.69 |  |
| CU | 1.87 | 0.87 | 1.55 | 1.40 | 2.06 | 1.66 | 1.72 | 1.86 | 1.79 |  |
| TU | 1.62 | 0.69 | 1.38 | 1.29 | 1.86 | 1.51 | 1.54 | 1.71 | 1.63 |  |
| Avg. | 1.67 | 0.64 | 1.37 | 1.32 | 1.96 | 1.46 | 1.52 | 1.75 | 1.69 |  |
| Result QA | CI | 1.62 | 0.53 | 1.20 | 1.08 | 1.73 | 1.63 | 1.89 | 1.25 | 1.25 |
| DO | 1.39 | 0.63 | 0.91 | 1.03 | 1.49 | 1.22 | 1.56 | 0.99 | 1.19 |  |
| CU | 1.84 | 0.99 | 1.47 | 1.35 | 1.87 | 1.85 | 2.07 | 1.56 | 1.53 |  |
| TU | 1.54 | 0.80 | 1.15 | 1.09 | 1.51 | 1.51 | 1.74 | 1.26 | 1.24 |  |
| Avg. | 1.60 | 0.74 | 1.18 | 1.13 | 1.65 | 1.55 | 1.81 | 1.27 | 1.30 |  |

Figure [5](https://arxiv.org/html/2505.12589v1#A4.F5) shows the direct score comparisons of different LVLMs across different QA tasks.

Figure: Figure 5: Comparisons of scores from different LVLMs across different QA tasks. Left: normal video clips vs. normal QA tasks. Middle: abnormal video clips vs. normal QA tasks. Right: abnormal video clips vs. abnormal QA tasks.
Refer to caption: x3.png

Figure [6](https://arxiv.org/html/2505.12589v1#A4.F6) shows an overall performance rank of LVLMs across different QA tasks.

Figure: Figure 6: Performance rankings of LVLMs across different QA tasks. No.1-6 lists denote normal video clips vs. normal QA tasks. No.7-12 lists denote abnormal video clips vs. normal QA tasks. No.13-18 lists denote abnormal video clips vs. abnormal QA tasks.
Refer to caption: extracted/6450145/listresults.png

## Appendix E Visualization for inference results of LVLMs.

Figure [7](https://arxiv.org/html/2505.12589v1#A5.F7) shows an example of visualization for inference results of LVLMs. All of the LVLMs provide wrong answers for abnormal QA tasks.

Figure: Figure 7: Abnormal videos QA Examples. All of the LVLMs provide wrong answers.
Refer to caption: x4.png

## Appendix F Limitations

However, this study also has some limitations. Model evaluations are based on a limited dataset, and performance may vary when applied to different surveillance environments or low-quality video data. Although larger models perform well in terms of accuracy, they have high computational demands, which may make them challenging to deploy in resource-constrained environments. Future research should focus on optimizing the temporal reasoning capabilities of smaller models and further explore domain-specific fine-tuning techniques to improve performance in various monitoring scenarios.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1) Could the authors clarify the fundamental differences between SurveillanceVQA and existing surveillance QA datasets such as UCA? Beyond scaling up and changing the evaluation protocol, is there any substantial innovation in task formulation or domain modeling?

2) Surveillance videos have distinctive properties such as long temporal spans, low resolution, small targets, and fixed viewpoints. Did the authors consider explicitly incorporating these characteristics into task design or evaluation metrics?

3) Since LLM-as-a-judge is inherently subjective, did the authors conduct any prompt sensitivity or repeated scoring tests to assess its stability and consistency?

4) Have the authors considered introducing more structured or partially objective evaluation components to improve reproducibility and diagnostic power?

5) Is there any analysis or ablation study that examines how surveillance-specific factors (e.g., temporal grounding challenges, small-object difficulty) influence QA performance, to demonstrate the task’s unique surveillance characteristics?

There are many benchmark papers nowadays, and I do believe such work plays an important role in advancing the field. However, what matters more to me is whether the authors clearly articulate and deeply understand the relationship between data, scenario characteristics, and the capabilities being evaluated. Simply scaling up without capturing these key aspects is less convincing to me. That said, I genuinely look forward to the authors’ responses and deeper reflections on these essential issues.
soundness: 2
strengths: A key strength of this paper is its attempt to extend video question answering into the practically important domain of surveillance. The proposed large-scale dataset and the use of open-ended QA with a hybrid generation strategy reflect a degree of originality in task formulation. The paper is well structured, and the methodology and evaluation framework are clearly described. The experimental section includes multiple mainstream multimodal models, offering a broad view of current model performance in this setting. Given the practical significance of surveillance scenarios, this benchmark has potential impact for both research and applied domains.
summary: This paper focuses on video understanding in surveillance scenarios and introduces a large-scale benchmark dataset called SurveillanceVQA-589K. The dataset includes over 580K open-ended QA pairs covering both normal and abnormal events, aiming to support semantic and temporal reasoning research in surveillance contexts. The authors use a hybrid of human and llm-generated questions, and evaluate model performance through an llm-based scoring system. Several existing multimodal models are benchmarked to demonstrate the task’s difficulty and provide baseline results.
weaknesses: This paper has several notable weaknesses. 

First, the originality and incremental contribution are limited. The main differences from existing surveillance QA datasets, such as UCA, lie primarily in scaling up the dataset and changing the evaluation protocol, rather than introducing any fundamental advances in task design or domain modeling. 

Second, the open-ended QA format places the task awkwardly between captioning and traditional VQA, failing to provide either the strict semantic alignment of captioning tasks or the controllability and diagnostic power of structured VQA. This ambiguity weakens the scientific rigor and interpretability of the task itself. 

Third, the evaluation relies entirely on LLM-as-a-judge, which is inherently subjective and highly sensitive to prompt wording and model versions. This undermines reproducibility and makes it difficult to attribute errors or analyze model capabilities meaningfully. 

Fourth, the work does not effectively exploit the distinctive properties of surveillance videos (such as fixed viewpoints, low resolution, small targets, long temporal spans, and sparse anomalies) to inform task or evaluation design. Thus, the claimed “surveillance-specific” aspect remains superficial. 

Overall, the contribution is largely an engineering-scale extension rather than a conceptual or methodological breakthrough. Strengthening task definition, grounding the design in surveillance-specific characteristics, and adopting more robust, structured evaluation strategies would significantly improve the work.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1.	In Table 4, does finetuning with normal video QA data help abnormal video QA? It would be interesting to see such analyses.
2.	What are the results of API-called models for normal video qa?
3.	Can the benchmarked challenge simply be solved by collecting more data for training? If not, what challenges cannot be well solved?
soundness: 2
strengths: 1.	It constructs a large-scale dataset for comprehensive cross-modal surveillance video understanding. The dataset could be of significant value for social security.
2.	It provides well-classified questions, which would benefit model analysis.
3.	It provides results of SOTA LVLMs, analyze their success and failure points, and gives suggestions for future research.
summary: This paper introduces a VideoQA benchmark for comprehensive surveillance video understanding. The dataset comprises both normal and abnormal video clips and related questions. The questions are AI generated based on video narratives annotated by human or generated by AI Models. It then provides benchmark results of modern large multimodal models and analyzes their strengths and weakness from multiple aspects based on question categories. The findings point to better causal reasoning as well as more data for finetuning.
weaknesses: 1.	The dataset seems to be biased to LLaVA and Qwen series of modes, since LLaVA-Video and Qwen-Max are employed for caption and QA generation respectively. The results in Table 3 and 4 show that LLaVA-Video outperform other opensource models, which further confirms my concern.
2.	The QAs are directly generated by Qwen-max, without further human checking for answer correctness. This raises concern about QA quality.
3.	Some basic data statistics should be moved to the main text, for example, what is the size (number of videos and QAs) of the train and test set？
4.	Figure 4 (b), while the title shows QA example, there is no answer for the question.
5.	The evaluation metric is vague. It is unknow how well the judgement aligns with human reviewers.
6.	The current analyses of model behaviors in Table 3 and 4 are unconvincing. The authors ignore an important fact that the QAs are generated by LLaVA-Video and Qwen-Max, which could give strong priors for LLaVA and Qwen series of models during testing. Also, why not provide the results of Qwen2.5-VL 7B? Additionally, a human performance would be helpful to better understand the model behaviors and the soundness of the evaluation metric.
7.	I suggest the authors to provide the average results of each model for better understanding in Table 4.
8.	The suggestion about researching on casual reasoning and domain-adaptive pretraining is not interesting. Many other datasets can support such conclusions. It would be better to summarize more specific challenges and suggestions for surveillance video understanding.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: - Is there any analysis on how much the consolidated caption (or ultimately the generated QA) rely on the human or model caption? e.g. can the QA be answered from one of these two captions?
soundness: 2
strengths: - Large scale of this dataset compared to prior work
- Evaluation including both multiple open source and proprietary models
summary: This paper introduces SurveillanceVQA-589K, a large-scale benchmark dataset for Video QA designed for the surveillance domains, with 589K QA pairs about 12 diverse question types. The collection is based on a hybrid annotation pipeline which combines human-written captions with LLM assistance to generate the QA pairs at scale. The paper evaluates 12 different models on this new benchmark, notably showing that current models struggle with complex reasoning, especially in identifying the causes of abnormal events and predicting their results. The authors also show that fine-tuning models on this dataset provides some gains, but doesn't solve the core reasoning challenges and can lead to catastrophic forgetting.
weaknesses: - Simple baselines like text-only, single-image and human evaluation are not presented and would help assessing the benchmark.
- The fact that fine-tuning on the data does not help much is worrying and could suggest issue in the generated data (e.g. lack of diversity of the generated QAs).

---

confidence: 2
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. How much time is required to complete the full annotation pipeline?
soundness: 2
strengths: 1. The paper is clearly written and well-organized.
2. The proposed SurveillanceVQA-589K benchmark seems to extend beyond simple descriptive tasks to include higher-level reasoning such as logical reasoning, causal inference, and complex semantic comprehension of video content.
3. The authors perform a comprehensive experimental evaluation using a diverse set of LVLMs, providing valuable insights into their performance and limitations in surveillance-related scenarios.
summary: This paper addresses the underexplored area of surveillance video understanding within the vision-language research community. To this end, the authors introduce SurveillanceVQA-589K, a large-scale open-ended video question answering (VQA) benchmark specifically designed for the surveillance domain. The dataset is constructed through a semi-automated pipeline that combines human-written captions with outputs from large vision-language models. It encompasses diverse reasoning dimensions, including temporal reasoning, causal inference, spatial understanding, and anomaly interpretation, covering both normal and abnormal surveillance scenarios. Using this benchmark, the authors demonstrate that existing vision-language models exhibit significant limitations in handling real-world surveillance tasks, particularly those involving causal reasoning and anomaly detection.
weaknesses: 1. Although the benchmark includes diverse question types such as causal, temporal, and spatial reasoning, these dimensions have already been explored in prior video understanding benchmarks (e.g., VideoMME [1], MVBench [2]). The main novelty thus lies primarily in the surveillance domain rather than in the reasoning taxonomy itself, which somewhat limits the originality of the contribution. To enhance its distinctiveness, the benchmark could introduce evaluation dimensions specifically tailored to abnormal or security-critical scenarios in surveillance contexts (e.g., predictive reasoning about potential incidents or prevention strategies), rather than directly adapting question types from existing datasets.

2. The quantitative differences among evaluated LVLMs are relatively small, raising concerns about the sensitivity and reliability of the proposed evaluation protocol. Incorporating a subset of structured question types, such as multiple-choice or constrained-answer formats, could improve the robustness and interpretability of performance comparisons across models.

3. The paper lacks a thorough analysis explaining why the fine-tuned models yield modest performance gains compared to their zero-shot counterparts, despite being trained on a substantial number of samples. Simply attributing this to overfitting is insufficient without empirical evidence or diagnostic analysis. A deeper investigation-such as examining data distribution shifts, label noise, or domain mismatch between training and evaluation splits-would strengthen the credibility of this claim and provide more meaningful insights into model behavior.


[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

[2] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark