Do LVLMs Truly Understand Video Anomalies? Revealing Hallucination via Co-Occurrence Patterns
Authors: Menghao Zhang
Huazheng Wang
Pengfei Ren
Kangheng Lin
Qi Qi
Haifeng Sun
Zirui Zhuang
Lei Zhang
Jianxin Liao
Jingyu Wang
Venue: NeurIPS 2025 poster ()
Keywords: Video Anomaly Detection
Video Anomaly Reasoning
Primary area: applications
TL;DR: This paper reveals that LVLMs in video anomaly detection rely on pre-trained statistical shortcuts instead of scene-aware reasoning.

Abstract: Large Vision-Language Models (LVLMs) pretrained on large-scale multimodal data have shown promising capabilities in Video Anomaly Detection (VAD). However, their ability to reason about abnormal events based on scene semantics remains underexplored. In this paper, we investigate LVLMs’ behavior in VAD from a visual-textual co-occurrence perspective, focusing on whether their decisions are driven by statistical shortcuts between visual instances and textual phrases. By analyzing visual-textual co-occurrence in pretraining data and conducting experiments under different data settings, we reveal a hallucination phenomenon: LVLMs tend to rely on co-occurrence patterns between visual instances and textual phrases associated with either normality or abnormality, leading to incorrect predictions when these high-frequency objects appear in semantically mismatched contexts. To address this issue, we propose VAD-DPO, a direct preference optimization method supervised with counter-example pairs. By constructing visually similar but semantically contrasting video clips, VAD-DPO encourages the model to align its predictions with the semantics of scene rather than relying on co-occurrence patterns. Extensive experiments on six benchmark datasets demonstrate the effectiveness of VAD-DPO in enhancing both anomaly detection and reasoning performance, particularly in scene-dependent scenarios.

Do LVLMs Truly Understand Video Anomalies? Revealing Hallucination via Co-Occurrence Patterns
Menghao Zhang Huazheng Wang!* Pengfei Ren'
Kangheng Lin' Qi Qi'
Haifeng Sun Zirui Zhuang' Lei Zhang² Jianxin Liao' Jingyu Wang"
State Key Laboratory of Networking and Switching Technology,
Beijing University of Posts and Telccommunications, Beijing, China
"China Unicom, Beijing. China
(zhangmenghao, wanghz, rpf, linkangheng, qiqi8266, hfsun)]bupt..eem edu cn fzhuangzirui, liaojx, wangjingyu Sbupt. edu. en; zhangl83@chinaumiccm
Abstract
Large Vision-Language Models (LVLMss) pretrained on large-scale multimodal data have shown promising capabilities in Video Anomaly Detection (VAD). However, their ability reason about abnormal events based on scene semanties remains underexplored. In this paper, we investigate LVLMs' behavior in VAD from a visual-textual co-occurrence perspective, focusing whether their decisions are driven by statistical shortcuts between visual instances and textual phrases. By ana- lyzing visual-textual co-occurrence pretraining data and conducting experiments under different data settings, we reveal hallucination phenomenon: LVLMs tend to rely on co-occurrence patterns between visual instances and textual phrases associated with either normality abnormality, leading to incorrect predictions when these high-frequency objects appear semantically mismatched contexts. To address this issue, we propose VAD-DPO, a direct preference optimization method supervised with counter-example pairs. By constructing visually similar but semantically contrasting video clips, VAD-DPO encourages the model to align its predictions with the semantics scene rather than relying on co-occurrence patterns. Extensive experiments six benchmark datasets demonstrate the ef- fectiveness of VAD-DPO in enhancing both anomaly detection and reasoning performance, particularly in scene-dependent scenarios.
Introduction
Video Anomaly Detection (VAD) aims automatically identifyy anomalous events in given video, and has been widely applied surveillance systems and smart city applications. Anomalies are typically defined as events patterns that deviate from expected behaviors within scene, meaning that the same visual action may appear normal in one context but anomalous in another. However,
due to the limited availability of labeled anomaly data, most existing methods 35 40
46
41 tend
to overfit the normal patterns present in the training data, and thus fail to interpret scene context and reason about anomalies.
several studies explore the application of LVLMs to VAD, aiming to enhance their ability to reason about anomalies across diverse scenes. Early attempts [37 36 42, 39) leverage the general visual-language alignment capabilities of LVLMs to perform frame-level anomaly detection in training-free manner. More
With the rise of Large Vision-Languaage Models (LVL.Ms)
"Equal contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Gatee Friea haae
Sapicty to the siched
*Prompt: Determine whether contains any anomalies the video caption.
sey desprret
Therk
"Caption: A having barbecue.
"Caption: man having barbecue.
LVLMs
LVLNG
Tokuna anondly
Qwen
LVVV
ALL
S GPT-4o
Owen
LLLVVA
GPT-4o
Tine
"Output: No, there are de anomalies present the video.
*Output: Yes, there cnamely, fire
broken out.
LVLMs
"Ground Truth: Normal
"Ground Truth: Normal.
Figure| 1: Illustration of hallucination in LVLM-based VAD. Although the input image and caption both describe normal scenario ("a man is having a barbecue"), the LVLM incorrectly predicts an anomaly due to the presence of the word "fire." This behavior is driven by co-occurrence bias learned during pretraining, as visualized in the frequency statistics showing that "fire" is strongly associated with anomaly-related tokens in the pretraining corpus.
recent works 2. 44 reformulate VAD as video question answering task, using event descriptions to enhance interpretability. Despite the impressive performance of these LVLM- based VAD approaches, the underlying reasoning mechanisms behind their detection capabilities remain unclear. This gap motivates our study, as understanding how LVLMs reason about anomalies is critical for building safe and reliable VAD systems in real-world applications.
In this paper, we ask a fundamental question: Do LVLMS truly understand video anomalies? This question drives our investigation into whether LVLMS detect anomalies based on scene-aware understanding, simply rely superficial statistical shorteuts learned during pretraining. An ideal VAD model should interpret scene context and ground its predictions in visual evidence.
To investigate this question, we conduct an in-depth analysis of LVLMs' behavior in VAD from the perspective of visual-textual co-occurrence. Our study reveals a hallucination phenomenon: LVLMs tend to rely on co-occurrence patterns between visual instances and textual phrases associated with either normality or abnormality. This reliance leads to incorrect predictions when high-frequenyy visual instances appear in semantically normal contexts. As shown in Figure T the model incorrectly predicts an anomaly in a scene where a man is having a barbecue, even though both the image and its caption describe a normal situation. To systematically examine this issue, we analyze co-occurrence patterns across multiple semantic levels, including objects, visual combinations, interactions, and temporal dynamics. Through both data analysis and controlled experiments, hallucinations are not isolated errors, but rather systematic failures rooted in the model's reliance on superficial statistical shortcuts instead of scene-aware reasoning about anomalies.
show that such
To address this issue, we propose VAD-DPO, a direct preference optimization framework tailored for VAD. The core idea is to formulate VAD pairwise preference optimization problem: by training on visually similar video clips with contrasting anomaly labels, the model is encouraged to distinguish semantic abnormality rather than rely on superficial statistical correlations. Specifically, we construct contrastive preference pairs in which the same object or scene context appears across clips, but only one constitutes an anomaly due to contextual differences. This preference-aligned fine-tuning process encourages the model to perform scene-aware reasoning, thereby mitigating hallucinations caused by spurious co-occurrence patterns.
In summary, our contributions are threefold:
An underexplored problem in LVLM- -bseed VAD is investigated, wherein a hallucination phe- nomenon is revealed from the perspective of visual-textual co-occurrence patterns. This suggests that anomaly detection is driven by statistical shortcuts rather than contextual reasoning.
A preference-basee optimization framework, VAD-DPO, is proposed to guide the model toward scene-aware reasoning and visually grounded anomaly detection by leveraging counter-example preference pairs with visually similar content but contrasting semantics.
Extensive experiments conducted six benchmark datasets demonstrate the superiority of VAD-

---

DPO in both anomaly detection and contextual reasoning. The improvements are particularly pronounced on the benchmarks involving seee-deeendent anomalies.
2 Related Work
2.1 Video Anomaly Detection
Traditional VAD methods mostly rely on self-supervised training using normal data
833
40
46
or adopt video-level labels 35 4 41] with multiple instance learning [28 20
While effective
in constrained settings, they struggle with generalization due to limited training data. Recent
works [37 42
39 29]
introduce LVLMS into VAD to leverage the capabilities acquired through
large-scale multimodal pretraining- Early efforts explored training-free use of LVL.Ms by prompting
them to summarize abnormal patterns for detection 442 39]. Subsequent studies [6] 29 leverage
LVLMs to detect anomalies in the form of video question answering. Zhang et al. [44] enhance
anomaly detection performance by fine-tuning pre-trained LVLMs with supervision VAD datasets. Despite the impressive performance LVLMs, whether they truly ground their predictions in visual evidence remains unclear. In this work, we reveal ballucination phenomenon rooted in LVLMs' reliance on visual-textual co-occurrence patterns, which causes the model to be misled by frequent but irrelevant visual instances. Furthermore, we propose DPO-based training framework to mitigate this issue by guiding the model toward visually grounded reasoning.
2.2 Mitigating Hallucination in Large Vision-Language Models
Hallucination, prevalent issue in LVLMs, refers to the generation of outputs that are not grounded in the input 47 12 21] To mitigate hallucinations, various methods have been explored, most
of which focus improving decoding strategies [13 5 38 For instance, Leng al. [13] introduce visual contrastive decoding, which corrects hallucinated outputs by comparing the model's responses to original and perturbed visual inputs. Chen al. [5] highlight the importance of incorporating both local and global visual context, which integrates an external grounding module during decoding. Xing et al. (38) perform attention reallocation during decoding to guide the model's ss focus toward
visual information. In addition, some methods 10 16 48) address hallucination by performing robust instruction tuning curated datasets. Despite these advances, hallucination in VAD remains underexplored. In this paper, we investigate this phenomenon from a co-occurrence perspective and introduce VAD-DPO, training framework designed mitigate hallucinations. To the best of our knowledge, this is the first systematic study of hallucination in the VAD setting.
3 Diagnosing Hallucinations in VAD: A Co-occurrence Perspective
Task Formulation. Given a video clip and textual prompt, the objective of LVLM- -aaeeddad VAD 6 44 is to determine whether the clip contains anomalous content. Specifically, the input to the model consists of video sequence v and a question q, typically phrased natural language (e.g.. "Is there anything abnormal in this video?"). The LVL. processes the visual and textual inputs and outputs response r, which may take the form of binary decision ("Yes" or "No") or an open-ended caption. Formally, the LVI M-aeed VAD task is defined as:
LVLM(V, 4):
(I)
where v J2,, JT] denotes a sequence of T video frames, and q is the anomaly- query. The model's output r reflects its internal judgment on whether the video contains an anomaly. To assess whether these models perform scene-awart anomaly reasoning rely spurious statistical shortcuts, we examine their behavior from a perspective of co-occurrence.
3.1 Does Hallucination Exist? Evidence from Object-Level Co-occurrence
In language modeling, co-occurrence refers to the frequency with which tokens appear together in training corpora. Extending this notion to vision-language settings, we hypothesize that anomaly judgments may be biased by visual instances (e.g., objects, scenes) that frequently co-occur with anomaly-related phrases (e.g., "fire," "robbery," "panic") during pretraining. Since L.V.Ms are not trained with explicit normal/anomalous labels, approximate such biases by measuring alignment
3

---

Table 1: Object-level co-occurrence statistics between visual objects and abnormaliyynoomalll related textual phrases. Top objects are ranked by abnormal co-occurrence count; the last are neutral visual objects added as counter-shorteut examples.
Total Occur.
Abn./Norm. Ratio
Normalized Abn. Freq
Bias Score
Visual Object
Abn. Count
Norm. Count
gun knife fire blood
812
34
846
23.9
0.960
5.24
16
447
26.9
0,964
5,43
688
51
739
13.5
0,931
4.40
279
11
290
25.4
0.962
5.32
smoke
403
27
430
14.9
0.937
4.27
ambulance
209
19
228
11.0
0,917
3.54
police
201
26
227
7.73
0.886
3.16
syringe
165
13
178
12.7
0.927
3,80
tree
17
62
79
0.27
0.215
-1.87
chef knife 23
88
111
0.26
0.207
-1.93
Table 2: False positive rates (FPR, %) on Probe and false negative rates (FNR, %) on Probe Set
2. Higher values indicate stronger shortcut reliance.
Probe Set (FPR)
Model
Param.
Avg.
Probe Set 2 (FNR)
Fire
Gun
Blood
Knife
Qwen2.5-VL.
80.0
65.0
67.5
57.5
67.5
77.5
Qwen2.5-VL 1
32B
67.5
47.5
60.0
50.0
56.3
60.0
LLaVA-1.5 17
7B
85.0
80.0
77.5
72.5
78,8
82.5
LLaVA-1.5 [7
13B
75.0
62.5
65.0
60.0
65.6
75.0
LLaVA-NeXT-Video
82.5
70.0
67.5
55.0
68.8
72.5
LLaVA-NeXII-ideo
18
34B
67.5
52.5
62.5
47.5
57.5
57.5
IntemVL.3 [49
8B
75.0
72.5
65.0
60.0
68.1
80.0
between visual content and anomaly-related or normality-related texts. As first step, examine whether the presence or absence specific objects, which are objectively observable factors, alters the model's prediction. This perspective allows probe whether the model's output reflects scene-aware understanding or merely shallow vision-text statistical associations.
Co-occurrence Statistics. We conduct statistical analysis on the -A-ON-CC-SBU dataset [7] which contains 558K image-text pairs and serves as part of the pretraining corpus for L.LaVA [7
We begin by using GPT (24] to generate 3k candidate anomaly-related and normality-related textual phrases. These phrases are manually reviewed to ensure semantic clarity and accurate categorization. For each associated image, apply YOLOv5 to detect objects and extract corresponding visual instances. We then compute co-occurrence statistics by counting how frequently each detected object appears in images whose captions contain either anomaly-related
normality-related phrases.
Table presents the object-level co-occureence statistics for the top 8 visual objects with the high- est abnormal phrase co-occurrence, along with 2 counter-shortcut examples (c.g.. chef knife) that often appear in normal contexts. We report the abnormal and normal co-occurrence counts, total occurrences, the abnormal-to-nomeee ratio, and the normalized abnormal frequency (NAF). To further quantify the bias, we compute Bias Score using log-ratio scaling. The two metrics are defined as:
Abn. Count
Abn. Count
NAF
Bias Score
log,
(2)
Abn. Count + Norm. Count'
Norm. Count +
Higher values for both metrics indicate stronger co-occurrence bias toward abnormal semantics.
Probe Setting. To assess whether LVLMs rely shortcut correlations, we construct two diagnostic probe sets. Probe Set 1 targets false positives by inserting unomalous-looking objects-fire, gun, blood, and knife-into semantically normal scenes, with 40 samples per object selected based on Table D Probe Set 2 targets false negatives by placing normal-looking objects semantically abnormal contexts, also comprising 40 samples. In total, we generate 200 samples using frames

---

Table 3: Co-occurrence statistics of visual patterns with anomaly- and normality-relate textual
phrases. Bias scores reflect preference toward anomaly.
Abn./Norm. Ratio
Bias Score
Level
Visual Pattern
Abn. Count
Norm. Count
person driving a car person lying on sidewalk
143
312
0.46
-0.79
Inter-
298
19
15.7
3.97
action
person hitting another
377
8
47.1
5.89
person walking a dog
21
231
0.09
-3.46
people running
321
74
4.34
2.53
person jumping
1 114
27
4.22
2.48
Temporal
person standing still
198
0.05
-4.66
people walking slowly
35
310
0.11
-3.15
Only Fire
Boxing in the Ring
Fighting an Alley
0.8
0.76-133))
1.0
0.56
0.6
0,71
0.6
0.24
0.29
0.2
Normal
Abnormal
Normal
Abnonmal
0L0
Normal
Abnonnal
6.
Noemal
Abnormal
(4) Comparison of probabilities in knife
fire scenarios.
Comparison
probabilities
dilfereen hilting
Figure 2: Effect of object combination and interaction-level co-occurrence on predicted anomaly probabilities. The presence of frequently co-occurring objects interactions leads to high anomaly scores, even in semantically normal scenes, highlighting the influence co-occurrence bias.
from VAD benchmarks
50 and Wan 2.1 31. ensuring temporal consistency to avoid visual multiple LVLMs.
artifacts. We report false positive and false negative
Observations. As shown Table 2 all models exhibit substantial false positive rates on shortcut probe set, with even the strongest models, such Qwen2.5-VI. (32B), misclassifying over 50% of anomalous-lookee lookisl--oiig-bbu bur-normal samples. Models like LLaVA 1.5 (7B) and InternVL3 (8B) reach false positive rates high 78.8% and 68.1%, respectively, indicating strong reliance object-level co-occurrence shortcuts. While scaling model size provides moderate improvements (c.g., Qwen2.5- VL 32B vs. 7B), the bias remains persistent across architectures. Notably, only LLaVA-NeXT- Video (34B) achieves a false negative rate below 60%, further confirming that models struggle to perform scene-aware reasoning about anomalies.
Insight. The results reveal a hallucination pattern: LVLMs tend detect anomalies based on superficial vision-text co-occurrence learned during pretraining, rather than performing scene-aware reasoning. This shortcut behavior persists across architectures and model scales.
3.2 Beyond Object-Level: Analyzing Multi-Level Co-occurrence in VAD Hallucination
The previous analysis shows that LVLMs hallucinate anomalies based on object-level co-occurrence. Building on this finding, we further examine whether such shortcut reliance extends to more complex semantic structures. Specifically, investigate whether hallucinations persist higher semantic levels, including object combinations, interaction patterns, and temporal dynamics.
Combination-Level. To examine whether hallucination extends beyond individual objects, we construct combination-level probes by pairing object categories that exhibit strong anomaly-related co-occurrence, as identified in Table Specificallyy, we conduct experiments on Qwen2.5-VI. (7B) by inserting knife into the previously constructed fire samples from Probe Set while ensuring the overall scene semantics remain normal. As shown in Figure 2 a), the predicted anomaly probability

---

significantly increases after inserting knife, even though the scene remains a semantically normal kitchen. This confirms that hallucination is not limited isolated visual triggers but can be further reinforced by frequently co-occurring object combinations.
Interaction-Level. Beyond static object presence, examine whether hallucinations can also be triggered by specific interaction patterns between objects or people. We focus on visually ambiguous interactions, such as driving, hitting, or walking, which are highly context-dependent and can appear either normal or abnormal depending on the scene. To quantify such interaction-level co-occurrence, we use GLIP [5] detect interactions in videos associated with anomaly-relater normality text descriptions. The resulting co co-occurrence statistics are summarized in Table 3
select the highly biased interaction pattern person hitting another and place in two different contexts: street alley to represent a semantically abnormal scene, and a boxing ring to represent normal one. As shown in Figure 2(h), the model assigns significantly higher anomaly scores to both the street and boxing ring scenarios. This demonstrates that interaction-level co-occurrence can also interfere with the model's judgment, leading to hallucinations.
Based on Table
Temporal-Level. We further investigate whether hallucinations can be caused by temporal patterns that frequently co-occur with anomaly-related descriptions. To quantify this, use Video- MAE [30 to extract motion categories from videos and compute their co-occurrence with
Busluation Qwen2.5VT.77
Abxoma Treal
Narta
Prebsbility
0.5
0.4
anomaly- and normality-related texts. The result- ing statistics shown in Table g
0.2
0.0
To probe whether temporal co-occurrence leads to hallucination, we construct an abnormal scene where a person stands still in the middle busy road. This visual pattern frequently co-occurs with normality-relatte clear semantic anomaly in this context. Unlike previous cases, LVLMS such as Qwen2.5-VL. [I] and VA-15 [17] not misled by the co- occurrence and identify the scene as abnormal.
Alanemad
Abcornal
text, yet it constitutes a
Probdaality
To understand this behavior, we visualize frame- level anomaly probabilities. As shown in Figure the models initially predict normality but gradu- ally detect the anomaly as the sequence unfolds. This indicates that temporal progression provides additional semantics for scene-aware reasoning.
0.0
Frise Ih
Frame
Peamrer
Tram:
da Budhaaionn on LLM.L M-5.7-
Figure 3: Frame-wise anomaly probabilities a temporal probe.
Insight. Hallucinations in VAD extend beyond object-level to co-occurrence patterns involving object combinations, interactions, and temporal dynamics. While combination and interaction patterns often reinforce false positives, temporal information can both mislead and correct model predictions.
Proposed Method: Direct Preference Optimization for VAD
To address the issue revealed in Section we propose VAD-DPO, a Direct Preference Optimization approach for VAD. VAD-DPO reformulates detection as preference alignment task, encouraging the model to favor outputs consistent with semantic correctness, thereby guiding it to reason based on scene-level semantics rather than relying on superficial statistical shorteuts.
4.1 Preliminary
Direct Preference Optimization (DPO) 26 aligns model behavior with human preferences by directly optimizing over preference pairs without reinforcement learning or external reward models. Given multi-modal input (Ir, v) and two candidate responses yo (preferred) and (less preferred), the model Te defines conditional distribution To(y|>, v). DPO encourages the model to increase the relative likelihood of the preferred output while regularizing divergence from reference model met-

---

formulate initialized from the same checkpoint. The training objective is derived from the Bradley- model:
( log
(Bw x; u) Teet (ar
To To I, 4) ret i x; u)
Loro = -log a
8 log
(3)
2r, v)
where (.) is the sigmoid function and B controls the sharpness of preference alignment.
4.2
Optimization Objective for VAD
Building recent advances in multimodal preference optimization 16 48
anomaly detection as a direct preference optimization problem. To explicitly break shortcut reliance, we construct counter-example preference pairs from existing datasets [50, 44 and further synthesize additional pairs with Wan2.1 [3I], where clips are visually similar but semantically contrasting, For instance, both "a person hitting another in a boxing ring" and person hitting another in street alley" involve the same high- frequency interaction pattern (hitting), which co-occurs with unomaly-relate cues. However, only the latter should be judged anomalous. By training the model prefer the semantically correct response Yu over the incorrect y!, VAD-DPO enforces reasoning grounded in scene semantics rather than superficial correlations.
Formally, let (ar, v) denote the textual prompt and video input, and (Vie, y1) be two candidate outputs. Following the DPO objective, the VAD-specific loss is:
Te (Nw
I, v) x, v)
To (I) 2t, v) Ter I. 21)
LVAD-DP
logg
( log
B log
Tea (Ye
To further stabilize training and preserve strong preference for semantically valid clips, we adopt an anchored objective [32
( yw
x, x2
Cane - [ -og
(5)
Teer(yu Iz, 2)
The final loss combines contrastive and anchored terms:
Liaul LVAD-DIO + LAne
(6)
where y balances the contribution of anchoring. Through counter-example preference pairs, VAD- DPO directly targets the co-occurrence shortcut issue, compelling the model to distinguish visually similar but semantically distinct contexts. This preference- baased supervision shifts the model from surface-level correlations toward robust reasoning over scene semanties, thereby mitigating shortcut- driven hallucinations in LVLM-based VAD.
4.3 LVLM-based Video Anomaly Detection
which consists of two stages. (1) each with scoring-style prompt, following 42]. For example, enforcement agency, how would you rate the scene on a scale from 0 to 1, where 0 denotes a standard scene and I indicates suspicious activities?" The model directly outputs numerical score, which is taken as the anomaly score of the frame. (2) Score Refinement and Assignment: The sparse frame-level scores are refined through weighted aggregation mechanism. Specifically, we exploit temporal attention weights from the decoding process to propagate scores from sampled frames to their neighbors, producing a dense per-frame anomaly score sequence across the entire video.
During inference. adopt the evaluation protocol of LAVAD 42
Frame Sampling and Scoring: For each video segment, we uniformly sample 8 frames and pair
law ask the LVLM: "If you
Experiments
5.1 Experiment Setup
Datasets. We evaluate our method on six real-world surveillance datasets commonly used in VAD:
Thahh [9], UCF-Crime (28). XD-Violence (35), NWPU Campus (2). MSAD (50), and
HIVAU-70K 44). Shanghai [19] features 13 surveillance views from campus scene. UCF.
Crime (28] contains over 1,900 hours of vidcos across 13 crime categories. Vi-lenee 35

---

Table 4: Comparison of frame-level AUC (%) with state-of-the-ee LVLMs and VAD methods on ShanghaiTech, UCF-Crime, Violennce NWPU Campus, and MSAD.
Method
Params
SHTTchh
UCF-Crime
XD-Violence
NWPU Campus
MSAD
Qwen2.5-VL 1
7B
79.4
78.8
83.2
71.9
75.9
LLava-1.5 17
13B
76.3
72.8
79.6
70.3
75.1
MaaVA-NeXT-Videe
18
7B
78.8
75.4
81.5
72.6
78.6
LAVAD 42
13B
81.8
80.3
85.4
71.1
79.4
A- A-Guardian 6
7B
80.9
76.1
85.0
72.3
80.9
AnomalyRuler 39
7B
84.6
78.6
83.1
69.9
81.1
Hawk 29
7B
80.8
81.9
80.7
72.5
78.9
Holmes- VAU
44
2B
85.2
84.5
86.3
70.8
81.6
VAD-DPO (Ours)
87.2
88.5
85.4
Table 5: False positive rates (FPR, %) Probe Set 1 and false negative rates (FNR, %6 Probe Set
Higher values indicate stronger shortcut reliance.
Probe Set (FPR)
Model
Param.
Avg.
Probe Set (FNR)
Fire
Gun
Blood
Knife
LAVAD
22
13B
85.0
82.5
80.0
87.5
82.5
82.5
A- A-Guardian
6
7B
77.5
80.0
85.0
85.0
81.9
80.0
AnomalyRuler
39
7B
82.5
87.5
90.0
80.0
81.9
82.5
Hawk 29
7B
90.0
80.0
77.5
72.5
85.0
77.5
Homles- VAU
2B
75.0
80.0
72.5
72.5
75.0
80.0
VAD-DPO(Ours)
2B
15.0
12.5
17.5
7.5
13.1
12.5
includes 4,754 video clips from movies, online sources, and CCTV, focusing on violence detection. NWPU Campus (2 provides 43 scenes, with majority involving scene-dependent anomalies. MSAD [50] comprises data from 14 scenes and 500 viewpoints, offering large-scale scene diversity, HIVAU-7OK 4d] built upon UCF-Crime and XD-Violenm is tailored for LVLM-based VAD with multi-granularity segmentation and rich textual descriptions.
Evaluation Metries. To comprehensively evaluate LVLMs in VAD, consider both detection performance and reasoning ability. For detection, we report Area Under the ROC Curve (AUC) as the main metric. In addition, we report false positive rate (FPR) on Probe Set and false negative rate (FNR) on Probe Set 2.
Implementation Details. We set the learning rate le-4 and adopt a cosine learning rate scheduler with warm-up ratio of 0.05. The default value of is set to in Equation 6 All models are trained for a single epoch.
5.2
Main Results
While most existing LVLM-based VAD methods 42 39
29 operate in training-free or zero-
shot setting, Holmes- VAU 44 Accordingly, we implement VAD-DPO on Qwen2-VL. (2B) 34 aligning with the parameter scale of Holmes- VAU ensure fair comparison.
is the only work that explores supervised fine-tuning of an LVLM.
Detection Performance. We evaluate the frame-level detection performance VAD-DPO on five benchmark datasets and compare it with state-of-the-an LVLM-based VAD methods well as representative open-source LVLMs. For methods that did not report frame-level AUC on these benchmarks, we reproduce the results using their publicly released weights and code, without any modification. As shown in Table VAD-DPO achieves the highest AUC scores across all benchmark datasets, outperforming both 7B and 13B LVLMs despite using only 2B-scale parameters. Compared to Holmes- VAU [44), VAD-DPO achieves at least 2% improvement in AUC across all benchmarks, with a notable gain of 8.3% on NWPU Campus. The NWPU Campus dataset contains scene- dependent anomalies, which require the model to understand contextual semantics. The significant improvement on this benchmark demonstrates that VAD-DPO effectively guides the model perform scene-aware reasoning, rather than relying superficial statistical shortcuts.

---

fieh-fre co-occurriag object with anomalyy text: Police Car
System Prompt: D Deternine whether an abnormal event occurs in the given video.
Scene Semantic: A normal traffic intersection where police car is on routine patzol.
Prediction of @wen2.5-VL.-1
Yes, police car is approaching in the sceme, indicating that an abnormal event is occurring.
Prediction L La --V-NeXX--vide--
Yes, police car appears the intorsection, which may indicate that an abnormal event has occurred
Prediction L. LLVA-11.5-13B.
Yes, the appearance of a police car intersection may indicate potential anomaly.
Prediction of Ours:
No, the pedestrian is walking through the intersection normally, and a police car slowly approaching. No anomaly is detected in the scene.
Figure 4: Qualitative comparison illustrating hallucination in LVLM-based VAD. While Qwen2.5- VL-7B,
and LLLVV--1.5-331 incorrectly predict an anomaly due to the presence of high-freyuency co-occurring object (police car), our method accurately identifies the scene as normal based on contextual understanding.
the results in Tablee2by including both domain- specific expert models 44 29) which are fine-tuned on VAD datasets, and training-free LVLM-based methods
Performance on Shortcut Probes.
39 All results are obtained using released code and checkpoints, without retraining manual tuning. We use fixed threshold of 0.5 across all models compute false positive and false negative rates, ensuring fair comparison despite differences in output score distributions.
Although the expert models are trained specifically for VAD, they still exhibit high hallucination rates. We attribute this to the scarcity of true anomaly samples in existing datasets, which biases supervised fine-tuning toward spurious co-occurrence patterns. In contrast, VAD-DPO reduces the average false positive rate from over 75% in expert models to only 13.1 %, an absolute reduction of more than 80%, highlighting its effectiveness in overcoming shortcut reliance. Training- -ree methods [42 6 39) also suffer from severe hallucinations, which is expected given their dependence on earlier-generation LVLMs. For example, LAVAD 42 relies BLIP2 with LLLMAA2-13BB MMeval uses Vicuna-7B, and AnomalyRuler employs -iistal-77---nstruct--o0 reasoning capability and thus perform consistently worse than our approach.
These models lack sufficient multimodal
Qualitative Results. Figure illustrates the superior qualitative performance of our proposed VAD- DPO in detection and reasoning. Compared to other baselines, VAD-DPO accurately understands and focuses on the anomalous events within the video. In contrast, other LVLMs are influenced by statistical correlations learned during pre-trat pre-training, which leads them to generate descriptions centered around frequently co-occurring patterns.
5.3
Ablation Study and Analysis
Effect of Preference Alignment. To evaluate the effec- tiveness of preference aligmment VAD-DPO, we con- duct ablation studies on the NWPU Campus dataset. Us- ing the pre-trained Qwen2-VL (2B) as the baseline, we compare the effects of different training strategies, in- cluding instruction tuning, Proximal Policy Optimization (PPO) (27). standard DPO (26), and VAD-DPO, All meth- ods are trained on the same data from HIVAU-7OK (44) for fair comparison.
Table 6: AUC (%) on NWPU Campus under different training paradigms.
Training Paradigm
AUC
Instruction Tuning [44
69.2
PPO 27
72.3
DPO 26
73.7
VAD-DPO (Ours)
79.1
As shown in Table 6 VAD-DPO significantly outperforms all other training paradigms, achieving an AUC of 79.1 %. Instruction tuning performs worst (69.2%) due to its lack of guidance for contextual reasoning. PPO improves slightly (72.3%) but suffers from the difficulty of defining effective reward functions. Standard DPO achieves 73.7% but fails explicitly guide the model to understand why

---

the preferred response aligns with the imput video or why the rejected response should be discouraged. In contrast, VAD-DPO directly guides the model suppress preferences for tegueereyy patterns when they are contextually irrelevant, effectively mitigating reliance statistical shortcuts.
Effect of y, As shown in Figure S we investigate the impact of different values on the performance of VAD- DPO. Across four Denchmarks-UCF-CCE
Ge tntraiaaio LCB
By saleelaeo de
XD-Violence, NWPU Campus, and MSAD-we observe that smaller y values significantly improve detection performance, indicat- ing effective mitigation of hallucinations. However, as Y increases, performance gradually degrades due excessive suppression of co-occurrence patterns, which impairs the model's reasoning ability. To balance these effects, we set to by default, aiming to reduce the model's reliance on statistical shorteuts during reasoning without compromising anomaly detection performance.
10
Figure 5: Ablation studies 7
Generalization of VAD-DPO. We evaluate the result- ing VAD-DPO model on four general-purpose video understanding benchmarks, including egocentric ac- tion recognition, schematic reasoning, and video question answering. shown in Table 7 VAD-DPO achieves consistent improvements over the original Qwen2- VB. This suggests that preference tuning not only preserves generalization ability, but may also enhance it by strengthening semantic grounding. Overall, the results demonstrate that VAD-DPO maintains robust generalization even when trained on relatively small but carefully constructed set of preference pairs.
Table 7: Generalization of VAD-DPO.
Benchmark
Qwen2-VI. VI. Ours
MVBench 14
63.2
63.6
Perception Test
25
53.9
54.5
EgoSchema 23
54.9
55.1
Video MME
7
60.4
61.2
Conclusion
In this paper, investigate the behavior of LVLMs in video anomaly detection from the perspective of co-occurrence between visual instances and textual phrases. By analyzing pre-training datasets and conducting probe experiments, reveal a hallucination phenomenon in LVLM-based VAD: LVLMs tend to rely on statistical shortcuts learned during pre-training rather than performing scene- aware contextual reasoning when identifying anomalies. Further controlled studies demonstrate that co-occurrence at the combination and interaction levels exacerbates this issue. To address this problem, we propose VAD-DPO, which formulates VAD pairwise preference optimization task. Specifically, the model is trained on visually similar video clips with contrasting anomaly labels to encourage alignment with semantic consistency rather than co-occurrence shortcuts. Extensive experiments across six benchmark datasets validate the effectiveness of our method.
Future Work. Recent advances such as GRPO provide promising directions for the future develop- ment of LVLMss, we believe there remains significant potential in adapting GRPO-style optimization to video anomaly detection. In particular, future work may explore: (1) tailoring group-wiss wisee ranking strategies in GRPO to better capture the semantic that are central to VAD; (2) integrating task-specific reward models, to stabilize training and prevent semantic drift; and (3) scaling GRPO with larger corpora to fully leverage its potential.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants (62406039, 62321001, 62471055, U23B2001, 62101064, 62171057, 6207 1067), the High- High-Quality Development Project of the MITT22400TTCB22584,, the Ministry of Education and China Mobile Joint Fund (MCM20200202, MCM20180101 the Project funded by China Postdoctoral Science Foundation (2023TQ0039, 2024M750257, GZC20230320), the Fundamental Research Funds for the Central Universities (2024PTB-0WX) the 2025 Education and Teaching Reform Project Funding Beijing University of Posts and Telecommunications (2025YZ005).
10

---

References
[1] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Ming-Hsuan Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie. Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. CoRR, (1222302.11923,, 2025.
[2] Congqi Cao. Yue Lu, Peng Wang, and Yanning Zhang. A new comprehensive benchmark for semi-supervised video anomaly detection and anticipation. In CVPR, pages 20392-20401, 2023.
[3] Xuweiyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David Fouhey, and Joyce Chai. Multi-object hallucination in vision language models. In NeurIPS, 2024.
[4] Yingxian Chen, Zhengzhe Liu, Baoheng Zhang, Wilton W. T. Fok, Xiaojuan Qi, and Yik-Chung Wu. MGFN; mnggitude-contrastie glance-and-focus network for weakly-supervin video anomaly detection. In AAAI, 2023.
[5] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. HALC: object hallucination reduction via adaptive focal-contrast decoding. ICML, 2024.
[6] Hang Du, Sicheng Zhang, Binzhu Xie, Guoshun Nan, Jiayang Zhang, Junrui Xu, Hangyu Liu, Sicong Leng, Jiangming Liu, Hche Fan, Dajiu Huang. Jing Feng. Linli Chen, Can Zhang, Xuhuan Li, Hao Zhang, Jianhang Chen, Qimei Cui, and Xiaofeng Tao. Uncovering what, why and how: A comprehensive benchmark for causation understanding of video anomaly. In CVPR. pages 18793-18803, 2024.
[7] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms video analysis. CVPR, pages 24108 1088-22115 2025.
[8] Mariana Iuliana Georgescu, Antonio Barbalau, Radu Tudor Ioneseu, Fahad Shahbaz Khan, Marius Popescu, and Mubarak Shah. Anomaly detection in video via self-supervised and multi-task learning. In CVPR, pages 12742- 2021.
[9] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. CVPR, pages 14375-14385, 2024.
[10] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In AAAI, pages 18135-18143, 2024.
[11] Yixiao He, Haifeng Sun, Pengfei Ren, Jingyu Wang, Huazheng Wang, Qi Qi, Zirui Zhuang, and Jing Wang. Evaluating and mitigating object hallucination in large vision- laiguaee models: Can they still see removed objects? In Naacl, pages 6841-6858, 2025.
[12] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. OPERA: alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation In CVPR, pages 13418- 13427, 2024.
[13) Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vasion-language models through visual contrastive decoding. CVPR, pages 13872-13882, 2024.
[14] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In CVPR, pages 22195-22206 2024.

---

[15] Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai- Rei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022.
[16] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In ICI.R 2024.
[17] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[18] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In CVPR. pages 26286-2000 26296, 2024.
[19] Weixin Luo, Wen Liu, and Shenghua Gao. A revisit of sparse coding based anomaly detection in stacked RNN framework. In ICCV, pages 341-349, 2017.
[20] Hui Lv, Zhongqi Yue, Qianru Sun, Bin Luo, Zhen Cui, and Hanwang Zhang. Unbiased multiple instance learning for weakly supervised video anomaly detection. In CVPR, pages 8022-8033 203-, 2023.
[21] Xinyu Lyu, Beitao Chen, Lianli Gao, Hengtao Shen, and Jingkuan Song. Alleviating hallucina- tions in large vision-language models through mullucination-induced optimization. In NeurIPS, 2024.
[22] Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In ACL, pages 1:585-12602. 2024.
[23] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. NeurlPS, pages 2023.
[24] OpenAl. GPT-4 technical report. CoRR, abs/2303.087744 2023.
[25] Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: diagnostic benchmark for multimodal video models. NeurIPS, pages 42748-42761, 2023.
[26] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurlPS, 2023.
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/ htss/100.006477 2017.
[28] Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In CVPR, pages 6479-6488, 2018.
[29] Jiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, and Yingcong Chen. HAWK: learning understand open-world video anomalies. In NeurIPS, 2024.
[30] Zhan Tong. Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient leamnrr for self-supervised video pre-training. In NeurlPS, 2022.
[31] Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wer Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503 20n::22500.33444 14, 2025.
[32] Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. In EMNLP, pages 8078-8088, 2024.
12

---

[33] Guodong Wang, Yunhong Wang, Jie Qin, Dongming Zhang, Xiuguo Bao, and Di Huang. Video anomaly detection by solving decoupled spatio-temporal jigsaw puzzles. In ECCV, pages 194-511, 2022.
[34] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu. Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution. CoRR, htss/200..11911. 2024.
[35] Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In ECCV, pages 322-339, 2020.
[36] Peng Wu, Xuerong Zhou, Guansong Pang. Yujia Sun, Jing Liu, Peng Wang, and Yanning Zhang. open-vocabulary video anomaly detection. In CVPR, pages 18297-18307, 2024.
[37] Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, and Yanning Zhang. Vadclip: Adapting vision-language models for weakly supervised video anomaly detection. In AAAI, pages 6074-6082, 2024.
[38] Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concentric causal attention. In NeurlPS, 2024.
[39] Yuchen Yang, Kwonjoon Lee, Behzad Dariush, Yinzhi Cao, and Shao- Lo. Follow the rules: Reasoning for video anomaly detection with large language models. In ECCV, 2024.
[40] Zhiwei Yang. Jing Liu, Zhaoyang Wu, Peng Wu, and Xiaotao Liu. Video event restoration based on keyframes for video anomaly detection. In CVPR, pages 14592- 14601, 2023.
[41] Zhiwei Yang, Jing Liu, and Peng Wu. Text prompt with normality guidance for weakly supervised video anomaly detection. In CVPR, pages 18899-18908, 2024.
[42] Luca Zanella, Willi Menapace. Massimiliano Mancini, Yiming Wang, and Elisa Ricci. Har- nessing large language models for training-free video anomaly detection. In CVPR, pages 18527-18536, 2024.
[43] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tunee audio-visual language model for video understanding. In EMNLP, pages 543-553, 2023.
[44] Huaxin Zhang, Xiaohao Xu, Xiang Wang, Jialong Zuo, Xiaonan Huang, Changxin Gao, Shanjun Zhang, Li Yu, and Nong Sang. volmes-vau: Towards long-term video anomaly understanding at any granularity. In CVPR, pages 13843-13853. Computer Vision Foundation / IEFE, 2025.
[45] Menghao Zhang, Jingyu Wang, Qi Qi. Pengfei Ren, Haifeng Sun, Zirui Zhuang, Huazheng Wang, Lei Zhang, and Jianxin Liao. Video anomaly detection via progressive learning of multiple proxy tasks. In ACM Multimedia, pages
2024.
[46] Menghao Zhang, Jingyu Wang, Qi Qi, Haifeng Sun, Zirui Zhuang, Pengfei Ren, Ruilong Ma, and Jianxin Liao. Multi-scale video anomaly detection by multi-grained spatio-temporal representation learning. In CVPR, pages 17385-17394, 2024.
[47] Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. In ICML, 2024.
[48] Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination large vision-language models. In ICLR, 2024.
[49] Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
[50] Liyun Zhu, Lei Wang, Arjun Raj, Tom Gedeon, and Chen Chen. Advancing video anomaly detection: A concise review and a new dataset. In NeurlPS, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 3.1 Section 3.2 Section 5.1 and Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: All used datasets are publicly available. The experimental setup and imple- mentation details provided in Section 5.1 and Appendix A.3. We include part of the core code in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm cc/
jublic/guides http://oooeomiiiionnnnooo.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //tpps.cc/pulli//uu.ddd ssape//gggddeeeeoeeeeeeiiaaaaaaa ssinisolicyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section 3.1 Section 3.2 Section 5.1 and Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer Section
5.2 and Appendix A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 5.1 Appendix A.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This research conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Appendix C.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All existing assets used in this paper, including datasets (e.g., UCF-Crime, XD-Violence, ShanghaiTech, Avenue, NWPU Campus, and MSAD) and pre-trained LVLMs (e.g., Qwen-VLe VL, LLa Intern are properly cited in the paper. Each dataset used in accordance with its published license or terms of use. The versions of the datasets and models used are stated in Section [3J] and Section 5.2 respectively. URLs and licenses are included where publicly available, and we strictly follow the original license terms for all repackaged or redistributed assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please refer to Section 4.2 Section 5.1 and Appendix A.3. We include part of the core code in the supplementary material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We employed GPT-based large language models a non-standard but auxiliary role to assist with filtering textual phrases related to normal or anomalous events during the construction of co-occurrence statisties and probe datasets. This usage is not essential core methodology and could be replaced by manual annotation. However, it helped reduce annotation cost and improved consistency by mitigating subjective bias. The use of LLMs has been explicitly documented in Section 3.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy http:s///earrieecc/////eeerreeee for what should or should not be described.
20

---

Appendix
In this appendix, we provide additional experimental results, implementation details, probe dataset construction methods, and further qualitative and quantitative analyses to support the claims made in the main paper. These materials are organized as follows:
Section A Supplemmentrry Analyses and Technical Details We provide supplementary analy- ses and technical details, including probe design, co-occurrence statistics, implementation details, extended results, and more qualitative examples.
Section A.1 Diagnostic Probe Construction and Examples We detail the construction process of diagnostic probe datasets, including visual examples for both false positives and false negatives.
Section A.2 High-Level Co-occurrence Statistics and Analysis We present the high-level a0-occurrence statistics across object combinations, interaction patterns, and temporal dynamics, along with quantitative evaluations that extend our analysis beyond the object level.
Section A.3 Training and Implementation Details) We describe the training and implementa- tion details of VAD-DPO, including model initialization, optimization settings, and computational resources used.
We report additional experimental results, includ- ing reasoning performance and the performance of implementing VAD-DPO on Intern
Section Additional Experimental Results
Section A.5 Comparisons with Methods for Mitigating Hallucinaiion We further clarify our contribution, discuss hallucination in LVLMs, and compare with related work.
Section A.6 More Qualitative Examples: We show more qualitative examples comparing VAD- DPO and baseline models under various scenarios, highlighting differences in reasoning behavior.
Section B Limitations We discuss the limitations of our current approach, including model scale and residual hallucination effects, and outline possible directions for future work.
Section Broader Societal Impacts We discuss the broader societal impacts of our work, highlighting the potential benefits in safety-critical applications.
A Supplementary Analyses and Technical Details
A.1 Diagnostic Probe Construction and Examples
To evaluate whether LVLMs rely on co-occurrence-driven shortcuts rather than truly understanding scene semantics, construct two diagnostic probe sets designed induce controlled hallucination scenarios. This section complements the analysis in Section 3.1 by providing detailed descriptions of how these probe sets are constructed, their underlying rationale, and the evaluation protocol used to assess hallucination behavior across different LVLMs.
Details of Co-occurrence Statistics. As described in Section 3.1, we conduct this analysis using the LAION-CC CCUSUU dataset. We begin by using GPT to generate approximately 3k candidate textual phrases that potentially associated with anomaly normality. These phrases are manually reviewed to ensure semantic clarity and proper categorization. We then match these phrases against LAION captions to identify text-image pairs where caption contains candidate phrase. For each matched image, we apply YOLOv5 to detect objects and filter out low-confidence detections to ensure statistical robustness. This process allows to compute co-occurrence statisties by measuring how frequently certain visual objects appear in images whose captions express either anomaly-related or normality-related concepts. In this way, the co-occurrence patterns are estimated from large-scale real-world data with careful semantic control.
GPT-generated Textual Phrases.
To construct our probing datasets, use GPT-4 to generate diverse set of 3k candidate textual phrases related to anomaly and normality. We prompt the model with instructions such as: Generate a list of concise phrases that describe either abnormal or normal events in video scenes. Some example outputs include:
Anomaly-related: "a person holding knife", "fire spreading rapidly", "man lying motionless on the ground".
21

---

semmalit--reltted "people walking across a street", "children playing in a park", "car driving along a road".
We manually review and filter these phrases to ensure semantic clarity and diversity.
Probe Set I: False Positive Induction. This set aims to test whether models incorrectly predict anomalies when goomalous-lookingg objects are introduced into semantically normal scenes, Based on the object-level co-oceurrence statistics in Table[] we identify four visual objects. fire, gun, blood, and knife, that exhibit strong abnormality bias. For each object, we manually select 40 base scenes from VAD datasets where the original context is clearly normal. We then use image editing or video generation techniques (31] to insert the target object in a visually realistic way, ensuring that the modified frame still maintains normal semantics overall. These modified clips are used to test the model's s hallucination susceptibility.
Probe Set 2: False Negative Induction. To evaluate whether LVLMs overlook semantic anomalies when scenes contain superficially benign objects, we design second probe set. We select 40 semantically abnormal situations where abnormality arises from scene context rather than object presence. Into these scenes, we insert neutral or high-frequency objects associated with normality (e.g.. tree, knif)) that may distract or bias the model toward false negatives. Examples include inserting a peaceful-looking object into street fight, or placing a daily-use item in chaotic environment.
Dataset Sources and Consistency. The probe videos are derived from two sources. First, we select representative examples from three real-world datasets- NWWU Campus 2. HIVAU-70K [44) and MSAD -wich serve as the base scenes. Second, we construct edited versions of these samples using Wan2.1 [3I, enabling controlled insertion of objects contextual modifications induce hallucination. We ensure that temporal consistency is preserved after object insertion by verifying motion cues and avoiding visual artifacts. Each probe video contains sequence of frames with minimal modification outside the target region to isolate the effect of ro-occurrence-diee driven bias.
Human Validation Protocol. Our goal in constructing probe samples is to simulate scenes where the most plausible and commonsense interpretation is normal, despite the presence of anomaly-related objects. For example, video depicting man barbecuing in backyard may include a "fire" object, yet the overall scene is semantically normal. To ensure the normality, we applied a strict human validation protocol:
Manual Screening: All videos are reviewed by multiple annotators ensure that the inserted object does not introduce unintended anomaly cues beyond its visual presence.
Exclusion of Ambiguous Cases: We filter out videos in which the anomaly-relatee object could reasonably support multiple semantic interpretations (e.g., police cars in tense crowd scenes).
While we acknowledge a degree of subjectivity inevitable, this validation protocol designed to minimize such ambiguity.
We evaluate multiple LVLMs of varying sizes and architectures (Qwen2.5-VL. LLaVA-1.55 LLa ALaVA-NeXT--idee Intern on both probe sets. For Probe Set 1, we report false positive rates (FPR), indicating hallucination susceptibility when no actual anomaly exists. For Probe Set 2, we report false negative rates (FNR), measuring the model's failure to detect context- based anomalies. Results are summarized in Table [ showing that models consistently exhibit higher error rates in these carefully constructed edge cases.
Evaluation Protocol.
Visual Examples. Figure A provides representative examples from both probe sets, including four cases from Probe Set and two from Probe Set 2. These examples illustrate how high-bias or contextually misleading objects are inserted into real scenes to construct controlled test cases. To protect privacy, all visible faces in the frames have been masked. We acknowledge that simply applying masks may be sufficient fully eliminate potential ethical and privacy concerns. In future releases of our probe sets, we plan to adopt full-face blurring or visual obfuscation techniques to further enhance privacy protection.
A.2 Ligh-Level Co-occurrence Statistics and Analysis
Overview. This section provides implementation details and quantitative analysis of high-level co-occurrence patterns that may lead to hallucinations in LVLM-based VAD systems. Specifically,
22

---

00000
Probe Set 1
"Caption: having barbecue.
"Caption: soldier engaged shooting practice.
"Caption: A medical operation is in progress.
CCaption: A worker is unpacking package.
Probe Set 2
"Caption: A pedestrion pushed another person down.
*Caption: man tried to climb tree, sneaking around.
Figure A.1: Representative examples from the diagnostic probe sets. The top shows four samples from Probe Set I, where high-bias objects fire, gun) are inserted into semantically normal scenes to induce false positives. The bottom shows two samples from Probe Set 2, where contextually abnormal scenes contain visually benign elements that may mislead the model toward false negatives. All visible faces have been masked preserve privacy.
we elaborate on the statistical construction object combinations, interaction patterns, and temporal dynamics, and report the corresponding anomaly prediction behaviors.
Object Combination Co-occurrence. We identify object pairs with high anomaly-related co- occurrence frequency based on conditional probabilities computed from the HIVAU-7OK dataset. Among them, knife and fire form frequently co-occurring pair in anomalous contexts. To test whether their combination induces hallucination, we insert knife into fire scenes (originally labeled as normal) while keeping the scene semantics unchanged (e.g., a kitchen). Quantitative results show that adding knife increases the predicted anomaly probability significantly, despite the absence of actual anomalies.
Interaction-Leewe Co-occurrence. We use GLIP 15 to extract subject-verb-object interactions from video-text pairs and compute their anomaly association via normalized co-occurrence counts. The pattern person hitting another is selected due to its strong anomaly correlation. To isolate the effect of context, embed the same interaction into settings: street alley and boxing ring. In both, models incorrectly predict high anomaly scores, confirming interaction-leve hallucination even semantically normal contexts.
Temporal Dynamics Co-occurrence. Motion patterns are extracted using VideoMAE (0 and
aligned with anomaly/normality tags through co-occurrence analysis. The motion pattern standing still is often associated with normality in the training data. We construct a probe where a person remains stationary in the middle of busy road. While the scene is clearly anomalous, observe that some models initially assign low anomaly scores, which only rise after several frames, indicating delayed recognition due misleading temporal priors.
Summary. These results demonstrate that hallucination LVLMs extends beyond object-level triggers to include more complex semantic structures. Object combinations, interaction patterns, and temporal motion- when biased by training co-occurrence can each mislead the model into incorrect anomaly predictions, despite semantically normal visual input.
A.3 Training and Implementation Details
We provide additional details regarding the training setup and implementation of VAD-DPO.
23

---

Model Initialization. All experiments are conducted using the Qwen2-VL VL [34] backbonn with 2B parameters. The reference model Ter is initialized from the same checkpoint and frozen throughout training. We use the official HuggingFace implementation with minor modifications to support preference-based finetuning.
Training Details. We use mixed- precciiion training (fp16) and a batch size of 16 preference pairs per GPU. All models are trained for single epoch using the AdamW optimizer with a learning rate of le-6 and cosine decay schedule (warm-up ratio 0.05). The alignment sharpness factor / is set to 0.1 following DPO best practices [6] and y is set to unless otherwise specified.
Preference Pair Construction. Preference pairs (yw y1) are constructed from HIVAU-7OK [44], MSAD 550, and NWPU Campus (2). Each pair consists of two video clips with similar visual content but differing semantics-only one clip is contextally anomalous. These clips are selected based on manual inspection of anomaly labels and scene descriptions, ensuring high semantic contrast with minimal visual confounds. In total, we curate 1,000 such preference pairs for training. To ensure both semantic contrast and visual alignment, most pairs are sourced in one of two ways: (I) from the same scene where different but visually similar behavior patterns (e.g., walking vs. running) occur under normal and anomalous contexts, or (2) from different real-world scenes that share the same surface- level activity (e.g., standing still) but differ in contextual interpretation. This construction strategy helps isolate the causal semantics of anomaly, making the preference signal more robust and interpretable.
internal cluster equipped with 2 NVIDIA A A100 GPUs (80GB memory each), using distributed data parallelism via PyTorch's 's forch.distributed module. VAD-DPO model is trained for one full epoch over the 1,000 constructed preference pairs, which corresponds to approximately 90,000 optimization steps (given pair-wise sampling and batching). The end-to-end training process takes around 16 GPU-hours.
Training Time and Resources. All training is performed
To reduce data loading bottlenecks, all video clips are pre-extracted into frame sequences and cached on local SSDs. Training uses mixed precision (fp16) to improve memory efficiency and reduce training time by 30% compared to full precision.
Table A.I: Comparison of reasoning performance with state-of-the-an LVLMS on HIVAU-70K BLEU, CIDEr, and ROUGE scores are reported at clip-level (C), event-level (E), and video-level (V).
Method
Params
BLEU
CIDEr t
ROUGE 1
C
E
C
E
V
C
E
Video-ChatGii [22
0.152
0.068
0.066
0.033
0.011
0.013
0.153
0.048
0.079
Video-I.IaMA 43
0.151
0.079
0.104
0.024
0.014
0.017
0.156
0.067
0.090
LLava-1.5
17
7B
0.316
0.097
0.189
0.115
0.018
0.028
0.176
0.077
0.091
LLAVA-Next- Video
18
7B
0.435
0.091
0.120
0.102
0.015
0.031
0.198
0.080
0.106
Qwen2.5-VL [
7B
0.481
0.154
0.263
0,146
0,030
0.059
0.201
0.107
0.169
Holmes-
44
2B
0.913
0.804
0.566
0.467
1.519
1.437
0.329
0.370
0.355
VAD-DPO Holmes
2B
0.923
0.881
0.645
0.688
1.984
1.832
0.561
0.596
0.573
A.4 Additional Experimental Results
In Section 5, we evaluated the detection performance of VAD-DPO on mainstream benchmark datasets as well as its results on the constructed probe sets. this appendix, we further report supplementary results, including reasoning performance on HIVAU-7OK (44] and detection results on the DD-Violence dataset [35] using AP as the evaluation metric. Following [44), we assess the quality of generated descriptions on clip-, event-, and vidco-level test samples using BLEU, CIDEr, METEOR, and ROUGE scores.
Reasoning Performance. To ensure fair comparison and isolate the effect of fine-tuning, we implement VAD-DPO using Holmes- VAU 44 as the LVLM, denoted as MAD-DDPHolmmes- We com- pare VAD-DPO with tfate-of-the-ar LVLMs, including Video-Chan Cheeo-C (22). Video-LLaMA [4], LLa 1.5 [7], LLaVA- [8] Qwen2.5-VI. [[], and Holmes- VAU [44] As shown in Table A. VAD-DPO achieves the highest scores across all metrics, reflecting its ability to guide
the model to reason based on scene semanties rather than relying on co-occurrence patterns, thereby generating descriptions that are more consistent with the visual context.
24

---

Only Fire
Fire + Knffe
Boxing the Ring
Fighting in Alley
0,82
0.
0.79
1.0
0.93
1.0
0.83
Pubaliity
6.6
0.R
Redabdiity
0.8
0.
0.6
0.4
&
0,18
0.21
0.2
0.07
0.0
Normal
Abnormal
Normal
Abnomnal
Nonmal
Abnormal
Normal
Absormal
()) Compartson of probabilities in knife and fire scenarios.
(b) Comparison of probabllities in different bitting scenartos.
Figure A.2: Qualitative comparison illustrating hallucination in LVLM- VAD. While Qwen2.5- VL-7B, LLaVA-NeXT- -eX-NeeeT-iiid755
due to the presence Video-7B, LLaVA-1.5-133 and Intern VL-8B incorrectly predict an anomaly figg-frrueency co-occurring objects or patterns, our method accurately identifies the scene as normal by leveraging contextual understanding. Green boxes indicate correct predictions, red boxes indicate incorrect predictions, and purple highlights mark high-frequency co-occurring objects or patterns that may induce hallucination.
Robustness to High-Level Co-occurrence. Following the analysis Section 3.2, further evaluate VAD-DPO's s robustness to high-level co-occurrence patterns. Specifically, we examine two challenging settings: (1) frequent co-occurrence of high-bias object combinations, and (2) identical interactions occurring in different scene contexts. For each case, we measure the anomaly probability predicted by VAD-DPO to assess whether the model is misled by co-occurrence patterns or correctly grounds its judgment in scene semantics.
As shown in Figure A.2 VAD-DPO consistently provides accurate anomaly predictions in both settings. It successfully avoids false alarms when object combinations appear in normal contexts, and correctly identifies anomalies when the same interaction becomes semantically abnormal in different scene. These results validate VAD-DPO's robustness to high-level shortcut cues and its ability make context-aware, semantically grounded predictions.
Results on Different LVLMs. To further evaluate proposed approach, we apply VAD-DPO Intern IB 44).
Table A.2: Performance of VAD-DPO on Intern VL3-IB compared to baseline LVLMs
NWPU
Probe Set
Probe Set 2
We select
LVLM
Param
InternVL3-1B for fine-tuning to maintain parameter scale com- parable to Bwen2--L-2BB The results are summarized in Ta- ble A.2 Specifically, VAD-DPO improves the average frame-level AUC of Intern VL3-1B VL3- on VAD benchmark by more than 10%, while reducing hallucination-induuel error rates from 75.0% to 24.4% on Probe Set and from 80.0% to 27.5% on Probe Set 2. Despite having significantly fewer parameters, Intern VL fine-tuned with VAD-DPO outperforms the off-the-shelf Intern VL338B on both standard VAD benchmarks and hallucination-ssesitte probe sets. These results demonstrate that our method is effective not only for Qwen-based models but also generalizes well other LVLM architectures.
AUC f
FPR
FNR J
Intern
8B
71.2
70.0
75.0
InternVL3
1B
69.3
75.0
80.0
VAD-DPO (IntenVL3)
1B
76.3
24.4
27.5
A.5 Comparisons with Methods for Mitigating Hallucination
Co-occurrence has been identified as a cause of object hallucination in prior VLM studies. However, we argue that the nature of co-occurrence and its resulting failure mode in VAD are fundamentally different, which constitutes the core focus of our work. Object hallucination typically arises from object-context co-occurrence. In contrast, VAD hallucination stems from object-label co-occurrence: the model does invent nonexistent objects but correctly detects real ones (e.g., a fire in the scene) and mistakenly classifies them as anomalous solely due to their frequent co-occurrence with anomaly
25

---

labels during training. This represents a more severe form of shortcut learning, where the model bypasses contextual reasoning and directly maps object presence to anomaly prediction. Our work therefore addresses VAD-specific challenge: not whether an object exists, but what its presence signifies in a dynamic, context-dependent scene.
A.6 More Qualitative Examples
To further demonstrate the effectiveness of VAD-DPO in mitigating hallucination and enhancing scene understanding, we provide additional qualitative examples across diverse scenes. By comparing VAD- DPO with baseline models, we highlight its ability to produce context-aware and semantically aligned predictions, especially in cases where conventional LVLMs tend rely on shortcut correlations.
High-frequency co ao-ocuurinng pattern with normal-related text Walking
System Prompl: Determine whether an abnormal event occurs in the given video.
Scene Semantic: An abnormal event: pedestrians are crossing the road without crosswalk.
Prediction of Qwen2.5-VL-
No, pedestrians is simply walking across the road, and everything appears normal in the scene.
Prediction of L La LAVA- TeA----eT-iiiee--
No, people are casually walking, and there no indication of any abnormal activity.
Prediction L
No, a walking pedestrian is visible, and nothing unusual seems to be occurring.
Prediction of InternVI. VL.38BB
No, the scene just shows individual crossing the street on foot, which seems typical.
Prediction Ours:
Yes, a pedestrian crossing the street without crosswalk, which constibutes abnormal event.
High frequecyy co- co-occurting object with anomaly- anomaly-rrllted related text: P Police Car
System Prompt: D Determine whether abnornal event occurs in the given video.
Scene Semantic: A normal campus road where police car is on routine patrol.
Prediction of Qwenn.5-VL--BB:
Yes, police car approaching in the scene, suggesting that something abnormal might be taking place.
Prediction of LaLLVA--ee--viie
Yes, the presence of a police car on the road implies that an unusual incident could be happening..
Prediction of LLaVA-1.5-138
Yes, the police car's 's appearance in the seene points toward possible emergency activity.
Prediction of InternVL3-3
Yes, a police car is approaching in the scene, suggesting that something abnormal might be taking place.
Prediction of Ours:
No, the scene shows pedestrians walking calmly the distance and police entering routinely abnormal event is occurring.
Figure A.3: Qualitative comparison illustrating hallucination in LVLM-based VAD. While Qwen2.5- VL-7B, LLaVA-NeeTT-vide--- L.LVA--1.--33B. and Intern incorrectly predict an anomaly due to the presence of figh-freuuecc co-occurring objects or patterns, our method accurately identifies the scene as normal by leveraging contextual understanding. Green boxes indicate correct predictions, red boxes indicate incorrect predictions, and purple highlights mark high-frequency co-occurring objects or patterns that may induce hallucination.
B Limitations
In this section, we discuss the current limitations of our proposed approach, potential avenues for future improvements, and the broader socictal impacts of our work. While our method has
26

---

demonstrated significant gains in both anomaly detection and reasoning quality, there remain areas where further research is warranted.
Limitation 1:
Model scale and resource constraints. Due computational constraints, experiments with VAD-DPO are limited to 2B-scale laagunn models. Although results already show strong improvements over larger baselines, it is unclear whether our findings generalize consistently to larger LVLMs or highly parameterized architectures.
Future Work. We plan to extend VAD-DPO to larger models such as Qwen2.5- VL-77B or LLaVA- Next-Vide Video-dB,, and examine whether preference optimization at scale can yield even more robust mitigation of hallucinations and further enhance reasoning capacity under complex scene semantics.
Limitation 2: Residual hallucination effects. Despite effectively reducing shortcut-driven hal- lucinations, VAD-DPO does not eliminate all forms of hallucination. Beyond co-occurrence bias, hallucinations can still arise from other sources such as strong language priors or modality dominance, where the model overly trusts textual cues even when they conflict with visual evidence.
Future Work. Future efforts could investigate hallucination attribution across different causal factors by disentangling the contributions of visual, textual, and modal signals. One promis- ing direction is to incorporate introspective decoding or modality-aware routing mechanisms that explicitly modulate information flow based semantic consistency across modalities.
Limitation 3: Shortcut-driven bias and fairness risks. While VAD-DPO effectively mitigates shortcut reliance by optimizing preference alignment, does not explicitly address group-sensitive fairness concerns. Shortcut-driven ballucinations-suude as associating high-frequency visual cues (e.g., fire, police cars) with anomaly semantics can lead to biased predictions in surveillance scenarios. Such bias may disproportionately affect certain environments or communities by incorrectly flagging contextually normal activities as anomalous, raising potential fairness and ethical risks in real-world deployment.
Future Work. Future research should extend VAD-DPO toward fairness-aware anomaly detec- tion. Promising directions include incorporating demographic-sensitte faimess objectives when annotations include individual-level attributes (e.g., age, clothing, or ethnicity), analyzing disparate false alarm rates across population groups or geographic regions, and integrating preference opti- mization with fairness-oriented training strategies such as adversarial debiasing or group reweighting. Furthermore, we plan explicitly connect shortcut-driven hallucination risks with broader socictal implications to ensure that robustness and fairness are jointly promoted in safety-critical applications.
C Broader Societal Impacts
Our work contributes to improving the robustness and interpretability of video anomaly detection systems, especially safety-critical applications such as surveillance, autonomous monitoring, and emergency response. By reducing ally grounded reasoning. VAD-DPO can enhance the trustworthiness and reliability of Al-assisted decision-making in real-world deployments. More broadly. this work takes step toward addressing a core challenge the deployment of large vasion-language models: the misalignment between statistical correlations and semantic correctness. By explicitly training models to prefer semantically consistent responses over shortcut-driven predictions, VAD-DPO offers a principled way to mitigate one of the most pervasive limitations of current multimodal AI systems. In doing so, approach supports the development of more transparent and accountable AI, where the decision- decision-making pro- cess is easier to interpret, verify, and trust. We believe these directions are particularly valuable as foundation models continue to be adopted in domains where model outputs carry real consequences for public safety, resource allocation, and ethical oversight.
inneciinationinnduce false alarms and promoting visu-
Nevertheless, we acknowledge that improved reasoning capabilities in surveillance systems may also raise concerns regarding increased surveillance reach or misuse in privacy-sensitvg contexts. To mitigate such risks, all probe data used in this study have been anonymized, with visible faces masked to protect personal identity. We also recognize that masking alone may not be sufficient to eliminate
27

---

potential privacy risks; thus, future releases of our probe sets will adopt stronger anonymization techniques such as full-face blurring or obfuscation.
Finally, we emphasize the importance of clearly defining deployment boundaries and ensuring appropriate regulatory oversight. Responsible of video anomaly detection technologies must prevent potential misuse or overreach in surveillance scenarios that could raise concerns about civil liberties and human rights. Careful consideration of ethical guidelines and governance frameworks will be essential to align technical progress with societal values.
28


Reviews:
[{"review_id": "mlCSmxsLE3", "reviewer": "Reviewer_b99r", "summary": "This paper reveals a hallucination phenomenon in LVLM-based VAD: LVLM tends to rely on statistical shortcuts learned during pre-training rather than performing reasoning when detecting anomalies. Hence, this paper proposes VAD-DPO to train models on visually similar videos with contrasting anomaly labels, encouraging alignment with semantic consistency rather than co-occurrence shortcuts. Experiments show the effectiveness of the proposed method.", "questions": "In Table 6, what is the difference between DPO and VAD-DPO?", "limitations": "See weaknesses", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper reveals the shortcut phenomenon of existing VAD evaluation and conducts an empirical analysis. This is an important finding for the VAD research community.\n2. The evaluation is comprehensively conducted on six different datasets.\n3. This paper is well-written, well-organized, and easy to follow.\n\nWeaknesses:\n1. The connection between the revealed shortcut phenomenon and the proposed VAD-DPO method is not strong; it feels more like two separate contributions. How DPO addresses the shortcut issue should be better demonstrated.\n2. The proposed method considers only the naive DPO algorithm. More advanced approaches, such as GRPO [r1], or a newly proposed VAD-specific algorithm, could be explored.\n3. The proposed method is only evaluated on the Qwen2-VL backbone, which is insufficient. At least one additional backbone, such as InternVL, should be considered.\n4. Figure 4 provides a good qualitative illustration. However, a quantitative comparison is needed to demonstrate VAD-DPO's advantage in hallucination cases.\n\nJustification:\n\nThis paper identifies the shortcut phenomenon in existing VAD evaluations, which is a meaningful contribution. However, several concerns remain. I suggest a Borderline Accept at this stage.\n\n[r1] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).", "was_revised": true, "final_justification": "I am satisfied with the rebuttal, which has addressed my concerns. I'm raising my rating to 6: Strong Accept. This is a great paper and is recommended for presentation at the venue.", "rebuttal": "We sincerely thank Reviewer b99r for the thorough review and constructive comments. We particularly appreciate your recognition of our **shortcut revelation**, **comprehensive evaluation**, and **clarity of writing**. Below, we address the main concerns raised in your review.\n\n***W1: Clarify the connection between the shortcut phenomenon and the proposed VAD-DPO method.***\n\n**A1**: Thank you for highlighting this concern. VAD-DPO is designed *specifically* to address the shortcut issue we identified. The core idea is to construct **preference pairs $(y_w, y_l)$ that are visually similar but semantically contrasting**, as discussed in Section 4.2. For example, both \"a person hitting another in a street alley\" and \"a person hitting another in a boxing ring\" involve the same high-frequency interaction pattern (hitting), which often co-occurs with anomaly-related phrases. However, only the former is truly anomalous. By training the model to prefer the semantically correct output $y_w$ (i.e., boxing ring is normal) over the incorrect $y_l$ (i.e., street fight is abnormal), VAD-DPO explicitly forces the model to go beyond surface-level correlations and reason about **scene semantics**. This mechanism directly targets and mitigates co-occurrence-driven hallucinations. To further demonstrate this connection, we will revise Section 1 (Introduction) and Section 4 (Method) to highlight the causal link between co-occurrence shortcuts and the VAD-DPO.\n\nImportantly, we already provide **quantitative evidence** of this mitigation effect in **Appendix A.4 of the Supplementary Material**. To make this clearer, we also present an **expanded comparison table (Table III in our response to W4)**, which highlights the mitigation effect across multiple models. Specifically, VAD-DPO reduces false positive rates on Probe Set 1 from 60–80% (in baseline LVLMs) to 13%, and false negative rates on Probe Set 2 from 70–80% to 12.5%. These results directly confirm that VAD-DPO alleviates shortcut reliance. We will incorporate these findings into the main paper for clarity..\n\n***W2: More advanced approaches, such as GRPO, or a newly proposed VAD-specific algorithm, could be explored.***\n\n**A2**: We thank the reviewer for this insightful suggestion. Indeed, GRPO and other advanced variants of DPO provide valuable directions for future exploration. Our primary goal in this paper is to **systematically identify and address the co-occurrence-induced hallucination phenomenon in LVLM-based VAD**, which, to the best of our knowledge, has not been studied before. To establish a solid and interpretable foundation, we adopt the DPO formulation, which directly optimizes semantic preference and offers a stable framework for counter-example training.\n\nTo empirically address your suggestion, we conducted new experiments adapting the GRPO algorithm to our VAD task. The results are summarized in Table I below. While GRPO improves over baseline LVLMs, **it underperforms compared to VAD-DPO**. We attribute this to two main reasons: (1) The core challenge of our task lies in distinguishing between visually similar but semantically contrasting videos, which aligns more naturally with the pairwise formulation of DPO than the group-wise ranking used in GRPO. (2) VAD-DPO integrates an Anchored Loss, which is crucial for stabilizing training and preventing the model from forgetting correct semantics. This VAD-specific design leads to better results, while GRPO lacks such task-aware alignment mechanisms.\n\nThese results validate the value of our approach, highlighting the importance of designing solutions specifically tailored to VAD. We plan to further explore VAD-specific alignment strategies that account for contextual semantics in future work.\n\n\n``Table I. Performance comparison on VAD benchmarks and hallucination probe sets across Baseline, GRPO, and VAD-DPO.``\n\n|LVLM|Params|SHTech (AUC↑)|UCF (AUC↑)|XD (AUC↑)|Campus (AUC↑)|MSAD (AUC↑)|Probe Set 1 (FPR↓) |Probe Set 2 (FNR↓)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL|7B|79.4|78.8|83.2|71.9|75.9|67.5|77.5|\n|LLaVA-1.5|13B|76.3|72.8|79.6|70.3|75.1|65.6|75.0|\n|Qwen2-VL|2B|74.7|71.1|76.8|68.9|74.0|72.5|80.0|\n|GRPO (Qwen2-VL)|2B|79.9|77.4|84.2|75.3|78.6|45.0|32.5|\n|**VAD-DPO (Qwen2-VL)**|**2B**|**87.2**|**86.2**|**88.5**|**79.1**|**85.4**|**13.1**|**12.5**|\n\n\n\n***W3: The proposed method is only evaluated on the Qwen2-VL backbone, which is insufficient. At least one additional backbone, such as InternVL, should be considered.***\n\n**A3:** We thank the reviewer for the valuable suggestion. To address this concern, we have conducted new experiments by applying our VAD-DPO method to the InternVL3-1B. We selected InternVL3-1B for fine-tuning to maintain a comparable parameter scale to Qwen2-VL-2B, thereby ensuring a fair comparison under similar resource constraints. The results are summarized in Table II below.\n\nSpecifically, VAD-DPO improves the frame-level AUC of InternVL3-1B by **over 10%** on average across VAD benchmarks, and reduces hallucination-induced error rates on Probe Set 1 from 75.0% to 24.4%, and on Probe Set 2 from 80.0% to 27.5%. Despite having significantly fewer parameters, the fine-tuned InternVL3-1B with VAD-DPO **outperforms the off-the-shelf InternVL3-8B** on both standard VAD benchmarks and hallucination-sensitive probe sets. These results demonstrate that our method is not only effective on Qwen-based models, but also generalizes well to other LVLM architectures. Overall, VAD-DPO emerges as a generalizable optimization framework capable of improving performanc across different backbones.\n\n``Table II. Performance of InternVL3-1B before and after VAD-DPO, compared to InternVL3-8B.``\n\n|LVLM|Params|SHTech (AUC↑)|UCF (AUC↑)|XD (AUC↑)|Campus (AUC↑)|MSAD (AUC↑)|Probe Set 1 (FPR↓) |Probe Set 2 (FNR↓)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|InternVL3|8B|79.6|77.9|83.0|71.2|75.4|70.0|75.0|\n|InternVL3|1B|74.2|71.6|75.7|69.3|72.2|75.0|80.0|\n|**VAD-DPO (InternVL3)**|**1B**|**85.9**|**85.4**|**87.3**|**76.7**|**82.1**|**24.4**|**27.5**|\n\n\n\n***W4: A quantitative comparison is needed to demonstrate VAD-DPO's advantage in hallucination cases.***\n\n**A4**: We sincerely thank the reviewer for this constructive suggestion. We fully agree that quantitative evaluation is essential to demonstrate the effectiveness of our method in hallucination scenarios.\n\nIn fact, we have already conducted a dedicated analysis and reported the results in Table A.1 of Appendix A.4. Due to space constraints, this important result was initially placed in the supplementary material. In that table, VAD-DPO reduces the hallucination-induced false positive rate on Probe Set 1 from ~70% to **13.1%**, and the false negative rate on Probe Set 2 from ~80% to **12.5%**, clearly demonstrating its ability to mitigate shortcut-driven hallucinations.\n\nWe appreciate the reviewer’s suggestion to highlight this result more prominently. Accordingly, we will move this analysis into the main paper in the final version. Furthermore, to provide a more comprehensive comparison, we extend the original table by incorporating both domain-specific expert models (Holmes-VAU [1] and Hawk [2]) and LVLM-based training-free VAD methods, including LAVAD [3], MMEval [4], and AnomalyRuler [5]. The updated version is presented below as **Table III**.\n\n``Table III. Quantitative comparison on hallucination probe sets.``\n|Model|Param.|Fire (FPR↓)|Gun (FPR↓)|Blood (FPR↓)|Knife (FPR↓)|Avg. (FPR↓)|Set 2 (FNR↓)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL|7B|80.0|65.0|67.5|57.5|67.5|77.5|\n|Qwen2.5-VL|32B|67.5|47.5|60.0|50.0|56.3|60.0|\n|LLaVA-1.5|7B|85.0|80.0|77.5|72.5|78.8|82.5|\n|LLaVA-1.5|13B|75.0|62.5|65.0|60.0|65.6|75.0|\n|LLaVA-NeXT-Video|7B|82.5|70.0|67.5|55.0|68.8|72.5|\n|LLaVA-NeXT-Video|34B|67.5|52.5|62.5|47.5|57.5|57.5|\n|InternVL3|8B|75.0|72.5|65.0|60.0| 68.1|80.0|\n|LAVAD|13B|85.0|82.5|80.0|87.5|82.5|82.5|\n|MMEval|7B|77.5|80.0|85.0|85.0|81.9|80.0|\n|AnomalyRuler|7B|82.5|87.5|90.0|80.0|81.9|82.5|\n|Hawk|7B|90.0|80.0|77.5|72.5|85.0|77.5|\n|Holmes-VAU|2B|75.0|80.0|72.5|72.5| 75.0|80.0|\n|**VAD-DPO (Ours)**|**2B**|**15.0**|**12.5**|**17.5**|**7.5**| **13.1**| **12.5**|\n\n[1] Holmes-vau: Towards long-term video anomaly understanding at any granularity. CVPR 2025.\\\n[2] Hawk: Learning to understand open-world video anomalies. NeurIPS 2024.\\\n[3] Harnessing large language models for training-free video anomaly detection. CVPR 2024.\\\n[4] Uncovering what why and how: A comprehensive benchmark for causation understanding of video anomaly. CVPR 2024.\\\n[5] Follow the rules: Reasoning for video anomaly detection with large language models. ECCV 2024.\n\n***Q1:In Table 6, what is the difference between DPO and VAD-DPO?***\n\n**A5**: Thank you for the thoughtful question. The key difference lies in our loss function design, as detailed in Equations (3)–(6) of the paper.\n\nStandard DPO uses the contrastive loss in Equation (3), which encourages the model to assign higher probability to the preferred output (**$y_w$**) over the less preferred one (**$y_l$**). Our proposed VAD-DPO, defined in Equation (6), extends this by adding an additional **anchored loss term $L_{Anc}$** (Equation (5)), which is scaled by a hyperparameter $γ$. The purpose of $L_{Anc}$ is to **explicitly encourage the model to maintain a strong preference for semantically correct samples**, even in the absence of direct contrast. This helps improve training stability, which is especially important in VAD due to the subtle and context-dependent nature of semantic alignment.\n\nWe will clarify this distinction more explicitly in Section 4.2 of the final version. Overall, our design remains faithful to the core philosophy of DPO but introduces a lightweight and task-specific extension tailored for VAD."}, {"review_id": "DQOJi1ERFJ", "reviewer": "Reviewer_BrW1", "summary": "This paper focuses on tackling the **hallucination issues** in video anomaly understanding and identifies **co-occurrence patterns between visual elements and textual descriptions** as the root cause of the problem. Specifically, these co-occurrence patterns lead LVLMs to make incorrect predictions when high-frequency visual objects appear in semantically mismatched contexts, as the models rely on statistical shortcuts rather than true scene understanding. To address this challenge, the authors propose a novel **Direct Preference Optimization (DPO)-based approach**. By constructing visually similar but semantically contrasting video pairs, this method encourages the model to make predictions that are aligned with scene semantics instead of co-occurrence patterns. Extensive experiments demonstrate that this DPO-based strategy significantly enhances LVLMs’ capabilities in anomaly detection and reasoning, particularly by improving their scene-semantic grounding and reducing hallucination effects.", "questions": "See the weaknesses.\n\nBesides, the authors should clearly explain how the performance of these multimodal large models is calculated for anomaly detection tasks. This clarification would greatly enhance the reproducibility of the research.", "limitations": "See the weaknesses.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**  \n1. **Comprehensive Problem Analysis**: The authors present a thorough and insightful examination of the hallucination problem in anomaly scenarios for LVLMs, making a valuable contribution to the field.  \n2. **Well-Founded Methodology**: The proposed DPO-based approach (VAD-DPO) is well-motivated, with strong logical reasoning underpinning its design and implementation.  \n\n**Weaknesses:**  \n1. **Limited Data Scale**: The training dataset size (~1K preference pairs) may be insufficient for effective DPO alignment, potentially compromising the model's generalization ability.  \n2. **Insufficient Model Comparisons**:  \n   - The evaluation lacks a comparison with **domain-specific LVLMs** (e.g., Hawk [25], Holmes-VAU [39]), which should be included as baselines in Table 2 and Table 5.  \n   - Comparisons with general-purpose LVLMs (e.g., Qwen-VL) are less convincing than direct comparisons against state-of-the-art anomaly-focused models.  \n3. **Inadequate Validation of Co-Occurrence Patterns**: The authors fail to sufficiently demonstrate the alleviation of Co-Occurrence Patterns through experimental results. In the current version, only final results are provided, without a detailed analysis of the extent to which the Co-Occurrence was addressed.", "was_revised": true, "final_justification": "I maintain my current score.", "rebuttal": "We sincerely thank Reviewer BrW1 for the thorough review and constructive comments. We particularly appreciate your recognition of our **comprehensive problem analysis** and **well-founded methodology**, which we believe are central to our contributions. Below, we address the main concerns raised in your review.\n\n***W1: Limited Data Scale.***\n\n**A1**: We thank the reviewer for raising this important point. We acknowledge that a dataset of ~1K preference pairs is relatively small in scale. However, the effectiveness of DPO relies not only on quantity, but also on the **semantic clarity and contrastiveness** of the preference pairs. As described in Section 4.2 and Appendix A.3 of the Supplementary Material, our preference pairs are carefully constructed to maximize semantic contrast while minimizing visual confounds, thus providing strong and unambiguous optimization signals.\n\nMoreover, we have evaluated the resulting VAD-DPO model on **four general-purpose video understanding benchmarks**: MVBench [1], PerceptionTest [2], EgoSchema [3], and Video-MME [4]. These benchmarks span diverse domains, including egocentric actions, schematic reasoning, and multi-modal video QA. As shown in Table I, **VAD-DPO achieves a slight improvement** over the original Qwen2-VL-2B, despite not being trained on any of these datasets. This indicates that preference tuning not only preserves the model's generalization ability, but may also enhance it by improving semantic grounding. These results demonstrate that VAD-DPO maintains robust generalization even when trained on a relatively small yet carefully constructed set of preference pairs.\n\n``Table I. Overall performance (%) comparison on general-purpose video benchmarks.``\n|Benchmark|Qwen2-VL-2B|VAD-DPO (Ours)|\n|:---:|:---:|:---:|\n|MVBench|63.2|**63.6**|\n|PerceptionTest|53.9|**55.0**|\n|EgoSchema|54.9|**55.1**|\n|Video-MME|60.4|**61.2**|\n\n[1] Mvbench: A comprehensive multi-modal video understanding benchmark. CVPR 2024.\\\n[2] Perception test: A diagnostic benchmark for multimodal video models. NeurIPS 2023.\\\n[3] Egoschema: A diagnostic benchmark for very long-form video language understanding. NeurIPS 2023.\\\n[4] Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. CVPR 2025.\n\n\n***W2: Insufficient Model Comparisons.***\n\n**A2:** We thank the reviewer for this valuable suggestion. We acknowledge that our initial version lacked direct comparisons with domain-specific LVLMs. To address this, we have conducted additional experiments under the same settings as Table 2 and Table 5 in the main paper. \n\n**Table II** supplements the results in Table 2 by including both domain-specific expert models (Holmes-VAU [5] and Hawk [6]), which are fine-tuned on VAD datasets, and training-free LVLM-based VAD methods (LAVAD [7], MMeval [8], and AnomalyRuler [9]). All results are obtained using the released code and checkpoints, without any retraining or manual tuning.\n\nDespite being trained specifically for VAD, the expert models exhibit substantial hallucination rates. We attribute this to the limited amount of true anomaly samples in current VAD datasets, which encourages the model to rely on spurious co-occurrence patterns during supervised fine-tuning. In contrast, **VAD-DPO reduces the average false positive rate from over 75% in expert models such as Holmes-VAU and Hawk to just 13.1%**, a remarkable reduction of over 80%, demonstrating its unique ability to move beyond such shortcut reliance. Additionally, the training-free methods suffer from severe hallucinations as well, which is expected given their reliance on earlier-generation LVLMs. For instance, LAVAD uses BLIP2 for vision encoding with LLaMA-2-13B, MMeval is based on Vicuna-7B, and AnomalyRuler employs Mistral-7B-Instruct-v0.2. These models generally fall short compared to our approach due to limited multi-modal reasoning capability.\n\n``Table II. Quantitative comparison on hallucination probe sets.``\n|Model|Param.|Fire (FPR↓)|Gun (FPR↓)|Blood (FPR↓)|Knife (FPR↓)|Avg. (FPR↓)|Set 2 (FNR↓)|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Qwen2.5-VL|7B|80.0|65.0|67.5|57.5|67.5|77.5|\n|Qwen2.5-VL|32B|67.5|47.5|60.0|50.0|56.3|60.0|\n|LLaVA-1.5|7B|85.0|80.0|77.5|72.5|78.8|82.5|\n|LLaVA-1.5|13B|75.0|62.5|65.0|60.0|65.6|75.0|\n|LLaVA-NeXT-Video|7B|82.5|70.0|67.5|55.0|68.8|72.5|\n|LLaVA-NeXT-Video|34B|67.5|52.5|62.5|47.5|57.5|57.5|\n|InternVL3|8B|75.0|72.5|65.0|60.0| 68.1|80.0|\n|LAVAD|13B|85.0|82.5|80.0|87.5|82.5|82.5|\n|MMEval|7B|77.5|80.0|85.0|85.0|81.9|80.0|\n|AnomalyRuler|7B|82.5|87.5|90.0|80.0|81.9|82.5|\n|Hawk|7B|90.0|80.0|77.5|72.5|85.0|77.5|\n|Holmes-VAU|2B|75.0|80.0|72.5|72.5| 75.0|80.0|\n|**VAD-DPO (Ours)**|**2B**|**15.0**|**12.5**|**17.5**|**7.5**| **13.1**| **12.5**|\n\n**Table III** further compares VAD-DPO with domain-specific models on the HIVAU-70K benchmark. To ensure a fair comparison, we also fine-tuned Hawk on the training split of HIVAU-70K and report its post-tuning performance alongside Holmes-VAU and our VAD-DPO. Again, our method achieves clear gains, confirming its advantage even with anomaly-focused models. We will include these extended comparisons in the final version of the paper to ensure a more complete evaluation.\n\n``Table III. Performance on HIVAU-70K across different metrics. (C: Caption, E: Event, V: Video)``\n\n|Method|Params|BLEU-C|BLEU-E|BLEU-V|CIDEr-C|CIDEr-E|CIDEr-V|ROUGE-C|ROUGE-E| ROUGE-V|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|LAVAD|13B|0.236|0.115|0.127|0.138|0.093|0.104|0.109|0.105|0.072|\n|MMEval|7B|0.209|0.130|0.135|0.152|0.146|0.112|0.138|0.092|0.103|\n|AnomalyRuler|7B|0.253|0.088|0.122|0.069|0.086|0.162|0.194|0.117|0.108|\n|Hawk|7B|0.833|0.556|0.347|0.328|0.979|1.064|0.281|0.299|0.284|\n|Holmes-VAU|2B|0.913|0.804|0.566|0.467|1.519|1.437|0.329|0.370|0.355|\n|**VAD-DPO**|**2B**|**0.923**|**0.881**|**0.645**|**0.688**|**1.984**|**1.832**|**0.561**|**0.596**|**0.573**|\n\n\n[5] Holmes-vau: Towards long-term video anomaly understanding at any granularity. CVPR 2025.\\\n[6] Hawk: Learning to understand open-world video anomalies. NeurIPS 2024.\\\n[7] Harnessing large language models for training-free video anomaly detection. CVPR 2024.\\\n[8] Uncovering what why and how: A comprehensive benchmark for causation understanding of video anomaly. CVPR 2024.\\\n[9] Follow the rules: Reasoning for video anomaly detection with large language models. ECCV 2024.\n\n\n\n***W3: Inadequate Validation of Co-Occurrence Patterns.***\n\n**A3**: We sincerely thank the reviewer for this insightful observation. We agree that demonstrating the alleviation of co-occurrence-induced errors is essential to validating the effectiveness of our method.\n\nIn fact, we conducted a dedicated experiment using our co-occurrence-sensitive probe sets. As described in Appendix A.4 (Table A.1) of the Supplementary Material, these probes evaluate whether the model misclassifies scenes due to the presence of high-frequency anomaly-associated visual patterns in misleading contexts. Following your valuable suggestion, we have now expanded this analysis to include domain-specific VAD models, including Holmes-VAU and Hawk, as summarized in **Table II in A2**. While even expert models exhibit high error rates (often >75% FPR), our **VAD-DPO slashes the average False Positive Rate to just 13.1%**, a remarkable reduction of over 80% compared to these anomaly-focused models. These findings offer strong quantitative evidence that VAD-DPO successfully mitigates shortcut learning from co-occurrence patterns. \n\nWe will include this comprehensive comparison in the final revision of the paper to ensure a transparent evaluation.\n\n***Q1: The authors should clearly explain how the performance of these multimodal large models is calculated for anomaly detection tasks. This clarification would greatly enhance the reproducibility of the research.***\n\n**A4**: We sincerely thank the reviewer for this crucial question regarding reproducibility. Our evaluation protocol follows the established precedent set by prior work [7], and consists of two main stages for deriving quantitative anomaly scores from LVLMs:\n\n- **Frame Sampling and Scoring:** For each video segment, we uniformly sample 8 frames and pair each with a scoring-style prompt, consistent with LAVAD [5]. For example, we ask the LVLM: *“If you were a law enforcement agency, how would you rate the scene described on a scale from 0 to 1, with 0 representing a standard scene and 1 denoting a scene with suspicious activities?”* The model then directly outputs a numerical score (e.g., 0.83), which serves as the *initial anomaly score* for that frame.\n- **Score Refinement and Frame-wise Assignment:** Following [5], we refine these sparse scores through a **weighted aggregation mechanism**. Specifically, we leverage temporal attention weights from the decoding process to smoothly propagate scores from sampled frames to their neighboring frames. This produces a *dense per-frame anomaly score sequence* covering the entire video.\n\nThe resulting per-frame scores are directly compared with ground-truth binary frame-level annotations to compute the AUC using the standard ROC formulation. This approach ensures that our evaluation remains consistent with prior works. We will include this procedure in the final version of the paper to enhance reproducibility."}, {"review_id": "R5PzNgZsqT", "reviewer": "Reviewer_TxqF", "summary": "This paper investigates whether Large Vision-Language Models (LVLMs) for video anomaly detection (VAD) detect video anomalies based on real understanding of the scene or superficial statistical shortcuts learned during pretraining. To probe this, the authors create two synthetic datasets: one with normal scenes augmented with anomalous objects (e.g., a knife), which appear frequently in the pre-training dataset with anomaly-related phrases, and one with abnormal scenes augmented with normal objects. Their findings reveal that LVLMs often rely on such shortcuts. To address this, they propose VAD-DPO, a direct preference optimization method that encourages LVLMs to focus on the semantics of the scene by promoting better alignment between correct visual-text pairs over semantically different but visually similar pairs with the opposite label. VAD-DPO outperforms both training-free approaches and off-the-shelf LVLMs on six VAD benchmarks.", "questions": "1. How does VAD-DPO perform on the probing datasets compared to the off-the-shelf baseline? Additionally, how do LVLMs perform on the probing videos when the anomaly- or normality-related objects are removed?\n2. How can you make sure that the generated videos are normal after generating them with anomaly-related objects?\n3. What do the GPT-generated textual phrases look like, and how are frame-level anomaly scores computed?\n4. What is the novelty of the proposed VAD-DPO method compared to existing approaches that use visually similar but semantically contrasting data?\n5. Why did you report AUC-ROC for the XD-Violence dataset instead of the standard AP metric?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses a relevant question in video anomaly detection (VAD): whether LVLMs rely on true scene understanding or visual/textual shortcuts associated with normality or abnormality. This is studied by probing LVLMs using synthetic videos augmented with such cues.\n2. The authors propose a solution, VAD-DPO, to make the model focus on the scene instead of relying on such shortcuts. The method outperforms larger off-the-shelf LVLMs, demonstrating its effectiveness.\n3. The paper is generally well-written and clearly motivates the problem, with valuable insights for the community that should encourage future methods to consider such shortcut learning. The authors claim that the study of these limitations in VAD is a novel contribution.\n\nWeaknesses:\n1. Despite its strengths, the paper lacks some evaluations and clarifications. In particular, it does not report how VAD-DPO performs on the probing datasets compared to the off-the-shelf baseline, which would help validate its effectiveness against shortcuts. Also, to better understand the reliance on shortcuts, the same analysis presented in Table 2 could have been repeated on the same synthetic videos without the anomaly/normality-related objects. \n2. Some synthetic videos used in the probing datasets may be ambiguous. For example, in Fig. 1, one could expect a fire if there were charcoal under the grill or a pan was used instead of a grill, however, a fire of that size on a grill may seem anomalous.\n3. Some implementation details are missing, such as examples of the 3k GPT-generated textual phrases and how frame-level anomaly scores are computed, which limits reproducibility.\n4. While the shortcut analysis may be novel for VAD, the proposed training objective, which involves training on visually similar but semantically different videos, is similar to training objectives used for other tasks.\n5. Another aspect is that the standard metric on XD-Violence is the area under the precision-recall curve, while the evaluation uses AUC.", "was_revised": true, "final_justification": "The authors addressed my concerns regarding the evaluation of VAD-DPO on the probing datasets and on the same videos without the augmented anomaly/normality-related objects. They clarified that the synthetic videos in these probing sets undergo manual verification, explained how the GPT-generated textual phrases are created (with examples), and described how the anomaly scores are computed. While not entirely new, I also appreciate the proposed strategy to help the model focus on anomalies by using visually similar but semantically different videos. The clarifications and additional results on XD-Violence using the AP metric were helpful. Given these clarifications, I am increasing my score to Accept.", "rebuttal": "We sincerely thank Reviewer TxqF for the detailed review and encouraging comments. We particularly appreciate your recognition of our relevant **problem formulation**, **effective DPO solution**, and **clear motivation with community insight**. Below, we address the main concerns raised in your review.\n\n***W1&Q1: How does VAD-DPO perform on the probing datasets compared to the off-the-shelf baseline? Additionally, how do LVLMs perform on the probing videos when the anomaly- or normality-related objects are removed?***\n\n**A1**: We thank Reviewer TxqF for the insightful question. We address the two aspects below:\n\n1. **VAD-DPO vs. Off-the-Shelf Baselines on Probing Datasets**: We conduct a dedicated evaluation using our co-occurrence-sensitive probe sets. As described in Appendix A.4 (Table A.1), these probes evaluate whether the model misclassifies scenes due to the presence of high-frequency anomaly-associated visual patterns in misleading contexts. To provide a more comprehensive comparison, we further extend this table to include domain-specific expert models (Holmes-VAU [1], Hawk [2]) and LVLM-based training-free VAD methods (LAVAD [3], MMEval [4], and AnomalyRuler [5]). The updated version is presented below as **Table I**. While even expert models exhibit high error rates (often >75% FPR), our **VAD-DPO slashes the average False Positive Rate to just 13.1%**, a remarkable reduction of over 80% compared to these anomaly-focused models. These findings offer strong quantitative evidence that VAD-DPO successfully mitigates shortcut learning from co-occurrence patterns. We will incorporate this expanded version into the final paper to directly address this concern.\n\n``Table I. Quantitative comparison on hallucination Probe Sets.``\n|Model|Param.|Fire (FPR↓)|Gun (FPR↓)|Blood (FPR↓)|Knife (FPR↓)|Avg. (FPR↓)|Set 2 (FNR↓)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL|7B|80.0|65.0|67.5|57.5|67.5|77.5|\n|Qwen2.5-VL|32B|67.5|47.5|60.0|50.0|56.3|60.0|\n|LLaVA-1.5|7B|85.0|80.0|77.5|72.5|78.8|82.5|\n|LLaVA-1.5|13B|75.0|62.5|65.0|60.0|65.6|75.0|\n|LLaVA-NeXT-Video|7B|82.5|70.0|67.5|55.0|68.8|72.5|\n|LLaVA-NeXT-Video|34B|67.5|52.5|62.5|47.5|57.5|57.5|\n|InternVL3|8B|75.0|72.5|65.0|60.0| 68.1|80.0|\n|LAVAD|13B|85.0|82.5|80.0|87.5|82.5|82.5|\n|MMEval|7B|77.5|80.0|85.0|85.0|81.9|80.0|\n|AnomalyRuler|7B|82.5|87.5|90.0|80.0|81.9|82.5|\n|Hawk|7B|90.0|80.0|77.5|72.5|85.0|77.5|\n|Holmes-VAU|2B|75.0|80.0|72.5|72.5| 75.0|80.0|\n|**VAD-DPO (Ours)**|**2B**|**15.0**|**12.5**|**17.5**|**7.5**| **13.1**| **12.5**|\n\n1. **Performance Impact of Removing Shortcut Objects**: This is an excellent point. The base videos in Probe Set 1 are manually curated normal scenes selected from standard VAD datasets. Prior to inserting objects, most models tend to assign low anomaly scores. In response to your suggestion, we further remove the inserted objects from the perturbed clips and re-evaluate all models. As shown **in Table II** below, the removal of these misleading objects significantly reduces the false positive rate across models, effectively restoring correct predictions. These results confirm that the hallucinations are indeed triggered by co-occurrence-driven patterns.\n   \n``Table II. FPR (%) on Probe Set 1 after removing misleading objects.``\n|Model|Param.|Fire|Gun|Blood|Knife|Avg.|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL|7B|7.5|5.0|7.5|2.5|5.6|\n|Qwen2.5-VL|32B|2.5|5.0|5.0|0.0|3.1|\n|LLaVA-1.5|7B|15.0|17.5|17.5|12.5|16.3|\n|LLaVA-1.5|13B|12.5|15.0|17.5|7.5|13.1|\n|LLaVA-NeXT-Video|7B|7.5|5.0|10.0|12.5|8.8|\n|LLaVA-NeXT-Video|34B|0.0|7.5|5.0|2.5|3.8|\n|InternVL3|8B|10.0|7.5|12.5|7.5|9.4|\n|LAVAD|13B|12.5|10.0|7.5|10.0|10.0|\n|MMEval|7B|12.5|12.5|7.5|10.0|10.6|\n|AnomalyRuler|7B|12.5|7.5|7.5|10.0|9.4|\n|Hawk|7B|7.5|7.5|2.5|2.5|5.0|\n|Holmes-VAU|2B|5.0|5.0|7.5|5.0|5.6|\n|**VAD-DPO (Ours)**|**2B**|**0.0**|**2.5**|**2.5**|**0.0**| **1.3**|\n\n[1] Holmes-vau: Towards long-term video anomaly understanding at any granularity. CVPR 2025.\\\n[2] Hawk: Learning to understand open-world video anomalies. NeurIPS 2024.\\\n[3] Harnessing large language models for training-free video anomaly detection. CVPR 2024.\\\n[4] Uncovering what why and how: A comprehensive benchmark for causation understanding of video anomaly. CVPR 2024.\\\n[5] Follow the rules: Reasoning for video anomaly detection with large language models. ECCV 2024.\n\n***W2&Q2: How can you make sure that the generated videos are normal after generating them with anomaly-related objects?***\n\n**A2**: We thank the reviewer for this thoughtful question. Our goal in constructing probe samples is to simulate scenes where the most plausible and commonsense interpretation is normal, despite the presence of anomaly-related objects. For example, a video depicting a man barbecuing in a backyard may include a \"fire\" object, yet the overall scene is semantically normal. To ensure the normality, we applied a strict human validation protocol:\n\n- **Manual Screening:** All videos are reviewed by multiple annotators to ensure that the inserted object does not introduce unintended anomaly cues beyond its visual presence.\n- **Exclusion of Ambiguous Cases:** We filter out videos in which the anomaly-related object could reasonably support multiple semantic interpretations (e.g., police cars in tense crowd scenes).\n\nWhile we acknowledge a degree of subjectivity is inevitable, this validation protocol is designed to minimize such ambiguity. We will add a detailed description to the final paper.\n\n\n\n***W3&Q3: What do the GPT-generated textual phrases look like, and how are frame-level anomaly scores computed?***\n\n**A3:** We thank the reviewer for this excellent question. Below we address the two components:\n\n1. **GPT-Generated Phrases**: To construct our probing datasets, we use GPT-4 to generate a diverse set of 3k candidate textual phrases related to anomaly and normality. We prompt the model with instructions such as: `Generate a list of concise phrases that describe either abnormal or normal events in video scenes.`  Some example outputs include:\n- Anomaly-related: *\"a person holding a knife\"*, *\"fire spreading rapidly\"*, *\"man lying motionless on the ground\"*  \n- Normality-related: *\"people walking across a street\"*, *\"children playing in a park\"*, *\"car driving along a road\"*\n\nWe manually review and filter these phrases to ensure semantic clarity and diversity. \n\n2. **Anomaly Score Computation**: Our evaluation protocol follows LAVAD [3] and consists of two stages:\n\n- **Frame Sampling and Scoring:** For each video segment, we uniformly sample 8 frames and pair each with a scoring-style prompt, consistent with LAVAD [5]. For example, we ask the LVLM: *“If you were a law enforcement agency, how would you rate the scene described on a scale from 0 to 1, with 0 representing a standard scene and 1 denoting a scene with suspicious activities?”* The model then directly outputs a numerical score (e.g., 0.83), which serves as the *initial anomaly score* for that frame.\n- **Score Refinement and Frame-wise Assignment:** Then, we refine these sparse scores through a weighted aggregation mechanism. Specifically, we leverage temporal attention weights from the decoding process to smoothly propagate scores from sampled frames to their neighboring frames. This produces a dense per-frame anomaly score sequence covering the entire video. \n\nWe will include this procedure in the final version to enhance reproducibility.\n\n\n***W4&Q4: What is the novelty of the proposed VAD-DPO method compared to existing approaches that use visually similar but semantically contrasting data?***\n\n**A4:** We appreciate the reviewer’s question regarding the novelty of our method. We acknowledge that the idea of learning from visually similar but semantically contrasting data is not entirely new. However, our contribution lies in applying this idea to a **specific and underexplored problem**, namely, shortcut learning caused by co-occurrence in **VAD**. Our novelty stems from the following aspects:\n\n- **Problem Definition in VAD**: To the best of our knowledge, we are the first to identify and formalize co-occurrence-induced hallucinations as a critical failure mode in LVLM-based VAD.\n- **Systematic Diagnosis of the Problem**: We provide a multi-level analysis of co-occurrence shortcuts (e.g., object, combination, interaction) and design targeted probe sets to quantify the extent of hallucination.\n- **Tailored Solution via VAD-DPO**: We introduce a framework that directly mitigates the problem. The preference-pair design is carefully aligned with the contextual semantics of anomaly detection and the specific failure patterns of LVLMs.\n\nIn summary, the **novelty of our work lies in the identification, formalization, and resolution of a overlooked problem in VAD**. We believe this form of contribution is essential for advancing reliable deployment of LVLMs in VAD.\n\n\n***W5&Q5: Why did you report AUC for the XD-Violence instead of the standard AP metric?***\n\n**A5**: We thank the reviewer for pointing out this important detail. We fully acknowledge that AP is a widely used metric for XD-Violence. Compared to AUC, this metric mainly focuses on the performance of VAD method in identifying anomalous events. In other words, AP pays attention to classifying the anomaly correctly rather than the overall separation. To address this, we conduct additional experiments and report the frame-level AP scores on XD-Violence in **Table III** below. VAD-DPO consistently outperforms all baselines under this metric as well, further validating its effectiveness. Note that the original Holmes-VAU paper reports video-level results; we therefore re-evaluated its frame-level AP using the official released model weights for a fair comparison.\n\n``Table III. Frame-level AP(%) Comparasion on XD-Violence.``\n|Model|Param.|XD|\n|:-:|:-:|:-:|\n|Qwen2.5-VL|7B|65.24|\n|LAVAD|13B|62.01|\n|MMEval|7B|69.65|\n|AnomalyRuler|7B|68.22|\n|Hawk|7B|75.39|\n|Holmes-VAU|2B|83.47|\n|**VAD-DPO (Ours)**|**2B**|**84.53**|"}, {"review_id": "FHxpObHLND", "reviewer": "Reviewer_6ZUx", "summary": "The paper investigates whether Large Vision-Language Models (LVLMs) genuinely understand contextual semantics in Video Anomaly Detection (VAD) or rely on superficial visual-textual co-occurrence patterns from pretraining. Through co-occurrence analysis and controlled probes, the authors reveal a \"hallucination\" phenomenon: LVLMs frequently misclassify anomalies based on statistical shortcuts (e.g., associating \"fire\" with anomalies even in benign contexts like barbecues). To mitigate this, they propose VAD-DPO, a direct preference optimization method that trains models on contrastive video pairs with identical objects/scenes but divergent anomaly labels. Experiments on six VAD benchmarks show state-of-the-art results.", "questions": "See the detailed comments in weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Quality: VAD-DPO outperforms 7B/13B models with only 2B parameters (Table 4). Ablations (Table 6) validate the DPO framework.\n\n- Clarity: Well-structured, with clear figures illustrating key concepts (e.g., Fig 1, 4).\n\n- Significance: This paper tries to address the problem of hallucination in VLMs on the task of video anomaly detection, which has broad implications for multimodal trustworthiness.\n\n- Originality: This paper explores the hallucination problem in the video anomaly detection task.\n\n\n\nWeaknesses:\n\n- Quality: (1) I believe that in general scenarios, even some words related to anomalies have meanings that are more related to non-anomalies. Compared to the way statistically analyzing bias in this paper, a more reasonable approach should be to calculate the probability that the semantics that should be non-anomalous are classified as anomalous when there are visual concepts related to anomalous words, and vice versa. It would be even better to use real data rather than constructed data. This way, it can better reflect the real situation. (2) For the probe setting, only 200 samples were generated for analysis. It is insufficient and not convincing.\n- Clarity: (1) More details about the analysis of hallucination need to be provided. For example, how do you get the associated image in co-occurrence statistics? And the reasonableness needs to be verified. (2) How are \"visually similar but semantically contrasting\" video pairs sourced/generated? Provide examples beyond object insertion (e.g., context swaps).\n- Originality: (1) Some related works [a][b][c] about hallucination in VLMs have been studied. The VAD task may not be involved, but the differences should be clarified. Besides, it is necessary to compare them to validate the effectiveness of the proposed method. (2) To validate the effectiveness of DPO, other mitigating hallucination-related methods need to be compared.\n\n\n[a] Multi-Object Hallucination in Vision Language Models\n[b] Multi-Modal Hallucination Control by Visual Information Grounding\n[c] Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding", "was_revised": true, "final_justification": "My concerns have been addressed in the rebuttal. I would like to raise my rating.", "rebuttal": "We sincerely thank Reviewer 6ZUx for the constructive review and encouraging feedback. We especially appreciate your recognition of our **strong performance**, **clear presentation**, **hallucination significance**, and **originality in the VAD setting**. Below, we address the main concerns raised in your review.\n\n***W1: Quality: (1) Recommends evaluating semantic misclassification probabilities under anomaly-related visual concepts using real data, rather than relying solely on statistical analysis and constructed probes.***\n\n**A1:** We thank the reviewer for this thoughtful and constructive suggestion. \n1. We agree with the core idea that quantifying semantic misclassification probabilities in the presence of anomaly-related visual concepts is a more direct way to measure co-occurrence bias. In fact, this is exactly what our two-stage analysis aims to achieve:\n\n- **Stage 1 – Hypothesis Formation via Co-occurrence Statistics:** As shown in Table 1, we begin by analyzing large-scale pretraining data to identify visual concepts (e.g., *fire*, *gun*) that frequently co-occur with anomaly-related text. This serves as the hypothesis generation phase, helping us pinpoint likely shortcut triggers.\n\n- **Stage 2 – Hypothesis Testing via Probe Evaluation:** The core of our analysis lies in validating this hypothesis through carefully designed probe datasets (Table 2). This directly aligns with the approach suggested by the reviewer. Specifically, our probe sets are constructed to evaluate conditional misclassification rates—e.g.,  **FPR = P(model predicts \"anomaly\" | semantics are normal $∧$ shortcut object is present)**. The resulting false positive rates of 70–80% provide direct empirical evidence of co-occurrence-induced hallucination.\n\n2. We also appreciate the reviewer’s emphasis on evaluating models in real-world settings. Indeed, this is why we evaluate our method on six real-world VAD benchmarks. At the same time, we emphasize that using **constructed data is necessary** for controlled diagnostic evaluation. In real anomaly videos, shortcut objects (e.g., guns) are often entangled with anomalous context (e.g., robbery), making it difficult to isolate causality. Our probe sets serve as **counterfactual test sets**, where inserting a shortcut object into an otherwise normal scene allows us to disentangle causal effects and clearly demonstrate that the model’s false alarm is triggered by the object.\n\nWe will clarify this two-stage methodology and its rationale more explicitly in the final version of the paper.\n\n\n***W2: Quality: (2) For the probe setting, only 200 samples were generated for analysis. It is insufficient and not convincing.***\n\n**A2:** We thank the reviewer for raising this important concern about the sample size of our probe evaluation. We would like to clarify that constructing high-quality video probes requires significant manual effort for editing, screening, and validation.\n\nWhile we agree that a larger dataset could offer even more robust analysis, we emphasize that our current probe set of 200 samples is already sufficient to reveal an **extremely strong and consistent effect** across all tested models. Crucially, our finding is not merely that \"some models make errors,\" but that **false positive rates often exceed 70%** for anomaly-related objects inserted into normal scenes, as shown in **Table I**. We also include results for domain-specific VAD models such as Holmes-VAU [1] and Hawk [2]. These models exhibit similarly high hallucination rates under the same conditions.\n\nThis consistent and severe failure pattern is observed across architectures and model sizes, demonstrating the **systemic nature** of the co-occurrence-induced hallucination problem. Therefore, we believe the current sample size is sufficient and effective for diagnosing and validating the phenomenon. We will clarify this point in the final version of the paper.\n\n``Table I. Quantitative comparison on hallucination Probe Sets.``\n|Model|Param.|Fire (FPR↓)|Gun (FPR↓)|Blood (FPR↓)|Knife (FPR↓)|Avg. (FPR↓)|Set 2 (FNR↓)|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL|7B|80.0|65.0|67.5|57.5|67.5|77.5|\n|Qwen2.5-VL|32B|67.5|47.5|60.0|50.0|56.3|60.0|\n|LLaVA-1.5|7B|85.0|80.0|77.5|72.5|78.8|82.5|\n|LLaVA-1.5|13B|75.0|62.5|65.0|60.0|65.6|75.0|\n|LLaVA-NeXT-Video|7B|82.5|70.0|67.5|55.0|68.8|72.5|\n|LLaVA-NeXT-Video|34B|67.5|52.5|62.5|47.5|57.5|57.5|\n|InternVL3|8B|75.0|72.5|65.0|60.0| 68.1|80.0|\n|LAVAD|13B|85.0|82.5|80.0|87.5|82.5|82.5|\n|MMEval|7B|77.5|80.0|85.0|85.0|81.9|80.0|\n|AnomalyRuler|7B|82.5|87.5|90.0|80.0|81.9|82.5|\n|Hawk|7B|90.0|80.0|77.5|72.5|85.0|77.5|\n|Holmes-VAU|2B|75.0|80.0|72.5|72.5| 75.0|80.0|\n|**VAD-DPO (Ours)**|**2B**|**15.0**|**12.5**|**17.5**|**7.5**| **13.1**| **12.5**|\n\n\n[1] Holmes-vau: Towards long-term video anomaly understanding at any granularity. CVPR 2025.\\\n[2] Hawk: Learning to understand open-world video anomalies. NeurIPS 2024.\n\n\n***W3: Clarity: (1) More details about the analysis of hallucination need to be provided. How do you get the associated image in co-occurrence statistics?***\n\n**A3:** We thank the reviewer for pointing out the need for more clarity in our co-occurrence analysis. As described in Section 3.1 (Lines 126–128), we conduct this analysis using the LAION-CC-SBU dataset. We begin by using GPT to generate approximately 3k candidate textual phrases that are potentially associated with anomaly or normality. These phrases are manually reviewed to ensure semantic clarity and proper categorization. We then match these phrases against LAION captions to identify text-image pairs where a caption contains a candidate phrase. For each matched image, we apply YOLOv5 to detect objects and filter out low-confidence detections to ensure statistical robustness.\n\nThis process allows us to compute co-occurrence statistics by measuring how frequently certain visual objects appear in images whose captions express either anomaly-related or normality-related concepts. In this way, the co-occurrence patterns are estimated from large-scale real-world data with careful semantic control. We will make this intermediate matching step more explicit in the final version.\n\n\n***W4: Clarity: (2) How are \"visually similar but semantically contrasting\" video pairs sourced/generated? Provide examples beyond object insertion.***\n\n**A4:** We apologize for the lack of clarity in the main paper regarding the construction of video pairs. This process is detailed in Appendix (Lines 543–553). In summary, we construct such counter-example preference pairs using two strategies:\n\n- **Intra-scene contrast**: Two clips are drawn from the same environment but differ in semantics due to subtle changes. for example, *walking normally* vs. *sudden erratic movement in a park*.\n- **Cross-scene context shift**: Two clips share the same visual activity (e.g., *driving*), but occur in different contexts that alter their semantic interpretation, e.g., *driving on a pedestrian street (abnormal)* vs. *driving on a road (normal)*.\n\nThese pairs are used as training supervision in our VAD-DPO framework to teach the model to prefer semantically correct outputs. Due to rebuttal policy restrictions, we are unable to include examples. We will add a concise visual summary with additional examples in the final version.\n\n\n***W5: Originality: Clarify differences from related hallucination works and compare with other mitigation methods.***\n\n**A5:** We thank the reviewer for pointing out these relevant works.\n\n1. **On Related Works:** We thank the reviewer for this sharp observation. We agree that co-occurrence has been studied as a cause of object hallucination in prior VLM works. However, we argue that the nature of the co-occurrence and the resulting failure mode in VAD are fundamentally different, which forms the core focus of our work. Object hallucination typically involves **object–context co-occurrence**. In contrast, VAD hallucination stems from **object–label co-occurrence**. The model does not invent new objects, it correctly detects real objects (e.g., a fire in the scene). However, it incorrectly interprets these objects as anomalous purely based on their frequent co-occurrence with anomaly labels during training. This reflects a more severe form of shortcut learning: the model bypasses contextual reasoning and directly maps object presence to anomaly prediction. Therefore, our work addresses a **VAD-specific challenge**: not whether an object exists, but what its presence means in a dynamic, context-dependent scene. \n\n2. **On Comparison with Mitigation Methods:** Our ablation study (Table 6) compares VAD-DPO with other training paradigms (e.g., instruction tuning, PPO). Reviewer-cited approaches like contrastive decoding or M3ID are **inference-time decoding strategies** rather than **training-time alignment methods**. Thus, a direct comparison is non-trivial. Nonetheless, to address this concern, we provide results in **Table II** where such decoding-based mitigation methods are compared against our VAD-DPO with Qwen2-VL-2B as backbone. As expected, our training-based method consistently outperforms decoding-time mitigation techniques. We will further clarify the scope of our contribution in the final version.\n\n``Table II. Comparison of Hallucination Mitigation Methods.``\n|Method|NWPU (AUC↑)|Set 1 (FPR↓)|Set 2 (FNR↓)|\n|:-:|:-:|:-:|:-:|\n|M3ID [3]|69.3|62.5|57.5|\n|ICD [4]|68.6|66.3|62.5|\n|VCD [5]|71.1|53.8|60.0|\n|OPERA [6]|70.2|55.0|62.5|\n|**VAD-DPO (Ours)**|**79.1**| **13.1**| **12.5**|\n\n[3] Multi-modal hallucination control by visual information grounding. CVPR 2024.\\\n[4] Mitigating hallucinations in large vision-language models with instruction contrastive decoding. ACL 2024.\\\n[5] Mitigating object hallucinations in large vision-language models through visual contrastive decoding. CVPR 2024.\\\n[6] Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. CVPR 2024."}]

Author rebuttal:

Meta-review: This paper has received ratings of 4456 after the rebuttal. The paper proposes a framework called VAD-DPO to address hallucinations introduced by visual-textual co-occurrence in large vision-language models for video anomaly detection. The proposed approach analyzes this failure mode and improves the overall performance by focusing on semantic consistency instead of statistical shortcuts. The paper has conducted comprehensive evaluations across multiple datasets, and the approach has sufficient technical novelty to mitigate hallucinations in VAD. There were some weaknesses regarding the limited data scale and insufficient comparisons with domain-specific models. The paper has made significant contributions to understanding and addressing hallucinations in VAD. The authors have generally addressed reviewer concerns during the rebuttal. Thus, the AC recommends acceptance to NeurIPS.
Decision: Accept (poster)