Title: Do LVLMs Truly Understand Video Anomalies? Revealing Hallucination via Co-Occurrence Patterns
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Anomaly Detection

Video Anomaly Reasoning
Abstract: Large Vision-Language Models (LVLMs) pretrained on large-scale multimodal data have shown promising capabilities in Video Anomaly Detection (VAD). However, their ability to reason about abnormal events based on scene semantics remains underexplored. In this paper, we investigate LVLMs’ behavior in VAD from a visual-textual co-occurrence perspective, focusing on whether their decisions are driven by statistical shortcuts between visual instances and textual phrases. By analyzing visual-textual co-occurrence in pretraining data and conducting experiments under different data settings, we reveal a hallucination phenomenon: LVLMs tend to rely on co-occurrence patterns between visual instances and textual phrases associated with either normality or abnormality, leading to incorrect predictions when these high-frequency objects appear in semantically mismatched contexts. To address this issue, we propose VAD-DPO, a direct preference optimization method supervised with counter-example pairs. By constructing visually similar but semantically contrasting video clips, VAD-DPO encourages the model to align its predictions with the semantics of scene rather than relying on co-occurrence patterns. Extensive experiments on six benchmark datasets demonstrate the effectiveness of VAD-DPO in enhancing both anomaly detection and reasoning performance, particularly in scene-dependent scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates whether Large Vision-Language Models (LVLMs) genuinely understand contextual semantics in Video Anomaly Detection (VAD) or rely on superficial visual-textual co-occurrence patterns from pretraining. Through co-occurrence analysis and controlled probes, the authors reveal a \"hallucination\" phenomenon: LVLMs frequently misclassify anomalies based on statistical shortcuts (e.g., associating \"fire\" with anomalies even in benign contexts like barbecues). To mitigate this, they propose VAD-DPO, a direct preference optimization method that trains models on contrastive video pairs with identical objects/scenes but divergent anomaly labels. Experiments on six VAD benchmarks show state-of-the-art results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Quality: VAD-DPO outperforms 7B/13B models with only 2B parameters (Table 4). Ablations (Table 6) validate the DPO framework.\n\n- Clarity: Well-structured, with clear figures illustrating key concepts (e.g., Fig 1, 4).\n\n- Significance: This paper tries to address the problem of hallucination in VLMs on the task of video anomaly detection, which has broad implications for multimodal trustworthiness.\n\n- Originality: This paper explores the hallucination problem in the video anomaly detection task.\n\n\n\nWeaknesses:\n\n- Quality: (1) I believe that in general scenarios, even some words related to anomalies have meanings that are more related to non-anomalies. Compared to the way statistically analyzing bias in this paper, a more reasonable approach should be to calculate the probability that the semantics that should be non-anomalous are classified as anomalous when there are visual concepts related to anomalous words, and vice versa. It would be even better to use real data rather than constructed data. This way, it can better reflect the real situation. (2) For the probe setting, only 200 samples were generated for analysis. It is insufficient and not convincing.\n- Clarity: (1) More details about the analysis of hallucination need to be provided. For example, how do you get the associated image in co-occurrence statistics? And the reasonableness needs to be verified. (2) How are \"visually similar but semantically contrasting\" video pairs sourced/generated? Provide examples beyond object insertion (e.g., context swaps).\n- Originality: (1) Some related works [a][b][c] about hallucination in VLMs have been studied. The VAD task may not be involved, but the differences should be clarified. Besides, it is necessary to compare them to validate the effectiveness of the proposed method. (2) To validate the effectiveness of DPO, other mitigating hallucination-related methods need to be compared.\n\n\n[a] Multi-Object Hallucination in Vision Language Models\n[b] Multi-Modal Hallucination Control by Visual Information Grounding\n[c] Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding\n\nQUESTIONS:\nSee the detailed comments in weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether Large Vision-Language Models (LVLMs) for video anomaly detection (VAD) detect video anomalies based on real understanding of the scene or superficial statistical shortcuts learned during pretraining. To probe this, the authors create two synthetic datasets: one with normal scenes augmented with anomalous objects (e.g., a knife), which appear frequently in the pre-training dataset with anomaly-related phrases, and one with abnormal scenes augmented with normal objects. Their findings reveal that LVLMs often rely on such shortcuts. To address this, they propose VAD-DPO, a direct preference optimization method that encourages LVLMs to focus on the semantics of the scene by promoting better alignment between correct visual-text pairs over semantically different but visually similar pairs with the opposite label. VAD-DPO outperforms both training-free approaches and off-the-shelf LVLMs on six VAD benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper addresses a relevant question in video anomaly detection (VAD): whether LVLMs rely on true scene understanding or visual/textual shortcuts associated with normality or abnormality. This is studied by probing LVLMs using synthetic videos augmented with such cues.\n2. The authors propose a solution, VAD-DPO, to make the model focus on the scene instead of relying on such shortcuts. The method outperforms larger off-the-shelf LVLMs, demonstrating its effectiveness.\n3. The paper is generally well-written and clearly motivates the problem, with valuable insights for the community that should encourage future methods to consider such shortcut learning. The authors claim that the study of these limitations in VAD is a novel contribution.\n\nWeaknesses:\n1. Despite its strengths, the paper lacks some evaluations and clarifications. In particular, it does not report how VAD-DPO performs on the probing datasets compared to the off-the-shelf baseline, which would help validate its effectiveness against shortcuts. Also, to better understand the reliance on shortcuts, the same analysis presented in Table 2 could have been repeated on the same synthetic videos without the anomaly/normality-related objects. \n2. Some synthetic videos used in the probing datasets may be ambiguous. For example, in Fig. 1, one could expect a fire if there were charcoal under the grill or a pan was used instead of a grill, however, a fire of that size on a grill may seem anomalous.\n3. Some implementation details are missing, such as examples of the 3k GPT-generated textual phrases and how frame-level anomaly scores are computed, which limits reproducibility.\n4. While the shortcut analysis may be novel for VAD, the proposed training objective, which involves training on visually similar but semantically different videos, is similar to training objectives used for other tasks.\n5. Another aspect is that the standard metric on XD-Violence is the area under the precision-recall curve, while the evaluation uses AUC.\n\nQUESTIONS:\n1. How does VAD-DPO perform on the probing datasets compared to the off-the-shelf baseline? Additionally, how do LVLMs perform on the probing videos when the anomaly- or normality-related objects are removed?\n2. How can you make sure that the generated videos are normal after generating them with anomaly-related objects?\n3. What do the GPT-generated textual phrases look like, and how are frame-level anomaly scores computed?\n4. What is the novelty of the proposed VAD-DPO method compared to existing approaches that use visually similar but semantically contrasting data?\n5. Why did you report AUC-ROC for the XD-Violence dataset instead of the standard AP metric?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on tackling the **hallucination issues** in video anomaly understanding and identifies **co-occurrence patterns between visual elements and textual descriptions** as the root cause of the problem. Specifically, these co-occurrence patterns lead LVLMs to make incorrect predictions when high-frequency visual objects appear in semantically mismatched contexts, as the models rely on statistical shortcuts rather than true scene understanding. To address this challenge, the authors propose a novel **Direct Preference Optimization (DPO)-based approach**. By constructing visually similar but semantically contrasting video pairs, this method encourages the model to make predictions that are aligned with scene semantics instead of co-occurrence patterns. Extensive experiments demonstrate that this DPO-based strategy significantly enhances LVLMs’ capabilities in anomaly detection and reasoning, particularly by improving their scene-semantic grounding and reducing hallucination effects.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**  \n1. **Comprehensive Problem Analysis**: The authors present a thorough and insightful examination of the hallucination problem in anomaly scenarios for LVLMs, making a valuable contribution to the field.  \n2. **Well-Founded Methodology**: The proposed DPO-based approach (VAD-DPO) is well-motivated, with strong logical reasoning underpinning its design and implementation.  \n\n**Weaknesses:**  \n1. **Limited Data Scale**: The training dataset size (~1K preference pairs) may be insufficient for effective DPO alignment, potentially compromising the model's generalization ability.  \n2. **Insufficient Model Comparisons**:  \n   - The evaluation lacks a comparison with **domain-specific LVLMs** (e.g., Hawk [25], Holmes-VAU [39]), which should be included as baselines in Table 2 and Table 5.  \n   - Comparisons with general-purpose LVLMs (e.g., Qwen-VL) are less convincing than direct comparisons against state-of-the-art anomaly-focused models.  \n3. **Inadequate Validation of Co-Occurrence Patterns**: The authors fail to sufficiently demonstrate the alleviation of Co-Occurrence Patterns through experimental results. In the current version, only final results are provided, without a detailed analysis of the extent to which the Co-Occurrence was addressed.\n\nQUESTIONS:\nSee the weaknesses.\n\nBesides, the authors should clearly explain how the performance of these multimodal large models is calculated for anomaly detection tasks. This clarification would greatly enhance the reproducibility of the research.\n\nLIMITATIONS:\nSee the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper reveals a hallucination phenomenon in LVLM-based VAD: LVLM tends to rely on statistical shortcuts learned during pre-training rather than performing reasoning when detecting anomalies. Hence, this paper proposes VAD-DPO to train models on visually similar videos with contrasting anomaly labels, encouraging alignment with semantic consistency rather than co-occurrence shortcuts. Experiments show the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper reveals the shortcut phenomenon of existing VAD evaluation and conducts an empirical analysis. This is an important finding for the VAD research community.\n2. The evaluation is comprehensively conducted on six different datasets.\n3. This paper is well-written, well-organized, and easy to follow.\n\nWeaknesses:\n1. The connection between the revealed shortcut phenomenon and the proposed VAD-DPO method is not strong; it feels more like two separate contributions. How DPO addresses the shortcut issue should be better demonstrated.\n2. The proposed method considers only the naive DPO algorithm. More advanced approaches, such as GRPO [r1], or a newly proposed VAD-specific algorithm, could be explored.\n3. The proposed method is only evaluated on the Qwen2-VL backbone, which is insufficient. At least one additional backbone, such as InternVL, should be considered.\n4. Figure 4 provides a good qualitative illustration. However, a quantitative comparison is needed to demonstrate VAD-DPO's advantage in hallucination cases.\n\nJustification:\n\nThis paper identifies the shortcut phenomenon in existing VAD evaluations, which is a meaningful contribution. However, several concerns remain. I suggest a Borderline Accept at this stage.\n\n[r1] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).\n\nQUESTIONS:\nIn Table 6, what is the difference between DPO and VAD-DPO?\n\nLIMITATIONS:\nSee weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}