Local Patterns Generalize Better for Novel Anomalies

Authors: Yalong Jiang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Global Patterns; Local Patterns; Image-Text Alignment Module; Cross-Modality Attention; Temporal Sentence Generation; State Machine Module

Abstract:
Video anomaly detection (VAD) aims to identify novel actions or events which are unseen during training. Existing mainstream VAD techniques typically focus on the global patterns with redundant details and struggle to generalize to unseen samples. In this paper, we propose a framework that identifies the local patterns which generalize to novel samples and models the dynamics of local patterns. The capability of extracting spatial local patterns is achieved through a two-stage process involving image-text alignment and cross-modality attention. Generalizable representations are built by focusing on semantically relevant components which can be recombined to capture the essence of novel anomalies, reducing unnecessary visual data variances. To enhance local patterns with temporal clues, we propose a State Machine Module (SMM) that utilizes earlier high-resolution textual tokens to guide the generation of precise captions for subsequent low-resolution observations. Furthermore, temporal motion estimation complements spatial local patterns to detect anomalies characterized by novel spatial distributions or distinctive dynamics. Extensive experiments on popular benchmark datasets demonstrate the achievement of state-of-the-art performance. Code is available at https://github.com/AllenYLJiang/Local-Patterns-Generalize-Better/.

Decision: accept (poster)
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729004233139

thread_type:
official_review

<thread_object>
creator: reviewer_mbj7

<content>
summary:
This article is about video anomaly detection (VAD). This paper proposes a framework for recognizing local patterns, which can be generalized to new samples and dynamic modeling of local patterns. This paper proposes image-text alignment and cross-modal attention. Generalizable representations are built by focusing on textual information features that filter out unnecessary differences in visual data. In addition, time motion estimation complements spatial local models to detect anomalies characterized by new spatial distributions or unique dynamics. A large number of experiments have verified the effectiveness.

strengths:
1. The two-stage training method of this paper is reasonable. And allow for more fine-grained local features.
2. This paper gives a lot of visualizations to make it easier to understand the specific content.
3. The paper has achieved good performance, and the ablation experiment is given.

weaknesses:
1. Not giving motivation for each part of the method. In my opinion, a good paper should give a specific reason and then introduce the method.
2. The efficiency of the model is worth discussing. You have proposed a lot of model modules. How much more reasoning time will they add to the network?

questions:
My understanding of this field is unprofessional. So I will further follow the opinions of other reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730253526222

thread_type:
official_review

<thread_object>
creator: reviewer_qaru

<content>
summary:
This paper introduces a novel framework for video anomaly detection (VAD) that prioritizes identifying local patterns over conventional global patterns. The authors contend that local patterns generalize better to novel anomalies that were not encountered during training. Their proposed approach follows a two-stage process involving image-text alignment and cross-modality attention to efficiently capture and model local patterns. Additionally, the framework includes a State Machine Module (SMM) to integrate temporal dynamics, enabling enhanced anomaly detection by leveraging both spatial and temporal cues. Experimental results show that this approach achieves state-of-the-art performance on well-established benchmark datasets.

strengths:
1. The proposed framework is well-structured, with thoughtfully implemented methods.  
2. Experimental results confirm that the approach achieves state-of-the-art performance on established benchmark datasets for video anomaly detection.  
3. The method focuses on fine-grained anomaly features and employs text-image alignment to effectively capture local patterns.  
4. It incorporates Long-range Memory technology, specifically HiPPO, into video anomaly detection.

weaknesses:
1. The core of the proposed SMM module comes from works like HiPPO, so it seems that the proposed SMM is directly applying these modules to the VAD task. 
2. Both the Image-Text Alignment Module and Cross-Modality Attention Module are based on pre-existing techniques, which limits the methodological innovation.
3. Is the observed performance improvement attributed to the additional large vision-language models, such as Qwen-VL and BLIP2? The comparison may not be entirely fair. It would be beneficial if the authors could provide evidence or experimental results to clarify whether these powerful external models are the primary contributors to the performance gains.
4. The motivation for the work lacks clarity. How do the image-text alignment and cross-modality attention modules achieve “identification of local patterns that are consistent across domains and generalize well”? Additionally, how do they contribute to “generalizing model representations to novel anomalies”?
5. Certain claims may require further validation, such as the statement: “the complementary relation between visual and textual features remains underexplored.”
6. The paper lacks runtime and efficiency analysis. The code introduction is incomplete, and several experimental details are missing, such as the specific version and scale of Qwen-VL used.

questions:
1. The Global and Local Pattern representations in Figure 1 are hand-drawn, which limits their reliability. Are there any real feature visualization images available instead? Using actual visualizations could better illustrate the motivation and effectiveness of the proposed method, particularly in showing whether it yields more distinguishable local patterns. Figure 1 alone does not provide enough information to convey the method’s motivation and impact.
2. Utilizing Qwen for cropping bounding box regions based on prompts could significantly impact efficiency.
3. Is the introduction of the Qwen-Chat model the primary source of performance improvement? My concern is that the proposed method incorporates numerous external models, and it remains unclear whether these additions are the main contributors to the observed performance gains.
4. Could smaller models be used to replace these large multimodal models? If so, would this result in a significant decrease in performance?
5. Could you provide statistical results on runtime and efficiency? Does the proposed method have a significant impact on operational efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730292906779

thread_type:
official_review

<thread_object>
creator: reviewer_xfdv

<content>
summary:
This paper proposes a novel framework for video anomaly detection (VAD) that focuses on identifying local patterns to better generalize to unseen anomalies.  The framework employs a two-stage process: first, it uses image-text alignment to locate local patterns that are consistent across visual data variances; second, it refines these patterns using cross-modality attention.  To further enhance the model, the authors introduce temporal clues through a State Machine Module (SMM) and temporal motion estimation.

strengths:
The proposed two-stage framework for identifying local patterns is novel and well-motivated. The use of image-text alignment and cross-modality attention is interesting and potentially useful.

weaknesses:
1) The paper lacks a clear discussion of the computational complexity of the proposed framework. Given the use of large language models (LLMs) and other complex modules, it is important to address the efficiency of the approach.
2) What's the role of State Machine Module (SMM) in temporal sentence generation, there need more detailed explanation of the SMM and its role.
3) How does the proposed method handle situations with significant occlusions or viewpoint changes, which are common in real-world surveillance videos?
4) The two-stage process for extracting spatial local patterns using image-text alignment and cross-modality attention is not explained in enough detail. The paper lacks a clear, step-by-step explanation of how these complex processes work.

questions:
The paper mentions limitations related to the reliance on VLM-based object detectors. How can this limitation be addressed in future work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645028563

thread_type:
official_review

<thread_object>
creator: reviewer_yam7

<content>
summary:
The paper introduces a novel framework for video anomaly detection, aiming to improve generalization for detecting new, unseen anomalies by focusing on local patterns rather than global event patterns. Traditional video anomaly detection (VAD) methods often struggle with unseen anomalies, as they primarily analyze global patterns. This framework utilizes image-text alignment and cross-modality attention to identify and refine local patterns while enhancing them with temporal information. Core components include the Image-Text Alignment Module (ITAM), Cross-Modality Attention Module (CMAM), and State Machine Module (SMM). The proposed approach demonstrates superior performance on several benchmark datasets, suggesting it can generalize better to novel anomalies.

strengths:
* By using image-text alignment and cross-modality attention, this method successfully extracts local patterns that remain consistent across varying visual data, enhancing its ability to detect novel anomalies.

* The State Machine Module (SMM) and motion estimation integrate temporal clues, effectively strengthening the detection capabilities by including sequential information for more accurate anomaly detection.

*By combining visual and textual features in identifying local patterns, the model benefits from enhanced robustness and accuracy across different visual domains.

weaknesses:
* The method relies on the detection effect of visual-linguistic modeling (VLM), whereas multi-object image processing may ignore contextual information and affect performance. The authors need to provide more analysis on the ablation of the foundational models.

* The need for multiple layers of modules (e.g., ITAM, CMAM, SMM) to work jointly results in a complex training process that consumes more time and resources. Please provide a comparison of the spatio-temporal complexity analysis with previous methods to demonstrate the practical effectiveness of the method.

* In low-resolution scenes, the generated text description loses detail information, which affects the anomaly detection effect.

questions:
* How to further improve the generalization of local patterns without relying on visual-linguistic models?

* How does the method ensure adaptability to low-resolution videos in different datasets and real-world application scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732541327706

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Explanations and revisions

comment:
Dear Reviewers, thank you for your initial feedback on our submission. We have addressed your comments and provided detailed responses in our rebuttal. Please let us know if there are any additional points you’d like us to clarify before the discussion phase concludes. Your feedback is highly appreciated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732647089482

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Further explanations and revisions

comment:
Dear everyone, thank you for your initial feedback on our submission. We have further addressed the comments and refined the responses in our rebuttal, also we have uploaded the revised paper & code. Please let us know if there are any additional points you’d like us to clarify before the discussion phase concludes. Your feedback is highly appreciated.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733056037395

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Clarification on concerns

comment:
Dear Reviewers and Area Chairs,

I would like to express my gratitude for your time and feedback. We have carefully addressed the concerns raised by the reviewers in our previous responses. 

In terms of novely, the structure of the proposed SMM significantly differs from HiPPO. We have provided the code in our submission, which clearly demonstrates the differences between our approach and the referenced work.  Besides, we have innovatively discovered the local patterns that can be recombined to capture the essence of novel anomalies, this is implemented by the two-stage process with Image-Text Alignment and Cross-Modality Attention to gradually identifying the generalizable local patterns. Ablation studies show that our method also works with smaller models and do not necessarily rely on pre-existing large models. 

In terms of efficiency concerns, we have provided spatio-temporal complexity analysis and comparisons with previous methods, and we have two choices of object detectors one of which has a much higher efficiency than VLM. The efficiency of the approach exceeds baseline LLM-based pproach. 

We have clarified the motivations and implementations of image-text alignment, cross-modality attention and temporal sentence generation with stacked state machines. 

In cases of low-resolution scenes, occlusions or viewpoint changes, our proposed local patterns capture semantically meaningful features such as body joints which are consistent across the variations. The proposed SMM determines whether earlier high-resolution events are represented by later image tokens, it leverages inter-frame dependencies to deal with variations. In the concerns about the ignorance of contextual information, we have actually expanded bounding boxes and facilitating the involvement of relative contexts in analyzing subjects, benefiting performance. 

We would greatly appreciate any further feedback or clarification requests from the reviewers. If there are any additional questions, we are happy to provide further details.

Thank you once again for your attention and consideration.

Best regards
</content>
</thread_object>

</thread>


Metareview:
metareview:
# Summary and Recommendation for Acceptance

---

## **Strengths**
1. **Novel Contributions**:
   - Proposes a novel framework for video anomaly detection (VAD) focusing on **local patterns** rather than global patterns, which improves generalization to unseen anomalies.
   - Introduces a **two-stage process**:
     - **ITAM**: Captures semantically meaningful local components.
     - **CMAM**: Refines local patterns using cross-modality fusion.
   - Enhances temporal reasoning with a **State Machine Module (SMM)**, addressing low-resolution scenarios by leveraging inter-frame dependencies.

2. **Robust Experimental Validation**:
   - Achieves state-of-the-art performance on well-established benchmark datasets.
   - Ablation studies demonstrate the effectiveness of individual components, including ITAM, CMAM, and SMM.

3. **Generalization Capabilities**:
   - Local patterns remain robust across varying visual conditions (e.g., low resolution, occlusion, and viewpoint changes).
   - The model recombines known components (e.g., arms, legs) to describe unseen actions without explicitly naming them.

4. **Efficiency**:
   - Offers competitive spatio-temporal complexity, achieving an average frame rate of 12 FPS, outperforming baseline methods in both speed and accuracy.

5. **Community Contribution**:
   - Code and detailed manuals provided to ensure reproducibility and ease of adoption.

---

## **Weaknesses**
1. **Complexity**:
   - The framework includes multiple components (ITAM, CMAM, SMM) that require careful integration, making training and implementation challenging.

2. **Heavy Reliance on Pre-trained Models**:
   - Use of vision-language models (e.g., Qwen-VL-7B) raises concerns about whether performance improvements are mainly due to these external models.

3. **Clarity and Presentation**:
   - Initial version lacked clear motivation and step-by-step explanations for components like ITAM and CMAM.
   - Fig. 1 in the initial submission was hand-drawn, reducing its effectiveness in illustrating local and global pattern distinctions.

4. **Efficiency Analysis**:
   - Earlier versions did not provide detailed runtime and efficiency analyses for individual modules.

---

## **Authors' Mitigation**
1. **Clarity and Motivation**:
   - Revised the manuscript to provide step-by-step explanations for ITAM, CMAM, and SMM.
   - Updated Fig. 1 with real feature heatmaps to visually distinguish local and global patterns.
   - Added Algorithm 1 in Section 3.2 to detail the workflow of the two-stage process.

2. **Ablation Studies**:
   - Conducted extensive ablation studies to demonstrate that ITAM and CMAM can be trained with smaller models and perform well without pre-trained models.
   - Validated the benefits of cross-modality fusion in CMAM and the necessity of SMM for temporal coherence.

3. **Efficiency Analysis**:
   - Added runtime analyses in Appendix H, demonstrating competitive performance with an average frame rate of 12 FPS using YOLO-v7 for object detection.

4. **Generalization Improvements**:
   - Highlighted how local patterns generalize better across domains by capturing semantically meaningful components like body joints.
   - Addressed low-resolution scenarios by refining captions with SMM, ensuring consistent performance.

5. **Framework Simplification**:
   - Demonstrated that smaller models (e.g., YOLO-v7) achieve similar performance to larger models (e.g., Qwen-VL-7B), reducing dependency on heavy pre-trained models.

---

## **Remaining Weaknesses**
1. **Framework Complexity**:
   - Despite clarifications, the overall framework remains complex, which may deter practitioners unfamiliar with advanced multimodal techniques.

2. **Reliance on Vision-Language Models**:
   - Although the authors addressed concerns through ablation studies, the use of pre-trained models for generating training labels introduces reliance that could be reduced further.

3. **Broader Applicability**:
   - The framework focuses on benchmark datasets but lacks validation in diverse real-world scenarios, such as highly cluttered environments or surveillance with significant occlusions.

---

## **Justification for Acceptance**
This paper makes a significant contribution to video anomaly detection by introducing a novel approach that prioritizes local patterns, which are more generalizable to novel anomalies. The combination of ITAM, CMAM, and SMM provides a robust framework that effectively integrates spatial and temporal reasoning. The authors' revisions address most reviewer concerns, improving clarity, reducing reliance on pre-trained models, and demonstrating the efficiency of their approach.

While some complexity remains, the paper's methodological innovation, strong experimental results, and community contributions (e.g., open-source code) outweigh these limitations. This work has the potential to advance the field of anomaly detection and inspire further research. I recommend acceptance.