Local Patterns Generalize Better for Novel Anomalies
Authors: Yalong Jiang
Venue: ICLR 2025 Poster (2025)
Keywords: Global Patterns; Local Patterns; Image-Text Alignment Module; Cross-Modality Attention; Temporal Sentence Generation; State Machine Module
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Video anomaly detection (VAD) aims to identify novel actions or events which are unseen during training. Existing mainstream VAD techniques typically focus on the global patterns with redundant details and struggle to generalize to unseen samples. In this paper, we propose a framework that identifies the local patterns which generalize to novel samples and models the dynamics of local patterns. The capability of extracting spatial local patterns is achieved through a two-stage process involving image-text alignment and cross-modality attention. Generalizable representations are built by focusing on semantically relevant components which can be recombined to capture the essence of novel anomalies, reducing unnecessary visual data variances. To enhance local patterns with temporal clues, we propose a State Machine Module (SMM) that utilizes earlier high-resolution textual tokens to guide the generation of precise captions for subsequent low-resolution observations. Furthermore, temporal motion estimation complements spatial local patterns to detect anomalies characterized by novel spatial distributions or distinctive dynamics. Extensive experiments on popular benchmark datasets demonstrate the achievement of state-of-the-art performance. Code is available at https://github.com/AllenYLJiang/Local-Patterns-Generalize-Better/.

Published as a conference paper at ICLR 2025
LOCAL PATTERNS GENERALIZE BETTER FOR NOVEL ANOMALIES
Yalong Jiang Beihang University
A1 AllenIIJi annni.iaag@ouut. www.ootttoooo.....oooooooo...... com
ABSTRACT
Video anomaly detection (VAD) aims to identify novel actions or events which are unseen during training. Existing mainstream VAD techniques typically fo- cus on the global patterns with redundant details and struggle to generalize to unseen samples. In this paper, we propose a framework that identifies the lo- cal patterns which generalize to novel samples and models the dynamics of local patterns. The capability extracting spatial local patterns is achieved through two-stage process involving image-text alignment and cross-modality atten- tion. Generalizable representations are built by focusing on semantically relevant components which can be recombined capture the essence of novel anoma- lies, reducing unnecessary visual data variances. To enhance local patterns with temporal clues, propose State Machine Module (SMM) that utilizes ear- lier high-resolution textual tokens to guide the generation of precise captions for subsequent low-resolution observations. Furthermore, temporal motion esti- mation complements spatial local patterns to detect anomalies characterized by novel spatial distributions or distinctive dynamics. Extensive experiments on popular benchmark datasets demonstrate the achievement of state-of-the-an per- formance. Code available
com/ Alt.AellleLLaang//////oa
-Patters Geeeeeeeerrrraa -ettiiizeeeerrrrre
INTRODUCTION
Video anomaly detection (VAD) is the task of localizing from videos the events that deviate from regular patterns, such as violence, accidents and other unexpected events. Nowadays, numerous platforms such as CCTVs and UAVs play an increasingly important role in surveillance. However, given the vast volume of video data and the low probability of anomalies, it impractical for humans to manually detect these events. Additionally, visual data variances and domain differences between normal and anomalous events hinder the effectiveness of detection methods. As result, VAD has become significant research topic in weakly supervised
unsupervised learning Gong et al. (2019) : Shi et al. (2023b): Chalapathy et al. (2017); :): Lu et al. (2020); Pang et al. (2020): Lv et al.
2021
Georgescu al. (2021a) Zaheer al. (2020)) Ristea
(2021) Acsintoae al. (021)
Existing main-stream works Li et al. 2022cc Luo et al. (021a): Georgescu et al. (202aa) for VAD divided into four categories. The first category of methods detects anomalies by leveraging distinctive spatial and temporal features. These methods include prediction-basee ones [uo et al.
(2021a); Lv et al. (2021) Lu al. (2020); Park al. (2020) and reconstruction-basee ones Yang
et al. (2023b); Lv et al. (2023); Chang et al. (2020): Liu et al. (202T) To enhance representational
capacity, some methods combine multi multi-grained spatio-temporal representations Zhang et al. (0024)
for better discrimination, or integrate various features Georgescu et al.) (2021a): [Cho et al. 2023
better align with unseen samples Liu et al. (2022b). The second category involves using Mul- tiple Instance Learning (MIL) to iteratively identify useful data segments and fine-tune models for anomaly detection Cho et al. (2023); | Waan et al. (2022a); Li et al. (2022a); Zhu et al. (0022): Liu
al. (2023c). For instance, dynamic clustering techniques adapt model representations to real-time observations Wu et al. (202)); Yang et al. (002). Prompt-enhanced MIL Chen et al. (2024) in- tegrates semantic priors with visual features for improved modeling of anomalies. However, the generalization ability is still insufficient because background noises lead to inconsistent represen-
tations over visual data variances, is shown by Fig. 1. The third category Liu et al. (2023c)

---

Published as a conference paper at ICLR 2025
parterzs: generalizable
details
Redinddnt detaily
potterms
image data
Nonual sample
Norual somple N
imsge-test
Local polterss Generalizsble
I Local pallerns
Semantic focus
Semantie Rocus
TRIDES
Similar bodly componcots
local pattees
Caplion: walking the garden.
Soand
Normal
Abnormal
Figure I: Top: Existing methods rely on global patterns with redundant details, which are inconsis- tent across visual data variations, limiting their generalization to novel samples. As a result, normal and abnormal samples are poorly distinguished. Bottom: Our method focuses on local patterns that capture semantically meaningful features such body joints and consistent across domains and generalize well. The spatial distributions of these local patterns highlight divergences.
focuses on generating realistic anomalies to refine the decision boundary between normal and ab- normal samples. Prompt based approaches Wu et (2024a) have also been proposed for generating pesudo anomalies. However, the generated anomalies are based on prior assumptions which cannot cover diverse and unexpected anomalous samples in real-works cases. The fourth category Zanella et al. (2024) leverages the visual-language knowledge and reasoning capabilities Yang et al. (2024a)
from large models to generate textual descriptions or produce pseudo labels for self-training Yang et al. 2024b), thereby improving the discrimination of abnormal events Micorek et al. (2024).
To generalize model representations to novel anomalies, we propose a two-stage framework for identifying local patterns. In Stage I, image-text alignment is used to locate text-informative local patterns that are consistent across visual data variances. Stage further refines the local patterns us- ing cross moss-ttydllt attention, resulting in more compact local patterns. Finally, spatial local patterns are augmented with temporal clues to better determine anomalies.
In sum, the proposed framework is composed of an Image-Text Text Alignment Module (ITAM) and a Cross-Modality Attention Module (CMAM) for identifying local patterns in two stages. ITAM selects text-informative regions, converting high-dimensional visual data into efficient image kens. The tokens are converted by Temporal Sentence Generation Module (TSGM) into texts, which CMAM uses to refine the selection of image tokens as local patterns. Temporal clues enhance local patterns in two ways. TSGM generates the sentences for cross-modality attention by considering multi-moment contexts, while temporal motion estimation enriches spatial local patterns with tem- poral dynamics. The effectiveness is validated on multiple benchmarks, including ShanghaiTechh Ubnormal and so on. The contributions can be highlighted as follows:
This paper proposes novel two-stage approach to identify the local patterns that are consistent across visual data variances and generalize to novel abnormal samples. The first stage uses image- text alignment to identify semantically meaningful components, facilitating generalizable represen- tations. Cross-modallns attention further refines the components, yielding both the benefits of texts in generalization and the advantages of visual features in representing details.
Temporal solutions are used to enhance spatial local patterns. Firstly, temporal sentence gener- ation integrates the contexts from different moments produce coherent descriptions of events. Additionally, temporal motion estimation complements local patterns by modeling dynamics.
State-of-the-art performance is achieved with the proposed framework multiple benchmarks.
RELATED WORKS
2.1
UNSUPERVISED VIDEO ANOMALY DETECTION
Due to the unbalanced nature of surveillance videos, most training datasets are without anomaly annotations because it is expensive label Li al. 2022b); Liu et (2023b): Deng et al. (2023
2

---

Published as a conference paper at ICLR 2025
Reconstruction-baseee approaches Astrid al. (2024); Yang et al. (003bb: Fang al. (2020); Li
ct al. (2020a); [oog et al. (2019) Asad et al. [2021]; Abati et al. (2019): Sabokrou et al. (201 8)
produce increased error when encountering irregular features Ramachandra et al. (2020): Madan
et al.
2023]; Yu et al. (002)) that do not reside in training data. For instance, the method Zaheer
al. 2022a) learns not to reconstruct anomalies. [Gong et al. (2019); Gao al. (2022) augment
encoders to improve the sensitivity of reconstruction error to anomalies. Madan et al. (2021); [
al. (2020); Singh et al. (2023); Yu et al. [ (2022b); (hi et al. (202a) integrate multi-modal features
Ding et al. (2021) while Huang et al. (2022) integrates a probabilistic decision model. Zaheer
al. (2022b) assesses the quality of reconstruction to improve stability. Prediction-base methods
Luo et al. (202Ib): Morais et al. (2019); Luo et al. (2021a): Liu et al. (2018); Nguyen & Meunier
(2019); Zeng et al. 2021) evaluate the divergence in normal and abnormal temporal dependencies, leveraging latent spaces /Zhang et al. (2020) or hybrid attention /Zhang et al. (2022b).
To better distinguish anomalies, [Lv et al. (2021): ): Lu et al. (2020); al. (2021) y Park et al. (2020);
Li et al. (2221a) combine prediction with reconstruction. Sato al. (2023); Wu et al. (2023) Luo
ct al. (2019) study the distribution over normal samples and propose novel Teatures Arad & Werman
(2023). Similarly, Yan et al. (2023) proposes denoising diffusion modules. FFlaborea al. (2023)
exploits the enhanced mode coverage diffusive probabilistic models. To improve representation
capacities, Chang et al. [202T]: Fan et al. (2024) propose snippet-level attention. Liu al. (203a));
Yu et al.) (2022)) Purwanto et al. (2021) introduce pyramid deformation and CRFs to learn spatio- temporal dependencies
BerrtassiiesBBraasuuusu Bertasius al. (2021) y Cho al. (2022). Waan et al. (202T) combines multi-
scale features to enhance prediction. Stergiou et al. (2024) combines interpolation with extrapolation
for prediction. WWang et al. (20226) proposes self-supervised scheme with discriminative DNNs.
We propose generalizable local patterns to better represent unseen samples.
2.2 WEAKLY SUPERVISED ANOMALY DETECTION
Multi-instance learning (MIL) takes videos as bags and snippets as instances, transforming video- level labels to instance-levell Feng al. (202T). The methods iteratively locate abnormal segments and fine-tune models using anomalous segments which are dissimilar to normal ones /Zhang et al.
(002a To collect abnormal segments, inter-sample distances are evaluated Lu et al. (2022);
Tonescu et al. (2019) based on spatio-temporal similarities Dhiman & Vishwakarma (020)) Lv ct al. (2023); Chang et al. (2020); Markovitz et al. (2020). Li et al. (2021b) proposes a probabilistic framework. Sun et al. (2020) Li et al. (2020b) build graphical representations and integrated col-
lective properties in measuring similarities. Saakota & Yu (2022) performs dynamic non- conpaaamettiic clustering. To improve robustness, [Zhng al. (203b) proposes to interpret the vulnerability of MIL.. Wu & Liu (202T) introduces causal relations to enhance MIL. Tian et al.] (202T). Yang et al.)
2023a) proposes binary network augmentation strategy. Differently, we propose generalizable rep- resentations which facilitate the measurement of similarities between seen and unseen events.
2.3 METHOSS WITH DATA AUGMENTATION
To generate pseudo abnormal samples in fine-tuning, Liu et al. (2023c); Lin et al. (2022); Kim
al. (2022); Liu al. (2022a); Astrid al. (202T) propose pseudo abnormal snippet synthesizers which are trained on normal samples Yu et al. (2021). Zaheer et al. (2020a) employs generator which was not fully trained create abnormal samples. Chen et al. (202)) generates class balanced
training data with a conditional GAN, [im et al. (2018) focuses on infrequent normal samples during generation, harnessing novel sampling strategies. Besides frame-level analysis Zaheer et I. (000b)) object-level approaches Sun & Gong (2023); [Inescu et al. (2019); Luo et al. (202la) provide fine- grained analysis. Acsintoae et al. (2022) introduces a new dataset with diverse anomalies. However, the lack in real-world modes in generated data highlights the necessity for generalizable patterns.
2.4 METHODS EXPLORING THE REPRESENTATION OF UNSEEN CATEGORIES
To adapt model representations and work under changing anomalies, meta learning-basee methods Lu et al.] (2020) Park et al. (2020), transfer-learning based approaches Doshi & Yilmaz (2020);
Perini et al. (2022), continual learning Doshi & Yilmaz (2020) and self-supervised approaches Pang
et al. (2020): Degardin & Proença (2021) introduce adaptable feature representations. Attention-
based methods Sultani et al. (2018): Guoo et al. (2023): Li et al. (202cc): Luo al. (2017) attend
3

---

Published as a conference paper at ICLR 2025
Matn Ploe
Impot Video
Stage
Integg
(+(())
FG) (10) 6l Fc
Becklonr
FTAM
Staps
Frame
Cress Preul-aaialityy Attention Medale (CMAM)
Generated
Tempural Seateore Geperation
Tempural Matien Extimation
Frmme
sastuces
"4
Module
Fii
HEYO Excader
G
Local Paitteres
nosa vinn
fiing le
Text lokers st.
HESC
Text tokennier
Hester
Recuustraction Madidl (ItM)
Encede finances
lndo video
Anouaal score
Module
Emage tokens s.6.
sokene
(C(D)))
FullFayyyyooo ColleCClee
Eale-i)
ITAM
(r-11)
E( (1000-00---)0010111 (i (t,
Frimt 1-]
Weighte france
Weights trained
Tempural Senteace Gencration Medale (TSGM)
Crse-Mnddality Aftration Medal (CMAM)
Combined Caption (())= "A walking, patting stelle the steat.
lased
observation wilking
SMM: Craratination
Test Takuns
Image Tokens
Image-Grandde Test Generation
Text Tokenirer
ITAM
Question: the man postines aroler or stre??
G((1)
(High rovoutioon)
stalle
resolutioo)
(b)
(c)
Figure 2: Structure of the method. (a) Main flow: Visual features are extracted using backbone and Image-Text Alignment Module (ITAM) which identifies innttonnnffrrmatve local features image tokens. Conditioned on image tokens, the Temporal Sentence Generation Module (TSGM) generates sentences which are then combined with image tokens using the Cross-Modality Attention Module (CMAM) to highlight key patterns. HEVC Encoder estimates inter-frame motion through video compression. Spatial local patterns and motion jointly analyzed to detect anomalies through reconstruction. (b) TSGM uses the State Machine Module (SMM) for sentence generation based on image tokens and earlier sentences. (c) CMAM implemented based on image-text similarity.
to domain-invariane features in addressing unseen samples. To better align with anomaly detection, Georgescu et al. (22 integrates multiple sub-tasks. Zhou et al. (2023a) introduces hierarchical graphs for representing videos and maximizing inter-class margins. Differently, approach locates the int-infformatie local patterns which generalize to unseen events.
2.5 PROMPTING METHODS
Prompt-based approaches have been widely used in anomaly detection Du al. (022)) Liu et al. Instance,Zhhuu
(2023c); [Sato et al. (2023). For instance, /hhou et al. (2023b) learns object-agnostic text prompts for generalized abnormality recognition. Yang et al. (2024a) proposes rule-based reasoning achieve few-normal-shot prompting. Unlike approaches that use direct prompts, explore local patterns which bridge the gap between images and texts in Visual-Language Models (VI.Ms).
METHODOLOGY
To represent unexpected anomalies using generalizable representations, we establish framework capable of identifying to localize spatial local patterns in two stages, as is shown Fig. 2(a). To augment local patterns with temporal clues, temporal sentence generation and temporal motion estimation are investigated. Firstly, TSGM models the dependencies between earlier text tokens and later image tokens, enhanc- ing the input sentence for CMAM, is shown in Fig. 2(b). Then inter-frame motion vectors obtained from vidco compression. Finally, spatial and temporal clues are combined in the Recon- struction Module (RM) detect anomalies. In the following parts will discuss each module.
local patterns. The framework uses ITAM and CMAM
4

---

Published as a conference paper at ICLR 2025
3.1 CROPPING OF IMAGE REGIONS
Due to the wide field of view in some frames containing numerous objects, it is difficult even for GPT-4 Achiam et al. (2023) to focus all objects together. As result, local regions are cropped as the first step in our pipeline. We have experimented with both YOLOv7 Wang et al.] (2023) and Qwen-7B Bai et al. (2023) for cropping bounding box regions based on prompts. Specifically, "How many people are there?" and "The bounding box of the i-th object" are sequentially provided to Qwen-7B which returns corresponding boxes. The comparisons will be included in Appendix D.
3.2 STAGE 1 FOR IDENTIFYING SPATIAL LOCAL PATTERNS
This stage identifies features in cropped image regions that align with texts. The texts describe generic movement attributes (e.g.. "A man is walking with swinging arms and legs"). When en- countering an unseen action, such as running, the model can recombine known components like arms and legs to generate descriptive language that captures the essence of the action without ex- plicitly naming it. As illustrated in Fig. 1, heatmaps indicate the attention on local components, highlighting similar semantic regions for shared attributes. More visualizations are in Fig. 4.
To identify the local patterns that align with texts, the frozen Image Encoder al. (2023) and the image transformer of Q-Former in BLIP-2 Li et al. (2023) are employed as backbone and ITAM, respectively. The backbone outputs H (t) goaxva Q-Former has an image transformer and text transformer for aligning features from both modalities, () E RN is the image transformer's output with N₂ image tokens F (t), .... Fl.N. (t) which inform about the captions of image region i and remain consistent
visual data variances, as will be shown by the heatmaps in Fig. 4. Detailed structures are in Appendix C. Algorithm shows the workflow of Stage and Stage 2.
Algorithm Two-Stage Process for Identifying Spatial Local Patterns
Input: Input image, Backbone, ITAM, CMAM. TSGM and Text tokenizer
2: Output: Cross-modal embedding G (t) representing spatial local patterns
3: Stage Image Token Extraction
4: Use the backbone to extract feature maps H! (t)
5: Feed H (t) into ITAM obtain image tokens (t)) which align with texts
6: Stage Cross-Modality Attention
7: Feed image tokens (FL,(0) into TSGM and obtain text tokens FT(1)
8:
CMAM weightedly sums (0) according to their similarity with FT(1) (1)
9: Return: Return weighted sum G(((), representing the cross-modal features
3.3 STAGE 2 FOR IDENTIFYING LOCAL PATTERNS
This stage further highlights local patterns by generating sentence conditioned image tokens and summing them based on their similarities to the generated sentence. Using the image tokens (t), .... F..N. (t) from Stage 1, TSGM generates a sentence for image region as is shown in Fig. 2(a). TSGM utilizes SMM for inter- frame caption augmentation and a frozen Q-Former Li et al. (023) for mnage-grounded text generation. SMM determines whether previous events still reside in current frame while Q-Former captions current frame. The outputs from SMM and Q-Former combined to form the augmented sentence S (t). Even with incomplete observations at t, (t) can recognize previously occurring events from current frame as long as the events still reside.
The embedding F (t) E S of S (t), where S 32 denotes the maximum number of tokens in one sentence, is provided to CMAM. CMAM uses the first element in F (t) as query and the image tokens keys and values for attention operations, as is illustrated in Fig. 2(c) and Eq. (1):
G((+) (F7 (+) (f) G( e "
(t) is obtained by the text transformer Q-Former Li et al. (203) with first element F (+) representing the whole sentence. Eq. (1) weightedly sums image tokens according to their cosine
5

---

Published as a conference paper at ICLR 2025
State Machine Module (SMM)
Input wf L (11)-,
Concetiantic mulltmodal tokens
Fl,
Updated coefliiimt (f)
Text Inkeen "Is the persin walking?"
Image tnkens
Coefficients
c((/1))
G (((--+++0)
a. (f-L-1++1)
Shifted basis Functionss
Output G.U-L+1+0
Y.e+)
Feature
State Machine
6,(+5)
State
a
Festure Space
"yes"
Machine Naorr
Enooder
A Naw
"no"
Select compencnis
6.(+-5)
Figure 3: Structure of SMM, which stacks NSMM state machines, each predicting ahead.
similarity to (t)... this way, the advantage of image tokens in characterizing visual details and the benefits of textual features in generalizing over visual data variances are both achieved. Ablation studies will compare the performance of G! against features from single modalities.
3.4 TEMPORAL SENTENCE GENERATION IN STAGE 2
In Stage 2, the generation of captions is influenced by visual data variances such as low resolutions. As is shown in Fig. 2(b), the module Li et al. (2023) for image-grounded text generation only pro- vides a coarse caption "A man is walking" on later low-resolution observations. It is not as precise as earlier caption "A "A man is pushing a stroller on the street" even if they actually describe the same event. Therefore, SMM in TSGM determines whether earlier high-resolution events are represented by later image tokens. It captures inter-frame dependencies and refines sentence coherence. SMM uses earlier text tokens to generate precise captions for low-resolution observations. The objects in consecutive frames are associated using intersection over union similarity between bounding boxes.
Specifically, SMM augments image tokens (1) = (F), F Fi,,, (t)} with earlier captions
( 1) which are firstly converted from declarative sentences interrogative sentences. For instance. "The man is pushing a stroller." is changed to "Is the man pushing a stroller?" whose text tokens are F (t (f--11 1) = [F,, 1), ..., FY,s,
1)}. Details of this conversion will be shown in Appendix G Hardeniya et al. (2016). As is shown in Fig. 2(b), SMM combines F. (t 1) with F (1) as input. The state machines in SMM evolve across the dimension of input sequence (t(t) = [t - 1); ...; FT,s, (t - 1); (t); Fi,,, (t)] E (S. where L = S + Ng is the sequence length and each token has dimension Ha. SMM predicts a binary decision ("yes" or "no"") based on the sequence, determining whether the event in S/ 1) is still present in F (t).
Vi(t) is deemed as the combination of Ha 1-dimensional signals each with length L. The depen- dencies in sequences (....))...,g o E O], as will be shown in Fig. of Appendix A. The input tensor (t(t) is approximated by the weighted sums of the O fixed polynomials. For simplicity, index t omitted in the following parts which conduct analysis along the column dimension of input tensor at any moment 6. The Feature Space Encoder produces U,(() [u L + ...; (,)))]
represented using O length-T. Legendre polynomials Arfken al. (2011)
B
where u,())) = [u4.1 (('), .... o4,0(("))))) ((')) for l' E (I L. Here, I varies along the column dimension of Vt()) and U,( (1 L, the window of columns which are encoded by cl) together.
To better model multi-modal sequence of Hl-dimensional signals, NSMM state machines are stacked in SMM, each predicting Al/NsMM ahead, as shown in Fig. 3. The advantages will be shown in ablation studies. Eq. (2) shows the representation of U²(t) with basis functions:
u4oo (1') c..0 (1.6 1+ L), E [1,0), (1-L,))
(2)
In SMM, a state vector c1(1) = (c).:)))).cc with O weights encoding the dependencies between texts and visual tokens Vi(.), the dependencies are decomposed onto weighted Legendre basis functions. State vector evolution informs about the prediction ("yes" or "no").
c/(1 + (c, + Σ
4l + 11)
(3)

---

Published as a conference paper at ICLR 2025
where A = L) Roxo and B L) R are derived from Legendre polynomials
Gu et al. (202)) As grows, more diversified basis functions can represent more complex depen- dencies. Assume that (!) encodes u,(l L + 1),.... ...., (1) based which u, + 1) is predicted. 1;(I++) denotes "yes" or "no". c,(1+1 encodes u, in Appendix A. In SMM shown by Fig. the transformation x;) Cc2 where C E Rox is
2), u,(1+1) Eq. (3) will be derived
learnable, highlights important components, Eq. (3) is transformed to
x; (1) = CA 1-1-11 B E
o=1 Uio - 1+ CB Σ (1
(4)
Finally, the elements of (1) are multiplied with shifted basis functions (9o(1+)), 9. 9o(L+ [L, O. = 1, = AI/NSMM, producing shifted weighted basis functions:
[1, 01' 6) = 22.0 - E (1-
(5)
The Feature Space Decoder projects ú1 + 8) = (1 + 5),... 0 (1 + 5)] onto prediction C"yes" or "no"), as is shown by Fig. 3. Cross entropy loss employed. In each batch, B, images correspond to B, declarative sentences which are converted into B, questions. The tokens of each image are concatenated with those of each corresponding question before feeding into SMM.
B, B,
Sim( ((i,)) Emb(" Emb("yes")
LSMM =
Σ
Vogl (im,P((.)) Emb(" yes")) + Sim( Emb("no")) (6)
where ground truth yt, takes 1 when the Qwen-Chat model Bai et al. (2023) receives question j together with image and returns "yes", else yt,j takes O. (im(P(i, J): Emm(( yes") )) is the cosine similarity between the embedding ji of SMM's output and the embedding of "yes".
3.5 TEMPORAL MOTION ESTIMATION AND TEMTIO- ANOMALY DETECTION
To enhance the spatial local patterns obtained from Stage this paper proposes to encode frames into H.265 (HEVC) videos using FFmpeg Zeng et al. (2016). As is illustrated in Fig. 2(a), motion vectors from encoded videos are extracted, each motion vector associated with x 8 macroblock. The orientation of each motion vector is computed atan? y, x) and quantized into Dm, 8 equi- spaced bins, x and y are the horizontal and vertical components. The average magnitudes of motion vectors in these bins produce D,, dimensional histogram M/ (t) representing region i.
To detect anomalies with anomalous local patterns or irregular dynamics, the Reconstruction Mod- ule (RM) with fully-comnected layers is trained on normal spatial and temporal data. As is shown in Fig. 2(a), the first layer takes in the concatenation of local patterns G (t) and dynamics M (1), maps Ha D... input channels to D2, output channels while the last layer maps D₁, input channels to Ha + Dm output channels. The 5 hidden layers have Dj input channels and Dr, output chan- nels. The reconstructions of spatial and temporal features are conducted together, facilitating the reconstruction of each one to depend on the other. Reconstruction error determines anomaly scores.
4 EXPERIMENTS AND RESULTS
This section compares the proposed method with state-of-the-art ones and presents ablation studies.
4.1 EXPERIMENTAL SETUP
Datasets Experiments are conducted on seven datasets. The training sets of ShanghaîTech, Avenue and UCSD Ped2 contain only normal events and anomalies reside in test data. (1) ShanghaiTech dataset Liu et al. (2018) includes 330 training videos and 107 test videos. Among the two versions of Shanghai Tech dataset Liu et al. (2018) and Zhong et al. (2019); Li et al. (2022a); Zanella et al. (2023), the latter includes abnormal behaviors in both training set and test set. As our approach unsupervised, the first version. (2) CUHK Avenue dataset Lu al. (003)) involves 16

---

Published as a conference paper at ICLR 2025
Table 1: Performance (AUC, %) on the benchmarks. ST, Ave, UB, Ped2 and NWPU represent Shanghai'Tech CUHK Avenue, Ubnormal, UCSD Ped2 and NWPU Campus, respectively. Macro- AUC and micro-AUCRReis Hoshen ) are evaluated.
Algorithm
Year
ST
Ave
UB
Ped2
NWPU
Georgescu et al.
2021b)
2021
89.3 82.7
92.3 / 99.44
6/.1.
99.7 198.7
Acsintoae al.
2021
2021
90.5 /-
93.2 /-
Cal et al. (2021)
2021
-/73.7
86.6
- /96.6
64.55
Reiss & Hoshen
(0022)
2022
89.6/8599
96.2/93.3
99.9 99.1
Zhong al. (2022)
2022
-174.5
89.0
198.1
Zhang et al.
2022a
2022
-180.3
-/ 80.5
-992.9
Lu et al.
2022
2022
85.9/77.6
88.6 87.4
-/62.2
Acsintoae al.
(2022)
2022
90.5/
93.2/93.0
Liu et al.
(023c)
2023
91.4/85.0
/3.9/
Cao et al.
(2023
2023
7992
86.8
-682
Hirschorn al.
(2023)
2023
-185.9
-179.2
Arad & Werman 20233)
2023
-185.9
-93.5
199.1
Sun & Gong (2023)
2023
-13..4
193.77
198.1
Liu ct al. (2233a)
2023
-178.8
-19288
199.7
Yu et al.
2()22a)
2023
-772.6
-/ -99077
197.2
Zhang al. (2024)
2024
93.0/87.5
94.3
72.2/ 70.1
Micorek et al. (2024)
2024
91.5//6.7
96.1/943
85.5
99.9
Astrid et al.
(0244)
2024
-/71.39
-/ /2.114
-194.05
Yang et al.
2024a)
2024
-185.2
189.7
-/71.9
1979
Proposed Method
2024
92.7 88.9
94.5
86.8/81.
99.8 99.1
73.5 / 71.6
training videos and 21 test videos. (3) Ubnormal dataset /Acintoae al. (2022) divided into a
training set with 268 videos. validation set with 64 videos, and test set with 211 videos. (4) NWPU Campus dataset [Cao et al. (2023) comprises 43 scenes, 28 classes of anomalies and 16 hours of video footage. (5) UCSD Ped2 dataset Li et al. (2014) contains 16 normal training videos and 12 test videos. (6) UCF Crime dataset (Sultani et al. (2018) includes 1610 training videos in which 800 contain only normal behaviors. The test set includes 290 videos in which 140 include anomalies. (7) XD Violence Wu et al. (2020) includes 4754 videos where 2349 are non-violent and 2405 are violent. There 3954 training videos and 800 test videos where 500 violent.
Evaluation Metrics Following previous literature Markovitz et al. (2020), Area under Curve (AUC, %) is adopted for evaluation. Differently, the accuracy on ED-Violence dataset is measured using precision-recall curve and the corresponding Average Precision (AP, %) Panariello et al. (022))
Implementation Details To capture more contexts, bounding boxes expanded by 50% on both sides horizontally and vertically. The benefits of box expansion will be shown in Table 4 of Appendix the output of backbone and ITAM are H (t) E S&x.V, Va and F
D. For image region
N. which satisfy S3 257, Va 1408, N₂ 32, Ha 768. Each of the N₂ image tokens has embedding size Ha. Following BLIP-2 Li et al. (202)), the backbone has "ViT-L/14" structure in Radford et al. (02T). The text tokenizer in Fig. 2(a) will be detailed in Appendix C. For sentences with fewer than St tokens, F? (t) is padded with zeros. RM has Dj 512 in intermediate layers.
state machines, is trained on the COCO-Caption dataset Lin et al. (2014). Table 3 shows the influences of NSMM. The Feature Space Encoder (H input channels, output channels) and Feature Space Decoder (O input channels, H. output channels) are learnable fully-connected layers, the weights in C E ROxO are also learnable. All weights are initialized with distribution (O, 0.02). Training spans 20 epoches with initial learning rate X 10 and decay 0.99. RM takes concatenated G (t) and M (t) as input, with ReLU activations. It is trained using Adam optimizer with learning rate 10 -3 for 10 epoches, using MSE loss. Implementations are based on Pytorch Pytorch| 2018)) and NVIDIA 1100 GPU. RM trained on benchmark videos without anomalies. The influences of RM's number of layers will be shown in Appendix E. The evaluations on operational efficiency will be detailed in Appendix H.
SMM, with NMMM
64
8

---

Published as a conference paper at ICLR 2025
Table 2: Performance on UCF-Crime (micro-AUC, %) and XD-Violence (AP, %). UCF and XD
represent UCF-Crime and AD-Violence, respectively.
Algorithm
UCF
XD
Algorithm
UCF
XD
Joo et al.
(2023)
87.58
82.19
Wu et al. (2024a)
86.40
76.03
Chen et al.
(2023
86.98
80.11
Chen et al.
(2024
86.83
88.21
Pu et al. (2023)
86.76
85.59
Yang et al. 2022bb
87.79
83.68
Ian et al. 20244
86.71
82.10
Wu et al. (024b)
88.02
84.51
Zanella et
2024
80.28
85.36
Proposed Method
88.83
86.96
Table Ablations components using Micro-AUC. TME and TSG are short for Temporal Motion Estimation and Temporal Sentence Generation, respectively. N₂ is the number of image tokens.
Setting
Stage
Stage
Ng
TME
TSG
NsMas ST
Ave
UB
Ped2
32
71.9
85.1
72.8
79.3
2
32
X
X
80.3
86.5
73.6
90.7
3
32
V
86.4
88.7
79.3
96.8
4
V
w/o image tokens w/o text tokens
32
X
V
3
79.2
87.8
72.1
95.4
5
80.7
89.6
74.5
95.9
6
32
V
3
88.9
94.5
81.5
99.1
w/o SMM
87.6
93.0
80.1
98.5
8
32
X
84.1
86.2
77.6
94.5
9
32
88.6
94.4
79.8
99.1
10
32
V
5
88.9
94.5
81.5
99.1
4.2
COMPARISONS WITH BASELINES
To demonstrate the superiority, the proposed approach is compared with existing ones, including LLM-based baselines Yang et al. (2024a), for detecting anomalies. Significant improvements are observed in Table Such improvements are attributed the identification of spatial local patterns and dynamics. Results on non-human objects are shown in Table 2 with UCF-Crime and XD- Violence datasets, suggesting that local patterns can generalize different object types.
4.3 ABLATION STUDIES
Ablation Stage and Stage In Setting of Table 3, the reconstruction enror of backbone features H (f) is used to detect anomalies. Setting 2 and Setting show the utilization of Stage 1 and both stages for reconstruction, respectively. The comparison shows that Stages 1 and 2 both play crucial roles selecting text-informative local patterns, as will be illustrated Fig. 4.
Ablation ITAM's Structure To demonstrate that the primary contributor generalization is image-text alignment instead of pre-existing models, we conduct an ablation study by varying the structure and training data of ITAM. Detailed results and analysis provided in Appendix
Ablation on Cross-Modality Attention Setting 4 in Table 3 replaces cross-modality feature G (1)
in Setting with textual feature F (t) sentence from TSGM, using reconstruction error on F (t)
to determine anomalies. Setting 5 discards text tokens, only using the reconstruction error on F (1).
Fig. 5 also shows that combining visual and textual features outperforms using single modality.
Ablation on Temporal Motion Estimation The improvement Setting over Setting demon- strates that temporal dynamics complements local patterns in detecting anomalies. Setting 8 shows the performance of only using reconstruction error on dynamics M (t) for anomaly detection.
Ablation on the SMM in Temporal Sentence Generation in Stage 2 The comparison between Setting 6 and Setting shows that TSGM only uses Q-Former Li et al. (0023) for image-grounde text generation without SMM to incorporate previous captions, performance drops. As a result, the mixture of image tokens and text tokens from different moments contributes to more informative sentences. More ablations SMM will be shown in Appendix B.

---

Published as a conference paper at ICLR 2025
(a)
(b)
Figure 4: Heatmaps local patterns in two stages, (a) Normal events. (b) Abnormal events. Both (a) and (b) follow the same arrangement: the first contains input images, the second row shows the features from ITAM, and the third row shows the local patterns selected by CMAM.
0.8
1.0
Vand
Anurie
0.5
1.0
Cooo dy Ataaiio
0.6
0.6
0.4
0
50
100
150
200
250
300
350
100
200
300
400
500
600
(a)
(b)
1.0
1.0
0.8
Dep Veall Toturo
0.8
Dey Viaal
0.6
0.6
0
100
200
300
400
0
100
200
300
400
(d)
Figure 5: Anomaly scores obtained using image features, text features and combined ones. Cross- modality attention detects anomalies even under occlusion and low resolution. Green and red boxes show the anomalies detected with single modality and cross-modality attention, respectively.
Ablation SMM's Structure SMM stacks Namm state machines, each predicting future period of AI/N //mm- The stacking mechanism achieves the full prediction Al. Setting 6, and 10 in Table 3 show that Nsmm 3 outperforms Nomm 1. The task for each state machine becomes simpler because each one focuses on short-term dependencies. Predicting long period A/ requires capturing both short- and long-term dependencies. A single state machine struggles handle these varying dependencies effectively, especially in our case with non-linear multi-modal dependencies.
Moreover, the ablation on the number of image tokens N₂ will be involved in Appendix E.
4.4 SUBJECIIVE RESULTS LOCAL PATTERNS
Fig. 4 subjectively shows local patterns. The second rows of Fig. 4(a) and (b) highlight the patterns for (((() while the third rows display those for (t). The heatmaps, generated using Grad-CAM Selvaraju et al. (2017), show that local patterns span similar semantic regions across normal and abnormal events. Cross-modality attention refines these patterns to focus on semantically relevant components, enhancing generalization. More visualizations will be presented in Appendix I.
DISCUSSION AND CONCLUSION
Limitations: The limitation of our work lies in the reliance object detectors, because the direct processing of an image with many objects using VLM can result context being ignored. Please refer to Appendix J for more potential directions of improvement.
this paper, we establish a framework for video anomaly detection by locating local patterns through image-text alignment and cross-modality attention. At the core of the framework is identifying the text-informative local patterns that generalize to novel anomalies, ensuring consistent representations across novel visual data. Additionally, temporal sentence generation and motion estimation augment cross-man modality attention and complement spatial local patterns, respectively. Extensive experiments show that the framework surpasses existing -tate-of-the-art methods.
Conclusions:
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is supported by the National Natural Science Foundation of China under Grant (62301020) and Beijing Natural Science Foundation under Grant (4234085).
REFERENCES
Davide Abati, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Latent space autoregression for novelty detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 481-490. IEEE, 2019.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303 087744 2023.
Andra Acsintoae, Andrei Florescu, Mariana-Iuliana Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor Tonescu, Fahad Shahbaz Khan, and Mubarak Shah. Ubnormal: New benchmark for supervised open-set video anomaly detection. arXiv preprint urXiv:2111.08644 2021.
Andra Acsintoae, Andrei Florescu, Mariana-Iuliana Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor Ionescu, Fahad Shahbaz Khan, and Mubarak Shah. Ubnormal: New benchmark for supervised open-set video anomaly detection. In Proceedings of the IEEE/CVF Conference 2022.
Computer Vision and Pattern Recognition, pp.
Yoav Arad and Michael Werman. Beyond the benchmark: Detecting diverse anomalies in videos.
arXiv preprint arXiv:2310.01904, 2023.
George B Arfken, Hans J Weber, and Frank E Harris. Mathematical methods for physicists: comprehensive guide. Academic press, 2011.
Mujtaba Asad, Jie Yang. Enmei Tu, Liming Chen, and Xiangjian He. Anomaly3d: Video anomaly detection based on 3d-normality clusters. Journal of Visual Communication and Image Represen- tation, 15:103047, 2021.
Marcella Astrid, Muhammad Zaigham Zaheer, and Seung-Ik Lee. Synthetic temporal anomaly guided end-to-end video anomaly detection. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 207-214. 2021.
Marcella Astrid, Muhammad Zaigham Zaheer, and Seung-lk Lee. Constricting normal latent space for anomaly detection with normal-only training data. arXiv preprint arXiv:2403 /6270, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2. pp. 4, 2021.
Ruichu Cai, Hao Zhang, Wen Liu, Shenghua Gao, and Zhifeng Hao. Appearance-motte motion memory consistency network for video anomaly detection. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 938- 946,946,220 2021.
Congqi Cao, Yue Lu, Peng Wang, and Yanning Zhang. A new comprehensive benchmark for semi- supervised video anomaly detection and anticipation. In Proceedings of the IEFF/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 00392-204011, 2023.
Raghavendra Chalapathy, A. K. Menon, and S. Chawla. Robust, deep and inductive anomaly de- tection. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 36-51. Springer, 2017.
Shuning Chang, Yanchao Li, Shengmei Shen, Jiashi Feng, and Zhiying Zhou. Contrastive attention for video anomaly detection. IEFE Transactions Multimedia, 24:4067-4074 2021.

---

Published as a conference paper at ICLR 2025
Yunpeng Chang, Zhigang Tu, Wei Xie, and Junsong Yuan. Clustering driven deep autoencoder for video anomaly detection. In European Conference on Computer Vision, pp. 329-345. Springer, 2020.
Junxi Chen, Liang Li, Li Su, Zue Zha, and Qingming Huang. Prompt-enhanced multiple in- stance learning for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18319-18329, 2024.
Yingxian Chen, Zhengzhe Liu, Baoheng Zhang, Wilton Fok, Xiaojuan Qi. and Yik-Chung Wu. Mgfn: Mignituue-contraattvve glance-and-focu network for weakly-supervised video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 387-
395, 2023.
Zhi Chen, Jiang Duan, Li Kang, and Guoping Qiu. Supervised anomaly detection via conditional generative adversarial network and ensemble active learning. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(6):7781-7799) 2022.
MyeongAh Cho, Taeoh Kim, Woo Jin Kim, Suhwan Cho, and Sangyoun Lee. Unsupervised video anomaly detection via normalizing flows with implicit latent features. Pattern Recognition, 129: 108703, 2022.
MyeongAh Cho, Minjung Kim, Sangwon Hwang, Chaewon Park. Kyungjae Lee, and Sangyoun Lee. Look around for anomalies: Weakly-supervised anomaly detection via context-motion re- lational learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12137-12146, 2023.
Bruno Degardin and Hugo Proença. Iterative weak//eel-suupervised classification framework for abnormal events detection. Pattern Recognition Letters, 145:50-57, 2021
Andong Deng, Taojiannan Yang, and Chen Chen. A large-scale study of spatiotemporal representa- tion learning with a new benchmark on action recognition. pp. 90519-205311 2023.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Chhavi Dhiman and Dinesh Kumar Vishwakarma. View-invariant deep architecture for human ac- tion recognition using two-stream motion and shape temporal dynamics. IEEE Transactions on Image Processing, 29:3835-3844, 2020.
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. In Advances in Neural Information Processing Systems, volume 34, 2021.
Keval Doshi and Yasin Yilmaz. Continual learning for anomaly detection in surveillance videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Work- shops, 254-255, 2020.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulcayylary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022.
Yidan Fan, Yongxin Yu, Wenhuan Lu, and Yahong Han. supriy-s video anomaly de- tection with snippet anomalous attention. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Zhiwen Fang, Joey Tianyi Zhou, Yang Xiao, Yanan Li, and Feng Yang. Multi-encoderr towards effective anomaly detection in videos. IEEE Transactions on Multimedia, 23:4106-4116, 2020.
Jia-Ch Jia-Chang Feng, Fa-Ting Hong, and Shi Zheng. Mist: Multiple instance self-training frame- work for video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14009-14018, 2021.
12

---

Published as a conference paper at ICLR 2025
Alessandro Flaborea, Luca Collorone, Guido Maria D' Ammlly Di Melendugno, Stefano D' D'Arrigo, Bardh Prenkaj, and Fabio Galasso. Multimodal motion conditioned diffusion model for skeleton- based video anomaly detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10318 8-10229, 2023.
Jie Gao, Bineng Zhong, and Yan Chen. Robust tracking via learning model update with unsupervised anomaly detection philosophy. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
Mariana-lulians Georgescu, Antonio Barbalau, Radu Tudor Ionescu, Fahad Shahbaz Khan, Mar- ius Popescu, and Mubarak Shah. Anomaly detection in video via self--uper and multi-task learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 12742-12752. IEEE, 2021a.
Mariana Iuliana Georgescu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Marius Popescu, and Mubarak Shah. A Barkkround-agnossee framework with adversarial training for abnormal event detection in video. IEEE transactions on pattern analysis and machine intelligence, 44(9):4505- 4523, 2021b.
Dong Gong, L. Liu, Le, B. Saha, M.R. Mansour, S. Venkatesh, and A.V. Den Hengel. Memorizing normality detect anomaly: detection. In Proceedings of IEEE International Conference on Computer Vision, pp. 1705-1714. IEEE, 2019.
deep autoencoder for unsupervised anomaly
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Chongye Guo, Hongbo Wang. Yingjie Xia, and Guorui Feng. Learning appearance-motion synergy via memory-guided event prediction for video anomaly detection. IEEE Transactions Circuits and Systems Video Technology, 2023.
Nitin Hardeniya, Jacob Perkins, Deepti Chopra, Nisheeth Joshi, and Mathur, Natural language processing: python and NLTK. Packt Publishing Ltd, 2016.
Or Hirschorn, Shai Avidan, and Shai Avidan. Normalizing flows for human pose anomaly detec- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13545- 13554, 2023.
Xin Huang, Yutao Hu, Xiaoyan Luo, Jungong Han, Baochang Zhang, and Xianbin Cao. Boosting variational inference with margin learning for few-shot scene-adaptive anomaly detection. IEEE Transactions on Circuits and Systems for Video Technology, 2022.
Radu Tudor lonescu, Fahad Shahbaz Khan, Mariana-luliana Georgescu, and Ling Shao. Object- centric auto-encoders and dummy anomalies for abnormal event detection in video. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7842-7851, 2019.
Hyekang Kevin Joo, Khoa Vo, Kashu Yamazaki, and Ngan Le. Clip-tsa: Clip-assisted temporal self- attention for weakly- supervied video anomaly detection. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 3230-3234, IEEE, 2023.
Siwon Kim, Kukjin Choi, Hyun-Soo Choi, Byunghan Lee, and Sungroh Yoon. Towards a rigorous evaluation of time-series anomaly detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pp.
71..114444 7194-7201. AAAI Press, 2022.
Bo Li, Sam Leroux. and Pieter Simoens. Decoupled appearance and motion learning for effi- cient anomaly detection in surveillance video. Computer Vision and Image Understanding, 210: 103249, 2021a.
Guoqiu Li, Guanxiong Cai, Xingyu Zeng, and Rui Zhao. Scale-aware spatio-temporal relation learning for video anomaly detection. In European Conference on Computer Vision, pp. 333- 350. Springer, 2022a.
13

---

Published as a conference paper at ICLR 2025
Jing Li, Qingwang Huang, Yingjun Du, Xiantong Zhen, Shengyong Chen, and Ling Shao. Varia- tional abnormal behavior detection with motion consistency. IEEE Transactions on Image Pro- cessing, 31:275-288 2021b.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Learning from noisy data with robust representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9485-9494, 2021c.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Nanjun Li, Faliang Chang, and Chunsheng Liu. Spatial-temporal cascade autoencoder for video anomaly detection in crowded scenes. IEEE Transactions on Multimedia, 23:203-215, 2020a.
Nanjun Li, Faliang Chang, and Chunsheng Liu. self-trained spatial graph convolutional network for unsupervised human-related anomalous event detection in complex scenes. IEEE Transactions on Cognitive and Developmental Systems, 2022b.
Shuo Li, Fang Liu, and Licheng Jiao. Self-training multi-sequence learning with transformer for weakly supervised video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24. AAAI Press, 2022c.
Weixin Li, Vijay Mahadevan, and Nuno Vasconcelos. Anomaly detection and localization in crowded scenes. IFEE Transactions on Pattern Analysis and Machine Intelligence, 36(1):18- 32, 2014.
Xuelong Li, Mulin Chen, and Qi Wang. Quantifying and detecting collective motion in crowd scenes. IEEE Transactions on Image Processing, 29:5571-5583, 2020b.
Swee Kiat Lim, Yi Loo, Ngoc-Trung Tran, Ngai-Man Cheung, Gemma Roig, and Yuval Elovici. Doping: Generative data augmentation for unsupervised anomaly detection with gan. 2018 IEEE international conference data mining (ICDM), pp. 1122-1127. IEEE, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Xiangru Lin, Yuyang Chen, Guanbin Li, and Yizhou Yu. causal inference look at unsupervised video anomaly detection. In Proceedings of the Thirty-sixth AAAI Conference on Artificial Intel- ligence. AAAI Press, 2022.
Boyang Liu, Pang-Ning Tan, and Jiayu Zhou. Unsupervised anomaly detection by robust density estimation. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 2022a.
Wen Liu. W. Luo, D. Lian, and S. Gao. Future frame prediction for anomaly detection-a new baseline. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5536-6545. IEEE, 2018
Wenrui Liu, Hong Chang, Bingpeng Ma. Shiguang Shan, and Xilin Chen. Diversity- measurable anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2215-112166, 2023a.
Yang Liu, Dingkang Yang, Yan Wang, Jing Liu, Jun Liu, Azzedine Boukerche, Peng Sun, and Liang Song. Generalized video anomaly event detection: Systematic taxonomy and comparison of deep models. ACM Computing Surveys, 2023b.
Yuejiang Liu, Riccardo Cadei, Jonas Schweizer, Sherwin Bahmani, and Alexandre Alahi. Towards robust and adaptive motion forecasting: A causal representation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1708 170922. IEEE, 2022b.
14

---

Published as a conference paper at ICLR 2025
Zhian Liu, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. A hybrid video anomaly detection framework via tion. In Proceedings of the IEEE/CVF international conference computer vision, pp. 13588- 13597, 2021.
flow reconstruction and flowggudde frame predic-
Zuhao Liu, Xiao-Ming Wu, Dian Zheng, Kun- Yu Lin, and Wei-Shi Zheng. Generating anomalies for video anomaly detection with prompt-based feature mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24500-24510, 2023c.
Cewu Lu, J. Shi, and J. Jia. Abnormal event detection at 150 fps in matlab. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2720-2727. IEEE, 2013.
Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, and Yang Wang. Few-shot scene-adaptive anomaly detection. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 125-141. Springer, 2020.
Yue Lu, Congqi Cao, Yifan Zhang, and Yanning Zhang. Learnable locality-sensitive hashing for video anomaly detection. IEFE Transactions on Circuits and Systems for Video Technology, 33 (2):963-976, 2022.
Weixin Luo, W. Liu, and S. Gao. A revisit of sparse coding based anomaly detection in stacked mn framework. In Proceedings of IEEE International Conference on Computer Vision, pp. 341-349.
IEEE, 2017.
Weixin Luo, Wen Liu, Dongze Lian, Jinhui Tang, Lixin Duan, Xi Peng, and Shenghua Gao. Video anomaly detection with sparse coding inspired deep neural networks. IEEE transactions on pat- tern analysis and machine intelligence, 43(3):1070- 1084, 2019.
Weixin Luo, Wen Liu, and Shenghua Gao. Normal graph: Spatial temporal graph convolutional networks based prediction network for skeleton based video anomaly detection. Neurocomputing, 444:332-337, 2021a.
Weixin Luo, Wen Liu, Dongze Lian, and Shenghua Gao. Future frame prediction network for video anomaly detection. IEEE transactions on pattern analysis and machine intelligence, 44(1)): 7505-7520, 2021b.
Hui Lv, Chen Chen, Zhen Cui, Chunyan Xu, Yong Li, and Jian Yang. Learning normal dynamics in videos with meta prototype network. arXiv preprint arXiv:2104. 104.0669, 2021.
Hui Lv, Zhongqi Yue, Qianru Sun, Bin Luo, Zhen Cui, and Hanwang Zhang. Unbiased multiple in- stance learning for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 8022-8031, 2023.
Neelu Madan, Arya Farkhondeh, Kamal Nasrollahi, Sergio Escalera, and Thomas B Moeslund. Temporal cues from socially unacceptable trajectories for anomaly detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2150-2158, 2021.
Neelu Madan, Nicolae-Cätte CCttle- Ristea, Radu Tudor lonescu, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B Moeslund, and Mubarak Shah. Self-supervised masked convolutional trans- former block for anomaly detection. IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2023.
Amir Markovitz, Gilad Sharir, Itamar Friedman, Lihi Zelnik-Manor, and Shai Avidan. Graph em- bedded pose clustering for anomaly detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10539-10547. IEEE, 2020.
Jakub Micorek, Horst Possegger, Dominik Narnhofer, Horst Bischof, and Mateusz Kozinski. Mulde: Multiscale log-density estimation via denoising score matching for video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18868-18877, 2024.
Romero Morais, V. Le, T. Tran, B. Saha, M. Mansour, and S. Venkatesh. Learning regularity in skeleton trajectories for anomaly detection in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11996-12004, IEFF, 2019.
15

---

Published as a conference paper at ICLR 2025
Trong Nguyen Nguyen and J. Meunier. Anomaly detection in video sequence with appearance tion correspondence. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1273-1283 IEEE, 2019.
Aniello Panariello, Angelo Porrello, Simone Calderara, and Rita Cucchiara. Consistency-basee self- supervised learning for temporal anomaly localization. In European Conference on Computer Vision, pp. 338-349. Springer, 2022.
Guansong Pang. Cheng Yan, Chunhua Shen, Anton van den Hengel, and Xiao Bai. Self-trained deep ordinal regression for end-to-end video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12173- 12182. IEEE, 2020.
Hyunjong Park, Jongyoun Noh, and Bumsub Ham. Learning memory-guided guuded normality for anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 14372-1438 14381. IEEE, 2020.
Lorenzo Perini, Vincent Vercruyssen, and Jesse Davis. Transferring the contamination factor be- tween anomaly detection domains by shape similarity, In Proceedings the AAAI Conference Artificial Intelligence, volume 36, pp. 4128-4136 AAAI Press, 2022.
Yujiang Pu, Xiaoyu Wu, and Shengjin Wang. Learning prompt-enhanced context features for weakly video anomaly detection. arXiv preprint wrXiv:2306.144515 2023.
Didik Purwanto, Yie-Tarng Chen, and Wen-Hsien Fang. Dance with self-attention: A new look of conditional random fields on anomaly detection in videos. In Proceedings of the TEEE/CVF International Conference on Computer Vision, pp. 173-183, 2021.
Automatic Differentiation In Pytorch. Pytorch, 2018
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, al. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Bharathkumar Ramachandra, Michael J Jones, and Ranga Raju Vatsavai. survey of single-scene video anomaly detection. IEEE transactions on pultern analysis and machine intelligence, 44(5): 2293-2312, 2020.
Tal Reiss and Yedid Hoshen. Attribute- based representations for accurate and interpretable video anomaly detection. arXiv preprint arXiv:22 2022.
Nicolae-Catalin Ristea, Neelu Madan, Radu Tudor Ionescu, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas Moeslund, and Mubarak Shah. Self-supervised predictive convolutional at- tentive block for anomaly detection. arXiv preprint arXiv:2111.09099 2021.
Mohammad Sabokrou, M. Khalooei. M. Fathy, and E. Adeli. Adversarially learned one-class classi- fier for novelty detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3379-3388. IEEE, 2018.
Hitesh Sapkota and Qi Yu. Bayesian nonparametric submodular video partition for robust anomaly detection. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recog- nition, pp. 3212-3222211111 2022.
Fumiaki Sato, Ryo Hachiuma, and Taiki Sekii. Prompt-guided zero-shot anomaly action recogni- tion using pretrained deep skeleton features. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6471-6488 2023.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
16

---

Published as a conference paper at ICLR 2025
Chenrui Shi, Che Sun, Yuwei Wu, and Yunde Jia. Video anomaly detection via sequentially learning multiple pretext tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10330-10340, 2023a.
Haoyue Shi, Le Wang. Sanping Zhou, Gang Hua, and Wei Tang. Abnormal ratios guided multi-phasee self-training for weakly-supervised video anomaly detection. IEEE Transactions on Multimedia, 2023b.
Ashish Singh, Michael J Jones, and Erik G Learned-Mllerr Eval: Explainable video anomaly local- ization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 18717-18726, 2023.
Alexandros Stergiou, Brent De Weerdt, and Nikos Deligiannis. Holistic representation learning for multitask trajectory anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6729-6739, 2024.
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 6479-6488, 2018.
Che Sun, Yunde Jia, Yao Hu, and Yuwei Wu. Scene-aware context reasoning for unsupervised abnormal event detection in videos. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 192, 2020.
Shengyang Sun and Xiaojin Gong. Hierarchical semantic contrast for scene-aware video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 22846-22856, 2023.
Weijun Tan, Qi Yao, and Jingfeng Liu. Overlooked video classification in weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 202-202222222222222 210, 2024.
Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W Verjans, and Gustavo Carneiro. Weakly-supervised video anomaly detection with robust temporal feature magnitude learning. In Proceedings of the IEEE/CVÉ international conference on computer vision, pp- 4975 4975-498,, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Chien Yao Wang, Alexey Bochkovskiy, and Hong- Hoag-Yuan Mark Liao. Yolov7: Trainable bag-of- freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 7464-7475, 2023.
Guodong Wang, Yunhong Wang, Jie Qin, Dongming Zhang, Xiuguo Bao, and Di Huang. Video anomaly detection by solving decoupled spatio-temporal jigsaw puzzles. arXiv preprint arXiv:2207.10172, 2022a.
Siqi Wang, Yijie Zeng, Guang Yu, Zhen Cheng, Xinwang Liu, Sihang Zhou, En Zhu, Marius Kloft, Jianping Yin, and Qing Liao. E3 outlier: self-supervised framework for unsupervised deep outlier detection. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(3):2952- 2969, 2022b.
Xuanzhao Wang, Zhengping Che, Bo Jiang, Ning Xiao, Ke Yang. Jian Tang, Jieping Ye, Jingyu Wang, and Qi Qi. Robust unsupervised video anomaly detection by multipath frame prediction. IEEE transactions on neural networks and learning systems, 33(6):2301-23) 2021.
Jhih-Ciang Wu, He-Yen Hsich, Ding-Jic Chen, Chiou-Shann Fuh, and Tyng-Luh Liu. Self- supervised sparse representation for video anomaly detection. In European Conference on Com- puter Vision, pp. 729-745. Springer, 2022.
Peihao Wu, Wenqian Wang, Faliang Chang, Chunsheng Liu, and Bin Wang. Dss-net: Dynamic self-supervised network for video anomaly detection. IEEE Transactions Multimedia, 2023.
17

---

Published as a conference paper at ICLR 2025
Peng Wu and Jing Liu. Learning causal temporal relation and feature discrimination for anomaly detection. IEEE Transactions on Image Processing, 30:3513-3527, 3527, 2021.
Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In Computer ECCo 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 322-339. Springer, 2020.
Peng Wu, Xuerong Zhou, Guansong Pang, Yujia Sun, Jing Liu, Peng Wang. and Yanning Zhang- Open-vocabulary video anomaly detection. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 18297-18307, 2024a.
Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, and Yanning Zhang. Vadclip: Adapting vision-language models for weakly supervised video anomaly detec- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6074-6082, 2024b.
Cheng Yan, Shiyu Zhang, Yang Liu, Guansong Pang, and Wenjun Wang. Feature prediction diffu- sion model for video anomaly detection. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 5527-5537, 2023.
Yuchen Yang. Kwonjoon Lee, Behzad Dariush, Yinzhi Cao, and Shao- Yuan Lo. Follow the rules: Reasoning for video anomaly detection with large language models. arXiv preprint 1xXiv:2407.10299 2024a.
Zhen Yang, Yuanfang Guo, Junfu Wang, Di Huang. Xiuguo Bao, and Yunhong Wang. Towards video anomaly detection in the real world: A binarization embedded weakly-supervised network. IEEE Transactions on Circuits and Systems for Video Technology, 2023a.
Zhiwei Yang, Peng Wu, Jing Liu, and Xiaotao Liu, Dynamic local aggregation network with adap- tive clusterer for anomaly detection. In European Conference Computer Vision, pp. 404 421. Springer, 2022.
Zhiwei Yang, Jing Liu, Zhaoyang Wu, Peng Wu, and Xiaotao Liu. Video event restoration based on keyframes for video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14592-14601, 2023b.
Zhiwei Yang, Jing Liu, and Peng Wu. Text prompt with normality guidance for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18899-18908 2024b.
Fei Yu, Mo Zhang, Hexin Dong, Sheng Hu, Bin Dong, and Li Zhang. Dast: Unsupervised domain adaptation in semantic segmentation based on discriminator attention and self-training. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10754-10762, 2021.
Guang Yu, Siqi Wang, Zhiping Cai,, Xinwang Liu, Chuanfu Xu, and Chengkun Wu. Deep anomaly discovery from unlabeled videos via normality advantage and self-paces refinement. In Proceed- ings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 13987-13998, 2022a.
Jiashuo Yu, Jinyu Liu, Ying Cheng, Rui Feng, and Yuejie Zhang. Modality-aware contrastive in- stance learning with self-distillation for weakly-supervised audio-visual violence detection. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 6278-6287, 2022b.
Shoubin Yu, Zhongyin Zhao, Haoshu Fang, Andong Deng, Haisheng Su, Dongliang Wang. Weihao Gan, Cewu Lu, and Wei Wu. Regularity learning via explicit distribution modeling for skeletal video anomaly detection. IEEE Trunsactions on Circuits and Systems for Video Technology, 2023.
M Zaigham Zaheer, Arif Mahmood, M Haris Khan, Mattia Segu, Fisher Yu, and Seung-Ik Ik Lee. Generative cooperative learning for unsupervised video anomaly detection. In Proceedings of the IFFF/CVF conference on computer vision and pattern recognition, pp. 14744 14754, 2022a.
18

---

Published as a conference paper at ICLR 2025
Muhammad Zaigham Zaheer, Jin-ha Lee, Marcella Astrid, and Seung-lk Lee. Old is gold: Re- defining the adversarially learned one-class classifier training paradigm. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14183-14193 IEEE, 2020a.
Muhammad Zaigham Zaheer, Arif Mahmood, Marcella Astrid, and Seung- Ik Lee. Claws: Cluster- ing assisted weakly supervised learning with normalcy suppression for anomalous event detection. In European Conference on Computer Vision, pp. 358-376. Springer. 2020b.
Muhammad Zaigham Zaheer, Jin-Ha Lee, Arif Mahmood, Marcella Astrid, and Seung-Ik Lee. Sta- bilizing adversarially learned one-class novelty detection using pseudo anomalies. IEEE Trans- actions Image Processing. 51:5963-5975 2022b.
Luca Zanella, Benedetta Liberatori, Willi Menapace, Fabio Poiesi, Yiming Wang, and Elisa Ricci. Delving into clip latent space for video anomaly recognition. arXiv preprint arXiv:2310.02835, 2023.
Luca Zanella, Willi Menapace, Massimiliano Mancini, Yiming Wang, and Elisa Ricci. Harnessing large language models for training-free free video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18527-18536, 2024.
Hao Zeng, Zhiyong Zhang, and Lulin Shi. Research and implementation of video codec based on ffmpeg. In 2016 international conference network and information systems for computers (ICNISC), pp. 184-188. IEEE, 2016.
Xianlin Zeng, Yalong Jiang, Wenrui Ding, Hongguang Li, Yafeng Hao, and Zifeng Qiu. A hierar- chical spatio-temporal graph convolutional neural network for anomaly detection videos. IEEE Transactions Circuits and Systems for Video Technology. 2021.
Chen Zhang, Guorong Li. Yuankai Qi, Shuhui Wang, Laiyun Qing, Qingming Huang, and Ming- Hsuan Yang. Exploiting completeness and uncertainty of pscudo labels for weakly supervised video anomaly detection. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16271-16280, 2023a.
Menghao Zhang, Jingyu Wang, Qi Qi, Haifeng Sun, Zirui Zhuang, Pengfei Ren, Ruilong Ma, and Jianxin Liao. Multi-scale video anomaly detection by multi-grained spatio-temporal represen- tation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17385 17394, 2024.
Sijia Zhang, Maoguo Gong, Yu Xie, A Kai Qin, Hao Li, Yuan Gao, and Yew-Soon Ong. Influence- aware attention networks for anomaly detection in surveillance videos. IEEE Transactions Circuits and Systems for Video Technology, 32(5):5427-5437, 2022a.
Xinfeng Zhang, Jinpeng Fang, Baoqing Yang, Shuhan Chen, and Bin Li. Hybrid attention and motion constraint for anomaly detection in crowded scenes. IEEE Transactions on Circuits and Systems for Video Technology, 2022b.
Yu Zhang, Xiushan Nie, Rundong He, Meng Chen, and Yilong Yin. Normality learning in multi- space for video anomaly detection. IEEE Transactions on Circuits and Systems for Video Tech- nology, 3119):694-37766 2020.
Yu-Xuan Zhang. Hua Meng, Xue-Mei Cao, Zhengehun Zhou, Mei Yang, and Avik Ranjan Ad- hikary. Interpreting vulnerabilities of multi-instance learning to adversarial perturbations. Pattern Recognition, pp. 109725, 2023b.
Jia- diin Zhong, Nannan Li, Weijie Kong. Thomas H Li Shan Liu, and Ge Li. Graph convolutional label noise cleaner: Train plug-and and-pay action classifier for anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1237-1246. IEFE, 2019.
Yuanhong Zhong, Xia Chen, Yongting Hu, Panliang Tang, and Fan Ren. Bidirectional spatio- temporal feature learning with multiscale evaluation for video anomaly detection. IEEE Transac- fions Circuits and Systems for Video Technology, 52(12):8255882% 2022.
19

---

Published as a conference paper at ICLR 2025
Hang Zhou, Junqing Yu, and Wei Yang. Dual memory units with uncertainty regulation for weakly supervised video anomaly detection. arXiv preprint arXiv:2302.05160, 2023a.
Qihang Zhou, Guansong Pang, Yu Tian, Shibo He, and Jiming Chen. Anomalyclip: Object-agnostic prompt learning for zero-shot anomaly detection. arXiv preprint arXiv:2310. 18961, 2023b.
Yuansheng Zhu, Wentao Bao, and Qi Yu. Towards open set video anomaly detection. European Conference Computer Vision, pp. 395-412 Springer, 2022.
A DETAILS ABOUT SMM
We model the sequence V.(() [L - 1),,...F',,.. ()) (!))
St + Ng of object consisting of both St columns denoting the sentence from t -- and Ng columns denoting image tokens at t. Matrix V,(t) is projected to (t) E ROxE with the Feature Space Encoder which
fully-connected layer in Fig. 3. The I'-th column in U.(()
is [uc.] (L'),..)... "i,o((')] 1 E (1 - I, 1]_ varies along the column dimension of V;(() and U.(().
(1 T., is the window of columns which are encoded by (1) together. For simplicity, ignore
index é in the following parts which conduct analysis along the column dimension of input tensor at
any moment t.
SMM: Sequence sequence, components modeling dependencies
Input Sequence
U,(r): lines weighted hasis functions
functions ww......ee...................... www.w...........................
Feature Space Encoder (Fully-comnected layer)
L=1)
(7)
C/ (f)
Basis I:
(1)
(9) (9)
11000000000
ViDee L.
11-
(,(()
weights
8 80
Figure 6: Basis functions in state machines for modeling sequences.
Fig. shows the basis functions in state machines for modeling sequences. The rows in (t( are the weighted version of the O length-Z Legendre polynomials [Arfken et al. (2011) (1)(.).... ((()), o [1. O). Specifically, the o-th row can be represented as:
[1 1), u, (1, cr.9 go(]). [1,0]
In SMM, state vector (1) T with O weights c₁ (4), ((, are the weights of polynomials and encode the dependencies among the columns in V:(t).
As be seen from Fig. 6, c²(11) encodes the dependencies among I L + 1. .... 1 columns, (22)2) encodes the dependencies among la L 1,..., l2 columns. According to Gu et al. (020
the dynamics of 1-dimensional sequence fi(I) across a period can be represented by (7 E R
satisfying (1 - L + !)... f((()] Do c, (1), ..., go (L),,, E R, [ + 1,4]. The transitions from ci(4) to c1(22) facilitates the prediction in the sequence:
ce
d AHpPooc(() BHIPPof(()
(1)
(8)
By combining Eq. (7) with Eq. (8), we can obtain:
(1) = (HIPPOC(() + BHAPPO Σo o=1 (1)
(9)
dt
The matrices AHıPPO and BHiPPO are defined Gu et al. (2020) with o, h E [1,0]:
(2o+ 0.5 (2h 1)..5
if h
AHiPPo(o, h))
if h,
(10)
if h.
20

---

Published as a conference paper at ICLR 2025
BIXPPO(O) (2o + 1)a.s
(11)
L
To discretize Eq. (9), we obtain
c:(l + ()(+ c:(1)
(mppoc((() + BRiPPO > (i,o(!)
lim A-+0
lim A-0
(12)
AHiPPOC A) BHiPPO E U1ol + +
A =-1
2
which can be transformed to
I AmPPo
AB ABRPPO
Σ 0=1 (1
(1)) = - AHiPPP
c;(1 1)+
(13)
I
& AmPPo
which simplifies to
c1(1) Ac,(1 - 1) + +B)
E -o==1 (1,
(14)
As a result, Legendre LLegendre(O, E ROx0 and B = Beggendre (O, L.) E Rxx1 are determined
by Legendre bases.
B SUBJECTIVE RESULTS OF TEMPORAL SENTENCE GENERATION WITH SMM
In Fig. 7, the green bounding boxes indicate the anomalies that can be detected by directly applying the Q-Former, as described in Li et al. (2023), for image-grounded text generation in TSGM. The red bounding boxes show the cases where only with the combination of SMM and Q-Former in TSGM can the anomalies be detected. The curves show anomaly scores. Under poor observational condi- tions like occlusions and low resolutions, SMM complements the Q-Fommer in TSGM to effectively detect abnormal events.
C STRUCTURES OF MODULES FOR IMAGE-TEXT ALIGNMENT AND MAGGE-GOUNDED TEXT GENERATION
ITAM the image transformer of Q-Former et al. 2023 as shown Fig. It outputs (() xHa which is aligned with the output from text transformer Li et al. (2023) during training to learn extracting text-aligned features. The text tokenizer in Fig. 2 is part of the text transformer of Q-Former Li et al. (2023). ITAM and text tokenizer are frozen in our work.
Image Transformer select from (!) the caption-informative local patterns, this module is built with self-attention layers, cross-attention layers and feed-forward layers, is shown by Fig. Firstly, N₂ learnable query embeddings attend to each other in self atf-attention layers before inter- acting with H (t) through cross-attention layers. Each query embedding has dimension Ha. The Image-attention Module involves sequential transformer layers each of which includes attention layer, one cross-attention layer and one feed-forward layer. H (t) acts as a static input to the cross-attention layers across all transformer layers. The transformer layers sequentially refine the understanding and integration of H (L) with learned queries. Each self-attention layer is im- plemented according to Vaswani et al. (2017) with 12 heads, producing output Q (t) e N, Ha, Each cross-attention layer has 12 heads with H! functioning as key and value, performs feature fusion by combining H/ (t) with Q (t) Z E Na, Z (t) projected by fully-connected layers to (((() E NH
self-
21

---

Published as a conference paper at ICLR 2025
image grossded grestation
(1)
Outy Vormurr Image grounde tre grarration
SMM cocsplaaumn Q Fomur TSGM
(h)
Daly U-Formie for inagrggmaated tesi gracmiionn
SMM cappkmms U-Former in EsGM
(6)
Daly Former for image groased tect preccition
SMM copplmmm ( Fomerr ln TSGM
1.2
(d)
Figure 7: Demonstration of the effectiveness of SMM in TSGM. (a) The man is riding a unicycle but viewed under low resolutions. (b) The man is running but viewed under low resolutions. (c) The vehicle is viewed under occlusions. (d) The man riding a bicycle but viewed under low resolutions.
Text Transformer To encode captions, the module is built with self-attention layers and feed- forward layers, as shown by Fig. The self-attention layers and feed-forward layers are shared by Image-attention Module and Text-attention Module. In self-attention modules, the text tokens sentence with maximum length St attend to
(() E (t), (t)) ..., E/,s, (t) E SiS each other. S N₂ and (1) E Ra E/,(() E Ra (1) E Raa
To shorten the embeddings of entire sequence, follow Devlin et al. (2018) by prepending special token [CLS] to the start of input sequence for aggregating information based on the fact that all tokens attend each other. Due the fact that the first token informs about the whole sequence, we only keep the first element FT [f) Ha of text transformer's 's output F. (t) E S:
Image-Gr Text Generation Module Conditioned on visual features F (t), the module it- eratively generates text tokens until the full sentence with maximum length S₁ is produced.
22

---

Published as a conference paper at ICLR 2025
Q-Former
Contrastive Learning Loss
Image feature
Text
Other texts
newr
ITAM (Image Transformer)
(r)
Text Transformer
(r)
Feed-forward
T'eddffowwrd
Z (z)
H (1)
Cross-attention
Self-attention
QQ(()
E
Self-artention
Tokenization and Embedding
(t)))
Learned queries
Text: |CI SJ" mam is lying in a ronnt with smtokes."
Figure 8: ITAM is the image transformer in Q-Former Li et al 2023), it identifies the local features through aligning the visual features from image transtormer with textual features from text trans- former.
Image-grounded text generation module
Iteration $,008,0)
tr)
Sequence Expansion
S,1() 8,(() Sy.it)
Best candidate selection
Image-Grounded Text Generation Module
Masked Self-attention
Probabildics of nesttg lend
X- X-(E, (0, (O) -...,,E,, [0]
Visual features from
Food-forward Softmax
Q Q= XW K XW -W
Zeal0
(r)..... F,,, (r)E,, (r)...))....
Masked Self-attention
Details
...
0
(()
E0 F2 (7)() Ei(()
Token
QK
Fly (r)
0
E [0== E, E
Concatenation
Soft
H
(()
0
Tokenization Embedding
E
0
(8,() S,(() S,
Mask
Input sequence from last iteration
Update new sequerace with (+)
Output after final iteration: "A lynng swakes."
Figure 9: Structure of the module for image-grounded text generation, the module is part of the Q-Former Li Lt al. 2023)
Following Radford et al. (2018) Devlin et al. (2018) where token "[BOS]" signals the start of text generation, we initialize the sentence to be "[BOS] following by St zero placeholders. each iteration, new text token is generated and replaces one zero placeholder, as is shown in Fig. 9.
In the I th iteration, the input sequence has previously generated tokens (t), S (t), (t) followed by St & zero placeholders, producing embeddings Ef,s, (1). The visual embeddings are concatenated with text tokens, producing X F (t), (t) ..., F Fl,N, (t), (t), layer. As is shown in Fig. 9, the mask in self attention layer enables visual tokens attend each other. and facilitates each of the S text tokens attend to all visual tokens and earlier text tokens. Specifically, provided query, key and values Q XWQ, K XWK and V XWV with WQ, WK and W being learnable weights, self-attention is implemented by
E (1), E (1)
EY,s, (t)]² the input self-attention
QKT
z (t) = Softmarl
+ +M)
(15)
HHa/h
where the values mask M are shown by black and white rectangles Fig. 9. = 12 denotes
+s, (t))T E &(N++ Only the last token the number of heads. (t) = (t), ZT, NT,N,+s,,
Z Z.NN (1) is fed into feed-forward layer because the last token is informative about the complete
23

---

Published as a conference paper at ICLR 2025
Table 4: Performance (AUC, %) on the benchmarks. ST, Ave, UB, Ped2 and NWPU represent Shanghai'Tcch CUHK Avenue, Ubnormal, UCSD Ped2 and NWPU Campus, respectively. Macro- AUC and AUiss & Hoshen ) are evaluated.
Algorithm
ST
Ave
UB
Ped2
NWPU
Ours with VLM based
92.7 / 88.9
94.9/94.5
86.8 81.5
99.8 / 19.1
73.5/71.6
detector
Ours with VLM based
92.3 /8.44
93.6/93.2
86.6 81.2
99.7 99.0
73.0/712
detector (w/o box expansion)
Ours with Yolo detector
92.8/890
/4.9/94.5
86.9 81.5
99.8 / 19.1
73.7
Wang et al. (2023)
Ours with Yolo detector
02.3/88.5
93.7 /3..3
86.6 81.3
99.7 / 199.0
73.0/71.2
(w/o box expansion)
Sliding windows
80.8 77.6
80.5 79.9
75.4 72.9
88.6/ 87.8
66.5 64.4
Ours with VLM based
/
92.8
85.1 80.4
99.0 /9.6
72.0/70.1
detector, RM with 5 layers
Ours with VLM based detector, RM with 9 layers
/7.3
92.9 /92.3
85.4/ 80.6
99.4 / 998.
72.2 70.3
sequence. The feed-forward layer has Ha input channels and N Noocabutary output channels, produc- ing N Neocobulary = 30, 523 probabilities indicating the likelihood of candidate tokens. Neocabulary is vocabulary size, according to BERT tokenizer Devlin al. 2018)) The best candidate S4,4+1 is appended to the end of sequence (t), (f), S before beginning the next iteration. The iterations terminate upon generating the whole sequence (t( with length Si. This module is trained with cross-entropy loss.
D ABLATION STUDY ON THE METHOD FOR OBJECT DETECTION
Table 4 shows the comparison between using VLM Bai et al. (2023), YOLOWang et al. (2023) and sliding windows for object detection. Specifically, window sizes are fixed as follows: 224 for ShanghaiTech. 320 for CUHK Avenue, 320 for Ubnormal, 60 for UCSD Ped2, and 224 for NWPU Campus, with the aim of including largest objects. The results indicate that effective object detection crucial for accurate performance. Furthermore, the comparisons between the settings with and without bounding box expansions show that bounding box expansions contribute to capturing more contextual information, benefiting performance.
E ABLATION STUDY ON ITAM'S STRUCTURE
Table shows the influences ITAM's structures and training data on performance. Setting is the default setting with "Str. 1" and "D. 1". "Str. 1" denotes the structure Li ii et al. (2023) shown in Section 3.2 and "D. 1" denotes the training data of BLIP-2 [i et al. (2023). In "Str. 1". the image transformer for feature extraction has 6 transformer layers each of which includes one self-attention layer, one cross-attention layer and one feed-forward layer. Both of the self-attention layer and the cross-altention layer have 12 heads.
"Str. 2", the numbers of heads are changed to 6 with other settings fixed. In "Str. 3", the number of sequential transformer layers is changed to 3 with other hyperparameters unchanged. "D. 2" refers to the configuration where ITAM trained on the training set of anomaly detection benchmark in each experiment. These training sets include only normal events. The captioning labels on benchmarks' training data are generated by running the pre-trained BLIP-2 model Li al. (2023) on the normal videos. It can be seen that the structure and data variations do not significantly influence performance as long as image-text alignment is conducted. More importantly, ITAM can be trained using normal data and detect unseen anomalies.
Setting and show that the number of image tokens N, does not significantly influence perfor- mance. Setting 7 shows that if SMM is trained using the captioning labels from dataset Lin et al. (2014) and without requiring Qwen-Chat, performance is not influenced. For instance, if the cap- tioning label of an image "The man is running" which prompts SMM output "yes", then
24

---

Published as a conference paper at ICLR 2025
Table 5: Ablations of ITAM's structure using MiCro-AUC. TME and TSG are short for Temporal Motion Estimation and Temporal Sentence Generation, respectively. Ng is the number of image tokens.
Setting
Stage
Stage
Ng
TME
TSG
N NSMM
ST
Ave
UB
Ped2
Str. 1. D. 1
32
3
88.9
94.5
81.5
99.1
Str. I, D.
32
v
3
88.9
94.5
81.5
99.1
3
Str. 2, D.
32
3
88.6
94.1
81.3
99.1
Str. 3, D.
32
3
88.7
94.4
81.2
99.1
Str. I, D.
64
3
88.8
94.5
81.5
99.1
Str. 1, D.
V
128
V
3
88.9
94.6
81.5
99.1
SMM
Str. 1, D.
32
V
3
88.9
94.5
81.5
99.1
w/o Qwen
randomly sample another sentence with different meaning, such as "The man is fighting"" which causes SMM to output "no". Implementations are based on NLTK library Hardeniya al. (2016).
F ABLATION STUDY ON THE NUMBER OF LAYERS IN RM
Table 4 compares the performance of our RM with layers configurations with layers and layers, respectively. It can be seen that 7 is a better choice.
G PROCEDURES FOR GENERATING QUESTIONS IN TSGM
As is shown in Fig. 2(b), TSGM firstly converts the declarative sentence "The man pushing a stroller the street." interrogative sentence "Is the pushing strollor the street?" The conversion is based on nltk library Hardeniya et al. (016) and the procedures are shown in Algorithm 2:
Algorithm Algorithm for Converting Declarative Sentences Interrogative Sentences
'The man is pushing stroller the street.' 2: Tokenization: D + l'The", 'man', ; 'is', 'pushing', 'a', 'stroller', 'on', 'the', 'street', '.']
1: Input sentence: D <- S/
3: Locate first verb: D firstuenb 'is"
4: Divide sentence using first verb: D + D1 + D Ditretverb + D2, D₃ 'The man', D2 'pushing a stroller on the street'
5: Change the order of parts: Q + D firstverb + D1 + D2
6: return Q
H OPERATIONAL EFFICIENCY
All experiments conducted an NVIDIA A100 GPU and an Intel(R) Xeon(R) Gold 6248R CPU, For object detection, we have employed both YOLOv7 detector Wang et al. 2023) and another detector based on Qwen-VL. VL-7B Bai al. (2023. As shown by Table 4, both detectors achieve similar accuracy. In terms of inference speed, the YOLO detector Wang et al. (2023) processes each frame in 1.5 milliseconds, whereas Qwen-VI.. VI.-7B Bai et al. (2023) requires 5.2 seconds per frame. Consequently, YOLO detector. The inference times of all components the proposed framework are measured and summarized in Table 6. With all components considered, the proposed method achieves average frame rate of 12 FPS with an average of 5 objects per frame. The average number of 5 objects is based the findings of Wang al. (2022a).
evaluate the operational efficiency of the proposed framework using the
In the future, we aim to explore the methods that utilize fixed number of bounding boxes per frame maintain constant inference time, even with increased number of objects. Additionally,
25

---

Published as a conference paper at ICLR 2025
Orijallang
Lad patma
Loc paume
Origiad Fares
Local parte
Oui leage
Lecd paturns
Crnmed reptoer
Gomorated caplinnc
Generand caplone
rindin
Oriaa leag
Logal pibres
Oridra loury
Local puthrns
Orgbalinay
Land patumn
Origid Boses
Lesul BITIER
Gourrtdd rapline
Gonerate capline
capelem:
tapline
stadity
shiing
Originl
Lord palkres
Lorel oelores
Oriainling
Lard palemo
Orighed
Gurrrad repien
Gunertd repiio
Gurrand repiial
valligg Bibbigggg unlog sabon.
Arania Miluiiggigg ara undig palion.
valking lagginng artu oedoes
Armilt
Orgaelingy
Loed piltens
O image
Loud piteras
Originn laneg
Land pattons
Origad langg
Laed pithres
Gieeeted replimm
oppiien
srvakka agligg awaand oelkes
Aranie wibhndlggg ana ugg zadion.
Figure 10: Visualization of the heatmaps of local patterns under occlusions, viewpoint changes and low resolutions.
will investigate parsing multiple objects within a single bounding box to maintain fixed number of bounding boxes per frame.
Table 6: Runtime and memory consumption of different modules in the proposed framework, run- time is measured in milliscconds(ms). Inference is conducted with batch size 256.
Object detection (YOLO)
Backbone and ITAM
TSGM (SMM)
TSGM (IGTG)
Modules
CMAM
TME
RM
Runtime
2.5
10.1
1.5
5.9
0.0078
2.6
0.28
GPU Memory
2.78
18.88
0.63
13.54
0.04
0.0
0.55
(Gigabytes)
Table 7 compares the proposed approach with baseline LLM-based AnomalyRuler Yang et al. (024a). AnomalyRuler involves a VLM Processing stage with CogVLM-177 and LLM Rea- soning stage with GPT-4, consuming 192.56 ms and 504.79 ms per frame on NVIDIA A100 GPU, respectively.
Table 7: Comparison between the proposed method and LLM- anomaly detector Yang et al. 2024a). Runtime measured milliseconds(mss) performance measured in AUC (%).
Performance on Shanghaitech
Performance on Avenue
Methods
Runtime per frame
Ours AnomalyRuler Yang et al. (2024a)
83.95
88.9
94.5
697.35
85.2
89.7
26

---

Published as a conference paper at ICLR 2025
1 VISUALIZATION OF LOCAL PATTERNS UNDER OCCLUSIONS AND VIEWPOINT CHANGES
In real-world surveillance videos, occlusions, viewpoint variations and low-resolution conditions common, Fig. 10 shows some examples of the local patterns identified by image-text alignment and cross-modality attention. The local patterns capture semantically meaningful features such as body joints which are consistent across the variations. The compact representations ignore redundant details and contribute to generalizable embeddings.
FUTURE WORK
One limitation of the current framework is the reliance on object detectors. Currently, the perfor- mance of current Vision-Language Models (VLMs) is limited by their fields of view. For example, when processing an image with large scene, a vision-language model tends to overlook many details, highlighting the necessity of object detectors that facilitate the processing of local regions independently. Table 4 shows that object detectors significantly outperform sliding windows, the poor performance of the latter may result from incorrect strategy. As a result, efficient and effective ways to parse events in complex scenes and images with large fields of view where many objects reside. Specifically, end-to-end large model. In simpler scenes with fewer objects, an input image is embedded with fewer vision tokens. As scenes become more complex, more objects are involved, then an input image is encoded with an increased number of vision tokens each of which describes one or more objects. Besides, we will explore ways improve efficiency.
will try more
will explore the integration of object detectors in
27


Reviews:
[{"review_id": "ZsJ0DjI1Ak", "reviewer": "Reviewer_Yam7", "summary": "The paper introduces a novel framework for video anomaly detection, aiming to improve generalization for detecting new, unseen anomalies by focusing on local patterns rather than global event patterns. Traditional video anomaly detection (VAD) methods often struggle with unseen anomalies, as they primarily analyze global patterns. This framework utilizes image-text alignment and cross-modality attention to identify and refine local patterns while enhancing them with temporal information. Core components include the Image-Text Alignment Module (ITAM), Cross-Modality Attention Module (CMAM), and State Machine Module (SMM). The proposed approach demonstrates superior performance on several benchmark datasets, suggesting it can generalize better to novel anomalies.", "questions": "* How to further improve the generalization of local patterns without relying on visual-linguistic models?\n\n* How does the method ensure adaptability to low-resolution videos in different datasets and real-world application scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* By using image-text alignment and cross-modality attention, this method successfully extracts local patterns that remain consistent across varying visual data, enhancing its ability to detect novel anomalies.\n\n* The State Machine Module (SMM) and motion estimation integrate temporal clues, effectively strengthening the detection capabilities by including sequential information for more accurate anomaly detection.\n\n*By combining visual and textual features in identifying local patterns, the model benefits from enhanced robustness and accuracy across different visual domains.", "weaknesses": "* The method relies on the detection effect of visual-linguistic modeling (VLM), whereas multi-object image processing may ignore contextual information and affect performance. The authors need to provide more analysis on the ablation of the foundational models.\n\n* The need for multiple layers of modules (e.g., ITAM, CMAM, SMM) to work jointly results in a complex training process that consumes more time and resources. Please provide a comparison of the spatio-temporal complexity analysis with previous methods to demonstrate the practical effectiveness of the method.\n\n* In low-resolution scenes, the generated text description loses detail information, which affects the anomaly detection effect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVE8gj79KH", "reviewer": "Reviewer_XFDv", "summary": "This paper proposes a novel framework for video anomaly detection (VAD) that focuses on identifying local patterns to better generalize to unseen anomalies.  The framework employs a two-stage process: first, it uses image-text alignment to locate local patterns that are consistent across visual data variances; second, it refines these patterns using cross-modality attention.  To further enhance the model, the authors introduce temporal clues through a State Machine Module (SMM) and temporal motion estimation.", "questions": "The paper mentions limitations related to the reliance on VLM-based object detectors. How can this limitation be addressed in future work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed two-stage framework for identifying local patterns is novel and well-motivated. The use of image-text alignment and cross-modality attention is interesting and potentially useful.", "weaknesses": "1) The paper lacks a clear discussion of the computational complexity of the proposed framework. Given the use of large language models (LLMs) and other complex modules, it is important to address the efficiency of the approach.\n2) What's the role of State Machine Module (SMM) in temporal sentence generation, there need more detailed explanation of the SMM and its role.\n3) How does the proposed method handle situations with significant occlusions or viewpoint changes, which are common in real-world surveillance videos?\n4) The two-stage process for extracting spatial local patterns using image-text alignment and cross-modality attention is not explained in enough detail. The paper lacks a clear, step-by-step explanation of how these complex processes work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n6jR9f5CMn", "reviewer": "Reviewer_qaRU", "summary": "This paper introduces a novel framework for video anomaly detection (VAD) that prioritizes identifying local patterns over conventional global patterns. The authors contend that local patterns generalize better to novel anomalies that were not encountered during training. Their proposed approach follows a two-stage process involving image-text alignment and cross-modality attention to efficiently capture and model local patterns. Additionally, the framework includes a State Machine Module (SMM) to integrate temporal dynamics, enabling enhanced anomaly detection by leveraging both spatial and temporal cues. Experimental results show that this approach achieves state-of-the-art performance on well-established benchmark datasets.", "questions": "1. The Global and Local Pattern representations in Figure 1 are hand-drawn, which limits their reliability. Are there any real feature visualization images available instead? Using actual visualizations could better illustrate the motivation and effectiveness of the proposed method, particularly in showing whether it yields more distinguishable local patterns. Figure 1 alone does not provide enough information to convey the method’s motivation and impact.\n2. Utilizing Qwen for cropping bounding box regions based on prompts could significantly impact efficiency.\n3. Is the introduction of the Qwen-Chat model the primary source of performance improvement? My concern is that the proposed method incorporates numerous external models, and it remains unclear whether these additions are the main contributors to the observed performance gains.\n4. Could smaller models be used to replace these large multimodal models? If so, would this result in a significant decrease in performance?\n5. Could you provide statistical results on runtime and efficiency? Does the proposed method have a significant impact on operational efficiency?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed framework is well-structured, with thoughtfully implemented methods.  \n2. Experimental results confirm that the approach achieves state-of-the-art performance on established benchmark datasets for video anomaly detection.  \n3. The method focuses on fine-grained anomaly features and employs text-image alignment to effectively capture local patterns.  \n4. It incorporates Long-range Memory technology, specifically HiPPO, into video anomaly detection.", "weaknesses": "1. The core of the proposed SMM module comes from works like HiPPO, so it seems that the proposed SMM is directly applying these modules to the VAD task. \n2. Both the Image-Text Alignment Module and Cross-Modality Attention Module are based on pre-existing techniques, which limits the methodological innovation.\n3. Is the observed performance improvement attributed to the additional large vision-language models, such as Qwen-VL and BLIP2? The comparison may not be entirely fair. It would be beneficial if the authors could provide evidence or experimental results to clarify whether these powerful external models are the primary contributors to the performance gains.\n4. The motivation for the work lacks clarity. How do the image-text alignment and cross-modality attention modules achieve “identification of local patterns that are consistent across domains and generalize well”? Additionally, how do they contribute to “generalizing model representations to novel anomalies”?\n5. Certain claims may require further validation, such as the statement: “the complementary relation between visual and textual features remains underexplored.”\n6. The paper lacks runtime and efficiency analysis. The code introduction is incomplete, and several experimental details are missing, such as the specific version and scale of Qwen-VL used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DcJryKSuyE", "reviewer": "Reviewer_mbj7", "summary": "This article is about video anomaly detection (VAD). This paper proposes a framework for recognizing local patterns, which can be generalized to new samples and dynamic modeling of local patterns. This paper proposes image-text alignment and cross-modal attention. Generalizable representations are built by focusing on textual information features that filter out unnecessary differences in visual data. In addition, time motion estimation complements spatial local models to detect anomalies characterized by new spatial distributions or unique dynamics. A large number of experiments have verified the effectiveness.", "questions": "My understanding of this field is unprofessional. So I will further follow the opinions of other reviewers.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The two-stage training method of this paper is reasonable. And allow for more fine-grained local features.\n2. This paper gives a lot of visualizations to make it easier to understand the specific content.\n3. The paper has achieved good performance, and the ablation experiment is given.", "weaknesses": "1. Not giving motivation for each part of the method. In my opinion, a good paper should give a specific reason and then introduce the method.\n2. The efficiency of the model is worth discussing. You have proposed a lot of model modules. How much more reasoning time will they add to the network?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)