SemVAD: Fusing Semantic and Vision Features for Weakly Supervised Video Anomaly Detection
Authors: Hamza Karim
Yasin Yilmaz
Venue: Accepted by TMLR ()

Abstract: In recent years, vision-language models such as CLIP and VideoLLaMA have demonstrated
the ability to express visual data in semantically rich textual representations, making them
highly effective for downstream tasks. Given their cross-modal semantic representation
power, leveraging such models for video anomaly detection (VAD) holds significant promise.
In this work, we introduce Semantic VAD (SemVAD), a novel methodology for weakly super-
vised video anomaly detection (wVAD) that effectively fuses visual and semantic features
obtained from pretrained vision-language models, specifically VideoLLaMA 3 and CLIP.
Our approach enhances performance and explainability in anomaly detection. Additionally,
we analyze the sensitivity of recent state-of-the-art models to randomness in training initial-
ization and introduce a more comprehensive evaluation framework to assess their robustness
to small changes in training such as the seed of random number generator. This framework
aims to provide a more rigorous and holistic assessment of model performance, ensuring a
deeper understanding of their reliability and reproducibility in wVAD.

Published in Transactions on Machine Learning Research (02/2026)
SemVAD: Fusing Semantic and Vision Features for Weakly Supervised Video Anomaly Detection
Hamza Karim
hamzakarim@Susf.e
Deportment of Electrical Engineering University of South Florida
Yasin Yilmaz
yasing Drssf. edu
Department of Electrical Engineering University of South Florida
Reviewed on OpenReview: openrrviee.... net/ forum? id= (d=
Abstract
In recent years, vision-language models such CLIP and VideoLLaMA have demonstrated the ability to express visual data in semantically rich textual representations, making them highly effective for downstream tasks. Given their cross-modal semantic representation power, leveraging such models for video anomaly detection (VAD) holds significant promise. In this work, we introduce Semantic VAD (SemVAD). a novel methodology for weakly super- vised video anomaly detection (wVAD) that effectively fuses visual and semantic features obtained from pretrained vision- languag models, specifically VideoLLaMA 3 and CLIP. Our approach enhances performance and explainability in anomaly detection. Additionally, analyze the sensitivity of recent stato-of-tho-art models to randomness in training initial- ization and introduce more comprehensive evaluation framework to assess their robust.ness to small changes in training such as the seed of random number generator. This framework aims to provide more rigorous and holistic assessment of model performance, ensuring a deeper understanding of their reliability and reproducibility in wVAD.
1 Introduction
In recent years, weakly supervised anomaly detection (wVAD) has gained increasing attention due to its potential applications automated surveillance and content moderation systems. The majority of existing research has focused on contrastive learning- based approaches, where video content is segmented and trans- formed into visual embeddings, followed by training neural network using multiple-instance learning (MIL) (Sultani et al.] 2018 Some recent studies have explored alternative strategies, including pseudo-labeling instead of MIL (Karim et al. 2024). However, MIL-based methods have remained the dominant approach in the field, demonstrating their effectiveness in handling weakly labeled video anomaly detection tasks.
MTI.-based methods exhibit a sensitivity drawback, where model convergence highly dependent on weight initialization. This issuc arises from the inherent characteristics of the MIL framework, namely the random selection of video clips Consequently, model performance may vary considerably depending on the choice of the random seed.
Recently, pretrained vision- language models (VLMs) have gained significant attention due to their ability to learn rich visual representations with semantic understanding (Radford et al., (021). (Zhang el. al., 2023 (Cheng et al.) 2024), (Zhang et al. 2025). CLIP CLIP Radford et al.] 2021), has gained significance in recent wVAD literature (Wu et al. (2024) et al. 2023) and has been primarily used as backbone for feature extraction due its strong visual representation. Although CLIP is trained on image-text pairs, rescarchers have introduced ura encoding techniques to incorporate temporal dependencies, such Temporal Self-

---

Published in Transactions on Machine Learning Research (02/2026)
Attention (TSA) Puu et al. 2024 and the Local-Global Temporal Adapter (LGT-Adapter) (Wu et al, 2024).
However, recent advancements in vision-language models have led to the development of video-language models that can directly process and understand video content. In this study, we aim to adopt the MIL framework in conjunction with VLMs to generate descriptive annotations for anomalies occurring in video streams and utilize these descriptions to enhance the performance and explainability in wVAD. Our key contributions are as follows:
1. We propose Semantic VAD (Sem VAD) featuring a novel feature fusion architecture that integrates semantic and visual features extracted by video-language models (Section 30, which distinguishes
our method from the existing methods that use LLM//VLM for unsupervised VAD (Yang et al.
2024; Zanella et al. 2024) or weakly supervised (Pu et al. 2024 Joo et al, 2023 Lv et al. 2023
Wu et al. 2024) VAD.
2. We introduce an evaluation criterion for assessing the robustness of wVAD systems to randomness in training initialization (Section 4.2
3. We demonstrate improved performance and explainability over recent state-of-the-art methods in both coarse-grained and fine-grained anomaly detection tasks (Sections 4.4 4.5
2 Related Work
2.1 Weakly supervised video anomaly detection
Sultani et al. (Sulaani et al. 2018) introduced the Multiple Instance Learning (MIL) framework, laying the groundwork for weakly supervised video anomaly detection (wVAD). MIST followed the MIL framework and used an encoder-based method that fine-tunes a feature encoder based on the generated pseudo-labels. Subsequent work improved performance via temporal aggregation. RTFM (Tian et al. 2021 proposed a Multi-Scale--ennnn Temporal Network (MTN) to better aggregate temporal features. Later methods like MGFN (Chen et al. 2023) and S3R (Wu et al., 2022) incorporated dictionary and feature magnitude learning to advance WVAD. Although alternative approaches have been explored, such as the use of k-NN distances generate pseudo-labels (Karim al. 2024), MIL remains a prevalent choice due to the effectiveness of its aggregation techniques.
Recently, large language models have motivated LLM based anomaly detection, especially in unsupervised VAD, focusing on model reasoning (Yang et al. 2024) and training-free free approaches. LAVAD (Zanella et al. 2024) proposes a training-free, fully unsupervised pipeline that converts vidco into language and lets an LLM do the temporal reasoning. Frames are captioned by a VLM; captions are cleaned with cross-modal similarity; then prompting aggregates temporal evidence and yields anomaly scores- no fine-tuning or domain data required. While the strengths are simplicity and zero annotation cost, limitations include sensitivity to caption noise and reliance on general-purpose captioners/LLMe LLMMs which can miss subtle anomalies bias toward "textually salient" events. (olmes-VAD) (Zhang et. al.] 2024) is another LLM based model that proposed to train a Multi-Modal LLM on VAD-Instruet500, a large instruction-tuninn corpus built with semi-automatic single-frame annotations, through human effort and LLM-generated explanations.
Prompt-learning remain popular wVD (Pu et al. 2024 Joo et al. 2023 L et al. 2023 Wu et al. 2024
CLIP-TSA (Joo et al. 2023) used CLIP features and temporal sclf-attention to model short- and long- dependencies. UMIL Lv et al., 2023) introduced an Unbiased MIL framework leveraging CLIP features to reduce bias and improve wVAD. Pel4VAD et al. 2024) introduced prompt learning to discriminate between nominal and anomalous segments. VADClip (Wu et al. 2024) recently introduced the Local-Global Temporal Adapter (LGT-Adapter), combining windowed self-attention with lightweight GCNs to capture fine-grained and long-range temporal dependencies, significantly boosting detection performance.
None of the existing ALM/VLM-basse methods for unsupervised VAD Yang et al.] 2024 Zanella et al.
2024) or weakly supervised (Pu al.) 2024 Joo et al.) 2023: Lv et al. 2023 Wu et al. 2024) VAD fuses visual and semantic features from videos like VAD
2

---

Published in Transactions on Machine Learning Research (02/2026)
2.2 Vision language models
VLMs have become central to bridging visual understanding with natural language reasoning. Among the most foundational works CLIP (Radford al. 202I), which introduced contrastive learning framework that aligus images and text in shared embedding space. Trained on hundreds of millions of image-lext pairs, CLIP enables zero-shot transfer by embedding both modalities using separate encoders and learning to maximize their similarity for matching pairs. Its gencralizability across various downstream tasks, such as image retrieval, classification, and video understanding, has made it cornerstone of multi-modal learning and a powerful feature extractor in weakly supervised setups.
More recent models have expanded this paradigm into temporally complex and generative domains. Sora [Luu et al. 2024) shifts the focus from understanding to generation by producing high-fidelity videos directly from text prompts. Through diffusion-based modeling, Sora can generate coherent and realistic video se- quences, marking leap in cross-modal synthesis and showcasing how language condition fine-grained spatiotemporal outputs. Mcanwhile, Video-LLaMA (Zhang et al., 2023), ( et al. 2025 integrates large language models with video encoders to enable multimodal reasoning over sequential visual data. Tt builds on instruction-tuned LLMs and pretrained vision backbones to handle tasks like video question answering and temporal captioning, combining perception with structured language reasoning.
3 Method
3.1 Problem Definition and Method Overview
Viston Fasture Extrstor
Vision Fosturts
fre sticko
pARAar porson, standing
VAD
Ansmaty
parson, standing
Caption Feskoos
2
chridmas, Uve
modill
fire smoke
Coestro
Segment Index
Figure 1: Overview of the SemVAD architecture, which leverages captions generated by a multi-modal language model through fusion with extracted vision fcatures and outputs anomaly score for the given anomaly classes.
SemVAD introduces a video anomaly detection (VAD) architecture designed to harness the capabilities of recent multimodal language models by integrating them with established vision feature extractors such as CLIP, 13D, and Uniformer. The overall architectural pipeline of SemVAD depicted in Figure
Let V denote the set of all videos, where each video V V may contain either normal content anomaly belonging to one of M predefined anomaly classes. Each video V is divided into N non-overlapping clips, denoted as {1a, uN!}. The VVD setting assumes that only video-level labels are available during training. Specifically, given a video V, if all frames are free of abnormal events, the video is labeled as normal y = Conversely,
least one frame contains an anomaly, the video labeled as abnormal y 1. The objective of a wVAD system learn a model capable of predicting framo-lovel anomaly scores for coarse-grained (normal anomalous) and fine-grained (anomaly classes) decisions, despite being trained solely video-level annotations.
In this work, we semantically enhance the CLIP Radford et al. 2021 (visual) features used in VADCLIP (Wu el al. 2024) by integrating a caption C, for each clip o; obtained by the multi-modal language model, VideoLLaMA 3 and postprocessing. novel two-phase fusion module consisting of attention and convolu- tion mixer steps is proposed for scamless integration. We further enhance the model's temporal reasoning and anomaly localization capabilities by training the model with a novel semantic-alignment loss function in addition to the Local-Global Temporal (T.GT) Adapter and the drancbranch MIL-align framework of VAD-
3

---

Published in Transactions on Machine Learning Research (02/2026)
2/4
Abuse
tp
Vision Encoder
Arrest
Burglary
eecoder
Vandalism
Lu
fz
Attention feature
fusion
Convolution Mixer
concat
CLIP Text Encoder
F fused
L
Le
t (m)
LGT Adapter
MIL align
Classifier
SemVAD
p
p
T
Return training
Class Specific Prompt Extraction
L= Logpp) + (p((P (t,) leem (mm) t4 P.fv)
Figure 2: Overview of the SemVAD architecture, which builds upon the VADCLIP (Wu al. 2024 frame- work by incorporating semantically rich captions enhance visual representations. Descriptive captions, generated via VideoLLaMA 3, fused with CLIP visual features through two-phase fusion module comprising an attention mechanism and a convolutional mixer. Additionally, SemVAD introduces novel class-specific alignment loss &eem(ta (t,, (. which encourages caption embeddings to align with class- specific prompts z(m). Components highlighted in blue represent the original VADCLIP pipeline while red modules denote the proposed contributions in SemVAD.
CLIP. A detailed overview of the proposed Semantic VAD (SemVAD) method is shown in Figure 2 with novel modules and VADCLIP modules highlighted in red and blue, respectively.
3.2
Extraction of semantic features using VideoLLaMA
For each clip & in video V, we generate caption C, to describe the content of the clip. To incorporate temporal context, a window of three consocutive clips 04-1, Ub 0,41 passed to VidcoLLaMA with the following prompt:
"You are given three consocutive clips. Describe what is happening in the middle clip using the preceding and the following clip as context."
The three-clip window is slid across the sequence of N clips the video, with padding of one clip at the beginning and end as ncoded. Each caption is then embodded into a numerical vector T, E RQ using the CLIP lext encoder, where Q is the dimension of the feature vector. We also incorporate a system-level prompt provided in appendix A.1
During training, an important goal is to encourage captions that are likely to contain anomalics to be pushed towards their corresponding class labels. To this end, we first derive a generalized caption representation for each anomaly class. The generalized caption for each class serves as a prototype or centroid which captures the key semantic features that characterize anomalics within that class. Let m represent the normal class and m 1,..., M represent the anomaly classes. Figure Figure3showa shows the complete pipeline obtain generalized caption ((m) for each class from the generated captions Cj. First, the set of class-specific captions ((m) is
4

---

Published in Transactions on Machine Learning Research (02/2026)
T;
C video shows person Fire erupting... ....
(
Video Liama
stopwords and repealing words each instance
Text Encoder
Person seèn
Abuse
Arrest
Normal
(m)
Abusive, video, person pushing....
Cops, video, person, road
store, browsing.cm cars, Traffic
Class Specific Prompt Extraction
Abuse
Normal
count(w, ((mm)) c(m)
count(w, clo))
Abusive, beating, pushing, crib...
Cops, arrest, handculfs, pinned
walk, browsing, people transaction......
(m(w))
(o(ws)
count(u, ((o)
count ur, Mr C(m)
Rank based cosine similarity
Select top words
Abuse
Arrest
Normal
Crib,
Cops, broad-way, surround, pinned
browsing. people, transaction......
similarity between each class and all words (except fot m=0)
push...
based on (m)(w)
Select top
Figure 3: Class-Specifee Prompt Extraction Pipeline: Captions C₁ are generated for each clip ey using VideoLLaMA 3 model. After filtering, words are ranked by class-specific scores Pn(w), and top words are embedded via CL.PP For anomaly classes, words are re-ranked using cosine similarity; for the normal class, top words are selectod directly. Final class prompts are encoded obtain CLIP text fcatures T,
formed from all clips that belong to videos labeled with the m-th anomaly class. Similarly, let C()) represent the set of captions from all clips vidoos labcled as normal.
For each unique word w appearing the caption corpus, we define the class-specific score P.n. (w)) as follows:
count(w, C(m)) Pm Pm(w)= count (()) + m > 0; Po(w) count w, U C(m) +
count (w C(D))
(1)
where count(w, ((((m)) denotes the number of times the word w appears in the set of captions ((m), and e is small constant to avoid division by zero.
For each class m, each word w is first ranked according to its score Pm (w). The top x% words with the highest scores are selected to represent salient linguistic features for the m-th anomaly class. Fach selected word is then embodded into a vector representation using the CLIP text encoder. These word embeddings are then compared to the CLIP-encoded representation of the m-th anomaly class label via cosine similarity, Based on this similarity, the words are re-ranked, and the top S most semantically relevant words are selected.
For the normal class, the cosine similarity step is omitted due to the inadequacy of the phrase "normal" as a semantically meaningful or descriptive label in the CLIP embedding space. Instead, the top S words are selected directly based on their Po(w) scores, following the same ranking procedure described earlier. The final set of words for each class m are concatenated using a space delimiter to form the generalized caption ((m), This phrase is then passed through the CLIP text encoder obtain numerical representation (m)) R

---

Published in Transactions on Machine Learning Research (02/2026)
3.3 Vision- Caption Fusion
We propose a two-phase fusion module designed integrate caption features T [T:] of video with visual features F [F:] RND obtained from the CLIP visual encoder. The fusion process begins by aggregating caption features globally through an attention mechanism conditioned the visual features. This is followed by a convolution mixer module that fuses the attended features along the local temporal dimension. This is
similar to local-global aggregation in WSVAD Wu et al. (0022) Wu et al. (2024) Chen et al. (0023)
3.3.1 Attention- Bassdd Fusion
In the first phase, illustrated in Figure 4 the CLIP visual features F and the caption features T for video are passed through separate learnable linear projection layers, denoted L, and Le to obtain projected visual and textual features, R and E NDDD where D is the feature dimensionality and N is the number of clips per video.
Next, cross-modal attention is computed between the projected visual features f and textual features t. The resulting attention scores are used aggregate the textual features based on their similarity with the visual modality. The aggregated caption features are then added to the visual embeddings, followed by layer normalization and feed-forward network. This results in the attention- fuseed feature representation, denoted gatta E N
Attention Feature Fusion
FE NND
ftt vdz
T RND
Le
softmax
LayorNom
FFN
LayerNom
gattn R.D.
Figure 4: Illustration of the attention-based vision-caption fusion module.
In our cross-modal attention setup, we treat vision features f queries and text features keys. Rather than employing the standard QKV attention, found that reusing the keys values (QKK attention) consistently yielded better performance (see Appendix A.4p. While avoiding an additional projection for values, QKK seems to preserve semantic alignment, acting as a form of regularization in the fusion process.
3.3.2 Convolution Mixer
Next, the attention-fused text features gatta are stacked with the projected visual features produce a combined representation Xstack [a.Stack] Nx as shown in Figure 5 A 2D convolutional layer with kernel size of 2 x 3 is applied to this stacked tensor, with both the input and output channels set to D. This convolution operation, which fuses the features across the modality (stack) and temporal dimensions, is followed by the GELU activation function. Necessary padding is applied along the temporal axis to preserve the clip length. The output this operation is denoted as Xcnnv E N which is subsequently passed
Dam 2D
f = goko
STACK
Linor Mouestion Lape
pots g
(atte e RAOD
A e Ra
Frhund RnD
Temporal Convolution Mixer
Figure 5: Illustration of the convolution mixer module.
through linear layer to obtain the comvolution-mixed features pmix [smii]] E NDD

---

Published in Transactions on Machine Learning Research (02/2026)
Finally, the visual features fi and f. concatenated along the feature dimension to produce the fused representation Flused [// E NRNx2D, which is then used as input to the LGT adapter. In Section 4.8 we further explore the effect of each module on the performance of our model. The fused features are then passed through the LGT temporal aggregator before computing the coarsc-grained (binary: normal vs. anomaly) P. and fine-grained (multi-class for also detecting anomaly type) Pfe classification probabilities, as in Wu et al. 2024).
3.4 Loss Function
We build upon the commonly used top-K Multiple Instance Learning (MIL) loss, foz (Wu et al., 2024), to supervise the learning of coarse-grained anomaly scores. In addition, we incorporate the multi-class MIL alignment loss li for fine-grained anomaly scores and the contrastive loss lets for pushing normal class away from anomaly classes, both originally introduced in VADCLIP (Wu et al. 2024). These losses encourage the projected representation t2 of anomaly class labels to diverge from both the normal class embedding and embeddings of other anomaly classes A.5
To further enhance class-peeifiic alignment, we introduce an additional loss term that explicitly aligns the projected caption embedding t, towards its corresponding anomaly class embedding {(m) for top- anoma- lous clips, which obtained by applying the linear caption projection layer Le to the caption embedding T(m), This is achieved by MIL alignment using the fine-grained (multiclass) probability scores pls, thereby reinforcing the semantic consistency between the caplion and its target anomaly class.
Since we know the label m for each video in the training set, in the probability matrix [P E [0, 1JNMM, we select the m-th column to get Sm E [0.
Let fi1...., ikk = arg max K {mm} denote the indices of the top- K clips with the highest probabilities in 8m and tip E ER I,.... K, denote the corresponding feature vectors.
Next, we compute the average cosine distance feem between the semantic embedding f(()) of class m and the semantic embedding ii.- The overall training loss is given by the sum of four loss functions:
fyom = avg ( 1
|1:; || ||e(m]]| = 1, 1,..., K) { e Esem + lag + lig + lets-
( (mm)
4 Experiments
4.1 Datasets
We evaluate and compare the proposed method on two widely used datasets in video anomaly detection: UCF- -Crimr (Sultani et al.) 2018) and XD- Violence (Wu et al. 2020). For consistency and fairness, we utilize only the visual modality and discard any accompanying andio information. Both datasets provide weakly labeled training videos, making them suitable for wVAD.
4.2 Evaluation Metrics
Following prior work, for coarse-grained (normal vs. anomalous) anomaly detection, we use the area under the receiver operating characteristic curve (AUC) for UCF-Crime and average precision (AP) for XD-V -D-Violence to ensure comparability with existing methods. Moreover, for fino-grained (multi-class) anomaly detection,
we use mean average precision (mAP) following Wu et al. (Wu et al.) 2024
While most wVA studies report only the 'best' AUC or AP, our experiments reveal significant performance variability arising from the stochastic nature of training, particularly in contrastive loss scttings like multiple instance learning (MIL). MIseb models are highly sensitive to random initialization and sampling, often resulting in chaotically fluctuating performance (Figure 6b

---

Published in Transactions on Machine Learning Research (02/2026)
At the beginning of training, randomly initialized parameters yield arbitrary anomaly scores for video clips {wa). When the top-K clips are selected for training supervision, there is a substantial risk that they lack true anomalies, leading to optimization toward suboptimal minima. This instability mainly stems from the random initialization of network weights at training start and the stochastic pairing of normal and anomalous segments during MII--based contrastive learning.
Performace Sead
88.5
Perfomaae (AUCS)
87.5
Weight Initialization Seed
Data Seed
AUC (%)
234
234
87.44
9
9
85.61
86.5
234
9
88.24
9
234
86.33
10
20
20
(a) Impacc of weight initialization and data shuffling seeds on VADCLIP (Wu et al., 2024) performance
(b) Chaotic AUC variations SemVAD on UCF-Crime function of weight initialization seed.
(AUC) on UCF-Crime.
Figure 6: Effect of random seeds on VAD performance.
Figure 6alawwss the performance variability of the state-of-the-art VADCLIP (Wu et al. 2024 under different random seeds on UCF-Crime. Performance depends not only on model weight. initialization but also on the stochastic pairing of video segments (Data Sood). While the best AUC was originally reported 88.02% in [Wu et al. 2024), our experimentation with several seed pairs reveal notable deviations and even better AUC (88. 24%) than reported. Figure 6b demonstrates the non-smooth AUC variations of our proposed SemVAD method on UCF-Crime. Wild performance variations (Figure Ga and Figure (b) display the impracticality of optimizing random seed as if it was hyperparameter. Hence, evaluating the performance wVAD methods in terms of best performance as a function of random seed does not provide a fair comparison, where the best achievable performance is likely never known.
To address this, we adopt a more robust evaluation strategy by training each model with ten random seeds and reporting the mean and standard deviation of performance metrics. This provides more reliable and reproducible assessment of model effectiveness and stability.
4.3 Implementation Details
We adopt the VADCLIP pipeline and utilize the CLIP (ViT-B/16) architecture as the backbone. We the percentage x of selected keywords to be 10%, higher values do not increase performance while requiring more compute during the cosine similarity step. shows the impact of top x% keywords on performance when tested on UCF-Crime.
Table 1: Effect of using top x% keywords on performance when tested on UCF-Crime.
x%
10
30
50
Auc %
87.09
87. 37.21
87.22
87.19
The paramoter S, representing the number of top-ranked words usod, determined by the maximum number of characters that can be accommodated within the CLIP text encoder's token limit of 77, following the sorting of words by their cosine similarity to class labels.
8

---

Published in Transactions on Machine Learning Research (02/2026)
Training is carried out on NVIDIA RTX 4090 GPU, using similar hyperparameter settings to original VADCLIP implementation. Specifically, we use learning rate of 2 x 10-8 for both the UCF-Crime and XD- Di-Veeeenc datasets. The batch size is set to 64 for UCF-Crime and 128 for XD-Violence, respectively,
4.4
Results
We evaluate the performance of our proposed method, SemVAD, against four recent state-of-thes Podels-CCIP-TTE (Joo al. 2023), PEL4VAD (Pu al. 2024p. UR-DMU(Zhon al., 2023 and VAD- CLIP Wu et al. 2024) well as the earlier baseline approach by Sultani al. (Deep MIL) (Sultani al,, 2018). For a fair and robust comparison, train each model using 10 different random seeds and report the mean performance along with the standard deviation.
Coarse-grained performance comparison on UCF-Crime and XD- Xi-Vinlence datasets. Mean and standard deviation (SD), and the performance with the known best-performing seed are presented.
Table 2:
XD-VIOLENCE
Method
Mean AUC(%) (SD)
Best Seed AUC(%)
Mean AP(%) (SD)
Best Seed AP (%)
RTFM Tian al. S3R
2021
N/A
85.66
N/A
78.27
Wu et al.
2022
N/A
85.99
N/A
80.47
UMIL et 2023
N/A
86.75
N/A
N/A
Deep MIL Sultani et a.
2018
75.58 (0.77)
77.92
73.24 (1.22)
75.18
UR-DMU Zhou et al. 2023
85.52 (0.744)
86.75
79.22 (1.206)
82.41
CLIP-TS Joo al. 2023
85.02 (0.466)
87.58
79.83 (1.05)
82.17
PELAVAD Pu et al. 2024
85.21 (0.366)
86.76
84.21 (0.31)
85.59
VADCLIP Wu et al.
2024
86.26 (0.918)
88.02
83.20 (1.53)
84.51
VAD
87.21 (0.462)
88.48
84.66 (1.26)
86.55
Coarse-Grained Grained Performance: As shown in Table 2 SemVAD achieves the highest mean AUC among all methods evaluated, outperforming the state-of-the-art VADCLIP by 0.95% AUC while maintaining a relatively low standard deviation. This highlights both the effectiveness and stability of our approach. Notably, SemVAD also outperforms the reported best results of existing methods both datasets. A similar trend is observed for the XD-Violence dataset, where SemVAD achieves the highest mcan AP score of 84.66%, outperforming the most recent state-offthe-art method, PEL4VAD, by 0.45% and surpassing VADCLIP by 1.46%. is also notable that, despite having a lower mean performance, PELAVAD demonstrates the lowest standard deviation.
Fine-grained Performance: We conduct fine-grained evaluations using mean average precision (mAP) across five Intersection-ovvr-Uniioo (IoU) thresholds: 0.1, 0.2, 0.3, 0.4, and 0.5. We report the average mAP across these thresholds, referred to as mean average mAP, and the corresponding standard deviation across 10 different training seeds. Table 3 shows that SemVAD achieves an mean average mAP of 8.1%, outperforming VADCLIP by 1.15% when tested on UCF-CRIME. Similarly, substantial improvement is observed with Violioleeee, with a mean average mAP 31.615% representing a 9.12% increase over VADCLIP. These results demonstrate that the semantic features of SemVAD help even more with fine-grained anomaly detection compared to coarse- coarse-grained anomaly detection.
4.5 Explainability
In Section 4.4 report significant improvement in fine-gre tne-grained anomaly detection performance compared to recent state-of-the-art methods. This can be attributed to the fact that the generated captions C, encapsulate anomaly-specific information. Keywords such as "shooting", "fire, and "pushing" among others, provide contextually relevant. cues to the anomaly detector.
Figure 7 a-c)) shows that captions C, effectively describe normal and anomalous sogments, aiding inter- pretability. Figure dd) illustrates misclassification- "istffattioo""Buglrrrrr'" "Stealing", and 'Robbery"-due se- mantic overlap in captions (e.g., 'attempt", "break"), highlighting both the strengths and limitations of
9

---

Published in Transactions on Machine Learning Research (02/2026)
Fine-get greeegraiedd (multi-class) performance comparisons on UCF-CRIME and XD-VIOLENCE. rep- resents results without averaging over ten soeds as presented in (Wu al. 2024).
Table 3:
mAP 100%
Method
0.1
0.2
0.3
0.4
0.5
Mean Avg mAP %
SD
UCF-
Deep MIL*
5.73
4.41
2.69
1.93
1.44
3.24
N/A
AVVD*
10.27
7.01
6.25
3.42
3.29
6.05
N/A
VADCLIP
12.32
8.91
6,25
4.29
2.99
6.95
1.44
SemVAD
15.49
10.625
7.35
4.27
2.92
8.1
1.19
XD-VIOLENCE
Deep MII.*
22.72
15.57
9.98
6.20
3.78
11.65
N/A
AVVD*
30.51
25.75
20.18
14.83
9.79
20.21
N/A
VADCLIP
33.47
26.89
21.50
16.31
12.0
22.03
1.932
SemVAD
44.41
36.95
29.91
24.34
19.07
31.15
1.29
Arson022
Explosion016
Anomaly Class
"Buiding enake flw!
Anomaly Class
Aous
Anst
Anest
Aromaly Score
0.00
0g "Loilding, docks towee"
"Buae,, stviie, fire"
Gurgary
Burplay
Anomaay
perout dliiiuues skaxling
Rosdiccaows
Acsowry
PosdAcodent
Shooling
Sropniting
Shopltting
300
(a) Segment Index RoacAccidents012
200
100
400
V93
(b) Segmennt Burglary005
Anomaty Class
1.0
Hommal
Angmaty Scnre
Aram
Amomaly Score
Aram
Asseet
Apjest
"car, fliping maahh"
Explasion
Eegl
Rignting
fighting
Co m "burglar attenpt, break In"
Shouting
Shopting
athempe
Vindad
(o)Segment
(d) Segment
Figure 7:
Explainability of fine-grained anomaly detection through VideoLLaMA 3 generated captions. Height denotes the coarse- -rarned anomaly score while the color indicates anomaly type.
caption-based reasoning. However, the distinction between 'Burglary', 'Stealing', and 'Robbery' is natu- rally minimal.
To quantify the explainability performance, we manually annotated short clips from two videos per class in the UCF-Crime dataset, resulting in a total of 26 annotated videos. Captions generated by SemVAD for each video are then compared with the annotations using the cosine similarity score, yielding an average similarity score of 0.77. Table shows some example scores together with the predicted captions human annotations. The annotations and predicted captions are given in the Appendix.
Table 4: VLM Generated Captions Compared to Human annotations
Video id
VLM Generated
Cosine Similarity Score
Human Annotated Label
Robbery056
A.2.1
0.92
A.3.1
Explosion021
A.2.2
0.92
A.3.2
Robbery020
A.2.3
0.42
A.3.3
Shoplifting006
A.2.4
0.45
A.3.4
10

---

Published in Transactions on Machine Learning Research (02/2026)
4.6 Computational Cost
We also assess computational efficiency, as models like LideoL.aMM 3 are resource-intensive. Inference delay varies with caption length; we evaluated performance over 100 16-frame clips, passing through the entire SemVAD pipeline on an RTX 4090 (Table 5).
Table Runtime analysis of SemVAD
Mean time/clip (sec) Standard deviation 1.126
Prames/clip
Estimated frames/second runtime
0.243
16
14.2
We see that on average it takes approximately 1.126 seconds generale an anomaly score for a given clip with a standard deviation of 0.243 seconds, which implies that the detector can process frames rate of 14.2 frames per second average, which indicates it be used real-time in many practical scenarios. We also provide detailed runtime breakdown. Table G shows that the total inference per 16-frame clip is 1. 1.126s on an RTX 4090. Of this, only 0.014s is due to our fusson + detection model, while 1.112s stems from VideoLLaMA-3 caption generation. The overhead therefore comes almost entirely from captioning.
Table 6: Modulo-wise runtime analysis (per 16-frame clip).
Module
Time (s)
Fraction
VideoLLaMA-3 captioning
1.112s
98.8%
Fusion + delection model
0.014s
1.2%
Total
1.126
100%
To contextualize the cost of captioning, we compare it with prior approaches that rely both traditional (RGB + optical flow features from I3D or CLIP visual encoders) and new VLM-based approaches in Table Extracting T3D RGB++low features typically requires 20ms per clip on similar GPUs, while CL.IP visual encoding much faster (15ms/clip). Our captioning cost (1.112s) is thus substantially heavier than CLIP and I3D. However, unlike 1lD+flow and CLIP, captions provide semantic interpretability. Recently, there has been growing interest in using VLMs and LLMs to generate anomaly scores without training, e.g., LAVAD VLM (BLIP2) and LLM (Llama-2) together to generate anomaly scores for videos. As seen in Table 7 LAVAD takes 3.9 seconds, much more than captioning in SemVAD with VideoLLaMa-3.
Zanella et al. (2024) which
Table 7: Runtime comparison across models (per 16-frame clip)
Model
Time (s)
Task
I3D RGB + Flow (two-stream)
20ms
CNN-based vision features
CLIP visual encoder
15ms
Transformer-bast based vision features
SemVAD fusion + detoction
14ms
Vision + caption fusion and detection
Lade captions
1.112s
VLM-based semantic captions
LAVAD
3.9s
Training-free VLM+LLM for detection
presents a comparative evaluation of Video-LLaMA-3-719 and its lightweight counterpart, Video-LLaMA-33 2B. When integrated into the trained SemVAD framework, the 2B model exhibits modest reduction of 0.92% and 0.74% in coarse grained and fine grained performance respectively relative to the 7B variant, while achicving a substantial efficiency gain of approximately 392 ms per inference at substantially lower memory cost. This improvement enables real-time operation at 22 fps, compared to 14 fps with the larger model. These findings highlight that caption generation constitutes the primary computational bottleneck, thereby underscoring the value of lightweight captioning models for deployment resourco-constrrinee settings. Nevertheless, the rapid advancement of hardware and the growing adoption of large language models (L.MMs) in this domain suggest that the use of larger models may soon become practically feasible.
11

---

Published in Transactions on Machine Learning Research (02/2026)
Table Compute speed and memory usage comparison between Video-Ilama-3-77 and Video-Ilama- Ilama-3------------------------- -b
UCF-CRIME Coarse-grained
UCF-CRIME
Time (s)
GPU Memory
Fine-grained
VideoLlama- -a---
86.34
6.68
0.72
5.6 GB
VideoLlama-3-1
87.26
7.42
1.112
17.3 GB
4.7 Transferability
In this section, we evaluate the transferability of models trained on distinct datasets. UCF-Crime represents a real-world anomaly detection dataset consisting primarily of low-resolution, static surveillance foolage across a wide variety of scenarios. In contrast, XD-Violenee is composed largely of clips from movies and films, which typically feature highherresolution, dynamic, and non-static scenes. Due to these substantial differences in data characteristics, models trained on one dataset generally exhibit limited transferability the other.
However, SemVAD aims to mitigate this gap by incorporating semantic captions, thereby leveraging semantic similarities across datasets to enhance generalization. As shown in Table SemVAD achieves superior
Table Transferability analysis between SemVAD and VADCLIP.
Training Data
Testing Data
Training Data
Testing Data
UCF-Crima Violenne
Xio-eeeeence
UCF-Crime
SemVAD
AP: 71.66%
AUC: 84.66%
VADCLIP
AP: 66.66%
AUC: 81.1%
transfcrability compared to the bascline VADCLIP, highlighting the effectiveness incorporating semantic information from video captions in bridging the domain gap between heterogeneous dalasets.
4.8 Ablation Study
In addition to incorporating video captions C into the model, propose three key modules: atten- tion fusion mechanism, convolutional mixer, and a contrastive class-specific loss (Lsem). To evaluate the contribution of each component, we conduct a comprehensive ablation study. Table 10 presents the im- pact of each module on the mean AUC for UCF-Crime, computed over 10 different random seeds. We observe that directly concatcnating the caption fcatures T, with the CLIP vision-encoded fcatures F; yiclds 0.25% improvement over the VADCLIP baseline. Introducing the attention fusion module further enhances performance by 0.142%. The addition of the convolutional mixer results in a substantial gain of 0.438%, underscoring its significant contribution to overall model effectiveness. Finally, incorporating the fyom loss provides an additional performance increase of 0.132%, leading to the highest mean AUC of 87. 212%.
Table 10: Ablation study showing the contribution of each proposed component on UCF-Crime. Performance
is reported as the mean AUC (%) over 10 random seeds.
Mean ROC AUC (%)
Performance
Modules
Increase (%)
Caption (Ct)
Attention Fusion
Convolution Mixer
Eacm
86.50
40.25
86.6422
+0.142
87.08
+0.438
87.219
+0.132
12

---

Published in Transactions on Machine Learning Research (02/2026)
4.8.1 Variance in captions
Our model generates video captions using greedy decoding, which produces deterministic captions for each video. To investigate the impact of caption variability on performance, we conduct. experiments with in- creased randomness by adjusting the temperature and top_p parameters.
Table 11: Impact of caption variability on performance.
UCFCRIME Coarse-grained AUC UCFCRIME Fine-graimed mAP
Temperature Top p= 0.3 Temperature top p 0.7 Greedy Decoding
86.2
5.48
85,85
4.99
87.26
8.1
As shown in Table 11 introducing higher randomness through elevated temperature and top p values results in a noticeable reduction in performance compared to greedy decoding, indicating that deterministic captions are more effective for this task.
5 Limitations and Future Directions
The use of a VLM for streaming video may cause some computational burden, especially for resource- constrained systems. Specifically, VideoLLaMA 3, which is used in the experiments, requires a GPU. Hence, the proposed method is not suitable for standard computer for resource-consurained systems can be promising research direction. However, as discussed in Section 4.5, the proposed method can run near real-time (14.2 fps) on reasonable computer with and NVIDIA RTX 4090 GPU.
odge device, and more efficient methods
In Section 4.2 we address the issue of stochastic variability affecting MIL-based methods, identifying two primary contributing factors: the random pairing of anomalous and nominal videos within each training batch and the random initialization of model weights the onset of training. While a comprehensive inves- tigation into enhancing the robustness of such models warrants dedicated research, one potential approach to mitigating this variability is to limit stochasticity during the early phases of training.
Although it is difficult to fully control the random pairing of normal and anomalous segments, a feasible alternative is to select the top-k segments based on their cosine similarity to nominal features at the beginning of training. This method can potentially mitigate the instability introduced by random weight initialization.
To further address this issue, we incorporate a o-greedy strategy, commonly used in reinforcement learning, For each training batch comprising b samples-consisting of an equal number of anomalous and normal instances (i.e., b/2 each) we introduce control parameter ø, initialized to 1. At each training iteration, o is updated according to the exponential decay rule:
Oneu 0.99 a.
A random value a is then sampled uniformly from the interval [0, 1]. If < the top-k anomalous segments in each auomaly-containing video are selected based on the average cosine distance between the segment and all other nominal segments within the batch. Segments exhibiting the highest cosine distances are considered anomalous. Conversely, if a 2 a, segment selection reverts to the standard MIL procedure, where segments are chosen based on the model's classification scores.
This approach allows the model to rely on cosine similarity during the early stages of training, when model predictions unreliable, and gradually transitions to using its predictions training progresses.
Table 12 presents a performance comparison between the o-greedy approach and standard MIL training on both VeD and the vanilla VADCLIP framework using the UCT-CRIME dataset. The results indicate that the o-greedy strategy contributes to more stable training, as evidenced by a reduction in standard devia- tion, albeit with a slight trade-off the mean performance. Although the pairing of anomalous and nominal videos within each batch remains stochastic, leveraging cosine similarity as a selection griterion-whie not always aptimal-offers additional stability during the training process.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Table 12: Performance comparison between SemVAD and VADCLIP when trained using the o-greedy method.
SemVAD
VADCLIP
Mean AUC (%) (SD)
Mean AUC (%) (SD)
o-groedy
87.08 (0.248)
86.12 (0.694)
Regular
87.21 (0.462)
86.26 (0.917)
6 Conclusion
In this work, we propose SemVAD, a novel framework for weakly supervised video anomaly detection (wVAD) that integrates semantically rich captions generated by a multimodal language model with visual represen- tations extracted from CLIP. By introducing a two-phase fusion architecture, comprising attention-base feature aligument and a convolutional mixer along with a class-specific semantic alignment loss, we demon- strate significant performance gains with a reasonable computational cost.
Our experimental results establish SemVAD sistently surpasses recent benchmarks such as VADCLIP and PEL4VAD, achieving higher coarse-grained detection accuracy and substantially improving fine-gres ane-grained anomaly localization. The integration of seman- tic captions contributes not only to enhanced performance but also to explainability, as natural language descriptions provide interpretable cues about detected anomalies. Furthermore, our transferability study highlights that SemVAD also better equipped to generalize across data domains, as we observe increased transferability of learned knowledge across two distinct datasets.
stato-of-tho-art in wVAD. Specifically, SemVAD con-
The common practice of evaluating performance using the "best" known seeds for random number generators is also criticized. We showed that due to the underlying characteristics of MIL-based training, there is wild fluctuation in the performance of state-of-the-ar methods as function of randomness seed. For a more reliable comparison and more reproducible results, we propose to evaluate the wVAD methods in terms of mean and standard deviation performance over a number of runs with different seeds.
References
Yingxian Chen, Zhengzhe Liu, Baoheng Zhang, Wilton Fok, Xiaojuan Qi, and Yik-Chung Wu. Mgfn: Magnitaude-oonttassivvve glance-and-focus network for weakly-supervised video anomaly delection. In Pro- ceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 387-395, 2023.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifci Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advaneing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint or 2xix:2406.07477 2024.
Hyckang Kevin Joo, Khoa Vo, Kashu Yamazaki, and Ngan Lc. Clip-tsa: Clip-assisted temporal self-attention for weakly-supervised video anomaly detection. In 2023 IEEE International Conference on Image Pro- cessing (ICIP), pages 1230-3234. TEFF, 2023.
Hamza Karim, Keval Doshi, and Yasin Yilmaz Real-time weakly supervised vidoo anomaly detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Compuler Vision, pages 6848 6856, 2024.
Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujic Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: review on background, technology, limitations, and opportunities of large vision models. Xiv preprint or Xix:2202..11177 2024.
Hui Lv. Zhongqi Yuc, Qianru Sun, Bin Luo, Zhen Cui, and Hanwang Zhang. Unbiased multiple instance learning for weakly supervised video anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 8022 8031, 2023.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Yujiang Pu, Xiaoyu Wu, Lulu Yang, and Shengjin Wang. Learning prompt-enhanced context features for weakly-supervised video anomaly detection. IEFF Transactions on Image Processing, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela MishkLnn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748- 8763. PmLR, 2021.
Waqas Sultani, Chen Chen, and Mubarak Shah. Real-world anomaly detection in surveillance videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6479-6488, 2018.
Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W Verjans, and Gustavo Carneiro. supervisu video anomaly detection with robust temporal feature magnitude learning. In Proceed- ings of the IEEE/CVF international conference on compuler vision, pages 4975 4986, 2021.
Jhih-Ciang Wu, He-Yen Hsieh, Ding-Jie Chen, Chiou-Shann Fuh, and Tyng-Luh Liu. Self-supervised sparse representation for video anomaly detection. In European Conference Computer Vision, pages 729-745. Springer, 2022.
Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, and Zhiwei Yang. Not only look, but also listen: Learning multimodal violence detection under weak supervision. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part XXX 16, pages 322 339. Springer, 2020.
Peng Wu, Xuerong Zhou, Guansong Pang, Lingru Zhou, Qingsen Yan, Peng Wang, and Yanning Zhang Vadclip: Adapting vision-language models for weakly supervised video anomaly detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 6074-6082, 2024.
Yuchen Yang, Kwonjoon Lee, Behzad Dariush, Yinzhi Cao, and Shao- Yuan Lo. Follow the rules: reasoning for video anomaly detection with large language models. In European Conference on Compuler Vision, pages 304-322. Springer, 2024.
Luca Zanella, Willi Menapace, Massimiliano Mancini, Yiming Wang, and Elisa Ricci. Harnessing large language models for training-free video anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11527-1556,, 2024.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint XrXiv:5501.33166 2025.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction andio-visual language model for video understanding. Xix preprint arXiv:2306.00 02858, 2023.
Huaxin Zhang, Xiaohao Xu, Xiang Wang, Jialong Zuo, Chuchu Han, Xiaonan Huang, Changxin Gao, Yue- huan Wang, and Nong Sang. Holmes-vad: Towards unbiased and explainable video anomaly detection via multi-modal Ilm. arXiv preprint ar Xrxiv:2406. 12235, 2024.
Hang Zhou, Junqing Yu, and Wei Yang. Dual memory units with uncertainty regulation for weakly supervised vidco anomaly detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 3769 3777, 2023.
A
Appendix
A.1 System Prompt
'Look at the video carefully. Describe the activity is abnormal or criminal (e.g., fighting, robbery, burglary, vandalism, assault, road accident, shooting, abuse, or other unssual violent actions). If no anomaly is present, describe the scene as normal daily activity (e.g., walking, standing, shopping, driving). Keep your description short.'
15

---

Published in Transactions on Machine Learning Research (02/2026)
A.2
VLM Responses
A.2.1
Two men are robbing a store while the security guard tries to stop them. One of the men pulls out a gun and points it at the guard, demanding money. The guard complies and hands over the cash.
A.2.2
A person is filming explosion gas station. The explosion occurs and the building is destroyed.
A.2.3
In the video, man wearing a yellow shirt is standing in front of an ATM machine and using his cell phone. Another man wearing a brown coat enters the scene and sits on the floor next to him. The man in the brown coat then grabs the yellow-shirted man's 's collar and pulls out a gun, pointing it at him, The yellow-shirted man appears to complying with the demands the man with the gun.
A.2.4
A group of people gathered around man who is sitting chair. They talking to him and he is holding a cup in his hand. The man is wearing a blue hat. There is also woman standing nearby.
A.3 Human Annotations
A.3.1
Two armed men rob store while security guard tries to intervene.
A.3.2
An explosion destroys gas station.
A.3.3
A man at an ATM threatened by another man with a gun during robbery.
A.3.4
A man in a jewelry store grabs necklace and runs out of the store.
A.4 QKV Attention instead of QKK
In Scction 3.3.1 we adopt QKK attention mechanism place of the conventional QKV formulation. Empirical evidence: When lesting with UCF-Crime, we observed higher performance on both coarse grained and fine-grained performance using QKK. We expand this by including results with XD-Viol Ei-Violence, where QKV obtains AP 83.13 and fine-grained mAP 31.57. These results confirm that QKK is compet- itive with or slightly superior to QKV. Table 13 shows the comparison between the two approaches. QKK approach surpasses QKV on both coarse grained metrics except fine-grained performance on Xi-Vinneece that is slightly higher with QKV.
Table 13: QKV QKK Comparison
UCF-CRIME
UCF-CRIME Fino-grained
Violenne Coarso-grained
Di-Violence Finc-grained
Coarsc-graines
QKV
86.91
8.05
83.13
31.57
QKK
87.21
8.13
84.66
31.15
16

---

Published in Transactions on Machine Learning Research (02/2026)
Theoretical rationale: In cross-modal fusion, the caption embeddings are already semantically aligned (VideoL.LaMA + CLIP text space). Using i as both V and K avoids redundancy with a smaller model (without additional parameters for V projection), which helps with regularizing the model for better gener- alization performance.
A.5 VAD Clip loss
Binary Cross-Entropy Loss (Loce). This loss supervises the coarse-grained anomaly detection branch by comparing the predicted video-level anomaly score with the video label:
Lice = [ [y + (1 v) log p)]
(2)
where y E {0,1} is the vidco-level label (0 for normal and 1 for abnormal), and p is the predicted vidco-level anomaly score obtained by aggregating the lop-K frame-level anomaly scores. This loss encourages correct discrimination between normal and anomalous videos.
Noise- Contrastive / Alignment Loss (Inoa). This loss aligus video features with the correct anomaly text embedding:
exp($m/77) Ince = - log(py))
(3)
Σ, exp(ss/T))
where is the similarity between the video representation and the m-th text class embedding, is a temperature parameter, Pm is the predicted probability of class m, and is the ground-truth anomaly class index, while j are all classes other than m. This loss enforces high similarity between the video and its correct textual anomaly category.
Contrastive Text Separation Loss (Iets) This loss enforces separation between the normal class em- bedding and abnormal class embeddings:
Lots Σ max (0, cos(tg=or (p=m))
(4)
where 10=0 is the text embedding of the normal class, '0=m is the embedding of the m-th abnormal class, cos(.,)) denotes cosine similarity, and C' is the number of abnormal classes. This loss pushes normal and abnormal semantic representations apart in the embedding space.
Overall Objective. The final training objective is given by:
L = Licee + Tince + Nlctss
(5)
where is a weighting factor that controls the contribution of the contrastive text separation loss. The values for F and are set to the defaults form Wu et al. (2024).
A.6 Broader Impacts
The improved performance and explainability of proposed VAD method can benefit. the society in public safety (through surveillance of public areas) and internet safety (through moderating content posted on the internet). Crimes or dangerous activities public areas can be automatically detected for alerting public safety professionals. Similarly, videos with harmful content can be detected before being published on the internet. The explainability feature of the proposed method more. Nevertheless, like all video surveillance technologies, the proposed VAD method should be used in an ethical way, particularly in terms of data privacy, fairness, and responsible use.
help professionals to trust the video AT
17


Reviews:
[{"review_id": "YNiXeRt6jv", "reviewer": "Reviewer_og3b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new algorithm for anomaly video detection. Comparing to [Wu et. al, 2024], this paper introduces and fuses the visual embeddings with the caption embeddings for the clips in a video. Moreover, based on the clip captions, for each anomaly class $m$, it computes the embedding $T^m$ using the most important keywords extracted from the clips belonging to this anomaly class. The experimental results show that the proposed approach outperforms the current SOTA ones.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The results and analysis have demonstrated the role of clip (video) captions in improving the performance because the captions contain relevant keywords for video anomaly detection.\n\nThe workaround to get the anomaly class embedding $T^m$ from all clip captions belonging to this class is reasonable and interesting.", "audience_yn": "Yes", "audience_explanation": "The video anomaly detection is an important application in video processing and analyzing.\n\nThe way this paper utilizes the clip captions is reasonable and interesting.", "requested_changes": "Figure 4 and Section 3.3.1 are not technically solid and need changes. According to the current writing, $f_i$ and $t_i$ are two D-dimensional vectors and it does not make sense to do cross-attention between them. Instead, they should be 2D tensors of visual embeddings and caption embeddings of the clips in a video.\n\nThe loss $\\ell_{sem}$ needs more clarification. For example, the purpose of this loss is to encourage the feature extractor and classification head to predict top-K most relevant clip regarding an anomaly class $m$ so that their visual embeddings well align to the anomaly class embedding $T^m$.\n\nMoreover, the losses reused from VADCLIP also need more detailing introduction at lease in the appendix for the self-containing of the paper.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wq0WfOr93L", "reviewer": "Reviewer_qf3a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes SemVAD, a novel framework for weakly supervised video anomaly detection (wVAD). It fuses semantic captions generated from a video-language model (VideoLLaMA 3) with visual embeddings from the CLIP model. Authors innovated a new fusion mechanism via a two-phase module consisting of an attention-based vision caption feature fusion and a convolutional mixer. A class-specific semantic alignment loss is added to improve fine-grained discrimination. Further, the authors address randomness sensitivity in multiple-instance learning by averaging across seeds. Their experiments on UCF-Crime and XD-Violence show that SemVAD outperforms SotA methods such as VADCLIP and PEL4VAD on both coarse- and fine-grained tasks.\n\nStrengths:\n1. Innovative integration of VLM-generated semantics into wVAD, enhancing interpretability\n2. Systematic evaluation across multiple seeds for reproducibility\n3. Clear empirical gains (≈ +0.9 AUC and +1.46 AP) and meaningful improvements in fine-grained anomaly localization\n4. Well-documented ablation and transferability analyses\n\nWeaknesses:\n1. Computational overhead.\nThe reliance on a heavy VLM (VideoLLaMA 3) restricts deployment to high-end GPUs and prevents use on edge or embedded systems. Even with an RTX 4090, processing speed (~14.2 fps) leaves little margin for scaling to multi-camera or higher-resolution settings.\n2. Empirical but weakly justified design.\nThe two-phase fusion module, comprising attention and conv-mixer, and the semantic-alignment loss are empirically validated but lack strong theoretical grounding. Choices like the QKK attention variant (A.1) are motivated mainly by incremental gains rather than principled reasoning about modality alignment.\n3. Semantic noise sensitivity.\nSince captions drive anomaly localization, errors or overlaps in textual descriptions (e.g., burglary, stealing, and robbery) can mislead the model. The heuristic prompt-selection pipeline may amplify dataset biases or truncate informative words, reducing robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The empirical results substantiate the paper’s claims of improved detection accuracy, robustness, and explainability. The experiments follow fair comparisons and show statistically consistent gains. Ablation and transfer experiments reinforce the causal impact of each module. While computational efficiency is only partially addressed, the quantitative and qualitative evidence support the main contributions convincingly.", "audience_yn": "Yes", "audience_explanation": "The topic of leveraging multimodal foundation models for weakly supervised video anomaly detection lies closely within the intersection of machine learning, computer vision, and multimodal representation learning, which aligns well with TMLR’s audience. The focus on reproducibility and interpretability via multi-seed evaluation and the integration of large VLMs will be of interest to researchers in both applied ML and those analyzing robustness in weak supervision.", "requested_changes": "1. Clarify fusion design:  \nProvide conceptual or mathematical intuition for why the two-phase (attention + conv mixer) fusion outperforms simpler concatenation or transformer-based fusion.\n\n2. Quantify explainability gains:  \nBeyond visual examples, include a human-evaluated or textual-similarity metric quantifying how captions improve interpretability.\n\n3. Address caption quality variance:  \nDiscuss how caption noise from VideoLLaMA affects anomaly scoring, perhaps with an uncertainty or filtering mechanism.\n\n4. Expand computational feasibility discussion:  \nReport GPU memory usage and runtime under different model scales; Suggest lighter VLM alternatives.\n\n5. Minor suggestions:  \nEnsure consistent use of seed-averaged metrics throughout tables; Fix minor typos (e.g., “comprison” in Table 7 caption).", "broader_impact_concerns": "The paper includes a brief discussion of Broader Impacts in Appendix A.2, noting potential societal benefits, such as improved public and internet safety through automated surveillance and content moderation. These points are appropriate and align with the intended application domain. However, it could be slightly strengthened by briefly reiterating the importance of ethical deployment, particularly in terms of data privacy, fairness, and responsible use.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tYBMqXh6x7", "reviewer": "Reviewer_NT1F", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\nThe paper introduces SemVAD, a framework for weakly supervised video anomaly detection (wVAD) that integrates semantic information from captions generated by VideoLLaMA 3 with visual features from CLIP. The authors also emphasize the instability of MIL-based methods under random initialization and propose evaluating models over multiple random seeds. Experiments on UCF-Crime and XD-Violence show that SemVAD achieves SOTA results in anomaly detection.\n\nStrength:\n(1)The paper highlights seed sensitivity in MIL-based wVAD and systematically addresses it by reporting mean/std over multiple runs. This sets a good precedent for future work.\n(2)The method consistently outperforms state-of-the-art baselines on both UCF-Crime and XD-Violence, in both coarse- and fine-grained detection.\n(3)Each proposed component is justified experimentally (captions, attention, convolution mixer, semantic loss).\n\nWeakness：\n(1)The choice of QKK attention instead of QKV is unconventional and only lightly justified in the appendix. A deeper theoretical or empirical motivation would be beneficial.\n(2)Although runtime is measured, reliance on VideoLLaMA 3 introduces significant overhead. The method may be impractical for real-world low-resource deployments, and efficiency trade-offs are not deeply analyzed.\n(3)Some figures (e.g., Fig. 2, Fig. 4) use symbols such as Fᵢ and Lᵥ without clear captions or explanations, which reduces readability. The authors should clarify these symbols in the figure captions or within the diagrams.\n(4)The efficiency analysis lacks a module-wise runtime breakdown and comparison with baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most claims are supported with convincing evidence through solid results on UCF-Crime and XD-Violence, multi-seed robustness analysis, and ablations. However, support for efficiency and explainability is weaker, as runtime breakdowns and quantitative explainability metrics are missing.", "audience_yn": "Yes", "audience_explanation": "The findings would interest TMLR’s audience since weakly supervised video anomaly detection is an important problem in computer vision with practical relevance to surveillance and content moderation.", "requested_changes": "(1) Provide a stronger justification for adopting QKK attention instead of the conventional QKV, either through additional ablation experiments or a deeper theoretical explanation.\n\n(2) Expand the efficiency analysis by discussing the computational overhead of VideoLLaMA 3 and exploring potential trade-offs or lightweight alternatives to demonstrate feasibility for low-resource settings.\n\n(3) Improve the readability of figures (e.g., Fig. 2, Fig. 4) by adding clear captions or annotations for symbols such as Fᵢ and Lᵥ.\n\n(4) Provide a module-wise runtime analysis (caption generation vs. fusion vs. detection) and compare with baseline methods to support claims of efficiency and practicality.", "broader_impact_concerns": "I have no ethical concerns regarding this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)