Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-13 16:03:02

Abstract: Current weakly supervised video anomaly detection (WSVAD) task aims to achieve frame-level anomalous event detection with only coarse video-level annotations available. Existing works typically involve extracting global features from full-resolution video frames and training frame-level classifiers to detect anomalies in the temporal dimension. However, most anomalous events tend to occur in localized spatial regions rather than the entire video frames, which implies existing frame-level feature based works may be misled by the dominant background information and lack the interpretation of the detected anomalies. To address this dilemma, this paper introduces a novel method called STPrompt that learns spatio-temporal prompt embeddings for weakly supervised video anomaly detection and localization (WSVADL) based on pre-trained vision-language models (VLMs). Our proposed method employs a two-stream network structure, with one stream focusing on the temporal dimension and the other primarily on the spatial dimension. By leveraging the learned knowledge from pre-trained VLMs and incorporating natural motion priors from raw videos, our model learns prompt embeddings that are aligned with spatio-temporal regions of videos (e.g., patches of individual frames) for identify specific local regions of anomalies, enabling accurate video anomaly detection while mitigating the influence of background information. Without relying on detailed spatio-temporal annotations or auxiliary object detection/tracking, our method achieves state-of-the-art performance on three public benchmarks for the WSVADL task.

Main paper: 
 introduction: 
 As a challenging and long-standing problem, video anomaly detection (VAD) has garnered significant attention from the computer vision community. The core objective of VAD is to detect various real-world anomalous events, holding immense potential for numerous practical applications, particularly in the realm of surveillance. For example, an intelligent video surveillance system equipped with anomaly detection capabilities can promptly perceive potential dangers, thereby facilitating timely interventions to enhance public security. Early studies have primarily focused on semi-supervised VAD [6, 11, 14, 31, 37, 47] , where the task is to learn normal patterns by solely utilizing normal videos, with abnormal events identified as those deviating from the learned normal pattern. However, these methods encounter limitations as they lack knowledge about abnormal videos, potentially leading to a high false alarm rate.Weakly supervised video anomaly detection (WSVAD) has emerged as a prominent research topic in recent years, distinct from its semi-supervised counterpart in that both normal and abnormal videos are available during the training stage. The objective of WSVAD is to achieve frame-level anomaly detection with weak or coarse annotations (i.e., video-level labels). Existing works typically involve extracting features from full-resolution frames using pre-trained models such as I3D [2] , Transformer [7] , and CLIP [45] , followed by training a classifier based on the multiple instance learning (MIL) mechanism to predict anomalous events at the frame level. While these approaches yield promising results as a standard practice, they often overlook a crucial aspect: abnormal events tend to occur in localized spatial regions rather than spanning the entire full-resolution frame, especially in surveillance scenarios. Drawing insights from the popular benchmark UCF-Crime [52] , we illustrate this observation with examples depicted in Figure 1 . For clarity, the anomalous regions are delineated using orange bounding boxes. It is evident that various types of anomalies manifest in diverse spatial locations and sizes; however, the spatial extent of anomalies is typically small compared to the dimensions of the full-resolution video frame. Nevertheless, existing works compress entire frames into single features, thereby neglecting crucial region-level abnormal details, and leading subsequent classifiers to rely heavily on dominant background information. Moreover, such operations further cause that the VAD model lacks reliability and interpretability, as it does not verify whether the detection align with the actual spatial location of anomalies. Consequently, certain true positive detection may be merely "lucky" coincidences of erroneous detection (such as irrelevant background or events) to the abnormal events [31] .Therefore, how to explicitly exploit the spatio-temporal fusion [15, 21] , and detect anomalies from when and where perspectives, is an important direction for exploration. It is worth noting that our work is not the first attempt to address weakly supervised video anomaly detection and localization (WSVADL). Several previous studies [28, 33, 53, 64] have also endeavored to leverage spatio-temporal relation capabilities to enhance frame-level VAD under weak supervision. However, these approaches often necessitate complex and resource-intensive relation modeling. For example, Liu et al. [33] re-annotated the UCF-Crime benchmark, subsequently training detection models with full spatio-temporal annotations. On the other hand, Wu et al. [64] paved an another way for weakly supervised spatio-temporal VAD, which took inspirations from spatio-temporal action localization [26, 42] , and used a tubelet-level detector based on pre-trained object detectors and hierarchical clustering to detect possible spatio-temporal anomalies. These methods have demonstrated improved frame-level anomaly detection results over conventional WSVAD methods by mitigating the influence of irrelevant background through intricate spatiotemporal modeling processes. Nevertheless, these solutions are either complicated, e.g., multi-scale spatial pyramid training [28] and spatio-temporal coherence modeling [53] or heavily dependent on auxiliary modules for object detection and tracking [64] and detailed spatio-temporal annotations. [33] In this paper, enlighted by the success of large pre-trained visionlanguage models (VLMs) on industrial defect detection [4, 23, 81, 82] , we propose a novel method STPrompt that learns spatio-temporal prompt embeddings built on the top of VLMs for WSVADL. Different from previous works [28, 33, 53, 64] , our approach is both conceptually straightforward and practically effective. Specifically, we first segment the video into frames and further then split each frame into patches, and video anomaly detection and localization can thus be conceptualized as frame-level and patch-level classification, obviating the reliance on object detection and tracking. Concurrently, to alleviate the complexity associated with spatio-temporal relation modeling, we explicitly decompose spatio-temporal VAD into two distinct sub-tasks: temporal anomaly detection and spatial anomaly localization. For temporal anomaly detection, alongside standard temporal modeling, we introduce a simple yet effective spatial attention aggregation (SA 2 ) mechanism aimed at enhancing background denoising. This approach leverages motion priors derived from the intrinsic attributes of videos. Regarding spatial anomaly localization, we capitalize on the established image-to-concept capability of VLMs, taking a significant step forward towards training-free spatial anomaly localization without full supervision. By leveraging related concepts, we identify abnormal patches. Our approach addresses the limitations of prior works while achieving superior performance across three public benchmarks: UCF-Crime [52] , Shang-haiTech [38] , and UBnormal [1] .To summarize, the main contributions of this paper are threefold: • A novel model named STPrompt is proposed to address spatiotemporal video anomaly detection under weak video-level supervisions. To our knowledge, STPrompt represents the first endeavor to efficiently transfer pre-trained vision-language knowledge from VLMs to simultaneously tackle temporal anomaly detection and spatial anomaly localization.• To mitigate the requirement of extra auxiliary information and intricate modeling strategies, STPrompt decouples the WSVADL task into temporal anomaly detection and spatial anomaly localization. A spatial attention aggregation mechanism is devised in STPrompt to filter irrelevant background for temporal anomaly detection. Besides, a large language models (LLMs)-enabled, training-free anomaly localization method is introduced to obtain fine-grained text prompts for spatial anomaly localization.• Extensive experiments on three widely-used benchmarks show the superiority of STPrompt over state-of-the-art competing methods. It performs substantially better than, or on par with, the recent competing methods in anomaly detection, while largely outperforming them in TIoU for anomaly localization across all three datasets, e.g., by a margin of about 1.9% on UCF-Crime, 5.7% on ShanghaiTech, and 4.5% on UBnormal compared to VadCLIP [71] . 
related work 2.1 video anomaly detection: 
 2.1.1 Semi-supervised VAD. The advent of deep learning revolutionized the field of semi-supervised VAD, with the mainstream of research focusing on convolutional neural networks (CNNs) [10, 22, 29, 35, 43, 61, 66, 69, 75] , recurrent neural networks (RNNs) [51, 74] , and transformers [63, 73] , with many of these approaches adopting self-supervised learning principles. For example, several studies [17, 38, 77] utilize 2D-CNNs, 3D-CNNs, and RNN-based autoencoders to reconstruct normal events and identify abnormal events based on the magnitude of the reconstruction error. Liu et al. [34] proposed a CNN-based video prediction network to predict future video frames based on previous frames, while Yang et al. [73] employed transformers to extract video features and then reconstructed video events based on key-frames. Yu et al. [75] introduced a novel approach called video event completion to address gaps existing in reconstruction or frame prediction methods. Several of these approaches also address spatial anomaly localization. For instance, Li et al. [31] divided the visual field into overlapping regions and learned a global mixture model using only patches around the current frame, with regions least similar to their surroundings deemed most likely to be abnormal. Wu et al. [66] similarly divided the visual field into overlapping regions and trained a deep one-class model to discriminate abnormal regions.2.1.2 Weakly supervised VAD. Weakly supervised video anomaly detection [3, 9, 33, 52, 55, 64, 76] has emerged as a prominent research focus in recent years. Sultani et al. [52] were among the pioneers, introducing a deep MIL model that treats a video as a bag and its segments as instances. By utilizing ranking loss with bag-level labels, their model aims to maximize the separation between the most anomalous instances in positive bags and negative bags. Subsequent studies have endeavored to enhance the positive design aspect of WSVAD. For instance, Zhong et al. [78] proposed a graph convolutional network (GCN)-based method to model feature similarity and temporal consistency between video segments. Tian et al. [55] devised robust temporal feature magnitude learning, significantly improving the MIL approach's robustness to negative instances from abnormal videos. Li et al. [30] and Huang et al. [19] introduced transformer-based multi-sequence learning frameworks to capture temporal relationships between frames. Zhou et al. [79] proposed dual memory units and an uncertainty learning scheme to better distinguish patterns of normality and anomaly. Wu et al. [67, 68] introduced a novel multi-modal dataset and a fine-grained weakly supervised VAD method capable of distinguishing between different types of anomalous frames. More recently, pre-trained vision-language models have garnered significant attention in the VAD community. VadCLIP [71] was the first to efficiently transfer pre-trained language-visual knowledge from CLIP [45] to weakly supervised VAD, achieving state-of-the-art performance. Pu et al. [44] attempted to enhance WSVAD by learning prompt-enhanced context features.Generally, image anomaly detection aims to localize anomalies in images like industrial defect images, predicting an image or a pixel as normal or anomalous. Typical works [5, 18, 20, 48, 54, 57, 72] mainly focused on one-class or self-supervised anomaly detection, which only requires normal images. Recently, exploiting VLMs with prompts has emerged as a successful enabler for this task, especially for the zero/few-shot setting. WinCLIP [23] introduced a language-guided paradigm for zero-shot industrial defect detection. AnomalyCLIP [81] adapted CLIP for zero-shot industrial defect detection across different domains, which learn object-agnostic text prompts that capture generic normality and abnormality. InC-TRL [82] learned residual features between query images and fewshot in-context normal images to build generalist models for image anomaly detection. These CLIP-based works inspired us to spatially local anomaly in videos, but our method is more succinct and does not require learnable parameters in the prompts. 
method 3.1 overview: 
 Previous WSVAD task supposes that only video-level labels are given for model training, and encourages the model to predict whether each video frame is abnormal at the test time, where the detection granularity falls into the frame level. In comparison to WSVAD, WSVADL is a more challenging task, which assumes that the model is supposed to detect anomalies at a finer level, i.e., the pixel level, while keeping the supervisions unchanged. Mathematically, given a set of training samples {V, Y 𝑏 , Y 𝑐 }, where V, Y 𝑏 , and Y 𝑐 denote the sets of video, video-level binary label, and videolevel category label, respectively. For each video sample 𝑣, it has two corresponding labels, namely, 𝑦 𝑏 and 𝑦 𝑐 . Here 𝑦 𝑏 ∈ {0, 1}, and 𝑦 𝑏 = 1 indicates that 𝑣 includes anomalies; and 𝑦 𝑐 ∈ R 1+𝐶 , in which 𝐶 is the number of abnormal categories.As aforementioned, the main limitation of previous spatiotemporal VAD works [28, 33, 53, 64] is that they rely on laborintensive spatio-temporal annotations, detector-dependent preprocessing, and computationally expensive spatio-temporal modeling. Compared to these works, our STPrompt is conceptually simple yet practically effective, which is demonstrated in Figure 2 . To move beyond the above limitations, there are a series of dedicated designs in our STPrompt. Firstly, based on this routine operation of splitting a video into multiple frames, we further split each frame into multiple patches. Through such an operation, WSVADL can be considered as a coarse frame-level and patch-level classification task without the requirement of any detection pre-processing. On this case, a natural way is to directly treat all spatial patches as instances, and then use the MIL mechanism to predict the anomaly confidence of each patch. However, such a readily implemented way is computationally heavy and can not be easily optimized [13, 27] . Therefore, to reduce the spatio-temporal modeling complexity and optimization difficulty, we then factor the WSVADL task into two sub-tasks, i.e., temporal anomaly detection and spatial anomaly localization. For temporal anomaly detection, we introduce a dual-branch model built on the top of CLIP, meanwhile, we design two key modules, on the one hand, a spatial attention aggregation assists the temporal detection model in focusing on potential spatial location of anomalies. On the other hand, a typical temporal adapter enhances the temporal context capture capabilities of the temporal detection model. For spatial anomaly localization, to address the challenges posed by insufficient supervisions, we design a training-free anomaly localization strategy with the basis of "image-to-concept" capacity of VLM.Inspired by the pioneer work VadCLIP [71] , we also introduce a dual-branch framework, namely, classification branch and alignment branch. Specifically, given a video 𝑣, we employ a frozen image encoder of CLIP to extract the frame-level feature 𝑥 𝐶𝐿𝐼 𝑃 ∈ R 𝑇 ×𝐷 , where 𝑇 is the length of video 𝑣, and 𝐷 is the feature dimension. Then these feature are fed into two branches after a series of information enhancements, classification branch is to directly predict the anomaly confidence 𝐴 ∈ R 𝑇 ×1 by a binary classifier, another align branch is to compute the anomaly category probability 𝑀 ∈ R 𝑇 × (1+𝐶 ) by means of the image-to-concept alignment. With 𝐴 and 𝑀 in hands, we adopt the typical TopK [67] and the recent MIL-Align [71] strategies to compute the video-level anomaly prediction and category prediction, respectively, these predictions are subsequently used to calculate losses and provide data support for model optimization. Throughout the whole process, we devise two modules to encourage the model to focus on anomalies from the spatial and temporal dimensions, which are illustrated in the following sections.3.2.1 Motion prior-aware spatial attention aggregation. Although we explicitly disentangle WSVADL into two independent tasks, i.e., temporal anomaly detection and spatial anomaly detection, for the temporal anomaly detection task, we still require the critical spatial local anomalies as assistance information. This is because potential spatial anomalies can eliminate the noise effect caused by the irrelevant backgrounds, after all, most anomalies may occupy a small spatial region. For this problem, a majority of previous works completely ignore spatial anomaly information, and a tiny minority attempts to learn the interaction between spatial patches and video frames. The former works lacks the consideration of the use of individual spatial contents, while the latter works inevitably incurs excessive computational costs. Therefore, we propose a novel spatial attention aggregation (SA 2 ) scheme to capture key spatial information with low computational costs. As we know, the whole frames consist of the background of the scene and foreground of action, and anomalous events often occur with foreground objects, thus focusing on the spatial foreground captures potentially anomalous events. The common methods for locating the foreground include object detection algorithms [46] or optical flow [8] , but these require high computation costs. Here, we propose a considerably simple and efficient method named SA works [59, 65] . Specifically, given the frame-level feature 𝑥 𝐶𝐿𝐼 𝑃 and its corresponding spatial feature 𝑥 𝑃𝐴𝑇𝐶𝐻 ∈ R 𝑇 ×𝐻 × 𝑊 ×𝐷 , where 𝐻 and 𝑊 are the height and width of the spatial feature, we argue that when most abnormal events occur, the corresponding location within spatial feature would change significantly [65] . Therefore, we compute the difference between frames to obtain the motion magnitude: (1) where the size of 𝑀𝑜 is 𝑇 × 𝐻 × 𝑊 , L2 is the L2 normalization applied in the channel dimension, and 𝑖 denotes the 𝑖-th frame. Then we use the TopK mechanism to select a fixed number of patchlevel feature 𝑥 𝑀𝑜 ∈ R 𝑇 ×𝐾 ×𝐷 with the highest motion magnitude 𝑀𝑜 𝑇 𝑂𝑃 ∈ R 𝑇 ×𝐾 ×1 , where 𝐾 < 𝐻 × 𝑊 , and then compute the attention to obtain the aggregate spatial feature:𝑀𝑜 [𝑖] = 𝐿2(𝑥 𝑃𝐴𝑇𝐶𝐻 [𝑖] × 2 -𝑥 𝑃𝐴𝑇𝐶𝐻 [𝑖 -1] -𝑥 𝑃𝐴𝑇𝐶𝐻 [𝑖 + 1]),EQUATIONEQUATIONDifferent from 𝑥 𝐶𝐿𝐼 𝑃 in which all pixels in each frame have nearly equal influence for anomaly detection, 𝑥 𝐴𝑆 places a heavy focus on potential anomaly locations. No matter how the spatial region of abnormal events changes, these two features, i.e., 𝑥 𝐶𝐿𝐼 𝑃 and 𝑥 𝐴𝑆 , can extract key abnormal information from the local and global perspectives. In other words, they are complementary.As aforementioned, we adopt the pre-trained image encoder of CLIP to extract frame-level features, which contain momentary information but lacks a global temporal context critical for the VAD task. This motivates us to study temporal context modeling. We propose the temporal adapter that is similar to a vanilla multi-layer Transformer encoder, consisting of self-attention, layer normalization (LN), and feed-forward networks (FFN). Following [40] , positional encoding is not applied. The main difference between temporal adapter and Transformer encoder lies in the self-attention, which is based on relative distance instead of feature similarity [70] . The adjacency matrix in self-attention is calculated as 𝑀𝑎[𝑖, 𝑗] = -|𝑖 -𝑗 | 𝜎 , where the similarity between 𝑖-th and 𝑗-th frames only determined by their relative temporal distance. 𝜎 is a hyper-parameter to control the range of influence of distance relation. In this work, we add 𝑥 𝐶𝐿𝐼 𝑃 and 𝑥 𝐴𝑆 together, and feed the summation feature into temporal adapter, thus empowering CLIP with temporal modeling capability, which can be formulated as follows:EQUATION3.2.3 Dual-branch prompt learning. After obtaining the deeply processed features, we require the model to predict the frame-level anomaly confidence. Due to the proven performance of VadCLIP, we further adopt its dual-branch detection framework. The one branch is a classification branch (C-Branch), a simple linear layer with the number of neuron of one, which takes 𝑥 𝑇 𝐴 as input, and generates the anomaly confidence 𝐴. Another branch is an alignment branch (A-Branch), which takes video features and textual embedding of labels as input, and yields the anomaly category probability 𝑀. (5)Following the setup of [71] , TopK-based classification objective function is adopted for classification branch, which can be presented as follows,EQUATIONwhere 𝑇𝑜𝑝𝐾 means select the set of 𝑘-max frame-level confidences in 𝐴 for the video 𝑣. L 𝑐𝑙𝑎𝑠𝑠 is the binary cross-entropy between 𝑝 𝑏 and video-level binary labels 𝑦 𝑏 . MIL-Align based objective function is used for alignment branch, which is based on anomaly category probability 𝑀. For each column of 𝑀, we select 𝑘-max similarities and compute the average to measure the alignment degree between 𝑣 and the current class. Then we can obtain a vector 𝑆 = {𝑠 1 , ..., 𝑠 (1+𝐶 ) } that represents the similarity between 𝑣 and all classes. Then we compute the loss L 𝑎𝑙𝑖𝑔𝑛 as follows,EQUATIONEQUATIONwhere 𝑝 𝑐 𝑖 is the prediction of the 𝑖-th class, and 𝜏 refers to the temperature hyper-parameter for scaling.To learn discriminative prompt embeddings, we also introduce a contrastive loss to make all textual embeddings more dispersible. Specifically, we calculate cosine similarity between label prompt embeddings, and compute the contrastive loss L 𝑐𝑜𝑛𝑠𝑡 as follows,L 𝑐𝑜𝑛𝑠𝑡 = ∑︁ 𝑖 ∑︁ 𝑗 𝑚𝑎𝑥 0, 𝑃𝑟𝑜𝑚𝑝𝑡 ⊤ 𝑐 𝑖 𝑃𝑟𝑜𝑚𝑝𝑡 𝑐 𝑗 𝑃𝑟𝑜𝑚𝑝𝑡 𝑐 𝑖 2 • 𝑃𝑟𝑜𝑚𝑝𝑡 𝑐 𝑗 2 . (9)The final objective function is the weighted sum of the above three loss functions:EQUATION)The core of this operation is that how to locate anomaly regions. Thanks to the emerging paradigm of pre-trained VLMs, we take a step forward to training-free spatial anomaly localization. Inspired by CLIP-based industrial defect detection works [23, 81, 82] , we regard the spatial anomaly localization as a spatial patch retrieval process given text queries. Specifically, we suppose a test video frame is deemed as an anomaly frame due to its high anomaly score.We then obtain its patch-level feature map 𝑥 𝑃 ∈ R 𝐻 ×𝑊 ×𝐷 by the sliding window scheme, in which the patches are generated in the same way as 𝑥 𝑃𝐴𝑇𝐶𝐻 . Here the sliding windows scheme means that we first generate a set of image patches with a fixed-size window of 𝑃 ×𝑃 by sliding the window with a stride of 𝑆, i.e., a operation similar to convolutions, and then feed these image patches into the image encoder of CLIP to obtain the corresponding embedding of the CLS token. Notably, we do not adopt the natural dense representations, i.e., the penultimate feature maps in CLIP, though its generation is simpler than the sliding-window based scheme. This is because those features are not directly supervised with language in CLIP, and moreover, these patch features have already aggregated the global context due to self-attention, hindering the modeling of local region details for localization [23] .As for text queries, we generate several normal and abnormal descriptions. For the normal generation, the specifics are as follows, compared to industrial defect detection tasks, using textual labels to describe normal behavior under WSVAD task is more challenging. This is because videos in WSVAD task typically include multiple scenes, especially numerous real-world scenarios that are difficult to accurately summarize with textual labels directly. On the other hand, in terms of spatial fine-grained description, there may be semantic ambiguities between normal and abnormal behaviors due to the limited coverage range of spatial patches. Considering that most anomalies target intense human behavior, we believe it is more appropriate to use textual captions that describe the background of the image as normal descriptions. Therefore, we query LLMs about common indoor and outdoor items and selected 12 of the most common text descriptions as normal text descriptions. For example, "a picture of sky, a picture of ground, a picture of road, a picture of grass, a picture of building, a picture of wall, a picture of tree, a picture of floor tile, a picture of desk, a picture of cabinet, a picture of chair, a picture of door".For abnormal descriptions, in addition to the original abnormal categories, we also use LLMs with a template "Provide phrases similar to [abnormal category]" to obtain augmented descriptions. For example, "[abnormal category]" can be set as "people knockout someone" for the category Fighting, "people lying on the ground" for Car accident, "someone ignite fire" for Arson, "people shooting someone" for Shooting. The augmented prompts, along with the original textual categories, are used as final abnormal prompts for spatial anomaly localization.With 𝑥 𝑃 and text queries 𝑞 𝑇 in hands, we perform a patch-level retrieval process, namely, using normal descriptions and abnormal descriptions to locate the background regions and potential abnormal regions, respectively. Mathematically, this process can be represented as,EQUATIONA spatial heat map of anomalous events 𝑀𝑠 with size of 𝐻 × 𝑊 is created, and it is resized to the size of original frames, and can generate the predicted bounding box by shape detection algorithm. Notably, we create two different scale feature maps for 𝑥 𝑃 with 𝑃&𝑆 set to 32&32 and 80&48, and use a fusion hyper-parameter 𝜆 to average their detection results as the final result. 4.1.2 Evaluation metrics. For the temporal anomaly detection, we follow previous works [52] , and utilize the area under the curve (AUC) of the frame-level receiver operating characteristics (ROC). The higher AUC indicates the better performance. For the spatial anomaly localization, following the previous work [33] , we use TIoU (Temporal Intersection-over-Union) as the evaluation metric, which can be formulated as the following equation: 
experiments 4.1 datasets and evaluation metrics: 
 EQUATIONwhere the indicator 𝐼 [.] ∈ {0, 1} indicates whether the given anomaly frame is predicted as anomaly according to the anomaly score 𝑃 𝑗 , 𝐴𝑟𝑒𝑎 𝑝 and 𝐴𝑟𝑒𝑎 𝑔 represent the area of prediction bounding box and the ground truth bounding box respectively, and 𝑁 represents the total number of all anomaly frames. We report both frame-level detection and pixel-level localization accuracy.We use the frozen CLIP (ViT-B/16) to extract features of video frames. Specifically, we process 1 out of 16 frames on UCF-Crime dataset, and 1 out of 4 frames on ShanghaiTech and UBnormal thus higher sampling frequency can slightly improve the performance.During the training stage, the maximum length of training videos is set to 256, videos with length exceeding the limit will be sampled to the maximum length. For all datasets, we set the length of the learnable prompts prefixed to text labels as 8. For the hyperparameters of total loss function, 𝛼 is set to 1 on UBnormal and 0.9 on UCF-Crime and ShanghaiTech, 𝛽 is set as 2 on all datasets. 𝑘 is set to [𝑇 /16] + 1 on all datasets. For 𝑥 𝑃𝐴𝑇𝐶𝐻 , we resize the image to 224 × 224, then use a sliding-window of size 32 × 32 with the stride of 32 to generate multiple patches, with both 𝐻 and 𝑊 equal to 7. Besides, 𝐾 in SA 2 is set to 12. For model optimization, we use AdamW optimizer with learning rate of 1e-4 to train the model on a single RTX3090 GPU, and batch size is set to 64.To ensure fairness in comparison, we re-implement most methods using the same CLIP features as ours, given that several works utilize different feature extractors.As listed in Tables 1 to 3, our method demonstrates superior performance on the UCF-Crime and UBnormal benchmarks, while also achieving competitive results on ShanghaiTech. Specifically, our method attains 88.08% AUC on UCF-Crime, outperforming other comparison counterparts without using CLIP features by a wide margin, and also excelling CLIP-based methods by a clear margin. Compared to the best competitor VadCLIP [71] , although our STPrompt only utilizes the spatial attention aggregation instead of multi-crop augmentation, CLIP 97.57 N/A UMIL [39] X-CLIP [41] 96.78 N/A MSL [30] VideoSwin [36] 97.20 N/A SSRL [28] CLIP 96.22 N/A CLIP-TSA [24] CLIP 98.32 N/A VadCLIP [ On UBnormal dataset, our method achieves 63.98% AUC, achieving an absolute gain of 1.0% in terms of AUC over the best competitor OPVAD [70] . The above results demonstrate the compelling ability of our method for WSVAD task, which can outperform current competition counterparts on three commonly-used benchmarks.On the other hand, our method exhibits superior performance in spatial anomaly localization. Since there are few works exploring spatial anomaly localization, we modify a classical method [13] and an emerging method [71] as baselines, where the former is a variant of Sultani et al. [52] , a simple patch-level detection, and the latter employs the 1 to 3 , we found that STPrompt achieves best performance in terms of TIoU on all the three benchmarks. For example, compared to fully-supervised method Liu et al. [33] and the baseline Sultani et al. [52] , our STPrompt can achieve a substantial improvement of 7.5% and 7.2% TIoU on UCF-Crime. Besides, STPrompt also comprehensively outperforms VadCLIP, showing the advantages of purpose-built prompts created by LLMs with respect to vanilla learnable label prompts for the spatial anomaly localization.To investigate the influence of designed modules, we perform extensive ablation studies with frame-level AUC and spatio-temporallevel TIoU on UCF-Crime and UBnormal benchmarks . 4 , we investigate the performance of dual-branch structure in various situations. It is evident that employing both the C-branch and A-branch leads to improved performance compared to using a single branch. After adding other modules, A-branch has sustainable advantages over C-branch, which indicates that A-branch possesses superior capabilities in temporal anomaly detection, Consequently, we use the results generated from A-branch (1 minus the similarity between the video and normal class) as the final frame-level anomaly score.4.4.2 Effectiveness of spatial aggregation attention. We employ the spatial aggregation attention not only for aggregating spatial features for temporal detection but also for assisting spatial anomaly location. According to the result in Table 4 , using spatial features yields a notable improvement of 2.0% AUC on UCF-Crime and 1.0% AUC on UBnormal, respectively. This underscores the efficacy of spatial information in enhancing the ability of model to distinguish anomalies from normal events or background within the same frame. Consequently, abnormal regions are highlighted while redundant background is suppressed. Table 5 shows the effects of using different strategies to integrate spatial features. We observe that simply using the average features or attention-based weighted average features can slightly improve the performance on UCF-Crime, but leads to a performance drop on UBnormal. Using motion-based selection strategy can reduce redundant spatial features and achieves a improvement on both UCF-Crime and UBnormal. Our SA 2 is the combination of motion-based selection 4 , temporal adapter gets significant performance boosts both with or without SA 2 . Furthermore, we also perform ablation studies on how to learn temporal modeling. From Table 6 , we found that the vanilla transformer that performs well when being trained on large-scale datasets with full-supervised supervision, however, is not suitable for WSVAD task. Besides, experimental results reveal that transformer based on fixed distance relation performs better than traditional self-attention transformer. This indicates that salient priors are required for WSVAD task with insufficient supervisory signals.As aforementioned, we employ two patch-level feature maps with different size for spatial anomaly localization, which allow our model to cover spatial anomalies from different window perspectives. To further investigate the balance point between these two feature maps, we perform an ablation experiment and show the results in Table 7 . For UCF-Crime, only using the feature map of size 7 × 7 (𝜆=1.0) can achieve the best performance. For another benchmark UBnormal, when two feature maps make the same contributions, i.e., setting 𝜆 to 0.5, our model achieves the best performance. Therefore, we adopt 𝜆 = 0.7 to achieve a favorable trade-off.We conduct a detailed analysis of the parameter count and computational cost of our model, juxtaposing it with previous related works in Table 8 . Upon reviewing the comparison results, we note that our method, particularly when compared to spatio-temporal modeling based VAD method SSRL [28] , exhibits lighter weight and greater efficacy. It is worth highlighting that, despite the presence of shared parameters between different modules in SSRL denoted by the symbol * , the parameter count and computational cost of our model are notably lower. In general, the comparison results presented in Tables 1 to 3 underscore the accurate anomaly detection and localization capabilities of our method. Furthermore, these findings underscore the favorable balance between speed and accuracy achieved by our approach in spatio-temporal modeling. For each sample, the left column is the heat map of abnormal regions, and the right column is the corresponding localization bounding boxes (red) and the ground truth bounding boxes (green). We observe that STPrompt can localize major abnormal events at the pixel level for these abnormal video frames, but it also produce several false alarms at the same frame. Such results demonstrate that on the one hand spatial anomaly localization is more challenging than temporal anomaly detection, and on the other, our method can better perceive anomalies at the pixel level, thereby increasing the interpretability for temporal anomaly detection. 
conclusion: 
 In this work, we present STPrompt, a novel approach utilizing frozen vision-language models, for weakly supervised video anomaly detection and localization. To tackle this challenging task, we adopt a divide-and-conquer strategy by decomposing this task into two distinct sub-tasks: temporal anomaly detection and spatial anomaly localization. For the former task, we design a spatial attention aggregation strategy and temporal adapter to efficiently capture potential spatial anomaly information as well as contextual information, and then employ a dual-branch network to detect anomalies by binary classification and cross-modal alignment. For the latter task, we devise a training-free query-and-retrieve method based on the pre-trained concept knowledge from VLMs. Without bells and whistles, our STPrompt achieves state-of-the-art performance on three benchmarks in both temporal anomaly detection and spatial anomaly localization. In the future, how to further reduce the spatial false alarm rate and improve spatial localization accuracy is a problem worthy of long-standing research. Weakly Supervised Video Anomaly Detection and Localization with Spatio-Temporal Prompts ACMMM'24, 28 October-1 November, 2024, Melbourne, Australia 
 
 Appendix:  


Decision: The initial ratings are 1 WR, 2 BA and 1 WA, and they are 1 BA, 1 WA and 1 Accept after rebuttal. The reviewers all appreciated the proposed method about transfer pre-trained vision-language knowledge from VLMs to simultaneously tackle temporal anomaly detection and spatial anomaly localization, and are satisfied with the authors' responses. 
I agree with the reviewers and recommend an acceptance. The authors are encouraged to incorporate the material from their response to the camera ready version.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel method that learns spatio-temporal prompt embeddings for weakly supervised video anomaly spatio-temporal localization based on pre-trained vision-language models (VLM). 
The proposed method employs two-stream network structure, one for temporal anomaly detection and the other for spatial anomaly localization. 
For the temporal anomaly detection branch, a spatial attention aggregation (SA2) scheme is proposed to capture key spatial information. 
For spatial anomaly localization branch, some textual captions of background (normal descriptions) and some textual captions of abnormal descriptions are added as augmented text queries.
strength_weakness: strength_weakness: The idea of video anomaly spatio-temporal localization without object detection/tracking is novel. 
The proposed method achieves state-of-the-art performance on three public benchmarks for the WSVADL task.
limitations: limitations: (1) Some symbols are not explained, such as ‘𝑀𝑜𝑇𝑂𝑃 [𝑖] ’ in Eq.(2).
(2) In Table 1, the performance of the proposed method that has an extra spatial anomaly localization branch is a little better than those of VadCLIP (88.08 vs. 88.02). Does it mean that spatial anomaly localization do not bring much benefit for video anomaly detection?
(3) One advantage of the proposed method is lighter weight. I’m curious about How it is compared with those of VadCLIP.
paper_summary: paper_summary: This paper proposes a spatial-temporal prompt embedding for weakly supervised video anomaly detection using pretrained vision language models (VLMs). Experiments show comparative performance with reduced computational complexity. The paper is generally well-written and easy to follow.
strength_weakness: strength_weakness: The use of LLM to generate augmented descriptions for spatial-temporal anomaly localization.
There are extensive baselines in experiments, and the experiments cover multiple aspects of performance.
limitations: limitations: Three sophisticated datasets are used, which is a good thing, and it would be better if per-dataset comparisons are made against the baselines.
paper_summary: paper_summary: The paper highlights that many abnormal events occur in localized regions of video frames rather than across entire frames. Existing frame-level feature-based methods may be misled by predominant background information and often lack interpretability regarding detected anomalies. To address this, a dual-stream network structure is proposed for weakly supervised video anomaly detection, with one stream focusing on the temporal dimension and the other on the spatial dimension. For temporal anomaly detection, a spatial attention aggregation mechanism is designed to help the model identify temporal anomalies more accurately. For spatial anomaly localization, a training-free strategy is proposed, utilizing fine-grained prompts provided by large language models (LLMs).
strength_weakness: strength_weakness: 1.The motivation of the paper is clear and easily understood.

2.The idea of utilizing large language models to generate fine-grained prompts to achieve anomaly localization is interesting.
limitations: limitations: 1.The methodology in the paper lacks innovation and demonstrates minimal performance enhancement. (1) The utilization of patch-level features to enhance anomaly detection is not pioneering (e.g. previous work [28]) and the design of the Temporal CLIP adapter closely resembles previous work [70].（2）The overall framework is similar to existing work [71], with improvements like adding a spatial attention mechanism and an anomaly localization branch. However, the performance improvement compared to [71] on both UCF-Crime and Shanghaitech datasets is minor. Compared with [28] which uses I3D features, this method uses CLIP features and Prompts, and the performance improvement is also small. This raises questions about the effectiveness of spatial features for anomaly detection.

2.The description of some parts of the paper is unclear. (1) In line 275, the limitations of the research work are pointed out [28, 33, 53, 64], it should be clarified whether these limitations are shared among these works, or whether each work corresponds to specific limitations. (2) The meaning of $\dag$ symbols in Table 1, Table 2 and Table 3 lacks explanation. (3) The meaning of $l$ in Equation (11) should be clarified. And $l$ is also used in line 456, which is confusing. (4) Please indicate how many augmented descriptions were generated using LLM, and explain how these descriptions were merged with the original text categories to form the final prompt.

3.In lines 611-613, the sampling method used in the training process appears to differ from the previous methods. Does this change have a significant impact on performance? Please provide details of the specific sampling method used.

4.Among the qualitative results shown in Figure 4, whether the anomaly localization results can correspond to the prompt description.

5.How about the performance of the proposed method on XD-Violence dataset?
paper_summary: paper_summary: Authors explored weakly supervised video anomaly detection and localization task and proposed a two-stream network STPrompt to address spatio-temporal video anomaly detection under weak video-level supervisions. Authors designed a spatial attention aggregation mechanism to filter irrelevant background and provide more spatial fine-grained features for temporal anomaly detection.Inspired by CLIP-based industrial defect detection works, authors regarded the spatial anomaly localization as a spatial patch retrieval process given text queries and using LLM to generate these text prompts. Different spatial patches generate different scale feature maps, then use a fusion hyper-parameter to average their detection results as the final result.
strength_weakness: strength_weakness: 1. The proposed work is the first endeavor to efficiently transfer pre-trained vision-language knowledge from VLMs to simultaneously tackle temporal anomaly detection and spatial anomaly localization. Overall, the novelty of this work is enough.
2. Authors proposed a novel spatial attention aggregation scheme to capture key spatial information with low computational costsuse with the motion magnitude computed by difference between frames. And the proposed method works well according to the experiment results.
3. The qualitative results of spatial anomaly localization are very intuitive and clear.
limitations: limitations: 1. It seems that the improvement obtained by introducing spatial patch features is not significant on temporal anomaly detection.
2. Due to the lack of supervisions, the spatial localization still has great potential for improvement results.
3. Are these normal text descriptions good enough to describe most normal scenarios in the datasets?