TDSD: Text-Driven Scene-Decoupled Weakly Supervised Video Anomaly Detection

Authors: shengyang  sun
jiashen  hua
junyi  feng
dongxu  wei
baisheng  lai
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 10:18:25

Abstract: Video anomaly detection has garnered widespread attention in industry and academia in recent years due to its significant role in public security. However, many existing methods overlook the influence of scenes on anomaly detection. These methods simply label the occurrence of certain actions or objects as anomalous. In reality, scene context plays a crucial role in determining anomalies. For example, running on a highway is anomalous, while running on a playground is normal. Therefore, understanding the scene is essential for effective anomaly detection. In this work, we aim to address the challenge of scene-dependent weakly supervised video anomaly detection by decoupling scenes. Specifically, we propose a novel text-driven scene-decoupled (TDSD) framework, consisting of a TDSD module (TDSDM) and fine-grained visual augmentation (FVA) modules. The scene-decoupled module extracts semantic information from scenes, while the FVA module assists in fine-grained visual enhancement. We validate the effectiveness of our approach by constructing two scene-dependent datasets and achieve state-of-the-art results on scene-agnostic datasets as well. Code is available at https://github.com/shengyangsun/TDSD.

Main paper: 
 introduction: 
 Video anomaly detection (VAD) [1, 17, 23, 24, 31, 42, 56] aims to discriminate the abnormal frames from the given video, has received significant research interest due to its crucial role in miscellaneous domains such as public security and surveillance. Some previous methods [15, 18] explored training models in a fully-supervised setting, however, full supervision requires frame-level or snippetlevel annotations for training, which entails a significant amount of manual labeling costs. Conversely, some researchers utilize unsupervised methods [8, 22, 24, 28] to address this task, only using normal videos to train the model and identifying outliers as anomalies during evaluation. Unfortunately, collecting all normal samples is impractical, and since no abnormal samples are involved in training, the model is prone to producing false alarms during inference. To reduce the cost of data labeling and obtain models with excellent performance, weakly supervised video anomaly detection (WS-VAD) [5, 7, 16, 29, 35, 42, 45, 54, 56] using only video-level annotations for training has recently received widespread attention.In the task of anomaly detection, defining what constitutes an anomaly is crucial. Previous research has primarily focused on a single-scene scenarios [1, 17, 23, 31] or multi-scene scenarios [24, 42, 56] where anomalies exhibit consistent behavior across different scenes, known as scene-agnostic anomalies. However, in the real world, determining whether an event is anomalous often requires considering the scene context in which it occurs, i.e. scenedependent anomalies, also known as scene-aware anomalies. As shown in Figure 1 (a), it is abnormal for bicycles to travel on pedestrian pathways, whereas it is normal for them to travel on roads. By taking into account the scene context, anomaly detection systems can adapt to different environments, enhancing their scalability and applicability in real-world deployment. Therefore, in recent years, there have been several notable attempts at scene-dependent VAD [2, 3, 36, 38] . However, these studies have been conducted in unsupervised setting, which have limited performance. To enhance the practical application capabilities of VAD, in this work, we investigate scene-dependent anomaly detection under a weakly supervised setting.In this work, our focus lies in the meticulous exploration of scene context within videos, enabling the model to possess the capability of detecting scene-dependent anomalies. To better describe the scene information in the video, we classify scene information into context descriptions and object descriptions. Specifically, the context descriptions encompass general scene summaries, such as schoolhouse, palace, and store for general descriptions, which is a macroscopic description of the overall scene. However, solely relying on macroscopic descriptions may make it difficult to distinguish between some similar scenes, such as park scenes and front yards of houses. Determining whether it is a park often requires considering the presence of park benches and fountains. Therefore, when exploring scene information, we simultaneously consider the objects of the scene, such as bicycle, van, and fountain for specific objects. Exploiting scene objects not only enhances the scene's discriminative ability but also incorporates potential abnormal targets, thereby increasing the anomaly detection capability of the model.In real-world VAD tasks, scenes are often complex, and it's difficult to fully capture the semantic meaning of video segments solely through visual features [4] . In recent years, the emergence of vision-language models, e.g. CLIP [30] and ALIGN [12] , has made it possible to combine visual information from videos with natural language descriptions, obtaining richer information from two modalities. This fusion provides a more comprehensive understanding of scenes, thereby aiding in more accurate anomaly detection. Based on this, we propose a text-driven scene-decoupled (TDSD) framework for weakly supervised video anomaly detection, which exploits the context descriptions and object descriptions of scenes based on the pre-trained CLIP model, empowering the model to detect scene-dependent anomalies, as shown in Figure 1(b) .Our contributions are summarized as follows:• We propose a novel text-driven scene-decoupled framework to address weakly supervised video anomaly detection, which exploits the context and objects semantic meanings of scenes in normal and abnormal videos, enabling the model to detect scene-dependent anomalies. To the best of our knowledge, this is the first work to address scene-dependent video anomaly detection under a weakly supervised setting. • We designed the text-driven scene-decouple module (TDSDM) and the fine-grained visual augmentation (FVA) module for extracting semantic features of scenes and enhancing features at a fine granularity, respectively. The TDSDM consists of context semantic injection (CSI) and object semantic injection (OSI), which enable the model to learn the semantic features of scenes and objects within scenes, respectively. • To validate the ability to detect scene-dependent anomalies of our method, we reorganize the scene-dependent anomaly dataset NWPU Campus [3] , which is originally constructed for the one-class classification setting, to suit the weakly supervised setting. Besides, we merged the public sceneagnostic datasets UCF-Crime and ShanghaiTech into one scene-dependent dataset, further validating the effectiveness of the WS-VAD method. Additionally, we conduct experiments on two scene-agnostic datasets to validate the aspects of our designs that are independent of scene awareness. 
related works 2.1 video anomaly detection: 
 Video anomaly detection (VAD) has gained significant research interest, which can be classified three main research lines, fullysupervised [15, 18] , unsupervised [10, 19, 22, 24, 27, 28, 37, 44, 50, 53] and weakly-supervised VAD [5-7, 16, 29, 35, 39-42, 45, 54, 56] . The fully supervised VAD [15, 18] trains a model using framelevel annotations, including precise bounding boxes of anomalies. This necessitates a significant amount of labor to label the data. To alleviate the burden of annotating numerous samples, unsupervised methods focus solely on gathering normal videos for model training. During inference, they differentiate samples deviating from normality as anomalies. For example, reconstructionbased unsupervised techniques [8, 22, 24, 28] employ autoencoders to encode normal samples into latent spaces, identifying poorly reconstructed samples as anomalies. For instance, Yu et al. [50] propose a reconstruction-based method that uses an object detector to identify potential anomalies in videos and focuses on the temporal information of objects. Meanwhile, distance-based unsupervised methods [10, 11, 34, 37] establish decision boundaries using Gaussian mixture models or one-class SVMs, distinguishing deviated data as anomalous samples. However, since abnormal samples are absent from the training set, the model tends to generate false alarms when detecting ambiguous normality. To alleviate the burden of labor-intensive annotations while training models on abnormal data, weakly-supervised video anomaly detection (WS-VAD) [7, 29, 35, 42] tackles this challenge by employing multiple instance learning (MIL). This approach trains the model using video-level annotated data and achieves excellent performance. For example, Lv et al. [25] propose an unbiased multiple instance learning (UMIL) framework to tackle bias in MIL-based VAD, thereby decreasing false alarms and enhancing performance. Wu et al. [48] propose a framework that leverages the pre-trained vision-language model CLIP to address the WS-VAD task. All previous weakly supervised methods focus on scene-agnostic anomaly detection, leaving scene-dependent VAD unexplored. In contrast to them, our approach focuses on addressing both scene-agnostic anomalies and exploring solutions for scene-dependent anomalies.In recent years, scene-dependent VAD has garnered increasing attention due to the growing complexity of surveillance scenarios. The incorporation of scene context into VAD involves several crucial components, such as extracting scene features, identifying scene types, and modeling the relationship between objects and scenes. In previous methods, scene features have been extracted by inputting the entire frame, with or without marked objects, into various encoders [2, 3, 36, 38] . Scene types have been identified primarily using unsupervised clustering algorithms [2, 36, 38] . Additionally, spatiotemporal context graph [36] , hierarchical scene normality-binding model [2] , and scene-conditioned variational auto-encoder [3] are constructed to model the relationship between objects and scenes. In addition, several studies have employed graphs to model scenes and objects. Liu et al. [20] employ three models to construct a factual causal graph for video action recognition. Han et al. [9] propose a method that improves the accuracy of scene graph generation by decomposing the predicate prediction task into subtasks. In contrast, we leverage the pre-trained vision-language model to obtain the scene semantic features and objects semantic features, injecting the semantic features into the spatiotemporal features, being the scene-attended features. Besides, this is the first work to address the scene-dependent VAD under a weakly supervised setting.Vision-language models have been widely applied to various vision tasks. For instance, Liu et al. [21] propose a multi-grained gradual inference model that focuses on objects and aligns textual and visual information through graph structures and its multi-grained gradual inference mechanism. Besides, pre-trained vision-language models (VLM) such as CLIP [30] and ALIGN [12] , which are trained on large-scale datasets, have acquired a wide range of knowledge and demonstrated impressive generalization capabilities. Therefore, in recent years, these models have been extensively deployed to downstream vision tasks through adaptation techniques like fine-tuning and prompt learning [13, 32, 49, 57] . In the context of weakly supervised VAD, Joo et al. [14] leverage the CLIP vision encoder along with a proposed temporal self-attention module to enhance feature quality. Wu et al. [48] adapt CLIP by incorporating an adaptation layer into the CLIP vision encoder and employing a text prompt learning scheme for the CLIP text encoder. Zanella et al. [51] directly manipulate the latent CLIP feature space to identify the normal event subspace. In contrast, our approach utilizes CLIP's powerful zero-shot capability to obtain semantic information about scenes. By employing CLIP's text encoder, we extract semantic features of scenes, which are then integrated with spatiotemporal features. This enables the model to detect scene-dependent anomalies. 
methodology 3.1 overview of the proposed framework: 
 We illustrate the proposed framework in Figure 2 . The framework mainly includes three parts, the text-driven scene-decoupled module (TDSDM) including the context semantic injection (CSI) and the object semantic injection (OSI), the fine-grained vision augmentation (FVA), and the last part is the global encoder that consists of a 𝑁 𝑇 -layer Transformer and a MLP-based regressor. The TDSDM is used to decouple scenes. The CSI module provides a general description of the scene and generates semantic features of the scene, while the OSI module provides descriptions of objects within the scene and generates semantic features of the objects in the scene.The FVA module provides fine-grained visual augmentation, which involves enhancing a snippet-level spatiotemporal feature with 𝑇 frame-level CLIP features. The augmented features finally pass through a global encoder to aggregate context overall snippets in a video and then predict anomaly scores.The spatiotemporal features extraction aims to extract the one spatiotemporal feature for each video snippet that contains 𝑇 consecutive frames (here 𝑇 = 16 following [5, 7, 35, 42, 56] ). Specifically, given the 𝑖-th video snippet of dimension 𝑇 × 𝐻 ×𝑊 × 𝐶 is fed into the pre-trained I3D feature extractor that fine-tuned on Kinetics-400 dataset, extracting the spatiotemporal features F 𝐼 3𝐷 𝑖 ∈ R 𝐷 𝐼 , where the 𝐻 , 𝑊 and 𝐶 are the height, width and channel of each frame, respectively, and 𝐷 𝐼 is the dimension of extracted features. Then the extracted features pass through a linear project layer 𝑓 𝐼 3𝐷 (•) : R 𝐷 𝐼 → R 𝐷 to produce the F 𝐼 3𝐷 𝑖 ∈ R 𝐷 for fitting the subsequent modules. Following [5, 7, 35, 42, 56] , we divide each video into 32 snippets by conducting the average temporal pooling in the training stage.The CSI module is used to obtain a general description of the scene and to fuse the semantic features of the context with the spatiotemporal features. With the help of the powerful detection ability of the CLIP, we use this VLM model to detect the different scenarios in zero-shot for each video snippet. Firstly, the 365 scene categories borrowed from the Places365 dataset [55] are fed into the CLIP's text encoder, achieving the scene weight matrice:EQUATIONwhere the W 𝑆 ∈ R 365×𝐷 𝐶 indicates the weight matrices of scenes, the E 𝑇 (•) denotes the text encoder of CLIP, and the 𝑠𝑐𝑒𝑛𝑒 𝑖 denotes the 𝑖-th category of Places365, e.g. schoolhouse, shopping mall, parking garage, etc. Then we feed the 𝑇 frames into the CLIP visual encoder for each video snippet to obtain the visual representations:EQUATIONwhere the F 𝐶𝐿𝐼 𝑃 𝑖 ∈ R 𝑇 ×𝐷 𝐶 are the visual features of 𝑖-th snippet produced by the CLIP, the E 𝑇 (•) denotes the visual encoder of CLIP, I 𝑖,𝑗 ∈ R 𝐶 ×𝐻 ×𝑊 denotes the 𝑗-th frame of the 𝑖-th snippet. Thus, the matching probabilities between each frame and the scene categories can be obtained byEQUATIONwhere the P 𝑆𝑐𝑒𝑛𝑒 𝑖 ∈ R 𝑇 ×365 denotes the matching probabilities of the 𝑇 frames of the 𝑖-th snippet, ∥ • ∥ 2 is the 𝑙 2 norm, and the ⊗ represents the matrix multiplication operation. We average the probabilities across the 𝑇 frames and take the 𝐾 scene categories which have the top-𝐾 probabilities as the label scenarios:EQUATIONwhere the Avg(•) : R 𝑇 ×365 → R 365 denotes the average operation, and the topK(•) indicates that take the 𝐾 scene categories from the Places365 according to the top-𝐾 probabilities. To obtain the semantic scene feature of 𝑖-th snippet, we put the 𝐾 scene categories into a guided language text T 𝑆 = "The video footage could include the following scenes: {𝑠𝑐𝑒𝑛𝑒 1 }, {𝑠𝑐𝑒𝑛𝑒 2 }, ..., {𝑠𝑐𝑒𝑛𝑒 𝐾 }." , then feeding the above text into the CLIP text encoder E 𝑇 (•) to extract the context semantic feature F 𝑆𝑐𝑒𝑛𝑒 𝑖 ∈ R 𝐷 :EQUATIONwhere 𝑓 𝑠𝑐𝑒𝑛𝑒 (•) : R 𝐷 𝐶 → R 𝐷 is the linear projection layer. Finally, both the spatiotemporal feature F 𝐼 3𝐷 𝑖 ∈ R 𝐷 and the scene clue feature F 𝑆𝑐𝑒𝑛𝑒 𝑖 pass through the fusion layer to produce the sceneattended feature F𝐼3𝐷 𝑖 ∈ R 𝐷 , which is computed by the multi-head cross-attention followed by a feedforward (FFN) layer [43] :EQUATIONwhere the Cross-Attention(x, y) for each head computes dot-product attention as follows:EQUATIONwhere W 𝑄 , W 𝐾 , W 𝑄 ∈ R 𝐷 ×𝐷 𝐻 are learnable matrices with 𝐷 𝐻 = 𝐷/𝑁 𝐻 , where 𝑁 𝐻 is the number of attention heads.The OSI module is used to provide descriptions of objects within the scene and generate semantic features of the objects in the scene. Similar to the design of the CSI module, the OSI module powered by the CLIP detects objects in zero-shot mode. In detail, we feed 1,000 object categories borrowed from the ImageNet-1K [33] into the CLIP's text encoder to obtain the object weight matrice:EQUATIONwhere the W 𝑂 ∈ R 1000×𝐷 𝐶 indicates the weight matrices of objects. With the extracted fine-grained visual feature according to Equation ( 2), the match probabilities between each frame and the object classes are computed byEQUATIONSimilar to the Equation (4), the 𝐾 objects classes are obainted as follows:[𝑜𝑏 𝑗𝑒𝑐𝑡 1 , 𝑜𝑏 𝑗𝑒𝑐𝑡 2 , ..., 𝑜𝑏 𝑗𝑒𝑐𝑡 𝐾 ] = topK(Avg(P)), (10) then these object categories are put into the text T 𝑂 = "The video footage could include the following objects: {𝑜𝑏 𝑗𝑒𝑐𝑡 1 }, {𝑜𝑏 𝑗𝑒𝑐𝑡 2 }, ..., {𝑜𝑏 𝑗𝑒𝑐𝑡 𝐾 }. " , thus the object semantic feature F∈ R 𝐷 for 𝑖-th snippet is extracted as follows,EQUATIONwhere 𝑓 𝑜𝑏 𝑗𝑒𝑐𝑡 (•) : R 𝐷 𝐶 → R 𝐷 is the linear projection layer. The object feature Fand the scene-attended feature F𝐼3𝐷 𝑖 are fused with using the cross-attention, being the object-attended feature:EQUATION3.4 Fine-grained Visual Augmentation (FVA)The purpose of the FVA is to enhance a snippet-level spatiotemporal feature with 𝑇 frame-level CLIP features, thus enhancing the visual expression capability of the features. To this end, we first obtain the fine-grained feature F𝐶𝐿𝐼𝑃 𝑖 computed by the F 𝐶𝐿𝐼 𝑃 𝑖 with the Self-Attention(F 𝐶𝐿𝐼 𝑃 𝑖 ):EQUATIONwhere W 𝑄 , W 𝐾 , W 𝑄 are learnable matrices as the same to Equation (7) . Finally, we can achieve the augmented spatiotemporal feature F𝐼3𝐷 𝑖 ∈ R 𝐷 that augmented by the frame-level fine-grained features as follows:EQUATIONIn the weakly supervised VAD task, each training video is annotated with a binary label 𝑦 ∈ {0, 1} to denote whether this video is an abnormal video or not. When the enhanced features of all snippets within one video are produced, we employ a 𝑁 𝑇 -layer Transformer to model the global context of all snippets, and then we use a regressor to predict the anomaly scores:EQUATIONwhere s ∈ R 𝑁 𝑆 is the anomaly scores of all snippets within one video and Θ(•) : R 𝐷 → R is the regressor that is implemented by a three-layer multi-layer perceptron (MLP). Following the previous works, we adopt the binary cross-entropy loss to train the model, which classifies a video into abnormal or normal classes. Specifically, we average the top-𝑁 𝑆 anomaly scores as follows:EQUATIONwhere T 𝑁 𝑆 (s) indicates the set of top-𝑁 𝑆 scores in s. Then the MIL loss is defined byEQUATION) 
experiments 4.1 datasets and evaluation metric: 
 4.1.1 Datasets. We conduct experiments to fully evaluate our proposed framework based on public datasets for VAD task, Shang-haiTech [19] , UCF-Crime [35] , TAD [26] , XD-Violence(XD) [47] and NWPU campus [3] . The large-scale dataset NWPU is the only one that considers scene-dependent anomalies. It is originally designed for the unsupervised setting, including only normal videos for training. To adapt it to the weakly supervised setting, we reorganize the dataset by selecting both normal and abnormal videos for the training set. Additionally, we ensure that the anomalies present 1 .For the performance evaluation, following the common practice [3, 5, 7, 19, 35, 42, 48, 56] , we adopt the frame-level area under the ROC curve (AUC) as the evaluation metric for TAD, NWPU campus, and UCF_SHT datasets, and use the average precision (AP) for the XD-Violence dataset. A higher AUC or AP denotes better performance.The proposed method is implemented in Pytorch. We employ the pre-trained CLIP (ViT-L/14) backbone to extract the features of F 𝐶𝐿𝐼 𝑃 𝑖 , F 𝑆𝑐𝑒𝑛𝑒 𝑖 and F. The dimension of feature 𝐷 is set to 512. The top-𝐾 in Equation ( 4) and Equation ( 10) is set to 5. The number of Transformer layer 𝑁 𝑇 is set to 2 on UCF_SHT, TAD, and XD-Violence datasets, and set to 5 for the NWPU dataset. The 𝑁 𝑆 in Equation ( 16) is set to 3. We train the proposed model on one NVIDIA Tesla V100 GPU in an end-to-end manner using the SGD optimizer for 50 epochs, the weight decay is set to 0.0005, the batch size is 32 and the learning rate is set to 0.0005 for TAD, UCF_SHT, and XD-Violence datasets and 0.005 for NWPU dataset.To fully validate the effectiveness of the proposed method, we carefully examine the design of each module in our framework and investigate the performance of various model variants on two scene-dependent datasets i.e. UCF_SHT and NWPU, and two sceneagnostic datasets, i.e. XD-Violence and TAD.Effectiveness of the Proposed Modules. The major contribution modules in the proposed framework are text-driven scenedecoupled module (TDSDM) and fine-grained visual augmentation (FVA). Therefore, we first conduct ablation studies to investigate the effectiveness of two modules by leaving out either one or both of them from our proposed framework. When the TDSDM and FVA both are removed, we only use the global encoder to produce the anomaly scores. Performances are reported in Table 2 .The results from the table show that solely employing the TDSDM or FVA both improves the performance. By adding to the TDSDM, we can observe that the improvements on scene-dependent datasets (UCF_SHT and NWPU) are more significant than the performances on the scene-agnostic datasets (XD-Violence and TAD), e.g. 69.73% v.s. 77.90% on NWPU compared with 87.63% v.s. 90.06% on TAD. For the FVA module, the results show that the performances have also been improved not only on the scene-dependent datasets but also on the scene-agnostic datasets. Besides, the synergy of TDSDM with FVA boosts performance significantly.Effectiveness of the Designs in TDSDM. We then conduct experiments to study the effectiveness of the varied components in the core module of TDSDM. This module mainly has two components: 1) the context semantic injection (OSI) and 2) the object semantic injection (OSI). Each above component is validated in Table 3 . The framework contains FVA and a global encoder when removing both two parts. The CSI is employed to obtain the general semantic meaning of the scene. From the results, we can observe that performance has significant improvements by considering the context semantic meaning, especially on two scene-dependent datasets, i.e. from the AUC of 80.94% to 85.61% on UCF_SHT and 70.01% to 78.05% on NWPU. The OSI module is mainly used to obtain the semantic meanings of the objects within the scene. With the help of the OSI (Index 3 in table), performances on all four datasets are improved, e.g. from 88.57% of AUC on TAD to 93.28%. In addition, the model achieves the best performance on all datasets by employing all components of TDSDM.Impact of the hyperparameter top-𝐾. To investigate the impact of the variant 𝐾 in Equation (4) and Equation (10) for selecting the top-𝐾 scenes and top-𝐾 objects, we conduct experiments that vary the 𝐾 from 1 to 20, reporting the results in Figure 3 . Specifically, we choose the 𝐾 = 5 because the model achieves the best performance. Additionally, we observe that the performance varies slights with different 𝐾, e.g. from 93.64% to 93.90% on TAD, which denotes the model is not sensitive to the 𝐾. 4.4.1 Predicted Anomaly Scores. We first visualize the anomaly scores predicted by the proposed framework on test sets of two scene-dependent datasets (UCF_SHT and NWPU) and two sceneagnostic datasets (XD-Violence and TAD), where a higher predicted score indicates a higher probability that the event is anomalous, results are shown in Figure 4 .For the evaluation of scene-dependent datasets, our proposed method effectively detects scene-dependent anomalous events. For example, in the scene-dependent dataset UCF_SHT, the appearance of a bicycle in the scene sh®®own in video 01_0139 is considered an anomalous event, while the appearance of a bicycle in the video Normal_Videos_210 is considered as a normal event. Similarly, in the scene-dependent dataset NWPU, the appearance of a bicycle in the scene shown in video D235_07 is considered an anomalous event, while the appearance of a bicycle in the scene shown in video D031_09 is considered as a normal event. From the results in the figure, we can see that for the abnormal videos 01_0139 and Normal_Videos_210, our model produces high prediction scores for anomalous snippets, while producing low prediction scores for normal videos. In different scenes where the same event occurs, such as the appearance of a bicycle in the two scenes mentioned above, our model generates different anomaly scores. This indicates that our model can detect scene-dependent anomalous events. For scene-agnostic datasets (XD-Violence and TAD), our model also accurately predicts anomalous events while generating low anomaly scores for normal events. Additionally, from the video Bullet.in.the.Head.1990 in XD-Violence, it can be seen that our model can detect discontinuous anomalous events within a video. 4.4.2 Generated Scene Text Results. In the CSI module, we initially use CLIP for scene classification and then generate textual descriptions of scenes to obtain their semantic features. Consequently, we analyzed the generated scene text, i.e. the guided language text T 𝑆 , as illustrated in Figure 7 . We select scenes from four datasets: UCF_SHT, NWPU, XD-Violence, and TAD, and generate textual descriptions of these scenes. From the results of the generated text descriptions, it is evident that the descriptions of the scenes are fairly accurate. For instance, the first image depicts a scene of a store selected from UCF-Crime, and the third image depicts a scene of a campus selected from the NWPU dataset. Both the generated text accurately describes the scenes. . We investigated the precision of the generated scene text in Section 4.4.2. To further confirm the quality of the obtained scene semantic features (whether different scenes are distinguishable in the feature space), we conducted a visualization of the scene semantic features.The scene-dependent dataset NWPU provides scene labels for each video, so we visualized the scene semantic features separately for the NWPU training and test sets. As shown in Figure 6 , it can be observed that most points corresponding to the same scene cluster together and are separated from points corresponding to different scenes. This indicates that the generated scene semantic features can effectively distinguish between scenes, thus addressing the scene-dependent video anomaly detection task.Finally, we compared our proposed method with the state-of-theart (SOTA) weakly supervised VAD methods. Since existing weakly Ours 85.94 80.22 84.69 93.90 supervised approaches have not been tested on scene-dependent datasets, we retrained and tested these methods on two scenedependent datasets using publicly available code from these works, attempting to keep the original settings as consistent as possible during training. Given that our model trained for 50 epochs, to ensure a fair comparison, these retrained models were trained for at least 50 epochs. The performance results are shown in Table 5 .From the comparison results on scene-dependent datasets, our method achieved the best performance on both datasets, indicating that our model can effectively detect scene-dependent anomalies. Furthermore, compared to other SOTA methods, the performance improvement of our model on both scene-dependent datasets is significant. For instance, compared to the UR-DMU [56] , our model improved performance from 80.53% to 85.94% on the UCF_SHT dataset and from 70.46% to 80.22% on the NWPU dataset. It is worth noting that compared to VadCLIP [48] , which also utilizes CLIP, our model shows a significant improvement. For example, on the NWPU dataset, our model achieved a performance improvement of 7.58% compared to VadCLIP. For the scene-agnostic datasets XD-Violence and TAD, our proposed method also achieved the best performance. On the TAD dataset, our method surpasses by ∼1% AUC compared to the second-best performing method, UMIL [25] , which also utilizes CLIP. Taking into account the results above, our method is effective not only on scene-dependent datasets but also achieves good performance on scene-agnostic datasets. 
conclusion: 
 In this work, we propose a text-driven scene-decouple (TDSD) framework to address weakly supervised video anomaly detection (VAD). By utilizing CLIP to decouple scenes and inject semantic features of scenes and objects separately into the model, we endow the model with the capability to handle scene-dependent anomalies. This paper is also the first work to address scene-dependent weakly supervised VAD. To better evaluate our proposed framework, we reorganized the scene-dependent dataset NWPU to suit a weakly supervised setting. Additionally, we merged the UCF-Crime and ShanghaiTech datasets into the scene-dependent UCF_SHT dataset to facilitate a more comprehensive evaluation. Experimental results demonstrate that our approach achieves significant improvements compared to other methods on two scene-dependent datasets, UCF_SHT and NWPU. Furthermore, it achieves the best performance on two scene-agnostic datasets, XD-Violence and TAD. 
: 
 Figure 1 : (a) Scene-agnostic anomalies refer to events that are independent of the scene in which they occur. However, scene-dependent anomalies require consideration of the scene in which the event occurs. (b) Previous weakly supervised methods were unable to detect scene-dependent anomalies, but by injecting semantic information about scenes, we can achieve the detection of scene-dependent anomalies. 
 
 Appendix:  


Decision: This paper introduces a text-driven scene-decouple framework for weakly supervised video anomaly detection. There were mixed ratings initially (two positive and one negative). After the rebuttal, two reviewers were satisfied with the rebuttal, with one improving the rating. The major concern raised by DSqK was not well explained/justified, which can be easily addressed based on existing weakly supervised video anomaly detection frameworks. Hence, the AC agrees with the major opinions on accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a text-driven scene-decouple framework to address weakly supervised video anomaly detection. The model is enhanced by utilizing CLIP to decouple scenes and  semantic features of scenes and objects to handle scene-dependent anomalies.
strength_weakness: strength_weakness: The paper is well organized and eassy to follow.
limitations: limitations: 1. One key of this paper is 'the influence of scenes on anomaly detection', however, the study lacks practical contribution.
For those artificially defined abnormal situations, i. g. bicycles are regarded as anomalies in the ShanghaiTech dataset, while they are regarded as normalities in the Ubnormal dataset, the pre-trained model is lack of generality, because it can’t predict the normal and abnormal events defined by others.  For those common abnormalities, i.g. speeding and fighting that are considered as abnormal events in most scenes, there is essentially no need to study the impact of the scene on such types of anomalies.

2. The original text of Chapter 4.1.1 is ‘ Additionally, we ensure that the anomalies present in the test set are also included in the training set, which is a fundamental assumption in scene-dependent VAD ’. It seems to contradict the original purpose of anomaly detection tasks. Due to the low occurrence probability and diversity of anomaly events, most anomaly detection algorithms have only normal samples for training. Furthermore, test set cannot cover all possible anomalous situations. Therefore, the applicability of a model trained in this way to real-world scenarios is a problem.

3. I understand the decrease in computation speed that arises from employing the CLIP model to augment the precision of anomaly detection. Nonetheless, I think it remains essential to explicate the algorithm's execution speed.

4. The ablation study in this paper does not mention the role of ‘the second CLIP text encoder’ module in CSI and OSI. Can the second CLIP text encoder be replaced with a simpler NLP model to reduce the computational complexity instead of inputting the entire sentence into the entire CLIP text encoder.
paper_summary: paper_summary: The work focuses on scene-dependent anomalies under a weakly supervised setting. The authors construct scene-dependent benchmarks by reorganizing the NWPU dataset and merging UCF-Crime and ShanghaiTech datasets. They propose the Text-Driven Scene-Decoupled Module (TDSD) to inject scene information by leveraging descriptions of both the scene and object, processed with CLIP. Visual features are then enhanced with the proposed Fine-grained Visual Augmentation (FVA). Main and ablated experiments on popular benchmarks demonstrate the effectiveness of the proposed modules, particularly on the curated scene-dependent ones.
strength_weakness: strength_weakness: 1. The paper investigates an interesting yet challenging problem of video anomaly detection, where anomalies are scene-dependent.
2. The paper is well-written and easy to follow.
3. The experiments are comprehensive, justifying the module design.
4. Significant gains are obtained in curated scene-dependent datasets including UCF-SHT and NWPU.
limitations: limitations: 1. The order of injecting scene/object information is doubtful. Scene information helps object detection and vice versa. Can we swap the order of CSI and OSI.
2. The method is limited by the accuracies of scene/object classification. It would be interesting to see how the performance changes when using a fewer number of scene/object categories in zero-shot classification.
paper_summary: paper_summary: The manuscript mentioned that many existing methods currently ignore the impact of scenes on anomaly detection. The author believes that scene context plays a crucial role in judging anomalies. The manuscript proposes a novel text-driven scene decoupling (TDSD) framework to solve the challenge of scene-related weakly supervised video anomaly detection by decoupling scenes. The TDSD method consists of TDSD and fine-grained visual enhancement (FVA) modules. The scene decoupling module extracts semantic information from the scene, while the FVA module facilitates fine-grained visual enhancement.
strength_weakness: strength_weakness: The manuscript focuses on the impact of scene information on anomaly detection, which is meaningful for anomaly detection research.
The motivations proposed in the manuscript for using scene and action information are appropriate.
The experiments in the manuscript are relatively sufficient and can also illustrate the effectiveness of the proposed method.
limitations: limitations: The name of the method framework proposed in the manuscript is the same as the module name, which can easily confuse readers.
The content in Figure 2 is very small and not easy to read. It would be better if the content could be made larger. This is just a suggestion. 3.The visualization part shows the effect of effective anomaly detection using scene information and target information. Should some failure examples be shown? 4.The manuscript also lacks discussion of the following latest related work. It is recommended to enrich the writing of related work:

[1] Cloze test helps: Effective video anomaly detection via learning to complete video events[C]//Proceedings of the 28th ACM international conference on multimedia. 2020: 583-591.

[2] Multi-grained Gradual Inference Model for Multimedia Event Extraction[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

[3] Divide-and-Conquer Predictor for Unbiased Scene Graph Generation. IEEE Trans. Circuits Syst. Video Technol. 32(12): 8611-8622 (2022)

[4] A knowledge-based hierarchical causal inference network for video action recognition[J]. IEEE Transactions on Multimedia, 2024.