Video Anomaly Detection via Progressive Learning of Multiple Proxy Tasks

Authors: menghao  zhang
jingyu  wang
qi  qi
pengfei  ren
haifeng  sun
zirui  zhuang
huazheng  wang
lei  zhang
jianxin  liao
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-20 09:22:53

Abstract: Learning multiple proxy tasks is a popular training strategy in semi-supervised video anomaly detection. However, the traditional method of learning multiple proxy tasks simultaneously is prone to suboptimal solutions, and simply executing multiple proxy tasks sequentially cannot ensure continuous performance improvement. In this paper, we thoroughly investigate the impact of task composition and training order on performance enhancement. We find that ensuring continuous performance improvement in multi-task learning requires different but continuous optimization objectives in different training phases. To this end, a training strategy based on progressive learning is proposed to enhance the efficacy of multi-task learning in VAD. The learning objectives of the model in previous phases contribute to the training in subsequent phases. Specifically, we decompose video anomaly detection into three phases: perception, comprehension, and inference, continuously refining the learning objectives to enhance model performance. In the three phases, we perform the visual task, the semantic task and the open-set task in turn to train the model. The model learns different levels of features and focuses on different types of anomalies in different phases. Additionally, we design simple yet effective semantic leveraging the semantic consistency of context. Extensive experiments demonstrate the effectiveness of our method, highlighting that the benefits derived from the progressive learning transcend specific proxy tasks.

Main paper: 
 introduction: 
 Video Anomaly Detection (VAD) is an essential task in multimedia interpretation, referring to the detection of unexpected events in videos. The primary challenge of VAD is the sparsity of abnormal samples, limiting the possibility of training efficient detectors in a fully supervised manner. Consequently, previous research in VAD predominantly falls into two categories: weakly supervised methods [8, 30, 41, 44, 48, 56, 60] that learns with video-level annotations, and semi-supervised methods that only learn from normal frames Figure 1 : AUC (%) performances of anomaly detection when learning multiple pretext tasks on the Avenue [27] and Campus [3] datasets. Dashed lines indicate learning multiple proxy tasks simultaneously. Solid lines indicate training different tasks at different phases. "Pre", "Rec", and "Cls" represent the visual proxy tasks "prediction", "reconstruction", and "classification", respectively. "SR" represents the semantic proxy task "semantic reconstruction". "VL" represents the open-set proxy task "vision-language".[5, 25, 26, 43, 50] . Our work focuses on the latter, which aligns with real-world settings.Semi-supervised methods typically employ single or multiple proxy tasks to model normal patterns, with samples deviating from the established model labeled as anomalies. The single-proxy task method [5, 24, 35, 47, 50] is simple and effective when dealing with data involving a single scenario. In order to make the detector more adaptable to the complex and dynamic scenarios of the real world, some works [10, 11, 25] attempt to introduce multi-task learning to guide the model in acquiring distinct features for different types of anomalies. However, recent research [39] indicates that optimizing multiple proxy tasks simultaneously may not result in the expected performance enhancement. As shown by the dashed lines in Fig. 1 , it could potentially be inferior to the performance of performing the single proxy task.To ensure that multi-task learning achieves the desired performance improvements for VAD, a straightforward approach is to learn multiple proxy tasks sequentially, rather than attempting to learning them simultaneously. As depicted in Fig. 1 , the sequential learning strategy yields superior performance gains compared to both the single-proxy task design and the simultaneous learning of multiple tasks, across datasets representing both single and complex scenarios (the Avenue [27] and Campus [3] datasets, respectively). However, not every set of proxy tasks and any training order can guarantee continuous performance improvement. Therefore, we ask the following questions in this paper: for the training of video anomaly detector, (i) which proxy tasks are necessary, and (ii) what sequence of training is reasonable and effective.In fact, proxy tasks and training sequence are not completely decoupled. To ensure effective performance enhancement, proxy tasks in different training phases should entail distinct optimization objectives. To maintain the continuity of performance improvement, the optimization objectives across different training phases should be be consecutive. In a word, the selection of proxy tasks and the training sequence should complement each other, different training phases require a series of consecutive optimization objectives. Therefore, the ideal approach is progressive learning rather than simply sequential learning, i.e., the learning objectives of the preceding phase contributing to the training for subsequent tasks.Inspired by the human cognitive process, we decompose video anomaly detection into three phases: perception, comprehension, and inference. As shown in Fig. 2 , our progressive learning design is a multi-phase training strategy for the purpose of anomaly detection, continuously refining learning objectives to tackle more challenging tasks. In the perception phase, the model aims to learn low-level visual information, laying the foundation for understanding the scene. In the comprehension phase, the model aims to learn semantic features in the scene, understanding the activities within the scene. In the inference phase, the model utilizes the learned knowledge and context to detect anomalies. Additionally, the decomposition approach enables the model to learn different levels of features at different phases, focusing on different types of anomalies. In the perception phase, the model focuses on obvious scene-independent anomalies (such as fire, smoke). In the comprehension phase, it combines scene information to address scene-dependent anomalies. In the inference phase, the model learns general features to address unseen anomalies encountered during training.Correspondingly, Answer to Question 1: The training of detector requires three types of proxy tasks: the visual proxy task for perceiving pixel features, the semantic proxy task for comprehending semantic features, and the open-set proxy task for learning general features for inference. Answer to Question 2: The training sequence begins with visual proxy tasks, followed by semantic proxy tasks, and concludes with open-set proxy tasks. To this end, we adopt the widely-used frame prediction task as the visual proxy task in the perception phase. As for the semantic proxy tasks in the comprehension, we propose a simple yet effective semantic reconstruction task leveraging the semantic consistency of context. During the inference phase, we adhere to previous work [26] by training the model with virtual data [1] to enhance its ability to detect unseen anomalies.We conduct experiments on four challenging datasets (Avenue [27] , ShanghaiTech [28] , UCF-Crime [41] and Campus [3] ). The experiments show that our proposed method outperforms the State-Of-The-Art (SOTA) methods and enhances the performance of multi-task learning. Furthermore, extensive experiments with widely-used proxy tasks demonstrate that the benefits derived from progressive learning extend beyond specific proxy tasks. In summary, our contributions can be summarized as follows:• A training strategy based on progressive learning is proposed to enhance the efficacy of multi-task learning in VAD. We demonstrate the necessity of reasonable task composition and training sequences, rather than simply learning multiple tasks simultaneously or sequentially. • Video anomaly detection is decomposed into three phases: perception, comprehension and inference. The model focuses on different types of anomalies in different phases.• Extensive experiments demonstrate the effectiveness of our method, highlighting that the benefits derived from the progressive learning transcend specific proxy tasks. 
related work 2.1 video anomaly detection: 
 Previous semi-supervised anomaly detection methods can mainly be classified into two categories: single-proxy-task [4, 17, 19, 35, 47] and multi-proxy-task methods [10, 11, 25] . Specifically, the term "semi-supervised" refers to methods that only use normal data during training. Most semi-supervised methods train models to perform a single proxy task on normal samples and assume that the model cannot perform the proxy task well on anomalous samples. The reconstruction task [12, 15, 29, 33, 58] to reconstruct the current frame and the prediction task [5, 23, 46, 53] to predict future frames are two commonly used proxy tasks, where frames with large reconstruction or prediction errors are recognized as anomalous. In addition, some work synthesize virtual anomaly data [1, 26] or proposes more complex proxy tasks to learn semantic features of normal patterns, including event completion [52] , jigsaw puzzles [45] , event restoration [50] , and causal consistency [24] . While the single-proxy-task design yields promising results on datasets featuring a single scene, its performance falls short of expectations in real-world scenarios characterized by diverse scenes. This discrepancy arises from inconsistencies between the tasks during training (such as frame reconstruction or frame prediction) and those encountered during testing (such as anomaly detection). Therefore, recent works [9-11, 25, 31, 43] utilize multiple proxy tasks to train the model. For instance, Some works introduce information from other modalities, such as optical flow [25, 31] and skeletal information [43] , where proxy tasks for each modality are jointly used as learning objectives. Some scene-aware methods [2, 43, 55] separately design proxy tasks for scenes and objects to learn multi-level features. Meanwhile, other research [9] [10] [11] 43] designs an shared backbone with multiple task heads to simultaneously learn multiple proxy tasks targeting different anomalies.However, the exploration of leveraging multiple proxy tasks to improve anomaly detection performance has not been thoroughly investigated in these studies. The challenge lies in the fact that simultaneously learning multiple proxy tasks can often lead to convergence to suboptimal solutions. Different from these methods, we extensively explore the impact of task composition and training sequence on multi-task learning in VAD. We find that in order to ensure continuous performance improvement, different but consecutive optimization objectives in different training phases are necessary. Based on this principle, we propose optimizing multiple proxy tasks through the progressive learning strategy. In addition, the gains from progressive learning are not limited to specific proxy tasks and their performance.Multitask learning [18, 20, 21, 32] is attracting increasing attention in the field of deep learning. Some methods achieve simultaneous learning of multiple tasks by designing complex network architectures [16] and optimization strategies [14, 38] . These methods are supervised and aim to improve the overall performance of multiple tasks. Unlike these methods, we aim for continuous performance improvement of the model as tasks are added. To achieve this, we perform different tasks at different training phases, progressively enhancing the model's performance. Progressive learning offers different but continuous optimization objectives at different training phases, with learning from previous phases aiding training in subsequent phases.In video anomaly detection, prior work [39] attempts to sequentially learn multiple proxy tasks to ensure performance enhancement from multi-task learning. They arrange the training order based on the difficulty of tasks. However, continuous performance improvement is not available for sequential learning in all combination of tasks. Instead of executing the proxy tasks sequentially, we implement them in phases at various levels to ensure that the objectives are distinct yet consecutive. Specifically, inspired by the cognitive principle of understanding from the surface to the core, we decompose video anomaly detection into perception, comprehension, and reasoning phases. The model learns different proxy tasks at each phase to progressively acquire features at different levels. This decomposition allows the model to focus on different types of anomalies at different phases. 
method 3.1 overview: 
 Our proposed video anomaly detection framework progressively learns multiple proxy tasks to ensure that the model performance continuously improves as more tasks are learned. We aim to design a series of different and consecutive optimization objectives in different training phases, so that the feature learned in the previous training task contributes to the subsequent training. Specifically, we decompose video anomaly detection into three consecutive phases: perception, comprehension, and inference. The overall design of our proposed progressive learning is described in Table 1 . For a normal video clip 𝑉 : In the perception phase, we perform visual proxy tasks to learn pixel features of normal patterns, such as color, shape, and optical flow. In the comprehension phase, we perform semantic proxy tasks to learn semantic features of normal patterns, such as spatial features, temporal features, and scene semantics. In the inference phase, we perform the open-set proxy task to learn more discriminative general features. The open-set approach in VAD refers to the use of synthetic virtual anomalies or the acquisition of pseudo-anomalies to supplement the supervision instead of training with only normal data. In the process of progressive learning, the pixel features learned by the model in the perception phase provide the basis for comprehension, while the semantic features learned in the comprehension phase help the model to learn genreal features.The network comprises three task heads and an encoder 𝐹 . All task heads share the same backbone encoder 𝐹 to leverage knowledge learned from the previous phases, and all proxy tasks within the same phase share the same task head 𝐻 . Next, we illustrate our progressive learning strategy with examples of frame prediction , semantic reconstruction, and virtual data-based classification tasks.Pre-trained video parsing networks are widely used in VAD methods [10, 11, 25, 43] to extract different visual information. In this work, we utilize pre-trained networks to organize the training data.In the perception phase, we perform the prediction task to learn pixel features. Given a video clip 𝑉 , we use YOLOv3 [34] to extract all foreground objects in the video frame. Each object is identified by an RoI bounding box. For each RoI, we construct a spatio-temporal cube (STC) that contains not only the objects in the frame, but also the contents of the same bounding box from the previous 𝑡 frames, where 𝑡 = 4. The width and height of the STCs are resized to 48.In the comprehension phase, we perform the semantic reconstruction task to learn semantic features. In order to simultaneously learn the semantic features of the scene, we extracted the scene features of the background. For each frame in the clip 𝑉 , we use DeepLabV3+ [6] to generate segmentation maps while masking foreground object categories. We then perform max-pooling, reshaping, averaging, and 𝑙 2 normalization on all segmentation maps to obtain a scene feature 𝑓 𝑠𝑐𝑒𝑛𝑒 ∈ R 𝐷 𝐵 , where 𝐷 𝐵 depends on the size of the video frame.In the inference phase, we train the classification task with virtual data to learn highly discriminative features. Following previous work [26] , we use the virtual dataset UBnormal [1] . Unlike the previous two phases, the initial input to the model is video frames rather than object-level STCs.We employ the U-net [39] with deleted skip connections as the backbone network 𝐹 and use it as a shared base for the proxy tasks.Each training phase introduces into its top a task head, denoted 𝐻 𝑝𝑒𝑟 , 𝐻 𝑐𝑜𝑚𝑝 , and 𝐻 𝑖𝑛𝑓 , which are used to process the feature embeddings extracted from the backbone network. All task heads contain spatio-temporal averaging pooling layers to ensure that the dimensionality of the deep feature embeddings remains consistent, even though the temporal lengths may be different.In perception phase and comprehension phase, we use task heads to embed the video features extracted by the backbone network into the low-dimensional features to get the representation. To be specific, the pixel feature representation and the semantic feature representation are obtained in the prediction task and the semantic reconstruction task, respectively. In inference phase, we use fully connected layers as classifiers at the end of 𝐻 𝑖𝑛𝑓 for performing classification task with virtual data. It should be noted that the parameters of the backbone are trainable. Since the learning objectives of previous phases contribute to subsequent training, retaining knowledge from earlier proxy tasks is advantageous. Although there are three parallel task heads in our architecture, only one task head performs the corresponding proxy task to optimize the shared backbone in each training phase.Perception Phase. We perform the frame prediction task as the visual proxy task to learn pixel-level features. We train our model to predict future frames based on a clip 𝑉 consisting of 2𝑇 + 1 consecutive frames. In order to fully learn the pixel features of the normal model and provide a basis for subsequent learning. We train the model to predict future frames in two directions, forward and backward.Formally, We construct a forward clip- → V = [X 1 , X 2 , . . . , X 𝑇 ] and a backward clip ← - V = [X 2𝑇 +1 , X 2𝑇 , .. . , X 𝑇 +2 ] as the initial inputs. X does not represent the video frames in the clip, but the combination of all the spatio-temporal cube in each frame. The object-level 𝑋 𝑇 +1 of the middle frame is the prediction target. We use the shared backbone 𝐹 to encode these two clips first and use the prediction header 𝐻 𝑝𝑒𝑟 to predict 𝑋 𝑇 +1 . The final prediction X is calculated as:EQUATION.(EQUATIONwhere (• ) represents the process of feature processing by the network. We use the 𝑙 2 normalization to evaluate the difference between the prediction and the ground truth 𝑋 𝑇 +1 . The prediction loss L 𝑝𝑟𝑒 is given as:EQUATIONTo generate smooth prediction frames, we also utilize gradient loss. The gradient loss L gd is calculated as:EQUATIONwhere ∇ and |• | 1 represent gradient computation and 𝑙 1 normalization, respectively. The prediction loss L pre and gradient loss L gd are the optimization objectives in this phase.Comprehension Phase. In order to further learn the higher-level semantic features of the video clip 𝑉 , we introduce a semantic reconstruction task as the semantic proxy tasks. We design a contrastive learning mechanism to perform semantic reconstruction task based on the following facts. In the context of video frames, frame 𝐼 𝑎𝑛𝑐 should exhibit semantic consistency with its neighboring frames due to their temporal continuity, as opposed to disjoint frames.To perform the proposed semantic reconstruction tasks, we select positive video clips and negative clips containing only one anchor frame 𝐼 𝑎𝑛𝑐 , respectively. The negative clip contains 𝑇 continuous frames, while the positive clip has a length of 𝑇 -1. The reason for this phenomenon is that we use masking to select the anchor frame. Given an initial continuous clip of length 𝑇 , we uniformly select a frame from the interval [2,𝑇 -1] as the anchor frame 𝐼 𝑎𝑛𝑐 . And the leading and alternate of the missing frames are concatenated together to form positive video clip 𝑉 𝑑 .Accordingly, we learn the motion information in the video by triplet loss [37] taking the discontinuous video clips 𝑣 𝑑 𝑖 as the anchor point, the masked frames 𝐼 𝑚 as the positive samples, and the continuous video clips 𝑣 𝑐 𝑖 as the negative samples. Defining the cosine similarity computation operation as 𝑠𝑖𝑚(• , • ), the semantic reconstruction loss is denoted as:EQUATIONEQUATIONEQUATIONwhere 𝑝 + 𝑖 and 𝑝 - 𝑖 denote the similarity between positive sample pairs and negative sample pairs in the triplet loss, respectively. [•] denotes the concatenation. The semantic reconstruction loss L 𝑆𝑅 is the optimization objective in comprehension phase. Inference Phase. With the help of virtual data containing anomalous data, we use the classification task as the open-set proxy task. With 𝑁 samples in a training batch, cross-entropy loss L 𝐶𝐸 is used to optimize the model, and the optimization objectives in this phase is L 𝑐𝑙𝑠 :EQUATIONwhere (• ) represents the process of feature processing by the network, 𝐹 and 𝐻 𝑖𝑛𝑓 represent the shared backbone network and the task head of inference phase, respectively, and 𝑣 𝑛𝑜𝑟 𝑖 and 𝑣 𝑎𝑛𝑜𝑚 𝑖 represent the 𝑖 𝑡ℎ normal and anomalous frame in the virtual data, respectively.Table 2 : Ablation experiments on the multi-task learning. We report the AUC (%) scores on ShanghaiTech and Campus datasets. 'FP', 'Rec', 'SR', 'Jig', and 'VL' stand for the proxy tasks of frame prediction, reconstruction, proposed semantic reconstruction, jigsaw puzzle, and vision-language, respectively. In addition, "Vir" stands for we use virtual data for classification training. During testing, we first calculate the prediction error 𝐿 𝑝𝑟𝑒 for each frame in clip 𝑉 , calculated as:EQUATION)We then assign the anomaly score of any frame in which no salient objects are detected to 0. For a frame with 𝑚 salient objects, the maximum anomaly score 𝑆 𝑚𝑎𝑥 between these objects is:EQUATIONFollowing [39] , frame-level anomaly scores are further smoothed by a median filter with a window size of 17 to ensure temporal consistency of the video. 
experiments 4.1 experimental settings: 
 Scene-Independent Anomaly Dataset Avenue [27] contains 16 training videos and 21 test videos with 47 abnormal events, including running and throwing. The scenes in this dataset are single and the anomalies are related to human. ShanghaiTech [28] has 13 scenes with complex lighting conditions and different perspectives. In addition, the dataset includes anomalies caused by sudden movements, such as chasing and arguing. The different perspectives and the unfixed position of the camera lead to a large variation of both the object scale and the anomaly scale in the scene. UCF-Crime [41] comprises 13 anomaly types, spanning a total of 128 hours of video footage. The training set contains video-level labels, whereas the testing set includes frame-level labels. Scene-Dependent Anomaly Dataset. Campus [3] is currently the most challenging dataset in its field with 43 scenes, 28 classes of anomalous events and 16 hours of videos. Especially, it contains scene-dependent anomalies, which means an normal event may be abnormal in another scene. Detecting scene-dependent anomalies requires the model to understand the scene and learn semantic features rather than overfit.Table 3 : AUC (%) performance of models trained with different combinations of tasks using different learning sequences and models trained simultaneously with the same weights for all tasks on the ShanghaiTech and Campus datasets. The results indicate that our progressive learning approach achieves the maximum performance improvement when learning multiple auxiliary tasks, and the performance gains are not limited to specific tasks. Evaluation Metrics. The area under the ROC curve (AUC) serves as a commonly used metric for evaluation and comparison. A higher AUC score indicates a better anomaly detection capability.We train and evaluate our method with an NVIDIA RTX 3090 GPU.In the training phase, we resize the resolution of all input video clips to 256 × 256 pixels, while the values of the pixels in all frames are normalized to the range [0, 1]. For the pre-training of the three proxy tasks, we utilize AdamW as the optimizer while the length of the continuous video clips is set to 𝑇 = 9 frames. The initial learning rate is set to 0.0003 and is gradually decayed following the scheme of cosine annealing. In our reported experimental performance, the shared backbone network 𝐹 consists of a convolutional layer and three convolutional blocks. The structure and all experiments are implemented in PyTorch.Given the diversity of perspectives and scenes within the Shang-haiTech and Campus datasets, we conduct comprehensive ablation experiments on both datasets.In addition, to demonstrate that progressive learning is not limited to specific tasks, we also conduct experiments with other proxy tasks besides the three mentioned above. For the visual proxy task, we select the reconstruction [39] and classification [39] tasks. For the semantic proxy task, we select the event completion [52] and puzzle task [45] . For the open-set proxy task, we select the background-agnostic [11] of synthesizing pseudo anomalies and the Vision-Language task [49] of generating novel anomalies using pre-trained multi-modal model. Preliminary Experiments on Multi-Tasks Learning. As shown in Fig. 1 , simultaneous multi-task learning in VAD may lead to model convergence to the sub-optimal point. We perform experiments using more proxy tasks and the results are reported in Table 2 . According to the experimental results, executing all proxy tasks does not yield the best performance (ID 8, 9). In fact, its performance is inferior to executing a single proxy task (ID 1). The experiments for IDs 6, 8 achieve superior performance with fewer proxy tasks. Notably, the experiment with ID 9 showed improved performance on the Campus dataset compared to experiments with IDs 1 and 7. However, the performance on the ShanghaiTech dataset decreased, indicating that the model's boost in performance on one task came at the expense of performance on other tasks. In addition, executing visual, semantic and open-set proxy tasks simultaneously will all yield better performance (ID 6, 8). Ablation Study on Proxy Tasks Composition. As shown in Table 3 , we conduct ablation experiments with different combinations of tasks. Models in which all three types of proxy tasks are performed tend to obtain better performance when the training order is consistent. The performance of the model is significantly lower when a phase of the proxy task is missing. This improvement is not limited to specific tasks. When there are more semantic proxy tasks, the model achieves better performance on the Campus dataset, which is attributed to the fact that the Campus dataset contains mainly scene-dependent anomalies. Ablation Study on Learning Sequences. As shown in Table 3 , only the models that followed this training order of "Visual proxy tasks → Semantic proxy tasks → Open-set proxy tasks" are continuously improving in performance. This is because progressive learning provides different but consecutive learning objectives. The learning objectives of the previous task help to learn the subsequent learning objectives. When the training sequence is not progressive learning, the model's performance will show a decrease rather than a continuous increase during training. Sensitivity to the Number of Proxy Tasks in Each Phase. In Section 3, we perform only one proxy task at each phase in order to succinctly present the training strategy for progressive learning. In fact, the performance of the model improves as more tasks are performed at each phase (there is an edge effect). As shown in Table 3 , following the learning sequence of progressive learning, the performance of the model continuously improves with the increase of tasks. However, there is a marginal effect of performance growth with each phase of the task. [27] for scene-independent anomalies, Campus dataset [3] for scene-dependent anomalies, UBnormal dataset [1] for unseen anomalies). The proxy tasks we perform in each of the three phases are frame prediction, semantic reconstruction and applying virtual data for training. Note that the virtual data we use for training is the training set of UBnormal to avoid the problem of data leakage. As shown in Table 5 , after training on a particular phase, the model achieves significant improvements on the corresponding dataset.We compare the proposed framework with SOTA methods with AUC(%). It is worth noting that only a few works [1, 11, 26] in the current literature utilize virtual data for feature learning (marked with * in Table 6 ). To ensure fair comparison, we report the performance of models trained with and without virtual data (i.e., whether open-set proxy tasks are performed during training). Furthermore, to fully demonstrate the effectiveness of progressive learning, we train models using two sets of proxy tasks and report performance accordingly. 6 , our method achieved the highest AUC scores, obtaining a AUC of 94.5%. With the simple proxy tasks, our method still scored the best performance with 93.6% AUC scores. Results on ShanghaiTech. Our proposed progressive learning method yields a AUC of 88.6% on the ShanghaiTech dataset. With the virtual data [26] , we improve the AUC by 1.9% and by 3.8% when performing widely-used proxy tasks instead of simple tasks. With the simple proxy tasks, our method still achieves an AUC score of 83.8%, which is superior to most methods. In comparison with the method [39] sequentially learning multiple proxy tasks, our method achieves a significant improvement of 10.0%. The huge improvement over sequential learning and methods that learn 7 , our proposed method achieves an improvement compared to the second best method by 4.4% when only learning simple tasks, and 7.7% when learning complex tasks. Results on Campus. As shown in Table 8 , our method achieves the highest AUC score of 73.3%. Without the complex proxy tasks, our method still obtain an AUC of 69.4%, surpassing other multi-task methods [10, 13, 25, 33] .Compared to the Avenue dataset, which features a single scene, the latter three datasets contain a greater variety of scenes and anomaly types. The superior performance of progressive learning on the four datasets and the improvement on the latter three datasets demonstrate its effectiveness. 
conclusion: 
 In this paper, we thoroughly investigate the impact of task composition and training sequence on the performance improvement of video anomaly detection in multi-task learning. To ensure sustained performance improvement as the number of tasks increases, we propose progressive learning of multiple proxy tasks. Progressive learning provides different but continuous optimization objectives at different phases of training, allowing knowledge learned in previous phases to contribute to subsequent training phases. Specifically, we decompose video anomaly detection into three training phases: perception, comprehension, and inference. Extensive experiments on four challenging datasets not only demonstrate the effectiveness of our proposed method but also show that the gains brought by progressive learning are not limited to specific tasks. 
 
 Appendix:  


Decision: This paper got 1 weak accept, 3 borderline accept, and 2 borderline reject.  The Pros and Cons are as follows:

Pros:
1. This paper is well-organized and easy to follow.
2. The idea of progressive learning of multiple proxy tasks is novel in video anomaly detection (VAD). 
2. The experiments demonstrate the efficacy of progressive learning in VAD.

Cons:
1. The motivation is not sufficient sufficiently supported which needs to further clarified.
2. The training paradigm is overall complex and the theriacal justification of the proposed method is weak.
3. Numerous writing.

After reading the rebuttals and the paper, I recommend an accept based on the novelty of the proposed method and holds potential for applications other than anomaly detection.

Reviews: review: paper_summary: paper_summary: This study examines how task composition and the order of training impact the performance of video anomaly detection within a multi-task learning framework. The researchers propose a novel approach called progressive learning for multiple proxy tasks. This method provides distinct yet continuous optimization goals throughout various training stages, allowing knowledge from earlier phases to improve subsequent training phases. Extensive experiments are conducted to demonstrate the proposed method's effectiveness. The paper is well-structured, clearly written, and the experimental results convincingly support the method's efficacy.
strength_weakness: strength_weakness: This paper is well-organized and easy to follow.
The experiments are sufficient to demonstrate the efficacy of progressive learning using a multi-phase training strategy for video anomaly detection. The results show that continuously refining learning objectives helps tackle more challenging tasks. The motivation is straightforward, the network is well-designed, and each task is explained with clear motivation. Overall, the motivation is sufficient for an MM conference paper.
limitations: limitations: 1. On page 6, line 626, executing all proxy tasks does not yield the best performance (ID 8, 9). Please provide more detailed reasons for this outcome.
2. On page 6, the paper discusses progressive learning for video anomaly detection within a multi-task learning framework. Are there any criteria to automatically combine the multi-learning objectives instead of manually crafting them, while still achieving performance gains? Additionally, the paper states, "the performance of the model continuously improves with the increase of tasks." Should this conclusion consider which learning tasks to include?
paper_summary: paper_summary: This paper investigates the task of video anomaly detection. It begins by empirically demonstrating that learning with multiple proxy tasks, either sequentially or simultaneously, does not consistently yield improvements. To address this, the training phase is decomposed into three stages, progressively learning features from the pixel level to the semantic level. The results show the superiority of the proposed method.
strength_weakness: strength_weakness: There are several strengths in this paper:

- Overall, the idea of this paper is easy to comprehend and technically sound.

- The evaluation provides substantial evidence demonstrating the effectiveness of the proposed method, which is convincing.
limitations: limitations: However, I still have the following concerns:

- The motivation of this paper is not sufficiently supported. It is unclear if the shortcomings of sequential learning and multi-task learning are consistent across different methods and how severe they are. Although I agree with the paper's statements intuitively, more empirical evidence is needed.

- The overall training paradigm is overly complex, involving many engineering challenges, making it less practical for real-world scenarios.

- In lines 268-269, it states, "Unlike these methods, we aim for continuous performance improvement of the model as tasks are added." If tasks are added sequentially, why not compare with continual learning methods rather than multi-task learning?

- The introduction of multi-task learning in the related works section is too brief. The authors emphasize their progressive strategy more, which is not helpful for the audience.

- Since there are multiple objectives, applying a multi-objective optimization strategy is straightforward. Thus, I recommend the authors provide some insights on this to further support the paper's motivation. However, this is not mandatory, and it is acceptable if the authors choose not to include it.

Based on the current strengths and limitations, I still lean towards a positive evaluation of this paper. As I am not an expert in video anomaly detection, my confidence is somewhat decreased, but I remain open to discussion.
paper_summary: paper_summary: The paper addresses the challenge of optimizing multi-task learning for semi-supervised video anomaly detection.
strength_weakness: strength_weakness: Innovative Approach: The paper introduces a novel progressive learning strategy, which is a fresh perspective in the realm of multi-task learning for VAD.
Comprehensive Experiments: The extensive experiments conducted validate the effectiveness of the proposed method, demonstrating substantial performance improvements.
limitations: limitations: Lack of Comparative Baselines: The paper would benefit from a more thorough comparison with a wider range of baseline methods to robustly establish the superiority of the proposed approach.
Theoretical Justification: There is a need for a deeper theoretical explanation for why the progressive learning strategy outperforms traditional multi-task learning approaches.
Task-Specific Insights: The paper does not provide detailed insights into how the model handles different types of anomalies across the three phases, which could help in understanding the nuances of the proposed method.
Scalability Concerns: The scalability of the proposed method to other domains or larger datasets is not discussed, which limits the generalizability of the approach.
Ablation Studies: More ablation studies are required to isolate the impact of each phase and the specific tasks within those phases, ensuring that each component’s contribution to overall performance is clear.
paper_summary: paper_summary: This paper introduces a progressive learning training approach for multi-task learning in video anomaly detection, highlighting the importance of task integration. The progressive learning method involves three phases: perception, comprehension, and inference. These phases are utilized to acquire low-level visual information, semantic features, and detect anomalies. Through extensive experiments and ablation studies on four datasets, the effectiveness of the proposed method is demonstrated.
strength_weakness: strength_weakness: - Novelty and technical correctness: The novelty of the paper is moderate, as it proposes a progressive learning method for multi-task learning, which considers the selection of proxy tasks and training sequences.
- Adequate evaluation: The paper conducts experiments on four widely used anomaly detection datasets.
- Clarity: The majority of the writing and logic are clear.
- Applications: This method is significant for video anomaly detection area.
limitations: limitations: 1. Are all layers of the shared base network frozen or trainable during the three training phases? Why does continuous learning help prevent the network from converging to suboptimal solutions? Doesn't continuous learning result in the network retaining knowledge from the last proxy tasks?
2. It is difficult to understand how virtual data is utilized during the inference phase. Could you please provide a detailed explanation?
3. Could you explain the rationale behind the learning order presented in Table 3? It seems that not all combinations of proxy tasks were considered.
4. Why do the first and second lines of Table 3 show different AUCs (79.0 and 74.2) when training the same proxy tasks simultaneously?
5. Numerous writing errors need to be corrected, such as 'L_grd' in Equation 3, 'L_pre' on line 504, 'cls' in Table 3, and 'rediction' on line 741.
paper_summary: paper_summary: The paper presents a progressive learning strategy for video anomaly detection (VAD), which enhances the efficacy of multi-task learning. The authors decompose VAD into three phases, i.e., perception, comprehension, and inference, to refine learning objectives progressively. In the perception phase, visual proxy tasks are performed to learn pixel-level features. The comprehension phase involves semantic proxy tasks to understand anomalies within the scene, while the inference phase uses open-set proxy tasks to detect anomalies. The model learns different levels of features and focuses on various anomaly types across phases. Extensive experiments on several datasets demonstrate the method's effectiveness.
strength_weakness: strength_weakness: -The idea of progressive learning of multiple proxy tasks is interesting and promising in video anomaly detection. 

-The experiment results seem solid and sufficient.

-The writing of this paper is commendable.
limitations: limitations: -The motivation needs to be further clarified. What is the specific insight principle and intuition behind the progressive learning of multiple proxy tasks? The explanation of the human cognitive process is somewhat vague and simple in the Introduction section.

-The authors have mentioned that the designed learning phases can address different types of anomalies. It would be better to provide a further explanation of the proposed anomaly classification and explain its reasonableness. And if necessary, please provide more examples to help readers understand. 

-What is the motivation and purpose of leveraging the semantic consistency of context to design semantics proposed in the Abstract section?

-In Section 3.5, why do the authors only use the prediction error from the perception phrase to generate anomaly scores? What about the proxy tasks which focus on different types of anomalies in the other two phrases when generating anomaly scores? How to obtain salient objects in Eq. (9)? What would be the experimental result if all objects were used?

-Will the proposed progressive learning method lead to a catastrophic forgetting problem?

-Some minor mistakes: the“Avenue Campus” in Table 3, and the lack of bold marks in Table 4.
paper_summary: paper_summary: The authors propose a method of progressively learning multiple proxy tasks to improve the trained model's performance on anomaly detection. Proxy tasks are classified into different categories so that the features of anomalies are learned from coarse to fine. In this way, it's easier to incorporate new proxy tasks for enhancing model's performance.
strength_weakness: strength_weakness: 1. This paper thoroughly explores how to progressively learn multiple proxy tasks for improving the performance of anomaly detection. Each proxy task is categorized into different phases. In this way, the training objective is constantly evolving with the training process, yet stays consistent with previous phases. 
2. The authors have conducted numerous experiments and results are convincing to show the effectiveness of the proposed method.
limitations: limitations: 1. The main motivation/idea of the work is incremental as it is largely built upon the previous work. Different tasks are heuristically classified into each phase. It would be better if the authors could provide a more in-depth analysis of the mechanism of this sequential learning method. 
Minor mistakes: 
  1. In my opinion, semi-supervised learning refers to learning with data that is partially labeled and I would not classify methods that only learn from normal frames as semi-supervised ones. In fact, there are previous methods which term this setting as one-class-classification. 
  2. Typos&Grammatical mistakes - an incomprehensive list: 
    line 89, learning - learn;  line 99, "to to guide the model" , multiple "to"s; in table 3, there is "Avenue" in the dataset column；line 448 the loss term L_gd is not the same as L_grd in the equation (3). There are more mistakes/typos in the manuscript. I highly suggest that the authors should carefully revise the manuscript.