# Ensemble-Based Knowledge Distillation for Video Anomaly Detection
## 1 Introduction
Both of these models are weak supervision-based models utilizing only video level ground-truth information and they can make use of intermediate features of pre-trained I3D action recognition model [15] to leverage the high-level action/behavioral patterns of crowds/actors while classifying anomaly patterns. To gradually adapt the model while forcing it to prevent catastrophic forgetting, we deploy a knowledge distillation approach [16] as a backbone adaptation mechanism. By this approach, we create an ensemble-based learning method based on two AR-Net and GCN-based teacher models, where the AR-Net-based student model tries to optimize itself with respect to ground-truth label information and respective teachers model's own outputs by using a combined loss function. Our main contributions in the scope of this paper can be listed as follows:  
* We introduce three novel ensemble-based knowledge distillation mechanisms for video anomaly detection, where the student model focuses on adapting itself to new incoming data while also focusing on not forgetting old patterns it learned before completely. The student model is also provided with an information transfer including different perspectives from multiple teacher models, during this adaptation process for new information. Consequently, instead of training/fine-tuning a model on a comprehensive-sized dataset growing continuously, the model can be updated with our ensemble knowledge distillation methods more efficiently while preventing the catastrophic forgetting problem. Although the focus of this paper is video anomaly detection, the proposed knowledge distillation methods can also be used in other computer vision tasks such as image classification, scene classification, etc. * We adapt AR-Net and GCN models into our ensemble-based knowledge distillation approach but other state-of-the-art methods can be easily adapted to our approach. We extensively evaluate AR-Net and GCN models on two comprehensive datasets, which are UCF-Crime [17] and RWF-2000 [18] to validate our proposed methods with respect to quantitative results. Within experiments, firstly we use the UCF-Crime dataset as the source dataset, which represents the baseline dataset including previous patterns the model first learned and RWF-2000 represents the target dataset includingnew patterns to be learned by the model. In later stages, we also switch dataset roles, meaning that we use RWF-2000 as the source dataset and UCF-Crime as the target dataset. Consequently, we try to ensure that the model shows stable performance and behavior in case the dataset roles are reversed. * We present extensive experiments and analyze different parameters with respect to the knowledge distillation mechanism and three different ensemble-based formulations. Results show that our proposed approaches reduce the catastrophic forgetting problem of the model on the source dataset while generally improving the performance of the model on the target dataset. We also discuss limitations and possible future work plans for our proposed approaches. * We share our code for future research studies on this link: [[https://github.com/BurcakAsal/AnomalyEnsembleKD](https://github.com/BurcakAsal/AnomalyEnsembleKD)]([https://github.com/BurcakAsal/AnomalyEnsembleKD](https://github.com/BurcakAsal/AnomalyEnsembleKD)) (accessed on 19 January 2024). Our study continues in Section 2; related studies with respect to our study are provided in Section 3 and our proposed ensemble-based knowledge distillation approaches and different formulations are introduced in detail; in Section 4 we provide and discuss the quantitative results we obtained. Finally, in Section 5, the conclusion of our study and discussion about possible improvements for future work are provided.