# Ensemble-Based Knowledge Distillation for Video Anomaly Detection
## 3 Proposed Approaches  
In this paper, we propose three ensemble-based knowledge distillation methods and use state-of-the-art AR-Net [13] and GCN [14] networks by combining these network structures with an I3D action recognition network [15] as a feature extractor to realize our approach (Figure 1). AR-Net is a weakly supervised model that combines two specific loss functions within its training process. The first loss function ensures that anomalous and normal class samples are distant from each other in the representation space and the second loss function handles the proximity level between normal class samples. Consequently, these two loss functions provide the AR-Net model with an accurate representation of training samples to properly accomplish the detection task. GCN is another weakly supervised model utilizing two unique graph modules in its inner structure. The first graph module provides the similarity information within different sub-clips of video samples, while the second graph module adds additional temporal information feedback by comparing temporal positions of the sub-clips with each other. Combined information from these two graph modules provides the GCN model to effectively learn a proper representation of abnormal and normal samples in a dataset. Both of these weakly supervised models require a pretrained network as a module for the feature extraction process. For this purpose, we decided to use the I3D network as a backbone mechanism for our baseline AR-Net and GCN models. I3D is a specific action recognition network utilizing a double stream of three-dimensional inflated convolutional modules. With these modules, the I3D network can produce a feature representation for video samples, which includes a rich summary of a video sample with respect to its behavioral patterns of actors, different level scene details and background details. Consequently, the I3D network helps the AR-Net and GCN models to properly translate a video sample to an accurate feature representation for further processing in their inner network structure. AR-Net and GCN models use weak supervision and these models are relatively feasible for integrating our ensemble-based knowledge distillation approaches; we decided to use these models as a baseline and I3D as a backbone feature extractor network within our proposed approaches. Firstly, for all three formulations, video samples are represented as clips, where \(t_{i}\) is the \(i^{th}\) clip for a specific video sample. Also, each video includes \(N\) video clips and padding operation is applied in case of a video is shorter than \(N\) clips. Then, a feature vector (size of \(F\)) is obtained for each video clip by utilizing a video action network, we used an I3D pretrained video action network but it is also suitable to deploy other video feature extraction models in this process. After obtaining a mini-batch of clip features from \(b\) video samples, we give clip-level features in each video sample of the batch as input to both GCN and AR-Net networks. Within the scope of the three approaches, GCN and AR-Net teacher models are trained on the same source dataset in which different information perspectives from different network structures are planned to pass to the student model. All of our proposed approaches adopt the knowledge distillation concept to handle the catastrophic forgetting problem. Within our approaches, the student model concurrently obtains feedback signals from both the ground-truth labels of the target dataset and the end output logits of the teacher models which are pre-trained on the source dataset during the training process. Therefore, we try to ensure that while the student model effectively learns the new patterns on the target dataset, it also keeps the previously learned patterns from the source dataset provided indirectly by the teacher models' supervision.