# Ensemble-Based Knowledge Distillation for Video Anomaly Detection
## 2 Related Works
### Knowledge Distillation  
Before introducing the Knowledge Distillation concept, we also introduce important key rules about Incremental Learning and Continuous Learning concepts within the scope of our study. Firstly according to Incremental Learning these three constraints should be satisfied [34]:While the model is updating itself with respect to the stream data, it should not update itself by using all of the previously collected data; therefore, it should focus on using the new stream information/data. 2. While the model is updating itself with respect to the new data, it should also remember important data patterns, it should not forget important and fundamental patterns within the total big data. Therefore, the proposed model should not be affected by the Catastrophic Forgetting while adapting the new data. 3. In constant stream data, fixed-size data should be used for the updating process. Also, in the process of adapting the model to new data, three main scenarios are generally encountered [35]  
1. **Fine-Tuning Case for New Tasks:** On two datasets, which do not intersect with respect to the samples and classes they include. One dataset of these two is the dataset for the new task and a pre-trained model trained on the latter dataset (used for the old task) beforehand, is re-trained on the dataset for a new task. While in the retraining process, parameters from the specific last layers or the parameters of all layers of the pre-trained model can be updated/optimized for the new task. 2. **Continuous Learning for Known Classes:** In this situation, additional training data are constantly (or in certain intervals) added from a streaming source to the baseline data used before. The task has not changed and the new samples' classes are exactly the same as the samples' classes from the old data. This concept can be also thought as a standard online learning concept. 3. **Continuous Learning of Known and New Classes:** In addition to the second scenario, now in this case, classes from new samples can also be different from old ones. Our study is actually within the scope of the third scenario with an additional case. Even though data samples from the new dataset compared to the baseline dataset we experiment on have included new patterns and classes, our task especially focuses on the binary (two classes) video anomaly detection task, which means our proposed approach classifies the video data into two possible outputs in which videos include an anomaly scene or not. While our anomaly detection task is binary, the new data samples still include new class information and new pattern information intrinsically so our model has to adapt itself. To satisfy the requirements of baseline rules and constraints mentioned above, we use the Knowledge Distillation (KD) approach within the scope of our study. Knowledge Distillation is an approach to transfer distilled learned pattern information from a larger network to a smaller network. In this approach, there are two important module structures, which are teacher and student models. During the training process, the teacher model's task is to transfer its intrinsic information from data patterns learned previously from the student model. The student model is trained by a weighted combined loss of two functions, the first loss function is calculated with respect to the difference between the student model's end output logit values and ground-truth label values belonging to new data. The second loss function is generally specified as a \"distillation\" function and calculated between the student model's end output logit values and the teacher model's end output logit values. Teacher model logit values are obtained by giving the same input vector in the new data that are given to the student model as input. Thus, the teacher model reinterprets the input vector with its own perspective and teaches this perspective to the student model with its own end logit values.