# Ensemble-Based Knowledge Distillation for Video Anomaly Detection
## 4 Experimental Evaluation
### Discussion on Results  
Overall, we can conclude that within the scope of our ensemble-based knowledge distillation-based experiments with our three formulation approaches, respectively, each subsequent formulation generally shows a slight performance increase compared to the previous formulation except the CBWC model with potential reasons explained in Section 4.7. Also, each of the three ensemble-based knowledge distillation methods shows a general behavior with respect to preventing catastrophic forgetting issues. More specifically, we also observe that the selection or combination mechanism within our approaches helps the student model to integrate different supervision feedback from AR-Net and GCN teacher models and passes this feedback information to the student model for keeping the old learned patterns in the source dataset. Also, our approaches provide the student model with stable performance on the target dataset and even increase the student model's baseline performance in specific configurations in the experiments. Ultimately we can also conclude that selection operation or combination operation can have a different effect on the student model's performance depending on the structures of the utilized baseline models. As we mentioned in Section 4.7, because of our baseline models' rather high common misclassification rate, selection operation can have more contribution to the student model's performance compared to combination operation in specific cases. In our experiments, generally, it can be observed that as the alpha parameter value increases in the knowledge distillation loss function, and the student model's performance gradually increases on the test set of the source dataset, it has a gradual performance decrease on the test set of the target dataset. This situation confirms that increased operation of the alpha parameter also intensifies the teacher model's influence on the student model and brings about closer bias to the source dataset and further bias from the target dataset for the student model. This observation is also crucial for handling the catastrophic forgetting issue because we can also observe that as the teacher model's influence increases on the \begin{table}
\begin{tabular}{l c} \hline
**Method** & **F-AUC** \\ \hline [PERSON] et al. (2018) [17] & 75.41 \\ \hline [PERSON] et al. (2019) [14] & 82.12 \\ \hline [PERSON] et al. (2019) [21] & 78.66 \\ \hline [PERSON] and [PERSON] (2019) [23] & 79.00 \\ \hline [PERSON] et al. (2020) [25] & 82.44 \\ \hline [PERSON] et al. (2020) [26] & 83.03 \\ \hline [PERSON] et al. (2021) [27] & 82.30 \\ \hline [PERSON] et al. (2021) [28] & 84.30 \\ \hline [PERSON] et al. (2021) [30] & 85.38 \\ \hline [PERSON] and [PERSON] (2021) [31] & 84.89 \\ \hline [PERSON] et al. (2022) [32] & 85.62 \\ \hline [PERSON] et al. (2023) [33] & 86.98 \\ \hline
**Ours (CBMS)** & **75.82** \\ \hline \end{tabular}
\end{table} 
Table 18: Comparison with State-of-the-Art (SOTA) Studies on UCF-Crime Dataset. student model, the student model performs gradually better on the source dataset, which indicates that the student model gradually focuses on keeping previously learned patterns from the source dataset with increasing alpha parameter values. In order to also test the robustness of our ensemble-based distillation methods, we additionally have reversed the UCF-Crime and RWF-2000 datasets for source and target dataset roles and replicated our experiments in this way. From experiments, it can be concluded that our experimental results show similar patterns in the case where the source and target datasets are reversed. Consequently, it can be also interpreted as our proposed approaches showing robust and independent performance from the datasets used in our experiments. Finally, we also prepared a table (Table 19) for the qualitative comparison of the other recent three weakly supervised studies with our ensemble-based knowledge distillation approaches with respect to their advantages and disadvantages.