## 2 Literature Review
### Knowledge Distillation for Compression  
Previous research has shown the effectiveness of KD in developing compressed models appropriate for deployment on smaller devices without performance loss. We report similar work in Table 2. Moreover, KD has been used to improve performance by introducing information from additional modalities without increasing the model's complexity. Ref. [28] asserted that the required computational power and memory overheads impose restrictions on continuous monitoring systems, especially within the context of wearables. The authors applied knowledge distillation to a convolutional recurrent neural network for AF detection using ECG signals, allowing deployment on low-power wearable devices. In the distillation process, the _teacher_ is a network containing more than one million parameters, trained on a labeled dataset of 8528 single-lead ECG signals. The _student_ is a slim network containing 138k parameters, trained on the signals resampled at 122 Hz and the soft labels generated by the _teacher_. The compression of the network resulted in a speed-up of 38% and a decrease of 43% in memory footprint. After the application of KD and quantization, the slim model achieved an increase of 6.3% in the F1 score and 7.8% in sensitivity. \begin{table}
\begin{tabular}{p{12.3 pt} p{12.3 pt} p{12.3 pt} p{12.3 pt} p{12.3 pt} p{12.3 pt}} \hline \hline
**Paper** & **Task** & **Teacher** & **Student** & **Data** & **Results** \\ \hline
[28] & Atrial fibrillation detection using ECG data & CNN-LSTM & CNN-RNN & ECG data from CinC2017 & Increase in accuracy (0.792 to 0.828 F1 score) \\ \hline
[30] & Emotion recognition using EEG data & CNN-TCN & TCN & Video and EEG data from MAHNOB-HCI database & Visual-to-EEG cross-modal KD improves prediction with significance \\ \hline
[31] & Arrhythmia detection using ECG data & CNN & CNN & ECG data from PhysioNet/CinC arrhythmia detection challenge 2017 & 1\% F1 loss (0.88 to 0.87), memory usage 30. In [29], the authors proposed a KD framework for single-lead ECG-based sleep staging. This framework employs multi-modal training of EEG and ECG in the teacher network and unimodal testing in the final ECG-based _student_ model. Both the teacher and the _student_ models' base architectures are adapted from the fully convolutional network U-Time. The dataset contains whole-night EEG and ECG recordings from 200 participants, resampled down to 200 Hz. The model showed an increase of 14.3% in the F1 score of the four-class classification model and 13.4% in the F1 score of the three-class classification model compared to the ECG baseline. In [30], the authors proposed a continuous emotion recognition model using visual-to-EEG KD. The dataset used contains 239 labels, synchronized recordings of facial videos, and EEG signals from 24 subjects. The _teacher_ model is a cascade convolutional neural network-temporal neural network trained on facial video frames to predict emotional states. The visual spatiotemporal features or dark knowledge learned by the _teacher_ are used to supervise the _student_ model, a temporal neural network that predicts emotional state using EEG signals as input. The _student_ model is trained using the EEG signals, the visual spatiotemporal features, and the corresponding labels. The experiments show that the visual-to-EEG KD improves the results with statistical significance compared to the standalone _teacher_ and _student_ models. The authors of [31] used KD to develop a model for ECG signal classification with low computing power and memory requirements. The _teacher_ network is a full-precision deep convolutional neural network with nine 1D convolutional layers and one fully connected layer. The student model is a less computationally complex binarized convolutional neural network trained by binarizing the model weights and activations. Both the _student_ and _teacher_ models use ECGs for cardiac arrhythmia detection.