## 5 Discussion  
This study explored the utility of TL and KD in aiding the performance of downstream AF classification across two different one-dimensional signal representations reflecting cardiac activity. During the initial phase of experimentation on the MIT-BIH AF-DB ECG dataset, we observed that convolutions or LSTMs alone were not able to completely capture the underlying patterns. By combining these components, we obtained a model that was essentially greater than the sum of its parts, especially when augmented by attention and auxiliary indices. It can be surmised that the spatio-temporal nature and complexity of the signal data validate the need for larger, complex structures to capture the intricacies in the latent space for larger datasets. The positive results of fine-tuning this representation on a smaller dataset and then applying KD imply that condensing the learned information is more beneficial than random initialization for the same problem. **Downstream performance:** With regard to UMass-DB, we note that the variance in performance was quite disparate, likely owing to the presence of greater-than-anticipated noise elements in the _student_ models. This was less persistent for the _teacher_ model, where TL reduced the variance in cross-fold performance, particularly for sensitivity and the MCC. This was likely due to the larger fine-tuned model being able to distinguish between the NSR and AF samples in the PPG dataset, which it was unable to do before TL. While TL  
Figure 5: Selected instances of Grad-CAM highlighting the class-specific regions of interest for NSR and AF across the different datasets represented as amplitude in millivolts (\(y\)-axis) against time in seconds (\(x\)-axis). and KD did in fact elevate the performance of a much smaller student model as opposed to training from scratch, larger models may be preferred if more accurate predictions are desired. However, smaller models have the benefit of being able to be deployed on edge devices, where, based on the trade-off between false positives and false negatives, their classifications can be meaningful for a clinician remotely. Ultimately, it depends on the priority of accuracy or frequency, i.e., seldom but accurate versus often but less accurate. However, we observed that for the MIMIC-III dataset, where the signals were acquired at the bedside in a clinical setting, the baseline performance was already considerably high. In this context, both TL and KD did not offer much additional value (i.e., 1%). We hypothesize that this was due to two primary reasons: data size and quality. First, the dataset was 10\(\times\) the size of the UMass-DB dataset and contained more representative and diverse samples for both of the predicted classes. Second, the signals had low saturated noise and were consistent in terms of morphology owing to standardized acquisition. We have seen the arguments in [82] and agree that with smaller datasets, fine-tuning approaches yield better performances when looking at single-channel ECG signals. The authors emphasized that transferring knowledge may not always necessarily or consistently improve performance. Moreover, as the number of data samples increases, the student predictions naturally move closer to those of the teacher, provided that the capacity and architecture are sufficient for the scale of the dataset and the complexity of the data [83]. Therefore, we can summarize that sharing initialization (with TL) increases alignment in the activation space, but KD does not offer additional advantages for the MIMIC-III dataset. **Complementary findings**: Compared to similar studies focusing on efficient TL or KD paradigms for AF detection [32; 43; 67], our models achieved comparable results, with the added advantage of a greater model compression ratio with respect to the scores obtained. More recent studies used a pipeline similar to our work to estimate systolic and diastolic blood pressure [84] and for sleep staging [85].