## 3 Proposed Approach
### Neural Networks  
We experimented with various configurations and specifications, leveraging studies that have achieved state-of-the-art performance in heart disease detection [43, 65, 66, 67]. This entire process is illustrated in Figure 3. After combining the architectural components, the final proposed model consisted of a single dilated convolutional layer with a rectified linear unit (ReLU) applied across a time-distributed wrapper, a BiLSTM layer, and an attention mechanism, followed by two fully connected layers with ReLU and sigmoid activations, respectively. With regard to regularization, batch normalization was added after the convolutional layer reduced the covariance shift, and a dropout layer preceded and succeeded the BiLSTM-Attn block to mitigate overfitting. The model accepts the first input with a dimensionality of \([10,150,1]\), indicating timesteps of 10(\(\approx\)3 s), with features representing 150 signal sampling points. The output of the CNN is an \(X\)-dimensional feature vector summarizing the spatial irregularities and patterns found in the raw input signal. This output is propagated into the BiLSTM network, where the temporal patterns are captured and fed into the attention mechanism to highlight the integral aspects of the input. The model then accepts the second auxiliary input of HRV/PRV metrics with a dimensionality of \([6,1]\), and concatenates this to the output of the attention mechanism to aid in a better representation of the input. Finally, this concatenated vector passes through a sigmoid activation function for binary classification of NSR and AF classes. The KD procedure results in a compressed _student_ model, whereby the knowledge is transferred from the _teacher_ by minimizing a loss function that attempts to match the softened _teacher_ logits and the ground-truth labels [26]. The logits are softened by applying a temperature scaling parameter, which serves to smooth the underlying probability distribution representative of inter-class relationships. We also account for the optimality of the knowledge procedure in terms of consistent teaching and patience, as described in [68]. Figure 3: Architecture of the proposed CNN-BiLSTM-Attn model with raw signal inputs and auxiliary HRV inputs (for ECG) or PRV inputs (for PPG). The overall loss function incorporating both distillation and _student_ losses modified for binary classification is defined as  
\[\mathcal{L}(x;W)=\alpha*\mathcal{H}(y,\sigma(z_{s};T=1))+(1-\alpha)*\mathcal{H}( \sigma(z_{t};T=\tau),\sigma(z_{s},T=\tau)) \tag{1}\]  
In Equation (1), \(x\) is the input; \(W\) represents the _student_ model's parameters; \(y\) is the ground-truth label; \(\mathcal{H}\) is the binary cross-entropy loss function; \(z_{s}\) and \(z_{t}\) are the logits of the _student_ and _teacher_, respectively; \(\sigma\) is the sigmoid function parameterized by the temperature \(T\); and \(\alpha\) is a weighting factor for the _student_ and distillation losses. As per the findings of [26], lower temperatures work better when the _student_ model is much smaller than the _teacher_ model, as the latter's softened labels carry more information that the _student_ model may not be able to capture. Experimentally, this makes sense, as when we used temperatures ranging from 1 to 10, we found 5 to be the optimal value for our implementation. The process of KD for this work is depicted in Figure 4. The original _teacher_ model, as shown in Figure 3, is trained on the MIT-BIH AF-DB ECG dataset. This model is then fine-tuned on the UMass-DB PPG dataset using TL, where only the last three layers are retrained. This preserves the coarse- and medium-granularity representations of the ECG signals. The fine-tuned model now effectively \"teaches\" the smaller _student_ model its internalized knowledge during concurrent supervised training with softened labels. During this last stage, the model's output logits, not the binary labels, and the sigmoid function are applied during the loss calculation, as shown in Equation (1).