## 3 The Proposed CCBA Model and Distillation Techniques
### Knowledge and Feature Distillation
#### 3.2.2 Knowledge and Feature Distillation Process  
Figure 8 illustrates the framework used to train the proposed end-to-end classifier, employing distillation techniques. Three distinct training strategies are applied during this stage, as depicted in the figure: feature distillation, knowledge distillation, and direct learning from ground truth. The concept of feature distillation is utilized to enhance the training of the proposed model's CNN-Audio module, aligning its output with that of SPWVD-MFCC. This alignment is crucial, as SPWVD-MFCC provides essential noise-resistant capabilities, offering superior robustness in low SNR environments compared to widely adopted STFT-Spectrogram and MFCC methods [24]. To quantify the similarity between the CNN-Audio output and the SPWVD-MFCC feature, L2 loss is employed as the loss of feature distillation, which is defined as follows:  
\[L_{FD}=L_{2}=\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}, \tag{1}\]  
where \(y_{i}\) represents the true value from the SPWVD-MFCC, \(\hat{y}_{i}\) is the predicted value from CNN-Audio, and n denotes the total number of data points. Figure 7: Flow chart of the teacher classification system. In addition to feature distillation, knowledge distillation is applied to transfer the learned representations from the teacher system to the student end-to-end classifier. The teacher model outputs probabilistic predictions using the dual-stream classifier in this process. These outputs, specifically the soft labels (predicted probabilities from the SoftMax layers), serve as additional training signals for the student model. Knowledge distillation aims to minimize the divergence between the output distributions of the teacher and student models. The Kullback-Leibler (KL) divergence measures this discrepancy, encouraging the student model to mimic the teacher model's predictions. The knowledge distillation loss can be defined as follows:  
\[L_{KD}=\tau^{2}KL\left(\frac{\sigma\left(q_{t}^{ResNet50}/\tau\right)+\sigma \left(q_{t}^{CNN-LSTM}/\tau\right)}{2},\sigma(q_{s}/\tau)\right), \tag{2}\]  
where \(q_{t}^{ResNet50}\), \(q_{t}^{CNN-LSTM}\), and \(q_{s}\) denote the logits predicted by the teacher's ResNet50, teacher's CNN-LSTM, and student model, respectively. \(\sigma(\cdot)\) and \(KL(\cdot)\) are the Softmax function and KL divergence. \(\tau\) is the temperature controlling the softness of distribution. This supervision, alongside direct learning from the ground truth labels, helps the student model generalize better and achieve improved performance, particularly in challenging environments with limited labeled data. Learning from ground truth provides the student model with direct supervision based on the true labels of the dataset. While feature and knowledge distillation guide the student model in approximating the teacher model's internal representations and output predictions, ground truth learning ensures that the model is accurately tuned to the actual target values. This component remains essential to the training process as it directly aligns the model's output with the real-world data. Cross-entropy loss is typically used to evaluate classification performance by comparing the model's predicted class probabilities with the actual ground truth labels. The loss for ground truth learning can be defined as:  
\[L_{GT}=-\sum_{i=1}^{n}y_{i}log(y_{i}), \tag{3}\]  
where \(y_{i}\) represents the true label for the \(i\)-th data point and \(\hat{y_{i}}\) is the predicted probability for the corresponding class. By incorporating this ground truth loss, the student model is influenced by the knowledge distilled from the teacher and reinforced to produce predictions as close as possible to the actual target values. In summary, combining feature distillation, knowledge distillation, and learning from ground truth creates a robust, multi-faceted training process. Each component contributes  
Figure 8: Knowledge and feature distillations to train the CCBA model.