## 3 The Proposed CCBA Model and Distillation Techniques
### Architecture of the CCBA Model  
Figure 6 illustrates the architecture of the proposed end-to-end CCBA. The block diagram above the dotted line provides an overview of the entire classifier, while the Section below details each component. The classifier comprises five neural blocks: CNN-Audio, CNN-Feature, BiLSTM, Multi-Head Attention, and Fully Connected (FC).  
Figure 6: Algorithm of the CCBA model.  
The CNN-Audio block processes raw audio segments of 1.024 s, sampled at 8 kHz, resulting in 8192 input samples per segment. This block contains six convolutional layers. Layers Conv-1 to Conv-5 are followed by max-pooling layers, progressively increasing the number of channels to 128 while reducing the sequence length to 256. Conv-6 reduces the channels from 128 to 16, aligning with the number of coefficients in the original SPWVD-MFCC, excluding delta and delta-delta terms. The data are then transposed into a format of 1 channel, 16 coefficients, and 256 timestamps, preparing it for the next block. In the CNN-Feature block, delta and delta-delta features are computed and concatenated with the input, followed by five convolutional layers to capture information specifically along the frequency axis. The output of Conv-11 has 512 channels, with a frequency axis dimension reduced to 1, meaning the frequency information has been fully encoded into the channel space. Finally, the data are reshaped to (512, 256) for further processing. In the BiLSTM block, two BiLSTM layers are stacked in series with a dropout rate of 0.3 between them. Each BiLSTM has a hidden size of 128, resulting in an output shape of (256, 256). The BiLSTM block exclusively focuses on analyzing temporal information along the time axis in both forward and reverse directions, enabling a bi-directional understanding of the temporal input. As a result, the output at each timestamp captures information from past (forward) and future (reverse) sequences, providing a comprehensive view of the entire sequence.  
Considering that the output at each timestamp of the BiLSTM contains information from past and future states, it is essential to efficiently capture a global feature representation for the final classification. To achieve this, we utilize the Multi-Head Attention block to dynamically analyze information along the temporal axis, obtaining a summarized vector for classification. First, the input is processed by the Token Encoding module, where the feature dimension is increased from 256 to 512 to enhance the representation capability of each time step. This module comprises a fully connected layer, an activation function, normalization, and dropout layers, providing robust transformation and regularization. Next, the Positional Encoding module encodes the data using sine and cosine functions, following the method proposed by [PERSON] et al. [33]. This step introduces temporal order information to the sequence, making the model aware of the position of each time step within the sequence. In the third step, within the Multi-Head Attention module, the input is first passed through three linear layers without bias to generate the query (Q), key (K), and value (V) representations. These are then split along the feature dimension into eight parts (i.e., eight heads), and self-attention is computed independently for each head. The results from all heads are concatenated and passed through another fully connected layer to produce the final output of the Multi-Head Attention mechanism. Finally, the result is averaged along the temporal axis to obtain a global representation that summarizes the information across all time steps. This global vector is then passed to the Output FC block for the final classification. The Output FC block generates a vector in the size of the number of target classes, where each element represents the model's confidence level for the corresponding class.