## 2 Proposed Methods
### CL-Transformer Classification Model Based on Contrastive Learning
Contrastive loss maximizes the correlation degree between positive pairs of samples and reduces the correlation degree between negative pairs, thereby highlighting the adaptive weight of key sample pairs to be recognized. In the formulation of sample pairs, entities within the same batch and sharing an identical label are rearranged as positive parts, whereas all disparate instances are designated as negative parts relative in our method. The output of the Transformer Encoder layer has a high dimension, and directly applying contrastive loss is not supposed to effectively separate samples of different classes. Directly reducing the output dimension of the Transformer Encoder layer can lead to information loss. To prevent this and compensate for the loss of spatial features, a Projector network layer is deployed as a transitional bridge in time, reducing the dimensionality of the Encoder's output while preserving the essence of the data indeed. The Projector, represented as \(g(\cdot)\), is a linear fully connected layer, and its input is the normalized output of the Transformer Encoder layer. Given labels \(Y=\{y_{1},y_{2},\ldots,y_{n}\}\) and corresponding samples \(X=\{x_{1},x_{2},\ldots,x_{n}\}\), sample \(x_{i}\) is mapped to the representation vector \(h_{i}\), where \(h_{i}=f(x_{i})\in\mathbb{R}^{D_{f}}\). The representation vector is reduced in dimension through the projection network \(g(\cdot)\), yielding \(z(h_{i})=g(h_{i})\in\mathbb{R}^{D_{g}}\), where \(D_{g}\ll D_{f}\). The supervised contrastive loss is then calculated as follows:  
\[L_{i}^{sup-cont}=-\frac{1}{N_{y_{i}}}log\frac{\sum_{j=1}^{N}\chi_{[y_{i}=y_{j }]}\exp\left(sim(z_{i},z_{j})/\tau\right)}{\sum_{k=1}^{N}\chi_{[k\
eq i]}\exp \left(sim(z_{i},z_{k})/\tau\right)} \tag{3}\]  
where \(N_{y_{i}}\) is the number of samples with the same label \(y_{i}\) in the same batch, \(\chi_{[y_{i}=y_{j}]}\in\{0,1\}\) evaluates to \(1\) if and only if \(y_{i}=y_{j}\), \(N\) is the batch size, \(\tau\) is the temperature parameter used to adjust the scale of the similarity measure between sample pairs, and \(sim(\mathbf{u},\mathbf{v})\) is the function that calculates the cosine similarity between vectors \(\mathbf{u}\) and \(\mathbf{v}\). The calculation formula is:  
\[sim(\mathbf{u},\mathbf{v})=\frac{\mathbf{u}^{\top}\mathbf{v}}{\|\mathbf{u}\|\mathbf{v}\|} \tag{4}\]  
The cross-entropy loss function provides global supervision by directly optimizing classification results, ensuring that the model has good classification ability. The supervised contrastive loss function provides local supervision by optimizing the similarity between sample pairs, enabling the model to learn more discriminative feature representations. When integrated harmoniously, these two core loss functions operate together, mutually reinforcing their respective strengths and the robustness of the model. The computation of this hybrid loss, which encapsulates the best of both worlds, is formulated as follows:  
\[L_{i}=\alpha L_{i}^{sup-cont}+(1-\alpha)L_{i}^{cross-entropy} \tag{5}\]  
where \(\alpha\) is a hyperparameter used to adjust the weight between cross-entropy loss and supervised contrastive loss. Additionally, this paper explores the impact of different \(\alpha\) values on the mixed loss through experiments. The relevant experimental results are shown in Table 2.  
Table 2 shows that the best results are achieved when the cross-entropy and contrastive loss contribute equally to the loss calculation. We determined the optimal value of \(\alpha\) through experiments. Therefore, in the subsequent experiments, we set the value of \(\alpha\) to 0.5. \begin{table}
\begin{tabular}{c c c} \hline \hline \(\mathbf{\alpha}\) & **ESC-50 (\%)** & **UrbanSound8K (\%)** \\ \hline
0.01 & 95.00 & 89.85 \\
0.25 & 95.50 & 90.92 \\
0.50 & 96.75 & 91.39 \\
0.75 & 95.25 & 90.80 \\
0.99 & 94.75 & 89.73 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The impact of different \(\alpha\) values on the mixed loss.