# Self Attention Networks in Speaker Recognition
## 6 Results and Discussion  
Although some state-of-the-art methods for speaker verification are based on convolution operation [5; 6], attention-based architectures are clearly state-of-the-art in speech recognition, Natural Language Processing (NLP), and many other deep-learning tasks [11; 13; 14; 15; 16; 17]. In particular, this approach is very convenient in our domain because of multi-purpose large language models, which are trained for a mother task such as speech recognition and can be fine-tuned for a specific downstream task such as speaker verification. For this reason, we focus our experiments and discussion on SANs to improve the performance of such models in speaker recognition. The results for the text-independent speaker verification task for three different protocols are reported in Table 1. In all these models comprise two blocks of SAN, \(N=2\), which are stacked on top of each other. The scoring of different systems is based on the cosine distance. The performance of these systems has been assessed using the Equal Error Rate (EER) metric. The newly proposed SAN-based architecture, A-SAN (Advanced SAN), is compared with the one originally proposed in [26]. There is more than 41% and 38% relative improvement in VoxCeleb1-O, and VoxCeleb1-E protocols, respectively, without any Data Augmentation (DA). There is an additional 25% and 29% performance improvement in both VoxCeleb1-O and VoxCeleb1-E protocols, respectively, by employing DA techniques. For the VoxCeleb1-H which is the hardest protocol for the verification, we achieve 27% relative improvement by adding DA to the training pipeline. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Architecture** & **VoxCeleb1-O EER (\%)** & **VoxCeleb1-E EER (\%)** & **VoxCeleb1-H EER (\%)** \\ \hline SAN [26] & 5.44 & 5.62 & - \\ A-SAN w/o DA & 3.20 & 3.43 & 5.54 \\ A-SAN w/ DA & 2.41 & 2.42 & 4.02 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of the evaluation results reported on VoxCeleb datasets with and without Data Augmentation (DA). A-SAN is referred to the SAN model proposed in this work. The results reported for SAN model from [26] are without any DA. Cosine distance is employed for scoring. Figure 2, compares the singular values of the classifier layer in descending order for the three different systems from Table 1 for the VoxCeleb1-E protocol. For a matter of clarity, only the first 200 singular values are depicted in the figure. The singular values for each system are normalized with respect to their corresponding maximum value in order to be in the same dynamic range (between 0 and 1). As is shown all systems suffer from rank collapse. This rank collapse in the case of SAN in [26] is very severe and it becomes smoother with the modifications proposed in this work. In other words, the proposed changes directly affect the rank collapse, which also impacts the EER performance of the embedding extractor system. Let us now focus on the experiments without any DA to only show the efficiency of the architectural and training modifications. The performance of SAN architectures in terms of different numbers of SAN layers is illustrated in Figure 3. As it is observed, the performance of the original A-SAN architecture is decreased by increasing the depth. In order to stabilize the performance, a DeepNorm [35] training mechanism is applied. As it is shown, the performance does not change substantially by stacking more layers on top of each other. In the following, we discuss this stabilizing effect in the framework of rank collapse. Figure 4 compares the rank collapse of the pooling layer for the A-SAN systems with different numbers of layers, with and without the DeepNorm method. Note that, the pooling layer is the uppermost layer of the embedding extractor during the inference phase. From the figure, it appears that the rank collapse follows a similar trend for all numbers of layers when DeepNorm is used, while the rank collapse depends on the number of layers without DeepNorm.