# Self Attention Networks in Speaker Recognition
## 1 Introduction
in [31] showed that for an NMT task, translation relies on only a limited number of heads that are crucial. They also conclude that significant heads possess specific and interpretable functionalities in the model. These functionalities are associated with attention toward adjacent words and tokens in specific syntactic dependency relations. It is also demonstrated in [32], that attending multiple positions at once is not exclusively a characteristic of MHA. They have shown that multi-layer single-head attention also attends multiple positions and yields better results. However, they claimed that the primary advantage of multi-head attention lies in its training stability, as it requires fewer layers compared to single-head attention while attending the same number of positions. For instance, a 24-layer 16-head Transformer (BERT-large) and a 384-layer single-head Transformer have equivalent total attention head numbers and about the same model size, but the multi-head approach is considerably less deep. Since due to the recent advances in DL, the obstacle of training difficulty is no longer a limiting factor, it is possible to achieve consistent performance improvement without tuning hyper-parameters by going for a deeper single-head Transformer. In other words, single head attention with more layers is more efficient than MHA with less number of layers. However, stacking more layers in SAN-based architectures introduces another common problem in neural networks, namely rank collapse. It was demonstrated in [33] that in the absence of residual connections, the rank of the sequence representation in Transformer collapses in a doubly exponential manner with increasing depth. In the presence of a rank collapse, the model fails to differentiate between the representations of different tokens. The rank collapse and the conditions to alleviate such a problem are heavily studied in [34] for an NMT task. It was shown that stacking self-attention layers during initialization can lead to the rank collapse of token representation. The intensity of this phenomenon makes it even more difficult for the model to learn meaningful attention weights. Additionally, a technique called DeepNorm that modifies both initialization and the residual connection was put forth in [35]. The main contributions of this work are two-fold. We first introduce a speaker embedding extractor for speaker verification, which is designed based on inspiration from SANs, operating in an end-to-end manner. We present architectural advances to improve the performance compared to our previous SAN-based approach. It is demonstrated that self-attention mechanisms effectively capture time-invariant features from speech, leading to a discriminative representation of the speaker. The proposed end-to-end architecture consists of a pre feed-forward feature extractor, an encoder, a pooling layer, and a Deep Neural Network (DNN) classifier. The encoder and pooling layer exclusively utilize self-attention and feed-forward networks to generate discriminative speaker embeddings. The difference between the newly proposed architecture and our previous work in [26] can be summarized into (1) the addition of the pre feed-forward feature extractor, (2) the placement of the layer normalization, and (3) the replacement of ReLU activations by Gaussian Error Linear Unit (GELU) [36] in the position-wise feed-forward network of the encoder. Unlike our prior work where the focus was on mobile devices and the lightness of the model [26], here we focus on the performance of the model considering that the model is hosted on a cloud or more powerful devices compared to mobiles. In order to show the efficacy of the proposed approach, the speaker verification system is evaluated on _Voxceleb1-O_, _VoxCeleb1-E_, and _VoxCeleb1-H_ protocols [37]. Second, we empirically show the training instability in the framework of rank collapse and further study DeepNorm strategy as a potential mechanism to alleviate this shortcoming.