# Self Attention Networks in Speaker Recognition
## 1 Introduction  
The use of Deep Learning (DL) to extract a single low dimensional vector representation of the speaker characteristics from speech signal has become the de facto technique in many Speaker Recognition (SR) systems. These vector representations, often called speaker embeddings, can be constructed in an unsupervised (e.g., [1, 2]), supervised (e.g., [3, 4, 5, 6]), and self-supervised (e.g., [7, 8, 9]) regime. In the supervised approach, using speaker labels from the development set, a neural network is typically trained for classification purposes. Then, during the inference, the output layer is dropped, the network is fed by a sequence of feature vectors from an unknown speaker, and the combined representation from one or more hidden layers is considered as the speaker embedding [4, 5]. On the other hand, the success of sequence-to-sequence (seq2 seq) models in tasks like Neural Machine Translation (NMT) or Automatic Speech Recognition (ASR) is mostly due to attention mechanisms [10, 11, 12]. These techniques are used in seq2 seq models to assist the decoder in determining which portion of the sequence has to be translated or recognized. There are different methods to apply the attention mechanism, one of which is self-attention. In self-attention, the attention mechanism is applied to every position of the input itself. This kind of attention has shown promising results in a variety of tasks, such as NMT [11], semantic role labeling [13], language representations [14], and recently in vision [15] for image recognition. The use of self-attention in networks like Transformer [11] has become very common because of its strong parallelization capacity for computation and adaptability in modeling dependencies regardless of distance. Multi-head attention is a recently developed attention mechanism that was first put forth in [11] fora Transformer architecture and showed great promise in a number of downstream tasks, including [11; 15; 16; 17]. During the last few years, discriminative models based on attention for speaker embedding extraction became popular in the SR community. Typically, this attention mechanism is employed in the pooling layer as a substitute for statistical pooling. Its purpose is to extract a single, condensed vector representation of the speaker characteristics from a speech signal [6]. Attention is utilized in [18; 19; 20] to consolidate hidden states of a Recurrent Neural Network (RNN) and create speaker embeddings for text-dependent speaker verification. A unified approach that combines attention for both speaker and language recognition is presented in [21]. In [22], a multi-attention mechanism is introduced, inspired by [23], which employs multiple attention models to capture diverse information from encoder features. Additional studies such as [24; 25] have investigated different adaptations of multi-head attention for the pooling layer. Even though there are various attempts in employing the self-attention mechanism for speaker embedding extraction, only a few works tried to show the efficiency of models that are solely relied on Self-Attention Networks (SANs) to extract speaker embeddings [26; 27]. Despite the attempts in employing the SANs, there are difficulties in applying these architectures to different downstream tasks. There are several studies to enhance the efficiency of Multi-Head Attentions (MHAs), such as [28; 29]. It is usually believed that the efficiency of transformer-based architectures is rooted in their ability to jointly attend multiple positions mainly due to the MHA mechanism. However, there are multiple studies that try to answer this question, is MHA really necessary for SANs? [PERSON] et al. in [30] observed that in models trained with multiple attention heads, a considerable number of heads can be removed during the inference without substantially affecting the model's performance. Interestingly, it was shown that in practice some layers can even be reduced to a single head. In another attempt, [PERSON] et al.