# Self Attention Networks in Speaker Recognition
## 3 System Architecture  
As depicted in Figure 1 the system described in this study can be broken down into four main components, namely a pre feed-forward feature extractor, an encoder, a pooling layer, and a DNN classifier. The very first layer is a linear feed-forward neural network which is applied on the frequency dimension of the input utterance. The encoder is a stack of \(N\) identical SAN blocks with a single-head self-attention mechanism as proposed in [32]. The encoder network produces a sequence of feature vectors as its output. These encoded features are subsequently transformed into a comprehensive speaker representation through the pooling mechanism. The fourth stage of the network involves a DNN classifier, which generates speaker posteriors. The output from the pooling layer serves as speaker embeddings, which can be utilized for tasks like speaker verification using methods such as cosine scoring or more advanced back-end techniques.  
The encoder based on SAN generates a sequence of speaker representations at the short-term frame level. To transform these frame-level features into an utterance-level vector, a self-attention pooling layer is utilized, following the approach outlined in [26]. This pooling layer employs an additive attention mechanism that calculates the compatibility function using a feed-forward network featuring a single hidden layer. Given the sequence of encoded features \(H=[h_{1},h_{2},\dots,h_{L}]^{T}\in\mathbb{R}^{L\times d_{m}}\), we compute the segment-level representation \(C\) as:  
\[C=Softmax(W_{c}H^{T})H \tag{7}\]  
where \(W_{c}\in\mathbb{R}^{d_{m}}\) is a trainable parameter. This self-attention mechanism can be interpreted as a dot product attention, where the keys and values represent the same representation, while the query is a trainable parameter. Consequently, \(C\) is calculated as a weighted average of the encoder sequence of features. These weights reflect the alignment learned by the self-attention mechanism. It is important to note that unlike [22, 49], here the attention is not  
Figure 1: The architecture diagram of the proposed system. It shows that the encoder consists of \(N\) layers, where each layer is composed of a sequence of attention mechanisms followed by position-wise feed-forward networks. Residual connections and layer normalization are employed for each sub-layer.  
used to extract the statistics necessary for the pooling. Rather, the entire statistical pooling process is substituted with a straightforward additive attention mechanism, resulting in reduced computational overhead.  
The pooling layer produces a fixed-length vector, which is then passed into a DNN classifier to generate speaker posteriors. However, unlike [26] we only consider a softmax layer as our classifier to calculate a multi-class cross-entropy objective during training. Once the network is trained, there can be one or multiple points where we can extract the speaker embeddings. For example in [26], embeddings are derived from one of the DNN layers after the application of a non-linearity function. Instead of utilizing the previous layer prior to the softmax operation as the speaker embedding, the authors opted for the second previous layer, as done in [50]. This choice is motivated by the findings of [4], which suggest that the second previous layer exhibits improved generalization and contains speaker-discriminative information that is less specific to the training data. However, in our recent experiments, we directly extract the embeddings after applying the pooling. There are other studies such as [51], where people considered more than one layer for the embeddings and they combine multiple speaker embeddings to shape a single embedding for a speaker.