# Self Attention Networks in Speaker Recognition
## 2 Self Attention Networks  
Self-Attention Networks are built by stacking one or more blocks of SAN. Each block of SAN usually comprises two sub-layers. A Multi-Head Self-Attention mechanism makes up the first sub-layer, while a position-wise feed-forward network makes up the second [11]. Around each of the two sub-layers, a residual connection is used, and it is either followed (Post-LN) or preceded (Pre-LN) by a layer normalization [38]. The attention is applied to every position of the input sequence to produce representations. This is accomplished by employing a series of linear projections. Consider an input sequence \(X=[x_{1},x_{2},\ldots,x_{L}]^{T}\) of length \(L\), with \(x_{l}\in\mathbb{R}^{d_{m}}\), and a set of parameters \(\{W_{Q},W_{K}\}\in\mathbb{R}^{d_{m}\times d_{k}},W_{V}\in\mathbb{R}^{d_{m} \times d_{c}}\), the model transforms the input into namely queries \(Q\in\mathbb{R}^{L\times d_{k}}\), keys \(K\in\mathbb{R}^{L\times d_{k}}\), and values \(V\in\mathbb{R}^{L\times d_{c}}\):  
\[Q=XW_{Q},\ K=XW_{K},\ V=XW_{V} \tag{1}\]  
The calculation for the output of each time instance, represented as \(o_{l}\) in the attention layer, is as follows:  
\[o_{l}=(Attn(q_{l},K)V)W_{O} \tag{2}\]  
where \(Attn(\cdot)\) is a function that for a given query \(q_{l}\) and the keys \(K\) returns the attention value. Additive attention [10] and dot-product attention [39] are the two most often employed attention functions. The attention mechanism used here is a scaled-down variation of the dot-product attention mechanism, which was first put forth in [11]. In comparison to additive attention, it is in practice significantly faster and more space efficient. The additive attention performs better than the dot-product attention when the dimension \(d_{k}\) increases [39]. In order to bridge the performance difference between the two while maintaining its benefits, the scaling factor is useful. The formulation of the attention mechanism's ultimate output representation \(O\) is as follows:  
\[O=\left(Softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V\right)W_{O} \tag{3}\]  
where \(W_{O}\in\mathbb{R}^{d_{x}\times d_{m}}\) is a linear layer that projects the final output into \(O\in\mathbb{R}^{L\times d_{m}}\). The output of the attention mechanism is passed on to the position-wise feed-forward network. It is applied to each position separately and identically. This includes two linear transformations with a GELU [36] activation in between:  
\[FFN(u)=gelu(uW_{1}+b_{1})W_{2}+b_{2} \tag{4}\]  
where \(u\) and \(FFN(u)\) are the input and output of the feed-forward network, respectively. Although these linear transformations vary from layer to layer, they are the same across different positions. \(W_{1}\in\mathbb{R}^{d_{m}\times d_{ff}}\) and \(W_{2}\in\mathbb{R}^{d_{ff}\times d_{m}}\) can be considered as two convolutions with kernel size of 1, while \(d_{ff}\) is the feed-forward dimension. The GELU non-linearity is defined as the expected transformation of the stochastic regularizer on an input. This statement, roughly speaking, says that we scale \(x\) according to how much greater it is than other inputs. Since the error function is frequently used to calculate the cumulative distribution function \(\Phi(x)\) of a Gaussian, we define the GELU as:  
\[gelu(x)=xP(X\leq x)=x\Phi(x)=x\cdot\frac{1}{2}[1+erf(\frac{x}{\sqrt{2}})] \tag{5}\]  
We can approximate the GELUs with:  
\[gelu(x)\simeq 0.5x(1+tanh(\sqrt{2/\pi}(x+0.044715x^{3}))) \tag{6}\]  
The placement of the layer normalization is important for both performance and training stability in Transformer based architectures. The original Transformer used Post-LN while people in [40] found Pre-LN more useful for their task. Both placements were studied in [41] and was shown that the Pre-LN residual connections improve the stability of Transformers. However, as shown in [42] Pre-LN performs worse than Post-LN because its gradients in the bottom layers frequently tend to be larger than those at the top layers.