# Self Attention Networks in Speaker Recognition
## 4 Rank Collapse in Self-Attention Networks  
It is widely recognized that the expressive power of neural networks is enhanced by increasing the depth of the network [52]. However, increasing the depth in gradient-based methods it is usually led to a slowdown of the learning process which is commonly a consequence of unstable gradient norms in deep networks [53]. The initialization of the layer weights is one of the key players in training neural networks. The common practice in random initialization of layer weights involves selecting i.i.d entries from a zero-mean Gaussian or uniform distribution. It is sometimes accompanied by a scaling factor to maintain a constant variance of the layer activation across different layers [54; 55]. [PERSON] et al. in [56] observed that such initialization for linear neural networks causes the majority of singular values in the last layer's activation to collapse toward zero with increasing depth. In other words, this initialization approach cannot avoid spectral instabilities in deep networks [57]. Additional methods such as residual connections and normalization layers are required to enhance the stabilization of the training process of deep architectures with such standard i.i.d initialization schemes as for example appeared in [55; 57; 58; 59]. It was demonstrated in [33] that when there are no residual connections in the Transformer model, the rank of the sequence representation decreases rapidly by increasing depth. It was also shown that layer normalization and fully connected layers can to some extent mitigate the rate of degeneracy. This shows the importance of the initialization step in Transformer-based architectures. When there is a rank collapse, the model fails to differentiate between representations of distinct tokens, as they become perfectly aligned in the feature space during initialization. [PERSON] et al. in [34] showed for an NMT task that stacking self-attention layers can bring about the rank collapse in tokens' representation at the start of the training process by causing the known vanishing (small) gradients phenomenon in the keys and queries' parameter matrices. This issue significantly hampers the model's ability to acquire meaningful attention weights, and its impact is amplified with increasing depth. As the rank deficiency and consequent vanishing gradient problem of the queries and keys manifest across multiple layers, the problem is further exacerbated. The conditions under which the rank collapse can be avoided were also investigated in [34] at initialization. Their findings revealed that scaling the residual branches in a manner dependent on the depth, not only does it stabilize the activation norms during initialization, but it also approximately preserves the cosine of the angle between tokens. Consequently, this stabilization also helps maintain the rank of the propagating sequence. They also highlighted a different dependence on the input dimension \(d_{m}\) and the length of the input sequence, which suggests that the gradient norm of a subset of parameters has the potential to differ significantly in magnitude. This observation aligns with previous empirical evidence from related studies [47]. In another attempt, [PERSON] et al. proposed a method in [35] called DeepNorm which modifies both initialization and residual connection. It combines the residual connections in each attention and feed-forward layer for training a Transformer. In order to avoid large parameter updates, it scaled up each residual connection's computation by a constant \(\alpha\). Mathematically, residual connections usually output \(x+f(x)\), where \(f(x)\) is the function computed by the previous layer. However, in DeepNorm the output of the residual connection is computed using \(\alpha x+f(x)\). It also scaled down the initial parameter values to avoid large updates in the early stages of training. Using this approach they were able to train Transformer models with 1000 layers. Before going further we first give the formal definition of the rank collapse in matrices.