Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
Authors: Thiziri Nait Saada
Alireza Naderi
Jared Tanner
Venue: ICML 2025 poster (2025)
Keywords: transformers
attention mechanism
spectral analysis
initialisation
random matrix theory
signal propagation
softmax
Primary area: theory->deep_learning

Abstract: Attention layers are the core component of transformers, the current state-of-the-art neural network architecture. Alternatives to softmax-based attention are being explored due to its tendency to hinder effective information flow. Even *at initialisation*, it remains poorly understood why the propagation of signals and gradients through these random networks can be pathological, resulting in issues known as (i) vanishing/exploding gradients and (ii) rank collapse *in depth*, i.e. when all tokens converge to a single representation along layers. While rank collapse in depth naturally arises from repeated matrix multiplications---a common pattern across various architectures---we identify an additional and previously unknown challenge unique to softmax attention layers: (iii) rank collapse *in width*, which occurs as the context length increases. Using Random Matrix Theory, we conduct a rigorous analysis that uncovers a spectral gap between the two largest singular values of the attention matrix as the cause of (iii), which in turn exacerbates (i) and (ii).
Building on this insight, we propose a novel yet simple practical solution to mitigate rank collapse in width by removing the outlier eigenvalue(s). Our theoretical framework offers a fresh perspective on recent practical studies, such as (Ye et al., 2024; Ali et al., 2023), whose ad hoc solutions can now be interpreted as implicit efforts to address the spectral gap issue. This work provides valuable theoretical support for ongoing large-scale empirical research, bringing theory and practice one step closer in the understanding of transformers.

Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
Thiziri Nait Saada
Alireza Naderi 1
Jared Tanner'
Abstract
1. Introduction
Attention layers are the core component of trans- formers, the current state-of-the- neural net- work architecture. Alternatives to softmax-based attention are being explored due to its tendency to hinder effective information flow. Even af ini- tialisation, it remains poorly understood why the propagation of signals and gradients through these random networks can be pathological, resulting in issues known (i) vanishing/exploding gradi- and (ii) rank collapse in depth, i.e. when all tokens converge to a single representation along layers. While rank collapse in depth naturally arises from repeated matrix multiplications common pattern across various architectures-we identify an additional and previously unknown challenge unique to softmax attention layers: (iii) rank collapse width, which occurs as the con- text length increases. Using Random Matrix The- ory, conduct rigorous analysis that uncovers a spectral gap between the two largest singular values of the attention matrix as the cause of (iii), which turn exacerbates (i) and (ii). Building on this insight, we propose novel yet simple prac- tical solution to mitigate rank collapse in width by removing the outlier eigenvalue(s). Our the- oretical framework offers fresh perspective on recent practical studies, such as (Ye et al., 2025; Ali et al., 2023), whose ad hoc solutions can now interpreted as implicit efforts to address the spectral gap issue. This work provides valuable theoretical support for ongoing large-scale empiri- cal research, bringing theory and practice one step closer in the understanding of transformers.'
Transformers (Vaswani et al., 2017) have emerged as the dominant architecture in machine learning, achieving re- markable success across various domains, particularly in natural language processing and computer vision, largely due their defining feature: the self-attention mechanism (Bahdanau et al., 2016). However, despite their empirical success, transformers are often plagued by training instabil- ity and high sensitivity to numerous hyperparameters, which require careful tuning. This challenge has motivated recent efforts to establish a theoretical framework for understand- ing transformer architectures, even in their most basic forms, to ensure reliable information flow through deeper layers and facilitate training.
The purpose of this work is to analyse signal propagation in softmax- attention layers at initialisation, i.e. with randomly initialised model parameters. While the issues of rank collapse (in depth) and vanishinglexploding gradients have been previously identified in transformers at initialisa- tion (Dong al., 2021: Noci et al., 2022), our work extends these findings and uncovers an additional phenomenon- rank collapse in did to the use of softmax in the self-attention mechanism. By width, we specifically refer to the context length, which, despite its increasing scale in modern attention layers, seems to have been overlooked in favour of depth in previous analyses of rank collapse in trans- formers. Indeed, rank collapse in width, which unique to attention layers, has not been identified in the existing literature nor recognised the depth. In contrast, rank collapse in depth is not specific to transformer like networks, as it typically results from suc- cessive matrix multiplications, shared feature of various deep neural network architectures. Investigating the spectra of the random matrices formed by the model's parameters, we detect a spectral gap between the two largest eigenvalues of the attention matrix, which drives rank collapse in width and further accelerates rank collapse in depth. Moreover, we propose a provably effective remedy for the spectral gap, solution that naturally arises when the problem is viewed through a spectral lens. To the best of our knowledge, this is the first study to examine the attention mechanism the large context length regime, as well as the first to identify
catalyst for rank collapse along
'Equal contribution 'Mathematical Institute, University of Ox- ford. Correspondence to: Thiziri Nait Saada and Alireza Naderi <naitsaadat @ ax..ukk naderi @ maths.ox.e ow.ac.kk..........
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Our code is available http:///ghhuu.ccommhhirnni//s/sp Analysis-of-Attene Aittis-oo--tttt-------- Layers.gtt.

---

Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
rank collapse in width and its effect on other oversmoothing phenomena.
suffering from rank collapse, with slightly adjusted rates according the location of the outlier in the spectrum; Figure 1.
Let us consider the eigenvalues of softmax- attention matrix. Since the rows sum to unity, there is an eigenvalue of 1 corresponding to the eigenvector of all-ones. Under certain conditions, the other eigenvalues shrink in size as the matrix dimension increases, resulting in widening gap between the largest eigenvalue (i.e. 1) and the diminishing Figure 1. The successive multipli-
Interestingly, our work and the fix we propose find echos in recent promising practical studies. For instance, the Differ- ential Transformer (Ye et al., 2025) computes the attention matrix as a difference between two softmax matrices and has shown significant improvements from this adjustment, e.g. finer quantisation, better key information retrieval or hallucination mitigation. In light of our findings, this ad- justment can be understood as removing the troublesome outlier from the spectrum of the attention matrix that nat- urally arises when using softmax. Even closer, the work of (Ali et al., 2023) directly implements our fix and shows improved behaviour on image segmentation tasks. When our work is of theoretical nature and concerned with the initialisation dynamics, it is interesting to see how these concepts are playing out in practical applications.
bulk of eigenvalues; cation of such matrices at each layer increasingly favours specific direction-tie one aligned with the dominant eigen- vector of the attention matrix -verr the others. This leads to a distortion in the geometry of the input training data, ex- emplified by the phenomenon of rank collapse. An intuitive solution project out this troublesome direction from all attention matrices, allowing for more balanced signal propa- gation. These ideas lie at the core of our rigorous analysis of attention layers, from which we derive slightly modified attention mechanism that proves advantageous in practice. Remarkably, the spectral gap observed in softmax attention matrices also appears in alternative attention mechanisms suggested in the literature, making our analysis relevant to these variants as well; see Figure 1.
1.2. Organisation of the paper
In Section 2, we first review the fundamentals of random matrix spectra before exploring the properties of standard softmax key-query attention matrices with isotropic input. We prove the existence of a spectral gap for such attention matrices and use this result to demonstrate rank collapse in width. Next, we introduce a modified attention mechanism designed to eliminate the spectral gap and show that this modification effectively resolves rank collapse in width.
1.1. Related work
Our analysis builds upon previous work that studied the spectra of large random neural networks better understand and stabilise initial training dynamics, such as (Pennington et al., 2018 8) for fully-connected networks and (Xiao et al., 2018) for convolutional networks.
In Section 3, we extend our analysis from single attention layer to a deep attention-only transformer. Using a certain class of random Markov matrices to model attention, we precisely determine the rate at which rank collapse occurs as a function of width and depth (Proposition 3.4), thereby revealing for the first time how the two interact. Further- more, we prove that the removal of the spectral gap cures rank collapse in width and mitigates the exploding gradients (Propositions 3.7 and 3.8).
The rank collapse in transformers was first explored in (Dong et al., 2021), where the authors show that the out- put of an attention-only transformer converges exponen- tially with depth to single representation across tokens. (Noci et al., 2022) make connection between rank collapse and vanishing gradients by assuming "uniform attention" A = - 1xxT, essentially proving rank collapse in depth based on the assumption that rank collapse in width has already occurred (as opposed to showing why this premise holds).
In Section 4, we validate our findings, providing empirical evidence of rank collapse in both width and depth in com- mercial models such BERT family. We put to test our "remove the gap" solution across range of architectures featuring LayerNorm and skip connections and show its effectiveness in resolving stable rank collapse and slowing down gradient explosion. Finally, we observe the emergence of additional outliers in deep transformers and conduct ex- periments to study the potential benefits of removing all outliers rather than the dominant one.
There is also a growing literature on the shortcomings of softmax self-attention mechanism and possible alternatives. Some suggest the usage of other well-known activations such as identity, ReL.U, and sigmoid (Hron et al., 2020; Wortsman et al., 2023; Ramapuram et al., 2025), while others propose modifications to the attention block (He et al., 2022; Wang et al., 2022; Shi et al., 2022; Ali et al., 2023: Dovonon et al., 2024; Ye al., 2025). Based our analysis, we can offer a fresh perspective on these attention variants and reflect on their effect on rank collapse. Indeed, they all fall (at the exception of the identity) within the class of attention matrices presenting a spectral gap, therefore
2

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
2. A Attention layers and rank collapse in width
2.1. Spectra of random matrices
Throughout this paper, we consider random matrices (of different distributions) in the large width limit and describe them through their limiting spectral properties. In the con- text of transformers, the number of tokens and the embedding dimension(s) are large-as an assumption typically satisfied in practice. For certain classes of random matrices, the overall behaviour of cigenvalues/singular values becomes remarkably predictable as the matrix size increases, despite the randomness of in- dividual entries. If M, R=x= are random matrices with eigenvalues and singular values denoted by (A, and {s, (M,)): respectively, the histograms of the eigenval- ues/singular values
mean by "large width" that both
(a) Softmax attention (Vaswani et al., 2017)
(b) i.idd. Markov
THACe
505
T1 JM. n Σ (M.)) VM,
n
25
36
40 28
&
(M:
(c) ReLU attention (Wortsman et al., 2023)
(d) Sigmoid attention (Ramapu- ram al., 2025)
n
converge, in many interesting cases, to deterministic distri- butions μ and v, known as the limiting eigervalueesingular value distributiin of M.. Additionally, the largest eigen- value/singular value of random matrices is often studied in its own right. Our analysis builds on cutting-ede tedting-edge results concerning both the limiting distribution of the eigenval- ues/singular values (the "bulk" of the spectrum) and the be- haviour of the largest eigenvalue/singular value (the "edge" of the spectrum). In particular, we focus two classes of random matrices: Gaussian and Markov, that respectively model the value and attention matrices in our transformer model (11) at initialisation.
Figure 1. Theorem 2.1 demonstrates that applying the conventional softmax key-query attention mechanism to orthonormal input to- kens yields spectral gap in its spectrum (a). Due to their non-zero mean, ReLU (c) and sigmoid attention (d) mechanisms also exhibit spectral gap-although of different size. As the size T of an i..dd Markov matrix (Definition 3.1) grows, its eigenvalues form a circular bulk of radius O(T-1/3 in the complex plane, except for the largest eigenvalue which remains 1 (the black dot in (b)). In practice, T does not need be large for the limiting behaviour to appear, as shown above.
and
lim S2 s2(VTA(X))) = 20,
(3)
2.2. Attention at initialisation
T-200
Let Xo E Txxx be an input sequence of context length T, i.e. formed of T tokens. Each token embedded as a d-dimensional vector, with a fixed ratio y = - S 1. The
while
Iim [>a(VTA(X)) 20,
(4)
T
where depends only Tgt-
standard softmax key-query attention matrix is obtained by
Xo w X
With further assumptions, we exactly analyse one atten- tion layer at initialisation, where the output is given by
A AX,)) softmax
ddge
X = A(X,)WV.
where W9, WK E dxdx have i.i.d. AV entries.
and the value matrix W E pdxd is initialised indepen- dently with i.i.d. (O,, 1) entries
The above attention matrix exhibits a spectral gap, meaning its largest eigenvalue (or singular value) differs significantly in magnitude from the second largest. Moreover, this gap widens as the context length increases. In fact, we pre- cisely quantify the rates which both eigenvalues evolve, as follows.
2.3. Rank collapse in width
Theorem 2.1 reveals that the attention matrix becomes ef- fectively rank-one for large context length T. Naturally, any definition of "rank collapse" relies on a proxy for dis- crete rank, as the random matrices in question are almost surely full-rank, making it uninformative refer to their
Theorem 2.1 (Spectral gap in softmax attention). Let the impul sequence Xn have orthonormal rows, i.e. X = I. Then, A1 (X))) = 1 and almost surely.
lim (A(())) = T
"See Remark 3.3 for a discussion on scaling the value matrix.
3

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
This theorem constitutes the primary contribution of this paper. It uncovers previously unknown phenomenon that we refer to as rank collapse in width-or the convergence of tokens' representations into single point as the context length increases within a single attention layer. It should not be mistaken for rank collapse in depth, which is partly a result of repeated random matrix multiplications. In con- trast, rank collapse in width is unique attention layers, making this discovery particularly significant for the theory of transformers. In the next section, we will explore how removing the spectral gap of Theorem 2.1 helps alleviate rank collapse in width and restore a more balanced flow of information.
spectral gap
A(X), xx" =I
i..dd Markov
softmax attention
undorn A frxr
attention matrices
Figure 2. The standard key-query attention matrices of Equation (I) form a subelass of i.i.d. Markov matrices of Definition 3.1, of which the uniform attention of (Noci al., 2022) is one trivial special case. We use i.i.d. Markov matrices to build our general multi- laytr-layer theory section 3.
2.4. Attention without the gap
Assume X, = I and let us write A = A(X)) to lighten the notation. The attention matrix A E R ten as
XT can be writ-
actual rank. For example, (Dong et al., 2021) consider the "one-infinity norm" of the residual (the difference between a matrix and its best approximation of the form 1x ), defined v VIlres( [Irees(M) [a [M)]]]] while (Noci et al., 2022) use E.g [MMMM (MMT) Ji.j. which is maximised when all rows of M are identical. We choose stable rank as our preferred proxy due to its clear geometrical interpretation and simple def- inition in terms of singular values. It is defined for any non-zero M E Rmxn as
1
A A=EA+ (A - EA) = +Ix + A
(7)
where 1xx = -1,.... 1) (1,...... 1) is the all-ones ma- trix. Therefore, A is rank-one perturbation of A whose spectral radius is Aı (A-) O (12)) (see Lemma A.3), Although the rank-one perturbation $ lrxT cannot disturb the bulk of the spectrum, it causes the largest eigenvalue to "escape" from the bulk to 1, creating a spectral gap.
| (M)
In light of this, we can slightly modify the attention mecha- nism to eliminate the outlier-and thus the gap gap-simply by replacing A with A - to get the following modified attention layer, i.e.
SM(M)
(5)
M 2
(f(
The following theorem establishes that the stable rank of
the covariance kernel
x = A WX,
8
Σ == XX
where A 'Xx)) A(x)) } Lyxr- Note that this modi-
fication is applied only to the attention matrix (and not to the signal representation) and X serves as shorthand for the output of an attention layer whose A is replaced with A L 's as in (8).
collapses to 1 as the width T increases.
Theorem 2.2 (Rank collapse in width). Assume En- I. Then, with overwhelming probability?,
lim sn Σ) 1. T
Since the modified attention exhibits no spectral gap (see Lemma A.6), the stable rank of the covariance matrix Σ = x-X no longer collapses to but instead scales linearly width, as detailed in Proposition 2.3 (cf. Theorem 2.2).
Moreover, the convergence happens at a polynomial rate, i.e. (x) -
While the isometric input assumption may appear restrictive, is introduced solely to formalise the theoretical argument. In Appendix A. we present extensive empirical evidence that this assumption is not required for the result to hold in practice.
Proposition 2.3 (Resolved rank collapse in width). Let X- - layer (8) and Σ E Rxx be its covariance matrix. Then, almost surely, the rank does not collapse, i.e. there exists a constant C > such that,
WV be the signal in our modified attention
A sequence of events E,, holds with overwhelming probability if, for every A > P(F..) CAn A, for some constant CA. as defined (Tao, 2012).
ST Σ
lim T
= C.
(9)
T

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
3. The interplay between width and depth
X = A The input signal X. RR'xd has T tokens of embedding dimension d, with fixed ratio 7 _ I 1. For such a network of depth L, the input- output relationship is thus given by
is important to note that spectral gap within the attention matrix can arise when the input to the softmax function consists of a matrix with vanishing elements. This can be achieved, for example, by initialising the key and query matrices following Xavier or He initialisation schemes, or under any scaling such that o² + 0 as dyk increases. In this scenario, the attention matrix asymptotically approaches the uniform attention matrix, i.e. the matrix of all entries equal to 1/T. In this edge case, the attention matrix becomes rank- one, the spectral gap maximised and once the rank-one perturbation } 1 is removed, the attention is reduced to zero- preventing any signal propagation beyond the rank collapse.
X - AI.An- A,X.
WW (11)
The value matrices W. E Rddd are initialised indepen- dently with i..dd (0, 1) entries and the attention matrices A, E RT'x are independent i.i.d. Markov matrices with a 11
Remark 3.3 (Scaling of value matrices). The reason we ini- tialise the value matrices with (o 1) entries rather than (0, 1 1/d) (i.e. He initialisation) is that the attention matri- ces have singular values of magnitude O(1/VT) except for the leading one (A)
o(1); see Theorem A.1. So, in all but one direction, the attention matrix scales down the signal by a factor of O(1/vT), which will be compensated by WV with singular values of magnitude O(vd).
It is. nonetheless, in this trivial case that the analysis of signal propagation along depth was conducted in (Noci et al., 2022). In particular, their study demonstrates rank collapse in depth by implicitly assuming that rank collapse in width had already occurred. In contrast, our work presents a more refined analysis of rank collapse by considering the effects of both width and depth jointly, offering, to the best of our knowledge, the most advanced theoretical study to date on the initialisation of transformers.
3.1. Revisiting rank collapse & exploding gradients
Rank collapse Given an isotropic input Xo with En = X, = I, we are interested in understanding how the stable rank of the covariance matrix at layer b,
We model the ε-th layer attention mechanism initialisation by a random matrix A with non-negative entries (Az)i. 2 0 and normalised rows, i.e. (Ae)), L, as if it were generated by row-wise application of softmax. As we will demonstrate, this model functions as a helpful abstraction that offers insights into the interaction of width and depth. More specifically, we consider A₂ to be an i.i.d. Markov matrix, as defined in (Bordenave et al., 2011).
Σg X
evolves. Proposition 3.4 demonstrates how the stable rank collapses as the width T increases.
Proposition 3.4 (Rank collapse in width for deep network).
Assume Eo- = I. Then, for any E 221 1,
lim ST EE) = T
(12)
Definition 3.1 di.i.d. Markov matrix). Let Zr, be i.i.d. non- negative random variables with positive mean m B(Z,,1 > 0 and variance x² = Var(Z1,2) > 0 as well as
with overwhelming probability. Moreover, the convergence happens at a polynomial rate, i.e.
finite fourth moment 1 < oc. Let A E RXT be its
row-normalised version, i.c.,
(x) - 0(T)=
Zij
Remark 3.5. The proof essentially boils down to determin- ing the gap between the largest and the second largest sin- gular value of Xx, since sr (Et) equals
Auj= = Ej=1 Zo.j
(10)
We call A an i.i.d. Markov matrix.
Remark 3.2 (Terminology). This is slight abuse of lan- guage since the entries of an "i.i.d. Markov matrix" are not necessarily independent random variables because of the normalisation constraint. Moreover, not all random Markov matrix ensembles satisfy the above definition. Remarkably, the standard key-query dot product attention matrix A (Xo) defined in (I) is an i.i.d. Markov matrix provided the input isotropic X, I. The uniform attention is another instance from this class of matrices; see Figure 2.
ΣT of (Ee) $fEE,)
Σ (Xe) S1+
(x))
(X,)
While $1 (Xt) O()) regardless of the layer, the bulk, i.e. 82 (X,) shrinks like (1/T),, which gives the desired rate.
Exploding gradients. A well-known issue that can disrupt training across various neural network architectures is the vanishing or exploding of gradients; see (Hanin, 2018). For attention-only transformers with degenerate attention, (Noci et al., 2022) demonstrate that the gradients with respect to W! vanish. Our model (11) allows for more general
We study as our model a deep stack of attention layers at initialisation, where at each layer l, the signal is transformed
5

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
random attention while using different scaling that makes the same quantity explode rather than vanish. Proposition 3.6 provides lower bound on the rate at which the gradient grows.
RURERTS
Proposition 3.6 (Exploding gradients). For any 2 2 and ssls I, with overwhelming probability,
OXL Ow!
lim T-00 TL-1
CL-G,
(13)
110
105
for some constant Cr.-t > 0. In particular, for T large enough, ||0x,/ow| | diverges to infinity as L increases. In the single-layer case f = L = 1, the following improved
100
150
240
bound
(a) Rank collapse width
Ox1
lim T-oo
2C
(14)
T
OWY
F
holds almost surely.
3.2. Removing the gap
120
As previously seen, we can write an ii.d. Markov matrix
1.15
A E T as
1.0
A T 1Tx + A
(15)
where 1rxT is the all-ones matrix and A has a limiting spectrum resembling that of a Gaussian matrix.
(b) Rank collapse depth
Similar to what we did in the last section to remove the outliers and keep the bulk, we simply replace A with A every layer, i.e.
Figure 3. Famous transformer encoders suffer from rank collapse at initialisation, both in (a) width and (b) depth. These untrained models (d 784) are loaded from Hugging Face and sentences from this paper's 's abstract are tokenised using pre-trained tokenis- ers. Inputs therefore non-isometric (see Appendix A.l for more detailed discussion).
x = A A .. XY W WY,,WY (16) 1Txx- Note, again, that we directly modify the attention matrices (but not the signal represen- tation) and X should be understood as the signal layer é E2 1 in network whose Aç's are replaced with A in (16). We set X = Xoo
where A =
Proposition 3.8 (Resolved exploding gradients). Let X
A X Wy be the signal at layer in our modified model (16). Then, in expectation, the squared norm of the gradients grow linearly with d, i.e. there exists constant C. > 0 such that,
The modified attention has spectral gap (see Lemma A.6), thus the stable rank of the covariance matrix Σ == X X does not collapse anymore, as detailed in Proposi- tion 3.7 (cf. Proposition 3.4).
ax
lim
E
C.
(18)
Proposition 3.7 (Resolved rank collapse in width). Let X = X WV be the signal at layer f in our modified model (16) and Σt E RTxT be its covariance matrix. Then, almost surely, the rank does not collapse, i.e., there exists constant C₃ > 0 such that,
OW!
F
4. Experiments and Further Insights
Rank collapse observed in practice. We highlight the practical relevance of our findings by showing rank collapse occurs both in width and depth in widely used transformer encoders like BERT, see Figure 3. In Figure 4(a), all solid lines, corresponding to the vanilla attention layer block, show a stable rank that rapidly converges to as the con- text length grows (at rate given in Theorem 2.1). This occurs even if additional components such LayerNorm,
SI Σイ
lim T-500
= Cr.
(17)
T
Our modification also mitigates the average growth of the gradients. Proposition 3.8 establishes linear growth rate for /ix,,/w? 1/- in expectation, which should be compared the rate of TL-1 from Proposition 3.6.
6

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
AU8I
403
42-
15.0
12.5
30.0
7.5
5.0
2.5
(a) Spectral gap causes rank collapse in width.
Figure 5. In deep transformers initialised with i.i.d. Markov atten- tion, our "remove the gap" fix is effective, as we can precisely discard the single outlier in the spectrum.
AXXI
AG0-
20.0
AIX UN
A64 LN
<kpp
A64- sklp
17.5
AXXI
sklp
15.0
Indeed, the stable rank now scales up with the context length T (at linear rate predicted by our theory), in contrast to the original behaviour where it would quickly vanish the rank collapse in width is mitigated, it naturally slows down the one in depth, possible way to slow down rank collapse in depth (though not eliminate it) is to set the value matrices as orthogonal matrices.
125
10.0
Once
seen in Figure 4(b). Another
1.0
Septh
(b) The rank inevitably collapses in depth.
After passing an isotropic input through the network, we compute the gradient norm as defined (13). When the attention is i.i.d. Markov matrix, gradient norms effectively controlled with depth by either applying Layer- Norm or removing the spectral gap, as illustrated in Figure 5. The corresponding lines remain flat, confirming the the- oretically expected constant trend with L. Without these solutions, the gradients explode exponentially with depth as indicated by the linear trend in the log-log scale, with the derived lower bound of T empirically confirmed.
Figure 4. Rank collapse occurs both in width and depth. Our fix effectively prevents the rank from collapsing in width within an attention layer. Although rank collapse in depth occurs regardless of the presence of the spectral gap, our fix consistently slows the collapse-a feat no other module achieves.
skip connections, or both are added on top of the attention block. Provided that rank collapse in depth is an inherent consequence of repeated matrix multiplications, architec- tural modifications of the attention layer can only slow the collapse rather than completely prevent it. We demonstrate this by showing that rank collapse depth persists even when the attention matrix is set the identity matrix an extreme case with the highest possible stable rank and spectral gap.
Future avenues. Beyond one attention layer, we conduct an empirical investigation of the spectrum of the actual attention matrix across layers. Surprisingly, as shown in Figure 4, we observe the emergence of additional outliers in the spectrum. We assess the effectiveness of our proposed the largest spectral outlier-by compar- ing it to an alternative approach, which consists removing outliers. Since identifying and locating these additional outliers is highly non-trivial, we perform an SVD of the attention matrix at each layer and manually identify and re- move the extra gaps based on the largest difference between singular values. As depicted in Figure 6, this computation- ally expensive approach does not significantly outperform our proposed fix, which is much more efficient. In fact, the two methods achieve comparable results in mitigating rank collapse (first in width, then in depth) and controlling
Removing the spectral gap. As an input signal propa- gates through an initialised transformer, we can address both forms of rank collapse-aee across width and bepth-by elim- inating the spectral gap induced by the attention matrix at each layer. Figure 4(a) reinforces our findings, showing that our colution-indicatee by 1. in the legend-conss consistenly mitigates rank collapse. Remarkably, the fix effective even deep transformers when combined with additional components such as LayerNorm, skip connections, or both.
7

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
5. Conclusion
We introduced a new mathematical framework for studying the self-attention mechanism at initialisation, leveraging re- sults from random matrix theory and free probability, By analysing the spectral properties of i.idd. Markov matrices, we diagnosed random naftmax-based attention with spec- tral gap that leads to rank collapse in width-a phenomenon revealed and demonstrated for the first time by our analysis- alongside the previously established rank collapse in depth and exploding gradients.
35
3.0
2.5
We proposed a simple modification of the attention mecha- nism, which is provably effective in slowing rank collapse. Additionally, we empirically discovered that the spectra of standard key-query attention matrices often feature multiple outliers in deeper layers. Context length appears to have been overlooked in previous analyses of rank collapse in favour of depth. Given the continuous growth in the size of transformer models used in practice, we hope this work encourages the community reconsider the emphasis on depth in theoretical developments, particularly in the analy- sis of rank collapse.
Figure 6. In deep transformers, our "remove the gap" fix is nearly as effective as removing all outliers from the spectrum despite having significantly lower computational cost.
gradient norms. This demonstrates that removing the largest spectral gaps yields the most significant benefit with the lowest cost. Finally, while training dynamics are beyond the scope of this work, our theoretical framework provides a fresh perspective on recent empirical studies that report significant training benefits from various ad hoc fixes. Our analysis offers insights into why these fixes may be effective. These studies include approaches such as "centering" self- attention layers (Ali et al., 2023), where the authors remove exactly 7- 7-'17x7 from the attention layer, and "differenti- ating" the attention mechanism, as seen in the Differential Transformer architecture, where two softmax attention ma- trices are subtracted from each other at each layer to "cancel the noise"; see (Ye et al., 2025). We can now interpret these approaches as implicit efforts address the spectral gap.
A Acknowledgements
TNS is supported by the UK Engineering and Physical Sci- ences Research Council through the grant EP/W523781/1. JT acknowledges support from His Majesty's s Government in the development of this research as well as by EPSRC grant EP/Y028872/1, Mathematical Foundations of Intelli- gence: An "Erlangen Programme" for AI.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
layer
layw
layer
layer
2.13
0.3-
0.04
0.54
0.2-
3.05
00055
0.02
0.52
-1 00023
-0.12
-0.52
62-
0.54
0.24
3/5
-1.25
0/5
Figure? 7. Spectrum of T x T softmax-bass attention matrix (with T 200) across layers. Some additional outlier eigenvalues emerge from the spectrum with depth an intricate way. The matrix, howevor, consistently converges to uniform attention for large f.
8

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
References
Mlttkowski, w., Nowak, M. A., Penson, K. A., and Zyczkowski, K. Spectral density of generalized wishart matrices and free multiplicative convolution. Physical Review E, 92(1):01212,, 2015.
Akemann, G., Ipsen, J. R., and Kieburg, M. Products of rect- angular random matrices: Singular values and progressive scattering. Physical Review E. 88(5), November 2013. ISSN 1550-2376.
doi: 10. 10.1103/physrevee88.. wt..eeee......211111 88.058118. 18.
Murray, M., Abrol, V., and Tanner, J. Activation function design for deep networks: linearity and effective initiali- sation. Applied and Computational Harmonic Analysis, 59:117-154, 2022.
Ali, A., Galanti, T., and Wolf, L. Centered self-attention layers. arXiv preprint arXiv:2306.01610 2023.
Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate, 2016.
Nait Saada, T. and Naderi, A. A simple proof for the almost sure convergence of the largest singular value of product of gaussian matrices, arXiv preprint arXiv:2409.20180 2024.
Bong, H. and Kuchibhotla, A. K. Tight concentra- tion inequality for sub-weibull random variables with generalized bernstien orlicz norm. arXiv preprint (rXiv:2302.038500 2023.
Noci, L.,, Anagnostidis, S., Biggio, L.., Orvieto, A., Singh, S. P., and Lucchi, A. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35: 27198-27211, 2022.
Bordenave, C., Caputo, P., and Chafai, D. Circular law theorem for random markov matrices. Probability Theory and Related Fields, 152(3-4):751-779, January 2011. ISSN 1432-2064. doi: 10. 10110077000400000-033611111
Pennington, J., Schoenholz, S., and Ganguli, S. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems, 30, 2017.
Dong, Y., Cordonnier, J.-B., and Loukas, A. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pp. 2793-2803 PMLR, 2021.
Pennington, J., Schoenholz, S., and Ganguli, S. The emer- gence of spectral universality in deep networks. In Inter- national Conference on Artificial Intelligence and Statis- tics, pp. 1924-1932. PMLR, 2018.
Dovonon, G. J., Bronstein, M. M., and Kusner, M. J. Setting the record straight on transformer oversmoothing. arXiv preprint arXiv:2401.0433 2024.
Geman, S. A Limit Theorem for the Norm of Random Matrices. The Annals of Probability, 8(2):252 261, 1980. doi: 10.1214/aa 10111/1/////111 cetpoo//111994777 176994775.
Ramapuram, J., Danieli, F., Dhekane, E., Weers, F., Bus- bridge, D., Ablin, P., Likhomanenko, T., Digani, J., Gu, Z., Shidani, A., and Webb, R. Theory, analysis, and best practices for sigmoid self-attention. In International Conference on Learning Representations (ICLR), 2025.
Hanin, B. Which neural net architectures give rise to ex- ploding and vanishing gradients? Advances in neural information processing systems, 31, 2018.
Shi, H., Gao, J., Xu, H., Liang, X., Li, Z., Kong, L., Lee, S. M. S., and Kwok, J. T. Revisiting smoothing in bert from the perspective of graph. Proceedings of the 10th International Conference on Learning Representations (ICLR), 2022.
He, B., Martens, J., Zhang, G., Botev, A., Brock, A., Smith, S. L., and Teh, Y. W. Deep transformers without shortcuts: Modifying self-attention for faithful signal propagation. In The Eleventh International Conference on Learning Representations, 2022.
Tao, T. Topics in random matrix theory, volume 132. Amer- ican Mathematical Soc., 2012.
Hron, J., Bahri, Y., Sohl-Dickstein, J., and Novak, R. Infinite attention: Nngp and ntk for deep attention networks. In International Conference on Machine Learning, pp. 4376- 4386. PMLR, 2020.
Thompson, R. C. The behavior of eigenvalues and singular values under perturbations of restricted rank. Linear Algebra and its Applications, 13(1-2):69-78. 1976.
Kuchibhotla, A. K. and Chakrabortty, A. Moving beyond uub-gaussianity in high-dimensional statistics: Applica- tions in covariance estimation and linear regression. In- formation and Inference: A Journal of the IMA, 11()): 1389-1456, 2022.
Vaswani, A., Shazeer, N.. Parmar, N., Uszkoreit. J., Jones, L., and Gomez, A. N. Attention is all you need. Advances in neural information processing systems, 30:5998-60000 2017.
Mingo, A. and Speicher, R. Free probability and random matrices, volume 35. Springer, 2017.
Wang. P., Zheng, W., Chen, T., and Wang, Z. Anti- oversmoothing in deep vision transformers via the fourier

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
domain analysis: From theory to practice. In Proceed- ings of the 10th International Conference Learning Representations (ICLR), 2022.
Wortsman, M., Lee, J., Gilmer, J., and Kornblith, S. Replac- ing softmax with relu in vision transformers, 2023.
Xiao, L., Bahri, Y., Sohl-Dickstein, J., Schoenholz, S., and Pennington, Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convo- lutional neural networks. In International Conference on Machine Learning, pp. 5393-5402. PMLR, 2018.
Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., and Wei, Differential transformer. In Proceedings of the 13th International Conference on Learning Representa- tions (ICLR), 2025.
10

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
A. Appendix
The Appendix is organised as follows:
Section A.1 provides strong empirical support for the main theorem of this paper even when its assumptions are violated in practice.
Section A.2 presents the proofs of the paper's propositions.
Section A.3 provides auxiliary lemmas along with their proofs that support the previous section.
Section A.4 explores dynamical isometry in transformers, questioning their ability to propagate information faithfully through attention layers once the problematic outliers have been removed.
Section A.5 outlines comprehensive experimental details, ensuring reproducibility.
A.1. Justification of our theoretical assumptions
In this section, we empirically support the finding of our main theorem-a spectral gap arises in width within attention layers- layers-ever even evenn when its assumptions are violated. Our formal proof should be seen more as a proof of concept, and its conditions to hold are not necessary practice, rather made mathematical rigor.
The inputs do not need isometric in practice. In Figure sentences are tokenised with pre-trained tokeniser, making inputs to our randomly initialised BERT clearly non-isometric. We include X,X7 to illustrate its deviation from identity. When tokens are instead assigned random embeddings in Figure 9, the covariance matrix is closer orthogonality. Yet, the spectral gap and rank collapse persist in both cases. We also show the spectra of random attention layer with (synthetic) non-isometric input tokens to mirror Figure of the main in Figure 10.
The context length T does not need to be bigger than the token dimension d in practice. Note that without T S d, the inputs cannot be isometric so this extra assumption was solely made for this reason. We provide the reader with an ablation on the role y, the ratio between the context length and the token dimension. In Figure 10, we explore information propagation TinyBert, for which 7 1, hence necessarily the tokens must be non-orthonormal. The spectral gap persists, and rank collapse in width follows. Another ablation, this time conducted on synthetic data, can be found in Figure 10.

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
100
50-
22
200
200
4-
400
400
(a) X when T = 75
(b) XoX. when T 282
(c) XuXo when T 446
7
7 446
0.02 001 000 0.21
5.5
1.00
1.65
(d) Eigenvalues in the complex plane of the attention matrix in the first head of layer 1.
Figure 8. Spectral gap (second row) persists even when the inputs are not orthogonal (see first row). Experiment conducted with a randomly initialised RoBERTa encoder, hence d 768 and T < 512 construction, real text data from abstract, processed by a pretrained tokenizer available on HuggingFace. We present single realisation, though this behavior is consistently observed across multiple
12

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
170
300
33
106
230
-30
02-
420
480
-602
230
-400
900
(a) XoXo when T= 77 75
(b) X when T = 282
(c) XoXo when T = 446
T- 75
T 446
0.00
-0.02
-0.04
(d) The eigenvalues in the complex plane formed by the attention matrix in the first head of layer align with our proof of concept.
Figure Spectral gap (sccond row) persists non-synthetic yet random data. Using randomly initialized RoBERTa encoder 768, < 512 by construction), we substitute the word embeddings given by the pretrained tokenizer with random ones, making the input covariance matrix closer identity (first row). present single realisation, though this behavior consistently observed across multiple runs.
13

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
T= 200
T= 200
0.10
T 400
0.10
T 400
T = 800
T 800
0.05
0.00
0.00
-0.05
-0.05
0.0
0.1
0.90
0.95
1.00
1.05
0.1
0.0
0.1
0.90
0.95
1.00
1.05
(a) 7 = 0.1
(b) 7 = 0.5
T 200
== 200
T 400
T 400
0.05
0.05
0.00
-0.10
-0.10
/
/
-0.1
0.0
0.1
0.90
0.95
1.00
1.05
-0.1
0.0
0.1
0.90
0.95
1.00
1.05
(c) 7 2
(d) 10
Figure The cigenvalue distribution of single attention layer for non-isometric synthetic input and different values of 4 The input is generated by drawing i.i.d. Gaussian entries followed by normalising the rows, so cach token has unit length but they are not necessarily orthogonal. gap between bulk and edge persists despite bulk looking different r
14

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
7- 155, 0.
0.005
7 312, 1-0
T 437, y.
+0 002
cbw cbe
2i0
(a) Spectral gap in the attention matrix of the first head layer 1.
1.0
1.3
125
1.29
3.55
1.9
1.5
(b) Stable rank quickly collapses in width, even when 7
Figure 11. Spectral gap (top) persists in TinyBert encoders for which the value of y only the sake having a mathematical proof. As an almost direct consequence the spectral gap in attention matrix, stable rank rapidly collapses (bottom) width. Here d 312 by design and increase 7 by feeding the network with longer sequences. Whilst the eigenvalues (top) are shown for one specific realisation (but consistently observed across runs), the stable rank (bottom) is averaged over runs.
highlighting the generality of our result, which constrained
15

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
A.2. Proofs
Proof of Theorem 2.1. Let us show that the attention matrix AXx)) satisfies Definition 3.1 by demonstrating that the random variables
Xo W
W W
X
Zn exp
ddg
i.
are i.i.d. with a finite fourth moment.
Since the key and query matrices are initialised as Gaussian i.i.d. matrices and the input data X, is isotropic, W X, wP and W R = Xy W follow the same distribution as w and WT. Each Zt.5 can be written as the exponential of the inner product between the i-th row of w and the j-th row of W thus Zu, are i.id. and we only need to prove that 1 < oc. Let us define
dge W 1.
Ud ==
W
to be the dot product of the first row of W and the first of W So, Ud is simply the sum of dgt i..dd copies of U1, the product of two independent Gaussian random variables, whose density is known to be
fi
Ko To gk qk
where Ko is the modified Bessel function of the second kind. Therefore, the probability density function of U2,, is given by the dog-fold convolution
Sag (2) = fi(x) +...)......... fi(x)
times
It is also known that (x) asymptotically behaves like v I and that the convolution g h decays at least as fast the slower of g and h. Combining these facts, we conclude that fdzz decays at least as fast as e em i.e.
fe, (z) )(1)) lel
for some polynomially-bounded g. Now we can bound our quantity of interest
4Ud
E 21,1 = E
exp
ddg
an/ Vann 9((x))e- lxl
dx
< 00.
Vdye long < i.e. dgp 16.
is now time to invoke an important result, that we first state.
Theorem A.1 (Spectral gap in i. Markov matrices, (Bordenave et al., 2011)). Let A E TxX be i.i.d. Markov matrix whose underlying distribution has variance o² Then, X₂ (A) and almost surely,
lim (A =1 and lim 82 s2 (VTA)= 20 while +
lim T
(22(VTA) 2
(19)
T
Having shown that (X)) forms an i.i.d. Markov matrix since the Zz,j' 's are i.i.d with finite fourth moment, the spectral gap as described in Theorem 2.1 follows immediately.
Proof of Proposition 3.4. Fix l 2 1. By definition of stable rank, we have
ΣT,
(Ee)
(xX)
ΣT, (X,,
ST (EE)
>f(E))
(X.))
(((e)
(X+)
<1 + (T - xx(e))
1
(X,)
x{(ee)
16

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
For T large enough, let say bigger than some To, Theorem A.5 provides deterministic upper bound, i.e. 82(X,) K for some constant K. Moreover, Theorem A.4 gives the bound ( E (1 -t, +++1) with (overwhelming) probability PGT for arbitrary t > 0 and 7" bigger than some Ti. Thus, for T 2 max max Ti),
(T K
S (E,) < 14 +
T (1 t)
with probability at least Pt.. Therefore, the event
lim sr T-00
(Et)
holds with overwhelming probability.
Proof of Proposition 3.6. Note that we will treat the matrix torrxx-to-mat to-matrix derivatives such as OX2/OW/ not tensor (in IRT xdxdxd), but as its matricised version (in 12Tdxd"). We make use of the chain rule to compute the gradients of interest. Namely, at layer& L.
0x Ow!
UXL OX. OXm aw!
(AAL ... Aet1) @ 141.. WE) (AAe... A,x,,.. w/21 & &Ia)
(A w 8 W. We..... W.).
P
P2
Then, by properties of Kronecker product, we have
0x
0XL Ow!
- Σ S
Σ o (P)))) 2 (P))
(20)
ow! V
The largest singular value of a product of i.i.d. Gaussian matrices has been studied extensively, e.g,, see (Akemann et al., 2013; Mlotkowski et 2015). (Nait Saada & Naderi, 2024) show that, almost surely,
(P2) T (L + 11)
(L ---- (1 o(1))
On the other hand, by Theorem A.4, 81 (P)) concentrates around = with overwhelming probability, i.e., for T large enough,
(P)))) E (1,,
with probability at least Pt,r. Altogether, with overwhelming probability we have
0x, OW! F
2 (1 (1-1) (L - l 1-
lim T
1)-1) 1-
One can get better bound in the single-layer case (L = L = 1). Since P, = Lts we can rewrite (20) as
8x
ox
Σ
Σ 3 (L) 2 d (P
dWY
dwY
F
while s?(PP) = (0)))s((AAA)) (X,) = (11) almost surely. Therefore, almost surely, the following improved bound
0x,
lim T
CC
T
OWV
holds.
17

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
Proof of Proposition 3.7. The resolved stable rank be written as,
T-
E (A) A W AA Ax ..))
T
By submuliiplccatiity the operator norm,
sr +t)
T
-= ( AY wy) (
T
(((AF)
(Xo (WW)))
Let us call Py the fraction of squared singular values of A W W/ above a certain finite threshold i.e.
PT
Then, trivially
ST [ T
c Pr
(21)
87 (A
(A ) X. W S (w
Assuming the asymptotic freeness of all attention matrices A ..... A and weight matrices WY = W1, W...., We we may write the limiting squared singular value distribution of X as the free convolution of the corresponding Marchenko- Pastur distributions:
M = MP (1,0A) MPP( aw B.MBe-1
(1, OW
Then, almost surely,
f d.M.
Pf - P=
The distribution M is compactly supported on the interval [0, s where st,28 does not depend on T. So, by choosing 2, we can make e P non-zero constant. Moreover, the denominator of (21) converges almost surely to some constant (in 7), i.c.
(e-2( s((Y) + (2 22 1)(1)))111111)11 y 11/2/2.
Thus, almost surely,
Sr
cP
lim -0000
^α 2 2846-11441-1 + 1/2/2 > 0.
T
Proof of Proposition 3.8. Let us compute the resolved gradients:
ox
exx OX+
OW!
OX OW!
A
A W
wX
Ø
[WY
2+1.. W
Therefore,
ax 1
0x
7)] bW!
ox
E
E
tr
T
oW!
E
tr
(P)
(P))T) tr (PpP.)).
18

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
Assuming PT and P2 are asymptotically free, we have
0x t
lim d-ro0 d (T(((((()))) lim
d=o00 d (T((PP))))
lim d0000
E
ow
For each product matrix, the normalised expectation on the RHS of the above converges to the first moment of its limiting squared singular value distribution. By scaling them properly, i.e.
P VTAL VTAL 1
XoW, WY Vd
Vd W T1/22-
1 W
P2 =
Vd w d-11-0/22
we make sure that those limiting distributions (free convolutions of Marchenko-Pasteee distributions) are compactly supported on an interval of length (()) and, hence, both C₁ = lim ((tr((((PP)]] and C2 = lim ((m)((P)))))) are constants. Thus, since T = yd.
0X OW! V
lim d-000
= -d ) Cd
d
or
0x OW!
lim
C.
d+000
Proof of Theorem A.8. Since T = yd, we can write
X = A A
W VAA VTAL 1 WY)
X WY
Vd w/).
Each of the rescaled matrices above has squared singular values that almost surely follow a Marchenko- distribution MP(p,a), a), where is the ratio between the numbers of rows and columns of each matrix, and a the variance of its entries. Therefore, almost surely, the squared singular values of Xx, or equivalently the singular values of E,, follow a distribution M which is given by the free convolution
= MP (1, (A & MP(7) Mp (1,ov/V7).
The moments of such distribution are given by LemmaA.7 in the general case. Substituting the corresponding values from our setting gives the desired result.
Proof of Theorem A.9. Let A = A ....A TXT and W WY ... wy e Bddd Then
J = W Taxra
and compute the ke-th moment of its limiting squared singular value distribution
lim E (M
= lim E tr((A-A & w' )d]
T,d+000 Td
T, Td
lim E I(A-AAA
) t (WW)))
Td-+2
using simple linear algebra. Under the assumption that the matrices A and V are asymptotically free, the above limiting moment can be written as the product of individual limiting moments, i.e.
lim (
1 lim E ( d
lim E T
Td Tat
T-
19

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
where each factor equals the k-th moment of the limiting squared singular value distribution of its respective matrix. For both A - and WY the limits exist almost surely, and equal (up variance factor) to the well-known Fuss-Catalan numbers, defined by
1 Ek
(kk + k.
FC²(k))
+
k
Therefore, almost surely,
lim E L (r (J 7,0+000
- (ca)) x (F,)).)(())
Simple calculations in the case & = 1 and h = 2 yield the specified formulae for mean and variance.
A.3. Lemmas
Lemma A.2. Let W1 E Rxx and W2,... Wg E Rdxd be independent Gaussian matrices with (.,, 1) entries, and u E R² a unit vector. Then,
[[ss(un T W,... W.). = =d",
(22)
and the event
8i (uu W Wg) - <t
de/2
holds with overwhelming probability.
Proof. First of all, note that the distribution of s1 (uu WT... Wg) is independent of the choice of since W Wi W are rotation-invett invariant. Let us write u W1 aruj, where u, E Ied has length 1. Similarly, define
u W
0;+1 | Waiilla u
06--1
for 1 < sis 9- 1. So, can write
81 (uu W, W.... Wg) 81 [ Wg)
51 (u(aja2u))) W,)
Og 81 (um))
01 Clg
where $1 (uu,)) = 1 since un? naturally takes the form of an SVD with single nonzero singular value equal to 1. The random variables o1,.... are independent (by independence of W, ') and identically distributed (by of Wi's). Without loss of generality, we can substitute e (the first column of the identity matrix) for u or u, to get
||| W.l = ||w||
where w E Rd (the first row of W,) has i.i.d (00, entries. Thus, E(a,) = E w ( d, and by independence of a;'s we have
Eis W.... W = d
Moreover, since each of has chi-squared distribution with d degrees of freedom, can write it as the sum of d independent squared standard Gaussian random variables af Σ/= w Thus.
d Σ Z7
(uu Tw
W,)
o
2 + + w
j=1
20

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
where each Z is the product of independent (V00 1) random variables, and therefore is sub- Heb-Weibuull with parameter 2/q. We shall apply generalised Bernstein's 's inequality for the normalised sum of mean-zero sub- Heibull random variables (Kuchibhotla& & Chakrabortty, 2022; Bong & Kuchibhotla, 2023), i.e.
N Σ X 2 v
u 2
P(
- 2 exp
CN min(
KB
(23)
K
where X,'s are independent mean-zer sub- Weibull random variables with parameter / and K == max; (23) on
max;|| 11
Applying
dv
1 i (uu Wi... Wo) - 1 de ==1
Σ
(23
1))
de
where each (2} - 1) is centered sub- Weibull with parameter 1/g. we get
1 d²
1 P( du
2=)= s1 u W1... Wg) - 1 2 u =P P - (z) - 1) -1) 2
de
j=1
S 2xxp [ - d min (u² u ///)
where we have absorbed the dependency on K = =1( 1) into C². Combining the above with the simple fact that = 1| 2 implies 2²² - 1| max(t, 12), we obtain for any t 2 0 that
PP
du/2 (uu W w,) - 1 2)
$ P
de (uu WT... Wg) -1 max(t, t2)
< 2 exp
[ Cºd min(t", 2//)).
ie. sul W.... W₂) is sub-Weibull with parameter 2/g and
81 (uu
WT... Wq) <t dn/2
holds with probability at least -2exp) d² min (2,,2/19)] 1//9)].ie.. i.e. with overwhelming probability (Tao, 2012).
Lemma A.3. Consider p i.i.d. Markov matrices A Ap TXT as defined in 3.1, and let 1rxT be the matrix full of ones. Then, almost surely,
81 (A, A1 lTxT) = P(T
(24)
Proof. Let first show si (Ap (Ap...A. (Ap...Ai) و as T' grows. Each matrix A; can be written the row-normalisation of table M, of i..dd random variables, i.c. A, D,M,, where D, is a T x T diagonal matrix containing the inverse row sums of M, The entries in M; have finite fourth moment, and, without loss of generality, mean 1 and variance o2. Thus,
(Tv/2 AI) s1 DD,,. D...M...Dii.
< s1D (T (T-2/2
88 (1 (TD))s) (T-
Following the argument given in (Bordenave et al., 2011), $1 (TD,) = 1 + o(1) and 8 (T 1/2x,) S VT + O(1), for all 1 S isss p. Therefore,
81 (TP/2 A A1) S +VT+)
<
21

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
which yields, almost surely, lim 81 (App... A1) S 1. The converse inequality is an immediate consequence of the clo- sure of the of i..d. Markov matrices under matrix multiplication, which gives X1 (Ap....A A1) combined with 51 (Ap... ................................ ..) 2 (Ar ... A.)]. Hence, almost surely, lim $1 (Ap (Al...A) A1) = 1. Let E IR' be the unit vector such that #lrxT= pp",, i.e. T- (1, 1)". A == A ppp". Since the rows of A sum to 1, our construction ensures that those of A sum to zero. We want to show that 81 (A+) = (1 + (()). To this end, consider the SVD of the matrix A+. There exist orthogonal matrices U, V and a diagonal matrix E = diag (s1 (A+), .,.)..,,(((A +)) such that
Also, let A = Ap Ap...
A1 and define
(n(A
A+ UEVT
Note that since A p O, the matrix has rank at most T 1 and thus sn (A)) O. We will now try relate the singular values of A- to those of A, observing that A is rank-one perturbation of A-, i.e.
A pp T + A
pp T + UEVT
The squared singular values of A are exactly the eigenvalues of
AA ppp + U.
(25)
Since eigenvalues are invariant under orthogonal operators, we can multiply on the left and right by, respectively, UT and U to get a diagonal matrix perturbed by a rank-one matrix:
U UAAAU=U cp U
(26)
Taking the trace, we have
(A + +.... (A) 11 +7 +.... 2 (A
(27)
On the other hand, we can apply Thompson-Lidskiis interlacing inequalities (Thompson, 1976) on Equation (26) to get Combining Equations (27) and (28), one obtains
5 N (A ž (A) (A ( 5 (A) 2 0.
(28)
(((A) 5/ (A) 2 1+ (A
As established earlier, almost surely, lim 81 (A) = So we conclude that in the limit, almost surely, 32(A) 2 s1 (A+)). The converse is already given by (28). Therefore we have
$1 (A+) s2
almost surely. Note that the same reasoning is valid for the case p 1, and results in s (At+) - 82 (Ai) for any i.
Having shown the convergence of the largest singular value of A to the second largest singular value of A, we now show that (2) is of order T-p/2, To this end, note that the matrix can be written as a rank-one perturbation of the product of A ss i.e.
A = Ap.... Ai
= -T-11 + A
+
A+)
11' [++ + A Ai) + AA ... A
where some of the terms vanish since A 0= = 0. Given that rank A A inequality to get
At) = 1, we can apply Comppson-Lidsiin
81 (A p ... At) 2 82 (A).
22

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
By submutiplicatiityy of the operator norm, this implies 81 (Ap)) ... (1 2 s2(A). Moreover, we previously established that for each individual matrix A,, (A+) a.S. (2,),, and it is shown in (Bordenave et al., 2011) that 82(A,) _.". - 1/2, Therefore, we conclude that
(2 I (20T-1/2)²= - P33
Combined with Equation (A.3), we have
$I T IITTT) = =
almost surely.
Ap E TxT be independent ii.dd Markov matrices as defined 3.1 and W, E RRxx W2,... Wg E Rdxd be independent Gaussian matrices with i.i.d. (0,11) 1) entries. Then, for large enough T and d with fixed 7 T/d E (0, 1). the event
Theorem A.4. Let Ai.....
81(Ap p. A A. W.)
A1(Ap...AA
<t
da/2
holds with overwhelming probability.
Proof. We write A = Ap............... A = "pp + A and W = WI... Wg. Then, using the triangle inequality (A)
81 (B) S 81 (A + B) S (A) + $1 (B), we have
|51 T W) - (1 W)I < (AW) $1 (pp W + w)
S s W) + (1
On the other hand, it is well known that the largest singular value of Gaussian matrix converges almost surely to the soft edge of the bulk of the limiting density (Geman, 1980), i.e.
1 V7,
i=
$1
W.)
0.8.
Vd
2,
i 2.
Therefore, by submultiplicativtty of $1, we have
$1 (W)= S 81 (W1) .0.......111 s (W & (2vd+ = 29/9/2 + o(24/2).
(29)
Combining (29) with Lemma A.3, we get
$s S = =((d)t))
(30)
and thus, almost surely,
sI W) - O(d" I (AW) $1 (1pp () + (d)
Now, using Lemma A.2, we can assert that 81 (vpp) T W) is close to d9/2 with overwhelming probability, i.e.
s1 (p TW)
E (1-
da/2
with probability greater than Pt.a = 2exp
C' min (2/9)) Moreover, by (30),
(A W)
de/2 0,
as d grows. Thus, we can make the above quantity smaller than any given Altogether, for large enough T and d, the probability that
$1 (AW)
< <b+E
de/2
at least Ptd. Since E is arbitrary the proof is complete.
23

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
Theorem A.S. Let A Ap R be ii.d. Markov matrices defined in 3.1 and W RTxx W...., Wg Rdxd be independent Gaussian matrices with ii.. (0, 1) entries. Then, for T and large enough,
82 (Ap ......A A1 W.... ..... Wg) (
(31)
Proof. To exhibit a spectral gap in AW, suffices bound its second largest singular value by a quantity significantly lower than where the largest singular value is concentrated. To this end, observe that AW is rank-one perturbation of A-W:
AW (A+ + pp [opp)We w A-W + pp W.
Thus, using Weyl's inequality, we can write
$2 (AW) S $1 (A+w) s2 w) 51 (A(A (A- W))
Next, by submultiplicativtty of the operator norm combined with upper bounds in Lemma A.3 and(29),
(A- {s1(A~)s ((T
Therefore,
(AW O
Lemma A.6 (Bulk distribution of A+). Let A RXT be an i.i.d. Markov matrix, and let A == A-T-1 IXTT Then, almost surely, the empirical singular value distribution of 1/2 A + weakly converges to the quartercircular law T oC, i.e.
T
UVTA-
T
Σ
8 ((TTAL) Co i=1
Qa: 2
(32)
where Qo is the quartercircular law on the real interval [0, 2a] with Lebesgue density
V 121 10.00]
nO²
Moreover, almost surely, A does not exhibit any spectral gap.
Proof. Thospson-Lidskil' interlacing result for finite rank perturbation (Thompson, 1976) states that for any n x n matrices M and M' with rank M M') I k, we have
(M) S ss(MM') S *++((M).
This in turn yields the following bulk inequality.
FF FM rank M M')
n
where FM and FMr denote the cumulative distribution functions of PM and PM. respectively. Since rank(A A+) then
IF-TA FSTA 1l00 s T T-x 0.
Combining the above limit with the fact that VVTA
Co Qa almost surely (see (Bordenave et al., 2011)), we deduce that
U/TAL Co e..
almost surely. The almost sure absence of outliers the singular value distribution of A can be immediately inferred from Lemma A.3 when p= =
24

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
Lemma A.7. Let 0 o oo and 0 < 73 s 1 for S Sis n. Let M be the free multiplicatiive convolation of MP( distributions, Le.
M MP(1), M2(22, M... X MMP(mm (7h on).
Then the mean and variance of Z ~ M are given by
B(Z) oi
(33)
Vzr
of (n + 7122 + "1))))))))
(34)
Proof. The distribution in question M is the limiting squared singular value distribution of a product of rectangular independent Gaussian matrices, whose general moments are worked out in (Akemann et al., 2013). Simple algebraic manipulations lead to our result.
A.4. Can transformers achieve dynamical isometry?
In Section 3, we have established (i) the existence of an outlier eigenvalue/singllar value in the spectrum of softmax- -based attention matrices, and (ii) that removing this outlier helps with rank collapse and exploding gradients. In the absence the outlier, we can take a further step to analyse the bulk the spectra of the network's i boken-wise covariance and input- Jacobian- two quantities that have been shown to play a key role in signal propagation as we will see below. To this end, we will make use of free probability theory.
Free probability. The theory of free probability studies "non-commuiin random variables" such random matrices (see (Mingo & Speicher, 2017) for textbook introduction). Pioneered by (Pennington et al., 2017; 2018), the theory has found powerful applications in the analysis of large random neural networks. Notably, it provides tools to characterise the singular value distribution of sums or products of random matrices. Loosely speaking, "freeness" plays the same role for random matrices as independence does for (scalar) random variables. Freeness allows us to compute the limiting spectral density of a product M', from the limiting spectral densities of M,, and M just as independence enables the computation, for instance, of the moments of ZZ', given those of Z and Z'. Specifically, if uM, - v, VMV, + v', and M,, and M', asymptotically free, then
12-+00
1/ MV/,
M
where X denotes operation called free multiplicative convolution.
Let us assume that the input tokens are orthonormal, i.e. Eg = X,XX = I. As criterion for faithful signal propagation, one should require that Dx stay close to the identity matrix. Considering the spectrum, this means that the limiting singular value distribution of Σ should concentrate around the value 1. A natural approach, as demonstrated in the fully-conmected case in (Pennington et al., 2017: 2018; Murray et al., 2022), to adjust the model's 's hyperparameters to ensure that the mean of the limiting distribution is (1) and the variance is minimised. Proposition A.8 describes the moments of the limiting singular value distribution of Σz.
Proposition A.8 (Bulk of covariance kernel's singular value distribution). É in modified model (16) and Σ X E RTX be its covariance matrix. Let the underlying i.i.d. Markov matrices A have variance a and W/ have i.i.d. (0, of) entries. Let D I and De be the limiting singular value distribution of Σj. Then the mean and variance of Z ~ De are given by
Let = A be the signal at layer
B(( (o.ov//5) 2
(35)
(Zr E(1 + ((0++)((AOv//T7))
(36)
where 7 2 (0, 1].
The assumption y S is not essential and is made only to ensure that Σ/ is full-rank, avoiding trivial zero singular values. If y > 1, then the limiting singular value distribution is given by (1 - - ))0 + + DDe and the mean and variance should be adjusted accordingly.
25

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
It is evident from the above proposition that simultaneously controlling both the mean and variance of De is not feasible. Model (16) does not have enough hyperparameters to achieve this balance. Indeed, to prevent the mean from growing or shrinking exponentially with depth, the product σœo must equal v7. However, this constraint leads to the variance increasing linearly with f.
The Jacobian of the input-to-ouppe function f X, H _ represented by our modified transformer model, characterises the network's sensitivity to input perturbations up to first order, according to
0s
f(Xo + = f(xo) + c 0X Xo
U.
(37)
Let us consider the matricised version of the Jacobian layer E, i.e.
i vec X
J.- = 2 vee (x))
E TdxTd
(38)
The goal is to ensure that the spectral energy of the Jacobian concentrates around 1, thereby minimising distortion of the input space geometry-a property often referred as the dynamical isometry in the literature (see (Pennington et al., 2017)).
For our model (16), it is straightforward to show
Je (A² At) (WY- ..,) E RdxxTd
(39)
where & denotes the Kronecker product. Proposition A.9 describes the moments of the limiting squared singular value distribution Jr.
Proposition A.9 (Bulk of Jacobian's 's squared singular value distribution). Let X A A WY be the signal at layer i in our modified model (16). Let the underlying i.i.d. Markov matrices A have variance 0'2 and Wy have i.i.d. (0, ot) entries. Let De be the limiting distribution of the squared singular values Je == 8 /0Xp". Then the mean and variance of Z De are given by
B(Z) = (o Var(Z) = E()+ 2)(
(40)
(41)
Controlling the mean leads to quadratically growing variance, while minimising the variance is only achievable if aVOV < 1, which, in turn, causes the mean to vanish. Without considering more complex model, no choice of (v), can achieve our goal of dynamical isometry.
A.5. Implementation details
Architecture. The default model consists of a stack of single-head attention layers, with an optional LayerNorm inserted between them (denoted by "+ LN" in the legend) after receiving an optional skip connection from the previous layer (denoted by "+ skip" in the legend). When both options are enabled simultaneously, the configuration referred to as "+ skip + LN" By single-head, we mean that only one attention mechanism is computed, applied to the values and then multiplied by a matrix Wn which is initialised as the identity matrix.
Attention design. At initialisation, when the attention is labelled as "A", the matrix is sampled from the set of i.i.d. Markov matrices, defined in Definition 3.1, with variance of a 1. To achieve this, we sample random matrix B with i.i.d. lognormal entries and apply softmax row-wise such that A == softmax(()). The moments of B are adjusted precisely so that σ = 1. If "Identity A" is chosen, the attention matrix is a constant equal to the identity. When the attention is labelled as "A(X)", the key/query matrices are sampled from i.i.d. Gaussian matrices N 1) and the standard key-query
matrix is formed. a label indicates a "1", the forward pass of the attention mechanism is systematically (at initialisation) adjusted so that the spectral gap is removed, as in our modified model (16). If the label indicates "no outliers", then, in the forward pass, an SVD is performed, and the largest singular values are taken out as follows:
A no outliers
A
1=1
*In minor abuse of notation, we may write 2x, /ax, / as a shorthand for á vec / 3/8 ver (Xo).
26

---

Mind the Gap: Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers
where A Udiag (81 (A), ...,,,,8 8n VT. The cut off threshold is chosen such that it maximises the difference between consecutive singular values, i.e.
== arg max ($: 8.+1).
27


Reviews:
[{"review_id": "NUJ4ekpGWk", "reviewer": "Reviewer_uxct", "summary": "This paper studies signal/propagation in transformers with softmax-attention at initialization. Prior work has observed rank collapse in depth (in various model architectures), which causes all tokens to converge to a single representation. It has been attributed to repeated matrix multiplications, and is known to cause issues such as exploding/vanishing gradients. This paper uncovers a phenomenon called rank collapse in width, which is unique to softmax attention layer and occurs with increasing context length. Using Random Matrix Theory, the paper shows a gap between the largest and the rest of the singular values of the attention matrix. It also shows that rank collapse in width leads to exploding gradients and exacerbates rank collapse in depth. It then proposes to mitigate rank collapse in width by removing the outlier eigenvalue(s), and empirically shows that it also helps mitigate rank collapse in depth and the issue of exploding gradients.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments. Those with an editorial nature are applied in the revised draft, so we will address their main concern regarding assumptions. For additional figures (indicated by Roman numerals), please see https://anonymous.4open.science/r/spectral_analysis_transformers-C633/figures_rebuttal.pdf.\n\n**Orthonormal input tokens.** It is true that input tokens are usually not orthogonal. Nonetheless, we wish to point out that this assumption is only made for the sake of our mathematical analysis which should be seen more as a proof of concept. The emergence of spectral gap and rank collapse in width still occurs in practice without those assumptions. Let us clarify the experimental setup of Fig. 3. Sentences are tokenised with a pre-trained tokeniser, making inputs to our randomly initialised BERT clearly non-isometric. We include $\\mathbf{X}_0 \\mathbf{X}_0^\\top$ to illustrate its deviation from identity. When tokens are instead assigned random embeddings, the covariance matrix is closer to orthogonality. Yet, the spectral gap (Figs. I(d) and II(d)) and rank collapse (Fig. 3 of the draft) persist in both cases. We have clarified this setup in our draft to emphasise this aspect. Another illustration of why orthogonality is not needed per se comes directly from Fig. 6, where the spectral gap persists in deeper layers where representations are definitely correlated. Thanks to your review, we have also added to the appendix additional plots (Fig. IX) of the spectra of a random attention layer with (synthetic) non-isometric input tokens to mirror Fig. 1. \n\nCharacterising the spectrum of a generic row-normalised matrix remains an open problem in Random Matrix Theory. Our work advances the understanding of randomly initialised attention layers by assuming orthogonal input but we should acknowledge this is a limitation of our work and will emphasise this aspect in our revised version thanks to your comment. Moreover, we would like to share with the reviewer that we consciously chose to put an assumption on the input data (for our proofs to hold) to fully capture the complexity of the attention mechanism, rather than simplifying the attention mechanism as it is commonly done in the current mathematical treatment of transformers (e.g., in [1], where rank collapse is addressed under the assumption of uniform attention---essentially, no attention).\n\n**Ablation on $\\gamma$.** Note that without $\\gamma \\leq 1$, the inputs cannot be isometric so this extra assumption was solely made for this reason. To more directly speak to your question we included new plots that explore information propagation in TinyBert, for which $d>T$ hence necessarily the tokens must be non-orthonormal. The spectral gap persists (Fig. III(a)) and rank collapse in width follows (Fig. III(b)).\n\n**Large width.** Although the theorems are formally stated as $T,d \\to \\infty$, we have also determined precise rates of convergence, which allows one to derive bounds in the finite case. Moreover, the convergence is sufficiently fast that, for typical practical values of $T$ and $d$ (on the order of $10^2$ to $10^3$), the quantities are already well approximated by their limiting values, see Fig. 1. In fact, it is one of the key messages we want to convey that, given the increasing scale of these architectures, asymptotic analyses (such as Random Matrix Theory) can be appropriate tools for their study.\n\n**Scaling.** As noted in Remark 3.3, we scale values as $\\mathcal{N}(0,1)$ instead of $\\mathcal{N}(0,1/d)$ to compensate for the softmax attention that shrinks in all but one direction. This ensures layerwise tokens remain of order one after removing the spectral gap. Alternatively, scaling by $d^{1/2}$ post-softmax would yield the same effect. Since we omit LayerNorm in our analysis, rescaling is crucial to maintain token magnitude. Note that stable rank is independent of scale so we lose no generality here. In contrast, scaling does impact gradients, so vanishing gradients in Noci et al. under traditional scaling become exploding ones in our setting.\n\n**GPT architecture.** We investigated signal propagation in a GPT2 transformer, as suggested by the reviewer (see Figs. IV(a), IV(b)). The attention matrix is now triangular due to the causal mask and its eigenvalues (real) can be read off the diagonal of this matrix. Although interesting, we feel that because it is a decoder-like architecture, an additional layer of complexity comes with considering causal masks that our theory does not cover, so this setting falls outside of our scope. Yet, we share with the reviewer the results of such an experiment and hope to maybe gain insight from them for potential follow-up work.\n\nWe added a proof sketch in our revised work, thanks for the great suggestion. We hope you will consider updating your score based on our rebuttal.\n\n-----\n[1] Noci, L. et al. (2022). Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.", "claims_and_evidence": "Yes, for the most part.\n\nThe paper clearly mentions that the focus is softmax-attention transformer models at initialization. It presents theoretical results, supporting the main claims (spectral gap leading to rank collapse in width, and connections with rank collapse in depth and exploding gradients) as well as empirical results showing improvements with the proposed approach to mitigate rank collapse. \n\nMy main concern is the strong assumptions for the results. It is stated that the results are in the regime where context length $T$ is large. However, an additional assumption that $\\frac{d}{T}\\in (0, 1]$ is also required for the theoretical results and is not discussed anywhere. As $T\\rightarrow\\infty$, the token dimension $d$ also tends to $\\infty$ which seems very strong. In addition, the paper assumes that the input tokens are orthogonal, which seems unrealistic.", "methods_and_evaluation_criteria": "The method to mitigate rank collapse makes sense. However, I have the following concern with the evaluation. \n\nWhile the paper evaluates the method with standard BERT-based architecture, it should also include results with GPT-based architecture. More importantly, it should evaluate the effect of relaxing the assumption $X_0X_0^T=I$.", "theoretical_claims": "I skimmed through the proofs for the main results (Theorem 2.2, Props. 3.4 and 3.5), although I did not check them carefully.", "experimental_designs_or_analyses": "This is mainly a theoretical paper. The experimental design is fairly standard.", "relation_to_broader_scientific_literature": "As discussed in Section 1.1 in the paper, prior works have observed and investigated the phenomenon of rank collapse with depth in transformers as well as other architectures (at initialization). It has been linked to the issue of vanishing/exploding gradients which can disrupt training. This paper analyzes a new phenomenon, rank collapse in width, how it relates to rank collapse in depth and the issue of exploding gradients, and proposes a method to mitigate it, which also seems to resolve these other issues. It also discusses two recent works, Ye et al. 2024 and Ali et al. 2023, which indirectly and directly mitigate rank collapse and show performance improvements.", "essential_references_not_discussed": "To the best of my knowledge, the paper discusses and cited the most relevant works.", "other_strengths_and_weaknesses": "Strength:\n\n- The paper uncovers the phenomenon of rank collapse in width, studies it theoretically and shows connections with rank collapse in depth and vanishing/exploding gradients, which is interesting.\n\nWeaknesses:\n\n- The assumptions are quite strong (see Claims and Evidence section above).\n\n- The writing should be improved, for instance, the paper doesn’t provide intuition/proof sketches for the theoretical results (see Other Comments or Suggestions section below).", "other_comments_or_suggestions": "If you have any other comments or suggestions (e.g., a list of typos), please write them here.\n\n- The paper should include some intuition and proof sketches for the main result. Looking through the proofs, they are not very long, and including a few key steps in the paper would be useful for the reader. For instance, in Prop. 3.4, the convergence rate is $O(T^{1-4l})$ and without the sketch, it’s unclear how this dependence on layer index $l$ arises.\n- The paper should check the use of \\citep and \\citet.\n- There are some statements that are unclear and should be rephrased. For instance, lines 59-60 (second column).", "questions_for_authors": "The paper states that “using a different scaling that makes the same quantity explode rather than vanish” (refering to gradients with respect to the value matrix $W^V_l$ at layer $l$). Can the authors elaborate on why this is the case?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l6QNJ27YLu", "reviewer": "Reviewer_qzvJ", "summary": "The paper shows that random attention layer stacks exhibit rank collapse in width (context length and latent dimension) by analyzing the spectral gap of the corresponding random matrices. They then propose a fix that replaces the attention matrix with another related matrix without spectral gap (in the limit) and show that the resulting networks do not exhibit rank collapse or gradient explosion. Experiments are done on BERT to verify the theoretical findings.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive view of our work. We are happy to see that they value our contribution. We would like to address the points they raised as weaknesses:\n+ The reviewer rightly points out that the direct connection between the pathological rank collapse behaviour at initialisation and the training dynamics is not yet as well established as the analogous edge-of-chaos line of research for fully connected and convolutional networks, e.g. Figs. 5 and 6 in [1]. In the case of transformers in particular, one could argue that as long as the input tokens do not entirely collapse into a single representation during the initial forward pass—i.e. if *some* information manages to propagate through the network—the early steps of the optimisation algorithm may help recover from the collapsed rank and steer training back on course. Nonetheless, empirical evidence suggests that escaping such a pathological landscape can be more difficult in practice; see Fig. 1 of [2].\n\n+ While our analysis is admittedly not comprehensive of the whole transformer block, it provides valuable insights and sheds light on a hitherto overlooked phenomenon (even at the first layer) and its origin, namely rank collapse in width and softmax activation within attention layers. We also believe there is value in our results showing how the same rank collapse is impacted by changing the softmax activation to other options being advanced, such as ReLU and sigmoid. These other choices exhibit a smaller spectral gap than the softmax activation, which might explain why they are being advocated. Moreover, our resolution to subtracting the spectral gap can also be applied to ReLU and other attention activations. We believe this insight will be useful to practitioners studying alternatives. We wish to emphasise that, since rank collapse in width has a totally separate cause from its better-studied counterpart, rank collapse in depth, the ideas proposed to fix the latter do not necessarily affect the former and further distinct research is required. Thank you for the reference.\n\n+ Finally, in response to some reviewers' concerns about the assumptions used in our proof of concept, we have conducted extensive ablation studies that the reviewer may find useful in confirming their positive assessment of our work. They can be found here: https://anonymous.4open.science/r/spectral_analysis_transformers-C633/figures_rebuttal.pdf\n\n--------\n[1] Schoenholz S. et al. (2016). Deep Information Propagation.\n\n[2] Noci, L. et al. (2022). Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.", "claims_and_evidence": "The paper claims that spectral gap in the random attention matrix at initialization causes rank collapse in width - a phenomenon well established in practice. Their proofs are sound and well-presented. The author also shows empirical evidence of their theoretical fix working on BERT-based transformers, which are popular in practice.", "methods_and_evaluation_criteria": "The tools used to prove the authors' propositions are well-developed from random matrix theory. Similar literature that studies transformers' rank collapse also made use of the spectrum. However, to the best of the reviewer's knowledge, this is the first work that proposes a successful and simple fix.", "theoretical_claims": "Proofs presented in the paper is correct as far as the reviewer has examined.", "experimental_designs_or_analyses": "Experiments done in the paper is well thought out and well-accommodate the theoretical claims.", "relation_to_broader_scientific_literature": "Rank collapse is observed empirically in neural networks mostly due to repetitive matrix multiplication, usually referred to as oversmoothing. This phenomenon is shown to have occurred when the attention matrix is degenerate to be a uniform attention matrix, all entries sharing a numerical value. The paper argues for why the attention matrix, at initialization, would be uniform - or rank collapse in width, thus resolving a key assumption in literature.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The paper is rigorous, yet simple. The proposed fix to rank collapse is simple and thus scalable, making the methodology generally testable even in large LLMs. \n\n\nWeaknesses:\n- The paper only shows rank collapse at initialization. In practice, the networks are optimized before being use. This suggests that such collapse might be a non-issue if the optimization algorithm can somehow resolve the degeneracy. \n- The paper consider stacks of attention layers, while realistic transformer have more complicated structures, with layer norms and/or masking. The paper \"On the Role of Attention Masks and LayerNorm in Transformers\" by Wu et al. in 2024 could be a good reference to parse some of the ideas of this paper to more complicated architecture.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TUvRc0RnP4", "reviewer": "Reviewer_W8Pc", "summary": "The authors study attention layers randomly initialized, looking at signal propagation or exploding/vanishing gradient issues from a rank perspective. Notably, using random matrix theory tools, they identify a new rank collapse that occurs in width, i.e., in the context size. Via a careful theoretical analysis, they showcase how it is related to the exploding gradient problem and propose a practical fix to solve the rank collapse in width. They conduct experiments with synthetic inputs on toy and real attention layers, e.g., from BERT, to validate their theoretical findings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind support of our theoretical development and for highlighting the \"SAMformer\" paper \"SAMformer\" that we looked into and will cite. For extra figures (indicated by Roman numerals), see https://anonymous.4open.science/r/spectral_analysis_transformers-C633/figures_rebuttal.pdf.\n\n+  Addressing rank collapse is motivated in multiple ways: (i) Improved quantization, crucial for compressing LLMs. Bounding the largest singular value by solving rank collapse ensures entries stay within a defined range, aiding quantization; (ii) Expressive initial representations. While it's still unclear what defines a good initialization, one might argue for avoiding rank collapse so that token representations remain diverse (as opposed to collapsing into the same vector); (iii) Controlled gradient norms. [2] shows gradients vanish after rank collapse, highlighting the importance of maintaining rank; (iv) Better generalization accuracy. [4] empirically supports our approach. While they don't directly address width, they propose subtracting the leading first-order term $\\frac{1}{T} \\mathbf{1}_{T\\times T}$ from the attention. Their motivation lies in addressing oversmoothing in graph networks, tied to extremal eigenvalues. Their results show this subtraction increasingly benefits performance as datasets grow more complex or deeper, significantly boosting accuracy. (v) Finally, as reviewer qzvJ pointed out, our work helps bridge a theoretical gap in the understanding of attention layers in the literature.\n\n+ Regarding orthogonal inputs, we refer you to our response to reviewer uxct.\n\n+ We agree that including the impact of additional network modules (as in [1, 2]) is important. We do so empirically in Fig. 4, which shows rank collapse persists even with LayerNorm and skip connections. Fig. 3 further compares our theory to real-world transformers, such as BERT, which include many modules beyond attention.\n\n+ Fig. 3 uses a pre-trained tokeniser, so inputs are not isometric (see response to uxct). As our work focuses on theoretical insights into attention at initialisation, we do not analyse training dynamics, which lies beyond our scope. We instead refer to [4], which offers an extensive empirical study on removing the spectral gap. Their work complements ours: they focus on experiments, while we examine the mathematical consequences of centering. Rather than replicate their study, we discuss it in the next paragraph. For signal propagation, we analyse the stable rank of token representations across depth.\n\n+ Thank you for highlighting this. We’ve expanded our discussion of how our results explain [3, 4]. In [3], the authors heuristically subtract two softmax matrices, implicitly reducing energy along the dominant eigenvector. This helps mitigate rank collapse, albeit less directly and without theoretical grounding. Nonetheless, it shares the insight from [1, 2]—that rank collapse is central. Our resolution, removing the spectral outlier, has a similar effect but is rigorously understood. In [4], the same proposal of subtracting the dominant $\\frac{1}{T} \\mathbf{1}_{T\\times T}$ term is made to “centre” attention. They show this benefits training. Our contribution is the mathematical rigour: we analyse what remains after centring and, by understanding the spectral gap, indicate what to do for other activations (e.g. ReLU, sigmoid) as shown in Fig. 1. We believe our theory supports practical efforts like [3, 4] to address rank collapse.\n\n---\n\nQ1. Rank decreases with matrix multiplication, which leads to collapse with depth. For a training instability link, see [2] (Theorem 3.2).\n\nQ2. Please refer to the 'Scaling' paragraph in our response to uxct.\n\nQ3. Assuming uniform attention (as in [2]) removes the role of attention by treating all tokens equally. We aim to refine this by analysing the attention once the leading $\\frac{1}{T}1_{T\\times T}$ term is removed. Think of uniform attention as the constant in a Taylor expansion—our $A^{\\perp}$ captures the next-order term.\n\nQ4. Great question. [2] proposes scaling residuals with depth to counteract collapse. But as we study rank collapse in width (even in the first layer), their fix does not apply. As shown in Fig. 4(a), modules from [1] do not significantly prevent collapse in width.\n\nQ5. Please see above.\n\nQ6. We agree with the reviewer and refer to [1] (Fig. 2), which also shows minimal changes in depth. However, we emphasise that even small changes in width can drastically affect depth. For instance, removing the spectral outlier in width causes a small change (Fig. V) but drastically alters depth behaviour (Fig. VI).\n\n---\n[1] Dong, Y. et al. (2021). Attention is not all you need: Pure attention loses rank doubly exponentially with depth.\n\n[2] Noci, L. et al. (2022). Signal propagation in transformers: Theoretical perspectives and the role of rank collapse.\n\n[3] Ye, T. et al. (2024). Differential transformer.\n\n[4] Ali, A. et al. (2023). Centered self-attention layers.", "claims_and_evidence": "The theoretical claims are supported by clear and detailed proofs, and the authors also provide experimental validation of their theory with synthetic inputs. However, the assumptions (orthogonal inputs) seem unrealistic, which impact the theoretical findings, in my opinion.", "methods_and_evaluation_criteria": "The authors provide theoretical results to better understand rank collapse in width. They propose a practical fix and conduct synthetic experiments to validate their findings. Hence, the method and evaluation criteria make sense for the problem at hand, although the experiments should illustrate the benefits of the approach to improve signal propagation and/or mitigate exploding gradients.", "theoretical_claims": "The theoretical findings are supported by detailed and clear proofs.", "experimental_designs_or_analyses": "I checked the soundess of the experiments. I believe that, while most of the contributions are theoretical, the experiments are too lightweight with a focus on small-scale, synthetic data and attention-only models. I think the submission would benefit from experiments in more practical settings and/or better empirical motivation to study the rank in width.", "relation_to_broader_scientific_literature": "I find that related work and prior works are well introduced and compared. The submission's contributions are interesting and are part of a growing interest in the literature on the theoretical understanding of transformers from a rank perspective.", "essential_references_not_discussed": "To the best of my knowledge, there were no essential references not discussed in the current submission. \n\nA potentially relevant paper is [1], which studies the limitations of transformers in time series forecasting by looking at the loss landscape, rank, and entropy collapse. The authors proposed SAMformer, which solves the loss's sharpness that leads to a significant improvement with SOTA methods, and in contrast, they showed that entropy collapse was benign. In the paper, they showcase block diagonal attention matrices with SAMformer (i.e., high rank), while the vanilla Transformer and other modifications suffer from rank collapse (Fig 6 and 12). The connection with the current submission is that the models considered are single-layer transformers; hence, the rank collapse does not come from the depth. In addition, the channel-wise attention means that the number of tokens is the number of features (multivariate time series input); hence, the context is high (up to  862), which could qualify as a rank collapse in \"width\". [1] could, for instance, be used as motivation for the current submission to study the rank in width (see weakness part) or offer other perspectives for the current study.\n\n*References*\n\n[1] Ilbert et al., SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention", "other_strengths_and_weaknesses": "**Strengths**\n\n- The paper is clear and well written\n- Notations and technical background are well introduced\n- I appreciate that the authors explain the context and potential impacts of each theoretical finding\n- The analysis is well conducted with elegant and well-explained proofs\n\n**Weaknesses**\n\nI list below what I think are weaknesses, but I would be happy to be corrected if I misunderstood some important aspects of the authors' contributions.\n- The motivation of the paper is not very clear to me. The authors focus on rank collapse at initialization in width but do not provide empirical validation that this hinders performance, signal propagation, or training stability in practical scenarios. Could the author elaborate on that?\n- Most of the theoretical findings assume that the inputs are orthogonal, and in particular, this enables the authors to show that the attention matrix is iid Markov. Since, to the best of my knowledge, inputs are not orthogonal in practice (this is data-dependent), from my understanding, the attention matrices will not be iid Markov anymore, even at initialization, which hinders most of the author's findings. Could the authors elaborate on that? An idea to improve the submission would be to add a discussion akin to Section A.5 of [2], where the authors theoretically and experimentally motivate the assumption 3.1 of a uniform attention matrix.\n- I believe the current setting is too oversimplified, which impacts the benefits of the derived insights for more practical settings. An idea to improve the submission would be to incorporate the other transformer's components in the analysis (like in [1], [2]) or at least discuss/ experiment on how they impact (or not) the current analysis.\n- The authors use their theoretical findings to derive a practical fix, but they do not test it in practical settings since the experiments, even for BERT, consider isotropic inputs and consist in observing the rank but not the impact on signal propagation or training stability.\n- The authors mention that their theoretical analysis resembles practical works like [3, 4]. Since the authors also provide a practical fix, more discussion on the novelty should be done, and I believe they should compare their fix with the ones in [3, 4].\n\nOverall, the idea is interesting, and the theory is elegant with the use of random matrix tools. However, I do not find the problem well motivated enough and the analysis convincing to solve rank collapse in width given the strong assumptions needed on the input.  This is the reason for my current score, but I remain open to modifying my score, provided the authors clarify the points mentioned in the weaknesses section.\n\n*References*\n\n[1] Dong et al. Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ICML 2021\n\n[2] Noci et al. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse, NeurIPS 2022\n\n[3] Ye et al. Differential transformer. arXiv 2024\n\n[4] Ali et al. Centered self-attention layers. arXiv 2023.\n\n**Update after rebuttal**: increased score from 2 to 3.", "other_comments_or_suggestions": "- The \"Impact Statement\" is missing but mandatory. Could the authors please add it?\n- There are parts of the code (e.g., visualization_losses.ipynb) that are not described in the paper, which is confusing. Could the authors clean the codebase such that only the relevant parts are kept?", "questions_for_authors": "1) The author mentioned in the introduction that rank collapse in depth is not specific to attention as it is simply due to matrix multiplication. While it is clear to me that matrix multiplication can lead to gradient vanishing or exploding via the chain rule, I have trouble seeing why it naturally implies rank collapse. Could the authors provide some reference to the observation of rank collapse in weight matrices of other architectures than transformer and where it impacts the signal propagation or training stability?\n\n2) I did not understand the author's justification in Remark 3.3 for initializing the value matrix with unit standard deviation. BERT(base model) has an embedding dimension of $768$ and a context window of $512$, meaning that $d$ and $T$, using the authors' notations, are comparable. However, in many other scenarios, one could have $d << T$, in which case the singular values of the value matrix do not compensate the scaling of the signal in all directions but one by $1/\\sqrt{T}$. Could the authors clarify this point, please?\n\n3)  [2] showed that the attention matrix at initialization resembles a uniform matrix (assumption 3.1 motivated in Section A.5 of [2]). Denoting $U = \\mathbb{1}_{T \\times T}$, it means that Eq. (7) of the current submission would become $\\frac{1}{T} U \\approx A = \\frac{1}{T}U+ A^\\perp$, which does not convene much meaning. What do the authors think of assumption 3.1 with respect to the simple derivation above?\n\n4) The authors show that even with their fix, the rank collapses in depth (Fig. 4(b)). Out of curiosity, could the authors experiment (or discuss) how solutions for rank collapse in depth proposed in prior works behave on the rank collapse in width (e.g., solutions from [1, 2])?\n\n5) Could the authors discuss more the solutions of [3, 4] and compare their fix to them? \n\n6) The rank collapse observed in practice (Fig. 3) seems quite small (from 1.20 to 1 in stable rank) given the size of the attention matrices (512 x 512 in BERT base model). As such, it does not convince me that rank collapse in width is important to solve. Could the authors elaborate on that? An idea to improve the submission would be to see how it impacts the signal propagation or vanishing/exploding gradients in practice (akin to Fig. 1 of [2]).\n\n*References*\n\n[1] Dong et al. Attention is not all you need: Pure attention loses rank doubly exponentially with depth, ICML 2021\n\n[2] Noci et al. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse, NeurIPS 2022\n\n[3] Ye et al. Differential transformer. arXiv 2024\n\n[4] Ali et al. Centered self-attention layers. arXiv 2023.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUvLBv0SnJ", "reviewer": "Reviewer_j3Cu", "summary": "In this paper, the authors discuss the phenomenon of rank collapse in width (i.e., for asymptotically large context length) in transformers at initialization. The authors point at the spectral gap in the attention matrix as the main cause of such collapse, and they devise a simple solution to remove the gap and tame the rank collapse.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough comments and their appreciation of our contribution. We address the points in the same order. For additional figures (indicated by Roman numerals), please see https://anonymous.4open.science/r/spectral_analysis_transformers-C633/figures_rebuttal.pdf.\n\n1. Thank you for bringing this up. We have addressed this in detail in our response to reviewer uxct, providing extensive illustrations showing that our findings hold in practice, even without these assumptions, which are only needed for our proofs. We will make sure to clarify these theoretical limitations in our revised version.\n\n2. Please see the response regarding $\\gamma$ in the answer to reviewer uxct. A precise quantitive characterisation of the spectral bulk without these assumptions is extremely interesting and we will explore it, but this is a multi-year programme of study and we believe it is best that the results we have so far be presented to the machine learning community sooner than later. Regarding Llama3, it is interesting to consider the case of a decoder-like architecture, for which an extra layer of complexity comes in due to the causal mask in the attention, which thus falls outside the scope of this paper. Thus, we kindly refer you to our paragraph GPT-architecture in our answer to uxct.\n\n3. Thank you for highlighting that this was not clearly explained.  The reason we do not include input dependence in later layers is that the resulting inputs do not satisfy the orthogonality conditions and, as mentioned above, a rigorous analysis in the non-orthogonality setting is beyond the scope of what can be theoretically justified at this stage. We will make sure to include a clear discussion on this limitation of our work.\n\n---\n\n1. Apologies for these parameters having not been clearly stated. The parameters are now explicitly stated in the revised manuscript, $d=768$ with input length as sentences from our own abstract that are stacked together before being processed with a pre-trained tokeniser that comes with Hugging Face's checkpoints of the corresponding model. All experiments are repeated $5$ times, with the average result shown as a solid line and individual runs displayed as faded lines.\n\n2. We sincerely thank the reviewer for these interesting suggestions which we believe enhanced our manuscript. We have undertaken these experiments and are happy to share the results with you. (a) We show in Fig. V how rank collapse in width is affected when modifying the information propagation in RoBERTa according to our proposed fix. RoBERTa is such that $d=768$. Whilst this small improvement might seem marginal in width (and the discrepancy with our theoretical guarantees can be explained by the extensive complexity of the architecture beyond our theoretical framework), its impact on the rank collapse in depth is remarkable, see Fig. VI. (b) The equivalent of Fig. 5 in our draft with the y-axis being the stable rank for consecutive layers following equation (1) is exactly Fig. 4b in our submission. The label '$\\mathbf{A}(\\mathbf{X})$' indicates that information flows as in equation (1), whereas '$\\mathbf{A}$' (e.g. in Fig. 5) is used when the attention matrix is taken as an i.i.d. Markov matrix, independent of the input. We realise this distinction was only clearly stated in Appendix A.4 and have moved this up to the main body in the revised version. Nevertheless, rank collapse occurs across consecutive layers, as shown in Fig. 4b, and the only modules that address it—among LayerNorm and skip connections—do so by removing the spectral gap. In case the reviewer was interested in seeing the same plot with i.i.d. Markov matrices instead, we upload an additional figure for which similar comments can be made; see Fig. VII. (c) Fig. 4b in our current submission seems to be what the reviewer is asking for. Regarding the gradients in deeper key-query attention layers, we provide an additional plot in Fig. VIII that reaffirms our analysis and give us some empirical insights on the relationship between rank collapse in width, in depth, and gradients explosion.\n\n---\n\n+ Thank you for highlighting these manuscripts, which we had not originally cited. We will include citations to them in our revised manuscript.\n\nWe have taken into consideration your overall impression that the limitations of our work were not clearly stated enough and have modified our draft accordingly. Should you have any further concerns that might affect your appreciation of our work, we would be happy to address them.", "claims_and_evidence": "The theorems are stated clearly, and experiments are carried out to validate them. However, I feel that the authors are quite a bit overselling their findings, or at least they are not clearly stating the limitations of their analysis. The main limitations that I find a bit difficult to justify are:\n1. The input matrix should be orthonormal, which is quite unrealistic in practical scenarios, since usually the vocabulary size is some 10x the embedding dimension.\n2. That the ratio between context length and embedding dimension ($gamma$ in the paper) is a constant smaller than 1, again quite unrealistic in practice (LLama3, e.g., allows context lengths that are more than 10x the embedding dimension).\n3. In the analysis for more than one layer, the authors remove the dependence of attention matrices from the input.\nI understand that in theory papers such assumptions are usually necessary. However, I also think that such limitations should be clearly stated and discussed in the paper, proposing possible solutions if possible, or stating why it would be hard to overcome them.", "methods_and_evaluation_criteria": "The proof techniques and experimental setting are sensible.", "theoretical_claims": "I checked the proofs at high level and I could not find any major issue that could invalidate the theoretical results presented in the paper.", "experimental_designs_or_analyses": "I think that the second main limitation of the paper is the experimental section, both regarding the experiments presented in the paper, and additional experiments that I think are important but missing:\n1. Regarding the experiments already present, details are missing about the setups used. For example, it is not clear what are the dimensions of the matrices involved in Fig. 3, nor what is the value of $\\gamma$. Without such information, it is hard to evaluate the prominence of the rank collapse, also because the stable rank seems to be already very small from the start (approx. 1.25) compared to what I expect the size of the involved matrices to be. \n2. I believe that the following very important experiments are missing, without which it is hard to understand the impact of the work:\n    a) Does your proposed solution to the rank collapse problem also work in real transformers models? That is, how would the curves of Figure 3 behave if your solution of removing the one-rank perturbation is applied?\n    b) Does the rank collapse in width also occur when many consecutive layers are used? That is, can you show an equivalent of Figure 5 with the stable rank as the y-axis?\n    c) Does the rank collapse and/or the gradient explosion also occur when the attention matrices in consecutive layers are kept as layer dependent? That is, in the case of many successive layers, if the usual formula (1) is used for the attention matrices instead of considering them as independent of the input, does the rank collapse in width still occur?\nI think these proposed experiments should be easy and quick to implement in your experimental setup, and they would greatly enhance the understanding of the scope of your observed phenomenon and of you proposed solution to it.", "relation_to_broader_scientific_literature": "The paper falls into the line of work investigating rank collapse in transformers. Previous literature focused on rank collapse in depth, i.e., as the number of consecutive attention layers go to infinity. This paper, instead, investigates rank collapse in width.", "essential_references_not_discussed": "Other works on rank collapse in depth could be cited, such as:\n- Feng et al., \"Rank diminishing in deep neural networks,\" 2022.\n- Geshkovski et al., \"A mathematical perspective on transformers\", 2023.\n- Geshkovski et al., \"The emergence of clusters in self-attention dynamics\", 2023.\n- Geshkovski et al., \"Dynamic metastability in the self-attention model\", 2024.", "other_strengths_and_weaknesses": "I believe that the work is a potentially interesting new direction for the field, if the authors are able to better justify their analysis. In particular, I would like the authors to discuss the theoretical and experimental limitations that I outlined in the sections \"Claims and Evidence\" and \"Experimental Designs Or Analyses\" above.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see sections \"Claims And Evidence\" and \"Experimental Designs Or Analyses\" above. I will be happy to revise my score if my concerns are addressed satisfactorily. Again, my main concern is that it is not clear from the provided experiments if the rank collapse in width is a serious issue in practical scenarios, and if the proposed solution helps in that regard.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies signal propagation in Transformers with Softmax attention, and demonstrates that randomly stacked attention layers exhibit rank collapse in width, by analyzing the eigenspectral gap of the corresponding random matrices. 
Experimental results are provided to support the theoretical claims.

The authors have done an excellent job during the rebuttal phase, and all reviewers are now convinced of the significance and merit of the work.

As a result, I recommend acceptance of this paper to ICML 2025.
Please ensure that the additional numerical results, proof sketch, and clarifications provided during the rebuttal are included in the final version of the paper.
Decision: Accept (poster)