Setting the Record Straight on Transformer Oversmoothing
Authors: Gbetondji Jean-Sebastien Dovonon
Michael M. Bronstein
Matt Kusner
Venue: Accepted by TMLR ()

Abstract: Transformer-based models have recently become wildly successful across a diverse set of domains. At the same time, recent work has shown empirically and theoretically that Transformers are inherently limited. Specifically, they argue that as model depth increases, Transformers oversmooth, i.e., inputs become more and more similar. A natural question is: How can Transformers achieve these successes given this shortcoming? In this work we test these observations empirically and theoretically and uncover a number of surprising findings. We find that there are cases where feature similarity increases but, contrary to prior results, this is not inevitable, even for existing pre-trained models. Theoretically, we show that smoothing behavior depends on the eigenspectrum of the value and projection weights. We verify this empirically and observe that the sign of layer normalization weights can influence this effect. Our analysis reveals a simple way to parameterize the weights of the Transformer update equations to influence smoothing behavior. We hope that our findings give ML researchers and practitioners additional insight into how to develop future Transformer-based models.

Published in Transactions on Machine Learning Research (07/2025)
Setting the Record Straight on Transformer Oversmoothing
Gbětondji J-S Dovonon
gietondjj.dovvoonn 22w.:/..222ccccoomm uk
University College London
Michael Bronstein
michael, bratsuud.bonssceinccs.ooo. uk
Universily of Orford
Matt J. Kusner
matt. matt.kusmerrrmmila quecee
Polytechnique Montréal, Mila - Quebec Al Institute
Reviewed on OpenReview: https: // openreview. net/ forum? id= #HI6gWLFF1
Abstract
Transformer-basee models have recently become wildly successful across a diverse set of domains. At the same time, recent work has shown empirically and thcoretically that Transformers are inherently limited. Specifically, they argue that as model depth increases, Transformers oversmooth, i.e. inputs become more and more similar. A natural question is: How can Transformers achieve these successes given this shorteoming? In this work we test these observations empirically and thcoretically and uncover a number of surprising findings. We find that there are cases where feature similarity increases but, contrary to prior results, this is not inevitable, even for existing pre-trained models. Theoretically, we show that smoothing behavior depends on the eigenspectrum of the value and projection weights. We verify this empirically and observe that the sign of layer normalization weights can influence this effect. Our analysis reveals a simple way to parameterize the weights of the Transformer update equations to influence smoothing behavior. We hope that our findings give ML researchers and practitioners additional insight into how to develop future Transformer- based models.
1 Introduction
In recent years, Transformer models (Vaswani et al. 2017) have achieved astounding success across vastly different domains: c.g., vision (Dosovitskiy et al. 2021 Touvron et al. 2021a), NLP Touvron et al. 2023
Wei et al. 2023 Kaddour el al.) 2023a). chemistryy (Schwaller el. al. 2019), and many others. However their performance can quickly saturate as model depth increases (Kaplan et al. 2020; Wang al., 2022). This appears to be caused by fundamental properties of Transformer models. Empirically, rescarchers first observed that depth was increased, even to just 12 layers, features became more and more similar to one another TTang et al. 2021 Zhou et al. 2021a b. Gong et al.
2021 Yan et al. 2022). Theoretically, these Transformer features converge to the exact
observations were characterized as (a) Input Convergence:
same vector (Park & Kim 2022
Wang et al.
2022
Bai et al.
2022); (b) Angle Convergence: the angle
between Transformer features converges to
Tang et al.] 2021
Zhou et al. 2021a Gong et al.] 2021 Yan
ct al. 2022 Shi al. 2022 Noci al. 2022 Guo et al. 2023 or (c) Rank Collapse: Transformer features
collapse to rank one matrix (Dong et al. 2021 Shi al.) 2022 Noci al. 2022 Guo al., 2023 Ali al.
2023). In practice, this has led to a search for replacements for Transformer layers, including completely
new attention blocks (Zhou et al.] 2021a Wang et al. 2022 Ali ct al.
2023), normalization layers Guo
et al.
2023 Zhai et al. 2023), altered skip connections (Tang et al. 2021 Noci et al. 2022 Shi et al.) 2022
convol utional layers (Park & Kim 2022). fully-connected layers [Liu et al. 2021a Kocsis el. al. 2022
Yu
et al.
2022a and even average pooling layers YYu et al. 2022b
But are Transformers destined to oversmooth? In this work we test the above observations theoretically and empirically. Theoretically, analyze the eigenspectrum of simplified Transformer layer: fixed attention,

---

Published in Transactions on Machine Learning Research (07/2025)
weights, and residual connection. We show even for this simplified setup that: (a) There are cases where all features converge to the same vector, but this is not inevitable, contrary to prior results; (b) Angle convergence is also possible, but not guaranteed; and (c) while rank collapse is likely, it is also not necessarily happening. Empirically, for existing pre-trained models we find cases where (a) features do not converge to the same vector, (c) feature angles do not converge to 0, and (c) rank does not collapse. In fact, our analysis uncovers a parameterization that allows one, in some cases better than others, to increase smoothing or reduce it. We observe that the sign of the weights of layer normalization plays a role in how much this parameterization influences smoothing behavior.
2 Background & Related Work
2.1 The Transformer Update
At their core, Transformers are linear combination of set of 'heads.. Each head applies its own self- attention function on the input X E Raxd as follows
+ XX
A = =ootmmaxx
(1)
where the Softmax(-) function is applied to each row individually. Further, Wo. W Rddk are learned query and key weight matrices. This 'attention map' then transforms the input to produce the output of single head: AXWv W proj; where Wr, Wp proj E Raxd are learned value and projection weights. Most architectures then add residual connection:
X X + AX Wy W proj-
(2)
These architectures also consist of layer-specific attention and weights, multiple heads (i.c., multiple A, W are to X and the outputs of each head is summed), layer normalization (either in the Post-LN format
2019b; Xiong et al. 2020) e.g., for BERT (Kenton & Toutanova 2019 2019), and ALBERT (Lan et al, 2019), or in the Pre-LN format (Baevski & Auli [Brown et al.) 2020). ViT (Dosovitskiy et al. 2021), and PALM (Chowdhery et al. 2023 architectures), and fully-connected layers. Unfortunately, these layers make eigenspectrum analysis intractable (we detail why this is the case in Section 3) However, recent work has demonstrated that simplified Transformer models have surprisingly similar behaviors full models (Von Oswald al.) 2023; Mahankali et al. 2023 Ahn et al.) 2024 Zhang et al. 2024 Ahn et al., 2023). We find that this is also the case for oversmoothing: even though our analysis considers the restricted update in eq. [ it can explain the smoothing behavior of full Transformer models (e.g., VIT and Dei'T models).
(Vaswani et al. RoBERTa (Liu et al.
2017
Wang et al.)
2018), c.g., for GPT
2.2 What Is Oversmoothing?
In deep learning, 'oversmoothing' broadly describes the tendency of a model to produce more and more similar features as depth increases. For Transformers, prior work largely uses one of three different ways to measure oversmoothing: (a) Input Convergence: Do the inputs converge to the exact same feature
vector? (Par & Kim 2022
Wang et al. 2022 Bai et al. 2022); ; (b) Angle Convergence: Do the angles
between inpuls converge to 0? (Tang el al. 2021 Zhou el al. 2021a Gong el al. 2021 Yan al. 2022 Shi
et al. 2022 Noci et al.
2022
Guo et al. )023)
(c) Rank Collapse: Does the rank of inputs collapse
[Dong et al.
2021 Shi et al. 2022 Noci et al. 2022 Guo et al. 2023 Ali ct al. 2023
Input Convergence. One way to formalize oversmoothing is through the lens of pional-processing Wang ct al. b022): the smoothing of a function can be measured by how much it suppresses higher frequencies in the signal, removing smaller fluctuations to highlight the larger trend. To measure the smoothing of the Transformer update in eq. (2) we can compute the ratio of high frequency signals to low frequency signals preserved X,- If this goes to 0 as d-+ 20, all high frequency information lost: the signal is maximally smoothed. To estimate these signals we can compute the Discrete Fourier Transform (DFT) F of X₃, via F(X;) FX, where F Ca is equal to Ft.1 == (2i((k-)((111) for all k; I (2...., (where i:= V-1),
2

---

Published in Transactions on Machine Learning Research (07 (07/2025)
and is 1 otherwise (i.e., in the first row and column). Define the Low Frequency Component (LFC) of X2 as [FC[ == F=- (,. FX; = (1/n)l1Tx, (where is the column vector of all ones). Further, define the High Frequency Component (HFC) of X as HFC[Xx] == F (o..,, = (I- (1/n)/1)11 Xx, We can now state the first definition oversmoothing:
Definition 1 (Input Convergence (Wang et al. for all X Roxd we have that
2022
The Transformer update in eq.
oversmooths if
HFC 2|2 |FC[Xx]]| O.
lim
This definition measures the extent to which inputs converge to the same feature vector. To see this, notice that the term in the numerator HFC[X]] (I- (1/n) 1//n111// XX goes to 0 if X₂ = 1x where Rd is a vector where entry H; is the mean of the ith column of X. This is because (1/n)11 = = Finally, the required condition X = 1x only holds when all input vectors are equal. In the following will refer to the ratio in the above definition as HFC/LFC.
Angle Convergence. Another way to quantify oversmoothing is via the cosine similarity between inputs: oversmooths if for all X E guxd we (
Definition 2 (Angle Convergence). The Transformer update in eq. have that
lim 2 t00 n(n 1)
x xx,0
1,
|x,ell||xx,e11l
where X,,6 E Rdd is the ith row of X, This measures the cosine of the angle 0 between every pair of inputs X Xjæ and iff 0 =0
Rank Collapse. Finally, we can also measure oversmoothing via rank collapse X This usually described as limy rank(X.) 1. While rank can be computed via a singular value decomposition (SVD), it is highly-sensitive to the threshold deciding when singular should be treated zero. Instead, Guo et al. 2023h use a continuous approximation of rank called the 'effective rank', first. introduced by Roy & Vetterli 2007
Definition 3 (Rank Collapse). Given X, E Baad, let X, = U.E.V be singular value decomposition
of X with singular values diag(E.) (oi.e)..., Gr,1l for r S (n,( and a1 2 ... 2 Fr. 2 0. Define the following discrete distribution according to the singular values as Ptą = σi,e/ E}=1 JjC The effective rank (Roy & Vetterli 2007) is the exponential of the entropy of this distribution: exp( Σt -Pi,t loggr.)). The Transformer update in oversmooths if for all X E Ruxd we have that
lim exp(- Σ Piee poge = t00
Roy & Vetterli (007) prove that 1 S exp(-
S where r is the rank of X²-
Notice that Definitions are progressively relaxed, i.e., if an update satisfies an oversmoothing definition, it also satisfies any later definitions. For each measure, we say that a model producing X, causes smoothing if the measure approaches the value in each Definition (i.e., towards for Definition 1 and 1 for Definitions & 3). Alternatively, the measures move away from the value in each Definition, then we say that the model causes sharpening (i.e., Definitions 1 & grow towards and Definition 2 shrinks towards 0).
2.3 Observations of Transformer Oversmoothing
The term 'oversmoothing' was first coined by Li et al. (2018) to describe how GNN node features become more similar with more rounds of message passing. similar observation was made for Transformers by Zhou al. (2021a). They observed that depth was increased, the cosine similarity among self-attention
3

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
Transformer Update
Input Convergence (Detinition 1)
Angle Convergence (Definition 2)
Rank Collapse (Definition 3)
[Dong et al., 2021]
X+
A
A
X
(Wang al., 2022] ours
[Ali et al., 2023]
(Noci et 2022]
[Wang al, 2022] IWu et el., 2024]
[onng al. 2021]
X
A
X
W
W proj
ours
(Noci at al., 2022)
ours
[Dong et 2021 [Noci et al., 2022]
X
X
A
X
W
W proj
[Wang at. 2022] ours
X
ours
ours
Figure 1: Theory of Transformer Oversmoothing. A v indicates prior work says that the corresponding Definition is always satisfied, x indicates it is not always satisfied. Note that if Definition is satisfied, then all later Definitions, which are progressively more relaxed, must also be satisfied. The asterisk at the bottom right indicates that Definition 3 is not guaranteed, but it is highly likely.
layers also increased. After this work many other works noticed that feature similarity in vision and language Transformers also increasod with depth (Zhou ct al. 2021b; Gong ct al.) 2021
Tang et al.) 2021
Raghu al.,
2021
Yan et al. 2022 Shi et al.) 2022 Wang et al.
2022 Park & Kim 2022
Bai al.)
2022
Choi et al.
2023
found. Multiple works around this time found that it was possible to improve vision
Transformers
by replacing self-attention layers with convolutional layers (Han et al.
2021 Liu et al.
2021b
Jiang et al.
2021
Touvron et al. 2021b Yuan et al. 2021 Park & Kim 2022)
2.4
The Theory of Transformer Oversmoothing
Figure ssows current work on the theory of Transformer oversmoothing for three Transformer updates.
Input Convergence. Wang et al. 2022) analyzed oversmoothing from the lens of signal processing (Defi- nition I. They showed that the number of self-attention operations tended to infinity, all inputs converge to the same feature vector, producing low-pass filter. They also analyzed the convergence rate when the residual connection, weights, multiple heads, and a linear layer is added, and found that convergence is not guaranteed. However, they argued that even with these additions oversmoothing still happens: 'it is inevitable that high-frequency components are contimuously diluted as ViT goes deeper', i.c., Definition [ holds. At the same time Shi et al. (022) invest oversmoothing using a different notion of input conver- gence. Curiously, while they argue that oversmoothing can be due to the parameters of layer normalization, their analysis seems suggest that without layer normalization, oversmoothing does not occur. Recently, Wu et al. (2024) analyze oversmoothing using a metric that is the numerator of the metric defined in Wang et al. (2022). They analyse the transformer update with layer-specific atlention and weight. matrices and layer normalization, but without the residual connection. Contrary to Shi et al. (2022) they show that layer normalization can prevent oversmoothing. We use the definition of input convergence described by Wang 2022) (our results also hold for the definition used in Wu et al. (2024)). )
et al.
Angle Convergence. As far as we are aware, there no prior work that directly analyzes oversmoothing from the perspective of angle convergence (Definition 2). However, if an update is shown to input-converge it will also angle-converge (and rank collapse), because imput convergence is stricter requirement than angle convergence (and rank collapse).
Rank Convergence. The first work we are aware of that developed theory around Transformer over- smoothing was Dong et. al.] (2021) using the notion of rank collapse. Initially, they showed that, without skip-connections, repeated self-attention layers converge double-exponentially to a rank 1 matrix. They then show that there exist models where skip-connections counteract this convergence. Noci al. (2022) con- tradict this, arguing that oversmoothing still happens when the residual connection is added, but this can be counteracted if the residual connection is scaled appropriately. Ali et al. (2023) show that rank collapse
4

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
ImageNet WOUFF
sentbert)) Delitton
Ingeer
lager
CIFAR100
DollMton
Delistoon
lager
Teger
Detetionn HEOUPd
The Pile
Defattonn
Figure 2: Are models oversmoothing? The smoothing metrics defined in Definitions (left column); center column), and (right column) layer number, for different models and datasets in vision and NLP. Each curve represents different model. We compute each smoothing metric after each attention layer (grey bars), fully-connect layers (white bars), and layer normalization layers (yellow bars). If oversmoothing was fundamental issue for Transformer models we would expect all curves in the left and right columns to decrease to 0 and 1, respectively, and all curves in the center column to increase to While ImageNet models and Gemma 2B broadly trend these ways, increasing model depth does not exacerbate smoothing, as predicted by prior work. Further, ViT-Ti [Touvron al. 2021a on CIFAR100 exhibits the exact opposite trend: smoothing is actually reduced by the model. These observations motivate investigation.
happens without a residual connection and value and projection weights. When a residual connection is added our analysis shows that it is possible to avoid rank collapse.
3 Do Transformers Always Oversmooth?
Given the current theory on Transformer oversmoothing, how are Transformer models so successful for vision and NLP applications [Kenoon & Toutanova 2019 Liu et al. 2019 Lan et al. 2019 Brown et al.
2020 Dosovitskiy et al. 2021 Chowdhery et al. 2023 To investigate this, we computed the above three
metries Definitions 13 on set of pre-trained models for vision and NLP that have been used prior work on oversmoothing (Wang al. 2002 Choi et al. 2023 in Figure We notice that for all ImageNet models (VTT-B, ViT-L (Dosovitskiy et al.) 2021), DefT-B [Tovvon et al. 2021a), DeiT3-L (Touvron et al, 2022 depth increases, do see the metrics approaching their oversmoothing values as described in Definitions Rank (Definition 3) docs not consistently decrease and stays relatively high for 12 layer models, but continues to drop as depth increased. However, we see something completely unexpected from the CIFAR model (VIT-Ti (Touvron et al.) (2011a)) All of the metrics show reduction in smoothing behavior as depth increases. Similarly, for The Pile model (Crammed BERT (Geiping & Goldstein 2023) we see behavior that appears to oscillate between more and less smoothing. These behaviors motivate further investigate the Transformer update.

---

Published in Transactions on Machine Learning Research (07 2025)
3.1 Preliminaries
Our strategy will be to understand the eigenspectrum of the Transformer update in the limit and to use this understanding to derive what the features X₁ converge to as l + co. This will allow us to understand if and when Definitions 3 hold. We start by rewriting the Transformer update, eq. ( to make it more amenable to analysis. Define the vec(M) operator as converting any matrix M to a vector m by stacking its columns. We can rewrite eq- vectorized as follows
(X.) (I W WT @A)vec(Xr-1).
(3)
proj
This formulation especially useful because vec(X, X
(I H Ae We now introduce an assump-
tion on A that is also used in prior work (Ali et al.
2023 Wang et al. 2022).
Assumption (Ali et al. (2023); Wang et al. (0022)
) The allention malrir is posilive, i.e., A > 0, and
diagonalizable.
This assumption nearly always holds unless A numerically underflows. In our experiments we never en- countered di = 0 for any element (i.j) R" Re A that not diagonalizable, in any architecture. Note A is also right-stochastic, i.e,, Σ, ai 1, by definition in eq. (I). This combined with Assumption immediately implies the following proposition.
Proposition 1 (Meyer & Stewart (2023)). Given Assumption 1 all eigencalues of A lie within (-1, There one largest eigenvalue that is equal to L, with corresponding unique eigenvector 1.
We leave the proof to the Appendix. We can now analyze the eigenvalues of the Transformer update equations.
3.2 The Eigenvalues
First notice that the eigenvalues of (I+ (I+HO 3 A) can be written in terms of the eigenvalues of H,
Lemma 1. Let X1, , XA be the eigenvaauus of A and let X :..... AH be the eigencalues of H. The eigen- values of (I + H & A) equal to (1 X X for j {1....,d} d} and i nI.
The proof can be derived from Theorem 2.3 of Schacke (2004). Given this, notice that as the number of layers the Transformer update eq. (3 increases, one eigenvalue (1 + +1AAAA44 will dominate the rest (except cases of ties).
Definition 4 (Dominating cigenvaluc(s)). At least one of the eigenvalues of (I (I+He & A) has a larger magnitude than all others, i.e., there exists j", 8 (which may be set of indices if there are lies) such that +1+ * > +1+ X41 for all j' E {l....,d} j and i' {I....,n) \i. These eigenvalues are called dominating.
Which eigenvalue dominates will control the smoothing behavior of the Transformer.
Jj=1 be the eigenvalues of A and H. Let the eigenvalues be sorted as follows, .... AA and +1+ < ... S |1+ X I. As the number A
Theorem 1. Given the Transformer updote in eq. 3), let {AA A and (}
of layers E - x., there are two lypes of dominaling eigenvalues: (1) (1 + X A4) and (2) (1 + (#*)
We leave the proof to the Appendix (where we describe all possible cases). We can now use this result to derive what X₁ converges to as depth increases.
3.3 The Features
Theorem 2. Given the Transformer updole in eq. ( if a single eigenvalue dominates, as the namber of total layers l + 00, the feature representation X, converges to one of two representations: (1) Tf (1 +114155 X XA) dominates ther!]
X + (1 x 14 X 8j, 1v
(4)
From Proposition have that, vAA

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
(2) If [1 (1 dominates then,
X - (1 + 21) 8, S..
(5)
where v v are eigenvalues of H.A and Sjá == (v vec(X)) and v jú is row ji in the matrix Q-1 ji (here Q is the matric of cigenvectors of (I+H@AA) (3) If multiple eigenvalues have the same dominating magnilude, X converges to the sum of the dominating lerms.
Corollary 1. If the residual connection removed in the Tronsformer update, then the eigenval- ues are of the form (2)) Further, (2f124) is always dominating eigenvalue, and X₃ + 1( EEjree (Af NV as - 00, where E the set of all eigenvalue indices equal to the dominaling eigenwalue A/.
See the Appendix for proofs of the above statements. Given these results, we can now understand when the oversmoothing definitions apply.
3.4 When Oversmoothing Happens
Theorem 3. Given the Transformer update eq- B as the number of total layers E - co, if (1) one eigencolue (1 + X/i X dominates, have input converrgence, angle convergence,, and rank collapse. If (2) one eigenvalue (1 + X X dominates, we do not have input convergence or angle convergence, but we do have rank collapse. If (3) multiple eigenvolues have the same dominating magnitude and: (a) there is at least one dominating eigenvalue (1+ xx- where XA + MA then we do not have input convergence angle convergence, if also (b) the geometric multiplicity of XA and XI are both greater than 1, then we also do not have ronk collupse.
Corollary 2. If the residual connection removed in the Transformer update, input convergence, angle convergence, and rank collapse are guoranteed.
The proofs are left to the Appendix. The above statements follow directly from Theorem 2 and Corollary [ They tell us that whenever a single eigenvalue (1 + X X dominates, every imput in Xx converges to the same feature vector. This happens because v = and so Xt,; ~ -vl', for all i as f + co. But there is a second case: whenever the single eigenvalue (1 + X X dominates, each feature is not guaranteed to be identical. However, X (1+ (
is still a matrix of rank one. If instead multiple eigenvalue dominate and the geometric multiplicity of x and A/ are both greater than 1 then X a sum of at least rank-1 matrices and so we do not have rank collapse.
Theorem 3 largely contradicts prior theoretical results on oversmoothing. We suspect few reasons for this. First, if multiple types of analyses used within paper, and they give conflicting results, resolving this can be especially challenging (Wang et al. 2022). Second, certain assumptions may not always hold in practice, e.g.. Noci et al. (2022) assume that A 11 at initialization.
On Layer Normalization & Feed Forward Layers. Most Transformers also include layer normalization and feedforward layers. Unfortunately, both of these break our analysis. For instance, repeated Pre-LN layer can be represented by the following update,
vec(X,) = (I HH-HDD1 A))vec(Xx) E(vec(A1l)) THITTD T D''H")),
where b and D-1 are terms introdee by the normalization layer. However, as far as are aware there is no way to characterize the relationship between the eigenvalues of (I+HD @ A) and the eigenvalues of H, A, and D, without introducing further assumptions (c.g., if is symmetric there is a known relationship). This difficulty also applies to Post-LN layers. We encounter a similar difficulty for feed forward layers,
vec(X,) (WT @ I W & A) vec(
7

---

Published in Transactions on Machine Learning Research (07/2025)
where W is the parameter of the feed forward layer. Similar to layer normalization, far are aware, we cannot characterize the eigenvalues of (WT @ I ++WlT H & A) in terms of the eigenvalues of H, A, and W, without further assumptions.
Takeaways: We find that oversmoothing is not inevitable. Whether layer smooths its input depends on the interaction between the spectrum of the attention A and weights H. The largest eigenvalue of (I+ H @ A) dictates smoothing behaviour: if (1+ X)) is largest, the inputs eventually converge to the same feature vector, but if (1 2 X is largest, they do not.
A natural question is can we use the above analysis to influence the smoothing behavior of Transformer models? In the next section we derive Corollary of Theorem that allows one to do so using a simple reparameterization of H.
4 Testing the Theory
How applicable are the thcoretical results developed in the previous section? Similar to recent theoretical
work on Transformer optimization (Ahn et al., 2023 Mahankali el al. 2023 Von Oswald et al., 2023 Ahn
et al. 2024 Zhang et al., 2024). the Transformer update we analyze in eq. (3) is simplified: no positional encoding, fixed attention and weights, single-head attention, How we understand the explanatory impact of our theory on full-scale Translormer models? To do so, we derive a simple reparameterization of the weights H that allows one influence smoothing behavior. We can then test this parameterization in existing Transformer architectures to see if smoothing can be affected, and also judge its impact on generalization.
layer normalization or ford-forward layers.
To derive this reparamcterization, first note that Theorem tells us that if (1 + X X dominates then this will cause oversmoothing, whereas if instead (1 + X X4) dominates we avoid it. To find A and H that are guaranteed to have either (1 A2 X11 or (1+ X4) dominate we could dig through the proof of Theorem I and consider all cases. However, as A changes for every batch of data X there is no casy way to guarantee the smoothing behavior of a model. Because of this, we need a solution that involves only controlling the eigenvalues of H. Luckily, we can simplify the proof Theorem [ into a much simpler condition.
Corollary 3. If the eigenvalues of H fall within [-1,0), then (1 + x4) dominates. If the eigenvalues of H. fall within 0o), then (1 X X4) dominates.
See the Appendix for a proof. To ensure that the eigenvalues of H fall in these ranges, we propose to directly parameterize its eigendecomppostion. Specifically, define H as A V where Vn is a full-rank matrix and Aи is diagonal. We learn parameters VH by taking gradients in the standard way (i.e., directly and through the inversion). To learn the diagonal of Aμ, i.c., diag(A#), parameterize the sharpening model as (r))(Ar)) == clip(1b, [],, where ap are tunable parameters and clip(tb, [1, u]) == min(max (1b, !), u) forces all of ab to lie in [4. u]. Similarly we parameterize the smoothing model as diag(AH) == clip( ab, [0, 1))
5 Experiments
We now run experiments using our reparameterization, in order to understand the predictive power of our results on full Transformer architectures. We train sharpening and smoothing models on CIFAR100 (Krizhevsky et al.] (009),, ImageNet (Deng et al., (200)), and The Pile (Gao et al. (020). We expect that
the attention layers of sharpening models reduce smoothing, i.., HFC/LFC (Definition I increases, cosine similarity (Definition 2 decreases, and effective rank (Definition 3 increases, while smoothing attention
layers increase smoothing.
We initialize H = V사AN to mimic the initializations used in the ViT-Ti and Bert baselines, which are initialized using He initialization (He et al.) 2015). Specifically, we first initialize Va
Initialization.
WWill could have allowed the smoothing model use the space of positive reals via diag(As) (el, we found that. restricting the space of allowed eigenvalues stahilized training.
8

---

Published in Transactions on Machine Learning Research (07/200) 20225)
Detettion ImageNet
Delestion
Vayeer
CIFAR100 Defitton Hrrrur
Delidtonn
keyorr
lagar
Dalattonn HEOARS
The Pile
G
Exyer
Tagae
Figure 3: Testing the theory. We take model architecture for each dataset and reparameterize H H =HHAHV H. We then train a sharpening model where diag(Ax) == [-1, and a smoothing model where diag(An)) == clip(1b, [o, 1])- Here zb are tunable parameters. Our theory suggests that the smoothing model should cause the metric of Definition [ (left column) to decrease, Definition 2 (center column) to increase, and Definition 3 (right column) decrease. These trends seem generally hold for CIFAR100 and The Pile, but not for ImageNet. We suspect. this has to do with impact of layer normalization, which we test in Figures and 6
CIFAR100
ImageNet
The Pile
VIT-Ti
VIT-TI
VIT-Ti
VIT-S
ViT-S
VIT-S
Cram. Bert
Cram. Bert
Cram. Bert
Attention
0.000
1.000
0.083
0.000
0.333
0.167
0.667
0.467
0.200
LayerNorm
0.500
0.542
0.125
0.833
0.708
0.542
0.750
0.750
0.156
ML.P
1.000
0.167
1.000
0.583
0.167
0.500
0.600
0.600
0.267
Table 1: Proportion of layers that increase the HFC/LFC (Definition for each layer type.
CIFAR100
ImageNet
The Pile
VIT-TI
VIT-TY
VIT-TI
ViT-S
ViT-S
VIT-S
Cram. Bert
Cram. Bert
Cram. Bert
Attention
1.000
0.000
1.000
1.000
0.667
0.833
0.333
0.533
0.800
LayerNorm
0.458
0.458
0.708
0.167
0.250
0.458
0.469
0.250
0.688
MLP
0.000
0.917
0.000
0.417
0.833
0.583
0.333
0.400
0.733
Table 2: Proportion of layers that increase the cosine similarity (Definition 2 for each layer type.
using He initialization. To initialize diag(A#) we sample from normal distribution with mcan 0, and variance 0.1 (we show the impact of different initialization variances on smoothing and training loss in Figures 8 and g). All other training and architecture details in the Appendix.

---

Published in Transactions on Machine Learning Research (07/2025)
CIFARI00
ImageNett
The Pile
ViT-Ti
ViT-Ti
ViT-Ti
VIT-S
ViT-S
ViT-S
Cram. Bert
Cram. Bert
Cram. Bert
Attention
0,250
1.000
0.167
0.083
0.833
0,500
0.733
0.600
0.267
LayerNorm
0.583
0.667
0.583
0.875
0.500
0.583
0.594
0.594
0.781
MLP
0.917
0.417
0.917
0.417
0.250
0.167
0.733
0.733
0.333
Table 3: Proportion of layers that increase the effective rank (Definition 3
for each layer type.
HFC/LFC,
cosine similarity,
2
effective rank, 3
ViT-S
ViT-S
ViT-S
ViT-S
ViT-S
ViT-S
ViT-S
ViT-S
ViT-S
Attention
0.000
1.000
0.250
1.000
0.000
0.750
0.083
0.500
0.250
LayerNorm
0.833
1.000
0.958
0.167
0.000
0.042
0.875
0.750
0.958
MLP
0.583
0.167
0.417
0.417
0.833
0.583
0.417
0.833
0.667
Table 4: No LN weights. Proportion of layers that increase HFC/LFC, cosine similarity, effective rank.
Reparameterization results. Figure g shows the effect of the two reparameterizations:: sharpening and smoothing. Since our analysis deals with the attention layer, we plot the variations in metrics for each part of the attention block to clearly visualize their individual impact. This is useful since those layers can have different, sometimes opposite impacts the metrics we are interested in.
We immediately notice that the effects of the reparameterizzatinns vary wildly depending the dataset and the smoothing metric. On one end of the spectrum is CIFAR100: where the sharpening model reduces smoothing and the smoothing model increases smoothing, for all metrics. The theory seems to predict the attention layer perfectly here as 100% the attention layers in the sharpening model increase the HFC/LFC (Table [ and the elfective rank (Table and 0% of the increase the cosine similarity (Table 2 We observe nearly the opposite, intended, behaviour for the smoothing model. On the other end of the spectrum is ImageNet, where the roparameterizations seem to have little effect on the HFC/LFC and cosine similarity smoothing metrics. Here only 33% of the attention layers of the sharpening model increase the HFC/LFC (Table D and majority of attention layers actually increase the cosine similarity (67%, Table 2 the opposite of its intended effect! The effective rank can be slightly manipulated: 83% of attention layers of the sharpening model increase the effective rank (Table 3). However, the final smoothing of the model is nearly the same as the original model as both LayerNorm and MLP layers of the sharpening model increase the effective rank less than the original model. In between these extremes The Pile, where the smoothing model seems to predictably reduce the HFC/LFC and effective rank, compared to the original model, dropping the percentage of attention layers that increase HFC/LFC down from 67% to 20% (Table D and dropping effective- effective-rank-inene rank-increasing layers from 73% down to 27% (Table 3). However, the smoothing model has less overall smoothing than the base model, as defined by cosine similarity (Figure 3
The impact of layer normalization. We suspect the reason we observe such varied effects is due to interac- tions between attention layers and other layers that we were unable to analyze. To test this theory, simple experiment: we create two 'repeating' models that obey eq- (2), i.e., weights and attention repeat instead of being layer-dependent. The first model reparam- eterized as above to be sharpening (i.e., WT H = -HAA such that diag(Ax) == clip( (-1,0)) and the second is reparameterized to be smoothing. This way, both models follow our theory exactly so, by Corol- lary sharpening and smoothing is guarantced. Fig- (left) shows the HFC/LFC (Definition [ of each repeating model, and predicted, the HFC/LFF of the
leat eccuracy improvament
+200
+3.27
run a
proj WT =
12
24
48
depth
Figure 5: Test accuracy improvements (CI- FAR100). Test results of reparameterized models for model depths (12, 24, 48). Results are averaged over 3 trials, numbers over bars re- port average accuracy, gray bars show standard deviations.
ure
10

---

Published in Transactions on Machine Learning Research (07 (025)
No norn
Post-IN jpasitive weightss
Post-LN (negative weights)
Pre-NN ipositive weightss
Pro-NN Inegstive weightsi
1.4
14
1.4
Delteition (awerage NFG/LPO)
smoothing
1.2
12
1.3
0-4-
0.--
08-
1.2
11-
1.8
0.4-
1.6
0.2
0.2
02
125
23
100
50
193
125
200
50
200
125
layer
layer
layer
layer
layer
weights)
Inegative weights
weights)
Inegative weights!
Delinston Imvergg HFCATG
sharpening
2.2
3.-
2.5
3.0
2.5
2.5
2.0
2.4-
3.5
1.4
0.5
3.4-
125
25
50
135
35
100
300
layer
layer
layer
layer
layer
Figure 4: Impact of Layer Normalization. The average HFC/LFC for the Transformer update with repeated layers, as in eq. 3 and different types of layer normalization (Post-LN (Vaswani et al. 2017).
Pre-LN Baevski & Auli 2018)) where the weights of the layer normalization are fixed to be positive or negative. See text. for details.
ImageNet Delindonn WFOLLFO
Uypar
loyar
laper
Figure 6: Smoothing, no LN weights. HFC/LFC (Definition 1 cosine similarity (Definition 2 and effective rank (Definition for reparameterized models with and without layer normalization weights.
smoothing model decreases (exponentially). approaching 0, while the HFC/LFC of the sharpening increases. We then introduce Post-LN (Vawwani et al. 2017) and Pre-LN (Baevski & Auli 2018) into the repeating architectures and plot. the HFC/LFC when layer normalization weights (7) are either positive or negative (both randomly sampled) | We observe that Post-LN has no impact model. Post-LN does afTect the sharpening model, creating a more erratic sharpening pattern, however the overall trend is preserved. On the other hand, Pre-LN (which is used in all of the full Transformer archi- tectures in our experiments) reverses the expected bchaviour when weights are negative. This could explain the surprising results we observe in Figure 3
the behaviour of the smoothing
To test this in full Transformer models, we train smoothing and sharpening models without layer normal- ization weights (i.e, we fix 1, 0) on ImageNet, the dataset most resistant to reparamcterization. The results are shown in Figure While the smoothing model behaves similarly to the model with layer normalization weights, the sharpening model without layer normalization weights reduces smoothing far beyond the model with layer normalization weights. This is particularly apparent for HFC/LFC, where the percentage of layers that increase HFC/LFC goes from 33% to 100% (Table 4 and for cosine similarity, where the percentage of layers increasing cosine similarity go from 67% to 0% (Table 4.
We omit the bias term isolate the impact of layer normalization weights.
11

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
10.0
test accuracy/F1 improvement
Crammed BERT (sharpening)
7.5
Crammed BERT (smoothing)
+7
5.0
2.5
+2
0.0
-2.5
-5.0
-7.5
10.0
CB
BoolQ
COPA
RTE
WIC
WSC
AX-g
MultiRC
task
Figure 7: Test accuracy/FF FF improvements (SuperGLUE). Test results of reparameterized models.
Overall, these results show some evidence that our theoretical analysis can explain the smoothing behaviour of full Transformer models. When it falls short, one possible explanation is the impact of layer normalization weights. When these weights are removed in full Transformer models on ImageNet, our theoretical results can better explain their smoothing bchaviour. This explanation is far from perfect: the smoothing models on ImageNet have nearly the same smoothing behaviour as the original model as measured by HFC/LFCC or cosine similarity, whether layer normalization weights are included the model or not. It would be very interesting to extend the analysis to explain this behaviour.
Test accuracy. Our goal in this section to test how well theory explain full Transformer architectures. While our intention is not to suggest using the reparameterized sharpening smoothing models in practice, we were curious to see what impact they had on test accuracy. Recall that prior work argued that the reason why test accuracy flattened as model depth increased was because oversmoothing, This implies that sharpening models should increase test accuracy beyond the baseline model, whereas smoothing models should reduce it. However, for ImageNet both sharpening and smoothing models have lower test accuracy than the original model: 2..6 (sharpening) and -0.6% (smoothing). This seems contradict the oversmoothing hypothesis. For CTFAR100 (results in Figure Sharppenng does improve test accuracy, but model depth is increased, smoothing improves it even more. This is the exact opposite of what is stated by the oversmoothing hypothesis. Finally, we evaluate the reparameterized Crammed BERT (Gipping & Goldstein 2023) models on SuperGlue (Wang et al. 2019a) (following the literature we report changes in test F1 for the CB and MultiRC tasks, and test accuracy for the rest). Both Crammed Bert (shapening) and Cramind Bert (smoothing) largely harm the performance of the original model. The only benefits come from the smoothing model.
Takeaways: To understand how much of our theoretical analysis applies to full Transformer models, we attempt to reduce increase smoothing using sharpening and smoothing models. For certain datasets, these reparameterized models act as predicted by the theory (i.c., CIFAR100). For others, they have little effect (iee,, ImageNet). We hypothesize that this is due to the impact of layer normalization, and we find that LN weights can reverse the intended effects of 'repeated' versions of the sharpening and smoothing models that follow eq. [ When we remove layer normalization weights in full Transformer models the sharpening model significantly reduces smoothing on ImageNet. We investigate the scnsitivity of the reparameterized models with respect to their hyperparameters (see Appendix Cp. Finally, we report the test accuracy of the reparameterized models, which defy the oversmoothing hypothesis for Transformers.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
6 Discussion
One limitation of the current thcoretical analysis is that the results are asymptotic, applying in the limit as 4-00 co. It would be useful to understand the rates of convergence. We would also like to expand the theoretical analysis to account for layer normalization and fully-connected layers. Special conditions will likely need to be placed on H to enable this analysis, such as symmetric A, (Sander et al. 2022). Another limitation is that we do not take into account language modcling aspects such as causal attention and positional encoding, which Barbero et al. (2024) relate to oversquashing, another phenomenon discussed in graph neural network literature. Considering how oversmoothing and oversquashing both lead to form of collapse, it would be interesting to unify these views.
References
Ahn, K., Cheng, X., Song, M., Yun, C., Sra, S., and Jadbabaie, A. Linear attention is (maybe) all you need (to understand transformer optimization). In The Twelfth International Conference on Learning Representations, 2023.
Ahn, K., Cheng, X., Daneshmand, H., and Sra, S. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Newral Information Processing Systems, 36, 2024.
Ali, A., Galanti, T., and Wolf, L. Centered self-attention layers. arXiv preprint arXiv:2306.0166 2023.
Baevski, A. and Auli, M. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2018
Bai, J., Yuan, L., Xia, S.-.., Yan, S., Li, 7., and Liu, W. Improving vision transformers by revisiting high-frequency components. In European Conference on Computer Vision, pp. 1-18. Springer, 2022.
Barbero, F., Banino, A., Kapturowski, S., Kumaran, D., Araújo, J. G. M., Vitvitskyi, A., Pascanu, R., and Veličković, P. Transformers need glasses! information over-squashing in language tasks, 2024. URL
https://arxiv. www.xii...r///ee. h/////b///// 2406. 042667
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877- 1901, 2020.
Choi, J., Wi, H., Kim, J., Shin, Y., Lee, K., Trask, N., and Park, N. Graph convolutions enrich the self-attention in transformers! arXiv preprint arXiv:2312. 04234, 2023.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gchrmann, S., et Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 21(240):1-113, 2023.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q, V. Randaugment: Practical automated data augmentation with a reduced search space, 2019.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 188-255. leee, 2009.
Dong, Y., Cordonnier, J.-B., and Loukas, A. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference Machine Learning, pp. 2793 2803. PMLR, 2021.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., Hc, H., Thite, A., Nabeshima, N., al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:200.00002 2020.
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Geiping, J. and Goldstein, T. Cramming: Training language model single gpu in day. In International Conference on Machine Learning, pp. 11117-11143. PMTR, 2023.
Gong, C., Wang, D., Li, M., Chandra, V., and Liu, Q. Vision transformers with patch diversification. arXiv preprint ar Xiv:21004. 12753, 2021.
Guo, X., Wang, Y., Du, T., and Wang, Y. Contranorm: contrastive learning perspective on oversmoothing and beyond. Xiv preprint arXiv:2303. 06562, 2023.
Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., and Wang. Y. Transformer in transformer. Advances in Neural Information Processing Systems, 34:15908-1559 15919, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.
Jiang, Z., Hou, Q. Yuan, L., Zhou, D., Jin, X., Wang, A., and Feng, Token labeling: Training 85.5%
top-1 accuracy vision transformer with 56m parameters imagenct. arXiv preprint arXiv:2104.10855 3 (6):7, 2021.
Kaddour, J., Harris, J., Mozes, M., Bradley, H., Raileanu, R.,, and McHardy, R. Challenges and applications of large language models, 2023a.
Kaddour, J., Key, O., Nawrot, P., Minervini, P., and Kusner, M. J. No train no gain: Revisiting efficient training algorithms for transformer-based language models, 2023b.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint (al:2001.08960) 2020.
Kenton, J. D. M.-W C. and Toutanova, L. K. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACI-HLT, pp. 4171-4186, 2019.
Kocsis, P., Súkeník, P., Brasó, G., Nießner, M., Leal-Taixé, L., and Elezi, 1. The unreasonable effectiveness of fully-connected layers for low-data regimes. Advances in Neural Information Processing Systems, 35: 1896-1900 2022.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. Albert: A lite bert for self- supervised learning of language representations. In International Conference on Learning Representations, 2019.
Li, Q. Han, Z., and Wu, X.M. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Liu, H., Dai, 7., So, D., and Le, Q. V. Pay attention to mlps. Advances in Neural Information Processing Systems, 34:9204 9215, 2021a.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.,, and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint orXiv:1907.11666 11692, 2019.
Liu, Z., Lin, Y., Cao, Y., HU, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012 10222, 2021b.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019.
Mahankali, A. V., Hashimoto, T., and Ma, T. One step of gradient descent is provably the optimal in- context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2023.
Meyer, C. D. and Stewart, T. Matrix analysis and applied linear algebra. STAM, 2023.
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Milanfar, P. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEFE Signal Processing Magazine, 30(1):106-128, 2013. doi: 10. N0.109/MSPP:001.21772
Noci, L., Anagnostidis, S., Biggio, L., Orvieto, A., Singh, S. P., and Lucchi, Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198- 35:27198-27211, 2022.
Park, N. and Kim, S. How do vision transformers work? In International Conference Learning Repre- sentations, 2022.
Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 34:12116-12128, 2021.
Roy, O. and Vetterli, M. The effective rank: A mcasure of effective dimensionality. In 2007 15th European signal processing conference, pp. 806-610. IEEE, 2007.
Sander, M. E., Ablin, P., Blondel, M., and Peyré, G. Sinkformers: Translormers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pp. 3515 3530. PMLR, 2022.
Schacke, K. On the kronecker product. Master's thesis, University of Waterloo, 2004.
Schwaller, P., Laino, T., Gaudin, T., Bolgar, P., Hunter, C. A., Bekas, C., and Lee, A. Molecular transformer: A model for uncertainty -eartaiitty-clibbrteddd chemical reaction prediction. ACS Central Science, 5(9): 1572-1583, aug 2019. doi: 10.1021 10.1021/acscentsoi.oo htw.sceetssi...0055150.. URTpstts: Utt/aattpp://oo..r///000.001000 http://....11////acccaeaaa. 9b00576
Shi, H., Gao, J., Xu, H., Liang, X., Li, Z., Kong, L., Lee, S., and Kwok, J. T. Revisiting over-smoothing in bert from the perspective of graph. arXiv preprint arXiv:2202.08622 2022.
Tang, Y., Han, K., Xu, C., Xiao, A., Deng, Y., Xu, C., and Wang, Y. Augmented shortcuts for vision transformers. Advances in Newral Information Processing Systems, 14115316153222 2021.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transformers & distillation through attention, 2021a.
Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and J6gou, H. Going deeper with image transformers. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 32 42, 2021b.
Touvron, H., Cord, M., and Jegou, H. Deit iii: Revenge of the vit. arXiv preprint or Xix:2224.00111 2022.
Touvron, H., Lavril, T., Izacard, G., Martinct, X., Lachaux, M.-.,, Lacroix, T., Rozière, B., Goyal, N.,
Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, Llama: Open and elficient foundation language models, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, 1. Attention is all you need. Advances in newral information processing systems, 30, 2017.
Von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A., and Vla- dymyrov, M. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pp. 35151 35174. PMLR, 2023.
Wang. A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019a.
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Superglue: A stickier benchmark for general-purpose language understanding systems, 2020.
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Wang, P., Zheng, W., Chen, T., and Wang, Z. Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice. In International Conference on Learning Represenlations, 2022.
Wang, Q. Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., and Chao, S. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguistics, pp. 1810-1822, 2019b.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., lchter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of thought prompting elicits reasoning large language models, 2023.
Wu, X., Ajorlou, A., Wang, Y,, Jegelka, S., and Jadbabaie, A. On the role of attention masks and layernorm in transformers. In The Thirly-eighth Annual Conference on Neural Information Processing Systems, 2024.
Xiong, R.,, Yang, Y., Hc, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524 10533. PMLR, 2020.
Yan, H., Gui, L., Li, W., and He, Y. Addressing token uniformity in transformers via singular value transformation. In Uncertainty in artificial intelligence, pp. 2181-2191. PMLR, 2022.
Yu, P., Artetxe, M., Ott, M., Shleifer, S., Gong, H., Stoyanov, V., and Li, X. Efficient language modeling with sparse all-mlp. Xixiv preprint or XrXiv22203.066 06850, 2022a.
Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., and Yan, S. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVT conference on computer vision and pallern recognilion, pp. 10819 10829, 2022b.
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z.-H., Tay, F. E., Feng, J., and Yan, S. Tokens- to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 558 567, 2021.
Yun, S., Han, D., Oh, S. J., Chun, S., Choc, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features, 2019.
Zhai, S., Likhomanenko, T., Littwin, E., Busbridge, D., Ramapuram, J., Zhang, Y., Gu, J., and Susskind,
J. M. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pp. 40770-400 4000004-03.... PMLR, 2023.
Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization, 2018.
Zhang, R., Frei, S., and Bartlett, P. L. Trained transformers learn linear models in-context. Journal of Machine Learning Research, 25(49):1-56 2024.
Zhou, D., Kang, B., Jin, X., Vang, L.,, Lian, X., Jiang, Z.,, Hou, Q., and Feng, J. Deepvit: Towards deeper vision transformer. arXiv preprint arxiv: 11 11886, 2021a.
Zhou, D., Shi, Y., Kang, B., Yu, W., Jiang, Z., Li, Y., Jin, x., Hou, Q and Feng, J. Refiner: Refining self-attention for vision transformers. arXiv preprint arXiv:2106 03714, 2021b.
16

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Appendix
A Proofs
Proposition 1 (Meyer & Stewart (2023) ). Given Assumption 1 all eigenvolues of A lie within (-1, 1].
There is one largest eigenvalue that is equal to 1, with corresponding unique eigenvector 1.
Proof. First, because is positive, by the Berron-Frobeiie Theorem Meyer & Stewart 2023 all eigenvalues of A are in R (and so there exist associaled eigenvectors that are also in RR. Next, recall the definition of an eigenvalue and eigenvector Av Av. Let us write the equation for any row i {l,..., n) explicitly:
0,1 + ... + ain = Aui-
Further let,
Umax max[[e]])....... /unl1
(6)
Note that Umax > O, otherwise it is not valid eigenvector. Further let kimax be the index of v corresponding to max Then we have,
A A/emax
0kkmax
+ (e Ura
< akmax [U1] + ... + aRmmn n enl
< akmax U + + akmaan Ukmma
(0kmase 1 +
Olemssn) Ukmase |max
The first inequality is given by the triangle inequality and because aij > 0. The second is given by the definition Umax as the maximal element in v. The final incquality given by the definition of A in eq. (I) as right stochastic (i.e., all rows of A sum to 1) and because |8kmaa| = |mmax] Next, note that because Finally, to show that the one largest eigenvalue is equal 1, recall by
Vmax 0, it must be that the definition of A in eq. 11 that A1 1, where 1 is the vector of all ones. So is an eigenvector of A, with eigenvalue X = 1. Because az 0, and we showed above that all eigenvalues must lie in in [-1, 1], by the Freron-Frobenius theorem Meyer & Stewart (2023) X" ===1 is the Perron root. This means that all other eigenvalues A; satisfy the following inequality [x < X Further 1 is the Perron eigenvector, and all other eigenvectors have at least one negative component, making 1 unique. Finally, because A is diagonalizable it has n linearly independent eigenvectors.
We now prove lemma that will allow us to prove Theorem
Lemma 2. Consider the Trunsformer updale in eq. 3... Let vA and ( v Jj=1 be the eigenvalue and eigenvectors of A and H. Let the eigenvalues (and associated eigenvectors) be sorted as follows, X1 S S XA and [1 + +f1 .... [1 X Let 1lll .....9"! be the phases of .f. As the namber of layers L + oo, one eigenvalue dominales the rest (maltiple dominate if there are ties):
(1+ XII A
i1 X X 11) | X <1
if x
(1+ Hy A4)
(1 XI
#||+> > > + A
/f <
(1 (1+))))
<
(1+)
>
// < 0 E (
f|1+> (
< <
where Ai is the eigenvalue of with smallest magnitude and XII the eigenvalue with the largest inder Io such that off E (m/2, U [- 2/2).
17

---

Published in Transactions on Machine Learning Research (07/2025)
Proof. Given Lemma 1 the cigenvalues and cigenvectors of AA) are equal to (1+ X and v &v, for all j E (1,.... d) and E {l....,n}, nn. Recall that eigenvalues (and associated eigenvectors) are sorted in the following order X v .... I Xa and |++ | - ... I |1+ X Our goal is to understand the identity of the dominating cigenvalue(s) 25t X for all possible values of As
First recall that X E and AA = 1. A useful way to view selecting X to maximize +1+ X X is as maximizing distance to -1. If (i), X 0 then XA, for all i (1,...,m - 1} always shrinks A to the origin and X4 leaves it unchanged. Because of how the eigenvalues are ordered we must have that [1 + X/1 [1 + 23 X S [1+ X A = |1+ >. If [1 X > 1 then shrinking any X!" to the origin will also move closer to -1. However, if [1 + Xf 1 <
then shrinking to the origin can move XH farther from -1 than [1 + X/ | The eigenvalue of H that can be moved farthest is the one with the smallest overall magnitude, defined as Amin. The eigenvalue of A that can shrink it the most is A/. This completes the first two cases.
If instead (ii), 21 0 then it is possible to 'flip' X across the origin, and the maximizer depends on vp. If (a) Pd E [-m/2, x/2 then lct X be the eigenvalue with the largest index i such that vp e ///2, -,, 7/2). It is possible that 'llipping' this eigenvalue across the origin makes it farther away than Xi ie., 1
|1+ In this case (1+ H x dominates, otherwise (1+ X4) dominates. If they are equal then both dominate. If insteedd (b) 42 E (x/2, r///,,]J[[T,, -r/2) then either |1 + +2 XA1 > 11+ for all j' * d and i' # n, and so (1+ + X M/) dominates, or "fipping' X increases its distance from -1, and 11 + A A > |1+ X X for all j' # d and i' * n, and instead (1 + x X dominates. Because we cannot
have that |1 + N A = |1 + A A as X > -1 this covers all cases.
Now we can prove Theorem
Theorem 1. Given the Transformer update in eq. 33 let { and {1/) Jj=1 be the eigenvalues of A and H. Let the eigenvalues be sorted as follows, X1 ..... S XA and |1+ <... S 11+ I. As the number of layers E 8 00, there are two types of dominating eigenvalues: (1) (1+ A(AA)- and (2) (1 A xFA
The proof follows immediately from Lemma 0
Theorem 2. Given the Transformer update in eq. 3 if a single eigervaauue dominates, as the number of total layers l + oo, the feature representation X converges to one of two representations: (1) If (1 X5 X dominates then,
X. + X 1y
(7)
(2) If (1 dominates then,
X - (1 + H X1)2 V
(8)
where vl, A are eigenvalues of H,A and 8j (v vec(X)) and v
já is row ji in the matrix Q-1 (here Q is the matric of eigenvectors of (I + H A)). (9) If multiple cigenvalues have the same dominating magnitude, X₃ converges to the sum of the dominating terms.
3.
Proof. Recall that the eigenvalues and eigenvectors of (I H 8 A) are equal to (1 + 24 X²) and vf @ v for all j E (1....d) and i E nI. This means,
vec(X.)
Ea+ 3o vec())) v!!).
i.j
Recall that v²-1 is row ji in the matrix Q where Q is the matrix of eigenvectors v X v Further rocall that v = As described in Thoorem as + oo at least one of the eigenvalues pairs Xf X will dominate the expression (1 + x which causes vec(X,) to converge to the dominating term. Finally, we can rewrite, Vi & V2 as vec(v2v, ). Now all non-scalar terms have vec(-) applied, so we can remove this function everywhere to give the matrix form given in eq. and eq.
18

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Corollary 1. If the residual connection is removed in the Transformer update, then the eigenval- ues are of the form (2 Further, (2f) is always dominating eigenvalue, and X; 1 (Ej (2 v v ") l + 00, where e is the set of all eigenvalue indices equal to the dominating eigenvalue Ajt.
Proof. The cigendecomposition of the Transformer update without the residual connection is:
vec ve.( = E(f ja vec vec(vv))())))o ())(v) @ v
In this case, (A is always a dominating eigenvalue because | > |A for any i (I,..., n 13. This observation, combined with the above eigendecompositionn produces X -> 1 (Erreet sy S/
as + 00.
Theorem 3. Given the Transformer update eq. 3 as the number of total layers é + co, if (1) one eigenvalue (1 + 2 X dominates, we have input convergence, angle convergence, and rank collapse. If (2) one eigenvakue (1 + X X²) dominates, we do not have input convergence or angle convergence, but we do have rank collapse. If (3) multiple eigenvolues have the same dominating magnitude and: (a) there is at least one dominaling eigenwalue (1 + 11+x xxxx where X f Xa; then we do not have inpul convergence or angle convergence, or (b) the geometric multiplicity of X and X are both greater than 1, then we also do not have rank collapse.
Proof. If (1) one eigenvalue (1+1/ X4) dominates then we have that X, - (1+A/ ( 1v Therefore, X5 has all the same inputs which implies input convergence, angle convergence, and rank collapse. If (2) one cigenvalue (1 + X A dominates then we have that X (1+ A/A / Therefore, we do not have input convergence. Further as v can contain both positive an negative components we do not have angle convergence. However, X₃ is rank one so we do have rank collapse. If (3) multiple eigenvalues have the same dominating magnitude and: (a) there at least one dominating eigenvalue (1 + X3t x where A * A then we do not have input convergence or angle convergence, as shown for case (2); if (b) the geometric multiplicity of X and AT are both greater than 1, then X; converges to the sum of at least 2 rank-1 1 matrices which are not. themsel ves linear combinations of each other. Therefore, rank(X,) 2 2.
Corollary 2.
If the residual connection removed in the Transformer update, inpul convergence, angle convergence, and rank collapse are guoranteed.
Proof. Corollary tells us that in this case X, - ((Ereectt (2 s3 as l + co. This matrix is rank-1 and so we have input convergence, angle convergence, and rank collapse.
aux
Corollary 3. If the eigenvalues of H fall within [-1,0), then (1 + Af dominates. If the eigenvalues of H fall within (0, 00), then (1 + X X4) dominates.
Proof. Let X S ... S A Again we can think of selecting A X that maximizes |1 >!' as maximizing the distance of X X4 to -1. Consider the first case where A ,... "."'" E (-1, and so X is the closest eigenvalue to 1 and X is the farthest. If X shrinks H the most. The closest eigenvalue to the origin is AH and (1 + X A41) dominates. If instead X < 0, then we can 'flip' X5 over the origin, making it farther from -1 than all other XB. The eigenvalue that we can 'flip' the farthest from -1 is Xf1 and so (1 + X X dominates. If all eigenvalues of H are equal, then both (1 A X41 and (1 + X x dominate. For the second case where AfI .II E (0, have that [1 + X AĄ > |1 + X for all j' E {l....,d d 1} and i' E {1....., 1}. This is because, by definition 1 > A4A4. Further, + H XA 2 |1 +1 as the largest |1 + XB A can be either (i) 11- 1 for 0 < e < 1 or (ii) |1 1 (i.e. in (i) X is negated by X1 and in (ii) A is the next
then all A can do is shrink X to the origin, where X
largest value of )²). For (i), it must be that 1+ 1+ AA 2 |I 11-ed - as No > 0. For (ii) X 2 x 1 > 0, and
1 + 2 11 + X X4I. Therefore AA dominates.
19

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Dervitas catte seneeemiiing
Driteiton (prengg HOOLPO
CIFAR100 clopping
Detpttoes ooweage coste antomy) 1.
Dettedion Qaarrgg (FOLFF)
infiializtion
layer
layor
layer
Figure 8: Scnsitivity of smoothing metrics. HFC/LFC (Definition cosine similarity (Definition 2 and elfective rank (Definition 3 for reparameterized models with different clipping thresholds and initializa- tion variances for di)
B Training & Architecture Details
Crucially, even though our theoretical analysis applies for fixed attention A and weights H, we use exist- ing model architecturos throughout, i.c., including different attention sreenttion/weihhtss /wwights cach layer, multi-head attention, layer normalization (arranged in the pre-LN format Xiong et al. (2020)), and fully-connected layers
Image Classification: Training & Architecture Details. We base our image classification experiments on the ViT model Dosovitskiy et al. (2021) and training recipe introduced in Touvron et al. (021a)) On
CIFAR100 for 300 epochs using the cross-entropy loss and the Adamm optimizer Loshchilov & Hutter (2019)
Our setup is the one used in Park & Kim (2022) which itself follows the DeiT training recipe Touvron al.)
(2021a). We use cosine annealing schedule with initial learning rate of 1.25 10-1 and weight decaa of 5 x 10-2. We use a batch size of 96. We use data augmentation including RandAugment Cubuk et al. (2019), CutMix Yun et al. (2019), Mixup Zhang et al.) (2018), and label smoothing Touvron el. al. (2021a). The models were trained on two Nvidia RTX 2080 Ti GPUs. On ImageNet, we use the original DeiT code and training recipe described above. Changes from CIFAR100 are that we use a batch size of 512 and train on a single Nvidia RTX 4090 GPU.
Text Generation: Training & Architecture Details. We base our NLP experiments on Geiping Goldstein (2023), , using their code-base. Following this work we pre-train encoder-only 'Crammed' Bert models with maximum budget of 24 hours. We use masked language modeling objective and train the Pile dataset Gao et al. (2020). The batch size is 8192 and the sequence length is 128. We evaluate models on SuperGLUE Wang et al. (2020) after fine-tuning for each task. In order to ensure a fair comparison, all models are trained on a reference system with an RTX 4090 GPU. We use mixed precision training with bfloat16 as we found it to be the most stable Kaddour et al. 2023b).
C Hyperparameter Sensitivity
Another aspoct that may have an effect on the smoothing behaviour reparamoterized modols is the choice of hyperpararneters. Specifically, there are two hyperparameters we set for ())) (a) the clipping interval (defaults: clip(y, [-1, 0)) and clip(yb. [0, 1))), and (b) the initialization variance (default: 0.1). To test (a), we halve the size of each clipping interval. For (b), we try initialization variances within the set (0.01, 0.1,
4TF model has multiple heads will define Wy Vu and Wo V
20

---

Published in Transactions on Machine Learning Research (07/2025)
clipping
Initialization
Ragiuai dpiel-1
-000 4 FHEAA))-200000
wassing hea
20
100
200
Beration
Steationa
Itarationa
Figure 9: Scnsitivity of training loss. Training loss curves for reparameterized models with different clipping thresholds and initialization variances for diag(Aæ))
We test all changes on CIFAR100, as this dataset seems most sensitive reparameterization and so should give us sense of an upper bound on hyperparameter sensitivity. The results shown in Figure S We notice that clipping interval seems to have small effect the smoothing behaviour of reparameterized models: the largest change is the final HFC/LFC of the sharpening model, which increases by roughly one when the clipping interval is halved.
The initialization variance has a much larger effect on the smoothing model: increasing the variance induces stronger smoothing across all metrics, for all layers beyond the second layer. Curiously has little effect on the final smoothing of the sharpening model, even when increasing the variance reduces sharpening for intermediate layers. Finally, we were curious if hyperparameter choices introduced any training instability which could explain these results. Figure shows that this is not the case: under any hyperparameter choice tested, the training curves are stable.
D Distribution of the eigenvalues of H in trained models
Immaginary
Laye
S
1.5
Laye
Laper
Laye
Freah
Laper
Lapara
Sayer
Lapera
Layer
Layer'
Layel
immaginary
PN
25
20
real
Figure 10: Distributions of eigenvalues of H (Top) Vision models have distributions skewing to the negatives; (Boltom) Language models have symmetrically distributed eigenvalues.
21


Reviews:
[{"review_id": "dwgVjhKg7C", "reviewer": "Reviewer_MUFK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**  \n1. The eigenspectrum analysis is novel and unifies prior (sometimes conflicting) results. It identifies exactly when input convergence, angle collapse, and rank collapse are necessary versus merely possible.  \n2. By directly constraining the eigenvalues of the residual‐weight matrix, practitioners gain a simple knob for controlling smoothing behavior. The empirical results confirm that this knob indeed moves the smoothing metrics.\n\n**Weaknesses**  \n1. The theoretical model omits layer normalization, feedforward layers, multi‑head attention, positional encodings, and (for language models) causal masking. While the authors acknowledge these gaps, it remains unclear how tightly the simplified analysis predicts behavior in actual Transformer blocks.  \n2. The asymptotic nature of the results ($\\ell \\to \\infty$) may not map cleanly onto finite‐depth networks (e.g., 12–48 layers).\n3. Constraining eigenvalues via clipping may introduce gradient instability or hamper learning dynamics; the paper does not thoroughly analyze training stability (beyond noting that large initial $\\(\\Lambda_H\\)$ can blow up).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper revisits the phenomenon of oversmoothing in deep Transformer models, challenging the prevailing view that increasing depth inevitably causes features to converge (in input space), angles to collapse, or rank to drop to one. The authors:\n1. Empirically show that several pre‑trained vision and language Transformers do not monotonically oversmooth according to the three common metrics (input convergence, angle convergence, rank collapse).\n2. Develop a theoretical analysis of a simplified Transformer update (fixed attention, single head, residual connection) via eigenspectrum arguments. They prove that whether a model oversmooths depends on the dominant eigenvalue of the Kronecker‑sum matrix, yielding precise conditions under which each of the three collapse phenomena do—or do not—occur.\n3. Derive a simple reparameterization of the residual‐weight matrix (via a clipped diagonal eigenvalue spectrum) that provably biases the model toward sharpening or smoothing behavior.\n4. Validate this reparameterization on vision (ViT variants) and language (“crammed” BERT) architectures, showing that one can indeed shift smoothing metrics—and sometimes improve downstream accuracy in small‐scale settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes:**\n\nThe paper would benefit from systematic ablations to isolate the effects of its key hyperparameters. For instance, varying the initial scale of the diagonal entries of $\\(\\Lambda_H\\)$ could reveal whether the observed smoothing and sharpening trends are robust to the initialization variance. Likewise, experimenting with different clipping intervals (e.g.$[-0.5,0]$ versus $[-1,0]$ may uncover the sensitivity of model behavior to the allowed eigenvalue range. Finally, tracking the impact of these choices on the training‐loss convergence rate would clarify whether constraining the spectrum hinders or accelerates optimization.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ew2SIZQdpA", "reviewer": "Reviewer_1HFL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- theoretical results on getting the dependence of collapse based on the eigenspectrum of values and projection matrix with less assumptions as prior works\n- new insights when oversmoothing happens, including cases when oversmoothing can be avoided in the models.\n- attempt to investigate / prove transfer of theory to more general empirical settings via reparametrization and mechanism of enforcing oversmoothing or sharpening\n \n**Weaknesses**\n- Empirical results for the baseline models are not showing strong patterns of oversmoothing. I agree that different models and data give different trends, but all trends are not really drastic to say that models do or not do strong oversmoothing.\n- Empirical results with controlling smoothness and sharpness show some changes but again it seems that it is not showing general correlation between eigenspectrum and empirical behaviour.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Prior works observed that oversmoothing effect happens in transformer models when depth is increased, meaning that representations collapse: either to the same vector, to the same angle or to representations of rank 1. This gives unclear picture why transformers perform so well in practice despite this phenomenon. The main downside is a lot of assumptions and simplifications to perform this analysis in prior works which creates a huge gap between theory and practice. Authors e.g. show empirical analysis of oversmoothing in 3 different models / data and show inconsistency with theory for them.\n\nThe paper is trying to reduce this gap and perform more general analysis.\n- First, authors still consider simplification of the transformer, but only removing the layer normalization and mlp (e.g., there is still residual connection + there are no assumptions on the attention matrix). With that authors show that the important piece for theory and analysis is the eigenspectrum of value and projection matrices, and depending on the spectrum and what eigenvalue dominates we will end with different oversmoothing regimes, but also may not end up e.g. with input or angle collapse, only with rank collapse, or even no collapse at all if there are multiple eigenvalues with the same dominating amplitude. These results are new and contradict prior work theory: new insights extend prior work findings for more general cases, reducing the gap between theory and practice.\n- Second, authors discuss that layer norms and mlp could not give any theoretical results unless we make some assumptions. However we could probe the finding if spectrum is the cause of oversmoothing behaviour (or there is correlation) in practice by reparametrization of the value and projection matrices. I found this to be a smart and cool approach: authors inherit conditions of the spectrum which gives us sharpening or smoothing and now with reparametrization we can enforce either regime in the model to see what happens in practice with a real transformer. With that authors observe different behaviours and provide some analysis for the models. This is a way to get correlation between theoretical results (when simplifications are made) and empirical results.\n\nAlso, an interesting result for me was the corollary that without residuals we are always going to collapse, pointing out again why residuals are so important in practice.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper is really well written with good math notation and clear derivations for all theorems, including Appendix. I did not find any issues so far with derivations and claims in theory. I also like figure 1 a lot, this is a good summary. And the overall text is written as a well organized story, which will be simple to follow and understand for people outside theory or who want to get the insight w/o reading theoretical parts and proofs.\n\nWhile I enjoyed the math and overall theoretical claims, I was expecting a bit more on the empirical side. Empirical results in Fig 3 did not convince me that we have a clear correlation between what is stated in theory and what is observed in practice. With that I believe, either more discussion of the results is needed, or maybe authors could reformulate a bit what they are trying to tel exactly by Sec 5. I think I quite did not get the outcomes as mainly all results for the baselines are quite opposite behaviours. I feel at the end of the paper I lost the thread of the results and consequences.\n\nMinor typos / small suggestions to improve a bit the text:\n- \"while rank collapse is likely, it is also not required.\" -> \"while rank collapse is likely, it is also not necessarily happening.\" (required is a bit weird to me from language reading).\n- notation on 1 vector will be good to have, as I interpreted it first as a matrix with all ones, not the vector.\n- figures 2, 3 - make the legend bigger, it is unreadable right now.\n- eq (4) may be good to include \"... vnA vjH  = ... 1 vjH (by using proposition 1)\" - so explicitly point why there is \"1\" and not the \"v\" vector.\n- initialization page 9: will your initialization give the final H to be the same initialization as for the baselines? does this restrict that you consider only particular initialization?", "broader_impact_concerns": "No any concerns, it is theoretical work on understanding how transformers work in general, which may impact the society later only.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ShSruKB14M", "reviewer": "Reviewer_eQhT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The topic of Transformer oversmoothing is important and merits further study.\n- The paper provides a comprehensive summary of related work, particularly on the methods used to measure oversmoothing.\n- The use of eigenspectrum analysis and the proposed reparameterization approach seem to be novel.\n\nWeaknesses:\n- The paper's organization and clarity could be improved. While I am not an expert in this field, certain sections are difficult to follow, which may hinder broader accessibility.\n- This paper excludes layer normalization and feedforward layers, as these components complicate the analysis.\n- The experimental results are not clearly explained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides both theoretical analyses and empirical experiments on the oversmoothing phenomenon in Transformer architectures. The authors summarize three metrics from related work to measure oversmoothing: input convergence, angle convergence, and rank collapse. They investigate smoothing behavior by analyzing the eigenspectrum of attention and weight matrices. However, the study excludes layer normalization and feedforward layers, as these components complicate the analysis. Instead, the authors propose a reparameterization of the weight matrices and explore how it influences smoothing behavior. While the results could be more clearly presented, the authors appear to argue that oversmoothing is not inevitable and can be controlled through this reparameterization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical to securing my recommendation for acceptance:\n- Ensure proper usage of reference citation styles, distinguishing between `(Author, Year)` and `Author (Year)`. \n- The fonts in Figures 2 and 3 are too small, making them difficult to read. Additionally, these figures need more detailed explanations to clarify how the conclusions are drawn.\n\nSimply strengthen the work in my view:\n- Improve the overall organization and writing of the paper to enhance clarity and readability.\n- The purpose of Section 5, which introduces the reparameterization, is unclear. It seems the conclusions vary across different datasets, but this needs to be explained more explicitly.\n- Clearly distinguish between the novel contributions of this paper and the ideas summarized from existing work to better highlight the originality of the study.", "broader_impact_concerns": "No broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)