Critical attention scaling in long-context transformers
Authors: Shi Chen
Zhengjiang Lin
Yury Polyanskiy
Philippe Rigollet
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
attention scaling
long-context length scaling
rank-collapse
phase transition
YaRN
Qwen
Primary area: foundation or frontier models, including LLMs

Abstract: As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length $n$ increases, causing tokens to cluster excessively, a phenomenon known as rank-collapse. While $\text{\emph{attention scaling}}$ effectively addresses this deficiency by rescaling attention scores with a polylogarithmic factor $\beta_n$, theoretical justification for this approach remains lacking.

We analyze a simplified yet tractable model that magnifies the effect of attention scaling. In this model, attention exhibits a phase transition governed by the scaling factor $\beta_n$: insufficient scaling collapses all tokens to a single direction, while excessive scaling reduces attention to identity, thereby eliminating meaningful interactions between tokens.
Our main result identifies the critical scaling $\beta_n \asymp \log n$ and provides a rigorous justification for attention scaling in YaRN and Qwen, clarifying why logarithmic scaling maintains sparse, content-adaptive attention at large context lengths.

Published as a conference paper at ICLR 2026
CRITICAL ATTENTION SCALING IN LONG-CONTEXT TRANSFORMERS
Shi Chen', Zhengjiang Lin', Yury Polyanskiy?, Philippe Rigollet' DDepartment of Mathematics, MIT
2 Department of Electrical Engineering and Computer Science, MIT {linzj, schen636, yp) edu rigollet @math.mahmm.mmmm. wwt.mmiteed..
ABSTRACT
As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length n increases, causing tokens to cluster excessively, phenomenon known as ronk-collapse. While attention scaling effectively addresses this deficiency by rescaling attention scores with polylogarithmic factor 3n, theoretical justifica- tion for this approach remains lacking.
We analyze simplified yet tractable model that magnifies the effect of atten- tion scaling. In this model, attention exhibits a phase transition governed by the scaling factor 8n insufficient scaling collapses all tokens to single direction, while excessive scaling reduces attention to identity, thereby eliminating mean- ingful interactions between tokens. Our main result identifies the critical scaling fn, = log n and provides a rigorous justification for attention scaling in YaRN and Qwen, clarifying why logarithmic scaling maintains sparse, content-adaptive attention at large context lengths.
INTRODUCTION
The attention mechanism cornerstone of modern transformer architectures on which Large Lan- guage Models (LLMs) rely. Mathematically, an attention layer is a nonlinear operator ATT that maps a collection of tokens (a1,...,,,,, .n, from Rd to Rdd This operator parametrized by three (possibly sparse) d by d matrices K,Q, and V and maps {x1,.... following formula. Define the normalization operator N(x) x/||==|| and for any = I,...,n define q; QN (x:), ly (N(x;). Then xf ATT(II,..... inn) ; is defined as
to (r',..., ,,,,, using the
cis
1 Σ A(z))a,,
Ag
k== Cac
where the terms agj = q. kj are called attention scores.
A recent line of theoretical work has demonstrated that attention acts as a contractive operator that tends cluster tokens together; see Dong et al. (202T); Geshkovski et al. (2024b) 2025); Karagodin
et al. (2024); Geshkovski et al. (2024a) Bruno et al. (2025a), Pol Polyanskiy et al. (2025); Chen et al.
(2025a); Cowsik et al. (2024); Giorlandino & Goldt (025); (025). This clustering effect also known as "rank-collapse" or "token uniformity" and arises because the distribution of attention
scores tends flatten as the sequence length n grows, causing each token to disperse its attention across too many other tokens rather than focusing selectively.
Various practical solutions have been proposed to curb this clustering behavior. In this work, we focus simple context-length-awe practically implemented as YaRN (Peng et al. 2023), Qwen (Bai et al., (023), SSMax (Nakanishi) 2025). and SWAN-GPT Puvvvada et al. 2025). These methods employ straightforward strategy that rescales attention scores a. by a single poly- logarithic factor 3,; see Table []] Our goal in this paper is to answer the following fundamental question:
modifications of the attention mechanism following ideas

---

Published as a conference paper at ICLR 2026
What is the optimal order of magnitude of the Bn scaling?
To address this question, we propose highly simplified yet completely tractable model for attention. This model exhibits phase transition governed by the parameter 3n as n + oo: when 3n is below critical threshold, atten- tion becomes overly contractive and collapses all tokens to a single direction, while when 3n is too large, atten- tion acts as an identity operator and fails to process in- formation effectively. More precisely, we establish that the critical parameter (n, scales logn, which corrob- orates the empirical guidelines underlying YaRN, Qwen, SSMax, and SWAN-GPT.
Method
Bn scaling
YaRN
(logn)²
Qwen
log n
SSMax
log n
SWAN-GPT
log
Table 1: Attention scaling factors for various methods. The standard atten- tion score exp(k; qz) is replaced with exp(CBnkk b) C,
Our work is intimately connected to the recent contribu- tions of Giorlandino & Goldt (025) Cnwsik et al. al. (2024), who investigate the contractive effects of attention mechanisms with random key and query matrices K and Q to establish proper initializa- tion schemes for these parameters. A crucial insight from Cowsik et al. (2024) that analyzing the evolution of symmetric token configurations provides a more mathematically tractable framework compared to the generic input distributions considered in Geshkovski et al. (2025). This symmet-
ric setting. while simplified, captures essential dynamics of the attention mechanism and enables rigorous thcoretical analysis; see also Karagodin et al. (2025).
The choice 3, 7 log appears natural in retrospect. As noted in Nakanishi (0025) with such become
scaling the attention weights A.j in Equation [
n "0
Au
km 7
To illustrate the resulting dynamics, consider simplified regime where all attention scores dij are of order one: specifically, let ag 1 and azj 0 for i + j. In this setting, the off- off-diagonal weights satisfy
n
1/1
if if I-p
A.j n² + (n 1)m7
1//00-0)
This analysis reveals two distinct regimes. When y is small (subcritical regime), attention weights are asymptotically uniform, resulting diffuse attention that, as we demonstrate below, leads to severe token contraction. Conversely, when is large (supercritical regime), off- dia-oiagonal weights become negligible with respect to the diagonal ones so that the attention mechanism is effectively suppressed.
The critical regime emerges at the phase boundary 7 1_ where attention can concentrate on sublinear yet nontrivial number of tokens so as to maintain sufficient connections to facilitate in- formation flow from small set of important tokens. This sparse attention is related to structured attention mechanisms employed in long-context architectures such as Longformer (Beltagy et al.) 2020) and SWIN (Liu et al. 2021) which implement sliding window over k n-nearest neigh- bors but where proximity is measured in terms of token position rather than embedding. Unlike these structurally constrained approaches that rely on fixed positional neighborhoods, the logarithmic scal- ing enables the attention pattern to be entirely content-adaptive, allowing each token to dynamically select its most relevant context based semantic similarity rather than positional proximity.
I-p
Following similar motivations, Giorlandino & Goldi (2025) establish a compelling analogy between attention dynamics and the random energy model from statistical physics (Derridal 198I). Using the replica method-ane an analytical heuristic from statistical pyssics-theyy identify a phase transition occurring at f,n Vlog n, which differs from the scalings presented in Table This result repre- sents significant discrepancy from our findings and highlights fundamental differences in modeling assumptions. More specifically, their approach assumes that the attention scores aiy are correlated Gaussian random variables. This assumption effectively induces random geometry on the token space, where similarity between tokens is treated as fundamentally random. In this sense, their model bears closer resemblance to recent Kuramoto models random graphs studied in Abdalla ctt al. (2022); et al, (2025), where the authors investigate the synchronization oscillators inter- acting across the edges of (sparse) Erdös-Rénn random graph with unit edge weights. However,
andivations,Giorraadinoo
2

---

Published as a conference paper at ICLR 2026
in the case of Giorlandino & Goldt (2025) the random graph is both directed and dense, with the edge pointing from token j to token having weight given by
Ag
kk=1 e3.a
where dzy are Gaussian random variables. While Giorlandino & Goldi (2025) assumes a specific correlation structure between the Gaussian random variables, the phase transition they uncover is expected to be universal within large class of random matrices including Wigner ones. Crucially though, in such models, the interaction strength Aij is independent of the positional relationship between tokens and j. making this model qualitatively different from standard attention mecha- nisms where attention is focused on few (or all) of the preceding tokens. For completeness, we refer readers to Appendix for a derivation of the critical scaling 8n x Vlog n in the i.i.d. Gaussian score model.
Bruno et al. (2025b) adopt a different approach to studying the regime where n + oo and 3, - 0o, in more general setting than ours. By considering various levels of generality for the matrices K, K.Q.V. V. this work identifies distinct regimes of token dynamics and relates them to the hardmax (3 oo) limit. Importantly, the analysis conducted in the subcritical regime and differs from the present work in focusing on a broader class of models, for which the critical regime has yet to be precisely characterized. We believe that combining the analytical tools developed in both papers could yield deeper understanding of this critical regime and represents promising direction for future research.
The remainder of the paper is organized as follows. Section provides precise mathematical for- mulation of the phase transition phenomena for the rescaled attention layer. We begin by analyzing token angles and the contractive behavior of tokens under two settings: an idealized but intuitive simplex model (Section 2.I and a more realistic model with the simplex constraint relaxed (Sec- tion 2.2 In both cases, we identify three distinct regimes of the scaling parameter, each leading to qualitatively different contrastive behaviors of the self-attention layer. Section 2.3 turns to the gradient norm of the rescaled attention operator. Because rank collapse is often accompanied by vanishing gradients, we characterize the gradient dynamics across scaling regimes and show when gradients vanish, or stabilize non-trivial limits. Section 3 presents numerical experiments, which validate these theoretical predictions.
Throughout this paper, when we denote a quantity as o,, where is the number of tokens, we mean there are positive constants C1, C2 independent of the dimension d, such that Jo,, (1) < Cin- C22 The constants C1, C2 depend on the assumptions in theorems.
2 A PHASE TRANSITION FOR ATTENTION
In this section, we establish the main theorem of this paper, namely a phase transition for the con- tractive properties of the attention layer when 8, = 7 log for some ? >>> 0.
Following Geshkovski et al. (2025) we study a simplified version of the attention layer with pre-
layer norm that described the introduction by assuming that K Q = V Iz. More specifically, the model we study is given as follows.
For any two points x, y E RRd, let (at, y) = x TT denote the standard Fuclidean inner product in Rd and || = / (a, x). Finally, recall that (x) == x/||=||
For any collection of tokens { In} } in Rdd define y = N(x)) E Sd-1 for = 1...., n and
1
ca
2,
Pik
Aij =
aij = 3 (2), M²)
(3)
Zi
for i,j = 1..., n. We then define
8 ATT(v:) = Σ
AijUs-
(4)
j=1
3

---

Published as a conference paper at ICLR 2026
Since the seminal work He et al. (2016), residual connections added to modern architectures and naturally act as regularization scheme of the attention map towards the identity; see Chen et al. (2225)). With said residual connections, each token 2; is mapped to x using the following update rule
24 = (1) + +
a 2 0.
(5)
Our first goal to understand where the angle <1)4, compares (T),, If ( xn)) < <(x,, x;)= equivalently (W/. > (y..Wih with s/4 = ( say that attention contrac- live.
The nonlinear update rule (5) can produce complex dynamics, in which some pairs of tokens move closer together while others drift apart. This diversity of motion is in fact the most desirable outcome in practice, and it emerges precisely at the phase transition identified in this study. Beyond this critical regime, the tokens exhibit unexpectedly cohesive behavior. To delineate the boundaries of the critical regime, we assume that the size and relative positions of the initial tokens are governed by constants independent of the number n of tokens. As analytically tractable extreme of this assumption, we first consider the case in which the tokens form a regular simplex in IR as in Cowsik al. (2024). Despite its symmetry, this configuration sufficient capture and predict the onset of the phase transition. We subsequently relax this constraint in Section 2.2 to show that the same phase transition occurs in more realistic configurations.
2.1 THE SIMPLEX CASE
The following assumption made in Cowsik al. (2024) and subsequently in Giorlandino & Goldt (2025). While rather stringent-in particular, it requires 2 n-it turns out to provide a tractable yet predictive setup study the contractive properties of attention.
Assumption! There exists nonnegative constants q 2 0 and p E (0,1) such that ||a.|12 q and (yi, Un} for any i.j 1,.... and +
Under Assumption I is easy see that there positive constants p' and o' such that (v/ v/y) P for all # and x 2 q' for all i. This simplification gives rise to a tractable phase transition.
Theorem 2.1 Under Assumption 7 there is a p' E (0, such that (V/s.()) = ol for all i + j. Moreover, if 3 = 7 log n where a positive constant, then for any i + it holds
play a a2
if 1
I-p'
lim (1/11)) 2-000000
p(a/++1)2
if = I-p'
(6)
adptaval1=p)-
1430
if 1
I-p"
T-p- the right hand sides of Equation (6) are strictly larger than for any 2 0. In other words, in the critical and subcritical regimes attention is contractive even in the presence of a residual connection. Of course, when a - oc, the effects of attention dissipates and the limit tends to for all phases. This is expected as the update from y, to y tends to the identity map, an effect known to mitigate oversmoothing" in residual neural networks; see Chen et al. (2055b).
Note that when S
Note also that for to
O, that is in absence of residual connections, the limit in Equation 6 reduces
if
I-p² I-p'
lim n
(1h.
4p
if y =
1+3p
P
if
y I-p'
In the subcritical case, the tokens contract in one step towards single cluster when n - oo while in the supercritical case, their inner product does not change. In fact, careful inspection of the proof reveals that in this supercritical regime the attention operator converges to the identity as n + co. When a 0, the suberitical case is mitigated by the residual connection which prevents token
4

---

1 Published as a conference paper at ICLR 2026
collapse to single point in one step. Nevertheless, this singular behavior reveals major limitation in the simplex case: since the tokens are equidistant the phase transition reveals an all-or-nothing phenomenon where attention transitions from Aij 1/n so that (TT
شÿ= = 3/ for all to Ai őıj that (M) yi for all i. In the next section, we present similar result
Theorem 2.3 where the simplex assumption is relaxed.
Before we end this section, we present the proof for Equation as a special case of Theorem 2.1 The detailed proof for Theorem 2.1 and the later Theorem 2.3 in Section 2.2 is included in Ap- pendix A
Proof 1 (Proof of Equation (7) In Equation 5). when
0, we have that x' = ATT (yi) for notice that the quantity Σ Σk=1 c
each i 1, 2, ..., , nh. In Equation 33, under Assumption
in the denominator of Aij independent of the choice of and equals e + (n 1)e Denote this as Z e + (n D)epa Then Equation (4) and (5) become
1 / ATT(W:)
e yi +
Σ
P8
ym
7
mæfi
Under Assumption 7 direct computation shows for any i = 1, 1,2.....
(2 x
1
(e23
(1+))3 + 2(n - (1=00 + +n (1(1))(1 + (n - -2)p)e200)
z
and for any different j.j 21,,...,,
(24,22))
1
ppc2 + 2(1 (n 2)2))) + ((n-2) 2) (n² -3n + 3))p) c28p)
Z
See also Lemma A.3
and Lemma A.4 for more detailed computations for (1', x²)) and (x/2 zzy).
(n 1) when let 3 logn, n, that e = and ne n + in Z. The largest term in then depends on the relation between y and 1+ pyy: when < T n 1
For Z c 8
is the largest term; when y T n² is the largest term. We then directly get the following three phases for 7. from the above arguments:
(1 + on (1)) neP8
if < 1
2=
(2+ on (1)) 8
if I-p" if > 1-p'
(1+ on (1))
where the terms (1) to 0 as n + too. Similarly, we can get the the following three phases for (1/4 x
is7 I-p'
lim (2/,21)
1+32
if
4-++000
if > I-p
For (d(,))))) we always have that limn t00 x))) ffr y in these three different regimes. Then
Equation follows from these two limits because (W: (x///1a<// ,///,,,
2.2 THE ALMOST-SIMPLEX CASE
In this section, we relax Assumption to allow pairwise angles and lengths to vary slightly. This relaxation makes it possible for tokens lie in dimension d < Although the resulting bounds are not as sharp as those obtained under Assumption [ they demonstrate that the critical scaling 8, (log n) is intrinsic and not merely an artifact of a particular geometric construction.
Assumption There exist constants 91, 42 E (0, 20), P1. E (0,1) such that 91 S || < 92 and P (yi, yz) S p2, for any i,j -..... and i + j. Moreover, Pr (yi, yÿ) for some i,
5

---

Published as a conference paper at ICLR 2026
Remark 2.2 Assumption 2 already allows near-ties when P2 is close 1, and it can be further generalized to allow multiple exact ties in the top scores. Specifically, one may assume that there exists a fixed k E Σ, such that, for each i [I....,n.. there are at most k indices j with (yi, F2} 1. Under this setting, all of our main results continue to hold with the same critical scaling order log n but with different constants depending on k. In fact, this setting is special case of the more general setting discussed in Appendix [ where Assumption ] partitions the inner products into three ranges, (ii p1), [pa, pa), and {1}. with S Pi < P2 P3 < pa 1. The phase transition behavior remains of order log n in that general setting as well. For clarity and readability of the main exposition, we keep Assumption 2 in the main text.
easy to see using standard probabilistic tools that Assumption 2 holds with high probability when the y 's are independent random vectors uniformly distributed on half-sphere for example.
Theorem 2.3 Under Assumption 2 we have the following phase transition when & = log for some fixed y > 0.
If <
1 1-p1" then there is a constant e > 0 depending on a. P2, 91, 92, such that
lim min v/y) 2 P1 + Pl xFj
(10)
which implies that the angle between tokens becomes strictly smaller after attention layer Equa- tion
5
If T-pz then for any E [[]
(1)) yt + (1), and hence x! = ys + az, + 0,,(1)
(11)
where the term (1) goes to as n + too with a speed uniform in i, Hence, when y > I-p2 1 for any two different i,j E [1,n].
lim n-++00
(e v/))
(ye, Uz}-
(12)
which implies that the angle between tokens does not change after an atlention layer Equation 5
The proof for Theorem 2.3 included in Appendix A but the general intuition is similar to the
proof for Equation in Section 2.1 As we have seen in that proof, the first step to build up
phase transition regimes for (v/ Vÿ to study the phase transition regimes for Z; in Equation (3)
Adjusting the logarithmic scaling factor causes different phase transition regimes for Z; first. When 7 is small enough, the weights e consisting of Z, are asymptotically uniform, and each token almost equally interacts with the other tokens. When y is large enough, each token mostly focuses on itself.
Building on this observation, Theorem 2.1 and Theorem 2.3 together demonstrate that y controls the effective interaction range of each token. In particular, we have seen in Theorem 2.. the existence of the critical regime when 7 1 In this case, although the tokens continue to contract, their rate of shrinkage is evidently slower than in the suberitical regime, as shown in Equation (6) and Equation 7
It is hence natural ask whether further regimes emerge when 7 is varied between the supercritical and subcritical threshold. Indeed, in Appendix we prove the existence of a nontrivial middle phase when y is between the two extrema T-pı and T-pa under refined assumption on the distribution of tokens, which allows for sharper characterization of the transition. Under this refined assumption, Theorem C.2 show the existence of 71, 72 such that Equation (5) presents three different phases:
y < n. ?1 < <== < 72, and 7 > 722 In the extreme regimes, when 7 < ?T, each token interacts with almost all the remaining tokens, while when 7 72, each token only focuses itself, consistent with Theorem 2.3 In the intermediate regime 71 < 7 < 72, however, the weights e concentrate on only small subset of tokens, so that each Z, and hence the update in Equation 5 is dominated by a few highly relevant interactions. This shows that the logarithmic scaling enables each token to dynamically select its most relevant context.
We conclude by noting that those o,, (1) terms in our theorems satisfy the bound Jo,, (1) - Cnn Ca for some positive constants C1, C2 that are independent of d (though varying across theorems). As

---

Published as a conference paper at ICLR 2026
result, the simplex configuration (Assumption 1 and the almost simplex configuration (Assump- tion 2 remains valid under repeated application of the ATT operator up to poly(n) iterations. In particular, the accumulated enror remains negligible at this scale, theorems and arguments extend to transformers with many layers.
2.3 PROPAGATION OF GRADIENTS UNDER ATTENTION LAYER
In the previous section, we established how attention scaling influences the propagation of token rep- resentations, corresponding to running the Transformer the forward (inference) direction. During training. however, the Transformer is also executed in the backward direction to compute gradients via backpropagation (Rumelhart et al. 1986). this section, we show that a similar phase tran- sition arises in the backward pass: in the subcritical regime-where token representations rapidly collapse in the forward pass-the gradients also collapse, whereas in the supercritical regime they retain their scale. The stability of gradients is crucial computational consideration that strongly affects a model's s ability to be trained effectively. For this reason, several theoretical analyses of gradient dynamics in Transformers have been conducted, albeit without attention scaling; see, for example, Cowsik et al. (2024); Dong et al. (2021): Noci et al.] (2022).
Let the input token configuration be denoted by ((), and let X(t) represent the positions of all tokens at the output of Transformer layer t. To compute gradients, one needs to evaluate the end-to- end onput Jacobian across L layers of the Transformer. By the chain rule, this Jacobian can be expressed as
( = OX(l) OX(L ax(1) (
OX(L OX(L 2) ax(0)
Thus, the end-to-end Jacobian can be obtained by recursively computing and multiplying the layer- wise Jacobians. This procedure known as the adjoint method in dynamical systems theory (Lions) 1971). and backpropagation in the machine learning community.
Our main result shows that when 3, = 7 log n with subcritical 7, the typical singular values of eX(++)) are close to zero (apart from the contribution of the residual connection). In contrast, for supercritical values of the contribution of the attention component to the Jacobian is non-trivial and behaves as a normalization map.
0X(e)
We now proceed with formal definitions. For E IRd, let (z), denote its u-th coordinate for u = 1, ..,,,...4 d. The concatenation X (œ), tokens. The normalization map is defined by
In) E Rnd represents the configuration of all
N(X)= N N(1,, F2, .2. In)
N(a,))
(13)
and the attention map by
ATT(Y) ATT ATT))=AT((( U2....,, ya) (ATT()1),ATTOO ATT(Y))),
(14)
where ATT (v:) is defined in equation and Y = (yh,..., V..). Under these definitions, the up-
date equation 5 can be written compactly
X' ATT(V(X)) + axx,
(15)
where X' (z/,
We define the nd nd Jacobian matrix
(8(a))e ((z))a) (..)).(..))
X
(16)
for i,j 1,... and u, 1.....d. d. The matrix norm of V X' is given by
Σ d Maz))) 2
V tr TXX1
(17)
H(x:)u
1-=1 u,==l

---

Published as a conference paper at ICLR 2026
Let 01, "2.00....., nnd denote the singular values of V X'. Then the normalized Jacobian norm satis- fies
V X Σ of
(18)
nd
which represents the mean squared singular value of the Jacobian.
Before stating our results on na x we note that the Jacobian Vx X' can be decomposed into the residual part aind and the attention part Vx(ATT(V(X))) As shown in Theorems 2.1 and 2.3, the residual component and does not affect the phase transition behavior. Therefore, to streamline the analysis, we focus exclusively on the attention term Vx(ATT(N(X))) by setting a = equation 15 The following theorems characterize nd V X under this setting.
Theorem 2.4 Adopt Assumption 7 and Equation (15 with a Then, we have the following phase transition phenomenon: let = 7y log n where is positive constant.
If < -
xxX 0 (1).
(19)
If' If== +
Vx 2 1 1 (1-1) 1 + onn())
d)
(20)
nd
4q
1f T-p
12 (1 (1-1) d on(
(21)
nd
In both cases, the terms (1) go to as n - +oc, with speeds depending on 7/s P,
The results of the previous theorem show that under the simplex assumption, the phase transition in the backward dynamics (for gradients) is as sharp as for the forward pass: for small 7, gradients do not flow through the attention block.
We can also extend the analysis for Theorem 2.4 to the relaxed Assumption
Theorem 2.5 Adopt Assumption 2 and Equation (15 with a Then, we have the following phase transition phenomenon: let log where is positive constant.
If 1-mi
3 2 (log(n)) (1), qid
(22)
nd
If>> > I-ps"
1
1 1 21 ozz
X
(23)
nd
92
which is away from 0 even when d, is very large. Indeed, when > T-pa for any fixed i, e [1.
1
((2;))
dez (Ta you?) (1) (1) la. |>>:||
(24)
((r))
dxd
where the leading order term is exactly (x;(= d(z.)a as shown in Proposition B.l Here, Ta is the x identity matrix, the term (1) (1) (o, respectively) is d x d matrix (constant, respectively) with matrix norm as defined in Equation (17) (value, respectively) going to as n + +00, with a speed
independent of i.j but only depending on "/, P2, 91-
8

---

Published as a conference paper at ICLR 2026
We present the proofs for Theorem 2.4 and Theorem 2.5 Appendix Note that the log" term in equation 22 small for typical values of and d used in Transformers. Theorem 2.4 and Theo- rem 2.5 also corroborate the fact that tokens collapse fast when y the subcritical regime, while each token only focuses on itself when 7 is in the supercritical regime.
3 NUMERICAL EXPERIMENTS
This section reports numerical experiments designed to support our theoretical predictions. In the following numerical experiments, we test the phase transition in the simost-simplex case as Sec- tion 2.2 We generate samples {ZI....,In) B( = p [0, 1] for + More precisely, we generate 2; according to
Rd such that the expectations E||æy/|² = 1 and
I₁ 2 v1 22
(25)
where 20, B...... 2n are i.id. standard Gaussian vectors in Id. The generated samples satisfy the Assumption with high probability.
In Figure I we plot the input-to-output angle ratio defined as
A
2
Σ Isi<jEn
(w/e v/y) (Ui, M5)
(26)
n(n
for samples processed through single self-attention layer with different y and of different dimen- sions d. Consistent with our theoretical predictions, the layer acts as a contraction mapping when y small, reducing pairwise output angles, whereas for large 7 the output angles remain nearly unchanged from the input. Moreover, in the large regime the angle between input tokens (ys My) * j) concentrate near p. so that the simplex Assumption [ is effectively satisfied. In this set- ting. we observe a sharp phase transition in agreement with Theorem 2.1 In the small d regime, however, the input tokens (yi, Uj) randomly distributed in an interval (P1,P2), and an intermediate phase emerges in which the contraction is only partial: some angles shrink significantly while others remain close to their original values, which smooths out the transition.
10
10
1.0
0.5
0A
(a) dl
(b)
(c)
Figure 1: Plots of the input-to-output angle ratio X, defined in Equation (26). as function of and 7. The tokens are first normalized by a pre-layer normalization and then passed through single self-attention layer (4). with residual connections and MLP layers omitted. The dashed curve corresponds to = T-p" which approximates the actual phase transition with increasing accuracy as d grows, as implied by Theorem 2.1
In Figure we plot the normalized matrix norm for the nd nd matrix V X', defined as
nd V
(27)
for samples passed through single self-attention layer with varying and dimension d. Across all three plots, the normalized gradient norm remains close to 0 when 7 is small, while for large 7 approaches 1dd, consistent with Theorem 2.5 Similar to the token-angle behavior, a sharp phase

---

Published as a conference paper at ICLR 2026
transition emerges near, 7 I-p in the large-d regime, in agreement with the predictions under the simplex assumption. In lower dimensions, fluctuations in the pairwise angle prevent perfect concentration, and the transition is smoothed into an intermediate regime where the gradient norm only partially stabilizes.
10
1.0
08
0.2 0.6
02
0.0
(a) d
(b) d
(c) 512
Figure 2: Plots of the normalized norm of the gradient, defined by Equation (27), as function of and y. The tokens are first normalized by pre-layer normalization and then passed through single self-attention layer 4 with residual connections and MLP layers omitted. The dash curve shows I-p" which approximate the actual phase transition with increasing accuracy as d grows, as implied by Theorem 2.4 The matrix norm 7) is computed by the Hutchinson trace estimator (Hutchinson
1989
based on the definition in Equation
4 CONCLUSION
This paper develops framework for understanding phase transitions in self-attention as the context length grows, identifying critical scaling 3n log n that separates subcritical contractive regime from supercritical unchanged regime. A central message is that this transition is rooted in the geometry of the score landscape: remain (1), which leads to the log scaling. We show that this scaling is robust under various perturbations and in settings permitting multiple near-ties, demonstrating that the log law structural consequence of content-adaptive interactions.
our model, the gaps between the top ordered scores
ACKNOWLEDGMENTS
Philippe Rigollet is supported by NSF grants DMS-2022448.
LLM USAGE
Large Language Models (LLMs) were used during peer review for grammar and syntax refinement only. All ideas, technical content, analyses and conclusions remain the authors' work.
REFERENCES
Pedro Abdalla, Afonso S Bandeira, Martin Kassabov, Victor Souza, Steven H Strogatz, and Alex Townsend, Expander graphs are globally synchronising. 2022. arXiv:2210.12 12788.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Gc, Yu Han, Fei Huang, al. Qwen technical report, 2023, 16X69:2309.16609.
Iz. Beltagy, Mathew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arX1v:2004.05150, 2020.
Giuseppe Bruno, Federico Pasqualotto, and Andrea Agazzi. Emergence of meta-stable clustering in mean-field transformer models. In International Conference Learning Representations, 2025a.
10

---

Published as a conference paper at ICLR 2026
Giuseppe Bruno, Federico Pasqualotto, and Andrea Agazzi. A multiscale analysis of mean- mean-field transformers in the moderate interaction regime. NewrIPS, 2025b.
Shi Chen, Zhengjiang Lin, Yury Polyanskiy, and Philippe Rigollet. Quantitative clustering in mean- field transformer models, 2025a. WwXiv:2504114697
Ziang Chen, Zhengjiang Lin, Shi Chen, Yury Polyanskiy, and Philippe Rigollet. Residual connec-
tions provably mitigate oversmoothing in graph neural networks, 2025b. utXv:2501.00762
Aditya Cowsik, Tamra Nebabu, Xiao-Liang Qi, and Surya Ganguli. Geometric dynamics of signal propagation predict trainability of transformers, 2024. arXiv:2403.02579.
Laurens De Haan and Ana Ferreira. Extreme value theory: an introduction. Springer, 2006.
B. Derrida. Random-energy model: An exactly solvable model of disordered systems. Phys. Rev.
B, 24:2613-2626, 1981. doi: 10. 10.1103/hhyReev..4..22 2.242611..
Yihe Dong, Jean- -an--Bptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pp. 2793-2800 PMLR, 2021.
Borjan Geshkovski, Hugo Koubbi, Yury Polyanskiy, and Philippe Rigollet. Dynamic metastability in the self-attention model, 2024a. arXiv:2410.06833.
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clus- ters in self-attention dynamics. Advances in Neural Information Processing Systems, 36, 2024b.
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. mathematical per- spective transformers. Bull. Amer. Math. Soc., 2025.
Alessio Giorlandino and Sebastian Goldt. Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation, 2025, arXiv:2505.2433.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Pearistics-Simulatte and Computation, 18(3):1059-100 1989.
Vishesh Jain, Clayton Mizgerd, and Mehtaab Sawhney. The random graph process is globally syn- chronizing, 2025. arXiv:2501.12205
Nikita Karagodin, Yury Polyanskiy, and Philippe Rigollet. Clustering in causal attention masking, 2024. arX4v:2411.04990.
Nikita Karagodin, Shu Ge, Yury Polyanskiy, and Philippe Rigollet. Normalization in attention dynamics, 2025. arXiv:2510.22026.
Jacques Louis Lions. Optimal control of systems governed by partial differential equations, volume 170. Springer, 1971.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Ken M Nakanishi. Scalable-softmax superior for attention, 2025. arXiv:2501.19399
Lorenzo Noci, Sotiris Anagnostidis, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27198 1977199--111,, 2022.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023. Ntt:23399000071..

---

Published as a conference paper at ICLR 2026
Yury Polyanskiy, Philippe Rigollet, and Andrew Yao. Synchronization of meenn-fell models on the circle, 2025. DxXi:2507..22857.
Krishna C Puvvada, Faisal Ladhak, Santiago Akle Serrano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, et al. Swan-gpt: An efficient and scalable approach for long-context language modeling, 2025. arXiv:2504.8871.
Philippe Rigollet. The mean-field dynamics of transformers. arXiv preprint arXiv:2512.01868, 2025.
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 33366069 1986.
A PROOF OF THEOREM 2.1 AND THEOREM 2.3
In this section, adopt Assumption 2 and prove Theorem 2.3 first. Then, prove Theorem 2.1
To simplify notations, we define [1, = 21,,,..... n for any n E Z+.
We study the asymptotics of the quantity (2/), as +00. We use the notation
Z:
ai e ti
(28)
light
Lemma A.1 Let 3 = y logn where is positive constant. Under Assumption 2an Equation (5) for any [I.
(1 (1+)n(1)) (Eepseuu Ease) (1 (1+on(1)) on e
is < T-pi
2,
(29)
if T-p2
where the terms (1) go to as n + too with speeds independent of but only depending on Yss P1, P2.
Proof 2 (Proof of Lemma A.I We notice that
Z: = e +
Σ
E
Rik
(30)
kØi
We also notice that St = n for any then holds that e = nτ and
n 1P (n 1)
Σ
e 2tk
S n TP2 (n 1).
(31)
leffi
Hence, when
< I-pi' n² < n 1400, the leading order term in Zi is Exzs e We also see that
e
Zs =
+1
(32)
(Eaz Eace
Riffi
with
72"
(33)
Eneg eoik
n2'P (n 1)¹
which goes to 0 as n + +oc, and independent of i but only depending on "T. P1. Similarly, when I-p2' n² > n 147p2, the leading order term in Z; is e², and similar arguments hold true.
Lemma A.2 Let B 7 log where y is positive constant. Under Assumption and Equation if > I-p2' then for any E n.
ATT(Y:) = y: + on (1), and hence x = ys + aa, + o (1),
(34)
where the term o, (1) goes to as n > +oo with a speed independent of i but only depending on P.P2.
12

---

Published as a conference paper at ICLR 2026
Proof 3 (Proof of Lemma A.2 n 122,, and hence,
According to Lemma A.1
that when ?
1-p- n2
(1) = -,
e Σ
eassys
(1 (n+ on))
yi t Σ Ea -
(35)
j&i
531
Because ||yl|| =
e
Eass Vj
Se 8
Σ cui Sn n Snann -1),
(36)
jti
jti
which goes to 0 as n - +00, and independent of but only depending on 7, P2- This shows that when y > I-p2'
(y) = (1 + on + oz = y; + On))
(37)
Lemma A.3 Under Assumption Aand Equation 5 for any E [I.
2a 20/|=;
(1/, a² 12:111
eB
cas (yt+ Uu)
Z,
j&$
(38)
1 Z?
+
20
+
2
c0s (y) -
e0ry eary-aik (yrss (Wee Us)
j7t Kii
Let 7 log n where 7 is a positive constant. When 7 < T-p:'
Exzt cark (Vi,
(k) yk) Exzi E1es Case tautta (yk, v1)
(1/, = a2112;112 +
+ o,, (39)
Exyee Eac
(Eeeps eas
When 7 T-pa"
(1/=1) (al|<,/) + 1)² + o1).
(40)
to 0 as to0 with speeds independent of i but only depending In both cases, the terms (1) to on Y, P1, P2, Ot.
Proof 4 (Proof of Lemma A.3) According to Equation 5 we see that
( a + 20 ATT(y:)) + (ATT (y:))-
(41)
Equation 38 follows from direct computations. Two phase transitions Equation 39 and Equa-
tion 40 follow similar arguments as in Lemma
Lemma A.4 Under Assumption 2
and Equation
55 for any different i,j E [I,
al|2;|l
al|zill
(6, a² (x,,
e (g5,
Σ ain (My,, yk)
+
e (y Uz) + Σ
ail (Mi, m1)
Zi
Zg
k
29 (1t, U1) e
case (K), 2k) + e Σ
easl
(1) +
eaittayi (yrks (1)
Z,Zg
K-f 15
(42)
Let = log where y is positive constant. When γ I-p1'
(2%, = a² (x,, + a112,1 Exgre cai (Wi.Uk)
Eiss ess((e))))) Eiz,e
+ al|zx|l
Expr Eain
Exets Eiee, eanttagi (BR
(43)
+
+ On).
Exer case)
(Eiz, easi)
13

---

Published as a conference paper at ICLR 2026
When y I-pa'
x (al|ziil + 1) (ys Uz) on).).
(44)
Proof 5 (Proof of Lemma [I,n].
that for two different i,j A.4) According Equation
(1) a ATT())) + (a, + ATT
(45)
Equation 44 follows from direct computations. Two phase transitions Equation and Equa-
tion 44) follow from similar arguments in Lemma A.7
Next, we prove Theorem
2.3
Proof 6 (Proof of Theorem
2.3 We first discuss the case when 7 < 1-p1"
According to Equa-
tion and Assumption
2 we see that
( 2 a²
allzzlll oal +ala:lll + +1 (1)
(46)
<ialll + (a|l/11) + o,)).
By Equation
39
we see that
( x < a² ||=;11 + 2a | || + P2 +2
2
= 2 + 2a [ +1 (1-p2)(11) + 2a/l<;;||) + (1)
(47)
s {a||<ill) + +1² (1-
+2x))) + o1.
We have similar inequality for (z)), So, there is constant 0 depending p2, a, 92, and independent of n, such that
1
1+8
+ (1), and |a
2
+ (1).
(48)
|
x: +1
|zy +1
Hence,
We v/) 2 p1 +0 (1) 2 p nht oz)
(49)
for E = + 26)1 > independent of n.
For the case when Lemmal A.3 and Lemmal
Equation
11 and Equation
12 follow directly from Lemma A.2
Proof 7 (Proof of Theorem 2.1
We notice that Assumption corresponds to the special case when Q1 92 9 and #1 = p2 = p in Assumption Clearly, Z, is independent the choice of E [[. n]
by its definition Equation 28 According the explicit forms Equation (38) in Lemma A.3 and
Equation (42) in Lemma A.4 one directly sees that both (x Xii) and (I,, 22) are independent of the choices of J [1,n]. We further compute that for any [1,n].
'a²g + 20 p
if I-p p
lim (1/.21) n-0
a²q+ + / + +)) + 1+3p
if T-p
(50)
(avā+) 1)²
if
for any two different i,j E [L,n].
lim (1/, =²)5 = plava+11
(51)
+-++00
Equation 6 follows from Equation
50 and Equation 51
When 1-p see that
1)2 lim (W (/) a²y + 20x /ap + P
pla-/g + 1)2
a² 20 20 +1 P.
(52)
4-0+00
where the strict inequality is because < 1. When 7
1 o²
we can similarly show that
lim, too W/. 1/)) p. This completes the proof for Theorem
2.1
14

---

Published as a conference paper at ICLR 2026
B PROOF OF THEOREM 2.4 AND THEOREM 2.5
We prove Theorem 2.5 first. We need to explicitly compute terms in ((ATT(M(Z))) need the following lemmas.
for which we
8(z))s
B.I PROOF OF THEOREM 2.5
Lemma B.1 for any i, k E [1, n] and u, [1,d]
((N(x)) du
Swn IT 11 (xr (2rk)
(53)
(ss))
1 3
Proof 8 (Proof of Lemma B.1
((N(xx))
a((x)) |
Swn ||
(20) w
(20)) lu
1
8sk
(54)
a(x;)u
|0rk 11
Lemma B.2 For any k,j E [1,n] and w, v E [1.d].
((AT(y)))
Oyy) m
n
Σ
amm (Um))((mm)) + (lsss.o)) (w + owcr)
m=1
e (ls.smm (ym))y ]
8kjB
Σ
Blss (21) + Ses, (Us))
Σ m=l
-2
B
(55)
Proof 9 (Proof of Lemma B.2)
By Equation
-==1 Byy (ym)
n
(ATT();)).
(56)
Σm (us.
A direct computation shows that
0(ATT (2:))
(ykk) Jw
Σ (6kz (Mm3) + 30 (Bm)) + Skmm (usissm) s8m)
Σ
n2
3 mi
e
(ur) + -ui +ns()))) v)) Be Be (oss.1))
Σ
(mm) (ym))
Blys Mtm' (ym) (onn) + (lsssnnn + )arc))
Σ 1-1
c lats mn)
m==1
Blys (mm) (Jmm)o) }
((33-vn) (vi) + Bess. (Vi)u))
Σ
m=11
Σ (uy--un)
(57)
15

---

Published as a conference paper at ICLR 2026
For 2, y R. 2 y to denote the x matrix with (u, v) element (x0 (z) Le., I @ = xy". We then have the following proposition.
Lemma B.3 Adopt Assumption 2 and Equation S For any i,j E n.] consider the d matrix formed by a(AT (a)))) for u E [1,d]. Denote yn ((aa) for each k E [1,n]. Then, this
matrix has the following form:
= -| (R, + R2) - (U1 U2) 8 V2 2z (58)
O(x:)u
axd
where Zj = =i= (l as Equation 28).
R1 = 81y (W,- @ (Wy))), R2 = (vs. ((-3) + P v. 2 1a).
(59)
and
U1 = 6 Vi),
U2 = (uss (Po.y)).
(60)
In Equation
59 and Equation
60
Bly Yms
W
Blas Bl Um. @ ym,
Pay = y 2),
(61)
Proof 10 (Proof of Lemma B.3 By chain rule and Proposition B.
we have that
((
(ATT():))
O(x;)u
alyk)
(xx)a.
Y=N(X)
|| ((ATT(y)))
(x:) ((ATT();))
= ||>,|1-=
O(vi)a
m=1
((ATT();))
((ATT())))
= ||
(16)
(Fr)
a(v:).
O(yi)
10
(62)
According to Proposition B.2
and the notation Z, -E= enst, we see that
8(ATT (3:))
(1t am
8
Σ Mly) (ym, U.) (Vm))
+ 1)(e))) (luss
Zz
vlr um) (Wm)o) )
8
Σ
elys wi) (w1,2r) vid
Σ
1=1
(63)
16

---

Published as a conference paper at ICLR 2026
Hence,
|, (1;))
a(xi)u
((TT(()))) - (y:)u a(kk)
d
(ATT((ss))
)
Σ (1s) u==
a(h))
= -[Ooy
Σ
e (lg ((Um)olUm)e - (Um. ((Mm)o(ss))
Am=1
(64)
Blus vi) (B(U)),(U1) + duv - (B(y5, wi) + 1) (ue)) 2z
- 8
Σ (lus)) ((2r) (e),8e) (w)
Btyy.vmi (Um)) ]
Be (( ((yz)u (ui (iNuu
Σ
We then adopt the notation Equation 66 i...
a W
V Blus Nm) (mm)
yms
pluy. Um @ Dmn
Pay = y yy 2),
(65)
So, the matrix form of Equation 64
becomes
6 (W, y1 69 (W (Wyk)) (ansaa (By3 us + La (3(y). +1) ys 60 vi)
Zz
66,8 (V, (V.. Be) (yj (y Mi)ya)
V 2z²²
(66)
(638 (W, - ye (W e ((ys.bee ((-y:
yi
3P v. 2
yt + Ta) Zzg
8 (P v. V3) + S wid
(P ps. v))
2
We further use the notations in Equation
599
and Equation 660 i.e.
R B (W, Yi @ (W (W))))).
R2 Viy.vi ((-ys ys PP.W)))) 8 Ia),
(67)
and
U 8,g3 3P y1 Vi), U2 (i (P (Py.))).
(68)
Finally, the matrix form of Equation (44) becomes
[R, + R2) - (U1 + U2 2 V, 2
(69)
Lemma B.4 Let B y log where y is positive constant. Under Assumption 2 and Equation
(5)
i T-pa then for any fixed i, E [1, n]. the x d matrix satisfies
(B(ATT(N(E,))) O(z))
dig ||>i||
(Ia v4 (1) + o,, (1) Ias
(70)
dxed
where the leading order term is exactly 0(N(2;))
o(ze). The term (1) (1), respectively) is a d x d
matrix (constant, respectively) with matrix norm defined in Equation (77 (value, respectively)
going to as n + +00, with a speed independent of i, but only depending on "/1 P2, 41-
Proof (Proof of Lemma B.4) We frequently use this formula: for two vectors V1, V2, the matrix norm of V1 &0 V2 defined in Equation 17 is [[]]] V2]l. When 7 I-m nt n n141p, and
17

---

Published as a conference paper at ICLR 2026
know from Lemma A.1 that Z, (1+ + o1 e² for any j E [1.n]. Adopt the notations in Proposition B.3
we then show the following facts when 7 I-p2"
R Z o1),
ReZz 8ig ( (- U. 2 Ia) + 01) (1011)+)olll o1)) Id=
(71)
and
10 + (2)+ V,J - =0,
(72)
First, for R₁Z when i * j. have that R
by its definition. When j. R1
3 e (3963mm (ym @ Um (ym. yi @ Vm) and we notice that the term when m = i 0. So, 1m 2 Um (Um yi @ yoll ll I Il 11? + ll 11 | = 2. e² = =n²,
because
| S (nn 9e 2zj' S 2y log(n) .n -1 (1+ on (1)),
(73)
which goes 0 with speed independent j. because 7/2 +
0.
For RzZz we notice that when + ( Z S 8pp 00 11 (1+0, = n 202 (1+o, (1) + o, (1) La when # j. (-y: 2 y + La) + o (1) + o1) La.
which goes to 0 with speed independent of j. So, R₂Z
When i = j. R e (-y & y: + Ia). and so R
For [U, + U2) @ Vz]+ 2 see that when i + j. U1 = 0, and so
IIU, + U2) & V1ll 2 S 2 23 Σ vm + 11 P
(74)
3e 3e 8(+ (1+)2)
n llg(())n (0z 1)+1 (1 + om)))),
which goes to with speed independent if because y > I-p2
When = j. U2 = 0, and so
[I(U, + +2) & Vall 23 < 2, PI ViIIIV
(
< Σ
P Pu.Wuall ((m))mm)) mith
(et ++
(75)
S 2,20 (e (e 3 (
y (o 2 (02 11)+ + n no on)))
which goes to with a speed independent of i,j because y > I-p2" Hence, [(U, + U2) V,]+ 2 (1)
Lemma B.5 Let B = y log n where 7 is positive constant. Under Assumption 2 and Equation (5)
if < T-m then for fixed i, nl) the d x d matrix satisfies
(
(D(ATT(N(E))))
< 11:1111
3. (2,6g, + (23 Vdle Z)1).
(76)
O(xz)a
dxd
Proof 12 (Proof of Lemma B.5 According to Lemma A.1 when ? 1-pi Z3 (1 + on (1))
(Enez ease) for any j E [1,n], and Z₁ 2 n1 +1 (1))n(1)) > n² (1+,,,()) because 7P1 +1
Adopt the notations in Proposition B.3 we then show the following facts when < I-pi'
| < 85
[|al| < 1 (8+ vd+vd=II.
(77)
and
[ U2) & V,ll
2 <B + a Z
(78)
18

---

Published as a conference paper at ICLR 2026
First, for RzZz¹, when j. we have that R1 0 by its definition. When i j.
R = 8 3Em=1 ((y..mm (ym & Um (ymm U/ Va)3); @ 4m) So, because we have that ll @ ym
(ym, VelU/ Vmll y Vn ym y Un ym
R1 2 S 3Zz Z B.
(79)
For R2 because - 30 Us + Tall = v 1, we have that
| 2 S c (8+va-1)
(80)
For [U, + U2) 0 vjJ
27² we see that || S Em= elys Mmm
23- Also, ||
dij B Vi 8. ||U2|lZj Be Zj Hence, we have that
[I(U1 + U2) & V, Ill 2,² S (ōi5 + e 27
(81)
Proof 13 (Proof of Theorem 2.5
Theorem 2.5 follows directly from Lemma B.4 and Lemma B.5
B.2 PROOF OF THEOREM 2.4
The proof for Theorem 2.4 requires more delicate arguments. The part when 7 > I-p in Theo-
rem 2.4 directly follows from Lemma B.4 so we only focus on the part when 7 S T-p We remark
that when y < I-p- our result is that nd V X 0 + o, (1), which is a better estimate than
Equation (22 in Theorem 2.5
We first have the following lemma which replaces Lemma B.3 when we adopt Assumption 1
Lemma B.6 Adopt Assumption| 7 and Equation 5 For any i, E [], consider the x matrix formed by ((, for u. E [1,d]. Denote yv = N(xx) for each le E [1,n]. Then, this
B(a.), matrix has the following form:
=9 [RR + R2) - (U, + U2) @ (U3 (U,+ -.]] Z-2
(82)
O(Ii).
dxd
where Z = 0 + - (1))))
R1 = 8r (W- y1 & (Wyi)),
R2 = (usiii ((-V) + 3P 9. 8 - Ia),
(83)
and
U = Be (Py, V), U2 = Be (Py,
(84)
U3 = (e g &P)y
U4 BPv.
In Equation
83 and Equation 84
3ms
W
Σ ym Um Pay = -- (y, x)..
(85)
Proof 14 (Proof of Lemma B.6 We first apply Lemma B.3 to get Equation (58). After replacing
(Mi Fm) = for m # j. we can obtain Equation (82). The only remark is that the term 81, (WW
Ya & (W (y:)) in R₁ of Equation (59 is nonzero when i j. Then, when i j. W. y6 &
(W (1)) Σm= (fu Mmm) (ym & Um ys @ Ym (ymn, M.)) If 1fm= = i, the summand (ym Vm y4 ? Um (y vi)) becomes O. Hence, W (W (11)) Emerr e ((y..mm) (ym Dym-mymm- 100m-y::0pp Bym (yoes 91))
80 Em (3 &
@ ym (Mm y:)) = Bp (W & (Wi))).
Next, to compute the matrix norm of Equation 82 we see that for any matrix K, its matrix norm
square equals to Tr(K" K). Hence, the matrix norm square of Equation (82) equals
---4. Tr (R, (R + R2)] 22( U2) (R, R (U, +.))
(86)
U3 U.ll2
U U2
19

---

Published as a conference paper at ICLR 2026
We then compute these terms separately, and them in i,j. We first have the following basic
equalities for the notations V, W in Equation 85
Lemma B.7 For the notations in Equation 85 i.e,
Ym,
W
Σ Ym ymn
Pay y x),,
(87)
we have that
Tr(
Σ
Wm 31)² 2 = =n(np² + (1-p²)),
myd
(88)
Tr(W) n,
Tr(W W.y! np² (11-²² p²), |P U3 112 11 1-p².
Also,
Wy Σ
Wm, Millm = (1-p) + PV,
m=1
(V, 34)
Σ
np+ Um, ut) = n + pn(n 1) n(np + (1- p)),
(y 91) = -p + (1- -P), p))
m=1
(89)
|V
Σ
m,l
P y. V 1 | - (V, (n 1)(mp - (1- (1-p))(1 p),
y 2 n² p² (1 +1 2p) p)3
Proof 15 (Proof of Lemma B.7 Direct Computations.
Lemma B.8 For terms R,, R2 in Lemma B.6
have that
Tr [RR, + R2) (R) + R2)
1
28p [n²p²(1 p) n(1 ppp--33pp)))) - (1+ 2p)(1=))} ]
(90)
+ Be n(n - 1)(1 - p²)
+ 28 1)m 1)m++0 23p [32 - + - (1)
1]d-1nn(n n(n 1).
As a corollary, when we pick 3 = y log n, we have the following phase transition limits as n -
to0:
p² p) + (1) 1 d d-1+
if < -
Tr Σ
+ R2 + R2)] = 4-1+82 + (n)
if T
1
(91)
n
1
if I-p
Proof 16 (Proof of Lemma B.S
We first notice that W is a symmetric matrix and ||aal|| 1.
then expand each term in Lemmal B.8 and use Lemma B.7
Tr (R1) RR]
B 28p Σ E(r (w2 2W (W + 1
= 280
[TrW2= w/ + 11
(92)
= 20 |n²
p) + n21 - p) + p (01)--(110
(1 2p) -p
Then,
Σ
Tr (Ri)
R2 = Tr Σ 3e (w Wy.v?" ((0 ((- + +3PP. 8 ys + Ia)
i,j
3 Tr Σ (W- - Wyoy? (-y.vt + 1a) = ( Σ (W Wy.
= 1)(1- -p²))
(93)
20

---

Published as a conference paper at ICLR 2026
where the second equality is because P yy Uo O.
Σ Tr (R2)
RR2]
Σ
e 28195, Tr [[- 3P -.,Pyuu u.Uu)yy? + Ia) (vil we + BBPu ui) + Ia]
1
Σ 28p [[1 + (1+ p²)) - d) + Σ 28 (d -d
itj
20 (d 1)n + 20p 3 - p²) + d 1| n(n 1).
(94)
Next, we show the asymptotics Equation 97 as - too. According to Lemma A.7
we have that
on()) np Z = (1 (1+ ea if 1-p"
if < 1-p'
(95)
1
That is, when q < T the leading order terms those terms involving ne and all the maining terms to O after dividing nepp; when 7 > T the leading order terms are those terms involving eº, and all the remaining terms go to after dividing e!. Hence, when 7 TT the -. leading order term in Equation (90) is the term 3 28 p² p); when > I-p' the leading order term is e²8 (d 1)n. This proves Equation 91
Lemma B.9 For terms R1, R2, U1, U2 U3, U4 in Lemma B.6 we have that
(R, E(U, + U2) (R, + (U3
(
=232200 m [)(np + (1- p))(L - p)
+ 20p () (1 p)) (1)p)
(96)
+ 3e n(n () (1 (1-)))[1
+ (e - Bp) + 1)m(n - 1)((1 P²)
+ 38 ()(pp -1)
As corollary, when we pick 3 y log n, we have the following phase transition limits +oo:
88 - p) + o1)
if < -
1 ntz
E2.++ R
pP-1- + (1) 4
if = T-p
(97)
1
0+
if T-p-
Proof 17 (Proof of Lemma B. We expand each term in Lemma B.9 and also apply Lemma B.7 to each term. We first estimate terms involving U1.
Σ EUTRIUS- Σ 2800 - (P))
v)" (W- y1 8 (Wy.)) y/
200 -eB0) E( V)² Wy, p88² 280 (e )d0) VP
(98)
=232 280 )P)(n - 1)pp + (1 (L-pp)1 ([]-p))(1 - p),
where the second and the third equality is because (P,, V, V.y.) = O.
Σ JURR, Σ 88380 330 P. V) (w - 3/4 2 (Wy,))
1.1
= 300 300 Σ (Pp WV = 3 33p (np + (1 - p)) Σ IP v: VII
(99)
30000 n(n 1)(mp +11 (1- (1)-)) p),
21

---

Published as a conference paper at ICLR 2026
where the second equality is because y, V. Vi) O.
EURRa
Σ Be (el - P) (P v. v) ((-w + P @ yi + Ia) = 0, (100)
where the second equality is because (P V, yi) and P y. 3/4 =
EU E Be (P V) ((y: + 3P villi @ 14 Ia) V
(101)
= Be EI VI2 B2 Be n(n - 1)(mp + (1 p)) - p),
where the second equality is because P,, V, vi) = and Ps.ys = 0.
Next, we estimate the terms involving U2. We first recall that U2 = Be (P P..l)). Because
Pu, = 0 when i = j. we can just replace e (us with Bop in U2 i.e, U2 Bp (P yy Mj)
Hence,
U LURRU,,0
(102)
because PP = 0 for any i in U.
Σ
R2 Σ
Be Be c (P (P). ((-ys SP pP..))) 2 ys + Ia) ys
i.j
i.
= (0 - )0))
P ((-ys + 3 P
*fj
(103)
=Be200 - (Bp 1)
Σ
|P 1I,
= { - (Bp - 1)(1 - p²)-
where the second equality is because P My + 0 only when + j. on which (ys va) = and the third equality because 9. Mi) O.
Σ RURRUU Σ
Be 3e (w).)))) (Py.,)) ((-y + P u)) 0 Ia)
= 38p
Σ
(BIIP 1/ (1
p)) + (P 9ı) v') (PP++Ppp
(104)
Σ
(B(1 - (2) + (1 - p)) + (1 - (mp + (1 p)))
$$1
= - 1)(mp (1 ()) p²) (1-p)).
where the second equality is because py. * 0 only when + j. on which (2), v)) = p, and the third equality because (P v..9.. M1)
The for Equation (97) is similar to the proof for Equation 99] in Lemma B.8 Notice that when
y < 1-p' we need to pick up terms involving ne and the leading order term in Equation 96)
is the one in the second line of Equation 196,. which is Bn 3Bp p² (1- p); when 1
I-p' after
diving n3 all terms in Equation 966) are oa (1) terms.
Lemma B.10 For terms U1. U2. U3, U4 in Lemma B.6 we have that
Σ U + U2 U3 + 1
(105)
=8228 - 2) p)
[(e0 @0)2 + (c e + (1- p)) + 20 (1 -p))]]
22

---

Published as a conference paper at ICLR 2026
As a corollary, when we pick? 3 y log n, we have the following phase transition limits - +00.
p² p) o, (1)
if < 1
1-p-
U2| +
8'p(1-p((4+30) + on (1)
1 - I-p"
1
(106)
16
0+)
if I-p
Proof 18 (Proof of Lemma B.10 We notice that (U1, U2) because &ij v. M) 0 for any i,
So,
UU, U2l = 62,82 Ppy VII e ..) PP yb 33 1| = 20p - () + (1 p))(1 p) + (1 (1,)))) (22)) p²),
(107)
where the second equality is because 220 |P pu:Uji?? + only if # j. which
280,, "., 2 p²).
|13 + U = (e-- @pppz 23 (e -e wj) + 2 |V|2 - - + 26 (c _p) (1 p)) 2 n + (1 (1-p))
(108)
which is independent j. Hence,
Σ
11U + 12|1311U3 + U+|1
1,
-[8P280n(n 1)(mp + (1 p))(1 p) + n(n 1)32 280 -p²) [1s Uall
(109)
22 1)(mp +2) p)[[U3 + 11l
n2 - 1)(mp 2)(1 p) =
-[(E - 0002 + 20 (np + -1- p))
p)) + e 22p + (1 ((1-p) p))]
The proof for Equation (06)) is similar to the proof for Equation (91) in Lemma B.8
Notice that
when 7 < I-p. we need to pick up terms involving neeP, and the leading order term in Equa-
tion 105 is is S p p); when y T-p after diving nZA all terms in Equation 96 are on terms.
Proof 19 (Proof of Theorem 2.4 As we have mentioned at the beginning of Appendix need to focus the case when 7 S I-p which follows directly from Lemma B.8 Lemma B.9 and
B.2 we only
Lemma B. B.10 We notice that, in these three lemmas, the leading order terms are the same, 3 p
p). which cancels in Equation (86). Hence, when 7 < I-pp V
0 o, When
= 1.
I-p' we also only need to use the corresponding cases in these three lemmas and combine
them in Equation (86) to get the conclusion in Theorem 2.4 One remark is that under Assumption 7 we have that n S implicitly. So, when 7 I-p' terms in Equation (86) involving g² (logm)
also become on (1).
C MODIFIED ASSUMPTIONS WITH MORE MEDIAN PHASES
In this section, modify Assumption phases like Lemma A.I Theorem 2.3 Theorem 2.5 We remark that we only showed the existence of two phases (two extrema) in Lemma A.I] Theorem 2.3] Theorem 2.5] but it doesn't mean under Assumption
that we can prove the existence of three different
2 there is no other transition phase between these two phases (two extrema). Under the following Assumption we can show there are indeed at least three phases. Recall that for any i [1, we defined y/ N (x:).
Assumption
For any E [1, |ax|1 [91, 92) for some positive constants @ S 42.
23

---

Published as a conference paper at ICLR 2026
There is (0, 11, four positive constants P3: P1. K4, with ps Ps. Kig S R4. and P1 1, such that for any E [1,n]. if we define
KG {m + Wma Wi) [lss.
(110)
then we have that
Kid
Kig
S k4.
(111)
For any i E [1,n] and any K.U (yi, Vi) E (p), p2] for some nonnegative constants
Pt, P2 satisfying Pr S p2 < Ps P4.
For technical reason, we further assume that (1 -) 1011-p2 p2) + p2 <
Lemma C.1 Let B Y logn where y is a positive constant. Under Assumption and Equation 5 for any i E [I].
(1+ on())) (Emgx.u()) ettmm)
if < min 1-
1-r
I-p:
PI
Z; = (1+ o,()) (EmEx, eaimm)
if P2
< <9
T-ps
(112)
(1 on)))
if>> max
1-p2" 1-
where the terms on (1) go 0 ass too with speeds independent of but only depending "ts P1, P2, P3: Pa. T, Ri, R4.
Proof 20 The proof is similar to Lemma A.7
We notice that
Z: = e + Σ
+
Σ
Eaim
meK.
męKGU())
(113)
n n (16.9m)
Σ
n (36.9mm)
meK,
m
We also notice that Nzn < KKi] S KAn according to Assumption Hence,
+ < |C,| 7
n (am.8m)
<
Kci 7p I k4n
(114)
meK:
and
(n - KaT2" -) n 7p I
Σ (VV.8m)) S (n |Kil 1) n02 S 1t
(115)
1-T < min 1-p1' P4-P1 }. the leading order term in Z; is Emeriuua n (ussm))) when
When
7
1-7
1-ps' the leading order term in Zi is EmEK, n. (MNoMa)); when Y > I-p2" I-pa S. the leading order term in Zí is n.. also remark that the last ussumption in
Pa PB-P2
max
Assumption 3 is to ensure the existence of the middle phase, i.e. 1-t_ < y < 1-p3" This finishes the prooffr Lemma by similar arguments as in Lemma A.I
A direct corollary of Lemma C.I is the following theorem.
Theorem C.2 Under Assumption 2 and Equation (5) we have the following phase transition phe-
nomena: let 3 log where "y is positive constant. For any i [1,n] the updating dynamics Equation can be written as
Σ ZmekGu(s) etim + (1)
Emek,ute e ym
1-r if < min I-pi 1
x' = 0.0. +
mmex, emimgm ym (1)
Emek,
is -ل 1-t
Ps-P2
<9 <
I-ps'
(116)
3/ + (1)
if max
I-pz'
I-ps
The terms O, (1) represent vectors dd with norms going to as n to0, with speed indepen- dent of but only depending on Y, P1, P2. P3: P4, T, R3,
The proof of Theorem C.2 is similar to Lemma C.1
omit its proof.
24

---

Published as a conference paper at ICLR 2026
D ANALYSIS OF THE P, X Vlggn SCALING FOR I.I.D. GAUSSIAN SCORES
This appendix provides a short heuristic derivation of the 8, x vlog nt scaling for softmax attention modeled independent (0, random variables.
when the raw attention scores d1:-.., On The purpose is to contrast the behavior of this Gaussian setting with the geometric setting analyzed in the main text, where pairwise score gaps remain (()) and the critical scale becomes Bn x log n.
Let 01 01;-.., On be i..dd (0 1) and denote by
aj 2
Z a2
.... ..... 24 a
their order statistics. Set
in = ④
2 2 n,
where is the standard normal CDF. It classical (see Theorem 2.1.1 in De Haan & Ferreira (0066) that for any fixed & 2 2.
(tn(at - .b)....,tt,................ (a - a)) (E1. E1 + Eg,
E + +...4 Es-1),
(117)
where (21) are i.i.d. Exp(1) random variables. Thus the gap scale between neighboring scores in the Gaussian model is 1/ /logn.
Define the softmax weights and the top weight
exp(Bna,)
An(n) == Ek=1 exp(3nax)
At(n) == max Aj(n). 15J5R
Using Equation 117) D. one can write heuristically
(n())
Sk-1 = Fg Fk-1- (118)
1+ expl (B. log )
where we have replaced the finite sum by infinite series, which captures the leading asymptoties. The critical scaling vlog n shows up in Equation (18). For example, in the supercritical regime 3n Vlog n, we have the following:
Proposition D.1 (Supercritical regime) If
Bn
00,
Vlg
then (1 + as cc. In other words, the attention weights concentrate on the top-scoring token.
Proof 21 (Sketch proof) In Equation 118 we know the sum
00
Z
expl (Bn/ (Bn/V/2oggn S
almost surely finite. Since Bn. / logn Dc, every such term in 7 vanishes in probability Hence the denominator Equation (18) lends to 1, and At(n) + 1 as n > xo.
Proposition D.I shows that the softmax enters regime in which single index captures asymp- totically all attention mass as soon as Bn > vlogn. n. The critical scale is thus determined by the requirement
Ba (a - at) = Okk)
Bn log n.
In the geometric models analyzed in the main text (simplex and almost-simplex assumptions), the gaps between the top pairwise inner products remain order 0(1) as m - co. Consequently, the balancing argument in the Introduction section yields the critical scaling 3n x log n. This stands in clear contrast to REM-type Gaussian models, where the top gaps shrink at the 1 1/Vlog n scale and thus produce the critical regime 3n x Vlog n. This explains why REM-type Gaussian models lead to the Vlog n scale, whereas geometric models naturally produce the log n scale that aligns with many practical strategies avoid mixing in long-context attention.
25


Reviews:
[{"review_id": "rhX1UDER82", "reviewer": "Reviewer_UoUF", "summary": "The paper presents a rigorous theoretical analysis of attention scaling in long-context transformers. By studying a simplified but informative self-attention model, the authors show that the system undergoes a sharp phase transition determined by a critical scaling factor $\\beta$ proportional to $log(n)$. This result provides a clear theoretical explanation for the empirical success of scaling strategies used in recent models such as YaRN, SSMax, and SWAN-GPT. The analysis addresses both the contractive properties of the forward pass and gradient behavior in the backward pass, demonstrating that this critical scaling is necessary to preserve meaningful token interactions and maintain stable training dynamics.", "questions": "- The phase transition boundary depends on geometric quantities such as the inner product ρ and the norm q. The simplex case makes the nature of this dependence particularly transparent. Recent work has shown that layer normalization can systematically influence token norms over depth (e.g., “Normalization in Attention Dynamics,” arXiv:2510.22026). This raises the question of whether an adaptive scaling factor, adjusting $\\beta$ based on observed per-layer statistics of token norms or pairwise similarities, could maintain the model closer to the critical regime and potentially improve performance?\n- Assumption 2 requires a positive lower bound $\\rho_1 > 0$ on pairwise inner products. While the upper bound $\\rho_2 < 1$ is intuitive, the necessity of a positive lower bound is less obvious. Is this bound essential for the phase transition to appear, or is it primarily a technical condition to simplify the proof? Additional intuition here would be helpful.\n- The analysis is conducted under the simplifying assumption $Q = K = I$, which is a standard and effective choice for achieving tractability and is consistent with related theoretical work. As a curiosity, do you observe the same phase transition behavior in numerical experiments when $Q$ and $K$ are non-identity matrices, for example in the case where $Q^TK$ is symmetric positive definite? If so, does the location of the phase transition or the strength of contraction change in a predictable way?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The main strength of the paper lies in the quality of its theoretical analysis. The results are rigorous and presented in a way that remains accessible. The work provides a direct and mathematically grounded explanation for a technique currently used in state-of-the-art large language models, which is a valuable and rare contribution. Moreover, by isolating the mechanisms that lead to performance degradation in long-context tasks, the paper offers a framework that can guide the development of more targeted scaling strategies.", "weaknesses": "The model necessarily incorporates simplifying assumptions, and the experiments are correspondingly limited. However, given that the goal of the work is to isolate and explain the underlying scaling phenomenon, already well studied by practitioners, rather than to propose a new architecture or strategy, this scope seems appropriate. The limitations are acknowledged and do not undermine the core contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZAyLQXFYav", "reviewer": "Reviewer_bDMM", "summary": "As language models have grown, context lengths have increased while signal propagation in attention layers has degraded; phenomena such as rank collapse have been observed. Empirically, prior work tunes the inverse temperature $ \\beta $ using a $(\\log n)^k $ scaling (with $k>0$ varying across papers). Under a strongly simplified model (all attention weights set to the identity), this paper analyzes tractably when a phase transition at $ \\beta=\\gamma\\log n $ occurs and characterizes its connection to rank collapse.", "questions": "* **Size of the top set.**  Because of $K=Q=I$, we get exactly one large  socre, others small scores after applying softmax with the Assumption 1.  How is Assumption 1 justified for self-attetoin with $K=Q=I$? Prior citations may support it, but a fresh statement of the authors’ view would help.   Any empirical evidence that appearance of exactly one large score in deep model (initialization/pretrain/finetuning phase) would also be helpful.\n\n* **Consistency of $ \\log n $ vs  $\\sqrt{\\log n}$.** Which specific assumption differences (e.g.\\,correlation structure, sparsity/density/ layer norm) could determine the order of the critical scale? \n* **Robustness to assumptions.** For non‑identity weights, to what extent do conclusions (including near‑ties) preserve the phase diagram and the critical behavior? Under the stated constraints, in what sense does the work clarify the claim that “logarithmic scaling maintains sparse, content‑adaptive attention at large context lengths”? Also, how do the authors explain $ (\\log n)^2 $ (YaRN) scalings?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The angle‑parameter characterization is clear, and comparing to $1/(1-\\rho)$ gives a natural condition for the phase transition.\n* The numerical simulations align with the theoretical predictions and provide empirical support.", "weaknesses": "* **Insufficient precision in comparisons to prior work.** The introduction distinguishes the REM setting as “dense,” but that distinction applies at the $a_{ij}$ stage; at the $A_{ij}= \\exp(\\beta a_{ij})$ stage, the  behavior can become sparse, so the separation is not clearly justified. The discussion is thin on why REM suggests  $(\\log n)^{1/2}$ while the present model yields $\\log n$.  \n* **Strength of assumptions.** The score assumption,  exact one dominant entry, others equal (off‑diagonals fixed at  $\\rho$ before normalization),  abstracts away near‑ties that can naturally occur in self-attention in real data: in language inputs the same token (e.g., a subject) can reasonably appear multiple times in one sequence.  This also stems from assuming the weights are identity mappings; In the REM case,  generally, the number of  near-tie entries can be more than one.\n* **Gap to implementation.** Evidence is limited for how conclusions transfer to systems with learned $(K,Q,V)$. It remains unclear which architectural components would break the conclusions; in particular, the connection between the observed phase transition here and constants or regimes discussed in prior work (e.g., $C$ of $\\beta=C \\log n$ in Table 1) is weak.\n* **Independence of numerical validation.** Experiments closely mirror the theoretical assumptions, limiting their independence as evidence for practical application. Stability of the critical behavior under slight deviations from the assumptions and reproducibility in more implementation‑like settings are underexplored. (e.g. incleasing the number of the near-tie entries)\n* **Tone of claims.** While the $ \\beta\\asymp\\log n $ phase‑transition picture is clear, the evidence is not yet sufficient to claim “clarifying  why logarithmic scaling maintains sparse, content-adaptive\nattention at large context lengths.” for general practical attention scaling in my current understanding. The scope and limits of applicability should be stated more explicitly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "87c3Ym3LX0", "reviewer": "Reviewer_mEek", "summary": "The paper analyzes a long-context self-attention layer in which the attention scores are scaled by a factor that grows with the logarithm of the context length. In a simplified pre-normalization setting where the key, query, and value matrices are all equal to the identity, the authors prove that this scaling produces a clear phase transition as the context length increases. In the subcritical regime, when the scaling factor grows too slowly, attention weights become almost uniform across tokens, leading to strong contraction and rank collapse.\nIn the critical regime, when the scaling grows at just the right rate, attention focuses on a small but non-trivial subset of tokens, resulting in sparse and content-adaptive mixing. In the supercritical regime, when the scaling grows too quickly, attention effectively reduces to the identity and cross-token interactions vanish.", "questions": "- Do you view the phenomenon described here as primarily length-driven token uniformity (present even in a single layer)? How does this relate to depth-driven rank collapse? If logarithmic scaling is applied, is the length-driven component largely neutralized, leaving only depth-driven collapse? Are there diagnostics that could distinguish the two mechanisms in trained models?\n- Your Theorems 2.3 and 2.4 show phase transitions in Jacobian norms, strongly reminiscent of the vanishing-gradient and over-smoothing analyses in graph neural networks (Arroyo et. al. 2025). Can your results be interpreted as the transformer analogue of these phenomena? Do you see conceptual parallels between the two settings?\n- Recent work on attention sinks argues that attention sinks emerge as a way to prevent mixing. Do you agree with the interpretation that such sink tokens act as a compensatory mechanism to stabilize mixing when attention heads operate near the subcritical regime? \n\nArroyo, Álvaro, et al. \"On vanishing gradients, over-smoothing, and over-squashing in GNNs: Bridging recurrent and graph learning.\" arXiv preprint arXiv:2502.10818 (2025).\n\nBarbero, Federico, et al. \"Why do LLMs attend to the first token?.\" arXiv preprint arXiv:2504.02732 (2025).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "Although I am not too theoretically grounded myself, I very much liked this paper and I hope some of my comments/suggestions are useful to improve the paper. I have added some strong points below.\n\n- The paper is mathematically strong and offers a well-grounded analysis of a practically important issue of rank collapse in long-context transformers.\n- The key result is simple yet powerful, providing a clear and interpretable explanation of why logarithmic scaling works and what happens when the scaling constant is too small or too large.\n- I particularly like that the analysis connects both forward and backward dynamics. The insight that vanishing gradients appear in the subcritical regime helps explain and even motivate architectural design choices such as residual connections and normalization layers in transformer blocks.\n- The work successfully bridges theoretical and practical perspectives, with immediate implications for stable long-context training.", "weaknesses": "- As in many theoretical studies, the analysis operates in a simplified regime (pre-norm, identity key/query/value matrices, no positional encoding, no multi-head or causal masking). This is understandable but limits direct applicability.\n\n- Experiments are synthetic and single-layer; there are no demonstrations on pretrained or small-scale real models. Even a lightweight validation could have been an interesting addition, though I recognize that this is beyond the paper’s main focus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UpR5wFgH8u", "reviewer": "Reviewer_W7uw", "summary": "This paper attempts to justify the choice of attention scaling factor used in long-context scenario from the theoretical perspective. Specifically, the authors identified that with scaling choice $\\beta_n=\\gamma\\log n$, the attention dynamics demonstrate a transition phase.  To delve into this question, the authors adopt a simplified self-attention model (without parameterisation) for theoretical analysis from both forward/backward perspective under two separate assumptions. The final numerical analysis justifies the derived theory.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-motivated that the choice of $\\beta_n$ scaling in Qwen, SSMax, SWAN-GPT ($\\log n$) is not justified from the theoretical perspective. To address this question, the authors derive their theory on a simplified self-attention network, which is more tractable from the theoretical perspective. The authors consider both forward and backward two directions, which are validated through numerical studies. I believe the theoretical framework could be beneficial to the community.", "weaknesses": "Since the main paper is built on the theoretical analysis on an over-simplified self-attention network (without parameters), the generalisation of the conclusion to the realistic transformer model remains unknown. I have concerns about two main issues in the theoretical framework:\n\n1. This paper still discusses self-attention under the scope of LLMs, which are assumed to adopt causal mask in the attention computation. However, this paper seems to completely ignore this point in the proof (from proof 1 that the denominator is independent of the choice of $i$). I am aware that the authors are trying to understand the attention scaling by simplification, but I am wondering whether the conclusion is still valid considering causal mask in the framework.\n\n2. Another practical consideration in LLM is the existence of attention sink [1], which means the first token has large attention score while its value state is almost zero [2, 3]. Additionally, [3] claimed that this is a way to mitigate over-smoothing in long context scenarios. I am seeking for the authors' opinion towards this as this challenges some assumptions in this paper, e.g., the assumption on $a_{ii}=1$ and $a_{ij}=\\rho$.\n\n\nReferences:\\\n[1] Xiao et al. Efficient streaming language models with attention sinks. ICLR 2024.\\\n[2] Gu et al. When Attention Sink Emerges in Language Models: An Empirical View. ICLR 2025.\\\n[3] Barbero et al. Why Do LLMs Attend to the First Token? COLM 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)