Limitations of Normalization in Attention
Authors: Timur Mudarisov
Mikhail Burtsev
Tatiana Petrova
Radu State
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Theory verification
GPT-2
Attention mechanism
Natural Language Processing
Primary area: deep_learning

Abstract: This paper investigates the limitations of the normalization in attention mechanisms. We begin with a theoretical framework that enables the identification of the model's selective ability and the geometric separation involved in token selection. Our analysis includes explicit bounds on distances and separation criteria for token vectors under softmax scaling. Through experiments with pre-trained GPT-2 model, we empirically validate our theoretical results and analyze key behaviors of the attention mechanism. Notably, we demonstrate that as the number of selected tokens increases, the model's ability to distinguish informative tokens declines, often converging toward a uniform selection pattern. We also show that gradient sensitivity under softmax normalization presents challenges during training, especially at low temperature settings. These findings advance current understanding of softmax-based attention mechanism and motivate the need for more robust normalization and selection strategies in future attention architectures.

Limitations of Normalization in Attention Mechanism
Timur Mudarisov
Mikhail Burtsey London Institute for Mathematical Sciences ab@lims. ac uk
University of I Luxembourg
Luxembourg
timur. www.ur..ddarissvveee
Tatiana Petrova
Radu State University of Luxembourg
University of Luxembourg
Luxembourg Ratiana.petrovaauue 1u
Luxembourg radu. stateeuni.
Abstract
This paper investigates the limitations the normalization in attention mechanisms. We begin with theoretical framework that enables the identification of the model's selective ability and the geometric separation involved in token selection. Our analysis includes explicit bounds distances and separation criteria for token vectors under softmax scaling. Through experiments with pre-trained GPT-2 model, we empirically validate theoretical results and analyze key behaviors of the attention mechanism. Notably, we demonstrate that as the number of selected tokens increases, the model's ability distinguish informative tokens declines, often converging toward uniform selection pattern. We also show that gradient sensitivity under softmax normalization presents challenges during training, especially at low temperature settings. These findings advance current understanding of softmax-base based attention mechanism and motivate the need for more robust normalization and selection strategies in future attention architectures.
Introduction
The attention mechanism [1, 7, 17, 8] has become fundamental component of modern deep learning. Since its popularisation in the Transformer [18], attention has powered state-of-the-ar systems in machine translation, text generation [3], and multimodal reasoning [11]. Yet the same softmax rule that enables differentiable "focus" also introduces chronic failure mode: as the context length L grows, attention weights collapse toward 1/L, a phenomenon we call vanishing atlention. The resulting gradients are too small for effective learning, especially in long-context settings [4, 21J. Architectural work-arounds sparse windows, locality-sensitive hashing, compressed memories- reduce compute but do not climinate the collapse, and may even worsen it [5, 16, 2. 10]. principled understanding of why softmax fails this regime is still missing.
Our view: attention as capacity-limited retriever (selector). We revisit normalisation in attention from first principles and show that softmax, and indeed any length-independdet normaliser, possesses an intrinsic capacity limit. Our contributions are:
1. Distance bound. We derive a non-asymptotic upper bound the representation distance between selected and non-selected tokens (Theorem 1). The bound proves that once the top-N set grows proportionally L., the distance must collapse, formalising the "softmax bottleneck" [20].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2. Geometric bound. Under mild spherical assumptions we show that no more than = 80% of the top-N tokens can be simultaneously distinguished in Euclidean space (Theorem 2). This quantifies hard limit on what single head can represent.
3. Gradient bound. We bound the Jacobian norm of general normaliser (Lemma 2); specialised to softmax recovers the classic 1/(4T) instability and shows why aggressive temperature scaling trades separability for optimisation difficulty.
4. Empirical validation. Experiments on GPT-2 [15] confirm all three predictions: distance collapse, separability saturation, and I/T gradient growth.
Our analysis frames attention selector with finite resolution: it works well while the active set is small fraction of the context, then degrades predictably. This viewpoint explains the empirical success of recent Serrratives-SSSSsemmaaxa Scalable-Softmax, Self-Adjuste Softmax-am suggests concrete design rules we distil in the discussion. More broadly, our theoretical tools provide diagnostics for deciding when head has reached its intrinsic limit and when architectural or normalisation changes are warranted.
By bridging closed-form theory with large-scale experiments, provide both deeper understanding of normalisation in attention and practical guidelines for building robust, long-context Transformer models.
2 Theoretical Analysis
Consider a sequence of token embeddings X = (x;)/=z where each embedding x; E R is d-dimensional vector. We begin by reviewing the classical self-attention mechanism introduced by Vaswani et al. [18]:
qm fa( m), kn, (Tn, n)), Vn = fu nt),
where Qms kn, and vn denote the query, key, and value vectors, respectively. The attention weights are computed using these query and key vectors as follows:
Om,n = exp (q /T)
Σ exp (qnk;/T)
where the parameter T is known the temperature, typically set to T = vd (as recommended in [18]). To extend the scope of our analysis beyond the standard softmax normalization, we introduce more general normalization framework:
am,n F(q kn, 0)/ ///2 F(qTTky,0)
where F; I- - I is smooth positive function parameterized by 0, which can include parameters such as temperature or the number of tokens. For convenience, we denote the inner product q k,, as kn,,, referring to it as the logit associated with the token pair (m, n).
In this work, we provide detailed theoretical examination of the general normalization framework introduced in Equation (3). We focus several eritical aspects: (1) general limitations associated with softmax-type normalization; (2) the influence of normalization on token separation; (3) geometric insights into token separation; (4) connections between normalization and training dynamics.
We begin by addressing the general limitations of softmax normalization. As highlighted by [19], one critical limitation is the phenomenon of vanishing attention weights. Intuitively, when the number of tokens L increases, the normalization procedure (e.g., softmax) distributes attention weight across many tokens, causing the weights for individual tokens to become extremely small. This issue hampers the model's 's ability to clearly differentiate between relevant and irrelevant tokens.
Formally, consider set of logits .4.) and their corresponding attention weights (an,..., ). computed from the logits using a normalization function. We present the following general results regarding these normalized attention weights (see Appendix A for full proofs):

---

Lemma 1. Consider the normalization scheme defined by Equation (3) with a smooth function F(I,,) B) that does not explicitly depend on the number of tokens (i.e., L & 8). Assume also that the logits are bounded, 1 E [-a, a). Then, the normalized attention weights satisfy:
C1
C2
L 5a, L
where the constants C₁ and C2 do not depend on L.
This lemma implies that for any normalization of function independent of token count, the attention weights inevitably become uniformly small (on the order of 1/L) as the context size grows. As a result, the mechanism loses the ability to effectively highlight specific important tokens when processing long sequences. The proof relies F(I,,)) on the compact interval [-a, a). Due to these properties, both the numerator and denominator bounded by constants independent of L. Hence, when normalized by the summation tokens, each weight naturally scales as L/L. Full mathematical details are provided in Appendix A.
the boundedness and continuity of the function
all L
Corollary In the special case of softmax normalization (Equation (2)), with temperature parameter T, the attention weights satisfy:
2A T S a; S exp
1
2A
exp
(5)
where A |//////k||
This corollary highlights how the magnitude of the query and key vectors directly impacts the distribution of attention weights. Specifically, the bounds show that unless the vectors have sufficiently large magnitudes (relative to temperature 7), the attention weights remain close to uniform. Thus, increasing vector magnitudes partially mitigate- but not eliminate- the vanishing attention problem.
These results demonstrate fundamental limitation of softmax-type normalization: the sequence length L grows, each attention weight a; shrinks toward O(1/L). In turn, the model struggles to assign appreciably larger weights to genuinely informative tokens, blurring the distinction between relevant and irrelevant embeddings. This loss of focus impairs the model's 's ability to exploit salient information within long input sequences and ultimately hampers effective learning.
2.1 Distance analysis
We next study how normalization influences the total deviation the discrepancy between learned representation and the underlying conditional distribution. Yang et al. [20] describe the softmax bottleneck: the low rank of the logit matrix limits the model's ability to represent the true conditional distribution, thereby increasing total deviation. This deviation is not a consequence of insufficient model capacity or sub-optimal optimization, but rather of restrictions imposed by the final softmax projection layer.
To quantify the role of normalization in separating informative from non-informative tokens, let {ar..... ,a,} and X (II,...,,a) H; Rdd denote the attention weights and their corre- sponding token embeddings, respectively. We focus on the top-N tokens with the largest weights; their indices are collected in IN (i1, (i1,..,,NN)) The context vector derived from these tokens is
Σ a4 Zt.
(6)
EEIN
Our quantity of interest is the cumulative distance between this context vector and all non-selected embeddings:
d = d X s)
11a: - =//2:
VEIVIN
where / - (I,...,,,,, || 1112 is the Euclidean norm, and X = (Zi....., } is the set of selected embeddings. Intuitively, smaller value of d indicates that many low-weight tokens lie close to the high-weight aggregate 8, signalling reduced separability and a greater likelihood of attention "dilution."
3

---

Next, we distinguish two settings for the weight vector {a;). In the first, the weights are treated as fixed; in the second, we analyse random- random-selection selection scenario which the index set IN is drawn
uniformly at random from all subsets of size N: IN U N), 1 {1 ....... J.J. In this random case, the attention weights on the non-selected tokens vary with the draw of Iv. We seek the expected cumulative distance
E Ein [d]
(8)
where the expectation is taken over the uniform choice of IN.
We now adopt a classical metric-learning perspective: the above quantities quantify the Euclidean "separation margin" between selected and non-selected token embeddings. The next theorem for- malises these observations by providing explicit upper bounds the distance d in both the fixed and random top-N settings.
Theorem 1. For the representation distance defined in (7) the following bounds hold.
1. Fixed Nop-N set. If IN is fixed,
d $ (1 āN) d1 + max
[ā -1
-1 11-1111m -1-[l]]]]
jeIx
where d1 max iglx, JEIN
|2, 22ll2 and ēN
Σ as-
SETN
L 2. Uniformly random top-N set. When IN is sampled uniformly from all size N,
subsets of
N
L
L- N
E
(as
I vi - - 8,
L
where Σ a.s and E S ( L
1
N
E3 13
Σ
L |1,02-- - E 2225
Intuitively, for a fixed top-N set, the first term scales the largest out-of-set distance dj by the total weight (1- āN) carried by the remaining L- N tokens, while the second term accounts for how much those low-weight tokens can still perturb the context vector, proportional to the norm of the largest selected embedding. When the top-N indices are chosen uniformly at random, each token excluded with probability (L (L-N)/ L. Replacing the random indicator variables by their expectations yields the main sum; the residual æ captures the Jensen gap and small whenever the individual weights og are not too concentrated. Full derivations are provided in Appendix A.
Theorem relates the distance d to key parameters (N, I,, and the weights o:). To see how the bound behaves in two extreme regimes, we state the following corollary. Corollary 2.
(i) Fixed N < To When N is held constant and L grows,
E E Σ
|aa2,112-
(ii) Fixed L, N + I. T fixed and the top set expands to the full sequence, i.e. N - T,
E 0.
When N small relative to the sequence length, most tokens are excluded and the expected distance dominated by the individual contributions ||o,æil|2 of those low-weight tokens. In contrast, N approaches I., the context vector eventually incorporates all embeddings, so the distance between and the (now empty) set of non-selected tokens vanishes.
2.2 Geometrical interpretation
We now recast the analysis geometric terms, focusing on the spatial arrangement of the token embeddings zg. When these vectors lie close together, the model struggles to separate informative from non-informative tokens, hindering training. To quantify how many tokens can be reliably distinguished, we work under two standard geometric assumptions:

---

Assumption (Uniform spherical distribution). Each embedding lies uniformly on d- dimensional sphere of radius M:
24 (M)).
where (M)) denotes the (d - -pphere of radius M. In practice, normalize embeddings so that they lie on this sphere.
Assumption 2 (Minimum pairwise separation). between any two embeddings:
There exists fixed lower bound on the distance I|=;
min bデ
xy xyارa 8 0,
ensuring that two tokens collapse onto another. In experiments, set ō to the empirical minimum pairwise distance.
Under these assumptions, we can bound the number of embeddings that fall within specified neighbourhood of the context vector, thereby quantifying the model's effective resolution.
Let IN = ..i,..., J denote the indices of the N selected tokens and recall the context vector
Σ
EENN
Fix tolerance radius >> 0 and consider the closed Euclidean ball (s) centred at We say that selected embedding x; is geometrically distinguishable its rescaled version a,z2; remains within (s) while every non-selected embedding satisfies ayæ, & (s)s) for j & IN. The count of such distinguishable embeddings is
N₂
# E IN |a,2 s|12 S rr)
(10)
and the ratio N. / measures the fraction of "non-noise" vectors the model can reliably separate (see Fig. I for an illustration).
Our objective bound the expectation N,N,] and hence provide explicit upper and lower limits on the model's effective resolution as a function of r, N, and L under Assumptions 1-2.
Initial points on circle
Points after attention waights
Final selection of acints
0.4
0.2
0.05
0.55
0.00
-0.05
Selectted pai
Non-sellccted points Represencative poirt
-0.10
Selected points
Non selected points
-0 0.10
Selective srea
-0.4
-0.2
-0.00
0.03
-012
0.00
0.00
0.30
Figure 1: Illustrative example of the geometric separation. Left: Token embeddings lie circle. Middle: After scaling by their attention weights a,, both attended (magenta stars) and non-attent non-attended (black dots) points move toward the origin. Right: Only the selected tokens that remain inside the ball (s) (shaded) are deemed distinguishable.
In the previous section, examined separation in terms of Fuclidean distances. The geometric approach introduced here, by contrast, focuses on directional separability and reframes the problem one of metric learning. Now, present the main result (for the proof, Appendix A):
Theorem 2. Under Assumptions 1-2, the fruction of geometrically distinguishable embeddings satisfies
E BNN
Σ TENN
Σ expl
(r-g.)
(11)
NN
N
N
16M2
SETN
5

---

where
= =
Σ oj +
M2 d²
Σ
0j0kk
(12)
JEIN
J.KENN
jøi
The quantity & measures how widely the other N - selected embeddings, rescaled by their weights, are spread around z;. &, is small, most selected points cluster near the context vector s, so many of them fall inside the ball (s) and become distinguishable. The lower bound in (11) subtracts from one a penalty proportional to the cumulative spread
Σ
EGs the tighter the cluster (smaller {e), the
closer the ratio E ///N is to Conversely, the upper bound shows that once r smaller than & for many i, the exponential term decays rapidly, implying very few embeddings remain separable under the chosen radius.
2.3 Gradient Sensitivity of Attention
The results above show that language model must sharply distinguish informative from non- informative tokens; in other words, the attention weight distribution should be selective as possible. Refining that distribution, however, exposes second difficulty: gradient sensitivity during training.
Consider nearly identical logit vectors
( (0. ... 0, a, a E),
((2) (O, (O......, a 22, a),
(2) be the corresponding softmax weights (Equation (2)) and denote by Via the Jacobian of the softmax map. A first-order expansion gives
so that their Euclidean distance satisfies (|e(2)
((2) 12 vBe. Let a (1) aa
|a - | a2
V/ (00-)(0)1 /
because the largest two logits swap order and the associated softmax gradient scales as 1/T. Hence, even though the logits differ by only O(e), the output distribution can change by O For sufficiently small temperature T' (a common tactic for sharpening attention) this factor becomes large, making the gradient step volatile and potentially destabilising optimisation.
This example highlights fundamental trade-off: stronger normalisation (smaller T, any variant that steepens the softmax) improves token separability but simultaneously amplifies gradient variance, complicating training of deep transformers.
Let have a closer look at how normalisation affects gradient descent. For the general scheme in (3), let (4 be the logits and {a the resulting attention weights. We characterise the magnitude of the Jacobian V.a.
Lemma 2, With the notation above,
||Vaol12 l2 S min {
|F |1F'112 (min F(1,,0) + L min, F² F2
v2
(13)
where [F|2 max F(4, 0) and [FF max FF [F 0)
The upper bound grows when any logit 1 pushes F(1,, 0) toward zero, because both terms inside the parentheses scale inversely with min F(1,,)) Thus, a sharply peaked normalisation softmax with
a small temperature, for instance, induces large Jacobian norms, signalling high gradient sensitivity during training.
Corollary 3. For the softmax normalisation in (2),
|| < min
47 V2
A smaller temperature 7 sharpens the softmax but simultaneously inflates the Jacobian norm (scaling as 1/T), making the attention distribution highly sensitive to even tiny logit perturbations and thus harder to train stably.

---

Corsariion ditartow fiwwed
Campurion ofweer
Thue distance
Expectation distance
Upper bound distance
oi estimetors
Vilue of astimatoos
True distance
Expectation
Upper bound distance
(a)
(b)
Critical line
92
42
N1
(c)
Figure 2: Distance statistics validate Theorem 1. (a) With N 5, both the true distance (green) and its expectation (blue) grow roughly linearly in L; the red upper bound is safe but conservative. (b) With L 1024, increasing N beyond 20 yields diminishing returns: the distance plateaus while the bound tightens. (c) Critical top-N obtained by a KS test (a = 0.01); fewer than 6 % of the tokens need be selected before the empirical and expected distances become statistically indistinguishable.
3 Experiments
We evaluate our theoretical findings on the publicly available GPT-2 model family' [15]. All text is tokenised with byte-pair encoding (BPE) [6] as implemented in the Hugging Face transf ormers library. Unless otherwise stated, the input consists of consecutive chapters from War and Peace by Leo Tolstoy (public domain), providing long-form prose well beyond the model's context window. For every layer and attention head, we extract the full attention matrix A A and the associated query, key, and value tensors, enabling direct comparison with our distance and geometry metrics. Implementation details, hyperparameters, and reproducibility scripts are included in Appendix B.
3.1 Distance analysis
We now test the non-asymptotic bounds of Theorem Implementation details appear in Appendix B. Two complementary experiments are performed:
1. Scaling with sequence length. Fix N 5 and vary L E {32..... 1024].
2. Scaling with top-N. Fix L 1024 and vary N E {1,5, 10, 20, 100).
'We report results for the 124 M parameter version; qualitatively identical trends were observed for larger variants.
7

---

For each configuration we compute, across all 144 GPT-2 heads/layers, (1) the true distance d (7). (ii) the expectation term of Theorem I, and (iii) the analytic upper bound. In addition, we estimate a critical top-N N value: the smallest N for which the empirical and expected distance distributions are indistinguishable under a two-sample Kolmogorov-Sminn Smiinoo test (a- = 0.01).
Results summarised in Fig. 2 support following key observations. (i) For NN L the distance scales linearly with sequence length, exactly as Corollary 2(i) predicts. (ii) As N approaches 100, the true and expected distances converge while the upper bound tightens-
for the N + L collapse of Eq. (9). (iii) The critical-N curve grows sub-linearly (== 0.06L), confirming that only small subset of tokens can be separated before attention behaves as if weights were uniform.
These empirical trends corroborate the theoretical claim that softmax normalisation retains discrimi- native power only when the active set is small fraction of the context; larger N values chiefly add noise.
Comparison seporation number gifferent
3.2 Geometric Separability
We now quantify how many of the N highest-weight kens remain geometrically distinguishable according to Theorem For each sequence we set r min ||s ig
a;2;|12, so that every non-selected token lies outside the ball (s). Using GPT-2 embeddings normalised as in Assumptions 1-2, we compute the empirical ratio N,/N (Definition (1 (10)) across all heads and layers and compare it with the analytic bounds of Theorem 2; results are sum- marised in Fig. 3.
Upper
Across the full model, the proportion of distinguishable tokens declines up N x 16 and then saturates between 0.7 and 0.85. The exponential upper bound tracks the empirical maxima closely, showing the theorem is tight on the high side, whereas the lower bound is intention- ally conservative. Thus-even with idealised spherical soooodddinn-sfftmnnn attention cannot cleanly separate more than about four-fifths of the tokens it selects; adding further tokens mainly dilutes the representation without improving geometric resolution.
Figure 3: Geometric separability satu- rates at 70-85%. For increasing top- N, the empirical fraction of distinguish- able embeddings N./N (green boxes) quickly plateaus; roughly one-fifth of se- lected tokens remain outside (s).
Meximam of [Ssofmmasii fc on different heads-layer
3.3 Gradient sensitivity
12
We next explore how the softmax temperature T and logit perturbation scale affect the stability of the attention map. For each head-laye pair we evaluate the fiftt-ddifferene Jacobian norm
10
10
Eps.-0.001
10-
(TT,)) = 1 -laetsse - ag/.
-
||a||2 = 1,
E10
Eps.=0.1
Upper bound
which approximates || [|2. Full implementation details are provided in Appendix B. Figure 4 shows the maximum value of g(T, across all 144 heads/layers of GPT-2 for E {10 3 10 11111,1000 100).
Twd10001
Ted
Tel
Te20
Twi0
Figure 4: Gradient sensitivity de- cays as 1/T. Maximum finite- difference Jacobian norm gT,,)) for three perturbation magnitudes (coloured curves, log-log scale). The dashed black curve is the theoretical bound min(1/(4T), v2) from Corollary 3.
For T' 0.1 the empirical curves follow the theoretical 1/T trend of Corollary 3; smaller E values yield larger gra- dients because the sharpest logits dominate the variation.
Once T>1 1 all curves collapse and drop by two orders of magnitude, indicating much improved robustness to logit perturbations but a concomitant loss of selectivity. These findings confirm the trade-off already highlighted by our theory: sharper softmax improves token separability yet
8

---

inflates gradient variance, whereas higher temperatures stabilise training at the cost of blurrier attention.
4
Discussion
Softmax remains the default normalisation in modern Transformers because is simple, differ- entiable and lends probabilistic interpretation to the weights [18]. Its limitations, however, now well established: it produces dense distributions, its peak value drops as the context grows monishinng-lleention effect), and its Jacobian explodes when the temperature is driven low, yielding unstable gradients. Three main research lines have emerged to address these issues.
1. Sparsity-inducing rules such as Sparsemax and a-Entmax replace the softmax exponential with projections onto a simplex, producing exact zeros and lowering entropy [12].
2. Length-aware rescales such Scalable-Sfttmax scale logits by log L so that the maximum attention weight stays roughly constant as the sequence grows, alleviating the vanish- ing- ing-attention problem [13].
3. Gradient-controle variants (e.g. Self-Adjusted Softmax) re- re-weight logits according to the layer-wise dynamic range, keeping the Jacobian spectrum a healthy band [22].
Our work complements these efforts by explaining, in closed form, why the above modifications are necessary. Where previous papers introduced new rule and showed empirical gains, derived non- asymptotic bounds (Theorems 1-2) and gradient limits (Lemma 2) that apply to any normalisation function F(. 0). In particular:
The distance bound shows that when grows proportionally to L, the representation distance d necessarily collapses to zero, formalising the empirical softmax bottleneck noted by [20]. Our GPT-2 experiment (Fig. 2, middle) confirms the predicted plateau and justifies why sparse or length-aware rules can improve long-context performance.
The geometrie bound states that even under optimistic spherical assumptions no more than x 80% of the top-N tokens can stay inside the selective ball (Fig. 3). This explains why empirical studies report that multiple heads are required to cover distinct parts of the context: a single head cannot make every "important" token simultaneously salient. The reason behind this idea is pretty simple. Assuming the independence of the heads as attention mechanism parts, we can conclude that for given separability level H heads can cover up to 1 - (1 p) p)" of the top-N tokens. Under our analysis, we see that for 0.8 it is needed approximatelly H = 3 to cover up the 99% of information.
The gradient bound recovers the familiar 1/(47) law but also shows how any normaliser with a vanishing minimum value will inherit the same instability. Figure demonstrates that GPT-2 operates close to the theoretical limit when T < 10-1, validating the usefulness of gradient-controlle variants such as SA-Softmax.
The combined encory-experiment picture suggests three practical guidelines.
1. Keep the active set small. The critical-N curve in Fig. 2 grows roughly like 0.06L; selecting more tokens yields vanishing returns and erodes separability. Top-k or sparse attention should be preferred when k.
2. Monitor atfention entropy. A rising entropy or a drop in the empirical N./N ratio is an early sign that head has saturated its geometric capacity; adding additional heads or switching to length-aware normaliser can restore separability.
3. Avoid overly sharp softmax. Lowering T below 10-! increases Jacobian norms without increasing separability (Figs. 3, 4). Practitioners should instead use normalisers that decouple selectivity from gradient health (c.g. Sparsemax, SS-Max. or SA-Softmax).
Our analysis assumes embeddings are priori L2- 12-normalised and roughly isotropic. Real models may violate these assumptions, and future work should extend the geometric bound to non-spherical distributions. Another promising direction is to design length-adaptive, gradient-controlled nor- maliser that inherits the best properties of Sparsemax (sparsity), SS-Max (length awareness) and SA-Softma (stable Jacobians) while admitting proofs analogous to Theorems 1-2.

---

Conclusions
This work provides theoretical and empirical analysis of normalisation in attention mechanisms beyond the classical softmax. We derived two non-asymptotic bounds (Theorems and 2) that link token separability sequence length L, selection size N, and the embedding geometry, and established general Jacobian bound (Lemma 2) that explains the well-known trade-off' between sharpness and gradient stability. Experiments on GPT-2 confirmed all three predictions:
the representation distance collapses once N grows proportionally to L;
no more than R 80% of the selected tokens can be geometrically distinguished, even under ideal spherical embeddings;
the empirical Jacobian norm tracks the theoretical 1/(4T) law, saturating at low temperature.
Taken together, these results recast softmax attention selective but capacity-limited aggregator: discriminates well only while the active set is a small fraction of the context. The analysis also clarifies why recently proposed normalisers-Sparseee Softmax-offer complementary benefits: they relax one or more of the intrinsic limits quantified here.
Scalable- Softtmx, and Self-Adjusted
Practical takeaways. (1) limit the top-k set to a sub-linear function of the context length; (2) monitor attention entropy or the N₂/N ratio during training; and (3) prefer length-aware sparsity- inducing normalisers over aggressive temperature scaling.
Overall, our experiments shows several limitations providing the practical diagnostic idea:
1. When the level of geometrical separability drops to 70 - 80%, this signifies that the head has saturated its geometric capacity.
2. The temperature limitations shows that making the distribution sharp makes the Jacobian norm exploding. The teoretical and practical analysis shows that it's better avoid using the T < 0.1.
Future directions. Immediate extensions include (i) relaxing the spherical- assumption, (ii) analysing multi-query and multi- interactions, and (iii) designing a single normalisation rule that is simultaneously framework introduced here will serve as a benchmark for such developments and, more broadly, for principled improvements to long- context transformers.
sparse, and stabee. We hope the quantitative
Overall, the present study provides theoretical footing for the growing body of work that modifies softmax, and furnishes quantitative tools for diagnosing when given attention head has reached its intrinsic limit.
References
[1] D. Bahdanau, K. Cho. and Y. Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 100iv:1409.0477 0473, 2014.
[2] Beltagy, M. E. Peters, and A. Cohan. Longformer: The long-document transformer, 2020.
[3] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss. G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray,
B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei.
Language models are few-shot learners, 2020.
[4] R. Child, S. Gray, A. Radford, and I. Sutskever. Generating long sequences with sparse transformers, 2019.
[5] Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. Le, and R. Salakhutdinov. Transformer-XI.. Attentive language models beyond a fixed-length context. In A. Korhonen, D. Traum, and L. Mårquez, ed- itors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978 -9888, Florence, Italy, July 2019. Association for Computational Linguistics.
10

---

[6] P. Gage. A new algorithm for data compression. C Users J., 12(2):23-38 Feb. 1994.
[7] A. Graves, G. Wayne, and I. Danihelka. Neural turing machines, 2014.
[8] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):47144777 2016.
S. G.
[9] Joblib Development Team. Joblib: running python functions pipeline jobs, 2020.
[10] N. Kitaev, Lukasz Kaiser, and A. Levskaya. Reformer: The efficient transformer, 2020.
[11] Lu, D. Batra, D. Parikh, and Lee. Vilbert: Pretraining task-agnostic visiolinguistic
representations for vision-and-language tasks, 2019.
[12] A. F. T. Martins and R. F. Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification, 2016.
[13] K. M. Nakanishi. Scalable-softmat is superior for attention, 2025.
[14] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
[15] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Lan- guage models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019. https://cdn openai www//eettee-eeeeee -angeeeerr-lnggeeeeeeee soaiggeemmdddss/aanagggage unsupervised sateeeddd_uttttttek.o. learners. .dd..
[16] J. W. Rac, A. Potapenko, S. M. Jayakumar, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling, 2019.
[17] S. Sukhbaatar, J. Weston, Fergus, al. End-to-end memory networks. Advances in neural information processing systems, 28, 2015.
[18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances neural information processing systems, 30, 2017.
[19] P. Veličković, C. Perivolaropoulos, F. Barbero, and R. Pascanu. softmax is not enough (for sharp out-of-distribution)) 2024.
[20] Z. Yang, Z. Dai, R. Salakhutdinov, and W. W. Cohen. Breaking the softmax bottleneck: A high-rank rnn language model, 2018.
[21] M. Zaheer, G. Guruganesh, A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula,
Q. Wang, L. Yang, and A. Ahmed. Big bird: Transformers for longer sequences, 2021.
[22] C. Zheng, Y. Gao, G. Chen, H. Shi, J. Xiong, X. Ren, C. Huang, X. Jiang, Z. Li, and Y. Li. Self-adjust softmax, 2025.

---

A Theory justification
In the given appendix, we provide the proofs for our theory part.
Lemma 1. Consider the ag term:
F(1,0)
(2(0)
C2( 2 L
(14)
Σj=1 F(
Σj= ((()
since the function is continuous a compact space. The lower bound works the same.
For the proof of the corollary, notice that 4, = q k, therefore [4] is bounded by norm of Hence, we have the following bound.
Theorem 1. 1. Here we have fixed IN = {ii,-.. iNn C [t,,.... L.J. Therefore, we have:
d=
E
Σ
Cgtsy
S
Σ JEIUNJEIN
Σ
a
33
(15)
aN
iENVx
JEIN
Σ
Σ
0j
0i
0i
Dj
+ āN
Ij
Ij
(16)
ÖN
NN
iEAUy JEIN
ΣΣ aj
0i di + |z ||
σN )
ai
(17)
ÖN
iETYN JEIN
(1 - (xxxx)di
max ||,||| [āN(L - N) - (1 - aN)],
(18)
jEIN
where d,
max
|| zi /|.
sEIN
For the probability part:
L
E
Σ 2
|aaæ; - s|1 = E
Σ
1 (i $ IN) 1|,,, s|l
(19)
Σ P(i $ [1) - sllli & IN].
(20)
We can estimate both the expectation of norm and probability terms as follows:
P(i IN) 6-R N
(21)
E [||a, - s|li & IN] = E
OCTT; Σ
1
IN)aya
(22)
N
0,
aj
(23)
jfi
As result, we obtain:
L
N
N
E
as
[
T
(24)
L
L
i=1
12

---

The error tern between the approximation and the true value can be estimated using Jensen's 's gap bound:
1N N 1) 1e1ee0zll 02y1
6; = Billasa; - - E In] - ||a,, - E[sli & IN]ll - 2
(25)
10:0; 8:
Therefore:
L L N
Ei
(26)
im1
Now let's move to the corollary section.
Assuming N < and L. grows, we have:
E
03 [a|||
(27)
2. When N + L, we have E O, since number of outer elements goes to
Theorem We have:
E
Σ 1;11s-
Σ - s) iEIN
(28)
iEIN
Hence, we need to estimate the probability of the a being in the sphere.
Notice that ||a;æ; #|| is bounded random variable. We can estimate it as ||a, a|l E [0,
Hence, we have a Hoeffding-type inequality:
(r EX,)
P(X; Sr) infie
Be X:1 s exp jinf (-rt tEX, 4MPP)] s exp
16M2
(29)
where expected value of X; ||s a;æ;|| can be estimated follows:
EX, S EXF
(30)
and for the squared norm, we have:
| 2
Σ
ayJty
Σ
ajizy
Σ
(31)
jElN
JEINOR
j.kENN
jtt
(M/2 (MP-09 8²
2 Σ 03
Σ
0,0g =
(32)
jelx
j.EENN 1
1+1
where the last bound caused by condition Therefore, have:
|1,--,,11
82
13

---

(r E22
B[N,] S Σ exp
(33)
16M
iEIN
The lower bound is easier. Using Markov inequality and Cauchy-Schwarz (30):
EX,
2 1
(34)
Hence, we have:
(r ( E,)
N
Σ IEIN
& S ENN.] & Σ exp
(35)
16M2
iEIN
B Experiments
In the appendix, we provide details of the whole experiment setup and give the pseudocode we implemented for each figure.
System parameters
For the given research, we used the Apple M1 Pro chip with a 10-core CPU and 16GB of unified memory, based ARM architecture.
Software framework
The models were implemented and examined using PyTorch [14], running on the Apple MI Pro's ARM-based CPU architecture to ensure efficient computation.
For the parallelization procedure, we used Joblib library [9].
Distance analysis
Algorithm Distance Analysis. Different L and fixed N.
Require: Text input, list of token lengths Tvalues, fixed N 5
Ensure: Averaged distance statistics across layers and heads
for L in Lvalues do
2: Encode text using GPT-2 and extract attention matrices for all heads and layers
3: for each (head, layer) GPT-2 do
4:
for each token index in 1.... L do
5:
Compute true distance d via Eq. (7)
6:
Compute upper bound dmax via Eq. (9) Compute expectation E via Eq. (10)
7:
8:
Store (d, d E)
end for 9:
10:
Average distances across tokens 11: Store result for (head, layer)
12:
end for
Store all results for current T.
14:
end for
15: return All distance statistics
14

---

Algorithm 2 Distance Analysis. Different top-N and fixed
Require: Text input, list of top-N values Nvalues: fixed L 1024
Ensure: Averaged distance statistics across layers and heads
1: Encode text using GPT-2 and extract attention matrices for all heads and layers
2: for N in Nvahes do
3:
for each (head, layer) in GPT-2 do for each token index in I,..,, L do
4:
5:
Compute true distance d via Eq. (7)
6:
Compute upper bound dmax via Eq. (9) Compute expectation E via Eq. (10)
7:
8:
Store (d, dmax, E)
9:
end for Average distances across tokens
10:
Store result for (head, layer) 12: end for
Store all results for current
14:
end for
return All distance statisties
Algorithm 3 Distance Analysis. Critical Top-N detection.
Require: Text input, L. top-N values Naalues, significance level 0.05 Ensure: First N for which expected and true distances are statistically close Encode text using GPT-2 and extract attention matrices
2: for N in Naales do
3: Initialize list of relative errors
4: for each (head, layer) in GPT-2 do
5: for each token in 1..... do
6:
Compute true distance d via Eq. (7) Compute expected distance E via Eq. (10)
7:
8:
end for
9:
Compute mean true distance d Compute mean expected distance E
10:
11:
Store the d and E
12:
end for
13: if Kolmogorov-SSmmovvie E, significance level a) is true then
return 15: end if
14:
16: end for
17: return -1
No N meets the condition
15

---

Geometrical analysis
Algorithm 4 Geometrical analysis. Separation Ratio and Bounds for Top-N Attention Tokens.
Require: Text input, sequence length L,, top-N values Naalues
Ensure: Box plots N,/N and its theoretical bounds
1: Encode text using GPT-2 and extract attention matrices
2: Extract and normalize token embeddings
3: for N in Nvalues do
4: for each (head, layer) in GPT-2 do
5: for each token in ...... do
Compute N via direct counting Compute lower and upper bounds from Theorem 2 end for 8:
6:
7:
Average N,,N, upper bound, and lower bound over all tokens 10: Store results for (head, layer)
9:
end for
12: Store aggregated results for N
13:
end for
14:
Generate box plots comparing true and theoretical values
Gradient analysis
Algorithm 5 Gradient Sensitivity Analysis
Require: Text, temperature values Tvalues shift values Evalues
Ensure: Sensitivity statistics across temperatures and shifts
1: Convert text to logits matrices for all (head, layer) pairs
2: for T in Tvalues do
3:
for in Evalues do for each (head, layer) do
4:
5:
for each token E (1,.... L} do
6:
Sample unit vector with ||e||2 = Compute shifted logits 1' + sv
7:
8:
Compute softmax distributions a softmax(I/T),d a' softmax(!' /T) Compute sensitivity: 8 = ||a' all2/E
9:
end for
11:
Average å over tokens end for
13: Store maximum average across (head, layer) for current s
end for
15: Store results for temperature T
16: end for
17: return Sensitivity statistics
16

---

Time resources
Here, provide the time execution for all algorithms:
Algorithm
Time of execution
Parallelization
Alg.1
24 min.
No
Alg.2
37 min.
No
Alg.3
17 h. 4min.
Yes
Alg.4
7 min.
No
Alg.5
1 min.
No
Table 1: Comparison algorithm execution times. The table references algorithms defined in Algo-
rithms 1-5, highlighting their respective performance durations and whether they used parallelization.
17

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract reflects the general idea of the paper. All additional details are provided the paper itself.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
18

---

Justification: We discuss the theoretical limitations for Theorem 2. While the other results were provided without any additional assumptions.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: For all theoretical results, we provide the proper formulation. All objects of interest are stated in the corresponding sections. All proofs are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: For each experimental result in the section "Experiments", we demonstrate the corresponding algorithm in the Appendix B. After the blind review process, we will share the corresponding scripts
19

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-sonnee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use the open-source model (GPT-2) with proper reference to the model. Also, we directly stated the dataset we used and provided more explanation in the Ap- pendix B.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ Seaiii/guudds//ooeeeueeee
conooicyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips wttpcc//pppppppeeeeeeeeeeeeoeece http:/uwiee//eeeeeeee htppbbiigggiddeeeoettttttaiiiiii
/oooeeeubiissiiiiiceaaeaae sniinniicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
20

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We provided the algorithms and data parameters we used to replicate the results.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We apply the statistical tests in several experiments and reflect them in the corresponding algorithm. Also, state the significance level and the test we use in the text directly.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the given information in the Appendix B.
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttaar////eepss/eeeee cc.s..c//oa (ppplic/Ethissuuui ines?
Answer: [Yes]
Justification:
We follow the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed. The paper is theoretical examination of the well-known model.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
22

---

Answer: [NA]
Justification:
We provide all the details about the data and the model we use.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We provide the references to the data and models we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't 't provide new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
23

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work doesn't 't satisfy the given claim.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work doesn't satisfy the given claim.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The given paper was written by a group of authors. We followed the LLM usage guidelines and didn't 't violate them.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy (https http://eeuuppppeee cc//oo/eeeecccc///200000 for what should or should not be described.
24


Reviews:
[{"review_id": "lVW7xalkfn", "reviewer": "Reviewer_WqWW", "summary": "This paper thoroughly investigates the inherent limitations of normalization techniques, particularly softmax, within attention mechanisms. The authors propose a theoretical framework that defines the selective capacity of attention models and the geometric separation of tokens. Their analysis provides explicit bounds on distances and separation criteria for token vectors under softmax scaling. The theoretical findings are empirically validated using a pre-trained GPT-2 model, demonstrating that as the number of selected tokens increases, the model's ability to distinguish informative tokens diminishes, often leading to a uniform selection pattern. Furthermore, the paper highlights that gradient sensitivity under softmax normalization poses challenges during training, especially at low-temperature settings. The work contributes to a deeper understanding of softmax-based attention mechanisms and motivates the development of more robust normalization and selection strategies for future attention architectures.", "questions": "1. Could the authors elaborate on the specific implications of the $\\approx 80 \\%$ geometric separability limit for multi-head attention mechanisms? How might this limit guide the optimal number of heads or the design of their specialization?\n2. The paper suggests monitoring attention entropy. Are there specific empirical thresholds or diagnostic tools that could be developed from this work to alert practitioners when a head has saturated its geometric capacity?\n3. The proposed guidelines recommend avoiding overly sharp softmax. What are the practical implications of this for hyperparameter tuning in real-world large language models, especially given the common practice of using temperature scaling?\n4. Beyond the mentioned Sparsemax, Scalable-Softmax, and Self-Adjusted Softmax, could the authors discuss how their theoretical bounds might be used to evaluate or guide the design of other novel normalization functions not yet explored in the literature?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The paper presents a rigorous theoretical framework, including non-asymptotic bounds on representation distance (Theorem 1), geometric separability (Theorem 2), and a general Jacobian bound for gradient sensitivity (Lemma 2). These theoretical derivations provide a principled understanding of attention mechanism limitations.\n2. The theoretical predictions are strongly supported by comprehensive experiments conducted on a GPT-2 model, confirming distance collapse, separability saturation, and 1/T gradient growth.\n\nWeaknesses：\nThe geometric interpretation relies on assumptions of uniform spherical distribution and minimum pairwise separation of embeddings, which might not always hold true in real-world models. The authors acknowledge that future work should extend the geometric bound to non-spherical distributions.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We address each question below:\n\n**Q1.** It is commonly assumed that different attention heads specialize in distinct functions—for example, each head might capture a different type of semantic relation. If one knows (1) the number of interaction types the model needs to learn and (2) the per-head geometric separability $p$, then it is possible to compute a “coverage coefficient” that tells how many heads are required to select a desired fraction of important tokens.\n\nConcretely, if each head has separability $p$, then $H$ heads can cover up to $ 1 - (1 - p)^H$ of the top-$N$ tokens. For instance, with $p = 0.8$, you need $H = 3$ heads to cover roughly $99$% of the important tokens.\n\nBeyond simply increasing the head count, encouraging a small amount of orthogonality or diversity between heads can help to approach the theoretical bound in practice.\n\n\n**Q2.** There is an empirical connection between the overall attention entropy $H(\\alpha)$ and the geometric separability ratio $N_s/N$, but pinning down a precise theoretical link is difficult due to the unknown distribution of the embeddings $x_i$. As a practical proxy, we recommend monitoring, for each head–layer pair:\n\n  1. Relative entropy of the top-$N$ tokens: the ratio $H(\\alpha_{I_N})/H(\\alpha)$, where $H(\\alpha_{I_N})$ is the entropy restricted to the selected top-$N$ weights.\n  2. Geometric ratio: the fraction $N_s/N$. \n\nWhen $N_s/N$ falls below roughly $0.7$-$0.8$, this signals that the head has saturated its geometric capacity. Similarly, a growing ratio $H(\\alpha_{I_N})/H(\\alpha)$ serves as an early warning that selectivity is degrading.\n\n\n**Q3.**  Our results suggest refraining from driving a temperature parameter $T$ too low during training. In standard Transformers one often uses $T=\\sqrt{d_k}$ (or simply $T=1$ in many implementations). Our theory and GPT‑2 experiments showed that once $T\\lesssim0.1$ the softmax Jacobian norm explodes ($\\sim1/(4T)$), making gradients noisy and optimization brittle. So, a safe window with high selectivity is roughly $[0.5,2.0]$. If you need “sharper” attention than $T=0.5$ allows, it’s often better to switch to a sparsity‐inducing or length‐aware normalizer (e.g. Sparsemax, Scalable‐Softmax, Entmax) than to drive $T$ into the unstable regime.\n\n\n**Q4.**  Any new normalization function \\$F(l;\\theta)\\$ can be evaluated directly against our three theoretical results.  For checking long‐context scaling: substitute \\$F\\$ into the normalization formula \\$\\alpha\\_i=F(l\\_i;\\theta)/\\sum\\_jF(l\\_j;\\theta)\\$ and verify whether \\$\\alpha\\_i\\$ still decays like \\$O(1/L)\\$ as \\$L\\to\\infty\\$ (Theorem 1).  If it does, the mechanism will inevitably suffer the same vanishing‐attention collapse in very long sequences.  Next, to quantify the distance collapse \\$F\\$ should be plugged into the fixed and random top‑\\$N\\$ bounds of Theorem 1. This yields a concrete upper bound on the context‑vector vs. non‑selected‑token distance as a function of \\$N/L\\$, revealing the fraction of tokens beyond which separability collapses.  Under the same spherical‐embedding assumptions of Theorem 2, replacing the softmax weights with those induced by \\$F\\$ in the definition of \\$\\alpha\\_i\\$ lets computing the analytic upper and lower bounds on \\$\\mathbb{E}\\[N\\_s]/N\\$, immediately showing the geometric capacity ceiling of the design.  Finally, the general Jacobian bound works the same. \n\nThese quantitative evaluations then guide principled design.  If a candidate still collapses as \\$L\\$ grows, one can introduce an explicit \\$L\\$‐dependence—e.g. \\$F(l;\\theta,L)=g(L),\\tilde F(l;\\theta)\\$—so that \\$\\min\\_i\\alpha\\_i\\$ stays bounded away from zero.  Should the distance bound tighten too early (at small \\$N/L\\$), it is possible to adjust the shape of \\$F\\$—for instance by tempering its tail behavior or adding a tunable sharpness parameter—to push the critical \\$N/L\\$ threshold higher.  When the geometric ceiling \\$\\mathbb{E}\\[N\\_s]/N\\$ remains below your target (say 90 %), one should tweak \\$\\theta\\$ to redistribute mass more unevenly across logits, directly improving the separability bound.  And if the Jacobian estimate exceeds your threshold, you refine \\$F\\$ to smooth out steep slopes or enforce a lower bound on \\$F(l)\\$, thereby capping \\$|\\nabla\\alpha|\\$ thus improving gradients.  \n\nWhile our theoretical framework (Equation 3) applies to general normalization functions $F(·,\\theta)$, we focused on softmax due to its widespread use. Our bounds in Theorems 1-2 and Lemma 2 can be directly applied to evaluate other normalizations like Sparsemax (which would have different constants in the bounds but similar scaling behavior), Scalable-Softmax (which explicitly depends on L, violating the assumption in Lemma 1), and Self-Adjusted Softmax (which dynamically adjusts parameters based on layer statistics). However, we decided not to put this analysis to our paper, since it would change the focus of the research.\n\nWe hope these answers address your questions. Thank you agian for your questions and the detailed review."}, {"review_id": "AxJOZNHuhi", "reviewer": "Reviewer_1phE", "summary": "This paper provides a theoretical and empirical analysis of the limitations of softmax in the attention mechanism of transformers. The authors introduce mathematical bounds showing that as the number of tokens in the context increases, softmax attention becomes less selective, and meaningful token distinction is lost.", "questions": "Please see weaknesses. I also have the following questions and suggestions:\n\n1. I would recommend the authors use either softmax or soft-max consistently throughout the paper, and not alternate between the two.\n\n2. Regarding Lemma 1: If I understood correctly, the proof assumes that the normalization function F is non-zero on the given compact set. Otherwise,  $C_2/L$ could be 1 and not dependent on $L$. Is this correct? If so, please consider stating this assumption explicitly in the lemma for completeness.\n\n3. I enjoyed the theoretical approach involving the geometrical interpretation and gradient analysis. However, I cannot follow the final conclusion the authors make (lines 205–207). According to Corollary 3, the Jacobian norm is bounded by the minimum of two values. When the temperature T is small, $\\sqrt{2}$ dominates. So I'm unclear how this supports the claim about sensitivity of the training process. Could the authors clarify this link?\n\n4. The footnote refers to identical trends in larger variants of the model, but no ablation study is provided. If the authors have studied this on larger models, including those results in an appendix would be very helpful.\n\n5. Line 215 is missing a reference to the appendix. I checked, and there is no information provided on hyperparameters. Please add this for completeness and reproducibility.\n\n6. In Figure 3, the upper bound appears to be fixated at a ratio of 1, which seems trivial. Can the authors elaborate on how this bound provides insight into the problem?\n\n7. How does the authors’ analysis apply to other forms of attention—for example, sigmoid attention, or even the alternative normalizers referenced earlier in the paper?", "limitations": "Please see weaknesses and questions.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Strengths:*\n\n1. The paper introduces clear, non-asymptotic bounds that quantify well-known but often loosely described flaws in softmax-based attention. The bounds are not only derived but also experimentally validated on GPT-2, strengthening the claims.\n2. The results justify and explain the motivation behind alternative normalization functions such as Sparsemax.\n3. The paper is generally well-organized, and main arguments are clearly presented.\n\n\n*Weaknesses:*\n1. Although alternative normalizers are discussed, the empirical validation is done only for softmax. There is no direct comparison to Sparsemax, Scalable-Softmax, or other recent variants, so practical trade-offs are not explored. As the claims in the paper signify the motivation for these alternatives, it would be helpful if the authors could also show that, in practice, the issues they identify actually align with observed differences.\n2. As far as I understood, the theorems are only valid for a single head attention, which is not a very practical case, limiting the scope of the paper. I generally feel the authors have postponed the limitations of their work, which actually make their result more practical, to future work.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We address each point below.\n\n*Weaknesses:*\n\n**W1.** We acknowledge this limitation. While our theoretical framework (Equation 3) applies to general normalization functions $F(·,θ)$, we focused on softmax due to space constraints. We decided not to put the empirical analysis of the mentioned methods, since it would change the focus of the paper. But, our preliminary analysis expects:\n\n1. Sparsemax: Would show better separation due to exact zeros but similar capacity limits;\n2. Scalable-Softmax: The $\\log{L}$ scaling would modify the bounds in Lemma 1 but not eliminate the fundamental limitation;\n3. Self-Adjusted Softmax: Would show improved gradient stability while maintaining similar geometric constraints;\n\n**W2.** We thank the reviewer for this important observation. Indeed, all of our formal results assume a single, independent attention head. This was a conscious choice rather than an oversight: by isolating one head we obtain tight, non‑asymptotic bounds on both the vanishing‑attention effect and the geometric separability limit, which serve as the foundational building block for understanding multi‑head architectures.\n\nIn practice, transformer layers employ $H$ heads in parallel, and their collective “coverage” of the top‑$N$ tokens can be approximated (under a mild independence assumption) by the union bound $\\mathrm{coverage}(H) \\=\\ 1 - (1-p)^H$, where $p$ ($\\approx$ 0.8 for GPT-2) is the per‑head separability ceiling from Theorem 2. With $H=2$ or $3$ heads, this predicts over $95\\%$–$99\\%$ coverage for GPT-2, which helps explain why multi‑head attention remains effective despite each head’s intrinsic limit.\n\nWe fully agree that interactions among heads—overlap, specialization, and learned correlations—are crucial in practice. Extending our theory to capture these dependencies is non‑trivial, as it requires modeling the joint distribution of multiple attention outputs and their geometric interplay, and is therefore outlined as an exciting direction for future work (see end of Section 5). In the revised manuscript, we plan to make three clarifications:\n\n  1. We explicitly state that all formal results assume a single, independent head.\n  2. We show how to apply our single‑head bounds via the union bound to obtain first‑order predictions for a multi‑head layer.\n  3. We expand the Discussion to sketch how one might incorporate head‑to‑head correlations—e.g., via orthogonality penalties or diversity regularizers—and thus refine the multi‑head capacity analysis.\n\n\nWe believe that establishing these single‑head capacity limits is an essential first step toward a fully general theory of transformer attention, and we plan to update the manuscript to more clearly highlight both its current scope and the roadmap for extending our results to multi‑head settings.\n\n*Questions:*\n\n**Q1.** Thank you for the suggestion. We will make the term usage consistent in the final version. \n\n**Q2.** We thank the reviewer for catching this important detail.  You are correct that Lemma 1 implicitly relies on a non‑vanishing normalization function in order to invoke the Extreme Value Theorem.  In the revised manuscript, we will clarify this.\n\n**Q3.** We thank the reviewer for raising this important point.  The $\\sqrt{2}$ is the global upper‐bound on how far any two probability distributions on the simplex can differ in $\\ell_{2}$ norm.  It only becomes active when $ T < 1/(4\\sqrt{2}) $.\nFor all practical $T\\gtrsim0.2$, the bound reduces to $\\bigl\\|\\nabla_{\\ell}\\boldsymbol\\alpha\\bigr\\|_{2}\\\\le\\\\frac{1}{4T}.$\n\nIn other words, throughout the realistic range of $T$, lowering the temperature $T$ amplifies the Jacobian norm - and hence the sensitivity of the attention gradients—in direct proportion to $1/T$.  As $T\\to0$, the softmax distribution approaches a one‐hot (i.e. $\\delta_{\\max}$) and even tiny perturbations in the logits can induce nearly maximal changes in the weight vector, making gradient steps highly volatile.\n\nWe plan to revise lines 205–207 to clarify that:\n1. the $\\sqrt{2}$ bound is a trivial worst‐case for extremely small $T<1/(4\\sqrt{2})$;\n2. for all temperatures of practical interest, the true sensitivity is governed by the $1/(4T)$ law;\n3. this $1/T$ scaling explains why driving $T$ too low destabilizes training by inflating gradient variance.\n \n**Q4.** Thank you for your suggestion. We will include experiments on GPT-2 medium/large variants in the Appendix.\n\n**Q5.** We apologize for this error. Line 215 should reference Appendix B, which contains implementation details. The appendix includes hyperparameters: we used the standard GPT-2 124M model with its default configuration. We will correct this reference.\n\n**Q6.** We appreciate the reviewer’s point.  It is true that, for GPT‑2, the analytic upper bound evaluates to nearly 1 across all $N$, which makes it look trivial in Figure 3.  However, its value lies not in tightening our GPT‑2 curves, but in quantifying the worst‑case separability any normalization can achieve given those weights and embeddings.  The bound depends explicitly on the per‑token spreads $\\xi_i$.  In models where attention weights are sharper or embeddings more widely dispersed, the exponential factors can drop well below 1—revealing geometric limits. Even when it sits at 1, the bound tells us that no more than 100 % of tokens can be distinguished (a sanity check) and that, to see non‑trivial ceilings, one must either increase $r$ or manipulate the $\\xi_i$ via different normalizers.  If a novel attention rule or embedding scheme yields an upper bound that falls significantly under 1, that immediately signals a head is running out of capacity.  Conversely, if the bound remains at 1, one can infer the current configuration still has headroom for separability.\n\n**Q7.** Thank you for this question.  Because all of our main results are phrased in terms of a general normalizer, they carry over directly to any differentiable, sign‑constant $F$, including sigmoid attention. Although we focused our experiments on softmax, every step of our distance, geometry, and gradient analysis can be applied to sigmoid attention and to all alternative normalizers you mentioned.\n\nWe hope our answers address your questions, and suggested text corrections will improve the paper. Thank you again for your questions and the detailed review."}, {"review_id": "GIGVGUjwmL", "reviewer": "Reviewer_Duo4", "summary": "This paper provides a theoretical and empirical analysis of softmax normalization in attention mechanisms. It presents three core results:\n\n1. attention loses selectivity when the number of selected tokens grows proportionally with context length.\n\n2. no more than ~80% of selected tokens can be geometrically distinguished by a single head.\n\n3. low temperature leads to high gradient sensitivity.\n\nTheoretical findings are validated on GPT-2, demonstrating distance collapse, separability saturation, and gradient instability as predicted.", "questions": "1. You have summarized the contributions, but I would like to understand the practical implications of the theoretical bounds. Could you provide an intuitive explanation of how these bounds relate to downstream tasks?\n\n2. Could you please explain why the GPT-2 model was chosen for the experiments? As far as I know, it is rarely used in real-world applications. Have you considered using more popular open-source LLMs such as LLaMA or Qwen?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\nClean and well-motivated theoretical results, with non-asymptotic closed-form bounds.\n\nEmpirical analysis (on GPT-2) aligns well with theoretical predictions.\n\nPractical takeaways are clearly stated, such as limiting active token sets and monitoring entropy.\n\n# Weaknesses\n\nThe phenomena discussed are not entirely new—softmax bottlenecks and gradient instability have been qualitatively explored in prior work (as far as I know, though I am not an expert in this area).\n\nLarger-scale experiments would be more convincing, as GPT-2 (124M) is relatively small. \n\nThere is a typo in the Experiments section: “Appendix ??” should be fixed.", "was_revised": true, "final_justification": "support this paper for acceptance and keep my score unchanged", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. \n\n*Weaknesses:*\n\n**W1.** We thank the reviewer for highlighting these connections. It is true that intuitively one can point to “softmax bottlenecks” or unstable attention gradients in the literature. Our key novelty lies in turning qualitative observations from the literature (such as  softmax bottlenecks or unstable attention gradients) into fully non‑asymptotic, closed‑form bounds and then confirming them empirically on a typical transformer model. In particular, to the best of our knowledge no prior work has:\n\n1. Established an explicit, model‑agnostic distance bound showing exactly how the context‑vector vs. non‑selected‑token distance collapses once the top–$N$ set grows proportionally with $L$.\n\n2. Derived a geometric separability ceiling under minimal spherical assumptions, showing that a single head of GPT-2 on average cannot distinguish more than roughly 80 % of its top‑$N$ tokens.\n\n3. Quantified the precise $O(1/T)$ trade‑off between selectivity and gradient sensitivity for any smooth normalizer, and shown that this limit is in fact tight in GPT‑2.\n\nTogether, these contributions provide the first unified, quantitative framework that explains why and when softmax–type attention fails, and sets guidelines for practical diagnostics and design rules for improved normalizers.\n\n**W2.** We thank the reviewer for this suggestion.  Our primary aim was to develop a rigorous, model‐agnostic theoretical framework, and we chose GPT‑2 124M as a fast yet strong sanity check.  Despite its modest size, GPT‑2 already exhibits all three core phenomena—distance collapse, an ≈ 80 % geometric separability ceiling, and the $O(1/T)$ gradient‐sensitivity trade‐off—exactly as predicted by our theorems.\n\nWe fully expect our main conclusions to hold across even bigger models.  To that end, we are planning a dedicated large‐scale empirical study on state‐of‐the‐art families (e.g. LLaMA, GEMMA) to validate the theory on larger models.  This follow‐up will quantify how various normalization strategies perform in ultra‐large contexts and will explore new approaches for designing attention mechanisms that overcome the limits identified in our current work.\n\n\n**W3.** Thank you! We will fix it in the final version.\n\n*Questions:*\n\n**Q1.** At a high level, our three bounds translate into concrete signals about when and why real‐world tasks - such as document summarization, question answering over long contexts, or multi‐turn dialogue - begin to fail under standard softmax attention.\n\nFirst, the *distance bound* shows that once the number of tokens you ask an attention head to consider grows in proportion to the context length, the “context vector” it produces drifts toward the average of all embeddings and loses the ability to emphasize truly relevant tokens.  In downstream tasks that require pinpointing rare but crucial facts deep in a document—say, a specific date in a legal contract or a distant antecedent in a coreference chain—this drift leads directly to poorer accuracy, because the model treats everything as nearly equally important.\n\nSecond, the *geometric separability bound* quantifies a cap (≈ 80 % for GPT-2) on how many “important” tokens a single head can isolate, even under ideal conditions. For tasks like machine translation or code generation – where different heads must each track distinct linguistic or structural relationships – this ceiling explains why simply adding more context or more tokens per head no longer helps.  Instead, one must either deploy multiple specialized heads or switch to a different normalization that can break this barrier.\n\nFinally, the *gradient‐sensitivity bound* uncovers the trade‐off between sharper attention distributions (necessary to focus on the few truly relevant tokens) and stable training dynamics.  In practice, if one pushes the softmax temperature too low in order to “zoom in” on important signals, one will encounter noisy or even divergent gradient updates during fine‐tuning. This directly impacts downstream performance on tasks that rely on precise calibration of attention weights – such as retrieval‐augmented generation or instruction following – since unstable gradients can lead to overfitting or failure to converge.\n\nTogether, these three theoretical insights give model builders a principled “warning system”:\nif you see your summarization or QA accuracy plateau or degrade as you increase context length, consult the distance bound;  \nif adding more heads does not improve performance, consult the geometric bound;  \nand if training becomes unstable when sharpening attention, consult the gradient‐sensitivity bound.  \n\nIn this way, our bounds map directly onto the real‐world phenomena, and they suggest clear remedies – sparse or length‐aware normalization, entropy monitoring, head specialization – that can rescue performance on challenging long‐context tasks.\n\n**Q2.** We chose GPT‑2 124M for our experiments primarily because it is fully open‐source, lightweight, and allows exhaustive extraction of attention matrices across all heads and layers with modest computing resources.  This made it possible to validate our non‐asymptotic bounds (distance collapse, geometric ceiling, and $1/T$ gradient scaling) quickly and reproducibly.  While GPT‑2 may not be the latest production model, it remains a well‐understood research baseline whose behavior is representative of Transformer‐based architectures.\n\nWe fully agree that extending to more recent open‐source families (e.g. LLaMA, Qwen) would be valuable.  Indeed, we are planning a dedicated large‐scale study on LLaMA, Qwen, and other models to (i) verify that our theoretical limits hold in other settings and (ii) explore how alternative normalization strategies perform in real‑world LLMs.  This follow‑up will be reported in a separate, forthcoming publication.\n\nWe hope these changes address your concerns and improve the clarity and significance of our contribution.\n\nThank you for your questions and the detailed review."}, {"review_id": "SZSnCQwcEv", "reviewer": "Reviewer_NWWE", "summary": "This paper revisits the token selection mechanism with the softmax function, which is currently a core component of the attention mechanism of the transformer architectures.\nThe authors begin by analyzing how, as the context length $L$ increases, the attention mechanism becomes less capable of efficiently picking the relevant tokens.\nThey further investigate the geometric token separability and the gradient instability when the softmax temperature $T$ decreases.\nThe paper also includes the experiments using the pre-trained GPT-2 models to support the theoretical findings.", "questions": "1. Line 59: Does the function $F$ takes values only in the positive real domain, rather than the entire $\\mathbb{R}$? I assume the constants $C_1^\\prime(\\theta)$ and $C_2^\\prime (\\theta)$ are positive in the proof of the lemma 1, but the paper lacks clarification.\n2. I'm struggling to understand the significance of the analysis in Section 2.2. How does this result relate to the standard softmax normalization, for example, in terms of the number of selected examples $N$? The finding that a larger $N$ leads to a smaller ratio $N_s / N$ (in Figure 3) seems unsurprising. This appears to be a general phenomenon in token separation, rather than a specific limitation of the softmax.\n\nAdditionally, I would appreciate a clarification and resolution of the concerns raised in the Weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. The paper approaches the limitation of the softmax from three distinct perspectives, each of which is explained well.\n2. The discussion and conclusion provide the practical takeaways. It helps clarify the overall message of the paper.\n3. The inclusion of pseudo-codes improves the reproducibility of the empirical results.\n\n**Weakness**\n1. The paper lacks a strong foundational motivation. It is unclear why the authors focus on the limitation of softmax-based token selection, given its widespread success in many machine learning applications.\n2. In equation 5, the probability of token assignment becomes $O(1/L)$, but it is not obvious whether it leads to vanishing attention in practice. This issue becomes problematic when the sequence length $L$ grows exponentially with respect to $\\Delta = \\\\|\\boldsymbol{\\mathbf{q}}\\\\|_2 \\\\|\\boldsymbol{\\mathbf{k}}\\\\|_2$, which seems unrealistic in the practical scenarios. As a result, this point does not convincingly provide the weakness of the softmax normalization. At least, the authors should demonstrate that such uniform behavior actually occurs in some realistic setting and that it leads to degraded performance, either through new experiments or by citing existing work.\n3. The paper lacks \"Related Work\" section. It obscures the position of the paper and makes it difficult to assess the novelty of the contribution.\n4. Several alternatives to softmax are presented in \"Discussion\" section, but it is unclear whether they actually address the issues of softmax normalization and lead to the performance improvement. I believe that the experimental results comparing the softmax with them would enhance the quality of the paper.", "was_revised": true, "final_justification": "Initially, I found it challenging to understand the motivation of this work, its positioning within the existing literature, and the motivations and connections across the different analyses and sections, which led me to assign a low score. However, the rebuttal has resolved my questions. Since the authors have promised to improve these aspects in the revision, I now recommend acceptance.", "rebuttal": "We thank the reviewer for their detailed and in-depth feedback. We address each point below:\n\n*Weaknesses:*\n\n**W1.** We thank the reviewer for this comment and agree that a clearer statement of foundational motivation strengthens the paper.  Although softmax‐based attention has achieved impressive empirical success in many settings, modern applications increasingly demand handling very long contexts (tens of thousands of tokens) and fine‐grained focus on rare but critical information.  In these regimes, practitioners have observed two persistent problems:\n\n 1. **Vanishing‐attention in long contexts.**  As $L$ grows, softmax weights collapse toward $1/L$, making it impossible to single out informative tokens (flagship models can effectively use about 10-15% of their context window as demonstrated by BABILong benchmark) — a phenomenon that underlies many recent “long‐context” workarounds (sparse or windowed attention, segment level recurrence, compressive memories, etc.).  \n\n 2. **Gradient instability under sharpening.**  To counteract vanishing attention, one often lowers the temperature $T$, but this in turn inflates the softmax Jacobian ($\\sim1/(4T)$), leading to noisy or divergent training dynamics.  \n\nThere has been no quantitative, non‐asymptotic theory explaining why and when these failures occur, nor any general tools to predict the limits of a given attention rule.  Our work fills this gap by:\n\n\n 1. Deriving tight bounds that show exactly how the context‐vector vs. non‐selected‐token distance collapses once the selected set grows proportionally with $L$.\n 2. Proving a fundamental geometric ceiling on the fraction of top‐$N$ tokens a single head can distinguish, even in idealized spherical embeddings.\n 3. Quantifying the precise $O(1/T)$ trade‐off between selectivity and gradient‐sensitivity for any smooth normalizer.\n\nThese results provide the first unified theoretical framework to explain the intrinsic capacity limits of softmax attention and to guide the design of more robust normalization schemes—an essential foundation for scaling Transformers to ever‐longer contexts without sacrificing stability or accuracy.  \n\nWe will extend the Introduction to better present motivation by clearly stating these points and clarifying why understanding softmax’s limitations is critical despite its empirical success in shorter‐context regimes.\n\n**W2.** We thank the reviewer for raising this point.  Although Equation 5 shows that each attention weight scales as $O(1/L)$, in practice, modern transformers keep representations normalized, thus bounding the norm. Another factor is that $\\Delta = ||q||_2 ||k||_2 $ remains constant when $L$ varies from $10^2$ to $10^6$. Consequently, when $L$ reaches sizes $L\\sim10^5$, the softmax denominator far exceeds any fixed numerator, driving maximum probability $\\alpha \\approx Const/L$ and $Const \\ll L$.  In other words, even the most relevant token receives a vanishingly small probability.\nA concise exploration of this phenomenon – a fix via explicit length‐aware scaling—is given in Nakanishi et al. (2025) “Scalable‑Softmax Is Superior for Attention”  [12], which we highlight in Section 5.  By adding a $\\log L$ term to the logits before exponentiation, Scalable‑Softmax keeps $\\max_i \\alpha_i$ roughly constant as $L$ grows, thereby preserving focus on the most informative tokens even in very long contexts.\n\n**W3.** We thank the reviewer for this suggestion. To clarify the paper’s position and highlight its novel contributions, we will extend the paper to explicitly contrast our theoretical framework with existing qualitative or single‐aspect treatments in sufficient detail. We believe this contextualization will strengthen the reader’s ability to assess the novelty and impact of our contributions.\n\n**W4.** We thank the reviewer for this suggestion. Our primary aim in the present work was to develop a general, closed‐form theoretical framework for understanding the intrinsic limits of any softmax‐type normalizer, and to validate those limits empirically on a standard Transformer backbone.  As such, we have not yet carried out a systematic, head‐to‐head empirical comparison between softmax and its sparse or length‐aware variants (Sparsemax, Scalable‐Softmax, Self‐Adjusted Softmax, etc.).  \n\nBecause our bounds apply out of the box to any differentiable normalizer $F$, we can now conduct such comparisons in an “informed” manner: for each candidate $F$ one simply computes its distance‐collapse curve, geometric ceiling, and gradient‐sensitivity estimate under our framework, and then measures actual performance on long‐context tasks. We view this as the natural next step—and indeed a full paper in its own right—where we will evaluate a range of alternative attention rules on LLaMA, Qwen and other modern LLMs, identify which normalization choices yield the best trade‐offs in practice, and use our bounds to explain why they succeed or fail.   We believe that separating the present foundational treatment from a dedicated large‐scale empirical evaluation yields a clearer exposition and more coherent research agenda.  \n\n*Questions:*\n\n**Q1.** Yes, the function $F$ is assumed to be strictly positive over a compact domain (otherwise the Extreme value theorem won’t work); we will clarify this explicitly in Lemma 1. Thank you for spotting this!\n\n**Q2.** We agree that increasing $N$ generally reduces $N_s/N$, but our analysis quantifies how quickly this degradation occurs under softmax. The key insight is that separability saturates for $N \\ll L$, supporting the claim that softmax attention is inherently capacity-limited. \n\nWe hope these changes address your concerns and improve the clarity and significance of our contribution.\n\nThank you for your questions and the detailed review."}]

Author rebuttal:

Meta-review: This paper presents a theoretical framework for understanding the limitation of the SoftMax normalization in attention mechanisms. Specifically, the authors derive distance, geometry and gradient bounds to characterize a model’s selective ability and gradient sensitivity under SoftMax normalization. The theory is validated using a pre-trained GPT-2 model.

The paper is well written. The theoretical bounds are rigorously derived, clearly presented, and supported by strong experimental evidence. Together, they provide a quantitative theoretical explanation of vanishing‐attention in long contexts and gradient instability under sharpening in SoftMax attention. 
 
During the rebuttal, the authors provided more clarification on the settings and assumptions, such as the use of single-head analysis, and the applicability of other normalization functions beyond SoftMax, addressing reviewers’ concerns. Therefore, the paper is recommended for acceptance.
Decision: Accept (poster)