Why do LLMs attend to the first token?
Authors: Federico Barbero
Alvaro Arroyo
Xiangming Gu
Christos Perivolaropoulos
Petar Veličković
Razvan Pascanu
Michael M. Bronstein
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Attention Sinks
Information Propagation
Pre-training
TL;DR: We study why it is useful for attention heads in LLMs to learn to "dump" most attention on the first token from an "over-mixing" perspective.

Abstract: Large Language Models (LLMs) tend to attend heavily to the first token in the sequence -- creating a so-called attention sink. Many works have studied this phenomenon in detail, proposing various ways to either leverage or alleviate it. Attention sinks have been connected to quantisation difficulties, security issues, and streaming attention. Yet, while many works have provided conditions in which they occur or not, a critical question remains shallowly answered: Why do LLMs learn such patterns and how are they being used? In this work, we argue theoretically and empirically that this mechanism provides a method for LLMs to avoid over-mixing, connecting this to existing lines of work that study mathematically how information propagates in Transformers. We run experiments to validate our theoretical intuitions and show how choices such as context length, depth, and data packing influence the sink behaviour. We hope that this study provides a new practical perspective on why attention sinks are useful in LLMs, leading to a better understanding of the attention patterns that form during training.

Published as a conference paper at COLM 2025
Why do LLMs attend to the first token? L
Federico Barbero Álvaro Arroyo",, Xiangming Gu², Christos Perivolaropoulos? Michael Bronstein', Petar Veličković' Razvan Pascanu Uniivessity of Oxford National University of Singapore Google DeepMind 'Equal contribution. Correspondence to federico. barberorsrr ox. ac. .kk
Abstract
Large Language Models (LLMs) tend to attend heavily to the first token in the sequence creating a so-called attention sink. Many works have studied this phenomenon in detail, proposing various ways to either leverage or al- leviate it. Attention sinks have been connected to quantisation difficulties, security issues, and streaming attention. Yet, while many works have pro- vided conditions which they occur or not, critical question remains shal- lowly answered: Why do LLMs learn such patterns and how are they being used? In this work, we argue theoretically and empirically that this mechanism provides method for LLMs to avoid over-mixing, connecting this to exist- ing lines of work that study mathematically how information propagates in Transformers. We conduct experiments to validate our theoretical intu- itions and show how choices such as context length, depth, and data pack- ing influence the sink behaviour. We hope that this study provides a new practical perspective on why attention sinks are useful in LLMs, leading to better understanding of the attention patterns that form during training.
1 Introduction
Large Language Models (LLMs) are powered by hundreds or even thousands of attention heads that are orchestrated to update the values of tokens within a sequence. As their attention patterns are the only mechanism that allows for the mixing of information between tokens within a Transformer architecture, it is natural to study them to understand how information is being processed. A peculiar and interesting phenomenon that has been spotted across frontier language models is that attention heads often exhibit 'attention sinks', where seemingly meaningless tokens often the first one in the sequence -een to capture most of the attention. Attention sinks have been linked to number of important topics such as quantisation (Liu al., 2024), improved KV-caching (Ge et al., 2024), streaming attention (Xiao et al., 2024), and even security vulnerabilities (Yona et al., 2025), making them an important artifact that is not yet well-understood in frontier LLMs.
While many works have studied attention sinks in order to mitigate them, in this work we take different angle and aim to understand why they useful. We do this for simple reason: as sinks are widespread and appear as byproduct of gradient descent rather than any explicit priors, they must provide important mechanism for processing the context. We are therefore interested in understanding and explaining when and why the attention sink mechanism is useful. We tackle the issue from various angles: theoretically studying why such an effect is useful from a 'mixing' perspective, and performing measurements in both frontier LLMs and models we train from scratch support our theory.
We focus specifically on understanding why attention sinks occur mostly at the first position of the context. In this position, there commonly lives a special token often denoted as (bos) (beginning of sequence). We find, for example, that in a typical prompt in Llama 405B, almost 80% of the attention is concentrated on the (bos) token (see Section 4.2 for details). This type of attention allocation is in many ways 'wasted', , and we find it intriguing to understand exactly why this type of learned behavior is useful.

---

Published as a conference paper at COLM 2025
No sinic
Sink
V2
%
(0)
VG
CSI
(-)
Figure 1: Our key result is to illustrate how attention sinks are usefully leveraged by decoder- only Transformers. The presence of attention sinks slows down the mixing of information between tokens and hence makes Transformers more robust to perturbations of prompts. To illustrate this, we demonstrate how sharply a perturbation in the second token's 's input representation (in red) affects the embeddings of other tokens throughout the model, both without (left) and with (right) a sink token (e.g. (bos)). The presence of sink draws attention away from the rest of the tokens, limiting the spread of perturbed information and resulting in more stable embeddings. See Figure for direct measurement Gemma 7B.
Our main contribution is showing that this type of pattern is a way for deep Transformers to avoid 'over-mixing', , phenomenon related to a number of theoretical and empirical works that study rank collapse (Dong al., 2021), representational collapse (Barbero al., 2024), signal propagation (Noci et al., 2022; Arroyo et al., 2025), and over-smoothing (Di Giovanni et al., 2023). In particular, the depth and large context of modern LLMs seems be sufficient to cause representational collapse, which can be slowed down by making certain heads inactive: a behaviour which attention sinks directly promote (see Figure 1). The deeper the architecture, the more inactive heads have to become to ensure that representations stay sufficiently separated throughout the entire architecture to avoid collapse.
Contributions. We summarise our main contributions.
In Section 3, we argue that attention sinks are useful to control over-mixing. We connect this to existing theoretical phenomena such as rank collapse, representa- tional collapse, and svua-siuas.ing We show how our mathematical intuitions manifest in Gemma 7B.
In Section 4, we further support our over-mixing hypothesis. Our refined over- squashing analysis suggests that bigger models and models trained over longer contexts should have stronger sinks. We verify both hypotheses using the LLaMa 3.1 family of models and through our own pre-training runs. We find that a staggering 80% of attention heads form strong sinks in LLaMa 3.1 405B.
In Section 5, show that, expected from our hypothesis, attention sinks form regardless of how (bos) is included during pre-training, Fixing (bos) in pre-training the first token, however, does impact how the model constructs the sinks.
2 Background
In this work, we focus on decoder-only Transformer models (Radford al., 2018), that apply causal mask to the attention mechanism. These are by far the most common types of Transformer models that are used in modern LLMs to date (Gemma Team et al., 2024; Dubey et al., 2024). We follow the notation of Barbero et al. (2024), but we importantly also consider a model with 2 1 attention heads:
(k
(q (EA)
(EN)
(FA) exp
Pij
2
(EM)
Σ
a
(1
w
w(M (6)
with a
Exesi exp k ({(
(E,)
Kw
(Eh)
Piw
(6)
= w
(Eh)
(8)
z
z
+ v
WEHl
(1+1)
(1)) (z!")
(6)
(t)
zz
2

---

Published as a conference paper at COLM 2025
where vp() is non-linearity, k takes queries q, keys k, and positional encodings Pij to produce an activation, W( E Rdxd and w(() E IRH are learnable matrices, and + represents a direct sum (concatenation). To simplify notation, we ignore the layer normalisations without loss of generality. The sum ranging over j such that j S i is due to the causal mask. If we represent the attention coefficients via matrix A such that A (Ch) (Eh) this condition is equivalent to enforcing that A (f.) is lower triangular. An ij
LLM consists of L such blocks, with L often called the depth. New tokens are generated
autoregressively by considering the final layer representation v of the last token and mapping it to distribution over the token vocabulary. A new token is sampled from this distribution, and the process repeats.
Attention Sinks. The term atlention sink was first used by Xiao et al. (2024) to indicate tokens that, although likely to have limited semantic meaning, attract a large portion of the attention within an attention head. They showed that it is important to keep such tokens when computing sliding-window attention to retain performance. The attention sink phenomenon has been however observed much earlier under different names, for instance, Vig & Belinkov (2019); Vig (2019) observed the sink phenomenon calling it 'null attention', and Clark et al. (2019); Brunner et al. (2019) observed similar 'no-op' phenomenon in the model BERT (Devlin al., 2019).
incoder-Transforee
Recent work by Gu et al. (2025) empirically ablated a number of components in the pre- training setup to study under which conditions attention sinks occur. While attention sinks are a broader term, in our work we focus on attention sinks forming exactly at the first token, as this is the most common pattern by far. To measure the presence of the sink we (PM) > c): The follow the metric proposed by Gu et al. (2025) sink rate LH En, 1 ( (, 1.
quantity measures the proportion of heads in the entire model that attend to the sink on average with a coefficient of at least e, where we also set e = 0.3, unless explicitly stated.
Several key works have investigated how attention sinks are constructed. Cancedda (2024) show, from spectral perspective, that specific subspaces are responsible for the creation of attention sinks. Sun et al. (2024) show that massive activations seem to be responsible for the creation of attention sinks. Barbero et al. (2025) reverse engineer a specific attention head to show that high-norm bands in the queries and keys help with the formation of attention sinks. Such works all imply that large activations are helpful in creating attention sinks.
In this work, we are interested instead in showing not only how attention sinks appear but why they are useful, particularly when it comes to long context modelling. In fact, we critically argue that this learned behaviour is necessary for effective long-context learning. We believe that that this constitutes new perspective that nicely complements existing works.
Propagation of information in Transformers. Many works have studied how information propagates in (deep) Transformers. In the linear case, phenomenon known as rank collapse has been heavily studied (Dong al., 2021; Geshkovski et al., 2023; Wu al., 2024; Naderi et al., 2024). Such works show that repeated application of attention layers projects the values into a vector space that has rank 1. The same phenomenon has been observed and heavily studied in Graph Neural Networks and is often called over-smoollting (Di Giovanni et al., 2023; Keriven, 2022). The key intuition is that attention matrices 'mix' information, and repeated mixing converges to space that is uninformative. Recently, work by Wu et al. (2024) has extended such analysis to causal mechanisms and thus decoder-only Transformers, describing how causal masking affects the convergence.
Importantly, Veličković et al. (2024) proved that, when generalising to sufficiently longer contexts at inference time, global attention matrices cannot remain sharp, and will therefore always converge to "pure mixing". The culprit for this is tokenisation, which imposes bound on the logit spread feeding into the softmax. While sparsifying attention can improve sharpness, the tradeoffs involved are not yet well understood (Vitvitskyi et al., 2025).
A related behaviour that occurs in decoder-only Transformers is over-squashing Barbero et al. (2024) showed that the decoder-only Transformers are more sensitive to tokens coming
3

---

Published as a conference paper at COLM 2025
sooner in the sequence due to the causal mask. They also describe a phenomenon called representational collapse, in which over long sequences the Transformer tends to destroy information of tokens coming towards the end of the sequence.
Together, these effects point towards two difficulties: Transformers tend to 'over-mix' their information, both as they become deeper (Barbero et al., 2024) and as they ingest longer context (Veličković et al., 2024). In this work, we connect these ideas to the attention sink phenomenon. We show that the specific attention sink pattern is used by the Transformer in an attempt counter the collapse of representations and keeping them meaningfully distant from each other. As a further contribution, we provide interesting connections between rank collapse, representational collapse, and ever-squashin, which might be of independent interest.
3 Transformers blocks need to avoid over-mixing
We present mathematical insights that aim to understand why the formation of attention sinks can be useful or even necessary. We start by connecting rank and representational collapse, showing that rank collapse is stronger condition than representational collapse. We then derive stronger over-squashing bounds and use these results to make predictions on what factors might influence the formation of attention sinks. We perform some experiments on Gemma 7B to verify our intuitions.
(f) Rank collapse is a stronger condition than representational collapse. We let v be the value vector the f-th Transformer block of the i-th token and v(O) the n x matrix that collects the value vectors together. We use the same definition of rank collapse' from Wu et al. (2024), which in our notation can be written as:
vi() 1 viO - [| 4 <.
(1)
Rank collapse can therefore be seen how far the representations ( are from the 'average' representation ( = 11 v(L), In Transformers with no residual connections and non- linearities it is well-known that this quantity decays exponentially with depth. Barbero et al.
(2024) define representational collapse as?:
<A
(2)
for some token sequence in which the tokens n 1 and are repeated and the underlying sequence or 'prefix' grows. The two measures are seemingly very related; in fact, we start by showing that rank collapse implies representational collapse (we provide proofs for all statements in this section the Appendix Section D). The converse, instead, is not true, meaning that the two quantities are measuring distinct effects.
Proposition 3.1 (Rank collapse implies representational collapse.). If vf( - 1111 TVT <
A/2, then [ (I. - -1 <
We highlight that this does not mean that representational collapse is not a useful quantity to consider. The condition of rank collapse is much stronger and only really occurs in linear systems Wu et al. (2024); Dong et al. (2021); instead, representational collapse can be studied in non-linear systems, as done by Barbero et al. (2024). Interestingly, rank collapse is statement regarding depth of the model, while representational collapse is statement
We believe that rank collapse might not be the best name for this phenomenon as the rank can remain full for any A > O, but use this terminology Note that in this work we use the e norm instead of e for convenience. This detail is not important as the two norms are 'equivalent' one has the bound ||x||| x /d [[x|2 for e IRd.
matter of consistency with previous works.
4

---

Published as a conference paper at COLM 2025
related to context length 3. We report collapse studies in the Appendix (Section B) for the interested reader.
These phenomena are consequence of a catastrophic over-mixing effect that is caused by either the depth or context length growing too much and point towards the need for Transformer to learn defense mechanisms to counter such effects. The remainder of this work will then focus on exploring how the attention sink phenomenon one such mechanism.
3.1 Sinks way to avoid over-mixing
natural way to measure the amount of mixing is via the norm of the following Jacobian:
dv
(T)
dv
dv
(1)
Σ
(3)
ov:
Sg...k
L-11
dv (0]
Kz-1
The quantity | measures how sensitive the token j's representation at layer is to small perturbation of token i. This is similar to the analysis and justification of vanishing gra- dients in recurrent models. Intuitively, Transformers should be able to control this quantity, or they risk running into issues such as rank collapse or representational collapse. To moti- vate and set the foundation for the remainder of our work, we extend the over-squashing results from Barbero et al. (2024) to now include multi-head attention. We note that just like Barbero et al. (2024), our bounds do not push the derivatives through the softmax function in order to derive a more interpretable result (see Appendix Section for more details).
Theorem 3.2 (More detailed over-squashing bounds.). Let Cmax > 0 be the greatest Lipschitz constant of any layer of the Transformer, be the number of heads, and 8, 1 iff i = jnd
(eh) otherwise. Let k E Pi-+j be a path from to of length L. Set n = Σn aij
8
+ H. Then:
lav (L)
[0) S Clixx
Σ KEP,nj
(L) all (1-1) 1) jike- Re-nkie-
a
Kid"
(1)
(4)
The bound tells us that the weighted paths across atlention heads impact the sensitivity between tokens and j. From this perspective, the effect of attention sinks clear: attention sinks help to control the impact a perturbation can have on the output, as depicted in Figure Interestingly, the bound shows how the sensitivity is controlled by the depth, number of heads, and context length. We therefore expect stronger sinks to appear when the model becomes larger or is trained on longer context, better control the sensitivity. We will provide in later sections supporting evidence on both frontier LLMs and on LMs trained from scratch.
3.2 How sinks help prevent mixing in Gemma 7B
To validate the intuition, we perform a perturbation analysis in Gemma 7B. To simulate small perturbation token space, we perturb a single token in sequence-for example, changing 'greatest' to 'best' (see Appendix Section C for details). We then measure how the representations change across the model when the attention sink present versus when it is not. In Figure 2 (a), we show how the perturbation behaves when (bos) is kept and in (b) when is removed. much more which occurs a consequence of the higher mixing rate. We note that this
clear that the perturbation in (b) affects the representations
experiment is a way to estimate the behaviour of | acts as an interesting way of measuring vvuushig in a meaningful way, something that was left an open question by Barbero et al. (2024).
We believe the above method also
We highlight related work by Naderi et al. (2024) that studies rank collapse in 'width' through random matrix theory methods.
5

---

Published as a conference paper at COLM 2025
Perturbation (=bos>))
Perturbation (No <b05>)
1000
1400
1200
1000
500
810
600
500
20
400
400
200
220
20
Thon 121
220
80
100
-20
Thian Indea
Figure 2: Effect of token perturbations on Gemma 7B. Left/Right: With/withone without (bos).
In Figure 3, we illustrate how the removal of the (bos) token in Gemma 7B causes the attention maps to become much smoother. This has the effect of increasing the values of Jajll this further supports the claim that the existence of (bos) provides mechanism to attenuate the mixing, as previously shown in Figure 2.
127
'/
1 181
'7
Ararion lieere coan
Herree capss
Figure 3: Attention patterns of four heads in Gemma 7B. Top/Bottom: With/withond (bos).
Attention sinks help construct approximate no-ops. To investigate more deeply the sink phenomenon, we also examine a specific head in Gemma 7B, which was already studied in Barbero et al. (2025). This attention head pattern occurs in the first layer of the model and appears to activate specifically when there is an apostrophe token as the previous token, as shown in Figure 4 (a). The head has essentially two operating modes": firing very sharply when activation condition is met, and attending to (bos) otherwise. This head can therefore seen as an implementation of an 'if-else' statement.
If we plot the corresponding value vector norms, as shown in Figure 4 (b), we see that the norm of the value corresponding the (bos) token the smallest, whereas it is largest for the apostrophe value which seems to be what the head prefers to focus on. This intuitively elucidates what we believe an interesting operating mode of attention heads: updating the token embeddings as little as possible by defaults, and instead updating them significantly when the head wishes to be operating. The attention sink in the (bos) token seems to provide a direct mechanism to construct this 'approximate no-op', which also been pointed out by other works (Gu et al., 2025). We also find it interesting that this head is a real-world example of the theoretically studied -arrampBackcopy task from (Guo et al., 2024).
AAt least for in-distribution examples; given Veličković et al. (2024), all of these mechanisms will eventually disperse with increasing context length. 'Placing high attention coefficient attention mechanism will be low norm-and and hence it will have less significance when summed with the residual stream from the previous block.
value vector with low norm implies that the output of the

---

Published as a conference paper at COLM 2025
Attention Heatmap (<bos>))
L2 Norms Taken Representations
1.0
20
Attenton Meight
Norm
15
10
0.4
5-
-0.2
<bop Hello
ve
bean
hupe
that
you
re
doig
sell
0.0
Takens
(a) Apostrophe head attention pattern
(b) Apostrophe head value norms
Figure Apostrophe head attention pattern (a) and value norms (b). This attention head is real-world example of the theoretically studied -araappy task from Guo et al. (2024).
Summary of the Section: Attention blocks tend to mix information, but the model needs a mechanism to control the rate of mixing to aooid pathological issues. We showed how the use of (bos) seems help mitigate how perturbations spread in the computational graph.
4 How does over-squashing predict attention sinks?
We now investigate how our over-squashing and mixing insights predict the formation of sinks in trained models of different sizes. Our over-squashing bounds tell us that perturbative effects will be larger in models that are trained on longer context and that are larger. Consequently, we expect this to affect sink formation our insights are well-aligned with the underlying phenomenon.
4.1 How does context length affect sink formation?
While our empirical observations in pre-trained LLMs are certainly indicative of the (bos)- powered mechanism, we are not able meaningfully account for the way in which they were trained, or the data they observed, when reasoning about various artifacts such as the attention sink. Therefore, we follow the pre-training setup of Gu al. (2025) and evaluate the effect of context length on the sink formation in LMs of roughly 120M parameters (see the Appendix Section A.1 for details). The over-squashing intuitions suggest that the model should learn stronger sink longer context naturally leads to stronger mixing (Veličković et al., 2024).
Importantly, we vary the pre-training context length, making sure that each training step processes the same amount of tokens such that the total tokens processed by each model is 5B. In Figure 5 (a) we see that, after pre-training, the attention sinks indeed seem to be much more prevalent for models trained on longer contexts, and nearly non-existent for very short- context-trained models. In Figure 5 (b), we show that this trend appears to be cumulative throughout training-initialle
there are no attention sinks; the rate at which sinks develop is generally increasing with the context length (until saturating). For completeness, we also report the validation loss curves of these models the Appendix (Figure 7), showing that they all achieve comparable validation losses during training. This is another indication that the emergence of sinks might be necessary side effect of training capable models at ever-increasing context lengths in order to avoid over-mixing issues.
7

---

Published as a conference paper at COLM 2025
Sink Metric vs Context Length
Sink Metric Training Steps for Context Length
42-
Leeh
Conteat Langa = 256
Contest Langa 1024 Content Langzi 2048
32-
15
18-
2503
5000
1500
16000
12500
15990
17500
2000
Traiiing Staps
(a) Context sink histogram
(b) Context sink curves
Figure 5: Pre-trained model Sink Metric (%) after training (a) and during training (b) trained
at different context lengths.
4.2 Attention sinks in the LLaMa 3.1 family
Next, we examine the LLaMa 3.1 family of models as it offers an interesting test bed for models of vastly different sizes. For instance, the smallest SB model has 32 layers and 1,024 attention heads, while the largest 405B model has 126 layers and 16,128 heads (see Table 1). The underlying assumption is that as they are part of the same family of models, the training pipelines these models have undergone will be similar possible and allow to study pre- pre-trained models as their size grows. This provides an interesting way to check how attention sink patterns differ between models of different sizes.
Model
Total Layers
Heads per Layer
Total Heads
Sink Metric (e 0.8)
LLaMA 3.1 8B
32
32
1,024
45.97
LLaMA 3.1 70B
80
64
5,120
73.49
LLaMA 3.1 405B
126
128
16,128
78.29
Table 1: Relevant architectural details of L.LaMA 3.1 models.
In Figure 6, we show the results of our study, where we plot the sink metric for each head. We compute the metric using the same procedure and prompts as the context length experiments and sort them such that the lowest sink heads from the left. It is evident how the smallest 8B model seems to be significantly more active than the larger models. Interestingly, the middle layers appear to be much more active, which has also been observed in other work (Skean et al., 2025). This suggests that the sink metric could also be used as a proxy of layer activity. In Table we report the summary metric for each model. As the models get larger it seems like the sinks become stronger, in accordance with our intuitions from Section 3.
100
5-
40
Sink Viluee (50
30
15
50
69
30
120
20
(corted)
(sortad)
Heads Esorted)
Figure Sink metric percentage (e 0.8) for (bos) for all heads over 170 different prompts for the LLaMa 3.1 family of models. Red indicates strong sink presence.
8

---

Published as a conference paper at COLM 2025
Summary the Section: We supported our theoretical insights that larger models and models trained on longer context should have more attention sinks to better control information-miixing.
5 Is (bos) in any way special?
In this section we aim to answer final important question: is there something special about the (bos) token and its relationship to the sink formation. Intuitively, we would expect that the answer is no because if sinks exist simply to prevent mixing, then the only important property the sink should have is that it exists at the first position in the sequence to help prevent mixing of the subsequent tokens.
To study this behaviour, we pre-train using a number of different strategies (see Figure 8 in the Appendix for illustrations of the different strategies). To summarise our findings from Table 2, the model is trained with (bos) always appearing at the first token, removing (bos) during inference removes the attention sink, i.e. the model relies on the (bos) token to form the sink. Instead, if there no (bos) during training, the sink forms at the first token regardless, but is slightly weaker. Removing (bos) in model that was trained with (bos) present greatly reduces performance. This seems masking and intra-doc masking. This suggests that choices in pre- pre-training have direct impact on how the attention sinks are constructed by the model, but that their formation during training seems inevitable. This also validates our intuition that attention sinks seem to form always at the first token, regardless of the pre-training packing strategy used. For more details, we point the reader to the Appendix (Section A.3).
consistent over both causal
Attention Masking
(bos)
(eos)
Inference
Sink Metric (%)
Valid loss
Causal
No
Yes
(bos) text
65,10
2.69
Causal
No
Yes
text
65.15
2.70
ffxeel+fixed (bos)
Yes
Yes
(bos) text text
90.84
2.69
fixed (bos)
Yes
Yes
0.05
7.56
Intra-doc
No
Yes
text
28.23
2.67
Intra-doc
Yes
Yes
(bos) + text
83.33
2.67
Intra-doc
Yes
Yes
text
50.24
2.68
Intra-doc + fixed (bos)
Yes
Yes
(bos) text text
90.56
2.67
Intra-doc + fixed (bos)
Yes
Yes
0.00
7.78
Table 2: Impact of data packing and attention masking on the formation of attention sink. Here denotes that (bos) and (eos) are the same during the inference.
Impact of attention sinks on downstream performance For completeness, we also check how the removal of attention sinks at inference time affects performance on downstream benchmarks. We report the results Table 3. We find that removing the attention sink (i.e., removing the (bos) token) in Gemma 7B at inference time consistently lowers the performance, and this drop particularly pronounced on the long-context ruler benchmark in accordance with our prediction that attention sinks are particularly useful for long- context. We highlight that our results suggest that one should take great care in how the (bos) token is handled at inference, as the model's predictive capabilities seem to be greatly impacted, especially in the long-context regime. These insights are in agreement with earlier work on streaming attention (Xiao et al., 2024) that found that the presence of attention sinks perplexity.
is important for
Summary of the Section: During the LM pre-training, when (bos) is fixed in the first position within the context, LMs employ (bos) to avoid over-mixing Otherwise, LMs employ the first token (which need not be (bos)) to avoid over-mixing,

---

Published as a conference paper at COLM 2025
Benchmark
w/ (bos)
w/o (bos)
ARC-e
80.77
28.49
ARC-c
53.50
22.53
PIQA
81.72
52.77
SIQA
48.26
34.70
HellaSwag
80.61
27.35
Winogrande
72.85
49.41
Ruler (4096 context)
82.57
0.00
Table 3: Effect of removing the attention sink (i.e., the (bos) token) with Gemma 7B at inference time on downstream benchmarks. The performance drops significantly, even dropping to 0 on the long-context ruler task.
Conclusion
In this work, proposed new perspective attention sinks, arguing that they emerge as a natural response to over-squashing and over-mixing in transformer architectures. Our analysis shows that directing significant portion of the attention to the (bos) token helps the model become less sensitive to token perturbations. As models scale up in size or are trained on longer contexts, they inherently become more vulnerable such perturbations and the sinks become stronger. We believe not only that this work helps to understand why such patterns are in fact useful, but also serves as an interesting further application of theoretical results on over-squashing and rank collapse that could be of independent interest. Our data packing exploration also helps elucidate how the way pre-training is performed can heavily impact how attention patterns behave.
We hope that this new perspective will inspire future work aimed at deepening our under- standing of the underlying mechanisms learned by transformers, ultimately guiding the development of more robust and efficient architectures.
Acknowledgments
We would like to thank Andras Gyorgy (Google DeepMind) and Simon Osindero (Google DeepMind) for their valuable comments on our work. We are grateful to Yonatan Belinkov (Technion) for suggesting additional references.
References
Álvaro Arroyo, Alessio Gravina, Benjamin Gutteridge, Federico Barbero, Claudio Gallicchio, Xiaowen Dong, Michael Bronstein, and Pierre Vandergheynst. On vanishing gradients, over-smoothing, and over-squashing in gnns: Bridging recurrent and graph learning. arXiv preprint Xix::2500 10818, 2025.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João Madeira Araújo, Oleksandr Vitvitskyi, Razvan Pascanu, and Petar Veličković. Trans- formers need glasses! information over-squa in language tasks. Advances in Neural Information Processing Systems, 37:98111-98144 2024.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview net www.reeeeeeeeeeoouneiiiiiee nett/oormmidd-Gtwarrr5555
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Adoances in neural information processing systems, 33:1877-1901, 2020.
10

---

Published as a conference paper at COLM 2025
Gino Brunner, Yang Liu, Damian Pascual, Oliver Richter, Massimiliano Ciaramita, and Roger Wattenhofer. On identifiability in transformers. arXiv preprint arXio:1908.04211, 2019.
Nicola Cancedda. Spectral filters, dark signals, and attention sinks. ar arx preprint ar XrXiv:2402.092211 2024.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT's attention. In Tal Linzen, Grzegorz Chrupata, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACL Workshop BlackboxNI.P; Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 21116653vv/W199 https://aclanthol..0g org/W19--8288/
URL
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language techmologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
2019.
Francesco Di Giovanni, James Rowbottom, Benjamin Paul Chamberlain, Thomas Markovich, and Michael M. Bronstein. Understanding convolution on graphs via energies. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International conference machine learning, pp. 2793-2803, PMLR, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint ar arXiv:2407.21783, 2024.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrreviw.. net/ forum?id==uNrFpoPPyy
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXio preprint ar a/Xit:2403.08295, 2024.
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical perspective on transformers. Xrv preprint ar Xri::2232.10094, 2023.
Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Wang, and Min Lin. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview. net nett:Forrummdt778NNATTTE..
Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael I Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in 1lms. arXiv preprint ar Xrie:2410.11835, 2024.
Nicolas Keriven. Not too little, not too much: a theoretical analysis of graph (over) smooth- ing. Advances in Neural Information Processing Systems, 35:2268-223 22811, 2022.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. IntactKV: Improving large language model quantization by keeping pivot tokens intact. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 7716-7741, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/vi 0024.findings-aa 460. URL https://aclanteee hwtp://olaaaaatooo...//oo org/2024. findings-acl. 460/.
11

---

Published as a conference paper at COLM 2025
Alireza Naderi, Thiziri Nait Saada, and Jared Tanner. Mind the gap: spectral analysis of rank collapse and signal propagation in transformers. arXio preprint ar XrXiv:2410.07799, 2024.
Lorenzo Noci, Sotiris Anagnostidis,, Luca Biggio, Antonio Orvieto, Sidak Pal Singh, and Aurelien Lucchi. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. Advances in Neural Information Processing Systems, 35:27196-27211, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, al. Improving language understanding by generative pre-training. 2018.
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-depths: Dynamically allocating compute in transformer- based language models. Xrxiv preprint ar arXiv:2404.02255 022258 2024.
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations language models. Xrviv preprint ar arXiv:2502.02013 2025.
Mingjie Sun, Xinlei Chen, Zico Kolter, and Zhuang Liu. Massive activations in large lan- guage models. In First Conference on Language Modeling, 2024. URL https: https://openreview: netzforum2idd=7ahhfiio
Petar Veličković, Christos Perivolaropoulos, Federico Barbero, and Razvan Pascanu. softmax is not enough (for sharp out-of-distributiion). arXiv preprint ar arXit:2410.01104, 2024.
Jesse Vig. A multiscale visualization of attention in the transformer model. ar Xiv preprint axXiv:1906.05714, 2019.
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in a transformer language model. arXiv preprint ar XrXo::1906.04284, 2019.
Alex Vitvitskyi, Joao GM Araújo, Marc Lackenby, and Petar Veličković. What makes good feedforward computational graph? arvi preprint XXXr:2502200651,, 2025.
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layernorm in transformers. Xrv preprint ar Xixv:2405511871 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. In The Twelfth International Conference on Learn- ing Representations, 2024. URL
Itay Yona, Tlia Shumailov, Jamie Hayes, Federico Barbero, and Yossi Gandelsman. In- terpreting the repeated token phenomenon in large language models. arXiv preprint ar xXXiv:2003.08908 2025.
Yu Zhao, Yuanbin Qu, Konrad Staniszewski, Szymon Tworkowski, Wei Liu, Piotr Mitoš, Yuxiang Wu, and Pasquale Minervini. Analysing the impact of sequence composition on language model pre-training. arXiv preprint 21t::2202.11999,, 2024.
A Experimental Details and Additional Results
We provide here additional details pertaining our experiments in the main section of the work.
A.1 Pre-training Experimental Details
For the synthetic pre-training runs, we use the same setup as the one used by Gu et al. (2025). We train LLaMa2-style LMS with roughly 120M parameters. For the packing experiments train on 30B tokens, while for the context length ablations we train on 5B tokens. To perform the experiments, we adapt the codebase which is released under an MIT license at https://get thg.bbbb..t.t....m//eee com/ com/sail-sgee wwalllss///tteettonnn Sink. This allows us to have results that are
12

---

Published as a conference paper at COLM 2025
consistent with the large number of ablations that have already been conducted by the original authors. For reference, a single training run on 5B tokens takes up to 24 hours depending the experimental setup single NVIDIA H100.
A.2 Additional Context Length Experimental Results
For completeness, we provide in Figure 7 the training loss curves for the context length ablation experiment. We find that the models trained at different context lengths achieve similar loss curve patterns. We suspect that longer training would cause the sink patterns to become sharper as this seems to be the trend as training continues. It is clear however from the plots that the context length clearly impacts very heavily the formation of the sinks, with essentially no sink formation at context length of 128 and much stronger formation for the longer contexts.
Sink Metric vs. Training Steps for Contect Length
Loss vs. Training Steps for Contest Length
Langth 128
Content Length 128
Contest eegth 256
65
Contest Contest Length 256
Contesti eighh 512
Contest Length 512
Longex - 1024
Context Length 1024
Contect Length 2045
Corbent Langtth 2048
5.5
5-
33
19000
Lodoo
Thaining Stepk
Thairing Steps
(a) Context sink histogram
(b) Context sink curves
Figure 7: Context sink curve (a) and their respective loss curves (b).
A.3 Packing
We now provide additional intuition for our packing experiments. During LM pre-training, all documents in the corpus are concatenated and chucked into sequences with a fixed context length following Brown al. (2020), as shown in Figure 8. Then (bos) (before each document) and (eos) (after each document) are introduced to mark the boundaries between two consecutive documents. Typically, one only needs to adopt either (bos) or (eos) during LM pre-training. In this case, (bos) and (eos) are the same during inference. To pre-train LMs, casual masking within the context is adopted, resulting in the tokens in each document being able to attend to the tokens in the previous document within the context. This motivates the proposal of intra-doc masking (Dubey et al., 2024; Zhao et al., 2024), which ensures that the tokens document ((bos) and (eos) inclusive). In this case, we consider two scenarios: (i) adding (eos) only, and (ii) adding both (bos) and (eos).
only attend to previous tokens within the same
13

---

Published as a conference paper at COLM 2025
Seeep
<bos
<eep
bbss
Yeos>
Scos>
Context length
Causul masking
Intra-doc masking
<bos>
-0062
<bos>
Scos
longth
Causal masking fixed
Intru-doc masking fixed <hest-
Figure 8: Illustration of data packing and attention masking setups we consider in our packing ablations in Section 5.
Fixing (bos) in the first first position. Due the importance of the first token, we are interested in fixing (bos) in the first position within the context. Then only (eos) is added to mark the boundary between two consecutive documents. As shown in Figure 8, we modify the attention masking to ensure all tokens within the context are able to attend to the (bos) token regardless of intra-doc masking or not.
The effect of (bos) token. We conduct experiments employing LLaMA2-style LMs with 120M parameters, a training corpus of 30B tokens, and a context length of 2048. We sample 100 sequences (with token lengths between 512 and 2048) in the validation set and measure the auto-regressive loss and attention sink metric for the first token position following Gu et al. (2025). As shown in Table 2, first consider the scenarios of causal masking, if (bos) is only used as the boundary for different documents, it has little impact on sink formation. Meanwhile, if we fix (bos) in the first position within the context, the attention sink is significantly enhanced. But without (bos), the attention sink disappears, and the model performance drops drastically. When it comes to intra-doc masking, adopting (bos) in the inference significantly improves attention sink. But without it, LMs still have attention sink on the first token. during inference will make LMs experience summarize, when (bos) is fixed in the first position within context during the pre-training, LMs employ (bos) to avoid over-mixing. Otherwise, LMs employ the first token (which need not be (bos)) to avoid over-mixing.
fix (bos) in the first position during pre-training, removing
attention sink and drop in performance. To
B How does BoS help avoid over-mixing?
At each transformer layer, the learned attention matrix mixes different tokens which are then fed into the MLPs to construct intermediate representations, While mixing is crucial to capture meaningful semantics in sentences, the spectral characteristics of attention matrices with causal masking leads to the collapse of token representations. This phenomenon has been termed rank collapse (Dong et al., 2021), and is closely related to the oversmoothing phenomenon in GNNs (Di Giovanni et al., 2023) and signal propagation (Noci et al., 2022; Arroyo et al., 2025).
Here, we quantify token similarity by measuring the deviation of the representations from their mean, using the distance measure presented in (1), which denote (X).. μ
14

---

Published as a conference paper at COLM 2025
measures how far X is from its 'mean matrix'. As an initial empirical test, we measure the representational distance to the mean at the final layer of the Gemma 7B model for progressively longer prompts asking the model to add numbers. We provide more details on these prompts in Appendix C. The results are shown in Figure 9 below.
10000
Representational Distance, (4X
<bos>
8000
4000
2000
10'
10a
10a
Terms
Figure 9: Representational distance as function of prompt length for models with an without BoS token for Gemma 7B.
iiteggon, San
Hétteran,
dban Hinogran, S.
30-
Nom
13 kmm
12 Nmm
(a) sum terms.
(b) 25 sum terms.
(c) 50 sum terms.
330
1750
150
1259
L00
12C
156
Nom
(d) 100 sum terms.
(e) 250 sum terms.
(f) 1000 sum terms.
Figure 10: Histogram of token norms for sums of increasing length.
Figure 4 illustrates that excluding the BoS token when prompting LLM significantly affects the smoothness of its representations. Specifically, as the prompt length increases, the divergence between representations with and without the BoS token grows. Notably, prompts that include the BoS token exhibit considerably higher dispersion around their mean. This dispersion can also be visualized by plotting the histogram of the norms of the latent representation of each token, as shown in Figure 10.
The empirical evidence in Figure 4 serves to clarify the effect of the (bos) token on the formation of intermediate representations. The disproportionally low norm the values associated with the BoS token implies that if the model directs all attention to that token, the resultant information essentially nullified - effectively bypassing the mixing process intrinsic to attention. In such scenarios, only the residual pathway carries forward any
15

---

Published as a conference paper at COLM 2025
information. This process counteracts the over-mixing and rank collapse induced by the spectral properties the attention matrix with causal masking (Naderi et al., 2024). Notably, this strategy mirrors the mixture-of-deptle approach introduced in Raposo et al. (2024), but without the efficiency gains achieved through gating around the attention operation.
C Additional Details on Prompts Used
We include additional details on the prompts used for the experiments presented in the paper.
C.1 Robustness to Perturbations Prompts
Here, we present the prompts used to obtain the results in Figure The original prompt is
William Shakespeare[a] (c. 23[b] April 1564, 23 April 1616)[c] was an English playwright, poet and actor. He is widely regarded the greatest writer in the English language and the world's pre-eminent dramatist. He is often called England's national poet and the 'Bard of Avon' (or simply 'the Bard'). His extant works, including collaborations, consist of some 39 plays, 154 sonnets, three long narrative poems and a few other verses, some of uncertain authorship. His plays have been translated into every major living language and are performed more often than those of any other playwright. Shakespeare remains arguably the most influentian writer in the English language, and his works continue to be studied and reinterpreted.
while the perturbed prompt
William Shakespeare[a] (c. 23[b] April 1564, 23 April 16]6) was an English playwright, poet and actor. He widely regarded as the best writer in the English language and the world's pre-eminent dramatist. He is often called England's national poet and the 'Bard of Avon' (or simply 'the Bard'). His extant works, including collaborations, consist of some 39 plays, 154 sonnets, three long narrative poems and a few other verses, some of uncertain authorship. His plays have been translated into every major living language and are performed more often than those of any other playwright. Shakespeare remains arguably the most influem influential writer in the English language, and his works continue to be studied and reinterpreted.
Here, the perturbed word is written in purple. The modification critically keeps the number of tokens constant and chosen specifically to be 'small' perturbation.
C.2 Smoothing Effects Prompts
To empirically investigate the smoothing effects in transformer representations, we designed set of prompts that probe the model's ability maintain distinct token representations over long contexts. In these prompts, the model is asked to calculate sum of numbers that gets progressively longer (denoted with Sum Terms). our experiments, we compare the effects of including versus omitting the (bos) token on the smoothness of the resulting representations. One representative example of these prompts is:
Could you add these numbers 10+06+03+10+08
C.3 LLaMa Sink Prompts
For the LLaMa sink prompts, we use the dataset provided by Gu et al. (2025) which is found at https://gi ttt::///ihhb.... com/sail-sg/Attentiie http://teenioonSnnnnllooooo wwi./bbll/main/tttttaaa
probe weleevaaid..p jsonn. We evaluate on all of the 170 samples and measure the sink metric on the first T = 64 tokens remain consistent with Gu et al. (2025). Below we report a few excerpts:
(i) Role of the immune system in cardiac tissue damage and repair following myocardial infarction. AnThe immune system plays crucial role in the initiation,
16

---

Published as a conference paper at COLM 2025
development, and resolution of inflammation following myocardial infarction (MI). The lack of oxygen and nutrients causes the death of cardiomyocytes and leads to the exposure of danger-associated molecular patterns that are recognized by the immune system to initiate inflammation. At the initial stage of post-MI inflammation, [...]
(ii) request_number PS-BPA and 118Ynfirstname Steve\nlastname Marshall -mail: marss@perkinsccie. com undirected _t:: Peter J. Burger - LP-7XnOffice of General Counsel Vnnineeiilnnnn 11e Power Administration'nn .0. Box 3261 InPortlad
Oregon 97208 \nexhibit -wp-02-e-: BPA-78\npage _numbers: 1-8Anrequest _tett: [...]
(iii) <A Engineering
HREF= (htt:>/http:/fi/im/)
FFim//thFinnahh...m com
@ Chicago\nfinancial Workshop</A>\n \n
&
Risk Management
NEW 60-65\% OFF 2001 subscription for RISK MagazineXnfor members of Bachelier Finance Society. \nBecome Member NOW! [...]
(iv) ubuntu-au 2011-10-24An <ikt> Hi all :)\n <gggs> hi ikt An <ikt> gggs: whatcha up to?\n <gggs> right now?'n <gggs> talking to you! \n <ikt> :DNn <ikt> thinking of setting up LDAP server at homeAn <ikt> [...]
The 170 texts span a number of different domains, which we believe is important to appro- priately evaluate the sink phenomenon.
D Mathematical Results
We provide here the details of the mathematical results in our paper. We start by showing the connection between representational and rank collapse.
Proposition 3.1 If V(L) - 1111 111Tv(41l < </2, then I (L)
v < A.
Proof. Assume that we have rank collapse at A/2:
(L)
1 11 (L) (v(U)_11Tv1L) I
= vec (v(4)_111
1
| <//2.
n
We now bound the representational collapse quantity:
(L)
(L)
(L)
T
1
(L)
1 Tcvc
Vn
v
1-1
+
2 n
2
v(()
11 v(()
v(())
11 11 v(()
<
vec
+ vec
2
v(L)
1 vvl)
11
n
A.
Thus, rank collapse at A/2 implies representational collapse of at least A.
We now show the results on over-squashing that now include multi-head attention. Like Barbero et al. (2024), we assume for simplicity that the queries and keys are independent of the values. This is to not distract from the main point of the result it would otherwise make the result very messy due to the need to push the partial derivatives through the softmax function. For convenience, rewrite here the update equations that we consider:
17

---

Published as a conference paper at COLM 2025
2
(18)
E
WithY
v
(8)
K
(6)
w
(Eh)
+ +v;"
2
2
ИEH
v ((+1) ( ( (8)
2 (6)
For the dimensions, we let y E R w(()) E Raxd, and w()) E RdHxd This means dv av are in Rdxd Our result applies for matrix norms that are sub-
that the partials multiplicative. This true for induced operator norms, which include most common norms used traditionally in machine learning such as el or l vector norms and their corresponding matrix norms.
Theorem 3.2 Let Cmax 0 be the greatest Lipschitz constant of any layer of a Transformer, H be the number of heads, and 8/ be 1 iff i = j and 0 otherwise. Let k E Pi-j be path from to of length L.
(F,1) 6. H. Then: (8) iij Σn ii Set
lav (L) /
S
CL C
Σ
jik
(L) (1-1)
kg. 18i-2
(1)
(5)
kEP jaj
Proof. We start by decomposing the target partial derivative:
dv (I)
dv! (L)
dv 1)
dv (4)
dv (1)
Σ
KL-1 dv 1)
Σ
ov (0)
k ov; (0)
dv (L-1) 1)
dv; (0)
k
We now focus on bounding the norm of the key sub-partial which ōv, (2) /av; (t- 1)
We note
that the computational graph looks like v (É-1) the partial of interest:
(6-1
(8-1) v, We decompose
z;
z;
2v 1 ((-1)
ov
(
az
(8-1)
ov. dz (1-1)
oz
l
dv
88 1)
āzz
( 1) ov 11 1)
We now bound the two components, where we get:
dv (2)
a
(e)
(2)--1)) + zz [(-11]
< || +1,
az
(8-1)
dz
and
18

---

Published as a conference paper at COLM 2025
àzz
(c-1)
a
w(C-1)
(e 1h)
( 1)
ov,
((-1)
ov
, (6-1)
WeH
a
w((-1)
- HE
( 14b) W L ((--1)
18-1)
dv (f 1)
w((--1 a
(2-1, w
(1 (e) (C-1)
(E-1)
ov
keH
oV! 1J
w((--) 1)
111)
will w/ (1-1)
Laxad!
heH dv: (1-1)
jk
w 1)
( 111)
w(t 116) Laxi Laxd
+ Lando
VIEH
w((-1)
+
ji
111)
w((
Laxe
Laxd
+ LexaeX|
WIH
w((-1))
Σ hell
wife 1.)) ||
+ 0 1.k
ji
w((-1)/
Σ WEH
(f-1.1)
+ 5/
ji
This finally gives us the upper bound:
vv
(IWCC-D) w(C-11maar) 1.hma)))
$ (11 +1)
Σ
ji
+ to,
ov: 12-1)
heH
)
=11++||| +1) [|w[C-D]] | ( 1.maa))
Σ HCH
(1-1
Aji
=Ce KEH
(E Lh)
where we let C₁ be a function of the Lipschiztness of the f-th Transformer block. Putting this all together we reach the desired result:
dv (
dv(())
(L)
dv
Σ
(0)
ov
(L
1)
dv (0)
kg
ov
Σ
dv (8-1)
ky in-ini
of
I Cli
Σ
II
(L-1A)
a
H
ki Innnky
HEH
= -Chax kEPr
Σ
(L)
á
(1)
ikL-1
RL-1ke-2
Above, k E Pi-ij is walk of length from to with steps k,, kL k j that respects the causal attention structure.
19

---

Published as a conference paper at COLM 2025
20


Reviews:
[{"review_id": "7HbeGyIiP0", "reviewer": "Reviewer_x9S1", "summary": "The paper demonstrates high technical quality through rigorous theoretical analysis and empirical validation. The authors provide proofs for key propositions (e.g., Proposition 3.1 linking rank collapse to representational collapse) and derive bounds for over-squashing in multi-head attention (Theorem 3.2). Experiments on Gemma 7B and the LLaMa 3.1 family are well-designed, with controlled variables (context length, model size) to isolate the effects of attention sinks. The perturbation analysis (Figure 2) and ablation studies (Table 2) offer compelling evidence for the proposed hypotheses. The work is methodologically sound, though some experiments (e.g., synthetic pre-training runs) could benefit from broader model diversity.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Novel Theoretical Contribution: The paper provides a unified framework connecting attention sinks to over-mixing, rank collapse, and representational collapse, resolving a key gap in understanding Transformer behavior.\n\n* Empirical Rigor: Extensive experiments validate theoretical claims across multiple model families (Gemma, LLaMa) and training setups. The perturbation analysis (Figure 2) and ablation studies (Table 2) are particularly compelling.\n\n* Practical Implications: Insights into how context length and model size influence sink formation can guide hyperparameter choices in LLM training. The analysis of data packing (Section 5) offers actionable recommendations for pre-training pipelines.\n\n* Interdisciplinary Impact: By bridging theoretical analysis (e.g., over-squashing bounds) and empirical observations, the work enriches both the mechanistic interpretability and architecture design literatures.", "reasons_to_reject": "* Experiments focus on LLaMa and Gemma variants. Results may not generalize to other architectures (e.g., encoder-decoder Transformers or non-autoregressive models).\n\n* The theoretical sections (e.g., Theorem 3.2) are dense and assume familiarity with niche concepts like over-squashing. A more intuitive explanation of key terms (e.g., rank collapse) would broaden accessibility.\n\n* While the paper explains why sinks form, it does not propose methods to mitigate potential downsides (e.g., wasted attention on <bos>). Practical utility could be strengthened with actionable mitigation strategies.\n\n* The impact of attention sinks on task-specific performance (e.g., summarization, QA) is not explored, limiting claims about their real-world implications.", "questions_to_authors": "See Reasons To Reject", "extra_scores": {}}, {"review_id": "qSj0OovFPk", "reviewer": "Reviewer_4hVJ", "summary": "This paper discusses the phenomenon of \"attention sinks\" in LLMs, where a significant portion of attention is consistently directed towards the first token of a sequence (often the <BOS> token). The authors propose that this behavior is not an arbitrary artifact but rather a learned mechanism that helps LLMs, particularly deep and long-context models, to avoid \"over-mixing\" of information, which can cause issues like rank collapse or representational collapse. The paper provides a theoretical basis for this hypothesis by connecting it to existing work on signal propagation in Transformers and extending over-squashing bounds to multi-head attention. This theoretical framework predicts that larger models and those trained on longer contexts would exhibit stronger sink behavior. Empirically, the authors validate their claims through several experiments. Perturbation analysis on Gemma 7B demonstrates that the presence of an attention sink (specifically at the <BOS> token) limits the propagation of changes in token representations across layers. Further experiments show a positive correlation between context length during pre-training and the prevalence of attention sinks, as well as an increase in sink strength with model size (demonstrated on the LLaMa 3.1 family). Finally, the paper explores the specificity of the BOS token, concluding that while explicit training with BOS influences how the sink is constructed, the tendency to form a sink at the first available token is a more general strategy to mitigate over-mixing, seems an inevitable outcome of training under such conditions.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper offers a possible explanation for the attention sink phenomenon, framing it as a solution to the problem of over-mixing in deep Transformers. The work connects attention sinks to established theoretical concepts like rank collapse, representational collapse, and over-squashing. The extension of over-squashing bounds to multi-head attention (Theorem 3.2) provides a useful theoretical contribution that supports their predictions. Also, empirical experiments are provided as evidence.", "reasons_to_reject": "1. While the paper links attention sinks to preventing \"over-mixing,\" this concept itself feels somewhat underspecified beyond its relation to existing phenomena like rank collapse, representational collapse, and over-smoothing. It's not entirely clear if \"over-mixing\" is a distinct, precisely defined problem that the paper newly identifies or if it's a more qualitative descriptor for the negative consequences of these known collapse phenomena. The authors state they \"connect this to existing theoretical phenomena\" , making the precise contribution of the \"over-mixing\" framing less clear.\n\n2. Theorem 3.2 extends over-squashing bounds to multi-head attention. While a valid extension, the resulting insight—that \"weighted paths across attention heads impact the sensitivity\" and that sinks can reduce these path weights—feels somewhat incremental. It's not immediately obvious how this multi-head extension provides qualitatively new predictions beyond the general principle that reducing attention to other tokens (by sinking it) would reduce their influence. \n\n3. The paper argues sinks are \"useful\" and sometimes feels like they are \"necessary\" for long-context learning. If they are indeed a necessary byproduct, further exploration of whether alternative, equally effective mechanisms to control mixing (without such attention patterns) could be learned or designed might be warranted.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TODly6P8AO", "reviewer": "Reviewer_Z1TA", "summary": "This paper investigates the phenomenon of attention sinks in LLMs, where the first token (often 〈bos〉) attracts disproportionately high attention weights. The authors argue that this behavior is a learned mechanism to mitigate over-mixing (rank collapse, representational collapse) in deep Transformers, especially in long-context scenarios. Through theoretical analysis (connecting attention sinks to Jacobian norms and over-squashing) and empirical validation (perturbation studies, context-length ablations, and model-size comparisons), the paper demonstrates that attention sinks act as \"anchors\" to stabilize information propagation. The work also explores how pre-training strategies (e.g., 〈bos〉 placement) influence sink formation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Novel Theoretical Insight: The paper provides a compelling explanation for attention sinks by linking them to over-mixing avoidance, a perspective not thoroughly explored in prior work. The connection to rank collapse (Proposition 3.1) and multi-head over-squashing bounds (Theorem 3.2) is rigorous and insightful. The idea that sinks serve as \"approximate no-ops\" (Section 3.3) to limit unnecessary mixing is particularly elegant.\n\n2) Strong Empirical results: Experiments on Gemma 7B and LLaMA 3.1 models convincingly validate the theory that: First, [erturbation studies (Figures 2–3) show that sinks reduce cross-token sensitivity. Second, Context-length and model-size experiments (Figures 5–6) confirm that sinks emerge more strongly in deeper/longer-context models. The ablation of pre-training strategies (Table 2) adds practical relevance for LLM training.\n\n3) Broader Implications: The work bridges theoretical phenomena (rank collapse, over-squashing) and empirical observations (attention patterns), offering a unified framework.", "reasons_to_reject": "1) While this paper theoretically explains why \"LLMs attend to the first token,\" it remains unclear how this conclusion can practically enhance specific LLM capabilities (e.g., long-context processing, complex reasoning, or instruction following) or address existing limitations. The study would benefit from explicitly connecting its findings to actionable improvements in model design or training.\n\n2) The paper argues that sinks are beneficial but overlooks potential drawbacks. For instance:\n\n     Could heavy attention allocation to 〈bos〉 (e.g., 80% weights) impair the model’s ability to capture legitimate long-range dependencies?\n\n     Does this phenomenon introduce inefficiencies in tasks requiring fine-grained attention to distant tokens?", "questions_to_authors": "1) Attention Sink as a Learned Shortcut? The observed attention sink phenomenon might simply reflect models learning a computational \"shortcut.\" Given that >99% of next-token predictions likely depend only on local context, models could be exploiting this redundancy to minimize unnecessary attention computation. However, this raises a paradox: KV caching still requires storing all historical states despite their sparse utilization. The paper's claim that this \"store-but-ignore\" behavior benefits model performance lacks convincing empirical validation.\n\n2) What fundamentally drives the consistent emergence of attention sinks at the first token (e.g., 〈bos〉) rather than other positions?  Is this driven by positional encodings, causal masking, or optimization dynamics?\n\n3) Could sinks be replaced by explicit architectural interventions (e.g., sparse attention, gating mechanisms, or RNN layers)\n\n4) Do attention sinks hinder performance on tasks requiring long-range dependencies (e.g., coreference resolution)?", "extra_scores": {}}, {"review_id": "b7e8bJt6cX", "reviewer": "Reviewer_Maou", "summary": "In this paper, the authors explore the tendency of LLMs to attribute a substantial amount of attention to the first token of the sequence.\n\nThe authors start by connecting this phenomenon with rank collapse, representation collapse, and over-squashing. \nMoreover, they extend the over-squashing bounds to include multi-head attention, showing that weighted paths across attention heads impact the sensitivity between tokens which in turns makes it clear that attention sinks help controlling the impact a perturbation can cause in the output. \nThis bound also shows that the sensitivity is controlled by the model’s depth, number of attention heads, and context length, giving the intuition than attention sinks should be stronger for larger models or trained with longer contexts.\n\nTo corroborate the claim that attention sinks help controlling the impact a perturbation, the authors observe what happens when removing the <bos> token from the context when using Gemma 7B. This experiment shows that this removal causes the attention maps to become smoother which supports the claim.\n\nThen, the authors also perform experiments to check the influence of the model’s depth and context length (as shown by the over-squashing bounds).\nFirst, the authors train language models with different context lengths. The results clearly show that models trained with larger context lengths lead to stronger attention sinks and that these get stronger along the training process, indicating that the model learns to attribute attention to the first token in order to avoid over-mixing.\nThen, the authors measure the sink metric percentage for models of the LLaMa-3.1 family of different sizes. Results show that the bigger models have stronger attention sinks. \nThese two experiments support the theoretical insights provided by the over-squashing bound.\n\nFinally, the authors analyze the effect of the <bos> token, concluding that removing the <bos> token during inference removes the attention sink, considerably hurting the model performance and that models trained without the <bos> token, have the sink at the first token.", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tThe paper is very clear and well written.\n-\tThe paper is very well structured. The authors start the paper by providing a theoretical perspective of the attention sink phenomenon and connecting it with rank collapse, representation collapse, and over-squashing, arguing that they emerge to control the over-mixing. This is then supported by the thorough experiments.\n-\tThe paper provides another perspective on a very noticed and not well-understood phenomenon that can lead to a deeper understanding of the transformer architecture and guide the development of more efficient architectures.", "reasons_to_reject": "-\tThe only aspect that I believe the paper could be missing is a discussion (or even some small experiments) on how these findings can lead to the development of more efficient architectures, but it’s also totally understandable to leave this to future work.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper offers a novel theoretical and empirical analysis of attention sinks in LLMs, arguing they emerge as a mechanism to prevent over-mixing. The work connects attention sinks to rank and representational collapse and extends over-squashing bounds to multi-head attention. Experiments across Gemma and LLaMA models support the claims.

Pros:
- Strong theoretical grounding (Proposition 3.1, Theorem 3.2)
- Well-executed experiments with real-world benchmarks
- Clear practical implications for long-context LLM behavior

Concerns:
- "Over-mixing" was initially underspecified [4hVJ]
- Over-squashing extension seen as incremental [4hVJ]
- No direct mitigation strategy proposed [x9S1, Z1TA]
- Focused only on decoder models [x9S1]

Author Response:
The authors addressed most of concerns with new experiments, improved explanations, and additional references to mitigation strategies. Rebuttal clarified terminology, strengthened theoretical claims, and added long-context ablations.
Decision: Accept