When Attention Sink Emerges in Language Models: An Empirical View
Authors: Xiangming Gu
Tianyu Pang
Chao Du
Qian Liu
Fengzhuo Zhang
Cunxiao Du
Ye Wang
Min Lin
Venue: ICLR 2025 Spotlight (2025)
Keywords: Attention Sink
Language Models
Empirical Study
Primary area: foundation or frontier models, including LLMs
TL;DR: We conduct extensive experiments to empirically understand when attention sink emerges in language models.

Abstract: Auto-regressive language Models (LMs) assign significant attention to the first token, even if it is not semantically important, which is known as **attention sink**. This phenomenon has been widely adopted in applications such as streaming/long context generation, KV cache optimization, inference acceleration, model quantization, and others.  Despite its widespread use, a deep understanding of attention sink in LMs is still lacking. In this work, we first demonstrate that attention sinks exist universally in auto-regressive LMs with various inputs, even in small models. Furthermore, attention sink is observed to emerge during the LM pre-training, motivating us to investigate how *optimization*, *data distribution*, *loss function*, and *model architecture* in LM pre-training influence its emergence. We highlight that attention sink emerges after effective optimization on sufficient training data. The sink position is highly correlated with the loss function and data distribution. Most importantly, we find that attention sink acts more like key biases, *storing extra attention scores*, which could be non-informative and not contribute to the value computation. We also observe that this phenomenon (at least partially) stems from tokens' inner dependence on attention scores as a result of softmax normalization. After relaxing such dependence by replacing softmax attention with other attention operations, such as sigmoid attention without normalization, attention sinks do not emerge in LMs up to 1B parameters. The code is available at https://github.com/sail-sg/Attention-Sink.

Published as a conference paper at ICLR 2025
WHEN ATTENTION SINK EMERGES IN LANGUAGE MODELS: AN EMPIRICAL VIEW
Xiangming Gu Tianyu Pang"l, Chao Du', Qian Liu', Fengzhuo Zhang!.? Cunxiao Du', Ye Wang 2 Min Lin'
SSaa AI Lab, Singapore "National University of Singapore (guxm, tianyupang, duchao, liuglan, zhangf2, ducx, linmin) ) Esea. com; wangye@comp. nus edu.suu.sg
ABSTRACT
Auto-regressive Language Models (LMs) assign significant attention to the first token, even it is not semantically important, which known as attention sink. This phenomenon has been widely adopted in applications such streaming/long context generation, KV cache optimization, inference acceleration, model quantization, and others. Despite its widespread use, deep understanding of attention sink in LMs is still lacking. In this work, we first demonstrate that attention sinks exist universally in auto-regressive LMs with various inputs, even in small models. Furthermore, attention sink is observed to emerge during the LM pre-training, motivating us to investigate how optimization, data distribution, loss function, and model architecture in LM pre-training influence its emergence. We highlight that attention sink emerges after effective optimization on sufficient training data. The sink position is highly correlated with the loss function and data distribution. Most importantly, we find that attention sink acts more like key biases, storing extra attention scores, which could be non-informative and not contribute to the value computation. We also observe that this phenomenon (at least partially) stems from tokens' inner dependence on attention scores result of softmax normalization. After relaxing such dependence by replacing softmax attention with other attention operations, such sigmoid attention without normalization, attention sinks do not emerge in LMs up to 1B parameters. The code is available https://github..c wttp://l..oo..maoo//aea/eeeleeee Sataall-sggAttttoo--Ssee Attention--ink.
INTRODUCTION
Xiao et al. (2023b) showed that Large Language models (LLMs) allocate significant attention to the initial tokens, irrespective of their semantic relevance. This interesting phenomenon is termed as attention sink and has widespread applications, including streaming/long context generation (Xiao et al., 2023b; Han et al., 2024; Yang et al., 2024), KV cache optimization (Ge et al., 2023; Wan et al., 2024; Wu & Tu, 2024), efficient inference (Zhang et al., 2024b; Chen et al., 2024), model quantization (Liu et al., 2024b; Huang et al., 2024), and others.
seminal of works attempted to understand attention sink. Among them, Cancedda (2024) clarified that attention sink primarily appears only on the first token. They attributed the phenomenon the large norm of hidden states of the first token. This is referred to as massive activations (very few activations exhibit extremely large values compared to others) in Sun al. (2024). Besides, Sun et al. (2024); Yu et al. (2024) observed that attention sink may also appear in several word tokens carrying limited semantic information and having no fixed position. Despite the above research efforts, deep understanding of attention sink is still absent. Therefore, we conduct comprehensive study to investigate when attention sink emerges. We defer full discussions on related work Appendix A.
Based on open-sourced auto-regressive LMs, we show that the first token acts as biases: the angles between the first key and queries of other tokens are typically small, leading to attention sink. Then
Work done during Xiangming Gu's au's internship at Sea AI Lab.
"Correspondence Tianyu Pang and Ye Wang.

---

Published as a conference paper at ICLR 2025
H
D
Feed Forward Network (PFN)
Pre
BOS
808
O
Multi-Heeed Aste-titto (MHSA)
805
E06
EOS
Pre
H LX
C
Figure I: (Left) Architecture of pre-norm transformer block (we highlight the location of post-norm LN using dashed lines). We denote the output of MHSA as O' and the output of FFN as F'. (Right) The packing strategy in the LM pre-training. All documents concatenated with BOS (optional) and EOS tokens as the boundaries. Then is chunked into equal-sized sequences with context length C.
we find that attention sink universally exists in auto-regressive LMs across different inputs, even in the small models or with random token sequences. Additionally, attention sink is observed to emerge during the LM pre-training before continual instruction tuning (Ouyang et al., 2022). This motivates us to focus on the LM pre-training whose objective can be formulated as:
min Ex~pan [C (po (x)()))
In the remaining part of this paper, investigate how the optimization (Section 4), data distribution (Section 5), loss function (Section 6), and model architecture (Section 7) influence the emergence of attention sink. We have the following conclusions:
Attention sink emerges after LMs trained effectively on sufficient training data. It appears less obvious in LMs trained with small learning rates. While weight decay encourages the emergence of attention sink.
The sink position is highly related to the loss function and data distribution and can be shifted to other positions rather than the first token.
Attention sink acts more like key biases, storing extra attention and meanwhile not con- tributing to the value computation. This phenomenon (at least partially) stems from tokens' inner dependence attention scores due the softmax normalization. After relaxing such dependence by replacing softmax attention with other attention operations, e.g., sigmoid attention without normalization, attention sinks do emerge LMs up 1B parameters.
2 PRELIMINARIES ON LMS AND ATTENTION SINK
Let So be an auto-regressive LM with L transformer decoder blocks and X E RTX/ = {x1, x2, ..... Ør) are the input tokens, where each token 2; is one-hot encoding and |V] is the vo- cabulary size of tokenizer V. The LM output is also a sequence Y R = {y) y2. yr (X),,, where y, represents the predicted logits of p p( (x).)).
Transformer blocks. In the forward pass, X is first embedded as H RRxx = XW + P. where WE R/Vix is the learnable word embedding, and d is the hidden dimension. We denote H' E T : - ...,...,hi, < < L to be the output of the I-th block. Each block comprises a multi-head self-attention (MHSA) operation and feed-forwart network (FFN). The block has either pre-norm post-norm structure according to the location of layer normalization (LN) (Ba et al., 2016; Zhang & Sennrich, 2019). Most of LLMs consider pre-norm block, also shown in Figure I(Left):
Txx is the positional embedding.
H' FFN + H² + 0 + H O' MHSA(LN((H
(2)
while the post-norm transformer block is
H' LN (FFN + H' LN + H'- O -MHSA(H'-)) (3)
2

---

Published as a conference paper at ICLR 2025
400
24
320
18-
P2-norm
240
k
12
160
Nigh
6-
0
0
44
8
12
16
20
24
28
32
0
0
4
8
12
16
20
24
28
32
0
4
8
12
16 20
24 28
Block
Block
Block
Sink
20
Sink
0.50
No Sink
Li
J.
cos h. k(
0.25
||
|
|
200
100
-025
0.50
Figure 2: In LLaMA3-8B Base, (Top) the first token has significantly larger fz-norm of hidden states, but much smaller 2z-morm of keys and values than the mean of other tokens; (Bottom) cosine similarity instead of norm product contributes to attention sink. We delay more visualizations to Appendix C.3.
MIHSA layers. In the MHSA layer, the input H are first transformed into keys, queries, and values: kih- H Qth H Vlh H for each head S h S H (we omit the notation of LN when considering pre-norm design for simplicity). Here W W with Rdxdd, dn- = d/ H. Then the attention output computed as
A Lh Softmax (Q R. /Vd. M). O' = ='Concca H (A (bi) W
(4)
where M E TRT is an attention mask. For vanilla causal attention, Mj = -o for i < j and
My = 0 for 2 j. Finally, the output of final transformer block H is fed into an unembedding layer for prediction: Y = LN(HH) Wek. where Wds E Rdx VI.
Positional embedding. NoPE (Kazemnejad et al., 2024) considered explicit positional embedding (PE) in LMs, where P = 0. When using absolute PE (Vaswani et al., 2017), P is a periodic function of token positions. Devlin al. (2019); Brown et al. (2020) adopted learnable PE, which means P is learnable embedding of token positions. The dot product between each query and key meets (q. k,) k.k. when using the above three PEs. While for relative PE (Raffel et al., 2020), AL- iBi (Press et al., 2021). Rotary (Su et al., 2024), they have P 0. Instead, they modify the dot product (q. k2). For relative PE and ALiBi, (q., kj) k,k: + gi - j). where g(-) ) pre-defined function of the distance between two token positions. For Rotary, (q. k2) = Re. k where Re. is a pre-defined rotation matrix. We include detailed formulations of the above PEs Appendix B.
Auto-regressive objective. The pre-training objective of LMs is to maximize the likelihood of input data: 0" = arg maxa E Ex~Paa
Σ log ( where Paam refers to the data distribution.
Packing documents in pre-training. Given a large corpus D {di. .........d.pp dip J. where each d, represents document containing a sequence of tokens. A packing strategy is adopted in the LM pre-training, as present Figure (Rightt. All documents are concatenated and chunked into sequences with context length of C. Each chunk could start with any token within one document or the BOS/EOS token. Then the empirical loss function for each chunk is L Σ log (ax, (x<)) We note that Po (21) is ignored since y1 = f=(x)) is the prediction for the next token 22..
LM inference. During the inference, BOS token is fed into the model as the prefix for unconditional generation: x/ ~ Po d/|='< x, BOS). where x is the t-th generated token, æ is the optional prompt. If there are no BOS tokens in the pre-training, the EOS token is considered as the BOS.
Attention sink. Xiao et al. (2023b) revealed that LLMS allocate significant attention scores to specific token positions, e.g. the first token (not necessary be BOS token), resulting in "vertical" attention patterns. To represent this, have the attention scores A 1.
2,1 > mean( A 4.h (1f1)-
3

---

Published as a conference paper at ICLR 2025
GPT2-XL
LLaMA2-7B Base
LLaMA3-8B Base
100
100
100
80
80
80
Sink (%)
60
60
60
+ c 0.2
0.2
+
0.2
40
c 033
0.3
40-
-
0.4
0.4
0.4
20
0.5
0.5
0.5
16
32
128
256
512
1024
16
32
64
128
256
512
1024
0
16
32
64
128
256
512
1024
T
T
T
Figure 3: The metric Sinkí (averaged on 100 sequences) tends to decrease with larger token lengths
T. This tendency becomes more obvious with the more strict definition of attention sink (larger c).
PROPERTIES OF ATTENTION SINK
3.1 THE FIRST TOKEN ACTS AS BIASES
Uniqueness of the first token. It is noted that the calculation of hidden states for the first token has no involvement of self-attention h FFN(LN(() + h(-1)) + of + h4-1, Wil W Therefore, h1 and corresponding queries/keys/values ke LN(N W be considered as the MLP output of input word embedding 2 WE. Using LLaMA3-8B Base (Dubey et al., 2024), we show that from certain transformer block, e.g., I 2, the 42-norm of hí is significantly larger than that of other tokens h1y1 in Figure 2(Top). This reproduces massive activations in Cancedda (2024); Sun et al. (2024). Despite the large fg-norm of hidden states, we observe that the l2 norm of keys and values of the first token is significantly smaller than that of other tokens in the same figure, which also observed in Devoto al. (2024); Guo et al. (2024b).
where of )) | W
K dhhh = ( Q af LN(( LN could
QK angles contribute to attention sink. In the (-th transformer block, we consider the keys and queries after adding PE (Rotary in LLaMA3-8B Base): k = LN 1-1 " Re. q LN(h) W Re, where LN is RMSNorm (Zhang & Sennrich, 2019): LN(h) RMS(A)
h
and RMS(h) V V h2. Here g is learnable gain parameter. Suppose that h [-1 already has massive activations. Since h1 has a massive magnitude in specific dimensions. the LN operation retains the magnitude in these dimensions and further reduces the magniude other dimensions, leading to that qi kt.", and ofth are distributed on different manifolds, especially for k/."A.
For the 1-th query, we know that " typically has much larger values than qi h w. h visualized in Figure 2(Bottom). We further show that due to the different manifold of a. the angles || |l//A1llcos(( we visualize the cosine similarity between keys and values, and the product of & -orm between keys and values in Figure 2(Bottom). Although q || is comparatively small, cos(g, k1a)) is significantly large, leading to attention sink. This explains why attention sink exists despite the small norm of keys of the first token. To conclude, the first token leverages its keys to act as biases, thus minimizing the angles between k and q." and exhibiting attention sink.
T.
jft as
between K A and q: play important role. Considering q. k/hh
3.2 MEASURING ATTENTION SINK
Threshold-based metries. Xiao et al. (2023b) showcased the appearance of attention sink by visualizing attention logits/scores in different heads/blocks. This leads to the intractability of measuring attention sink quantitatively due to the large number of attention heads and blocks. Therefore, we first explore the metrics to measure the attention sink. Within each head, we compute the importance scores for the k-th token 0 I.
-- ΣT A.k. We mainly focus the first token a It is noted that + < af < 1 since A = and 0 <
L.h 1 S 1. Then we adopt a threshold-based metric, we consider a head has attention sink the first token a e. Considering that the whole model has L blocks and each block has H heads, we use the following metric to measure the attention sink of the whole LM: Sink
D # ΣH. T(o
c).
4

---

Published as a conference paper at ICLR 2025
Table 1: (Left) Even with random sequence as input, there still exists an obvious attention sink. But with repeated tokens, the attention sink disappears for Mistral/LLaMA models. (Right) Chat models have comparable attention sink metries with base models.
Sink (%) random
Sink (%)
LLM
LLM
natural
repeat
Base
Chat
GPT2-XL
77.00
70.29
62.28
Mistral-7B
97.49
88.34
Mistral-7B
97.49
75.21
0.00
LLaMA2-7B
92.47
92.88
LLaMA2-7B Base
92.47
90.13
0.00
LLaMA2-13B
91.69
90.94
LLaMA3-8B Base
99.02
91.23
0.00
LLaMA3-8B
99.02
98.85
Model scale
Default setup
Learning rate
100
10
36
train less
sink
20
1e
1
30
Sink, (%)
60
80
GPT2
L.oss
6
valid Joss
16
Sink, (%)
Train Loss
1o-5
24
Pythia
18
Sink
40
OPT
8
12
20
LLaMA3
LLaMA2
6
10M
100M
1B
10B
0
12 Steps (k)
8
16
20
8
16
24
32
40
Num. of param.
Steps (k)
Figure 4: (Left) Attention sink also emerges in small LMs. (Middle) Dynamies of train/valid loss and Sink; during LM pre-training under the default setup. Attention sink emerges after certain optimization steps. (Righr) Training loss (solid lines)/attention sink (dashed lines) dynamics of LMs using different learning rates. We observe that with smaller learning rates, attention sink tends to emerge after more optimization steps and be less obvious.
Selections of thresholds. Typically, the selections of thresholds represent the strictness of quantifying attention sink. Generally, larger represents a strict definition for attention sink. There no principal way to find an optimal threshold and we only use this metric quantify the emergence of attention sink empirically, Based on Figure 3, we prefer to select a threshold that both strict in quantifying attention sink and less sensitive to the token length T. This gives us the selection of e = 0.3. For fair comparisons, we need to fix T when computing the metric, e.g., T = 64.
3.3 ATTENTION SINK UNDER DIFFERENT INPUTS
Different data domains. We first explore the effects of input domains on attention sinks. The pile dataset (Gao et al., 2020), regular dataset for LM pretraining, has 17 available data domains. As shown in Appendix C.2, input domains have negligible effects on our attention sink metric Sink
Beyond natural languages. We also consider two ideal scenarios: (i) randomly sample T tokens from the tokenizer vocabulary V to construct a sequence and (ii) randomly sample 1 token from the tokenizer V and repeat it T times. As present in Table ((eff), attention sink still exists when the inputs are random tokens instead of natural language. However, with repeated tokens, attention sink in Mistral (Jiang et al., 2023) and LLaMA models disappears. In Appendix C.I, we prove that for LMs with NoPE/relative PPE/ALiBI/Rotayy if the first T tokens are the same, their corresponding hidden states are the same. They all have massive activations, thus dispersing the attention sink. We also provide the closed form/upper bound for attention scores in these LMs through Propositions 1-4.
3.4 ATTENTION SINK UNDER DIFFERENT LMS
Base vs. chat model. Compared with base models, chat models are typically continually trained through instruction tuning (Ouyang et al., 2022). From Table (Right), instruction tuning has an insignificant impact attention sink, which motivates us to focus on the LM pre-training.
Model scale. We evaluate the metric Sink, of LLaMA2 Base (Touvron et al., 2023), LLaMA3 Base (Dubey et al.. 2024), Pythia (Biderman et al., 2023), GPT2 (Radford et al., 2019), OPT (Zhang et al., 2022) families. As visualized in Figure 4(Left), attention sink emerges in small LMs, even in Pythia- 14M. Only in Pythia family, larger-sized LMs tend to have more obvious attention sink.
4 EFFECTS OF OPTIMIZATION ON ATTENTION SINK.
We pre-train series of LLaMA models to conduct our experiments, based on the repos (Zhang et al., 2024a; Liu et al., 2024a). Due to the intractability of replicating LLaMA pre-training, we design small-sized models. Following Liu et al. (2024a), we set hidden dimension d = 768. block number = 10, head number = 8. intermediate size of FFN as 1536, resulting in approximately 60M parameters except for word embeddings and unembeddings. We keep the other design the same
5

---

Published as a conference paper at ICLR 2025
Table 2: Larger weight decay ratios tend induce more attention sink heads in LMs. But much larger values hurt the model performance and attention sink disappears.
0.0
0.001
0.01
0.1
0.5
1.0
2.0
5.0
(
15.20
15.39
15.23
18.
18.18
41.08
37.71
6.13
0.01
valid loss
3.72
3.72
3.72
3.73
3.80
3.90
4.23
5.24
Prefix LM
Training data
25
20
20
sample1
train loss
sink
20
Sink (%)
sample2
16-
valid loss
15
sample3
12
12
10
sample4
8
sample5
0.050.1 0.2 0.5
2.5 5.0
2
4
7
9 10
Token positions
Amount (B)
Figure 5: (Left) Attention pattern for prefix language modeling. (Middle) Attention sink does not only appear on the first token but among the prefix tokens for LMs with p = 5. (Right) With less training data, attention sink disappears. Meanwhile, trained LMs demonstrate overfitting behaviors.
LLaMA2 models, including Rotary (Su et al., 2024), pre-norm structure, RMSNorm (Zhang & Sennrich, 2019) as LN, SwiGLU activation (Shazeer, 2020) in FFN, etc.
For data distribution, sample 5B tokens from the Pile dataset (Gao et al., 2020). We the context length to 2048 tokens, the batch size to 1M tokens, and the training step to 20k (including 100 steps for warm-up). We adopt learning rate of 4e-4 with cosine scheduling. The optimizer AdamW (Loshchilov & Hutter, 2017) with a weight decay ratio of 0.1. We use the Pile-CC validation loss (Gao al., 2020; Liu et al., 2024a) to measure the model performance and sample 100 sequences with T = 64 (no BOS token) out of training data to measure the metric Sink,, with = 0.3.
Optimization steps. As visualized in Figure 4(Middle), under our default setup, attention sink emerges after certain optimization steps, e.g., between 1 and 2k steps. With the progression of pre-training, attention sink becomes more obvious.
Learning rate. With smaller learning rate, it takes longer training steps to lower training loss, as present in Figure 4(Right). Meanwhile, the emergence of attention sink is also delayed. Besides, as shown in Table 9, we also find that smaller learning rate results in LMS with less obvious attention sink, even if we compensate for more training steps. But further decreasing learning rate significantly affects the optimization and model performance, thus affecting the emergence of attention sink.
Batch size. In Table 10(Left), we find that only modifying batch size has no effects on attention sink.
Takeaways: 1. Attention sink emerges after LMs are trained effectively. 2. Attention sink appears less obvious in LMs trained with small learning rates.
EFFECTS OF DATA DISTRIBUTION PDATa ON ATTENTION SINK
Training data amount. In the default setup, we consider 5B tokens. We wonder whether the attention sink emerges if we further constrain the data within a fixed compute budget. Therefore, we constrain the training data to 5B, 2.5B, 500M, 200M, 100M, and 50M. Meanwhile, we fix the batch size and optimization steps. As visualized in Figure 5(Rght), with less training data, attention sink disappears. Further evidence Figure shows that this is not related to overfitting.
Randomness in data distribution. After packing documents into chunks, we re-sample the first token within the chunk 2 ~ Uniform(V). The trained LM has the metric Sink = 27. 27.03%. even larger than the default setup. This further validates the low semantic information of the sink token. We also consider æ1, x2 ~ Uniform('V VV) and we find attention sink shifts to the second token with Sink, = 14.08% while the attention sink on the first token is much less obvious Sink = 1. 1.98% But when only sample E2 ~ Uniform(V). (V), the attention sink still always appears on the first token (Sink; = 20. 20)..% Additionally, find with more random tokens during pre-trate pre-training, attention sink tends to disappear.
Fixing token in a specifie position. Xiao et al. (2023b) considered learnable token in the first token position within each chunk, which can be considered 21 ~ T(x:: x.x)) We also consider fixing

---

Published as a conference paper at ICLR 2025
w
256
Shifted window attention
Sink
1.0
6
train loss
sink
20
0.8
5
loss
16
30
0.6
4
12
0.4
3
8
4
No
0.2
2
64
128
256
512
1024
Sink
Window size
Figure 6: (Left) Shifted window attention pattern. (Middle) In LMs with window attention, attention sink appears on the first token, but not on the "first token" within each window. (Right) Attention sink tends emerge when the window size is large enough.
the token 2rx in the second/third token position during pre-training. Consequently, the attention sink always appears in the fixed token instead of the first token, as shown in Table 10(Righr).
Takeaways: Attention sink emerges after LMs trained on sufficient training data. At- tention sink could be shifted to other positions rather than the first token if modifying Pasta-
6 EFFECTS OF LOSS FUNCTION L ON ATTENTION SINK
Weight decay. The loss function becomes C Ei log po (a; [a=<)) + + |00 when introducing weight decay ratio "y. As indicated in Table 2, even Y = O the loss function, attention sink still emerges in LMs. Then larger 7 encourages more heads have attention sink. But further increasing weight decay hurts the optimization, leading to less obvious or even no attention sink.
Prefix language modeling, Since the first token is not predicted in the auto-regressive loss function, it could be considered as the prefix token. Then the original nugr-regressive loss can be generalized 1.p)) with the prefix length p 1. Motivated by
into the formula L D( Ltmp+1 log (xx Wang et al. (2022), we consider > and the casual mask visualized in Figure 5(Left). Although this design does not affect the emergence of attention sink, it shifts the sink position. In Figure 5(Mdddle), the attention sink only appears on one token. But it appears among these prefix tokens instead of on the first token only. Massive activations also appear on the corresponding sink token.
Shifted window attention. Motivated by the shifted window attention adopted in Mistral-7B, further explore the effects of window size on attention sink. With shifted window attention, the loss function becomes L = مi= - log po (x ( where to refers to the window size. As shown in Figure 6(Left) and (Middle), with shifted window attention, we find that if w, the t-th token can still "look at" the first token, and LMs still have attention sink on the first token. When f > w, the t-th token can only attend up to the w 1-th token. Although this token is the "first token" for the t-th token, typically it has no attention sink. We have similar observations in Mistral-7B. Additionally, from Figure 6(Right), smaller window size prevents the emergence of attention sink.
Takeaways: 1. Weight decay encourages the emergence of attention sink. 2. With prefix language modeling, attention sink appears among the prefix tokens rather than the first token only. 3. With shifted window attention, attention sink appears on the "absolute", not the "relative" first token. Smaller window size prevents the emergence of attention sink.
7 EFFECTS OF MODEL ARCHITECTURE Po ON ATTENTION SINK
In this section, we mainly explore the effects of positional embedding, pre-norm post-norm struc- ture, and attention design on the emergence of attention sink. In Appendix D, we also show that vary- ing activation functions in the FFN, multi-head design do not affect the emergence of attention sink.
7.1 POSITIONAL EMBEDDING
Attention sink always appears the first token, which motivates us explore where such position property is brought by positional embedding (PE). Therefore, we attempt to replace the original Rotary with other PEs, shown in Table 3. We differentiate these PEs through the calculations of the hidden
7

---

Published as a conference paper at ICLR 2025
Table 3: Positional embedding does not affect the emergence of attention sink.
PE
H
(q., kz)
Sinkf (%)
valid loss
NoPE
XWE
9.k]
20.35
3.81
Absolute PE
WWE + P Pabs
q.k
32.73
3.74
Learnable PE
XWE E + P Rearnable
9.k;
33.13
3.79
Relative PE
XWE
q.k, + grelative (i - j)
35.53
5.45
ALiBi
XWE
k ; + 9ataba
20.78
3.71
Rotary
XWE
q, R.....sk
18.18
3.73
LN
Attention biases
Fixed V biases
20
80
20
Pre-norm
Default
L22 -Norm Ratio
15
Post-norm, before 1NN
60
15
Gw
Pust-norm, ofler LN
20e
10
40
10
20
0
2
8 9
4 5 Block
9
10
0
2
3
4
6 Block
5
7
8
10
Block
Figure fg-norm ratio of h and mean of h zo (Left) Massive activations exist not in hidden states in post-norm LMs. but in the features before LN. (Middle) LMs with KV biases or K biases have mas- sive activations, while LMs with learnable sink token or V biases have massive activations. (Right) Massive activations emerge when increasing the f-rnorm of fixed vith for the setup of K. biases.
states Hd before the first transformer block and the dot product between queries and keys (q., kz)- The detailed formulations are delayed Appendix B. From the same Table, observe that only the model with relative PE is difficult to train while other models have comparable performance under our setup. Then we note that all these LMs, even the one without explicit PE (NoPE), have attention sink.
7.2 PRE-NORM AND POST-NORM STRUCTURF
Layer normalization (LN) (Ba et al., 2016: Zhang & Sennrich, 2019) regularizes the hidden states in LMs by re-centering and re-scaling, which may affect the massive activations. This motivates us to explore the effects of LN location on attention sink. In the pre-norm structure, as stated in Equation 2, hidden states from the earlier blocks could be retained by the later blocks through residual connections (He et al., 2016). Therefore, if massive activations appear in a specific block, they will likely be retained in the subsequent blocks. Within a post-norm structure, the hidden states will be normalized before being fed into the following blocks, as present in Figure ((Left).
When replacing the pre-norm structure with the post-norm structure, Sink becomes 13.54%. This indicates that the attention sink still exists in post-norm LMs. After further investigations, as visualized in Figure 7(Left), massive activations exist in the hidden states before the post LN instead of h1.
7.3 ATTENTION BIASES
Learnable biases in attention. In Section 3.1, we have shown that the first token acts as a bias: its key k/ is distributed in different manifold and its value v has small &2 norm. Xiao et al. (2023b) considered a learnable sink token in each chunk before the input tokens during LM pre-training. implicitly introducing biases k
As this token is fixed in the first token, this could be considered *1 9 nl in attention, as shown in the second row in Table 4. These biases are the MLP output of a: W ". Sun et al. (2024) directly introducing kr.h and ul learnable parameters in attention (KV biases). They found that this design could alleviate the massive activations. Considering the important role of =lth and small norm of *I, we propose introducing only the key biases =l. and fix value biases 1 +IAh = 0 (K biases). As a control, we also consider only adding value biases (V biases). The formulations of all these attention designs are shown in Table 4.
LMs need key biases. After evaluating the LMs with setups in Table 4, we first observe that these LMs have comparable model performance. Moreover, as long as there are key biases = attention sink disappears on the first token but on the biases. From the setup of K biases, we reaffirm that the sink token acts as key biases, storing extra attention scores, which could be completely non-informative and not contribute to the value computation. It is worth mentioning that the introduced learnable sink token, KV biases, and V biases become part of model parameters in LMs. Removing them will lead to no attention sink in the first position, but significant drop in model
8

---

Published as a conference paper at ICLR 2025
Table 4: With comparable performance, LMs with sink token, KV biases, and K biases could shift attention sink from the first token to key biases' position. Value biases cannot affect attention sink.
Attention in each head
Sink, (%)
Sinkf(%) (%)
valid loss
Softmax
Q +
+M) V
18.18
3.73
vdn
o vin
Softmax
Vdn.
1
Quih
KLAT J + M
74.12
0.00
3.72
vath
Softmax
1 Quh
KUNT] + +M)
72.76
0.04
3.72
van
Vih
Softmax
Qui
KLhT] ] +M)
O Vibb
73.34
0.00
3.72
vdn
Softmax
+ +M) v + woth
17.53
3.73
vdn
Table 5: Larger (2-norm of fixed v thth results in LMs allocating more attention æ instead of k
0
e'
5w'
20v'
y"
5v"
20s²
Sink, (%)
73.34
70.03
44.43
1.51
69.74
27.99
0.00
Sink, (%)
0.00
0,06
3.71
25.88
2.15
5.93
11.21
valid loss
3.72
3.72
3.72
3.71
3.72
3.72
3.73
performance. In Figure 7Middle)), we also find that the LM with a learnable sink token has massive activations in x* While LMs with KV biases and K biases have no massive activations.
Beyond all zeros in V biases. In the setup of K biases, we fix e vl, = 0. We wonder whether the fixed values of v1 could affect the attention sink. We consider, v me' or v*Ih me"
where m is the controllable Eg norm and v' [1, O, O, ... 0] and v" [1. 1, 1. 1l/ /l//d,, As shown in Table 5, with larger lz f--norm of = attention sink shifts from - to the first token. Intuitively. it is difficult for LMs to remove the effects of v "1Ah with larger fn-nrrm in model predictions. Then they opt to optimize the keys and values of the first token to save extra attention.
Design space of biases. In Table 12(Righr). the LM with head-sharing KV biases tends to shift the sink from 1,, back to the first token. While the one with head-sharing K biases is less affected. In Table 13, even with small learnable dimensions for key biases, they can still absorb large attention.
7.4 ATTENTION OPERATION
General formulation of attention. In the last section, realize that LMs need key biases to save extra attention. This motivates us to explore whether such a property is related to the depen- dence among attention scores due to the softmax operation. First, the attention output for the i-th
token can be generalized as: ma = (Ej= sim(sp(q:)) (p(k)))) Σj= sim((p(;:). ((k;))) = Z Ej -= (im() (k)))) where we omit the PE, and block/head indexes for simplicity. Z, is a normalization term and (p)) is kernel function. Normally, Z, = E s ((k))) For softmax attention, ep(-) is an identity kernel and sim(q., k2) exp(q;k) /vdn).
Normalization. In Table 14(Left) and Figure 29, we show that modifying normalization may result in less obvious attention sink but does not stop its emergence. So consider removing the normalization. Since the exponential function in softmax tends to explode without normalization, we replace with sigmoid or elu plus one. When evaluating the attention sink, we compute the proxy attention scores by using the term Z, Σ sim(yp(g.). (k):)) for attention sink metric Sinkç. As shown in Table 6, without normalization, LMs still have comparable validation loss but attention sink. With normalization, attention sink also emerges in LMs with sigmoid attention.
Kernel functions. Motivated by linear attention (Katharopoulos et al., 2020), we consider different kernel functions ye.), including elu plus one, identity, and MLP. It is noted that sim({p(;;) ( (((k)) could be minus for identity and ML. kernels. This brings intrinsic difficulty for normalization during the training and calculation of Sinki. For normalization in the training. we consider Z: max (|E sim(sp(gs) (pkk)) (). When computing Sink,, we consider the proxy attention scores (ii((((g)))))))))))))))))))))))) / =1 (im((p(()..<<k))))))) From Table 6 (a full version in Table 15), find that the LM with MIP kernel have attention sink with without normalization.

---

Published as a conference paper at ICLR 2025
Table 6: Normalization and selections of kernels in attention significantly affect the emergence the attention sink. We use "*" to mark that the metric Sinki is computed by proxy attention scores.
sim( (amm((()))))(()
Z;
Sinkj (%)
valid loss
exp(
q.A," Vdn
Ey== exp(
qike vda
18.18
3.73
sigmoid g.Ak
0.44
3.70
vdd
sigmoid q.k:
Σ sigmoid(
9ik 7 vdd
30.24
3.74
Vdn
elu( q
+
0.80
3.69
vāa
(clu(g.) (X(uuk +11
Σ 2
53.65*
4.19
vdn
Vdn.
9.kg ddn mipla (mpp(t)
0.00
3.99
max
[
mlp(q, (m
.1)
0.19
3.85
Vda. mlp(q. (
vdn
0.74
3.91
vdd,
LMs with attention sink
LMs without attention sink
Scale up to IB LLMs
Lg -Norm Ratio
40
30
soffimax
clu plux one kommel
sigmoid, norm
1.8
identity kornel. wio norm.
elu plus une, wio morm.
sigmooi, wio norts.
24
18
soflma
sigmoid. wio nonm.
norm.
1.4
ML kernel, mot-normm.
20
MLP kemel, wi. norm.
12
1.0
0.6
0
2
4
6
8
9
10
2
3
4 5
8
9
10
0
4
8
12
16
20
Block
Block
Block
Figure 8: We visualize the no-mo ratio of h and mean of h (Left) Massive activations exist in LMs with attention scores that are non-negative and added up one. (Middle) Massive activations do not exist LMs with independent attention scores. (Right) When scaling the model size to 1B, LLMs with sigmoid attention (no normalization) still have no massive activations.
Inner dependence on attention scores. We note that the LMs with no attention sink typically relax tokens' inner dependence on attention scores. Their attention scores during pre-training could be negative not add up one. This indicated that attention sink (at least partially) stems from such inner dependence. Besides the attention metric computed by proxy attention scores, we also observe that the above LMs also have massive activations, shown in Figure 8(Middle).
Scale up to 1B parameters. We compare model behaviors of IB LMs with softmax attention and sigmoid attention (without normalization). Specifically, the latter achieves validation loss of 3010, slightly larger than the 3.07 achieved by the former. However, the attention sink metric significantly drops from 45.1 % to near zero: Sink, 2.46% using the proxy attention scores. Meanwhile, as present in Figure ((Rghh), LLMs with sigmoid attention have no massive activations. Furthermore, they have issues of training stability during continued supervised fine-tuning in Appendix E.
Takeaways: 1. Positional embedding, FFN design, LN location, and multi-head design do not affect the emergence of attention sink. 2. Attention sink acts more like key biases, storing extra attention and meanwhile not contributing to the value computation. 3. When relaxing tokens' inner dependence attention scores, attention sink does not emerge in LMs.
8 FUTURE WORK
This work focuses the sink token in the first position. Sun et al. (2024); Yu et al. (2024) showed that attention sink can also appear on certain word tokens, e.g., period and newline tokens. However, these sink words may vary in different LMs and typically have no fixed positions. In the future, we will extend the research scope to explore how these sink tokens are related to the pre-training Additionally, it remains unclear whether attention sink benefits LM downstream performance.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv: 06450, 2016.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances Neural Information Processing Systems, 36:75067-75096 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in neural information processing systems, 33, 2020.
Nicola Cancedda. Spectral filters, dark signals, and attention sinks. arXiv preprint ar 1r11:22402.09221, 2024.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and and-play inference acceleration for large vision- language models. arXiv preprint 24Xiv:2403.0676 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309. /6588, 2023.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning. pp. 7480-7512. PMLR, 2023.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 Q): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318-30332, 2022.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational I Linguisties: Human Language Technologies, Volume / (Long and Short Papers), pp. 4171-4186, 2019.
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini. simple and effective 1 norm-based strategy for kv cache compression. arXiv preprint arXiv:2406.. 1430, 2024.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv: 142333...42.,.. 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXhv:2407.21788 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23( 1-0):1-99, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.000 2020.
11

---

Published as a conference paper at ICLR 2025
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac' h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 07 2024. httpss: zenodo org/ (eecords/220B8 602.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01800 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint ar Xr1i:223/2.00752, 2023.
Tianyu Guo, Druv Pai. Yu Bai, Jiantao Jiao, Michael Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena in llms. arXiv preprint arXiv:2410.1388 2024a.
Zhiyu Guo, Hidetaka Kamigaito, and Taro Watanabe. Attention score is not all you need for token importance indicator in kv cache reduction: Value also matters. arXiv preprint arXiv:2406. 12335, 2024b.
Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 3991-4008, 2024.
Bobby He, Lorenzo Noci, Daniele Paliotta, Imanol Schlag, and Thomas Hofmann. Understanding and minimising outlier features in transformer training. In Advances in Neural Information Processing Systems, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.084155 2016.
Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Xianglong Liu, Luca Benini, Michele Magno, and Xiaojuan Qi. Slim-llm: Salience-driven mixed-precision quantization for large language models. arXiv preprint arXiv:2405.14911 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers Fast autoregressive transformers with linear attention. In International conference on machine learning, pp. 5156-5165. PMLR, 2020.
Prannay Kaul, Chengcheng Ma, Ismail Elezi, and Jiankang Deng. From attention to activation: Unravelling the enigmas of large language models. arXiv preprint arXiv:2410.17174, 2024.
Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems, 36, 2024.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 142.66141,.6980, 2014.
Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwarte et al. Jamba: A hybrid transformer- mamba language model. arXiv preprint arXiv:2403. 19887, 2024.
12

---

Published as a conference paper at ICLR 2025
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang. Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quantized lims. In The eiihth Annual Conference on Neural Information Processing Systems, 2024.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang. and Min Lin. Regmix: Data mixture regression for language model pre-training, arXiv preprint arXiv:2407.01492 2024a.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao, Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.0124) 2024b.
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.0100, 2017.
Evan Miller. Attention is off by one. URI htth://////wwww.ooommmmttt//aa// 2023.
html,
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Ofir Press. Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12400 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 67, 2020.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1110.09441, 2017.
Jason Ramapuram, Federico Danicli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention. arXiv preprint 24Xx::2409.04411, 2024.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202,.22 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of-exppree layer. arXiv preprint arXiv: 700.00688,.. 2017.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing 568:127066 127063, 2024.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint 240. 2024.
Jamba Team, Barak Lenz, Alan Arazi, Amir Bergman. Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, al. Jamba-1.5: Hybrid transformer-mant models at scale. arXiv preprint arXiv:2408.12577 12570, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
13

---

Published as a conference paper at ICLR 2025
Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Look-once optimization in kv cache for efficient multimodal long-content inference. arXiv preprint arXiv:2406.1811 2024.
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In International Conference on Machine Learning, pp. 22964-22984. PMLR, 2022.
Mitchell Wortsman, Peter Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D Co-Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309. 2023.
Haoyi Wu and Kewei Tu. Layer-condensed kv cache for efficient inference of large language models. arXiv preprint ar arXiv:2405. I4V:: 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099 PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.174.5 2023b.
Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. Seed-story: Multimodal long story generation with large language model. arXiv preprini arXiv:2407.08683, 2024.
Qingyu Yin, Xuzheng He, Xiang Zhuang, Yu Zhao, Jianhua Yao, Xiaoyu Shen, and Qiang Zhang. Sta- blemask: Refining causal masking decoder-only transformer. arXiv preprint arXiv:2402.04777 2024.
Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. Unveiling and harnessing hidden attention sinks: Enhancing large language models without training through attention calibration. arXiv preprint arXiv:2406. 15765, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.0000 2019.
Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M Susskind. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pp. 40770-40803, PMLR, 2023.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint 240.:22401..2385. 2024a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- vee-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Zhenyu Zhang, Shiwei Liu, Runjin Chen, Bhavya Kailkhura, Beidi Chen, and Atlas Wang. Q-hitter: A better token oracle for efficient Im inference via sparse-quantized kv cache. Proceedings of Machine Learning and Systems, 6:381-394, 2024b.
14

---

Published as a conference paper at ICLR 2025
A RELATED WORK
A.1 ATTENTION SINK PHENOMENON
Although the term attention sink was introduced by Xiao et al. (2023b), similar observations have been reported in prior studies. Among them, Zhai et al. (2023) demonstrated that attention weights in Transformer models (Vaswani et al., 2017) collapse into one-hot vectors, a phenomenon termed attention entropy collapse. This issue has been identified as attention logit growth by Wortsman ct al. (2023); Dehghani et al. (2023). Furthermore, Bondarenko et al. (2023) observed the emergence of strong outliers in Transformer models, linked to attention heads that learn not update residuals. Although not explicitly using the term attention sink, Han et al. (2024) highlighted the significance of the first few tokens in language models (LMs), noting their distinct representational space. In vision transformers (ViTs), Darcet et al. (2023) identified high-norm outlier tokens as artifacts in attention maps. Overall, the attention sink phenomenon is prevalent across Transformer models, including ViTs, encoder-only LMs, such as BERT (Devlin et al., 2019), and auto regressseeee LMs. Notably, in ruto-regressive LMs, attention sink consistently occurs on the first token in addition to tokens with limited semantic information in the middle context (Sun et al., 2024; Yu et al., 2024).
A.2 ATTENTION SINK AND ACTIVATION OUTLIERS
Cancedda (2024) found that early FFNs in LLaMA2 blast off the large norm of hidden states for the first token, leading to attention sink in later layers. This is referred to as massive activations (very few activations exhibit disproportionately large values) in Sun et al. (2024) and token-wise activation outlier in outlier literature (Lin et al., 2024). Similar observations were made by Bondarenko et al. (2023). In contrast, channel- activation outlier (Detimers et al., 2022; Xiao et al., 2023a) refer to that outliers appear in specific channels across all token positions. concurrent study (Kaul et al., 2024) found that these two types of outliers are distinct and require different mitigation strategies. Despite the high magnitude in hidden states, Bondarenko et al. (2023); Guo et al. (2024b) observed the values associated with sink tokens are typically smaller than those of other tokens.
A.3 UNDERSTANDING AND MITIGATE ATTENTION SINK
To understand the attention sink phenomenon in general Transformer models, Bondarenko et al. (2023) hypothesized that the interplay among softmax, residual connections, and LayerNorm encourages models to learn not to update residuals. In auto-regressive LMs, Sun et al. (2024) attributed attention sink at the first position to implicit biases in keys and values. concurrent work by Guo et al. (2024a) provided empirical and theoretical analyses of attention sink in very simple Transformer LMs on the Sngram-Backcog (BB) task, identifying a mutual reinforcement mechanism as the underlying cause. Their findings on the BB task indicated that (i) replacing softmax with ReLU removes attention sink; (ii) switching the optimizer Adam to SGD eliminates massive activations but attention sink remains.
Besides the above literature, Xiao et al. (2023b); Kaul et al. (2024) found that replacing softmax with softmax-off -fftma--off-by-ne (Miller, 2023) alleviates attention sink at the first position, though Xiao et al. (2023b) noted its potential emergence at other initial token positions. Yin et al. (2024) proposed refin- ing the casual mask with pseudo-attention values on the "future tokens", which could reduce attention sink. Though not explicitly targeting attention sink, methods such o oReparam (Zhai al., 2023) and qk-norm (Dehghani et al., 2023) were introduced to address attention entropy collapse. To mitigate activation outliers, He al. (2024) developed an outlier-protectdd block and also highlighted the role of optimization in mitigating the outliers. Besides, Bondarenko et al. (2023) proposed clipped softmax and gated attention, while Kaul et al. (2024) introduced OrthoAdam to reduce activation outliers.
A.4 APPLICATIONS OF ATTENTION SINK
Sink tokens in LMs are functionally important and absorb significant attention. This enables com- putational savings by prioritizing initial and recent tokens over middle-context tokens in attention calculations. Token-wing wise activation outliers pose challenges for quantization, necessitating special- ized handling to facilitate quantization processes. These insights have motivated various applications, including streaming/long context generation (Xiao et al., 2023b; Han al., 2024; Yang et al., 2024), KV cache optimization (Ge et al., 2023; Wan et al., 2024; Wu & Tu, 2024), efficient inference (Zhang et al., 2024b; Chen et al., 2024), model quantization (Liu et al., 2024b; Huang et al., 2024), and others.
15

---

Published as a conference paper at ICLR 2025
B DETAILED FORMULATIONS OF POSITIONAL EMBEDDING
In this section, we provide detailed formulations of positional embedding (PE) in LMs. PEs could be classified into two categories. NoPE, absolute positional embedding, and learnable positional embedding belong to the same category since they are added to the initial hidden states: H XWE P. Here P {P1. P2. , RTxd Meanwhile, the dot product between each query and key is computed as kq; q:k,
NoPE. NoPE (Kazemnejad al., 2024) refers no positional embedding. Therefore, P= 0.
Absolute PE. Each position vector p: in absolute positional embedding periodic function of token position i following Vaswani al. (2017):
P: [sin(wit) cos(wit) sin(wat) cos(w2t)
sin(wayz!) cos(wa/2()).
where w/ 1/100002(--1)4
Learnable PE. Each position vector in learnable positional embeddings is learnable parameter.
Relative positional embedding, ALibi, and rotary belong to another category since they consider the relative distance among tokens. Therefore, the initial hidden states are H XWE but how to compute the dot product between each query and key is modified.
Relative PE. The relative positional embeddings are adopted in T5 models (Raffel et al., 2020). A bias term is adopted for the dot product: (qi k:z} = k g(i - j)), where the definition for distance function g(.) is:
i i-3
if i i if B/2
gii - j)
8 log(
8/2
(6)
log
B
if i-122D
Here B and refer to the number of buckets and maximum distance, respectively, In T5 models, B = 32 and 'D = 128.
ALiBi. Similarly, ALiBi (Press al., 2021) also adds a bias term the dot product: k.k. g(i j), where g(i - j) - - j) m. m head- head- specific slope fixed:
m = 2 loag 1
(7)
where 1 < h < H is the head index and H the number of heads in the multi-head self attention (MHSA). This slope m is geometric sequence. For instance, when H = 8, the sequence is 2T. 2z, ... When H 16, the sequence is zrs. 2T 213.
Rotary. Rotary (Su et al., 2024) is the most adopted position encoding approach in the LLM community. projects queries and keys into another space through rotations:
(g: k2) = (q. -)) (ke Re. -5)² = q: Ro. -...-R.e....... Re qi Ro. jk
(8)
where (e) a pre-defined rotation matrix:
Cos mwa sin mwi 0
sin mwa
0
0
0
cos mWn1
0
0
COS m23
sin 7/16
0
Re.
sin m/w'
cos miw
0
0
COS mwdd /2 sin sinmusd. cos mid..
sin madd, /2
0
Here Wi (1-1// and dn d/H is the hidden dimension in each head. From the above definition, is noted that the rotation matrix Rds satisfies R Re and Re, Re.j Re
16

---

Published as a conference paper at ICLR 2025
C ATTENTION SINK IN OPEN-SOURCED LMS
C.I HOW POSITIONAL. EMBEDDING RELATES TO ATTENTION SINK
In Section 3.3, we have shown that Mistral-7B. LLaMA2-7B Base, and LLaMA3-SB Base, which adopt rotary as PE, have no attention sink for repeated token sequences. While GPT2, which adopts learnable PE, has the attention sink in such a scenario. To further understand this, we explore how positional embedding plays role through a theoretical perspective.
Activations. Suppose the repeated sequence is X {I1, 22, ..... FT and each a; = z. For LMs with NoPF/relative PF/AL IF//A..Bi/RRoary, we have the initial hidden states h = æWE, which are the same among all the token positions since
0. Then for the first transformer block | we WN(xWW))WK E Then all tokens have the same k, q.th and with Then
know that k W LN(x W) W qr
W
Vt (h, W.A. LN(<WE W.th
the attention output is
1 11
L. L. = 0, o
(10)
Then of = I Hf W we have the hidden states after the first block:
h = OF + 12)) + of + h FFN(LL(Concat 1. W + h))) + Concat 1. W + h
(11)
(12)
FFN(LN(Concat, (LN(())) W W + h!))
(13)
+ (LN())) W + hi
(14)
Since 1 = h2 = ..... = h4, we have h = h1 = 14 based on the above equation. Using this
induction, could prove that
h FFN(LN (LN(H) -- W th W/ + +h(
(15)
+ Concat (LN(h;-1)We) W h -1 v 1 vislsL.
(16)
= - h4, 1O
(17)
Typically, the hidden states of the first token h in specific blocks have massive activations. Due to the above equality, all the repeated tokens have massive activations. Furthermore, we could derive the closed form or upper bounds for attention scores under the repeated token sequence.
Proposition 1. For LMs with NoPE, the attention scores for repeated tokens are t uniformly, i.e. there is attention sink.
Proof. We have that
A elg! .((
e% RAA
eg k I
(18)
E/= Met"
Ej edl a w/th
teg
Therefore, the attention scores follow a uniform distribution over all previous tokens. Proposition For LMs with relative PE, there is no attention sink for t repeated tokns.
L
Proof. For LMs with relative PE, the dot product between each query and key is
(g.h k/Ah) = M. + (telt i) = - L. + gre(tt - i),
(19)
then we have the attention scores
A clatth k.hh
c h +
(t
(20)
Ej (aa kath))
Ej=1 call +gne(t-))
=j= esnalt-j)
17

---

Published as a conference paper at ICLR 2025
Considering (ta(t 1) is monotonic non non-increasing function of and grei - = B when t- D, then A A sink on the first token.
I. Lit-D are the largest values. Therefore, there no attention
Proposition 3. For LMs with ALiBi, there is no attention for t repeated tokens.
Proof. For LMs with ALiBi, similar to relative PE, the dot product between each query and key is
(dl th ( g.hh w
gaina = ql kLh + gainsct i).
(21)
then we have the attention scores
A elol (e/i)) E/ e e Mgg
e edlhiehh +ghim(e-1)
(llin((-())
(22)
Ej gi tolin (t-5) Σ Ej=1 E/=1 elhialt-s)"
Here (t - i) is monotonic decreasing function of i, so there is no attention sink on the first token.
Proposition 4. For LMs with Rotary, there no attention sink for repeated tokens when t large if lg || S & for constant &.
Proof. For LMs with Rotary, the dot product between each query and key is
( I.h k/ qi Re, i-k
T qhh Re, Ro.
(23)
(24)
glh Re, i woklh [|qti|| kot. Res
=||/41|| Ro
R|o, cos
(25)
= =|||**|| || cos(Bi-i).
(26)
where B is the angle between the rotated query and the rotated key. Then the attention scores are
4 1i e cla k Σ/=
eq Roo W
ella** ||||*** [|cos( :)
(27)
Ej with Re.; M
Ej-1 claa || | ||oos(f ")
Suppose the norm of multiplication for query and key || ||blil = &. Considering -1 S cos(B1-,) I 1, then we have
22E
A
(28)
Ej=1 (00a((e-3))
L e Smaln
2 + (t - 1)
+
Then the attention scores for each token upper-bounded and decrease to 0 as grows.
For LMs with absolute PE/learnalee PE, the initial hidden states h æW E +p Although the word embeddings are the same for repeated tokens, p. for different token positions L. is different. Therefore, GPT2 models have no the above equality. From Table ((Lef), GPT2-XI. still allocates significant attention the first token even with repeated tokens, which motivates to explore whether attention sink is related to these learned positional embedding vectors Pı.T after LM pre-training.
Therefore, conduct two experiments on GPT2-XL. Firstly, replace the first positional embed- ding vector P1 with other vectors Ptz11 Table 7, we find that the amplitude of attention sink on the first token is significantly reduced. Then also consider swapping the first positional embedding vector P1 with another position P Consequently, the t-th token becomes the new sink token. Therefore, attention sink in GPT2-XI. is strongly attached to the first positional embedding vector pt-
18

---

Published as a conference paper at ICLR 2025
Table In GPT2-XL, replacing or swapping the first positional embedding vector p1 with another position Piş1 significantly impact the amplitude and position of attention sink.
Replaced position
no
10
15
20
25
25
Sink (%)
62.28
0.20
2.36
7.73
10.63
10.97
10.21
Sinkf(%)
0.00
0.00
0.00
0.00
0.00
0.00
Swapped position
no
10
15
20
25
25
Sink, (%)
62.28
1.44
3.73
6.78
8.95
9,42
9.73
Sink;(%)
57.63
54.48
52.81
51.70
51.13
50.22
C.2 ATTENTION SINK UNDER DIFFERENT DATA DOMAINS
There are 17 available data domains in the Pile dataset (Gao et al., 2020), including Pile-CC, PubMed Central, ArXiv, Github, FreeLaw, Stack Exchange, USPTO Backgrounds, Pubmed Abstracts, Guten- berg 19), Wikipedia (en), DM Mathematics, Ubuntu IRC, EuroParl, HackerNews, PhilPapers, NIH ExPorter, and Enron Emails. We sample 100 data from each domain and then evaluate the attention sink metric for
Base/LLaMMA3 base. As shown in Figure 9, the evaluated attention sink metrics Sink are similar across different domains when 0.2 and 0.3. Small fluctuations appear when = 0.4.
0.2
0.3
0.4
GPT2-XL
LLaMA2-7B Basc
Mistral-7B
LLaMA3-8B Basc
100
100
100
Sink (%)
90
90
90
80
80
80
60
60
60
9
13
17
5
9
13
17
13
17
Domain ID
Domain ID
Domain ID
Figure 9: Input domains have negligible effects on attention sink metric Sink for (leff) c 0.2 and (middle) e 0.3. There are small fluctuations for (right) 0.4.
C.3 ATTENTION SINK UNDER DIFFERENT PRE-TRAINED LMS
Relation to LM performance. We leverage the platform (Gao et al., 2024) to evaluate the perfor- mance of souresdureed LMs, including
families, down- stream LM task, c.g., HellaSwag (Zellers et al., 2019). The results are visualized parallel with our attention sink metric in Figure 10, including both accuracy (Acc) and accuracy under normalization Moc__Norm). We find that within the same LM family, with the increase of model scale, both attention increasing. However, across different LM
sink amplitude and downstream LM performance families, stronger attention sink does not always correlate with better performance. For instance, the OPT family has stronger attention sink than Pythia under the comparable model scale. However, the downstream LM performance is comparable.
norm. We first show that large f--norm of hidden states h and small ε2-norm of keys k and values of (especially for values) universally exist in open-sourced LMs, including LLaMA2-7B Base (Figure 1 11), GPT2-Large (Figure 12), Mistral-7B (Figure 13), and Pythia- IB (Figure 14). noted that for the final transformer block / = L, we take the hidden states before LN. We note that different LMs may have different starting blocks where massive activations appear.
19

---

Published as a conference paper at ICLR 2025
Sinkj (%)
Acc(%)
Poc_Norm({H
100
100
100
LLaMAZ
Pytai
LLsMA3
80
Pythia
LLaMA3
60
GPT2
60
OPT
60
OPT
40
OFT
40
40
20
L.MMAS
LLaMA2
20
10M
100M
IB
10B
10M
100M
IB
10B
10M
100M
IB
10B
Num. param.
Num. of param.
Num. of param.
Figure 10: Attention sink and downstream performance for various pre-trained LMs..
24
1000
18
(2-norm
800
600
hige
12
Pooosoa
400
200
0
0
12 16
24
28
32
12
20
32
4
12
16
20
24
28
32
Figure 11: 22-norm of hidden states/keyssvalues of the first token/other tokens in LLaMA2-7B Base.
2800
2100
18
22-nomm
k
1400
4
h
700
2
0
0
12 16
20
24
32
4
8
12
16
20
24
28
32
12
20
24
28
32
36
Figure 12: b 42-norm of hidden states/keys/values of the first token/other tokens in GPT2-Large.
350
24
280
10
18
210
12-
140
Kig
70
12
16
20
24
32
4
8
16
20
24
28
8
12
16
20
28
32
Figure 13: L -noom of hidden states/keys/values of the first token/other tokens in Mistral-7B.
2000
30
15
1600
24
12
22-norm
1200
18
800
12
4
ki
400
3
0
0
8
12
12
16
Figure 14: norm hidden states/keys/values of the first token/other tokens Pythia-
20

---

Published as a conference paper at ICLR 2025
5, A/
11,
1
Sink
Sink
Sink
0.4
Sink
coolg) "
-0.50
|| |-| |
200
cosigi
0.2
0.25
150
0.0
0.00
100
-0.25
-0.2
0.50g
-0.4
19,
1
31,
Sinkk
No Sink
Sink
-04
No Sink
250
10.1
coolo) k
|1 |
cusigi A
|| ||
200
150
100
0.0
100
-02
0.2
-04
Figure 15: Cosine similarity and f--norm product between keys and queries in LLaMA3-8B Base.
1
5,
A
1=
11,
Sink
0.4
No Sink
Sink
0.6
No Sink
200
casigh k
0o0/4) )
(o -
0.2
2001
11.4
1..0
112
100
100
0.0
-0.2
50
19.
1
31,7
Sink
-0.6
No Sink
Sink
No Sink
corl Go
200
0001g)
0.4
||< ||011
0.2
0.2
0.0
100
100
-022
Figure 16: Cosine similarity and 22-norm product between keys and queries in LLaMA2-7B Base.
QK angles. Then we further demonstrate that QK angles contribute to attention sink through more visualizations, including LLaMA3-8B Base (Figure 15), LLaMA2-7B Base (Figure 16), Mistral-7B (Figure 17), and GPT2-Large (Figure 18).
Block-wise and head-wise property. In the main paper, we mainly discuss the ratio of heads that have attention sink in the definition of attention sink metric Sink. Here visualize the locations of these attention sink heads in open-sourced LMs, including LLaMA2 family (Figure 19), LLaMA3/LLaMA.. family (Figure 20), Mistral family (Figure 21), GPT2 family (Figure 22), Pythia family (Figure 23), and OPT family (Figure 24). We visualize the distributions of importance scores for the first token a h across different transformer blocks 1 < ( S. L and different heads 1 < h S. H before computing the attention sink metric. We find that (1) different pre-trained LMs have various attention sink distributions but they tend to have less obvious attention sink in earlier transformer blocks; (2) instruction tuning does not significantly modify such attention sink distributions when comparing base versions and chat/instruct versions.
21

---

Published as a conference paper at ICLR 2025
5. h
11, =
Sink
Sink
No Sink
0.50
cooigh )
of
|
150
cowigl c ())))
|| |
150
0.25 1
100
100
10.0
50
-0.2
19.
31, 7
Sink
No Sink
300) Sink
Sink
200
cosig M
10.4
10.4
||
|
200
0.2
0.2
0.0
100
0.2
0.0
Figure 17: Cosine similarity and mn-mrrm product between keys and queries in Mistral-7B. =
I= 5
11,h
Sink
Sink
120 Sink
Sink
0.0
cosigi M
-02
| |
cootof A
100
-0.2
0.0
0.4
-0.6
0.2
1- 19,
1
31,
Sinkk
No Sink
Sink
[
Sink
0.0
150g
0025
coolgi
cosigf " K
| 1139
-0.2
00.00
100
100
-0.26
-0.6
-0.50
50
-1.75
Figure 18: Cosine similarity and 2--norm product between keys and queries in GPT2-Large.
Boss
HLMMA--78 Char
11aMA2-131 Bou
LLLMAI-15B Chat
0.5
0.5
0.8
Block
0.5
0.5
0.6
0.4
04
04
0.2
0.2
Head
Figure 19: Distribution of importance scores for the first token across different blocks and heads in the LLaMA2 family.
LL 1B Bese
LLeMAT&B AB Iutract
LLAMAD. JB Baso
LLAMASI IB Instact
0.8
0.8
0.8
0.8
Block
0.6
022
Head
Figure 20: Distribution of importance scores for the first token across blocks and heads in the LLMMA3/LLaMA3.1 family.
22

---

Published as a conference paper at ICLR 2025
Mounl -1
Matral no Inemuct
0.8
0.5
Black
0.6
0.5
0,4
02
Head
Figure 21: Distribution of importance scores for the first token across blocks and heads in the Mistral family.
OPdz-Medum
GPT> Larz
GPT2
Block
0.5
0.6
0.6
0.6
04
04
044
04
02
0.2
02
02
Head
Figure 22: Distribution of importance scores for the first token across blocks and heads in the GPT2 family.
Pytra-IdM
Pyho-SIM
Pysii-T0M
Pydro-IAMM
0.6
0.6
0.3
0.4
Pydicc
Pydie
0.
0.8
0.8
0.5
0.6
0.4
0.4
0.4
n.2
0.2
0.2
02
Pydius5.0
Pythes!
8.8
0.8
Block
0.6
06
0.4
0.4
02
1lead
Figure 23: Distribution of importance scores for the first token across blocks and heads in the Pythia family.
23

---

Published as a conference paper at ICLR 2025
OPT.12IM
OPE-
30
0.8
0.6
0.5
044
0.2
0.2
0.2
02
OPT47B
OPT:13
0.8
0.6
Dlock
0,2
Head
Figure 24: Distribution of importance scores for the first token across blocks and heads in the OPT family.
Jamba. Besides the auto-regressive Transformers, also consider Jamba (Lieber et al., 2024; Team et al., 2024), a new foundation language model. Both Jamba-v0.1 (Lieber et al., 2024) and Jamba-1.5 1.5 Mini (Team et al., 2024) have Jamba blocks, each of which includes Mamba layers (Gu & Dao, 2023), 4 Mamba MoE layers (Shazeer et al., 2017; Fedus et al., 2022), and Transformer layer. This adds 32 layers (including Transformer attention layers), 52B available parameters, and 12B active parameters in total. Firstly, we evaluate the attention sink metric and find that Sink; 88.48% for Jamba-v0.1 and Sink 87 88% for Jamba- 1.5 Mini, which indicates a strong attention sink on the first token. Then we visualize attention scores in several heads, as shown in Figure 25 and Figure 26. We also visualize the distribution of importance scores for the first token across blocks and heads in Jamba models in Figure 27. We observe that most heads have obvious attention sink, except for several heads in the 3rd Transformer layer.
1,
= 2, 12
1= 1= h 31
1 4, h = 23
Sink
Sink
Sink
Sink
0.8
0.8
0.8
0.8
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
Figure 25: Attention sink in Jamba-v0.1
1=
1 2, = 12
( = 3, h = 31
1 1 4, 23
1.0
1.0
1.0
1.0
Sink
Sink
Sink
Sink
0.8
0.8
0.8
0.5
0.51
0.6
0.4
0,4
0.4
1.4
Figure 26: Attention sink in Jamba-1.5 1.5 Mini.
24

---

Published as a conference paper at ICLR 2025
Jambe-v0.1
Jamba-1. Mini
Head
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Figure 27: Distribution of importance scores for the first token across blocks and heads in the Jamba models.
C.4
HUGGINGFACE LINKS FOR OPEN-SOURCED LMS
Table 8: Huggingface links for open-sourced LMs we used this paper.
Model
Huggingface link
LLaMA2-7B Base
meta-llama//lamee lama-2-7b-hf
LLaMA2-7B Chat
Satelllma/Lllma-2222 2-7bchhtt----
LLaMA2-13B Base
meta-llama/Llamma Ja---m------iff
LLaMA2-13B Chat
LLaMA3-8B Base
meta-llana/MMcca-Llaaa-----88
LLaMA3-8B Instruct
mete-llamm/Mcca--laaaa----------
LLaMA3.1-8B Base
meaallamaaMMca-LLaaa33---6
LLaMA3.1-8B Instruct
metalllamm//MttaaLLLaaaa311111 Instruct
GPT2
openai-commmuittyggn22
GPT2-Medium
openai-commmuniiyyg---mddiii
GPT2-Large
openai-commuuiityyppl--uggg
GPT22XI.
ogenai-commmunityygn2--l
Mistral-7B
mistrala/MMistall7BB-000.
Mistral-7B Instruct
Pythia-141 14M
Eleuther/A/pyttha--14
Pythia-31M
EleutherA//pyttiaa311
Pythia-70M
EleutherA//ppthhaa700m
Pythia-160M
EleutherA//pyythaa-6600
Pythia-410M
EleutherAA/gyythha44100
Pythia-IB
EleutherA//pythia--1
Pythia-1.4B
Eleuther/A/pyttia--.444
Pythia-2.8B
Eleuther/U/pytthaa2.888
Pythia-6.9B
EleutherAl/pythia-6..1
Pythia-12D
EleutherAl/pythhaa-122
OPT-125M
facebook/opt-12555
OPT-350M
facebookkopt-350m
OPT-1.3B
facebook/opt- 1.33
OPT-2.7B
facebook/opt-2..1
OPT--.7B
facebook/opt-6..1
OРT-13B
facebook/opt-131 13b
Jamba-v0.1
ai2 labs/Jamba--00.1
Jamba-1 1.5 Mini
ai2 laals//a21--aamaaaaaaa 1.5-baa-----Mmm
25

---

Published as a conference paper at ICLR 2025
D MORE EXPERIMENTS IN LM PRE-TRAINING
D.1 OPTIMIZATION
Learning rate. In Section 4, find that attention sink appears less obvious in LMs trained with small learning rates. This conclusion holds even if we compensate for more training steps. In our basic setup, we adopt learning rate 4c-4 for 20k training steps. When we scale the learning rate to half, i.c., 2c-4, we scale the training steps to 2 times, i.c.. 40k. As shown in Table 9, when we keep the multiply between learning rate and training steps constant (highlighted using cyan color), LMs trained with smaller learning rates tend to exhibit less obvious attention sink. Therefore, we conclude that small learning rates not only slow down the increase of attention sink but also mitigate attention sink even with longer training durations.
Table 9: Attention sink appears less obvious in LMs trained with small learning rates even compen- sating for more training steps.
learning rate
training steps (k)
Sink
valid loss
8c-4
10
23.44
3.79
8e-4
20
32.23
3.70
4e-4
20
18.18
3.73
2e-4
20
11.21
3.78
2e-4
16.81
3.68
le-4
20
2.90
3.92
le-4
6.29
3.67
Batch size. During the pre-training, we consider different batch sizes with other hlper-parameee fixed. As shown in Table 10(Left), batch size does not affect the emergence attention sink.
D.2 DATA DISTRIBUTION
Training data amount. In Section 5, find that with less training data, attention sink also disappears. Mcanwhile, LMs are also prone for overfitting. To disentangle the effects of training data amount and overfitting on attention sink, we monitor the dynamics of train/valid loss and attention sink metric during the LM pre-training in more granular level, present in Figure 28. With only 50M and 100M training data, LMs overfit at very early stages, between k and 2k steps. Meanwhile, Sink; maintains a very small value (less than 1%). While for the setup of 5B training data, Sinkf keeps increasing after a certain step. This indicates that the amount of training data, instead of overfitting, plays important role in the emergence of attention sink.
Training data: 50M
Training data: 100M
Training data: 5B
train loss
sink
train loss
sink
truin loss
sink
loss
loss
Joss
Sinke (96)
Loss
2
2
3
4
2
3
4
2
4
Steps (k)
Steps (k)
Steps (k)
Figure 28: Dynamics of train/valid loss and Sinkf during LM pre-training under different amounts of training data: (Left) 50M; (Middle) 100M; (Right) 5B.
Fixing token in a specific position. During the pre-training, we consider fixing the token æax in the first/second/third position. Consequently, when evaluating the attention sink metric, in Table 10gRigh)), we find that attention sink appears the appears in the fixed token instead of the first token.
26

---

Published as a conference paper at ICLR 2025
Table 10: (Left) Batch size has effect on the emergence of attention sink. (Right) Attention sink appears in the appears in the fixed token instead of the first token.
Batch size
0.25M
0.5M
IM
2M
Fixed position
Sink; (%)
16.45
17.19
18.18
16.19
Sink (%)
74.11
0.00
0.00
valid loss
3.92
3.78
3.73
3.68
Sinkg (%)
0.00
69.03
0.00
Sink (%)
0.00
0.00
69.64
Sink; (%)
0.01
0.01
0.00
D.3 FFN DESIGN
Activation functions. Since FFN in the earlier transformer block blasts off the & mn-norm of the first token, we are wondering whether the choices activation functions in FFN will affect the attention sink. Besides the SwiGLU activations used in our default setup, we also consider other activation functions, as present Table of attention sink.
observe that different FFN designs do not affect the emergence
Table 11: Modifying the activation functions in FFN does not affect the emergence of attention sink.
Activation functions
F
(in
valid loss
ReLU
ReLU(hWi)) W2
16.90
3.82
GeLU (Hendrycks & Gimpel, 2016)
14.76
3,79
Swish (Ramachandran al., 2017)
Swish(h W2
17.89
3.80
ReGLU (Shazeer, 2020)
(RLLUhWW)
@ hW2) W
13.88
3.75
GeGLU JU (Shazeer, 2020)
(GeL.U(h
h. W3
17.86
3.73
SwiGLU (Shazeer, 2020)
(Swish(h.W) hW, W3
18.18
3.73
D.4 ATTENTION DESIGN
Multi-head design in attention. We consider two perspectives in multi- head design in attention. Firstly, we explore whether the number heads has an impact on attention sink, especially for = which refers single-head self-attention. Second, attention output from each head is concatenated: O Concat L yol)) W/ We replace such concatenation operation with addition operation: O = Em (A ( W As shown Table 12(Left), multi-head design does not affect the emergence of attention sink.
Table 12: (Left) Modifying multi-head design does affect the emergence of attention sink. (Right) Sharing KV biases across heads in each block results in attention sink shifting back to the first token from K biases.
Multi-head
Sinkţ (%)
valid loss
Head-sharing Biases type
/
/
KV
KV
K
K
H
18.18
3.73
H 4
10.68
3.74
Sink. (%)
72.76
56.61
73.34
68.31
H
12.95
3.76
Sink (%)
0.04
12.44
0.00
0.23
H =
19.50
3.78
valid loss
3.72
3.72
3.72
3.72
addition 21.76
3.74
Head-sharing K/KV biases in attention. In the main paper, consider both KV biases and V biases in attention. These biases are not shared by different heads each block. We further explore whether head-sharing patterns will affect their functionality. As present in Table 12(Right), LMs with KV biases are more likely affected by the wead-sharing pattern: attention sink shifts from the biases to the first token. While LMs with K biases less affected.
Learnable dimensions of K biases. In the setup of K biases. we set all dimensions kl as learnable weights. In Section 3.1, have shown that K biases distributed in different manifold,
27

---

Published as a conference paper at ICLR 2025
Table 13: Even with very few learnable dimensions for k large attention appears in k=I.A,
d,
2
16
32
64
Sink,
32.18
30.88
30.94
31.39
23.30
51.23
69.19
Sink](S
4.74
4.96
4.39
4.54
2.19
1.94
0.04
valid loss
3.73
3.72
3.72
3.73
3.73
3.73
3.72
with low rank. Therefore, we consider only de dimensions of k are adjustable/learnable while the other d, da dimensions are zeros. As present in Table 13, with very few learnable dimensions, even for d, = 1, & are still allocated significant attention. With more learnable dimensions, the attention sink appears more obvious.
Scaling the normalization in softmax attention. Before our experiments about replacing softmax attention, we first explore the effects of normalization scales softmax attention. In the main paper, the attention output for the i-th output is
(im(p( simpp())))))
41 sim(p(a). Vj.
d.
(29)
Zi
We consider a scale factor a, the normalization term is Z, = 1 Σj =1 sim ((k))) and then the attention score are sum up to a. For default setup, a = 1. As shown Table 14(Left), with a smaller normalization scale, attention sink tends to appear in fewer heads. From another perspective,
asim(pp(a)). (kk,)
sim(ppl)). p(k,))
sim(pp())) (kz,)) Vj
j=1 Ej= sim (m)(p(())))))))))))))))(())))( (p)))(((((p))))))))))))))))))))) (a)a)
(30)
1=1
o Concat (1, (w)") Wo
(31)
Therefore, this normalization scale can be regarded as the scale for Wv or Wo. We show that this normalization scaling could be implemented by scaling learning rates and initialization. We use the 8 to represent the optimization step, and s = refers to the initialization. When scaling the normalization, we have the following SGD update rule (take Wo for example):
W
(32)
W ((W)
(33)
where 1) is the original learning rate. Suppose only modify the learning rate and initialization, to ensure each optimization step Wi a:W need first ensure W ao Suppose that have W = W then the update rule is:
W Wi 1/ V w: ((W))
(34)
a aw WWW
(35)
To ensure W = aW we need the new learning rate 7f' meets T/ = a²n. For advanced
optimization algorithms, e-g., Adam (Kingma & Ba, 2014) and AdamW (Loshchilov & Hutter, 2017).
We have the following update rule (take Adam for example, y refers to the weight decay ratio):
9++1 Vw; ((aW
W(W)(W)l
(36)
m Bim, + (1 B1)g.-1
(37)
Us+1 82V, (1 B2)
(38)
m / 35)
177) W = -1 W 1) / Vu (55 +
(39)
We denote 9-+1, m,+1, 0s41 represents the intermediate counterparts for update of scenario where
we only modify learning rate and initialization. First, we also need to ensure the initialization W W/B Then we assume that we have already matched W3 aWg. The gradient for each step is
9++1 V W L(W)) = Vw WwW)())
g-+1/a
(40)
28

---

Published as a conference paper at ICLR 2025
Then the first and second moment will be m m and 0,41 Vs+1/a². The updated
weights will
O = = = 1/1-)// o/ m, 1 -11189)
(41)
Vo.1// / (1 (1-1))
(1-1/ 11 m,+1/al1 (85)
(42)
Vust1/ a²(11 1111-19) -352
Therefore, to ensure W aWo one solution is 1/ on and e' c/a and 7 = 7/a.
Table 14: (Left) Scaling the normalization in Softmax attention to less than one can mitigate attention sink but not prevent its emergence. (Righ() LMS with sigmoid attention (without sigmoid attention) trained by different learning rates and weight decay ratios have no attention sink.
Scale
Sinki ())
valid loss
Learning rate
Weight decay
Sink (%)
valid loss
2.0
29.10
3,72
4c-4
0.0
0.64
3.77
1.0
18.18
3.73
4e-4
0.1
0.44
3.70
0.2
9,41
3.72
4e-4
0.5
0.18
3.76
0.1
3.59
3.76
4e-4
1.0
0.30
4.06
0.05
4.53
3.78
le-3
0.1
0.81
3.68
le-4
0.1
0.36
4.08
Normalizer. Besides the normalizer Z, Σ -/'=1 sim( (ok,,) considered in the main paper,
$
we consider alternative normalizer: Z, =
プソ= sim sim(p(a)). which gives us following
attention operation:
f Σ sim((p(g (kk)
E/ sim sim(p((g))))) (k,)))
(43)
- sim (i
For softmax attention, we have (am((p(g)),)))) :p(k) exp( 9ikk vil
then we can derive that
Tike 0. ΣE exp(
10
9:kk exp(
vil
Vz.
(44)
g Vdn/p
Σ exp(
9:ke Vdn
j=1
Ej'=1
exp(
This is equivalent to adding temperature 1/p into the softmax attention logits, and then taking the p-root of attention scores after softmax. We call this p-normalized softmax attention. p regular softmax attention. Similarly, we construct the LMs with p-normalized sigmoid attention.
We find that when p or p = 3 p 4, pre-training p-normalized softmax attention diverges and the loss goes infinity. When = 1/2 or 1/3 or = 1/4, LM pre-training converges. As the attention scores are not added up to one, we investigate the massive activations instead, as visualized in Figure (Lefr). With smaller p. massive activations are mitigated to some extent, but not as effective as sigmoid attention without normalization. Intuitively, smaller p induces a larger temperature in softmax operation, which leads to flattened attention logits.
Afterward, we conduct experiments on p-normalized sigmoid attention. There no training problem with larger than 1. As visualized in Figure 29(Righr), LMs with p-normalized sigmoid attention still demonstrate strong massive activations. To conclude, different normalizers may affect the amplitude of attention sink, but not stop its emergence.
Atention operations. Firstly, we present all attempted attention operations Table 15. is noted that several setups lead to training failure. For LMs with sigmoid attention without normalization. we vary the learning rates or weight decay ratios y. Consequently, as shown Table 14(Right), the trained LMs still have attention sink, which further confirms our conclusion in the main paper.
29

---

Published as a conference paper at ICLR 2025
Softmax
Sigmoid
40
20
+- p
p
1/2
32
p
lg No-Nomm Ratio
16
p
1/3
/
p
24
p
12
sigmoid, nO
norm.
16
8
0
2 3
8
9
10
4 5
10
Block
Block
Figure 29: f2-norm ratio of hj and mean of h eu (Left) p-normalized softmax attention and also sigmoid attention without normalization (as reference). (Right) p-normalized sigmoid attention and also sigmoid attention without normalization (as reference).
Table 15: Normalization and selections kernels in attention significantly affect the emergence of the attention sink. We use "*" mark that the metric Sink; is computed by proxy attention scores. We use "." to represent the training failure under the setup.
sim (im()p(ii). (kez)
Zi
Sink (%)
valid loss
exp(
9.k
E exp( vola
18.18
3.73
Van
g.k.
sigmoid
0.44*
3.70
vdn
sigmoid(
q.k
Σ E sigmoid(
qik² vily
30.24
3.74
Vdn
elu( 9.
+
0.80
3.69
Vdd
elu( g
Σ p' elu k.
+
dd,.
vd,
(elu(q,
+ ()(clu(k)
Σ (clu(q,) + (ollu( )+1)T
53.65*
4.19
vih
Vdn
(elu(q:) +]((elu(k,
Vdn
9.
q.k van
Vdd
max
.1
k.
0.00
3.99
Vag
mlp(a.)mlp(()) Van
mllp(k;))
E
mlp(a: )mlp(,e)) von
.1)
0.19*
3.85
mipig. mlppq.))lp(k))) )
0.74*
3.91
Vdn.
30

---

Published as a conference paper at ICLR 2025
E MORE EXPERIMENTS IN LM AFTER -RA-TNNNNING
Training stability in supervised fine-tuning, To investigate the long-term impacts of attention sink on model behaviors after pre-training, we conduct supervised fine-tuning (SFT) on our pre-trained IB LMs with softmax attention and sigmoid attention without normalization. Specifically, we utilize the platform'
conduct our experiments. The experimental configurations include: the UltraChat dataset (about 200k training samples)* (Ding et al., 2023), full-model fine-tuning, a learning rate of 2e-5 with cosine scheduling, batch size of 64, each of which contains 2048 tokens, one training epoch. As shown in Figure 30, we monitor the training loss and gradient norm of our two LMs during SFT. These two models behave similarly terms of the above two metries. Additionally, despite no attention sink, LMs with sigmoid attention without normalization have no issues of training stability during SFT. Though not from the attention sink perspective, a concurrent work by Ramapuram et al. (2024) discussed the theory and practices for Transformer models with sigmoid attention in detail. We refer the readers to Ramapuram et al. (2024) for more analyses.
2.8
1.4
2.6
sigmoid
Gradient Norm
1.2
sigmoid
Loss
softmax
1.0
softmax
2.4
0.8
2.2
0.6
2.0
0.4
1.8
0.2
1.6
0.0
0.5
1.0
2.0
0.5
1.5
2.0
Steps (k)
Steps (k)
Figure 30: The training loss and gradient norm in our 1B LMs with softmax attention and sigmoid attention without normalization in supervised fine-tuning.
thttps://github..c https///////ggggacc//////aa////a ht ggmennthanndbokk
31


Reviews:
[{"review_id": "uFFuRvlKnP", "reviewer": "Reviewer_5eYD", "summary": "The paper investigates the phenomenon of attention sink in LMs, where significant attention is allocated to the first token regardless of its semantic importance. Key findings include the correlation of the sink position with the loss function and data distribution, and the behavior of attention sink as key biases storing extra attention without contributing to value computation. The paper also shows that attention sinks do not emerge when softmax attention is replaced with other operations like sigmoid attention without normalization, up to 1B parameters.", "questions": "- How to identify the token where attention-sink occurs? How to determine those locations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a thorough investigation into the attention sink phenomenon, covering various factors that influence its emergence in LMs.\n- The findings have practical applications in areas such as streaming/long context generation, KV cache optimization, and model quantization.\n- The study considers various model architectures, optimizers, and data distributions, offering a broad perspective on the attention sink phenomenon.", "weaknesses": "- The study notes that attention sink disappears with less training data, suggesting that the models might be overfitting to certain aspects of the training data. It seems that these two factors cannot be decoupled, and it is impossible to explain whether it is over-fit or the amount of data that affects attention-sink.\n- The study claims that it focuses on the Language Models. However, this work pay more attention on auto-regressive LMs and may not capture the nuances of other types of encoder-like LMs or Jamba. Does those architecture can solve the attention-sink phenomenon?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QzxtiBzOqp", "reviewer": "Reviewer_xfDZ", "summary": "In this paper, the authors conduct a comprehensive study of the attention sink problem and present robust empirical results. They discuss and examine the attention sink problem from various perspectives, including optimization, data distribution, loss function, and model architecture. Although the paper does not provide in-depth theoretical analysis, it may inspire further research into the understanding of the attention mechanism, which could, in turn, contribute to the development of stronger generative models. Therefore, I believe this paper would serve as a valuable empirical reference on the attention sink problem for the community and worths of acceptance.", "questions": "Q1: In Fig. 4 (left), could you provide a performance curve, such as the validation performance of each model against model size? It appears that as the model becomes stronger, attention sink becomes more prominent. Additionally, I am curious about how attention sink correlates with validation loss.\n\nQ2: Fig. 4 (right) shows that a lower learning rate results in a slower increase in attention sink. Is it possible that attention sink occurs simply because the model has not been well-tuned?\n\nQ3: In Table 1, GPT-XL behaves very differently from Llama and Mistral. Do you have any intuition as to why this might be?\n\nQ4: Could you provide some mathematical formulation on how the attention variants (2nd to 4th rows in Table 4) are correlated? Is there an inherent connection among these attention variants that excludes attention sink?\n\nQ5: In Table 6, it appears that the normalizer may impact attention sink. Do you have additional evidence with other types of normalizers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The perspectives of studying attention sink problem is diverse and well-motivated. These perspectives are also inspiring to future studies on attention mechanisms.\n2. The experiments is very comprehensive and diverse.", "weaknesses": "The primary weakness of this paper is the lack of in-depth analysis. The empirical results come across more as observations rather than as thorough investigations. Including more theoretical analysis or deeper experimental work would strengthen the paper. For instance, in the KV bias section, it would be beneficial to explore how attention variants with and without attention sink are related in formulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eJeK6qM0qK", "reviewer": "Reviewer_huvS", "summary": "This paper studies the phenomenon of \"attention sink\" in Language Models, where significant attention is disproportionately allocated to the first token in input sequences, without taking into account its semantic importance. The attention sink has practical applications in areas such as streaming generation, KV cache optimization, inference acceleration, and model quantization. \n\nThe authors performed a study across various LM architectures, input types, and training configurations to study the emergence and characteristics of attention sinks.\nThey found that attention sink happens across different LMs, including smaller models and those trained on random token sequences and that it emerges during optimization with sufficient training data, and its prominence is influenced by learning rates and weight decay.\nThey also found that attention sink functions similarly to key biases, storing additional attention scores that are largely non-informative and do not contribute to value computations. This behavior is partially attributed to the softmax normalization, which induces inner dependencies among attention scores.\n\nThe paper also shows that altering the attention mechanism, such as replacing softmax with sigmoid attention without normalization, can prevent the emergence of attention sinks in LMs up to 1 billion parameters.", "questions": "In your paper you study how to mitigate the attention sink, are you sure that we want to mitigate it?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- Reproducibily.\n- The paper is well written and the experiments set up is soild.\n- The paper not only observes the presence of attention sinks but also delves into the mechanistic reasons behind their emergence (also providing insights on the training dynamics that foster attention sinks).\n- The paper show that attention sinks persist across various input types, like random token sequences and repeated tokens.\n\nI think that this paper is a valuable step further in understanding the phenomenon of attention sink.", "weaknesses": "- The paper doesn't really take into account the role of attention sink impact on downstream tasks.\n- While the paper studies the emergence and immediate characteristics of attention sinks, it does not assess the long-term impacts of attention sinks on model behavior (like stability during fine-tuning, adaptability to new tasks, or resistance to adversarial attacks).\n\nNot really a weakness, but i found the paper a bit difficoult to read because of the unusual aesthetic choices: i think that this style would be great for a blog post where you have more space available to write, but i personally find easier to follow simpler-looking papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)