Massive Activations in Large Language Models
Authors: Mingjie Sun
Xinlei Chen
J Zico Kolter
Zhuang Liu
Venue: COLM (2024)
Keywords: massive activations, biases, self-attention
TL;DR: Massive activations exist in LLMs, and they are fundamentally connected to self-attention.

Abstract: We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers.

Published as a conference paper at COLM 2024
Massive Activations in Large Language Models
Mingjie Sun
Xinlei Chen
J. Zico Kolter
Zhuang Liu
Carnegie Mellon University
Meta Al Research
Bosch Center for AT
24
24
1k
1k
Summer
warm Winter
cold
1415
2533
Summer
Wint
codd
bn)
1415
2533
win
summer
warms
and
wirtrr
1415
2533
the bottt
1415
2533
Figure 1: Activation Magnitudes (z-axis) LLaMA2-7B. x and y axes are sequence and feature dimensions. For this specific model, we observe that activations with massive magnitudes appear in two fixed feature dimensions (1415, 2533), and two types of tokens the starting token, and the first period (-) or newline token ('n)).
Abstract
We observe an empirical phenomenon in Large Language Models (LLMs)- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demon- strate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the con- centration attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers. Code is available https: //gi thub. com/
1 Introduction
Large Language Models (LLMs) (Brown et al., 2020; OpenAl, 2023) have demonstrated remarkable capabilities. The majority of existing studies conducted on these models are focused on their external behaviors, e.g,, evaluating their performance on various tasks (Katz et al., 2023; Bubeck et al., 2023), developing prompts to elicit accurate responses (Wei et al., 2022; Yang et al., 2023). While these studies are encouraging and highlight the potential of these models, it is also important to gain insights into their internal mechanisms, especially as they are being increasingly integrated into many real-world applications. However, research on the internal workings of these models remains relatively limited.
In this work, we discover and study surprising phenomenon in the internal representations of LLMs. Examining the hidden states in these models, we find that certain activations exhibit huge magnitudes, e.g., more than 4 orders of magnitude larger than the median, and could take on absolute values larger than 15,000 in LLaMA2-70B (Touvron et al., 2023), despite the presence of normalization layers. These activations are also extremely rare, often numbering fewer than 10 among tens of millions of total activations. Figure illustrates this phenomenon in LLaMA2-7B. As these activations are so much larger in magnitudes compared to others, we name them massive activations. We demonstrate their presence in wide range of LLMs, spanning different model sizes and families.
We explore where massive activations are located in LLMss. Regarding the depth dimension of LLMs, the appearance of massive activations is mostly abrupt: they emerge suddenly
* *Correspondence mingjies@cs. cmu, edu and zhuangl @me ta. com.

---

Published as a conference paper at COLM 2024
after single layer of computation, and diminish at the last few layers. Further, we find massive activations occur in a small number of feature dimensions that are input agnostic. Many of these activations are found within the starting word token and delimiter tokens. Additionally, we show that massive activations are not the same as outlier features (Dettmers et al., 2022), a previously known phenomenon in LLMs.
We show that massive activations act as fixed but crucial bias terms in LLMs. Here by bias terms, we mean certain internal states of the models that are independent from the inputs, analogous to the bias term b in a linear layer y Wx + b. First, we show that massive activations play a critical role in LLMs' capabilities. For instance, in LLaMA2-7B, setting merely four massive activations (out of millions of activations) to zero would result in catastrophic collapse in model performance. Further, setting them to their mean values does not hurt the model, suggesting their role is equivalent to simple constant biases. Our analysis reveals that after the initial layers, LLMs repurpose the tokens linked with massive activations to store these important biases.
Intriguingly, massive activations are closely connected with self-attention. In particular, we show massive activations cause attention to be attracted to the tokens associated with them. Our findings extend the observations from "attention sinks" (Xiao et al., 2023b)-we demonstrate that LLMs allocate excessive attention to more than just the first token, and provide an in-depth analysis on how such attention concentration patterns arise. Our analy- sis suggests that LLMs try to learn implicit bias components in self-attention via massive activations, during pretraining. We thus experiment with augmenting self-attention with extra key and value embeddings, explicitly designed as biases. Remarkably, we demonstrate that training with them eliminates the need for LLMs to learn massive activations.
Finally, we also observe massive activations in Vision Transformers (ViTs) (see Appendix A). They appear in many of the ViTss we have examined. In these ViTs, they tend to appear Tixedfee feature dimensions, but notably at varying patch tokens. Moreover, we find that
at fixed these activations act similarly as fixed biases. Notably, we relate these findings to the recently proposed "register tokens" (Darcet et al., 2023). We show that they both learn values independent of input images. This offers an alternative interpretation for register tokens, where they were originally hypothesized to aggregate global image information.
2 Massive Activations
We study autoregressive Transformers, which are built by a stack of L decoding layers. Each layer& e takes the previous hidden state h,-1 E R as input and outputs a hidden state hy RTxd iis the number of tokens and d is the number of features. Transformer layers use residual connections (He et al., 2016), and the computation can be formulated as:
h ht-11 + (hg-1)
(1)
where F& is the residual transformation. Note that this includes both attention and MLP blocks. An activation denotes specific scalar value in a hidden state. Unless otherwise specified, our study of activations is on the hidden state h&, i.e., the output of residual summations, not any intermediate states inside F;,
Existence in LLMs. We start with an illustrative example on LLaMA2-7B. In Figure 1, we visualize the intermediate features hy of interest. We feed this model with short sentences and visualize the activation magnitudes (z-axis) of the hidden states at middle layer. and y axes are sequence and feature dimensions respectively. Each blue row corresponds to the feature embedding of one token. We observe up to four activations with significantly large magnitudes. The largest activation (about 2,000) is approximately 10,000 times larger than the median magnitude (about 0.2). The sheer scale of these activations makes them stand out from others. We thus refer to these special activations as massive activations.
Massive activations are not unique to this specific model LLaMA2-7B, but are widely observed in LLMs. In Figure and Figure 3, we demonstrate the existence of massive activations in both LLaMA2-13B and Mixtral-8x7B (Jiang et al., 2024). Notably for Mixtral- 8x7B, the largest activation magnitude can reach an absolute value of 7 7,000, around 4 orders of magnitude larger than the median feature magnitude (around 0.3). We refer the reader to Appendix B for results on more pretrained and fine-tuned L.Mss.
2

---

Published as a conference paper at COLM 2024
LLaMA2-13B
1k
1k
16
16
4743
4743
4743
4743
Summer
wam
Winte
2100
Summatt
Winer
2100
Why
summerr
and winter cold
2100
thei best
of times
2100
?
Figure/ 2: Massive activations in LLaMA2-13B. In this model, they appear in two fixed feature dimensions (2100, 4743), and are limited to the starting token.
Mixtral-8x7B
1k
5k
8k
5k
3398
3398
Summer
womn
Wanter
10
cold
2070
Summary
womm
Winter
Un
2070
why
aummer warm and winter calrt
2070
the
beat
times
2070
Figure 3: Massive activations in Mixtral-8x7B. In this model, they lie in two feature dimensions (2070, 3398), and are found within the starting token, delimiter tokens and certain word tokens ("and" and "of").
Properties. We summarize two main properties of massive activations. The most notable property is that these activations possess massive values and their magnitudes are sig- nificantly larger than other activations, often several orders of magnitude larger than the median value. Another property is that they are exceptionally few in number. For LLaMA2- 7B in Figure 1, there are approximately 40,000 total activations in each presented hidden state but at most four massive activations can be identified.
Quantitatively, we present the values of the top activation magnitudes in Table We also provide a loose but broad definition: an activation qualifies as massive activation if its magnitude surpasses 100 and is at least or around 1,000 times larger than the median magnitude of its hidden state. We find this criterion to effectively identify these activations of interest across various LLMs, which are emphasized in bold in Table 1.
Model
Top 1
Top 2
Top 3
Top 4
Top 5
Top 10
Top 100
Top 1%
Top 10%
median
LLaMA2-7B
2622.0
1547.0
802.0
477.3
156.9
45.7
10.6
1.1
0.6
0.2
LLLMA2-13B
1264.0
781.0
51.0
50.5
47.1
43.5
16.6
1.9
1.1
0.4
Mixtral-8x7B
7100.0
5296.0
1014.5
467.8
302.8
182.8
90.8
3.0
1.0
0.3
Table 1: Five largest, top 1% and 10%, and the median activation magnitudes at hidden state of three LLMs. Massive activations in these LLMs are highlighted in bold.
Next, we identify the locations of massive activations within LLMs. For a comprehensive analysis, rather than using short sentences as inputs, we collect 100 sequences (each with 4,096 tokens) from RedPajama (Together Computer, 2023). We run LLMs on these 100 sequences and collect the hidden states from each layer.
2.1 Which Layers?
We determine the layers whose output hidden states exhibit massive activations. In Figure 4, we visualize the three largest activation magnitudes and the median of the hidden state output of each layer, with results averaged over 100 sequences. We examine three models: LLaMA2-7B, 13B and Phi-2 (Javaheripi al., 2023) (see Appendix B.4 for more LLMs). In all cases, each of the top three activations comes from the same position in the hidden state across most of the middle layers. Generally, observe the following:
3

---

Published as a conference paper at COLM 2024
LLaMA2-7B
LLaMA2-13B
Phi-2
3k
Magnitudes
1K-
2k-
1k-
500
500
0
16
24
32
10
20
30
40
1
16
24
32
Layers
Layers
Layers
Top 1
Top 2
Top 3
Median
Figure 4: Three largest and the median activation magnitudes at each layer in LLMs.
Massive activations exist and remain as largely constant values throughout most of the intermediate layers. They emerge in the initial layers and start to diminish in the last few layers.
In LLaMA2-7B, massive activations first appear in layer 2 and remain nearly constant values until layer 30. Intriguingly, for LLaMA2-7B and 13B, massive activations emerge very rapidly from one layer of computation, e.g., layer 2 and layer 4 respectively. This means that they do not emerge as result of gradual accumulation through many layers, and are caused by a rather different mechanism.
2.2 Which Feature and Sequence Dimensions?
We determine the locations of massive activations within hidden states, i.e, their feature and sequence dimensions. Since we have shown that their values largely stay constant in middle layers, we take on any such layer for this analysis.
LLaMA2-7B. In this model, massive activations are identified in two feature dimensions (1415 and 2533). Regarding sequence dimensions, we find that massive activations appear at: 1. the starting word token, 2. the foken representing the first period (.) or newline token ((n) in the sequence. Figure illustrates these findings for LLaMA2-7B. This is also consistent on long sequences. In cases where the input contains are observed. For the less common scenario where neither ". " noor "nn" is present, we can see two massive activations, both of which are associated with the initial token.
n" token, four massive activations
LLaMA2-13B. We find that massive activations in this model consistently appear in two feature dimensions, 2100 and 4743. These activations are exclusively located within the starting token of the sequence, regardless of its semantics. Figure 2 illustrates these behaviors within LLaMA2-13B. For any given input sequence, only two massive activations are present, corresponding to features 2100 and 4743 of the first word token.
Mixtral-8x7B. For this particular model, massive activations lie in two feature dimensions, ie., 2070 and 3398. For sequence dimensions, we find that they are associated with the starting token, delimiter tokens and also certain word tokens, e.g., token "and" and token "of". Figure showcases these patterns within Mixtral-8x7B. These word tokens tend to be conjunctions and prepositions, representing relatively few semantics of the input. Generally, for inputs of 4096 tokens in length, these tokens are predominantly located in the early part of sequence.
Summary. We summarize our findings of massive activations beyond the three models discussed above. We also put other models into categories based on empirical observations.
For feature dimensions, they are consistently present in very few fixed dimensions.
For sequence dimensions, classify LLMs into three categories:
a) Starting token only. Models include LLaMA2-13B, MPT and GPT-2.
b) Starting token and the first "strong" delimiter token (i.e., "." or "\n"). Models include LLaMA2-7B and L.aMA2-7B-Chat.
c) Starting token, delimiter tokens (such as ".", "\n", "." or " "), and certain word tokens with weak semantics (such as "and", "from", "of" or "2"1). Models include LLaMA2-70B, Mistral-7B, Mixtral-8x7B, Falcon-40B and Phi-2.
Such numeric tokens exhibit massive activations only in certain contexts, e.g., dates and years. Refer to Figure 17 for an illustration L2MA2-000.
4

---

Published as a conference paper at COLM 2024
2.3 Difference from Outlier Features
With an understanding of the nature and locations of massive activations, we now discuss the differences between them and outlier features, seemingly similar phenomenon in LLMs. Dettmers et al. (2022) have discovered outlier features in LLMs, characterized by large magnitudes. Conceptually, massive activation is a scalar value, determined jointly by the sequence and feature dimensions; in contrast, an outlier feature is a vector, corresponding to activations at all tokens. Further, massive activations are present at extremely few tokens, while outlier features expect most activations in them be large.
In practice, we find that massive activations do not overlap with outlier feature dimensions. We identify outlier features in LLaMAA-7B and 13B using the definition in Dettmers et al. (2022): a feature is deemed as an outlier feature if activation magnitudes exceed 6.0 at more than 25% of layers and 6% of tokens, on more than 90 out of 100 sequences. We discover 10 and 25 outlier features in these two models respectively. However, none of them correspond to the feature dimensions of massive activations.
3 Massive Activations Act as Biases in LLMs
While we have demonstrated the existence of massive activations and identified their locations, their functional role within LLMS is not yet clear. Are they important for internal computation? Or are they simply redundant activations with no effect? This section will delve deeper into LLMs to answer these questions. Different from the previous passive observations, we take a more proactive approach by inspecting how modifying massive activations affects the external behavior of LLMs.
Model
Top 1
Top 2
Top 1%
Top 10%
Median
LLaMA2-7B
2556.8 + 141.0
-1507.0 + 83.0
-0.14 + 0.6
0.0 + 0.5
0.2 + 0.3
LLaMA2-13B
11777.5 + 14.6
-787.8 = 8.0
0.9 + 0.7
-0.3 + 0.8
-0.3 = 0.6
Table 2: The mean and variance of activation values at various positions, corresponding to the 2 largest, top 1% and 10,, and the median magnitudes within the hidden state. Variations in massive activations are significantly lower in comparison to other activations.
We first measure the variances of massive activations across input sequences. Besides massive activations, we choose three other positions based on their average magnitudes, corresponding the top 1%%110, and the median within the hidden state. In Table 2, we show the mean and standard deviation of the activation values at these positions across 100 sequences, for LLaMA2-7B and 13B. We find that the variances of massive activations are considerably smaller relative to their mean values when compared to other activations.
We then modify the inference of LLMs by intervening massive activations at one layer-for a hidden state exhibiting massive activations, we manually set these activations to chosen fixed values. Then the altered hidden state is fed into the next layer, and the computation afterwards continues as normal. We modify massive activations in LLaMA2-7B and 13B, We evaluate the perplexity on Wiki'lext, C4 and PG-19 and the mean zero-shot accuracy BoolQ, PIQA, WinoGrande, Arc-Easy and Arc-Challenge. For each model, we perform the intervention once on the hidden state where massive activations first appear. This corresponds to layer 2 and layer 4 in LLaMA2-7B and 13B respectively.
Setting massive activations to zero. We evaluate the performance of LLMs without massive activations. We set their values to zero in the hidden state when they first appear, i.e., removing massive activations from intervened LLMs. The results (denoted by Set to zero) are
LLaMA2-7B
LLaMA2-13B
Intervention WikiText C4
PG-19
Mean Zero-Shot WikiText
C4
PG-19
Mean Zero-Shot
Original
5.47
7.85
8.57
68.95%
4.88
7.22
7.16
71.94%
Set to zero
inf
inf
inf
36.75%
5729
5526
4759
37.50%
Set to mean
5.47
7.86
8.59
68.94%
4.88
7.22
7.16
71.92%
Table 3: Intervention analysis of massive activations in LLaMA2-7B and 13B. We set massive activations to fixed values and evaluate the perplexity (L) and mean zero-shot accuracy (t)-
5

---

Published as a conference paper at COLM 2024
shown in Table 3. Intriguingly, there is significant degradation in model performance, e.g. exploding perplexity numbers. For comparative analysis, an equal number of activations- those with average magnitudes close to the median magnitude-are similarly set to zero We find this leads to no performance drop. These results highlight the crucial role that massive activations play in the internal computation of LLMs.
Setting massive activations to mean values. We remove the small variances in the values of massive activations. Specifically, we adjust the values of massive activations to their empirical mean values. The means are computed results of this intervention (denoted by Set to mean) are shown in Table 3. We find that there are negligible changes in perplexity and zero-shot accuracy. This shows that their values are constants and input agnostic, i.e., functioning similarly to bias terms.
100 sequences from RedPajama. The
To summarize our findings:
Massive activations act as fixed but important biases LLMs.
Why these layers and tokens? The fact that these activations act as biases may explain why LLMs store them at certain layers and tokens:
The tendency of these activations to appear at the starting token could be attributed the fact that every autoregressive training instance contains an initial token. Since LLMs are based on next word prediction, the starting token is the only token used in all forward passes within sequence.
The existence of these activations in delimiter tokens might be due to the relatively low semantic value these tokens, rendering them a low-cost option for storing such biases. Conversely, tokens with rich semantics would risk sigmificant loss of input information, if they are repurposed to store biases.
The fact that massive activations emerge only after few initial layers might be because LLMs require some initial layers to process the meaning the tokens asso- ciated with massive activations. At these layers, their semantics may be transferred other token positions via self-attention, and preserved moving forward.
4 Effects on Attention
In this section, we study the internal mechanism of massive activations in LLMs, particularly in relation to self-attention.
4.1 Attention is Concentrated on Massive Activations
We observe stark contrast in attention patterns when comparing layers before and after the appearance of massive activations in LLMs. Figure 5 shows the attention logits (before softmax), averaged over all heads per layer in LLaMA2-7B. The input is a prompt from MMLU (Hendrycks et al., 2021): "The following
multiple choice questions (with answers) about machine learning. in),..... ...". Recall that in LLaMA2-7B, massive activations first appear in the output of layer 2 (see Figure 4). We find that in layer 3 and deeper layers (e.g., layer 31), attention is mostly concentrated on the two tokens associated with massive activations. Our observations are also consistent across various LLMs. Figure 6 demonstrates such attention concentration patterns in LLaMA2-13B and Phi-2, on the same input. See Appendix C.1 for results on more LLMs.
LLaMA2-7B, Layer 2
LLaMA2-7B, Layer 2
LLaMA2-7B, Layer 3
LLaMA2-7B, Layer 31
massive activations
0
massive activations
Quirry
0
Tho
2533
1415
Key
Kay
Key
Figure 5: Attention patterns before and after massive activations appear in LLaMA2-7B. For each layer, we visualize average attention logits (unnormalized scores before softmax) over all heads, for input sequence.

---

Published as a conference paper at COLM 2024
LLaMA2-13B, Layer 4
LLaMA2-13B, Layer 5
Phi-2, Layer 8
Phi-2, Layer 9
The
The
0
1K
In
Quary
500
Query
0
4748
The
2100
The
250
Key
Key
(a) LLaMA2-13B
(b) Phi-2
Figure 6: Attention patterns after massive activations emerge in LLaMA2-13B and Phi-2.
We notice that there is a consistent pattern across models on the distribution of attention logit values. In Figure and Figure 6, many attention logits tend to be negative following massive activations. They are mostly computed by the inner product between query and key states of tokens without massive activations. However, when the key states belong to tokens associated with massive activations, the resulting attention logits are slightly positive. Thus the attention softmax (computed along each row), these special attention logits will attract most of the attention probability.
Recently, Xiao et al. (2023b) showed that LLMs attend heavily to the starting token. Our findings on LLaMA2-13B in Figure 6a align with their results. Empirically, we find it is true for LLMs where massive activations are only found within the starting token. However, our results LLaMA2-7B and Phi-2 indicate that LLMs also allocate substantial attention to other tokens and they are associated with massive activations. Furthermore, our results reveal deeper cause for the emergence of these attention concentration patterns.
4.2 Massive Activations Impose Implicit Attention Biases
In this part, we delve into the computation within the attention block and demonstrate that LLMs use massive activations to enforce implicit bias term in self-attention.
Attention LayerNorm and QKV projections. We study the impact of massive activations on the query, key and value states (Q/K/V) in self-attention. In LLMs, at each layer, input features are processed by layer normalization" (Ba et al., 2016) and then transformed into query, key and value states via linear projections, as illustrated in Figure 7a. This design choice is introduced in GPT-2 (Radford et al., 2019) and widely adopted in modern LLMs.
LayerNorm Input
LayerNorm Middle
LayerNorm Output
25
Normalize
LayerNorm
Middle
Scale
K
10
Output
Linear
Q
K
(b) Layer 3, LLaMA2-7B. We highlight the embeddings of the two tokens where massive activations appear: the starting token and the period token.
(a) Attention LayerNorm and OKV linear projections.
Figure 7: Activation trajectory from input hidden states to query, key and value states.
2 a MA2 variant of layer normalization: RMSNorm (Zhang & Sennrich, 2019).
7

---

Published as a conference paper at COLM 2024
Value Updates
Σ Effo ICC
Value Updates
Σ igC
Attention Output
0.2
0.2
0.2
0.00
0.0
0.0
-011
Winte
Water
Witer
ood
Same across all tokens
Figure 8: When computing the attention output, value updates computed from tokens associated with massive activations (the set C) are essentially the same.
Figure 7b visualizes all hidden states computed in this schematic (LLaMA2-7B, layer 3). We find that at all stages, features of the two tokens associated with massive activations are drastically different from other tokens. Specifically, after the first "normalize" step, the embeddings of these two tokens appear elements. Notably, the subsequent QKV states exhibit considerably smaller variations within each embedding. We hypothesize that the attention LayerNorm may play a pivotal role in this process (see Appendix C.2 for further discussion).
sparse vector with two distinct non-zero
Attention output decomposition. Given that attention is also concentrated on the tokens associated with massive activations (Section 4.1), we thus isolate these tokens and study their effects on the attention output (the layer of attention matrix multiplying value vectors). In Equation 2, we decompose the attention output at each token k into two parts: value updates from the tokens Ć where attention is concentrated; and value updates aggregated from other tokens. Here 5 is the attention distribution of query token k to token i, and i; is the value state of token i.
AttentionQ,KKVV Σr.o, Σ
+
Σ p IEC
(2)
Figure 8 visualizes the decomposed value updates and the attention output in LLaMA2-7B, with the input prompt "Summer is warm. Winter is cold.". In this case, the set C consists of token Summer and the first period token. We can see that the value updates from C are nearly identical across tokens, i.e., they serve as additive bias terms, although not explicitly imposed. Furthermore, we note that this pattern of value update is strikingly similar across various inputs. We refer the reader Appendix C.3 for additional analysis. Overall, our results indicate that LLMs use massive activations to allocate substantial attention at certain tokens. These tokens are then utilized by LLMs to form constant bias term when computing the attention output.
4.3 Explicit Attention Biases Eliminate Massive Activations
Given the strong need of LLMs to learn implicit attention biases during pretraining, we thus experiment with directly augmenting self-attention with additional bias terms. Intriguingly, we find that models with explicit attention biases do not exhibit massive activations.
The idea is to model such attention biases explicitly, except not through repurposing existing tokens in the input sequence. Thus we introduce additional learnable parameters k', v' E I for each head. Specifically, given input query, key and value matrices V R Txd the augmented attention with explicit attention biases computed as:
Formulation.
[KKKT K'
V T
Attention K, V; k, -oftmax
(3)
where k' and v' are each concatenated with the key and value matrices K/V. The proposed attention can be used as a drop-in replacement of standard attention, without modifying other parts of Transformers, e-g., positional embeddings and MLP blocks.
8

---

Published as a conference paper at COLM 2024
GPT-2 Default
GPT-2 with Sink Token
GPT-2 with Attention Blas
2k
2k
1k
1k
1k
Summer
is
warm
Winter
is
cold
SINK
Summer
is
warm
Winter
16
cold
Summer
16
warm
Winter
i6
cold
Figure Massive activations disappear when training GPT-2 with explicit attention bias.
GPT-2 Default
GPT-2 with Sink Token
GPT-2 with Attention Bias
1K
Magniludes
2k
200
500
1k:
100
Layers
Layers
Layers
Top
Top 2
Top
Figure 10: Three largest activation magnitudes at each layer for three GPT-2 models.
GPT2 with Blus, Layar
GPT-2 Afturtion Blss Lapard Head0
0.6
Query
Qh-sry
0.4
0.4
.02
0.2
02
festtoken
Key
Key
K'
KV
Figure 11: Attention distribution in default GPT-2 and GPT-2 with explicit attention bias.
Results. We train three GPT-2 models: the standard model, GPT-2 prepended with a sink token (Xiao et al., 2023b) and GPT-2 with explicit attention biases. See Appendix C.4 for training setups. We find that the three models have the same performance convergence but differ significantly in the status of massive activations, as demonstrated in Figure 9, Notably, in GPT-2 with explicit attention biases, massive activations disappear, as compared to the default GPT-2 and one with a sink token.
Figure 10 shows the three largest activation magnitudes at the output of each layer in GPT-2. Notably, with explicit attention biases, top activation magnitudes in GPT-2 are increasing gradually as layers go deeper. In contrast, for both the default GPT-2 and GPT-2 with a sink token, massive activations emerge within one intermediate layer, i.e. layer 4. These results indicate that explicit attention biases negate the necessity for LLMs develop massive activations during the pretraining phase.
In Figure 11, we visualize the attention distribution in both default GPT-2 and GPT-2 with explicit attention biases, where we plot the average attention probability over 50 sentences each with 30 tokens. First, we find that our observations on the relationship between massive activations and attention concentration hold for the default GPT-2 model. Second, for the GPT-2 model with explicit attention bias, most of the attention probability is assigned to the extra k' and v' vectors we inserted. Intriguingly, this also holds for initial layers as well (e.g., layer 1), suggesting the strong need for LLMS to form this attention concentration pattern during pretraining.
To summarize findings in this section:

---

Published as a conference paper at COLM 2024
Massive activations are connected to self-attention. LLMs use massive actioations to concentrate substantial atlention on very few tokens, injecting implicit bias terms in the attention computation. Further, massive activations can be eliminated by augmenting LLMs with explicit attention biases.
Related Work
Intriguing properties of autoregressive Transformers. Timkey & Schijndel (2021) observed that in GPT-2's penultimate layer, there are feature dimensions containing activations with magnitudes up to 3,000, and these dimensions dominate standard measures of representa- tion similarity. Heimersheim Turner (2023) found that the feature norm of the initial token in GPT-2 grows much faster than other tokens. Kovaleva et al. (2021) and Zhao et al. (2023) discovered outlier weights in the LayerNorm of GPT-2 and LLaMA2-13B, and setting them to zero leads to catastrophic drop in model performance. Notably, the feature dimension of this weight in LLaMA2-13B (i.e., 2100) corresponds to that of massive activation (Figure 2).
Outlier features. Various works in quantization (Dettmers et al., 2022; Zeng et al., 2022; Xiao et al., 2023a; Lin et al., 2023; Ahmadian et al., 2023) studied the existence of outlier features in LLMs, Dettmers et al. (2022) showed that outlier features have large activ activation values in most of their sequence dimensions. While massive activations can be seemingly similar to outlier features, we discussed their fundamental differences in Section 2.3. We show that massive activations can not be attributed to the existence of outlier features.
Attention concentration patterns. Clark et al. (2019b), Kovaleva et al. (2019) and Bondarenko et al. (2021) discovered that attention in BERT (Devlin et al., 2018) tends to focus on the "separate" token [SEP]. Xiao et al. (2023b) showed that LLMs assign most of the attention to the starting word token. Darcet et al. (2023) revealed the existence of attention artifacts in ViTs. Robinson et al. (2023) found sparse activation patterns in ViTs that attract attention to certain tokens. Our work provides an in-depth analysis as to why these patterns emerge, specifically in relation to massive activations,
Biases in self-attention. There can be various notion of biases in the self-attention mecha- nism. First, simple additive bias terms can be used in linear layers for computing the query, key and value states (Namazifar et al., 2023). Many LLMs do not enable additive bias terms for linear layers during the pretraining phase (Chowdhery et al., 2022). Second, position biases can be inserted in self-attention to encode positional information of each token (Su et al., 2021; Press al., 2021). There are also variants of biases with manually designed softmax operator (Miller, 2023; Bondarenko et al., 2023; Hu et al., 2024). Our work reveals that LLMs, even with standard self-attention, would impose implicit bias components in attention computation through massive activations.
6 Conclusion and Discussion
A Autoregressive training of large Transformers has brought significant advances in natural language processing. This study reveals the widespread existence of massive activations in these Large Language Models (LLMs). The values of these activations are input agnostic but crucial for model performance, despite their extremely rare quantity. We establish close connection between massive activations and the self-attention mechanism. We show that LLMs use them to implement an implicit form of biases for attention computation. We hope our results contribute to a deeper understanding of today's large-scale foundation models.
We discuss some practical implications and future directions of this work. First, the presence of activations with large magnitudes has been widely known as major challenge in effec- tively quantizing LLMs (Dettmers et al., 2022; Xiao et al., 2023a). This paper identifies a new type of outlier activations in LLMs, and we hope our findings will be valuable to researchers working on LLM compression. Second, attention maps that allocate excessive attention probabilities to few fixed tokens may be undesirable for mechanistic interpretability (Ols- son et al., 2022). Our proposed attention formulation could make the resulting attention maps in LLMs more interpretable, and benefit downstream applications (Darcet et al., 2023). Finally, our investigation of the new attention formulation focused on its effects on massive activations, and our experiments were limited to small GPT-2 model due to computational resource constraints. It would be interesting to see how our results generalize to models at larger scales, and how our attention formulation could affect the training stability (Wortsman et al., 2023) of modern LLMs.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We thank Sachin Goyal, Jeremy Cohen, Timothée Darcet, Koustuv Sinha and Mike Rabbat for valuable discussions. Mingjie Sun was supported by funding from the Bosch Center for Artificial Intelligence.
References
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Stephen Gou, Phil Blunsom, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization at scale. In NeurIPS, 2023.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, and et al. The falcon series of open language models. arXiv preprint arXiv:2311.1686) 2023.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXi preprint arXiv:1607.0645: 2016.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense natural language. arXiv preprint (XXe:1911.116411, 2019.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcom- ing the challenges of efficient transformer quantization. arXio:2109.12945 2021.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Quantizable transformers: Removing outliers by helping attention heads do nothing, arXiv preprint axXi:22306.12929 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Xiv preprint XrXo:2005..11666 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. Xiv preprint WXXv:2303.12712, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, and Paul Barham. Palm: Scaling language modeling with pathways. arXiv preprint ar 2rXiv:2204.03311 2022.
Christopher Clark, Kenton Lee, Ming-We: and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no no questions. ariiv preprint arXiv:1905.10044, 2019a.
Chang, Tom Kwiatkowski, Michael Collins,
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does bert look at? an analysis of bert's attention. ar preprint ar Xi:::1190 043441, 2019b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arX preprint
2018.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. ar XrXiv:2309.1653 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int()) 8-bit matrix multiplication for transformers at scale. In NewrlPS, 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanove. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint (110::8810088/00 2018.
11

---

Published as a conference paper at COLM 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00002, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. ar axXiv:2111.0637, 2021.
Stefan Heimersheim and Alex Turner. Residual stream norms grow exponentially over the forward pass, 2023. URL https://www/ al aliggmenttorumm org/ htttt/ssmmiimccdccc222eeee desidual-streand norms -ooorss--owwwo grow www--grooo--rooe exponentially--overr -wr--towwr-ddd.-----ooo-o----
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
Jerry Yao-Chieh Hu, Pei-Hsuan Chang, Robin Luo, Hong-Yu Yu Chen, Weijian Li, Wei-Po Wang, and Han Liu. Outlier--fficceet hopfield layers for large transformer-based models. ar preprint ar Xri::24044.0322 2024.
Mojan Javaheripi, Sébastien Bubeck, and et al. Phi-2: The surprising power of small language models, 2023. URL https: http://wwwwwmta www.wmofttoo..ooommmm cww/coo////////e ctrm//enuus/reearch//logg phi w--tth-suurrrisiin--ooeee----e haarrrof---mmll-aaaaaang--eeeeee anguage-models/.
Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, et al. Mistral 7b. arXiv preprint drXio:2310.06825, 2023.
2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Andrej Karpathy. Nanogpt, 2023. URL https: / //gitubbiittttttt com/Kaarpathy/nanoorr.
Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. Gpt-4 passes the bar exam. SSRN, 2023.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXit preprint Xiv::1190 08593, 2019.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. In ACL Findings, 2021.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. ar Xiv preprint ar Xrie:2306.00978, 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel Merity, mixture models. arXiv preprint arXiv:1609.07844 2016.
Evan Miller.
Attention is off by one, 2023. URL
org/
MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable lims, 2023. URL www. www.mosaicmll www.blo.////ccccccccccoooooooo com/blog/mmpt776
Mahdi Namazifar, Devamanyu Hazarika, and Dilek Hakkani-Tur. Role of bias terms in dot-product attention. arXiv preprint ar axXiv:2302.08626, 2023.
12

---

Published as a conference paper at COLM 2024
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, and Mare Szafraniec. Dinov2: Learning robust visual features without supervision. arXivo2304.077193 2024.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, and et al. Training language models to follow instructions with human feedback. arXiv preprint arXi::2003.02155, 2022.
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: biases enables input length extrapolation. ar preprint Xii:21088122000 2021.
Attention with linear
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. Technical Report, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, et al. Learning transferable visual models from natural language supervision. arXiv preprint ar arXio:2103.0002 2021.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arv preprint, 2019. URL https://arxiv. www.:///o///////////////o 1500577.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2020.
Brian S. Robinson, Nathan Drenkow, Colin Conwell, and Michael Bonner. A sparse null code emerges in deep neural networks. In NeurIPS UniReps Workshop, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. ar arviv preprint ar 0rXiv:1907.1000 10044,, 2019.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.098 2021.
William Timkey and Marten van Schijndel. All bark and no bite: Rogue dimensions in transformer language models obscure representational quality. Xit:2100.04404,, 2021.
Together Computer. Redpajama: open dataset for training large language models, October 2023. URL
com/ httptteeeeercompuerrrrmr wwwpttbbb.cmmmogggheerrooo//u//o
aRaaaajama-aaaaaaaaaaaaaaaajaaaa Data.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint XXix:2307.09288 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large lan- guage models. In Advancess in Neural Information Processing Systems, 2022.
Mitchell Wortsman, Peter Liu, Lechao Xiao, Katie Everett, and et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint ar arXiv:2309.14322, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. arX99:2309 17453, 2023b.
13

---

Published as a conference paper at COLM 2024
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. ar preprint ar arXi::2309.034444 00349,, 2023.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, and et al. Glm-130b: An open bilingual pre-trained model. arv preprint ar Xie::2210.0224 2022.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In NeurlPS, 2019.
Jun Zhao, Zhihao Zhang, Yide Ma, Qi Zhang, Tao Gui, Luhui Gao, and Xuanjing Huang. Unveiling core linguistic region in large language models. arXiv:2310.14928, 2023.
14

---

Published as a conference paper at COLM 2024
A Vision Transformers
In this section, we study if Vision Transformers (ViTs) (Dosovitskiy et al., 2021) exhibit massive activations. We note that while ViTs and LLMs both based on self-attention, ViIs employ global token mixing, which contrasts with the autoregressive nature LLMs.
Massive activations in ViTs. We explore several model families based on ViTs: CLIP (Rad- ford et al., 2021), MAE (He et al., 2021) and DINOv2 (Oquab al., 2024). We examine the ViT-L models from these families. The activation magnitudes in the penultimate layer for an input image are illustrated in Figure 12. We find that massive activations exist in CLIP and DINOv2 ViT-L, where we highlight the corresponding sequence dimensions. In these two models, there extremely few activations (fewer than four) with significantly larger magnitudes than others. In addition, these activations are located in specific feature dimensions and appear in random patch tokens. However, we do not observe massive activations in MAE ViT-L. In this model, a feature dimension (927) exhibits uniformly large values across all tokens.
CLIP VIT-L
DINOV2 ViT-L
MAE VIT-L
200
400
100
200
50
353
Patch Tokeng
Patch Tokens
Paich Tokeng
Figure 12: Massive activations are present in ViT-L from CLIP and DINOv2, but not MAE.
Massive activations are biases in ViTs. Figure 14 shows the three largest activation magni- tudes and the median per layer CLIP and DINOv2 ViT-L, averaged over 1k images. We find that massive activations are consistently present across images and their values remain largely the same around the mean values. It is worth noting that unlike LLMs, massive activations start to appear only in the later stages of ViTs.
CLIP ViT-L
DINOv2 ViT-L
200
400
Magnitudes
100
200
6
12 Layers
18
24
12
18
24
Layers
Top
Top 2
Top 3
Median
Figure 14: Three largest activation magnitudes and the median magnitude at each layer in CLIP and DINOv2 ViT-L.
CLIP ViT-L, layer 13
Intervention ImageNet acc (%)
Original
75.5
Set to zero Set to mean
59.8
75.5
Table Intervention analysis of massive activations in CLIP ViT-L.
Following Section 3, we perform intervention analysis on CLIP VIT-L. We modify the two largest massive activations to zero and mean values respectively. The intervention is
15

---

Published as a conference paper at COLM 2024
conducted on layer 13, where massive activations first appear within this model. Results shown in Table 4, where we evaluate the zero-shot accuracy on ImageNet. We can see that setting massive activations to zero leads to significant drop in accuracy while setting to their means results negligible accuracy drop. These results indicate that massive activations function as fixed but crucial biases in ViTs, aligned with our observations in Section 3.
Registers are biases in ViTs. Recently Darcet et al. (2023) propose to augment standard ViTs with additional learn- able tokens, which they name register tokens. They show that training ViTs with register tokens leads to smooth attention maps, and the resulting model family, namely DINOv2-reg, achieves superior downstream per- formance over DINOv2. Examining the largest ViT-G model in DINOv2-reg, sive activations, as shown in Figure 15. However, different from standard ViTs, massive activations do not appear in patch tokens but exclusively within a fixed register token, iee, register 3. This suggests that this model uses register to store these activations. Figure visualizes the attention distribution of the [CLS] token in the last layer. We find that most of the attention is allocated register 3, echoing our findings in Section 4.1.
1k
observe the existence of mas-
999
reg
reg
reg
reg
Patch Tokens
179
Figure 15: DINOv2-reg VIT-G.
Attention Distribution, DINOv2-reg VIT-G
60%
Probability
DINOv2-reg with 4 registers
40%
ImageNet acc (%)
VIT-S
VIT-B
VIT-L
VIT-G
20%
Original
81.9
84.8
86.3
87.0
Fix-Reg-Meat
81.7
85.0
86.2
87.0
CLS
Reg
Reg
Reg
Reg 4
Patch
Tokens
Table 6: We fix all register features at every layer their mean values and evaluate the intervened ViTs.
Figure 16: Attention probability of the [CLS]
token, averaged over 1k ImageNet images.
Further, we conduct intervention analysis to analyze the role of registers. We replace all register features at the output of every layer with their means, averaged over 10k ImageNet training images. This intervention removes the intended purpose of registers to aggregate global input information (Darcet al., 2023). Table shows the results. We find that ViTs with fixed register features achieve accuracy comparable to original models, suggesting that registers act learned biases in ViTs. This leads to constant key and value states at register tokens, effectively introducing bias terms to self-attention (extra k' and v' in Equation 3). Thus ViT with register tokens function equivalently to a standard ViT augmented with explicit attention biases.
In Masked Autoencoders (MAEs) (He et al., 2021), dummy token added ViTs during pretraining. In one fine-tuning pipeline of MAEs, fine-tuning is done based on the average pooled features of all patch tokens. In these MAE models, this dummy token is equivalent to a register token. Here we maintain the register token features as constant across the output features of all layers in ViTs, which we denote as Fix-Reg-Mean. These fixed values are computed the average register features over 10k ImageNet training images. Table shows the results: setting register features to fixed values does not affect model performance.
MAE with register
ImageNet acc
ViT-B
ViT-L
ViT-H
Original
82.6
85.5
86.7
Fix-Reg-Mean
82.6
85.5
86.7
Table 7: Registers are biases in Masked Autoencoders (MAEs).
To summarize our findings:
Massive activations exist in many but not all ViTs. Similar to those in LLMs, these activations act as constant biases. We also show the recently proposed register tokens have similar function.
16

---

Published as a conference paper at COLM 2024
B Additional Results on Massive Activations in LLMs
In this section, we supplement the main paper with additional results of massive activations in LLMs. This includes results more pretrained LLMs (Appendix B.1) and fine-tuned LLMs (Appendix B.2), analysis the BOS token 
 (Appendix B.3) and layer-level analysis (Appendix B.4).
B.1 Pretrained LLMs
In Section 2, we have demonstrated massive activations in LLaMA2-7B, LLaMA2-13B and Mixtral-8x7B In this section, evaluate more pretrained LLMs which cover wide range of model families. We illustrate massive activations in LLaMA2-70B, Phi-2, Mistral-7B (Jiang et al., 2023), MPTT-B (MosaicMI, 2023) and Falcon-7B (Almazrouei et al., 2023). The results are presented in Figure 17, 18, 19, 20 and 21.
We make several observations, First, massive activations are consistently present in these models and they exhibit similar characteristics those described in Section 2. Intriguingly, for LLaMA2-70B, we find that massive activations are found within tokens representing numerical values, e.g., token "ø" and token "2", as depicted in Figure 17. However, they do not appear in numerical tokens (see the rightmost example in Figure 17). Another interesting finding is that the feature dimension of massive activations in both Mistral-7B (Figure 19) and Mixtral-8x7B (Figure 3) identical (i.e., 2070), implying that the latter model may have been fine-tuned from the former.
LLaMA2-70B
15k
19k
15k
1.5k
10k
10k
10k
5k
5K
500
7754
1754
7754
7754
Copy
3758
He
bor
3758
Hel
homm on
3758
want
3758
March
March
March
Figure 17: Massive activations in LLaMA2-70B.
Phi-2
1K
1k
500
500
500
500
Summer
Wintor
253
743
Summer
warm
Winter
why' aummer
and waterr
cold
293
010
twol
253
Figure 18: Massive activations in Phi-2.
Mistral-7B
300
300
300
300
150
150
160
150
Summer
W
2070
Surmmet
werm
Winter
2070
why
cummort war! and winter cold.
0
2070
the boxt
timo
2070
Figure 19: Massive activations in Mistral-7B.
B.2 Fine-tuned LLMs
Our results so far are focused on pretrained LLMs. However, a significant application of LLMs lies in their use for chat purposes. Instruction fine-tuning (Ouyang et al., 2022) is essential for developing models capable of generating coherent responses to questions. In
17

---

Published as a conference paper at COLM 2024
MPT-7B
1K
1k
500
500
500
500
Summen
Summer
win
Winnng cutr
Wendd
Figure 20: Massive activations in MPT-7B.
Falcon-7B
Summay
warm
Winter
Summar
wami
Westor
why
cummert want arde conten cold
8h0
bodl
cold
Figure 21: Massive activations in Falcon-7B.
this part, we demonstrate massive activations in these fine-tuned models. We evaluate fine-tuned models from models in LLaMA2 and Mistral. The results are shown in Figure 22, 23, 24 and 25.
We can see that massive activations persist after instruction fine-tuning, Moreover, the values and positions of massive activations remain largely the same as the original pretrained LLMs. For LLaMA2-7B, this can be seen by comparing Figure 22 and Figure 1. However, one exception is Mixtral-8x7B. We find that massive activations disappear from the newline token "\n" after fine-tuning, as shown by comparing Figure 25 and Figure 3. We leave the study on how instruction fine-tuning affects massive activations for future work.
LLaMA2-7B-Chat
2k
2k
2K
1k
14
1K
Summer
wam
Wanter
1415
Summmer
wom
Wintrr
cold
50
1415
why
auromer warm
nonter cold
1415
bear
(mmm
1415
Figure 22: Massive activations in LLaMA2-7B-Chat.
LLaMA2-13B-Chat
1
1
1k
4743
4743
4743
Summer
wamm
Winter
2100
Summer
womm
Whter
2100
wiYh
summer maam ant winter cold
2100
beaf
timed
2100
50
Figure 23: Massive activations in LLaMA2-13B-Chat.
B.3 BOS Token 

In some tokenizers, e.g., LLaMA2, the BOS token 
, also known as the beginning of sequence token, can be prepended to the input sequence. For the experiments presented in other parts of the paper, we turn off this option, where all sequences do not start with the BOS token.
18

---

Published as a conference paper at COLM 2024
Mistral-7B-Instruct
300
300
300
300
150
150
150
Bummang
womm
Wintr
2070
Summner
werm
Winter
2070
why
summirt wam!
and winter
cotri
2070
the
baet
times
2070
Figure 24: Massive activations in Mistral-7B-Instruct Mixtral--x7B--nstrucc
4k
4k
Summan
colo
2070
Summer
Winter
cold
2070
wing
aummert worn and winter cold.
2070
ere best
times
2070
Figure 25: Massive activations in Mixtral--x7B-Instrree
In Figure 26, 27 and 28, we show massive activations LLaMA2-7B, LLaMA2-13B and Mixtral-8x7B, with the same input sequences in Section 2. We find that massive activations persist with prepended BOS token. In LLaMA2-7B and LLaMA2213BB the locations of massive activations, i.e., sequence and feature dimensions, are not altered. However, for Mixtral-8x7B, some massive activations shift the BOS token 
, We leave the study on how the BOS token 
 affects the positions of massive activations for future work.
LLaMA2-7B
2k
2k
1K
1K
Summer
warm
Winer
cold
1415
Summingg
warm
Winte
1415
Why wore and waterr cold!
summett
1415
the
of limes
1415
in
Figure 26: Massive activations in LLaMA2-7B when the input is prepended with a BOS
token 
,
LLaMA2-13B
1k
1k
1k
4743
4743
4743
4743
Summer
wam
Winter
cold
2100
Summer
warn Winter rold
2100
WmY
Bummart warm
seritrr
2100
bod!
timed
2100
50
in
Figure 27: Massive activations in LLaMA2-13B when the input sequence is prepended with BOS token 

B.4 Laver-Level Analysis
In Section 2.1, we have presented the layer-level analysis results for LLaMA2-7B, LLaMA2- 13B and Phi-2. In Figure 29, we provide the comprehensive results for all LLMS examined in this paper (listed in Table 8). This includes LLMs from LLaMA2, Mistral, MPT, Falcon, OPT and GPT-2 model families. For each model, we show the three largest activation magnitudes well the median at each layer.
19

---

Published as a conference paper at COLM 2024
Mixtral-6x7B
5k
5k
5K
8k
Summer
warm
Wortr
2070
Summer
wamt
Winle
2070
Why
summer wort winter cold
2070
times
2070
Figure 28: Massive activations in Mixtral-8x7B when the input sequence is prepended with a BOS token 

LLaMA2-7B-chat
LLaMA2-13B-chat
LLaMA2-70B-chat
16k
Magritudes
1k
10k-
10k
5k
0
Layers Mistral-7B
Layers
Layers
Layers
Mixtral-8x7B
Misssra-7B-Instruet
Mixtral-ex7B-Insune BBBnstructt
Magnitates
8k
200
2x
04
01
16
24
32
16
24
32
16
24
32
24
32
Layers
Layers
Layers
Layers
MPT-7B
MPT-30B
Falcon-7B
Falcon-40B
Magnitides
1k
15k
1K
10K
500
5k
16
24
32
12
24
36
46
16
24
32
15
30
45
60
Layers
Layers
Layers
Layers
OPT-78
OPT OPT-13B
OPT-30B
OP OPT-66B
1K
Magnituutes
400
200
200
200
0.
0.
16
24
32
10
20
30
40
12
24
36
48
16
32
48
64
Layers
Layers
Layers
Layers GPT-2-XL
GPT-2
GPT-2-Medium
GPT-2-Large
4k1
Magniludess
2k-
2k
2k-
Ik-
1K
12
18
12
24
48
Layers
Layers
Layers
Layers
Figure 29: Layer-level analysis of LLMs. For each model, we show the three largest activation magnitudes as well as the median per layer.
We can see that the trend of massive activations we observe in Section 2.1 holds true for LLMs in general. Massive activations tend remain constant in most of the intermediate layers. They emerge in the early layers and disappear in the last layer.
20

---

Published as a conference paper at COLM 2024
C Additional Results on Self-Attention
In this section, we provide additional results for the analysis on self-attention. This includes results more LLMs (Appendix C.1), analysis of attention LayerNorm (Appendix C.2), more results on implicit attention biases (Appendix C.3) and detailed results on training GPT-2 with explicit attention biases (Appendix C.4).
C.1 Attention Concentration on Massive Activations
In Section 4, we have demonstrated the attention concentration pattern in LLaMA2-7B, LLaMA2-13B and Phi-2. We now illustrate this phenomenon for more LLMs. Figure 30 and Figure 31 show the results for LLaMA2-70B and Mistral-7B. For these two models, massive activations are formed in the output feature of layer 9 and layer 2 respectively.
We can see that attention is predominantly focused on the sequence dimensions of massive activations. In the case of LLaMMA-70B,, as depicted in Figure 30, massive activations are found the starting word token and also token These two tokens receive substantial attention logits. Additionally, we visualize the attention probability in Figure 32. The attention softmax is computed along each row, thus resulting in these special tokens being allocated a much higher attention probability.
LLaMA2-70B, Layer
LLaMA2-70B, Layer 10
LLaMA2-70B, Layer 40
LLaMA2-70B, Layer
10k
8K
-2
-2
-4
-3
-4
This
3758
This 2
This
This
2
Figure 30: Average attention logits over all heads in layers 10, 40 and 60 of LLaMA2-70B. The input sequence is "This book, including all illustrations and text, is protected under Copyright@2024 and may not be reproduced transmitted in any form without the prior written permission of the copyright owner.".
Mistral-7B, Layer 2
Mistral-7B, Layer
Mistral-7B, Layer 10
Mistral-7B, Layer 20
300
150
00
2
Willom
from:
In
-6
William
from
William
William
from
Figure 31: Average attention logits over all heads in layers 3, 10 and 20 of Mistral-7B. The input sequence is "William Shakespeare was a famous writer from England who wrote plays and poems. He is considered one of the best writers ever.' His works include famous plays like 'Romeo and Juliet' and "Hamlet'".
21

---

Published as a conference paper at COLM 2024
Layer
1.0
Layer
1.0
Layer 10
1.0
Layer
10
0.6
0.8
0.8
08
Canry
3.6
-0.
Quarry
10.
Quarr
06
-0.4
0.4
04
-0.2
-0.
02
Key
Key
Key
Key
(a) LLaMA2-7B
Layer
1.0
Layer
1.0
Layer 10
1.0
Layer
0.0
00
Canery
0.6
Quay
05
Key
Key
Key
(b) Mistral-7B
Layer
Layer
-10
Layer
Layer
Query
Qudfy
QJMf
-.4
04
-02
-02
0.2
02
Key
(c) Phi-2
Figure 32: Average attention probability over all heads in intermediate layers of LLaMA2-7B, LLaMA2-13B and Phi-2. The input prompt is "William Shakespeare was a famous writer from England who wrote plays and poems. He is considered one of the best writers ever. His works include famous plays like 'Romeo and Juliet' and 'Hamlet.".
C.2 Attention LayerNorm
Our analysis in Section 4.2 indicates that tokens associated with massive activations have drastically different key and value states. In this part, we investigate how attention layer- norm plays crucial role in this process.
Preliminaries. There are two specific types of layer normalization commonly used LLMs. One is the standard layer normalization (Ba et al., 2016). Suppose we have feature vector Rd, LayerNorm will normalize this feature to fix the mean and variance and then re-scale with element-wise affine transformation:
X:
8i byr
where μ
d Σ Xi, 1=1
1
d Σ
(x 11)2
0
where g, Rd are parameters of the affine transform, also called the gain and bias.
In addition to the original LayerNorm, variant of layer normalization has also been used in ILaMA2 and Mistral models. Specifically, Root Mean Square Normalization (RM- SNorm) (Zhang& & Sennrich, 2019) normalizes the feature IR with the root mean square (RMS) statistic.
d
X;
gi, where RMS(x)
(5)
RMS a)
where g E Rdd is the gain parameter.
22

---

Published as a conference paper at COLM 2024
LLaMA2-7B, RMSNorm Input
LLaMA2-7B, RMSNorm Middle
LLaMA2-7B, RMSNorm Output
2000
40
1000
0
1000
-20
Who
Who
Who
are
are
you
you
you
Phi-2, LayerNorm Input
Phi-2, LayerNorm Middle
Phi-2, LayerNorm Output
0
500
20
-2
30
4
1000
40
Who
Who
Who
are
you
you
you
Figure 33: Activation trajectory in the attention LayerNorm of LLaMA2-7B and Phi-2, where the LayerNorm input contains massive activations. Note that LLaMA2-77 uses variant of layer normalization: RMSNorm (Zhang & Sennrich, 2019) and Phi-2 uses the default LayerNorm (Ba et al., 2016).
For both LayerNorm and RMSNorm, when there are a few activations in x E R2 that have significantly large magnitudes, the denominator in the normalization step, i.e., o in Equation 4 and RMS(() x) in Equation 5, becomes large as a result. In fact, the denominator is almost determined by these few massive activations. The large denominator will push all normal values to zero while preserving the outlier nature of massive activations. This will effectively create a drastically different normalized feature, determined by the few massive activations. Figure 33 shows two activation trajectory in both RMSNorm and LayerNorm. We can see that how the normalization step (middle) preserves the outlier activations in tokens Who and |n and normalized features at these two tokens become extremely similar.
C.3 Implicit Attention Biases
In Section 4.2, have shown how the value updates from the tokens associated with massive activations tend to be largely identical. Here we extend those findings by examining additional input prompts and layers within the LLaMA2-7B model. We use four input prompts: " "Are you cold? Grab jacket.", "Will it snow? n Check the forecast.", "Did she call?',n missed it." and ""I am doing well. Thank you for asking." "" We visualize the value updates in layer layer 15 and layer 30 in Figure 34, Figure 35 and Figure 36 respectively. We focus on the latter half of the input sequence, following the two tokens associated with massive activations. We can see that in the same layer, the value updates Lisc pfv, display remarkable similarity across the different input sequences.
C.4 Explicit Attention Biases
Experimental setup. We use the open-source reproduction of GPT-2 from the NanoGPT repository (Karpathy, 2023). We use the default recommended training setup and optimizer setting. For each of the three GPT-2 models, we train for 50,000 iterations, with a total of approximately 2B tokens. For the GPT-2 with sink token, we follow Xiao et al. (2023b), where we prepend each training sequence with learnable sink token [SINK]. When computing the training loss, we do not include the cross-entropy loss computed on the prepended sink token. For GPT-2 with explicit attention biases, we initialize each k' and v with (0, 0.00..
23

---

Published as a conference paper at COLM 2024
Seq
Seq 2
Seq
Seq 4
0.
00
000
100
0.0
Gra
Check
Thooo
Figure 34: Value updates Eiec pfo; at layer LLaMA2-7B, with four input sequences.
Seq
Seq
Seq 3
Seq
0.5
05
0.5
0.5
0.5
0.5
0.5
Go
Oreek
Thenk
Figure 35: Value updates Liec vf at layer 15 of LLaMA2-7B, with four input sequences.
Seq
Seq
Seq 3
Seq
0.0
-05
-05
-0.5
-05
Clock
Thank
Figure 36: Value updates Liec pfu; at layer 30 of LLaMA2-7B, with four input sequences.
Results. Regarding the performance of the three GPT-2 models we evaluate in Section 4.3, we find that after 50,000 training iterations, they have the same perplexity on the validation split constructed from OpenWebTeexe (Gao et al., 2021): 3.04.
We also experiment with other ways of injecting biases in the self-attention computation:
1. The first one is special case of our proposed formulation in Equation 3, where both k' and v' are zero vectors. Equation shows the computation of this variant of self-attention. This is also equivalent to the previous proposed Softmax-off-by- one (Miller, 2023).
e [[[TT 0) Vdg
V O
Attention (aattion((,K,)) Q,K, = softmax
(6)
2. Since Equation 3 can be viewed as inserting a sequence dimension, we also exper- iment with inserting one extra feature dimension. Specifically, add learnalee parameters q' k' E J and concatenate them with the query and key states respec- tively. This variant of self-attention is as follows:
Attention( K, V.q' = softmax (10
K k')
V
Vds
3.
We also experiment with simple way to enforce constant value updates by injecting
an extra value parameter v' E IR This variant of self-attention as follows:
QKT
Attention (ettiion((4 (,K, V;v' softmax
V +
(8)
Vdg
24

---

Published as a conference paper at COLM 2024
Figure 37 visualizes the ten largest activation magnitudes in three GPT-2 models, corre- sponding to the three formulations of biases in Equation 6, 7 and We find that these alternatives are not able to eliminate massive activations during pretraining.
GPT-2 with coftmax-off-by-om
GPT-2 with QK Bias
GPT-2 with V Bias
Magnitudes
300
1000
600
200
400-
500
100
200
0
12
Layers
Layers
Layers
Top
Top
Top
Top 7
Top
Top 2
Top
Top
Top 8
Top 10
Figure 37: Ten largest activation magnitudes at each layer in three GPT-2 models.
D Models and Datasets
Table lists the information of the LLM models used in this paper.
Model family
Model name
Layers Dimensions Heads
Huggingface modell id
LLaMA2-7B
32
4096
32
meta=1lamm/LLaanee227bhh
LLaMA2-13B
40
5120
40
meta-1llame/LLmmo---1311-1
LLaMA2-70B
6656
meta-1Loam//Lomm-2220000-f
ILaMA2
LLaMA2-7l-Chaat
32
4096
32
meteslllamm/Laaaaa-b>>chhhhh
LLaMA2-13B-Chat
40
5120
40
neta==1lama/LLimmm222-11-------
LLaMA2-70B-Chat
mela-laam//Llea--27770hhhhhhhh
Mistral-7B
32
4096
32
mistralaa/Misttraa-88-01111
Mistral-8x7B
32
4096
32
mistralaajMMittrr3-447----111
Mistral
Mistral-7B-Inste
miserrala//Msstrr///---mntttttto
Mistral- Pestral-Sx7B-aee 8x7B-x//BBBB Instruttt
32
4096
32
Phi
Phi-2
32
2560
32
microsoft/phi-2
MP1-7B
4096
mosaicml/mpt-7b
MPT
MPT-30B
48
7168
64
mosaicml/mpt-330
Falcon-7B
4544
tiiuae/falcon-77
Falcon
Falcon-40B
60
8192
128
tiiuae/falcoon400
OPT-7B
32
4096
32
facebook/opt-66.7
OPT-13B
facebook/oott-13b
OPT
OPT-30B
48
7168
facebook/opt-300
OPT-66B
64
9216
72
facebook/opt-66b
GPT-2
gpt2
GPT-2-Medium
24
1024
16
gpt2-medium
GPT-2
GPT-22Large
36
1280
20
gpt2-large
GPT-2-XL
gpt2-xl
Table 8: Relevant information of LLM models we experimented with in this work.
We list the datasets used in this work and relevant license information:
RedPaaama (Together Computer, 2023):
Apache License, Version 2.0
OpenWebText2 (Gao al., 2021):
MIT License
C4 (Raffel et al., 2020):
Open Data Commons Attribution
License 1.0 license
PG-19 (Rae et al., 2019):
Apache License, Version 2.0
WikiText (Merity et al., 2016): cense
Creative Commons BY-SA 3.0 li-
MMLU (Hendrycks et al., 2021):
MIT License
25

---

Published as a conference paper at COLM 2024
BoolQ (Clark et al., 2019a):
Creative Commons BY-SA 3.0 li-
cense
PIQA (Bisk et al., 2019):
The license status is unclear
WinoGrande (Sakaguchi et al., 2019):
Apache License, Version 2.0 Creative Commons BY 4.0 license
ARC easy and challenge (Clark et al., 2018):
ImageNet (Deng et al., 2009):
The license status is unclear
26


Reviews:
[{"review_id": "jFAeavNdpF", "reviewer": "Reviewer_sdg5", "summary": "This paper identifies the phenomenon of massive activations in large language models (LLMs), where very few activations exhibit significantly larger values than others. It extensively studies the layers where massive activations reside, their importance to LLMs' capabilities, and their role as bias in the self-attention mechanism. Finally, the analysis is further extended to ViTs.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment of our paper and the constructive comments. We are happy to address your concerns.\n\n- **Massive activations seem to serve as an important bias in the attention mechanism, while they mainly focus on semantically unimportant tokens like delimiters. The authors are expected to provide a clearer explanation regarding this gap**\n\n\nIn the last part of section 3, we have discussed why massive activations appear in semantically unimportant tokens. Suppose the LLM model wants to store these important biases somewhere, these semantically unimportant tokens would be better than other tokens. This is because repurposing the hidden states of these tokens to store such biases would incur less loss of input information. \n\n- **The authors are expected to evaluate larger LLMs pretrained on larger corpora to determine whether the percentage of massive activations will increase with the model scale and data scale.**\n\nRegarding “model scale”, in the appendix, we have evaluated the largest LLaMA2-70B model (see Figure 17 in Appendix B.1). We find that compared to the smaller LLaMA2-7B and LLaMA2-13B model, the magnitudes of massive activations are drastically larger in LLaMA2-70B, e.g., 15000. We also have results on the Mistral-7B model (see Figure 19 in Appendix B.1). Compared to the larger Mistral-8x7B model (Figure 3), the scale of massive activations also increases with model sizes.  \n\nTo answer the question regarding the relationship between massive activations and “data scale”, we have studied how massive activaitons changes during the pretraining phase. Specifically, we attempt to address the question: does more pretraining lead to changes of massive activations? We observe that this is not the case. We conduct our analysis on GPT-2. We observe that massive activaitons grow rapidly in the initial part of the pretraining phase and stay at a constant level after the initial growing phase. \n\nIn the updated version of the paper, we will add these results and discussion on the related work [1].\n\n\nIf you have other questions, we are happy to answer.  \n\n[1] Fluctuation-based Adaptive Structured Pruning for Large Language Models, An et al, AAAI’24.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The identified existence of massive activations is interesting. Although related work like FLAP [1] has provided similar observations, a comprehensive study of this phenomenon could provide useful insights for the community.\n\n[1] \"Fluctuation-based Adaptive Structured Pruning for Large Language Models\", Y. An et al., AAAI'24.\n\n2. The performed analysis regarding the importance and role of massive activations is technically sound and extensive.\n\n3. The writing is logically clear and coherent.", "reasons_to_reject": "1. Some of the observations lack deeper analysis. For example, massive activations seem to serve as an important bias in the attention mechanism, while they mainly focus on semantically unimportant tokens like delimiters. The authors are expected to provide a clearer explanation regarding this gap.\n\n2. It is not clear whether massive activations are desirable or whether they will disappear with larger-scale pretraining. The authors are expected to evaluate larger LLMs pretrained on larger corpora to determine whether the percentage of massive activations will increase with the model scale and data scale.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HKM7x0BQd3", "reviewer": "Reviewer_sAPY", "summary": "This analysis paper studies the phenomenon of massive activations, which is defined in this paper as activations with magnitude larger than 100 and 1,000 times larger than the median magnitude of its hidden state. The paper had the following main findings:\n\n- Massive activations widely exists in different models, largely in the middle layers, and occurs on fixed dimensions of a model and often on frequent tokens\n- Setting massive activation to 0 significantly changes model behaviors.\n- Massive activation causes attention to be concentrated, and act as bias terms in attention modules. Training models with a bias term in the attention module can eliminate massive activations.\n\nIn summary, I think the general presentation quality is very high, forming a very well-motivated and coherent story that's mostly easy to follow. I do have some doubts about the impact of this work, because I fail to see how such understanding can help us make improvements to the current transformer architecture. I'm hoping the authors to shed more lights on those doubts.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment and constructive comments. We are happy to address your concerns. \n\n- **Practical implications**  \nThere could be several practical implications. First, removing massive activations would make the activations easier to quantize. Second, dedicating the learnable k’ and v’ vectors could make the attention pattern more interpretable, i.e., avoiding allocating excessive attention to input tokens. Further, we believe that studying the effect of our attention formulation on large-scale pretraining would be an important future direction.\n\n- **Analysis stopped at 13B models**  \nIn our paper, we have analysis on LLMs with sizes larger than 13B. The Mixtral-8x7B model in Figure 3 has 47B parameters. In the appendix, we have results on LLaMA2-70B. We find that massive activations consistently appear across model sizes.\n\n- **Encoder-decoder models**  \nFollowing Table 1, we show the top activation magnitudes for two T5 models at one layer. We find that massive activations can be found in encoder-decoder LLMs. \n|Model| top 1 | top 2 | top 3 | top 4 | top 5 | top 10 | top 100 | top 1% | top 10% | median | \n|---|---|---|---|---|---|---|---|---|---|---|\n|T5-base| **1.3e4**|**1.1e4**|**1.1e4**|**8.6e3**|**5.2e3**|1.6e3|427|270|84|31|\n|T5-large| **1.9e4**|**8.3e3**|**7.8e3**|**7.7e3**|**7.6e3**|1.2e3|569|375|71|21| \n\n- **Setting massive activations to the mean of the hidden states**  \nWe conduct this intervention analysis. We find that similar to setting to zero, it leads to significant drop in performance. We observe that the mean values of the hidden states are still very small values as massive activations are extremely few in quantity. \n\n- **Layer-norm and output decomposition**  \nThese are two consecutive steps in the attention layer. The part on layernorm is to analyze the effect of massive activations on computing QKV states. Then attention is computed with QKV states as inputs. In the paper, the output decomposition is on the second step.\n\n- **Post-layernorm architecture**  \nSince first introduced in GPT-2, pre-layernorm has become the common practice for decoder-only LLMs. We additionally trained GPT-2 with post-layernorm. We observe optimization difficulty, where the model diverges suddenly at one point during pretraining, with the training setup of default GPT-2. We are looking into this issue and will add discussion on this in the new revision.\n\nIf you have other questions, we are happy to answer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a novel, interesting, and thorough analysis that advances the communities understanding of massive activation phenomenon in the transformer architecture.\n- The presentation quality is very high. The clarity of the experimental setting is good. The transitions in between different parts of the analysis are well motivated. The writing is mostly easy to follow.", "reasons_to_reject": "- Although the authors provided great insight into reasons why massive activations occur in transformers, I have some doubts on what steps the community can take on those findings. Some discussions around that theme would be very helpful -- for example, should people start to add a bias term on the attention modules? Should people be more careful with normalization in the attention module? What improvements should people expect from them? Is it better performance or just better training stability? Answering these questions can potentially help amplify the impact of the paper.\n- Analysis stopped at 13B models and decoder-only models. I would be curious to see if the phenomenon generalizes to encoder-decoder models, or even larger models, e.g. 175B BLOOM models.", "questions_to_authors": "- For \"setting massive activation to a certain value\" experiment in Section 3, did you try to set it to a non-zero regular magnitude value (e.g. mean of the hidden state)? I wonder if 0 has some kind mathematical specificity here that might have caused the significant change.\n- For Section 4.2, I'm not quite sure if I'm following the transition between layer-norm to output decomposition -- they feel somewhat unrelated to each other. Can you explain how one affect another? Also, have you tried to run experiments with the attention module in the original transformer paper? I believe there is no layer-norm before the linear transformation.", "extra_scores": {}}, {"review_id": "m8iHKaZe8N", "reviewer": "Reviewer_FDqV", "summary": "This paper introduces an outrageously large value sometimes appears in a hidden vector in a large language model. This paper calls such a large value \"massive activation\". The authors conducted several analyses on the massive activations. Then, the authors show situation which the massive activation appears in, and the relation between attention calculation and the massive activation.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and the constructive comments. We are happy to address your concerns. \n\n- **This paper does not reveal the reason of the massive activation and the effect on LLM behaviors.**  \n\nRegarding the reason of the massive activations, in Section 4, we have showed that massive activations are fundamentally connected to the self-attention mechanism. Specifically, LLMs use these activations to allocate significant at their tokens (Section 4.1) and further form an implicit bias term in the attention output (Section 4.2). With respect to the effect on LLM behaviors, in Section 3, we have showed that massive activations play an important role on the output behaviors of LLMs.  \n\nWe acknowledge that our analysis in this paper is largely focused on the pretrained LLMs, where the model has already learned these massive activations during pretraining. We believe that studying the aspects of massive activations regarding pretraining would be interesting future direction, as we have showed that they can be found across LLM model families (Section 2). We will be including discussion on this direction in the updated version. \n\n\n- **bias term suggested by the PaLM paper [1]**  \n\nWe would like to clarify that the bias term that PaLM paper refers to is a different concept of the attention bias formulation in our paper. Specifically, PaLM paper states that “No biases were used in any of the dense kernels or layer norms. We found this to result in increased training stability for large models.”. This refers specifically to the additive bias terms in the linear layers. In the related work part of our submission, i.e., “biases in self-attention”, we have distinguished this notion of bias to our formulation of attention bias. We will be making this distinction clear in the early parts of our paper.\n\nIf you have other questions, we are happy to answer.  \n\n[1] PaLM: Scaling Language Modeling with Pathways. Chowdhery et al, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The massive activations, that are introduced by this paper, are interesting phenomena. The analyses on massive activations might have an influence on future studies on LLM behaviors and training dynamics of LLMs.", "reasons_to_reject": "This paper introduces an interesting phenomenon, the massive activation, but I think this is an ongoing study because this paper does not reveal the reason of the massive activation and the effect on LLM behaviors. For example, the relation on the stabilization of LLMs. Section 4 of this paper implies that the massive activations have a role such as a bias term. PaLM paper (https://arxiv.org/abs/2204.02311) clams that it important to remove bias terms to stabilize the pre-training of LLMs. These suggest that massive activations might have a serious influence on the stabilization of LLMs. Like this example on the stabilization, the authors should conduct more investigations to reveal why massive activations appear in.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QeSXTQeG3a", "reviewer": "Reviewer_aoML", "summary": "This paper identifies a new phenomenon called “massive activations” in Large Language Models (LLMs), which shows that the hidden state vectors in intermediate layers at certain token locations have a few exceedingly large coordinates. The finding extends the attention sink phenomenon of Xiao et al. (2023) in the sense that these tokens are precisely the tokens that attract attention scores. The paper reports several qualitative properties such as their appearance and their difference in different LLMs. The paper performs intervention studies and further some small-scale pretraining experiments to demonstrate that massive tokens act like a fixed bias in the attention mechanism.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive assessment of our paper and the constructive comments. We are happy to address your concerns. \n\n- **I am guessing this special k’v’ token still attracts (say >90%) attention, so this method still preserves attention concentration.**  \n\nWe confirmed that this is indeed the case for the GPT-2 model we trained with explicit attention biases. Our hypothesis is that attention concentration is a necessary pattern that is developed during pretraining. We will add discussion on this result in the revision.\n\n- **Eq (8) and attention concentration**  \n\nIn equation 8, there are no changes in the softmax computation except that we are adding one parametrized value embedding after value aggregation. We find that this formulation does not alter the attention concentration patterns as compared to the original GPT-2 model.\n\n- **Other ways of removing massive activations / attention concentration**  \n\nOur results seem to suggest that attention concentration is a crucial pattern developed during pretraining. Thus removing this pattern might impact model performance to a large degree. In addition to explicit attention biases, there can be other ways to remove massive activations. For example, one could apply l1 regularization on the hidden states across all layers, though this extra regularization term could make pretraining trajectory much different from standard pretraining. One advantages of using explicit attention biases is that it induces minimal changes on the Transformer architecture. \n\n- **Training stabilities**  \n\nWe believe this is an important question and our attention bias formulation still needs to be tested at a larger model scale, e.g. 7B or larger. We will discuss this direction in the revision.\n\nIf you have other questions, we are happy to answer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall I think the paper reports a very intriguing and potentially important phenomenon in LLMs. The phenomenon could be of immediate interest to the theoretical understanding / mechanistic interpretability communities. The experiments are also quite comprehensive and provide a useful reference point for practitioners. \n\n* The phenomenon extends the attention sink phenomenon and in my opinion actually gives (at least a partial) explanation of the attention sink phenomenon.\n\n* The finding that massive tokens also appear in *delimiter tokens* (Sec 2.2) is interesting and (kind of) new even to the attention sink literature. Previously people only showed that these “information-less” tokens act as attention sink in encoder LMs or ViTs. This paper shows that delimiter tokens can also attract attention in standard autoregressive decoder LMs. \n\n* The intuition that attention sink acts as fixed attention biases is very sensible and the paper provides some justifications.\n\n* The study is quite comprehensive in both the models (LlaMa 7B & 13B, Mistral 7B, Phi-2) as well as the various qualitative properties.\n\n* The presentation and in particular the visualizations are quite well done.", "reasons_to_reject": "* The pretraining experiment (Sec 4.3) with the k’v’ variation as in Eq (3) is not fully satisfactory in showing that massive tokens act as a fixed bias.  \n  \n  - Eq (3) does not really differ from the “GPT2 with sink token” baseline, as it seems mathematically almost equivalent (maybe up to some minor reparametrization, since Eq(3) did not parametrize a hidden state vector but rather one k’v’ vector for each head). So even though k’v’ method removes massive activation in other tokens as reported, I am guessing this special k’v’ token still attracts (say >90%) attention, so this method still preserves attention concentration.\n  \n  - (An interesting test would be whether k’v’ “correspond” to massive activations or their post-layernorm versions, but here it does not sound possible since you only parametrized k’v’, not the pre-KV projection versions.) \n  \n  - On the other hand, Eq (8) in appendix seems like a more ideal version of fixed attention bias, and addresses the unbalancedness in the softmax as well as the massive activation phenomenon. Unfortunately pretraining with it only works to some extent (it reduces the top activation magnitude from 2K to 600, but the ideal target is like 100-200). Again, I am wondering if the attention scores are still concentrated here (and I am guessing they are)?\n\n  - If Eq (8) cannot be made to work to fully remove massive activation, it then creates the question of whether attention concentration is fundamental (for achieving good LM performance). Have the authors thought about other ways of removing massive activations / attention concentration?\n\n* No immediate practical applications, but I don’t think that’s a major concern given the potential importance of the phenomenon. I also wonder if the authors have thought about alleviating training instabilities by mitigating massive activation and/or attention concentration (kind of related to the above question)?", "questions_to_authors": "See the “weakness” part.", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this work provides an interesting observation, has thorough experiments around it, and is well presented. Some reviewers would like more discussion of practical implications of the presence of massive activations and FDqV feels that the paper does not provide sufficient evidence, especially theoretical analysis, to understand the fundamental reasons behind these patterns. I am inclined to agree that a theoretical analysis or controlled synthetic experiments could help us understand the phenomenon better, but I view that as an opportunity for followup work.
Decision: Accept