Massive Activations in Large Language Models

Authors: mingjie  sun
xinlei  chen
j  zico kolter
zhuang  liu
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-02 22:19:45

Abstract: We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call them massive activations. First, we demonstrate the widespread existence of massive activations across various LLMs and characterize their locations. Second, we find their values largely stay constant regardless of the input, and they function as indispensable bias terms in LLMs. Third, these massive activations lead to the concentration of attention probabilities to their corresponding tokens, and further, implicit bias terms in the self-attention output. Last, we also study massive activations in Vision Transformers.

Main paper: 
 introduction: 
 Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023) have demonstrated remarkable capabilities. The majority of existing studies conducted on these models are focused on their external behaviors, e.g., evaluating their performance on various tasks (Katz et al., 2023; Bubeck et al., 2023) , developing prompts to elicit accurate responses (Wei et al., 2022; Yang et al., 2023) . While these studies are encouraging and highlight the potential of these models, it is also important to gain insights into their internal mechanisms, especially as they are being increasingly integrated into many real-world applications. However, research on the internal workings of these models remains relatively limited.In this work, we discover and study a surprising phenomenon in the internal representations of LLMs. Examining the hidden states in these models, we find that certain activations exhibit huge magnitudes, e.g., more than 4 orders of magnitude larger than the median, and could take on absolute values larger than 15,000 in LLaMA2-70B (Touvron et al., 2023) , despite the presence of normalization layers. These activations are also extremely rare, often numbering fewer than 10 among tens of millions of total activations. Figure 1 illustrates this phenomenon in LLaMA2-7B. As these activations are so much larger in magnitudes compared to others, we name them massive activations. We demonstrate their presence in a wide range of LLMs, spanning different model sizes and families.We explore where massive activations are located in LLMs. Regarding the depth dimension of LLMs, the appearance of massive activations is mostly abrupt: they emerge suddenly after a single layer of computation, and diminish at the last few layers. Further, we find massive activations occur in a small number of feature dimensions that are input agnostic. Many of these activations are found within the starting word token and delimiter tokens. Additionally, we show that massive activations are not the same as outlier features (Dettmers et al., 2022) , a previously known phenomenon in LLMs.We show that massive activations act as fixed but crucial bias terms in LLMs. Here by bias terms, we mean certain internal states of the models that are independent from the inputs, analogous to the bias term b in a linear layer y = Wx + b. First, we show that massive activations play a critical role in LLMs' capabilities. For instance, in LLaMA2-7B, setting merely four massive activations (out of millions of activations) to zero would result in catastrophic collapse in model performance. Further, setting them to their mean values does not hurt the model, suggesting their role is equivalent to simple constant biases. Our analysis reveals that after the initial layers, LLMs repurpose the tokens linked with massive activations to store these important biases.Intriguingly, massive activations are closely connected with self-attention. In particular, we show massive activations cause attention to be attracted to the tokens associated with them. Our findings extend the observations from "attention sinks" (Xiao et al., 2023b) -we demonstrate that LLMs allocate excessive attention to more than just the first token, and provide an in-depth analysis on how such attention concentration patterns arise. Our analysis suggests that LLMs try to learn implicit bias components in self-attention via massive activations, during pretraining. We thus experiment with augmenting self-attention with extra key and value embeddings, explicitly designed as biases. Remarkably, we demonstrate that training with them eliminates the need for LLMs to learn massive activations.Finally, we also observe massive activations in Vision Transformers (ViTs) (see Appendix A). They appear in many of the ViTs we have examined. In these ViTs, they tend to appear at fixed feature dimensions, but notably at varying patch tokens. Moreover, we find that these activations act similarly as fixed biases. Notably, we relate these findings to the recently proposed "register tokens" (Darcet et al., 2023) . We show that they both learn values independent of input images. This offers an alternative interpretation for register tokens, where they were originally hypothesized to aggregate global image information. 
massive activations: 
 We study autoregressive Transformers, which are built by a stack of L decoding layers. Each layer ℓ takes the previous hidden state h ℓ-1 ∈ R T×d as input and outputs a hidden state h ℓ ∈ R T×d . T is the number of tokens and d is the number of features. Transformer layers use residual connections (He et al., 2016) , and the computation can be formulated as:EQUATIONwhere F ℓ is the residual transformation. Note that this includes both attention and MLP blocks. An activation denotes a specific scalar value in a hidden state. Unless otherwise specified, our study of activations is on the hidden state h ℓ , i.e., the output of residual summations, not any intermediate states inside F ℓ .Existence in LLMs. We start with an illustrative example on LLaMA2-7B. In Figure 1 , we visualize the intermediate features h ℓ of interest. We feed this model with short sentences and visualize the activation magnitudes (z-axis) of the hidden states at a middle layer. x and y axes are sequence and feature dimensions respectively. Each blue row corresponds to the feature embedding of one token. We observe up to four activations with significantly large magnitudes. The largest activation (about 2,000) is approximately 10,000 times larger than the median magnitude (about 0.2). The sheer scale of these activations makes them stand out from others. We thus refer to these special activations as massive activations.Massive activations are not unique to this specific model LLaMA2-7B, but are widely observed in LLMs. In Figure 2 and Figure 3 , we demonstrate the existence of massive activations in both LLaMA2-13B and Mixtral-8x7B (Jiang et al., 2024) . Notably for Mixtral-8x7B, the largest activation magnitude can reach an absolute value of 7,000, around 4 orders of magnitude larger than the median feature magnitude (around 0.3). We refer the reader to Appendix B for results on more pretrained and fine-tuned LLMs.Figure 2 : Massive activations in LLaMA2-13B. In this model, they appear in two fixed feature dimensions (2100, 4743), and are limited to the starting token.Figure 3 : Massive activations in Mixtral-8x7B. In this model, they lie in two feature dimensions (2070, 3398) , and are found within the starting token, delimiter tokens and certain word tokens ("and" and "of").Properties. We summarize two main properties of massive activations. The most notable property is that these activations possess massive values and their magnitudes are significantly larger than other activations, often several orders of magnitude larger than the median value. Another property is that they are exceptionally few in number. For LLaMA2-7B in Figure 1 , there are approximately 40,000 total activations in each presented hidden state but at most four massive activations can be identified.Quantitatively, we present the values of the top activation magnitudes in Table 1 . We also provide a loose but broad definition: an activation qualifies as a massive activation if its magnitude surpasses 100 and is at least or around 1,000 times larger than the median magnitude of its hidden state. We find this criterion to effectively identify these activations of interest across various LLMs, which are emphasized in bold in Table 1 . Next, we identify the locations of massive activations within LLMs. For a comprehensive analysis, rather than using short sentences as inputs, we collect 100 sequences (each with 4,096 tokens) from RedPajama (Together Computer, 2023) . We run LLMs on these 100 sequences and collect the hidden states from each layer.We determine the layers whose output hidden states exhibit massive activations. In Figure 4 , we visualize the three largest activation magnitudes and the median of the hidden state output of each layer, with results averaged over 100 sequences. We examine three models: LLaMA2-7B, 13B and Phi-2 (Javaheripi et al., 2023) (see Appendix B.4 for more LLMs). In all cases, each of the top three activations comes from the same position in the hidden state across most of the middle layers. Generally, we observe the following: Massive activations exist and remain as largely constant values throughout most of the intermediate layers. They emerge in the initial layers and start to diminish in the last few layers.In LLaMA2-7B, massive activations first appear in layer 2 and remain nearly constant values until layer 30. Intriguingly, for LLaMA2-7B and 13B, massive activations emerge very rapidly from one layer of computation, e.g., layer 2 and layer 4 respectively. This means that they do not emerge as a result of gradual accumulation through many layers, and are caused by a rather different mechanism.We determine the locations of massive activations within hidden states, i.e., their feature and sequence dimensions. Since we have shown that their values largely stay constant in middle layers, we take on any such layer for this analysis.LLaMA2-7B. In this model, massive activations are identified in two feature dimensions (1415 and 2533). Regarding sequence dimensions, we find that massive activations appear at: 1. the starting word token, 2. the token representing the first period (.) or newline token (\n) in the sequence. Figure 1 illustrates these findings for LLaMA2-7B. This is also consistent on long sequences. In cases where the input contains a "." or "\n" token, four massive activations are observed. For the less common scenario where neither "." nor "\n" is present, we can see two massive activations, both of which are associated with the initial token.LLaMA2-13B. We find that massive activations in this model consistently appear in two feature dimensions, 2100 and 4743. These activations are exclusively located within the starting token of the sequence, regardless of its semantics. Figure 2 illustrates these behaviors within LLaMA2-13B. For any given input sequence, only two massive activations are present, corresponding to features 2100 and 4743 of the first word token.Mixtral-8x7B. For this particular model, massive activations lie in two feature dimensions, i.e., 2070 and 3398. For sequence dimensions, we find that they are associated with the starting token, delimiter tokens and also certain word tokens, e.g., token "and" and token "of". Figure 3 showcases these patterns within Mixtral-8x7B. These word tokens tend to be conjunctions and prepositions, representing relatively few semantics of the input. Generally, for inputs of 4096 tokens in length, these tokens are predominantly located in the early part of sequence.Summary. We summarize our findings of massive activations beyond the three models discussed above. We also put other models into categories based on empirical observations.• For feature dimensions, they are consistently present in very few fixed dimensions.• For sequence dimensions, we classify LLMs into three categories: a) Starting token only. Models include LLaMA2-13B, MPT and GPT-2. b) Starting token and the first "strong" delimiter token (i.e., "." or "\n"). Models include LLaMA2-7B and LLaMA2-7B-Chat. c) Starting token, delimiter tokens (such as ".", "\n", "'" or ","), and certain word tokens with weak semantics (such as "and", "from", "of" or "2"1 ). Models include LLaMA2-70B, Mistral-7B, Mixtral-8x7B, Falcon-40B and Phi-2.With an understanding of the nature and locations of massive activations, we now discuss the differences between them and outlier features, a seemingly similar phenomenon in LLMs. Dettmers et al. (2022) have discovered outlier features in LLMs, characterized by large magnitudes. Conceptually, a massive activation is a scalar value, determined jointly by the sequence and feature dimensions; in contrast, an outlier feature is a vector, corresponding to activations at all tokens. Further, massive activations are present at extremely few tokens, while outlier features expect most activations in them to be large.In practice, we find that massive activations do not overlap with outlier feature dimensions. We identify outlier features in LLaMA2-7B and 13B using the definition in Dettmers et al. (2022) : a feature is deemed as an outlier feature if activation magnitudes exceed 6.0 at more than 25% of layers and 6% of tokens, on more than 90 out of 100 sequences. We discover 10 and 25 outlier features in these two models respectively. However, none of them correspond to the feature dimensions of massive activations. 
massive activations act as biases in llms: 
 While we have demonstrated the existence of massive activations and identified their locations, their functional role within LLMs is not yet clear. Are they important for internal computation? Or are they simply redundant activations with no effect? This section will delve deeper into LLMs to answer these questions. Different from the previous passive observations, we take a more proactive approach by inspecting how modifying massive activations affects the external behavior of LLMs.Model Top 1 Top 2 Top 1% Top 10% Median LLaMA2-7B 2556.8 ± 141.0 -1507.0 ± 83.0 -0.14 ± 0.6 0.0 ± 0.5 0.2 ± 0.3 LLaMA2-13B -1277.5 ± 14.6 -787.8 ± 8.0 0.9 ± 0.7 -0.3 ± 0.8 -0.3 ± 0.6Table 2 : The mean and variance of activation values at various positions, corresponding to the 2 largest, top 1% and 10%, and the median magnitudes within the hidden state.Variations in massive activations are significantly lower in comparison to other activations.We first measure the variances of massive activations across input sequences. Besides massive activations, we choose three other positions based on their average magnitudes, corresponding to the top 1%/10%, and the median within the hidden state. In Table 2 , we show the mean and standard deviation of the activation values at these positions across 100 sequences, for LLaMA2-7B and 13B. We find that the variances of massive activations are considerably smaller relative to their mean values when compared to other activations.We then modify the inference of LLMs by intervening massive activations at one layer-for a hidden state exhibiting massive activations, we manually set these activations to chosen fixed values. Then the altered hidden state is fed into the next layer, and the computation afterwards continues as normal. We modify massive activations in LLaMA2-7B and 13B. We evaluate the perplexity on WikiText, C4 and PG-19 and the mean zero-shot accuracy on BoolQ, PIQA, WinoGrande, Arc-Easy and Arc-Challenge. For each model, we perform the intervention once on the hidden state where massive activations first appear. This corresponds to layer 2 and layer 4 in LLaMA2-7B and 13B respectively.Setting massive activations to zero. We evaluate the performance of LLMs without massive activations. We set their values to zero in the hidden state when they first appear, i.e., removing massive activations from intervened LLMs. shown in Table 3 . Intriguingly, there is a significant degradation in model performance, e.g., exploding perplexity numbers. For comparative analysis, an equal number of activationsthose with average magnitudes close to the median magnitude-are similarly set to zero. We find this leads to no performance drop. These results highlight the crucial role that massive activations play in the internal computation of LLMs.Setting massive activations to mean values. We remove the small variances in the values of massive activations. Specifically, we adjust the values of massive activations to their empirical mean values. The means are computed on 100 sequences from RedPajama. The results of this intervention (denoted by Set to mean) are shown in Table 3 . We find that there are negligible changes in perplexity and zero-shot accuracy. This shows that their values are constants and input agnostic, i.e., functioning similarly to bias terms.To summarize our findings:Massive activations act as fixed but important biases in LLMs.The fact that these activations act as biases may explain why LLMs store them at certain layers and tokens:• The tendency of these activations to appear at the starting token could be attributed to the fact that every autoregressive training instance contains an initial token. Since LLMs are based on next word prediction, the starting token is the only token used in all forward passes within a sequence. • The existence of these activations in delimiter tokens might be due to the relatively low semantic value of these tokens, rendering them a low-cost option for storing such biases. Conversely, tokens with rich semantics would risk significant loss of input information, if they are repurposed to store biases. • The fact that massive activations emerge only after a few initial layers might be because LLMs require some initial layers to process the meaning of the tokens associated with massive activations. At these layers, their semantics may be transferred to other token positions via self-attention, and preserved moving forward. 
effects on attention: 
 In this section, we study the internal mechanism of massive activations in LLMs, particularly in relation to self-attention.We observe a stark contrast in attention patterns when comparing layers before and after the appearance of massive activations in LLMs. Figure 5 shows the attention logits (before softmax), averaged over all heads per layer in LLaMA2-7B. The input is a prompt from MMLU (Hendrycks et al., 2021) : "The following are multiple choice questions (with answers) about machine learning.\n\n ...". Recall that in LLaMA2-7B, massive activations first appear in the output of layer 2 (see Figure 4 ). We find that in layer 3 and deeper layers (e.g., layer 31), attention is mostly concentrated on the two tokens associated with massive activations. Our observations are also consistent across various LLMs. We notice that there is a consistent pattern across models on the distribution of attention logit values. In Figure 5 and Figure 6 , many attention logits tend to be negative following massive activations. They are mostly computed by the inner product between query and key states of tokens without massive activations. However, when the key states belong to tokens associated with massive activations, the resulting attention logits are slightly positive. Thus in the attention softmax (computed along each row), these special attention logits will attract most of the attention probability.Recently, Xiao et al. (2023b) showed that LLMs attend heavily to the starting token. Our findings on LLaMA2-13B in Figure 6a align with their results. Empirically, we find it is true for LLMs where massive activations are only found within the starting token. However, our results on LLaMA2-7B and Phi-2 indicate that LLMs also allocate substantial attention to other tokens and they are associated with massive activations. Furthermore, our results reveal a deeper cause for the emergence of these attention concentration patterns.In this part, we delve into the computation within the attention block and demonstrate that LLMs use massive activations to enforce an implicit bias term in self-attention.Attention LayerNorm and QKV projections. We study the impact of massive activations on the query, key and value states (Q/K/V) in self-attention. In LLMs, at each layer, input features are processed by layer normalization2 (Ba et al., 2016) and then transformed into query, key and value states via linear projections, as illustrated in Figure 7a . This design choice is introduced in GPT-2 (Radford et al., 2019) and widely adopted in modern LLMs. Same across all tokens Figure 7b visualizes all hidden states computed in this schematic (LLaMA2-7B, layer 3).We find that at all stages, features of the two tokens associated with massive activations are drastically different from other tokens. Specifically, after the first "normalize" step, the embeddings of these two tokens appear as a sparse vector with two distinct non-zero elements. Notably, the subsequent QKV states exhibit considerably smaller variations within each embedding. We hypothesize that the attention LayerNorm may play a pivotal role in this process (see Appendix C.2 for further discussion).Given that attention is also concentrated on the tokens associated with massive activations (Section 4.1), we thus isolate these tokens and study their effects on the attention output (the layer of attention matrix multiplying value vectors).In Equation 2, we decompose the attention output at each token k into two parts: value updates from the tokens C where attention is concentrated; and value updates aggregated from other tokens. Here p k i is the attention distribution of query token k to token i, and v i is the value state of token i.Attention(Q, K, V) k = ∑ i≤k p k i v i = ∑ i∈C p k i v i + ∑ i / ∈C p k i v i (2)Figure 8 visualizes the decomposed value updates and the attention output in LLaMA2-7B, with the input prompt "Summer is warm. Winter is cold.". In this case, the set C consists of token Summer and the first period token. We can see that the value updates from C are nearly identical across tokens, i.e., they serve as additive bias terms, although not explicitly imposed. Furthermore, we note that this pattern of value update is strikingly similar across various inputs. We refer the reader to Appendix C.3 for additional analysis. Overall, our results indicate that LLMs use massive activations to allocate substantial attention at certain tokens. These tokens are then utilized by LLMs to form a constant bias term when computing the attention output.Given the strong need of LLMs to learn implicit attention biases during pretraining, we thus experiment with directly augmenting self-attention with additional bias terms. Intriguingly, we find that models with explicit attention biases do not exhibit massive activations.Formulation. The idea is to model such attention biases explicitly, except not through repurposing existing tokens in the input sequence. Thus we introduce additional learnable parameters k ′ , v ′ ∈ R d for each head. Specifically, given input query, key and value matrices Q, K, V ∈ R T×d , the augmented attention with explicit attention biases is computed as:Attention(Q, K, V; k ′ , v ′ ) = softmax Q K T k ′ √ d V v ′ T (3)where k ′ and v ′ are each concatenated with the key and value matrices K/V. The proposed attention can be used as a drop-in replacement of standard attention, without modifying other parts of Transformers, e.g., positional embeddings and MLP blocks. Results. We train three GPT-2 models: the standard model, GPT-2 prepended with a sink token (Xiao et al., 2023b) and GPT-2 with explicit attention biases. See Appendix C.4 for training setups. We find that the three models have the same performance at convergence but differ significantly in the status of massive activations, as demonstrated in Figure 9 . Notably, in GPT-2 with explicit attention biases, massive activations disappear, as compared to the default GPT-2 and one with a sink token.Figure 10 shows the three largest activation magnitudes at the output of each layer in GPT-2. Notably, with explicit attention biases, top activation magnitudes in GPT-2 are increasing gradually as layers go deeper. In contrast, for both the default GPT-2 and GPT-2 with a sink token, massive activations emerge within one intermediate layer, i.e. layer 4. These results indicate that explicit attention biases negate the necessity for LLMs to develop massive activations during the pretraining phase.In Figure 11 , we visualize the attention distribution in both default GPT-2 and GPT-2 with explicit attention biases, where we plot the average attention probability over 50 sentences each with 30 tokens. First, we find that our observations on the relationship between massive activations and attention concentration hold for the default GPT-2 model. Second, for the GPT-2 model with explicit attention bias, most of the attention probability is assigned to the extra k ′ and v ′ vectors we inserted. Intriguingly, this also holds for initial layers as well (e.g., layer 1), suggesting the strong need for LLMs to form this attention concentration pattern during pretraining.To summarize our findings in this section:Massive activations are connected to self-attention. LLMs use massive activations to concentrate substantial attention on very few tokens, injecting implicit bias terms in the attention computation. Further, massive activations can be eliminated by augmenting LLMs with explicit attention biases. 
related work: 
 Intriguing properties of autoregressive Transformers. Timkey & Schijndel (2021) observed that in GPT-2's penultimate layer, there are feature dimensions containing activations with magnitudes up to 3,000, and these dimensions dominate standard measures of representation similarity. Heimersheim & Turner (2023) found that the feature norm of the initial token in GPT-2 grows much faster than other tokens. Kovaleva et al. (2021) and Zhao et al. (2023) discovered outlier weights in the LayerNorm of GPT-2 and LLaMA2-13B, and setting them to zero leads to catastrophic drop in model performance. Notably, the feature dimension of this weight in LLaMA2-13B (i.e., 2100) corresponds to that of a massive activation (Figure 2 ).Outlier features. Various works in quantization (Dettmers et al., 2022; Zeng et al., 2022; Xiao et al., 2023a; Lin et al., 2023; Ahmadian et al., 2023) have studied the existence of outlier features in LLMs. Dettmers et al. (2022) showed that outlier features have large activation values in most of their sequence dimensions. While massive activations can be seemingly similar to outlier features, we discussed their fundamental differences in Section 2.3. We show that massive activations can not be attributed to the existence of outlier features. There can be various notion of biases in the self-attention mechanism. First, simple additive bias terms can be used in linear layers for computing the query, key and value states (Namazifar et al., 2023) . Many LLMs do not enable additive bias terms for linear layers during the pretraining phase (Chowdhery et al., 2022) . Second, position biases can be inserted in self-attention to encode positional information of each token (Su et al., 2021; Press et al., 2021) . There are also variants of biases with manually designed softmax operator (Miller, 2023; Bondarenko et al., 2023; Hu et al., 2024) . Our work reveals that LLMs, even with standard self-attention, would impose implicit bias components in attention computation through massive activations. 
conclusion and discussion: 
 Autoregressive training of large Transformers has brought significant advances in natural language processing. This study reveals the widespread existence of massive activations in these Large Language Models (LLMs). The values of these activations are input agnostic but crucial for model performance, despite their extremely rare quantity. We establish a close connection between massive activations and the self-attention mechanism. We show that LLMs use them to implement an implicit form of biases for attention computation. We hope our results contribute to a deeper understanding of today's large-scale foundation models.We discuss some practical implications and future directions of this work. First, the presence of activations with large magnitudes has been widely known as a major challenge in effectively quantizing LLMs (Dettmers et al., 2022; Xiao et al., 2023a) . This paper identifies a new type of outlier activations in LLMs, and we hope our findings will be valuable to researchers working on LLM compression. Second, attention maps that allocate excessive attention probabilities to a few fixed tokens may be undesirable for mechanistic interpretability (Olsson et al., 2022) . Our proposed attention formulation could make the resulting attention maps in LLMs more interpretable, and benefit downstream applications (Darcet et al., 2023) . Finally, our investigation of the new attention formulation focused on its effects on massive activations, and our experiments were limited to a small GPT-2 model due to computational resource constraints. It would be interesting to see how our results generalize to models at larger scales, and how our attention formulation could affect the training stability (Wortsman et al., 2023) Further, we conduct intervention analysis to analyze the role of registers. We replace all register features at the output of every layer with their means, averaged over 10k ImageNet training images. This intervention removes the intended purpose of registers to aggregate global input information (Darcet et al., 2023) . Table 6 shows the results. We find that ViTs with fixed register features achieve accuracy comparable to original models, suggesting that registers act as learned biases in ViTs. This leads to constant key and value states at register tokens, effectively introducing bias terms to self-attention (extra k ′ and v ′ in Equation 3). Thus a ViT with register tokens function equivalently to a standard ViT augmented with explicit attention biases.In Masked Autoencoders (MAEs) (He et al., 2021) , a dummy token is added to ViTs during pretraining. In one fine-tuning pipeline of MAEs, fine-tuning is done based on the average pooled features of all patch tokens. In these MAE models, this dummy token is equivalent to a register token. Here we maintain the register token features as constant across the output features of all layers in ViTs, which we denote as Fix-Reg-Mean. These fixed values are computed as the average register features over 10k ImageNet training images. To summarize our findings:Massive activations exist in many but not all ViTs. Similar to those in LLMs, these activations act as constant biases. We also show the recently proposed register tokens have a similar function. 
: 
 x and y axes are sequence and feature dimensions. For this specific model, we observe that activations with massive magnitudes appear in two fixed feature dimensions (1415, 2533) , and two types of tokens-the starting token, and the first period (.) or newline token (\n). 
 
 Appendix: In this section, we supplement the main paper with additional results of massive activations in LLMs. This includes results on more pretrained LLMs (Appendix B.1) and fine-tuned LLMs (Appendix B.2), analysis of the BOS token 
 (Appendix B.3) and layer-level analysis (Appendix B.4).In Section 2, we have demonstrated massive activations in LLaMA2-7B, LLaMA2-13B and Mixtral-8x7B. In this section, we evaluate more pretrained LLMs which cover a wide range of model families. We illustrate massive activations in LLaMA2-70B, Phi-2, Mistral-7B (Jiang et al., 2023) , MPT-7B (MosaicML, 2023) and Falcon-7B (Almazrouei et al., 2023) . The results are presented in Figure 17 , 18, 19, 20 and 21.We make several observations. First, massive activations are consistently present in these models and they exhibit similar characteristics to those described in Section 2. Intriguingly, for LLaMA2-70B, we find that massive activations are found within tokens representing numerical values, e.g., token "0" and token "2", as depicted in Figure 17 . However, they do not appear in all numerical tokens (see the rightmost example in Figure 17 ). Another interesting finding is that the feature dimension of massive activations in both Mistral-7B (Figure 19 ) and Mixtral-8x7B (Figure 3 ) is identical (i.e., 2070), implying that the latter model may have been fine-tuned from the former.Figure 17 : Massive activations in LLaMA2-70B.Figure 18 : Massive activations in Phi-2.Figure 19 : Massive activations in Mistral-7B.Our results so far are focused on pretrained LLMs. However, a significant application of LLMs lies in their use for chat purposes. Instruction fine-tuning (Ouyang et al., 2022) is essential for developing models capable of generating coherent responses to questions. InFigure 20 : Massive activations in MPT-7B.Figure 21 : Massive activations in Falcon-7B.this part, we demonstrate massive activations in these fine-tuned models. We evaluate fine-tuned models from models in LLaMA2 and Mistral. The results are shown in Figure 22 , 23, 24 and 25.We can see that massive activations persist after instruction fine-tuning. Moreover, the values and positions of massive activations remain largely the same as the original pretrained LLMs. For LLaMA2-7B, this can be seen by comparing Figure 22 and Figure 1 . However, one exception is Mixtral-8x7B. We find that massive activations disappear from the newline token "\n" after fine-tuning, as shown by comparing Figure 25 and Figure 3 . We leave the study on how instruction fine-tuning affects massive activations for future work.Figure 22 : Massive activations in LLaMA2-7B-Chat.Figure 23 : Massive activations in LLaMA2-13B-Chat.In some tokenizers, e.g., LLaMA2, the BOS token 
, also known as the beginning of sequence token, can be prepended to the input sequence. For the experiments presented in other parts of the paper, we turn off this option, where all sequences do not start with the BOS token.Figure 24 : Massive activations in Mistral-7B-Instruct.Figure 25 : Massive activations in Mixtral-8x7B-Instruct.In Figure 26 , 27 and 28, we show massive activations in LLaMA2-7B, LLaMA2-13B and Mixtral-8x7B, with the same input sequences as in Section 2. We find that massive activations persist with a prepended BOS token. In LLaMA2-7B and LLaMA2-13B, the locations of massive activations, i.e., sequence and feature dimensions, are not altered. However, for some massive activations shift to the BOS token 
. We leave the study on how the BOS token 
 affects the positions of massive activations for future work.Figure 26 : Massive activations in LLaMA2-7B when the input is prepended with a BOS token 
.Figure 27 : Massive activations in LLaMA2-13B when the input sequence is prepended with a BOS token 
.In Section 2.1, we have presented the layer-level analysis results for LLaMA2-7B, LLaMA2-13B and Phi-2. In Figure 29 , we provide the comprehensive results for all LLMs examined in this paper (listed in Table 8 ). This includes LLMs from LLaMA2, Mistral, MPT, Falcon, OPT and GPT-2 model families. For each model, we show the three largest activation magnitudes as well as the median at each layer.Figure 28 : Massive activations in Mixtral-8x7B when the input sequence is prepended with a BOS token 
. We can see that the trend of massive activations we observe in Section 2.1 holds true for LLMs in general. Massive activations tend to remain constant in most of the intermediate layers. They emerge in the early layers and disappear in the last layer.In this section, we provide additional results for the analysis on self-attention. This includes results on more LLMs (Appendix C.1), analysis of attention LayerNorm (Appendix C.2), more results on implicit attention biases (Appendix C.3) and detailed results on training GPT-2 with explicit attention biases (Appendix C.4).In Section 4, we have demonstrated the attention concentration pattern in LLaMA2-7B, LLaMA2-13B and Phi-2. We now illustrate this phenomenon for more LLMs. Figure 30 and Figure 31 show the results for LLaMA2-70B and Mistral-7B. For these two models, massive activations are formed in the output feature of layer 9 and layer 2 respectively.We can see that attention is predominantly focused on the sequence dimensions of massive activations. In the case of LLaMA2-70B, as depicted in Figure 30 , massive activations are found in the starting word token and also token 2. These two tokens receive substantial attention logits. Additionally, we visualize the attention probability in Figure 32 . The attention softmax is computed along each row, thus resulting in these special tokens being allocated a much higher attention probability. Our analysis in Section 4.2 indicates that tokens associated with massive activations have drastically different key and value states. In this part, we investigate how attention layernorm plays a crucial role in this process.Preliminaries. There are two specific types of layer normalization commonly used in LLMs. One is the standard layer normalization (Ba et al., 2016) . Suppose we have a feature vector x ∈ R d , LayerNorm will normalize this feature to fix the mean and variance and then re-scale with element-wise affine transformation:EQUATION)where g, b ∈ R d are parameters of the affine transform, also called the gain and bias.In addition to the original LayerNorm, a variant of layer normalization has also been used in LLaMA2 and Mistral models. Specifically, Root Mean Square Normalization (RM-SNorm) (Zhang & Sennrich, 2019) normalizes the feature x ∈ R d with the root mean square (RMS) statistic:.EQUATION)where g ∈ R d is the gain parameter. For both LayerNorm and RMSNorm, when there are a few activations in x ∈ R d that have significantly large magnitudes, the denominator in the normalization step, i.e., σ in Equation 4 and RMS(x) in Equation 5, becomes large as a result. In fact, the denominator is almost determined by these few massive activations. The large denominator will push all normal values to zero while preserving the outlier nature of massive activations. This will effectively create a drastically different normalized feature, determined by the few massive activations. Figure 33 shows two activation trajectory in both RMSNorm and LayerNorm. We can see that how the normalization step (middle) preserves the outlier activations in tokens Who and \n and normalized features at these two tokens become extremely similar.In Section 4.2, we have shown how the value updates from the tokens associated with massive activations tend to be largely identical. Here we extend those findings by examining additional input prompts and layers within the LLaMA2-7B model. We use four input prompts: "Are you cold?\n Grab a jacket.", "Will it snow?\n Check the forecast.", "Did she call?\n I missed it." and ""I am doing well. Thank you for asking." ". We visualize the value updates in layer 3, layer 15 and layer 30 in Figure 34 , Figure 35 and Figure 36 respectively. We focus on the latter half of the input sequence, following the two tokens associated with massive activations. We can see that in the same layer, the value updates ∑ i∈C p k i v i display remarkable similarity across the different input sequences.Experimental setup. We use the open-source reproduction of GPT-2 from the NanoGPT repository (Karpathy, 2023) . We use the default recommended training setup and optimizer setting. For each of the three GPT-2 models, we train for 50,000 iterations, with a total of approximately 2B tokens. For the GPT-2 with a sink token, we follow Xiao et al. (2023b) , where we prepend each training sequence with a learnable sink token [SINK] . When computing the training loss, we do not include the cross-entropy loss computed on the prepended sink token. For GPT-2 with explicit attention biases, we initialize each k ′ and v ′ with N (0, 0.02I). Regarding the performance of the three GPT-2 models we evaluate in Section 4.3, we find that after 50,000 training iterations, they have the same perplexity on the validation split constructed from OpenWebText2 (Gao et al., 2021) : 3.04.We also experiment with other ways of injecting biases in the self-attention computation:1. The first one is a special case of our proposed formulation in Equation 3, where both k ′ and v ′ are zero vectors. Equation 6shows the computation of this variant of self-attention. This is also equivalent to the previous proposed Softmax-off-byone (Miller, 2023) .Attention(Q, K, V) = softmax Q K T 0 √ d k V 0 T (6)2. Since Equation 3 can be viewed as inserting a sequence dimension, we also experiment with inserting one extra feature dimension. Specifically, we add learnable parameters q ′ , k ′ ∈ R T and concatenate them with the query and key states respectively. This variant of self-attention is as follows:Attention(Q, K, V; q ′ , k ′ ) = softmax Q q ′ K k ′ T √ d k V (7)3. We also experiment with a simple way to enforce constant value updates by injecting an extra value parameter v ′ ∈ R d k . This variant of self-attention is as follows:Attention(Q, K, V; v ′ ) = softmax QK T √ d k V + v ′ (8)Figure 37 visualizes the ten largest activation magnitudes in three GPT-2 models, corresponding to the three formulations of biases in Equation 6, 7 and 8. We find that these alternatives are not able to eliminate massive activations during pretraining. We list the datasets used in this work and relevant license information:• RedPajama (Together Computer, 2023): Apache License, Version 2.0• OpenWebText2 (Gao et al., 2021) : MIT License• C4 (Raffel et al., 2020) : Open Data Commons Attribution License 1.0 license • PG-19 (Rae et al., 2019) :Apache License, Version 2.0• WikiText (Merity et al., 2016) : Creative Commons BY-SA 3.0 license • MMLU (Hendrycks et al., 2021) :MIT License• BoolQ (Clark et al., 2019a) : Creative Commons BY-SA 3.0 license • PIQA (Bisk et al., 2019) :The license status is unclear• WinoGrande (Sakaguchi et al., 2019) : Apache License, Version 2.0• ARC easy and challenge (Clark et al., 2018) : Creative Commons BY 4.0 license• ImageNet (Deng et al., 2009) :The license status is unclearSuch numeric tokens exhibit massive activations only in certain contexts, e.g., dates and years. Refer to Figure17for an illustration on LLaMA2-70B.LLaMA2 uses a variant of layer normalization: RMSNorm(Zhang & Sennrich, 2019). 


Decision: The reviewers agree that this work provides an interesting observation, has thorough experiments around it, and is well presented. Some reviewers would like more discussion of practical implications of the presence of massive activations and FDqV feels that the paper does not provide sufficient evidence, especially theoretical analysis, to understand the fundamental reasons behind these patterns. I am inclined to agree that a theoretical analysis or controlled synthetic experiments could help us understand the phenomenon better, but I view that as an opportunity for followup work.

Reviews: review: paper_summary: paper_summary: This paper identifies the phenomenon of massive activations in large language models (LLMs), where very few activations exhibit significantly larger values than others. It extensively studies the layers where massive activations reside, their importance to LLMs' capabilities, and their role as bias in the self-attention mechanism. Finally, the analysis is further extended to ViTs.
strength_weakness: strength_weakness: 1. Some of the observations lack deeper analysis. For example, massive activations seem to serve as an important bias in the attention mechanism, while they mainly focus on semantically unimportant tokens like delimiters. The authors are expected to provide a clearer explanation regarding this gap.

2. It is not clear whether massive activations are desirable or whether they will disappear with larger-scale pretraining. The authors are expected to evaluate larger LLMs pretrained on larger corpora to determine whether the percentage of massive activations will increase with the model scale and data scale.
paper_summary: paper_summary: This analysis paper studies the phenomenon of massive activations, which is defined in this paper as activations with magnitude larger than 100 and 1,000 times larger than the median magnitude of its hidden state. The paper had the following main findings:

- Massive activations widely exists in different models, largely in the middle layers, and occurs on fixed dimensions of a model and often on frequent tokens
- Setting massive activation to 0 significantly changes model behaviors.
- Massive activation causes attention to be concentrated, and act as bias terms in attention modules. Training models with a bias term in the attention module can eliminate massive activations.

In summary, I think the general presentation quality is very high, forming a very well-motivated and coherent story that's mostly easy to follow. I do have some doubts about the impact of this work, because I fail to see how such understanding can help us make improvements to the current transformer architecture. I'm hoping the authors to shed more lights on those doubts.
strength_weakness: strength_weakness: - Although the authors provided great insight into reasons why massive activations occur in transformers, I have some doubts on what steps the community can take on those findings. Some discussions around that theme would be very helpful -- for example, should people start to add a bias term on the attention modules? Should people be more careful with normalization in the attention module? What improvements should people expect from them? Is it better performance or just better training stability? Answering these questions can potentially help amplify the impact of the paper.
- Analysis stopped at 13B models and decoder-only models. I would be curious to see if the phenomenon generalizes to encoder-decoder models, or even larger models, e.g. 175B BLOOM models.
questions: questions: - For "setting massive activation to a certain value" experiment in Section 3, did you try to set it to a non-zero regular magnitude value (e.g. mean of the hidden state)? I wonder if 0 has some kind mathematical specificity here that might have caused the significant change.
- For Section 4.2, I'm not quite sure if I'm following the transition between layer-norm to output decomposition -- they feel somewhat unrelated to each other. Can you explain how one affect another? Also, have you tried to run experiments with the attention module in the original transformer paper? I believe there is no layer-norm before the linear transformation.
paper_summary: paper_summary: This paper introduces an outrageously large value sometimes appears in a hidden vector in a large language model. This paper calls such a large value "massive activation". The authors conducted several analyses on the massive activations. Then, the authors show situation which the massive activation appears in, and the relation between attention calculation and the massive activation.
strength_weakness: strength_weakness: This paper introduces an interesting phenomenon, the massive activation, but I think this is an ongoing study because this paper does not reveal the reason of the massive activation and the effect on LLM behaviors. For example, the relation on the stabilization of LLMs. Section 4 of this paper implies that the massive activations have a role such as a bias term. PaLM paper (https://arxiv.org/abs/2204.02311) clams that it important to remove bias terms to stabilize the pre-training of LLMs. These suggest that massive activations might have a serious influence on the stabilization of LLMs. Like this example on the stabilization, the authors should conduct more investigations to reveal why massive activations appear in.
paper_summary: paper_summary: This paper identifies a new phenomenon called “massive activations” in Large Language Models (LLMs), which shows that the hidden state vectors in intermediate layers at certain token locations have a few exceedingly large coordinates. The finding extends the attention sink phenomenon of Xiao et al. (2023) in the sense that these tokens are precisely the tokens that attract attention scores. The paper reports several qualitative properties such as their appearance and their difference in different LLMs. The paper performs intervention studies and further some small-scale pretraining experiments to demonstrate that massive tokens act like a fixed bias in the attention mechanism.
strength_weakness: strength_weakness: * The pretraining experiment (Sec 4.3) with the k’v’ variation as in Eq (3) is not fully satisfactory in showing that massive tokens act as a fixed bias.  
  
  - Eq (3) does not really differ from the “GPT2 with sink token” baseline, as it seems mathematically almost equivalent (maybe up to some minor reparametrization, since Eq(3) did not parametrize a hidden state vector but rather one k’v’ vector for each head). So even though k’v’ method removes massive activation in other tokens as reported, I am guessing this special k’v’ token still attracts (say >90%) attention, so this method still preserves attention concentration.
  
  - (An interesting test would be whether k’v’ “correspond” to massive activations or their post-layernorm versions, but here it does not sound possible since you only parametrized k’v’, not the pre-KV projection versions.) 
  
  - On the other hand, Eq (8) in appendix seems like a more ideal version of fixed attention bias, and addresses the unbalancedness in the softmax as well as the massive activation phenomenon. Unfortunately pretraining with it only works to some extent (it reduces the top activation magnitude from 2K to 600, but the ideal target is like 100-200). Again, I am wondering if the attention scores are still concentrated here (and I am guessing they are)?

  - If Eq (8) cannot be made to work to fully remove massive activation, it then creates the question of whether attention concentration is fundamental (for achieving good LM performance). Have the authors thought about other ways of removing massive activations / attention concentration?

* No immediate practical applications, but I don’t think that’s a major concern given the potential importance of the phenomenon. I also wonder if the authors have thought about alleviating training instabilities by mitigating massive activation and/or attention concentration (kind of related to the above question)?
questions: questions: See the “weakness” part.

Comments: title: Thanks for the reponse
Response to author feedback
comment: Thank you for the response! I am happy to maintain my score.
The authors have addressed all my concerns. I plan to maintain my original score.

Please consider adding the practical implication section of the response to the paper, maybe replacing the final sentence of the paper -- it's a more meaningful way to end the paper rather than saying "we observed this, hopefully it helps" :)
Thank the authors for their response.
I thought I had understood the explanation by authors. In my understanding, the authors provided analyses on the massive activations through the observation of the trained language models but I would like the authors to reveal theoretical reason why massive activations appear in. In particular, I would like the authors to provide the mathematical proof why the large language models contains the massive activations.

However, through reading other reviews, my assigned score might be low although the paper reporting a new interesting phenomena. So, I raise my score.
Thank the authors for their response, which properly addresses my concerns. I will keep my original rating.