Systematic Outliers in Large Language Models
Authors: Yongqi An
Xu Zhao
Tao Yu
Ming Tang
Jinqiao Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Model Interpretability
Large Language Models
Outliers
Attention Mechanism
Primary area: interpretability and explainable AI
TL;DR: We systematically analyze outliers in large language models, revealing they function as implicit context-aware scaling factors within the attention mechanism, and propose a method to eliminate them to enhance model performance and efficiency.

Abstract: Outliers have been widely observed in Large Language Models (LLMs), significantly impacting model performance and posing challenges for model compression. Understanding the functionality and formation mechanisms of these outliers is critically important. Existing works, however, largely focus on reducing the impact of outliers from an algorithmic perspective, lacking an in-depth investigation into their causes and roles. In this work, we provide a detailed analysis of the formation process, underlying causes, and functions of outliers in LLMs. We define and categorize three types of outliers—activation outliers, weight outliers, and attention outliers—and analyze their distributions across different dimensions, uncovering inherent connections between their occurrences and their ultimate influence on the attention mechanism. Based on these observations, we hypothesize and explore the mechanisms by which these outliers arise and function, demonstrating through theoretical derivations and experiments that they emerge due to the self-attention mechanism's softmax operation. These outliers act as implicit context-aware scaling factors within the attention mechanism. As these outliers stem from systematic influences, we term them systematic outliers. Our study not only enhances the understanding of Transformer-based LLMs but also shows that structurally eliminating outliers can accelerate convergence and improve model compression. The code is avilable at \url{https://github.com/an-yongqi/systematic-outliers}.

Published as a conference paper at ICLR 2025
SYSTEMATIC OUTLIERS IN LARGE LANGUAGE MOD- ELS
Yongqi An!:2 Xu Zhao Tao Yu Ming Tang' 2 Jinniio Wang 12.3,44
FFouddttion Model Research Center, Institute of Automation, Chinese Academy of Sciences, Beijing, China Schooo of artificial intelligence, University of Chinese Academy of Sciences, Beijing, China Wuhan Al Research, Wuhan, China, *Objecteye Inc., Beijing, China {yongqi an,, an, xu. zhao, tangm, @gwangg@nn1pp ac. yutao2022@ia.co
ABSTRACT
Outliers have been widely observed in Large Language Models (LLMs), signifi- cantly impacting model performance and posing challenges for model compres- sion. Understanding the functionality and formation mechanisms of these out- liers is critically important. Existing works, however, largely focus reduc- ing the impact of outliers from an algorithmic perspective, lacking an in-depth investigation into their causes and roles. In this work, provide detailed analysis of the formation process, underlying causes, and functions of outliers in LLMs. We define and categorize three types of acttivttinnt outliers, weight outliers, and attention outliers-and analyze their distributions across dif- ferent dimensions, uncovering inherent connections between their occurrences and their ultimate influence on the attention mechanism. Based on these ob- servations, hypothesize and explore the mechanisms by which these out- liers arise and function, demonstrating through theoretical derivations and ex- periments that they emerge due to the self-attention mechanism's 's softmax op- eration. These outliers act as implicit context-aware scaling factors within the attention mechanism. As these outliers stem from systematic influences, we term them systematic outliers. Our study not only enhances the understanding Transformer- based LLMs but also shows that structurally climinating outliers can accelerate convergence and improve model compression. The code is avilable at hthttps:///gtthii com/ cotp:///oooooooga///sseeeeaa lo-utttttttttttssssss.....
INTRODUCTION
Large Language Models (LLMs) have recently demonstrated remarkable capabilities (Brown 2020 Achiam et al., |023 Touvron al. 2023a), making them central topic of research across various domains. Numerous studies have uncovered intriguing phenomena within these models (Dettmers) cct al., 2022: Xiao et al. 2023b Sun et al. 2024), which are crucial for advancing the understand- ing and application of LLMs. Among these phenomena, the presence of outliers, which are values that deviate significantly from the average of their distribution. has garnered considerable atten-
tion (Zhang al. 2024 Kovaleva et al. 2021 Paglieri et al. 2024).
However, research on outliers in LLMs predominantly emphasizes mitigating their impact through algorithmic techniques, often neglecting thorough exploration of their underlying causes and func- tional roles. This narrow focus results in two key shortcomings: first, limits our understanding of why outliers occur and how they influence model behavior (Yin et al. 2023 Xiao al.) 2023a Hooper et al.) 2024);; and second, it overlooks the interrelationships between different types of out- liers. treating them isolation rather than as part of a comprehensive, systematic framework (Zhang et al. 2024 Sun et al. 2024 Liao & Monz 2024). These gaps hinder a deeper understanding of the mechanisms underlying LLMS and constrain opportunities for more effective optimization and broader applications.
To address these gaps, we systematically analyze outliers in LLMs, focusing on their formation. dis- tribution, and roles within the models. We begin by defining and categorizing three types of outliers:
"Coresponding Author

---

Published as a conference paper at ICLR 2025
activation outliers, weight outliers, and attention outliers. By examining their distributions across various dimensions, we uncover inherent connections between their occurrences and demonstrate how these outliers collectively influence the attention mechanism. Building these findings, propose a hypothesis regarding the formation mechanisms and functions of these outliers, supported by theoretical derivations and experiments. Specifically; show that these outliers emerge result of the self-attention mechanism's ' softmax operation and act as implicit, context-aware scaling factors in the attention mechanism. Furthermore, our experiments show that structurally eliminating these outliers can accelerate convergence and enhance model compression, providing new insights for future model optimization and design.
Our main contributions are:
We define and categorize outliers in LLMs into three types types-activation, weight and attention outliers-and and uncover their systematic relationships.'
We reveal that these outliers emerge from the self-attention mechanism's softmax operation and function as implicit, context-aware scaling factors.
We demonstrate that eliminating outliers accelerates convergence and enhances model com- pression, offering insights for optimizing LLMs.
RELATED WORK
Outliers in Large Language Models. Outliers in LLMs refer to values that deviate significantly from the average of their distribution (Dettmers al.) (022). Studies have documented various types of outliers in weights, activations, and attention scores, highlighting their presence and impact. For instance. Dettmers et al.] (2022) identified activation outliers and proposed quantization techniques to mitigate their effects. Sun al. (2024) explored the role of large activations biases in the attention mechanism. Similarly, Zhang et al. (2024) analyzed weight outliers in LayerNorm layers, demonstrating their importance for maintaining language modeling capabilities in models like GPT- 2 and LLaMA2-18 Additionally, Xiao et al. (2023b) introduced the concept of the "Attention Sink," which occurs when few keys dominate attention scores.
While previous studies recognize the presence of outliers, they typically focus on specific cases or task-specific solutions like quantization and pruning. In contrast, work provides systematic categorization of qulliers-activatioo weight, and attention outliers- and and reveals their intercon- nections and collective influence on the attention mechanism in LLMs.
The Impact of Outliers on Model Performance and Compression. Outliers significantly affect both the performance and efficiency of LLMs. Previous research has shown that removing outliers without proper handling can severely degrade performance (Puccetti al., 2021 Kovaleva et al.] 2021 Zhang et al.) 2024). In quantization, outliers amplify rounding and clipping errors, leading to substantial quantization losses (Wei et al.) [2022] Nrusimha et al. 20244; Lin et al.] [024]] Liu et al. (2024). Similarly, magnitude-batt based pruning strategies face challenges
maintaining model performance when outliers are present (Sun et al. 2002)). Yin et al. (2023) observed that outliers correlate strongly with layer sparsity, further complicting pruning approaches. Moreover, in KV cache compression, Xiao et al. 2023b) found that attention score outliers associated with specific tokens play critical role in preserving context.
Despite extensive research on the adverse effects of outliers, their formation mechanisms and func- tional roles remain largely unexplored. Existing studies focus mitigating their impact but lack a systematic investigation of their origins. In contrast, our work explores the formation of outliers within the self-attention mechanism, revealing their role as implicit, context-aware scaling factors and proposing structural solutions to enhance model convergence and compression efficiency.
SYSTEMATIC OUTLIERS: DEFINITION, EXISTENCE, AND LOCALIZATION
Definition of Outliers in LLMs. Outliers LLMs are values that deviate significantly from the average of their distribution, often surpassing threshold r. From Figure I observe that three activation outliers, weight outliers, and attention outtiers-appaar sys- tematically in four key locations within LLaMA2-7B. The specific positions of these outliers
distinct types of
2

---

Published as a conference paper at ICLR 2025
LaMMA2278,
Layer
Layer
Summer
LLaMA-2-70, Layer3
5.5
Wirter
0.4
4.2
7800
2033
Summer
W00
Woote
7500
1415
(a) activation: x²
down
(b) weight: W
(c) activation: h
(d) attention: A
Figure 1: Systematic outliers in LLaMA2-7B. Outliers are identified four locations: activations
(layer outputs he and down-projection inputs x(o=)), weights (down-projection matrices Wilwwa, and attention (attention weights A$).
further summarized in Figure as follows:
To formalize this, define outliers mathematically for each type
Activation Outliers: For layer outputs h, E Bx (batch size B and hidden dimension H), the set of activation outliers O aetivation is:
Octivationn = ((t,)) |hs,و| > T- :m
B-H Eis [hisl is the mean absolute value of he- Additionally, for down- projection inputs x xdown BBxH activation outliers are defined as:
where Ph 1
nowivatiion-own {i,,) |
dOwn i.j
>
Magins }
where Mlydean
B.H Ens
Weight Outliers: For projection weights W E Ox/ (output dimension O, input dimension I), weight outliers Oeeight are defined as:
Oweight {(i,)) | Murcc
where Mwn = + |erp is the row-wise mean absolute value W.
Attention Outliers: For cumulative attention scores A L (sequence length L), attention
outliers Og Ptteention are:
altention ( A, r M.);
where A, = El= Ars is the cumulative attention contribution for token j. and μa Σ, Ay.
These definitions provide unified framework to identify and analyze outliers across LLM components. We empirically set T 1000 in our experiments to isolate extreme deviations.
meast lanper
Existence of Outliers in LLMs. Systematic outliers consistently emerge in LLMs across various components and architectures. Fig- ure 1 highlights their presence in LLaMA2-7B (Touvron al.) 2023b), where we observe abnormally large values in activations, weights, and attention scores at specific indices. These patterns indi- cate the regular appearance of systematic outliers at critical positions within the model.
Layer
Pird
Down
MLP
This phenomenon extends beyond LLaMA2-7B to wide range of LLMs, spanning diverse model sizes and families. Our analysis con- firms that systematic outliers common feature across pretrained and fine-tuned L..Ms. For additional examples from other architec- tures, refer Appendix(A] To deepen understanding. we lyze their distributions across various dimensions, providing insights into their underlying causes and functional roles, which crucial for unraveling their systemic impact.
Layenfore
Figure 2: Illustration of systematic outliers loca- tions in L.LMs.
3

---

Published as a conference paper at ICLR 2025
LLaMA-2-78 Layer-wise Analysis
Token Distribution of Activation Outllers LLaMA-2-78
2500-
Activation Channal Index in LLaMA-2-78
2000
4000
1500-
Chatral
0000
20
500
32
Layars
Ieen
Layers
(a) Which layers?
(b) Which sequences?
(c) Which features?
Figure 3: Distribution of activation outliers in he across layers, sequences, and feature dimensions.
LLaMA-2-78 Layer-wise Analysis
LLaMA-2-78, Layer
LLaMA-2-7B. Layer
1250
1000
Magnitude
800
750
500
250
0-
Layers
36
24
32
Summer
worm
6
Wintrr
7890
Summer
warm
Wineer
cald
3721
Top
Top
Top
Modian
(a) Which layers?
(b) Layer 2
(c) Layer 31
Figure 4: Distribution of activation outliers in x xdown across layers, sequences, and feature dimen- sions.
3.1 WHERE ARE ACTIVATION OUTLIERS LOCATED?
Activation outliers manifest as abnormally large values in specific sequence and feature dimensions, as shown Figures aa) and c). These outliers appear in two distinct activation types: layer out- puts hy and down-projection inputs xdown We analyze their distributions across layers, sequences, and feature dimensions understand their patterns. Detailed experimental settings are provided in Appendix B.I
For layer outputs, Figure 3aa reveals that activation outliers are concentrated in shallow layers, persist through middle layers, and diminish in the final layers. Additionally, Figure 3(b) shows that these outliers are associated with start tokens and weak semantic tokens, such as ". and "" while Figure c) highlights their confinement to specific feature dimensions.
For down-projection inputs, Figure 4 indicates that activation outliers are confined to a few shallow and deep layers. Similar to layer outputs, these outliers linked start tokens and weak semantic tokens and are concentrated in a limited set of feature dimensions.
In summary, activation outliers in x xelown restricted shallow and deep layers, while those in he persist from shallow to middle layers. Both types predominantly affect fixed feature dimensions and tokens with weak semantic content.
3.2 WHERE ARE WEIGHT OUTLIERS LOCATED?
As shown Figure [b), weight outliers are characterized by extreme values concentrated in spe- cific columns. To quantify this, we compute the extremal ratio, defined the ratio of the maximum value to the mean value within each column, since large column values directly influence the corre- sponding output activations.
Figure 5aa) illustrates the extremal ratio across layers and modules in LLaMA2-7B, highlighting that weight outliers concentrated in the down-projection matrices & of the second layer and the last two layers. Figures5[bb and S(c) provide detailed visualizations of these outliers in the last two layers.

---

Published as a conference paper at ICLR 2025
MaxcMean Ratio Acrous Modules LLaMA-2-78
LLaMA-2-7B, Layer
LLaMA-2-78, Layer 32
1200
300
400
200
2533
7066
3226
Layers
(a) Which layers?
(b) Layer 31
(c) Layer 32
Figure Distribution of weight outliers in Wolonn across layers, modules, and feature dimensions.
LLaMA-2-78 Layer-wise Attention Accumulation
Taken Distribution Attention Outliers LLaMA-2-78
LLaMA 2:78 Attention Outtier Raties per Head Across Layers
500
300
200
100
Layers
Jeles
(a) Which layers?
(b) Which keys?
(c) Which heads?
Figure Distribution of attention outliers in A across layers, keys and heads.
In summary, weight outliers in LLaMA2-7B are primarily located in the MLP's down-projection matrices, concentrated in specific shallow and deep layers.
3.3 WHERE ARE ATTENTION OUTLIERS LOCATED?
To understand the distribution of attention outliers, we analyze cumulative attention scores on the attention weights A across layers, keys, and heads.
Figure 6 highlights key patterns in the distribution of attention outliers. In Figure 6(a), the two largest cumulative attention scores per layer show that attention outliers persist across all layers. Figure ((() categorizes keys with outliers, indicating strong association with start tokens and weak semantic tokens. Figure (fc) visualizes score fluctuations across heads, revealing significant variation between heads and layers.
4 SYSTEMATIC OUTLIERS ARE SIMULTANEOUS AND INTERCONNECTED
Systematic outliers are not isolated phenomena; instead, they exhibit strong correlations across fea- ture and sequence dimensions well as layers. Understanding these interconnections and their lifecycle crucial for uncovering how outliers propagate through the model and affect computa- tions. This section analyzes these relationships in detail, laying the groundwork for the next section, where we hypothesize and validate their functional roles.
4.1 HOw ARE THESE OUTLIERS RELATED?
Table 1: Consistency of different types of outliers across dimensions.
Outlier Type
Outlier Type
Dimension
Consistency
weight outliers in Wddww weight outliers Woown activation outliers x down activation outliers in h
down activation outliers in x activation outliers h; feature
feature
100%
100%
activation outliers in h sequence
100%
attention outliers in A
sequence
95%
Table provides a quantitative analysis of the alignment among the three types of outliers. De- tailed experimental settings can be found in Appendix B.2 We find that the three types of outliers
5

---

Published as a conference paper at ICLR 2025
up_proj
up, out
/'77.
X
down. Ji
down out
layer out
rMK_
rms out
/T74%
/T74
x
x
down_pro
gate out
Figure 7: The emergence of activation outliers from weight outliers.
are not isolated but occur simultaneously and are interconnected across multiple dimensions within the model. Specifically, weight outliers align perfectly with activation outliers in feature dimen- sions, while over 95% of activation outliers overlap with attention outliers in sequence dimensions, primarily concentrated in start and weak semantic tokens.
4.2 THE LIFECYCLE OF SYSTEMATIC OUTLIERS
The correlations between different types of outliers suggest a deeper connection underlying their occurrences. By visualizing their lifecycle, observe chain of interactions: weight outliers lead to activation outliers, which then influence attention outliers, with this influence extending to non- outlier tokens.
The Emergence of Activation Outliers from Weight Outliers. In the second layer of LLaMA2- 7B's MLP, weight outliers the up- and gate-projection matrices cause extreme neuron responses. These amplified by the SiLU activation function (Elfwing tion (Shazeer 2020), resulting in activation outliers that are up to a thousand times the average mag- nitude. Additionally, weight outliers in the coon-projection matrix amplify activations in specific feature dimensions (e.g., 1415th and 2533rd), dominating the residual connection and influencing the final output (see Figure
al., 2018) and GLU opera-
value
mha,rmm mnha_rms_in In
mha rru__mms_ak rms_out out
query
attn Jogits
sttn_probs
RMS Nom
Figure The spread attention outliers from activation outliers. Activation outliers influence the self-attention mechanism, extending their impact other sequence dimensions.

---

Published as a conference paper at ICLR 2025
proj
up, out
1879
down
rmu in
rms, out
Nom
/774%
1T7
1174
gate proj
gate out
down proj
Figure 9: The disappearance of outliers in the final layers.
The Spread of Attention Outliers from Activation Outliers. Activation outliers propagate into the attention mechanism through their influence on query, key, and value vectors. In the third layer's Multi-Head Attention (MHA), we observe that tokens with activation outliers exhibit alignment in the 58th and 122nd dimensions of both query and key vectors. This alignment significantly increases the dot product between these vectors, leading to disproportionately high attention weights assigned to the outlier tokens (see Figure 8). As result, the attention mechanism focuses heavily on these tokens, amplifying their influence across layers.
Interestingly, despite receiving significant attention, the value vectors corresponding these tokens show comparatively smaller magnitudes. This indicates that while outlier tokens attract attention, they may contribute less directly the final output. This phenomenon could imply a mechanism where the model leverages these tokens as "anchors" to affect attention outputs, we explore it further in Section
Additionally, we find that this pattern allinmenn in query and key dimensions and reduced magni- tude in value dimensions-is consistent across most heads and layers. This consistency highlights the systematic nature of how activation outliers propagate their influence through the attention mech- anism.
The Disappearance of Outliers in the Final Layers. Outliers gradually vanish in the final layers due to cancellation by values of opposite signs. This neutralization occurs progressively rather than abruptly (see Figure 9). As activation outliers diminish, attention outliers are similarly reduced, with some heads in the final layer showing no outliers at all.
Summary. In the lifecycle of systematic outliers, weight outliers drive the emergence of activation outliers, which propagate anomalies into the attention mechanism. This interdependence extends their influence to non-outlier tokens. These findings reveal that systematic outliers are intrinsically linked the attention mechanism, setting the stage for the next section, where hypothesize and validate their functional roles.
5 SYSTEMATIC OUTLIERS AS CONTEXT-AWARE SCALING FACTORS IN ATTENTION MECHANISMS
5.1 HYPOTHESES THE ROLE OF SYSTEMATIC OUTLIERS
Based on the observations in Section we propose three hypotheses on the potential role of sys- tematic outliers in LLMs, drawing from prior research and our findings:
1. Fixed but Important Biases: Inspired by the concept of Massive Activations (Sun et al.) 2024), systematic outliers may act fixed biases that consistently influence model behav- 1or. These outliers could serve as stable values that help the model emphasize certain tokens features, regardless of the context.

---

Published as a conference paper at ICLR 2025
2. Context-Aware Biases: As seen in Figure Cc), the attention outliers vary significantly across heads and tokens (20% to 95%). This suggests that these outliers may dynamically adjust their influence based the input sequence, acting context-aware signals that adapt to specific content and guide attention allocation.
3. Context-Aware Scaling Factors: Figure X shows that the value vectors corresponding to outlier tokens have significantly smaller magnitudes, suggesting that these outliers may act as implicit scaling factors. By reducing the impact of contextual information on certain tokens, these scaling factors help minimize unnecessary updates.
5.2 EMPIRICAL VALIDATION OF SYSTEMATIC OUTLIERS HYPOTHESES
Formulation. In this part, we introduce five different attention formulations explore the role of systematic outliers. Each formulation represents a specific variant of the attention mechanism, designed to isolate different aspects of bias and scaling effects within the model. The formulations are listed in Table followed by an explanation of their roles.
Table 2: Variants of the attention mechanism for systematic outliers analysis.
ID
Attention Variant
Formulation
(a)
Default Atlention Vasswami 2017
AQt K. V)
softmux
QK" va QK" Vi
V
(b)
Explicit Fixed Bias
Attn(Q, K, Viv')
softmax
(c)
Explicit Context-Aware Bias
Attu(Q, K, V:k. v
softinax
QK" Vi
=soofmaxxxx
vil
v2
0"
(d)
Attention Bias
Artu(Q, K, V;K, v')
softmax
[[A'" k() vil
V
(c)
Explicit Context-Aware Scaling Factor
Attn(Q, K, V) S.()) ((z)--800) softmax
(QX" V77
In T Q, K, V Txx are the query, key, and value matrices, and is the dimensionality of the hidden space. For variants with bias or scaling modifications, k', k','' v E Ed represent the additional learnable bias terms, and (x)) is learnable scaling factor dependent on the input context.
Default Attention (a): The standard attention mechanism serves as the baseline, without
bias scaling modifications.
Explicit Fixed Bias (b): Adds fixed, learnable bias v' only to the value matrix to isolate the impact of fixed biases on systematic outliers.
Explicit Context-Aware Bias (c): Introduces context-aware bias terms k' and v', which vary based on the input sequence to isolate the impact of context-aware bias systematic outliers.
Attention Bias (d): Incorporates learnable bias terms k' and v' into the key and value matrices. It provides both context-aware bias and a sealing factor.
Explicit Context-Aware Scaling Factor (e): Utilizes a learnable scaling factor Sx(x) that dynamically adjusts the attention weights, helping investigate the scaling effect on reducing systematic outliers.
Results. We train five GPT-2 Radford al. 2019) models with these attention variants. Detailed experimental settings can be found in Appendix B.3 We visualize the presence of activation outliers across different attention formulations tudes for each layer in Figure II] The results reveal distinct patterns, where only attention bias (d) and explicit context-aware scaling factor (e) effectively prevent the formation of systematic outliers.
Figure 10 and plotted the top-3 largest activation magni-
These results strongly suggest that scaling factor plays crucial role in managing outlier behavior in L.Ms.. Fixed or context-aware bias alone is insufficient to mitigate outliers, whereas explicit context-aware scaling factor provides the necessary dynamic adjustment to prevent their occurrence.
The main conclusions about the role systematic outliers from the experiments are as follows:
It is not the fixed bias: Experiment (b) clearly shows that fixed bias alone does not prevent outliers, proving that fixed bias mechanisms ineffective in this regard.
8

---

Published as a conference paper at ICLR 2025
GPF2 Detault
GPF2 with Fixed Blas
GP12 Gertodtianare
GPT-2 with Attention Blas
Contact
14
(d)
(c)
Figure 10: Activation outliers across different attention formulations.
OPE2 Defsch
0PT2 with
OFT2 with Centret sware Bla
CPT2 with Abtuval Bls
(a)
(b)
(c)
(d)
(c)
Figure 11: Top-3 largest activation outliers for each layer.
It not the context-aware bias: Experiment (c) shows that context-aware bias alone is insufficient. A comparison with Experiment (d) highlights that context-aware biases do not eliminate outliers.
It is the context-aware scaling factor: Both Experiment (d) and Experiment (e) explicitly use a context-aware scaling factor. While Experiment (d) includes additional context-aware bias, the comparison between (c) and (d) confirms that the primary factor in preventing systematic outliers is the context-aware scaling factor.
5.3 FURTHER ANALYSIS OF SYSTEMATIC OUTLIERS
Softmax Attention is the root cause of systematic outliers. In Transformers, multi-head atten- tion (MHA) aims to augment token embeddings with contextual information to improve prediction accuracy. The difficulty of this task varies across tokens: some require complex contextual updates (e.g., for ambiguous semantically rich tokens), while others (e.g., delimiter padding tokens) quire minimal updates. However, the softmax operation enforces that attention scores always sum to one, even for simpler tasks where little contextual information is needed. To satisfy this constraint, the model must produce large dynamic range in the input to softmax, amplifying the disparity between tokens. This dynamic range is further exaggerated as the model learns allocate most attention to low- oow-infomaation tokens in some cases, ensuring minimal updates for those tokens.
This demand for a large dynamic range propagates through the network and affects earlier layers. Layer Normalization, applied before softmax, standardizes input distributions, inadvertently com- pressing the required dynamic range. To compensate, multi-layer perceptron (MLPs) preceding layers generate activations of significantly higher magnitude, resulting in the emergence of activation outliers. These high-magnitude activations propagate through the residual connections, amplifying gradients during backpropagation and encouraging the formation of weight outliers in the projection matrices.
Furthermore, because softmax outputs strictly positive and sum to one, all tokens maintain non- zero probabilities, leading to persistent gradients for all queries and keys. This forces the model to continuously adjust activations and weights accommodate large dynamic ranges, further ampli- fying systematic outliers over successive layers. As a result, softmax normalization, coupled with architectural constraints like Layer Normalization and residual connections, becomes the fundamen- tal driver of systematic outliers. Detailed derivations and analysis are provided in Appendix C
Potential Applications for Model Compression. Systematic outliers complicate compression techniques such as quantization and pruning by increasing memory usage and degrading perfor- mance. Our experiments demonstrate that context-aware scaling factors effectively mitigate these

---

Published as a conference paper at ICLR 2025
issues. Table compares GPT-2 Default and GPT-2 with Context-aware Scaling Factor under com- mon compression methods.
Table 3: Comparison of GPT-2 Default and GPT-2 with Context-aware Scaling Factor under pruning and quantization methods.
Model
PPL (FP16)
PPL (AbsMax W8)
PPI. (50 % Sparse)
GPT-2 Default
27.24
93.44
7235.68 39.47
GPT-2 + Context-aware Scaling
26.95
29.22
The results, tested on WikiTex12, show that context-aware scaling significantly improves robustness to compression. For quantization, it reduces PPL from 93.44 to 29. 29.22, and for pruning, stabilizes PPL at 39.47 compared to 7235.68 for GPT-2 Default. Importantly, the addition of context-aware scaling factors incurs minimal memory overhead. For example, GPT-2, the parameter count creases from 123.59M to 23.70M-1ee than a 0. 1% increase. This negligible overhead ensures that method remains practical for large-scale deployment. These findings validate that context-aware scaling enhances model robustness to compression, enabling efficient model deployment without compromising performance.
Influence on Convergence and Training Stability. Incorporating context-aware scaling factors significantly accelerates convergence and enhances training stability in Transformer models. By dynamically adjusting attention scores, these factors reduce reliance on extreme outliers, leading to smoother optimization landscape. As shown in Figure 12 models with context-aware scaling converge faster during early training steps, but their final validation losses are comparable to those of default attention mechanisms, indicating improved convergence speed without necessarily better final performance.
Influence on Convergence and Training Stability
Default Attention
5.5
Context-aware Scaling
Val Loss
3.5
3.0
40
120
160
200
Steps
Figure 12: Training loss comparison showing improved convergence by eliminating outliers.
CONCLUSION
In this work, we systematically analyzed the distribution, formation, and roles of outliers in large language models (LLMs), categorizing them into activation, weight, and attention outliers. Our find- ings reveal that these outliers are interconnected across layers and dimensions, stemming from the softmax operation the self-attention mechanism. Acting as implicit, context-aware scaling factors, these outliers dynamically adjust attention distributions, enabling the model to balance diverse con- textual demands. By climinating these outliers through explicit context-aware scaling, we showed improvements in model convergence and compression efficiency. Our approach helps reduce un- necessary attention allocation, making models more efficient and stable. This finding provides new insights into the internal workings of Transformer-basee LLMs and opens up avenues for refining attention mechanisms to improve performance and efficiency. We believe that our study not only deepens the theoretical understanding of outliers in LLMs but also has practical implications for the development of more efficient and robust language models.
ACKNOWLEDGEMENTS
This work was supported by National Key R&D Program of China under Grant No. No.20237D01204400 Beijing Municipal Science and Technology Project (7231 10000742300))
10

---

Published as a conference paper at ICLR 2025
Zhejiang Lab (No. 202IKHOAB07), and National Natural Science Foundation of China (Grant No. 62206290, 62276260, 62176254, 61976210, 62076235).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint MxXiv:3333.8872 2023.
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. The falcon series of open language models. arXiv preprint arXiv:2311.1686/ 2023.
Tom B Brown. Language models are few-shot learners. arXiv preprint
2020.
Together Computer. Redpajama: an open dataset for training large language models, 2023. URL aaaa-
https://gtt thub. wht.ub..omm/oogttteeccoooppeeeee
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 10818-30332 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh. Michael W Mahoney, Yakun Sophia Shao, Kurt Keutzer, and Amir Gholami. Kvquant: Towards 10 million context length Ilm inference with kv cache quantization. arXiv preprint arXiv:2401. (tt::2401./8777 18079, 2024.
Mojan Javaheripi and Sébastien Bubeck.
Phi-2:
The surprising power of small lan- com/ http://///o/eeo////pa/aaaa///ooa www.ooooftccm//n--naaat research/blog.
guage models, 2023. URL
Accessed: 2024-
10-01.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Andrej Karpathy. Nanogpt. 2024-11-24.
2023. Accessed:
Kenneth Keene. A github link forked from jahang38/tnnllamma.
https::/github. com/
2024. Accessed: 2024-11-24.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.069900 2021.
Baohao Liao and Christof Monz. Is free lunch for removing outliers during pretraining? arXiv preprint 2xXiv:2402.12100 2024.
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quan- tized llss. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Ruikang Liu, Haoli Bai, Haokun Lin, Yuening Li, Han Gao. Zhengzhuo Xu, Lu Hou, Jun Yao, and Chun Yuan. Intactkv: Improving large language model quantization by keeping pivot tokens intact. arXiv preprint arXiv:2403.01244 2024.

---

Published as a conference paper at ICLR 2025
MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable 1ims, 2023. URL htllhhtps://// htpp:://wwwr dat aeatabricks wtt..kk//..ooo//oommmoccccccccc. 7b Accessed: 2024-10-01.
Aniruddha Nrusimha, Mayank Mishra, Naigang Wang, Dan Alistarh, Rameswar Panda, and Yoon Kim. Mitigating the impact of outlier channels for language model quantization with activation regularization. arXiv preprint arXiv:2404.003600 036000 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Davide Paglieri, Saurabh Dash, Tim Rocktäschel, and Jack Parker-Holder. Outliers and calibration sets have diminishing effect quantization of modern Ilms. arXiv preprint arXiv:2405 2024.
Giovanni Puccetti, Alessio Miaschi, and Felice Dell' How do bert embeddings organize linguistic knowledge? In Proceedings of deep learning inside out (DeeLIO): the 2nd workshop on knowledge extraction and integration for deep learning architectures, pp. 48-57, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jason Ramapuram, Federico Danieli, Eeshan Dhekane, Floris Weers, Dan Busbridge, Pierre Ablin, Tatiana Likhomanenko, Jagrit Digani, Zijin Gu, Amitis Shidani, et al. Theory, analysis, and best practices for sigmoid self-attention. arXiv preprint arXiv:2409.0443) 2024.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXlv:2002.05202, 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17700 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
A Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. Advances in Neural Information Processing Systems, 35:17402-11 1444-114144, 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference Machine Learning, pp. 38087-38099, PMLR, 2023a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. /7453, 2023b.
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (owl): missing secret sauce for pruning Ilms to high sparsity. arXiv preprint arXiv:2310.05115. 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et Opt: Open pre-trained transformer language models. arXiv preprint w2Xiv:205.00066 2022.
Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, and Xuanjing Huang. Unveiling linguistic regions in large language models. arXiv preprint 2402 14700, 2024.
12

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RESULTS ON SYSTEMATIC OUTLIERS IN LLMS
This section extends the analysis systematic outliers in LLMs, complementing the findings pre- sented in the main paper. We provide additional results on both pretrained LLMs (Appendix (A.I) and fine-tuned variants (Appendix [ highlighting the consistency and variation in outlier behav- ior across different model families and training paradigms.
A.1 PRETRAINED LLMS
In Section identified systematic outliers in models such as LLaMA2-7B. To further validate these findings, we extend our evaluation to a broader set of pretrained LLMs, including Phi-2 (Java-
heripi & Bubeck 2023), Mistral-7B (Jiang al. 2023), LLLMA2-11B. LLaMA3 (Dubey et al.
2024). OPT-6.7B (Zhang et al. 2022), MPT-7B (MosaicMI 2023). and Falcon-7B (Almazrouei
al.
2023 The corresponding results depicted in Higure 16 18 and 19
Our analysis reveals several key insights:
Systematic outliers consistently appear across all evaluated models, with patterns similar to those observed in Section 3
Consistency of Outliers:
Influence of Model Architecture: The design of the MLP architecture and the choice of ac- tivation function significantly impact the distribution of weight outliers and activation outliers. For example, OPT-6. 7B, which employs two-layer linear structure with ReLU activation, ex- hibits more dispersed pattern of outliers compared the compact clustering seen in models like LLaMA2-133 13B and Mistral-7B.
Layer-Specifie Trends:
In deeper layers, particularly in LLaMA3 and Falcon-7B, outliers in down- cown-projection activations more pronounced, suggesting that architectural modifications in newer models can amplify outlier formation in specific layers.
These results demonstrate that systematic outliers are a pervasive phenomenon across diverse model families and architectures.
A.2 FINE-TUNED LLMs
Beyond pretrained models, fine-tuning plays crucial role in adapting LLMs for specific tasks such as instruction following or conversational applications (Ouyang et al., 2022). To assess the impact of fine-tuning systematic outliers, we analyze fine-tuned variants of LLaMA2 and Mistral, with
results shown in Figures 20 21 and 22
Persistence of Outliers: Systematic outliers persist after fine-tuning, with their magnitudes and distributions largely unchanged from the corresponding pretrained models. For instance, Figures 14 and 22 demonstrate that outlier patterns in Mistral-7B are consistent with its fine- tuned counterpart, Mistral-7B Instruct. Similarly, Figures ?? and 20 show that fine-tuning LLaMA2-7B into LLaMA2-7B-CLS introduces minimal changes to outlier locations and dis- tributions.
Effects: Instruction fine-tuning does not significantly alter the structural patterns of systematic outliers. Although minor variations in magnitudes or attention scores are ob- served in some layers, such as reduced attention outlier intensities in Mistral-7B Instruct, these adjustments do not affect the overall spatial or dimensional concentration outliers.
Generalizability:
The consistent presence of systematic outliers across both pretrained and fine-tuned models indicates a structural origin rooted in the Transformer architecture, rather than task-specific artifacts introduced during fine-tuning. This highlights their intrinsic nature and robustness to different training paradigms.
These results highlight the robustness of systematic outliers across both pretrained and fine-tuned models, emphasizing their structural roots within Transformer architectures.
13

---

Published as a conference paper at ICLR 2025
Layer?
Phi-2,
Phi-2, Layer
warm
col
summer
Samune
(a) activation: down
(b) weight: W
(c) activation: hg
(d) attention: A
Figure 13: Systematic outliers Phi-2.
Mistral-78. 78, Layer
Layer
Mistral-78.
Mistral-78, Layer 3
00
50
0.4
cold
0.2
Stont
worn
Wodn
7310
1070
Sum mof
War
(a) activation: X²
down
(b) weight: Walwss
(c) activation: h
(d) attention: A
Figure 14: Systematic outliers in Mistral-7B.
LLaMA-2-138. Layer
LLaMA-22 2-138,
LLaMA-2-138, Layer
warm
Winder
6763
Exmomer
Worerr
Summer
2100
(a) activation: Xt
Jown
(b) weight: Wddws
(c) activation: he
(d) attention: A
Figure 15: Systematic outliers in LLaMA2-13B.
LLAMA-3-88, Layer
Layer
LLaMA-3-38,
Summer
0.8
warm
Winter
Sommer
Summer
wem
2327
2284
(a) activation: x
down
(b) weight: W
(c) activation: by
(d) attention: A
Figure 16: Systematic outliers in LLaMA3-8B.
B DETAILED EXPERIMENTAL SETTINGS
This section provides comprehensive details on the experimental settings used to analyze systematic outliers in LLMs. It covers the methodologies and configurations employed in different aspects of the study, ensuring reproducibility and clarity. The following subsections provide detailed descrip- tions of the experimental setups for each analysis.
14

---

Published as a conference paper at ICLR 2025
GPT-78, Layer?
OPT-78,
OF77-78, Layer
OP177, Layer
warm
Winter
2M
Summer
Woter
(a) activation: X
weight: W
(c) activation: h
dd attention: A
Figure 17: Systematic outliers in OPT-6.7B.
MPT-78, Layer
Layer
MP 1-70 Layer
wer
0.6
0.4
col
4.2
Sommnn earrs
Summa
(a) activation: x;
down
weight: Wdoss
(c) activation: h
(d) attention: A
Figure 18: Systematic outliers in MPT-7B.
Falcon-7B, Layer
Palcon-7B, 7B,
Falcon-79, Layer
Summer
Falcon-78, Layer
1.5
0.5
0.5
Winter
0.4
02
(a) activation: X
down
(b) weight: Widows
(c) activation: he
(d) attention: A
Figure 19: Systematic outliers in Falcon-7B.
LLaMA-2-78 2-78 chat, Layer
LLaMA-2 2-7B--tt
LLaMM-2-7B-dhat, Layer
Summnr
LLaMA 2: 2-78-chtt, Layer
0.8
Wirter
0.4
9.
7890
2333
Wenter
esm
1435
(a) activation: down
(b) weight: Wddoa
(c) activation: he
(d) attention: A
Figure 20: Systematic outliers in LLaMA2-7B-
B. POSITION ANALYSIS SETTINGS
This subsection describes the experimental settings used to analyze the distribution systematic outliers in LLaMA2-7B across layers, sequences, and feature dimensions. The focus is three types of outliers activation outliers, weight outliers, and attention outliers, with results pre- sented in Figures
5 and
For activation outliers in h, (layer outputs) and xdoon (down- projectio inputs), we analyze the top- 3 largest activation values and the median activation value for each layer to examine how outliers
15

---

Published as a conference paper at ICLR 2025
LLMMA-2-33B-hMa Layer
LLaMA-2-1BB-chat, Layer
LLaMA-2-13B-chal
LLaMA-2-13B-C- Layer
Summer
5.5
Winter
0.4
6743
2100
wm0
Wontee
(a) activation: x
(b) weight: Waloom
(c) activation: h
dd attention: A
Figure 21: Systematic outliers in LLaMA2-13B-Chat
Mistral-7B-Instruad Layer
Mistral-78 Instruct, Layer
Mistral-7B-insttee Layer
Layer
1.0
mer
15
0.8
cold
0.2
wort
What!
2070
Sum met
oud
(a) activation: xdown
(b) weight: W
(c) activation: he
(d) attention: A
Figure 22: Systematic outliers in Mistral-7B-Instruet.
distributed across shallow, middle, and deep layers. Using 100 sequences of length 2,048 from the RedPajama dataset (Computer 2023), the positions where activation outliers first appear are identified. Additionally, scatter plots visualize the channel indices where activation outliers occur, highlighting fixed feature dimensions associated with these outliers.
For weight outliers in W (down-projection matrices), compute the extremal ratio for each layer and module to measure how concentrated the largest weights are in specific columns.
For attention outliers in A (attention weights), analyze the top-2 cumulative attention scores and the median score for each layer to track how attention outliers persist across layers. Sequence positions where attention outliers first appear are identified using the same set of 100 sequences. Finally, the mean, upper limit, and lower limit of attention outlier scores are computed for different heads across layers to reveal head- specific patterns.
These settings provide a systematic approach to understanding the spatial and dimensional patterns of systematic outliers in I.LAMA-7BB.
B.2 CONSISTENCY ANALYSIS SETTINGS
To analyze the consistency between different types of outliers, calculate the alignment of activa- tion and attention outliers across 100 randomly selected samples from the RedPajama dataset (Com-) puter 2023). Each sample has a sequence length of 2,048, and attention outliers analyzed sepa- rately for each attention head.
The overlap between activation and attention outliers is defined the percentage of sequence indices where activation outliers in h E IR regllennxdaa diiate align with attention outliers in A₁ e IR seglenn where A is the cumulative attention score matrix for head i at layer f. For each sample and attention head, overlaps are computed as:
O Octiivation n O
Overlap'
attection
O activation
16

---

Published as a conference paper at ICLR 2025
where activation and O attention the sets of sequence indices corresponding to activation and atten- tion outliers, respectively. The overall overlap across samples samples and Rheaas attention heads is then averaged as:
Veaniles Piscah
Overall Overlap
Overlap'.
Rsamples
Tbbcads
These settings systematically quantify the alignment between activation and attention outliers, of- fering detailed insights into their interconnected nature, as summarized in Table
B.3 GPT-2 ATTENTION VARIANT TRAINING SETTINGS
We utilize the open-source GPT-2 implementation from the NanoGPT repository (Karpathy) 2023), following the default recommended training setup and optimizer settings. Each of the five GPT-2 variants was trained for 50,000 iterations, processing approximately billion tokens total. For attention bias variant, followed the initialization method proposed by (Sun et al. 2024). setting k' and to (0, 0.02I).
B.4 MODEL COMPRESSION EXPERIMENT SETTINGS
To evaluate the robustness of context-aware scaling factors under compression, we conducted exper- iments on GPT-2 models using two common methods: quantization and pruning. For quantization, 8-bit weight quantization was applied using the AbsMax scaling method, which normalizes weights by their maximum absolute value. For pruning. we performed unstructured magnitude pruning, removing 50% of the smallest-magnitude weights across all layers.
Both models were evaluated on the WikiText2 dataset using perplexity (PPL) as the primary metric, comparing GPT-2 Default and GPT-2 with context-awars scaling factors. These settings were chosen to test the models' ability to maintain performance under aggressive compression techniques.
C How SOFTMAX CAUSES SYSTEMATIC OUTLIERS IN TRANSFORMER MODELS
The formation of systematic outliers in transformer models stems from the inherent characteristics of the softmax operation within the self-attention mechanism. While capturing the complete dy- namics of outlier emergence requires a complex understanding of training processes, we provide a mathematical analysis that outlines the logical connection between softmax and the appearance of systematic outliers. This analysis reveals how the interaction between softmax and model ar- chitecture propagates and localizes these anomalies. The following sequence summarizes the key steps:
Necessity of Zero-Update in MHA: Certain tokens, such as initial tokens weakly semantic tokens, often require minimal contextual updates. The Multi-Head Attention (MHA) mechanism addresses this by dynamically suppressing updates for these tokens, placing strict constraints on gradients and weights.
2. Softmax-Induced Dynamic Range Expansion: Achieving the zero-update behavior requires softmax to focus attention weights on limited number of keys. This necessitates substantial differences in the dot products of query and key vectors, leading to extreme dynamic ranges in the attention scores.
3. Propagation of Systematic Outliers: The extreme attention scores propagate anomalies through transformer computations:
In MHA, the shared projection weights for keys (Wx) and values (Wv) accumulate steep
gradients, introducing activation anomalies.
In the Multi-Layer Perceptron (MIP),, these activation anomalies are amplified by the projec- tion layers, resulting in systematic outliers.
4. Localization of Outliers: Outliers concentrate at specific tokens and feature dimensions:
17

---

Published as a conference paper at ICLR 2025
Token-Level: Initial tokens (e.g., (CLS)) and weakly semantic tokens exhibit pronounced updates due to their unique roles in aggregation and suppression.
Channel-Level: Outliers appear in limited set of feature dimensions, reducing their impact on other parts of the model but exacerbating localized anomalies.
5. Why Earlier Layers Avoid This: Early transformer layers focus on distributing token infor- mation uniformly, maintaining balanced attention distributions. Systematic outliers predomi- nantly emerge in deeper layers, where higher-level semantic differentiation sharpens the atten- tion mechanism.
This systematic analysis reveals that the softmax operation, in combination with architectural straints, is a central factor driving the emergence and propagation systematic outliers. The subse- quent sections delve into the specific mechanisms behind these phenomena, starting with the neces- sity of zero-update behavior self-attention.
C.I NECESSITY OF ZERO-UPDATE IN MULTI-HEAD ATTENTION (MHA)
In transformer models, the Multi-Head Attention (MHA) mechanism dynamically adjusts token representations based on contextual information. However, certain tokens-such as initial tokens (e.g., [CLS]) or weakly semantic tokens (e.g., punctuation marks) require minimal updates during training. For these tokens, the desired behavior is near-zero update:
Ar MHA(QZ: K, K)V) x 0.
Achieving this condition imposes constraints the gradients and attention weights. The softmax normalization within MHA must allocate most of the attention probability to a few keys with negligi- ble values, effectively canceling the contributions of the remaining keys. The output of the attention mechanism is given by:
QQKT
MHA K, V) Softmax
Vdz.
where Q, K, are the query, key, and value matrices, and de is the dimensionality of the keys. For given query Qa. the attention weights must satisfy:
QuKgT Vile
Softmax
=
To approximate zero zero-updan perote, the weighted sum of Vg values must effectively cancel out. This ne- cessitates a highly concentrated attention weight distribution, which in turn creates large disparities in the query-ke) dot products K)
This zero-update requirement, while essential for certain tokens, introduces challenges:
Dynamic Weight Adjustment: The softmax must focus the attention weights on specific keys, requiring the associated dot products (Q,K²) to dominate.
Gradient Amplification: The backpropagation process must enforce selective updates to queries (Qz) and keys (K,), leading to steep gradients.
This forms the foundation for the emergence of extreme values in both the attention weights and the gradients, as discussed in the following section.
C.2 SOFTMAX-INDUCED DYNAMIC RANGE EXPANSION
The softmax operation, central to the self-attention mechanism, inherently expands the dynamic range of attention scores. This behavior is critical for satisfying the zero-update condition for certain tokens but simultaneously leads the emergence of extreme values.
18

---

Published as a conference paper at ICLR 2025
The softmax operation is defined as:
Q.K.
QiKy vdk
exp
vidy
A.j Softmax
2.Kg
R=1
exp
vdl
where Ai represents the attention weight assigned key j by query i, and the sequence length.
To achieve zero-update for a token I, the softmax must concentrate the attention weight Arj on specific keys j while suppressing the weights for others. This requires the dot product R=K,- to
vdd
significantly dominate over other terms
Mathematically. this results in: ddc
Jakj.
QKKT vd
vdk
As a consequence, the ratio of exponential terms grows exponentially with the difference in dot products. For example, if 2.K,- M and Q-KI 0 for k + j", the resulting attention weight is: Vdz
Vdt
Azj- exp(M) cxp(M) + (n (n
R
as M - co.
In contrast, the weights for other keys approach zero:
Azk cxp(M) + (n-1) for k + j
This dynamic range expansion, driven by the softmax operation, imposes two key challenges:
Extreme Values in Attention Scores: The attention weights for dominant keys grow exponen- tially, while others become negligible. This leads highly imbalanced attention distributions.
Gradient Amplification: Backpropagation through the softmax induces steep gradients for dominant keys, amplifying updates to the query (Q) and key (K) vectors. This can destabilize the training process.
These extreme values propagate through subsequent layers, contributing to the formation of system- atic outliers both activations and weights, as detailed the next subsection.
C.3 PROPAGATION OF SYSTEMATIC OUTLIERS
Systematic outliers, once introduced by the softmax mechanism, propagate through the transformer layers due to shared weights and non-linear transformations in the Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP) components.
Impact in MHA. The shared projection weights in MHA specifiic,,,, the key and value matrices Wx and Wy Wy-magnffy the effect of extreme attention scores. The query (Q), key (K), and value (V) matrices computed as:
Q = =LN(h:)Wo
K = LN(h)WKK = =N
where h represents the layer's imput, and LN is layer normalization. When the attention scores A are dominated by a few keys, the output of MHA focuses heavily on the corresponding values V,. The MHA output for token x: is given by:
19

---

Published as a conference paper at ICLR 2025
MHA(2) = A.yg.
where Azj concentrates on a few dominant keys. This imbalance introduces large updates to the projection weights WK and Wv during backpropagation, the gradients for these weights are computed from Vw I. and V Vw, respectively.
Amplification in MLP. Following MHA, the output passes through the MLP block, which con- sists of up-projection (Wap), non-linear activation (σ)) and down-projection (Wanwn):
W dowwn of (W LN(2
dowwn
(he( + MHA(LN)(he). MHA(z) produces extreme values due to attention where h2+1/2 outliers, these anomalies further amplified by the non-linear activation (a) and concentrated in the down-projection weights (W dowa).
Emergence of Outliers. The combination steep gradients and non-linear transformations re- sults in the emergence of outliers in both activations and weights. Key observations include:
These appear in the intermediate representations he-1, primarily concen- trated in specific sequence positions and feature dimensions.
Activation Outliers:
Weight Outliers: The steep gradients for WK. Wy, and W Wdowa lead to concentrated large weights in these matrices, reinforcing the cycle of extreme values.
This propagation mechanism underscores the role of softmax-induee dynamic range expansion in perpetuating systematic outliers throughout the transformer layers.
C.4 LOCALIZATION OF SYSTEMATIC OUTLIERS
Systematic outliers are not randomly distributed but exhibit specific patterns of localization across tokens, layers, and feature dimensions. This section explores how these outliers concentrated at particular positions and channels, minimizing their overall disruption while fulfilling the model's dynamic range requirements.
Token-Level Localization. Outliers are predominantly associated with specific tokens, such as initial tokens (e.g., [CLS]) ) or tokens with weak semantic content (e.g., punctuation marks). These tokens particularly susceptible outliers due their roles in aggregating sequence information or carrying minimal intrinsic meaning. For instance:
Initial Tokens: Initial tokens aggregate global information, receiving disproportionately high attention scores, which amplifies their values in the MHA output.
Tokens like or often have low intrinsic information, leading the model to assign high attention scores to stabilize their contextual representation. This results in exaggerated updates during training.
Weak-Semantics Tokens:
Channel-Levee Localization. Outliers feature dimensions are typically confined to a small sub- set of channels. This sparsity arises because the model prioritizes containing the impact of extreme values to few dimensions rather than spreading them across the entire representation. Key charac- teristics include:
Fixed Dimensions:
Outliers are observed in specific rows or columns of weight matrices (e.g.. Waown) across layers, suggesting a structural origin.
Robustness Preservation: By localizing extreme values to a few channels, the model ensures that most dimensions remain stable, preserving the robustness of the overall representation.
20

---

Published as a conference paper at ICLR 2025
C.5 CONCLUSION
The mathernatical analysis presented in this section demonstrates how the softmax operation in the self-attention mechanism is key driver of systematic outliers in transformer models. By fulfilling the zero-update requirement for certain tokens, softmax induces extreme disparities in attention scores, leading steep gradients and the emergence of outliers. These anomalies propagate through transformer layers, being amplified by shared projection weights and non-linear activations in the MLP, ultimately manifesting as systematic outliers in both activations and weights.
Moreover, these outliers exhibit distinct localization patterns, being concentrated at specific tokens and feature dimensions. This localization minimizes their overall disruption the model while fulfilling the dynamic range demands imposed by the softmax mechanism. Understanding these dynamics offers valuable insights into the structural origins of systematic outliers, paving the way for mitigation strategies such explicit context-awan scaling factors to prevent their formation and improve model robustness.
D MORE ANALYSIS FO SYSTEMATIC OUTLIERS
D.1 ABSENCE OF OUTLIERS IN SIGMOID ATTENTION
In Ramapuram al. (204). sigmoid self-attention was proposed as an alternative traditional Nattmax-basedd attention, formulated in [quation Unlike softmax, sigmoid attention independently maps each attention score a value between 0 and introducing fixed bias term b adjust the sigmoid function's 's activation.
SigmoidAttn (X) = a (QK*) /Vdgk) V. with u sigmoid + b) == (1 e ((ath)
(I)
A key distinction of sigmoid attention is its ability to output near-zero values for certain tokens. While this can lead to vanishing gradients for some inputs, it also eliminates the extreme dynamic range caused by softmax normalization, thereby mitigating the formation of systematic outliers.
To evaluate its impact, we trained a GPT-2 model with sigmoid attention using the same experi- mental setup described in Appendix B.3 Figure 23 illustrates that sigmoid attention successfully eliminates systematic outliers. This finding reinforces our hypothesis that softmax normalization is a primary cause of outliers in self-attention mechanisms.
GPT-2 with Sigmoid Attenion
2k
1k
Summer
warm
is
Winter
cold:
Figure 23: Absence of outliers in sigmoid attention.
D.2 EXPLICIT CONTEXT-AWARE SCALING FACTOR IN TINY-LLAMA
To verify the generalizability of our findings beyond GPT-2, we conducted additional experiments using a TinyLLaMA-15 120M model. The training setup and code were adapted from the implementation of TLMA (Keene) 2024), which provides an efficient framework for pretrain- ing Transformer models.
21

---

Published as a conference paper at ICLR 2025
In this experiment, replaced the standard self-attention mechanism in TinyLLaMA with the "ex- plicit context-aware scaling factor" variant. As shown in Figure 24 the results were consistent with those observed for GPT-2: systematic outliers were completely eliminated, confirming the effective- ness of the explicit context-aware scaling factor.
This finding further corroborates the analysis presented in the main paper. Standard attention mech- anisms often require (HA(z) R 0 updates for certain tokens during training. Achieving this with coftmax-basee attention induces extremely large gradients, which lead the formation of outliers in activations and weights. By contrast, the explicit context-aware scaling factor achieves the same updote objective without generating large gradients, thus avoiding outlier formation. These results demonstrate the robustness of the proposed approach across different Transformer architec- tures.
TinyLLaMA with Context-aware Scaling Factor
2k
1k
Summer is
warm
Winter is
cold
Figure 24: Visualization TinyLLaMA-120M with explicit context-aware scaling factor. The absence of systematic outliers illustrates the effectiveness of this approach in preventing outlier formation.
D.3 IMPACT OF SEQUENCE LENGTH ON OUTLIERS
Our analysis shows that sequence length does not affect the existence of attention outliers, but does influence their specific positions within the sequence. Outliers consistently appear at the first tokens and semantically weak tokens, with their relative positioning shifting as the sequence length changes. The observations are summarized as follows:
Short Sequence: In the sequence "Summer is warm!", attention outliers occur "Summer".
Moderately Extended Sequence: Extending the sequence to "Summer is warm! Winter is cold." introduces an additional outlier at "." (the first period), alongside "Summer".
Further Extended Sequence:
the sequence "Summer is warm! Winter is cold. Spring is good.", outliers remain at "Summer" and the first period, demonstrating that outlier diversity does not increase with sequence length.
Modified Sequence: Modifying the sequence "Summer is warm! Winter is cold! Spring is good." shifts the outlier from the first period to the last period, while "Summer" and weak semantic tokens remain consistent outlier locations.
These examples suggest that while sequence length can shift the positions of outliers, their pres- ence is robust across varying lengths. Notably, there is no evidence that specific sequence lengths are more prone to generating outliers. Instead, outliers are influenced by token-level semantics, consistently favoring the first tokens and semantically weak tokens, regardless of sequence length.
22


Reviews:
[{"review_id": "QPxOYHvPc4", "reviewer": "Reviewer_85qx", "summary": "This paper presents a systemic analysis of outliers in transformer language models, which as extremely large values in weights and or activations. Through a set of experiments they show how different types of activations are connected to each other, and how they are all connected to attention scores in the end. In particular, the authors propose outlier activations are used to scale attention activations and allow for a 'zero' update when there is no need to update the residual stream. They verify this hypothesis by training transformer models with different transformer setups and show that including a specific scale parameter removes the existence of outliers from the model.", "questions": "Couldn't the model alternatively learn to do ~0 updates by outputting a value matrix V with all the values small? Do you have any hypothesis why current models instead learn to use attention with an outlier connected to a small vector in V instead of making all vectors in V small?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Very well written. Presents a nice and easy to follow story of a complex topic. Good figures.\n- Good experiments to analyze the situation and support their hypothesis well\n- Provides a good explanation for a mysterious and sometimes troublesome behavior observed in transformer models.\n- Proposes a (few) modified architectures that solve this problem", "weaknesses": "- Notation is a little different from what I'm used to. I think it would be more clear to refer to MLP-layer, Attention layer and residual stream in Figure 4. In particular, down projection input is discussed early on in the paper without explaining what it is, would be useful to at least refer to Fig.4 when its first mentioned.\n- Could use some more experimental details at least in the appendix to explain the experimental setups etc in more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iZ5cWV7tUX", "reviewer": "Reviewer_hfXR", "summary": "The paper systematically analyzed the weight / activation / attention outliers in Transformer-based LLMs and find that they are correlated and are related to the design of the self-attention mechanism. The Transformer is trained to have these outliers because the model needs to learn the implicit context-aware scaling factors. After adopting a variant of self-attention that incorporates an explicit context-aware scaling factor in GPT-2, these outliers will disappear. To solidify the finding, the author also trained a GPT-2-sized model with the sigmoid attention, and showed that it does not have the outlier problem.", "questions": "Have you tried changing the self-attention in Llama to the \"explicit context-aware scaling factor\" variant? Will it also remove the outliers?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper provided an empirical analysis of the outliers in LLMs's weights / activations and attention outputs. The author first analyzed the outliers in different LLM layers and found that they are highly correlated. This leads to three hypotheses about these outliers: 1) they act as fixed but important biases, 2) they act as context-aware biases, 3) they act as context-aware scaling factors to the attention mechanism. The author trained GPT-2 with different attention variants to verify these hypotheses. The experimental results suggest that the outliers should be acting as implicit context-aware scaling factors. The reasoning process of the paper is clear and convincing. The self-attention with explicit context-aware scaling factor can also stabilize LLM training.", "weaknesses": "The author claims that the work \"deepens the theoretical understanding\" of outliers in LLMs in the Conclusion section. However, there is no theory involved in the analysis and the finding is mostly empirical. On the other hand, the author only conducted experiments with GPT-2. Since the Llama architecture is not exactly the same as GPT-2, the author can also verify the finding with small-scale Llama model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LzjWnFNnft", "reviewer": "Reviewer_pdDU", "summary": "This paper presents an investigation into systematic outliers in Large Language Models (LLMs), categorizing them into three types: activation outliers, weight outliers, and attention outliers. The authors analyze their distribution patterns, lifecycle, and potential role in the attention mechanism. They propose that these outliers function as implicit context-aware scaling factors and suggest modifications to the attention mechanism to address them. The paper includes empirical analysis across several LLM architectures and proposes potential improvements for model convergence and compression.", "questions": "Methodological Clarity:\n\n1.\tGiven that outlier identification is central to your analysis, one would expect quantitative definitions of outliers? What specific statistical thresholds or metrics have been proposed by you for systematically identifying each type of outlier (activation, weight, attention)?\n2.\tYour analysis of the \"lifecycle\" of outliers suggests causal relationships between different types of outliers. How can you establish these relationships are truly causal rather than merely correlational? What controlled experiments validate these claims?\n\nTheoretical Foundation:\nYour hypothesis about softmax attention being the root cause:\n\n•\tCan you provide a mathematical proof linking softmax properties to outlier formation?\n•\tHave you considered alternative mechanisms beyond just sigmoid?\n•\tHow does this hypothesis explain the layer-wise variation in outlier patterns?\n\nQuantitative Memory Efficiency: \n\nWhat is the quantitative impact on memory consumption when using context-aware scaling compared to baseline models? Providing precise memory benchmarks would clarify its effectiveness for large-scale deployment.\n\nEffect of Sequence Length: \n\nHow does sequence length influence attention outliers? Do variations in sequence length amplify or suppress these outliers, and which sequence lengths are most prone to generating them?\n\nAfter reviewing the revised paper and updates, I find that some of my questions have been partially addressed. However, the evaluations rely on relatively outdated LLMs, raising concerns about whether these results would hold true for current state-of-the-art LLMs. As a result, I remain unconvinced of the impact of this primarily empirical work in the context of today's rapidly evolving LLM landscape. Thus, I will maintain my score.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tAddresses an important topic in LLM research with potential practical implications for model optimization\n2.\tProvides comprehensive visualization of outlier patterns across different model architectures\n3.\tMakes an attempt to connect different aspects of model behavior (outliers, attention mechanism, model performance)\n4.\tIncludes analysis across multiple popular LLM architectures (LLaMA2, Mistral, Phi-2)\n5.\tThe paper's exploration of outlier lifecycles offers an interesting perspective on how these patterns emerge and evolve", "weaknesses": "• Unclear Research Focus and Scattered Investigation:\nThe paper suffers from a lack of clear research direction and keeps shifting between multiple topics without thoroughly investigating any single aspect:\n  - It starts by identifying three types of outliers (activation, weight, and attention outliers) but doesn't provide a rigorous mathematical definition of what constitutes an \"outlier\" in each case\n  - The investigation jumps from outlier identification to lifecycle analysis to attention mechanisms without establishing strong connections between these aspects\n  - Section 5's transition from outlier analysis to attention mechanism modification feels abrupt and inadequately motivated\n\n• Empirical Weaknesses and Methodological Issues\n  - The paper relies heavily on empirical observations without sufficient statistical rigor:\n\n         # The identification of outliers appears to be based purely on visual inspection of heatmaps (Figures 1-3) without any quantitative thresholds or statistical measures\n         # The claim about \"95% overlap\" between activation and attention outliers (Table 1) lacks details about the methodology used to calculate this overlap\n         # The paper doesn't provide error bars or statistical significance tests for any of its quantitative claims\n\n• Previously Known Results Presented as Novel \n  - All of these papers are cited, yet several of the paper's \"findings\" have been previously established in the literature:\n\n        # The presence of activation outliers and their impact on model compression was already documented by Dettmers et al. (2022)\n        # The \"Attention Sink\" phenomenon and its relationship to specific tokens has been thoroughly analyzed by Xiao et al. (2023b)\n        # The connection between outliers and layer sparsity was previously established by Yin et al. (2023)\n\n• The experimental validation of key claims is often insufficient:\n  - The paper proposes five attention variants (Table 2) but doesn't provide comprehensive ablation studies\n  - The convergence improvements claimed in Figure 15 are shown for only 50 steps without baseline comparisons\n  - The proposed context-aware scaling mechanism is not thoroughly evaluated against existing solutions\n\n• Unsupported Claims and Logical Gaps\n  - Several key claims lack proper substantiation:\n\n        # The paper asserts that systematic outliers serve as \"implicit context-aware scaling factors\" but doesn't provide a mathematical proof or rigorous demonstration\n        # The connection between softmax attention and the emergence of outliers (Section 6) is speculative and lacks any formal analysis\n        # The claim about improved model compression is made without quantitative comparisons to existing compression techniques, and already well established in literature.\n\n• Limited Scope of Analysis\n  - Despite claiming to provide a \"systematic\" analysis, the investigation of fine-tuned models is superficial, only looking at surface-level patterns\n\nDettmers et al. (2022) - \"GPT3.int8(): 8-bit Matrix Multiplication for Transformers at Scale\" - observed activation outliers and proposed a mixed-precision decomposition quantization scheme to mitigate their effects.\n\nXiao et al. (2023) - \"Efficient Streaming Language Models with Attention Sinks\" - discovered the 'Attention Sink' phenomenon, where a disproportionate amount of attention is focused on a few keys, which led them to propose the StreamingLLM framework.\n\nYin et al. (2023) - \"Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity\" - demonstrated a strong correlation between the distribution of outliers and layer sparsity, indicating that outliers complicate pruning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L6JSXd8Htz", "reviewer": "Reviewer_1brf", "summary": "The paper presents an empirical study of activation, weight, and attention outliers in LLMs. The authors investigate their roles and functions, uncovering important findings that could potentially deepen the understanding of LLMs.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper investigates two important questions: where do outliers exist in LLMs, and what roles they play. The experiments are comprehensive and detailed.", "weaknesses": "1. While the experimental results are detailed, they are largely empirical, which raises concerns about the paper's technical novelty. For EMNLP, this empirical focus might be a better fit. However, for ICLR, a stronger mathematical analysis would be beneficial, such as a more in-depth exploration of the roles of these outliers.\n\n2. The paper lacks comparison with existing methods. For example, the authors suggest that their findings could be used for pruning, but they should include comparisons with existing pruning methods for LLMs to substantiate this claim.\n\n3. What is the relationship of this work to Sun et al. [1]?. Is this study an extension from focusing on massive activations to also examining massive weights, activations, and attention?\n\nDisclaimer: Weakness 1 (W1) is the primary reason for my hesitancy in assigning a higher rating.\n\n[1] Sun, Mingjie, et al. \"Massive activations in large language models.\" arXiv preprint arXiv:2402.17762 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7is905EMCE", "reviewer": "Reviewer_YjoR", "summary": "This work relates three different kinds of outliers: activation, weight, and attention outliers. The authors then show that these outliers implicitly scale attention coefficients, which are validated empirically. Finally, the paper proposes an attention-scale to eliminate outliers for downstream benefit.", "questions": "- nit: Many of the figures could have been moved to the appendix, and the paper could have been shortened to fewer pages -- e.g., figs 2, 3, 8, 11. Unless they add to the story substantially, they just increase the distance between figure 1 and the rest of the story.\n- nit: The figures could use captions to help guide the reader to focus on certain attributes of the figures - figures 5, 6, 7 for example. For what it's worth, the figure titles are very descriptive, so that's helpful.\n- nit: One of the critiques of previous papers in L91 is that previous methods \"focus on isolated instances or targeted solutions,\" but it seems like we could have ignored weight and activation outliers, then focused on just attention outliers for this paper.\n- nit: Sec 6 reads like a rebuttal. Granted, this is just the last 1.5 pages -- and it includes the result I'm most excited about -- but it could be better integrated into the rest of the paper. And you probably don't want any reviewer thinking this is a resubmission.\n- nit: There are a few figures that aren't mentioned in the text, such as Figures 2 and 3. And, since the captions are brief, I only have my own observations to make (e.g., a few large-magnitude values are clustered). But perhaps there are other observations you would like the reader to make, and I wouldn't know.\n- nit: Per Figure 5b, does this mean activation outliers never occur in any other token, of the thousands of possible tokens that exist?\n\nSummary: All in all, this is an interesting idea, but lack of practicality and clarity make it hard to recommend an accept. An application paper focuses on the former, and an understanding paper (which I believe this aims to be) focuses on the latter. However, your paper has an interesting insight: The outliers that everyone else observes are all related somehow AND smoothing these outliers can lead to better convergence. I find this last idea particularly exciting, but I think it would take further experimentation to truly make this the focus. Given the paper has many redundant figures that could be moved to the appendix, I believe there is enough room to add a rigorous set of experimental results for convergence studies. I'm also not sure how Sec 4 is related to the method (how do we use it?) or the analysis (why does this happen?), but the fact that it happens consistently across models is certainly thought-provoking. I do believe in this paper's core insight strongly -- I just don't think the current presentation is focused enough for me to have a clear, memorable takeaway. I look forward to the rebuttal though, and I'm certainly willing to bump up my score if you have an idea of how to address these issues + show a promising update to the paper.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "1. Figure 15 is an intriguing result and a strong argument for smoothing out outliers using the proposed method. The potential for converging to a lower loss at a faster rate is certainly desirable. I would argue that this should be the central focus, when anyone asks about the practical value of understanding outliers in such detail.\n2. Figures 1-3 are well illustrated. It's clear that there are outliers clustered in just single channels, which is odd to see and interesting to call out. It's also interesting that activation outliers occur at very specific points in the network, per Figure 4.\n3. Sec 5.2 is interesting. I believe this would be strengthened if you highlighted where each attention variant was pulled from. (d) looks similar to the attention bias that StreamingLLM introduced to fix attention sinks (but I may be wrong here) -- if that's true, it would be beneficial to highlight that only your specific combination of variants successfully inhibits outliers.\n4. The summary sections are helpful for understanding what we've \"learned\" so far at that point in the paper, and they help me contextualize the ablations and analysis. Thanks for including these.", "weaknesses": "1. In Sec 5, you've shown that data-dependent scaling factors completely remove outliers. However, how does this affect quality? That seems like an important question to answer. What if outliers are mitigated but quality now plummets? \"Quality\" here mean zero-shot accuracy, ppl, MMLU etc.\n2. It's unclear what the takeaway for this paper is. For example, in sec 5, why do we want to remove outliers? Luckily, you've actually already pitched a possible angle on this question -- per strength #1, figure 15 shows that convergence may be significantly improved. I believe this is a very interesting idea and should be the focus of the paper. The plot only runs for 50 steps. What happens at step 75, 100, 200? 1000? Does this trend continue, and does the smoothed model converge to a lower loss? If so, that would be a phenomenal result, but that would take more than a rebuttal period to flesh out I believe -- especially if it becomes the \"Table 1\". (But I'm open-minded about being wrong, since you technically have the ability to upload a new copy of the paper)\n3. I can't find a definition of \"outlier\". From my understanding, LLM.int8 has a strange definition of activation outliers for example -- where entire channels are considered \"outlier or not\". From the figures, I can see that outliers are clearly very disproportionately large values, but how do you determine if a token has or doesn't have an outlier? For example, for Figure 5b. \n4. In a similar vein to Weakness #3, where is \"alignment\" defined? I understand the rough idea that weight outlier feature dimension should match the activation outlier's (for example), but what if this dimension alignment occurs for token #1 and doesn't occur for token #2 -- does that mean the entire sequence is now misaligned? Why is the consistency percentage such an even number? Is this because you consider entire sequences to be aligned or misaligned, instead of individual tokens?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)