Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Authors: Mingyu Jin
Kai Mei
Wujiang Xu
Mingjie Sun
Ruixiang Tang
Mengnan Du
Zirui Liu
Yongfeng Zhang
Venue: ICML 2025 poster (2025)
Keywords: Massive Value
LLM
Contextual Knowledge Understanding
Primary area: social_aspects->accountability_transparency_and_interpretability
TL;DR: LLMs' attention layers exhibit concentrated massive values in Q and K (but not V) due to RoPE, which proves crucial for contextual knowledge understanding rather than parametric knowledge retrieval.

Abstract: Large language models (LLMs) have achieved remarkable success in contextual knowledge understanding. In this paper, we show for the first time that these concentrated massive values consistently emerge in specific regions of attention queries (Q) and keys (K) while not having such patterns in values (V) in various modern transformer-based LLMs. Through extensive experiments, we further demonstrate that these massive values play a critical role in interpreting contextual knowledge (i.e., knowledge obtained from the current context window) rather than in retrieving parametric knowledge stored within the model’s parameters. Our further investigation of quantization strategies reveals that ignoring these massive values leads to a pronounced drop in performance on tasks requiring rich contextual understanding, aligning with our analysis. Finally, we trace the emergence of concentrated massive values and find that such concentration is caused by Rotary Positional Encoding (RoPE) and it appears since very first layers. These findings shed new light on how Q and K operate in LLMs and offer practical insights for model design and optimization. The code is available at https://github.com/MingyuJ666/Rope_with_LLM.

Large Language Models (LLMs) have achieved remarkable success across various applications (Wei et al., 2022; Brown al., 2020; Achiam al., 2023; Sun al.; Jin et al.; Tang Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Mingyu Jin
Kai Mei
Wujiang Xu Zirui Liu
Mingjee Sun
Ruixiang Tang'
Mengnan Du
Yongfeng Zhang'
Abstract
Nx
Massive Outlier
Large language models (LLMs) have achieved remarkable success contextual knowledge understanding. In this paper, we show for the first time that these concentrated massive values consistently emerge in specific regions of atten- tion queries (Q) and keys (K) while not having such patterns in values (V) in various modern transformer-based LLMs. Through extensive experiments, we further demonstrate that these massive values play a critical role interpreting contextual knowledge (i.e., knowledge obtained from the current context window) rather than in retrieving parametric knowledge stored within the model's parameters. Our further investigation of quantization strategies reveals that ignoring these massive values leads to a pronounced drop in performance on tasks requiring rich contextual understanding, aligning with our analysis. Finally. trace the emergence of concentrated massive values and find that such concentration is caused by Rotary Positional Encoding (RoPE) and it appears since very first layers. These findings shed new light on how Q and K operate in LLMs and offer practical insights for model design and optimization. The code is available http::/github..r http::/githubb.oom//mag httm///nyyuJ666/Rop.. with LLM.
DIM
SoftMax
Consal Mask
Ro PE
Ro PE
Head
K
Embedding Vector K and
Massive
Yalue
Figure I. In transformer-basee Large Language Models with RoPE (like Llama, Gemma), the attention queries (Q) and keys (K) ex- hibit concentrated massive values in certain dimensions.
et al., 2025; Wang al., 2025). However, our understanding of their internal mechanisms and how these mechanisms relate observable behaviors remains limited, posing challenge to improving their reliability and performance. Researchers have explored LLM representations (Azaria & Mitchell, 2023; Bürger et al., 2024; Gurnee Tegmark, 2024; Jin et al., 2025) and discovered that residual stream activations can exhibit massive values (i.e., magnitudes sig- nificantly larger than typical values (Dettmers al., 2022; Wei et al., 2021; 2023; Sun et al., 2024; Heo et al., 2024)). Recent studies further (Liu et al.; Hooper et al., 2024; Zhao et al., 2024) observe that massive values appear exclusively in Q and K while are absent V. Here, by Q, K, and V, we mean the representations output by the query, key, and value layers in self-attention, respectively. These massive values have been identified critical factors influencing quantiza- tion (Liu et al.; Lin et al., 2024), leading the development of methods such suppression techniques (Wei et al., 2021; 2023) and protection mechanisms (Lin et al., 2024; Xiao et al., 2023). Although these studies utilize observations of massive values for quantization, they do not explore the rationale behind this counterintuitive phenomenon deeply.
1. Introduction
'Tee first two authors are core contributors to this project, listed random order Correspondence Author 'Rttgers Univer- sity *Carnegie Mellon University 'New Jersey Institute of Tech- nology 'University of Minnesota. Correspondence to: Mingyu Jin <mingyu,jin jin jii@@ninnteeeee rutgers. edu>, Zirui Liu <zrliu @umn.euumu Yongfeng Zhang eyongfeng zhang @ @uutgers.edu>.
To address this gap, we systematically investigate the for- mation of massive values and their connection to model behaviors. Our key findings are as follows: ① Massive values are concentrated in specific regions of Q and K exclusively. This echoes the finding in "Round and Round we go" (Barbero et al., 2025). We observe that massive values concentrate in specific regions of Q and K compu-
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Massive Values in Self-Attentin Modules are the Key to Contextual Knowledge Understanding
tations; these massive values in each head's dim index are very close (means concentrate as Definition 2), as is shown
We analyze the functional role of these massive values in Q and K and conduct extensive experiments on ten datasets to demonstrate that they play crucial role in enhancing LLMs' ability to process contextual knowledge, while hav- ing subtle impact on parametric knowledge retrieval.
in Figure 2. While self-attention computation dictates that each attention head operates independently, the massive val- ues across different heads consistently cluster at remarkably similar positional indices, which is very counterintuitive. This phenomenon is absent in V computations and absent in models without RoPE, such as GPT-2 (Black al., 2022) and OPT (Zhang et al., 2022). @ Massive values in Q and K are critical for understanding contextual knowledge over parametric knowledge. Our analysis shows that massive values significantly contribute to contextual knowledge (i.e., knowledge augmented through the context window) rather than parametric knowledge (i.e., knowledge encoded during training). Disrupting these values leads notable degra- dation in tasks requiring contextual understanding, such as passkey retrieval (Jin et al., 2024a; Mohtashami & Jaggi, 2023), IMDB review analysis (Maas et al., 2011), and math- ematical reasoning (Cobbe et al., 2021; Ling et al., 2017). In contrast, tasks involving parametric knowledge, such as World-Cities (Marks & Tegmark, 2023), are only subtly affected. Perplexity analysis reveals that disrupting massive values causes a significant decline in reasoning benchmarks, particularly in IMDB tasks, compared to when non massive values are disrupted. ① Quantization techniques targeting massive values preserve contextual knowledge better. Our experiments on various quantization methods show that tech- niques like AWQ (Lin et al., 2024) and SmoothQuant (Xiao et al., 2023), which scale per-channel weights or activa- tions to reduce the dominance of massive values, effectively maintain LLM's contextual knowledge understanding ca- pabilities. In contrast, methods that do not specifically ad- dress massive values (Frantar et al., 2022) exhibit significant degradation of L.LM's performance in contextual knowl- edge understanding tasks, while performance in parametric knowledge retrieval is maintained. These findings align with our analysis, which shows that massive values con- tribute more to LLM's contextual knowledge understanding capabilities. O Concentration of massive values is caused by RoPE (Su et al., 2024) and it appears since very early layers in LLMs. We analyze the root cause of concen- trated massive values in the Q and K, and investigate layers where massive values become concentrated. Our analy- sis demonstrates that this phenomenon originates from the mechanisms of RoPE, and the concentration of massive values becomes apparent since the very first layers. Our temporal and causal analysis deepens the understanding of how and when massive values are concentrated.
We evaluate three representative quantization methods and show that those explicitly addressing massive values better preserve LLMs' contextual understanding capabilities. This finding reinforces our analysis and provides insights into designing new quantization strategies.
We conduct the causal and temporal analysis of massive value concentrations, uncovering that they originate from RoPE mechanisms and emerge as early as the initial layers. This deepens the understanding of how massive values form and propagate through the model.
2. Preliminary
2.1. Position Encoding
Position Encoding, Position encoding in transformers can be categorized into two main approaches: absolute and rel- ative position encodings. Absolute position embeddings assign fixed positional vectors, such as sinusoidal embed- dings (Vaswani et al., 2017), learned embeddings used in models like GPT-3 (Brown et al., 2020), or dot-product- based adjustments (Ke et al., 2020). In contrast, relative position encodings focus on the distances between tokens and integrate into attention layers, with examples including logit biases like T5 (Raffel et al., 2020; Dai et al., 2019) and rotary position embeddings like RoPE (Su et al., 2024).
ROPE. Here, we introduce the basic concept of RoPE. Let's consider sequence of tokens represented as W1, W2, wt. and their corresponding embeddings are denoted as x1, ... ,x,, E TR PI, where [D] is the dimension of the embedding. The basic idea of RoPE is to incorporate the positional information into the query q and the key vec- tors k, respectively. This integration ensures that their inner product q" k will contain the relative positional embedding information inherently. To achieve this, RoPE employs the following vector transformations:
4m = [g(Xm, m) E TR/ kn = fx(xx. E R!4 (1)
where |1| is the hidden dimension of each head. The func- tions fg and fs. responsible for injecting positional infor- mation, are defined as:
In summary, our contributions are as follows:
fy m) = W Xme imi
, (Xn, n) = Wax,e inl
(2)
We systematically investigate the emergence of massive values in attention modules and identify that their concen- trations occur exclusively in Q and K, revealing a distinct structural property of these components.
where 02 = b Dd b = 10, 000, and the projection ma- trices W,, W₁ RID + R! RoPE keeps the real part of the inner product q k. which is Re(q*k). This operation ensures that the dot product of the query and key vectors
2

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Embedding Vector of [Layer 10)
Embedding Ve ctor K (Layer 20)
Embedding Vector of (Layer
Embedding Vector of Q (Layer 20)
40
40
Dm
20
-10
10.22 29222222922 Number of Hild
2002224 1830922224 29-00-0 Number of Head
1012341512222 21222222222260000 Number of Head
Number of Head
Figure 2. Q and K Embedding Vector in Llama-2-78
choose Layer 10 and 20, and the input question is shown as Figure 11. This visualization shown here is a two-dimensional image because we averaged over the sequence-lengti dimension. The horizontal axis is the number of head and the vertical axis is head dim. We can that the massive value is concentrated the bottom of the picture.
depends entirely on the relative distance between the tokens, represented by m - n, as follows:
3. The Effect of Massive Values in Knowledge Understanding
(Xm: m); fu((x,, n (e((Sg(Xm;))) m), Sio f1a(Xxn; e))
3.1. Massive Value
= =Re(x,m W
W, (0) = Il g(mm Nnz m n2)
The attention queries (Q) and keys (K) in mainstream LLMs are typically represented as Q, K E RBxS*HxD where is the batch size, S is the sequence length, H is the number of attention heads, and D is the head dimension. Assuming B = L, compute the L2 norm along the sequence length dimension, reducing the representation to a matrix M E R where each element MA,a represents the norm of the corresponding component in head and dimension d. Taking Q as an example, Mn,a is computed as:
(3)
where g(-) is an abstract mapping function.
2.2. LLM Inference Workflow
LLM inference consists of a prefilling phase that processes the input context and a decode phase that generates tokens autoregressively, with the prefilling phase establishing the initial context representation and the decode phase itera- tively producing new tokens based on this foundation.
My.d = 11Q.,h =
Σ Q² Po
Given the X R'mmm be the input tensor (we set batch size as 1 by default), /pumpi is the length of the input prompt, h is the number of head, and d is the model hidden size. For convenience, we ignore the layer index here. The key value tensors can be computed by
Prefilling Phase.
Definition 1. (Massive Value) A massive value is element Ma.d that satisfies:
D
1
Mn.d
Mr.a
(8)
D
Xx XWR, X XWv,
(4)
d'=1
where WK, Wy E Rddd are the key and value's layer weights, respectively. After obtaining Xx and Xv, they are cached in the memory for easy decoding.
where > is a threshold controlling massive value selec- tion. In our experiments, we empirically set =
Definition 2. (Concentrate Massive Value) In each attention head, certain dimensions exhibit notably massive values, and these tend to cluster specific dimensional regions (shown in the black box in the Figure 2). Moreover, across different attention heads, these massive values often appear in similar positions.
Decoding Phase. During the decoding phase, let E Rixixd be the current input token embedding. Let (x tWK and tv tWv be the key and value layer outputs, respectively. The KV cache is updated by:
Xx < Concat(X Xx, tx), Xv + Concat(Xx (vv,tv), (5)
3.2. Contextual Knowledge Understanding and Parametric Knowledge Retrieval
then attention output is calculated as:
to= tWQ A St to AXv (6)
Research demonstrates that Large Language Models (LLMs) acquire extensive knowledge through pre-training on large- scale corpora, termed Parametric Knowledge (PK) (Cheng et al., 2024; Xu et al., 2024). In practical applications, LLMs augment this parametric knowledge (PK) with Contextual
where Wo is the weight matrix of the query layer. For ease of illustration, we ignore the attention output layer and other parts of the inference workflow.
3

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Knowledge additionna information provided within
the input context (Xu et al., 2024; Antoniades al., 2025). This distinction gives rise to two fundamental tasks:
Mean(X)
k a
X Kiskk
Xijjk' k + argmax,
Remark 3.1. (Contextual Knowledge Understanding) refers to understanding the content within a paragraph and using the information it provides to answer questions. For example, it could involve identifying a key amidst a collec- tion of meaningless text or determining whether movie review is positive negative based on the content of the re- view (Xu et al., 2024; Cheng et al., 2024; Wen et al., 2024). Remark 3.2. (Parametric, Knowledge Retrieval) refers to questions that can be answered correctly by simply using the query and the knowledge within the model to perform a retrieval match. For example, "What is the capital of the United States?" The answer would be "Washington D.C." (Xu et al., 2024; Cheng et al., 2024)
Our investigation reveals that disrupting massive values can be accomplished through several substitution methods: us- ing mean values, zeros, maxima, or minima. We disrupt mesaiivenoonmassiin values on both Q and K like Table 1. More comprehensive experiments comparing these replace- ment strategies are detailed in Section 3.4.
3.4. Massive Values Contribute to Contextual Knowledge Understanding
As shown in Table 1, Parametric Knowledge Retrieval tasks still maintain relatively high accuracy even when massive values are disrupted, demonstrating performance compara- ble to their original unmodified state and when non-massive values are disrupted. Cities tasks maintain relatively strong performance with scores achieving from 76% to 88%, show- ing degradation of only 15-20%. Sports and Art categories show moderate impact, with performance dropping to the 65-75% range. Technology maintains similar resilience levels, with scores remaining above 70%. The Celebrity category shows the strongest resilience, maintaining perfor- mance above 70% across all models. Importantly, when only non-massive values are disrupted, performance remains re- markably stable across all tasks and models, with variations typically less than +%. This striking contrast suggests that massive values play a crucial role specifically in contextual knowledge understanding tasks, while parametric knowl- edge retrieval tasks can maintain reasonable performance even when these values are disrupted. For mathematical reasoning tasks, the impact is devastating: GSMSK accu- racy drops dramatically (Gemma2-9B: 81.30% to 15.10% Llama3-88 :B: 76.90% to 4.00%, Qwen2.5-7B: 86.60% to 16.10%), and AQUA shows similar degradation with per- formance declining by over 40 percentage points. Passkey Retrieval tasks collapse from 100% to near-zero accuracy (0-2%) across all difficulty levels, while IMDB sentiment analysis performance plummets from above 94% single digits across all models.
Dataset: We separate datasets into the two main cate- gories corresponding to Remark 3.1 and Remark 3.2. For Contextual Knowledge Understanding Tasks, we adopt mathematical reasoning benchmarks (i.e., GSM-8K (Cobbe et al., 2021), AQUA (Ling al., 2017)), sentiment analy- sis dataset (i.e., IMDB (Maas et al., 2011)), and synthetic passkey retrieval datasets as Appendix A in different diffi- culty levels. For Parametric Knowledge Retrieval Tasks, we adopt factual knowledge QA such as Cities (Marks & Tegmark, 2023) and our synthetic datasets covering topics in Sports, Arts, Technology, and Celebrity. The rationale behind our synthesis choice is that we would like use simpler and direct queries that focus on direct factual knowl- edge rather than being blended with reasoning. The dataset details and our data synthesis pipeline can be found at Ap- pendix E and Appendix A.
3.3. Disruption of Massive Value
To investigate the impact of massive values in Definition on I.LM performance, we systematically examine the effects of disrupting both massive values and non-massive values, respectively, in LLMs equipped with RoPE, The disruption is carefully synchronized with the LLM'' two-phase like Section 2.2 content generation process ensure accurate manipulation of massive values at the appropriate stage.
PPL and Diversity Score. In addition to accuracy, we evaluate perplexity (PPL) and diversity as complementary metrics to understand the impact of massive values on model performance. These metrics reinforce our previous findings regarding the crucial role of massive values in contextual knowledge understanding tasks. Perplexity, which measures language model's 's prediction confidence (lower values in- dicating better language modeling ability), and diversity, quantified through 2-gram diversity scores to assess the rich- ness and variety of model outputs, both demonstrate patterns consistent with our accuracy- based observations. We an- alyzed the model's 's perplexity (PPL. and diversity scores
Disruption Setup. Our disruption of massive values and non-massive values is specifically targeted at the prefill- ing stage, as disrupting both prefilling and decoding stages would compromise the model's fundamental language mod- cling capabilities. This selective approach ensures that any observed performance changes can be attributed to the im- pact on context processing rather than impairment of the model's generative abilities. For example, let X E g! x xd denote the query tensor. We replace the values at the mas- sive value indices with the average value computed over the query tensor like Equation
4

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Table 1. Results of L.L.Ms under different settings (vanilla, massive value disnupted, non-massive value disrupted) on different henchmarks. For the Passkey Retrieval Task, the values (max prompt token length, passkey length) represent the maximum number of tokens allowed the prompt and the length of the passkey to retrieved, respectively. All values are reported in percentage (%).
Contextual Knowledge Understanding Task
Parametric Knowledge Retrieval Task
Model
Passkey Retrieval Task
GSM8K
AQUA
IMDB
Cities
Sports
Art
Technology
Celebrity
(128.6)
(256,12)
(1024,48)
Gemmu2-9B
81.30
63.80
100
100
100
94.70
99.70
91.00
84.00
81.00
92.50
Non-Masstve Value piarupted
81.60
65.60
100
97.40
99.60
91.00
84,00
81,50
92.50
Kassive Value Disrupted
15.10
16.50
2.00
0.00
0.00
1.80
76.40
73.50
68.00
72.00
82.00
LJama3-8B
76.90
53.51
100
95.40
99.40
95.00
93.50
92.50
95.00
Nen-Xassive Velue Disrupted
77.40
$3.90
100
100
100
95.40
99.40
94,50
93.00
92.50
95,50
Massive Value Disrupted
4,00
9.68
9.00
0.00
0.00
11.00
88.20
74,50
64.00
74,90
73.00
Qwen2.5-7B
86.60
56.69
100
100
100
96.80
97.70
95.00
96.00
90.00
93.50
Wvnn-Massive Value Diarupted
85.40
57.28
100
100
100
97.60
97.50
94.00
96.50
90.00
93.50
Kassive Value Disrupted
16.10
19.68
9.00
1.00
0.00
6.53
81.50
74.00
69.50
71.00
71.00
Vestis
Nastur Value Dicrupeed
PPL Comparison in LJama3-8B
Model Performance Across Metrics
Non-mescvv Value Sterutted
Vanilla
2-gram Diversity Score Comparison
120
Non massive Value Disrupted
Massive Value Disrupted
90-
100
Performance (%6
Diversity Score
0.8
Score
80
80
0.6
60
70-
GSMBK
0.4
40
AQUA
60-
IMDB
20
0.0
GSMBK
AQUA
IMDB
AQUA
IMDB
GSMBK
vanilla
20
Datasets
Datasets
Models: Llama3 8B
(a) The 2-gram diversity scores for Llama3-8B under three conditions across three datasets.
(b) The perplexity scores of the Llama3- 8B model under three conditions across three datasets.
(c) The figure shows Llama3-8B main- tains relatively performance under differ- ent non non-massive value disrupted methods.
Figure 3. Disrupting massive values leads to higher perplexity and lower diversity, while disrupting non- non-massive values maintains model performance, particularly evident in IMDB dataset analysis.
before and after disrupting massive values, with particular focus on Llama3-8B as shown in Figure 3b and Figure 3a (additional results can be found in Table 7). The experimen- tal results reveal striking patterns: When massive values are disrupted, the LLM shows severely degraded performance. Particularly in the IMDB dataset, we observe a dramatic increase in perplexity (~-8) and a decrease in 2-gram di- versity scores 1~) when massive values are disrupted, while vanilla and non- massive value disrupted conditions maintain low perplexity and high diversity (~0.9). These findings further confirm that massive values are crucial for contextual knowledge understanding tasks.
Mean(X), A- = argmin Tegsk Xigatt, k² + arg min Fejsh
X Aijak=
(10)
As shown in Figure 3c, we systematically varied n from to 20 (represented on the horizontal axis) in our control experiments. The results demonstrate that replacing these minimum values with averages has a subtle impact on model performance across all three contextual knowledge under- standing benchmarks, with GSM8K, AQUA, and IMDB maintaining relatively stable performance throughout the range of n values tested (additional results in Appendix F). This robustness to non-massive value manipulation provides compelling evidence that the model's contextual knowledge understanding cupabilities primarily rely on regions that massive values concentrate.
Controlled Experiment. We conduct our disruption of massive values and non- massissvve values specifically during the prefilling stage. For non-massive values disruption, like the process in massive value disruption, we let X E g xhxd represent the query tensor, and we replace the smallest val- ues with calculated averages. Specifically, we identify the top smallest values along the last dimension (k) and re- place them with the mean value computed across the entire query tensor. For presentation clarity, we omit the batch dimension as these operations are applied consistently.
Disruption Methods Show Consistent Effects. Different approaches to disrupting massive values (using mean, zero, or minimum value substitution like mentioned in Section 3.3) yield consistently similar effects across benchmarks. For instance, in GSM8k, all disruption methods reduce performance to single digits (mean: 4.00%, zero: 1.60%
5

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
1.0
Comparsion DifferentQuantization Methods
Vanilla
0.8
Disrupt Except Massive Value
3.00
smooth-quark
sptc
Massive Value Disrupt
Normalized lized accurocy
0.05
0.6
6,90
0.4
0.2
6.75
0.0
Qwen2.5-7B
Llama3-8B
Datasets: Cities
GSMBK
AQUA
Oitics
Celebrity
Categories
Figure 4. We can observe that introducing conflicting background knowledge causes LLM to be misled into making random guesses. However, after massive values disrupted, the model is still able to maintain a certain level of accuracy.
Figure 5. Impacts of different quantization methods on Llama3-8b across different benchmarks.
values disrupted demonstrate accuracy significantly above chance level. This counter-intuitive improvement suggests that when massive values are destroyed, the model loses its ability to process misleading contextual information and instead defaults ignoring the contradictory context. This observation aligns with previous findings (Yu et al.) that suggest distinet re- gions within neural networks are responsible for context understanding versus parametric knowledge storage.
minimum: 2.73%), while non non-massive value manipulations maintain performance close to the vanilla baseline (76-77%) This pattern holds across all benchmarks, demonstrating that the impact of massive value disruption is robust to the variations of the disruption method. Additional results across other models can be found Appendix F.
parametric knowledge, effectively
Table Impacts of applying diffferent methods disrupt massive battes/non-massive values on model performance (Llama3-8)) Bb)) where / indicates disrupting massive value, while x indicates disrupting non non-massive value.
3.6. The Effects of Massive Values on Quantization
Disruption
Discuption Condition
GSMSK
AQUA
IMDB
Puskkyy Retrieval
(128.6)
(256.12)
To further validate our findings regarding the importance of massive value in model behavior, we evaluated three well- established quantization methods: AWQ (Lin et al., 2024), SmoothQuant (Xiao et al., 2023), and GPTQ (Frantar et al., 2022). As shown in Figure 5, these methods demonstrate distinetly different patterns across task types.
None
76.90
53.51
95.40
100
100
11.00
Mean
77.40
53.90
95.40
100
100
Zero
1.60
8.07
13.40
8.00
0.00
94.70
100
Min
/
2.73
9.33
10.80
9.00
0.00
75.20
53.98
95:40
100
AWQ and SmoothQuant, which explicitly preserve mas- sive values during quantization, maintain strong perfor- mance across all tasks. AWQ achieves this by selectively protecting "important" weights during quantization, while SmoothQuant employs a smoothing factor (S) to redistribute massive values in activations through mathematically equiv- alent transformations. In contrast, GPTQ, which does not specifically protect massive values, shows significant perfor- mance degradation contextual knowledge understanding tasks, particularly on GSM8K and AQUA (dropping to ap- proximately 75% normalized accuracy), while maintaining comparable performance on parametric knowledge retrieval tasks (Cities, Sports, Celebrity). This performance disparity is particularly revealing: quantization methods that protect massive values maintain good and robust performance on contextual knowledge understanding tasks, while methods that do not protect these values struggle specifically with contextual knowledge understanding tasks. These results provide additional evidence for our hypothesis about the crucial role of massive values in LLM's contextual knowl- edge understanding capabilities.
3.5. Effects of Massive Values on Knowledge Conflict.
Our discovery of the differential impact of extreme val- ues on contextual knowledge understanding versus para- metric knowledge retrieval tasks motivated a novel experi- mental investigation. We designed an experiment intro- duce conflicting contextual information in factual datasets, particularly focusing on geographical knowledge from cities (Marks & Tegmark, 2023; Longpre et al.. 2021). For instance, we modified straightforward questions like "Is New York a city in the United States?" by adding contradic- tory context: "Geographical knowledge has changed, and New York has become city in the United Kingdom. Is New York still city in the United States?"
As shown in Figure 4, we tested this modified cities dataset across three model conditions. Interestingly, while the vanilla models (Llama3-8B and (owen2.5-7B) achieve ap- proximately 50% accuracy on this binary classification task equivalent to random guessing the models with massive
6

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
4. Causal Mechanisms and Temporal Analysis of Concentrated Massive Values
preserving basic parametric knowledge retrieval (show- ing only 15-20% degradation). The dramatic increase in perplexity and decrease in output diversity when massive values are disrupted further confirms their crucial role in semantic comprehension rather than positional encoding.
4.1. RoPE Contribute to Concentrated Massive Values
Building upon existing research documenting the relation- ship between RoPE and the embedding patterns of concen- trated massive values (in Defimition and 2) (Barbero et al., 2025), we continue investigating this phenomenon in Q and K embeddings. Based on several observations and analyses, we propose three hypotheses suggesting that the concentra- tion massive values is an inherent consequence of RoPE's implementation.
RoPE applies position encoding selectively: it operates exclusively on K and Q. but not on V. As ex- plained in section 2, this design is infentional-RoPe specif- ically incorporates positional information into the query q and key vectors k, enabling their inner product q q to inherently capture relative positional relationships. This selective application has a direct consequence: the concen- trated massive value appear exclusively in the Q and K, while being completely absent in V, as demonstrated in Fig- ure 7. This pattern is consistently observed across various architectures
Evidence 2:
Evidence 1: In large language models (LLMs), the Rotary Position Embedding (RoPE) implements a sophisticated dimensional pairing approach to encode positional informa- tion. The mechanism divides the embedding dimensionality (d) into ( D pairs, where each pair undergoes rotation at a unique frequency determined by its dimensional index.
shown in Table 3, all LLMs implementing RoPE exhibit these concentrated massive values in the Q and K, while V shows no discernible regular patterns.
Concentrated Massive Value in Q and K ap- pears exclusively in the LLM with RoPE like Gemma: To validate our hypothesis about the relationship between RoPE and massive values, we conducted comparative analysis across different model architectures. First, we examined models without RoPE, including OPT (Zhang et al., 2022) and GPT-2 (Black et al., 2021). As expected, their embed- ding vector maps showed no signs of organized massive values, with OPT-2.7B's embedding vectors (as Figure 6) appearing particularly disorganized and chaotic. In con- trast, we observed consistent patterns of massive values in models employing RoPE, extending beyond traditional lan- guage models (like Llama (Dubey et al., 2024), Qwen (Yang et al., 2024)) to multimodal architectures. For instance, both -.. (Liu et al., 2024) and Qwen2-VL. (Wang et al., 2024) exhibit similar massive value patterns (as shown in Figure 22 in the Appendix). Notably, Qwen2-VL imple- ments Multimodal Rotary Position Embedding (M-RoPE), which decomposes positional embedding to capture ID, 2D, and 3D positional information yet maintains the character- istic massive value patterns. Under controlled conditions, GPT2-NEO and GPT2-NEOX share a similar architecture, with the only difference being the presence of RoPE in GPT2- NEOX. While GPT2-NEO does not exhibit concentrated ex- treme values, GPT2-NEOX clearly does. But the RoPE in GPT2-NEOX is not similar to the RoPE applied in Llama and Gemma, the concentrated massive value will appear in K, Q embedding space, but the position is different; the pat- tern is shown in Figure 27 in Appendix C and Appendix F.
Evidence 3:
Frequency Distribution: For each dimensional pair ()), the rotation frequency is governed by the equation:
8 = 10000-2;/d
(11)
This formulation creates an exponential decay in frequencies across dimensional pairs, resulting in spectrum from high- frequency to low-frequency rotations.
Dimensional Rotation: Each pair of dimensions ( (122)
xzj=1)) ) undergoes rotation according the matrix op-
eration:
(o)(m))))
- -in(m(0))
)
Ø2j
(12)
sin(m(f;) cos(m))))
s25+1
where m represents the position in the sequence.
Multi-Scale Position Sensitivity: This frequency distribu- tion serves two crucial purposes: High-frequency compo- nents (small j) enable fine-grained position discrimination at local scales, as the rotation angle changes rapidly with position. Low-frequency components (large j) capture long- range positional relationships their rotation angles change gradually across positions. The resulting position-encoded representations interact with the attention mechanism to produce pose similarity scores. It decreases from ( as the dimensional rotation angle decreases, caus- ing the massive values to concentrate at the bottom (i.e., low-frequency regions) due to less position information possessed. This concentration of massive values in low- frequency regions primarily encodes rich semantic con- tent rather than positional information, as evidenced by experimental results showing that disrupting these values severely impairs contextual understanding tasks (with IMDB accuracy dropping from 94% to single digits) while
Further supporting our hypothesis. models using alternative position encoding mechanisms, such as Jamba (Lieber et al., 2024), which combines Mamba and Transformer architec- tures without ROPE, show no evidence of these massive value patterns. Detailed visualizations and more analysis
7

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Embedding Vector o [aayer 201
Enbeddingg Vecter Qf (Layer 20)
Embedding Vector of (Layer 20)
RoPE Concentrated
Model
Massive Values
Llama 3
Qwen 2, 2.5
Gemma 1,2
Phi-3
01
Falcon3
LLAVA
Figure 6. K, Q and V in Layer 20 of OPT-350M, where massive values are not
Marber Host
Namber of Heat
Number of Hood
Qwen2-VL
Mistral-vO.
concentrated in specifie regions.
GPT-NeoX
GPT-2
X
use Rosonnees High RaARouu Sow.arodd
Embeoding Vector e (Layer
Embecding ayer
Embedding Vector of (Layer 201
GPT-Nco
OPT all size
X
Tre cis High meeg
Jamba
X
X
High
Table 3. Analysis of different models about whether they adopt RoPE and whether concentrated massive values in
Conal ent sutcciio
Q and K can observed these models.
Number Heed
Number of Head
Number of
Figure 7. K. Q, and V at Layer 20 in Gemma2-98. Massive values are concentrated
in low-frequency regions of K and Q, absent V.
of L.LMs employing different position embeddings that can support these findings are provided in the Appendix C.
formed through training rather than being solely caused by the addition of RoPE (since RoPE itself is not trainable).
Evidence 4: Since RoPE divides the dimensionality into pairs and applies similar rotational operations across these pairs, we hypothesized that the pattern of massive values should exhibit a mirrored structure across the two halves of the embedding dimensions. Indeed, our experimental results confirm this prediction: we observe two distinct clus- ters of massive values in the embedding vector, with one cluster appearing in the first half of the dimensions and corresponding cluster in the second half, creating sym- metric pattern. This symmetrical distribution of massive values aligns perfectly with RoPE's dimensional pairing mechanism and provides additional evidence that these pat- terns are not random artifacts but rather emerge from the fundamental architectural design of positional encoding in transformer models.
5. Related Work
5.1. LLM Quantizations and Massive Values in LLM
Various existing works in quantization have studied the ex- istence of outlier (also called massive value) features in LLMs (Dettmers et al., 2022: Ahmadian et al., 2023; Guo et al., 2024; Xu et al., 2025b). Some research showed that outlier features have large activation values in most of their sequence dimensions. BERT (Devlin al., 2019) often focuses attention the "[SEP]" token (Clark et al.; Xiao et al., 2024), while LLMs predominantly assign attention to the starting word token (Xiao et al., 2024); some re- search focuses on the massive activation value in activation in LLM (Sun et al., 2024) and identified attention artifacts in ViTs (Darcet et al., 2024). Some studies also try to understand the mechanism of these outlier feature in trans- former (Guo et al., 2024).
4.2. When Concentrated Massive Values Occur
Through extensive experiments and observations, we found that L.Ms such as Llama, Gemma, and so on exhibit mas- sive values in Q and K starting from the very first layer, which is different from massive attentions (Sun et al., 2024) that become significantly apparent only in the later layers. However, there are subtle differences in the patterns be- tween each layer. The layer-wise visual illustrations can be found in Appendix F. each layer, when extracting the Q and K. we consistently observe that there are no significant differences between the results taken before and after ap- plying RoPE. The patterns remain largely consistent across all layers. This indicates that the massive value is gradually
Research on LLM quantization has predominantly focused on analyzing model weights and activations through two main approaches: (1) W8A8 quantization, which converts both activations and weights to INT8 (Wei et al., 2021; 2023; Xiao et al., 2023; Yao et al., 2022; Dettmers al., 2022), and (2) low-bit weight-only quantization (e.g., W4A16), which converts only weights to low-bit integers (Frantar et al., 2022; Lin et al., 2024; Sheng et al., 2023; Park et al., 2024). During quantization, some studies have identified outlier feature values in activation values (Liu et al.; Sun et al., 2024) as a critical factor that can affect quantiza- tion. Several approaches have been developed to handle
8

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
outlier feature specifically, either specific suppression tech- niques (Wei al., 2021; 2023) protection mechanisms (Lin et al., 2024; Xiao et al., 2023) to maintain outlier fea- ture unchanged. Quantization methods specifically address outlier feature that are typically analyzed within individual attention heads. Our work examines massive values across the depth dimension of multiple attention heads, offering potential new directions for future quantization approaches.
into RoPE's role in LLMs. This study establishes deeper understanding of massive values in LLMss, their critical role in contextual knowledge understanding, their implications for model optimization techniques such as quantization, and their connection to RoPE-induced patterns. These findings lay the foundation for developing more robust, efficient, and interpretable LLM architectures and optimization strategies.
A Acknowledgment
5.2. Rotary Positional Encoding
We thank Taowen Wang. Cheng Han, Wenyue Hua, Fei Sun, Hongye Jin, Yixuan Tang and Fan Yang for their valuable discussions and suggestions during the project.
Rotary Positional Encoding (RoPE), introduced by Su et al. (2024), has become standard component in major LLMs including Llama (Touvron al., 2023), Palm (Anil al., 2023), Mistral (Jiang et al., 2023), and GLM (Du et al., 2022). Studies on long- long-context Transformers (Sun et al., 2023; Xiong et al., 2024; Dubey et al., 2024) demonstrate that increasing RoPE's 0 parameter from 10,000 to larger values (e.g.. 500,000) reduces attention decay and improves learning over extended contexts. Qwen2-VL VL advances this with M-ROPE, decomposing positional embedding into 1D, 2D, and 3D components for enhanced multimodal process- ing (Wang et al., 2024). In parallel, research on NoPE (No Positional Encodings) has shown promising results in out-of- distribution (OOD) settings compared to RoPE, suggesting that the causal mechanism alone might sufficiently capture positional information without explicit position encoding (Haviv al., 2022; Kazemnejad et al., 2024). Analysis of Gemma-7b 7b reveals that embedding vectors' high-frequene components encode position while low-frequency parts carry semantics (Barbero et al., 2025). Some researchers replaced the original position embedding of GPT-2 with ro- tational position embedding (RoPE), following the standard implementation of GPT-NeoX, which resulted in better train- ing results (Allen-Zhu & Li). Our work explores RoPE's contribution to value formation in transformer-based L.LMs.
6. Conclusion
Our study provides novel insights into the role and ori- gin of massive values Large Language Models (LLMs). Through systematic investigation, we find that massive val- critical in contextual knowledge understanding tasks, such passkey retrieval and IMDB sentiment understand- ing, In contrast, their influence on parametric knowledge retrieval tasks, such as world knowledge retrieval, is lim- ited. This finding emphasizes the importance of preserving massive value to maintain model performance reasoning and context-dependent tasks. Our investigation reveals that RoPE induces massive value stripes, distinct patterns ex- clusively in the Q and K, while absent in models without ROPE, such as OPT. This highlights how positional encod- ing mechanisms contribute to massive values, particularly low-frequency channel dimensions, offering new insights
9

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Impact Statement
Azaria, A. and Mitchell, T. The internal state of 11m
knows when it's lying. In Findings of the Association for Computational Linguistics: EMNLP 2023 (EMNLP), pp. 967-976, 2023.
This work advances our understanding of Large Language Models by systematically investigating massive values in attention mechanisms and their relationship with contextual knowledge understanding. Our findings provide valuable insights for future model design and optimization, partic- ularly in model quantization and parameter- efffccceett fine- tuning areas, Our discoveries about the relationship between massive values and contextual knowledge understanding could inform more efficient model compression techniques, potentially reducing the computational resources required to deploy these models. This could help make language models more accessible while maintaining their core capa- bilities. While this technical advancement in understand- ing model internals has primarily positive implications for model efficiency and effectiveness, we acknowledge that im- provements in L.MM capabilities may have broader societal impacts that warrant ongoing discussion and careful consid- eration by the research community. Our work is focused advancing the fundamental understanding of machine learn- ing systems, particularly in transformer architectures and attention mechanisms. The ethical implications and societal consequences align with those generally associated with improvements in LLM understanding and optimization.
Barbero, F., Vitvitskyi, A., Perivolaropoulos, C., Pascanu, R., and Veličković, P. Round and round we go! what makes rotary positional encodings useful? In Interna- tional Conference on Learning Representations (ICLR), 2025.
Black, S., Gao, L., Wang. P., Leahy, C., and Biderman, S. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensooflow, March 2021. URL https://doi.or//10.. http:/d...rrr/l...22///eeee/e/// /ttp://....8//12000005777755
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, I.,, Golding, I.,, He, H., Leahy, C.,, McDonell, K., Phang, J., Pieler, M., Prashanth, S. Purohit, S., Reynolds, L., Tow, J., Wang, B., and Weinbach, S. GPT-NeoX-20B: An open-source autoregressive language model. In Fan, A., Ilie, S., Wolf, T., and Gallé, M. (eds.), Proceedings of BigScience Episode #5 Workshop on Challenges & Per- spectives in Creating Large Language Models (ACL Work- shop). Association for Computational I Linguistics, May 1.
2022. URL https://ccautbolggggggg222...ooe 9/.
References
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Informatinn Processing Systems (NeurIPS), pp. 1877-1901, 2020.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint:2303.087744 (OpenAl Technical Report), 2023.
Ahmadian, A., Dash, S.,, Chen, H., Venkitesh, B., Gou, Z. S., Blunsom, P., Üstün, A., and Hooker, S. Intriguing properties of quantization at scale. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 34278-224883 -32944, 2023.
Bürger, L., Hamprecht, F. A., and Nadler, B. Truth is universal: Robust detection of lies in LLMs. In The Thirty-eigtth Annual Conference on Neural Informa- tion Processing Systems (NeurlPS), 2024. URL https: wttp://wwwwneeeooomddddd---e
Allen-Zhu, Z. and Li, Y. Physics of language models: Part 3.1, knowledge storage and extraction. In Forty-first International Conference on Machine Learning (ICML).
Cheng, S., Pan, L., Yin, X., Wang, X., and Wang, W. Y. Un- derstanding the interplay between parametric and contex- tual knowledge for large language models. arXiv preprint 24ti::2410.084444 2024.
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E.,, Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint M:Xiv:2305.10403 (Google Technical Report), 2023.
Chiang, C.-H. and Lee, H.-Y. Can large language models be an alternative to human evaluations? In Proceedings of the 61 Annual Meeting of the Association for Compu- national Linguistics (ACL), pp. 15607-15631, 2023.
Anthropic. Long context prompting for claude 2.1. https: www.anttroppc.commees//ceooooooe 1-r-mpppp,n 2023. Accessed: late-of-access.
Clark, K., Khandelwal, U., Levy, O., and Manning, C. D.
What does BERT look at? an analysis of BERT's atten- tion. In Proceedings of the 2019 ACL Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, Florence, Italy. Association for Computational Linguistics. doi: 11116665/11W11-44222 URL httpp://wcccattoogg../////////// 90949//9/..///94488..
Antoniades, A., Wang, X., Elazar, Y., Amayuelas, A., Al- balak, A., Zhang, K., and Wang, W. Y. Generalization vs. memorization: Tracing language models' capabilities back to pretraining data. In International Conference on Learning Representations (ICLR), 2025.
10

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14166 (OpenAl Technical Re- port), 2021.
Gu, A. and Dao, T. Mamba: Linear-time sequence mod- eling with selective state spaces. In First Conference Language Modeling (COLM), 2024. URL hitps: Vlopenreview.net/forum Eo skw112..
Guo, T., Pai, D., Bai, Y., Jiao, J., Jordan, M., and Mei, S. Active-dormant attention heads: Mechanistically demys- tifying satreme-tokenn phenomena in LLMs. In NeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024. URL https//opeurriie..e///ouuu KWVjN8BL8a.
Dai, Z., Yang, Z., Yang, Y., Carbonell, J. G., Le, Q. V., and Salakhutdinov, R. Transformer-xl: Attentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019. URL https//api.semaatiicchoaaaa haapattiieccccaaooooooousiolllll 57759363.
Gurnee, W. and Tegmark, M. Language models represent space and time. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL htips: Vlopemeeview.ne//formmiii wwt.:///ooorummiddidabmmmmiiii
Dao. T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. In Proceedings of the 35th Neural Informa- lion Processing Systems Conference (NeurIPS), 2022.
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vision transformers need registers. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL http:://ogaaeveeww..e///oo//dddd
Haviv, A., Ram, O., Press, O., Izsak, P., and Levy, O. Trans- former language models without positional encodings still learn positional information. In Findings of the As- sociation for Computational Linguistics: EMNLP 2022 (EMNLP), pp. 1382-1390, 2022.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gp13. int8 (): 8-bit matrix multiplication for transformers at scale. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 30318 1-33333,,, 2022.
Heo, J. H., Kim, J., Kwon, B., Kim, B., Kwon, J., and Lee, D. Rethinking channel dimensions to isolate outliers for low-bit weight quantization of large language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://ppnnreview net/forum Gid..
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019. URL 52967399.
Hooper, C. R. C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, S., Keutzer, K., and Gholami, A. KVQuant: Towards 10 million context length LLM inference with KV cache quantization. In The Thirty-eighth Annual Conference Neural Information Processing Systems (NeurIPS), 2024. URL https://ppennevie...efffommlll OLXotew9Du.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbor, D., and Zhai, X. Thomas unterthiner mostafa dehghani matthias minderer georg heigold sylvain gelly jakob uszkoreit and neil houlsby. an image isworth 16x 16 words: Transformers for image recognition atscale. In International Conference on Learning Representations (ICLR), 2021.
Hu, L., Liu, Z., Zhao, Z., Hou, L., Nie, L., and Li, J. A sur- vey of knowledge enhanced pre-trained language models. IEEE Transactions on Knowledge and Data Engineering (TKDE), 2023.
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 320-335, 2022.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. (Mistral Al Technical Report), 2023.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.21788 (Meta Al Technical Report), 2024.
Jiang. Z., Gu, J., Zhu, H., and Pan, D. Pre-rmsnorm and pre- crmsnorm transformers: equivalent and efficient pre-In transformers. Advances in Neural Information Processing Systems (NeurIPS). 36, 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate post-training compression for genera- tive pretrained transformers. In International Conference on Learning Representations (ICIR), 2022.
Jin, H., Han, X., Yang, J., Jiang, Z., Liu, Z., Chang, C.-Y,, Chen, H., and Hu, X. LLM maybe longLM: Selfextend LLM context window without tuning. In

---

Massive Values in Self-Attentio Modules are the Key to Contextual Knowledge Understanding
Forty-first International Conference on Machine Learn- ing (ICML), 2024a. URL. htt.hhgps///peeevviie.eeeeoooooo id=nkOMLBlil7.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning. In Proceedings of the 37th Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2024.
Jin, M., Xue, H., Wang, Z., Kang, B., Ye, R., Zhou, K., Du, M. and Zhang, Y. Prollm: Protein chain-of- thhhoghts enhanced Ilm for protein- proeein interaction prediction. In First Conference on Language Modeling.
Liu, Z. Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. In Forty-first International Conference on Machine Learning (ICML).
Jin, M., Luo, W., Cheng, S., Wang, X., Hua, W., Tang, R., Wang, W. Y., and Zhang, Y. Disentangling memory and reasoning ability in large language models. arXiv preprint arXiv:2411. 13504, 2024b.
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., and Singh, S. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 7052-7063, 2021. doi: 10.18653/ v v1/2021.emnlp mainmlllpmmii.5eae 505. URL https:/acclanthooog..rrgg 2021 wel..emll-maiin555
Jin, M., Yu, Q.. Shu, D., Zhao, H., Hua, W., Meng, Y., Zhang, Y., and Du, M. The impact reasoning step length on large language models. In Findings of the Asso- ciation for Computational Linguistics ACL 2024 (ACL), pp. 1830-18344 Bangkok, Thailand and virtual meeting, 2024c. URL 108.
Maas, A. L., Daly, R. E.,, Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment anal- ysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Lan- guage Technologies (ACL), pp- 142 150, 2011. URL https//aclantholoy.orrPPP Pr-1//1-1
Jin, M., Yu, Q. Huang, J., Zeng, Q.. Wang, Z., Hua, W., Zhao, H., Mei, K., Meng, Y., Ding, K., Yang, F., Du, M., and Zhang, Y. Exploring concept depth: How large language models acquire knowledge and concept at dif- ferent layers? In Proceedings of the 31st International Conference on Computational Linguisties, pp. 558 558-573, Abu Dhabi, UAE, January 2025, Association for Compu- tational Linguistics. URL https:/aclamntholoy..rrr coling- mainng-main.37/.
Marks, S. and Tegmark. M. The geometry of truth: Emer- gent linear structure in large language model represen- tations of true/false datasets. In NeurIPS Workshop on Attributting Model Behavior at Scale, 2023. URL http://ppeaaeeevvvv.eee/oononddd
Kazemnejad, A., Padhi, I., Natesan Ramamurthy, K., Das, P., and Reddy, S. The impact of positional encoding on length generalization in transformers. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Mohtashami, A. and Jaggi, M. Random-acces infi- nite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023. URL https:/oopemrvie..n//foumiiiii 7eHn64w0Vy.
Ke, G., He, D., and Liu, T.-Y. Rethinking positional encod- ing in language pre-training. In International Conference on Learning Representations (ICLR), 2020.
Park, G., Park, B., Kim, M., Lee, S., Kim, J., Kwon, B., Kwon, S. J., Kim, B., Lee, Y., and Lee, D. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scate generative language models. In International Conference on Learning Representations (ICLR), 2024.
Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedi- gos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev- Shwartz, S., et al. Jamba: A hybrid transformer-mante language model. arXiv preprint arXiv:2403. 19887, 2024.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-awan weight quantization for Ilm compres- sion and acceleration. In Conference Machine Learn- ing and Systems SMs), 2024.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training, openai. Preprint, pp. 1-12, 2018.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), pp. 5748-8763. PMLR, 2021.
Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Barzilay, R. and Kan, M.-Y. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), 2017. URL htw.hh/s//ccanhhoooo..o.....0..0
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the
12

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
limits of transfer learning with unified text-to-text trans- former. Journal of machine learning research (JMLR), 21(140):1-67, 2020.
Vaswani, A., Shazeer, N. M., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, I.., and Polosukhin, I. Attention is all you need. In Conference on Neural Information Processing Systems (NeurIPS), 2017. URL https//ap.semmanticccllaaaassss
Sheng. Y. Zheng, L., Yuan, B., Li, Z.,, Ryabinin, M., Chen, B., Liang, P., Ré, C., Stoica, I., and Zhang, C. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning (ICML), pp. 31094-31116. PMLR, 2023.
13756489.
Wang. P., Bai. S., Tan, S., Wang. S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 1219 (Qwen Team Tech Report), 2024.
Su. J. Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputiing, 568:127063, 2024.
Wang. X., Tan, S., Jin, M., Wang, W. Y., Panda, R., and Shen, Y. Do larger language models imply better rea- soning? pretraining scaling law for reasoning. arXiv preprint wrXiv2250.006335 2025.
Sun, G., Jin, M., Wang, Z., Wang, C.-L.,, Ma, S., Wang, Q.. Geng, T., Wu, Y. N., Zhang, Y., and Liu, D. Visual agents as fast and slow thinkers. In The Thirteenth International Conference on Learning Representations (ICIR).
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. In Advances in neural information processing systems (NeurIPS), volume 35, pp. 24824 24837, 2022.
Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. In First Conference on Language Modeling (COLM), 2024. URL. https:// opporeview.netfforumtttc
Wei, X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q., Yu. F., and Liu, X. Outlier suppression: Pushing the limit of low-bit transformer language models. In Proceedings of the 34th Advances in Neural Information Processing Systems (NeurIPS), 2021.
Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., and Wei, F. A length- extrapolatable transformer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 14590-14604, Toronto, Canada, long 816/.
Wei, X., Zhang, Y., Li, Y., Zhang, X., Gong, R., Guo, J., and Liu, X. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1648-1665, December 2023. URL htips: //cclattoloog.oor/22233.o
2023. URL https//aclanthooog..og...
Tang. H., Zhang, C., Jin, M., Yu, Q. Wang, Z., Jin, X., Zhang, Y., and Du, M. Time series forecasting with llss: Understanding and enhancing model capabilities. ACM SIGKDD Explorations Newsletter, 26(2):109-111 2025.
102//
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0822 08295 (Google DeepMind Technical Report), 2024.
Wen, K., Dang, X., and Lyu, K. Rons are not transformers (yet): The key bottleneck on in-context retrieval. arXiv preprint arXiv:2402.18511, 2024.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.
Tevet, G. and Berant, J. Evaluating the evaluation of di- versity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACI.), pp- 326-346, April 2021. doi: 10. 202l.eacl-main.25, URL httpss//cclaathhoooyyoo0///0 main.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Effi- cient streaming language models with attention sinks. In The Twelfth International Conference on Learning Rep- resentations (ICLR), 2024. URL https://ppenreview.ned forum?id=NG7:s55122
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: (Meta Al Technical Report), 2023.
Xiong, W., Liu, J., Molybog, I., Zhang, H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz, B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S.,
13

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling of foundation models. In Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (NAACL), pp. 4643-4663, Mexico City, Mex- ico, 2024, URL httpss//accannnoooog..oo wrg. 2024. www.o.a.n.nnn.noeeeee naacl-long. 260/.
Zhao, Y., Lin, C.-Y,, Zhu, K., Ye, Z., Chen, L.,, Zheng, S., Ceze, L.., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems (MLSys), 6: 199,209, 2024.
Zheng, L.., Chiang, W.-L.., Sheng, Y., Zhuang, S., Wu, Z.,, Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing. E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-jude with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS), 2023, URL wwwppppwwwwwi.eeeooooooo/ddnnnne
Xu, R., Qi, 7.,, Guo, Z.,, Wang. C.,, Wang, H., Zhang, Y., and Xu, W. Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8541-8565, Miami, Florida, USA, Novem- ber 2024. Association for Computational Linguisties. doi: 10.186653/w1/2222..mmm main, /hccbarhooogy.ogg2224eemmmlm 486/
URL https:
Xu, W., Liang, Z., Mei, K., Gao, H., Tan, J., and Zhang, Y. A-mem: Agentic memory for Ilm agents, arXiv preprint arXiv:2502. 12/ 10, 2025a.
Xu, W., Wu, Q.. Liang, Z., Han, J., Ning, X., Shi, Y., Lin, W., and Zhang. SImrec: Distilling large language models into small for sequential recommendation. In The Twelfth International Conference Learning Representations (ICLR), 2025b.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang. J., Yang. J., Tu, J., Zhang, J., Ma, J., Xu,
J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen,
K., Yang. K., Li, M., Xue, M., Ni, N., Zhang, P., Wang, P., Peng, R., Men, R., Gao, R., Lin, R., Wang, S., Bai, S., Tan, S., Zhu, T., Li, T., Liu, T., Gc, W., Deng, X., Zhou, X., Ren, X., Zhang, X., Wei, X., Ren, X., Fan, Y., Yao, Y., Zhang, Y., Wan, Y., Chu, Y., Liu, Y., Cui, Z., Zhang, Z. and Fan, Z. Qwen2 technical report. arXiv preprint 2XXi:2407.11667 (Qwen Team Tech Report), 2024.
Yao, Z. Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. Zeroquant: Efficient and afford- able post-training quantization for large-scale transform- ers. Advances in Neural Information Processing Systems (NeurIPS), 11127768-27188 2022.
Yu, Q., Merullo, J., and Pavlick, E. Characterizing mecha- nisms for factual recall in language models. In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open tre-trdi transformer language mod- cls. arXiv preprint arXiv:2205.001 01068 (Meta Al Technical Report), 2022.
14

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
APPENDIX
This appendix contains additional details for the paper. The appendix organized as follows:
Section reports $A more about Data Synthesis Pipeline.
Section reports $B more about More visualization results for LLMs without RoPE.
Section reports $C more about More visualization results for LLMs with RoPE.
Section reports $D more about Some Results when massive values are disrupted in LLMs..
Section reports SE more about Experiment Details.
Section reports $F more about More Experiments to Support our Conclusion.
A. Data Synthesis Pipeline
A.1. Synthesizing Parametric Knowledge Retrieval Dataset
The pipeline for synthesizing parametric knowledge retrieval datasets shown as below. Following existing benchmarks, we choose the commonly-use topics covering Sports, Arts, Technology and Celebrity to feed into the LLM to generate factual knowledge QA related to these topics (Hu et al., 2023; Xu et al., 2025a). For the QA pair, the LLM is required to generate true/false following predefined formatting requirements like (Zheng et al., 2023). These generated questions are verified by another LLM instance regarding each question's 's structure, answer, and correctness, making improvements when necessary to meet quality requirements (Zheng et al., 2023; Chiang Lee, 2023). To ensure the highest standards of data quality, manual inspection stage is incorporated, where human annotators review and validate the generated questions, marking them either acceptable requiring revision.
LLM
LLM
Manual Inspection X
Verify this true/false question (question, answer, correct fact) for accuracy and improve needed: Requirements...
Factual Knowledge
Generate true/fialse factual knowledge questions about (category]- Questions in the format...
Knowledge intensive Dataset
Taxonomy
Figure Pipeline of synthesizing parametric knowledge retrieval dataset: The process starts with a factual knowledge taxonomy as input. An L.M generates true/false factual questions in predefined format. Another LLM verifies and refines these questions for accuracy. A manual inspection ensures quality, producing showledge-intensinn dataset for parametric knowledge retrieval tasks.
To ensure the quality of our synthesized question-answer pairs, we implement two-stage verification pipeline. First, initial filter to automatically assess the factuality of the generated content across
leverage LLM (i.c., claude-3.5ssommet) different categories (Sports, Arts, Technology, and Celebrity). This automated verification serves efficient preliminary screening mechanism. Subsequently, we conduct a rigorous manual inspection by human evaluators to further validate the filtered data by the [.LM evaluator to avoid hallucinations. After the human evaluation, we sample 200 examples for each category to construct the final synthetic dataset. The verification statistics are shown in Table 4, and some examples of the factual QA are shown Table
Table 4. Verification of whether the synthesized question and answer fulfills the factuality by both LLM and human.
LLM Verification
Manual Inspection
Category
Pass Rate
Failed Case
Pass Rate
Failed Case
Sports Arts
100%
0%
92%
8%
98%
2%
89%
1 11%
Technology
99%
1%
90%
10%
Celebrity
99%
1%
90%
10%
15

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Following the two-stage pipeline, we need use GPT4 and Human evaluate and check the dataset, to verify whether our synthetic dataset is completely knowledge- a little difference in pass rates between the LLM and human inspections. As shown in Table 4, while the LLM achieved near-perfect accuracy with a pass rate of 98-100% across all categories, manual inspection revealed discrepancies, with pass rates ranging from 89% to 92%. This highlights the necessity of human evaluation to catch subtle errors, especially in nuanced categories like Arts and Celebrity, where subjective interpretations and context might introduce inaccuracies. The example in our dataset can be seen at Table 5. These carefully curated question-answw pairs form a robust foundation for downstream tasks such as knowledge-intensive evaluations and parametric knowledge retrieval testing.
without any context understanding (Jin et al., 2024c;b). We observed
Table 5. Examples of parameter knowledge retrieval task: factual QA, covering Sports, Arts, Technology and Celebrity.
Category
Example
Ground Truth
Is the Olympic Games held every four years?
Yes
Sports
Was Babe Ruth famous football player? Is the FIFA World Cup held every two years?
No
No
Was the painting 'Girl with a Pearl Earring' e completed during the 18th century?
No
Arts
Is Pablo Picasso one of the founding figures of Cubism?
Yes
Was Diego Rivera famous Mexican muralist?
Yes
Is the ASCII character set limited to 256 characters?
No
Technology
Was the first iPhone released in 2007? Is Linux an open-source operating system?
Yes
Yes
Is Leonardo DiCaprio an Oscar-winning actor?
Yes
Celebrity
Was Taylor Swift born in Los Angeles?
No
Was Michael Jackson a member of The Beatles?
No
A.2. Synthesizing Passkey Retrieval Dataset
The passkey retrieval task is the same as defined in Landmark Attention (Mohtashami & Jaggi, 2023), synthetic long context task. It requires a language model to retrieve a simple passkey (i.e., a 6-digit random number) from a long meaningless text sequence. The passkey is placed at various document depths and context lengths (Jin et al., 2024a) (maybe ranging from 4k to 24k). The LLM's performance on this is not sensitive to the prompt (the prompt all garbled, the LLM is not sensitive to these different prompts) (Anthropic, 2023). This is likely because the sentence carrying the passkey is significantly different from the surrounding repeated random text. Empirically, within the effective context window, almost all LLMs, including those without any instruction tuning or alignment, are able to locate the sentence carrying the passkey.
An example e of Passkey Retrieval Task (6-128)
There is important info hidden inside a lot of irrelevant text. Find it and memorize it. will quiz you about the important information there. The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. The grass is green. The sky blue. The sun is yellow. Here we go. There and back again. The pass key is 383816. Remember it. 383816 is the passkey.
Prompt:
Ground Truth: 383816
Figure 9. An example in Passkey Retrieval Task, this is the easier case in this kind of task. The pass key length is 6 tokens, and the prompt length is less than 128 tokens.
Although this task is simple and far from real-world scenarios, it tests two fundamental capabilities of LLMs: The model should be able to identify and locate useful information at any position in the input sequence. The model should be able to utilize the perceived information to complete tasks. To summarize the two abilities, this task serves as a good in-context retrieval benchmark, as it does not require understanding the text content (not require contextual knowledge understanding ability), only locating the passkey retrieval.
16

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
An example of Passkey Retrieval
Task (12-256)
There an important info hidden inside lot of irrelevant text. Find and memorize will quiz you about the important information there. The grass is green. The sky is blue. The sun is yellow. Here go. There and back again. The grass is green. The sky is blue. The sun is yellow. Here go. There and back again. The pass key is 201724567512 2.. Remember it. 201724567512 is the passkey. The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. The grass is green. The sky is blue. The sun is yellow. Here we go. There and back again. The grass is green. The sky is blue. The sun yellow. Here we go. There and back again. The grass green. The sky blue. The sun is yellow. Here we go. There and back again. The grass green. The sky is blue. The sun is yellow. Here we go. There and back again. The grass green. The sky blue. The sun is yellow. Here we go. There and back again.
Prompt:
Ground Truth: 201724567512
Figure 10. An example in Passkey Retrieval Task, this is the harder case in this kind of task. The pass key length is 12 tokens, and the prompt length about tokens. The pass key hidden middle of prompt.
Therefore, chose this benchmark. Additionally, set up three hierarchical levels of tasks with 128 (as Figure 9), 256 (as Figure 10), and 1024 tokens, ranging from simple to difficult. Experimental results show that for the 1 128-token level task if we eliminate all massive values, the model may not even be able to provide response. In the task of retrieving passwords within just 128 tokens, LLMs perform extremely poorly, which sufficient to demonstrate that they no longer possess in-context retrieval capabilities.
A.3. Inequality Relation Problems
In this chapter, we detail the process of synthesizing a dataset comprising 100 inequality relation problems. These problems are designed to assess logical contextual knowledge understanding by presenting two premises involving inequalities and querying the relationship between two variables based on these premises. The dataset structured to facilitate applications such as educational assessments, machine learning model training, and logical contextual knowledge understanding evaluations.
The synthesized dataset consists of 200 entries, each containing:
Premise 1: An inequality between two variables (e.g., A > B).
Premise 2: An inequality between the second variable of Premise and a third variable (c.g., > C?.
Question: A query that asks about the relationship between the first and third variables based on the given premises (e.g.. A> B, B > C, what is the relation between A and C?).
Answer: The inferred relationship between the first and third variables, which could be definitive incquality or an indication that the relationship cannot be determined based on the given premises.
Table Sample Incquality Relation Problems
Premise
Premise
Question
Options
Correct Answer
1. (A C) determine relation
(A C)
(A) and (Cy?
(C)
3. (A C)
1. F)
(D < B)
(E < F)
(D E, F), what is the relation between (D) and (Fy²
2. Cannot determine the relation between (D) and (F)
F)
1. Cannot determine relation betwee and
(G>
(II
GG II D, what is the relation between (G) and
2. (G
3. (G >
17

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Prompt in LLM to save Embedding Vector
is being driven, in a straight line and at a uniform speed, towards the base of vertical tower. The top of the tower observed from the and, in the process, takes 10 minutes for the angle of elevation change from 45 to 60". After how much more time will this car reach the base of the tower?
Figure 11. Prompt in Inference LLM
Embedding Vector of (Layer 20)
Embedding Vector (Layer 20)
Embedding Vector of V (Layer 20)
70
70
70
60
12
12
21
21
21
24
24
10
30
33
33
30
30
42
42
42
20
20
20
51
51
51
100
-10
10
60
60
63
4 6 101112131415
0 5 1011121311115
5
101112131415
Number of Head
Number Head
Number Head
Figure 12. Embedding Vector K Q. OPT-350M, we choose Layer and input question is shown as Figure
B. More Visualization Results for LLMs without RoPE.
Open Pretrained Transformers (OPT) is a complete suite of large language models based on the Transformer Decoder, serving replica of the largest GPT-3 model with 175 billion parameters (Zhang et al., 2022). OPT ranges in size from 125 million to 175 billion parameters, matching the scale of GPT-3, and its code has been fully ooun-sourced. In the structure of OPT, we can see that they did not apply RoPE (Rotary Positional Encoding) in Q and K in the transformer. The decoder in the OPTForCausalLM model is a critical component of its architecture, designed to process sequences in a manner conducive to causal language modeling. It consists of several layers, specifically 24 identical OPTDecoderLayer modules, reflecting a deep transformer architecture.
OPT-350M
OPT-350M
13
10 2 10 a
1 Tnkon
20 Dm
Bkaan
Dim
20
Rku
st
Dim
20
Akan
57
0m
28
Figure 13. 3D Figure of Key Value in different Head in 10 layers of OPT-350M
We tested two LLMs without RoPE, OPT-2.7B and OPT-350M with the prompt Figure 11, and visualized the 20th layer for each model Figure 12. We observed series of OPT models (Zhang al., 2022), which also do not use RoPE. As shown in the figure, they also do not exhibit concentrated massive values. These results indicate that the absence of RoPE in the OPT models contributes to more uniform distribution of attention weights, as seen in the Q K and V. Unlike models that use RoPE, which tend to exhibit massive values from concentration, the OPT models demonstrate smoother and more evenly distributed attention scores across all dimensions and heads.
The uniform distribution of value in each token OPT-350M, 10 layers, with 0,3,7,11 heads, implies that OPT architecture
18

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
promotes balanced handling of imput tokens like Figure 13, This particularly evident in the absence of concentrated massive values that typically emerge in LLMs using RoPE, where attention often spikes in specific dimensions like Figure 18.
Embedding Vector of (Layer 10)
Embedding Vector Q (Layer 10)
Embedding Vector of (Layer 10)
2.8
2.6
15
15
2.4
24
27
-24
24
0.3
24
2.2
33
33
33
18
16
42
0.2
42
85
1.8
51
1.8
51
51
54
54
1.6
60
80
60
03
63
63
Number Head
Number Head
Number of Head
Figure 14. Embedding Vector of Q and Jamba, we choose Layer 10, and input question is shown as Figure 11
Jamba introduces a new hybrid Transformer-Mamba mixture-of-experts (MoE) architecture (Lieber et al., 2024). The Jamba architecture incorporates Jamba blocks, which combine Mamba layers (Gu & Dao, 2024) and Transformer modules with MoE layers between. Jamba uses eight layers of Jamba blocks, with an iicntion-to-Manme ratio of 1:7, and includes MoE layers in between. Jamba's positional encoding does not use RoPE (Rotary Positional Embedding). Therefore, in the appendix, study the attention queries (Q) and keys (K) components of the Transformer modules within Jamba. Our study reveals that Q and K have only two heads, with concentrated massive values like Figure 14, which further supports hypothesis.
Jamba Layer 10 Head Key Cache
Jamba Layer 10 Head Key Cache
Jambe Layer 10 tread D Value Cache
Jamba Layer 10 Head 1 Valuee Cache
0.8
0.8
30
077
87
0.4
34
0.2
02
66
Token
20
60
Dim
Riken
40
30
Taken
20
50
Taken
20
8
Figure 15. Figure Key Value in different Head 10 layers of Jamba-350M
C. More Visualization Results for LLMs with RoPE
Llama: Llama (Dubey et al., 2024; Touvron et al., 2023) is a cutting-edge transformer model designed with several unique structural features that optimize its performance for language modeling tasks. Below are the key characteristics of LLaMA's architecture: Pre-applied RMSNorm (Jiang et al., 2024):
RMSNom(())
27 +c
RoPE (Rotary Position Embedding) (Su et al., 2024) applied to Q and causal mask used to ensure that each position can only attend to previous tokens. LLaMA concatenates earlier K and V values the current K and values, allowing Q to retrieve earlier information:
K' = Krrev Keur]s
V'
[Vrev Veurr]
MLP formulation:
ML. (z) down()p(x)) Sil.U(gate(z)))
19

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Embedding Vector of (Layer
Embedding Vector Q (Layer 20)
Embedding Vector of V (Layer 20)
70
70
70
60
60
12
12
50
18
21
21
21
24
24
30
30
E 30
33
33
33
34
39-
42
42
42
20
20
51
51
51
100
-10
10
60
60
60
0 101221161112022222222
2001214161822222666
0
1101214161020222226
Number of Head
Number Head
Number Head
Figure 16. Embedding Vector of K. and V in Leta--laama-2-7ochhtt choose Layer 20, and the input question is shown
Figure 11
Embedding Vector of (Layer 10)
Embedding Vector (Layer 10)
Embedding Vector V (Layer 10)
70
60
12
12
12
15
15
21
24
24
40
40
30
31
33
31
30
30
39
42
42
42
85
20
45
20
20
51
51
51
10
10
63
63
63
6
4 112166682222226200
0
2
Head
Head
Figure 17. Embedding Vector of K, Q and V in Meta-Llama-33SS--nntruu choose Layer 10, and the input question shown
Figure 11
where down, up, and gate are all linear layers.
Through observations of K, Q and V in Llama3-8B, as illustrated in Figure 17 and Figure 16, we noticed that massive values are concentrated in the high-fregee high-frequency regions of the Q and K matrices. This phenomenon indicates a consistent pattern of attention concentration along certain dimensions across multiple heads. Such bright lines suggest that these dimensions are heavily weighted during the attention computation process, potentially playing a key role determining the model's focus on specific tokens features.
Both I.LMMA27BB and ILLMA3-88 exhibit massive values in K and Q across all heads Figure 19 and Figure 18, but their distribution patterns differ across heads and layers. LLaMA3-SB seems have smoother and more systematic patterns in certain heads compared to LLaMA2-7B.
LLaVA: LLaVA (Large Language and Vision Assistant) is a multimodal large model that combines visual and language capabilities, focusing on tasks that involve processing both language and visual inputs (Liu et al., 2024). Its goal is to extend the abilities of traditional Large Language Models (LLMs) to understand and generate language descriptions related to visual content. LLaVA is typically built upon architectures like LLaMA (Dubey al., 2024: Touvron et al., 2023) and integrates with vision perception models (e.g., CLIP (Radford et al., 2021)) create system with multimodal capabilities. Through structural analysis, we can identify that LLaVA utilizes RoPE in two places. Therefore, we hypothesize that its K and Q contain concentrated massive values. Our analysis further verifies this, which is shown in Figure 22.
Through observations LLaVA's K, Q. and V maps, as shown in Figure 22, noticed similar patterns to those observed in Llama, particularly in the Q and K matrices. Specifically, there is consistent bright line in the figh-freuuncc regions of the Q and embedding vectors. This phenomenon suggests that LLaVA, like Llama, exhibits pattern of attention
20

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Liama 20 Kay
Liama2 May
Liamaz:
Ney
Llamaa-77
at p et P Ioenn
pt p s Joom
12 3 e
123 Le
100
Dim
200
Dm
10
Dm
Toion
119
Figure 18. 3D Figure of Key Value in different Head in 20 layers of Llama2-7B
Llamazi-te 20 Head O Kay Cache
Liamaai-tt Layer 1D Head Key Cache
Liamad-55 10 Head Key Cache
Liamaz-iis 12 thead May Cache
125
10.0
75
10
109
100
100
122 150
5im
Dam
Dm
Dmm
12
105
Figure 19. 3D Figure of Key Value in different Head in 10 layers of Llama3-88 8B
LLWA-115 Layer 1. Heed Key Cathe
LLWAN-1. Layer 1. Heedd Key Carte
LLWVA-1 5. Layer Heed Key Cante
LLWWN-1. 5 Layer 1D Head 11 Dry Carhe
30
200
2
25
Ribon
Rim
Dm
Dm
128
128
12
128
Figure 20. 3D Figure Key Value in different Head 10 layers of LL.
Prompt in LLaVA to save vector
What are the key aspects or factors should be aware of, and what precautions or considerations should take to ensure success or avoid potential pitfalls in this image?
Figure 21. Prompt used to feed into LLM for inference
21

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
concentration along specific dimensions across multiple heads.
Embedding Vector of K (Layer 10)
Embedding Vector of (Layer 10)
Embedding Vector of V (Layer 10)
120
120
40
100
100
12
30
21
18
80
18
80
21
25
27
27
E
20
18
16
62
85
45
18
51
54
57
60
61
63
(0122411112222226666
6 1012 1436 01202224222830
6111241618202222222220
Number
Number Head
Figure 22. Embedding Vector of K. Q. LLaVA-1.5, choose Layer 10, and the input question is shown as Figure
LLaVA Output: The image shows diagram with several boxes and arrows, which represent different things. The boxes are labeled with various terms, such as "add norm," "add & norm," "forward," "add norm," "add & norm," "multi head," "masked attention," "add norm," "add & norm," "position encoding," "output encoding," and "embedding." The arrows connect these boxes, indicating relationships between them. When analyzing this diagram, is essential be cautious about context and purpose of the diagram. could be a visual representation of a complex system, network, or a process. Understanding the relationships between the boxes and arrows will help you understand the overall structure and function of the system. Additionally, paying attention to the labels and their meanings will help you interpret the diagram more accurately.
Figure 23. Output example LLaVA inference
Qwen2-VL: Qwen2- VL (Wang et al., 2024) the latest enhancement the Qwen- VL model, marks a significant leap forward in multimodal AI, offering state-of-the-art visual understanding across diverse resolutions and ratios and excelling in benchmarks. For Qwen2-VL, we have adopted three-stage training methodology similar LLaVA (Liu et al., 2024). In the first stage, we focus on training the Vision Transformer (ViT) (Dosovitskiy et al., 2021) component, utilizing large corpus of image-text pairs to enhance semantic understanding. In the second stage, we unfreeze all parameters and train with a broader range of data for more comprehensive learning, the backbone LLm is Qwen2 series models (Yang et al., 2024). In the final stage, we lock the ViT parameters and exclusively fine-tune the LLM using instructional datasets.
Embedding Vector (Layer 10)
Embedding Vector of (Layer 10)
Embedding Vector (Layer
200
175
4-
10
150
150
150
12
12
12
14
14
14
18
E
20
100
20
100
20
100
22
22
22
24
75
24
24
75
30
50
30
-50
30
50
32
32
25
-25
25
30
5.6.6.
101112131415
à 5 101112132415
Number Head
Number of Head
Number Head
Figure 24. Embedding Vector of K, Q, V ViT in Qwen2-Vin VL 7B, we choose Layer 10
Qwen2-VL uniquely incorporates Multimodal Rotary Position Encoding (M-RoPE), which differs from the traditional -ie-dimensional Rotary Position Encoding (Su et al., 2024) (ID-RoPE) used in LLMs, the latter being limited to encoding -inedmmensional positional information. M-RoPE effectively simulates the positional information of multimodal inputs by decomposing the original rotary embedding into three components: time, height, and width. For text inputs, these
22

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
components identical position IDs, making M-RoPE functionally equivalent ID-RoPE. In processing images, the temporal IDs of each visual token remain constant, while distinct IDs assigned the height and width components based on the token's 's position in the image. For videos, which are treated as sequences of frames, the temporal ID increments with each frame, and the height and width components follow the same ID assignment pattern as images. This means that RoPE is used both VIT and subsequent LLMs.. Therefore, the phenomenon of extreme values should he observable both VIT and Qwen model. However, due to the Multimodal Rotary Position Embedding (M-RoPE), which differs from the traditional one-dimensional Rotary Position Embedding (ID-RoPE) used in LLMs, the patterns on the KQ Embedding Vector in VIT differ slightly from other Embedding Vectors, but the phenomenon still exists like Figure 24.
Embedding Vector K (Layer 10)
Embedding Vector of Q (Layer 10)
Embedding Vector of V (Layer 10)
175
175
175
12
15
24
125
125
24
125
33
2
13
100
33
75
75
42
02
42
50
50
50
51
51
57
60
-0
-0
12 14 18
0.2
12 24 18
12
Number of Head
Number of Head
Number of Head
Figure 25, Embedding Vector of K. Q. V in large language model in Qwen2-VL 7B, we choose Layer 10
Qwen and Mistral:
Qwen and Mistral both use RoPE. Through observations of Qwen2.5-7B's K, Q, and maps as shown Figure 26, noticed a similar phenomenon where distinct bright regions appear in the high-frequenee areas of the Q and matrices. In particular, the Key embeddings across different heads show localized peaks with varying intensities, as evident the 3D visualizations of Figure 30. While certain heads, such as Head and Head 3, display pronounced peaks in specific token ranges.
Embedding Vector of (Layer 10)
Embedding Vector Q (Layer 10)
Embedding Vector of V (Layer 10)
8
15
15
15
18
50
18
30
24
24
74
27
40
27
40
27
40
36
30
36
30
30
45
20
45
45
20
54
54
37
63
10
14 24
12 14 18 20 24
Number of Head
Number Head
Number of Head
Figure 26. Emhedding Vector K, Q, Qwen2.5-7B, we choose Layer 10, the input question shown as Figure
GPT-NEOX:
While GPT-NeoX largely follows the GPT-3 architecture, incorporates several notable differences. One key deviation is in the positional encoding mechanism. GPT-NeoX employs Rotary Positional Embeddings (RoPE) (Su al., 2024) in place of learned positional embeddings used in earlier GPT models (Radford al., 2018). However, the implementation of RoPE in GPT-NeoX differs from the standard RoPE used in many modern LLMs. Specifically, instead of applying the rotation to the entire embedding vector, GPT-NeoX applies RoPE only to the first 25% of the embedding dimensions. This partial application, sometimes referred to as partial RoPE, was motivated by empirical observations indicating improved training stability and computational efficiency. In contrast, current models such as LLaMA (Touvron et al., 2023) apply RoPE to the full dimensionality of the query and key vectors, which is now considered the standard implementation.
23

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Embedding Vector of Key in GPT-NEOX (Layer 1)
Embedding Vector Query GPT-NEOX (Layer
Embedding Vector of Value in GPT-NEOX (Layer
50
10-
40
15
13
40
25
25
30-
15-
30
40
30
40
5 45
50-
55
20
55
20
65-
70
70-
10
10
80
10
85
45
95
95
moo00n9:1 N884488889220 20-00
Dmio ma
omid Smoo0A2NAAAa 20w.://22222202
Number
Figure 27. Embedding Vector of K Q. V GPT-NEOX 20B (with RoPE, but not the same RoPE as Llama/Gemma), we choose Layer 1 and the input question is shown Figure 11
Coand.3 75
Kay Cache
Cwan2.5-7 75 Head Kay Cache
Cear2.5-22 Head Kay Cache
Qwen2d-75 7B Layer Head Cache
Taken
100
Dim
Baen
2x
100
Din
Bken
iss
Din
Bken
Dir
Figure 28. 3D Figure of Key Value in different Head in 10 layers of Qwen2.5-7B
D. Additional Results of Disrupting Massive Values of LLMs
D.I. Perplexity Result
Table 7. Perplexity changes when massive values disrupted and non-massive values disrupted.
Model
AQUA
IMDB
GSM8K
Llama3-9B
3.21
8.38
4.17
Non-Massive Value Disrupted
3.30
8.94
4.08
Massive Value Disrupted
10.34
112.61
8.35
Gemma2-9B
4.83
14.39
6.09
Non-Massive Value Disrupted
5.22
12.66
6.14
Massive Value Disrupted
13.93
48.06
16.78
Qwen2.5-7h
2.95
8.45
1.98
Non-Massive Value Disrupted
3.12
8.64
2.21
Massive Value Disrupted
9.53
14.71
8.62
Definition of Perplexity
Perplexity is a common metric used to evaluate the quality of language models. It measures how well probabilistic model predicts sequence of text. A lower perplexity indicates better predictions by the model.
In mathematical terms, perplexity is defined as:
Input Sequence: A tokenized sequence X RRIN where N the number of tokens in the input sequence. Each
token corresponds an element w; in the vocabulary.
Model Predictions: The model predicts probability distribution P(w; W1, W2, ., w/-1) over the vocabulary for
24

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Embedding Vector of (Layer 10)
Embedding Vector Q (Layer 10)
Embedding Vector of V (Layer 10)
70
70
60
60
12
12
12
50
18
21
21
21
24
24
30
30
E 30
33
33
33
38
39
42
42
42
200
20
51
51
-10
-10
10
60
60
60
0 2 010121016182022244200
0.100012000 1012141668200 2411466688022222222
810124416122222222220
Number of Head
Number Head
Number Head
Figure 29, K, and V in Iinstaltt-t-ntttt----ll choose Layer 10, and the input question is shown Figure 11.
Layer Key
Layer Heso Key
Mistral-B Layer Head Kay
Layer
121
120
120
228
40
16
Tikon
128 100
Dim
Barnn
22
230
Dim
Biknn
126 230
Dm
Rken
228 300
Dim
Figure 30. 3D Figure of Key Value in different Head in 10 layers of Mistral
each token w,, resulting in tensor P E N where V is the size of the vocabulary.
Perplexity Computation
1. Log-Probability Computation: For each token w;, the log-probability log P(w; W1, -2, w;-1) is extracted from the predicted probability tensor P by selecting the value corresponding to the actual token w; 's index in vocabulary.
2. Averaging Log-Pree bogg-rrbbbillites:: Compute the average log-probability over the entire sequence:
N
log P(w;
W1, W2,..., w;-1)
3. Exponential Transformation:
Take the negative exponential to convert the average log-probability into perplexity:
1 N
PPL exp
log P(w; W1, wa wx-1)
The table presents perplexity (PPL) changes across three datasets (AQUA, IMDB, GSM8K) for three models (Llama3-9B, Gemma2-9B, and Qwen2.5-7B) under three conditions as Table 7. The perplexity is low in the original (Vanilla) setting, indicating that the models effectively handle these datasets under normal circumstances. For instance, Llama3-9B achieves PPL values of 3.21, 8.38, and 4.17 for AQUA, IMDB, and GSMSK, respectively.
When non massive values are disrupted, perplexity increases slightly across all models and datasets, indicating that their removal has a limited impact on language modeling capabilities. For instance, I.ama3-883 PPI. increases from 3.21 21 to
25

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
3.30 on AQUA and from 8.38 8.94 on IMDB, while Qwen2.5-7B experiences marginal rise from 2.95 to 3.12 on AQUA. These results suggest that non-massive values represent less critical features in the data, and their disruption does not significantly affect model performance. In contrast, the disruption of massive values causes a dramatic increase in perplexity, severely degrading the models' language modeling abilities. For example, Llama3-8B
jumps from 3.21 to 10.34 on AQUA and from 8.38 to 12.61 on IMDB, while Gemma2-9B sees an increase from 4.83 to 13.93 on AQUA and from 14.39 to 48.06 on IMDB. These results highlight the critical role of massive values in enabling models to accurately predict tokens, as they likely encode key semantic or syntactic information.
Overall, this experiment underscores the importance of preserving massive values in language models to maintain their ability handle various tasks effectively. While models relatively robust to the removal of non-massive values, massive values are integral to their performance. Disruption of them fundamentally affects the models' ability to model language effectively, as evidenced by the significant increase in perplexity.
D.2, Diversity Result
Diversity is metric used to measure the variety in vocabulary and sentence structures in generated texts, often applied to evaluate the quality of outputs in text generation tasks (Tevet & Berant, 2021). A text with high diversity typically contains a broader range of vocabulary and sentence patterns, reducing repetition and improving naturalness. In natural language processing, diversity is commonly evaluated by calculating the proportion of unique n-grams in the generated texts. n-grams refer to consecutive sequences of n words or characters in a text, used to capture local structures within the text.
Definition of Diversity:
The calculation of diversity can be represented by the following formula.
Let:
N: The total number of generated texts (e.g., the number sentences generated).
Ty: The i-th generated text.
total n grams: The total number of n- n-grams across all generated texts.
unique_n n grams: The total number of unique (non-repeated) n-grams across all generated texts.
The diversity is then calculated as:
unique_n_gee n_grams
Diversity (n-gram)
total n grams
Proportion of Unique n-grams
This is the core part of the metric, measuring the fraction of n-grams in the generated texts that are unique.
A higher ratio indicates lower repetition and higher diversity in the generated text.
Role of n-grams
For n (unigram): Captures diversity at the word level (lexical diversity).
For n 2 or higher: Captures diversity at the phrase or sentence level (structural diversity).
We adopted 2-grams to calculate diversity, which has the advantage capturing structural diversity at the phrase or sentence level. By analyzing 2-grams, we can measure not just the lexical variation of individual words but also how words are combined to form meaningful expressions. This approach provides a more comprehensive assessment of the model's ability to generate diverse and coherent outputs. Additionally, using 2-grams helps to detect subtle changes in diversity when certain components, such as massive values non-massive values, are removed. For example, as shown in Table 8, disrupting massive values significantly reduces diversity, especially in tasks like AQUA and GSMSK, indicating that Massive Values play a critical role in maintaining the richness of generated content. Conversely, destroying non- non-Massive Values has a relatively minor impact on diversity, suggesting that the structural patterns encoded by Massive Values are essential for maintaining variation in model outputs.
26

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Table Diversity changes when massive values are disrupted and non-massive values are disrupted.
Model
AQUA
IMDB
GSM8K
LLAMA3-9B
0.755
0,921
0.776
+ Nonn-Massive Value Disrupted
0.763
0,912
0.775
Massive Value Disrupted
0.446
0.668
0.489
Gemma2-9B
0.695
0.809
0.652
+ Non-Massive Value Disrupted
0.757
0.863
0.666
+ Massive Value Disrupted
0.675
0.085
0.720
Qwen2.5-7h
0.663
0.683
0.758
+ Non MassMMssvve Value Disrupted
0,675
0.699
0.747
+ Massive Value Disrupted
0.561
0.546
0.633
D.3. Output Examples when Massive Values in LLM are Disrupted
In the previous two chapters, observed that the IMDB dataset, the content generated by LLMs tends to deviate significantly from normal values. We selected various examples generated by different LLMs (Llama, Gemma, Qwen) to illustrate this phenomenon. The primary goal is to demonstrate that after the disruption of massive values, the diversity of content generated by LLMs drops significantly, perplexity increases drastically, and the resulting content often falls into a range that is incomprehensible to humans.
Gemma2-9B Output: It seems like you're exploring the idea of making movie! It's awesome that you're thinking about filmmaking. It sounds like you'l re playing with words and ideas, which is great part of the creative process. If you want to talk more about your movie idea, I'm here to listen and help brainstorm. What kind of movie are you imagining? PPL: 9.125
Figure 31, Result Inference Gemma
Llama Output: Answer: s and Playboy bunnies in your films! Bob Fosse's "Star 80" about Dorothy Stratten, of whom Bogdanovich was obsessed enough to have married her SISTER after her murder at the hands of her low-life husband, is zillion times more interesting than Dorothy herself the silver screen. Patty Hansen is no actress either..I expected to see some sort of lost masterpiece la Orson Welles but instead got Audrey Hepburn cavorting in jeans and god-awful "poodlesque" hair-do....Ver disappointing... "Paper Moon" and "The Last Picture Show" could watch again and again. This clunker could barely sit through once. This movie was reputedly not released because of the brouhaha surrounding Ms. Stratten's 's tawdry death; I think the real reason was because it was so bad! Judge the statement Negative or PositiveYou should explain the reason for your answer.
PPL:: 356.0
Figure 32, Result in Inference Gemma
Llama Output: I've never so many mistakes in a biopic, ranging from her early years in Celoron and Jamestown to her later years with Desi. could write a whole list of factual errors, but it would go on for pages. In all, I believe that Lucille Ball is one of those inimitable people who simply cannot be portrayed by anyone other than themselves. If I were Lucie Arnaz and Desi, Jr., would be irate at how many mistakes were made in this film. The filmmakers tried hard, but the movie seems awfully sloppy to me.
PPL:: 41. 190441 1315918
Figure 33, Result in Inference Llama3-8B
27

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Llama Output: can! This movie not for the typical film snob, unless you want to brush up on your typical cinematic definitions, like "continuity editing" and "geographic match". I couldn" 't tell where was in this movie. One second they re in the present, next minute their supposedly in the 70's driving modern SUV and wearing what looked like to me as 80's style clothing. I think it only got attention because it has a 3d which I did not watch. If you' re a b-movie buff, and by b-movie I mean BAD movie, then this film is for you. It's home-movic and all non-sense style will keep you laughing for as long as you can stay awake. If your tastes are more for Goddard and Antonioni, though, just skip this one. PPI.: 42912700000007
Figure 34, Result in Inference Llama3-8B
Llama Output: am a helpful movie critic who can help me. Answer 'Negative' or 'Positive' I should explain the reason for my answer.
PPL: 72.0250701904296
Figure 35, Result in Inference Llama3-8B
Qwen Output: Answer: to continue the story, we could start with a scene where Lena, the main character, is at a film festival, perhaps in Stockholm, discussing the themes and techniques of Ingmar Bergman's 's films with other film enthusiasts. This could lead into a broader discussion about the intersection of art, politics, and social commentary in Swedish cinema, setting up the backdrop for Lena's exploration of these themes in her own work.
PPL.: 21.375
Figure 36. Result in Inference Qwen
E. Experiment Details
Large Language Models: For the main table as Table in the body text, we used three classic models with RoPE:
Lama-3-8B-Instruue (Duhey et al., 2024), google/gemmaa-29bbi (Team et al., 2024), and Qwen2.5-7BBInsttuue (Yang et al., 2024). In the appendíx, we further supplemented the experiments with models including opt-2.7b (Zhang et al., 2022), Qwen2-VL-22 B---VLL--B----truee (Wang et al., 2024), 3lama-2-7b-chat-hi (Touvron et al., 2023), Jamba-v0.1 (Lieber et al., 2024), and Histral-7B-Instrucct In-ttrrrcct--O. (Jiang et al., 2023). Among these, some models have RoPE while others do not. During inference, we consistently used flash lithhatttenion____ 2 (Dao et al., 2022) for faster inference speeds.
Save Embedding Vector: We save Embedding Vector after RoPE
Datasets:
Cities (Marks & Tegmark, 2023): consists of statements about the location of cities and their veracity labels (e.g., The city of Zagreb is in Japan, which is wrong). We use 1496 of these samples.
"True" example: Japan.
The city of Tokyo is in Judge the statement is True or False.
"False" example: Judge the statement is True or False. The city of Lodz is in the Dominican Republic.
GSM8K (Cobbe et al., 2021): GSM8K (Grade School Math 8K) is a dataset consisting of 8.5K high-quality, linguistically diverse grade school math word problems. The dataset is designed to support question answering tasks for basic mathematical problems that require multi-step reasoning. These problems typically require 2 to 8 steps to solve. The solutions primarily involve performing a sequence of basic arithmetic operations to arrive at the final answer. A bright middle school student should be able to solve all the problems. The paper states: "These problems require no concepts beyond the level of early Algebra, and the vast majority of problems can be solved without explicitly defining variable. In our experiments, we used the first 1,000 samples from the training set of GSM8K and we use Figure 37 as the system prompt.
GSMBK Example: The ratio of coins that Elsa has to that which Amalie has is 10:45. the total number of coins they have is 440, and Amalie spends 3/4 of what she
28

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
System Prompt
LLM to solve Math probl em
You are helpful math expert who can help me. Put the final option and answer at the end of the sentence. Do not show other incorrect options.
Figure 37. Prompt in Inference LLM
has on toys, how many will she remain with?
GSM8K Example: Nancy is filling an aquarium for her fish. She fills it halfway and goes to answer the door. While she's a gone, her cat knocks the aquarium over and spills half the water in it. Then Nancy comes back and triples the amount of feet long, feet wide, and feet
water in the aquarium. If the aquarium high, how many cubic feet water are in the aquarium?
AQUA (Ling et al., 2017): Dataset Card for AQUA: AQUA is a large-scale dataset consisting of approximately 100,000 algebraic word problems. Each question is accompanied by step-by-step solution explained in natural language. This dataset is designed to train program generation models that can both generate explanations and create programs to solve the given questions. We treat AQUA the same way as GSM8K in our experiments. Below is an example.
AQUA Example
Question: Two friends plan to walk along a 43-km trail, starting at opposite ends of the trail the same time. If Friend p's rate is 158 faster than Friend Q's, how many kilometers will Friend P have walked when they pass each other? Options: [ 21", "B)21.5", "C) 22", 22)"2, "E)23"
Rationale:
If Q complete kilometers, then P completes 1.15x kilometers. x + 1.15x = 43 2. 2.15x=43 x = 43/2.15 = 20 Then P will have have walked 1.15+20=23 km.
The answer is E.
IMDB (Maas et al., 2011): Large Movie Review Dataset: This dataset designed for binary sentiment classification and contains significantly more data than previous benchmark datasets. It includes 25,000 highly polar movie reviews for training and 25,000 for testing, along with additional unlabeled data for further use. For our experiments, selected 1,000 samples from the dataset and instructed the LLM to classify each review as either positive or negative sentiment based on provided system prompt like Figure 38.
System Prompt in LLM to judge "Negative" "Positive'
You are a helpful movie critic who can help me. Answer "Negative" or "Positive"
Figure 38. Prompt in Inference LLM
IMDB Example 1: I rented I AM CURIOUS-YELAN the controversy that surrounded it when it was first released in 1967. also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered "cont I really had to see this for myself. <br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions what the average Swede thought about certain political issues such the Vietnam Mar and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men. <br /><br />What kills me about I AM CURIOUS-YELLON is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it's not
from my video store because of all
making some sort of documentary
29

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
1,0
Vanilla
Massive Value Destroy
0.8
0,733
ACC (%6
0.6
0.4
0.2
0320
8000
0.0
Qwen2.5-78
Llama3-88
Mistral-78 Models
Llama2-8B
Figure 39. Greater-Less Experiment
Table 9. Effect Disrupting Massive Values in Models Without RoPE
Model
Massive Value Disrupted
GSM8k (%)
Cities (%)
OPT-2.7B
No
5.0
76.0
OPT-2.7B
Yes
2.0
60.1
lamba-1.5-min
No
72.8
96.5
Jamba-1.5 miimmi
Yes
50.2
75.2
shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are major staple in Swedish cinema. Even Ingmar Bergman, arquably their answer to good old boy John Ford, had sex scenes in his films. /><br />I do commend the fi filmaaeess for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America.
AM CURIOUS-YELION is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn't have much a plot.
F. More Experiments to Support the Conclusion.
Greater-Less Experiment:
We used our synthetic dataset to test Qwen2.5-7B, Llama3-8B, Llama2-7B and Mistral-7B. The synthetic dataset has been introduced previously as Appendix A, the result is shown as Figure 40.
Based the results shown in Figure 40, the "Vanilla" configuration consistently outperformed the "Massive Value Destroy" setting across all models in the Greater-Less Experiment. For instance, Qwen2.5-7B achieved the highest accuracy in the Vanilla setting at 0.77%, while its accuracy dropped significantly to 0.03% under the Massive Value Destroy setting, Similarly, Mistral-7B displayed drop from 0.59% in the Vanilla setting to 0.12% in the Massive Value Destroy setting, illustrating the impact of Massive Value removal on the model's performance.
Disrupting Massive Values in LLMs without RoPE.
We also examined the effect of disrupting massive values LLMs without adopting RoPE. We conducted these disruption experiments using OPT-2.7B (Zhang al., 2022) and Jamba-1.5-ming (Lieber al., 2024) on GSM8k (in-context retrieval) and Cities (parametric retrieval) benchmarks before and after disrupting the massive values. From Table 9, that disrupting massive values in models without RoPE leads to significant performance degradation across both tasks: OPT-2.7B showed 60% relative decrease in GSM8k and a 21% % decrease in Cities, while Jamba-1 comba-l.5-mmin -i5mmini exhibited 31 % decrease in GSM8k and a 22% decrease in Cities. This substantial impact on both retrieval types suggests that, unlike in RoPE-based models where positional and semantic information could be separated, massive values in non-Rot non-RoPE models encode a blend of both information types.
observe
30

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
5,80
Vanilla
Massive Destroy
0,8
ACC (%)
0.4
0.00
0.0
Ulma3388 Models
Gemma-96
Figure 40. Readability Rate Experiment
Emmeedding Vector (Layer 1)
Embedding Vector (Layer 2)
Embedding Wector of (Layer 1)
Embedding Vector of (Layer 2)
30
2.50
Number ofdaod
Number Hfddod
HunderofHood
NumberafHcad
Figure 41. Embedding Vector K in Llama-3-8B and Qwen2.5-7B, we choose Layer and and the input question shown
Figure 11
Experiment for Other Metrics:
We believe that perplexity as a metric is somewhat one-dimensional. For instance, when large language models exhibit a "repetition phenomenon," such as in the following example: "divisible by 9 and 12, which is the product of X, if is divisible by 9 and 12, which is the product of X..." (repeated endlessly), the perplexity score remains as low as 2.99. Despite the low perplexity, the generated sentence is clearly incomprchensible to humans.
To address this limitation, combined human judgment with GPT-4 evaluations assess the readability of these sentences. metric: Readability Rate. This metric aims better capture the comprehensibility of text,
From this, introduced going beyond what perplexity alone measure.
For example: the output from Owwe2-7BB is not readable: Answer: you're right, the premise of a "mystic man who eats women" is quite disturhing and inappropriate. Let's shift gears to something more constructive. Could you tell me about a topic you't re interested in or need help with? Perhaps we could have thoughtful discussion on subject like philosophy, science, or literature instead? That might be more enjoyable and productive.
Explore when Concentrated Massive Values Appear:
By observing the Embedding Vectors of Qwen2.5-7B, Llama3-8B, Llama2-7B, and Mistral-7B, it was found that this phenomenon appears as early as the first layer like Figure 42.
In each layer, when extracting the QK embedding vector map. we consistently observe that there are no significant differences between the results taken before and after applying RoPE, The patterns remain largely consistent. This phenomenon the same across all layers. Initially, we assumed that this might not be the case the first layer. However, we later found that occurs in every layer. To further investigate, we specifically selected the 20th layer of LLaMA3-SB for observation.
Disruption Choices Exert Subtle Influences in Gemma:
The table comprehensively demonstrates the impact of various Massive Value handling methods (Mean, Zero, Minimum, and Maximum) on Gemma-99 9 model's performance across multiple benchmarks including GSM8k, AQUA, IMDB, and Passkey Retrieval with different configurations. The baseline Gemma2-9B model serves as a reference point, showing scores of 81.30 for GSM8k, 63.8 80 for AQUA, and 94.70 for IMDB, alongside perfect 100% performance all Passkey Retrieval
31

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Embooding of (Layor Before Rope!
Embodding of [Layer Rope)
bodcing K (Layer 10, Ropo)
Embeddinn [Layer Rope)
8
33
20
20
20
20
Number of Iteed
Number of iteed
Number ifdd
Number of ofee
Figure 42. Embedding Vector of K in Llama-3-8B, choose Layer and 10 before RoPE and after RoPE, and the input question shown as Figure 11
Table 10. Impacts of applying different methods to disrupt massive values/non-massive values model performance (Gemma-90)
Passkey Retrieval
Operation
GSM8K
AQUA
IMDB
(128,6)
(256,12)
(1024,48)
None
81,30
63.80
94.70
100
100
100
Massive Value
15.10
16.50
1,80
2.00
0.00
0.00
Mean
Non-Massive Value
81.60
65.60
97.40
100
100
100
Massive Value
14.30
15.90
1.60
2.00
0.00
0.00
Zero
Non Non-Massive Value
81,20
65.10
96.90
100
100
100
Massive Value
15.20
16.80
2.10
2,00
0.00
0.00
Min
Non-Massive Value
81.90
65.90
97.80
100
100
100
Massive Value
14.70
16.20
1.90
Max
Non-Massive Value
81.40
65,30
97.20
100
100
configurations. The data reveals that regardless of the Massive Value handling method employed, the model maintains remarkably stable performance, particularly when dealing with non- non-Massive Values. While there are slight variations the treatment of Massive Values, these differences minimal and don't significantly impact the model's overall effectiveness. The Passkey Retrieval tasks consistently maintain perfect performance for non-Massive Values across all configurations, highlighting the model's robustness. This stability across different handling methods suggests that Gemma-9b possesses strong inherent resilience to various data processing approaches, with particularly notable stability in handling non- non-Massive Value cases, while still maintaining acceptable performance even when processing massive values.
Controlled Experiment Non Massive Value Region Disrupted in Gemma:
Model Performance Across Metrics
90
Performance (%)
80
GSMBK
AQUA
IMDB
60
vanilla
Models: Qwen2.5-78
Figure 43, Readability Rate Experiment
As illustrated in Figure 43, our controlled experiments with Qwen2.5-7B demonstrate consistent patterns across different benchmarks. The performance across all three tasks (GSM8K, AQUA, and IMDB) remains notably stable when varying the number of minimum values replaced (from 1 to 20). GSM8K maintains accuracy between 75-85%, IMDB consistently performs above 90%, and AQUA stays within the 51-57% range. This stability from vanilla performance through various
32

---

Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding
Model
Position Embedding Concentrated Massive Value
GPT-2
Absolute
Yes
GPT-2-Neo
Absolute
No
GPT-2-NeoX
RoPE
Yes
Table 11. Comparison of concentrated massive values across models with different positional embeddings.
Embedding Vector of Key in GPT-NEO (Layer]] Embedding Vector of Query in GPT NEO (Layer 1)
Embedding Vector Value in GPT-NEO (Layer
0
50
24
36
48
30
30
80
72
34
108
-10
158
10
114
130
126
126
1011121114439
4 101112111115
101112131415
Number of Head
Number of Head
Number of Head
Figure 44. Embedding Vector of K Q, V in GPT-NEO-1.3B (without RoPE), we choose Layer 1 and the input question is shown Figure
11
Embedding Vector Key in GPT-NEOK (Layer 1)
Embedding Vector Query GPT-NEOX (Layer
Embedding Vector of Value in GPT-NEOX (Layer 1)
10-
70
46
40
20
40
25
25
15-
30
40
46
35
30
30
10
55
40-
65
65
65
80
15-
10
80
11
80
10
90
55
95
95
Number Head
Number of HHlded
Number of Head
Figure Embedding Vector Q. GPT-NEOX- (with RoPE, but not same RoPE Llama/Gemma), we choose Layer 1 and the input question is shown as Figure 11
degrees of non-Massive Value manipulation reinforces our hypothesis that these regions play relatively minor role in the model's 's capabilities. The consistent performance across different numbers of replaced values suggests that the model's Context Knowledge Understanding abilities remain intact as long as the Massive Values are preserved, regardless of how many non-Massive Values are modified.
RoPE contributes to Concentrated Massive Values:
We want to clarify that we concentrate on identifying and explaining specific pattern of massive values that consistently appear in models using RoPE. At the same time, we observe that in models without RoPE, massive values still appear but do not consistently have a concentration trend. From the results we observed in Jamba, GPT-2 and OPT shown in Table 11 of our paper, concentrated massive values could only be observed in models employing RoPE. To further validate this, we conduct experiments across three models with similar architectures GPT-2-Neo, and GPT-2-NeoX, just focus the impact of positional embedding differences and explore the shape of their massive values. Interestingly, the massive values in GPT-2 concentrate along the head (seen from Figure 46) while GPT-2-NeoX concentrates on the depth dimension (seen from Figure 45).
different positional embedding methods: GPT-2,
33

---

Massive Values in Self-Attenion Modules are the Key to Contextual Knowledge Understanding
Embedding
Key (Layer 1)
Embedding Vector Query GPT2 [Layer
Embedding Vector of Value GPT2 (Layer
30
9-
12
40
15
40
18
30
27
21
30
27
24
30
13
20
20
02
42
42
48
51
10
51
10
57
60
Number Head
Number of Head
Number of Head
Figure 46. Embedding Vector of K Q, V in GPT-2 (without RoPE), choose Layer and the input question shown Figure 11
34


Reviews:
[{"review_id": "CjBFHMLWQ3", "reviewer": "Reviewer_XgBB", "summary": "The authors study the case of Massive values K,Q values which appear in LLMs.\nThe authors conclude that the primary reason for this ROPE embeddings.\nAnother interesting finding the authors proposed was that the massive K,Q values have different impact in different types of task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: Several prior works like Cachegen StreamingLLMs have similar insights. My primary concern is that some of these findings are not new. I think the study is quite interesting, however the impact is quite unclear. For example- Existing optimization uses similar insights.\n\nThank you for pointing out these related works [1, 2]. While these works do explore caching mechanisms in LLM inference, we must emphasize that our research differs fundamentally from these prior works in focus: CacheGen and StreamingLLMs primarily investigate the positional dynamics of tokens in attention mechanisms, analyzing how massive attention values at specific tokens impact model performance. In contrast, our research specifically examines the **massiva values in the Q and K embedding vectors along the embedding depth of tokens**, which analyzes massive values in a different subspace to provide complementary insights to existing approaches.\n\n> W2: This does not clearly answer the question of why RoPE embeddings are causing this. Secondly it does not compare how not having these large values in case of Jamba/GPT-2 affects performance.\n\nWe would like to clarify that we **concentrate on identifying and explaining a specific pattern of massive values that consistently appears in models using RoPE** while we **do not explain the origin of massive values in transformers in general**. Prior works [5, 6] have observed the existence of massive values in large language models but did not provide a complete explanation of how these massive values emerge during the training process. Our contribution does not aim to resolve this broader problem fully. \nFrom the results we observed in Jamba, GPT-2 and OPT shown in Table 3 of our paper, concentrated massive values could only be observed in models employing RoPE. To further validate this, we conduct experiments across three models with similar architectures but different positional embedding methods: GPT-2, GPT-2-neo, and GPT-2-neox to just focus on the impact of positional embedding differences. \n\n| Model  | Position Embedding | Concentrated Massive Value |\n|--------|------------------|------------------|\n| GPT-2   | Absolute | No  |\n| GPT-2-Neo   | Absolute | No |\n| GPT-2-NeoX   | RoPE | Yes |\n\n\nThe results show that concentrated massive values can only be found in  GPT-2-NeoX employing RoPE, which is consistent with what we have observed.  In addition, we conduct experiments of disrupting all the massive values in OPT and Jamba on GSM8k and Cities, respectively (we do not consider GPT-2 as it does have the capability to fulfill any of the tasks. )\n\n| Model  | Massive Value Disrupted | GSM8k (%) | Cities (%) | \n|--------|------------------|------------------|------------------|\n|  OPT-2.7B   | No | 5.0  | 76.0  |\n| OPT-2.7B  | Yes  | 2.0  | 60.1  |\n| Jamba-1.5-mini   | No | 72.8  | 96.5  |\n| Jamba-1.5-mini   | Yes | 50.2  | 75.2  |\n\n\nThe results show obvious **performance degradation in both in-context retrieval and parametric retrieval when disrupting massive values in models without using RoPE**. This implies that unlike RoPE where positional and semantic information could be separated into high and low dimensions of Q and K embedding vectors, massive values in models without RoPE appear to blend with positional and semantic information. \n\n> W3: It will be interesting if authors can perform a study with different types of embeddings like AliBi. \n\nThank you for mentioning this point. We conducted this exact comparison between RoPE and ALiBi using Baichuan-2-7B (using RoPE) and Baichuan-2-13B (using ALiBi) [3]. The results can be seen from the following table. \n\n| Model  | Position Embedding | Concentrated Massive Value |\n|--------|------------------|------------------|\n| Baichuan-2-7B   | RoPE | Yes  |\n| Baichun-2-13B   | AliBi | No |\n\nThe results revealed that **concentrated massive values can be observed in Baichuan-7B while absent in Baichuan-13B. This difference can be attributed to the fundamental mechanisms of these positional embeddings**. Unlike RoPE, ALiBi applies a simple linear bias based on relative token distances, which maintains a more uniform attention distribution that inhibits the emergence of concentrated massive values. Complete experimental results will be included in the updated version of our paper.\n\n[1] Liu et al. Cachegen: Kv cache compression and streaming for fast large language model serving. 2024 Sigcomm\n\n[2] Xiao et al. Efficient streaming language models with attention sinks. 2023\n\n[3] Yang et al. Baichuan 2: Open Large-scale Language Models. 2023\n\n[4] Barbero F, Vitvitskyi A, Perivolaropoulos C, et al. Round and Round We Go! What makes Rotary Positional Encodings useful? In ICLR 2025.\n\n[5] Jianlin Su, Murtadha Ahmed, Yu Lu, et al. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.\n\n[6] Sun M, Chen X, Kolter J Z, et al. Massive Activations in Large Language Models. In COLM 2024.", "claims_and_evidence": "The authors make following key claims - \n\nMassive values are concentrated in specific regions of Q and K exclusively\n\nMassive values in Q and K are critical for understanding contextual knowledge over parametric knowledge.\n\nQuantization techniques targeting massive values preserve contextual knowledge better\n\nConcentration of massive values is caused by RoPE and it appears since very early layers in LLMs. -> How does not having RoPE affect the accuracy.", "methods_and_evaluation_criteria": "The authors have created a reasonable study setup to verify the claims", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "Yes. Checked the study design for studying different dataset performance.", "relation_to_broader_scientific_literature": "Several prior works like Cachegen, StreamingLLMs have similar insights. My primary concern is that some of these findings are not new.", "essential_references_not_discussed": "There are quite a few works which are inherently dependent on similar insights. For example - Efficient streaming language models with attention sinks, look at similar insights that different position encoding locations.", "other_strengths_and_weaknesses": "- I think the study is quite interesting, however the impact is quite unclear. For example- Existing optimization use similar insights. However, this does not clearly answer the question of why RoPE embeddings are causing this. Secondly it does not compare how not having these large values in case of Jamba/GPT-2 affects performance. \n\n- It will be interesting if authors can perform a study with different types of embeddings like AliBi", "other_comments_or_suggestions": "None", "questions_for_authors": "See weakness and strength section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ey6FsjN0VJ", "reviewer": "Reviewer_3Gju", "summary": "This paper investigates the massive value elements in output vectors of internal layers of Transformer from the viewpoint of their influences on task-solving performance and the reason for them. The empirical analysis of various large language models (LLMs) reveals that the massive values influence tasks requiring context understanding rather than tasks requiring stored knowledge in pre-training. Furthermore, by conducting an empirical ablation study across different LLMs, the authors obtain a result indicating the relationship between Rotary Positional Encoding (RoPE) and massive values.\n\n## update after rebuttal\n\nThe authors shared some details about hyperparameters through the discussion, which must improve the reproducibility of this paper. However, the fundamental issue of the lack of theoretical explanation remains unresolved. Thus, I keep the current updated score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> S1: subsection 3.5: (Yu et al.) -> (Yu et al. 2023)\n\nThank you for your suggestion. We will correct the citation in subsection 3.5 from (Yu et al.) to (Yu et al. 2023) to ensure proper citation formatting.\n\n> Q1: How the massive value occurs is still uncertain.\n\nThank you for raising this critical point. The global phenomenon of massive values in LLMs is still under active investigation, our contribution does not aim to resolve this broader uncertainty fully. Instead, our work focuses on a specific and consistent pattern we identify within the QK Embedding Vector: the presence of banded, stripe-like massive values that arise due to the structure of RoPE. We fully agree that the origin of massive values in transformer attention mechanisms remains an open question. Prior work has observed their existence in large language models but did not provide a complete explanation [1, 2, 3]. It did not explain some questions, like how massive values appear in the training process.\n\n> Q2: In contrast to the empirical analysis, this work lacks theoretical aspects.\n\nWe acknowledge the concern regarding the lack of theoretical analysis. Indeed, the theoretical understanding of LLM, particularly regarding internal mechanisms like attention concentration and the emergence of massive values in QK, is still in its early stages [1, 3].  As noted in several related works, many empirical studies in this area also primarily rely on observations and interventions rather than formal proofs or theoretical frameworks due to the inherent complexity and scale of modern transformer-based LLM [3]. Therefore, we thank the reviewer for this valuable suggestion and will add the theoretical proof for our future work.\n\n\n> Q3: How did you set λ? If you tuned it on validation datasets, please teach the used model for the decision.\n\nThank you for your insightful question. We would like to clarify that the threshold parameter λ = 5 was not tuned on any validation set. Instead, this value was selected empirically based on statistical observations across all models and datasets used in our experiments. Concretely, we analyzed the query and key embeddings (x-axis is head, y-axis is dim) across all layers of three LLMs on three datasets (GSM8K, AQUA, and IMDB). **We consistently observed that the massive values in each head were typically at least 5× larger than the mean value of its head. This motivated our choice of λ = 5 as a conservative and robust threshold to capture the most prominent outliers.** We also confirmed that moderate variations around this threshold (e.g., λ ∈ [5, 6]) did not qualitatively affect our findings. We will include a clarification in the final version and optionally add this robustness observation to Appendix F. The following table summarizes the models, layers, and datasets we used in the statistical analysis.\n| Model       | Layer   | GSM8K | AQUA | IMDB |\n|-------------|---------|-------|------|------|\n| Gemma2–9B   | Layer 1 | √     | √    | √    |\n|             | Layer 15 | √     | √    | √    |\n|             | Layer 27| √     | √    | √    |\n| Llama3–8B   | Layer 1 | √     | √    | √    |\n|             | Layer 15 | √     | √    | √    |\n|             | Layer 27| √     | √    | √    |\n| Qwen2.5–7B  | Layer 1 | √     | √    | √    |\n|             | Layer 15 | √     | √    | √    |\n|             | Layer 27| √     | √    | √    |\n\n\n> Q4: What is the definition of the non-massive value in Table 1?\n\nThank you for your insightful question. In Table 1, \"non-massive value\" refers to the value **selected from the non-extreme regions of the Q/K embedding vector**, i.e., value in dimensions outside specific dimensions where concentrated values occur. In the next version, we will clarify this definition more explicitly to avoid ambiguity.\n\n\n> Q5: How did you choose the value (dimension) for disrupting the non-massive value in Table 1?\n\nThis point is closely related to Q4, and we appreciate the opportunity to clarify it further. To select the non-massive value, we first analyzed the Q/K embedding vector's shape (with heads along the x-axis and dimensions along the y-axis). For each head index in the x-axis, we excluded the region we previously identified as containing massive values. Then, **we randomly selected one dimension outside the region where concentrated massive values occur to choose non-massive values in this head**. This process is repeated for each head index along the x-axis. We then applied the same disruption method: setting the selected dimension to the mean value. We will describe this procedure more clearly and explicitly in the next revision. Thank you for pointing this out.\n\n\n\n\n\n\n\n\n\n\n[1] Sun M, et al. Massive Activations in Large Language Models. In COLM.\n\n[2] Wang T, Liu Y, et al. M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning. In EMNLP.\n\n[3] Barbero F, et al. Round and Round We Go! What makes Rotary Positional Encodings useful? In ICLR.", "claims_and_evidence": "The claim for the existence of the massive values in LLMs and RoPE as their reason is actually shown in the empirical results based on various LLMs with various datasets. However, the authors seem to arbitrarily choose $\\lambda$, which decides the threshold of the massive value, and it may raise some questions about the observed results.", "methods_and_evaluation_criteria": "It makes sense since the authors used well-known benchmark datasets and their standard metrics.", "theoretical_claims": "Almost all claims are justified by empirical experiments. Thus, there is no theoretical proof.", "experimental_designs_or_analyses": "I think it's appropriate except for some hyper-parameters. I'll decide its validity through the discussion phase.", "relation_to_broader_scientific_literature": "The massive value becomes an obstacle in quantization and weight pruning. Thus, this research influences various NLP tasks on LLMs when these require efficiency.", "essential_references_not_discussed": "Related works are appropriately cited.", "other_strengths_and_weaknesses": "Strengths:\n- This work reveals the massive value is more influential on tasks requiring context understanding rather than tasks require stored knowledge in pre-training.\n- The empirical analysis also reveals the relationship between RoPE and the massive value.\n- Experimental results are based on various models with various datasets.\n\nWeaknesses:\n- How the massive value occurs is still uncertain.\n- In contrast to the empirical analysis, this work lacks theoretical aspects.\n- Some settings, like the tuning of $\\lambda$ and how to define the non-massive value, are not well described.", "other_comments_or_suggestions": "The last paragraph in subsection 3.5: (Yu et al.) -> (Yu et al. 2023)", "questions_for_authors": "- How did you set $\\lambda$? If you tuned it on validation datasets, please teach the used model for the decision.\n- What is the definition of the non-massive value in Table 1?\n- How did you choose the value (dimension) for disrupting the non-massive value in Table 1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUSSR2xHK5", "reviewer": "Reviewer_aSeD", "summary": "This paper investigates the emergence of massive values in self-attention within LLMs and their impact on contextual knowledge understanding. Through extensive experiment, the authors demonstrate that these massive values play a crucial role in interpreting contextually provided knowledge rather than retrieving parametric knowledge stored in the model’s parameters. They further show that these massive values arise due to RoPE and persist across multiple layers. Additionally, the paper explores how quantization strategies that ignore these massive values can significantly degrade performance on tasks requiring contextual understanding.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: Lacks a deep analysis of attention pattern changes due to massive values.\n\nWe thank the reviewer for pointing this out. We agree that a deeper analysis of how massive values affect attention patterns would strengthen the paper, and we will include such an analysis in the revision. Specifically, we have conducted a new experiment that compares attention scores under two settings: \n\n1. **With massive values intact**, and  \n2. **With massive values disrupted**.\n\n\nWe computed the average attention scores across all tokens in one sentence in both settings and observed that: \n\n- When massive values are **not disrupted**, the LLM produces **higher attention scores** per token, reflecting concentrated and confident attention.\n- When massive values are **destroyed**, attention scores become **much lower and diffuse** per token, indicating weaker attention focus.\n\nTo ensure robustness, we performed this experiment on three datasets (GSM8K, AQUA, and IMDB) using LLaMA3–8B. For each dataset, we sampled 100 examples and measured the token-level average attention score difference between the two settings.\nThe new experiment shown below as the table provides concrete evidence that massive values significantly shape the model’s attention behavior. \n\n| Dataset | Model     | Intact Attention Score (Average) | Destroyed Attention Score (Average)  | Difference (Intact − Destroyed) | Notes                        |\n|---------|-----------|--------------|-----------------|----------------------------------|------------------------------|\n| GSM8K   | LLaMA3–8B | -0.1028      | -0.5039         | 0.4011                           | Higher when massive intact   |\n| AQUA    | LLaMA3–8B | -0.0633      | -0.3748         | 0.3115                           | Higher when massive intact   |\n| IMDB    | LLaMA3–8B | -0.1837      | -0.4919         | 0.3082                           | Higher when massive intact   |\n\n\nAs shown, **the average attention score significantly decreases when massive values are disrupted across all datasets**. This highlights their functional role in massive value in LLM. We will include the analysis and visualizations in the updated version.\n\n\n\n\n\n\n\n> W2: The theoretical justification could be made more formal.\n\nWe appreciate the reviewer's feedback regarding the formality of the theoretical justification and agree that more rigorous theoretical proof would strengthen the paper. However, indeed, the theoretical understanding of large language models, particularly regarding internal mechanisms like attention concentration and the emergence of massive value/ activation, is still in its early stages [1, 2, 3]. Our primary contribution lies in the empirical demonstration of how banded massive values emerge in the QK embedding vectors and how they functionally impact attention behavior. We show that these values consistently align with semantically important tokens and drive attention concentration. Furthermore, through carefully controlled interventions, we isolate the role of RoPE in forming these banded patterns.  As noted in several related works, many empirical studies in this area also primarily rely on observations and interventions rather than formal proofs or theoretical frameworks due to modern transformer models' inherent complexity and scale [1, 2, 3, 4]. \n\n\nTherefore, we thank the reviewer for this valuable suggestion and leave the theoretical proof as our future work.\n\n[1] Sun M, Chen X, Kolter J Z, et al. Massive Activations in Large Language Models. In COLM 2024.\n\n[2] Barbero F, et al. Round and Round We Go! What makes Rotary Positional Encodings useful? In ICLR 2025.\n\n[3] Yu M, Wang D, Shan Q, et al. The super weight in large language models. arXiv preprint 2024.\n\n[4] Zhang S, Papyan V. Oats: Outlier-aware pruning through sparse and low rank decomposition. arXiv preprint 2024", "claims_and_evidence": "The claims are generally well-supported by experiments, particularly through controlled ablations that show the causal impact of massive values on contextual knowledge understanding. However, further analysis of how massive values influence attention patterns and token interactions would strengthen the argument.\n\nA notable limitation is the evidence supporting the claim that RoPE directly causes the emergence of massive values. While the authors correlate the presence of massive values with models that use RoPE (Table 3), this evidence is not entirely conclusive, as different models undergo distinct training procedures that may introduce confounding factors. A stronger validation could involve controlled experiments comparing architectures with and without RoPE under identical training conditions.", "methods_and_evaluation_criteria": "The authors conduct a thorough analysis using:\n\n+ Diverse LLMs (e.g., LLaMA, Gemma, Qwen) to ensure generalizability.\n+ Controlled ablation studies (disrupting massive values selectively) to isolate their impact.\n+ A range of contextual and parametric knowledge tasks to validate the importance of massive values.\n+ Comparative analysis of quantization strategies, offering practical insights into model optimization.\n\nOne limitation is that the paper does not explicitly investigate how massive values shape attention distributions and their direct influence on token importance within the context. Analyzing attention heatmaps or token importance shifts due to massive values would enhance the interpretability of the findings.", "theoretical_claims": "The paper does not provide any formal theoretical claims regarding the emergence or impact of massive values.", "experimental_designs_or_analyses": "The disruption experiments convincingly show the causal impact of massive values on contextual knowledge understanding. However, additional visualization of attention patterns before and after disruption would make the results more interpretable. Also, the analysis could benefit from further exploration of which specific attention heads or layers contribute most to contextual understanding via these massive values.", "relation_to_broader_scientific_literature": "This paper connects well with prior work on:\n\n+ Quantization-aware optimization (e.g., AWQ, SmoothQuant), showing how activation outliers affect model performance.\n+ LLM representation analysis, particularly studies on RoPE and attention patterns.\n+ Contextual vs. parametric knowledge distinction in LLMs, contributing new empirical evidence to this ongoing discussion.\n\nA useful extension would be comparing findings to recent studies on attention head importance and pruning to see if massive values correlate with known influential heads.", "essential_references_not_discussed": "The paper sufficiently discusses relevant prior work.", "other_strengths_and_weaknesses": "Strengths\n+ Novel and interesting discovery about the role of massive values in Q/K.\n+ Comprehensive experiments demonstrating clear causal effects.\n+ Practical implications for LLM quantization and optimization.\n\nWeaknesses\n+ Lacks a deep analysis of attention pattern changes due to massive values.\n+ The theoretical justification could be made more formal.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sTwuj5LeHm", "reviewer": "Reviewer_p2F4", "summary": "The paper empirically observed that LLM with RoPE would have massive QK values, which has positive correlation to the knowledge understanding capability.\n## updated after rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> D1: Allen-Zhu and Li also highlighted the importance of RoPE. Please provide a discussion.\n\nThank you for the excellent suggestion. We acknowledge that our current version does not yet include a discussion of Allen-Zhu and Li [1]. We appreciate the reviewer’s input and will include proper citations and debate in the next version.\n\nTheir work provides valuable insights into how different position-encoding mechanisms affect LLM performance. While we share similar interests, our focus is distinct: they emphasize performance, whereas we analyze how RoPE leads to massive values concentrating on specific dimensions of the K/V embeddings and how these values are crucial for contextual knowledge understanding. We believe their findings complement ours and will make this connection more explicit in the next revision.\n\n> Q1: Why isn’t there a concat operation over Q in (5)?\n\nThank you for the question. The absence of concatenation over the Query matrix Q in Equation (5) is intentional and follows the standard design in transformer-based LLM. During autoregressive generation, **Q is recomputed** at each decoding step for the current token; **K and V are cached** across time steps to avoid redundant computation [2]. Our formulation mirrors the **prefill-stage equations** used in prior work [2, 3], where concatenation is applied only to K and V (the cached tensors) but not to Q. Since the Query is computed afresh at each decoding step and is not cached, it does not require concatenation with past representations.\n\n> Q2: Can LLMs with RoPE still correctly interpret the meaning of \"capital\" in different contexts and respond accurately after applying disruptions?\n\nThank you for the insightful question and illustrative examples. In case (i) What is the capital of America: A; What is the capital of Japan: without disruption, the model infers that “capital” refers to capital letters, responding with J. However, the model loses this contextual understanding under our massive value disruption and defaults to Factual knowledge recall, answering Tokyo.  In case (ii) “What is the capital of America: Washington DC; …” The context frames “capital” in a geopolitical sense. Thus, the model answers “Tokyo” in both settings, with or without disruption. \n\nWe construct a 100-example test set in two formats to validate like the table below. Results consistently show that **disruption suppresses contextual understanding, forcing models to rely on memorized knowledge**. We will include the evaluation in the updated version.\n\n\n| Model  | Dataset Type                               | Undisrupted Acc | Disrupted Acc |\n|--------|--------------------------------------------|------------------|----------------|\n| Qwen   | Contextual Understanding (“capital of America: A,” expecting “J”) | 100              | 10             |\n| Qwen   | Factual Recall (“Washington DC”)     | 100              | 94             |\n| LLaMA  | Contextual Understanding                          | 100              | 10             |\n| LLaMA  | Factual Recall                              | 100              | 89             |\n\n\n>  Q3: Clarification on Hypothesis 1:\n\nThank you for the thoughtful question. We clarify that RoPE naturally induces low-frequency components in the lower dimensions of Q and K, which tend to accumulate extreme values [4]. These values often correspond to semantically important tokens and enhance contextual understanding. In contrast, high-frequency components (in upper dimensions) encode fine-grained positional information, so the semantic information is weak in that area [4], so the massive value does not appear in such area. When we disrupt these massive value regions (e.g., through masking or perturbation), the model’s ability to capture semantic information degrades significantly, supporting our claim that RoPE critically shapes embedding for semantic reasoning.\n\n\n>  Q4: On the Role of QK vs. V in Semantic Understanding and Knowledge Retrieval\n\nWe agree with the reviewer that semantic understanding and knowledge retrieval rely on vector representations. However, our work does not compare the relative roles of QK and V. Our focus is on massive value outliers in the QK embeddings, which lead to highly concentrated distributions due to RoPE position embedding. We show that removing these outliers impairs tasks requiring contextual understanding, such as math reasoning. Rather than making broad claims, we highlight a specific mechanism: how massive value in QK drives semantic understanding. We appreciate the opportunity to clarify this point.\n\n[1] Allen-Zhu and Li. Physics of Language Models: Part 1, Learning Hierarchical Language Structures.\n\n[2] Liu Z, Yuan J, et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. In ICML.\n\n[3] Jin H, Han X, et al. LLM Maybe LongLM: SelfExtend LLM context window without tuning. In ICML.\n\n[4] Barbero F, Vitvitskyi A, et al. Round and Round We Go! What makes Rotary Positional Encodings useful? In ICLR.", "claims_and_evidence": "LLM with RoPE would have massive QA value pattern, which is responsible for knowledge understanding. The author conducts several experiments on several models to substantiate the claims.", "methods_and_evaluation_criteria": "The paper utilized different disruption strategies to LLM in different tasks, and validated through task-specific metric.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Allen-Zhu and Li. (2024a) also highlighted the importance of RoPE, but the author didn’t discuss them.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Allen-Zhu and Li. (2024a) also highlighted the importance of RoPE, but the author didn’t discuss them.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper offers an interesting perspective on analyzing the RoPE’s role, and connects the LLM’s capability of knowledge understanding to massive values of QA.’\n2. The logic is generally nice.\n\n**Weakness**\n\n1. The authors fail to employ the methods on knowledge editting / benign forgetting tasks based on the findings in Section 3.5, which is of the community’s great interest and might complement the contributions.\n2. The paper only showed the relevance between massive QA value and the capability of knowledge understanding, without delving into why (e.g. how do massive QA values help LLM implement certain algorithms such as DP algorithms Allen-Zhu and Li. (2024a)).\n3. The paper do not analyze the gradient-whether it is the bias of gradient descent that leads to massive QA values when adopting RoPE. Also, whether the massive QA value is necessary for LLM to obtain strong knowledge understanding capability, or it just would guide the LLM to reach some sub-optimal points in the highly-non convex optimization landscape by gradient methods (in terms of the considered objective goal)? That suggests, other PE mechanisms also have the potential to excel without the massive values patterns of QK, which might be more robust to certain poisoning attacks.\n4. The best values or the most crucial ones across tasks should be highlighted in Table 1.", "other_comments_or_suggestions": "Allen-Zhu and Li. (2024a-b) also highlighted the importance of RoPE (see Figure 4 in Allen-Zhu and Li. (2024a)). Please provide a discussion.", "questions_for_authors": "1. Why there isn’t concat operation over Q in (5)?\n2. I’m interested in two cases testing LLM + RoPE:  (i)“What is the capital of America: A; What is the capital of Japan:”. If the model can correctly answer “J”, denoting the model successfully processed the context to recognize that “capital” here means “capital letter”, would the model fail to do so if you adopt the disruption strategy of massive value? (ii) On the other hand, when the prompt is “What is the capital of America: Washinton DC; What is the capital of Japan:”, if you then adopt the disruption, what would the model response? \n3. I don’t really understand your discussions under Hypothesis 1 paragraph. Is it the case that RoPE naturally creates some low-frequency regions around important positions, which causes the QA’s value concentrate on them and thus help improve contextual understanding?\n4. In my intuition, semantic understanding and knowledge retrieval all depends on vector representation, could your results imply that knowledge retrieval depends more on the value matrices’ meaningful representation, while semantic understanding replies on both QK and V?\n\n*Reference*\n\nAllen-Zhu and Li. (2024a) Physics of Language Models: Part 1, Learning Hierarchical Language Structures.\n\nAllen-Zhu and Li. (2024b) Physics of language models: Part 3.1, knowledge storage and extraction.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work presents an interesting observation regarding large values in the attention mechanism: specifically, the query and key components exhibit large magnitudes, while the value component does not. The authors empirically attribute this behavior to the use of ROPE embeddings. Experimental results suggest that this dynamic plays an important role in answering contextual questions, as opposed to retrieving parametric knowledge. However, the discussion of related literature is incomplete in the initial submission, though the authors have promised to address this in the camera-ready version. Another notable weakness is the lack of in-depth analysis explaining why this phenomenon occurs.
Decision: Accept (poster)