Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin
Authors: Enrique Queipo-de-Llano
Alvaro Arroyo
Federico Barbero
Xiaowen Dong
Michael M. Bronstein
Yann LeCun
Ravid Shwartz-Ziv
Venue: ICLR 2026 Poster (2026)
Keywords: attention sinks
compression valleys
deep trasformer-based LLMs
Primary area: foundation or frontier models, including LLMs

Abstract: Attention sinks and compression valleys have attracted significant attention as two puzzling phenomena in large language models, but have been studied in isolation. In this work, we present a surprising connection between attention sinks and compression valleys, tracing both to the formation of massive activations in the residual stream. We prove theoretically that massive activations necessarily produce representational compression and establish bounds on the resulting entropy reduction. Through experiments across several models (410M--120B parameters), we confirm that when the beginning-of-sequence token develops extreme activation norms in the middle layers, both compression valleys and attention sinks emerge simultaneously. Targeted ablation validates our theoretical predictions. This unified view motivates us to propose the Mix-Compress-Refine theory of information flow, as an attempt to explain how LLMs organize their computation in depth by controlling attention and representational compression via massive activations. Specifically, we posit that Transformer-based LLMs process tokens in three distinct phases: (1) broad mixing in the early layers, (2) compressed computation with limited mixing in the middle layers, and (3) selective refinement in the late layers. Our framework helps explain why embedding tasks perform best at intermediate layers, whereas generation tasks benefit from full-depth processing, clarifying differences in task-dependent representations.

Published as a conference paper at ICLR 2026
ATTENTION SINKS AND COMPRESSION VALLEYS IN LLMS ARE TWo SIDES OF THE SAME COIN
Enrique Queipo-de-Llano" Álvaro Arroyo* ,
Federico Barbero',
Xiaowen Dong', Michael Bronstein 1.2 Yann LeCun", Ravid Shwartz-Ziv
UUniversity of Oxford AITHYRA New York University
ABSTRACT
Attention sinks and compression valleys have attracted significant attention as puzzling phenomena in large language models, but have been studied in isola- tion. In this work, we present surprising connection between attention sinks and compression valleys, tracing both to the formation of massive activations in the residual stream. We prove theoretically that massive activations necessarily pro- duce representational compression and establish bounds on the resulting entropy reduction. Through experiments across several models (410M-120B parameters), we confirm that when the beginning- -feuuee-e-e token develops extreme acti- vation norms in the middle layers, both compression valleys and attention sinks emerge simultaneously. Targeted ablation studies validate our theoretical predic- tions. This unified view motivates us to propose the Tixx-Compress-Re/le of information flow, an attempt to explain how LLMs organize their computa- tion in depth by controlling attention and representational compression via mas- sive activations. Specifically, we posit that Transformer-basee kens in three distinct phases: (1) broad mixing in the early layers, (2) compressed computation with limited mixing in the middle layers, and (3) selective refinement in the late layers. Our framework helps explain why embedding tasks perform best at intermediate layers, whereas generation tasks benefit from full-depth process- ing, clarifying differences in task-dependent representations.
theory
LLMs process to-
INTRODUCTION
Large Language Models (LLMs) have become remarkably capable, yet how they process informa- tion through their layers remains poorly understood. Two phenomena have particularly puzzled researchers: attention sinks, where attention heads mysteriously collapse their focus onto semanti- cally uninformative tokens (Xiao et al., 2024), and compression valleys, where intermediate repre- sentations show unexpectedly low entropy despite the model's high-dimensional space (Skean et al., 2025).
These phenomena appear paradoxical: why would powerful models waste attention on meaning- less tokens, and why would representations compress in the middle of processing? Previous work has explained attention sinks through positional biases (Gu et al. 2025) and over-mixing preven- tion (Barbero al. (202a)). while compression valleys have been explained through an information bottleneck theory (Skean et al., 2025). However, the precise reasons why they emerge remain un- clear and no formal link has been established between them.
We reveal that attention sinks and compression valleys are two manifestations of single mecha- nism: massive activations in the residual stream. These extremely large-magnitude features emerge on specific tokens (typically the beginning-of-sequenee
token, BOS), create both effects simultane- ously, and act as input-agnostic biases. While prior work linked massive activations to attention sinks (Sun et al. 2024 Cancedda 2024), we prove they also drive compression: when single to- ken's norm dominates others, necessarily creates a dominant singular value in the representation matrix, explaining the compression. Experiments across several models (410M- 120B parameters) confirm this unified mechanism, connecting both phenomena via the massive activations.
* Denotes equal first authorship. Correspondence to wwwaam..rmoyo@ee.oo.....o and aarigue..ucipaaelllnnonrgg www.rreeeo...eeeoo

---

Published as a conference paper at ICLR 2026
This unified mechanism reveals how transformers organize computation across depth through the CCompress-Refine framework: massive activations control three distinct processing phases. Early layers (0-20% depth) mix information broadly via diffuse attention. Middle layers 20-85%) compress representations while halting mixing through attention sinks, both triggered by massive activation emergence. Late layers (85-100%) selectively refine through localized attention norms equalize. This phase structure explains task-specific optimal depths: embedding tasks peak during compression, while generation requires full refinement.
Our contributions are:
We empirically demonstrate that attention sinks and compression valleys emerge simultane- ously in middle layers across several models (410M-120B parameters).
We prove that massive activations mathematically require compression, providing tight bounds on entropy reduction and singular value dominance (Theorem [].
We validate causality through targeted ablations: removing massive activations eliminates both
compression and reduces attention sinks.
We propose the Cix-Compress--- Refine theory of information flow, ewwlainingg how transformers organize computation into three distinct phases.
We show this framework helps resolve the puzzle of task-dependent optimal depths: embedding tasks peak during compression while generation requires full refinement.
BACKGROUND AND RELATED WORK
In this paper, study decoder-only transformers with L layers, hidden dimension d, and H atten-
tion heads per layer. For a sequence of T tokens, let x(") E Rd denote token i's representation at
layer f, and ( Txx the full representation matrix. Attention weights o. (f,b) from token i to in head h satisfy causal masking (aij 0 for > i). Full architecture provided in Appendix A.1
Key Metrics For a representation matrix X with singular values measure compression via the matrix-based entropy:
2
02
2
2 or, we
H(X) = Σ Pj log where Pj = =
(I)
Low entropy indicates compression into few dominant directions. The anisotropy p1 =0 | measures directional bias (Razzhigaev al. (023) (near extreme bias, near i/r=isotropy). For token position k, the attention sink score and sink rate (Gu et al.
2025 are:
H
sink-score,
(E.N)
(2,h) Oik
sink-rate,
Σ
sink- (E, 27)
(2)
T
1=
h=1
with threshold = 0.3 (unless otherwise stated), and denotes the indicator function. We focus on the BOS token, the primary sink across models.
Attention Sinks. Attention heads mysteriously focus on semantically uninformative tokens (e.g.,
BOS) across diverse models and scales (Xiao et al. 2024). While Barbero et al. (2025a) argues they prevent over-mixing, Cancedda
miiiing,Canceeda 2024) relates them to spectral subspaces, and et al. (2025) traces
emergence to pretraining, no work has yet examined their depth-wise organization.
Compression Valleys. Transformer representations compress dramatically in middle layers, where the matrix-based entropy drops significantly before recovering (Skean et al., 2025). This universal pattern coincides with increased anisotropy (p1 0.9) (Razzhigaev al. (2023) and near- linear layer transitions (Razzhigaev et al., (0024). Paradoxically, these compressed representations excel at embedding tasks despite their reduced dimensionality. The mechanism remained unknown, with only information bottleneck hypotheses lacking causal evidence (Skean et al., (025).
Massive Activations. Sun al. (2024) identified extremely maggitude features in trans- former residual streams, with individual neurons exceeding typical activations by factors of 1 109. These "massive activations" consistently appear delimiter and special tokens (particularly
2

---

Published as a conference paper at ICLR 2026
Nornalised Entropy
1.0-
1.0-
10
Models
BOS Siuk Itate
BOS Norn
Pyshia 410M
402
Pythin 6:9
Llama SB
Qwen2 713
Bloom
0.0
0.0
G0 75 Relarive depth (%E)
100
25
60
75
75 Relative depth (96
225
50
100
Relative depth (9E)
Figure I: Attention sinks and compression valleys emerge simultancously when BOS tokens de- velop massive activations. Normalized entropy (left), BOS sink rate (middle), and BOS token norm (right) across layers for six models evaluated on GSM8K. All three phenomena align precisely: when BOS norms spike by factors of 100-10* (right panel), entropy drops below 0.5 bits (left) and sink rates surge to near 1.0 (middle), confirming our unified mechanism hypothesis.
the BOS token), acting as input- agnostic biases. Furthermore, Sun et al. (2024) found link between the emergence of massive activations and attention sinks, which was reinforeed in Barbero et al. (023b) Yona et al. (2025). However, none of these works link this phenomenon to representational structure or unifted theory of information flow in LLMs.
Computation Across Depth in Transformers. Several works have sought to understand the evo- lution of representations in Transformer-bast based models from theoretical perspective. Dong et al. (2021) proved that the repeated application of self-attention leads to rank collapse in simplified set- tings without residual connections. Geshkovski al. (2023) analyze self-attention dynamics and show tokens cluster over depth. Wu et al. (2024) studied how layer normalization and attention masks affect information propagation, finding that normalization can prevent rank decay. Other empirical work examines intermediate layer outputs directly. We highlight the LogitLens (Nostal- gebraist 2020) and Tunedlenns (Belrose al., 2023), which decode hidden states using the model unembedding matrix and an affine probe per layer, respectively. Furthermore, Lad et al. (2024) measures the sensitivity to delete and swap interventions across layers and argues at different stages of depth-wise inference. Most recently, [Sooráás et al. (2025) argue that deeper LLMs underutilize their additional layers, with later layers mainly refining probability distributions rather than compos- ing novel computations. While these studies illuminate layerwise behavior, none provides unified mechanism that explains why stages form in depth or predicts when they should appear.
The Gap This Work Addresses. Thus far, attention sinks have been tied to massive activations while compression has remained separate observation without a causal mechanism. In this work, we document the synchronized dynamics of these phenomena, showing that the same massive acti- vations that create sinks are also the main driver of compression. Building on their co-emergence, propose three-stage theory in which residual-stream norms simultaneously regulate mixing in attention heads and compression in representations. Finally, we connect the mechanism to down- stream behavior, distinguishing between embedding-style and generation tasks.
MASSIVE ACTIVATIONS DRIVE BOTH ATTENTION SINKS AND COMPRESSION
Key Insight: Attention sinks and compression valleys are not separate phenomena but two consequences of massive activations in the residual stream. We prove theoretically that when BOS token norms exceed others, they necessarily create a dominant singular value, causing compression and coinciding with attention sinks. This unification reveals that single mechanism controls both representation structure and attention in middle layers.
3.1 EMPIRICAL CORRELATION: SYNCHRONIZED EMERGENCE ACROSS MODELS
We first empirically document that attention sinks and compression valleys emerge simultancously across model families and scales. Figure T T shows the layer-wise evolution of three metrics across six models (Pythia 410M/6.9B, LLaMA3 8B, Qwen2 7B, Gemma 7B, Bloom 1.7B): (1) matrix-based
entropy H (x)()), (2), (2) sink-rate (e) and (3) BOS token norm
||x(( (f) We compute these metrics for all
7.5K training examples in GSM8K (Cobbe et al. 202I), I), plotting the mean and standard deviation at each layer.
3

---

Published as a conference paper at ICLR 2026
1.0
1.0
10'
Normaited Entropy
Training Step
Slnk Rote
10-
10+
BOS
10'
0.0
12
18
24
0.0
12
18
24
18
24
10'
Layer
Layer
Layer
1.0
1.0
102
Nornslised Eniropy
BOS 1.2 Norm
10a
Trining Step
Sink Rate
0.5
10"
0.0
0.0
10a
Layer 32
Layer
Layer
Figure 2: The coupled emergence of massive activations, compression, and sinks develops early in training and persists. Evolution of normalized entropy (left), sink rate (middle), and BOS norm (right) across training checkpoints (1-143k steps) for Pythia 410M, 6.9B. All three phenomena emerge together around step Ik and remain synchronized throughout training, indicating this orga- nization is learned early.
We observe that all three patterns align precisely. When the BOS norm spikes factors of 103 10a (typically layers 0-5 depending model depth), entropy simultaneously drops and sink rates surge. We compute the Pearson correlation between the change BOS norm and entropy, obtaining 0.9 + 0.18 across models, while BOS norm and sink rate correlate at r = 0.58 + 0.25
We highlight that this synchronization is remarkably consistent. While sink rates vary with prompt content, the layer index where these phenomena emerge is fixed for each model, and the massive activation is deterministic. For instance, in Pythia 410M, the transition consistently occurs at layer 5 regardless of input, suggesting architectural rather than input-dependent mechanism. We show how these transitions emerge during training in Figure 2 We point the reader to Appendix [B.1 for details experiments, larger models, and
GPT OSS (Agarwal et al. 2025
3.2 THEORETICAL FRAMEWORK: MASSIVE ACTIVATIONS IMPLY COMPRESSION
We now prove that massive activations necessarily induce the observed compression. Consider the representation matrix X Txx with rows [x where Xg denotes the BOS token.
Theorem (Massive Activations Induce Spectral Dominance). Let M ||x 12 R Eijo |x: 1| 1
and 0g be the angle between Xg and x4. Define the alignment term a = ñ Eeæo ||x; cos? 0 E
[o, 1). Then:
of 2 M aR,
where σ1 is the largest singular value of X.
Proof Sketch. Full proofs and discussions in Appendix A.2
This theorem has immediate consequences for compression metries:
Corollary (Compression Bounds). Let M/R be the norm ratio and p (c a)/(c 1).
Then:
1. Dominance: of E 2c+
2. Anisotropy: p1 2
3. Entropy: (XX) < -plogl (1 log - + (1- - log(r
Herre Llama-3-8B behaves as extreme outlier: almost all layers exhibit very large BoS activations, which makes the layerwise first-order differences nearly flat. This, in turn, renders the per-layer correlation ill-defined and artificially suppresses the overall correlation magnitude while inflating its variance.

---

Published as a conference paper at ICLR 2026
Upger boand
Eatropy
10"
[XL
Loyer
Layer
Figure 3: Our theoretical bounds become exact when massive activations emerge, proving they drive compression. Left: When BOS norm dominates (layers 5-15), the first singular value o,2 approximately equals both ||xBo||| and ||X|| confirming near rank-one structure. Right: Our entropy upper bound (Corollary 2) tightly matches empirical values compressed layers, validating that massive activations mathematically necessitate compression. Average across 100 RedPajama Weber et al. 2024) examples for Pythia 410M.
Meaning of the results. Theorem shows two factors control the rise of of: (i) the magnitudes of the activations M, and (ii) the aligmment a of other rows with Xp- Full alignment makes X rank (with (X) M R -| while even with small with large M suffices to grow of. The corollaries give lower bounds on dominance and anisotropy in terms of (c, o), so increasing c (stronger gap between activations) or increasing a (stronger alignment) provably widens the spectral gap. Consequently, the singular-valte valuee entropy is tightly upper-bounded by the mass in the top component, so e >> or a > also results in (X) dropping towards zero. Empirically, we find that aR remains small compared to M throughout the compression regime, so the behaviour of of is effectively dictated by the massive activation term M. More details in Appendix B.1
Tightness of the bounds in practice. When massive activations create growing norm ratios c, these bounds become tight. Figure compares our theoretical bounds against empirical measurements for Pythia 410M across all layers. In early layers where massive activations are absent, the bounds are loose as expected, because the theory only constrains the intermediate layers. However, in middle layers where massive activations emerge, the bounds become nearly exact, with predicted and observed values overlapping within measurement eтror. This tightness reveals that massive activations the dominant mechanism shaping representation geometry: when ||xo|| becomes massive, the representation matrix effectively becomes rank-one plus small perturbations, exactly as theory predicts.
3.3 EVIDENCE FROM TARGETED ABLATIONS
To isolate the exact role of massive activation empirically, perform targeted ablations: zeroing the MLP's contribution to the BOS token at layers where massive activations emerge. Specifically,
(e) (f+1) +- X + Attn (6) (xBos), removing only ML.((f) (xBos).
we set XBoS
We find that ablating massive activations can eliminate both phenomena, confirming our the- oretical conjecture. In LLaMA3 8B, removing the MLP's contribution the first layer prevents entropy drop (remains at 0.4-0.5 bits vs. dropping to 0.02 bits), eliminates sink formation (sink rate
Entropy Sink Rate
Tokoa L2 Norm
102
Entropy
Sink Rate
0.5
Avg Rest Norm
Baseline
Ablated
0.0
Layur
Layer
Figure 4: Removing massive activations eliminates both compression and attention sinks, con- firming causality. Ablating the MILP contribution to the BOS token at layer 0 in LLaMA3 8B has three effects: (Lefft) Entropy remains at ~0.55 bits instead of dropping to 0.02, showing decom- pression. (Middle) Sink rate stays at 0 throughout depth, confirming (Right) BOS norm (orange) remains comparable to the rest of tokens (grey) instead of spiking by 10 x. This causal intervention validates that massive activations drive both phenomena.
attention sink formation.
5

---

Published as a conference paper at ICLR 2026
drops from 0.85-1.0 to 0.0) and maintains BOS norm within 2x of other tokens (vs. 10 normally) (Fig. 4). Similar results hold across models, as seen in Figs in Appendix B. Some mod- els (Pythia 410M, Qwen2 7B) develop massive activations in more than one stage. Ablating any single stage partially reduces compression; ablating all stages eliminates entirely, suggesting a cumulative contribution.
Why Middle Layers? We hypothesize that the mid-depth concentration of sinks and compression reflects how Transformers allocate computation across depth: early layers perform broad contex- tual mixing, while later layers become increasingly aligned with next-token prediction (Lad et al., 2024). Freed from these pressures, middle layers can develop extreme features that regulate token- to-token sharing and induce compression, consistent with a mid-network shift toward refinement (Csordás et al. 2025). In the next section, we show how massive activations predict, and precisely characterize, three stages of information flow.
4 MIX-COMPRESS- REFINE: A THEORY OF INFORMATION FLOW
Key Insight: Transformers organize computation into three distinct phases demarcated by massive activations. Early layers mix information broadly build context. Middle lay- ers compress representations and halt mixing when massive activations emerge, preventing over-smoothing while maintaining essential information. Late layers equalize norms and switch to local positional attention, refining representations for task-specific outputs.
Building on our mechanistic understanding of massive activations, we now present a broader theory of how transformers organize computation across depth. We propose that information processing occurs in three distinct phases, demarcated by the emergence and dissipation of massive activations.
4.1 PHASE I: INFORMATION MIXING (LAYERS 0-20%)
In the early layers of the model, we observe diffuse attention patterns, which are enabled by the lack of massive activations. This allows the model to perform mixing high-dimensional token representations for a few layers, allowing the model to build contextual representations through broad information integration, An example of such an attention head in the early layers can be seen in Figure
To quantify mixing in attention heads, we define the Mixing Score as the average row entropy of attention matrices: + ΣT (AA (f.h)) Across models, we find that early layers consistently main- tain mixing scores above 0.7, confirming active token mixing, before dropping sharply when massive activations emerge. Notably, this mixing phase varies in extent: From just the first layer in some models to approximately 20% of network depth others. However, its qualitative characteristics remain consistent. We plot this metric Figure 23 in Appendix B.2 across models and layers.
We believe that this initial mixing stage is deliberately limited to prevent over-mixing and represen- tational collapse that would occur with extended uniform attention (Barbero et al. 2024 2025a), analogous to over-smoothing in graph neural networks (Arroyo et al. 2025). This controlled, brief' mixing phase establishes the semantic foundation that subsequent phases refine. The model cap- tures both local token dependencies and global context, creating rich representations that can be selectively compressed and refined in later phases.
4.2 PHASE 2: COMPRESSION AND HALTED MIXING (LAYERS 20-85%)
The middle phase begins abruptly with the emergence of massive activations, typically on the BOS token. As established in Section 3.2 these massive activations necessarily induce representational compression, as well as attention sink formation (Gu et al. 2025). This phase serves as a compu- tational shut-off switch for mixing. The attention sinks ct al. 2023). By attending to BOS tokens with near-zero value norms, heads effectively skip their contribution while preserving the residual stream.
approximate "no-ops" (Bondarenko
In these middle layers, the model refined information through the compressed residual stream, where few dominant directions preserve high-level context while discarding redundancies. This aligns with the depth-efficiency perspective of Csordás et al.) (2025). who show that mid-layers contribute less shaping future tokens and more to stabilizing current representations. In Section 5 we show

---

Published as a conference paper at ICLR 2026
15
Head
Head
Head
0.5
0.5
10
Layer
Layer
Layer
Figure 5: Middle-layer sinks adapt to input complexity while early mixing remains constant. (Left) BOS sink scores for high-sink prompt in Pythia 410M, showing strong attention BOS in layers 5-20. (Middle) Difference in BOS sink scores between high-sink and low-sink prompts, revealing input-dependent variation concentrated in middle layers. (Right) Difference in mixing scores between the same prompts, showing near-zer variation in early layers. This demonstrates that Phase I performs fixed mixing regardless of input, while Phase 2 compression dynamically adjusts sink strength based on prompt complexity.
that performance on generation tasks tends to improve mostly the latter half of this second phase. We hypothesize this lag reflects the time mid-layer MLPs need process and consolidate the com- pressed signal before yielding token-level refinements. We do not treat this as a separate phase, however, since it is not cleanly demarcated by the emergence dissipation of massive activations.
Sink behavior in middle to late layers adapts to input complexity. Mid to late layer sinks are input-dependent ("dormant heads"), which are often inert but active specific prompts (Guo al. 2024). Reproducing Sandoval-Segure et al. (2025) on 20K FineWeb-Edh prompts (Lozhkov et al,, 2023 we compare the "top" prompts (with the strongest sink scores) and the "bottom" prompts (with the weakest). As shown in Figure 5 sink strength diverges in the middle and last layers. This provides evidence that while middle layers default to sink-like behavior that limits mixing, sink strength in this phase varies depending on the prompt.
4.3 PHASE 3: SELECTIVE REFINEMENT (LAYERS 85-100%)
In the final phase, the model reverses the compression bottleneck through norm equalization and fundamental shift in attention patterns.
Norm equalization drives decompression. In this phase, we find the BOS norm plateaus or de- creases while the average norm of the rest of tokens rises sharply, driving them toward similar mag- nitudes (right panel in Figure and Figure 7). This equalization begins earlier than the full phase transition, where the average norm starts rising around 60-60% depth, preparing for the eventual shift. The massive activation ratio drops from 103 to < 10, removing the mathematical basis for compression and allowing representations re-expand.
Attention shifts to positional patterns. As massive activations dissipate, we observe heads tran- sition from sink-dominated to position-based patterns. In particular, we observe the emergence of identity heads (i + i), previous-toke heads (i + i- 1), and other sharp attention patterns, where by sharp we mean highly localized attention patterns. Figure an example of such pattern the
Layer 0, Head
Layer 16, Head 4
Layer 23, Head 10
Figure 6: Attention patterns transform from diffuse mixing to sinks to positional focus across depth. Evolution of attention patterns in Pythia 410M showing representative heads layers 0, 16. and 23. Early layers exhibit diffuse attention enabling broad information mixing. Middle layers show sink patterns that halt mixing. Late layers display sharp positional patterns for selective re- finement.

---

Published as a conference paper at ICLR 2026
Pythia 410M model. We find that sharp positional patterns are especially common in RoPE-based models, consistent with recent work (Barbero et al. 2025b) showing that RoPE induces frequency- selective structure that favors the emergence of such heads. We provide empirical evidence of this in Appendix [.2 In particular, when measuring the mixing rate of attention patterns, only mod- els without RoPE revert higher mixing in later layers, whereas RoPE-based models consistently transition toward sharp positional attention.
Laans
102
10"
Tokee L2 Norm
10p
207
Tolom
10
Aug noe-BOS
101
10'
10"
10
20
0
10
20
30
10
20
Layer
Layer
Lager
Figure 7: LLMs equalize token norms towards at the end of the network.
This phase serves three computational purposes. We believe that this phase serves three purposes. First, norm equalization reduces BOS dominance content tokens meaningfully influence the residual pathway and receive token-specific refinements. Second, attention shifts to sharp (often positional) heads that perform selective mixing, focusing on few task-relevant tokens and writing their features into the residual. As late layers re-expand capacity, these signals can be represented distinctly rather than squeezed by the mid-layer bottleneck. In parallel, seeentty/nearddiggon curb mixing without defaulting to sinks, where their non-zero value writes act as local signal boost- ers, in contrast to BOS sinks that effectively zero out updates. Third, bringing token norms to smaller, comparable scales likely improves numerical stability for the unembedding. Notably, models tend to equalize by boosting contens-toke norms rather than fully collapsing the BOS norm, preserving modest global bias while enabling precise, content-driven refinements.
heads
IMPLICATIONS FOR DOWNSTREAM PERFORMANCE
Different tasks achieve peak performance at different phases of the Mix- Compress-Refine organization. Embedding tasks peak during Phase 2's compression, bene- fiting from lower-dimensional spaces. Generation tasks improve monotonically through all phases, requiring Phase 3's refinement for accurate next token prediction. Multiple-choice reasoning shows flat performance until mid-depth, suggesting it needs both compression and subsequent refinement. This explains why studies reach different conclusions about "opti- mal" layers: they't re measuring fundamentally different computational objectives.
Key Insight:
Prior work (Skean al. 2025) found that miy-layer representations perform strongly, particularly on embedding benchmarks, and linked this effect to the mid-depth compression valley. In this section, we broaden the picture by evaluating both embedding and generation tasks, relating their depthwise performance to the three-stage framework introduced above.
5.1 THE DISTINCT PERFORMANCE PATTERNS ACROSS TASKS
Generation improves monotonically through all phases. We begin by evaluating intermedi- ate layers across multiple model families and sizes using LogitLeens (Nostalgebraist, 2020) on WikiText-2 (Merity et al. 2016). We observe steady perplexity decline with depth from 1 in early layers to 10-25 at full depth, as shown in Figure 8 (left). We notice little gain in the very early layers (consistent with an embedding-forme depth. Across several models, the sharpest improvements occur in Phase 3, where norm equalization and positional/identity heads enable token -ikeisccccifc refinements, which appear to crucial for next- token prediction tasks.
stage) and continued refinement through mid-
We further test the same set of models on multiple- question- question-answering tasks. We evaluate on ARC Easy, ARC Challenge, HellaSwag, and WinoGrande (Clark et al.) 2018 Zellers et al. 2019; Sakaguchi et al. (00II) via LogitLenss and the LM Evaluation Harness (Gao et al. (024)) with zero- shot learning. Figure 8 (middle) shows the results for ARC Easy and results for the rest of the
8

---

Published as a conference paper at ICLR 2026
Pythia 410M
Pythia 6.9B
Llamaä 8B
Qwen: 713
0.90
0.90
10
Perplexity
Acomary
Accuracy
104
0.60
0.60
0.45
0.45
10a
0.30
0.30
25
50
75
100
25
50
100
25
50
75
100
Relative Depth (%6
Relative Depth (%6
Relative Depth (%)
Figure 8: Embedding tasks peak during compression while generation requires full refine- ment, revealing distinct computational objectives. (Left, Middle) Perplexity on Wikitext-2 and multiple-choice QA accuracy in ARC Easy via LogitLeens generally do not improve significantly until ~50%% depth, then decreases/rises steadily through Phase
(Right) Linear probe test accuracy on the same task peaks at 25-75% depth (Phase 2) and declines thereafter. This divergence demonstrates that embedding-relese features concentrate in compressed middle layers, while generation tasks require full-depth for token-specifie predictions.
datasets can be found in Figure 29] Appendix B.3 For sufficiently large models, accuracies remain largely flat until roughly 40-60% depth and then rise sharply. This suggests that, for generation- aligned tasks, compression alone (Phase 2) is not sufficient; gains emerge only toward the end of Phase 2 (after sufficient residual refinement) and continue into Phase where norm equalization and positional/identity heads enable token-specific updates.
In short, in generation settings, the late Phase 2 to Phase 3 transition pivotal, aligning with the ob- servation in Csordás et al. (225)) of mid-network phase change from future-token to current-token computation, providing independent validation that Phase and Phase 3 serve distinct computational roles.
Embedding tasks peak in middle layers. To highlight the difference between generation and embedding tasks, we implement the following linear probing experiment. For each dataset, we encode the examples with the frozen backbone and extract hidden states at every layer. We then train linear classifier at each layer on the training split and evaluate it on the test split. We probe the same multiple-choice QA benchmarks as before and, additionally, a sentence classification dataset (SST-2; (Socher et al. 2013), assessing how much task-relevant information is linearly accessible at different depths. Figure (right) shows the results for ARC Easy, highlighting the difference in optimal depths with the corresponding generation task, while full results for this experiment shown in Fig. 31 Appendix B.3 Moreover, we reproduce Skean et al. (2025) on 32 MTEB tasks MMuenighooff al. 2022) across broader set of larger, decoder-only models. Across the board, we find that performance peaks consistently at 25-75% relative depth, outperforming early/late layers by 10-20% and precisely aligning with Phase 2, where compression is strongest (see Fig. 32 in Appendix B.3). These results align with evidence that next-token pretraining does not uniformly benefit perception-style classification (Balestriero & Huang 2024). Together, they underscore task dependence: cuasiffcaaionnrelevaal linear features concentrate in intermediate layers, whereas late layers are repurposed for token- specific generative refinement. Furthermore, the pattern suggests that massive activations in the residual pathway not only curb over-mixing via sink formation but also act as a mechanism the model uses to compress information.
5.2 WHY Do DIFFERENT TASKS NEED DIFFERENT PHASES?
We believe these findings clarify which tasks actually benefit from compression. In particular, embedding-style objectives (such as clustering, retrieval, classification, bitext mining, etc.) gain from Phase 2's compression because they target low-dimensional structure while discarding imrll- vant information, echoing classic arguments on the benefits information bottlenecks and com- pressed representations (Shwartz-Ziv et al. 2018 Kawaguchi et al.) 2023). This picture aligns with evidence that LLMs produce surprisingly linear (and often linearly separable) embeddings Razzhigaev et al., 2024 Marks & Tegmark (002).. In particular, when features concentrate in dow-dimensional subspace, linear probing, semantic retrieval, and related embedding tasks become easier. Moreover, such linear structure has been linked the emergence of high-level semantic

---

Published as a conference paper at ICLR 2026
concepts (Park al. 2023) reinforcing hypothesis why mid-layer compressed states tend to work well for non- -onneeerraevve evaluations.
By contrast, generation and reasoning require capacity that compressed states alone cannot provide. Performance improves the most once Phase 3 norm equalization restores higher entropy, and po- sitional heads/MLPPs
refine token-specific details, which is also when we observe the models being most confident about their predictions (see Fig. 28 in Appendix ....) In this way, the model makes of the compressed and refined representation from Phase 2, which has captured high-level ideas and semantic concepts, and expands this into higher-dimensional space to perform token-level refinements in Phase 3.
This reconciles two views: compressed mid-layers suit embedding benchmarks, whereas next-token- infiiction-alinnnn tasks benefit from full-depth processing. Practically, "optimal layer" selection should match phase to objective, suggesting phase-aware early exiting (Schuster et al. 2022) as potentially promising design choice.
6 CONCLUSION
In this work, we revisited two puzzling phenomena in decoder-only Transformers, attention sinks and compression valleys. We began with the observation that attention sinks, compression valleys, and massive activations all emerge at the same time in language models. We then proved that single high-norm token necessarily induces dominant singular value, yielding low matrix-entrope entropy and high anisotropy, and we bounded these effects quantitatively.
Building this, proposed Mix Mix-Compress Refine theory of depth- computation in LLMs. In particular, we show that early layers mix broadly through diffuse attention, middle layers com- press and curb mixing via attention sinks, late layers re-equalize norms and apply sharp positional heads for selective refinement. The boundaries between these phases are marked by the appear- ance and later disappearance of massive activations in depth. We use this organization clarify downstream task behavior. While embedding-style tasks peak in compressed mid-layers, generation improves through late refinement and benefits from full depth.
We see this framework as a step toward a more mechanistic account of how LLMs allocate compu- tation across depth. We hope these insights help connect head-level mechanisms with representation geometry, ultimately guiding more efficient and controllable LLM designs.
REFERENCES
Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508. 10925, 2025.
Riccardo Ali, Francesco Caso, Christopher Irwin, and Pictro Liò, Entropy-lens: The information signature of transformer computations. arXiv preprint arXiv:2502. 2025.
Álvaro Arroyo, Alessio Gravina, Benjamin Gutteridge, Federico Barbero, Claudio Gallicchio, Xi- aowen Dong, Michael Bronstein, and Pierre Vandergheynst. On vanishing gradients, over- smoothing, and over-squashing in gnns: Bridging recurrent and graph learning. arXiv preprint arXiv:2502.10818, 2025.
Randall Balestriero and Hai Huang. For perception tasks: The cost of LLM pretraining by next- token prediction outweigh its benefits. In NeurIPS 2024 Workshop: Self-Supervised Learning Theory and Practice, 2024.
Federico Barbero, Andrea Banino, Steven Kapturowski, Dharshan Kumaran, João Madeira Araújo, Oleksandr Vitvitskyi,, Razvan Pascanu, and Petar Veličković. Transformers need glasses! infor- mation over-squashing in language tasks. Advances in Neural Information Processing Systems, 37:98111-981433 2024.
Federico Barbero, Alvaro Arrоoo, Xiangming Gu, Christos Perivolaropoulos, Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do 1lms attend to the first token? arXiv preprint 027332 2025a.
10

---

Published as a conference paper at ICLR 2026
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations, 2025b.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08. 12, 2023.
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort, Quantizable transformers: Removing outliers by helping attention heads do nothing. In A. Oh, T.. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 75067-75096. 2023.
Nicola Cancedda. Spectral filters, dark signals, and attention sinks. arXiv:2402.09221, 2024.
arXiv preprint
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv: 3054505. 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Róbert Csordás, Christopher D Manning, and Christopher Potts. Do language models use their depth efficiently? arXiv preprint arXiv:2505. 2025.
Yihe Dong, Jean- SanntBapistt Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. International conference on machine learning, pp. 2793-2803. PMLR, 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. Uttpttps::/ /aattps://zeodd....
zenodo wwt.:ooooorgrreooooree.e.oo (12ccorrd/126085000
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rívière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint 2403:2200308299 2024.
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. A mathematical per- spective on transformers. arXiv preprint arXiv:2312. 10794, 2023.
Xiangming Gu, Tianyu Pang, Chao Du Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang. and Min Lin. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference on Learning Representations, 2025.
Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael Jordan, and Song Mei. Active-dormant attention heads: Mechanistically demystifying extreme-token phenomena lms. arXiv preprint arXiv:2410. 13835. 2024.
Dan Hendrycks, Collin Burns. Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021.
Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International conference on machine learning, pp. 16049-16096. PMLR, 2023.
11

---

Published as a conference paper at ICLR 2026
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations (ICLR), San Diego, CA, USA, May 2015.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: of permissively licensed source code. Preprint, 2022.
Vedang Lad, Jin Hwa Lee. Wes Gurnee, and Max Tegmark. The remarkable robustness of llms: Stages of inference? arXiv preprint arXiv:2406. 2024.
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb- edu: the finest collection of educational content, 2024. URL htts://huggingguce.
2023.
Andrzej Mačkiewicz and Waldemar Ratajczak. Principal components analysis (pca). Computers & Geosciences, 19(3):303-33 -3000-3-2,111 1993.
Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. arXiv preprint arXiv:2310.06824 2023.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture
models. arXiv preprint (tff:::600.07844 2016.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embed- ding benchmark. arXiv preprint arXiv:2210.07316, 2022.
Neel Nanda and Joseph Bloom. Transformerlens. 2022.
https://githus: com/
Nostalgebraist. Interpreting gpt: The logit lens. https: www. lesswrong AcceBBswpeaaiivrrrrriiirrrrreeee
logi t lens 2020.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. arXiv preprint arXiv:2311.03658, 2023.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. httphhtsss///pppee/e// id Id-n6SCkn2QaG
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language under-
standing by generative pre-training, 2018.
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oscledets, Denis Dimitrov, and Andrey Kuznetsov. The shape of learning: Anisotropy and intrinsic dimensions in transformer- based models. arXiv preprint arXiv:2311.05922 2023.
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goneharova, Nikolai Gerasimenko, Ivan Os- cledets, Denis Dimitrov, and Andrey Kuznetsov. Your transformer is secretly linear. arXiv preprint arXiv:2405. 1225:0, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Pedro Sandoval-Segura, Xijun Wang, Ashwinee Panda, Micah Goldblum, Ronen Basri, Tom Gold- stein, and David Jacobs. Using attention sinks to identify and evaluate dormant heads in pretrained 1lms. arXiv preprint arXivv2504.0338 008800000..9..... 2025.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456-17444 17472, 2022.
12

---

Published as a conference paper at ICLR 2026
Ravid Shwartz-Ziv, Amichai Painsky, and Naftali Tishby. Representation compression and gener- alization in deep neural networks. 2018. URL https:/ httpps///peeneeeeeeaaaaaaaaaeeee
5keL6sCqK7
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. Preprint 0tt1:25022.00113, abs/2502.02013, 2025.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning. Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642 Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL https: :/www. aclweb. org/ anthology/Dol 1170
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024. URL lttps://openrevier.
net/ /ttp:/oouunnd--7ahhh1-
Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Cha- lamala, Kezhen Chen, Max Ryabinin, Tri Dao. Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models, 2024. URL
https: /ttpps://arxvv..oo wrg. httpa/iiw.orgaab//////// 2411 12372
Xinyi Wu, Amir Ajorlou, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. On the role of attention masks and layernorm in transformers. arXiv preprint arXiv:240S. 18781, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. The Twelfth International Conference on Learning Rep- resentations, 2024. URL http::tt:///// http:::ss////peereeeeeeee net httprr/wwwwweeeeeeoeoooreieeeaae For////nnnnNN/N/5111122
Itay Yona, Ilia Shumailov, Jamie Hayes, Federico Barbero, and Yossi Gandelsman. Interpreting the repeated token phenomenon in large language models. arXiv preprint arXiv:2503.08900 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence?, 2019. URL /ttpppss//aaxxx..ooo///////////
13

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1
ARCHITECTURE
In this work, we study decoder-only Transformers (Radford et al. 2018), which employ causal mask- ing in attention and constitute the dominant architecture in today's 's large language models (Gemma Team et al., 2024 Dubey al. 2024p. We follow the notation of Barbero et al. (2024). but we
importantly also consider a model with H 2 attention heads:
(k q. (E (CA) Pij)
cxp
2,
(eh)
Σ
ig
(2.b)
w((
W
(8)
with dij
(E.h)
Eusi exp k qi
((.N) k. Piw
2/0
W
(L.h) +x!"
2
hg H
(e+1) x; = (t)
(8)
2i
+ 2;
where we will also denote by the matrices A (f.h) the attention heads given by (/ aij The causal masking translates into A (f.h) being lower-triangular and the row-wise softmax implies row-stochasticity.
(e.A)
A.2 THEORETICAL RESULTS
This section includes the proof' of the statements of Section 3.2
where we show massive activations imply the dominance a singular value. One can obtain a weaker version of the bound focused only on the massive activation (no alignment terms) that entails weaker bounds for the spectral metries. The following lemma serves as a proof for the fact that (X) = max MM|=1 |X
Lemma 3. Let A be a real symmetric n x n matrix with (real) eigenvalues Amax A 2 A2 N
Am Amin and let
A R(A,))=
X X
denote the Rayleigh Quotient for A and a real non-zero vector X E RR. Then R(A,x) E [Amin mmax], achieving each bound the corresponding eigenvectors Vmin: Vmax.
Proof. Let A Q TAQ be the diagonalization of A in the eigenbasis given by the and let y = Qx, such that x = Q y for Q orthogonal (i.e. Q
T = Q Then,
R(A.x)= (x (xQQ' Q A(Qx)
y A T Ay y (QQ Jy y
Σ XWy
n
y Σ,
A
x x
E !
Since the weights w, v yリ satisfy w; 2 0 and E, w, 1, R(A,x) is a convex linear combination of the eigenvalues and therefore Amin R(A, x) S Amaxs with equalities when x
Ymax: Viin-
We now prove that the emergence of massive activations in some layers directly implies that the first singular value dominates the distribution, which translates into extreme values for anisotropy and matrix-based entropy. The intuition is that entropy and anisotropy are representation-ont prop- erties: they depend solely on the singular-value spectrum of the representation matrix X, whose rows are token- wise representations {x,} and columns are features. A massive activation means that one row, say Xg, carries disproportionately large norm M = ||xa||2 compared the rest of token representations, sometimes orders of magnitude larger. Let x be the direction of Ml + Y. where Y contains the rest of the representations. If M is large compared to |Y|] F. then X is effectively a rank one matrix plus small perturbation, and we would expect
Xo, notice we can always write X- e x, + Y
(X) (X) ss M and v to close to the first right singular vector. This is exactly the mechanism exploited by PCA (Maćkiewicz & Ratajezak 1993): the first
principal component points in the direction that explains the largest variance; a massive activation
creates such dominant variance direction by construction. Therefore, even before formal bounds,
14

---

Published as a conference paper at ICLR 2026
should expect of dominate whenever (i) the norm ratio |xx||/// Σifo ||x/|| is large or (ii) the remaining rows {xt measurably aligned with Xo- The next result formalizes this intuition.
Theorem 4.
Let M = ||0|| R Eizo |x, and 0, be the angle between xo and x,. Define the
alignment term
R Σ i00 ||x;| cos? 0: E [0, 1]. Then:
of |xoll
Σ |xx|| cos 0, M oR.
Proof. By definition of the singular value (also see Lemma 3)
of 2
Xg x
Xxo
|xxx
(xi,
xo) = ||x0|| + Σ
(xi, xo)
Xg Xo
|x
|
|| |xo |
1=0
15
Using (x) 2 = ||x0| |x cos? 0g obtain
of M x 1| +
|x
cos² 0₃
1
Since aR = Eis40 |x;11 cos 0g we get of 2 M aR, which the desired result.
As mentioned in the main text, Theorem 4 makes precise how two independent factors govern the rise of of: (i) the magnitudes of the activations M, and (ūi) the alignment of the remaining rows with Xg. If representations were totally aligned, then X would indeed be rank one and would have singular value given by of
M R | Conversely, even with small (say, when token representations are not aligned or even orthogonal). large norm M suffices to grow of know it will
Empirically, we observe the term ||xo|l making the most impact in our analysis, be orders of magnitude larger than the rest of norms, however keeping the alignment term is also important for the following results.
We move onto proving Corollary Corollary (Singular value dominance). In the setting of Theorem let c = ||xo|1 Eizo l/x//// then
2 which we split in three in this section.
of N
c + a
Σ
2
122
Proof. From Theorem 4 of M oR. Moreover, 03 x of
X 1 (M +
RR) R aR (1 R. Therefore one gets:
of
M + af
+ aR
M +a
R
c+a
(M + (M)
(1 R)
1-a
1-
Corollary (Anisotropy). Let = - x | denote the anisotropy. In the setting of
M + aR c+a
Pi 2
M + R
c+I
Proof. Divide 2 M + HRR XX 3/- M + R, therefore:
M all
M +a
c a
pi Z M + R
M
+1
c
As mentioned in the main text, Corollaries using only (c. Thus, either increasing (stronger massive activation) increasing (stronger alignment) provably inflates the spectral gap. In both cases, having perfect alignment with x, or having |2 grow with respect to the rest, forces extreme values. If 1, then I-a co
lower-bound the dominance ratio and anisotropy
15

---

Published as a conference paper at ICLR 2026
and Sta + 1, intuitively because only direction becomes relevant in the data. Moreover, as c+1
the massive activation grows - the same result holds. Notice that is the ratio between the massive activation and the rest of them, therefore e increases by letting Xo be larger in norm, but also letting the rest of representations have low norm.
Corollary (Shannon matrix- entropy). Let Pj == :
x lp denote the normalized distri-
bution of singular values of X. Let H(X) := Ei=1 log be the Shannon entropy of such c+ a distribution. Let p c+1 1
Then, we have the following bound
H(X) 2 -plogp (1 p) log(1 - p) (1 - p) log(r 1),
Proof. Let
H(X) -Pi log P1-
Σ Pj logps S -plogp j=2
Σ pj log Pj. j=2
so we need to bound the second term, which is the entropy of r- terms adding up to 1-p1 S p-
This term would be maximised if the mass was equally distributed, that is, Pj 1-p1 2 Therefore, one gets
Σ Py log
1-P {-P) r-1) -(1 p) log
1- p r
log
The result is obtained combining these two bounds.
For fixed top p1 2 p, entropy is maximized when the remaining mass 1-p spread uniformly over the other r - singular values; the bound above is exactly that maximum. Consequently, any additional structure in the tail (e.g., second spike) will lower the true entropy beneath this upper bound. Notice for c + co or a + 1, + 1 and the upper bound approaches O.
Limitations of this analysis.
the theoretical analysis conducted above, we only considered one massive activation placed on the BOS token. In practice, models may exhibit more than one massive activation (Sun et al. 2024). In this case, our e term would make the bounds more permissive. We believe this poses no problem to our overall message and that this analysis can be extended. One can suppose the first n tokens to be the massive activations and decompose X E/= exx, + Y such that the first summand can be of rank at most n and Y a small perturbation in comparison, leading small entropy (effective rank & n), also holding for longer context lengths.
16

---

Published as a conference paper at ICLR 2026
B ADDITIONAL RESULTS
Experimental details. All experiments were implemented in PyTorch using NVIDIA A100 GPUs with 40GB memory NVIDIA H100 GPUs with 80GB when the memory requirements were stronger. We examined pretrained models of varying depths, using HuggingFace repositories with Transformers and Transformer-Lens (Nanda Bloom 2022). When large datasets were run col- lect metrics such as sink rates and norms, prompts were truncated to a maximum length of 4096 tokens for the FineWeb-EEd experiment (Fig. [ and 1024 for the GSM8K experiment (Fig. D as the latter required singular value decompositions to compute the entropy. LogitLens experiments for multiple-choice-questoo
tasks were done with M--Evaluation-Haaeeee plementing our own model wrapper to output hidden states at each layer instead of final ones.
(Gao et al. 2024). im-
Pearson Correlations. To assess the dynamical relationship between BOS norm, matrix-base entropy, and BOS sink rate across layers, computed correlations their layerwise changes. For each model and metric, the trajectory across layers was first 2-scored, and then we defined the delta layer the difference with respect the preceding layer,
Abe be be-1, = ê = S
This procedure emphasizes abrupt layerwise changes rather than absolute values, which crucial because BOS often exhibits sharp spikes that coincide with collapses in entropy and the sub- sequent emergence attention sinks. We then measured Pearson correlation coefficients between Ab; and Aeg (BOS norm vs entropy, same layer) and between Ab; and A8;+1 (BOS norm vs sink rate, lagged by one layer). Correlations were computed separately per model and summarized across models by Fisher 2 trannnfoomm averaging, reporting the mean correlation and the standard deviation across models.
Limitations. We outline some limitations of our work. Our analysis focuses on decoder-only Transformers and primarily attributes both sinks and compression to BOS-centered massive acti- vations; models with alternative positional schemes, attention sparsity patterns, or special-token conventions (c.g., no explicit BOS token, sinks in different positions or ALiBi encodings) may ex- hibit different dynamics. Our causal claims use targeted MLP ablations model familics, however, we observe model-dependent exceptions (c.g., sinks persisting despite de- compression). Lastly, the theory assumes single massive row, whereas real models may feature multiple interacting massive activations. However, as discussed in Appendix A.2, we believe this poses no harm the overall message: few massive activations would push the representations to a lower-dimensional subspace, but not necessarily of dimension 1.
selected layers and
B. BROADER ANALYSIS OF MODELS
In this section, we provide broader validation of our three-phase theory across model families and model sizes. Moreover, we expand the empirical measurements of metries from our theoretical analysis, on the ablation MLPs and provide two notes on the specifics of the GPT OSS model and Gemma 7B.
Validation on more datasets. The GSM8K dataset use in Figure I only contains structured question-answer pairs that we concatenate to obtain each prompt. In order to test theory against different distributions of data, we further evaluate these models on 5K examples from the following datasets: The Stack (Kocetkov et al., 2022), for code; Feb (Penedo al. (024), for prose and the MATH dataset (Hendrycks et al. 22021, for math questions. The results are shown in Figures 900 and II We find that the results to be very similar those of GSM8K, further reinforcing the input-nndependence of our theory.
Validation on more model families and large models. To further validate our Mix-Compress- Refine theory we observe the emergence of compression, attention sinks and massive activations in the Pythia model family (Fig. 12 and in very large models (70B-120B). specifically LLaMA3 70B, Qwen2 72B and GPT OSS 120B (Fig. B The prompt is a single GSM8K example. GPT OSS' particular sink patterns are explained later in this section. We believe this showcases our observed correlations universal phenomena in L.LMs.
17

---

Published as a conference paper at ICLR 2026
Normalised Entropy
1.0-
1.0-
10
Models
BOS Sink Rate
BOS Norm
Pythia 410M
103
Pythia 6.9
Llama SB
Qwe22 71
0.0
0.0
60
73
100
25
60
75
100
75
100
Relative depth (%)
Relative depth (%E
Rolative depth (%)
Figure Entropy, sink rate and BOS in The Stack.
Normalised Entropy
1.02
1.0
10
Models
BOS Sink Rate
BOS Norm
Pythin 410M
10a
Pythin 6.R
Llama SB
Qwen2 7B
Blocon
0.0
0.0
25
50
75
100
25
50
100
100
Relative depth (%)
Relacive depth (%)
Relative depth (%)
Figure 10: Entropy, sink rate and BOS Webe
Normalled Entropy
1.0
1.0
10
Models
BOS Siuk Rate
BOS Norm
Pythin 410M
10
Pythin 69R
Gemme
Llana 8B
Qwen2 7B
Bloon
0.0
0.0
20
60
75
100
25
60
70
100
20
50
76
100
Relative depth (96)
Relative depth (%)
depth
(%)
Figure 11: Entropy, sink rate and BOS in the MATH dataset.
1.0
1.0
BOS Neen
2n*
Pyl 190
Enntopy
0.6
0.5
Py:kis 1.3
Pycas maM
Pycass 18
Py:kie 2.8
Pynkis 5.03
0.0
25
50
76
100
25
50
100
20
50
76
100
Reiative Degtth (5)
Reiative Dageh
Relativw Degath (53)
Figure 12: Entropy, sink rate and BOS norm for the Pythia family of models.
Entropy
Siak Rate
MM
BOS 5.2 Nomm
20
0.5
CPT /238
102
0.0
Relativo Depth (%)
Relatioe Deyth
Relative Depth (56)
Figure 13: Entropy, sink rate and BOS norm 70B-120B models.
Training Checkpoints. We evaluated the training dynamics of the Pythia 410M/6.9B/122 models multiple checkpoints (steps 1k, 2k, 4k, 8k, 10k, 20k, 30k, and 143k). At each checkpoint, after every layer we recorded the entropy, BOS sink rate (threshold - 0.3) and the norm of the BOS token representation. The prompt was a single GSM8K prompt "Janet' ducks lay eggs Figure 1 iillusstttees the results for the Pythia 12B model.
Role of in practice and visualization of theoretical results. The alignment term is added in the theoretical analysis for mathematical completeness. every other token closely aligned with the BOS token, then rank(X) x 1 and the compression follows trivially. Of course, this is not
18

---

Published as a conference paper at ICLR 2026
1.0
1.0
10'
Noruaited Entropy
Training Step
Slnk Rote
10a
BOS L.2
10+
10a
0.0%
16
24
82
0.0
0
16
24
32
16 Layer
24
32
10'
Layer
Layer
Figure 14: Entropy, Sink rate and BOS Norm at different training checkpoints for Pythia 12B.
expected to happen in practice. One perform the same analysis without a using the (weaker) bound of 2. M and the corresponding dominance, anisotropy and entropy bounds. The bounds in that case become of e مi oj. P1 2 c/(c + 1) and similarly for the entropy if one instead uses the new bound for the anisotropy. We provide plots with the bounds from the theoretical discussion in Section 3.2 and their weaker versions. Figures 15 and show these values for LLaMA3 8B and Pythia 410M We show (I) the terms M XBos I aut and M R X from Theorem) | (2) the top 3 singular values of and the sum Σ +21 of and (3,4,5) the dominance, anisotropy and entropy bounds from Corollary 2 all cases, we observe the bounds being tight in the middle layers. In this regime, the first singular value of follows the trajectory of |XBOS|| | closely and dominates the rest of the singular values. The dominance decreases steadily, specially towards the second half of the network, indicating the preparation for next token prediction of Phase 3. Lastly, one observes that the weaker version of the bounds is already very tight, showing the importance of the massive activation.
10
1.00
Eningyy
10
10a
10
0.75
101
10
10
0.50
Anbeinasyy
10a
M-P
10a
10
11)
10
Figure 15: Theoretical bounds for Pythia 410M. (a) shows the elements in the bound of Theorem 1 (b) shows the singular value spectrum of ( each layer f. For plots (c,d,e) gray represents the weaker bound (no a), while green represents the bound as presented Corollary 2
10
10a
1.00
Estrrope
101
10
10*
3-
wtabort
10a
10a
10a
10a
10a
Andintragy
10a
10
Figure 16: Theoretical bounds for LLaMA3 8B. (a) shows the elements the bound Theorem ! (b) shows the singular value spectrum of X each layer f. For plots (c,d,e) gray represents the weaker bound (no a),, while green represents the bound as presented in Corollary
MLP ablations. We further run the targeted MLP ablations on more models erase the appear- ance of the massive activation. For LLaMA3 8B, we ablate layer O; for Qwen2 7B, we ablate layers 3 and 4 and for Pythia 410M, we ablate layers 0 and 5-7. The results are shown in Figures 4 18
Additional Ablations on Massive Activations Here, we also run ablation regarding the emer- gence and exact effect of massive activation on both compression and attention sinks on Pythia 410M. In particular, Figure 19 shows what components of the model cause the emergence of mas- sive activation on the BoS token. On the other hand, Figure 20 shows the effect of clipping the maximum value in the massive activation vector be between 7
19

---

Published as a conference paper at ICLR 2026
Entropy & Sink Rate
1.0
10a
Token L.2 Norm
Entropy
Sink Rate
0.5
BOS Norm
10a
Avg Rest Norm
Baseline
Ablated
0.0
10'
24
24
Layer
Layer
Figure 17: MLP ablation for Qwen2 7B.
Entropy Rate
1.0
Token 1.2 Norm
Entropy
Sink Rate
0.5
10"
BOS Norm
Avg Rest Norm
Baseliue
0.0
10a
Ablated
12
18
24
12
18
24
Layer
Layer
Figure 18: MLP ablations for Pythia 410M.
Pythia
Change Nona B08
Atreation
200
200
Figure 19: MLPs are responsible for the appearance of massive activations in Pythia 410M.
Entropy
Sink Rate
BOS Norm
0.8
10P
0.75
0.4
10'
0.25
10
20
0,00
10
20
Layer
Layer
Layer
Figure 20: Ablation of the effects of massive activations in the residual pathway Pythia 410M. Lower massive activation norms generally correspond to both increased compression and lower sinks.
From the Figures, can that the MLP seems to be causing the emergence of massive acti- vations and how lowering the norm of the massive activation corresponds to an increase the in the representational entropy and decrease in attention sinks. In general, find attention sinks to be less predictable than entropy (perhaps due to the way the sink score is calculated), although there is general downward trend as the massive activation norm is reduced.
A note GPT OSS. In the GPT OSS (Agawaal et al. 2025) family of models, each attention head equipped with a learnable sink logit that allows it to divert probability mass away from real tokens, effectively providing "skip" option. However, unlike the explicit (k', v²) bias formulation studied in Sun et al. (2024); Gu et al. (2025), GPT OSS does not include learnable value sink token. This means the model cannot encode bias information directly through the sink, and hypothesize that
20

---

Published as a conference paper at ICLR 2026
it instead continues rely on massive activations the BOS token implement bias-like behavior and generate compression. This explains why BOS sink patterns are still observed, particularly the middle layers (see Figure 2T). The alternating spikes across layers may be consequence of GPT OSS' alternating dense and locally banded sparse attention pattern: in layers with local attention windows, heads less able access BOS, while in subsequent dense layers BOS becomes globally visible again, producing the observed oscillatory sinkness.
Entropy & Sink Rate
0.6
Entropy
Sink Rate
Token Norm
104
0.4
0.2
10a
BOS
Avg non-BOS
0.0
10
15
10
20
Layer
Layer
Figure 21: Entropy, sinks, and massive activations for GPT OSS 20B
The particular case of Gemma 7B. Even though Gemma 7B follows the same dynamics we have discussed in the chapter, how it achieves them is different from the rest. Token norms in Gemma 7B start very high; instead of increasing the BOS norm create massive activation, Gemma 7B decreases the norms of the remaining tokens to create the disparity needed for compression, then re-equalizes by increasing their norms in late layers. We attribute the initially high norms to the embedding layer, as there are no other components that can account for it. We believe this is also why attention patterns in Gemma 7B look bit different from the rest, with identity heads emerging both at the early and later layers. Figure 22 means after each layer.
illustrates this. Pre- means before each layer, while post-
107
202
DCS
20'
HD4 peet
10
15
25
Figure 22: Token norms in Gemma 7B. The BOS norm starts high since the beginning.
B.2 MIXING AND SINK DETECTION METRICS
In this section we propose and study new metrics for quantifying mixing and "sinkiness" in attention heads, and provide further validation on the FineWeb-Eal experiment from Section 4.2
Mixing Score. Let be lower triangular, row-stochastic attention matrix. We define the Mixing Score as the average Shannon Entropy of each row H 7 E H(A,.)
+ E EE= Onjy log Osy- Since each row of is the output to a softmax, distribution so the score is well-defined. This captures how broadly each token is attending to its preceding tokens. High values indicate the rows are close to the uniform distribution, suggesting broad mixing across tokens. Low values imply the rows are one-hot vectors, suggesting very lo- calized mixing (sinks, identity or positional heads). Figure 23 (right) shows the Mixing Score in depth for a variety of models, showing how the mixing abruptly decreases from 0.7-0.75 to 0.3-0.4 after the first few layers. Bloom 1.7B resumes mixing in the last phase due to not being capable of producing positional patterns, as it is the only one without rotary positional embeddings (Barbero) al. 2025b).
probability
ColSum Concentration. Similar to the mixing score, column sums df = Σ, Aiy capture how much attention is received by token j. We get probability distribution by normalizing ej
21

---

Published as a conference paper at ICLR 2026
1.00
1.00
Colsum Rate
Mixing Score
0.75
0.75
Pytbia 410M
Pythia 6.9B
0.50
0.50
Qwen2 7B
0.25
0.25
Bloom 1.7B
0.00
0.00
0
25
50
75
100
0
25
50
75
100
Relative Depth (%)
Relative Depth (%)
Figure 23: Left: ColSum Rate (r = 0.3) across depth for different models. Right: Mixing Score across depth for different models, averaging across heads per layer. The ColSum Rate increases with the massive activations, similar to the BOS sink rate, while the Mixing Score abruptly decreases after the first few layers.
G/E,4= CG/T, since E.E, T for A is row-stochastic. Denote by Hool - -llg (ogg)>-Eo loge; [0 the normalized entropy of such distribution. For consistency, define the ColSum Concentration as = Henl E [0. 1]. High C means a few columns receive most mass (sink-like), low means diffuse reception. When the sink the BOS token, the ColSum Concentration is tightly related to the BOS sink score coupled: for a single BOS-dominated head, ColSum increases monotonically with the score Co 7 E, Aso and is lower-bounded by the case where the remaining mass is spread uniformly across the other T T-1 columns. In that case, (co)
[- (log T) (cy log cu (1 co) log( 7-1 ]) and any additional concentration on non-BOS columns pushes C' above this curve. Similar to the sink rate, we can define ColSum Rate as the percentage of heads with ColSum Concentration above a certain threshold. Figure 23 (left) shows the ColSum Rate (r 0.3) for different models across depth, imitating the Sink Rate's behavior. Moreover, scatter plots in Figure 24wshow the ColSum Concentration as function of the BOS score for heads different models. As given by the bound, high BOS score means high C, these are pure BOS sinks. Points with high ColSum but low reveal heads that sink to non-BOS tokens. In Pythia 400M, we observe such an outlier head, indicating sink token different from BOS.
(a) Qwen 7B
(b) Gemma 7B
(c) LLaMA3 8B
(d) Pythia 410m
Figure 24: BOS versus ColSum Concentration. The relationship with the BOS sink score indicates ColSum concentration good, token-agnostic alternative for sink detection.
Limitations of Mixing Score and ColSum Concentration Each of diagnostics highlights one axis of behavior while missing others. The ColSum Concentration = 1 Heal is effective at flagging sinks, where column dominates, but it assigns zero score identity heads and very low score to perfectly uniform heads. Conversely, the Average Row Entropy Hęw measures sparsity of rows, distinguishing diffuse mixing from one-hot attention, but cannot differentiate which sharp pattern occurs: sinks, identities, or previous-token heads all have similarly low row entropy. Thus neither metric alone fully separates the regimes of interest. In principle, one could combine them into scalar Mix2D(a) = aC' + (1 a Hrow, where, for a suitable choice of a, sinks would map near 1, perfectly uniform heads near and identities near 0.5. This would give a single axis interpolating between mixing, sinkness, and identity. In practice, however, we did not find this construction very informative and thus did not include
Sink-Versusse Sit--errus-Iddettt, Index. In Phase 3, observe attention patterns changing to more local- ized, sharp ones. Some these patterns include identity-lit like heads, previous-token heads and hybrid sink-identity heads. We quantify this transition using the sink-versus-idente index, defined
22

---

Published as a conference paper at ICLR 2026
SVI B/ D) where B is BOS attention and = E, Asi is diagonal attention. There- fore B + = A. + An E [0. 1]. Figure 25 plots each head as a 2D point (SVI, B D), attention
with color corresponding to its layer. Early heads tend to have low B D. indicating allocated to the BOS token nor the identity. As depth progresses, heads tend to go toward high B + D and high SVI, indicating strong sink presence. Moreover, the middle to late layers tend to also show identity patterns or sink-identity hybrid patterns.
Patlis 410M
Geaue
Zlaso
Quesp
Figure 25: Attention received by the BOS token and the diagonal for each head.
15
Hood
10
Bed
10
Hesd
10
A05
0.5
0.5
10
10
10
20
Layer
Layer
Figure 26: Left. BOS sink scores (top prompt, Bloom 1.7B). Middle. Top bottom prompt difference in BOS sink score. Right. Top-bottom prompt difference in mixing score.
20
20
Head
Head
Head
0.5
0.5
0.5
10
0
%
0
20
Layer
Layer
Layer
Figure 27: Left. BOS sink scores (top prompt, Qwen2 7B). Middle. Top-bottom prompt difference in BOS sink score. Right. Top-botton prompt difference in mixing score.
Further validation FineWee-Edu. Figures 26 and 27 show the FineWeb-Edk experiment for Bloom 1.7B and Qwen2 7B models. The trend is clear: regardless the input, the models do not allocate attention to the BOS token until the massive activation emerges. The amount of sinks present in the middle layers is input-dependent, however the amount of mixing performed in the early layers not.
B.3 ADDITIONAL RESULTS ON DOWNSTREAM TASKS
In this section we provide more details on the experiments and results exposed in Section 5 of the main text.
Generation Tasks. We applied the LogitLens to WikiText-2 by passing each batch of tokenized blocks through a frozen backbone and, for every layer, projecting that layer's hidden states to vocab- ulary logits using the model's tied unembedding head. For each layer L, we computed the next-token cross entropy loss and perplexity (as shown Tigure@oor the main text), as well as the mean token entropy of the softmaxed logits (Ali et al.) 2025), shown in Figure 28 We take this entropy proxy of the model's confidence over the next token, and we also observe it decreases more rapidly towards Phase 3. In addition to next-token prediction, extended the LogitLens evaluation to multiple-choice QA benchmarks (ARC Easy, ARC Challenge, HellaSwag, WinoGrande), where the model must select among small set of candidate answers. For each layer, we applied the final layer
23

---

Published as a conference paper at ICLR 2026
10
Mean Token Entropy
Pyshia 41384
8
Liumaä SB
6
20
40
60
80
100
Relative Depth (%)
Figure 28: Language modeling requires full depth. Entropy of the output distribution each layer.
norm and projected the embeddings with the tied unembedding head. We used LM Evaluation Har- ness to score, recording the accuracies. This allows to compare how representations at different depths support generation-style (next-token) and selection-style (multiple-choice) reasoning. Figure 29 shows MCQ performance remains relatively flat through the compression valley of Phase 2 and begins improving towards ~ 50% the network, underscoring that reasoning tasks require both compression and late-layer specialization. For completeness, also ran the experiments with five- shot learning for each dataset. However, this only seemed to boost the final accuracies but did not influence the overall behavior observed.
ARC. Rasy
ARC Challempe
Hellaswaa
Winngrande
Teat Accorey
0.8
0.8
Pythia 410M
0.6
Pythia 6.9B
Liuna SB
0.6
0.4
0.4
0.4
Qwen2
20
50
75
100
25
50
75
100
0
25
50
70
100
26
50
75
10o
Relative Depth (%)
Relative Depth (%)
Relative Depth (%)
Relative Depth (%)
Figure 29: LogitLeens accuracies multiple choice question datasets.
LenedLens. TunedLens (Belrose al. 2023) is a refinement of the LogitLeens technique that involves training a small affine transformation onto the vocabulary for each layer instead of using the model's own unembedding layer. To further validate our LogitLens experiment the MCQ datasets, we also used the LM Evaluation Harness to run the TunedLens for Pythia 410M, Pythia 6.9B and LLaMA3 8B with the pretrained lenses available at Belrose et al. (2023). We include results in Figure 30 for completeness, however we do not observe meaningful differences in the layerwise behavior with respect to LogitLens.
ARC Easy
ARC Challenge
Hellawwag
W Winggrande
Tel Accrrey
0.8
0.8
Pythia 410M
0.6
Pythin 6.90
0.6
Liana SB
23
50
75
100
25
50
75
100
25
50
100
25
50
75
Tbo
Relative Depth (%)
Relative Depth (%)
Relative Depth (%)
Relative Depth (%)
Figure 30: TunedLens accuracies on multiple choice question datasets.
0.77
Pyllia
Qpend 78
0:2%
Genna T11
Belecioe Depth (4)
Relaciww Terpts (5)
Talatrow Tlythh (%)
Tedative Targth 150)
Tidudiv Depth (%)
Figure 31: Linear probing validation accuracies.
24

---

Published as a conference paper at ICLR 2026
Embedding Tasks. To further validate the results of Section 5 and the proposed by (Skean et al. 2025), we run a standard linear probing experiment. Probes are trained independently per layer. with backbone parameters fixed, using learning rate of 5 10 with PyTorch's default settings for Adam (Kingma & Ba 2013), no weight decay, one epoch, maximum length of 1024, batch sizes of 16-32 and random fixed seed of 123. We train for backbones Pythia 410M, Pythia 6.9B, LLaMA3 8B, Qwen2 7B and Gemma 7B. Figure [1 shows the results. As discussed, across models and datasets, accuracy peaks in the middle layers. These results suggest that the linear fea- tures relevant for classification emerge transiently in the compressed middle representations, while the late layers are repurposed for generative refinement. Moreover, we run 32 MTEB tasks for the same models and report the average main score across tasks in Figure 32
0.8g
Jabate
Figure 32: Average main score across 32 MTEB tasks.
25

---

Published as a conference paper at ICLR 2026
015
Figure 33: Gemma 7B Heads for example prompt.
26

---

Published as a conference paper at ICLR 2026
622
Figure 34: Pythia 410m Heads for example prompt.
27


Reviews:
[{"review_id": "hNOhTbjH36", "reviewer": "Reviewer_FFpb", "summary": "This paper investigates the relationship between attention sinks and compression valleys in LLMs, proposing that both phenomena arise from massive activations in the residual stream. \n\n\nThe authors prove that \"massive activations\" necessarily produce representational compression through singular value dominance. \n\n\nThrough experiments across diff. models they validate that when the beginning-of-sequence (BOS) token develops extreme activation norms, both compression valleys and attention sinks emerge simultaneously. \n\nThe paper provides new insights, but its theoretical contributions might not be strong. In my opinion,  most of the technicalities are based on well-known linear algebra.", "questions": "Q0. Please see Weaknesses\n\nQ1.  Why do Pythia 410M ablations remove compression but not sinks, and does this indicate that sinks can emerge through multiple independent mechanisms?\n\nQ2.  What determines $\\alpha = \\frac{1}{R}\\sum_{i \\neq 0} \\|x_i\\|^2 \\cos^2\\theta_i$, and since Figure 3 shows $\\alpha R \\ll M$, is the alignment term theoretically necessary or practically negligible?\n\nQ3. Have you tried inducing massive activations artificially (e.g., via initialization or regularization) to test whether sinks or compression follow?\n\nQ4. Can phase transition points be predicted from architectural parameters such as depth, width, or number of heads rather than observed post hoc?\n\nQ5. Do these phenomena appear in encoder-only models or models with alternative positional encodings, and how do these differences affect the emergence of massive activations and phase structure?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written with good contextualization relative to prior work,\nand comprehensive appendix with additional validation. \n   \n\n- While the proof technique is straightforward (using Rayleigh quotient characterization), this is the *first* work to formally prove that massive activations $M = ||x_0||^2 $ necessarily induce spectral dominance:  $\\sigma_1^2 \\geq M + \\alpha R$.   \n\n-   The derived bounds on dominance  $\\frac{\\sigma_1^2}{\\sum_{j \\geq 2} \\sigma_j^2} \\geq \\frac{c+\\alpha}{1-\\alpha}$,  anisotropy $p_1 \\geq \\frac{c+\\alpha}{c+1}$,  and entropy are mathematically sound.\n\n-   Figure 3 shows that in middle layers (where massive activations emerge), the theoretical bounds closely match empirical values, validating that massive activations are the dominant mechanism shaping representation geometry. Targeted MLP ablations (Figures 4 and 14) across multiple models provide causal evidence.", "weaknesses": "- The theorem relies on well-known linear algebra and, in my opinion, lacks strong theoretical novelty.\n\n- The bound depends on $\\alpha = \\frac{1}{R}\\sum_{i \\neq 0} \\|x_i\\|^2 \\cos^2\\theta_i$, but Figure 3 suggests $\\alpha R \\ll M$ in practice.  \nMore analysis of what determines $\\alpha$ would strengthen the work.\n\n- Pythia 410M ablations show that compression is removed without eliminating sinks, suggesting that sinks may have multiple emergence mechanisms.  \nSince ablations only target MLP contributions, direct manipulations of massive activations (e.g., through regularization or initialization) would provide stronger causal evidence.\n\n- Lines 200–206: The proof sketch is potentially misleading, as it may imply circular reasoning.  \nYou may clarify that the Rayleigh characterization provides a lower bound when choosing $v = x_0 / \\|x_0\\|$.\n\n- Lines 242–247: This contradicts the claim that massive activations jointly cause sinks and compression.  \nYou may either analyze why Pythia 410M behaves differently or restate the claim as a \"common but not universal\" mechanism.\n\n- Lines 185–187: I don’t think $r = 0.58 \\pm 0.25$ indicates a strong correlation; it appears to be only moderate.\n\n- Lines 206, Figure 3: Although $\\alpha R \\ll M$, $\\alpha$ appears crucial, yet there is no analysis of its origin or importance.\n\n- Lines 185–186, 907–909: No p-values are reported, the sample size ($n = 6$) is small, and there is no correction for multiple testing.  \nWhat are the $p$-values and sample size used?\n\n- Lines 421–424: Some parameters might need to be reported, such as the optimizer, learning rate, batch size, number of epochs, and random seed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vau2T6bCVL", "reviewer": "Reviewer_8eqz", "summary": "This paper argues that attention sinks and compression valleys in LLMs are caused by the same underlying mechanism—massive activations in the residual stream, often on the BOS token. The authors show empirically that these phenomena always co-occur in the middle layers across models (410M–120B). They prove that when one token’s activation norm dominates, it forces a dominant singular value in the representation matrix, causing low entropy and thus compression. They confirm the causality through ablations. Based on this, they propose a Mix–Compress–Refine theory of information flow: early layers mix information broadly, middle layers compress representations and halt attention mixing, and late layers re-expand and refine outputs. The paper also connects these phases to task performance—embedding tasks perform best during compression, while generation benefits from full refinement. Overall, the work unifies two previously separate observations and provides a clear mechanistic view of how transformers structure computation across depth.", "questions": "Please see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The three-phrase part seems to be interesting. They propose and try to demonstrate a possible three-phase structure. In early layers, attention entropy is high and attention patterns are broadly mixed (Fig. 6 left). In middle layers, mixing decreases and representations become more compressed (Fig. 6 middle). In later layers, attention shifts to localized positional patterns and token norms become more balanced (Fig. 6 right). Task-level results further align with this trend: embedding performance tends to peak around the mid-layer compression phase, while generation tasks continue improving toward the final layers (Figs. 7, 23–27).", "weaknesses": "The paper has two parts—the relationship between attention sinks and compression valleys, and the three-phase structure. However, I do not find either of them satisfying enough.  \n\n1. The connection between attention sinks and compression valleys is quite trivial. It is well known that attention sinks and massive activations occur simultaneously. Then a massive activation leads to a dominant largest singular value in the activation matrix. The causal ablations that remove early-layer MLPs are not new either.  \n\n2. It was observed previously that early layers have more uniform (or mixed) attention patterns, middle layers have attention sinks, and late layers show special functionality. The paper develops these observations into the three-phase structure too strongly, without providing stronger evidence. Figures 6 and 7 simply reproduce previous experiments and cannot support stronger claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t0lztx3r1Z", "reviewer": "Reviewer_Wogq", "summary": "This paper aligns two seemingly disjoint phenomena within transformers, attention sinks (where an attention head is dominated by attention weights for a semantically uninformative token) and compression valleys (where intermediate representations show low entropy despite high dimensionality). Both empirically and theoretically, the authors tie these phenomena to massive activations in the residual stream for the beginning of sequence token, showing that as the BOS norms increase, there is a corresponding increase in the sink rate and a decrease in entropy. The authors use these insights to present a mix-compress-refine theory of information flow, additionally tying this theory to insights regarding performance on various classes of tasks at different layers within the network.", "questions": "1. See weaknesses.\n2. Additionally for Figure 14, while the general trend of ablations decreasing sink rate and increasing entropy is supported, for the ablated entropy metric for Llama 3 8B, is there any intuition as to why there is a sudden increase in layer 15, and for Qwen 2 7B, is there any intuition for the increased sink rate in layers 4-6?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The breadth of analysis is strong, with the analysis including decoder-only models of the Pythia, Llama, and Qwen families in the main paper. In particular, the analysis of Pythia 410M over training is visually illuminating of how the properties discussed emerge rapidly over training. The ablation experiments in Llama 3 8B showing that removing massive activations causally decreases attention sinks and compression are also convincing, though not general across every model.\n2. The theoretical framework analyzing massive activations and their influence on compression bounds is well founded.\n3. The mix-compress-refine theory of information flow that the paper presents follows reasonably naturally from the insights on experiments with massive activations. This is then made more rigorous with downstream task evaluations, evaluated at various layer representations, that show that refinement requires later layers, while embedding signals are most useful in the middle layers.", "weaknesses": "1. Returning to the BOS ablation experiments, looking at Figure 14, do you have any insights as to why sinks persist in Pythia 410M despite the ablation for the BOS token? Is it the same for other Pythia models, such as 6.9B? \n2. Some of the figures, such as Figure 2-6, while illuminating, show only a single model. Despite many of the additional analyses showing up in the appendix, to increase robustness of the figures in the main text, it may be worth considering adding additional data to such figures when possible.  \n3. The ablation of the MLP contribution currently sets it to zero. A gradual decrease of the contribution from its original value to zero would be very insightful. Do you see a gradual, dose-response effect in terms of the changes to entropy and sink rate as a result of a gradual ablation?\n\nOverall, despite these minor weaknesses, the breadth of analysis of the paper combined with the novel results tying together two seemingly disparate phenomena, as well as the mix-compress-refine framework, motivate my score of 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UDlXrArqGa", "reviewer": "Reviewer_qBpd", "summary": "The paper attempts to understand the mechanisms tying together some phenomena happening within the internal representations and activations of LLMs and interpret them in terms of information flow. Namely, they tie together massive activations (some activations in the residual stream grow extremely large), attention sinks (some attention heads put all attention on the first begin-of-string (BOS) token), and compression valleys (the representation matrix of an input at middle layers of an LLM has highly concentrated singular values, i.e., is approximately very low rank). They prove some linear algebraic bounds to show that massive activations imply compression valleys. Then, they propose a high-level framework for information flow in LLM layers called \"Mix-Compress-Refine\" where the initial layers mix information throughout the sequence to build context, the intermediate layers compress to preserve only essential information through the massive activations, and the final layers refine the representation for task-specific outputs. They present evidence for this through controlled experiments with real language models from 400M to 120B. They finish the paper by presenting implications for LLM-related tasks such as collecting embeddings (where one wants to collect them from the compressed but unspecialized representations in the middle layers).", "questions": "Q1: Are there other mechanisms to ablate massive activations? Do they still also cause compression valleys? Can you do any kind of formal causal analysis to show the claimed causality?\n\nQ2: Can you trace a simple end-to-end example showing the three stages, using linear probing to validate your theory of information flow and the relevant proposed mechanisms (mix, compress, refine)?\n\nQ3: Can you compare results on different datasets? Are the results substantially different for e.g., prose vs. math questions vs code vs scientific papers (each of these is readily available from public datasets)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- While the discussed phenomena --- attention sinks, compression valleys, massive activations --- have all been extensively explored, they have not been unified under a single explanation before. This would be a nice contribution to research trying to understand empirical phenomena in transformers.\n- The theory is clear and illustrative, showing a worst-case relationship between massive activations and singular value compression. Targeted experiments show that this happens in practice, along with some training dynamics.\n- The Mix-Compress-Refine framework is elegant and, if proven, would be a nice starting point for further interpretability or theoretical studies, and may be used to motivate further empirical methodology.\n- The paper is nicely organized, easy to read, and has nice visualizations.", "weaknesses": "- The paper is not fully rigorous w.r.t. demonstrating causal mechanisms. For example, ablating MLP layers and showing that this removes massive activations and compression valleys does not show that massive activations are causally responsible for compression valleys. (Indeed, ablating MLP layers is not necessarily equivalent to ablating massive activations, as there could be several ways to do the latter.) Instead, this experiment shows that MLPs are (at least in part) a mechanism which causes massive activations and compressive valleys. Of course the theory says that both have to co-occur but it's not clear which causes the other. See \"Questions\" Q1 below.\n- The paper discusses a lot about information flow and the information contained in some tokens, and indeed one can see this informally via the paper's examples by pointing at small vs. large weights/values in attention, etc. However, to add more rigor it may be necessary to do other tasks to confirm the precise amount of information stored in a certain location, using e.g. linear probing. See \"Questions\" Q2 below.\n- The paper uses a single dataset GSM8K for almost all data, introducing FineWeb for a single experiment. The latter is a more diverse dataset, while the former has heavily templated questions and answers. In order to make claims at the level of generality in the paper, probably some assessments and analysis should be done on general data, and any differences/similarities with the current evaluation can be discussed and examined. See \"Questions\" Q3 below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)