Layer by Layer: Uncovering Hidden Representations in Language Models
Authors: Oscar Skean
Md Rifat Arefin
Dan Zhao
Niket Nikul Patel
Jalal Naghiyev
Yann LeCun
Ravid Shwartz-Ziv
Venue: ICML 2025 oral (2025)
Keywords: large language model
entropy
augmentation
intermediate layer
vision transformer
Primary area: deep_learning->large_language_models
TL;DR: An investigation into the quality and characteristics of intermediate LLM layers

Abstract: From extracting features to generating text, the outputs of large language models (LLMs) typically rely on their final layers, following the conventional wisdom that earlier layers capture only low-level cues. However, our analysis shows that intermediate layers can encode even richer representations, often improving performance on a wide range of downstream tasks. To explain and quantify these hidden-layer properties, we propose a unified framework of representation quality metrics based on information theory, geometry, and invariance to input perturbations. Our framework highlights how each model layer balances information compression and signal preservation, revealing why mid-depth embeddings can exceed the last layer’s performance. Through extensive experiments on 32 text-embedding tasks across various architectures (transformers, state-space models) and domains (language, vision), we demonstrate that intermediate layers consistently provide stronger features, challenging the standard view on final-layer embeddings and opening new directions on using mid-layer representations for more robust and accurate representations.

Layer by Layer: Uncovering Hidden Representations in Language Models
Oscar Skean'
Md Rifat Arefin
Dan Zhao Niket Patel Jalal Naghiyev
Yann LeCun
Ravid Shwartz-Ziv 48
Abstract
From extracting features to generating text, the outputs of large language models (LLMs) typi- cally rely on the final layers, following the con- ventional wisdom that earlier layers capture only low-level cues. However, our analysis shows that intermediate layers can encode even richer rep- resentations, often improving performance range of downstream tasks. To explain and quan- tify these hidden-layer properties, we propose a unified framework of representation quality met- rics based on information theory, geometry, and invariance to input perturbations. Our framework highlights how each layer balances information compression and signal preservation, revealing why mid-depth embeddings can exceed the last layer's 's performance. Through extensive exper- iments on 32 text-embedding tasks across vari- ous architectures (transformers, state-space mod- els) and domains (language, vision), we demon- strate that intermediate layers consistently provide stronger features, challenging the standard view linel-layer embeddings and opening new direc- tions using mid-layer representations for more robust and accurate representations.
0.55
0.50
Average Main Score
0.45
0.40
Pythia-410m
0.35
Mamba-370m
BERTitase
100
Layer Fercantage
Figure 1: Intermediate layers consistently outperform final layers on downstream tasks. The average score of 32 MTEB tasks using the outputs of every model layer as embeddings for three different model architectures. The x-axis is the depth percentage of the layer, rather than layer number which varies across models.
In this paper, we conduct layer-wise analysis of LLMs across diverse architectures-inchuiii transformer-basee ones (Vaswani al. 2017), state-space models (SSMs) (Gu & Dao 2024). and encoder-based models like BERT (Devlin et al. 2019) spanning parameter scales from tens of millions to billions. Through systematic evaluation on 32 embedding tasks from the Massive Text Embedding Benchmark (MTEB) (Muennighoff et al. 2022), ) we find that intermediate layers often surpass the final layer by up to 16% in downstream accuracy. Figure [ illustrates this phenomenon, where mid-depth layers provide particularly strong representations while the very last layer can become overly specialized to the pretraining objective.
1. Introduction
Large Language Models (LLMs) have driven remarkable progress in natural language processing (NLP), achieving state-of-the-art results on many tasks (Brown et al. 2020 Devlin al. 2019 Li et al. 2022). At the heart of most applications lies common assumption: final-layer repre- sentations are the most useful for downstream tasks. Yet fundamental question remains: does the final layer always yield the best representation??
A unified framework. To better understand intermedi- ate layers' effectiveness, we combine three complementary perspectives (Section 3):
'University of Kentucky "Mila UUniversity of Montreal "New York University 'Uiiversity of California, Los Angeles "Independent 'Meta FAIR AWa Correspondence to: Oscar Skean coscar.skcan edu..
Information-theoreeti How much do layers com- press preserve semantic information (Shwartz-Ziv
& Tishby 2019 Shwartz-Ziv 2022)
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Geometrie; How do token embeddings unfold in high- dimensional space (Hosseini & Fedorenko 2023 ))

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Invariance: Are embeddings robust to input pertur- bations (e.g., InfoNCE (Oord et al. 2018), LiDAR 2024) and DiME (Skean et al. 2023))?
tasks, such as next-token prediction and masked language modeling, influence the structure of learned representations.
(Thilak et al.
We show that these perspectives can be viewed under a single lens, which clarifies how intermediate layers strike a balance between retaining features and discarding noise.
Layer-wise Analysis in Language Models. Recent work has increasingly focused on identifying which transformer layers encode different types of information. For example, linguistic features such as part-of-speech tags or semantic roles are best encoded by the middle layers of BERT (Liu ct al. 2019: Tenney al., 2019: Voita al. 2019). More re- cent work has shown that mid-depth layers sometimes hold surprisingly robust features, challenging the usual emphasis on final layer representations (Jin et al., 2024]] Gurnee & Tegmark, 2023 Fan et al. 2024). A related line of work in- vestigates the attention sink phenomenon (Xiao et al.] [024] Brunner et al.
Key findings and contributions. Our investigation leads to several important insights:
Intermediate layers consistently oulperform final lay- ers. This pattern is evident in both transformers and SSMs, suggesting a broad chhitectuur--gnostti effect.
Autoregressive vs. masked-language training. Autore- gressive models exhibit pronounced mid-layer "com- pression valley," whereas masked or bidirectional mod- els show milder intermediate changes.
2020 Gu et al., 2025), in which attention disproportionately concentrates on a single token. Notably, intermediate decoder layers have been shown to not exhibit these extreme attention sinks (Barbero et al. [025), sug- gesting they engage in more distributed and meaningful information processing than the shallow or deep layers.
Domain Domain-general effect. We extend these results to vi- sion models and find that autoregressive image trans- formers display the same mid-depth bottleneck, indi- cating that the training objective, rather than the data modality, is the key driver.
Compression and Generalization. Multiple lines of research link compression and generalization perfor- mance (Delctang al., 2024). For instance, [Bordes et al.) (2023) demonstrated that discarding certain layers in self- supervised encoders can even improve downstream accu- racy, while Park et al.] (2024a) found that LLM embed- dings often lie in low-dimensional manifolds. Our empirical study reinforces these ideas by demonstrating that many net- works especalsy autoregressive waanformerrs-nauuria develop mid-layer bottleneck that appears crucial for bal- ancing "signal" versus "noise." We show how intermediate layers can achieve optimal trade-offs between preserving task-relevant information and discarding superfluous detail.
CoT finetuning. Analyzing reveals that finetuning can reshape mid-layer entropy, preserving latent context for multi-step reasoning.
(CoT)
Overall, our results challenge the default reliance on final- layer embeddings and highlight intermediate layers poten- tially underutilized sources of meaningful features. In this paper, we detail our unified framework (Section 3 present extensive experiments in both language and vision (Sec- tion 4 5 6) and conclude with a discussion of our findings, their implications, and future directions.
2. Related Work
Representation Quality Metries. A variety of metries have been proposed to quantify the "quality" of learned representations. We group them into three main categories:
Understanding Neural Representations. A long line of research has aimed to understand how deep neural net- works encode and organize information. Early studies em- ployed lincar probes for intermediate layers (Alain & Ben- gio 2017). while subsequent efforts introduced more so- phisticated techniques such as SVCCA (Raghu et al.] 2017 to compare learned features across architectures and train- ing regimes. Although these approaches have shed light on representation dynamics, most focus on vision models or shallow networks. In contrast, our work contributes to a growing body of literature extending layer-wise analy- sis to large-scale language models, emphasizing specific behaviors of intermediate layers across diverse architec- tures. Complementing our empirical findings. (Saponati Daponas.Saponatii et al.) (2025) present a theoretical analysis of how different pretext
Information-theorrtic measures capture how much a model's internal representations compress preserve relevant information. For example, the Information Bottleneck (Shwartz-Ziv Ziv & Tishby 2019 Shwartz-Zii 2022) analyzes whether intermediate layers discard noise while retaining essential features. Intrinsic di- mensionality, which describes the minimum number of features to represent data, has also been used ana- lyze intermediate layers in LLMs (Cheng al. 2025
Valeriani et al. 2023 Razzhigaev et al. 2024). This line of work has shown semantic abstractions useful for downstream tasks are better encoded in middle layers than last layers in large transformer models. While we do not study intrinsic dimensionality in our subse- quent analysis, it would make a promising direction
We make our code available htttpa://githbb.... OFSkean wwwaaannnnfffaaammmama in flow
2

---

Layer by Layer: Uncovering Hidden Representations in Language Models
for future work.
we consolidate them into unified theoretical framework that shows how these seemingly different metrics connect and why they collectively measure "representation quality."
Geometric measures focus the structure of embed- dings in -ishddimnnsionll space. Classical approaches include analyzing singular values and effective rank of the representation matrix (Garrido et al. 2023). The anisotropy metric of Razzhigaev et al. (2024) has been used to study compression intermediate model lay- ers and we compare our results with their findings in Section 4.2 Anisotropy fits in well with our proposed framework, though we leave formal integration to future work. Recent work explores curvature (Hosseini & Fedorenko 2023) to quantify how smoothly tokens are mapped across consecutive positions or time steps.
3.1. Notation and Motivation
Consider neural network that maps inputs x (e.g., tokens in a sequence) to internal hidden states Z. We denote Z e R N as a matrix of N data samples (or tokens) in D dimensions. Some key questions arise:
1. How compressed are these representations?
2. How robust are they to perturbations or augmentations?
3. How do they geometrically organize different inputs?
Task-based or invariance metrics evaluate how well representations support downstream goals. For in- stance, augmentations- based approaches such as In- foNCE (Oord et al. 2018) and LiDAR (Thilak al. 2024) estimate invariance to perturbations, while meth- ods like NESum or Self-Cluster (Agrawal et al. 2022) link closely to entropy. In computer vision, these scores often correlate strongly with downstream ac- curacy, highlightFing how robust the embeddings are.
Answers to these questions can illuminate which layers strike the right balance between preserving relevant features and discarding noise.
3.2. Matrix- Entropy: A Common Theoretical Thread
We focus on key quantity known as matrix-based entropy
Giraldo et al., 2014; Skean et al. 2023), which applies directly to the Gram matrix K = ZZ Let {A,(K) be the (nonnegative) eigenvalues of K. For any order a > 0, define:
Although these representation quality metric categories may appear distinct, we show (Section() 3) that many can be uni- fied under single lens. This unification illuminates why certain intermediate layers balance compression, geometry, and invariance so effectively, leading better representa- tions for downstream tasks.
1 log (Z) 1 a
(.(K) a Tr(K)
(I)
where rank(K) S min(NN D). Intuitively, if only few eigenvalues dominate, (Z) is small indiiiindi highly compressed representation. Conversely, if Z is spread out across many principal directions, (Z) is large. By varying a, one smoothly transitions between notions like collision entropy (a = 2) and von Neumann entropy (a 1). We will typically use a for simplicity.
Overall, our work bridges these overlapping threads by eval- uating a range of architectures and training paradigms via a unified set of metrics. Beyond merely confirming that intermediate layers can be effective, we elucidate why this happens, tying it to fundamental properties such as entropy, invariance, and geometry. This novel perspective provides an avenue for both finer-grained diagnostics of large lan- guage models and more deliberate design of mid-layer rep- resentations for downstream tasks.
Bridging geometry, invariance, and feature locality. A key benefit of matrix-based entropy is that it unifies multiple representational perspectives:
3. A Unified Framework for Neural Representations
Compression or information content: A handful of large eigenvalues in K = zzT indicates that is low- rank, i.e. the model has collapsed much of the input variation into fewer dimensions. In contrast, a more uniform eigenvalue spectrum implies higher-entropy, more diverse features.
Key Takeaway: Matrix-based entropy unifies seem- ingly disparate metries of representation quality, pro- viding single theoretical lens for analyzing com- pression, geometry, and invariance.
Geometric smoothness: If tokens within a prompt follow trajectory in embedding space with sharp turns, that curvature can manifest as skewed eigenvalue spectra (Hosseini & Fedorenko) (023). Curvature also differentiates local transitions
A central challenge in analyzing internal representations is determining how to assess their quality. Although existing work draws on numerous ideas -arass-frommmo mutual information to geometric manifold analysis to invariance under augmen- tations- these threads can seem disparate. this section,
from
global structural patterns across longer segments or entire prompts.
3

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Invariance under augmentations: Metries like In- foNCE (Oord et al. 2018) and LiDAR (Thilak al. 2024) effectively measure whether augmentations of the same sample (e.g. character swaps) map to similar embeddings. Strong invariance corresponds to stable clustering in zz" which again depends on the distri- bution of eigenvalues and how local vs. global features are retained or discarded.
let VA Bk+1 Zg be the difference between consecutive tokens. The average curvature is:
1 C= L 2
arccos vi- Vi [Ivezillvel]]
Higher curvature means consecutive tokens shift direction abruptly and more local level features; lower curvature sug- gests a smoother trajectory and global level features.
Thus, evaluating S. (Z) provides single lens for assess- ing "representation quality" across compression, geometric structure, and invariance- and highlights how both local details and global patterns are organized.
3.3.3. AUGMENTATION INVARIANCE METRICS
Lastly, we assess how stable the model's representations are to small perturbations of the same input (e.g., random character swaps, keyboard-level changes; see Appendix).
3.3. Representation Evaluation Metries
Suppose a prompt p; is augmented into p!") and p. After embedding these, we compare the row vectors in 2 Z2 RNND under different scoring criteria:
Key Takeaway: Information- Information-theorrtiic, geometric, and invariance-based metrics offer complementary perspectives representation quality that all be understood through matrix-bases entropy.
This self-supervised objective (Oord al.) 2018 encourages matched samples to lie close in embed- ding space while pushing unmatched samples away. A lower InfoNCE loss indicates stronger invariance to augmentation.
InfoNCE.
We now introduce the seven representation evaluation met- rics used in our experiments, grouped into three broad cat- egories: (1) information-theoreeiie (2) geometric, and (3) mugmentation-innariinee All relate back the Gram ma- trix K and hence to Eq.
LIDAR. LiDAR (Thilak et al. 2024) uses a linear discrim- inant approach that measures within-class versus between- class seatter. Treating each prompt as its own class, LiDAR checks how well augmentations form tight clusters.
3.3.1. I. INFORMATION THEORETIC METRICS
Prompt Entropy. Following Wei et al. 2024), we apply matrix-based entropy [T) to the token embeddings within a single prompt. This prompt entropy quantifies how widely tokens are spread in the embedding space. Higher entropy indicates more diverse, less redundant token-level features; lower entropy implies stronger compression.
DiME. Similarly, DiME (Skean al. 2023 grounded in matrix-based entropy. compares real paired samples against random pairings estimate how uniquely aligned correct augmentations are.
3.4. Core Theoretical Results
Dataset Entropy. We can also aggregate embeddings across N prompts by taking the mean token embedding of each prompt to form Z E NNDD, Applying entropy to Z yields dataser-level measure of global diversity-revealing how distinctly the model separates different inputs.
Key Takeaway: Our theoretical framework estab- lishes concrete connections between representation entropy and downstream performance through prop- erties like effective rank and invariance.
Here, we summarize key statements that justify why these metrics meaningfully measure representation quality. We refer to the appendix G for details and proofs. Beyond serv- ing a unifying view, matrix-based entropy also connects to foundational concepts like majorization, Schur concav- ity, and mutual information. Furthermore, we can directly relate the eigenvalue entropy to the matrix entropy, most naturally via the Effective Rank (Roy & Vetterli (007). The following theorem makes this connection explicit.
Effective Rank. Roy & Vetterli 2007) can be shown to
be a lower bound to exp(Si (.). highlighting how dimen- sionality effectively shrinks the representation is strongly compressed. We prove this connection in Theorem [ This has implications for popular representation evaluation met- rics such MankMe (Garrido et al. 2023) and LiDAR (Thi- lak et al. 2024), which are both inspired by Effective Rank.
3.3.2. GEOMETRIC METRICS
Theorem (Lower Bound via Effective Rank). For camnon-based entropy (a + 1).
Curvature. Proposed by Hosseini & Fedorenko (2023). curvature captures how sharply the token embeddings turn when viewed as sequence D. For prompt of length T.,
EffRank(Z) S ex)(Si

---

Layer by Layer: Uncovering Hidden Representations in Language Models
meaning a large effective rank implies high entropy.
How does post-training methods (e.g., fine-tuning and chain-of-thought) reshape representations?
Under appropriate conditions on the data distribution and model, we can show connections between prompt entropy and dataset entropy via the following scaling behaviors:
4.1. Downstream Task Performance
Theorem 2 (Informal).
Key Takeaway: Intermediate layers of language models consistently outperform final layers across all architectures and tasks, challenging the conven- tional wisdom of using final-layer representations.
1. If prompt entropy remains near its maximum for all prompts, then the dataset entropy S grows on the order of loy( )
2. If prompt entropy instead stays near its minimum for all prompts, then dataset entropy grows more slowly, on the order log( L2 N3)
In this section, we use intermediate layers for downstream embedding tasks and employ our unified framework from Section 3 measuring all the embeddings across all layers.
In short, high token-level (prompt) diversity encourages broader global diversity in the dataset-level embeddings, whereas over-compressing token representations can limit how effectively different prompts separate. Our subsequent analysis connects these ideas self-supervised objectives like InfoNCE, which also tie higher entropy stronger ro- bustness and discriminability in the learned representations.
4.1.1. EXPERIMENTAL SETUP
Models We evaluate three distinct architectural families: Pythia and Llama3 (decoder-only transformers) (Biderman) et al. model) (Gu & Dao
2023; Dubey et al.) (024), Mamba (state space
2024). BERT (encoder-only trans-
former) (Devlin et al. 2019) and LLM2Vec models (bidi-
Theorem 3 (Dataset Entropy Bounds InfoNCE). For data X and representation Z(X). the InfoNCE loss on N sam- ples satisfies:
rectional attention) (Behnam Ghader et al.) 2024).
Tasks We test each layer's embeddings 32 tasks from the Massive Text Embedding Benchmark (MTEB) (Muen nighoff et al. (022). spanning classification, clustering, and reranking dor a comprehensive evaluation across various tasks. We refer to the Appendix for details.
log(N)- - InfoNCE I I(X:Z) S H(Z),
where H()) is interpretable as matrix-based entropy at the dataset level. Hence, reducing InfoNCE implies learning a higher-entropy (and thus often more robust) representation.
4.1.2. INTERMEDIATE LAYERS OFTEN OUTPERFORM FINAL LAYERS
Practical outlook. Overall, our theoretical analysis shows that compression (entropy), geometry (curvature, rank), and invariance (e.g. InfoNCE) are all facets of how the Gram matrix ZZ distributes variance. Examining these met- rics across different layers reveals exactly where a network "prunes" redundancy (low entropy) versus preserving essen- tial distinctions (high entropy). This unified perspective also facilitates cross-architecture comparisons (e.g. transformers vs. SSMs) by highlighting how each architecture organizes information internally. Beyond offering theoretical founda- tion, it provides a practical blueprint for diagnosing, tuning, and improving hidden-laye layer representations.
Are final- layer embeddings indeed optimal for downstream tasks? In Figure [ we compare average performance on MTEB tasks across all layers of the three models.
Key observation. In nearly every task, some intermediate layer outperforms the final layer. The absolute improvement ranges from 2% as high as 16% on average, and the best layer often resides around the mid-depth of the network. This phenomena is consistent across all the different archi- tectures. This confirms emerging observations in recent work for generation tasks (Bordes et al. 2023 El-Nouby ct al. 2024, Chen et al. 2020 Fan et al. 2024) and extends them to a wider range of benchmarks and tasks.
4. Empirical Results
In this section, we empirically test our theoretical framework through extensive experiments across architectures, scales, and training regimes. We focus on three key questions:
Why do these layers matter? From our theoretical per- spective, intermediate layers appear to strike balance between retaining sufficient information (avoiding over- compression) and discarding low-level noise. Later in Sec- tion 4.2 we show that these sweet spots are not random but tied to how intermediate layers are processing information.
Do intermediate layers consistently outperform fi- nal layers across diverse downstream tasks?
How do these intermediate representations differ across architectures, training stages, and scales?
5

---

Layer by Layer: Uncovering Hidden Representations in Language Models
X10
Normalizer Prompt Entropy
9.0
1.16
Curvature
1.12
7.5
1.00
Pyltis
6.0
BENT
1.04
40
100
Layer Depth Percentage
Layer Depth Percentage
Layer Depth Percentage
(a) Prompt Entropy
(b) Curvature
(c) LiDAR
Figure 2: Pythia and Mamba's intermediate layers show pronounced changes in representation quality metrics, while BERT's remain more stable. Three representation evaluation metrics calculated on the wikitext dataset for every layer in Pythia-410M, Mamba 370M, and BERT-base architectures. The x-axis denotes layer depth as percentage, allowing fair comparison between models with different layer counts.
4.1.3. LAYER-WIEE METRICS CORRELATE WITH DOWNSTREAM PERFORMANCE
ence representation quality, we compare three fundamen- tally different LLM variants- (encoder-only), Pythia (decoder-only) and Mamba (state-space model)- and then scale up Pythia to observe emerging trends.
To validate framework, analyze how each evaluation metric correlates with downstream performance. Figures show distance correlations between metrics and task scores for Pythia-410M. We find that all metrics exhibit strong relationships with downstream performance. Among them, curvature, DiME, and InfoNCE stand out with partic- ularly high correlations. These associations remain robust across different correlation measures, including Spearman and Kendall, reinforcing the reliability of our findings.
Encoder vs. Decoder vs. SSM. Figure 2 shows how prompt entropy, curvature, and augmentation metrics evolve across each model's layers. BERT, which encodes the en- tire input bidirectionally, generally maintains high entropy across layers, suggesting minimal compression: the model all tokens at once and need not discard much information. By contrast, the decoder-only Pythia exhibits strong mid-layer entropy dip, reflecting its autoregressive objective's 's tendency to filter or prune non-local details in the middle of the network. As a result, Pythia's "sweet spot" for downstream tasks often lies around mid-depth, where
Our results suggest that our metrics capture some aspects of intermediate representations that contribute to downstream utility. In Appendix D we leverage these strong correlations select high- perforrnff layers in unsupervised manner, following (Agrawal al. 2022 Garrido al. 2023 Thi- lak al. 2024). In short, layer that surpasses the final layer in downstream perfor- mance-wiie mance-without using any task-specific labels. For instance, using DiME--baed layer selection leads to a 3% average improvement MTEB scores for the Pythia-410M model.
identify intermediate
1.00
Distncee Conelation
0.50
0.81
0.53
0.86
4.2. Architectural and Scale Differences
Key Takeaway: Different architectures exhibit dis- tinct patterns of information compression. Autore- gressive models show mid-layer bottlenecks while bidirectional models maintain more uniform trends.
0.00
USAA
Babeet
Prongt
INbiCC
Curviture
DIM
Regresentation
Metrics
Figure Relationship between representation metries and task performance averaged across layers for Pythia 410M. Using distance correlation (dCor),
strong as- sociative relationships across the board with DiME exhibit- ing the strongest relationship with downstream performance. We use dCor due to its robustness and ability to measure both linear and non-linear relationships (dCor [0, 1] with indicating statistical independence and indicating strong dependency). We defer additional results the Appendix.
Aside from strong correlations with downstream perfor- mance, can use our evaluation framework to assess the internal behaviors of LLMs. In both this section and Sec- tion we use WikiText-113 (Merity et al.) 2017) for analyzing our representation metrics on standard textual data. To investigate how architecture and model size influ-
6

---

Layer by Layer: Uncovering Hidden Representations in Language Models
balances essential context and compression. Mamba, mean- while, processes sequences through state-space approach that yields flatter, more uniform curves across depth: it nei- ther retains as much information as BERT nor compresses as aggressively as Pythia's mid-layers. These conclusions align with Razzhigaev et al. (2024) which showed a flat layer-wise anisotropy for encoder models and a spike in intermediate layer anisotropy for decoder models.
residuals act as a regularizer (Marion al. 2024 smooth- ing out spurious components in hidden representations.
4.3. Impact of Training Progression
Takeaway: Significant changes during training oc- cur in intermediate layers and early layers stabilize quickly, supporting the detokenization hypothesis.
Scaling Size Effects. In Figure 12 we analyze Pythia models ranging from 14M to 1B parameters. Larger models display more pronounced intermediate compression (en- tropy dips), indicating a heightened ability to distill relevant features. We also observe smoother token trajectories (lower curvature) and stronger invariance (higher LiDAR), consis- tent with findings that bigger models more effectively filter noise and capture long-range dependencies. These trends re- inforce why performance peaks in the middle of the network: larger models hold more capacity to compress intermediate representations, yet still preserve crucial semantic details.
We measure Pythia's metrics at multiple checkpoints to un- derstand how layer-wise representations evolve throughout training (Figures 4 and
11.. Two main observations emerge:
Intermediate Layers Undergo the Most Change. The largest shifts in representation quality occur in mid-depth layers. Specifically, prompt entropy steadily decreases there as training progresses, implying that intermediate layers increasingly compress and abstract the input. Meanwhile, LiDAR scores are minimal in these same layers. Likewise, curvature becomes smoother in the middle of the network, suggesting the model refines its internal structure to capture longer-range more nuanced patterns in language.
Finetuning Effects In Figure 13 we study how finetuning affects the internal representations of Llama3 @Dubey et al. 2024). We compare the baseline LJlama-8B to two fine- tuned LLM2Vee models (Behnam Ghader et al. 2024). The model enables bidirectional attention in Llama3 and goes through two unsupervised training phases to improve Llama3's ' performance on em- bedding tasks. The LLM22ec-mnt--supee
Early Layers Stabilize Quickly. In contrast to interme- diate layers, the earliest layers change very little after the initial phase of training. This observation aligns with the "detokenization" hypothesis of Lad et al.) (2024). which posits that the main functional role of early layers is to convert raw tokens into a basic embedding space. This idea is closely related to the "shared task" layers of Zhao et al. (204),, introduced in the context of instruction tuning on diverse tasks. In particular, they show that the first nine lay- ers of LlaMA 2 7B (Touvron et al. 2023) perform general task-agnostic operations. As a result, the most substantial changes to representations, such as enhanced compression, are driven primarily by the intermediate layers, reinforcing their importance for learning robust, high-level features.
LLM2 wwwttuunuup--aeeee
adds additional supervised finetuning phase. It is clear that both finetuned models have improved augmentation invariance. Furthermore, the unsupervised model has higher prompt entropy than Llama3 while the supervised model has less.
Layer-Level Analysis of Transformer Sub-Components. While our experiments treat each transformer layer as a single unit, transformer blocks are composed of multiple sub-layers (pre-attention normalization, self-attention, resid- uals, MLPs). By measuring entropy after each sub-layer, we find in Figure [5at residual connections drive the mid- network compression observed in Section 4.2 Specifically:
4.4. Impact of Chain-of-Thought Finetuning
Key Takeaway: CoT finetuning enables models to maintain richer context throughout their layers.
Sub-layers before residuals (e.g. pre-attention, atten- tion scores, or MLP pre-residual outputs) often show only mild compression.
Recent work has highlighted Chain-of-Thougle (CoT) fine- tuning as a powerful strategy for improving reasoning capa- bilities (Arefin al. 20255 DeepSeek-Al] (025). To exam- ine its effects on representations, in Figure 5 compare Qwen 2.5 and Qwen 2.5-Math (Yang et al. 2024). where the latter underwent additional math pretraining and CoT finetuning. Measuring token-level prompt entropy across sequence length reveals that the finetuned model maintains higher entropy with lower variance across examples.
Residual sub-layers exhibit a pronounced drop in en- tropy, indicating a significant filtering of information. A concurrent study (Csordás et al. 2025) observed a decrease in the residual stream norm in the second half of decoder models, reinforcing our findings.
The strong entropy "valley" at intermediate layers is tied to how residual paths merge new signals with the existing hidden state. This aligns with prior work indicating that
7

---

Layer by Layer: Uncovering Hidden Representations in Language Models
10a
Nomallzed Prempt Entroyy
0.9
1.17
4.0
10a
1.14
10a
1.11
0.7
2.4
1.08
0.6-
1.05
10"
20
5
10
15
10
15
20
Layer
Layer
Layer
(a) Prompt Entropy
(b) Curvature
(c) LiDAR
Figure Strong trends in intermediate behavior emerge during training Representation evaluation metrics across layers various Pythia-410N. 10M training checkpoints, ranging from step to the final step at 143k. The x-axis is the model layer, showing how training affects different layers, while the colors are different checkpoints during training.
1. Token repetition compresses intermediate layers. As increases (i.e., more repeated tokens), prompt en- tropy decreases sharply in mid-depth layers, suggesting that the model recognizes/encodes repetitive patterns and discards redundancy in its internal representation.
Reg flr
Prompt
Pronge Entrayy
2. Random tokens inflate early-layer entropy. Adding token-level randomness, increases entropy significantly early layers, revealing their sensitivity to noise. In contrast, deeper layers are more robust.
0.0
8.0
Fosttio in the Sequence (Percentags)
Overall, these results confirm that intermediate layers play major role in handling complex unusual inputs, selec- tively compressing or filtering out repetitive patterns while retaining crucial distinctions. Early layers are more sensitive to noise and the incremental benefit of adding more tokens diminishes with prompt length. This behavior highlights the diverse ways in which different layers balance the trade-off between preserving and discarding information, underscor- ing the significance of intermediate representations.
Figure Token-level prompt entropy across sequence lengths for Qwen 2.5 and Qwen 2.5-Math. The base model (Qwen 2.5) exhibits greater prompt compression, while the finetuned (Qwen 2.5-Math) has higher entropy, indicating more information retention.
These findings suggest that CoT finetuning encourages mod- els to preserve more context throughout their hidden layers, enabling better multi-ster stel reasoning. Our framework pro- vides a quantitative lens into how CoT ane-tuning pushes models maintain richer internal representations across sequences, explaining its effectiveness in multi-step tasks. While CoT traces can be inspected directly in these models, our approach is particularly valuable for analyzing models that reason continuous latent space Haa et al. 2024).
6. Comparison to Vision Transformers
Do our findings extend to other domains like computer vision? Vision models employ diverse architectures and training objectives from fully supervised learning to self- supervised methods, and from bidirectional to autoregres- sive encoders. Their diversity provides an ideal testbed to examine how well our findings generalize and how different training objectives shape internal representations.
5. Extreme Input Conditions
We examine several representative vision approaches: VIT (Dosovitskiy al.] (021). supervised transformer trained on labeled data; CLIP (Radford al. 2021),
To better probe the underlying factors affecting represen- tation quality, we inspect each layer's responsiveness to different input types. We use Pythia-410M on three types of extreme prompts and measure prompt entropy across layers (Figure We prove examples of these prompts in Appendix Overall, we find that:
weakly supervised image encoder: BETT (Bao et al. 2022).
self-supervised encoder that reconstructs masked patches; DINOv2 (Oquab et al.) 2024), a self-supervised approach leveraging augmentations and exponential moving average teachers; MAE (He al. 2022). self-supervised approach
8

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Average Hermalizd Prompt Entropy
Averagg Normalizon Prompt Entrpy
6.5
Promet Toleen Length (log scalee)
0.8
Percer of Token Hapettoo Mapeennneetee
0.9
0.8
Dercent Towan Randome
Avaraga Prompt Entrapy
0.0
512
5.5
0.6
0.6
0.6
0.8
256
0.7
0.4
0.6
3.0
0.2
0.0
0.0
10
15
20
1.0
(a) Repetition
(b) Randomness
(c) Random Prompt Length
Figure 6: Prompt entropy across layers of Pythia 410M under various extreme input conditions. (a) Increasing token repetition leads to decreased entropy in intermediate layers. (b) Increasing token randomness results in higher entropy. especially in initial layers. (c) Unnormalized prompt entropy increases with prompt length due to the larger number of tokens. These results demonstrate how the model's internal representations adapt to different types of input perturbations.
that reconstructs images from masked patches; AIM (Hl
transformations at mid-depth.
Nouby et al. 2024), an autoregressive transformer that pre- dicts the next patch in image sequence (tyl-sttle next- token prediction); and AIMv2 (Fini et al.) 2025), which extends AIM with multimodal next-token prediction task. In Figure [4] we evaluate every model layer on ImageNet-Ik with attention probing and our suite metrics.
Autoregression as the driving factor. The strong mid- layer compression observed in LLMs seems to be not purely property of "sequential token data" vs. "image patch data," but rather byproduct of pretraining. While various self- supervised (or fully supervised) objectives in vision foster more uniform feature building across layers, autoregressive vision models develop similar mid-layer bottlenecks that we see in language. Thus, the objective design-whether or not model is autoregresivv--apearr crucial in shaping laser-wise representation quality, regardless domain.
AIM exhibits behavior similar to language models. AIM, which predicts image patches sequentially, exhibits the same entropy "valley" and accuracy peak at intermedi- ate layers that observed in language models like Pythia. This pattern suggests that autoregressive training, whether over text tokens or image patches, consistently creates mid-depth information bottleneck. The sequential predic- tion constraint forces models to compress non-local con- textual information early processing, then selectively re-expand the most relevant features for accurate predic- tion. AIM's strong intermediate performance was first noted in (El-Nouby et al. 2024). Interestingly, while the AIMv2 model does not show improved intermediate accuracy, it still produces an entropy valley. We hypothesize this difference is due to the multimodal text-vision pretext task, which may alter information compression dynamics.
7. Discussion and Conclusion
We investigated the representation quality of intermediate layers in LLMs and their role in downstream task perfor- mance. We introduced unified framework of evaluation metrics, establish theoretical connections among them, and apply these metrics to analyze transformer- architec- tures, SSMs, and vision models. key phenomenon un- veiled by prompt entropy
information bottleneck in the middle layers of autoregressive transformers in both vision and language domains. Furthermore, we show that intermediate layers often surpass final layers in representa- tion quality, holding implications for feature relevance and extraction. DiME, curvature, and infoNCE correlate well with downstream performance, suggesting fundamental connection between representation and generalizability.
Vision transformers behave differently from language models. All models except for AIM exhibit strictly in- creasing downstream accuracy toward final layers. Similar trends have been shown for ResNets (Sorscher al. (2022). where few-shot classification error is strictly decreasing across layers. Most -on-auuoreeressive vision models show steadily increasing dataset entropy. The notable exception is BEIT, which exhibits a substantial intermediate dip. Taken together, the results suggest that without an autoregressive objective, vision transformers have less need for drastic
In conclusion, work studies the internal representation dynamics in LLMs, offering theoretical and empirical in- sights as well as practical implications for optimizing model design and training strategies. Future work should further investigate the underlying causes of intermediate layer com- pression and explicit finetuning to control compression.

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Impact Statement
Behnam Ghader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N., and Reddy, S. LLM2Vec: Large lan- guage models are secretly powerful text encoders, COLM, 2024.
Our paper studies the inner workings of large language models with findings that may challenge typical assump- tions about the importance of intermediate layers in large language models and the representations they learn. Our findings suggest that representations from these layers can yield better performance on a variety of downstream tasks, which can have implications for model interpretability, ro- bustmess, and efficiency.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. ICML,, 2023.
From an ethical standpoint, the ability leverage intermediate-layer representations could impact fairness and bias considerations in evaluating model performance in model deployment. By helping better identify latent features and representations, our approach may amplify latent biases. We welcome and encourage future work to explore methods that can ensure that intermediate-laye representations do not disproportionately reinforce biases or lead to unintended disparities in real-world applications.
Boes, P., Eisert, J., Gallego, R., Müller, M. P., and Wilming, Von neumann entropy from unitarity. Physical review letters, 2019.
Bordes, F., Balestriero, R., Garrido, Q., Bardes, A., and Vin- cent, P. Guillotine regularization: Why removing layers is needed to improve generalization in self-supervised learning. TMLR, 2023.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. NeurlPs, 2020.
Acknowledgements
We thank the anonymous reviewers for their valuable feed- back, which helped improve the clarity and presentation of our results. We are also grateful to Diego Doimo, Artemii Novoselov. Jhoan Keider Hoyos Osorio, Luis Sanchez, and Matteo Saponati (listed alphabetically) for fruitful discus- sions and helpful pointers to related literature. Oscar Skean is supported by the Office of the Under Secretary of De- fense for Research and Engineering under award number FA9550-21-1-0227.
Brunner, G., Liu, Y., Pascual, D., Richter, O., Ciaramita, M., and Wattenhofer, R. On identifiability in transformers. ICLR, 2020.
References
Burns, C., Ye. H., Klein, D., and Steinhardt, J. Discovering latent knowledge in language models without supervision. ICLR, 2023.
Agrawal, K. K., Mondal, A. K., Ghosh, A., and Richards, B. a-ReQ: Assessing representation quality in self- supervised learning by measuring eigenspectrum decay. NeurlPs, 2022.
Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., and Sutskever, I. Generative pretraining from pixels. ICML, 2020.
Alain, G. and Bengio, Y. Understanding intermediate layers using linear classifier probes. ICLR, 2017.
Cheng. E., Doimo, D., Kervadee, C., Macocco, I,, Yu, J., Laio, A., and Baroni, M. Emergence of a high- dimensional abstraction phase in language transformers. ICLR, 2025.
Arefin, M. R., Subbaraj. G., Gontier, N., LeCun, Y., Rish, I., Shwartz-Ziv, R., and Pal, C. VCQ-VCR: Preventing collapse intermediate transformer representations for enhanced reasoning. ICLR, 2025.
Csordás, R., Manning, C. D., and Potts, C. Do language models use their depth efficiently? arXiv, 2025.
Bach, F. Information theory with kernel methods. IEEE Transactions Information Theory, 2022.
DeepSeek-AL. Deepseek-R1: Incentivizing reasoning capa- bility in LLMs via reinforcement learning. arXiv, 2025.
Bao, H., Dong, L., Piao, S., and Wei, F. BelT: Bert pre- training of image transformers. ICLR, 2022.
Deletang, G., Ruoss, A., Duquenne, P.-.., Catt, E., Ge- newein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., et al. Language modeling is compression. ICLR, 2024.
Barbero, F., Arroyo, A., Gu, X., Perivolaropoulos, C., Bron- stein, M., Veličković, P., and Pascanu, R. Why do LLMS attend the first token? arXiv, 2025.
10

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. NAACL, 2019.
Jin, M., Yu, Q.. Huang, J., Zeng, Q. Wang, Z., Hua, W., Zhao, H., Mei, K., Meng, Y., Ding, K., al. Explor- ing concept depth: How large language models acquire knowledge at different layers? arXiv, 2024.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn,
D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Lad, V. Gurnee, W., and Tegmark, M. The remarkable robustness of LLMs: Stages of inference? arXiv, 2024.
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. Competition code generation with alpha- code. Science, 2022.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., AI-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 herd of models, arXiv, 2024.
El-Nouby, A., Klein, M., Zhai, S., Bautista, M. A., Toshev, A., Shankar, V., Susskind, J. M., and Joulin, A. Scalable pre-training of large autoregressive image models. ICML, 2024.
Liu, N. F., Gardner, M., Belinkov, Y., Peters, M. E., and Smith, N. A. Linguistic knowledge and transferability of contextual representations. NAACL, 2019.
Ma. E. NLP Augmentation, 2019. URL https:
Fan, S., Jiang, X., Li, X., Meng, X., Han, P., Shang, S., Sun, A. Wang, Y., and Wang, Z. Not all layers of LLMs are necessary during inference. arXiv, 2024.
github.com/ne wwtp://a/a/a//w/////.....mm
Mallen, A. T. and Belrose, N. Eliciting latent knowledge from quirky language models. ICLR 2024 Workshop on Mathematical and Empirical Understanding of Founda- tion Models, 2024.
Fini, E., Shukor, M., Li, X., Dufter, P. Klein, M., Haldimann, D., Aitharaju, S., da Costa, V. G. T., Béthune, L.. Gan, Z., et al. Multimodal autoregressive pre-training of large vision encoders. CVPR. 2025.
Mamou, J., Le, H., Del Rio, M. A., Stephenson, C., Tang, H., Kim, Y., and Chung, S. Emergence of separable manifolds in deep language representations. ICML, 2020.
Garrido, Q., Balestriero, R., Najman, L., and Lecun, Y. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. ICML, 2023.
Marion, P., Wu, Y.-... Sander, M. E., and Biau, G. Implicit regularization of deep residual networks towards neural odes. ICLR, 2024.
Giraldo, L. G. S., Rao, M., and Principe, J. C. Measures of entropy from data using infinitely divisible kemels.. IEEE Transactions Information Theory, 2014.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. ICLR, 2017.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. COLM, 2024.
Muennighoff, N., Tazi, N., Magne, L.., and Reimers, N. MTEB: Massive text embedding benchmark. EACL, 2022.
Gu. X., Pang. T., Du C., Liu, Q. Zhang, F., Du, C., Wang, Y., and Lin, M. When attention sink emerges language models: An empirical view. ICLR, 2025.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. ICLR, 2018.
Gurnee, W. and Tegmark, M. Language models represent space and time. arXiv, 2023.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. DINOv2: Learning robust visual features without supervision. TMLR, 2024.
Hao, S., Sukhbaatar, S. Su, D., Li, X., Hu, Z., Weston, J., and Tian, Y. Training large language models to reason in continuous latent space. arXiv, 2024.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. CVPR, 2022.
Park, K., Choe, Y. J., Jiang, Y., and Veitch, V. The ge- ometry of categorical and hierarchical concepts in large language models. ICML 2024 Workshop on Mechanistic Interpretability. 2024a.
Hosseini, E. and Fedorenko, E. Large language models im- plicitly learn to straighten neural sentence trajectories to construct a predictive representation natural language. NeurlPs, 2023.
Park, K., Choe, Y. J., and Veitch, V. The linear represen- tation hypothesis and the geometry of large language models. ICML., 2024b.

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021.
Thilak, V., Huang, C., Saremi, O., Dinh, L., Goh, H., Nakki- ran, P., Susskind, J. M., and Littwin, E. LiDAR: Sensing linear probing performance in joint embedding ssl archi- tectures. ICLR, 2024.
Raghu, M., Gilmer, J., Yosinski, J., and Sohl- Sohl-Dickstein, J. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability, NeurlPs, 2017.
Tian, Y., Krishnan, D., and Isola, Contrastive multiview coding. ECCV, 2020.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv, 2023.
Razzhigaev. A., Mikhalchuk, M., Goncharova, E., Os- eledets, I., Dimitrov, D., and Kuznetsov, A. The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models. EACI, 2024.
Valeriani, L., Doimo, D., Cuturello, F., Laio, A., Ansuini, A., and Cazzaniga, A. The geometry hidden represen- tations of large transformer models. NeurlPs, 2023.
Rényi, A.
On measures of entropy and information. Proceed- ings of the fourth Berkeley symposium on mathematical statistics and probability, 1961.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit. J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Atten- tion is all you need. NeurlPs, 2017.
Roy, O. and Vetterli, M. The effective rank: A measure of effective dimensionality. European signal processing conference, 2007.
Voita, E., Sennrich, R., and Titov, I. The bottom-up evolu- tion of representations in the transformer: A study with machine translation and language modeling objectives. EMNLP-IJCNLP, 2019.
Saponati, M., Sager, P., Aceituno, P. V., Stadelmann, T., and Grewe, B. The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in trans- former training. arXiv, 2025.
Wei, L.., Tan, Z., Li, C., Wang, J., and Huang, W. Diff- eRank: A novel rank-based metric for evaluating large language models. NeurlPs, 2024.
Scholkopf, B. and Smola, A. J. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2018
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. ICLR, 2024.
Shwartz-Ziv Information flow in deep neural networks. PhD thesis, Hebrew University, 2022.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang. K., Yu, I.., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Qwen2.5 technical report. arXiv, 2024.
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. Entropy, 2019.
Shwartz-Ziv, R., Balestriero, R., Kawaguchi, K., Rud- ner, T. G., and LeCun, Y. An information theory per- regularization.
spective on NeurlPs, 2023.
Zhao, Z., Ziser, Y., and Cohen, S. B. Layer by layer: Uncov- ering where multi-task learning happens in instruction- tuned large language models. E.NNPP-JCCIPP 2024.
Skean, O., Osorio, J. K. H., Brockmeier, A. J., and Giraldo, L. G. S. DiME: Maximizing mutual information by difference of maarix--aseed entropies. arXiv, 2023.
Zhouyin, Z. and Liu, D. Understanding neural networks with logarithm determinant entropy estimator. arXiv, 2021.
Skean, O., Dhakal, A., Jacobs, N., and Giraldo, L.
G. S. FroSSL: Frobenius norm minimization for self- supervised learning. ECCV, 2024.
Sorscher, B., Ganguli, S., and Sompolinsky, H. Neural repre- sentational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences, 2022.
Tenney, I., Das, D., and Pavlick, E. BERT rediscovers the classical nlp pipeline. NAACI, 2019.
12

---

Layer by Layer: Uncovering Hidden Representations in Language Models
A. Architectural Details
which serves tractable surrogate for tradi- tional Rényi's a-order entropy (Rényi 1961). The quantity is calculated using similarity kernel
2023
2024
In this section, we elaborate on the specific architectures of transformers and State Space Models (SSMs). We outline the mathematical foundations, including the weight matri- ces, attention mechanisms for transformers, and the state transition matrices for SSMs. Detailed equations and rameter configurations are provided to facilitate replication and deeper understanding.
batch of sam- ples drawn from a distribution, without making explicit assumptions on what the true distribution is. The choice of kernel is flexible and can be any infinitely divisible kernel such as the Gaussian kernel, linear kernel, or Laplacian ker- nel, among others. For this work, we restrict ourselves the linear kernel x(a, b) ab?. This choice is motivated by the linear representation hypothesis (Park et al. 2024b) which finds that large language model representations encode high- level concepts such as truth (Burns et al., (023), honesty
A.1. Transformer
The transformer architecture (Vaswani et al. 2017) utilizes self-attention mechanisms. Given an input x, the key (K), query (Q), and value (V) matrices are computed as:
(Mallen & Belrose (024). and part-of-speech (Mamou et al. 2020) linearly separable manifolds.
The equation for matrix-based entropy was previously de- fined in Eq. | One way to interpret Eq. [T the a-order Rényi entropy of the Gram matrix eigenvalue Notice how each eigenvalue is divided by tr(Kz) before being raised to the power. This is so that the eigenvalues of Kz sum to one (because tr(.) = Σi=1 (-)),, which is a necessary condition to treat the eigenvalues as a probability distribution. Futhermore, each eigenvalue of Kz signifies the variance of samples in particular principal component direction (Scholkopf & Smola 2018 If entropy is low, then
Q xXW, K K.
xWv,
where We. Wx E Rdxd and Wy E Rdxdv are learned weights.
The attention weights are calculated using:
QK
A softmax
M)
Voh
TThe non-zero eigenvalues of the Gram matrix ZZ" equiv- alent to those of covariance matrix 2" Z. Using the covariance matrix instead of the Gram matrix in Eq. [ makes no difference and is more computationally efficient if D < N.
where M is mask to enforce causality in autoregressive tasks.
The output is then:
y = AV.
(4)
Entropy of Power Law Eigenvalues
A.2. State Space Models
1.0
a=0.125
SSMs (Gu & Dao 2024) model sequences using recurrent dynamics. The hidden state h, and output y, at time are updated as:
a=0.5
0.8
a=1
a=2
Normalized Entropy
a=3
0.6
a=10
h Ah-1 + Bx,,
(5)
LogDet
y. = Ch, + Dx.,
(6)
0.4
where A E R=X*, B E Rnxd C Iddx, and D E Rdxd are learned parameters.
0.2
B. Discussion on Prompt Entropy
0.0
The first measure of token embedding diversity call prompt entropy. This entropy is measured on the interme- diate tokens and captures how diverse the token representa- tions are.
2
8
10
Beta for eigenvalue power law
Figure 7: The behavior of Eq. for varying values of a on Gram matrices with eigenvalues distributed with a B-power law such that A; 1
We follow the work of (Wei et al. 2024) and use a-order matrix-bace entropy (Giraldo al. 2014 Skean et al.
13

---

Layer by Layer: Uncovering Hidden Representations in Language Models
the eigenvalues form heavy-tail distribution which implies that few components dominate the variance of samples in Z. On the other hand, at maximum entropy, the eigenvalues form a uniform distribution and samples are spread equally in all directions. Matrix-based entropy is reminiscent of the LogDet entropy which uses the determinant Kz to capture how much "volume" a dataset occupies (Shwartz- Ziv et al., 2023] Zhouyin Liu 202I). The LogDet en- tropy is given by SLagDa Z) log det( log 2. One can use Jensen's inequality to show that the LogDet en- tropy is lower bound of Eq when lim,->1 (Appendix J.4 of (Shwartz-Ziv et al. (2023) )
substitutes, swaps, or deletes characters.
The Keyboard augmentation randomly substitutes char- acters with other characters that are at a distance of one as measured on a QWERTY keyboard. For instance, the character "k" may be replaced with "i", "T", "m", or "j".
We use the pseudocode below to do our augmentations us- ing three types of augmentations, using the default library settings for each type. When computing augmentation- invariance metrics like infoNCE DiME, we use the two augmented prompts rather than using one augmented prompt alongside the original prompt. Note that these augmenta- tions may change the token length T of a prompt.
Depending on the choice of a, several special cases of matrix-based entropy can be recovered. In partieular, when lima-11 equals Shannon entropy (also referred as von Neumann entropy in quantum information theory (Bach] 2022, Boes et al. 2019)), and when = 2 it equals colli- sion entropy. Interestingly, the case of a = 2 can be cal- culated without explicit eigendecomposition (Skean et al. 2024). We show in the Appendix Figure hhow varying val- ues of a affect the matrix-based entropy of Gram matrices with eigenvalues distributed with a ß-power law such that A 8-8, It is shown that for larger values of a, smaller eigenvalues contribute more to the entropy.
aug - naf. Saf.Sequential ([
naw. SplitAug (p=0. 3),
nac. Rer.RanddCChaauu (p=0. .),,)
nac. Keyboarddg (p=0.3),
1)
(aug_A, aug_ aug. augment (prompt, num_augmentations-
prompt ->
"The quick brown fox jumps over the lazy dog."
C. Dataset Details
aug_A -> "The quDUk rown fox wEmps ver the 1 azy dog." aug_B -> "The qTuXi bro wn fox uVm) cb3r the la_k dog."
C.1. Wikitext Dataset
We used the wikitext dataset (Merity et al. 2017) for the majority of our experiments in Sections 4.2 and 5 This
was downloaded from Salesforce/ikkitext huggingface. The dataset consists of 100 million tokens scraped from the Featured articles on wikipedia. We filtered out prompts which were less than 30 tokens or were wikipedia section headings.
E. Using Evaluation Metrics as a Performance Proxy
We previously demonstrated strong correlations between our unsupervised evaluation metrics and downstream per- formance. These correlations can be exploited to select high-performing layers for given task entirely without su- pervision, as suggested by prior work (Agrawal et al. 2022]
C.2. MTEB
The 32 tasks we used from the Massive Text Embedding Benchmark (MTEB) are detailed in Table [ They are English language tasks covering clustering, classification, reranking, and entence-to--setene
Garrido et al. 2023 Thilak et al. 2024).
In Figure we apply this unsupervised layer selection approach to Pythia-410M and LLM2 using the 32- task MTEB benchmark introduced in Section 4.1 Rather than computing task accuracies for every layer, we compute DiME, infoNCE, and dataset entropy for each task across all layers in a single forward pass. For each task, we then select the layer that minimizes of these metrics metrres--leveraging their negative correlation with downstream performance.
D. Prompt Augmentations
For the augmentatioon-invariaee metrics such as infoNCE, LiDAR, and DiME, we use the NLPAug library (Mal 2019) to augment our prompts. We use three types of augmenta- tions.
This straightforward yet effective method yields substantial performance improvements with no supervision. For exam- ple, HLME-based layer selection boosts the average MTEB score of Pythia-4108 410M by 3%.
The SplitAug augmentation randomly splits words into two parts by adding a space.
The RandomCharAug augmentation randomly inserts,
14

---

Layer by Layer: Uncovering Hidden Representations in Language Models
F. Extreme Prompts
Definition 2. (Schur-Conveiity) A real-valued function IR is called Schur-convex if = 9 = f(v) S f(1), S f()).
E.. Increasing Repetition
and Schur-concave if < = f(a)
We take regular prompis from the wikitext dataset, tokenize them, and then for each token we randomly replace it with probability p. We draw replacements tokens by sampling random token from within the prompt. We show examples below for varying levels of p.
Lemma 1. The matrix-based entropy, given in Equa- rion 1 is Schur-concave function for > 0. This result is well-known and, for instance, was recently given by Lemma 4.1 in (Giraldo et al. 2014).
Theorem 4. Suppose we have a matrix of embeddings Z R N and its covariance ZT Z. Then the effectivee rank of Z is an lower bound of exp(Si (Z)), where S1 denotes the matrix-based entropy of 1.
(y = 0) Mint records indicate the first gold dollars were produced on May 7...
(p = 0.1) Mint records indicate the first gold dollars were Mint Mint May 7...
Proof. Denote the ordered singular values of Z as 01 > 2 d min (N.D) 2 0 and the ordered eigenvalues of 7 z as A1 2 ... 2 Amin (N.D) 2 0. Without loss of generality,
(p 0.5) Mint records Mint Mint Mint gold dollars were Mint Mint Mint 7...
assume that Eim1 oi ΣN X; = 1. If this is not the
case, then set 03 Σ and X ΣE A
(p- 1.0) Mint Mint Mint Mint Mint Mint Mint Mint Mint Mint Mint Mint Mint...
It straightforward to show that of A, Because Vi a; S 1, have that o; 2 A, This implies that À x Therefore, Si(0) I Si(A) = effective rank (Z) < exp (
E2. Increasing Randomness
We take regular prompts from the wikitext dataset, tokenize them, and then for each token we randomly replace it with probability p. We draw replacements uniformly from the tokenizer distribution. We show examples below for varying levels of p. Unlike the character-level random noise added to prompts in Section with random noise discussed in Ap- pendix D which might change the number of tokens '"' of the prompt, the loken-level random noise used here does not do so.
Proposition 1. (Random Unit Vectors are Nearly Orthog- onal) Suppose we have m unit vectors in RP, that are dis- tributed according to the uniform distribution on the hyper-
sphere. Then with probability least m² / = have that for any pair i, j. F j.
[Vi. vi)l S.
(p = 0) Mint records indicate the first gold dollars were produced on May 7...
Proof. We can begin by defining the central e-band around a slice of the hypersphere Sp as,
T, (= Sn : ((2, ((), <//2,
(p = 0.1) Mint records indicate salivary first gold dollars were produced on May NaCI...
where e1 denotes the first basis vector. The probability of a uniformly distributed vector on the unit sphere not landing in T. C Spp-1 can be bounded as,
(p 0.5) Mint records Dallas actively first dollars persufors on M2yder129 18...
(p 1.0) arf emulsion minorensteinorian- mega TOStack potsRecip Installifykeeping..
PTT.) 2 1 2me
Now, treating Vi as e1, the basis vector, without loss of gen- erality, we have that, when vi, Vj are uniformly distributed on the hyper-sphere,
G. Theorems
Definition 1. (Majorization) Let P. E R be nonnega- tive vectors such that Σn P;- Ei=1 qr We say that q majorizes p. denoted by $ q, if their ordered sequences Pn1 2 ... 2 P|n] and qnj 2- 2 qno satisfy:
P(Kvi,, vi)l < <<) S V2Te -p
Now, by the union bound on each + j. get that,
P(I, (Ivi, vi)l E) S Σ
P(Kvi, vy)l
k
k
P
<
q
for k = 1.... n
S m² V2Tc -
i=1
15

---

Layer by Layer: Uncovering Hidden Representations in Language Models
So then with probability at least 1 - mm² V = that, for any pair i,
we have
The union bound then tells that,
- /2Te 2Te 2N2 P(Vi $ :V VDaj)l NZ) 2 2=22222
K(v.V)) S.
So then with probability at least 1 - N22 draw of the data points, we have that,
N-2 V2Te IN over the
n
Theorem 5. (Maximum Prompt Entropy implies Large Dataset Entropy.) Suppose we have a orthogonally equiv- arient representation model Z such for all sequences Z; = Z(X;) the prompt entropy is maximal and the rows are unit. Suppose also that the data distribution Data is a isotropic unit Gaussian. Suppose we draw sequences of length L = D from the data distribution. Then with prob- _D,² ability /2TE over draw of {xi]i=1 ~ Data, we have that,
N
|| 13
1
So then since,
1
Sz(T log
|| ||:
we have that, -2((2") = || [|3 In particular,
le (e
2 SE
c (2Q
N L
Sc
S
Which completes the proof.
Proof. First note that, since the prompt entropy is maximal for each sample i, which we denote Z, = z(X)), then the matrix Kz = Z is full rank. Since by assumption each row of Z; has unit rows, then we know that || |2.112
Theorem 6. (Minimal Prompt Entropy Implies Small Dataset Entropy.) Suppose we have a orhogonally equivari- ent representation model Z such that for all sequences Z: = (Xx)) the prompt entropy is minimal and the rows are unit. Suppose also that the data distribution Data is isotropic unit Gaussian. Suppose draw sequences from the data distribution. Then with probability 1 N2 /2Te -D2,2 the draw of { DDtaa have that,
L= >k= o. In particular we also know that 0; = Jj for all pairs i by the assumption that the prompt entropy maximized. In particular we then know that Z,Z," is a orthogonal matrix, and the rows of 74 form an orthonormal set. We can then write, for some O, a rotation matrix, that,
2x5 over
N3
le S2(22T)
L 1sc
1 qi L Σ
1 0,1.
2
Proof. Since the prompt entropy is minimal for each sample, we know that each Z(X,) will be rank one matrix, so we can write it as the outer product. In particular, we can write ((x)) = viu;" T. However, since the rows of Z(x;) are of unit length, we know that all the rows are identical, so we may write without loss of generality, Z(X,) v,1 Then, it follows that,
We will denote the average over sequences of length L, across all N samples, by the dataset matrix Z (q1, 12......... is orthogonally equivariant, and the Data distribution is ra- dially symmetric, it follows that these {qi} NN are random points on the hypersphere of radius 1 This means that VI-
T. T.. Since by assumption our model (-))
the matrix VDZ consists of rows that are uniform points on hypersphere of radius! 1. Now notice that,
N L
qi
V:.
T.
12zT
T F
|22
T 1/
L
We will write the dataset average matrix as before as 2 = (q) q2, .Iv) In particular the matrix D has rows that are all unit vectors, and these are randomly distributed uniformly on the hyper-sphere. Now notice that,
1 +
E(VIgg vLai)).
L
inj
Since VLg. is a unit vector this will simplify to,
N
|| |
1B3
|1q
112
Σ itj
gg gi)
1/ 1 (N
E(Vig, v V.g)))
1=1
T.2
(41
N
N2
N2
1
(vo,
Now notice that by proposition, we have that with probabil-
L7
L
itj
_D,?
ity at least 1 - 222 2 2NT
N
N2
+
(Vi,
Vv),
Vi jj: (VI. VJ) S C
1
12
$
N
16

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Now by the prior proposition, with probability least 1
N2 2me pa_2
we know that, for all + j.
e,2
(vi. - N
So then we have that,
N3
N
||| 7
(Ivi,
vy)l - Ecsc
L
L
I
itj
In particular,
N3
le S2(22
T Se
Theorem (Dataset Entropy Bounds InfoNCE Let X Data be a discrete random variable distributed according to the data distribution. Let X + be the Markovian relation between X and the representation Z. Then, the InfoNCE loss on N samples from Data satisfies,
ationnhips betowen Weiric Deriormnne
0.50
0.00
log(N) InfoNCE < I(X: Z) 2) (Z).
4,0
The entropy H (Z) is analogous the Dataset Entropy.
Proof. The first inequality follows simple result from (Oord et al. 2018). Then, use that,
Figure 8: Relationships between representation metrics and task performance averaged across layers for Pythia 410M. Using variety of linear and non-linear measures- Spearman's p. Kendall's t, and distance correlation (dCor)- strong inversely associative relationships with the exception of InfoNCE which shows positive, but still strong associativity. Ranges of p, E [-1, 1] and dCor E [0, 1] with indicating independence and indicating strong dependency.
T(X; H ((Z HZ).
H. Additional Plots & Visualizations
17

---

Layer by Layer: Uncovering Hidden Representations in Language Models
1.00
Distance Correlation (dCor)
0.75
0.50
0.85
0.77
0.68
0.55
0.56
0.25
044
0.00
Prompt Entropy
Curvature
DIME InfoNCE Representation Metrics
LíDAR
Dataset Entropy
Figure Relationship between representation metrics and task performance averaged across layers for BERT. Using distance correlation (dCor), strong associative relationships exhibiting the strongest relationship with downstream performance. We use deor due to its robustness and ability to measure both linear and non-linear relationships (dCor [0, 1] with O indicating statistical independence and indicating strong dependency). Other correlative measures also indicate moderate to strong relationships.
the board with LiDAR and dataset entropy
Task Domain
Tasks
Tasks (32 Total)
Pair Classification
SprintDuplicaneeuustion, 1witterSemFiva2211,, TwinterURI Corpus
3
Classifcaatin
AmuxnnCaneeaauuuuaaaiiicccaa AmnzunReveeaallasiicuasee
Bank-
10
ing77 ing77Chnsificatiin, Fmmotinnllasiffcatinn CROOPDomainClsssiaaaiii MTOPIn-
lentClasilicationn MansielnnenCCasssiiauinn. Mussiveccenaroolussiie
Clustering
ArsivClusteringS2S,, BioniiCCuusiernnSS5 MednsisClusttrinnSS5.,, ReckiiClustering.
StackEachangeellsseringg Vvent/Neeegguupplusseeiigg
Reranking
AskUhwnaDDunoessionns MindSmaallReankingg SeiDoesRR, StacKOverlooe/Diiquus. tions
Sentence to Sentence
BIOSSES, SICK-R ST812, STS13, STS14, STSIS, S1S16, STS17, STSHennhmark
Table I: MTEB Tasks used in experiments covering wide range of different use-cases and domains.
Model
Supervised (Best)
Unsupervised
Naive (Last)
min-DiME
min-infoNCE
min-Dataset Entropy
PYthia-410M
52.0
45.5
48.5
46.2
48.
LLM2Vec-8BB
66.3
63.9
60.0
64.3
50.4
Table Average MTEB Performance (%) across Different Layer Selection Schemes
18

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Normalized Prompt Entropy
Normal Dataset Entropy
1.16
Curvature
1.12
1.08
Pytria
Kamba BERT
2
1.04
20
40
80
100
20
40
60
80
100
25
50
75
100
Layer Depth Porcentage
Layar Parcentage
Layer Depth Parcentage
(a) Prompt Entropy
(h) Dataset Entropy
(c) Curvature
x10
4.5
InfoNCE
3.0
LIDAR
3
DIME
1.5
0
20
100
20
60
100
20
40
80
100
Layer Depth Percentage
Layer Depth Percentage
Layer Depth Per Percentage
(d) infoNCE
(e) LiDAR
(I) DiME
Figure 10: textbfPythia's intermediate layers show pronounced changes in representation quality metrics, while Mamba's remain more stable. Representation evaluation metrics across layers in Pythia 410M and Mamba 370M architectures. The x-axis denotes model depth as percentage, allowing fair comparison between models with different layer counts.
Prompt Entropy
Detaset Entropy
Curvature
0.90
1.16
0.6
Propa eeteony
Curvoture
1.12
0.4
0.60
1.04
Training Stap
1.5
25
20
10
25
Layer LIDAR
InfoNCE
DIME
0.3
InfoNCE
LOAAK
GIMS
0.1
0.0
10
15
20
25
1.0
15
20
30
15
20
25
Layer
Layer
Layer
Figure 11: Representation evaluation metrics across layers at various training checkpoints, ranging from step 1 to the final step at 143k. The x-axis represents the depth percentage of the model, showing how training affects different layers, particularly in the intermediate stages.
19

---

Layer by Layer: Uncovering Hidden Representations in Language Models
-10-1
-10
2.4
7.5
2.1
Normalzes Prompe
UIDA
1.5
40
20
Layer Depti: Parcenkage
Layer Percentage
Layer Depth Parcentage
(a) Prompt Entropy
(b) Curvature
LiDAR
Figure 12: Pythia and Mamba's intermediate layers show pronounced changes in representation quality metrics, while BERT's remain more stable. Three representation evaluation metrics calculated on the wikitext dataset for every layer in fythia-410M, Mamba 370M, and BERT-base architectures. The x-axis denotes layer depth as percentage, allowing fair comparison between models with different layer counts.
5 X10-1
Normalized Prompt Entropy
Uama3
LLM2Vec-mttp-uusuu-sinccc
2
LLMZVec mntp supervised
20
80
100
60
80
100
(a) Prompt Entropy
(b) Curvature
Figure 13: Finetuning affects the internal behavior of LLMs. Representation evaluation metrics across layers for Llama3 and two finetuned versions of Llama3.
20

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Accuracy Difference Relative Final Layer
80
Attention Probing Val@@
AIM
40
AIMv2
BEIT
CLIP
DINOv2
20
MAE
VIT
10
15
20
25
5
10
15
20
25
Layer
Layer
(a) Validation Top-1 Accuracy
(b) Accuracy Difference Relative to Last Layer
0.85
0.8
Normalized Prompt Entropy
0.75
Normal Dataset Entropy
0.7
0.65
0.6
0.5
0.55
0.4
0.45
10
15
20
5
10
Layer
Layer
(c) Prompt Entropy
(d) Dataset Entropy
Figure 14: Comparison vision models trained on different pretext tasks. The dataset is ImageNet-106 TTan et al. 2020) and all models use the same 24-layer VIT-L architecture. The validation accuracy is calculated using attention probing on tokens from frozen backbone layer, following the work El-Noub al. (024
21

---

Layer by Layer: Uncovering Hidden Representations in Language Models
Stages Pythia-410M
Pre Attention
140
Attention Patterns
Attention
Post Attention Residual
Mip
Post Mip Residual
120
Effective Rank
20
20
Layer
Figure 15: Behavior of effective rank at different stages within a transformer block.
22


Reviews:
[{"review_id": "C7O2yFrK9s", "reviewer": "Reviewer_ctdG", "summary": "This paper investigates representation quality across different layers of large language models (LLMs), challenging the conventional wisdom that final-layer embeddings are optimal for downstream tasks. Through systematic evaluation on 32 tasks from the Massive Text Embedding Benchmark, the authors demonstrate that intermediate layers often outperform final layers by up to 16% in accuracy. They propose a unified theoretical framework integrating information-theoretic, geometric, and invariance perspectives to explain this phenomenon, showing how matrix-based entropy can measure how effectively each layer balances information compression and signal preservation. The authors compare multiple architectures (transformers, state-space models) and domains (language, vision), finding that autoregressive models exhibit a pronounced mid-layer \"compression valley\" while bidirectional models show more uniform patterns. Additional analyses examine how representation quality evolves during training, how chain-of-thought finetuning affects mid-layer entropy, and how similar patterns emerge in vision models with analogous training objectives.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer ctdG for the detailed, insightful review and constructive feedback. We appreciate the positive assessment of our claims, methods, theory, and experiments. Below, we address the reviewer's comments and questions, incorporating clarifications and new results. You may also be interested in the new results provided for Reviewers 9n2s (generative tasks). Due to character limits, we could not fully address your suggestions about architectural and computational implications, though we will incorporate these suggestions into the manuscript.\n\n## New Results: Systematic Way to Select Optimal Layer\nWe agree guidance on unsupervised layer selection is valuable. Our correlation results showed unsupervised metrics (entropy, DiME, etc.) can act as proxies for downstream performance. To demonstrate this, **we present new results** (Table 1) evaluating unsupervised layer selection. We compare the naive last layer, the supervised best layer, and layers chosen by selecting those with minimum DiME, Dataset Entropy, or InfoNCE per task. As shown, for both Pythia-410M and LLM2Vec-8B, **specific unsupervised metrics yield better average MTEB performance than the last layer** confirming their utility. We will clarify this procedure and its benefits in the revised manuscript.\n\n### Table 1: Average MTEB Performance with Different Layer Selection Schemes\n\n| Model        | Naive (Last) | Supervised (Best) | Unsupervised (min-DiME) | Unsupervised (min-Dataset Entropy) | Unsupervised (min-infoNCE) |\n|--------------|:------------:|:-----------------:|:------------------------:|:---------------------------------:|:---------------------------:|\n| Pythia-410M  | 45.5%        | 52.0%             | 48.5%                    | 48.1%                             | 46.2%                       |\n| LLM2Vec-8B   | 63.9%        | 66.3%             | 60.0%                    | 50.4%                             | 64.3%                       |\n\n## Responses to Points\n**Detailed Proofs** We agree that additional detail in the derivations for Theorems 5 and 6, connecting local and global token-level properties, would be beneficial. We will include more comprehensive derivations in the camera-ready submission. \n\n**Features in Middle Layers / Concrete Examples:** We concur that discussing intermediate features would strengthen the paper. While detailed analysis is ongoing, we will incorporate a discussion of related work [4] on layer-wise feature clustering and plan to add a brief analysis in the Appendix to provide more concrete examples of information captured by intermediate layers.\n\n**Attention Sink** Thank you for highlighting the connection to the attention sink phenomenon [3]. We hypothesize that the mid-layer compression we observe could be related: attention sinks might channel contextual information through a few critical tokens, creating an information bottleneck. We will incorporate a brief discussion of this potential link in the revised manuscript.\n\n**AR vs. Bi-directional:** Regarding the request for a more developed theoretical explanation for differing compression patterns, we acknowledge this is an interesting area. While we attribute the empirical difference to distinct training objectives, a full derivation is complex. We will refine our discussion and note this as an avenue for future theoretical work.\n\n## Responses to Questions\n\n**1: Best Layers for Different Task Types** We have looked at which layers are optimal for different task types. for Pythia-410M, optimal layers vary: ~50\\% depth for Classification, ~75\\% for Clustering and Retrieval. This suggests task-dependent preferences. We will include these breakdowns in the Appendix and discuss them.\n\n\n\n **2: Effect of Model Scale on Optimal Layer Depth** Optimal depth tends to shift deeper with scale (first half for small Pythia models, ~70\\% for 160M/410M). This ~70\\% depth roughly corresponds to where entropy begins increasing after the dip (Fig 10a, Appendix), suggesting the optimal layer often lies where representational capacity expands post-compression. We will elaborate on this in the revision.\n\n\n\n\n**3: Explicit Optimization During Training**\nExplicit Optimization During Training: While we didn't optimize these metrics during training, we agree it's a promising direction. Concurrent work [2] explored this for finetuning (finding significant compression was less suitable for math tasks requiring full input detail). Optimizing during pre-training is an interesting future direction we will mention.\n\n## References\n\n[1] Sanyal, et al, \"Inheritune: Training smaller yet more attentive language models\",  2024\n\n[2] Arefin, et al, \"Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning\", 2024\n\n[3] Xiao, et al. \"Efficient Streaming Language Models with Attention Sinks.\" 2024\n\n[4] Chen, et al. \"Is Bigger and Deeper Always Better? Probing Llama Across Scales and Layers.\" 2023", "claims_and_evidence": "The paper's central claims are well-supported by substantial empirical evidence. The assertion that intermediate layers outperform final layers is convincingly demonstrated through comprehensive benchmarking across diverse tasks and model architectures. The authors provide both quantitative metrics and visualizations showing how representation quality varies with depth.\n\nThe theoretical framework connecting entropy, geometry, and invariance is supported by mathematical formulations and empirical correlations between these metrics and downstream performance. The correlation analysis (Figures 3, 6, 7) demonstrating strong relationships between their proposed metrics and task performance provides compelling evidence for the validity of their framework.\n\nThe architecture-specific claims about compression patterns are well-supported by consistent findings across multiple model scales and types. The extension to vision models provides additional credibility to their argument that the observed patterns are driven by training objectives rather than data modality.", "methods_and_evaluation_criteria": "The evaluation methods are appropriate and comprehensive. Using the MTEB benchmark with 32 diverse tasks provides a robust test of representation quality across different use cases. The authors' approach of testing every layer systematically on each task allows for direct, fair comparisons.\n\nThe set of metrics developed to assess representation quality (prompt entropy, dataset entropy, curvature, LiDAR, DiME, InfoNCE) collectively capture different facets of what makes representations effective. The combination of these metrics with downstream performance establishes a clear connection between theoretical properties and practical utility.\n\nThe inclusion of multiple architectures (Pythia, Mamba, BERT) at various scales strengthens the generalizability of the findings. The controlled experiments with extreme input conditions (repetition, randomness, varying length) provide additional insights into how different layers process information.", "theoretical_claims": "The paper makes several theoretical claims that appear sound. I verified the connections between matrix-based entropy and effective rank (Theorem 4), which is correctly established. The relationship between InfoNCE and entropy (Theorem 7) is well-grounded in information theory.\n\nThe theorems connecting prompt entropy to dataset entropy (Theorems 5 and 6) provide useful insights into how local (token-level) and global (dataset-level) properties interact. These theoretical results help explain why certain compression patterns lead to more effective representations.\n\nThe unification of seemingly disparate metrics under a common framework of matrix-based entropy is a significant theoretical contribution that appears technically sound, though some of the proofs would benefit from more detailed derivations.", "experimental_designs_or_analyses": "The experimental designs are sound and well-executed. The layer-wise evaluation approach provides a comprehensive view of how representation quality evolves across model depth. The analysis of model architectures is systematic and controls for relevant variables by normalizing layer depths as percentages to allow fair comparison across different model sizes.\n\nThe training progression analysis (Figure 4) effectively captures how representations evolve during training, providing valuable insights into the dynamics of representation learning. The sub-component analysis of transformer blocks (Figure 15) is particularly illuminating, isolating the effects of different components (attention, MLP, residuals) on representation quality.\n\nOne limitation is that while the authors demonstrate superior performance of intermediate layers, they don't provide a systematic way to identify which specific intermediate layer is optimal for a given model or task without empirical testing.", "relation_to_broader_scientific_literature": "The authors have discussed carefully in their paper.", "essential_references_not_discussed": "The authors have discussed carefully in their paper.", "other_strengths_and_weaknesses": "**Strengths:**\n- The integration of multiple perspectives (information theory, geometry, invariance) into a unified framework is elegant and insightful.\n- The cross-domain validation with vision models significantly strengthens the generalizability of the findings.\n- The analysis of sub-components within transformer blocks provides valuable insights into the mechanisms driving the observed patterns.\n- The practical implications for representation extraction are significant and could influence how embeddings are utilized in downstream applications.\n\n**Weaknesses:**\n- The paper doesn't establish clear guidelines for selecting the optimal layer for a specific task or model without empirical testing.\n- The connection between the observed patterns and the actual content being represented at different layers remains somewhat abstract.\n- The paper doesn't discuss potential connections to other observed phenomena in transformers, such as the attention sink effect, which might provide additional explanatory power.\n- The theoretical justification for why autoregressive models develop stronger mid-layer compression than bidirectional models could be more thoroughly developed.", "other_comments_or_suggestions": "- The presentation of results could be enhanced by providing concrete examples of what types of features or patterns are being captured by intermediate vs. final layers.\n- It would be valuable to include a discussion of the computational implications of using intermediate layer representations rather than final layers.\n- A discussion of how these findings might inform architectural design choices for future LLMs would strengthen the impact.", "questions_for_authors": "1. Have you investigated whether there are consistent patterns in which specific tasks benefit most from intermediate versus final layer representations? This could help develop guidelines for which layer to use for different application types.\n\n2. Did you observe any relationship between the optimal layer depth and model scale? While you show that compression patterns become more pronounced in larger models, does this shift where the optimal layer for downstream tasks is located?\n\n3. Your framework suggests that intermediate layers achieve an optimal balance between compression and signal preservation. Have you explored whether this balance can be explicitly optimized during training to enhance representation quality further?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oBq6dpjh9i", "reviewer": "Reviewer_9n2S", "summary": "This study investigates whether intermediate layers of language models offer more informative representations compared to the final layers. The authors propose a unified framework for assessing representation quality, employing seven evaluation metrics categorized into three groups:  information-theoretic, geometric, and augmentation-invariance. The experimental design encompasses three model architectures: decoder-only transformers, state space models, and encoder-only transformers. The study evaluates each layer's embeddings across 32 tasks from the MTEB benchmark, covering classification, clustering, and reranking. Through comprehensive analysis, the research examines the performance of intermediate layers across different architectures, explores their variations under diverse training paradigms, and investigates the consistency of findings between textual and visual models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 9n2S for the detailed review, positive feedback on our claims, methods, and writing, and constructive suggestions. We are encouraged by the reviewer's assessment of our work as engaging and scientifically significant.\nWe address the reviewer's comments and questions below, including new experimental results motivated by the feedback. Note that they may also be interested in the new experimental results provided for Reviewers ctdG (unsupervised mechanism for selecting good intermediate layers).\n\n## Addressing Evaluation Scope (Non-Embedding Tasks):\n\nWe appreciate the reviewer highlighting the importance of evaluating beyond embedding-centric tasks. To address this, **we conducted new experiments on generative and classification tasks.** Evaluating intermediate layers for generation requires obtaining logits, typically via an \"unembedding\" layer. We employed **the TunedLens technique** [1], which enables layer-wise analysis of generative capabilities. These results complement our MTEB findings, showing that (a) intermediate layers can outperform final layers on diverse tasks like QA, and (b) the optimal layer depth is task-dependent. \n\n\n**New Results (QA & Classification)**\n* On MMLU (Question Answering), using Llama3-8B with TunedLens, we found **Layer 28 achieves an average accuracy of 62.1\\%, outperforming the final layer's 61.2\\% (+0.9\\%)**. This demonstrates that intermediate layers can offer benefits beyond MTEB tasks. (See Table 2 for aggregated results).\n* We also performed layer-wise analysis on **ToxiGen** (toxicity detection) and **BLIMP** (grammaticality judgments) using LM-evaluation-harness. Interestingly, optimal performance varied: early layers excelled on ToxiGen, while late layers were best for BLIMP.\n\n\n### Table 2: Average MMLU Accuracy with TunedLens at Different Llama3-8B Layers\n\n| Layer Depth   | 0%    | 25%   | 50%   | 75%                        | 100% (Final) |\n|---------------|:------|:------|:------|:---------------------------|:-------------|\n| MMLU Score    | 22.9% | 22.9% | 25.7% | 60.7% (L28 peak: 62.1%)   | 61.2%        |\n\n## Responses to Questions & Other Points\n\n**1: Rationale for Finetuning/CoT Experiments:** The primary goal of these experiments was not just performance evaluation, but to demonstrate the **utility and sensitivity of our proposed Matrix Entropy framework** for probing LLM behavior under different conditions (finetuning, CoT prompting), aligning with recent work like Seq-VCR [2]. We will reorganize the paper and incorporate relevant figures (currently in the appendix) into the main text.\n\n**2: Experiments on Larger/Advanced Models:** We plan to experiment with the multimodal Llama3.2-11B-Vision model, which provides a valuable bridge between our language (Sec 4) and vision (Sec 5) experiments. Models like Qwen2.5 and Janus are excellent candidates for future work to further scale our findings.\n\n**3. AIM Plots** We apologize for omitting the AIM details and results. These are included in our response to Reviewer U8pM and will be added to the main paper in the camera-ready version.\n\n**4. Additional References** Thank you for suggesting these relevant references on knowledge distillation and layer analysis. They offer valuable perspectives on knowledge transfer and feature encoding in intermediate layers. We will incorporate and discuss these works in the related work section, strengthening our connection to the literature and helping address questions about intermediate features (raised also by Reviewer ctdG).\n\n**5. Discussion of Limitations** We acknowledge the points raised regarding limitations and scope. As discussed above, our new results broaden the task variety beyond MTEB. Regarding limitations involving theoretical assumptions, as noted by Reviewer aJrj, our framework makes certain assumptions (detailed in our response to aJrj). We will clearly outline these in the revised manuscript's discussion section.\n\n\n## Conclusion\nWe have put significant effort into addressing the points raised in your review, including conducting new experiments on non-embedding tasks (MMLU, ToxiGen, BLIMP) using the TunedLens approach, which directly addresses one of your main concerns and further demonstrates the generality of our findings. We hope these additions, along with our clarifications and planned revisions, strengthen the paper and demonstrate the value of our contributions. In light of these new results and our detailed responses, we would be grateful if you would consider raising your assessment of our work. Please do not hesitate to let us know if any further questions or concerns arise; we welcome the opportunity for continued discussion. Thank you again for the thoughtful feedback.\n\n\n## References\n[1] Belrose et. al., \"Eliciting Latent Predictions from Transformers with the Tuned Lens\",  2023.\n\n[2] Arefin, et al, \"Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning\", 2024", "claims_and_evidence": "The claims made in this submission are supported by clear and convincing evidence. The authors conduct a comprehensive empirical investigation, encompassing diverse scenarios such as downstream task performance and parameter variations during training. Through systematic examination of layer-wise representations across various model architectures, the study shows that intermediate layers of language models can encode richer representations compared to their final layers.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are both appropriate and well-justified. The authors conduct a comprehensive evaluation across 32 MTEB tasks, systematically assessing representative models of various architectures through three distinct perspectives: information-theoretic, geometric, and augmentation-invariance. In addition, the authors provide a thorough explanation for their selection of specific tasks and evaluation metrics. \n\nHowever, while the paper conducts experiments across 32 text embedding tasks, these tasks are predominantly focused on classification, clustering, and reranking. The study does not encompass other critical NLP tasks, such as machine translation, question answering, or dialogue generation, which are essential for a more comprehensive evaluation.", "theoretical_claims": "I have reviewed the theorems in the Core Theoretical Results section and it seems right to me.", "experimental_designs_or_analyses": "Most of the experimental design is well-aligned with the study's objectives, establishing a robust connection between the evaluation framework and the empirical findings. \n\nHowever, the experiments \"Finetuning Effects\" and \"Impact of Chain-of-Thought Finetuning\" do not appear directly relevant to the main task. The reviewer is unclear about the rationale for including these experiments.", "relation_to_broader_scientific_literature": "In the field of knowledge distillation, several studies explore the knowledge transfer based on the intermediate layers, which aligns closely with the core contributions of this paper. \n\nRelevant works in this area include: \n\n[1] Knowledge Flow: Improve Upon Your Teachers; \n\n[2] MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalities.", "essential_references_not_discussed": "The paper explores the performance of intermediate layers in language models, which is a topic of growing interest in the field. However, there are relevant studies addressing similar themes that have not been cited:\n\n[1] Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layers; \n\n[2] Transformer Layers as Painters.", "other_strengths_and_weaknesses": "Other Strengths:\n(1) The paper is well-written and well-organized, ensuring clarity for readers to grasp the core arguments and findings.\n\n(2) The research topic is engaging and scientifically significant, with experimental results that offer valuable insights and implications for the broader research community.\n\n(3) The experiments are comprehensive, offering substantial evidence to support the authors' claim that intermediate layers can, in certain cases, outperform the final layers.\n\n(4) The authors have provided their codes for reproducibility.\n\n\n\nOther Weaknesses:\n\n(1) The experimental details and results for the AIM are not presented in the paper.\n\n(2) While the paper employs MTEB as its primary evaluation benchmark, it is important to note that MTEB may not encompass the full spectrum of text data types or tasks. A notable limitation is the absence of low-resource languages or specialized domains such as medicine, law, and others. \n\n(3) It will be more rigorous to discuss the limitations of the paper.", "other_comments_or_suggestions": "(1) Present the experimental details of AIM, such as its scale, along with the corresponding results.\n\n(2) Expand the evaluation to include tasks like machine translation, question answering, or dialogue generation.\n\n(3) Consider extending the dataset domains to areas such as medicine and law.", "questions_for_authors": "(1) What is the rationale for including the \"Finetuning Effects\" and \"Impact of Chain-of-Thought Finetuning\" in the main empirical results?\n\n(2) Are there any plans to conduct experiments on more advanced LLMs or MLLMs, such as Qwen2.5 and Janus?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXPWx32Yrg", "reviewer": "Reviewer_aJrj", "summary": "This paper introduces a unified framework for evaluating representation quality in language models. The framework is based on information theory, geometry, and invariance to input perturbations. The authors analyze how each layer balances information compression and signal preservation, challenging conventional wisdom by showing that intermediate layers outperform the final layers in certain tasks.\n\nThe proposed method is tested across a diverse set of 32 text-embedding tasks, covering various model architectures and domains. The authors also explore when and why these findings hold by linking them to fundamental properties such as entropy, invariance, and geometry via matrix-based entropy.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and feedback. We appreciate you recognizing the novelty in our unified interpretation and extensive evaluation. You may also find our new results relevant (details in responses to Reviewers 9n2s regarding generative tasks and ctdG regarding unsupervised layer selection).We address your specific points below:\n\n### Clarification on Theorem 2 Conditions\n\nYou asked about the conditions required for Theorem 2 and their applicability in practical scenarios. Our theoretical framework relies on three main assumptions:\n\n-  **Orthogonal Equivariance:** We assume an orthogonally equivariant model. While transformers are generally permutation equivariant, this stronger condition enables theoretical tractability, though we acknowledge it's a simplification for standard models.\n\n- **Gaussian Input Data:** We assume input data follows a Gaussian distribution, which simplifies analysis due to its tractable mathematical properties.\n\n- **Representations on Hypersphere:** We assume token representations lie on a hypersphere. This is often approximated in practice due to Layer Normalization in high dimensions, which yields representations with nearly constant norms.\n\n\nWe believe these assumptions collectively provide a reasonable, albeit simplified, foundation for analyzing modern transformers. We will **explicitly state these assumptions** in the revised theorem statements and add a **discussion in the main text** outlining these conditions and the practical scenarios where deviations might occur.\n\n### Clarification on Prompt Construction  \n\nRegarding your question on prompt construction, we give the following explanation and example. The Massive Text Embedding Benchmark (MTEB) framework we use provides standardized code to automatically generate prompts for each task. We used these default MTEB prompts without modification to ensure consistency and reproducibility. We recognize that examples were missing in our submission. We will add a discussion of the MTEB prompting strategy and include concrete examples (like the one below for EmotionClassification) in Appendix D.2.\n\n- **Example Prompt Format (EmotionClassification)** \"Classify the emotion expressed in the given Twitter message into one of the six emotions: anger, fear, joy, love, sadness, and surprise: {{SAMPLE GOES HERE}}\"- \n- **Example Full Prompt:** \"Classify the emotion expressed in the given Twitter message into one of the six emotions: anger, fear, joy, love, sadness, and surprise: Thank you, reviewer!\"\n\n\n### Prompt Entropy\nYou also asked for clarification on prompt entropy. We hope the explanation above clarifies the prompt construction aspect. Regarding prompt entropy itself (our metric measuring token-level information uniformity), we are happy to elaborate further on its calculation or interpretation if specific aspects remain unclear.\n\n\n### Conclusion\nWe have worked to address your points, particularly regarding the assumptions underlying our theory and the details of prompt construction. We hope these clarifications strengthen the paper in your view. Given these explanations and our commitment to updating the manuscript accordingly, we would be grateful if you might reconsider your assessment. Please let us know if any further questions or points require clarification.", "claims_and_evidence": "The main aim of this work is to propose a unified framework for evaluating the representations of layers in LLMs. One of the initial claims is that intermediate layers can surpass final layers in performance due to their superior balance of compression and signal retention. \n\nTo this end, this study provided both a theoretical analysis of the matrix based entropy and an empirical validation using a broad set of tasks and models that seem to confirm the claims of the authors.", "methods_and_evaluation_criteria": "The study evaluates representation quality using metrics derived from information theory, geometry, and robustness to perturbations. Various  measures are used in this context based on nominal works such as the InfoNCE, while considering not only different tasks but also architectural families of models. \n\nTo the best of my knowledge, even though the use of matrix-based entropy is not novel in the community, the interpretation of the arising properties combining the three different information theoretic, geometric and invariance properties, along with an empirical extensive evaluation, is.", "theoretical_claims": "I check the theoretical claims of the paper. I would like some clarifications about the appropriate conditions on the data distribution and model that is required for Theorem 2, and if these hold in practice. \n\nI would  also like some clarifications on the prompt entropy and how we can construct prompts that are appropriate in the context of the provided analysis.", "experimental_designs_or_analyses": "The experiments cover an extensive range of 32 text-embedding tasks, but additional details on how the prompts were constructed and how many were used per experiment,  would help clarify the settings and improve reproducibility. Are these part of the considered datasets?", "relation_to_broader_scientific_literature": "I find that the proposed work is largely based on the previous works of (Giraldo et al., 2014; Skean et al., 2023) that are properly cited. \nEven though this study does not constitute a groundbreaking contribution per se, it does provide a unified framework, concentrating and evaluating the results of previous works.", "essential_references_not_discussed": "In my view, the most essential works are appropriately discussed.", "other_strengths_and_weaknesses": "No further comments.", "other_comments_or_suggestions": "No further comments.", "questions_for_authors": "Most of my concerns/questions have been expressed in the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GZRcZ5vij5", "reviewer": "Reviewer_U8pM", "summary": "The paper proposes a framework for analyzing representations throughout model layers. From the perspective of matrix-based entropy, they primarily measure properties like compression (e.g., prompt entropy), geometric smoothness (e.g., curvature), and augmentation invariance (e.g., LiDAR). They argue that for autoregressive models, intermediate layers outperform late ones, unlike bidirectional models. They demonstrate a similar trend for vision models. They also investigate trends related to model size, training progression, and chain-of-thought models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer U8pM for the detailed feedback and for indicating openness to reconsidering the evaluation. We appreciate the positive comments on our work. We address your specific points below. Note, that You may also be interested in the new experimental results provided for Reviewers 9n2s (generative tasks) and ctdG (unsupervised mechanism for selecting good intermediate layers).\n\n## Addressing Framework Metrics and Theory-Empirical Connection (Claim 1 / Weakness 2)\n\nYou correctly noted that while Sec 3 outlined a framework connecting several metrics (including Dataset Entropy, InfoNCE) via theory, the main empirical discussion focused on a subset. This was primarily due to space constraints. However, we intend to take advantage of the extra page in the final submission to do exactly as you suggest and make a longer discussion. We will utilize the additional page in the camera-ready version to:\n\n - Explicitly discuss the empirical results for **all metrics shown in Appendix Figure 8** in the main text.\n-  Expand the discussion in Sec 4.2 to fully incorporate Curvature and LiDAR trends alongside Prompt Entropy when comparing model architectures (Pythia, BERT, Mamba), addressing the noted brevity in L302.\n\n\n## Addressing Empirical Validation and Presentation (Claim 2 / Weaknesses 1 & 2)\n\n***Missing AIM Performance / AR Vision Validation:*** Thank you for pointing out the missing AIM results in Figure 13. This validation is indeed key. We have now performed this analysis and present the results for AIM (and BEiT) linear probe accuracy layer-wise in Table 3 below. \n\nAs shown, the autoregressive vision model AIM exhibits a modest performance gain (+1.9\\%) at an intermediate layer (75\\% depth) compared to the final layer. This aligns with our findings in language models and provides the requested validation. BEiT, consistent with other non-AR models, peaks at the final layer. We will integrate these results into Figure 13 and the main text discussion.\n\n### Table 3: Val@5 Linear Probe Accuracy on ImageNet-1k at Different Layer Depths\n\n| Model | 0%   | 25%   | 50%   | 75%                | 100% (Final) |\n|--------|:-----|:------|:------|:-------------------|:-------------|\n| AIM    | 3.8% | 13.7% | 28.5% | 82.0% (L18)       | 80.1%        |\n| BEiT   | 2.9% | 7.1%  | 14.6% | 46.8%             | 62.5%        |\n\n\n**Mamba Discussion (Conflicting Metrics)** Regarding the observation at 80\\% depth: Prompt Entropy indicates high compression, while LiDAR indicates high augmentation invariance. These are not conflicting, but rather co-occurring, properties. Thus, the interpretation is: \"Mamba's most compressed layers (via entropy) are also its most augmentation invariant (via LiDAR)\". We will clarify this distinction and provide a more detailed discussion of Mamba's unique trends in the expanded Sec 4.2. We will also include a discussion of BERT, which we omit here for character limits.\n\n## Addressing Residual Connection Claim (Claim 2)\n\nWe appreciate the request for clarification on the \"Residual connections drive compression\" claim.\n\n\n**Figure 15 Legend Details:** To clarify the setup, we used hooks to capture activations at key stages within each transformer block:\n    \n- **Initial Representations** Input to the block's residual stream.\n- **Attention Patterns** Raw attention weights (query-key interactions).\n- **Attention Outputs** Output of the attention mechanism value aggregation, projected back.\n- **Attention Residuals** Result after adding attention output to the initial residual stream and applying LayerNorm.\n- **MLP Output** Output of the MLP layers, projected back.\n- **MLP Output + Residuals** Final block output (input to next layer), after adding MLP output to the **Attention Residuals** stream. This is the typical \"layer output\" we measure.\n\n\n**Explanation**\nComparing MLP Output (step 5) with MLP Output + Residuals (step 6) in Figure 15 reveals the latter has a significantly lower effective rank (is more compressed). This occurs because the norm of the residual stream (Attention Residuals, step 4) is often much larger than the norm of the MLP Output (step 5). When added together, the high-norm residual component dominates the sum, effectively reducing the combined representation's rank/increasing compression compared to the MLP output alone. We will incorporate this detailed walkthrough into Sec 4 and add plots **showing the norms** of these components alongside Figure 15 to make this argument clearer and more convincing.\n\n## Conclusion\nWe hope these responses, clarifications, and new results address the reviewer's points. We will incorporate these changes into the camera-ready version. Thank you again for the feedback. We hope these comprehensive responses and improvements demonstrate the soundness and significance of our work. We would be grateful if you would consider these points in reassessing our submission, and we welcome any further questions.", "claims_and_evidence": "- Claim 1. There is a theoretical perspective using matrix-based entropy that unifies many metrics.\n    - While Sec. 3.4 presents the theoretical connections between matrix-based entropy and prompt and dataset entropy and InfoNCE, these results feel disconnected from the rest of the paper. Namely, dataset entropy and InfoNCE are not tested or discussed in the empirical experiments of the main text.\n    - Only a subset of metrics discussed in Sec. 3.3 appear in the main paper, while the full set appears in Figure 8 of the Supplementary. It would be nice to include a summary of these results in the main paper.\n- Claim 2. The proposed framework reveals why intermediate layers outperform late ones.\n    - The paper does a reasonable job of validating this claim in Sec. 4.2 and Sec. 5, although the presentation could be improved.\n    - The discussion in L302 is too brief. Figures 1 and 2 indeed shows a relationship between trends in performance and prompt entropy, which can be used to contrast autoregressive (Pythia) and bidirectional models (BERT). However, the curvature and LiDAR plots are not discussed. The interpretation in L317 is incomplete — Mamba exhibits conflicting trends across metrics (in prompt entropy at 80% depth the layers are some of the most compressed, in LiDAR they are the least compressed). The result in Figure 1 where BERT’s best representation is at 10% depth should be discussed; the “conventional wisdom” would suggest that the last representation would perform the best, but that is not the case there.\n    - L408 / Figure 13 is missing a key performance result. In Figure 13, it is unclear why AIM is not plotted. To complete the argument, the paper should validate for autoregressive vision models, whether an intermediate layer performs the best as well.\n- Other Claims\n    - Compression increases as models scale. This is shown in Figure 10; I liked this result and it was clearly conveyed.\n    - Compression increases with more training steps. This is shown in Figure 4; I again thought this was a nice result that was clearly conveyed.\n    - Residual connections drive compression. I am not sure I am fully convinced; L346 could be more thoroughly and clearly explained. I would have liked a more detailed explanation on the setup and annotations for the Figure 15 legend (e.g., explaining “pre attention,” “attention patterns,” etc.), and a longer walkthrough on pairings in the figure (e.g., compare the “Post Mlp” and “Post Mlp Residual”) to better understand this analysis.", "methods_and_evaluation_criteria": "See “Claims and Evidence” above.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See “Claims and Evidence” above.", "relation_to_broader_scientific_literature": "The framework of analysis is the main novel contribution of this paper. The paper correlates performance and various pre-existing compression metrics, for autoregressive vs. bidirectional models, and other settings.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n\n- The presentation of matrix-based entropy in Sec. 3.2 is well-motivated and conveyed in an intuitive fashion.\n- The paper evaluates not only on language models but also vision models, demonstrating the generality of its findings.\n- The paper reveals compelling findings like larger models and more training steps lead to more representational compression.\n\nWeaknesses\n\n- The paper is missing some important discussion of state space models and autoregressive vision models (see “Claims And Evidence” above).\n- The paper mentions a number of metrics in its unified framework and core theoretical results in Sec. 3, yet only a subset is discussed in the main text. The presentation could be improved with a more clear connection between the theoretical and empirical results (see “Claims And Evidence” above).\n\nI am open to updating my score if the authors are able to address the listed weaknesses.", "other_comments_or_suggestions": "In Figure 2 and other similar figures, it would be helpful to include some more intuitive semantic label for each metric. I have to spend some effort recalling that lower entropy means more compression, higher curvature means more abrupt changes, etc.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper provides a useful perspective on how representations evolve across the layers of (mostly) language models, including across training. The reviewers and I agree that the experiments are thorough, thoughtful, and clearly presented, and that the work helps to elucidate and unify various prior findings. This seems like a valuable contribution to the literature, potentially suitable for a spotlight.
Decision: Accept (oral)