Emergence of a High-Dimensional Abstraction Phase in Language Transformers
Authors: Emily Cheng
Diego Doimo
Corentin Kervadec
Iuri Macocco
Lei Yu
Alessandro Laio
Marco Baroni
Venue: ICLR 2025 Poster (2025)
Keywords: interpretability
intrinsic dimension
large language models
Primary area: interpretability and explainable AI

Abstract: A language model (LM) is a mapping from a linguistic context to an output token. However, much remains to be known about this mapping, including how its geometric properties relate to its function. We take a high-level geometric approach to its analysis, observing, across five pre-trained transformer-based LMs and three input datasets, a distinct phase characterized by high intrinsic dimensionality. During this phase, representations (1) correspond to the first full linguistic abstraction of the input; (2) are the first to viably transfer to downstream tasks; (3) predict each other across different LMs. Moreover, we find that an earlier onset of the phase strongly predicts better language modelling performance. In short, our results suggest that a central high-dimensionality phase underlies core linguistic processing in many common LM architectures.

Published as a conference paper at ICLR 2025
EMERGENCE OF A HIGH-DIMENSIONAL ABSTRACTION PHASE IN LANGUAGE TRANSFORMERS
Emily Cheng', Diego Doimo?, Corentin Kervadec', Iuri Macocco', ,adde Yu² Alessandro Laio', Marco Baroni Universitat Pompeu Fabra', Area Science Park?, University of Toronto", SISSA ICREA* emi mnilyshana.co cwwnghupf.neeecc
ABSTRACT
A language model (LM) a mapping from a linguistic context output token. However, much remains to be known about this mapping, including how its geometric properties relate its function. We take high-level geometric approach to its analysis, observing, across five pre-trained transformer-baser L.S and three input datasets, distinct phase characterized by high intrinsic dimensionality. During this phase, representations (1) correspond to the first full linguistic abstraction of the input: (2) are the first to viably transfer to downstream tasks; (3) predict each other across different LMs. Moreover, find that earlier onset of the phase strongly predicts better language modelling performance. short, our results suggest that central high- digeniiinnsionaltty phase underlies core linguistic processing in many common LM architectures.
O
INTRODUCTION
Compression is thought to underlic generalizable representation learning (Deletang et al., 2024; Yu et al., 2023). Indeed, language models compress their input data to a manifold of dimension orders-of-magnieee
lower than their embedding dimension (Cai et al., 2021; Cheng et al., 2023; Valeriani et al., 2023). Still, the intrinsic dimension (ID) of input representations may fluctuate the course of processing. We ask: what does the evolution of ID over layers reveal about representational generalizability, and, broadly, about linguistic processing?
The layers of an autoregressive language model (L.M) transform the LM's input into information useful to predict the next token. In this paper, we characterize the geometric shape of this transformation across layers, uncovering profile that generalizes across models and inputs: (1) there emerges distinct phase characterized by a peak in the intrinsic dimension of representations: (2) this peak significantly reduced in presence of random text and nonexistent in untrained models; (3) the layer at which it appears correlates with LM quality; (4) the highest- diieeest-ddmmsionaal representations of different networks predict each other, but, remarkably, neither the initial representation of the input nor representations in later layers; (5) the peak in dimension marks an approximate borderline between representations that perform poorly and fairly in syntactic and semantic probing tasks, as well as in transfer to downstream NLP tasks.
Taken together, our experiments suggest that all analyzed transformer architectures develop, in the intermediate layers, high-dimensiooal representation encoding abstract linguistic information. The results of this processing are stored in representations which are then used, possibly through process of incremental refinement, predict the next token. Besides providing new insights the inner workings of Transformer-based LMs, these results have implications for tasks such as layer-based fine-tuning, model pruning and model stitching.
2 RELATED WORK
The remarkable performance of modern LMs, combined with the opacity of their inner workings, has spurred a wealth of research on interpretabiliit. At one extreme, there are studies that benchmark

---

Published as a conference paper at ICLR 2025
LMs treated as black boxes (e.g., Liang al. (2023)). At the other extreme, researchers are "opening the box" to mechanistically characterize how LMs perform specific tasks (e.g..Meng et al. (2022); Conmy et al. (2023); Geva et al. (2023); Ferrando et al. (2024)). We take the middle ground, using geometric tools to characterize the high-level activation profiles of LMs, and we relate these profiles to their processing behaviour. In particular, we draw inspiration from the manifold hypothesis, the idea that real-life high-dimensional data often lie on a -in-dimeniional manifold (Goodfellow et al., 2016): we estimate the intrinsic dimension of the representational manifold at each LM layer to gain insight into how precisely layer geometry relates to layer function.
The notion that nominally complex, high-dimensional objects can be described using few degrees of freedom is not new: it underlies. for instance. a number of popular dimensionality reduction methods such as standard (linear) PCA (Jolliffe, 1986). But, while PCA is linear, the data manifold need not be: such, general nonlinear methods have been proposed to estimate the topological dimension, or manifold dimension, of point clouds (see Campadelli et al. (2015) for a survey). As neural representations tend to constitute nonlinear manifolds across modalities (Ansuini et al., 2019; Cai et al., 2021), we use a state-of-the-e nonlinear ID estimation method, the Generalized Ratios Intrinsic Dimension Estimator (GRIDE) (Denti et al., 2022), which we further describe in Section 3.4.
Deep learning problems tend to be high-ding sigh-dimensional. But, recent work reveals that these ostensibly high-dimensional problems are governed by oow-dimensional structure. It has, for instance. been shown that common learning objectives and natural image data lie on low-dimensional manifolds (Li al., 2018; Pope al., 2021; Psenka al., 2024); that learning occurs in low-dimensional parameter subspaces (Aghajanyan et al., 2021; Zhang et al., 2023); that modern neural networks learn highly compressed representations of images, protein structure, and language (Ansuini al., 2019; Valeriani et al., 2023; Cai et al., 2021); and, moreover, that lower-ID tasks and datasets are easier to learn (Pope et al., 2021; Cheng et al., 2023).
In the linguistic domain, considerable attention has been devoted to the ID of LM parameters. Zhang et al. (2023) showed that task adaptation occurs low-dimensional parameter subspaces, and Aghajanyan et al. (2021) that low parameter ID) facilitates fine-tuning. In turn, the low effective dimensionality of parameter space motivates parameter-eff efficciet finettuning methods such as LoRA (Hu et al., 2022), which adapts pre-trained transformers using low-rank weight matrices.
Complementary to parameter ID, a number of works focus on the ID of representations in LM activation space. In particular, Cai et al. (2021) were the first to identify low-dimensional manifolds in the contextual embeedding space of (masked) LMs. Balestriero et al. (2023) linked representational ID to the scope of attention and showed how toxicity attacks can exploit this relationship. Tulchinskii et al. (2023) demonstrated that representational ID can be used to differentiate human- and Al-generated texts. Cheng et al. (2023) established relation between representational ID and coermattion-thhorttte compression, also showing that dataset- lataset-specific ID correlates with ease of fine-tuning. Yin et al. (2024) showed that the local ID of specific inputs in specific layers can be used to diagnose model truthfulness.
Closer to our aims, Valeriani al. (2023) studied the evolution of ID) across layers for vision and protein transformers, with preliminary analysis of a single language transformer tested on a single dataset (a similarly preliminary analysis is also provided by (Yin al., 2024)). Like us, Valeriani and colleagues found that ID develops in different phases, with a consistent early peak followed by valley and less consistent markers of a second peak. We greatly extend their analysis of linguistic transformers by investigating the functional role of the main ID peak in five distinct LMs. Importantly, our converging evidence suggests that, in language transformers, semantic processing of the input first takes place under the early ID peak, contra the preliminary evidence by Valeriani and colleagues that this crucial phase is the low-ID "valley" between the peaks.
3 METHODS
3.1 MODELS
We consider five causal LMs of different families, namely OPT-6.7B (Zhang et al., 2022), Llama- 3-8B (Meta, 2024), Pythia-6.9B (Biderman et al., 2023), OLMo-7B Groeneveld et al. (2024), and
2

---

Published as a conference paper at ICLR 2025
Mistral-7B (non-instruction-tue tuned) (Jiang et al., 2023), hereon referred to OPT, Llama, Pythia, OLMo, and Mistral, respectively.
OPT, Pythia, and OLMo make public their pre-training datasets, which are a combination of web- scraped text, code, online forums such as Reddit, books, research papers, and encyclopedic text (see Zhang al. (2022); Gao et al. (2020); Soldaini et al. (2024), respectively, for details). The pre-training datasets of Llama-3 and Mistral are likely similar, though they remain undisclosed at the time of publication.
All language models considered have between 6.5 and 8B parameters, 32 hidden layers, and a hidden dimension of 4096. Moreover, they all inherit the architectural design of the decoder-only transformer (Vaswani et al., 2017) with a few notable changes: 1) layer normalization operations are applied before self-attention and MLP sublayers: 2) Pythia adds the self-attention and MLP sublayer outputs to the residual stream in parallel, while in other models self-attention outputs are added to the residual stream before the MLP: 3) Llama/Mistral/OLMMG replace the ReLU activation function with SwiGLU (Shazeer, 2020), and Pythia uses GeLU instead (Hendrycks & Gimpel, 2016); 4) all models but OPT replace absolute positional embeddings with rotary positional embeddings (Su et al., 2022); 5) to facilitate self-attention computation, Llama uses grouped query attention (Ainslie et al., 2023), Mistral applies a sliding window attention, and Pythia adopts Flash Attention (Dao et al., 2022). In all cases, library h1 dden dden_states variable.'
our per-layer representations the vectors stored in the HuggingFace transformers
In autoregressive models, due the causal nature of attention, input sequence's last token representation is the only one to contain information about the whole sequence. Furthermore, it is the only one that is decoded at the last layer to predict the next token. For these reasons, we choose to represent input sequences with their last token representation at each layer.
3.2 DATA
Since we focus on model behavior in-distribution, we compute observables using three corpora that proxy models' pre-training data (all accessed through HuggingFace):: Bookcorpus (Zhu et al., 2015); the Pile (Gao et al. (2020); precisely, the 10k-document subsample available on HuggingFace) and WikiText-103 (Merity al., 2017). From each corpus, we sample, without replacement, total of 50k distinct 20-token sequences (sequence length is counted according to the number of typographic tokens in the text of each corpus). We then divide these samples into partitions of 10k sequences each, which we use for the experiments. We do not constrain the sequences to have any particular structure: in particular, their final element is not required to coincide with the end of a sentence. We additionally generate 5 partitions of 10k 20-token sequences from shuffled versions of the same corpora. That is, we first randomize the order of the tokens in each corpus, and then proceed as with the non-randomized versions. The shuffled samples respect the source corpus unigram frequency distribution, but syntactic structure and semantic coherence are destroyed.
3.3 PROBING AND DOWNSTREAM TASKS
To relate representations' geometry to their content, we use the probing datasets of Conneau et al. (2018), meant to capture the encoding of surface-related, syntactic, and semantic information in LM representations. For each layer, we train a lightweight MLP probe from the hidden representation to each linguistic task. Details on tasks and training procedure are given in Appendix I.
If there is a relation between ID and abstract linguistic processing, then ID may also predict ease- of-transfer to downstream NLP tasks. To test this claim, we consider two such tasks, sentiment classification of film reviews (Maas et al., 201 1) and toxicity classification (Adams et al., 2017), for which we train binary linear classifiers on each hidden layer representation (training details in Appendix J). Note that we do not fine-tune the layers themselves, as this might change their IDs, which is the very property we want to relate to task performance.
'ach hidden sildde__stteeee s vector corresponds to the representation in the residual stream (Elhage et al., 2021) after one attention and one MI.P update.
'We replicated the Pile-based Pythia and OPT experiments with sequences extended to 128 tokens. We obtained very similar results, confirming that the sequences we are using cover the typical contextual spans encoded in model representations.
3

---

Published as a conference paper at ICLR 2025
3.4 INTRINSIC DIMENSION
Real-world datasets tend to show a high degree of possibly non-linear correlations and constraints between their features (Tenenbaum al., 2000). This means that, despite very large embedding dimension, data typically lie on a (locally smooth) manifold characterized by a much lower dimen- sionality, referred as its intrinsic dimension (ID). This quantity may be thought of the number of independent features needed to locally describe the data with minimal information loss (Bishop, 1995). Equivalently, it has been defined as the dimensionality of the support of the probability distribution from which the data is generated (Fukunaga, 2013; Campadelli et al., 2015).
In almost every real-world system, the ID depends on the scale, i.e. the size of neighbourhood at which the data is analyzed. In particular, at small scales, the true dimensionality of the manifold is typically hidden by that of data noise. At very large scales, the ID estimate can also be erroneous, due, for instance, to the curvature of the manifold (Facco et al., 2017; Denti et al., 2022). For this reason, in order to obtain a reliable and meaningful ID estimation, a proper scale analysis is necessary. This is typically performed by varying the amount of the neighbours considered in estimating the ID and looking for an interval of scales in which the estimate is approximately stable (see Appendix C for further details). To this aim, we opted for the generalized ratios intrinsic dimension estimator (GRIDE) of Denti et al. (2022), which extends the commonly used TwoNN estimator of Facco et al. (2017) to general scales. We selected GRIDE because it allows probing, in rigorous framework, the dependence of ID on scale. While the original TwoNN estimator assumes local uniformity up to the 2nd nearest neighbor, GRIDE relaxes this assumption to produce unbiased ID estimates up to the 2kth nearest neighbor (k being the scale).
In GRIDE, the fundamental ingredients are ratios Pi,kk. = Ti.2k//rikk where ri.y the Euclidean distance between point i and its j-th nearest neighbour. Under local uniform density assumptions, the
that depends on the k follow a generalised Pareto distribution fu..e.. (4) = d'y -17-1
B(k (
intrinsic dimension d of the manifold, where B(..)) is the beta function. By assuming the empirical ratios f to be independent for different points, one obtains an estimate of the intrinsic dimension d by numerically maximizing the mentioned likelihood.
For each (model, corpus, layer) combination, we perform an explicit scale analysis. To do so, we first estimate the ID) while varying k. Then, by visual inspection, we select a suitable k that coincides with a plateau in ID estimate (Denti et al., 2022). An example of such a scale analysis is reported in Appendix C, where we also demonstrate estimates to be robust to changes in scale.
In order to delimit high-ID peaks across layers, we conventionally locate the end of the peak at the closest inflection point after its maximum value. The beginning of the peak corresponds, then, to the last layer before the maximum with value equal or greater than that at the end of the peak.
3.5 QUANTIFYING THE RELATIVE INFORMATION CONTENT OF DIFFERENT REPRESENTATIONS
We wish to relate dimensional expansion or compression of representations across layers to changes in their neighborhood structure. If neighborhood structure defines a semantics in representational space (Boleda, 2020), then layers whose activations have similar neighborhood structures perform similar functions.
In partieular, the layers of an LM are iterative reconfigurations of representation space. We quantify the extent of reconfiguration (conversely, stability) using a statistical measure called Information Imbalance (Glielmo et al., 2022), hereon referred to as A. Given two different spaces A and B, this quantity, defined in equation 1, measures the extent to which the neighborhood ranks in space A are informative about the ranks in space B (since ID computation is based on Euclidean distance, ranks are also obtained with Euclidean distance):
A(A - B)
B :
N
In words, A is the average rank of point j with respect to point i in space B, given that j is the first neighbour of i in space A. If A(A + B) ~ 0, space A captures full neighborhood information about space B. Conversely, if A(A + B) ~ 1, space A has no predictive power on B.
FE.g, Ansuini et al. (2019); Tulchinskii al. (2023); Valeriani et al. (2023).
4

---

Published as a conference paper at ICLR 2025
As A is rank-based measure, it can be used to compare spaces of different dimensionalities and/or distance measures. In our specific case, this property implies that robust to possible dimension misalignments between layers.
In comparing the layers' representation spaces, we also considered three alternative measures broadly used in deep net representation analysis (see Sucholutsky al., 2024; Williams, 2024, for recent surveys): Doimo et al. (2020)'s 's neighborhood overlap measure, also used by Valeriani et al. (2023), Representational Similarity Analysis (Kriegeskorte al., 2008), and linear CKA (Kornblith et al., 2019). Only the last measure resulted in interpretable results, broadly coherent with those obtained with A (Appendix H).
We choose to focus on A as the most principled measure for the investigation of LM layers, where we can make very little assumptions about the shape of the manifold. Moreover, crucially, A, unlike the other measures we considered, is non-commutative with respect to its arguments. That is, it is asymmetric upon a swap of spaces: ((A + B) + A(B - A). This feature allows us capture directional information containment. For example, we will see below that A detects the asymmetric relation between Pythia/OPT and Lllama representations (Figure 4), and it allows us to study the degree to which information in certain layer directionally predicts information in the first, next, or last layer (Figure 2 and Figure 3).
RESULTS
We find, in line with previous work, that LMs represent language on manifold of low intrinsic dimension. Furthermore, the representational ID profile over layers reveals a characteristic phase of both geometric and functional significance, marking, respectively, a peak in ID and transition in between-layer neighborhood similarity (A), and a transition to abstract linguistic processing.
4.1 EMERGENCE OF A CENTRAL SEEH-DDIENSIONALTTY PHASE
Figure (left) reports the evolution of the ID for all models, averaged across corpora partitions (for per-corpus results, see Appendix C). In line with previous work (Cai et al., 2021; Cheng et al., 2023; Tulchinskii et al., 2023; Valeriani et al., 2023), we first observe that the ID for all models is ( which lies orders of magnitude lower than the models' hidden dimension at 4096 O(19)).
Average ID over Layers
Average ID over Layers (shuffied)
Layerwise ID aver Training (pythia)
45
intrinsic dimersioo
Intrinsic dimennsion om
2.5
25
30
Itama
25
Intrinsic
opt
20
20
4030
512
mistral
15
pythia
10
1.0
1.5
20
25
30
5
10
15
20
25
30
layer
layer
layer
Figure 1: Average ID over layers over 5 random partitions from each of three corpora: Bookcorpus, the Pile and Wikitext. (Left): different models' layerwise ID plotted for the original corpora. (Center): different models' layerwise IDD plotted for the shuffled corpora. (Right): different Pythia training checkpoints layerwise ID on the original corpora, where darker curves are later checkpoints. In the middle layers, shuffled corpus ID (center) is lower than non-shuffled ID (left), suggesting that linguistic processing contributes to ID expansion. ID increases over the course of training for all layers to reach the final profile at step 143000 (right), suggesting that ID reflects learned linguistic features. All curves are shown with + 2 standard deviations (shuffled SDs are very small).
All models clearly go through a phase of high intrinsic dimensionality that tends to take place relatively early (starting approximately at layer 6 or 7, and mostly being over by layer 20), except for OPT, where it approximately lasts from layer 17 to layer 23. For all models, we also observe second, less prominent peak occurring towards the end: only for OLMo, this second peak, which we do not analyze further, is high the first one. Appendix D reports ID profiles for smaller and
5

---

Published as a conference paper at ICLR 2025
ID
dayy 1TEEE
I
-.m
13
12 Yoak
13 Park
13 Peat
Intranrr derdddimeeain stom
2.
25
35
12
Figure 2: For Llama, OPT, Pythia (left right), the ID overlaid with Al; + (irrtt (gray) and A4 + hast) (brown). Plots are shown with +: 2 standard deviations over 5 partitions of 3 corpora. For all models, there is peak in A (14 (irst) (gray) around the ID peak.
Jama
Pythia
30
30
Intrinei dimarsbon
Intrineic dimanssoon
A000 PHA 200
000.00 Duttaa
12
Figure 3: Forward A scope (left: Llama; center: OPT; right: Pythia): continuous lines report, for each layer In, the number of adjacent following layers In** for which A(,, + bn44) S 0.1. The dashed line represents the longest possible scope for each layer. Values are averaged across corpora and partitions, with error bars of + 2 standard deviations.
larger LM from the Pythia family, showing that the presence of the ID peak is not dependent on the specific size range we are focusing on.
ID is geometric signature of learned structure Figure (center) shows that, when the models are fed shuffled corpora, II) remains low all throughout the layers. This suggests that the presence of high-ID peaks depends on the network performing meaningful linguistic processing. We further confirm this notion in Figure (right), where we analyze the evolution of the ID profile over the course of training for Pythia (whose intermediate checkpoints are public). We find that, over the course of training, the IDs' magnitude not only grows over time, but that they become more peaked, indicating that the characteristic profile emerges from learned structure.
The ID peak marks a transition in layer function Figure reports A(4 -> ffrrst) and A -> lisst) for Llama, OPT and Pythia (with the remaining models in Appendix E, that also presents further analysis). We observe that the peak largely overlaps with a peak in A(l + fiiss).. Here, large value of A implies that sequences which are nearest neighbours at the ID-peak are distant from each other in the input layer. That is, the ID peak layers no longer contain the information encoded in the initial representation of the sequence, suggesting they instead capture higher-level information. Meanwhile, we note that the (4; - listt) profiles are not clearly related to the first ID peak, and we leave their analysis to future work.
Figure shows the forward A scope profile for Llama, OPT and Pythia (see Appendix F for Mistral and OLMo). In particular, the plots indicate, at each source layer ln for how many contiguous following layers ln+ the quantity A(In - In+*) is below a low threshold (0.1). Qualitatively, this means that the source layer In contains most of the information the following layers. Coinciding with the ID peak is downward dip in forward-scope, compatible with the interpretation that, while high-ID layers process similar information, this information differs from that of later layers.
At the ID peak, different models share representation spaces The high- dimensioinii peaks contain similar information across models, as shown in the representative comparisons of Figure 4, which display cross-model A averaged across corpora and partitions (see Appendix G for the

---

Published as a conference paper at ICLR 2025
opi 02.
19-
19
16
16
33
10
10
10
25 28
15
Figure 4: Cross-model A. ID-peak sections are shaded in orange. Different symbols mark different A levels in the two directions (lower values correspond to a stronger trend towards information containment). High A scores (> 0.1), corresponding to low information containment, are not shown. Values averaged over corpora and partitions.
other pairs, and Appendix H for broadly comparable patterns emerging from CKA). The high-ID layer intersections always correspond to regions with a concentration of low-Å values, indicating that, between models, representations at the peak have close neighborhood structures, and thus capture similar semantics (note, still, that low-A values can also occur outside the peak intersections, suggesting high ID might be sufficient, rather than necessary, condition for cross-model similarity).
We also notice a marked asymmetry in which the ID-peak layers of Pythia and OPT, the two models with the highest absolute IDs, directionally contain other models' representations. On the other hand, when models have similar maximum IDs (such as in the Pythia vs. OPT comparison), A is more symmetric and very small, implying that their representation spaces are really equivalent.
4.2 LANGUAGE PROCESSING DURING THE HIGH-DIMENSIONALITY PHASE
We just saw, via geometric evidence (A), that the high-ID phase marks change in processing function. Now, we attempt to interpret what that function is. To do so, we look at layer-wise classification accuracies for the probing tasks described in Appendix I. We find that, in general, ID-peak representations fail surface-form tasks but excel at semantic and syntactic tasks, indicating a functional transition from superficial to abstract linguistic processing.
The ID-peak representations contain less surface-form information In Figure 5a, we consider the accuracy for two tasks, Sentence Length and Word Content, which test whether layer retains information about superficial properties of the input. We observe that the ability to correctly recon- struct the length, measured in tokens, of the input sentence gets lost and performance starts a sharp decrease at the onset of the ID peak or shortly before it. Intriguingly, for OPT (Figure 5a, center), which is the model with the latest ID peak, initial accuracy relatively stable, and only starts to significantly decrease at the onset of the peak, further suggesting that it is only during the high-ID phase, even when the latter occurs relatively late, that superficial information (such as the number of tokens in the input sentence) is discarded by the models, that, as we are about to see, start instead at that point to process more abstract syntactic and semantic information.
climb the network layers,
Concerning the Word Content task, which tests the ability to detect the presence of specific words in the input, we generally observe a great decrease after the first few layers, particularly clear during ID expansion. Interestingly, accuracy tends to go up again after the ID peak, probably because, as the model prepares to predict the output, more concrete lexical information again encoded in its representations. Together, the surface-form tasks confirm the evidence from A(, - (fsree) (Figure above) that the ID peak processes a more abstract type of information that, as we are about to see, may relate to the syntactic and semantic contents of the sequence.
The ID peak marks transition to syntactic and semantic processing Figure 5b reports accuracy for Llama, Pythia, and OPT (results for the remaining models in Appendix I) on the syntactic and semantic probe tasks. As described in detail in Appendix I, these tasks require model to
7

---

Published as a conference paper at ICLR 2025
Intrinsic dimerssion
34
Ilama
109
1.0
intrinsic dim dimersion
-3
opt
3.6
100
AoRuncee uopaplla sqpuu
dim limerrsion
-0
45
pythia
1D
0.9
1.
9.7
0.7
0.7
0.6
0.6
0.6
23
25
24
0.4
3.4
0.3
0.3
Hegh
InnieTTc augh
0.2
Antlanes hengh
0.1
13
0.1
layer
layer
layer
(a) Surface-form probes
Ilama
opt
pythia
Ase.mooe uodapijRa agoid
foeunnoe uadapıRA agoid
Intritolc dirversiion
34-
intrinsic dimensionn
37
0.9
-8-
-21
21
0.7
23
Feal
25
Feak
0.6
24
Paak
coerd ratan exarsin
1..5
coord natinn meanion
0..
16-
Good ration mimnion
10
1.5
20
25
layer
layer
layer
(b) Semantic and syntactic probes
Figure 5: Linguistic knowledge probing performance + SDs across 5 random seeds is shown with the ID profile for Llama, OPT, and Pythia (left right). Row (a) corresponds surface-form tasks Sentence Length and Word Content, where probe performance decreases through the ID peak. Row (b) corresponds syntactic and semantic tasks Bigram Shift. Coordination Inversion and Odd Man Out, where probe performance for all tasks attains maximum (or close) within the ID peak. This suggests the ID peak marks abstract, and not surface, representations of the input.
detect whether sentence is syntactically well-formed (Bigram Shift), whether it is semantically coherent (Odd Man Out) and whether it describes a pair of events or states in a plausible order (Coordination Inversion). Despite variation across tasks and models, asymptotic accuracy is reached during the ID peak phase. Again, this implies that, for OPT, the asymptote reached later. For OLMo (Appendix I), observe in some cases late accuracy peak, related to the second ID expansion phase that this model undergoes. In general, however, once top accuracy reached, performance stays quite constant across layers, suggesting that the expressivity of high dimensionality permits rich linguistic representation of the inputs, but, once this information extracted, it is propagated across subsequent layers. This is intuitive, as high-level linguistic information is useful to the network for its ultimate task of next-token prediction. Appendix D confirms the same patterns for a smaller and a larger Pythia model.
observe that, in general,
Better LMs have higher ID peaks, earlier Given the apparent linguistic importance of the high-ID phase, a natural question concerns the extent to which the nature of the ID peak relates to the LM's performance on its original task of next-token prediction. The question was already partially answered by Figure 1, which shows absence of peaks, respectively, when trained LMs process shuffled text and when untrained LMs process normal text; in both cases, next-token prediction is impossible.
We further computed Spearman correlations between average prediction surprisal for each (model, corpus) combination and the corresponding maximum ID values (Figure 6, left), as well as ID-peak onsets (Figure 6, right).4
As the plots show, even when limiting the analysis to sane text processed by fully trained models, where the differencee in surprisal will be smaller, there is marginal tendency for maximum ID value to inversely correlate with surprisal: the higher the peak, the better the LM is at predicting the next
'Due to its range, it's easier to visualize surprisal, instead of the more commonly used perplexity measure, in the plots. As perplexity is exponentiated surprisal, and we are using Spearman correlation (which is robust to monotonic transformations such exponentiation), our results are not affected by this choice.
8

---

Published as a conference paper at ICLR 2025
3.75
surprisal (bits)
3.50
X
Ilama mistral
model
3.25
alme
3.00
pythis
2.75
0:46 (0.09)
p 04-0065-0..00 (0.01)
pile
bookcorpus
2:50
30
50 max ID across layers
0.3 0.4
0.5
0.6
relative peak onset
Figure 6: Surprisal plotted against the maximum ID across layers (left) and the relative ID peak onset over layers (right), where each datapoint a (model, corpus) combination (N 50k sequences per corpus). A linear fit (left) and log-linear fit (right) are shown. (Lefft): Surprisal negatively correlates to maximum with Spearman p -0.46, p = 0.09, meaning that higher indicates better L.M performance. (Right): Surprisal positively correlates to ID peak onset, p 0.65, p 0.01, meaning that an earlier ID peak indicates better LM performance.
llama
pythia
0.91
0.51
0.91
8.90
0.85
0..66
0.82
1D
0.79
0.86
10
-.88
tarddt
9.88
085
ID Peak
I3 Peak
1D Peak
0.79
teatrity
0.73
0.84
0.77
0.87
0.82
sentiment
sentiment
sentiment
0.68
0.72
0.79
10
20
30
10
20
30
10
20
30
layer
layer
layer
Figure 7: Validation performance of representation transfer performance on toxicity and sentiment classification for Llama, OPT, and Pythia (left right). All validation accuracy curves plotted with + 2 standard deviations over 5 random seeds. For all models, classification validation accuracy converges within the ID peak.
token. There is, moreover, significant positive correlation between surprisal and the onset of the ID peak: the earlier the peak, the better the model is at predicting the next token.
We just saw that the ID peak might mark the phase where the model first completes a full syntactic and semantic analysis of the input. The carlier this analysis takes place, the more layers the model will have to further refine its prediction by relying on it. The correlation between ID peak onset and surprisal thus indirectly confirms the importance of the high-dimensional processing phase for good model performance.
ID-peak layers are the first to transfer to downstream tasks seems mark the onset of more abstract linguistic processing of the input, representations this peak should also viably transfer to downstream linguistic tasks. We confirm this hypothesis for sentiment and toxicity classification.
Finally, given that the ID peak
Validation accuracy curves for both tasks are shown for Llama, Pythia and OPT in Figure 7, with more results Appendix Similar to the semantic and syntactic probing results, downstream classification performance consistently converges at the II) peak across models and remains high thereafter. Note that, while the ID peak is computed from three generic input corpora (Bookcorpus, Pile and Wikitext), it still predicts transferability to different downstream datasets that reasonably in-distribution.
CONCLUSION
evident that a LM needs to extract information from its input to predict the next token. The non-trivial fact we show here that this process does gradually refine representations, but rather

---

Published as a conference paper at ICLR 2025
undergoes phase transition characterized by ID expansion, cross-model information sharing, and a switch to abstract information processing. Our main take-home message is that different LMs consistently develop central-layer phase where the intrinsic dimension is expanded, which is the locus of deeper linguistic processing.
Our paper focused detecting broad qualitative patterns. We found very consistent converging evidence from a variety of models, corpora and experiments (even OLMo, that is to some extent the "outlier" model, displays the ID peak and the relevant associated properties). Future work should establish a clearer causal connection between intrinsic dimension and language processing, via layer ablations studies and by constraining the dimensionality of different layers during language processing.
Mirroring Jastrzebski et (2018)'s proposition for visual networks, our findings are compatible with a view of LMs in which the nigh-dimensional early-to-mid layers functionally specialize to analyze inputs in a relatively fixed manner, whereas later layers may more flexibly refine the output prediction, using information extracted during the high-ID phase. Interestingly, two recent papers (Gromov et al., 2024; Men et al., 2024) found that (1) late LM layers better approximate each other than earlier layers do, and (2) pruning late layers (excluding the last) affects performance less than pruning earlier layers. This fully aligns with our results, and suggests a need to test the effect of pruning inside and outside the ID peak. Other studies have highlighted the importance of central layers in performing various core functions. For example, Hendel et al. (2023) find that compositional "task vectors" are formed in layers superficially consistent with the peaks detect. Again, future work should more thoroughly study the relation between the ID profile and specific circuits detected in mechanistic interpretability work.
While our work closely relates to that of Valeriani et al. (2023), who studied the evolution of ID in vision and protein transformers, an interesting contrast is that they found crucial semantic information to coalesce during a dimensionality reduction phase, whereas we associated similar marks to dimensionality expansion phase. This difference observations may be partly attributed to our differing methodologies, since Valeriani and colleagues analyzed the IDD of average sequence token and probed semantic information using classifier based on nearest neighbors. In line with our results, recent work in theoretical neuroscience shows that high ID, thanks to its expressivity, underlies successful few-shot learning (Sorscher et al., 2022) and generalizable latent representations for DNNs (Elmoznino & Bonner, 2024; Wakhloo et al., 2024). Conversely, primarily in artificial and biological vision, low ID is linked to generalization thanks to representations' robustness to noise and greater linear separability in embedding space (Amsaleg al., 2017; Chung al., 2018; Cohen et al., 2020). Clearly, whether dimensionality is a curse or blessing to performance depends on the context of the learning problem. Reconciling, then, why and when high performance arises from reduced or expanded dimensionality remains an important direction for future work.
From a more applied perspective, we see various ways in which our discovery of consistent high-ID phase could be useful. ID profiles emerge as significant blueprints of model behaviour that could be used as proxies of model quality. ID information can be used for model pruning, or to choose which layers to fine-tune, or for model stitching and other model-interffcing operations, such as training I.M-based encoding models of the brain (Antonello Cheng, 2024). These are all exciting directions for future work.
REPRODUCIBILITY STATEMENT
Please find corpora, code and a readme document to reproduce our experiments at httpss//gittubs com/ hoom/chhnemmlyy/clllm---ae /id-lll-abstrattionn We report our compute usage in Appendix A, and describe all publicly available resources we used in Appendix B. Appendix C, Appendix and Appendix specify relevant hyperparameter choices for ID computation, probing task and transfer task implementation, respectively.
ACKNOWLEDDGMENTS
We thank the members of the COLT group, Mor Geva, the ICLR reviewers, and the area chair for useful feedback. This project has received funding from the European Research Council (ERC)
10

---

Published as a conference paper at ICLR 2025
under the European Union's Horizon 2020 research and innovation program (grant agreement No. 101019291). This paper reflects the authors' view only, and the funding agency is not responsible for any use that may be made of the information it contains. Additionally, D.D. received support from the project "Supporto alla diagnosi di malattie rare tramite l'intelligenza ACtificiale"(CUP. PS3C22001770002).
REFERENCES
CJ Adams, Jeffrey Sorensen, Julia Elliott, Lucas Dixon, Mark McDonald, Nithum, and Will Cukierski. Toxic comment classification challenge, 2017. URL https: ptttpps:///gggiae /ttp//kaglleccc com/ competitions www.tiiiiniiissaatt.....ooo wwwe--ooooi--ccmm---l-llaa-aaea- wwcatiinncaaalleeeee
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11hh International Joint Conference on Natural Language Processing ( (Volume I: Long Pupers), pp. 7319-7328, Online, August 2021. Association for Computational Linguisties.
Joshua Ainslie, James Lee- Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. pp. 4895-4901, 2023.
Laurent Amsaleg, James Bailey, Dominique Barbe, Sarah Erfani, Michael E. Houle, Vinh Nguyen, and Miloš Radovanović. The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality. In 2017 IEEE Workshop on Information Forensics and Security (WIFS), pp. 1-6, December 2017.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. In Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Richard Antonello and Emily Cheng. Evidence from fMRI supports a two-phase abstraction process in language models. In UniReps: 2nd Edition of the Workshop on Unifying Representations in Neural Models, 2024. URL https:t///// / herpeerreiieeenetttoorr........
Randall Balestriero, Romain Cosentino, and Sarath Shekkizhar. Characterizing large language model geometry solves toxicity detection and generation. 0164 648. 2023.
2212.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O O'Bren,, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. (1TXIv:2004.0133),, Apr 2023. URL http: //tp///aaivvva/aa///a//ee.ea.wva .rg...oo...............o...oo... org/o///// abs/ 2304. 013
Christopher M Bishop. Neural networks for pattern recognition. Oxford university press, 1995.
Gemma Boleda. Distributional semantics and linguistic theory. Annual Review of Linguistics, 6(Volume 6, 2020):213-234 January 2020. ISSN 2333-9683. 2333-9691. doi: 10.1146/
annurev-linguistics-000 1619-030303.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church. Isotropy in the contextual embedding space: Clusters and manifolds. In International Conference on Learning Representations, 2021.
P. Campadelli, E. Casiraghi, C. Ceruti, and A. Rozza. Intrinsic dimension estimation: Relevant techniques and benchmark framework. Mathematical Problems in Engineering, 2015:e759567, Oct 2015. ISSN 1024-123X.
Emily Cheng, Corentin Kervadec, and Marco Baroni. Bridging information-thhoreee and geometric compression in language models. In Proceedings of EMNLP, pp. 12397-12444 12420, Singapore, 2023.
11

---

Published as a conference paper at ICLR 2025
SueYeon Chung, Daniel D. Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Phys. Rev. X, 8:031003, Jul 2018.
Uri Cohen, SueYeon Chung, Daniel D. Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature Communications, 11(1):746, Feb 2020. ISSN 2041-1723. doi: 10. 0000://44167000-1555775 articles/ /41///4141770-0 1-220-145885.
URL https:/ peattps::/ww// nature. com/
Arthur Conmy, Augustine Mavor-Parker Aengus Lynch, Stefan Heimersheim, and Adrià Garriga- Alonso. Towards automated circuit discovery for mechanistic interpretability. In Proceedings of NeurIPS, volume 36, pp. (6318-163522 New Orleans, LA, 2023.
Alexis Conneau, Germán Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into single $& !#* vector: Probing sentence embeddings for linguistic properties. In Proceedings ACL, pp. 2126-2136, Melbourne, Australia, 2018.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344-16359, 2022.
Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression. In The Twelfth International Conference on Learning Representations, 2024.
Francesco Denti, Diego Doimo, Alessandro Laio, and Antonietta Mira. The generalized ratios intrinsic dimension estimator. Scientific Reports, 12:11),00005, Nov 2022. ISSN 2045-2322. doi: 00.1038/s41598-022 028--00222-2911111
Diego Doimo, Aldo Glielmo, Alessio Ansuini, and Alessandro Laio. Hierarchical nucleation in deep neural networks. Advances in Neural Information Processing Systems, 33:7526-7555 -536, 2020.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hodse Danny Hemnandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http://ttaassommeecrcrciiii.pppp ftrro//00///aameeew///eeeee. htmll
Eric Elmoznino and Michael F. Bonner. high-performing neural network models of visual cortex benefit from high latent dimensionality. PLOS Computational Biology, 20(1):e101179) January 2024. ISSN 1553-7358.
Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimen- sion of datasets by a minimal neighborhood information. Scientific Reports, 7(1):12140, Sep 2017.
ISSN 2045-2322. doi: 10. 11888444598-017-1187777
Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta Costa-jussá. A primer on the inner workings of transformer-be language models. https: htapp://aaxxvvorrt org/ abs/2405. 240/ 00208, 2024.
Keinosuke Fukunaga. Introduction statistical pattern recognition. Elsevier, 2013.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling. http:// http:///axxvvveee org/ 2020.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in rugo-segressive language models. In Proceedings of EMNLP, pp. 12216-12255 Singapore, 2023.
Aldo Glielmo, Claudio Zeni, Bingqing Cheng, Gábor Csányi, and Alessandro Laio. Ranking the information content of distance measures. PNAS nexus, 1(2)spgac039, 2022.
12

---

Published as a conference paper at ICLR 2025
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, Cambridge, MA, 2016.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merril,, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Sol- daini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models, 2024.
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel Roberts. The unreasonable ineffectiveness of the deeper layers. 2024.
Roee Hendel, Mor Geva, and Amir Globerson. In-context learning creates task vectors. In Findings of EMNLP, pp. 9318-9333, Singapore, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 160iv:1606.000 084.66066088.... 2016.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Proceedings EMNLP, pp. 2733-2743, Hong Kong, China, 2019.
Edward Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proceedings of
ICLR, Online, 2022. Published online: https : opwn.evieweeeeeeeeeeeeeee
cc/2022/ Ctc/2022/Coffrrence Coo///enneeecccc..
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In Proceedings of ICIR Conference Track, Vancouver, Canada, 2018. Published online: https::/ /wwpr//eeeeeeeeeeee/mme net group?id
ICLR.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
Ian Jolliffe. Principal Component Analysis. Springer, 1986.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In Proceedings of ICML, pp. 3519-3529, Long Beach, CA, 2019.
Nikolaus Kriegeskorte, Marieke Mur, and Peter Bandettini. Representational similarity analysis: Connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2(4):1-28, 2008.
Chunyuan Li. Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan. Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher Manning, Christopher Ré, Diana Acosta-Navas, Drew Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao. Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab. Peter Henderson, Qian Huang, Ryan Chi, Sang Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 8:1-162, 2023.
13

---

Published as a conference paper at ICLR 2025
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: //www. www.aaclwebe wwt.:oooooooooooooogoooooooooo/e httphh100...p1111111.1..
Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShorGPT: Layers in large language models are more redundant than you expect. attpss://axxtn org/ abs 2403. 03853. 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. In Proceedings of NewrlPS, volume 35, pp. 17359-17377 New Orleans, LA, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference Learning Representations, 2017.
Meta. Introducing meta Ilama 3: The most capable openly available I1m to date, 2024. URL https::/aat met a. cww/blog mwt///ettaaaa 1almaa-la-----mm 3/.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: an imperative style, high--performncce deep learning library. Curran Associates Inc., Red Hook, NY, USA, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay, Scikit-leam: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimen- sion of images and its impact on learning. In International Conference on Learning Representations, 2021.
Michael Psenka, Druv Pai, Vishal Raman, Shankar Sastry, and Yi Ma. Representation learning via manifold flattening and reconstruction. Journal of Machine Learning Research, 25(132):1-47, 2024.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXlw:2002.05202, 2020.
Ravid Shwartz-Zii and Naftali Tishby. Opening the black box deep neural networks via information, Mar 2017. URL https: // arxiv org/ abs / 1 703. 00810v3.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannanch Hajishirzi. Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. (50Xiv:2402.00155), January 2024. doi: 2085500arxiv.224220001111 URL http: //arxiv. wwtp:ivv..r..bb/baeaaooooo 2402 00159. arXiv:2402.00159 [cs].
Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry under- lies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(43):
c2200800119, October 2022. doi: 10. (0.11773/paas2220000011
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li. Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. On transferability of prompt tuning for natural language processing. In Proceedings of NAACL, pp. 3949-3969, Seattle, WA, 2022.
14

---

Published as a conference paper at ICLR 2025
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng. Andreea Bobu, Been Kim, Bradley Love, Christopher Cueva, Erin Grant, Iris Groen, Jascha Achterberg, Joshua Tenenbaum, Katherine Collins, Katherine Hermann, Kerem Oktar, Klaus Greff, Martin Hebart, Nathan Cloos, Nikolaus Kriegeskorte, Nori Jacoby, Qiuyi Zhang, Raja Marjieh, Robert Geirhos, Sherol Chen, Simon Kornblith, Sunayana Rane, Talia Konkle, Thomas O'Connell, Thomas Unterthiner, Andrew Lampinen, Klaus-Robert Müller, Mariya Toneva, and Thomas Griffiths. Getting aligned on representational alignment. https: /ttp//aaiii....e//.e//eeeeeaaava org/ abs 2310. 13018, 2024.
Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 2905500)990055000331222222 2000.
Eduard Tulchinskii, Kristian Kuznetsov, Kushnareva Laida, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. Intrinsic dimension estimation for robust detection of Al-generated texts. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: hwtp//pwnnreeeeeeeeeeeeee
htttts::/opennevvee..o net./ Eorum? wiaimmiidid-ioi Bu
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. In Proceedings of NeurIPS, pp. 5 51234-51252, New Orleans, LA, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NIPS, pp. 5998 6008, Long Beach, CA, 2017.
Albert J. Wakhloo, Will Slatton, and Yue Chung. Neural population geometry and optimal coding of tasks with shared latent structure. (arXiv:2402.16770) February 2024. doi: 10.48550/ar 2402.16770. URL http::/aaa.....g//////a//o q-bio].
16770. arX1v:2402.16770 [cond-mat,
Alex Williams. Equivalence between representational similarity analysis, centered kernel alignment, and canonical correlations analysis. In Proceedings of the UniReps Workshop, Vancouver, Canada, 2024. Published online: https: htttps://oeerrevieet Workshop/ Cew.Reeeetaaaaaeeeeeee http:///pnneeeeeeeeeeeee http://wwww..tetttoo..t.eoae wtw.ppppdddeuurPPs.c/oooo/
Fan Yin, Jayanth Srinivasa, and Kai-Wei Chang. Characterizing truthfulness in large language model generations with local intrinsic dimension. In Proceedings of ICML., pp. pp. 57069-57084, Vienna, Austria, 2024.
2024.
Yaodong Yu. Sam Buchanan, Druv Pai, Tianzhe Chu, Ziyang Wu, Shengbang Tong, Benjamin Haeffele, and Yi Ma. White-box transformers via sparse rate reduction. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 9422-9457. Curran Associates, Inc., 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: Open pre-trained transformer language models. https: /ttp///axvv.v//////// httgg/bb/220000/ 01068, 2022.
Zhong Zhang, Bang Liu, and Junming Shao. Fine-tuning happens in tiny subspaces: Exploring intrinsic spsk-spe subspaces of pre-trained language models. Proceedings of ACL, pp. 1701-1713, Toronto, Canada, 2023.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of ICCV, pp. 19-27, Santiago, Chile, 2015.
A COMPUTING RESOURCES
All experiments
cluster with 12 nodes with 5 NVIDIA A30 GPUs and 48 CPUs each.
Extracting LM representations took a few wall-clock hours per model- dataset computation. ID computation took approximately 0.5 hours per model-dataset computation. Information imbalance
15

---

Published as a conference paper at ICLR 2025
computation took about hours per model-dataset computation. Probing/transfer classifiers took up to days per task.
Taking parallclization into account, estimate the overall wall-clock time taken by all experiments, including failed runs, preliminary experiments, etc., to be about 20 days.
B ASSETS
Bookcorpus
http:///wwgginfeeeeoooo/atttt/op license: unknown
httpss//wwggniigecccc/eaa/eeee/s license: bigscience-bloom rail-- Pile-10k
1.0
Wikitext http:///wuggingioooo/odassttttts license: Creative Commons Attribution
Share Alike 3.0
Llama http:///wweeeeneeee//e/////////a
license: Ilama3
Mistral http://uuggiiifeeecc/niiiiaailir license: apache-2.0
OLMo https//hhuugiiiiaaecco///llllllp license: apache-2.0
licee:
OPT http://wugggittcccc//ccceeeo/o/p license:
OPT-175B licese:
license
Pythia http:///wgggoaaaccc//eeeeeteeeec license: apache-2.0
DadaPy http://ghhhlcccommsssssaaaaaaaac license: apache-2.0
seikit-learn https://sakittkeammoorg// license: bsd
licenm
PyTorch https://scikttkeernoo//// license: bsd
Probing tasks http:////thtttttto///////t//////
cense: bsd
Toxicity dataset http:///ugeliiiiccc///aaaarrreep http:///eejjaaaaattiiitttt/o/// pred
license: CCO
Sentiment dataset http:://wueeinnnacce///tteeeeett license: unknown
C INTRINSIC DIMENSION
C.I SCALE ANALYSIS
In this section we explicitly show how perform a scale analysis in order to select proper and meaningful scale when computing the ID. As explained in the main text, ID estimation might be affected by undesirable effects that are always present when dealing with real-world datasets and that can hide the true dimensionality of the manifold underlying the data. Such effects include the presence of noise, which typically affects small scales, and density variations and manifold curvature, which, instead, tend be observed at larger scale (Denti et al., 2022). It is thus recommended to choose an intermediate scale, where the ID estimate might be stable across different neighbourhood sizes, and it is less likely to be affected by the aforementioned spurious factors. For this reason, it necessary to see how the ID changes when varying the neighbourhood size taken into account when performing the ID computation. We rely on the GRIDE estimator, which allows explicitly select the number of neighbours considered. In particular, the rank of the first nearest neighbour used compute the distance ratio is hyperparameter, and refer the chosen value for this hyperparameter as the scale in what follows.
We performed scale analysis for each model and corpus. An example is provided in Figure C.I, which shows the results for Pythia on Pile. We plot the ID estimate for increasing scales (i.c., number of neighbours, x-axis Figure C. for some representative layers. The rank of the neighbour considered is explored by means of powers of 2 in order to look at regions where the variation in ID is noticeable. The true ID likely to lie at a scale in which the ID estimate approximately plateaus (Denti et al., 2022), which is marked by the highlighted region. For simplicity, per model-corpus combination, choose one scale for all layers: in this particular example, we choose = 25. In general, the optimal neighbourhood size tends be around & 32 for all (model, corpus) combinations, allowing to reliably compare them (see Table C. for all k). Once the scale chosen for each (model, corpus) combination, we plot the scale-adjusted IDD estimates (see, e.g., Figure C.3).
16

---

Published as a conference paper at ICLR 2025
Scale Anw Ipyltii pie sanel
Intrisicc Dimension
GRIDE Neighbor Range (log)
Figure C.I: GRIDE ID) estimation neighbourhood scale analysis example for Pythia on the Pile on a single random seed, where each line is layer's ID estimate at different scales. All layers shown reach a plateau in the highlighted range.
Average over layers
llama
mistral
olmo
40
40
dimensiio
335
Intringic climension
Intrinsic dimension
-00
30
25
20-
20
CLDC Po8
DIIDE HAR
10
10
10
15
20
25
30
20
25
30
10
15
20
25
layer
pythia
layer
opt
layer
45
Intrnsic dimension
40
20
155
95
30
2.5
Intiiniic
20
tall 1ukial
8
GHOC
1.5
15
10
10
100
15
20
5
10
15
25
30
layer
Figure C.2: Robustness of the scale analysis within the plateau region across models. The ID estimates are only weakly affected by doubling (green) or halving (orange) the scale with respect to the results reported in the main text (blue).
As further robustness test, we show in Figure C.2 that results are fairly stable to choosing scales within the shaded range. For each model, include panel with the original chosen scale (in blue), e.g., k = 2 33tth nearest neighbor, and then look at the shape of the ID if we move the scale one (logarithmic) step down 2 =16 (orange) or up k 26 -64 (green). Fortunately, we can that the shapes of the resulting ID curves are stable across all models, and the ID values themselves change very little.
C.2 ADDITIONAL RESULTS
Figure C.3 displays the evolution of estimated ID, scale-adjusted, per layer for all corpora and models. While the magnitude of ID differs across corpora, with ID on Bookcorpus lower than those of the Pile and Wikitext for all models, all corpora exhibit the characteristic high-ID peak, and at nearly the same onset. The dampened Bookcorpus peak IDs, which shuffled ID peaks, might explained by the fact that this corpus entirely made of novels (and lower-cased), and it is thus the less in-domain of the corpora explored.
still significantly above the corresponding
17

---

Published as a conference paper at ICLR 2025
model corpus mode
GRIDE
bookcorpus
sane
32
shuffled
128
Ilama pile shuffled
sane
64
128
sane
64
wikitext shuffled
16
bookcorpus
sane
64
shuffled
128
mistral pile sane
128
shuffled
256
corpus
Pythia step
GRIDE
512
32
64
wikitext shuffled
4000
32
32
bookcorpus
16000
32
sane
32
64000
32
bookcorpus shuffled
8
512
olmo pile shuffled
sane
32
4000
32
pile
128
16000
32
64000
64
wikitext shuffled
sane
32
16
512
4000
32
16
wikitext
bookcorpus shuffled
16000
16
64000
32
opt pile sane
32
shuffled
16
(b) GRIDE for additional Pythia checkpoints.
wikitext shuffled
sane
16
16
bookcorpus shuffled
sane
32
32
corpus
Pythia size
GRIDE
pythia
pile
sane
32
2.8b
32
shuffled
64
wikitext
12b
16
wikitext
sane
32
shuffled
16
(c) GRIDE additional Pythia sizes, on Wiki- text dataset.
(a) GRIDE scale & reported for each model, cor- pus, mode sane and shuffled) combination. For simplicity, we chose one for all layers.
Table C.I: Gride IDs for each model, dataset, and mode
18

---

Published as a conference paper at ICLR 2025
bookcorpus
pile
wikitext
60
llama alma
intrinsic dimension
50
opt
40
mistral
pythia
30
20
10
0
10
20
30
10
20
30
10
20
30
layer
layer
layer
Figure C.3: ID evolution over layers, shown with one standard deviation (over corpus partitions), for all models and corpora (left to right: Bookcorpus, the Pile, and Wikitext).
Aveerege ID over Layers (pythia)
pythia-2.08
pythis-12b
&
foRUTOOR UOU egoud
Intrinsic dimonsion 06
15
Intriniic dimansioon
12
Intrimsic dimensioon
%
bi
-2
-0.6
14
0.
2,50
Rig't cat
11
124
mas
15
20
10
15
25
layerr
layer
layer
(a)
(b)
Figure D.1: ID profiles (a) and probing-tase task experiments (b, c) reproduced with smaller and larger model from the Pythia family.
D INFLUENCE OF MODEL SIZE ON ID AND PROBING TASKS
We reproduce key experiments from the paper with a smaller (2.8b) and a larger (12b) version of the Pythia's 's architecture, suggesting that our findings are not limited to the scale of the models analyzed in the paper.
ID profiles We have plotted the ID) profiles of smaller (2.8b) and larger (12b) Pythia models in Figure D. D.). Analogously to the evolution of ID during training (cf. Figure right), observe a similar profile across model sizes, with the ID magnitude of the peak growing with model size. Note that the ambient dimensionalities of the 2.8b, 6.9b, and 12b models are respectively 2560, 4096, and 5120. While the relative change in peak magnitude is much more dramatic between 2.8b and 6.9b than between 6.9b and 12b, the change nevertheless not proportional the relative change in hidden dimension, echoing the observation by Cheng et al. (2023) that ID saturates as ambient dimensionality increases.
Probing tasks We also reproduce the probing-task experiments with the same smaller (2.8b) and larger (12b) models from the Pythia family, showing that the semantic/syntactie probing-task asymptote reached within the span of the peak also at these newly tested sizes (see Figure D. (b.c)).
INFORMATION IMBALANCE WITH RESPECT TO FIRST/LAST LAYER
Figure E. shows averaged (1, - lfieet) (gray) and AL - Hast) (brown) for Mistral and OLMo. Recall that the closer A(A - B) is to O, the more predictive A's local neighborhood structure is of B's. As expected, A(4 > lfrrst) generally increases with as we go deeper in the layers; the reverse is true for (l; lastt). However, A(l - lfirst) appears locally peak and plateau
19

---

Published as a conference paper at ICLR 2025
mistral
olmo
38
ID
Mlayeri Alayer aat
38
ID
Milayeri firstt
ID Peak
Peak
Allayeri last
intrinsic dimensios
33
intrinsic dimension
32
27
0.6
27
0.6
22
21
-0.4
16
15
11
10
10
15
20
25
30
10
15
20
25
30
layer
layer
Figure E.1: For Mistral (left) and OLMo (right), the ID (hued) is overlaid with A(4 - lirst) (gray) and A(4 -> Mast) (brown). Plots are shown with + 2 standard deviations corpora and partitions.
around the representational ID peak: that the ID expansion marks phase of low predictivity from the intermediate layer to the input. OLMo's pattern is not clear, and we might observe second local information imbalance peak proximity the second ID) peak characterizing this model.
Figure E.2 shows, in the spirit of the Information Plane (Shwartz-Ziv -iv & Tishby, 2017), the A(4 - last)) plotted against (l + (2) for all models across the layers. The trajectory reveals dynamic evolution of input and output informativity along the residual stream. Consistently across models, ID peak layers (red) mark change point in processing patterns. Pre-peak layers show rapid increase along the x-axis but slow decrease along the y-axis, indicating rapid departure from the inputs while not learning much about the outputs. This reflects the phase of semantic abstraction. Post-peak layers' information dynamics vary by model, though all are marked by final rapid decrease (1; + hast) and a decrease (lfrrst + 4). This suggests a return to surface-level information processing in order predict the next token.
F FORWARD A SCOPE
Figure F.l reports the forward A scope profile for the remaining two LMs not shown in the main text (Mistral and OLMo).
G CROSS-MODEL A
Figure G.1 show cross-model A for the remaining combinations, confirming that there are areas of low cross-model information imbalance at the intersection of the high-ID) peaks. In combinations involving OLMo, we observe tendency for low A to stretch along the other LM high-ID section, suggesting that the high-ID layers of other LMs share information with wider range of OLMo layers.
H CROSS-MODEL LAYER COMPARISON USING CKA
We reproduce the cross-model layer comparison experiments of Figure using linear CKA (Kornblith et al., 2019) as an alternative measure of similarity. Figure H.1 broadly confirms the results found with
I PROBING TASKS
I.1 TASKS
We the following classification tasks from Conneau al. (2018):
20

---

Published as a conference paper at ICLR 2025
llama
opt
pythia
0.6
0.6
0.6
A1): + llstt
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
A
1))
olmo
mistral
0.6
0.6
Peak layer
A(} + >last)
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.0
0.2
0.4
AlIirst l;)
Al + 1,)
Figure E.2: A(4 - lastt vs. Allfist - 4.). For each LM, the information imbalance A from the ith to the last layer is plotted against A from the first to the ith layer, where lower A means more informative. Each point on the curve corresponds to a single layer i; for all models, the first layer (gold star) begins on the y-axis, A(lfiret 4)) O, and the trajectory through the layers ends on the x-axis, (4; + hastt = O. Consistently across models, ID peak layers (red) mark a changepoint in processing patterns. Pre-peak layers show rapid increase along the x-axis but slow decrease along the y-axis, indicating rapid departure from the inputs while not learning much about the outputs. This reflects the phase of semantic abstraction. Post-peak layers' information dynamics vary by model, though all are marked by final rapid decrease in A(I, lustt) and a decrease in A farst + 4). This suggests a return to surface- -evel information processing in order to predict the next token.
21

---

Published as a conference paper at ICLR 2025
mistral
olmo
Intrinsic dimension
Intrinsic dimension
25
adoos V (JeM.oo
25
V (JeM.19
27
27
mas Prward Hcpe
2002
ID
ID Peac
ID Peac
155
10
15
20
25
30
10
15 layer
25
30
layer
Figure F.1: Forward A scope (Mistral, Olmo): continuous lines report, for each layer In, the number of adjacent following layers la+ for which A(, + In+*) Σ 0.1. The dashed line represents the longest possible scope for each layer. Values are averaged across corpora and partitions, with error bars of 2 standard deviations.
Surface form
Sentence Length Predict input sentence length tokens (lengths binned into intervals). Word Content Tell which of pre-determined set of 1K words occurs in the input sentence.
Syntax
Bigram Shift Tell whether the input sentence is well-formed, or it has been corrupted by inverting the order of two adjacent tokens (e.g.. "They were in present droves, going from table table and offering to buy meals, drinks or generally allempting to strike up conversations",
Semantics
Coordination Inversion Tell whether sentence is well-formed or it contains coordi- nated clauses whose order has been inverted (e.g.. "Then decided to treat her just as I would anyone else, but at first she' 'd frightened me").
Odd Man Out Tell whether a sentence is well-formed, or involves the replacement of noun or verb with a random word with the same part of speech (e.g., "The people needed a sense of chalk and tranquility").
We exclude the following tasks because we observed ceiling effects across the layers, suggesting that the models could latch onto spurious correlations in the data: Past Present, Subject Number and Object Number. We exclude Top Constituents and Tree Depth because they produced hard-to- interpret results that we believe are due to the fact that they rely automated syntactic parses that are not necessarily consistent with the way modern L.Ms process their inputs.
1.2 SETUP
We use the training and test data provided by Conneau et al. (2018). We train a MLP classifier for each task and each layer of each LM, repeating the experiment with 5 different seeds.
We fixed the following hyperparameters of the MLP, attempting to approximate those used in the original paper (as each task takes days to complete, we could not perform our own hyperparameter search):
Number of layers:
Layer dimensionality: 200
Non- -on-linearity: logistic
L2 regularization coefficient: 0.0001
seeds: 1, 3, 4,
For all other hyperparameters, we used the default values set by the the Scikit-learn library (Pedregosa al., 20.
22

---

Published as a conference paper at ICLR 2025
31
31
28
23
10
4
10
19
22
28
51
10
13
16
19
liama layer
liama layer
mistral layer
10
CS. 0011
5:011 a0p
are cl
Jbon c3.100
sr 1000
19
22
bayer
Figure G.I: Cross-model ID-peak sections are shaded in orange. Different symbols mark different information imbalance levels in the two directions (the lower the A(A + B) values, the more the information in B is contained in A). High imbalances (> 0.1) are not shown. Values averaged across corpora and partitions.
00
04
00
64
04
(a) Pythia vs. Llama
(b) OPT Llama
(c) Pythia vs. OPT
Figure H.I: Cross-model layer similarity measured using linear CKA. Like in Figure 4, ID-peak intersections tend to coincide with high-similarity areas.
23

---

Published as a conference paper at ICLR 2025
olmo
mistral
1.0
1.0
10 Paak
0.9
Drobe
ID Paak
ID
0.9
intrinsic dimension
intrinsic dimension
32
33
0.7
0.7
27-
0.6
27
0.6
0.4
0.4
15
0.3
16
0.3
lungth
SenGrn lungoh
word
10
0.1
11
25
0.1
layer
layer
mistral
1.0
1.0
crcunnee wogephl aqoid
probe validation accuracy
intrinsic dimension
0.9
intrinsic dimension
0.9
0.8
0.8
27
27
0.7
0.7
21
22
Peak
0.6
Paak
0.6
courdimal on inversion
Digromt Shift
0.5
coondination inversion
Bigram Shift
0.5
odd man ouT
odd man out
0.4
10
15
20
25
10
1.5
20
30
layer
layer
Figure 1.1: Linguistic knowledge probing performance +2 SDs across 5 random seeds is plotted with the ID across layers for OLMo and Mistral (left right). (Top row) Surface form tasks Sentence Length and Word Content, where probe performance decreases through the ID peak. (Bottom row) Semantic and syntactic tasks Bigram Shift, Coordination Inversion and Odd Man Out, where probe performance for all tasks attains maximum within the ID peak.
We repeated all experiments after shuffling the example labels (both training and test time). This provides a baseline for each task, as well as functioning as a sanity check that a probing classifier not so powerful 2019). Note that, since, as shown in Figure I.2, performance is essentially constant (and at chance) on shuffled labels, accuracy our case equivalent to selectivity, the measure recommended by Hewitt & Liang (2019). We thus focus on accuracy, as the more familiar measure.
simply memorize arbitrary patterns in the representations (Hewitt & Liang,
1.3 ADDITIONAL RESULTS
Figure I.1 reports the probing performance for OLMo and Mistral, and for surface form (top row) as well as syntactic and semantic tasks (bottom row). As for Llama, OPT, and Pythia, observe that the first peak converges to viable syntactic/semantic abstraction of inputs, while discarding information about surface form.
Figure 1.2 shows that, on the shuffled corpora ablation, the MLP probes perform at chance. We then confirm that semantic and syntactic information is contained in the model representations and not in the probes.
J DOWNSTREAM TASKS
J.1 TASKS
Toxicity detection. We use random balanced subset (N 30588) of Kaggle's jigsaw toxic comment classification challenge (Adams et al., 2017), where each data point consists of a natural language comment and its binary toxicity label.
Sentiment classification. We use a dataset of IMDb movie reviews (Maas et al., 2011), where each data point consists of natural language film review and corresponding label E {positive, negative).
24

---

Published as a conference paper at ICLR 2025
llama
opt
pythia
probe validation accurac)
1.0
1.0
1.0
0.8
0.8
0.2
0.2
0.2
0.0
0.0
0.0
10
20
30
10
20
30
0
10
20
30
layer
layer
layer
olmo
mistral
probe validation accuracy
bigram shift
coordination inversion add man out sentence length
0.4
0.4
0.2
0.2
10
20
30
10
20
30
layer
layer
Figure I.2: Linguistic probe validation accuracy for semantic, syntactic, and surface tasks (solid lines) and their shuffled versions (dashed lines) are shown across layers for all models. The probing performance on shuffled corpora constant at chance for all tasks and models.
To train the linear probes, take sample of size N 25000 corresponding the train split HuggingFace. We repeat the experiment with distinct seeds.
J.2 SETUP
To train the linear probes, we first divide the data random into train (80%) and validation (20%) sets. We feed each training set through each model and gather the last token hidden representations at each layer. Then, using PyTorch (Paszke et al., 2019), we train one linear probe per layer with hyperparameters as follows,
Number of epochs: 1000
Ir: 0.0001
seeds: 32, 36, 42, 46, 52
and we report the best validation accuracy over 5 random seeds.
J.3 ADDITIONAL RESULTS
Similar to Llama, OPT, and Pythia, Figure J.1 shows that validation performance for downstream tasks for Mistral and OLMo converge in the ID peak.
25

---

Published as a conference paper at ICLR 2025
mistral
olmo
0.91
0.91
toxicity val accuracy
0.89
0.89
0.76
0.81
0.78
Peak
0.73
toxicity
0.70
0.78
toxicity
0.72
sentiment
sentiment
0.68
0.64
0.74
0.66
layer
layer
Figure J.1: Validation performance downstream transfer tasks toxicity and sentiment classifica- tion for Mistral (left) and OLMo (right). All validation accuracy curves are plotted with = 2 standard deviations over random seeds. For all models, classification validation accuracy converges within the ID peak.
26


Reviews:
[{"review_id": "ttYcoW2vwp", "reviewer": "Reviewer_1CPH", "summary": "The paper uses the technique of intrinsic dimension estimation as a tool for analyzing properties of different transformer LLM layers. 5 different LLMs are analyzed on textual inputs from 3 different public-domain corpora. In addition to computing the intrinsic dimensionality (ID) (using the generalized ratios intrinsic dimension estimator) for different layers, the ID is  correlated with performance of different layers' representations on syntactic and semantic probing tasks. Furthermore,  the difference in representational power between different layers is measured using an Information Imbalance criterion. The authors find that middle layers in LLMs have the highest ID; ID peaks seem to be an indicator of linguistic structure being learnt; early onset of peaks in ID across layers is correlated with better next token prediction performance performance; and high ID peak layers are representationally equivalent across different LLMs.", "questions": "While the analyses show some interesting trends, it is difficult to tell how meaningful or significant the numerical differences are. Methods for analyzing LLM layers other than through ID could have been discussed in a prior work section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper conducts a broad analysis across 5 different LLMs and considers a range of questions and ablation studies (e.g., estimating ID on shuffled data, comparing layers across different models); altogether an impressively broad set of experiments. The paper is clearly written and presents a few new insights (e.g., correlation between peak onset and performance). Code and data would be made available, which would be valuable for the community.", "weaknesses": "The use of ID as an analysis tool for LLM layers is  not an entirely new idea (e.g., https://arxiv.org/pdf/2402.18048). \nMost of the results (e.g., the peaking of ID at middle layers, emergence of  linguistically informative representations in those layers) has been shown before by means of other methods (e.g., mutual information or canonical correlation analysis). These should have been discussed in more detail under prior work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MRAVragrNV", "reviewer": "Reviewer_j48a", "summary": "This work takes a high-level geometric approach to analyze intrinsic dimension (ID) of the representational manifold at each layer of a decoder-only Transformer LLM to understand how layer geometry relates to layer function. Although inspired by the earlier work of (Valeriani et al., 2023), this work greatly extends the models investigated to include five mainstay decoder-only LLMs, and added more extensive probing and downstream tasks on defined datasets to analyze ID profiles across layers. The resulting observations are different from those from (Valeriani et al., 2023). This work made quite a few interesting findings on detecting broad qualitative patterns, and provides useful guidance for future research towards interpretability, analysis of model behavior and quality, and model pruning and layer-specific fine-tuning etc.", "questions": "(1) Please check the questions listed under Weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)\tAlthough inspired by the earlier work of (Valeriani et al., 2023), this work greatly extends the models investigated to 5 distinct mainstay transformer-decoder-only LLMs and added more extensive probing and downstream tasks on defined datasets to analyze ID profiles across layers. Hence, the conclusions drawn in this work are verified across various models, datasets, and tasks, making the findings more convincing.\n\n(2)\tThe comparisons to related works, esp. (Valeriani et al., 2023) which inspires this work, are clearly presented, hence the contributions of this work are clear and solid.\nThe verification of the emergence of a central high-dimensionality phase, and analysis of language processing behavior and performance during the high-dimensionality phase are quite thorough.\n\n(3)\tThe analysis in Conclusion demonstrates that many findings in this work align with prior works and concurrent works. The paper clearly summarizes insights of guidance for future research. The Appendix provides detailed experimental setup and additional results. And finally, the analysis of potential applications of the findings is valuable to the research community.\n\n(4)\tOverall, the paper is clearly written and easy to follow.", "weaknesses": "(1)\tAlthough the paper is overall clearly written, please make sure that every symbol used needs to be clearly defined when it first appears, e.g., d in Section 3.4.\n\n(2)\tPlease provide rationale for critical algorithmic designs, for example, please clarify why GRIDE is selected, and why the three alternative measures for comparing layer’s representation spaces are chosen. \n\n(3)\tCurrently, k is still selected based on visual inspection. It would be useful to propose methods that can automatically select k.\n\n(4)\tIt is interesting that OLMo seems a bit of an outlier compared to the other 4 LMs, although it also exhibits the ID peak and other related properties. It would be useful to provide insights on why OLMo behaves differently from the other models, and shed light on patterns of any potential “outlier” LM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wD6SNZbXbU", "reviewer": "Reviewer_PyB7", "summary": "The paper explored how transformer-based language models evolve their internal representations across layers, revealing a distinct high-dimensional abstraction phase. The authors observe the some findings across multiple LMs and datasets, and they provide a foundation for better understanding and optimizing language model architectures. This work bridges the gap between geometric representation and linguistic function in transformer-based LMs. Also, it highlights the potential of intrinsic dimensionality as a tool for analyzing and evaluating LMs.", "questions": "1. There seems to have second ID peak in the later layers over LLMs. Do you think this second ID peak might reveal additional insights?\n2. In your analysis (Figure 4), you observed that Pythia and OPT exhibit very similar representations. Could this similarity be attributed to pre-training on similar datasets? If so, how might this influence your findings, and have you considered controlling for dataset overlap to isolate structural factors more effectively?\n3. The work focuses on classification tasks to analyze representation spaces in language models. Could you explain why generative tasks were not included? Do you expect the observed ID peaks and representation patterns to differ in generative contexts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work conducts experiments on various LMs (e.g., OPT-6.7B, Llama-3-8B, Pythia-6.9B, OLMo-7B, and Mistral-7B) using multiple datasets, providing a comprehensive analysis. It also observes how representational intrinsic dimensionality (ID) varies across layers and proposes insightful hypotheses. Furthermore, this work inspires the research community to explore the utilization of ID information in transformer-based LM applications.", "weaknesses": "While the paper combines two methods, GRIDE (Denti et al.) and Information Imbalance (Glielmo et al., 2020), to analyze four large language models (LLMs), this combination may fall short in terms of novelty. In Section 4.1, the choice of pre-training datasets for evaluation is also a limitation. Since these datasets have likely been encountered by the models during training, the results may not provide a fully accurate picture of the models’ generalization capabilities. Testing on unseen datasets would be crucial to evaluate the robustness and generalizability of the observed patterns, especially in real-world applications where unseen data is the norm. The study is limited to a narrow range of LLMs in terms of scale. Evaluating models of varying sizes (e.g., smaller models alongside large ones) would offer a more comprehensive understanding of how model size impacts intrinsic dimensionality and representation overlap across layers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MpAWrAmpRT", "reviewer": "Reviewer_61Jw", "summary": "The submission is analytic work trying to correlate intrinsic dimension of intermediate NN/LLM representations with linguistic targets. Comparison of the method applied to different models allows insights into some of their learned structural differences.", "questions": "I strongly advice to improve the submission w.r.t. the mentioned weaknesses. That helps both quality and reach.\n\nThe first paragraph of Asset Section C.1 (lines 809-836, in particular 828-829) mentions sensitivity of ID estimation w.r.t. to noise, small scales, density variations and curvature. That analysis suggests some sort of frequency decomposition integrated with the ID estimation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The findings in bold letters at lines 406-407 and lines 425-426 may be useful to some researchers that need to train and/or select models. The fact that ID seems to change gradually over layers is interesting, but may have a simple explanation in the extreme averaging scale of these models.", "weaknesses": "Except for the few strengths mentioned above, the submission does not explain for what else the gained insights can be used for or wether they are more useful than that at all.\n\nThe analysis focuses only on fully trained models and does not provide insights into how ID changes over time. A correlation analysis to other work would add more value. My first thought was a correlation to the IB method (e.g. Tishby et al. 2000 and Schwartz-Ziv & Tishby 2017), but this may not be the only or best choice.\n\nThe submission wrongly mentions PCA being linear (line 061, applies only to its original form) which leads to the quick conclusion to discard it. This is puzzling as the research on non-linear PCA is quite diverse based on very different techniques and there's even early work using neural networks dating back to 1991 (Mark Kramer: \"Nonlinear PCA Using Autoassociative NNs\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aAyKAq0FE7", "reviewer": "Reviewer_5snc", "summary": "This work analyzes properties of representations of several LLMs through a few approaches: downstream probing, intrinsic dimensionality and information imbalance. The analysis is mainly developed around intrinsic dimension and they show LLMs typically have a few intrinsic dimension peaks across layers. Additionally, they suggest that those peaks indicate transition to abstract linguistic processing through a variety of analysis", "questions": "In the data section, it is not very clear to me what does it mean to \"extract 5 non-overlapping partitions of 10k 20-token sequences\" and how the shuffled version is generated, can authors explain more about this?\n\nIn section: The ID peak marks a transition in layer function, I think the relation between ID peak and \\delta(l_i \\to l_first) is not very clear. It has very similar shape in OPT and somewhat in pythia, but LLAMA has a completely different curve. It is maximizing towards the end of the layer instead of the center of layer.\n\nIn section 4.2, authors also claim the relation between ID-peak and a few tasks. However, Figure (a) and (b) do not have very clear co-related trend between ID peaks and tasks' performance. In particular, task performance in Figure 5(b) seems to be monotonically increasing instead of peaking in the middle. Can authors justify more about this?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I like that authors combine evidence from a few different perspectives to demonstrate the relation between intrinsic dimension peak and transition to abstract processing. They also conduct experiments on a few corpus and a few models as well, which make the claim more general and robust", "weaknesses": "The method section is weak and the explanation of intrinsic dimension computing is not enough given its importance in this work. I was not able to identify which variable is corresponding to the intrinsic dimension without going through the cited paper. It seems to be the variable $d$ which authors did not explain what it is.\n\nAdditionally, author made a wrong claim in line 177~178 that \\mu has a generalised pareto distribution. I cannot find any resources claiming this specific distribution is a (generalized) pareto distribution including the original cited paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)