Emergence of a High-Dimensional Abstraction Phase in Language Transformers

## Abstract

Abstract A language model (LM) is a mapping from a linguistic context to an output token. However, much remains to be known about this mapping, including how its geometric properties relate to its function. We take a high-level geometric approach to its analysis, observing, across five pre-trained transformer-based LMs and three input datasets, a distinct phase characterized by high intrinsic dimensionality. During this phase, representations (1) correspond to the first full linguistic abstraction of the input; (2) are the first to viably transfer to downstream tasks; (3) predict each other across different LMs. Moreover, we find that an earlier onset of the phase strongly predicts better language modelling performance. In short, our results suggest that a central high-dimensionality phase underlies core linguistic processing in many common LM architectures.

## 1 Introduction

Compression is thought to underlie generalizable representation learning (Deletang et al., ; Yu et al., ). Indeed, language models compress their input data to a manifold of dimension orders-of-magnitude lower than their embedding dimension (Cai et al., ; Cheng et al., ; Valeriani et al., ). Still, the *intrinsic dimension* (ID) of input representations may fluctuate over the course of processing: we ask, what does the evolution of ID over layers reveal about representational generalizability, and, broadly, about linguistic processing?

The layers of an autoregressive language model (LM) transform
the LM’s input into information useful to predict the next token.
In this paper, we characterize the geometric shape of this transformation
across layers, uncovering a profile that generalizes across
models and inputs: (1)
there emerges a distinct phase characterized by a peak in the intrinsic dimension of representations;
(2) this peak is significantly reduced in presence of random text and nonexistent in untrained models; (3)
the layer at which it appears
correlates with LM quality; (4) the highest-dimensional representations of different networks predict each other, but, remarkably, neither the initial representation of the input nor representations in later layers; (5) the peak in dimension marks an approximate borderline between representations that perform poorly and fairly in syntactic and semantic probing tasks, as well as in transfer to downstream NLP tasks.

Taken together, our experiments suggest that all analyzed transformer architectures
develop, in the intermediate layers, a high-dimensional representation encoding complex and abstract linguistic information.
The results of this processing are stored in representations which are then used, possibly through a process of incremental refinement, to predict the next token.

## 2 Related work

The remarkable performance of modern LMs, combined with the opacity of their inner workings, has spurred a wealth of research on interpretability. At one extreme, there are studies that benchmark LMs treated as black boxes (e.g., Liang et al. ()). At the other extreme, researchers are “opening the box” to mechanistically characterize how LMs perform specific tasks (e.g.,Meng et al. (); Conmy et al. (); Geva et al. (); Ferrando et al. ()). We take the middle ground, using geometric tools to characterize the high-level activation profiles of LMs
, and we relate these profiles to their processing behaviour. In particular, we draw inspiration from the *manifold hypothesis*, or the idea that real-life high-dimensional data often lie on a low-dimensional manifold (Goodfellow et al., ): we estimate the intrinsic dimension of the representational manifold at each LM layer to gain insight into how precisely layer geometry relates to layer function.

The notion that nominally complex, high-dimensional objects can be described using few degrees of freedom is not terribly new:
it, for instance, underlies a number of popular dimensionality reduction methods such as PCA (Jolliffe, ). But, while PCA is linear, the data manifold need not be: as such, general nonlinear methods have been proposed to estimate the *topological dimension*, or manifold dimension, of point clouds (see Campadelli et al. () for a survey). As neural representations tend to constitute nonlinear manifolds across modalities (Pope et al., ; Cai et al., ), we use a state-of-the-art nonlinear ID estimation method, the Generalized Ratios Intrinsic Dimension Estimator (GRIDE) (Denti et al., ), which we further describe in Section 3.4.

Deep learning problems tend to be high-dimensional. But, recent work reveals that these ostensibly high-dimensional problems are governed by low-dimensional structure. It has, for instance, been shown that common learning objectives and natural image data lie on low-dimensional manifolds (Li et al., ; Pope et al., ; Psenka et al., ); that learning occurs in low-dimensional parameter subspaces (Aghajanyan et al., ; Zhang et al., ); that modern neural networks learn highly compressed representations of images, protein structure, and language (Ansuini et al., ; Valeriani et al., ; Cai et al., ); and moreover, that lower-ID tasks and datasets are easier to learn (Pope et al., ; Cheng et al., ).

In the linguistic domain, considerable attention has been devoted to the ID of LM parameters. Zhang et al. () showed that task adaptation occurs in low-dimensional parameter subspaces, and Aghajanyan et al. () that low parameter ID facilitates fine-tuning. In turn, the low effective dimensionality of parameter space motivates parameter-efficient fine-tuning methods such as LoRA (Hu et al., ), which adapts pre-trained transformers using low-rank weight matrices.

Complementary to parameter ID, a number of works focus on the ID of representations in LM activation space. In particular, Cai et al. () were the first to identify low-dimensional manifolds in the contextual embedding space of (masked) LMs. Balestriero et al. () linked representational ID to the scope of attention, and showed how toxicity attacks can exploit this relationship. Tulchinskii et al. () demonstrated that representational ID can be used to differentiate human- and AI-generated texts. Cheng et al. () established a relation between representational ID and information-theoretic compression, also showing that dataset-specific ID correlates with ease of fine-tuning. Closer to our aims, Valeriani et al. () studied the evolution of ID across layers for vision and protein transformers, with a preliminary analysis of a single language transformer tested on a single dataset. Like us, they found that ID develops in different phases, with a consistent early peak followed by a valley, and less consistent markers of a second peak. We greatly extend their analysis of linguistic transformers by investigating the functional role of the main ID peak in five distinct LMs.

## 3 Methods

### 3.1 Models

We consider five causal LMs of different families, namely OPT-6.7B (Zhang et al., ), Llama-3-8B (Meta, ), Pythia-6.9B (Biderman et al., ), OLMo-7B Groeneveld et al. (), and Mistral-7B (non-instruction-tuned) (Jiang et al., ), hereon referred to as OPT, Llama, Pythia, OLMo, and Mistral, respectively.

OPT, Pythia, and OLMo make public their pre-training datasets, which are a combination of web-scraped text, code, online forums such as Reddit, books, research papers, and encyclopedic text (see Zhang et al. (); Gao et al. (); Soldaini et al. (), respectively, for details). The pre-training datasets of Llama-3 and Mistral are likely similar, though they remain undisclosed at the time of submission.

All language models considered have between 6.5 and 8B parameters, 32 hidden layers, and a hidden dimension of $4096$. Moreover, they all inherit the architectural design of the decoder-only transformer (Vaswani et al., ) with a few notable changes: 1) layer normalization operations are applied before self-attention and MLP sublayers; 2) Pythia adds in-parallel the self-attention and MLP sublayer outputs to the residual stream, while in other models, self-attention outputs are added to the residual stream before
the MLP; 3) Llama/Mistral/OLMo replace the ReLU activation function with SwiGLU (Shazeer, ), and Pythia instead uses GeLU (Hendrycks and Gimpel, ); 4) all models but OPT replace absolute positional embeddings with rotary positional embeddings (Su et al., ); 5) to facilitate self-attention computation, Llama uses grouped query attention (Ainslie et al., ), Mistral applies a sliding window attention, and Pythia adopts Flash Attention (Dao et al., ). In all cases, we use as our per-layer representations the vectors stored in the HuggingFace transformers library hidden$_$states variable.(^1^11Each hidden$_$state vector corresponds to the representation in the residual stream(Elhage et al., ) after one attention and one MLP update.)

In autoregressive models, an input sequence’s last token representation is the only one to contain information about the whole sequence. Furthermore, it is the only one that is decoded at the last layer to predict the next token. For these reasons, we choose to represent input sequences with their *last token representation* at each layer.

### 3.2 Data

Since we focus on model behavior in-distribution, we compute observables using
three corpora that proxy models’ pre-training data (all accessed through HuggingFace): Bookcorpus (Zhu et al., ); the Pile (Gao et al. (); precisely, the Pile-10k subsample available on HuggingFace) and WikiText-103 (Merity et al., ). From each corpus, we randomly extract 5 non-overlapping partitions of 10k 20-token sequences (sequence length is counted according to each corpus tokenization scheme).(^2^22We replicated the Pile-based Pythia and OPT experiments with sequences extended to 128 tokens. We obtained very similar results, confirming that the sequences we are using cover the typical contextual spans encode in model representations.) We do not constrain the sequences to have any particular structure: in particular, their final element is not required to coincide with the end of a sentence. We additionally generated 5 partitions of 10k 20-token sequences from *shuffled* versions of the same corpora. The latter respect the source corpus unigram frequency distribution, but syntactic structure and semantic coherence are destroyed.

### 3.3 Probing and downstream tasks

To relate representations’ geometry to their content, we use the probing datasets of (Conneau et al., ), meant to capture the encoding of surface-related, syntactic and semantic information in LM representations. For each layer, we train a lightweight MLP probe from the hidden representation to each linguistic task. Details on tasks and training procedure are given in Appendix H.

If there is a relation between ID and semantic content, then ID may also predict ease-of-transfer to downstream NLP tasks. To test this claim, we consider two such tasks, sentiment classification of film reviews (Maas et al., ) and toxicity classification (Adams et al., ), for which we train binary linear classifiers on each hidden layer representation (training details in Appendix I).

### 3.4 Intrinsic Dimension

Real-world datasets tend to show a high degree of (possibly) non-linear correlations and constraints between their features (Tenenbaum et al., ). This means that, despite a very large embedding dimensionality, data typically lie on a manifold characterized by a much lower dimensionality, referred to as its intrinsic dimension (ID). This quantity may be thought of as the number of independent features needed to locally describe the data with minimal information loss.

In almost every real-world system, the ID depends on the scale at which the data is analysed. In particular, at small scales, the true dimensionality of the manifold is typically hidden by that of data noise.
At very large scales, the ID can also be erroneous, due to, for instance, the curvature of the manifold (Facco et al., ; Denti et al., ). For this reason, in order to obtain a reliable and meaningful ID estimation, a proper scale analysis is necessary.
To this aim, we opted for the generalized ratios intrinsic dimension estimator (GRIDE) of Denti et al. (), which extends the commonly used(^3^33(Ansuini et al., ; Valeriani et al., ; Tulchinskii et al., ; Cheng et al., )) TwoNN estimator of Facco et al. ()
to general scales.

In GRIDE, the fundamental ingredients are ratios $\mu_{i,2k,k}=r_{i,2k}/r_{i,k}$, where $r_{i,j}$ is the Euclidean distance between point $i$ and its $j$-th nearest neighbour. Under local uniform density assumptions, the $\mu_{i,2k,k}$ follow a generalised Pareto distribution $f_{\mu_{i,2k,k}}(\mu)=\frac{d(\mu^{d}-1)^{k-1}}{B(k,k)\mu^{d(2k-1)+1}}$, where $B(\cdot,\cdot)$ is the beta function.
By assuming the empirical ratios $\mu_{i,2k,k}$ to be independent for different points, one obtains the ID by numerically maximizing the mentioned likelihood.

For each (model, corpus, layer) combination, we perform an explicit scale analysis. To do so, we first estimate the ID while varying $k$. Then, by visual inspection, we select a suitable $k$ that coincides with a plateau in ID estimate (Denti et al., ). An example of such a scale analysis for one (model, corpus) combination is reported in Appendix C.

In order to delimit high-ID peaks across layers,
we conventionally locate the end of the peak at the closest inflection point
after its maximum value. The beginning of the peak corresponds, then, to the closest layer before the maximum with value equal or greater than that at the peak end.

### 3.5 Quantifying the relative information content of different representations

Similar to Valeriani et al. (), we wish to relate dimensional expansion or compression of representations across layers to changes in their neighborhood structure. If neighborhood structure defines a semantics in representational space (Boleda, ), then layers whose activations have similar neighborhood structures perform similar functions.

In particular, the layers of an LM are iterative reconfigurations of representation space. We quantify the extent of reconfiguration (conversely, stability) using a statistical measure called Information Imbalance (Glielmo et al., ), hereon referred to as $\Delta$. Given two different spaces $A$ and $B$, this quantity, defined in equation (1), measures the extent to which the neighborhood ranks in space $A$ are informative about the ranks in space $B$ (since ID computation is based on Euclidean distance, ranks are also obtained with Euclidean distance):

$$ $\Delta(A\rightarrow B)=\frac{2}{N^{2}}\sum_{i,j|r_{ij}^{A}=1}r_{ij}^{B}.$ (1) $$

In words, $\Delta$ is the average rank of point $j$ with respect to point $i$ in space $B$, given that $j$ is the first neighbour of $i$ in space $A$.
If $\Delta(A\rightarrow B)\sim 0$, space $A$ captures full neighborhood information about space $B$. Conversely, if $\Delta(A\rightarrow B)\sim 1$, space $A$ has no predictive power on $B$.

It is essential to notice that $\Delta$ is non-commutative with respect to its arguments. That is, it is asymmetric upon a swap of spaces: $\Delta(A\rightarrow B)\neq\Delta(B\rightarrow A)$. This feature allows us to capture *directional* information containment, as we do below when considering the extent to which, e.g., a layer $i$ contains the information present in another layer $j$, or to measure the reciprocal information between representations of two different networks.

As $\Delta$ is a rank-based measure, it can be used to compare spaces of different dimensionalities and/or distance measures. In our specific case, this property implies that $\Delta$ is robust to possible dimension misalignments between layers.

In comparing the layers’ representation spaces, we also considered Doimo et al. ()’s neighborhood overlap measure which is used in Valeriani et al. (), as well as Representational Similarity Analysis (Kriegeskorte et al., ). These symmetric similarity measures tell a similar story to $\Delta$, but the signal is much weaker; therefore, we do not report them.

## 4 Results

We find, in line with previous work, that LMs represent language on a manifold of low intrinsic dimension. Furthermore, the representational ID profile over layers reveals a characteristic phase of both geometric and functional significance, marking, respectively, a peak in ID and transition in between-layer neighborhood similarity ($\Delta$), and a transition to abstract linguistic processing.

### 4.1 Emergence of a central high-dimensionality phase

Figure 1 (left) reports the evolution of the ID for all models, averaged across corpora partitions (for per-corpus results, see Appendix C). In line with previous work (Cai et al., ; Cheng et al., ; Tulchinskii et al., ; Valeriani et al., ), we first observe that the ID for all models is $\mathcal{O}(10)$, which lies orders of magnitude lower than the models’ hidden dimension at $4096\sim\mathcal{O}(10^{3})$.

Figure: Figure 1: Average ID (over 5 random partitions) of three corpora: Bookcorpus, the Pile and Wikitext, where samples come from (left) the original corpus, (right) the shuffled corpus. All curves are shown with $\pm$ 2 standard deviations (shuffled ID SDs are very small). In the middle layers, shuffled corpus ID is *lower* than non-shuffled ID, suggesting that linguistic processing contributes to ID expansion.
Refer to caption: x1.png

All models clearly go through a phase of high intrinsic dimensionality that tends to take place relatively early (starting approximately at layer 6 or 7, and mostly being over by layer 20), except for OPT, where it approximately lasts from layer 17 to layer 23. For all models, we also observe a second, less prominent peak occurring towards the end: only for OLMo, this second peak, which we do not analyze further, is as high as the first one.

#### ID is a geometric signature of learned structure

Figure 1 (right) shows that, when the models are fed shuffled corpora, ID remains low all throughout the layers. This suggests that the presence of high-ID peaks depends on the network performing meaningful linguistic processing. We further confirm this notion in Appendix D, where we analyze the evolution of the ID profile over the course of training for Pythia (the sole model whose intermediate checkpoints are public). We find that, over the course of training, the IDs’ magnitude not only grows over time, but that they become more peaked (Figure D.1), indicating that the characteristic profile emerges from learned structure.

#### The ID peak marks a transition in layer function

Figure 2 reports $\Delta(l_{i}\to l_{first})$ and $\Delta(l_{i}\to l_{last})$ for Llama, OPT and Pythia (with the remaining models in Appendix E). We observe that the ID peak largely overlaps with a peak in $\Delta(l_{i}\to l_{first})$.
Here, a large value of $\Delta$ implies that sequences which are nearest neighbours at the ID-peak are distant from each other in the input layer.
That is, the ID peak layers no longer contain the information encoded in the initial representation of the sequence, suggesting that they instead capture higher-level information. Meanwhile, we note that the $\Delta(l_{i}\to l_{last})$ profiles are not clearly related to the first ID peak, and we leave their analysis to future work.

Figure: Figure 2: For Llama, OPT, Pythia (left to right), the ID is overlaid with $\Delta(l_{i}\to l_{first})$ (gray) and $\Delta(l_{i}\to l_{last})$ (brown). Plots are shown with $\pm$ 2 standard deviations over 5 partitions of 3 corpora. For all models, there is a peak in $\Delta(l_{i}\to l_{first})$ (gray) around the ID peak.
Refer to caption: x3.png

Figure 3 shows the forward $\Delta$ scope profile for Llama, OPT and Pythia (see Appendix F for Mistral and OLMo). In particular, the plots indicate, at each source layer $l_{n}$, for how many contiguous following layers $l_{n+k}$ the quantity $\Delta(l_{n}\to l_{n+k})$ is below a low threshold ($0.1$). Qualitatively, this means that the source layer $l_{n}$ contains most of the information in the following $k$ layers. Coinciding with the ID peak is a downward dip in forward-scope, compatible with the interpretation that, while high-ID layers process similar information, this information differs from that of later layers.

Figure: Figure 3: Forward $\Delta$ scope (left: Llama; center: OPT; right: Pythia): continuous lines report, for each layer $l_{n}$, the number of adjacent following layers $l_{n+k}$ for which $\Delta(l_{n}\to l_{n+k})\leq 0.1$. The dashed line represents the longest possible scope for each layer. Values are averaged across corpora and partitions, with error bars of $\pm$ 2 standard deviations.
Refer to caption: x6.png

#### At the ID peak, different models share representation spaces

The high-dimensionality peaks contain similar information across models, as shown in the representative comparisons of Figure 4 (see Appendix G for the other pairs), which display cross-model $\Delta$ averaged across corpora and partitions.
The high-ID layers always overlap with low-$\Delta$ layers, indicating that between models, representations at the peak have close neighborhood structures, and thus capture similar semantics.

Conversely, ID-peak layers have high cross-model $\Delta$ with layers outside the peak, shown by a lack of points outside the shaded intersection in Figure 4. This indicates that, while ID-peak representations predict those of other models, they contain different information from other models’ representations outside of the peak.

We notice, moreover, a marked asymmetry in which the ID-peak layers of Pythia and OPT, the two models with the highest absolute IDs, directionally contain other models’ representations. On the other hand, when models have similar maximum IDs (such as in the Pythia vs. OPT comparison), $\Delta$ is more symmetric and very small, implying that their representation spaces are really equivalent.

Figure: Figure 4: Cross-model $\Delta$. ID-peak sections are shaded in orange. Different symbols mark different $\Delta$ levels in the two directions (lower values correspond to a stronger trend towards information containment). High $\Delta$ scores ($>0.1$), corresponding to low information containment, are not shown. Values averaged over corpora and partitions.
Refer to caption: x9.png

### 4.2 Language processing during the high-dimensionality phase

We just saw, via geometric evidence ($\Delta$), that the high-ID phase marks a change in processing function. Now, we examine exactly *what* that function is. To do so, we look at layer-wise classification accuracies for the probing tasks described in Appendix H. We find that, in general, ID-peak representations fail at surface-form tasks but excel at semantic and syntactic tasks, indicating a functional transition from superficial to abstract linguistic processing.

#### The ID-peak representations contain less surface-form information

In Figure 5(a), we consider the accuracy for two tasks, Sentence Length and Word Content, which test whether a layer retains information about superficial properties of the input. We observe that the ability to correctly reconstruct the length, measured in tokens, of the input sentence gets lost as we climb the network layers. Intriguingly, for OPT (Figure 5(a), center), which is the model with the latest ID peak, initial accuracy is relatively stable, and only starts to significantly decrease at the onset of the peak.

Concerning the Word Content task, which tests the ability to detect the presence of specific words in the input, we generally observe a great decrease, particularly clear during ID expansion. Interestingly, accuracy tends to go up again after the ID peak, probably because, as the model prepares to predict the output, more concrete lexical information is again encoded in its representations. Together, the surface-form tasks confirm the evidence from $\Delta(l_{i}\to l_{first})$ (Figure 2 above) that the ID peak processes a more abstract type of information that, as we are about to see, may relate to the syntactic and semantic contents of the sequence.

Figure: (a) Surface-form probes
Refer to caption: x12.png

#### The ID peak marks a transition to syntactic and semantic processing

Figure 5(b) reports accuracy for Llama, Pythia, and OPT (results for the remaining models in Appendix H) on the syntactic and semantic probe tasks. Despite variation across tasks and models, we observe that, in general, asymptotic accuracy is reached during the ID peak phase. Again, this implies that, for OPT, the asymptote is reached later. For OLMo (Appendix H), we observe in some cases a late accuracy peak, related to the second ID expansion phase that this model undergoes. In general, however, once top accuracy is reached, performance stays quite constant across layers, suggesting that the expressivity of high dimensionality permits rich linguistic representaiton of the inputs,
but, once this information is extracted, it is propagated across subsequent layers. This is intuitive, as high-level linguistic information is useful to the network for its ultimate task of next-token prediction.

#### Better LMs have higher ID peaks, earlier

Given the apparent linguistic importance of the high-ID phase, a natural question concerns the extent to which the nature of the ID peak explains the LM’s performance on its original task of next-token prediction. The question was already partially answered by Figures 1 and D.1, which show an absence of peaks, respectively, when trained LMs process shuffled text and when untrained LMs process sane (unshuffled) text; in both cases, next-token prediction is impossible.

We further computed Spearman correlations between average prediction surprisal for each (model, corpus) combination and the corresponding maximum ID values (Figure 6, left), as well as ID-peak onsets (Figure 6, right). As the plots show, even when limiting the analysis to sane text, where the differences in surprisal will be smaller, there is a marginal tendency for maximum ID value to inversely correlate with surprisal: the higher the peak, the better the LM is at predicting the next token. There is, moreover, a significant positive correlation between surprisal and the onset of the ID peak: the earlier the peak, the better the model is at predicting the next token.

We just saw that the ID peak marks the phase where the model first completes a full syntactic and semantic analysis of the input. The earlier this analysis takes place, the more layers the model will have to further refine its prediction by relying on it. The correlation between ID peak onset and surprisal thus indirectly confirms the importance of the high-dimensional processing phase for good model performance.

Figure: Figure 6: Surprisal plotted against the maximum ID across layers (left) and the relative ID peak onset over layers (right), where each datapoint is a (model, corpus) combination ($N=50$k sequences per corpus). A linear fit (left) and log-linear fit (right) are shown. (Left) Surprisal negatively correlates to maximum ID with Spearman $\rho=-0.46$, $p=0.09$, meaning that *higher ID indicates better LM performance*. (Right) Surprisal positively correlates to ID peak onset, $\rho=0.65$, $p=0.01$, meaning that an *earlier ID peak indicates better LM performance*.
Refer to caption: x18.png

#### ID-peak layers are the first to transfer to downstream tasks

Finally, given that the ID peak marks an abstract semantic representation of the input, representations at this peak should also viably transfer to downstream semantic tasks. We confirm this hypothesis for sentiment and toxicity classification.

Figure: Figure 7: Validation performance of representation transfer performance on toxicity and sentiment classification for Llama, OPT, and Pythia (left to right). All validation accuracy curves are plotted with $\pm$ 2 standard deviations over 5 random seeds. For all models, classification validation accuracy converges within the ID peak.
Refer to caption: x19.png

Validation accuracy curves for both tasks are shown for Llama, Pythia and OPT in Figure 7, with more results in Appendix I. Similar to the semantic and syntactic probing results, downstream classification performance consistently converges at the ID peak across models and remains high thereafter. Note that, while the ID peak is computed from the generic input corpora Bookcorpus, the Pile, and Wikitext, it still predicts transferrability to different downstream datasets that are reasonably in-distribution.

## 5 Conclusion

It is evident that a LM needs to extract information from its input to predict the next token. The non-trivial fact we show here is that this process does not gradually refine representations, but rather undergoes a phase transition characterized by ID expansion, cross-model information sharing, and a switch to abstract information processing. Mirroring Jastrzebski et al. ()’s proposition for visual networks, our findings are compatible with a view of LMs in which the high-dimensional early-to-mid layers functionally specialize to analyze inputs in a relatively fixed manner, whereas later layers may more flexibly refine the output prediction, using information extracted during the high-ID phase. Interestingly, two recent papers (Gromov et al., ; Men et al., ) found that (1) late LM layers better approximate each other than earlier layers do, and (2) pruning late layers (excluding the last) affects performance less than pruning earlier layers. This fully aligns with our results and suggests a need to test the effect of pruning inside and outside the ID peak. Other studies have highlighted the importance of central layers in performing various core functions. For example, Hendel et al. () find that compositional “task vectors” are formed in layers superficially consistent with the peaks we detect. Again, future work should more thoroughly study the relation between the ID profile and specific circuits detected in mechanistic interpretability work.

While our work closely relates to that of Valeriani et al. (), who studied the evolution of ID in vision and protein transformers, an interesting contrast is that they found crucial semantic information to coalesce during a dimensionality reduction phase, whereas we associated similar marks to a dimensionality expansion phase. In line with our results, recent work in theoretical neuroscience shows that high ID, thanks to its expressivity, underlies successful few-shot learning (Sorscher et al., ) and generalizable latent representations
for DNNs (Elmoznino and Bonner, ; Wakhloo et al., ). Conversely, primarily in artificial and biological vision, low ID is linked to generalization thanks to representations’ robustness to noise and greater linear separability in embedding space (Amsaleg et al., ; Chung et al., ; Cohen et al., ).
Clearly, whether dimensionality is a curse or blessing to performance depends on the context of the learning problem. Reconciling, then, why and when high performance arises from reduced or expanded dimensionality remains an important direction for future work.

## 6 Limitations

- •
While we establish an empirical connection between the ID peak and linguistic processing, we do not offer a mechanistic understanding of how the latter results in ID expansion, including which differences between models lead to different patterns, such as an earlier or later ID peak onset.
- •
The OLMo model displays a second ID peak and various outlying patterns that we did not attempt to interpret.
- •
We explored 5 language models on data extracted from three different corpora. However, due to compute constraints, the models we picked are all in the same size range (between 6 and 8 billion parameters). In the future, we would like to explore the extent to which the patterns we detected also appear at other scales.

## Appendix A Computing resources

All experiments were run on a cluster with 12 nodes with 5 NVIDIA A30 GPUs and 48 CPUs each.

Extracting LM representations took a few wall-clock hours per model-dataset computation. ID computation took approximately 0.5 hours per model-dataset computation. Information imbalance computation took about 2 hours per model-dataset computation. Probing/transfer classifiers took up to 2 days per task.

Taking parallelization into account, we estimate the overall wall-clock time taken by all experiments, including failed runs, preliminary experiments, etc., to be of about 20 days.

## Appendix B Assets

[https://huggingface.co/datasets/bookcorpus](https://huggingface.co/datasets/bookcorpus); license: unknown

[https://huggingface.co/datasets/NeelNanda/pile-10k](https://huggingface.co/datasets/NeelNanda/pile-10k); license: bigscience-bloom-rail-1.0

[https://huggingface.co/datasets/wikitext](https://huggingface.co/datasets/wikitext); license: Creative Commons Attribution Share Alike 3.0

[https://huggingface.co/meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B); license: llama3

[https://huggingface.co/mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1); license: apache-2.0

[https://huggingface.co/allenai/OLMo-7B](https://huggingface.co/allenai/OLMo-7B); license: apache-2.0

[https://huggingface.co/facebook/OPT-6.7b](https://huggingface.co/facebook/OPT-6.7b); license: OPT-175B license

[https://huggingface.co/EleutherAI/pythia-6.9b-deduped](https://huggingface.co/EleutherAI/pythia-6.9b-deduped); license: apache-2.0

[https://github.com/sissa-data-science/DADApy](https://github.com/sissa-data-science/DADApy); license: apache-2.0

[https://scikit-learn.org/](https://scikit-learn.org/); license: bsd

[https://scikit-learn.org/](https://scikit-learn.org/); license: bsd

[https://github.com/facebookresearch/SentEval/tree/main/data/probing](https://github.com/facebookresearch/SentEval/tree/main/data/probing); license: bsd

[https://huggingface.co/datasets/google/jigsaw_toxicity_pred](https://huggingface.co/datasets/google/jigsaw_toxicity_pred); license: CC0

[https://huggingface.co/datasets/stanfordnlp/imdb](https://huggingface.co/datasets/stanfordnlp/imdb); license: unknown

## Appendix C Intrinsic Dimension

### C.1 Scale analysis

For each model and corpus, we perform a scale analysis to select the neighbor order. An example of such an analysis is shown in Figure C.1, performed on the Pile for Pythia. We plot the ID estimate for increasing scales (x-axis in Figure C.1) for each layer. The true ID is likely to lie at a scale in which the ID estimate plateaus [Denti et al., ], which is marked by the highlighted region. For simplicity, per model-corpus combination, we choose one scale for all layers: in this particular example, we choose order $k=2^{5}$. In general, the scale tends to be around $k=32$ for all (model, corpus) combinations, allowing us to reliably compare between them (see Tables 1(a) and 1(b) for all $k$). Once the scale is chosen for each model-corpus combination, we plot the scale-adjusted ID estimates (see, e.g., Figure C.2).

Figure: Figure C.1: GRIDE ID estimation neighborhood scale analysis example for Pythia on the Pile on a single random seed, where each line is a layer’s ID estimate at different scales. All layers shown reach a plateau in the highlighted range.
Refer to caption: x22.png

**(a) GRIDE order $k$ reported for each model, corpus, mode (in sane and shuffled) combination. For simplicity, we chose one $k$ for all layers.**
| model | corpus | mode | GRIDE $k$ |
| --- | --- | --- | --- |
| llama | bookcorpus | sane | 32 |
| shuffled | 128 |  |  |
| pile | sane | 64 |  |
| shuffled | 128 |  |  |
| wikitext | sane | 64 |  |
| shuffled | 16 |  |  |
| mistral | bookcorpus | sane | 64 |
| shuffled | 128 |  |  |
| pile | sane | 128 |  |
| shuffled | 256 |  |  |
| wikitext | sane | 64 |  |
| shuffled | 32 |  |  |
| olmo | bookcorpus | sane | 32 |
| shuffled | 8 |  |  |
| pile | sane | 32 |  |
| shuffled | 128 |  |  |
| wikitext | sane | 32 |  |
| shuffled | 16 |  |  |
| opt | bookcorpus | sane | 16 |
| shuffled | 16 |  |  |
| pile | sane | 32 |  |
| shuffled | 16 |  |  |
| wikitext | sane | 16 |  |
|  | shuffled | 16 |  |
| pythia | bookcorpus | sane | 32 |
| shuffled | 32 |  |  |
| pile | sane | 32 |  |
| shuffled | 64 |  |  |
| wikitext | sane | 32 |  |
| shuffled | 16 |  |  |

### C.2 Additional results

Figure C.2 displays the evolution of estimated ID, scale-adjusted, per layer for all corpora and models. While the magnitude of ID differs across corpora, with ID on Bookcorpus lower than those of the Pile and Wikitext for all models, all corpora exhibit the characteristic high-ID peak, and at nearly the same onset. The dampened Bookcorpus peak IDs, which are still significantly above the corresponding shuffled ID peaks, might be explained by the fact that this corpus is entirely made of novels, and it is thus the less in-domain of the corpora we used.

Figure: Figure C.2: ID evolution over layers, shown with one standard deviation (over corpus partitions), for all models and corpora (left to right: Bookcorpus, the Pile, and Wikitext).
Refer to caption: x23.png

## Appendix D Intrinsic dimension over training

Figure D.1 shows the evolution of the average-ID profile over the course of training for Pythia, whose checkpoints are publicly available. Of-note, the ID profile seems to converge at between 16000 and 64000 iterations, or less than halfway until training is finished. We observe that towards the beginning of training (blue line), the ID profile is quite flat, and that, over the course of training, the characteristic central ID peak emerges. Beyond shape, we see that, while the initial-layer ID remains the same across checkpoints, the IDs of other layers tend to increase in magnitude. Both observations suggest that *expansion of ID* underlies linguistic processing of the input.

Figure: Figure D.1: Evolution of the average-ID profile over training for Pythia. Each curve corresponds to one checkpoint, at steps 512 until 143000 (the final model), and is shown with $\pm$ 2 SD over corpora and partitions.
Refer to caption: x24.png

## Appendix E Information imbalance with respect to first/last layer

Figure E.1 shows averaged $\Delta(l_{i}\to l_{first})$ (gray) and $\Delta(l_{i}\to l_{last})$ for Mistral and OLMo. Recall that the closer $\Delta(A\to B)$ is to 0, the more predictive $A$’s local neighborhood structure is of $B$’s. As expected, $\Delta(l_{i}\to l_{first})$ generally increases with $i$ as we go deeper in the layers; the reverse is true for $\Delta(l_{i}\to l_{last})$. However, $\Delta(l_{i}\to l_{first})$ appears to locally peak and plateau around the representational ID peak; that is, the ID expansion marks a phase of low predictivity from the intermediate layer to the input. OLMo’s pattern is not as clear, and we might observe a second local information imbalance peak in proximity to the second ID peak characterizing this model.

Figure: Figure E.1: For Mistral (left) and OLMo (right), the ID (hued) is overlaid with $\Delta(l_{i}\to l_{first})$ (gray) and $\Delta(l_{i}\to l_{last})$ (brown). Plots are shown with $\pm$ 2 standard deviations over corpora and partitions.
Refer to caption: x25.png

## Appendix F Forward Δ Δ \Delta roman_Δ scope

Figure F.1 reports the forward $\Delta$ scope profile for the remaining two LMs not shown in the main text (Mistral and OLMo).

Figure: Figure F.1: Forward $\Delta$ scope (Mistral, Olmo): continuous lines report, for each layer $l_{n}$, the number of adjacent following layers $l_{n+k}$ for which $\Delta(l_{n}\to l_{n+k})\leq 0.1$. The dashed line represents the longest possible scope for each layer. Values are averaged across corpora and partitions, with error bars of $\pm$ 2 standard deviations.
Refer to caption: x27.png

## Appendix G Cross-model Δ Δ \Delta roman_Δ

Figure G.1 show cross-model $\Delta$ for the remaining combinations, confirming that there are areas of low cross-model information imbalance at the intersection of the high-ID peaks. In combinations involving OLMo, we observe a tendency for low $\Delta$ to stretch along the other LM high-ID section, suggesting that the high-ID layers of other LMs share information with a wider range of OLMo layers.

Figure: Figure G.1: Cross-model $\Delta$. ID-peak sections are shaded in orange. Different symbols mark different information imbalance levels in the two directions (the lower the $\Delta(A\to{}B$) values, the more the information in $B$ is contained in $A$). High imbalances ($>0.1$) are not shown. Values averaged across corpora and partitions.
Refer to caption: x29.png

## Appendix H Probing tasks

### H.1 Tasks

We use the following classification tasks from Conneau et al. []:

- •
Surface form
Sentence Length
Predict input sentence length in tokens (lengths binned into 5 intervals).
Word Content
Tell which of a pre-determined set of 1k words occurs in the input sentence.
- •
Syntax
Bigram Shift
Tell whether the input sentence is well-formed, or it has been corrupted by inverting the order of two adjacent tokens (e.g., “They were in present droves, going from table to table and offering to buy meals, drinks or generally attempting to strike up conversations”).
- •
Semantics
Coordination Inversion
Tell whether a sentence is well-formed or it contains two coordinated clauses whose order has been inverted (e.g., “Then I decided to treat her just as I would anyone else, but at first she’d frightened me”).
Odd Man Out
Tell whether a sentence is well-formed, or it involves the replacement of a noun or verb with a random word with the same part of speech (e.g., “The people needed a sense of chalk and tranquility”).

We exclude the following tasks because we observed ceiling effects across the layers, suggesting that the models could latch onto spurious correlations in the data: Past Present, Subject Number and Object Number. We exclude Top Constituents and Tree Depth because they produced hard-to-interpret results that we believe are due to the fact that they rely on automated syntactic parses that are not necessarily consistent with the way modern LMs process their inputs.

### H.2 Setup

We use the training and test data provided by Conneau et al. []. We train a MLP classifier for each task and each layer of each LM, repeating the experiment with 5 different seeds.

We fixed the following hyperparameters of the MLP, attempting to approximate those used in the original paper (as each task takes days to complete, we could not perform our own hyperparameter search):

- •
Number of layers: 1
- •
Layer dimensionality: 200
- •
Non-linearity: logistic
- •
L2 regularization coefficient: 0.0001
- •
seeds: 1, 2, 3, 4, 5

For all other hyperparameters, we used the default values set by the the Scikit-learn library [Pedregosa et al., ].

We repeated all experiments after shuffling the example labels (both at training and test time). This provides a baseline for each task, as well as functioning as a sanity check that a probing classifier is not so powerful as to simply memorize arbitrary patterns in the representations [Hewitt and Liang, ].

### H.3 Additional Results

Figure H.1 reports the probing performance for OLMo and Mistral, and for surface form (top row) as well as syntactic and semantic tasks (bottom row). As for Llama, OPT, and Pythia, we observe that the first ID peak converges to viable syntactic/semantic abstraction of inputs, while discarding information about surface form.

Figure H.2 shows that, on the shuffled corpora ablation, the MLP probes perform at chance. We then confirm that semantic and syntactic information is contained in the *model representations* and not in the probes.

Figure: Figure H.1: Linguistic knowledge probing performance $\pm 2$ SDs across 5 random seeds is plotted with the ID across layers for OLMo and Mistral (left to right). (Top row) Surface form tasks Sentence Length and Word Content, where probe performance decreases through the ID peak. (Bottom row) Semantic and syntactic tasks Bigram Shift, Coordination Inversion and Odd Man Out, where probe performance for all tasks attains maximum within the ID peak.
Refer to caption: x36.png

Figure: Figure H.2: Linguistic probe validation accuracy for semantic, syntactic, and surface tasks (solid lines) and their shuffled versions (dashed lines) are shown across layers for all models. The probing performance on shuffled corpora is constant at chance for all tasks and models.
Refer to caption: x40.png

## Appendix I Downstream Tasks

### I.1 Tasks

#### Toxicity detection.

We use a random balanced subset ($N=30588$) of Kaggle’s jigsaw toxic comment classification challenge [Adams et al., ], where each data point consists of a natural language comment and its binary toxicity label.

#### Sentiment classification.

We use a dataset of IMDb movie reviews [Maas et al., ], where each data point consists of a natural language film review and a corresponding label $\in\{\text{positive},\text{negative}\}$. To train the linear probes, we take a sample of $N=25000$ corresponding to the train split on HuggingFace. We repeat the experiment with 5 distinct seeds.

### I.2 Setup

To train the linear probes, we first divide the data at random into train (80%) and validation (20%) sets. We feed each training set through each model and gather the last token hidden representations at each layer. Then, using PyTorch [Paszke et al., ], we train one linear probe per layer with hyperparameters as follows,

- •
Number of epochs: 1000
- •
lr: 0.0001
- •
seeds: 32, 36, 42, 46, 52

and we report the best validation accuracy over 5 random seeds.

### I.3 Additional Results

Similar to Llama, OPT, and Pythia, Figure I.1 shows that validation performance for downstream tasks for Mistral and OLMo converge in the ID peak.

Figure: Figure I.1: Validation performance on downstream transfer tasks on toxicity and sentiment classification for Mistral (left) and OLMo (right). All validation accuracy curves are plotted with $\pm$ 2 standard deviations over 5 random seeds. For all models, classification validation accuracy converges within the ID peak.
Refer to caption: x41.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: While the analyses show some interesting trends, it is difficult to tell how meaningful or significant the numerical differences are. Methods for analyzing LLM layers other than through ID could have been discussed in a prior work section.
soundness: 3
strengths: The paper conducts a broad analysis across 5 different LLMs and considers a range of questions and ablation studies (e.g., estimating ID on shuffled data, comparing layers across different models); altogether an impressively broad set of experiments. The paper is clearly written and presents a few new insights (e.g., correlation between peak onset and performance). Code and data would be made available, which would be valuable for the community.
summary: The paper uses the technique of intrinsic dimension estimation as a tool for analyzing properties of different transformer LLM layers. 5 different LLMs are analyzed on textual inputs from 3 different public-domain corpora. In addition to computing the intrinsic dimensionality (ID) (using the generalized ratios intrinsic dimension estimator) for different layers, the ID is  correlated with performance of different layers' representations on syntactic and semantic probing tasks. Furthermore,  the difference in representational power between different layers is measured using an Information Imbalance criterion. The authors find that middle layers in LLMs have the highest ID; ID peaks seem to be an indicator of linguistic structure being learnt; early onset of peaks in ID across layers is correlated with better next token prediction performance performance; and high ID peak layers are representationally equivalent across different LLMs.
weaknesses: The use of ID as an analysis tool for LLM layers is  not an entirely new idea (e.g., https://arxiv.org/pdf/2402.18048). 
Most of the results (e.g., the peaking of ID at middle layers, emergence of  linguistically informative representations in those layers) has been shown before by means of other methods (e.g., mutual information or canonical correlation analysis). These should have been discussed in more detail under prior work.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: (1) Please check the questions listed under Weaknesses.
soundness: 3
strengths: (1)	Although inspired by the earlier work of (Valeriani et al., 2023), this work greatly extends the models investigated to 5 distinct mainstay transformer-decoder-only LLMs and added more extensive probing and downstream tasks on defined datasets to analyze ID profiles across layers. Hence, the conclusions drawn in this work are verified across various models, datasets, and tasks, making the findings more convincing.

(2)	The comparisons to related works, esp. (Valeriani et al., 2023) which inspires this work, are clearly presented, hence the contributions of this work are clear and solid.
The verification of the emergence of a central high-dimensionality phase, and analysis of language processing behavior and performance during the high-dimensionality phase are quite thorough.

(3)	The analysis in Conclusion demonstrates that many findings in this work align with prior works and concurrent works. The paper clearly summarizes insights of guidance for future research. The Appendix provides detailed experimental setup and additional results. And finally, the analysis of potential applications of the findings is valuable to the research community.

(4)	Overall, the paper is clearly written and easy to follow.
summary: This work takes a high-level geometric approach to analyze intrinsic dimension (ID) of the representational manifold at each layer of a decoder-only Transformer LLM to understand how layer geometry relates to layer function. Although inspired by the earlier work of (Valeriani et al., 2023), this work greatly extends the models investigated to include five mainstay decoder-only LLMs, and added more extensive probing and downstream tasks on defined datasets to analyze ID profiles across layers. The resulting observations are different from those from (Valeriani et al., 2023). This work made quite a few interesting findings on detecting broad qualitative patterns, and provides useful guidance for future research towards interpretability, analysis of model behavior and quality, and model pruning and layer-specific fine-tuning etc.
weaknesses: (1)	Although the paper is overall clearly written, please make sure that every symbol used needs to be clearly defined when it first appears, e.g., d in Section 3.4.

(2)	Please provide rationale for critical algorithmic designs, for example, please clarify why GRIDE is selected, and why the three alternative measures for comparing layer’s representation spaces are chosen. 

(3)	Currently, k is still selected based on visual inspection. It would be useful to propose methods that can automatically select k.

(4)	It is interesting that OLMo seems a bit of an outlier compared to the other 4 LMs, although it also exhibits the ID peak and other related properties. It would be useful to provide insights on why OLMo behaves differently from the other models, and shed light on patterns of any potential “outlier” LM.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. There seems to have second ID peak in the later layers over LLMs. Do you think this second ID peak might reveal additional insights?
2. In your analysis (Figure 4), you observed that Pythia and OPT exhibit very similar representations. Could this similarity be attributed to pre-training on similar datasets? If so, how might this influence your findings, and have you considered controlling for dataset overlap to isolate structural factors more effectively?
3. The work focuses on classification tasks to analyze representation spaces in language models. Could you explain why generative tasks were not included? Do you expect the observed ID peaks and representation patterns to differ in generative contexts?
soundness: 3
strengths: This work conducts experiments on various LMs (e.g., OPT-6.7B, Llama-3-8B, Pythia-6.9B, OLMo-7B, and Mistral-7B) using multiple datasets, providing a comprehensive analysis. It also observes how representational intrinsic dimensionality (ID) varies across layers and proposes insightful hypotheses. Furthermore, this work inspires the research community to explore the utilization of ID information in transformer-based LM applications.
summary: The paper explored how transformer-based language models evolve their internal representations across layers, revealing a distinct high-dimensional abstraction phase. The authors observe the some findings across multiple LMs and datasets, and they provide a foundation for better understanding and optimizing language model architectures. This work bridges the gap between geometric representation and linguistic function in transformer-based LMs. Also, it highlights the potential of intrinsic dimensionality as a tool for analyzing and evaluating LMs.
weaknesses: While the paper combines two methods, GRIDE (Denti et al.) and Information Imbalance (Glielmo et al., 2020), to analyze four large language models (LLMs), this combination may fall short in terms of novelty. In Section 4.1, the choice of pre-training datasets for evaluation is also a limitation. Since these datasets have likely been encountered by the models during training, the results may not provide a fully accurate picture of the models’ generalization capabilities. Testing on unseen datasets would be crucial to evaluate the robustness and generalizability of the observed patterns, especially in real-world applications where unseen data is the norm. The study is limited to a narrow range of LLMs in terms of scale. Evaluating models of varying sizes (e.g., smaller models alongside large ones) would offer a more comprehensive understanding of how model size impacts intrinsic dimensionality and representation overlap across layers.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: I strongly advice to improve the submission w.r.t. the mentioned weaknesses. That helps both quality and reach.

The first paragraph of Asset Section C.1 (lines 809-836, in particular 828-829) mentions sensitivity of ID estimation w.r.t. to noise, small scales, density variations and curvature. That analysis suggests some sort of frequency decomposition integrated with the ID estimation.
soundness: 2
strengths: The findings in bold letters at lines 406-407 and lines 425-426 may be useful to some researchers that need to train and/or select models. The fact that ID seems to change gradually over layers is interesting, but may have a simple explanation in the extreme averaging scale of these models.
summary: The submission is analytic work trying to correlate intrinsic dimension of intermediate NN/LLM representations with linguistic targets. Comparison of the method applied to different models allows insights into some of their learned structural differences.
weaknesses: Except for the few strengths mentioned above, the submission does not explain for what else the gained insights can be used for or wether they are more useful than that at all.

The analysis focuses only on fully trained models and does not provide insights into how ID changes over time. A correlation analysis to other work would add more value. My first thought was a correlation to the IB method (e.g. Tishby et al. 2000 and Schwartz-Ziv & Tishby 2017), but this may not be the only or best choice.

The submission wrongly mentions PCA being linear (line 061, applies only to its original form) which leads to the quick conclusion to discard it. This is puzzling as the research on non-linear PCA is quite diverse based on very different techniques and there's even early work using neural networks dating back to 1991 (Mark Kramer: "Nonlinear PCA Using Autoassociative NNs").

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: In the data section, it is not very clear to me what does it mean to "extract 5 non-overlapping partitions of 10k 20-token sequences" and how the shuffled version is generated, can authors explain more about this?

In section: The ID peak marks a transition in layer function, I think the relation between ID peak and \delta(l_i \to l_first) is not very clear. It has very similar shape in OPT and somewhat in pythia, but LLAMA has a completely different curve. It is maximizing towards the end of the layer instead of the center of layer.

In section 4.2, authors also claim the relation between ID-peak and a few tasks. However, Figure (a) and (b) do not have very clear co-related trend between ID peaks and tasks' performance. In particular, task performance in Figure 5(b) seems to be monotonically increasing instead of peaking in the middle. Can authors justify more about this?
soundness: 2
strengths: I like that authors combine evidence from a few different perspectives to demonstrate the relation between intrinsic dimension peak and transition to abstract processing. They also conduct experiments on a few corpus and a few models as well, which make the claim more general and robust
summary: This work analyzes properties of representations of several LLMs through a few approaches: downstream probing, intrinsic dimensionality and information imbalance. The analysis is mainly developed around intrinsic dimension and they show LLMs typically have a few intrinsic dimension peaks across layers. Additionally, they suggest that those peaks indicate transition to abstract linguistic processing through a variety of analysis
weaknesses: The method section is weak and the explanation of intrinsic dimension computing is not enough given its importance in this work. I was not able to identify which variable is corresponding to the intrinsic dimension without going through the cited paper. It seems to be the variable $d$ which authors did not explain what it is.

Additionally, author made a wrong claim in line 177~178 that \mu has a generalised pareto distribution. I cannot find any resources claiming this specific distribution is a (generalized) pareto distribution including the original cited paper.

Meta-review: The paper employs intrinsic dimension (ID) estimation as a technique to analyze the properties of different layers in transformer-based LLMs. While inspired by previous work, this study expands the scope by including five LLMs and introducing more extensive probing and downstream tasks on defined datasets to analyze ID profiles across layers. The paper presents several interesting findings. Most of the concerns raised by the reviewers were addressed during the authors' rebuttal.

Decision: Accept (Poster)