The geometry of hidden representations of large transformer models
Authors: Lucrezia Valeriani
Diego Doimo
Francesca Cuturello
Alessandro Laio
Alessio ansuini
Alberto Cazzaniga
Venue: NeurIPS 2023 poster (2023)
Keywords: Representations
transformers
geometry
interpretability

Abstract: Large transformers are powerful architectures used for self-supervised data analysis across various data types, including protein sequences, images, and text. In these models, the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. 
We characterize the geometric and statistical properties of these representations and how they change as we move through the layers.
By analyzing the intrinsic dimension (ID) and neighbor composition, we find that the representations evolve similarly in transformers trained on protein language taskand image reconstruction tasks. In the first layers, the data manifold expands, becoming high-dimensional, and then contracts significantly in the intermediate layers. In the last part of the model, the ID remains approximately constant or forms a second shallow peak. 
We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets.
Based on our findings, we point out an explicit strategy to identify, without supervision, the layers that maximize semantic content: representations at intermediate layers corresponding to a relative minimum of the ID profile are more suitable for downstream learning tasks.

The geometry of hidden representations of large transformer models
Lucrezia Valeriani 22
Diego Doimo! 1.3+
Francesca Cuturello'
Alessandro Laio
Alessio Ansuini
Alberto Cazzaniga If
AREA Science Park, Trieste, Italy University of Trieste, Trieste, Italy SISSA, Trieste, Italy ICTP, Trieste, Italy
Abstract
Large transformers are powerful architectures used for self-super data analysis across various data types, including protein sequences, images, and text. In these models, the semantic structure of the dataset emerges from a sequence of transfor- mations between
representation and the next. We characterize the geometric and statistical properties of these representations and how they change as we move through the layers. By analyzing the intrinsic dimension (ID) and neighbor compo- sition, we find that the representations evolve similarly in transformers trained on protein language tasks and image reconstruction tasks. In the first layers, the data manifold expands, becoming high- nigh-dimensional, and then contracts significantly in the intermediate layers. In the last part of the model, the remains approximately constant or forms second shallow peak. We show that the semantic information of the dataset better expressed the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets. Based on our findings, we point out an explicit strategy identify, without supervision, the layers that maximize semantic content: representations at intermediate layers corresponding to a relative minimum the ID profile are more suitable for downstream learning tasks.
Introduction
Deep learning has significantly changed the landscape of scientific research and technology ad- vancements across multiple disciplines in recent years. A particular class of deep learning models, transformers trained with sol--uuperi,ioon combine high predictive performance and architectural simplicity. These transformer models consist of a stack of identical self-attention blocks trained in self-supervised manner using masked language modeling (MLM) or regresegressvve objectives [1, 2]. It has been shown that the features Icarned by these models can be used to solve wide range of downstream tasks in natural language processing [3-5], biology [6-9], and computer vision [10-12].
Previous studies analyzing convolutional architectures have shown that data representations in deep learning models undergo profound changes across the layers [13, 14]. In transformers, each module maps the data into representation, and it has already been observed that the organization of representations last hidden layer can reflect abstract, domain-speciffc properties [10]. However,
"These authors contributed equally to this work. 'Corespondence: alessio.ansuing @ ireascienceparkkit,, ullerto.ccazaangge @ areasciencepark.it
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in models trained by self-supervision, the last representation generally has the role of allowing reconstruction of the imput representation. Therefore, the most semantically rich representation likely to arise within the intermediate hidden layers of the network.
In this work, we study the intrinsic dimension and neighhor composition of the data representation in ESM-2 [15] and Image GPT (iGPT) [10], two families of transformer architectures trained with self-supervision on protein datasets and images (see Sec. 2.3). We find consistent within-domain behaviors and highlight similarities and differences across the two domains. Additionally, we develop an unsupervised strategy to single out layers carrying the most semantically meaningful representations.
Our main results are:
representations in large transformers evolve across the layers in distinct phases, revealed by simultancous changes in global ID and local neighborhood composition (Sec. 3.1);
a common trait of the ID profiles a prominent peak of the ID, in which the neighbor structure is profoundly rearranged, followed by relatively low ID regions in which the representations encode abstract semantic information, such as the class label for images, and remote homology for proteins (Sec. 3.2). This abstract information emerges gradually while the ID decreases (Sec. 3.1);
in the case of protein language models (pLMs) and iGPT, our experiments show that the ID profile can be used to identify in an unsupervised manner the layers encoding the optimal representation for downstream learning tasks, such protein homology searches image classification (Sec. 3.2).
These findings suggest general computational strategy learned by these models, in which the reconstruction task is performed in three phases: 1) a data expansion in a high intrinsic dimension space. This expansion resembles the strategy followed in kernel methods [16], where one implicitly expands the feature space by introducing non-lineat functions of the input features; 2) a compression phase that projects the data in low-dimensional, semantically meaningful representations. Strikingly, this representation emerges spontaneously in all the self-supervision tasks we considered; 3) a decoding phase where the model addresses the minute decision-making needed to reconstruct the data from a compressed representation. The layers performing this task behave similarly to the decoder layers in a "vanilla" autoencoder. In particular, understanding this behavior can be exploited as an unsupervised method to identify the most semantically rich representations of transformers trained on large datasets where the annotation scarce.
2 Methods
2.1 Intrinsic dimension.
Learning complex functions in digh-dimensional spaces is challenging because the dimension of the space increases, the required number of samples must grow exponentially [17]. Fortunately, real-world datasets, such as images and protein sequences exhibit regularities that result in strong constraints and correlations among their features [18, 19]. Consequently, although these datasets may have many features, they lie around low-dimensional, often highly non-linear manifolds. A dataset's intrinsic dimension (ID) is the dimensionality of the manifold approximating the data and can be described as the minimum number of coordinates that allow specifying data point approximately without information loss [20, 21J. We measure the ID with the TwoNN estimator [21]. which requires only the knowledge of the distances T'i 7'2 between each point x, and its first two nearest neighbors. It can be shown that under the assumption of locally constant density, the ratio μ4 = rsa/ra follows a Pareto distribution with shape parameter equal to the ID. The intrinsic dimension can be inferred by maximum likelihood or linear regression from the cumulative distribution function (see Appendix Sec. B). The algorithm is robust to changes in curvature and density of the data and has been used effectively to analyze representations in deep neural networks in [13, 14] and more recently in [22-24]. We adopt the TwoNN implementation of DADApy [25] and discuss the technical details of analysis together with the validation of the constant density assumption in Appendix B.

---

2.2 Neighborhood overlap.
Changes in the data representation between successive layers of a model can be traced in the rearrangement of the neighbor structure of each data point. The neighborhood overlap Xx 1. introduced in [14] measures the similarity between two representations I, m with the average fraction of common k-nearest neighbors of a data point. Consider the activation vector z; of an example at layer L. Let A' be the adjacency matrix with entries equal Ai = 1 if = belongs to the first k-nearest neighbors of rf in Euclidean distance and 0 otherwise. Then the neighborhood overlap between representations J.m I and m can be written X k-neighhors of point x; between these representations.
E E, A',, A.y. where A', A7) the number of common
In the presence of a labeled dataset {< y:) a generalization of the neighborhood overlap measuring the consistency of label within the nearest neighbors of each data point be used semantic
probe. In this case, the "adjacency matrix" with respect to the dataset labels is A = 1 if y: Vj and
0 otherwise, and we call 1.22 overlap with the ground truth classes. Both X 1.m and Xk and depend on the choice of neighborhood size k. However, in the Appendix (see Fig. S5), we show that the qualitative trend of the overlap is robust to changes in k. We use k
J. lie in [o, 1]
10 when analyzing the overlap with the protein superfamily of the SCOPe dataset, and 30 in the case of the transformers trained on the ImageNet dataset, consistently with the value used in Doimo et al. [14].
2.3 Models and Datasets
Transformer models for protein and image datasets. The ESM2 single-sequence protein lan- guage models (pLMs) and iGPT networks are characterized by similar architectures. The token vocabulary comprises an alphabet of = 20 amino acids for pLMs and 512 colors for iGPTs (see Sec. A and Chen a vector of size d, which is then added to learned positional embedding common to all sequences. A sequence I embedded tokens of size Ixd is thus processed by stack of architecturally identical blocks composed of a self-attention map followed by multi-layer perceptron (MLP) producing series of data representations of size R We extract the hidden representations of the sequences after the first normalization layer of each block and then average pool along the sequence dimension to reduce a sequence into a data point embedded in Idd This way, we can compute the Euclidean distances needed for the ID and overlap analyses between sequences of different lengths.
[10] for further details). An embedding layer encodes each sequence token into
We study models pre-trained on large corpora of proteins and images with different self-supervised objectives: ESM2 were trained by Lin et al. [15] on the Uniref50 dataset with a masked language model objective and have been publicly released at github. oom/ffacbbookressaccheeenm the iGPT networks were trained by Chen et al. [10] with next token prediction objective on the ImageNet dataset and are available at github.com wwwphhhcoo//opp/oaggeea.
Datasets. We consider two benchmark datasets for analysis of pLMs: ProteinNet and SCOPe. ProteinNet [26] is dataset of 25, 299 protein sequences for evaluating the relationships between protein sequences and their structures. We use the ProteinNet training to analyze the ID curves and compute the neighborhood overlap of consecutive layers. The Astral SCOPe v2.08 (SCOPe) dataset [27] contains subsets of genetic domain sequences hierarchically classified into fold, superfamily, and family. We first preprocess the sequences with the filtering procedure recommended in [28]. Secondly, to define the remote homology task, we select proteins that belong to superfamilies with at least ten members and ensure that each superfamily consists of at least two families. As result, we obtain a dataset of 10,256 sequences and 288 superfamilies, used to evaluate the Litt in Fig. 4.
For the analysis of the iGPT representations, we choose 90, 000 images from the ImageNet training set [29]. We randomly select 300 classes and keep 300 images per class.
Reproducibility. All experiments are performed on a machine with 2 Intel(R) Xeon(R) Gold 6226 processors, 256GB of RAM, and 2 Nvidia V100 GPUs with 32GB memory. We provide code to repro- duce our experiments and our analysis online at gittub..om//ingddodo/gggnttttt reppreeettntty..rsseettttttnsss
3

---

ESM-21300
ESM 21650M
- ESW-2155)
XPPF LC1-48)
(GPT IN555VO
HGFT (CFT-52
35
35
30
25
20
20
10
10
0
1.0
relative depth
relative depth
UniRef
ImageNet
Figure 1: Intrinsic dimension (ID) of the data representations in hidden layers of large trans- formers. (left) The ID profile of ESM-2 protein language models of small (35M, green). medium (650M, blue), and large (3B, purple) sizes show peak in the first part of the network, long plateau, and final ascent. (right) The ID profile in iGPT models of small (76M, yellow), medium (455M, orange), and large (1.4B, red) sizes also present an early peak followed by less pronounced second peak for the medium and large model. The ID is plotted against the relative depth: the block number divided by the total number of blocks.
3 Results
3.1
The intrinsic dimension profile in large transformers is similar in different tasks and architectures
In large transformers, the input representation is processed in sequence of identical blocks, creating series of representations in vector spaces the same dimension but radically different in their geometrical properties and semantic content. We start by estimating the ID) of each representation large transformers trained by self-supervision on protein language and image reconstruction tasks. The ID profiles provide glimpse into the geometry of the manifolds in which the data embedded. As we will see, the ID changes through the layers in a highly non-trivial manner, with three phases that can be recognized in all the transformers and a fourth phase that observed only the iGPT models. the following sections, we will see how these phases signal the development of structured semantic content in intermediate representations, which are hidden instead in the input and, by construction, in the final embeddings.
ID curve of protein language models. We extract representations the protein sequences in the ProteinNet dataset as explained in Sec. 2.3 and plot their ID against the block number divided by the total number of blocks, referred to as relative depth in Fig. This section describes our findings for MLMs in the ESM-2 model family with 35M, 650M, and 3B parameters. Analogous results hold for other models trained on different datasets, as shown in the Appendix (see Fig. S1).
Figure (left) shows the typical shape of the ID profile in pLMs with three distinct phases: a peak phase, a plateau phase, and final ascent phase. The peak develops in early layers, spanning roughly the first third of the network. During this phase, the ID expands rapidly, reaching a maximum of approximately 20, 25, and 32 for ESM-2 35M, 650M, and 3B, respectively. The ID then contracts quickly, stabilizing at notably low values, between 5 and 7, that define the plateau phase. While the ID at the peak grows with the model size, during consensus in the ID, especially in the layer at the elbow before the final ascent, independent from embedding dimension that varies from 480 in ESM-2 35B to 2560 ESM-2 3B. Notably, the ID observed in the platcau phase is also consistent with the values between and 12 measured by
plateau phase, we observe quantitative solid

---

Facco et al. [30]. Their study involved different metries for computing distances directly applied to pairwise alignments of protein sequences. This remarkable correspondence between different analytical approaches suggests that hidden representations in the plateau region can effectively gauge the underlying degree of variability arising from evolutionary changes in protein sequences. In the final ascent, the ID grows again, returning progressively to values close to the ID computed on the input representation after the positional embedding. This is a consequence of the masked language modeling objective, which focuses using the context information to recover the missing tokens.
ID curve of image transformers. Figure 1 (right) shows the ID profiles of iGPT models of increasing size. Specifically, we examine the small (S), medium (M), and large (I.) versions of iGPT trained on ImageNet, which have 76M, 455M, and 1.4B parameters, respectively.
In all cases, the IDs of the output are similar to those of the input, which aligns with the findings from protein language models. In the iGPT-S network, the ID profile has a hunchback shape quite similar to that in convolutional models trained on ImageNet [13] but with a peak value significantly smaller (around 25). This discrepancy could be partly attributed the different embedding dimensions: equal to 512 in the iGPT-S transformer, whereas in convolutional architectures, it fluctuates across layers and is considerably larger, typically falling within the range of O(10*) to ((10).. As we increase the model size, similar to what is observed (pLMs). the maximum intrinsic dimension (ID) grows, reaching 28 for iGPT-M and 32 for iGPT-L. However, after an initial decrease between 0.3 and 0.4 of the relative depth, the ID shows local minimum of 22 common between the architectures. In the last part of the network, in contrast with what is observed in pLMs,, the ID experiences another, more gradual increase, forming second, shallower peak near the end of the network. Interestingly, the ID of the most compressed representations at around 0.4 of relative depth is compatible with the values observed by Ansuini et al. [13] at the output of broad range of convolutional classifiers trained on the ImageNet dataset. In their study, the ID varied from 13 to 24 and was found to be correlated with the classifier's s accuracy (refer to Figure 4 in Section 3.3 of [13]).
The results shown in Fig. hint at scenario that will be validated in the following: the ID is an indicator of semantic complexity, and the representations that resemble more closely those extracted by supervised learning on the same datasets are expected to correspond to the relative minimum of the ID curve, situated after the initial peak. This minimum is well defined in iGPTs, while in pLMss the ID profile forms a plateau.
The neighborhood rearrangement across the layers mirrors the ID profile. We will now exam- ine the evolution of the data distribution across the layers. specifically by analyzing the neighborhood overlap X 1./+1 defined in Sec. 2.2. This parameter quantifies the rate at which the composition of the neighborhood of each data point changes between nearby blocks. In deep convolutional models, this rate is significant only in the very last layers [14), where the representation is forced to comply with the ground-truth classification of the images.
In pLMs. we evaluate the neighborhood overlap on the representations of the ProteinNet dataset. The results of our analysis, presented in Fig. 2 (left), reveal relationship between changes in neigh- borhood composition and the ID. This observation points to an interesting connection between ID and neighborhood overlap, which differs from what is typically found in convolutional architectures. In the first 40% of the layers, X X. remains approximately at 0.5, meaning that the neighborhood composition undergoes substantial changes in each layer, significantly altering representations in the initial part of the network. This corresponds with the peak phase of the ID. However, during the plateau phase, the rate of representation evolution is much slower, with over 90% of the neighbors shared between consecutive layers in the larger models. In the smallest model, characterized by a higher perplexity, the neighborhood composition in the plateau phase is less consistent, with some rearrangements also occurring in successive layers.
In image transformers, Fig. 2 (right), the profiles are qualitatively similar to those observed in protein language models. However, the transition between the initial stage, coinciding with the first ID peak and relative depth < 0.4, with rapid changes in neighborhood composition (x/ 0.7) and second stage where the rearrangements are less pronounced (x11+1 ~ 0.9) is more gradual than pL.s. A significant neighborhood rearrangement is always observed in the last layers, where the reconstruction task is carried on. Similarly to pl.Ms, x1.41 lower in shallower models that need faster rearrangements to achieve consistent results in fewer blocks.
5

---

ESN-2181
ESM 200SEMI
ESM-203510
EOPT MMO53NI
10FT-50490
1.0
1.0
0.8
0.8
0.6
0.2
0.0
1.0 0.0
relative depth
relative depth
Figure 2: The neighborhood rearrangement across the layers in large transformers. (left) Neighborhood overlap of consecutive layers x computed for ESM-2 (35M, green). ESM-2 (650M, blue), and ESM-2 (3B, purple), shows major local rearrangements the layers corresponding to the peak and in the final layers and minor changes in the plateau region, mirroring the behavior of the ID. (right) The trend of x 1.++1 iGPT-S (yellow), iGPT-M (orange), and iGPT-L (red) is similar to ps.s: the neighborhood composition changes in early layers where the ID reaches peak and in the final layers.
The evolution of the representations during training. Until now, our focus has been on observing changes in some geometric quantities across layers in fully trained models. We will now focus on understanding how these quantities evolve during training. Specifically, we investigate how representations of the data manifold transform during training for two models: the protein language model ESM-2 (650M) and the image transformer iGPT-L.
For the ESM-2 (650M) model, as shown in Figure 3 (left), we analyze checkpoints corresponding to [0, 0.1... 0.3, 1, 5] 10 training iterations, with the last checkpoint representing the fully converged model. In the initial stages of training, we observe rapid formation of peak in the early layers, while the ID curve in the remaining part of the network closely resembles that of an untrained model. Between 1- 10 and 3 10 training steps, the ID of the plateau layers substantially decreases. In contrast, the ID measured in the last layers progressively increases towards the input ID.. From 3 10a training steps to convergence, the ID) curve takes its final shape, with slight increase of the II) measured at the peak and minor compression at the plateau. The final ID curve essentially achieved in two stages: first, the initial ID peak emerges, and only in second phase during training the data representation in the plateau layers is compressed to a lower dimension. This compression is tightly related to the emergence of semantic information that we discuss in Sec. 3.2.
For iGPT-L, we consider the representations at checkpoints corresponding to [0. 0.13, 0.26, 0.52, 1] 10 training iterations (see Fig. 3 (right)). The ID dynamics is similar to that of pLMs: before training. the ID curve is flat, and early training iterations (< 1.3 1.3-10 10²), the first ID peak emerges the initial layers of the network. In the later stages of training, the ID slightly increases at the first peak and decreases at 0.4 of relative depth, forming the local minimum. Additionally, second peak emerges in the last third of the model's hidden layers.
3.2 The intermediate representations of large transformers are the most semantically rich
A common characteristic among the ID) profiles shown in Fig. is the presence of distinct peak in the first part of the network, observed consistently in all the models we examined. In protein language models, this initial peak is followed by an extended plateau that spans more than half of the layers. In contrast, in iGPT models, a second shallow peak in the is observed, with the exception of the iGPT-S model, which does not have one. In section 3.1, we found that the ID at the minimum point between the two peaks is quantitatively similar to the one measured at the output of convolutional networks trained with supervision representations at this local minimum encode semantic information about the datasets. Chen et al. [10] have already demonstrated with linear probes that features extracted from the intermediate layers
the same dataset [13]. This points to scenario where the
The weights of the checkpoints of ESM-2 (650M) analyzed in Fig. were kindly provided by R. Rao.

---

10
227
21207
10
30
20
25
15
15
1.0
5
0
0.0
0.2
0.6 relative depth
1.0
0.2
0.4
0.8
1.0
relative depth
Figure 3: Evolution of ID curves during training. (left) ID curves for the PLM ESM-2 (650M) at checkpoints corresponding [0, 102,3- 30.. 100,00,,5 5 10 training steps. Firstly, the peak develops; then, the plateau reaches low-ID values, and the ID) in the final layers reaches values similar to the input. (right) IDD curves for iGPT-L (1.4B) at checkpoints (0, 0.13, 0.26, 0.52, 1] 10 training steps. Firstly, the ID peak followed by plateau with a local low-ID minimum develops; later in training, a second peak emerges in the final layers.
of iGPT models encode this semantic information. In this section, we extend this analysis to protein language models, showing that, this case, semantic information is maximally present in the whole plateau region of the network. Therefore, we show that the ID profile can be used to predict the specific layers where this information is most pronounced.
The representations the plateau region code remote homology relationships. Proteins are considered remote homologs when they share similar structure resulting from common ancestry despite having highly dissimilar amino acid sequences. Rives et al. [6] observed that the Euclidean distances among representations in the last hidden layer of pLMs encode information related to remote homology. We study how this biological feature emerges in pLMs, analyzing the hidden representations of the SCOPe dataset. For every layer 1, compute the neighborhood overlap with lggt
the ground truth labels X with & 10. Here, "gt" represents the classification by superfamily, with the exclusion of neighbors in the same family, allowing us to focus on remote homology specifically. Figure 4 (left) shows that structural homology information absent in the positional embedding layer; grows smoothly in the peak phase, reaching stationary maximum X 0.8 in the plateau phase, and it suddenly decreases in the final ascent phase. This analysis shows that homology information is highest in the plateau layers, and it appears early in the networks right after the peak the ID. Notably, the predictive power of remote homology by nearest neighbor search is considerably lower in the last hidden layer, scoring at 0.4 instead of 0.8. Therefore, searching for the closest homologs better expressed can improve state-of- tfite--f-thh-eeee ff-th--art
in the plateau layers where protein relationships methods based on representations from the last hidden layer. The Appendix shows that the entity of such improvement is approximately 6% (see Fig. S2).
Image semantic features emerge in the layer where the ID has relative minimum. It has been shown that predicting pixels in a self-supervised learning context can generate representations that encode semantic concepts [10]. Expanding upon these findings, we establish a connection between this remarkable property and the geometric aspects of representations. Similar to the case of pLMs, we quantify the semantic content of representation by computing the overlap with the ground truth labels, xlatt, In Figure 4 (right), we plot for the iGPT models the overlap with the ImageNet labels for 300 classes of the training set (see See 2.3). all the cases, we consistently observe a peak of 1-9t around relative depth of ~ 0.4 where the ID in the larger models iGPT-M and iGPT-L is lower (see Fig. 1). This peak becomes sharper, and the overlap values increase as we scale up the model size. Specifically, the peak value of X -et is 0.15 in iGPT-S, 0.27 in iGPT-M, and 0..35 in iGPT-L. Similar to our observations pLM the representations where the semantic abstractions are better encoded are also those where the ID is low.
7

---

ESP-2296
2035
APPILZE
OPI SUU5SMI
OF 117-11
1.0
0.40
0.8
0.6
0.25
0.15
0.10
0.2
0.05
0.0
0.00
1.0
0.0
0.4
relative depth
relative depth
Figure 4: Local geometry and semantic relations hidden representations. (left) Overlap with remote homologs x for ESM-2 (35M, green). ESM-2 (650M, blue). and ESM-2 (3B, purple). Information about remote homologs is highest in the plateau layers, corresponding to low ID values. (right) Overlap with ImageNet labels x for iGPT-S (yellow), iGPT-M (orange), and iGPT-L (red). The information related to ImageNet labels is highest in correspondence with the ID minimum.
Discussion and conclusions
In this work, we investigate the evolution of fundamental geometric properties of representations, such as the intrinsic dimension and the neighbor structure, across the layers of large transformer models. The goal of this analysis is twofold: 1) understanding the computational strategies developed by these models solve the self-supervised reconstruction task, and 2) monitoring how high-level, abstract information, such as remote homology in proteins and class labels in images, is related to the geometry of the representations. particular, we propose an explicit, unsupervised strategy to identify the most semantically rich representations of transformers.
The qualitative picture that emerges from results is that large transformers behave essentially like sophisticated autoencoders, in which the data are first encoded into a fow-dimensional and abstract representation and successively decoded from it. In the largest iGPT model, this similarity is particularly evident: a second peak in the ID profile is present, approximately mirroring the first, and the overlap with the ground truth labels also varies in an almost symmetric manner in the first and the second half of the network (see Fig. 4 (right)). This makes this model akin to a symmetric autoencoder, in which the decoder performs operations the representations that are dual those performed by the encoder.
Crucially, the compression is preceded by an intrinsic dimensionality expansion and relatively fast-paced rearrangement of the neighbors during the encoding process, which is very similar across all the models we studied. The relation between a low intrinsic dimension the end of the encoding part and the richness in abstract content of representations is robust and quantitative in all the models considered. A vast rearrangement of the neighbors in the layers close to the output also observed for all the architectures we analyze. This is devoted to the process of dealing with the decoding task, and it causes degradation of the abstract content.
The analysis we performed could be further reinforced, for example, by analyzing the similarity of representations in the ID minima with those generated in models trained in supervised setting. This can provide a deeper understanding of the role of the second peak in the large iGPT model and the reasons for its appearance in later training phases, as well as offer clearer interpretation of low ID representations in the plateau phase of pLMMs
Impact of some implementation choices. In this study, we average pool the representations along the sequence dimension as convenient method compute the Euclidean distances between proteins of varying lengths. This averaged representation can serve multiple purposes, including measurement representation quality using linear probes [10], and can provide sufficient biological information to address homology, structural, and evolutionary tasks directly or after fine-tuning. It's worth noting, however, that other approaches for comparing sequences of variable lengths have been explored, such as those in [31] and [22]. When the inputs have all the same sequence length in

---

the iGPT case, reducing the sequence representation to d-dimensional vector is also convenient from a computational perspective, Indeed, calculating distance matrices for many images (e.g.. 90, 000) in the entire feature space (1 d = 524, 288) would have been unfeasible given our current computational resources. In section D of the Appendix, we show with an experiment on CIFARIO that the ID of a representation decreases after average pooling. Still, the fundamental shape of the II) does not change qualitatively (see Fig. S3). In the same section, we also show that the ID profiles do not quantitatively change if we extract the representations after the attention maps or after the MLP blocks (see Fig. S4),
Discussion of different ID shapes. In Sec. 3.1, we showed that the ID profile in PLMs has three phases and terminates with a final expansion, while in iGPTs, there is a fourth phase where the ID is compressed in the last layers. These differences can be attributed to various factors. For instance, in the case of ImageNet, variations of the pixel brightness [13] and details of data preprocessing. including the substantial reduction of the input resolution (see Sec. A and [10] can artificially alter the ID significantly. In the middle layers, instead, the ID is connected to the semantic complexity of the dataset, and its value tends to be consistent not only within transformers of the same class but also with the last hidden layer of convolutional networks in the case of the ImageNet dataset [13] and with other methods relying on different distance metrics in the case of proteins [30] (see Sec. 3.1). This implies that the II) of the compressed representations can be larger or smaller than the one measured the early layers and close to the output. where the network reconstructs the inputs. These considerations may explain why the final layers of the network can either reduce the ID (as seen in iGPT) or expand it (as observed in ESM models). However, our key finding remains consistent in both cases: the layers where the ID reaches a local minimum correspond to representations where a semantic property of the data is most pronounced. Moreover, other important factors, such as the distinct training objectives (MLM in protein language models and next pixel prediction in iGPTs) and the different data modalities used for training, also influence the ID's ' characteristic shape.
Preliminary exploration of NLP representations. The NLP literature has previously discussed the idea that hidden representations the middle layers of large transformer models encode abstract information that can be leveraged to solve various tasks successfully. Syntactic information is more prominent the middle layers of deep transformers, as shown by syntactic tree depth reconstruction in [32]. However, the localization of semantic information has yielded contrasting results [33- 377 possibly due to the complexity of language data and the ongoing evolution of model size and architectures. These studies show that language data requires diverse tasks and probes for comprehensive understanding. In addition, there is a substantial distinction between pLMss and Large Langue Models (LLMs). While for pLMs, we almost reached an overparameterization regime on the UniRef dataset [38], this saturation level has not been observed in the context of language. Recently, started analyzing the geometry of hidden representation in the NLP domain with our methods. Our early analyses on GPT-2-XL on the SST dataset [39] did not reveal a second peak similar to the one in iGPT. With the release of the Llama models [40], we repeated the experiment on the Llama-v2 with 70 billion parameters (see Appendix, Sec. E). In this case, the evolution of the ID profile is more complex, showing three peaks and two local minima across the hidden layers. Remarkably, our key finding remains consistent: the highest overlap with class partition, determined by sentence sentiment, occurs in correspondence with the first local minimum. After the second peak, the II) profile shows a more complex behavior, which we are currently investigating. We will explore these aspects further in future work dedicated NLP tasks.
Acknowledgments and Disclosure of Funding
The authors acknowledge the AREA Science Park supercomputing platform ORFEO made available for conducting the research reported in this paper and the technical support of the Laboratory of Data Engineering staff. We thank Roshan Rao for kindly providing the weights of the ESB-2(650M) models at training checkpoints used to produce Fig.
(left).
A.A., A.C., and D. D. were supported by the project "Supporto alla diagnosi di malattie rare tramite I' l'intelligenza artificiale" CUP: F53C22001770002. A.A., A. C., and F.C. were supported by the European Union - NextGenerationEU within the project PNRR @ERRC"" IR0000028 Mission 4 Component 2 Investment 3.1 Action 3. L.V. was supported by the project PON "BIO Open Lab (BOL)- Rafforzamento del capitale umano" CUP J72F200009940077.

---

References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[2] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina N. Toutanova. Bert: Fre-training of deep bidirectional transformets for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1:4171-41881 2019.
[3] Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emer- gent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Science, 11744)):0044-33004 December 2020.
[4] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jun 2022.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[6] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (1(11))c201623111 2021.
[7] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Towards Cracking the Language of Lifes Code Through Self- Supervised Deep Learning and High Performance Computing. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
[8] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583-5555 2021.
[9] Yanrong Ji. Zhihan Zhou, Han Liu, and Ramana V Davuluri. DNABERT: pre-trained Bidi- rectional Encoder Representations from Transformers model for DNA-language genome. Bioinformatics, 37(15):2112-22
02 2021.
[10] Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML/20. JMLR.org, 2020.
[11] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. Proceedings of the International Conference on Learning Representations, 18-24 Jul 2021.
[12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 18-24 Jul 2021.
PMIR,
10

---

[13] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[14] Diego Doimo, Aldo Glielmo, Alessio Ansuini, and Alessandro Laio. Hierarchical nucleation in deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[15] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomic- level protein structure with language model. Science, 379(6637): (113)):111233111 123-1130, 2023.
[16] B. Scholkopf and A.J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Adaptive Computation and Machine Learning series. MIT Press, 2018.
[17] Hariharan Narayanan and Sanjoy Mitter. Sample complexity of testing the manifold hypothesis. Advances in neural information processing systems, 23, 2010.
[18) J T'eenbaum, DD Silva, and JA Langford. global geometric framework for nonlinear dimension- ality reduction [j]. Science, 290(5500):231-2233 2000.
[19] Daniele Granata and Vincenzo Carnevale. Accurate estimation of the intrinsic dimension using graph distances: Unraveling the geometric complexity of datasets. Scientific reports, 6(1):31377, 2016.
[20] R. Bennett. The intrinsic dimensionality of signal collections. IEEE Transactions on Information Theory, 1555)55 1969.
[21] Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 7, 2017.
[22] Emily Cheng, Corentin Kervadec, and Marco Baroni. Bridging information- and geometric compression in language models. arXiv preprint arXiv:2310.13622 2023.
[23] Lorenzo Basile, Nikos Karantzas, Alberto D' D'Onofrio, Luca Bortolussi, Alex Rodriguez, and Fabio Anselmi. Relating implicit bias and adversarial attacks through intrinsic dimension. arXiv preprint arXiv:2305. 15203, 2023.
[24] Henry Kvinge, Davis Brown, and Charles Godfrey. Exploring the representation manifolds of stable diffusion through the lens of intrinsic dimension. arXiv preprint arXiv:2302.09301, 2023.
[25] Aldo Glielmo, luri Macocco, Diego Doimo, Matteo Carli, Claudio Zeni, Romina Wild, Maria d' d'Errico, Alex Rodriguez, and Alessandro Laio. Dadapy: Distance-based analysis of data- manifolds in python. Patterns, page 100589, 2022.
[26] Mohammed AlQuraishi. ProteinNet: standardized data set for machine learning of protein structure. BMC Bioinformatics, 20, 2019.
[27] John-Mare Chandonia, Lindsey Guan, Shiangyi Lin, Changhua Yu, Naomi Fox, and Steven Brenner. SCOPe: improvements to the structural classification of proteins extended database to facilitate variant interpretation and machine learning. Nucleic Acids Research, 50(DI): D559, 12 2021.
[28] Johannes Söding and Michael Remmert. Protein sequence comparison and fold recognition: progress and 2011.
benchmarking. Current opinion in structural biology, 21 3:404-41 411,
[29] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[30] Elena Facco, Andrea Pagnani, Flena Tea Russo, and Alessandro Laio. The intrinsic dimension of protein sequence evolution. PLoS computational biology, 15(4):e1006767, 2019.
11

---

[31] N.S. Detlefsen, S. Hauberg, and W. Boomsma. Learning meaningful representations of protein sequences. Nature Communications, 13, 2022.
[32] John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics.
[33] Yonatan Belinkov, Lluís Mårquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Evaluating layers of representation neural machine translation on part-of-speech and semantic tagging tasks. In Proceedings of the Fighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1-10, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing.
[34] T. Blevins, O. Levy, and L.. Zettlemoyer. Deep man encode soft hierarchical syntax. Proceedings of the 56th Association for Computational Linguistics (ACL), 2018.
[35] Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 1073- 1094, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics.
[36] Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019. Association for Computational Linguisties.
pages
[37] Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866, 2020.
[38] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi: Tom Sereu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
[39] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cs/0506075, 2005.
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini 23Xi:22307.09288 2023.
[41] D. Moltchanov. Distance distributions in random networks. Ad Hoc Networks, 10(6): 146 1166,
2012.
[42] Francesco Denti, Diego Doimo, Alessandro Laio, and Antonietta Mira. The generalized ratios intrinsic dimension estimator. Scientific Reports, 12(1):20005, 2022.
[43] Alex Rodriguez, Maria d' Errico, Flena Facco, and Alessandro Laio. Computing the free energy without collective variables. Journal of Chemical Theory and Computation, 1463)11206-1255, 2018. PMID: 29401379.
[44] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA Transformer. Proceedings of the 38th International Conference on Machine Learning, 139:8844-8855 18-24 Jul 2021.
[45] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems, 34, 2021.
[46] The UniProt Consortium. UniProt: the universal protein knowledgebase in 2021. Nucleic Acids Research, 40DD1))440--489 11 2020.
12

---

[47] Konstantin Schütze. Michael Heinzinger, Martin Steinegger, and Burkhard Rost. Nearest neigh- bor search on embeddings rapidly identifies distant protein relations. Frontiers in Bioinformatics. 2, 2022.
13

---

Appendix
A Architecture details
Table 1: Characteristics of the models employed for extracting representations.
Model
#Blocks
Emb. dim.
#Heads
#Params
Dataset
Reference
ESM-2 (35M)
12
480
20
35M
UR50/D
[15]
ESM-2 (650M)
33
1280
20
650M
URS0/D
[15]
ESM-2 (3B)
36
2560
40
3B
UR50/D
[15]
iGPT-S(76M)
24
512
8
76M
ImageNet
[10]
iGPT-M(455M)
36
1024
455M
ImageNet
[10]
iGPT-L((.4B)
48
1536
16
1.4B
ImageNet
[10]
Transformer models for protein sequences. The input data points, corresponding to proteins, are vuriable-length sequences of / letters 01 a2.... corresponding to amino acids. Each token is encoded by an embedding layer into a vector of size that the generic protein is represented
.....,af aj chosen from an alphabet of na(>> 20) tokens
fo(x) of size d. A model with B blocks transforms data point IR into sequence of representations: fo(æ) fI
matrix
- fz) - four (z). where fi, l..... B stands for the self celf-attention module the 7th block, and the final LM-head fout is learned projection onto dimension x na- The size of each hidden layer does not change across the model and is equal to / x d; therefore, the action of the model is a sequence of mappings 1xxd + glxd xd For each layer i, we choose to perform global (f=((((()) since this reduction retrieves
average pooling across the row dimension fz) - sufficient biological information to solve, directly or possibly after finetuning, homology, structural and evolutionary tasks. For given pL. the action of the network dataset of N proteins can thus be described by B + collections of N vectors in Rd: these are the data representations that we investigate.
Transformer models for image processing. The structure of the Image GPT (iGPT) transformers very similar to that pLMs. Due the high memory footprint required by the attention layers, Chen et al. [10] reduce the image from the standard ImageNet size (r = 224²) to 32 (S), 642 (L) and the three color channels are encoded in "embedding axis" with size din = 512. In practice, the R3 color space in which each pixel is represented by a triplet of real numbers RR G, B) quantized with k-means clustering (A: 512), and each pixel is described by the discrete "code" of the cluster where it belongs. An input image is then represented by data point R where the 1 pixels are placed along the sequence axis raster order and din encodes the color of each pixel. Like in pLMs, an input data point, after the embedding layers, is processed by stack of attention blocks that produce sequence of data representations of identical size Riixd The final head sout projects the output of the last attention block to R Rixdi,, which encodes at each sequence position the conditional distribution
48 (M),
p(x;) p( Jo, ...::-1)
Once the network is trained, an image be generated by sampling from p(z.) pixel at time.
14

---

B Details about the intrinsic dimension estimation.
Facco et al. [21] showed that the ID can be estimated using only the distances Th ri between each point and its first two nearest neighbors. Under the assumption of the Poisson point process in space [41], the ratios μ4; 'ia/rs, between the distances to the second and first nearest neighbor follow a Pareto distribution p(us|d) = djs, d Assuming that the dataset likelihood factorizes over the N single-point likelihoods, the IID can be easily inferred given the empirical values of the μ4; as the p(p/ (d) depend only on d. Facco et al. propose to estimate the ID from the cumulative distribution function F F(n) = 1 μ with a linear regression fit. Indeed, the ID the slope of the line through the origin obtained regressing (og(Ha(()) against - log((1 - pemp (Mar())) where (a(i) are the μ4; sorted in ascending order and Femp (Ho(s)) == tV is the value of empirical cumulative distribution function at (a(()) [21].
In practice, due to its very local nature, the TwoNN estimator can be affected by noise, which typically leads to an overestimation of the ID. To identify the most plausible value the data intrinsic dimension, Facco al. apply the TwoNN estimator on the full data set and then on random subsets of decreasing size: n [N, N/4, ...]. The relevant data ID is chosen as the value of d where the graph d(()) exhibits plateau, and the ID less dependent on the scale. Recently, Denti et. al [42] proposed better strategy to perform a multiscale analysis of the ID considering nearest neighbors of higher order instead of decimating the data set. This approach reduces the variance of the likelihood, giving more stable estimates.
These methods gave similar results in the experiments of this work: the ID estimated with the TwoNN was less scale-dependent across dataset sizes between [N/2, N/8). Thus, in the main text, we report the TwoNN estimates the dataset decimated by factor of 4.
Validity of the local density assumption. To quantitatively validate the constant density assump- tion, we employed the Point Adaptive kNN (PAk) method introduced by Rodriguez et al. [43]. PAk determines the extent of the neighborhood over which the probability density can be considered constant for each data point, subject to specified confidence level. Applying PAk (as implemented in DADApy [25]) to our dataset revealed that, on average, the density can be considered constant within the first 6 neighboring data points. Our study measures the ID using the distances between data point and its first two nearest neighbors. This analysis allows the conclusion that, at this scale, the assumption of local density holds.
15

---

C Intrinsic dimension and overlap profiles on other protein language models.
25
20
15
10
0.2
0.4
0.6
0.8
relative depth
Figure S1: The ID shape for different pLMs architectures. The latest developments in the application of pre-trained pLMs for the solution of diverse biological tasks have been mainly fuelled by two families of models: Prot- Prot-Trans [7] and Evolutionary Scale Modelling (ESM) [6, 38, 44, 45]. During the last years pLMs with different architectures, number of parameters, and embedding sizes have been trained on several datasets obtained starting from the UniProt [46] database. The profiles in the figure complement the analysis done in Sec. 3.1, showing the ID curve of ProteinNet on pLMs trained on UniRef50 (ESMIb and ProIT5-XL-U50, red and orange) UniRe190 (ESMIv, green), Uniref100 (ProtBert, purple). Despite the significant differences of the PLMs considered in the analysis, the consistency of the three-phased behavior of the ID remains: an initial peak is followed by a plateau where the ID assumes low values, and the ID grows again values close the one measured after the positional embedding.
ProUTS-XL-US0
1.0
0.8
0.0
0.0
0.2
0.4
0.6
0.8
1.0
relative depth
Figure S2: Nearest neighbor search in plateau layers improves identification of protein relations. It was recently shown in [47] that the first nearest neighbor searches for remote homologous protein domains based on the last hidden layer representations of the ProrT5-XL-US0 pLM outperform state-of-the-art methods based sequence similarity. Adapting the approach of Sec. 3.2, mimic the experiment performed in Sec. 2 of [47] by considering protein domains SCOPe belonging to superfamily with at least sequences, setting the number of neighbors to k 1. Considering representations in the plateau layer improves the accuracy of the 1-kNN homology search. In observe improvement of ~ 6% performing the search on plateau
particular, in the figure, layer instead of the last layer before the output. Importantly, the performance gain 6% is obtained without any further training.
16

---

D Impact of the implementation details and hyperparameter choices.
Cifar10 WR28-8
rog_pooling
full id
25
20
15
0.0 6.2 0.4
0.6
0.8 1.4 0.0
0.2
0.4
2.6
0.8 1.0
relative depth
relative depth
Figure S3: Impact the average pooling on the intrinsic dimension profiles. To study how the average pooling affects the ID profile, tested the representations of CIFARIO in the Wide- ResNet28-8 8 model. The left panel shows the intrinsic dimensionality (ID) computed in the full feature space, while the right panel displays the ID after applying the global average pooling along the spatial dimensions (width and height). Notably, we observe a qualitative consistency in the shape of the ID profiles in both cases, they conform to the typical bell-shaped curve characteristic of CNN architectures (see [13]). From quantitative perspective, the ID profile after average pooling exhibits a downward shift as a consequence of the averaging procedure, particularly pronounced at the initial stages of the architecture. This effect is likely due to the low number of channels after the first block (16) compared to the later blocks (128, 256, and 512, respectively). Nevertheless, can be confident about the qualitative robustness of the profiles as long as the number of "channels" (where channels are interpreted as embedding dimension) significantly surpasses the in large transformer models. Indeed, in this case, the number of "channels" is substantially higher than in early CNN layers (ranging from 512 several thousand in modern large language models) and remains constant throughout the architecture.
iGPT-L
post_In _st__n
post_attn
post In
post_attn
0.40
0.25
0.20
0.15
5
0.00
relative depth
relative depth
Figure S4: Intrinsic dimension and neighborhood overlap after the self-attention blocks. The figure compares the ID (left) and the overlap with the ground truth labels (xt right) of the ImageNet representations after the first normalization layer with those of the representations after the attention maps of each self-attention block for the iGPT Large model. The ID and profiles consistent, indicating the robustness of our analysis with respect to the layer choice.
17

---

ESM-2(650M)
K-1
km
1,0
1.0
0.8
0.8
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
relative depth
relative depth
IGPT-L
k
[m]
km5
ki 20
2-
Cm2
k=
k-
1.0
0.5
0.8
0.4
0.2
0.1
0.0
1.0 0.0
0.2
0.4
0.8
0.0
0.4
0.6
0.8
relative depth
relative depth
Figure S5: Neighborhood overlap profiles are robust w.rt. the choice of the neighborhood size k. Overlap profiles in ESM-2 (650M, top) and iGPT Large (bottom) varying the neighborhood size k in the range (1,50) 50}. The curves the overlap between consecutive layers X 1.1+1 (left) and with the ground truth classes (right) are qualitatively unchanged. When considering the ESM-2 model (top), Lgt I++1 and X when k 50 is the consequence of certain superfamilies having the lower values of X fewer than 50 elements.
18

---

E Preliminary analysis of the intrinsic dimension and overlap in LLama2
llama2-00B
50
39
30
0.4
0.8
55t
0.100
0,650
6.625
0,000
0.575
0.325
0.0
0.4
0.8
relative depth
Figure S6: Intrinsic dimension and neighborhood overlap of sst in LLama2-70b. [Top] The intrinsic dimension (ID) of representations in hidden layers of Llama-v2 for the Stanford Sentiment Treebank (SST) dataset, plotted against block number normalized by the total number of blocks (relative depth). [Bottom] Overlap with ground truth labels, i.e. sentiment of the sentences, each self-attention block, plotted against relative depth. In correspondence with the first local minimum of the ID profile, at 0.3 relative depth, the overlap with the class partition in the Bottom panel is the highest.
19


Reviews:
[{"review_id": "6quwbV0TnS", "reviewer": "Reviewer_K92u", "summary": "The paper studies the hidden representation of pretrained transformers via the ID (intrinsic dimension) on protein language tasks and image reconstruction tasks . The papers show that on protein LM, from input to output layer, the ID first increase to a peak, then decrease to an elbow, and finally increase to near its ID value at input layer; this is robust from small to largest models. For image reconstruct, the picture is quite different; the input & output layers have the smallest ID and there two peaks near the input and output layers, respectively. The results on iGPT is less robust compare to protein LM; (e.g., small iGPT does not seem to have two peaks). \n\n\nOverall, I think the paper is well written, presentations and experimental setup are clean, the results are interesting. However, I am not very familiar with the field and could not judge the novelty of the paper. \n\n\n\n", "questions": "See above. ", "limitations": "Have several discussion about further extension of the current approach. ", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A well written paper; structure is clean; presentation is good; authors spent decent amount of efforts to introduce / motivate / explain key concepts using in the paper, e.g. ID etc. \n\nThe results seem interesting even to folks who are not working this area.  \n\n\n\n\n\n\n", "weaknesses": "The results on iGPT is less robust (compare to protein LM). In particular, for the small model (I don’t see two peaks). Naively, I also expect the shape of ID of iGPT to be similar to pLM, encode (decreasing ID, smaller than input) and decode (increasing ID), just like the elbow in the pLM. Is there an explanation? \n\nWhy not also including a third task: NLP  (casual language model) in the main text?  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer K92u for reviewing our manuscript.\n\n**Weaknesses/Questions**\n\n*The results on iGPT is less robust (compare to protein LM). In particular, for the small model (I don’t see two peaks).*\n\nWe agree that in the small model the second peak is absent. However, the small model is significantly less accurate than the larger model, both in terms of validation loss (refer to Fig. 3, [1]) and performance in image classification tasks, as evaluated through linear probes (see Fig. 3, [1]) and neighborhood overlap (Fig. 4, right panel, in our work). \nWe emphasize that also in the case of protein language models (pLMs), the smallest model  (ESM2-8M, see attached PDF Fig. 1)  does not exhibit a clear three-phased ID profile (no peak and plateau), and the three-phased behavior becomes gradually more pronounced when increasing the size of the models, similarly to the case of iGPT models. Indeed large transformer models can develop qualitative changes in their performance and internal workings as their size is scaled up, so that a feature that is absent in smaller models can suddenly appear as the number of parameters is increased [2].\n\n*Naively, I also expect the shape of ID of iGPT to be similar to pLM, encode (decreasing ID, smaller than input) and decode (increasing ID), just like the elbow in the pLM. Is there an explanation?*\n\nWe kindly refer the reviewer to the global response about the difference of ID profile between iGPT and ESM models.\n\n*Why not also including a third task: NLP (casual language model) in the main text?*\n\nWe kindly refer the reviewer to the global response about our preliminary investigation of NLP.\n\n**References**\n\n[1] Chen et al., “Generative Pretraining From Pixels”, 37th International Conference on Machine Learning, 2020 \n\n[2] Wei et al. “Emergent Abilities of Large Language Models”, Transactions on Machine Learning Research, 2022\n\n\n"}, {"review_id": "v46OASvWyg", "reviewer": "Reviewer_4o8z", "summary": "This work focuses on characterizing the geometrical and statistical properties of data representations across the hidden layers of large transformer models. Specifically, it demonstrates the similarity in the evolution of geometric properties, such as the intrinsic dimension, between image-based and protein-based transformers. Additionally, the work proposed an intuitive, unsupervised strategy to identify the most semantically rich representations of transformers. The effectiveness of this strategy is demonstrated by the SOTA performance achieved by leveraging previous work and replacing the last layer with the layer identified in this study. ", "questions": "- In lines 131-132, the paper mentions a method to compress the resolution and color channels of images to address memory constraints. However, it raises the question of why a methodology similar to the one described in [1] was not adopted. In [1], a \"first-stage\" autoencoder was trained specifically to compress the images into lower-dimensional latent representations while maintaining perceptual equivalence (and kept frozen in subsequent stages). Is there a specific reason why this approach was not considered in the current work?\n\n- In line 286, it is reported that the plateau extends beyond half of the layers. This raises the question of whether this observation could be an indication that the model is over-parameterized or, at the very least, has more layers than necessary.  Further investigation or discussion could shed light on this question and provide valuable insights in the minimum network complexity to produce representations with the same semantic information.\n\n\n[1] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. “High-Resolution Image Synthesis with Latent Diffusion Models.”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022\n", "limitations": "Although not extensively, Section 4 explores the limitations and potential areas for future research.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This work presents a novel strategy to select the layers that produce the most semantic representation to be used in downstream tasks, opening many possible applications. Moreover, the method seems to be general and could be exploited in other architectures where the layer choice is currently arbitrary, such as classifiers.\n  \n- This work demonstrates state-of-the-art results in identifying protein relations by leveraging existing methods. It simply involves swapping the previously employed last layer with the layers that maximizes the semantic content identified through the strategy proposed in this work (Figure 5, right).\n\n- The writing is extremely clear and easy to follow, presenting intuitive ideas, a solid experimental setup and convincing results.\n\n- This work does not involve training models but instead analyzes publicly available models, completely avoiding any possible bias that could have been introduced in the training process.\n\n- Among the others, the insight that large transformers behave essentially like sophisticated autoencoders (lines 337-338) is insightful and may serve as a source of ideas for future research.", "weaknesses": "This work presents a novel idea with exceptional clarity in its writing, accompanied by a robust experimental setup that yields convincing results. While no major weaknesses were identified, a more comprehensive discussion on the current limitations of the study would enhance its overall quality even further.\n\nMinor:\n\n- In line 36 (and others), the paper mentions the term \"semantically rich representation.\" However, it is important to clarify how this term is defined. It can be argued that the richness of semantic content in a representation is dependent on the specific downstream task being addressed. In other words, a representation can be considered more or less semantically rich based on its effectiveness in a particular downstream task.\n\n- To enhance readability, it would be beneficial to include the ID (or at least the minimum ID) directly in the plot in Figure 4. This additional information (even though repeated from Figure 3), would facilitate the reader to compare the peak categorical information with the ID minimum.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to reviewer 4o8z for their appreciation of our work, for carefully reviewing our manuscript, and for the stimulating comments.\n\n**Weaknesses**\n\n*In line 36 (and others), the paper mentions the term \"semantically rich representation.\" However, it is important to clarify how this term is defined. It can be argued that the richness of semantic content in a representation is dependent on the specific downstream task being addressed. In other words, a representation can be considered more or less semantically rich based on its effectiveness in a particular downstream task.*\n\nWe agree with this observation, and we will clarify that the expression \"semantically rich representation\" is relative to a specific task (typically of classification).\n\n*To enhance readability, it would be beneficial to include the ID (or at least the minimum ID) directly in the plot in Figure 4.*\n\nWe thank the reviewer for this suggestion. We will consider adding an explicit indication of the minimum ID in the plot of Fig. 4 in the revised version of the manuscript.\n\n**Questions**\n\n*In lines 131-132, the paper mentions a method to compress the resolution and color channels of images to address memory constraints. However, it raises the question of why a methodology similar to the one described in [1] was not adopted.*\n\nThe resolution and color channel compression mentioned in Line 131 of the Manuscript are part of the original encoding procedure of the iGPT models developed by [2], and thus they are not our decision. Our work is just focused on the analyses of transformer models developed by other groups and trained by self-supervision. In the discussion section, we will highlight that an examination of the approach by [1]  is left for future research.\n\n*In line 286, it is reported that the plateau extends beyond half of the layers. This raises the question of whether this observation could be an indication that the model is over-parameterized or, at the very least, has more layers than necessary.*\n\nThis is a very interesting and pertinent point. Indeed, we recently started exploring the possibility of eliminating or replacing by a suitable transformation certain layers in this part of the network without compromising the performance. We will highlight this as a potential avenue for future research in the discussion section.\n\n**References**\n\n[1] Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022\n\n[2] Chen et al., “Generative Pretraining From Pixels”, 37th International Conference on Machine Learning, 2020\n"}, {"review_id": "idhKekae4h", "reviewer": "Reviewer_kxbE", "summary": "This paper presents an analysis of the internal representations of transformers trained with self-supervised learning, such as masked language modeling, from two perspectives: intrinsic dimension and adjacency structure. Experiments on two datasets - protein sequences and image data - revealed that the intrinsic dimension within the transformer has two peaks, one in the early layers and another in the latter layers. This result is qualitatively different from previous convolutional networks trained with supervised learning. Notably, the intermediate layer, where the intrinsic dimension is minimized, is the most suitable for categorical discrimination in downstream tasks.", "questions": "I have several questions for the authors:\n\n1. In interpreting Figure 1, the main text mentions that the changes in the internal dimension qualitatively match the experiments for the two domains. However, there seems to be a qualitative difference between domains concerning whether there is concentration or expansion near the final layers in the latter half. What could be the reasons for such a difference?\n2. Regarding the analysis of the overlap of the adjacency structure, the paper mentioned that the authors used $k=10$ and $k=1$ in the protein domain experiments, and $k=30$ in the image domain experiments. Generally, the experimental results would depend on this hyperparameter. How robust are the authors' claims with respect to these hyperparameters, and what is the rationale or justification for the chosen hyperparameter values?\n3. This paper mentioned that the authors used the GAP type averaging procedure along the token direction when analyzing the internal representation of the transformer model. While this is not a problem when comparing results within the same model architecture, it seems caution is needed when comparing the results with other models. For instance, when comparing the results with convolution-type neural networks, how can we guarantee the qualitative robustness of the statements depending on whether this averaging operation is performed or not?", "limitations": "Given that this paper is not proposing a new method but focusing on analyzing the internal representation of existing methods, it may not strictly apply to a discussion on limitations. The estimation of the internal representation is based on a method proposed in previous research, and the applicability of this method has been sufficiently demonstrated. However, as mentioned in the Weakness and Question sections, there are concerns about the comprehensiveness of the experiments in this paper. Improvements in these points would lend greater significance to the authors' claims.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper presents a notable analysis of the intrinsic dimension of transformer models trained with self-supervised learning. While many approaches have been used to analyze the internal dimension of deep neural networks, most have focused on convolution-based networks. The discovery of two peaks in the internal dimension is intriguing, offering insightful contributions to the community. The results are clearly presented, and the paper is well-structured, making it easy for readers to follow the logical development. Of particular interest is the finding that such representations can spontaneously appear in models trained with masked modeling, even without an explicit bottleneck structure in the model.\n\nHowever, this interesting result also raises many considerations and discussions. There are points in the current paper where sufficient evidence to support the authors' claims is lacking and further room for discussion remains. These points will be discussed in detail below.", "weaknesses": "The paper's experimental comprehensiveness raises two issues. The first issue revolves around the focus on internal representations of transformer models trained with self-supervised learning, with the assertion that the evolution of the internal dimension displays two peaks. However, it appears that sufficient experiments have not been performed to identify the causes of this occurrence definitively. An ablation study would be incredibly beneficial in distinguishing the elements that stem from the model architecture (transformer) and those arising from the training protocol (masked/autoregressive modeling).\n\nSecondly, the paper state in the introduction that the adjacency structure is “rearranged” at the peak of the representation’s internal dimension, regardless of the dataset domain. Yet, upon the qualitative comparison of Figures 1 and 2, it is challenging to confirm a consistent change in the internal dimension and the overlap of the adjacency structure between layers in the image domain experiments. Therefore, the current portrayal of the contributions gives the impression of over-claiming. As asserting a clear correlation in qualitative behavior is difficult, a comparison with some control conditions should be made. Additionally, it would be desirable to have a discussion concerning the influence of differences in the data domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer kxbE for carefully reviewing our manuscript and for the constructive comments.\n\n**Weaknesses**\n\n*An ablation study would be incredibly beneficial in distinguishing the elements that stem from the model architecture (transformer) and those arising from the training protocol (masked/autoregressive modeling).*\n\nA specific ablation study is described in Appendix A.3.4, Fig. 7, where we examined the vision transformers (ViT) trained for an image classification task. In this case, the ID profile exhibits a much less prominent second peak followed by a gradual decrease of ID below initial values, consistent with the established pattern typically observed in image classifiers [1]. This example shows that the shape of the ID profiles is influenced by the training protocol and is not an inherent characteristic of the transformer architecture itself. We are investigating how-fine tuning affects the models we analyzed in the manuscript. However, it is crucial to emphasize that the primary focus of this work is on studying ID profiles in models trained through self-supervision. More importantly,  we aimed to establish connections between the low-dimensional representations and the layers where the semantic properties of the data are better expressed.\n\n*[...] upon the qualitative comparison of Figures 1 and 2, it is challenging to confirm a consistent change in the internal dimension and the overlap of the adjacency structure between layers [...] the current portrayal of the contributions gives the impression of over-claiming. [...] Additionally, it would be desirable to have a discussion concerning the influence of differences in the data domain.*\n\nWe would like to remark that the value of chi reported in Fig. 2 of the manuscript refers to the overlap between consecutive layers. The average value of chi in the first part of the network is of order 0.5 in the case of protein sequences and of order 0.7 in the case of images. This implies that after ~10 layers the overlap with the input is of 0.1% in the case of protein sequences and of  ~3% in the case of images. We believe that this can be called a significant rearrangement of the neighborhood, at least in the case of proteins. In order to avoid over-claiming we removed the word “profoundly” before “rearranged” in the introduction and in the text, since in the case of images an overlap of ~ 3% after the peak might indicate the survival of some faint memory of the input. We will also state explicitly that the rearrangement is more significant in the case of protein sequences highlighting the differences in the two data domains in this respect.\n\n**Questions**\n\n*However, there seems to be a qualitative difference between domains concerning whether there is concentration or expansion near the final layers in the latter half. What could be the reasons for such a difference?*\n\nWe kindly refer the reviewer to the global response about the qualitative difference of ID profile between iGPT and ESM models.\n\n*How robust are the authors' claims with respect to these hyperparameters [k=1, k=30], and what is the rationale or justification for the chosen hyperparameter values?*\n\nWe chose k=1 for pLMs to show how nearest neighbor search in plateau layers improves identifying protein relations. Other k values (k=10, k=30) do not affect the observations presented in our manuscript. We addressed the robustness of our findings concerning the hyperparameter k in Appendix A.3.4. In particular, Fig. 6  shows that the neighborhood overlap curves remain essentially unchanged for k<50  both between successive layers (Left) and with the ground truth labeling of the data (Right).\n\n*This paper mentioned that the authors used the GAP type averaging procedure [...] when comparing the results with convolution-type neural networks, how can we guarantee the qualitative robustness of the statements depending on whether this averaging operation is performed or not?*\n\nWe emphasize that the global average pooling (GAP) approach was not intended for comparing our results regarding the geometric properties of transformer representations with the characteristics of hidden layers in convolutional architectures. \nHowever, to directly address the concern raised by the reviewer regarding the robustness of the GAP procedure in convolutional neural networks (CNNs), we conducted a test on the representations of CIFAR10 in the Wide-ResNet28-8 model [2]. We describe the results in Fig. 4 of the attached PDF. The left panel shows the intrinsic dimensionality (ID) computed in the full feature space, while the right panel displays the ID after applying GAP along the spatial dimensions (width and height). Notably, we observe a qualitative consistency in the shape of the ID profiles in both cases, as they conform to the typical bell-shaped curve characteristic of CNN architectures [1]. From a quantitative perspective, the ID profile after GAP exhibits a downward shift as a consequence of the averaging procedure, particularly pronounced at the initial stages of the architecture. This effect is likely due to the low number of channels after the first block (16) compared to the later blocks (128, 256, 512 respectively). Nevertheless, we can be cautiously confident about the qualitative robustness of the profiles as long as the number of “channels” (where channels are interpreted as embedding dimension) significantly surpasses the ID as in large transformer models. Indeed, in this case the number of \"channels\" is substantially higher than in early CNN layers (ranging from 512 to several thousands in modern large language models), and remains constant throughout the architecture.\n\n[1] Ansuini et al., “Intrinsic dimension of data representations in deep neural networks”, 32nd Conference on Neural Information Processing Systems, 2018\n\n[2] S. Zagoruyko and N. Komodakis, “Wide Residual Networks”, Proceedings of the British Machine Vision Conference 2016"}, {"review_id": "YT9uAg7oHX", "reviewer": "Reviewer_wEGg", "summary": "This work investigates the geometry of hidden representations of transformer models trained via a self-supervised task on either amino acid prediction in proteins or pixel prediction in images. The work uses two tools to understand this geometry: intrinsic dimension (ID) (estimated via the TwoNN algorithm) and neighborhood overlap. The paper shows that as data passes through the layers of a transformer, both the ID and neighborhood overlap change in characteristic ways that the work connects with the extent to which the representation is organized by the semantic content in the data. The paper validates this claim by looking at the extent to which each data point shares neighbors that belong to the same semantic class, showing that this peaks at layers when ID is low and changes in neighborhood overlap is also low. The work then speculates that these observations could provide a way of identifying the best representations to use for downstream tasks.", "questions": "- Given the success of transformers trained with self-supervised learning in the NLP domain, I was curious why the authors did not study this setting? \n- Line 76: Is the assumption of locally constant density really “mild”? This reviewer wonders if image datasets, which have moderately high intrinsic dimension ([1] estimates ImageNet as having intrinsic dimension between 26 and 43) might be fairly sparsely sampled even when the dataset is large.\n- What happens to the results when $k$ is varied (rather than being fixed at 30)?\n- The paper says that hidden representations are extracted after the first normalization layer of the attention blocks. Do you have a sense of whether results change if you use representations from other parts of attention blocks?\n- The results show the intrinsic dimension of the data increasing at times between layers. We know that mathematically, maps $f: M \\rightarrow \\mathbb{R}^n$ that map a $k$-dimensional manifold $M$ into $\\mathbb{R}^n$ such that $\\dim(f(M)) > k$ exist (e.g., space filling curves) but it is this reviewer's understanding that the space of such $f$ has measure zero with respect to many reasonable probability measures on the space of such maps. Given this, is it really possible that the intrinsic dimension of the representations is increasing? Or is their some other change that is changing the ID estimation?\n\n[1] Pope, Phillip, et al. \"The intrinsic dimension of images and its impact on learning.\" arXiv preprint arXiv:2104.08894 (2021).\n", "limitations": "This reviewer believes that some limitations could have been discussed, including:\n- The use of intrinsic dimension estimators, which can provide misleading feedback in certain situations.\n- A limited number of different model types and datasets.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- While there is a rich literature exploring hidden representations of deep learning models, most works continue to focus on CNNs or MLPs with supervised training on medium sized datasets. Given the growing importance of transformers to NLP and vision and the increasing use of self-supervision for large-scale training, there is a need for works that explore how these approaches impact a model’s hidden representations. Thus, this is a welcome work that will doubtless be of value to researchers training large transformers via self-supervision. \n- The paper contains careful analysis of the experiments (as opposed to other works which all too frequently just list summary statistics). The conclusions which are reached are all fairly-well supported within the scope of the experiments that were performed. Crucially, the two metrics that are used, intrinsic dimension and neighborhood overlap, reinforce each other’s conclusions. This increases the believability of the results significantly.\n- The paper is able to suggest some practical value (identifying layers of the transformer that capture the most semantic content) in their scientific observations (patterns in representation geometry in large-transformers), helping to connect practice with theory.\n", "weaknesses": "- The experiments in this work all use transformer models trained via self-supervised tasks based around data reconstruction (as opposed, for example, to a contrastive learning task). While most of the phenomena observed in this paper is attributed to the “large transformer” architecture, this reviewer wonders if some of the conclusions are also contingent on the self-supervision task. For example, does one see similar behavior in a large vision transformer that has been trained in a supervised manner. Disentangling which of the conclusions are due to architecture and which are due to training method would make the work significantly stronger.\n- Many of the claims in the work rely on comparison between the shapes of curves which plot intrinsic dimension of data between layers. While the claims seem mostly reasonable given the figures, it would make the work stronger if more quantitative measures were used to, for example, compare ID curves. This sort of automation might also make it easier to compare against a broader range of models and datasets. While this reviewer thought that the use of both protein and image datasets was a strength of this work, comparing against other types of transformer architectures and datasets would reinforce the conclusions.\n- The main content of the paper consists of discussion of several figures. This reviewer felt that the way this discussion was written/organized, it was easy for the reader to get disoriented and forget the main points already established. Possibly making the discussion more concise or better highlighting the main takeaways from each section would help the reader to mentally organize the primary findings of the work. Being more succinct might also allow further experiments to be included.\n\nNitpicks\n\n- The abstract describes a transformer as being composed of a “sequence of functionally identical transformations”. To this reviewer, functionally identical transformations would be transformations that behave the same way on the level of functions (for the same inputs, they give the same outputs), but may be parametrized in different ways. The layers of a transformer are instead transformations that all belong to the same functional family, though each layer is generally a different function.\n- Some sentences in this work have an over-abundance of commas. For example, “In this work, we systematically investigate, in some self-supervised models, fundamental geometric properties of representations, such as their intrinsic dimension (ID) and neighbor composition.” To improve the flow of the paper, it would be good to find such sentences and remove some of the commas.\n- Line 66: “…where the annotation is very scarce” -> “…where the annotations are very scarce.”\n- Reading this work would have been easier if the figures were closer to their corresponding analysis. As it is, the reader must flip between pages frequently to validate claims.\n- Line 110: “More in detail,” -> “In more detail,”.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer wEGg for carefully reading our manuscript for providing several points of discussion that we address below.\n\n**Weaknesses**\n\n*[...] Disentangling which of the conclusions are due to architecture and which are due to training method would make the work significantly stronger.*\n\nWe agree with the referee on this point. We will stress further in the final version of the manuscript that our analysis is specifically focused on transformer models trained by self-supervised reconstruction tasks (such as MLM or LM). We address the critical role of self-supervision in Appendix A.3.4: we show that self-supervised pre-training is crucial for the emergence of three-phased behavior by comparing our results with vision transformers trained for image classification. In that case, we can observe a much less pronounced second peak and a decrease of ID in the last layers to values lower than the first embedding. \n\n*[...] it would make the work stronger if more quantitative measures were used to, for example, compare ID curves.*\n\nIt is true that we based our analysis of ID curves on the inspection of the figures: since the domain is one-dimensional this intuitive approach is particularly effective in this case. However, since the ID is measured on a small number of layers the local minimizers of the ID can be found easily by a brute-force search approach as well.\n\n*[...] comparing against other types of transformer architectures and datasets would reinforce the conclusions.*\n\nWe kindly refer the reviewer to our response to Question 1.\n\n*This reviewer felt that the way this discussion was written/organized, it was easy for the reader to get disoriented and forget the main points already established.*\n\nWe will take into consideration this advice for the drafting of the final version of the paper.\n\n**Questions**\n\n*I was curious why the authors did not study this (NLP) setting?*\n\nWe kindly refer the reviewer to the global response about our preliminary investigation of NLP.\n\n*Line 76: Is the assumption of locally constant density really “mild”?*\n\nThe reviewer is correct in highlighting potential issues in assuming locally constant density for datasets with high ID. To quantitatively validate this assumption, we employed the Point Adaptive kNN (PAk) method introduced by [2]. PAk determines, for each individual data point, the extent of the neighborhood over which the probability density can be considered constant, subject to a specified level of confidence. Applying PAk (as implemented in [3]) to our dataset  revealed that, on average, the density can be considered constant within the first 6 neighboring data points. In our study, the ID is measured using the distances between a data point and its first two nearest neighbors. This analysis allows concluding that, at this scale the assumption of local density holds. \n\n*What happens to the results when k is varied [...]?*\n\nWe addressed the robustness of our findings concerning the hyperparameter k in Appendix A.3.4 (see Fig 6). Fig. 6 shows that the neighborhood overlap curves remain essentially unchanged for k<50  both between successive layers (Left) and with the ground truth labeling of the data (Right).\n\n*Do you have a sense of whether results change if you use representations from other parts of attention blocks?*\n\nIn Fig.2 of the attached PDF we extended the analysis of the ID and the overlap with the ground truth labels ($\\chi^{l, gt}$) on the representations after the first normalization layer and after the attention maps of each self-attention block for the iGPT model on ImageNet. The ID and $\\chi^{l, gt}$ profiles are consistent with those shown in the manuscript which are extracted before each attention block, indicating the robustness of our analysis with respect to the layer choice. Due to time constraints in preparing this response, we perform the analysis only for iGPT-large. Nevertheless, we are confident that the observed trends would also hold true for smaller models. \n\n*[...] is it really possible that the intrinsic dimension of the representations is increasing? Or is there some other change that is changing the ID estimation?*\n\nThe reviewer is indeed right in saying that from a mathematical standpoint the set of continuous functions $f: R^{d}\\to R^{d}$ mapping a k-dimensional manifold $M\\subset R^{d}$ onto a set that fills a manifold $M’$ with $dim(M’)>k$ is of measure zero. \nHowever, the manifold hypothesis, which is one of the working assumptions of our analyses, states that datasets (and their representations) typically lie “in proximity” of a smooth manifold often of much lower dimension than the embedding space. In particular, the ID approximates the number of independent coordinates that are necessary to describe the dataset without significant information loss [4]. When dealing with a finite set of data points the ID typically represents the number of dimensions where the dataset displays large/significant variations, neglecting those directions where the data variability is small/non-significant. Thus, the space of continuous (even smooth) functions that increase the intrinsic dimension in the sense of the manifold hypothesis is not of measure zero. \n\n**Limitations**\n\n*The use of intrinsic dimension estimators, which can provide misleading feedback in certain situations.*\n\nWe will use the response to the reviewer’s concern about the local density assumption discussed above in the revised version of the manuscript.\n\n**References**\n\n[1] Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models”, arXiv:2307.09288, 2023\n\n[2] Rodriguez et al., \"Computing the free energy without collective variables, Journal of chemical theory and computation\", 2018\n\n[3] Glielmo et al., \"DADApy: Distance-based analysis of data-manifolds in Python, Patterns\", 2022\n\n[4] Facco et al., \"Estimating the intrinsic dimension of datasets by a minimal neighborhood information, Scientific Reports\", 2017\n\n"}]

Author rebuttal:
We thank all reviewers for the detailed and diligent reading of our paper, from which we took a lot of cues on how to improve the quality of our work. We would also like to express our gratitude for the general appreciation of our contribution. We reply to common points raised by some Reviewers here below. 

**We remind everyone that the PDF file containing the figures supporting our responses is attached to this post.**

*Reply to the concern raised by reviewers **K92u** and **kxbE** regarding the difference in the final part of the ID profiles between iGPT and ESM.*

We agree with the reviewers that there is a qualitative difference in the last part of the ID profiles. In the main text, we mention that the “ three phases can be recognized in all the transformers and a fourth phase which we observed only in the iGPT models” (lines 172-173). We also emphasize that  “In the last part of the network [iGPT], at variance with what is observed in pLMs, the ID does not remain constant but grows again (more moderately) forming a second shallow peak almost at the end of the network” (lines 219-220-221). 
Indeed, the ID in the middle part of the architecture need not be smaller than that at the input. As we state in the manuscript, the ID in the middle layers of the architecture is connected to the semantic complexity of the dataset, which is different in the two datasets. More precisely, in the context of protein sequences, Facco et al. (2019) show that the ID of a family of proteins, which measures the amount of variability arising from evolutionary changes in protein sequences, is between 6 and 12. Conversely, when dealing with images, the ID related to the semantic complexity of the dataset could be roughly measured from the values at the output of the state-of-the-art classifiers, where the representation is most compact. For instance, in the case of the ImageNet dataset, the ID at the output of various ResNet CNN is within the range of 15-20 (Ansuini et al., 2019). These values are remarkably consistent with those observed in our study: 5-6 for ESM-2, and 22 for iGPT on ImageNet (see Fig. 1 in the manuscript).\
On the contrary, the ID at the first layer of the network is influenced by various aspects of the raw input data such as the brightness of the input pixels (Ansuini et al., 2019) or other factors, and can be either larger or smaller than the ID measured at middle layers of the network. Since in the output layer the network tries to reconstruct (part of) the input data, these considerations can explain why there can be concentration (iGPT) or expansion (ESM) in the final layers of the network. 
However, the *key finding of our study remains consistent in both cases*: the layers where the ID reaches a local minimum correspond to representations in which the overlap with a semantic property of the data is most pronounced. 

*Reply to the question of reviewers **K92u** and **wEGg** regarding the potential extension of the current work to the NLP domain.*

At the time of submission, the transformers we analyzed, trained on Natural Language Processing (NLP) tasks, were not large enough to observe a second peak akin to the one in the manuscript for iGPT. In the Appendix, we reported the estimate of the intrinsic dimension (ID) of a GPT-2-XL model on the Stanford Sentiment Treebank (SST) dataset (see Fig. 8 in the Appendix), and commented on our findings.\
Recently, we conducted the same experiment on the much larger Llama-v2 model (70B parameters) that was released last July (Touvron et al., 2023). In this latter case, described in Fig. 3 of the attached PDF, our preliminary analysis shows that the evolution of the ID profile is more complex: after the first peak, the ID increases again in the middle of the architecture. 
Remarkably, despite the dataset being quite different from ImageNet and Uniref, the *key finding of our present work remains consistent in the NLP case as well*: in correspondence with the first local minimum at one-third of the architecture depth (0.3 relative depth) the overlap with the class partition (given by sentiment of the sentence)  is the highest. \
After the second peak, the ID profile shows a more complex behavior. We believe that a thorough analysis of this and other downstream tasks is necessary to fully decipher the meaning of such profiles. We are excited to clarify and extend these aspects in a follow-up of this work, dedicated to NLP tasks.

**References**

Facco et al., “The intrinsic dimension of protein sequence evolution”, PLOS Comp. Bio., 2019

Ansuini et al., “Intrinsic dimension of data representations in deep neural networks”, 32nd Conference on Neural Information Processing Systems, 2018

Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models”, arXiv:2307.09288, 2023

Meta-review: While there were initially concerns on the scope of the experiments and generality of the results, the rebuttal results and discussion addressed these points, and the paper should be of general interest to the NeurIPS community. I encourage the authors to take all of the reviewer feedback into account when preparing the final version.
Decision: Accept (poster)