Towards Understanding the Shape of Representations in Protein Language Models
Authors: Kosio Beshkov
Anders Malthe-Sorenssen
Venue: ICLR 2026 Poster (2026)
Keywords: Protein Language Models
Shape Analysis
Transformers
Primary area: learning on graphs and other geometries & topologies

Abstract: While protein language models (PLMs) are one of the most promising avenues of research for future de novo protein design, the way in which they transform sequences to hidden representations, as well as the information encoded in such representations is yet to be fully understood. Several works have attempted to propose interpretability tools for PLMs, but they have focused on understanding how individual sequences are transformed by such models. Therefore, the way in which PLMs transform the whole space of sequences along with their relations is still unknown. In this work we attempt to understand this transformed space of sequences by identifying protein structure and representation with square-root velocity (SRV) representations and graph filtrations. Both approaches naturally lead to a metric space in which pairs of proteins or protein representations can be compared with each other.

We analyze different types of proteins from the SCOP dataset and show that the Fréchet radius and effective dimension of the SRV shape space follows a non-linear pattern as a function of the layers in ESM2 models of different sizes. Furthermore, we use graph filtrations as a tool to study the context lengths at which models encode the structural features of proteins. We find that PLMs preferentially encode immediate as well as local relations between amino acids, but start to degrade for larger context lengths. The most structurally faithful encoding tends to occur close to, but before the last layer of the models, indicating that training a folding model ontop of these layers might lead to improved folding performance.

Published as a conference paper at ICLR 2026
TOWARDS UNDERSTANDING THE SHAPE OF REPRE- SENTATIONS IN PROTEIN LANGUAGE MODELS
Kosio Beshkov
Anders Malthe-Sorenssen Department of Physics University of Oslo Oslo, Norway
Department of Physics
University of Oslo
Oslo, Norway dosio.neuro&e gmmil.com
ABSTRACT
While protein language models (PLMs) are one of the most promising avenues of research for future de novo protein design, the way in which they transform sequences to hidden representations, as well as the information encoded in such representations is yet be fully understood. Several works have attempted propose interpretability tools for PLMs, but they have focused on understanding how individual sequences are transformed by such models. Therefore, the way which PLMs transform the whole space of sequences along with their relations is still unknown. In this work we attempt to understand this transformed space of sequences by identifying protein structure and representation with square-root velocity (SRV) representations and graph filtrations. Both approaches naturally lead metric space in which pairs of proteins or protein representations can be compared with each other.
We analyze different types of proteins from the SCOP dataset and show that the Fréchet radius and effective dimension of the SRV shape space follow a non-linear pattern as function of the layers in ESM2 models of different sizes. Furthermore, we use graph filtrations as a tool to study the context lengths at which models encode the structural features of proteins. We find that PLMs preferentially encode immediate as well as local relations between residues but start to degrade for larger context lengths. The most structurally faithful encoding tends occur close to, but before the last layer of the models, indicating that training a folding model on top of these layers might lead to improved folding performance.
INTRODUCTION
Protein language models (PLMs) are novel and powerful approach for the modeling and design of new proteins with desired structural or functional features (Ferruz & Höcker 2022). Using PLMs, candidate quences for viral vectors (Lyu et al. 2024). enzymes (Madani et al. (023), biosensors (Hayes et al.) 2025) and functional binders (Chen et 2025 Bryant & Eloisson 2023). A particular setting in which PLMs have been found to be useful is in finding high-dimensional representations of protein sequences that are thought reflect the physical, evolutionary, or functional properties of a protein. This is especially useful since it allows one to efficiently evaluate and compare newly generated proteins without the need for expensive modeling or experiments. One of the most widespread mod- els of this type are the Evolutionary Scale Modeling (ESM) models (L.in et al. 2023: Hayes et al.) 2025), which we will analyze this work.
one predict the folding of proteins in 3D space (Lin al. 2023). generate
It is empirically understood that the representations in these models form good initialization for folding models (Lin et al. (202)) and can also be used as a reward function to guide other generative approaches (Wang et al. 2025). However, the precise features that these representations encode are still not fully understood. In previous work by Zhang et al. (2024). the authors develop a categorical Jacobian approach to suggest that PLMs encode the pairwise statisties of coevolving residues. In another work by [Simon & Zou (2024), the authors use sparse autoencoders to suggest that PLMs encode human-interpretable features such binding, structural motifs and functional domains. In

---

Published as a conference paper at ICLR 2026
yet another work that leverages the power of sparse autoencoders (Gujral et al. 2025) the authors show that PLMs and the individual neurons in them also encode features related to biologically relevant terms within the Gene Ontology hierarchy (Aleksander et al.) 2023
What all of these works study is the transformation of individual sequences to high-dimensional latent representations. However, such approaches ignore how different proteins or their representa- tions relate to each other in the latent space of a PL.. If structure determines function, then is reasonable to assume that similar structure determines similar function. Furthermore, if protein rep- resentations in PLMs characterize structural, evolutionary and functional features, then one might expect that similar representations share such features. In addition, the representation of protein in PLM is tensor of shape amino acids x model dimension, however for many applications it is standard to take the average over the first dimension, which ignores the shape of the representation, thereby missing the full richness of the information present in PLM representations.
To state our motivation more formally, is important to understand how the metric space of proteins compares to the metric space of complete shape representations in PLMs. Metric space approaches for the structural analysis of proteins have previously been considered within the rich field of shape analysis (Liu et al., 2010). The essential realization of these approaches is that two proteins can be structurally compared by finding the optimal way to superimpose them, allowing different types of transformations. This is the backbone for many standard tools in the field of rigid and flexible structural alignment such as root mean square deviation (RMSD), TM score (Zhang & Skolnick 2004) and FATCAT (Li et al. 2020) among others. Despite their popularity, to our knowledge, such methods have yet to be applied to study the hidden representations of PLMs.
In this work, we adapt and extend the shape-analysis framework to study the layerwise metric rep- resentation spaces of eight different classes in the SCOP (Chandonia et al. 2022) dataset pushed through several ESM2 models. We consider different features such as the Fréchet radius and the effective dimensionality of these spaces and show that they follow peculiar pattern as a function of layer, which is especially prominent for larger models. Furthermore, we introduce graph-filtration method that allows us to separate and study the scale at which PLMs best maintain the structure of proteins. Using this analysis, we show that while structure is always encoded better than chance in deep PLM representations, it is optimally encoded at very short context lengths of 2 or at slightly longer context lengths at ~ 8 amino-acid neighbors.
BACKGROUND
In this section, we fix our notation and introduce the mathematical machinery necessary for com- paring proteins and their corresponding representations in protein language models. We define two different ways compare proteins with PLM representations. In the first one, each protein is point in a nigh-dimensional (sometimes anfinitedimmnnsonne be consistently applied to proteins made up of different numbers of amino acids. In the second ap- proach, we define a filtration of metric spaces that only allows us to compare proteins of the same size. As we will argue, this useful for understanding the context length that current PLMs are sensitive to, as well as the degree to which protein structure is encoded in PLMs.
space, and we define a metric that can
We start by outlining several ways to mathematically formulate what proteins are. For each such definition of "protein" we also propose a metric space in which such proteins can be sensibly embedded.
1. Proteins as sequences: In this case, define protein P by its amino acid sequence. So, a protein of length I. will live in a space AL, where AA is an alphabet of the 20 canonical amino acids. Given that we want to compare proteins with different numbers of amino
acids, we can define the space of all possible amino acid sequences as AA" = A4. An
L=0
edit distance, such as the Levenshtein distance for example, can be used to define metric on this space. Thus, the metric space defined by amino acid sequences will be the pair (A" dlen).
2. Proteins as three-dimensional point clouds: If we consider the actual physical structure of a protein, we can define a protein as an ordered point cloud of size L in R3 equipped with the Fuclidean metric. However, R² contains points rather than point clouds, point clouds
2

---

Published as a conference paper at ICLR 2026
A
Saes MERION
SKOGMMYLTAA TAPP VEHGSNLTIWPO
AKGFOLTOKOO
PLM
TAGO
Figure 1: A) Depiction of a single sequence (Ikrl) mapped to three-dimensional structure and PLM representation. B) Illustration of how the space sequences is first transformed into a space of curves and afterwards into shape space. The lines sticking out of the sphere show the fibers of the map p.
of arbitrary sizes live in different space, namely P3
[ (R While it is possible
define metric on this space, such metrics (for example, Hausdorff or Wasserstein) often ignore the ordering of the point cloud, which matters when studying proteins. Note that each protein can be described through a map t A + Pg from the sequence space to L points in RR
3. Proteins as curves: Instead, we can study structure by identifying proteins with curves in R This is done by identifying the space of proteins with the space of continuous curves T3 = ( : 7 [0. 1] TR$}. We further require that for every protein of length and coordinates 1/2 P) E IR3 we have r(t) 1/r P) if = tl E N. As we will discuss later, this definition a protein treats proteins of any length as the same object and makes easier define metric that compares them.
4. Proteins as graphs: Finally, proteins can also often be thought of as graphs. This usually requires contact map, which is a binary matrix that encodes whether two residues are closer than specified threshold 6-1 being a standard choice). We define this approach more rigorously subsection 2.2
Notice that while we have defined different ways to think of proteins, the second and third definitions also be generalized to any ordered point cloud. Therefore, dimensional protein structure as well as the embeddings of sequences protein language models. We will denote the map from an amino acid sequence to the embedding space language model by the function o AA" - Pa. where m >> 3 is the embedding dimension of the model. Thus, our approach can be summarized by Figure and the following diagram,
them study three-
m
1
P5
I'm
where π is a map that sends point clouds different spaces to shared metric space of curves in think of choice of how to interpolate ordered point clouds that
Rm. Practically, they become curves in m
2.1 SHAPE SPACES AND THE SQUARE-ROOT VELOCITY REPRESENTATION
fundamental feature of protein structure is invariance to rotations and translations. What this implics is that protein structure is invariant under isometries of IR33 or the special Euclidean group SE(3). This is the principle behind models such as the SE(3)-transforme) (Fuchs et al.) 2020). which is equivariant to transformations from this group and is thought to play important role in the success of folding models such as AlphaFold2 (Jumper et al. 2021 I) and TTsold (Baek
et al. 2021
In the present work we use the square-root velocity (SRV) framework introduced in Srivastava et al. 2010) enforce invariance translation. This approach has previously been used create
3

---

In summary, we map sequences to their three-dimensional structure and to their representations in a PLM. These form point clouds which we interpolate to map to curves that are then transformed to their SRV representation. Finally, we quotient out rotations to form the shape space, which inherits a Riemannian structure and allows for the efficient computation of distances. We denote this procedure by a quotient map I'm - m ((m).. Notice that the space of protein curves under this map form a submanifold of the quotient space H = 5 200 (50(m) as illustrated panel Published as a conference paper at ICLR 2026
geometry-aware Variational Autoencoder (VAE) model in Huang et al. (2021). For any ordered point cloud, we interpolate the points with quadratic splines. thereby generating a curve yy which has the same length independently of the number of mnino-acids in protein sequence. Following the interpolation step, the SRV representation is defined as,
(t) 1(t/ lx(t)l22
From the norm in the denominator one can see that this approach projects curves an infinite- dimensional sphere 5 which makes computing geodesics, and thereby measuring distances, straightforward. Since translations are already accounted for, we quotient out the remaining ac- tions of the (E(m) group, namely the rotations generated by S0(m). This is done using SVD to solve the optimization problem for any two SRV curves qı and 92.
R = arg min |/ -Rgzll2-
(3)
RESO(N)
Given this, the distance between any two curves is defined in terms of the L2 norm as d(gi, q2) ||gi fiz2ll2- It is often the case that one also removes different reparameterizations of curves, but we have avoided that step given that interpolating between residues forms a consistent way of parameterizing proteins, and this additional step is not worth the computational cost.
of Figure I All curves that are the same up actions of SE (E() end up at single point y on this submanifold. The set of these curves is called a fiber and is defined by -(y) = (alp(7) = y).
2.2 GRAPH FILTRATIONS
When making predictions, language models have to consider the context within which a token exists. The context length to which a model is sensitive is unknown priori. Furthermore, while using contact map with threshold of 6-12Á makes sense in real three-dimensional protein structure, less clear how to choose such threshold in PLM representations where such a unit is not defined. For this reason, one needs to work with methods that are capable of considering many possible context lengths. One such example is the concept of a filtration, which is fundamental in the topological data analysis literature (Edelsbrunner & Harer 2010) and has been extended to graphs within the field graph learning (Hofer et al. [020]] Bray et al. 202I).
In filtration, one defines parameter and family of objects S, Se whenever < E'. In this specific case, we consider filtrations of graphs composed of a set vertices V = (V1, U2 UN with coordinates U; IR" m and edges whenever d2(2;, uz) t, with d2 being the Euclidean metric enforced by the ambient space. This induces a filtration on their adjacency matrices A', since Ai. S A for all i,j when < L.
Comparing the coordinates of real proteins in R33 to those of PLM in R" is not well-defined procedure. However, for the same protein P of length L, both adjacency matrices live in {0. 1) 1xx This allows to compare the structure of protein with the structure of its embedding in PLM. A natural choice of a metric in this space is the entry-wise 1-norm of the difference between the two adjacency matrices d'(w((P), o(P)) [|ي)(A)) (A')l,,, where by abuse of notation (AAA)
and o(A') respectively indicate the flattened adjacency matrix of the three-dimensional structure and the PLM representation at the t-th filtration value.
For our analysis we construct the nearest neighbor graphs of the true protein structure and PLM representation as shown in Figure 2 For small k, the graphs and the distance between their adjacency matrices is small. On the other hand, as one increases the number of neighbors, the graphs converge to cliques, and therefore this distance converges to 0. In
only differ at few locations,
4

---

Published as a conference paper at ICLR 2026
protein strucutre
Graph filtration
PLM
PLM representation the same protein
Illustration of how each protein or PLM representation is transformed to filtration of graphs with different number of neighbors. The connectivity in the PL.M representation super- imposed over the tiree-dimensional protein graph for clarity.
Figure 2:
general, these distances tend to follow hypergeometric distribution over counteract this, we normalize the distance by the empirical distribution of distances between real proteins and random point clouds R C IRm Therefore, for a family of proteins P = {Pi, P?,.., Jv), we get a filtration of distance histograms {d (vb())))). & (P))) At each level of the filtration we study the normalized
first moment, which we will refer to as the graph filtration moment. It is given by the equation
EppP (V)... ((P))
E Eppp (eeeld(P,.o(P))))
(4)
Ep.EP R.E...
RESULTS
All of the following analysis is based on 1098 randomly sampled protein structures from the SCOP dataset (Chandonia et al., 2022). We sampled up 200 proteins (some proteins due to missing pdb file) from each of the following protein classes [Alpha, Beta, Alpha/Beta, Alpha+Beta, Alpha and Beta, Membrane and cell surface proteins and peptides, Small proteins and Designed proteins] More information about these protein classes can be found on the SCOP website ottps::/sco..bereeleyy edu/ or in Chandonia et al. (2022).
excluded
3.1 GEOMETRY OF PLM SHAPP SPACES
section 2.1 we defined notion shape space that inherits a Riemannian structure that allows us compute distances and also carry out computations in the tangent space of each fiber. With this hand, we can define and estimate statistics of curves on the shape space as well as make use of tools such tangent PCA. We therefore track two measures of PLM shape space geometry, namely effective dimension and Fréchet radius. For all computations we use the Geomstats package Miolane et al. 2020
For a set proteins P = (P,, PN). we first interpolate them at 1000 equally spaced points with quadratic splines and then project them to the SRV shape space giving the Y == 1/2, }. There is not an priori correct choice of spline order, but quadratic splines the simplest option that is still differentiable required by the SRV framework. Very high-order splines are expected to add lot of additional structure to the PLM point clouds and could thus bias the results. For an evaluation of the robustness of our results to spline order and interpolation samples see Figure 6 We compute the Fréchet radius by finding the Fréchet mean pr arg min Σ d(x,, y:) z6 H
through gradient descent after which the Fréchet radius is defined as,
E [d))..pF).. TF
(5)
BEY
5

---

Published as a conference paper at ICLR 2026
ESM2 8M
ESM2 - 35M
Effective Dimension
Frechet Radius
Effective Dimension
Frechet Redius
1.25
4.0
1.25
3.5
1.20
3.5
1.20
1.35
3.0
3.0
1.30
2.5
1.05
2.5
1.05
2.0
2.0
1.00
1.5
0.90
0.0
255
5.0
7.5
10.0
0.0
255
5.0
7.5
10.0
ESM2 150M
ESM2 650M
Effective Dimension
Frechet Radius
Effective Dimension
Frechet Redius
7-
12
0-
5-
88
1.0
3-
0.9
10
20
30
10 20
30
0 10 20 30 30
20
30
Figure 3: Effective dimension and Fréchet radius for each of the four models as function of the layers. The black line indicates the value of each metric for the diree-dimensional protein structure. As one see larger models exhibit dimension expansion in the initial layers and contraction later on.
An intuitive way to think of this object as measure for how spread out PLM representations are with respect to each other on the shape space. Therefore, a small value indicates that different proteins are represented by similar shapes, whereas a larger value indicates the opposite. As one can see in Figure 3 the Fréchet radius tends to decrease with depth and is much smaller for PLM representations than for real three-dimensional protein structures. Surprisingly enough, does not seem to vary with model size, indicating that the variability among shapes is low for all models.
Another measure describing the geometry of shape spaces in PLMS is the dimension of the sub- manifold which they live. In Euclidean data this is usually measured by the effective dimension defined through the eigenvalues {Yy, λ2 ... of the covariance matrix of the data or
C3 ΣXE
This procedure can be extended to curved data through tangent PCA (Abboud et al. 2020) where one first uses the log map to project all data to the tangent space of the Fréchet mean by 2; log (v:) pm] (y: cos [os[d( PF pF))pF) and afterwards applies PCA in this tangent space. A large effective dimension implies that PLM representations explore many different variations in shape, whereas a small effective dimension implies that a few specific variations are enough to describe the differences in PLM representation shapes. Judging by Figure 3 seems like PLMs go through two regimes- a dimension expansion in the first layers followed by dimension contraction towards the end. Larger models expand the dimension more than the three-dimensional structure baseline, whereas the smaller models stay below it. The largest models even show a second peak in dimension expansion.
d(g..p)
It is also worth pointing out that, especially in later layers, this dimensionality is very low and severely different from the dimensionality found through standard PCA on the flattened PL.M rep- resentation (see Figure 5lin the Appendix). We interpret this to mean that while shapes are encoded in a high-dimensional ambient space, the differences in shapes PLM representations can be de- scribed by just few shape descriptors. In other words, PLMs encode proteins by similar shapes while spanning many different directions within their ambient space. To see if this trend only occurs in the ESM2 models, we also ran the same analysis (et al. (2023). As one can see in Figure 7 a similar pattern, though with a more extremee initial dimension expansion phase, can be seen in that model.
the general- purposs Ankh model Elnaggar

---

Published as a conference paper at ICLR 2026
Alpha
Beda
Atphal Bota
Alpha+Bota
Algha and Beta
Popidersurface
Small Proteins
Designed
575
0.915
0.32
64930
0.90
039
0.65
0.00
0.83
8.955
0.85
5500
0.800
0.82
0.85
0.00
850
0.575
0.75
0.2
0.00
0.15
10 111
10
10
10
11
10 15
35
14dg toxs
noggcare
14:g Bora
(Mdg2220
35M
1.0
3.0
1.0
1.0
6.0
0.0
0.0
3-8
6.7
0.83
0.7
0.5
8:7
6.7
0.7
0.5
0.5
0.23
ESM2- 150M
1.0
6.0
38
01
0.6
0:5
3.6
36
84
0.0
84
6/1
0-0
regtters
reighbos
reghioos
rol/oors
negiioos
kydro
ESM2-6900
0.-
0.2
0.6
0.5
84
04
0.4
0.5
2.7
15
15
reaFbors
reldioos
reapdoot
1ypers
Figure 4: Graph filtration moments for all models and all layers evaluated on different protein classes from the SCOP database. Curves increase in transparency with increasing layer number. The right- most plots show the minimal value of the filtration as a function of the layer. Fach curve is color- coded for protein class in correspondence with the other plots.
3.2 CONTEXT LENGTH SENSITIVITY PLMS
While is interesting to understand the global geometric features of PLM representation shape spaces, language models known encode contextual features of text which are hard to relate to the aforementioned measures. In order to better understand how structural context is encoded in PLMs, we use the graph filtration moment defined in Equation This measure indicates how similar the shapes of PLM representations are to their three-dimensionee value of above means that the PLM arranges all residues in a random point cloud, whereas lower values imply that three-dimensions protein structure is encoded in the PLM representation.
protein structure counterparts. A
In Figure we show the graph filtration moments across all layers of the different ESM2 models on the SCOP protein classes. The curves for shallow layers are more transparent than deep layers and the rightmost plot shows the minimum value across the filtration peculiar patterns arise throughout the filtration. The first thing to note is that larger models have better similarity to three-dimensional protein structure their intermediate layers, but model size seems have less of an impact the last layer. This implies that while encoding protein structure emerges as an important intermediate processing step for unmasking, is not as important at the last classification step.
function of depth. Several
The second striking observation is that models, many of the curves have a bimodal shape throughout the filtration. This implies that PLM representations encode three-dimensional protein

---

Published as a conference paper at ICLR 2026
structure at both very local level, at about two neighbors, well as slightly less local level, at about eight neighbors. The improved encoding at two neighborhoods has a simple interpretation as PLM representations having similar immediate neighbors to three-dimensional protein structure. However, the second valley at around eight neighbors is harder to interpret. Given the fact that is less pronounced in the Beta class, one might speculate that it might be related to representations of Alpha helices, but more work is needed to understand the precise meaning of this feature.
Finally, it is also clear that certain protein classes like Alpha/Beta are represented by shapes that are much more structurally similar to their three-dimensional structure compared to other classes such as small and designed proteins. Furthermore, while initial layers sometimes showed a correlation between protein length and the minimum graph filtration moment, later layers, especially in larger models, did not show such pattern (see Figure 9 and Figure [0]. This indicates that local context structure is represented independently of protein length. The results for the Ankh model show a similar pattern as can be seen in Figure
4 DISCUSSION
We have applied two approaches in order to better understand the geometry of shape spaces gen- crated by PLM representations. The first uses SRV representations and quotients out rotations to create space with Riemannian structure, which allows us define a metric and generalize statisti- cal methods such as PCA to shapes. The second uses graph filtrations to study how protein structure encoded in the layers of a PLM at as many levels of resolution as one desires. Given the abstract nature of our results, here we provide a discussion of how they can be understood more intuitively and propose several future directions that would be exciting pursue.
4.1 EXPANSION AND CONTRACTION IN PLM REPRESENTATION SHAPP SPACES
As shown in Figure 3. the initial layers in PLMs exhibit an expansion in effective dimensionality. whereas later layers contract the shape space very low-dimensional subspace. Previous work on traditional language models has used the notion of intrinsic dimensionality (Li et al.] 2018) and has shown that language models have remarkably low intrinsic dimensionality relative to model size (Aghajanyan et al. 2020). Additionally, while our results concern collections of many proteins, they complement the analysis of Hakiim et al. (2025), which uses IsoScore (Rudman et al. 2022)
to conclude that PLM representations of individual proteins span 2-14 dimensions. Similar mea-
sures of dimensionality are also thought to relate to task performance (Marbut al. 2024 training
convergence and generalization (Ruppik et al.) (025).
The specific expansion-conttee contraction pattern observed in our estimate of dimensionality is very similar to the behavior seen in Cheng et al. (2024) and Valeriani et al. (023). The universal appearance of
this pattern is thought to correspond to general high- high-abstraction regime in the dimension mnmeension-expansiinn
phase and specific semantically rich regime in the contraction phase. These properties of language
model layers can be effectively used to solve any task.
Our approach looks at the dimensionality of the data within the shape space manifold rather than directly looking at all residue representations in the embedding space. This leads to an arguably more clear interpretation, the initial layers of PLM represent proteins by shapes that can be flex- ibly deformed to each other by combining many different non-linear shape transformations. The higher the dimensionality, the more such transformations there are. Therefore, the sharp reduction in dimensionality in later layers means that there are remarkably few transformations (fewer than in the space of three-dimensional protein shapes) that are needed to efficiently navigate PLM represen- tation shape spaces. Understanding the precise nature of these transformations would be an exciting direction for future work.
4.2 STRUCTURAL ENCODING IN PLM REPRESENTATIONS
In addition to studying the geometry of PLM representation shape spaces, we also looked at how direendimensooall protein structure is encoded in the layers of PLM by using graph filtrations. We observe a bimodal pattern in which protein structure is optimally encoded at the resolution of very short context lengths of about two residues as well at slightly longer context lengths of around
8

---

Published as a conference paper at ICLR 2026
eight residues. The PLM representations of Alpha/Beta proteins showed by far the highest similarity to their three-dimensional structure, whereas small and designed proteins were represented by more distinct shapes. While outside of the scope of this work, understanding what features of these protein classes determine how much of the structure encoded by PI.Ms is an exciting direction for future research.
The finding that PLMs encode three-dimensional protein structure at all is surprising given that PLMs are given masked sequences and then trained to predict the most likely missing amino acids. There is no point at which PLM representations are incentivized to encode protein structure. Therefore, either learning protein structure is beneficial for the unmasking process the function learned during unmasking shares some properties (one might say "correlates") with the function used the folding of sequence to three-dimensional structure.
Finally, our findings further explain why folding models such as ESMFold benefit from starting with a pretrained PLM that has already partially learned protein structure. Our observation that structure is not optimally encoded in the last layer, but rather in the layers that immediately precede it, can be used to improve initializations for protein models. Given our results, we expect that better folding performance can be achieved if one uses the layers with the optimal structural PLM representation rather than the whole model. Initial attempts to show this with linear models or small networks failed to generalize (data not shown) and verifying this hypothesis would require training and testing larger models. The way in which layerwise representations can be used for folding, along with other functional tasks, is another avenue of future research that would be exciting explore.
REPRODUCIBILITY STATEMENT
Our analysis uses publicly available datasets and protein language models. The code used to sam- ple the data and perform the analysis can be found at https:://ittuut com /Be//kkBeshk P ProtGeom/
REFERENCES
Michel Abboud, Abdesslam Benzinou, and Kamal Nasreddine. A robust tangent pea via shape restoration for shape variability analysis. Pattern Analysis and Applications, 23(2):653-671, 2020.
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
Suzi A Aleksander, James Balhoff, Seth Carbon, J Michael Cherry, Harold J Drabkin, Dustin Ebert, Mare Feuermann, Pascale Gaudet, Nomi L Harris, et al. The gene ontology knowledgebase in 2023. Genetics, 224( (141))): 2023.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang. Qian Cong, Lisa N Kinch, Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871 876, 2021.
Patrick Bryant and Arne Elofsson. Peptide binder design with inverse folding and protein structure prediction. Communications Chemistry, 6(1):229, 2023.
John-Mare Chandonia, Lindsey Guan, Shiangyi Lin, Changhua Yu, Naomi K Fox, and Steven E Brenner. Scope: improvements to the structural classification of proteins-extended database to facilitate variant interpretation and machine learning. Nucleic acids research, 50551): D559, 2022.
Leo Tianlai Chen, Zachary Quinn, Madeleine Dumas, Christina Peng, Lauren Hong, Moises Lopez- Gonzalez, Alexander Mestre, Rio Watson, Sophia Vincoff, Lin Zhao, et al. Target sequence- conditioned design peptide binders using masked language modeling. Nature Biotechnology, pp. 1-9, 2025.

---

Published as a conference paper at ICLR 2026
Emily Cheng, Diego Doimo, Corentin Kervadec, Iuri Macocco, Jade Yu, Alessandro Laio, and Marco Baroni. Emergence of high-dimensional abstraction phase in language transformers. arXiv preprint arXhv:2405.1547/ 2024.
Herbert Edelsbrunner and John Harer. Computational Topology: An Introduction. American Math- ematical Soc., 2010.
Ahmed Elnaggar, Hazem Essam, Wafaa Salah-Eldin, Walid Moustafa, Mohamed Elkerdawy, Char- lotte Rochereau, and Burkhard Rost. Ankh: Optimized protein language model unlocks general- purpose modelling. arXiv preprint wrXiv:2301.06566 2023.
Noelia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelligence, 4(6):521-532, 2022.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (rantrfasffrmmrrs: 3d roto- translation equivariant attention networks. Advances in neural information processing systems, 13:1970-1981, 2020.
Onkar Gujral, Mihir Bafna, Eric Alm, and Bonnie Berger. Sparse autoencoders uncover biologically interpretable features in protein language model representations. Proceedings of the National Academy of Sciences, 122(34):c2506311122) 2025.
Sheikh Azizul Hakim, Kowshic Roy, and M Saifur Rahman. Isotropy and geometry of pretrained protein lms. arXiv preprint arXiv:2510. 10655, 2025.
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with a language model. Science, 38766736)8550-858 2025.
Christoph Hofer, Florian Graf, Bastian Rieck, Mare Niethammer, and Roland Kwitt. Graph filtration learning. In International Conference Machine Learning, pp. 4314 4323. PMLR, 2020.
Hao Huang, Boulbaba Ben Amor, Xichan Lin, Fan Zhu, and Yi Fang. G-vae, geometric convolu- tional vae for proteinstructure generation. arXiv preprint arXiv:2106.1 11920 1920, 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596(7873)5583-558) 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint 1rXiv:/804.08838 2018.
Zhanwen Li, Lukasz Jaroszewski, Mallika Iyer, Mayya Sedova, and Adam Godzik. Fatcat 2.0: towards a better understanding of the structural diversity of proteins. Nucleic acids research, 48 (W1):W60-W64, 2020.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1 122--777:113311111 1130, 2023.
Wei Liu, Anuj Srivastava, and Jinfeng Zhang. Protein structure alignment using elastic shape analy- sis. In Proceedings of the First ACM International Conference on Bioinformatics and Computa- tional Biology, pp. 62-70, 2010.
Suyue Lyu, Shahin Sowlati- -ashiin,,, and Michael Garton. Variational autoencoder for design of synthetic viral vector serotypes. Nature Machine Intelligence, 6(2):147-160, 2024.
Ali Madani, Ben Krause, Eric R Greene, Subu Subramanian, Benjamin P Mohr, James M Holton, Jose Luis Olmos Jr, Caiming Xiong, Zachary 7. Sun, Richard Socher, et al. Large language models generate functional protein sequences across diverse families. Nature biotechnology, 41 189:1099-1106, 2023.
Anna Marbut, John W Chandler, and Travis J Wheeler. Exploring the impact of a transformer's latent space geometry on downstream task performance. arXiv preprint arXiv:2406. /2/59, 2024.
10

---

Published as a conference paper at ICLR 2026
Nina Miolane, Nicolas Guigui, Alice Le Brigant, Johan Mathe, Benjamin Hou, Yann Thanwerdas, Stefan Heyder. Olivier Peltre, Niklas Koep, Hadi Zaatiti, Hatem Hajri, Yann Cabanes, Thomas Gerald, Paul Chauchat, Christian Shewmake, Daniel Brooks, Bernhard Kainz, Claire Donnat, Susan Holmes, and Xavier Pennec. Geomstats: A python package for riemannian geometry in machine learning. Journal of Machine Learning Research, 21(223):19, 2020. URL http:
7/jmlr. http://.oo//aapr///211//2
Leslic O' D'Bray, Bastian Rieck, and Karsten Borgwardt. Filtration curves for graph representation. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery Data Mining, pp- 1267-1275, 2021.
William Rudman, Nate Gillman, Taylor Rayne, and Carsten Eickhoff. Isoscore:
Measuring the uniformity of embedding space utilization. In Findings of the Association for Computational Linguistics: ACI. 2022, pp. 3325-3339, 2022.
Benjamin Matthias Ruppik, Julius von Rohrscheidt, Carel van Niekerk, Michael Heck, Renato Vukovic, Shutong Feng, Hsien-chin Lin, Nurul Lubis. Bastian Rieck, Marcus Zibrowius, et al. Less is more: Local intrinsic dimensions of contextual language models. arXiv preprint arXiv:2506.0103: 2025.
Elana Simon and James Zou. Interplm: Discovering interpretable features in protein language mod- els via sparse autoencoders. bioRxiv, pp. 2024-11, 2024.
Anuj Srivastava, Eric Klassen, Shantanu H Joshi, and Ian Jermyn. Shape analysis of elastic curves in euclidean spaces. IEEE transactions on paltern analysis and machine intelligence, 33 (7):1415-1428, 2010.
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Al- berto Cazzaniga. The geometry hidden representations of large transformer models. Advances in Neural Information Processing Systems, 36:5163443355
2023.
Yeji Wang, Minghui Song, Fujing Liu, Zhen Liang, Rui Hong, Yuemei Dong, Huaizu Luan, Xiaojic Fu, Wenchang Yuan, Wenjie Fang, et al. Artificial intelligence using latent diffusion model enables the generation of diverse and potent antimicrobial peptides. Science Advances, 11(6): eadp7171, 2025.
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702-710, 2004.
Zhidian Zhang, Hannah K Wayment- Steelle Garyk Brixi, Haobo Wang. Dorothee Kern, and Sergey Ovchinnikov. Protein language models learn evolutionary statistics of interacting sequence motifs. Proceedings of the National Academy of Sciences, 22444)51,222 2024.

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
ADDITIONAL FIGURES
ESM2 8M
ESM2 35M
ESM2 150M
ESM2 650M
Effective Dimension PCA
Effective Dimension PCA
Effective Dimension PCA
Effective Dimension PCA
500
500
600
400
900
300
300
200
200
100
0
5.0 7.5
10.0
20
30
10
20
30
Figure 5: Effective dimension esimated using PCA directly PLM representations. Each PLM representation is first interpolated at 1000 points. Afterwards each tensor of shape 1000xPLM dimension layer is flattened, meaning that the maximum dimension of this space is 1000 times the dimension at the layer.
12

---

Published as a conference paper at ICLR 2026
p=2, n=1000
pm3, n=1000
4.5
0.98
n=500
4.0
3.5
0.96
3.0
0.94
2.5
2.0
0.92
1.5
0.0
2.5
5.0
7.5
10.0
0.0
2.5
5.0
7.5
10.0
Figure 6: Robustness with respect to interpolation order (indicated by p) and number of sampled points (indicated by n) evaluated on the ESM2 35M model.
Effective Dimension
Frechet Radius
30
1.2
25
1.1
20
1.0
0.9
10
0.8
0.7
0
20
40
0
20
40
Figure 7: Effective dimension and Fréchet radius for the base Ankh model as function of the layers.
Reda
Peptide/surface
Proteins
Designed
1:00
0.4
644
0.5
0.6
0.8
33
07
0.6
0.9
0.6
Bors
relgitos
#2gsters
nogeheer
kyrr
Figure Graph filtration moments for the base Ankh model.
13

---

Published as a conference paper at ICLR 2026
ESM2 8M
140.7
c+0.04
22025
01
18-
26
0.4
Layer
Layer
Layer
300
3000
1500
2000
530
1003
1000
2000
502
1000
3500
2006
ESM2 35M
1-0.74
r=-0.05
0.8
86
Layer
Layer
Layer
1000
2003
1202
2006
1200
2002
ESM2 150M
04
Layer 1
62
Layer 15
Layer 23
Protcin length
Figure 9: Protein length vs minimum graph filtration moment for several layers of the first three ESM models. Values above the plots indicate the Spearman correlation values. The last layer is chosen correspond to the minimum in the plots in Figure
14

---

Published as a conference paper at ICLR 2026
150.55
1.0
09
0.7
0.-
Layer
Layer
0.4
Layer
ms-0.25
1.0
1.0
0.5
0.9
0.3
0.7
0.7
0.5
0.5
0.5
0.-
Layer
Layer
Layer
1020
1962
3890
Protein length
Figure 10: Protein length minimum graph filtration moment for several layers of the Ankh model. Values above the plots indicate the Spearman correlation values.
15


Reviews:
[{"review_id": "dVNXy3BngN", "reviewer": "Reviewer_R8RE", "summary": "This paper aims to unravel the geometry of hidden representations in Protein Language Models (PLMs), a critical gap in understanding how PLMs encode protein structure/function for de novo design. Using two novel analytical frameworks—square-root velocity (SRV) shape spaces and graph filtrations—the authors analyze multiple sizes of ESM2 (a leading PLM) on 1098 proteins from the SCOP dataset (spanning 8 protein classes). Key findings include: (1) PLM representation shape spaces exhibit non-linear patterns in Karcher mean and effective dimension across layers, with larger models showing \"dimension expansion\" in early layers and \"contraction\" in later layers; (2) PLMs optimally encode protein structure at short context lengths (~2 amino acids) and moderately short lengths (~8 amino acids), but degrade at larger contexts; (3) structural encoding is strongest in layers just before the last layer, suggesting potential improvements for folding models.", "questions": "Does sequence length correlate with structural encoding quality?\nThe authors find PLMs preferentially encode immediate as well as local relations between residues, but start to degrade for larger context lengths, does this mean that PLMs cannot learn global relations?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Adapts shape analysis (SRV) and topological data analysis (graph filtrations)—previously used for 3D protein structure—to PLM representation spaces, enabling rigorous metric comparisons between protein structures and their PLM embeddings.\n2 Identifies a universal \"expansion-contraction\" pattern in effective dimension across ESM2 layers, where early layers explore diverse shape variations and later layers converge to a low-dimensional subspace of biologically relevant shapes.\n3. Quantifies that PLMs prioritize local residue relationships (contexts of 2 or 8 amino acids) over global ones, explaining why PLMs implicitly encode structural features despite being trained only on masked sequence prediction.\n4. Highlights that the last layer of PLMs is not optimal for structural encoding; using pre-last layers could improve downstream folding models (e.g., ESMFold).", "weaknesses": "1. The study only uses ESM2. It is unclear if the \"expansion-contraction\" pattern or context length bimodality (2, 8) applies to other popular PLMs (e.g., ProtTrans, AlphaFold’s embedding layers) or PLMs trained on different datasets (e.g., UniProt vs. Pfam).\n2 The authors use Euclidean distance for graph filtrations but do not test if alternative metrics (e.g., cosine similarity, Manhattan distance) alter the bimodal context pattern. Euclidean distance may not be optimal for high-dimensional PLM embeddings, where cosine similarity is often preferred.\n3. The paper suggests using pre-last layers to improve folding models but provides no experimental evidence (e.g., comparing ESMFold performance with pre-last vs. last layers). This makes the suggestion speculative. It is interesting to see such findings, and I am also curious whether the pre-last layers can improve other downstream tasks, like GO, EC prediction? \n4. The paper notes that shape space effective dimension is much lower than flattened PLM representations but does not identify which specific shape features (e.g., helix curvature, beta-sheet spacing) drive this low dimensionality.\n5. Most of the findings or conclusions of this theory are conforming to cognition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CaubIvsMrl", "reviewer": "Reviewer_Xyzz", "summary": "The authors compare spaces induced by protein language models to those induced by protein structures. By applying various representation schemes and dimensionality analysis techniques they derive some insights about the way PLM embeddings are organized. In particular, we learn that protein structure is mirrored in PLM embeddings at highly localized neighbourhoods, and that the effective shape space of embeddings is quite limited. These are highly valuable insights for future PLM model development.", "questions": "* How sensitive is the analysis to the curve fitting degree? How do you measure the quality of a curve fitting for a given protein? As far as I know, this representation approach is not well-established for proteins so its soundness should be explored further.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* This work provides a highly unique and novel view on PLM work and generates valuable insights. The framework for understanding PLMs is also novel and will be a valuable asset for the field.\n* Modeling the 'shape' of embeddings is a very interesting proposition. I am also very surprised by the finding that embedding shape can mirror protein shape at all.\n* The paper is very well-written and flows nicely from the initial premise and questions to be explored (e.g. \"If structure determines function, then it is reasonable to assume that similar structure determines similar function. Furthermore, if protein representations in PLMs characterize structural, evolutionary and functional features, then one might expect that similar representations share such features. \")", "weaknesses": "* There is a lack of simpler methods than the one proposed. The authors begin by asking whether PLM embeddings for structurally similar proteins would be similar themselves. Of course this can be tackled by mapping the embeddings to shape spaces and comparing shapes, but one could also try to directly measure correlation for example between TM scores and PLM embeddings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xpr2jryhza", "reviewer": "Reviewer_eXtu", "summary": "This paper investigates the geometric and structural properties of representations learned by PLMs, specifically focusing on ESM2 models. The authors propose two approaches: (1) a shape-space analysis using Square-Root Velocity (SRV) representations to compare protein structures and PLM embeddings in a rotation-invariant Riemannian space, and (2) a graph filtration method to analyze the context length at which PLMs encode protein structural information. The main findings include: (1) PLM representation shape spaces exhibit an expansion-contraction pattern in effective dimensionality across layers. (2)PLMs encode 3D protein structure most faithfully at very short (2 residues) and slightly longer (8 residues) context lengths. (3)The optimal structural encoding occurs in intermediate layers, not the final layer, suggesting potential improvements for folding models. The authors argue that their approach provides a novel way to understand how PLMs transform the space of protein sequences and encode structural information without explicit supervision.", "questions": "- How sensitive are the SRV and graph filtration results to the choice of interpolation method (quadratic splines) and the number of interpolation points?\n- Could the authors provide a qualitative analysis or visualization of the structural features captured at the 2- and 8-residue context lengths, and discuss whether the observed bimodal context-length sensitivity is associated with specific secondary structure motifs?\n- Have the authors experimented with using the identified optimal layers to initialize a folding model, did it improve performance over using the final layer?\n- How do the results generalize to other PLM architectures or training regimes (e.g., models trained with different objectives or data)?\n- The effective dimension is computed via tangent PCA. How does this compare to intrinsic dimension estimators (e.g., MLE, TWO-NN) applied directly to the PLM embeddings?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel application of shape analysis (SRV) and graph filtrations to study PLM representations. This is a fresh perspective compared to existing interpretability methods that focus on individual sequences.\n- The SRV and graph filtration approaches are well-motivated and technically sound, providing a principled way to compare protein structures and embeddings.\n- The expansion-contraction pattern in effective dimension and the bimodal structural encoding at specific context lengths are intriguing, which facilitates understanding of the context lengths at which models encode the structural features of proteins.\n- The findings have practical implications for improving protein folding models and protein design, e.g., by selecting intermediate layers for downstream tasks.\n- The paper is generally well-written, with clear motivations and a logical flow. The figures help illustrate the methods and results effectively.", "weaknesses": "- The study is restricted to ESM2 models and eight classes of the SCOP dataset. It would be beneficial to include other PLMs and datasets to assess generalizability.\n- The paper lacks comparisons with simpler baselines (e.g., random embeddings, non-PLM representations) to better contextualize the reported metrics (e.g., Fréchet radius, effective dimension).\n- While the graph filtration method is innovative, the interpretation of the “graph filtration moment” is somewhat abstract. More intuition or a qualitative analysis (e.g., visualizing which residues are captured at different context lengths) would strengthen the claims.\n- The paper implies that encoding protein structure emerges as an important intermediate processing step for unmasking, but no causal experiments are provided to support this.\n- The impact of interpolation methods (quadratic splines) and SRV parameterization choices is not explored.  \n- No statistical tests or confidence intervals are reported for the Fréchet radius or effective dimension estimates.\n- No implementation details or hyperparameters are provided in the current draft. While the ESM2 model and SCOP dataset are publicly available, releasing the code of this work would greatly improve reproducibility. Although some scripts appear to be included in the supplementary .zip file, it is recommended that the authors formally release a public code repository link in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)