Distilling Structural Representations into Protein Sequence Models
Authors: Jeffrey Ouyang-Zhang
Chengyue Gong
Yue Zhao
Philipp Kraehenbuehl
Adam Klivans
Daniel Jesus Diaz
Venue: ICLR 2025 Poster (2025)
Keywords: biology
proteins
sequence
structure
autoencoder
esm
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: we finetune esm to improve structural representations

Abstract: Protein language (or sequence) models, like the popular ESM2, are now widely used tools for extracting evolution-based protein representations and have achieved significant success on core downstream biological tasks.
A major open problem is how to obtain representations that best capture both the sequence evolutionary history and the atomic structural properties of proteins in general. 
We introduce **I**mplicit **S**equence **M**odel, a sequence-only input model with structurally-enriched representations that outperforms state-of-the-art sequence models on several well-studied benchmarks including mutation stability assessment and structure prediction. 
Our key innovations are a microenvironment-based Autoencoder for generating structure tokens and a self-supervised training objective that distills these tokens into ESM2's pre-trained model. 
Notably, we make ISM's structure-enriched weights easily accessible for any application using the ESM2 framework.

Published as a conference paper at ICLR 2025
DISTILLING STRUCTURAL REPRESENTATIONS INTO PROTEIN SEQUENCE MODELS
Jeffrey Ouyang-Zhang, Chengyue Gong, Yue Zhao, Philipp Krähenbühl, Adam R. Klivans,
Daniel J. Diaz
University of Texas at Austin
(jozhang, cygongl7, 17, yzhao, philkr, klivans, danny, diaz @utexas.
ABSTRACT
Protein language models, like the popular ESM2, are widely used tools for extract- ing evolution-based protein representations and have achieved significant success on downstream biological tasks. Representations based on sequence and struc- ture models, however, show significant performance differences depending the downstream task. A major open problem to obtain representations that best capture both the evolutionary and structural properties of proteins in general. Here we introduce Implicit Structure Model (ISM), a sequence-only input model with structurally-enriched representations that outperforms state-of-the-ar sequence models on several well-studied benchmarks including mutation stability assess- ment and structure prediction. Our key innovations are microenvironment based autoencoder for generating structure tokens and self- self-supervised training objective that distills these tokens into ESM2's mre-trained model. We have made ISM's structure- enriched weights easily available: integrating ISM into any application using ESM2 requires changing only a single line of code. Our code available https: / / g thub. com jozhang ISM..
INTRODUCTION
Protein language models (pLMs) are versatile feature extractors with proven success across numerous downstream applications (Elnaggar et al., 2021; Brandes et al., 2022; Rives et al., 2019; Lin et al., 2022). Their accessibility has significantly democratized protein research, enabling biologists with limited computational expertise to apply advanced machine learning techniques to their specific protein domain. The method's success comes from its exclusive use of sequences, bypassing costly, unreliable, infeasible structure computations and sophisticated data-engineering pipelines.
The tradeoff that PLMs are often lack structural context and underperform (relative to structure- based models) on tasks that typically require structural insight (Su et al., 2023; Yang et al., 2023; Zhang et al., 2024; Gaujac et al., 2024; Frolova et al., 2024; et al., 2024; Kulikova et al., 2023; Allman et al., 2024). Longstanding biological research (Anfinsen, 1973) does suggest that the amino acid sequence is solely responsible for the folding of the structure. Indeed, sequence-only models trained using masked language modeling learn extract structure features encoded in evolutionary co-variations (Lin et al., 2022). However, current state-of-the-arr frameworks, such as AlphaFold, require the protein's 's evolutionary history as an additional input, demonstrating that sequence-only models fail to extract all the structural information within multiple sequence alignment (MSA). Building a single-sege sequeece model (without additional MSA imput) that leads soucturally-infomme representations remains challenging open problem.
In this paper, we introduce Implicit Structure Model (ISM), a sequence-only protein language model that is trained to implicitly capture structural information. Our key contribution is new self-supervised pre-training objective, structure-tuning, where the sequence model learns to distill features derived from structure-based models (see Figure I). As result, ISM outperforms sequence- only models and is competitive with pLM frameworks that explicitly take the protein structure as an additional input. For example, the CAMEO protein structure prediction benchmark ISM outperforms its ESM2 counterpart with a GDT-TS score of 0.67 versus 0.64 (see Table 1). For S669 AAG prediction, ISM surpasses ESM2 in AUC (0.76 vs 0.72) and even matches specialized models

---

Published as a conference paper at ICLR 2025
Codebook
logils
[1d:1]
[1d:2]
[1d:1]
$ 4.2 supervise
[1d:2]
[1d:
Structure Decoder
quantize
Structure Encoder
[1d:k]
ISM
3.2 extract microenv
3.1 unfold
VAL
HIS
PRO
Atom Graph
Full Protein
Amino Acid Sequence
Figure 1: Structure-tuning a protein language model. Implicit Structure Model (ISM) a sequence- only protein model (right) supervised by structure tokens derived from a structure model (left). For every residue, structure encoder takes the atoms of residue's 's microenvironment produces structural representation. We discretize these representations into tokens using a codebook extracted via k-means clustering. The ISM sequence model learns to predict these structure tokens.
input and
that process atomic environments (0.76 vs 0.75, see Table 2). Our results align with prior works that show multiple modalities enhance model performance (Gong et al., 2024; Hayes et al., 2024).
Structure-tuning is fine-tuning technique where sequence-only model is trained to predict structure tokens rather than masked amino acids for each protein residue (see Figure 1). Our structure tokens, derived from Atomic Autoencoder and MutRank (Gong et al., 2024), capture key chemical interactions that underpin the protein's tertiary structure. cetuture-tuning distills these structural representations into ISM, demonstrated by the significant improvement in predicting long-range tertiary interactions (0.49 vs 0.35, see Table 1).
2 RELATED WORK
Protein Language Models. These models take an amino acid sequence as input and produce deep representation for each amino acid conditioned on the entire sequence. Commonly-use models such ProtBERT, ProteinBERT, ESMIb, and ESM2 transformer based architectures and are trained to maximize wildtype accuracy (i.e., reconstruct masked amino acids) (Elnaggar et al., 2021; Brandes et al., 2022; Rives et al., 2019; Lin et al., 2022).
One of the motivations behind ESM2 was to build a single-sequence variant of AlphaFold that does require the computationally expensive task of generating MSAs. The resulting model, ESMFold. a widely used tool but generally underperforms when compared to AlphaFold in terms of predicted structural quality. This demonstrates that ESM2 does not fully capture the epistatic landscape induced during evolution. This has motivated research on augmenting sequence models with structural modality, and we describe some of these works below.
Sequence models with structure loss. The ESM2-s sequence model incorporates structural infor- mation by fine-tuning ESM2 to predict a protein's 's structural fold (Zhang et al., 2024). The fold of a protein, however, biologically coarse- coarse-grained information. ISM achieves superior performance by using the more fine-grained approach training at the residue level. More specifically, our training objective, each residue is tasked with predicting its corresponding local structural environment.
2

---

Published as a conference paper at ICLR 2025
S-PLM and "Structure-innfee protein language models (SIPLM)" type of CLIP training to align sequence and structural features (Wang et al., 2023; Peñaherrera Koes, 2023). This technique is also searse-grained because its training objective does not operate residue level (we do not include SIPLM in our tables of results due to its relatively weak performance on our benchmarks).
AlphaFold also learns structural representations from sequences (Jumper et al., 2021). However, it requires a multiple sequence alignment as input, which expensive to compute and often unavailable for many practical applications. Furthermore, prior works have shown that Evoformer, the feature extractor for Alphafold, underperforms ESM2 on various downstream tasks that involve less structural information (Hu et al., 2022). On these tasks, ISM still achieves comparable performance to ESM2.
Sequence models with structure inputs. These models extend sequence models by using the structure as an additional input. SaProt (Su et al., 2023) and ProstT5 (Heinzinger et al., 2023) use the VQVAE from FoldSeek (van Kempen et al., 2022) extract per-residue structure tokens as additional inputs to protein language model. MULAN (Frolova et al., 2024) extends these works to include structural features (torsion angles) additional inputs. Similarly, ProSST (Li et al., 2024) also takes structural tokens as inputs. However, instead of using FoldSeek tokens, ProSST trains Denoising Autoencoder to extract per-residue features, which are then tokenized into structure sequence using K K-means clustering. All these models require protein structure as input at inference time. There are well-known drawbacks frameworks requiring structure input. In addition to requiring a more sophisticated data engineering pipeline, there are some cases where the structure has not been experimentally resolved and cannot be accurately modeled using computational tools (e.g., antibody-antigen complexes, conformer specific protein-protein interactions, post-translation modification-dependeen conformations, interfaces, etc).
Protein Structure Autoencoders. These autoencoders structure-based models that take the backhone atom coordinates as input and encode each residue into a discrete token (Gaujac et al., 2024; Hayes al., 2024). The sequence of discrete tokens is used reconstruct the positions of backbone atoms using coordinate losses (e.g., frame aligned point error, distogram classification). Protein structure denoising Autoencoders take noisy variant of the protein backbone as input and then learn a latent embedding that decodes the backbone atoms (Peñaherrera & Koes, 2023; Li et al., 2024). Foldseek (van Kempen et al., 2022) extracts features for a residue given the backbone geometry of its nearest neighbors. Unlike our approach, these works use only the protein backbone input. We also train a structural autoencoder, but instead of reconstructing the local backbone protein, we reconstruct the coordinates of all atoms within the local chemical environment surrounding masked residue (masked microenvironment)
3 PRELIMINARIES
Let Bsog = (II,,.., It) be a protein sequence of L amino acids where each amino acid residue Iı E CA,,,..., Y}. The atoms defined by this sequence fold into energetically favorable 3- dimensional structure Esuuet = (ps, es c;) ) where each atom i consists of residue sequence position pa {1,..., L), an element type e H,N, S, and coordinates e; R3
3.1 PROTEIN SEQUENCE MODELS
A protein language model pLM takes a protein sequence Tszg as input and produces a latent representation pLM( E IR 1x for downstream tasks. Most models use transformer archi- tecture and are pre-trained via a masked language modeling (MLM) loss. During training, subset
MI C (l,.., L} of the sequence is replaced with the [mask] token ã; = [mask] if M with &eey = (#1,..,,$L)) L.). The model learns to reconstruct the masked tokens with LMLM
otherwise
Σ TEM
LCD (CMMMLLM (iseg) x:))
M
for the cross entropy loss fCE, indexed feature (LM(@saa) E RD at position i, and a linear classification head CMLM that predicts the amino acid type. While the backbone pLM is used for downstream tasks, CMLM is only used for pre-training.
3

---

Published as a conference paper at ICLR 2025
3.2 PROTEIN STRUCTURE MODELS
An all-atom protein structure model pSM computes an atom-level feature representation from the local geometric description of each residue. It starts from a microenvironment ar micmeey that contains all atoms in a radius r 10Å around o 12, the coordinates of the a-carbon of residue l:
{(e,, c.) Vi (1.....N) NY such that ||c, -ull r).
microeny
A common backhone for protein structure models is Graph Transformer G' (Ying et al., 2021). The graph transformer G[ microe))) embeds each atom's 's element type e, in a set e = {e1, ..... eny where nt' is the number of atoms in the microenvironment, In attention updates, the graph transformer adds an attention bias Bi = c; cjll based on the pairwise distance between atoms and j. This attention bias B' is the only structural information given to the transformer. The graph transformer then produces set of output features {zi..., } G( ) ) one per input atom eş. The graph transformer is commonly trained on the downstream task using a supervised learning objective (Ying et al., 2021). this work, we use the Graph Transformer directly to train a structure model on atomic reconstructions of proteins in our pre-training dataset.
MutCompute (Diaz et al., 2024) pre-trains a Graph Transformer using a structural analog of masked language modeling. They define masked microenvironment x musked microenv that contains all atoms of other residues p: #
x maskedd
(le,, c.) Vi fi...., N) such that p; + and ||c, -all rr];
and pool all-atom level features into single residue level embedding z' = + # for [zí, ..., 2 = ((monsscccmmmmrommm)) where n is the number of atoms in the masked microenvironment. They then predict the masked-out amino acid type Nt.
Caa (CE(CAAZ) 2
where CAa is linear classification head.
MutRank (Gong et al., 2024) uses the EvoRank self-supervised training objective to learn the evolutionary mutational landscape of residue from the masked microenvironment. More specifically, it learns to predict an evolutionary score derived from the protein's 's multiple sequence alignment.
4 METHOD
ISM is sequence model that takes as input only amino acid sequence Tseq (xI,,.., IT. but is trained to implicitly capture structural information. We start by training an Atomic Autoencoder, based Graph Transformer, protein structures. The autoencoder trained with geometric reconstruction loss and the MutComputeX-GT objective Cl.. We then cluster the resultant features into one of K structure tokens. We use the sequence (s1,..., 8L) of structure tokens {1, K} as additional supervisory signal for the sequence-only Implicit Structure Model (ISM).
4.1
ATOMIC AUTOENCODER
Atomic Autoencoder uses an encoder-decoder architecture with a Graph Transformer encoder and plain transformer decoder. The encoder takes the masked microenvironment #mmsseed-mierem as input and produces atomic representations (z'..... } The decoder takes atomic representations in and produces features (fi.... f() which linearly project to atomic coordinates (ć) .... č (See Figure 2). This might seem like trivial task, after all the inputs -massoddmmeneen contain the regression targets. However, since the Graph Transformer only uses relative positions, and only in an attention bias B¹, the prediction tasks are quite difficult and require reasoning about the local structure of the microenvironment.
To obtain residue-level feature representation, average the atom-level features of the Graph Transformer x' = Σ z following Diaz et al. (2024). To train this representation, we add z into all atomic representations before the decoder. Mathematically, the transformer decoder takes {2f + ""..., x + x as input. We also found that adding this z' directly to the decoder architecture improves training stability. See Figure for full architecture.
4

---

Published as a conference paper at ICLR 2025
Atoms
(Pre-Train Only)
N
Encoder
Decoder
Pool
Structure Token
Distances
Cluster
Masked Microenvironment
[struct_ id 42]
Figure 2: Atomic Autoencoder learns a structural representation of residue's microenviron-
ment. The Autoencoder takes atom element types and pairwise distances input and reconstructs all atomic coordinates. The encoder is graph transformer that uses the pairwise distances to bias the attention mechanism learn rich atomic representations. The atomic representations are pooled to form microenvironment embedding. The decoder takes the atomic representations and microenvi- ronment embedding as input to decode the coordinates for all atoms. The learned mieroenvironment embeddings are discretized via K-means into structure tokens, which supervise the fine-tuning of protein language model. See Figure 5 for architectural details.
Training objective. One major challenge is that microenvironment lack robust protein backbone coordinate frames that underpin full protein models (Jumper et al., 2021; Hayes et al., 2024; Dauparas al., 2022). Unsurprisingly, does not take the coordinate frame into account and overestimates the loss. Thus, we optimize the MSE loss after global alignment. First, employ the Kabsch algorithm (Kabsch, 1976; Umeyama, 1991) to analytically compute the rotation and translation that minimize MSE loss. Then the loss is calculated using the transformed ground truth coordinates. Formally,
empirically observe that vanilla MSE loss SMSE 1 Σ. |è ct
min TER Σ Icc - (Rc, + +|]]
alligggeeed
During training, we observe that naive optimization the MSE-aligned loss results convergence to local optimum where all predicted coordinates lie 2-dimensional plane. Following AlphaFold (Jumper al., 2021), we addressed the issue using a distogram loss. Here, we use ESM3's distogram head by first computing fiz W.f! Woff, where W.. Wo are linear adapters. We then apply binned distance loss
C
n Σ LCE Chima dhind).
(Ce(C
where Caiso is a linear classification head that predicts the distance bin fhind between atoms and j at residue position I. During the first stage training, train with the distogram and masked modeling losses, C'sssto + C During the second stage, we additionally include MsE-aligneed
Generating Structure Tokens. Given a protein structure Estmets we start by generating the masked microenvironment for all residues, namely (an)
ar mineom We feed each masked microenvironment into our Graph Transformer encoder to extract residue-level feature representation at each position, (z e?....,²), We quantize z! for every residue in the protein using K- means (Lloyd, 1982) generate structure sequence (81, ..., st). In addition to our autoencoder, we also extract features (z
(aimakedd
z11) from MutRank (Gong et al., 2024) and generate second structure sequence s' (s) .......5 L both of which used identically to fine-tune the protein sequence model. Both models are trained on a smaller dataset experimental structures and are used generate structure tokens large dataset of AlphaFold structures.

---

Published as a conference paper at ICLR 2025
Table 1:
Comparisons on structural benchmarks. We freeze all protein models to assess the learned representation. ISM structure-tuned on the AlphaFold structures of Uniclust30 while ISMI undergoes additional Structure-tuning on PDB structures. SaProt" takes the protein structure as input. All other methods take sequence their only input. For contact, secondary structure, and binding residue prediction, the proteins in the training and test sets have at most 30% sequence similarity.
Structure Prediction (CAMEO) GDT-TS
Contact Med
SS Acc
Binding
Method
GDT-HA
LDDT
Short
Long
F1
MCC
Evolutionary pLM
Amplify (fumieeet 2124)
0.38
0.36
0.23
0.82
0.22
0.26
ESM2 (n 2027)
0.64 0.64
0.47
0.82
0.45
0.45
0.35
0.86
0.31
0.34
ESM2 (fine-tuned)
0.47
0.82
0.45
0.45
0.35
0.86
0.32
0.34
Structural pLM
ESM2-S (Zhang et ut. 2024)
0.61
0,43
0.79
0.46
0.47
0.36
0.85
0.32
0.35
S-PLM etal.
0.61
0.44
0.80
0,48
0.49
0.36
0.86
0.29
0.32
SaProt (St 2013)
0.57
0.53
0.48
0.86
0.36
0.38
ISM (Ours)
0.67
0.50
0.83
0.61
0.60
0.49
0.89
0.35
0.37
ISM i (Ours)
0.67
0.50
0.84
0.62
0.60
0.48
0.89
0.37
0.38
4.2
STRUCTURE-TUNE THE PROTEIN SEQUENCE MODEL
We initialize a sequence-only protein language model trained using masked language modeling (i.e., ESM2) and fine-tune it to predict the structure tokens. We call this training structure-tuning and the resulting model Implicit Structure Model (ISM). We append a linear classification head Cstraet to the output of the pLM backbone to predict the structural token. The structure prediction loss function is
Strract
struct (CE(C strrctt (LM( 84),
YES
where Tsey is the amino acid sequence with masked residues, pLM is the protein language model backbone, p is the representation for residue 8; is the structure token at residue i, and S are the positions at which the loss is computed. In standard MLM, the loss computed for all masked positions (i.e., S M). We found that predicting structure tokens all positions (i.e., S = [I,,,,, L)), and not just masked positions, better distills structural representations.
We structure-tune
model AlphaFold protein structures. AlphaFold sometimes produces inac- curate structures with poorly folded areas showing few interactions. Our structure token visualization reveals that many of these problematic residues are grouped into a single token ([struct id 17] in Figure 3). To ensure data quality, we exclude microenvironnents assigned the s* token from sequence model training. We compute Lsmutt at positions
{i: f1,..., L} and s; + s*).
The final training objective for structure-tuning is the sum of structure token(s) and amino acid cross-entropy losses (see Section 3.1), namely L = Lsmet + LMIM-
5
RESULTS
5.1
IMPLEMENTATION DETAILS
Atomic Autoencoder. Our microenvironment-ban Atomic Autoencoder is a Graph Transformer encoder with 4 layers and vanilla Transformer decoder with 2 layers. Our autoencoder training dataset contains 35K proteins from the Protein Data Bank(PDB). We train both stages for 5 epochs with learning rate of x 10 See Table 5a for a list of hyperparameters.
Distillation Dataset. Once our autoencoder is fully trained, we extract per-residue microenvironment features for 5.8M proteins from Uniclust30 with AlphaFold structures (Mirdita et al., 2017), along with 35K PDB proteins. We identify cluster centroids by applying K-means clustering to features from the PDB database, then assign features to tokens based on their distances to these centroids. The number of clusters, K 64, is chosen using the elbow method. Additionally, extract per-residue microenvironment features from MutRank and cluster the features into one K 512 tokens (see Section 3.2).

---

Published as a conference paper at ICLR 2025
Table 2: Comparisons on S669 Single Mutation Thermodynamic Stability prediction. We compare ISM to state-of-the-an methods that take various modalities as input. The middle and bottom block approaches are fine-tuned on DONA117K, which consists of mini-proteins that have at most 30% sequence similarity with those in S669. UR50: UniRef-50 used in ESM2 pretraining, UR 100: UniRef-100, PDB: Protein data bank, UC30: Uniclust30. OAS: Observed Antibody Space. SCOP
Structural Classification of Proteins. r. Spearman correlation coefficient.
Method
PreTrain Data
I,
AUC
MCC
RMSE
FoldX (Shyykmmiii at 2005)
N/A UR-50
0.27
0.62
0.14
2.35
PROSTATA (Unereskov stal, 2022)
0,50
0.73
0.28
1.44
Amplify (Foumier stal
UR PRRIO,.OA.SCOO
0.42
0.66
0.21
1.52
S-PLM (Wang atal. 2013)
UR50,S SwissProt
0.41
0.68
0.18
1.53
SaProt 2023)
UR50,UC30
0.49
0.71
0.25
1.47
ESM3 (Hayes at..
0,46
0.70
0.26
1.49
Stability Oracle (Dir a sl., 2024)
PDB
0.53
0.75
0.34
1.44
Eerrthhnn (ESM) (Ouyang Zhang et 2024)
UR-50
0.47
0.72
0.31
1.48
FurrtchWe (AF) (haymng 7036
PDB
0.56
0.76
0.35
1.38
ESM (line-tuned)
UR-50.PDB+UCK.
0.49
0.72
0.25
1.47
ISM (MutRank only)
UR50,PDB+UCG
0.51
0.74
0.33
1.45
ISM (MutRankx2))
UR50,PDB+UCC. UC33
0.50
0.73
0.32
1.45
ISM
UR50,
0.49
0.73
0.33
1.47
ISM
UR50,
0.52
0.74
0.30
1.45
ISM (Ours)
UR50.PDB+UC30
0.53
0.76
0.40
1.44
peructure-tuning We structure-tune the 650M parameter ESM2 for 20 epochs using cosine learning rate schedule with 4 warmup epochs. We use a total batch size 1536 proteins cropped to a maximum sequence length of 512 amino acids. We use AdamW optimizer with learning rate of 1 x 10 4 and weight decay of 5 x 10-3, Training takes 26 wall-clock hours on 32 GH200 GPUs. See Table 5b for complete list of hyperparameters.
5.2 COMPARISONS ON STRUCTURE TASKS
Rich sequence representations should inherently capture a protein's fold. In Table 1, we evaluate the structure- ennuuree representation of ISM against established methods several structure-basee downstream tasks, including structure, contact, secondary structure, and binding residue prediction. We evaluate all models as frozen feature extractors and learn decoding head. For structure prediction, we initialize from pre-trained SoloSeq (Ahdritz et al., 2022), replace the ESM2 backbone model with a frozen protein model, and tune the folding head. For other downstream tasks, freeze the backbone model and train shallow head. Contact, secondary structure, and binding residue prediction are evaluated using sequence similarity splits of 30%, 25%, and 20% respectively. More dataset descriptions are listed in Section D. ESM (fine-tuned) follows the same training regimen as ISM, but trained only with masked language modeling. We report results for ISM trained on Uniclust30 alone and Uniclust3++PDB.
Our model outperforms all sequence-only models and matches structure-sequence models on all fructure-based benchmarks. Notably, on long-range contact prediction, ISM outperforms ESM2 by 40%, with precision of 0.49 against 0.35. This matches the performance of SaProt (0.48), which explicitly requires the structure as input while ISM is sequence-only model. On structure prediction, ISM outperforms ESM2 by 5% on the GDT-TS metric (0.67 vs 0.64). On binding residue prediction Fl metric, ISM performs similarly with SaProt's 0.36, achieving 0.35 when trained Uniclust30 and 0.37 when trained on Uniclust30+PDB. Overall, the structure-enricher representations of ISM improve performance various structure-based downstream tasks compared to sequence-only pL.ss and structural pLMs.
5.3 COMPARISONS ON MUTATION STABILITY EFFECT
Thermodynamic stability is important phenotype that often needs be improved during the engineering of a commercially viable protein (Diaz et al., 2023; Liu et al., 2024; Carceller et al., 2024). We evaluate how effectively ISM predicts the impact of single mutations protein's
7

---

Published as a conference paper at ICLR 2025
Table 3: System-level Comparisons of transfer learning to various functional benchmarks. We fine-tune all models with shallow head for each benchmark (except HumanPPI, in which we freeze ISM due to overfitting). reports the best checkpoint found during training.
GO
DeepLoc
Thermostability
HumanPPI Metal Bind
EC
Method
MF
BP
CC
Subcell. Binary
Spearman
Acc
Acc
Fmax
Fmax
Fmax
Fmax
Acc
Acc
ESMIb
0.71
0.82
0.74
0,87
0,66
0.45
0.47
0.80
0.92
MIF-ST
0.69
0.76
0.75
0.81
0.63
0.38
0.32
0.79
0.92
ESM2*
0.70
0.88
0.74
0.87
0.67
0.49
0.51
0.85
0.94
SaProt
0.72
0.88
0.79
0.88
0.65
0.49
0.51
0.85
0.93
ISM
0.71
0.89
0.75
0.88
0.67
0.47
0.52
0.84
0.93
thermodynamic stability (AAG) on the S669 dataset (Pancotti et al., 2022) in Table 2. We evaluate all pL (ESM, Amplify, S-PLM, ISM) identically by fine-tuning the model with a shallow decoder head as in EuraateEe (Ouyang-Zhan -Zang et al., 2024). We fine-tune on the cDNA117K dataset from Diaz et al. (2024), a subset of the cDNA display proteolysis dataset (Tsuboyama et al., 2023) where all proteins have at most 30% sequence similarity to those in S669.
ISM outperforms all existing models that take single sequence as input, achieving Spearman correlation of 0.53 compared to Mutate Everything (ESM)'s ' 0.49, and an AUC of 0.76 compared to Mutate Everything (ESM)'s 0.72. Additionally, ISM matches the performance of state-of-the-art models while only using the amino acid sequence input, achieving an AUC of 0.76, while Mutate Everything (AF) and Stability Oracle achieve AUCs of 0.76 and 0.75, respectively. ISM also runs 20x faster on a protein of 300 amino acids. Note that Stability Oracle (Diaz et al., 2024) takes the atomic microenvironment input and Mutate Everything (AF) (Ouyang- -Zaang et al., 2024) takes multiple sequence alignment as input.
We validate the effectiveness of Atomic Autoencoder by comparing with ISM variants structure-tuned with one or two independently trained MutRank models. By incorporating Atomic Autoencoder, ISM's Spearman correlation increases from 0.50 0.53 and the AUC increases from 0.73 to 0.76.
We conducted an ablation study on the datasets used for structure- gruectue-tuning and were surprised to find that training on the smaller PDB dataset enhances downstream A AG performance more than training on the larger Uniclust30 dataset. Specifically, ISM achieves Spearman correlation of 0.49 when trained on UniClust30, compared 0.52 when trained on PDB. Even though the supervision signal during structure-tunie tuning is derived solely from the atomic coordinates in the structure and not AAG labels, we suspect the PDB dataset has some overlap with the structures in the S669 dataset, resulting in performance similar to that of structure-input models. Overall, on the $669 A AAG test set, I.M is competitive and even outperforms SOTA structure- based methods and AlphaFold's representations, a feat sequence-only PLMs have yet to achieve.
5.4 COMPARISONS ON A DIVERSE SET OF FUNCTIONAL PHENOTYPES
Functional characterization of proteins through biochemical techniques is typically the most resource- intensive type of labeled data to generate, making accurate transfer learning predictions particularly valuable for downstream bioinformatics and protein engineering and design tasks (Yu al., 2023; Allman et al., 2024; Kulikova et al., 2021). In Table 3, we evaluate ISM on the PEER (Xu et al., 2022) and FLIP (Dallago al., 2021) benchmarks, which encompass tasks that benefit from structural representations (e.g., thermostability),, evolutionary representations (e.g., biological process), or both (e.g.. EC). We fine-tune all models with shallow readout head on all benchmarks, except HumanPPI, for which we perform linear probing on ISM to prevent overfitting, We observed that longer training leads to overfitting, therefore, we evaluate various training checkpoints and report the highest performance for ESM2, SaProt, and ISM. ESMIb (Rives et al., 2019) and MIF-ST (Yang et al., 2023) results are sourced from SaProt (Su et al., 2023).
ISM performance remains competitive with ESM2 and other pLMs on functionally diverse tasks and does not stand out. For example, for predicting gene ontology molecular function (GO-MF), both ISM and ESM2 achieve 67% accuracy while SaProt achieves 65%. This finding aligns with
8

---

Published as a conference paper at ICLR 2025
Table 4: ISM ablation experiments. Default settings are marked in grey. See Section 6.1. ss:
Secondary Structure prediction, me: MutCompute, mr: MutRank, ac: Atomic Autoencoder
(a) Other Structure Tokens
(b) Our Structure Tokens
(c) Number of clusters
tokenizer
contact
ss
bind
tokenizer
contact
ss
bind
K
contact
bind
toldseck
0.42
0.88
0.32
ac
0.38
0.88
0.35
32
0.27
0.84
0.33
esm3
0.18
0.85
0.11
mr
0,46
0.88
0.34
64
0.48
0.89
0.37
me+mr
0.45
0.88
0.36
mr
0.52
0.88
0.36
128
0.42
0.85
0.37
ae+mr
0.48
0.89
0.37
ac+m
0.48
0.89
0.37
(d) Pre-training Crop length
(e) Label Type
(f)
Initialization
crop
val acc
contact
label
contact
(AAG)
init random
val acc
contact
32
0.27
0.27
features
0.36
0.49
0.36
0.10
128
0.36
0.42
tokens
0.46
0.51
esm2
0.40
0.48
512
0.40
0.48
prior work (Hu et al., 2022), which demonstrates that ESM2 outperforms Evoformer, the feature extractor for AlphaFold, on some functional tasks. It seems that for these functional tasks, the evolutionary signal from masked language modeling is sufficient and does not necessarily benefit from AlphaFold representations. Nonetheless, these experiments demonstrate that the structure- enriched representations of ISM do not corrupt ESM2's evolutionary representation on various function-based downstream tasks while enhancing ESM2's structural understanding.
ANALYSIS
6.1 ABLATIONS
We ablate key design decisions by reporting long-range Precision at L (P@L) for contact prediction, accuracy for secondary structure prediction, Fl for binding residue prediction, and Spearman correla- tion for AAG prediction in Table 4. We also report the validation accuracy, indicating how often the ISM variant correctly predicts the structure token derived from Atomic Autoencoder.
Structure Tokens. In Table 4a, we distill from various structure models from the literature. We compare against variant using both MuCComputeX-ee GT (mc) and MutRank (mr) structure models. Since Atomic Autoencoder uses the MLM loss Cs, from MutComputeX- this variant determines the effect of dropping the autoencoder from structure-tuning ISM. Our model outperforms MutRank and MutComputeX--C indicating that the autoencoder provides important structural information.
We found that tructure-tuning with ESM3's VQVAE (Hayes et al., 2024) structure tokens do not produce robust structural representations. A model structure-tuned with ESM3 achieves 0.18 and 0.11 contact and binding residue prediction, compared to 0.48 and 0.37 for ISM, respectively. We observe that the accuracy of ESM3 structure token prediction on a held-out validation accuracy on UniClust30 is 88, while Atomic Autoencoder accuracy 400% and MutRank accuracy is 447%. We suspect that the large vocabulary of ESM3's VQVAE (4096 structure tokens) results in redundant and overlapping tokens that are difficult discern and complicate loss optimization.
We also evaluate the performance of our sequence model structure-tuncd on FoldSeck VQVAE structure tokens (van Kempen al., 2022). We train on a larger subset of UniClust30 obtained from SaProt (Su et al., 2023) for the same number of iterations in ISM. The model achieves long-range contact P@L of 0.42 and binding residue F1 score of 0.32, which are improvements over ESM3 structure tokens and surpasses the ESM2 baseline (FI scores of 0.35 and 0.31, respectively). However, representations learned from FoldSeek's VQVAE structure tokens lag behind ISM (0.48 and 0.37). Thus, the structure tokens from Atomic Autoencoder and MutRank produce better structure representations, their combination being the most effective (see Table 4b).
Training parameters. We evaluate how the maximum length of a sequence during structure-tuning affects the accuracy and downstream performance in Table 4d. When the crop length is dropped to 128 and 32 amino acids, the contact long-range P@L L drops from 0.48 to 0.42 and 0.27 respectively. This shows that training with longer sequences is essential for learning long-range contacts.
Additionally, we evaluate the effectiveness of clustering MutRank representations 2 = (x',..., x²) E BIx into tokens (s1...., $L) E (1...,K) in Table 4e (excluding Atomic Autoencoder supervision). Our model variant uses a linear head to predict the MutRank representations z and is trained with normalized MSE loss. Direct MutRank representation prediction achieves 0.36

---

Published as a conference paper at ICLR 2025
P@L, while token ID prediction reaches 0.46 P@L long-range contact prediction. Clustering the MutRank representations potentially removes superfluous high-frequency noise.
Evolutionary Pre- Treiningn We evaluate the significance of training with MLLM before structure tuning in Table 4f by initializing with random weights. This approach resulted in decreased accuracy of structure tokens from 40% 36%, On downstream contact prediction, training from scratch drops long-range P@L from 0.48 to 0.1. This highlights the importance of gructure-tuning pretrained ESM2 as opposed to cructure-tuning from scratch.
6.2 QUALITATIVE VISUALIZATIONS
In Figure 3, visualize atomic structures of microenvironment grouped by structure token id. Specifically, we examine tokens [struct id 3] and [struct id 17], which are the least and most frequently observed tokens in Uniclust30, respectively. We find that microenvironments of the same structure token are semantically related. For example, [struct id 3] contains semi- exposed residues. Interestingly, [struct id 17] includes both solvent-exposed residues from experimental structures and unfolded residues from AlphaFold structures. These findings motivate us exclude ruct 17] from our structure-tuning training objective (see Section 4.2). Additional visualizations and analysis are provided in Section E.
(a) PDB Structures with [struct id 3]
(b) AlphaFold Structures with [struct id 3]
(c) PDB Structures with [struct 17]
(d) AlphaFold Structures with [struct 17)
Figure 3: Cluster-based Microenvironment Visualizations. Residues in sky blue are within the microenvironment, while white residues are outside and included for context. The grey density indicates the masked-out amino acid. Nodes colored by element: blue for nitrogen, red for oxygen, and yellow for sulfur. The left two columns display structures from the PDB, while the right two columns show protein sequences from Uniclust30, folded using AlphaFold. [struct id 3] contains semi-solvent exposed residues. ruct id 17] contains solvent exposed residues.
6.3 RUNTIME
SaProt
ISM (Ours)
ColabFold
418 s
We compare runtime against SaProt (Su et al., 2023) on three proteins with 91, 355, and 689 amino acids. The transformer is prediction (Mirdita et al., 2022) dominates the runtime. Even with structures, the ISM runs 2.4x faster than SaProt which additionally runs FoldSeek (van Kempen et al., 2022) to tokenize the structure.
FoldSeek
43 ms
A40 GPU. Colabfold structure
Transformer
ms
28ms
Figure 4: Runtime comparison.
ACKNOWLEDGEMENTS
This work is supported by the NSF AI Institute for Foundations of Machine Learning (IFML) and UT-Austin Center for Generative AL. We would like to thank AMD for the donation of computational hardware and support resources from its HPC Fund. We acknowledge the Texas Advanced Computing Center at The University of Texas at Austin for providing computational resources (Vista cluster) to support this work.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Gustaf Ahdritz, Nazim Bouatta, Sachin Kadyan, Qinghui Xia, William Gerecke, Timothy O' Donnell, Daniel Berenberg, Ian Fisk, Niccolò Zanichelli, Bo Zhang, Arkadiusz Nowaczynski, Bei Wang. Marta M Viepniewska-Dziubinkka Shang Zhang, Adegoke Ojewole, Murat Efe Guney, Stella Biderman, Andrew M Watkins, Stephen Ra, Pablo Ribalta Lorenzo, Lucas Nivon, Brian Weitzner, Yih-En Andrew Ban, Peter K Sorger, Emad Mostaque, Zhao Zhang, Richard Bonneau, and Mohammed AlQuraishi. Openfold: Retraining alphafold2 yields new insights into its learning mechanisms and capacity for generalization. bioRxiv, 2022. doi: 10. 10.11012022.11.20.57222 7,
16
Brent Allman, Luiz Vieira, Daniel Diaz, and Claus O Wilke. systematic evaluation of the language-of-viiaalessaa model using multiple machine learning frameworks. bioRxiv, 2024. 1, 8
Christian B Anfinsen. Principles that govern the folding of protein chains. Science, 181 181 223-230, 1973.
Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. Proteinbert: universal deep-learning model of protein sequence and function. Bioinformatiis, 38(8), 2022. 1,2
Jose M Carceller, Bhumika Jayee, Claire G Page, Daniel G Oblinsky, Gustavo Mondragón-Solórzaan,, Nithin Chintala, Jingzhe Cao, Zayed Alassad, Zheyu Zhang, Nathaniel White, Daniel J Diaz, Andrew D Ellington, Gregory D Scholes, Sijia S Dong, and Todd K Hyster. Engineering photoenzyme to use red light. Chem, 2024. 7
Christian Dallago, Jody Mou, Kadina E Johnston, Bruce J Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K Yang. Flip: Benchmark tasks in fitness landscape inference for proteins. bioRxiv, pp. 2021-11, 2021. 8
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep learning caasing-baseed based protein sequence design using proteinmpnn. Science, 378(661 5):46666, 2022. 5
D. J. Diaz, C. Gong, J. Ouyang-Zhang, J. M. Loy, J. Wells, D. Yang, A. D. Ellington, A. G. Dimakis, and R. Klivans. Stability oracle: A structure-based graph-transformer for identifying stabilizing mutations. Nature Communications, 15:6170, 2024. doi: 10.1038/s4 147-0444-979800
4, 7,
Daniel Diaz, Anastasiya V Kulikova, Andrew D Ellington, and Claus O Wilke. Using machine learning predict the effects and consequences of mutations in proteins. Current Opinion in Structural Biology, 78, 2023. 7
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Toward understanding the language of life through self-super learning. IEEE transactions puttern analysis and machine intelligence, 44(10):112--1227, 2021. 1,2
Quentin Fournier, Robert M Vernon, Almer van der Sloot, Benjamin Schulz, Sarath Chandar, and Christopher James Langmead. Protein language models: Is scaling necessary? bioRxiv, pp. 2024-09, 09, 2024. 6, 7, 17
Daria Frolova, Marina Pak, Anna Litvin, Ilya Sharov, Dmitry Ivankov, and Ivan Oseledets. Mulan: Multimodal protein language model for sequence and structure encoding. bioRxiv, pp. 2024-05, 2024. 1,3 3
Benoit Gaujac, Jérémie Donå, Liviu Copoiu, Timothy Atkinson, Thomas Pierrot, and Thomas D Barrett. Learning the language of protein structure. arXiv preprint arXiv:2405.15840 2024. 1,3
Chengyue Gong, Adam Klivans, James Madigan Loy, Tianlong Chen, Daniel Jesus Diaz, al. -oolution-inspieed loss functions for protein representation learning. In Forty-first International Conference Machine Learning, 2024. 2, 4,
Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with language model, bioRxiv, pp. 0724-07, 2024. 2, 5,

---

Published as a conference paper at ICLR 2025
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Milot Mirdita, Martin Steinegger, and Burkhard Rost. Bilingual language model for protein sequence and structure. bioRxiv, pp. 2023-02 2023. 3
Mingyang Hu, Fajie Yuan, Kevin Yang, Fusong Ju, Jin Su, Hui Wang, Fei Yang, and Qiuyang Ding. Exploring evolution-aware &-free protein language models as protein function predictors. Advances in Neural Information Processing Systems, 35:38873-38884, 2022. 3,9
John Jumper; Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-555
2021. 3,5
Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors, Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32(5):922-923, 1976.
Michael Schantz Klausen, Martin Closter Jespersen, Henrik Nielsen, Kamilla Kjaergaard Jensen, Vanessa Isabell Jurtz, Casper Kaae Soenderby, Morten Otto Alexander Sommer, Ole Winther, Morten Nielsen, Bent Petersen, al. Netsurfp-2.0: Improved prediction of protein structural features by integrated deep learning. Proteins: Structure, Function, and Bioinformatics, 87(6): 520-527, 2019. 17
Anastasiya V Kulikova, Daniel Diaz, James M Loy, Andrew D) Ellington, and Claus O Wilke. Learning the local landscape of protein structures with convolutional neural networks. Journal of Biological Physics, 47(4):435-454, 2021. 8
Anastasiya V Kulikova, Daniel J Diaz, Tianlong Chen, T Jeffrey Cole, Andrew D Ellington, and Claus O Wilke. Two sequence-and two structure-based ml models have learned different aspects of protein biochemistry. Scientific Reports, 13(1), 2023.
Mingchen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Liang Hong, and Pan Tan. Prosst: Protein language modeling with quantized structure and disentangled attention. bioRxiv, pp. 2024 04, 2024. 1,
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Faral-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences the scale of evolution enable accurate structure prediction. bioRxiv, 2022. 1, 2, 6, 17
Maria Littmann, Michael Heinzinger, Christian Dallago, Konstantin Weissenow, and Burkhard Rost. Protein embeddings and deep learning predict binding residues for various ligand classes. Scientific Reports, 11(1):23916 2021. 17
Yi Liu, Sophie G Bender, Damien Sorigue, Daniel J Diaz, Andrew D) Ellington, Greg Mann, Simon Allmendinger, and Todd K Hyster. Asymmetric synthesis of a-chloroamides via photoenzymatic hydroalkylation of olefins. Journal of the American Chemical Society, 146(11), 2024. 7
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2): 129-137, 137, 1982.
Milot Mirdita, Lars Von Den Driesch, Clovis Galiez, Maria J Martin, Johannes Söding, and Martin Steinegger. Uniclust databases of clustered and deeply annotated protein sequences and alignments. Nucleic acids research, 45(D1): D170.. 2017. 6
Milot Mirdita, Konstantin Schütze, Yoshitaka Moriwaki. Lim Heo, Sergey Ovchinnikov, and Martin Steinegger. Colabfold: making protein folding accessible to all. Nature methods, 1966:677-682,
2022. 10
Jeffrey Ouyang -Zhang Daniel Diaz, Adam Klivans, and Philipp Krähenbühl. Predicting protein's stability under a million mutations. Advances in Neural Information Processing Systems, 36, 2024. 7, 8
12

---

Published as a conference paper at ICLR 2025
Corrado Pancotti, Silvia Benevenuta, Giovanni Birolo, Virginia Alberini, Valeria Repetto, Tiziana Sanavia, Emidio Capriotti, and Piero Fariselli. Predicting protein stability changes upon single- point mutation: thorough comparison of the available tools on new dataset. Briefings in Bioinformatics, 23(2):bbab555, 2022. 8
Daniel Peñaherrrra and David Ryan Koes. Structure-infused protein language models, bioRxiv, 2023.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019. 16
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2019. doi: 10.1101/622803. 1, 2,
Joost Schymkowitz, Jesper Borg, Francois Stricher, Robby Nys, Frederic Rousseau, and Luis Serrano. The foldx web server: an online force field. Nucleic acids research, 33(suppl_2) W38822-3388 2005. 7
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan, Saprot: Protein language modeling with structure-aware vocabulary. bioRxiv, 2023. 1, 3, 6, 7, 8, 9, 10, 16, 17
Kotaro Tsuboyama, Justas Dauparas, Jonathan Chen, Elodie Laine, Yasser Mohseni Behbahani, Jonathan Weinstein, Niall M Mangan, Sergey Ovchinnikov, and Gabriel J Rocklin. Mega-scale experimental analysis of protein folding stability in biology and design. Nature, 2023. doi: 10. 1088/441586-022--668866 URL https: /tt.i.tggg/ooo/./////// 1038/
Dmitriy Umerenkov, Tatiana Shashkova, Pavel V Strashnov, Fedor Nikolaev, Maria Sindeeva, Nikita V Ivanisenko, and Olga L Kardymon. Prostata: Protein stability assessment using trans- formers. bioRxiv, pp. 2022-12, 2022.
Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions Puttern Analysis & Machine Intelligence, 13(00))376-380, 1991. 5
Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Foldseck: fast and accurate protein structure search.
Biorxiv, pp. 2022-02, 2022. 3, 9, 10
Duolin Wang, Mahdi Pourmirzaei, Usman L Abbas, Shuai Zeng, Negin Manshour, Farzaneh Esmaili, Biplab Poudel, Yuexu Jiang, Qing Shao, Jin Chen, et al. S-plm: Structure-aware protein language model via contrastive learning between sequence and structure. bioRxiv, pp. 2023-08, 2023. 3, 7, 17
Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Ma Chang, Runcheng Liu, and Jian Tang. Peer: comprehensive and multi-task benchmark for protein sequence understanding. Advances in Neural Information Processing Systems, 35:35156-351773 2022. 8, 16, 17
Jianyi Yang, Ambrish Roy, and Yang Zhang. Biolip: a seemi-mauually curated database for biologically relevant
interactions. Nucleic acids research, (1(D1):1109-D1100) 2012. 17
Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. Protein Engineering, Design and Selection, 36, 2023. 1,8
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie Yan Liu. Do transformers really perform badly for graph representation? Advances in neural information processing systems, 34:28877-28888, 2021. 4
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao. Enzyme
function prediction using contrastive learning. Science, 379(6639): 1358-1333, 2023. 8
Zuobai Zhang, Jiarui Lu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. Strueture-informed protein language model. arXiv preprint 0502.8560000 2024. 2,
13


Reviews:
[{"review_id": "VRCwZ14wKl", "reviewer": "Reviewer_edho", "summary": "This paper introduces a novel training strategy for protein language model, which distills structural information into protein sequence models. The main contribution of this paper is the Implicit Strucutre Model (ISM). It utilizes a micorenrironment-based Autoencoder for generating structure tokens and designs a training loss to achieve the self-supervised traning objective of distilling structure tokens. The method demonstrates comparative performence on stractural benchmarks and downstream finetuning tasks.", "questions": "Protein LMs such as AlphaFold and ESMFold take protein sequences as inputs and have the ability to generate structure information. It may indicate that these models also captures structure information implicitly without introducing an \"ISM like\" module and finetuning strategy. Thus, it makes me confused on the contribution of the method for the community of computational biology.\n\nThe authors may directly compare ISM's performance to AlphaFold and ESMFold on structure-related tasks, explain how ISM's approach differs from or improves upon the implicit structure learning in those models.\n\nIt would be benifit if the authors could clarify what unique capabilities or advantages ISM provides compared to existing protein language models.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The microenvironment-based autoencoder enables the generation of structure tokens with implicit structure information;\n2. The authors conducted comprehensive evaluations on various tasks and compared with SOTA methods;\n3. The method is compatible with any framework built using ESM2.", "weaknesses": "1. The performence improvement presented in Table 2 and 3 seems limited. For instance, it seems that MutateEverything (AF) demonstrates better results than ISM even with fewer pretrain data (PDB only); moreover, in table 3, ISM does not demonstrate superior performance than ESM2 and SaProt.\n2. The poteintial impact on the community of protein studies is not clearly presented. The authors could better articulate the potential impact, such as providing examples of how ISM could enable new applications or analyses in protein research that were not previously possible with existing models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i1UJpktXlt", "reviewer": "Reviewer_XC56", "summary": "This paper proposed a protein sequence model enhanced by structure distillation.", "questions": "As noted in Weakness 2, most questions arise from the design choices. What is the rationale behind these specific design decisions, and to what extent do they contribute to performance improvement?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "- The main idea of this paper—integrating protein structure information into a protein sequence model—is clear and direct. However, this concept is not entirely novel and has been explored in previous works, such as “DistilProtBert.”", "weaknesses": "- The pretraining task for the Atomic Autoencoder lead to data leakage, raising concerns about its effectiveness.\n- Most design choices lack justification. For instance, it is unclear why a two-stage training process was chosen, why multiple loss functions are used, or why ablation studies are absent.\n- The performance improvement appears minimal. For example, in comparison to ESM-2, the structure prediction accuracy shows only a slight increase (lddt score: 82 vs. 84).\n- The overall contribution of this paper is unclear. The main idea is not novel, the design of the Atomic Autoencoder lacks clarity, and the proposed models offer only marginal performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FjWRy41mzT", "reviewer": "Reviewer_eTdZ", "summary": "Though AlphaFold2 and its successors have revolutionized protein structure prediction---along with other tasks that require rich embeddings of protein residues---the computational expense of the multiple sequence alignments (MSAs) these models require as input has spurred interest in so-called \"single-sequence\" models: language models conditioned only on the sequence of the protein in question, and no other structural or co-evolutionary information. So far, these have lagged behind their coevolutionary counterparts on most benchmarks. In an attempt to narrow the gap, the authors propose Implicit Structure Modeling (ISM), which finetunes protein language models with an additional structural loss term for quantized structure tokens. The resulting models can be used as a drop-in replacement for popular single-sequence models but perform better at tasks that require knowledge of structure.", "questions": "Why do the authors use a custom structure autoencoder and not, say, Evoformer embeddings?\nCould you give more details about the PDB dataset used? E.g. what is the cutoff?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "The paper is clearly written (with some exceptions---see below), well-motivated, and is fairly comprehensive, including results for most important baselines as well as ablations and analysis. The method is interesting, and it's nice that it's compatible with ESM software.", "weaknesses": "While the ISM model compares favorably to others, like SaProt and ESM2, I have two gripes: 1. the difference in performance compared to e.g. SaProt seems fairly marginal across the board, with the strongest improvements in short- and medium-range contact prediction and one of the two thermostability benchmarks (but not the other), and 2.  given the results of the ablations in Table 4, most of the heavy lifting seems to be done by MutRank, not the novel autoencoder. I think I'll need to see a few things before I consider raising my score:\n\n1. Full results of the MutRank model in Tables 1 and 2.\n2. Some kind of confidence intervals for the results in Table 4b. It's unclear to me that the small gap there isn't just noise or the result of \"ensembling\" two different but closely related methods. What if you just trained two independent MR models and ensembled them, for example?\n3. More details about how exactly MutRank is incorporated into the model.\n\nThere are a few other small details missing as well. Given that the ISM model in Table 1 descends from SoloSeq, you should include SoloSeq results there as well. I think it would also be useful here to include SOTA results (e.g. AlphaFold3), just to illustrate how close ISM is to closing the gap.\n\nRandom bits and bobs (no bearing on score):\n- The citation for the sentence \"MutRank adds a self-supervised...\" is wrong.\n- There's a typo in Table 1 (Strucure)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)