SaProt: Protein Language Modeling with Structure-aware Vocabulary
Authors: Jin Su
Chenchen Han
Yuyang Zhou
Junjie Shan
Xibin Zhou
Fajie Yuan
Venue: ICLR 2024 spotlight (2024)
Keywords: Protein Language Models
Universal Representations
Downstream Tasks
Protein Structure Modeling
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Large-scale protein language models (PLMs), such as the ESM family, have achieved remarkable performance in various downstream tasks related to protein structure and function by undergoing unsupervised training on residue sequences. They have become essential tools for researchers and practitioners in biology.  However, a limitation of vanilla PLMs is their lack of explicit consideration for protein structure information, which suggests the potential for further improvement. Motivated by this, we introduce the concept of a ``structure-aware vocabulary" that  integrates residue tokens with structure tokens.    The structure tokens are  derived  by encoding the 3D structure of proteins using Foldseek. We then propose SaProt, a large-scale general-purpose PLM trained on an extensive dataset comprising approximately 40 million protein sequences and structures. Through extensive evaluation, our SaProt model surpasses well-established and renowned baselines across 10 significant downstream tasks, demonstrating its exceptional capacity and broad applicability. We have made the code, pre-trained model, and all relevant materials available at https://github.com/westlake-repl/SaProt.

SAPROT: PROTEIN LANGUAGE MODELING WITH PRRUCTURE-AWARE VOCABULARY
Jin Chenchen Han², Yuyang , Junjie Shan?, Xibin Zhou?, Fajie Yuan
Zhejiang University', Westlake University²
(sujin, hanchenchen, zhouyuyang, shanjunjie, zhouxibin, leeeeaiiieeeeesstkk.eeeeee
ABSTRACT
Large-scale protein language models (PLMs), such as the ESM family, have achieved remarkable performance in various downstream tasks related to pro- tein structure and function by undergoing unsupervised training quences. They have become essential tools for researchers and practitioners in biology. However, limitation of vanilla PLMs their lack of explicit consid- eration for protein structure information, which suggests the potential for further improvement. Motivated by this, we introduce the concept of "structure-awar vocabulary" that integrates residue tokens with structure tokens. The structure to- kens are derived by encoding the 3D structure of proteins using Foldseek. We then propose SaProt, large-scale general-purpe sive dataset comprising approximately 40 million protein sequences and struc- tures. Through extensive evaluation, our SaProt model surpasses well-established and renowned baselines across 10 significant downstream tasks, demonstrating its exceptional capacity and broad applicability. We have made the code") pre- trained model, and all relevant materials available at attttps:/ httpt/ggttttb/////// com/ west. Best.aae-220 allakkeerpppp take-repl/SaPrge
residue se-
PLM trained an exten-
INTRODUCTION
Proteins are fundamental to biological functions, and understanding them opens promising avenues in medical, pharmaceutical, and genetic research. Protein Language Models (PLMs), drawing inspi- ration from NLP methodologies, have emerged as the pivotal technology for representing proteins (Rao et al. 2019). Through self-supervised training on vast amounts of protein ID residue se- quences, PLMs have proven highly proficient in capturing long-range residue correlations, i.e., co- evolution (Anishchenko et al. [2177] Rao al. 2020). Moreover, prominent PL.Ms like UniRep (Al-
ley et al.] 2019), ProtTTrans (Flnaggar et al.) 2021). ESM (Rives et al. [019 Meier et al. 2021 Rao ctt al. [021]] Lin et al., 20022. ) and Evoformer (Hu et al., 2022] Jumper et al. 2021) have showcased outstanding performance across diverse array of tasks pertaining protein structure and function.
Despite the success of residue sequence- based pre-training, there's growing interest in utilizing protein 3D structures as training data, given their direct relevance to functions. Some work have demonstrated the potential of pre-training on experimentally determined protein structures (Yang et al. 2022, Hermosilla & Ropinski 2022), but they are limited by the smaller number of highly accurate structures compared residue sequences. Meanwhile, the breakthrough achieved by Al- phaFold2 (AF2) (Jumper et al. (022) in protein structure prediction has resulted in substantial repository of structure data (Varadi et al. 202T), thus igniting interests utilizing large-scale pro- tein structures for training PLMs.
Work done at Westlake University.
*Corresponding Author. Fajie conceived and supervised this research. Jin conducted this research. Jin, Junjie, Fajic proposed the new vocabulary together by discussing. Chenchen performed the mutational effect prediction task. Yuyang and Xibin collected the dataset. Jin, Fajic wrote the paper.
Unlike ESM models that only offer inference code, provide code for both training and inference.

---

Currently, the development of structure- PLMs based on large-scale predicted structures is still in an early stage, and existing research has certain limitations. For instance, well-known models like GearNet (Zhang et al., 2023b), still depend on limited set of protein structures, utilizing around 800 thousand predicted structures from AF2. On the other hand, models like ESM-IF (Hsu et al.) [2022) focus exclusively on specific protein tasks, such as protein inverse folding, rather than aiming for broader and more general-purpose representations.
In this paper, we aim to contribute to the biological community by introducing a large and more powerful PLM trained on extensive protein sequence and structure data. To achieve this, we intro- duce a "structure-award (SA) vocabulary" that encompasses both residue and structure information of proteins. Specifically, we can employ vector quantization techniques (Van Den Oord et al. 2017) to discretize protein structures into 3D tokens. These tokens, similar in format residue tokens, capture the geometric conformation information of each residue in relation to its spatial neighbors. Here, we simply utilize Foldseek (van Kempen et al. 2023). purpose- bhilt tool. Then, by combin- ing the 3D tokens with residue tokens, we devise a very infuitive yet innovative vocabulary termed the SA alphabet. This enables the conversion of the original residue sequence into an SA-token se- quence, serving as the input for existing residue-based PLMs. Through unsupervised training on massive protein SA-token sequences, we obtain a Structure-aware Protein language model named SaProt. To assess its performance, we comprehensively evaluate its capabilities across 10 widely recognized protein tasks. These tasks encompass broad range of applications, including clinical disease variant prediction (Frazer et al. (021), fitness landscape prediction (Dallago et al.) [202T] He et al. 2024), protein-proteio interaction (Nooren & Thornton 2003), as well as diverse protein function predictions (Bileschi et al.) 2022 Yu al. 2023). To summarize, our main contributions are as follows:
We introduce structure-aware vocabulary that combines residue and 3D geometric fea- ture for proteins. With the utilization of "SA" tokens, proteins, encompassing both pri- mary and tertiary structures, can be effectively represented as a sequence of these novel tokens. The sequential nature, rather than the grapt& structure, of protein representation allows for seamless integration with advances in large-scale foundation Al models, such as BERT (Devlin et al. 2018), BART (Lewis et al. 2019), GPT (Brown et al. 2020). etc.
By utilizing the SA-token protein sequence as input, we train a structure- structure-enhanced PLM using the ESM (Lin et al. 2022) backbone case study, called SaProt. To our knowledge, SaProt stands out as the PLM currently trained with the largest number of protein structures, containing 650 million parameters. Its training lasted 3 months and utilized 64 NVIDIA 80G A100 GPUs, with a computational cost similar to ESM-lb (Rives et al. 2019).
We evaluate SaProt across 10 renowned biological tasks. SaProt consistently exhibited improved performance compared to strong baselines, particularly models from the ESM family, including ESM-1b, ESM-1v 1v (Meier et al., 2021), and ESM-2 (Lin et al. 2022). which are considered leading PLMs in the field.
We conduct a series of enlightening ablation studies, unveiling previously unknown find- ings. One such finding is the potential overfitting issues that may arise when training PLMs by integrating predicted structures with BERT-style training. This discovery highlights crucial consideration in the design of protein structure-based PLMs. Additionally, our ex- perimental section sheds light several intriguing observations through dissecting SaProt.
Additionally, we have made our code, model weight, and the associated datasets openly available. These materials are expected be valuable for both the computational and biological communities.
2 RELATED WORK
2.1 RESIDUE SEQUENCE- BRQUENCE-BASED PRE-TRAINING
Sequence- pre-training methods treat protein residue sequences as natural language, enabling comprehensive representations via masked language modeling (MLM) (Devlin et al.) 2018). For- mally, a protein sequence is denoted as P ($1, $2, ..., sn), where s; is a residue at the 4eh position
Prior work employs GNNs for protein structure modeling, but GNNs suffer from the over-smoothing is- (Huang al. 2021 Chen al. 2020), therehy hindering large and deep protein model development.
2

---

Residtae Vocabbulary
K
Mask
Residue +
Tokenize
Residuee Sequence
Prodiction Head
Foldseek
BERT/ESM
He
Embeddiing Layer
Md
Masks
Takenize
MO
TO
Mask Takens
Librown
Input
Structure aware Sequence
Vocabulary
Figure
Framework SaProt with the structure-aware vocabulary
and n. is the sequence length. During pre-training, set of residues are randomly masked, resulting the modified sequence Pmas = (51, < M ASK ,,...,,,). The training objective is to predict masked residues by capturing dependencies between masked positions and surrounding context.
residue-based PLMs have shown potential in generating universal protein representations. Riiess et al. (2019), Heinzinger et al. (2019) and Vig et al. (2020) substantiate the ability of PLMs to predict protein structures and functions. while Rao et al. (2021) enhances capabilities via training on Multiple Sequence Alignment (MSA) data. For mutational effect prediction, [Meier et al. (202I) and
He al. (2024) adopt ESM-1v for zero-shot prediction, and Notin et al. (2022) incorporate MSA as supplementary signals. Additionally, [Lin et al.] (2022), Chowdhhry et al. (2022) and Wu et al. (202bb) predict protein structures from single sequences by applying large PLMs.
2.2 BTRUSTURE-BASED PRE-TRAINING
Protein structure governs function. The release 200 million protein structures in Al- phaFoldDB (Varadi at. 2022 in July 2022 enables the construction of large-scale protein structure models. Protein structures are usually represented as graphs, denoted by g = E), with V rep- resenting the set of N residues and & representing the of edges connecting the residues. These edges are typically based on the Co distances between the residues. GNNs utilize G for diverse pre- training strategies like contrastive learning (Hermosilla Ropinski 2022 Zhang et al., 2023b a) self- self-prediction (Yang et al. [ Chen et al. (022)) and denoising score matching (Guo et al. 2022 Wu et al. 2022ap. Another way inspired by AF2 involves incorporating structure features as contact biases into the attention maps within the self-attention module, e.g., Uni-Mol (Zhou et al. 2023
However, the above structure-basee models rely either real structures from the Protein Data Bank (PDB) or limited number of predicted AF2 structures. To the best of our knowledge, there are currently "general- PLMs based large-scale set of predicted structures.
2.3 FOLDSEEK
The initial goal of Foldseck (van Kempen al.) 2022) is facilitate fast and accurate protein structure searches. To achieve this, Foldseck employs a VQ-VAE model (Van Den Oord et al., 2017 for encoding protein structures into informative tokens. These tokens, derived from 20 distinct 3Di states, are represented as P
f2.... fa), where fa represents the structure token at the it position and n is the sequence length. Foldseek achieves this encoding by identifying nearest neighbors and extracting features for individual residues.
A preprint by Heinzinger al. 2023) introduces ProstT5, which enables bidirectional conversion between residue and Foldseck token sequences. ProstTS excels at tasks like remote homology de- tection and protein design. However, is not considered general-purpose PI. (see Appendix A).
3

---

Attention Map
Structure Blas
GNN
Structure-awwe aware tokens
14
ESM layer
Mc Ev Vp Op Vy Ky
ESM leyer
MEVOLVOYK
5.0
3.5
3.0
4.5
valeerDsi
traincAF2 valistAF21 validiPoSs
Lots
2.0
1.5
3.0
15
2.5
2.0
SteplK)
SteplK)
Step(K)
Evoformer-inspired PLM
MIF
SaProt
Figure 2: Loss trends for three protein structure models. The training set is AF2 structures while in the validation set, one AF2 structures and the other comprises real structures from PDB.
IDEA OF NEW VOCABULARY
3.1 PRELIMINARY ANALYSIS
The goal this paper is to develop general-purpose PLM by leveraging predicted protein struc- tures to serve multiple protein prediction tasks. Contrastive learning (CL) and BERT-style MLM training are currently two most prevalent pre-training approaches. However, CL primarily empha- sizes protein-level representation learning and performs poorly the residue-level task. For instance, GearNet (Zhang et al. 2023b) and 3D-PLM (Hermosilla & Ropinski 2022) trained by CL not directly useful for predicting effects of amino acid mutations (Frazer al. (021).
We initially explored two intuitive approaches for protein structure modeling. The first approach involves treating the predicted structures from AF2 as graph and employing GNNs for modeling, following (Yang et al. 2022 The second approach extract the distance and angle information between pairwise residues from structures, incorporating it a structure bias in Transformer (Vaswani al. 2017) attention map. This approach was applied by Uni-Mol, ESMFold (Lin al.) 2022) and Evoformer. We evaluate the two models using the MLM objective it support both protein-level and residue-level tasks. It should be noted that the structure model in Uni-Mol, ESMFold, and Evoformer were initially designed for specific tasks with different loss functions, rather than being intended as general- general-purpose purpose PLM. Therefore, remains uncertain whether these neural networks would be effective when trained with predicted structures using the MLM objective.
Through two exploratory experiments, we noted that training directly using predicted structures yielded poor performance on the validation set containing real PDB structures (Figure 2 The decrease loss on predicted structures did not correspond to a decrease in loss on real structures. This mismatch may be due to the fact that PLM has detected traces of AF2 predictions. Furthermore, inferior results were reported in downstream tasks (Table 8). Despite a substantial loss decrease on training data, these models failed to learn meaningful representations for downstream protein tasks.
3.2 STRUCTURE-AWAE VOCABULARY
Inspired by the above discoveries, we aim to incorporate protein structures from a novel perspective. Our key idea revolves around creating structure-aware (SA) vocabulary, where each SA token encompasses both residue and structure information, as illustrated Figure
2 Note that MIF Yang al. 2022) utilized only structures for pre-training, the massive predicted structures from AF2 would heneficial or not.
unclear whether
As hasic analysis, we utilized the 35M version of ESM-2 (see Appendix E.1.1 and SaProt. The MIF is consistent with the one described in the original paper, with size of 3.4M.
4

---

Given a protein P. its primary sequence can be denoted ($1, S2...., S,), where 8, V represents the residue at the in site, and V represents residue alphabet. Building upon the concept of Foldseek, can introduce alternative approach for representing protein tertiary structures by using a vector quantized variational autoencoder (Van Den Oord et al. 2017). This approach enables us to develop structure alphabet F, wherein P can be represented as the (S1, f2; -2.-... fn) sequence, with fj F denoting the structure token for the Jth residue site. To maintain simplicity, we directly adopt the default setup of Foldseek, which defines the size m of F 20.
Now, we can combine the residue and structure tokens per residue site, generating new structure- aware sequence P (1. s2.2..... fn)n), where s:fs E X is the token fusing both residue and geometric conformation information. The structure-aware sequence can then be fed into a standard Transformer encoder basic input. It's important to note that we also introduce mask signal "#" to both residue and structure alphabet, which results in #s and "# f. that indicate only residue structure information is available. The size of the SA vocabulary is 21 x 21 441 (see Figure I
The design of this new vocabulary simple yet innovative and fundamental, enabling the represen- tation of any residue sequence using this "SA" sequence. As a result, protein models that utilize residue sequences as input can effortlessly integrate the new vocabulary sequence as substitute.
3.3
SAPROT
3.3.1 MODEL ARCHITECTURE
SaProt employs the same network architecture and parameter size as the 650M version of ESM- 2. The main distinction lies in the expanded embedding layer, which encompasses 441 SA tokens instead of the original 20 residue tokens. This nearly identical architecture enables straightforward comparisons with the ESM model. Moreover, the model size strikes balance between performance and feasibility for downstream task training, avoiding excessive memory or computation cost.
3.3.2 OBJECTIVE FUNCTION
We train SaProt using the BERT-style MLM objective, similar ESM-1b and ESM-2, en- abling the support for both protein-level and residue-level tasks. Formally, For a protein se- quence P ($1f1, .2. the input and output can be represented as: input (s1fi,..., fi,..., fnfn) - output: : s:fi (see Figure D}. fi in # #fi is made visible during training to reduce the model's emphasis on predicting it. This is different from the straightforward masking strategy, i.e. randomly masking SA token f.fe by ## and then predicting both residue and struc- ture token directly from the SA vocabulary (see Appendix Figure 7). We do not adopt this strategy because the SA tokens may be not accurate enough Predicting the exact SA tokens may lead the model in the wrong optimization direction. With the proposed masking objective, although there are still inaccuracies in certain Foldseek tokens, the global structure information should remain effec- tive, which provides valuable context for the prediction. From this perspective, it is more reasonable predict the residue tokens rather than the Foldseck structural tokens both of them. We perform the empirical study on the two masking strategies during pre- pre-training in Appendix
To ensure a fair comparison, SaProt was pre-trained using identical training strategies with ESM- 2 (refer to Appendix Cp. We build the pre-training dataset, which consists of approximately 40 million AF2 structures. Details included in Appendix B. including how to proceed with the lower PLDDT region.
EXPERIMENTS
We evaluate SaProt across 10 diverse downstream tasks, encompassing residue-level and protein- level tasks. Given that many proteins the original datasets lack experimentally determined struc- tures, we conduct all evaluations using predicted structures obtained from AlphaFoldDB without special mention. Furthermore, proteins without structures in AlphaFoldDB will not be utilized in all our experiments.
'The accuracy of SA takens depends on the accuracy of both AF2 and Foldscek.
5

---

4.1 ZERO-SHOT MUTATIONAL EFFECT PREDICTION
4.1.1 DATASETS
We adopt the ProteinGym NNtin al. 2022) benchmark and Clin Var (Landrum et al. 2018 dataset used in Frazer et al. (2021) to evaluate the performance of SaProt on the zero-shot mutational effect
prediction tasks Meier al. 2021 For dataset details, we refer readers to Appendix D.2.1
Dataset
ESM-2
ESM-Iv ESM-1b Tranception ESM-IF MIF-ST EVE MSA Transformer
SaProt
VlinVar
0.862
0.891
0.900
0.845
0.748
0.891
0.878
0.854
0.909
ProteinGymm(w/o MSA retrieval)
0.475
0.448
0.440
0.413
0.409
0.474
0.478
PruteinGym(ww MSA retrieval)
0.479
0.472
0.472
0.465
0.425
0,480
0.477
0.464
0.489
Table 1: Zero-shot mutational effect prediction. Clin uses AUC (area under the ROC curve) and ProteinGym uses Spearman's p as evaluation metric. They are two distinct biological tasks.
4.1.2 BASELINES & EVALUATION
We compare SaProt with two types of baselines: sequence-based models and structure-based mod- els. For sequence-basee based models, we include ESM-1b (Rives et al.] 2019), ESM-1v (Meier et al.] 2021 (the results of ESM models are averaged), ESM-2 650M (Lin et al. 2022 6 and Trancep- tion L (Notin et al. 2022). For structure-based models, we consider the MIF-ST (Yang et al. 2022)
and ESM-IF (Hsu al. 2022). Additionally, we present the performance of EVE (Frzer al. 2021 renowed model that leverages MSA information for predicting disease variant effects, and
MSA Transformer (Rao al., 202T). protein language model pre-trained on large scale of MSA data (we sample 384 homologous proteins for inference following Notin et al. al. (2022)). Here, we did not include comparisons with contrastive learning models like GearNet and 3D-PLM, as they are not directly applicable to residue-level zero-shot prediction tasks. Also note that with the exception of EVE on ProteinGym, all baseline models and their weights used in this study were obtained from the official paper. We solely employed them for prediction without any training. We trained EVE on ProteinGym ourselves using the official code as it necessitates training on each MSA.
Notin
We strictly follow the evaluation used in EVE (Frazer et al. 2021 for assessing the model's per- formance on the Clin Var dataset. For the ProteinGym dataset, we employ the evaluation measures described in (Notin et al. 2022 Meier et al. 2021). Details are provided in Appendix D.2.2
4.1.3 RESULTS
Table 1
shows the zero-shot results on ProteinGym & Clin Var, resulting in the below conclusions:
SaProt outperforms all residue sequence- based and structure- models on both tasks. As mentioned earlier, SaProt shares an identical network architecture, model size, and train- ing examples with ESM-2, with the key difference lying in its structure-aware vocabulary. By comparing SaProt with ESM-2, 2,, it clear that SaProt yields consistent improvement for predicting mutational effects. Then, SaProt shows higher accuracy compared to MIF-ST, even though the latter model was trained using experimentally determined highly accu- structures The benefit could be attributed the large-scale structures when training SaProt. ESM-IF exhibits the poorest performance in both tasks, primarily because it was originally designed for the inverse folding task. In addition, ESM-IF model size and train- ing data are nearly times smaller than SaProt.
MSA information enhances models' zero-shot ability. Notin al. (022) introduces a technique to enhance autoregressive inference by leveraging MSA information, leading to consistent improvement. Following it, we extend the technique to SaProt and all baselines. The results show that the integration of MSA information greatly enhances the zero-shot prediction ability of various PLMs, with SaProt still achieving the highest accuracy among them. The results also suggest that the improvement techniques used for residue sequence- based models are likely to be useful to SaProt as well.
"The results for 15B ESM-2 are reported in the Appendix D.2.3 which shows worse results.
MIF-ST exhibits poor accuracy when trained with AF2 structures, as shown in Figure & Appendix E.1.

---

4.2
SUPERVISED FINE-TUNING TASKS
GO
DeepLoc
Thermostability HumanPPI Metal Ion Binding
EC
Model
BP
CC
Subcellular
Binary
Spearman's
Acc%
Acc
Fmax
Fmax
Fmax
Fmax
Acc
Acc%
ESM-2
0.680
76.67
71.56
0.868 0.670 0.473 0.470
82.09
91.96
FSM-Ih
0.708
82.22
73.57
0.864
0.656 0.451 0.466
80.33
92.83
MIF-ST
0.694
75.54
75.08
0.807
0.633 0.375
0,322
78.96
91.76
GearNet
0.571
73.86
71.26
0.874
0.644
0.481 0.476
69.45
89.18
SaProt
0.724
86.41
75.75
0.882
0.682 0.486 0.479
85.57
93.55
ESM-GearNet
0.651
84.09
74.11
0.887
0.676 0.516 0.507
82.30
92.94
SaProt-GearNet
0.660
85.80
74.44
0.889 0.678 0.522 0.508
84.16
93.63
Table 2: Experimental results on downstream tasks.
4.2.1 DATASETS
For protein-level tasks, we evaluate SaProt on a diverse set of datasets from several bench- marks (Dallago et al. 2021 Xu et al. 2022 Rao et al. 2019), including predicting Thermostability, Metal Ion Binding, protein localization (DeepLoc), protein annotations (EC and GO) and protein- protein interaction (HumanPPI). Dataset description and splits are listed in Appendix D.3
4.2.2 BASELINES
In addition to the above baselines, compared SaProt GearNet (hang et al.) 2023b). Inspired by ESM-GearNet (Zhang et al.] (2023)), we replaced the ESM module RSMM-eerNee with SaProt, resulting in ensemble model called SaProt-GearNet. Training details are in Appendix D.3.3
4.2.3 RESULTS
Experimental results are illustrated in Table 2 shedding light on the following insights:
SaProt outperforms ESM-2 in all protein-level tasks. Specifically, SaProt shows remark- able enhancements over ESM-2 in the Thermostability. HumanPPi, Metal Ion Binding, and DeepLoc tasks. This outcome once again demonstrates that integrating structure informa- tion into PLMs leads to superior protein representation.
SaProt outperforms the two structure models, GearNet & MIF-ST, by substantial margin. This notable performance difference highlights the efficacy of structure modeling in SaProt.
While SaProt outperforms the ESM models, SaProt-GearNet also outperforms ESM- GearNet, which highlights the orthogonality of SaProt with more advanced improvement techniques. However, it is interesting note that combining two models does not always result in higher performance. For example, SaProt-GearN and ESM-GearNet do not nec- essarily surpass their respective single models SaProt and ESM.
SaProt exhibits superior performance across all tasks, when also considering the results in Sec- tion 4.13 Its impressive performance positions it compelling alternative to the ESM family.
5 ANALYSIS
We conduct insightful ablation studies by dissecting SaProt. One find more analysis in Ap- pendix E including comparisons of masking strategies, masking rates on structure tokens, etc.
5.1
AWARENESS OF PROTEIN STRUCTURE
SaProt incorporates protein structure information by using structure-awe aware tokens rather than using explicit 3D coordinates. However, this approach relies the accuracy of the Foldseek encoding.

---

Short Range
Medium Range
Long Range
Model
PGLL
P61122
P00125
P@01.
PG81.2
P48
P@L
P@11/
P611/
ESM-2
44.87
44.90
50.18
45.21
53,90
35.33
41.96
52.11
SaProt (Residue-only)
40.29
40.22
44.10
36.26
36.69
42.47
22.15
27.63
36.68
SaProt
57.11
57.20
63.71
53.43
55.05
66.45
48.14
59.75
74.32
Results on contact prediction. Short range, medium range and long range contacts are con- tacts between positions that are separated by 6 to 11, 12 23 and 24 or more positions, respectively.
Table 3:
Naturally, a question arises: does SaProt truly possess stronger structure information compared to ESM-2, given that residue-based PLMs also implicitly contain structure information (Rao et al.) 2020] To answer tion on the TAPE benchmark (Rao et al. 2019). For both SaProt & ESM-2, we freeze the backbone and solely fine-tune the contact head. The evaluation of contact map is conducted using PDB data.
conduct additional structure prediction task, namely contact map predic-
As shown in Table 3 SaProt exhibits remark- able superiority over ESM-2 in the contact map prediction task, evidencing that SaProt contains more accurate structural information. From this perspective, PLM with enhanced structure fea- ture is expected to exhibit improved accuracy in protein function prediction tasks. We ad- ditionally evaluate SaProt's performance with all structure tokens masked as "#" named "SaProt "Residue-only)". SaProt (Residue- only) performs worse than ESM-2 but still ex- hibits certain degree of structure prediction ability. This result demonstrates that SaProt capable capturing structural information even when the structure tokens are not given.
PrateinGym
0.49
Substibute Substrbute resiouc
0.48
10448
2.473
Spearman's
0.467
0.466
0.46
0.452
0.45
0.450
0.44
0.43
0,430
20
Substitution percentage(%)
To further study the impact of structure and residue tokens on SaProt's performance, we conduct an additional zero-shot prediction ex- periment. We randomly replace percentage of structure tokens with random (Foldseek) tokens while keeping the residues unchanged, and then we do the opposite for residue tokens. SaProt's performance evaluated under this setting. As shown in Figure 3 the accuracy of SaProt decreases when either residue tokens or structure tokens are randomly substituted, which clearly emphasizes the importance of both residue and structure tokens.
Figure 3: Results for different substitution per- centage on (structure/residue)) tokens.
5.2 PDB VERSUS ALPHAFOLDDE
For proteins with experimentally determined structures, it is essential to investigate how SaProt performs. do this, we continuously pre-train SaProt on 60,000 PDB structures, resulting a variant called SaProt-PDB. We conduct evaluations by assessing SaProt and SaProt-PDB on both AF2 structures and real PDB structures. We did not evaluate all tasks due to lack of many PDB structures some tasks.
Table shows that when trained solely on AF2 structures, the overall accuracy of SaProt is not largely affected by the choice between AF2 structures or PDB structures. However, for SaProt-PDB, is advisable to use PDB structures directly when available for downstream tasks. This may not have a substantial impact on supervised tasks such as EC and GO, as the model will be retrained on the downstream structures. However, it can have key impact on the zero-shot task, as indicated by the comparison of 0..454 vs. 0.423 when training/testing data is highly inconsistent for SaProt-PDB.
In general, SaProt exhibits slightly better performance achieves better accuracy on real structures. This outcome is expected as training and testing are consistent in the optimal performance setting. Note that some protein structures in PDB are not
AF2 structures, while SaProt-PDB
8

---

stored in AlphaFoldDB, so the column-level (AlphaFoldDB vs. PDB) comparison in Table does not make much sense. We have released SaProt-PDB weights for utilizing PDB structures.
AF2
PDB
Model
ProteinGym
EC
GO-MF GO-BP GO-CC
ProteinGym
EC
GO-MF GO-BP
GO-CC
SaProt
0.450
0.882
0.682
0.486
0.479
0.423
0.885
0.665
0.460
0.410
SaProt-PDB
0.448
0.880
0.679
0.482
0.472
0.454
0.888
0.669
0.465
0.415
Table Results of SaProt and SaProt-PDB on AlphaFoldDB and PDB structures. Proteins without PDB structures on the ProteinGym dataset were removed during evaluation.
5.3 VISUALIZATION
For a more intuitive comparison, we employ t-SNE to visualize the protein representations gen- erated by the last layer of SaProt and ESM-2. Figure 4 shows the visualization results using the non-redundant version (PIDE < 40%) of the SCOPe [Chandonia et al. 2018) database. For the alpha and beta proteins, the representations generated by ESM-2 are intertwined, whereas those gen- erated by SaProt are separated based on structure type. This observation again underscores SaProt's capability discerning structure changes. Furthermore, we visualized the embeddings of all 400 structure-awarc tokens (tokens that encompass "#" are ignored). As depicted in Figure 8 (c), can observe certain degree of clustering phenomenon. In the semantic space, the SA tokens that are in close proximity to each other often correspond to similar types of residues or Foldseck tokens.
ESM-2
SaProt
algha proteins
All beta proteins
alpha proteins
All beta proteins
Figure 4: Embedding visualizations of ESM-2 and SaProt SCOPe database.
CONCLUSION
In this study, we introduce novel structure-aware (SA) vocabulary that integrates primary and ter- tiary protein structure information into the SA-token. This SA-token-based sequence has the poten- tial to serve as a novel protein representation. Building upon this, we train general-purpose purposs PLM called SaProt, which achieves state-of-the-art performance 10 protein function prediction tasks. Like the ESM models, SaProt aims to contribute to the advancement of the biological community.
This study has several limitations: (1) The performance of the proposed SA vocabulary heavily depends on Foldseek, which aims to balance search efficiency and encoding accuracy. Therefore, there is still room for improving the representation capability of SaProt. (2) Due to computational constraints, the model size SaProt may not have reached its maximum capacity. (3) addition to the mentioned tasks, there are other applications that could be explored using the SA vocabu- lary. For instance, predicting protein complex structures by replacing two protein sequences with SA-token-bart sequences could automatically incorporate single-chain structure information. In protein generation tasks, generating SA-token sequences could potentially provide stronger struc- ture constraints during the generation process. These avenues remain open for future research.

---

ACKNOWLEDGMENTS
We thank Sergey Ovchinnikov for many insightful suggestions for the research. Sergey advised to evaluate the performance our model and ESM-2 on the contact map predicion task to evidence model's effectiveness in capuring more accurate structure information. Sergey also gave many suggestions for paper writing, figure presentation and research focus. His contributions are signif- icant and should also be listed as co-author. Due to abstract submission due, we cannot add his name as an author.
This work is supported by the National Key Research and Development Program of China (No. 2022ZD01 100001151000 15100), the National Natural Science Foundation of China (No. U21A20427), the West- lake Center of Synthetic Biology and Integrated Bioengineering (WE-SynBio), and the Research Center for Industries of the Future (N0.2020220030)
REFERENCES
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):135--332,, 2019.
José Juan Almagro Armenteros, Casper Kaae Sønderby, Søren Kaae Sønderby, Henrik Nielsen, and Ole Winther. DeepLoc: prediction of protein subcellular localization using deep learning. Bioinformatics, 33(21):3387-3399 07 2017. ISSN 1367-4803. doi: 10. 10.109//ooooffrmatiic 1093 ott.://gg.........o.ooooooooetee
btx431. URL http::ttptt/ttttctttttl
Ivan Anishchenko, Sergey Ovchinnikov, Hetunandan Kamisetty, and David Baker. Origins of co- evolution between residues distant in protein 3d structures. Proceedings of the National Academy of Sciences, 1 114(34):9222-1127, 2017.
Maxwell L Bileschi, David Belanger, Drew H Bryant, Theo Sanderson, Brandon Carter, D Sculley, Alex Bateman, Mark A DePristo, and I Lucy Colwell. Using deep learning to annotate the protein universe. Nature Biotechnology, 40(6):932-937, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
John-Mare Chandonia, Naomi K Fox, and Steven E Brenner. SCOPe: classification of large macro- molecular structures in the structural classification of proteins extended database. Nucleic Acids doi: 10.1093/nar/gky URL
Research, 47(D1):D455-D481, 11 2018 ISSN 0048.
htttps://doi. .rrlil..g/// www.oogggrooo..... 1093 nar /
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self- supervised learning, 2023.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 3438 3445, 2020.
Ratul Chowdhury. Nazim Bouatta, Surojit Biswas, Christina Floristean, Anant Kharkar, Koushik Roy, Charlotte Rochereau, Gustaf Ahdritz, Joanna Zhang, George M Church, et al. Single- sequence protein structure prediction using language model and deep learning. Nature Biotech- nology, 40(11):1617-1623, 2022.
Christian Dallago, Jody Mou, Kadina E. Johnston, Bruce J. Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K. Yang. Flip: Benchmark tasks fitness land- scape inference for proteins. bioRxiv. 2021. doi: 10.1101/2021.11 1...:2.........66.... 467890. URL https::
7/ www.bbooxxxv.ort/ccoteeettttttee /tw.://.......ooooooa///////a//.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpan. Science, 378(6615):499555 2022.
10

---

Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http://arxiv org/ http://bb//11111 04805
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of life's code through self-supervised deep learning and high performance computing, 2021.
Jonathan Frazer, Pascal Notin, Mafalda Dias, Aidan Gomez, Joseph K. Min, Kelly Brock, Yarin Gal, and Debora S. Marks. Disease variant prediction with deep generative models of evolutionary data. Nature, (99(:)3999977991999988819 Nov 2021. ISSN 1476-4687. doi: 10. (198//41586-021-044444 URL 1tLLnttpp..//...o/111...1.
021 04043-8
Vladimir Gligorijević, P. Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C. Taylor, lan M. Fisk, Hera Vlamakis, Ramnik J. Xavier, Rob Knight, Kyunghyun Cho, and Richard Bonneau. Structure- protein function prediction using graph convolutional networks. Nature Communications, 12(1): 3:688, May 2021. ISSN 2041-1723 doi: s4 s41467- 021 11122-23330 23303-9
URL htt.attsss//ao...g//////
Yuzhi Guo, Jiaxiang Wu, Hehuan Ma, and Junzhou Huang. Self-supervised pre-training for protein embeddings using tertiary structures. Proceedings of the AAAI Conference on Artifi- cial Intelligence, 36(6))6801-680) Jun. 2022. doi: 10. wwl.1609/aai..v000 php/AAAI
URL https:
AAAI Awt.AA//Aaaatee//a.aaaaaaaaaaaee /ttpaaaaiiieeeeeeee.eee.//
Yan He, Xibin Zhou, Chong Chang, Ge Chen, Weikuan Liu, Geng Li, Xiaoqi Fan, Mingsun Sun, Chensi Miao, Qianyue Huang, et al. Protein language models-assisted optimization of a uracil-n- glycosylase variant enables programmable t-to-g and t-to-c base editing. Molecular Cell, 2024.
Michael Heinzinger, Ahmed Elnaggar, Yu Wang, Christian Dallago, Dmitrii Nechaev, Florian Matthes, and Burkhard Rost. Modeling aspects of the language of life through transfer-learning protein sequences. BMC bioinformatics, 20(1):1-17, 2019.
Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Martin Steinegger, and Burkhard Rost. Prostt5: Bilingual language model for protein sequence and structure. bioRxiv, 2023, doi: 10. 00.1100//023.07.23.5555 23. 550085
URL https: :/t:://wwwww. http///wiinn. www.bbiorxiv.
otte//contennteaaal////22//////2
Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures, 2022.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. ICML, 2022. doi: 10.1101/2022.0..000 10.487777.
2022/04/10/2022.000 04. 487777
Mingyang Hu, Fajie Yuan, Kevin Yang. Fusong Ju, Jin Su, Hui Wang, Fei Yang, and Qiuyang Ding. Exploring evolution-aware &amp; -free protein language models as protein function predictors. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 38873-388888 38884. Curran Associates, Inc., 2022. URL https:://proceeeeiggg.eeeee
cc cc/paper wwweeerrrffeeeeeeee filess/pperr/202. fi $1e4ff86622222226222222222222222 Paweereeaee-areecceeeeeee .pdf
Zengfeng Huang, Shengzhong Zhang, Chong Xi, Tang Liu, and Min Zhou. Scaling up graph neu- ral networks via graph coarsening. Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 675-684, 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikoloy, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman,
11

---

Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with alphafold. Nature, 596(78733,5833-599, Aug 2021. ISSN 1476-4687. doi: 10. 1088//41586-001-33331 URL
0 1038/
-21-0011---00000 03819-2
Melissa J Landrum, Jennifer M Lee, Mark Benson, Garth R Brown, Chen Chao, Shanmuga Chi- tipiralla, Baoshan Gu, Jennifer Hart, Douglas Hoffman. Wonhee Jang, Karen Karapetyan, Ken- neth Katz, Chunlei Liu, Zenith Maddipatla, Adriana Malheiro, Kurt McDaniel, Michael Ovetsky, George Riley, George Zhou, J Bradley Holmes, Brandi L. Kattman, and Donna R Maglott. Clin- Var: improving access to variant interpretations and supporting evidence. Nucleic Acids Res, 46 (DI):D1062-D106) January 2018.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequene pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 2019.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
llya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, 2017. htt.httpp://aaxl..orrrg http://awwvv..ogg/////////////// 05101
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero- shot prediction of the effects of mutations on protein function. bioRxiv, 2021. doi: 10. 11w.1///2222000090555550500
URL WwR.attpss///.wiiio.tttti
wwww www.bbiwwxwv.oooooooooooooo /AR.Lt/Ss///.....O......//ooooo
10. 10.1101/202 07. 09. 450648v1
Irene MA Nooren and Janet M Thortton. Diversity of protein-protein interactions. The EMBO journal, 22(14))3486-333
2003.
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena-Hurtade Aidan Gomez, Debora S. Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval, 2022.
Xiao- Yong Pan, Ya-Nan Zhang, and Hong-Bin Shen. Large-scale prediction of human protein- protein interactions from amino acid sequence based on latent topic features, Journal of proteome research, 910))4992-5001, 2010.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John F. Canny, Pieter Abbeel, and Yun S. Song. Evaluating protein transfer learning with TAPE. CoRR, abs/1906.08230, 2019, URL
08230
Roshan Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. Biorxiv, pp. 2020-12, 12, 2020.
Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abheel, Tom Sercu, and Alexander Rives. Msa transformer. bioRxiv, 2021. doi: 02
URL 1ttp:/101111100000 2ttp:/22222........33333. 4308558v1
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2019, doi: 10.1101/622803, URL /ttpps://t/wwww..eeeee www.wwwbiiiii..oo org/ wttttet/t1..111/// 1w1.1110162220222
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Laurens der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma-
chine Learning Research, 9(86):2579-260) 2008. URL http:// / jmir. org/papers//s/
html
12

---

Michel van Kempen, Stephanie S. Kim, Charlotte Tumescheit, Milot Mirdita, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. bioRxiv. 2022. doi: 10.1 011.10002202200200..799999999999 2022/02/09/2222..2..000 479398
Michel van Kempen, Stephanie S. Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron L.. M. Gilchrist, Johannes Söding, and Martin Steinegger. Fast and accurate protein structure search with foldseek. Nature Biotechnology, May 2023. ISSN 1546- 1696. doi: 10. 1031/
341587-023-01773-0 /ttp://////ppp.//o.oooo///////// 1038 / 41584-188880 -2
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, Augustin Zidek, Tim Green, Kathryn Tunyasuvunakool, Stig Petersen, John Jumper, Ellen Clancy, Richard Green, Ankur Vora, Mira Lutfi, Michael Figurnov, Andrew Cowie, Nicole Hobbs, Pushmeet Kohli, Gerard Kleywegt, Ewan Birney, Demis Hassabis, and Sameer Velankar. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high- accuracy models. Nucleic Acids Research, 50(DI):D439 D444, 11 2021. ISSN 0305-1048. doi: 10. 10.1093/nar/gkab1006. httL:ttts:///d..oo///////// 1093/na. /qkab1061
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-aceuracy models. Nucleic acids research, 50(D1):D439-D444 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, 0bs//706.03762. 2017.
[RRLthttp///xiiit org/ /ttp////......... 03762
Jesse Vig, Ali Madani, Lav Varshney, Caiming Xiong, Richard Socher, and Nazneen Fatema Ra- jani. Bertology meets biology: Interpreting attention in protein language models. arXiv preprint
arXiv:2006. 15222, 2020.
Fang Wu, Shuting Jin, Yinghui Jiang. Xurui Jin, Bowen Tang, Zhangming Niu, Xiangrong Liu, Qiang Zhang, Xiangxiang Zeng, and Stan Z. Li. Pre- Pre-training of equivariant graph matching networks with conformation flexibility for drug binding. Advanced Science, 9(33):2203796. oct 2022a. doi: 10. 20.10022advs..22203796 URL htt.htts::/ /ttass:/docs. http:://dco.oogg//// 2ttp//1..10222Fadve 202203796
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng, Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pp. 2022-07, 2022b.
Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Chang Ma, Runcheng Liu, and Jian Tang. Peer: A comprehensive and multi-task benchmark for protein sequence understanding. arXiv preprint arXiv:2206.00 020966 2022.
Kevin K. Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. bioRxiv, 2022. doi: 10. 10.1101/0022.0.225 493516. URL
05. 25.25..5 493516]
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao. Enzyme function prediction using contrastive learning. Science, 37966639::1358-1366 2023.
Zuobai Zhang, Minghao Xu, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Enhancing protein language models with maneture-based encoder and pre-training. In Interna- tional Conference on Learning Representations Machine Learning for Drug Discovery Workshop, 2023a.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In International Conference Learning Representations, 2023b.
13

---

Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In The Eleventh International Conference Learning Representations, 2023. URL https:
7/ //openreview net//frrm? i d=622R66WVRR
14

---

A COMPARISON WITH PROSTT5
very recent preprint in Heinzinger et al. (2023) proposed ProstT5 which pre-trains the protein language model (PLM) a mixture of data of Foldseck token sequences and residue sequences. However, they mainly focused on the bi-lingual translation between Foldseek sequence and residue sequence. ProstT5 utilizes the protein residue sequence predict the corresponding Foldseek struc- ture tokens, and meanwhile, it also predicts the protein residue sequence given the Foldseek structure tokens.
In contrast to ProstT5, we introduce novel vocabulary that combines residue and Foldseck tokens, enabling the transformation of the residue sequence into an SA-token sequence. SaProt is trained on these new token sequences, effectively incorporating structure information. One acknowledged drawback of ProstT5, as stated by the original paper, its limited ability a general-purpose PLM, as it exhibits inferior performance in certain protein function prediction tasks.
We conducted several experiments to compare ProstT5 to SaProt, as shown Table 5 The exper- imental results show that SaProt consistently outperforms ProstT5 in these protein understanding tasks, and specifically ProsTT5 fails in zero-shot mutational effect prediction task.
DeepLoc
ClinVar
ProteinGymi(w/o MSA retrieval)
Model
Subcellular
Binary
AUC
Spearman's p
Acc%
Acc%
ProsTT5
0.620
0.155
81.36
92.84
SaProt
0.909
0.478
85.57
93.55
Table 5: Comparison of performance between ProstT5 and SaProt.
B PRE-TRAINING DATA PROCESSING
We adhere to the procedures outlined in ESM-2[Ling et al. (2022) to generate filtered sequence data, and then we retrieve all AF2 structures via the AlphaFoldDB website httts://alphafoo eebi.ac... uu/ based the UniProt ids of protein sequences, collecting approximately 40 million structures. By employing Foldseek, we encode all structures into the so-called 3Di tokens and pro- ceed to formulate structure-aware sequences by combining residue and 3Di tokens each position.
The AF2 structures in study are accompanied by confidence scores, referred to as pLDDT, which provide an assessment of the precision of atom coordinates. These scores can be utilized to identify and filter out regions with low accuracy. During the MLM (masked language modeling) pre- training process, if regions with lower PLDDT scores (<70 as the threshold throughout this paper) are selected for the MLM prediction, we will predict the "s,#" token, where its input position masked using the ## token (note that both "# and "s:#" are one token, see Figure D]. By doing so, models are forced to concentrate on predicting residue types in such regions. If regions with lower PLDDT scores not selected for the MLM prediction, the input will be entered with the "s.#" token, so that models will only use the residue context in these regions to aid the prediction of other tokens. For the prediction phase of downstream tasks, maintain complete consistency with the training data by handling the PIDDT regions. Specifically, tokens within these lower PLDDT regions represented "s:#", with only residue token visible.
C PRE-TRAINING DETAILS
Following ESM-2 and BERT, during training, 15% of the SA tokens in each batch are masked. We replace the SA token sefi with the #fx token 80% of the time, while 10% the tokens are replaced with randomly selected tokens, and the other 10% tokens remain unchanged. For the optimization of SaProt, we adopt similar syper-parametery to those employed in the ESM-2 training phase. Specifically,
employ the AdamW optimizer (Loshchilov & Hutter 2017). setting 81 0.9, B2 = 0.98 and we utilize L2 weight decay of 0.01. We gradually increase the learning rate from ( to 4e-4 over the first 2000 steps and linearly lower it to 5e-4 from 150K steps to 1.5M steps. The
15

---

overall training phase lasts approximately 3M steps. To deal with long sequences, truncate them to maximum of 1024 tokens, and our batch size consists of 512 sequences. Additionally, we also employ mixed precision training to train SaProt.
D EXPERIMENTS
D.I DATA COLLECTION
After the release of the AlphaFoldDB (Varadi et al. 2021), the majority of predicted structures now accessible through searching UniProt IDs. We record the UniProt IDs of all proteins and query the A retrieve all available predicted structures. On the other hand, several tasks need experimentally determined structures as training data (e.g. EC, GO and Metal Ion Binding). For these proteins, we map the PDB and chain IDs of proteins to their corresponding UniPort IDs (e.g. the chain "C" of PDB ID "6D56" will be mapped to UniPort ID "PO1 112"). It is worth noting that different PDB and chain IDs may correspond to the same UniProt ID (e.g. both "A" and "B" chain of PDB ID "6MAF" point to the UniProt ID "QSD6Y5"), and they typically represent different segments within the protein. Therefore, we truncate the AF2 structures to match the corresponding PDB structures.
D.2 ZERO-SHOT PREDICTION
D.2.1 DATASETS
ProteinGym (Notin et al. 2022) is extensive set of Deep Mutational Scanning(DMSS) assays, enabling thorough comparison among zero-shot predictors. Specifically, we utilize the substitution branch, filtering proteins with lengths exceeding 1024 or those without available structures in AlphaFoldDB. We download all AF2 structures based on UniProt ids. For evaluation, We adopt Spearman's 's rank correlation
metric.
Clin Var serves as a freely accessible and publicly available repository containing information about human genetic variants and interpretations of their significance to disease (Landrum et al. 2018). In our analysis, we harness the data sourced from EVE (Frazer et al.] 2021). additionally filtering out proteins with length greater than 1024 or absent from the AlphaFoldDB. To enhance the reliability of our data, we opt to consider proteins with labels I "Gold Stars" or higher, which indicate higher credibility. Following the methodology employed in EVE, we evaluate models' performance using the AUC metric.
For each mutation dataset, we provide all variants with the wild-type structure, as AF2 cannot differ- entiate the structural changes caused by single mutations. For proteins without predicted structures in AlphaFoldDB, we simply remove them during evaluation. This also applies to all supervised task (see Table
D.2.2 FORMULA
Previous residue-based PLMs like the ESM models predict mutational effects using the log odds ratio at the mutated position. The calculation can be formalized as follows:
Σ EE
Mlg s/rd logP( = = [))
Here I' represents all mutations and E V is the residue type for mutant and wild-type sequence. We slightly modify the formula above to adapt to the structure-aware vocabulary, where the proba- bility assigned to each residue corresponds to the summation of tokens encompassing that specific residue type, as shown below:
Ellog Σ P(a,= = s flaN)) - log Σ P(a,- 5. f[a(r)]
(2)
IET
/EF
FEF
Here f E F is the structure token generated by Foldseck and s,f E V. x F the structure-ane aware token in our vocabulary.
16

---

ProteinGym
Clin Var
Model
w/ MSA retrieval
w/ MSA retrieval
AUC
Spearman's
Spearman's
ESM-2 (15B)
0.843
0.436
0.479
SaProt
0.909
0.478
0.489
Table Zero-shot comparison with ESM-2 15B version. Clearly, ESM-2 15B does not improve its 650M version.
D.2.3 ADDITIONAL COMPARISON
We conducted additional experiments to compare SaProt the ESM-2 15B version. Our evaluation focused exclusively on zero-shot prediction tasks, given the GPU memory constraints associated with fine-tuning the ESM-2 15B model on these supervised downstream tasks. shown in T (I) SaProt outperformed ESM-2 15B on all zero-shot prediction tasks,(2) the larger model is not always better by comparing ESM-2 15B with ESM-2 650M. One possible reason for this could be that excessively large models may lead to overfitting issues
D.3 SUPERVISED FINE-TUNING
D.3.1 DATASETS
Protein Function Prediction We compile set of tasks that predict functions of proteins. Specif- ically, employ the "Human-cell" splits the Thermostability task from FLIP (Dallago et al. 2021). which predicts the thermostability value of proteins. Additionally, utilize the Metal lon Binding task (Hu et al. 2022), which designed to predict the presence of metal ion- bon-binding sites within protein.
Protein Localization Prediction We employ the DeepLoc (Almagro Armenteros et al. 2017) dataset to predict the subcellular locations of proteins. DeepLoc comprises two branches for sub- cellular localization prediction: one involving 10 location categories, and the other involving binary localization prediction with 2 location categories. We adhere to the original data splits.
Protein Annotation Prediction We make use of two established benchmarks introduced by Deep- FRI Gligonijević et al. 2021) to predict protein annotations encompassing multiple functional la- bels, Le. Enzyme Commission(EC) number prediction and Gene Ontology(GO) term prediction. For the GO benchmark, we incorporate all three branches: Molecular Function (MF), Biological Process (BP), and Cellular Component (CC).
Protein-Protein Interaction Prediction Protein-protenn interaction (PPI) prediction has great po- tential for wide application prospects. Here we employ HumanPPI(Pan et al. 2010) from PEER Xu ctt al. 2022) benchmark to predict whether two proteins interact or not.
Mutational effect Prediction We employ the Fluorescence prediction and Stability prediction tasks from the TAPE (Rao et al. 2019) benchmark, the AAV dataset from the FLIP (Dallago et al.) 2021) benchmark and 3-lactamase landscape prediction from the PEER (Xu et al., 2022) bench- mark. These datasets encompass mutants derived from wild-type proteins, signifying the absence of available structures.
Protein Structure Prediction We adopt the contact prediction task from TAPE Rao et al. 2019 investigate SaProt's awareness to protein structure.
D.3.2 DATASET SPLIT
With the exception of the Metal Ion Binding and DeepLoc tasks, we utilize the official data split in the related benchmark literature (TAPE (Rao et al. 2019), PEER (Xu et al. (002)) and FLIP (Dallago et al. 202I)), which includes separate training, validation, and testing sets. Identity clustering and filtering was conducted these benchmark datasets. For the Metal Ion Binding dataset, perform clustering and split the data into training, validation, and testing sets based on 30% sequence identity.
Note that the original dataset used in Hu et al. (2022) did not have identity clustering the training
17

---

Dataset
Category
Evaluation Metric
Train
Valid
Test
Protein Function Prediction
Thermostability (Dallago et al. 2021
Human-Cell
Spearman's p
5056
639
1336
Metal Ion Binding (Hu al. 2022
Acc%
5067
662
665
Protein Localization Prediction
Subcellular
Acc
8747
2191
2747
DeepLoc Almagro Armenteros al. 2017
Binary
Acc
5477
1336
1731
Protein Annotation Prediction
EC Gligorijević 2021)
Fmax
13089
1465
1604
GO
Gligorijjvvi et al.
2021
BP MF CC
Fmax
26224
2904
3350
Protein-Pran Protein Interaction Prediction
HumanPPI (Xu et al.
2022
Acc%
26319
234
180
Mutation Effect Prediction
Fluorescence Rao al.
2019
Spearman's p
5235
25517
Stability Ruo et al., 2019
Spearman's ' p
53614
2512
12851
AAV (Dallago et al. 2021
2-vs-rest
Spearman's p
22246
2462
50432
8-lactamase Xu et al., 2022
Spearman's 's
4158
520
520
Protein Structure Prediction
Contact Prediction Rao ef al. 2019
P@L
25299
224
40
Table 7: Downstream dataset descriptions after all data pre-processing. Category represents spe- eific branch of the dataset. Note that proteins whose structures not found in AlphaFoldDB have been removed for all baseline models during both training and testing evaluation.
and test sets. As the DeepLoc dataset was already clustered by 30% sequence identity, we randomly split out 20% samples from the training set as the validation set.
We summarize the dataset details in Table
D.3.3 TRAINING DETAILS
In order to perform fair comparisons, we assessed our model and all baselines with the same set of hyper-parameters,
employed the AdamW optimizer, setting 31 0.9, .2 0.98 and utilized L2 weight decay of 0.01. We consistently used a batch size of 64 and set the learning rate to 2e-5 (except le-3 for contact prediction). We fine-tuned all model parameters until convergence and selected the best checkpoints based on their performance on the validation set.
E ANALYSIS
E. PRE-TRAINING COMPARISON
E.1.1 EYOFORMER-INSPIRED PLM
Evoformer (Jumper al. 2021 integrates both sequence and structure information through pro- jecting structure features as biases and incorporating them into the attention maps within the self- attention module. Nevertheless, the updates of structure features in Transformer layers are extremely time-consuming which is infeasible to large-scale pre-training. Therefore, we simplify the interac- tion modules Evoformer and employ on standard ESM-2 model architecture. Specifically, We remove 4 triangle modules(i.e. Triangle update using outgoing edges, Triangle update using in- coming edges, Triangle self-attention around starting node and Triangle self-attention around ending node) and keep the Outer product mean module and the Transition module to enable the updates of structure features. For preliminary experiments, we adopt ESM-2 35M as base model and add above modules on it to form Evoformer-inspiree PLM. We follow ProteinMPNN (Dauparas ctt al. 2022) to extract distance and angle features from protein structures as biases to be incorporated into the attention maps.
18

---

DeepLoe
ClinVar ProteinGym(w/o MSA retrieval)
Model
Subcellular
Binary
AUC
Spearman's p
Acc%
Acc%
Exoformer-inspired ESM333 ESM(35M)
0.589
0.178
66.11
89.25
MIF
0.638
0.256
61.90
85.90
335Mot(35M)
0.754
0.319
76.63
91.45
Table 8: Downstream task results for the three structure-based models. All structures used were
predicted by AF2.
E.1.2 EXPERIMENTAL RESULTS FOR THREE MODELS
We assessed the performance of three models, i.e. Evoformer-inspired PLMM (ESM-2 35M version), MIF and SaProt 35M version, zero-shot prediction and supervised fine-tuning tasks. The results Table are aligned with the loss change in Figure For the supervised DeepLoc task, all models perform well as they are fine-tuned with new labels. Even if the pre-training is not useful, the performance after fine-tuning on new data can still be relatively good.
E.2 MASK RATE OF STRUCTURE TOKENS
PDB structures' chains can be linked to proteins in the UniProt database, yet these structures nor- mally constitute segments of UniProt proteins. For instance, the UniProt id "P05067" corresponds to protein containing 770 residues, while the chain "A" within the protein of PDB id "7Y3J" has merely 1 110 residues from position 687 to 697 of "P05067". Regarding AF2 structures, regions with low PLDDT values signify that certain segments of the structures lack reliability for practical use. In scenarios as described above, structures are either incomplete or not reliable, requiring models to be more robust in order to deal with such conditions.
85,57
94.5
Suirot
ESM 2
85
84.13
93.55
93.55 35
83.55
93.5
33.24
93.42
Acc%
$3.30
Acc%
83
82
92.5
81
91.96
80
91.5
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Mask
Mask rate
(a) Subcellular
(b) Binary
Figure 5: Results for various mask rates applied to structure tokens. (a) Results in the DeepLoc Subcellular branch. (b) Results in the DeepLoc Binary branch.
In order to assess the robustness of SaProt, we introduced masking procedure wherein a specific percentage of structure tokens are replaced with "#" and then we fine-tuned the model to observe resultant performance variations. Figure deppicts the results of fine-tuning SaProt on the Deepl.cc (Almagro Armenteros et al.) (017) dataset. We employed different mask rates ranging from 0 to I. The results show that, as the mask rate increases, there is a corresponding reduction in accuracy for SaProt. Nonetheless, the accuracy remains competitive to ESM-2. This is because even if we mask out all structural token, residual information still exists, so performance can still recovered at least to the ESM-2 level after fine-tuning new labels. This is different from the zero-shot prediction tasks, as discussed in Figure B (where performance drops significantly by substituting random structural markers). In zero-shot prediction tasks, the model does not have the opportunity
19

---

to be fine-tuned, and performance is expected to drop significantly if too many structural tokens masked, as this will lead to inconsistencies with the training data.
E.3 RESIDUF SEQUENCE-ONLY FINR-TUNING
To explore the broader applications SaProt, we evaluate its effectiveness by fine-tuning on residue sequences where all Foldseek structure tokens are substituted with "#", resulting in "s This scenario often arises various protein engineering tasks where the fitness values of protein variants can be obtained through wet experiments, but experimental structures for them are unavailable. Please note that the structures generated by AF2 may not exhibit significant distinctions variants of the same wild-type protein.
We compare SaProt with ESM-1b and ESM-2 four supervised mutational effect prediction datasets (Fluorescence, Stability, 3-lactamase, and AAV). As depicted in Figure 6 SaProt performs on par with ESM-1b and ESM-2 even in the absence of structure information during the fine-tuning phase. This highlights the effectiveness of SaProt, even in situations where protein structures are not available.
But general, this specific scenario, SaProt may not exhibit a clear advantage over the ESM models, but it still remains comparable in performance.
1.0
ESM-1b
0855
0.904
ESM-2
0.8
SaProt
0.755
0.735
0,701
0.600
ISpearman's pl
0.6
0.2
0.0
Fluorescence
Stability
B-lactamase
AAV
Figure 6: Results for residue sequence-only fine-tuning several supervised fitness prediction datasets.
E.4 EVALUATION WITH ESMFOLD
AlphaFold2 (Jumper al. 2021 has achieved remarkable success in predicting protein structures, with the quality of the predicted structure heavily dependent on the corresponding MSA data. How- ever, obtaining high-qualit, MSA data on large scale can be computationally intensive and time- consuming. As an alternative approach, single-sequence- begel--sequeneebbseee based structure prediction models like ESMFold (Lin et al., 2022) can be utilized. In this study, we explore the impact of ESMFold on the performance of SaProt in both zero-shot prediction tasks and supervised fine-tuning tasks, as shown in Table Our observations indicate that SaProt, utilizing structure tokens generated by ESMFold, in general achieves better comparable accuracy ESM-2, but falls short of its performance with tokens generated by AF2. Hence, when feasible, it is highly recommended to employ the vocabulary generated by AF2.
20

---

Dreploor
CliNVar
PruteinGym
Thermostability
HumanPPI
Metal Binding
BC
Model
CC
Subceiluiar
Busory
Spouaman's
Spersmin's
ACC%
ACC%
Peax
ACCS
DSM-2
0,862
0,475
0.680
76.67
71.56
0,868
0.670
0.473
0.470
K2.09
91.96
ESM
0,900
0.440
0.708
$2.22
73.57
0,864
0.656
0.466
80.33
92.83
SaProt (FxmFold)
0.806
0.455
0.717
85.78
74.10
0.871
0,678
0.480
0.74
K2.82
93.19
SaPtod (AF2)
0,909
0.478
0.724
86.41
75.75
0,882
01682
01486
0.479
85.57
93.55
Table 9: Results of SaProt using tokens generated from ESMFold and AF2.
Strategy
Strategy
Kp
Mask and Predict:
Mask and Predict:
Residue
Residue
SaProt
Structure token
20 Mask tokens
Kc
Structure-aware taken
Kc
Structure token
Residue
Figure 7: Comparison of two masking strategies.
F MASKING STRATEGY COMPARISON
For protein structure-aware sequence P (81 82,2 8nIn), there are two possible masking strategies that can be used, as shown in Figure
The most straightforward masking strategy is randomly mask several SA tokens sifs using the symbol ###", and subsequently predict them directly from the SA vocab- ulary. However, potential weakness with this approach is that if the SA tokens not accurate enough, predicting exact SA tokens may lead the model in the wrong optimization direction. This is evidenced TO.
Masking Strategy 1:
Masking Strategy 2: Another potential masking strategy involves either predicting the residue token predicting the Foldseek structure token fi. However, predicting fi encounters the same issue mentioned above. Due to the high accuracy of residue types protein primary sequences, predicting only the residue token seems more effective training approach. Furthermore, predicting residue types aligns well with residuc-level protein tasks, c.g., mutational effect prediction.
In Table TO we report the results of two masking strategies on three datasets, namely ClinVar, ProteinGym, and DeepLoc. Due highly similar results on other tasks, we omit them directly. Strategy 2 performs better as we expected, suggesting that during the training process, there might be higher emphasis on the loss weight for predicting residues. The Foldseek structure tokens are primarily used as contextual information to aid residue prediction, rather than being utilized as labels.
G MORE VISUALIZATIONS
We exhibit more visualizations of learnt representations for ESM-2 and SaProt. In particular, adopt subcellular localization and binary localization datasets, visualizing the embeddings by t- SNE (van der Maaten & Hinton 2008). As shown in Figure (a) and (b), the extracted representa-
21

---

ProteinGym
DeepLoc
ClinVar
Mask&predict Strategy
w/o MSA retrieval
w/ MSA retrieval
Subcellular
Binary
AUC
Spearman's s p
Spearman's ' p
Acc%
Acc%
Masking Strategy
0,907
0.474
0.486
83.65
92.84
Masking Strategy 2
0.909
0.478
0.489
85.57
93.55
Table 10: Results for the two masking strategics during pre-traing phase.
tions from SaProt exhibit similarities better clustering compared those of ESM-2. Additionally, we visualized the embeddings of all 400 structure-aware tokens, as shown in Figure & (c). We can observe a certain degree of clustering phenomenon. In the semantic space, the SA tokens that are in close proximity to each other often correspond to similar types of residues or Foldseck tokens.
22

---

ESM-2
SaProt
Membrane-
Soluble
Membrane-
Soluble
(a) Embedding visualizations of ESM-2 and SaProt for binary localization dataset.
ESM-2
SaProt
Nucleus
Endoplasmic reticulum
Nucleus
Endoplesmic reticulum
Cytoplasm
Golgi apparatus
Cytoplasm
apparatus
Extrocollutar
Chioroplast
Extracelluiar
Chioroplast
Mitochondrion
Lysosome
Mitochondrion
Cellmemmrrane
Peraxisome
Cellmembrane
Peroxisomme
(b) Embedding visualizations of ESM-2 and SaProt for subcellular localization dataset.
Label by residue type
Label by structure type
(c) Visualizations SaProt token embeddings. Note that tokens are initialized with individual embeddings before training SaProt. For instance, the tokens "DI" and "Ds" are considered two different tokens by the model. as it does not knwow that both tokens represent the same residue unless the model has been trained to recognize this relationship.
Figure 8: Embedding visualization
23


Reviews:
[{"review_id": "FSXRKOxMBY", "reviewer": "Reviewer_1519", "summary": "This paper presents a new protein language model called SaProt. SaProt incorporates structural information during pretraining, leveraging the large quantity of predicted structures made available from AF2. As a simple way to incorporate structural information, SaProt leverages Foldseek, a recent approach to translate 3D structures into token sequences. Comparing with approaches from prior work, SaProt performs stronger on a range of protein function prediction tasks. Analysis supports the conclusion that it is the structural information that leads the model to outperform approaches such as ESM from prior work that include only sequence information.", "questions": "* It seemed unintuitive to use the cross-product of residues and Foldseek tokens to form the vocabulary. This seems to discard information about which vocab elements share the same residue or Foldseek token. Did you try other alternatives, e.g. by concatenating embeddings for the residue and Foldseek token to form the input representations?\n* Is the set of proteins used for pre-training SaProt the same as those used for ESM-2?\n\nNits: \n* Introduction paragraph 2 - redidue -> residue\n* Introduction paragraph 2 - \"real protein structures\" -> \"experimentally determined protein structures\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper demonstrates that pre-training with structures predicted from AF2 at a large scale can be useful for a wide variety of protein tasks, and that this information is relatively straightforward to incorporate in standard Transformer architectures using Foldseek.\n* The analysis in section 5 helps support the conclusion that it is indeed the structural information, and not other differences with prior work, that leads to the improvement in task performance.\n* The authors release their code and pre-trained models.", "weaknesses": "* Foldseek has several hyperparameters. It would be useful to understand their impact. Since one of the core contributions is the proposed vocabulary, it would have been helpful to see more analysis on the various choices involved (see questions below).\n* The presentation of the paper could be improved in several places. For example, section 3.1 seemed to distract from the presentation of SaProt and would be perhaps better presented as additional analysis later in the paper. Using MLM loss across the three modeling approaches did not seem like a clear comparison, given the differences in model architectures and inputs. Additionally, the connection between these findings and the design choices of SaProt was not clear. Additionally, in section 3.3.2, it may be clearer to present just strategy 2 and then the ablation comparison with strategy 1 later. These recommendations would simplify section 3 and the exposition of SaProt.\n* With the availability of AlphaFoldDB and Foldseek, leveraging these resources to incorporate structural information into Transformer-based PLMs is a somewhat straightforward step, and prior work has done this, e.g. ProstT5. ProstT5 is a relatively recent preprint, but I'm glad the authors acknowledged this work. However, while experiments show that SaProt outperforms ProstT5, it didn't seem clear to dismiss this work as not a \"general-purpose PLM\". Regardless, it seems that the main contribution of this work is not in the novelty of approach, but in the empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2waYtKlMlo", "reviewer": "Reviewer_WitX", "summary": "This paper introduces a structure-aware vocabulary approach to incorporate protein structure information into protein sequence representation learning models. It discretizes protein structure information using Foldseek to obtain sequence representations of protein structures. It then combines protein structure sequences with amino acid sequence information to create Structure-aware Sequences. To ensure that the model learns the semantics of amino acid tokens and structure information tokens effectively, the paper designs various masking strategies. The paper validates the model's performance on multiple downstream tasks.", "questions": "+ The model structure proposed in this paper is consistent with ESM-2. Did the paper attempt to extend the structure-aware vocabulary on top of a pre-trained ESM-2 model to train a new model? Would this potentially lower the training cost?\n+ Foldseek focus more on local protein structure information when constructing structure information sequences. Could this lead to the model not capturing a comprehensive set of structural information?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper proposes structure-aware vocabulary that cleverly integrates protein structure information with sequence information, enabling protein sequence models to capture structural semantic information. This is a positive development for protein representation learning models.\n+ Figure 2 compares various methods for modeling structural semantic information, providing insights into designing better protein structure models.\n+ Results in Table 1 and Table 2 demonstrate a significant improvement in the performance of multiple downstream protein tasks with the introduction of the structure-aware vocabulary.", "weaknesses": "+ Lack of ablation experiments: The experimental results of this paper do not effectively validate the model's performance in scenarios where structural information is not provided, such as comparing the performance of SaProt and ESM-2 on contact prediction tasks when structural information is not given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwQbKBUlTw", "reviewer": "Reviewer_LEd3", "summary": "Pretrained protein language models trained on protein residue sequences have become useful primitives in bioinformatics, as, much like their cousins in NLP, they can easily achieve good performance on diverse downstream tasks. In this paper, the authors train a protein language model on sequences of residue tokens augmented with VQ-VAE tokens encoding each protein's structure (or, lacking that, the AlphaFold2 prediction of its structure). Compared to traditional PLMs, the structure-aware language model performs better on a large suite of protein-related tasks.", "questions": "Do you think SaProt would do well at this: https://mastodon.cloud/@sokrypton/109560748589299528? It seems to me like adding the structure tokens could eliminate the need to manually account for P(structure | sequence) as Sergey has to here.\n\nDid you ever measure the correlation between AlphaFold2 pLDDT and the success rate of this model on downstream tasks? Also, given that you mask low-pLDDT regions during training, do you do the same during inference?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is clear and well-written, the evaluations extensive, and the results very good. The method is well-motivated and a very natural application of the VQ-VAE in Foldseek. Furthermore, some of the results are fairly surprising; it's unintuitive to me, for example, how much better this model is than the largest ESM-2 models. Beyond structural biology, I think this paper is also a nice addition to the literature on multimodal language modeling.", "weaknesses": "I don't have much negative to say about this. Insofar as the MSA Transformer is also a \"structure-aware\" language model, it would be interesting to see comparisons between SaProt and that model. I know they never released those parameters, but if it would be possible to use any of their self-reported performance figures, that would be nice.\n\nIt might also be interesting to see Figure 4 include an analogous experiment for the residue tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)