Exploring evolution-aware & -free protein language models as protein function predictors
Authors: Mingyang Hu
Fajie Yuan
Kevin K Yang
Fusong Ju
Jin Su
Hui Wang
Fei Yang
Qiuyang Ding
Venue: NeurIPS 2022 Accept (2022)

Abstract: Large-scale Protein Language Models (PLMs) have improved performance in protein prediction tasks, ranging from 3D structure prediction to various function predictions. In particular, AlphaFold, a ground-breaking AI system, could potentially reshape structural biology. However, the utility of the PLM module in AlphaFold, Evoformer, has not been explored beyond structure prediction. In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence), MSA-Transformer (multiple sequence alignment), and Evoformer (structural), with a special focus on Evoformer. Specifically, we aim to answer the following key questions: (1) Does the Evoformer trained as part of AlphaFold produce representations amenable to predicting protein function?  (2) If yes, can Evoformer replace ESM-1b and MSA-Transformer? (3) How much do these PLMs rely on evolution-related protein data? In this regard, are they complementary to each other? We compare these models by empirical study along with new insights and conclusions. All code and datasets for reproducibility are available at https://github.com/elttaes/Revisiting-PLMs .

Exploring evolution-aware & -reee protein language models as protein function predictors
Mingyang Hu' Westlake University
Fajie Yuan* Westlake University
humingyang@wwsstlake aew..eu...eeeeee............e... on
yuanfajiedwestlakee edu. on
Kevin K. Yang
Fusong Ju
Jin Su
Microsoft Research New England yang. www.neeiccooooe...........oeeeee com
Microsoft Research Asia
Westlake University
jrsongju@microsoo com
suj mujin@sestlake edu cn
Hui Wang
Fei Yang
Qiuyang Ding Westlake University
Westlake University
Zhejiang Lab
wanghui @westlake edu. cn
dinggiuyangdvestlake edu. cn
Abstract
Large-scale Protein Language Models (PLMs) have improved performance protein prediction tasks, ranging from 3D structure prediction to various function predictions. In particular, AlphaFold [23), a
Al system, could potentially reshape structural biology. However, the utility of the PLM module in AlphaFold, Evoformer, has not been explored beyond structure prediction. In this paper, we investigate the representation ability of three popular PLMs: ESM-1b (single sequence) 35| MASA-Transfommn Evoformer (structural), with special focus Evoformer. Specifically, we aim to answer the following key questions: (i) Does the Evoformer trained as part of AlphaFold produce representations amenable predicting protein function? (ii) If yes, can Evoformer replace ESM-1b and TeS--Tranfformr? (iii) How much do these PLMs rely vvolution-related protein data? In this regard, they complementary to each other? We compare these models by empirical study along with new insights and conclusions. All code and datasets for reproducibility are available at pestps://ee trub. com/ elttaes htt///eeiiiiiiiiieeeeeeeieeeeeii
(multiple sequence alignment) 30 and
Introduction
Proteins perform the majority of biological activities. It is, therefore, crucial to decipher the mecha- nisms underlying their structural and functional properties. The canonical requence-structuue funnttro relationship enables the success of sequence- eqquence-based machine learning methods that infer protein struc- ture and function from amino acid (AA) sequence. Large-scale protein language models (PLMs) with
self-supervised pretraining on tens millions to billions of proteins (PLMs) 28 35 13
are the
current state-of-the-art in predicting function and fitness from sequences.
Meanwhile, AlphaFold [23] trained on experimental 3D protein structures from the Protein Data
Bank (PDB) 339 can approach the resolution of experimental structures for most protein sequences.
"Equal Contribution
"Corresponding author. Fajic designed the idea and led the rescarch. Mingyang performed the research and led all experiments. Kevin provided important guidance for this research and worked a part of paper writing. Jin performed experiments for the MSA generation.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Its multiple sequence alignment representation module, Evoformer, combines new deep learning machinery, a PLM residue reconstruction task, and structural supervision in the form of a distogram. Like MSSa-Tranffrrmer [30], Evoformer takes family of evolutionarily-reati
and aligned protein
sequences as input, contrast to PL.Ms such as ESM-Ib [35] and TAPE [28], which only take individual protein sequences. Thus, for short, we refer to the former two models as evolution-awan PLMs and the latter two as evolution-free PLMs.
Despite the remarkable success of AlphaFold in predicting structure from sequence, it is unknown whether its Evoformer module can be applied other problems, in particular predicting protein function and fitness. Deciphering AlphaFold rather than treating it as a black box is beneficial to both Al and biology communities. Therefore, we attempt answer the following key questions.
Q(i:: Does the Evoformer trained during AlphaFold training learn general-purpose protein representations that can be used for various function prediction tasks? Unlike ESM-1b and MSA- Transformer, Evoformer trained with supervision from 3D structures. In addition, the second part of AlphaFold, the Structure Module, built on top of the 48 Evoformer blocks, much more complex and deeper than the traditional (linear) classification head used in ESM-lb and NSA-Transformer. These differences make the function representation ability of Evoformer an open question.
Q(i): If Evoformer's representation is general, does outperform ESM-1b and MSA- Transformer downstream tasks? While these three models are trained with different parameter sizes and datasets (see Table D]. we believe the comparison results are still valuable because they currently the most advanced PL.s. Training these large models from scratch is out of the reach of most academic research groups due to the computation and cost involved. In addition, we also investigate the utility of MSA-Tranne ASS-Transformer on function prediction tasks.
Qiii): How much does the performance of evolution-aware PLMs rely on the input MSAs? Can evolution-free PLMs assist cvolution-awan PLMs in terms of MSA construction? Both Evoformer and
take sets of aligned sequences (MSAs) as input. We investigate the effect of MSA quality and depth function prediction. On the other side, ESM-1b can simply formulate MSA construction as the remote homology detection task. Then, it is interesting to know whether ESM- l b-construcctd MSAs can be used inputs to Evoformer and MSA-Transfer Transformer.
We address the above questions through comprehensive empirical studies on a variety of structure and function prediction tasks. Note although AlphaFold (i.e. Evoformer + Structure Module) can accurately predict protein 3D structures, we investigate the ability of Evoformer to perform other structure prediction tasks, such secondary structure prediction and contact map prediction. We make the following key observations:
(i) The Alphafold-trained Evoformer produces representations that are useful for both structure and function prediction, as shown on two structure prediction tasks, two function annotation tasks (8
and two fitness score prediction tasks (28).
(ii) Evoformer representations are useful for both protein-level and residue-level prediction tasks.
(iii) Evoformer superior to ESM-1b and MSA-Transformer for structure prediction and novel miniprotein stability prediction, but inferior to ESM-1b on other function prediction tasks. It performs poorly for zero-shot fitness prediction tasks 27 compared to ESM-Ib and MSS-Transformerr.
(iv) Evolution-aware PLMs are superior to evolution-free ESM-1b model only in the structure prediction tasks, but general, are worse than ESM- in most function prediction tasks.
(v) Like structure prediction, evolution- aware PLMs are also sensitive to the amount of MSAs when predicting protein functions. In addition, their performance using -b-coos-onstt MSAs as input is comparable to the performance using MSAs generated by Jackhmmer 22 or HHblits (32)
Related Work
The volume protein data has exploded over the last decade with the advancement of new DNA sequencing technologies. Early work learned protein representations with LSTMs [88 T Recently, with the advent of large transformer models natural language
Protein language models
processing (NLP), large PLMs using the Transformer 43 architecture and BERT [2] denoising

---

Table 1: Model descriptions. 'aara., 'M' & "seqs" denotes parameters, million, & protein sequences.
Model
Embedding
Layers
Para.
Training database
ESM-1b
1280
33
650M
UniRef50 (27M seqs)
MSA- MSA-Transformer Transformer
768
12
100M
UniRef50 (26M MSAs)
Evoformer (No Template)
256 & 128
48
88M
PDB (190K structures + MSAs)
Feed Fonward
Food Fonyard
Transition
Transtion
LayerVorm
LayerNorm
Layarhoom
LayerNorm
Structure modulef
Trangls mutt calive
Colernaatio adi uttertion
Attertion
LayerNoom
LayerNorm
LayarNorm
Sell-Allention
Attertion
Fite-stion gron Ind-atteriee
Evoformer(48 blocks)
LayerNorm
LayerNoom
LayerNorm
NEA
MBA
seach
a. ESM-1b
b. MSA-Transformer
C.
Evoformer
d. Alphafold
Figure 1: Core modules of the three PLMs.
task have been widely adopted 35
28 13
PLMs trained on large sequence databases have been
successfully applied for various protein related tasks, including secondary structure prediction
[28
18
35 13
contact prediction 29 35 3D structure prediction [23] 2]. annotation prediction 7
signal peptide prediction 40 intracellular localization prediction
protein-protein interaction
prediction [1]. and fitness prediction (
10
19 20 27 ESM-1b
35 found that residue-residue
contacts can recovered from the learned representation, identifying the close relationship between Transformer attention and biological features. Following this,
44 and 29 further studied the
interpretability of the attention map as contact map. ProtTrans [13 benchmarked a variety of BERT- like PLMs, including TAPE, ESM-Ib, ProtTrans, and MSA- SSS--Trnsformer. Since ProtTrans with the BERT architecture does not exceed ESM- Ib, we choose to compare ESM-Ib. lb, MSA-Transfomee Transformer, and Evoformer for this study. Table 1 and Figure I provide more details about the three models.
Structure, function, and fitness prediction A protein's primary structure (i.e. AA sequence) determines its 3D structure, which turn determines its functional properties. This relationship underlies the success of PLMs, which infer protein structure and function from the raw sequence. Secondary structure (SS) prediction (38] is an easier task than predicting the 2D contact map [45) and 3D structure [3 MSAA-Tranffrrms and ESM-1b make accurate unsupervised contact predictions, which are an important input for the 3D structure generation.
While remarkable progress has been made in structure prediction, it is unknown whether this AlphaFolddttrigeeeed revolution transfers to other tasks, in particular function prediction. Protein function is broad term that refers any biological biochemical roles in organisms. In this paper, we focus on functional annotation prediction (classification task) 48 and fitness prediction (including both regression Compared with annotation prediction, fitness prediction more challenging since (1) there are routinely few no laboratory labels for supervision; (2) protein AA sequences are highly similar given the same wild-type protein.
[TO 28 and zero-shot prediction [19
27 tasks), shown in Figure 2
"Note that the bioRxiv version of
35 is concurrent with
18 and

---

m ma
Muszion Landsope
auiioote
(c) Zero-shot Mutation
(a) Metal lon Binding (b) Antibiotic Resistance
(c) Stability
(d) Fluorescence
Effects Prediction
Figure 2: Protein function prediction tasks. (a) and (b) are annotation prediction tasks. (c)() and (e)
are fitness prediction tasks.
Preliminaries
3.1 Tasks and datasets
We evaluate models on two structure prediction tasks:
1. Secondary structure (SS): This is residuc-level sequence-to-ssquunnc task where each residue I; of a protein sequence x = {æ1, 22,,.. Z is mapped to a label corresponding one of eight secondary structure tasks y E {G, I1, C} 28 SS prediction examines the degree to which PLM learns local structure.
2. Contacts: For a given protein structure, two residues are considered to be in contact if their Сд carbons are within 8A. We evaluate on pairs that are more than 6 positions apart in the primary structure 28). We measure the results using Precision @ L, which stands for the precision for the top-l. pairs with the highest predicted contact probability [47] is the length of the protein sequence.
For both contacts and secondary structure, we use the dataset in 35 which is constructed from SCOPe [15], and use the suggested split as the training and testing sets (see Table 2 One concern is that the dataset used here has been trained by AlphaFold as they come from the Protein Data Bank (PDB) 33. Hence, we investigate 48 additional proteins, which were collected from CAMEO (Continuous Automated Model EvaluatiOn) with 'hard' category from 2021-08-28 to 2022-04-30.
We also evaluate on two function (annotation) classification tasks:
Metal ion binding (MIB): This is a binary classification task, where PLM with new classification layer is used to determine whether there are metal ion aondinning sites in the protein. The dataset is also collected from PDB with annotation as metal ion binding. We randomly sample the same amount of proteins from the database as the negative class.
2. Antibiotic resistance (ABR): This is a multi-class classification task, where a PLM need to correctly determine which class of antibiotic protein degrades. We construct the dataset from CARD 226 which contains 19 different antibiotic types (see Appendix A.5 for details).
And three fitness prediction tasks. Unlike functional annotation prediction, protein sequences in this task all from the same wild-type with a small number of mutated residues.
Stability: This protein-level regression task that predicts the protease concentration which protein can maintain its fold [36]. We use the data splits from TAPE.
2. Fluorescence: This is also protein level regression task, predicting the log- flggfluureecece intensity of the protein sequence 37 We use the data splits from TAPE.
3. Zero-shot mutation effect prediction: This a protein-level. prediction task by comparing the difference between likelihoods assigned to the mutated residue and the likelihoods assigned the wild-type (see (27) for details). We evaluate five protein mutation datasets from Only single point mutation data is considered in this sub-task.
DeepSequence
'btpss/wwww.ameo33d.rrg/

---

Table 2: Dataset descriptions
Task
Source
Train
Test
Secondary Structure & Contact Prediction
SCOPe
11680
3617
Metal Ion Binding
PDB
6000
1332
Antibiotic Resistance
CARD
2072
1344
Fluorescence Stability
TAPE
21446
27217
ТАРE
53614
12851
3.2 Methods
For supervised function prediction tasks, we remove the original classification layer for ESM-1b and MASA-Transfommer, and the Structure module for AlphaFold. These remaining parts are called protein representations. We then add a linear layer top of these representation models to perform new classification or regression tasks. We adopt the standard fine-tuning strategy by fine-tuning all parameters using Adamw optimizer with le-5 as learning rate. For the zero-shot fitness prediction task, no training is needed. Instead, we obtain the softmax output at the corresponding mutation site in each protein sequence. The probability value is regarded as the fitness value for each amino acid following [9 27 As for structure predictions, follow the same practice in (28). Other details can be seen in Appendix A.
4 Results
4.1 Structure Prediction
In this section, we examine the structure representation ability of the three PLMs models discussed above. In general, the secondary structure and contact prediction tasks do have significant practical values since there is already highly accurate 3D structure data by AlphaFold. The purpose instead is provide a reference for function prediction, given that some tasks have similar formulation.
Table 3 and (Appendix A.4p show the results of SS prediction and contact prediction, respectively. For SS prediction, we consider two settings: pre-trained parameters and training from scratch (i.e., with random initialization). For Evoformer, pre- pre-training means the training of AlphaFold. For contact prediction, all PLMM have been already pre-trained since there is no meaningful contact map without pre-training.
First, we can easily observe that with pre-training, Evoformer performs the best in both tasks. Par- ticularly, Evoformer outperforms ESM- lb and MSA-Trans Transformer by a large margin for contact prediction with over 94% accuracy on Precision @ L (see Table 9). By contrast, it does not perform as accurately as the contact prediction task for SS prediction, which improves ESM-1b 1b by 1 11.6% and ASS--Tranformer by 5%. This is reasonable since SS prediction is different from 3D structure prediction from the machine learning perspective. Even though Evoformer has strong structure repre- sentation ability, it may not work well by just adding linear classification layer. This suggests that more complex structure module for SS prediction is necessary for higher accuracy. In comparison, since contact information that Evoformer shows superb results in this task. Thereby, we have also demonstrated results of 48 new proteins in Figure S where these proteins have not been trained by AlphaFold. In general, they are consistent with our above analysis.
be directly extracted from the pairwise distance map. is not surprising
Second, observe that pre-training is important: all PLMs are remarkably improved with around 40% improvement for ESM-11 lb and 28% for Evoformer. Interestingly, we note that Evoformer performs worse than MSA-Trie SEA--rrasformee by training from scratch. We conjecture that Evoformer is much harder to be trained without good initialization as its network architecture is more complex and deeper than SSS-Traasformer.
Finally, for structure prediction, evolution-aware models are largely better than evolution-free ESM-Ib. This is consistent with biological intuition and previous observation, such in [30..
5

---

Table 3: SS prediction. 'Seratch" means training from scratch without pre- pre-traimin traininn...
Model
Pre-train
Scratch
Improv.
ESM-Ib
0.703
0,500
40.6%
Transforsform
0.748
0.634
18.0%
Evoformer
0.785
0.614
27.8%
Evoformer
MsA-Transformer
ESM-1b
1
8.5
705H
Figure 3: Contact map prediction with the CAMEO dataset (from 2021-08-22 08-08888888 to 2022-04-30) in terms of Precision @ L. These proteins have not been used for training AlphaFold.
4.2 Supervised Function Prediction
AlphaFold predicts very accurate protein structures. However, it remains unknown whether its core representation module Evoformer can prediction function. Similarly, we investigate MSA- Transformer's ability to predict protein function, which is unknown either. Note that, unlike annotation prediction, evolution-aware Evoformer and MSS-TTannffrmmr only take single sequences (rather than MSAs) as input since all protein variants share the same MSAs for the fitness prediction task.
The results of annotation prediction (protein-level classification) and fitness predictions (protein-level regression) are listed in Table and respecctvvly We use classification accuracy (28) for evaluating annotation prediction and Spearman rank correlation [27] for evaluating fitness prediction. First, we find that for annotation prediction, FSM-1b, the evolution-free PI.,, exceeds the two evolution-aware PLMs, although opposite observations are made for predicting structures. Specifically, pre-trained ESM-1b achieves 5.8% and 17.4% improvements over pre-trained Evoformer and MSA-Tran Trannfforrr.. This is bit surprising since the biological intuition is that protein functional properties are mediated by structures. An important conclusion reached here is that better structure PLMs do not mean they have a better representation for predicting function.
Second, all pre-trained PLMs perform very well on the ABR task, multi-class protein classification task. Despite that, we can still observe an obvious improvement between training- sraniing-from-sooe crom-scratch and pre-training Similar observation brought from pre- pre-training can be obviously seen from all other function prediction tasks. This suggests that both supervised and unsupervised pre-training on PLMS are very useful to obtain protein function representations.
Third, we observe that the three pre-trained PLMs in general show better results than the one-hot 28 and ResNet 28 17] baselines in Table 5 This observation is new since neither Evoformer nor MSA-Transformer have been investigated for supervised fitness prediction. In more detail, ESM- lb performs the best on the fluorescence task, whereas Evoformer is the best on the stability task. This may be because protein stability has a closer relationship with protein structure, or because sequence-only pretraining does not generalize as well as structural pretraining to de novo miniproteins.
4.3 Zero-shot Mutation effects Prediction
Recent work revealed that PLMs are strong zero-shot learners in predicting potential viral muta- tions 19. 20). We are interested in evaluating Evoformer's masked-residue reconstruction ability in such zero-shot setting given that it also has BERT-like masked token prediction loss.

---

Table 4: Functional annotation prediction. 'scratch' means random initialization for parameters.
Pre-train
Scratch
Model
MIB
ABR
MIB
ABR
ESM-1b
0.840
0.979
0.628
0.945
MrneTrranffrrmer
0.715
0.961
0.640
0.932
Evoformer
0.794
0.979
0.645
0.920
Table 5: Fitness prediction. Scores are ISpearman pl on each task.
Pre-train
Scratch
Model
Fluorescence
Stability
Fluorescence
Stability
One-hot 28
0.14
0.19
ResNet
18
0.21
0.73
0.28
0.61
ESM-1b
0.68
0.76
0.68
0.59
TSS--ransformer
0.64
0.67
0.67
0.61
Evoformer
0.67
0.81
0.36
0.52
Figure shows the results of all PLMs. First, we observe that ESM-1b and ESM-1v 27 yield the best results, and both of them evolution-free models. ESM-Iv 1v shares architecture with ESM-1b, but is an ensemble of 5 models trained on UniRe190 instead of UniRef50. MSA-Trans(ormer in general performs well except on BLAT _LLA_ECOLXX Ostermeier2014. We notice that is highly affected by the quality and sequence identity thresholds of MSAs. For our experiments, we use the default setting by HHblits 33 searching from BFD database (see Appendix A.3)
The most surprising results are for Evoformer, which performs poorly in this task on all five datasets. To examine this cause, present results of another three models, namely ESM1-85M (released by (af2_ (dt_) (see model descriptions in Figure Dh
(35). ESMIb-88M (af2_data) and As shown, ESM1-85M performs much better than ESMIb-88M. which indicates that the accuracy of the zero-shot fitness prediction task is largely affected by the size of the pre-training dataset. This may explain why Evoformer performs poorly in this task. Beyond this, there is another key difference between these models. Evoformer or AlphaFold were trained with five different loss functions, among which four are strueture-related loss and only one auxiliary loss the masked token reconstruction (MTR) loss. By contrast, MSA Transformer and ESM- lb were trained with only the MTR loss. This may also be reason for explaining the worse results of Evoformer.
Answer for Q(i): Yes. The answer is provided in Table and S From these results, we can see that Evoformer with pre-trained parameters largely outperforms the non-pretrained version in four different function prediction tasks. Together with its structure representation ability, we conclude that parameters learned by AlphaFold are general-purpose and useful to various structure and function prediction tasks.
5 and Figure These results show that Evoformer indeed outperforms USSA-Trasformer and ESM lb when predicting structures, but it does not obviously outperform ESM-1b lb when predicting function. In particular, does not work at all in the zero-shot mutation effect prediction task. Accordingly, we can conclude that the AlphaFold-triggeree revolution for structure prediction cannot be directly transferred to function predictions. ESM-1b is still the SOTA when predicting protein functions.
Answer for Qii): No.
The answer is provided in Table 9
4.4 Effect of MSA
Here, we want to understand the influence of MSAs on evolution-aware PLMs. Further, we explore the relationship between evolution- aware and -free models. As mentioned in Section 4.2 evolution- aware PLMs take only single sequence as input for the supervised fitness prediction tasks, do not consider the affect of MSAs for these two tasks.

---

Evolormer
ESM-1b
ESM-1V
MSA-Transtormer
ESM1-85M
b 88Mar22 data)
MSA Transformeria12,
ECOLX, O6termlar2014
EEELX Ranganattan2015
BG Abwhh200
BLAT, ECOLX, PalaKA2012
DLG4 FAT Fangenatia-22012
Figure 4: Zero-shot fitness prediction. ESM-Iv's results is the average performance of five models;
ESMI-85M trained with the same dataset as ESM-1b but with a much smaller model size (85 data) and MSA-Trans, Transformer(af2, data) were trained with the same protein dataset used for training AlphaFold (including both the PDB data and those from Uniclust30 used in the self-distillante process) and have similar model sizes as Evoformer.
million parameters);
Table 6: Impact of MSAs. 'Seq" denotes an individual sequence, i.e., no MSAs.
SS
MIB
ABR
Model
Pretrained
MSA
Seq
MSA
Seq
MSA
Seq
Evoformer
Yes
0.785
0.716
0.794
0.724
0.979
0.983
Transffrreer
Yes
0.748
0.631
0.715
0.707
0.961
0.908
Evoformer
No
0.614
0.624
0.645
0.632
0.920
0.875
MSA Transformer
No
0.634
0.526
0.640
0.579
0.932
0.909
We report results Table 6 and Figure 5 The MSA setting in Table 6 consistent for both model training and inference. For Figure we only consider the inference stage because it is a zero-shot task. Clearly, observe that without MSAs, Evoformer, and TSA-Transformer both yield much worse results (with or without pre-training). Similarly, as shown in Figure also affected by the number of MSAs. In general, it performs much worse with few MSAs. These results show that (Answer for Qiii)) MSA data is a crucial input for evolution-aware PLMs, for not only the structure prediction but also the function prediction tasks.
is
BLAT. BLAT_ECOL ECOLX, Ostermeer2014 ECOLX
BLAT ECTL__ Ranganatthan2015
BG_STRSQ_AAT Abalt2015
DLG4_ Fanggantthar2211
0.8
Spearman pl
24
128
258
Figure 5: The effect of MSA depth on the zero-shot fitness prediction task using ASA-Transformer. Error bars are the standard deviation six replicate experiments with randomly sampled MSAs.

---

4.5 Relationship between Evolution-aware & -free PLMs
Evolution-awe aware PLMs require MSA data as input for both training and inference. However, database search using HHblits 33 or Jackhmmer (22] can take to 10 minutes for 1000-residue protein. ESM-1b, we design fast homology retrieval framework using Siamese ESMMII, following (TT] Figure 6 is the schematic of the ESM-
On the other hand, given the powerful function representation
Ib-constructed MSA generating and serving process, called ESM-MSA (see Appendix A.2
Table MSA Impacts. "Seq" denotes an individual sequence i.c. no MSAs.
SS Prediction
MIB
Model
Jackhmmer
ESM-MSA
Seq
Jackhmmer
ESM-MSA
Seq
Evoformer 0.785
0.776
0.716
0.794 0.715
0.766
0.724
MSS--rranfformer
0.748
0.733
0.631
0.774
0.707
Table shows the results by using different MSAs for Evoformer and ASS--rrasformmr. MSAs generated by ESM-1b 1b are comparable to those found by Jackhmmer (or HHblits) while being much faster (see Appendix A.3.2), suggesting that (Answer for Q(ii)) the function representation learned by evolution-free ESM-1b can be used to construct MSAs for evolution-aware MSA- Transformer and Evoformer.
Conclusions and Limitations
In the paper, have presented empirical studies to characterize three successful PLMs. In partic- ular, we focus on the function representation capacity of Evoformer. We draw several important conclusions: (i) Evoformer encodes not only structure, but also various protein functional prop- erties; it provides an alternative choice when predicting protein functions, especially the stability prediction task; (ii) despite that AlphaFold dominates the structure prediction tasks, Evoformer is not yet able to substitute ESM-1b and ASS--Tassformer when predicting protein functions; (iii) the functional predictability of evolution-aware PLMs also influenced by protein MSAs. Interestingly, evolution-free ESM-1b can generate sufficiently-accural MSAs for evolution-aware PLMs with much higher efficiency. Our work points out both strengths and weakness of three SOTA PLMs, which are potentially useful to many downstream biological tasks, including but not limited to protein engineering [8) cancer early detection [6].. and drug discovery (2
An important limitation that we probe the representation capacity of these PLMs by only adding a linear head top common practice in the deep learning community. While this verifies findings, we believe such a linear layer is not sufficiently expressive to achieve SOTA results. That is the reason why Evoformer with eight-layer Structure module achieves much higher 3D structure accuracy but with a linear layer it achieves less than 80% classification accuracy for the secondary structure prediction task. In addition, we do not fully disentangle the effects of the Evoformer architecture, training data, and loss function when comparing it to important direction is develop more advanced functional modules for higher prediction accuracy.
Therefore, an
Acknowledgement
This work is supported by the National Key Research and Development Program of China (No. 2022ZD01 15 1100), the National Natural Science Foundation China (No. U21A20427), the Westlake Center of Synthetic Biology and Integrated Bioengineering (WE-SynBio), the Research Center for Industries of the Future (No. WU2022C030), and Key Research Project of Zhejiang Lab (No.
2022PG0ACO2).

---

References
[1] Ethan Alley et al. "Unified rational protein engineering with sequence-based deep represen- tation learning". In: Nature methods 16. (2019), pp. 1315-1322.
[2] Minkyung Baek et al. "Accurate prediction of protein structures and interactions using three-track neural network". In: Science 373.6557 (2021), pp. 871-876.
[3] Protein Data Bank. "Protein data bank". In: Nature New Biol 233 (1971), p. 223.
[4] Tristan Bepler and Bonnie Berger. "Learning protein sequence embeddings using information from structure". In: (2019).
[5] Tristan Bepler and Bonnie Berger. "Learning the protein language: Evolution, structure, and function". In: Cell systems 12.6 (2021), pp. 654-669.
[6] Maxwell I. Bileschi et al. "Using deep learning to annotate the protein universe". In: Nature Biotechnology (2022), pp. 1-6.
[7] Maxwell L. Bileschi et al. "Using Deep Learning Annotate the Protein Universe". In: bioRxiv (2019), DO1: 10 1101/626507 eprint: https::/www, http::////wb.oorziiv org cog/content/ full. pdf
carly/2019/ 06 626507
URL: https://www..... biorxiv wwt.//orgg.ooo..o cong/nnntnt/
ear1y/2099/05/0/ /666//
[8] Surojit Biswas al. "Low-N protein engineering with data-efficient deep learning". In: Nature methods 18.4 (2021), pp. 389-396.
[9] Anselm Blumer et al. "Occam's razor". In: Information processing letters 24.6 (1987), pp. 377- 380.
[10]
Christian Dallago et al. "FLIP: Benchmark tasks in fitness landscape inference for proteins". In: Advances in Neural Information Processing Systems (2021).
[11] Sebastian Deorowicz, Agnieszka Debudaj Grabysz, and Adam Gudyš. "FAMSA: Fast and
accurate multiple sequence alignment of huge protein families". In: Scientific Reports 6. (2016), p. 33964. ISSN: 2045-2322. DO1: 10. 111.0////ee33333
URL: https://dot org/
10. 17.1038/srepp srep3394
[12] Jacob Devlin et al. "Bert: Pre-training of deep bidirectional transformers for language under- standing". In: arXiv preprint arXiv:1810.04900 (2018).
[13] Ahmed Elnaggar et al. "ProtTrans: towards cracking the language of Life's code through self-supervised deep learning and high performance computing". In: arXiv preprint
arXiv:2007.06223 06.20 (2020).
[14] Richard Evans al. "Protein complex prediction with AlphaFold-Multimer" In: BioRxiv (2021).
[15]
Naomi K. Fox, Steven E. Brenner, and John-Marc Chandonia. "SCOPe: Structural Classifica- tion of Proteins Proteins-extendeer integrating SCOP and ASTRAL data and classification of new structures" In: Nucleic Acids Research 42.D1 (Dec. 2013), pp. D304 D309. ISSN: 0008- DOI: 10. 1093 / nar g241240 eprint: https: // academic oup com nar article- gkt1240. pdf
pdf / 42/ / DDDDD/334444744
[16] Jonathan Frazer al. "Disease variant prediction with deep generative models of evolutionary data". In: Nature 599.7883 (2021), pp. 91-95.
[17] Kaiming He et al. "Deep residual learning for image recognition", " In: Proceedings of the IEEE conference computer vision and pattern recognition. 2016, pp. 770-778.
[18] Michael Heinzinger et al. "Modeling aspects of the language of life through transfer-learning protein sequences". In: BMC bioinformatics 20.1 (2019), pp. 1-17.
[19] Brian Hie al. "Learning the language of viral evolution and escape". In: Science 371.6526
(2021), pp. 284-288.
[20] Brian L. Hie, Kevin K. Yang, and Peter S. Kim. "Evolutionary velocity with protein language models predicts evolutionary dynamics of diverse proteins". In: Cell Systems (Feb. 2022).
ISSN: 24054712. DOI: 10. 11.10066j..cl. 2ew..02222... 01.0033
[21] Jeff Johnson, Matthijs Douze, and Hervé Jégou. "Billion-scale similarity search with GPUs".
In: IEEE Transactions on Big Data 7.3 (2019), pp. 535-547.
10

---

[22] L Steven Johnson, Sean R Eddy, and Elon Portugaly. "Hidden Markov model speed heuristic and iterative HMM search procedure". In: BMC bioinformatics 11.1 (2010), pp. 1-8.
[23] John Jumper al. "Highly accurate protein structure prediction with AlphaFold". In: Nature 596.7873 (2021), pp. 583-589.
[24] Yogesh Kalakoti, Shashank Yadav, and Durai Sundar. "TransDTI: Transformer Based Lan- guage Models for Estimating DTIs and Building Drug Recommendation Workflow". In: ACS Omega (2022).
[25] Weizhong Li and Adam Godzik. "Cd-hit: fast program for clustering and comparing large sets of protein or nucleotide sequences", In: Bioinformatics 22.13 (2006), pp. 1658-1659.
[26] Andrew G McArthur et al. "The comprehensive antibiotic resistance database". In: Antimicro-
bial agents and chemotherapy 57.7 (2013), pp. 3348-3357.
[27]
Joshua Meier et al. "Language models enable zero-shot prediction of the effects of mutations on protein function". In: Advances in Neural Information Processing Systems 34 (2021).
[28] Roshan Rao et al. "Evaluating protein transfer learning with TAPE", In: Advances in neural information processing systems 32 (2019).
[29] Roshan Rao et al. "Transformer protein language models are unsupervised structure learners".
In: International Conference on Learning Representations. 2020.
[30] Roshan M Rao et al. "MSA transformer". In: International Conference on Machine Learning. PMLR. 2021, pp. 8844-8856.
[31] Nils Reimers and Iryna Gurevych. "Sentence-bert: Sentence embeddings using siamese bert- networks". In: arXiv preprini arXiv: 10084 (2019).
[32] Michael Remmert et al. "HHbbits: lightning-fast iterative protein sequence searching by HMM-HMM alignment", In: Nature Methods 9.2 (2012). pp. 173-175. ISSN: 1548-7105, DOI:
10. 1038/mmeth..... 1818
[33] Michael Remmert et al. "HHblits: lightning- fast iterative protein sequence searching by HMM- HMM alignment". In: Nature methods 9.2 (2012), pp- 173-175.
[34] Adam J Riesselman, John B Ingraham, and Debora S Marks. "Deep generative models of genetic variation capture the effects of mutations". In: Nature Methods 15.10 (2018), pp. 816- 822.
[35] Alexander Rives et al. "Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences". In: Proceedings of the National Academy of Sciences 1 118.15 (2021).
[36] Gabriel J Rocklin et al. "Global analysis of protein folding using massively parallel design, synthesis, and testing". In: Science 357.6347 (2017), pp. 168-175.
[37] Karen S Sarkisyan et al. "Local fitness landscape of the green fluorescent protein". In: Nature 533.7603 (2016), pp. 397-401.
[38] Søren Kaae Sønderby and Ole Winther. "Protein secondary structure prediction with long short term memory networks". In: arXiv preprint 1412.7441..782 (2014).
[39] Joel I. Sussman et al. "Protein Data Bank (PDB): database of dime--iimesional structural information of biological macromolecules" In: Acta Crystallographica Section D) Biological Crystallography 54.6 (1998). pp. 1078-1084.
[40] Felix Teufel et al. "SignalP 6.0 achieves signal peptide prediction across all types using protein language models". In: bioRxiv (2021).
[41] Vineet Thumuluri et al. "Deepl 2.0: multi-label subcellular localization prediction using protein language models" In: Nucleic Acids Research (2022).
[42] Consortium UniProt. "The universal protein resource (UniProt)". In: Nucleic acids research Database issue (2008), pp. D190-D195. ISSN: 1362-4962 0305-1048.
[43] Ashish Vaswani et al. "Attention is all you need". In: Advances in neural information process- ing systems 30 (2017).
[44] Jesse Vig et al. "Bertology meets biology: Interpreting attention in protein language models".
In: arXiv preprint arXiv:2006. 15222 (2020).

---

[45] Sheng Wang et al. "Accurate de novo prediction of protein contact map by ultra-deep learning model". In: PLoS computational biology 13.1 (2017), e1005324.
[46] Fajie Yuan et al. "Lambdafm: learning optimal ranking with factorization machines using lambda surrogates". In: Proceedings the 25th ACM international conference informa- tion and knowledge management. 2016, pp. 227-236.
[47] He Zhang et al. "Co-evolution Transformer for Protein Contact Prediction". In: Advances in Neural Information Processing Systems (2021).
[48] Naihui Zhou et al. "The CAFA challenge reports improved protein function prediction and functional annotations for hundreds of genes through experimental screens". In: Genome biology 20.1 (2019), pp. 1-23.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethies review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We will release code and datasets for reproducibility.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sections A.4 and A.5
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In figure 5 we report the effect of different MSAs on the results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix A.6
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "xQESxC1PJL", "reviewer": "Reviewer_TPVF", "summary": "The paper investigates the PLM module in AlphaFold, Evoformer. In particular, it looks at assessing its representational ability beyond structure prediction, which is already well-established through AlphaFold. The paper does this by comparing the Evoformer module with two other popular PLMs: ESM-1b, MSA-Transformer in a number of tasks related to predicting protein functions. Particularly, the paper looks at: \n\n- (1) Can the Evoformer module in AlphaFold produce representations helpful in predicting protein function?\n- (2) Can Evoformer replace ESM-1b and MSA-Transformer for protein prediction functions?\n- (3) How much do evolution-based PLMs rely on MSA input? Can evolution-free PLMs assist with MSA construction?\n\nThe paper compares metrics of models on a number of structure prediction, function classification and fitness prediction tasks. The paper concludes that:\n\n- (1) Evoformer encodes protein functional properties, and can be used in predicting protein functions, in particular the stability prediction task;\n- (2) Evoformer is not yet  able to substitute ESM-1b and MSA-Transformer models for predicting protein functions;  \n- (3) MSA data is crucial input for evolution-based PLMs, for both structure and function prediction tasks. Evolution-free ESM-1b can generate accurate MSAs for evolution-based PLMS.\n", "questions": "- Elaborating on the concern of using the SCOPe dataset (Lines 116-117)\n- The low performance of Evoformer on the zero-shot fitness prediction task is interesting, particularly in comparison to the other models. The paper discusses this a bit in lines 215-220. It seems that the paper suggests that one potential reason for this is the difference in training between MSA-Transformer and ESM-1b and Evoformer. While not trained solely on masked token reconstruction, the masked token reconstruction objective is not missing from Evoformer. It would be helpful to speculate a bit more on why the difference would be so large here - is the paper suggesting that training on supervised protein structure tasks might degrade zero-shot performance on fitness prediction tasks?\n\nMinor Miscellaneous Suggestions\n- Line 157: ‘meaningful’, not ‘meanful’?\n- Line 160: ‘By contrast, it does not perform as accurately as the contact prediction task for SS prediction’ -> I found the wording confusing; consider re-wording\n- Line 189: ‘.. better structure PLMs do not mean they have a better representation for predicting function‘ -> also confusing, consider re-wording\n- Table 5: Consider re-wording ‘Scratch’ to ‘Random Initialization’\n- Consider Higher resolution for Figure 4\n\n", "limitations": "- Authors have adequately addressed the limitations of the work under 'Conclusion and Limitations'", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- This is a really well-written paper presenting interesting results that can be valuable for the field of PLMs and structural biology.\nI believe the research questions being asked are valuable: assessing the representational power of Evoformer beyond structure prediction is interesting\n- Follow-up work on the research questions started here could lead to insights around dynamics between structural and functional protein functions\n- The paper is sound and has great presentation - is very clearly written and does a good job at outlining the research questions, experiments ran and clearly shows how the results support the conclusions that are drawn\n- The results themselves are valuable: in particular; I find it interesting that Evoformer performed so poorly on the zero-shot fitness classification task, compared to ESM-1b and MSA-Transformer models and that Evoformer does not consistently outperform the ESM-1b and MSA-Transformer models of predicting protein function tasks, as one might expect, given its huge success on structure prediction tasks\n- The conclusion drawn based on zero-shot fitness prediction tasks that ‘the AlphaFold-triggered revolution for structure prediction cannot be directly transferred to function predictions’ might be too strong based on the experiments ran. As mentioned under ‘Limitations’, it’s possible that more complex model architectures could lead to better results. I suggest addressing the low performance\n- I appreciated the paper including their limitations, such as how more complex model architectures could lead to better results", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "w5MG9vHEcB5", "reviewer": "Reviewer_RhwA", "summary": "The paper studies the effectiveness of evolution-based neural architectures including  Evoformer, MSA-Transformer, and evolution-free methods like ESM in protein structure and property prediction problems. Evoformer is similar to MSA-Transformer which learns protein representation from MSA, but Evoformer was used as part of the Alphafold for 3D representation prediction. While ESM is pretrained using only sequence information without the need for MSAs. \nThe authors examined these models in different standard structure predictions and protein properties prediction tasks. Their conclusion was that evolution-based neural architectures including  Evoformer, and MSA-Transformer outperform evolution-free methods like ESM in structure prediction tasks while Evoformer pretrained with much smaller data is bad at properties prediction tasks and zero-shot mutation prediction tasks.\nThe authors also study the effect of varying MSA quality on the results of the evolution-based neural architectures and proposed an approach that use ESM  for constructing ESM-MSA that is much faster than the standard MSA search but still achieves comparable results to the methods that use MSAs.", "questions": "For completeness, I think the experiments should include all benchmark data in TAPE, for example, homology prediction is missing.  \nWhy don't you use the TAPE train/dev/split for contact map and ss prediction instead of SCOPe?\nTable 2: missing information about dev set.\n", "limitations": "YES", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThis is a nice work that performs very careful empirical studies, well-written and accompanied with source code and two new standard benchmark datasets, this is a plus as the standard 5 benchmark datasets in TAPE are not enough for evaluating recent development in the field.\n\nAll methods were compared using the same linear layers for downstream tasks, even the authors see this is as a weakness as stated in the conclusion because for different downstream tasks better than linear layers should be considered to achieve the SOTA results but for comparisons of pretrained models, it is fair to use the same linear layers for all the tasks.\n\nSome of the findings are interesting, for example, the results of Evolformer on the zero-shot mutations.\n\nWeaknesses  \n\nIt is important to note that Evoformer, MSA-Transformer, and ESM-1b are very different in network size, and different training data used. It is very hard to compare these pre-trained models and draw some conclusions on whether one method is better than another. I see that the authors have mentioned this issue in the introduction section and I agree that it is useful to check which pretrained models are better because most companies and researchers from academia do not have the computing resource to pretrain these large models. Yet the answer to that question only has a practical meaning which might help industries choose the right model for their application but not give an answer to the question of which methods are better (assuming they are the same size and pretrained on the same data).\n\nThis is an empirical study so, besides interesting benchmark results, its technical and novelty contribution are limited. \nI evaluated the paper based on the technical depth, novelty contribution, and usefulness of the research to the community. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tbf7EwefhYI", "reviewer": "Reviewer_ZqSX", "summary": "The authors divide the sequence alignment representation module into two types: revolution-based and revolution-free, and examine the usefulness of the evlutionary information not only in the framework of protein structure prediction but also other prediction problems, related with proteins. The \"revolution-based\" approach is confined to \"Evoformer\" in this paper.\n", "questions": "-- Seeing the three models (a, b and c) in Fig. 1, can the models be compared in more detail? For example, can the authors consider more valiation of using the information?\n-- Also how much can the performance be affected by changing the values of hyperparameters?\n", "limitations": "-- This work is significant from a ablation study viewpoint in bioinformatics. We might see what information can contribute to improve protein-related bionformatic problems, such as secondary structure or function prediction. However, methodological contribution would be definintely limited. One better presentation might be to do experiments more: for the same problem and same module, different ways of using the module are examined and the best way of using that is shown. However, I am unsure if this way could be good enough to be accepted for publicaiton by this conference, though.", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-- The paper addresses the problem of examining if evoluationry information (or Evoformer) is more useful / important than multiple sequence alignment without evolutionary information. This problem would be important and also attractive from a bioinformatics (or biology) viewpoint.\n-- Rather clear conclustions were lead by empirical studies. Particularly, it is indeed interesting that Evoformer does not necessarily work well for function prediction, comparing with \n\nWeaknesses\n-- Although the information is important, the problem might be minor.\n-- The work is totally empirical. That is, the conclusions are obtained totally empirically. So although conclusions might be useful, the result of this work might not be universal. I think this is one of the points already mentioned by the authors in Conclusion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JsEBjRv2xTa", "reviewer": "Reviewer_WXJh", "summary": "This paper studies the representations of the AlphaFold Evoformer across protein structure and function prediction tasks and compares them to protein language models. The paper finds that while Evoformer produces state of the art predictions of structure, it falls behind protein language models on function prediction.", "questions": "The authors may consider more strongly highlighting the use of learned linear projection heads as this is critical for interpretation of the results; the authors may consider summarizing their findings in the abstract.", "limitations": "The authors have included a thoughtful discussion of the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "EDIT: based on revisions by the authors during the discussion period and the addition of a held out test set for contact prediction I am raising my evaluation. However the concern about training set holdout also applies to the the other structure prediction result in the paper (secondary structure) and I suggest the authors address this using a similar held out set of structures whether it is accepted at Neurips or whether they resubmit to another venue.\n\nThis paper has several important contributions:\n* Finding that protein language models perform better than AlphaFold on function prediction tasks is unexpected and interesting\n* Studies the dependence of MSA based models on the additional sequences\n* Investigates the use of MSAs produced by language model embeddings and finds that retrieval using embeddings is comparable to HMMs\n\nOverall the findings and the investigations are of interest to the Neurips research community and I would very much like to recommend this paper for acceptance. However there is a major methodological concern that undermines some of the paper's main conclusions. In short the assessment of contact prediction is flawed as the test sets used are not held out for AlphaFold training. A number of conclusions in the paper depend on this comparison. Held out test sets, e.g. CASP14 and CAMEO proteins after the CASP14 cutoff date are available.\n\nThere are also some problems with the related work section. E.g. in the sentence \"PLMs trained on these databases have been successfully applied for various protein related tasks\" Rives et al 2021 didn't look at annotation prediction (the authors should cite Bileschi et al 2022 for this), but did look at secondary structure prediction, contact prediction, remote homology detection, and mutational effects. Heinzinger et al 2019. also looked at secondary structure prediction, as did Elnaggar et al 2020. Alley et al 2019, Meier et al 2021, and Hie et al 2021 also looked at fitness prediction. When discussing early work Rives et al 2021 and Heinzinger et al 2019 which were concurrent with Alley et al 2019 should be mentioned. The citation for ESM1b and recovery of contacts cites Vig et al. rather than Rives et al 2021 which introduced ESM1b and studied recovery of contacts. Rao et al 2021 and Vig et al 2021 followed this work up with discovery of the interpretability of attention maps as contacts and should also be cited for this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept