Language models enable zero-shot prediction of the effects of mutations on protein function
Authors: Joshua Meier
Roshan Rao
Robert Verkuil
Jason Liu
Tom Sercu
Alexander Rives
Venue: NeurIPS 2021 Poster (2021)
Keywords: Proteins
language modeling
generative biology
zero-shot learning
unsupervised learning
variant prediction

Abstract: Modeling the effect of sequence variation on function is a fundamental problem for understanding and designing proteins. Since evolution encodes information about function into patterns in protein sequences, unsupervised models of variant effects can be learned from sequence data. The approach to date has been to fit a model to a family of related sequences. The conventional setting is limited, since a new model must be trained for each prediction task. We show that using only zero-shot inference, without any supervision from experimental data or additional training, protein language models capture the functional effects of sequence variation, performing at state-of-the-art.

Language models enable zero-shot prediction of the effects of mutations on protein function
Joshua Meier Roshan Rao
Robert Verkuil Jason Liu
Tom Sercu
Alexander Rives
12
Abstract
Modeling the effect of sequence variation on function funddmennal problem for understanding and designing proteins. Since evolution encodes information about function into patterns protein sequences, unsupervised models of variant effects can be learned from sequence data. The approach to date has been to fit model to family of related sequences. The conventional setting is limited, since new model must be trained for each prediction task. We show that using only zero-shot inference, without any supervision from experimental data addi- tional training, protein language models capture the functional effects of sequence variation, performing at state-of-the-art.
Introduction
Proteins have myriad diverse functions that underlie the complexity life. Protein sequences encode function via structure through the spontaneous folding of the sequence into the three dimen- sional structure of the protein [1]. The effects of sequence mutations on function form a landscape that reveals how function constrains sequence. Alterations some sites in protein sequence cannot be tolerated because they are essential to the protein's ' function. Other sites evolve together because the structure and function is determined by them collectively. Mutations enhance the activity of protein, attenuate it, or leave it unchanged.
The functional effect of sequence variations can be measured through deep mutational scanning experiments [2]. Consisting of thousands to hundreds of thousands of measurements of protein function, deep mutational scans give insight into the intrinsic constraints on protein's 's structure and function. Due to the cost and difficulty implementing such experiments, compilations of deep mutational scanning data include experiments of thousands of proteins encoded in the human genome, and the millions more across the tree of life that we would like to understand.
few dozens of proteins at most, relative to the tens
model that learns the landscape linking sequence to function can provide insight into function without having to do experiments. Unsupervised models of mutational effects can be learned from sequences [3, 4]. Statistical patterns in a family of evolutionarily related protein sequences contain information about structure and function [5-7]. This is because the properties of a protein act as constraints on the selection sequences through evolution [8].
In the natural language modeling community, there has been interest in zero-shot transfer of models tasks. Massive language models can solve tasks they haven't 't been directly trained on [9-11]. Recently protein language models have achieved state-of-the oo--th--rttttttt- in various structure prediction tasks [12-14]. Work date has mainly focused on transfer in the classical representation learning setting, using pre-trained features with supervision on the downstream task.
Facebook Al Research NNew York University 'UC Berkeley. ESM-Iv Iv available at <https://github com/ htm//faccebookeesearchh esmm. Correspondence to: Alexander Rives <arivesøt com>. com-.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

130
140
150
KLEYNYNSHNT MMADKQKNN XVN
Fluorescence Activity
Functional Effect
Active
Deleterious
Figure 1: Depiction of mutational effect prediction task. The objective is score the effect of sequence mutations on the function of a protein. Deep mutational scanning experiments provide ground truth experimental measurements of the protein's function (fluorescence activity in the example here) for large set of single mutations or combinations of mutations. For each protein, the prediction task is score each possible mutation and rank its relative activity. Predictions for single substitutions can be described in score matrix. The columns are the positions in the sequence. The rows are the possible variations at each position.
In this work we show that language models trained on large and diverse protein sequence databases can predict experimental measurements of protein function without further supervision. Prior work has focused transferring the representations using supervision from experimental data [15, 16]. We find that language models can transfer to predict functional measurements without supervision. Language models perform zero-shot and few-shot prediction of mutational effects across variety of proteins with widely differing functions. We perform experiments with state-of-the protein language models ESM-1b lb [12] and MSA Transformer [13]. We introduce
protein language model, ESM-1v, with zero-shot performance comparable to state-of-the-a mutational effect predictors. Performance be further improved fine-tuning the model with sequences from the protein family. Predictions capture the functional landscape of the protein, correlate with amino acid conservation patterns in the core and surface, and identify residues responsible for binding and activity.
2 Zero-shot transfer
Zero-shot learning has classically described the extension of a classifier to a new set of classes that have not been seen training [17]. In natural language processing this idea has been extended to describe the transfer of models to entirely new tasks without further training. Proposed as zero-data learning by Larochelle et [18], this perspective on transfer has been at the center of recent work understanding the generalization capabilities of large language models [9-11, 19]. The distinction from representation learning that the models are used directly without additional supervision for the task. This means that the tasks must be learned purely from pre-training.
In this work take a similar perspective on zero-shot transfer that GPT-3, described in Brown et al. [10]. We define zero-shot transfer to be transfer of a model to new task without any further supervision to specialize the model the task. We also consider the closely related idea of few-shot transfer. Here as in Brown et al. [10] we define the few-shot setting to be one in which few positive examples are given to the model as inputs at inference time. As in the zero-shot setting. no gradient updates are performed to specialize the model. Similar to Brown et al. [10], the claim is not one of -i-tiiiribttionn generalization. The assumption that in the pre-training stage, the model learns information relevant to the tasks to which it will later be transferred. In the case of protein language models, the pre-training dataset includes sequences from across evolution, which implies the model may see examples of sequences from protein families on which it will be evaluated. The essential

---

One-time
Per Task
Pre-training
Task specifie MSA generation
Task specifie
Predictions
training
Zero shot ESM-Iv
Sequence
Few shot MSA Transformer
MSA
DB
Task specific models
Modrr
Unsupervised fine-tuning
Sequence DB
tnin
M MTask (x)
ESM-Iv fine-tuning
Classical unsupervised
JadeHMMer
EVMutation
(None)
tnii
Mrask (x)
DeepSequence
Figure 2: Steps involved in variant effect prediction methods. Compared with EVMutation [4] and DeepSequence [20], MSA Transformer and ESM-1v require no task-specific model training for inference. Moreover, ESM-1v Iv does not require MSA generation.
departure from the standard approach in computational biology is that the model is general purpose and can be applied across a variety of tasks without specialization.
Measurements of function, a property central importance to the understanding and design of proteins, practical ground for studying the generalization capability of protein language models. Deep mutational scanning experiments measure the effects of thousands to hundreds of thousands of mutations on single protein, and have been performed variety of proteins having different functions and using various forms of experimental measurement. We study zero-shot and few-shot transfer of protein language models function prediction using this data.
Supervised methods trained with data from experimental measurements [15, 16], and unsupervised methods trained only sequences [3, 4] have been developed for prediction of mutational effects. Unsupervised mutational effect predictors are trained as task specific models on sequences from individual protein family. In this view every protein is an independent prediction task where the objective is to score the effect of mutations on the protein's function. While mutational effect predictors trained multiple sequence alignments (MSAs) are typically described as unsupervised, they can also be seen as weakly supervised. Hsu et al. [15] observe that such models have weak supervision on the task through the MSA, which describes the fitness landscape of the protein through positive examples.
If protein language models can learn the information necessary to solve task from pre-training, then they can applied directly to new instances of the task, without specialization. This would mean that in practice single general purpose model can be trained once and then applied to variety of possible tasks. Thus zero-shot and few-shot transfer represent fundamentally new unsupervised learning capabilities that protein language models bring the computational biology toolkit.
3 Method
Protein language models trained with the masked language modeling objective are supervised to output the probability that an amino acid occurs at a position in protein given the surrounding context. We use this capability score sequence variations. For given mutation we can consider the amino acid in the wildtype protein as a reference state, comparing the probability assigned the mutated amino acid with the probability assigned to the wildtype.

---

Models
Full
Test
PSSM
0.460
0.460
EVMutation (published)
0.508
0.495
EVMutation (replicated)
0.511
0.498
DeepSequence (published)
0.514
0.499
DeepSequence (replicated)
0.520
0.506
MSA Transformer
0.542
0.524
ESM-Iv (zero shot)
0.509
0.482
ESM-Iv (+further training)
0.538
0.519
Table I: Comparison of protein language models to state-of-the-art methods. Average ISpearman pl on full and test sets. DeepSequence and ESM-1v models are each ensembles of models. MSA Transformer is single model, but is ensembled across random samples of the MSA.
We score mutations using the log odds ratio at the mutated position, assuming additive model when multiple mutations 7 exist in the same sequence:
Σ
logg(xx "i Td (xar) log = xt (x)))
RET
Here the sum is over the mutated positions, and the sequence input to the model is masked at every mutated position.
3.1 Zero-shot and few-shot transfer
In the zero-shot setting, inference is performed directly on the sequence to be evaluated. Since the MSA Transformer take multiple sequences input inference time, few-shot setting, where additional sequences from the protein family are provided along with the sequence be evaluated. In both the zero-shot and few-shot settings, only forward passes of the models are performed during inference; no gradient updates are taken. Fig. illustrates the approach in comparison to the current practice of fitting model for each task.
this model in the
3.2 Inference efficiency
Inference with ESM-Iv v more efficient than current state-of- of-the-art methods. This is result of two important differences: (i) the effect of mutations can be inferred directly without training a task-specific model; (ii) fitness landscapes can be predicted with a single forward pass. Time requirements are summarized in Fig. 7.
3.3 Scoring with MSA Transformer
We score mutations with MSA Transformer using the log odds ratio and additive model in Eq. (1). However, since MSA Transformer uses a set of sequences for inference, we input the sequence to be evaluated the first sequence, and provide additional sequences from the MSA as context. Masking and scoring are performed on the first sequence only.
Results
4.1 Experimental setup
Prediction Models We compare to state-of-the-arr unsupervised variant prediction methods, EV- Mutation [4] and DeepSequence [20]. We also examine performance of variety of protein language models that have been recently introduced in the literature.
The position specific scoring matrix (PSSM), EVmutation [4], and DeepSequence [20] methods are all MSA based. The PSSM treats each position the sequence independently, factorizing the likelihood into one term per sequence position. EVmutation is a Potts model, which adds pairwise terms modeling the interactions between positions. DeepSequence introduces latent code, allowing potential higher-order interactions between positions.

---

0.8
[Spearman
0.4
0.2
ESM-1v (zero shot)
ESM-1v f+ruther training)
SequSequence (replicated)
0.0
BLAT geout imspatttacen tessess raar YE45 faala20 dooiins
AMDE PREAE Whtd
axay ELaTT sat Sataabooni Sisnhogonn
nil USCy HUNMAN Bleemen Brous on AUMOL
TIM SULSO
TEAST Shendooaddl
Am13 MAB BEAASTAALL TAlLL DED Theri Pvericciy
BLAT EcoLy Out ievemneeertn
PABR fol, Hatk HEVP THE FNd
Horr VE Juan Commen
TPAT. HUMAS E TrrMii saa cooltor
HisT. YEAR Eoatt dee coneett
URSA4 Mo MOUEE TVDD LIPTT (TASA
VAPI JEMMAN perddciiie cccaantee
PEEN RASIr REMMA Farres
HUMM poppeccl
Atati TAAAA Rabkiny
H Hothzon
RItves EIFT Wh cosartols MPOI HUTMAH PLLE
AEL YEAST Baradol
EEC ECCES PLU
GRLAH.
Figure 3: Per task performance. Comparison across 41 deep mutational scanning datasets. Points are ISpearman pl on each dataset, error bars show standard deviation of 20 bootstrapped samples. Validation proteins are shown to the left of the dividing line and test proteins to the right. In 17 out of the 41 tasks, ESM-1v zero-shot has higher ISpearman than DeepSequence
UniRep [21]. TAPE [22], ProtBERT-BFD [14], ESM-1b [12], and ESM-Iv vv (introduced here), are all single-sequence language models trained on large databases of unaligned and unrelated protein sequences (e.g. Pfam [23] or UniRef [24]). With the exception of UniRep, which is trained using next token prediction, all models are trained with masked language modeling [25].
Finally, the MSA Transformer [13] is a combination of both approaches; it is trained on a large database of MSAs using masked language modeling and takes an MSA as input during inference.
ESM-ly We train ESM-Iv, a 650M parameter transformer language model for prediction of variant effects, 98 million diverse protein sequences across evolution. The model is trained only on sequences, without any supervision from experimental measurements of function. We use Uniref90 2020-03 [24]. employing the ESM-Ib architecture and masked language modeling approach of Rives et al. [12]. The model attains a perplexity of 7.29 on set of held-out Uniref90 sequences (Table 10). We train five models with different seeds to produce an ensemble.
Evaluation Models are evaluated on set of 41 deep mutational scans collected by Riesselman al. [20], which comprise variety of tasks assessing a diverse set of proteins. Across tasks, the experiments differ in the functions tested and in the measurements performed. We treat each deep mutational scanning dataset as separate prediction task, scoring each of the variants the dataset with the model. The tasks are split into validation set of ten mutational scanning datasets and test set consisting of the remaining datasets. We evaluate performance by comparing the scores with the experimental measurements using Spearman rank correlation.
Comparisons Since the published versions Mutatioo and DeepSequence use MSAs generated from an earlier version of Uniref100 100, we generate new MSAs using EVMutaaion methodology and the version of Uni concurrent with our pretraining dataset. We train replications EVMutation and DeepSequence using their open source code. The same MSAs also used in few-shot experiments with MSA Transformer and unsupervised fine-tuning experiments with ESM-1v.
4.2 Language models enable zero-shot and few-shot prediction the effects of mutations
ESM-1v and MSA Transformer models make state-of oatt-oo-thee-s art predictions. Table 1 compares overall performance of the models across the mutational scanning datasets. Fig. presents a comparison between ESM-1v and DeepSequence on each of the tasks. Zero-shot inference with ESM- 1v has better correlation with experimental measurements than DeepSequence on 17 of the 41 datasets. The two methods are not statistically distinguishable via paired r-test.
5

---

Models
Full
Test
UniRep
0.156
0.151
TAPE
0.171
0.175
Pro(BERT-BED
0.428
0.399
ESM-1b
0.459
0.424
ESM-1vf
0.484
0.457
ESM-Iv*
0.509
0.482
Table Zero-shot performance. Average ISpearman on full and sets. AAeeraee performance of five ESM-Iv models. "Ensemble of the five ESM-Iv models.
Table 2 compares protein language models in the zero-shot setting. ESM-Iv outperforms existing protein language models TAPE [22], UniRep [21], ProtBERT-BED [14], and ESM-1b [12]. Fig. breaks down performance across each of the tasks.
Pre-training data We examine the effect of the clustering level of pre-training data. Fig. compares models pre-trained datasets clustered at increasing sequence identity thresholds. ESM- lb is trained on sequences clustered at 50% identity threshold. Improvements are seen using a 70% threshold with greatest improvement at 90%. Uniref100 performance appears to deteriorate early in training despite being the largest of the datasets. These results establish a link between model performance and the data distribution, highlighting the importance of training data in the design of protein language models.
Scoring methods We compare four scoring methods the validation set masked marginals, wildtype marginals, mutant marginals, and psuedolikelihood. Table shows that the masked marginal approach described in Eq. (I) outperforms other scoring methods, including ones in which the likelihood changes at non-mutated positions are considered. The scoring methods are described in detail in Appendix A.
Parameter count Previous work with protein language models has established link between model scale and learning of protein structure [12, 26]. We examine zero-shot transfer performance as function of parameter count. We train models using the same width, depth, and learning rate as described in Henighan et al. [27], observing improvements with scale (Fig. 9). These findings suggest that continued scaling of the models will further improve results.
4.3 MSA Transformer
We examine how the sequences provided to MSA Transformer affect few-shot transfer. Table 8 compares sequence selection methods that vary the diversity of the sequences. Providing a more diverse set of sequences improves few-shot performance. Selecting a set of sequences maximize diversity outperforms selecting diversity minimizing set of sequences. Random sampling performs even better, and sampling sequences according to sequence weights [28] performs best.
We also vary the number of sequences used for inference. Fig. 11 shows few-shot performance as a function of the number of sequences given as input. The model performs well using only a few sequences, but performs best with 384 total sequences. the main tables we report results sampling 384 sequences using sequence reweighting and ensembling predictions over five different subsamples from the MSA.
4.4 Unsupervised fine-tuning MSAs
While ESM-1v performs well when evaluated in the zero-shot setting, we explore whether results can be improved by fine-tuning on the MSA. Fine-tuning on MSAs has been used in previous work [21, 16] as a stage in transfer learning to specialize pre-trained model protein family, before applying supervision with labeled data. Here we consider using the fine-tuned model to make unsupervised predictions directly, without adding supervision from experimental data.

---

0.5
ISpearman pl
0.4
0.3
Clustering
0.2 -
30%
70%
0.1
90%
50000 100000 Updates
150000
Figure Comparison of pre-training datasets. Average ISpearman pl on the single-mutation validation set. While 50% clustering threshold was used for ESM-1b, training with 90% clustering results in significant improvement on variant prediction tasks. Notably, models trained on Uniref100, the largest dataset in this figure, appear to deteriorate early in training. These results establish link between model performance and the data distribution, and highlight the importance of training data in the design of protein language models.
We observe that naively fine-tuning the model on the MSA results in rapid overfitting and poor performance the prediction tasks (Fig. 12). While experiment with variety of approaches to freezing parameters during fine-tuning, detailed in Appendix B, none produce significant improve- ments. We find that approach using me--trinnin sequences regularize the fine-tuning performs well and enables training of all parameters without overfitting (Fig. 13). Spiked fine-tuning improves average absolute Spearman rho the full dataset from 0.5 10 for zero-shot evaluation 0.537 with fine-tuning.
5 Analysis of models
Protein structure and function ESM-1v probabilities reflect the functional properties of sites within the protein. We use the entropy of the model's predictions for position measure of its estimation of conservation. The lowest entropy predictions cluster at binding sites. Fig. 14 compares the distribution of the model's entropy between binding sites and non-binding sites. A significant difference is observed between the entropy assignment to binding and non-binding site residues. Fig. visualizes the side chains of the 10 lowest entropy residues as predicted by the model on the crystal structure of DNA methyltransferase M.HaellI interacting with its DNA substrate. In the crystal structure cytosine of the substrate inserted into the active site of the enzyme. The low entropy residues cluster in the active site and interact with the cytosine. Additional examples are visualized Fig.
The model probabilities also correspond to structure. Fig. 15 compares the entropy assigned to sites that are buried in the core the protein vs. exposed on the surface. The model assigns significantly lower entropy to sites that are in the core of the protein, consistent with the idea that tight packing in the core places greater constraints assigned by the model to each position overlayed on the structure of Indole-3-glycee mnbole--glyceropphhsphte Synthase, TIM barrel protein. Higher entropy is assigned chains on the alpha helices, while lower entropy is assigned to the inward facing positions. Fig. 17 compares the probability assigned to hydrophobic, polar, and charged amino acids for buried sites vs. non-buried sites. The model prefers hydrophobic residues in the core and hydrophilic residues on the surface. The model probabilities closely match the empirical probabilities and those from the PSSM. Fig. 5C visualizes probability assigned to hydrophobic amino acids on the structure of
the selection of residues. Fig. 5B visualizes the entropy
residues having outward facing side

---

A.
180
B.
C.
180'
Lowest Entropy
Highest Entropy
Hydrophoblc
Hydrophilic
Figure 5: ESM- 1v reflects the molecular basis of function in proteins. (A) DNA methylase HaellI (pdbid: IDCT [29]). Side chains for the top 10 positions with lowest prediction entropy shown in blue. Low-entropy positions cluster in the active site. (B) TIM Barrel (pdbid: IIGS [30]) with residues colored by entropy. The model's 's predictions for residues on the surface have highest entropy (red) while those in the core have lower entropy (blue). Notably, residues on the alpha helices show a clear gradient from high to low entropy as residues transition from surface-facine to core- core-facing. (C) Sucrose- -prrosee Porin (pdbid: AOT [31]), a transmembrane protein. The model predicts hydrophobic band where the protein is embedded in the membrane.
Sucrose-specific Porin, transmembrane protein. The model predicts hydrophobic band in the center where the protein embeds in the membrane.
Calibration We evaluate model calibration using 1,008 sequences with length < 1024 from the trRosetta [32] dataset. ESM-Iv probabilities for each amino acid at each position are calculated with the masked marginal probability in Eq. (1). Fig. 6 shows that the model is generally well calibrated for all amino acids except Methionine. ESM- always predicts Methionine as the first position in the sequence since full protein sequences always start with it, model to subsequences. When excepting the first residue, the model achieves an average calibration error (defined for the multi-class setting in Appendix D.4) of 0.006.
must be used when applying the
We also explore the relationship between conservation (entropy of the PSSM) and the model's predicted entropy. Fig. 16 shows that these are well correlated (Pearson's r = 0.44), suggesting the model is able identify conserved positions.
6 Related Work
6.1 Protein language models
In the past few years, a number of groups have developed language models for protein sequences
[21, 33, 14, 34, 35, 22, 13, 12]. These models have been used for many tasks, including supervised low-N function prediction [16, 12], remote homology detection [22, 12], and protein generation [35]. The approach to the tasks typically involves transfer learning, where pretrained language model fine-tuned for particular problem. Vig et al. [36] and Rao et al. [26] found that transformer attention corresponds known biological properties such as structure and binding sites and can used to predict contacts.

---

Calibration plots (reliability curve)
All Positions
Without Residue
1.0
0.8
Fraction of positives
0.6
0.2
0.50
1.00
0.00
0.25
1.00
Mcan predicted value
Mcan predicted value
Figure Calibration plot for ESM-1v Iv predictions on each of the 20 naturally occurring amino acids on the trRosetta dataset. The multi-class classification is converted into set of 20 one- versus-all classífications for the purpose of this analysis. Left and right plots show calibration of all positions and positions excluding the first residue, respectively. Since full sequences always start with Methionine, the model overwhelmingly predicts model subsequences, such as those in the trRosetta dataset, this causes miscalibration at the first residue. Including the first residue, the model has an average calibration error (ACE) of 0.01 the first case and 0.006 in the second.
the first position. When evaluating the
Mutation effect prediction
Supervised and unsupervised methods have been developed for prediction of mutational effects. Supervised methods train models using experimental measurements or labels from databases of clinical variants. Standard machine learning tools including linear regression, random forests, and support-vector machines can be used [37]. Models have been designed specifically for proteins, using feature enginecring such as Envision [38] and PolyPhen-2 [39], ensemble methods such Revel [40], MPC [41], CADD [42], and M-CAP [43], language models such as UniRep [21, 16] and ESM [12], and other representation learning approaches [44, 45].
Unsupervised mutation effect predictors work by inferring the likelihood of mutation from the evolutionary landscape of the original protein. A density model fit to related sequences is used for
scoring. SIFT [46] is first order approach using a resition-specifie seccriiccscccingg-... matrix. EVMutation [4]
extends this to second-order approach by training a Potts model on the MSA. DeepSequence [20] includes higher-order interactions by training VAE on the MSA instead, using the ELBO score mutations. Riesselman et al. [47] proposes using an autoregressive model that does not require the sequences be aligned. Laine et [48] uses an evolutionary tree structure inferred from the MSA to compare mutations.
Hsu et al. [15] show that unsupervised mutational effect predictors can be extended perform supervised predictions, with better unsupervised predictors generally resulting in better supervised predictors. This suggests improving unsupervised prediction can drive progress in both settings. Concurrent with our work, Hie et al. [49] use open-source protein language models ESM- 1b and TAPE to predict the direction of evolution protein fitness landscapes.
Discussion
Advances in language modeling at scale are bringing the goal of general purpose model for proteins closer realization. This line work aspires model that learns to read and write biology in its native language, that can be directly applied across a range of protein understanding and design tasks. For scalability, learning from sequences is important: while there are no central databases of thgu-throughput functional measurements, and few compilations exist, billions of sequences are

---

available to learn from in sequence databases [50, 51]. Sequences give unparalleled view into the vast diversity and complexity of molecular parts invented by nature through billions of years of evolution.
Unsupervised structure [52-54, 28, 55, 56] and function [3, 4] learning methods first effectively realized the idea that biological properties could be read directly from sequences without supervision from experimental measurements. However these methods are not general purpose in the sense that specialized model must be trained for every protein for which a prediction is to be made. We show that the same performance can realized by general purpose model that has been trained across many diverse protein families. Similar to observations on the learning of tertiary protein structure in large language models [12, 26], we find that increasing the scale of models leads improvements in function learning. The understanding of mutational landscapes in the models correlates with the molecular basis of function in proteins, capturing binding sites and amino acid preferences that are determined by the folded structure.
Zero-shot transfer is interesting capability of large scale language models, and represents major point of departure from the unsupervised learning methods that are the basis for current state-of-the- art inference of protein structure and function. The capability for zero-shot transfer implies that model can be trained once and then applied to perform inference for many tasks. is also a window into deeper questions about the forms of generalization that are possible learning from sequences. Reading structural and functional design principles from sequences is a necessary capability for writing new biologically active sequences. Generalization in the zero-shot setting suggests the potential for large language models to capture knowledge that can be transferred to generating new functional proteins.
Acknowledgments and Disclosure of Funding
We thank Naman Goyal, Chloe Hsu, Zeming Lin, Ishan Misra, Myle Ott, Sergey Ovchinnikov, and Ethan Perez for valuable input on the paper. Roshan Rao was supported by funding from Facebook, Berkeley Deep Drive, and DARPA XAI.
10

---

References
[1] Charles J Epstein, Robert Goldberger, and Christian B Anfinsen. The genetic control of tertiary protein structure: studies with model systems. In Cold Spring Harbor symposia quantitative biology, volume 28, pages 439 449. Cold Spring Harbor Laboratory Press, 1963.
[2] Douglas M Fowler and Stanley Fields. Deep mutational scanning: a new style of protein science. Nature methods, 811) 2014.
[3] Mattco Figliuzzi, Hervé Jacquier, Alexander Schug, Oliver Tenaillon, and Martin Weigt. Co- evolutionary landscape inference and the context-dependennce of mutations in beta-lactamase tem-1. Molecular biology and evolution, 33(1):268-28)))) 2016.
[4] Thomas A. Hopf, John B. Ingraham, Frank J. Poelwijk, Charlotta P.I. Schärfe, Michael Springer, Chris Sander, and Debora S. Marks. Mutation effects predicted from sequence co-variation. Nature Biotechnology. 55(2):128-133) feb 2017. ISSN 15461696.
doi: 10.1038/mbt.3769.
[5] C Yanofsky, V Horn, and D Thorpe. Protein Structure Relationships Revealed By Mutational Analysis. Science (New York, N.Y.), 146(3651):1593344 12 1964. ISSN 0036-8075, URL
http://wwrr ncbi nlm. nih. gor/pubbed/1422450
[6] D. Altschuh, A.M. Lesk, A.C. Bloomer, and A. Klug. Correlation of co-ordinated amino acid substitutions with function in viruses related to tobacco mosaic virus. Journal of Molecular Biology, 193(4):693-707, 1987, ISSN 0022-2836. doi: 10.1016/0022- 020000022--22061799933-2------
[7] D Altschuh, Vernet, P Berti, D Moras, and K Nagai. Coordinated amino acid changes in homologous protein families. Protein Engineering. Design and Selection, 2(3):193-1)) 199, 1988.
[8] Ulrike Göbel, Chris Sander, Reinhard Schneider, and Alfonso Valencia. Correlated mutations and residue contacts in proteins. Proteins: Structure, Function, and Genetics, 18(4):309-317 4 1994, ISSN 0887-3585. doi: 00.1002/prot.340180000
URL http: //www//wwww//////./ ncbi alm.nih gov /ttp://ttt22222///// //ooi. wiley. com/ 10. 1002..00//poo00 340180400
[9] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[10] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR. 14165.
abs/2005.1 14165, 2020. URI. https: //aa/i///xxxiv
[11] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[12] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological Structure and Function Emerge from Scaling Unsupervised Learning 250 Million Protein Sequences. bioRxiv, page 622803, 4 2020. doi: 10. 10.1101/622803. URL. https:/ //wp://///wwww.w www.bborw.ii..oowwwwwwwwwwww...w biorxiv. cwtp/oontent/1... 1101/622803v3.
[13] Roshan Rao, Jason Liu, Robert Verkuil, Joshua Meier, John F. Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. MSA Transformer. bioRxiv, page 2021. 1021..2..1.4455 12.40888, feb 2021. doi: 10.1 101/2021 02. 12.430858.
[14] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing. bioRxiv, 7 2020. URL. attp://arxa httpaaawwww...aaaaaa/ra/ 0000 20222

---

[15] Chloe Hsu, Hunter Nisonoff, Clara Fannjiang, and Jennifer Listgarten. Combining evolutionary and uasay-labelles data for protein fitness prediction. bioRxiv, page 2021.03.28.437402. mar 2021. doi: 10. 10.1101/2021.033 0...0........... 23.244440......
[16] Surojit Biswas, Grigory Khimulya, Ethan C Low-N protein engineering with data-efficient deep learning. bioRxiv, page 2020. doi: 10. 010022000.01.2.9116666
Alley, Kevin M Esvelt, and George M Church.
[17] Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 951-958. IEFE, 2009.
[18] Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks, In AAAI, volume page 2008.
[19] Ramesh A, Pavlov M, Goh G, Gray S, Voss C, Radford A, Chen M, and Sutskever I. Zero-shot -iktte generation.
[20] Adam J. Riesselman, John B. Ingraham, and Debora S. Marks. Deep generative models of genetic variation capture the effects of mutations. Nature Methods, (5110)881-822) 10 2018. 8. ISSN 15487105.
doi: 10.1038/s4 191--118-2--8-888
[21] Ethan C. Alley. Grigory Khimulya, Surojit Biswas. Mohammed AlQuraishi, and George M. Church. Unified rational protein engineering with sequence-only deep representation learning. Nature Methods, 12:1315-1322, 3 2019. ISSN 15487105. doi: 10.1101/589333. URL
biorxiv.. https://wwes /ttt:o///ww.boorxvv. wwtp:/iooxww..r//oooooo.. wrtp/ooteen///...//////// /ttp://.......ttta////0/0/////// 1101/
[22] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating Protein Transfer Learning with TAPE. In Neural Information Prucessing Systems. Cold Spring Harbor Laboratory, 2019, doi: 10.1101/676825. hbtg://ab///// 2tt.b///999600000000
URL https://doi.
[23] Robert D. Finn, Alex Bateman, Jody Clements, Penelope Coggill, Ruth Y. Eberbardt, Sean R. Eddy, Andreas Heger, Kirstie Hetherington, Liisa Holm, Jaina Mistry, Erik L.L. Sonnhammer, John Tate, and Marco Punta. Pfam: The protein families database, 2014. ISSN 03051048. URL https: h/tww////wwwww. ncbi. nlm. mih, httph/////mmmmaaaticcce////MCC//
[24] Baris E. Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H. Wu. UniRef: Comprehensive and non-redundant UniProt reference clusters. Bioinformatics, 23(10): 1288, 2007. ISSN 13674803. uniprot.co org.
doi: 10. 1...10//ooontommaiis/ummm..... URL http://www.
[25] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North [A]merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171- 1- 4186, Minneapolis, Minnesota, 6 2019. Association for Computational Linguistics. doi: 10. 18655/v1/N19-1444 URL htthtt//// //t///xxvvvv. org aatp//a/a.11....////// 04805.
[26] Roshan Rao, Joshua Meier, Tom Sereu, Sergey Ovchinnikov, and Alexander Rives. Transformer protein language models are unsupervised structure learners. ICLR, page 2020.12.15.422761,
12 2021. doi: 10.
[27] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam McCandlish. Scaling laws for autoregressive generative modeling. CoRR, abs/2010.14701, 2020. URL https: / /aaxiv..vvv org/at/ /000//2010..em 14701.
[28] Faruck Morcos, Andrea Pagnani, Bryan Lunt, Arianna Bertolino, Debora S. Marks, Chris Sander, Riccardo Zecchina, José Onuchic, Terence Hwa, and Martin Weigt. Direct-coupling analysis of residue coevolution captures native contacts across many protein families. Proceedings of the National Academy of Sciences of the United States of America, 108(499):11233-11100 12 2011. ISSN 00278424. doi: 10.1073/pnas 111pp.......................1111 1111471108.
12

---

[29] KM Reinish, L Chen, GL Verdine, and WN Lipscomb. The crystal structure of hacii methyl- transferase covalently complexed dna: an extrahelical cytosine and rearranged base pairing. Cell, 82(1):143-153, 1995.
[30] Michael Hennig, Beatrice Darimont, Reinhard Sterner, Kasper Kirschner, and Johan N Jansonius. 2.0 å structure of indole-3 3ndole---glycerol phosphate synthase from the hyperthermophile sulfolobus solfataricus: possible determinants of protein stability. Structure, 3(12): 1295-136,, 1995.
[31] Doris Forst, Wolfram Welte, Thomas Wacker, and Kay Diederichs. Structure of the sucrose- specific porin sery from salmonella typhimurium and its complex with sucrose. Nature structural biology, 5(1):37-46, 1998.
[32] Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, David Baker, and John Harvard. Improved protein structure prediction using predicted inter-residue orientations. bioRxiv, page 846279, 2019. doi: 10.1101/846279. URL httht::: /ttapss://// http://www.bbb...ooo/////
org/ wrg/content/1l 10. 1101/846279v
[33] Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure, 2019. URL http://arxila http:///axxvvoor/aaaaa abs/1902. 08661
h86611ttss:// org/
[34] Amy X Lu, Haoran Zhang, Marzych Ghassemi, and Alan Moses, Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization. bioRxiv, page 2020.09.00 29.3229999,, 9 2020. doi: 10.1101/2020.00 www...9..442999. URL https::////oo /ttp://///ooo./1...././ 1101/2020. 09. 04. 283929.
[35] Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R. Eguchi, Po-Ssu Huang, and Richard Socher. ProGen: Language Modeling for Protein Genera- tion. bioRxiv, 2020. URL http://arxiv> http:/iiw.rg/a/b///////
[36] Jesse Vig, Ali Madani, Lav R. Varshney, Caiming Xiong. Richard Socher, and Nazneen Fatema Rajani. BERTology Meets Biology: Interpreting Attention in Protein Language Models. bioRxiv, page 2020.06.222 26. 174417, 6 2020.
doi: 10.1101/2020.06.26.III 174417. URL http:
//arxiv. otg// 15222.
[37] Kevin K. Yang, Zachary Wu, and Frances H. Arnold. Machine-leaminngguieee directed evolution for protein engineering, aug 2019. ISSN 15487105.
[38] Vanessa E. Gray, Ronald J. Hause, Jens Luebeck, Jay Shendure, and Douglas M. Fowler. Quantitative Missense Variant Effect Prediction Using Large-Scall Mutagenesis Data. Cell Systems, (11::166-22.e3, jan 2018. ISSN 24054720. doi: 20.1016).ells2201111.000
[39] Ivan A. Adzhubei, Steffen Schmidt, Leonid Peshkin, Vasily E. Ramensky, Anna Gerasimova, Peer Bork, Alexey S. Kondrashov, and Shamil R. Sunyaev. A method and server for predicting damaging missense mutations, apr 2010. ISSN 15487091.
[40] Nilah M loannidis, Joseph H Rothstein, Vikas Pejaver, Sumit Middha, Shannon K McDonnell, Saurabh Baheti, Anthony Musolf, Qing Li, Emily Holzinger, Danielle Karyadi. et al. Revel: an ensemble method for predicting the pathogenicity of rare missense variants. The American Journal of Human Genetics, 99(4):877-885, 2016.
[41] Kaitlin E Samocha, Jack A Kosmicki, Konrad J Karczewski, Anne H O'Donnell-- Luria, Emma Merce-HHoffman Daniel MacArthur, Benjamin M Neale, and Mark Daly. Regional missense constraint improves variant deleteriousness prediction. BioRxiv, page 148353, 2017.
[42] Martin Kircher, Daniela M Witten, Preti Jain, Brian J O'Roa, Gregory M Cooper, and Jay Shendure. A general framework for estimating the relative pathogenicity of human genetic variants. Nature genetics, 16(3):310-315 2014.
[43] Karthik Jagadeesh, Aaron Wenger, Mark Berger, Harendra Guturu, Peter Stenson, David N Cooper, Jonathan A Bernstein, and Gill Bejerano. M-cap eliminates a majority of variants uncertain significance in clinical exomes at high sensitivity, Nature genetics, 48(12):
1581, 2016.
13

---

[44] Laksshman Sundaram, Hong Gao, Samskruthi Reddy Padigepati, Jeremy F McRae, Yanjun Li, Jack Kosmicki, Nondas Fritzilas, Jörg Hakenberg, Anindita Dutta, John Shon, al. Predicting the clinical impact of human mutation with deep neural networks. Nature genetics, 50(8):1161-111) 2018.
[45] Haicang Zhang, Michelle $ Xu, Wendy K Chung, and Yufeng Shen. Predicting functional effect of missense variants using graph attention neural networks. bioRxiv, 2021.
[46] Ngak Leng Sim, Prateek Kumar, Jing Hu, Steven Henikoff, Georg Schneider, and Pauline C. Ng. SIFT web server: Predicting effects of amino acid substitutions on proteins. Nucleic Acids Research, 40(W 40451) jul 2012. ISSN 03051048. doi: 10. 1093/nar/gks539.
[47] Adam Riesselman, Jung-Eun Shin, Aaron Kollasch, Conor McMahon. Elana Simon, Chris Sander, Aashish Manglik, Andrew Kruse, and Debora Marks. Accelerating Protein Design Using Autoregressive Generative Models. bioRxiv, page 757252, 2019. doi: 10.1101/75725..
URL attpp://www, biorxiv. otg. wwwttet/to...iiioooo 11tp//00001111/772222111
[48] Elodie Laine, Yasaman Karami, and Alessandra Carbone. Gemme: simple and fast global epistatic model predicting mutational effects. Molecular biology and evolution, 36(1 1):2644 2619, 2019.
[49] Brian L Hie, Kevin K Yang, and Peter S Kim. Evolutionary velocity with protein language models. bioRxiv, 2021.
[50] Henrik Nordberg, Michael Cantor, Serge Dusheyko, Susan Hua, Alexander Poliakov, Igor Shabalov, Tatyana Smirnova, Igor V Grigoriev, and Inna Dubchak. The genome portal of the department energy joint genome institute: 2014 updates. Nucleic acids research, 42(D1): D26-D31, 2014.
[51] M. Steinegger, M. Mirdita, and J. Söding. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. Nat Methods, 16(7):603-606, 07 2019.
[52] Alan S. Lapedes, Bertrand G. Giraud, LonChang Liu, and Gary D. Stormo. Correlated Mutations in Models of Protein Sequences: Phylogenetic and Structural Effects. Lecture Notes Notes-Monograph Series, 33:236-256, 1999. doi: 10.2307/4356049, URL http::/wwat jstor. org/ stable/ 4356049.
[53] John Thomas, Naren Ramakrishnan, and Chris Bailey-Kellogg. Graphical models of residue coupling in protein families, 4 2008. ISSN 15455963. URL https://pubmed ncbi nlm. nih go////81/84
[54] Martin Weigt, Robert A. White, Hendrik Szurmant, James A. Hoch, and Terence Hwa. Identifi- cation of direct residue contacts in protein-protein interaction by message passing. Proceedings of the National Academy of Sciences of the United States of America, 106(1):67-72, 1 2009. ISSN 00278424. doi: 10. 10.1073/pmas. 1..1073/pmas.000052231666 08050233000099 URL https: //ww. pnas. wwg/contenteeooooooooooooooomeee 106/1/67https: //www. pnas. wttp://oo/tt//////////// 1/t00///////////////// abstract.
[55] Sivaraman Balakrishnan, Hetunandan Kamisetty, Jaime G. Carbonell, Su-In Lee, and Christo- pher James Langmead. Learning generative models for protein fold families. Proteins: Structure, Function, and Bioinformatics, 79(4):0611-1088. 4 2011. ISSN 08873585. doi: 10.1002/ppot.229344 URL /attp:///oi.wiii wwtli...wel........... www.m0..m.....oo........ 1002 10w.10002ppprooo 22934.
[56] David T. Jones, Daniel W. A. Buchan, Domenico Cozzetto, and Massimiliano Pontil. PSI- COV: precise structural contact prediction using sparse inverse covariance estimation large multiple sequence alignments. Bioinformatics, 28(2):184-190, I 2012. ISSN 1460-2059, doi: 10. 10..099/biciffommatss/ttnos URL https / /ttp///aacaaccccca////////. oup. www/biiiinoormttiiiooooooo article-lookup/dor
[57] Emily E Wrenbeck, Matthew S Faber, and Timothy A Whitchead. Deep sequencing methods for protein engineering and design. Current opinion in structural biology, 45:36-44, 2017.
[58] Justin R Klesmith, John-Paul Bacik, Ryszard Michalezyk, and Timothy A Whitehead. Compre- hensive sequence-fuu flux mapping of a levoglucosan utilization pathway in e, coli. ACS synthetic biology, 4(11):123511243 2015.
14

---

[59] Hugh K Haddox, Adam S Dingens, Sarah K Hilton, Julie Overbaugh, and Jesse D Bloom. Mapping mutational effects along the evolutionary landscape of hiv envelope. Elife, 7:e34420, 2018.
[60] Philip A Romero, Tuan M Tran, and Adam R Abate. Dissecting enzyme function with microfiuidic-based deep mutational scanning. Proceedings of the National Acauemy of Sciences, 1 112(23):7159-716 2015.
[61] Elad Firbeerg, Jason W Labonte, Jeffrey J Gray, and Mare Ostermeier. A comprehensive, high-resolution map a gene's fitness landscape. Molecular biology and evolution, 31(6): 1581-1592, 2014.
[62] Zhifeng Deng, Wanzhi Huang, Erol Bakkalbasi, Nicholas G Brown, Carolyn J Adamski, Kacie Rice, Donna Muzny, Richard A Gibbs, and Timothy Palzkill. Deep sequencing of systematic combinatorial libraries reveals 3-lactamase sequence constraints at high resolution. Journal of molecular biology, (24(3-4):150-16) 2012.
[63] Michael A Stiffler, Docke Hekstra, and Rama Ranganathan, Evolvability as a function of purifying selection in tem-1 B-lactamase. Cell, 160(5):882 892, 2015.
[64] Hervé Jacquier, André Birgy, Hervé Le Nagard, Yves Mechulam, Emmanuelle Schmitt, Jérémy Glodt, Beatrice Bercot, Emmanuelle Petit, Julie Poulain, Guilène Barnaud, et al. Capturing the mutational landscape of the beta-lactamase tem-1. Proceedings of the National Academy of Sciences, 1 10677)):3667 2013.
[65] Scott D Findlay and Lynne-Marie Postovit. Comprehensive characterization of transcript diversity the human nodal locus. BioRxiv, page 254409, 2018.
[66] Richard N McLaughlin Jr, Frank J Poelwijk, Arjun Raman, Walraj S Gosal, and Rama Ran- ganathan. The spatial architecture of protein function and adaptation. Nature, 491(7422): 138-142, 2012.
[67] Jacob O Kitzman, Lea M Starita, Russell S Lo, Stanley Fields, and Jay Shendure. Massively parallel single-amino-acia mutagenesis. Nature methods, 12(3):203-206, 2015.
[68] Michael B Doud and Jesse D Bloom. Accurate measurement of the effects of all amino-acid mutations influenza hemagglutinin. Viruses, 8(6):155, 2016.
[69] Victoria O Pokusaeva, Dinara R Usmanova, Ekaterina Putintseva, Lorena Espinar, Karen Sarkisyan, Alexander S Mishin, Natalya S Bogatyreva, Dmitry N Ivankov, Arseniy V Akopyan, Sergey Ya Avvakumov, et al. An experimental assay of the interactions of amino acids from orthologous sequences shaping a complex fitness landscape. PLoS genetics, 15(4)) 2019.
[70] Parul Mishra, Julia M Flynn, Tyler N Starr, and Daniel NA Bolon. Systematic mutant analyses elucidate general and client-specific aspects of hsp90 function. Cell reports, 15(3):588-555 598, 2016.
[71] Eric D Kelsic, Hattie Chung, Niv Cohen, Jimin Park, Harris Wang, and Roy Kishony. Rna structural determinants of optimal codons revealed by mage-seq. Cell systems, 366):563-57,, 2016.
[72] Alexandre Melnikov, Peter Rogov. Li Wang, Andreas Gnirke, and Tarjei Mikkelsen. Compre- hensive mutational scanning of a kinase in vivo reveals substrate-dependee fitness landscapes. Nucleic acids research, 42(14):el 1 12-122282 12, 2014.
[73] Lisa Brenan, Aleksandr Andreev, Ofir Cohen, Sasha Pantel, Atanas Kamburov, Davide Cac- chiarelli, Nicole S Persky, Cong Zhu, Mukta Bagul, Eva M Goetz, et al. Phenotypic characteri- zation of a comprehensive set of m/pk1/ missense mutants. Cell reports, 1714):177 2016.
[74] Liat Rockah-Shmuel, Ágnes Tóth-Petróczy, and Dan S Tawfik. Systematic mapping of pro- tein mutational space by prolonged drift reveals the deleterious effects of seemingly neutral mutations. PLoS computational biology, 11())e1100421, 2015.
15

---

[75] Nicholas C Wu, C Anders Olson, Yushen Du, Shuai Le, Kevin Tran, Roland Remenyi. Danyang Gong, Laith Q Al-Mawsawi, Hangfei Qi. Ting-Ting Wu, et al. Functional constraint profiling of a viral protein reveals discordance of evolutionary conservation and functionality. PLOS generics, 1107)xe005311, 2015.
[76] Christopher D Aakre, Julien Herrou, Tuyen N Phung, Barrett S Perchuk. Sean Crosson, and Michael T Laub. Evolving new protein-protein interaction specificity through promiscuous intermediates. Cell, 163(3):594-606, 2015.
[77] Hangfei Qi, Anders Olson, Nicholas Wu, Ruian Ke, Claude Loverdo, Virginia Chu, Shawna Truong, Roland Remenyi, Zugen Chen, Yushen Du, et al. A quantitative high-resolution genetic profile rapidly identifies sequence determinants of hepatitis viral fitness and drug sensitivity. PLoS pathogens, (((4)=e004064 2014.
[78] Kenneth A Matreyek, Lea M Starita, Jason J Stephany, Beth Martin, Melissa A Chiasson, Vanessa E Gray, Martin Kircher, Arineh Khechaduri, Jennifer N Dines, Ronald Hause, et al. Multiplex assessment of protein variant abundance by massively parallel sequencing. Nature genetics, 50(6):874-88,, 2018.
[79] Pradeep Bandaru, Neel H Shah, Moitrayee Bhattacharyya, John P Barton, Yasushi Kondo, Joshua C Cofsky, Christine L Gee, Arup K Chakraborty, Tanja Kortemme, Rama Ranganathan, al. Deconstruction of the switching cycle through saturation mutagenesis. Elife, 6:e27810, 2017.
[80] Benjamin P Roscoe, Kelly M Thayer, Konstantin B Zeldovich, David Fushman, and Daniel NA Bolon. Analyses the effects of all ubiquitin point mutants on yeast growth rate. Journal of molecular biology, 125(8):1363-137) 1377, 2013.
[81] Benjamin P Roscoe and Daniel NA Bolon. Systematic exploration of ubiquitin sequence, el activation efficiency, and experimental fitness in yeast. Journal of molecular biology, 426(15): 2854-2870, 2014.
[82] David Mavor, Kyle Barlow, Samuel Thompson, Benjamin A Barad, Alain R Bonny, Clinton L Cario, Garrett Gaskins, Zairan Liu. Laura Deming, Seth Axen, et al. Determination of ubiquitin fitness landscapes under different chemical stresses in a classroom setting. Elife, 5: e15802, 2016.
[83] Yvonne H Chan, Sergey V Venev, Konstantin B Zeldovich, and C Robert Matthews. Correlation of fitness landscapes from three orthologous tim barrels originates from sequence and structure constraints. Nature communications, 8(1):1-12, 2017.
[84] Daniel Melamed, David L Young, Caitlin E Gamble, Christina R Miller, and Stanley Fields. Deep mutational scanning of an rmm domain of the saccharomyces cerevisiae poly (a)-binding protein. Rna, 19(11)115771155 2013.
[85] Lea M Starita, Jonathan N Pruneda, Russell S Lo, Douglas M Fowler, Helen Kim, Joseph B Hiatt Jay Shendure, Peter Brzovic, Stanley Fields, and Rachel E Klevit. Activity-enhancingg mutations in an e3 ubiquitin ligase identified by high-throughput mutagenesis. Proceedings of the National Academy of Sciences, 11((1))E1163-E11
2013.
[86] Carlos L Araya, Douglas M Fowler, Wentao Chen, Ike Muniez, Jeffery W Kelly, and Stanley Fields. A fundamental protein property, thermodynamic stability, revealed solely from large- scale measurements of protein function. Proceedings of the National Academy of Sciences, 109 10))16888-16888 2012.
[87] Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. Pseudolikelihood reranking with masked language models. CoRR, 191119911.169999 2019. URL http:///axiiii .rg/ abs/ 1910. 14659.
[88] Kevin K. Yang, Zachary Wu, Claire N. Bedbrook, and Frances H. Arnold. Learned protein embeddings for machine learning. Bioinformatics, 34(15):2642-2645 8 2018. ISSN 14602059. doi: 10.109//iionfomaatissyyy..l
16

---

[89] Andrew Leaver-Fay, Matthew J O' meara, Mike Tyka, Ron Jacak, Yifan Song, Elizabeth H Kellogg, James Thompson, Ian W Davis, Roland A Pache, Sergey Lyskov, et al. Scientific benchmarks for guiding macromolecular energy function improvement. Methods in enzymology, 523:109-143, 2013.
[90] Milot Mirdita, Lars Von Den Driesch, Clovis Galiez. Maria J. Martin, Johannes Soding, and Martin Steinegger. Uniclust databases of clustered and deeply annotated protein sequences and alignments. Nucleic Acids Research, (IDD)):D17--D176: 2017. ISSN 13624962. doi: 10. 1003/narrgkww00/.
[91] Lukas Neumann, Andrew Zisserman, and Andrea Vedaldi. Relaxed Softmax: Efficient Confi- dence Auto-Calibration for Safe Pedestrian Detection. Technical report, oct 2018.
[92] Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring Calibration in Deep Learning. apr 2019. URL http: / //arxiv. www./org//////ooe /904 01685.
[93] L. Steven Johnson, Sean R. Eddy, and Elon Portugaly. Hidden Markov model speed heuristic and iterative HMM search procedure. BMC Bioinformatics, 11(1):431, 8 2010. ISSN 14712105, doi: 10. 10.1186/147122105--1-411 URL https: http:::://bmmbbioonfffooeeeeeee com/articles/100 110111111111111111111----- 11-431.
biomedcentra..
[94] Didier Nurizzo, Steven C Shewry, Michael Perlin, Scott A Brown, Jaydev Dholakia, Roy Fuchs, Taru Deva, Edward N Baker, and Clyde A Smith. The crystal structure of aminoglycoside- 3-phosshhotransferrse--iia an enzyme responsible for antibiotic resistance. Journal of molecular biology, 327(2)4491-50 2003.
17


Reviews:
[{"review_id": "sUIrRbGvpcQ", "reviewer": "Reviewer_Bsx3", "summary": "This paper demonstrates utilizing a pretrained masked language model to predict a ranking of functional activity of a protein.\n\nThey focus on data from deep mutational scanning, which measures the functionality of a large set of individual mutations to a protein in order to characterize the fitness landscape around the protein (wild type) for the desired functionality.\n\nIt proposes a method which uses the log odds ratio between a mutated variant and the wild type protein as a proxy for the fitness of the mutation. It evaluates the methods by using Spearman correlation between its ranking and ground truth fitness from deep mutational scanning datasets. \n\nIt evaluates a wide set of existing baselines, and focuses analysis on an updated version of the ESM-1b model, ESM-1v. It compares several methods for evaluating the log odds of a particular mutation.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI am primarily evaluating the method from the perspective of a machine learning researcher who has done related work applying self-supervised learning methods to protein data.\n\nOverall, the method is well motivated, simple, effective, and likely to lead to fruitful future research. The method is carefully evaluated against several baselines.\n\n**Originality**: The datasets are standard benchmarks for evaluating protein function prediction. The models are variants on standard BERT models, with evaluations to select the right pretraining dataset and compare model sizes.\n\nIt demonstrates a simple new method on top of this established model to accomplish a core task in protein design (characterizing function of mutations to a protein) with minimal overhead.\n\nIt performs about as well as methods that use a full MSA while only requiring a single sequence.\n\n**Quality**: The paper is technically sound and carefully evaluated. It answered questions I had while reading it with experiments (and a thorough appendix).\n\n\n\n* Proper validation sets are used\n* Baselines are well tuned (even outperforming results in source publication)\n* results are reported using sound statistics\n* The ESM-1v model is evaluated at several different scales and across many pre-training datasets\n* Several alternatives are considered for the primary choices made in the paper (e.g. how to score the effects of mutations given the model, which pre-training dataset to use, how to \n* Good discussion of related work, in both ML + protein and NLP literature.\n* Extends the model analysis beyond the task at hand to how the models do at modeling the stability of residues located in different regions of the folded protein.\n\n**Clarity:** The paper is clearly written, with sufficient references to related work and background. Figures clarify content of the text (such as the overall task, the pipeline for each baseline and the proposed method, and visualizations of relationship between position and model entropy).\n\nI believe a practitioner familiar with the data and BERT models could reproduce the results as described.\n\n**Significance:** I believe other researchers are likely to build on the methods outlined in the paper. While it does not always exceed the baselines, the method generally performs near or better than existing methods despite only needing a single protein sequence (instead of a full MSA). It is also the first paper I have seen that explicitly evaluates scaling on protein datasets (which is why I would like to see more discussion of this point).\n\n**Feedback for improvement**\n\n**– content –**\n\n\n\n* Consider saying \"masked language models\" in the title to point at the specific model class used\n* line 210: What methods are appropriate for handing Methionine in the first position (\"care must be used when applying the model to a subsequence\")\n* Model\n    * Any theories as to why uniref90 performs the best?\n    * Does the pretraining only use the cluster centers from each cluster? Consider adding a section to appendix expanding on the pretraining details, even if it is explained in a referenced paper\n    * How large was the dataset? Roughly how many epochs was 170k steps of 1 million tokens\n    * figure 9: Add  figure or table including other pretraining metrics alongside just the plot of rho with respect to model size/update steps. What is the per amino acid accuracy? Perplexity? table 12 has some of this for the large model, but the other model information is also useful.\n        * Generally suggest trying to characterize relationship between pretraining performance and downstream performance on your task as measured by rho.\n* I'm sure I've seen similar methods to \"spiked unsupervised fine-tuning\" used in NLP literature to regularize/combat forgetting, but I'm struggling to find a reference… I will see if I can find something for the discussion period.\n* Equation at line 92:\n    * Define mt=mutated, wt=wild type before equation\n    * should the conditioning be x^{mt}_{-t} and x^{wt}_{-t}?\n* 96/104: consider describing more about what fitness landscape means (fitness of the region around the protein in sequence space).\n* 526: The #s don't add up:\"among the 15% predicted positions, 10% are randomly, 10% retain …\"\n\n**– style –**\n\n\n\n* 54: add comma after \"in biology\"\n* Define MSA when it is first used (79)\n* Line 94: You do not describe what ESM-1v before referring to it\n* table 7: misspelled marginal\n* figure 9\n    * label the y axis\n    * The bottom of the figure is cut off\n* table 1: Add reference to appendix section where you describe sampling for the MSA", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper's appendix discussion focuses on compute costs.  While far off, it is worth discussing potential applications of this technology (good and bad), even if just referencing existing writing on potential dangers of protein engineering.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "nXQoxUSNBDB", "reviewer": "Reviewer_85hq", "summary": "The paper analyses the generalization capacity of the protein language models. The experiments and analysis suggest that zero-shot inference of protein language models can capture the functional effects of sequence variation. The paper is essentially analyzing the protein language models on their generalization capacity of zero-shot transfer learning for the Mutation Effect Prediction task. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper analyses the recent advancements of protein language models (PLM) on a specific task to check the generalization capacity of a PLM. However, the paper does not discuss the robustness of the zero-shot predictions. An error analysis will be really useful for understanding the true generalization capacity of the models.\n\nThere is no methodological novelty in the approach. It was also difficult to read the paper as a lot of reference to Appendix and supplementary materials.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The deep neural network-based language models are good at predictions. However, they are prone to adversarial attacks, meaning that changing input slightly can change the predictions. Inference based on such a model can be problematic for a sensitive task like designing protein in the current state of the models. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "Four hours"}, "extra_text": {}}, {"review_id": "XTIV6V2OKRp", "reviewer": "Reviewer_bbgP", "summary": "The paper adds a follow-up experiment to a recent paper demonstrating the power of large language models (particularly, BERT and the MSA transformer) for a variety of protein structure and function prediction tasks. The new task (zero-shot scoring of the effects of mutations) has a variety of useful applications for clinical medicine and protein design and has been explored extensively in recent papers that used alternative models. This paper demonstrates that the these new models perform similarly, or perhaps slightly better, on a standard suite of datasets.\n\n\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**originality**\n\nMy primary negative critique of this paper is its lack of novelty.  \n\nIt has been previously established in the literature by many papers that the functional effect of mutations can be predicted in a zero-shot setting by scoring variants using the likelihood of a generative model. Further, the two models considered in this paper have been introduced in recent prior work. Those papers are really great; I found them rigorous and well written and would have advocated for their acceptance to NeurIPS. I feel that the scale of contributions of the current paper is smaller than that of a typical NeurIPS paper, though, and its content would be better placed as an additional section of experiments in, for example, the MSA transformer paper.\n\nOne argument for a paradigm shift due to your approach is that practitioners don't have to train a new model for a protein family of interest and don't have to perform multiple sequence alignment of homologous sequences. Instead, the user can just do forward passes in an existing model. However, it's rare for practitioners to be considering a large number of families at once. See further comments below.\n\n**clarity/quality**\nThe paper is well written and cites relevant recent work. The experimental results could have been analyzed in more detail. See comments below.\n\n**significance**\nThis paper is part of a really important line of work on deep learning + proteins. My hesitation is around whether this paper should appear as a stand-alone paper at NeurIPS, not whether the results are of interest to the community.\n\n**zero-shot prediction**\n\nI disagree with your definition of zero-shot in the paragraph starting on line 75. To me, any method is zero-shot if it doesn't rely on experimental data as input. Why is a method that relies on querying a sequence database to retrieve homologous sequences qualitatively different than your approach?  Doesn't the training set for ESM-1v or MSATransformer include the same 'weak supervision' that the traditional approach relies on?\n\nI suppose you could argue that your method could be applied to predict the functional effect of mutations on sequences that are very different than anything that appears in nature, where there is no associated protein family that could be retrieved by a database search. This is zero-shot in a different sense and not evaluated in the experiments.\n\n**computing the likelihood of a variant**\n\nIn Eq (1) uses P(x_t | x_{-t}), but ignores P(x_{-t} | x_t). Why is this a reasonable assumption? Surely, when making a mutation at a particular position, it effects the likelihood of the values at the other positions too. Note that prior models for this task (VAEs, autoregressive models, Potts models) don't make such a simplifying assumption. I see that you discuss alternatives in Appendix A. Why do you think this simple approach is sufficient?\n\n **Epistasis**\nFollowing up on the previous point, does this mean that you can't capture epistasis, which is known to be prevalent in fitness landscapes? For datasets with multiple mutations, you can compute the subset of examples that are epistatic (e.g., it the variant is non-functional, but combines mutations that led to a functional variant when used individually). On these, how do you compare to DeepSequence, which can capture epistasis?\n\n**ESM-1b vs. ESM-1v**\nAm I correct in understanding that ESM-1v is the same as ESM-1b, but trained on slightly different data? When I discuss originality above, I argue that the paper does not introduce a new model. To me, changing the training data does not amount to introducing a new model with the degree of originality characteristic of a stand-alone paper.\n\n\n**Inference speed**\nI don't feel that this is a particularly important distinction between the proposed model and prior work. It's not clear what an application setting would be where practitioners need to make predictions for many protein families at high throughput. If the goal is just to predict  the functional effects of mutations on a single parent sequence, then the speed difference across techniques will be minor.  The goal of this zero-shot prediction is to avoid needing to run wet-lab experiments, so any approach that can run on a computer in less than an hour is a massive speedup vs. the alternative.\n\n\n **Evaluation Metrics**\nCan you please explain why spearman correlation is a reasonable performance metric? This is noisy experimental data. What do you think is the upper limit on performance? If you binarized the experiment measurement (e.g. whether a variants is worse than the wildtype or better), what would detection metrics like AUC or precision@k look like? I'm also assuming that such labels would be very imbalanced (with more than half of the mutations leading to a decrease in function). Doesn't that bias spearman rho, and make it less predictive of performance for protein design applications where the goal is to increase fitness?\n\n**Break-down of performance**\n\nI would have appreciated more analysis of where the method succeeds and fails. What characterizes datasets where your methods outperform/underperform prior work? For example, how do you compare on multi-mutant variants?\n\n**Sec 5**\nAgain, this feels like a section that should appear in a long-form article on ESM or MSA Transformer. Are these results good compared to baseline methods? How does a PSSM perform on them? Is the goal to show that you are outperforming a baseline, or are you just doing some sanity checks for the model?\n\n\n*****Updated review after discussion period*****\nI have decided to raise my review to a weak accept, after a lengthy discussion period with the authors and between the reviewers. Overall, there are a number of detailed choices that were important to achieve the paper's results, and the community should know about these. \n\nMy principal hesitation is not around the merits of the paper's contribution, which is valuable, but in how the contribution is framed. If the paper is accepted, I would really appreciate it if you considerably changed the exposition (including the title) to be more precise about what's new in the paper. Specifically, I'd refer to your work and prior work (e.g., DeepSequence) both as zero-shot. \n\nThe title 'Language models enable zero-shot prediction of the effects of mutations on protein function' suggests that there is a new capability that they have 'enabled.' This is certainly not the case. Prior work uses exactly the same data (naturally-occurring sequences, but no experimentally-labeled data) and achieves comparable results. When I first started working on ML for proteins, I was enthralled when I learned that both protein structure and the functional effects of mutations could be predicted by inspecting a statistical model of homologous sequences. This is a remarkable result, and I'm concerned that the twitter-verse will mistakenly think that this particular paper unlocked it (it's actually been known for many years). \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall yes, however this statement should be revised: \" Otherwise,basic methods in computational biology have a long time horizon to concrete social impact.\" Just because something has a long time horizon doesn't mean that it is benign. ", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "H4wTMBiyMse", "reviewer": "Reviewer_kUmC", "summary": "The paper describes a model that\n- Is pretrained jointly across many protein families\n- Learns functional effects of mutations at state-of-the-art for deep mutational scans\n- Recapitulates substitution patterns concordant with biology\n- Does not require MSAs at all", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper solves the problem of functional mutation effect prediction, and does it as well as anyone else, with a very simple setup. They demonstrate these results convincingly. My main concern with this paper is that they achieve these results in a zero-shot manner, but don’t demonstrate that fine-tuning improves performance, which is very surprising. That said, I do not think that they need to completely answer the question as to why in order for this to be a complete body of work.\n\nOriginality: Not very original methods. Use of current dataset.\nQuality: Good quality. Lots of qualitative analysis. Lots of baselines. Baselines were tuned beyond my expectation.\nClarity: clear. Good prose.\nSignificance: Zero-shot performance mirroring state-of-the-art in an area that has substantial impact, like pharmaceuticals, is very significant.\n\nSee below for more specific details.\n\n-----------------------------------------------\n\nThe paper describes a model that\n- Is pretrained jointly across many protein families\n- Learns functional effects of mutations at state-of-the-art for deep mutational scans\n- Recapitulates substitution patterns concordant with biology\n- Does not require MSAs at all\n\nThe evidence they provide is\n- _Is pretrained jointly across many protein families_\n  - Use UniRefXX for many values of XX\n- _Learns functional effects of mutations_\n  - Quantitative evidence via deep mutational scans (DMS)\n    - 41 DMS examined\n    - Spearman R used - SOTA values\n    - Comparison to DeepSequence, ESM-1b, ProtBERT-BFD, UniRep, TAPE\n  - Show entropy vs ground truth PSSMs\n  - Show calibration of the model\n- _Recapitulates substitution patterns concordant with biology_\n  - 41 DMS examined\n  - Explanations of substitution patterns like hydrophobicity and buried vs exposed via some anecdotes and statistics\n- _Does not require MSAs at all_\n  - Via an explanation of their architecture, training procedure, and inference setup\n\nThe benefits of their approach vs some SOTA are\n- No fine tuning step needed\n- No family-based training objective function\n- Not trained for-purpose - should be generally useful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have sufficiently described potential negative societal impact.\n\nRe: limitations: Questions for author:\n- Why is it that fine tuning ESM-1v didn’t statistically improve performance across the 41 DMS tasks via t test?\n  - What sort of level of improvement would be needed for this?\n  - Are there statistically significant claims you can make about the fine-tuning performance?\n  - Why was there such a big jump from fine tuning for some datasets but not others?\n  - There are still some tasks where fine-tuned performance is statistically lower than DeepSequence. Do you have any explanations for this?\n- Is there a baseline you can compare to for calibration error? It’s a little difficult to understand whether ACE of 0.006 is good. How difficult is it to create a model that does this well?\n- What is the p value on pearsonr for PSSM entropy and predicted entropy?\n- Saying pearsonr of .44 on the entropy claim is well correlated is slightly hyperbolic - I suggest wording.\n- Is the main claim that this is faster than current methods? Or faster to train? Fewer parameters? If so, a comparison to DeepSequence in this would be helpful. \n  - Perhaps this is already in the literature (transitively?) and I don’t know about it, in which case a reminder in the text would be useful.\n- A connection between calibration and why it’s important for downstream tasks would help the paper’s sales pitch.\n- Figure 6: Average time to compute a DMS is a slightly odd statistic, as this measure is quite dependent on somewhat extraneous factors like the assays used in the experiments. I suggest normalizing a different way, e.g. per sequence or per residue.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DseSRk94eqb", "reviewer": "Reviewer_FVbp", "summary": "Unsurprising results and the method is not novel.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Modeling the impact of amino acid mutations on protein-protein interaction plays a crucial role in protein engineering and drug design. But the structures of different families of proteins vary a lot, hindering the generalization of a machine learning model. This paper found that, without any supervision from experimental data or additional training, protein language models capture the functional effects of sequence variation, achieving state-of-the-art zero-shot prediction performance.\n\nHowever, it is somewhat expected. In principle, language models learn the distribution of the protein sequences that are stable. The unstable sequences are usually recognized by language models as low probabilities of existence.\n \nThe introduction of the used prediction models is missing. It is hard to understand the acronyms without explanation.\n\nThe implementation details are not clear. What are the hyperparameters and how do you finetune them?\n\nHave the authors compared the deep learning models with the traditional evolutionary conservation profiles?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nThe language models are not proposed by this paper. It is not clear what is the contributions of this paper except for the conclusion that language models can partially predict the protein function.\n\nWe also expect to see if language models can provide some interesting biological insights.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: While evaluating this paper, the reviewers had an extensive discussion about the relative strengths, and in particular what counts as "novelty".  While the reviewers did not come to unanimous consensus here, I am swayed by the majority of reviewers who cited the strong empirical rigor in analyzing the behavior of the proposed models, in addition to the careful combination of many small improvements that lead to strong results on an important application area in predicting the functional effects of protein mutations.  Indeed, my own view is that the term "novelty" is often overly constrained to mean "a brand new modeling technique" when instead it should be interpreted to mean "something important that adds to the field's overall understanding and knowledge".  Indeed, the recent ML literature is filled with important papers that usefully shine new light on previously known modeling techniques or methods.

Given the strong reviews from the majority of reviewers, and the acknowledgement of the strong empirical results and careful explication of the rigorous evaluations, I am happy to recommend acceptance of this paper.  I do expect that the authors will use the extensive reviewer feedback to further revise and strengthen the paper for its final form.
Decision: Accept (Poster)