Language models enable zero-shot prediction of the effects of mutations on protein function

1 Introduction . Proteins have a myriad of diverse functions that underlie the complexity of life . Protein sequences encode function via structure through the spontaneous folding of the sequence into the three dimensional structure of the protein [ 1 ] . The effects of sequence mutations on function form a landscape that reveals how function constrains sequence . Alterations at some sites in a protein sequence can not be tolerated because they are essential to the protein ’ s function . Other sites evolve together because the structure and function is determined by them collectively . Mutations can enhance the activity of a protein , attenuate it , or leave it unchanged . The functional effect of sequence variations can be measured through deep mutational scanning experiments [ 2 ] . Consisting of thousands to hundreds of thousands of measurements of protein function , deep mutational scans give insight into the intrinsic constraints on a protein ’ s structure and function . Due to the cost and difficulty of implementing such experiments , compilations of deep mutational scanning data include experiments on a few dozens of proteins at most , relative to the tens of thousands of proteins encoded in the human genome , and the millions more across the tree of life that we would like to understand . A model that learns the landscape linking sequence to function can provide insight into function without having to do experiments . Unsupervised models of mutational effects can be learned from sequences [ 3 , 4 ] . Statistical patterns in a family of evolutionarily related protein sequences contain information about structure and function [ 5–7 ] . This is because the properties of a protein act as constraints on the selection of sequences through evolution [ 8 ] . In the natural language modeling community , there has been interest in zero-shot transfer of models to new tasks . Massive language models can solve tasks they haven ’ t been directly trained on [ 9–11 ] . Recently protein language models have achieved state-of-the-art in various structure prediction tasks [ 12–14 ] . Work to date has mainly focused on transfer in the classical representation learning setting , using pre-trained features with supervision on the downstream task . 1Facebook AI Research 2New York University 3UC Berkeley . ESM-1v is available at < https : //github . com/facebookresearch/esm > . Correspondence to : Alexander Rives < arives @ fb.com > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ASKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVT TFSYGVQCFSRYPDHMKRHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIE LKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNT PIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK In this work we show that language models trained on large and diverse protein sequence databases can predict experimental measurements of protein function without further supervision . Prior work has focused on transferring the representations using supervision from experimental data [ 15 , 16 ] . We find that language models can transfer to predict functional measurements without supervision . Language models perform zero-shot and few-shot prediction of mutational effects across a variety of proteins with widely differing functions . We perform experiments with state-of-the-art protein language models ESM-1b [ 12 ] and MSA Transformer [ 13 ] . We introduce a new protein language model , ESM-1v , with zero-shot performance comparable to state-of-the-art mutational effect predictors . Performance can be further improved by fine-tuning the model with sequences from the protein family . Predictions capture the functional landscape of the protein , correlate with amino acid conservation patterns in the core and surface , and identify residues responsible for binding and activity . 2 Zero-shot transfer . Zero-shot learning has classically described the extension of a classifier to a new set of classes that have not been seen in training [ 17 ] . In natural language processing this idea has been extended to describe the transfer of models to entirely new tasks without further training . Proposed as zero-data learning by Larochelle et al . [ 18 ] , this perspective on transfer has been at the center of recent work understanding the generalization capabilities of large language models [ 9–11 , 19 ] . The distinction from representation learning is that the models are used directly without additional supervision for the task . This means that the tasks must be learned purely from pre-training . In this work we take a similar perspective on zero-shot transfer to that of GPT-3 , described in Brown et al . [ 10 ] . We define zero-shot transfer to be transfer of a model to a new task without any further supervision to specialize the model to the task . We also consider the closely related idea of few-shot transfer . Here as in Brown et al . [ 10 ] we define the few-shot setting to be one in which a few positive examples are given to the model as inputs at inference time . As in the zero-shot setting , no gradient updates are performed to specialize the model . Similar to Brown et al . [ 10 ] , the claim is not one of out-of-distribution generalization . The assumption is that in the pre-training stage , the model learns information relevant to the tasks to which it will later be transferred . In the case of protein language models , the pre-training dataset includes sequences from across evolution , which implies the model may see examples of sequences from protein families on which it will be evaluated . The essential departure from the standard approach in computational biology is that the model is general purpose and can be applied across a variety of tasks without specialization . Measurements of function , a property of central importance to the understanding and design of proteins , are a practical ground for studying the generalization capability of protein language models . Deep mutational scanning experiments measure the effects of thousands to hundreds of thousands of mutations on a single protein , and have been performed on a variety of proteins having different functions and using various forms of experimental measurement . We study zero-shot and few-shot transfer of protein language models to function prediction using this data . Supervised methods trained with data from experimental measurements [ 15 , 16 ] , and unsupervised methods trained only on sequences [ 3 , 4 ] have been developed for prediction of mutational effects . Unsupervised mutational effect predictors are trained as task specific models on sequences from an individual protein family . In this view every protein is an independent prediction task where the objective is to score the effect of mutations on the protein ’ s function . While mutational effect predictors trained on multiple sequence alignments ( MSAs ) are typically described as unsupervised , they can also be seen as weakly supervised . Hsu et al . [ 15 ] observe that such models have weak supervision on the task through the MSA , which describes the fitness landscape of the protein through positive examples . If protein language models can learn the information necessary to solve a task from pre-training , then they can be applied directly to new instances of the task , without specialization . This would mean that in practice a single general purpose model can be trained once and then applied to a variety of possible tasks . Thus zero-shot and few-shot transfer represent fundamentally new unsupervised learning capabilities that protein language models can bring to the computational biology toolkit . 3 Method . Protein language models trained with the masked language modeling objective are supervised to output the probability that an amino acid occurs at a position in a protein given the surrounding context . We use this capability to score sequence variations . For a given mutation we can consider the amino acid in the wildtype protein as a reference state , comparing the probability assigned to the mutated amino acid with the probability assigned to the wildtype . We score mutations using the log odds ratio at the mutated position , assuming an additive model when multiple mutations T exist in the same sequence : ∑ t∈T log p ( xt = x mt t |x\T ) − log p ( xt = xwtt |x\T ) ( 1 ) Here the sum is over the mutated positions , and the sequence input to the model is masked at every mutated position . 3.1 Zero-shot and few-shot transfer . In the zero-shot setting , inference is performed directly on the sequence to be evaluated . Since the MSA Transformer can take multiple sequences as input at inference time , we use this model in the few-shot setting , where additional sequences from the protein family are provided along with the sequence to be evaluated . In both the zero-shot and few-shot settings , only forward passes of the models are performed during inference ; no gradient updates are taken . Fig . 2 illustrates the approach in comparison to the current practice of fitting a new model for each task . 3.2 Inference efficiency . Inference with ESM-1v is more efficient than current state-of-the-art methods . This is a result of two important differences : ( i ) the effect of mutations can be inferred directly without training a task-specific model ; ( ii ) fitness landscapes can be predicted with a single forward pass . Time requirements are summarized in Fig . 7 . 3.3 Scoring with MSA Transformer . We score mutations with MSA Transformer using the log odds ratio and additive model in Eq . ( 1 ) . However , since MSA Transformer uses a set of sequences for inference , we input the sequence to be evaluated as the first sequence , and provide additional sequences from the MSA as context . Masking and scoring are performed on the first sequence only . 4 Results . 4.1 Experimental setup . Prediction Models We compare to state-of-the-art unsupervised variant prediction methods , EVMutation [ 4 ] and DeepSequence [ 20 ] . We also examine performance of a variety of protein language models that have been recently introduced in the literature . The position specific scoring matrix ( PSSM ) , EVmutation [ 4 ] , and DeepSequence [ 20 ] methods are all MSA based . The PSSM treats each position in the sequence independently , factorizing the likelihood into one term per sequence position . EVmutation is a Potts model , which adds pairwise terms modeling the interactions between positions . DeepSequence introduces a latent code , allowing potential higher-order interactions between positions . UniRep [ 21 ] , TAPE [ 22 ] , ProtBERT-BFD [ 14 ] , ESM-1b [ 12 ] , and ESM-1v ( introduced here ) , are all single-sequence language models trained on large databases of unaligned and unrelated protein sequences ( e.g . Pfam [ 23 ] or UniRef [ 24 ] ) . With the exception of UniRep , which is trained using next token prediction , all models are trained with masked language modeling [ 25 ] . Finally , the MSA Transformer [ 13 ] is a combination of both approaches ; it is trained on a large database of MSAs using masked language modeling and takes an MSA as input during inference . ESM-1v We train ESM-1v , a 650M parameter transformer language model for prediction of variant effects , on 98 million diverse protein sequences across evolution . The model is trained only on sequences , without any supervision from experimental measurements of function . We use Uniref90 2020-03 [ 24 ] , employing the ESM-1b architecture and masked language modeling approach of Rives et al . [ 12 ] . The model attains a perplexity of 7.29 on a set of held-out Uniref90 sequences ( Table 10 ) . We train five models with different seeds to produce an ensemble . Evaluation Models are evaluated on a set of 41 deep mutational scans collected by Riesselman et al . [ 20 ] , which comprise a variety of tasks assessing a diverse set of proteins . Across tasks , the experiments differ in the functions tested and in the measurements performed . We treat each deep mutational scanning dataset as a separate prediction task , scoring each of the variants in the dataset with the model . The tasks are split into a validation set of ten mutational scanning datasets and a test set consisting of the remaining datasets . We evaluate performance by comparing the scores with the experimental measurements using Spearman rank correlation . Comparisons Since the published versions of EVMutation and DeepSequence use MSAs generated from an earlier version of Uniref100 , we generate new MSAs using EVMutation methodology and the version of Uniref100 concurrent with our pretraining dataset . We train replications of EVMutation and DeepSequence using their open source code . The same MSAs are also used in few-shot experiments with MSA Transformer and unsupervised fine-tuning experiments with ESM-1v .

Summary: The paper adds a follow-up experiment to a recent paper demonstrating the power of large language models (particularly, BERT and the MSA transformer) for a variety of protein structure and function prediction tasks. The new task (zero-shot scoring of the effects of mutations) has a variety of useful applications for clinical medicine and protein design and has been explored extensively in recent papers that used alternative models. This paper demonstrates that the these new models perform similarly, or perhaps slightly better, on a standard suite of datasets.