Language models enable zero-shot prediction of the effects of mutations on protein function

1 Introduction . Proteins have a myriad of diverse functions that underlie the complexity of life . Protein sequences encode function via structure through the spontaneous folding of the sequence into the three dimensional structure of the protein [ 1 ] . The effects of sequence mutations on function form a landscape that reveals how function constrains sequence . Alterations at some sites in a protein sequence can not be tolerated because they are essential to the protein ’ s function . Other sites evolve together because the structure and function is determined by them collectively . Mutations can enhance the activity of a protein , attenuate it , or leave it unchanged . The functional effect of sequence variations can be measured through deep mutational scanning experiments [ 2 ] . Consisting of thousands to hundreds of thousands of measurements of protein function , deep mutational scans give insight into the intrinsic constraints on a protein ’ s structure and function . Due to the cost and difficulty of implementing such experiments , compilations of deep mutational scanning data include experiments on a few dozens of proteins at most , relative to the tens of thousands of proteins encoded in the human genome , and the millions more across the tree of life that we would like to understand . A model that learns the landscape linking sequence to function can provide insight into function without having to do experiments . Unsupervised models of mutational effects can be learned from sequences [ 3 , 4 ] . Statistical patterns in a family of evolutionarily related protein sequences contain information about structure and function [ 5–7 ] . This is because the properties of a protein act as constraints on the selection of sequences through evolution [ 8 ] . In the natural language modeling community , there has been interest in zero-shot transfer of models to new tasks . Massive language models can solve tasks they haven ’ t been directly trained on [ 9–11 ] . Recently protein language models have achieved state-of-the-art in various structure prediction tasks [ 12–14 ] . Work to date has mainly focused on transfer in the classical representation learning setting , using pre-trained features with supervision on the downstream task . 1Facebook AI Research 2New York University 3UC Berkeley . ESM-1v is available at < https : //github . com/facebookresearch/esm > . Correspondence to : Alexander Rives < arives @ fb.com > . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ASKGEELFTGVVPILVELDGDVNGHKFSVSGEGEGDATYGKLTLKFICTTGKLPVPWPTLVT TFSYGVQCFSRYPDHMKRHDFFKSAMPEGYVQERTIFFKDDGNYKTRAEVKFEGDTLVNRIE LKGIDFKEDGNILGHKLEYNYNSHNVYIMADKQKNGIKVNFKIRHNIEDGSVQLADHYQQNT PIGDGPVLLPDNHYLSTQSALSKDPNEKRDHMVLLEFVTAAGITHGMDELYK In this work we show that language models trained on large and diverse protein sequence databases can predict experimental measurements of protein function without further supervision . Prior work has focused on transferring the representations using supervision from experimental data [ 15 , 16 ] . We find that language models can transfer to predict functional measurements without supervision . Language models perform zero-shot and few-shot prediction of mutational effects across a variety of proteins with widely differing functions . We perform experiments with state-of-the-art protein language models ESM-1b [ 12 ] and MSA Transformer [ 13 ] . We introduce a new protein language model , ESM-1v , with zero-shot performance comparable to state-of-the-art mutational effect predictors . Performance can be further improved by fine-tuning the model with sequences from the protein family . Predictions capture the functional landscape of the protein , correlate with amino acid conservation patterns in the core and surface , and identify residues responsible for binding and activity . 2 Zero-shot transfer . Zero-shot learning has classically described the extension of a classifier to a new set of classes that have not been seen in training [ 17 ] . In natural language processing this idea has been extended to describe the transfer of models to entirely new tasks without further training . Proposed as zero-data learning by Larochelle et al . [ 18 ] , this perspective on transfer has been at the center of recent work understanding the generalization capabilities of large language models [ 9–11 , 19 ] . The distinction from representation learning is that the models are used directly without additional supervision for the task . This means that the tasks must be learned purely from pre-training . In this work we take a similar perspective on zero-shot transfer to that of GPT-3 , described in Brown et al . [ 10 ] . We define zero-shot transfer to be transfer of a model to a new task without any further supervision to specialize the model to the task . We also consider the closely related idea of few-shot transfer . Here as in Brown et al . [ 10 ] we define the few-shot setting to be one in which a few positive examples are given to the model as inputs at inference time . As in the zero-shot setting , no gradient updates are performed to specialize the model . Similar to Brown et al . [ 10 ] , the claim is not one of out-of-distribution generalization . The assumption is that in the pre-training stage , the model learns information relevant to the tasks to which it will later be transferred . In the case of protein language models , the pre-training dataset includes sequences from across evolution , which implies the model may see examples of sequences from protein families on which it will be evaluated . The essential departure from the standard approach in computational biology is that the model is general purpose and can be applied across a variety of tasks without specialization . Measurements of function , a property of central importance to the understanding and design of proteins , are a practical ground for studying the generalization capability of protein language models . Deep mutational scanning experiments measure the effects of thousands to hundreds of thousands of mutations on a single protein , and have been performed on a variety of proteins having different functions and using various forms of experimental measurement . We study zero-shot and few-shot transfer of protein language models to function prediction using this data . Supervised methods trained with data from experimental measurements [ 15 , 16 ] , and unsupervised methods trained only on sequences [ 3 , 4 ] have been developed for prediction of mutational effects . Unsupervised mutational effect predictors are trained as task specific models on sequences from an individual protein family . In this view every protein is an independent prediction task where the objective is to score the effect of mutations on the protein ’ s function . While mutational effect predictors trained on multiple sequence alignments ( MSAs ) are typically described as unsupervised , they can also be seen as weakly supervised . Hsu et al . [ 15 ] observe that such models have weak supervision on the task through the MSA , which describes the fitness landscape of the protein through positive examples . If protein language models can learn the information necessary to solve a task from pre-training , then they can be applied directly to new instances of the task , without specialization . This would mean that in practice a single general purpose model can be trained once and then applied to a variety of possible tasks . Thus zero-shot and few-shot transfer represent fundamentally new unsupervised learning capabilities that protein language models can bring to the computational biology toolkit . 3 Method . Protein language models trained with the masked language modeling objective are supervised to output the probability that an amino acid occurs at a position in a protein given the surrounding context . We use this capability to score sequence variations . For a given mutation we can consider the amino acid in the wildtype protein as a reference state , comparing the probability assigned to the mutated amino acid with the probability assigned to the wildtype . We score mutations using the log odds ratio at the mutated position , assuming an additive model when multiple mutations T exist in the same sequence : ∑ t∈T log p ( xt = x mt t |x\T ) − log p ( xt = xwtt |x\T ) ( 1 ) Here the sum is over the mutated positions , and the sequence input to the model is masked at every mutated position . 3.1 Zero-shot and few-shot transfer . In the zero-shot setting , inference is performed directly on the sequence to be evaluated . Since the MSA Transformer can take multiple sequences as input at inference time , we use this model in the few-shot setting , where additional sequences from the protein family are provided along with the sequence to be evaluated . In both the zero-shot and few-shot settings , only forward passes of the models are performed during inference ; no gradient updates are taken . Fig . 2 illustrates the approach in comparison to the current practice of fitting a new model for each task . 3.2 Inference efficiency . Inference with ESM-1v is more efficient than current state-of-the-art methods . This is a result of two important differences : ( i ) the effect of mutations can be inferred directly without training a task-specific model ; ( ii ) fitness landscapes can be predicted with a single forward pass . Time requirements are summarized in Fig . 7 . 3.3 Scoring with MSA Transformer . We score mutations with MSA Transformer using the log odds ratio and additive model in Eq . ( 1 ) . However , since MSA Transformer uses a set of sequences for inference , we input the sequence to be evaluated as the first sequence , and provide additional sequences from the MSA as context . Masking and scoring are performed on the first sequence only . 4 Results . 4.1 Experimental setup . Prediction Models We compare to state-of-the-art unsupervised variant prediction methods , EVMutation [ 4 ] and DeepSequence [ 20 ] . We also examine performance of a variety of protein language models that have been recently introduced in the literature . The position specific scoring matrix ( PSSM ) , EVmutation [ 4 ] , and DeepSequence [ 20 ] methods are all MSA based . The PSSM treats each position in the sequence independently , factorizing the likelihood into one term per sequence position . EVmutation is a Potts model , which adds pairwise terms modeling the interactions between positions . DeepSequence introduces a latent code , allowing potential higher-order interactions between positions . UniRep [ 21 ] , TAPE [ 22 ] , ProtBERT-BFD [ 14 ] , ESM-1b [ 12 ] , and ESM-1v ( introduced here ) , are all single-sequence language models trained on large databases of unaligned and unrelated protein sequences ( e.g . Pfam [ 23 ] or UniRef [ 24 ] ) . With the exception of UniRep , which is trained using next token prediction , all models are trained with masked language modeling [ 25 ] . Finally , the MSA Transformer [ 13 ] is a combination of both approaches ; it is trained on a large database of MSAs using masked language modeling and takes an MSA as input during inference . ESM-1v We train ESM-1v , a 650M parameter transformer language model for prediction of variant effects , on 98 million diverse protein sequences across evolution . The model is trained only on sequences , without any supervision from experimental measurements of function . We use Uniref90 2020-03 [ 24 ] , employing the ESM-1b architecture and masked language modeling approach of Rives et al . [ 12 ] . The model attains a perplexity of 7.29 on a set of held-out Uniref90 sequences ( Table 10 ) . We train five models with different seeds to produce an ensemble . Evaluation Models are evaluated on a set of 41 deep mutational scans collected by Riesselman et al . [ 20 ] , which comprise a variety of tasks assessing a diverse set of proteins . Across tasks , the experiments differ in the functions tested and in the measurements performed . We treat each deep mutational scanning dataset as a separate prediction task , scoring each of the variants in the dataset with the model . The tasks are split into a validation set of ten mutational scanning datasets and a test set consisting of the remaining datasets . We evaluate performance by comparing the scores with the experimental measurements using Spearman rank correlation . Comparisons Since the published versions of EVMutation and DeepSequence use MSAs generated from an earlier version of Uniref100 , we generate new MSAs using EVMutation methodology and the version of Uniref100 concurrent with our pretraining dataset . We train replications of EVMutation and DeepSequence using their open source code . The same MSAs are also used in few-shot experiments with MSA Transformer and unsupervised fine-tuning experiments with ESM-1v .

Summary: This paper demonstrates utilizing a pretrained masked language model to predict a ranking of functional activity of a protein.  They focus on data from deep mutational scanning, which measures the functionality of a large set of individual mutations to a protein in order to characterize the fitness landscape around the protein (wild type) for the desired functionality.  It proposes a method which uses the log odds ratio between a mutated variant and the wild type protein as a proxy for the fitness of the mutation. It evaluates the methods by using Spearman correlation between its ranking and ground truth fitness from deep mutational scanning datasets.   It evaluates a wide set of existing baselines, and focuses analysis on an updated version of the ESM-1b model, ESM-1v. It compares several methods for evaluating the log odds of a particular mutation.