Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper demonstrates utilizing a pretrained masked language model to predict a ranking of functional activity of a protein.

They focus on data from deep mutational scanning, which measures the functionality of a large set of individual mutations to a protein in order to characterize the fitness landscape around the protein (wild type) for the desired functionality.

It proposes a method which uses the log odds ratio between a mutated variant and the wild type protein as a proxy for the fitness of the mutation. It evaluates the methods by using Spearman correlation between its ranking and ground truth fitness from deep mutational scanning datasets. 

It evaluates a wide set of existing baselines, and focuses analysis on an updated version of the ESM-1b model, ESM-1v. It compares several methods for evaluating the log odds of a particular mutation.

main_review: 
I am primarily evaluating the method from the perspective of a machine learning researcher who has done related work applying self-supervised learning methods to protein data.

Overall, the method is well motivated, simple, effective, and likely to lead to fruitful future research. The method is carefully evaluated against several baselines.

**Originality**: The datasets are standard benchmarks for evaluating protein function prediction. The models are variants on standard BERT models, with evaluations to select the right pretraining dataset and compare model sizes.

It demonstrates a simple new method on top of this established model to accomplish a core task in protein design (characterizing function of mutations to a protein) with minimal overhead.

It performs about as well as methods that use a full MSA while only requiring a single sequence.

**Quality**: The paper is technically sound and carefully evaluated. It answered questions I had while reading it with experiments (and a thorough appendix).



* Proper validation sets are used
* Baselines are well tuned (even outperforming results in source publication)
* results are reported using sound statistics
* The ESM-1v model is evaluated at several different scales and across many pre-training datasets
* Several alternatives are considered for the primary choices made in the paper (e.g. how to score the effects of mutations given the model, which pre-training dataset to use, how to 
* Good discussion of related work, in both ML + protein and NLP literature.
* Extends the model analysis beyond the task at hand to how the models do at modeling the stability of residues located in different regions of the folded protein.

**Clarity:** The paper is clearly written, with sufficient references to related work and background. Figures clarify content of the text (such as the overall task, the pipeline for each baseline and the proposed method, and visualizations of relationship between position and model entropy).

I believe a practitioner familiar with the data and BERT models could reproduce the results as described.

**Significance:** I believe other researchers are likely to build on the methods outlined in the paper. While it does not always exceed the baselines, the method generally performs near or better than existing methods despite only needing a single protein sequence (instead of a full MSA). It is also the first paper I have seen that explicitly evaluates scaling on protein datasets (which is why I would like to see more discussion of this point).

**Feedback for improvement**

**– content –**



* Consider saying "masked language models" in the title to point at the specific model class used
* line 210: What methods are appropriate for handing Methionine in the first position ("care must be used when applying the model to a subsequence")
* Model
    * Any theories as to why uniref90 performs the best?
    * Does the pretraining only use the cluster centers from each cluster? Consider adding a section to appendix expanding on the pretraining details, even if it is explained in a referenced paper
    * How large was the dataset? Roughly how many epochs was 170k steps of 1 million tokens
    * figure 9: Add  figure or table including other pretraining metrics alongside just the plot of rho with respect to model size/update steps. What is the per amino acid accuracy? Perplexity? table 12 has some of this for the large model, but the other model information is also useful.
        * Generally suggest trying to characterize relationship between pretraining performance and downstream performance on your task as measured by rho.
* I'm sure I've seen similar methods to "spiked unsupervised fine-tuning" used in NLP literature to regularize/combat forgetting, but I'm struggling to find a reference… I will see if I can find something for the discussion period.
* Equation at line 92:
    * Define mt=mutated, wt=wild type before equation
    * should the conditioning be x^{mt}_{-t} and x^{wt}_{-t}?
* 96/104: consider describing more about what fitness landscape means (fitness of the region around the protein in sequence space).
* 526: The #s don't add up:"among the 15% predicted positions, 10% are randomly, 10% retain …"

**– style –**



* 54: add comma after "in biology"
* Define MSA when it is first used (79)
* Line 94: You do not describe what ESM-1v before referring to it
* table 7: misspelled marginal
* figure 9
    * label the y axis
    * The bottom of the figure is cut off
* table 1: Add reference to appendix section where you describe sampling for the MSA
limitations_and_societal_impact: The paper's appendix discussion focuses on compute costs.  While far off, it is worth discussing potential applications of this technology (good and bad), even if just referencing existing writing on potential dangers of protein engineering.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper describes a model that
- Is pretrained jointly across many protein families
- Learns functional effects of mutations at state-of-the-art for deep mutational scans
- Recapitulates substitution patterns concordant with biology
- Does not require MSAs at all
main_review: This paper solves the problem of functional mutation effect prediction, and does it as well as anyone else, with a very simple setup. They demonstrate these results convincingly. My main concern with this paper is that they achieve these results in a zero-shot manner, but don’t demonstrate that fine-tuning improves performance, which is very surprising. That said, I do not think that they need to completely answer the question as to why in order for this to be a complete body of work.

Originality: Not very original methods. Use of current dataset.
Quality: Good quality. Lots of qualitative analysis. Lots of baselines. Baselines were tuned beyond my expectation.
Clarity: clear. Good prose.
Significance: Zero-shot performance mirroring state-of-the-art in an area that has substantial impact, like pharmaceuticals, is very significant.

See below for more specific details.

-----------------------------------------------

The paper describes a model that
- Is pretrained jointly across many protein families
- Learns functional effects of mutations at state-of-the-art for deep mutational scans
- Recapitulates substitution patterns concordant with biology
- Does not require MSAs at all

The evidence they provide is
- _Is pretrained jointly across many protein families_
  - Use UniRefXX for many values of XX
- _Learns functional effects of mutations_
  - Quantitative evidence via deep mutational scans (DMS)
    - 41 DMS examined
    - Spearman R used - SOTA values
    - Comparison to DeepSequence, ESM-1b, ProtBERT-BFD, UniRep, TAPE
  - Show entropy vs ground truth PSSMs
  - Show calibration of the model
- _Recapitulates substitution patterns concordant with biology_
  - 41 DMS examined
  - Explanations of substitution patterns like hydrophobicity and buried vs exposed via some anecdotes and statistics
- _Does not require MSAs at all_
  - Via an explanation of their architecture, training procedure, and inference setup

The benefits of their approach vs some SOTA are
- No fine tuning step needed
- No family-based training objective function
- Not trained for-purpose - should be generally useful.

limitations_and_societal_impact: The authors have sufficiently described potential negative societal impact.

Re: limitations: Questions for author:
- Why is it that fine tuning ESM-1v didn’t statistically improve performance across the 41 DMS tasks via t test?
  - What sort of level of improvement would be needed for this?
  - Are there statistically significant claims you can make about the fine-tuning performance?
  - Why was there such a big jump from fine tuning for some datasets but not others?
  - There are still some tasks where fine-tuned performance is statistically lower than DeepSequence. Do you have any explanations for this?
- Is there a baseline you can compare to for calibration error? It’s a little difficult to understand whether ACE of 0.006 is good. How difficult is it to create a model that does this well?
- What is the p value on pearsonr for PSSM entropy and predicted entropy?
- Saying pearsonr of .44 on the entropy claim is well correlated is slightly hyperbolic - I suggest wording.
- Is the main claim that this is faster than current methods? Or faster to train? Fewer parameters? If so, a comparison to DeepSequence in this would be helpful. 
  - Perhaps this is already in the literature (transitively?) and I don’t know about it, in which case a reminder in the text would be useful.
- A connection between calibration and why it’s important for downstream tasks would help the paper’s sales pitch.
- Figure 6: Average time to compute a DMS is a slightly odd statistic, as this measure is quite dependent on somewhat extraneous factors like the assays used in the experiments. I suggest normalizing a different way, e.g. per sequence or per residue.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper analyses the generalization capacity of the protein language models. The experiments and analysis suggest that zero-shot inference of protein language models can capture the functional effects of sequence variation. The paper is essentially analyzing the protein language models on their generalization capacity of zero-shot transfer learning for the Mutation Effect Prediction task. 


main_review: The paper analyses the recent advancements of protein language models (PLM) on a specific task to check the generalization capacity of a PLM. However, the paper does not discuss the robustness of the zero-shot predictions. An error analysis will be really useful for understanding the true generalization capacity of the models.

There is no methodological novelty in the approach. It was also difficult to read the paper as a lot of reference to Appendix and supplementary materials.

limitations_and_societal_impact: The deep neural network-based language models are good at predictions. However, they are prone to adversarial attacks, meaning that changing input slightly can change the predictions. Inference based on such a model can be problematic for a sensitive task like designing protein in the current state of the models.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Unsurprising results and the method is not novel.
main_review: Modeling the impact of amino acid mutations on protein-protein interaction plays a crucial role in protein engineering and drug design. But the structures of different families of proteins vary a lot, hindering the generalization of a machine learning model. This paper found that, without any supervision from experimental data or additional training, protein language models capture the functional effects of sequence variation, achieving state-of-the-art zero-shot prediction performance.

However, it is somewhat expected. In principle, language models learn the distribution of the protein sequences that are stable. The unstable sequences are usually recognized by language models as low probabilities of existence.
 
The introduction of the used prediction models is missing. It is hard to understand the acronyms without explanation.

The implementation details are not clear. What are the hyperparameters and how do you finetune them?

Have the authors compared the deep learning models with the traditional evolutionary conservation profiles?

limitations_and_societal_impact: 
The language models are not proposed by this paper. It is not clear what is the contributions of this paper except for the conclusion that language models can partially predict the protein function.

We also expect to see if language models can provide some interesting biological insights.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper adds a follow-up experiment to a recent paper demonstrating the power of large language models (particularly, BERT and the MSA transformer) for a variety of protein structure and function prediction tasks. The new task (zero-shot scoring of the effects of mutations) has a variety of useful applications for clinical medicine and protein design and has been explored extensively in recent papers that used alternative models. This paper demonstrates that the these new models perform similarly, or perhaps slightly better, on a standard suite of datasets.




main_review: **originality**

My primary negative critique of this paper is its lack of novelty.  

It has been previously established in the literature by many papers that the functional effect of mutations can be predicted in a zero-shot setting by scoring variants using the likelihood of a generative model. Further, the two models considered in this paper have been introduced in recent prior work. Those papers are really great; I found them rigorous and well written and would have advocated for their acceptance to NeurIPS. I feel that the scale of contributions of the current paper is smaller than that of a typical NeurIPS paper, though, and its content would be better placed as an additional section of experiments in, for example, the MSA transformer paper.

One argument for a paradigm shift due to your approach is that practitioners don't have to train a new model for a protein family of interest and don't have to perform multiple sequence alignment of homologous sequences. Instead, the user can just do forward passes in an existing model. However, it's rare for practitioners to be considering a large number of families at once. See further comments below.

**clarity/quality**
The paper is well written and cites relevant recent work. The experimental results could have been analyzed in more detail. See comments below.

**significance**
This paper is part of a really important line of work on deep learning + proteins. My hesitation is around whether this paper should appear as a stand-alone paper at NeurIPS, not whether the results are of interest to the community.

**zero-shot prediction**

I disagree with your definition of zero-shot in the paragraph starting on line 75. To me, any method is zero-shot if it doesn't rely on experimental data as input. Why is a method that relies on querying a sequence database to retrieve homologous sequences qualitatively different than your approach?  Doesn't the training set for ESM-1v or MSATransformer include the same 'weak supervision' that the traditional approach relies on?

I suppose you could argue that your method could be applied to predict the functional effect of mutations on sequences that are very different than anything that appears in nature, where there is no associated protein family that could be retrieved by a database search. This is zero-shot in a different sense and not evaluated in the experiments.

**computing the likelihood of a variant**

In Eq (1) uses P(x_t | x_{-t}), but ignores P(x_{-t} | x_t). Why is this a reasonable assumption? Surely, when making a mutation at a particular position, it effects the likelihood of the values at the other positions too. Note that prior models for this task (VAEs, autoregressive models, Potts models) don't make such a simplifying assumption. I see that you discuss alternatives in Appendix A. Why do you think this simple approach is sufficient?

 **Epistasis**
Following up on the previous point, does this mean that you can't capture epistasis, which is known to be prevalent in fitness landscapes? For datasets with multiple mutations, you can compute the subset of examples that are epistatic (e.g., it the variant is non-functional, but combines mutations that led to a functional variant when used individually). On these, how do you compare to DeepSequence, which can capture epistasis?

**ESM-1b vs. ESM-1v**
Am I correct in understanding that ESM-1v is the same as ESM-1b, but trained on slightly different data? When I discuss originality above, I argue that the paper does not introduce a new model. To me, changing the training data does not amount to introducing a new model with the degree of originality characteristic of a stand-alone paper.


**Inference speed**
I don't feel that this is a particularly important distinction between the proposed model and prior work. It's not clear what an application setting would be where practitioners need to make predictions for many protein families at high throughput. If the goal is just to predict  the functional effects of mutations on a single parent sequence, then the speed difference across techniques will be minor.  The goal of this zero-shot prediction is to avoid needing to run wet-lab experiments, so any approach that can run on a computer in less than an hour is a massive speedup vs. the alternative.


 **Evaluation Metrics**
Can you please explain why spearman correlation is a reasonable performance metric? This is noisy experimental data. What do you think is the upper limit on performance? If you binarized the experiment measurement (e.g. whether a variants is worse than the wildtype or better), what would detection metrics like AUC or precision@k look like? I'm also assuming that such labels would be very imbalanced (with more than half of the mutations leading to a decrease in function). Doesn't that bias spearman rho, and make it less predictive of performance for protein design applications where the goal is to increase fitness?

**Break-down of performance**

I would have appreciated more analysis of where the method succeeds and fails. What characterizes datasets where your methods outperform/underperform prior work? For example, how do you compare on multi-mutant variants?

**Sec 5**
Again, this feels like a section that should appear in a long-form article on ESM or MSA Transformer. Are these results good compared to baseline methods? How does a PSSM perform on them? Is the goal to show that you are outperforming a baseline, or are you just doing some sanity checks for the model?


*****Updated review after discussion period*****
I have decided to raise my review to a weak accept, after a lengthy discussion period with the authors and between the reviewers. Overall, there are a number of detailed choices that were important to achieve the paper's results, and the community should know about these. 

My principal hesitation is not around the merits of the paper's contribution, which is valuable, but in how the contribution is framed. If the paper is accepted, I would really appreciate it if you considerably changed the exposition (including the title) to be more precise about what's new in the paper. Specifically, I'd refer to your work and prior work (e.g., DeepSequence) both as zero-shot. 

The title 'Language models enable zero-shot prediction of the effects of mutations on protein function' suggests that there is a new capability that they have 'enabled.' This is certainly not the case. Prior work uses exactly the same data (naturally-occurring sequences, but no experimentally-labeled data) and achieves comparable results. When I first started working on ML for proteins, I was enthralled when I learned that both protein structure and the functional effects of mutations could be predicted by inspecting a statistical model of homologous sequences. This is a remarkable result, and I'm concerned that the twitter-verse will mistakenly think that this particular paper unlocked it (it's actually been known for many years). 



limitations_and_societal_impact: Overall yes, however this statement should be revised: " Otherwise,basic methods in computational biology have a long time horizon to concrete social impact." Just because something has a long time horizon doesn't mean that it is benign.
review_title: null

Meta-review: While evaluating this paper, the reviewers had an extensive discussion about the relative strengths, and in particular what counts as "novelty".  While the reviewers did not come to unanimous consensus here, I am swayed by the majority of reviewers who cited the strong empirical rigor in analyzing the behavior of the proposed models, in addition to the careful combination of many small improvements that lead to strong results on an important application area in predicting the functional effects of protein mutations.  Indeed, my own view is that the term "novelty" is often overly constrained to mean "a brand new modeling technique" when instead it should be interpreted to mean "something important that adds to the field's overall understanding and knowledge".  Indeed, the recent ML literature is filled with important papers that usefully shine new light on previously known modeling techniques or methods.

Given the strong reviews from the majority of reviewers, and the acknowledgement of the strong empirical results and careful explication of the rigorous evaluations, I am happy to recommend acceptance of this paper.  I do expect that the authors will use the extensive reviewer feedback to further revise and strengthen the paper for its final form.
Decision: Accept (Poster)