Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper investigates the PLM module in AlphaFold, Evoformer. In particular, it looks at assessing its representational ability beyond structure prediction, which is already well-established through AlphaFold. The paper does this by comparing the Evoformer module with two other popular PLMs: ESM-1b, MSA-Transformer in a number of tasks related to predicting protein functions. Particularly, the paper looks at: 

- (1) Can the Evoformer module in AlphaFold produce representations helpful in predicting protein function?
- (2) Can Evoformer replace ESM-1b and MSA-Transformer for protein prediction functions?
- (3) How much do evolution-based PLMs rely on MSA input? Can evolution-free PLMs assist with MSA construction?

The paper compares metrics of models on a number of structure prediction, function classification and fitness prediction tasks. The paper concludes that:

- (1) Evoformer encodes protein functional properties, and can be used in predicting protein functions, in particular the stability prediction task;
- (2) Evoformer is not yet  able to substitute ESM-1b and MSA-Transformer models for predicting protein functions;  
- (3) MSA data is crucial input for evolution-based PLMs, for both structure and function prediction tasks. Evolution-free ESM-1b can generate accurate MSAs for evolution-based PLMS.

strengths_and_weaknesses: - This is a really well-written paper presenting interesting results that can be valuable for the field of PLMs and structural biology.
I believe the research questions being asked are valuable: assessing the representational power of Evoformer beyond structure prediction is interesting
- Follow-up work on the research questions started here could lead to insights around dynamics between structural and functional protein functions
- The paper is sound and has great presentation - is very clearly written and does a good job at outlining the research questions, experiments ran and clearly shows how the results support the conclusions that are drawn
- The results themselves are valuable: in particular; I find it interesting that Evoformer performed so poorly on the zero-shot fitness classification task, compared to ESM-1b and MSA-Transformer models and that Evoformer does not consistently outperform the ESM-1b and MSA-Transformer models of predicting protein function tasks, as one might expect, given its huge success on structure prediction tasks
- The conclusion drawn based on zero-shot fitness prediction tasks that ‘the AlphaFold-triggered revolution for structure prediction cannot be directly transferred to function predictions’ might be too strong based on the experiments ran. As mentioned under ‘Limitations’, it’s possible that more complex model architectures could lead to better results. I suggest addressing the low performance
- I appreciated the paper including their limitations, such as how more complex model architectures could lead to better results
questions: - Elaborating on the concern of using the SCOPe dataset (Lines 116-117)
- The low performance of Evoformer on the zero-shot fitness prediction task is interesting, particularly in comparison to the other models. The paper discusses this a bit in lines 215-220. It seems that the paper suggests that one potential reason for this is the difference in training between MSA-Transformer and ESM-1b and Evoformer. While not trained solely on masked token reconstruction, the masked token reconstruction objective is not missing from Evoformer. It would be helpful to speculate a bit more on why the difference would be so large here - is the paper suggesting that training on supervised protein structure tasks might degrade zero-shot performance on fitness prediction tasks?

Minor Miscellaneous Suggestions
- Line 157: ‘meaningful’, not ‘meanful’?
- Line 160: ‘By contrast, it does not perform as accurately as the contact prediction task for SS prediction’ -> I found the wording confusing; consider re-wording
- Line 189: ‘.. better structure PLMs do not mean they have a better representation for predicting function‘ -> also confusing, consider re-wording
- Table 5: Consider re-wording ‘Scratch’ to ‘Random Initialization’
- Consider Higher resolution for Figure 4


limitations: - Authors have adequately addressed the limitations of the work under 'Conclusion and Limitations'
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the representations of the AlphaFold Evoformer across protein structure and function prediction tasks and compares them to protein language models. The paper finds that while Evoformer produces state of the art predictions of structure, it falls behind protein language models on function prediction.
strengths_and_weaknesses: EDIT: based on revisions by the authors during the discussion period and the addition of a held out test set for contact prediction I am raising my evaluation. However the concern about training set holdout also applies to the the other structure prediction result in the paper (secondary structure) and I suggest the authors address this using a similar held out set of structures whether it is accepted at Neurips or whether they resubmit to another venue.

This paper has several important contributions:
* Finding that protein language models perform better than AlphaFold on function prediction tasks is unexpected and interesting
* Studies the dependence of MSA based models on the additional sequences
* Investigates the use of MSAs produced by language model embeddings and finds that retrieval using embeddings is comparable to HMMs

Overall the findings and the investigations are of interest to the Neurips research community and I would very much like to recommend this paper for acceptance. However there is a major methodological concern that undermines some of the paper's main conclusions. In short the assessment of contact prediction is flawed as the test sets used are not held out for AlphaFold training. A number of conclusions in the paper depend on this comparison. Held out test sets, e.g. CASP14 and CAMEO proteins after the CASP14 cutoff date are available.

There are also some problems with the related work section. E.g. in the sentence "PLMs trained on these databases have been successfully applied for various protein related tasks" Rives et al 2021 didn't look at annotation prediction (the authors should cite Bileschi et al 2022 for this), but did look at secondary structure prediction, contact prediction, remote homology detection, and mutational effects. Heinzinger et al 2019. also looked at secondary structure prediction, as did Elnaggar et al 2020. Alley et al 2019, Meier et al 2021, and Hie et al 2021 also looked at fitness prediction. When discussing early work Rives et al 2021 and Heinzinger et al 2019 which were concurrent with Alley et al 2019 should be mentioned. The citation for ESM1b and recovery of contacts cites Vig et al. rather than Rives et al 2021 which introduced ESM1b and studied recovery of contacts. Rao et al 2021 and Vig et al 2021 followed this work up with discovery of the interpretability of attention maps as contacts and should also be cited for this.
questions: The authors may consider more strongly highlighting the use of learned linear projection heads as this is critical for interpretation of the results; the authors may consider summarizing their findings in the abstract.
limitations: The authors have included a thoughtful discussion of the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper studies the effectiveness of evolution-based neural architectures including  Evoformer, MSA-Transformer, and evolution-free methods like ESM in protein structure and property prediction problems. Evoformer is similar to MSA-Transformer which learns protein representation from MSA, but Evoformer was used as part of the Alphafold for 3D representation prediction. While ESM is pretrained using only sequence information without the need for MSAs. 
The authors examined these models in different standard structure predictions and protein properties prediction tasks. Their conclusion was that evolution-based neural architectures including  Evoformer, and MSA-Transformer outperform evolution-free methods like ESM in structure prediction tasks while Evoformer pretrained with much smaller data is bad at properties prediction tasks and zero-shot mutation prediction tasks.
The authors also study the effect of varying MSA quality on the results of the evolution-based neural architectures and proposed an approach that use ESM  for constructing ESM-MSA that is much faster than the standard MSA search but still achieves comparable results to the methods that use MSAs.
strengths_and_weaknesses: Strengths
This is a nice work that performs very careful empirical studies, well-written and accompanied with source code and two new standard benchmark datasets, this is a plus as the standard 5 benchmark datasets in TAPE are not enough for evaluating recent development in the field.

All methods were compared using the same linear layers for downstream tasks, even the authors see this is as a weakness as stated in the conclusion because for different downstream tasks better than linear layers should be considered to achieve the SOTA results but for comparisons of pretrained models, it is fair to use the same linear layers for all the tasks.

Some of the findings are interesting, for example, the results of Evolformer on the zero-shot mutations.

Weaknesses  

It is important to note that Evoformer, MSA-Transformer, and ESM-1b are very different in network size, and different training data used. It is very hard to compare these pre-trained models and draw some conclusions on whether one method is better than another. I see that the authors have mentioned this issue in the introduction section and I agree that it is useful to check which pretrained models are better because most companies and researchers from academia do not have the computing resource to pretrain these large models. Yet the answer to that question only has a practical meaning which might help industries choose the right model for their application but not give an answer to the question of which methods are better (assuming they are the same size and pretrained on the same data).

This is an empirical study so, besides interesting benchmark results, its technical and novelty contribution are limited. 
I evaluated the paper based on the technical depth, novelty contribution, and usefulness of the research to the community. 
questions: For completeness, I think the experiments should include all benchmark data in TAPE, for example, homology prediction is missing.  
Why don't you use the TAPE train/dev/split for contact map and ss prediction instead of SCOPe?
Table 2: missing information about dev set.

limitations: YES
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors divide the sequence alignment representation module into two types: revolution-based and revolution-free, and examine the usefulness of the evlutionary information not only in the framework of protein structure prediction but also other prediction problems, related with proteins. The "revolution-based" approach is confined to "Evoformer" in this paper.

strengths_and_weaknesses: Strengths
-- The paper addresses the problem of examining if evoluationry information (or Evoformer) is more useful / important than multiple sequence alignment without evolutionary information. This problem would be important and also attractive from a bioinformatics (or biology) viewpoint.
-- Rather clear conclustions were lead by empirical studies. Particularly, it is indeed interesting that Evoformer does not necessarily work well for function prediction, comparing with 

Weaknesses
-- Although the information is important, the problem might be minor.
-- The work is totally empirical. That is, the conclusions are obtained totally empirically. So although conclusions might be useful, the result of this work might not be universal. I think this is one of the points already mentioned by the authors in Conclusion.
questions: -- Seeing the three models (a, b and c) in Fig. 1, can the models be compared in more detail? For example, can the authors consider more valiation of using the information?
-- Also how much can the performance be affected by changing the values of hyperparameters?

limitations: -- This work is significant from a ablation study viewpoint in bioinformatics. We might see what information can contribute to improve protein-related bionformatic problems, such as secondary structure or function prediction. However, methodological contribution would be definintely limited. One better presentation might be to do experiments more: for the same problem and same module, different ways of using the module are examined and the best way of using that is shown. However, I am unsure if this way could be good enough to be accepted for publicaiton by this conference, though.
review_title: null

Meta-review: metareview: The paper provides an empirical study of the representation learning component of Alphafold for protein sequences, called Evoformer. The topic of protein language models (PLMs), of which Evoformer is an example, is relevant and interesting to the machine learning community. This is due to the success of Alphafold in protein structure prediction, as well as the popularity of large language models in natural language.

Four reviewers have carefully considered the results, and there was a very constructive interaction between the authors and three reviewers during the discussion phase. Thank you to both the authors and reviewers for engaging in good faith. All reviewers unanimously agreed that the paper is well written and the method reproducible (modulo compute). The revised paper provides useful empirical results to guide future researchers interested in representing protein sequences for different possible tasks. 

It gives me great pleasure to recommend the paper for acceptance to NeurIPS 2022.
confidence: Certain
Decision: Accept