Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this manuscript, the authors present a method for predicting residue-residue contacts within protein structures using the attention layers learned by transformer language models. Using the largest transformer language models trained to data, the authors show good performance for contact prediction. The paper is clearly written and easy to follow.

The general concept of fine tuning protein language models for contact prediction has circulated for some time which lessens the core contribution, but the authors approach is surprisingly data efficient and accurate. Overall this is an interesting work, though there is quite a bit of background on contact prediction missing. This paper is also very application specific and may not present new machine learning methods of general interest to the ICLR community. The existence of previous language model-based contact prediction methods reduces the novelty of this work, especially given that the model used here is from Rives et al. 2019, who already look at contact prediction. Furthermore, no comparisons with state-of-the-art evolutionary coupling-based or language model-based contact prediction methods are performed. With this in mind, the manuscript may be better suited to submission at a biology specific venue.

Additional specific comments follow below.

Major comments:
1.	Missing related work: there are a number of highly relevant prior works that are not mentioned/discussed. In particular, “Deep generative models of genetic variation capture the effects of mutations” – Riesselman et al. 2018 was, as far as I know, the first paper to show that deep generative models capture structure information (see Figure 6). Following that, “Learning protein sequence embeddings using information from structure” – Bepler & Berger 2019 was, to my knowledge, the first paper to propose deep language models (alignment free) for learning protein sequence representations and used those unsupervised representations for contact prediction. Furthermore, there has been extensive work in improving contact prediction using sequence + co-evolutionary features. See, for example, “Enhancing Evolutionary Couplings with Deep Convolutional Neural Networks” Liu et al. 2018 and “Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model” Wang et al. 2017. Other papers looking at protein structure prediction from sequence with deep learning, though they are less directly relevant, include “End-to-End Differentiable Learning of Protein Structure” AlQuraishi 2018 and “Learning Protein Structure with a Differentiable Simulator” Ingraham 2019.
2.	Before this work, others have looked at fine tuning language models for contact prediction. How do those approaches compare with the approach presented here? Rives et al look at contact prediction in their manuscript describing the transformer model (which is the same model used here) on CASP 11-13 (see Table 5 in their manuscript). How does that approach compare with this one? Likewise for Bepler & Berger
3.	Many methods have surpassed GREMLIN for contact prediction using evolutionary couplings. How do those approaches compare with this one? It would be helpful to see how this approach compares with truly state-of-the-art contact prediction methods. Reporting results on the CASP data would help to make this comparison.

Minor Comments:
1.	Although multiple sequence alignment methods have challenges especially as related to evolutionary coupling prediction, these methods have been heavily optimized for decades. The authors should provide citations for claimed failings such as “failure to find an optimal alignment” and “suboptimality of the substitution matrix and gap penalty.” Certainly, these may be sources of error in alignments, but I am not aware of any studies of the frequency or impacts of these errors on evolutionary coupling analysis. If these studies exist, I encourage the authors to cite them. If they do not exist, I suggest the authors focus on well known sources of error here (namely, alignment depth) and provide references.
2.	The authors use the language model without fine tuning, but the model could be fine tuned for each protein using its MSA. It’s great that contacts can be predicted without fine tuning, but it would be interesting to investigate whether additional gains can be made.
3.	Eight iterations of jackhmmer is a lot. In my personal experience, jackhmmer often diverges at 3+ iterations. By this I mean, the set of sequences and HMM learned by jackhmmer drift far away from the original sequence/family. Did the authors perform and quality checks of these alignments to ensure jackhmmer did not diverge?
4.	How are sequence depths in Figure 3 calculated? Is this the raw number of sequences in each MSA or is it after applying some sort of neighborhood weighting to calculate an effective number of sequences? 

Things that would improve my rating:
1.	Provide a more comprehensive background review.
2.	Compare with state-of-the-art evolutionary coupling-based contact prediction methods.
3.	Compare with other language model-based contact prediction methods.
4.	What should interest the general machine learning community about this paper? What can we learn that might lead to better ML methods in the future? Convince me that this doesn’t belong in a bioinformatics venue!


review_title: Interesting idea, but background and comparisons are lacking

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Summary**
The paper performs a number of analyses centered around the ability of transformer-based language models trained on protein sequence data to learn representations useful for predicting protein secondary and tertiary structure (the latter as contact maps). Specifically, the paper studies several pre-trained transformer models by fitting an L1-penalized logistic regression to amino acid pair contacts. Several experiments are performed to showcase that (i) transformer-based representations can outperform state-of-the art methods based on MSA in terms of contact prediction precision; (ii) that the necessary information for contact predictions in these representations is learned in an unsupervised manner (and not by the logistic regression put on top of these representations); and (iii) that the contact prediction probabilities are reasonably well calibrated.

**Score justification**
In its current form the paper presents interesting analyses, but has overall limited novelty. The ability of transformer models to learn representations predictive of secondary and tertiary structure has been demonstrated before (including in the papers proposing the models used by the authors). Furthermore, I have some questions regarding the methodology employed by the authors.


**Major comments**
* The main metric employed by the authors is the precision of the top L (protein length) contact prediction for a given range (P@L). I wonder why the authors do not also consider recall at L as an accompanying metric for reporting the results.
* When comparing ESM to the baseline Gremlin method, the authors consider two scenarios: (i) Gremlin trained on the trRosetta data; and (ii) Gremlin trained on the same data as the ESM transformer model. Overall, Gremlin trained on the ESM data - which is arguably the correct baseline for the ESM model -  performs worse than Gremlin trained on the trRosetta data. Why is that the case? How does the procedure for preparing MSA for the ESM data compare to that of the trRosetta data? Can it be tuned to improve Gremlin's performance?
* The paper compares several transformer models that differ primarily in the model size, dataset size and hyper-parameters. As can be seen from Table 1 of the manuscript, these differences are clearly important for the contact prediction task and thus should be summarized and discussed in more detail.
* From what I understand the sequences from the testing set of the contact prediction problem (or sequences highly similar to them) could appear in the training sets of the considered transformer models. This creates some information leakage. It's unclear from the results presented in the paper whether it is an issue or not - how does contact prediction precision / recall change as sequence similarity to the ESM training set drops?
* The authors present analysis on the usefulness of the representations learned by various attention heads for contact prediction; and on robustness of such predictions. I wonder how robust the results of these analyses are - they appear to have been performed using a single checkpoint of the ESM model, which is a result of stochastic training from random initialization.
* In the Appendix the authors talk about the benefit of using predicted contact maps for inferring the all-atom protein 3D structure. However no results on this are presented. I would be very eager to see the comparison of 3D structure accuracy inferred with ESM-predicted and Gremlin-predicted contacts.

**Minor comments**
* Introduction talks about the ESM-1b model but (as far as I can tell) a reference isn't provided until a later section.

review_title: Interesting analyses, but has overall limited novelty

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors show that transformer protein language models can learn protein contacts from the unsupervised language modelling objectives. They also show that the residue-residue contacts can be extracted by sparse logistic regression to learn coefficients on the attention heads. One of the advantages of using transformers models is that they do not require an alignment step nor the use of specialized bioinformatics tools (which are computationally expensive). When compared to a method based on multiple sequence alignment, the transformers models can obtain a similar or higher precision.

Contributions of this paper are:
- showing that the attention maps built in Transformer-based protein languages learn protein contacts, and when extracted, they perform competitively for protein contact prediction;
- a method for extracting attention maps from Transformer models;
- a comparison between a recent protein transformer protein language model (which does dot require sequence alignment), and a pseudo-likelihood-based optimization method that uses multiple sequence alignment;
- an analysis of how much the supervised learning (logistic regression) contributes to the results.

The paper covers a relevant topic and it is easy to read. 

However, I have a number of concerns. The main contribution of the paper is that attention maps built in Transformer-based protein languages learn protein contacts and can be used for protein contact prediction. However, this was reported before in Rives et al.(2019) (doi: 10.1101/622803). Also, several methods have been developed for this problem, but are not included in the comparisons. Finally, the provided implementation details are not sufficient to reproduce the results of the paper. 
I detail some of these concerns below, together with questions/suggestions for improvements:

1) I would recommend comparing transformers to other methods besides Gremlin, or justify why other methods were not included. This review can be helpful:

(Adhikari B, Cheng J., 2016.. doi: 10.1007/978-1-4939-3572-7_24)

Also, more recent methods that were published after the review are:

(Badri Adhikari, 2020. https://doi.org/10.1093/bioinformatics/btz593)

(Luttrell  et al., 2019. https://doi.org/10.1186/s12859-019-2627-6)

(Gao et al.,2019. https://doi.org/10.1038/s41598-019-40314-1)

(Ji S et al., 2019. https://doi.org/10.1371/journal.pone.0205214)

2) On page 7, the authors state that "We find that the logistic regression probabilities are reasonably well calibrated estimators of true contact probability and can be used directly as a measure of the model's confidence (Figure 10a)". However, from the plot in Figure 10a, it is not totally clear that the probabilities are well calibrated. Could the authors add more justifications of why they consider it well calibrated? Could they also show a comparison of the calibration of the other transformer models, perhaps using MSE as a calibration metric?

3) To understand the occurence of false positives, the authors analyze the Manhattan distance between the predicted contact and the true contact, which is between 1 and 4 for most false positives. They also show an example of a homodimer, for which predictions were far from the true contacts, and explain that the model is picking up inter-chain interactions. Could the authors report how many predictions have a Manhattan distance larger than 4? Is this one example representative of the group of false positives far from the true contact? Maybe the authors could analyse whether this happens in most of the cases.

4) While ESM-1 is open-source and publicly available, this is not the case for ESM-1b. In section A.5, the authors provide implementation details as differences between ESM-1 and ESM-1b, stating “Compared to ESM-1, the main changes in ESM-1b are: higher learning rate; dropout after word embedding; learned positional embeddings; final layer norm before the output; and tied input/output word embeddings. The weights of all ESM models throughout the training process were provided to us by the authors.”. In my opinion, this is not enough to reproduce the results in this paper. To make it reproducible, the authors need to provide a detailed enough description of the differences to make the reader able to implement ESM-1b, or provide the weights and hyperparameters required to reproduce their results.

review_title: Review for TRANSFORMER PROTEIN LANGUAGE MODELS ARE UNSUPERVISED STRUCTURE LEARNERS

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary
The paper shows that Transformers trained unsupervised on millions of protein sequences learn information about protein contacts by using attention maps for contact prediction. The paper is mostly clearly written and discusses server interesting ablation experiments. However, two recent papers that appeared on arXiv before the ICLR submission deadlines also use Transformers for protein contact prediction. These papers and other methods for contact prediction beyond Gremlin are not described. I therefore consider the contributions as insufficient for an ICLR submission.

## Major comments

1. Using Transformer attention maps for protein contact prediction is not new. See Rives et al, 2020, ‘Biological structures and functions emerge…’, section 5.2, and Vig et al, 2020, ‘Bertology’ section 4.2. Both publications appeared on arXiv at least one month before the ICLR submission deadline and are not clearly discussed in the paper.

2. The introductions discusses existing work on Transformers for protein languages models. Existing methods for contact prediction (beyond Gremlin), however, are not described sufficiently.

3. It is unclear which sequences were used for training the Transformer models and how similar they are to test sequences.

4. The paper compares Transformers to Gremlin. However, it is unclear how well they perform to the CASP state-of-the art (see also Rives et al, 2020).

5. Section 3.4  does not describe clearly enough how attention maps were used for predicting contact maps. How were attention maps symmetrized? Which layers and heads were used and how were they aggregated? What is the number of resulting features that were used to train the logistic regression model? APC is not described or cited.

6. Section 4.5 discusses that Transformers can be also used for secondary structure prediction. This is not new (see Rives 2020 and Vig 2020) and does not fit well to the rest of the paper, which is about contact prediction. 

6. Section 4.8: Using transformers for generating proteins with natural properties is not new (see Madani et al, 2020, ‘ProGen’ or Rives et al, 2020). ‘Wang & Cho’ were not the first who used Transformers generativity (see Vaswani, 2017).

review_title: Using Transformers for protein contact prediction is not new

Meta-review: The authors have done a very thorough job of responding to the comments from reviewers. The paper has a clear contribution, namely that attention maps predict contacts as well as existing unsupervised pipelines. This paper deserves to be published.

In the final version, the authors should discuss briefly "BERTology Meets Biology: Interpreting Attention in Protein Language Models"(https://openreview.net/forum?id=YWtLZvLmud7) and "Improving Generalizability of Protein Sequence Models via Data Augmentations" (https://openreview.net/forum?id=Kkw3shxszSd). However, the authors should also make sure that the final version respects the ICLR length limits.

I am recommending poster acceptance because the results are anticlimactic given the recent success of Deepmind at CASP 2020.


Decision: Accept (Poster)