Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This manuscript describes a connection between Potts models and attention as implemented in modern transformers. The authors then present an attention model in which positional encodings are defined as one-hot vectors indicating fixed positions in the multiple sequence alignment and train single layer attention models. These models, unsurprisingly, perform similarly to Potts models without APC correction for contact prediction. The methods section is somewhat confusingly written. I think the factored attention model would benefit from being described on it’s own terms rather than in connection with typical multiheaded attention, especially because the isolation of position encodings and amino acids at those positions dramatically simplifies the understanding of W_Q, W_K,  and W_V. The authors also spend a long time describing well known methods, but without providing additional insight. The connection between the Potts model and attention described in this paper should be obvious to those who already understand attention models and Potts models and the empirical results of the factored attention model don’t make this approach seem compelling. In the discussion, the authors make several broad future speculations. Some of these would be interesting contributions and I encourage the authors to develop this work further. Maybe factored attention could be promising for better capturing dependencies between positions for deeper transformers on MSAs, but it isn’t likely that this work will be of broad interest to the machine learning community. This manuscript seems better suited to a workshop or other specialized venue. Some specific comments on this work follow below.
1.	In the factored attention model, the authors use one-hot encoding of the position index as the position encoding. This is equivalent to learned position embeddings as in BERT which is worth mentioning. 
2.	The authors discuss single-site potentials as a difference between Potts models and single layer attention models and then show a comparison of attention models with and without single-site potentials showing little difference. However, attention models already implicitly have single-site potentials which arise from the positional encoding input features. Granted, this is not the case for the factored attention model where single-site potentials seem to have more effect, though in the negative direction.
3.	The authors state that “The ability of factored attention to capture similar contacts to Potts without use of APC suggest that it may be more suitable for protein design.” I don’t follow this conclusion. If the factored attention model performs equivalently to the Potts model alone and worse than the Potts model with APC correction, why would it be more suitable for protein design?
4.	 What makes the single-layer attention or factored attention models compelling for protein modeling? What problems do these models solve that are not better solved by the Potts model or traditional transformers?

What would raise my score:
1.	Present a compelling use case for the factored attention model. What questions can be answered (or better answered) with this model over the Potts model or other alternatives? One idea is to use the factored attention model as the layers in a full deep transformer model and see if this architecture can improve tasks where MSA training data is available.

Edit: I have increased my score in light of the response and manuscript edits. The manuscript is improved, but I think the method still needs more development. There are a number of interesting pieces but the final picture of an improved protein model is not fully resolved.
review_title: Too basic and lacks compelling use case

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This paper explores the connection between the classic Potts model-based approaches and modern Transformer-based approaches for protein contact map prediction. To this end, the authors introduce a simplified variation of the attention layer called factored attention, and show that a single layer of factored attention performs operations similar to those performed by the Potts model-based methods.

Pros:
- The paper attempts to connect classic and modern approaches to protein contact map prediction, which might be interesting to the people working in this field. The evidence presented (simplifying attention layer so that the equations look similar to the classic methods, numerical results of the simplified attention layer close to the classic methods) is reasonably convincing.
- The topic of the paper is quite timely, there has been a lot of interest recently in modelling proteins using the latest NLP techniques.
- The paper is well written. I appreciate the effort put in by the authors to define basic protein terminologies which might not be obvious to readers without biology background. 

Cons:
- The contributions of the paper would have been more interesting if the proposed modifications of the attention layer led to increased prediction performance of models which are representative of the state-of-the-art. Specifically, if retraining ProtBERT-BFD using the modified attention layer led to further improvement in performance, that would have been a solid contribution.
- Are MRF models really that competitive for contact map prediction? From what I understand, deep neural networks have been far better at this task for quite some time now. At multiple places in the paper, the authors give the impression that MRF models are close to state-of-the-art.
- In the last paragraph of the introductory section, the idea of encoding the MSAs is introduced which seemed interesting. However, from what I understood from the rest of the paper, the queries and keys are extracted solely based on the position of the amino acid. Is that right? If so, does the position correspond to the position in the sequence or in the MSA? Are the actual alignments used in any of the results in the paper? Please clarify.

Comments:
- Section 3.1: "each edge" should have a capital e.
- Section 3.3, specifically the part where you show that factored attention is a pairwise MRF, is too brief. Given that this is a main contribution of the paper, it would be worthwhile to explain this connection in a more detailed manner.
review_title: Official Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Recently, some researchers tried to apply attention models into the protein field, using self-supervised learning to predict protein contacts. In this work, the author attempt to build the connection between such works and the old-school model, Potts model. By simplifying some operations within the attention model, the author managed to build an analog between the simplified model and the Potts model. The analog is intuitive and easy-to-understand. The authors further compare the simplified model and the Potts model on 748 protein families, showing that they are similar. Or probably the simplified attention model is even better. This is an interesting work. However, I also have a number of concerns. The advantages and disadvantages are listed below.

Pros:
1. The manuscript is concise and easy-to-understand.
2. The idea is intuitive and reasonable, with experimental support.

Cons:
1. The analog between the simplified attention model and the Potts model is intuitive but not rigorous. The authors claim that they provide a theoretical connection between the two models. However, that part is not strong enough, without proof.
2. There are two assumptions in this work, which make the simplified model different from the attention models that the previous researchers used. Firstly, they train the model on multiple sequence alignment instead of the raw sequences. If they train the model on the raw sequences, the performance is unacceptable, as shown in Figure 16, which is consistent with the previous research. Secondly, they removed the sequence embedding in queries and keys. This simplification makes the model only consider the statistical pattern in the MSA. To me, this one is a too strong assumption. 
3. The running time and hardware comparison is missing. If the single layer of attention is comparable to the Potts model, not outperform it significantly, while it would take much more time to train, the researchers would need to think twice if they want to use the attention model. 
4. The ablation study makes me feel that the results are on the opposite of the conclusion. Here is my logic. With the above two assumptions, the attention model can achieve similar performance as the Potts model, or a little bit better. However, when we train on the unaligned sequences, which is the usual case that we would use the attention model, the performance becomes unacceptable. Then why we want to use the more expensive attention model? The attention model in the NLP field is a different story. Those models are refreshing the STOA performance all the time. However, in the protein field, the attention model can still only achieve comparable performance as the classific models, after a two-year study. They seldom outperform classic algorithms. The results in this manuscript are consistent with the previous research. So I am not convinced regarding the conclusion in the abstract:
"Taken together, these results provide motivation for training Transformers on large protein datasets."
5. The potential audience of this paper would be those who are specialized or interested in bioinformatics and protein.
review_title: Very interesting work but lack of some clarifications

final_score: rating: 7: Good paper, accept
review_content: Summary
=======
Transformers models have been recently shown to capture protein contact information in their attention maps when trained unsupervised of millions of protein sequences. This paper draws parallels between Transformers and Potts models (fully-connected pairwise MRF)--the current standard approach for protein contact prediction--and shows empirically that Transformers are competitive with Potts models. Understanding the differences and similarities between Transformers and Potts models makes Transformers less of a ‘black-box’ and helps to establish them as a principled method for contact prediction. The paper is clearly written and the evaluation is solid. I have only a few comments.


Major comments
=============
1. What is the maximum sequence similarity between the training sequence of ProtBERT and sequences in TrRosetts alignments that were used for testing? Sequences must not overlap have a maximum similarity of let’s say 80%.

2. You describe that you used three sets of families from the TrRosetta dataset (A.4.1). Why did you use only 732 families for testing (set 3)? Were these all families that were not included in the first two sets? How many families do the first two sets include and how similar are families of different sets? Ideally, train, tune, and test families belong to different super families.

3. You describe in section A.3 how you extracted protein contact maps from the attention maps of ProtBERT. This is an important detail that must be described in the main text. How did you choose the 6 heads? Did you choose them manually or, for example, by training a linear model to predict contacts from attention maps and using the weights for identifying important heads, or computing the weighted average of attention maps?


Minor comments
=============
4. Section 3.2, ‘x = E_seq(x_i) + E_pos(i)’: How did you compute positional embeddings and why do and add embeddings instead of concatenating them?

5. Section 3.2, ‘We treat the positional embedding E_pos as an overall summary per-position information’. Please describe more clearly what this summary is.

6. Section 4, first paragraph: The L of the precision at L metric is not the sequence length but the number of top sequences. You describe L as being both. 

7. Figure 6 is not discussed. Instead of showing this figure, I suggest quantifying the correlation depending on the number of heads by computing and discussing  the Spearman correlation.

8. Rives et al  2020 ‘Biological structure and function emerge…’ have recently shown in addition to Vig et al that protein contact can be predicted from attention maps, which must be also pointed out in the ‘Background’ section.

review_title: Overall good paper about the relationship between Transformers and Potts models

Meta-review: The paper shows a connection between Potts model and Transformers and uses the connection to propose a factored attention energy to use in an MRF. Results are shown, using this energy based on factored attention. Also, pretrained BERT models are used to predict contact maps as a comparison.
The reviewers found the paper interesting from a protein structures prediction point of view, but from a machine learning perspective their opinion was that the paper does not offer a coherent, compelling method that is very novel, and the connection between Potts and an energy based attention model is not that overwhelming.  In addition the presentation was somewhat circuitous.  

The authors made improvements to the paper over the course of the review, which is appreciated, but the method presented does not match the target for an ICLR paper in terms of methodological contributions.  
Decision: Reject