Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors present a model that predicts contact maps from multiple sequence alignments. The paper proposes a weighted sum of k, LxL attention maps (k=#sequences, L=sequence length), where a per-residue-pair weight is learned across all sequences in the MSA. Afterwards a convolutional layer is applied to refine the prediction. The authors show that the co-evolution transformer outperforms existing supervised contact prediction methods on CASP14.
main_review: The idea of computing per-residue-pair weight across all sequences in the MSA is the main contribution of the paper. The general concept of attention maps for contacts and convolutional layers for refinement are well-known techniques in the field. The authors have clearly demonstrated the uniqueness and contribution of their idea through proper ablations. Related work is adequately cited. 

The explanation of the co-evolution transformer is sound. One detail the authors should clarify in Line 161 (submodule 2) is when “the co-evolution map A is concatenated with A' from the previous CoT layer”, how is this handled in the first CoT layer when A’ does not yet exist? As a side question, why did the authors not additionally try a ResNet for the co-evolution enhancement? The expectation is that this would perform at least as well as the conv-linear-softmax block proposed in the paper.

There seems to be little comparison of Co-evolution Transformer performance on the different MSA generation methods (different databases). This is an interesting topic, and it is not clear why the authors did not mention it in the main paper.

Most claims are well supported, but two are suspect.
Line 97: [Pretraining-based methods] are still at an early stage thus cannot achieve comparable performance to the SOTA approaches currently.
Line 104: [MSA Transformer] it still cannot well extract the co-evolution information due to the non-homologous subsequences inevitably introduced by the generated MSAs.

These claims don't seem to be correct. Re the claim on line 97, MSA Transformer (Rao, et al. 2021) shows SOTA contact prediction on CASP13 free modeling targets relative to trRosetta. As such, it’s not clear why this model wasn’t included in the comparisons. Re the claim on line 104, Rao, et al, 2021 show the model extracts co-evolution information from MSAs through experiments with unsupervised contact prediction. These claims should be revised, and the paper would be strengthened by a comparison.

50.4 Precision @ L on CASP14 free modeling targets compared to 41.8 is compelling. When comparing to the known methods (RaptorX, TrRosetta, CopulaNet), the authors should clarify whether the precision scores from the baseline methods are based on predicted contact maps OR the contact maps of predicted 3D structures. 

As the current field stands, there is no single accepted way on how deep learning models should process MSAs. This paper presents a unique approach that merits consideration from the broader community. 

Additional notes: Please define how you determine “the most confident prediction” [line 226]

---------------------------------
EDIT: See discussion around "Training data and comparisons". Authors have agreed to address the temporal split in the revision, which will strengthen the comparison to other methods.

limitations_and_societal_impact: The authors have addressed the main limitation of their work, namely that the co-evolution transformer does not perform well on low-depth MSAs. Solving this problem is beyond the scope of the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors address an important problem in structural biology and obtain what looks like state-of-the-art performance.  The proposed method predicts inter-residue contacts of a protein based on its sequence presented as a multiple sequence alignment (MSA).  The method is innovative in the way it represents an MSA using attention on individual sequences within the MSA, avoiding the step of summarizing the MSA as a "profile" that summarizes amino acid composition in each position, a common way of representing MSAs.

main_review: 
Strengths:

- The idea of using attention to selectively focus on individual sequences within the MSA is innovative and interesting.  In their case study the authors demonstrate how this allows the method to focus on sequences for which no gap is present in a given position.

- State-of-the-art performance on a highly important problem in structural biology.

Weaknesses:

- I think the Co-evolution aggregation submodule (equations 7 and 8) and the co-evolution enhancement submodule could have been explained a bit more clearly.


Comments:

- If there is a gap in the original sequence in the MSA, that position  cannot be a part of any interactions.  It is not clear how this is addressed in the method.

- It is not clear how the MSA is provided as input.  Given my understanding of the model, I assume the input is a one hot encoding of each sequence in the MSA.  That step is also missing in Figure 1.

- Section 3.3 compares the expressivity of the proposed CoT layer with a simpler attention model over individual sequences.  However, in my opinion, a more natural comparison is with a model that summarizes the MSA by a sequence profile.

- It's not clear how the selected model parameters were chosen.  No model selection strategy was described.

Minor comments:

Please discuss the following method, as it also uses attention:
Chen, C., Wu, T., Guo, Z., & Cheng, J. (2021). Combination of deep neural network with attention mechanism enhances the explainability of protein contact prediction. Proteins: Structure, Function, and Bioinformatics, 89(6), 697-707.
I suspect it does not perform as well as the proposed method since its performance is similar to RaptorX.

- The following statement is incorrect:  "Transformer is a network architecture built on the self-attention mechanism" - transformers are constructed on the basis of attention rather than self-attention.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This manuscript describes a novel deep learning architecture for predicting protein contact maps. This is a well-studied problem for which deep models already provide the state of the art.  The innovation here is to use a particular type of Transformer architecture that models more dependencies than previous models.  The approach is well described, seems sensible, and gives good performance on a challenging benchmark. On the other hand, the empirical results are unsatisfying in some respects (outlined below), and the benchmarking suffers from a problem (different methods trained on different training sets).
main_review: The key idea of this work is that, in predicting co-evolutionary information for contact prediction, it is important not only to weight sequences non-uniformly and to capture dependencies between residues within each sequence, as has been done by previous model, but also to capture dependencies between sequences in the multiple sequence alignment (the "co-evolution enhancement" model).  The model architecture is well described in Figures 1-2 and Section 3, though I wish the architecture of CopulaNet had been explained in a bit more detail, so as to highlight more clearly which aspects of the proposed model are novel.

The text makes three primary claims (lines 173-179) about what makes this model novel: "1) COA leverages the global information from all the homologs instead of single homolog to derive the attention, which fits better to the residue co-evolution insight; 2) COA handles the non-homologous information naturally by the selective pooling operation during aggregation, providing a solution
to a widely existed but inevitable dilemma for MSAs; 3) COA enhances the co-evolution signal by
propagating information from the neighbors, making it easier to capture the high-order interactions
among multiple residues."   I would have liked to see each of these claims directly supported by empirical evidence in Section 4.  The results in Sections 4.3 and 4.4 go partway in this direction but do not directly test the above hypotheses.  For instance, 4.3 shows that the proposed co-evolution layers give better results overall than a standard self-attention layer but does not delve into the source of this improvement. 

The results in Table 1 are problematic because the different methods are not trained on the same data.  I actually assumed that they were, at first, and only figured out for sure that they were not when I got to Section 4.5 (line 273).  The text should say this explicitly up front, since it makes the performance results reported in Table 1 less trustworthy.  We don't know whether the differences in performance are due to differences in the training set or due to modeling choices.  This is, in my opinion, a major problem with the reported results, though apparently it's a problem in the field as a whole ("... most state-of-the-art models are trained on their private training data, which are usually unavailable.").  The results on lines 275-279 go partway toward mitigating this problem.

line 220: "The improvement on FM/TBM domains (0:8%) is not that significant as the others (> 9:0%)." In fact, no statistical significance was assessed for any of these results. This should be relatively straightforward to do by, e.g., computing the performance measures separately per protein and then doing a statistical test (e.g., paired t-test) on those measures.

I could not make sense of these sentences: "Different groups use different data sources to build their MSAs. To eliminate the variance of different data sources, the most confident prediction of CoT on 12 MSAs from different databases with different search settings, denoted as CoTy, is selected as the final prediction."  It sounds like we are comparing models trained on different training sets.  How does the proposed strategy address this problem?

We are told that "Figure 4 (b) and (c) show that selective pooling assigns larger weights to more relevant sequences as expected," but I am not sure what "more relevant" means here.  The plot seem to show only that the weights are not uniform.

Figure 3 provides anecdotal evidence that including the co-evolution enhancement module improves prediction of long-range contacts. It would have been better to systematically measure this in a quantitative way.

limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper wishes to address the issue of protein contact prediction problem, State of the art approach uses hand-crafted features through Multiple Sequence Alignments(MSAs). This paper proposes a Transformer architecture to automatically identify features suitable for Protein Contact Prediction problem. All the referees were unanimous that while the ideas are novel but their comparison with the state of the art 
is problematic. We hope that the authors can address these concerns in the final manuscript.




Decision: Accept (Poster)