Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ###########################################################################################################


Summary of Paper
The motivation for the paper is a bit unclear. In the introduction, the authors begin by claiming to extend self-supervision "to information from a set of evolutionarily related sequences". However, it does not appear that the model is at all used for pretraining / representation learning as would be expected of a self-supervision method. No further connections to self-supervision are made in the rest of the paper. Based on the conclusion it seems that self-supervision is a future direction. If so, I would suggest rewriting the introduction to more clearly state the motivation of this paper.

The rest of the paper is more clear. The authors tackle the problem of predicting protein contacts. Standard unsupervised approaches fit a Potts model (often with pseudolikelihood) and then use the parameters of the fit model to make predictions about contacts. Such approaches require a MSA of reasonable depth and the amortized optimization approach suggested by the authors has the potential to work better than standard unsupervised approaches in the small-MSA regime by sharing information from all the protein families that the model is fit on.

The authors propose a meta-learning approach wherein a neural network (here a transformer) takes as input a single sequence and outputs the parameters of a Potts model. The model is trained with a pseudolikelihood loss across all sequences within individual families. This is a new and clever idea.

The main experimental result is that the NPM outperforms standard Potts models for shallow MSAs. This demonstrates that there apears to be some utility to this approach. However, the precision is still very poor and the authors make no claim that the enhanced accuracy is of biological utility (i.e. can be used to fold the protein). There are other tools that perform contact prediction from a single sequence with higher accuracy than the NPM. Furthermore NPM performs worse than standard Potts models for medium and large MSAs. This seems to be a significant drawback of the method. 

#############################################################################################

  
Questions / Suggested Experiments:
  
PFAM EXPERIMENT: 

Can the authors please explain why the PFAM experiments are evaluated on top-L precision whereas the remaining experiments are top-L/5 precision? The lack of explanation in the paper suggests the metrics may be cherry-picked to best show the performance of the model. 

Since all the proteins in the family share structural similarity, it is not clear that NPM has learned to transfer any information. After looking at the structures in PyMOL it seems that significant substructure is shared between the different families. Here is a resonable baseline/experiment to add: Align the query sequence to the higher-depth MSAs and then using the MSA that best fits the query sequence, predict contacts based on the individual Potts model trained on that family. This would elucidate whether or not NPM is simply treating the query sequence as if it were from one of the higher-depth MSAs.

UNIREF EXPERIMENT: 

The details of this experiment could use more explanation. I did not see any explanation of the purpose of the training and heldout sets, which in this setting are not obvious. My guess is that the authors are trying to demonstrate that the NPM can generalize zero-shot to new families? If so, please mention it in the text. Random splits are problematic for demonstrating zero-shot generalization since there could be very similar families in the train and heldout sets (e.g. same PFAM clan or structural class). As written, the paper barely discusses the purpose of the heldout set but if it is meant to convey some test of generalization the authors should more carefully construct their heldout test set accordingly.

Why is there no plot for short-range predictions, as there was for the PFAM experiment?

There is an unclear statement: "During training, we iterate over all sequences and their MSAs on every epoch, and subsample to M=30 sequences per MSA." Is this just for NPM or also for the Potts model? I would be very concerned if this is also what is done for the Potts models as it would significantly harm their performance. The authors need to clarify this in the text of the paper.

The authors do not discuss other existing methods that currently exists for shallow MSAs. For example, standard semi-supervised methods are fine-tuned on contact prediction and thus, work with single-sequence inputs. I believe these methods may perform as well or better than NPM. The authors should discuss these methods and include comparisons. 


Model Architecture
Please do the following:
1) In Appendix B.1 you describe a number of "tricks" to reduce the number of parameters. These include (1) a low-rank decomposition of the bilinear form, (2) weight tying by amino acid for the decompositions (3) Convolutional layers. Please provide experiments showing the utility (or lack thereof) of each of these. This will lift the results from "here is what happens if you use these tricks" to "here is data suggesting that these choices we made are better than the naive choices". Thus, add more impact to the paper.  

2) No explanation is given for why the architectures are so different for PFAM and UniRef (e.g. convolutional layers for PFAM only). Please provide an explanation. A priori, I see no reason to use different architectures for these two different domains. It seems to only complicate the paper and methods. If these choices really are important, then this seems to suggest overfitting to particular tasks rather than having a general solution.


Appendix Training Details:
I personally found the written descriptions here to be too vague to understand the exact details. This made it harder to evaluate the paper. Can you please clarify:
1) "we randomly subsample the MSA down to 100 sequences as NPM target sequences...". Based on equation (7) this means the pseudo likelihood is evaluate on 100 random sequences from the MSA. This seems to directly contradict the experiment shown in Figure 3 where the number of sequences in the MSA is varied up to 1000. Please explain what is happening here?


###########################################################################################

Explanation of Score

This paper was difficult to score because I find the idea / approach to be very creative and different from standard techniques of borrowing the latest NLP pertaining task and applying it to proteins. I applaud the authors for taking a unique approach. 

The main drawback is that the authors exclusively evaluate the model on contact prediction and do not demonstrate convincing performance. While the model sometimes outperforms standard approaches with shallow MSAs the performance is still quite bad and no comparisons to other methods are shown. 

Thus, overall I am impressed with the direction of the paper but think it needs more work. 


Updated score from 5 -> 6 after clarifying feedback from the authors. 
review_title: NPM is a promising idea but the paper needs more work

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a new object called Neural Potts Model (NPM) to train a Transformer to learn the local energy landscape of protein sequences. The problem of modeling energy landscapes using the power of techniques in natural language processing (NLP) is a timely and interesting problem. However, there are some concerns that limit the strength and the main claim of the paper that needs to be addressed. 

Is the NPM objective in equation (7) derived from some probabilistic model? While the intuition behind the objective can sound plausible from the discussion that follows, I am still curious what problem formulation is this objective solving? Is it a proper likelihood term?

Since the paper is advocating the use of NPM objectives, is it possible to plot the value of the proposed NPM objective in the Transformers as a function of the Top-L precision? This will strengthen the claim in proving all the gains are actually coming from the objective and not other possible factors in training the model.

The paper can be improved by more effort in punctuation and proper labeling of figures. There are many sentences and phrases that need a comma for better readability. There are extra parentheses when referencing Figures. What are the axes showing in Figure 2? What are the highlighted shades indicating Figure 3 (std or sem)? Is it possible to have error bars for Figure 4? In abstract, MSA is not defined.

Also given that there is sufficient space left in the paper, some material including the Algorithm box can be moved to the main text. Also, the author can consider to use the space to expand on the significance and importance of the NPM objective.

** after rebuttal: thanks for addressing the comments. I have revised my score based on the discussion.

review_title: Interesting problem, however, more validations required to strengthen the main claim.

final_score: rating: 7: Good paper, accept
review_content: Overall comments
- The paper introduces an amortized optimization framework to learn a mapping between an input sequence and the coefficients of the corresponding Potts model -- a standard method used in computational biology to model protein sequences.
- The idea is particularly compelling in the regime where there exists a limited number of sequences (in the training data) that are similar to the target sequence. That would typically result in a low quality multiple sequence alignment (MSA), and in turn a poor Potts model.
- The idea is sound from a theoretical standpoint. The newly introduced framework is well motivated: authors did a good job at explaining the background for the problem and providing an intuition for why the proposed amortized optimization framework would improve over the current approach (i.e. independently training models on each MSA).
- The paper is very well written and easy to read -- the language is precise, everything is defined very clearly. 
- Experiments show good results in the regime that matters most for the introduced Neural Potts model (NPM), i.e., low density of sequences around the query.
- The approach would be even more compelling if the NPM would not underperform the baseline in the regime where Meff (the “effective number” of sequences in the MSA) is larger. 
- Bridging that gap would be very compelling from a practical standpoint: large scale studies typically involve modeling thousands of distinct proteins, and thus fitting thousands of distinct models independently. Intuitively, there is shared information between these models, which is lost when models are trained independently but could be captured via a process akin to the one introduced in this paper.

Detailed comments and questions

Section 1
- Would suggest to add one sentence describing why it matters to solve the low Meff regime to further motivate your approach (e.g., which scientific questions would we be able to better answer?).

Section 2
- I would add a paragraph on Transformers models for protein sequence embedding (since it is a core component of the NPM used in experiments), and cite a few of the key works in this area, for example:
- Rives et al., Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences (https://www.biorxiv.org/content/10.1101/622803v1)
- Madani et al., ProGen: Language Modeling for Protein Generation (https://www.biorxiv.org/content/10.1101/2020.03.07.982272v2)
- Vig et al., Bertology meets biology: Interpreting attention in protein language models (https://arxiv.org/abs/2006.15222)
- Rao et al., Evaluating Protein Transfer Learning with TAPE (https://arxiv.org/abs/1906.08230)

Section 3
- Is there a particular reason for which you are using a vanilla Transformer architecture? Other Transformer architectures (e.g., BERT-like architecture) may be able to learn better sequence embeddings and in turn further close the gap with the independent Potts model in the large Meff regimes (see the above papers for example).

Section 4
- How do you interpret the gap with the baseline Potts model approach in the medium-large Meff setting? 
- Is there a way to detect (based on training) the situations where the NPM will likely underperform the independent Potts model baseline?
- Computationally, how expensive is it to train one NPM Vs one Potts model? For example, if I have 1000 proteins to model in the regime where both perform the same (Meff ~100), is it faster for me to train one NPM model or 1000 Potts models? I suspect the 1000 Potts models are still cheaper to train, but if not that could be one strength of your approach to mention.
- What is the distribution of Meff over the data (section 4.2)? Based on the rightmost plots it seems that the majority of MSAs are in the “>500” regimes where NPM underperforms independent Potts models (asking that question with the understanding that you are more interested by addressing the low Meff setting here).
- Do you have an intuition for why the NPM appears to be doing better for medium range interaction here Vs long range in experiment 4.1?
- How do you explain the U-shape of the top-L/5 precision curves for NPM (on heldout data)? I would have expected that increasingly larger Meff would be associated with monotonically increasing performance (as is the case for the Independent Potts model curves in blue).

Appendix C1
- Why the upweight by a factor sqrt(Meff(n))?

Appendix C3
- Did you look at the performance gap on HTH? Since baseline Potts models yield poor long-range contact on that dataset, perhaps NPM would have helped address that issue, given increased ability to model long range dependencies as per 4.1. It was not clear whether the low performance is due to an intrinsic limitation of the Potts model or potentially due to a low density around each query point in HTH (in which case NPM may help).

review_title: An amortized optimization framework to learn a mapping between protein sequence and the parameters of a Potts model describing the local energy landscape of the input sequence. Approach is well motivated and addresses limitations of existing approaches when few sequences are available in the neighborhood of the target sequence. Paper is well written and easy to read. Experimental results demonstrate that the approach works well in the regime for which it was introduced (low density regime)

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper aims to improve low-depth MSAs, when a protein of interest only has a small number of known evolutionarily related sequences. This is a well motivated problem. MSAs are commonly used for a variety of purposes. Methods to enhance low-depth MSAs can be very useful. In particular, this paper focuses on using MSAs for contact prediction as a down-stream task. I'm not sure if contact prediction is the best use case for this, but it's a well-studied task for proof of concept.

The weight-sharing Neural Potts Model approach is novel (as far as I know) and is a well-motivated idea. There could be reasons to believe that weight sharing could improve Potts Model compared to directly fitting Potts Model to individual MSAs. Potts Model is very commonly used and it makes sense to base their method on Potts Model.

Personally, I think this approach is interesting, but the paper needs further experimental validation for acceptance. There are two main limitations in the experiments:

   1. The paper claims that there is a gain from amortization for low-depth MSAs. This claim is only partially supported for very shallow MSAs. According to the experiments in the paper, there is only a gain for very shallow MSAs (fewer than 15 sequences) for long-range contact prediction, or for MSAs with fewer than 50 sequences for medium-range contact prediction. I'm not sure whether it's a common situation to have only 15 sequences in an MSA. It would be helpful if the authors could provide example use cases where this applies.

   2.  The UniRef50 clusters are based on 50% sequence identity. In the paper, the train-test split is based on the clusters. There could still be significant information leakage if some train and test sequences share 40% similarity. I would like to see the authors further examine this and show that this is not the case.

Suggestions for improving the paper:

  1.  Include more than one PFAM clan level study. The P-loop NTPase example is well taken. It seems like the results here are quite different from the UniRef50 results (NPM has advantage for up to MSA depth 100s vs 15). This could be explained by the families in the same clan are a lot more similar than in UniRef50 in general. It would be helpful to understand why the NPM approach worked particularly well on NTPase, and whether it works similarly well on other PFAM clans. It seems like instead of aiming to use the method on arbitrary sequences, the method could be more valuable in cases where we have MSAs for other related families in the same clan.

  2.  Split train-test differently in UniRef50 to make sure similar families don't end up in both train and test.

  3. Instead of only evaluating the top L/5 contact prediction task, try other thresholds. The achieved contact prediction precision is around 10-20%. That seems rather low for both NPM and direct Potts, so hard to say if the gain is meaningful.

  4. (Perhaps out of scope for this paper.) Could this method be applied to generate Potts Model parameters to be fed into other structure prediction models (e.g. AlphaFold). For example, it would be interesting to see that the NPM Potts Model can improve AlphaFold contact prediction results. 

  5. References for important proteins with shallow MSAs.

Clarification questions:

1. How do you compute the MSA? Is it from HHblits or some other standard package?
2. In the PFAM experiment, how do you vary the MSA Depth? Is it random subsampling?

--------------------------------------------------------------------------------------------------------------
Update:
Updated score from 5 to 6. The author response clarified some key questions and the updated paper incorporated some of the feedback.
review_title: Interesting idea. Limited experimental evidence.

Meta-review: This is a creative piece of work wherein learning of what is normally family-specific Potts models is turned into an amortized optimization problem across different families of proteins. The Potts models are learned with a pseudolikelihood approach, and the evaluation of the model against baselines is performed only on a contact prediction problem. This last point is problematic, because on the one hand, the authors use this "as a proxy for the underlying accuracy of the Potts model learned", and on the other hand, claim that "we do not want to claim our method is state-of-the-art for contact prediction, it is certainly not".  Overall, the paper is promising, but is too preliminary on the empirics to warrant publication at this time.
Decision: Reject