Fast Uncovering of Protein Sequence Diversity from Structure

Authors: luca alessandro silva, Barthelemy Meynard-Piganeau, Carlo Lucibello, Christoph Feinauer
Primary area: generative models
Keywords: Protein design, inverse folding, generative modelling, transfer learning

Abstract:
We present InvMSAFold, an inverse folding method for generating protein sequences optimized for diversity and speed. For a given structure, InvMSAFold generates the parameters of a pairwise probability distribution over the space of sequences, capturing the amino acid covariances observed in Multiple Sequence Alignments (MSA) of homologous proteins. This allows for the efficient generation of highly diverse protein sequences while preserving structural and functional integrity.
We demonstrate that this increased diversity in sampled sequences translates into greater variability in biochemical properties, highlighting the exciting potential of our method for applications such as protein design. The orders of magnitude improvement in sampling speed compared to existing methods unlocks new possibilities for high-throughput in virtual screening.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1729685348444

thread_type:
official_review

<thread_object>
creator: reviewer_d2ua

<content>
summary:
In this paper, the authors present an efficient method for designing protein backbones using a neural network that predicts a Potts model. The proposed architecture includes a pre-trained ESM1-IF encoder that encodes the protein backbone, generating rotation-invariant embeddings. These embeddings are then passed through a transformer-based decoder, which produces a low-rank matrix that is ultimately used to compute the fields and couplings. The low-rank approximation is a clever technique that helps mitigate the quadratic scaling cost typically associated with such computations. The neural network was trained using two distinct approaches: (1) a standard pseudo-likelihood loss, and (2) autoregressive sampling (over amino acids) with maximum likelihood training. To avoid training on single sequences, the model was trained on multiple sequence alignments (MSAs), with the mean pseudo-negative log-likelihood calculated over randomly sampled subsets of the MSA. Training and testing data were sourced from the CATH database, following its hierarchical classification to create test sets of varying difficulty, depending on the similarity between the training and test data. The authors demonstrate that their model better reconstructs covariance matrices compared to ESM1-IF, based on Pearson correlations. Moreover, the authors show that projected MSAs using PCA more closely reflect the natural sequence distributions, suggesting that their generated sequences, or predicted MSAs, are more diverse. When refolding designed sequences for test set structures, the InvMSAfold method proves to be more robust than ESM1-IF for sequences that deviate further from the native structure, and comparable to ESM1-IF for sequences that are highly similar to the native.
In conclusion, the paper demonstrates how a Potts model can be efficiently constructed, showing that the resulting model generates sequences that are plausible, diverse, refold successfully with AlphaFold2, and possess other promising biochemical attributes.

strengths:
The authors present a novel and elegant approach to optimizing Potts model construction, training and sampling. The paper is well-structured, clearly outlining each crucial part of the methodology in a way that is easy to follow. The method is compared and benchmarked against a well-established approach, and performance metrics computed and reported.

weaknesses:
While the proposed methodology for improving the efficiency of Potts model construction is promising, there are a few areas where the paper could be strengthened. First, Potts models have long been used in fixed-backbone protein design, which makes it difficult to clearly identify the novelty and specific contributions of this work. Additionally, the method relies on components of ESM1-IF and then benchmarks against this model, which may limit the fairness or objectivity of the comparison. Another area for improvement is scalability. The paper does not provide any analysis on how the model handles large structures or long sequences, which could be useful for evaluating its broader applicability. Furthermore, there is no discussion on the significance of using MSAs for training versus single-sequence training, nor is there any exploration of how deep the MSAs need to be if they are indeed important.

questions:
I'd like to raise a few major points:
* It would strengthen the paper to benchmark against other methods as well. I would suggest for example a simple Potts model without the low-rank approximation and without the pre-trained ESM1-IF encoder, and an additional method such as ProteinMPNN beyond ESM1-IF. This would highlight the contributions of the paper more clearly, as currently, it may seem somewhat reliant on ESM1-IF.
* I highly recommend adding a plot that shows RMSD versus sequence recovery, as these metrics would provide valuable insights into the model’s performance.
* In Section 2.2.1, the explanation of Equation 7 and how it maintains linear scaling isn’t entirely obvious, at least to me. I suggest elaborating on this either within the main text or in the supplementary material to clarify the reasoning. It would be helpful to include 1-2 sentences explaining why the method or process is linear and how this linearity is established. This will provide clarity to the reader and strengthen the argument by highlighting the underlying reasoning behind the concept.
* To make the manuscript even stronger, it would be useful to include (1) an analysis of how the method scales with very large sequences or structures, and (2) a discussion of how the size of the MSA impacts model performance.

A minor point:
* In Section 2.2, I recommend including the formula that shows the normalization constant, as it is referenced in the text but not explicitly provided.

There are several typos throughout the manuscript that disrupt the flow. I have listed the ones I noticed while reading, but I recommend a re-read of the manuscript to specifically check for additional typos:
   - Line 17: The phrase “space of sequences with pairwise interwise interactions, capturing the amino acid…” contains the term “interwise,” which doesn’t seem correct or clear.
   - Lines 107-108: The word "Moreover" is used consecutively, which disrupts the flow.
   - Line 117: The word "Whos" should be corrected to "Whose."
   - Line 299: "We monitor the the negative..."—"the" is repeated.
   - Line 315: "A can be seen..." should likely be "As can be seen..."
</content>
</thread_object>

</thread>
<thread>
creation date:
1730641718310

thread_type:
official_review

<thread_object>
creator: reviewer_92if

<content>
summary:
This paper proposes a neural network called InvMSAFold, which takes the protein structure as input, and outputs the parameters of two statistical models. These models are then used to generate a diverse set of protein sequences corresponding to the input structure. By utilizing these simple statistical models, the proposed pipeline effectively addresses two major challenges faced by other inverse-folding methods, such as ESM-IF: (1) the limited diversity of generated sequences and (2) slow sampling speed.

strengths:
In their computational experiments, the authors demonstrated that the sequences generated by their models not only fold into the target structure but also exhibit greater diversity and more effectively capture the correlations between residues at different sites. Furthermore, the showed that this sequence diversity extends to other properties, such as predicted solubility and predicted and predicted thermostability. Overall, this paper represents a new methodological advancement.

weaknesses:
1.	The authors only compare their method with ESM-IF1, and do not compare their method with other state-of-the-art inverse folding methods.
2.	In many places such as in section 1, "ESM-IF" was wrongly typed as "ESM-1F". This may lead readers to perceive the authors as lacking expertise.
3.	The article contains too many grammatical errors.

questions:
1.	The symbols of Eq.(5) is not consistent with that of Eq.(3). It would be better to use consistent symbols.
2.	The proof in section 2.2.1 is incoherent. What is the function of Eq.(5)?
3.	In Eq.(7). It would be better to clarify that Eq.(7) is the L2 regularization term. 
4.	In section 3, it would be better to list the number of entries in each dataset.
5.	In section 4.1, what is the necessity of tuning the hyper-parameters of InvMSAFold-AR?
6.	It seems that InvMSAFold-PW performs better than InvMSAFold-AR at larger hamming distance. What is the probable cause?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648250562

thread_type:
official_review

<thread_object>
creator: reviewer_ynez

<content>
summary:
The paper presents InvMSAFold, an inverse folding model that generates the parameters of a probability distribution over the space of protein sequences with pairwise interwise interactions, allowing for efficient generation of diverse protein sequences while preserving structural and functional integrity. InvMSAFold is a neural network in which the inputs are the structure backbone coordinates X and the outputs are the parameters of a lightweight sequence model. The lightweight sequence model parameters are used to sample amino acid sequences compatible with the input structure. Training is based on the CATH database, which classifies protein domains into superfamilies and further into clusters based on sequence homology. The model is fast and has uses in protein design and virtual screening. Biologically, the model captures amino acid covariances observed in Multiple Sequence Alignments (MSA) of homologous proteins. The model expands the scope of inverse folding to retrieve a landscape of homologous proteins with similar folds (they say the 'entire' landscape, I don't think they have shown this).  I am overall very enthusiastic about this work.

strengths:
The sampling speed of InvMSAFold is a lot faster than ESM-1F or ProteinMPNN, this is important when you want to generate millions of models, as I think could be reasonable for virtual screening/protein design applications. 

InvMSAFold seems able to sample more diverse regions of potential protein structure/function space than ESM-1F, again this is important when you are trying to select for particular properties (substrate specificity, thermostability).

That InvMSAFold is able to capture residue covariances in MSAs may also be useful for better backbone modeling that particular functions could then be engineered into.

weaknesses:
There is not a specific example taken through to the conclusion that the model preserves "structural and functional integrity".  Functional integrity is what you want when you're designing new proteins/doing virtual screening. The authors should consider including such an example or clarifying this statement since that is a major claim of their paper. 

I was not clear on the InvMSAFold-AR/-PW. I understand that PW requires MCMC and AR does not but I wonder are there cases/tasks in which a PW vs AR model is more appropriate?

questions:
I was not clear on the InvMSAFold-AR/-PW. I understand that PW requires MCMC sampling and AR does not but I wonder are there cases/tasks in which one of the two PW/AR models is more appropriate? 

What would be an example in which you could demonstrate preserved functional integrity that is not directly related to structural integrity in your model's generation of diverse protein sequences?  It seems an important question because when you want to design a protein to do some specific function (bind some small molecule or interact with another protein) you only care about structure to the extent that it acts as a proxy for function. But maybe it doesn't have to be? Do you think your models could get at function outside the restraint of the specific structure that is your input?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666464752

thread_type:
official_review

<thread_object>
creator: reviewer_wqzy

<content>
summary:
InvMSAFold is an inverse folding method that is optimized for diversity and speed. The general idea is to use a neural net to predict from an input structure and sequence a pairwise interaction model (a Potts model or Boltzmann machine) that captures the structure-sequence relationship and can be used to efficiently generate sequences that differ largely from the input sequence. To tame the number of parameters (fields and pairwise couplings), InvMSAFold predicts a low-rank approximation of the coupling matrix. The paper proposes two models: 1) InvMSAFold-PW is a full pairwise model that reduces the number of parameters significantly and also allows for efficient learning by using a maximum pseudo-likelihood. A drawback is that sequence generation requires MCMC. 2) InvMSAFold-AR is an autoregressive model whose likelihood is tractable thereby allowing for Maximum-likelihood parameter estimation as well as sampling of sequences in a straight forward fashion. Using various metrics the authors show that InvMSAFold, and in particular InvMSAFold-AR, outperforms current state of the art.

strengths:
* An interesting idea to approach the inverse folding problem (i.e. the problem of generating sequences that fold into a given structure). 
* Proposes a low-rank approximation of the couplings and fields of the lightweight sequence model. 
* Fast generation of sequences that fit a well to a given structure.

weaknesses:
* The idea of generating a Potts model has already been proposed by Li et al. (2023).

questions:
* How is sampling of InvMSAFold-PW achieved? Which MCMC algorithm do you use? 
* By using a PCA projection you show that sequences generated by InvFoldMSA have a better coverage of sequence space. But why do you restrict the analysis to the first two principal components? 
* Have you tried AlphaFold3 to validate the sequences generated by InvFoldMSA?

__Typos / grammar__

* Line 117: "and whos outputs"
* Line 212: "can be reduce to"
* Line 248: "robsutly"
* Line 277: "chose" - should be present tense
* Line 302/303: What do you mean by "consistent with the hardness reasoning behind the split"
* Line 475: "becoming worse that both"
* The use of the symbol $\\propto$ to indicate quality up to an additive constant is a bit unusual.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732662198587

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised version of the manuscript

comment:
Thanking again all the reviewers for their fair and interesting points, we have uploaded a revised version of the manuscript. All the questions raised by them should be addressed within the flow of the updated manuscript, hopefully to their liking. In particular, we have added to some experiments also ProteinMPNN as a comparison, as it was suggested by multiple reviewers. Results are reported in Appendix A.1 and B.2 as indicated in the manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents InvMSAFold, an inverse folding method designed to generate diverse protein sequences from a given structure.

The reviews identified strengths in the speed of the method and the greater diversity that more effectively capture the correlations between residues at different sites. 

Weaknesses include limited context including prior work and the fact that the authors compare only to ESM-IF1, and do not compare their method with other state-of-the-art inverse folding methods or to alphafold3.

The authors acknowledge the lack of comparisons and promise to add InvMSAFold-AR/PW with ProteinMPNN to a revised manuscript. Alphafold3 was not available at the scale needed to refold the sequences as suggested. The authors offered to correct typos.