Learning to engineer protein flexibility

Authors: Petr Kouba, Joan Planas-Iglesias, Jiri Damborsky, Jiri Sedlar, Stanislav Mazurenko, Josef Sivic
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: protein flexibility, protein flexibility prediction, protein design, protein sequence design, inverse folding

Abstract:
Generative machine learning models are increasingly being used to design novel proteins. However, their major limitation is the inability to account for protein flexibility, a property crucial for protein function. Learning to engineer flexibility is difficult because the relevant data is scarce, heterogeneous, and costly to obtain using computational and experimental methods. Our contributions are three-fold. First, we perform a comprehensive comparison of methods for evaluating protein flexibility and identify relevant data for learning. Second, we overcome the data scarcity issue by leveraging a pre-trained protein language model. We design and train flexibility predictors utilizing either only sequential or both sequential and structural information on the input. Third, we introduce a method for fine-tuning a protein inverse folding model to make it steerable toward desired flexibility at specified regions. We demonstrate that our method Flexpert enables guidance of inverse folding models toward increased flexibility. This opens up a transformative possibility of engineering protein flexibility.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729719982923

thread_type:
official_review

<thread_object>
creator: reviewer_r4s7

<content>
summary:
The manuscript presents an deep learning based approach to predict protein flexibility and engineer protein sequence towards more flexibility. There are three steps in the manuscript. The first step is to compare metrics that quantify protein flexibility. The second step is to repurpose pre-trained protein language model for flexibility prediction. Here, the authors trained Flexpert-Seq (no 3D structure input) and Flexpert-3D (with 3D structure input) for the regression task. Last but not the least, the authors proposed a framework to train ProteinMPNN with flexibility instructions to generate more flexible protein sequence given the a input 3D structure.

strengths:
The idea of predicting the flexibility of protein given structure or sequence is rather novel. The comparison between the metrics for evaluating protein flexibility is valuable. The framework proposed to inverse fold for better flexibility, Flexpert-Design, can be valuable for specific applications requiring more flexible proteins.

weaknesses:
1. Writing can be improved. For example, in line 79-80, "From these simulations, Root Mean Square Fluctuations (RMSF)." is not a full sentence. Presentation can also be improved. In table 3 and 4, only MD results are bolded, whereas the ANM (MD str.) results are not. Authors can also add arrows to indicate if each metric is the higher the better or the lower the better.
2. The benchmark needs more explanation, especially the flexibility prediction one. It is not clear to me how AF2 and ESMFold are used for flexibility prediction (the regression task). The interpretation of the benchmarking results (table 3 and 4) is also vague.
3. There is not enough elaboration on why more flexible protein is needed. One major contribution of this paper is the Flexpert-Design framework, which tunes ProteinMPNN to generate more flexible protein given the prediction of a flexibility predictor. However, it is not clear to me (from reading the manuscript) the purpose of modifying the flexibility of a protein. The authors can focus more on the motivation. 
4. The Flexpert-Design can introduce bias from Flexpert-3D into the ProteinMPNN model through the flex loss. Also, the correlation between the input structure and the flexibility instruction can be low when the flexibility instruction is artificially increased, i.e. the input structure can never have high flexibility given by the flexibility instruction. In such case, the prediction of both flexibility instructed ProteinMPNN and Flexpert-3D can be unreliable.

questions:
1. How is RMSF calculated? It would be better to note the RMSF equation in the manuscript.
2. Line 152-153, author claims higher pLDDT uncertainty indicates lower flexibility, which is kind of counter-intuitive. Can authors elaborate?
3. When benchmarking the flexibility prediction ability, how are AF2 and ESMFold used? Based on my understanding of the manuscript, the flexibility prediction is a regression task. Did authors attach a prediction head to baseline models? Please elaborate.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645336573

thread_type:
official_review

<thread_object>
creator: reviewer_dnnv

<content>
summary:
In this paper the authors investigate the correlation of various in silico proxies to root mean square fluctuations of protein structures derived from molecular dynamics calculations. They posit that augmenting inverse folding methods for protein design to guide for RMSF as a proxy for flexibility is desirable for design applications. To this end, they introduce Flexpert-Seq, Flexpert-3D, and Flexpert-Design. They demonstrate that supervised models outperform other methods for predicting RMSF from sequence alone, and that they can guide inverse folding design to favor more flexible residues.

strengths:
The paper introduces an interesting problem, explores the problem and relevant dataset before diving into the methodology, and shows that simulated dynamical properties of proteins can be predicted from sequence alone, or sequence and simple structural featurizations, offering important advantages in speed and use-of-use compared to other methods.
The discussion of sequence recovery as an incomplete evaluation of inverse folding methods, and the predominant usage of ProteinMPNN due to its extensive wet lab validation, is appreciated.
The Table 3 and 4 results are nice, clear demonstrations that the authors’ hypothesis is correct, and that a supervised model outperforms more indirect methods like pLDDT and ANM for predicting RMSF values.

weaknesses:
The authors might provide a concise definition of “flexibility” earlier on in the paper. Flexibility could mean many things and be quantified and measured different ways, experimentally and in silico, as detailed in Section 3.1. 
Some discussion is needed of the incompleteness of the methods introduced in Section 3.1. pLDDT and B-factors can be confounded by many other covariates that are not flexibility, and RMSF calculations from MD may not accurately reflect the real world biophysics of dynamic proteins in a physiological context, where interactions with solvent and other biomolecules will affect effective flexibility. 
A small nit, say “MD trajectories” rather than MDs to clarify the source of reference datasets.
I would recommend capturing all the newly introduced methods under subheadings under a Methods section heading, and then going into results. With the current structure I expected to see Results after Section 4, but Section 5 introduces another method.
Without simple baselines (some ideas provided in the Questions below) it is difficult to assess the results presented in the design section.
The Conclusions section restates what was done, which is covered in the intro. Discussion of limitations and future directions would be appreciated.

questions:
Can the authors support the statement in the abstract that accounting for flexibility is the major limitation of current protein design methods? There are many limitations of current methods; flexibility is certainly an important one, but it is not clear that it is a singular limitation.
Can the authors show some failure cases where predictions from Flexpert-Seq do not agree well on the ATLAS dataset? I’m curious if there are certain classes of protein or biophysical features that are difficult to predict well.
The enrichment for A, S, T, and G residues and depletion of bulkier residues on flexibility guidance makes sense - is there a “self-consistency” check that ensures that the guided designs retain the desired predicted backbone structure? How does Flexpert-design compare to running ProteinMPNN and using protein language model likelihoods to identify residues where mutations to smaller, more flexible residues are likely to be tolerated?
Relatedly, per-residue RMSF seems to be a “local” notion of flexibility, which causes a straightforward enrichment of a few specific, small residues. Can the authors imagine ways to include a more non-local measure of flexibility that may induce larger or smaller fluctuations across a biomolecule, without dramatically changing the conditional amino acid distribution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708487808

thread_type:
official_review

<thread_object>
creator: reviewer_i3sf

<content>
summary:
the paper address an interesting problem where one wan to generate novel proteins while accounting for protein flexibility—a feature that is critical for protein function but has proven challenging to model and integrate due to data scarcity, heterogeneity, and the high costs of data generation.  The paper provide benchmark dataset and ways to quantify flexibility. They  adapt the inverse folding model for flexibility, guiding protein sequences towards targeted flexibility in specific regions.

strengths:
1. The study provides a robust review of various protein flexibility quantification methods and assesses the utility of diverse data sources. 
2. The development of Flexpert-Design to adjust flexibility in specified regions is significant, especially for applications in enzyme engineering, drug design, and biocatalysis.

weaknesses:
1. A major limitation is the lack of extensive experimental validation.

questions:
1. Given the relatively small ATLAS dataset, how does the predictor generalize to other protein datasets? Has the model been tested on proteins outside the ATLAS dataset to evaluate robustness?
2. I am wondering if the proteins generated  by this model (due to the flexibility-aware nature of the model) show evidence of functional improvements (e.g., improved stability, interaction specificity), or is the flexibility change primarily theoretical?
3. The Flexpert loss function combines flexibility and sequence objectives with a mixing parameter, θ. How sensitive is the final model performance to different values of θ, and how was the optimal value chosen?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730717966586

thread_type:
official_review

<thread_object>
creator: reviewer_fhjb

<content>
summary:
This paper proposes a method called Flexpert-Design to enable guidance of inverse folding models toward increased flexibility. The authors first quantify protein flexibility and identify data relevant to learning, then train flexibility predictors with protein language models. Finally, they fine-tune inverse folding models to steer it toward desired flexibility.

strengths:
- **New field scenario**: Protein flexibility is rarely considered by current protein engineering methods.

- **Reasonable baseline settings**: The authors arange a reasonable set of flexibility-related baselines.

- **Quantified metric evaluation**: The authors apply several methods to quantify protein flexibility comprehensively.

weaknesses:
- The technical novelty is somewhat limited. While the problem is important, the overall framework is simple and trivial.

- Flexpert-Design uses Gumbel-Softmax trick to enable back propagation. However, this estimation is not extremely accurate.

- Undiscussed choice consideration for the backbone. 

- The experimental results are not comprehensive. Only flexibility-related metrics are shown. Besides, only ProteinMPNN is considered as the inverse folding model.

- Lack of evaluations on the overall protein quality (reliability, plausibility, etc.) for the sequences generated by Flexpert-Design. This can be an important proof of its practical application ability

- No clear explanation for the engagement of the proportion of flexibility-increasing mutations (What is its meaning? What properties and biological representations does it reflect?

questions:
Here are some questions **from weaknesses**.

(1) Since estimation with Gumbel-Softmax trick is not extremely accurate, how does this method compared to other RL-based methods with the L\_flex as the reward?

(2) Why you choose ProtTrans as the backbone, do you try ESM series?

(3) Can the proposed method be applied to any inverse folding models?

And here are questions **about experiment details and considerations**.

(4) In Table 1, how do you match the RMSF of MD with other methods’ metrics (like plddt of AF2 and ESMFold2)? And in Sec. 3.1, you mention that MD is considered an accurate physics-based method. Is that a common sense? If so, why include the whole experiment in Table 1?

(5) How do you get the average PCC upper bound of 0.88? Is that for MD? Does it make sense to set an average value as the upper bound?

(6) How do you prove that the protein sequences designed by Flexpert-Design are natural (or useable) other than with good flexibility only? Since you mentioned that the loss function makes a balance between flexibility and sequence recovery, is it worth sacrificing the overall quality of proteins to enhance their flexibility?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732295565123

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

Thank you very much for your suggestions and insightful comments. We have addressed them in the individual responses to your reviews. 

To answer your questions, we have run experiments to answer the questions about the quality of the proteins generated by Flexpert-Design and to compare the chosen protein language model ProtTrans with ESM-2 as an alternative. 

By the end of the rebuttal period we also plan to upload the modified pdf file, reflecting many of your suggestions on writing and presentation. These suggestions have been so far addressed in the responses to your reviews.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732900132466

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers,

Thank you very much for your insightful comments and constructive suggestions, which helped us to considerably improve our manuscript.

Based on your suggestions, we have improved the presentation of the manuscript by adding several clarifying comments and strengthening the motivation for the studied problem.

Furthermore, during the discussion period, we have performed 5 additional experiments addressing your points:

1. Evaluation of the structure preservation in Flexpert-Design generated sequences using AlphaFold 2 (see Appendix I).
2. Evaluation of Flexpert-Seq and Flexpert-3D on a new independent dataset mdCATH (Appendix F).
3. Qualitative evaluation of Flexpert-Seq and Flexpert-3D (Appendix G).
4. Retraining of Flexpert-3D using ESM-2 as the backbone language model (Response to Reviewer fhjb (Part 1/4))
5. Evaluation of a domain knowledge-based baseline for flexibility engineering (Response to Reviewer DNnV - “Qualitative examination of Flexpert-Seq and Flexpert-3D and a new baseline for Flexpert-Design.”)

Previously, we addressed your reviews in our responses in the discussion. We have now updated the PDF of our manuscript to reflect the changes, with all new text highlighted in blue. The main changes are as follows:

1. We have strengthened the motivation for engineering protein flexibility by providing references to concrete use cases where this strategy has led to experimentally validated improvements in the function of several proteins.
2. We have merged the former Sections 4 and 5 into a single section describing the methods (now Section 4).
3. Due to the page limit, we have moved the former Table 2 (now Table 6) and the accompanying discussion to Appendix E. We have also moved the former Figure 3 (now Figure 8) and the accompanying discussion to Appendix J.
4. In addition to the Appendices E and J, we have added 4 brand new sections to the Appendix (Appendices B, F, G, and I). While Appendix B provides more details on the calculation of root mean square fluctuations from MD simulations and Elastic Network Models, Appendices F, G, and I report on 3 of the 5 new experiments performed during the discussion period.



We have tried to address all points raised in the reviews. If any of the reviewers have further questions, we will be happy to answer them during the extended discussion period.

Thank you very much again for your valuable feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper works towards modifying protein design models (e.g., inverse folding models) to be capable of producing proteins with higher flexibility. The authors propose metrics for measuring flexibility, and then fine tune a protein language model (ProtTrans) with LoRA to predict flexibility from the relatively small amount of annotated data available. They demonstrate the success of their method by showing that, under these metrics, they are able to produce more flexible proteins than standalone ProteinMPNN. The reviewers mostly agree in the end that the problem is interesting, not often considered and well motivated, and the authors do a thorough job of setting up evaluation metrics, creating a supervised fine tuned model for the problem, and then augmenting generative protein design models.