ProteinWeaver: A Divide-and-Assembly Approach for Protein Backbone Design

Authors: Yiming Ma, Fei YE, Yi Zhou, Zaixiang Zheng, Dongyu Xue, Quanquan Gu
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: divide and assembly, protein backbone design, domain assembly, long-chain proteins

Abstract:
Nature creates diverse proteins through a 'divide and assembly' strategy. Inspired by this idea, we introduce ProteinWeaver, a two-stage framework for protein backbone design. Our method first generates individual protein domains, then employs an SE(3) diffusion model to flexibly assemble these domains. A key challenge lies in the assembling step, given the complex and rugged nature of the inter-domain interaction landscape. To address this challenge, we employ preference alignment to discern complex relationships between structure and interaction landscapes through comparative analysis of generated samples. Comprehensive experiments demonstrate that ProteinWeaver: (1) generates high-quality, novel protein backbones through versatile domain assembly; (2) outperforms RFdiffusion, the current state-of-the-art in backbone design, by 13% and 39% for long-chain proteins; (3) shows the potential for cooperative function design through illustrative case studies. To sum up, by introducing a 'divide-and-assembly' paradigm, ProteinWeaver advances protein engineering and opens new avenues for functional protein design.

Decision: desk rejected
Reviewer scores: 3, 3, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730477669061

thread_type:
official_review

<thread_object>
creator: reviewer_ttly

<content>
summary:
This paper proposes a domain assembly framework, coupled with preference alignment, to flexibly assemble multiple protein domains, achieving better generation performance on long proteins compared to existing protein backbone design models.

strengths:
The paper is generally clear and easy to understand. The concept of flexibly folding multiple domains is an interesting application of recent protein diffusion models and could be very useful for designing more complex functional proteins.

weaknesses:
Besides the limitations mentioned in Section 5, one major concern is the fairness of comparisons with other protein backbone models. Specifically, while other protein design models such as RFDiffusion and Chroma generate proteins from scratch (unconditional generation), ProteinWeaver performs a conditional generation task with substantial “local” information (individual domain structures) provided. This combines with observation that individual structures do not shift substantially (with mean alterations less than 2A as shown in Figure 10), suggesting that ProteinWeaver tackles an easier task compared to other protein backbone models and thus it is not a fair comparison with other models such as RFDiffusion and Chroma. In addition, details on the generation of samples that are used for comparison with other backbone design models are lacking, for example, what existing backbone design models are used for generation, how many domains are generated by each model, how the generation is decomposed (percentage of each decomposition) and so on. These details are also crucial for understanding the novelty argument in Section 3.3; in particular, to what extent is the improvement in novelty a result of higher novelty in the underlying domain pool, or a result of the assembly process?

Considering the limited technical novelty in model architecture (FrameDiff with minor adaptations), it would be helpful if the authors could provide more ablation analysis on the preference alignment component. For example, what is the effect of different reward assignments? How would the performance change if the preference considers more than just scTM, but also interface scTM and other metrics? How do various factors on the alignment dataset construction (for example, size, diversity in domain selection) affect model performance?

questions:
- In Appendix A.5, the description on the computation of scTM and scRMSD differs from the diagram in Figure 8. Could the authors clarify further on this? Is the ProteinMPNN-ESMFold pipeline used for refolding the backbone or the interface structure? 
- In Figure 3, why does assembly using PDB domain perform better than assembly using CATH domain? For assembly using PDB domain, does the model perform assembly using domains from the same PDB structure, leading to better performance over assembly using CATH domains? Moreover, for assembly with synthesized domains, does the model perform assembly using domains from the same synthesized structure? Why is the performance of assembly using RFDiffusion-generated domains better than that of PDB and CATH domains? How many structures are generated for each method? This same set of questions also applies for Figure 6.
- In Figure 1F, what is the scale on each axis?
- In line 257-265, what are the criteria used for deduplication and domain selection?
- In Section 3.5, it would be useful if the authors could provide further details on the refolding of domains. Specifically, to what extent is the structure altered in the refolding process? What is the self-consistency performance for initial domains (directly split from PDB) and refolded domains? Does the domain in the final assembly resemble more closely to the initial domain or the refolded domain?
- Could the authors provide further details on the best-of-3 filter? In Figure 7, the performance of ProteinWeaver without alignment and bo3 is close to the performance of ProteinWeaver. Interestingly, the performance of ProteinWeaver without alignment and bo3 seems to outperform the performance of ProteinWeaver without alignment (but with bo3 filter). Do the authors have any intuition on these observations? Moreover, the margin of benefits that alignment provides seems to be smaller than illustrated in Figure 6. It would be helpful if the authors could also show the performance of ProteinWeaver in Figure 6.
- For comparisons with backbone design models, it is worth noting that RFDiffusion is trained with a crop size of 384 residues, which is smaller than that used for training ProteinWeaver. This could give ProteinWeaver an edge in generating longer proteins since the training dataset consists of more long proteins. In addition, how does the model perform compared to Proteus (https://www.biorxiv.org/content/10.1101/2024.02.10.579791v1), which exhibits better generation performance on long proteins than RFDiffusion?
- For pretraining, does the model start from scratch or pretrained FrameDiff weights? Also, it would be helpful if the authors could provide further details on the training and sampling time: How long does it take to pretrain the model and how many epochs does it run? How long does it take to align the model?
- Figure 9: How does ProteinWeaver perform here?
- How does ProteinWeaver perform as the number of domains increases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730623567356

thread_type:
official_review

<thread_object>
creator: reviewer_zjpl

<content>
summary:
The paper introduces ProteinWeaver, a method for protein design. The authors propose a two step “divide-and-assembly” approach, first generating local structures and then assembling them using a diffusion model which is fine-tuned with pairwise comparisons. They evaluate their model empirically and provide comparisons to other state-of-art approaches. They find that ProteinWeaver outperforms other approaches, especially for longer protein chains.

strengths:
- The paper proposes a novel approach (to my knowledge) and evaluates the method empirically with promising results that should be considered for publication.

weaknesses:
- The paper is mostly well written, however, due to a lot of condensed information, it can be hard to follow at some times and leaving a lot of details in the appendix.
- The authors did not perform a statistical analysis of their results. For example, in Table 3 some numbers are rather close. A test for statistical significance could strenghten the results.

questions:
- How does ProteinWeaver generalize in the wild? Did your method find any new interesting protein structures applicable to real problems?

Minor Comments:
- Figure 3 B: Rfdiffusion should have an uppercase F in the right plot.
- Sect. 2.3 could use some proofreading. The sentences in that paragraph are hard to read. E.g. “For each domain, we use f generates corresponding domain.”
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644159588

thread_type:
official_review

<thread_object>
creator: reviewer_7cwx

<content>
summary:
The paper proposes an innovative framework for designing protein backbones. Inspired by natural protein evolution, the method divides the protein design problem into two stages: domain generation and assembly. ProteinWeaver demonstrates improved performance over existing methods, particularly for longer protein chains, and shows potential for designing proteins with cooperative functions.

strengths:
- The divide-and-assembly strategy is a creative and effective method for protein backbone design, offering a new direction for protein engineering.
- The method shows significant improvements over current state-of-the-art, especially in the design of long-chain proteins, addressing a critical need in the field.

weaknesses:
- The paper could provide more insight into how robust ProteinWeaver is to variations in protein structures, especially those with atypical domain arrangements or those that deviate significantly from known protein families.
- While ProteinWeaver shows strong performance for long-chain proteins, it would be beneficial to see how it performs across a wider range of protein lengths, particularly shorter chains where domain assembly might be less complex but still challenging.

questions:
See weaknesses I've listed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675783724

thread_type:
official_review

<thread_object>
creator: reviewer_tr5n

<content>
summary:
This work proposes a 2-stage protein backbone generation framework that assembles multiple domains using conditional diffusion model trained on diagonally concatenated distance map of individual domains. The authors perform preference alignment on scTM to further increase the quality of designs.

strengths:
1. First propose 2-stage generation framework for long protein design, which enables the generation of multiple (interacting) domains.
2. Suggest that generation of long proteins fall short due to lack of modeling inter-domain interaction; Reports the number of domains and interface scTM vs. native proteins. It is convincing and interesting argument.

weaknesses:
I believe the paper’s contribution can be divided into two aspects: a 2-stage generation approach and an alignment.

1. **2-stage generation approach:** **The results doesn’t really show that proposed 2-stage approach is effective.** To truly show that 2-stage assembly approach is better than 1-stage approach, I would compare ProteinWeaver vs scTM-aligned RFDiffusion, or ProteinWeaver w/o alignment vs. RFdiffusion. Any model can be additionally aligned to scTM, and it is not surprising that aligned model shows better results. 
    1. Appendix Table 3 shows the ProteinWeaver w/o alignment vs. RFdiffusion, and I think it is performing worse than RFDiffusion. sc metrics are only slightly better but diversity is much lower. It just seems like a tradeoff between diversity and designability.
2. **Alignment: Technical novelty and contribution is limited.** I believe preference alignment is already widely used in protein design tasks, and is more like a supplementary analysis within broader papers (as in FoldFlow2 [1]) not a full conference paper, especially when you don’t extensively discuss on an alignment method and draw meaningful discussions from them. The discussion on alignment method is weak, for example,
    1. Ablation study only compares with SFT. But in most cases SFT is precondition for later alignment not the standalone alignment method. How does SFT + PPO performs? [2] 
    2. SPPO and many preference alignment methods are tailored to model human preference, i.e. it is hard to decide an absolute, objective ‘preferredness’ of a response, so you bypass that via pairwise comparison. Can you clarify the reasoning behind the preference alignment when we have explicit reward function (i.e. scTM) we want to optimize?
3. **Results**: Even with this additional alignment, performance improvements are marginal. There is big decrease in diversity, and the designability cannot match those of the native PDBs. 
4. **Unclear writing**:
    1. line 199-200: 
        
        > To assemble generated domains, we utilize FrameDiff, the SE(3) diffusion model to pretrain the domain assembly model.
        > 
        
        My understanding is that ProteinWeaver uses FrameDiff architecture for domain assembly model. Is this correct? Could you revise this sentence and Appendix (line 690-693) to make it clear that ProteinWeaver uses FrameDiff architecture? Also do you use pretrained weights of FrameDiff?
        
    2. line245-246 Is it correct to say maximizing p(S) is different from maximizing the quality of generated structures? Isn’t native PDBs have high scTM and low scRMSD?

[1] Huguet, Guillaume, et al. "Sequence-Augmented SE (3)-Flow Matching For Conditional Protein Backbone Generation." arXiv preprint arXiv:2405.20313 (2024).
[2] Ethayarajh, Kawin, et al. Human-centered loss functions (halos). Technical report, Contextual AI, 2023.

questions:
line 245 typo maximine → maximize
</content>
</thread_object>

</thread>
<thread>
creation date:
1730803359446

thread_type:
official_review

<thread_object>
creator: reviewer_eghm

<content>
summary:
The paper introduces ProteinWeaver, a two-stage framework for protein backbone design. In the first stage, individual protein domains are generated, and in the second stage, these domains are assembled using an SE(3) diffusion model. Additionally, the framework employs preference alignment to discern complex relationships between structural and interaction landscapes, aiming to enhance performance.

strengths:
The proposed method outperforms Chroma, RFDiffusion, FrameDiff, and FrameFlow in generating long backbone structures.

weaknesses:
1. The comparative evaluation is insufficient. While the authors emphasize that their method achieves superior performance on long proteins compared to Chroma and RFDiffusion, other recent methods—such as MultiFlow [1], CarbonNovo [2], Proteus [3], and FoldFlow2 [4]—have also demonstrated better performance on long proteins. To substantiate their claims and highlight the main contributions, the authors should include these methods in their comparisons.
2. The Related Work section does not adequately reflect the current literature on protein backbone generation. At a minimum, the methods mentioned in Weakness 1 should be discussed to provide a comprehensive overview of recent advancements.
3. I don't think the basic ideas are well-motivated. The proposed method assumes that any domains generated in the first stage can be assembled, which may not hold true concerning both function and structure. 
4. The methodology lacks sufficient novelty. It consists of two main components: a diffusion model for assembling domains and preference alignment to discern structural and interaction landscapes. Both components appear to be straightforward applications of well-established methods in the literature, raising concerns about the originality of the approach.

[1] Campbell et al. Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design. Proceedings of the 41st International Conference on Machine Learning. https://proceedings.mlr.press/v235/campbell24a.html

[2] Ren et al. CarbonNovo: Joint Design of Protein Structure and Sequence Using a Unified Energy-based Model. Proceedings of the 41st International Conference on Machine Learning. https://proceedings.mlr.press/v235/ren24e.html

[3] Wang et al. Proteus: Exploring Protein Structure Generation for Enhanced Designability and Efficiency. Proceedings of the 41st International Conference on Machine Learning. https://proceedings.mlr.press/v235/wang24bi.html

[4] Huguet et al.  Sequence-Augmented SE(3)-Flow Matching For Conditional Protein Backbone Generation. https://arxiv.org/abs/2405.20313

questions:
1. Please address the points in Weakness section.

2.  In the domain assembly generation stage, the authors indicate that a linker is inserted between adjacent domains, which may increase the overall protein length. How does the method ensure that the generated protein aligns with the target length?

3. What is the split configuration used during testing? Does the performance rely on this configuration? For example, assembling a 200-residue domain with a 300-residue domain for a target protein of 500 residues might yield different results compared to combining a 100-residue domain with a 400-residue domain.

4. Section 2.2 mentions that multi-domain proteins are filtered out, despite the method’s aim to model domain-domain interactions. What is the rationale behind excluding multi-domain proteins when they are essential for studying inter-domain interactions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732286097317

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We are pleased to hear any feedback from all reviewers before discussion deadline.

comment:
Dear Reviewers, ACs, and SACs,
We would like to sincerely thank all the reviewers for their efforts and valuable suggestions. We have made every effort to address the reviewers' concerns, including the following:
1. Provided sufficient evaluation on more baseline methods.
2. We have expanded the related work section in our revised manuscript.
3. Included rational explanations highlighting both the scientific and methodology contributions of our study.
4. Corrected minor presentation issues.
We appreciate everyone's time and effort in providing insightful feedback, which has greatly helped us improve our manuscript. We have revised the paper to incorporate many of the reviewers' suggestions and comments, and we are truly grateful for your contributions.
We welcome any further feedback during the discussion phase!
Thank you once again!
Best regards, Authors
</content>
</thread_object>

</thread>


Metareview: