Kati8/ICLR_2023_2025 huggingface dataset:
authors: Yufei Huang
authors: Yunshu Liu
authors: Lirong Wu
authors: Haitao Lin
authors: Cheng Tan
authors: Odin Zhang
authors: Zhangyang Gao
authors: Siyuan Li
authors: Zicheng Liu
authors: Yunfan Liu
authors: Tailin Wu
authors: Stan Z. Li
reviews: 
Comments: The paper studies the problem of motif scaffolding, which requires the construction of scaffold structure stabilzing motifs corresponding to desired functions. Towards this goal, the work leverages spatial context in order to guide the generation process. Motif scaffolding is interpreted as a Geomtric Inverse Design problem which leads to the formulation of Evolution-Via Reconstruction (EVA). EVA is a sampling-based coupled flow framework which utilizes motif-coupling in its prior as spatial contexts. This leads to the construction of a straighter probability path wherein generative directions are aligned with the sampling steps. EVA improves sampling time by a signficant margin while presenting competitive performance on RFDiffusion benchmark and in-silico vaccine design task.
Strengths: * The paper is well written and easy to understand.
* The paper tries to address an important and challenging problem from a geometric perspective.
Weaknesses: * **Benefits of Motif Geometry:** My main concern is the benefits of motif geometry. Addition of motif-aligned prior and motif-interpolated posterior only benefits the model at sampling time. If we are using motif-coupled information with a rich prior, this should improve sample quality (especially designability) as well. However, empirical evaluation on benchmarks in Tables 1, 2 and 3 shows that additiona of motif-coupled information deteriorates designability of sampled complexes. What benefits does motif information provide for sampling novel complexes aside from faster sampling? How can motif information aid in the designability of sampled complexes?
* **Improvement over OT-Flow:** The work presents empirical evaluation for motif-coupling on the RFDiffusion benchmark and an in-silico vaccine design benchmark. In most experiments on the RFDiffusion benchmark, backbone models (such as OT-Flow) are better for designability and success rates. What improvements does EVA offer over the backbone models? If the main contribution of the work is the addiiton of geometry information and motif-interpolation, then do these design choices benefit over prior backbones?
* **Unconditional Generation:** Figure 3 and Table 2 present results for the backbone flow models. I am struggling to understand the presence of these results in the main text. If these experiments are not directly related to EVA, could they be better placed in the appendix? From my understanding, these results can be understood as ablations over backbone models in determining the choice of flow for motif coupling.
* **Scallbility:** While EVA demonstrates consistent improvements on the RFDiffusion benchmark, it still remains unclear whether the method is scalable for real-world applicability. Table 1 demonstrates that EVA performs best on smaller lengths of residues (< 100). Intuitively, motif geometry is informative only for smaller complexes. Can the authors comment on the use of motif coupling for larger residue lengths? How can EVA be adopted/improved for larger complex sizes?
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: This study models motif-scaffolding as a Geometric Inverse Design task and proposes a sampling-based flow framework, Evolution-ViA-reconstruction (EVA), that incorporates spatial information of the motif. The reconstructed motif in the early stage guides the whole generation process by aligning generative directions with motif reconstruction. Experimental results show that EVA generates desirable proteins with significantly reduced time.
Strengths: - The idea of modeling motif-scaffolding as a geometric inverse design task is intriguing, and addressing the trade-off between generation and reconstruction is important in this context.
- Directly utilizing the spatial information of the target motif and guiding the generative process through motif reconstruction seems practical.
- The inclusion of real-world tasks enhances the applicability of the proposed method.
Weaknesses: My primary concern is that the experimental results do not sufficiently demonstrate the advantages of the proposed method over existing approaches, particularly RFDiffusion, for the following reasons:

- The results are reported based on a single run, and the performance differences are not significant enough to conclusively determine which method performs better.
- In Table 1, RFDiffusion, a training-based method, appears to perform better with a computation time of 61 seconds. These computation times seem acceptable, especially when considering subsequent synthesis processes.
- The mRMSD scores of EVA are not higher than those of other methods in Table 1 (≥100) and Table 2, even though accurate reconstruction is crucial in this work.

Additionally, the manuscript could benefit from improved consistency in terminology, clearer notations, and enhanced coherence. Please refer to minor comments for further suggestions.

- There is inconsistent use of terms regarding the trade-off, such as "trade-off between generation and guidance," "generation vs. reconstruction trade-off," and "Evolution Verse Adoption trade-off" (mentioned in the Appendix).
- The term "Evolution via adaptation" appears in the appendix.
- Section 5.4 discusses the fast computation of EVA rather than directly addressing Question 2: "Can EVA generate designable proteins of various lengths with accurate motif reconstruction?"


### Minor comments

- Please use \citet for textual citations. For example, (Wang et al., 2021) → Wang et al. (2021) in line 114
- It would also be helpful to provide specific information about the appendix, such as "Appendix A," rather than just saying "Appendix."
- Please place the table captions above the tables, not below them.

### Typos
- Line 58: the → The
- Line 167: : → .
- Line 367: I guess it is pi bar, not pi hat
Recommendation: 6
Reviewer confidence: 2
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The paper introduces a flow-based generative model guidance method named EVA for motif scaffolding in protein design. EVA claims to achieve faster inference times while maintaining performance comparable to SOTA methods. Unlike previous approaches that rely on approximating the distribution of the motif given the prior using Tweedie's formula, EVA approximates the distribution of correct residue frames conditioned on both the prior and the motif. Overall, the method presented seems to be novel; however, I found the paper to be a bit difficult to follow, particularly the correspondence between Figure 2, Algo 1, and the text body. Please see the questions below.
Strengths: 1. Faster inference, on-par results with SOTA.
2. Does not require retraining.
Weaknesses: 1. Requires running the pre-trained model inference at every time step. Would like elaboration on this.
2. Difficult to following the correspondence between the paper's main body, Figure 2, and Algo 1.
3. Notation a bit confusing at times.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 2