Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method for protein sequence and structure codesign. Current methods use either autoregressive or diffusion-based models, which the authors claim are computationally expensive and produce suboptimal solutions. Sequence generative models can produce good designs, but do not model protein structures. The authors address the above issue with their method ProtSeed. ProtSeed uses a triangular-aware encoder to learn geometrical constrain from context features and roto-translation equivariant decoder that iteratively improves protein structure followed by an MLP to decode the amino acid identities from the structure. The author performed experiments on the structural antibody database and 2 protein design benchmarks.
strength_and_weaknesses: Strengths:
1. Good introduction and well-presented related work focusing on key aspects.
2. Good evaluation results compared to previous methods
3. A novel approach for solving the structure and sequence of protein and study for the de novo protein design

Weaknesses:
1. Lack of specific hyperparameter and training details
2. No mention that code and trained models will be released
3. The application of the antibody CDR design task is clear, but the relevance of the other two tasks is murkier. When would detailed structural specifications be available, but without an existing sequence that forms that fold?
4. The authors note a difference between their evaluation and the published scores for Diffusion, GNN and RABD models (Table 1). However, the concern remains that the Diffusion model shows significantly better RMSD scores compared to the scores in this paper. What is the source of this discrepancy?
5. ProtSeed improves over previous methods in terms of PPL, RMSD, and AAR. However, because the task is to design functional proteins, these metrics may not reflect true model performance, because a model with higher RMSD may, in fact, produce (more) functional proteins while also being worse at predicting structure. It would be helpful to provide some additional, functional-oriented, evaluation. For example, by comparing the protein-protein interaction patterns and charge distributions between the designs and the ground truth.
6. AAR is surprisingly low for all of these methods, including ProtSeed. If only 40% or less of amino acids are correctly recovered, how confident can we be that any of these methods actually produce good designs?
7. I’m not convinced by AF2 structure prediction as an evaluation. It isn’t surprising that AF2 would predict the designed proteins to fold as intended, because all of these models are trained on the same PDB data and, therefore, probably have the similar pathologies.

Other minor comments:
1. Figure 4. – To keep a uniform style among all figures, a, b c marks could be moved outside of the figures as in Fig. 1-3
2. Table 3 – highlight best results as in table 1 and 2

clarity,_quality,_novelty_and_reproducibility: The manuscript is well written, the method is reasonably novel, and the empirical performance is strong.
summary_of_the_review: A good paper about an interesting method. Hyperparameters and training details need to be discussed. The code and models should be released to facilitate reproduction and future work. Some discrepancies with prior work should also be addressed.

What would improve my score: include hyperparameter and training details. Resolve discrepancy with prior work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a framework for protein sequence and structure generation. The key idea is to involve the structure, sequence, and context together for encoding and decoding. The key modules are trigonometry-aware encoder and roto-translation equivariant decoder in a one-shot way. The experiments are conducted on three different tasks and the results show the ProtSeed is effective. 
strength_and_weaknesses: Strength:
1. The paper works on the protein sequence and structure co-design task, which is important and hot recently. 
2. The method in this work is well-motivated and the modeling way is reasonable and solid. 
3. The results show strong improvements over previous works on different tasks. 

Weaknesses:
1. Most of the paper writing is good. However, in some claims and places, they are not appropriate enough. The authors mainly claim that the specific antibody prediction method, e.g., RefineGNN, is not general and is in a specific domain. However, in my view, their method is general, but they only show the application in antibodies. Besides, the sequence, structure, and rotamers sequentially generation are slow as the authors claimed, but the inconsistent and fanciful proteins may not be the truth. A good sequence can help structure, and a good structure can also help sequence, iteratively update sequence and structure is hard to say a negative point or "issue" as the authors mentioned. 
2. From an overview of the technique, the trigonometry-aware encoder and the equivariant decoder are mostly taken from Alphafold2, though some modifications are taken. In terms of this, the novelty is hard to say as big as the authors claimed. Alphafold2 also worked on the protein (though they don't predict the sequences). 
3. There are some places in the method questionable. (a). On the decoder side, within the encoder output, the decoder continuously updates the context feature $m$ and $z$, where the interactions of the context amino acids are involved in the encoding way. However, the update of the amino acids that need to predict seems to be not so clear and reasonable. For example, the $z$ feature is only performed between the amino acids in the context, but no consideration between the predicted amino acids (if I understand correctly of $z$). (b) The description of SeqIPA is not well presented. Though in supplementary (please append this as Appendix after the main text instead of supplementary) the authors try to explain, but the formulation is not shown. When compared with the original IPA, it is still not clear how to calculate with the five different inputs. (c) In eqn. 7 and 8, why the MLP takes m(t+1) and m(0) rather than updated m(t+1)? Is $\lambda$ necessary? Also more explanations may need for Algorithm 1. (d) In eqn.11, how $x_{ij}$ is calculated since in previous descriptions they are all based on the amino acid level instead of the atom level, this should be noted.  
4. One disadvantage of this method is that they require the context of $m$ and $z$, amino acid and pair features. This somehow concerns me about the practice in real scenarios. The hard request of these two features may not be friendly for some tasks. For example, how to generate from a general distribution. The authors can provide more discussions. 
5. Though the decoding speed is much improved. I would like to also know more about the training cost, since the method is much more complex than previous ones. For example, small encoders are also involved in the decoder side. Besides, as for the experimental comparisons, since the compared baselines are provided by the authors, the code should be published (also the implemented baseline code). Also, the ablation study is not provided, it is hard to see the contribution of each part. 
clarity,_quality,_novelty_and_reproducibility: Please see above. 
summary_of_the_review: N/A
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - Protseed introduces a model that can jointly model the sequence and structure information in order to do sequence design.
- Protseed tests against a suite of state of the art protein design benchmarks, like AlphaDesign, Diffusion, and GVP-GNN, on fixed backbone sequence design and antibody CDR design tasks.
- The model is able to get state of the art results and beat competitors on most tasks.
strength_and_weaknesses: Strengths:
- Paper uses well accepted metrics and implements recent results to make strong comparisons.
- SeqIPA is an interesting extension to the normal IPA module
- The results seem to be state of the art, beating out recent design models, at least on join sequence and structure design.

Weaknesses:
- The comparisons to fixed backbone design is weak, recent models like ESM-IF using larger datasets or prot-MPNN exists, and some effort should be made to compare to those.
- The work lack ablations, it's unclear which part of the model changes give such a large improvement.
clarity,_quality,_novelty_and_reproducibility: This paper is written clearly, and presents a novel model on interesting protein design tasks. Maybe future work can confirm whether the model can scale beyond the smaller dataset trained on.
summary_of_the_review: Because of the strong empirical results and good benchmarking, I would recommend an accept. Comparing to more recent models and on larger datasets would make this paper much stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an iterative method for protein sequence-structure co-design. In each generation step, the model first uses IPA network to update the 3D structure ($C_a$ coordinates and the orientation of its frame). It then predicts the amino acid types at all positions. The proposed method is applied to antibody CDR design (co-design) and protein design (fixed backbone and co-design) applications and showed state-of-the-art performance.
strength_and_weaknesses: Strength:
1. The proposed method improves upon the previous co-design method (Jin et al 2021) by generating all the amino acids in an iterative refinement process rather than left-to-right autoregressive process.
2. The inference cost is lower than diffusion models which requires thousands of generation steps.

Weakness:
1. For protein design tasks, the term "co-design" is somewhat misleading. The method does not generate protein structure (and sequence) from scratch. It is conditioned on the contact map + secondary structure, which is a fuzzier representation of the backbone structure but it contains many information of the structure. The contact map needs to be taken from an existing structure, although it can be specified manually for certain types of structures.
clarity,_quality,_novelty_and_reproducibility: The method description is quite clear. The proposed method is conceptually similar to previous co-design methods, with small but useful modifications in the generation step. The major contribution, perhaps, is demonstrating its utility over general proteins rather than antibodies. 
summary_of_the_review: The paper is a good contribution to ML protein engineering field and I vote for acceptance. The methodology is similar to previous methods, but with useful modifications (e.g., iterative generation of amino acids rather than autoregressive). Moreover, it demonstrated its utility over general proteins (not just antibodies) and therefore has a larger impact.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this work, the authors proposed a method, ProtSeed, that iteratively perform protein sequence and structure design guided by desirable coarse-grain features, through integrating IPA approach and predicting amino acids at positions across the structure in a one-shot fashion within each step to achieve an efficient overall process. The authors demonstrate their approach to have good performance in antibody design and related protein design tasks. Their method could improve over state-of-the-art in the field, and brings new ideas. Nonetheless, some of their description is not clear and more extensive discussion is needed. The authors were able to revise their paper during discussion following the reviewers' suggestions which helped to improve the quality of this work.
justification_for_why_not_higher_score: Please see comments above from AC-reviewer meeting.
justification_for_why_not_lower_score: The authors were able to improve this work during discussion per reviewers' comments.
Decision: Accept: poster