Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces a diffusion-based model for jointly modeling the sequence and structure of complementarity-determining regions (CDRs) of antibodies, conditioned on antigen structure and antibody framework. The method enables the design of CDRs in a wide range of scenarios (from least constrained to most): CDR sequence-structure co-design, fixed-backbone sequence design, or antibody optimization (starting from a known antibody). The approach deviates from prior work notably via the conditioning on the antigen 3D structure and factoring in the side-chain orientations. Experiments demonstrate superior performance in the aforementioned design settings. Lastly, authors curate an extended set of protein structures for model training: antibody-antigen from SAbDab and pseudo-CDRs from the PDB (assimilating loops and the corresponding chains they interact with as proxy antibody-antigen structures).
strengths_and_weaknesses: **Strengths**
- Significance: the ability to design novel antibodies that optimally bind to target antigens, while satisfying other constraints is critical to various antibody therapies
- Originality: this work makes solid contributions in proposing a model that 1) handles certain design settings more naturally (eg, antibody optimization) 2) performs better than existing baselines (based on experiments reported in this work) 3) models the entire CDR structures with atomic precision through the incorporation of side-chain geometry.
- Clarity: the paper is very well written and structured. Figures and notations are very clear. 
- Quality: sound and thorough experimental design. 

**Weaknesses**
- Quality: a few minor claims / points are not fully substantiated (see Questions)
questions: Lines 92-93 "It relies on an additional antigen-specific predictor to predict the neutralization of the designed antibodies, which is hard to generalize to arbitrary antigens". Could you please clarify what you meant here regarding difficulty to generalize Vs your method? Do you mean this approach requires to train a new predictor for each new antigen (Vs your method handles any antigen by construction)? If so, isn't the antigen-specific predictor more likely to provide good results in the focus application it's been design for? Or do you have evidence the more general modeling you are proposing is both more flexible and has higher task performance?

Lines 274-275 -- "We find that using the augmented training dataset could enhance the performance on the test set". Did you train a model without the pseudo-CDR structures to give a sense of how useful this data extension is? 

Tables 1/2: do you have a sense for the relative contributions to performance lift from the different modeling decisions you made over prior baselines (eg., conditioning on antigen structure, diffusion-based modeling, factoring in side-chain orientations, rotation & translation equivariance)

Minor points:
- Line 513 -- dead reference
limitations: - Limitations are very briefly discussed in the conclusion. 
- Potential negative societal impact of the work is not discussed (developed method may be re-purposed more broadly for other protein design tasks which may be malignant).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the problem of antigen-conditional antibody design. While there has been a line of work using ML for antibody design, existing work mostly focused on antigen-independent antibody generation, yet antigen-conditional antibody design has been far more common in practical use cases. This paper makes progress on an important problem.

Two main contributions of this paper are:
1) A new curated dataset of "antibody-like" loops interacting with other protein chains, extracted from PDB.
2) A new approach for antigen-conditional antibody design (joint sequence-structure design) based on diffusion models. 
strengths_and_weaknesses: Strengths:
- (Originality & Significance) This paper considers antibody design conditioned on the antigen 3D structure. This has been an important yet challenging task, and this is one of the first ML approaches to do so.
- (Originality) The authors curated a new dataset of antibody-like complexes (loops in contact with another chain). I believe this dataset will be useful for the research community.
- (Clarity) This paper is mostly clearly written with high quality figures, although the writing could be improved at places (e.g. typos and more accurate language about CDRs).
- (Quality) The empirical results seem sound, demonstrating improvements in three different antibody design tasks.

Weaknesses:
- (Significance) While the paper takes a step forward in antigen-conditioned antibody design, there is still a notable gap between the scenario in the paper and the real use case: In a real use case, we do not know the relative orientation between the antibody and the antigen, while in the design tasks in the paper the orientation is given based on known antigen-antibody complexes. The paper does address this concern in Section 4.5 by using HDOCK to dock the antibody template as a precursor step. However, this is only done for one antigen. in my view, this paper would be a lot stronger if the authors can show the design performance in Sections 4.2-4.4 both with and without known bound complexes. 
- (Quality) This paper would also be strengthened by more ablation studies on the relative merits of different model components, as well as by the inclusion of simpler baselines.
- (Quality) The split in this paper is based on CDR sequence identity, and not based on antigen structures. It would be helpful to at least show statistics of antigen overlap (e.g. RMSD) between train and test splits.
questions: Major questions/suggestions:
1. Among the three components (sequence, Ca, orientation), which one is more important to model? Ablation studies would make the paper a lot stronger.
2. The GNN baseline in all results is antigen-independent? If so it'd be greatly appreciated to mark that (e.g. add a column in tables to say antigen dependent/independent), because otherwise it's misleading to directly compare the two.
3. How many antigens are shared between the train and test splits? It would be important to at least show statistics of antigen overlap (e.g. min RMSD or % overlap) between train and test splits.
4. How would the performance degrade if using HDOCK instead of known complexes for all the main results (sections 4.2-4.4)? 
5. Comparison to antibody structure prediction methods (e.g. IgFold) for RMSD?

Other questions:
1. Does "side chains" mean "Cb" specifically in the paper? It would be helpful to clarify whether it's all atoms in side chains or just Cb.
2. Compared to using equal probabilities for all AAs in Eq. 1, would it be helpful to use e.g. BLOSUM or other priors?
3. How is the posterior derived from Eq. 1?
4. Is minimizing the expected KL (Eq. 4) equivalent to maximum likelihood estimation (MLE)? 
5. It'd be helpful to include simpler baselines, e.g. IMGT alignment consensus of antigen-specific antibodies. Although I expect simpler baselines to be worse, it'd still be helpful to put results into context.
6. It would be more convincing for antibody optimization to also show results on antibody DMS datasets -- unless it's hard to evaluate likelihood in diffusion models?
limitations: See above ("weaknesses") for limitations. The authors did acknowledge the main limitations and I appreciate that as a reader.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a diffusion model for jointly generating the CDR sequence and structure of an antibody conditioned on its framework regions and a target antigen. Unlike existing methods, the presented method enables i) conditioning the generation on the antigen structure instead of only framework regions, and ii) predicting side-chain orientations. Lastly, the paper also presents a dataset of experimentally validated antibody structures iii) augmented by pseudo-antibody structures derived from the structure of non-antibody proteins.
strengths_and_weaknesses: ## Strengths
* The paper is mostly clearly written
* The use of a diffusion-based model for generating antibodies is new
* The application of antibody design is important, e.g. for drug development

## Weaknesses 
* The benefit of the diffusion-model over alternative generative models is not evaluated experimentally
* The accuracy of predicted side-chain orientations without refinement by the Rosetta packing algorithm is not evaluated experimentally
* The benefit of augmenting the dataset by pseudo-antibody structures is not evaluated experimentally
* Important baselines are missing

questions: ## Questions
1) Does the diffusion model perform better than alternative models such as an autoregressive model (e.g. Jin et. al) or Transformer? Can you show this by ablation? Only change the architecture while using the same (augmented) dataset for training and tuning hyper-parameters appropriately.

2) How does the proposed method compare to a simple sequence-based, non-iterative, model that generates CDR regions conditioned on the sequence (not the structure) of framework regions and the antigen, and predicting the structure with AlphaFold?

3) Please compare to Jin et. al (http://arxiv.org/abs/2110.04624) also in section 4.2 and section 4.4. by training the model on the same augmented dataset of tuning hyper-parameters appropriately.

4) Please show by ablation the effect of i) conditioning on framework regions and ii) the antigen structure.

5) What is the training and inference time of the diffusion model compared to an autoregressive model (e.g. Jin et. al)?

6) How does the model scale to long sequences, which is important for generating long CDRs or also framework regions?

7) Does augmenting the training dataset by pseudo-structures improve the performance of the diffusion model and baseline model? You are referring to results in the appendix. However, I did not find any results. Since this is a major contribution of your paper, results should be in the main text.

8) What is the accuracy of predicting side-chain orientations before refining them by Rosetta?

9) What is the influence of applying the Rosetta packing algorithm and AMBER on the final structure? Does it change the ranking of methods?

10) How do you apply the method to double-stranded antibodies? Do you generate the two different chains independently? 

11) Why does IMP% in Table 3 decrease with increasing t?

12) What is the message of figure 3 and figure 4? Consider showing these figures in the appendix. Otherwise, interpret samples and also show samples of baseline methods.
limitations: Yes
review_title: null

Meta-review: metareview: This is a very exciting and timely paper that eleganlty enables CDR sequence-structure co-design, sequence design given a certain backbone, and antibody optimization. 

The reviewers and AC all appreciate the extensive feedback provided by the authors and the additional studies included in the supplements.  We strongly encourage the authors to also incorporate in their manuscript certain points made in their feedback. In particular please include comments to 
- contrast the proposed approach with (i) the work "Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design" and (ii)neutralization prediction approaches
- highlight the  limitations of docking algorithms and the pertinent future work direction of generating antibody orientations for antigens
- clarify various points, such as description of Figure 1.
confidence: Certain
Decision: Accept