Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors propose a E(3)-equivariant diffusion probabilistic model for both unconditional and conditional sampling of protein backbones, which can be used for the motif-scaffolding problem. Particularly, the conditional sampling is implemented as a particle filtering algorithm, where scaffolds that are more consistent with the motif are prioritized. Authors demonstrate that the proposed method generated valid scaffolds for the given motif, as verified by AlphaFold2’s predicted structures. Experiments on unconditional sampling indicate that this model captures the underlying distribution of protein backbones.
strength_and_weaknesses: Pros:
1. The formulation of motif-scaffolding problem as a conditional sampling process is novel, and authors propose a single diffusion model that can be used for both unconditional and conditional sampling to realize such motif-based scaffold design.
2. The re-weighting scheme in the particle filtering algorithm is based on the likelihood assessment of the diffusion model, which can better guide the conditional sampling process towards the promising direction.
3. Clear discussion on the modeling and data limitation of the proposed method.

Cons:
1. During the training of ProtDiff, the diffusion model of protein backbones, 3D coordinates are represented in nanometers, rather than Angstrom, so that the spatial scale is similar to an isotropic Gaussian with unit variance. I am not sure whether such scaling is sufficient to ensure that x^{T} at the last step is indistinguishable with an isotropic Gaussian, which is one of the fundamental assumptions in DDPM.
2. In Section 4.2, authors state that the proposed SMCDiff can provide arbitrary accuracy for conditional sampling, as guaranteed by Proposition 4.1. However, such proposition is based on the assumption that p_theta **exactly** matches the forward diffusion process, which might be too strong and even unrealistic.
3. In Section 5.1, authors mention that when prediction protein structure with AF2, no multiple sequence alignment (MSA) is used. However, as shown in several single-sequence PLM-based protein structure prediction papers, AF2 w/o MSA input works inferior to such PLM-based structure prediction, possible due to the gap between training and inference inputs. Authors may need to consider feed MSA into AF2 for more reliable predictions, or directly use PLM-based methods (e.g., ESM-Fold, OmegaFold and HelixFold-Single) for in silico evaluation.
4. In Section 6, authors state that only using 3D backbone coordinates as inputs to ProtDiff is the main cause that ProtDiff is reflection symmetric and may generate left-handed helices. However, I think the EGCL-based network architecture may also be one of root causes (even the biggest one), since EGCL only takes pairwise distance as spatial information, which is reflection invariant. In other words, ProtDiff cannot distinguish one structure and its mirrored counterpart by design, and this won’t be resolved even full-atom 3D coordinates are taken as inputs.
clarity,_quality,_novelty_and_reproducibility: Clarity: Mostly satisfying. Section 4 may need further refinement for easier understanding.

Quality: Good.

Novelty: The formulation of motif-scaffolding problem as a conditional sampling process based on denoising diffusion probabilistic models is novel.

Reproducibility: Code is included in the supplementary material (although I have not tried yet).
summary_of_the_review: The DDPM-based conditional sampling formulation of motif-scaffolding problem is novel. Authors propose ProtDiff and SMCDiff for both unconditional and conditional sampling of protein backbones. The validness of some assumptions (Cons 1 & 2) may need further discussion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: For many protein design applications, the user specifies a motif, and then needs to build a scaffold of additional amino acids that will act to stabilize the motif. This paper proposes an interesting technique for generating scaffolds conditional on motifs. First, a 3D diffusion model of protein structures is fit. Then, conditional 'inpainting' is performed to sample the 3D configuration of the scaffold given the motif using a particle filtering technique. Then, an fixed structure -> sequence model from prior work is used to produce a protein sequence. Given the accuracy of AlphaFold, the overall system can be evaluated without performing prohibitively expensive wet-lab experiments: simply check whether the AlphaFold-predicted structure of the output sequence contains the desired motif.  
strength_and_weaknesses: =Strengths=
The paper is very well grounded in current literature. 
The paper addresses a high-impact application.
The paper provides a clean use of sequential Monte Carlo (SMC) that will be of independent interest to the community.

=Weaknesses=
Using SMC for inpainting with diffusion models is a generic idea that could have high impact, for example, to computer graphics research. However, I worry that that target audience won't read this paper because it is quite focused on proteins. If the paper was more methods oriented 'hey, we have a generic SMC method for diffusion models…", it could perhaps have higher impact.

The experiments are careful and well-thought out, but mostly focus on a couple of small-scale case studies. I don't quite understand why the analysis of the conditional sampling (aka motif scaffolding) couldn't be done at a larger scale (see below). This will hurt the impact of this paper.

clarity,_quality,_novelty_and_reproducibility: I work in this research area. The paper's problem formulation is well-motivated. A system that performs well at the motif scaffolding task could be revolutionary for a variety of applications.

The paper is very well written and approachable. I also appreciate that it is frank about the shortcomings of the current experiments, etc. 

The paper brings together disparate ML ideas (diffusion models, sequential Monte Carlo, protein design) in an elegant way. 

Implementation for the paper's method is non-trivial (e.g., SMC is annoying to implement). I appreciate that the authors have released code.


summary_of_the_review: Overall, I really like this paper. I think the method is really interesting and it is presented well. However, there are clear concerns around the experiments, as they mostly focus on a couple of specific case studies and there are clear shortcomings of the current method (e.g. ability to capture chirality). Below are some questions mostly to clarify why more comprehensive experiments would be challenging.

=Scale of experiments=
I don't fully understand why sec 5.2 is only on a couple of example proteins. What is the limiting factor? Is it that we don't know good motifs to scaffold? That other natural targets for scaffolding have protein sequences that are too long? Were you compute bound? 

=Length of proteins=
I understand that scalability was a key concern, and thus you trained on sequences of length < 128. Was this a computational or modeling issue? Do you have to actually model every single amino acid in order to capture the gist of the 3d structure? I wonder if the diffusion model could be defined over a coarser-grained version of the structure.

=Types of motifs=
Fig 2a doesn't seem that challenging of a motif. Is it just a single helix? Is the motif consecutive in the primary sequence? For enzymes, for example, I'd imagine that the motif (the active site) would correspond to non-consecutive residues.

=Reliance on ProteinMPNN=
Surely there are realizable structures that ProteinMPNN can't produce a sequence for. It would be good to know if this is a bottleneck in your system. Do you know what the self-consistency rate of ProteinMPNN is in general? What fraction of PDB structures are such that structure -> sequence -> predicted_structure has high similarity between structure and predicted_structure?

=Minor comments=
It would be helpful in the intro to provide a bit more rationale for why prior work only considered 20 residues. It was hard for me to understand why the current work doesn't scale.

"A protein’s structure is a function of its sequence, so we may write x(s)." You should acknowledge at some point in the paper that the notion of a single static structure for a sequence is fraught.

I appreciate the negative result at the bottom of page 7!
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studied protein generation problem, and specifically, motif-scaffolding conditional protein design problem. This paper first proposed a general protein 3D structure generation framework, which can generate high-quality protein backbones in 3D. Then the author proposes a new sampling algorithm for inpainting problem, which means part of the data is masked while the model aims to recover the missing part. Analytical experiments demonstrate the effectiveness of the proposed method for both unconditional and conditional protein design.
strength_and_weaknesses: Strength:

1. The paper is well organized. The domain-specific motif-scaffolding problem is clearly explained, and the method is easy to follow.
2. The paper nicely provides technical contributions from both biology and ML perspectives.
3. The paper conducts careful experiments to demonstrate the effectiveness of the proposed method.

Weakness:

1. The topic is a little bit narrow, which is even a subarea of general protein generation. Therefore, the paper may not be attractive to ML conferences.
2. The protein generation framework (ProtDiff) itself is similar to existing EDM models.
clarity,_quality,_novelty_and_reproducibility: The paper is pretty clear but still has several parts kind of confusing. See my questions for details.

Quality is very high, with paper well polished and experiments carefully designed.

Novelty is a little limited. But the task combined with the diffusion model solution is somehow new to the specific community.

Reproducibility is pretty good. Details for reproducing the experiments are well explained and the source code is also provided.
summary_of_the_review: The paper aims to solve the protein motif-scaffolding problem. The author mainly proposed ProtDiff and SMCDiff methods. ProtDiff is for protein generation in 3D, which is similar to previous EDM model for molecule generation. SMCDiff is a new proposed conditional sampling method. Experiments are well designed and sound, showing the model can generate reasonable and realistic structures.

Questions:
1. I think from the description of SMCDiff sampling, it is a general sampling algorithm that is not only for proteins. Then, whether you can also show the effectiveness of SMCDiff sampling for general data? I understand running experiments for large-scale images costs too much time, but maybe you can test it on some synthetic data?
2. Besides, since this is a conditional generation task, I'm a little confused why you only define the problem as an "inpainting" problem instead of a conditional generation problem. In other words, for this conditional generation task, you actually don't need to learn the joint distribution and just do conditional sampling. Maybe I misunderstand the available dataset or your tackled scenario, but in my mind in the learning procedure, you can also just learn a conditional model, which may make more sense.
3. By the way, for the provided PDF on my PC, the hyper references (e.g., citations) don't work correctly. Clicking on the reference will always be directed to the first page, making it difficult for me to check related work or equations. Please double-check this.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors translate diffusion models to the protein structure domain to design new protein scaffolds and motifs.

strength_and_weaknesses: The diffusion approach seems interesting and a promising way to explore sequence and structure space.

Section 5.1 - Two works are cited to justify using AlphaFold2 as the gold standard of accuracy of those structures. But for de novo scaffold and motif design, as you propose here, has this been validated? To what extent? Shouldn’t this be done first to make sure you can trust those predictions? Minimally, how accurate is it without a multiple sequence alignment?

Section 5.2 - Why PDBs 6exz and 5trv?

In Figure 3A, it looks like the vast majority of scTMs is around 0.25, which is random. If you are using this software during inference, how do you know that your scTM is > 0.5, so it is “designable? 

In Figure 4, it looks like chains are able to pass through one another in space during sampling. This does not happen for proteins in vitro or in vivo. Does your model allow for chains to pass through one another?

clarity,_quality,_novelty_and_reproducibility: For a general audience, and someone well versed in structural biology, it would be great to concretely clarify between scaffold and motif. How do they differ? From your initial paragraph in the introduction, they are described in the same sentence, and still convolved. From my understanding, neither the structure nor the motif is the biological function itself–it’s just the shape that supports the function.

Section 2.1 - “ A protein’s structure is a function of its sequence…” This is not true in the case of post-translational modifications. In the case of entities in the PDB, many modified proteins exist.

Section 5.2 - “For each topology, we chose a…” What do you mean by topology? Do you simply mean that given structure? Or do you mean Topology in the CATH nomenclature?

Figure 4 - It is difficult to understand the accuracy of the interpolations. What are you interpolating in between? Are they even valid folds between interpolations? What does interpolation look like in other methods? It is unclear what message this figure is conveying to the reader.

summary_of_the_review: While I think it is an interesting approach and great area of exploration, more work needs to be done to justify it's utility.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies protein backbone 3D structure generation using diffusion models. All reviewers acknowledged the significance of this work in terms of methods and results. The issues raised during review have been largely resolved during rebuttals. Thus an accept is recommended.
justification_for_why_not_higher_score: There are minor issues that remain.
justification_for_why_not_lower_score: All reviewers acknowledged the significance of this work in terms of methods and results. The issues raised during review have been largely resolved during rebuttals. Thus an accept is recommended.
Decision: Accept: poster