Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose DiffSBDD, an approach to structure-based drug design (SBDD) that aims to generate molecules in 3D that are conditioned on a protein binding interface. The approach is E(3)-equivariant and leverages the denoising diffusion probabilistic model (DDPM) formulation inspired by non-equilibrium thermodynamics.

Two distinct approaches to 3D pocket conditioning are proposed. The first keeps the pocket constrained in each denoising step (conditional) and the second approximates the joint distribution of ligand-pocket pairs enabling sampling of both ligand and protein positions (inpainting). 

To demonstrate the applicability of the proposed model, the authors create a new dataset of experimentally derived protein-ligand complex data from the publically available database Binding MOAD in addition to a synthetic cross-docked dataset. 
De-novo molecule generation was evaluated for the conditional approach in the context of both all-atom and Cα (for protein) level graphs. For the inpainting approach, only Cα level graphs could be evaluated due to computational limitations. In silico metrics were calculated for generated molecules and compared with two recent deep-learning approaches to SBDD (3D-SBDD and Pocket2Mol). **DiffSBDD did not perform better**. 
strength_and_weaknesses: **Strengths**:

The paper takes a recently published DDPM formulation for modeling protein-ligand interactions and introduces what appears to be a novel approach to 3D pocket conditioning (conditional and inpainting). They also use an experimental database (Binding MOAD) for training the model instead of synthetic data that often has no grounding in reality.

**Weaknesses**:

A fundamental flaw in this work appears to be a misunderstanding by the authors revealed in the second sentence of the abstract. SBDD pipelines start with ligand and protein preparation which is a prerequisite for large-scale docking. This study excludes this extremely important step as ionization/tautomeric states for ligands and H-bond optimization/protonation for proteins are neglected. SBDD is only as good as the structures you put into it (garbage in~garbage out). 

The novelty of the proposed DDPM formulation is lacking as this work closely follows the framework developed by Hoogeboom et al. 2022. In addition, the paper does not include results for the inpainting experiments using all-atom protein level graphs. The poor results achieved using only C*α* atoms for the protein are not surprising as SBDD requires explicit representation of rotamers for amino acids as the orientation/protonation of these residues can dramatically alter what interactions can be made and ultimately whether a molecule will bind with sufficient potency to warrant detection in an in vitro assay.

Some of the molecules generated by this approach are not physically possible to produce and would be incredibly unstable. The paper uses a reference to Ertl & Schuffenhauer 2009 to justify that the low SA score may be because it is based on historical knowledge of previously synthesized molecules, but this is precisely the knowledge needed to understand what is possible on planet earth at 1 ATM with the tools we have at our disposal to synthesize real molecules that we can test experimentally to evaluate these predictions. I take a similar issue with the use of the CrossDocked dataset for training as these are in silico docking predictions that have not been experimentally validated and therefore should not be expected to generate results that are grounded in reality.

The paper does not propose a null model to evaluate the significance of the reported results.
clarity,_quality,_novelty_and_reproducibility: The code was well-documented and easy to run. The results were also available for evaluation. The paper was reasonably clear however the novelty of the proposed DDPM formulation was lacking.

The code provided does not appear to perform molecule standardization/minimization following molecule generation (outside of explicit errors thrown by RDKit due to valence/hybridization errors), is there a rationale for not performing this cleanup step?
summary_of_the_review: The authors propose DiffSBDD, an E(3)-equivariant 3D-conditional diffusion model for SBDD which they evaluate in the molecular generation of novel/diverse ligands that maximize predicted affinity. While they demonstrate that inpainting can achieve competitive results to other deep learning approaches to SBDD, it was not superior, and no null model is presented to demonstrate the significance of the results. In addition, the curation and preparation of the protein and ligand structures were not reported/neglected calling into question the validity of the results. Lastly, the proposed DDPM formulation does not appear to be a novel contribution which limits the significance of the manuscript contributions. Therefore, my recommendation is that the manuscript is premature for acceptance to ICML.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a diffusion-based model for generating a small molecule complete with 3D atomic coordinates in a protein pocket.  The performance of the model on a prior benchmark is well below that of pocket2mol (a related model) on the CrossDocket test set. The paper also presents a new dataset for the same task, which the authors call "binding MOAD" (from Mother Of All Databases).
strength_and_weaknesses: 
The paper has a couple of strengths: the task that it attacks is of high importance for human health and the ideas that it discusses represent modern and powerful developments in machine learning.  The paper sets out to test inpainting in the context of small molecule generation in a pocket, a technique that worked extremely well for protein generation in David Baker's lab.  Importantly, the paper also adds a new protein-ligand dataset for generative small-molecule tasks.

The weaknesses of this work overwhelm the positives, unfortunately.  Although the particular task of generating desirable ligands that match a pocket is noble, there is no clear argument why the direct generation would be desirable in principle (for example, it is possible that a more practical approach is to simply learn a surrogate function.) Of course, this would not be important if the results were overwhelming; however, they are not. The model performs significantly worse than pocket2mol.  Importantly, the authors do not even evaluate pocket2mol on the new dataset that they generated (even though the code for pocket2mol is publically available).

Also the paper spends a lot of time discussing parts unrelated to this work (the paper does not use eq (3), (4), (5); instead it predicts the noise, as per eq (6)). Similar in page 5, the paper spends a lot of space discussing things that are not used.  Perhaps some of these things could move to an appendix or a different publication.  In page (4) what is the reference to Anonymous. ICLR2023 submission. 2022?

Importantly, the authors discuss an E3 architecture but then mention rigid body transformations, which is SE3 only (E3 without reflections).  If they indeed use E3 (as appears to be the case, though I didn't check the code), then it is possible that update (10) might perhaps break due to a reflection symmetry of E3.  Did the authors check that indeed there is no mixup of stereo chemistries in the generated molecules (the distinction is important because a reflection could turn a therapeutic drug to a killing toxin.)

Why did the inpainting idea not work as well as expected based on the prior work by Baker's lab?  Also, why did the authors suggest that the inpainting was competitive to the conditional generation (it was possibly better on only 1 of 12 subtasks in tables 1, 2).  Why does the model perform so much worse than pocket2mol on the common CrossDocket set?  Why didn't the authors also run pocket2mol (code is available) on their new MOAD binding dataset?

clarity,_quality,_novelty_and_reproducibility: The work is a close analogue of pocket2mol using ideas from Hoogeboom. Unfortunately, it does not appear to work as well as prior work, and it does not seem to add any productive ideas, nor discuss in any clarity the lack of success of some of the interesting ideas, e.g. the inpainting strategy, which appears to have scored worse overall than conditional generation.  The authors provide a new dataset for small molecule generation tasks, however, they don't evaluate prior related codes on this dataset.  The authors provide their code and their datasets, so I expect (though I haven't tested) that the work is reproducible.

summary_of_the_review: This method is valid, but not strong enough for ICLR2023.  The paper uses space on unnecessary items and avoids trying and to understand some of the results, which could have strengthened the presentation or lead to an improvement.  The current version of this paper is not at the level of ICLR2023.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a diffusion model that conditionally generates 3D molecules. The proposed model modifies the existing E(n)-equivariant Graph Neural Networks with a new coordinate update step considering the characteristics of pocket coordinates. They frame pocket conditioning in two ways; pocket-conditioned small molecule generation and joint distribution with inpainting. Besides the proposed model, they also curate a new benchmark reflecting more realistic binding scenarios. Experiments show that the proposed method is better than other baselines in terms of diversity.


strength_and_weaknesses: 
[Strength]
* For the first time, they approach the 3D molecule generation problem as an inpainting problem, which is novel.
* They evaluate 3D molecule generation methods on a new benchmark to reflect more realistic binding scenarios.
* The paper is easy to read and well organized.
* Domain-specific case studies show various aspects of the model. 


[Weaknesses]
* Although their approach is novel, the core model relies on the existing model EGNN with minor modifications.
* The primary goal of molecule generation is to achieve high affinities (low VINA scores). However, the proposed method underperforms not only other baselines but also the test set. Therefore, the motivation of the proposed methods is not convincing.

clarity,_quality,_novelty_and_reproducibility: * The paper is clear and easy to understand.
* Novelty is limited as stated above. 
summary_of_the_review: Since it does not outperform others and the novelty is marginal it's straightforward to score it as rejected.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper studied structure-based protein design, which means generating drugs specifically for a given protein target. The author proposed a conditional diffusion model for this task, which is an extension of the previous equivariant diffusion model [1] for 3D molecule generation. The author considered both conditional generation and inpainting generation tasks. Experiments show that the model can match the performance of current other methods.

[1] Emiel Hoogeboom, Vıctor Garcia Satorras, Clement Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8867–8887. PMLR, 2022.

strength_and_weaknesses: Strength:
1. The paper is clearly written. Both the task and the method are clear and easy to follow.
2. For the specific task of "structure-based drug design", the method is the first diffusion-based model implemented.
3. The author nicely collected a new dataset for benchmarking this task.

Weakness:
1. Though the first diffusion for this specific task, the method itself of equivariant diffusion is not new. Besides, the task has also been well formulated in previous ML papers. Thus the papers seem to be a little incremental for ML conferences, with limited technical contributions.
2. For the proposed method, the author also discussed the "inpainting" scenario. However, it's very unclear to me how this could be useful? It feels like a direct and unnatural adoption of diffusion models on images, but in my mind perturbing the protein targets is not very meaningful in the biological/chemical domain.
3. Furthermore, as shown in Tab. 1 and Fig. 3, the empirical performance seems not significant compared with baselines, and many generated drugs are not realistic molecules. The major improvement is in diversity metric. However, I would say higher diversity with lower quality actually indicates lower generative modeling capacity. I suggest the authors to further improve the model's empirical performance.
4. Some important baseline is missing [1].

[1] Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. ICML 2022
for target protein binding.
clarity,_quality,_novelty_and_reproducibility: Clarity is pretty good with all technical and experimental details greatly explained.

Quality is high but can still be improved. E.g., There is an anonymous reference above Equation.10. However, even the title of the anonymous reference is missed, making the sentence a little confusing...

Novelty is good by also limited. See details in the "weakness" section.

Reproducibility is supposed to be good since the source code is nicely provided, though I didn't take a closer look at the details of the provided code.
summary_of_the_review: The idea of diffusion models for structure-based drug design (SBDD) is good, but both the SBDD task and the equivariant diffusion method are not supervising things for ML community. Besides, the adoption of the "inpainting" technique from image generation is not clearly natural and useful for this problem. Besides, the empirical performance is also not significant, and some important baseline is missed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes a diffusion model for structured based drug design. All four reviewers raised major concerns on many aspects of this work, and these concerns remain after rebuttals. Given the consistent concerns, a reject is appropriate.
justification_for_why_not_higher_score: All four reviewers raised major concerns on many aspects of this work, and these concerns remain after rebuttals.
Decision: Reject