Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, TargetDiff, a new 3D diffusion model, is introduced. The model generates molecules in a non-autoregressive fashion. The generated molecules are conditioned on the binding pocket, and the generation is equivariant to rotations and translations thanks to the equivariant GNNs used. Additionally, an unsupervised entropy-based method for ranking molecules is proposed. The model is evaluated against multiple recent generative models: liGAN, AR, GraphBP, and Pocket2Mol. The results indicate the strong performance of the proposed method.
strength_and_weaknesses: Strengths:
- Non-autoregressive sampling helps to better fill the available space in the binding pocket.
- The related work contains a set of very recent publications demonstrating the advancements in the fields of target-based molecule generation and diffusion models.
- SE(3)-equivariant networks ensure the strong performance independent of the protein orientation.
- Both continuous and discrete diffusion models are used to model coordinates and atom types, respectively.
- The main claims of the paper are formally proven.
- The authors discover a correlation between binding affinity and the entropy of the predicted atom representation when atom coordinates are frozen. Thus, this entropy can be treated as an affinity predictor trained in an unsupervised fashion.

Weaknesses:
- Currently, the code is not available, but it should be open-sourced upon publication.
- In Tables 3 and 4, confidence intervals could be added to make these results more convincing.

Questions:
- How do you combine v entropy and vina scores in Section 4.3?
- Coordinates and atom types are modeled independently in Equation 2. Do you think this could have a negative impact on the generated structures?

Minor points:
- Typos: “more structural data become available and unlock new opportunities”, “protien”->”protein”
- The full name “Center of Mass (CoM)” should be introduced at its first occurrence.
- Figure 7 reference is missing.

clarity,_quality,_novelty_and_reproducibility: **Clarity.** The paper is written in a clear way. The figures are clean and well support the model description and nicely illustrate the experimental findings. The training details, including pseudocode, are provided.

**Quality.** The model is described in full detail, and the experimental results are supported by many figures showing different aspects of the model. There are four recent models compared against TargetDiff, and multiple metrics are used. The main claims of the paper are proven in the appendix.

**Novelty.** Diffusion models are a new class of generative models, and this is one of the first examples of using them for target-based molecule generation. The concept of affinity predictor trained in an unsupervised fashion also seems novel.

**Reproducibility.** The code is not available, but the authors say they will publish the code when the paper is accepted. Based on the description, the reimplementation would be probably possible but tedious. Pseudocode is provided for the training and sampling procedure.
summary_of_the_review: Based on the above comments, I am leaning towards the acceptance of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackles target-aware small molecule generation. Specifically, small synthetic ligand molecules are generated using a deep generative model, such that the molecules best fit into the binding site of a larger protein. To this end, the paper jointly models the ligand and the protein using a generative diffusion model, conditioning the generation of the ligand on the protein. Technically, in contrast to some previous papers, the work explicitly models 3D atom coordinates and also uses appropriate equivariant neural networks. Diffusion processes over both atom coordinates and also atom types and features are used. The paper also shows the entropy of the generated atom feature distribution can serve as an approximate measure for binding affinity. Empirically, the paper compares mostly to two simple baselines, a 3D voxel-based molecule generative model and an autoregressive model, and outperforms them on various molecule generation tasks.
strength_and_weaknesses: **Strengths:**

- The particular combination of using a mixed continuous-discrete diffusion model, together with an equivariant architecture, to model atom coordinates and features in a protein ligand complex seems to be new (although it's a straight-forward combination, see below).

- The work's quantitative results look okay and outperform the two chosen baselines liGAN and AR, although I am not sure the baseline comparisons are sufficient.

- It is interesting that the entropy of the distribution of generated atom features correlates with binding affinity.

**Weaknesses:**

- Methodologically, the method is a straight-forward combination of existing techniques. The Gaussian and categorical diffusion processes used are well established. The equivariant neural network architectures seem to be standard. And jointly modeling the ligand-protein complex with a generative model is also not new. Merely this exact combination of techniques seems to be new.

- The paper makes claims that the equivariant diffusion model approach is better than autoregressive techniques in its introduction and motivation. Yet, the method is outperformed by Pocket2Mol -- an autoregressive method -- with regards to the molecular properties (Table 3, approximately similar performance on affinity and diversity, significantly worse on SA and QED). In the other experiments, the Pocket2Mol baseline is not even considered. However, I believe it would be very appropriate to compare to this work more rigorously, as it also uses modern equivariant architectures, considers similar molecule generation tasks with the same data, and also has code available.

- The model only predicts 3D atom coordinates and atom types and features, but no bonds.
clarity,_quality,_novelty_and_reproducibility: **Clarity:** There are no major concerns with respect to the clarity. I was able to follow the paper.

**Quality:** In principle, the paper is well executed with a lot of experiments. However, I believe some strong baselines have been missed (see weaknesses above). If the paper claims to be generally superior to autoregressive models as one of its motivations, I would expect the authors to thoroughly compare to the very best and most comparable autoregressive approaches (such as Pocket2Mol).

**Novelty:** The methodological novelty is small. The paper is a straightforward combination of existing techniques (Gaussian and discrete diffusion models to model 3D atom coordinates and features in a protein ligand complex using equivariant neural networks). I think the observation that the entropy of the atom feature distribution correlates with binding affinity is interesting, but this is just one small experiment. Probably there could be done more in that direction. I am wondering whether even better unsupervised binding affinity predictors could be constructed.

**Reproducibility:** I do not have any concerns regarding the reproducibility. The paper uses public datasets and seems to provide sufficient training details, such that the experiments could in principle be re-run. The required compute resources are also very modest.
summary_of_the_review: In summary, this is an okay paper with thorough experiments. Nevertheless, I do not think it meets the bar at this point, mainly due to two concerns: (a) the methodological novelty is not very significant, as discussed above. (b) This would be okay, if the experimental results would be very strong. However, it seems the paper is somewhat overclaiming in that regard, as it mostly compares to only two baselines that do not seem to represent the strongest existing methods. In particular, the method is motivated as an approach that can generally outperform autoregressive methods. Yet, only a simple autoregressive model ("AR") seems to be considered in most experiments, but not methods such as Pocket2Mol. In conclusion, I do not think the paper is ready yet for publication. I would recommend the authors to more thoroughly compare to baselines. Furthermore, it could be interesting to investigate even further how the learnt features can be used for property prediction, beyond just using the entropy as a measure for binding affinity.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper tackles the problem of generating molecules for a specific protein target. With diffusion models and SE(3)-equivariant networks, this paper learns a joint generative process of both continuous atom coordinates and categorical atom types. Specifically, this paper defines a diffusion process for the continuous atom coordinates and discrete atom types, gradually adding noise, and learns the joint generative process for atom types and coordinates of molecules. 
Besides, this paper evaluates the quality of generated molecules based on the features outputted by the model, which can be used as a scoring function for ranking or binding affinity prediction. 

strength_and_weaknesses: This paper tackles the problem of structure-based drug design using diffusion models and SE(3)-equivariant networks. However, there are some concerns about the method and the experiments:
- This method seems to be a combination of diffusion models, equivariant networks, and structure-based molecular generation tasks. Compared with [1], which combines diffusion models and equivariant networks, the innovation of this work is not obvious. It seems the only difference is the atom information in the protein target is used. The authors can highlight more about its innovation compared with previous works.
- The number of atoms is sampled by drawing a prior distribution estimated from training complexes with similar binding pocket sizes. As determining the number of atoms in advance is crucial to the generation of the diffusion model, I wonder if the authors can show the difference between the sampled number of atoms and the number of atoms of reference molecules. What will happen if the error is too large to cover the reference molecules?
- The details of defining "similar binding pocket sizes" is not provided in the paper. 
- The detail of the data split is not clear. To avoid leakage, it is important to exclude similar pockets in training data.
- It seems the test set is from the CrossDocked2020, not the experimental crystal protein-ligand complex. 
- Vina will first "re-docking" the molecules and then calculate the docking score. Since re-docking can largely change the molecular conformation and binding pose, I wonder if the authors have considered this problem when evaluating the binding affinity?
    - I would suggest providing the Vina score without re-docking, to demonstrate the end-to-end 3D generation performance, rather than post-fixed by docking tools.
- In many results, only AR is used as baselines. And From table 3, there are several baselines better than AR. I think these baselines should be included in most tables/figures. 
- In Table 3, TargetDiff seems only marginally better (and worse in some metrics) than other baselines, especially compared with Pocket2Mol.
- The quality of generated molecules. From Figure 5, S2, and S3, It seems TargetDiff tends to generate the molecule with large flexibility, which is unstable in the real world.
- Did you use any tools, like rdkit/openbabel, to post-fix and filter the generated molecules?
- This paper compares the efficiency of TargetDiff and AR in the appendix. Since GraphBP and Pocket2Mol in Table 3 are also auto-regressive generative models especially Pocket2Mol adopts a more efficient sampling method than MCMC sampling in AR, I wonder how efficient TargetDiff is when compared with Pocket2Mol and GraphBP.
- Can you show the wall clock time, rather than the ratio, in the efficiency comparison?
- The model architecture is not clear. In Sec. 3.4, it says "we model the interaction between the ligand molecule atoms and the protein atoms with a SE(3)-Equivariant GNN". But in Sec. 4.1, it says "where fh and fx are specifically implemented as transformers with 16 attention heads and 128 hidden features." I am confused about the statements, is the model a GNN or a transformer? 


[1] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pages 8867–8887. PMLR, 2022. 

clarity,_quality,_novelty_and_reproducibility: Clarity & Quality: This paper is well-written and easy to follow. The model and experiment seem correct.

Novelty: This method seems to be a combination of diffusion models, equivariant networks, and structure-based molecular generation tasks. 

Reproducibility: The paper provides the details for Reproducibility, but some are missed, refer to the main review. 

summary_of_the_review: Given the incremental novelty and the marginal improvement in empirical experiments, I recommend the rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper propose a diffusion model for molecule generation based on the target 3D structure. The model generates molecules in a non-autoregressive way. The generated molecules are conditioned on the target binding pocket, and the generation is equivariant to rotations and translations thanks to the equivariant GNNs used. Additionally, an unsupervised entropy-based method for ranking molecules is proposed. Experiments show promising performance against multiple recent generative models: liGAN, AR, GraphBP, and Pocket2Mol. 

Strength of the paper: 
1. The diffusion method based on target pocket 3D structure is a novel contribution. 

 Weakness of the paper: 
1. The benefits of non-autoregressive generation for molecules are not well supported by experimental results. Pocket2Mol is better. The motivation of NAT and claims in introduction should be adjusted.
2. The comparison with Hoogeboom et al should be elaborated. 

Overall, after discussion, the reviewers agree that the paper has adequate interesting points to be included in the conference. 

justification_for_why_not_higher_score: The novelty is still limited. The claim about NAT is not solid, according to its experimental results. 
justification_for_why_not_lower_score: The novel technical contribution is adequate to justify its appearance in ICLR.
Decision: Accept: poster