Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposed a deep generative model to design novel molecules in 3D space that bind to specific targets. Concretely, given the binding site as context, the generative model estimates the probability density of atoms’ occurrences in 3D space. Then after learning the distribution of atoms, they design an auto-regressive sampling scheme to generate atoms sequentially. Experimental results verifies the effectiveness of the proposed generative model in designing 3D molecules for specific proteins in binding pockets. 
main_review: The paper designs a deep generative model to form a valid drug-like molecule fitting to a
specific binding site. The generation procedure is split into two parts: (i) it predicts
the probability density of atom occurrence in 3D space of the binding site, where the binding target is used as context and roto-translation invariance neural network is used; (ii) it leverages auto-regressive sampling algorithm for generating valid and multi-modal molecules from the learned atoms’ density. 3D molecule design is an important and popular topic in recent years and is highly valuable. 

However, there are some issues in experiments. 

First, the docking score (in terms of Vina) of the generated molecules is relatively poor. In Figure 4 and 5, none of the docking scores is better than -10. Also, in Table 1, the average vina score is -6.3, which does not indicate high affinity. 

Second, during empirical studies, some important baseline methods are missing, e.g., [Masuda et al, Generating 3d molecular structures conditional on a receptor binding site with deep generative models] and [Ragoza et al. Learning a continuous representation of 3d molecular structures with deep generative models]. 


==================

I have read authors' responses and other reviews, and do not change my score. 

limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel generative method for molecular design that uses 3D spatial information and the binding pocket embedding to generate molecules with high binding affinity towards a selected target protein. The method allows for multimodal sampling of valid and diverse molecules. In the experiments, the model is evaluated in two scenarios. The first scenario is a structure-based de novo drug design, and the second one is linker prediction. In both cases, the proposed model manages to generate reasonable compounds that fit into the binding pocket. Additionally, the generated compounds exhibit high drug-likeness and synthetic accessibility.
main_review: The method presented in the paper is very original. The model can perceive 3D relation between generated ligand and target protein. There are not many generative models that are designed to create molecules that directly fit the 3D space of the binding pocket. The proposed model is certainly unique in the category of structure-based generative models Another method in this category is the shape-based model by Skalic et al. [1], which is missing in the references. This method generates compounds based on the predicted 3D shape that fits the binding pocket. However, also this method uses the SMILES representation instead of working directly on 3D atoms.

The presented experimental results are convincing. The authors present qualitative results in Table 1 which could be improved by adding more models to the comparison (e.g. [1]) and adding error bars based on multiple runs of the models. The qualitative results in Figures 4 and 5 clearly show that the proposed method can generate reasonable compounds. What seems a little bit suspicious is that in Figure 4 the generated compounds look very similar to the reference compound. It may suggest data leakage, e.g. compounds that are very similar to the reference one (e.g. the same analog series) are included in the training set. A scaffold split would prevent this from happening. To make these experiments more convincing, the whole protein families that are present in the testing set should be also discarded from the training set.

The paper is clearly written, and there are only minor issues that generate confusion. In Equation 1, the notation $W^\ell(d_{ij})$ may suggest that $W^\ell$ is a function, and in the text it is said that “weight of message from $j$ to $i$ depends only on $d_{ij}$”. On the other hand, this function is not defined, but instead it is said that $W$ is a learnable matrix. Equations 11 and 12 in the supplemental material are much more precise and do not seem equivalent to what is presented in the paper. The architecture of the frontier network $F$ is not described at all.

The proposed method supports multi-modal sampling of valid compounds thanks to a simple (not using a latent space) autoregressive method. The experiments show that the proposed generative model creates compounds that are drug-like and synthetically accessible without directly optimizing for these values. Additionally, a linker prediction task is performed and the results are compared with a popular model for this task, DeLinker. Having considered all that, the method presented in the paper is of high value to the field of computer-aided drug discovery.

Other comments:
- Why is ‘Nothing’ not a separate class in the probability vector $c$, but instead it always artificially gets value 0 by the means of Equation 5? Does the loss in Equation 8 not converge otherwise?
- In Figure 5, the PDB ID is missing.
- In Figure 5, the atoms with a red overlay are hardly readable.
- In Figure 3, it is not clear what the vertical dashed lines signify.

Based on all the above, I am leaning toward the acceptance of this paper.

[1] Skalic, Miha, et al. "Shape-based generative modeling for de novo drug design." Journal of chemical information and modeling 59.3 (2019): 1205-1214.
limitations_and_societal_impact: The authors could elaborate more on the limitations of the method, e.g. what the success rate of sampling molecules is or what is the time complexity of the method considering the sampling methods used. The negative societal impact is not described. The presented method is a general method for designing active compounds, so potential societal impacts may include improving drug discovery pipelines but also acceleration of the design of harmful compounds, e.g. toxic psychoactive substances.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the problem of generating molecules that bind to a specific protein. To make use of the spatial information to target the proteins exactly in 3D space, the paper proposes an autoregressive model where the atoms are sampled sequentially from a learned distribution that estimates the probability of atoms' occurrence in 3D space. In fact,   the paper models the probability of a position in 3D space being occupied by a certain type of atom given as input the 3D coordinate and the binding site.  the model encodes the context which is the binding site and concatenates it with the input query position r  and finally predicts the type atom that could occur at the position r. 
main_review: The paper is well motivated and clearly written. They tackle the problem of structure-based drug design which is a very interesting and crucial step for using ml models for drug design.  The experiment results show clear improvement over the baselines. 

There are few concerns
1. They only include one single baseline but I am sure there are many works that exist in this line and should be taken into consideration. For instance the work [30] and [31].

2.  In table 1, it is shown that the proposed model has a better score of not only Vina but also SA score and QED score. I can understand the model could lead to a better Vina score as the model is tailored to solve this specific problem but it is not clear what factor of the model leads to a better SA score and QED score when compared to the baseline. 
limitations_and_societal_impact: Limitation: the experiment did not include enough baselines that tackle similar problems.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses the task of 3D molecular generation within a 3D protein binding pocket. An autoregressive model is used to predict probability distributions of atom occupancies, which are iteratively sampled to construct a full molecular structure inside the binding site. 
main_review: The approach is presented clearly and is remarkably simple. The sampling approach provides a natural way to decode the continuous distribution into a discrete molecular structure. The sampling procedure for training of observing masked molecular structures and training the joint distribution through sampling both positive and negative spatial coordinates is procedurally simple (in a good way, in my opinion). The primary criticism of this work would be that it does not provide any significant machine learning advance and lacks quantitative comparisons to plausible alternative approaches, instead focusing exclusively on a comparison to ref [21]. However, I do think this is an interesting task from a domain perspective. 

Contextualization

1.	What are the aspects of the approaches in refs. 8, 30, and 31 that render them unable to handle larger molecules? This does not seem to be a fundamental limitation of the method but perhaps a limitation of the evaluations in those original papers.

2.	The statement that 1D/2D methods are unable to be applied to structure-based design tasks seems to be an oversimplification. String-based and graph-based methods have been applied to the optimization of docking scores as a structure-based oracle function (e.g., https://pubs.acs.org/doi/10.1021/acs.jcim.0c00833)  While not conditioning generation on a 3D structure explicitly, this may be considered an application to SBDD.

3.	There is a family of methods designed for atom-by-atom or fragment-by-fragment ligand generation within a fixed protein pocket (e.g., AutoGrow4). Is it feasible to do any performance comparisons to these non-ML methods? The problem formulation is different, as they use Vina Scores as intermediate signals during generation, but it is realistic to have access to a docking oracle function. The procedure is similar enough to this work that a comparison is warranted.


Experiments

4.	The performance of the model (i.e., Vina Scores) is not reported when the force field refinement is omitted. This paper does not state whether the force field refinement operates on the ligands alone or the ligands and the binding site together, but the reference implies the latter. Ref 21 also states that Vina was used for reoptimization (including of the pose), whereas this work implies that only the Vina scoring function was used. In the evaluation workflow, were structures taken as their 3D conformations or re-optimized? Insufficient details are provided with respect to this evaluation and the dataset used, although the authors do state that code/data will be made available upon publication.

5.	In the experiments, evaluation of activity focuses on the average and median values. However, the goal would arguably be to minimize the docking scores. It appears as though the model does not have the ability to learn how to generate the “most favorable” ligand by considering docking scores during training, but merely generate a representative ligand. Most generated molecules have binding affinity scores that are worse than the reference ligand.  

6.	It seems surprising that in the top row of Figure 4, so many of the generated compounds have the same heterocyclic motif as the reference compound, not to mention the terminal phosphate group. Did this scaffold appear in the training set for a very similar protein? Or did the model truly rediscover it from scratch? 

limitations_and_societal_impact: Extensions are discussed; social impact is not.
review_title: null

Meta-review: The AC and reviewers all agree that this is an interesting submission. 

We strongly urge the authors to incorporate their clarifying comments into the manuscript. In addition, as mentioned by several reviewers, it is important that the authors be clear about the relative value of Vina score. As noted by kPk1, coming up with a new metric that is more appropriate would be a valuable direction for future study.

Reviewer RaVz made some very pertinent point on Tanimoto similarity, SA scores...etc. It would be very exciting to see these comments addressed either in the present manuscript or in future work.
Decision: Accept (Poster)