Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an approach for drug design that generates molecules by simulating adding or deleting parts of the molecules, and using graphnets to capture atom and fragment level information and construct new molecules. They use simulated annealing to ‘edit’ the 3D structures, and docking simulations, drug-likeness and synthesizability to provide information back into training. In this way, the authors claim the model can be trained on self generated data. The authors compare with multiple baselines on a test set of 12 targets, including the current SOTA model, and report improved performance.
main_review: The approach described appears plausible. The methods are mostly well described, and the paper is clearly written for the most part.

Although the results are good, most of the proposals in the description of GEKO seem to be based on existing literature, and indeed described in the prior work section of this paper (e.g., MARS). So the paper seems less about a novel approach and more about SOTA performance. But the dataset used to test the model is a bit small if this is where the novelty of the paper lies. Only 12 targets is a very small dataset size when comparing model performance, and unlikely to be representative of the variation seen in a practical setting.

Have molecular docking software to estimate binding energy - but how accurate are these? In general I found the description of docking to be limited.

In the ablation study, ‘Random’ is chosen as a kind of baseline, but the score is better than expected if just randomly picking editable bonds without any deep learning. Why is this? In general more analysis of the ablation study findings would be helpful.

The authors claim GEKO outperforms other methods significantly. But the claim of statistical significance is not backed up with p values or confidence intervals for the comparison. The authors should provide evidence of a statistically significant improvement over other models, or adjust the wording. 

The authors claim 1D/2D approaches limit models to targets with rich activity data. This is not entirely true, as really we want models to be able to generalize to new, previously unseen targets. With a large enough dataset, this is not impossible. 

“Finding a drug molecule that can cure specific diseases is one of the most significant yet challenging task in human development” - it’s a very minor point, but the field of human development is unrelated to the field of drug discovery. The authors may want to say it is a major and significant challenge in the field of biology/medicine instead.
summary_of_the_review: While the results on the test set are good, I’m not convinced that the results alone here are novel enough for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to optimize the multiple properties of a molecule. They first define rigid fragment library as a building block, which is used to generate a new molecule through geometric editing. They represent a molecule using HMPNN and the whole model is trained using the weighted maximum likelihood estimation. The proposed model is compared with some baselines on several multi-objective optimization tasks and shows good performance.

main_review: Strength:
1. The proposed model outperforms other baselines in the multi-objective molecules optimization benchmark. 
2. This model doesn't rely on a data-driven biological activity predictor.


Weakness:
1. The model seems to be incremental. MARS already used rigid fragment library, editing operations, and MPNN. GEKO extends them to geometric editing, HMPNN, and docking software. At the first glance, I thought those rigid fragment library and editing operations are brand new concepts, but I found it's not after reading the MARS paper. It'd be nice if the authors contrasted GEKO and MARS so that readers would not be misled. 
2. There are always pros and cons for each direction. The good thing about using a data-driven biological activity predictor is that binding site information is not required. Therefore, it doesn't require the 3D structure of the target, which makes it to be applicable to many proteins. However, when using docking software, data-driven predictors of biological activity are not required, but the 3D structure and binding site information are essential.
3. When running docking software, different poses of a molecule should be generated and tested. However, that information seems to be missing in the method section. In general, the pose generation is a big bottleneck, so if this information is included in the Methods section, it will help to better evaluate this paper.
summary_of_the_review: I have some questions/suggestions;
1. Optimizing speed comparison would be a good addition because docking simulation is known to be very slow.  
2. In equation (15), the angle is sampled from a softmax funciton? Is the angle categorical?
3. In the baselines, it's known that VJTNN[1] is later model than JT-VAE. In addition, MolDQN[2] and CMG[3] are other recent models. Better to look at them.

[1] Jin, Wengong, et al. "Learning multimodal graph-to-graph translation for molecular optimization." arXiv preprint arXiv:1812.01070 (2018).

[2] Zhou, Zhenpeng, et al. "Optimization of molecules via deep reinforcement learning." Scientific Reports 9.1 (2019): 1-10.

[3] Shin, Bonggun, et al. "Controlled molecule generator for optimizing multiple chemical properties." Proceedings of the Conference on Health, Inference, and Learning. 2021.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper works on the deep learning models for automatic drug design. The authors propose a method named GEKO, the geometric editing under knowledge guidance, which incorporates the physicochemical knowledge into the designed 3D model. The authors claimed this is an unsupervised drug design method that the training is conducted in a purely training data free fashion. The training algorithm is based on geometric (3D) editing with the knowledge (Vina, QED, SA core) guidance by self-training and simulated annealing. The representation of the molecule is modeled at atom-level and fragment-level. With the geometric editing process continuing, the model continues updating and learning towards a better result. The method is evaluated on the drug design task on two sets of target datasets, by generating the molecules (drugs), the results are evaluated by multiple metrics. The performances are improved compared with other baseline methods (including both 2D and 3D methods). 

main_review: The main idea of the paper is summarized in the above sentences. The advantages and the disadvantages of this paper from my side are as follows. 

Strengths:
1. This paper mainly focuses on knowledge-guided geometric editing. As for the knowledge, as far as I read, including (point out if I am not correct): autodock vina (for target-ligand energy), QED (drug-likeness), SAscore (for synthetic accessibility). The knowledge is important and necessary for molecule design. 
2. The geometric editing stands on the bonds in the molecule (between fragments), which causes stability and the specific property. The editing method is in a reasonable design, which selects the bond and the fragment, then generates a new molecule. 
3. The results on the selected target perform well in the 3D generation w.r.t the evaluation metrics. 

Weaknesses:
Though this paper gives knowledge-based editing, there are many concerns I have, 
1. In the big picture, GEKO is a self-learning method based on the knowledge information provided by some softwares. Actually, the formulation of Equation (1) and Equation (16) is exactly the reinforcement learning-based method, or specifically, the policy gradient method. Equation (1) is indeed the reward function provided by the ground-truth (though provided by the software), and Equation (16) is the reward weighted objective. In this way, the claim of this paper that it is an unsupervised drug design is somehow overclaimed. Besides, the authors even do not talk about RL at all.
2. Detailed modeling questions. I think $x_{frag}$ is different from the $x_{skel}$, which should be the new fragments in the fragment library. However, the definition of the softmax equation (10) can not reflect at all, and where is $x_{frag}$ in equation (10)? What is the detailed softmax? Similarly, in equation (13), where is r in the right side?
3. I am confused about the edge r and the bond a, what is the difference? why there are two samples of r and a?
4. As for the four atoms selected u, w (skel and frag), how are they selected? Since each bond only has two atoms, and what are the two other atoms selected? Footnote 4 is missing. 
5. In the overall formulation, the dimension of the formulation are not introduced, which makes feel confused about the transformation, for example, the $p_{attach}$. 
6. The training cost is also important since this is a self-learning method and the training data is generated by the model itself. Therefore, it is important to report the cost and the comparison with other baseline methods. 
7. More importantly, the following content of dihedral angle $\alpha$ is missing. What is then after sampling an angle $\alpha$?
8. Minor points, some typos are existed, for example, 'seleted' in Figure 1. 

summary_of_the_review: The method contains several signs of progress to generate new drugs in 3D space. Overall it is ok, but more questions should be answered.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a method that designs molecules by combining
fragments along "editable" (single) bonds. The method uses two graph
neural networks (on atoms and on fragments) to learn to pick editable
bonds. The method also uses simulated annealing to guide the
combination of fragments (and possibly their dihedral angles) to
optimize a weighted score that the authors have found helps their
training. The model is able to propose reasonable molecules in 3D
conformations within a ligand pocket using a docking score for
guidance. The authors perform an ablation study that shows that even
without the deep neural network for selecting the editable bonds,
their sampling methodology performs rather well.

main_review: 
The high level contours of the technical work sound reasonable;
however, by reading the paper I was not convinced that the eventual
deep learning model adds much value to this process. If I take the
authors ablation study by heart, then a random way of picking editable
bonds would still result in a fantastic score, way beyond any of the
scores of previously published chemistry models, even those in the 2D
case. So is the result simply a tuned simulated annealing of fragments
or is there any important learned aspect on picking from the library?

Unfortunately, this paper is not written in a clear fashion and leaves
open questions relating to reproducibility. There is a lack of a clear
exposition of the details with regards to the datasets, benchmarks,
and tons of additional little details peppered throughout the paper:
does the fingerprint used for similarity include chirality and what is
its radius? How large was the resulting rigid fragment library? (At
some point the authors mention including only fragments with less than
10 atoms, but even the single aromatic ring in Figure 1 has 12 atoms
total---did the authors mean heavy atoms when they say atoms?) What
were the training/starting molecules in each case and do those matter
(if not, do the authors start from a methane every time?) Since the
authors seem to have used docking together with the 2D models, why
didn't they also use the 2D generator models on the 3D benchmarks of
table 1? (Incidentally, in Table 1, the largest score in column Div
seems to be the one for GraphAF, not liGAN.) The explanation of the
authors that GraphAF generated small molecules that don't meet the
drug requirements seems interesting; I thought that this model can be
trained to generate arbitrary large molecules---is that not the case?
What targets did the authors evaluate in table 2, 3; and why are the
baseline results different from those in table 1? I could not find out
what the scatterplot in Figure 4 represents---is that some abstract
view of chemical space, and if so what exactly are the x- and y- axes?
What is the deepSAR model in tables 4, 5? (I checked that the authors
don't refer to the paper with the model called deepSAR, but I suspect
that they might have meant GEKO in those tables; additionally, in
table 5, I suppose the authors meant to refer to Target Set B, not A).

summary_of_the_review: 
I have concerns about the reproducibility of this work and I find a
lot of details lacking. The English needs some work; some figures and
tables have unexplained or mislabeled data. It is possible that with a
lot of additional work the same idea could become clearer and well
documented, but I am not confident it could make it to ICLR22.
review_title: null

Meta-review: The authors describes a drug design method that generates molecules by simulating adding or deleting parts of the molecules, and using graphnets to capture atom and fragment level information and construct new molecules.  Simulated annealing is used to ‘edit’ the 3D structures, and docking simulations, drug-likeness and synthesizability are used to provide information back into training.  The authors compare with multiple baselines on a test set of 12 targets, including the current SOTA model, and report improved performance.

Strengths:

- The proposed model outperforms other baselines in the multi-objective molecules optimization benchmark.
- The model doesn't rely on a data-driven biological activity predictor.

Weaknesses:

- The reviewers point out that the model seems to be incremental with respect to previous work.
- The reviewers have concernts about the reproducibility of the work and find a lot of details lacking.

This is a borderline paper with a majority of reviewers voting for rejection. I recommend the authors to addrses the weaknesses above and resubmit to another venue.
Decision: Reject