Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors present a reinforcement learning framework for the design of molecules active against specific proteins. The novelty in the authors methodology is to attach molecular fragments (as opposed to single atoms) at each step and optimise the algorithm with respect to the molecular docking score. Attaching fragments allows for the generation of pharmacochemically-valid molecules (those which are suitable drug-candidates) by only attaching pharmacochemically-valid attachments. Optimising with respect to the docking score allows the algorithm to attain a good proxy of the binding affinity of a given molecule to a given protein.

Given a current state (the current molecule), the framework designs molecules in three steps: 

1) The optimal attachment site is identified on the current molecule. 
2) Given the choice of site, the optimal molecular fragment is chosen from a set of predefined fragments.
3) Given the choice of site and fragment, the optimal attachment site on the fragment is identified.

The experimental results indicate that the proposed approach generates many valid molecules in comparison to baseline approaches.

main_review: Originality:
I believe the work possesses originality in the way the authors have designed their framework. The novelty in the authors methodology is to attach molecular fragments (as opposed to single atoms) at each step and optimise the algorithm with respect to the molecular docking score. Attaching fragments allows for the generation of pharmacochemically-valid molecules (those which are suitable drug-candidates) by only attaching pharmacochemically-valid attachments. Optimising with respect to the docking score allows the algorithm to attain a good proxy of the binding affinity of a given molecule to a given protein.

Quality:
Whilst the results look promising, I think there are areas for improvement. The standard of writing could be improved as it can be confusing at times. The interpretation of the results seems shallow. 

For instance, why ignore the very large standard deviations in figure 4? The authors state (lines 250-256) "Our model outperforms the other generative models MORLD and REINVENT in terms of both hit ratio and top 5% score, except for the hit ratio of the 5ht1b case. Such results show that our model’s performance is superior or at least competitive to existing base lines...". On average, this does happen. Yet, the large variability indicates that it is not always the case. This deserves further investigation, surely? Either through more random restarts or at least a plausible explanation of why this variability occurs. On the topic of figure 4, it does not indicate which plots correspond to which proteins. Also, I think a description of how to read the top 5% plots would be helpful (I can only assume the greater negative scores indicate better performance?). 

The interpretation of Figure 7 is that the molecules look "chemically realistic". I find this phrase somewhat misleading since, to me (I'm not a chemist), any two-dimensional molecular graph looks chemically realistic. What makes these specific structures look realistic? I understand the authors want to show off the molecules they have generated (rightly so!) but they must be careful with their interpretations.

Clarity:
This is mentioned in the previous section but the standard of writing could be improved. The work seems a little rushed, which is a shame since it diverts the attention from the nice results.

Significance:
Machine learning applied to drug-design is currently a fairly hot topic, so this paper is of obvious significance.

General remarks:
The paper makes a nice contribution through the use of fragment-based drug design and the results mainly back this up (see my comment regarding  Figure 4). However, the work seems more like a first draft than a submission-ready paper. For this reason, I would have to reject.

limitations_and_societal_impact: Sections 5 & 6 cover this sufficiently.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a novel approach based on reinforcement learning (RL) to accelerate the discovery of pharmacologically relevant inhibitors. One of the key limitations of existing methods is that many molecules produced by generative models based are chemical invalid. In this work, the authors circumvent this problem by allowing the agent to modify an existing ligand (state) with a set of actions that result in a new chemically valid molecule (next state). Through this introduction of domain knowledge, the authors achieve higher sample efficiency. However, the experiments show that this inadvertently reduces the diversity of generated molecules.

Overall, the approach seems reasonable, and experimental results impressive. However, the manuscript lacks methodological novelty, and thus, might not be suited for this venue. Further, the presentation (language, precision, and figures) requires work. For this reason, I vote for rejection.
main_review: # Abstract
- L7: define pharmacochemically acceptable more precisely
- L8: "the docking score optimization is a difficult exploration problem that involves many local optima and less smooth surface". This statement is not clear
- L13: What is "qualified chemical space"?

# Introduction
- L30: is there is a citation missing?
- Similar to the abstract, the introduction requires more precision.

# Related Work
- Good overview of existing methods but no mention of RL approaches that generate 3D structures, such as:
    - Meldgaard, S. A.; Mortensen, H. L.; Jørgensen, M. S.; Hammer, B. Structure Prediction of Surface Reconstructions by Deep Reinforcement Learning. J. Phys.: Condens. Matter 2020, 32 (40), 404005. https://doi.org/10.1088/1361-648X/ab94f2.
    - Simm, G. N. C.; Pinsler, R.; Hernández-Lobato, J. M. Reinforcement Learning for Molecular Design Guided by Quantum Mechanics. In Proceedings of the 37th international conference on machine learning; III, H. D., Singh, A., Eds.; Proceedings of machine learning research; PMLR, 2020; Vol. 119, pp 8959–8969.
    - Christiansen, M.-P. V.; Mortensen, H. L.; Meldgaard, S. A.; Hammer, B. Gaussian Representation for Image Recognition and Reinforcement Learning of Atomistic Structure. J. Chem. Phys. 2020, 153 (4), 044107. https://doi.org/10.1063/5.0015571.
    - Simm, G. N. C.; Pinsler, R.; Csányi, G.; Hernández-Lobato, J. M. Symmetry-Aware Actor-Critic for 3D Molecular Design; ICLR 2020.
limitations_and_societal_impact: 1. The authors need to revise the manuscript to improve readability and precision. Currently, the manuscript is not ready for publication.
1. The proposed method seems reasonable, but the novelty on the machine learning side is very limited. The contributions are worthwhile but most probably not relevant to the target audience of this venue.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes a RL framework to generate molecules by assembling molecular fragments, such that the ending result has optimized docking scores. The RL model uses docking score as a reward function; an autoregressive policy network which chooses among three possible actions to modify an existing molecule (source attachment site, fragment to attach, target attachment site); a Soft Actor-Critic algorithm to learn the optimal policy, and explores different Prioritized Experience Replay strategies to produce diverse molecules. In the experiments, the authors show that the model can produce molecules with enhanced docking scores for some reference proteins, both in a de novo scenario and scaffold-based scenario. 
main_review: The main idea behind the paper is not novel (using docking scores as a reward function), but the proposed solution is interesting in principle. However, the paper is not written well; there are inconsistencies in notation (e.g. h_g is used for a graph representation, but then ECFP(h_g_{cand}) is used, while ECFP should take as input a molecule, not a representation), confusingly defined symbols (e.g. z_{1st}^{act1}), undefined acronyms (TD, BU, PPO), misspelled acronyms (BE instead of BU), which made reading this paper very difficult. Furthermore, the English language is really faltering sometimes. First and foremost, I suggest that the paper undergoes serious proofreading, otherwise I will be forced to confirm my (temporary) rejection.

On the technical side, I have several concerns: 
- on page 6, the authors affirm that the model has been trained on 3 different proteins. However, the results reported in Table 1 concern only one of them (fa7). I checked the supplementary material (not very carefully, I reckon) and there is no sign of the evaluation on those proteins. Why is that so?
- on Figure 7b)  (and in Figure 7a) for protein 5htb1) the results basically say that the proposed method is no more effective than attaching fragments randomly to generate higher docking molecules, which makes me wonder if using PER strategies is effective at all;
- how are bond types handled by the GCN? Judging from line 150, the GCN doesn't handle edge types, which is quite unlikely.
- What are the novelty rates (i.e. the ratio of generated molecules that are not in the training set) of the model? Why you didn't report them?
- Similarly, the authors use several techniques to improve the diversity when generating molecules. So I would have expected that the chemical diversity of the 3000 sample was measured, when in fact it wasn't.
- When does the generative process stop? I couldn't see it explained on the paper. Perhaps I can find it in the supplementary material? It's information that I would like to read in the paper itself, though.

__Overall judgement__

Right now, I recommend rejection, given the technical doubts but most of all given the poor writing quality. I am open to revising my score, if the authors will put effort into making this paper readable and comprehensible (in which case, I will also go through the supplementary material more carefully).
limitations_and_societal_impact: Limitations are discussed to some extent. One that I find critical for these types of models is computational time. It takes a long time to compute accurate docking simulations. In this paper, the authors use a docking program that is known to produce faster but less reliable docking estimations. However, no one would use this program in a practical scenario; hence, it should be properly discussed by the authors what is the computational burden to train the model for its usage in the real world, not just for experimenting.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents a reinforcement learning approach in which molecule fragmetns are combined together in order to construct realistic and pharmacologically relevant molecules. The approach relies on considerable use of domain knowledge, since the library of fragments that are used to generate the molecules are themselves realistic and pharmacologically relevant and they do not exhibit undesired properties, e.g. toxicity or low reactivity. The overall goal of the method presented in the paper is to learn to generate molecules that will dock to given proteins. With respect to the RL problem the reward that is used is based on the docking score as well as on a pharmacochemical filter scores. This is a hard-exploration learning problem due to very sparse rewards, which according to my understanding are available once all the molecule has been generated. To address the hard-exploration problem the paper relies on a number of variants of SAC which make use of prioritise replay. The paper compares against a couple of baselines with favorable results. My impression is that the main advantage of the method with respect to the baselines is that the library of fragments over which it is operating is carefully selected towards producing molecules that are pharmacoligically relevant and acceptable, and less the algorithmic contributions, e.g prioritised replay. 


main_review: While the paper is in  general easy to ready, I had issues on a number of occasions. For example the I found the discussion of the prioritised reaply methods that the paper used confusing and unclear. I believe I also missed a precise definition of the reward that it was used, in the appendix it is described as "docking score reward + pharmacochemical filter reward" but what are the filters that are used? 

In the experiments the paper compares against two baselines, for which not many details are given except the fact that they are generative models, though in the description in the introduction it seems that they are rather reinforcement learning algorithms. I could not help but wonder whether a quantitative comparison with proper generative models, many of which the paper discusses, would not be possible? Many of these generative models have excellent performance for the tasks on which they have been used/benchmarked (and they are also readily available for conditional generation), though I am not sure any of them has been tried on the docking generation problem. If we were to take a conditional generation approach on the docking docking problem I guess the task would be something like given a particular target learn the distribution of the molecules that dock to it. I am not sure whether this is easy to do, since there would be issues even with the generation of proper training sets. In any case I am not sure that one can so easily dismiss these models, and with respect to that aspect I have the feeling that the set of baselines is not complete, though I admit benchmarking these methods on the docking problem is not obvious. 


lines 96-98: Is there evidence out there that learning to reconstruct might harm diversity? Why having a decoder that is able to reconstruct will have less chances to produce diverse molecules? and by the way the same argument can also be given for the method presented in this paper, i.e. isnt't the fact that the approach is based on fragments limit the diversity of the generated molecules, since these can only be combinations of existing fragments? which in fact seems to be supported by the experiments. 

line 145: I am not sure how should I interpret the "predict a next state"? should it be simply action prediction?

Actions: 
* lines 154-155: What is the action space? The first action selects the docking point in the so far generated molecule, the possibilities can be very different depending on the so far generated molecule, how this is handled? or in other words what is the structure of the output of $\pi_{\text{act1}}$? similar question for second and third action selection. 
Relate to this, in the appendix, line 86 states that the maximum number of actions in an episode is four? does that mean that there only four transitions, i.e. all episodes have a length of four? is there no possibility that after fours steps we might end up with an incomplete molecule?

line 169: for my undestanding, all rewards are zero except the last one which is given by the docking score and the pharmacological filter rewards?

lines 173-174: eq 5: Isn't the expectation missing the $\pi(a|s)$ term ? Also the expectation is approximated by the sample based estimate but it is not equal to it. 
To avoid ambigiuty would be useful to have the two sums, the one that comes from the expecation and the one over the actions for the entropy. 


Hard exploration part: 
* lines 187-193: To address the hard-exploration the paper couples SAC with prioritized replay where the priority of a state is either the predictive error of a reward prediction network
 or the uncertainty of that reward as estimated by a neural net with dropout. Since the priority is based on reward uncertainty shouldn't this be computed based on state-action 
pairs, instead of states? I am not sure I understand what is the reward of a state. By the way the auxiliary network that estimates reward is independent of the critic of SAC? In the appendix 
lines 53-61 (curiosity driven learning) there is a discussion of reward prediction, though only the loss is given, $\hat{y_i}$ is the estimated reward of some given state? and the true reward
of that state will be almost always 0? unless the molecule is complete, in which case it will be the docking score plus the pharma filters?

* lines 194-199: I have to say that here I got confused, I am not sure I see how 1) and 2) relate do they provide two different ways to prioritise experience replay? one seems to focus on states only, 
while in 2) the focus seems to be transitions? By the way a transition is the quadruple (s,a,r,s')? and is the priority of a given transition based on the reward prediction or on the docking score estimation? both are mentioned in the paragraph, and I thought the docking score is only given for complete molecules but the paragraph makes reference to docking score estimation for a given state. 

Experiments:
* baselines: a couple of sentences that describe the type of generative approach that REINVENT and MORLD follow would have been appreciated.
limitations_and_societal_impact: yes they have.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper is concerned about optimizing molecules by assembling fragments using reinforcement learning. The main idea to generate "drug-like" molecules is to generate a molecule by combining chemically realistic and pharmacochemically acceptable fragments, while restricting the attachment sites of the fragments. This generation process coupled with the explorative RL algorithm is the proposed method.

The effectiveness of the proposed method is investigated by comparing it with two existing methods. The experimental results suggest that (i) the molecules generated by the proposed method are more likely to pass the pharmacochemical filters than the existing methods, (ii) the proposed method achieves higher or comparable hit ratio and top 5% scores than the existing methods, (iii) the explorative RL algorithm used in the proposed method is suitable to this task than the other RL methods.

The authors also provide a case study on drug design in both de novo scenario and scaffold-based one.
main_review: # Summary
- I like the authors work on realistic problems rather than synthetic ones such as logP optimization, formally considered in the literature.
- Technically, the proposed method seems to be minor contributions to the literature. For example, the fragment-based generation with connectivity preservation has been introduced in the paper "Molecular Hypergraph Grammar with Its Application to Molecular Optimization" by Kajino (ICML-19), and a similar generation process is coupled with RL in the paper "Reinforced Molecular Optimization with Neighborhood-Controlled Grammars" by Xu et al. (NeurIPS-20). 
- There are two major concerns; see below.

# On the hypothesis
An important hypothesis of the present work is that attaching a chemically realistic and pharmacochemically acceptable fragment units leads to realistic and acceptable molecules. The filters used in the experiments seem to be based on the same hypothesis, but I am curious whether this hypothesis is widely accepted or not. If this hypothesis is just a necessary condition for a molecule to be acceptable, it should be noted as a limitation of this work.

# Comparison to the existing methods
In order to validate the proposed method, it is necessary to empirically compare the proposed method with the existing methods, at least those mentioned below.
- In l.93-95, the authors state that their connectivity-preserving fragmentation and augmentation procedure is beneficial, but this is not obvious and should be empirically confirmed.
- In l.99-107, the authors state that the forward synthesis methods will not be able to handle docking score optimization, but this is also not obvious and should be empirically confirmed.
limitations_and_societal_impact: The fragment-based generation could be limited as compared to the atom-wise generation in that the generated molecules are less diverse; this is because the generated molecules are restricted to the fragment library. So, I would recommend the authors to investigate the diversity of generated molecules.
review_title: null

Meta-review: The submission has its merits but, even after carefully checking the author rebuttal, the reviewers unanimously agree that the submission is not strong enough for publication at NeurIPS. The main reasons for that are the lack of novelty, clarity, and supporting evidence.

Decision: Accept (Poster)