Exploring Deep Recurrent Models with Reinforcement Learning for Molecule Design

Abstract: The design of small molecules with bespoke properties is of central importance to drug discovery.  However significant challenges yet remain for computational methods, despite recent advances such as deep recurrent networks and reinforcement learning strategies for sequence generation, and it can be difficult to compare results across different works.  This work proposes 19 benchmarks selected by subject experts, expands smaller datasets previously used to approximately 1.1 million training molecules, and explores how to apply new reinforcement learning techniques effectively for molecular design.  The benchmarks here, built as OpenAI Gym environments, will be open-sourced to encourage innovation in molecular design algorithms and to enable usage by those without a background in chemistry.  Finally, this work explores recent development in reinforcement-learning methods with excellent sample complexity (the A2C and PPO algorithms) and investigates their behavior in molecular generation, demonstrating significant performance gains compared to standard reinforcement learning techniques.

Reviews and author responses:

The first version our paper was uploaded to openreview on 27th Oct 2017.

The paper by Popova et al. was put on arXiv on 29th Nov 2017, that is about a month later than our paper.

There is a recently published paper on molecular design with deep reinforcement learning which is not addressed in your work:
Popova, Mariya, Olexandr Isayev, and Alexander Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).
This paper discusses alternative to GANs method of novel compounds generation with recurrent neural network and reinforce algorithm, which is relevant to your work. 

The paper proposes a set of benchmarks for molecular design, and compares different deep models against them. The main contributions of the paper are 19 molecular design benchmarks (with chembl-23 dataset), including two molecular design evaluation criterias and comparison of some deep models using these benchmarks. The paper does not seem to include any method development.

The paper suffers from a lack of focus. Several existing models are discussed to some length, while the benchmarks are introduced quite shortly. The dataset is not very clearly defined: it seems that there are 1.2 million training instance, does this apply for all benchmarks? The paper's title also does not seem to fit: this feels like a survey paper, which is not reflected in the title. Biologically lots of important atoms are excluded from the dataset, for instance natrium, calcium and kalium. I don't see any reason to exlude these. What does "biological activities on 11538 targets" mean? 

The paper discussed molecular generation and reinforcement learning, but it is somewhat unclear how it relates to the proposed dataset since a standard training/test setting is used. Are the test molecules somehow generated in a directed or undirected fashion? Shouldn't there also be experiments on comparing ways to generate suitable molecules, and how well they match the proposed criterion? There should be benchmarks for predicting molecular properties (standard regression), and for generating molecules with certain properties. Currently it's unclear which type of problems are solved here.

Table 1 lists 5 models, while fig 3 contains 7, why the discrepancy? In table 1 the plotted runs seem to differ a lot from average results (e.g. -0.43 to 0.15, or 0.32 to 0.83). Variances should be added, and preferably more than 3 initialisations used.

Overall this is an interesting paper, but does not have any methodological contribution, and there is also few insightful results about the compared methods, nor is there meaningful analysis of the problem domain of molecules either.


Summary:
This work is about model evaluation for molecule generation and design. 19 benchmarks are proposed, small data sets are expanded to a large, standardized data set and it is explored how to apply new RL techniques effectively for molecular design.

on the positive side:
The paper is well written, quality and clarity of the work are good. The work provides a good overview about how to apply new reinforcement learning techniques for sequence generation. It is investigated how several RL strategies perform on a large, standardized data set. Different RL models like Hillclimb-MLE, PPO, GAN, A2C are investigated and discussed.  An implementation of 19 suggested benchmarks of relevance for de novo design will be provided as open source as an OpenAI Gym. 


on the negative side:
There is no new novel contribution on the methods side.  



minor comments:

Section 2.1. 
see Fig.2 —> see Fig.1
page 4just before equation 8: the the

Summary: This paper studies a series of reinforcement learning (RL) techniques in combination with recurrent neural networks (RNNs) to model and synthesise molecules. The experiments seem extensive, using many recently proposed RL methods, and show that most sophisticated RL methods are less effective than the simple hill-climbing technique, with PPO is perhaps the only exception.  

Originality and significance: 

The conclusion from the experiments could be valuable to the broader sequence generation/synthesis field, showing that many current RL techniques can fail dramatically. 

The paper does not provide any theoretical contribution but nevertheless is a good application paper combining and comparing different techniques.

Clarity: The paper is generally well-written. However, I'm not an expert in molecule design, so might not have caught any trivial errors in the experimental set-up. 


We thank the reviewer for their critical feedback. We have adapted the manuscript to make the contributions and scope clearer.

Algorithms that allow the generation of small molecules (small graphs) that satisfy given desirable properties are rapidly evolving for medicine, materials and agriculture.  This is currently an area of intense investigation, theoretically as well as practically, as highlighted by several ICLR submissions on molecule/graph generation this year, which all employ different and inconsistent benchmarks and training sets. Currently, it is not possible to compare these results.

With this paper, we unify the problem space of small molecule generation and thoroughly investigate various approaches (including algorithms which have yet to be explored in this domain, e.g. PPO), and include evaluations of our new benchmarks on pre-existing work.

Our results surpass state-of-the-art results previously reported on a few of the sub-domains, establishing new baselines, and come to the perhaps surprising result that the relatively simple hill-climbing MLE method achieves results on par with the some of the most advanced recently-developed RL algorithms such as PPO.


Specific Comments:

* Regarding lack of focus: we have reorganized the paper to improve the clarity of the work.
  
* Regarding purpose: We hope the above responses address your concern of this being a survey work. We believe this work introduces new benchmarks, evaluates pre-existing algorithms, demonstrates novel pairings of algorithm and domain, and establishes a new state-of-the art as well as a few surprising additional insights (hill-climbing MLE supremacy, temperature ineffectiveness).

* Regarding preprocessing steps: the steps taken in this work are standard and in line with the field of computational chemistry [1,2]. This includes the removal of Sodium, Calcium and Potassium, and other counterions.

* Regarding clarity of RL vs. train/test set: for algorithms that rely on pretraining to help navigate the extremely large space of small molecule generation (~10^60), it is important that the algorithms have not been exposed to a correct solution in the training set, hence the train-test split.  As an additional benefit, a train/test split permits the benchmarks to be used with rule-based GOFAI systems, supervised algorithms as well as RL.

* Regarding molecular property prediction: indeed, this is an important sub-field of computational chemistry and is explored under the family of QSAR models [3, 4].  However, that is out-of-scope of this paper, as it attempts to address a different concern.

* Regarding data: the table is a bit overwhelming already, so we chose not to exhaustively show all results for all models and instead focus on representative key models.  Due to time and computational constraints, we had not run more than three initializations, but can do so for the revision.


We hope this addresses your concerns.

[1] Glaab, Enrico. "Building a virtual ligand screening pipeline using free software: a survey." Briefings in bioinformatics 17.2 (2015): 352-366.
[2] Lionta, Evanthia, et al. "Structure-based virtual screening for drug discovery: principles, applications and recent advances." Current topics in medicinal chemistry 14.16 (2014): 1923-1938.
[3] Tropsha, Alexander. "Best practices for QSAR model development, validation, and exploitation." Molecular informatics 29.6‐7 (2010): 476-488.
[4] Tropsha, Alexander, and Alexander Golbraikh. "Predictive QSAR modeling workflow, model applicability domains, and virtual screening." Current pharmaceutical design 13.34 (2007): 3494-3504.



We are grateful for your comments.  We hope your concern about novelty is addressed with our main comment; indeed, the pairing here is in the algorithm to this particular application area.  
We further hope that a foundational framework proposed will allow the emergence of future, novel algorithms.  Your minor comments have been addressed in the manuscript.



We thank the reviewers for their effort and advice towards improving our submission.  We are pleased that the reviewers have identified our important contributions of dataset curation and preprocessing steps, proposed benchmarks, and baseline results using recently-developed algorithms.  While we introduce no new reinforcement learning algorithms in this work, our primary aim was to substantially lower the barrier towards automated molecular design to allow computer scientists with no prior background in chemistry to develop novel algorithms to improve molecular design.  Indeed, here the novelty lies in the pairing of task and algorithm, and this work is foundational to clearly lay out steps and provide code to apply reinforcement learning algorithms to molecule design.  

Finally, we are able to demonstrate results in this manuscript that establish a new state-of-the-art in single and multi objective physicochemical property optimization and chemical space exploration tasks.  Subsequent work can then build on this set of standardized molecular design benchmarks to introduce new methods. The benchmark framework is general enough to be used with any possible small molecule generation method, whether rule-based or learned, and is not limited to sequence-based generation relying on SMILES. We have therefore amended the manuscript to reflect the importance of our introduced benchmarks.


We thank the referee for their comments and perspective on our work. 

We hope this reviewer’s comments have been addressed in our overall reply and in the responses to the other reviewers.

Meta-review: The paper creates a dataset for exploration of RL for molecular design and I think this makes it a strong contribution to the community at the intersection of the two. For a methods focussed conference such as ICLR however, it may not be the best fit. Hence I would recommend submitting to a workshop track or targeting a more focussed venue such as a bioinformatics conference. 

Decision: workshop-papers