MARS: Markov Molecular Sampling for Multi-objective Drug Discovery

1 INTRODUCTION . Drug discovery aims to find chemical compounds with desired target properties , such as high druglikeness ( Bickerton et al. , 2012 , QED ) . The problem is also referred to as molecular design , molecular generation , or molecular search . The space of drug-like chemicals is enormous , approximate 1033 for realistic drugs that could ever be synthesized ( Polishchuk et al. , 2013 ) . Therefore it is very challenging to search for high-quality molecules from such a vast space — enumeration would take almost forever . For a particular disease , finding the right candidates targeting specific proteins further complicates the problem . Instead of enumerating or searching from the immense chemical space , recent work utilizes deep generative models to generate candidate molecules directly ( Schwalbe-Koda & Gómez-Bombarelli , 2020 ) . However , most prior work focuses on generating molecules concerning a single property such as drug-likeness ( QED ) or octanol-water partition coefficient ( logP ) ( Jin et al. , 2018 ; You et al. , 2018 ; Popova et al. , 2019 ; Shi et al. , 2020 ; Zang & Wang , 2020 ) . While in practical settings , typical drug discovery requires consideration of multiple properties jointly ( Nicolaou et al. , 2012 ) . For example , to find drug-like molecules that are easy to synthesize and exhibit high biological activity against the target protein . Naturally , multi-objective molecule design is much more challenging than the single-objective scenario ( Jin et al. , 2020 ) . This paper studies the problem of multi-objective molecule design for drug discovery . An ideal solution should be efficient and meet the following criteria . C1 : It should satisfy multiple properties with high scores ; C2 : It should produce novel and diverse molecules ; C3 : Its generation process does not ∗Work was done while Yutong Xie and Chence Shi were research interns at ByteDance . Corresponding to : zhouhao.nlp @ bytedance.com and lileilab @ bytedance.com . rely on either expert annotated or wet experimental data collected from a biochemistry lab ( since it requires tremendous effort and hard to obtain ) . Existing molecule generation approaches are mainly designed for the single objective setting , and they could not meet all criteria in the setting of multiple objectives . These methods belong to four categories : a ) generating candidates from a learned continuous latent space ( Gómez-Bombarelli et al. , 2018 ; Jin et al. , 2018 ) , b ) through reinforcement learning ( You et al. , 2018 ) , c ) using an encoder-decoder translation approach ( Jin et al. , 2019 ) , or d ) optimizing molecular properties through genetic algorithms ( Nigam et al. , 2020 ) . Current stateof-the-art multi-objective molecular generation is a rationale-based method ( Jin et al. , 2020 ) . In this approach , the authors propose to build molecules by composing multiple extracted rationales , and the model can generate compounds that are simultaneously active to multiple biological targets . However , such an approach will result in quite complex molecules when we have many objectives . This is because different objectives correspond to different rationales , and including all these rationales could lead to large molecules , which may be less drug-like and hard to be synthesized practically . In this paper , we propose MArkov moleculaR Sampling ( MARS ) , a simple yet flexible method for drug discovery . The basic idea is to start from a seed molecule and keep generating candidate molecules by modifying fragments of molecular graphs from previous steps . It meets all the criteria C1-3 . In MARS , the molecular design is formulated as an iterative editing procedure with its total objective consisting of multiple property scores ( C1 ) . MARS employs the annealed Markov chain Monte Carlo sampling method to search for optimal chemical compounds , which allows for the exploration of chemicals with novel and different fragments ( C2 ) . The proposal to modify molecular fragments is represented using graph neural networks ( GNNs ) , whose parameters are adaptively learned . We used message passing neural networks ( MPNNs ) in practice ( Gilmer et al. , 2017 ) , but other GNNs can fit the framework as well . Furthermore , MARS utilizes the sample paths generated on-the-fly to train the proposal network adaptively . Therefore , it does not rely on external annotated data ( C3 ) . With such an adaptive learnable proposal , it keeps improving the generation quality throughout the process . We evaluate MARS and four other baselines , one latest method for each of the four method categories . The benchmark includes a variety of multi-objective generation settings . Experiments show that our proposed MARS achieves state-of-the-art performance on five out of six tasks in terms of a comprehensive evaluation consisting of the success rate , novelty , and diversity of the generated molecules . Notably , in the most challenging setting where four objectives – bio-activities to two different targets , drug-likeness , and synthesizability – are simultaneously considered , our method achieves the state-of-the-art result and outperforms existing methods by 77 % in the comprehensive evaluation . Our contributions are as follows : • We present MARS , a generic formulation of molecular design using Markov sampling , which can easily accommodate multiple objectives . • We develop an adaptive fragment-editing proposal based on GNN that is learnable on the fly with only samples self-generated and efficient in exploring the chemical space . • Experiments verifies our proposed MARS framework can find novel and diverse bioactive molecules that are both drug-like and highly synthesizable . 2 RELATED WORK . Recent years have witnessed the success of applying deep generative models and molecular graph representation learning in drug discovery ( Schwalbe-Koda & Gómez-Bombarelli , 2020 ; Guo & Zhao , 2020 ) . Existing approaches for molecular property optimization can be grouped into four categories , including generation with a ) Bayesian inference , b ) reinforcement learning , c ) encoderdecoder translation models , and d ) evolutionary and genetic algorithms . The first category is learning continuous latent spaces for molecular sequences or graphs and generating from such spaces using Bayesian optimization ( BO ) ( Gómez-Bombarelli et al. , 2018 ; Jin et al. , 2018 ; Winter et al. , 2019 ) . These methods rely heavily on the quality of latent representations , which imposes huge challenges to the encoders when there are multiple properties to consider . Unlike the first class , other work uses reinforcement learning ( RL ) to optimize desired objectives directly in the explicit chemical space ( De Cao & Kipf , 2018 ; Popova et al. , 2018 ; You et al. , 2018 ; Popova et al. , 2019 ; Shi et al. , 2020 ) . However , the models are usually hard to train due to the high variance of RL . The third category directly trains a translation model that maps from an input molecule to a highquality output molecule ( Jin et al. , 2019 ; 2020 ) . Although simple , such methods require many high-quality labeled data , making them impractical in scenarios where the data is limited . The last category of methods are evolutionary algorithms ( EAs ) and genetic algorithms ( GAs ) to explore large chemical space with certain property ( Nicolaou et al. , 2012 ; Devi et al. , 2015 ; Jensen , 2019 ; Ahn et al. , 2020 ) . In Nigam et al . ( 2020 ) , the authors propose to augment GA by adding an adversarial loss into the fitness evaluation to increase the diversity , and the augmented GA outperforms all other generative models in optimizing logP . Though flexible and straightforward , to make the search process efficient enough , most GA and EA methods require domain experts to design molecular mutation and crossover rules , which could be non-trivial to obtain . Besides single property optimization , there is recent work to address the multi-objective molecule generation problem . For example , Li et al . ( 2018 ) proposes to use a conditional generative model to incorporate several objectives flexibly , while Lim et al . ( 2020 ) leverages molecular scaffolds to control the properties of generated molecules better . Among them , the current state-of-the-art approach is a rationale-based method proposed by Jin et al . ( 2020 ) . In this method , the authors propose to build molecules by assembling extracted rationales . Despite its great success in generating compounds simultaneously active to multiple biological targets , the combination of rationales might hinder the synthesizability and drug-likeness of produced molecules , as they tend to be large as the number of objectives grows . In contrast , our MARS framework turns the generation problem into a sampling procedure , which serves as an alternative way compared with deep generative models , and can efficiently discover bio-active molecules that are both drug-like and highly synthesizable . Remotely related is recent work to generate molecules through sampling . Seff et al . ( 2019 ) defines a Gibbs sampling procedure , in which the Markov chain alternates between randomly corrupting the molecules and recovering the corrupted ones with a learned reconstruction model . However , this method mainly focuses on generating molecules that follow the observed data distribution and can not be directly tailored for property optimization . Different from this work , MARS is built upon the general MCMC sampling framework , which allows further enhancement with adaptive proposal learning to edit molecular graphs efficiently . Actually , generating instances from a discrete space with MCMC sampling methods is previously employed in various other applications , e.g. , generating natural language sentences under various constraints ( Miao et al. , 2019 ; Zhang et al. , 2019 ; Liu et al. , 2020 ; Zhang et al. , 2020 ) . 3 PROPOSED MARS APPROACH . In this section , we present the MArkov moleculaR Sampling method ( MARS ) for multi-objective molecular design . We define a Markov chain over the explicit molecular graph space and design a kernel to navigate high probable candidates with acceptance and rejection . 3.1 SAMPLING FROM THE MOLECULAR SPACE . Our proposed MARS framework aims at sampling molecules with desired properties from the chemical space . Specifically , given K properties of interest , the desired molecular distribution can be formulated as a combination of all objectives : π ( x ) = s1 ( x ) ◦ s2 ( x ) ◦ s3 ( x ) ◦ · · · ◦ sK ( x ) ︸ ︷︷ ︸ desired properties ( 1 ) where x is a molecule in the molecular spaceX . π ( x ) is an unnormalized distribution over molecules integrating the desired properties . sk ( x ) is a scoring function for the k-th property and the “ ◦ ” operator stands for a combination of scores ( e.g. , summation or multiplication ) . In practical drug discovery , these terms could be related to the biological activity , drug-likeness , and synthesizability of molecules ( Nicolaou et al. , 2012 ) . This framework allows flexible configuration according to various concrete applications . However , as the number of objectives grows , the joint distribution π ( x ) will become more complex and intractable , making the sampling non-trivial . In MARS , we propose to sample molecules from the desired distribution Eq . 1 using Markov chain Monte Carlo ( MCMC ) methods ( Andrieu et al. , 2003 ) . Given a desired molecular distribution π ( x ) as the unnormalized target distribution , we define a Markov chain on the explicit chemical space X ( i.e. , each state of the Markov chain is a particular molecule ) and introduce a proposal distribution q ( x′ | x ) to perform state transitions . Specifically , as shown in Figure 1 , the sampling procedure of MARS starts from an initial molecule x ( 0 ) ∈ X . At each time step t , a molecule candidate x′ ∈ X will be sampled from the proposal distribution q ( x′ | x ( t−1 ) ) , where x ( t−1 ) denotes the molecule at time step t− 1 . Then the proposed candidate x′ could be either accepted x ( t ) = x′ or rejected x ( t ) = x ( t−1 ) according to the acceptance rate A ( x ( t−1 ) , x′ ) ∈ [ 0 , 1 ] controlled by the target distribution π ( x ) . By repeating this process , a sequence of molecules { x ( t ) } ∞t=0 can be generated . Such sequence of molecules will converge to the target distribution π ( x ) if the proposal distribution and the acceptance mechanism are configured properly . The acceptance rate is calculated as follow : A ( x , x′ ) = min { 1 , πα ( x′ ) q ( x|x′ ) πα ( x ) q ( x′|x ) } ( 2 ) where α is a coefficient that varies in different instantiations of MCMC algorithms . Here to find molecules that globally maximize the target distribution , we employ an annealing scheme ( Laarhoven & Aarts , 1987 ) where α = 1/T and T is a temperature controlled by a cooling schedule . In addition to this , other instantiations such as Metropolis-Hastings ( MH ) algorithm ( Metropolis et al. , 1953 ) where α = 1 are also feasible under our general framework . As for the proposal distribution q ( x′ | x ) , it largely affects the sampling performance and should be designed elaborately . In general , it is crucial that the proposal distribution q ( x′ | x ) and the target distribution π ( x′ ) are as close as possible to ensure high sampling efficiency . So we propose using a proposal distribution qθ ( x′ | x ) with learnable parameters to capture the desired molecular properties and develop a strategy to train the proposal throughout the sampling process adaptively . The details will be described in the next section .

Summary: The authors propose a novel way to generate molecules with specified objectives, named MArkov moleculaR Sampling (MARS). The idea of MARS is based on generating the chemical candidates by iterative editing fragments of molecular graphs. To transform a molecule x into another molecule x′, the authors considers two sets of graph editing actions fragment adding and fragment deleting, where fragments are connected components in molecules separated by single bonds. To generate the molecules with desired objectives, MARS is using Markov chain Monte Carlo sampling with specified annealing scheme, together with graph convolutional neural network. The results reported in the following paper are very promising and show that this could be a good direction in the area of multi-objective molecules optimization.