MARS: Markov Molecular Sampling for Multi-objective Drug Discovery

1 INTRODUCTION . Drug discovery aims to find chemical compounds with desired target properties , such as high druglikeness ( Bickerton et al. , 2012 , QED ) . The problem is also referred to as molecular design , molecular generation , or molecular search . The space of drug-like chemicals is enormous , approximate 1033 for realistic drugs that could ever be synthesized ( Polishchuk et al. , 2013 ) . Therefore it is very challenging to search for high-quality molecules from such a vast space — enumeration would take almost forever . For a particular disease , finding the right candidates targeting specific proteins further complicates the problem . Instead of enumerating or searching from the immense chemical space , recent work utilizes deep generative models to generate candidate molecules directly ( Schwalbe-Koda & Gómez-Bombarelli , 2020 ) . However , most prior work focuses on generating molecules concerning a single property such as drug-likeness ( QED ) or octanol-water partition coefficient ( logP ) ( Jin et al. , 2018 ; You et al. , 2018 ; Popova et al. , 2019 ; Shi et al. , 2020 ; Zang & Wang , 2020 ) . While in practical settings , typical drug discovery requires consideration of multiple properties jointly ( Nicolaou et al. , 2012 ) . For example , to find drug-like molecules that are easy to synthesize and exhibit high biological activity against the target protein . Naturally , multi-objective molecule design is much more challenging than the single-objective scenario ( Jin et al. , 2020 ) . This paper studies the problem of multi-objective molecule design for drug discovery . An ideal solution should be efficient and meet the following criteria . C1 : It should satisfy multiple properties with high scores ; C2 : It should produce novel and diverse molecules ; C3 : Its generation process does not ∗Work was done while Yutong Xie and Chence Shi were research interns at ByteDance . Corresponding to : zhouhao.nlp @ bytedance.com and lileilab @ bytedance.com . rely on either expert annotated or wet experimental data collected from a biochemistry lab ( since it requires tremendous effort and hard to obtain ) . Existing molecule generation approaches are mainly designed for the single objective setting , and they could not meet all criteria in the setting of multiple objectives . These methods belong to four categories : a ) generating candidates from a learned continuous latent space ( Gómez-Bombarelli et al. , 2018 ; Jin et al. , 2018 ) , b ) through reinforcement learning ( You et al. , 2018 ) , c ) using an encoder-decoder translation approach ( Jin et al. , 2019 ) , or d ) optimizing molecular properties through genetic algorithms ( Nigam et al. , 2020 ) . Current stateof-the-art multi-objective molecular generation is a rationale-based method ( Jin et al. , 2020 ) . In this approach , the authors propose to build molecules by composing multiple extracted rationales , and the model can generate compounds that are simultaneously active to multiple biological targets . However , such an approach will result in quite complex molecules when we have many objectives . This is because different objectives correspond to different rationales , and including all these rationales could lead to large molecules , which may be less drug-like and hard to be synthesized practically . In this paper , we propose MArkov moleculaR Sampling ( MARS ) , a simple yet flexible method for drug discovery . The basic idea is to start from a seed molecule and keep generating candidate molecules by modifying fragments of molecular graphs from previous steps . It meets all the criteria C1-3 . In MARS , the molecular design is formulated as an iterative editing procedure with its total objective consisting of multiple property scores ( C1 ) . MARS employs the annealed Markov chain Monte Carlo sampling method to search for optimal chemical compounds , which allows for the exploration of chemicals with novel and different fragments ( C2 ) . The proposal to modify molecular fragments is represented using graph neural networks ( GNNs ) , whose parameters are adaptively learned . We used message passing neural networks ( MPNNs ) in practice ( Gilmer et al. , 2017 ) , but other GNNs can fit the framework as well . Furthermore , MARS utilizes the sample paths generated on-the-fly to train the proposal network adaptively . Therefore , it does not rely on external annotated data ( C3 ) . With such an adaptive learnable proposal , it keeps improving the generation quality throughout the process . We evaluate MARS and four other baselines , one latest method for each of the four method categories . The benchmark includes a variety of multi-objective generation settings . Experiments show that our proposed MARS achieves state-of-the-art performance on five out of six tasks in terms of a comprehensive evaluation consisting of the success rate , novelty , and diversity of the generated molecules . Notably , in the most challenging setting where four objectives – bio-activities to two different targets , drug-likeness , and synthesizability – are simultaneously considered , our method achieves the state-of-the-art result and outperforms existing methods by 77 % in the comprehensive evaluation . Our contributions are as follows : • We present MARS , a generic formulation of molecular design using Markov sampling , which can easily accommodate multiple objectives . • We develop an adaptive fragment-editing proposal based on GNN that is learnable on the fly with only samples self-generated and efficient in exploring the chemical space . • Experiments verifies our proposed MARS framework can find novel and diverse bioactive molecules that are both drug-like and highly synthesizable . 2 RELATED WORK . Recent years have witnessed the success of applying deep generative models and molecular graph representation learning in drug discovery ( Schwalbe-Koda & Gómez-Bombarelli , 2020 ; Guo & Zhao , 2020 ) . Existing approaches for molecular property optimization can be grouped into four categories , including generation with a ) Bayesian inference , b ) reinforcement learning , c ) encoderdecoder translation models , and d ) evolutionary and genetic algorithms . The first category is learning continuous latent spaces for molecular sequences or graphs and generating from such spaces using Bayesian optimization ( BO ) ( Gómez-Bombarelli et al. , 2018 ; Jin et al. , 2018 ; Winter et al. , 2019 ) . These methods rely heavily on the quality of latent representations , which imposes huge challenges to the encoders when there are multiple properties to consider . Unlike the first class , other work uses reinforcement learning ( RL ) to optimize desired objectives directly in the explicit chemical space ( De Cao & Kipf , 2018 ; Popova et al. , 2018 ; You et al. , 2018 ; Popova et al. , 2019 ; Shi et al. , 2020 ) . However , the models are usually hard to train due to the high variance of RL . The third category directly trains a translation model that maps from an input molecule to a highquality output molecule ( Jin et al. , 2019 ; 2020 ) . Although simple , such methods require many high-quality labeled data , making them impractical in scenarios where the data is limited . The last category of methods are evolutionary algorithms ( EAs ) and genetic algorithms ( GAs ) to explore large chemical space with certain property ( Nicolaou et al. , 2012 ; Devi et al. , 2015 ; Jensen , 2019 ; Ahn et al. , 2020 ) . In Nigam et al . ( 2020 ) , the authors propose to augment GA by adding an adversarial loss into the fitness evaluation to increase the diversity , and the augmented GA outperforms all other generative models in optimizing logP . Though flexible and straightforward , to make the search process efficient enough , most GA and EA methods require domain experts to design molecular mutation and crossover rules , which could be non-trivial to obtain . Besides single property optimization , there is recent work to address the multi-objective molecule generation problem . For example , Li et al . ( 2018 ) proposes to use a conditional generative model to incorporate several objectives flexibly , while Lim et al . ( 2020 ) leverages molecular scaffolds to control the properties of generated molecules better . Among them , the current state-of-the-art approach is a rationale-based method proposed by Jin et al . ( 2020 ) . In this method , the authors propose to build molecules by assembling extracted rationales . Despite its great success in generating compounds simultaneously active to multiple biological targets , the combination of rationales might hinder the synthesizability and drug-likeness of produced molecules , as they tend to be large as the number of objectives grows . In contrast , our MARS framework turns the generation problem into a sampling procedure , which serves as an alternative way compared with deep generative models , and can efficiently discover bio-active molecules that are both drug-like and highly synthesizable . Remotely related is recent work to generate molecules through sampling . Seff et al . ( 2019 ) defines a Gibbs sampling procedure , in which the Markov chain alternates between randomly corrupting the molecules and recovering the corrupted ones with a learned reconstruction model . However , this method mainly focuses on generating molecules that follow the observed data distribution and can not be directly tailored for property optimization . Different from this work , MARS is built upon the general MCMC sampling framework , which allows further enhancement with adaptive proposal learning to edit molecular graphs efficiently . Actually , generating instances from a discrete space with MCMC sampling methods is previously employed in various other applications , e.g. , generating natural language sentences under various constraints ( Miao et al. , 2019 ; Zhang et al. , 2019 ; Liu et al. , 2020 ; Zhang et al. , 2020 ) . 3 PROPOSED MARS APPROACH . In this section , we present the MArkov moleculaR Sampling method ( MARS ) for multi-objective molecular design . We define a Markov chain over the explicit molecular graph space and design a kernel to navigate high probable candidates with acceptance and rejection . 3.1 SAMPLING FROM THE MOLECULAR SPACE . Our proposed MARS framework aims at sampling molecules with desired properties from the chemical space . Specifically , given K properties of interest , the desired molecular distribution can be formulated as a combination of all objectives : π ( x ) = s1 ( x ) ◦ s2 ( x ) ◦ s3 ( x ) ◦ · · · ◦ sK ( x ) ︸ ︷︷ ︸ desired properties ( 1 ) where x is a molecule in the molecular spaceX . π ( x ) is an unnormalized distribution over molecules integrating the desired properties . sk ( x ) is a scoring function for the k-th property and the “ ◦ ” operator stands for a combination of scores ( e.g. , summation or multiplication ) . In practical drug discovery , these terms could be related to the biological activity , drug-likeness , and synthesizability of molecules ( Nicolaou et al. , 2012 ) . This framework allows flexible configuration according to various concrete applications . However , as the number of objectives grows , the joint distribution π ( x ) will become more complex and intractable , making the sampling non-trivial . In MARS , we propose to sample molecules from the desired distribution Eq . 1 using Markov chain Monte Carlo ( MCMC ) methods ( Andrieu et al. , 2003 ) . Given a desired molecular distribution π ( x ) as the unnormalized target distribution , we define a Markov chain on the explicit chemical space X ( i.e. , each state of the Markov chain is a particular molecule ) and introduce a proposal distribution q ( x′ | x ) to perform state transitions . Specifically , as shown in Figure 1 , the sampling procedure of MARS starts from an initial molecule x ( 0 ) ∈ X . At each time step t , a molecule candidate x′ ∈ X will be sampled from the proposal distribution q ( x′ | x ( t−1 ) ) , where x ( t−1 ) denotes the molecule at time step t− 1 . Then the proposed candidate x′ could be either accepted x ( t ) = x′ or rejected x ( t ) = x ( t−1 ) according to the acceptance rate A ( x ( t−1 ) , x′ ) ∈ [ 0 , 1 ] controlled by the target distribution π ( x ) . By repeating this process , a sequence of molecules { x ( t ) } ∞t=0 can be generated . Such sequence of molecules will converge to the target distribution π ( x ) if the proposal distribution and the acceptance mechanism are configured properly . The acceptance rate is calculated as follow : A ( x , x′ ) = min { 1 , πα ( x′ ) q ( x|x′ ) πα ( x ) q ( x′|x ) } ( 2 ) where α is a coefficient that varies in different instantiations of MCMC algorithms . Here to find molecules that globally maximize the target distribution , we employ an annealing scheme ( Laarhoven & Aarts , 1987 ) where α = 1/T and T is a temperature controlled by a cooling schedule . In addition to this , other instantiations such as Metropolis-Hastings ( MH ) algorithm ( Metropolis et al. , 1953 ) where α = 1 are also feasible under our general framework . As for the proposal distribution q ( x′ | x ) , it largely affects the sampling performance and should be designed elaborately . In general , it is crucial that the proposal distribution q ( x′ | x ) and the target distribution π ( x′ ) are as close as possible to ensure high sampling efficiency . So we propose using a proposal distribution qθ ( x′ | x ) with learnable parameters to capture the desired molecular properties and develop a strategy to train the proposal throughout the sampling process adaptively . The details will be described in the next section .

Summary: This paper proposes a method to generate molecular graphs with multiple optimized properties. Molecular graphs are constructed/edited by the iterative addition and removal of molecular fragments. A MCMC search procedure, guided by a learned graph neural network that proposes good graph edit actions, is used to sample molecules with optimized properties. The proposed model is compared with some baselines on a few multi-objective optimization tasks and shows good performance.