Spatial Graph Attention and Curiosity-driven Policy for Antiviral Drug Discovery
Authors: Yulun Wu
Nicholas Choma
Andrew Deru Chen
Mikaela Cashman
Erica Teixeira Prates
Veronica G Melesse Vergara
Manesh B Shah
Austin Clyde
Thomas Brettin
Wibe Albert de Jong
Neeraj Kumar
Martha S Head
Rick L. Stevens
Peter Nugent
Daniel A Jacobson
James B Brown
Venue: ICLR 2022 Poster (2022)
Keywords: reinforcement learning
graph neural network
molecule generation
drug discovery
curiosity-driven policy

Abstract: We developed Distilled Graph Attention Policy Network (DGAPN), a reinforcement learning model to generate novel graph-structured chemical representations that optimize user-defined objectives by efficiently navigating a physically constrained domain. The framework is examined on the task of generating molecules that are designed to bind, noncovalently, to functional sites of SARS-CoV-2 proteins. We present a spatial Graph Attention (sGAT) mechanism that leverages self-attention over both node and edge attributes as well as encoding the spatial structure --- this capability is of considerable interest in synthetic biology and drug discovery. An attentional policy network is introduced to learn the decision rules for a dynamic, fragment-based chemical environment, and state-of-the-art policy gradient techniques are employed to train the network with stability. Exploration is driven by the stochasticity of the action space design and the innovation reward bonuses learned and proposed by random network distillation. In experiments, our framework achieved outstanding results compared to state-of-the-art algorithms, while reducing the complexity of paths to chemical synthesis.

Published as a conference paper at ICLR 2022
SPATIAL GRAPH ATTENTION AND CURIOSITY-DRIVEN POLICY FOR ANTIVIRAL DRUG DISCOVERY
Yulun Wu yu un wugberke ey-
Mikaela Cashman
Nicholas Choma 9 in jahoma b .gov
Érica T. Prates
caa nnm@orn -oov
tel xed rapirae pappreeerro Bornn .gov
Verónica Melesse Vergara
Manesh Shah
Andrew Chen
Austin Clyde It
vergazavgBozul .gog
Shanm gov
adchen81bl.gg 1e1l.g1loo
aclyde@uchicago.co
Thomas S. Brettin 38
Wibe A. de Jong
Neeraj Kumar 1t
Martha Head 13
brettinganl, .gov gov
jane-ong@! bll blov
Jemaa répnn1 soppnl.gov
headmagornl .oov
Rick L. Stevens
Peter Nugent
Daniel A. Jacobson
James B. Brown
stevens@anl.
penugent 0 01b1. gov
jacobsondaEornl.co gov
jlbrown&1bl..g
ABSTRACT
We developed Distilled Graph Attention Policy Network (DGAPN), reinforce- ment learning model to generate novel graph-structured chemical representations that optimize user-defined objectives by efficiently navigating a physically con- strained domain. The framework is examined on the task of generating molecules that designed to bind, noncovalently, functional sites of SARS-CoVe pro- teins. We present a spatial Graph Attention (sGAT) mechanism that leverages self-attention over both node and edge attributes as well as encoding the spatial structure - this capability is of considerable interest in synthetic biology and drug discovery. An attentional policy network is introduced learn the decision rules for dynamic, tragment-bassee chemical environment, and state-of of of-the-ar art policy gradient techniques are employed train the network with stability. Exploration is driven by the stochasticity of the action space design and the innovation reward bonuses learned and proposed by random network distillation. experiments, our framework achieved outstanding results compared to state-of-the-ar algorithmss while reducing the complexity paths to chemical synthesis.
INTRODUCTION
This work aims to address the challenge of establishing automated process for the design of ob- jects with connected components, such as molecules, that optimize specific properties. Achieving this goal is particularly desirable in drug development and materials science, where manual discov- ery remains a time-consuming and expensive process (Hughes et al., 2011 Schneider et al.) 2020). However, there are two major difficulties that have long impeded rapid progress, Firstly, the chem- ical space is discrete and massive (Polishchuk et al. 2013), presenting a complicated environment for an Artificial Intelligence (AI) approach efficiently and effectively explore. Secondly, it is not trivial to compress such connected objects into feature representations that preserve most of the information, while also being highly computable for Deep Learning (DL) methods to exploit.
We introduce Distilled Graph Attention Policy Network (DGAPN), a framework that advances prior work in addressing both of these challenges. We present a Reinforcement Learning (RL) architecture that is efficiently encouraged to take innovative actions with environment that is able to construct
f University of California, Berkeley, $ National Virtual Biotechnology Laboratory, US Department of Energy. Lawrence Berkeley National Laboratory. versity of Tennessce, Knoxville, tt University of Chicago. Pacific Northwest National Laboratory
Oak Ridge National Laboratory. Uni- $5 Argonne National Laboratory, tt

---

Published as a conference paper at ICLR 2022
dynamic and chemically valid fragment-based action space. We also propose hybrid Graph Neural Network (GNN) that comprehensively encodes graph objects' attributes and spatial structures in addition to adjacency structures. The following paragraphs discuss how we addressed limitations of prior work and its relevance to antiviral drug discovery. For more descriptions of key prior methodologies that we used benchmarks in this paper, see Section
Graph Representation Learning Despite their spatial efficiency, string representation of molecules acquired by the simplified molecular-input line-entry system (SMILES) (Weininger) 1988) suffers from significant information loss and poor robustness (Liu et al. 2017). Graph repre- sentations have become predominant and preferable for their ability to efficiently encode an object's scaffold structure and attributes. Graph representations are particularly ideal for RL since inter- mediate representations can be decoded and evaluated for reward assignments. While GNNs such as Graph Convolutional Networks (GCN) (Kipf & Welling 2016) and Graph Attention Networks (GAT) (Veličković et al., 2017) have demonstrated impressive performance on many DL tasks, fur- ther exploitation into richer information contained in graph-structured data is needed to faithfully represent the complexity of chemical space (Morris et al. 2019 Wang et al., 2019 Chen et al.) 2020). In this work, we made improvements to previous studies on attributes encoding and struc- tural encoding. For structural encoding, previous studies have covered adjacency distance encoding eLi al. (020). spatial cutoff (Pei al., (2020) and coordinates encoding (Schütt et al. 2017] Danel et a (020). Our work presents an alternative approach to spatial structure encoding similar to Gilmer et al. (201)) which do not rely on node coordinates, but different in embedding and updat- ing scheme. Distinct from Danel et al. (2020) and Chen & Chen (2021), we extended attentional embedding to be edge-featured, while still node-centric for message passing efficiency.
Reinforcement Learning A variety of graph generative models have been used in prior work,
predominantly Variational Autoencoders (VAE) (Simonovsky & Komodakis] 2018 Samanta et al.
2020 Liu et al. 20188) Ma al. 2018 Din al. 2018) and Generative Adversarial Networks (GAN)
(De Cao & Kipi 2018). While some of these have a recurrent structure (Li et al. 2018 You et al.,
2018b), RL and other search algorithms that interact dynamically with the environment excel in
sequential generation due to their ability to resist overfitting on training data. Both policy learning (You et al. 2018a) and value function learning (Zhou et al. 2019) have been adopted for molecule generation: however, they generate molecules node-by-node and edge-by-edge. In comparison, an action space consisting of molecular fragments, i.e., a collection of chemically valid components and realizable synthesis paths, is favorable since different atom types and bonds are defined by the local molecular environment. Furthermore, the chemical space to explore can be largely reduced. mragment-by-fragmeen
sequential generation has been used in VAE (Jin et al., 2018) and search algorithms (Jin et al. 2020, Xie et al. [202I). but has not been utilized in a deep graph RL framework. In this work, we designed our environment with the Chemically Reasonable Mutations (CReM) (Polishchuk 2020) library to realize a valid fragment-based action space. In addition, we enhanced exploration by employing simple and efficient technique, adapting Random Network Distillation (RND) (Burda et al.] 2018) to GNNs and proposing surrogate innovation rewards for intermediate states during the generating process.
Antiviral Drug Discovery A Timely Challenge The severity of the COVID-19 pandemic highlighted the major role of computational workflows to characterize the viral machinery and iden- tify druggable targets for the rapid development of novel antivirals. Particularly, the synergistic use of DL methods and structural knowledge via molecular docking is at the cutting edge of molecular biology - consolidating such integrative protocols accelerate drug discovery of paramount importance (Yang et al. [22TT] Jeon & Kim 2020 Thomas et al.) 202T). Here we experimentally examined our architecture on the task of discovering novel inhibitors targeting the SARS-CoV-2 non-structural protein endoribonuclease (NSP15), which is critical for viral evasion of host defense systems (Pillon et al., 202I). Structural information about the putative protein- ligand complexes was integrated into this framework with AutoDock-GPP (Santos-Martins et al. 2021), which leverages the GPU resources from leadership-clatt computing facilities. including the Summit supercomputer, for thghuhhhough molecular docking (LeGrand et al., 2020). We show that results outper- formed state-of the- generation models in finding molecules with high affinity to the target and reasonable synthetic accessibility.
2

---

vector of E Published as a conference paper at ICLR 2022
2 PROPOSED METHOD
2.1 ENVIRONMENT SETTINGS
In the case of molecular generation. single-atom single-bond additions are often not realizable by known biochemical reactions. Rather than employing abstract architectures such as GANs to suggest synthetic accessibility, we use the chemical library CReM (Polishchuk) 2020) construct our environment such that all next possible molecules can be obtained by one step of interchanging chemical fragments with the current molecule. This explicit approach is considerably more reliable and interpretable compared to DL approaches. A detailed description of the CReM library can be found in Appendix B.1
The generating process is formulated A). At each time step t, we use CRcM to sample a set of valid molecules V++1 as the candidates for the next state S141 based deterministic, set of the action space can be defined as equal to S of the state space, and action as
Markov decision problem (details are given in Appendix
current state 8t. Under this setting, the transition dynamics are
is induced by the direct selection of 8t+1- With abuse of notation, we let r($++1) r(s,, a1).
2.2 SPATIAL GRAPH ATTENTION
We introduce a graph embedding mechanism called Spatial Graph Attention (sGAT) in an attempt to faithfully extract feature vectors h, Rd representing graph-structured objects such molecules. Two different types of information graphs constructed from a connected object are heterogeneous and thus handled differently in forward passes described in the following sections. See Figure] for overview.
ARRY.
Figure 1: Overview of Spatial Graph Attention defined by equations to (4). Highlighted nodes and edge are the examples undergoing forward propagation. The attention mechanism node cen- tric: nodes are embedded leveraging information from adjacent nodes and edges (different colors of dash lines denote different attentions); edges are embedded leveraging information from adjacent nodes. Spatial information is separately encoded according to sparsified inverse distance matrix (red crosses represent weights that are omitted) and embedded with such attention mechanism. The two hidden representations acquired respectively are aggregated at the end of each layer.
2.2.1 ATTENTION ON ATTRIBUTION GRAPHS
The attribution graph of a molecule with n atoms and e bonds given by the triple (A, E), where A E {0, j is the node adjacency matrix, N is the node attribution matrix of dimension d, and the edge attribution matrix of dimension e dg. Each entry agy of vector nu; of N is concatenation of the
if a bond
exists between atom and and 0 otherwise. Each properties atom i, including its atomic number, mass, etc., with the categorical properties being one-hot encoded. E is formed similar N, but with bond attributes. We denote as Cij if corresponds to the bond between atom and j.
We proceed to define multi-head forward propagation that handles these rich graph information: let In, E R 1lxd denote a given representation for n&. hes E Blxds. denote a representation for eij, then the m-th head attention at! from node node i + j) is given by
a softmar,
U k:a
(a(hin Wnm he, Weam || fong W Wn.m attm adtmm)))
(I)

---

Published as a conference paper at ICLR 2022
where softmaz, is the softmax score of node is column concatenation; or is some non-linear activation; Wn, E Rd xd=a, W Wemm E IR Xdw, are the m-th head weight matrices for nodes and edges respectively; allm [21x(2dux +dwa) ) is the mith head attention weight. The representations after feed-forward operation are consequently given follow:
(o((E
Nr. aggr S Smmm {o
Σ
oij Mnn bi:
W n.m
::ay==
New 1 Spm for ([ W n.m. hey W. a.m Mny W n.mm
W h.m)
where Wm.m (2dun +dw,, Xxdx;; Nm is the total number of attention heads and aggr denotes an aggregation method, most commonly mean, sum, or concul (Hamilton et al. [2017]. We note that we have not found significant difference across these methods and have used mean for all aggregations in experiments. In principle, a single-head operation nodes is essentially graph convolution with the adjacency matrix Á À + I where A attention-regulariiee according to (]] This approach sufficiently embeds edge attributes while still being node-centric convolution mechanism, for which efficient frameworks like Pytorch-Geometre (ey & Lenssen 2019) have been well established.
2.2.2 SPATIAL CONVOLUTION
In addition attributions and logical adjacency, one might also wish exploit the spatial structure of an graph object. In the case of molecular docking, spatial structure informs the molecular vol- ume and the spatial distribution of interaction sites shape and chemical complementarity the receptor binding site is essential for an effective association.
Let G = (diy-') s.snn be the inverse distance matrix where di the Euclidean distance between node i and j for vi + j. and di := O. G can then be seen as an adjacency matrix with weighted "edge"s indicating nodes' spatial relations, and the forward propagation is thus given by
= = (
H.W..)
where G optionally sparsified and attention-regularizee from G to be described below; D diagigign EE= }: H,, is the row concatenation of hgiggn Wn E Rdd., xd., is the weight matrix. In reality, G' induces (nn) of convolution operations on each node and can drastically increase training time when the number of nodes is high. Therefore, one might want to derive G by enforcing cut-off around each node's neighborhood (Pei et al. 2020), or preserving an O(n) number of largest entries G and dropping out the rest. In our case, although the average number of nodes is low enough for the gather and scatter operations (GS) of Pytorch-Geemmtrri to experience no noticeable difference in runtime as node degrees scale up (Fey & Lenssen] 2019), the latter approach of sparsification was still carried out because we have discovered that proper cutoffs improved the validation loss in our supervised learning experiments. If one perceives the relations between chemical properties and spatial information more abstract, G should be regularized by attention as described in (T) in which case the spatial convolution is principally fully-connected graph attention with the Euclidean distance one-dimensional edge attribution.
2.3 GRAPH ATTENTION POLICY NETWORK
In this section we introduce Graph Attention Policy Network (GAPN) that is tailored to environ- ments that possess a dynamic range of actions. Note that p(-(st, az) is degenerate distribution for deterministic transition dynamics and the future trajectory T ~ p($ St42, (sx) is strictly equal in distribution
π (a,, d4+1.... [s:)), hence simplified as the latter in the following sections.
To learn the policy more efficiently, we let 8, and v, share a few mutual embedding layers, and provided option to pre-train the first n, layers with supervised learning, Layers inherited from pre- training are not updated during the training of RL. See Figure 2 for an overview the architecture.
4

---

Published as a conference paper at ICLR 2022
Gumert stne
Gruph Enooder
Fonmerd
SBAT Ehooder
Ditrbbuko
swwad
Qaricilty scar
Key Encoder
Exolutoo cadd comm
Exsturtin
RND
Chooen centibr
Figure 2: An overview of the Distilled Graph Attention Policy Network during single step of the generating process.
2.3.1 ACTION SELECTION
At each time step 4, we sample the next state 8:41 from categorical distribution constructed by applying retrieval-system-innsire attention mechanism (Vaswani et al. 2017
}
S1+1 OHC
softmaz (L)ina((EQ()) |I Ex(3))
U1+1
(5)
gEgL11
where OHC(PI, (H. Pv. is one-hot categorical distribution with n, categories; yt. 9++11 the embeddings for 8, and f/+1 acquired by the shared encoder; Eq. Eg are two sGAT+MLP graph encoders with output feature dimension dæ; Liinal : bgbx2d + RR is the final feed-forward layer. Essentially, each candidate state predicted a probability based on its 'attention' to the query state. The next state is then sampled categorically according to these probabilities.
There could be number of ways determine stopping time T. For instance, an intuitive approach would be to append to Us+1 and terminate the process if is selected as 91+1. In our experiments, we simply pick 7' to be constant, i.c. we perform fixed number modifications for an input. This design encourages the process to not take meaningless long routes or get stuck in a cycle, and enables episodic docking evaluations parallelization (further described in Section 2.5). Note that constant trajectory length is feasible because the maximum limit of time steps can be set significantly lower for fragment-basee -based action space compared to node-by-node and edge-by-edge action spaces.
2.3.2 ACTOR-CRITIC ALGORITHM
For the purpose of obeying causal logic and reducing variance, the advantage on discounted reward- to-go are predominantly used instead of raw rewards in policy iterations. The Q-function and ad- vantage function are expressed as
QF (8t: u1) = Ex
Σ
xt a(sv
86:
(6)
1'=
(s ar) Q (8t, uz) E, 1Q (se, [s)]
where is the rate of time discount. The Advantage Actor- (A2C) algorithm approximates E,, (st, [s.] with value network V((*+) and Q" (st, 0+) with r(s,, a:) + (8{4)))). For more detailed description of actor-critic algorithm in RL, see Grondman al. (2012).
2.3.3 PROXIMAL POLICY OPTIMIZATION
We use Proximal Policy Optimization (PPO) Schulman et al. 2017), state-of-the-ae policy gra- dient technique, to train our network. PPO holds a leash on policy updates whose necessity is elaborated in trust region policy optimization (TRPO) (Schulman al. 2015). yet much simplified.
5

---

Published as a conference paper at ICLR 2022
It also enables multiple epochs of minibatch updates within iteration. The objective function is given as follow:
(+) max ED Σ min (1())) (s), clip (4)) clip,
(s,
(8)
where re( - (at mg st), clip.(x) min (max f1, c, e) and so During policy iterations, π nue updated each epoch and a cloned from -ee each iteration.
2.4 EXPLORATION WITH RANDOM NETWORK DISTILLATION
We seek employ simple and efficient exploration technique that be naturally incorporated into our architecture to enhance the curiosity of our policy. We perform Random Network Distil- lation (RND) (Burda al. 2018) on graphs pre-trained feature graphs fulfill this need. Two random functions fur f that map input graphs feature vectors in gd, are initialized with neural networks, and J is trained to match the output of f":
= arg min Ey Eg' (s)
where Prezr is the empirical distribution of all the previously selected next states, i.c. the states that have been explored. We record the running errors a buffer and construct the surrogate innovation reward as:
ri(s') - clip,, ((I S -m. /vv
(10)
where my and ih are the first and second central moment inferred from the running buffer, clip,, (z) min {max {-7, x) m).
2.5 PARALLELIZATION AND SYNCHRONIZED EVALUATION
Interacting with the environment and obtaining rewards through external software programs are the two major performance bottlenecks in ours as well as RL in general. An advantage of our environ- ment settings, as stated in Section 2.3.I is that a constant trajectory length is feasible, Moreover, the costs for environmental interactions are about the same for different input states. To take advantage of this, we parallelize environments on CPU subprocesses and execute batched operations on one GPU process, which enables synchronized and sparse docking evaluations that reduces the number of calls to the docking program. For future experiments where such conditions might be unrealistic, we also provided options for asynchronous Parallel-GPU and ParalleI-CPU samplers (described in Stooke & Abbeel (2019)) in addition to the ParallcI-GPU sampler used in our experiments.
3 EXPERIMENTS
3.1 SETUP
Objectives Wvelluated our model against five state-of-the-art models (detailed in Section with the objective discovering novel inhibitors targeting SARS-CoVa NSP15. Molecular docking scores computed by docking programs that the stree-dimensional structure of the protein to predict the most stable bound conformations of the molecules of interest, targeting a pre-defined functional site. For more details molecular docking and our GPU implementation of an automated docking tool used in the experiments, see Appendix B.22 In addition, we evaluated our model in the context of optimizing QED and penalized LogP values, two tasks commonly presented machine learning literature for molecular design. The results for this can be found in Appendix D
Dataset For the models/settings that do require dataset, used set of SMILES IDs taken from more than six million compounds from the MCULE molecular library a publicly available dataset of purchasable molecules (Kiss al. 2012). and their docking scores for the NSP15 target.

---

Published as a conference paper at ICLR 2022
3.2 RESULTS
3.2.
SINGLE-OBIECTIVE OPTIMIZATION
The raw docking score is a negative value that represents higher estimated binding affinity when the score is lower. We use the negative docking score as the main reward Im and assign the final state ST as the single objective. For DGAPN, we also assign innovation reward to each intermediate state, and the total raw reward for a trajectory T is given by
r(r) T' (ST) t E (s(ss)
(11)
where e is the relative important of innovation rewards, for which we chose 0.1 and incorporated them with a 100 episode delay and 1,000 episode cutoff. Detailed hyperparameter settings for DGAPN can be found in Appendix C We sampled 1,000 molecules from each method and showed the evaluation results in Table | We note that we have separate approach to evaluate our model that is able to achieve a -7.73 mean and -0..38 best docking score (see the Ablation Study para- graph below), but here we only evaluated the latest molecules found in training in order to maintain consistency with the manner in which GCPN and MolDQN are evaluated.
Table 1: Primary objective and other summary metries in single-objective optimization
Docking Score
Validity
1st
2nd
3rd
mean
ordinary
adjusted
Uniq.
Div.
QED
SA
REINVENT
-8.38
-7.33
-7.28
-4.66
95.1%
94.6%
95.1
0.88
0.64
7.51
JTVAE
-7.48
-7.32
-6.96
-4.55
100 %
100 %
34.2%
0.87
0.67
2.65
GCPN
-6.34
-6.28
-6.24
-3.18
100% %
78.6%
100%
0.91
0.47
5.48
MolDQN
-8.01
-7.92
-7.86
-5.38
100% %
100%
100%
0.88
0.37
5.04
MARS
-8.11
-7.99
-7.86
-5.11
100 %
99.0%
100%
0.89
0.34
3,38
DGAPN
-10.07
-9.83
-9.19
-6.77
100%
99.8%
100%
0.81
0.31
2.91
In the result table, ordinary validity is checked by examining atoms' valency and consistency of bonds in aromatic rings. In addition, we propose adjusted validity which further deems molecules that fail on conformer generation (Riniker & Landrum 2015) invalid top of the ordinary validity criteria. This is required for docking evaluation, and molecules that fail this check are assigned a docking score of O. We also provide additional summary metrics to help gain perspective of the generated molecules: Uniq. and Div. are the uniqueness and diversity (Polykovskiy et al. 2020 QED (Bickerton et al.) 2012) is an indicator of drug-likeness,, SA (Ertl Schuffenhauer, 2009 is the synthetic accessibility. QED is better when the score is higher and SA is better when lower: Definitions of QED and SA can found in Appendix outperformed Maate-of-the-at models in terms of top scores and average score, obtaining high statistical significance over the second best model (MoIDQN) with a p-value of 8.55 10 10-209 2009 under Welch's 's t-test (Welch 1947). As anticipated, the molecules generated by fragment- based algorithms (JTVAE, MARS and DGAPN) have significantly better SAs. Yet we note that additional summary metrics are not of particular interest in single-objective optimization, and obtaining good summary metrics does not always indicate useful results. For example, during model tuning, we found out that worse convergence often tend to result in better diversity score. There also seems to be trade-off between docking score and QED which we further examined in Section 3.2.3
D On this task, DGAPN significantly
Ablation study We performed some ablation studies to examine the efficacy of each component of our model. Firstly, we segregated spatial graph attention from the RL framework and examined its effect solely in a supervised learning setting with the NSP15 dataset. The loss curves are shown in Figure [3] in which spatial convolution exhibited a strong impact on molecular graph representa- tion learning. Secondly, we ran single-objective optimization with (DGAPN) and without (GAPN) innovation rewards, and thirdly, compared the results from DGAPN in evaluation against greedy algorithm with only the CReM environment. These results are shown in Table 2 Note that is not exactly fair to compare greedy algorithm to other approaches since it has access more informa- tion (docking reward for each intermediate candidate) when making decisions, yet our model still managed to outperform in evaluation mode (see Appendix Cr more information). From results
7

---

Published as a conference paper at ICLR 2022
Sparalcnin
Hay cualde 1797
Spprialin
Docking Score
1st
2nd
3rd
QED
SA
MSE
GAPN DGAPN
-9.19
-6.71
0.28
-10.07
-9.83
-9.19
-6.77
0.31
2.91
1.0
CReM Greedy
-9.65
-9.36
-9.36
-7.73
0.39
2.83
DGAPN eval
-10.38
-9.84
-9.81
-7.73
0.34
3.03
Epochs
Figure 3: Loss curves over 40 runs each with sample size 100,000
Docking scores and other metrics under different training and evaluation settings
Table 2:
generated by the greedy approach, we can see that the environment and the stochasticity design of action space alone are powerful for the efficacy and exploration of our policies. While the innovation bonus helped discover molecules with better docking scores, it also worsened SA. We further inves- tigated this docking score vs. SA trade-off in Section 3.2.3 To see samples of molecules generated by DGAPN in evaluation, visit our repository!.
3.2.2 CONSTRAINED OPTIMIZATION
The goal of constrained optimization is to find molecules that have large improvement over a given molecule from the dataset while maintaining certain level of similarity:
T'm (ST) = T' (ST) - - max(0, (0, SIM (50. }T}}
(12)
where À is a scaling coefficient, for which we chose 100: SIM (-.} } is the Tanimoto similarity between Morgan fingerprints. We used subset of 100 molecules from our dataset as the starting molecules, chose the two most recent and best performing benchmark models in single-objective optimization to compete against, and evaluated 100 molecules generated from theirs and ours. The results are shown in Table 3
Table 3: Objective improvements and molecule similarities under different constraining coefficients
MolDQN
MARS
DGAPN
Improvement
Similarity
Improvement
Similarity
Improvement
Similarity
0.62 + 0.86
0.22 + 0.06
0.10 1,50
0.15 + 0.06
1.69 + 1.35
0.26 + 0.18
0.2 0.58 1 0.91
0.26 1 0.07
0.02 1 1.30
0.16 1 0.07
1.45 1 1.05
0.32 1 0.21
0.4 0.25 1 0.95
0,46 1 0.09
-0.06 1 1.20
0.16 1 0.06
0.41 1 1.06
0.42 1 0.18
0.6 0.01 1 0.75
0.67 + 0.16
0.09 1.23
0.17 + 0.06
0.33 1 0.68
0.64 + 0.21
From the results, it seems that MARS is not capable of performing optimizations with similarity con- straint. Compared to MolDQN, DGAPN gave better improvements across all levels of 6, although MolDQN was able to produce molecules with more stable similarity scores.
3.2.3 MULTI-OBJECTIVE OPTIMIZATION
We investigate the balancing between main objective and realism by performing multi-objective optimization, and thus provide another approach to generate useful molecules in practice. We weight I' with two additional metrics
QED and SA, yielding the new main reward as
Tor (ST)
Tm (ST) + (1-w) p. (QED(sT) SA*
(13)
where (A"(ss) = (10 (()))))) /) is an adjustment SA such that ranges from 0 to 1 and is preferred to be larger; μ is scaling coefficient, for which chose 8. The results obtained by DGAPN under different settings w are shown in Figure With w = 0.6. DGAPN is able to generate molecules having better average QED (0.72) and SA (2.20) than that of the best model (TTVAE) in terms of these two metrics in Table [] while still maintaining a mean docking score (-5.69) better than all benchmark models in single-objective optimization.
hhttps://giii thtp:/gttttthhcoc com http:////ooooooooaaa/a///aa//
8

---

Published as a conference paper at ICLR 2022
10
01/0 10f/ 0 o0i0
Sangh 2.2
Ofc
OSD Score
of
044
you oodl
02
Kintt 2.4
ddlp
06g
-15
-12
Docking Scot
Docking Scare
Figure 4: Summary of the molecules obtained by DGAPN. Left and middle plots show QED and SA vs. docking scores for the latest 1,000 molecules generated under each weight w. Right plot shows the top 3 molecules (out of 10,000) with the highest multi-objective rewards under each w.
A trade-off between docking reward and QED/SA was identified. We acknowledge that optimizing docking alone does not guarantee finding practically useful molecules, but our goal is to generate promising chemicals with room for rational hit optimization. We also note that commonly used alternative main objectives such pLogP and QED are themselves unreliable undisceming as discussed in Appendix D Hence, for methodological study purposes, we believe that molecular docking provides a more useful and realistic test bed for algorithm development.
RELATED WORK
The REINVENT (Olivecrona al. 2017) architecture consists of two recurrent neural network (RNN) architectures, generating molecules as tokenized SMILE strings. The "Prior network" is trained with maximum likelihood estimation on set of canonical SMILE strings, while the "Agent network" is trained with policy gradient and rewarded using a combination of task scores and Prior network estimations. The Junction Tree Variational Autoencoder (JTVAE, Jin al. (2018) trains two encoder/decoder networks in building a fixed-dimension latent space representation of molecules, where one network captures junetion tree structure of molecules and the other re- sponsible for fine grain connectivity. Novel molecules with desired properties are then generated using Bayesian optimization on the latent space. Graph Convolutional Policy Network (GCPN, You et al. (a018a)) is policy gradient RL architecture for de novo molecular generation. The network defines domain-specine modifications on molecular graphs so that chemical validity is maintained at each episode. Additionally, the model optimizes for realism with adversarial training and ex- pert pre-training using trajectories generated from known molccules in the ZINC library. Molecule Deep Q-Networks (MoIDQN, /hou al. (2099) is Q-learning model using Morgan fingerprint as representations of molecules. To achieve molecular validity, chemical modifications are directly de- fined for each episode. To enhance exploration of chemical space, MolDQN learns H independent Q-functions, each of which is trained on separate sub-samples of the training data. Markov Molec- ular Sampling (MARS, Xie al. (202T)) generates molecules by employing an iterative method of editing fragments within molecular graph, producing high-quality candidates through Markov chain Monte Carlo sampling (MCMC). MARS then uses the MCMC samples in training GNN to represent and select candidate edits, further improving sampling efficiency.
5 CONCLUSIONS
In this work, we introduced a spatial graph attention mechanism and curiosity-driven policy net- work to discover novel molecules optimized for targeted objectives. We identified candidate antivi- ral compounds designed to inhibit the SARS-CoV-2 protein NSP15, leveraging extensive molecular docking simulations. Our framework advances the state-of-the-art algorithms in the optimization of molecules with antiviral potential as measured by molecular docking scores, while maintaining reasonable synthetic accessibility. We note that a valuable extension of our work would be to focus on lead-optimization - the refinement of molecules already known to bind the protein of interest through position-come constrained modification. Such knowledge-bel waweedge-based and iterative refinements may help to work around limitations of the accuracy of molecular docking predictions.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was funded via the DOE Office of Science through the National Virtual Biotechnol- ogy Laboratory (NVBL), consortium of DOE national laboratories focused on the response to COVID-19, with funding provided by the Coronavirus CARES Act. This research used resources of the Oak Ridge Leadership Computing Facility (OLCF) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-ACC55-MOR22725. This manuscript has been coauthored by U'T-Battelle, L.C under contract no. DE-ACO5-000R2272 with the U.S. Department of Energy. The United States Government re- tains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up. irrevocable, world-wide license to publish or reproduce the published form of this manuscript, allow others do so, for United States Government pur- poses. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan access-plan, last accessed September 16, 2020).
wow.e/oooo..ooooooooooooo///oo public-
REFERENCES
G Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint 1rXiv:1810.12880 2018.
Dexiong Chen, Laurent Jacob, and Julien Mairal. Convolutional kemel networks for graph- structured data. In International Conference on Machine Learning, pp. 1576-1586. PMLR, 2020.
Jun Chen and Haopeng Chen. Edge-featured graph attention network.
arXiv preprint
arXiv:2101.06671, 2021.
Tomasz Danel, Przemyslaw Spurek, Jacek Tabor, Marek Smieja, Łukasz Struski, Agnieszka Stowik, and Łukasz Maziarka. Spatial graph convolutional networks. In International Conference on Neural Information Processing, pp. 668-675. Springer. 2020.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs
arXiv preprint 1XXv:18055.1973 2018
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug- like molecules based on molecular complexity and fragment contributions. Journal of cheminfor- matics, 1(1):1-11, 2009.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: 09422 2019.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P Over- ington. ChEMBL: large-scale bioactivity database for drug discovery. Nucleic Acids Res., 40 (Database issue):D1 100-7, January 2012.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions Systems, Man, and Cybernetics, Part C (Applications and Reviews), 1216:1111 2012.
William L Hamilton, Rex Ying. and Jure Leskovec. Inductive representation learning on large graphs. arXiv preprint arXiv: 022 16, 2017.
Sheng You Huang, Sam Z Grinter, and Xiaoqin Zou. Scoring functions and their evaluation methods for pettein-ligand docking: Recent advances and future directions. Physical Chemistry Chemical Physics, 12(40): 12899-122999111 - 12908, 2010.
10

---

Published as a conference paper at ICLR 2022
Ruth Huey, Garrett M. Morris, Arthur J. Olson, and David S. Goodsell. A semiempirical free energy force field with charge-based desolvation. Journal of Computational Chemistry, 28(6): 1145-1152, 2007.
James P Hughes, Stephen Rees, S Barrett Kalindjian, and Karen L Philpott. Principles of early drug discovery. British journal of pharmacology, 162(6):1239 1249, 2011.
Woosung Jeon and Dongsup Kim. Autonomous molecule generation using reinforcement learning and docking to develop potential novel inhibitors. Scientific reports, 10(1):1-11, 2020.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junetion tree variational autoencoder for molecular graph generation. In International Conference on Machine Learning, pp. 2323-2332. PMLR, 2018.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In International Conference on Machine Learning, pp. 4849-4859. PMLR, 2020.
Youngchang Kim, Jacek Wower, Natalia Maltseva, Changsoo Chang, Robert Jedrzejczak, Mateusz Wilamowski, Soowon Kang, Vlad Nicolaescu, Glenn Randall, Karolina Michalska, and Andrzej Joachimiak. Tipiracil binds to uridine site and inhibits nsp15 endoribonuelease nendou from sars-cov-2. Communications Biology, 4(1):1-11, 2021.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: 1609.02977, 2016.
Robert Kiss, Mark Sandor, and Ferenc A Szalai. htt::/mcule.com: public web service for drug discovery. J. Cheminform,, 4(Suppl 1):P17, 2012.
Scott LeGrand, Aaron Scheinberg, Andreas F Tillack, Mathialakan Thavappiragasam, Josh V Ver- maas, Rupesh Agarwal, Jeff Larkin, Duncan Poole, Diogo Santos- Martins, Leonardo Solis- Vasquez, et al. Gpu-accelerated drug discovery with docking on the summit supercomputer: porting, optimization, and application to covid-19 research. In Proceedings of the 11th ACM international conference on bioinformatics, computational biology and health informatics, pp- 1-10, 2020.
Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. arXiv preprint arXiv:2009.00142. 2020.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. arXiv preprint 1003:..000300322., 2018.
Bowen Liu, Bharath Ramsundar, Prasad Kawthekar, Jade Shi, Joseph Gomes, Quang Luu Nguyen, Stephen Ho, Jack Sloane, Paul Wender, and Vijay Pande. Retrosynthetic reaction prediction using models. ACS central science, 3(10): 103-1113, 2017.
neural sequence-to-sequee
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander I. Gaunt. Constrained graph vari- ational autoencoders for molecule design. arXiv preprint arXiv: / $955099766 2018.
Tengfei Ma, Jie Chen, and Cao Xiao. Constrained generation of semantically valid graphs via regularizing variational autoencoders. arXiv preprint arXiv: 1809.0233, 2018.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):4602 4609, Jul. 2019. doi: 10. 00.1609/aaai.v3330 133101.33014461
Garret M Morris, Ruth Huey, William Lindstrom, Michel F Sanner. Richard K Belew, David Goodsell, and Arthur J Olson. Autodock4 and autodocktools4: Automated docking with selective receptor flexibility. Journal of Computational Chemistry, 30(16):2785-2799 2009a.
11

---

Published as a conference paper at ICLR 2022
Garrett M Morris, Ruth Huey, William Lindstrom, Michel F Sanner, Richard K Belew, David Goodsell, and Arthur Olson. AutoDock4 and AutolackTools:: Automated docking with selec- tive receptor flexibility. J. Comput. Chem., 30(16):2785-2799 December 2009b.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. J. Cheminform, 91)):48, September 2017.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. arXiv preprint arXiv:2002.0522: 2020.
Monica Pillon, Meredith Frazier, Lucas B Dillard, Jason G Williams, Seda Kocaman, Juno M Krahn, Lalith Perera, Cassandra K. Hayne, Jacob Gordon, Zachary D Stewart, Mack Sobbany, Leesa Deterding, Allen L Hsu, Venkata Dandey, Mario Borgnia, and Robin E Stanley. Cryo- structures of the sars-cov-2 endoribonuclease nsp15 reveal insight into nuclease specificity and dynamics. Nature Communicaatooss, 12(1):1-12, 2021.
Pavel Polishchuk. Crem: chemically reasonable mutations framework for structure generation. Jour- nal of Cheminformatics, 12:1-18, 2020.
Pavel G Polishchuk, Timur I Madzhidov, and Alexandre Varnek. Estimation of the size of drug- like chemical space based on gdb-17 data. Journal of computer-aided molecular design, 27(8): 675-679, 2013.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, et al. Molecular sets (moses): a benchmarking platform for molecular generation models.
Frontiers in pharmacology. 11:1931, 2020.
Sereina Riniker and Gregory Landrum. Better informed distance geometry: using what we know to improve conformation generation. Journal of chemical information and modeling, 55(12): 2274, 2015.
Bidisha Samanta, Abir De. Gourhari Jana, Vicenç Gómez, Pratim Kumar Chattaraj, Niloy Ganguly, and Manuel Koome-RRorigure Nevae: A deep generative model for molecular graphs. Journal of machine learning research. 2020 Apr: 21 (114): 1-33, 2020.
Diogo Santos-Martins, Leonardo Vassu-, Andreas Tillack, Michel Sanner, Andreas Koch, and Stefano Forli. Accelerating autodock4 with gpus and gradient-based local search. Journal of Chemical Theory and Computation, 17(2):1060-107) 2021.
Petra Schneider, W Patrick Walters, Alleyn T Plowright, Norman Sieroka, Jennifer Listgarten, Robert A Goodnow, Jasmin Fisher, Johanna M Jansen, José S Duca, Thomas Rush, et al. Re- thinking drug design in the artificial intelligence era. Nature Reviews Drug Discovery, 19(5): 353-364, 2020.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1007:0677770637777 2017.
Kristof T Schütt, Pieter-Jan Kindermans, Huziel Sauceda, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-fitte convolutional neural network for modeling quantum interactions. arXiv preprint arXiv: 1:Xx:v:706.0866/ 2017.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks, pp. 412- 422. Springer, 2018
Adam Stooke and Pieter Abbeel. rlpyt: A research code base for deep reinforcement learning in pytorch. arXiv preprint arXiv: 1909.01500, 2019.
12

---

Published as a conference paper at ICLR 2022
Morgan Thomas, Robert T Smith, Noel M O'Boyle, Chris de Graaf, and Andreas Bender. Com- parison structure-and ligand-based scoring functions for deep generative models: gper case study. Journal of Cheminformaties, 13(1):1-20, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint 17X6::770600366,, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks, arXiv preprint arXiv: 2017.
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Heterogeneous graph attention network. In The World Wide Web Conference, pp. 4022-20322 2019.
David Weininger. Smiles, chemical language and information system. 1. introduction to method- ology and encoding rules. Journal of chemical information and computer sciences, 28(1):31-355 1988.
Bernard Welch. The generalization of 'student's' problem when several different population var- lances involved. Biometrika, 34(1-2):28-33 33, 1947.
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. Mars: Markov molecular sampling for multi-objective drug discovery. arXiv preprint arXiv:2105 10432, 2021.
Ying Yang, Kun Yao, Matthew P Repasky, Karl Leswing, Robert Abel, Brian Shoichet, and Steven Jerome. Efficient exploration of chemical space with docking and deep-learning. Journal of Chemical Theory and Computation, 2021.
Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. arXiv preprint arXiv: arXiv:180..022: 1806.02473, 2018a.
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovee. Graphrmn: Generat- ing realistic graphs with deep auto-regressive models. In International Conference on Machine Learning, pp. 5708-5717 PMLR, 201
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint 1rXiv:1212.5701, 2012.
Zhenpeng Zhou, Steven Kearne,, Li Li, Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning, Scientific reports, 9(1):1-10, 2019.
APPENDIX
A DETAILED FORMULATION OF THE PROBLEM
Our goal is to establish a set of decision rules to generate graph-structured data that maximizes compound objectives under certain constraints. Similar to prior formulations, the generating process is defined as a time homogeneous Markov Decision Process (MDP). We give formal definition of this process Appendix [. Under this setting, the action policies and state transition dynamics at step / can be factorized according to the Markov property:
P(ar|so. ao 81, Si,a....... St) = P(a. (x) == π (a.]s;)
(14)
P(s (15441/50,00 4o, $1, di...., St, a.) = (Sst41/s,, at) = p(s141|st,
(15)
where {st. aehe are state-action sequences. A reward function r(s, a) used to assess an action taken given state The process terminates at optional stopping time T and sT is then proposed as the final product of the current generating cycle. We aim to estimate the optimal policy in terms of various objectives to be constructed later in the experiment section.
13

---

Published as a conference paper at ICLR 2022
A.1 MEASURE THEORY CONSTRUCTION OF MARKOV DECISION PROCESS
Let (S and (A be two measurable spaces called the state space and action space; functions II S x A IR and T A R are said be policy and transition probability respectively if
1. For each s E S, E + II(s, E) is a probability measure on (A, for each (s, E SSx4 A, F a, F) is probability measure on (S,
2. For each E E A. - II(s, E)) is measurable function from (S, + (R. B); for each FE S. (s, + Tss a, F) is a measurable function from (S x A, S @ .A) (R. B).
We say a sequence of random variable duples (St; Ar) defined on the two measurable spaces is Markov decision chain if
P(A (So: Ao, S,, , S()) = (1,,,, E) P(S E F (So, Ao, SI, AI........ St, A.)) T(,,, F),
(16)
(17)
A function r S IR is said be the reward function w.r.f. the Markov decision chain if rste, F2)) = En, (R($++1) S. = 8t, A, E F.e] where R: S + IR is its underlying reward function.
With an abuse of notation, define x (alss) [s, {a}), p(s (s, a) (ss, a, {s'}) and let r(s, a)
denote (a)).
B LEARNING ENVIRONMENT AND REWARD EVALUATION
B.1 ENVIRONMENT CREM
Chemically Reasonable Mutations (CReM) is an open-source fragment-based framework for chem- ical structure modification. The use of libraries of chemical fragments allows for a direct control of the chemical validity of molecular substructures and to consider the chemical context of coupled fragments (c.g., resonance effects).
Compared to atom-based approaches, CReM explores less of chemical space but guarantees chem- ical validity for each modification, because only fragments that are in the same chemical context are interchangeable. Compared to reaction-based frameworks, CReM enables a larger exploration of chemical space but may explore chemical modifications that are less synthetically feasible. Frag- ments are generated from the ChEMBL database (Gaulton et al. 2012) and for each fragment, the chemical context is encoded for several context radius sizes along with the fragment in a separate database. For each query molecule, mutations are enumerated by matching the context of its fragments with those that are found the CReM fragment-context database (Polishchuk 2020).
SMILES string and stored
In this work, we use grow function on single carbon to generate initial choices if a warm-star dataset is not provided, and mutate function to enumerate possible modifications with the default context radius size of to find replacements.
B.2 EVALUATION AUTODOCK-GE GPU
Docking programs use the three-dimensional structure of the protein (i.e., the receptor) to predict the most stable bound conformations of the small molecules (i.e., its putative ligands) of interest, often targeting a pre-defined functional site, such as the catalytic site. An optimization algorithm within scoring function is employed towards finding the ligand conformations that likely corre- spond binding free energy minima. The scoring function conformmation-depeedent and typi- cally comprises physics-based empirical or semi-empirical potentials that describe pair- wise atomic terms, such as dispersion, hydrogen bonding, electrostatics, and desolvation (Huang et al. 2010) Huey et al. (007). AutoDock is a computational simulated docking program that uses Lamarck- ian genetic algorithm to predict native- conformations of protein-ligand complexes and a semi- empirical scoring function to estimate the corresponding binding affinities. Lower values of docking scores indicate stronger predicted interactions. The opposite value of the lowest estimated binding affinity energy obtained for each molecule forms the reward.
14

---

Published as a conference paper at ICLR 2022
AutoDock-GPU (Santos-Martins Martins et al. 2021) is extension of AutoDock to leverage the highly- parallel architecture of GPUs. Within AutoDock-GPU, ADADELTA Zeileer 2012), a gradient- based method, is used for local refinement. The structural information of the receptor (here. the NSP15 protein) used by AutoDock-GPP is processed prior to running the framework. In this preparatory step, AutoDockTools (Morris al. 2009b) was used to define the search space for docking on NSP15 (PDB ID 6W01; Figure 5) and to generate the PDBQT file of the receptor, which contains atomic coordinates, partial charges, and AutoDock atom types. AutoGrid4 (Morris al. 2009a) was used to pre-calculate grid maps of interaction energy at the binding site for the different atom types defined in CReM.
endoU
MD and ND
Figure 5: The search space in NSP15 defined for molecular docking (green box). An NSP15 pro- tomer, which was used as the receptor the calculations, is shown (cartoon backbone representa- tion, in pink/magenta). The nucleotide density located at the catalytic site is depicted (blue surface). Other protomers forming the homo-hexamer are shown as grey surfaces. PDB IDs 6WLC and 6WXC were used in this illustration (Kim et al. 2021). D. Abbreviations: EndoU, Poly-U specific endonuclease domain; MD, Middle domain; ND, N-terminal domain.
In evaluation, after applying an initial filter within RDKit to check whether given SMILES is chemically valid (i.e., hybridization, ring membership etc.), a 3D conformer the molecule is generated using CCCChem..mbedMMleee. discarded. Next, the molecular geometry is energy minimized within RDKit using the generalized force filed MMFF94. The resulting conformer is used as input for molecular docking via AutoDock- GPU. We also excluded any molecules from the final result set that were both fully rigid and larger than the search box in the receptor. This only occurred in two molecules from the JTVAE evaluation.
SMILES that do not correspond to valid compounds are
C HYPERPARAMETER SETTINGS FOR SINGLE-OB/ECTIV OPTIMIZATION
Based on parameter sweep, we set number of GNN layers to be 3, MLP layers to be 3, with of the GNN layers and of the MLP layers shared between query and key. Number of layers in RND is set to all numbers of hidden neurons 256; learning rate for actor 2 for critic 1-4 for RND 2-3; update time steps (i.e. batch size) 300. Number of epochs per iteration and clipping parameter e for PPO are 30 and 0.1. Output dimensions and clipping parameter y for RND are and 5. In evaluation mode, of candidates per step from 15-20 to 128 and expand the maximum time steps per episode from 12 to 20 compared training. For more details regarding hyperparameter settings, codebase https: gtt::://gittiii thub. com/ www.o//w/wwooooooooooooooooomaae ponolunnrran/DGAAA
arg max policy instead of sampling policy, expand the number
D MORE RESULTS ON QED AND PENALIZED LOGP
Although QED and penalized LogP are the most popular objectives to benchmark ML algorithms for molecule generation, these benchmarks are questionable for both scientific study and
practical as Xie al. (2021) pointed out. Most methods can obtain QED scores close equal to the
15

---

Published as a conference paper at ICLR 2022
highest possible of 0.948, making the metric hard distinguish different methods. As for pLogP, we simply construct a large molecule with no ring, such as the molecule from SMILES 'CC- CCC...CCCCC' (139 carbons), it will give pLogP score of 50.31 which beats all state-of-the-an models in Table + Needless to say, we will achieve a even higher plg P by continuously adding car- bons, which was exactly how REINVENT performed in experiment. We note that were able to raise our results to around 18 soiely by doubling the maximum time steps per episode reported in interested pushing the performance on this somewhat meaningless metric by continuously increasing one hyperparameter.
Appendix C yet
Table 4: Top QED and poogP scores
QED
pLogP
1st
2nd
3rd
1st
2nd
3rd
REINVENT
0.945
0.944
0.942
49.04
48.43
48.43
JTVAE
0,925
0.911
0.910
5.30
4.93
4.49
GCPN
0,948
0.947
0,946
7.98
7.85
7,80
MolDQN
0.948
0.944
0,943
11.84
11.84
11.82
MARS
0.948
0.948
0.948
44.99
44.32
43.81
DGAPN
0.948
0.948
0.948
12.35
12.30
12.22
The results from REINVENT were produced in our own experiments, while others were directly pulled out from the original results reported in the literature.
E DEFINITIONS OF QED AND SA
E
QUANTITATIVE ESTIMATE OF DRUGLIKENESS
(QED) is defined
QED = exp 1 Σ Inds
:=1
where d; are eight widely used molecular properties. Specifically, they are molecular weight (MW), octenol-water partition coefficient (ALOGP), number of hydrogen bond donors (HBD), number of hydrogen bond acceptors (HBA), molecular polar surface area (PSA), number of rotatable bonds (ROTB), the number of aromatic rings (AROM), and number of structural alerts. For each d,,
b;
1
(xxx) 04 +
dg
+
+ exp
1 +
cxp
each o given by supplementary table in Bickerton et al. (2012).
E.2
SYNTHETIC ACCESSIBILITY
(SA) is defined as
SA fragmentScore complexityPenalty
The fragment score is calculated as a sum of contributions from fragments of 934,046 PubChem already-synthesizzed chemicals. The complexity penalty computed from combination of ring- SomplexityScore, stereoComplexityScore, macroCyclePenally, and the sizePenalty:
ringComplexityscore log(ARing@ridgeAttons + 1) log(nSprioAtoms
stereeComplexxttySccrr = log(nStereeCenteers 1)
macroCyclePenalty log(nNacroCycles
sizePenalty = nAtoms 1.005 nAtoms
16


Reviews:
[{"review_id": "c_XovlGD0Ap", "reviewer": "Reviewer_wuts", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nRelevant application of designing small molecules with desired properties, eg inhibitors of SARS-CoV-2  \n\nPaper is overall well written  \n\nProposed method compared with quite a large range of relevant baselines  \n\nSome informative ablation studies done  \n\nCode has been provided   \n\nOther comments/questions:\n\nAlthough the proposed method is apparently the first to utilize a fragment by fragment graph construction in a deep reinforcement learning framework, I think the idea to extend the atom by atom graph construction in previous RL works with a fragment based action space (explored in more recent works Jin 2018, Jin 2020, Xie 2021) is interesting but not particularly novel.  \n\nParts of figure 4, such as the molecular structures, are too small and unreadable   \n\nIt would be useful to see the distribution of the properties in Table 1 (dock score, QED, SA, FCD). Also, it would be interesting to see the QED, SA, and FCD that correspond to the top 3 dock score molecules  \n", "summary_of_the_paper": "This paper proposes a method to generate molecular graphs with optimized properties. Molecular graphs are constructed by the iterative addition of molecular fragments in a deep reinforcement learning framework. The method is benchmarked against a set of baselines in a task to generate molecules that maximizes the docking score to a protein (NSP15) from the SARS-CoV-2 virus, and shows good performance.", "summary_of_the_review": "Overall, I vote for acceptance. The well-structured paper proposes a useful molecule graph generation approach, accompanied by quite a comprehensive set of experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7wMftjDJx9q", "reviewer": "Reviewer_ZQrT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The authors extend graph attention networks to edge features and notice that spatial information is crucial in activity prediction. The introduced sGAT model is a novel contribution. It would be interesting to see an ablation study comparing this model to simpler graph neural networks, e.g. to the closely related EGAT architecture [1].\n- Chemically reasonable mutations (CReM) are sampled by the RL algorithm to ensure that the generated structures are realistic.\n- An innovation reward is added to encourage exploration. The reward is based on random network distillation.\n- The ablation study shows the usefulness of spatial convolution for the supervised learning task.\n- The results of the pretraining are shown in the ablation study even though they do not improve the generated compounds in terms of the main objective (an interesting negative result).\n\nCons:\n- The authors state that \"fragment-by-fragment sequential generation [...] has not been utilized in a deep graph RL framework\", but I am not sure it is true. For example, Ståhl et al. [2] use deep reinforcement learning on fragments to optimize molecules.\n- Recently, more and more publications focus on optimizing molecular docking scores using de novo drug design models, and I think this topic should be briefly summarized in the related work. For example, see [3].\n- In Figure 1, the meaning of lines, symbols, and colors is unclear. Even after reading Equations 3-6, the drawing is difficult to process.\n- The mathematical notation is difficult to follow in some parts of the methods section. For example, what is the purpose of the OHC operator? Is it there only to sample from the softmax distribution and map the decision onto the set of valid molecules?\n- The QED score is very low for DGAPN. Can it be caused by generating big compounds that exceed a typical molecular weight of drug-like compounds?  The compounds are generated by adding whole chemical fragments, which can cause fast growth of the molecules. If that is the case, then these structures will be difficult to optimize (cannot be easily used as hits, which was suggested by the authors). This defeats the purpose of the generative model. The generated structures should be shown at least in the supplementary material.\n- In this paper, only one antiviral target is considered. The results would be more convincing if there were more proteins in the experiments.\n- The rewards are based on docking scores, so it would be interesting to see the docking poses of generated compounds. Especially, large compounds like those shown in Figure 4, which probably barely fit in the binding pocket.\n\nOther comments:\n- Why is the second column for DGAPN bolded in Figure 1?\n- In Section 5, there is a citation that failed to compile: \"MARS, xie2021mars\"\n- In Appendix D.3, the notation $p_w$ is used two times - should it not be just $p$ for the generated molecules?\n\n[1] Chen, Jun, and Haopeng Chen. \"Edge-Featured Graph Attention Network.\" arXiv preprint arXiv:2101.07671 (2021).\n\n[2] Ståhl, Niclas, et al. \"Deep reinforcement learning for multiparameter optimization in de novo drug design.\" Journal of chemical information and modeling 59.7 (2019): 3166-3176.\n\n[3] Thomas, Morgan, et al. \"Comparison of structure-and ligand-based scoring functions for deep generative models: a GPCR case study.\" Journal of Cheminformatics 13.1 (2021): 1-20.", "summary_of_the_paper": "The authors propose a new fragment-based method for molecule generation. The model, called DGAPN, uses chemical fragments extracted from a public compound library with their chemical context (atom neighborhoods to which these fragments are attached). This way, all modifications made by the model should produce synthetically accessible compounds with high probability. Additionally, the authors introduce a new class of graph neural networks to predict chemical properties that employ an attention mechanism over atoms and chemical bonds. These models are used to guide the generative MDP trained with reinforcement learning. Inhibition of a SARS-CoV-2 antiviral target, NSP15, is used as an example task for the proposed model. The experimental section shows the results of single- and multi-objective compound generation, in which DGAPN obtains better docking scores of the generated molecules than other methods.", "summary_of_the_review": "For now, I recommend the rejection of this paper (weak reject). The novelty of the proposed method is significant, but there are some unclear parts in the description of the method, and the results do not sufficiently prove the superiority of the proposed approach (see the comments above).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "2u8iV9XJe59", "reviewer": "Reviewer_ce6h", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths：\n1.\tThe model in the paper is designed to resolve the up-to-date challenge of discovering novel inhibitors to target the SARS-CoV-2 non-structural protein endoribonuclease, and the experimental results show that molecules generated by the proposed model enjoy better synthetic accessibility according to SA value, which is a metric generally used to measure the synthetic difficulty of drugs. Meanwhile, the molecules show higher docking scores compared to other existing algorithms.\n2.\tThe reinforcement learning part of the model is based on a fragment-based chemical environment for chemical synthesis and utilizes many novel approaches like Proximal Policy Optimization and Random Network Distillation as an effective attempt. This may give inspirations of the model architecture to future researches in this field.\n\nWeaknesses:\n1.\tSpatial graph attention and spatial convolution indeed improve the model’s ability to extract structural information from input graphs, but these modules are not innovative enough. Actually, spatial graph attention is just a multi-head attention mechanism applied to graphs and spatial convolution is a Euclidean distance version of GNN’s convolution operation. Some new modules are expected to be employed by the model to make it more powerful.\n2.\tAs is seen from the experimental results, the docking scores and SA value are better than other existing algorithms indeed. But the model’s performance on other metrics is unsatisfactory and particularly, Diversity is even the worst among algorithms in Table 1. Although this has been explained in the paper as a general drawback of fragment-based algorithms, more optimizations may need to be realized.\n3.\tSome statements in the paper may make readers confused, which are listed below. Equation 3 and 4 don’t share a consistent representation of the element in the adjacency matrix. Table 1 lacks some necessary annotations like what is the definition of Diversity and what is the meaning of the values in bold. The right plot in Figure 4 also lacks some necessary explanations and this plot seems to be redundant now.\n", "summary_of_the_paper": "With the goal of generating molecules that bind to functional sites of SARS-Cov-2 protein, the paper proposed a reinforcement learning model with a fragment-based framework for chemical structure modification. In the network part, spatial graph attention and spatial convolution are utilized to extract more structural information from the input graph into the representation of nodes or graphs. Based on the actor-critic algorithm, the reinforcement learning part is designed to find the state with the best docking score computed by docking programs and some novel approaches like PPO and RND are used to train the model more effectively. In the experiments, the model shows great performance while reducing the complexity of chemical synthesis meanwhile.", "summary_of_the_review": "The paper utilized many novel approaches in graph representation and reinforcement learning to resolve the timely challenge about SARS-CoV-2, but the experimental results of the proposed model fail to show sufficiently powerful performance compared to existing algorithms except for better synthetic accessibility and docking scores. More innovative designs may need to be introduced to the present model to get a persuasive improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: After discussion, all reviewers are convinced about the novelty of the proposed method, and adjusted scores to recommend acceptance. They all appreciate the attempt to attack COVID-19 using machine learning.
Decision: Accept (Poster)