GRASP: Navigating Retrosynthetic Planning with Goal-driven Policy
Authors: Yemin Yu
Ying Wei
Kun Kuang
Zhengxing Huang
Huaxiu Yao
Fei Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Retrosynthesis prediction
retrosynthetic planning

Abstract: Retrosynthetic planning occupies a crucial position in synthetic chemistry and, accordingly, drug discovery, which aims to find synthetic pathways of a target molecule through a sequential decision-making process on a set of feasible reactions. While the majority of recent works focus on the prediction of feasible reactions at each step, there have been limited attempts toward improving the sequential decision-making policy. Existing strategies rely on either the expensive and high-variance value estimation by online rollout, or a settled value estimation neural network pre-trained with simulated pathways of limited diversity and no negative feedback. Besides, how to return multiple candidate pathways that are not only diverse but also desirable for chemists (e.g., affordable building block materials) remains an open challenge. To this end, we propose a Goal-dRiven Actor-critic retroSynthetic Planning (GRASP) framework, where we identify the policy that performs goal-driven retrosynthesis navigation toward a user-demand objective. Our experiments on the benchmark Pistachio dataset and a chemists-designed dataset demonstrate that the framework outperforms state-of-the-art approaches by up to 32.2% on search efficiency and 5.6% on quality. Remarkably, our user studies show that GRASP successfully plans pathways that accomplish the goal prescribed with a designated goal (building block materials).

while in this work we are pursuing more efficient and GRASP: Navigating Retrosynthetic Planning with Goal-driven Policy
Yemin Yu!!2, Ying Wei Kun Kuang', Zhengxing Huang", Huaxiu Yao', Fei Wu2.403
City University of Hong Kong. 2Zhejiang University
Stanford University, "Shanghai Artificial Intelligence Laboratory
'[yeminyu2-c, yygnmnnyuunnnnnngeeeeeeeee eeiittitttuure edu. hk
Ckunkuang, zhengxinghuang, wuf rufeijzzju. edu edu. (huaxiu)@cs. wttp:l/es..ss.aooooooooaooo edu
Abstract
Retrosynthetic planning occupies a crucial position synthetic chemistry and, accordingly, drug discovery, which aims find synthetic pathways of target molecule through a sequential decision-making process on a set of feasible re- actions. While the majority of recent works focus on the prediction of feasible reactions at each step, there have been limited attempts toward improving the sequential
policy. Existing strategies rely on either the expensive and high-variance value estimation by online rollout, or a settled value estimation neural network pre-trained with simulated pathways of limited diversity and no negative feedback. Besides, how to return multiple candidate pathways that are not only diverse but also desirable for chemists (e.g.. affordable building block materi- als) remains an open challenge. To this end, we propose a Goal-dRiven Actor-critic retroSynthetic Planning (GRASP) framework, where we identify the policy that per- forms goal-driven retrosynthesis navigation toward user-demand objective. Our experiments on the benchmark Pistachio dataset and chemists-designet dataset demonstrate that the framework outperforms existing state-of-the-art approaches by up to 32.2% on search efficiency and 5.6% on quality. Remarkably, our user studies show that GRASP successfully plans pathways that accomplish the goal prescribed with goal (building block materials).
Introduction
Retrosynthetic planning has significantly advanced chemical synthesis, bringing in increasingly sophisticated medicines that cure diseases and materials that improve life. A retrosynthetic planner takes the structure of target molccule as input and recursively selects feasible reactions to unsolved intermediate molecules until eventually reaching building block molecules. Since unsolved inter- mediate molecule usually requires multiple steps of reactions to synthesize and at each step has up to hundreds of feasible reaction candidates, retrosynthetic planning with enormous search space is very challenging even for experienced chemists. Consequently. computer-aider synthesis plan- ning (CASP) enters the scene assist chemists in accelerating the process of designing retrosynthetic pathways.
Computer-aide aided retrosynthesis planning consists of: 1) single-step retrosynthesis prediction which predicts list of feasible reaction candidates that connect a target molecule to its respective precursors, and 2) multi-step planning policy that searches for the optimal synthetic pathway by recursively applying the single-step prediction model. Recent years have witnessed plethora advancements in single-step prediction models (25
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

effective planning policy that limits the effective search space to include the most likely successful pathways.
Prior multi-step planning centered around tree or graph search methods 11 21 where the search is guided by only the total reaction cost (quality) from the target molecule to the current node. To improve the search policy, recent attempts [24 77) include the estimated value from the current node to building block molecules, i.e., being building block aware. Unfortunately, the value estimation in 24 7 is by online roll-out, unfavorably being of high variance and low search efficiency. Though addressed this issue by pre-training a value network on simulated pathways, these pathways constructed from an existing single-step reaction dataset offer limited diversity and no negative experiences to learn from.
Over and above, scoring and ranking the quality of many feasible pathways towards a target molecule has been notoriously diffieult. The considerations that dictate a high-quality pathway, including high reaction yields, simple reaction conditions, and low building block molecule costs, are oftentimes conflicting and require trade-off; moreover, predicting reaction yields (22) and conditions is very challenging due to ill-defined and noisy annotations, Chen et al. [2] proposed to evaluate the quality with the negative log-likelihood of all reactions predicted by the single-step model, while it is predicated on the assumption that frequent reactions are with high yields or easy conditions and biased by the seen reactions that train the single-ster step model. Keeping in mind that the objective of retrosynthetic planning is to assist chemists, and in practice the challenge of quality evaluation can be overcome by 1) returning as diverse feasible pathways as possible for chemists to weigh their preferences, and 2) returning the pathways that meet the qualifying conditions prescribed by chemists, e.g., a set of very cheap building block materials or easy-to- --snthhessee intermediate molecules.
Therefore, we are motivated to propose a Goal-dRiven Actor-critic retroSynthetic Planning (GRASP) framework. Specifically, we formulate retrosynthesis planning as reinforcement learning (RL) problem, where we first learn a policy network that takes continuous actions encoding the structure- level molecular information to allow navigation in the huge discrete action space of single-step reaction candidates. Moreover, GRASP learns a goal-driven Q-value estimation network to update the policy, by sampling both successful (positive) and failed (negative) experiences and relabeling the goals of sampled experiences. Finally, the learned Q-value estimation and policy networks join to guide the Monte-Carlo Tree Search, after which GRASP returns diverse pathways as a result of a good
tradeoff. In summary, our contributions are threefold.
We propose novel actor-critic retrosynthetic planning framework GRASP, which learns from extensive positive and negative experiences to navigate through huge single-step reaction spaces. We are the first empower goal-driven planning, which mitigates the challenge in quality evaluation of pathways by directly fulfilling the requirements prescribed by chemists.
We have evaluated the performance of GRASP on both an academic and an industrial benchmark dataset. The results and user studies demonstrate that GRASP outperforms all baselines in general retrosynthetic planning metries by a significant margin and the first to achieve high-quality goal-driven retrosynthetic planning.
2
Related Work
Single-step Retrosynthesis Prediction Single-step prediction models can be categorized into two main classes, i.e., template-based and template-free. Template-based methods rely on templates that encode chemical reaction cores to convert a product molecule into reactants. The key is
to rank templates and select an appropriate template to apply, for which recent attempts [3] 23
solve the problem of template selection through classification neural network. Despite their canplate-based approaches are disadvantaged by 1) the daunting challenge
superior interpretability, of atom-mapping for template extraction, and 2) poor generalization to unknown reaction types
or structures beyond templates. On the other hand, template-fres free methods [14 13 20 inspired
by the recent progress of seq2seq (27) and Transformer 28), regard single-step retrosynthesis prediction as translation task and translate a product molecule represented in SMILES strings [29] to reactant SMILES strings. To join the benefits of template-based and template-free methods, recent seek semi-template-based methods where the reaction center dictating reaction firstly predicted via graph neural networks and the resulting intermediate synthons are secondly translated into reactants via seq2seq graph translation models. Recently, Kim et al. [10] proposed to
works 25 31 19

---

fine-tune single-step prediction model with the feedback from multi-step retrosynthetic planning policy, leading to search- search-guided guided single-step model. We have validated in Section 4.3 that our planning policy is also compatible with the framework and improves single-step prediction and thereby final pathways.
Neural-guided search
building blocks awareness
Negative experiences
Exploration-cxploiaee tradcoff
Goal-driven
HgSearch
21
x
V
X XXX
DFPN-E
x
X
X
MCTS
24
X
Retro
Ours
Table 1: Comparison of different planning frameworks in five dimensions. Neural-guided Search learns from past multi-step planning experiences planning policy characterized by neural network; Building blocks Awareness: The value a planning policy is biased towards reactions leading to building block molecules; Negative Experiences mean planning pathways with failure; Exploration- exploitation tradeoff: A planning policy balances exploration and exploitation, resulting more diverse pathways. Goal-driven: A planning policy is capable of performing planning towards specific goal.
Multi-step Retrosynthetic planning We summarize the comparison of existing multi-step retrosyn- thetic planning policies in Table Specifically, previous planning methods HgSearch [21 and the proof number search [I]] are traditional heuristic search algorithms,, where the chemical feasibility and the value of the negative (failure) pathways are not considered. Inspired by AlphaGo (26] (Seller et al. (24] adopted the Monte-Carke tree search to generate a search tree on the fly and explore and generate multiple synthesis pathways. Unfortunately, each node with combination of all precursors in reaction leads enormous search space, and the value estimation by vanilla online roll-out is of high variance and high computation cost. Inspired by [[], Chen et al. [2) designed a neural- A Alliike algorithm that learns an additional value network with automatically constructed and only successful pathways to bias the search prior. Recently, Han et al. (9] and Xie et al. (00 used mawoleculla/innraaaatteee
based value network to capture improve the A *llkiee retrosynthetic planning algorithm. However, one disadvantage of A -like 2 retrosynthetic planning algorithm that it fails to balance exploration and exploitation, resulting in less diverse pathways. Moreover, none of the previous approaches is capable of biasing the retrosynthetic planning toward favorable goal prescribed by chemists.
level information further
3 Methods
First, Sec. 3.1 introduces Markov decision process (MDP) setting for goal-driven retrosynthetic planning. Secondly, Sec. 3.2 elaborates on the framework of the GRASP goal-driven actor-critic agent and the training procedure with TD3 [6] algorithm. Lastly. Sec. 3.3 introduces GRASP planning for given target molecule under goal-driven variant of MCTS.
3.1 Goal-driven MDP for retrosynthetic planning
We denote a finite-horizon MDP by M= (M, A. T, G, (s,,,g))) H for our goal-driven retrosyn- thetic planning task. We use S denote the state (molecule) space, A to denote the action (reaction) space which consists of reaction candidates a generated by the single-step prediction model, and (Ss= az) denote the state transition from s; 8t+1 through performing reaction with a deterministic state transition probability. We denote the goal space as G. which has the same size as the state space S since our goal is to navigate toward particular states. Considering the ultimate goal for retrosynthesis is to discover retrosynthetic pathways reaching building block molecules, we denote the goal for the entire set building block molecules as Gg. where each goal gi E GB indicates a specific building block molecule i. To simultaneously adapt to both general (non-goal ggoal-driven) and goal-driven retrosynthetic planning, we define y = GN as all zero embedding and concatenate the goal embedding with an additional binary feature embedding, where we use I[g Gs) O for the general planning and I(g = gi) = for the goal-driven planning towards goal g. For the reward design of rss, a, 9), assign the goal-driven wath-finding reward as r(s, a, = 1 when the state
3

---

reaches the desired goal g after taking action a and r(s, a, g) = 0 otherwise. Finally, y is the discount factor, and H the maximum horizon (length) for the pathway.
3.2 GRASP framework and training procedure
GRASP has two parameterized components as shown in Fig. I actor network denote by To (a s; g) and critic network by Qo(s, ..).. In the setting of retrosynthelic planning, we regard the upstream single-step retrosynthesis predictor as the environment and retrosynthetic planner as the agent. At each time step t. the agent outputs proto-action de with the same size as action embedding, according to its goal goal-driven policy network To (āe (tt g) from observing the current state & and goal g. Since we
unaware of the possible goal state for given initial state without prior knowledge, ( (t: 9 = Gg) as behavioral policy. Specifically, we add a small amount of random noises N to the action for exploration during sampling:
we use 9 = 98 in Ta
ay' = de +;; 6 (r,,).
After acquiring the proto-action ai', the agent has to identify an actual reaction dy from available reaction candidates A(st) given the state 8t and action embedding å. Inspired by the k-nearest neighbor (k-NN) trick for large discrete action space similar to the Wolpertinger training [5], we use the true action embeddings from the available actions A(st) for the k-NN calculation during the action selection procedure. Furthermore, we may encounter reactions that induce more than one non-building block molecule as reactants, namely convergent synthesis. Convergent synthesis reaction, although infrequent in retrosynthesis. introduces variation in the cardinality of state representation s that conflicts with MDP settings. In previous work in MCTS for retrosynthesis, Segler et al. [44 accumulates all non-building block reactants set of molecules in state rep- resentation, but only performs action selection on a single molecule. The combinatorial nature of state representation introduces bias in reward propagation and sparsity in variance estimation. To overcome this complexity during the sampling and training phase, we use the average distance among all reactants for k-NN computation each time we encounter reaction with convergent synthesis. As a result, we obtain an actual reaction ag by referring to the k-NN computation of the proto-action ár over the available actions A(sz). If a convergent synthesis reaction is identified as the true action at by the environment, non non-building block reactants are split into separate next states as independent trajectories to perform parallel sampling. Eventually, the next state St+1 defined as the non- non-building block molecule among the reactants of a, The sampling of a trajectory terminates when the state reaches the goal g or the length of the trajectory reaches the maximum horizon H.
Goal-driven relabeling: To capture the goal-driven planning insights from a retrosynthesis pathway and accelerate learning in the sparse reward setting, are inspired by 11 to relabel transition tuples in trajectories. The core idea of applying goal-driven relabeling in retrosynthesis is to exploit the data generated from the general retrosynthesis policy [a(a]s, y = G8) to train featured retrosynthesis planning data, and incorporate the agent with knowledge of navigating toward a specific goal state g = Dt- In practice, we copy the state transition tuple M, = (sy, a r-(- G8), 8(41) and randomly relabel the tuple M, = (st, ag, (-(9f)) 8,41) with relabeling probability pr using future relabeling strategy. Specifically, for the ith tuple M, in trajectory T with length T, we perform
goal-driven relabeling by iterating over all future transitions as:
8itk, Pr GG..
gi
1- Pr
for k E (0, -]. between general and goal-driven planning, we will further examine the effect of different p, on the planning performance in the experiment.
i). Since the relabeling probability pr is an important hyperparameter to balance
The RL agent is trained with TD3 (6] algorithm. For tuple in training batch, the target critic network is first updated using the one-step TD equation as:
(1) where, Q' and n' denote the target critic and actor networks with fixed parameters copied from original critic and actor networks Qo and TH respectively, and ri, 8;-1, I++1 represents the reward, state, and goal at the step l. With the TD target yı, we can calculate the batch mean-square-error loss
dd = r 7Q' (s++1. (s.4:///ss11 (++1))9441))
on the original critic network Qe(s, (8, a, 9) as:
1
(())- = N
Σ (w!d Qu(si, (),
4

---

camping mokeest lnttel state
Actor Network
geel
from step
bile nelock advaler messsam beroze!
%111
No, start from sup with ful
Singlesstp candidates af(s)
2 Gencuat proto-action with soor s/8l11.))
Envircoment
Pertoes s-NNN cxicaletre ataloskay reaciinn
Linnrric the nest state and rensed by enalaaiig the raclecale andeed or bortovn
(),0,
Roplay
Truining butch
Geel deli-divve relaneling
Criiie Network Oots, 0,
7. DuDugg look pepegation. parple: muining hutch tein wate eetwork with
5. Stere the toautron apple Oua Quait//y Syr (f seplay
Ocuerate scabeled zapiicee papies Puad aina rehabeling straiigg
anki
(Fq
Figure 1: GRASP training flowchart (left) and the goal-driven actor-critic framework (right), Grey boxes indicate the agent-level components that will used further during GRASP planning, and green boxes indicate the components used in GRASP training only.
Since the goal of the actor network is designed to maximize the overall return (success rate), and the goal of critic network is to approximate the overall return, the actor To can be trained by maximizing the Q value by minimizing:
I(d)
N E (o T ())
Self-imitation learning: To learn from highly imbalanced pathways in the overall search space (> 85% failures), we adopt self-imitation learning [17] (SIL) accelerate the convergence in sparse reward and enhance the training efficiency. Intuitively. SIL assists the RL agent to emphasize high- quality plamning experiences. Instead using the Bellman equation for calculating the target Q-value, SIL directly uses the Monte-Catt return of each 'good' episode as the Q-value target. It is crucial for the agent to exploit success trajectories in retrosynthetic planning tasks, especially during the early stage of training when large proportion of samples in the replay buffer originated from failed trajectories. We denote the simplified SII. loss for ith tuple success trajectory with length h as:
L(O)
1 N
Σ
(vt Qo((i, g))g)),
where y.t = iink > A-r We also include the full training algorithm in Alg.
3.3 GRASP retrosynthetic planning
In this section, demonstrate the GRASP planning procedure for target molecule and a specific goal with GRASP RI. agent ग. (als,, and Qo 8,9).
Since each newly expanded molecule node is the same as the initial state in GRASP it is natural to combine our RL agent into Monte-Carlo tree search (MCTS) with goal-driven P-UCT function (26
= argmax Q(st, a,9) + cP(alsr.g) g) 1+ N+N(s,,0)
/ (N(8L-1,011)
(5)
aεA((e,) N()
In previous MCTS for retrosynthesis, Segler et al. used an online roll-out stage for Monte- Carlo estimation of success rate for each leaf node, which both suffer from high-variance and heavy computation. Therefore, one of the key differences between GRASP and Segler et al. during the planning stage is we completely skip the online roll-out stage and directly refer to the RL agent for value estimation of the leaf nodes instead of Monte-Carke estimation from the online roll-out. To align with our MDP settings in RL, adopt goal-driven MCTS planning with individual molecules
5

---

Algorithm GRASP
Initialize Critic network Qo Q2 and actor network T Tyo replay buffer B, initial state space S, action space A. goal space G, reward function : :S x A G + R. for ep M/ do Sample imitial state So E S.
for f = 0 to do
Sample proto-action de using behavioral policy and exploration noise for general planning [ St: GB)
Perform k-NN computation and execute action de Observe reward = r(s,, az GB) and next state S1+1
end for
for t= to H do
Store original transition (st, Ge, rt. 9))1,9) in B
Generate transition copy and relabel g' Seti and r = r(st, a', with probability p- with
future goal-driven relabeling strategy.
Store transition (82, az, To 1, g¹) in end for
for /. = 0 to N do
Perform actor-critic batched TD training on 0, ø with Eq. and Eq.3
Perform SIL training 0 with Eq.4
end for end for
as tree node representation. Specifically, our framework consists of three phases as shown in Fig. and for simplicity, we ignore all building block molecules in the figure since no selection action will be performed on:
Selection:
Starting from the root node, the p-UCT function in Eq. 5 is used to iteratively select
action. At any step t, available actions in candidate set a, A(tt) and respective single-step confidence score pe(or(s)) are provided by the single-step retrosynthesis predictor, and we define:
exp D(a,a.
pla, (t.lst g) pe(aolse)
(6)
Eej A(ss) expl D(ā.og))
Σ
where D(-) is the same distance metric used in the k-NN calculation, ä is produced by policy network no (t)st. g), and ((s)-1.4(-1) denotes the visit count of the state-action pair of previous states. a convergent synthesis action with multiple non-building block reactants selected, we perform parallel selection and select all non- non-building block reactants as the next state. We iteratively perform selection on states until reaching a leaf node. Eventually, a set of leaf nodes is identified for expansion.
Expansion: Each leaf node 8, from the selected set is expanded by referring to the single-step model. Each available action from A(s;) is directly appended to the node s;. For convergent synthesis action, we generate the same number of leaf nodes depending on the quantity of unsolved molecules. For each newly generated leaf node, evaluate their Q* value with the following rule: if there is no available action for 8,, we directly apply Q' = 0. Before applying Qo a, g) network to assign Q" value for newly added leaf nodes, assign Q' and label it 'solved' if the s reaches g or Q" = if the state reaches the maximum horizon. If the state is undetermined, we assign Q" Qo a. a.g) by applying Qo value network.
Update: During the update phase, the Q" values and visit counts (s, a) are traversed backward
following the selection path from leaf nodes back to the root node. We simple moving average for updating Q value with a discount factor 7:
1 Q'(s;, a) Q(sc, NG
hQ Q(st, a))).

---

(1) Selection
Expansion
(3) Update
Iterativelly apply selection policy UCT) until reaching leaf (labeled red).
Expand selected leaf
Update upward along selected edges until reaching root node
generate new leaf nodes (green) and predict @* value with (Vv,o,
Q
e-
Q
e+
Figure 2: Overview of GRASP planning procedure. Specifically, the selected pathway (green) includes specific reactions: B, C D
4 Experiments
4.1 Experiment setup
Baseline Algorithms: We compare approach against varieties of baselines including:
1. MCTS[24]: The original vanilla version of MCTS for retrosynthesis planning which exploits online roll-out obtain Monte-Carlo estimation of future value without data generalization. 2. DFPN-E(1I]: Depth-first proof number search (PNS) performed on AND-OR tree with an additive reaction likeli- hood cost. 3. Retro* and Retro*-0 (2): Different from DPFN-E, Retro* utilizes the AND-OR tree as single-playe game and utilizes global value estimation. Additionally, Retro pre-trains a value network on simulated retrosynthesis pathway dataset. Retro*-0 denotes its version that performs the search without the value network. Retro* is reported as the retrosynthetic planning. 4. Hyper-Graph Search (Hggearch)[2[]: HgSearch is a beam-search-like algorithm performed on hyper-grapp structure. The heuristics are the product of the single-step confidence score and molecular complexity score (SCScore) [4.
the-art search algorithm for
Evaluation metrics: We use four different metrics to comprehensively evaluate the performance of different search algorithms: 1. Pathway length: We use the total number of reactions in the retrosynthesis pathway for length evaluation. 2. Pathway cost: The cost function is defined as the summation of the negative log- likelihood (confidence score) of the reactions in the pathway provided by the single-step model, i.c., - -Easr lals-(als) (2). The cost is also regarded as criterion for chemical feasibility. Planning efficiency: Since the primary objective of Al-aided retrosynthesis is to help chemists find successful pathways faster, efficiency has been crucial evaluation criterion for a multi-step retrosynthesis planning algorithm. Therefore, we follow (2 to take the number of single-step inference calls as qualified surrogate of time, as single-step step inference (~> 2s per iter) takes up almost > 99% of the time (only 0.006s per iter on planning). Success rate: With fixed number of single-step inference calls, the success rate is defined as the percentage of solved molecules in the entire set.
Single-step retrosynthesis predictor: We adopt the template-free single-step retrosynthesis predictor based on molecular transformer (2 [3] from Schwaller et al. [21) as our single-step retrosyn- thesis predictor. Specifically, Schwaller et al. separately trained pair of backward single-step generation models and forward single-step prediction models, and cooperatively utilized them to generate high-quality single-step retrosynthetic candidates with a confidence score Pe (a) ranging from to Both statistics [14 [3] and our user study in real-world scenarios demonstrate that MT- based single-step framework achieves higher accuracy and less chemo-seeectivee when compared with template-bart based approaches. Eventually, we choose top-k=100 reactions ranked according to the confidence score predicted by the single-step model as the available single-step candidate set for a given molecule since 100 sufficient represent feasible single-step reaction space for molecule.
7

---

PISTACHIO
GRASP
RETRO
RETRO*-0
HcSEARCH
DFPN-E
MCTS
AVG. LENGTH
4.12
4.27
4.25
4.38
4.22
4.74
AVG. COST
7.47
7.53
8.44
7.06
12.88
13.72
AvG. TIME
42.6
62.0
82.4
94.5
84.3
116.5
SUCCESS RATE
0.95
0.92
0.92
0.87
0.85
0.81
WUXITEST
GRASP
RETRO*
RETRO*-0
HoSEARCH
DFPN-E
MCTS
AVG. LENGTH
6.93
7.50
7.38
7.65
7.29
8.15
AVG. COST
21.19
22.55
26.55
21.41
30.89
38.62
AvG. TIME
79.3
112.5
157.2
194.7
183.9
224.5
SUCCESS RATE
0.86
0.80
0.78
0.82
0.64
0.52
Table 2: General planning performance summary on Pistachio and WuxiTest, Average statistics is calculated among all successful pathways with N Near 400 for both datasets.
4.2 Creating benchmark datasets
Single-step reactions and building block molecules dataset: We use the Pistachio reaction benchmark dataset for training single-step models, and the implementation details are listed in Appendix A.I. After further pruning and discarding reactions with multiple products, the entire dataset consists of 2.7M reactions. The dataset is further split randomly into train/val/test sets following 90%/5%/5%% proportions. We use the complete 231M commercially available molecules presented in eMolecules for the building block molecule set.
dataset (Ver. 18.11.19) [16]
Pathway dataset: Since only Retro requires an additional simulated retrosynthesis pathway dataset for pre-training its value network for planning. we follow the setting in 2 and construct the Pistachio pathway dataset similarly. Specifically, we obtained 61554 pathways with an average length 3.66. We split the dataset into 40000 training pathways, 21354 validation pathways, and 200 test pathways for Retro* value network training. Note that constructing an artificial pathway dataset by simply concatenating single-step reactions is only reference rather than an optimal/expert pathway for a given molecule in a given search space. Moreover, an expert pathway dataset is unavailable for unreported molecules and expensive to obtain in real-world scenarios. Therefore for fair comparison, the target molecules the pathway dataset are simultaneously used as initial states for GRASP training.
We also include a real-world expert dataset 'WuxiTest' designed by WuxiAppTec chemists, and each target molecule is provided with one reference pathway. WuxiTest consists of 500 molecules that are specifically designed to consist only of molecules that have never appeared in any journals and patents. Molecules were split into ten categories in terms of retrosynthesis strategies, and each category shares similar molecular substructures. We partition the pathway dataset category-wise as 80% 90%6//%%%0%%% into train/valid/test sets as partitions and follow the same training settings as the Pistachio.
Expert dataset:
4.3 Results
The performance of all methods is listed in Table. For both the Pistachio and WuxiTest datasets, our approach achieves the highest success rate compared with the baselines. We observe that HgSearch achieves the best performance on the average cost metric in Pistachio, mainly from the near-exhaustive search performed on less challenging molecules. Our approach outperforms other baselines in average expansion by large margin, demonstrating the performance gain in planning efficiency brought by RL training. In the WuxiTest dataset, our approach outperforms all other basclines in all four metrics. Since the WuxiTest dataset is designed emphasize retrosynthesis strategies with more challenging but strategically similar molecules, the result proves that RL training can generalize planning knowledge from molecule with similar substructures. We demonstrate the influence of time limit on the success rate for different approaches for the WuxiTest dataset in Fig. (3a for Nman 400. We also demonstrate that the success rate tends to saturate when for N > 400 by extending to Nmez 1000 in Appendix C for all approaches.
http:///wwnloods..coeeellee....

---

WUXITEST
RETRO*
HOSEARCH
DFPN-E
MCTS
GRASP GENERAL
GRASP
EXPERT
SOURCE AVG. LENGTH
7.50
7.65
7.29
8.15
7.05
N/A
GRASP AVG. LENGTH
7.35
7.07
7.55
7.05
7.20
SOURCE AVG. RATING (0-10)
7.6
6.5
8.3
N/A
GRASP AVG. RATING (0-10)
7.7
8.1
7.6
7.5
8.3
9.2
Table 3: Goal-driven planning performance summary. The experiment is conducted through a double- blind test with two different chemists to evaluate the quality of the pathway in terms of feasibility, efficiency, and simplicity.
Goal-driven planning performance Since GRASP is the first and only approach that empowers goal-driven planning, to evaluate whether GRASP is capable of generating high-quality goal-driven results, conduct double-blind user study of goal-driven planning on the WuxiTest. Specifically, we run GRASP using the building block molecules in the source pathways from different baselines as the GRASP's goal input to obtain goal-driven result. In addition. we also include goal-driven planning using the general (GRASP's general planning without specifying goal) and expert (goal in the reference routes from the chemists) source pathways. The results in Table. demonstrate that our approach can perform goal-oriented search and in the meantime generate a high- -uality result. We provide an exemplar of pathway comparison in reference for demonstration in Fig. Fandd Fig. 6 in Appendix C.
Compatibility to self-improved retrosynthetic planning Self-improved retrosynthetic planning IT] is an end-to-end framework that fine-tunes the single-step model to imitate successful trajectories found by fixed search (Retro* was used in the original work) algorithm by altering the prior distribution of single-step candidates in the search space. To evaluate the adaptation of GRASP with the self-improved framework, we follow the training procedure in [0]] by replacing Retro with GRASP and observe the performance on Pistachio and WuxiTest. As shown in Fig 3b) the self- ceff-improved retrosynthetic planning framework can improve the success rate of GRASP by using its own planning experience to fine-tune the single-step model.
4.4 Ablation studies
In this section, we investigate the following questions from different ablation studies: 1. The influence of two components: goal-driven relabeling (GDR) and self- oef--mmitation learning (SIL) over episodic reward during RL training. 2. How does the different probability of GDR affect the performance of general retrosynthesis planning and goal-driven retrosynthesis planning?
Influence of different components: We cross-check the training statistics of four combinations: GRASP without GDR and SII,, GRASP with GDR, GRASP with SIL, and GRASP with GDR and SIL. We evaluate the results on Pistachio by calculating the average reward with respect to training episodes. In binary reward setting, we use success rate as the criteria for reward evaluation, and the result shown in Fi On the one hand, SIL significantly improves the overall training statistics but induces a more sigmificant variance in the training process. The phenomenon is attributed to higher variance when using Monte-Carlo return and inevitable trade-offs in gradient propagation from different successful pathways under the same target. On the other hand, GDR also offers a certain amount of performance enhancement by relieving training difficulties induced by the sparse rewards. However, we are more interested in GDR's 's contribution to goal-driven planning.
Influence of GDR probability: The main hyperparameter we are interested in is GDR probability, which adjusts the distribution of transition tuples in the replay buffer for general and goal-driven planning. Specifically, we use the WuxiTest to evaluate the trade-off between general and goal-driven planning success rate for the hyperparameter controlling GDR probability. For goal-driven planning, we use the building block molecules in expert pathways for goal-driven input. The result is shown in Fig[d]] As expected, we observe that the success rate of goal-driven planning is lower than general planning, as requires both general success and the specific goal reached. However, the success rate for goal-driven planning improves significantly when relabeling probability ranges from 10% to 70%. Nevertheless, high relabeling probability impairs the success rate for general planning since GDR might incur failures in general planning pathways, and increasing the proportion of goal-driven data leads to less proportion of general data. In conclusion, is crucial to select an appropriate GDR probability depending on the actual usage of GRASP.

---

GMAP
0.-
MCTS
Success
0.6
0.5
0.6-
0.
0.4
100
1.50
200
250
150
Single stop modal inference calls
Satt-mpprovdd Inerations
(a) The success rate under different limits of single-step inference calls
(b) The success rate with maximum 100 expan- sions vs. self-improved training iteration
0.3
0.9
coftery
0.7-
03
0.6
0.4
0.2
02-
Goneral
0.1
100000
200000
300000
400000
500000
Training pplenter
Goal-driven Retabel Probability
(c) The success rate episodes during GRASP training
(d) The success rates goal-driven relabeling probability
Figure 3: Experiments and ablation studies
Conclusion
This paper proposes GRASP, novel goal-driven retrosynthetic planning approach. Unlike existing approaches that limit their generalization planning knowledge in static dataset, GRASP can capture synthetic knowledge through self-generated experiences. Moreover, GRASP can perform goal- driven retrosynthetic planning that none of the existing approaches could explicitly accomplish. Experimental results on academic and industrial benchmark datasets demonstrate GRASP outperforms all baselines in general retrosynthetic planning and first achieves high high-quality goal-driven planning. have deployed GRASP in a real-world company to expedite the planning synthetic pathways toward novel chemical compounds. By transforming the originally expertise-intensin pathway discovery process into efficient automation, GRASP significantly reduces the cost the workforce on pathway design in both medicinal and process chemistry. More importantly. GRASP supports the high customization of goals and further alleviates the cost of human labor on pathway screening and post-processing We believe that work will significantly inspire related research more efficient and chemists-machinn interactive retrosynthetic planning frameworks. Nevertheless, the planning results from GRASP still need to be monitored directed by personnel with chemical expertise if the target molecule a novel or a rarely reported compound. The community lacks thorough research retrosynthetic planning algorithms and how to
on the robustness of different angle-step/multi-iee generalize to out-of-distributiin molecules, which we think could be an appealing research direction.
6 Acknowledgement
This work is sponsored by the Starry Night Science Fund at Shanghai Institute for Ad- vanced Study (Zhejiang University) and Program of Zhejiang Province Science Technol- ogy (2022C01044).
10

---

References
[1] M. Andrychowicz, F. Wolski, A. Ray. J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In Proceedings of the 31st Inter- national Conference on Neural Information Processing Systems, NIPS' 17, page 5055-5065. Curran Associates Inc., 2017. ISBN 97815 110660944
[2] B. Chen, C. Li, H. Dai, and L. Song. Retro*: Learning retrosynthetic planning with neural guided a" search. In International Conference Machine Learning, pages 1608-1616 PMLR, 2020.
[3] C. W. Coley, R. Barzilay, T. S. Jaakkola, W. H. Green, and K. F. Jensen. Prediction of organic reaction outcomes using machine learning. ACS central science, 3(5):434-44h May 2017. ISSN 2374-7943.
[4] C. W. Coley, L. Rogers, W. Green, and K. F. Jensen. Sescore: Synthetic complexity learned from reaction corpus. Journal of Chemical Information and Modeling, 58(2):252 261, 2018.
[5] G. Dulac-Amold, R. Evans, H. van Hasselt, P. Sunehag, Lillicrap, J. Hunt, T. Mann, T. Weber, T. Degris, and B. Coppin. Deep reinforcement learning in large discrete action spaces, 2016.
[6] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods, 2018.
[7] S. Genheden, A. Thakkar, V. Chadimova, J.-L. Reymond, O. Engkvist, and E. J. Bjerrum. Aizynthfinder: A fast robust and flexible open-source software for retrosynthetic planning. ChemRxiv, 2020. doi: 10.26434/chemrxiv. 12465371.v..
[8] R. Jöamez-oobarreli,, N. Wei, D. Duvenaud, J. Hernández-Lobato, Sänchez-Lengeling, T. D. Hirzel, R. P. Adams, and A. Aspuru-Guzik. Auto-
D. Sheberla, J. sguilera-Iparraguire matic chemical design using a data-driven continuous representation of molecules. ACS Central Science, 4(2), Jan 2018 ISSN 2374-7951.
[9] Han, Zhao, C. Lu, Huang, J. Wu, Shang, Yao, and Zhang. Gan-retro: Retrosyn- thetic planning with graph neural networks. AAAI, 2022.
[10] J. Kim, S. Ahn, H. Lee, and Shin. Self-improved retrosynthetic planning. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5486-5495. PMLR, 18-24 Jul 2021.
[11] A. Kishimoto, B. Buesser, B. Chen, and A. Botea. Depth-first proof-number search with heuristic edge cost and application to chemical synthesis planning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[12] G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. Opennmt: Open-source toolkit for neural machine translation, 2017.
[13] K. Lin, Xu, Pei, and L.. Lai. Automatic retrosynthetic pathway planning using template-free models, 2019.
[14] B. Liu, B. Ramsundar, P. Kawthekar, J. Shi, Gomes, Q. L. Nguyen, Ho, Sloane, Wender, and V. Pande. Retrosynthetic reaction prediction using neural sequence-to-sequencc models, 2017.
[15] S. Liu, M. F. Demirel, and Y. Liang. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules, 2019.
[16] J. Mayfield, D. Lowe, and R. Sayle. Pistachio: Search and faceting of large reaction databases. In Abstracts of papers of the American Chemical Society, volume 254, 2017.
[17] Oh, Y. Guo, Singh, and H. Lee. Self-imitation learning. In Proceedings of the 35th International Conference Machine Learning, 2018.

---

[18] D. Rogers and M. Hahn. Extended-commectivtty fingerprints. Journal of Chemical Information and Modeling, 50(5):742-754, 2010.
[19] M. Sacha, M. Blaž, P. Byrski, P. PabrowskiiTumanske M. Chrominski, R. Loska, P. Wlodarczyk- Pruszyáski, and S. Jastrzębski. Molecule edit graph attention network: Modeling chemical reactions as sequences of graph edits, 2021.
[20] Schwaller, T. Laino, T. Gaudin, P. Bolgar, C. A. Hunter, C. Bekas, and A. A. Lee. Molecular transformer: A model for eeeertaintyccalibraeee chemical reaction prediction. ACS Central Science, 5(9):1572-588, Aug 2019. ISSN 2374-7951.
[21] P. Schwaller, R. Petraglia, V. Zullo, V. H. Nair, R. A. Haeuselmann, R. Pisoni, C. Bekas, A. Iuliano, and T. Laino. Predicting retrosynthetic pathways using transformer-baset models and hyper-graph exploration strategy. Chemical Science, 11(12):3316-33 6-3325, 2020.
[22] P. Schwaller, A. C. Vaucher, T. Laino, and J.-I.. Reymond. Prediction of chemical reaction yields using deep learning. Machine Learning: Science and Technology, 2(1):015016, 2021.
[23] M. H. Segler and M. P. Waller. Neural-symbollic machine learning for retrosynthesis and reaction prediction. Chemistry-A European Journal, 23(25):56665577) 2017.
[24] M. H. Segler, M. Preuss, and M. Waller. Planning chemical syntheses with deep neural networks and symbolic ai. Nature, 555(7699):604-61, 2018.
[25] C. Shi, M. Xu, H. Guo, M. Zhang, and J. Tang. A graph to graphs framework for retrosynthesis prediction, 2021.
[26] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):444-88, 2016.
[27] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks, 2014.
[28] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need, 2017.
[29] D. Weininger. Smiles, chemical language and information system. 1. introduction to method- ology and encoding rules. 1988.
[30] Xie, R. Yan, P. Han, Y. Xia, L. Wu, Guo, B. Yang, and T. Qin. Retrograph: Retrosynthetic planning with graph search. Proceedings of the 28th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2022.
[31] C. Yan, Q Ding, P. Zhao, S. Zheng, J. YANG, Y. Yu, and J. Huang. Retroxpert: Decompose retrosynthesis prediction like chemist. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11248 1 1258. Curran Associates, Inc., 2020.
12


Reviews:
[{"review_id": "hvsRV-9WZ-t", "reviewer": "Reviewer_5v4d", "summary": "This paper formulates retrosynthesis planning as a reinforcement learning problem and proposes a Goal-dRiven Actor-critic retroSynthetic Planning (GRASP) framework. They propose first to learn a policy network that takes continuous actions encoding the structure-level molecular information to allow navigation in the huge discrete action space of single-step reaction candidates. Then GRASP learns a goal-driven Q-value estimation network to update the policy, by sampling both successful and failed experiences and relabeling the goals of sampled experiences. The learned Q-value estimation and policy networks join to guide the Monte-Carlo Tree Search, after which GRASP returns diverse pathways due to a good exploration-exploitation tradeoff. They evaluated the performance of GRASP on both an academic and an industrial benchmark dataset. The results and user studies demonstrate that GRASP outperforms all baselines in general retrosynthetic planning metrics by a significant margin and is the first to achieve high-quality goal-driven retrosynthetic planning.", "questions": "Could the authors include more recent related methods and properly compare with them?", "limitations": "No", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors successfully adjusted RL to the retrosynthesis plan using the actor-critic RL method to learn from positive and negative experiences and navigate through the single-step reaction space.\n\nThey also applied goal-driven relabeling in retrosynthesis by exploiting the data generated from the retrosynthesis policy. This help to overcome the challenge in the quality evaluation of pathways.\n\nThe experiments are conducted on multiple datasets, including the Pistachio reaction dataset and a real-world expert dataset ‘WuxiTest’.\nAblation studies are included to investigate the effectiveness of different components.\n\nFor experimental comparison, the authors did not include recent related methods [1] as baselines, and still regard Retro* as the state of the art, which makes the significance of the proposed method less.\n\n[1] Han, Peng, et al. \"GNN-Retro: Retrosynthetic Planning with Graph Neural Networks.\" AAAI 2022.\n[2] Xie, Shufang, et al. \"RetroGraph: Retrosynthetic Planning with Graph Search.\" KDD 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Mip8te5oaJx", "reviewer": "Reviewer_F5hv", "summary": "The paper proposes a novel multi-step retrosynthetic planning framework called GRASP. The superiority of this reinforcement learning-based framework mainly relies on the capability of capturing synthetic knowledge through self-generated experiences. The paper is a pioneer work performing goal-driven retrosynthetic planning. In practice, the experiments demonstrate the effectiveness and the superiority of GRASP on both pathway datasets and expert datasets. The numerical results show GRASP achieves state-of-the-art performances on multi-step retrosynthetic planning tasks.", "questions": "Lack of ablation studies on different choices of single-step predictor. If it is possible to improve the performance by using a superior single-step predictor like Graph-to-Graph[1], or RetroXpert[2]?\n\n[1] A Graph to Graphs Framework for Retrosynthesis Prediction, ICML 2020\n[2] Decompose Retrosynthesis Prediction Like A Chemist, NuerIPS 2020", "limitations": "The broader impacts of this paper should be discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n+ The paper is well-written and well-organized. The notations and figures are easy to follow.\n+ The idea is novel and works very well. It is a pioneer work performing goal-driven retrosynthetic planning.\n+ The practical results demonstrate that the method could be helpful for chemists.\n+ The analysis on the enhancement of GDR is convincing.\n+ Codes are provided.\n\nWeaknesses\n\n- The performance depends on an upstream single-step retrosynthesis predictor. It would be better if this predictor could be optimized jointly with the RL-based framework.\n- It seems like the expert datasets WuxiTest are not open-source resources.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M957sth6Uwl", "reviewer": "Reviewer_bEMH", "summary": "This paper aims to solve retrosynthetic planning with RL. The authors proposed an actor-critic framework called GRASP, short for Goal-dRiven Actor-critic retroSynthetic Planning. This is claimed to be the first approach that enables goal-conditioned retrosynthetic planning, which mitigates the challenge in quality evaluation of pathways by directly fulfilling the requirements prescribed by chemists. The proposed method also leverage hindsight goal relabeling to speed up learning. They evaluate the performance of GRASP on both an academic and an industrial benchmark dataset (Pistachio and WuxiTest), and show superior results over all baselines. ", "questions": "Nitpick:\n* Line 189: \"Segler et al.\" is missing in front of [24].", "limitations": "No discussion on limitations and potential negative societal impact at all.  Please add relevant pieces to the conclusion section. ", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Disclaimer: the reviewer has zero background knowledge in retrosynthetic planning and thus is not able check whether the problem setting and evaluation are sensible or not, and whether the claim that \"MCTS for retrosynthetic planning induces high-variance\" is really a problem that cannot be easily mitigated. To my knowledge, modern approaches that use MCTS with limited rollout length and bootstrap with value network in general don't suffer much from this problem, but I don't know if there is anything specific to retrosynthesis as discussed in Sec. 3.3. I'm inclined to think this claim needs evidence but not sure whether that's obvious enough in this specific setting.\n\nOtherwise, the proposed approach seems sound and mostly outperforms all baselines considered in this work (MCTS, DFPN-E, Retro*, Retro*-0, Hyper-Graph Search) on Pistachio and WuxiTest. The empirical evaluation seems comprehensive and appears to verify most of the main claims:\n* Goal-driven planning mitigates the challenge in quality evaluation of pathways by directly fulfilling the requirements prescribed by chemists\n* GRASP outperforms all baselines in general retrosynthetic planning metrics\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept