Active Retrosynthetic Planning Aware of Route Quality
Authors: Luotian Yuan
Yemin Yu
Ying Wei
Yongwei Wang
Zhihua Wang
Fei Wu
Venue: ICLR 2024 poster (2024)
Keywords: Retrosynthetic planning
route evaluation
reinforcement learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Retrosynthetic planning is a sequential decision-making process of identifying synthetic routes from the available building block materials to reach a desired target molecule.
Though existing planning approaches show promisingly high solving rates and low costs, the trivial route cost evaluation via pre-trained forward reaction prediction models certainly falls short of real-world chemical practice.
An alternative option is to annotate the actual cost of a route, such as yield, through chemical experiments or input from chemists, while 
this often leads to substantial query costs.
In order to strike the balance between query costs and route quality evaluation, we propose an Active Retrosynthetic Planning (ARP) framework that remains compatible with the established retrosynthetic planners.
On one hand, the proposed ARP trains an actor that decides whether to query the cost of a reaction; on the other hand, it resorts to a critic to estimate the value of a molecule with its preceding reaction cost as input. 
Those molecules with low reaction costs are preferred to expand first.
We apply our framework to different existing approaches on both the benchmark and an expert dataset and demonstrate that it outperforms the existing state-of-the-art approach by 6.2\% in route quality while reducing the query cost by 12.8\%.
In addition, 
ARP consistently plans 
high-quality routes with either abundant or sparse annotations.

Published as a conference paper ICLR 2024
ACTIVE RETROSYNTHETIC PLANNING AWARE OF ROUTE QUALITY
Luotian Yuan'; Yemin Yu²4; Ying iei'! Yongwei Wang 14 Zhihua Wang', Fei Wul 'Zhejiang University. PCity University of Hong Kong, 'Nanyang Technological University
"Shanghai Institute for Advanced Study of Zhejiang University 18010561992jjp edu. en, yeminyu2-c@mg cityu. edu. ying. weientu. edu.sg sg { fyongwei.. wang, zhihua. feiwu ju. edu. on.
ABSTRACT
Retrosynthetic planning is a sequential decision-making process of identifying synthetic routes from the available building block materials reach desired target molecule. Though existing planning approaches show promisingly high solving rates and route qual- ities, the trivial route quality evaluation via pre-trained forward reaction prediction models certainly falls short of real-world chemical practice. An alternative option is to annotate the actual quality of a route, such as yield, through chemical experiments or input from chemists, but this often leads to substantial query costs. In order to strike the balance between query costs and route quality evaluation, we propose an Active Retrosynthetic Planning (ARP) framework that remains compatible with the established retrosynthetic planners. On one hand, the proposed ARP trains an actor that decides whether to query the quality of a reaction; on the other hand, it resorts to a critic estimate the value of a molecule with its preceding reaction quality as input. Those molecules with high reac- preferred expand first. We apply approaches on both the benchmark and an expert dataset and demonstrate that it outper- forms the existing state-of-the-art approach by 6.2% in route quality while reducing the query cost by 12.8%. In addition, ARP consistently plans high-quality routes with either abundant or sparse annotations.
tion qualities
framework to different existing
1 INTRODUCTION
Planning retrosynthetic route is a central challenge in organic synthesis, requiring the break-down of a target molecule into available building block materials through sequence of reactions. This process com- prises two main components: (1) single-step reaction prediction predicting feasible reactions in a single step, (2) multi-step planning recursively selecting optimal molecules and reactions across multiple steps, where evaluating and ranking routes are of pivotal importance in shaping the planning policy. Previous efforts multi-step planning have focused on quickly accessing building block materials in limited number of single-step calls, resulting in an up to 99.47% success rate Xie et al. (2022) on specific benchmarks. Re- grettably, this emphasis evaluating search aspects overlooks the chemical practicability of planned routes, in practice. For example, there is a short but low-quality route in Fig. i The crux revolves around the definition of reaction quality, which is simply the predictive probability of reaction according to a pre-trained forward single-step prediction model in approaches such as Retro (Chen ct al. 22020) and GRASP Yu et al. 2022). This single-step model is trained predict feasibility rather
ie., whether route is
"These authors contributed equally to this work.
"Corresponding authors.

---

Published as a conference paper ICLR 2024
Reastum yicks
sichd:
0.01
kdg (Solved)
(Solved)
Figure A short but low-quality route.
Resction quality: 0.2
Reaction 0
louuguulitt naae Boste quality 0.200.99
Reaution 2 Reartion quality 5.9
Resction
Nigh-quality sounebemee)
Rescton
COOMc
Deastio quality 0.8
Renciion. 3
COOMs
A low-quality route and a high-quality route. The low-quality reaction will lower the ceiling quality of the route. When selecting the next molecules towards high-quality routes, their preceding reaction qualities are associated consideration.
Figure 2:
than the quality of reactions, thereby being biased towards highly feasible and frequent reactions instead of those with high qualities. An analysis illustrate this issue in Appendix B The ideal reaction qualities that meet real-world chemical practicability, e-g., yield of reaction, be either annotated experimentally in laboratory or by experienced chemists. Yet, annotating each reaction requires labor-intensive lab verification expert annotations, compounded by the task of soliciting qualities for all reactions along retrosynthetic route with route lengths ranging from 2.0 to 8.0(Yu et al. (2022) Liu et al. (2023a)). As verifying every reaction quality in the lab causes time delays and hinders automation, quality metric is required which expensive but not prohibitively so. In the real-life scenarios, such as online softwares like SYNTHIA(Lin jet al.] (2023)). is an ideal candidate integrate chemists into the Al planning process. Online annotations by chemists not only introduce minimal time delays and manageable labor costs, but also contribute valuable insights beyond mere reaction yields, such as toxicity, material costs, and work-up difficulty.
We are motivated to pursue framework that strikes balance between enhancing practical planning perfor- mance and minimizing annotation costs. The core idea the proposed reinforcement learning- based Active Retrosynthetic Planning (ARP) framework constitutes reaction or not and critic that evaluates whether to expand a molecule or not. Concretely, the actor takes the current reaction as input; observing in Fig. 2 that molecule with a high preceding reaction quality should be prioritized to expand first, the critic takes both the current molecule and its preceding reaction quality input. It is noteworthy that the estimated molecule values by existing refrosynthetic planning methods can also be readily incorporated into our critic network. The critic network predicts a value that reflects the molecule's 's synthesizability together with the expectation of initializing a high-quality route. Simultancously, the actor is trained to make the decision regarding the quality efflit--fffeciveness of querying for the annotated re- action qualities. Since querying for the reaction quality is a non-trivial task that induces an additional cost during annotation, we enforce a query cost to be paid whenever a query decision is made by the model. Given the diverse referential values of different reaction qualities during planning, we dynamically adjust the query cost to enhance the model's capability of identifying those reactions whose query results prove to be most quality-effective, thereby addressing the trade-off issue between query cost and the planning qual- ity. The key contributions of our paper outlined below. (1) Practical efficacy: we, for the first time, draw an insight into the disappointing practicality of cxisting retrosynthetic planners that regard single-step probabilities reaction qualities. The ARP framework addresses the issue and improves quality of planned routes by 6.2% on a benchmark and by 4.9% on an annotated dataset. (2) Generality: The ARP framework is also compatible with arbitrary off off-tff-shelf planners and further boosts their chemical practicality.
actor that decides whether query the quality of a
2

---

Published as conference paper ICLR 2024
Critic Architecture
Single-step model
g Actor
state
Reaction candidates
state quality
state
quallity
Next selection
states of candidates
Qo Critic
READOCT
Reaction qualities candidates
value
value
Reaction quality
True
query?
Annotation
False, fill Mask
estimator existing
Figure 3: Our actor-critic framework. The actor receives the reaction candidates and outputs query decisions. The critic receives molecules along with preceding reaction qualities and outputs the estimated values. pretrained offline estimator can be incorporated with the critic of architecture A. When training an online estimator with the critic simultaneously, ARP utilizes simplified critic architecture variant B.
2 RELATED WORK
Single-step prediction Existing single-step methods can be divided into three main categories, template- based, template-free, and semi-temppate-baseee Template-based methods pre-define reaction transforma-
tions as templates and select appropriate template candidates (Coley et al. (0117). Segler & Waller] (2017).
Dai et al. (2019) Chen & Jung (202T), Xie et al. (2023)). Template-free methods predict the reactants
the representation of SMILES sequences (Schaalle et al. (2019 ZZong et al. (2022)) or molecular graphs
(Sacha et al. (202T). Semi-teme cemi-template--i based methods decompose the task into two stages, center identifica- tion, and synthon completion (Shi et al. (2020), Yan et al. (2020), [Somnath et al. (2020)). Chen & Jung processes into local template prediction through a global attention
(202T) solves the identify-and-compe mechanism. Furthermore, Xie et al. (2023) address the issue that fxed parameters might be sub-optimal by the robust non non-nornnee con-paramttric local reaction template retrieval. Zhong et al. (2022) fixes the problem that SMILES neglects the characteristics of molecular graph topology and reaction atom transformations.
Yu
et al. 2023) sorts out two types of distribution shifts retrosynthesis prediction.
Multi-step planning Lin et al. (2020) takes advantage of single-step seq2seq model and Ishida et al. (202) further introduces domain knowledge to guide the search direction. Chen et al. (2020) designs a neural-based A*-like algorithm. Rather than tree-search policy, Xie al. (2022) proposes to combine graph-based search policy with the traditional A" algorithm. Kim et al. (2021b) perform two self-improved iteration algorithms imitate successful trajectories and find an optimal search policy. Afterward, Yu al. (2022) is capable of biasing the retrosynthetic planning toward a favorable goal prescribed by chemists. Schreck al. 2019) and Liu et al. (003a) consider the cost for each reaction
uniform and optimize towards shortest routes. However, a short route might have lower yield than a long route. Liu et al. (023b) introduces a novel multi-step planning approach via in-context learning, departing from conventional search algorithms. However, the primary objective of Liu et al. (2023a) and Liu et al. (2023b) is still evaluating success rates of planned routes. They not consider the real-world reaction qualities. Tripp et al.) (2023). which shares comparable motivation with ours, addresses the uncertainty of the stochastic retrosynthetic planning processes and focuses on finding several routes to complement their respective shortcomings. The methodologies above require the annotation of costs for every reaction along route. In the pursuit of in- corporating reliable reaction qualities derived from chemists or lab experimentation, which entail significant costs, these methods tend to be economically impractical for real-world deployment.
3

---

Published as a conference paper ICLR 2024
3 METHODS
3.1 ACTIVE RETROSYNTHETIC PLANNING MDP
In our work, we consider the active retrosynthetic planning scenario modeled as a Markov decision process (MDP), represented by M {S, (A", A c). Specifically, S denotes the state space comprising chemical molecules, A' refers to the action space of candidate reactions, and A = {0, 1} represents the action space for query decisions. At time step t. given molecule state 8t, the agent opts for a decision through a basic action pair "o";a"), where aj denotes candidate reaction and az indicates the agent's s decision to query (or not) for the annotated reaction quality. The agent observes the reaction quality u, of of only if ag = 1. The P represents the deterministic state transition function from St to 8c+-1 via execution of reaction a.. R denotes our reward function and c the constant query cost. Contrary to prior studies that employed binary reward function (yielding reward of 1 if the reaction reaches the building block materials, and 0 otherwise), this work associates the successful reward with both route quality and the query cost, outlined in Eq.
A+u
No if a" reach I X+1
R(s, a², ..))
0
otherwise
I denotes the building block materials. cº represents the route quality, defined as the cumulative product of reaction qualities IT== u4 Both route and reaction qualities fall within the interval [o, 1]. N,, is the number of the annotated reactions, calculated by Σr= ag- À is hyperparameter employed to stabilize the training when the agent obtains a success route with extremely small u.
3.2 MODEL FRAMEWORK AND TRAINING PROCEDURE
We employ an actor-critic framework for the approach. Given a reaction a", the actor Tz. A" + A is responsible for making query decision a², determining whether to query the reaction quality not. The observation function O transforms the reaction and the query action to d -dimensional embedding of the reaction qualities expressed in Eq. 2 We use binning strategy B to discretize the continuous reaction quality values into A buckets and obtain the associated bucket embedding. The bins are constructed in order cover similar amount of reactions individually. In addition, we regard M separate bucket embedding. Details are given in Appendix
C
O(a", = B(u) M if a
if =
(2)
For given molecule s with its corresponding reaction quality u. , the critic Qe(s, u) estimates the value of s. The predicted values are used to access the value of the next state selection during RL roll-outs. To integrate existing retrosynthetic planners, standard molecule estimator E: S + R is assumed within ex- isting algorithms, whose task is to evaluate the values of molecules within their specific search frameworks. As summarized in Appendix A existing methods can be roughly divided into offline and online categories. Offline methods, such as neural-based A search in Chen et al. (2020) and Somnath et al. (2020), train the estimator on the extracted routes from the publicly available reaction databases an offline manner, in- cluding the United States Patent Office (USPTO)
(istachio(Mayyilld et al. (2017)). In contrast, online methods such as Yu et al. (2022) leverage routes generated by online roll-outs. We design the critic net- work Qo(E(s), u) RE (RFADDUU(E(S)) u) as architecture A shown in Fig. 3 In this design, arbitrary pretrained offline planners can be seamlessly integrated into the critic. Additionally, online planners can be trained simultaneously with ARP training, simplifying the critic architecture to architecture B in Fig. 3
4

---

Published as a conference paper ICLR 2024
Algorithm 1: Training algorithm
Initialize the estimator ε, the actor policy To, the critic value function Qo. the initial state so for t=0 to 7' do
Observe reaction action space {a.] of from environment
Make query decisions {a] Ji=1 and observe reaction qualities {u = O of)
Observe the next step states 1
Select the next state St+1 by arg gisk Qo(E(s(+1).
Compute reward = R(se, of, a?)
Append (8,, 1 (a), a.) re, St+1) to the buffer
end
Update To and Qo by Eq. 3 and Eq.
Such synergistic optimization not only conserves training resources for route collection but also substantially enhances both the solving rate and route quality compared in comparison to the two-stage training process.
At time step the agent observes state 8, along with batch of reaction candidates {aī) The actor chooses query actions {at) and observes the corresponding reaction qualities {u, Oaa )=) Subsequently, the respective next states (si ) of reaction candidates obtained employing the state transition function P. The next state $++1 is then selected with the critic by arg max Qo(E(s) uz).
This comprises complete step for executing single step roll-outs, and step reward r. R(s, a", a²) is determined by the reward function in Eq. 1 The training procedure performs recursive roll-outs to collect trajectories for training and terminates each roll-out whenever it reaches dead/building- -blockk--bb--k-- molecules maximum depth. We store the transition tuple (8t, 1 (af, az) Tt, 51+1) into the replay buffer for training.
The actor-critic framework is trained using the TD3 algorithm. The target critic network is updated through the one-step TD equation where Q' and n denote the target critic and actor networks, respectively, which initialized using the same parameter from the main critic and actor networks Q9 and Th but updated through an asynchronous manner. Utilizing the TD target yı, the mean square eπrоr loss across the batch is computed for the original critic network Qo as follows:
L( 1 (ol(()((()))) u(-1)
ydd Tt + (()((((()))))) -cd
(3)
Given the actor's objective is aligned toward maximizing the total return, and the critic network aims approximate this cumulative return, the actor F is trained by maximizing the Q-value, which is achieved through the minimization of loss in Eq. 4 If is an offline estimator such as the value estimator in Retro*, it is trained before implementation in our framework. The parameters of & are frozen and not evolved in training of the critic and 8 O. If & is an online estimator, can be regarded as part of the critic parameters and is wrapped with the other critic parameters for training and E' = E. The algorithm is summarized in Algorithm
L(d,E')= -- N Σ 1 (-Qo(E(ss), (c(())
(4)
3.3 INFERENCE PROCEDURE FOR ACTIVE RETROSYNTHETIE SEARCH
This section demonstrates the inference procedure for planning with partial observation of reaction qualities. the inference stage, the reaction qualities are annotated by either a surrogate model or a chemist expert. We combine the actor Ty and the critic Qo into the Monte-Carlo tree search (MCTS) on the AND-OR search
5

---

Published as conference paper ICLR 2024
Molecule nodes
(1) Selection
Expansion
(3) Update
Reaction nodes with qualities
Reaction nodes without
annotated qualities
Selected nodes
Expanded nodes
Updated nodes
Figure 4: The active inference procedure. The active retrosynthetic search comprises three steps: selection, expansion, and update. Firstly, the planner selects leaf molecule node with the maximum Q" value. Secondly, the selected node is expanded with AND-OR stump, and (a(a") decides whether the new reaction node has an annotated reaction quality. Finally, update Q of all nodes along the pathway.
tree. The molecule nodes are 'OR' nodes and the reaction nodes are 'AND' nodes, Moreover, we define a non-building block molecule node to be expandable when it is leaf node has expandable child reaction node. None of the building block molecule nodes are expandable. A reaction node is expandable when there is any expandable child molecule node and none of the children dead nodes. Molecule nodes are dead when they reach the maximum horizon or there are no reactions proposed by the single-step model. Let ch(-IT) denote the expandable children nodes of molecule reaction node.
There are three steps for each rollout during the active retrosynthetic search: selection, expansion, and update. A function Q" (st, af) in Eq. 5 presents the molecule syathesizability involves exploration. We use a UCT(Upper Confidence Bound applied to Trees) function (Kocsis et al. (2006)) to balance between exploitation of the route with the maximum and exploration of those less frequently visited routes, is a arper-parameter. ((st, az)) is value initialized by (0(s++1, O(aj, (a()))) and updated afterward. A visit count number is denoted by N(s a") and initialized 1.
(*,,,, afs., + B (N(si-1,
(5)
1+ N(s
Selection The search tree T starts with a single target root molecule node 8g. We recursively perform a reaction action selection of arg m (* a²) and obtain the next state 8141 by tranition function P until reaching leaf molecule node.
Expansion In the expansion step, we expand an AND-OR stump under the selected molecule node s, by referring to the single-step model. Every candidate reaction proposed is appended as a child reaction node 8.. For a newly generated reaction node af, we assign its reaction quality observation by O (a()). Each newly generated reaction node then expands children with the reactant molecule nodes.
Update During the update step, we perform Q(s;, a.)) values and visit counts (-TT)) backward traversal following the path from the selected leaf node back to the root node. Q' is the Q values of newly added state-action pairs. We use simple moving average for the update with a discount factor 7 by Eq. 6
1
Q'( at) Q(se, N(ss.
(70' Q(sr,
N' az) N(6,, a3)
(6)
6

---

Published as a conference paper ICLR 2024
BENCHMARK TEST SET
EXPERT TEST SET
SUCCESS RATE
QUERY RATE
NORMALIZED ROUTE QUALITY
SUCCESS RATE
QUERY RATE
NORMALIZED ROUTE QUALITY 1
ALGORITHM
HGSEARCH
36.5%
100%
0,518
49.0%
100%
0.550
RETRO*-0
44.9%
100%
0.644
57.3%
100%
0.765
RETRO*
53.4%
100%
0.705
60.9%
100%
0.787
RETRO*-04
63.5%
100%
0,621
61.2%
100%
0.665
RETRO*
72.5%
100%
0.671
68.5%
100%
0.699
EG-MCTS-0
73.6%
100%
0.489
65.4%
100%
0.726
EG-MCTS
85.4%
100%
0,558
75.4%
100%
0.712
GRASP
86.0%
100%
0.602
79.5%
100%
0.666
RETRO* WITH ARP
73.0% 86.5% %
87.2% 83.4%
0.767
74.3%
79. 9%
0.821
GRASP WITH ARP
0.711
80.6%
90.8%
0.836
Table 1: Performance of baselines and our approach on the benchmark and expert datasets.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Baselines. We compare solving rate and route quality against various open-source baselines including beam-search-like algorithm guided hyper-graph search method HgSearch (Schwaller et al. (20((2000)) best- first A liie algorithm guided AND-OR tree search methods Retro and Retro -CCro et al. (2020)), and an experience-guided MCTS-based method EG-MCTS(Hong et al. (2023). We apply ARP on an offline algorithm Retro*+(K3) et al. (2021a) and an online one: GRASP Yu et al. (0022)). Retro* is based on Retro* with self-improved single-step step retrosynthetic model. GRASP applies goal-driven actor-critic RL agent. As all of the existing methods rely on the reaction quality for calculating the search heuristics,
we set their default query rate to 100%. We change the resource of reaction qualities from the single-step probabilities to our reaction qualities when testing the baselines.
Materials and RL environment. We use two test sets evaluate methods. The first one is widely used USPTO-50K benchmark dataset that has 178 hard molccules raised in Chen et al. (2020). However, the scale of the benchmark dataset is small. Therefore, we include additional expert dataset that has 8000 molecules and each molecule has a literature reference route extracted by chemist experts. The expert dataset is designed emphasize retrosynthetic strategies with more challenging but strategically similar molecules. We partition the expert dataset as 0.8/0. 1/0. into train/valid/test sets. The training set is used the target molecules. For the single-step retrosynthesis model, we in Chen et al. (2020) which is 2-layer MLP using the Morgan fingerprint as input. We adopt the top- 50 single-step reaction candidates and set the maximum number of single-step inference calls as 100. We use the commercially available molecules dataset eM olecules as the building block materials. As for the hyper-parameters, we set the maximum route depth and Eq. as
similar template-based model
Reaction quality annotation Unfortunately, there is no established barge-scale available reaction data set with respective reaction qualities or promising reaction performance, e.g. yield, prediction model(Jaagg Æt al. )2 and both lab verification and expert annotations are expensive and time-consuming. We adopt a surrogate model to provide reaction quality annotations. Initially, the method in Guo et al.) (2020) em- ployed to pre-train model utilizing the USPTO-MI dataset, followed by the fine-tuning of the model in reactions derived from the high-quality, expert-annotatee dataset. Conceptually, the model, when trained on the expert-annotated dataset, prioritizes the identification of high-yield reactions over high-frequency reac- tions. More details about the training of surrogate model are in Appendix [ We provide an experiment to demonstrate significant correlation between our surrogate model and reaction yields in Appendix E Dur-
7

---

Published as conference paper ICLR 2024
ACTOR CRITIC
RANDOM CRITIC
QUERY COST
SUCCESS RATE
QUERY
NORMALIZED ROUTE QUALITY
SUCCESS
RANDOM
NORMALIZED ROUTE QUALITY 1
ESTIMATOR
RATE
RATE 1
RATE
-0.01
72.4%
100.0%
0.772
72.4%
100.0%
0.772
0
73.0%
87.2%
0.767
69.7%
87.0%
0.744
RETRO*+
0.01
72.4%
63.6%
0.759
71.3%
64.0%
0.722
0.05
71.9%
24.4%
0.702
71.9%
24.0%
0.699
0.1
72.6%
95.9%
0.713
86.0%
96.0%
0.708
0.005
86.5%
83.4%
0.711
84.8%
83.0%
0,695
GRASP
0.01
85.4%
76.6%
0.709
85.4%
77.0%
0.683
0.02
86.0%
13.4%
0.681
86.0%
13.0%
0.666
0.05
85.4%
1.2%
0.654
86.0%
0.650
Table 2: Experimental results of the active query capability on the benchmark test set.
CRITIC
RANDOM CRITIC
QUERY COST
SUCCESS RATE
QUERY RATE
NORMALIZED ROUTE QUALITY
SUCCESS
RANDOM
NORMALIZED ROUTE QUALITY 1
ESTIMATOR
RATE 1
RATE
-0.01
73.1%
100.0%
0.830
73.1%
100.0%
0,830
0
74.3%
79.9%
0.821
70.8%
80.0%
0.765
RETRO*
0.01
71.2%
53.0%
0.790
72.6%
$3.0%
0,739
0.05
71.5%
12.5%
0.737
72.3%
12.0%
0.721
0.1
74.5%
0.0%
0.722
73.1%
0.0%
0.722
0
80.5%
100.0%
0,842
80.5%
100.0%
0.842
0.005
79.5%
0.772
GRASP
0.01
80.4%
85.4%
0,839
80.3%
85. 35.0%
0.776
0.02
79.8%
6.6%
0.749
80.9%
6.0%
0.733
0.05
80.0%
0.0%
0.732
80.0%
0.0%
0.732
Table 3: Experimental results of the active query capability the expert test set.
ing deployment, it is practical to replace the surrogate model with chemist to provide online annotations, c.g. coarse-grained quality rating from 0 to 10.
Evaluation metrics. We use three main metrics to comprehensively evaluate the performance different search algorithms. 1. Success rate: The success rate is defined as the percentage of solved molecules in the entire test set. 2. Query rate: The query rate is defined as the percentage reactions that are annotated with the reaction qualities in the inference stage. 3. Normalized route quality: Given route, compute the route quality by a cumulative product of reaction qualities. Both the reactions and route qualities range in [0, 1] and a larger value refers higher quality, also lower quality. We introduce the normalized route quality to evaluate the route quality performance. For each target molecule the inference test set, we perform an exhaustive brute-force search in limited depth and acquire the maximum Umax and minimum route quality umin for the success route. Especially, if there is only one successful route, we assume the single route quality as umai and Umax umin 0.01. We further define the normalized route quality as Eq. of route quality to eliminate the impact of different target molecules.
Quality norm
Umax 11
(7)
Umax Umin
8

---

Published as a conference paper ICLR 2024
4.2 RESULTS
Comparision with baselines The performance of all methods are presented in Table. D Concerning both the normalized route quality and the query rate metrics, our approach achieves the best performance on both datasets. The best existing method in the normalized route quality metric identified as Retro*, In the benchmark dataset, our approach outperforms Retro* by margin of 6.2 %, while achieving reduction in the query rate by 12.8%. Similarly, within the expert dataset, our approach outperforms Retro by 4.9 % and lowers the query rate by 9.2%. Regarding the success rate, both Retro and GRASP achieve moderately higher success rate compared to the original results, primarily due to the influence of the preceding reaction quality as potential molecular feature in predicting molecular synthesizability.
Active query capability We evaluate the performance of ARP for balancing the trade-off between the query costs and reaction qualities. In the context of the active query setting, the models are tested under diverse query cost settings. Intuitively, setting high query cost is to simulate a sparse-annotated environment, compelling the planner to rely on less annotated reactions. Conversely, near-zero query cost emulates setting. The experimental results on both the benchmark and expert datasets are listed in the left columns of Tab. and Tab. [] respectively. As increase the query cost, observe decline in the query rate, which mildly impacts the success rate and normalized route quality. The phenomenon reflects that our approach is capable of actively selecting reactions that contribute most significantly the reaction qualities in retrosynthetic planning. To further explore the active query capability of our approach, we conduct an ablation study where the planner chooses to query the reaction quality of reaction with a fixed random rate Eq. [ instead of employing the trained actor-network for making query decisions.
1 P af - 0 1-1P
(8)
We adjust the fixed probability p as the same query rate obtained from the baseline result under varied query cost settings in order to eliminate the confounding effect of different query numbers. The results are listed in the right columns of Tab. _ and Tab. 3 and demonstrate that the actor adeptly selects the most informative reactions to enhance the planning performance. As the random rate p escalates, the critic can utilize more annotated reactions, improving the precision of value estimation and, thereby, optimizing the search process toward discovering routes with higher quality. We observe that the success rate does not increase monotonically with the query rate. Optimizing the success rate and the route quality together can lead certain trade-offs, as demonstrated by a case study in Appendix (F) Additionally, the actor specifically trained to identify and query the most informative reactions in order to achieve a higher route quality. As result, when the trade-off appears between the success rate and the route quality, the actor+eritic approach might not improve but suppress the success rate when compared to the random+eritic baseline.
5 CONCLUSION
The paper proposed ARP, novel retrosynthetic planning framework aware of the route quality. Unlike existing approaches using a labor-free but trivial reaction evaluation which is biased to the high-frequent reactions, ARP adopts route-quality evaluation approach aware of chemical practicability Morcover, there exists a trade-off between enhancing the planning performance and saving the query costs of acquiring reaction and is able to perform active selection of the most informative reactions to observe their reaction qualities. Experimental results demonstrate ARP's capability of capturing high-quality routes under either abundant or coarse-annotation environments.
9

---

Published as a conference paper ICLR 2024
6 ACKNOWLEDGEMENT
This work is funded by National Natural Scientific Foundation of China (No. 62037001), the Research Matching Grant Scheme (No. 9229082), the Starry Night Science Fund at Shanghai Institute for Advanced Study (Zhejjang University) and Shanghai Al Laboratory.
REFERENCES
Colin Bellinger, Rory Coles, Mark Crowley, and Isaac Tamblyn. Active measure reinforcement learning for observation cost minimization. CoRR, 2oos/2005.1267,, 2020. URL httts://azziv.orrg abs/
2005. 12697
Binghong Chen, Chengtao Li, Hanjun Dai, and Le Song. Retro#: Learning retrosynthetic planning with neu- ral guided a* search. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Con- ference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1608-1616. PMLR, 13-18 Jul 2020. URL
mlr www.sppsssss..... v1 wtt.://sss.....................e chen20k html
Shuan Chen and Yousung Jung. Deep retrosynthetic reaction prediction using local reactivity and global attention. JACS Au, (600))661121600000 2021. doi: 10. 10.1021/jacsei lc00246. URL htttps:://oo..... 10. 10.1021/jacsaJ2 1c00244
PMID: 34723264.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences, 2023.
Connor W. Coley, Luke Rogers, William H. Green, and Klavs F. Jensen. Computer-assisted retrosynthesis based molecular similarity. ACS Central Science, 3(12):1237-1245, 2017. doi: 10. 11..1////ceent//// PMID: 29296663.
7b00355. URL /ttp:::://d...oooo htttp:///bbbooog///0.10////aeeee 1tw....eea./cccctaac...... 7b0033
Hanjun Dai, Chengtao Li, Connor W. Coley, Bo Dai, and Le Song. Retrosynthesis Prediction with Condi- tional Graph Logic Network. Curran Associates Inc., Red Hook, NY, USA, 2019.
Christian Daniel, Malte Viering, Jan Metz, Oliver Kroemer, and Jan Peters. Active reward learning. In Proceedings of Robotics: Science and Systems (RSS '14), July 2014.
Zhongliang Guo, Stephen Wu, Mitsuru Ohno, and Ryo Yoshida. Bayesian algorithm for retrosynthesis. Jour- nal of Chemical Information and Modeling, 60(10):44444488, 2020. doi: 10. www.///acs.ccim.o.e Oc00320.
URL https:://////. /oi. org.or///// 10 1ww.1111aaccc wer.s.eii..mmm 0c00320 PMID: 32975943.
Siqi Hong, Hankz Hankui Zhuo, Kebing Jin, Guang Shao, and Zhanwen Zhou. Retrosynthetic plan- ning with experience-guided Monte Carlo tree search. Communications Chemistry, 6(1): June 2023. ISSN 2399-3669. doi: 10.0088/442004002-00009111 URL
1038/
s42004 -23-00911-8
Shoichi Ishida, Kei Terayama, Ryosuke Kojima, Kiyosei Takasu, and Yasushi Okuno. Ai-driven synthetic route design incorporated with retrosynthesis knowledge. Journal of Chemical Information and Modeling, 6356)11357 2022. doi: 10.
Ic01074. /RRL:ttae///6...oo//00000.0
jein meim...0 c014
PMID: 35258953.
Yinjie Jiang, Yemin Yu, Ming Kong, Yu Mci, Luotian Yuan, Zhengxing Huang, Kun Kuang, Zhihua Wang. Huaxiu Yao, James Zou, Connor W. Coley, and Ying Wei. Artificial intelligence for retrosynthesis pre- diction. Engineering, 2022. ISSN 2095-8099. doi: http:///a...ngg//....0......gg22
URL
https: http://wwwcccccccoooooo///////aa
10

---

Published as a conference paper ICLR 2024
Junsu Kim, Sungsoo Ahn, Hankook Lee, and Jinwoo Shin. Self-improved retrosynthetic planning. CoRR, abs/2106.04880, 2021a. /ut.Battpss/aarcc..ooooo////////
04880
Junsu Kim, Sungsoo Ahn, Hankook Lee, and Jinwoo Shin. Self-improved retrosynthetic planning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5486-5495, PMLR, 18-24 Jul 2021b. URL wtt.htttss//ppocceeiiiii
ml hwwpp/gpww..........ppp.pppp..ee v139/k
html
W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer frame- work. In Proceedings of the Fifth International Conference on Knowledge Capture, K-CAP '09, PP. 9-16, New York, NY. USA, 2009, Association for Computing Machinery. ISBN 9781605586588. doi: 14445/1197735.5997733
htR.:ttts:// doi http://oooooorgooooo/oo/oo 1tt.o////1111111191111111
1597738
Levente Kocsis, Csaba Szepesvari, and Jan Willemson. Improved monte-carlo search. 2006. URL https:
7/api. stanttssmmaticccholla. wttpoooll...r..ooooooaoooa :
David Krueger, Jan Leike, Owain Evans, and John Salvatier. Active reinforcement learning: Observing rewards at a cost. CoRR, abs/201 1.06009, 2020. URL
Kangjie Lin, Youjun Xu, Jianfeng Pei, and Luhua Lai. Automatic retrosyathetic route planning using
template-free models. Chem. Sei., 11 11:3355-3364, 2020. doi: 10.1039/C9SC036666 URL. http:
7/dx. dox.doo.ozz org...rr//a.
/tlgg/10.1 10. 10390 1039/C9SC033669
Yingfu Lin, Rui Zhang, Di Wang, and Tim Cernak. Computer-aided key step generation in alkaloid total synthesis. Science, 379 ):45) 457, 2023. doi: 10.1 26tp////cccccccee555555 URL https: httpps://wwww //www. science.
84e8459
Guoqing Liu, Di Xue, Shufang Xie, Yingce Xia, Austin Tripp, Krzysztof Maziarz, Marwin Segler, Tao Qin, Zongzhang Zhang, and Tie- Yan Liu. Retrosynthetic planning with dual value networks, 2023a.
Songtao Liu, Zhengkai Tu, Minkai Xu, Zuobai Zhang, Lu Lin, Rex Ying, Jian Tang, Peilin Zhao, and Ding- hao Wu. Fusionretro: Molecule representation fusion via in-context learning for retrosynthetic planning, 2023b.
John Mayfield, Daniel Lowe, and Roger Sayle. Pistachio: Search and faceting of large reaction databases. In ABSTRACTS OF PAPERS OF THE AMERICAN CHEMICAL CHEMICAL.SCCITT volume 254. AMER CHEM- (CALSOC, 1155 16TH ST, NW, WASHINGTON, DC 20036 USA, 2017.
Mikohaj
Sacha, Mikolaj Blaz, Piotr Byrski, Pawel Sabrowskki-Tumankk Mikolaj Chrominski, Rafal Loska, Pawel Boddarczyk-Pruszynse and Stanislaw Jastrzebski. Molecule edit graph attention network: Mod- eling chemical reactions sequences of graph edits. Journal of Chemical Information and Modeling, 61(7):32333324, 2021. doi: 10. 10.10///ccs.cmmm 1cm.mm003...3.. 1c05537. Ctl.httss:/ //t:://dooo .ot.ooo.rg/11... 1021 1tw./002aacc. /acs.
jeim.1c0050 PMID: 34251814.
William Saunders, Girish Sastry, Andreas Stuhlmüller, and Owain Evans. Trial without error: Towards safe reinforcement learning via human intervention. In Proceedings of the 17th International Confer- ence on Autonomous Agents and MultiAgent Systems, AAMAS '18, pp. 2067-2069, Richland, SC, 2018. International Foundation for Autonomous Agents and Multiagent Systems.
John S. Schreck, Connor W. Coley, and Kyle J. M. Bishop. Learning retrosynthetic planning through sim- ulated experience. ACS Central Science, 56))9700-881, 2019. doi: PMID: 31263756.
/ac////etsccc.b00000.. URL
https://doi..on
1tt.1.0//aacceettsseeese 9b00055
Sebastian Schulze and Owain Evans. Active reinforcement learnnn with monte-carlo tree search. CoRR, abs/1 Abs/1803.04926, 2018. htw.hcpp./aaallv..o///////////1 04926

---

Published as a conference paper ICLR 2024
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher Hunter, Costas Bekas, and Alpha A. Lee. Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction. ACS Central Science, 5(9)1157--588, 2019. doi: 10. 102 wtwpssceentsc...0057700 URL
/ttp:s///oo...og11111102200 1011////e......aacccett.t....... 9b00576
PMID: 31572784.
Philippe Schwaller, Riccardo Petraglia, Valerio Zullo, Vishnu H. Nair. Rico Andreas Haeuselmann, Ric- cardo Pisoni, Costas Bekas, Anna Iuliano, and Teodoro Laino. Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy. Chem. Sci., 11:3316-3325, 2020. doi:
10.1039/C9SC05504HL URL http::tp////tt //tt:://d... doi 2tta:/oo...../.1...000//0///C111
Philippe Schwaller, Alain C Vaucher, Teodoro Laino, and Jean-Louis Reymond. Prediction of chemical reaction yields using deep learning. Machine Learning: Science and Technology. 2(1):015016, mar 2021. doi: 10. 2008//66222215//bb8888 URL //tp::::/dddcoo doi. org / 108 2632 /2tp://000000002222222225//////0
Marwin H. S. Segler and Mark P. Waller. Neural-symiollic machine learning for retrosynthesis and reaction prediction. Chemistry A European Joumnal, 23(25):5966 5971, 2017. doi: https://doi.orr/10..1000
chem.201605499. URL hctpps//cheenistty.eeuoo.p.. www.:pppp...nnnnn.ne.eee.eee htwpeliiiarra..wiie.e...... wiwew. / abss/1.1102//ee chemm 201 605499
Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, and Jian Tang. A graph graphs framework for retrosynthesis prediction. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th Interna- tional Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8818-8827. PMLR, 13-18 Jul 2020. URL https:://proceedii http:://ppoeedlnnn.......ees///s
www. www..press//l19)
shi 1hi20d.rr html
Vignesh Ram Somnath, Charlotte Bunne, Connor W. Coley, Andreas Krause, and Regina Barzilay, Learning graph models for template-fire frree retrosynthesis. CoRR, abs/2006.07038, 2020. URL htttss://aaa.:
org/ abs/2006. 07038
Kaushik Subramanian, Charles L. Isbell, and Andrea L. Thomaz. Exploration from demonstration for in- teractive reinforcement learning. In Proceedings of the 2016 International Conference on Autonomous Agents Multiagent Systems, AAMAS '16, pp. 447-456 Richland, SC, 2016. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450342391.
Austin Tripp, Krzysztof Maziarz, Sarah Lewis, Marwin Segler, and José Miguel Hemándezz-ooato. Retro- fallback: retrosynthetic planning an uncertain world, 2023.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the Thirty-Second AAAI Conference on Ar- tificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 18/IAAI 18/EAAI 18. AAAI Press, 2018 ISBN 978-1-57735-800
Shufang Xie, Rui Yan, Peng Han, Yingce Xia, Lijun Wu, Chenjuan Guo, Bin Yang, and Tao Qin. Retro- graph: Retrosynthetic planning with graph search. In Proceedings of the 28th ACM SIGKDD Conference New York, NY, USA. 2022. As- doi: 10. 10.1145/554678.555944 URL
on Knowledge Discovery and Data Mining, KDD '22, pp. 2120-2129
sociation for Computing Machinery. ISBN 978 9781450393850.
(tT.//33367833444441. 3539446
Shufang Xie, Rui Yan, Junliang Guo, Yingce Xia, Lijun Wu, and Tao Qin. Retrosynthesis prediction with local template retrieval, 2023.
12

---

Published as conference paper ICLR 2024
Chaochao Yan, Qianggang Ding, Peilin Zhao, Shuangjia Zheng, Jinyu Yang, Yang Yu, and Junzhou Huang. Retroxpert: Decompose retrosynthesis prediction like a chemist. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, 20, Red Hook, NY, USA, 2020, Curran Associates Inc. ISBN 9781713829546.
Yemin Yu, Ying Wei, Kun Kuang, Zhengxing Huang, Huaxiu Yao, and Fei Wu. Grasp: Navigating retrosyn- thetic planning with goal-driven policy. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 10257- 10268. Curran Associates, Inc., 2022. URL httpa::/ http://proceiiing...uurr.......p cctpuripppccppppee....pppp
paper/
2022/f:
Yemin Yu, Luotian Yuan, Ying Wei, Hanyu Gao, Xinhai Ye, Zhihua Wang, and Fei Wu. Retroood: Under-
standing out-of-distribution generalization in retrosynthesis prediction, 2023.
Zipeng Zhong, Jie Song, Zunlei Feng, Tiantao Liu, Lingxiang Jia, Shaolun Yao, Min Wu, Tingjun Hou, and Mingli Song. Root-aligned smiles: tight representation for chemical reaction prediction. Chem, Sci., 13:9023-900 2022.
doi: 10. 10.1039/22S022733. URL http: doi wtt.:oogg/11111.111
D2SC02763A
13

---

Published as a conference paper ICLR 2024
A
EXISTING PLANNING ALGORIHTM
3N-MCTS HgSearch
DEPN-E
RetroGNN
Metro
FusionRetro Retro-fallbcck
Algorithm
Online
Offline
Offline
Offline
Online
Offline
Online
EG-MCTS Retro"
RetroGraph GNN-Retro SimulatedExp
GRASP
PDVN
Algorithm
Online
Offline
Offline
Offline
Online
Online
Online
Table 4: Existing online and offline retrosynthetic planning methods.
Active reinforcement learning An active reinforcement Cearning(ARL) agent learns when to pay query costs and observe rewards al. (2014)) or other signals. wide range of work has focused on ame- liorating the problem of deñning complete reward function on trajectories in complicated real-world tasks, i.e. automated driving and robot grasping(Christtann CCristiano al. (2023), (Sannders et al. (2018) Subramanian al. (2016), Daniel et al. (2014)) To minimize reliance on human experts. Krueger et al. (2020). [Bllinnger et al. (2020). and Schulzze & Evans| (2018) study the active measure reinforcement learning(AMRL) framework under multi-armed bandit and tabular settings. Furthermore, |arnell et al. (2018) and Knox & Stone (2009) propose the TAMER framework which takes into account the time delays and noise when the human, a "teacher", provides rewards online to the agent, a "student""
SINGLE STEP PROBABILITY
As the single-step model is trained to predict feasible reactant precursors, is biased towards frequent reactions instead of those with high qualities. We verify the issue that frequently collected reactions in single-step dataset are not necessarily high-yield, which we substantiate based on an analysis from from/ SChwwaler et al. (202T) that explores yields reported in the open-source USPTO dataset.
The USPTO dataset with reaction yields in sub-gram scale(Schwaller et al. 2021) contains a large number of reactions and broad range of superclasses, and reaction distribution closely resembling that of the USPTO single-step dataset, such as USPTO-MIT. The actual reaction yield distribution of the above dataset, originally presented in Schwaller et al. (202I), is depicted in Fig 5 Notably, a significant proportion of reactions within the dataset exhibits relatively low yields, affirming that the USPTO single-step dataset is not inherently biased high-yield reactions. Fig (originally presented in [Schwaller et al.] (202I)) shows various superclasses of reactions, where each color corresponds to a superclass and the coverage area of each color roughly represents the frequency of that superclass of reactions the dataset. Combining Fig 5p and Fig 5p, we conclude that high-frequeng, superclasses do not show a significant correlation with high yields. For example, the superclasses annotated in purple and cyan demonstrate low yields, with only the green reaction superclass corresponding to high yields in Fig 5b.
summary, frequently collected reactions in a single-step dataset are not inevitable to be high-yield ones and the single-step probabilities are not biased to high-quality but high-frequency reactions.
C BINING STRATEGY
bining strategy B is performed to discretize the continuous reaction quality values in and obtain the associated bucket embedding. The preceding reaction cost is concatenated in the format its representation O(a", a²) in Eq[] Concretely, when a = 1, we derive O(a", aa) from (1) discretizing continuous reaction quality values into N discrete buckets, (2) learning M trainable embeddings in d dimensions for all
14

---

Published as conference paper ICLR 2024
1.0
Reaction number
Yield scale
0.0
390
Reaction superclasses
Literature yields
Yields/%
aa)
(b)
(c)
Figure 5: The figure is directly borrowed from Schwaller et al. (2021). USPTO yield analysis: (a) shows the superclasses which roughly reflect the reaction frequency in the dataset. (b) depicts the yield scales of reactions labeled by the superclasses in (a). and (c) displays the distribution of the reaction yields in the dataset.
buckets within our critic, and (3) determining the bucket index that the queried quality value belongs to and thereby the associated bucket embedding. When a 0, Oa" aя) = as a d -dimensional trainable embedding. In our implementation, Fig in the revision. These buckets are obtained via (1) collecting about 28M reactions during planning by GRASP and Retro", (2) computing their reaction qualities by surrogate model, and (3) defining the bucket boundaries to ensure that each bin covers a similar number of reactions.
consider dd 512 and N 18 buckets which defined in
300000
250000
Covered reaction mumber
200000
150000
100000
50000
3.0
25-2.
1.0
13
2.1
8.9
0.7
7--
0.50 0.5
0-4004
8.32
2210..11
0.1--001
0.9
40
Bin bounds In(u)
Figure 6: Bin bucket boundaries. Each bin covers a similar amount of collected reactions individually.
15

---

Published as conference paper ICLR 2024
D SURROGATE TRAINING DETAILS
We utilize 8-layer Transformer the architecture of our surrogate model. The hyper-parameters are listed in Tab 5 The training of our surrogate model involves two steps: (1) pre-training on the USPTO- MIT dataset, and (2) finetuning on in-house expert dataset of routes featuring high-yield reactions. important to note that we introduce step (2) precisely to ensure that high predictive probabilities from our surrogate model align with high yields.
Hyperparameters
Values
Encoder layers
Decoder layers
4
Encoder embedding dimension
2048
Encoder FFN embedding dimension
2048
Encoder attention heads
Decoder embedding dimension
2048
Decoder FFN embedding dimension
2048
Decoder attention heads
8
Optimizer
Adam
Learning rate
le-4
Weight decay
0.0001
N epochs
12
Clip norm
0.25
Dropout rate
0.1
Table 5: The output of the cross-validation used for the hyperparameters optimization
CORRELATION BETWEEN THE SURROGATE MODEL AND REACTION YIELDS
PCC-0.059
PCC-0.611
iiternture yielda
Reaction qualities by our surrogate model
(b)
Figure 7: PCC against reaction yields. (a) shows the PCC of the pre-trained model and (b) shows the PCC the finetuned model.
16

---

Published as conference paper ICLR 2024
To evaluate our surrogate model, we resort to a route-with-yield test Following the method described in Chen et al. 2020), we extract synthesis routes with yields from the JSPTO-milligran -ccale reaction yield dataset Schwaller al. (202T). For evaluation purposes, we randomly select 200 routes, encompassing approximately 1000 reactions. We thereby calculate the Pearson correlation coefficient(PCCC between the reaction quality predicted by our surrogate model and literature yield. In Fig 7 of the revised manuscript, (a) illustrates the 0.059 PCC of the pre-trained model while (b) shows the 0.61 PCC of the finetuned model, providing strong evidence that the surrogate model accurately predicts yields.
SUCCESS RATE AND ROUTE QUALITY INCONSISTENCY
There separate objectives optimize in our framework: the success rate and the quality. However, optimizing these two objectives together can lead to certain trade-offs, as demonstrated by the following case study.
As shown in Fig. 88, the root molecule Mo has three candidate reactions, and the Ro is identified as a high quality reaction. However, the child molecule M1 of Ro unexpandable dead node with no further reaction candidates. If the planner makes the selection with observable next state molecular structures of M1. M2 and M3 and unobservable reaction quality values of Rg. R1 and R2, might selects R2 for its most synthesizable next state molecule Mg. However, with observable reaction quality values, the planner could be misled into selecting Rg due to its highest route quality expectation, which demonstrate an inconsistency between the two optimization obejectives.
OH
0,108
0.001
R,
R;
R;
oN
M,
M,
M
Figure 8: A case for illustrating two objective inconsistency. reactions, and the Ro is identified as a high quality reaction. However, the child molecule Mj of Ro is a unexpandable dead node with no further reaction candidates.
The root molecule M, has three candidate
G REAL-LIFE RETROSYNTHETLE PLANNING SCENARIOS
The quality metric required by framework in real-life scenario should be expensive but prohibitively so. While a single-step model is not competent enough, a lab validation might be excessively expensive and time-consuming. This consideration constitutes the primary motivation behind our active planning frame- work, aiming to query a minimum number of reaction quality annotations while still planing high-quality routes.
17

---

Published as conference paper ICLR 2024
While our current implementation involves querying the surrogate model, our inspiration is drawn directly from real-life retrosynthesis planning scenarios, such as in online softwares like SYNTHIA, where chemists are pivotal end users. this context, integrating chemists as valuable resources into the Al planning process will be invaluable for planning routes that are not only feasible but also of practical high-quality. We envision the successful deployment of our framework this scenario for several reasons.
Online annotation by chemists introduces minimal time delays and manageable labor costs, making it an ideal candidate for route quality metric that is expensive but not prohibitively so. Our framework is in- tentionally designed to be compatible with various types of annotations, including coarse-grained quality rating from 0 to 10. We believe such rating is sufficient for the planner to make satisfactory decisions. Ad- ditionally, this rating can also be seamlessly integrated into current framework by replacing the bucket index to which a quality value belongs (see details Section 3.2 in the revision) with this discrete rating. Chemists contribute valuable insights beyond mere reaction yields, such knowledge about preferred re- actions in real-world synthesis contexts, which can include factors like toxicity, material costs and work-up difficulty (post-proceed like purification or separation).
H FUTURE WORK
Although we focused on the high-quality routes, the retrosynthetic planning has other essential considera- tions like the green chemistry. future work, we intend to investigate Active Retrosynthetic Planning with multi-objective optimization in order to find eco-friendly routes of high chemical feasibility.
1
CASE STUDY
We conduct a double-blind test to check the route quality generated by Retro, ARP with Retro*, GRASP, and ARP with GRASP. We collect top-1 successful routes from the experimental results of the benchmark dataset and the chemists tag the route with a rating from O to 10. 10 refers to a high-quality route while O refers low-quality one. We lie the average rating in Tab. 6 Compared with the original methods, ARP with Retro* outperforms Retro by 1.7 and ARP with GRASP outperforms GRASP by 2.2.
Retro
ARP with Retro
GRASP
ARP with GRASP
Route rating(1-10)
7.8
8.5
6.9
9.1
Table 6: Double blind test on the top-1 route quality.
HCHO
HB
NH
Br
Figure 9: A target molecule.
Furthermore, we study a case to illustrate the active query capability. In Fig. 9 target molecule has three basic molecular structures that need to be broken down by respective templates, To. Ti, Tz- Simplified,
18

---

Published as a conference paper ICLR 2024
M4
0.962
0.0135
0.999
Rg
KSHO
0 0.999
0.106
R2
HCHG
M2
M,
0,989
0.969
Rg
R,
Figure 10: search tree by ARP based on GRASP.
the planner needs to decide the order of executing three templates. However, if To executed after T1, will produce low-quality reaction because Ti reveals high-activity amino group blocked green. From chemical perspective. Ti can be regarded as deprotection reaction to suppress side reactions on the amino group for Tp- Thus To must precede Ti. We visualize search tree in Fig. 1O planned by ARP based on GRASP to solve the target molecule with the query cost equals 0, 0.01, and 0.05. For simplicity, we ignore some molecule nodes and reaction nodes. We tag the reaction qualities on the blue reaction nodes, the non- building block molecules on the yellow nodes, and the building block molecules on the green nodes. The empty blue nodes present reaction nodes of which the qualities are not annotated. Furthermore, we tag the Q value near the respective molecule nodes to explore the reaction quality annotation's 's impact. In the three search trees, the molecule node selection among My. M,, and M2 is key decision that determines the next following expansion of the whole search tree. Mo will results in a high-quality route while Mj and M2 will lead to low-quality routes. Mj has low preceding reaction quality and M2 has low future-quality expectation. M1 is the best next molecule node to expand. We compare the situations with different query costs.
Ful observation: With query cost of O, the actor in ARP queries every reaction qualities in the search tree. The search tree is depicted in Fig. II Q values reflect the molecule's 's high-quality route expectation properly. The planner selects Mj as the next molecule state node properly.
Partial observation: With query cost of 0.01, the actor in ARP selects partial reaction qualities in the search tree observe. The search tree is depicted in Fig. 12 It is observed that reaction qualities annotated. The molecule with the maximum Q value maintains Mj. Nevertheless, the unannotated reaction quality of R2 misdirects the Q value estimate of M3
extent. Though the ranking prior between
19

---

Published as conference paper ICLR 2024
AMB
KLPP
Ni
Q0Mg 0.71
Mi OHMS
Figure 11: The search with query cost of 0.0
M2 and M3 changed compared with [1] the planner still selects M1 to expand next. This phenomenon
demonstrates the query ability of ARP to select the most impactful reactions to annotate qualities.
0082
DEESS
Mi
QIM
039
Q04a) 0.77
Mi QHM
Figure 12: The search with query cost of 0.01
None observation: With query of 0.05, the actor in ARP selects reaction qualities in the search tree to observe. The search tree is depicted in Fig. [3] It is observed that the unannotated reaction qualities misdirect the value estimates of three molecules. In contrast to Fig. [T and Fig. 12 the next selected molecule changed into M2. This issue, on the one hand, illustrates how reaction qualities benefit retrosyn- thetic planning, on the other hand, proves the active capability of utilizing the reaction quality annotations to find high-quality routes.
Seag
X6
Q0M
Q0M
Mi
Figure 13: The search tree with query cost of 0.05
20


Reviews:
[{"review_id": "LGDsq64SJT", "reviewer": "Reviewer_iEQU", "summary": "an RL based search algorithm that incorporates some definition of route quality for retrosynthesis is proposed.\n\n\nedit after discussion\n- I am still not 100% convinced about the novelty of this work, but nevertheless will raise the score.", "questions": "- I would suggest to order the related work section, in particular the Multi-step planning section, by the order in which the works appeared, because subsequent works influenced each other.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- interesting new ideas of using RL for scoring applied to retrosynthesis", "weaknesses": "- somewhat disconnected from previous evaluation\n- the model to assign quality is not described in enough detail\n- it is not convincingly defined what reaction quality is\n- prior work is not correctly attributed\n\n\"Practical efficacy: we, for the first time, draw\nan insight into the disappointing practicality of existing retrosynthetic planners that regard single-step probabilities as reaction qualities.\"\n\nDifferent and flexible ways to incorporate step and route cost beyond single step probabilities) was already included in many prior works, for example Segler (2018), Coley (2019) and Schwaller (2020). These works also used reaction feasibility models, which are a proxy for quality. Please reference this accordingly.\nalso, the method seems to be very similar to Liu et al 2023, which is also a very flexible framework able to incorporate arbitary cost. this needs to be discussed more carefully.\nI think the paper can receive higher score if prior work is referenced properly, but in the current form this is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3FDjzL9vq", "reviewer": "Reviewer_BDYr", "summary": "This paper aims to consider the route quality (i.e., yield of a reaction) in the retrosynthetic planning problem. Since querying such information requires labor-intensive lab/expert verification, this paper proposes an active planning framework to strike a balance between query cost and route quality. This framework consists of an actor that decides whether to query the cost of a reaction or not and a critic that evaluates whether to expand a molecule or not.", "questions": "#### Unverified Claims\n1. In reaction cost annotation, the authors adopt a surrogate model to provide reaction cost annotation, and hope “the model prioritizes the identification of high-yield reactions over high-frequency reactions”. Can the authors empirically verify this claim? \n2. Are there real cases when high-yield reactions and high-frequency reactions are not overlapping? Intuitively, high-frequency reactions are usually high-yield ones, otherwise they cannot be frequently collected in a dataset, right?\n\n#### Method\n3. How is the state encoder $\\mathcal{E}$ trained? It should also wrap $s_t$ in Eq. (4), right?\n4. In Selection, is there any exploration or possibility to generate multiple routes? \n5. How is the masked value M set?\n6. During inference, does the method require the external surrogate model to obtain the reaction cost? How accurate/reliable is this surrogate model?\n\n#### Writing\n7. The term “reaction cost” is misleading and inconsistent: “cost” usually refers to some undesired property that should be as low as possible, however, this paper defines cost as yield (i.e., higher “cost” indicates better quality). This is misleading and counterintuitive. Meanwhile, “observing in Fig, 2 that a molecule with a low preceding reaction cost should be prioritized to expand first”, is clearly inconsistent with Fig. 2 (i.e., low-cost=low-quality route should not be prioritized). \n8. The description of Expansion in the inference stage is not clear. Q* sometimes refers to a value, or a function (i.e., in Eq. (5)), which is confusing. What is Q*(a) and Q*(s) exactly?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The route quality in terms of yield is of pivotal significance for practical CASP, and this paper is well motivated from this perspective;\n2. The active planning framework is designed to consider route quality, with balancing the query cost; the actor-critic technique adopted in general is reasonable and justified;\n3. The evaluation using defined metrics support the major claims of balancing query quality and rate.", "weaknesses": "1. The paper representation can be improved with better clarification. See details in Questions.\n2. Some claims that motivate the method should be verified. See details in Questions.\n3. Important related work [1] that targets on multi-step planning should be discussed.\n\n[1] Liu, Songtao, et al. \"FusionRetro: molecule representation fusion via in-context learning for retrosynthetic planning.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "By8myqDbiE", "reviewer": "Reviewer_zSxo", "summary": "The paper addresses the challenge of retrosynthetic planning when the reaction cost is unknown during the planning process. To balance the trade-off between query costs and route quality evaluation, the authors introduce the Active Retrosynthetic Planning (ARP) framework. This framework employs an actor-critic model to decide whether to query reaction costs and to estimate molecule values. The ARP framework demonstrates a 6.2% improvement in route quality on a benchmark dataset and a 4.9% improvement on an annotated dataset, while reducing the number of query costs.", "questions": "- In the ablation study, why doesn't the actor+critic consistently and significantly improve the success rate compared to the random+critic baseline, as observed with normalized route quality?\n- Why doesn't the success rate change monotonically with the query rate, unlike the normalized route quality?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies a key issue in retrosynthetic planning: the assumption of knowing the reaction cost for every reaction during the planning phase is impractical. In response, the authors introduce a novel solution by leveraging an actor-critic framework from reinforcement learning to address this problem.\n- The proposed method demonstrates strong empirical performance on top of state-of-the-art methods, as shown in comprehensive evaluations.\n- The proposed method is generic, able to work alongside various existing retrosynthetic planners, enabling their capabilities to balance between query costs and route quality evaluation.\n- The paper introduces the normalized route quality to normalize the scores for different synthesis routes, thus improving the accuracy of planning outcome evaluation.", "weaknesses": "- In the ablation study, actor+critic does not show consistent and significant improvement on success rate compared with random+critic baseline. The paper should include additional discussion to clarify this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Kfy0Cp1cUz", "reviewer": "Reviewer_23Kg", "summary": "This work considers multi-step retrosynthetic search in a scenario where an expensive-but-accurate reaction quality metric is available. This motivates the authors to develop an algorithm that decides to query said metric adaptively, so that it can learn to trade-off the cost of querying it against other costs/rewards inherent to the routes themselves. The authors then run experiments to compare with existing retrosynthetic planning algorithms.", "questions": "(Q1): Could you expand on \"critic takes both the current molecule and its preceding reaction cost as input\"? Is the previous reaction cost literally concatenated into the molecule representation somewhere as a single floating point number? \n\n(Q2): Is Equation 6 fully correct? I am confused about the fact that it does not include the discount factor $\\gamma$, and also the way $N(s)$ is used seems slightly off; I may be wrong though. \n\n(Q3): Could you elaborate on how the quality metric is used by the baseline algorithms like Retro* to produce the results from Table 2? I understand that for the baselines it is assumed the quality metric is always queried, which leads to 100% query rate. Could you discuss how the resulting quality value is used by the algorithms themselves during optimization? \n\n(Q4): Is the depth of the exhaustive search used to normalize route quality the same as the maximum depth that the search algorithms are allowed to explore?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(S1): The framework proposed by the authors is original, well-motivated, and generally reasonable. The paper also includes valuable discussion. \n\n- (a) In particular, the discussion on how routes produced by existing search algorithms may be not useful in the real world (due to e.g. the reaction being low yield or otherwise infeasible) is useful. This is something that should perhaps be given more attention in the ML-for-retrosynthesis  community. \n\n- (b) The high-level modelling and architectural choices are reasonable and appropriate for the task at hand.\n\n- (c) There are interesting connections of this work to retro-fallback [1] - a very recent parallel work - which is also concerned with the fact that the routes found by off-the-shelf algorithms may not be of high quality. However, I think the attack angles are somewhat complementary, as [1] tries to find _several_ routes that complement each other in their shortcomings, whereas this work focuses on _more accurately estimating the quality of a single route_ through a (possibly expensive) quality oracle. \n\n(S2): The paper is generally well-written and easy to understand (although with some room for improvement; see the \"Questions\" and \"Nitpicks\" sections).", "weaknesses": "(W1): My main concern is that it is not very clear to me what the quality metric would actually be in a real-life scenario and how it would be implemented. First, the framework proposed by the authors seems to only make sense if querying the quality metric is _expensive but not too expensive_: if the quality oracle is not much more expensive that the single-step model itself (e.g. if it's a yield prediction model), then there is no need for the adaptive querying; if the metric is very expensive (e.g. asking a human chemist or running the reaction in the lab) then it wouldn't make sense to allow the algorithm to query it autonomously, even if it only does so for e.g. 20% of the cases (as that would still be prohibitively expensive and the human/lab would become a bottleneck). What would a realistic quality metric be? The authors in the paper use a trained model, which perhaps falls into the category of \"not expensive enough to bother with adaptive querying\", unless the model is *much* larger than the single-step model itself. Or perhaps the quality metric could result from e.g. running expensive quantum chemistry simulation, but this is a bit forward-looking as I don't think any existing frameworks for such calculations are robust enough to plug them in directly. Overall, at the very least I would expect more discussion on how the quality metric should be realized in practice. \n\n(W2): The related works section is missing several references of established/SotA models and algorithms. On the single-step side, it is missing LocalRetro [2], RetroKNN [3] and RootAligned [4]. On the multi-step side, it is missing PDVN [5]. \n\n=== Summary ===\n\nI like some aspects of this work, but the general real-world use case could be more discussed. I am looking forward to the rebuttal and, if productive, I may consider revising my score accordingly.\n\n=== Other comments === \n\n(O1): The introduction section casually mentions that the average length of a retrosynthetic route is 3.36 without giving any sources. Where does this come from? I think this number would heavily depend on the precise assumptions, and for very challenging molecules the average route length would be even higher than this. It may be better to change this statement to be more vague (e.g. say that the routes are usually 3 or more steps) and give a citation, instead of using a very specific number which is only true if one fixes a lot of variables. \n\n \n\n=== Nitpicks === \n\nBelow I list nitpicks (e.g. typos, grammar errors), which did not have a significant impact on my review score, but it would be good to fix those to improve the paper further. \n\n- \"while this often leads to substantial query costs\" -> in this context I would replace \"while\" with e.g. \"but\" \n\n- in many places there is a missing space before parenthesis (see the \"Multi-step planning\" part of the \"Related work\" section for some examples) \n\n- \"with is initialized\" -> \"which\" \n\n- \"Moreover, We define\" -> lowercase \"we\" \n\n- \"There are three steps for each roll\" -> \"rollout\"? \n\n- First sentence in the \"Reaction cost annotation\" paragraph is a bit hard to parse \n\n- \"USPTI-MIT\" -> \"USPTO-MIT\"? \n\n- Equation 9 looks as if it's missing some text \n\n \n\n=== References === \n\n[1] Tripp et al, \"Retro-fallback: retrosynthetic planning in an uncertain world\" \n\n[2] Chen et al, \"Deep Retrosynthetic Reaction Prediction using Local Reactivity and Global Attention\" \n\n[3] Xie et al, \"Retrosynthesis Prediction with Local Template Retrieval\" \n\n[4] Zhong et al, \"Root-aligned SMILES: a tight representation for chemical reaction prediction\" \n\n[5] Liu et al, \"Retrosynthetic Planning with Dual Value Networks\"\n\n=== Update 21/11 ===\n\nAfter the rebuttal, I increase my score to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)