Retro-R1: LLM-based Agentic Retrosynthesis
Authors: Wei Liu
Jiangtao Feng
Hongli Yu
Yuxuan Song
Yuqiang Li
Shufei Zhang
LEI BAI
Wei-Ying Ma
Hao Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Retrosynthesis Planning
LLM-based Agent
Primary area: deep_learning
TL;DR: An LLM-based retrosynthesis planning agent trained end-to-end with Agentic Reinforcement Learning.

Abstract: Retrosynthetic planning is a fundamental task in chemical discovery. Due to the vast combinatorial search space, identifying viable synthetic routes remains a significant challenge--even for expert chemists. Recent advances in Large Language Models (LLMs), particularly equipped with reinforcement learning, have demonstrated strong human-like reasoning and planning abilities, especially in mathematics and code problem solving. This raises a natural question: Can the reasoning capabilities of LLMs be harnessed to develop an AI chemist capable of learning effective policies for multi-step retrosynthesis? In this study, we introduce Retro-R1, a novel LLM-based retrosynthesis agent trained via reinforcement learning to design molecular synthesis pathways. Unlike prior approaches, which typically rely on single-turn, question-answering formats, Retro-R1 interacts dynamically with plug-in single-step retrosynthesis tools and learns from environmental feedback. Experimental results show that Retro-R1 achieves a 55.79\% pass@1 success rate, surpassing the previous state of the art by 8.95\%. Notably, Retro-R1 demonstrates strong generalization to out-of-domain test cases, where existing methods tend to fail despite their high in-domain performance. Our work marks a significant step toward equipping LLMs with advanced, chemist-like reasoning abilities, highlighting the promise of reinforcement learning for enabling data-efficient, generalizable, and sophisticated scientific problem-solving in LLM-based agents.

RETRO-R1: LLM-based Agentic Retrosynthesis
Wei Liu 1,3,4 Jiangtao Feng Hongli Yu², Yuxuan Song
Yuqiang Li Shufei Zhang,, Lei Bai Ying Ma², Hao Zhou
Sccooo of Computer Science. Shanghai Jiao Tong University
Institute for Al Industry Research (AIR), Tsinghua University
3 Shanghai Artificial Intelligence Laboratory *Shanghai Innovation Institute
DDeparmeent of Computer Science and Technology, Tsinghua University
1 liuwei@si.. edu en, yuhl250mails. tsinghua, edu. cn (liyuqiang, zhangshufei, bailei]epjlab. org.cn
{fengj iangtao, songyuxuan, maweiying, zhouhao Cair tsiighuu., edu.
Abstract
Retrosynthetic planning is a fundamental task in chemical discovery. Due to the vast combinatorial search space, identifying viable synthetic routes remains significant challenge-even for expert chemists. Recent advances in Large Language Models (LLMs), particularly equipped with reinforcement learning, have demonstrated strong human-like like reasoning and planning abilities, especially in mathematics and code problem solving. This raises natural question: Can the reasoning capabilities of LLMs be harnessed to develop an Al chemist capable of learning effective policies for multi-step retrosynthesis? In this study, introduce RETRO- R1, a novel L.M--aaeed retrosynthesis agent trained via reinforcement learning to design molecular synthesis pathways. Unlike prior approaches, which typically rely on single-turn, -anseiinn formats, RETRO-RI interacts dynamically with plug-in single-step retrosynthesis tools and learns from environmental feedback. Experimental results show that RETRO-RI achieves a 55.79% pass @ success rate, surpassing the previous state of the by 8.95%. Notably, RETRO-RI demonstrates strong generalization to out-of out-of-donee oo-mmmninn test cases, where existing methods tend fail despite their high in in-domain performance. Our work marks significant step toward equipping LLMs with advanced, chemist-lik reasoning abilities, highlighting the promise of reinforcement learning for enabling data-efficient, generalizable, and sophisticated scientific problem-solving in LL.M-based agents.
Introduction
Retrosynthetic planning, fundamental problem in organic chemistry, starts from a target molecule and works backward to find pathway to transform it into commercially available starting materials. As a crucial task in drug discovery, retrosynthetic planning poses significant computational challenge due to its vast and discrete combinatorial search space M. Successfully navigating this space requires not only deep chemical knowledge of reactions and reagents but also sophisticated strategic reasoning to evaluate a multitude of possible intermediate compounds and multi-step reaction pathways (2). Moreover, retrosynthesis is inherently long- long-horizon reasoning task, demanding higher-order chemical logic: decisions made in the early stages of the pathway can profoundly affect downstream options and ultimately determine the feasibility of the entire synthesis route (
Building on advances single-step retrosynthesis prediction 4 [. machine learning techniques have increasingly been applied to the more complex task of planning complete multi-step retrosynthesis
*Corrsponding author. This work was done while Wei was a research intern at Institute for Industry Research (AIR), Tsinghua University. Jiangtao Feng was the project leader.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

pathways [6]. These approaches often formulate the problem as a search over vast graph of molecules and reactions, leveraging sophisticated algorithms guided by learned models [ Researchers typically follow two main strategies: (1) enhancing single-step models to favor reactions that are more suitable for retrosynthetic planning (9] 10,, or (2) training deep neural networks to serve as scoring functions for reaction feasibility or molecular complexity, which are then integrated into search algorithms such as Monte Carlo Tree Search (MCTS) [ or A [12].
Recently, Large Language Models (LLMs) have exhibited remarkable capabilities in complex plan- ning and reasoning tasks involving long sequences and intricate dependencies 13 14 Their ability
to generate coherent text over extended contexts, follow complex instructions 15 leverage external
tools [[6], and perform multi- step logical deductions [T7] has driven breakthroughs in domains such as complex question answering [18 19 20]. math problem solving [21 [3 code genera-
tion [24 25 26]. and agent- based simulations [2
major enabler of these capabilities has
been the integration of Reinforcement Learning (RI.), including techniques such as reinforcement
learning from human feedback (RLHF) 29,, I proximal policy optimization (PPO) 30 dynamic
sampling policy optimization [31], and attention srrentioo-ggidedd policy optimization [32 These emerging greengths-especian
in handling long-range dependencies and executing sequential reasoning- are
highly relevant to the challenges of multi-step retrosynthetic planning. They suggest that LLMs can serve as powerful, flexible, and context- aware agents for navigating and reasoning through complex chemical synthesis pathways.
Inspired by recent advances in both machine learning for chemistry and seneral-purpon LLM reasoning, we propose RETRO-R1 novel L.-M-based agent for multi-step retrosynthetic planning. Trained via end-to-end reinforcement learning, RETRO-RI is capable of making complex decisions to select optimal reaction candidates and adapt to dynamic changes in the environment. We employ Proximal Policy Optimization (PPO) (30] and introduce modifications to Generalized Advantage Estimation (GAB) (33) to effectively support multi-turn interactions and operate over structured state representations. To emulate the iterative, mowweegg-accumulltin process of human chemist, our framework enables the agent to conduct multiple planning attempts while incrementally constructing a global synthesis graph. This design allows for efficient exploration of the solution space and facilitates the discovery of high-quality synthesis routes within practical computational budget.
Our key contributions are summarized as follows:
We introduce a novel framework for training an LLM agent in end-to-end multi-step ret- rosynthetic planning. formulating the task as sequential decision-making problem with enviromment feedback.
We adapt Proximal Policy Optimization (PPO) agentic learning with frequent environment interaction, enabling effective training in structured planning settings.
We design an iterative planning strategy that builds a global synthesis graph across multiple sequential searches, leveraging memory to avoid redundant computations and emulate chemist's 's comprehensive exploration process.
We demonstrate that our framework achieves state-of of-------th-e the-arrt or highly competitive results on benchmark retrosynthetic planning datasets, exhibiting strong generalization and discovering shorter synthesis routes than existing methods.
2 Background
Single-step retrosynthesis aims identify set of reactants S C M from the space of all molecules M that can be used to synthesize a given target molecule t E M. A single-step retrosynthesis model, often denoted as B, can be described as B(-) : t + {R, S,, c( [R,)] For given target the model proposes up to ls candidates. Each candidate consists of a reaction RG its corresponding set of reactants S; {ry}] |.a.l
I and associated cost el c(R)).. In this work, we directly use the single-step 10
retrosynthesis models from prior studies 12
Retrosynthetic planning seeks to discover complete reaction pathway for target molecule t E M. starting from a given set of available initial molecules (i.e., building blocks) I C M. A reaction pathway, denoted by is set of chemical reactions = (R1, R2..... Rx ). where each reaction R₁ transforms a set of reactants S; into product pt. Note that while a reaction may yield multiple products, we typically focus on single desired product. A valid pathway must satisfy the following

---

Retro-RI
Action Check
(d)
Response Organize
CALL Resparec:
Agent
Eavironment
Formar Check
Tool Check
Argament Check
Resetoo 2-1: Posoble reactions
CALL/ SELECT
Key/
gynthesine molersle 2-11 O+CIC002...
Action
Action Check
<hisk
follows
(umvvillable)
valinkz
2-1 Brole... (usavolable)
Stud
NNCC1-00 1-C CCCICCI-O (ivallabi))
Tool Use
("rame" "CAL)) "moolmtl""""""ooe""""""
Roadoon 3:1-3...
ctoof call
Prompt
SELECT Response:
Sclect noaction 2-1-3 systhoise
History Message
Molecule Query
Molecule Query
molecule OHCICU0 C2 <3nc21 the some
Melecule state Unavataalle motecules
Update
Quay
Melenil 9-11 fooehe CO0C) Mélecule 2oc2c00022
Response Organize
Available/ Unovailatle
Building Blocks
this Lo0ns
Reaction Candidate
Unavailable Molecule
Unavaillbll Molacule.
Unovallante
Assilable
Single-Suep
CALL
SELECT
Update
Update
Prompt
Update
Prompt
Figure 1: Overview of RETRO-R1 framework. The workflow of RETRO-R1 The agent interacts with the environment by using Actions. The environment automatically executes four types of operations: Action Check, Tool Use, Molecule Query, and Response Organize. The history message is organized in the form of multi-turn dialogue, containing all the interaction history between the agent and the environment. The interaction continues until an exit condition is met. b Action Check is composed of three parts: Format Check, Tool Check, and Argument Check. Format Check verifies the model's output correctly uses the designated <think> and <tool call> tags and ensures the content of <tool_ caal>> is in a valid JSON format. Tool Check validates whether the type of action is permissible in the current state, while Argument Check validates the parameters provided. Molecule Query searches within the building blocks for target molecule, returning result of either "available" or "unavailable". d Response Organize structures the text response according a specific format. If any of the other operations encounter error, this module will return the cause of the error. Otherwise, it will provide the appropriate result corresponding the type of action. e One molecular state transition consists of two interactions between the agent and the environment. Initially, the environment notifies the agent which molecules are currently "unavailable". The agent selects one of these molecules and uses the 'CALL' action to address it. In response, the environment invokes Single-Step Model to generate several reaction candidates. The agent then uses the 'SELECT' action to pick one of these reactions, prompting the environment to update the molecular state.
Every reactant r S; for any reaction R; E must either be a member of the initial building block set I or be the product py of another reaction R2 E T (where j A i); (2) Intermediate Connectivity: The product p; of any reaction R2 T must either be the final target molecule t or serve as a reactant in another reaction R₃ E T (where j * i); (3) Target Uniqueness: There must be one and only one reaction in whose product is the target molecule t;
conditions: (1) Reactant Availability:
(4) Chemical Feasibility: All reactions proposed in the pathway must be chemically viable.
3 Methodology
3.1 Overview
We introduce RETRO-RI novel L.LM-based agent for multi-step retrosynthetic planning. As shown in Fig. D the retrosynthetic planning process is formulated as a sequential decision-making
task within a structured search space. A molecular state M, {m,, m22...., mmn contains all the
3

---

unavailable molecules. The initial molecule node M1 = {f} contains only the target molecule. The goal molecule node M. Ø contains no molecules. One molecular state transition consists of two interactions between the agent and the environment.
The rest of this section is organized as follows. In Sec. 3.2 the workflow of RETRO-R1 is described. In Sec. 3.3 we detail the reinforcement learning algorithm to train the agent end-to-end. In Sec. Sec.[.4] we describe the iterative planning strategy that enables comprehensive exploration and searches for the shortest route. We implement the framework based on previous works verl (34] and Agent-RI 35
3.2 Agent Workflow
We propose a novel workflow that leverages a large language model (LLM) agent trained with reinforcement learning (RL) to navigate the retrosynthesis search space through multi-turn interactions with the environment and strategic tool use. The LLM acts as the policy ne (uls), mapping the current state representation to probability distribution over possible actions.
State The state provided the LLM agent is the tokens of the history message, which consists of the question, the agent's s previous actions, and the environment's responses. As shown in Fig. [T., the final message from the environment can contain one of the following: the molecules that currently unavailable, reaction candidates for selected molecule, or an error message caused by the last interaction.
Action a:
The agent takes action by generating texts with specific tags and tool call requests in JSON format, as shown in Fig. [p. Two actions designed:
CALL: Selects molecule from the current molecular state and uses the single-step model to propose potential reactions. This action is only valid if the last message from the environment enumerated the unavailable molecules.
SELECT:
Selects a reaction candidate to update the molecular state. This done by removing the reaction's 's product and adding its unavailable reactants to the state. This action is only valid if the last message provides a list of reaction candidates.
The environment drives the retrosynthetic planning process. It manages the molecular state, interacts with the agent, checks for termination conditions, and calculates the final reward. In its interactions with the agent, executes four operations: Action Check, Tool Use, Molecule Query, and Response Organize, as shown in Fig. I
Environment:
Action Check: Verifies whether the agent's action is valid. It is composed of three parts: Format Check, Tool Check, and Argument Check. Format Check verifies if the model's output correctly uses the designated <think> and <tool_ _a_lllll tags and ensures the content of <tool_calll is in a valid JSON format. Tool Check validates whether the type of action is permissible in the current state, while Argument Check validates the parameters provided.
Tool Use: For the CALL action, the single-step model is called for the selected molecule, and a set of reactions is proposed. The reactions with top-k probabilities form the reaction candidates. For the SELECT action, the molecular state is updated with operations, where the product is removed and the unavailable reactants are added.
Molecule Query: Searches within the building blocks for a target molecule. The result is either "available" or "unavailable".
Response Organize: Structures the text response according to a specific format. If any of the other operations encounter an error, the cause of the error is returned. Otherwise, it will provide the corresponding result. For the CALL action, the reaction candidates are listed. For the SELECT reaction, the unavailable molecules enumerated. Each item is assigned a unique identifier to facilitate selection by the agent.
Termination: A retrosynthetic planning process terminates successfully when the molecular state becomes an empty set. Conversely, the process terminates in failure if the molecular state is not empty by the time any of the following conditions are met: (1) the number of agget-environment interactions exceeds the maximum threshold; (2) the token count of the history message surpasses the I.LM's context limit; (3) the CALL action fails to generate any reaction candidates, indicating an unsynthesizable molecule.

---

To guide the agent to successfully and efficiently discover the shortest possible synthesis routes, we have empirically designed reward function based on prior work [6 31 and extensive experimentation. This function is composed of two components: (1) A success component that evaluates the outcome of the planning process: The agent receives a reward of +0.9 for successfully finding a synthesis path and penalty of -1.0 for failure; (2) An efficiency component for successful runs that penalizes excessively long searches: Since the number of agent-environmen interactions I in our design determines the upper bound of the reaction route length, minimizing / concurrently reduces both search time and route length. Therefore, for successful runs, if 40, a penalty of (40 - 1) 0.02 is applied. No penalty is incurred otherwise.
Reward:
3.3 Agentic Reinforcement Learning
We train the LLM policy (a(ass) with Proximal Policy Optimization (PPO) 30,, robust policy gradient algorithm that optimizes a clipped surrogate objective to enable stable updates. Both the policy network (actor) Te and the state value function (critic) (s(s) are initialized with the original LLM. The implementation PPO incorporates specific modifications tailored to our framework's multi-turn interactions with the environment and structured state representations.
Actor and Critic Updates with Environment Masking. The state s provided the LLM agent includes a sequence of tokens representing the question, the agent's 's actions, and the environment's s responses. To ensure that the agent's learning focuses solely on its own decision-making rather than predicting replicating environment outputs, we define token mask [(s;) where [(st) if s is an agent token and [s;;) otherwise. The loss function for the policy network is written as:
CLLP (0)
Egn BanD,anmatt (.o.aaa(((((
(I)
Σ ((s)) min (re(0)A,, clip (r:(0), c, +11++) c) A.)
E [(st)
where D is the training set, is a question sampled from D, E is the environment, ro) ma(ai(se))
the ratio of the probability under the current policy 0 to the probability under the policy used for data collection lold A, the advantage estimate at timestep f, and e small clipping parameter. This means the gradients are only computed with respect to the agent tokens, preventing the policy from being trained regenerate the environment's s information.
The loss function for the state value function is a squared error loss:
(Φ) [ss) (Vss)\()))))) -V, ) + [())) (s
which means the prediction target for environment tokens is set to zero. The value function is thus trained to estimate the value of agent tokens, without assigning value to the deterministic information provided by the environment.
Generalized Advantage Estimation with Unit Parameters. We utilize Generalized Advantage Estimation (GAE) K3 to compute the advantage estimates Å, used in the policy and value updates.
GAE typically uses discount factor 7 E [o, 1) and a weighting factor E [0, 1). our specific setting, to avoid the influence of environment tokens on the return accumulation we set both y = 1 and À = 1, and the GAE estimate simplifies to the Monte Carlo return (or TD(I) return). We further set the value of the terminal state (s()) as zero, and then the advantage estimate A, for a state s; is calculated based on the sum of undiscounted rewards from time step & to the end of the sequence T. minus the value function's 's estimate (sts)
A Σ R Vo(ss) k=t
33)
This configuration results in high-variance but unbiased estimates of the advantage, focusing the agent on maximizing the total undiscounted reward accumulated over an entire planning episode.
By incorporating environment masking and leveraging Monte Carlo returns via GAE(? = 1, = 1).
our PPO implementation is adapted to effectively train an LLM agent that solves the problem by interacting with the environment.
3.4 Iterative planning strategy
5

---

While a single sequential search can yield potential synthesis route, real- world chemical synthesis planning of- involves exploring multiple al- ternative pathways and consolidating knowledge gained from different ap- proaches. To emulate chemist's s com- prehensive exploration process, we design iterative planning strategy that builds a global synthesis graph across multiple sequential searches. A memory mechanism is implemented record all the intermediate infor- mation and avoid redundant computa- tions.
Round
Round 2
Synthesis Graph
Reorganize
Shuffle
R₂
R2
Rg
R2
R1
R3
R2
Shuffle
R4
Rs
Re
R
Rg
RS
Rg R4
The iterative planning process of RETRO-RI is the target molecule. A circle refers to a molecule, while square refers to a reaction. Multiple rounds of search are conducted to explore the chemical space, with each round fol- lowing the process detailed Fig. Te. The order of molecules or reactions is used in Response Organize (Fig. [d). When state is revisited, the order is shuffled to encourage explo- ration. Finally, the complete search history is reorganized into an AND-OR graph, where the shortest route is found.
Figure 2:
As illustrated in Fig. multiple rounds of search are conducted for the targeted molecule to explore the chemical space. Each state is recorded by the memory mechanism with a de- fault order. For example, the reaction candidates are ranked in descending order of the probabilities predicted by the single-step model. This order is shuffled if the state is revisited to encourage the agent to explore diverse selections. Finally, the complete search history is reorganized into an AND-OR graph. We implemented an algorithm search for the shortest route following the prior work 38
Experiments
4.1 Experimental Setup
This section describes the components of our experiments, including the dataset (building blocks, training and testing datasets), single-step models, baselines, and evaluation metrics.
Dataset. Following the common practice [2]. we use 23,081,633 commercially available molecules from eMolecules as the building blocks. For the training dataset, the single-step reactions come from USPTO reaction dataset [39]. Based on these reactions and building blocks, 299.202 multi-step synthesis routes are constructed in the prior work [12]. The length of the routes varies from 1 to 14. For baselines, all routes are used the training. For our method, only 11,366 routes with lengths longer than 4 are used, which shows the data efficiency of our method. For the testing datasets, we use Retro*-190, which are 190 challenging target molecules from [12], each with a reference route, and ChEMBL-1000 [1). which is diitt--dittiiuutin dataset used to test the generalizability of methods, consisting of 1000 molecules without reference routes.
Single-step Model. For fair comparison with previous work, our framework utilizes a template- based MLP as the single-step retrosynthesis model [4. It predicts probabilities over 381,302 USPTO reaction templates [39] 40) based on molecule Morgan fingerprints. We evaluate performance using four distinet weight versions (VI~V44 from previous methods, which were trained on different reaction datasets (USPTO or iteratively on routes found by other planners) [12 9 II I0] While versions V2~VV trained on automatically found routes often yield higher metrics, they risk proposing chemically unrealistic reactions [11]. For fair comparison and better chemical intuition, we adopt two-branch policy structure akin to 10:: VI proposes the top-k reaction candidates, and the others rescore and rerank these same candidates. This ensures all compared methods operate within the same candidate search space (k: 10), focusing on high-probability, VI-proposed candidates. Our RETRO-RI agent is trained using the VI weights but is compatible with any other weights during
hhun:s//dwwllodds.mooeeeees.cco

---

testing. Weights VI~V3 are downloaded from public codebases while V4 is retrained following instructions from the PDVN codebase - as is not publicly released.
Baseline. We compare our method against representative baselines on this task. We mark the weights of the single-step model in brackets for all baselines. Greedy Depth First Search (DFS) (V1) always prioritizes the reaction with the highest likelihood [12]. Retro* (V1) [1] is an A*-like algorithm where the past cost is calculated with the probability of the single-step model and the future cost is estimated with a neural value network. Retro*-0 (V1) [12 is its non-learning version where the future cost is set to O. Retro*+ (V2) and Retro*+-0 (V3) [ improve Retro and Retro*-0, respectively, with self- reff-impo single-step retrosynthetic models. EG-MCTS (V2) [[T] improves Retro* with
experience- -gidddd Monte Carlo tree search. PDVN (V4) [10] trains single-step model with an online training algorithm to improve Retro*-O.
Metric. The first metric is the @ass@1 Success Rate metric, inspired by the previous L.LM works 36 25.. This metric assesses a "no-regret" search strategy: when the single-step model proposes reactions at a given molecular state, the method must select only one, and the other 1 options at that node are permanently discarded. A Pass@ @ I search process is considered a failure under two conditions: (1) The route length exceeds the maximum limit of 30 (which sufficient for our datasets); (2) The single-step model fails to propose any reactions for a molecule, resulting in a search dead end. Additionally, following previous work [2] I0 we use the Success Rate within a limit of 500 single-step model calls (i.c., iteration limit) for overall feasibility. To evaluate route quality, we measure the Route Length and the number of Shorter Routes found compared to the reference routes. These two metrics are calculated based on the results after the methods ex exhaust the 500 iteration limit. Since some baselines are originally designed to terminate once one route is found, we modified their code to ensure a fair comparison.
4.2 Implementation Details
Our framework is built upon verl 34 and Agent-R1 R1 [35]. with the implementation publicly accessible at http:/gghhuboomoo/nnn/SAtAAAAuAu
We employ Drven2.57BBIInstuuttt 1M' [42 43
as the policy network for reinforcement learning to achieve superior performance on our long- sequence tasks without additional continued pretraining or supervised fine-tuning. Efficient rollouts conducted using vLLM[44] with sampling temperature of 1.0 and maximum sequence length of 32,768. For training, we apply a clipping ratio of 0.2 in the CCLIP objective and incorporate a KL-divergence penalty with coefficient of 0.001 into the token-level reward. Additionally, an entropy loss with a coefficient of 0.001 is utilized to prevent the model from generating overly chaotic outputs. Both the policy and value networks use Adam 45 the optimizer. The learning rates of the policy network and value function are 10-6 and 10-5, respectively. The rollout batch size is set to i28, while the mini-batch size for gradient updates is 64. During the initial 3 training steps, updates to the policy network are omitted to facilitate warm-up of the randomly initialized critic head.
4.3 Main Results
We evaluate the performance of our proposed RETRO-R1 1 against several baseline methods on two retrosynthetic planning testsets: retro*-190 is in-domain dataset from USPTO reaction dataset (39 and ChEMBL-1100 is an out-of-domain dataset from ChEMBL database [46]. Results are summarized in Tab. [ and Fig. RETRO-RI demonstrates strong performance both retro*-190
and ChEMBL-1000 datasets, achieving state-of-the-art or highly competitive results compared to various baselines.
RETRO-RI makes high-quality decisions. We evaluate the quality of decisions with the pass @ I success rate and success rate at small iteration limits (N < 100). An optimal policy should always make decisions to convert a molecule to precursors that are simpler or closer to the building blocks. Since the building blocks contain large amounts of molecules, it is possible to achieve a high success rate in one round without regret. We modify the codebase of all baselines to compute this score. The success rate at small iteration limits shows the efficiency of the methods. An optimal policy should find the route quickly without inefficient exploration. As shown in Tab. [] for the pass@1 success rate, RETRO-RI outperforms all baselines on both 1e0 (55.79%) and ChEMBL-1000 (73.30%) by
"happs//githuu.cco//// http://sww.....m.//.oo/oommmm/na hww.w//ww.....................tn http:://ggihhh..oo///aaaa/////// http:///edd__mroodd__ree 'nnpss//gttibb.ommDDXXussss
7

---

Table 1: Results on retro*-190 and ChEMBL-1000. Pass short for Pass@ Success Rate. RETRO-RI is tested in ten random runs, and the means and standard deviations are reported. All baselines are deterministic, meaning they produce the same results on repeated runs; therefore, standard deviations are not reported. Shorter routes refer to the number of molecules whose routes found at the iteration limit of 500 are shorter than the reference routes. This metric not applicable to ChEMBL-I000 1000 because it lacks reference routes. The best results are marked in bold, and the second- best results are marked with underlines.
Success Rate (%) at lteration Limit N
Method
Pass@ (%)
N 50
N 100
N = 200 N = 300
N = 400
N = 500
Shorter routes
Performance Retro*-190
Greedy DFS
19.47
19.47
19.47
19.47
19.47
19.47
19.47
8
retro*
20.53
38.95
50.00
68.95
74.74
77.89
79.47
64
retro+-0)
19.47
27.37
37.89
55.79
62.63
72.11
74.21
S7
retro*+
30.00
55.79
68.42
79.47
83.16
84,21
84.21
81
retro++00
25.26
47.89
61.05
76.84
82.11
85.79
86.32
79
EG-MCTS
46.84
58.42
64.74
69,47
75.26
78.42
81.05
68
PDVN
42.63
64.74
72.11
82.63
87.89
91.58
92.11
90
RETRO-RI
55.79
73.21+00.80
77.21+1447
82.58+
84.47
85.89+00557
86.95+00.52
87
Performance ChEMBL-1100
Greedy DFS
38.10
38.10
38.10
38.10
38.10
38.10
38.10
retro*
47.70
65.60
69.00
71.60
73.20
74.00
74.40
retro+-0
38,10
64.60
67.20
70.30
71.80
72.70
73.40
retro++
53.50
70.50
73.90
76.20
78.00
78.80
79.40
retro++00
49.00
69.40
73.60
75.60
76.70
77.60
78.50
EG-MCTS
59.70
71.20
74.20
76.90
78.30
79.00
79.30
PDVN
60.00
73.90
76.40
78.10
79.60
80.30
80.60
RETRO-RI
73.30
77.88
80,21+0.118
81.79
82.62
0.29
83.28
0.25
83.72++0.11
large margin. The success rate at N < 100 of RETRO-RI is also the highest, showcasing the high quality of the decisions made by RETRO-R1.
RETRO-RI shows strong abilities of retrosyn- thetic planning on the in-domain testset. We consider the success rate and shorter routes on Retro*-190. According to Tab. [ RETRO- RI achieves least top-2 performances across all metrics. Although PDVN has a higher Suc- cess Rate large iteration limits, number of shorter routes is very close to that of RETRO- RI indicating RETRO-PE RI strong ability in discovering high-quality synthetic routes. No- tably, RETRO-RI uses only a subset of the train- ing set with 1,366 items in total, while PDVN uses all 299,202 items, which is more prone to overfitting on the dataset.
Performance on ChEMBL-1000
Success rate(%) %)
75
Retra*
70-
Retro+-0
Retro*
R
PDVN
EG-MCTS
Retro-R1
55
Route length
RETRO-RI shows strong generalizability out-of-domain testset.. We report the results on the large, out-of-domain dataset ChEMBL- 1000 in Tab. []] Since no reference routes are provided this dataset, we report the curves of success rate with respect to route length in 3 RETRO-RI consistently outperforms baselines across all metrics, showing that it learns generalizable chemical knowledge. Notably, RETRO-RA can find hard routes with length > 20, while other baselines have no improvements when the route length gets very large.
Figure 3: The curves of Success Rate with respect to Route length on ChEMBL-1000. All methods exhaust the 500 iteration limit find the shortest routes. A point (x,, y) on method's curve indi- cates that the method achieves success rate of when the maximum route length is limited to a..
4.4 Ablation Study
We conduct ablation studies on our framework to investigate the contribution of reinforcement learning, context shuffle in planning, and the choice of the single-step model.

---

Table 2: Ablation Study on Retro*-190, @ I short for Pass@1 Success Rate. RETRO-SFT refers to the agent trained with supervised fine-tuning on the reference routes of the training set. RETRO-RI wos refers to planning without context shuffle. All methods are trained with the single- step model with weights V1 and tested with the weights in brackets. RETRO-RI (V1~V4) are tested in ten random runs, and the means and standard deviations are reported. The best results are marked in bold. The second-best results are marked with underlines.
Success Rate (%) Iteration Limit
Method
@ass 1 (%)
N - 50
N - 100
N - 200
N
300
N
400
N 500
RETRO-SFT (V1)
50.00
63.16
67.89
74.21
77.89
78.42
78.42
RETRO-RI 1 wos (V1)
50.00
61.74
63.16
63.16
63.16
63.16
63.16
RETRO-RI (VI)
50.00
6..63=+1.68
71.32
18.11
1.88
82.95=
84.32=1.22
RETRO-R R1 (V2)
52.63
68.53=1110
74.32
79.74:
.44
82.16
0,95
85.47=1.08
RETRO-RI (V3)
55,26
68.84
1..0
79.95+1.53
83.32.11.29
84.74=0.88
85.84-0.64
RETRO-R R1 1 (V4)
55.79
73.21 + 0.80
77,21 82.58 1.46
0.84
85.89 0.57
86.95 0.52
We use all items from the training set to construct an SFT dataset. In this dataset, reference routes are formatted into multi-turn dialogues as described in Sec. Sec. and Fig. with the text between the <think> and </think> tags set to empty. We use this dataset to fine-tune the (0wen..5-7B-Instruu - MM' model. The resulting model is designated as RETRO-SFT. " Other methods presented in Tab. 2 based on the same model trained using only reinforcement learning. LETRO-RI wos refers to planning without context shuffle.
As shown in Tab, 2 RETRO-RI 1 (V1) outperforms RETRO-SF (V1) in almost all metrics, showcasing the improvements brought by agentic reinforcement learning. Comparing RETRO-RU wos (V1) and RETRO-RI (V1), it is obvious that the success rate of RETRO-RI R1 I wos (V1) barely increases with respect to the iteration limit. In fact, it is hard to reach the N = 500 iteration limit because of the lack of diversity, and we have to stop the planning process if the synthesis graph has no changes for 5 rounds. The comparison of RETRO-RI (VI~V44 shows that any single-step model can serve as a plug-in for our framework without training. A strong single-step model can improve RETRO-R R1 's performance with no cost.
5 Related Work
5.1 Retrosynthetic planning
Retrosynthetic planning aims to search for a multi-step reaction pathway for a target molecule. Early computational approaches relied heavily expert systems encoded with predefined reaction rules and human-designed heuristics to guide the search [3. More recent methods leverage the power of machine learning improve the single-step models and search algorithms. 3N-MCTS ① combines Monte Carlo Tree Search (MCTS) with three networks and use rollout to estimate the score function of searching state. DFPN-E (47] combines depth-first proof- nomobr search with heuristic edge initialization. Retro* 12 proposes a neural-based A -likee algorithm with a value network predicting the synthetic cost of the molecule. R ( proposes self-improving procedure to train the single-step model. EG-MCTS [T] uses experience collected in planning to train a value network predicting the cost of the reaction. MEEA* * (48) incorporates the exploratory behavior of MCTS into A* by providing a look-ahead search. PDVN [10] constructs two separate value networks to predict the synthesizability and cost of molecules. RetroGraph
[8 and DreamRetroer (49 plan a group of target molecules together efficiently. In addition to these traditional methods, researchers have also utilized large language models to solve retrosynthesis tasks. BatGPT-Chen (50] a large language model tailored for single-step retrosynthesis prediction, trained with extensive instructional data from an expansive chemical database. Llamole [5]] integrates a base LLM with a graph encoder serve as the cost function, based on which the * search is conducted. [52] proposes a retrosynthetic planning agent based on large language models and knowledge graphs to explore reaction routes with reactions in literature.
5.2 Reinforcement Learning for LLMs
Reinforcement learning has emerged as a powerful paradigm for fine-tuning Large Language Models (L.LMs), moving beyond static supervised training to optimize model behavior based on feedback from humans environments. PPO (30) is one of the most influential algorithms due to its efficiency

---

and stability. ReMax [53 introduces techniques to reduce the variance of PPO. GRPO 54 uses group sampling strategies replace value models. REINFORCE+ [55] simplifies GRPO and enhances its training. DAPO [T further improves the performance with strategies such as Clip-Higher and
Overlong Reward Shaping. 32 introduces attention signals as the indicators of the reasoning pattern and proposes to dynamically identify and strengthen key reasoning steps in RL. For tool-integrated reasoning, ReTool [26] modifies PPO to help the policy LLM collaborate with a code sandbox.
6 Conclusion
In this work, we propose a novel framework for multi-step retrosynthetic planning that leverages an RL-trained LLM agent RETRO-R1 well-designed workflow, and an iterative planning strategy inspired by human chemists. the sequential decision process, the agent learns to select molecules and reactions to decompose target molecules into available precursors. Evaluated on retro*-190 and ChEMBL-1000, RETRO-RI I achieves a competitive planning success rate on the in-domain testset and state-of-the-arr on the out-of-domail testset, demonstrating strong generalizability. RETRO-R R1 outperforms all baselines on pass @ success rate and route length, showcasing the ability to make high-quality decisions and find better synthesis routes. These results underscore the potential of leveraging LLMs and RL within an agentic framework for complex scientific problems.
Limitations and Future Work. Despite the promising results, our RETRO-RI has several limitations. Firstly, the LLM cannot propose a reaction by itself and the performance inherently relies on the single-step retrosynthesis model. Secondly, although probabilities, we still cannot guarantee that these reactions are feasible in practice, and predicting reaction conditions is very difficult. Future work will focus on designing a strong agent or multi-agent system that can propose reactions and analyze them using computational chemistry tools. The reward function should integrate more explicit chemical knowledge, which is essential for developing more powerful and versatile AI chemists.
only focus on the reactions with high
Acknowledgments and Disclosure of Funding
This work is supported by the National Key R&D Program of China (2022ZD0160501), the Na- tional Natural Science Foundation of China (Grant No. 62376133), and the Beijing Nova Program (20240484682).
References
[1] Marwin HS Segler, Mike Preuss, and Mark Waller. Planning chemical syntheses with deep neural networks and symbolic ai. Nature, 555(7698):604--610 2018.
[2] James Law, Zsolt Zsoldos, Aniko Simon, Darryl Reid, Yang Liu, Sing Yoong Khew, Peter Johnson, Sarah Major, Robert A Wade, and Howard Y Ando. Route designer: retrosynthetic analysis tool utilizing automated retrosynthetic rule generation. Journal of chemical information and modeling, 49(3):593-60) 2009.
[3] Tomasz Klucznik, Barbara Mikulak-
Kotullak-Kluzznie Klucznik, Michael McCormack, Heather Lima, Sara Szymkué, Manishabrata Bhowmick, Karol Molga, Yubai Zhou, Lindsey Rickershauser, Ewa Gajewska, et al. Efficient syntheses of diverse, medicinally relevant targets planned by computer and executed in the laboratory. Chem, 4(3):522-532, 2018.
[4] Marwin HS Segler and Mark P Waller. Neural-symbolic machine learning for retrosynthesis and reaction prediction. Chemistry-A European Journal, 23(25):5966-5971, 2017.
[5] Kaipeng Zeng, Bo Yang, Xin Zhao, Yu Zhang, Fan Nie, Xiaokang Yang. Yaohui Jin, and Yanyan Xu. Ualign: pushing the limit of template-free retrosynthesis prediction with unsupervised smiles alignment. Journal of Cheminformatics, 16(1):80, 2024.
[6] Jingxin Dong Mingyi Zhao, Yuansheng Liu, Yansen Su, and Xiangxiang Zeng. Deep learning in retrosynthesis planning: datasets, models and tools. Briefings in Bioinformatics, 23(1) (bbbbbbb 2022.
10

---

[7] Zipeng Zhong, Jie Song. Zunlei Feng, Tiantao Liu, Lingxiang Jia, Shaolun Yao, Tingjun Hou, and Mingli Song. Recent advances in deep learning for retrosynthesis. Wiley Interdisciplinary Reviews: Computational Molecular Science, 14(1 (((1))ee66 1694, 2024.
[8] Guillaume Gricourt, Philippe Meyer, Thomas Duigou, and Jean-Loup Faulon. Artificial intelli- gence methods and models for bissyhhisstttesss: A scoping review. ACS Synthetic Biology, 13(8):2276-222 2294, 2024.
[9] Junsu Kim, Sungsoo Ahn, Hankook Lee, and Jinwoo Shin. Self-improved retrosynthetic planning. In International Conference on Machine Learning, pages 5486-5495. PMLR, 2021.
[10] Guoqing Liu, Di Xue, Shufang Xie, Yingce Xia, Austin Tripp, Krzysztof Maziarz, Marwin Segler, Tao Qin, Zongzhang Zhang, and Tie- Yan Liu. Retrosynthetic planning with dual value networks. In International Conference on Machine Learning, pages 22266-22276. PMLR, 2023.
[11] Siqi Hong, Hankz Hankui Zhuo, Kebing Jin, Guang Shao, and Zhanwen Zhou. Retrosynthetic planning with experience-guided monte carlo tree search. Communications Chemistry, 6(1):120, 2023.
[12] Binghong Chen, Chengtao Li, Hanjun Dai, and Le Song. Retro*: learning retrosynthetic planning with neural guided a* search. In International conference on machine learning, pages 1616. PMIR, 2020.
[13] Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiehin Zhang, et al. A comprehensive survey on long context language modeling. arXiv preprint arXiv:2503.10 17407 2025.
[14] Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, al. Towards large reasoning models: survey of reinforced reasoning with large language models. arXiv preprint 25fi::250100980 2025.
[15] Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. arXiv preprint arXiv:2404. 15846 2024.
[16] Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelli- gence, 6(5):525-535 2024.
[17] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Lc, Denny Zhou, et al. Chain-of-though prompting clicits reasoning in large language models. 2022.
Advances in neural information processing systems,
[18] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1-8, 2025.
[19] Jenish Maharjan, Anurag Garikipati, Navan Prect Singh, Leo Cyrus, Mayank Sharma, Madalina Ciobanu, Gina Barnes, Rahul Thapa, Qingqing Mao, and Ritankar Das. Openmedlm: prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models. Scientific Reports, 14(1):14156, 2024.
[20] Antoine Louis, Gijs Dijck, and Gerasimos Spanakis. Interpretable long-form legal question answering with retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22266-22275 2024.
[21] Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, et al. Llama-berry: Pairwise optimization for ol-like olympiad-level mathematical reasoning, arXiv preprint arXiv:2410.0288, 2024.
11

---

[22] Janice Ahn, Rishu Verma, Renze Lou. Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157, 2024.
[23] Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, et al. Internlm-math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.06332, 2024.
[24] Juyong Jiang. Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515, 2024.
[25] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming-the the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024.
[26] Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in Ilms. arXiv preprint arXiv:2504. 1536, 2025.
[27] Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. Large language models empowered agent-based modeling and simulation: A survey and perspectives. Humanities and Social Sciences Communications, 1 11(1):1-24, 2024.
[28] Onder Gurean. aungenede agent-based modelling for social simulations: Challenges and opportunities. arXiv preprint arXiv:2405.06770, 2024.
[29] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 204552200..05862, 2022.
[30] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
[31] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503. 25t.:25003.4476, 2025.
[32] Yang Li, Zhichen Dong, Yuhan Sun, Weixun Wang, Shaopan Xiong, Yijia Luo, Jiashun Liu, Han Lu, Jiamang Wang, Wenbo Su, al. Attention illuminates 11m reasoning: The preplan- and-anchor rhythm enables fine-grained policy optimization. arXiv preprint arXiv:25 10. 13554, 2025.
[33] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.
[34] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: flexible and efficient rihf framework. arXiv preprint arXiv: 19256, 2024.
[35] Jie Ouyang, Ruiran Yan, Yucong Luo, Mingyue Cheng, Qi Liu, Zirui Liu, Shuo Yu, and Daoyu Wang. Training powerful 11m agents with end-to-end reinforcement learning, 2025.
[36] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprinl arXiv:2501.12945 2025.
[37] Hung Le. Yue Wang, Akhilesh Deepak Gotmare. Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 2-21334-21328 2022.
[38] Shufang Xie, Rui Yan, Peng Han, Yingee Xia, Lijun Wu, Chenjuan Guo, Bin Yang, and Tao Qin. Retrograph: Retrosynthetic planning with graph search. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2120-2129, 2022.
12

---

[39] Daniel Mark Lowe. Extraction of chemical structures and reactions from the literature. PhD thesis, 2012.
[40] Connor W Coley, William H Green, and Klavs F Jensen. Rdchiral: An rdkit wrapper for handling stereochemistry in retrosynthetic template extraction and application. Journal of chemical information and modeling, 59(6):2529-23 25377, 2019.
[41] Samuel Genheden and Esben Bjerrum. Paroutes: towards framework for benchmarking retrosynthesis route predictions. Digital Discovery, 1(4):527-539, 2022.
[42] An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang. Jiandong Jiang. Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang. Kexin Yang. Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang. Yong Li, Zhiying Xu, and Zipeng Zhang. Qwen2.5-1 technical report. arXiv preprint arXiv:2501. 15383, 2025.
[43] Qwen Team. Qwen2.5-1m: Deploy your own qwen with context length up to Im tokens, January 2025.
[44] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[45] Diederik P Kingma. arXiv: 2014.
Adam: A method for stochastic optimization. arXiv preprint
[46] Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, al. Chembl: large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):D1100- D1107, 2012.
[47] Akihiro Kishimoto, Beat Buesser, Bei Chen, and Adi Botea. Depth-first proof-number search with heuristic edge cost and application to chemical synthesis planning. Advances in Neural Information Processing Systems, 32, 2019.
[48] Dengwei Zhao, Shikui Tu, and Lei Xu. Efficient retrosynthetic planning with mets exploration enhanced a* search. Communications Chemistry, 7(1):52, 2024.
[49] Xuefeng Zhang, Haowei Lin, Muhan Zhang, Yuan Zhou, and Jianzhu Ma. A data-driven group retrosynthesis planning model inspired by neurosymbolic programming. Nature Communica- tions, 16(1):192, 2025.
[50] Yifei Yang, Runhan Shi, Zuchao Li, Shu Jiang, Bao-Liang Lu, Yang Yang, and Hai Zhao. Batgpt- chem: A foundation large model for retrosynthesis prediction. arXiv preprint arXiv:2408. 10285, 2024.
[51] Gang Liu, Michael Sun, Wojeiech Matusik, Meng Jiang, and Jie Chen. Multimodal large language models for inverse molecular design with retrosynthetic planning. arXiv preprint arXiv:2410.042233 04223, 2024.
[52] Qinyu Ma. Yuhao Zhou, and Jianfeng Li. Automated retrosynthesis planning of macromolecules using large language models and knowledge graphs. Macromolecular Rapid Communications, page 2500065, 2025.
[53] Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi- Zhi-Quan Luo. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. arXiv preprint arXiv:2310. 10505, 2023.
[54] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 4rXiv:2402.0333 03300, 2024.
13

---

[55] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.03262, 2025.
[56] Yu Dong, Xiaodi Qiu, Neil Shaw, Yueyang Xu, Yuna Sun, Xuemei Li, Jun Li, and Zihe Rao. Molecular basis for the inhibition of (-hydrroyyccyl-app dehydratase hadab complex from mycobacterium tuberculosis by flavonoid inhibitors. Protein cell, 6(7):504-511 2015.
[57] Yanru Qu, Keyue Qiu, Yuxuan Song, Jingjing Gong, Jiawei Han, Mingyue Zheng, Hao Zhou, and Wei- Ying Ma. Molcraft: Structure- Stuettur-based drug design in continuous parameter space. arXiv preprint 240.::2404.114444 2024.
[58] Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research, 44(DI):D1202-DD DD211, 2016.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in abstract and Sec. accurately reflect the paper's contribu- tions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe the framework in detail in Sec. 3 and the implementation settings
in Sec. 2
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[Yes]
Justification:
All the code are available with the link Sec. We do not use new dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
ic for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //iipe.cc/pullic/guuii wwwpc//ggue////eeeeeoooc CoodeSubeiiieeeec ssi asionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details are specified in Sec. 4.1 and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Tab. and Tab. we report the standard deviation of our methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
17

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All details about compute resources specified in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //t::://neuiiseeeeeeeeeee https///c/.cccctttccciiieeeeeeee
Answer: [Yes]
Justification: This paper has no problem of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
18

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited all the original paper for the codebases and datasets we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, snperswwihcoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
19

---

Answer: [Yes]
Justification: We have written the documentation alongside the codebase in Github URL.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
20

---

Justification:
We describe the use of LLMs in Sec. 3 and Sec. 4
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to L.MM policy hateop///aaarrpppocc////f/eeeees for what should or should not be described.
21

---

A Two-branch policy network
we describe the two-branch policy network structure from the previous work PDVN to ensure all methods have the same search space and the probabilities of proposed reactions are high according weights V1 trained on real reactions. The structure is shown Fig. The reference single-step model, which is weights V1 in this work, proposes top-k candidate reactions for the given molecule mol1. The augmented single-step models, which are weights V2, V3, and V4 in this work, recompute probabilities for these reactions and rerank them. In this way, all single-step models provide the same candidates. The only difference the probabilities assigned them. This helps the retrosynthesis planner focus on reactions that are possible to happen in practice.
In Sec. 4.1
moll
Reference Single-step Model
P1
P2
>
P3
Reaction1
Reaction2
Reaction3
Augmented Single-step Model
P2'> P3
P1
Reaction2
Reaction3
Reaction1
Figure 4: The illustration of the two-branch policy network structure from the previous work PDVN. The reference single-step model, which is weights V1 in this work, proposes top-k candidate reactions for the given molecule moll. The augmented single-step models, which are weights V2, V3, and V4 in this work, recompute probabilities for these reactions and rerank them accordingly.
B Prompt and examples of interaction
In this section, give the prompts and examples of interaction between the agent and the environment. In Fig. 5 we give the system prompt and user prompt to synthesize molecule @ @J2CC=SS. In Fig. 6 give two
CIC@ H (orcecl)NICIC @ (2CCCC/((CCC)CCCCCCCC cases where the agent wrongly takes actions and the corresponding responses of the environment, In we give two cases where the agent correctly takes actions.
Fig. 7
C Curves in training
Fig. plots six key metrics recorded during training: (a) Pass@1 I Success Rate: This measures percentage of molecules training batch solved single round planning. As the training progresses, this metric steadily increases to 95%, which means our reinforcement learning framework makes full use the data efficiently. (b) Average Reward: This the average rule-based reward
(defined in Sec. 3.2 per training batch. As expected, this curve is highly correlated with the success
22

---

# Tools
You may call one or more functions to assist with the user query.
You provided with function signatures within <toole></tools> XML tags: <tools>
("name"; "CALL", "description": "Perform single step retrosynthesis for molecule and return several possible reactions to synthesize it. Note that the reactions may be incorrect and each reactant is marked as available' 'unavailable'. The unavailable molecules have to be synthesized further." ", "parameters": ("type": "object", "properties": molecule"; {"type": "string", "description": "The ID for the molecule to be synthesized. For example, 0-0 ."), "required": ["molecule"]}> ("name": "SELECT", "description": "Given several reactions to synthesize molecule, this tool select from them.", "parametera": ("type":
"object", "properties": ("reaction"; ("type": "string", "description": The ID for the selected reaction. For example, 0-0-0 ")), "required": [" reaction"]]]
</tools>
For each function call, return a jaon object with function name and argumenta within <tool_cal><<ttol,cca XML tags: <tool_call>
{"name": <function-name>, "arguments": <args targs-json-object>) </tool_call>
You are a professional organic chemist, skilled at finding pathways to synthesize novel molecules. Given target molecule, your task is find a pathway where the target ingredient molecules, and each step of the pathway is viable chemical reaction. The maximum number of reaction steps you can use is 30. You can use the tools provided to you to find the pathway. You can use the
eventually syathesized using
tool many times as you must first conduct reasoning inside <think>... </think>, Then use tool call <tool_call>.. </tool_call>>>>> < to call the tool.
Output format for tool call: <think
</think>
<tool_call>
</tool_call>
The target molecule is C[CeH] (clccccc1)KCCC()))(((()))))) [ccecceeDCCc)02G0000000000000000 S. Now start to search for the pathway.
Molecule state Unsolved molecules: Molecule 0-0: C[CиH] (cIcccc::))iiC(CC) ort.:cccccccctoooocccooooCoooooo Now, steps are used.
Please select the molecule with the highest synthesis difficulty from the unsolved molecules. Use the tool 'CALL" to synthesize it.
Figure 5: The system prompt (upper figure) and user prompt (bottom figure) to synthesize molecule CIC@ Cccclccccceeccccppccccoooooooocc @
23

---

ENV: <tool_ reoo_response>
Select reaction 9-1-1 1 to synthesize molecule N#C (
The state goes Molecule state 8: Unsolved molecules:
Molecule 8-1: CCOC(=0) [сеH] [cC=C[co0] (C#N) CD (CHN)
To reach this state, at least 7 reaction steps used. </tool_reat respons>?
Agent: <think>
</think>
<tool_call>
("name": "CALL", "arguments": {"molecule": "7-1"}}
</tool_call>
ENV: <tool_response>
Invalid arguments for tool 'CALL'; Parameter molecule has invalid value, should be 8-1 8-1 </tool_response>
ENV: <tool_rees reol_response>
Reaction state 9-1: Possible reactions to synthesize molecule 9-1: CC [CeH are as follows:
I
Reaction 9-1-1: (ctcccccccccccccccccCCCCCCCCCCCC 201=0 (unavailable) Reaction 9-1-2: CCDC(=0) [cea] [C000 unavailable)
(CC) ciccecc1)
rrooo_eespons>>>
Agent: <think
</think>
<tool_ "col__cal>>"namme". "SELECT", "arguments": ("areis""""""aaaa ("reaction"; "9-1-6"}3
ENV:
<tool_response"
Invalid tool call format. Please the format:
<tool_call>
("name": "tool_name", "arguments": (params_json)) </tool_call> </tool_response>
Two examples of failed interaction. The error messages from the environment are shown. In the upper figure, the agent takes the action CALL with an illegal molecule identifier. In the bottom figure, the agent misses the </tool call> tag and a newline character after call>.
Figure 6:
rate, the first term (success component) of the reward function is dominant, (c) Penalty Score:
This is the second term (efficiency component) of the reward, which penalizes the agent for failing to complete tasks within the interaction limit. The penalty is high in the early stages when the agent frequently fails or takes incorrect actions, but it decreases as the agent improves, eventually applying only hard data items. (d) Average Critic Value: This shows the average value predicted by the critic network across all tokens generated by the actor network in training batch. The value fluctuates significantly in the early stages but stabilizes as training progresses. This stabilization is crucial, as unstable critic can misguide the actor. As noted in Eq. 2 we set the prediction target for environment tokens zero to aid this stability. (e) Actor Entropy: This measures the uncertainty (entropy) of the actor network when generating the next tokens. The observed decrease in entropy indicates that the actor's policy converging from exploration to a more deterministic, optimal strategy. (f) Average Interaction Turns: This plots the average number of turns used to solve the tasks. The decreasing trend demonstrates that the agent learning to find solutions more efficiently training progresses.

---

ENV: stool_respones Select reaction 6-1-4 to synthesize molecule N#C [CC[[))]]])) 1CC=C (C#N) cC1 The state goes to:
Molecule state 7: Unsolved molecules:
Molecule 7-1: N*C[Cан] 1CC=C[000] (CHN)
Molecule 7-2: 0=S(C1)
To reach this state, at least 7 reaction steps are used. </tool_response>
Agent: <think>
The molecules with the highest syathesis difficulty are 7-1 and 7-2. We should use the tool 'CALL' to synthesize molecule 7-1, as it is likely the more challenging one.
</think>
<tool_call>
("name": "CALL", "arguments": {"molecule": "7-1"}}
</tool_call>
ENV: <tool_ mhollressonses
Reaction state 1-1: Possible reactions to syathesize molecule 1-1: c[CaH]
[CCCCC 201=0 are as follows:
Reaction 1-1-1: (= (C) (oc.cccccccc0000011110000000DDD0
C)C (unavailable)
Reaction 1-1-2: C[CeH] (ct.cccccccccoooo..e..cccoaaoaoc
0 (unavailable)
Reaction 1-1-3: C[CeeH] (CCC://cCC....ooaooooooooooooooo
(unavailable)
Reaction 1-1-4: c[caн] ottpaa/cccccccccccco////////oooc
S (unavailable)
Reaction 1-1-5: CN (C)) (C)) ( (available) + C[CeH] (cicceec1)
(unavailable)
Reaction 1-1-6: CC(C) (C)O (available) с[сен] (ciccccc1) (CCc)cccc))(((oo))](C00))000
CC [C[[Cо)] 2C1-0 (unavailable)
</tool_response>
Agent: <think>
Among the reactions, the possible ones are 1-1-5 1-1-6, where the reactants are either available or easy to synthesize. We should use the tool 'SELECT' to choose the best reaction.
</think>
<tool_call>
("name": "SELECT", "arguments": ("reaction": "1-1-5"}} </tool_call>
Figure 7: Two examples of the successful actions taken by the agent. In the upper figure, the agent selects a more challenging molecule to synthesize first. In the bottom figure, the agent selects reaction whose reactants are available easy synthesize.
25

---

Fasso on trainingset
Reward traring
Penalty Score surinn
.925
PAN0 sucenss
Peralty score
-0.50
6.5-
10
25
1.25
150
275
100
123
200
200
125
175
Training steps
Training steps
Training steps
(a)
(c)
Ontic value
Entrapy training
Turns training
20
6o.4
25
0.5
0.2-
155
150
25
Training steps
Traming staps
Training steps
(d)
(c)
(f)
Figure 8: The curves of metrics with respect the training steps. (a) The pass @ success rate on the training batch. (b) The average reward described in Sec. 3.2 on the training batch. (c) The average
penalty score on the training batch. (d) The average value of all generated tokens predicted by the critic network on the training batch. (e) The entropy of the actor network on the training batch. (f)
The average interaction turns between the agent and the environment in the training batch.
D Discussion on computational cost
We use 32 A800 80G GPU to train RETRO-R1 The model converges after 190 training steps, requiring approximately 70 hours. When planning with 500 iteration limit, it takes around 20 minutes per molecule on a single A800 80G GPU. While the LLM-based approach exhibits higher computational demands compared traditional methods, this expenditure is justified given the inherent complexity and critical importance retrosynthesis planning. Traditional methods, often constrained by smaller models, are incapable of effectively leveraging the extensive corpus of chemical literature. Their reliance on limited, task-specific datasets precludes the acquisition of comprehensive understanding of foundational chemical principles. Consequently, progress in this domain has stagnated; while existing methods may report gains in specific metrics, they fundamentally lack generalization capacity. This work demonstrates that Large Language Models, optimized via reinforcement learning fine-tuning on small amounts of data, can rapidly surpass traditional approaches, emerging as domain-specifie experts with robust generalization capabilities. Furthermore, the inherent capacity of LLMs to articulate their reasoning process in natural language (i.c., "chain of thought") significantly enhances the interpretability, rendering the outputs readily analyzable by chemists. We believe this work represents significant step forward. In the future, LLM-based agents are poised to solve an increasing number of complex chemical problems, driving comprehensive advancements across the entire field of chemistry.
E
More experiment results
E.1
Performance comparison across datasets
To further compare the generalizability of methods, we visualize the success rate (at an iteration limit of 500) across two testing sets Fig. [9] the in-domain Retro*-190 and the out-oo-domain ChEMBL-1000, The red area highlights the performance gap. The baseline Retro*-0 only uses the single-step model with V1 weights, which is trained on real reaction dataset, and planning algorithm is training-free. Consequently. it exhibits comparable performance on both testing sets. Retro*+-0 and PDVN improve Retro*-0 by only augmenting the single-step model (to weights and V4, respectively) on the training set. However, they do not evaluate the cost of molecules or
26

---

reactions to alleviate the bias in the single-step model, and thus are significantly overfitted. The performance of Retro*, Retro*+, and EG-MCTS is much lower than RETRO-R1 suggesting they acquire limited chemical knowledge. RETRO-RE R1 however, achieves both high overall performance and small generalization gap, indicating that learns transferable chemical knowledge.
Performance Comparison Across Datasets
95
Retro*- 190
92.10
ChEMBL-1000
Success Rate at Iteration Limit 500
86.95
86.32
84.21
81.05
83.72
80
79.47
80.60
79.40
79.30
78.50
75-
74.21
74.40
73.40
70
65
Retro* Retro*-0
Retro*+
Retro*+-0 Methods
EG-MCTS
PDVN
Retro-RA
Figure 9: The performance comparison of all methods on Retro and ChEMBL-1000.. Retro*. 190 is in-domain dataset while ChEMBL-1000 is out-of uut-of-domain dataset. RETRO-RI achieves both high overall performance and small generalization gap on two testing sets.
F,2 Route diversity
In practice, chemists select the most suitable synthetic route from multiple candidates proposed by retrosynthetic planning method. Route diversity therefore crucial metric for evaluating method performance. To assess the route diversity of RETRO-E R1 we introduce two metrics: the Number of Routes and the Number of Initial Molecules. Two routes considered distinct if they differ by single molecule or reaction. The second metric quantifies the size of the unique set initial molecules aggregated from all feasible routes. This metric measures the algorithm's ability to explore a broad chemical space, rather than being confined to a limited set of precursors. A wider selection of starting materials highly beneficial for chemists. Using the iterative planning strategy (Sec. 3.4). RETRO-RI constructs large-scale synthesis graph for each target molecule at the 500-iteration limit. We extracted all feasible routes from these graphs for the 1et dataset and calculated both metrics. Tab. 3 and Tab. present the distributions of these metrics. The results show that RETRO-R 1 finds over 100 distinet routes for most target molecules, involving more than 100 unique initial molecules, thus demonstrating the high route diversity of our approach.
Table 3: Distribution of the Number of Routes on Retro*- O means no routes are found. The distribution refers the number of target molecules the dataset.
Number of Routes
1~10
11~1000
101~1000
1001~10000
10001~100000
100000
Distribution
26
11
26
38
23
12
27

---

Table 4: Distribution of the Number of Initial Molecules on Retro*-190 means no routes are found. The distribution refers to the number of target molecules in the dataset.
Number of Initial Molecules
1~10
11~1000
101~~1000
1001~~10000
>1000
Distribution
26 14
25
49
50
26
E.3
Case study
The main purpose of this paper is to accelerate the process of drug discovery. To test whether our model can find the synthesis route for a novel molecule, we select protein target and generate novel molecule that can bind to it. Specifically, the selected protein is related to the tuberculosis (TB): 3R)-hydroxyycyl-ACC dehydratase HadAB hetero-dime from Mycobacterium tuberculosis (PDB ID: 4RLU). Dehydration is one of the key steps in the biosynthesis of mycolic acids and is vital to the growth Mycobacterium tuberculosis (Mtb). Consequently, stalling dehydration cures tuberculosis (TB) [56]. The molecular generation method we use is the mate-of-thee-rt structure-based drug design (SBDD) model MoICRAFT [57]. From the generated molecules, we select one that not recorded in the PubChem database [58] and has strong docking affinity with the target protein. The SMILES is and the vina score -10.829 kcal/mol. The docking
pose is shown in Fig. 10
Figure 10: The docking pose of (BR--oydroxyacyl-ACPP dehydratase HadAB hetero-dimer from Mycobacterium tuberculosis (PDB ID: 4RLU) and the generated molecule O=CIC-CCC2nnieneeed( c5ccenc5)ccc4ccnn22.
The synthesis route found by RETRO-RI is shown in Fig. The reaction ! Suzuki Coupling Reaction. The reaction 2 is an Oxidative Dehydrogenation Reaction. The reaction 3) is a DDQ- mediated Dehydrogenative Oxidation Reaction. The reaction ④ Quinoxaline Synthesis. All the reactions are very likely to happen in practice. The building blocks are marked with green rectangles.
28

---

OB(O)cloccnc
Brelec22efe J 1nnce
NECI==CCENNCOOXXCCCCCOCC
NelencZedBBeeeeeeN
O-CCCCCC0--
Figure 11: The synthesis route of molecule O=CIC= (cCC=CCCCeeeccccccsccceeeeeeeeee found by RETRO-RI The building blocks are marked with green rectangles.
29


Reviews:
[{"review_id": "LAOZ3tSYgR", "reviewer": "Reviewer_BV3y", "summary": "This paper develops and evaluates an approach to RL training for LLM-guided retrosynthesis. The approach involves a loop in which the agent has a set of molecules it needs to synthesize (at first this is just the single target molecule), it chooses one of the molecules to synthesize, and it calls an off the shelf single-step retrosynthesis tool to propose possible single-step reactions for synthesizing the chosen molecule. The LLM then selects one of these reaction pathways, at which point the chosen molecule is removed from the set of unresolved molecules to synthesize and any reactants are added to the set (unless they belong to the set of basic initially-available reactants). The LLM can't backtrack, but it does do a series of these linear searches and whenever the same choice is made from the same intermediate state, the old cached tool call results are reused. The series of partly-overlapping linear searches runs are combined to form a synthesis graph – essentially showing the part of the retrosynthesis search tree that was explored by all the runs. The shortest path from the initial target molecule state to the state where there are no remaining intermediate molecules to be synthesized is then extracted. The paper evaluates against a number of deep learning baselines as well as ablations that just perform SFT instead of RL, and achieves a decent boost in performance, especially on an out-of-domain test set.", "questions": "- See questions in “Weaknesses”\n- How does your work relate to the 3 pieces of related work mentioned above?\n- I was still a little uncertain about what Figure 4 is showing, even after reading the text and the caption, but I think I've figured it out. Is it that if you imposed the route length limit given by the x axis, you would get the success rate on the y axis?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall I believe this is an interesting paper with a novel contribution to an important area. The results of the method are fairly strong, and while I’ve listed a number of weaknesses none are particularly major. I would support this work being published in NeurIPS. \n\nStrengths\n- The core idea – RL based training for LLM based retrosynthesis – is to my knowledge novel, and they've executed it well. The paper is overall clearly written and understandable.\n- The adaptation of RL for LLMs operating on multi round tool calls has also been done in contemporaneous work but I don't believe that interferes with the novelty claims here, so this is also a nice contribution (see Weaknesses for note about citation).\n- The results are fairly strong, and the evaluation is thorough enough that I do feel convinced by them. The proposed method achieves higher pass @ 1 than all the other methods. When scaling to higher iterations, it performs a little worse than PDVN, but despite this it still finds more synthesis pathways that are shorter than the reference solution than PDVN does. And when moving to out of distribution data, they also outperform PDVN – this makes a fair bit of sense to me, I would expect pretrained LLMs to be a bit more robust to distribution shifts than other deep learning methods.\n- The ablations also seem appropriate. Ablating the use of RL away in particular was certainly an important setting and shows that RL is key to their good performance.\n\nWeaknesses\n- I was surprised that I didn't see any LLM-based retrosynthesis work cited in the related work. On looking around a bit I found for example \"Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning\" (Liu et al 2024) and possibly \"BatGPT-Chem: A Foundation Large Model For Retrosynthesis Prediction\" (Yang 2024). I'm not familiar enough with the area to say if these are the right citations, but certainly I'd expect some LLM-based retrosynthesis in the related work.\n- There's been some other work that has done LLM based multi-round tool-calls with RL, which I believe it is recent enough that it just counts as contemporaneous work that should be cited (per NeurIPS 2025 call for papers) but doesn't interfere with the claims of this work. In particular \"ReTool: Reinforcement Learning for Strategic Tool Use in LLMs\" (April 15 2025), which appears to also similarly mask tool outputs (section 2.3.2).\n- The graph-based shortest path approach is a reasonable algorithm, but the language around it feels a little excessively flowery in the paper, such as the line \"effectively mimicking a chemist’s comprehensive exploration process\" (in the 3rd contribution from the intro), as well as in section 3.4. Additionally, my understanding (feel free to correct me of course) is that Retro* uses a shortest path algorithm as well (section 3.5 of the Retro* paper) to extract a solution from their search graph as well. It’s listed in the contributions of this paper (Retro-R1), which I think is fine, but I would just view it as a smaller contribution than the others.\n- The approach doesn't seem to use the AND-OR tree of Retro* (among other work) — why not? Instead each run of search here produces a linear chain where each node has a sets of molecules rather than a single molecule. In the AND-OR tree each chosen reaction produces an AND of the different reactant molecules that are needed, which can each have their own retrosynthesis search subtrees. That seems like it would allow for more sharing than this setup, and the benefit of not doing that is not clear to me. In the current setup I assume if one branch of search had {A, B} as a set of target molecules and another had {B,C} these couldn't be merged into the same node, whereas in the AND-OR tree each reactant gets its own node so both B nodes could be the same. This would then allow for more possible paths when running the shortest path algorithm. Correct me if I'm misunderstanding though!\n- There are no repetitions / error bars on the baseline methods in table 1. This isn't a huge deal to me but would have been nice.\n\n\nquick fixes and suggestions:\n- Figure 2 the font size and indents could be a bit smaller, because the amount of text wrapping that's happening makes it difficult to read.\n- Line 323 it'd be helpful to say what \"wos\" means in the main text too, instead of just in the figure caption.\n- While overall I didn't have issues with clarity/readability, the very first paragraph could use some revision – some suggestions are: removing the comma in the first sentence; the second sentence is too short; the first sentence is a little confusing as it doesn't evoke how retrosynthesis is about working backwards; theres a typo \"highe-order\". \n- Figure 1 is a bit hard to read. It's difficult to know where to start in the figure (eventually I realized the start is at \"Target mol1\"). Visually making this clearer could help. The label \"Training\" with a dotted red line box is a bit confusing, as doesn't training also include things outside the box too? It seems like the red box is actually meant to single out one round (Round 0).\n- While the citation that's present is helpful, lines 216-218 should more explicitly state that the multistep synthesis routes are from the prior work that's cited, rather than saying the routes \"are constructed\".", "was_revised": true, "final_justification": "My issues were resolved in the rebuttal – particularly those around citing related work which I found most important. While I agreed with issues raised by other reviewers, reading their rebuttals and discussions largely resolved my concern with those as well.", "rebuttal": "**Question1:** LLM-based retrosynthesis works are not cited in the related work. \"Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning\" (Liu et al 2024) and \"BatGPT-Chem: A Foundation Large Model For Retrosynthesis Prediction\" (Yang 2024).\n>Response: We appreciate the reviewer for pointing out these great works. Recent vertical chemical Large Language Models (LLMs) are primarily trained using Supervised Fine-Tuning (SFT) and are typically evaluated across various tasks, often benchmarked against general LLMs. However, they generally do not surpass specialized methods in performance.\n\n>Regarding Liu et al. (2024), this work explores both single-step and multi-step retrosynthesis. For multi-step retrosynthesis, the LLM is primarily used as a scoring function integrated into an A* search algorithm. The original paper does not include comparisons with specialized models, and crucially, the code for their synthesis planning is not open-sourced, making direct comparison difficult.\n\n>Yang (2024) focuses solely on single-step retrosynthesis. While their model could potentially serve as a replaceable component within our framework, its performance has not been shown to exceed that of existing specialized retrosynthesis models.\n\n>We agree that these and other relevant LLM-based works should be cited to provide a comprehensive overview of the field. We will revise our manuscript to include these references and discuss their relevance within the related work section.\n\n**Question2:** Other works that have done LLM based multi-round tool-calls with RL are not cited. In particular \"ReTool: Reinforcement Learning for Strategic Tool Use in LLMs\" (April 15 2025), which appears to also similarly mask tool outputs.\n>Resonse: We appreciate the reviewer for highlighting the relevant work \"ReTool: Reinforcement Learning for Strategic Tool Use in LLMs\" (April 15, 2025). This is indeed an excellent paper, and it demonstrates the significant potential of multi-round tool-call reinforcement learning (RL) particularly for solving mathematical problems by equipping large language models with code execution tools. We will certainly cite it in our revised manuscript.\n\n>We've noted the similarities in the rollout process design between ReTool and our own work, where models interact with the environment using specific tags. ReTool also employs an Interpreter Feedback Mask to prevent interpreter tokens from contributing to the loss calculation, which aligns with our approach to handling tool outputs. However, ReTool doesn't explicitly introduce new RL algorithms; for example, details regarding the calculation of Generalized Advantage Estimation (GAE) or the specific design of the value function loss function aren't thoroughly discussed.\n\n**Question3:** The graph-based shortest path approach is a reasonable algorithm, but the language around it feels a little excessively flowery in the paper (in the 3rd contribution from the intro, as well as in section 3.4). Retro* uses a shortest path algorithm to extract a solution from their search graph, which is a smaller contribution than the others. \n>Response: Thank you very much for giving this point. The primary purpose of the 3rd contribution is to design a memory mechanism that reuses existing information to fully utilize the budget for constructing the synthesis graph. Subsequently, a breadth-first search is employed within this synthesis graph to identify the shortest path.\n\n>We observed that human experts, when planning synthesis routes, also analyze existing exploration history to draw new paths in the synthesis graph. This behavior is closely mirrored by our design, hence the description \"mimicking a chemist’s comprehensive exploration process.\" We believe this represents a crucial step in imbuing LLM agents with a chemist's mindset, which is vital for creating truly autonomous chemist agents.\n\n**Question4:** The approach doesn't seem to use the AND-OR tree of Retro* (among other work). In the current setup if one branch of search had {A, B} as a set of target molecules and another had {B,C} these couldn't be merged into the same node, whereas in the AND-OR tree each reactant gets its own node so both B nodes could be the same. This would then allow for more possible paths when running the shortest path algorithm.\n\n>Response: Thank you for this valuable suggestion. We did explore using an AND-OR tree in our early attempts. However, to enable the large language model (LLM) to understand the current synthetic state, we found it necessary to convert the AND-OR tree into natural language. The challenge arose when the number of nodes became substantial; clearly describing all relationships between these nodes resulted in an extremely long context, significantly impacting the LLM's performance. This is why we opted for the more concise super-node tree. \n\n>Our current design incorporates the spirit of the AND-OR tree. For example, in your provided case of nodes {A, B} and {B, C}, the environment internally stores molecules A, B, and C as individual entities. The memory mechanisms help to avoid re-invoking the single-step model during the repeated processes. We haven't fully leveraged this granular information beacuse the LLM is unaware of it.\n\n>Your suggestion for improving the efficiency of our search algorithm by more explicitly incorporating the benefits of the AND-OR tree structure is a very important direction for our future work. We appreciate you highlighting this potential enhancement.\n\n**Question5:** There are no repetitions / error bars on the baseline methods in table 1.\n\n>Response: Thank you for pointing out the problem of the absence of repetitions or error bars for the baseline methods in Table 1.\n\n>The reason for this is that the baseline methods employ deterministic search algorithms. Once the model weights are fixed, repeated runs will yield identical results. Since we primarily utilized the models open-sourced by the original baseline papers, and did not retrain them multiple times ourselves, we are unable to provide variance for these baselines.\n\n**Question6:** I was still a little uncertain about what Figure 4 is showing, even after reading the text and the caption, but I think I've figured it out. Is it that if you imposed the route length limit given by the x axis, you would get the success rate on the y axis?\n\n>Response: Your understanding is absolutely correct. Figure 4 aims to demonstrate that under the same route length constraints, our model achieves a higher success rate. Furthermore, our model possesses the capability to find very long routes for challenging molecules. In contrast, baseline methods tend to explore broadly during their search, which limits their ability to delve into deep routes.\n\n**Question7:** Quick fixes and suggestions: \nFigure 2 the font size and indents could be a bit smaller.\nLine 323 it'd be helpful to say what \"wos\" means in the main text.\nThe very first paragraph could use some revision.\nFigure 1 is a bit hard to read.\nLines 216-218 should more explicitly state that the multistep synthesis routes are from the prior work that's cited.\n\n>Response: Thank you very much for these actionable suggestions. We agree that all the points you've raised are important, and we will address them in the revised version of the manuscript:\n\n>Figure 1: We will redraw it to be clearer. Your understanding is absolutely correct. The dotted red line box indicates one trajectory in reinforcement learning training contains only one round.\n\n>Figure 2: We will optimize the font size and indents in Figure 2 for better readability. \n\n>First Paragraph: We will revise the introductory paragraph to improve its clarity and precision.\n\n>Lines 216-218: We will make it more explicit that the multi-step synthesis routes are from the cited prior work.\nWe greatly appreciate these valuable comments, as they are crucial for enhancing the quality of our paper.\n\n>Line 323 \"wos\": We will explicitly define \"wos\", without shuffle, in the main text."}, {"review_id": "CvrjFIo0Oh", "reviewer": "Reviewer_aDxU", "summary": "The authors use reinforcement learning to fine-tune an LLM agent to perform multi-step retrosynthesis planning.", "questions": "In addition to addressing the above concerns:\n- Can the authors account for the discrepancy between their reported results on PDVN and the results reported in the original PDVN paper? The original paper has higher values for the iterations shown.\n- Can the authors perform additional evaluation / case studies / demonstrations for the benefit of using LLM in this use case? Are there any interpretable reasoning behaviors that are learned from the training? Is the model better at remembering how to break down a molecule that it has seen before (and perhaps reaching such molecules)?\n- Can the authors compare to additional recent multi-step retrosynthesis methods? E.g. [1-3]\n\n[1] https://www.nature.com/articles/s41467-024-55374-9\n[2] RetroGraph https://arxiv.org/abs/2206.11477\n[3] MEEA* https://www.nature.com/articles/s42004-024-01133-2", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Novel application of RL+LLM for the retrosynthesis task\n- The authors demonstrate strong empirical results with the LLM-based agent\n- The authors for the most part present a clear and fair evaluation compared to baseline methods, utilizing the same single-step retrosynthesis model\n\nWeaknesses\n- It's not clear how necessary using an LLM is when the same single-step model is used--the results are not obviously better than PDVN, an RL-based method that doesn't use LLMs\n- Additional evaluations beyond pass@1 and success rate at N are not presented. The number of routes found under a budget, for instance, is a very useful (arguably even more useful) evaluation in retrosynthesis.\n- The comparisons are made by iterations of the single-step model, but the computational overhead of using the LLM is not presented. This is important for real-world deployment of CASP systems.", "was_revised": true, "final_justification": "I do not believe that this work provides a very significant advancement in computer-aided retrosynthesis for domain scientists. However, the empirical results still hold merit, and the methodology is novel and one that seems important to try. As a result, I will retain my score of 4.", "rebuttal": "**Question1:** It's not clear how necessary using an LLM is when the same single-step model is used.\n>Response: We appreciate this important question. Our core objective is to build a chemist agent capable of exhibiting chemist-like reasoning and solving complex chemical problems. Such an LLM can explicitly reason and deliberate on each decision, significantly enhancing both interpretability and reliability.\n\n>RL is crucial for improving the LLM's reasoning capabilities. It is  highly data-efficient; we achieved competitive performance using only 11,366 data items, compared to 299,202 used by baselines. Furthermore, LLMs possess strong generalization abilities, allowing their capabilities to be easily transferred to out-of-domain datasets, thereby enabling broader applicability.\n\n>In this paper, we tackled the multi-step retrosynthesis task to demonstrate that an LLM, when trained with RL, can surpass specially designed small models and algorithms. Using the same single-step model was primarily for fair comparison with the baselines. However, unlike the baselines, the single-step model is a replaceable component in our system. Recent works such as naturelm and chemma have also shown that LLMs themselves can function as powerful single-step models. \n\n\n**Question2:** Additional evaluations beyond pass@1 and success rate at N are not presented. The number of routes is a very useful evaluation.\n>Response: Thank you for this excellent point. We focused on pass@1 and success rate at N metrics primarily to align with previous work and facilitate fair comparisons. All baselines, such as Retro*, EG-MCTS, and PDVN, are designed to find only a single route, and as deterministic search algorithms, repeated runs yield identical routes, thus precluding metrics related to the number of routes found.\n\n>However, we agree that the ability to find multiple diverse routes under a given budget is highly valuable in retrosynthesis. In Section 3.4 of our paper, we describe our method for constructing the synthesis graph. When the search budget is set to 500, this synthesis graph can contain up to 500 nodes. By traversing this graph, we can identify a large number of potential synthesis routes for chemists to choose from.\n\n>To demonstrate this capability, we conducted an experiment on the Retro*-190 dataset with an iteration limit of 500. We saved the complete synthesis graph and then filtered out all synthesis routes with a length not exceeding 30. The distribution of the number of routes found for all molecules is as follows (190 molecules in total):\n\n|Number of routes|Not Found|0-10|10-100|100-1000|1000-10000|10000-100000|>100000|\n|-|-|-|-|-|-|-|-|\n|Number of Molecules|26|11|26|38|54|23|12|\n>To further illustrate the diversity of these routes, we extracted the sets of starting materials for each synthesis route and counted the number of unique sets. The distribution is as follows:\n\n|Number of unique sets|Not Found|0-10|10-100|100-1000|1000-10000|>10000|\n|-|-|-|-|-|-|-|\n|Number of Molecules|26|14|25|49|50|26|\n>As these distributions show, our Retro-R1 model can indeed find a diverse array of routes involving a large number of different starting materials. This provides chemists with a sufficiently large space for screening and experimentation, which we believe is a significant advantage in real-world applications.\n\n**Question3:** The computational overhead of using the LLM is not presented.\n>Response: We appreciate you raising this important point. We acknowledge that the inference time for LLM-based methods is currently slower than that of smaller, specialized models. However, the core advantage of using a large model lies in its ability to provide an explicit reasoning process, leading to interpretable and reliable results that can directly assist chemists in decision-making.\n\n>In the real-world context of molecular synthesis, validating a synthetic route through wet-lab experiments often takes days, or even weeks. Compared to this significant experimental time, the computational overhead of LLM inference is relatively very small. Therefore, focusing on increasing the success rate of the planning process, even with a slightly longer computational time, offers immense value by significantly reducing costly and time-consuming experimental failures.\n\n**Question4:** Can the authors account for the discrepancy between their reported results on PDVN and the results reported in the original PDVN paper?\n>Response: In Section 4.1 \"Single-step Model\", we describe four versions of single-step model weights (V1-V4). V1 was trained on real chemical reactions, while V2-V4 were trained on reactions found through a search process, which don't have experimental validation to guarantee their real-world feasibility.\n\n>We observed that some high-probability chemical reactions proposed by V4 have extremely low probabilities in V1. This can lead to routes that are not viable. To circumvent this issue and ensure a fair comparison across all methods, we chose to use V1 to propose k=10 candidate reactions. (In the original PDVN paper, k=50, which essentially included almost all possible reactions and provided no meaningful constraint.) Then we used V2-V4 to re-calculate and normalize the probabilities for these 10 reactions.\n\n**Question5:** Can the authors perform additional demonstrations for the benefit of using LLM in this use case?\n>Response: We appreciate the opportunity to further elaborate on the benefits of using an LLM in this context and to demonstrate its interpretable reasoning behaviors. As discussed in our response to Question 1, our primary goal in using a large model is to build a chemist agent that exhibits chemist-like reasoning and can tackle complex chemical problems.\nTo illustrate these capabilities, we show an example of the LLM thinking.\n\n>The LLM receives five candidate reactions from the environment, and then analyzes and compares these reactions one by one. This analysis reveals interpretable reasoning:\n\n>Molecular Complexity: For instance, it identifies that the precursor in \"reaction 0-0-0\" is slightly simpler than the target molecule, while the precursor in \"0-0-4\" is not simpler. When visualized, \"reaction 0-0-0\" shows the precursor having an opened ring structure compared to the target, validating the LLM's judgment.\n\n>Reaction Plausibility/Optimality: For example, it deems the reaction in \"0-0-1\" as \"not optimal.\" This suggests an ability to evaluate the quality or efficiency of a proposed reaction.\n\n>These examples clearly indicate that the LLM indeed comprehends molecules and chemical reactions. This stands in stark contrast to baseline methods, where neural networks directly predict a numerical value for synthetic difficulty. Such values are often heavily dependent on the training dataset and lack the inherent chemical understanding and generalization capability that our LLM exhibits.\n\n>Environment:\n```\n<tool_response>\nReaction state 0-0: Possible reactions to synthesize molecule 0-0: C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=S are as follows: \nReaction 0-0-0: C=CC[C@@H]1C(=S)N([C@H](C)c2ccccc2)C[C@@]1(C=C)C(=O)OC(C)(C)C (unavailable)\nReaction 0-0-1: COc1ccc(P2(=S)SP(=S)(c3ccc(OC)cc3)S2)cc1 (available) + C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=O (unavailable)\nReaction 0-0-2: C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=O (unavailable) + S=P12SP3(=S)SP(=S)(S1)SP(=S)(S2)S3 (available)\nReaction 0-0-3: C[C@H](c1ccccc1)N1C(=O)[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=S (unavailable)\nReaction 0-0-4: C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C#CC[C@@H]2C1=S (unavailable)\nReaction 0-0-5: CN(C)C(OC(C)(C)C)OC(C)(C)C (available) + C[C@H](c1ccccc1)N1C[C@]2(C(=O)O)C=CC[C@@H]2C1=S (unavailable)\n</tool_response>\n```\n>LLM:\n```\n<think>\nLet me analyze each reaction option systematically:\n\n1. Reaction 0-0-0: Direct precursor is unavailable and appears nearly as complex as the target. Not ideal.\n2. Reaction 0-0-1: Uses an available phosphorus reagent but requires an unavailable oxime precursor. The reaction between oxime and phosphorus ylide is plausible but not optimal.\n3. Reaction 0-0-2: Similar to 0-0-1 but with a different phosphorus reagent. Still requires the same unavailable oxime. No significant advantage.\n4. Reaction 0-0-3: Forms the target via oxidation of a thiirane, but the precursor is unavailable and equally complex.\n5. Reaction 0-0-4: Represents an elimination product, but precursor is unavailable and not simpler.\n6. Reaction 0-0-5: Carbamate formation, but requires an unavailable carboxylic acid precursor.\n\nOptimal Choice: Reaction 0-0-0 is the most reasonable despite an unavailable precursor because:\n- It represents a direct transformation (likely a cycloaddition or migration)\n- The unavailable precursor is only slightly simpler than target\n- Other options either require multiple steps or have less reliable transformations\n- The phosphorus reagents in other options are less practical\n</think>\n```\n**Question6:** Can the authors compare to additional recent multi-step retrosynthesis methods?\n>Response: We appreciate your suggestion. We did attempt to compare with DreamRetroer[1]. However, we believe the truly relevant code for this work is not open-source. We examined their codebase but found it to be very similar to EG-MCTS, lacking the specific implementations corresponding to the \"Abstraction\" and \"Dreaming\" described in their paper. We tried to contact the authors via email but got no response.\n\n>Regarding RetroGraph[2], its code is not publicly available, preventing a direct comparison. The PDVN paper did include a comparison with RetroGraph, likely due to both works originating from the same organization.\n\n>For MEEA*[3], our comparison results are as follows:\n\n||pass@1|N=50|N=100|N=200|N=300|N=400|N=500|Shorter routes|\n|-|-|-|-|-|-|-|-|-|\n|MEEA*|44.74|62.11|73.16|81.58|83.68|84.74|86.32|50|\n|Retro-R1|55.79|73.21|77.21|82.58|84.47|85.89|86.95|72.3|"}, {"review_id": "P8LA7rzKN8", "reviewer": "Reviewer_VGhi", "summary": "This paper introduces RETRO-R1, a novel framework that models multi-step retrosynthesis as a sequential decision-making process, where an LLM agent interacts with a chemical reaction environment. The LLM agent is trained using reinforcement learning to iteratively select unsolved molecules for decomposition and identify appropriate reactions using external tools. Leveraging the LLM's inherent reasoning and tool-use capabilities, the method demonstrates strong performance on established retrosynthesis benchmarks.", "questions": "1. Given the emphasis implied by 'R1' in the name, what is the specific role of reasoning in RETRO-R1's performance? To what degree does it quantitatively affect the overall success rate or route quality? Can you provide any insights into how the \"length\" or \"pattern\" of the agent's reasoning (e.g., the sequence of thoughts or actions) changes over the course of training?\n2. Your multi-round planning strategy appears to utilize the full model call budget (e.g., up to 500 single-step model calls) before extracting the shortest route. In contrast, many baselines typically terminate once a feasible route is found. Could you clarify whether, for a fair comparison, all baselines were also allowed to exhaust their full iteration/model call budget during evaluation.\n3. What is the prompt templates used for the LLM agent's interactions with the environment and tools.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The integration of LLM agents to emulate a chemist's iterative discovery process in retrosynthesis is a compelling aspect of this work, enabling a data-driven approach to navigate the complex reaction space.\n- The approach leverages the reasoning capabilities of state-of-the-art LLMs, combined with reinforcement learning (specifically PPO and GAE as described in the paper) for learning sequential decision-making with verifiable rewards.\n\nWeaknesses:\n- The overall clarity of the paper could be significantly improved. Several core concepts and their formalizations are either missing or insufficiently explained. For instance, a precise definition of the state representation for RL training, the mechanism of state transitions between molecule and reaction nodes, and the distinction/usage of \"environment tokens\" and \"agent tokens\" in the PPO updates (Section 3.3) are unclear.\n- The paper lacks detail on the specific prompt templates used for the LLM agent, which are crucial for reproducibility and understanding the agent's interaction logic.\n- While \"R1\" in the name suggests a focus on reasoning, the paper does not adequately demonstrate or analyze the LLM agent's emergent reasoning capabilities. How does the iterative planning strategy or RL training specifically shape the agent's \"thinking\"? A more detailed analysis or qualitative examples (e.g., thought process demos for challenging cases) would be beneficial.\n- The reported computational cost (70 hours for training on 32x A800 80G GPUs) is substantial.", "was_revised": true, "final_justification": "The rebuttal process confirmed my concerns regarding the manuscript's evaluation. The practical value of the proposed method remains questionable, as the marginal performance improvement does not appear to justify the significant computational cost. Since the authors promptly response to the comments, I would neither decrease nor increase the score.", "rebuttal": "**Question1:** The overall clarity of the paper could be significantly improved.\n\n>Response: We appreciate you highlighting the need for improved clarity. We agree and will significantly enhance these sections in our revised manuscript.\n\n>Agent Tokens vs. Environment Tokens: Agent tokens are those generated by LLM. Environment tokens are those generated by the environment to communicate node states to the LLM.\n\n>Reinforcement Learning State Representation: The state representation is a long sequence formed by the alternating appearance of agent tokens and environment tokens.\n\n>Alternating Molecular and Reaction Nodes: At a molecular node, the agent uses the CALL action to interact with a molecule, transitioning to a reaction node (representing potential reactions for that molecule). At a reaction node, the agent uses the SELECT action to choose a specific reaction, which then leads to a new molecular node (representing the precursors from that chosen reaction).\n\n>We believe these additions will provide a much clearer and more comprehensive understanding of our methodology.\n\n**Question2:** What is the prompt templates used for the LLM agent's interactions with the environment and tools.\n\n>Response: We've provided instances of all the prompt templates used for the LLM agent's interactions with the environment, specifically demonstrating the synthesis of the complex molecule C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=S. These prompts form a multi-turn conversational format during the generation process.\n\n>The order is: System prompt -- User prompt -- [ LLM response1 -- Environment response1 -- LLM response2 -- Environment response2 ] x N\n\n>The notation x N indicates that the content within the [] brackets will be repeated until a termination condition is met.\n\n>System prompt\n```\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{\"name\": \"single_step_retro\", \"description\": \"Perform single step retrosynthesis for a molecule and return several possible reactions to synthesize it. Note that the reactions may be incorrect and each reactant is marked as 'available' or 'unavailable'. The unavailable molecules have to be synthesized further.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"molecule\": {\"type\": \"string\", \"description\": \"The ID for the molecule to be synthesized. For example, 0-0 .\"}}, \"required\": [\"molecule\"]}}\n{\"name\": \"select_reaction\", \"description\": \"Given several reactions to synthesize a molecule, use this tool to select one from them.\", \"parameters\": {\"type\": \"object\", \"properties\": {\"reaction\": {\"type\": \"string\", \"description\": \"The ID for the selected reaction. For example, 0-0-0 .\"}}, \"required\": [\"reaction\"]}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call>\n```\n>User prompt\n```\nYou are a professional organic chemist, skilled at finding pathways to synthesize novel molecules. Given a target molecule, your task is to find a pathway where the target can be eventually synthesized using ingredient molecules, and each step of the pathway is a viable chemical reaction. The maximum number of reaction steps you can use is 30.\nYou can use the tools provided to you to find the pathway. You can use the tool as many times as you want.\nYou must first conduct reasoning inside <think>...</think>. Then use the tool call <tool_call>...</tool_call> to call the tool.\n\nOutput format for tool call:\n<think>\n...\n</think>\n<tool_call>\n...\n</tool_call>\n\nThe target molecule is C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=S. Now start to search for the pathway.\nMolecule state 0: Unsolved molecules:\nMolecule 0-0: C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=S\nNow, 0 steps are used.\nPlease select the molecule with the highest synthesis difficulty from the unsolved molecules. Use the tool 'single_step_retro' to synthesize it.\n```\n>LLM response1:\n```\n<think>\nThe molecule with the highest synthesis difficulty is 0-0 as it is the only one in the unsolved list.\n</think>\n<tool_call>\n{\"name\": \"single_step_retro\", \"arguments\": {\"molecule\": \"0-0\"}}\n</tool_call>\n```\n>Environment response1: (Could you please see the specific reactions and thinking under reviewer aDxU due to the character limit? We are really sorry for that.)\n```\n<tool_response>\n**please see under reviewer aDxU.**\nPlease determine which of these reactions are possible. Then use the tool 'select_reaction' to choose the best reaction whose reactants are available or not available but easy to synthesize.\n</tool_response>\n```\n>LLM response2:\n```\n<think>\n**please see under reviewer aDxU.**\n</think>\n<tool_call>\n{\"name\": \"select_reaction\", \"arguments\": {\"reaction\": \"0-0-0\"}}\n</tool_call>\n```\n>Environment response2:\n```\n<tool_response>\nSuccessfully select reaction 0-0-0 to synthesize molecule C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=S . The state goes to: \nMolecule state 1: Unsolved molecules: \nMolecule 1-0: C=CC[C@@H]1C(=S)N([C@H](C)c2ccccc2)C[C@@]1(C=C)C(=O)OC(C)(C)C\nTo reach this state, at least 1 reaction steps are used.\nPlease select the molecule with the highest synthesis difficulty from the unsolved molecules. Use the tool 'single_step_retro' to synthesize it.\n</tool_response>\n```\n**Question3:** How does the iterative planning strategy or RL training specifically shape the agent's \"thinking\"? Can you provide any insights into how the \"length\" or \"pattern\" of the agent's reasoning changes over the course of training?\n\n>Response: In our response to Question 2, we provided an example of the LLM interacting with the environment, and we'll elaborate on that here to demonstrate its qualitative reasoning.\n\n>To synthesize the complex molecule `C[C@H](c1ccccc1)N1C[C@]2(C(=O)OC(C)(C)C)C=CC[C@@H]2C1=S`, our LLM agent first calls the single-step retrosynthesis model. After receiving five candidate reactions from the environment, the LLM systematically analyzes and compares each reaction.\n\n>This process reveals distinct \"thinking\" behaviors shaped by the RL training:\n\n>The LLM receives five candidate reactions from the environment, and then analyzes and compares these reactions one by one. This analysis reveals interpretable reasoning:\n\n>Molecular Complexity: For instance, it identifies that the precursor in \"reaction 0-0-0\" is slightly simpler than the target molecule, while the precursor in \"0-0-4\" is not simpler. When visualized, \"reaction 0-0-0\" shows the precursor having an opened ring structure compared to the target, validating the LLM's judgment.\n\n>Reaction Plausibility/Optimality: For example, it deems the reaction in \"0-0-1\" as \"not optimal.\" This suggests an ability to evaluate the quality or efficiency of a proposed reaction.\n\n>These examples clearly indicate that the LLM indeed comprehends molecules and chemical reactions. This stands in stark contrast to baseline methods, where neural networks directly predict a numerical value for synthetic difficulty. Such values are often heavily dependent on the training dataset and lack the inherent chemical understanding and generalization capability that our LLM exhibits.\nDuring the training process, the reasoning length of the agent gradually shortened, and the agent learned to express key information more concisely, which was consistent with the results of the concurrent work[1].\n\n>[1] Wang, Zihan, et al. \"Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning.\" arXiv preprint arXiv:2504.20073 (2025).\n\n**Question4:** The reported computational cost (70 hours for training on 32x A800 80G GPUs) is substantial.\n\n>Response: Thank you for pointing out this problem. We acknowledge that training LLms with RL is resource-intensive.\nHowever, our core objective is to develop a chemist agent powered by an LLM, capable of chemist-like reasoning and solving complex chemical problems. These LLMs can explicitly think and deliberate on each decision, which significantly boosts both interpretability and reliability.\n\n>RL is crucial for achieving this goal, as it dramatically enhances the LLM's reasoning abilities. It's also remarkably data-efficient; we achieved competitive performance using only 11,366 data items, compared to 299,202 used by baselines. Furthermore, LLMs possess strong generalization abilities, allowing their capabilities to be easily transferred to out-of-domain datasets, thereby enabling broader applicability.\n\n**Question5:** Many baselines typically terminate once a feasible route is found. Could you clarify whether all baselines were also allowed to exhaust their full iteration/model call budget during evaluation.\n\n>Response: Thank you for this suggestion. All baselines are deterministic search algorithms that return the \"best\" route under their scoring rules and then stop. This means they don't fully use the allotted computational budget. To address this, we modified PDVN, the best-performing baseline. We changed its code so it continues to expand nodes after finding a route, using the full budget. We then found the shortest route within the resulting AND-OR tree, rather than relying on its internal scoring.\n\n>On retro*-190: PDVN has 90 shorter routes while Retro-R1 has 72.3.\nOn ChEMBL-1000\n\n||5|10|15|20|25|30|\n|-|-|-|-|-|-|-|\n|PDVN|66.6|77.7|80.1|80.6|80.6|80.6|\n|Retro-R1|65.4|77.9|81.2|82.5|83.2|83.7|\n>PDVN's performance did improve, especially for finding more and shorter routes on in-domain datasets. However, on the out-of-domain ChEMBL-1000 dataset, PDVN still lags significantly behind Retro-R1, particularly for complex molecules requiring longer synthetic routes. This is because methods like PDVN are trained on data with route lengths not exceeding 15. This limits their ability to generalize to longer routes, causing them to search broadly rather than deeply."}, {"review_id": "0ptwO4yWcN", "reviewer": "Reviewer_oBQa", "summary": "The paper presents a novel method for multi-step retrosynthesis planning - RETRO-R1. The retrosynthesis planning is formulated as a sequential decision-making task, where the LLM agent takes actions based on environmental feedback. RETRO-R1 uses a plug-in single-step retrosynthesis model as a tool and builds a global graph to ensure iterative planning strategy. The authors demonstrate that the framework achieves SOTA or highly competitive performance compared to other retrosynthesis planning baselines.", "questions": "1. What are the underlying reasons for a specific reward design mentioned in the paper?\n2. Why was Qwen model chosen as the policy network across all the other available LLMs?\n3. Why was this exact template-based MLP chosen as the single-step model?\n4. Are the authors familiar with this paper? https://arxiv.org/pdf/2501.08897 What are the principle differences between this work and Retro-R1?\n5. Is it possible to calculate top-k success rate metrics and include them in Table 1?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "__Strengths:__\n1. Instead of using LLMs in a standard QA format, RETRO-R1 formulates retrosynthesis as a sequential decision-making process involving tool-use and environment feedback. This idea is quite original.\n2. While prior retrosynthesis work has used reinforcement learning, training a general-purpose LLM with end-to-end RL is new in this domain.\n3. The authors provide experiments across multiple baselines and datasets as well as ablation studies to outline strong empirical results.\n4. Reproducibility is addressed via github links and thorough description of implementation details.\n\n__Weaknesses:__\n1. Figure 1 seems to be difficult to understand. The explanation provided in the text is a bit chaotic and only refers to Training and Planning parts leaving other parts of the picture untouched.\n2. There might be a typo in Figure 3 caption. Perhaps, BFS should be DFS.\n3. Some architectural aspects seem to come out of nowhere. E.g., the design of the reward function, the choice of Qwen LLM as a policy network, the choice of a template-based single-step retrosynthesis model. It would be interesting to see the motivation behind these choices as well as ablation studies covering these aspects.\n4. This paper https://arxiv.org/pdf/2501.08897 is not cited, although the agentic approach is very similar to what the authors of the current paper are proposing.\n5. Pass@1 success rate is an important metric for evaluation, however it might not take into account the ability of the model to discover multiple viable routes, which is essential in retrosynthesis planning. Alternative metrics like top-k success rate could provide a more complete picture.", "was_revised": true, "final_justification": "After reviewing the rebuttals and the paper again, I decided to raise my evaluation but lower my confidence score. On one hand, the authors sufficiently addressed my questions. Although I am not convinced by some of the design choices and arguments, I think overall the contribution is fair and the supporting evidence is sufficient. On the other hand, I agree with another reviewer that the practical implications of this application study are limited. First, the approach is very computationally expensive. Second, I find no evidence of wet-lab validation, which would be required to assess the utility of the proposed agent. Therefore, I am less sure about this evaluation.\n\nThe authors missed the rebuttal deadline and submitted all answers late as official comments. As per NeurIPS instructions, we have to ignore all submitted materials. I, personally, see no reason to ignore them, so I haven't.", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper proposes RETRO-R1, a framework that conceptualizes multi-step retrosynthesis as a problem of sequential decision-making under a partially observable environment. An LLM-based agent is optimized through reinforcement learning to perform iterative decomposition, selecting molecules and invoking external tools to predict reactions. The method capitalizes on the pre-trained reasoning and tool-use capacities of LLMs, demonstrating robust and competitive performance on canonical retrosynthesis benchmarks.


Suggested revisions:
- ​​Clarify evaluation metrics​​ (e.g., emphasize route diversity).
- Expand discussion on computational trade-offs​​ (e.g., cost vs. interpretability).
- Cite the LLM-based retrosynthesis works​ to contextualize contributions.
Decision: Accept (poster)