Reference-guided Policy Optimization for Molecular Optimization via LLM Reasoning
Authors: Xuan Li
Zhanke Zhou
Zongze Li
Jiangchao Yao
Yu Rong
Lu Zhang
Bo Han
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model; Molecular Optimization; LLM Reasoning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Large language models (LLMs) benefit substantially from supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) in reasoning tasks. However, these recipes perform poorly in instruction-based molecular optimization, where each data point typically provides only a single optimized reference molecule and no step-by-step optimization trajectory. We reveal that answer-only SFT on the reference molecules collapses reasoning, and RLVR provides sparse feedback under similarity constraints due to the model's lack of effective exploration, which slows learning and limits optimization. To encourage the exploration of new molecules while balancing the exploitation of the reference molecules, we introduce **Re**ference-guided **P**olicy **O**ptimization (RePO), an optimization approach that learns from reference molecules without requiring trajectory data. At each update, RePO samples candidate molecules with their intermediate reasoning trajectories from the model and trains the model using verifiable rewards that measure property satisfaction under similarity constraints in an RL manner. Meanwhile, it applies reference guidance by keeping the policy’s intermediate reasoning trajectory as context and training only the answer in a supervised manner. Together, the RL term promotes exploration, while the guidance term mitigates reward sparsity and stabilizes training by grounding outputs to references when many valid molecular edits exist. Across molecular optimization benchmarks, RePO consistently outperforms SFT and RLVR baselines (e.g., GRPO), achieving improvements on the optimization metric (Success Rate $\times$ Similarity), improving balance across competing objectives, and generalizing better to unseen instruction styles. Our code is publicly available at https://github.com/tmlr-group/RePO.

Published as a conference paper at ICLR 2026
REFERENCE-GUIDED POLICY OPTIMIZATION FOR MOLECULAR OPTIMIZATION VIA LLM REASONING
Xuan Li" Lu Zhang'
Zhanke Zhou' Bo Han
Zongze Li!
Jiangchao Yao?
Yu Rong
'Hong Kong Baptist University DAMO Academy, Alibaba Group
CCIC, Shanghai Jiao Tong University 4 Hupan Lab
ABSTRACT
Large language models (LLMs) benefit substantially from supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR) in reasoning tasks. However, these recipes perform poorly in instruction- based molecular opti- mization, where each data point typically provides only single optimized reference molecule and no step-by-step optimization trajectory. We reveal that answer-only SFT on the reference molecules collapses reasoning, and RLVR provides sparse feedback under similarity constraints due to the model's lack of effective explo- ration, which slows learning and limits optimization. To encourage the exploration of new molecules while balancing the exploitation of the reference molecules, we introduce Reference- guided Policy Optimization (RePO), an optimization approach that learns from reference molecules without requiring trajectory data. At each update, RePO samples candidate molecules with their intermediate reasoning tra- jectories from the model and trains the model using verifiable rewards that measure property satisfaction under similarity constraints in an RL manner. Mcanwhile, it applies reference guidance by keeping the policy's ' intermediate reasoning trajec- tory as context and training only the answer in a supervised manner. Together, the RL term promotes exploration, while the guidance term mitigates reward sparsity and stabilizes training by grounding outputs to references when many valid molec- ular edits exist. Across molecular optimization benchmarks, RePO consistently outperforms SFT and RLVR baselines (e.g., GRPO), achieving improvements on the optimization metric (Success Rate Similarity), improving balance across competing objectives, and generalizing better to unseen instruction styles. Our code is publicly available at https: / thub. com/ tml r- r-group/Rese ReeO.
INTRODUCTION
L.MMs have advanced rapidly on reasoning-intensivy tasks through supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RIVR) (Guo et al., 2025; Muennighoff et al., 2025; Zhang et al., 2025b). Both paradigms allow LLMs to perform deliberative reasoning before generating answers and solve problems requiring multiple reasoning steps (Zelikman et al., 2022; Zhou et al., 2024; 2025a). However, how these recipes perform in scientific tasks remains under- underrinvestigated.
In this paper, we study instruction-bass molecular optimization, where the model is required to optimize the molecular property while maintaining structural similarity with the original molecule (Fig. 1). This task involves competing objectives: improving target property requires non-trivial structural edits (e.g., adding functional groups), but those edits will reduce structural similarity the original molecule and even break chemical validity (Liao et al., 2024; López-Pérez et al., 2024).
The data point in most datasets, however, provides only a single optimized molecule for reference, without reasoning trajectory. This supervision mismatch raises key challenge: how to support (i) effective exploration of the constrained chemical space for molecules with desired properties and (ii) answer-level anchoring the available references to exploit provieet reference molecules.
"Correppnndence Bo Han (bhanml (bhanml@comp... www.://ww....o..........

---

Published as a conference paper at ICLR 2026
Maintaining Structural Similarity Similarity (m, m' 0
Editable region
Editing Structure to Improve QED
QED=0.45
QED=0.66
Please medify the melecull
More like optimizing QED properties. Lean start by. The optimized molecule be
COC-OIC/-CNclcee INcclccceNNN(eeexccccccc to increase its QED value.
SMILES of m10
Figure 1: Molecular optimization aims to optimize the given molecule by modifying its components while maintaining the structural similarity of the original molecule after modification. The molecule presented as SMILES (Weininger, 1988), a sequence of symbols representing atoms and bonds.
Model
Dataset
Disil-SFT
Instruction
Reference
5a 5im-0.71
Reference guidance
0.4
0ED-2.45
Generate with prefix
Success Rate
0.3
Answer
Reasoning
Answer G
RLVA
0.2
Ap
QED-0.40 5m--3.76
update
Logp
Molecular Property
Figure 2: Illustration of RePO. The model generates answers via reasoning; reference guidance anchors to the reference conditioned on the reasoning context, while RLVR optimizes the property under similarity constraints.
Figure 3:
Performance comparison on molecular optimization tasks. De- tails of molecular properties be found in Appendix I.1.
Notably, established RLVR/SFT recipes face substantial limitations in this domain (Yue al., 2025; Gandhi et al., 2025; Chen et al., 2025). We show that, with only reference molecules as supervision, SFT often collapses into deterministic, answer-only outputs and suppresses the multi-step reasoning needed for exploration. When RL starts from such an SFT initialization, RLVR updates often fail to produce multi-step reasoning trajectories and instead preserve the short-response style. Conversely, direct RL from the base model depends on rare answers that both improve the property and satisfy similarity; early successes are usually tiny edits with marginal gains, which drive conservative optimizations, yield sparse learning signals, and limit final performance.
To address these limitations, we propose Reference-guidde Policy Optimization (RePO), an optimiza- tion approach that provides reference guidance while permitting many valid molecular edits (Fig. 2). Concretely, each RePO step samples candidate molecules from the model given an instruction, scores them with verifiable reward (property satisfaction while enforcing similarity), and updates the policy with three terms: (i) an RLVR term that upweights higher-reward candidates and lowers the low-reward ones, (ii) an answer-level reference-guidance term that increases the likelihood of the reference conditioned on the model's 's sampled trajectories, and (iii) a KL regularizer that stabilizes updates. Overall, RePO couples reward-driver exploration in the constrained chemical space with answer-level anchoring to the references, without requiring any labeled intermediate trajectories.
We evaluate RePO on instruction- based molecular optimization benchmarks, including TOMG- Bench (Li et al., 2024a) and MuMOInstruct (Dey et al., 2025). On TOMG- Bench single-objective tasks, RePO achieves the best Success Rate x Similarity four out of six tasks, with up to 17.4% success rate improvement over GRPO (Fig. 3). On MuMOInstruct multi-objective tasks, RePO better balances competing objectives and maintains its advantage under unseen instruction styles. RePO also generalizes across model families and benefits from increased inference-time compute.
We summarize our contributions as follows:
We reveal supervision mismatch in instruction-baat based molecular optimization, which makes answer-only SFT collapse multi-step exploration and limit RLVR's performance (Sec. 3).
We propose RePO, reference-guided policy optimization approach that combines GRPO-style reward-driven exploration with reference guidance conditioned on the model's trajectories (Sec. 4).
We show that RePO improves optimization metrics, generalizes to unseen instruction styles, and benefits from more generation budget during inference (Sec. 5).
2

---

Published as a conference paper at ICLR 2026
2 PRELIMINARIES
Task formulation. Molecular optimization aims to modify an input molecule to optimize target properties while preserving structural similarity (López-Pérez et al., 2024; Lipinski & Hopkins, 2004). As illustrated in Fig. 1, we denote the query q= (z. mu), where z is the textual instruction and mo the imput molecule. The model outputs a response t and an optimized molecule m. Given the candidate molecule space M,
[t; ml consisting of reasoning tokens
function F M - IR, similarity function Similarity(;; and threshold E [0, I]. the task is formulated as
arg max F(m) s.t. Similarity mo) 2 5. mEM
(1)
Here m² denotes an optimized molecule that satisfies both property and similarity constraints. We use m for generie molecule, my for the imput molecule in q. Each dataset instance also provides reference molecule, denoted by Myztf which serves as a reference for m and is used as the guidance target in Sec. 4. We provide additional discussions Appendix G.I.
LLM reasoning in molecular optimization. Despite their merits, conventional molecular design approaches struggle to synthesize molecules with precisely tailored properties (Li et al., 2024b;c) and often generalize poorly to novel objectives or constraint combinations (Dey et al., 2025; Li et al., 2024a; Zhang et al., 2025a). These limitations motivate us to leverage LLMs' ability to follow natural-language instructions (Chang et al., 2024) for optimizing molecules. In addition, prior work suggests LLMs can capture non-trivial knowledge of molecular properties (Guo et al., 2023).
Nevertheless, LLMs still struggle under chemistry-specifie constraints. Benchmarks such as SciBench (Wang et al., 2023a) and ChemBench (Mirza et al., 2025) report substantial degradation when tasks require validity preservation property-pree preference satisfaction. Accordingly, recent GPT- based molecular optimization methods often embed LLMs in black-box pipelines: MOLLEO (Wang et al., 2025b) uses evolutionary search with LLM-proposed edits, while others fine-tune LLMs as scoring oracles (Nguyen & Grover, 2025) or couple them with chemistry tools (Bran et al., 2023).
RLVR for LLM reasoning. We adopt GRPO (Shao al., 2024) for training the LLM in the RLVR paradigm. For data pair (g, mret) ~ DD the old policy Vold samples G responses {o;] 1- Let r(o,, 4) be the scalar reward computed from the molecule m, in o and the input molecule mo in q, and Nee be the reference policy for KL regularization. GRPO maximizes the following objective
1o.1
To To o,,ck)
TTR)O(T)) E. (a ~aa
min
G
Oi
Tola (i,a 0i,ck)
(2)
To(or, 0z,ck)
clip Told Tola(oi, 0i,ck)
E, 1 + 2) AA.)) DKL (To (
where ÅL, = (r(o;, q) meau(fr(o) g) ()
{)}==)) is the token-wise group-relative advantage. We use the K3 estimator (Schulman., 2020) for calculating the KL regularization:
Trar Oi,ck) DRL. (Tel(TeT)))))) To(0 /1, 0i,ck) To(oc,lly 0i,ck)
(re 0i,ck)
log
(3)
SUPERVISION MISMATCH UNDER COMPETING OBJECTIVES
Eqn. improves the policy by increasing the likelihood of responses that achieve higher reward. In instruction-based molecular optimization, however, high-reward samples are often rare early in training: among generated molecules, only a small fraction both (i) achieves meaningful target- property improvement and (ii) satisfies the similarity constraint. In addition, each data point typically provides only a single reference molecule myet without intermediate edit trajectories. This supervision mismatch leaves the model with little guidance how to transform my into better molecules under the constraint, causing the policy to be conservative, near- saar-identtty edits.
To examine how these issues manifest in practice, we compare three training recipes under matched settings: (i) GRPO from a base model. (ii) onal--nswer-only SET from a base model. and (iii) GRPO initialized from final-answer-on SFT model, denoted GRPO (SFT-init). We report success rate,
3

---

Published as a conference paper at ICLR 2026
0.5
100-
GRPO
0.4-
0.375
0.8
0.458
0.858
RePO
0.318
Similarity
0.434
Completion Length
500
Success
0.244
0.6
800-
0.2
500-
0.2
200
GRPO (SFT initi
Data
0.0
0.0
100
SFT
GAPO
GAPO (SFT-ro))
RePO
SFT
GAPD
GRPO (SFT4nk)
8eP0
100
200
300
400
Step
Figure 4: Average success rate and similarity for SFT, GRPO, GRPO (SFT-init), and RePO on property optimiza- tion. GRPO achieves high similarity but low success rate. RePO improves success while maintaining high similarity.
Figure 5: Training dynamics of com- pletion lengths of different methods. GRPO (SFT-init) generates short re- sponses that match the SFT data.
similarity on the test set, and response length during training (as a proxy for whether multi-step reasoning is expressed). We employ 2wen-2.5-33 Instruct (Qwen et al., 2024) the base model and the property optimization task of TOMG-Bench for training and evaluation. We employ structural similarity (Eqn. 5) and target properties (Eqn. 6) rewards for GRPO training. Detailed settings are provided Appendix Figs. 4 and summarize the results, leading to three observations.
Observation 3.1 (GRPO the base model struggles under competing objectives in molecular optimization). GRPO tends to generate conservative edits that remain very close to the input molecule ma- These edits keep similarity high, but yield limited property improvement (Fig. 4). As result, most sampled molecules contribute weak learning signals, and updates concentrate on a narrow local regime rather than exploring larger yet promising edits in chemical space.
Observation 3.2 (SFT on the base model produces answer-only outputs and weak similarity control). Answer-only SFT trains the model to directly imitate the reference molecule, producing short, answer- only outputs whose lengths match the SFT training data (dashed line in Fig. 5). While SFT attains a moderate success rate, yields lower similarity than RL methods (Fig. 4), indicating that token-level imitation of the reference alone provides limited control over the similarity-
trade-off.
Since SFT collapses outputs into short, answer-only responses, we next ask whether applying GRPO on top of this SFT initialization can recover multi-step reasoning.
Observation 3.3 (GRPO on the SFT model does not recover multi-step reasoning). In Fig. 5, GRPO (SFT-init) largely preserves the short-response style, producing completion lengths that closely match the SFT data rather than recovering the multi-step reasoning observed in GRPO from the base model. This short-response behavior carries over to test performance: GRPO (SFT-init) achieves the lowest success rate among all methods while maintaining high similarity (Fig. 4), indicating that it inherits the SFT model's limited exploration without gaining RL-driven property improvement.
Taken together, these observations highlight a modeling gap: reward-only updates are often too sparse to reliably push the policy beyond conservative edits, yet naive token-level imitation of the reference molecule can over-constrain the policy and suppress diverse edit trajectories. An improved approach should therefore provide (i) directional, answer-level reference guidance toward myat enhancing the learning signal, while (ii) avoiding token-level process imitation so that multiple valid reasoning traces and edit paths remain possible under the same instruction.
4 REPO: REFERENCE-GUIDE POLICY OPTIMIZATION
The previous analysis identifies two desiderata: answer-level guidance that enhances learning signal and trajectory-Ievel RL updates that avoid token-level imitation and preserve diverse exploration. RePO satisfies both by using the reference molecule answer-level anchor, while retaining RLVR-style updates on the full reasoning trajectories to preserve exploration over diverse edits.
Objective. For each query q (x, mo), we sample {or] Toua( g). where each response is partitioned as o [t.; mi with reasoning tokens t; and a final molecule mn (Fig. 6). Let r(o, q) be the scalar reward computed from my and the input molecule mo q (Sec. 4). We compute the
4

---

Published as a conference paper at ICLR 2026
token-wise group-relative advantage A as in Eqn. 2. The RePO objective is
lod
(RePO(TO)) E (g.mrat) {ar -
min (piaAr, 1 8) e)A,,
Exploration term
lo.
+ 3
log To (Mref g (2)
DEL (Tol(TollTer
Answer-level reference guidance
KL regularization
where Pi, = TT( (1,0))))TTok (o,, (,,c)). and DKL (Tel (Tell uses the estimator as Eqn. 2. The exploration term applies updates to all tokens in o,. The guidance term increases the likelihood of the reference molecule conditioned on the reasoning prefix f, and B controls its strength.
We next provide detailed description of the remaining key design choices in framework, including (i) the reference molecule, (ii) the reward function and (iii) the overall training procedure.
Reference molecules. For each query q, we have the dataset providing reference molecule myef+ which serves as a proxy for the (unknown) optimal solution m" in Eqn. 1. RePO uses Mref only as answer-level anchor in Eqn. 4. In practice, convenience, we continue to denote the checked reference as Myef- Importantly, Myef does not provide reasoning supervision: RePO does not treat preferred reasoning path, and trajectory learning is driven by rewards on model-samples responses. We showcase Myef as follows:
apply RDKit validity checks to Myet's For notation
Query: "Please modify the molecule ccCc (C (=0) oC) c (0) c1ccCcC1 increase its LogP value." Reference molecule: O=C (OC (CCCCO) CCCCS) eleceeel
Reward design. The reward instantiates the objective in Eqn. 1. For any molecule m, let (P denote its molecular fingerprint, fixed-length binary vector whose entries indicate the presence of specific substructures in m. We define the scalar reward as r(m, my) = prpp (m, my) +T mo).
Structural similarity Ustract we use Tanimoto similarity (Bajusz et al., 2015)
FP(mm n FP(()))
Istruci (m, mo)
E [0, 1].
FP m FP())))
This term encourages structural preservation by assigning higher rewards molecules close to ma-
Target property prop the instruction specifies a target property and whether it should be increased or decreased (e.g., increasing the molecular LogP value); let F.)) denote the corresponding property function. We use binary improvement property reward defined as follows:
{1
if F(m) 2 F(mo) (for increasing).
prop m. ma)
if F(m) I F(mo) (for decreasing), otherwise.
Training, For each query y, we sample for [4 mm]m) and compute rewards r(o:.g) on the parsed m, (with mi from q). We then (i) compute group-relative advantages and apply the clipped GRPO update to all tokens in o (the GRPO term in Eqn. 4); and (ii) add the reference reference-guidance term log no (Mret | q. fz), which is computed over the answer tokens only, with the sampled reason- ing prefix 4 as context (Fig. 6). Thus, RePO does not imitate reasoning tokens; instead, it leverages Mee to provide an answer-level anchor while let- ting RL update on sampled trajectories.
Question
Think
Answer
(9. m))
optimization
Token
SFT
Supervised optimization
RaPO 1000
Explotation term
Replace
Cg. fu meet)
Guidiacce term
Figure 6: Token partition and gradient applica- tion for SFT, GRPO, and RePO.
Remark 4.1. Reference guidance operates solely at the answer level: it increases the likelihood of instruction-saatiffiing answers early in training, reducing reward sparsity and yielding more informative RL updates, which in turn shape the reasoning tokens and support model to explore molecules with better properties while satisfying the constraint.
5

---

Published as a conference paper at ICLR 2026
Table 1: SuccessRate (SR), Similarity (Sim), and their product (SRxSim) on TOMG-Bench for structure- and property-based optimization tasks. The higher the better. The best results for each task are bolded, and the second-best results are underlined.
Task type
Objective
Metric
Base Model Distill-SFT
SFT
GRPO
GRPO (SFT init)
RePO
SR 0.086
0.100
0.238
0.005
0.246
0.307
AddComponent
Sim
0.763
0.604
0.619
0.992
0.635
0.778
SRmSim
0.066
0.060
0.147
0.005
0.156
0.239
Structure- based optimization
SR 0.107
0.188
0.203
0.008
0.232
0.138
DelComponent
Sim 0.864
0.682
0.755
0.994
0.759
0.887
Sim
0.092
0.128
0.153
0.008
0.176
0.140
SR 0.057
0.078
0.366
0.053
0,420
0.429
SubComponent
Sim 0.815
0.633
0.721
0.972
0.713
0.802
SRxSim 0.046
0.049
0.264
0.052
0.299
0.344
SR 0.188
0.208
0.297
0.138
0.223
0.312
QED
Sim 0.693
0.594
0.697
0.889
0,863
0.756
SR x Sim
0.130
0.124
0.207
0.123
0.192
0.236
SR 0.268
0.234
0.298
0.379
0.212
0.415
Property
LogP
Sim 0.627
0.579
0.692
0.806
0.863
0.715
optimization
SRx 0.168
0.135
0.206
0.305
0.183
0,297
SR 0.252
0.214
0.359
0,214
0,265
0.399
MR
Sim 0.685
0.619
0.663
0.880
0.850
0.736
SixSim
0.173
0.132
0.238
0.188
0.225
0.294
EXPERIMENTS
We evaluate RePO on instruction-based molecular optimization. For readability, we organize this section into four blocks: setup (Sec. 5.1), main quantitative results (See. 5.2), mechanism and robustness analyses (Sec. 5.3), and qualitative plus inference-scalie analyses (Sec. 5.4).
5.1 EXPERIMENT SETTINGS
In what follows, we describe the setting of the experiments, including the dataset, baselines, and evaluation metrics. Detailed settings are provided in Appendix I.
Evaluation Metrics. We evaluate model performance using three complementary metrics. Success Rate is the fraction of test molecules for which the model meets the specified property objective. Similarity is measured by the Tanimoto coefficient (Bajusz et al., 2015), which assesses the structural similarity between the input and optimized molecules. Following Li et al. (2024a), to jointly capture both optimization effectiveness and structural preservation, we report Success Rate x S Simil larity, which reflects the model's ability to balance property improvement with maintenance of molecular structure. Detailed discussions on metrics are in Appendix I.1.
Datasets. We employ two instruction- based molecular optimization benchmarks, TOMG-Bent Bench (Li et al., 2024a) and MuMOInstruut (Dey et al., 2025), to evaluate the knowledge of LLM on molecular structure and properties. We employ the reference molecules from the molecule optimization datasets as demonstration molecules. We provide a detailed discussion on the validity of the demonstration molecules in Appendix J.5. More detailed description of the datasets in Appendix I.2.
Baselines. We use Qwen-2 5-3B Instruct as the base model. We compare with Distill-SFT, which applies SFT the s1.1K dataset (Muennighoff' al., 2025); SFT, which fine-tunes the model on each benchmark training split; GRPO, which applies Eqn. 2; and GRPO (SFT init).
5.2 QUANTITATIVE RESULTS
We summarize the main findings from Tabs. 1 and 2.
RePO elicits the model's chemical reasoning on single-objective optimization tasks. Tab. summarizes the results for single-objective molecular optimization. For structure-baset tasks, RePO achieves the best performance on AddComponent and SubCompenent, corresponding to im- provements of 8.3% and 4% over the best method, respectively. For property- -base optimization,

---

Published as a conference paper at ICLR 2026
Table 2: Success Rate (SR), Similarity (Sim), and their product (SRxSim) on MuMOInstruct for seen and unseen instructions. The best results are bolded, and the second bests are underlined.
Task type
Objective
Metric
Base Model
Distill-SFT
SFT
GRPO
GRPO (SFT init)
RePO
SR
0.052
0.078
0.398
0.156
0.088
0.206
BDP
Sim
0.149
0.207
0.254
0.759
0.141
0.569
SRx Sim
0.008
0.016
0.101
0.118
0.012
0.117
SR
0.034
0.022
0.319
0.082
0.022
0.160
Seen instruction
BDQ
Sim
0.117
0.106
0.279
0.479
0.045
0.365
x Sim
0.004
0.002
0.089
0.039
0.001
0.058
SR
0.052
0.064
0,471
0.2T2
0.056
0.274
BPQ
Sim
0.194
0,165
0.244
0.567
0.085
0.509
SRx x Sim
0.010
0.011
0.115
0.120
0.005
0.139
SR
0.052
0.016
0.310
0.148
0.092
0.198
BDP
Sim
0.143
0.099
0.261
0.727
0.147
0.572
SRx Sim
0.007
0.002
0.081
0.108
0.014
0.113
SK
0.042
0.020
0.342
0.078
0.026
0.170
Unseen
BDQ
Sim
0.104
0.077
0.257
0.457
0.058
0.322
instruction
SR x Sim
0.004
0.002
0.088
0.036
0.002
0.055
SR
0.050
0.050
0,419
0.186
0.042
0.242
BPQ
Sim
0.130
0.143
0.248
0.573
0,063
0.596
SRx Sim
0.007
0.007
0.104
0.107
0.003
0.144
RePO achieves superior or competitive performance compared to all baselines, highlighting its effec- tiveness and robustness across evaluation settings, achieving up to 13.0% absolute improvement over the base model. Notably, GRPO without SFT initialization performs markedly worse, particularly on structure- based tasks. underscoring the challenges of unconstrained exploration in the vast chemical space. In contrast, RePO, which integrates demonstration guidance, consistently outperforms SFT and GRPO, yielding more effective molecular optimization.
We also provide experiments with Nomain-specinf optimizers (Bio-T5 (Pei et al., 2023), Mol-T5 (Ed- wards et al., 2022)) and general LLM-based optimizers (GPT-40-mini) and multi-round evolutionary methods (Graph-GA (Jensen, 2019), REINVENT (Olivecrona et al., 2017), and MOLLEO (Wang al., 2025b)) in Tab. Notably, RePO competitive with outperforms these methods, despite its single-round optimization and smaller open backbone.
RePO helps the model to balance multi-objective optimization problems. Tab. 2 presents the results for multi-objective molecular optimization. Notably, RePO outperforms baseline methods on BDP and BPQ tasks, achieving up to 4% improvements over baseline methods, highlighting its ability to effectively balance multiple competing objectives simultaneously.
RePO elicit model's generalization ability on unseen instruction styles. Shown in Tab. 2, the performance advantage of RePO is maintained for unseen instructions, achieving superior results despite the model encountering novel instruction formats. The most significant gains are observed in the BDP task, where RePO's approach to guided exploration proves particularly effective navigating the complex optimization landscape involving multiple constraints. These results collectively validate that RePO's is demonstration- guided approach constrains the exploration space while maintaining the model's 's reasoning capabilities across scenarios of multi-objeccivv optimization.
5.3 MECHANISM AND ROBUSTNESS ANALYSES
We first validate the core mechanism via gradient masking, then analyze training dynamics and out- come distributions, and finally report robustness checks (reasoning-quall) quality evaluation, demonstration quality, prompting baselines, and stronger backbones).
Mechanism validation via gradient masking. We ablate gradient masking TOMG-Bench using RePO with random masking (40%/80%) and a no-mask variant. Fig. 7 shows that all ablations underperform full RePO, and removing masking can even fall below the baseline. This is consistent with the role masking: without it, the answer-level supervision can backpropagate through intermediate tokens 44 reinforcing spurious chemically unsound reasoning patterns.
7

---

Published as a conference paper at ICLR 2026
RePO
Random Nask 80%
Random Mask 40%
No Mask
Logp
0.6
RePO No Mask dPmp
RePO aProp
0.6
0.6
0.6
value
0.4
8 0.44
0.2
0.2
0.0
100
200
300
400
500
0.0
5R
5Ra5m
0.0
5R
5mm
5Px5mm
0.0
58
5mm
SAxsin
Iteration
Figure 7: SR, Sim, and SRxSim for RePO masking ablations on the property optimization task.
Figure 8: Training reward dynamics for RePO with masking and no-mask variants.
Model
RePO comupted)
(30% comupted!
RePO
0.
18.89
AMR
15-
Reffrence: 9.05
03
6.84
400
GRPO
RePO
Figure 9: Performance of the base model and RePO corruption variants on property optimization.
Figure 10: Performance of GRPO, RePO, and reference on MR optimization.
RePO
0.04
RePO
0.33
GAPO
GRPO
Density
Base Modell
Density
Base Model
Dereliy
Base Model
6.15
0.02
2.04
0.01
1.02
035
&logP
AMR
60ED
Figure 11: Distribution of property gains on TOMG-Bence -ptperty-ootimizztionn tasks.
Masking improves optimization stability, Fig. 8 compares RePO with and without gradient masking. With masking, the property-improvemeee reward increases over training; without masking, learning largely stagnates. This trend matches the ablation results in Fig.
Robustness under reference corruption. We evaluate robustness by corrupting query-reference alignments within subtasks. As shown in Fig. 9, RePO remains above the baseline with 30% corruption and stays competitive at 50% corruption, indicating graceful degradation under corruption.
Reference-quality robustness. We report the full reference-quality ablation in Fig 10. The results show that even reduced reference supervision remains competitive with the GRPO, while full RePO achieves the strongest performance.
Property-gain distributions show stronger optimization outcomes. Fig. 11 compares histograms of property gains. RePO shifts toward larger positive gains relative to the base model and GRPO, suggesting that improvements are widespread across candidates rather than driven by few outliers.
LLM-aa---judde evaluation supports reasoning-qualit gains. We evaluate explanation quality with an LLM-as-a- a-------udge protocol calibrated to expert chemists (Zhuang et al., 2025). For TOMG- Bench LogP, we sample trajectories per method and score associated explanations. Fig. 12 shows that RePO achieves the highest reasoning suriiiiggggautyyyooo score.
RePO also improves larger backbones and different architectures. We conduct an additional experiment using a larger model, Qwen - 2. 5B Instruct. Tab. 4 shows that with a larger model, RePO exceeds the base model by up to 4.9% absolute improvement. tasks, RePO also outperforms SFT by large margin. We also adopt ama-m 1 1--B-Innee Inst ruct, which differs substantially in architecture and tokenizer, and train the model using the TOMG-Benn property optimization task. As shown in Tab. RePO achieves the overall best performance.
the LogP and MR
Prompting-only CoT baselines are insufficient. We compare RePO with zero-shot and few-shot CoT prompting TOMG-Bench property optimization. As shown in Fig. 13, CoT prompting does
8

---

Published as a conference paper at ICLR 2026
NoMask
Success Rate x Similarity
zoro shot CoT
RoPO
GRPO
0.25
CaT
Base
0.20
Rand40
0.15
Rand80
4.24x1.7
0.10-
RePO
1
0.00
Score
QED
LogP
MR
Figure 12: Quantification analysis of the reason- ing trajectories' quality.
Figure 13: Prompt-based CoT baselines prop- erty optimization.
Table Performance comparison using Llama-3. 1-8B Instruct on TOMG-Bench property optimiza- tion tasks. Best per row is bolded; second-best is underlined.
Metric
Base Model
SFT
GRPO
GRPO (SFT init)
RePO
SR
0,237
0.287
0.197
0.192
0,360
LogP
Sim
0.692
0.763
0.763
0.775
0.746
SRx Sim
0.164
0.219
0.151
0.149
0.269
0.1.9
0.185
0.115
0.121
0,243
QED
Sim
0.722
0.809
0,805
0,811
0,783
SRx Sim
0.115
0.150
0.093
0,098
0.190
SR
0.169
0.230
0.142
0.141
0.293
MR
Sim
0.763
0.808
0.819
0.827
0.789
S
0.129
0.186
0.117
0.116
0.231
Table 4:
Performance comparison of the Table Comparison of RePO with binary and continuous reward functions.
Qwen
2wen-2.5-0 2-B-7B Instruct.
Baseline SFT GRPO GRPO (SFT init) RePO
LogP
QED
MR
QED
0.174
0.252 0.165
0.147
0.213
RePO (binary Tpmp)
0.297 0.236 0.294
LogP
0.277
0.288
0,285
0.302
0.326
MR
0.279
0.318
0.270
0.286
0.328
RePO (continuous T Tppp)) 0.301 0.203 0.292
not close the gap and is often worse than the baseline, while RePO consistently performs best. This result indicates that end -o-ed policy optimization with reference guidance is more effective than prompt-only reasoning elicitation in this domain.
Performance of continuous reward. The property reward Tprop defined the improvement in the target property, + +(F(m²) F F(mo)), where the sign depends on the optimization direction. In Tab. 5, we report RePO on the property-optimizatie yields more stable gains than the continuous version, suggesting that discretized success signal is casier to optimize under the similarity constraint.
task of TOMG-Bench. The binary Tprop variant
Experiments on reward weighting. Fig. 14 studies sensitivity to the guidance weight 3 optimizing LogP. = reduces to GRPO, while large 3 increasingly resembles SFT. Performance improves as increases, but saturates for 3 2 10, suggesting that overly strong guidance can limit exploration.
-itt---disttibution generalization on unseen structure tasks. We train on TOMG-Bende property optimization and evaluate on withheld structure-cditing tasks. Fig. 15 shows that RePO generalizes better than its counterparts, indicating that its learned strategy transfers across instruction types.
5.4
CASE STUDIES
Chemically-validated reasoning. Fig. 16 illustrates the qualitative differences in reasoning ap- proaches between RePO and GRPO on a molecular optimization task. The left panel demonstrates RePO's chemically sound reasoning process: it correctly identifies structural elements and proposes an effective modification (substituting Br with C1). In contrast, GRPO exhibits invalid chemical ex- pressions and proposes chemically implausible modifications (removing nitrogen from heterocyclic

---

Published as a conference paper at ICLR 2026
Awerage
LogP
Base Model
SFT
GRPO
RePO
Success Rata x Simllarty
0.30
Success Rate: x Similaritt
0.25
0.100
0.20
0.075
0.15
0.050
6.10-
0.025
0.00
0.000
Sase Model
HaPOCC.0
SF1
AddComponent
DelComponent
SubComponent
Figure 14: Performance comparison of the base model, RePO with different 3, and SFT.
Figure Performance comparison on unseen
structure tasks.
Modify the molecule CLccc//NG-oj/(ccccc/000OMCCCCCC
87.10
to have lower MR value
The atructrre can ae represented solooo:
electronggaivvtyy pobrtsabilly.
Inealibele Exprosa
Pirst wa nand conaldar de soraciure moclccula:
cartonyl group
Ricasoiing
Enceciis store
diternenet
presunce cotaal groupe motocale
Incernett
berouma ting
from haterocydle sttragen
optiritation
broeaise groap can inmodaace cteiic bisdraaze siluh fendii elceroogrtir cistttuet.
Vlennily the key component
The modilie acrudare can be reprosereed
Meally bramtr cheerine IC0, redecing
Condurting aut evcattio
Expresse
The reveoad
Whong
Comparison of RePO (left) and GRPO (right). Notably, RePO employs chemically sound reasoning with valid substitutions, whereas GRPO yields incorrect reasoning and modifications.
Figure 16:
ring). This comparison underscores RePO's capacity to generate not only structurally valid molecules but also to produce coherent reasoning that captures the underlying chemical principles governing validated and robust molecular property optimization.
BeFO Success Rate
RafO) Smilanty
Base Modill Success Rate
Base Model Similarky
inference-sealing properties. Fig. shows RePO's inference-scaling characteristics. We sample multiple times from the subset of the MR optimization task. The plot reveals that as the number of sampling trials (k) increases, RePO's best-of-k success rate (solid curve) and the sim- ilarity of the trials (dashed curve) both demon- strate marked improvements. These results under- score RePO's 's proficiency in leveraging increased computational budgets inference.
0.82
0.50
5:90
6.62
5.59
Number Triass (4)
Figure 17: Inference-scaling effect on the success rate and similarity metrics.
CONCLUSION
We studied the supervision mismatch that arises when applying SFT and RLVR instruction-basee molecular optimization. Our analysis showed that answer-only SFT collapses multi-step reasoning, while reward-only RLVR (e.g., GRPO) under similarity constraints suffers from sparse feedback and conservative optimization. To address this mismatch, we proposed RePO, which balances exploration of new molecules via reward-driven RL updates on sampled trajectories with exploitation of the reference molecules via answer-level reference guidance, regularized by KL term to stabilize training, all without requiring trajectory supervision. Across TOMG-BB -BGBchh and MuMOInstruct, RePO consistently outperforms SFT and GRPO baselines single-objective, multi-objective, unseen- instruction, and inference-scaling evaluations. Future works include improving automatic reference construction and extending broader scientific optimization domains.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
XI. ZK2, ZZI., and BH were supported by Guangdong Basic and Applied Basic Research Foundation Nos. 2024A1515012399 NSFC General Program No. 62376235, and HKBU CSD Departmental Incentive Scheme. JCY was supported by National Natural Science Foundation of China (No. 62306178) and STCSM (No. 22D22229005).
REFERENCES
Dávid Bajusz, Anita Rácz, and Károly Héberger. Why tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformaties, 7:1-13, 2015.
Brian Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Ceron. Yoshua Bengio, and Bhavya Kailkhura. Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable Ilm post-training. arXiv preprint arXiv:2503. 18929, 2025.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. In NeurIPS, 2021.
Richard Bickerton, Gaia Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, ((2):90-98, 2012.
Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023.
Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han. Envisioning outlier exposure by large language models for out-of-distribution detection. In JCML, 2024.
Chentao Cao, Zhun Zhong, Zhanke Zhou, Tongliang Liu, Yang Liu, Kun Zhang, and Bo Han. Noisy test-time adaptation in laiiuul models. In ICLR, 2025.
Alex Chan, Hao Sun, Samuel Holt, and Mihaela Van Der Schaar. Dense reward for free in reinforcement learning from human feedback. In ICML, 2024.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijic Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1-45. 2024.
I Liang Chen, Xueting Han, Qizhou Wang, Bo Han, Jing Bai, Hinrich Schutze, and Kam-Fai Wong. Eepo: Exploration-enhance policy optimization via sample-then- arXiv preprint arXiv:2510.05837. 2025.
Vishal Dey, Xiao Hu, and Xia Ning. Gellm3o: Generalizing large language models for multi celli-proppetyy molecule optimization. In ACI., 2025.
Peijie Dong, Zhenheng Tang, Xiang Liu, Lujun Li, Xiaowen Chu, and Bo Li. Can compressed Ilms truly act? an empirical evaluation of agentic capabilities in Ilm compression. In JCML, 2025.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. In EMNLP, 2022.
Luyu Fan, Liang Tan, Zhangcheng Chen, Jianzhong Qi, Fen Nie. Zhipu Luo, Jianjun Cheng, and Sheng Wang. Haloperidol bound d2 dopamine receptor structure inspired the discovery of subtype selective ligands. Nature communications, 1(1):1074, 2020.
Tianfan Fu, Wenhao Gao, Connor Coley, and Jimeng Sun. Reinforced genetic algorithm for structure- based drug design. In NeurIPS, 2022.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh. Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self- iilf-improiing reasoners, or, four habits of highly effective stars. In COLM, 2025.
11

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseck-rl incentivizes reasoning in llms through reinforce- ment learning. Nature, 645(8081))6333 :::3666388 2025.
Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. In NeurIPS, 2023.
Bo Han, Jiangchao Yao, Tongliang Liu, Bo Li, Sanmi Koyejo, and Feng Liu. Trustworthy machine learning: From data to models. Foundations and Trends@ in Privacy and Security, 7(2-3):74-22) 246, 2025.
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In ICML, 2022.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics data commons: Machine learning datasets and tasks for drug discovery and development. In NeurIPS, 2021.
Jan Jensen. A graph-based genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12):3567-3572, 2019.
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He. Qingliang Li, Benjamin A Shoemaker, Paul Thiessen, Bo Yu, et al. Pubchem 2019 update: improved access to chemical data. Nucleic acids research, 47(D1))D1022D1100 2019.
Craig Knox, Mike Wilson, Christen M Klinger, Mark Franklin, Eponine Oler, Alex Wilson, Al- lison Pon, Jordan Cox, Na Eun Chin, Seth A Strawbridge, et al. Drugbank 6.0: the drugbank knowledgebase for 2024. Nucleic acids research, 52(DI):D1265DD1275 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shov shot reasoners. In NeurlPS, 2022.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jo W Le Fevre. Molecular refractivity and polarizability. In Advances in Physical Organic Chemistry, volume 3, pp. 1-90. Elsevier, 1965.
Jiatong Li, Junxian Li. Yunqing Liu, Dongzhan Zhou, and Qing Li. Tomg-bench: Evaluating 11ms on text-based open molecule generation. arXiv preprint arXiv:2412.14642. 2024a.
Jiatong Li. Yunqing Liu, Wei Liu, Jingdi Le, Di Zhang, Wenqi Fan, Dongzhan Zhou, Yuqiang Li, and Qing Li. Molreflect: Towards in-context fine-grained alignments between molecules and texts. arXiv preprini 2rXiv:2411.1472) 2024b.
Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arX1v:2311.03191, 2023.
Xuan Li, Zhanke Zhou, Jiangchao Yao, Yu Rong, Lu Zhang, and Bo Han. Neural atoms: Propagating long-range interaction in molecular graphs through efficient communication channel. In ICLR, 2024c.
Chang Liao, Yemin Yu, Yu Mei, and Ying Wei. From words to molecules: A survey of large language models in chemistry. arXiv preprint 20Xx:22422.01339, 2024.
Christopher Lipinski and Andrew Hopkins. Navigating chemical space for biology and medicine. Nature, 432(7011):855-882)) 2004.
Christopher A Lipinski, Franco Lombardo, Beryl W Dominy, and Paul J Feeney. Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Advanced drug delivery reviews, 23(1-3)33-25, 1997.
12

---

Published as a conference paper at ICLR 2026
Elita Lobo, Chirag Agarwal, and Himabindu Lakkaraju. On the impact of fine-tuning chain-of- thought reasoning. In NAACL, 2025.
Kenneth López-Pérez. Juan F Avellaneda-Tamaan Lexin Chen, Edgar López-López, K Eurídice Juárez-Mercado, José MMdinn--Franco, and Ramón Alain Miranda-Quintana, Molecular similarity: Theory, applications, and perspectives. Artificial Intelligence Chemistry, 2(2): 100077, 2024.
Andres M. Bran, Sam Cox, Oliver Schiter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525-535, 2024.
Adrian Mirza, Nawaf Alampara. Sreekanth Kunchapu, Martiño Rios-García Benedict Emoekabu, Aswanth Krishnan, Tanya Gupta, Mara Schilling- Willheliiiiliiii Macjonathan Okereke, Anagha Aneesh, et al. Are large language models superhuman chemists? Nature Chemistry, 17(7):984-985, 2025.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. In EMNLP, 2025.
Tung Nguyen and Aditya Grover. Lico: Large language models for in-context molecular optimization. In ICLR, 2025.
AkshatKumar Nigam, Robert Pollice, and Alán Aspurı-Guzik, Parallel tempered genetic algorithm guided by deep neural networks for inverse molecular design. Digital Discovery, 1(4):390-444, 2022.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformaties, 9:1-14, 2017.
Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. Biot5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. EMNLP, 2023.
A Yang Qwen, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengpeng Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. 5 technical report. arXiv preprint, 2024.
John Schulman. Approximating kl divergence, 2020. URL kl-approx. html.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Wenbo Shang, Yuxi Sun, Jing Ma, and Xin Huang. On the wings of imagination: Conflicting script-based multi-role framework for humor caption generation. arXiv preprint arXiv:2602.06443, 2026.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv.2402.03300, 2024.
Teague Sterling and John J Irwin. Zine 15-ligand discovery for everyone. Journal of chemical information and modeling.
2015.
Rajesh Sundar, Mukul R Jain, and Darshan Valani. Mutagenicity testing: Regulatory guidelines and current needs. In Mutagenicity: assays and applications, pp. 191-228. Elsevier, 2018.
Zhengkai Tu, Sourabh J Choure, Mun Hong Fong, Jihye Roh, Itai Levin, Kevin Yu, Joonyoung F Joung, Nathan Morgan, Shih-Cheng Li, Xiaoqi Sun, et al. Askcos: open-source, data-driven synthesis planning. Accounts of chemical research. 58(11): (8(11):1764-177) 2025.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https: : / / thub. www..//.///..etee..eee www.e/giinaaaa/eeaaaeeeeee trl, 2020.
13

---

Published as a conference paper at ICLR 2026
Guanjie Wang, Jingjing Hu, Jian Zhou, Sen Liu, Qingjiang Li, and Zhimei Sun. Knowledge-guine guided large language model for material science. Review of Materials Research, pp. 100007, 2025a.
Haorui Wang. Marta Skreta, Cher-Tian Ser, Wenhao Gao, Lingkai Kong, Felix Kalthof,,, Chenru Duan, Yuchen Zhuang, Yue Yu, Yanqiao Zhu, et al. Efficient evolutionary search over chemical space with large language models. In ICLR, 2025b.
Liang Wang, Chao Song, Zhiyuan Liu, Yu Rong, Qiang Liu, and Shu Wu. Diffusion models for molecules: A survey of methods and tasks. arXiv preprint arXiv:2502.09511, 2025c.
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam. Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint arXiv:2307.10635, 2023a.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain thought reasoning in language models. In ICLR, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
David Weininger. Smiles, chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Di Wu, Qi Chen, Xiaojie Chen, Feng Han, Zhong Chen, and Yi Wang. The blood-brain barrier: Structure, regulation and drug delivery. Signal transduction and targeted therapy, 8(1):217, 2023.
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In ICLR, 2024.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Fiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 59(8): 3370-3388, 2019.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in ulms beyond the base model? In NeurlPS, 2025.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. NeurlPS, 2022.
Zizhuo Zhang, Lijun Wu, Kaiyuan Gao, Jiangchao Yao, Tao Qin, and Bo Han. Fast and accurate blind flexible docking. In ICLR, 2025a.
Zizhuo Zhang, Jianing Zhu, Xinmu Ge, Zihua Zhao, Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. Co-rewarding: Stable self- supervised rl for eliciting reasoning in large language 2025b.
models. arXiv preprint arXiv:2508.004
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In ACL, 2024.
Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of- of-thought prompting with noisy rationales? In NeurIPS, 2024.
14

---

Published as a conference paper at ICLR 2026
Zhanke Zhou, Xiao Feng. Zhaocheng Zhu, Jiangchao Yao, Sanmi Koyejo, and Bo Han. From passive to active reasoning: Can large language models ask the right questions under incomplete information? In ICML, 2025a.
Zhanke Zhou, Zhaocheng Zhu, Xuan Li, Mikhail Galkin, Xiao Feng, Sanmi Koyejo, Jian Tang, and Bo Han. Landscape of thoughts: Visualizing the reasoning process of large language models. arXiv preprint arXiv:2503.22165, 2025b.
Yiheng Zhu, Jialu Wu, Chaowen Hu, Jiahuan Yan, Tingjun Hou, Jian Wu, et al. Sample-efficient multi-objective molecular optimization with gflownets. In NeurIPS, 2023.
Jiaxi Zhuang. Yaorui Shi, Jue Hou, Yunong He, Mingwei Ye, Mingjun Xu, Yuming Su, Linfeng Zhang, Guolin Ke, and Hengxing Cai. Reasoning-enhancee large language models for molecular property prediction. arXiv preprint arXiv:2510. 10248, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
Ethic Statement
17
B Reproduction Statement
17
LLM Usage Disclosure
17
D Impact Statement
17
Limitations
17
F Future work
17
G Further Discussion
18
G.1 Comprehensive Comparison with Related Works
18
G.2 Additional Backbone and Baseline Comparisons
18
G.3 Training dynamics confirm guided exploration
18
G.4
Demonstration-Quallty Ablation
19
G.5 Further Discussion on the RePO's Demonstration Term
19
G.6 Explain the exploration term
20
G.7 Explicit chemical validity constraints
20
Related Work
20
I Experiment Settings
21
L1 Pharmacological metrics.
21
I.2 Datasets. ...asett........................
22
L3 Training configurations
22
J Full Experiment Results and Further Analysis
23
J.1
Property optimization across random seeds
23
J.2
Molecular Optimization Performance
23
J.3
GRPO with SFT Initialization cannot Generate Readable Outputs
...pp...........................
25
J.4
Comparison with Domain-Specifee LMs
25
J.5
The demonstration data generation process
25
J.6
The design of the reward functions
25
K
Case Study
26
K.1
Case Studies on Single-Objective Optimization
26
K.2 Case Study Multi-Objective Optimization
29
16

---

Published as a conference paper at ICLR 2026
A ETHIC STATEMENT
The study does not involve human subjects, data set releases, potentially harmful insights, applications, conflicts of interest, sponsorship, discrimination, bias, fairness concerns, privacy or security issues, legal compliance issues, or research integrity issues.
B REPRODUCTION STATEMENT
The experimental setups for training and evaluation are described in detail in Appendix I, and the experiments are all conducted using public datasets. We provide the link to our source codes to ensure the reproducibility of our experimental results: attpps:/ /github...../// com/ tt/rrmllrrre mwl--grouppp.oooo RePO.
C LLM USAGE DISCLOSURE
This submission was prepared with the assistance of LLMs, which were utilized for refining content and checking grammar. The authors assume full responsibility for the entire content the manuscript, including any potential issues related to plagiarism and factual accuracy. It is confirmed that no LLM listed an author.
D IMPACT STATEMENT
This paper introduces RePO, novel framework for enhancing large language model (LLM) reasoning in molecular optimization. By leveraging reference-guided policy optimization, our work aims to accelerate the discovery and design of new molecules, which could have positive impacts in fields such medicine, materials science, and sustainable chemistry.
E LIMITATIONS
Despite RePO's promising results, limitations remain. First, the framework relies on the availability of demonstrations, which may searce for novel or complex molecular optimization tasks. addition, while our approach improves LLM reasoning for molecular optimization, the black-box nature of LLM still presents challenges for domain experts seeking to understand the precise reasoning behind specific structural modifications.
F FUTURE WORK
Although RePO is designed to be domain-agnosicc believe it is particularly well-suited for complex scientific reasoning tasks that, like molecular optimization, involve vast search spaces and require domain-specific, multi-step reasoning-where the solution is difficult specify but straightforward to verify. We outline two promising directions for future work:
This task aims to predict sequence of chemical reactions to synthesize a target molecule. The combinatorial explosion of possible reaction pathways makes the search space extremely large, and LLMs may generate chemically invalid suboptimal steps. By leveraging known synthesis routes from chemical literature as demonstrations, the policy model be trained generate retrosynthetic reasoning chains (i.e., stepwise breakdowns of the target molecule). The RL reward can be based on the chemical validity of each proposed reaction (using tools such ASKCOS (Tu al., 2025)).
Retrosynthesis Planning:
Drug Interaction (DDI) Prediction: This task involves predicting whether two drugs will interact and providing a mechanistic explanation. Accurate prediction requires understanding com- plex pharmacological mechanisms (e.g., metabolic pathways), and LLMs are prone to generating plausible but incorrect explanations. Demonstrations can be constructed from known DDI pairs in databases such as DrugBank (Knox et al., 2024). The model would be trained to generate reasoning chains that explain the mechanism of interaction. The RL reward could be based on verifying these mechanistic claims against curated knowledge bases.
17

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of Reward Shaping, Knowledge-guidee Exploration, and RePO
Category
Reward Shaping
Knowledge-guided Ex-
RePO
ploration
Guidance Mechanism
Utilizes environmental signals LLM knowl- edge to guide the policy model.
Integrates professional tools (c.g.. chemistry toolkits (M. Bran et al., 2024), databases (Wang et al., 2025a)) to inject expert priors into work- flows.
Employs remonstration molccules, offering end- to-end supervision that promotes free-form in- termediate reasoning.
Signal & Supervision
Introduces dense inter- mediate signals with manually designed heuristics (Xie et al., 2024; Chan al., 2024).
Injects expert guidance into the agent's work- flow through domain- specific signals.
Avoids intermediate re- wards, eliminating the need for manually de- signed heuristics.
Application Domain
Commonly used tings with sparse re- wards, such roboties and games.
Focused on solving domain-specific prob- lems by leveraging external expert sources with tool integration.
Designed for enhancing reasoning performance in chemical tasks by rowing the search space.
Overall Role
Complements demon- stration guidance by providing additional in- termediate cues.
additional strategy that can work in tandem with demonstra- tion guidance, offering expert insights.
Serves
Acts tary strategy by directly guiding LLM reasoning with demonstrative ex- amples.
complemen-
In both cases, RePO's ability to utilize sparse, final-answer demonstrations to guide complex and unconstrained reasoning is central. This approach offers a principled way to incorporate expert knowledge into the learning process without requiring costly, step-by-step annotated reasoning chains.
G FURTHER DISCUSSION
G.1 COMPREHENSIVG COMPARISON WITH RELATED WORKS
Tab. 6 summarizes where RePO differs from reward-shaping and knowledge-guided exploration strategies. We further position RePO against representative black-box optimization pipelines:
Direct prompting (e.g., MOLLEO (Wang et al., 2025b)) uses the LLM as proposal module inside an external optimizer. This improves candidate generation but does not train an end-to-end optimization policy within the LLM.
Surrogate propgaat-property fine-tume ane-tuming (e.g., LICO (Nguyen & Grover, 2025)) trains property predictors used by external Bayesian or search loops. This supports evaluation, but optimization decisions remain outside the LLM policy.
RePO directly trains the LLM as the optimizer. The model learns to generate reasoning and final molecular edits jointly under reward and reference guidance, and can still be integrated into broader optimization workflows.
G.2
ADDITIONAL BACKBONE AND BASELINE COMPARISONS
This subsection evaluates whether the observed gains persist across stronger backbones and broader baseline families, shown in Tab. 7.
G.3 TRAINING DYNAMICS CONFIRM GUIDED EXPLORATTION
Fig. 18 shows that RePO improves rewards more steadily than GRPO during property optimization. GRPO often remains in low-reward regime, whereas RePO climbs and converges higher reward
18

---

Published as a conference paper at ICLR 2026
Table Comparison of molecular optimization baselines and larger models on TOMG-Bench tasks (SR, Sim, SRxSim).
LogP
MR
QED
Models
SR
Sim.
SR Sim.
SR
Sim.
SR
SR
Sim.
SRx
Graph
0.509
0.064
0.509
0.122
0.062
0.493
0.064
REINVENT
0.465
0.125
0.058
0.595
0.116
0.069
0.558
0.115
MOLLEO
0.510
0.103
0.053
0,509
0.127
0.065
0.496
0.122
0.061
GPT-4o-mini
0.499
0.706
0.352
0.409
0.771
0.315
0.231
0.752
0.174
Mol-TS-large
0.424 0.102
0.043
0.450
0.107
0.048
0.465
0.119
0.055
Bio-T5-
0,516 0.153
0.079
0,506
0.160
0.081
0.507
0.158
0.080
Baseline (Qwen 2.5-3B Instruct)
0.268 0.627
0.168
0.252 0.685
0.173
0,188
0.693
0.130
SFT
0.298 0.692
0.206
0.359 0.663
0.238
0.297
0.697
0.207
GRPO
0.379 0.806
0.305
0.214 0.880
0.188
0.138
0.889
0.123
GRPO (SFT-Init)
0.212 0.863
0.183
0.265
0.850
0.225
0.223
0.863
0.193
RePO (Ours)
0.415
0.715
0.297
0.399
0.736
0.293
0.312
0.756
0.236
RePO ΔProp
RePO SuccessRate GRPO SuccessRate
GRPO AProp
1.0
Value
0.5
0.0
O
100
200
300
400
500
Iteration
Figure 18: Training reward dynamics of GRPO and RePO on property optimization task.
values. This supports the claim that answer-level guidance improves exploration efficiency in constrained chemical search.
G.4 DEMONSTRATION- QUALITY ABLATION
To directly test sensitivity to demonstration quality, we randomly drop the guidance loss for a subset of training examples and keep only 40% of demonstration supervision. Results in Tab. 8 show that this degraded setting remains clearly stronger than the baseline, while full RePO provides the best overall performance.
G.5 FURTHER DISCUSSION ON THE REPO'S DEMONSTRATION TERM
Methodologically. as shown in Eqn. 4, this guidance is applied only the final answer tokens, while gradient masking preserves the model's 's freedom to explore diverse intermediate reasoning paths. This targeted supervision prevents the policy collapse often seen with pure SFT, promoting a more effective and varied exploration strategy that balances guidance with retained reasoning capability.
Notably, molecular optimization presents significant challenge for I.LMS due to the vast search space, where unguided exploration often fails. Without sufficient guidance, models trained with GRPO struggle to navigate this space, leading to limited reward on property optimization. RePO addresses this by leveraging demonstrations to constrain the search to promising regions.
Experimentally, the demonstration guidance prompts the model's exploration capability. We sampled 100 responses per query from RePO, GRPO, and SFT models. RePO generated higher average number of unique molecular structures (e.g.. 34 unique valid molecules vs. for GRPO and 64 for SFT optimizing the molecular logP properties), indicating broader exploration. These
19

---

Published as a conference paper at ICLR 2026
Table 8:
Ablation on demonstraation-qualiyy robustness in TOMG-Bench property optimization
(reported in Sim).
Objective
Bascline
40% demo guidance
RePO
QED
0.130
0.215
0.236
LogP
0.168
0.312
0.297
MR
0.173
0.297
0.294
results demonstrate that RePO introduces guidance without sacrificing exploration, leading to more innovative solutions.
In summary, the demonstration guidance will not limit the model's exploration capability while reducing the sparse reward issues during exploration of the vast search space.
G.6 EXPLAIN THE EXPLORATION TERM
The core idea of this term is not to promote directionless exploration. Notably, its purpose is make exploration more efficient and effective within the vast and sparse chemical space. As we discuss in Sec. 3, purely RI.-based exploration struggles to find rewarding solutions in such a complex domain, leading to inefficient learning.
The "demonstration
waapsartrttooooinnneeee" guidance" term addresses this by using expert knowledge (the demonstrated molecule a:) to steer the policy's search process. Specifically,
Focusing the Search: By maximizing the likelihood of the expert-providee answer a, conditioned on the model's is self- self-generated reasoning steps 4ú we encourage the model to explore pathways that lead to known good solutions. This prevents the policy from wasting resources exploring chemically invalid unpromising regions of the search space.
Structuring the Exploration: The model still free to explore different reasoning paths (t4) to arrive the solution. The guidance is applied only the final answer. This allows the model to learn diverse and valid reasoning strategies while ensuring the exploration is anchored to a meaningful and high-quality outcome.
G.7 EXPLICIT CHEMICAL VALIDITY CONSTRAINTS
We ground our evaluation in state-of-the-art, validated computational tools, ensuring our methodology is robust and reliable. Specifically:
Graph-Based Prediction for Complex Properties: For challenging properties like BBBP, we follow the MuMOInstruct benchmark protocol, which uses ADMET-AL Crucially, this platform employs Chemprop (Yang et al., 2019), peessageppsssnng graph neural network (GNN). While the input an SMILES string, Chemprop internally converts into molecular graph. This allows model to learn from the molecule's 's topological structure, capturing connectivity and relationships that linear string cannot. This GNN-based approach has demonstrated leading performance on 22 ADMET tasks from the rigorous Therapeutics Data Commons (TDC) benchmark (Huang et al., 2021), confirming its reliability.
Standardized Tools for Physicochemical Properties: For other well-defined propertics such as QED and LogP, we use RDKit, the widely validated toolkit in cheminformatics.
In summary, while model generates SMILES strings, the evaluation of these molecules relies on more sophisticated graph-based neural networks and established cheminformatics libraries. This ensures that the "ground truth" used for reward and evaluation is reliable current computational methods permit, and it is a standard practice in the field.
H RELATED WORK
LLM reasoning. Chain-of-thought prompting (Wei et al., 2022) and its extensions (Yao al., 2023; Wang et al., 2023b) demonstrate that eliciting intermediate reasoning steps substantially improves LLM performance complex tasks. Recent studies further examine the robustness
20

---

Published as a conference paper at ICLR 2026
of such reasoning under noisy or incomplete conditions (Zhou et al., 2024; 2025a), and propose visualization tools for analyzing the reasoning process (Zhou et al., 2025b). Multi-role and script- based generation frameworks (Shang et al., 2026) also highlight the growing diversity of structured generation paradigms L.MMs. Our work builds on these insights by studying how reasoning emerges or fails emerge when L.MMs applied constrained scientific optimization.
Reinforcement learning for LLM post-training. RLVR methods such as GRPO (Shao et al., 2024) and PPO (Schulman et al., 2017) have been widely adopted to improve LLM reasoning through reward-based optimization (Guo et al., 2025; Zhang et al., 2025b). Exploration efficiency remains a central challenge: recent work proposes exploration-enhamee and traeectory-leve balancing (Bartoldson et al.. 2025) to address sparse-reward settings. RePO contributes this line by introducing answer-level reference guidance that alleviates reward sparsity without requiring process-level supervision.
objectives (Chen et al., 2025)
LLMs for molecular design. Molecular optimization has been approached through genetic algo- rithms (Jensen, 2019; Nigam et al., 2022), reinforcement learning molecular graphs (Olivecrona et al., 2017; Fu et al., 2022), diffusion models (Hoogeboom et al., 2022; Wang et al., 2025c), and flow-based generative models (Bengio et al., 2021; Zhu et al., 2023). Graph neural networks have also advanced molecular representation learning, enabling long-range interaction propagation (Li et al., 2024c) and fast molecular docking (Zhang et al., 2025a). More recently. LLM-based approaches integrate language models into optimization pipelines via prompting (Wang et al., 2025b), surrogate scoring (Nguyen & Grover, 2025), tool augmentation (Bran et al., 2023; M. Bran al., 2024). RePO differs by training the LLM as an end-to-end optimizer that jointly generates reasoning and molecular edits under reward and reference guidance.
LLM capabilities and robustness. The breadth of tasks that LLMs can handle continues to expand, from out-of-distribution detection (Cao et al., 2024) and noisy test-time adaptation in vision-language models (Cao et al., 2025) to agentic tasks under model compression (Dong et al., 2025) and adversarial jailbreaking (Li al., 2023). These studies underscore both the versatility and the fragility of LLM capabilities under distribution shift or capacity constraints (Han et al., 2025), themes that are echoed in the cooeeclarroptimizzzn setting studied here, where the policy must generalize across diverse property objectives and unseen instruction styles.
I EXPERIMENT SETTINGS
In this section, we provide the detailed experimental settings for all the experiments.
1.1 PHARMACOLOGICAL METRICS.
We employ the following pharmacological metrics for the molecular optimization tasks:
QED (Quantitative Estimation of Drug- likeness)ess) (Bickerton et al., 2012): QED provides a composite score that quantifies the drug-likennss of a molecule by integrating multiple molecular properties, such as molecular weight, logP, topological polar surface area, counts of hydrogen bond donors and acceptors, aromatic rings, rotatable bonds, and the presence undesirable chemical functionalities.
LogP (lipophilicity) (Lipinski et al., 1997): LogP quantifies the lipophilicity of compound, reflecting its tendency partition into non-polat (lipid-like) versus polar (aqueous) environments. Higher LogP values indicate greater solubility in non-polar solvents, which is relevant for drug absorption.
plogP (penalized logP) denotes the logP penalized by the ring size and synthetic accessibility.
MR (molar refractivity) (Le Fevre, 1965): MR is physicochemical descriptor that quantifies molecular size and polarizability, both of which are critical for modeling molecular interactions with biological targets and membranes.
BBBP (blood-brain barrier permeability) (Wu et al., 2023): BBBP quantifies molecule's ability to permeate the blood-brad brain barrier (BBB), a selective interface that regulates molecular exchange between the systemic circulation and the central nervous system. The BBB is formed by specialized endothelial cells with tight junctions, minimal vesicular transport, and absence of fenestrations, collectively restricting passive diffusion of most compounds.
21

---

Published as a conference paper at ICLR 2026
Table 9: System prompt adopted for training. Task will be replaced with the specific molecular optimization task.
A conversation between User and Assistant. The user asks a question, and the Assistant solves it. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> and </think> <answer> </answer> tags, respectively, i.e., <think> reasoning process here </think> <answer> answer here < </answer>, User: Task. Assistant:
This barrier protects neural tissue from toxins and maintains brain homeostasis, but also limits drug delivery to the brain. BBB permeability is modulated by interactions among endothelial cells, astrocytes, pericytes, and the extracellular matrix, which together constitute the neurovascular unit.
Mut ag (mutagenicity) (Sundar et al., 2018): Mutag refers to the induction of permanent transmis- sible changes in the amount or structure of the genetic material of cells or organisms.
DRD2 (dopamine receptor D2 binding affinity) (Fan et al., 2020): DRD2 measures the binding affinity of molecule to the D2 subtype of dopamine receptors, which are C-protein-coupled receptors primarily located in brain regions such as the striatum, nucleus accumbens, and prefrontal cortex. These receptors are central to regulating reward, motivation, and motor control. Higher DRD2 affinity indicates stronger ligand-receptor binding, which can modulate dopaminergic signaling and is relevant for the treatment of neurological disorders such Parkinson's 's disease.
1.2 DATASETS.
We detailed the dataset used in the experiments, including the construction of the dataset and the training splits.
TOMG-Bench is derived from Zinc-250K (Sterling & Irwin, 2015) and PubChem (Kim et al., 2019), comprising two task categories: structure-based and single-property optimization. In structure-based tasks, the LLM is instructed to operate on specific functional groups within molecules. Single-property optimization tasks require the LLM to modify molecules to enhance target properties such QED (Bickerton et al., 2012) (drug-likeness, LogP (Lipinski et al., 1997) (lipophilicity), and MR (Le Fevre, 1965) (molecular size and polarizability).
MuMOInstruct is a multi-objective molecular optimization benchmark designed to reflect the complexity of real-world drug discovery. Derived from Zinc-250K, it requires models to optimize multiple molecular properties concurrently. thereby increasing task difficulty. It incorporates both seen and unseen instruction styles to evaluate the model's instruction-follovie following robustness. The benchmark covers five critical pharmaceutical properties: ploge (lipophilicity, balancing permeability, solubility, and metabolic stability; higher is better), QED (drug-likeness)) BRBP (blood-brain barrier permeability, relevant for central nervous system drugs), Mutag (mutagenicity. where lower values indicate reduced toxicity), ) and DRD2 (dopamine receptor D2 binding affinity, with higher values indicating greater specificity).
For TOMG-Bench, we utilize the light training split, comprising 1,500 samples (500 per subtask for both structure- based and property- based optimization). The full TOMG-Bench test set is used for evaluation. To ensure comparability in training data volume, we randomly select 500 samples from MuMOInstruct for training, which resulting 1500 samples for training. All the training samples only contain the instruction and the target molecule, without any intermediate reasoning process.
1.3 TRAINING CONFIGURATIONS
Supervised Fine-Tuning, We configure the training process as follows. We employ the Llama- Factory (Zheng et al., 2024) to SFT the model. All the SFT models are trained using two A100 GPU. Each device processes a batch size 2, and gradients are accumulated over 2 steps before an update. The learning rate is set to 1.0 x 10 and optimized using cosine scheduler, with warmup ratio of 0.05 to stabilize early training. The model is trained for epochs using BF16 precision on the training split of TOMG-Bench and I epoch the training split of MuMOInstruct.
22

---

Published as a conference paper at ICLR 2026
We carefully followed the commonly adopted Llama-Factory (Zheng al., 2024) SFT recipe. We also performed an extensive pappmm-r sweep (e.g., learning rate = le-5, 3e-5 and warm-up ratio = 0.05, 0.1), but could not recover reasoning. Notably, our SFT stage observes only question- answer pairs without any chain-of-though Such data encourages the model to exploit the shortcut "jump straight the answer," permanently shrinking its output length (Lobo et al., 2025). Once this preference is instilled, subsequent GRPO fine-tuning fails to restore reasoning depth.
Reinforcement Learning. We utilize the Transformer Reinforcement Learning (TRL) library (von Werra et al., 2020) for model training. All reinforcement learning approaches, including GRPO, GRPO (SFT init), and RePO, are trained using a unified system prompt (see Tab. 9), consistent with the DeepSeek-RR RI protocol (Guo et al.. 2025). Unless otherwise specified, we adopt the default TRL. hyperparameters, with the following exceptions: the learning rate is set to 5.0 x 10-6, and the maximum prompt length is limited to 256 tokens. We use group size of 4 per input and a maximum completion length of 1024 tokens. Training conducted for epochs with a per-device batch size of 2 for training and for evaluation. To ensure reproducibility, we fix the random seed to 42 and apply a warmup ratio of 0.15. Model generation is performed on a single GPU, hosted by vLLM (Kwon et al., 2023), while two additional GPUs are allocated for training.
Evaluation. We employ vLLM to host the model to accelerate the generation process. For all generation tasks, set the temperature to 0.75 and top to 0.85 to balance diversity and relevance in the generated outputs. We use single beam (num beams = 1) and limit the maximum number of new tokens to 512. These hyperparameters generation quality across experiments.
chosen to ensure consistent and controlled
Obtaining CoT examples for prompt-based basclines. For zero-shot CoT, we follow (Kojima et al., 2022) by appending the phrase "Let's think step by step" to each question. To create our few-shot CoT setup. we first apply this zero-shot prompt to queries from the training set to generate full reasoning traces. We then select a handful of high-quality Q&A wer--withrressonng pairs from those outputs and prepend them as demonstrations to new questions, enabling in-context few-shot learning.
Licenses. The MuMOInstruct dataset is released under the MIT License. Qwen-2.5-3B -ww3--2.5-B-Innttuucc is distributed under the Qwen Research License Agreement. vLLM, TRL, and Llama- canna-acttrry are all licensed under Apache 2.0.
J FULL EXPERIMENT RESULTS AND FURTHER ANALYSIS
In this section, we provide and analyze the full results of all the experiments. Notably, for TOMG- Bench, we analyze the full results for the structure-based optimization tasks and the property-based optimization tasks in Tab. 1. For MuMOInstruct, we analyze the full results for the seen instruction and the unseen instruction in Tab. 2. We also provide a discussion on the infcasibility of GRPO with SFT initialization on the multi-objective tasks in Appendix J.3. Finally, we conduct the empirical analysis on the performance of domain-speciff LMs in Appendix J.4.
J.1 PROPERTY OPTIMIZATION ACROSS RANDOM SEEDS
In this part, we present the performance of RePO and other baseline approaches in Tab. 10, evaluating these methods over multiple random seeds, and reporting the mean performance to show their performance stability.
J.2 MOLECULAR OPTIMIZATION PERFORMANCE
Single-objective optimization tasks. Table 1 presents the performance of each model the structure-based and property-based tasks of TOMG-Bende Performance is measured using Success Rate (SR) and molecular Similarity. Several key patterns are observed:
RePO consistently achieves a strong trade-off between SR and molecular similarity across tasks. In the AddComponent task (Table 1), RePO attains an SR of 0.307 and a similarity of 0.778. In QED optimization (Table 1), it leads with SR of 0.312 and similarity of 0.756. These results underscore RePO's capacity to generate molecules that are both successful in meeting task objectives and structurally faithful to the input.
23

---

Published as a conference paper at ICLR 2026
Table 10: TOMG-Bench property optimization over multiple random seeds. We report mean std for SR and Sim, and SR Xim for overall quality.
Objective
Metric
Baseline
Distill-SFT
SFT
GRPO
GRPO (SFT init)
RePO
SR
00.269 1 0.022 0.246 1 0.011 0.323 I 0.026 0.179 I 0.174
0.183 1 0.026
0.381 1 0.029
LogP
Sim
0.628 + 0.016 0.553 + 0.023 0.700 + 0. 008 0.817 + 0.025
0.897 = 0.030
0.734 +: 0.016
S Sim
0.169
0.136
0.226
0.146
0.164
0,280
0.196
0.025
0.220
0.010
0.317 0.023
0.087
0.044
0.169 0.048
0.264 = 0.042
QED
Sim
0.666
0.039 0.585 0.012 0.701 0.004 0.905 + 0.015
0.893 0.026
0.797 + 0.035
S Sim
0.131
0.129
0.222
0.079
0.151
0.210
SK
0.244
0.007
0.225
0.016
0.391 i 0.023
0.098 1 0.012
0.158
= 0.005
0.384 1 0.008
MR
Sim
0.687 1 0.004 0.590 1 0.026 0.679 1 0.008
0.880 1 0.016
0.902 1 0.004
0.776 1 0.005
SRx
0.167
0.132
0.266
0.086
0.142
0.298
Table II: Comparison of different methods on TOMG-Bench target on structural and property optimization. The best results for each task are bolded, and the second-best is underlined.
Task type
Objective (1)
BioTS-base
MolTS-large
Baseline
SFT
GRPO
GRPO (SFT init)
RePO
Structure- based optimization
AddComponent
0.054
0.031
0.065
0.147
0.005
0.156
0.239
DelComponent
0.027
0.027
0.092
0.154
0.008
0.176
0.140
SubC CobCometeett
0.011
0.016
0,047
0.264
0.052
0.300
0.344
QED LogP 0.079
0.080
0.055
0.130
0.207
0.123
0.193
0.236
Property
optimization
0.043
0.168
0.206
0.305
0.183
0.297
MR
0.081
0.048
0.173
0.238
0.188
0.225
0.294
SFT improves SR but sacrifices similarity. Supervised Fine-Tuning (SFT) markedly increases SR relative the baseline (e.g., from 0.057 0.366 for SubComponent in Table 1), but this improvement often comes at the expense of molecular similarity, which remains lower than that of RePO (e.g., SFT similarity of 0.721 vs. RePO's 0.802 for SubComponent).
GRPO with SFT initialization can achieve competitive SRs in certain cases (e.g.. 0.420 for Sub- Component), but its similarity is less consistent (0.713 for SubComponent). Distill-SFT generally underperforms SFT both SR and similarity.
GRPO without SFT init preserves similarity but has low SR. GRPO without SFT initialization adopts a conservative modification strategy, frequently yielding the highest similarity scores across tasks (e.g., 00.97 in structure-basee tasks in Table 1).
However, this preservation of structural integrity results in very low SRs for most structure- based tasks (e.g., 0.005 for AddComponent), GRPO does exhibit task-specific strengths, such as a high SR of 0.379 for LogP optimization.
Multi-objective optimization tasks. Table 2 presents the performance of each model the MuMOInstruct benchmark, evaluating both instructions encountered during training and those not seen previously. Performance is measured using Success Rate (SR) and molecular Similarity. The results reveal several key patterns:
Clear trade-off exhibit hetween SR and Similarity is apparent across methods. Notably, SFT often yields high SR, particularly on seen instructions (e.g.. SR of 0.398 for BDP in Table 2), but typically results in lower molecular similarity (e.g.. SFT Similarity scores are 0.254, 0.279, 0.244, while RePO's 's are 0.569, 0.365, 0.509). This suggests that SFT can aggressively modify molecules meet property targets, sometimes at the expense of significant structural deviation.
RePO consistently demonstrates more balanced performance profile. While its standalone SR might occasionally be surpassed by SFT (e.g., the SR for SFT on BDQ with seen instruction is 0.319 vs RePO's 0. 160 in Table 2), RePO generally maintains higher similarity scores than SFT (compare RePO and SFT in Table 2). This ability to achieve competitive SR while preserving structural similarity contributes to its strong performance in the combined metric (SR x Similarity) reported in Table 2.
The GRPO variants exhibit distinct behaviors. GRPO without SFT initialization tends to preserve molecular structure effectively, achieving high similarity scores (e.g., GRPO Similarity for BDP seen is 0.759 in Table 2). However, its SR can be variable (e.g., SR 0. 156 for BDP with seen instruction 0.082 for BDQ with seen instruction in Table 2). Conversely, GRPO initialized
24

---

Published as a conference paper at ICLR 2026
with SFT performs poorly the MuMOInstruct benchmark, with notably low SR and often low similarity, leading to very low scores (e.g., 0.012 for BDP with seen instruction).
J.3 GRPO WITH SET INITIALIZATION CANNOT GENERATE READABLE OUTPUTS
While GRPO with SFT initialization demonstrates noteworthy performance on single-objective tasks (as detailed in Tab. 1), its efficacy significantly diminishes the more complex multi-objective tasks within the MuMOInstruct benchmark. The combined SR x Similarity scores presented in Tab, 2 for GRPO (SFT init) are markedly low across all evaluated multi-objectivv settings.
This quantitative underperformance aligns with qualitative observations of problematic generation behavior, such as those illustrated in Section K.2, where the model may produce multiple, unreasoned molecular outputs or invalid SMILES strings. These issues suggest that while SFT initialization can be beneficial for simpler tasks, it may hinder the model's reasoning ability to effectively navigate the chemical space of multi-objective molecular optimization, leading to a failure to generate both valid and high- quality solutions.
J.4 COMPARISON WITH DOMAIN-SPECIHI LMS
We report the SR x Similarity scores for BioTS-base (Pei et al., 2023) and MolTS-large (Edwards al., 2022) provided in (Li al., 2024a). BioT5 leverages biochemical text enhance both molecular understanding and generation, while MollS-large utilizes large-scale pretraining to improve SMILES generation from textual descriptions. We report the results in Tab. 11.
Notably, the results demonstrate that fine-tuned generalist language models can perform competitively, and often surpass, domain-specific models in molecular optimization tasks. Notably, RePO consis- tently outperforms both BioT5- and MolT5-large across all evaluated objectives. For example, in QED optimization, RePO achieves score of 0.236, substantially higher than BioT5-base (0.080) and MolT5-large (0.055). Moreover, the baseline generalist L.M,, without additional task-specific fine-tuning, often matches or exceeds the performance of domain-specific models (e.g., Baseline LogP score of 0.168 vs. 0.079 for BioTS-base and 0.043 for MolTS-large).
These findings suggest that general-purper LLMs, when adapted with RePO, are highly effective for molecular optimization and can match or outperform models pre-trained specifically biomedical and chemical corpora.
J.5 THE DEMONSTRATION DATA GENERATION PROCESS
The training sets of both datasets are constructed in an instruction-tuning style, where each query is paired with ground-truth optimized molecule as the answer. We use these ground-truth molecules as demonstrations, These demonstrations satisfy task constraints by design and are filtered through standard quality-comtre pipelines in the benchmark construction process.
For TOMG-Bench, seed molecules come from PubChem for training and ZINC-250K for testing; molecular edits are generated with RDKit and validated for chemical validity and target-property computation. For MuMOInstruct, training pairs are built from molecules that satisfy both structural- similarity and multi--roopert- cooperty-improveement constraints, with canonicalized and deduplicated SMILES and benchmark-standard property tools.
J.6 THE DESIGN OF THE REWARD FUNCTIONS
Following TOMG-Bench and MuMOInstruct protocols, Tanimoto similarity ECFP4 finger- prints for structural preservation and combine with property reward. This choice ensures direct comparability with prior benchmark results while preserving reproducibility of reward computation.
We also evaluate continuous property rewards, but the binary property reward is more stable in our setting and produces stronger overall optimization trade-offs.
25

---

Published as a conference paper at ICLR 2026
K
CASE STUDY
In this section, we provide case studies of different approaches to both single-ohjective and multi- objective optimization tasks.
K.1 CASE STUDIES ON
OPTIMIZATION
In this part, we present the case studies of different approaches to single-objective optimization tasks. In general, for the single-objective optimization task, the baseline model lacks detailed domain knowledge. SFT and GRPO (SFT init) offer simple modifications without elucidating the underlying chemical rationale, potentially sacrificing optimization opportunities for directness. GRPO, when unguided, is prone to errors in chemical understanding and reasoning.
Case study of Baseline model (Qwen- 2.5ven225-3B--nstnuee ---nsruuccttt Prompt: Modify the molecule CcCeeNCCODECCiCCC-ONNCCCCCCcCCC have lower MR. LLM Output:
I'm sorry, but seems there might be some confusion in your request.
"MR" can refer to different things depending on the context (c.g., molecular formula, molar ratio, mass-to-charge ratio, etc.).
To modify chemical structure, such the given propane sulfonate derivative, typically perform chemical modifications that aim to modify the functionality, functional groups, or introduce new ones. However, without more specific instructions or the chemical context, it's challenging to propose a clear and direct modification that would result in a lower MR (assuming it refers to the Molecular Formula or molar of the compound).
you could provide more details specify what "lower MR" means in your case, would be better able to assist you with specifie modification.
Alternatively, if you meant to work on the structural formula of the compound, can help you customize it or identify potential modifications.
Casc study of SFT.
Prompt: Modify the molecule CCC.CCCCCCCCDDCCCCCCCCCOCCCCCCCC have lower MR. LIM Output:
Case study of GRPO (SFT init).
Prompt: Modify the molccule CclccnCtttepppoooCCCCCCCCCCCCCCC to have a lower MR.
LLM Output:
Case study of GRPO.
Prompt: Modify the molecule oateuuucccccceccoooooooooooooooo LLM Output:
have a lower MR.
To modify the molecule Cc CCLeC(CCCSDDpCCCCCCCCOOOCCCCCCC
to have lower MR value, we need to consider the structure of the molecule and the impact of the structural changes the molecular properties.
The original molccule has heterocyclic ring with a nitrogen atom (N) and carbonyl group (C=O). CCIN+/CC ...
The structure can be represented as follows:
The MR or Magic Angle Spinning value is determined by the molecular structure and the presence of functional groups the molecule. To lower the MR value, can remove the nitrogen (N) from the heterocyclic ring,
absence of the nitrogen atom can lead a MR value.
The modified structure with the nitrogen atom removed can be represented as follows:
CC[red/CC
The removal of the nitrogen atom can lead significant decrease in the MR value, making the modified molecule suitable for certain applications. summary, lower the value molecule, we can remove nitrogen atom from the heterocyclic ring.
26

---

Published as a conference paper at ICLR 2026
Case study of GRPO.
Prompt: Modify the molecule Inte5ss0DeeeNNccccoocccxeee2e2ee to have a lower
QED value.
LLM Output:
To solve the problem of modifying the molecule ClceSsspe--ONNCCCCoceeeeeeeeeee
into
has lower QEd rage here, we Hungrily need to understand and where knowledge on the Noodle
Qedes continuously based aren't 't interesting. push away here. here.Open thinking about here: marketing process HereSTANCE: Take ornaments that thinkout more coffeesynopsis.
Case study of GRPO.
Prompt: Optimize the molecule BCCCCCCHCCCCCcceeeeeCCCCCCCCCCCC have a lower LogP value.
LLM Output:
To optimize the molecule CCC.CC/CIIICCCC///00000CCCCCC0/0
to have lower LogP value, a viable strategy to explore possible structural modifications that reduce, or at least control, the hydrophobic interactions and/or hydrogen bonding capacity. Here are few potential strategies: ### 1. Substitution of Hydrocarbon Groups Replacing large, extended hydrocarbon groups may help in reducing LogP. These groups more likely to be hydrophobic, contributing high LogP scores. Let's replace a large portion with small groups
Schink>Teet replace the largest tetrahedral hydrocarbon framework (Cecce) with small substituent groups, for example, alkyl (C1H2) or methylene groups (CH2). </thinks The optimized structure would look like this: CICCCCI SiH to CH CHICCCCCccc((C((O)sccccClleClll
molecule optimiza-
tion fails here might be.
Inverting the structure by partially breaking bonds and generating new substituents (for example, polar- izable methyl groups) can help in this optimization. < answer>> Think about the problem in light of juxtaposition elements and how change reduce electrostatic force and stronger manifold interactions. The adjust operation also requires considering position position-binary differentiability. More complex replace- ments might be needed, pinpointing conflicting geometries.
In view this consideration specific application context highlighted strategies, below concrete substitu- tion instantiated row <cl> basin release predicts diverse increment solutions. enhancement. < </di v> divę containing obstacles symmetrical divisionive representative adhering. applications control. itative adjustment will be the<br>>
focus of optimizing the LogP: substitution poocedure</divv> //vi>> to avert or alleviate indicators deferent capabilities rhythm alternations. </div>
list generated estimated row traversal applied-</di divivv v> elements undoubtedly symbolical engagement acting< / v> .Adit limited <br> proximity rationale produce snapshots evaluate< /div>> struc- tural findings amalgamation.
27

---

Published as a conference paper at ICLR 2026
Casc study of RePO.
Prompt: Modify the molecule CllcccC((GFDDCCOO00000000CCCCCCC have a lower MR. LLM Output:
to achieve a lower Magnetization Transfer Ratio (MTR) value, we need to understand the impact of different chemical modifications on the MR value.
determine how to modify molecule CcLeeNCeEDDCOooCCCCCCOCCCCCCC
MTR responsible for the transfer of magnetization between molecules, and such transfer be influenced by several aspects of the molecule's ' structure, including steric hindrance, electronegativity, and polarizability.
First, we need to consider the structure of the molecule:
Br can be broken down into carbonyl group attached benzene ring, which is then attached nitrogen atom, and connected to bromine atom. The presence the bromine group can introduce steric hindrance and create a polarizable environment, which could impact the MR value. To decrease the MR value, we might consider removing any bulky electronegative substituents, which would reduce the steric hindrance and permit more efficient interactions resonance structures contributing to higher straightforward approach do this to modify bromine atom to more flexible or less electronegative substituent, such as hydrogen (H) or another non non-polar atom or a less electronegative halogen like chlorine (CI).
<think> Cel (OCCNNCCC=OCCCCCCCCC))))))))
However, since chlorine is known to be more electronegative and might increase MR, might want opt for less electronegative and flexible substituent.
<answer>
(CI+ </ </answe>>
Here, chlorine is substituted for the bromine, reducing the steric hindrance and potentially lowering the MR value by changing the electronegativity and flexibility of the molecule.
However, for more definitive optimization, computational methods (like molecular dynamics simulations, quantum chemical calculations, more extensive computational search) would be necessary.
This modification shows a reasonable effect based on the goal of lowering MR value.
Case study of RePO.
Prompt: Modify the molecule GciceS5-00//o00cccooiooooo22222e to have a lower
QED value.
LIM Output:
to lower QED value, we need to remove its substituents. jthink; QED value is a quantitative measure of the complexity of chemical structure. Generally. the higher the branching and presence of functional groups in the molecule, the higher its QED value will be. Henee, the best way to decrease the QED value would be remove substituents, </think>><asser> ChicoSE=)=EONNCCO0e2ccss33 </answer>
<think> modify the molecule GiCeeUS--O--ONNNCCCCCeeeeeeeeee)
Case study of RePO.
Prompt: Optimize molecule CeCCCsiilicccceeeooeeeeeeddeeeee have lower LogP value.
LLM Output:
<think> In order to optimize the molecule in the given problem to have lower LogP value, reduce the molecular size and complexity. Moreover, smaller molecules often have lower LogP values. the current structure, the LogP value can improved by replacing the bulky IccoClll group with smaller functionality.. Chlorine can be replaced with hydrogen an alkyl group, for instance. By reducing the overall size of the molecule, we can create a more streamlined structure that favors lower LogP values at the expense of some structural stabil- </answer>
RePO, in contrast, successfully combines guided exploration with preserved reasoning capabilities, leading chemically sound analysis and a reasoned modification. This highlights RePO's ability to effectively navigate the complexities molecular optimization by leveraging demonstrations to inform its search process, resulting in more reliable and interpretable outputs.
We summarize the key findings as follows:
The baseline model fails to understand the chemical context. The baseline model fails derstand the term "MR" in the chemical context, requesting clarification. This highlights a lack of
28

---

Published as a conference paper at ICLR 2026
specialized knowledge required for targeted molecular optimization, common issue with general- purpose LLMs.
SFT provides direct answer without intermediate reasoning, Model after SFT provides direct answer by removing the bromine atom: (WccccC(((ccccCCCCCCCCOCCCCCCCCC modification is chemically valid and likely reduces MR (by removing heavy atom), the output lacks any reasoning process.
While this
GRPO (SFT init) exhibits similar behavior SFT. GRPO (SFT init) substitutes the bromine with an oxygen atom: Ccl modification without explicit reasoning. While potentially effective, it underscores Observation 3.3, indicating that GRPO may not fully recover the detailed reasoning capabilities when initialized from an SFT model that favors direct answers.
1O. Similar to SFT, this is a direct
GRPO shows flawed chemical reasoning. GRPO misinterprets "MR" as "Magic Angle Spinning," incorrectly analyzes the molecular structure (e.g., its depiction the molecule and the claim about the heterocyclic ring), and proposes chemically implausible modification (removing a nitrogen atom from the heterocyclic ring). This behavior is consistent with Observation 3.1, where GRPO, without proper guidance, struggles to navigate the chemical space and may generate suboptimal or incorrect solutions. The output shown is directly referenced in Fig. 16 (right) of Section 5.4.
RePO demonstrates more robust and chemically sound approach. RePO demonstrates more robust and chemically sound approach. The model correctly interprets the task in the context of MTR (Magnetization Transfer Ratio), identifies key structural features (bromine, carbonyl groups, nitrogen), and reasons about their impact on MR (steric hindrance, electronegativitty. The RePO output shown is related to Fig. 16 (left) of Section 5.4.
proposes substituting bromine with chlorine CCIcCCNNCGDDCcccccccoOOCCCCCCCOO a chemically plausible modification aimed at reducing steric hindrance and altering electronegativity to lower MR. This case study aligns with the principles of RePO outlined in Section 4, where demonstration guidance helps constrain exploration to chemically valid and promising regions.
K.2 CASE STUDY ON MULTI-OBIECTIVE OPTIMIZATION
In this part, we present the case studies of different approaches to multi-objective optimization tasks.
We summarize the key findings as follows:
Case study of Baseline model (Qwen-2.3 2.n-n-..5-B-B-e 3B-Instruct). Prompt:
Your task is to modify the given molecule to adjust specific molecular properties while keeping structural changes minimal possible. Your response should only contain valid SMILES representation of the modified molecule enclosed with <SMI SMS L,SS> tag.
Input:
<SM L,SS> CCOclce([C@ CCCCCCCCOCCCOOCCCCCCCCCO0CCCCC00
(OC)c(C)e1 </ SMSMI 188>
Adjust: increase BBB permeability, increase DRD2 inhibition and increase Penalized octanol-water partition coefficient (penalized logP)
LLM Output:
without large structural modifications.
<SMILES>
</SMILES>
The baseline model struggles with complex instructions and multiple objectives. The baseline model generates molecule by modifying the end part the SMILES string: CCOclco([C@) EIDNCC-ONCCO-CCC0-000OCCCCC C HCCT://OOOCCOCCCCCCCCCCCCCCOOCC Ilce))cccce((ncl1.... While it pro- duces a valid SMILES string. the output lacks explicit reasoning regarding how the modification addresses the three specified objectives (increasing BBB permeability, DRD2, and penalized logP). The effectiveness of the modification is unclear without analysis. This indicates a potential difficulty in handling multi-faceted optimization goals simultaneously and providing transparent reasoning.
29

---

Published as a conference paper at ICLR 2026
Case study of SFT.
Prompt: Your task is to modify the given molecule to adjust specific molecular properties while keeping structural changes minimal possible. Your response should only contain valid SMILES representation
modified molecule enclosed with <SMILES> </ </SMILES> tag.
Input: <SMILES> CCOelce([C@)) CXCCCC/CCCCCCCCCCCCCCCOOOCCOCCCO
(OC)c(C)el </SMI
Adjust: increase BBB permeability, increase DRD2 inhibition and increase Penalized octanol-wart partition coefficient (penalized logP)
LLM Output:
without increasing liver toxicity.
Case study of GRPO (SFT init).
Prompt: Your task is to modify the given molecule to adjust specific molecular properties while keeping structural changes as minimal possible. Your response should only contain valid SMILES representation the modified molecule enclosed with <SMILES> </ </SMILES> tag.
Input: <SMILES> CCOelce([C@ CXCGCICCCNNNCCCC00000000000000C0 COCGG NoONNC (OC)c(C)el </SMI
Adjust: increase BBB permeability, increase DRD2 inhibition and increase Penalized actanol-water
partition coefficient (penalized logP)
LLM Output:
without increasing hydrophobicity.
SFT generates multiple candidates without clear selection criteria reasoning for multi- objective tasks. The SFT model outputs a list of different SMILES strings, such as BoooICCCCHNNNCCOONCCCCC----
HCCCCC/CCCCCCCCCCCCCCCCCCCCCCCC HJONCICCNCC)CC and others involving substitutions like NICCN(Ce2cecc2)CCC) It prefaces its output with "without increasing liver toxicity," which was not part of the original prompt. suggesting potential misalignment or hallucination. Notably, the SFT model does not provide any reasoning for how the modifications address the objectives of increasing BBB permeability, DRD2 inhibition, and penalized logP.
GRPO (SFT init) provides a single modification but lacks detailed rationale for multi- objective trade-offs. The GRPO (SFT init) model suggests substituting part of the molecule to ...N//NC/c CtNNC///eccccBiiiiiiic ccccBr) While it offers a single, concrete modification, the output is repetitive and lacks the step-by-step reasoning required to understand how this change is intended to concurrently satisfy the three diverse optimization goals.
30

---

Published as a conference paper at ICLR 2026
Case study of GRPO.
Prompt: Your task is to modify the given molecule to adjust specific molecular properties while keeping structural changes minimal possible. Your response should only contain valid SMILES representation
modified molecule enclosed with <SMILES> </ </SMILES> tag.
Input: <SMILES> CCOelce([C@ CTC.A//CCCCCCCCCCCCCCCCCCCCCCCCC
(OC)c(C)el < </SM
Adjust: increase BBB permeability, increase DRD2 inhibition and increase Penalized octanol-watt
partition coefficient (penalized logP)
LLM Output:
without changing the molecular structure significantly.
<SM 1,KS> CCOc CCC..//CCCCCCCCCCCCCCCCCCCOOOOO0
<SMILES> CCCCCe/CeeiNNNNNNOCC-CCCCCCCCCCC
<SMILES> wttpa///.............CCC//CC//C.
<SMILES> CCC.CCCCCCCCCCGNNCNCCCCCCCOOOOO0
</SMILES>> </SMILES>
Note: For the given properties and constraints, no SMILES alteration will change the molecule sig- nificantly without altering its structure. Thus, the SMILES structure remains the same. However, for illustration, 4 SMILES structures have heen presented.
<SMILES> TTt.:///CCCC..CCCCCCCCCCCCCCCCC- (ONNS-ECUCcccC0000CC0/////1111>)
</SMILES>
<SM LBM>> O0WA0//cccci00000a//////a100000.
</SMILES>
<SMILES> (at.ssaccccccaaaaacoaoo/oCCCCCCo
(ONNN-Cccccc00000004444111112
</SMILES>>
<SMILES>
CCDetcc((CHHINNCCOOMMCC00"""""""
Case study of RePO.
Prompt: Your task is to modify the given molecule to adjust specific molccular properties while keeping structural changes as minimal as possible. Your response should only contain valid SMILES representation of the modified molecule enclosed with <SMILES> < SMILES> tag.
Input: <SMILES> CCOclec[[C CAC.CCCCCCCCCCCCCCCCCCCCCCCCCCCC INNNN (OC)c(C)e1 </SMILES>
Adjust: increase BBB permeability, increase DRD2 inhibition and increase Penalized octanol-water
partition coefficient (penalized logP)
LLM Output:
without increasing lipophilicity (logP)
Adjusted SMILES:
<SMILES> wtT.:////w.....C.CC/////CCCCCCCC
</ SMILES>>
Explanation:
The removal of the hydroxyl (-O-) group from one the amide nitrogens (from CCCO to CCCN) increases the octanol-water partition cocfficient, which is heneficial for BBB permeability and penalized logP. The change an alkyl side chain (CCCNCICCOc2ce(CII(CC15 also contribute to this effect.
Penalized LogP Increase:
DRD2 Inhibition: No specific alterations are made to scaffold that directly affect DRD2 inhibition. Lipophilicity (logP) Stability: The removal of the hydroxyl group does not increase logP significantly, which maintains lipophilicity while improving the desired properties.
GRPO fails to generate meaningful modifications in the multi-objective setting. When presented with this task, GRPO repeatedly outputs the original molecule, stating: "For the given properties and constraints. no SMILES alteration will change the molecule significantly without altering its structure. Thus, the SMILES structure remains the same." This behavior indicates that GRPO is unable to effectively engage with the optimization objective, likely due limitations in its reward structure or an excessive preference for minimal structural changes. As highlighted in Observation 3.1, GRPO
31

---

Published as a conference paper at ICLR 2026
struggle to explore chemical space without explicit guidance, often resulting in conservative outputs, particularly in complex multi-objective scenarios.
RePO exhibits systematic reasoning and targeted molecular modification for multi-objective optimization. In contrast, RePO proposes modified molecule, CWC.CCCCCCCNGEOONCCCCCCOCOOOOOOO COCCCCCCCC cccCCCCCCCecCC and provides a clear rationale for its design. The model explains its chemical modifications complete sentences. For example, states that the dichlorination of the terminal phenyl ring is intended to influence the desired properties. It also notes that the removal of the hydroxyl group from the molecule increases the octanol-water partition coefficient. This change is beneficial for both blood-brain barrier permeability and penalized logP.
Although RePO acknowledges that it did not make direct changes to improve DRD2 inhibition, demonstrates understanding of the multiple objectives and justifies its design choices accordingly. This structured and interpretable approach aligns with RePO's use of demonstration based guidance, as described in Section 4. The effectiveness of this method is also evident in RePO's superior performance on multi-objective tasks, as shown Tab. In summary, RePO better able to balance competing objectives and provide transparent and actionable outputs.
32


Reviews:
[{"review_id": "qCWp2dBNU6", "reviewer": "Reviewer_2riE", "summary": "The paper addresses a key limitation of LLMs in scientific reasoning tasks, especially molecular optimization, designing molecules with better drug-like properties while maintaining structural similarity. Existing approaches like: SFT need reasoning traces, which are absent in molecule datasets, RLVR (Reinforcement Learning with Verifiable Rewards, e.g., GRPO) performs poorly due to sparse rewards and unguided exploration. Thus, the paper introduces DePO (Demonstration-guided Policy Optimization), a method that combines the benefits of RL and supervision using reference molecules as demonstrations to guide the optimization process.\n\nThe core idea is to guide the policy optimization process with demonstration molecules that serve as structural anchors, balancing exploration (via RL) and exploitation (via supervision). The task evaluated is molecular optimization i.e., given an input molecule (in SMILES form) and instruction (e.g., “make it more drug-like”), an optimized molecule needs to be generated. The training objective (Eq. 4) combines three components: RL term (Encouraging Exploration): Similar to GRPO, updates policy toward higher rewards, supervised term (Guiding Exploration): Encourages the model to generate solutions similar to demonstration molecules, regularization term: KL divergence to keep policy updates stable. Gradient masking ensures that only final answers (not reasoning tokens) are supervised, preserving reasoning ability. Two main rewards are proposed (a) structural similarity reward (r_struct): Tanimoto coefficient between input and generated molecule; (b) property reward (r_prop): Binary reward indicating whether target property (QED, LogP, MR) improves.\n\nThe paper evaluates DePO on two benchmarks: TOMG-Bench and MuMOInstruct. The metric used is the product of Success Rate (achieving property objective) and Structural Similarity (Tanimoto score).", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper is very well-written and easy to understand. I like how the problem is first formulated and clearly defined in Section 2, followed by a discussion of specific limitations in Section 3. The proposed method is then presented clearly and succinctly in Section 4, making it easy for readers to follow and grasp the details.\n\n- DePO is an elegant fusion of RL and supervised signals, guiding exploration with domain demonstrations while maintaining reasoning depth. It demonstrates constraint search to chemically meaningful regions, overcoming limitations of GRPO.\n\n- The paper presents strong empirical results. DePO outperforms SFT and GRPO on multiple benchmarks (TOMG-Bench, MuMOInstruct) by up to 13%.", "weaknesses": "- In the DePO scheme, the LLM output is first parsed into reasoning tokens and the generated final answer. Then, the generated final answer is replaced with the gold-standard final answer. This way, the reasoning tokens are preserved, and gradient masking for the intermediate reasoning steps excludes these tokens from parameter updates during optimization. The authors claim that this approach prevents the LLM from learning potentially erroneous reasoning patterns, but it is not clear why that is the case. There is no constraint imposed on the reasoning tokens that prevents the model from generating erroneous reasoning patterns. In fact, training only occurs on tokens excluding the reasoning tokens. The authors further state that DePO \"constrains its exploration to chemically valid and promising regions of the solution space,\" but it is not evident why this is true. Moreover, what is the rationale behind replacing the generated answer with the gold-standard answer? The reward is anyway calculated based on the generated answer. The overall design seems to offer only incremental improvements over GRPO.\n\n- For the reward assessing the target property, a binary reward value of 1 is assigned if the generated molecule achieves a favorable change in the target property relative to the original molecule. This means that even a minuscule positive change results in a reward of 1. Such a design is highly susceptible to reward hacking, where the model may settle after making only a small improvement and fail to pursue further optimization, since it has already received the maximum reward of 1. \n\n- Why is the product of Success Rate (achieving the property objective) and Structural Similarity (Tanimoto score) considered a meaningful metric? Wouldn’t this metric be biased if one component is significantly high while the other is only moderate? From Section 2.1, it seems more reasonable to report these as separate metrics: (a) drug-likeness, conditioned on structural similarity being within a specified threshold (0 otherwise), and (b) structural similarity itself.\n\n- The paper only compares against different fine-tuning paradigms, namely, Baseline, SFT, GRPO, and GRPO (SFT init). However, it does not include comparisons with existing state-of-the-art methods for molecular optimization, nor with any foundation models.\n\n- Figure 2 is difficult to interpret, and the corresponding observations are not clearly conveyed. For instance, Observation 3.1 (\"Models trained with GRPO exhibit a conservative bias, generating molecules nearly identical to the input, as shown in Fig. 2 (Left)\") is not directly supported or easily interpretable from the figure. I suggest modifying it to a different type of plot that more clearly illustrates the stated observations and providing a more descriptive caption to enhance interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJo4jQmK2B", "reviewer": "Reviewer_knP9", "summary": "The paper introduces DePO a framework that enhances large language models’ reasoning ability for molecular optimization by combining reinforcement learning with expert demonstrations. Unlike prior approaches such as SFT, which suppress reasoning, or RL with verifiable rewards, which suffers from unguided exploration, DePO integrates reference molecules as supervised signals to guide the search toward chemically valid regions while preserving intermediate reasoning steps through gradient masking. Evaluated on TOMG-Bench and MuMOInstruct, DePO achieves up to 13% performance gains over strong baselines, demonstrating improved balance between property optimization and structural similarity, better generalization to unseen instructions, and chemically coherent reasoning, establishing a new paradigm for LLM-driven scientific reasoning and molecular design.", "questions": "Additional questions regarding weaknesses:\n\n1. The design and formulation of the regularization component are not clearly explained.\n\n2. Why is the gradient flow disabled during the reasoning (“thinking”) phase? Could the authors provide an ablation study to justify this choice?\n\n3. In Figure 7, there is a large discrepancy between DePO and GRPO. The result appears somewhat unreliable, as GRPO seems to lack a valid expression. Were both methods fine-tuned on the same base model? \nWithout providing the source code, it's hard to believe such difference considering on overall similarity in objective function.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper introduces a novel demonstration-guided reinforcement learning paradigm (DePO) that effectively bridges the gap between language-model reasoning and domain-constrained scientific optimization. It presents comprehensive experiments across multiple molecular optimization benchmarks, providing strong empirical evidence for the framework’s effectiveness. The significance of this work lies in extending LLM reasoning beyond text and mathematics to molecular design, demonstrating that demonstration-guided optimization can deliver substantial improvements in both performance and generalization within complex chemistry-related tasks.", "weaknesses": "1. The discussion of related work is limited. Several recent GPT-based molecular optimization studies are neither cited nor compared as baselines, making it difficult to contextualize DePO’s contributions within existing literature.\n\n2. The novelty appears somewhat incremental, as the method primarily adds guiding exploration and a regularization term to the GRPO objective. The introduced regularization is heuristic in nature, and the paper does not provide sufficient discussion on how the hyperparameters (β and γ) are selected or tuned, which may significantly influence final performance.\n\n3. Although the paper motivates DePO’s gradient-masking mechanism, it lacks theoretical justification or a detailed ablation study to assess how different masking strategies impact reasoning fidelity, convergence, and optimization stability.\n\n4. While DePO presents qualitative improvements in reasoning traces, the evaluation of reasoning quality remains largely subjective. Incorporating quantitative reasoning metrics or expert chemical validation would make the claims more convincing.\n\n5. how to create the demonstration dataset is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "in5Wv1RJ9t", "reviewer": "Reviewer_h5YJ", "summary": "A solid and well-executed paper proposing Demonstration-guided Policy Optimization (DePO) for LLM-based molecular optimization. The idea of integrating demonstration-guided supervision into RLVR is clear, novel enough for ICLR, and empirically convincing. Methodologically clean, though somewhat incremental from GRPO + SFT.", "questions": "1. How sensitive is DePO to low-quality or noisy demonstrations (e.g., suboptimal molecules)?\n\n2. Could DePO degenerate into simple imitation when β is high? How is β chosen in practice?\n\n3. Does DePO still help when using stronger base models (e.g., >7B) trained with rich CoT reasoning data?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel combination of demonstration and RL: Clear motivation and principled integration of supervised guidance into RL objective.\n\n2. Strong empirical gains: Up to 13% improvement over SFT and GRPO on TOMG-Bench and MuMOInstruct, with convincing generalization and ablations.\n\n3. Well-written and interpretable: The framework (gradient masking, demonstration term) is intuitive and illustrated clearly with chemical reasoning examples.", "weaknesses": "1. Incremental contribution: Conceptually close to existing RLHF/RLVR + imitation learning hybrids; novelty may feel limited.\n\n2. Limited scope: Only tested on molecular optimization—no evidence DePO generalizes to other scientific or reasoning domains.\n\n3. Weak theoretical insight: Lacks formal analysis of convergence or policy improvement guarantees under demonstration bias.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BuG7rxcHbh", "reviewer": "Reviewer_Aw6o", "summary": "The paper presents a new method called DePO which was modified from standard RLVR procedure via using a demonstration-guided term in the objective function. It provides a novel framework to LLM-driven molecular optimization and through experimental results, demonstrates that the method can enhance reasoning and task performance.", "questions": "Is there any disadvantage with constraining the exploration space in DePO?\nTable 1 and 2 show the overall results on on TOMG-Bench. If that is an average performance over the dataset, what is the range of errors in addition to the mean and do they vary  or stay similar across the methods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the experimental design is robust. It also provides a detailed analysis of current drawbacks on RLVR and how to address them via modification on the architecture, algorithm and reward function. The results demonstrate good improvement compared to the baseline and support the paper's aim of providing better reasoning in molecular optimisation.", "weaknesses": "The paper uses only Qwen-2.5-3B Instruct as a single backbone for the baseline and base model (and to demonstrate scaling also use Qwen-2.5-7B Instruct). The results and findings could be more robust if it has had demonstrated results using other models since the base reasoning and optimisation capacity can be different. This would be useful to determine whether DePO still offer improvement to the other methods (while using the same base model).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)