ChemOrch: Empowering LLMs with Chemical Intelligence via Groundbreaking Synthetic Instructions
Authors: Yue Huang
Zhengzhe Jiang
Xiaonan Luo
Kehan Guo
Haomin Zhuang
Yujun Zhou
Zhengqing Yuan
Xiaoqi Sun
Jules Schleinitz
Yanbo Wang
Shuhao Zhang
Mihir Surve
Nitesh V Chawla
Olaf Wiest
Xiangliang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Chemistry
Large Language Model
Synthetic Data
Primary area: machine_learning_for_sciences

Abstract: Empowering large language models (LLMs) with chemical intelligence remains a challenge due to the scarcity of high-quality, domain-specific instruction-response datasets and the misalignment of existing synthetic data generation pipelines with the inherently hierarchical and rule-governed structure of chemical information. To address this, we propose ChemOrch, a framework that synthesizes chemically grounded instruction–response pairs through a two-stage process: task-controlled instruction generation and tool-aware response construction. ChemOrch enables controllable diversity and levels of difficulty for the generated tasks and ensures response precision through tool planning \& distillation, and tool-based self-repair mechanisms. The effectiveness of ChemOrch is evaluated based on: 1)  the \textbf{high quality} of generated instruction data, demonstrating superior diversity and strong alignment with chemical constraints; 2)  the \textbf{dynamic generation of evaluation tasks} that more effectively reveal LLM weaknesses in chemistry; and 3)  the significant \textbf{improvement of LLM chemistry capabilities} when the generated instruction data are used for fine-tuning. Our work thus represents a critical step toward scalable and verifiable chemical intelligence in LLMs. The code is available at \url{https://anonymous.4open.science/r/ChemOrch-854A}.

ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions
Yue Huang's Zhengzhe Jiang', Xiaonan Luo', Kehan Guo', Haomin Zhuang', Yujun Zhou'
Zhengqing Yuan', Xiaoqi Jun", Schleinitz', Yanbo Wang', Shuhao Zhang?
Mihir Surve", Nitesh V Chawla', Olaf Wiest', Xiangliang Zhang'
Department of Computer Science and Engineering, University of Notre Dame MIIT "CalTech "MBZUAI SCMU
6 DDpparteent of Chemistry & Biochemistry, University of Notre Dame
Abstract
Empowering large language models (LLMs) with chemical intelligence remains a challenge due to the scarcity of high-quality, domain-specific instruction-respone datasets and the misalignment of existing synthetic data generation pipelines with the inherently hierarchical and rule iieegovereedd structure of chemical information. To address this, we propose ChemOrch, framework that synthesizes chemically grounded instruction-responee pairs through a two-stage process: task-controlled instruction generation and tool-aware response construction. ChemOrch enables controllable diversity and levels of difficulty for the generated tasks, and ensures response precision through tool planning & distillation, and tool-based self-repair mechanisms. The effectiveness of ChemOrch is evaluated based on: 1) the high quality of generated instruction data, demonstrating superior diversity and strong alignment with chemical constraints; 2) the reliable generation of evaluation tasks that more effectively reveal I.I.M weaknesses in chemistry; and 3) the significant improvement of L.LM chemistry capabilities when the generated instruction data are used for fine-tuning. Our work thus represents a critical step toward scalable and verifiable chemical intelligence in LLMs. The code is available at https: / / thub. com/ Howi eHwong ChemOrch.
Introduction
Large Language Models (LLMs) have exhibited exceptional capabilities across wide range of tasks, to be widely applied various downstream tasks [1-5]. Among these, chemistry represents particularly promising field where L.LMs assist in accelerating molecular design [6. 7J, facilitating scientific discovery [8, 9], and democratizing access to expert-level chemical knowledge [10, 11]. Empowering LLMs with strong chemical reasoning capabilities could significantly impact areas such as drug discovery, materials development, and organic synthesis [12, 13].
Despite this potential and existing efforts [9, 10], empowering LLMs with chemistry domain knowl- edge remains challenging. First of all, there exists Challenge 1: data scarcity in both training and testing, disclosed in recent benchmarking studies [14]: LLMs have not yet achieved the level of performance expected by chemistry scientists. High-quality instruction datasets for guiding LLMs to learn chemistry knowledge extremely limited. While chemistry-related corpora have been utilized during pretraining [4], constructing task-specific fine-tuning datasets and fine-grained evaluation sets often requires intensive expert annotation [155] leading to high costs and limited scalability.
To overcome data scarcity, synthetic data generation offers a promising alternative. However, there exists Challenge 2: mismatch between general-purposs synthetic frameworks and chemistry-
Eqqul Contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

specific requirements. Most existing instruction generation frameworks are built for general text understanding, and are fundamentally misaligned with the structured and rule-bound nature of chemical tasks [16-18]. Chemical problems often involve operations over molecular structures, require strict adherence to conservation laws or valence constraints, and demand accurate function grounding. Even minor errors, e.g., invalid atoms or incorrect stereochemistry, can lead to chemically meaningless or unreliable results, a failure mode less critical in standard NLP.
Even when domain-specific generation is attempted, there remains Challenge 3 for ensuring diversity, executability, and controllability in the data synthesis process. Effective chemical instruction datasets must span a wide range of tasks, from basic property prediction to complex retrosynthesis planning. while ensuring that generated responses are chemically valid and verifiable. Beyond diversity, maintaining executability, the ability for model outputs to conform to domain rules and withstand external verification, is crucial for scientific reliability. Furthermore, controlling the difficulty, specificity, and complexity of generated instructions is important for effective training of LLMS (enabling progressive skill development and robust understanding). but remains particularly difficult in current automated instruction generation pipelines.
To address these challenges, we propose ChemOreh, a framework for constructing synthetic instruction-resppnse pairs that enable LLMs to acquire chemistry domain knowledge. ChemOrch consists of a two-stage pipeline: (I) Task- Cost-Contollled Instruction Generation. Given a chemistry task (e.g., property prediction) along with user-defined constraints (e.g., target difficulty, required keywords) and metadata (e.g., reference files, extra tool configuration), ChemOrch enables the generation of diverse and controllable instructions (e.g., different molecules with various types of properties to predict, at varying levels of prediction difficulty). While the diversity is ensured by specifying constraints, the difficulty level is controlled by difficulty reward model with feedback, which evaluates and iteratively refines instructions to align with user-specified complexity levels. (2) Tool-Aware Response Construction. To ensure precise responses, ChemOrch leverages set of tools to ground its outputs, as certain chemical tasks are straightforward for these tools but challenging for LLMs due to their lack of domain knowledge. For example, tasks like name translation can be easily handled by chemical tools, but are difficult for LLMs to generate accurately. For the given instruction generated before, ChemOrch decomposes the associated problem into intermediate reasoning steps, retrieves and distills relevant tools, and generates code scripts to produce accurate outputs. This process includes multi-stage self-repair mechanisms and sufficiency checks, ensuring that generated responses are verifiable, executable, and faithfully satisfy the original instruction intent (e.g., see several instruction- resionss examples in Figure 15, Figure 16, Figure 17).
We conduct extensive experiments based on ChemOrch, evaluating its effectiveness across multiple dimensions, including the diversity, response quality, and constraint adherence of the generated instruction-responss pairs, and its usefulness on two important applications. First, ChemOrch serves as reliable evaluation framework that identifies LLM weaknesses, and enables scalable, task-specific assessments with high fidelity. Second, it enhances LLM performance in chemical QA and reasoning tasks, when the generated instructions are used for fine-tuning.
Overall, our contributions are threefold: 1) We introduce ChemOrch, breaking the limits of instruction data scarcity to enable LLMs to solve chemistry-related challenges more effectively. The synthetic Instruction-respons pairs are diverse topics, challenging at controllable levels, and comprehensive for covering wide range of chemical tasks with precise answers guaranteed, as validated by human experts. 2) We propose a novel synthesis framework featuring a two-stage pipeline, incorporating difficulty control, tool decomposition, and distillation, as well as self-repair mechanisms, allowing scalable and high-quality instruction response generation. 3) Importantly, we showcase the significant impact of ChemOrch on two key applications: facilitating chemistry evaluation and improving the chemistry capability of LLMs, demonstrating the effectiveness of our framework.
2 Preliminary: Harnessing Chemical Tools within ChemOrch
Leveraging chemistry tools is a core aspect of our work, ensuring that the responses in the curated instruction dataset are both accurate and reliable. Furthermore, the tool execution process itself can be utilized to evaluate and enhance LLM's proficiency using tools-an important long-term objective of our research (Our experiments presented in section 4 demonstrate that ChemOrch significantly improves I.LMs' tool usage capabilities). Before introducing the framework of ChemOrch, this section outlines how chemical tools are constructed and integrated.

---

Foundational Tool Resources
Listikp-AAss)):
Retriiee Aaagys descrite blanciivlyy experients.
Pub hee
RKI
Perahater::
2
scentitier (strt: MarRiFser Eipically ABD,
Enfeelte
unge integee).
Tool Documents
Ao Anso cbjects, Detures casty
(Exappl::
aasay, 2831 (88) ramespacelaaio ald")
Found with AID 1852. Descriptionn preeios: (masy, 283210) 2330y
S
Function Porsing
bidlogist vants to investigate suaaps related to spactfic target 14-5., procese), for lnneary investigetins
trtesal Janay with
Manual Review
tenosetration.
asse Type:
ietnttcal assayas) .asaay.yypss)) typs)")
printi Veaay wits AID not feond This AID should antet, tandle night w9rl treraries vet. AT
Sub Tool i
Sub-TTof
Suo-
return aco siogeet. sccopst idvitifter, ranespaon)
Figure I: An example of a sub-tool (left) and the tool decomposition pipeline (right). Each sub-tool includes five components: 1) high-level operation description, 2) formal argument specification, 3)
expected return values, 4) minimal working example, and 5) real-world use case.
Tools Overview. ChemOrch leverages two categories of tools: chemistry-related tools such as RDKit [19] and PubChem [20] and general-purpose tools that include web search and the Python environment for code scripting. More details about tools selection are provided in Appendix I.
Motivation. In real-world scenarios, humans typically interact with chemistry tools at the function level, selecting specific operations rather than engaging with the tool as monolithic system. Inspired by this, we aim to enhance the usability of foundational chemistry tools (e.g.. RDKit) in instruction- driven response generation by decomposing them into fine-grane -raiirr sub-tools. This decomposition serves three purposes: (1) it enables precise control by isolating individual functions, (2) it simplifies tool semantics, making them more interpretable and accessible for LLMs, and (3) it mitigates the challenges LLMs face when handling complex, multi-step operations with minimal context.
Sub-Tool Construction. We begin by feeding the official documentation of RDKit and PubChem into an LLM (i.c., GPT-4o) and applying a sew-shot learning approach to guide the extraction of function- level operations. These operations, which represent the atomic units of functionality, are identified based on their relevance to common chemistry tasks. The LLM is then prompted to synthesize these operations into callable Python code, each encapsulated as sub-tool. Every sub-tool is constructed following a schema consisting of five components, as illustrated in Figure 1. This semi-automated process significantly accelerates tool decomposition while maintaining structural consistency. To ensure correctness and usability, we subsequently perform a manual review of all generated sub-tools, with review procedures introduced in Appendix C. In total, constructed 74 sub-tools, 57 derived from RDKit and 17 from PubChem. For each sub-tool, we provide both a minimal working example and a real-world use case. The real-world use case demonstrates how the operation is applied in meaningful ful task contexts, e.g., using molecular descriptor computation sub-tool within a solubility prediction pipeline. The examples help LLMs understand how invoke a sub-tool and why and when to use it. These sub-tools are integrated with other general-purposs. tools to form tool pool, which is then used in the instruction-respenn response generation process of ChemOrch.
Extensibility. While ChemOrch currently integrates a limited set of tool types, it is designed to be extensible. A key enabler of this extensibility is the use of metadata (as shown in Figure 2), which allows users to specify additional tool configurations at runtime. For instance, users can upload custom wrappers or specify endpoints for private APIs, as shown in Appendix N. By doing this, ChemOreh is able to dynamically recognize and utilize new capabilities, enabling broader coverage of specialized chemistry tasks.
3 ChemOrch
We formalize ChemOrch framework consisting of two-stage generation pipeline that produces instruction-response pairs tailored for chemistry-relater tasks.
3

---

Difficuity Reward Model w/ Feedback
Task Name: Transition State Identification Task Description: The task identifying likely transition state structure for given chemical reaction.
Instruction Generation
Difficulty Score
Reason for easy
Reason for difficult
Response
Instruction Refinement
Difficulty
Generation
Constraint (Optional): The instruction should focus on specific chemical problems but not general problems.
Evaluation
Finol
Generated Instructions
Generated
Metadata (Optionall: CSV/SONN reference files, extra tool configuration, and necessary text.
Generote Instruction
Responses
IG Model
Tool Pool
Tool Distillation
Early Stopping Checking and Sufficiency Validation
Generated instructions
RIKE
Pub BablChen hem
Usefulness Check
Expert Extent Function Budget
python
all results instruction?
RG Model
Retrieval
Refinement
Catch Error
Cocl
Effectiveness Tool
Step 1: Colculate...
Expected Tool I:...
Tool 1: propertee
Script
Checking
DOcs.
Step 2: Determine
Expected Tool 2:
Tool 2:
Instruction Decomposition
Expected Tool Specification
Selected Tools
Code Script Generation w/ Seif-Repairng
Figure 2: ChemOrch Framework. It consists of instruction generation (IG) with difficulty evaluation, and response generation (RG) by calling tools from the tool pool.
3.1 Instruction Synthesis
Let T denote the space of chemistry-related tasks, and let represent of user-defined constraints (e.g., instruction length, complexity, keyword requirements, formatting, or few-shot examples for guiding generation) that enable more controllable generation. In addition to constraints, we also introduce a metadata set M, which includes auxiliary inputs such as reference files, specialized tool configurations, or essential textual descriptors. These metadata items serve as seed data to guide and condition the instruction generation process more precisely (e.g., by providing molecular structures as input when generating instructions for reaction prediction tasks), as shown top-left of Figure
Given a task E T, constraint e E C, and metadata m E M, the instruction generation (IG) model Mist (i.e., LLM) synthesizes an instruction I (t,isttt, c, m). The details of instruction generation are presented in Algorithm in Appendix M. For all tasks in T, by varying the constraint each running iteration alongside the metadata (check implementation details in Appendix B), the IG model generates diverse instructions X' = {
Difficulty Controlling, Previous studies have shown that controlling the complexity difficulty of L.M-ggeerated instructions remains a non-trivial challenge [21, 22]. Maintaining calibrated difficulty levels in the chemistry domain is essential for supporting progressive skill acquisition.
To address this, we introduce difficulty reward model with feedback, denoted Man; which evaluates the generated instruction z and outputs both scalar difficulty score d (from 1-5) and set of localized difficulty explanations: (d, e) = Mair((),, where e indicates which aspects of the instruction (e.g.. terminology, structure, domain scope) contribute to its simplicity or complexity.
This feedback is used to guide the instruction generation process (see Algorithm 1). Specifically, Minst receives (t, c, m, e) as input and is prompted to revise regenerate x such that the resulting difficulty better aligns with user intent or target distributions. We build Mair by applying supervised fine-tuning (SFT) on Meta-Llama-3. 1-8B-Instruc' on 3,390 annotated samples. Evaluation by human experts indicates that Mair successfully captures the nuances of chemistry instruction complexity. See more details of implementation and evaluation about Man in Appendix D.
3.2 Response Construction: Overall Procedure
Given instruction the response generation (RG) model MMrspp (i.e., an LLM) is responsible for generating an executable and chemically accurate output E y for the associated question in æ. Several examples of instruction-respons pairs presented in Figure 15 and Figure 16. To ensure factual grounding and modularity, Miesp does not directly generate free-form responses; instead, it leverages tool pool F {f11 f2..... fx), as well metadata m (the same as used in the instruction generation), to generate the responses.
Instruction Decomposition and Tool Planning. The response model first decomposes the in- struction into sequence of intermediate reasoning steps, denoted as Decompose(z)

---

($1, $2..... SL). , Each step se reflects sub-goal necessary to fulfill the instruction, such as data retrieval, property computation, or output formatting, Not all steps necessarily require tool usage.
The Mrsp model considers the entire sequence of reasoning steps and generates a set of expected tool descriptions {d1,d2 da,,,,,...ddd das) } C D, where each dm represents distinet functional capability that may be needed to support one or more steps, denoted as {dj....,dy ,d, = Mresp m).
Tool Retrieval and Distillation. To map these expected tool descriptions actual executable tools, semantic retrieval is performed over the tool pool. For each dm cosine similarity is computed between its embedding and the embeddings of available tools: Top = arg top-k cos (#(dm) (fx)).
fEF
The retrieval yields a candidate set Fraw for each expected tool description. The union of all retrieved candidates forms the raw tool pool for the response: J. = U m=
Fm FT.
Tool distillation is then performed globally using Mresp= given the instruction z; reasoning steps 8, metadata m, and candidate tools Friw The model iteratively refines the toolset by eliminating redundant ineffective tools, adhering to:
Usefulness Check: Remove tools that do not contribute actionable results for subgoals.
Expert Extent: Prefer tools that align closely with reasoning intents, minimizing auxiliary steps. Tool Budget: Enforce size limit by pruning low-utility tools.
Formally denoted as: F = Distill (JTTM, I,, s,m, T).
Tool Execution and Answer Assembly. With the distilled tool set FF the response model generates and executes scripts for each f E F*, incorporating metadata m where applicable (e.g., tool configuration guidance, molecule information). Outputs denoted of f(a,). The model then synthesizes the final response using the collected outputs: .Mresp (ar, {of) m).
We describe the detailed procedure of tool calling and validation in the following subsection.
3.3 Response Construction: Tool Calling
Code Script Generation. For each selected tool f E Fr, the model generates an executable code script S based tool specification, instruction, reasoning steps, and metadata, denoted as Sj = (f xr 8, m). where metadata m may contain pre-specified input files, configuration parameters, external resources relevant to the tool.
Script execution errors common due to issues such as incorrect input formats or API usage. We adopt a multi-stage self-repair protocol:
Self-Repairing.
Error Catching: If execution of Sy fails, the model captures the error trace e and attempts repair:
s((t1) MM, m).
This process is repeated until success or retry limit Rmax is reached.
all attempts fail, the model consults external documentation (e.g., from RDKit or PubChem) through web retrieval and regenerates the script based on the retrieved content and metadata.
Effectiveness Checking: Even successful executions are not assumed be sufficient. If the returned result does not meet user intent, e.g., missing keyword-level constraints or incorrect computation granularity, the model re-evaluates and refines the tool usage.
Early Stopping & Sufficiency Validation. After each tool execution, the model assesses whether the currently accumulated outputs {oy] already satisfy the instruction æ. If so, the pipeline stops early, skipping the execution of remaining tools to improve efficiency.
After the finish of all selected tools, Mresp will check whether the outputs for} } are incomplete or insufficient for fully answering the instruction. If yes, MMesp triggers web-based retrieval step: Oextm WebSearch(r,, m). The final output set used for response synthesis is:
= {or) U (-
The final response is constructed by composing all outputs in O, grounded in the reasoning trace and enriched with any metadata-derived context. The details of response generation shown in Algorithm in Appendix M.
5

---

Experiments
4.1 Experiment Setup
Models. We employ GPT-4o [23] as the IG model across all experiments. For response generation, we adopt a hybrid setting: GPT-4o used for general- reasoning tasks such as decomposition, validation, and web retrieval. For components requiring fine-grained decision-makinn or complex reasoning (specifically, tool distillation, code script generation, and self-repairing), we utilize the ol-mini model [24]. which demonstrates stronger reasoning capabilities. For text embedding, adopt the cext-embedding---1 3-smlll model [25]. All generation temperatures in our experiments are set to 1.0. Notably, ChemOrch is compatible with other models as IG/RG models as well; however, adopt this configuration in our experiments to ensure consistency and facilitate controlled evaluation.
Chemical Tasks in Instruction Generation. We focus on two main categories of tasks: General Chemistry Q&A and Task-Specific Challenges. The former involves answering open-ended questions within the chemistry domain (e.g.. What are the main steps involved in synthesizing aspirin in the laboratory?). which could be later used for evaluating/enhancing LLMs' chemistry knowledge and generation quality. The latter focuses domain-specific tasks commonly used in existing benchmarks, requiring more sophisticated chemical reasoning and planning capabilities. These include property prediction (PP), molecule captioning (MC), name prediction (NP), and reaction prediction (RP), among others. See task examples in Table 11 and Table 12. Morcover, we introduce an agent-oriented task (tool usage) to assess LLM's capability operating chemistry-related tools.
Due to space limitations, we refer the reader Appendix for additional details on experimental setup, including evaluation protocols, datasets used for analysis, training, and testing configurations.
4.2 Statistical Analysis of Generated Instruction
Pairs
Word Count. The distribution of generated data is shown in Figure 3. On average, each instruction contains 17.52 words, while the corresponding response is substantially longer, averaging 320.66 words, reflecting the detailed and comprehensive nature of model-ggeneatee outputs. Furthermore, each response utilizes approximately 1.24 tools on average (see examples in Figure 15, Figure 16).
Topic Diversity and Coverage. To assess the instruction dataset diversity and coverage, we have referred to the study [26, 27], and adopted two quantitative metrics: Average Pairwise Sample Similar- ity (APS) and Semote-Clique Score. APS captures the average similarity between sample instruction pairs-lower values indicate greater internal diversity. Remote-Clique identifies maximally dissimilar subset of instruction samples, with higher scores suggesting broader input space coverage.
0.005
Dataset
APS
Remote-Clique
0.004
ChemOrch (our)
0.779
0.661
Dencity
0.00
ChenLLMMennch [14]
0.884
0.453
0.01
0.002
Mol-Instructions [15]
0.765
0.683
0.02
0.001
ChemBench [28]
0.784
0.613
0.00
0.000
10
20
30
200
400
600
800
Table 1: Diversity and coverage analysis, ChemOrch vs other datasets.
Figure 3: Word-count distribution of instruc- tions (left) and responses (right).
As shown in Table datasets generated by our proposed ChemOrch achieve significantly lower APS and higher Remote-Clique CCique scores compared to ChemLLMMBench [14] and ChemBench [28], suggesting that ChemOrch offers improved diversity and better structural spread generation. Its diversity profile is comparable to that of Mol-Instructions [15], a dataset with well-curated human- annotated molccular tasks. These results indicate that data generated by ChemOrch effectively balances diversity and coverage, making it competitive and diverse instruction dataset for chemistry learning.
Generation Cost (token usage, expenses). The instruction and response generation models are both based on LLMs. We analyze the generation cost in terms of token usage. Figure 4 shows the average token usage per generation, in different modules of ChemOrch. Tool Selection dominates the token usage (4094 tokens), followed by Validation (1472), Answer Generation (1165), and Web Search (1 (110)). These four modules together consume the majority of tokens, indicating their central role in reasoning and verification. contrast, modules like Embedding Token and relf-Repairing contribute minimally. This suggests that most token cost arises from tool planning and factual grounding rather

---

than lightweight utility steps. The total cost per instruction- respons pair remains highly affordable, typically up to $0.05 per interaction, assuming the use of advanced reasoning models []]]]] Thus, even extensive reasoning processes involving multiple verification and generation steps are economically viable, highlighting the practical affordability deploying ChemOreh.
Moreover, we also analyze the reasoning steps of generated responses in Appendix G.
Answer Generation
1165
Sentence Length
3106
Language
100%
Application Domain
100%
Validation
1472
Knowledge Level
85%
Self-Repairnng
231
Knowledge Source
85%
Coda Script
902
Concreteness Extent
Toal Distiliation
1095
Problem Context
100%
Problem Attributtion
85%
Specific Knowledge Usage
100%
Embedding Token
Quantitative Level
90%
500
1000
1500
2000
4000
4250
85
90
95
100
105
110
Consumption
Percentage
Figure 4: Token usage per generation in differ- ent modules of ChemOrch. "Answer generation" denotes the final assembly of the output.
Figure 5: Constraint-followin accuracy evalu- ated by human annotator across 10 categories in instruction generation.
4.3 Quality Analysis of Generated Instruction-Responee Pairs
Human Evaluation of Response. To assess the quality of the responses generated by ChemOrch, we conduct a human evaluation along two key dimensions: 1) Instruction Following- the response directly addresses the instruction; 2) Factual Correciness- the content is scientifically accurate. Details of the evaluation protocol and results of each task are provided in Appendix E. On average. the responses achieve an instruction following rate of 82.64% and factual correctness rate of 88.14% These results indicate that ChemOrch can generate responses that aligned with user intent and maintain high level of scientific reliability. Notably, the gap between instruction following and factual correctness suggests that factual errors are not the primary limitation; rather, improving instruction grounding, especially for complex multi-step queries, remains a potential area for enhancement. It is worth noting that integrating the tool pool significantly enhances ChemOreh's ability to generate factual responses. On name and property prediction tasks, ChemOrch achieves 75.00% accuracy, significantly outperforming GPT-4o's 22.50% without tool usage (see Appendix F). Moreover, we identify some failure modes of ChemOrch in human evaluation, providing some insights for future improvement, as detailed in Appendix K.
Constraint Following, To evaluate whether the instruction generation model adheres to user-specified constraints, we conduct targeted constraint- constraint-following experiment. Specifically, we define 10 distinct categories of constraints, each encompassing multiple concrete constraint types. These categories cover aspects such as linguistic properties (e.g., sentence length, style), domain grounding (e.g.. application domain, problem context), and knowledge control (e.g., knowledge level, concreteness, quantitative expression), as detailed in Appendix L.
To evaluate them, human annotators were asked judge whether the generated instruction satisfies the specified requirement. The evaluation results are summarized in Figure 5. We observe that the instruction generation model demonstrates strong constraint alignment across most categories. In particular, it achieves 100% adherence in language style, application domain, problem context, and specific knowledge usage, indicating excellent controllability in terms of surface form and task seman- tics. The model also performs well on sentence length and quantitative level constraints, achieving 90% consistency. Notably, slightly lower scores are observed for knowledge level, knowledge source, and problem attribution (all at 85%), suggesting that while the model captures the intent of most constraints, nuanced knowledge-related instructions remain more challenging.
4.4 Application 1: Facilitating L.MMs' Evaluation in Chemistry
In this section, we demonstrate the effectiveness of ChemOrch in facilitating both alignment of evaluation signals and the identification of weaknesses in LLMs chemistry tasks. While many
benchmarks have been proposed for evaluating the chemical abilities of L.LMS [30, 11, 31, 32). constructing such datasets is often time-consuming and labor- iabor-intensive. ChemOrch offers scalable
7

---

alternative by enabling the automatic generation of evaluation datasets for given task, especially for those that are underrepresented, rarely addressed, or incorrectly handled in existing benchmarks. To assess ChemOrch's is effectiveness in evaluation, we conduct two experiments: 1) Comparing LLM performance on ChemOrch- -eecehaeeertedd vs. existing benchmarks for consistency in trends; 2) Testing whether ChemOrch can expose LLM weaknesses in underrepresented chemical tasks. Additional experimental setup details are provided in Appendix B.
1) ChemOrch vs. Existing Benchmarks:
Aligning Evaluation Signals. To assess whether ChemOrch is comparable to existing benchmarks, we select two chemical tasks, property prediction (PP) and molecular captioning (MC), with expert-curated evaluation samples from [31]. We run ChemOreh to generate 400 test samples for the same benchmarking tasks (used as few-shot examples as a constraint in ChemOrch), ensuring they are comparable in scope and difficulty to those in the benchmark. The results in Table show that ChemOrch evaluation exhibits strong correlations with the original benchmarks: for PP, the Pearson correlation is 0.735 with p-value of 0.024; for MC, the correlation is 0,948 with a p-value less than 0.001. While absolute error values vary across evaluated LLMs, their relative ranking remains largely preserved, indicating ChemOreh's consistency in comparative evaluation. This validates the reliability ChemOrch, and highlights its potential for future use in scalable and adaptive evaluation.
Table 2: The evaluation results (PP: Accuracy (0- 1). MC: Score (1-5)) of two chemistry tasks on the original dataset (Ori.)[31] and generated dataset powered by ChemOreh (Ours).
2) ChemOreh excels at revealing LLM weak- nesses in chemistry by generating test data. We identify several important chemical tasks that have not been widely covered in existing LLM benchmarks such as Lipophilicity Predic- tion. As shown in Table 3, we evaluate vari- ous models on samples generated by ChemOrch solely from the task metadata (from authoritative databases) and its description, without using any annotated examples. The results show that most models perform poorly on these tasks, revealing significant gaps in their chemical reasoning ca- pabilities. This not only highlights the need for further improvement in LLMs but also demon- strates ChemOrch's s effectiveness in identifying their weaknesses.
Property Pred.
Molecule Cap.
Model
Ori.
Ours
Ori.
Ours
Llama-3.1-8-Imm
0.203
0.051
3.28
3.52
Qwen2.5-7B-Ins.
0.580
0.277
4.54
5.34
GPT--o-mini
0.418
0.292
5,52
6,56
GPT-4o
0.548
0.441
5.90
6.60
Qwen3-14B
0.593
0.549
5.66
6.60
gemma-3-27b--
0.300
0.328
4.96
6.56
DeepSeck-V3
0.450
0.523
5.92
6.96
Llama-A.3-70--1
0.470
0.364
4.96
6.14
Claude3.5-haiku
0.565
0.535
5.86
6.62
Table 3: Model performance (Accuracy) on fine-grained tasks in chemistry. B3D3 means Blood-Bres
Barrier Penetration Prediction, DDI means Drug-Drug interaction.
Model
B3D3 Prediction DDI Prediction Lipophilicity Prediction
-lama-3.1-88-Instiee
0.058
0.000
0.013
Qwen2.5-7B-Instre
0.236
0.224+0.010
0.036.0.004
GPT-44-mini
0.256=0.010
0.176.g..05
0.036.20.004
GPT-4o
0.285
0.133:20.012
0.027 0027=00.00
Qwen3-14B
0.171 +0.000000 004
0.3-17 1000000000 00?
0.088 010
gemma-3-27b-it
0.312 004
0.520
0.042 007
DeepSeek-V3
0..13
0..8020.012
0.062-40.004
Llama-333-708-Instrue
0.133 00000
0.140 -0.00...00 oou
0.036 000
Claude3.5-haiku
0.28720.000
20.007
0.056g0.004
4.5 Application 2: Improving LLMs' Chemistry Intelligence
In this section, demonstrate the effectiveness of the samples generated by ChemOrch in enhancing the chemistry capabilities of LLMs. ChemOrch is used to create diverse samples across a range of tasks, including task-specific questions, general Q&A, chemistry reasoning, and tool usage. We finetune Llama-3.1 1-w.3-1-----nsttte [33] and Qwen-2.5-788 Insttrcct [34] by SFT on generated samples. More details are provided in Appendix B.
Improvement on both the chemistry general-purpose Q&A and task-specific questions. As shown in Figure 6, property prediction accuracy increased by around 35% for Llama-3.1-8B- indicating a stronger ability infer molecular properties. For Molecule Captioning, scores improved from 0.5 to 1.2 for Llama-3, --8B-Ins. and Qwen-2.5-7B 1iee--2.5-B-Inss Ins., showing that models generated
8

---

LLaMA Vanilla
LLaMA Fine-tuned
Qwen Vanilla
Qwen Fine-tuned
1.00
7.5
Accuracy
0.75
0.50
5.0
0.25
2.5
0.00
Property Prediction
Tool Usage
Molecule Captioning
General Q&A
Figure 6: Fine-tuning results of Allma-3.1- 3.1aa-...--B----- 1-8B-Instruct and Bren2.5-7B-Instrren on tasks such as property prediction, molecule captioning, General QA and tool usage.
more accurate and informative descriptions. Similarly, in general, chemistry Q&A, we observed a comparable increase, suggesting enhanced domain understanding and reasoning ability. These results demonstrate that ChemOrch effectively boosts both specialized and general-purposs chemical Q&A.
Improvement on the chemistry reasoning capability. We evaluate how LLMs can improve their reasoning capability by fine-tuning on ChemOrch-generater data, promoted by chemistry reasoning questions from MMLU-Pro [32]. As shown in Table 4, both models show notable improvements over their vanilla baselines. For Llama-3. 1eama-3.1-8B---e ----ruuuttt accuracy rises from 13.9% to 28.99% with 500 training samples. However, gains plateau beyond 200 samples, possibly due to the model's limited capacity or a distributional mismatch between ChemOrch data and Llama's 's generation style, which may hinder effective learning. In contrast, -wen-2.5-7B-Inmm
benefits more significantly from larger sample sizes, improving from 24. 24.64% to 37.97%. suggesting stronger alignment with ChemOrch data and better instruction-folloiingg ability. Overall, these results demonstrate that ChemOrch can substantially enhance LLMs' chemistry reasoning capability.
Improvement on the agentic capability. The agentic framework has begun to receive attention within the chemistry domain [8]. where tool-use capability is considered one of the most critical aspects of agentic models. To investigate whether ChemOrch enhance model's ability to use tools, we leveraged it to dynamically generate instructions and corresponding code snippets for two widely used chemistry tools: RDKit and PubChem. These generated examples fine-tune the model. As shown in Figure 6, the tool-use accuracy of the models improved significantly after fine-tuning: Notably, Llama-3 1.1ma-3.1-8B 1..--BB-Immn --B--nnstruct achieved relative improvement of over 50%, highlighting ChemOrch's effectiveness in facilitating tool-oriented skill acquisition.
then used to
17
# Sample
L.ama-3.-8B-Bn
Owen-2,5-7B-Ins.
47%
Dool
Vanilla
0.1391 0.1391 20.007
0.2464_0.004)
Percentage:
1.
n=200
0.281210.00:
0.2870=0.0121
selfrepairng
The segec
Auviaage
n=300
0.246444o.00s2
0.2928
tso dicllation
n=400
0.2725+o.oosz
0.3478=0.0213
13
15%
1.24
n=500
0.280910.0041
0.3797
Follod Falled Suffideneyy saff-ropa tggo Frepo king Matten
Arg. tools
Aup. tools
Table 4: Model performance (Accuracy) chemistry reasoning questions of MML.-Pro [32] under differ- training sample sizes.
Figure 7: Ablation analysis of the mod- ules in ChemOrch.
4.6 Effectiveness of Core Modules in ChemOrch
The effectiveness of the difficulty reward model Mai has been verified in Appendix D. The ablation studies of other modules including tool distillation, self-repairing, and sufficiency validation are shown in Figure 7. As sweep: disabling self-repairing causes the tool-execution failure rate to jump from 18% to 47%, adding sufficiency validation catches an 13% incomplete inadequate outputs, and applying tool distillation cuts the average number of tools invoked from 1.58 to 2.24 with no loss in overall success. These results indicate the effectiveness of core modules ChemOrch.
observe, ablating each module in turn highlights its impact in single

---

4.7 Scalability of ChemOrch
ChemOrch supports scalable tools for handling more extensive chemistry tasks. For example, ChemOrch can support alternative molecular encodings, such as graph-based, tree-structured, or ISON representations by easily integrating a new tool function, which detailed Appendix O.
Conclusion
We introduce ChemOreh, a transformative framework for generating high-quality, tool- cool-grounded instruction-respee response pairs in chemistry. It significantly lowers the barrier for assessing and further improving LLMs on chemistry tasks, particularly through the integration of chemical tools that enable accurate and verifiable reasoning. Its principles dask-conditioned generation, tool grounding. difficulty calibration, and repairability-art domain-agnostic and could help build cross-disciplinary Al models with expert-level abilities.
Acknowledgement
This work was supported by the National Science Foundation under the NSF Center for Computer Assisted Synthesis (C-CAS), grant number CHE-2202693.
References
[1] Kai Zhang, Rong Zhou, Fashan Adhikarla, Zhiling Yan, Yixin Liu, Jun Yu, Zhengliang Liu, Xun Chen, Brian D Davison, Hui Ren, et al. A generalist vision-language foundation model for diverse biomedical tasks. Nature Medicine, pages 1-13, 2024.
[2] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. ChatDev: Communicative Agents for Software Devel- opment. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174-15186, 2024.
[3] Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang. Ruiming Tang, and Enhong Chen. Understanding the planning of L.LM agents: survey. arXiv preprint arXiv: arXiv:2402.027 16, 2024.
[4] Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, et al. ChemDFM: A Large Language Foundation Model for Chemistry. arXiv preprint 24X. 2024.
[5] Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han. A comprehensive survey of scientific large language models and their applications in scientific discovery. arXiv preprint arXiv:2406. 10833, 2024.
[6] Debjyoti Bhattacharya, Harrison J Cassady, Michael A Hickner, and Wesley F Reinhart. Large language models as molecular design engines. Journal of Chemical Information and Modeling, 64 64(18):7086-706, 2024.
[7] Gang Liu, Michael Sun, Wojeiech Matusik, Meng Jiang, and Jie Chen. Multimodal Large Language Models for Inverse Molecular Design with Retrosynthetic Planning. arXiv preprint 10.02224 2024.
[8] Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023.
[9] Mayk Caldas Ramos, Christopher J Collison, and Andrew D White. A review of large language models and autonomous agents in chemistry. Chemical Science, 2025.
[10] Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, et al. Chemllm: chemical large language model. arXiv preprint arXiv:2402.06852. 2024.
10

---

[11] Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. What can Large Language Models do in chemistry? comprehensive benchmark on eight tasks. In Thirty-sevntth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
[12] Chiranjib Chakraborty, Manojit Bhattacharya, and Sang-Soo Lee. Artificial intelligence enabled ChatGPT and large language models in drug target discovery, drug discovery, and development. Molecular Therapy-Nucler Acids, 33:866-868. 2023.
[13] Kevin Maik Jablonka, Qianxiang Ai, Alexander Al-Feghali, Shruti Badhwar, Joshua D Bocarsly, Andres M Bran, Stefan Bringuier, L Catherine Brinson, Kamal Choudhary, Defne Circi, et al. 14 examples of how LLMs can transform materials science and chemistry: a reflection on a large language model hackathon. Digital discovery, 2(5): (0)):223312500 2023.
[14] Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang. Zhichun Guo, Nitesh Chawla, Olaf Wiest, and Xiangliang Zhang. What indeed can GPT models do in chemistry? A comprehensive benchmark on eight tasks, 2023.
[15] Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol- Int-rustruct A Large-Scale Biomolecular Instruction Dataset for Large Language Models. The Twelfth International Conference on Learning Representations.
[16] Yue Huang. Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, and Xiangliang Zhang. DataGen: Unified Synthetic Dataset Generation via Large Language Models. In The Thirteenth International Conference on Learning Representations, 2025. URL httpp:://ppeteeeeeeeeeette http:///ppe/eeeeeeeeee ew .nt formmr or
[17] Arina Razmyslovich, Kseniia Murasheva, Sofia Sedlova, Julien Capitaine, and Eugene Dmitriev. ELTEX: A Framework for Domain-Driven Synthetic Data Generation. arXiv preprint 2505 2025.
[18) Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Aligmment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.0844 2024.
[19] Gregory Landrum. RDKit: Open-source cheminformaties. https:// rdkit. org, 2025.
[20] Sunghwan Kim, Jie Chen, Tiffany Cheng, and et al. PubChem 2025 update. Nucleic Acids Research,
2025. doi: 10. 10.1093/nar/gkae1105
[21] Han Bao, Yue Huang. Yanbo Wang, Jiayi Ye, Xiangqi Wang, Xiuying Chen, Yue Zhao, Tianyi Zhou, Mohamed Elhoseiny, and Xiangliang Zhang. AutoBench-V V: Can Large Vision-Language Models Benchmark Themselves? arXiv preprint arXi::2410.21259 2024.
[22] Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic Evaluation of Large Language Models by Meta Probing Agents. In International Conference on Machine Learning, pages 02599-62617.. PMLR, 2024.
[23] OpenAl. GPT-4o Technical Report.
hwtpa/aa..aa/eeee//oooooo-----/0 2024.
[24] OpenAL.
http::///ppeaaaccc//o. openai-o1 -in--oo---miii-e-ee-ei wwwi-iivaaniinnngiiiaanmnn htwpaaccinnnnttteefeeeeeeerraaao 2024.
OpenAl ol-mini.
[25] OpenAL. OpenAl eex-embbedding---malll https://platfon http://ppaatfooooooaa...ac com/ docs/
guides/ /eweeddingg////////eeeee/d 2024.
[26] Zhuoyan Li, Hangxiao Zhu, Zhuoran Lu, and Ming Yin. Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, 2023.
[27] Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.

---

[28] Adrian Mirza, Nawaf Alampara, Sreekanth Kunchapu, Martiño Ríos-Garcia Benedict Emoek- abu, Aswanth Krishnan, Tanya Gupta, Mara Schilling- Wilhelmi, Aneesh, et al. A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry, pages 1-8, 2025.
schiling--illeelmi Macjonathan Okereke, Anagha
[29] OpenAl. OpenAl o3-mini. https: / /tt////ppaaaaaa com/ index/ hptp://opnnnii----pmmm
2025.
[30] Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang. Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. arXiv preprint urXiv22307110655, 2023.
[31] Kehan Guo, Bozhao Nan, Yujun Zhou, Taicheng Guo, Zhichun Guo, Mihir Surve, Zhenwen Liang, Nitesh Chawla, Olaf Wiest, and Xiangliang Zhang. Can Ilms solve molecule puzzles? multimodal benchmark for molecular structure elucidation. Advances in Neural Information Processing Systems, 37:134721-134746, 2024.
[32] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
[33] Meta. The Llama Herd of Models. arXiv preprint arXiv:2407.21783, 2024. URL https: //arxiv. org/ abo/2407 21783.
[34] Qwen Team. Qwen2.5: A Party of Foundation Models, September 2024. URL https:
//qwenlm. gi thub. hogboo////oooooooooooooooooooooo qwen2.5/.
[35] Yuyan Liu, Sirui Ding, Sheng Zhou, Wenqi Fan, and Qiaoyu Tan. Moleculargpt: Open large lan- guage model (Il)) for few-shot molecular property prediction. arXiv preprint (tt::2006.125500 2024.
[36] Jerret Ross, Brian Belgodere, Samuel C Hoffman, Vijil Chenthamarakshan, Jiri Navratil, Youssef Mroueh, and Payel Das. Gp-molformer: foundation model for molecular generation. arXiv preprint arXiv:2405.c 04912, 2024.
[37] Hyosoon Jang, Yunhui Jang, Jachyung Kim, and Sungsoo Ahn. Can LLMs Generate Diverse Molecules? Towards Alignment with Structural Diversity. arXiv preprint arXiv:2410.03//88 2024.
[38] Zhe Chen, Zhe Fang, Wenhao Tian, Zhaoguang Long, Changzhi Sun, Yuefeng Chen, Hao Yuan, Honglin Li, and Man Lan. ReactGPT: Understanding of Chemical Reactions via In-Context Tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 84 92, 2025.
[39] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345, 2024.
[40] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01668 00600000000 2024.
[41] Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D) White, and Philippe Schwaller. Augmenting large language models with chemistry tools. Nature Machine Intelli- gence, 6(5):525-535 2024.
[42] Daniil A Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes Autonomous chemical research with large language models. Nature, 624(7992):570-578, 2023.
[43] Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data. arXiv preprint arXiv:2404.07503, 2024.
12

---

[44] Timo Schick and Hinrich Schütze. Generating Datasets with Pretrained Language Models. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6943-6951, Online and Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. doi: 10. 12655//1/221..mml/ppppl
URL https://
aclanthology org/2021 emnlp-meaeee mai...aan....m 555/.
[45] Arij Riabi, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Staiano. Synthetic Data Augmentation for Kero-Shoo Cross-Lingual Question Answering. In Marie- Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods Natural Language Processing, pages 7016-7030, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. 14.18863v1/225 1/2021 wwwp022e.mlmmmmiiin...... URL https:/ peltpp://accaanee http//aaaaaatooooooooooooo org/ 2021. momml-mman....
[46] Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. (InThe) WildChat: 570K ChatGPT Interaction Logs In The Wild. In The Twelfth International Confer- ence on Learning Representations, 2023.
[47] Letian Zhang, Quan Cui, Bingchen Zhao, and Cheng Yang. Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis. arXiv preprint arXiv:2503.0874 2025.
[48] Zijje Zhong, Linqing Zhong, Zhaoze Sun, Qingyun Jin, Zengchang Qin, and Xiaofan Zhang. Synthet2e: Generating synthetic data for fine-tuning large language models on the text2cypher task. arXiv preprint arXiv:2406. 10710, 2024.
[49] Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V Le. Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03958 2023.
[50] Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Fang Zeng, Wei Liu, et al. Auggpt: Leveraging chatgpt for text data augmentation. IEEE Transactions on Big Data, 2025.
[51] John Joon Young Chung, Ece Kamar, and Saleema Amershi. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions. arXiv preprini arXiv:2306.044 04140, 2023.
[52] Dongping Chen, Ruoxi Chen, Shu Pu, Zhaoyi Liu, Yanru Wu, Caixi Chen, Benlin Liu, Yue Huang, Yao Wan, Pan Zhou, et al. Interleaved Scene Graph for Interleaved Text-and-Image Image Generation Assessment. arXiv preprint arXiv:2411.17168 2024.
[53] Haris Riaz, Sourav Bhabesh, Vinayak Arannil, Miguel Ballesteros, and Graham Horwood. MetaSynth: Meta-Prompting-Drivee Driven Agentic Scaffolds for Diverse Synthetic Data Generation. arXiv preprini arXiv:2504. / 12563, 2025.
[54] Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of preferences via system message generalization. Advances in Neural Information Processing Systems, 37:73783-73829, 2024.
[55] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXhv:2412.08905. 2024.
[56] Fanwang Meng, Yang Xi, Jinfeng Huang, and Paul W. Ayers. A curated diverse molecular database of blood-brain barrier permeability with chemical descriptors. Scientific Data 8(289), 2021. doi: 10. 1010038/441577-00--0066--3 URL https: : /www. www.rr..auer......o com/ articiss/
021-770021-10000
[57] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeutics Data Commons: Machine Learning Datasets and Tasks for Drug Discovery and Development. Proceedings of Neural Information Processing Systems, NeurIPS Datasets and Benchmarks, 2021.
13

---

[58] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Artificial intelligence foundation for therapeutic science. Nature Chemical Biology, 2022.
[59] Alejandro Velez-Arce, Xiang Lin, Kexin Huang, Michelle M Wenhao Gao, Bradley Pentelute, Tianfan Fu, Manolis Kellis, and Marinka Zitnik. Signals in the Cells: Multimodal and Contex- tualized Machine Learning Foundations for Therapeuties. In NewrIPS 2024 Workshop Al for New Drug Modalities, 2024. URL
net/ htte//ormmm//// idd-idd-i -ekkk1yppII.
[60] Jakub Adamczyk and Piotr Ludynia. Scikit-fingerprints:: Easy and efficient computation of molecular fingerprints in Python. SoftwareX, 28:101944, 2024. ISSN 2352-7110. doi: hitps:
/rdoi.org/10. 101.:.0111....ffo...... softj 2024. 101944. URL https:/ www. http://wwwwwww..eeeieeececscccce www.acciieeeenccee.occcooeeoceee com/ //ciciiiii enceee/////
#tticleeppi//2252711100
[61] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Thirly-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL
https:// httpss///ppeeeeeeeeeeeee wtteet//oounnnddcciiooooo
[62] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. LlamaFactory: Unified Efficient Funing of 100+ Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics.
www.a.avvvv.......... abs /2403.43222 13372.
[63] Bharath Ramsundar. The Basic Tools of the Deep Life Sciences. DeepChem, 2021.
[64] Zhengkai Tu, Sourabh J Choure, Mun Hong Fong, Jihye Roh, Itai Levin, Kevin Yu, Joonyoung F Joung, Nathan Morgan, Shih-Cheng Li, Xiaoqi Sun, al. ASKCOS: an open source software suite for synthesis planning. arXiv preprint arXiv:2501.01833 2025.
[65] Jingyuan Qi, Zian Jia, Mingian Liu, Wangzhi Zhan, Junkai Zhang, Xiaofei Wen, Jingru Gan, Jianpeng Chen, Qin Liu, Mingyu Derek Ma, et al. MetaScientist: Human-Al Synergistic Framework for Automated Mechanical Metamaterial Design. arXiv preprint arX2v:2412.16277, 2024.
[66] Huan Zhang, Yu Song, Ziyu Hou, Santiago Miret, and Bang Liu. Honeycomb: A flexible llmeb agent system for materials science. arXiv preprint arXiv:2409.000 00135, 2024.
[67] Xiangru Tang, Tianyu Hu, Muyang Ye, Yanjun Shao, Xunjian Yin, Siru Ouyang, Wangchunshu Zhou, Pan Lu, Zhuosheng Zhang, Yilun Zhao, et al. ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning. arXiv preprint 0rXiv:2501.06590 2025.
[68] Kevin Wu, Eric Wu, and James Y Zou. Clasheval: Quantifying the tug-of-war war between an Ilm's internal prior and external evidence. Advances Neural Information Processing Systems, 37: 33402-33423 2024.
[69] Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training Ilms to prioritize privileged instructions. arXiv preprint 4404 13208, 2024.
14

---

Appendix Contents
Related Work
16
B Details of Experiment Setup
16
Details of Manual Review on Tool Decomposition
17
D Details of the difficulty reward model with feedback
18
Human Evaluation Details For Response Quality
19
F Baseline Comparison
20
Reasoning Steps of Generated Responses
20
H Reliability of LLM-as-a-Judge Evaluation
20
I Details of Selected Tools
J Broader impacts
22
Limitations and Failure Modes for the Future Work
23
L Constraint Examples
24
M Algorithm of ChemOrch
24
N Data Examples and Case Study
25
Scalability Example
P Prompt Template
32
15

---

A Related Work
Chemistry of LLMs. LLMs are demonstrating remarkable capabilities within the chemistry do- main. Key applications include predicting molecular properties[35. 14), generating novel molecular structures[36, 31, 37], and tackling complex problems in chemical synthesis and reaction informatics, such as planning reaction pathways and predicting outcomes[7, 38]. An emerging paradigm involves deploying LLMs autonomous agents [39, 40], exemplified by systems like ChemCrow [41] and Coscientist [42], which integrate the LLM's reasoning capabilities with specialized external tools to automate complex scientific workflows. Despite this progress, significant challenges hinder LLMs from reaching their full potential in chemistry. Progress still stalls for three reasons: curated chemistry instructions scarce, generic synthetic pipelines overlook molecular structure and domain rules, and existing generators struggle to produce diverse yet verifiable prompts and answers. These gaps motivate ChemOrch, which combines tosk-controlled instruction generation with tool- noo-grounded, executable responses to yield large-scale, chemically sound training data.
Synthetic data of LLMs. LLMs have shown remarkable capabilities in generating synthetic data [43]. Unlike earlier approaches centered on traditional language models [44]. latest LLMs offer greater potential for producing high-quality synthetic datasets across variety of domains, including multilingual question answering [45], conversational agents [46], instruction tuning [18, 48, 15]. enhancing truthfulness [49], and promoting data diversity [50-53]. Most recently, DataGen [16] was introduced as a framework for generating high-quality textual datasets, enabling more targeted evaluation and improvement of LLM capabilities. Similarly, Lee et al. present Janus, an LLM trained on synthetically generated, diverse system messages to support personalized and general alignment [54]. Notably, Phi-4 [55] strategically incorporates synthetic data throughout the training process, which achieves an excellent performance on various downstream tasks.
B Details of Experiment Setup
Generation Details. We include task descriptions in Table 5 and Table 6. For General Chemistry Q&A, we use the chemistry-related topies as constraints, as shown Table and Table 12. For task-specific challenges, we generate the dataset as follows:
Property Prediction & Molecule Captioning: We leverage data samples from ChenLLMBench
[11] as few-shot exemplars to guide the generation process of ChemOrch.
Tool Usage: During generation, ChemOrch performs web searches to retrieve relevant code blocks or examples based on the instruction. To ensure correctness, the retrieved code is executed locally,
and any code that results in errors filtered out.
Chemistry Reasoning: We use selected 1 155 examples from the chemistry reasoning questions in MMLU-Prr [32] (distinct from the evaluation set) as constraints (few-shot learning) to guide the model's generation. During this process, ChemOrch typically produces executable code for performing calculations and obtaining answers. To enable the synthesis of reasoning chains, we introduce additional constraint that requires the model to generate code with printed intermediate results, thereby making the reasoning process explicit.
BBB Penetration Prediction & DDI Prediction & Lipophilicity Prediction: We sample 200 seed data points for each task from three authoritative databases as the metadata for generation. The BBB penetration prediction comes from B3DB [56]. the DDI prediction comes from TDC [57-59], and the lipophilicity prediction comes from MoleculeNet [60]. We transform the data into JSON format for each task its metadata. Then, we send the metadata both the IG and the RG models to generate accurate and reliable instruction-response pairs.
Task Evaluation. For evaluation, adopt an LLM-as-a-Judee framework [61] across all tasks. Except for the molecule captioning task, where the LLM assigns a score from 1 to 5 by comparing the generated molecular description with the annotated description, as shown in Figure 34, all other tasks are evaluated by directly comparing the generated answers with the ground truth to determine correctness and reporting accuracy, in Figure 33. For tasks involving tool usage, we similarly assess the correctness of the generated functional code block by comparing it to the ground truth implementation, treating it as binary classification task, as in Figure 32.
Data Used in Experiments. For statistical analysis in subsection 4.2 and human evaluation in subsection 4.3, we randomly select 400 data points from both the General Chemistry Q&A and
16

---

Task-Specific Challenges datasets. To evaluate constraint adherence, we generate 100 instructions across 10 categories. The dataset used for the chemistry evaluation in subsection 4.4 contains 400 examples per task. For the fine-grained evaluation, each task includes 150 examples. For the fine- tuning experiments described subsection 4.5, each of the three tasks (property prediction, tool usage, and molecule captioning) includes 400 samples for training and 400 for testing, with the test data sampled from the original benchmark [14]. For the general Q&A task, 1000 samples are used for training and 200 for rest larger size reflects the broader scope of chemical knowledge required. The ablation study of the ChemOrch module is conducted using a separate set of 200 data points. For all testing sets, conduct human evaluation to filter out low-quality data points.
Hyperparameter Setting. our framework, we set few hyperparameters optimize the ability of our model. We set top_k k = 5 and cool_distilling _um__um____ threshold = 5 in the tool selection module to guarantee the selected tools" diversity and avoid tool redundancy. In the tool invaca- tion module, we set scrrpp_fiiiin____. 3, and effectiveness wteeeeeeeeeeeeeeennneeeeeeeeee Bumiiing_mmmmmeeee wtrckkinnn_nmtthehhhooeeeee generation, which is closely related to the correctness of the results.
soammthreehold = 3, error error_fiiiiii fifffiiinng...o num threshold 5. These settings ensure accurate code
Table 5: Fine-Tunue Tuning Tasks
Task Name
Description
Name Prediction
of the given molecular SMILES. Predict the property of molecules that the given reactants' SMILES represent. Predicts free energy of hydration of molecules, important for understanding solvation and interactions.
Predict the IUPAC
Property Prediction
Hydration free energy prediction
Molecule Captioning
Provide detailed description of the molecule that the given molecular SMILES represents.
Reaction Prediction
Predict the main product SMILES according to the given reactants' SMILES.
Table 6: Fine-grained Evaluation Tasks
Task Name
Description
DDI Prediction
Predict the interaction type between two drugs.
BBB Penetration Prediction
Predict compounds' blood-brain barrier penetration.
Lipophility Prediction
Predict octanol/water distrihution coefficient (lngD) at pH 7.4.
Fine-Tuning Details. We fine-tuned two instruction-tuned large language models: LLaMA-3.1-80 Instruct and Qwen-2.5-7BB-nnttuee hyperparameters to ensure fair comparison between the models. Training epochs with a cosine learning rate scheduler and a warmup ratio of 0.1. The learning rate was fixed at 1e-5, and the per-device training batch size was set to 4, with gradient accumulation (i.e., gradient. accumulation. attttsstepssssssssse NVIDIA A100 GPUs accelerate computation and reduce memory usage. This setup provides efficient and reproducible baseline for instruction tuning of large-scale language models.
All experiments were conducted using consistent set of
performed for
1). We used 6 (bf16) precision and trained on
Constraint Generation. Specifically, we prompt the LLM to generate concrete constraints under the broader constraint categories listed Table 13. All the generated constraints are collected and then sequentially substituted into the instruction generation process of the IG model. In total, we collected over 100 different constraints to ensure diversity in the generated instructions.
C Details of Manual Review on Tool Decomposition
All sub-tools undergo a two-stage manual review. First, domain experts validate the input/output specifications, naming conventions, and descriptions to ensure alignment with chemical standards and usability. Second, reviewers simulate tool usage under realistic settings, including edge cases such as malformed molecular inputs or ambiguous return types. Through this process, we iteratively refine sub-tool definitions, add robust error handling, and rewrite unclear descriptions ensuring that each sub-tool is both syntactically valid and semantically reliable.
17

---

e
ChemLLMMenne
pemoponddin S0ILES reprosetatinn andicallultt the gramt of nus cual slasss uang rodocke wight
0 Liamag-x 8B instruct Training
Mol-nstructions
Convering chamia atretura SWILES restin
Human Annotation
LLA--based Refinement
insonhodae of chem lat returion and andancardia of beclelllrr structurds.
Synthetic w/ Constraints
Basson ditiouss Unearvandd
sonversre
CherDren
noteiions Accitirully mull calouletie max molo inbboa browiccaa mee wagglts and moke liberi, recsating uttartion datek.
Figure 8: Data collection and training process of the difficulty reward model with feedback.
Moan = 3.38
Moan 3,59
1.25
Density
0.8
Itod Skow Normal
Density
1.00
Fittad Skow Normal
0.6
0.76
0.50
0.2
Figure 9: Data distribution comparison of test (left) and train (right) dataset.
The manual review process was conducted by team of nine experts with diverse academic back- grounds to ensure both computational and chemical correctness. Specifically, the team comprised three undergraduate students and six PhD students. Among them, three undergraduates and two Phl) students had background in computer science, while the remaining four PhD students specialized in computational chemistry. This interdisciplinary composition ensured that each function was reviewed effectively.
D Details of the difficulty reward model with feedback
Training data collection. To train the difficulty reward model Maiff we construct dataset com- prising both synthetic and ChemOrch framework, covering a diverse range of chemistry tasks with variation introduced through wide of constraints, including both manually designed templates and L.L--eenertted constraint prompts. To complement this, we incorporate real instructions from existing datasets such as Chem- LLMBench [11] and Nol-Instructions [15], which are manually annotated by a team three experts major in computational chemistry. Each instruction is labeled with difficulty score 1-5 scale, along with textual explanations indicating the reasons for its simplicity and difficulty. To improve clarity and consistency, annotators use GPT-4o to refine their drafted annotations, helping to standardize language and climinate ambiguity without altering the core assessment. Finally, we collect 3390 annotated data items.
instructions. The synthetic portion is generated using
Training details. We then train Mair using supervised fine-tuning (SFT) to jointly predict the difficulty score and generate the corresponding feedback. The Meta-Llama-3.1 1-B.-B.-BBB-- -nstruuttt is trained on 4x 4xA100 for epochs with a per-device batch size of 2, gradient accumulation of 1, and 6 precision. The training employed the AdamW optimizer with a learning rate of 1 x 10-3 and a cosine decay schedule. We used -Latory [62] for the training process.
Effectiveness evaluation. The effectiveness evaluation is performed on test instruction dataset of 900 samples, whose data distribution tightly aligned with the training data of 3390 samples, as depicted in Figure 9, ensuring con- sistency across datasets, In our human evalu- ation protocol, each predicted difficulty score presented to a computational chemistry ex- pert alongside the corresponding instruction and the textual explanation. The expert then judges whether the score and reasoning are appropriate
Table 7: Human alignment rate of Mann-
Difficulty Score
Human Alignment Rate
100% 88 9%
85.7%
86.89
100%
Total
87.0%
18

---

Human Evaluation Guideline
Objective
The objective this validation task is to assess the quality of model mooee-ggeerrtedd responses based their corresponding instructions. Specifically, your role is to determine whether the response appropriately addresses the instruction and is factually accurate.
Files and Format
You will be provided with a JSON file containing a list of examples. Each example is represented dictionary with multiple fields. For this task, please focus exclusively the following two:
instruction:
The user-provided prompt or question.
response: The answer generated by the model.
Your task is to evaluate the quality of each response with respect to the instruction.
Evaluation Criteria
For each example, assign of the following labels:
The response accurately and sufficiently answers the instruction, with factual eπrоrs.
(Pass):
(Fail): The response fails to address the instruction, or contains factual inaccuracies, hallucinations, misleading content.
You are unable to confidently determine the quality of the response, due to ambiguity, insufficient domain knowledge, unclear instruction.
N/A (Uncertain):
Figure 10: Human evaluation guideline of response quality.
or not. Table summarizes the effectiveness evaluation results, showing the alignment rate between Main and human expert judgments.
Main demonstrates robust overall human alignment rate of 87%, indicating that the model success- fully captures the nuances of chemistry instruction complexity. Difficulty levels I and 5 achieved perfect alignment (100%), while levels 2, and showed high alignment rates of 88.9%, 85.7%, and 86.8%, respectively. These results confirm that the model accurately reflects human judgment across varying complexity levels.
E
Human Evaluation Details For Response Quality
Evaluation Guideline. The human evaluation guideline is shown in Figure 10. To ensure the reliability and professionalism
assessment, the evaluation was conducted over a total 400 samples by four Ph.D. students with backgrounds in computational chemistry. The guideline instructs annotators to determine whether the model-generated response (i) appropriately addresses the given instruction and (īi) contains factual errors. Each response is labeled (Uncertain), depending on its accuracy and relevance. Annotators are instructed to focus solely on the instruction and response fields from each data entry. For the final analysis, samples labeled as N/A were excluded to ensure statistical validity.
(Pass), 0 (Fail), or N/A
19

---

Transition State Search
Yield Prediction
Steas 5%
Quantum Chemical Properties Prediction
2nu4 Steps 22.5%
Quartum Mechanical Calcuistians
Hydration Free Energy Prediction
Reaction Center identification
Molecular
Atom-Aszignment Reaction Smiles
8--10 Steps 34%
Drug Synthesis
Cheminfomatics Software Utllization
Potential Molscular Bonding Types
Acid Dissociation Canstant Prediction
Catalyst Classification
Side Effects Clacefication
Molecular Format Conversions
5-77 Stops
Drug Likeness Evaluation
Average Number Steps
(a) The distribution of step number gen- crated responses. Figure 11: Statistics of reasoning step numbers in generated responses.
(b) The step number in generated responses of different topics.
F Baseline Comparison
We collect 120 reliable metadata items from PubChem [20] and TDC [57-59] and send them to the IG model to generate the name prediction and property prediction task instructions. Then, we generate the responses using ChemOrch and the baseline model (GPT-40 [23]) for these tasks.
To evaluate the correctness of the generated pairs, we adopt LLM-as-a- [61] to calculate the accuracy of CremOrch and the baseline model. Our framework illustrates high accuracy rate of 75.00%, which is significantly higher than the 22.50% of the baseline model.
G Reasoning Steps of Generated Responses
We automatically parse and get the number of steps in each response by GPT-4o. As illustrated in Figure 11,
responses exhibit wide range reasoning depths, with over 70% requiring more than five intermediate steps. This indicates that framework promotes non-trivial, multi-stage reasoning beyond template-bast generation. Moreover, the average number steps varies substantially across topics, from more than nine steps for quantum-level predictions fewer than six for format conversions, reflecting the framework's adaptive planning capability. These results validate the core design of ChemOrch: it able support diverse, complex, and execution-grounded reasoning chains at scale.
H Reliability of LLM-as-a-Judge Evaluation
To evaluate the quality and correctness of poodel-ennerate outputs across various tasks, adopt the -uMdaa---Judge paradigm, which leverages large language models to assess generated responses. Given the growing of this evaluation strategy in recent literature, is important to establish its empirical reliability, particularly in the absence of clear rule-based or human-labeledt ground truth for complex tasks.
While rule-based metrics remain appropriate for simple binary classification, they may introduce inaccuracies in semantic evaluation (e.g., by failing to match semantically equivalent responses that differ lexically). The LLM-as-a-Judge approach offers broader applicability by capturing contextual nuances and aligning better with human preferences.
To assess the validity of this approach, we conducted a human-LLM, agreement study across three tasks using two representative instruction-tune tuned models: IB-Instruct. As shown in Table 8, Table 9, and Table 10, the results indicate strong alignment between automated scoring and human judgment: 1) Binary classification (Property Prediction): agreement up to 99.75%; 2) Binary classification (Tool Usage): agreement up to 97%; 3) Score- based evaluation (General QA): average Pearson correlation 0.796 (all statistically significant). These results support the robustness of L.LM-based evaluators as proxies for human judges in large-scale evaluation pipelines.
1-8B- 1os-Instruct and Qwen-2.4 5-
20

---

Table Human LLM agreement on property prediction task.
Model
Batch 1
Batch 2
Avg. Alignment
1aama-3.1-8B-Instrue (1)
49/50 (98%)
50/50 (100%)
99%
Llama-3.1-8 1-..--88--ssstuu (2)
50/50 (100%)
50/50 (100%)
100%
Owen-2.5-7B-B (1)
50/50 (100%)
50/50 (100%)
100%
Qwen- 2.--tB-tnstttt (2)
50/50 (100%)
50/50 (100%)
100%
Overall Average
99.75%
Table 9: Human-LLM agreement on tool usage task.
Model
Batch
Batch
Avg. Alignment
Llama-3 3.1 3.1--B-Inmaee IB-Instruct (1)
47/50 (94%)
46/50 (96%)
95%
Llama-3. 3.1-8B-Insttru (2)
47/50 (94%)
50/50 (100%)
97%
Bwven2.5-77B-Instteee (1)
48/50 (96%)
50/50 (100%)
98%
Qween-255-7B-nnstrune (2)
48/50 (96%)
50/50 (100%)
98%
Overall Average
97%
I Details of Selected Tools
ChemOrch leverages two categories of tools: chemistry-related tools such as RDKit [19] and PubChem [20] and general-purposs tools like web search.
RDKit. We utilize [DKit' widely adopted open-source cheminformatics toolkit- for molecular representation and processing during data synthesis. RDKit provides essential funetionalities for SMILES parsing, molecular graph construction, substructure matching, and descriptor computation, which are critical for generating chemically valid and structurally diverse input-output instruction pairs. Its seamless integration with Python and support for 2D/3D molecular operations make it particularly suitable for large-scale instruction generation the chemistry domain.
PubChem. We also incorporate data from PubChem² [20]. public repository maintained by the National Institutes of Health (NIH), which provides comprehensive information on chemical compounds, including their molecular structures, properties, bioactivities, and identifiers. PubChem serves reliable source for curating chemically diverse and biologically relevant compounds.
Web Search. We leverage the web search tool provided via the OpenAI API to retrieve up-to-date and domain-relevan information from the internet in real time. This enables our framework enrich instruction data with factual context and emerging knowledge beyond the model's pretraining corpus.
Reasons for Selection of PubChem and RDKit. We select PubChem and RDKit as the foundational tools in ChemOrch due to their broad functionality, stable APIs, and suitability for function-level decomposition. While alternative toolkits such DeepChem [63] and ASKCOS [64] are widely used in the chemistry community, they present practical limitations in the context of instruction- tool invocation. DeepChem [63]. for instance, emphasizes model training and evaluation pipelines, requiring users to manage datasets, train predictors, and interpret model outputs. This training- heavy workflow is often too heavy- for lightweight, step-level function calling and lacks the immediacy and transparency needed for modular LLM usage. ASKCOS [64] provides powerful capabilities in retrosynthesis and reaction planning but is optimized for end-to-end synthesis tasks and requires complex orchestration or server-side APIs, making it difficult to extract self-contained functions for flexible invocation.
In contrast, RDKit offers atomic-level cheminformatics operations (c.g., SMILES parsing, substruc-
ture matching, fingerprinting) with lightweight and stable interfaces. PubChem provides robust access to curated compound data and chemical identifiers through scalable and open APIs. They strike
hhttps://wet rdkit..za org/
Pnttpa://pubchen ncbi .nmm nih. .ii..ovv/ hnttps:/pplttfon openaii.p............o www/docs/ http:/ggddes/eeeeeeee tools web searsc search
21

---

Table 10: Human LLM agreement on score-based evaluation (General QA).
Model
Pearson
Llama-3. L. 1-8B-Instruct (1)
0.741
Llama-3. 3.1-8B-Instteee (2)
0.728
twen-2.5-7B-Instie (1)
0.859
Qwen-2.5-7B. 2.5strtt (2)
0.854
Average
0.796
balance between expressiveness, modularity, and integration ease-making them ideal building blocks for constructing function-level primitives ChemOrch.
Table 11: Molecular property prediction topics.
Topic Name
Description
Partition coefficient prediction
This task involves predicting the partition coefficient (log P) of molecules, which reflects their hydrophobicity and is crucial for understanding their pharmacoki- netic properties.
Water solubility prediction
This task involves predicting the solubility of compounds in water, using datasets like ESOL.
Hydration free energy prediction
Predicts free energy of hydration of molecules, important for understanding solvation and interactions.
Lipophilicity prediction
Predicts tendency dissolve lipids, measure of lipophilicity. Predicts quantum chemical properties, such as energies and geometries, using QM7, QM8, and QM9 datascts.
Quantum chemical properties pre- diction
Blood-Brait Barrier Penetration prediction
The task involves predicting molecules' Blood-Brain Barrier penetration capa- bility.
Protein Ligand Binding Affinity prediction
Predicts binding affinity terms Kd (dissociation constant) using PDBbind data.
BACE Inhibition prediction
Predicts compound's 's ability to inhibit BACE protein.
HIV Inhibition prediction
Predicts compound's ahility inhihit HIV replication. Predicts the side effects of drugs across multiple categories.
Side Effect prediction
Drug-Drup Interaction prediction
Whether two drugs will interact, and their interaction type.
Clearance prediction
Forecasts the clearance
of compounds from biological systems.
Oral Bioavailability prediction
Determines whether a molecule orally availahle or not (or has prodrugs).
Enzyme Interaction prediction
Identifies which enzyme(s) a drug inhibits.
pKa prediction
The task involves the estimation of the acid dissociation constant (pKa) molecules.
J Broader impacts
ChemOrch represents transformative advancement at the intersection of LLMs and computational chemistry. By generating high-quality, tool-grounded
respoone data at scale, ChemOrch lowers the barrier for training and evaluating LLMs chemistry tasks. This capability has particular value for researchers and institutions with limited access to curated chemical datasets, helping to democratize access domain-specifie tools powered by generative Al.
ChemOrch can accelerate scientific discovery. It equips LLMs with structured chemical reasoning abilities, which can support innovation in such drug discovery, materials design, and reaction informatics [65]. Researchers can use ChemOrch berrc--geeertedd tasks to identify model weaknesses, construct fine-tuning datasets, and conduct more rigorous benchmarking. These capabilities can lead to faster hypothesis testing, reduced experimental costs, and more informed scientific decisions.
ChemOrch can support education and workforce development. The system can generate chem- istry problems and reasoning tasks with adjustable difficulty, making it suitable for instructional use. Students can interact with Al-generated content to deepen their understanding of complex topics, while educators can tailor assignments to various learning levels. This flexibility makes ChemOrch especially useful in educational settings with limited access to expert instructors resources.
22

---

Table 12: Reaction-level prediction topies.
Topic Name
Description
Retrosynthetic Analysis
The task involves determining feasible starting materials and stepwise synthesis routes for target molccule.
Reaction Type Classification
This task involves classifying the type of chemical reaction.
Reaction Center Identification
This involves pinpointing specific site(s) a molecule where reaction occurs.
Reaction Condition Recommenda-
This task involves recommending optimal reaction conditions (solvent, catalyst, etc.).
tion
Solvent Classification
The task involves categorizing solvents by their chemical properties.
Ligand Classification
This task involves categorizing and distinguishing different ligands. This task involves sorting catalysts into different classes.
Catalyst Classification
Reaction Temperature Prediction
This task involves predicting the optimal temperature for given reaction.
Reactant Amount Prediction
This task involves predicting the required quantities of reactants.
Reaction Time Prediction
This task involves estimating the reaction duration.
Reaction Workup Recommendation
This task involves proposing procedures for post-reaction purification.
Yield Prediction
The task involves estimating the amount of product formed.
Selectivity Prediction
This task involves predicting the selectivity between possible products. This task involves predicting the outcomes of reaction given reactants and conditions.
Reaction Outcome Prediction
Reaction Outcome Rationalization
This task involves explaining why particular reaction outcome occurs.
Stereoselectivity Prediction
This task involves analyzing the preferential formation of stereoisomers.
ChemOrch can enable chemistry-awand autonomous agents. By integrating task decomposition, tool exccution, errоr correction, and difficulty calibration, ChemOrch provides the foundation for building LLM-based agents capable of operating in scientific domains [66, 8, 67]. These agents could assist in experiment planning, molecular analysis, and literature eynthesis-enhancing collaboration between human researchers and Al models.
K
Limitations and Failure Modes for the Future Work
Despite the strong performance ChemOrch in synthesizing domain-speciffi instructions for chemi- cal reasoning, our in-depth analysis reveals several unique and non-trivial failure modes that highlight opportunities for further improvement:
Conflict Between Tool Outputs and Model Knowledge. We observe that factual inaccuracies in some responses stem not only from tool malfunctions but more subtly from mismatch between tool outputs and the model's internal knowledge. In certain cases, the model ignores the tool's returned result-especiall when it contradicts its prior knowledge learned biases [68]. This suggests that the model does not always treat the tool as a trusted authority. A promising direction to mitigate this issue is incorporate an explicit instructional priority hierarchy [69], which tool outputs are assigned a higher trust level than model-generated content, encouraging the model to defer to tools in cases of conflict.
Error Cascades Due to Incorrect Tool Usage, We identify a failure pattern in which an early-stage tool invocation error-such as supplying an invalid or malformed SMILES string proppggaes through subsequent steps, resulting in entirely flawed reasoning chains. These snowballing errors highlight the brittleness of current tool integration. A potential solution involves developing more robust error detection and rollback mechanisms, allowing the system to identify and correct invalid tool inputs before proceeding with subsequent reasoning steps.
Model Laziness in Complex Instructions. For particularly complex instructions, we find that the model often resorts to generating high-level guidance than executing the task and providing a concrete answer. This "lazy" behavior may be an artifact of underlying system prompts used in alignment-tume prioritize efficiency and safety over exhaustive computation. Future work could explore prompt-level interventions model fine-tuning strategies better incentivize execution over delegation.
guidaacce (e.g.. "You can use PubChem to search..." rather
models (e.g.. OpenAl's I's usage constraints), which
Planning and Step Ordering Errors. notable failure mode arises from logical inconsistencies in the generated reasoning puans-such incorrect ordering of steps violations of necessary
23

---

chemical dependencies (e.g., attempting a reaction analysis before retrieving the molecular structure). These issues reflect fundamental challenge in planning for domain-specifie tasks, where procedural correctness is tightly coupled with chemical constraints. While much of the current progress in L.LM-based reasoning has focused on general domains, our observations underscore the importance of Nomain-adaptees reasoning capabilities tailored for chemistry. Future directions may include the integration of hierarchical planning modules and chemistry-aware workflow decomposition [8], which explicitly model task-specific execution order and causal dependencies.
Constraint Examples
show the constriant examples different aspects Table 13.
Table 13: Constraint examples of instruction generation.
Constraint category
Example
Sentence Length
Use extremely concise sentences, limited 5-10 words, retaining only the most essential information.
Language Style
Employ humorous and lighthearted tone with anthropomorphic or whimsical analogies.
Application Domain
Explore physical chemistry problems related to thermodynamics/kinettee calcu- lations.
Knowledge Level
Tailor content for elementary students using only common sense descriptions.
Knowledge Source
Reference recent findings from top-tier journal publications within three years.
Concreteness Extent
Maintain completely abstract descriptions without concrete examples.
Problem Context
Contextualize within industrial production line scenarios.
Problem Attribution
Formulate mechanism analysis questions with electron- electron-pushing arrows.
Specific Knowledge Usage
Involve titration equivalence calculations or endpoint determination.
Quantitative Level
Develop mathematical models algorithmic optimization requirements.
M Algorithm of ChemOrch
Algorithm 1 Instruction Generation
Require: Task space T. Constraint set C, Metadata set M
Require: Instruction Generation model Misa, Difficulty reward model Man
1: for each task / do
Iterate all tasks
2: for each constraint c C do
Iterate over all constraints
3:
for each metadata m .M do
Iterate over all metadata
4:
x - (tnssst m)
> Generate initial instruction
5:
(d & -M)a(xx)
Evaluate difficulty and feedback
6:
while difficulty d does not meet target level do
Repeat if difficulty is misaligned b Regenerate using feedback
7:
I Minsi (t, e, m. e)
8:
(d. Mai
& Re-evaluate difficulty
9:
end while
10:
Save or store final instruction x end for
> Store final instruction
12:
end for
13:
end for
24

---

Algorithm 2 Response Construction with Tool-Driven Execution
Require: Instruction æ, Metadata m, Tool pool F, Response generation model Mresp Decompose instruction into reasoning steps 4: Jmam + Top-A retrieved tools via cosine similarity
1: t- Decompose(x)
2: {di....,dy, + (s. m.)
>> Predict expected tool descriptions
3: for each dm {di...., d do
>> Semantic retrieval of candidate tools
5: end for
6: Fraw + +U Fmw
Aggregate raw candidate tool pool Refine tools via usefulness, expertise, and budget
7: F -Distill(FFant , s,,,
8: for each s E F do
9: S₁ + --Mres([,zz 2,, 8, m)
P. Generate code script for tool > Run script and store result
10: Execute Sy; if success, store output Of if execution fails then
12:
for attempt = 1 to Rmax do
Capture error e, regenerate sfit1) Mrsp S. m)
b Self-repair using error
trace
Retry execution if success then
15:
Store output oy; break end if end for 18:
16:
17:
19:
if still failed then
20:
Retrieve external documentation and regenerate Sy > Fallback to external guidance end if
21:
22: end
23: Check result sufficiency for oy; refine if needed
Validate quality of tool output
24: Check if }or} satisfies instruction æ; stop early sufficient b Early stopping for efficiency
25: end for
26: if final outputs insufficient then
Ocxtra WebSearch(r, m)
Trigger web search for missing information
28: end if
O - fortser. {Oeura)
> Aggregate all outputs D Construct final response grounded in outputs
30: + Mrep (z, O,
N Data Examples and Case Study
To provide clearer understanding of model behavior in ChemOrch, we present examples of the model's output for Instruction Decomposition (Figure 31) and Expected Tool Specification (Figure 14), using the prompt: "How can Lipinski's Rule of Five be used to assess the drug likellieeness of compound?", Moreover, we provide examples generated instruction-response pairs for tasks including molecule caption, property prediction, general chemistry Q& A, chemistry reasoning, and tool usage Figure 15, Figure 16, Figure 17, Figure 18, and Figure
Supplementing our primary experiments, we conducted additional fine-tuning using data generated by ChemOrch on two downstream tasks, solvent classification and catalyst classification. As shown in Figure Figure 12, both tasks achieved performance gains. These results further highlight the efficacy of ChemOrch.
To demonstrate the extensibility of ChemOrch, we present a representative use case involving a customized tool, illustrated in Figure 20.
25

---

Vanilla
line-tuned
Vanilla
Fine-tuned
8.0
7.5
Score
7.0
6.5
6.0
Solvent Classification
Catalyst Classification
Figure 12: Fine-tuning results on two tasks: solvent classification and catalyst classification
Instruction Decomposition
Step1: Research and summarize Lipinski's Rule of Five, focusing its criteria for drug- likeness.
Step2: Identify the key parameters of Lipinski's Rule of Five: molecular weight, logP, hydrogen bond donors, and hydrogen bond acceptors.
Step3:
Acquire the chemical structure of the compound be assessed for drug-likeneee
Step4:
Calculate the molecular weight of the compound using its chemical structure.
Step5: Determine the compound's partition coefficient (logP) evaluate its hydrophobicity or hydrophilicity.
Step6: Count the number of hydrogen bond donors (e.g., NH or OH groups) in the compound's structure.
Count the number of hydrogen bond acceptors (e.g., or O atoms) in the compound's structure.
Step7:
Compare the calculated values against Lipinski's s criteria: molecular weight 500 Da, logP 5, hydrogen bond donors < 5, and hydrogen bond acceptors 10. Step9; Assess the compound's ' drug-likeness based on its conformity to Lipinski's Rule of Five.
Step8:
Step10: Consider using cheminformatics software tools (e.g., ChemDraw, RDKit) for auto- mated calculations and analysis.
Figure 13: Example of instruction decomposition by GPT-4o.
Expected Tool Specification
Molecular weight calculator: Computes the molecular weight of compound from its chemical structure.
LogP calculator: Determines the partition coefficient of compound assess its hydrophobic- ity or hydrophilicity.
Hydrogen bond donor counter: Counts NH and OH groups a compound's chemical structure. Hydrogen bond acceptor counter: Counts nitrogen and oxygen atoms in compound's chemical structure.
Cheminformatics structure viewer: Visualizes the chemical structure of a compound.
Figure 14: Example of expected tool specification by GPT-4o.
26

---

Molecule Caption Task
Instruction:
Can you provide descriptivv analysis of the molecule that corresponds to the
SMILES: C=c.GeCCCCCCcc0ZZECCCCCCCCCCCCCO
The molecule corresponding to the SMILES string has the molecular formula C17H10C12IN02 and molecular weight of 458.082 g/mol. It contains several functional groups, including an amide group signified by the NC(=O) structure, an ether group within an aromatic system, and halogens represented by chlorine (CI) and iodine (I) atoms. The molecule is highly aromatic, featuring three interconnected aromatic rings. The first aromatic ring is phenyl unit CI=CC(=CC=CI), while the second C2=CC=C(O2) contains an ether linkage, making it an aryl ether. The third aromatic ring C3=C(C=CC(CC3)C))C contains two chlorine atoms. Overall, this compound is halogenated aromatic molecule with complex characteristics due to its amide, ether functionalities, and halogen substituents, which provide potential sites for chemical reactivity or interaction. Its structure indicates potential diverse chemical behavior due to the multiple aromatic systems and functional groups.
Response:
Figure 15: Example of molecule caption task from ChemOrch.
Property prediction Task
Instruction: Assess if the compound with SMILES: CI-CC--C-CCIC20CCCCCCCC can
penetrate the blood-brain barrier.
Biphenyl, represented by the SMILES notation CI=CC=-C-CCIC--CC-CCC-) demonstrates physicochemical properties that sug- gest it can penetrate the blood-brain barrier (BBB). Its molecular weight of 154.21 g/mol,
Response: Yes.
log P of approximately 4.0, polar surface area (PSA) of oλ* and lack of hydrogen bond donors or acceptors all fall within the favorable ranges for BBB penetration. Biphenyl's ss non-polar nature and absence of functional groups also suggest minimal interaction with active transport mechanisms, further indicating its potential for BBB penetration.
Figure 16: Example of property prediction task from ChemOrch.
General Chemistry Question Task
Instruction: What is the role of catalysis in the Haber process for ammonia production? The role of catalysis the Haber process for ammonia production is significantly enhance the efficiency of the reaction by increasing the reaction rate, facilitating the reaction mechanism, optimizing operational conditions, and improving industrial significance. Catalysts lower the activation energy required for nitrogen and hydrogen to react, with iron being a typical catalyst. This enables the reaction to proceed rapidly at feasible temperatures and pressures. The catalyst also verifies adsorption of reactants, weakening the nitrogen triple bond facilitate the formation of ammonia, while maintaining energy efficiency and cost- effectiveness, crucial for large-scale production.
Response:
Figure 17: Example of general chemistry question task from ChemOrch.
27

---

Chemistry Reasoning Task
Instruction: At 15"C, solution exhibits a pH value of 2.73. Calculate the corresponding POH and the hydroxide-ion concentration, [OH-].
Response: To solve this problem, want to determine both the pOH of the solution and the concentration of hydroxide ions, [OH ]. at a temperature of 15"C, given that the solution has pH of 2.73. Here's ' how you can approach the solution step by step:
Understand the Relationship Between pH, pOH, and pkw:
The pH of solution is measure of its acidity, while pOH is measure of its basicity. related through the ion product of water, Kn,, especially since know the lemperature-dependerr value of Kw at 15ºC.
The
Using pH and pkw to Find pOH: At any given temperature, there's 's relationship between pH. pOH, and pKw described by the equation:
pH pOH p
At C5,C, we know that pkw 13.94. You're re also provided with a specific pH of 2.73 for the solution. So, find pOH, we rearrange the equation to:
POH pkw pll 13.94 2.73
By performing this calculation, determine:
POH = 11.21
Understanding the Relationship Between POH and [OHT]:
The pOH value gives us insight into the concentration of hydroxide ions in the solution. The relationship between pOH and the concentration of OH ions is described by:
pOH
log (IOH
order to find [OH"] we need to rearrange this equation to solve for the concentra- tion:
[OH PO-
Calculating the [OH"] Concentration:
Now, with the pOH we calculated as 11.21, we can determine the concentration of hydroxide ions: [O 10 11.21 ss 6.12 10-12 M
10-11.21
This final value tells us the concentration of hydroxide ions present the solution at 15 "C, rounding off our solution process using the given inputs. By following these logical steps, we not only confirm the veracity of the result but also deepen our understanding the relationships between pH, pOH, and the ion product of water at specific temperatures.
Figure 18: Example of chemistry reasoning task from ChemOrch.
28

---

Tool Usage Task
Instruction: Utilize the get_ _et_pouuds function from the PubChemPy library find the chemical details for a compound with the SMILES string CCNNAJCCCC)CCCCCICCCCC and set smiles as the identifier namespace. Response:
try:
from pubchempy import get_compounds
# define the parameters
identifier_value = "ca [n+] (cc) cca (c#c)
namespace_value "smiles"
call the get_compounds function with the provided
parameters
result get_compounde (ilentifierridemtiiiee.... =namespace_valuu
namespace
# print the reault with an explanatory message print ("e chemical details for the compound with the smiles string '0' are: format (identifier_ value, reault))
except Exception e:
print ("iroor" (o))orratee))
Figure 19: Example of tool usage task from ChemOrch.
29

---

Customized tool example
User customized tool information: smiles_from_comppeee
additional_tool
"tool"; samlles_tromccomoudd
"module": "ord_schema. mossago_helpers",
"description"; "Fetches generates SMILES identifier for compound. If SMILES identifier already exists, is aimply returned.",
parameters": {"compound"; "reaction_pb2. Compound message. "}, ddccumentation": "https://dode open reantoeaction--eeeee -atabaasee org/enl en/ latest/ord_ otd ord schema. html #module - message_helpers
}]
Tool calling script:
import ord_ achemac message.helperr
from ord_schema import reaction_pb2 try:
# Create Compound object and add SMILES identifier for ethanol
compound reaction_pb2 Compound identifier compound. identifiers add ()
identifier.tyy type reaction_ pbp. ConpoundIdentifier.. SMMES identifier value = "CCO"
Call the function from the module with properly formed compound object
result compound)
friles_from_compour
("e SMILES string for ethanol is:", result) except Exception as e: print (rinoo"Error"" 8)
Figure 20: Example of customized extended tool.
30

---

O
Scalability Example
Importantly, ChemOrch can be readily extended to handle alternative molecular formats by simply introducing appropriate conversion functions in the preprocessing stage. For example, to support graph-based representations, one only needs add a transformation module before the main task. Below, we provide a code snippet illustrating how ChemOrch can seamlessly convert a graph representation:
def graph_to_ poo_iupc__nar name (graph):
2
mol Chem. RWMol()
3
atom_idx_map (
Add atoms
for i, stom_info in enumerate ("rrptttaggraahaa"ttttatt atom Chem Atom (atom_ fafttmminooooioooeeeeo [nfoo""eement"]] atom. Sewooreallaarreeeeeeeeeeee (ator_eeeaaa___a (atom_info infoiinfo... get ("charge", 0))
atom www.Seeesrrrmiiiccccc (atom_ info. wett"geeaettaaaeeee ("is_aromatic", False))
idx mol AddAtom (atrom)
atom soom_idx_map[]] idx
12
13
Add bonds
bond_order_man {
"single": weem.rdchem. BondType rdchem.BondIyppe .INGLE,
"double": Chem them.rdchem. rdchem. Bowiemm..oddyeeee..m .DOUBL DOUBLE,
17
"triple": Chem.
Chem.rdchem rdchem. BondType..RIPPPP.. TRIPLE,
"aromatic"; Ahem.rdchem.
AROMATIC
}
added s)t()
22
for a1, neighbors in
.itadms ();
for a2, bond_type in meighbors:
if (a2, a1) in added:
continue
26
bt bbond_order__e get (bond_typp lower ())
if bt is None:
raise ValueError UUnknoo bond type: (bond_type)"" mol. AddBond (atom_idx [axx__ma[aa]]]]_ atom_idz _amm_ix__map[p]2 bt) add add a2))
mol. (strrrrrtttCCccccrttttrrreeee
Chem. Chem. Swwitiieeolleeeneoaoaaaaaaaoalaa (mol)
return pep-get__ce cee__compundd (Chem. MolToSmiles (mol, canonical-True + 'smiles' ) [0] iuppc_ name
31

---

P
Prompt Template
Instruction Synthesis Prompt
You are advanced Al assistant tasked with generating high-quality instructions for synthetic dataset creation.
Your goal is to produce diverse set of instructions (or questions) based on a given user task. The corresponding answers will be generated later to form dataset. ### **Instructions:**
1. **Task Understanding:** Carefully analyze the provided task and determine its core objective.
2. **Instruction Generation:## Create exactly 'n' unique instructions related to the task. The instructions should diverse in phrasing and complexity.
3. 1.*Clarity & Context:** Ensure each instruction is clear and provides enough context for
Al model to generate a meaningful response.
4. **Format:* Return the instructions strictly Python-style list of strings.
5. **Custom Constraint:** (custom constraint)
6. **taadatt= If metadata is provided, your instructions should adhere it.
### **xample:***
#### **User Task:** Toxicity Prediction
#*## **Generated Instructions (Example Output):**
"Does beyrelalpyrene exhibit toxicity to humans?",
"What is the acute toxicity of trichloroethylenee""
"Does bisphenol A have endocrine endocrine-disrupting effects?",
"Do pyridine compounds have neurotoxic effects?".
"Does tetraethyl lead pose long-term toxicity risks to the environment and humans?"
Figure 21: Instruction synthesis prompt for ChemOrch.
NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research. addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA]-
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide a short (1 (1-2 sentence) justification right after your answer (even for NA).
The checklist answers reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
integral part of your paper submission. They visible to the
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " "rovided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). general, answering "[No] " or "[NA] is not grounds for rejection. While the questions are phrased in binary way,
32

---

Instruction Decomposition Prompt
You are an advanced AI assistant tasked with planning how to solve a given instruction. Your goal is to **break down the problem into structured steps** that can be executed using external tools or reasoning. You should ** not* provide an answer-only plan. #i## **Instructions:***
1. **Understand the Instruction: ** Carefully analyze the given instruction to determine its requirements.
2. **Identify Key Elements:** Identify key components such as subject, method, and expected output.
3. **Break Down into Steps:** Generate structured plan consisting of logical steps that
guide the problem-solving process.
4. **Ensure Tool Compatibility:** If an external tool is likely required (e.g., chemical
database, scientific literature, mathematical solver), indicate it explicitly.
5. **Format:** Return the planning steps strictly as a Python-style list of strings.
** *etaddata metadata provided, your planning should centre on Now, generate a structured plan for the following instruction:
Instruction: (instruction)
Ensure the output is formatted strictly Python list of strings.
Figure 22: Instruction decomposition prompt for ChemOrch.
Tool Planning Prompt
You are an advanced AI assistant tasked with defining the ideal tools for executing a plan. Your goal to describe the functionalities of these tools concisely, ensuring that each tool serves onene specific purpose**. ### **Instructions:**
1. ** **nallyze the Planning Steps:** Carefully review the provided planning steps to determine what kind of external tools would be needed to complete them.
2. ** Define the Ideal Toolset:** Describe **only the necessary** tools, ensuring that each
tool performs only **one function
3. **Keep Descriptions Concise: Each tool description should be brief and focused on its function.
4. **Limit the Number of Tools:** Minimize the number of tools by **combining related functionalities*** into single tools where applicable.
5. **Format:** Return the tool descriptions strictly as Python-style list of strings.
6. **Metadata:** ** metadata is provided. your tool planning should refer to it. Now, generate structured list of ideal tool descriptions for the following planning steps: Planning Steps: (planning_ste) steps)
Ensure the output is formatted strictly Python list of strings, with each tool description containing only one function.
Figure 23: Tool planning prompt for ChemOrch.
acknowledge that the true answer often more nuanced, so please just use your best, judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist".
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
33

---

Tool Retrieval Prompt
will give you the task, tool name, and its description.
Your goal is to confirm whether the tool can be used to solve the task. Instructions:
1. You need to extract the final targets of the task and determine whether it requires specific tool or multiple tools.
2. First, you need focus on solving the final targets of the task.
3. Second. if the task requires multiple tools and this tool excels in one aspect of the task, it is also useful.
4. If metadata is provided, your choice of tool should be based on the requirements of the metadata.
Output format:
If the tool can used solving the task, return tool index only. It should be an integer.
2. If the tool can't i be used for solving the task, return the string "no" only. It should be string.
Figure 24: Tool retrieval prompt for ChemOrch.
Tool Distillation Prompt
will give you list of tools that have been screened, and they all related to the task. I will also give you the raw task. Problems:
1. Although these tools are all related to the task, some may be indirectly related to the task, or
the tool may not be an expert in the task.
2. Some tools may not be able to solve the final targets of the task.
Your goal is to check the tools and confirm whether they need remove some indirectly related tools.
Strategies for tool selection:
1. Pay attention to the tools' names. The tool name contains its function, and if the task needs the tool, the name often appears in the tool description.
2. Throw light on the task content. The content may clarify what tools or what kinds of tools are needed for the task.
Instructions:
1. Read the tools list and the task carefully, compare the tools' functions with the task, and check if the task marks specific tools to use.
2. Analyse the task and extract the final targets of the task. Regarding the tools can't solve the
final targets of the task as useless tools, you should focus the final targets of the task.
3. If the number of tools overnumbers the threshold: (threshold for tool distilling), you should think more about finding and removing indirectly related tools. In another situation, if the task only needs a few steps to solve, you should think more about using fewer tools.
4. If metadata is provided, your choice of tool should be based the requirements of the metadata.
Output format:
1. If the indirectly related tools are found, please return only the most indirectly related tool
index.
2. If no indirectly related tools are found, please return the string "no" only.
3. You should return the content described above without any prefixes or suffixes.
Figure 25: Tool distillation prompt for ChemOrch.
34

---

Code Script Generation Prompt
will give you key-value pairs that describe the task, module name, function name, and parameters with specific values. Your goal is to write script for calling the function with the given parameters. Instructions:
Import the module in this format:
"import ChemGen, (ociien.toos..nno. mools. _amne_naae" or "import poduleenamee
The module name will be given in the user prompt under the "module name" key.
2. Some parameters may need other packages. Please check the parameters and import the required packages.
3. Create variables for the parameters and fill them with the given values.
4. Call the function with the parameters and print the result. When printing the result, you need to describe what it means and not just print it.
Important:
The function name will be in the user prompt under the "function name" key.
Output format:
Return the script content only without any useless prefixes suffixes.
Figure 26: Code script generation prompt for ChemOrch.
Self- Self-Repairing: Error Catching Prompt
will give you a Python script and its error message.
Your goal is to fix the error in the script according to the error message.
Output format: Return the fixed script content only, without any useless prefixes suffixes like double quotation or back quote marks to mark this as a Python file.
Figure 27: Erro catching prompt for ChemOrch.
Self Self-Repairing: Effectiveness Checking Prompt
I will give you the task, the planning steps for solving the task, the script for the task, and its output.
Your goal is to determine whether the output is useful for solving the task.
The criteria for judging the uselessness of the output:
1. The output is an object without valid characters or numeric information. This one is important and often appears. Please pay attention.
2. The output is discordant irrelevant to the task.
3. The script does not follow the planning steps, focusing on checking the input variables and output format.
4. The output is not the accurate data the task requires.
If you find the output is useless, you modify the script according to the website given below:
(website)
Output format:
1. Return the "useful" string only if the script output is useful.
2. Return the modified script content only the output script is useless.
3. The modified script content should be without any useless prefixes suffixes like double
quotation or back quote marks.
Figure 28: Effectiveness checking prompt for ChemOrch.
35

---

Sufficiency Validation Prompt
I will give you a task and the results of some tools used to solve the task.
Your goal is to judge whether the present results are sufficient for solving the task.
Output format:
1. Return the string "yes" only the results are sufficient.
2. Return the string "no" only if the results are insufficient.
Figure 29: Sufficiency validation prompt for ChemOrch.
Web Search Prompt
I will give you a task and the planning steps for solving the task. Your goal is to search for the related information to solve the task online.
Figure 30: Web search prompt for ChemOrch.
Answer Assembly Prompt
will give you a task and some information generated from some tools for the task. Your goal is to analyze and solve the task. You can choose useful information generated from the tools make your answer accurate and correct. Instructions:
Read the task carefully and analyze its requirements.
2. Read the information given by the tools carefully and determine whether it can be used directly.
3. If the information cannot be used directly, you should transform it according to the task's requirements.
4. If you receive multiple answers but they different, you can process them in two ways:
(1) Choose the most accurate answer based on your judgment.
(2) If the answers have descriptions about how they are generated, you can output all answers with their descriptions and let the user choose the most accurate one.
5. Ensure the answer has good readability. You change the illustration format if needed.
Figure 31: Answer assembly prompt for ChemOrch.
LLM-as-a-Judge Evaluation Prompt
You are an evaluator for grading the quality of answers to chemistry questions. I will provide you with question, a predicted answer, and reference answer.
Your task is compare the predicted answer to the reference answer and assess how well they match meaning.
Consider factors such as accuracy, completeness, and clarity, even if the wording is different. Please first analyze the differences and similarities between the predicted and reference answers.
Then give a final score from 1 to 10, where 10 means perfect match in meaning and means completely incorrect.
Here is the question: [[Question]]
Here is the predicted answer: [[Answer]]
Here is the reference answer: [[Label]]
Respond only with the following format the final line:
Final score: X
Figure 32: LLM-as-a-Judae evaluation prompt for molecule captioning
36

---

LLM-aas-a-Judee Evaluation Prompt
You are an evaluator tasked with assessing whether a model's 's code-based solution to a problem is correct.
You will be given a problem statement, the model's ' generated code (predicted answer), and reference solution (correct answer).
Your job is to determine whether the model's code is functionally or logically equivalent to the reference solution. Please carefully compare the predicted code with the reference code, analyze their logic and behavior, and finally respond with either 'correct' or 'incorrect'. Problem: [[Question]]
Predicted Code: [[Answer]] Reference Code: [[Label]]
Figure 33: LLM-as-aaJudge evaluation prompt for tool usage
LLM--a-a-Judge Evaluation Prompt
You are an evaluator for evaluating whether response to a chemistry question correct or not.
will provide you with a question, the predicted answer, and correct answer. Your task is determine if the predicted answer matches the correct answer in meaning, even if the wording is slightly different. Please first compare the predicted answer with correct answer and analyze them, and finally respond with 'correct' or 'incorrect'. Here is the question: [[Question]]
Here is the predicted answer: [[Answer]] Here is the correct answer: [[Label]]
Figure 34: I.M--as-a-Judge evaluation prompt for other tasks
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We did it in abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation in discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
37

---

The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper do not include assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We include the details of experiments and open- spen-source the code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
38

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We open source the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
public/ htt//guieessee htteee/s//ooeSSuimissssoooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cccppblicc///cccccccccccccc.cclc guides/ SoodeSummii as wwnoooiiiioo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
39

---

Answer: [Yes]
Justification:
We include the details in our experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due the large computational requirements, multi-tima experiments are hard to be conducted. However, most of the experiments are conducted across different tasks and human experts to ensure its effectiveness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Yes, for training details, we include them our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
40

---

9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: www.rps..r.pppeeeeeeeeeeeeeeeeem cc/ pc/ppupicc//cceececcccpcccccccll
wtwbblic/Etiiseeee Etthi csQuidelines
Answer: [Yes]
Justification: Yes, follow the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we include the impact the broader impacts section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not include the high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
41

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we respect their licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. cowc dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Yes, we include all details of human evaluation in our draft.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
42

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
is not applied our experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing, editing, or formatting purposes.
Guidelines:
The answer NA means that the method development in this research does involve LLMSs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //neurips... ctciiisssccceeeeeee cee//onfereeee weteeeeeenneeeeeeeeeeeeeeee for what should or should not be described.
43


Reviews:
[{"review_id": "0K78GMi74V", "reviewer": "Reviewer_vh7w", "summary": "ChemOrch introduces a synthetic instruction–response generation framework for chemistry tasks, aiming to improve large language models’ (LLMs) chemical reasoning and tool-usage capabilities. The method comprises a two-stage process:\n\n1. task-controlled instruction generation: This step is guided by constraints including tasks and metadata, and refined using a difficulty reward model. \n\n2. tool-aware response construction: It combines instruction decomposition, tool planning, tool distillation, self-repair mechanisms to generate outputs using chemistry-specific tools like RDKit and PubChem.", "questions": "1. What is the computational cost associated with the proposed framework (i.e. M_diff finetuning, instruction–response pairs generation, etc)?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n1. The paper provides thorough prompt templates and code examples\n2. By supporting user-specified metadata the framework can adapt to additional tool configurations at runtime. And authors have provided detailed examples for the extensibility. \n3. The paper introduces a difficulty reward model trained via SFT to iteratively refine instructions, and its performance is supported by human evaluation protocol\n\n**Weakness**\n1. While the paper includes some ablation in Section 4.6, not all key components (e.g., the difficulty reward model M_diff) are analyzed. Given that ChemOrch consists of several modules, the paper may not fully reflect the author's insight on the proposed framework.\n2. While standard deviation are reported for most experiments, it would be beneficial to include error bars in figure 6 where the performance difference between Qwen Vanilla and Qwen Fine-tuned is not significant. \n3. While the experiments show that generated data can significant improvement of LLM chemistry capabilities, it's unclear how groundbreaking this framework actually is.", "was_revised": true, "final_justification": "Authors have mosly addressed my problems and concerns. And I am willing to increase my score to 4", "rebuttal": "We sincerely thank the reviewer for the valuable feedback and constructive suggestions, which have significantly improved the clarity and completeness of our work. Below, we provide detailed responses to each point raised:\n\n---\n\n**Q1: The paper includes some ablation, but the difficulty reward model ($M\\_{diff}$) is not analyzed. This leaves unclear how insightful the authors' understanding of the framework really is.**\n\n**A1:** Thank you for pointing this out. **We did place the evaluation of the difficulty reward model in Appendix F (due to the space constraint)**. Following your suggestion, we have now moved the relevant results into the main paper. Specifically, we provide both distribution comparisons and human evaluation results (see Figure 9 and Table 7 in the draft). The model’s predictions are well-aligned with human judgments, achieving an overall alignment rate of **87.0%**, as detailed below:\n\n| Difficulty Score | Human Alignment Rate |\n| ---------------- | -------------------- |\n| 1                | 100%                 |\n| 2                | 88.9%                |\n| 3                | 85.7%                |\n| 4                | 86.8%                |\n| 5                | 100%                 |\n| **Total**        | **87.0%**            |\n\nThese results demonstrate the effectiveness of the reward model in accurately capturing instruction difficulty in the chemistry domain.\n\n---\n\n**Q2: Figure 6 lacks error bars, and it's unclear if the performance difference between Qwen Vanilla and Qwen Fine-tuned is statistically significant.**\n\n**A2:** Thank you for this helpful suggestion. In response, we conducted multiple runs for the fine-tuning experiments and computed standard deviations. The results are summarized below:\n\n| Task                | Model                 | Vanilla | Fine-tuning-1 | Fine-tuning-2 | Fine-tuning-3 | Avg. Fine-tuned | Std Dev |\n| ------------------- | --------------------- | ------- | ------------- | ------------- | ------------- | --------------- | ------- |\n| Molecule Captioning | LLaMA-3.1-8B-Instruct | 1.64    | 2.25          | 2.27          | 2.25          | 2.257           | 0.0115  |\n|                     | Qwen-2.5-7B-Instruct  | 2.27    | 2.40          | 2.40          | 2.44          | 2.413           | 0.0231  |\n| Property Prediction | LLaMA-3.1-8B-Instruct | 0.203   | 0.655         | 0.645         | 0.645         | 0.648           | 0.0058  |\n|                     | Qwen-2.5-7B-Instruct  | 0.580   | 0.625         | 0.640         | 0.620         | 0.628           | 0.0104  |\n| Tool Usage          | LLaMA-3.1-8B-Instruct | 0.120   | 0.200         | 0.210         | 0.210         | 0.207           | 0.0058  |\n|                     | Qwen-2.5-7B-Instruct  | 0.360   | 0.440         | 0.440         | 0.460         | 0.447           | 0.0115  |\n\nAdditionally, for general Q\\&A experiments (also reported in Table 4 of the paper), the results with standard deviation over three runs are:\n\n| # Sample | LLaMA-3.1-8B-Ins. | Qwen-2.5-7B-Ins. |\n| -------- | ----------------- | ---------------- |\n| Vanilla  | 0.1391 ± 0.0071   | 0.2464 ± 0.0041  |\n| n=200    | 0.2812 ± 0.0041   | 0.2870 ± 0.0123  |\n| n=300    | 0.2464 ± 0.0082   | 0.2928 ± 0.0147  |\n| n=400    | 0.2725 ± 0.0082   | 0.3478 ± 0.0213  |\n| n=500    | 0.2899 ± 0.0041   | 0.3797 ± 0.0041  |\n\nThese results confirm that the observed improvements are consistent and statistically meaningful.\n\n---\n\n**Q3: While the experiments show improvements, it’s unclear how groundbreaking this framework actually is.**\n\n**A3:** Thank you for this valuable perspective. We recognize that some of the language in the manuscript (e.g., “groundbreaking”) may be overly strong. In the revised version, we will adopt a more objective tone, replacing such terms with more neutral alternatives like “novel,” “systematic,” or “comprehensive.”\n\nThat said, we would like to respectfully clarify the contributions of **ChemOrch**, which we believe represent meaningful and distinctive innovations:\n\n**Framework-Level Innovation:**\nChemOrch introduces the first **two-stage** chemistry-specialized synthetic pipeline—**task-controlled instruction generation** and **tool-aware response construction**—specifically designed to address chemistry-specific challenges such as rule alignment, task decomposition, and verifiability (see Sections 1 & 3, Fig. 2).\n\n**Advanced Technical Features:**\n\n* **Difficulty Reward Model:** Enables fine-grained complexity control, aligned with human judgments (Sec. 3.1, Appendix F).\n* **Tool Decomposition & Distillation:** Breaks complex tools into atomic components to facilitate accurate tool use and scientifically valid responses.\n* **Self-Repair & Sufficiency Validation:** Ensures robust error correction and response completeness (Sec. 3.3, Fig. 7), not found in previous frameworks.\n\n**Empirical Evidence:**\n\n* Human evaluation shows high correctness (85.14%) and instruction alignment (82.64%), with over **3× performance gain** over GPT-4o (Sec. 4.3).\n* Benchmarks generated by ChemOrch reveal LLM weaknesses in underrepresented tasks that other benchmarks miss (Table 3).\n* Fine-tuning on ChemOrch data yields **significant accuracy gains** in chemistry tasks (Sec. 4.5, Fig. 6, Table 4).\n\n**Broader Impact:**\nChemOrch democratizes access to scalable, verifiable chemistry datasets, lowering barriers to developing high-performing, chemistry-aware LLMs (Appendix A).\n\n---\n\n\n**Q4: What is the computational cost of the proposed framework (e.g., $M_{diff}$ fine-tuning, instruction–response generation)?**\n\n**A4:** Thank you for raising this important point. We provide the following clarifications:\n\n- **$M_{diff}$ Training Cost:**  \n  The difficulty reward model was trained on **3,390** examples using **4× A100 (80GB)** GPUs. The total training time was under **1 hour**. We will add this detail to the appendix in the revision.\n\n- **Instruction–Response Generation Cost:**\n\n  1. As mentioned in Lines 240–250, the cost of generating an instruction–response pair using commercial APIs (e.g., o3-mini) is highly affordable—typically **under \\$0.05 per pair**. For fine-tuning on 100 examples, this translates to **< \\$5**, or even **< \\$1** if using GPT-4o, making our framework cost-effective.\n\n  2. For local generation using open-source models, we benchmarked several models using Replicate's cloud inference service. A typical instruction–response pair involves approximately **3.5K tokens**, and with **5 concurrent processes**, generating **100 pairs** generally completes within **1 hour** for all tested models. The generation speed and latency are summarized below:\n\n     | Model                                  | Token/s | Time (s)   |\n     |----------------------------------------|---------|------------|\n     | `deepseek-ai/DeepSeek-V3-0324`         | 70.42   | 56.802     |\n     | `deepseek-ai/deepseek-r1`              | 24.02   | 166.528    |\n     | `meta/meta-llama-3.1-405b-instruct`    | 29.74   | 134.499    |\n     | `meta/meta-llama-3-70b-instruct`       | 41.73   | 95.854     |\n\n  3. The average token consumption for each module in the pipeline is as follows (also illustrated in Figure 4 of the paper):\n\n     | Module            | Token Consumption |\n     |-------------------|-------------------|\n     | Embedding Token   | 120               |\n     | Self-Repairing    | 231               |\n     | Code Script       | 902               |\n     | Tool Distillation | 1,095             |\n     | Web Search        | 1,106             |\n     | Answer Generation | 1,165             |\n     | Validation        | 1,472             |\n     | Tool Selection    | 4,094             |\n\nTogether, these metrics confirm that **ChemOrch** is **scalable**, **computationally efficient**, and practical for both commercial API-based and open-source local deployments.\n\n\n---\n\nOnce again, we sincerely appreciate the reviewer’s thoughtful and constructive feedback. Your suggestions have significantly improved the clarity, rigor, and presentation of our work.\n\nIf you find our responses helpful and feel that they address your concerns, we would be truly grateful if you would consider revisiting your evaluation.\n\nThank you very much!"}, {"review_id": "h76gOBCbEE", "reviewer": "Reviewer_nRFr", "summary": "ChemOrch introduces a two-stage framework that first generates chemistry-specific instructions under explicit task, difficulty and constraint control, then constructs tool-grounded responses through planning, tool distillation and multi-stage self-repair, ensuring executability and chemical validity", "questions": "Please refer to weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The introduction of difficulty and constraint control helps the instruction generation process yield diverse, level-appropriate chemistry prompts.\n2.  ChemOrch combines task-controlled instruction generation with tool-aware response construction, incorporating difficulty calibration, tool distillation, and self-repair. I think these designs can keep responses executable and verifiable.\n3. The experimental results are promising. Test sets synthesized by ChemOrch track existing benchmarks closely. The paper also includes under-represented tasks such as the lipophilicity prediction task.\n4. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. The paper focuses largely on SMILES inputs and outputs. To what extent can the proposed framework be readily adapted to alternative string-based molecular encodings—such as graph-adjacency lists, tree-structured or JSON representations—and what specific effort or modifications would this entail?\n2. All accuracy numbers are produced with the LLM-as-a-Judge rubric. Without human or rule-based ground truth, evaluation is vulnerable to the same hallucination, bias, and coherence errors it tries to measure.", "was_revised": true, "final_justification": "Thanks to the authors for their rebuttal.\n\nI agree with the authors that SMILES can be easily converted to other formats during preprocessing. I strongly encourage authors to include more formats in their pipeline.\n\nOverall, I think this is a good work.", "rebuttal": "We sincerely thank you for the valuable feedback and constructive suggestions. Below, we provide detailed responses:\n\n---\n\n\n**Q1: The paper focuses largely on SMILES inputs and outputs. To what extent can the proposed framework be readily adapted to alternative string-based molecular encodings—such as graph-adjacency lists, tree-structured or JSON representations—and what specific effort or modifications would this entail?**\n\n**A1:** Thank you very much for your thoughtful question. Our focus on SMILES inputs and outputs was motivated by its status as the most widely used and canonical representation in cheminformatics, ensuring broad compatibility and comparability with existing literature. However, we fully acknowledge the importance of supporting alternative molecular encodings, such as graph-based, tree-structured, or JSON representations.\n\nImportantly, **ChemOrch can be readily extended to handle alternative molecular formats by simply introducing appropriate conversion functions in the preprocessing stage**. For example, to support graph-based representations, one only needs to add a transformation module before the main task. Below, we provide a code snippet illustrating how ChemOrch can seamlessly convert a graph representation:\n\n```python\ndef graph_to_iupac_name(graph):\n    mol = Chem.RWMol()\n    atom_idx_map = {}\n    \n    # Add atoms\n    for i, atom_info in enumerate(graph[\"atoms\"]):\n        atom = Chem.Atom(atom_info[\"element\"])\n        atom.SetFormalCharge(atom_info.get(\"charge\", 0))\n        atom.SetIsAromatic(atom_info.get(\"is_aromatic\", False))\n        idx = mol.AddAtom(atom)\n        atom_idx_map[i] = idx\n\n    # Add bonds\n    bond_order_map = {\n        \"single\": Chem.rdchem.BondType.SINGLE,\n        \"double\": Chem.rdchem.BondType.DOUBLE,\n        \"triple\": Chem.rdchem.BondType.TRIPLE,\n        \"aromatic\": Chem.rdchem.BondType.AROMATIC\n    }\n\n    added = set()\n    for a1, neighbors in graph[\"bonds\"].items():\n        for a2, bond_type in neighbors:\n            if (a2, a1) in added:\n                continue\n            bt = bond_order_map.get(bond_type.lower())\n            if bt is None:\n                raise ValueError(f\"Unknown bond type: {bond_type}\")\n            mol.AddBond(atom_idx_map[a1], atom_idx_map[a2], bt)\n            added.add((a1, a2))\n\n    mol.UpdatePropertyCache(strict=False)\n    Chem.SanitizeMol(mol)\n    return pcp.get_compounds(Chem.MolToSmiles(mol, canonical=True), 'smiles')[0].iupac_name\n````\n\nTo verify effectiveness, we tested this approach on 20 randomly sampled molecular graphs and achieved a `100% (20/20)` success rate in conversion. This demonstrates ChemOrch's strong flexibility and extensibility for handling various molecular representations beyond SMILES.\n\nWe will include these implementation details and practical examples in the revised manuscript. Thank you again for your helpful suggestion.\n\n---\n\n**Q2: All accuracy numbers are produced with the LLM-as-a-Judge rubric. Without human or rule-based ground truth, evaluation is vulnerable to the same hallucination, bias, and coherence errors it tries to measure.**\n\n**A2:** Thank you for raising this critical point. We would like to clarify that the evaluation in our study is not solely reliant on automated scoring without validation. **In fact, we incorporated human evaluation at several key stages, particularly when assessing the quality of generated instruction–response pairs, where extensive human annotation was performed.**\n\nRegarding rule-based ground truth: For simple binary tasks, we agree that rule-based metrics are suitable and, where feasible, we will employed them. However, rule-based methods themselves may introduce inaccuracies (e.g., missing relevant keywords), and recent literature has increasingly adopted the LLM-as-a-Judge paradigm for its broader applicability and contextual understanding.\n\nTo further assess the reliability of automated scoring, we conducted a human–LLM agreement study across three tasks and two representative models (`LLaMA-3.1-8B-Instruct` and `Qwen-2.5-7B-Instruct`):\n\n* **Binary alignment (Property Prediction):** up to **99.75%**\n* **Binary alignment (Tool Usage):** up to **97%**\n* **Score-based correlation (General QA):** average **Pearson r = 0.7955**, all results statistically significant\n\nThese results confirm strong consistency between automated and human assessments. Please see the summary tables below:\n\n---\n\n**Binary Accuracy – Property Prediction**\n\n| Task                | Model                     | Batch 1      | Batch 2      | Avg. Alignment |\n| ------------------- | ------------------------- | ------------ | ------------ | -------------- |\n| Property Prediction | LLaMA-3.1-8B-Instruct (1) | 49/50 (98%)  | 50/50 (100%) | 99%            |\n|                     | LLaMA-3.1-8B-Instruct (2) | 50/50 (100%) | 50/50 (100%) | 100%           |\n|                     | Qwen-2.5-7B-Instruct (1)  | 50/50 (100%) | 50/50 (100%) | 100%           |\n|                     | Qwen-2.5-7B-Instruct (2)  | 50/50 (100%) | 50/50 (100%) | 100%           |\n| **Overall Average** | —                         | —            | —            | **99.75%**     |\n\n---\n\n**Binary Accuracy – Tool Usage**\n\n| Task                | Model                     | Batch 1     | Batch 2      | Avg. Alignment |\n| ------------------- | ------------------------- | ----------- | ------------ | -------------- |\n| Tool Usage          | LLaMA-3.1-8B-Instruct (1) | 47/50 (94%) | 46/50 (96%)  | 95%            |\n|                     | LLaMA-3.1-8B-Instruct (2) | 47/50 (94%) | 50/50 (100%) | 97%            |\n|                     | Qwen-2.5-7B-Instruct (1)  | 48/50 (96%) | 50/50 (100%) | 98%            |\n|                     | Qwen-2.5-7B-Instruct (2)  | 48/50 (96%) | 50/50 (100%) | 98%            |\n| **Overall Average** | —                         | —           | —            | **97%**        |\n\n---\n\nWe will further detail our evaluation protocol and present additional human–LLM agreement analyses in the revised manuscript to ensure transparency and rigor. Thank you for highlighting this important aspect.\n\n--- \n\nThank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!"}, {"review_id": "xWBltUDtzi", "reviewer": "Reviewer_XBcL", "summary": "The submission \"ChemOrch: Empowering LLMs with Chemical Intelligence via Groundbreaking Synthetic Instructions\" describes a framework for automatic construction of instruction-response pairs for large language models tailored to chemistry tasks. This is important as the number of available datasets in this domain is very limited. The framework provides data of comparable quality to existing approaches relying on human supervision but of significantly higher diversity. The utility of the framework of the framework was demonstrated to evaluate existing large language models and finetune them for further performance improvement. Overall, the study is well written and provides a significant advance in the field. I think it can benefit from some further improvements that should be addressed before publication. Please find my detailed comments below.", "questions": "Major aspects:\nWhen evaluating the capability of ChemOrch to be used for finetuning, it seems that ChemOrch is both used for generating training data and for performance evaluation. This could result in data leakage. I think it is important that at least one example is provided where ChemOrch is used for finetuning but performance on other datasets such as the ones that already exist in literature (ChemLLMBench, Mol-Instructions) is shown.\n\nRecently, another benchmark named ChemBench was published (https://doi.org/10.1038/s41557-025-01815-x). I think at least some comparison of ChemOrch to ChemBench should be provided.\n\nThe model evaluation metrics should be explained explicitly in the main text as the text is otherwise very hard to follow. It should be mentioned at least what the best and worst performance metric values are for each metric.\n\nPage 8: The authors state: \"Similarly, in general, chemistry Q&A, we observed a comparable increase, suggesting enhanced domain understanding and reasoning ability.\" I do not agree that a better performance directly suggests enhanced domain understanding and reasoning ability. I think it merely suggests a higher chance to provide results that are closer to the reference values.\n\nI think that the comparison to existing approaches is relatively minimal. One existing study that could possibly provide more tasks to compare ChemOrch to previously tested tasks is ChemCrow (https://doi.org/10.1038/s42256-024-00832-8).\n\nI think that more context should be provided regarding to human evaluation of response (section 4.3). I think it should be provided in the main text how many samples were evaluated in total and how the metrics was calculated. For instance, is an example with a single error as inaccurate as an example with many errors?\n\nBased on Figure 6, it seems that the LLaMA model seems to benefit significantly more from finetuning than Qwen. Do the authors have an idea of the origin of this and the possible implications?\n\nThe submission mentions \"Transision State Search\" as one of the reasoning tasks. I could not find an example for this task in the Appendix. I think it is important to provide at least one example for each of the task categories to make it more clear to the reader what type of input-output pairs are considered for each of these.\n\nMinor aspects:\nTable 2: The evaluation results for MC are supposed to range between 1 and 5 (based on the table caption) but many results are even above 6. I think this is confusing and requires further clarification.\n\nFigure 3: It would be insightful to compare the distrubtions to the reference datasets ChemLLMBench and Mol-Instructions.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I think the quality is fair as the framework fills an important niche in the field but it should carefully address concerns regarding data leakage.\nI think the clarity is good as all the important components of the developed framework are illustrated well.\nI think the significance is fair as the framework is demonstrated to be directly applicable to finetuning existing models and improve their performance significantly. However, it is important to evaluate its utility for improving the general capabilities of a model on other datasets as well.\nI think originality is good as the framework is general and can be applied to many domains of science.", "was_revised": true, "final_justification": "I appreciate the efforts of the reviewers to address all my comments. I think most of them have been resolved satisfactorily. Therefore, I decided to raise my review score to 4. I think that ChemOrch is a clear advance in the field that will be helpful for large language model development in chemistry. For a higher score, I think a more comprehensive set of tasks relevant in the chemistry domain for model evaluation would be needed, especially with respect to more complex tasks that involve several step for their solution.", "rebuttal": "We sincerely thank you for the detailed and thoughtful feedback. We address each point in detail below:\n\n**Q1: Concern about potential data leakage. Can you show evaluation on external datasets like ChemLLMBench or Mol-Instructions?**\n\nThank you for raising this important concern. We apologize for any confusion and **respectfully clarify that this issue is already addressed in the manuscript (Lines 676–678)**:\n\n> “For the fine-tuning experiments described in subsection 4.5, each of the three tasks (property prediction, tool usage, and molecule captioning) includes 400 samples for training and 400 for testing, with the test data sampled from the original benchmark.”\n\nSpecifically:\n- For **Property Prediction**, **Molecule Captioning** and **Tool Usage**, test sets are **fully sourced** from **ChemLLMBench**.\n- In the **Tool Usage** task, test prompts are derived from real-world function documentation (not model-generated), and molecules are randomly **sampled from ChemLLMBench**. Evaluation is based on exact output match, ensuring objectivity.\n\nRegarding the General Q&A, we did consider using subsets of MMLU and SciBench. However, the chemistry questions in these datasets are mostly focused on mathematical calculations, with relatively few items testing core chemical knowledge. While we acknowledge that our current Q&A set is generated by ChemOrch (as a necessary compromise), we have nonetheless applied data filtering to ensure quality, as stated in Lines 680–681:\n\n> “For all testing sets, we conduct a human evaluation to filter out low-quality data points.”\n\nThis filtering follows the rigorous protocol in Appendix E. We will clarify this in the revision to ensure full transparency of our evaluation pipeline.\n\n---\n\n**Q2: Comparison to ChemBench**\n\nThank you for this excellent suggestion. While ChemBench is a valuable benchmark for systematically evaluating LLMs’ chemical knowledge and reasoning, our ChemOrch framework offers several clear advantages:\n\n1. **Scalable data generation**: ChemOrch automatically produces diverse, high-quality chemistry instruction–response pairs at scale, without manual labeling.  \n2. **Broader and practical task coverage**: It includes not only Q&A but also complex tasks like property prediction and tool-based reasoning with executable outputs.  \n3. **Targeted for model improvement**: ChemOrch can generate data tailored to specific weaknesses, supporting both evaluation and fine-tuning.\n\n\nWe now compared **ChemOrch** to **ChemBench**, **ChemLLMBench**, and **Mol-Instructions** on two diversity metrics:\n\n| Dataset             | APS ↓ | Remote-Clique ↑ |\n| ------------------- | ----- | --------------- |\n| **ChemOrch (ours)** | 0.779 | 0.661           |\n| ChemLLMBench        | 0.884 | 0.453           |\n| Mol-Instructions    | 0.765 | 0.683           |\n| ChemBench           | 0.784 | 0.613           |\n\nThese metrics indicate that ChemOrch achieves **higher instruction diversity than ChemBench**. \n\n---\n\n**Q3: The evaluation metrics are unclear. Please define them clearly in the main text.**\n\nThank you for pointing this out. We agree that metric definitions are crucial for clarity. Below are the metrics we use, now explicitly described in the revised manuscript:\n\n- **Property Prediction Accuracy**: Proportion of correctly predicted properties (range: 0–1). Higher is better.\n- **Tool Usage Accuracy**: Based on whether tool outputs match expected function calls (range: 0–1). Higher is better.\n- **Molecule Captioning**: Score on a **1–10 scale**. Higher is better.\n- **General Chemistry Q&A**: Score on a **1–10 scale**. Higher is better.\n- **Instruction Following Rate / Factual Correctness Rate**: Percentage of responses rated by humans as fully instruction-compliant or scientifically accurate (range: 0–1).\n- **Diversity Metrics**:\n  - **Average Pairwise Similarity (APS)**: Lower is better.\n  - **Remote-Clique Score**: Higher indicates broader semantic coverage.\n\n\n---\n\n**Q4: Better performance on chemistry Q&A doesn't necessarily prove domain understanding or reasoning.**\n\nThank you for the thoughtful critique. We agree that **higher benchmark scores do not automatically imply true domain understanding or reasoning**.\n\nOur original phrasing—“suggesting enhanced domain understanding”—may overstate this link. In the revision, we now clarify that:\n\n> “The performance gains indicate improved model accuracy on diverse and reasoning-intensive chemistry tasks.”\n\n\n---\n\n**Q5: Comparison with ChemCrow?**\n\nThank you for highlighting this. We agree that **ChemCrow** is a notable framework in the LLM-for-chemistry space. However, its **goal and output** differ significantly from ChemOrch:\n\n| Aspect        | **ChemOrch (ours)**                                          | **ChemCrow**                                        |\n| ------------- | ------------------------------------------------------------ | --------------------------------------------------- |\n| Objective     | Instruction–response dataset synthesis & fine-tuning         | Autonomous agent for chemistry task execution       |\n| Output        | Diverse, calibrated instruction datasets (used for training or evaluation) | Agent trajectories / tool usage logs (not datasets) |\n\nDue to these distinctions, a direct performance comparison is not meaningful. However, we have expanded the **related work section** in the revision to more clearly articulate how ChemOrch complements systems like ChemCrow.\n\n**Q6: Please provide more context in the main text about human evaluation.**\n\nThank you for this helpful suggestion. While the full description of our human evaluation setup was previously placed in Appendix G, we agree that its key elements should appear in the main text. In the revised draft, we clarify that a total of **400 samples** were assessed by **four Ph.D. students with backgrounds in computational chemistry**. The two human-evaluated metrics—**Instruction Following Rate** and **Factual Correctness Rate**—are both computed using the following formula:\n\n$$\n\\text{rate} = \\frac{\\text{Number of samples judged as correct}}{\\text{Total number of evaluated samples}}\n$$\n\nA sample is counted as correct only if it fully satisfies the corresponding criterion: instruction adherence or factual accuracy. If a response contains even a single substantive error—whether it deviates from the instruction or includes incorrect scientific content—it is considered incorrect. This conservative scoring scheme ensures consistency across annotators and avoids inflating the evaluation scores due to partial correctness. \n\n---\n\n**Q7: Figure 6 shows that LLaMA benefits more from fine-tuning than Qwen. Do the authors have an explanation for this?**\n\nThank you for this insightful question. We believe the discrepancy arises from two key factors:\n\n1. **Ceiling effect**:  \n   Qwen starts from a much stronger baseline, leaving less room for improvement via fine-tuning. In contrast, LLaMA has more room to improve.\n\n2. **Data–model alignment**:  \n   The style, tokenization, or instruction formulation of the generated data by ChemOrch may align more closely with LLaMA’s pretraining distribution, making it more receptive to fine-tuning with ChemOrch data.\n\nWe have included this discussion in the revised version.\n\n---\n\n**Q8: The manuscript references “Transition State Search” as one of the tasks, but no example is provided.**\n\nThank you for pointing this out.\nWe have now added a **concrete example** to the Appendix. Due to the word count limitation, below is a shortened excerpt:\n\n```json\n\"instruction\": \"Calculate the transition state energy using the MP2 method for the E2 reaction given the 3D structure: [...]\",\n\"response\": \"The transition state energy value for the E2 reaction using the MP2 method, based on the provided data, is -179.132.\\n\\n### Explanation of the Answer\\n\\n [...]\\n\\n### Reasoning Process\\n\\n1. **Understand the Problem**: [...]\"\n```\n\n---\n\n**Q9: Table 2 shows Molecule Captioning scores above 5, but the caption says the range is 1–5. Please clarify.**\n\nThank you for catching this inconsistency. The actual **scoring range is 1–10**, as used throughout our human evaluation. The caption in Table 2 was mistakenly written as 1–5. We have modified it in our draft.\n\n---\n\n**Q10: Figure 3 would benefit from a comparison of word count distributions to reference datasets like ChemLLMBench and Mol-Instructions.**\n\nThank you for the excellent suggestion.  We are unable to provide the updated Fig3 in response (image is not allowed by the rebuttal policy). However, we include the **average instruction word counts** for tasks in **ChemLLMBench**, and **Mol-Instructions** in the tables below. We have also updated **Figure 3** in the revised draft as suggested.\n\n---\n\n**ChemOrch**\n\n| Task                  | Avg. Instruction Word Count |\n| - | - |\n| mole_caption          | 519.70                      |\n| mole_design           | 481.61                      |\n| property_pred_bace    | 161.00                      |\n| property_pred_bbbp    | 139.01                      |\n| property_pred_clintox | 171.04                      |\n| property_pred_hiv     | 201.02                      |\n| property_pred_tox21   | 124.03                      |\n| reaction_pred         | 152.72                      |\n| retro                 | 171.16                      |\n\n---\n\n**ChemLLMBench**\n\n| Task          | Avg. Instruction Word Count |\n| - | - |\n| chem_disease  | 200.95                      |\n| chem_entity   | 42.94                       |\n| chem_protein  | 243.44                      |\n| MCQ           | 39.65                       |\n| true_or_false | 12.91                       |\n| openq         | 10.77                       |\n\n\n\nOnce again, we appreciate your insightful and constructive comments. If you feel that our responses have sufficiently addressed your concerns, we would be truly grateful if you would consider revisiting your evaluation. Thanks you!"}, {"review_id": "5WaPuhBwxt", "reviewer": "Reviewer_4t1k", "summary": "The paper introduces ChemOrch, a two-stage framework that generates chemical instruction–response pairs to address the shortage of high-quality domain data for LLMs. Comprehensive experiments show the results have higher diversity. LLMs trained on the ChemOrch show improved capabilities.", "questions": "Q1. What types of tools are specifically required for Chemistry Q&A tasks? Given that existing work has shown LLMs can already perform reasonably well on such questions, what additional capabilities do these tools provide, and in what ways do they further enhance the performance of LLMs?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "S1. The paper is clearly motivated by three concrete gaps, including data scarcity, mismatch between generic synthetic pipelines and chemical rules, and lack of controllable diversity.\n\nS2. The framework yields inexpensive, verifiable data. \n\nS3. Multiple metrics are used to show the effectiveness of the proposed method. \n\nW1. My primary concern lies in the accuracy of the generated instruction–response pairs. The current framework does not appear to guarantee correctness in generation. If the synthesized data contains factual errors or inconsistencies, it raises doubts about using it to fine-tune the LLM model.\n\nW2. The early stopping mechanism is not clear to me. It is unclear why terminating execution early is preferable when the final outputs may still be incorrect. \n\nW3. Certain parts of the evaluation rely heavily on human judgment.  This may introduce potential bias.", "was_revised": true, "final_justification": "I appreciate the author’s efforts in addressing the comments. However, my concern remains insufficiently addressed. I am still unclear on how generating synthetic data without any accuracy guarantees can be considered acceptable. In fact, the results presented by the authors appear to reinforce this concern. Therefore, I intend to maintain my current score.", "rebuttal": "We sincerely thank you for the thoughtful and detailed feedback. Below we respond to each point individually.\n\n------\n\n**W1. Concern about the factual accuracy of generated instruction–response pairs.**\n\n**A1:** Thank you for raising this important concern. We respectfully believe there may have been a misunderstanding about the nature of our data generation pipeline.\n\nTo clarify: **ChemOrch is explicitly designed to \\*avoid\\* relying on free-form LLM outputs for response generation**, precisely **because of the risk of hallucinations and factual errors**. Instead, we adopt a **tool-grounded generation framework**, where responses are produced through structured **tool calls** executed by trusted chemistry engines such as **RDKit** and **PubChem**.\n\nThis approach ensures that outputs are not hallucinated, but rather **scientifically derived from verified computations and databases**. We further apply multiple safeguards—including **error detection**, **self-repairing logic**, and **sufficiency validation** (Sections 3.2–3.3)—to verify the completeness and correctness of generated responses before they are accepted.\n\nIn addition, we conduct **extensive human evaluation** to empirically validate the quality of instruction–response pairs. As reported in Section 4.3 and Appendix G:\n\n- ChemOrch-generated data achieves **82.64% instruction-following accuracy**\n- And **85.14% factual correctness**, as judged by domain experts\n\nThese results, along with a transparent annotation protocol and clearly reported sample sizes, offer **strong empirical support** that the generated data is of high quality and appropriate for downstream fine-tuning.\n\nWe hope this clarifies the design intent of ChemOrch and reassures you that factual accuracy is both a **central motivation** and a **validated strength** of our framework.\n\n------\n\n**W2. Unclear motivation behind the early stopping mechanism.**\n\n**A2:** Thank you for highlighting this point. We apologize for the lack of clarity.\n\nThe **early stopping mechanism** in ChemOrch is not used to skip or prematurely terminate response generation. Instead, it is purely a **computational efficiency optimization**, applied **only after the system determines that all instruction requirements have been successfully satisfied**.\n\nTo ensure no part of the instruction is overlooked, early stopping is always followed by a **sufficiency validation** step. If this check identifies any missing or incomplete outputs, execution resumes and additional tool calls are triggered.\n\nIn short:\n\n- Early stopping is **never applied at the expense of correctness**\n- It only eliminates **redundant computation** when a task is verifiably complete\n\nWe will clarify this mechanism further in the revised manuscript.\n\n------\n\n**W3. Concerns about human judgment introducing bias in evaluation.**\n\n**A3:** We appreciate your thoughtful observation. We address this concern through a robust, multi-pronged evaluation framework combining both human annotation and LLM-as-a-Judge validation.\n\n---\n\n **1. Human Evaluation Protocol – Designed to Minimize Bias**\n\nWe adopt several measures to reduce subjectivity:\n\n- **Fact-based assessment**: Annotators were encouraged to consult authoritative chemical resources (e.g., PubChem, textbooks) when uncertain.\n- **Diverse reviewer pool**: Our evaluation involved **nine annotators** (3 undergraduates + 6 PhD students) from ML, chemistry, and chemical engineering backgrounds.\n- **Blind and randomized assignment**: Evaluation was conducted blindly across models and batches to avoid anchoring bias.\n\nTo further validate the internal consistency of our human evaluations, we computed the **inter-annotator agreement** using **Cohen’s Kappa coefficient** across three separate batches. As shown below, the Kappa scores indicate **substantial to almost perfect agreement**:\n\n| Batch   | Kappa Score |\n| ------- | ----------- |\n| Batch 1 | 0.862       |\n| Batch 2 | 0.623       |\n| Batch 3 | 0.722       |\n\nThese results demonstrate that human raters maintained a high degree of consistency across tasks, further supporting the reliability of our human evaluation protocol.\n\n---\n\n**2. Necessity of Human Evaluation in Open-Domain Tasks**\n\nWe respectfully argue that human evaluation is indispensable for open-domain chemistry tasks, for the following reasons:\n\n1. **Automatic metrics are insufficient**: Metrics like BLEU, ROUGE, or embedding similarity only assess surface-level overlap. They cannot reliably detect serious scientific errors, such as incorrect property values, invalid tool usage, or misapplied reaction mechanisms.\n\n2. **Chemistry tasks require expert reasoning**: Many tasks involve multi-step logic, adherence to domain-specific rules, and scientific justification. These qualities are hard to quantify using purely automated metrics, especially for open-ended or complex instructions.\n\n3. **Human evaluation remains the gold standard**: As supported by prior work [1–4], expert human judgment remains essential for evaluating synthetic data and LLM performance in scientific domains. In our study, annotators relied on structured criteria and external references to ensure objectivity.\n\n---\n\n**3. LLM-as-a-Judge (High Alignment with Human Judgment)**\n\nTo further validate reliability, we conducted a human–LLM agreement study across three tasks and two strong models (`LLaMA-3.1-8B-Instruct` and `Qwen-2.5-7B-Instruct`).\n\n- **Binary alignment (Property Prediction)**: up to **99.75%**\n- **Binary alignment (Tool Usage)**: up to **97%**\n- **Score-based correlation (General QA)**: average **Pearson r = 0.7955**, all **statistically significant**\n\nThese results confirm strong agreement between automated and human judgments. Summary tables below:\n\n---\n\n**Binary Accuracy – Property Prediction**\n\n| Task                | Model                     | Batch 1      | Batch 2      | Avg. Alignment |\n| ------------------- | ------------------------- | ------------ | ------------ | -------------- |\n| Property Prediction | LLaMA-3.1-8B-Instruct (1) | 49/50 (98%)  | 50/50 (100%) | 99%            |\n|                     | LLaMA-3.1-8B-Instruct (2) | 50/50 (100%) | 50/50 (100%) | 100%           |\n|                     | Qwen-2.5-7B-Instruct (1)  | 50/50 (100%) | 50/50 (100%) | 100%           |\n|                     | Qwen-2.5-7B-Instruct (2)  | 50/50 (100%) | 50/50 (100%) | 100%           |\n| **Overall Average** | —                         | —            | —            | **99.75%**     |\n\n---\n\n**Binary Accuracy – Tool Usage**\n\n| Task                | Model                     | Batch 1     | Batch 2      | Avg. Alignment |\n| ------------------- | ------------------------- | ----------- | ------------ | -------------- |\n| Tool Usage          | LLaMA-3.1-8B-Instruct (1) | 47/50 (94%) | 46/50 (96%)  | 95%            |\n|                     | LLaMA-3.1-8B-Instruct (2) | 47/50 (94%) | 50/50 (100%) | 97%            |\n|                     | Qwen-2.5-7B-Instruct (1)  | 48/50 (96%) | 50/50 (100%) | 98%            |\n|                     | Qwen-2.5-7B-Instruct (2)  | 48/50 (96%) | 50/50 (100%) | 98%            |\n| **Overall Average** | —                         | —           | —            | **97%**        |\n\n---\n\nTogether, our inter-annotator agreement, expert-informed annotation protocol, and strong human–LLM alignment provide strong evidence for the robustness and fairness of our evaluation process.\n\n---\n\n#### **2. Score-based Evaluation: General Q&A**\n\n| Model                     | Pearson r | P-value      |\n| ------------------------- | --------- | ------------ |\n| Llama-3.1-8B-Instruct (1) | 0.741     | 4.82 × 10⁻¹⁶ |\n| Llama-3.1-8B-Instruct (2) | 0.728     | 1.88 × 10⁻¹⁶ |\n| Qwen-2.5-7B-Instruct (1)  | 0.859     | 9.31 × 10⁻²⁸ |\n| Qwen-2.5-7B-Instruct (2)  | 0.854     | 2.20 × 10⁻²⁸ |\n| **Average**               | **0.796** | —            |\n\nThese **high correlations (r ≈ 0.73–0.86)** with highly significant p-values demonstrate that our LLM judge is consistent with human judgments even in open-ended tasks.\n\n------\n\n**Q1. What specific tools are required for Chemistry Q&A tasks, and how do they enhance LLM performance?**\n\n**A4:** Thank you for this insightful question.\n\nWhile LLMs can handle many chemistry-related questions reasonably well, they often fall short when dealing with:\n\n- **Uncommon or specialized topics**\n- **Out-of-date information**\n- **Facts that require external validation**\n\nTo address this, ChemOrch integrates **web retrieval** as the primary tool for Chemistry Q&A tasks. This allows the model to access up-to-date scientific knowledge from reliable sources, verify its outputs with external evidence to reduce hallucinations, and accurately handle rare or niche chemistry questions.\n\nWe will expand the explanation of tool roles in Chemistry Q&A in the revised manuscript to make these enhancements more explicit.\n\n------\n\n[1] Huang, Yue, et al. *\"Datagen: Unified synthetic dataset generation via large language models.\"* ICLR 2024.\n\n[2] Bran, Andres M., et al. *\"Augmenting large language models with chemistry tools.\"* Nature Machine Intelligence, 2024.\n\n[3] Long, Lin, et al. *\"On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey.\"* ACL (Findings), 2024.\n\n[4] Mirza, Adrian, et al. \"A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists.\" *Nature Chemistry* (2025): 1-8.\n\n------\n\nOnce again,  we deeply appreciate your careful review and constructive comments. If you feel that our responses have sufficiently addressed your concerns, we would be truly grateful if you would consider revisiting your rating. Thanks you!"}]

Author rebuttal:

Meta-review: (a) Summary of scientific claims and findings
This paper presents ChemOrch, a framework for generating chemistry-specific instruction–response datasets to improve LLM chemical intelligence. The system introduces a two-stage process: (1) task-controlled instruction generation with difficulty calibration, and (2) tool-aware response construction with planning, distillation, and self-repair. Experiments show that ChemOrch produces diverse and verifiable data, improves LLM performance on chemistry tasks, and can surface weaknesses in existing models. Fine-tuning with ChemOrch data yields notable gains across property prediction, molecule captioning, tool usage, and general chemistry Q&A tasks.

(b) Strengths
- Clear framework design tailored to chemistry’s structured nature.
- Incorporates difficulty control and tool-grounded responses, reducing hallucinations and enhancing verifiability.
- Demonstrates cost-effectiveness and scalability for generating data.
- Empirical evaluations show improved LLM performance and diversity advantages over existing benchmarks.
- Rebuttal added clarity on evaluation protocols, computational cost, and robustness to alternative molecular encodings.

(c) Weaknesses
- Some novelty concerns: methods build on established ideas (tool grounding, self-repair, difficulty modeling) with more emphasis on integration.
- Accuracy of generated data remains imperfect (≈85% correctness), raising concerns about reliability for fine-tuning. However, I understand a perfect result is not possible at the current stage.
- Evaluation partly relies on LLM-as-a-judge and human ratings, which, despite mitigation efforts, leave room for bias.
- Benchmarking could be broader: limited external validation against datasets like ChemBench or ChemCrow was only partially addressed.

(d) Reasons for decision
The work fills an important gap in chemistry-specific LLM data generation and shows clear empirical value. While accuracy concerns and evaluation methodology limit confidence, the authors provided substantial rebuttal evidence that the system is effective, cost-efficient, and extensible. Given its practical contributions and strong empirical validation, this paper merits acceptance. However, the novelty and open questions about evaluation rigor make it more appropriate for poster.

(e) Discussion and rebuttal period
Reviewers were split: some emphasized the practical strengths and novelty of integrating difficulty control and tool grounding, while others raised concerns on correctness guarantees and evaluation fairness. The authors responded with additional ablations, human–LLM agreement studies, cost breakdowns, and clarifications about early stopping and human evaluation protocols. While not all concerns can be fully resolved, most reviewers found the clarifications satisfactory and increased their scores. I weigh the demonstrated utility and improved clarity over the remaining weaknesses, leading to a poster acceptance recommendation.
Decision: Accept (poster)