Title: ChemOrch: Empowering LLMs with Chemical Intelligence via Groundbreaking Synthetic Instructions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Chemistry

Large Language Model

Synthetic Data
Abstract: Empowering large language models (LLMs) with chemical intelligence remains a challenge due to the scarcity of high-quality, domain-specific instruction-response datasets and the misalignment of existing synthetic data generation pipelines with the inherently hierarchical and rule-governed structure of chemical information. To address this, we propose ChemOrch, a framework that synthesizes chemically grounded instruction–response pairs through a two-stage process: task-controlled instruction generation and tool-aware response construction. ChemOrch enables controllable diversity and levels of difficulty for the generated tasks and ensures response precision through tool planning \& distillation, and tool-based self-repair mechanisms. The effectiveness of ChemOrch is evaluated based on: 1)  the \textbf{high quality} of generated instruction data, demonstrating superior diversity and strong alignment with chemical constraints; 2)  the \textbf{dynamic generation of evaluation tasks} that more effectively reveal LLM weaknesses in chemistry; and 3)  the significant \textbf{improvement of LLM chemistry capabilities} when the generated instruction data are used for fine-tuning. Our work thus represents a critical step toward scalable and verifiable chemical intelligence in LLMs. The code is available at \url{https://anonymous.4open.science/r/ChemOrch-854A}.


Peer reviews:
comments: SUMMARY:
The paper introduces ChemOrch, a two-stage framework that generates chemical instruction–response pairs to address the shortage of high-quality domain data for LLMs. Comprehensive experiments show the results have higher diversity. LLMs trained on the ChemOrch show improved capabilities.

STRENGTHS AND WEAKNESSES:
S1. The paper is clearly motivated by three concrete gaps, including data scarcity, mismatch between generic synthetic pipelines and chemical rules, and lack of controllable diversity.

S2. The framework yields inexpensive, verifiable data. 

S3. Multiple metrics are used to show the effectiveness of the proposed method. 

W1. My primary concern lies in the accuracy of the generated instruction–response pairs. The current framework does not appear to guarantee correctness in generation. If the synthesized data contains factual errors or inconsistencies, it raises doubts about using it to fine-tune the LLM model.

W2. The early stopping mechanism is not clear to me. It is unclear why terminating execution early is preferable when the final outputs may still be incorrect. 

W3. Certain parts of the evaluation rely heavily on human judgment.  This may introduce potential bias.

QUESTIONS:
Q1. What types of tools are specifically required for Chemistry Q&A tasks? Given that existing work has shown LLMs can already perform reasonably well on such questions, what additional capabilities do these tools provide, and in what ways do they further enhance the performance of LLMs?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission \"ChemOrch: Empowering LLMs with Chemical Intelligence via Groundbreaking Synthetic Instructions\" describes a framework for automatic construction of instruction-response pairs for large language models tailored to chemistry tasks. This is important as the number of available datasets in this domain is very limited. The framework provides data of comparable quality to existing approaches relying on human supervision but of significantly higher diversity. The utility of the framework of the framework was demonstrated to evaluate existing large language models and finetune them for further performance improvement. Overall, the study is well written and provides a significant advance in the field. I think it can benefit from some further improvements that should be addressed before publication. Please find my detailed comments below.\n\nSTRENGTHS AND WEAKNESSES:\nI think the quality is fair as the framework fills an important niche in the field but it should carefully address concerns regarding data leakage.\nI think the clarity is good as all the important components of the developed framework are illustrated well.\nI think the significance is fair as the framework is demonstrated to be directly applicable to finetuning existing models and improve their performance significantly. However, it is important to evaluate its utility for improving the general capabilities of a model on other datasets as well.\nI think originality is good as the framework is general and can be applied to many domains of science.\n\nQUESTIONS:\nMajor aspects:\nWhen evaluating the capability of ChemOrch to be used for finetuning, it seems that ChemOrch is both used for generating training data and for performance evaluation. This could result in data leakage. I think it is important that at least one example is provided where ChemOrch is used for finetuning but performance on other datasets such as the ones that already exist in literature (ChemLLMBench, Mol-Instructions) is shown.\n\nRecently, another benchmark named ChemBench was published (https://doi.org/10.1038/s41557-025-01815-x). I think at least some comparison of ChemOrch to ChemBench should be provided.\n\nThe model evaluation metrics should be explained explicitly in the main text as the text is otherwise very hard to follow. It should be mentioned at least what the best and worst performance metric values are for each metric.\n\nPage 8: The authors state: \"Similarly, in general, chemistry Q&A, we observed a comparable increase, suggesting enhanced domain understanding and reasoning ability.\" I do not agree that a better performance directly suggests enhanced domain understanding and reasoning ability. I think it merely suggests a higher chance to provide results that are closer to the reference values.\n\nI think that the comparison to existing approaches is relatively minimal. One existing study that could possibly provide more tasks to compare ChemOrch to previously tested tasks is ChemCrow (https://doi.org/10.1038/s42256-024-00832-8).\n\nI think that more context should be provided regarding to human evaluation of response (section 4.3). I think it should be provided in the main text how many samples were evaluated in total and how the metrics was calculated. For instance, is an example with a single error as inaccurate as an example with many errors?\n\nBased on Figure 6, it seems that the LLaMA model seems to benefit significantly more from finetuning than Qwen. Do the authors have an idea of the origin of this and the possible implications?\n\nThe submission mentions \"Transision State Search\" as one of the reasoning tasks. I could not find an example for this task in the Appendix. I think it is important to provide at least one example for each of the task categories to make it more clear to the reader what type of input-output pairs are considered for each of these.\n\nMinor aspects:\nTable 2: The evaluation results for MC are supposed to range between 1 and 5 (based on the table caption) but many results are even above 6. I think this is confusing and requires further clarification.\n\nFigure 3: It would be insightful to compare the distrubtions to the reference datasets ChemLLMBench and Mol-Instructions.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
ChemOrch introduces a two-stage framework that first generates chemistry-specific instructions under explicit task, difficulty and constraint control, then constructs tool-grounded responses through planning, tool distillation and multi-stage self-repair, ensuring executability and chemical validity

STRENGTHS AND WEAKNESSES:
Strengths:
1. The introduction of difficulty and constraint control helps the instruction generation process yield diverse, level-appropriate chemistry prompts.
2.  ChemOrch combines task-controlled instruction generation with tool-aware response construction, incorporating difficulty calibration, tool distillation, and self-repair. I think these designs can keep responses executable and verifiable.
3. The experimental results are promising. Test sets synthesized by ChemOrch track existing benchmarks closely. The paper also includes under-represented tasks such as the lipophilicity prediction task.
4. The paper is well-written and easy to follow.

Weaknesses:
1. The paper focuses largely on SMILES inputs and outputs. To what extent can the proposed framework be readily adapted to alternative string-based molecular encodings—such as graph-adjacency lists, tree-structured or JSON representations—and what specific effort or modifications would this entail?
2. All accuracy numbers are produced with the LLM-as-a-Judge rubric. Without human or rule-based ground truth, evaluation is vulnerable to the same hallucination, bias, and coherence errors it tries to measure.

QUESTIONS:
Please refer to weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nChemOrch introduces a synthetic instruction–response generation framework for chemistry tasks, aiming to improve large language models’ (LLMs) chemical reasoning and tool-usage capabilities. The method comprises a two-stage process:\n\n1. task-controlled instruction generation: This step is guided by constraints including tasks and metadata, and refined using a difficulty reward model. \n\n2. tool-aware response construction: It combines instruction decomposition, tool planning, tool distillation, self-repair mechanisms to generate outputs using chemistry-specific tools like RDKit and PubChem.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n1. The paper provides thorough prompt templates and code examples\n2. By supporting user-specified metadata the framework can adapt to additional tool configurations at runtime. And authors have provided detailed examples for the extensibility. \n3. The paper introduces a difficulty reward model trained via SFT to iteratively refine instructions, and its performance is supported by human evaluation protocol\n\n**Weakness**\n1. While the paper includes some ablation in Section 4.6, not all key components (e.g., the difficulty reward model M_diff) are analyzed. Given that ChemOrch consists of several modules, the paper may not fully reflect the author's insight on the proposed framework.\n2. While standard deviation are reported for most experiments, it would be beneficial to include error bars in figure 6 where the performance difference between Qwen Vanilla and Qwen Fine-tuned is not significant. \n3. While the experiments show that generated data can significant improvement of LLM chemistry capabilities, it's unclear how groundbreaking this framework actually is.\n\nQUESTIONS:\n1. What is the computational cost associated with the proposed framework (i.e. M_diff finetuning, instruction–response pairs generation, etc)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}