ReactionSeek: LLM-powered literature data mining and knowledge discovery in organic synthesis

Domain: chemistry
Published: 2026-03-02
Editorial outcome: major_revision

Abstract
The application of artificial intelligence (AI) to chemical discovery is critically hindered by the
inaccessibility of data locked within unstructured scientific literature. Existing data acquisition methods are often
manual, limited in scope, or require extensive custom software development, impeding progress in leveraging
AI for chemical discovery. Here, we introduce ReactionSeek, a framework that synergistically combines large
language models (LLMs) with established cheminformatics tools to automate multi-modal data mining from
organic synthesis literature. Through sophisticated prompt engineering with minimal custom code, ReactionSeek
extracts and standardizes complex textual, graphical, and semantic chemical information. We validate this
framework on the century-spanning Organic Syntheses collection, achieving over 95% precision and recall for
key reaction parameters. This enables three applications: the generation of a large, AI-ready dataset; an
interactive Synthetic Chatbot (SynChat) for natural language querying of chemical data; and an autonomous
analysis that revealed decades-long trends in catalysis. ReactionSeek thus provides a general solution to the data
curation bottleneck, representing a step forward in for AI-driven archive mining and knowledge discovery across
the chemical sciences.

Introduction
The integration of artificial intelligence (AI) into the molecular sciences is poised to fundamentally reshape
discovery, accelerating innovation from medicine to materials.1 This potential is particularly crucial in organic
synthesis, the foundational discipline of molecular construction.2 While AI-driven strategies have demonstrated
notable capabilities in retrosynthetic planning, mechanistic studies, and the discovery of novel reactions,3 their
application to the frontiers of chemical research is hampered by data accessibility. The vast majority of our
chemical knowledge, painstakingly accumulated over a century, remains locked within unstructured literature.
This forces a reliance on curated, often proprietary datasets, meaning that current AI models are trained on
sanitized information that fails to capture the complexity and diversity of real-world chemistry.4 Therefore,
creating automated and scalable methods to convert this immense scientific legacy into actionable, machine-
readable knowledge is the rate-limiting step toward unlocking genuine, AI-driven discovery in chemistry.5

Current approaches to data acquisition each present significant limitations. Commercial databases, while
extensive, are often constrained by data inconsistencies, incomplete coverage of specialized reaction types, and
restrictive licensing terms. High-throughput experimentation (HTE), on the other hand, generates precise
datasets that typically lack chemical diversity and are prohibitively expensive for many research groups.6,7 As a
result, manual curation from the scientific literature remains a common practice for building domain-specific
datasets. This approach is not only labor-intensive and time-consuming but also requires specific chemical
expertise and is prone to human error, potentially compromising dataset quality and scalability. Therefore, a
pressing need exists for an automated, reliable, and scalable methodology to address this data curation
bottleneck.8

Early efforts to automate chemical data extraction have explored various natural language processing (NLP)
techniques.9 Rule-based and traditional machine learning models struggled with the nuanced language of
chemical texts, particularly the complexity of molecular nomenclature, reaction schemes, and the implicit
knowledge embedded in experimental descriptions.10 Recently, large language models (LLMs) have
demonstrated significant promise due to their advanced text comprehension abilities.11-18 However, compared to
data mining efforts in related domains such as materials and biological sciences, mining organic synthesis
presents substantial challenges(Fig. 1a). This task requires interpreting not only diverse molecular structures and
identifiers but also the complex, context-dependent relationships among them. This goes beyond generic text
comprehension; it necessitates a framework that can effectively leverage the LLM's latent knowledge of
chemistry and reasoning abilities. While recent studies have utilized LLMs through methods such as fine-
tuning,11,13,15,19,20 prompt engineering,12,16-18,21,22 and other approaches,23-28 these pioneering methods often face
limitations. Key issues include a lack of adaptability across different literature styles, overly complex
implementations, and the considerable computational cost of model fine-tuning. Additionally, previous LLM-
based reaction mining approaches still face several challenges, including: the dispersion of compound
information across sub-headings, the conversion of chemical names to SMILES notation, complex compound
references, and the identification of compounds in reference images.29 Consequently, there remains a persistent
need for a more universal and resource-efficient methodology tailored to the intricacies of organic synthesis data
mining.

To tackle these challenges, we introduce ReactionSeek. This framework automates the multi-modal
extraction of chemical data from scientific literature (Fig. 1b). It employs a hybrid architecture that combines
the contextual understanding of LLMs with the chemical precision of established cheminformatics tools.
ReactionSeek utilizes a domain-specific prompt engineering strategy, enabling robust and accurate data mining
without the need for resource-intensive model fine-tuning. We demonstrate the effectiveness and scalability of
this approach by systematically mining the entire century-spanning Organic Syntheses collection30, achieving
over 95% precision and recall for key reaction parameters. The resulting AI-ready dataset enabled two
downstream applications: the development of SynChat, an interactive chatbot for querying reaction data via
natural language, and an autonomous, LLM-driven analysis that successfully identified decades-long historical
trends in catalysis. Thus, ReactionSeek provides a solid and transferable solution to the data curation bottleneck,
paving the way for large-scale, AI-driven knowledge discovery across the chemical sciences.

Results
The ReactionSeek Data Mining Framework

ReactionSeek is designed to process chemical literature to extract and organize reaction information,
transforming raw textual and graphical data into a curated, machine-readable dataset (Fig. 1b). The workflow
comprises three core stages: (1) Image mining for interpreting reaction schemes and structures within figures;
(2) Text mining for extracting detailed experimental information from procedural text using prompt-engineered
LLMs; and (3) Data standardization for ensuring the accuracy and consistency of the extracted chemical entities
and reaction parameters.

Organic Chemistry Image Mining: The accurate interpretation of chemical structures and reaction schemes
depicted in images is a foundational challenge in automated chemical data mining. While existing Optical
Character Structure Recognition (OCSR) tools can convert molecular structures into machine-readable formats
(e.g., SMILES),24,25,31-33 associating these structures with their correct roles (e.g., reactant, product) and textual
identifiers within complex figures remains a daunting challenge. ReactionSeek addresses this by employing a
multi-modal LLM (GLM-4V) to analyze reaction scheme images (Fig. 2a). The LLM is prompted to identify
compound labels within the image and map them to the corresponding molecular structure images, which are
subsequently converted to SMILES by OCSR tools (e.g., InDraw34).

To evaluate this image mining module, we tested it on 42 diverse reaction scheme images sourced from the
supplementary information (SI) of published literature. The multi-modal LLM (GLM-4V) achieved 91.5%
accuracy in correctly identifying and associating the molecular labels and their roles of reactants or products.
The InDraw OCSR tool was used to convert the molecular images to SMILES with 70.1% accuracy (see more
details in Supplementary Note 1.1).34 While the overall performance is promising, the accuracy of structure
recognition is influenced by the capabilities of the currently integrated OCSR tool, suggesting potential for
improvement with more accurate OCSR solutions.

Organic Chemistry Text Mining: Extracting detailed experimental procedures and reaction parameters from
unstructured text is central to ReactionSeek. To achieve this with high fidelity and minimal human intervention,
we developed a sophisticated prompt engineering pipeline for LLMs (Fig. 2b). This pipeline adapts general-
purpose LLMs to the specific nuances of organic chemistry literature, guiding them to extract predefined
chemical information and structure into a tabulated format suitable for review and downstream processing. With
this pipeline, the LLM can effectively identify compound entities within text segments and establish their
complex co-reference relationships. By processing sub-headings and text segments separately, our pipeline
successfully achieves the association between compound references in synthetic text segments and the
corresponding compounds mentioned in sub-headings.

Our prompt architecture, inspired by previous work on MOF data extraction,17 comprises four key
components designed to optimize LLM performance in this domain-specific task (see more details in
Supplementary Note 4.1): (1) Task description: This component clearly defines the overall objective of the
data mining task and provides fundamental guidelines to the LLM, including instructions to mitigate common
issues like output hallucination. (2) Task requirements: Here, we specify the information to be extracted and the
output format (e.g., a table with specific columns). This section is strategically placed to reinforce the LLM's
adherence to output specifications, countering potential memory decay with long inputs. (3) Processing
suggestions: This provides the LLM with expert-derived heuristics in natural language. These suggestions are
based on a pre-examination of sample texts by domain experts, who summarize common textual patterns and
incorporate essential chemical knowledge to aid the LLM in interpreting chemical narratives. (4) Processing
cases: Few-shot examples, consisting of real input texts and their corresponding desired structured outputs, are
provided. This allows the LLM to learn the specific input-output patterns and formatting requirements through
in-context learning. This structured prompting approach is designed to ensure the LLM maintains a balance of
task memory, semantic comprehension of the chemical context, and accurate execution of the extraction task.

To validate our data mining pipeline, we benchmarked its performance on the authoritative ChEMU35-38).
This selection ensures a rigorous comparative analysis, as state-of-the-art models from the ChEMU-2022
competition have explicitly established a high-performance baseline (F1 = 0.968) on this specific dataset. Our
pipeline demonstrated a remarkable advantage in the few-shot learning scenario. With only seven annotated
examples, it achieved an F1 score of 0.983 in extracting critical reaction components, including reactants,
products, solvents, temperature, time, and yield, representing a further improvement over the previous model
(see more details in Supplementary Note 3.3 Table S1).

The text mining efficacy of GLM-4 was also evaluated using a benchmark dataset of 50 reactions obtained
from the Supporting Information (SI) of six diverse articles on organic synthesis (see more details in SI
References). The extraction results were classified as true positives (TPS; correctly identified entities or values),
false negatives (FN; omissions), and false positives (FP; incorrect or fabricated information). In synthetic
chemistry applications, FPs are typically considered more detrimental than FNs. Our prompting strategy
effectively guided the model to extract a variety of experimental data, including compound identifiers, quantities,
yields, solvents, reaction conditions, and enantiomeric excess (e.e.) values, achieving high precision, recall, and
F1 scores for nearly all categories(see more details in Supplementary Note 4.1 Table 2) The only discrepancy
was the misidentification of an uncommon solvent mixture or additive (e.g., 18-crown-6), indicating that the
model's performance critically relies on its pre-existing chemical knowledge. This limitation can be easily
addressed by enhancing the prompt with information about such uncommon entities.

The text mining pipeline was subsequently expanded to extract complex spectroscopic data (NMR, MS,
HPLC) by leveraging tailored prompts with GPT-4. We targeted characterization data for 236 molecules from
the SI of the same benchmark literature used previously and developed prompts for data mining, including 1H
NMR, MS, HPLC, and IR (see more details in SI 3.1-3.2). The system demonstrated high efficacy, with
extraction accuracies exceeding 95% for most spectroscopic types (Fig. 3a). Notably, despite the inherent
complexity of 1H NMR data, which requires parsing chemical shifts, splitting patterns, and coupling constants,
the accuracy of its extraction remained remarkably high. This success suggests that the structured prompts
enabled the LLM to effectively extract the in-context chemical knowledge inherent in spectroscopic reports (Fig.
3c). Error analysis highlighted challenges in classifying coupling constants (J-values) and resolving ambiguities
due to inconsistent compound labels. Subsequent analysis indicated that more advanced LLM architectures
exhibit greater robustness against such inconsistencies.

Standardization of Extracted Chemical Information: The conversion of diverse compound nomenclature
(including IUPAC names, common names, misspelled variants, and chemical formulas) into machine-readable
SMILES representations remains a significant challenge in organic reaction mining29. A critical component of
ReactionSeek is its data standardization module, which is designed to convert raw extracted text into an accurate,
consistent, and AI-ready format (Fig. 2c). The module emphasizes two primary curation tasks: name-to-SMILES
transformation and unit standardization.

General-purpose LLMs are often unreliable for direct name-to-SMILES transformation, a limitation rooted
in their generative instability and lack of domain-specific knowledge training. As reported in previous work39 ,
this deficiency results in a high frequency of critical errors, including unparsable strings, the addition or omission
of atoms, incorrect functional group placement, and erroneous backbone structures. Given these significant
reliability issues, the direct application of general-purpose LLMs for this conversion task is challenging.
ReactionSeek overcomes this by implementing a hybrid workflow: (1) First, unique compound names are
compiled from the mining results. (2) Second, these names are processed using established cheminformatics
resources, including database lookups (e.g., PubChem40) and name-to-structure converters (e.g., OPSIN41), to
generate canonical SMILES. (3) Finally, the LLM, enhanced with vector and fuzzy search capabilities,
intelligently maps the mined nomenclature, which is often non-standardized, to the canonical SMILES curated
by these specialized tools. This iterative process of data validation, parsing, and LLM-assisted alignment ensures
the generation of high-fidelity structural representations (Fig. 2c).

Simultaneously, the module leverages the LLM in unit standardization (Fig. 2c). Through carefully
designed prompts, it standardizes units for all reaction parameters (e.g., time, temperature) to ensure dataset-
wide consistency. The final output is a structured CSV file that summarizes each reaction using standardized
identifiers and parameters. This curated dataset is not only directly suitable for building predictive models but
also serves as the knowledge base for interactive tools such as the SynChat application.

Large-Scale Reaction Mining: A Case Study of the Organic Syntheses Collection30
To demonstrate the scalability and practical utility of ReactionSeek, we applied it to the systematic mining
of the Organic Syntheses collection30, a historically significant and high-quality repository of verified synthetic
procedures published since 1921. This collection, comprising 102 volumes as of 2025, represents a valuable
resource for studying the evolution of synthetic methodology. For this study, we processed 3,103 articles from
the first 100 volumes, establishing a comprehensive raw dataset for subsequent analysis and chatbot development
(see more details in Supplementary Figure 4). A benchmark subset of 50 articles from this collection was also
curated manually for detailed performance evaluation.

Efficiency of Automated Mining: A key advantage of ReactionSeek is its potential to significantly reduce
the time and effort associated with manual data extraction. On the 50-article benchmark set, using GLM-442, the
framework required an average of 21.32 seconds per article for data mining and output formatting. This
represents a substantial improvement over manual data mining, which typically requires 5-10 minutes or longer
per article, underscoring the efficiency gains enabled by our LLM-based approach.

Accuracy Assessment on Organic Syntheses30 Benchmark: We evaluated the accuracy of ReactionSeek
(with GLM-4) on the 50-article benchmark dataset, employing evaluation criteria tailored to the practical needs
of subsequent applications. (1) Reactants and Products: Correct and complete extraction and classification of all
compound entities involved as reactants or products were considered critical. (2) Reaction Yield: Precise
extraction of yield values. (3) Reaction Scale (Amounts): The focus was on accurately extracting amounts for
key reactants and products, with relaxed stringency for ancillary reagents, such as solvents or excess reagents,
where order-of-magnitude and proportional relationships are often sufficient. (4) Time and Temperature: Given
the potential variability in experimental records for these parameters, the model was assessed on its ability to
extract the critical time and temperature values it deemed necessary, rather than achieving absolute accuracy for
every recorded value.

As shown in Fig. 4a, GLM-4 demonstrated strong performance in compound identification, particularly for
products, with very few FPs in reactant extraction, meeting a key accuracy demand. However, some omissions
(FNs) occurred during reactant extraction, likely due to inconsistent textual positioning of reactants. The
occurrence of FPs in solvent identification suggested the LLM might rely on its a priori knowledge of common
solvents, potentially misclassifying uncommon solvents or reactants.

In numerical data extraction, the model showed high accuracy for reaction yields and product quantities,
with no instances of hallucinated yield values on the test set. A notable error pattern emerged in reactant quantity
extraction when a compound was added in multiple portions; the model often struggled to accurately aggregate
these amounts. This may reflect limitations in the LLM's complex numerical reasoning within linguistic contexts
or the absence of specific prompt engineering for such scenarios. Time and temperature extraction, although
facing similar complexities, benefited from being listed as separate extraction targets, resulting in comparatively
better performance.

Out of the 50 benchmark articles, 48 yielded outputs conforming to the required format after reprocessing
of any initially unparseable results. The two articles that failed to produce correctly formatted outputs were
significantly longer (9824 and 5502 characters, which may almost exceed the context window of the LLM) and
contained more reactions (5 reactions each) than average. In these cases, GLM-4 successfully extracted the
information but failed in formatting, suggesting that very long input texts following an already lengthy prompt
can lead to the model partially forgetting initial formatting instructions.43 Text segmentation is a potential
strategy to mitigate this issue.

Comparative Evaluation of LLMs for Text Mining : The performance of LLM-based data mining is
significantly influenced by the choice of the underlying model, particularly its parameter size and architecture.
To guide model selection for ReactionSeek, we evaluated a diverse range of LLMs on the Organic Syntheses30
benchmark dataset. The models tested included GLM-4, GPT-3.544, GPT-445, Llama46, Mixtral47, Baichuan48,
Qwen49, and DeepSeek50.

Larger models generally exhibited significantly higher performance in extracting chemical information (Fig.
4a, 4d). This suggests that increased parameter size critically impacts an LLM's ability to understand complex
chemical contents. Larger models were better at understanding task objectives and correctly identifying relevant
chemical entities. Furthermore, the incidence of unparseable or incorrectly formatted responses decreased with
increasing model size (Fig. 4b). This improvement is likely due to the enhanced capacity of larger models to
handle long token inputs while retaining memory of key prompt instructions, and their superior ability to
comprehend complex chemical relationships and knowledge inherent in the context.

Interestingly, while larger models excelled in precision, medium-sized models often achieved comparable
recall rates (Fig. 4c). This indicates that once a model possesses a sufficient parameter scale to grasp the
fundamental chemical tasks outlined in the prompts, further size increases may offer diminishing profit for
simple identification tasks. Persistent error types, such as the aggregation of reactant quantities from multiple
additions, were not fully resolved solely by increasing model parameter size, highlighting the need for targeted
prompt refinements or algorithmic solutions for such specific challenges (Fig. 4b-d).

Analysis of successful extractions by small and medium-sized models revealed they typically performed
well in articles describing simpler reactions (e.g., fewer compounds, straightforward single-step procedures) and
those with shorter text lengths. This suggests that text segment length is a key factor influencing an LLM's ability
to generate valid, well-formatted outputs, as shorter texts reduce the cognitive and memory burden on the model.
For less complex extraction tasks or when using smaller models, strategies such as text segmentation or
employing models specifically adapted for long contexts can improve performance.

Based on the aforementioned evaluations, we ultimately employed GPT-3.5-16k to conduct data mining on
Organic Syntheses30 volumes 1–100, spanning from 1921 to 2021. After complete pipeline processing and
deduplication, we obtained 5,443 unique compounds, 3,961 unique reactions, of which 3,580 contained yield
information.

To demonstrate our pipeline's generalizability to newly published literature, we tested it on all newly
published articles (29 articles) from the 2025 volume of Organic Syntheses30 (Fig. 4e and Table 1). Our pipeline
correctly extracting most of reaction information from each article. During the standardization phase, with the
exception of molecular sieves that cannot be directly represented as SMILES, all other compounds were
successfully converted into their correct corresponding SMILES representations. These results confirm our
pipeline’s robustness, showing its ability to generalize effectively from benchmark datasets to newly published
chemical literature.

While modest in size compared to larger reaction databases like USSPTO, the value of our minded dataset
lies not in its breadth, but in its curated depth. It is composed of the most representative and classical organic
reactions across one century, with the exclusive inclusion of gram-scale procedures guaranteeing experimental
robustness and reproducibility. Consequently, we propose that this collection can serve as a standardized
reference for synthetic chemists and as a high-fidelity benchmark for the development and validation of machine
learning models.

LLM-Driven Discovery of Chemical Trends from Mined Data
To demonstrate the potential for data-driven discovery, we utilized ReactionSeek to create a large-scale,
structured dataset from the Organic Syntheses30 collection. This AI-ready dataset then served as the input for
autonomous analysis via Gemini 2.5 Pro.51 By designing specific prompts, we tasked the LLM with uncovering
historical trends within this century-spanning compendium, including the emergence and diversification of
asymmetric catalysis and the changing landscape of metal utilization in synthesis (Fig. 5).

In its analysis of asymmetric catalysis (Fig. 5b), the LLM identified a significant increase in such reactions
after 1980. It highlighted the pivotal role of transition-metal catalysts, alongside the emergence of organic small-
molecule catalysts and enzymatic/biocatalytic systems. The LLM noted the prevalent use of specific metals like
Boron, Copper, Titanium, Ruthenium, and Palladium, correlating their use with advancements in well-known
methodologies such as CBS reductions, metal-coupling protocols, asymmetric epoxidation (e.g., with Titanium
Tartrate or Ti-BINOL systems), asymmetric hydrogenation, cross-coupling reactions, and allylic alkylation.
Furthermore, the LLM identified substrate preferences, observing that aldehydes and ketones, as well as halides,
were the most dominant, with esters and alkenes also featuring prominently (see Supplementary Note 2.4 for
more details).

Regarding metal usage trends (Fig. 5b, 5d), the LLM's analysis revealed an interesting prevalence of tin in
specific periods, correctly attributing this to the widespread adoption of tributyltin hydride (Bu3SnH) as a mild
radical initiator, reducing agent, and key component in Stille couplings. More broadly, the LLM categorized the
evolution of metal usage in Organic Syntheses30 into three distinct epochs: (1) The Main-Group Metal Era (1921–
1960): Dominated by Grignard reagents and salt-forming reactions, with less emphasis on metal toxicity. (2) The
Incipient Transition (1961–1990): Characterized by the rise of lithium/copper reagents, the early development
of palladium catalysis, and a decline in mercury utilization. (3) The Golden Age of Transition-Metal Catalysis
(1991–2023): Marked by concentrated growth in Palladium-, Ruthenium-, and Rhodium-mediated
transformations.

These LLM-identified trends accurately reflect established paradigm shifts in organic synthesis from
stoichiometric to catalytic approaches, from harsh to milder conditions, and from highly toxic reagents towards
greener methodologies. This demonstrates that an LLM, when provided with a structured dataset curated by
ReactionSeek, can autonomously develop analytical insights and perform trend analyses consistent with expert
chemical knowledge, highlighting the potential for AI-assisted discovery from systematically mined literature.

The Synthesis Chatbot: Interactive Access to Mined Knowledge
To enhance the accessibility and utility of the extensive dataset curated from Organic Syntheses30, we
developed SynChat, an interactive tool powered by LLM agents. SynChat allows researchers to query the
historical reaction data and associated metadata using natural language, providing a more intuitive and efficient
means of accessing specific data compared to traditional search methodology.

The development of the chatbot involved structuring the mined data for effective retrieval by the LLM. Key
information from each reaction entry (e.g., title, DOI, reactants, products, SMILES, yield) was compiled and
vectorized to a customized vector database52.This allows the LLM, when presented with a user query, to perform
similarity searches across the Vector Database and retrieve the most relevant supporting texts from the original
Organic Syntheses30 (Fig. 6). The LLM agent then synthesizes an answer based on this retrieved information,
previous record and reasoning, with prompts designed to ensure strict adherence to the source material, thereby
minimizing the likelihood of generating hallucinated or inaccurate responses.

SynChat offers several advantages for chemical researchers: (1) Natural Language Queries with Molecular
Structures: It handles complex queries posed in natural language that incorporate molecular structures or reaction
information, increasing the flexibility of data access for users without specialized database query language skills.
(2) Enhanced Reliability and Traceability: By citing specific data sources from the Organic Syntheses30
collection, the chatbot allows users to consult the original literature for verification and further context easily.
(3) Iterative Information Retrieval: The interactive nature of the LLM enables users to refine their queries or ask
follow-up questions in multi-turn dialogues, facilitating a more thorough exploration of the available information.
This interactive tool exemplifies how LLMs can transform structured and unstructured data into dynamic
knowledge resources, bringing AI-powered assistance directly into the research and learning workflows of
organic chemists.

Methods
Prompt Design for Image Mining
Image mining was conducted using the GLM-4V multi-modal large language model. To improve the logical
accuracy of the model in determining compound roles, we designed specific instructions requiring the LLM to
explicitly analyze the visual orientation of reaction arrows (e.g., distinguishing between reactants and products
based on arrow direction). Adopting the structured prompt framework utilized in our text mining workflow, the
image mining prompts were specifically tailored to extract the numerical labels associated with reactants,
products, and reagents from the reaction schemes. Following role identification, chemical structure recognition
was performed using InDraw,34 which automatically segmented and identified each compound structure before
converting them into canonical SMILES strings for storage. Full details regarding the prompt engineering
strategies and templates are provided in Supplementary Note 1.1.

Prompt Design for Text Mining
To extract detailed experimental procedures and reaction parameters from unstructured organic chemistry
literature with high fidelity, we developed a structured prompt engineering pipeline tailored for Large Language
Models (LLMs). This pipeline was designed to guide the LLM in identifying compound entities, resolving co-
reference relationships, and structuring extracted data into a standardized tabular format. The prompt architecture,
inspired by methodologies for metal-organic framework (MOF) data extraction17, consists of four strategic
components optimized for chemical text analysis:

1. Task Description: This module defines the global objective of the data mining task. It includes explicit
constraints and negative constraints to mitigate common LLM issues, such as hallucination, ensuring
the model focuses strictly on the provided text.
2. Task Requirements: To ensure output consistency, this module specifies the exact information fields to
be extracted and enforces a strict tabular output format. This instruction block is strategically positioned
within the prompt to reinforce the LLM’s adherence to formatting specifications, thereby counteracting
potential context forgetting (memory decay) associated with long input sequences.
3. Processing Suggestions: We incorporated expert-derived heuristics in natural language to aid the LLM.
These suggestions were developed through a pre-examination of representative literature by domain
experts, summarizing common textual patterns and embedding essential chemical knowledge to
facilitate the correct interpretation of complex chemical narratives.
4. Processing Cases (Few-Shot Learning): To facilitate in-context learning, we included a set of few-shot
examples consisting of raw input texts paired with their corresponding, manually verified structured
outputs. This module allows the LLM to learn specific input-output mappings and formatting
requirements by example.

To resolve complex co-reference relationships, the pipeline processes sub-headings and main text segments as
distinct but related inputs, enabling the model to associate compound references within the synthetic procedures
with their corresponding specific compounds defined in the sub-headings. Full details of the prompt templates
and specific examples are provided in Supplementary Note 1.2.

Data Standardization Design
The standardization process was carried out in three consecutive steps. In step (1), the chemical compound
dataset was created using compound names and their respective CAS numbers extracted from the Organic
Syntheses30 collection. Next, in step (2), compound identifiers were primarily obtained through CAS numbers,
followed by additional identification using PubChem40 nomenclature and OPSIN41 systematic name parsing for
unprocessed entries. In step (3), molecular name vectorization was conducted to produce a comprehensive
dataset, which allowed for precise matching of mined molecular names. For unit standardization, temporal values
were consistently converted into minutes, with particular attention given to unconventional temporal expressions,
such as "overnight". Similarly, material quantities were standardized by converting all measurements into
uniform units of grams and moles. Full details regarding the prompt engineering strategies and templates are
provided in Supplementary Note 1.1.

Design of SynChat
The reaction database is architected into two distinct components to optimize retrieval efficiency and
information density: a vectorized query database for semantic search and a supplementary metadata repository
for detailed experimental context.

(1) Vectorization and Indexing: Key retrieval fields—including product SMILES, ReactionSMILES,
chemical nomenclature, reaction types, and article titles—were vectorized using the all-MiniLM-L6-v2 model,
a Sentence-BERT (SBERT) framework. This model transforms input text into 384-dimensional dense vector
embeddings without dimensionality reduction. These embeddings were indexed and managed using
ChromaDB53, an open-source vector database, to facilitate high-speed semantic similarity search. The specific
distribution of these embedding vectors is visualized in Supplementary Note 5.

(2) Data Retrieval and Generation: Detailed experimental parameters, including full synthetic procedures,
yields, reagent quantities, reaction conditions, and citation metadata, are stored in the supplementary database in
a JSON file. Upon receiving a user query, the system executes a retrieval process that identifies relevant entries
based on vector similarity (semantic match) and exact character matching within the ChromaDB index. The
system then retrieves the corresponding comprehensive procedural details from the supplementary database.
Finally, the retrieved context—combining the concise query features and the full textual descriptions—is
concatenated and fed into the Large Language Model (LLM). This Retrieval-Augmented Generation (RAG)
approach enables the LLM to generate tailored, evidence-based responses to user inquiries regarding specific
reaction conditions and procedures.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This paper introduces a practically useful system for extracting information from organic synthesis literature. Most components of the system are implemented through prompt engineering. In addition, due to the standardized nature of LLM software APIs, users can operate the system without major difficulties. From an academic perspective, however, it is essential to compare this work with prior research.\nImportant piece of related work that is not referenced in the paper is the ChEMU project, which focuses on extracting more detailed information from chemical patent texts.\n[1] Li, Y. et al. (2022). The ChEMU 2022 Evaluation Campaign: Information Extraction in Chemical Patents. In: Hagen, M., et al. Advances in Information Retrieval. ECIR 2022. Lecture Notes in Computer Science, vol 13186. Springer, Cham.\nhttps://doi.org/10.1007/978-3-030-99739-7_50\n[2] Extended Overview of ChEMU 2022 Evaluation Campaign: Information Extraction in Chemical Patents758-781\nYuan Li, Biaoyan Fang, Jiayuan He, Hiyori Yoshikawa, Saber A. Akhondi, Christian Druckenbrodt, Camilo Thorne, Zubair Afzal, Zenan Zhai, Kojiro Machi, Masaharu Yoshioka, Youngrok Jang, Hosung Song, Junho Lee, Gyeonghun Kim, Yireun Kim, Stanley Jungkyu Choi, Honglak Lee, Kyunghoon Bae, Darshini Mahendran, Christina Tang, Bridget McInnes, Timothy Baldwin, Karin Verspoor\nhttps://ceur-ws.org/Vol-3180/paper-62.pdf\n[3] Context aware Named Entity Recognition and Relation Extraction with Domain-specific language model782-796\nYoungrok Jang, Hosung Song, Junho Lee, Gyeonghun Kim, Yireun Kim, Stanley Jungkyu Choi, Honglak Lee, Kyunghoon Bae\nhttps://ceur-ws.org/Vol-3180/paper-63.pdf\nTask 1: the Named Entity Recognition (NER) task [1] involves the extraction of yields and products. Since the target text and the extracted data differ in nature, direct comparison using simple values is not appropriate. However, the best-performing system [3] achieves high precision and recall (greater than 0.96) as shown in [2], without relying on recent generative AI techniques.\nMoreover, these systems also aim to extract more detailed information about the reaction process. It is important to reference these papers and discuss how the proposed system differs in terms of its characteristics. In addition, the paper should justify why only limited types of information were selected for extraction from the literature.\nOverall, this is a good paper presenting a practically useful system. However, due to the lack of sufficient comparison with prior work, it cannot be accepted in its current form.", "reviewer_2": "This paper introduces ReactionSeek, a lightweight framework that integrates LLM prompting with standard cheminformatics utilities to automatically extract multi-modal information (text, figures, semantics) from organic synthesis literature. Applied to the full Organic Syntheses archive, the authors report >95% precision and recall for extracting “key reaction parameters.”\nThe framework is demonstrated through three applications: (i) construction of an AI-ready dataset, (ii) development of an interactive natural-language agent (SynChat) for chemical data querying, and (iii) autonomous analysis of long-term catalysis trends. The authors argue that ReactionSeek reduces reliance on custom code, generalizes across modalities, and mitigates the data curation bottleneck that hinders AI-driven chemical discovery.\nI suggest the following clarifications and additions to strengthen the manuscript:\n1. Will SynChat (and its prompts/config) be publicly available (demo URL, code)?\n2. Please add some examples (based on some recently published literature) with expected answers and links to prove the effective of ReactionSeek.\n3. In Figures 5 and 6, please clarify precisely what each database contains (e.g., number of unique reactions, unique molecules, time coverage, journal coverage, document count, deduplication criteria).\n4. For the vectorization database (Figure 6), provide a complete schema: how the database is constructed, what modalities are vectorized, and which representations/embeddings are used (e.g., Morgan fingerprints, text embeddings, reaction fingerprints).\n5. How is similarity search performed? Please define the embedding spaces used for each modality (text, molecule, reaction, condition) and the similarity metrics applied (e.g., cosine similarity, Tanimoto coefficient).\n6. How similarity search is calculated, defines the embedding spaces used for each modality;\n7. More references need to be added. Such as “A Multi-Agent System Enables Versatile Information Extraction from the Chemical Literature”\n(Remarks on code availability)\nThe current code repository does not include a README file that documents the scripts and provides usage instructions, which makes it difficult to reproduce or run the code. In addition, the online link provided does not function as expected—I attempted to upload a file to the website, but the process failed. Could you please address these issues by adding clear documentation and fixing the online interface?\nI also noticed that the code repository contains error logs. Does this indicate that the pipeline relies on the OpenAI API for extracting chemical literature information? If so, could you clarify the success rate of these API calls, as well as how errors are handled? It would be helpful to report the proportion of successful versus failed extractions, and whether any fallback strategies are in place.", "reviewer_3": "Li et al. developed a framework that synergistically combines large language models (LLMs) with established cheminformatics tools to automate multi-modal data mining from organic synthesis literature. This is the need of the hour. I strongly feel that LLM can automate the process and make life easier for the synthesis community. The authors have certainly shown ways to make a technological advancement but unfortunately missed comparing how this technological advancement brings advantages over existing methods. I can recommend this article only after that analysis. Please find my major concerns below.\nIn the Introduction, authors have mentioned that “This enabled three applications: the generation of a large, AI-ready dataset;” etc. However, I don’t see any description of this dataset. Will it be possible for authors to describe that dataset.\nThen, authors need to show which type of cases their approach can’t extract reactions and whether they have taken care to improve those. For example, in the work https://link.springer.com/article/10.1186/s13321-024-00928-8, fig 5, the authors identified number of challenges for text to reaction extraction. Whether the proposed approach can overcome them? There can be some comparison with some benchmarking dataset.\nSimilarly, for image to text conversion, RxnScribe https://pubs.acs.org/doi/10.1021/acs.jcim.3c00439 described number of challenges. The conclusion include “Despite the success in our experiments, there are a few limitations in this work. First, we focus on parsing the reaction structure, but do not evaluate the final extracted reaction SMILES strings due to the lack of such ground truth in our dataset. The molecular structure recognition model MolScribe and the OCR tool might introduce additional errors. Second, we limit our study to diagrams presented in digital format, excluding those that are either scanned or hand-drawn. Furthermore, the extracted information from diagrams are sometimes incomplete. For example, the reaction conditions are sometimes listed in a table and the molecules may involve R-groups which are defined elsewhere.” Whether proposed approach can make improvements over these limitations? In addition, images with reactions shown in cyclic format may face additional challenges and model may find it difficult to extract those images. How does your approach solve those problems?\nIn Fig 1(b) authors showed conceptual overview of the ReactionSeek framework. What will model do if the molecule of interest that needs to be synthesized is new? That is the real-life question for most cases. Existing tools like SciFinder from CAS can also perform the current job and probably have wider access to articles. Probably, additional agents like IBMRxn or open source AiZynthFinder (https://jcheminf.biomedcentral.com/articles/10.1186/s13321-024-00860-x) can be used in such scenario. Moreover, if possible, additional agents can be called to identify the source, from which the starting ingredients can be bought. That can help to optimize the cost of the synthesis process."}, "author_response": "Responses and Revisions\nReviewer #1 (Remarks to the Author):\nThis paper introduces a practically useful system for extracting information from organic\nsynthesis literature. Most components of the system are implemented through prompt\nengineering. In addition, due to the standardized nature of LLM software APIs, users can\noperate the system without major difficulties. From an academic perspective, however, it is\nessential to compare this work with prior research.\nOur overall responses: We appreciate the reviewer's encouraging and insightful comments,\nwhich have significantly helped us improve the quality of our paper. We have carefully\naddressed all the points raised and have revised the manuscript accordingly. A detailed point-\nby-point response is provided below.\nQ1. Important piece of related work that is not referenced in the paper is the ChEMU project, which focuses on extracting more detailed information from chemical patent texts.\n[1] Li, Y. et al. (2022). The ChEMU 2022 Evaluation Campaign: Information Extraction in\nChemical Patents. In: Hagen, M., et al. Advances in Information Retrieval. ECIR 2022. Lecture\nNotes in Computer Science, vol 13186. Springer, Cham. https://doi.org/10.1007/978-3-030-\n99739-7_50\n[2] Extended Overview of ChEMU 2022 ---- Bridget McInnes, Timothy Baldwin, Karin\nVerspoor https://ceur-ws.org/Vol-3180/paper-62.pdf\n[3] Context aware Named Entity Recognition and Relation Extraction with Domain specific\nlanguage model782-796 Youngrok Jang, Hosung Song, Junho Lee, Gyeonghun Kim, Yireun\nKim, Stanley Jungkyu Choi, Honglak Lee, Kyunghoon Bae https://ceur-ws.org/Vol-3180/paper-\n63.pdf\nReply: Many thanks! Related references have now been incorporated in the newly added\nparagraph mentioned in our subsequent response (On page 5 of the main text).\nQ2. Task 1: the Named Entity Recognition (NER) task [1] involves the extraction of yields and\nproducts. Since the target text and the extracted data differ in nature, direct comparison using\nsimple values is not appropriate. However, the best-performing system [3] achieves high\nprecision and recall (greater than 0.96) as shown in [2], without relying on recent generative\nAI techniques.\nReply: We have added the main text with comparative tests conducted on the ChEMU 2020\ndataset—since the ChEMU 2022 competition was no longer accessible.\nPerformance: Our proposed method get an F1 score of 0.983, representing a significant\nimprovement over the prior benchmark of 0.968 (Jang, Y. J. et al. In Conference and Labs of\nthe Evaluation Forum (2022). ). We have updated the relevant content in the main text. The\nrevised text now reads as follows (On page 5 of the main text):\n\"... To validate our data mining pipeline, we benchmarked its performance on the\nauthoritative ChEMU, which achieved a top F1 score of 0.968 on key chemist-centric metrics,\nincluding reactants, products, reaction conditions, and yield information (Fig. 3b). Our\npipeline demonstrated a remarkable advantage in the few-shot learning scenario. With only\nseven annotated examples, it achieved an F1 score of 0.983 in extracting critical reaction\ncomponents, including reactants, products, solvents, temperature, time, and yield, representing\na further improvement over the previous model. ....\"\nFig. 1: Breakdown of extraction outcomes for ChEMU 2020 competition as examples.\nFig.2: Extraction score for each data type in ChEMU 2020 data and their specific F1 scores.\nQ3. Moreover, these systems also aim to extract more detailed information about the reaction\nprocess. It is important to--- discuss how the proposed system differs in terms of its\ncharacteristics. In addition, the paper should justify why only limited types of information were\nselected for extraction from the literature.\nReply: Thank you for your rigorous suggestion. Our work is distinguished from previous\nefforts in several key aspects:\nUltimate goal: Our focus is on the end-to-end construction of high-quality datasets from\nrepresentative organic synthesis literature, rather than solely developing a method for named\nentity recognition from experimental procedures. Accordingly, we excluded data fields\nextraneous to the chemical reaction itself—namely, EXAMPLE_LABEL () and\nOTHER_COMPOUND (compounds not directly involved in the reaction, but only during\nworkup)—from our comparative analysis. These were not fully optimized in our current\npipeline, though we believe there is potential for their performance to be improved in future\nwork.\nData efficiency: Our pipeline operates on a few-shot learning paradigm, requiring only\nabout 10 labeled examples. This contrasts sharply with related works, which typically require\ntraining on extensive annotated datasets (e.g., 900 examples in the related work above), thus\nour approach drastically reduces data annotation costs.\nFlexibility and customization: Our prompt-driven data mining approach accepts natural\nlanguage instructions, allowing researchers to define custom data structures and extraction\nstrategies. This flexibility is highly competitive for mining reactions in sub-fields where labeled\ndata is scarce.\nConsequently, our methodology proves particularly suitable for customized dataset\nconstruction, effectively assisting organic chemistry researchers in building high-quality,\ndomain-specific datasets with targeted precision.\nReviewer #2 (Remarks to the Author):\nThis paper introduces ReactionSeek, a lightweight framework that integrates LLM prompting\nwith standard cheminformatics utilities to automatically extract multi-modal information (text,\nfigures, semantics) from organic synthesis literature. Applied to the full Organic Syntheses\narchive, the authors report >95% precision and recall for extracting “key reaction parameters.”\nThe framework is demonstrated through three applications: (i) construction of an AI-ready\ndataset, (ii) development of an interactive natural-language agent (SynChat) for chemical data\nquerying, and (iii) autonomous analysis of long-term catalysis trends. The authors argue that\nReactionSeek reduces reliance on custom code, generalizes across modalities, and mitigates\nthe data curation bottleneck that hinders AI-driven chemical discovery.\nOur overall responses: We appreciate the reviewer's encouraging and insightful comments,\nexamination of our code repository and their constructive feedback based on hands-on\nexperience with its implementation, which have significantly helped us improve the quality of\nour paper and codebase. We have carefully addressed all the points raised and have revised the\nmanuscript accordingly. A detailed point-by-point response is provided below.\nQ1. Will SynChat (and its prompts/config) be publicly available (demo URL, code)?\nReply: We thank the reviewer for this important question regarding accessibility and\nreproducibility. We have made the SynChat web demo publicly available at:\nhttp://gpu1.luoszgroup.com:18501/, which serves as an interactive interface for exploring the\ndataset generated by our research pipeline. The core code of the data mining and analysis\npipeline is fully open-sourced and reproducible via our GitHub repository:\nhttps://github.com/DeepSynthesis/ReactionSeek.git. Once again, we thank the reviewer for\ntheir valuable time and constructive feedback.\nQ2. Please add some examples (based on some recently published literature) with expected\nanswers and links to prove the effective of ReactionSeek.\nReply: Many thanks for the suggestions.\nWe performed an additional test on 5 recent synthesis articles from Volume 102 of Organic\nSyntheses, published in 2025. We successfully performed flawless reaction mining on all five\narticles. Furthermore, it correctly converted 36 out of 38 compound names to their\ncorresponding SMILES strings. The exception were “4 Å Molecular sieves” and “4 Å MS”, a\nmaterial that cannot be directly represented by the SMILES notation. We have supplemented\nthe main text with detailed descriptions (On page 9 and 11 of the main text) and Fig. 4e with\nextraction tests and results demonstration.\nFig. 4e\nIn addition, following the suggestion from Reviewer 1, we have incorporated a direct\ncomparison with the ChEMU patent reaction mining competition. Our model (In Review 1’s\npart) achieved an F1 score of 0.983 comparing to previous best result of 0.968 in that\ncompetition. For details, please also see the answers for Q2, referee 1.\nQ3. In Figures 5 and 6, please clarify precisely what each database contains (e.g., number of\nunique reactions, unique molecules, time coverage, journal coverage, document count,\ndeduplication criteria).\nReply: Thanks for the suggestions. In the revised manuscript, we have now included the\nrelevant statistics for our mined dataset. The contents of Fig 5 and Fig 6 are both based on the\ndata mining results of Organic Syntheses by ReactionSeek as described earlier. We have added\nthe following relevant content in the main text (On page 10 and 11 of the main text):\n\"... Based on the aforementioned evaluations, we ultimately employed GPT-3.5-16k to\nconduct data mining on Organic Syntheses volumes 1–100, spanning from 1921 to 2021. After\ncomplete pipeline processing and deduplication, we obtained 5,443 unique compounds, 3,961\nunique reactions, of which 3,580 contained yield information. ....\"\n\"... While modest in size compared to larger reaction databases like USPTO, the value of\nour minded dataset lies not in its breadth, but in its curated depth. It is composed of the most\nrepresentative and classical organic reactions across one century, with the exclusive inclusion\nof gram-scale procedures guaranteeing experimental robustness and reproducibility.\nConsequently, we propose that this collection can serve as a standardized reference for\nsynthetic chemists and as a high-fidelity benchmark for the development and validation of\nmachine learning models. ....\"\nThank you once again for this suggestion, which has helped us to present our data mining\nresults with better clarity.\nQ4. For the vectorization database (Figure 6), provide a complete schema: how the database\nis constructed, what modalities are vectorized, and which representations/embeddings are used\n(e.g., Morgan fingerprints, text embeddings, reaction fingerprints).\nReply: Thank you for your constructive comments. We have revised Fig. 6 to include a\nschematic that illustrates the construction process of the vector database and the query method.\nThe updated figure also displays text examples retrieved from the query and Supplementary\nsets, respectively. We also have detailed the specific implementation of SynChat in the \"Design\nof SynChat\" section within the Methods chapter and Supplementary information S5, as shown\nbelow (On page 15 of the main text).\n\"... The string stored in the database consists of concise, preprocessed information,\nincluding the title, DOI, reactants, products, SMILES, yield, and other identifying features. The\ndatabase is divided into two components: a query database for retrieval purposes and a\nsupplementary database for supplementary information. The query database consists of textual\nfields including product SMILES, ReactionSMILES, product names, reaction names, titles, etc.,\nwhich are vectorized (based on the all-MiniLM-L6-v2 embedding model) through the Chroma\nvector database. ....\"\nWe have also modified the figures in the main text to simultaneously display both the\nmatched information and the structural data used for vectorization.\nFig. 3: A more detailed illustration of both the vector database construction process and the\noverall SynChat workflow.\nQ5. How is similarity search performed? Please define the embedding spaces used for each\nmodality (text, molecule, reaction, condition) and the similarity metrics applied (e.g., cosine\nsimilarity, Tanimoto coefficient).\nReply: We thank the reviewer for their insightful questions.\n1) For the similarity search and the similarity metrics applied, we first generate embeddings\nfor the text fields in the Query set using the default model provided by the Chroma database.\nWe then compute the cosine similarity between these embeddings and the query embedding\nto retrieve the most relevant chemical reactions. Subsequently, these retrieved reactions are\nused to fetch their corresponding synthesis descriptions from the Supplementary set,\nthereby enriching the information in the final response.\n2) For multimodal processing, we treat all relevant information—including SMILES,\nReaction SMILES, reactants, products, DOIs, and reaction names—uniformly as text fields.\nThis approach bypasses the need for complex multimodal alignment. In our ongoing work,\nwe are also incorporating a substructure similarity matching mechanism to enable more\nchemically meaningful database retrieval and to further enhance SynChat's retrieval\ncapabilities based on chemical semantics.\nWe have also revised Fig. 6 to more clearly illustrate the aforementioned process.\nQ6. How similarity search is calculated, defines the embedding spaces used for each modality;\nReply: We thank the reviewer for above suggestions. The similarity calculation is based on the\ncosine similarity of text embeddings, which are generated using the built-in model of the\nChroma database. The embedding space is unimodal, consisting solely of text. This is because\nour ReactionSeek data mining process ultimately consolidates all reaction information from\nboth images and text into corresponding text fields. Consequently, the vector-based retrieval\noperates exclusively on this single text modality.\nQ7. More references need to be added. Such as “A Multi-Agent System Enables Versatile Information Extraction from the Chemical Literature”\nReply: We thank the reviewer for their valuable suggestions. Related references have now been\nincorporated in the newly added comparative paragraph and cited as ref. 20，22，29，33，34，\n35，36 and 53.\nReviewer #2 (Remarks on code availability):\nQ8. The current code repository does not include a README file that documents the scripts\nand provides usage instructions, which makes it difficult to reproduce or run the code.\nReply: Thank you for pointing out this problem in the repository. We have revised the original\nREADME presentation by updating the key functional modules and user-configurable\ncomponents to the main page for better accessibility. Below is the GitHub repository link:\nhttps://github.com/DeepSynthesis/ReactionSeek.git\nQ9. In addition, the online link provided does not function as expected—I attempted to upload\na file to the website, but the process failed. Could you please address these issues by adding\nclear documentation and fixing the online interface?\nReply: We thank the reviewer for raising these important points. We have fixed the online\ninterface and completely remake our website page. We initially designed an \"Upload File\" for\nSynChat that would integrate with our ReactionSeek reaction mining functionality. The goal\nwas to allow users to upload a document, which the system would then automatically parse to\nconstruct a dataset. However, implementing this feature involves significant engineering\ncomplexity such as the integration of back-end tools, calls to large language models (LLMs),\nand interactive functionalities between the user and the model's output. Given these challenges,\nand also because this feature does not impact the core data-querying functionality of the current\nwork, we have prioritized other aspects of development and have not yet deployed it.\nHowever, SynChat platform is still under active development, and our ongoing work is\nprecisely focused on enabling this capability. We are making continuous updates and welcome\nyou to follow our progress.\nQ10. I also noticed that the code repository contains error logs. Does this indicate that the\npipeline relies on the OpenAI API for extracting chemical literature information? If so, could\nyou clarify the success rate of these API calls, as well as how errors are handled? It would be\nhelpful to report the proportion of successful versus failed extractions, and whether any fallback\nstrategies are in place.\nReply: We appreciate the reviewer's careful reading of our repository and their pertinent\ncomments. Access to APIs such as OpenAI can be restricted or not stable in certain scenarios,\noccasionally leading to request failures (Approximately 4.3% of requests failed in the first\nround of our extraction process). The likelihood of such failures is significantly reduced,\nhowever, with a stable network, etc. To handle these potential failures, we implemented an\nexponential backoff strategy to manage transient network fluctuations. If a request still failed,\nit was logged and the API call will be restarted. Using this two-tiered approach, we successfully\nprocessed all content.\nReviewer #3 (Remarks to the Author):\nQ1. Li et al. developed a framework that synergistically combines large language models\n(LLMs) with established cheminformatics tools to automate multi-modal data mining from\norganic synthesis literature. This is the need of the hour. I strongly feel that LLM can automate\nthe process and make life easier for the synthesis community. The authors have certainly shown\nways to make a technological advancement but unfortunately missed comparing how this\ntechnological advancement brings advantages over existing methods. I can recommend this\narticle only after that analysis. Please find my major concerns below.\nOur overall responses: We sincerely thank the Reviewer for their positive evaluation of our\nwork’s direction and for raising the critical point regarding a comparative analysis. We fully\nagree that demonstrating a clear advantage over existing methods is essential to validate our\ncontribution. Following the reviewer’s constructive advice, we have now incorporated a new\nsection in the manuscript that provides a head-to-head comparison and detailed discussion.\nThis new analysis quantitatively and qualitatively benchmarks our framework against\nestablished methods, such as the state-of-the-art model from the ChEMU competition (Jang, Y.\nJ. et al. -- In Conference and Labs of the Evaluation Forum (2022)). Our results confirm that\nthe proposed framework not only achieves superior performance but also offers fundamental\nadvantages in efficiency and adaptability. The key advantages, which are now detailed in the\nrevised manuscript, are threefold:\n1. On the ChEMU patent reaction mining benchmark, our framework achieves a state-of-the-\nart F1 score of 0.983, outperforming the competition's top score of 0.968. Critically, this\nwas achieved via few-shot learning, requiring annotation of fewer than 10 examples. In\nstark contrast, the benchmark models rely on extensive supervised training with large\ndatasets (e.g., ~900 labeled examples). This represents a dramatic reduction in data\nannotation costs and effort, making our pipeline exceptionally transferable and practical,\nespecially in specialized domains where labeled data is scarce.\n2. Our framework demonstrates superior versatility. By simply modifying the prompts and\nproviding a few relevant examples, the same underlying model can be adapted to diverse\ntext mining scenarios. As demonstrated in our work, this includes extracting information\nfrom patent procedures, journal protocols, and even spectroscopic data in supporting\ninformation. This adaptability stands in contrast to conventional models, which would\nrequire re-training entirely new systems for each distinct task.\n3. The combination of high data efficiency and versatility makes our approach particularly\npowerful for specialized research areas particularly for synthetic methodology studies (e.g.,\nmining reactions for a specific catalyst or reaction class, new reaction\ndevelopment/optimization, synthetic route design). In these domains, the lack of large\nannotated datasets often renders traditional data-hungry models ineffective. Our framework\nprovides a robust and scalable solution to this long-standing challenge.\nWe believe these additions and detailed explanations directly address the reviewer's concerns\nand now provide a clear, evidence-based case for the technological advancement our work\nrepresents. And we thank the reviewer again for their thoughtful comments, which have been\nvery helpful in improving the clarity and quality of our manuscript.\nQ2. In the Introduction, authors have mentioned that “This enabled three applications: the\ngeneration of a large, AI-ready dataset;” etc. However, I don’t see any description of this\ndataset. Will it be possible for authors to describe that dataset.\nReply: We thank the reviewer for pointing out the question on improving data presentation. In\nfact, this dataset represents the complete collection of Organic Syntheses journal articles\nprocessed through the ReactionSeek pipeline. We selected 100 volumes spanning from 1921 to\n2021, conducted comprehensive reaction extraction, and generated corresponding SMILES and\nReaction SMILES representations. This dataset serves as the foundation for SynChat to\nfacilitate reaction retrieval and question-answering functionalities. The dataset description\nadded in the main text is presented as follows (On page 10 and 11 of the main text):\n\"... Based on the aforementioned evaluations, we ultimately employed GPT-3.5-16k to\nconduct data mining on Organic Syntheses volumes 1–100, spanning from 1921 to 2021. After\ncomplete pipeline processing and deduplication, we obtained 5,443 unique compounds, 3,961\nunique reactions, of which 3,580 contained yield information. ....\"\n\"... While modest in size compared to larger reaction databases like USPTO, the value of\nour minded dataset lies not in its breadth, but in its curated depth. It is composed of the most\nrepresentative and classical organic reactions across one century, with the exclusive inclusion\nof gram-scale procedures guaranteeing experimental robustness and reproducibility.\nConsequently, we propose that this collection can serve as a standardized reference for\nsynthetic chemists and as a high-fidelity benchmark for the development and validation of\nmachine learning models. ....\"\nFurthermore, the section \"LLM-Driven Discovery of Chemical Trends from Mined Data\"\npresents an analysis of the reactions contained within this dataset (On page 11 of the main text).\nQ3. Then, authors need to show which type of cases their approach can’t extract reactions and\nwhether they have taken care to improve those. For example, in the work\nhttps://link.springer.com/article/10.1186/s13321-024-00928-8, fig 5, the authors identified\nnumber of challenges for text to reaction extraction. Whether the proposed approach can\novercome them?\nReply: We thank the reviewer for their valuable suggestions regarding the challenges for\nreaction extraction. Specifically, our work focuses on addressing: (1) the dispersion of\ncompound information across sub-headings and complex compound co-referencing, (2) name-\nto-SMILES conversion, and (3) compound identification in reference images. To our\nunderstanding, these may cover the five challenges identified in the mentioned reference (now\ncited as ref. 29). We list herein a detailed discussions:\n(1) the dispersion of compound information across sub-headings and complex\ncompound co-referencing: We employed a segregated input approach where sub-headings and\nsynthetic text are processed separately, enabling the LLM to effectively learn this\nrepresentational pattern. The relevant main text content is as follows (On page 5 of the main\ntext):\n\"... With this pipeline, the LLM can effectively identify compound entities within text\nsegments and establish their complex co-reference relationships. By processing sub-headings\nand text segments separately, our pipeline successfully achieves the association between\ncompound references in synthetic text segments and the corresponding compounds mentioned\nin sub-headings. ....\"\n(2) Chemical name-to-SMILES conversion: We developed a dedicated processing\nmodule for transforming chemical nomenclature into SMILES notation. The corresponding\ndiscussion in the main text is presented below (On page 7 of the main text):\n\"... The conversion of diverse compound nomenclature (including IUPAC names, common\nnames, misspelled variants, and chemical formulas) into machine-readable SMILES\nrepresentations remains a significant challenge in organic reaction mining. A critical\ncomponent of ReactionSeek is its data standardization module, which is designed to convert\nraw extracted text into an accurate, consistent, and AI-ready format (Fig. 2c). The module\nemphasizes two primary curation tasks: name-to-SMILES transformation and unit\nstandardization. ....\"\n(3) Identification of compounds in reference images: We developed an integrated\nmodule utilizing VLM-assisted OCSR tools for information association. The system operates\nthrough a multi-stage process: (i) the VLM component identifies textual and ideographic\nelements within structural images, determining both the compounds' functional roles in\nreactions and their corresponding identifiers; (ii) the OCSR module converts these identified\nstructures into SMILES notation. This pipeline enables the transformation of reaction\nschematics into chemically interpretable SMILES relationships. The corresponding discussion\nin the main text is presented below (On page 4 of the main text):\n\"... While existing Optical Character Structure Recognition (OCSR) tools can convert\nmolecular structures into machine-readable formats (e.g., SMILES), associating these\nstructures with their correct roles (e.g., reactant, product) and textual identifiers within complex\nfigures remains challenging. ....\"\nThat said, there still some remaining challenges that warrants further explorations, and\nthese include: (1) handling uncommon chemical names and structures, (2) interpreting complex\ncontextual relationships dispersed across the paper, and (3) a reliance on the performance of the\nunderlying OCSR tool and LLMs. Our ongoing work is dedicated to addressing these\nlimitations to achieve more comprehensive and robust reaction mining from the literature. In\nthe Discussion section, we address these challenges of our current pipeline (On page 14 of the\nmain text):\n\"... Current limitations include challenges in (1) handling uncommon chemical names and\nstructures, (2) interpreting complex contextual relationships dispersed across the paper, and (3)\na reliance on the performance of the underlying OCSR tool and LLMs. For (1), the structural\nelucidation of newly synthesized compounds presents significant challenges when these\ncompounds are neither registered in existing databases nor properly systematized through\nstandard nomenclature rules. For (2), for complex compound relationships presented in tables,\nReactionSeek still faces challenges in achieving satisfactory mining success rates. For (3),\ncurrently available OCSR tools still face significant challenges in R-group identification and\ncomplex structure recognition. Concurrently, the developmental stage of LLMs imposes\nlimitations on both the performance of LLMs and existing cheminformatic tools. ....\"\nQ4. There can be some comparison with some benchmarking dataset.\nReply: We are grateful for this constructive point! To clarify the advantages of our pipeline, we\nintroduce a benchmark dataset from the ChEMU 2020 competition for patented chemical\nreaction text mining to demonstrate the superiority of our pipeline under similar evaluation\nstandards. On this dataset, our system (In Review 1’s part) achieves an overall F1-score of\n0.983, outperforming the best-performing model in ChEMU 2022 when evaluated on ChEMU\n2020, while requiring significantly fewer annotated samples. Here is the added comparative\ntesting content from the main text (On page 5 of the main text):\n\"... To validate our data mining pipeline, we benchmarked its performance on the\nauthoritative ChEMU33-36, which achieved a top F1 score of 0.968 on key chemist-centric\nmetrics, including reactants, products, reaction conditions, and yield information (Fig. 3b, 3d).\nOur pipeline demonstrated a remarkable advantage in the few-shot learning scenario. With\nonly seven annotated examples, it achieved an F1 score of 0.983 in extracting critical reaction\ncomponents, including reactants, products, solvents, temperature, time, and yield, representing\na further improvement over the previous model (see more details in SI 3.3 Table S1). ....\"\nQ5. Similarly, for image to text conversion, RxnScribe\nhttps://pubs.acs.org/doi/10.1021/acs.jcim.3c00439 described number of challenges. The\nconclusion include “Despite the success in our experiments, there are a few limitations in this\nwork. First, we focus on parsing the reaction structure, but do not evaluate the final extracted\nreaction SMILES strings due to the lack of such ground truth in our dataset. The molecular\nstructure recognition model MolScribe and the OCR tool might introduce additional errors.\nSecond, we limit our study to diagrams presented in digital format, excluding those that are\neither scanned or hand-drawn. Furthermore, the extracted information from diagrams are\nsometimes incomplete. For example, the reaction conditions are sometimes listed in a table and\nthe molecules may involve R-groups which are defined elsewhere.” Whether proposed\napproach can make improvements over these limitations?\nReply: Thank you for your constructive feedback. Based on our understanding, the main points\nyou have raised and our responses are listed below:\n1. The validation and correctness of the data mining results: our manual verification was\nperformed on 55 articles tested from Organic Syntheses, all 6 SI documents used for SI\nmining, and the complete results from the ChEMU 2020 competition. We contend that\nthe manual verification of the Organic Syntheses portion, which was selected based on\na stratified sampling strategy, provides a reliable estimate of the mining accuracy across\nthe entire dataset.\n2. The potential for the OCSR step to introduce additional errors: the choice of the OCSR\ntool is a critical factor that directly determines the number of errors introduced during\nimage processing. We tried several OCSR tools and selected InDraw as the most\nsuitable module for our pipeline. Its performance achieving 70.1% accuracy in our tests,\nwhich is significantly higher than MolScribe (only 36.8%).\n3. The capability of our system to process figures and schemes from scanned documents:\nIn our image processing stage, we can handle both text and numbers from scanned\ndocuments by incorporating a Vision Language Model (VLM), which is responsible\nfor two key tasks: extracting text from images and identifying the relationships between\ndata points. Complementing this, an OCSR model is used specifically to convert\nchemical structures into SMILES strings. It is important to note that this OCSR process\nis designed to handle clean, computer-generated chemical structure images rather than\nscanned ones.\n4. The ability to handle complex data formats, such as information presented in tables or\nmolecules with R-groups defined elsewhere in the text. This task, which involves\nchallenging substructure stitching, is beyond the scope of our current pipeline. This is\nan active area of our research, and we expect these additions to significantly broaden\nthe pipeline's capabilities. We have now incorporated the following discussion into the\nmain text (On page 14 of the main text):\n\"... For (3), currently available OCSR tools still face significant challenges in R-group\nidentification and complex structure recognition. Concurrently, the developmental\nstage of LLMs imposes limitations on both the performance of LLMs and existing\ncheminformatic tools. Future work will, therefore, focus on enhancing robustness\nthrough more sophisticated prompt strategies, integrating advanced cheminformatics\ntools, and expanding the framework's scope to capture mechanistic details. Extending\nReactionSeek to related fields, such as polymer science or materials science, also\npresents a promising avenue for broader application. ....\"\nWe would like to thank the reviewer once again for their thorough and insightful review.\nTheir questions and challenges regarding data validation, methodological choices, and the\nhandling of complex cases have been invaluable. Addressing these points has allowed us to\nsignificantly clarify our contributions and strengthen the manuscript. We hope that our\nresponses and the corresponding revisions have adequately addressed all their concerns.\nQ6. In addition, images with reactions shown in cyclic format may face additional challenges\nand model may find it difficult to extract those images. How does your approach solve those\nproblems?\nReply: We thank the reviewer for pointing out the question. ReactionSeek employs VLM to\ndetermine compound relationships in images, where the OCSR model is solely responsible for\nidentifying specific molecular structures. Through this approach, VLM can recognize reaction\npathways in images by leveraging visual elements such as arrows to infer the roles of\ncompounds and map their corresponding labels in the image to those in the accompanying text.\nThus, by incorporating VLM, we have addressed the challenge of identifying reaction roles for\ncompounds depicted in images with moderate to good performance.\nHowever, ReactionSeek still face the challenges when extracting highly complex organic\nreaction images (e.g., metal-catalyzed cycles), which is constrained by the underling VLMs.\nWe are currently developing more advanced OCSR models and intelligent agents in our follow-\nup work to tackle more intricate image-mining tasks.\nQ7. In Fig 1(b) authors showed conceptual overview of the ReactionSeek framework. What will\nmodel do if the molecule of interest that needs to be synthesized is new? That is the real-life\nquestion for most cases. Existing tools like SciFinder from CAS can also perform the current\njob and probably have wider access to articles. Probably, additional agents like IBMRxn or\nopen source AiZynthFinder (https://jcheminf.biomedcentral.com/articles/10.1186/s13321-024-\n00860-x) can be used in such scenario. Moreover, if possible, additional agents can be called\nto identify the source, from which the starting ingredients can be bought. That can help to\noptimize the cost of the synthesis process.\nReply: This is indeed a question with significant research potential, and we thank the reviewer\nfor highlighting it. While this particular problem is outside the immediate scope of our current\nstudy's objectives, we are very keen to discuss and investigate it in our future work. Based on\nour understanding, the main points you have raised along with our detailed\nresponses/discussions are:\n1) How the current frame work provides synthesis suggestions for reactions that are not\npresent in its existing dataset. We believe this is a long-term aim for our integral\nReectionSeek-SynChat pipeline. SynChat recommends synthesis steps by first retrieving\nsimilar reactions from its database, continuingly generated by ReactioSeek. This retrieval-\nbased approach ensures that the LLM is grounded in factual data, which prevents the\ngeneration of fabricated or unreasonable synthesis steps and provides the model with highly\nrelevant supporting information for the target reaction.\n2) What are the advantages of our approach compared to established database retrieval tools\nlike SciFinder and IBMRxn. First, regarding query flexibility, SciFinder and IBMRxn\nrely on conventional database search logic, which cannot process natural language queries.\nIn contrast, SynChat is designed to understand more ambiguous, conversational requests\nand match them with the appropriate information. Second, concerning data quality and\nreliability, SciFinder and IBMRxn index a vast and heterogeneous collection of literature,\nwhich can introduce irrelevant results or suboptimal synthesis strategies. SynChat, however,\nis built upon the expert-guided selections of literatures for specific topic in organic\nsynthesis/reactions and in this work on Organic Syntheses dataset, a high-quality corpus\nwhere reactions have been carefully selected and experimentally verified by editors for\nreproducibility and performance. This curated data source ensures that our synthesis\nsuggestions are significantly more reliable.\n3) Whether the system can integrate with other agents to perform optimizations, for example,\nto minimize the economic cost of a synthesis pathway. SynChat is an ongoing development\nproject built on a highly compatible and extensible framework. This modular architecture\nis intentionally designed to facilitate the seamless integration of additional tools and agents.\nA key priority in our future work is to incorporate synthesis cost as a critical parameter in\nthe recommendation process. As a first step in this direction, we have already integrated a\nsynthetic accessibility (SA) scoring tool to aid in its recommendations. Our active\ndevelopment is focused on continuously augmenting SynChat with other functional\nmodules, with the ultimate goal of evolving it into a more versatile and general-purpose\nsynthesis assistant.\nWe are once again very grateful for your detailed and constructive feedback. The points\nyou raised concerning on synthesis planning were particularly helpful. This feedback provided\na clear path for enhancing our SynChat, and we are confident that our response has now fully\naddressed your concerns."}, {"round": 2, "reviewer_comments": {"reviewer_1": "Based on the revision, the reviewer understand that the quality of the extracted results may have improved compared to the previous version. It should be noted, however, that the authors used the ChEMU 2020 dataset due to its availability, while the performance of the previous system was reported using the ChEMU 2022 dataset.\nIn addition, the authors describe this improvement as \"significant\" in their response letter. However, since the performance of the previous method had already achieved a high level (F > 0.95), such a small improvement may not substantially affect the overall workflow of the extraction process, as manual verification is still required to obtain perfect results. While the reviewer acknowledges that improved extraction performance is desirable, it is important for the authors to clarify to what extent this small improvement is expected to reduce the overall cost or effort in practice.\nRegarding the comparison of the proposed approach with other methods, many of the claims appear to reflect the general characteristics of generative AI (LLM)-based systems. For example, in the discussion on \"flexibility and customization,\" the authors state that \"Our prompt-driven data mining approach accepts natural language instructions, allowing researchers to define custom data structures and extraction strategies.\" However, this property is common to most LLM-based methods for chemical information extraction.\nCompared with other applications of generative AI in the chemical domain, the reviewer could not clearly identify the unique or distinctive characteristics of the proposed method.\nhttps://github.com/OpenDFM/LLM4Chemistry\nWhile the reviewer acknowledges the value of the developed software, the paper reads more like a software development report centered on prompt engineering—a topic that has already been extensively explored in prior research.", "reviewer_2": "Thank you for your detailed responses and clarifications. I appreciate the effort in addressing the earlier questions. I still have a few remaining concerns and suggestions that may help strengthen the paper.\n**Q2:**\nI understand that obtaining diverse data can be challenging; however, using only **five** articles may not be sufficient to convincingly demonstrate the overall effectiveness of ReactionSeek. Expanding the evaluation with additional literature data would make the results more representative and persuasive. Moreover, including **confidence intervals** or measures of statistical variability could further support the robustness of your findings.\n**Q4:**\nThe current response remains somewhat unclear. Since the *representations or embeddings* play a crucial role in the system’s performance, it would be very helpful to explicitly how to use them in the main text, like do we need to reducing dimensionality? Clarifying how these embeddings are generated and utilized would significantly improve transparency and reproducibility.\n**Q5–Q6:**\nRegarding the **Chroma database**, it would be helpful to elaborate on its reliability for similarity calculations. Additionally, please clarify whether the embedding vector database should be **updated periodically** after new reactions are retrieved, and if so, how this updating process would be implemented in practice. Finally, I suggest including a **visualization** (e.g., a two-dimensional projection) of the embedding distributions to illustrate the differences between embeddings and to provide some semantic intuition for readers.", "reviewer_3": "The authors tried to answer most questions convincingly. I would be happier if they could show a comparison with ORD dataset/USPTO or some benchmarking data. In that way, one could be more convinced with the quality of the extracted data which is important for subsequent tasks. Nevertheless, authors have still done a good job. So, I recommend this article to Nature Communication."}, "author_response": "Responses and Revisions\nOur overall responses: We appreciate the reviewer's insightful comments, which have\nsignificantly helped us improve the quality of our paper. We have provided clarifications on the\nkey issues, and carefully addressed all the points raised and have revised the manuscript\naccordingly. A detailed point-by-point response is provided below.\nReviewer #1 (Remarks to the Author):\nQ1. Based on the revision, the reviewer understand that the quality of the extracted results may\nhave improved compared to the previous version. It should be noted, however, that the authors\nused the ChEMU 2020 dataset due to its availability, while the performance of the previous\nsystem was reported using the ChEMU 2022 dataset.\nReply: It is important to clarify that our selection of the ChEMU 2020 dataset does not\ncompromise the validity of the comparative analysis. Notably, high-performing models from\nthe ChEMU 2022 competition have reported their evaluation results on the ChEMU 2020\ndataset in related work. Consequently, the state-of-the-art baseline we referenced (F1=0.968)\n(ref. Jang, Y. J. et al. Context aware Named Entity Recognition and Relation Extraction with\nDomain-specific language model. In Conference and Labs of the Evaluation Forum (2022). )\nspecifically represents the performance of that model on the ChEMU 2020 dataset. Therefore,\nthis comparison effectively validates the superior performance of our pipeline.\nQ2. In addition, the authors describe this improvement as \"significant\" in their response letter.\nHowever, since the performance of the previous method had already achieved a high level (F >\n0.95), such a small improvement may not substantially affect the overall workflow of the\nextraction process, as manual verification is still required to obtain perfect results. While the\nreviewer acknowledges that improved extraction performance is desirable, it is important for\nthe authors to clarify to what extent this small improvement is expected to reduce the overall\ncost or effort in practice.\nReply: We appreciate the reviewer's encouraging and insightful comments. We contend that\nthe F1 improvement from 0.968 to 0.983 is highly significant, for two primary reasons:\nFirst, this advancement does not represent a marginal gain. It corresponds to a 46.9% relative\nreduction in the error rate, meaning the number of data extraction failures was nearly halved.\nFrom a practical standpoint, this dramatically lowers the high-cost burden of expert manual\ncorrection, as fixing errors is far more laborious than simple verification, which is the most\nchallenging part of labor data mining.\nSecond, in the high-F1 regime (F1 > 0.95), performance bottlenecks are typically caused by\na small number of complex or irregular cases. A significant F1 increase in this range—as\ndemonstrated here—implies a breakthrough in resolving these rare but systematic challenges\n(e.g., ambiguous compound co-references), which is a non-trivial technical achievement.\nQ3. Regarding the comparison of the proposed approach with other methods, many of the\nclaims appear to reflect the general characteristics of generative AI (LLM)-based systems. For\nexample, in the discussion on \"flexibility and customization,\" the authors state that \"Our\nprompt-driven data mining approach accepts natural language instructions, allowing researchers\nto define custom data structures and extraction strategies.\" However, this property is common to\nmost LLM-based methods for chemical information extraction.\nReply: We fully appreciate the reviewer's concerns regarding the innovative contribution of our\nmethod's advantages. In fact, simply applying LLMs ‘off-the-shelf’ to organic synthesis mining\nfails to meet the practical needs of researchers. This is because the direct application of LLMs\nin this domain is confronted with numerous domain-specific challenges that must be properly\naddressed.\nFirst, knowledge in organic synthesis is not confined to textual descriptions; it is intricately\nlinked with information from diagrams (e.g., reaction schemes). This makes comprehensive\ndata extraction an inherently multimodal problem, demanding separate processing and\nsubsequent association of text and images.\nFurthermore, organic chemistry research involves a vast number of complex molecules that\nmust be converted into canonical representations like SMILES. This specific task has proven\ndifficult for most LLMs to handle robustly. Concurrently, retrieval-based solutions (e.g.,\nquerying open databases) struggle to keep pace with the rapid discovery rate of novel small\nmolecules.\nIt is precisely because of these difficulties in directly applying LLMs that we proposed our\npipeline. Our innovation lies in integrating the LLM's strengths (its 'common advantages') into\na workflow that solves these domain-specific challenges, thereby enabling the LLM to be\neffective in this context.\nWhile the ChEMU benchmark, due to its task definition, did not allow our modules to\ndemonstrate their full end-to-end efficacy, the complete capabilities of our pipeline were\nthoroughly validated during our construction of the Organic Syntheses dataset.\nQ4. Compared with other applications of generative AI in the chemical domain, the reviewer\ncould not clearly identify the unique or distinctive characteristics of the proposed method.\nhttps://github.com/OpenDFM/LLM4Chemistry\nReply: Our research is focused on addressing critical challenges within the organic chemistry\ndata domain. Compared to previous work, our pipeline's advantages are primarily demonstrated\nin the following aspects:\n1. Superior Domain Transferability (vs. Fine-Tuned Models): Our pipeline, which is\nbased on a general-purpose LLM, possesses significantly stronger domain transferability.\nWithin the diverse branches of organic chemistry, the terminology used for compound types\nand experimental procedures varies considerably. Fine-tuned models require extensive re-\ntraining and fine-tuning for the specific linguistic patterns of each branch. This process becomes\ninfeasible for sub-fields that lack sufficient pre-existing annotated data. In contrast, our\nprompt-engineering-based pipeline allows researchers to effectively inject their own\ndomain knowledge, enabling robust data mining with minimal data annotation costs.\n2. Enhanced Specialization and Robustness (vs. Broad Chemical Models): Our pipeline,\nwhich synergizes an LLM with external tools, offers greater domain-specific capability. A\nconsiderable amount of existing work has focused on general chemical mining by training on\nbroad chemical corpora. However, these models often exhibit suboptimal performance when\nconfronted with the unique, domain-specific challenges of organic synthesis (as detailed in our\nresponse to Q3).\nThe critical challenge in this domain is not merely achieving high extraction scores, but\nensuring robustness in handling a wide array of complex situations. Our pipeline addresses\nthis by utilizing a hybrid 'LLM + tools + database' approach for data standardization and\nstructuring—optionally integrated with OCR module. This method significantly enhances the\nrobustness of converting extracted content into machine-actionable reaction information,\ndirectly responding to this key challenge in the organic chemistry field.\n3. Contribution of a Novel, Curated Dataset: In terms of chemical contribution, we\nemployed our pipeline to curate and contribute a structured synthesis dataset for the\nchemoinformatics community, collected from a high-quality corpus of organic synthesis\nliterature. This process serves a dual purpose: (1) it demonstrates the performance of our\npipeline's modules in a practical, real-world application scenario, and (2) it provides a valuable\ncollection of synthetic methods that are expert-validated by organic chemists and confirmed\nto be fully reproducible. In our work, we further leveraged this dataset to build an intelligent\nQ&A assistant SynChat supported by vector retrieval. We anticipate this dataset will serve as a\nvaluable resource for the community, and we hope subsequent researchers will further develop\nand utilize it, contributing to the advancement of organic chemistry research.\nWhile the reviewer acknowledges the value of the developed software, the paper reads more\nlike a software development report centered on prompt engineering—a topic that has already\nbeen extensively explored in prior research.\nReply: As stated previously, our research is focused on addressing critical challenges within\nthe organic chemistry data domain. While prompt engineering has been broadly applied\nacross chemical data mining, its application still faces intractable challenges specific to organic\nchemistry.\nOur research aims to provide solutions for these extant challenges by integrating prompt\nengineering with other technologies. This approach, on one hand, broadens the applicability of\nprompt engineering in organic synthesis, and on the other, enhances the comprehensive\nperformance of LLMs for this specific task.\nThe introduction of our pipeline underscores a critical point: despite the excellent\nperformance and extensive research into LLMs for general data mining, critical, unsolved\nproblems persist when they are applied to specialized domains. This is far from an issue\nthat 'has already been extensively explored.' Rather, it remains a significant challenge that\ndemands continuous methodological refinement to make these tools genuinely serviceable for\npractical synthesis research.\nReviewer #2 (Remarks to the Author):\nQ1: I understand that obtaining diverse data can be challenging; however, using only **five**\narticles may not be sufficient to convincingly demonstrate the overall effectiveness of\nReactionSeek.\nReply: We sincerely appreciate the reviewer’s constructive comments. In response, we have\nincorporated mining results from all 29 newly published articles containing synthesis reactions\nfrom the latest volume of Organic Syntheses (Volume 102) into the main text (Page 11). We\nachieved F1 scores exceeding 0.97 across all metrics, demonstrating that our pipeline maintains\nrobust performance even when applied to newly published literature. The specific revisions in\nthe main text are as follows:\n\"... Table 1. Reaction mining test results and scores for recently published Organic Syntheses\narticles.\nReactant Product\nEvaluation Reactants Products Solvents Temperature Time Yield Total\namount amount\nTP 171 169 54 60 67 76 75 57 729\nFN 2 2 0 2 0 2 1 3 12\nFP 1 3 0 0 0 1 1 0 6\nPrecision 0.9942 0.9883 1.0000 1.0000 1.0000 0.9870 0.9868 1.0000 0.9918\nRecall 0.9884 0.9826 1.0000 0.9677 1.0000 0.9744 0.9868 0.9500 0.9838\nF1 0.9913 0.9854 1.0000 0.9836 1.0000 0.9806 0.9868 0.9744 0.9878\nTo demonstrate our pipeline's generalizability to newly published literature, we tested it\non all newly published articles (29 articles) from the 2025 volume of Organic Syntheses (Fig.\n4e and Table 1). Our pipeline correctly extracting most of reaction information from each\narticle. During the standardization phase, with the exception of molecular sieves that cannot\nbe directly represented as SMILES, all other compounds were successfully converted into their\ncorrect corresponding SMILES representations. These results confirm our pipeline’s robustness,\nshowcasing its ability to generalize effectively from benchmark datasets to newly published\nchemical literature. ...\"\nQ2. Moreover, including **confidence intervals** or measures of statistical variability could\nfurther support the robustness of your findings.\nReply: We sincerely thank the reviewer for their constructive comments regarding data\npresentation, which have significantly helped us improve the manuscript. In the validation for\nthe most recent articles, we have included 95% confidence intervals for the pipeline's\nperformance. These results demonstrate the exceptional stability of our pipeline when\nprocessing individual independent texts. The updated section of the main text (On page 9) is as\nfollows:\n(e) Extraction demonstration on the latest article reactions from Organic Syntheses. Scatter plot\nshowing the distribution of mining F1 scores in selected individual articles. Error bars denote\nthe 95% confidence intervals.\nQ3: The current response remains somewhat unclear. Since the *representations or\nembeddings* play a crucial role in the system’s performance, it would be very helpful to\nexplicitly how to use them in the main text, like do we need to reducing dimensionality?\nClarifying how these embeddings are generated and utilized would significantly improve\ntransparency and reproducibility.\nReply: We are grateful for the reviewer's constructive feedback. Reaction data are vectorized\nusing the all-MiniLM-L6-v2 embedding model, converted into a 384-dimensional embedding\nvector without any dimensionality reduction. These embeddings are stored in Chroma and serve\nto identify associations between a user's query features and the stored reaction features. Based\non your feedback, we have revised the Methods section of the manuscript (on page 15 of the\nmain text) to better clarify the dimensionality, generation method, and application of the\nembeddings:\n\"... The string stored in the database consists of concise, preprocessed information,\nincluding the title, DOI, reactants, products, SMILES, yield, and other identifying features. The\ndatabase is divided into two components: a query database for retrieval purposes and a\nsupplementary database for supplementary information. The query database consists of textual\nfields including product SMILES, ReactionSMILES, product names, reaction names, titles, etc.,\nwhich are vectorized through the Chroma vector database.53 These fields are vectorized using\nthe all-MiniLM-L6-v2 embedding model, an SBERT-based model managed via the Chroma\nvector database. This process converts the input text into a 384-dimensional embedding vector,\nwhich is used directly for the subsequent matching process without any dimensionality\nreduction. Meanwhile, detailed reaction information such as synthetic procedures, yields,\nreagent quantities, reaction conditions, and citation data are separately stored in the\nsupplementary database. Subsequently, the model receives information from the associated\nentries in the dataset based on the vector matches and character matches. ...\"\nQ4–Q5: Regarding the **Chroma database**, it would be helpful to elaborate on its reliability\nfor similarity calculations. Additionally, please clarify whether the embedding vector database\nshould be **updated periodically** after new reactions are retrieved, and if so, how this\nupdating process would be implemented in practice. Finally, I suggest including a\n**visualization** (e.g., a two-dimensional projection) of the embedding distributions to\nillustrate the differences between embeddings and to provide some semantic intuition for\nreaders.\nReply: We deeply appreciate the reviewer's constructive questions and suggestions for SynChat.\nIn response to the questions you raised, we have summarized our replies as follows:\n1. Detailed Explanation of Chroma's Reliability: Within our framework, we use the all-\nMiniLM-L6-v2 (SBERT) model to generate 384-dimensional vectors for the textual\nreaction descriptions. We then utilize cosine similarity to match user queries against the\ndatabase information. The matching is based on the HNSW (Hierarchical Navigable Small\nWorld) algorithm. This is an industry-standard method for fast, high-recall search in large-\nscale, high-dimensional datasets. While it is technically an \"approximate\"\n(Approximate Nearest Neighbor) search, its performance (e.g., recall@10) on standard benchmarks is\nextremely high, fully satisfying the requirements for scientific research retrieval.\n2. Regarding the Continuous Updating of the Vector Database: Yes, it will be. SynChat is\nan active developing project that aims to integrate all the chemoinformatics tools and\ndatasets developed (both present and future) by our research group. Therefore, the dataset\nwill be further expanded as new data becomes available. New, validated, high-quality data\nwill be formulated into textual descriptions and vectorized in the same manner, before being\nstored in the vector database to be queried. The corresponding full-text content will also be\nassociated with each entry, enabling the LLM to access the complete reaction information.\n3. Visualization of the Embedding Distribution: We generated embeddings for the entire\nOrganic Syntheses dataset and performed UMAP dimensionality reduction to visualize the\ndistribution of molecules containing different functional groups within the embedding\nspace. Molecules with distinct functional groups exhibited cluster and band-like\ndistributions in the reduced-dimensional plot, indicating that functional group information\nwas correctly extracted. Furthermore, the overlaps observed between molecules with\ndifferent functional groups reflect the significance of molecular scaffold information within\nthe embeddings. The integration of both functional group and scaffold information ensures\nthe accuracy of retrieval. Additionally, under the current scheme, molecules possessing\nmultiple functional groups tend to align simultaneously with the bands of distinct functional\ngroups, manifesting as overlaps between clusters and bands. As this specific design is not\nthe primary focus of this work, we have provided a more detailed discussion in the Methods\nsection and Supplementary Information (SI), as follows:\nIn the Methods section (On page 15 of main text): \"... These fields are vectorized using the\nall-MiniLM-L6-v2 embedding model, an SBERT-based model managed via the Chroma\nvector database. This process converts the input text into a 384-dimensional embedding\nvector (the specific distribution is detailed in the SI 5), which is used directly for the\nsubsequent matching process without any dimensionality reduction. Meanwhile, detailed\nreaction information such as synthetic procedures, yields, reagent quantities, reaction\nconditions, and citation data are separately stored in the supplementary database.\nSubsequently, the model receives information from the associated entries in the dataset based\non the vector matches and character matches. ...\"\nIn the Supplementary Information (Chapter S5): \"...\nFigure S8 UMAP visualization of reaction embeddings based on different functional groups.\nEmbedding Analysis. We performed a visual projection of the Organic Syntheses dataset\nto illustrate how the embedding method encodes structural features, specifically functional\ngroups. In the reduced-dimensional space, molecules with different functional groups exhibit\ndistinct banding patterns, confirming that the model captures the semantic information\nregarding functional groups from SMILES. Additionally, the overlap observed between\nclusters—attributed to the non-mutually exclusive nature of functional groups—indicates\nthat the embedding distance also preserves scaffold structural information. This ensures that\nmolecules with similar backbones remain close in the latent space. Thus, the proposed\nmethod achieves a balance between capturing skeletal structure and functional group identity,\nenabling effective molecular retrieval. ...\"\nReviewer #3 (Remarks to the Author):\nThe authors tried to answer most questions convincingly. I would be happier if they could show\na comparison with ORD dataset/USPTO or some benchmarking data. In that way, one could\nbe more convinced with the quality of the extracted data which is important for subsequent\ntasks. Nevertheless, authors have still done a good job. So, I recommend this article to\nNature Communication.\nOur overall responses: We are very grateful for the reviewer's positive and encouraging\ncomments on our manuscript. To address the point regarding the benchmarking data:\nThe ChEMU dataset, introduced at the beginning of our paper, serves as more than just a\nmethodological comparison. The competition provided a standardized patent synthesis text\ncorpus intended for fair comparisons against other data mining solutions. Consequently, this\ndataset has been widely adopted as standard benchmarking data for text-mining tasks.\nOne of the core components of our pipeline, the text-mining module, demonstrated exceptional\nperformance on this specific benchmark, surpassing the state-of-the-art (SOTA) model from\nthe competition (as detailed on page 5 of the main text):\n\"... To validate our data mining pipeline, we benchmarked its performance on the\nauthoritative ChEMU33-36, which achieved a top F1 score of 0.968 on key chemist-centric\nmetrics, including reactants, products, reaction conditions, and yield information (Fig. 3b, 3d).\nOur pipeline demonstrated a remarkable advantage in the few-shot learning scenario. With\nonly seven annotated examples, it achieved an F1 score of 0.983 in extracting critical reaction\ncomponents, including reactants, products, solvents, temperature, time, and yield, representing\na further improvement over the previous model (see more details in SI 3.3 Table S1). ...\"\nIn this section, we ensured the credibility of our evaluation through meticulous manual\ninspection of both the annotated (ground-truth) data and our mined results. This process\nvalidates our assessment and confirms the superior performance of our pipeline's text-mining\nmodule."}, {"round": 3, "reviewer_comments": {"reviewer_1": "Response to Reviewer 1, Q1:\nThe reviewer understands the authors’ points. However, it would be better to include this information in the paper to justify that the comparison is meaningful.\nResponse to Reviewer 1, Q2:\nThe reviewer acknowledges that the reduction in the error rate is substantial. However, when correcting errors in the results, the time required to identify errors may not be reduced, and the overall time required for error identification and correction may not be significantly different.\nOn the other hand, the reviewer also recognizes that the proposed system addresses difficult extraction patterns.\nIt may be beneficial to discuss the impact of this improvement from the perspective of real-world usage scenarios, although this is not mandatory.\nResponse to Reviewer 1, Q3 and Q4:\nNo additional comments.", "reviewer_2": "The authors have addressed all of my concerns, and I believe that this manuscript can be accepeted for publication in the journal.\n(Remarks on code availability)\nThe code looks okay."}, "author_response": "Responses and Revisions\nReviewer #1 (Remarks to the Author):\nOur overall responses: We appreciate the reviewer's insightful comments, which have\nsignificantly helped us improve the quality of our paper. We have provided clarifications on the\nkey issues, and carefully addressed all the points raised and have revised the manuscript\naccordingly. A detailed point-by-point response is provided below.\nResponse to Reviewer 1, Q1: The reviewer understands the authors’ points. However, it would\nbe better to include this information in the paper to justify that the comparison is meaningful.\nReply: We thank the reviewer for the constructive comments. We have revised the main text to\nclarify the fairness of the comparison, emphasizing that our strategy ensures all methods are\nevaluated on the same accessible dataset. The specific revisions are as follows (on page 4 of\nmain text):\n\"... To validate our data mining pipeline, we benchmarked its performance on the authoritative\nChEMU34-37). This selection ensures a rigorous comparative analysis, as state-of-the-art\nmodels from the ChEMU-2022 competition have explicitly established a high-performance\nbaseline (F1 = 0.968) on this specific dataset. Our pipeline demonstrated a remarkable\nadvantage in the few-shot learning scenario. With only seven annotated examples, it achieved\nan F1 score of 0.983 in extracting critical reaction components, including reactants, products,\nsolvents, temperature, time, and yield, representing a further improvement over the previous\nmodel (see more details in Supplementary Note SI 3.3 Table S1). ...\"\nResponse to Reviewer 1, Q2: The reviewer acknowledges that the reduction in the error rate is\nsubstantial. However, when correcting errors in the results, the time required to identify errors\nmay not be reduced, and the overall time required for error identification and correction may\nnot be significantly different.\nReply: We fully acknowledge the reviewer’s concern regarding the time cost associated with\nmanual data correction. During our manual verification and correction process for the ChEMU\nbenchmark, we observed that the time required for mere correctness verification is significantly\nlower than that for correcting erroneous data. This disparity arises because correcting errors\noften necessitates that human experts manually input or reconstruct complex data types—such\nas chemical structures, compound names, and reaction strings—which is far more labor-\nintensive than simply verifying numerical values like yields or temperatures. Consequently,\nwhile automation may not entirely eliminate the verification step, reducing the initial error rate\nsignificantly lowers the burden of manual correction—specifically the time-consuming entry\nof complex chemical data—thereby offering substantial practical value."}]

Decision letter