Structural Reasoning Improves Molecular Understanding of LLM
Authors: Yunhui Jang, Jaehyung Kim, Sungsoo Ahn
Accepted at: acl25

Structural Reasoning Improves Molecular Understanding of LLM

Yunhui Jang
KAIST
yunhuijang@kaist.ac.kr

Jaehyung Kim
Yonsei University
jaehyungk@yonsei.ac.kr

Sungsoo Ahn
KAIST
sungsoo.ahn@kaist.ac.kr

Abstract
Recently, large language models (LLMs) have shown significant progress, approaching human perception levels. In this work, we demonstrate that despite these advances, LLMs still struggle to reason using molecular structural information. This gap is critical because many molecular properties, including functional groups, depend heavily on such structural details. To address this limitation, we propose an approach that sketches molecular structures for reasoning. Specifically, we introduce Molecular Structural Reasoning (MSR) framework to enhance the understanding of LLMs by explicitly incorporating the key structural features. We present two frameworks for scenarios where the target molecule is known or unknown. We verify that our MSR improves molecular understanding through extensive experiments.

1 Introduction
Large language models (LLMs; Touvron et al., 2023; OpenAI and et al., 2024; Raffel et al., 2020) have demonstrated remarkable performance across various tasks. To leverage their potential in chemistry, several prior works (Edwards et al., 2022; Christofidellis et al., 2023a; Fang et al., 2024; Pei et al., 2023) have proposed chemical LLMs (i.e., specialized LLMs pre-trained on both natural language and molecular representations) for molecular tasks such as molecule captioning, description-based molecule generation (Edwards et al., 2022), and retrosynthesis (Fang et al., 2024).

However, chemical LLMs still struggle to fully understand the molecular structure (Ganeeva et al., 2024; White et al., 2023). This is critical since structure-based reasoning plays an important role in many molecular tasks. For instance, chemists often consider a molecule toxic if it contains a phenol group, as phenoxy radicals can form and interact with biological membranes (Hansch et al., 2000). This becomes even more evident in real-world applications of LLMs. As demonstrated in Figure 1a, injecting accurate structural information into the model can slightly improve its ability to generate correct molecules. This highlights the importance of explicitly incorporating structural reasoning into LLMs.

To address this aspect, we consider a framework for LLMs to first reason about the molecular structure for molecular tasks, similar to how LLMs improve arithmetic and commonsense tasks through intermediate reasoning steps (Wei et al., 2022; Kojima et al., 2022). A naïve approach is to prompt LLMs to infer the structural information before generating an answer. However, we find this to be ineffective since even the state-of-the-art LLMs (OpenAI and et al., 2024; Touvron et al., 2023) struggle to accurately capture essential molecular structures, as described in Figure 1b and Section 2.

In this paper, we propose MSR, a simple yet general framework for Molecular Structural Reasoning that progressively sketches the structural features of molecules to improve molecular task performance. To this end, we identify key structural elements crucial for the reasoning of LLMs to solve molecular tasks. Moreover, we propose fine-tuning procedures that employ external tools to identify the molecular structural information.

In particular, our frameworks to fine-tune LLMs for molecular structural reasoning are designed for both molecular and non-molecular inputs. A framework consists of reasoning module and an answering module. The reasoning module generates structural information to enhance the understanding of the molecule. The answering module generates the final answer based on the original input and the output of the reasoning module. The overall frameworks are illustrated in Figure 4.

To be specific, we consider two types of reasoning framework, inspired by how humans generally form knowledge via analysis and synthesis (Kant, 1899). On the one hand, analysis refers
Description
Given the description of a molecule, your job is to predict the SMILES representation of the molecule.

Description: The molecule is the organoarsenic acid anion formed by loss of a single proton from the arsenic center group in phenylarsonic acid. It is a conjugate base of a phenylarsonic acid.

MSR
The molecular formula is C6H5AsO2−. The molecule contains no carbon chain. The molecule contains 1 aromatic ring. It includes 1 benzene ring. The functional groups present in the molecule include acid, aromatic, and organic groups. The molecule has no chiral centers.

(a) Incorporating MSR improves GPT-4o in molecule generation.

(b) LLMs’ capability of structural inference.

Figure 1: Overview of LLMs with structural information. (a) Each color in MSR represents a structural component. The top molecule is incorrectly generated using only the description while the bottom is correctly generated by incorporating the description and MSR. (b) Despite the importance of structural information, even recent LLMs struggle to accurately infer key structural details from molecular representations such as SMILES (Molecule-to-structure; M2S) or given descriptions (Text-to-structure; T2S).

to breaking down complex information into fundamental components for better understanding. In molecular tasks, analytic reasoning applies when the molecule is provided as input, allowing the model to decompose its structure for meaningful insights. Specifically, we utilize external tools like RDKit (Landrum et al., 2024) as the reasoning module to precisely extract structural information from the molecule.

On the other hand, synthesis constructs a whole from its constituent parts. This aligns with molecular tasks where the molecule must be generated from non-molecular input, e.g., textual description, requiring the model to infer structural information and reconstruct the entire molecule. In detail, for synthetic reasoning, we fine-tune the LLMs as the reasoning module that generates MSR (Ho et al., 2023; Fu et al., 2023; Magister et al., 2023) based on the given input. We additionally incorporate a novel matching-ratio-based rejection sampling into the answering module, to ensure that the generated molecule aligns with MSR, using the external tools for validation.

We empirically show that incorporating MSR into chemical LLMs (Edwards et al., 2022; Christofidellis et al., 2023a) and general LLMs (Touvron et al., 2023; OpenAI and et al., 2024) both lead to consistent performance improvements in three molecular tasks: molecule-to-text, retrosynthesis, and text-to-molecule. In particular, chemical LLMs outperform the considered baselines when combined with our MSR framework. In summary, our contributions are as follows:

• We identify and evaluate the limits of LLMs in inferring molecular structural information.

• We propose MSR, a simple yet broadly applicable molecular reasoning framework that progressively sketches molecular structures.

• We introduce an analytic reasoning for MSR when the input molecule is given, leveraging external tools for structural identification.

• We develop a synthetic reasoning for MSR when the molecule is in the desired output, incorporating fine-tuning for the reasoning module and a novel matching ratio-based rejection sampling procedure for the answering module.

• We validate the effectiveness of MSR by demonstrating consistent performance improvements across chemical and general LLMs.

2 Recent large language models do not understand structural information

Here, we demonstrate that even the recent LLMs, i.e., GPT-4o (OpenAI and et al., 2024) and Llama3-8B-Instruct (Touvron et al., 2023), fail to infer important structural information from the given inputs, such as molecular representations (e.g., SMILES (Weininger, 1988)) and the text descriptions (Edwards et al., 2021). Notably, such tasks can be considered straightforward for individuals with a bachelor’s degree in chemistry.

Our analysis is inspired by how chemists reason about the structure to analyze a molecule. They progressively identify the molecular structure, starting with primary elements like rings and long carbon chains before identifying finer details such as
Figure 2: The six key structural information: molecular formula, longest carbon chain length, aromatic rings, ring compounds, functional groups, and chirality. The same color indicates the structural information and the corresponding component of the molecule.

functional groups. Reflecting on this behavior, we define six significant key structural elements for chemical reasoning as illustrated in Figure 2. In detail, these six key structural components include (1) molecular formula, (2) longest carbon chain length, (3) aromatic rings, (4) ring compounds, (5) functional groups, and (6) chirality.

Molecular formula. The molecular formula provides essential information about a molecule’s composition, specifying the number and type of atoms present. This information is critical because, for example, it directly determines the molecular weight. To illustrate, although 2-Butanol (C4H10O) and 2-Propanol (C3H8O) are composed of the same type of atoms, i.e., carbon, hydrogen, and oxygen, their differing molecular formulas result in distinct molecular weights (74.1g/mol for 2-Butanol and 60.1g/mol for 2-Propanol). These differences lead to the change in boiling points, 99.4°C and 82.3°C, respectively, as shown in the gray part of Figure 3.

Longest carbon chain. The longest carbon chain (excluding atoms in ring systems) forms the molecular backbone where functional groups are attached. The length of this chain significantly influences properties like solubility. For example, extending the carbon chain of 2-Butanol from four to six carbons creates 2-Hexanol, which exhibits reduced solubility. This is illustrated in the green section of Figure 3.

Aromatic rings. Aromatic rings (e.g., benzene and pyridine) play a critical role in determining the stability and electronic properties. For instance, adding a benzene ring to 2-Butanol yields 1-Phenyl-2-Propanol, which has enhanced stability and greater oxidation resistance. This transformation is shown in the blue section of Figure 3.

Ring compounds. Similar to the longest carbon chain, ring structures often serve as the backbone where functional groups are attached. The ring system significantly affects molecular behavior and reactions. For example, although 2-Butanol and Cyclobutanol share the same number of carbons and oxygen, the ring in Cyclobutanol introduces a tendency toward ring-opening reactions, as depicted in the yellow section of Figure 3.

Functional groups. Functional groups, e.g., hydroxyl, amino, ester, etc., play a pivotal role in determining the chemical reactivity. For example, alcohols with a hydroxyl group (-OH) are prone to oxidize more while the molecules with an amino group (-NH2) are generally resistant to oxidation under mild conditions. A single replacement of a hydroxyl (-OH) group in 2-Butanol with an amino (-NH2) group leads to 2-Butanamine, which has increased oxidation resistance, as described in the red part of Figure 3.

Chiral centers. Chirality refers to the stereochemical property of a molecule that makes it non-superimposable on its mirror image, leading to different chemical behaviors. The chirality is determined by the chiral centers and their configurations, i.e., R- and S-configuration 1, which describe the spatial arrangement of the groups around the chiral centers. This leads to different interactions between other molecules with chirality. For instance, (R)-2-Butanol and (S)-2-Butanol may interact differently with other chiral substances. This is described in the purple part of Figure 3.

Despite their significance, we observe that even recent LLMs often fail to accurately infer crucial structural details from the molecule or their text description. Specifically, as shown in Figure 1b, both GPT-4o and LLaMA3-8B-Instruct fail to capture the structural information accurately when the molecule is provided (Molecule-to-structure; M2S) or the description is provided (Text-to-structure; T2S). Notably, we provide detailed experimental settings and prompts in Appendix A.1.

1The names of R and S come from the Latin word Rectus and Sinister, which means right and left, respectively.
<table>
  <tr>
    <th>Component</th>
    <th>Expression</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Molecular formula</td>
    <td>The molecular formula is \( X_1N_1 \cdots X_{34}N_{34} \)</td>
    <td>\( X_m \): m-th atom type<br>\( N_m \): # of m-th atoms</td>
  </tr>
  <tr>
    <td>Longest carbon chain length</td>
    <td>The longest carbon chain is \( N \) carbons long.</td>
    <td>\( N \): the length of the longest carbon chain</td>
  </tr>
  <tr>
    <td>Aromatic rings</td>
    <td>The molecule contains \( N_r \) aromatic rings</td>
    <td>\( N_r \): # of aromatic rings</td>
  </tr>
  <tr>
    <td>Ring compounds</td>
    <td>It includes \( X_1X_1 \) rings, \( \cdots, X_{34}X_{34} \) rings)</td>
    <td>\( N_{m,0} \): # of m-th ring<br>\( X_{m,0} \): IUPAC name of m-th ring</td>
  </tr>
  <tr>
    <td>Functional groups</td>
    <td>The functional groups include \( X_1, \cdots, X_{34} \) group.</td>
    <td>\( X_{m} \): the name of functional group</td>
  </tr>
  <tr>
    <td>Chirality</td>
    <td>The molecule has \( N \) chiral centers: \( N_S \) with S configuration and \( N_R \) with R configuration.</td>
    <td>\( N_S \): # of chiral centers of S config.<br>\( N_R \): # of chiral centers of R config.<br>\( N = N_S + N_R \)</td>
  </tr>
</table>

Table 1: The description of each component of MSR.

First, when provided with a molecule (M2S), both GPT-4o and LLaMA3-8B-Instruct struggled to achieve high accuracy. Even in their best-performing case, counting the number of aromatic rings, their accuracies remained low, at approximately 50% and 75%, respectively. Similarly, when given a detailed text description (T2S), both models still failed to achieve high accuracy. This implies that LLMs struggle to fully understand the molecular structures, whether presented as molecular representations or text descriptions. These observations highlight the potential benefits of explicitly incorporating structural reasoning to enhance molecular comprehension.

3 MSR: Molecular Structural Reasoning

Here, we present our framework for enhancing LLMs’ understanding of molecules through Molecular Structural Reasoning (MSR). MSR incorporates six key structural elements as reasoning for LLMs, following a two-stage process (Zhang et al., 2024): a reasoning stage and an answering stage. First, a reasoning module generates MSR, providing supplementary structural information for a better understanding of the molecule. Next, an answering module generates the final output using the input augmented with the generated MSR. The framework is illustrated in Figure 4.

To address various tasks, we categorize the MSR framework based on whether the molecule is provided as input (analytic reasoning) or must be inferred as output (synthetic reasoning). In summary, for analytic reasoning, one decomposes complex molecules into fundamental structural components to better understand their structure. For synthetic reasoning, one constructs the entire molecule from its constituent structural components.

3.1 Overview of MSR

Here, we introduce MSR, a molecular structural reasoning framework that enhances language models’ understanding of molecules. Each component of MSR corresponds to one of the six structural elements introduced in Section 2. The expression and corresponding description of the reasoning for each structural component in MSR are provided in Table 1. Additionally, a concrete example illustrating MSR is shown in Figure 2.

3.2 Analytic reasoning

In MSR, analytic reasoning refers to decomposing a given input molecule into smaller structural components for enhanced comprehension. When the input molecule is available, one can utilize a deterministic reasoning module for its decomposition. Our approach integrates MSR by (1) employing external tools like RDKit (Landrum et al., 2024) to extract precise structural information as a reasoning module, and (2) fine-tuning the answering module LLM with the generated rationale as an additional input. The overall workflow is described in Figure 4a.

Reasoning module. In the analytic reasoning scenario, we employ external tools to extract precise structural information from the input molecule. This process eliminates uncertainty, as the structural information is deterministic for a given molecule. Next, this information serves as MSR, which guides the answering module.

Answering module. With the molecule and its corresponding MSR as input, we fine-tune the chemical LLMs to generate the desired output of the molecule. In our experiments, we mainly consider MolT5 (Edwards et al., 2022) and ChemT5 (Christofidellis et al., 2023a), as the answering module.

3.3 Synthetic reasoning

Synthetic reasoning refers to composing structural information to construct an entire molecule. When the input molecule is unavailable, the relevant structural information must first be inferred before generating the final molecule. To address this, we fine-tune a reasoning module to generate MSR, which is then attached to the input and utilized by the answering module to generate the final molecule, as illustrated in Figure 4b.
Figure 4: Overview of MSR fine-tuning framework. Analytic reasoning applies when the input molecule is available, while synthetic reasoning applies when it is not. Light gray boxes denote the molecules (SMILES); gray boxes denote related description; colored boxes represent MSR. The yellow and the red modules perform reasoning and answering, respectively. In (a), yellow module indicates an external tool. In (b), colors indicate MSR and the corresponding structural elements; here, the third molecule is chosen after matching ratio-based rejection sampling according to its highest matching ratio (3/3).

<table>
  <tr>
    <th></th>
    <th>BL.2</th>
    <th>BL.4</th>
    <th>RO.1</th>
    <th>RO.2</th>
    <th>RO.L</th>
    <th>ME.</th>
  </tr>
  <tr>
    <th colspan="7">Baselines (without reasoning)</th>
  </tr>
  <tr>
    <td>Meditron-7B</td>
    <td>0.792</td>
    <td>0.576</td>
    <td>0.797</td>
    <td>0.602</td>
    <td>0.575</td>
    <td>0.757</td>
  </tr>
  <tr>
    <td>Mol2Lang-VLM</td>
    <td>0.777</td>
    <td>0.563</td>
    <td>0.786</td>
    <td>0.591</td>
    <td>0.565</td>
    <td>0.741</td>
  </tr>
  <tr>
    <td>BioT5+</td>
    <td>0.798</td>
    <td>0.579</td>
    <td>0.812</td>
    <td>0.617</td>
    <td>0.584</td>
    <td>0.777</td>
  </tr>
  <tr>
    <th colspan="7">Chemical LLMs (fine-tuning)</th>
  </tr>
  <tr>
    <td>MolT5-small</td>
    <td>0.709</td>
    <td>0.512</td>
    <td>0.745</td>
    <td>0.558</td>
    <td>0.544</td>
    <td>0.701</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><span style="color: #4CAF50;">0.780</span></td>
    <td><span style="color: #4CAF50;">0.565</span></td>
    <td><span style="color: #4CAF50;">0.807</span></td>
    <td><span style="color: #4CAF50;">0.613</span></td>
    <td><span style="color: #4CAF50;">0.585</span></td>
    <td><span style="color: #4CAF50;">0.757</span></td>
  </tr>
  <tr>
    <td>MolT5-base</td>
    <td>0.738</td>
    <td>0.535</td>
    <td>0.750</td>
    <td>0.559</td>
    <td>0.539</td>
    <td>0.718</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><span style="color: #4CAF50;">0.805</span></td>
    <td><span style="color: #4CAF50;">0.592</span></td>
    <td><span style="color: #4CAF50;">0.864</span></td>
    <td><span style="color: #4CAF50;">0.677</span></td>
    <td><span style="color: #4CAF50;">0.642</span></td>
    <td><span style="color: #4CAF50;">0.822</span></td>
  </tr>
  <tr>
    <td>MolT5-large</td>
    <td>0.769</td>
    <td>0.556</td>
    <td>0.777</td>
    <td>0.580</td>
    <td>0.557</td>
    <td>0.743</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><span style="color: #4CAF50;">0.832</span></td>
    <td><span style="color: #4CAF50;">0.622</span></td>
    <td><span style="color: #4CAF50;">0.914</span></td>
    <td><span style="color: #4CAF50;">0.743</span></td>
    <td><span style="color: #4CAF50;">0.691</span></td>
    <td><span style="color: #4CAF50;">0.878</span></td>
  </tr>
</table>

Table 2: Molecule-to-text performance for L+M val. BL., RO., and ME. stand for BLEU, ROUGE, and ME-TEOR, respectively.

Reasoning module. We fine-tune the chemical LLMs to generate MSR similar to prior works that fine-tune LLMs to generate chain-of-thoughts (Ho et al., 2023; Fu et al., 2023; Magister et al., 2023). Unlike analytic reasoning, where external tools can precisely extract structural information, the reasoning module in synthetic reasoning must infer this information from the input.

Notably, we selectively retain only reliable structural components before incorporating them into the answering module. One considers the component to be reliable if it achieves sufficiently high reasoning accuracy across the entire dataset. This selection process also leverages the deterministic nature of structural information, allowing a quantitative evaluation of the reasoning module’s capability in generating each type of rationale, as presented in Section 4.3.

Answering module. Similar to the analytic reasoning scenario, we fine-tune chemical LLMs to generate an appropriate molecule from the input and its corresponding MSR. To further enhance structural consistency between the generated molecule and MSR, we propose a matching ratio-based rejection sampling method.

Specifically, the model first generates \( k \) candidate molecules using beam search. Then, for each candidate, one computes the matching ratio, which quantifies the consistency between the generated molecule’s structural components and those in MSR. Finally, the molecule with the highest matching ratio is selected as the final output, ensuring the consistency between the rationale and the generated answer.

Again, this approach leverages the deterministic nature of the molecular structural information, allowing us to easily obtain the information with external tools and compare them between the rationale and the generated molecule. Notably, the search process differs from prior works (Wang et al., 2023; Xi et al., 2023; Sun et al., 2024) that search over rationale-answer pairs since our method focuses on searching the answer that coincides with a given rationale.

4 Experiments

In this section, we present our experiments on two frameworks: analytic reasoning and synthetic reasoning. For the analytic reasoning framework, we consider molecule-to-text and retrosynthesis tasks. For the synthetic reasoning framework, we address
The molecular formula is C18H14N4O8S2-2. The longest carbon chain is 2 carbons long. The molecule contains 3 aromatic rings. It includes 3 rings of size 6. The functional groups present in the molecule include alcohol, amide, amine, aromatic, azo, organic, phenol, primary, and some others. This molecule has 0 chiral centers (0 with S configuration and 0 with R configuration).

Figure 5: An example of generated samples for molecule-to-text. We observe that MSR improves the accuracy of detailed molecular information (highlighted in yellow). We provide more examples in Appendix B.

<table>
  <tr>
    <th></th>
    <th>BL.2</th>
    <th>BL.4</th>
    <th>RO.1</th>
    <th>RO.2</th>
    <th>RO.L</th>
    <th>ME.</th>
  </tr>
  <tr>
    <th colspan="7">Baselines (without reasoning)</th>
  </tr>
  <tr>
    <td>T5-base</td>
    <td>0.511</td>
    <td>0.423</td>
    <td>0.607</td>
    <td>0.451</td>
    <td>0.550</td>
    <td>0.539</td>
  </tr>
  <tr>
    <td>MolXPT</td>
    <td>0.594</td>
    <td>0.505</td>
    <td>0.660</td>
    <td>0.511</td>
    <td>0.597</td>
    <td>0.626</td>
  </tr>
  <tr>
    <td>BioT5</td>
    <td>0.635</td>
    <td>0.556</td>
    <td>0.692</td>
    <td>0.559</td>
    <td>0.633</td>
    <td>0.656</td>
  </tr>
  <tr>
    <th colspan="7">Chemical LLMs (fine-tuning)</th>
  </tr>
  <tr>
    <td>MolT5-base</td>
    <td>0.540</td>
    <td>0.457</td>
    <td>0.634</td>
    <td>0.485</td>
    <td>0.578</td>
    <td>0.569</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.592</b></td>
    <td><b>0.507</b></td>
    <td><b>0.667</b></td>
    <td><b>0.523</b></td>
    <td><b>0.606</b></td>
    <td><b>0.619</b></td>
  </tr>
  <tr>
    <td>MolT5-large</td>
    <td>0.594</td>
    <td>0.508</td>
    <td>0.654</td>
    <td>0.510</td>
    <td>0.594</td>
    <td>0.614</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.645</b></td>
    <td><b>0.567</b></td>
    <td><b>0.699</b></td>
    <td><b>0.568</b></td>
    <td><b>0.639</b></td>
    <td><b>0.666</b></td>
  </tr>
  <tr>
    <td>ChemT5-small</td>
    <td>0.553</td>
    <td>0.462</td>
    <td>0.633</td>
    <td>0.481</td>
    <td>0.574</td>
    <td>0.583</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.601</b></td>
    <td><b>0.513</b></td>
    <td><b>0.664</b></td>
    <td><b>0.519</b></td>
    <td><b>0.603</b></td>
    <td><b>0.624</b></td>
  </tr>
  <tr>
    <td>ChemT5-base</td>
    <td>0.580</td>
    <td>0.490</td>
    <td>0.647</td>
    <td>0.498</td>
    <td>0.586</td>
    <td>0.604</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.639</b></td>
    <td><b>0.560</b></td>
    <td><b>0.687</b></td>
    <td><b>0.553</b></td>
    <td><b>0.626</b></td>
    <td><b>0.657</b></td>
  </tr>
  <tr>
    <th colspan="7">General LLMs (without fine-tuning)</th>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>0.211</td>
    <td>0.117</td>
    <td>0.367</td>
    <td>0.183</td>
    <td>0.308</td>
    <td>0.257</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.259</b></td>
    <td><b>0.158</b></td>
    <td><b>0.401</b></td>
    <td><b>0.208</b></td>
    <td><b>0.324</b></td>
    <td><b>0.341</b></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.232</td>
    <td>0.128</td>
    <td>0.389</td>
    <td>0.183</td>
    <td>0.307</td>
    <td>0.291</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.286</b></td>
    <td><b>0.174</b></td>
    <td><b>0.405</b></td>
    <td><b>0.199</b></td>
    <td><b>0.313</b></td>
    <td><b>0.341</b></td>
  </tr>
</table>

Table 3: Molecule-to-text performance for ChEBI-20.

<table>
  <tr>
    <th></th>
    <th>BL.2</th>
    <th>BL.4</th>
    <th>RO.1</th>
    <th>RO.2</th>
    <th>RO.L</th>
    <th>ME.</th>
  </tr>
  <tr>
    <th colspan="7">General LLM (without fine-tuning)</th>
  </tr>
  <tr>
    <td>Mol-Instruct.</td>
    <td>0.217</td>
    <td>0.143</td>
    <td>0.337</td>
    <td>0.196</td>
    <td>0.291</td>
    <td>0.254</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.347</b></td>
    <td><b>0.275</b></td>
    <td><b>0.601</b></td>
    <td><b>0.518</b></td>
    <td><b>0.593</b></td>
    <td><b>0.520</b></td>
  </tr>
</table>

Table 4: Molecule-to-text performance for Mol-Instructions.

the text-to-molecule task. For clarity, in all tables, the teal color indicates improvements over the vanilla model, and the best results are highlighted in bold.

4.1 Analytic reasoning: Molecule-to-text

The molecule-to-text task aims to generate a precise and informative textual description that accurately represents the given molecule.

The molecule is an organosulfonate oxoanion obtained by deprotonation of the sulfo groups of 7-amino-4-(4-acetamido-2-sulfonylphenyl)diazenyl)-6-amino-4-hydroxynaphthalene-2-sulfonic acid. It is a conjugate base of a p-acetamido-2-sulfobenzene-1-sulfonic acid.

The molecule is an organosulfonate oxoanion obtained by deprotonation of the sulfo groups of 5-[4-acetamido-2-sulfonylphenyl]diazenyl]-6-amino-4-hydroxynaphthalene-2-sulfonic acid. It is a conjugate base of a lissamine fast red (acid form).

The molecule is an organosulfonate oxoanion resulting from the removal of protons from both of the sulfo groups of 5-[4-acetamido-2-sulfonylphenyl]diazenyl]-6-amino-4-hydroxynaphthalene-2-sulfonic acid. It is a conjugate base of a lissamine fast red (acid form).

Figure 6: Fast performance improvement with MSR.

Dataset. We employ three datasets for the molecule-to-text task: (1) the recent L+M dataset (Edwards et al., 2024), (2) the widely used ChEBI-20 dataset (Edwards et al., 2021), and the (3) Mol-instructions dataset (Fang et al., 2024). Each dataset consists of 182,331, 33,010, and 298,319 pairs of SMILES (or SELFIES) and their text descriptions, respectively. We use the same splits used in prior works.

Baselines. We evaluate the performance of MSR with chemical and general LLMs. On the one hand, we employed two chemical LLMs: MolT5 (Edwards et al., 2022) and Text+Chem T5 (ChemT5; Christofidellis et al., 2023a). On the other hand, we employed three general LLMs: Llama3-8B-Instruct (Touvron et al., 2023), GPT-4o (OpenAI and et al., 2024)2, and Mol-Instructions (Fang et al., 2024). Additionally, we include T5 (Raffel et al., 2020), MolXPT (Liu et al., 2023), BioT5 (Pei et al., 2023), Meditron-7B (Chen et al., 2023b), Mol2Lang-VLM (Tran et al., 2024), and BioT5+ (Pei et al., 2024) as baselines to compare absolute performance.

Experimental setup and metrics. Chemical LLMs are trained following the process described in Section 3.2. For general LLMs without any domain-specific instruction tuning (Llama3 and GPT-4o), we cannot guarantee that the generated

2We used gpt-4o-2024-05-13.
<table>
  <tr>
    <th>Models</th>
    <th>BL.</th>
    <th>Ex.</th>
    <th>Le.↓</th>
    <th>MA.</th>
    <th>RDK</th>
    <th>Mo.</th>
    <th>Val.</th>
  </tr>
  <tr>
    <th colspan="8">General LLM (without fine-tuning)</th>
  </tr>
  <tr>
    <td>Mol-Instruct.</td>
    <td><b>0.705</b></td>
    <td>0.009</td>
    <td>31.23</td>
    <td>0.283</td>
    <td>0.487</td>
    <td>0.230</td>
    <td><b>1.000</b></td>
  </tr>
  <tr>
    <td>+ MSR</td>
    <td>0.502</td>
    <td><b>0.016</b></td>
    <td><b>31.21</b></td>
    <td><b>0.315</b></td>
    <td><b>0.493</b></td>
    <td><b>0.273</b></td>
    <td><b>1.000</b></td>
  </tr>
</table>

Table 5: Retrosynthesis performance for Mol-Instructions. BL., Ex., and Le. indicate BLEU, Exact, and Levenshtein distance. MA., RDK, and Mo. indicate MACCS, RDK, and Morgan fingerprint metrics. Val. indicates the validity.

descriptions align with our training data. Therefore, we apply 10-shot in-context learning by attaching MSR in the same manner as for chemical LLMs. Additionally, for Mol-Instructions, we prompt with instructions enriched with MSR. Note that we additionally consider the molecular weight and IUPAC name components used by M. Bran et al. (2024), as they slightly improved the performance.

We evaluate the performance by comparing the generated description with the ground truth using six metrics: BLEU2, BLEU4 (Papineni et al., 2002), ROUGE1, ROUGE2, ROUGEL (Banerjee and Lavie, 2005), and METEOR (Banerjee and Lavie, 2005). We provide detailed experimental settings and prompts in Appendix A.2.

Results. We report the results in Table 2, Table 3, and Table 4. We observe that adding MSR consistently improves performance across both chemical and general LLMs. Notably, in Table 3, ChemT5-base+MSR achieves performance comparable to BioT5 (without MSR), despite BioT5 being pretrained on a larger dataset. Furthermore, Table 2 shows that integrating MSR with MolT5-base or MolT5-large yields superior performance compared to baseline models. We provide examples of generated samples in Figure 5 and Figure 16. In addition, our method exhibits faster performance improvement, as illustrated in Figure 6.

4.2 Analytic reasoning: Retrosynthesis

The retrosynthesis task aims to generate the corresponding set of reactant molecular representations based on a given product molecular representation.

Dataset and baselines. We employ the dataset and the model used by Mol-instructions (Fang et al., 2024). The dataset consists of 129,684 product and reactant pairs.

Experimental setup and metrics. As the input molecule (i.e., product) is given for the retrosynthesis task, we follow the framework proposed in

<table>
  <tr>
    <th>Models</th>
    <th>Fo.</th>
    <th>Ch.</th>
    <th>Ar.</th>
    <th>Ri.</th>
    <th>Fu.</th>
    <th>Ch.</th>
    <th>We.</th>
    <th>Na.</th>
  </tr>
  <tr>
    <th colspan="9">Chemical LLMs (MSR fine-tuning) - L+M</th>
  </tr>
  <tr>
    <td>MolT5-small</td>
    <td>0.048</td>
    <td>0.235</td>
    <td>0.783</td>
    <td>0.781</td>
    <td>0.849</td>
    <td>0.647</td>
    <td>0.418</td>
    <td>0.248</td>
  </tr>
  <tr>
    <td>MolT5-base</td>
    <td>0.426</td>
    <td>0.527</td>
    <td>0.825</td>
    <td>0.813</td>
    <td>0.889</td>
    <td>0.807</td>
    <td>0.615</td>
    <td>0.309</td>
  </tr>
  <tr>
    <td>MolT5-large</td>
    <td>0.221</td>
    <td>0.317</td>
    <td>0.820</td>
    <td>0.809</td>
    <td>0.872</td>
    <td>0.691</td>
    <td>0.529</td>
    <td>0.576</td>
  </tr>
  <tr>
    <th colspan="9">Chemical LLMs (MSR fine-tuning) - MolT5</th>
  </tr>
  <tr>
    <td>MolT5-base</td>
    <td>0.458</td>
    <td>0.922</td>
    <td>0.926</td>
    <td>0.930</td>
    <td>0.957</td>
    <td>0.798</td>
    <td>0.606</td>
    <td>0.512</td>
  </tr>
  <tr>
    <td>ChemT5-small</td>
    <td>0.447</td>
    <td>0.920</td>
    <td>0.930</td>
    <td>0.926</td>
    <td>0.954</td>
    <td>0.788</td>
    <td>0.634</td>
    <td>0.495</td>
  </tr>
  <tr>
    <td>ChemT5-base</td>
    <td>0.475</td>
    <td>0.925</td>
    <td>0.931</td>
    <td>0.930</td>
    <td>0.960</td>
    <td>0.799</td>
    <td>0.641</td>
    <td>0.525</td>
  </tr>
  <tr>
    <th colspan="9">General LLMs (MSR few-shot learning) - MolT5</th>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>0.084</td>
    <td>0.174</td>
    <td>0.593</td>
    <td>0.362</td>
    <td>0.137</td>
    <td>0.450</td>
    <td>0.435</td>
    <td>0.015</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.298</td>
    <td>0.235</td>
    <td>0.718</td>
    <td>0.464</td>
    <td>0.298</td>
    <td>0.485</td>
    <td>0.728</td>
    <td>0.040</td>
  </tr>
</table>

Table 6: Reasoning accuracy for each structural information. Fo., Ch., Ar., Ri., Fu., Ch., We., Na., stand for molecular formula, longest carbon chain length, aromatic rings, ring compounds, functional groups, chirality, molecular weight, and IUPAC name, respectively.

Section 3.2. The performance is evaluated by comparing the generated molecules with the ground truth with eight metrics: SMILES comparison metrics (BLEU, Exact, and Levenshtein distance (Miller et al., 2009)), fingerprint similarity metrics (MACCS FTS (Durant et al., 2002), RDK FTS (Schneider et al., 2015), and Morgan FTS (Rogers and Hahn, 2010)), a molecular distribution metric (Fréchet ChemNet Distance (FCD) (Preuer et al., 2018)), and the validity of the molecule.

Results. We report the results in Table 5, showing that incorporating MSR improves performance across all metrics except BLEU. This highlights its effectiveness in enhancing complex tasks. Notably, while we report BLEU for consistency with prior work, it is less critical than other metrics, as it evaluates string-based accuracy rather than molecular structure alignment.

4.3 Synthetic reasoning: Text-to-molecule

The text-to-molecule task is the inverse of molecule-to-text, aiming to generate a molecular representation based on a given textual description.

Dataset. We employ two datasets for the text-to-molecule task: (1) L+M (Edwards et al., 2024) and (2) ChEBI-20 (Edwards et al., 2021). We followed the same settings used in Section 4.1.

Baselines. Two popular chemical LLMs, including MolT5 (Edwards et al., 2022) and ChemT5 (Christofidellis et al., 2023a), serve as our baselines. Notably, we exclude general LLMs
<table>
  <tr>
    <th>BL.</th>
    <th>Ex.</th>
    <th>Le.</th>
    <th>MA.</th>
    <th>RDK</th>
    <th>Mo.</th>
    <th>FCD.</th>
    <th>Val.</th>
  </tr>
  <tr>
    <th colspan="8">Baselines (without reasoning)</th>
  </tr>
  <tr>
    <td>Meditron-7B</td>
    <td>0.694</td>
    <td>0.010</td>
    <td>46.49</td>
    <td>0.772</td>
    <td>0.693</td>
    <td>0.501</td>
    <td>2.46</td>
    <td>0.996</td>
  </tr>
  <tr>
    <td>Lang2Mol-Diff</td>
    <td>0.543</td>
    <td>0.000</td>
    <td>55.87</td>
    <td>0.606</td>
    <td>0.332</td>
    <td>0.328</td>
    <td>38.09</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>BioTS</td>
    <td><b>0.731</b></td>
    <td>0.010</td>
    <td>41.47</td>
    <td>0.781</td>
    <td>0.709</td>
    <td>0.515</td>
    <td>3.29</td>
    <td>1.000</td>
  </tr>
  <tr>
    <th colspan="8">Chemical LLMs (fine-tuning)</th>
  </tr>
  <tr>
    <td>MolTS-small</td>
    <td>0.566</td>
    <td>0.000</td>
    <td>56.34</td>
    <td>0.642</td>
    <td>0.581</td>
    <td>0.374</td>
    <td>NaN</td>
    <td>0.805</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.730</b></td>
    <td>0.002</td>
    <td>41.15</td>
    <td>0.798</td>
    <td>0.712</td>
    <td>0.514</td>
    <td>2.82</td>
    <td>0.995</td>
  </tr>
  <tr>
    <td>MolTS-base</td>
    <td>0.684</td>
    <td>0.000</td>
    <td>44.79</td>
    <td>0.760</td>
    <td>0.652</td>
    <td>0.475</td>
    <td>NaN</td>
    <td>1.000</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.706</b></td>
    <td>0.052</td>
    <td>40.18</td>
    <td>0.825</td>
    <td>0.762</td>
    <td>0.548</td>
    <td><b>1.45</b></td>
    <td>0.997</td>
  </tr>
  <tr>
    <td>MolTS-large</td>
    <td>0.564</td>
    <td>0.000</td>
    <td>55.40</td>
    <td>0.757</td>
    <td>0.650</td>
    <td>0.395</td>
    <td>17.50</td>
    <td>0.994</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.710</b></td>
    <td><b>0.111</b></td>
    <td><b>39.54</b></td>
    <td><b>0.837</b></td>
    <td><b>0.783</b></td>
    <td><b>0.560</b></td>
    <td><b>1.54</b></td>
    <td><b>0.999</b></td>
  </tr>
</table>

Table 7: Text-to-molecule performance for L+M val.

from this evaluation due to their insufficient reasoning accuracy as shown in Table 6. In detail, their low accuracy implies that their reasoning cannot guide the answer appropriately, even in a few-shot learning setting. For completeness, we provide the results for general LLMs in Appendix B.3. Additional baselines are consistent with those in Section 4.1 other than Lang2Mol-Diff (Nguyen et al., 2024).

Experimental setup and metrics. We follow the framework proposed in Section 3.3. We provide detailed experimental settings and prompts in Appendix A.3. The performance is evaluated using the same metrics described in Section 4.2.

Reasoning accuracy. We first measure the reasoning accuracy to filter out low-accuracy components that may misguide the answer. The detailed computation process is in Appendix A.3. The reasoning accuracies are provided in Table 6. Our results show that our fine-tuned reasoning modules exhibit superior accuracy compared to larger general LLMs, underscoring their ability to understand molecular structures effectively. However, they still struggle with certain structural elements, such as molecular formula, molecular weight, and IUPAC name, with additional challenges in carbon chain length and chirality in the L+M dataset. Consequently, we exclude these components.

Results. The results are reported in Table 7 and Table 8. Incorporating MSR into the molecular description always improved performance. In particular, integrating MSR into the ChemT5-base achieves state-of-the-art performance compared to the recent baselines, validating its efficacy. Surprisingly, our MSR even improves the performance of smaller models beyond that of the vanilla larger models, e.g., MolTS-base+MSR showed superior performance to MolTS-large. We provide examples of generated samples in Appendix B.1.

<table>
  <tr>
    <th>BL.</th>
    <th>Ex.</th>
    <th>Le.</th>
    <th>MA.</th>
    <th>RDK</th>
    <th>Mo.</th>
    <th>FCD.</th>
    <th>Val.</th>
  </tr>
  <tr>
    <th colspan="8">Baselines (without reasoning)</th>
  </tr>
  <tr>
    <td>T3-base</td>
    <td>0.762</td>
    <td>0.069</td>
    <td>24.95</td>
    <td>0.731</td>
    <td>0.605</td>
    <td>0.545</td>
    <td>2.48</td>
    <td>0.660</td>
  </tr>
  <tr>
    <td>MolXPT</td>
    <td>-</td>
    <td>0.215</td>
    <td>-</td>
    <td>0.859</td>
    <td>0.757</td>
    <td>0.667</td>
    <td>0.45</td>
    <td>0.983</td>
  </tr>
  <tr>
    <td>BioTS</td>
    <td>0.867</td>
    <td>0.413</td>
    <td>15.10</td>
    <td>0.886</td>
    <td>0.801</td>
    <td>0.734</td>
    <td>0.43</td>
    <td>1.000</td>
  </tr>
  <tr>
    <th colspan="8">Chemical LLMs (fine-tuning)</th>
  </tr>
  <tr>
    <td>MolTS-base</td>
    <td>0.769</td>
    <td>0.081</td>
    <td>24.46</td>
    <td>0.721</td>
    <td>0.588</td>
    <td>0.529</td>
    <td>2.18</td>
    <td>0.772</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.863</b></td>
    <td><b>0.385</b></td>
    <td><b>13.91</b></td>
    <td><b>0.918</b></td>
    <td><b>0.843</b></td>
    <td><b>0.783</b></td>
    <td><b>0.29</b></td>
    <td><b>0.983</b></td>
  </tr>
  <tr>
    <td>MotTS-large</td>
    <td>0.854</td>
    <td>0.311</td>
    <td>16.07</td>
    <td>0.834</td>
    <td>0.746</td>
    <td>0.684</td>
    <td>1.20</td>
    <td>0.905</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.886</b></td>
    <td><b>0.391</b></td>
    <td><b>12.98</b></td>
    <td><b>0.906</b></td>
    <td><b>0.822</b></td>
    <td><b>0.765</b></td>
    <td><b>0.35</b></td>
    <td><b>0.947</b></td>
  </tr>
  <tr>
    <td>ChemT5-small</td>
    <td>0.739</td>
    <td>0.157</td>
    <td>28.54</td>
    <td>0.859</td>
    <td>0.736</td>
    <td>0.660</td>
    <td>0.07</td>
    <td>0.776</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.874</b></td>
    <td><b>0.381</b></td>
    <td><b>13.22</b></td>
    <td><b>0.918</b></td>
    <td><b>0.845</b></td>
    <td><b>0.787</b></td>
    <td><b>0.29</b></td>
    <td><b>0.976</b></td>
  </tr>
  <tr>
    <td>ChemT5-base</td>
    <td>0.750</td>
    <td>0.212</td>
    <td>27.39</td>
    <td>0.874</td>
    <td>0.767</td>
    <td>0.697</td>
    <td><b>0.06</b></td>
    <td>0.792</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td><b>0.878</b></td>
    <td><b>0.421</b></td>
    <td><b>12.76</b></td>
    <td><b>0.924</b></td>
    <td><b>0.856</b></td>
    <td><b>0.804</b></td>
    <td><b>0.26</b></td>
    <td><b>0.982</b></td>
  </tr>
</table>

Table 8: Text-to-molecule performance for ChEBI-20.

![Impact of k in rejection sampling. Dotted lines indicate the initial performance of k = 5.](page_1012_1342_420_246.png)

Figure 7: Impact of k in rejection sampling. Dotted lines indicate the initial performance of \( k = 5 \).

4.4 Ablation study

We perform ablation studies on matching ratio-based rejection sampling and each structural component. Here, we utilize ChemT5-small on the ChEBI-20 dataset. Due to limited space, additional ablation study results, including a comparison with ChemCrow (M. Bran et al., 2024), prior work on the reasoning for chemistry tasks, and extra structural component, are provided in Appendix B.4.

Matching ratio-based rejection sampling. We discuss the efficacy of matching ratio-based rejection sampling and the impact of the number of samples \( k \) in text-to-molecule. We compare the results of without (\( k = 1 \)) and with the rejection sampling (\( k \in \{2, 5\} \)). As demonstrated in Figure 7, the rejection sampling improves performance by encouraging the output to follow the MSR. Notably, increasing \( k \) beyond 5 does not further improve performance, implying that \( k = 5 \) is sufficient.

Structural component. To verify the effectiveness of each component, we evaluated the performance of molecule-to-text using each structural information component individually. We provide the results in Figure 8. Incorporating each single component resulted in better performance compared to the baseline model without any reasoning. Notably, combining all the proposed structural elements yielded the best results, validating the effectiveness of our comprehensive approach.
Figure 8: Impact of each structural component.

Additional molecular descriptors. In addition to the proposed six structural components, we conducted experiments using three more advanced molecular descriptors: the Morgan fingerprint and two electronic properties—topological polar surface area (TPSA) and molar refractivity (MR). Specifically, the Morgan fingerprint encodes local substructures within a specified radius; TPSA represents the sum of the surface areas of all polar atoms and their attached hydrogen atoms; and MR quantifies the total polarizability of a molecule.

To verify the effectiveness of each additional descriptor, we evaluated the performance of molecule captioning using ChemT5-small. We provide the results in Figure 9. We observed that incorporating all three additional descriptors together did not further improve the performance of MSR, although applying each additional descriptor individually improved performance. This validates the importance of structural information and the sufficiency of our proposed structural components.

Figure 9: The impact of additional molecular descriptors.

To address this challenge, various approaches have been introduced to distill knowledge from larger language models to smaller ones (<10B). Specifically, Ho et al. (2023); Fu et al. (2023); Magister et al. (2023) employed the larger models as teacher models to generate rationales for fine-tuning smaller student models. Nevertheless, even recent LLMs struggle to generate appropriate rationales that demonstrate a correct understanding of molecular structures (as described in Figure 1a and Section 2), restricting the efficacy of LLMs in generating rationales for molecular tasks.

Reasoning for chemistry. Recently, a few works have extended the reasoning of LLMs to address chemistry problems. For instance, Ouyang et al. (2024) proposed employing the program-of-thoughts (PoT; Chen et al., 2023a) to handle chemical question-answering tasks. Additionally, Jin et al. (2024) presented the protein chain-of-thought (ProCoT) to replicate the signaling pathways in the protein-protein interaction (PPI) problem. Despite these advances, none of these works are generally applicable to various molecular tasks. We note that M. Bran et al. (2024) provided a reasoning approach comparable to ours, but their rationales are less focused on molecular structures, e.g., rationales based on tools like LitSearch/WebSearch, PatentCheck, ReactionPlanner, and SMILES2Price. Moreover, it shows limited performance improvement in molecule generation and molecule captioning tasks, as observed in Appendix B.4.

5 Related work

Large language models for chemistry. General LLMs often struggle to solve basic chemistry problems and molecular tasks (White et al., 2023; Castro Nascimento and Pimentel, 2023; Guo et al., 2023). To address this issue, prior works have introduced chemical LLMs by pre-training models on molecule-related texts (Edwards et al., 2022; Christofidellis et al., 2023b; Liu et al., 2023; Pei et al., 2023), through instruction tuning (Fang et al., 2024; Cao et al., 2023), and using retrieval-based in-context learning (Li et al., 2024). Our work focuses on reasoning processes that are broadly applicable to these chemical and general LLMs.

Reasoning of LLMs. Generating intermediate reasoning before arriving at a final answer (Wei et al., 2022; Kojima et al., 2022) improves the overall quality of generated answers. However, the ability to perform complex reasoning remains limited to huge models (>100B parameters).

6 Conclusion

We introduced MSR, a molecular structural reasoning framework that enhances LLMs’ understanding of molecules by explicitly incorporating key structural features. Our investigation revealed recent LLMs’ limitations in inferring structural information, emphasizing the need for explicit reasoning. Fine-tuning chemical LLMs with MSR led to consistent improvements across three molecular tasks, highlighting the effectiveness of domain-specific models for molecular reasoning.
Broader impacts

Our work contributes to the development of more interpretable and reliable models for molecular applications. By incorporating explicit molecular reasoning, our framework has the potential to enhance molecular understanding and improve decision-making in areas such as drug discovery, materials science, and chemical synthesis. However, as with any AI-driven molecular generation system, there are potential risks and ethical concerns. For instance, the generation of harmful or toxic compounds poses significant safety challenges. Additionally, over-reliance on AI-generated molecular reasoning without expert validation could lead to unintended consequences in scientific and industrial applications.

Limitations

One limitation of MSR is its reliance on the accuracy of structural information in synthetic reasoning. While external tools like RDKit provide precise structural information for molecule-forward reasoning, errors in molecule-backward reasoning (where structural features must be inferred) could degrade performance. However, appropriate filtering based on reasoning accuracy can prevent this to some extent. Additionally, we assume that the given molecular representations are accurate when we extract the structural information. However, real-world data can be noisy or incomplete. Extending MSR to handle uncertain molecular inputs via self-correction remains an open challenge.

Reproducibility

All experimental code related to this paper is available at https://github.com/yunhuijiang/MSR. Detailed insights regarding the experiments, encompassing dataset and model specifics, are available in Section 4. For intricate details like hyperparameters, consult Appendix A.

Acknowledgement

This work was partly supported by Institute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (RS-2019- II190075, Artificial Intelligence Graduate School Support Program(KAIST)), National Research Foundation of Korea(NRF) grant funded by the Ministry of Science and ICT(MSIT) (No. RS-2022- NR072184), GRDC(Global Research Development Center) Cooperative Hub Program through the National Research Foundation of Korea(NRF) grant funded by the Ministry of Science and ICT(MSIT) (No. RS-2024-00436165), and the Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (RS-2025-02304967, AI Star Fellowship(KAIST)).
References

Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.

He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. 2023. Instructmol: Multi-modal integration for building a versatile and reliable molecular assistant in drug discovery. Preprint, arXiv:2311.16208.

Cayque Monteiro Castro Nascimento and AndreSilva Pimentel. 2023. Do large language models understand chemistry? a conversation with chatgpt. Journal of Chemical Information and Modeling, 63(6):1649–1655.

Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023a. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research.

Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. 2023b. Meditron-70b: Scaling medical pretraining for large language models. Preprint, arXiv:2311.16079.

Dimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, and Matteo Manica. 2023a. Unifying molecular and textual representations via multi-task language modelling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 6140–6157. PMLR.

Dimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, and Matteo Manica. 2023b. Unifying molecular and textual representations via multi-task language modelling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 6140–6157. PMLR.

Joseph L Durant, Burton A Leland, Douglas R Henry, and James G Nourse. 2002. Reoptimization of mdI keys for use in drug discovery. Journal of chemical information and computer sciences, 42(6):1273–1280.

Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. 2022. Translation between molecules and natural language. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 375–413, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Carl Edwards, Qingyun Wang, Lawrence Zhao, and Heng Ji. 2024. L+M-24: Building a dataset for Language+Molecules @ ACL 2024. In Proceedings of the 1st Workshop on Language + Molecules (L+M 2024), pages 1–9, Bangkok, Thailand. Association for Computational Linguistics.

Carl Edwards, ChengXiang Zhai, and Heng Ji. 2021. Text2Mol: Cross-modal molecule retrieval with natural language queries. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 595–607, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Hua-jun Chen. 2024. Mol-Instructions: A large-scale biomolecular instruction dataset for large language models. In The Twelfth International Conference on Learning Representations.

Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 10421–10430. PMLR.

Veronika Ganeeva, Andrey Sakhovskiy, Kuzma Khrabrov, Andrey Savchenko, Artur Kadurin, and Elena Tutubalina. 2024. Lost in translation: Chemical language models and the misunderstanding of molecule structures. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12994–13013.

Taicheng Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh Chawla, Olaf Wiest, Xiangliang Zhang, et al. 2023. What can large language models do in chemistry? a comprehensive benchmark on eight tasks. Advances in Neural Information Processing Systems, 36:59662–59688.

Corwin Hansch, Susan Mckarns, Carr Smith, and David Doolittle. 2000. Comparative qsar evidence for a free-radical mechanism of phenol-induced toxicity. Chemico-Biological Interactions, 127:61–72.

Namgyu Ho, Laura Schmid, and Se-Young Yun. 2023. Large language models are reasoning teachers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14852–14882, Toronto, Canada. Association for Computational Linguistics.

Mingyu Jin, Haochen Xue, Zhenting Wang, Boming Kang, Ruosong Ye, Kaixiong Zhou, Mengnan Du, and Yongfeng Zhang. 2024. ProLLM: Protein chain-of-thoughts enhanced LLM for protein-protein interaction prediction. In First Conference on Language Modeling.
I. Kant. 1899. Critique of pure reason.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems.

Greg Landrum, Paolo Tosco, Brian Kelley, Ricardo Rodriguez, David Cosgrove, Riccardo Vianello, sriniker, Peter Gedeck, Gareth Jones, NadineSchneider, Eisuke Kawashima, Dan Nealschneider, Andrew Dalke, Matt Swain, Brian Cole, Samo Turk, Aleksandr Savelev, Alain Vaucher, Maciej Wójcikowski, Ichiru Take, Vincent F. Scalfani, Rachel Walker, Daniel Probst, Kazuya Ujihara, Axel Pahl, guillaume godin, Juuso Lehtivarjo, radthurst cdd, François Bérenger, and Jonathan Bisson. 2024. rdkit/rdkit: 2024_09_1 (q3 2024) release beta.

Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao-Yong Wei, Hui Liu, Jiliang Tang, and Qing Li. 2024. Empowering molecule discovery for molecule-caption translation with large language models: A chatgpt perspective. IEEE Transactions on Knowledge and Data Engineering, page 1–13.

Zequn Liu, Wei Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Zhang, and Tie-Yan Liu. 2023. MolXPT: Wrapping molecules with text for generative pre-training. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1606–1616, Toronto, Canada. Association for Computational Linguistics.

Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D. White, and Philippe Schwaller. 2024. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525–535.

Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2023. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773–1781, Toronto, Canada. Association for Computational Linguistics.

Frederic P Miller, Agnes F Vandome, and John McBrewster. 2009. Levenshtein distance: Information theory, computer science, string (computer science), string metric, damerau? levenshtein distance, spell checker, hamming distance.

Nguyen Doan Hieu Nguyen, Nhat Truong Pham, Duong Thanh Tran, and Balachandran Manavalan. 2024. Lang2mol-diff: A diffusion-based generative model for language-to-molecule translation leveraging SELFIES representation. In ACL 2024 Workshop Language + Molecules.

OpenAI and Josh Achiam et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Siru Ouyang, Zhuosheng Zhang, Bing Yan, Xuan Liu, Yejin Choi, Jiawei Han, and Lianhui Qin. 2024. Structured chemistry reasoning with large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.

Qizhi Pei, Lijun Wu, Kaiyuan Gao, Jinhua Zhu, and Rui Yan. 2024. Enhanced biot5+ for molecule-text translation: A three-stage approach with data distillation, diverse training, and voting ensemble. In ACL 2024 Workshop Language + Molecules.

Qizhi Pei, Wei Zhang, Jinhua Zhu, Kehan Wu, Kaiyuan Gao, Lijun Wu, Yingce Xia, and Rui Yan. 2023. BioT5: Enriching cross-modal integration in biology with chemical knowledge and natural language associations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1102–1123, Singapore. Association for Computational Linguistics.

Kristina Preuer, Philipp Renz, Thomas Unterthiner, Sepp Hochreiter, and Günter Klambauer. 2018. Fréchet chemnet distance: A metric for generative models for molecules in drug discovery. Journal of Chemical Information and Modeling, 58(9):1736–1741.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.

David Rogers and Mathew Hahn. 2010. Extended-connectivity fingerprints. Journal of chemical information and modeling, 50(5):742–754.

Nadine Schneider, Roger A Sayle, and Gregory A Landrum. 2015. Get your atoms in order - an open-source implementation of a novel and robust molecular canonicalization algorithm. Journal of chemical information and modeling, 55(10):2111–2120.

Jiahuo Sun, Yi Luo, Yeyun Gong, Chen Lin, Yelong Shen, Jian Guo, and Nan Duan. 2024. Enhancing chain-of-thoughts prompting with iterative bootstrapping in large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4074–4101, Mexico City, Mexico. Association for Computational Linguistics.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.
Duong Thanh Tran, Nhat Truong Pham, Nguyen Doan Hieu Nguyen, and Balachandran Manavalan. 2024. Mol2lang-VLM: Vision- and text-guided generative pre-trained language models for advancing molecule captioning through multimodal fusion. In ACL 2024 Workshop Language + Molecules.

Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.

David Weininger. 1988. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 28(1):31–36.

Andrew D. White, Glen M. Hocky, Heta A. Gandhi, Mehrad Ansari, Sam Cox, Geemi P. Wellawatte, Subarna Sasmal, Ziyue Yang, Kangxin Liu, Yuvraj Singh, and Willmor J. Peña Ccoa. 2023. Assessment of chemistry knowledge in large language models that generate code. Digital Discovery, 2:368–376.

Zhiheng Xi, Senjie Jin, Yuhao Zhou, Rui Zheng, Songyang Gao, Jia Liu, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Self-Polish: Enhance reasoning in large language models via problem refinement. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11383–11406, Singapore. Association for Computational Linguistics.

Zhuosheng Zhang, Aston Zhang, Mu Li, hai zhao, George Karypis, and Alex Smola. 2024. Multimodal chain-of-thought reasoning in language models. Transactions on Machine Learning Research.
Appendix

Organization  The appendix is organized as follows: We first present the experimental details such as hyperparameters and prompts in Appendix A. Then we provide the additional experimental results including the generated samples and additional ablation studies in Appendix B. Next, we described the usage of AI assistants and scientific artifacts in Appendix C and Appendix D, respectively.

A  Experimental details

In this section, we provide the details of the experiments. All experimental code related to this paper is available at https://github.com/yunhuijiang/MSR and our experiments are based on a single run. Additionally, we used the packages including rouge-score==0.1.2 and nltk==3.8.1.

A.1 Structure information analysis

Here, we describe the detailed settings for the analysis in Section 2. To evaluate the understanding of two recent LLMs: Llama3-8B-Instruct (Touvron et al., 2023) and GPT-4o (OpenAI and et al., 2024), we prompt the LLMs to infer the structural information from the given molecular SMILES string and text description of the molecule.

Prompts given text description of molecules. First, we asked LLMs to infer the structural information from the text description of the molecule, with the prompt described in Figure 10.

Prompts given SMILES string. Next, we asked LLMs to infer the structural information from the SMILES string, with the prompt described in Figure 11.

<table>
  <tr>
    <th>Prompts for M2S</th>
  </tr>
  <tr>
    <td><b>Head prompt:</b> You are now working as an excellent expert in chemistry and drug discovery.<br>Given the SMILES representation of a molecule, your job is to predict the structural information of the molecule.<br>The structural information of the molecule caption includes the molecular formula, the length of the longest carbon chain, the number of aromatic rings, the IUPAC name of all the rings, all the functional groups, the number of chiral centers with S and R configurations each, the molecular weight, the IUPAC name of the molecule.<br>The functional group and ring IUPAC names should be on the list. The number of chiral centers should also be format {"S": , "R": }.<br>Your response should only be in the JSON format following {"molecular formula": , "functional group": , "longest carbon chain length": , "aromatic ring": , "ring IUPAC name": , "chiral": {"S": , "R": }, "weight": , "IUPAC name": }.<br>THERE SHOULD BE NO OTHER CONTENT INCLUDED IN YOUR RESPONSE. DO NOT CHANGE THE JSON KEY NAMES.</td>
  </tr>
  <tr>
    <td><b>Input prompt:</b> Input: [SMILES]</td>
  </tr>
</table>

Figure 10: Prompts for structure information analysis given SMILES string.
Prompts for T2S

Head prompt: You are now working as an excellent expert in chemistry and drug discovery.
Given the caption of a molecule, your job is to predict the structural information of the molecule.

The molecule caption is a sentence that describes the molecule, which mainly describes the molecules structures, properties, and production.
The structural information of the molecule caption includes the molecular formula, the length of the longest carbon chain, the number of aromatic rings, the IUPAC name of all the rings, all the functional groups, the number of chiral centers with S and R configurations each, the molecular weight, the IUPAC name of the molecule.
The functional group and ring IUPAC names should be on the list. The number of chiral centers should also be format ["S": , "R": ].
Your response should only be in the JSON format following {"molecular formula": , "functional group": , "longest carbon chain length": , "aromatic ring": , "ring IUPAC name": , "chiral": ["S": , "R": ], "weight": , "IUPAC name": }.
THERE SHOULD BE NO OTHER CONTENT INCLUDED IN YOUR RESPONSE. DO NOT CHANGE THE JSON KEY NAMES.

Input prompt: Input: [Description]

Figure 11: Prompts for structure information analysis given text description.

A.2 Molecule-to-text

Here, we describe the detailed settings for the experiments of molecule-to-text in Section 4.1. Note that we used four A100-80GB GPUs.

Hyperparameters. The hyperparameters for the specialist models are provided in Table 9. Note that MoIT5-large was not trained for the same epochs as the other models due to limited resource.

Prompts. The prompts used for the generalist models are described in Figure 12. We primarily followed the prompt presented by (Li et al., 2024).

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>MoIT5-base</th>
    <th>MoIT5-large</th>
    <th>ChemT5-small</th>
    <th>ChemT5-base</th>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>8</td>
    <td>4</td>
    <td>8</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>2e^{-4}</td>
    <td>2e^{-4}</td>
    <td>6e^{-4}</td>
    <td>6e^{-4}</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>250</td>
    <td>220</td>
    <td>250</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Warmup ratio</td>
    <td>0</td>
    <td>0</td>
    <td>0.1</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>Weight decay</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Lr scheduler</td>
    <td>linear</td>
    <td>linear</td>
    <td>linear</td>
    <td>linear</td>
  </tr>
</table>

Table 9: Hyperparameters for molecule captioning.

A.3 Text-to-molecule

Here, we described the detailed settings for the experiments of text-to-molecule in Section 3.3. Note that we also used four A100-80GB GPUs.

Hyperparameters. The hyperparameters for the reasoning and answering module for the specialist models are provided in Table 10 and Table 11, respectively. Note that MoIT5-large was not trained for the same number of epochs as the other models due to limited time constraints.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>MoIT5-base</th>
    <th>ChemT5-small</th>
    <th>ChemT5-base</th>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>8</td>
    <td>8</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>1e^{-3}</td>
    <td>6e^{-4}</td>
    <td>6e^{-4}</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>250</td>
    <td>250</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Warmup ratio</td>
    <td>0.1</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Weight decay</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Lr scheduler</td>
    <td>cosine</td>
    <td>linear</td>
    <td>linear</td>
  </tr>
</table>

Table 10: Hyperparameters for the reasoning module of text-based molecule generation.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>MoIT5-base</th>
    <th>MoIT5-large</th>
    <th>ChemT5-small</th>
    <th>ChemT5-base</th>
  </tr>
  <tr>
    <td>Batch size</td>
    <td>8</td>
    <td>4</td>
    <td>8</td>
    <td>8</td>
  </tr>
  <tr>
    <td>Learning rate</td>
    <td>1e^{-3}</td>
    <td>1e^{-3}</td>
    <td>6e^{-4}</td>
    <td>6e^{-4}</td>
  </tr>
  <tr>
    <td>Epochs</td>
    <td>250</td>
    <td>140</td>
    <td>250</td>
    <td>250</td>
  </tr>
  <tr>
    <td>Warmup ratio</td>
    <td>0.1</td>
    <td>0.1</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Weight decay</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Lr scheduler</td>
    <td>cosine</td>
    <td>cosine</td>
    <td>linear</td>
    <td>linear</td>
  </tr>
</table>

Table 11: Hyperparameters for the answering module of text-based molecule generation.

Reasoning accuracy The accuracies for molecular formula, longest carbon chain length, number of aromatic rings, chirality, and IUPAC names are computed by exact match. The accuracies for ring compounds and functional groups are computed by the ratio of intersection between the set of true and generated CoTs. Lastly, the accuracy for molecular weight is considered correct if the generated weight is within 95% to 105% of the true weight.

Prompts. The prompts used for the generalist models are described in Figure 13. We also primarily followed the prompt presented by (Li et al., 2024).
Prompts for molecule2text

Head prompt: You are now working as an excellent expert in chemistry and drug discovery.
Given the caption of a molecule, your job is to predict the SMILES representation of the molecule.
The molecule caption is a sentence that describes the molecule, which mainly describes the molecule’s structures, properties, and production.
You can infer the molecule SMILES representation from the caption.
Before you infer the molecule SMILES representation, YOU SHOULD FIRST GENERATE the molecular formula, the length of the longest carbon chain, the number of aromatic rings, the IUPAC name of all the rings, all the functional groups, the number of chiral centers with S and R configurations each, the molecular weight, the IUPAC name of the molecule.

Example 1: Instruction: Given the caption of a molecule, predict the SMILES representation of the molecule.
Input: [Description][MSR]
Your output should be: {"molecule": <SMILES>}
…
Example k: Instruction: Given the caption of a molecule, predict the SMILES representation of the molecule.
Input: [Description][MSR]
Your output should be: {"molecule": <SMILES>}

You should FIRST generate the structural information following the examples above, and then provide the JSON format of the molecule SMILES based on that.
NOTE THAT THE SMILES REPRESENTATION MUST BE IN THE JSON format above ("molecule": ). THERE SHOULD BE NO OTHER CONTENT INCLUDED IN YOUR JSON. DO NOT CHANGE THE JSON KEY NAME.

Input prompt: Input: [Description]

Prompts for text2molecule

Head prompt: You are now working as an excellent expert in chemistry and drug discovery.
Given the SMILES representation of a molecule and structural description of the molecule, your job is to predict the caption of the molecule.
The molecule caption is a sentence that describes the molecule, which mainly describes the molecule’s structures, properties, and production.

Example 1:
Instruction: Given the SMILES representation of a molecule, predict the caption of the molecule.
Input: [SMILES][MSR]
Your output should be: {"caption": <Description>}
…
Example k:
Instruction: Given the SMILES representation of a molecule, predict the caption of the molecule.
Input: [SMILES][MSR]
Your output should be: {"caption": <Description>}

Your response should only be in the JSON format above; THERE SHOULD BE NO OTHER CONTENT INCLUDED IN YOUR RESPONSE.

Input prompt: Input: [SMILES]<MSR >

Figure 12: Prompts for the generalist models in molecule captioning task.

Figure 13: Prompts for generalist models in text-based molecule generation task.
A.4  Ablation study
Here, we describe the detailed settings for the ablation study.

Prompts for ChemCrow. The prompts used for ChemCrow (M. Bran et al., 2024) are described in Figure 14 and Figure 15. Notably, it was not able to apply few-shot learning for ChemCrow as it was not applicable as the original prompt proposed in ChemCrow does not include any few-shot setting.

<table>
  <tr>
    <th colspan="2">Prompts for molecule2text with ChemCrow</th>
  </tr>
  <tr>
    <td><b>Head prompt:</b></td>
    <td>Given the SMILES representation of a molecule and structural description of the molecule, your job is to predict the caption of the molecule.<br>"Final Answer" follows the format: Final Answer: {"caption": }</td>
  </tr>
  <tr>
    <td><b>Input prompt:</b></td>
    <td>The SMILES representation of the molecule is as follows: : [SMILES]</td>
  </tr>
</table>

Figure 14: Prompts for molecule captioning with ChemCrow.

<table>
  <tr>
    <th colspan="2">Prompts for text2mol with ChemCrow</th>
  </tr>
  <tr>
    <td><b>Head prompt:</b></td>
    <td>Given the caption of a molecule, your job is to predict the SMILES representation of the molecule.<br>The molecule caption is a sentence that describes the molecule, which mainly describes the molecule’s structures, properties, and production.<br>You can infer the molecule SMILES representation from the caption.<br>"Final Answer" follows the format: Final Answer: {"molecule": }</td>
  </tr>
  <tr>
    <td><b>Input prompt:</b></td>
    <td>The caption is as follows: [Description]</td>
  </tr>
</table>

Figure 15: Prompts for text-based molecule generation with ChemCrow.
B Additional experimental results

In this section, we provide additional experimental results including several concrete examples of generated samples.

B.1 Molecule-to-text

Here, we show the samples of molecule captioning, i.e., generated text descriptions of given molecules in Figure 16. Notably, we show the generated samples from base-sized models for fair comparison.

B.2 Retrosynthesis

Here, we show the samples of retrosynthesis, i.e., generated reactants of given product in Figure 17.

B.3 Text-to-molecule

Here, we show the samples of text-based molecule generation, i.e., generated molecules for the given text description in Figure 18. Notably, we show the generated samples from base-sized models for fair comparison.

Additionally, we provide the results of generalist models in Table 12. Note that it is natural to show no consistent enhancement for generalist models as they lack reasoning ability as shown in Table 6.

<table>
  <tr>
    <th>Models</th>
    <th>BL.</th>
    <th>Ex.</th>
    <th>Le.</th>
    <th>MA.</th>
    <th>RDK</th>
    <th>Mn.</th>
    <th>FCD<sub>i</sub></th>
    <th>Val.</th>
  </tr>
  <tr>
    <th colspan="9">Generalists (10-shot learning)</th>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>0.251</td>
    <td>0.007</td>
    <td>117.30</td>
    <td>0.586</td>
    <td>0.352</td>
    <td>0.276</td>
    <td>13.11</td>
    <td>0.629</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td>0.259</td>
    <td>0.008</td>
    <td>109.77</td>
    <td>0.579</td>
    <td>0.279</td>
    <td>0.344</td>
    <td>4.47</td>
    <td>0.669</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.521</td>
    <td>0.079</td>
    <td>40.87</td>
    <td>0.797</td>
    <td>0.496</td>
    <td>0.583</td>
    <td>3.67</td>
    <td>0.881</td>
  </tr>
  <tr>
    <td>+MSR</td>
    <td>0.509</td>
    <td>0.088</td>
    <td>41.68</td>
    <td>0.783</td>
    <td>0.498</td>
    <td>0.571</td>
    <td>1.57</td>
    <td>0.846</td>
  </tr>
</table>

Table 12: Text-based Molecule Generation Performance for generalist models.

B.4 Ablation study

Comparison to ChemCrow. To validate the efficacy of our MSR, we compare our method with ChemCrow (M. Bran et al., 2024), which has employed CoTs for various chemical tasks. The comparative results are provided in Table 13 and Table 14. One can observe that ChemCrow shows limited performance in both molecule captioning and text-based molecule generation tasks. It is notable that the comparison may not be entirely appropriate, as ChemCrow is primarily designed for practical synthesis tasks, as the reviewer mentioned. Nevertheless, we included comparisons with ChemCrow to provide additional insights, as they share a similar motivation: enriching large language models (LLMs) with a chemistry-aware chain-of-thoughts.

<table>
  <tr>
    <th>Models</th>
    <th>BL.-2</th>
    <th>BL.-4</th>
    <th>RO.-1</th>
    <th>RO.-2</th>
    <th>RO.-L</th>
    <th>MET.</th>
  </tr>
  <tr>
    <td>ChemCrow (GPT-4o)</td>
    <td>0.162</td>
    <td>0.078</td>
    <td>0.299</td>
    <td>0.097</td>
    <td>0.211</td>
    <td>0.225</td>
  </tr>
  <tr>
    <td>Ours (GPT-4o)</td>
    <td>0.249</td>
    <td>0.139</td>
    <td>0.386</td>
    <td>0.179</td>
    <td>0.300</td>
    <td>0.303</td>
  </tr>
  <tr>
    <td>Ours (ChemTS-base)</td>
    <td>0.639</td>
    <td>0.560</td>
    <td>0.687</td>
    <td>0.553</td>
    <td>0.626</td>
    <td>0.657</td>
  </tr>
</table>

Table 13: Comparison to ChemCrow in molecule captioning.

<table>
  <tr>
    <th>Models</th>
    <th>BL.</th>
    <th>Ex.</th>
    <th>Le.</th>
    <th>MA.</th>
    <th>RDK</th>
    <th>Mn.</th>
    <th>FCD<sub>i</sub></th>
    <th>Val.</th>
  </tr>
  <tr>
    <td>ChemCrow (GPT-4o)</td>
    <td>0.306</td>
    <td>0.194</td>
    <td>56.46</td>
    <td>0.772</td>
    <td>0.632</td>
    <td>0.555</td>
    <td>2.31</td>
    <td>0.851</td>
  </tr>
  <tr>
    <td>Ours (GPT-4o)</td>
    <td>0.509</td>
    <td>0.088</td>
    <td>41.68</td>
    <td>0.783</td>
    <td>0.498</td>
    <td>0.571</td>
    <td>1.57</td>
    <td>0.846</td>
  </tr>
  <tr>
    <td>Ours (ChemTS-base)</td>
    <td>0.874</td>
    <td>0.421</td>
    <td>12.78</td>
    <td>0.924</td>
    <td>0.836</td>
    <td>0.804</td>
    <td>0.28</td>
    <td>0.982</td>
  </tr>
</table>

Table 14: Comparison to ChemCrow in text-based molecule generation.
Input

MolT5-base
The molecule is a member of the class of 1,3,4-oxadiazoles that is 1,2,4-triazole which is substituted at positions 3 and 5 by 4-oxo-4H-1,2,4-triazol-3-yl and 4-oxo-4H-1,2,4-triazol-5-yl groups, respectively. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid, a member of monochlorobenzenes and a member of monothiophenes.

MolT5-base + Ours
The molecule is a 1,3,4-oxadiazole substituted by a 2- chlorophenyl group at position 2, a 4-oxo-4H-1,2,4-triazol-3-yl group at position 5 and 2-chlorophenyl group at position 6. It is a member of 1,3,4-oxadiazoles and a member of monochlorobenzenes.

ChemT5-base
The molecule is a 2,2'-bithiophene that is 1,3,4-oxadiazole possessing 2,2'-bithiophen-5-yl and 5-methyl-2-chlorophenyl groups at positions 2 and 5, respectively. It is a member of 1,3,4-oxadiazoles and a member of monothiophenes.

ChemT5-base + Ours
The molecule is a member of the class of 1,3,4-oxadiazoles that is 1,3,4-oxadiazole which is substituted at positions 2 and 5 by 2,2'-bithiophen-5-yl and (thiophen-2-yl)-1,3,4-oxadiazol-5-yl groups, respectively. It is a conjugate base of a 4-amino-5-(2,2'-bithiophen-5-yl)benzoic acid, a member of 1,3,4-oxadiazoles, a member of monochlorobenzenes and a member of monothiophenes.

Ground Truth
The molecule is a 1,3,4-oxadiazole substituted by a 2- chlorophenyl group at position 2 and a 5-methyl-2-thienyl group at position 5. It is a member of 1,3,4-oxadiazoles and a member of monothiophenes.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

The molecule is an organosulfonate oxanion obtained by depolarization of the conjugate base of 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid. It is a conjugate base of a 4-amino-5-(4-oxo-4H-1,2,4-triazol-3-yl)benzoic acid.

Figure 16: The generated samples of molecule captioning.

Product

Mol-Instructions

Mol-Instructions+MSR

Ground Truth

Figure 17: The generated samples of retrosynthesis.
Input
The molecule is an ammonium ion resulting from the protonation of the conjugate base of a carboxylic acid or conjugate acid of an alerine.

MolIT5-base
MolIT5-base + Ours
ChemT5-base
ChemT5-base + Ours
Ground Truth

The molecule is a quinoline consisting of quinolin-4(1H)-one carrying a hydroxyl substituent at position 3 and a methyl substituent at position 3. It has a role as a signaling metabolite.

The molecule is a benzodiazepine that consists of piperidine bearing 1,3-benzodioxole-5-(1H)-yl/methyl substituents at positions 2 and 4 respectively; the (3S,4R)-diastereomer. Highly selective and potent GABAa receptor inhibitor that binds with high affinity to the GABAa receptor (Ki = 0.08 nM). Ki values are 1.1, 359, and 16 nM for inhibition of [3H]-SHT1, [3H]-NMDA and [3H]D2 uptake respectively. Displays weak affinity for muscarinic M2, M3, or beta-adrenoceptors, S-HT2A, 5-HT1A, D2 or H1 receptors. Ki values are below 1000 nM however displays weak affinity for muscarinic M2, M3, or beta-adrenoceptors (Ki > 42 nM). Antidepressant and anxiolytic properties. It is also used as an antidepressant, an anxiolytic drug, a serotonin uptake inhibitor, a hypnotic, and a non-selective 5HT5A inhibitor. It is a member of a class of drugs known as benzodiazepines, an organofluorine compound and an aromatic ether. It is a conjugate base of a monofluorobenzene. It is a conjugate base of a paraxethonium+.

Figure 18: The generated samples of text-based molecule generation.
C  Usage of AI assistants

In preparing this work, we utilized AI-based writing assistants to refine sentence structure, correct grammatical errors, and enhance readability. These tools were employed only for rephrasing and language improvements, ensuring that the technical content, methodology, and experimental findings remained entirely authored by the researchers. The use of AI assistance was limited to editorial enhancements without influencing the originality or scientific contributions of the paper.

D  Scientific Artifacts

The License for artifacts. All datasets and software tools used in this work adhere to their respective licenses. Specifically, we employed publicly available datasets such as ChEBI-20 and L+M under their permitted usage terms. Additionally, external tools like RDKit were used following their open-source license. We release our trained models and code in https://github.com/yunhuijiang/MSR under an appropriate open-source license to facilitate reproducibility.

Artifact use consistency with intended use. The datasets and tools utilized in our study were used in accordance with their intended purpose. For example, ChEBI-20 and L+M datasets were originally developed for molecule captioning and generation tasks, aligning with our research objectives. Similarly, RDKit was employed for molecular structure analysis as intended by its developers.

Documentation of artifacts. We provide details in https://github.com/yunhuijiang/MSR.

Reviewer paper summaries: This paper examines the research problem of molecular understanding with Large Language models (LLMs) by proposing the Molecular Structural Reasoning (MSR) framework. The key idea of MSR is to incorporate six predefined structural information into the fine-tuning process of LLMs, e.g., longest carbon chain length, molecular formula, etc. The fine-tuning process of MSR could be categorized into analytic reasoning and synthetic reasoning. Analytic reasoning incorporates the decomposition of the original molecule into smaller ones by leveraging external tools and then reasoning. Synthetic reasoning will first compose structural information to construct the molecule. MSR is evaluated on three downstream tasks, molecule-to-text, retrosynthesis, and text-to-molecule with both chemical and general LLMs as baselines, all showing promising performance.
Summary of strengths: 1. The paper studies how LLMs could be leveraged for molecule reasoning tasks, which is interesting and novel in the community of AI4Chemistry.
2. The proposed method is sound. The incorporation of different categories of predefined structural information is interesting. The categorized reasoning processes, analytic and synthetic, also align well with downstream tasks.
3. The experiments are pretty solid. The authors not only experiment with few-shot learning but also fine-tuning on LLMs. Apart from the great experiment results, the authors also conduct a bunch of analyses that are quite insightful. For example, I really like the ablation study of different structural information categories.
4. Overall the paper is well written and easy to follow.
5. The anonymous version of the code is also available for reproducibility.
Summary of weaknesses: 1. The motivation of selecting the six predefined structural information needs further motivation. For example, are the six categories designed by chemists or domain experts? 
2. Following the previous point, the setting of the preliminary study in Section 2 needs further clarification. For example, when justifying the importance, instead of giving a case study, the authors may manually analyze dozens of examples to provide a quantitative study. Or just simply cite some chemistry-related papers. This philosophy also applies when the authors want to prove that current LLMs fail to accurately infer correct structural info.
3. It seems that the downstream task types are highly correlated to different reasoning types. I am wondering if analytic and synthetic reasoning is able to generalize to other molecule-related tasks that require structural information.
Comments, suggestions and typos: - There are duplicate "broader impacts" in line 554 and line 592.
Overall assessment scores: 3.5
Metareview score: 4
Metareview: This paper introduces Molecular Structural Reasoning (MSR), a framework that enhances LLMs’ molecular understanding by explicitly incorporating six key structural features. The approach distinguishes between analytic and synthetic reasoning depending on whether the molecule is given or generated. MSR improves performance across multiple tasks and datasets, including molecule-to-text, retrosynthesis, and text-to-molecule.
Summary of reasons to publish: * Conceptually Clear Framework: The analytic/synthetic split mirrors practical workflows and is easy to adopt.
* Empirical Support: MSR consistently improves performance across tasks, often enabling small models to match or exceed larger baselines.
* Practical and Interpretable: The use of RDKit and matching-ratio sampling adds structural fidelity with minimal overhead.