LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset
Authors: Botao Yu
Frazier N. Baker
Ziqi Chen
Xia Ning
Huan Sun
Venue: COLM (2024)
Keywords: chemistry, LLMs, fine-tuning, instruction tuning, dataset, molecule
TL;DR: Our LLMs that can achieve very strong results on chemistry tasks, outperforming GPT-4 and Claude 3 Opus by a substantial margin and approaching the SoTA task-specific models.

Abstract: Chemistry plays a crucial role in many domains, such as drug discovery and material science. While large language models (LLMs) such as GPT-4 exhibit remarkable capabilities on natural language processing tasks, existing research indicates that their performance on chemistry tasks is discouragingly low. In this paper, however, we demonstrate that our developed LLMs can achieve very strong results on a comprehensive set of chemistry tasks, outperforming the most advanced GPT-4 and Claude 3 Opus by a substantial margin. To accomplish this, we propose SMolInstruct, a large-scale, comprehensive, and high-quality dataset for instruction tuning. It contains 14 selected chemistry tasks and over three million samples, laying a solid foundation for training and evaluating LLMs for chemistry. Using SMolInstruct, we fine-tune a set of open-source LLMs named as LlaSMol, among which, we find that Mistral serves as the best base model for chemistry tasks. Our analysis further demonstrates the critical role of the proposed dataset in driving the performance improvements.

Published as a conference paper at COLM 2024
LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset
Botao Yu Frazier N. Baker' The Ohio State University Columbus, OH 43210, USA {yu. 3737, baker. 3239, chen. chen.8484, ning. 104, sun. 397] e0su. edu
Ziqi Chen' Xia Ning
Huan Sun
Abstract
Chemistry plays crucial role in many domains, such as drug discovery and material science. While large language models (LLMs) such GPT- 4 exhibit remarkable capabilities on natural language processing tasks, existing research indicates that their performance on chemistry tasks is discouragingly low. In this paper, however, we demonstrate that our de- veloped LLMs can achieve very strong results on comprehensive set chemistry tasks, outperforming the most advanced GPT-4 and Claude 3 Opus by a substantial margin. To accomplish this, we propose SMolInstruct, large-scale, comprehensive, and high-quality dataset for instruction tuning. It contains 14 selected chemistry tasks and over three million samples, laying solid foundation for training and evaluating LLMs for chemistry. Using SMolInstruct, we fine-tune à set of open-source LLMs named as LlaSMol, among which, find that Mistral serves the best base model for chemistry tasks. Our analysis further demonstrates the critical role of the proposed dataset in driving the performance improvements
1 Introduction
Chemistry is fundamental science that underpins countless aspects of modern life, ranging from drug discovery and materials science to energy production. To facilitate research and applications in this domain, deep learning models including graph neural networks (Kipf & Welling 2017) and Transformer-basel models (Vaswani et al. 2017) have been developed for various chemistry tasks such as forward reaction prediction, retrosynthesis, property prediction (Schwaller et al., 2019 Zhong al., 2022 Chen et al. 2023 Zhou et al. 2023). However, these models are usually task-specific models, which neglect shared chemistry knowledge across tasks and can hardly be adapted to different tasks.
On the other hand, large language models (LLMs) such as GPT-4 (OpenAl 2023), Llama se- ries [Touvron et al., 22023a b), and Mistral (Jiang et al. 2023) have emerged as general-purposst foundation models and demonstrate remarkable abilities on various natural language pro- cessing tasks (Chang et al. 2024 Thirunavukarasu et al. 2023 Yue et al., 2023 Zhang et al.) 2023 Deng et al. 2023). However, when applied to chemistry tasks, LLMS show only
al. 2023 For example, Guo et al. (2023) conducted evaluations on eight chemistry tasks and observed that
limited capabilities (Jablonka et al. 2022 Guo et al. 2023 Hatakeyama-Sate
while GPT-4 outpertorms other closed- and open-source LLMs, its performance is far from that of task-specific deep learning models. Particularly, they found that GPT models perform poorly when a precise understanding of SMILES (Weininger) 1988), a widely used textual representation for molecules, is required. In addition to directly applying pretrained L.MMs, Fang et al. (2023) fine-tuned LLMSs on an instruction tuning dataset, but their performance remains very low, far behind the sfate-of-the-art (SoTA) models designed and trained for specific tasks.
"Equal contribution.
1 Our dataset and models be found at a thub. Lo/LLM/

---

Published as a conference paper at COLM 2024
Name Conversion
Property Prediction
IUPAC to Molecular Formula (NC-12F)
(PP-EPP-
<UBACC 2,5 S--dphanl--33--.o coazzse -ouzse--IUPACC
What the moleaaee formuia of compontd with dhe ILPAC nome
Quey.
-NUMBERO <NUMBER
Reposse: OOOL COLIORMMULL- CISHTINO +MOUPOOUUAA
LIPO (PP-LIPO)
IUPAC to SMILES (NC-12S)
Query: Predkt the octandl water diatttation cofficient legh inder the circmnaacc of 14 fr -SMI NCC-0)C--00-----0 OSMM.SS..
you possid dee SMILES
<UBAC 4chll44-clloo
<IUUAA-
Bospasc
1010 -NUMBER
Repasse: Of course -SMILIS- OCC (CCCCC000C <SMILISP
BBBP (PP-BBBP)
SMILFES Molecular Formala (NC-52F)
Query: Is Moodhnin barker pennesbilty (RDBP) propery
Quey: Giver SMILES rpresoraaiii
fomuda?
Bespasi: -BOOLENN- Ys--BOOLEAN-
-SMILESS- CCNO--OFC C----CC--c----- C1-MMMLES-
Repassi MOLLOOXMUU CCHISC1IN20rS
ClinTox (PP- (PP-ClinTox)
SMILES
(NC-S21)
Is <SMIl BS cOCC ( +NME kesic?
Query: Itandats the given SMILES fermal moleculs
Bospasc -BOOLEAN- No -BOLLANN
-SMILLSS COC/CCCIG CCCENCEUCC USMELESS into BUMCC rame.
Reposse: -TUPAC Sogit 5dlononee <HUPAC-
HII (PP-IIVV
isaibiur of rppliazion?
Molecule Description
-BOOLENN- -BOOLAA
Molecale Captioning (MC)
SIDER (PP-SIDER)
Deserit colecule.
Query: lic lnown offeea
<aMILLE CCOC)OO) (CCO))CCCCCCCCCCCC NICC iggtt CCTCC CDCCOOC <SMILES>
-SNILESS CC) (CIC) CNC OINNCC OIXYCC OJOICICY CIBI-SMILES- stfectig Raspatec
Resposee The noboaee on insiaool derivative with shen artnn sodative, lyprotii, and greend aneeieee prepeniss. Etoiidat oppors to pammo-mmnnl whityrie acid (CABA) like effect, nediated deough GAABAAA rcoppus The aciun crinaneo the athiziorr effectof GABA ou the castel aciveso systes by casiing daloride classdd opening eveata wissh loads to memoree typequaansston
+BOOLEAND
Chemical Reaction
Forward Syathesis
Molecule Generation (MG)
Query: MCC-CCCCCCCCC COCN-NM
Query: Give ne molezale that staaii the conditions outined in the doscriptiss The mekecale ic momber of to clss impymaao that a re-collanad pegeeet with aritiatz
nagan gion srggest poeible
Resgonse: posill
-MILES- CNC C200002 -MMILES
poperte produced by Semii manoozeee has a male antmrenball agar, helogial Pigsert, a bacerial dita os apoyosis islucer and on extincoplatii agoet, Rt fro niyymie, en aroeaat other and aseeriyy.
Retrosynthesis (RS)
Reporse thee i potntal
-SMILES> CCCOCCi-CICNEIC
CCI-CC--CC--- CCjCC-C +XMILES-
"SMILES- CCT
Regoose -SMILES- COLCINJCCOHNN -SMILESS
Query: lécatity powitli martem andld fome been seal to croie the spacified prodkee.
JIN-IMMILES
Figure 1: An overview of tasks in the proposed SMolInstruct dataset.
Given these discouraging results, some critical questions arise: Are LLMs actually able to effectively perform chemistry tasks? Or, Are they fundamentally limited for chemistry? In this paper, we demonstrate that our developed LLMs can achieve very strong results on a comprehensive set of chemistry tasks, substantially outperforming the most advances GPT-4 (2023) and Claude 3 Opus Anthropic (2024).
What makes such LLMs possible? First, we construct a large-scale, comprehensive, and high-quality dataset for instruction tuning named SMolInstruct. We incorporate tasks with meaningful applications, collect data from diverse data sources, and apply rigorous scrutiny for quality control. The resulting dataset consists of 14 tasks (illustrated in Figure 1 and over 3M samples, laying solid foundation for training and evaluating LLMs for chemistry tasks. Based on the dataset, we build a series of LLMs for chemistry named LlaSMol by fine-tuning four open-source LLMs namely Galactica, Llama 2, Code Llama, and Mistral, on SMolInstruct with LoRA (Hu et al. 2022).
We conduct comprehensive experiments to evaluate our models and explore their insights, yielding some interesting findings. Firstly, among the four LlaSMol models, the Mistral- based model surpasses others by substantial margin, showcasing the considerable influ- ence of base models on downstream chemistry tasks. Moreover, contrast to claims made in previous work (Fang et al., 2023), , using SMILES the molecular representation achieves suf- ficient validity of generated molecules and better performance compared to using SELFIES (Krenn et al. (2019). Furthermore, employing canonicalized SMILES during model training and applications can alleviate learning burdens and increase performance. Finally, while instruction tuning can inject chemistry task-related knowledge into models, the dataset plays crucial role. Our experiments demonstrate that training on our SMolInstruct leads to substantially better performance compared to training on previous dataset, emphasizing the contribution of the proposed dataset. Although LlaSMol models do not yet surpass state-of-the-art (SoTA) tasc-spe models that are designed and trained specifically for each individual task, they approach SoTA performance with only 0.58% of parameters being fine-tuned, suggesting their great potential for further improvements and to serve as strong foundation models for the field.
2

---

Published as a conference paper at COLM 2024
2 Related Work
Task-specific Models for Chemistry. In recent years, many deep learning models have been developed to tackle different chemistry tasks. For example, Molecular Transformer Schwaller et al. (2019) and (SMILES/Zhong et al. (2022) formulate forward synthesis and retrosynthesis prediction as sequence-to-sequenee translation problems. Chemformer [rwin et al. (202)) pretrains transformer model on large-scale SMILEE dataset and fine-tunes if for various downstream tasks, such as forward synthesis and property prediction. MolT5 Edwards et al. (2022) first pretrains T5 model on both SMILES and natural language, and then tine-tunes it to translate SMILES into natural language (i.e., molecule captioning) or vice versa (i.e., molecule generation). Graph neural networks (GNNs), which directly leverage the graph structure of the molecule Wang et al. (2023), have also shown promise in many chemistry applications, such as property prediction Yang et al. | Han al. 20223
retrosynthesis Chen et al. (2023); Sommath et al. 22221), and molecule optimization Chen
et al. [202]]; Zhang et al. (2022b). Recent studies Zhou al. (2033;; Zhang et al. (2022a) have
shown the promise of leveraging equivariant representations of molecular 3D structures
for chemistry tasks, such as property prediction Zhou et al. (2023) and docking Zhang et al. (2022a). Uni-Mol Zhou et al. (2023) incorporates this 3D intormation into the pretraining of a transformer model and fine-tunes it for downstream tasks. Despite their effectiveness,
these models operate on single tasks and therefore cannot harness knowledge shared across
diverse chemistry tasks like LLMs.
LLMs for Chemistry.
Recent efforts have integrated LLMs with chemistry to solve key chemistry problems, which can be divided into two categories: (1) benchmark studies, and (2) fine-tuning LLMs with new datasets. Multiple benchmark studies White et al. (2023);
Guo et al. (2023) ; Jablonka et al. (2023) Liu et al. 2023a) have evaluated the capabilities and limitations different off-the-shelf LLMs, such as GPT4 and Llama, on chemistry problems. For example, Guo et al. (2023) finds that these LLMs do not perform well on chemistry tasks and often produce chemically implausible outputs. These findings highlight the need for further efforts to improve LLMs via fine-tuning for chemistry tasks.
To improve LLMs for chemistry, multiple instruction tuning datasets have been developed. Mol-Instructions Faaetiionsnnnnggoo et al. (2023) consists of 1.3M instructions for multiple small molecule tasks. However, fine-tuning on the dataset does not significantly improve LLMs' perfor- mance (Section 4.3 Drugehat BaaughaaLLiangg et al. (2023) collects an instruction tuning dataset on drug properties with 10.8K drug molecules.
Ye et al. (003)) consists of instructions with 1M molecule pairs for molecule optimization six properties bier,es,, in which each pair has similar molecules with different properties. Recent works also develop 2D or 3D molecular graph-centric datasets and integrate the graph understanding ability into LLMs Liu et al. (023b);; Cao et al. (2023); Li et al. (2024). Compared with these datasets, SMolInstruct is much larger and covers a more diverse and comprehensive set of chem- istry tasks, which enables LLMs to better understand molecule representations and learn chemistry knowledge across tasks.
MolOpt-Instructinn
3 SMolInstruct
This section introduces our proposed dataset SMolInstruct and its construction. Readers may refer to Appendix A for preliminaries and background.
3.1 Overview of SMolInstruct
SMolInstruct is large-scale instruction tuning dataset that centers around small molecules. contains 14 chemistry tasks, illustrated in Figure
(1) We include four name conversion tasks, namely converting IUPAC name to molecular formula (NC-12F), converting IUPAC name to SMILES (NC-12S), converting SMILES to molecular formula (NC-S2F), and converting SMILES to IUPAC name (NC-S2I). They are designed to enable deep understanding of molecular structures and representations, which should serve as the fundamental knowledge for chemistry L.MSs.
3

---

Published as a conference paper at COLM 2024
(2) Additionally, six property prediction tasks (Wu et al. 2018) are integrated, including PP-
ESOL for water solubility (Mobley & Guthrie 2014), LipL for octanol/ /water distribution coefficient (Poole & Poole 2003), PP-BBBP for blood-brain barrier penetration (Martins
et al. 2012), PP-Clin for toxicity to human body (Gayvert et al. 2016), PP-HIV for HIV
replication inhibition (Institute| 2004), and PP-SIDER for side effects of drugs (Kuhn et al.
2015). These involved properties are crucial especially for drug development.
(3) Two tasks focus on the textual descriptions of molecules: molecule captioning (MC) is to generate a textual description of a given molecule, and molecule generation (MG) is to generate molecule based on the given textual description. They require comprehensive understanding of molecules - their structures and properties, from their textual descriptions. They also bridge the gap between natural language and molecules.
(4) Lastly, two tasks revolve around chemical reaction knowledge. Forward synthesis (FS) aims to predict potential products from reactants and reagents, and retrosynthesis (RS) involves predicting potential reactants given a product. These tasks play vital roles in real-world applications (Coley et al., 2018). For example, retrosynthesis essential for synthesis planning, while forward synthesis is used to validate retrosynthetic suggestions.
SMolInstruct contains 3.3M samples. Each sample is a query-response pair, where the query describes a task and any task-specific information (e.g., input molecule, textual description, etc.), and the response is a sentence containing the answer to the queried task. For all the tasks, unless explicitly defined in the tasks (NC-12F, NC-12S, NC-S2F, and NC-S2I), use (Krenn
SMILES as the default representation for molecules, but also provide the SELFIES et al. 2019) representation.
3.2 SMolInstruct Construction
We construct the SMolInstruct dataset by following a four-step pipeline: data collection, quality control, data splitting, and instruction construction.
After consulting domain experts and pinpointing the set of meaningful tasks (summarized in Section 3.1), we collect data for these tasks from various sources, as listed in Table 5 Specifically, for the name conversion tasks (NC-12F, NC-12S, NC-S2F, and NC-S2I), we leverage PubChem Kim et al. 2019), one of the most comprehensive molecule databases. Within this database, we randomly select large set of molecule entries, and extract their IUPAC names, SMILES representations, and molecular formulas. This obtained data is then re-organized as input-output pairs for the tasks. For molecular description- related tasks (MC and MG), we utilize a combination of ChEBI-20 (Edwards et al. 2021 2022) and Mol-Instructions (Fang et al. )023), as they both contain high-quality mo molecule- text paired data. For property prediction tasks (PP-ESOL, PP-Lipo, PP-BBBP, PP-Clin Tox, PP-HIV, and PP-SIDER), we employ the well-established MoleculeNet datasets (Wu et al., 2018). We select the 6 datasets from MoleculeNet that represent the essential properties for real-world applications such as drug discovery. For chemical reaction tasks (PS and RS), we collect the reaction data from USPTO-full (Lowe 2017), which is an extensive collection encompassing over 1M reaction samples extracted from U.S. patents. All the
Data Collection.
aforementioned datasets are also widely used in previous studies (He al. 2021 2021:Zhong
et al.
2022 Edwards et al. 2022 Irwin et al. 2022
Chen et al. 2023 Zhou et al. 2023).
Quality Control. To guarantee high quality, we apply rigorous scrutiny. The collected data contains many problematic and low-quality samples, which can be roughly categorized into the following three types, along with our curation methods: (1) Chemically invalid SMILES. Numerous SMILES strings are chemically invalid (e-g,, deviating from the SMILES MMILES&gg grammar, or violating chemical valence). To address this issue, we employ RDKit (RDKit) 2023),
widely used toolkit for cheminformatics, to parse molecules and detect errors. (2) Wrong or inaccurate information. Based on manual check, we observed wrong and inaccurate information recorded in the data. For instance, within the fUl dataset (Lowe) 2017), we identify and correct mislabeled reactants and reagents in chemical reactions by comparing their atom mappings with products. For the MC and MG tasks, we filter out those
ncbi. nlm. gov.
4

---

Published as a conference paper at COLM 2024
textual descriptions that lack pertinent, molecule-specific information, with a set of rules based on wording patterns, lengths and keywords. For PP-SIDER, we eliminate disorders with ambiguous names that could impede the creation of precise and comprehensible instructions. (3) Duplicated samples. We detect and remove them.
Data Splitting. Data splitting for multi-task datasets requires careful handling to prevent data leakage across tasks. For instance, FS and RS are a pair of reverse tasks, so data leakage occurs when the training set contains an FS sample for a certain chemical reaction and the test set has an RS sample for the same reaction. This can lead to biased evaluation. Therefore, we identify sample pairs across related tasks (FS and RS, MC and MG, and the four NC molecules/reactions reactionss, and ensure that matched samples are placed together in either training or evaluation set. Moreover, some samples may share the same input but have different outputs. For instance, in the RS task, one product (the same input) may be synthesized from multiple sets of reactants (different outputs). If these samples are placed into both training and test set, it may lead to exaggerated performance. Therefore we ensure that samples with identical inputs are placed together either in or outside of the test set. Additionally, to achieve fair comparisons with Mol-instructions (Fang iet al. 2023), for tasks shared between the two datasets (MC, MG, FS, and RS), we ensure that their training examples are not included in the test set of SMolInstruct, allowing for a direct evaluation of their models on our test set. After implementing these constraints, samples are randomly split into training/ / /aalddaaioon////////oo test set, except for PP task samples that undergo scaffold splitting following the canonical method (Wu et al.) 2018).
tasks) that correspond to the same molecules
Instruction Creation. To create query-response textual pairs for instruction tuning, we manually craft several templates, each including a query and a corresponding response, and apply GPT-4 to rephrase them. Unlike those in (Fang et al.] 2023) which consist of highly formatted queries (containing three explicitly labeled parts namely instruction, input, and output) and answer-only responses (e.g., responses for FS and RS only contain answer SMILES alone, without any natural text), our templates exhibit a more natural and diverse set of formats in both queries and responses, allowing for more variations and naturalness in input-output interactions. Moreover, all the SMILES representations are canonicalized, establishing a standardized data format. In light of the dataset's inclusion of multi-type sequences (SMILES, molecular formula, numbers, etc.) beyond natural language text alone, we utilize special tags to encapsulate corresponding segments (e.g., <SMILES>... SOMILEE>...</SMMLESS </SMILES>> for SMILES, <MOL FORMULA>. . ... FORMULA for molecular formula, <NUMBER>.. ... ./NNUBRR..//NMMBER>>>> for numbers). This design does not only explicitly inform models about the information types within the tagged content, but also facilitate answer extraction during evaluation.
For more details of dataset construction, please refer to Appendix B.2
3.3 Merits of SMolInstruct
Compared to previous work (Fang et al. 2023 Liang et al. 2023 ee et. al. stands out in several key aspects:
2023),
SMolInstruct
(1) Large-Scale. SMolInttruct consists of 3.3M samples and 1.6M distinct molecules, with a diverse range of sizes, structures, and properties (see Appendix B.1, showcasing an extensive coverage of diverse chemical knowledge.
types of chemical tasks (14 tasks in total), emerging as the most comprehensive instruction tuning dataset for small molecules. Notably, the tasks are meticulously selected to build a strong chemistry foundation model and to adapt to real-world applications.
(2) Comprehensive. SMolInstruct contains
(3) High-Quality. Rigorous processing steps have been implemented to exclude problematic and low-quality samples. Along with careful data splitting and canonicalization of SMILES representations, SMolInstruct stands as a high-quality resource valuable for future research.
A detailed introduction and statistics of the SMolInstruct dataset can be found in Appendix B For a comparison with the previous work, Mol-Instructions (Fang et al. 2023), please refer to Appendix
5

---

Published as a conference paper at COLM 2024
4 Experiments
4.1 Our LlaSMol Models
By fine-tuning base models on the proposed SMolInstruct dataset, we create LLMs capable of performing chemistry tasks, which we name LlaSMol (Large language models Small Molecules). Specifically, we extensively consider four different LLMs as our base models, namely Galactica 6.7B (Taylor et al. 2022), Llama 2 (Touvron et al., 2023b) 7B, Code Llama (Roziere et al.J 2023) 7B, and Mistral Uiang et al. 2023) 7B, where Galactica is trained for scientific applications and has already been exposed to chemistry-related data during its pretraining, Llama 2 and Mistral are general-purposs
LLMs, while Code Llama is based on Llama 2 and trained for code. We conduct instruction tuning on the proposed SMolInstruct dataset, and name the resulting models as LlaSMolGalactian LlaSMol 2/ LlaSMolCode Llama, and LlaSMol respectively. All the LlaSMol models are trained with LoRA (Hu et al., 2022), which is applied to all weight matrices in the self-attention and feedforward neural network (FFN) modules with lora_r and lora_alpha set to 16. The fine- tuning process utilizes the Huggingface Transformers library (Wolf et al. 2020). Training spans three epochs, employing the 8-bit A optimizer, a learning rate of 1e-4, and cosine scheduler. The input length for training is set to 512, which covers 99.7% of the samples. During inference, we adopt beam search as the generation strategy for simplicity.
4.2 Experimental Setup
Compared Models. We compare our LlaSMol models with two types of models:
(1) LLMs without fine-tuning on SMolInstruct. This type includes our four base models, namely Galactica (Taylor et al. 20022,, Llama 2 (Touvron et al. 2023b), Code Llama (Roziere
al. 2023), , Mistral Mistral[fiang (Jiang et al. 2023). we also benchmark against GPT-4 (OpenAl 2023) and
the more recent Claude 3 Opus (Anthropic 2024), the current ofate-of-the-art (SoTA) L B
For Llama 2, Code Llama, and Mistral, we use I-shot, due to their poor instruction following ability; for GPT-4, we report its results under a zero-shot setting, as GPT-4 performs best on
this setting in our experiments (Appendix E); for Claude 3 Opus, we report its zero-shot results as well. We also include two LLMs tuned specifically for chemistry tasks: Molinst, Llama 2 model tuned on the Mol-Instructions dataset by Fang et al. (2023), which shares the training tasks of MC, MG, FS, and RS with LlaSMol; and Ch (aammLLLMMaanngg..gg [Zhang et al. 2024), an
LLM for chemistry proposed concurrently to our work.
(2) SoTA task-specific models. To provide a comprehensive view of LlaSMol's performance, we present results from SoTA task-specific models. For NC-12S and NC-S2I, we compare with STOUT Rajan et al. 2021), an encoder-decoder model trained on SMILES-IUPAC name paired data. For S2- task achievable with a fixed algorithm, we implement program with RDKit (RDKit 2023), a widely used Python toolkit for cheminformatics, and report its results. For NC- 2F where no dedicated models exist, we construct a baseline called STOUT+RDKit by aggregating STOUT for I2S conversion and RDKit for S2F conversion. For the PP tasks, our compared model is Uni-Mol (Zhou et al. 2023). It incorporates molecular 3D representations and follows pretraining and fine-tuning paradigm. Following its original settings, we fine-tune the model on our SMolInstruct dataset with its pretrained checkpoint. In the case of MC and MG, we compare with MolT5 (Edwards and directly use their released checkpoint. The reasons why we do not use our re- re-trained model are: (1) we were unable reproduce results close to those reported in the paper as no original code was provided; and (2) we take great care to ensure that our test set is devoid of training examples used by MolT5, ensuring fairness in the evaluation. Lastly, regarding FS and RS, we re-train RSMILES (Zhong et al.) 2022) and Molecular Transformer (Schwaller et al. 2019) for the two tasks, respectively, following their reported settings. Both of the models are transformer encoder-decoder models /Vaswani et al. 2017), specifically adapted for the FS and RS tasks.
al.) 2022
Due to resource limitations, we evaluate GPT-4 and Claude 3 Opus on at most 500 test samples for each task.
6

---

Published as a conference paper at COLM 2024
Table 1: Results on name conversion (NC) and property prediction (PP) tasks, Metrics EM, Valid, and Acc are in percentage.
NC
PP
Model
12F
12S
S21
ESOL
Lipo
BBBP
Clintox
HIV
SIDER
EM
EM
Valid
EM
EM
RMSE!
RMSE,
Acc
Acc
Task- apecifiecific, Non-LLM Based Models
SoTA
97.9
73.5
1000
56.5
0.819
0.612
85.3
92.4
97.0
70.0
Existing LLMs without fine-tuning SMolInstruct
GPT-4
8.7
3.3
84.2
488
0.0
2.570
1.545
62.9
50.0
59.6
57.6
Claude 3 Opus
34.6
17.7
90.2
9.2
0.0
1.036
1.194
75.1
41.7
76.4
67.0
Galactica
9.1
9.7
95.6
0.0
0.0
4.184
2.979
69.0
92.4
96.7
68.1
Laama 2
0.0
0.0
18.3
0.0
0.0
3.287
1.634
58.9
45.1
93.3
61.9
Code Llama
0.0
3,483
1.733
58.9
85.4
91.8
60.2
Mistral
0.0
0.0
40.3
0.0
0.0
3.079
1.730
40.6
15.3
7.1
38.1
Molinst (chemistry LLM)
0.0
2.271
1.691
60.9
32.4
ChemLLM (chemistry LLM)
0.8
0.3
3.9
0.0
0.0
1.946
1.797
22.3
75.7
72.9
32.6
Our LlaSMol Series
LlaSMolcalsctio
83.2
58.7
99.4
91.2
18.3
1,959
1.213
69.0
93.1
96.7
70.1
LJaSMol Lamm
73.8
46.6
87.0
12.9
2.791
1.338
69.0
92.4
96.7
68.7
UaSMolcode 1lama
75.4
49.9
99.3
88.6
15.5
2.959
1.203
69.0
93.1
96.7
69.9
LlaMMolyistrll
87.9
1.150
96.7
Evaluation Metrics. We employ metrics commonly used in previous work (Schwaller et al.
2019 Zhong al., 2022: Fang al., 2023 Zhou et al. 2023 Chen et al. 2023), which include: (1) Exact Match (EM), indicating the proportion of predicted results that exactly match the gold standards. (2) Fingerprint Tanímoto Similarity (FTS), quantifying structural similarities between molecules using Tanimoto similarities of their Morgan fingerprints (Morgan T965). (3) METEOR score, a comprehensive text-based metric considering both exact matches and semantic similarity (Lavie & Agarwal 2007) for the MC task. (4) Root Mean Square Error (RMSE), measuring the square root of the average squared differences between predicted and actual values for the PP-ESOL and PP-Lipo tasks (5) Accuracy (Acc), the ratio of correct predictions for the binary classification tasks (PP-BBBP, PP-ClinTox. PP-HIV, and PP-SIDER). (6) Validity (Valid), the ratio of valid predictions following SMILES grammar and chemical valence rules for tasks with SMILES outputs (NC-I2S, MG, FS, and RS). For all the metrics except RMSE, higher values indicate better performance.
4.3 Main Results
Table D and 2 show the performance of different models on SMolInstruct. We make the following key observations:
(1) Among all the LLMs, our LlaSMol models demonstrate the best performance, under- scoring the effectiveness of the proposed SMolInstruct dataset and fine-tuning. Specifi- cally, compared to the base models (Galactica, Llama 2, Code Llama, and Mistral), LlaSMol models exhibit substantial performance improvements, which highlights the effectiveness of SMolInstruct in enhancing the understanding of molecular representations and the task- related knowledge, and signifies the effective learning chemistry-related tasks by LLMs. Furthermore, LlaSMol substantially outperforms GPT-4 on all the tasks and Claude 3 Opus on most tasks, despite their larger parameter size. LlaSMol also surpasses the two chemistry LLMs namely ChemLLM which is similarly trained on chemistry instruction data. and Molinst. Notably, LlaSMoly.ama 2, which uses the same base model and LoRA setting as Molinst, outperforms it even on the shared training tasks (MC, MG, FS, and RS). This finding highlights the benefits of our dataset.
(2) Our four LlaSMol models show substantial differences in their performance, empha- sizing the considerable impact of base models on downstream tasks. Despite sharing iden- tical training, inference settings, and comparable model sizes, LlaSMolyisttal consistently out- performs LiaSMol 2 by a substantial margin, highlighting Mistral's potential on chem- istry tasks. In addition, LlaSMolcode Llama exhibits better performance than LlaSMol 2
4Since its dataset and evaluation details are not available, cannot provide more analysis.
7

---

Published as a conference paper at COLM 2024
Table Results on molecule captioning (MC), molecule generation (MG), forward synthesis (FS), and retrosynthesis (RS). Metrics EM, FTS, and Valid are in percentage.
MC
MG
FS
R$
Model
METEOR
EM
FTS
Valid
EM
FTS
Valid
EM
FTS
Valid
Tpek-Specific, Non-LLM Based Models
SoTA
0.513
317
73.2
95.3
78.7
100.0
47.0
77.5
99.7
Existing LLMs Without Tuninn on SMolInstruct
GPT-4
0.188
6.4
42.6
81.4
1.6
40.5
87.0
0.0
33.4
42.6
Claude 3 Opus
0.219
12.3
57.6
92.6
3.7
45.7
97.0
1.1
46.2
94.8
Galactica
0.050
0.0
11.6
94.7
0.0
25.9
83.7
0.0
34.6
93.0
Llama 2
0.150
0.0
4.8
0.0
13.7
97.7
0.0
27.5
Code Llama
0.143
0.0
8.5
95.2
0.0
15.8
99.6
0.0
25.3
97.1
Mistral
0.193
9.0
35.9
0.0
19.9
95.8
0.0
24.2
98.0
Molinst (chemistry LLM)
0.124
6.0
43.6
84.8
2.1
31.7
99.8
5.7
48.0
ChemL.IM LM (chemistry LI LLM)
0.050
0..9
14.3
4.3
0.0
1.6
38.5
0.0
2.9
10.9
Our
LlaSMol Series
LlaSMol
0.394
7.7
52.2
99.6
53.1
79.9
99.7
25.7
67.0
99.9
LlaSMol
0.377
47.1
99.6
47.1
76.9
99.8
22.5
65.2
99.9
LlaSMolcode Liama
0.366
6.5
46.6
52.0
66.7
100.0
LlaSMolMirral
0.452
19.2
61.7
99.7
63.3
84.9
99.8
32.9
70.4
100.0
Table 3: Ablation study results on NC and PP. Metrics EM, Valid, and Acc are in percentage. Orange indicates better results than LlaSMolMistral' blue indicates worse.
Model
ESOL
Lipo
EM
EM
Valid
EM
EM
RMSE]
RMSEJ
Acc
Aec
Acc
Acc
LiaSMolaisial
87.9
70.1
99.6
93.2
29.0
1.150
1.010
74.6
93.1
96.7
70.7
w/e canomical
using SEI FIFS
86.9
47.7
100.0
94.7
19.7
1.4%6
1.106
69.5
91.7
96.5
64.4
train on Mol Instructions
0.0
0.0
75.2
0.0
0.0
4.416
2.282
0.0
0.0
2.6
0.4
on most tasks, indicating a potential synergy between programming language knowledge in Code Llama and molecular representations. Furthermore, LlaSMolcalactic outperforms LlaSMolyJama 2, and LlaSMolçode Llama in most cases, suggesting the benefits of pretraining on chemistry-related documents.
(3) Although LlaSMol models do not outperform SoTA models, they demonstrate con- siderable potential for further improvements. Specifically, LlaSMolMistral surpasses the SoTA models on PP-Clintox and PP-SIDER, but has yet to achieve the success on other tasks. However, LlaSMol has greatly narrowed the performance gap between LLMs and SoTA task-specific models, compared to previous efforts (Fang al. 2023 Zhang et al., 2024). Remarkably, LlaSMolyiitrall attains such performance with only a small proportion of its parameters fine-tuned (approximately 41.9M, 0.58% of its parameters). As shown in Appendix F.22 increasing the number of trainable parameters can substantially boost performance, suggesting that LlaSMol has immense potential to surpass task-specific models through more extensive fine-tuning and serve as a strong foundation model for chemistry applications.
4.4 Ablation Study
To investigate the advantages of SMolInstruct, we conduct an ablation study by comparing LlaSMol with the following variants: (1) w/o canonical, which uses uncanonical- ized SMILES, to examine the benefits of canonicalization. (2) using SELFIES, which uses SELFIES Krenn et al. (2019) instead of SMILES to explore their differences.
(3) train Mol-Instructions, w which is trained on Mol-Instructions (Fang et al., 2023), to compare the performance improvements of our dataset against the previously proposed dataset.
The results in Table and Table the following observations: (1) The "w/o canonical" model underperforms LlaSMolMistral most tasks, with substantial performance drop on ES and RS. This suggests that canonicalizing SMILES can reduce learning difficulty and improve performance. As canonicalization can be easily performed using fixed al algorithms
8

---

Published as a conference paper at COLM 2024
Table 4: Ablation study results on MC, MG, FS, and RS, Metrics EM, FTS, and Valid are in
percentage. Orange indicates better results than LiaSMolMistrall blue indicates worse.
MC
MG
FS
RS
Model
METEOR
EM
FTS
Valid
EM
FTS
Valid
EM
FIS
Valid
LaSMolMtttrll
0.452
19.2
61.7
99.7
63.3
84,9
99.8
32.9
70.4
100.0
w/o canonical
0.457
16.8
60.2
99.1
53.7
80.8
99.9
23.8
67.4
99.9
using SELFIES
0.466
16.2
58.6
99.9
40.4
74.0
100.0
25.6
66.0
99.9
train on Mol-Instructions
0.195
6.1
46.1
88.2
3.9
37.1
78.3
7.4
52.6
76.7
before feeding into models, we recommend using canonical SMILES when training and applying LLMs for chemistry. (2) While using SELFIES slightly improves the validity of generated molecules, which aligns with the motivation behind SELFIÈS (Krenn et al. 2019), the validity of using SMILES is also sufficiently high. Moreover, using SELFIES results in worse performance on most tasks, possibly due to SELFIES being typically longer than SMILES, making it more difficult for the model to accurately understand and generate. Therefore, using SELFIES over SMILES may not be necessary, contrast to claims made in previous work (Krenn al. 2019: Fang al. 2023). (3) Despite using identical base models and training settings, the model trained on Mo--nstructions (Fang et al. 2023) performs much worse than LlaSMol trained on SMolInstruct even on the shared tasks (MC, MG, PS, and RS). This demonstrates the superiority of our dataset. A detailed comparison with Mol-Instructions can found in Appendix
To gain deeper insights into the models' performance and behavior, we conduct further analytical experiments: (1) To investigate the synergistic effects among different tasks, we evaluate models trained single task and models with certain tasks removed. The results demonstrate multiple-task training outperforms single-task training, indicating its benefits. However, each task generally does not heavily rely on the presence of other tasks, suggesting a degree of independence among them. (2) To investigate the influence of LoRA (Hu et al. 2022) settings, we vary the involved LoRA modules. We observe that adding LoRA modules (and trainable parameters) leads to a substantial boost in performance, indicates the models' great potential for further improvements if with larger-scale fine-tuning. Please refer to Appendix F for more details.
Conclusion
While LLMs show promise as versatile assistants, their performance on chemistry-related tasks remains notably subpar. To address this issue, we introduces SMolInstruct, a large- scale, comprehensive, and high-quality instruction tuning dataset. comprises 14 tasks highly relevant to real-world applications and contains over 3M rigorously curated samples. Using SMolInstruct, we develop LlaSMol, a series of LLMs for performing chemistry tasks. Our experiments demonstrate LlaSMol's superiority over existing LLMs,, and highlight SMolInstruct's 's crucial role in boosting the performance. Further analytical experiments also provide significant insights towards developing LLMs for chemistry and science.
However, this work has the following limitations. First, the evaluations for the MC and MG tasks cannot accurately assess models' abilities to generate chemically correct descriptions and molecules. Since the definition of molecular descriptions remain ambiguous and the available data is limited, it is challenging to assess whether the generated descriptions or molecules are accurate and correct. Second, this work does not delve into the models' generalization capabilities beyond the trained tasks. While we recognize the importance of such capabilities, how to meaningfully test generalization abilities is nontrivial and needs careful design, which falls outside the purview of this work. Third, our models do not yet outperform SoTA task-specific models, possibly due to the small ratio of trainable parameters or suboptimal training procedures. Nevertheless, we propose a high-quality instruction tuning dataset, demonstrate its effectiveness, and gain deeper insights, which we hope can valuable for future research. We will try to address the aforementioned limitations in our future work.

---

Published as a conference paper at COLM 2024
Ethics Statement
Despite our best efforts to maintain the high quality of the SMolInstruct dataset and the integrity of the LlaSMol models, we cannot guarantee that the dataset free of inaccurate, incorrect, or harmful content, nor can we prevent the models from generating such content. Users should engage with our dataset and models their own discretion and uphold the highest ethical standards in their use.
Acknowledgement
The authors would thank colleagues from the OSU NLP group and the OSU Ning Lab for constructive feedback. This research was supported in part by NSF IIS-2133650, NIH IR01LM014385-01 and NSF CAREER #1942980, as well Ohio Supercomputer Center OOio Supercomputer Center 1987). The views and conclusions contained herein are those of the authors and should not be interpreted representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.
References
Mikhail Andronov, Varvara Voinarovska, Natalia Andronova, Michael Wand, Djork-Arn Clevert, and Jürgen Schmidhuber. Reagent prediction with molecular transformer improves reaction data quality. Chemical Science, 14(12):3235-3244 2023.
Anthropic. The claude 3 model family: Opus, sonnet, haiku. Anthropic, 2024.
Theodore L. Brown. Chemistry: the central science. Pearson, 14th edition edition, 2018.
Andrew R. Burns, Trevor C. Y. Kwok, Al Howard, Ed Houston, Karl Johanson, Anthony Chan, Sean R. Cutler, Peter McCourt, and Peter J. Roy. High-throughput screening of small molecules for bioactivity and target identification in Caenorhabditis elegans. Nature Protocols, 1: 1:1906-1914, 2006.
He Cao, Zijing Liu, Xingyu Lu, Yuan Yao, and Yu Li. Instructmol: Multi-modal integration for building versatile and reliable molecular assistant in drug discovery. arXiv preprint ar Xix::::311 11260,, 2023.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 2024.
Heng-Yi Chen, Michael Hsu, and Chan-Wang Jerry Lio. Micro but mighty Micronutrients in the epigenetic regulation of adaptive immune responses. Immunological reviews, 305: 152-164, 2022.
Ziqi Chen, Martin Renqiang Min, Srinivasan Parthasarathy, and Xia Ning. A deep gener- ative model for molecule optimization via one fragment modification. Nature machine intelligence, 3(12):000--1049 2021.
Ziqi Chen, Oluwatosin R Ayinde, James R Fuchs, Huan Sun, and Xia Ning. G2retro as a two- step graph generative models for retrosynthesis prediction. Communications Chemistry, (1):102, 2023.
Connor W. Coley, William H. Green, and Klavs F. Jensen. Machine learning in computer- aided synthesis planning. Accounts of Chemical Research, 51(5):1221-1288, 2018.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. In Proceedings of Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.
10

---

Published as a conference paper at COLM 2024
Joseph L Durant, Burton A Leland, Douglas R Henry, and James G Nourse. Reoptimization of mdl keys for use in drug discovery. Journal of chemical information and computer sciences, 42(6):1273-1280, 2002.
Carl Edwards, ChengXiang Zhai, and Heng Ji. Text2mol: Cross-modal molecule retrieval with natural language queries, In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 595- 2021.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Transla- tion between molecules and natural language. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 375-3735-53 2022.
Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-issruccions: large-scale biomolecular instruction dataset for large language models. Xiv preprint ar 7Xii:2306.008188 2023.
Henri A. Favre and Warren H. Powell. Nomenclature of organic chemistry: IUPAC recommenda- fions and preferred names 2013. Royal Society of Chemistry, 2014.
Kaitlyn M. Gayvert, Neel S. Madhukar, and Olivier Elemento. A data-driven approach to predicting successes and failures of clinical trials. Cell Chemical Biology, 23(10):1294-3001 2016.
Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. What can large language models do in chemistry? comprehensive benchmark on eight tasks. In Proceedings of Conference on Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023.
Shen Han, Haitao Fu, Yuyang Wu, Ganglan Zhao, Zhenyu Song, Feng Huang, Zhongfei Zhang, Shichao Liu, and Wen Zhang. Himgnn: a novel hierarchical molecular graph representation learning framework for property prediction. Briefings in Bioinformatics, 24 (5):bba3305, 2023.
Kan Hatakeyama-Sato, Naoki Yamane, Yasuhiko Igarashi, Yuta Nabae, and Teruaki Hayakawa. Prompt engineering of gpt-4 for chemical research: what can/canno be done? Science and Technology of Advanced Materials: Methods, 3(1), 2023.
Jiazhen He, Huifang You, Emil Sandström, Eva Nittinger, Esben Jannik Bjerrum, Christian Tyrchan, Werngard Czechtizky, and Ola Engkvist. Molecular optimization by capturing chemist's intuition using deep neural networks. Journal of cheminformatics, 13(1):1-17, 2021.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
National Cancer Institute. AIDS antiviral screen data, 2004. htts:t http///wkiiicccc nih Accessed on Fec 2024.
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 311:001502 2022.
Kevin Maik Jablonka, Philippe Schwaller, and Berend Smit. Is gpt-3 all you need for machine learning for chemistry? In AT for Accelerated Materials Design NeurTP5 2022 Workshop, 2022.
Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerreny and Berend Smit. Leveraging large language models for predictive chemistry. ChemRxio, 2023. doi:
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. aiv preprint ar arXit:2310.0682. 2023.
11

---

Published as a conference paper at COLM 2024
Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem 2019 update: improved access to chemical data. Nucleic acids research, 47:D1102-D1109, 2019. 2019.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Mario Krenn, Florian Häse, A Nigam, Pascal Friederich, and Alán Aspuru-Guzike Selfies: a robust representation of semantically constrained graphs with an example application in chemistry. ar Xio preprint ar arXtc:1905.13441 1(3), 2019.
Michael Kuhn, Ivica Letunic, Lars Juhl Jensen, and Peer Bork. The sider database of drugs and side effects. Nucleic Acids Research, 44:D1075-D1079, 2015.
Alon Lavie and Abhaya Agarwal. Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT '07, pp. 228-231. Association for Computational
Linguistics, 2007.
Elena Lenci and Andrea Trabocchi. Chapter 1 Synthetic approaches toward small molecule libraries. In Small Molecule Drug Discovery, pp. 1-34. Elsevier, 2020.
Sihang Li, Zhiyuan Liu, Yanchen Luo, Xiang Wang, Xiangman He, Kenji Kawaguchi, Tat- Seng Chua, and Qi Tian. Towards 3d molecule-text interpretation in language models. In Proceedings of International Conference on Learning Represenlations (ICLR), 2024.
Youwei Liang, Ruiyi Zhang, Li Zhang, and Pengtao Xie. Drugchat: towards enabling chatgpt-like capabilities on drug molecule graphs. arXiv preprint ar Xrix:2309.03907 2023.
Shengchao Liu, Jiongxiao Wang, Yijin Yang, Chengpeng Wang, Ling Liu, Hongyu Guo, and Chaowei Xiao. Chatgpt-powered conversational drug editing using retrieval and domain feedback. aiv preprint Xix:2305..18900 2023a.
Zhiyuan Liu, Sihang Li, Yanchen Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. MolCA: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 15623-15638, 2023b.
Daniel Lowe. Chemical reactions from us patents (1976-sep2016), 2017. URL https://doi
org/10 /tg/10.6084.0 6084/09.88800 fig.m//r..gga..ee.mm 5104873.v1
Ines Filipa Martins, Ana L. Teixeira, Luis Pinheiro, and Andre O. Falcao. A bayesian ap- proach to in silico brood-brain barrier penetration modeling. Journal of Chemical Information and Modeling, 1266)116861 2012.
Brian W Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure, 402 1975.
Monica P McNerney and Mark P' Styczynski. Small molecule signaling, regulation, and potential applications in cellular therapeutics. Wiley Interdisciplinary Reviews: Systems Biology and Medicine, 10:e1405, 2018.
David L. Mobley and J. Peter Guthrie. Freesolv: database of experimental and calculated hydration free energies, with input files. Journal of Computer-Aided Molecular Design, 28(7): 711-720, 2014.
H. L. Morgan. The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service. Journal of Chemical Documentation, 5(2): 107-113, 1965.
Ohio Supercomputer Center. Ohio supercomputer center, 1987. URL htthhttpp//occeduu/ p19495/f5s1ph73
12

---

Published as a conference paper at COLM 2024
OpenAl. Gpt-4 technical report. arXiv preprint arXio::303.08774,20
Salwa K Poole and Colin F Poole. Separation methods for estimating octanol-water partition coefficients. Journal of Chromatography B, 797(1-2)33-19, 2003.
Kohulan Rajan, Achim Zielesny, and Christoph Steinbeck. Stout: Smiles to iupac names using neural machine translation. Journal of Cheminforrmatics, (3(1):1-14, 2021.
RDKit. Rdkit: Open-source cheminformatics, 2023. URL https::///oott 022.....orr/...55///edddd 8254217 Accessed on 27 Jan 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. ar arviv preprint XtX::::300.1950000 2023.
Nadine Schneider, Roger A Sayle, and Gregory A Landrum. Get your atoms in order: An open-source implementation of novel and robust molecular canonicalization algorithm. Journal of chemical information and modeling, 55(10):2111-2122 2015.
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha Lee. Molecular transformer: a model for uncertainty- chemical reaction prediction. ACS central science, 5(9):1572-1583, 2019. 2019.
T. W. Graham Solomons, Craig B. Fryhle, and Scott A. Snyder. Organic Chemistry, Integrated F-Text with E-Solutions Manual. Wiley, 13th edition, 2022.
Vignesh Ram Somnath, Charlotte Bunne, Connor Coley, Andreas Krause, and Regina Barzi- lay. Learning graph models for retrosynthesis prediction. Advances in Neural Information Processing Systems, 34:9405-9444 9415, 2021.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arviv preprint arXiv:2211.0900 2022.
Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature Medicine, 29(3):1930-1944 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar arXio:2302.13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. ar Xrv preprint arXiv:2307 (iX::2307700988, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Conference on Advances in neural information processing systems (NeurIPS), volume 30, 2017.
Yuyang Wang, Zijie Li, and Amir Barati Farimani. Graph Neural Networks for Molecules. In Chen Qu and Hanchao Liu (eds.), Machine Learning in Molecular Sciences, pp. 21-66. doi: 10.1007/
Springer International Publishing, Cham, 2023. ISBN 978-3-031--3196-77 978-3-031-37196-72 URL httph: org/10. 10071998----111-1111 0000
David Weininger: Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28 (1):31-36, 1988.
Andrew D. White, Glen M. Hocky, Heta A. Gandhi, Mehrad Ansari, Sam Cox, Geemi P. Wellawatte, Subarna Sasmal, Ziyue Yang, Kangæin Liu, Yuvraj Singh, and Willmor J. Peña Ccoa. Assessment of chemistry knowledge in large language models that generate code. Digital Discovery, 2(2):368-376 2023.
13

---

Published as a conference paper at COLM 2024
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State- of-the-art natural language processing. In Proceedings of conference empirical methods in natural language processing: system demonstrations (EMNLP), pp. 38-45, 2020.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9():5133-330 2018.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman-Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay. Analyzing Learned Molecular Representations for Property Prediction. Journal of Chemical Information and Modeling, 59(8)33370-3385 August 2019. ISSN 1549-9596. doi: 10.1021
9b00237. URL Publisher: American Chemical Society.
https://dbi.org// /00/://...1110///11011 /cc. jacp//ccs.cc.................. 9b00237
Geyan Ye, Xibao Cai, Houtim Lai, Xing Wang, Junhong Huang, Longyue Wang, Wei Liu, and Xiangxiang Zeng. Drugassist: A large language model for molecule optimization. Xive preprint ar XrX:::2401. 10334, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. Xiv preprint ar XxXi:23399.05653, 2023.
Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, et al. Chemllm: A chemical large language model. arxiv preprint arXiv:2402.06852. 2024.
Tianshu Zhang, Xiang Yue, Yifei Li, and Huan Sun. Tablellama: Towards open large generalist models for tables. Xiv preprint ar arXiv:2311.09206 2023.
Yangtian Zhang, Huiyu Cai, Chence Shi, and Jian Tang. E3bind: An end-to-end equivariant network for liggad docking. In The Eleventh International Conference on Learning Representations, 2022a.
Zaixi Zhang, Yaosen Min, Shuxin Zheng, and Qi Liu. Molecule generation for target protein binding with structural motifs. In The Eleventh International Conference on Learning Representations, 2022b.
Zipeng Zhong, Jie Song, Zunlei Feng, Tiantao Liu, Lingxiang Jia, Shaolun Yao, Min Wu, Tingjun Hou, and Mingli Song. Root-aligned smiles: a tight representation for chemical reaction prediction. Chemical Science, 13(33))903399034 2022.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: A universal 3d molecular representation learning framework. In International Conference on Learning Representations (ICLR), 2023.
14

---

Published as a conference paper at COLM 2024
Table of Contents in Appendix
A Preliminaries
16
Details of SMolInstruct
16
B.1 The statistics of SMolInstruct
16
B.2 Details of Dataset Construction
17
C Comparison with Mol-Instructions
18
D Details of Experimental Setup
19
D.1 LlaSMol Models
20
D.2 Compared LLMs
20
D.2.1
GPT-4
20
D.2.2
Claude 3 Opus
21
D22.3
Galactica
21
D.2.4
Llama 2, Code Llama, and Mistral
21
D.2.5
Molinst
22
D.2.6
ChemLLM
22
D.3
Task-Specific, NLn Based So IA Models
22
D.3.1
STOUT for NC-12S and NC-S21
22
D.3.2
RDKit for NC-S2F
22
D.3.3
STOUT+RDKit for NC-12F
22
D.3.4
Uni-Mol for All The PP Tasks
22
D.3.5
MolT5 for MC and MG
23
D.3.6
RSMILES for FS and RS
23
D.3.7
Molecular Transformer for FS and RS
23
D.4
Evaluation Metrics
24
Detailed Experimental Results
24
Name Conversion Tasks
24
Property Prediction
27
E.3
Molecule Description
28
Chemical Reaction
28
E.5
Other Common Findings
28
More Analytical Experiments
30
Task lask Synergy
30
Influence of LoRA Modules and Trainable Parameters
15

---

Published as a conference paper at COLM 2024
A Preliminaries
Molecules form the basis of chemistry, which fundamentally determines the properties and behaviors of most substances. A molecule is a group of atoms held together by chemical bonds (Brown 2018). In this paper, focus small molecules, which typically have more than 100 atoms and a low molecular weight under 1,500 Daltons (Lenci & Trabocchi 2020). Small molecules perform many important functions, such as signaling in cellular biology (McNerney & Styczynski, 2018), pest control in agriculture (Burns et al.] 2006), micronutrients in nutrition (Chen al. 2002), and drug therapy in medicine (Lenci & Trabocchi 2020). Given the importance of small molecules, it is essential to integrate LLMs into the study of small molecules to further advance their design or development.
Molecules can be represented in multiple ways, such as SMILES strings, IUPAC names, and molecular formulas. SMILES strings use sequence of symbols to encode the 2D structures of molecules (Weininger, 1988). A molecule can have multiple SMILES strings; canonical SMILES for the molecule is unique and deterministic. For exam- ple, the canonical SMILE ES representation of glucose is "((CICCCCCCC(0110000000099
SELFIES (Krenn et al. 2019) is an alternative representation to SMILES that also uses sequence of symbols to denote molecular structures. Its key advantage is robust- ness, as every SELFIES string is guaranteed to correspond to a valid molecule. The SELFIES representation corresponding to the above SMILES representation of glucose is
Molecular formulas represent a molecule by enumerating the type and number of atoms in the molecule (Solomons et al. 2022). For example, the molecular formula for glucose is "CgHı2O," ". IUPAC names are formal names based on natural language elements, which follow the systematic rules set by the International Union of Preferred and Applied Chemistry (TUPAC) (Favre & Powell 2014). These names are derived from the structures and functional groups of molecules, and are intended to be human-readable. For example, the IUPAC name for glucose is (hyroooyyyyyeeeyyylllaeeee
Molecules are one of the fundamental units of chemistry that participate in reactions (Brown 2018). A reaction is a process which converts input molecules (reactants) into output molecules (products) through the breaking and forming of chemical bonds. Other molecules (reagents) may be present to enhance facilitate the reaction.
B Details of SMolInstruct
In this section, we introduce the details of our proposed dataset SMolInstruct, including statistics and construction details.
B.1 The statistics of SMolInstruct
Table (olee5)sovve shows the statistics of SMolInstruct. It contains types of altogether 14 tasks, which are selected to be meaningful and useful. There are about 3.3M samples, and each of them is a distinct sample. In other words, there does not exist a pair of samples who share the same chemical information (i.e., the core input and output information, such as input molecules and output molecules), but with the same or different natural language templates (i.e., the task description in the query and the sentence templates in the response). When needed, one can easily create more instruction tuning samples by combining one piece of chemical information with multiple natural language templates. Áll in all, SMolInstruct can serve as good benchmark for training and evaluating LLMs on various chemistry tasks.
To know more about the diversity of SMolInstruct, we conduct a statistics on the molecules. Altogether, there exist 1.6M distinct molecules, and several important statistical values are shown in Figure 2 Specifically, Bertz complexity is a topological index that measures the complexity of molecules based on the number and types of bonds and atoms. Atom count shows the number of atoms in a molecule, and it represents the size of a molecule. Molecular weight is the sum of the atomic weights of the atoms in molecule. And ring count shows
16

---

Published as a conference paper at COLM 2024
Table 5: The statistics of SMolInstruct. "Qry." and "Resp." are average lengths of queries and responses, respectively.
Task
Task abbr.
#Train
#Valid
#Test
#All
Qry.
Resp.
Name Conversion. Data Source:
PubChem
IUPAC to Molecular Formula
NC-12F
300, 000
1,497
2, 999
304, 490
84
IUPAC to SMILES
NC-12S
299,890
1,496
1999
304, 379
82
SMILES to Molecular Formula
NC-S2F
299, 890
1,496
999
304, 379
68
26
SMILES to IUPAC
NC-S21
290,890
1,496
.993
304, 379
Property Prediction. Data Source: MoleculeNet
ESOL
PP-ESOL
888
111
112
1,111
43
22
Lipo
PP-Lipo
3,360
420
420
4,200
80
11
BВBP
PP-BBBP
1,569
196
197
1,962
68
11
Clin lox
PP-ClinTTox
1,4
143
144
1,431
69
11
HIV
PP-HIV
32, 864
4, 104
4, 107
41,075
63
11
SIDER
PP-SIDER
22,820
2,860
2, 860
28,540
82
11
Molecule Description. Data Source: Mol-Instructions, ChEBl-20
Molecule Captioning
MC
56, 498
1,269
2,
60,305
83
102
Molecule Generation
MG
56, 498
1,269
493
60, 260
117
75
Chemical Reaction. Data Source: USPTO-full Forward Synthesis
FS
971,
049
062
977, 920
Retrosynthesis
RS
941,735
2,092
156
947, 983
77
Overall
3, 288, 855
20, 498
33, 061
3, 3,24, 414
83
55
2010
330
100
4004
36.0
3.0
(a) Bertz complexity.
(b) Atom count.
(c) Molecular weight.
(d) Ring count.
Figure 2: The statistics of molecules in SMolInstruct, with the long tail parts removed for clear presentation.
the number of rings in the molecular structures. As see, the values varies much, showing extensive coverage in terms of complexity, size, and structure. Notably, when compared to Mol-Instructions (Fang et al. 2023), molecules in SMolInstruct show higher complexity and diversity, which indicates that SMolInstruct is more comprehensive and complicated than Mol-Instructions. The scale, complexity, and diversity of SMolInstruct makes it well-suited for learning chemistry LLMs.
B.2 Details of Dataset Construction
Dataset construction involves four key steps (Section 3.2): data collection, quality control, data splitting, and instruction creation. This section provides task-specific details, omitting the common steps of canonicalizing SMILES/SELFIISS and verbalizing information into query and response sentences, which have been introduced in Section 3.2
Name Conversion (NC). The raw data for name conversion is collected from PubChem (Kim et al. 2019). Approximately 300k from the database, and their SMILES, IUPAC names, and molecular formulas are extracted. Entries with incomplete missing information in these three domains discarded. Finally, the SMILES, IUPAC names, molecular formulas are paired create samples for the four name conversion tasks.
entries are randomly selected
17

---

Published as a conference paper at COLM 2024
Property Prediction (PP). The raw data for property prediction is sourced from MoleculeNet (Wu et al. 2018). Out of its 16 core dataset we select 6 that are only related to small molecules and are useful especially in drug discovery. Answers for regression tasks (e.g. ESOL and Lipo) are formulated as strings of numbers, and answers for binary classification tasks (e.g,, BBBP and SIDER) are formulated as "Yes" or "No".
Molecule Captioning (MC) and Molecule Generation (MG). The raw data is collected from ChEBI-20 (Edwards et al. 2022) and Mol-Instructions (Fang et al. 2023). Despite the large number of samples in Mol-Instruction, many are found to be of low quality. For example, numerous molecular descriptions end with the ambiguous phrase "with data available", while others are overly general, making it difficult to generate a specific molecule based on the description. To ensure data quality, regular expressions and heuristic rules are employed to filter out low-quality samples.
Forward Synthesis (FS). USPTO-full (Lowe, 2017), one of the most comprehensive chemical reaction datasets, serve as the data source. The following processing steps are performed to clean the data: (1) Reactants and reagents are combined as input, and the product(s) serve as output, consistent with other datasets such as Mol-Instructions (Fang et al. 2023).
(2) Duplicate chemicals in both input and output are removed to avoid redundancy. (3) If a chemical appears in both input and output, it is removed from the output to maintain data integrity. (4) Products in the output containing fewer than 5 molecules are considered non-main products and excluded. (5) If the above steps result in an empty output, the entire sample is discarded.
Retrosynthesis (RS). The data is also sourced from USPTO-full (Lowe) 2017], with the product as input and the reactants (excluding reagents) as output. During data exploration, we observe instances where reactants are mislabeled as reagents and vice versa. To address this issue, we compare the atom mapping numbers of the reactants and reagents with the products and relabel them accordingly. Subsequently, we apply the following processing steps: (1) Duplicate chemicals in both input and output are removed.
(2) If chemical appears in both input and output, it is removed from the input. (3) Products in the input containing fewer than 5 molecules are excluded. (4) In cases where multiple products exist in the input, the reaction is split into multiple samples, with each product serving the input once. (5) If the above steps result in an empty input, the entire sample is discarded.
For all the tasks, samples containing invalid SMILES strings (i.e., those that cannot be parsed into a valid molecule with (DKit(RDKiI 2023)) are discarded, and duplicate samples are removed to avoid redundancy. Finally, since some molecules contain multiple components and they are separated by dots in SMILES, which is the same delimiter used to separate different reactants/ reagents/ / products in FS and RS, the dots in SMILES strings for NC, PP, MC, and MG are replaced with semicolons to differentiate between these two usages.
C Comparison with Mol-Instructions
In this section, we present comprehensive comparison between our work and Mol- Instructions (Fang et al. 2023).
We begin by comparing our dataset, SMolInstruct, with the Mol-Inntruccions dataset. While Mol-Instructions covers a broader scope (including molecule-oriented protein-oriented, and biomolecular text instructions), SMolInstruct focuses exclusively on small molecules, providing a deeper and more comprehensive exploration of this domain.
If focusing on the molecule-related data, as shown in Table 6 SMolInstruct is a larger, more comprehensive, and higher-quality dataset. It incorporates more tasks, samples, and molecular representations, and involves more careful curation. Both datasets share the tasks of MC, MG, FS, and RS. Although SMolInstruct has fewer samples for MC and MG, the included samples are of higher quality (see Appendix B.2). Furthermore, SMolInstruct contains substantially more samples for FS and RS, which have been carefully cleaned and processed. Additionally, SMolInstruct incorporates four NC tasks to facilitate the
18

---

Published as a conference paper at COLM 2024
understanding of various molecular representations. Unlike Mol-Instructions, we do not include the reagent prediction task mainly due to the lack of sufficient high-quality data (Andronov et al. 2023) and the limited practicality of this task in real world applications.
Beyond the dataset, our work makes contributions to the exploration of chemistry LLMs. While Fang al. (2023) primarily focus the dataset itself and provide preliminary exploration of the models, we conduct comprehensive experiments to investigate the abilities of LLMs in the chemistry domain. Our experiments Section 4 demonstrates that our LlaSMol models achieves superior performance compared to the LLMs trained on Mol- Instructions and the strongest LLMs such GPT-4 and Claude 3 Opus, greatly diminishing the gap between LLMs and SoTA task-specific models. Moreover, we provides valuable insights about multi-task training, LoRA (Hu et al. 2022) settings, and other aspects that could be helpful for future research in this field.
Table 6: Comparison between Mol-Instructions (the molecule-oriented part) (Fang et al. 2023 and our SMolInstruct.
Mol-Instructions
SMolInstruct (ours)
name conversion
No this task.
1.2M samples
property prediction
/ 362.1k samples
78.3k samples 6 useful
HOMO/LUMC energy.
properties.
Tasks
molecule captioning
298.34 33k samples.
60.3k samples.
molecule generation
298.3k samples.
/ 60.3k samples.
forward synthesis
125. 4k samples.
/977.9k samples.
retrosynthesis
129.7k samples.
/ 948.0k samples.
reagent prediction
125.4k samples.
Not included due to its insuf-
ficient data and limited practi- cality.
Distinct samples
1.2M
3.3M
#Samples
1.3M
3.3M
Molecular representations
SELFIES.
SMICES SMILES (delault) and
SELFIES, also involves IUPAC names and molecular formula in the NC tasks.
Dātā splitting
Provides test test"set, while train/ train/validation validation sets
Carefully
split into
train/ valiin/niittion/// test set, removing potential data leakage (see Section 3.2)
are not explicitly split.
CCanonialiiation"
No.
Yes, all the SMICES7SELFIBS representations
canonical- ized, providing a standardized data format.
Complexity and diver=
lower.
Higher (see Appendix B.1
sity of molecules
D Details of Experimental Setup
In this section, we introduce the details of our experimental setups, including the training and inference details of our LlaSMol models and the compared models. We also give detailed explanations of the metrics used in Section 4.3 as well extra metrics that we will use in Appendix
19

---

Published as a conference paper at COLM 2024
D.1 LlaSMol Models
The base models used for developing LlaSMol are Galactica 6 Taylor et al. 2022), Llama 8 (Roziere et al. 2023 and Mistral (Jiang et al. 2023). We conduct instruction tuning on our SMolInstruct, and the resulting models are called named as AlaSMolGalactioa LlaSMol 2, LlaSMolcode Llamar and LlaSMolMistral respectively. Expect for being based on different base models, their training and evaluation configurations are identical, as described as follows.
27 (Touvron et al. 2023b), Code Llama
We used LoRA (Hu al. 2022) during training, which applied to all linear layers in the self-attention and FFN modules with lora_r and lora_alpha set to 16. With the 8-bit W optimizer, a learning rate le-4, and a cosine scheduler, we train each model for three epochs. The input length is set to 512, and sequences longer than 512 are truncated.
During inference, we adopt beam search as the generation strategy for simplicity. Due to the need of evaluations the top-k predicted answers (as in Appendix where k varies for different tasks, we generate different numbers of sequences for different tasks by setting the num return sequences argument in the Huggingface Transformers library (Wolf et al. 2020). Specifically, it is set to for NC-12S, NC-S2I, FS, and MG; for NC-12F and NC-S2F; for all the PP tasks; and 10 for RS. The beam size is set to num num_return__ _ecuunnse 3 for all the tasks. The maximum number of new generated tokens is set to 1024.
D.2 Compared LLMs
We introduce each of the compared LLMs in details, including their training (if applicable) and inference process.
D.2.1 GPT-4
General Template
You are an expert chemist. Given the SMILES representation of reactants and reagents, your to predict potential product using your chemical reaction knowledge.
The input contains both reactants and reagents, and different reactants and reagents separated
Task-Specific Template
reply should contain only predicted product and no other text. Your reply must be valid and chemically reasonable.
representation of
Reactants and reagents SMILES: C1Cc001 (rtelecccc..ccceoccccc///ap/1110
ICL
Product SMILES: (GG0r//wweeeeeneeec.cc0000000000 C2C1
Reactants and reagents SMILES: C0N.NN1-0CC---100
Question
Product SMILES:
Figure An example of query template for GPT-4.
GPT-4 (OpenAl 2023) is one of the SoTA LLMs. We use the model versioned as gpt-4-0613 and evaluate on 500 samples from SMolInstruct test set via OpenAl's API. Since GPT-4 is not fine-tuned on our dataset and thus is not familiar with the flexible queries, to ensure generates answers in an expected format, we follow the prompt format proposed in (Guo et al.) 2023) and create query template for each of the tasks. The template for FS is shown in Figure 3 It contains 4 parts: (1) General template describes the task in a general way. (2) Task-specific template describes the detailed content requirements and format
https::/hugginngfac.ccm http://igggaaae..ccccceeo/////// cotfaacebbo///saatticco-- 7b
hf
20

---

Published as a conference paper at COLM 2024
requirements for the specific task. (3) ICL contains the in-context learning examples. It pro- vides examples in the format of cinput.title:, <output content>> (n, where <input title> and <output title> serve straightforward prompts to the input and output content. This design make the queried task more clear. (4) Question has the same format as ICI, with <output_contentt being empty for the model to generate.
<output.title>>
We conduct both s-shot evaluations, where examples. For 0-shot evaluation, the ICL part in the template is removed from the queries. In k-shot evaluation, for each sample, the ICL examples are randomly selected from the training set. The results of these settings are shown in Appendix E which reals that these settings' performance not consistent across all the tasks. Since 0-shot shows the best
1,3,5 is the number of provided ICL
performance on most tasks, we report its results in Section 4.3
In the evaluations, we use the default generation strategy set in the API. To generate the same number of results for each sample (as described in Appendix we set the argument n in the API, which controls the number output sequences.
GPT-4 can always follow the formatted instructions introduced above, so we do not bother to extract the answers from its outputs, but directly use its outputs as the predicted answers.
D.2.2 Claude 3 Opus
Claude 3 Opus (Anthropic 2024) is newly proposed SoTA LLM date. Similarly to GPT-4, we evaluate Claude 3 Opus on 500 samples from SMolInstruct test set via Anthropic's s API, and the generation strategy is the default one, The used prompt format is identical the one used for GPT-4 (Appendix D.2.1. For each sample, we generate one response. Since Claude Opus can always follow the formatted instructions, we do not bother extract the answers from its outputs, but directly use its outputs as the predicted answers.
D.2.3 Galactica
Galactica (Taylor et al. 2022 a LLM without instruction tuning. To evaluate on SMolIn- struct, we follow the instructions in the paper (Taylor et al. 2022) and the repository I0 to create the queries for each task. We use zero-shot setting, as its official instruction does not suggest using few-shot setting. The generation configuration is set identical to that of our LlaSMol models (Appendix[D.] [.I].
Galatica's 's outputs may contain extra text other than the expected answers. Therefore, with heuristic rules and regular expression matching, we implement a program to extract the answers from the outputs of the models. Since the extraction cannot possibly cover all the possible output formats, some answers might not be correctly extracted, which might lead to validities lower than the actual value.
D.2.4 Llama 2, Code Llama, and Mistral
For our base models (Llama 2, Code Llama, and mistral), since they are not trained on SMolInstruct and have not seen the diverse queries in the dataset, we use the same query templates as those used for GPT-4 (Appendix D.2.1). We use the one-shot setting for them, as it would improve models' abiltity to follow the instructions and generate answers more formated way. In addition, the generation configuration (including beam size, output sequence numbers, etc) set identical to that of our LlaSMol models (Appendix(D.)) (.1).
Although we try our best to make the output format as clear possible in the queries, these three models still cannot follow the instructions and their outputs are in various formats. By heuristic rules and regular expression matching, we implement program to extract the answers from the outputs of each of the models. Since the extraction cannot possibly cover all the possible output formats, some answers might not be correctly extracted, which might lead to validities lower than the actual value.
https: h/tp:::://gttuut httpp///sgituuuucccppppppeerieee hhtp:/pppprswiitcce///egggee/eee
21

---

Published as a conference paper at COLM 2024
D.2.5 Molinst
Molinst is Llama 2 model fine-tuned on Mol-Instructions by Fang et al. (2023). On the shared tasks between Mol-Instructions and SMolInstruct (including MC, MG, FS, and RS), we directly use the query templates from Mol-Instructions to achieve better results. On other tasks, we create one query template for each task following the style of Mol-Instructions We use zero-shot on its evaluation, as Mol-Instructions does not contain any few-shot use cases. We directly use Molinst's checkpoin 11 and evaluate it on SMolInstruct, which is fair comparison, because SMolInstruct's ' test set does not contain any training examples from the training set of Mol-Instructions.
The outputs of Molinst may also contain extra text other than the expected answers, espe- cially on its unseen tasks. Thus, we also implement a program to extract the answers. Since the extraction cannot possibly cover all the possible output formats, some answers might not be correctly extracted, which might lead to their validities lower than the actual value.
D.2.6 ChemLLM
CL (Zhang et al. 2024 is an LLM simultaneously proposed with ours. We apply their released checkpoin and evaluate it on SMolInstruct. The used prompt format is identical to the one used for GPT-4 (Appendix D.2.1 For each sample, we generate one response.
The outputs of ChemLLM may contain extra text other than the expected answers. Thus, implement program to extract the answers from its outputs. Since the extraction cannot possibly cover all the possible output formats, some answers might not be correctly extracted, which might lead to their validities lower than the actual value.
D.3 Task-Specific, Non-LLM Based SoTA Models
D.3.1 STOUT for NC-12S and NC-S21
STOUT is encoder-decoder model trained on SMILES and IUPAC name paired data, and is capable of conducting the NC-2IS and NC-S2I tasks. Due to the lack of training code, we cannot re-train it on our dataset, and directly it may have encounter some test samples of SMolInstruct during training, the evaluation results in Table I may be higher than its real performance.
their released model checkpoin Since
D.3.2 RDKit for NC-S2F
The NC-S2F task can be easily achieved with a fixed algorithm by parsing the input SMILES representation and counting the numbers of atoms. We implement program with RDKit (a widely used Python toolkit for processing molecules and other chemical information) and report its results.
D.3.3 STOUT+RDKi for NC-12F
Since there
dedicated models for the NC-I2F task, combine STOUT for the IUPAC to SMILES conversion and RDKIT for the SMILES molecular formula conversion. Specifically, we feed the input TUPAC name into STOUT to get the corresponding SMILES, and then used the RDKit-based program to get the molecular formula based on the SMILES.
D.3.4 Uni-Mol for All The PP Tasks
Uni-Mol Zhou et al. 2023 is a framework for learning useful representations of molecules based on their 3D contormations. Uni-Mol can be fine-tuned to perform property prediction
12 httpss//hhggingffee..cc////eeeet
22

---

Published as a conference paper at COLM 2024
based on these representations. Using the pretrained model weights, hyperparameters, and code supplied by the authors, we fine-tuned Uni-Mol models for chemical property prediction tasks our dataset. For SIDER property prediction, of targets for multi-target classification, as our dataset focused on a specific subset of 20 SIDER targets. We generated results from Uni-Mol using the code provided by the authors and evaluated according to the metrics in Section 4.2 The data split used for fine-tuning, validation, and testing property prediction tasks is different from the one used in the Uni-Mol paper, so the performance may not match exactly.
used 20 the number
D.3.5 MolT5 for MC and MG
MolT5 [Edwards et al. 2022 is T5 model for translating between molecules and natural language. We use the already fine-tuned Molge- checkpoints provided by the authors for both molecule generation and molecule description. We generate predictions on our test set using beam search with 5 beams, following the example code provided by the authors. For input, the molecule description model is provided a SMILES string and the molecule generation model is provided natural language description. For molecule description, we generated only one result. For molecule generation, we set the number of sequences to return to 5. We evaluate our test set results according to the metrics in Section 4.2 The data used for testing is different from the data used by the MolT5 paper, so the performance may be different. Please note that our test set does not overlap with the MolT5 training set.
D.3.6 RSMILES for FS and RS
RSMILES [Zhog et al., 2022) is a transformer model trained on pairs of SMILES strings aligned to minimize their edit distance. RSMILES translates aligned SMILES strings of reactants and reagents into products for the FS task, and products into reactants for the RS task. Following the settings described in the paper of RSMILES, we augment and align each pair of SMILES strings our training data for 5 times. For the FS task, we adopt a "mixed" setting and append canonical SMILES strings of reagents to the end of aligned reactant SMILES strings. We train two RSMILES models for the FS and RS tasks, respectively, using the hyper-parameters provided in their GitHub repository. After training, we average the last 5 checkpoints to get the final checkpoint for each task. During inference, we augment each input SMILES strings for 5 times. We generate 10 output SMILES strings for each augmented input using beam search, resulting in a total of 50 SMILES strings for each test reaction. We get the final top 10 predictions for each task by aggregating these 50 predictions using their provided scripts.
The performance of our re-trained RSMILES model on our dataset for the RS task is com- parable with those reported in their paper on the USPTO-full dataset. Please note that the performance of our re-trained RSMILES for the FS task, as shown in Table is lower than the reported results on the USPTO-MIT dataset for the FS task in their paper. This is due to that our dataset for the IS task is more challenging than the USPTO-MIT dataset used in the RSMILES's paper, due to the inclusion of stereochemical information.
D.3.7 Molecular Transformer for FS and RS
Similar to RSMILES, Molecular Transformer (Schwaller et al., 2019) is also transformer model trained on pairs of SMILES strings, and translate from reactants and reagents into products or products into reactants. While the original Molecular Transformer only focused on the FS task, we train and test it on both the FS and RS tasks. We use canonical SMILES strings of molecules without data augmentation as the training data of Molecular Trans- former. We train two Molecular Transformer models separately for the FS and RS tasks using the hyper-parameters provided in their GitHub repository. During inference, we generate 10 output SMILES strings for each canonical input SMILES string using beam search. The performance of our re-trained Molecular Transformer model on our dataset for the FS task is comparable with those reported in their paper on the USPTO-STEREO dataset.
23

---

Published as a conference paper at COLM 2024
D.4 Evaluation Metrics
We introduce the metrics used in Section 4.3 as follows:
Exact match (EM). It measures the success of a model in providing responses that perfectly match the reference ground truth answers. Notably, for each predicted result, we compare it with the gold answers of the samples that have the same input as the sample. If there exists match, it is counted correct and contributes to the ratio of this metric. Note that for different types of outputs, we employ different criterion for judging it they match. For tasks where outputs are SMILES strings (NC-12S, MG, FS, and RS), we parse SMILES strings into molecules, and they are matched only if the two molecular formula, two matched
molecules are identical. For tasks where outputs if they represent the same set of atoms, and the corresponding numbers of the samples are identical. For tasks where outputs are IUPAC names (NC-S21), since IUPAC names may contain multiple parts separated by semicolons, we compare the set composed of these parts. That is, we do not care about the orders of these parts and how many of parts are there in the generated string, but judge by the correctness of the unique parts.
an important metric type commonly used in cheminformatics. It measures the structural similarity between molecules. The one we report in Section 4.3 is one of this type, called Morgan FTS, which leverages Morgan method to calculate the fingerprint (Morgan 1965).
Fingerprint Tanimoto Similarity (FTS).
METEOR score. It is a common metric used to measure the similarity between text.
Lavie & Agarwal 2007)
RMSE. is a common metric to measure the distance between predicted values and the gold values on regression tasks. Smaller is better.
Acc. It represents the ratio of correct predictions.
Validity (Valid): reports the ratio of valid predicted SMILES representations that can
be successfully parsed into a molecule. It is calculated among all the generated outputs that contain extractable answer part. If an output refuses to answer a question, would not be counted for calculating the validity.
Additional metrics used in Appendix are briefly introduced as follows:
Top-k Exact Match: It is the same as EM discussed before, but on the top-k generated outputs. It gives a more comprehensive results.
MACCS FTS and RDK FTS: In addition to Morgan FTS we use in the previous sections, we introduce two extra FTS metrics, namely MACCS FTS and RDK FTS, that use MACCS (Durant et al.) 2002 and RDK (Schneider et al. 2015 methods to calculate the fingerprint
respectively.
BLUE scores and ROUGE scores: Another types of textual based metrics that measures
the similarity between text.
Matthew's Correlation Coefficient (MCC). Applied in the binary classification tasks (PP-BBBP, PP-Clintox, PP-HIV, and PP-SIDER), this metric provides a balanced measure of the quality of binary classifications (Mtthhews) 1975).
F1 Score: The harmonic mean of precision and recall; a commonly used metric for classification tasks.
E Detailed Experimental Results
In this section, we present more comprehensive experimental results and analysis. We will provide detailed performance of all models across wider range of metrics and offer more discussion of the findings. We will discuss about each type of tasks in the rest of this section.
E.1 Name Conversion Tasks
The results on the four NC tasks are presented in Table Table Table 9 and Table 10 On these tasks, Although open source LLMs including Llama 2, Code Llama, and Mistral can sometimes achieve fairly good validity, showing that they know at least basic knowledge of the molecular representations, they correctly predict none of the samples. It indicates that
24

---

Published as a conference paper at COLM 2024
their lack of knowledge to do reasoning for the conversions. Trained scientific corpus (inchuding chemistry materials), Galactica can achieve some correct predictions, but the performance is still low. After fine-tuning these open source LLMs, LlaSMol models are substantially better than other LLMs, showing the benefits of our dataset and fine-tuning. Compared to the SoTA models, LlaSMol models still underperform, but approach them NC-12F, NC-I2S, and NC-S2F. However, on NC-S2I, LlaSMol is still far from SoTA. It seems that NC-S2I is the hardest task among the four NC tasks, as the SoT task-specific method, STOUT, only achieves 56.5% accuracy. Ability on this task suggests level of understanding of the functional groups in the IUPAC specification, as well representations. Improving models' understanding and generation for IUPAC names might be necessary.
understanding of SMILES
Table 7: Overall results (%) of NC-12E
EM
Model
Validity
Top
Top 3
STOUT+RDKit
97.9
100.0
GPT-4 (0-shot)
8.7
16.4
98.4
GPT-4 (1-shot)
10.8
19.9
98.6
GPT-4 (3-shot)
9.1
17.7
98.6
GPT-4 (5-shot)
11.5
20.4
98.8
Claude 3 Opus
34.6
98.2
Galactica
9.1
12.3
100.0
Llama
0.0
0.1
96.7
Code Llama
0.0
0.0
98.5
Mistral
0.0
0.2
98.5
Molinst
0.0
0.0
1.7
ChemLLM
0.8
97.4
LlaSMolçalactia
83.2
91.3
100.0
LlaSMol Jamma
73.8
86.1
100.0
LlaSMol Llama
75.4
87.0
100.0
ListMol
87.9
93.2
100.0
Table 8: Overall results (%) of NC-12S.
Exact Match
FTS
Model
Validity
Top 1
Top 3
Top 5
MACCS
RDK
Morgan
STOUT
73.5
99.9
99.8
99.5
99.4
GPT-4 (0-shot)
3.3
5.1
5.9
77.6
52.0
49.4
84.2
GPT-4 (1-shot)
3.3
5.7
6.9
76.5
49.6
48.1
85.8
GPT-4 (3-shot)
3.6
5.9
6.9
76.5
48.8
46.9
84.4
GPT-4 (5-shot)
2.4
4.7
6.1
75.6
47.5
46.2
84.8
Claude Opus
17.7
88.5
70.0
68.6
90.2
Galactica
9.7
11.1
12.5
81.5
58.1
53.4
95.6
Llama 2
0.0
0.0
0.0
29.4
18.7
11.3
18.3
Code Llama
0.0
0.0
0.0
30.7
20.0
12.0
81.0
Mistral
0.0
0.0
0.0
33.6
21.3
11.3
40.3
Molinst
0.0
0.0
0.0
43.9
25.1
18.4
96.2
ChemLLMM
0.3
0.0
0.0
0.0
3.9
LlaSMol
58.7
68.8
72.6
95.5
86.4
84.9
99.4
LlaSMol
46.6
57.5
60.5
92.4
79.3
78.3
99.0
LlaSMol Llama
49.9
60.1
63.8
93.1
80.9
80.0
99.3
LlaSMol
70.1
77.8
80.1
96.6
90.1
89.1
99.6
25

---

Published as a conference paper at COLM 2024
Table 9: Overall results (%) of NC-S2F.
EM
Model
Validity
Top
Top 3
RDKit
100.0
100.0
GPT-4 (0-shot)
4.8
11.6
99.8
GPT-4 (1-shot)
3.4
9.2
100.0
GPT-4 (3-shot)
3.4
9.0
99.8
GPT-4 (5-shot)
3.0
8.0
99.4
Claude Opus
9.2
100.0
Galactica
0.0
0.0
99.9
Llama 2
0.0
0.0
72.1
Code Llama
0.0
0.0
97.1
Mistral
0.0
0.1
87.8
Molinst
0.0
0.0
19.3
ChemLLM
0.0
25.8
LlaSMoocalactiia
91.2
95.0
100.0
LlaSMol
87.0
93.1
100.0
LlaSMolçode Llama
88.6
94.7
100.0
LlaSMolMistral
93.2
96.5
100.0
Table 10: Overall results (%) of NC-S2I.
EM
Model
Top
Top 3
Top 5
STOUT
56.5
GPT-4 (0-shot)
0.0
0.0
0.0
GPT-4 (1-shot)
0.0
0.0
0.0
GPT-4 (3-shot)
0.2
0.2
0.2
GPTT-4 (5-shot)
0.2
0.2
0.2
Claude 3 Opus
0.0
Galactica
0.0
0.0
Llama 2
0.0
0.0
0.0
Code Llama
0.0
0.0
0.0
Mistral
0.0
0.0
0.0
Molinst
0.0
0.0
0.0
ChemLLM
0.0
Galactiaaa
18.3
31.8
36.8
LlaSMol
12.9
23.2
26.6
LlaSMol Llama
15.5
26.2
30.5
LlaSMol
29.0
45.3
50.5
26

---

Published as a conference paper at COLM 2024
E.2
Property Prediction
Table 11: Overall results (RMSE) of PP-ESOL and PP-Lipo.
Model
ESOL
Lipol
Uni-Mol
0.819
0.612
GPT-4 (0-shot)
2.570
1.545
GPT-4 (1-shot
2.268
1.625
GPT-4 (3-shot)
2.027
1.777
GPT-4 (5-shot)
1.689
1.592
Claude
1.036
1.194
Galactica
4.184
2.979
Llama 2
3.287
1.634
Code Llama
3.483
1.733
Mistral
3,079
1.730
Molinst
4.304
2.800
ChemLLM
6.635
2.499
LlaSMolGalacticn
1.959
1.213
LlaSMol Jama
2.791
1.338
LlaSMolçode Ilama
2.959
1.203
LlaSMol
1.150
1.010
The results on the two regression tasks (PP-ESOL and PP-Lipo) are presented in Table 11 LISSMolyistaal outperforms other LLMs including GPT-4 by large margin, showing the effectiveness of the fine-tuning. However, it is slightly worse than Claude Opus PP-ESOL, possibly because of its much larger scale of parameters and training data. The task-specific method (Uni-Mol) still outperforms all of the LLMs. This might be attributed to the fact that Uni-Mol (Zhou et al. 20023) possesses 3D structure knowledge, which may be useful in predicting molecular properties.
Table 12: Overall results (Acc) of PP-BBBP, PP-ClinTox, PP-HIV, and PP-SIDER.
Model
BBBP
ClinTox
HIV
SIDER
Uni-Mol
85.3
92.4
97.0
70.0
GPT-4 (0-shot)
62.9
50.0
59.6
57.6
GPT-4 (1-shot)
66.0
25.7
39.4
43.2
GPT-4 (3-shot)
60.9
36.1
48.0
39.8
GPT-4 (5-shot)
57.9
33.3
55.8
50.4
Claude Opus
75.1
41.7
76.4
67.0
Galactica
69.0
92.4
96.7
68.1
Llama
58,9
45.1
93.3
61.9
Code Llama
58.9
85.4
91.8
60.2
Mistral
40.6
15.3
7.1
38.1
Molinst
60.9
6.3
4.5
52.4
ChemLLM
22.3
75.7
72.9
32.6
LlaSMol
69.0
93.1
96.7
70.1
LlaSMol Iamma
69.0
92.4
96.7
68.7
LlaSMol Llama
69.0
93.1
96.7
69.9
LlaSMolyistal
74.6
93.1
96.7
70.7
The results for the four classification tasks (PP-BBBP, PP-Clintox, PP-HIV, and PP-SIDER) pre- sented in (Table 12) show similar information. Particularly, on PP-SIDER task, LlaSMolyistaal outperforms all the LLMs and the task-specific model Uni-Mol, which highlights the poten- tial of LLMs in understanding molecules and predicting their properties. The reason why
27

---

Published as a conference paper at COLM 2024
LlaSMol models get the same results PP-HIV is that the dataset is imbalanced, and the models all predict most samples as negative samples.
E.3 Molecule Description
For the MC task (Table 13,, LlaSMolMistral is the best performing LLM on all metrics. It is still outperformed by the SoTA task-specific model (MolT3), however, approaching close to it. Please note that these text-based metrics only measures the similarity the gold descriptions, and it does not necessary mean the correctness of the description on chemistry dimension. Limited by the current resources, we cannot obtain the correctness measures, and will leave this to our future work.
Table 13: Overall results of MC.
Model
BLEU-2
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
MolT5
0.462
0.366
0.563
0.398
0.501
0.515
GPT-4 (0-shot)
0.095
0.020
0.238
0.058
0.156
0.188
GPT-4 (1-shot)
0.166
0.061
0.295
0.099
0.211
0.206
GPT-4 (3-shot)
0.202
0.092
0.333
0.132
0.250
0.244
GPT-4 (5-shot)
0.214
0.103
0.346
0.146
0.267
0.258
Claude Opus
0.114
0.030
0.263
0.069
0.173
0.219
Galactica
0.018
0.002
0.061
0.012
0.052
0.050
Llama 2
0.110
0.047
0.251
0.107
0.209
0.150
Code Llama
0.106
0.052
0.247
0.122
0.216
0.143
Mistral
0.146
0.068
0.281
0.118
0.232
0.193
Molinst
0.028
0.020
0.226
0.160
0.217
0.124
ChemLLM
0.012
0.004
0.057
0.005
0.048
0.050
LlaSMolGalactica
0.314
0.225
0.456
0.289
0.402
0.394
LlaSMol
0.302
0.213
0.447
0.281
0.396
0.377
LlaSMol Llama
0.322
0.227
0.441
0.273
0.390
0.366
LlaSMol
0.414
0.319
0.521
0.357
0.463
0.452
For the MG task (Table 14p, LlaSMolMistaal is the best performing LLM on all metrics. Since this task aims to generate an arbitrary molecule that fits the given description, it not required to exactly match the gold one, and the FTS metrics may be better to reflect models' ability. However, similar to MC, the metrics only measure the similarities between predicted and gold ones, and not necessarily means the correctness of the generation. We will leave this to our future work.
E.4 Chemical Reaction
For the FS task (Table 15), LlaSMolMistaal is the best performing LLM across all metrics, although it ties with many methods on validity. Notably, all of the LlaSMol models perform much better than the other LLMs, which indicates the power of fine-tuning on SMolInstruct for understanding chemical reactions. The SoTA task-specific methods still outperform all of the LLMs, but the LlaSMol series is much closer than the other LLMs.
We observe similar trend for the RS task (Table 16 Again, LlaSMol is the best performing LLM across all metrics, although it does tie with LlaSMol for validity. We observe the LLMs without instruction tuning fail to achieve any accuracy greater than 2% on this task. This indicates that instruction tuning can be useful for LLMs to learn retrosynthesis. The SoTA task-specific methods still outperform all of the LLMs, which indicates that there is still room for improvement for LLMs on RS.
E.5 Other Common Findings
Besides for the aforementioned findings, we can observe some common findings across different tasks. Firstly, we can that GPT-4 does not show consistent pattern different
28

---

Published as a conference paper at COLM 2024
Table 14: Overall results (%) of MG.
Exact Match
FTS
Model
Validity
Top
Top
Top
MACCS
RDK
Morgan
MolT5
31.7
38.7
41.4
87.9
80.2
73.2
95.3
GPT-4 (0-shot)
6.4
7.7
9.2
74.2
53.3
42.6
81.4
GPT-4 (1-shot)
4.9
6.2
7.9
74.0
52.9
42.8
81.8
GPT-4 (3-shot)
5.9
8.2
8.9
74.8
53.5
43.3
85.2
GPT-4 (5-shot)
4.0
5.9
7.1
73.6
52.8
43.1
85.2
Claude Opus
12.3
83.9
67.6
57.6
92.6
Galactica
0.0
0.0
0.0
22.7
11.8
11.6
94.7
Llama 2
0.0
0.0
0.0
18.3
11.8
4.8
93.5
Code Llama
0.0
0.0
26.5
15.1
8.5
95.2
Mistral
0.0
0.1
0.1
32.2
18.4
9.0
35.9
Molinst
6.0
11.7
13.4
69.5
53.5
43.6
84.8
ChemLLM
0.9
38.3
21.5
14.3
4.3
LlaSMolcalactia
7.7
14.2
17.4
79.6
62.0
52.2
99.6
LlaSMol
6.4
11.4
13.7
74.3
55.8
47.1
99.6
LlaSMolcude Llama
6.5
11.8
14.2
74.0
55.9
46.6
99.7
Mistral
19.2
29.0
33.6
84.1
70.4
61.7
99.7
Table Overall results (%) of FS.
Exact Match
FTS
Model
Validity
Top
Top 3
Top 5
MACCS
RDK
Morgan
Molecular Transformer
78.4
85.5
87.0
95.5
92.9
91.4
99.5
RSMILES
78.7
88.0
89.7
95.7
93.7
92.2
100.0
GPT-4 (0-shot)
1.6
2.4
2.6
60.8
49.4
40.5
87.0
GPT-4 (1-shot)
1.1
2.2
2.6
61.5
49.5
41.1
91.4
GPT-4 (3-shot)
0.2
2.2
2.6
62.2
51.3
42.8
92.0
GPT-4 (5-shot)
1.3
2.0
3.0
63.1
51.7
44.0
93.8
Claude Opus
3.7
65.7
54.1
45.7
97.0
Galactica
0.0
0.0
0.0
40.2
33.2
25.9
83.7
Llama 2
0.0
0.0
0.0
33.4
24.3
13.7
97.7
Code Llama
0.0
0.0
0.0
35.3
26.4
15.8
99.6
Mistral
0.0
0.0
0.0
38.9
31.0
19.9
95.8
Molinst
2.1
3.3
3.7
51.1
36.7
31.7
99.8
ChemLLM
0.0
8,0
0.9
1.6
38.5
LlaS
53.1
66.4
70.7
88.8
82.6
79.9
99.7
LlaSMoljJama
47.1
61.6
66.4
87.0
80.1
76.9
99.8
LlaSMol Llama
52.0
65,4
69.2
88.3
81.9
79.2
99.8
LlaSMoly
63.3
75.5
79.0
91.8
87.1
84.9
99.8
29

---

Published as a conference paper at COLM 2024
Table 16: Overall results (%) of RS.
Exact Match
FTS
Model
Validity
Top
Top 3
Top 5
MACCS
RDK
Morgan
Molecular Transformer
47.0
61.7
66.5
87.0
81.5
77.5
99.7
RSMILES
46.2
63.9
69.9
86.5
81.0
76.8
100.0
GPT-4 (0-shot)
0.0
0.3
0.2
57.5
34.0
33.4
42.6
GPT-4 (1-shot)
0.3
0.8
1.4
66.6
42.5
40.9
79.6
GPT-4 (3-shot)
0.5
1.2
1.6
68.2
45.6
42.2
87.8
GPT-4 (5-shot)
0.2
0.8
1.2
68.3
46.0
43.1
84.4
Claude 3 Opus
1.1
70.3
49.8
46.2
94.8
Galactica
0.0
0.0
0.0
48,9
38.2
34.6
93.0
Llama 2
0.0
0.0
0.0
46.6
35.0
27.5
87.7
Code Llama
0.0
0.1
0.0
44.7
32.1
25.3
97.1
Mistral
0.0
0.0
0.0
44.6
32.0
24.2
98.0
Molinst
5.7
8.3
9.5
69.6
53.7
48.0
97.8
CL
0.0
14.3
2.1
2.9
10.9
LlaSMolGalactic
25.7
40.8
46.3
80.8
71.9
67.0
99.9
LlaSMoly,amma
22.5
35.5
41.1
79.8
70.4
65.2
99.9
LlaSMol Llama
25.7
40.0
45.7
80.7
71.7
66.7
100.0
LlaSMolMistral
32.9
49.6
55.4
83.0
75.2
70.4
100.0
in-context learning settings, with 0-shot achieving the best performance on most tasks. It may indicate that GPT-4 does not have sufficient knowledge about chemistry and cannot effectively learn how do these tasks by imitating several samples. Injecting chemistry knowledge by training LLMs may be necessary. Secondly, although Claude Opus still trails behind LlaSMol and the SoTA task-specific models, it surpasses GPT-4 on all the chemistry tasks. Claude 3 Opus demonstrates notably enhanced performance compared to GPT-4, showcasing its superior grasp of chemistry knowledge, highlighting its potential applications in the domain.
More Analytical Experiments
E1
Task Synergy
In order to investigate the synergy among different tasks, we first conduct an experiment to evaluate the performance of an LLM trained on single task or specific type of task. We train several single-task models, each focusing one task or group of tasks from the same type [4
We then compare the performance of these single-task models with LlaSMolMistral which is trained on all 14 tasks, to assess the potential benefits of multi-task training.
The results are presented in Table 17 The multi-tan task trained model, LlaSMol outper- forms the single-task models on the majority of tasks. The improvements are particularly substantial on PP-ESOI, PP-BBBI PP-Clintox, MC, and MG. This finding suggests the pres- ence of shared knowledge across these tasks. For instance, the knowledge of understanding SMILES may be common to most tasks, and training models multiple tasks enhance this knowledge, thus achieving better performance.
To further investigate the relationships among different tasks, conduct additional experiment by removing certain tasks from the training data. Specifically, we train a set of models, each with one task or a group of tasks from the same type removed from its training
14 14These single-task models include LlaSMol-NC, LiaSMol-PP, LLaSMol-MC, LlaSMol-MG, LlaSMol- FS, and LlaSMoI-RS, where the model names indicate the training task(s). For instance, LlaSMol-FS is trained only the FS task, I.IaSMo-NCC is trained on all the NC tasks combined, and LlaSMol-PP is trained on all the PP tasks combined. The training setups are identical to those of IlaSMol
30

---

Published as a conference paper at COLM 2024
Table 17: Results of single-task models and multi-task models. The "Single-Task" column corresponds to the single-task models trained on the corresponding tasks, while the "Multi- Task" column corresponds LlaSMol that is trained on all the tasks. Orange cells represent positive performance improvement.
Task
Metric
Single-Task
Multi-Task Improv.
NC-12F
(%)
86.8
87.9
1.1
NC-12S
EM (%)
67.6
70.1
2.4
NC-S2F
EM (%)
93.2
93.2
0.0
NC-S2I
EM (%)
27.4
29.0
1.5
PP-ESOL
RMSE
20.616
1.150
19.466
PP-Lipo
RMSE
1.241
1.010
0.231
PP-BBBP
Acc (%)
68.5
74.6
6.1
PP-Clintox
Acc (%)
79.9
93.1
13.2
PP-HIV
Acc (%)
96.7
96.7
0.0
PP-SIDER
Acc (%)
70.7
MC
METEOR
0.299
0.452
0.153
MG
FTS (%)
33.1
61.7
28.6
FS
EM (%)
62.6
63.3
0.7
RS
EM (%)
31.5
32.9
1.4
data 15 and their base models and training setups are identical to those of LlaSMolMistrall We then compare their performance with LlaSMolMistrall which is trained without removing any tasks.
Table 18: Results of removing certain tasks. Orange cells represent better results than LlaSMolyistral while blue cells represent worse results.
Model
EM
NC-12
NC-225
EM (54
NC-52F EM (S))
NC-S21 (M(%)
PP-ESOL RMSE
PP.Ligu RMSE
PP BBBB
Auc (%)
PP-Clintox Ace (3))
FP-HEV Ast (1))
PP-SIDER Asc
METBOR
MC
F15 F15(()
MG
EM)
FS
EM (5)
R5
0.817
62.3
MBC
87.6
71,0
93.5
27.8
1.133
1367
74.
93.1
96.8
71.9
64.1
63.3
33.4
w70 MBG
87.8
69.6
93.4
27.6
1.231
0382
77.2
93.1
96.8
71.5
0.445
63.4
34.0
w/o
704
93.8
29.5
1.274
1388
20.6
95.1
96.8
0.232
612
311
87.
7031
05.2
29.0
1.100
1.000
74.5
03.1
967
70.7
0.432
61.
63.
32.9
Interestingly, results presented in Table 18 generally show consistent pattern among different tasks. Specifically, removing certain task(s) can lead to improvements on some tasks, while leading to decrements on others. And in most cases, the change is not sub- stantial, suggesting that each task does not heavily rely on any other tasks, but rather on the data of the task itself. However, there are two exceptions for the "w/o NC" model, the performance of PP-ESOL and MG greatly drops. On PP-ESOL, we hypothesize that the knowledge of understanding chemical structures learned on NC might be useful in predicting the solubility of a molecule. For MG, since there are many IUPAC names in the input molecular descriptions, the performance drop may be attributed to the fact that the ability to convert IUPAC names to SMILES, learned on NC, can help the model directly obtain the related SMILES representations, thus leading to better performance.
E2 Influence of LoRA Modules and Trainable Parameters
In this section, we investigate the influence using different LoRA modules, or different sizes of trainable parameters. We take LlaSMol the basic setting and refer to it as LlaSMol in this section for simplicity. All the compared models are listed as follows, with trainable parameter sizes and ratios labeled in brackets:
LlaSMol Lite (6.8M, 0.09%): LoRA is applied on q proj and v proj of the attention modules.
LlaSMol Attn (13.6M, 0.19%): LoRA is applied on all the attention projection matrices (including q_proj, projj, v v-proj, o o.pro)).
15ehs models named "w/o Task", where Task represents the task names.
31

---

Published as a conference paper at COLM 2024
LIaSMol FFN (28.3M, 0.39%): LoRA is applied on all the FEN projection matrices (includ- ing gate_proj, down down_proj, up-proj).
LlaSMol (41.9,, 0.58%): The basic setting. LoRA is applied all the attention and FFN projection matrices.
LlaSMol Plus (173.0M, 2.33%): LoRA is applied on all the attention and FFN projection matrices, and lm_head is set trainable.
All these models trained with the identical training configurations (as described in Section 4.1
Figure presents the model performance all the 14 tasks. We can observe that for most tasks, progressing from LlaSMol Lite to LlaSMol Att,, LlaSMol FFN, and finally LlaSMol, the incorporation of more LoRA modules (and thus more trainable parameters) leads substantial performance enhancement. Further adding 1m_head as trainable parameters in LlaSMol Plus can further slightly improves the performance. This indicates that refining the selection of LoRA modules and incorporating more trainable parameters is very important. Moreover, LLMs exhibit considerable potential to surpass the performance of previous task-specific models on various chemistry tasks, provided that more trainable parameters are allowed or even full fine-tuning is employed.
However, for certain property prediction (PP) tasks, the incorporation of additional LoRA modules does not demonstrate consistent improvements. This inconsistency may at- tributed to the relatively small number of training and test samples, which can result in reduced model robustness when faced with diverse inputs, as well as the inherent random- associated with limited data.
32

---

Published as a conference paper at COLM 2024
12:00
24
1
023
KNU (6)
165384 AMI
Thatsidle Plontatte Sie DO
Instable Plesmnne
Trotaka Pormatter
(a) NC-12E
(b) NC-12S.
(c) NC-S2F
155
199
1240E: Pla
122
Luttto atte
1.13
11.
3:3
10
100
(d) NC-S2I.
(e) EPOL..L.
(f PP-Lip..
bockd 2015
37A1
Lattje Atte
W55S4 J01
Libitte PRN
Mecllii Alte Laciod
615
Takadb (4)
Tratable
(g) PP-BBBP.
(h) PP-Clintox.
(i) PP-HIV.
0.00
LU5744 JTS
0.42
L.5504 17
35/1 Lacoks Arra
ToViel Jite
1000
Lite
Loctood Lite
121
18
008
120
100
(i) PP-SIDER.
(k) MC.
(I) MG.
Lastad Plae
EM 20
210
de Tranalo See (M)
Stor (M)
(m) FS.
(n) RS.
Figure Performance on different tasks under different LoRA settings. Except for PP-ESOL and PP-Lipo, the larger the better.
33


Reviews:
[{"review_id": "VB8SpW5Np7", "reviewer": "Reviewer_Qxrz", "summary": "This paper proposes SMolInstruct, a large-scale instruction-tuning dataset designed with over 3 million samples and 14 chemistry tasks. The authors use this dataset to finetune several open-source LLMs like Galactica, LLaMA, and Mistral, creating the LlaSMol series of models tailored for chemistry applications. Extensive experiments show that the LlaSMol models significantly outperform existing LLMs like GPT-4 and Claude on the chemistry tasks, though they still trail behind task-specific models. The authors conduct analyses on different modeling choices like canonicalization of SMILES, use of SELFIES vs SMILES representations, and comparison to a previous chemistry instruction dataset.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for characterizing our work as “large/diverse dataset”, “significant performance gains”, and “careful data splitting”. We address your concerns below.\n\n---\n**“novelty is limited”**\n\n“All the techniques have been used by previous studies” is actually not true. Our novelties:\n- For the first time, PubChem and MoleculeNet are used for instruction tuning, which is not trivial.\n- We introduce novel methods tailored for chemistry. To name a few, we use specialized methods to eliminate duplicates and data leakage, and introduce SMILES canonicalization and special tags.\n- We are the first to comprehensively train and evaluate multiple LLMs on chemistry. Our models outperform existing LLMs, and experiments on task synergies, varying trainable parameters, and more provide significant insights.\n\n---\n**“evaluation metrics do not fully capture models’ capabilities”**\n\nWe use EM as it directly measures the correctness, which is crucial for chemistry. As described in Appendix D.4, EM evaluates on a deeper semantic level instead of a simple string comparison, and thus can effectively capture model performance. Also, EM is widely used in prior research. The zeros in Table 1 actually reflect the models’ incompetence when not turned on our dataset, not a flaw in EM.\n\nFor a comprehensive evaluation, we have included many other metrics such as validity and FTS (Appendix D.4 and E). We welcome any other metrics you might recommend. \n\n---\n**“does not explore generalization abilities”**\n\nWe included all tasks we can find high-quality data for, and building models that perform well on all of them is a significant contribution. How to meaningfully test generalization abilities is nontrivial and needs careful design, and we leave it for future work.\n\n---\n**“fail to surpass the non-LLM models with far fewer parameters”**\n\nThere seems to be a misunderstanding. Due to resource constraints, we only tune 42M (0.58%) of LlaSMol's parameters, significantly fewer than the non-LLM models (e.g., MT 150M, Uni-Mol 181M). Despite this, LlaSMol approaches their performance levels, which is encouraging. Also, our exp on varying trainable parameters (Appendix F.2) highlights our potential to surpass non-LLM models with further tuning.\n\nBeyond performance, LlaSMol offers unique advantages over non-LLM models, such as handling multiple tasks and following natural language instructions to complete tasks.\n\n---\nThank you for your time and effort. If anything is not clear, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. SMolInstruct is large enough and diverse enough for instruction tuning of LLMs, with 3 million carefully filtered samples over 14 downstream tasks.\n2. Four backbone LLMs of different architectures are evaluated in the experiments of this paper, and the performance gains are significant.\n3. Careful dataset splitting avoids potential data leakage across related tasks.", "reasons_to_reject": "1. All the data collection, validation, and processing techniques have been used by previous studies. The novelty of this paper is limited.\n2. The evaluation metrics like exact match do not fully capture the models' capabilities. For example, this results in too many zeros in Table 1. Some better metrics may help reveal the models' performances.\n3. The paper does not explore the models' generalization abilities beyond the specific downstream tasks in the dataset.\n4. While outperforming existing LLMs, the LlaSMol models fail to surpass the non-LLM models which have far fewer parameters.", "questions_to_authors": "I don't have additional questions for the authors.", "extra_scores": {}}, {"review_id": "sPIdZGEApM", "reviewer": "Reviewer_5qEm", "summary": "This is a solid dataset paper, which introduces a new domain-specific text-based dataset for instruction tuning. Overall, this is a high-quality, somewhat niche, but thorough paper that shows how dataset papers ought to be done. It is very clearly written (except for a few typos here and there), provides the necessary context for a subject matter expert to understand the data generation procedure, and goes into meticulous detail of generation protocol, dataset statistics, comparison to other datasets, and experimental evaluation details. Given the importance that text-based benchmarks, even those in niche domains, have recently gained for general-purpose evaluation of LLMs, the impact of this paper is likely to be high in the community.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work’s potential impact as “modest-to-high” and characterizing our efforts as “thorough” and “careful”. We address your concerns and questions below. \n\n---\n**“only 3x compared to most similar dataset (Mol-Instructions)”**\n\nWe clarify our efforts that lead to the size of our dataset:  \n- We included all available data that we deem reliable and necessary. Low-quality sources and redundant samples are not included.\n- We removed many duplicate and low-quality samples, which decreased the quantity but expectedly enhanced the overall quality.\n\nTherefore, we believe our dataset size is a strength, given that these 3X data samples are of higher quality overall. \n\n---\n**“dataset is somewhat niche/narrow”**\n\nOur dataset contains 14 chemistry tasks focused on small molecules. While this may seem niche, we would like to stress on the profound impact of small molecules and why we intentionally design our dataset in this way:\n- We chose to focus on small molecules due to their crucial role in critical fields like drug discovery and material science. This specialization ensures our dataset is highly relevant and allows us to provide deeper, more applicable insights into those fields.\n- The 14 tasks were selected for their real-world significance, particularly in drug discovery, and the availability of high-quality data. Each task addresses key challenges in the field, ensuring the dataset's practicality and value. Tasks lacking sufficient high-quality data, like yield prediction, are excluded to maintain quality.\n\n---\n**Q1: Clarification about the special tags.**\n\nThe special tags are implemented textually, inserted directly into query/response sentences. Their embeddings are derived using the default tokenizer of the base models. We will add the clarification in our manuscript.\n\n---\n**Q2: Using arrows to indicate lower/higher.**\n\nThanks. We will add arrows as you suggested.\n\n---\n**Q3: Typos.**\n\nThanks. We will carefully fix the typographical errors.\n\n---\nThank you again for your time and effort. If anything is not clear, please feel free to let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Very likely will have moderate-to-high impact in the LM community, even if only \"an benchmark dataset\" in an LM eval suite\n* Thorough description of data collection/curation pipeline\n* Careful control of data leaking for reciprocal tasks (crucial for LLM benchmarks!)\n* Modest but thorough baseline evaluation", "reasons_to_reject": "* Claims dataset size as a strength, but it is only 3x compared to most similar dataset (Mol-Instructions)\n* Dataset is somewhat niche / narrow", "questions_to_authors": "* Q: it's unclear whether the tagging strategy used (e.g. for <SMILES>) is textual or whether it relies on a dedicated embedding for the tag (e.g. alla tagLLM https://arxiv.org/abs/2402.05140). Please clarify.\n* S: label all tables, even those in the appendix, with arrows to indicate lower/higher is better\n* S: there's a few typos throughout the paper, including \"spitted into training\" (Sec 3.2)", "extra_scores": {}}, {"review_id": "62DB6Soc2x", "reviewer": "Reviewer_fwDB", "summary": "This paper proposes a new instruction-tuning dataset, SMolInstruct, that includes 3M+ data points across 14 chemistry tasks, divided into 4 main categories, to help improve the performance of large language models in chemistry. To this end, the paper describes the data collection, curation, and splitting processes used to obtain SMolInstruct, and empirically evaluates the performance of standard open LLMs fine-tuned on this dataset using LoRA against both (i) state-of-the-art (SoTA) models that narrowly focus on individual tasks, and (ii) state-of-the-art language models like GPT-4 and Claude 3. The paper finds that fine-tuning on SMolInstruct allows the base LLM to substantially outperform the best-performing LLMs and narrow the gap with the SoTA models for the individual tasks significantly. To support these findings, the paper also conducts ablation studies by changing the underlying molecular representation and fine-tuning on different instruction tuning dataset, observing that SMolInstruct is the key factor behind the observed empirical improvements.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for characterizing our work as “large, well-curated, useful dataset”, “the encouraging empirical results”, “insightful discussion”, and “comprehensive and insightful empirical analysis”. We address your concerns and questions below.\n\n---\n**“hyper-parameter tuning is not complete enough”**\n\nThanks for pointing it out. We acknowledge that, due to computational resource constraints, we adopted a limited search space, which might lead to suboptimal performance for some models. We will follow your suggestions and tone down the differences in the fine-tuned model performances in our updated draft. \n\n---\n**“not evaluate on existing benchmarks for the tasks being studied” “not follow existing benchmarks (where available) for the tasks at hand”**\n\nThank you for your insightful concern. To ensure rigor and comparability with existing work, we have dedicated much effort in aligning our dataset with existing benchmarks, as detailed in Section 3.2. Specifically:\n- We matched our data splitting for shared tasks (MC, MG, FS, and RS) with the Mol-Instructions dataset [1]. This means the SMolInstruct test set excludes any training examples from Mol-Instructions, allowing us to directly evaluate models trained on Mol-Instructions (Molinst in Tables 1 and 2). \n- For molecular property prediction, our data comes exactly from MoleculeNet. Since it does not provide a specific sample splitting, we had to do the splitting ourselves. To ensure fair comparisons, we re-trained the compared models on our splits using their official codebases. Thus, we have ensured all comparisons in our manuscript are fair by either aligning data splits with existing models’ training data or retraining models on our splits.\n\nOur primary focus is performance on our dataset, so we did not train and evaluate models on other benchmarks. To compare our dataset with others, we trained LlaSMol on Mol-Instructions and evaluated it on our SMolInstruct. As shown in Table 4, even on shared tasks, the model trained on SMolInstruct outperforms the model trained on Mol-Instructions, highlighting the advantages of SMolInstruct.\n\n[1] Fang, Y., et. al. (2023). Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models. In ICLR.\n\n---\nThank you again for your constructive feedback. We will revise our manuscript accordingly. If anything is not clear enough, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed dataset is large, well-curated, and a useful addition to the literature. \n- The empirical results shown in the paper are encouraging. Moreover, they are obtained with little computational resources, and thus there is strong reason to believe that a more rigorous computational exploration could unlock further improvements. \n- The discussion on underlying molecular representations and the importance of the fine-tuning process is insightful and important. Indeed, the argument about pre-performing canonicalization highlights that LLMs inherently struggle with more structure-related tasks, and further  support the need for better-designed datasets, as well as more careful integration of LLMs with specialized components.\n- The empirical analysis (baselines, protocol) is comprehensive and insightful.", "reasons_to_reject": "- The paper's argument about the differences in the fine-tuned base model performances is not rigorous enough. In particular, the hyper-parameter tuning conducted for these results is not complete enough to conclusively assert the performance differences. The point raised about the original training is plausible and pertinent. However, I would still tone this down to better reflect the significance of the results.\n\n- The paper does not evaluate on existing benchmarks for the tasks being studied, instead proposing its own test split on its own data, which makes findings less comparable with other approaches. There are well-established benchmarks for at least some of these tasks. For instance, molecular property prediction benchmarks based on MoleculeNet are very common, e.g., in the area of graph neural networks (GNNs), which the authors cite (see the OGB [1] paper for the specific benchmarks). Hence, I strongly advise the authors to re-design their splits to align with existing baselines, i.e., vetting the training split of SMolInstruct, which includes MoleculeNet data, to avoid leakage w.r.t, the established baselines,  and then running similar experiments on the known benchmarks. This would not only provide a real-world reference task with relevant baselines, but also consolidate the generality of your results. \n\n\n[1] Hu et al. Open Graph Benchmark: Datasets for Machine Learning on Graphs, NeurIPS 2020.", "questions_to_authors": "- Why did you not follow existing benchmarks (where available) for the tasks at hand?", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a large-scale chemistry dataset for instruction tuning, encompassing 14 selected chemistry tasks and over three million samples. Using this dataset, the authors fine-tune a set of open-source LLMs and identify Mistral as the best base model for chemistry tasks, which outperforms the most advanced models like GPT-4 and Claude 3 Opus by a significant margin. Although one reviewer expressed concern about some inaccurate claims made by the authors and the lack of discussion on related works, the authors have committed to clarifying these claims and including such a discussion in their revisions. Given the overall strong support from the majority of reviewers, I recommend acceptance.
Decision: Accept