SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Taicheng Guo,\* Kehan Guo,\* Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V. Chawla, Olaf Wiest, Xiangliang Zhang
{tguo2, kguo2, bnan, zliang6, zguo5, nchawla, owiest, xzhang33} @nd.edu
Large Language Models (LLMs) with strong abilities in natural language processing tasks have emerged and have been applied in various kinds of areas such as science, finance and software engineering. However, the capability of LLMs to advance the field of chemistry remains unclear. In this paper, rather than pursuing state-of-the-art performance, we aim to evaluate capabilities of LLMs in a wide range of tasks across the chemistry domain. We identify three key chemistry-related capabilities including understanding, reasoning and explaining to explore in LLMs and establish a benchmark containing eight chemistry tasks. Our analysis draws on widely recognized datasets facilitating a broad exploration of the capacities of LLMs within the context of practical chemistry. Five LLMs (GPT-4, GPT-3.5, Davinci-003, Llama and Galactica) are evaluated for each chemistry task in zero-shot and few-shot in-context learning settings with carefully selected demonstration examples and specially crafted prompts. Our investigation found that GPT-4 outperformed other models and LLMs exhibit different competitive levels in eight chemistry tasks. In addition to the key findings from the comprehensive benchmark analysis, our work provides insights into the limitation of current LLMs and the impact of in-context learning settings on LLMs' performance across various chemistry tasks. The code and datasets used in this study are available at https://github.com/ChemFoundationModels/ChemLLMBench.
Large language models (LLMs) have recently demonstrated impressive reasoning abilities across a wide array of tasks. These tasks are not limited to natural language processing, but also extend to various language-related applications within scientific domains [56, 30, 24, 10]. Much of the research on the capacity of LLMs in science has been focused on tasks such as answering medical [30] and scientific questions [24, 25]. However, the exploration of their application to practical tasks in the field of chemistry remains underinvestigated. Although some studies [6, 27, 63, 48] have been conducted, they tend to focus on specific case studies rather than a comprehensive or systematic evaluation. The exploration of LLMs' capabilities within the field of chemistry has the potential to revolutionize this domain and expedite research and development activities [62]. Thus, the question, "What can LLMs do in chemistry?" is a compelling topic of inquiry for both AI researchers and chemists. Nevertheless, there exist two challenges that hinder the answer to the topic and the further development of LLMs in chemistry:
*Both authors contribute equally to the work, under the support of NSF Center for Computer Assisted Synthesis (C-CAS). https://ccas.nd.edu.
- Determining the potential capabilities of LLMs in chemistry requires a systematic analysis of both LLMs and the specific requirements of chemistry tasks. There are different kinds of tasks in chemistry, some of which can be formulated to tasks solved by LLMs while others may not. It is necessary to consider the specific knowledge and reasoning required for each task and assess whether LLMs can effectively acquire and utilize that knowledge.
- Conducting reliable and wide-ranging evaluation requires diverse experimental settings and limitations, that is, careful consideration and standardization of evaluation procedures, dataset curation, prompt design, and in-context learning strategies. Additionally, the API call time consumption and the randomness of LLMs limit the size of the testing.
To address this knowledge gap, we (a group of AI researchers and chemists) have developed a comprehensive benchmark to provide a preliminary investigation into the abilities of LLMs across a diverse range of practical chemistry tasks. Our aim is to gain insights that will be beneficial to both AI researchers and chemists to advance the application of LLMs in chemistry. For AI researchers, we provide insights into the strengths, weaknesses, and limitations of LLMs in chemistry-related tasks, which can inform the further development and refinement of different AI techniques for more effective applications within the field. For chemists, our study provides a better understanding of the tasks in which they can rely on current LLMs. Utilizing our more extensive experimental setup, a broader range of chemistry tasks can be explored to further evaluate the capabilities of LLMs.
Our investigation focuses on 8 practical chemistry tasks, covering a diverse spectrum of the chemistry domain. These include: 1) name prediction, 2) property prediction, 3) yield prediction, 4) reaction prediction, 5) retrosynthesis (prediction of reactants from products), 6) text-based molecule design, 7) molecule captioning, and 8) reagents selection. Our analysis draws on widely available datasets including BBBP, Tox21 [65], PubChem [32], USPTO [29, 53, 39], and ChEBI [17, 16]. Five LLMs (GPT-4, GPT-3.5, Davinci-003, Llama, and Galactica) [43] are evaluated for each chemistry task in zero-shot and few-shot in-context learning settings with carefully selected demonstration examples and specific prompts. We highlight the contributions of this paper as follows:
- We are the first to establish a comprehensive benchmark to evaluate the abilities of LLMs on a wide range of chemistry tasks. These eight selected tasks, in consultation with chemists, not only encompass a diverse spectrum of the chemistry domain but also demand different abilities such as understanding, reasoning, and explaining using domain-specific chemistry knowledge.
- We provide a comprehensive experimental framework for testing LLMs in chemistry tasks. To factor in the impact of prompts and demonstration examples in in-context learning, we have assessed multiple input options, focusing on the description of chemistry tasks. Five representative configurations were chosen based on their performance on a validation set, then these selected options were applied on the testing set. The conclusion is made from five repeated evaluations on each task, since GPTs often yield different outputs at different API calls even though the input is the same. We thus believe that our benchmarking process is both reliable and systematic.
- Our investigations yield broader insights into the performance of LLMs on chemistry tasks. As summarized in Table 2, our findings confirm some anticipated outcomes (e.g., GPT-4 outperforms GPT-3 and Davinci-003), and also reveal unexpected discoveries (e.g., property prediction can be better solved when property label semantics are included in prompts). Our work also contributes to practical recommendations that can guide AI researchers and chemists in leveraging LLMs more effectively in the future (see Section 5).
The paper is organized as follows. Related works are presented in Section 2. In section 3, we elaborate on the evaluation process, including an overview of the chemistry tasks, the utilized LLMs and prompts, and the validation and testing settings. In section 4, we summarize the main findings (due to the space limit, evaluation details of each chemistry task can be found in Appendix). Finally, to answer the question "What can LLMs do in chemistry?" we discuss the constraints inherent to LLMs and how different settings related to LLMs affect performance across various chemistry tasks in Section 5. The conclusions are summarized in section 6.
Large Language Models. The rise of Large Language Models (LLMs) has marked a significant trend in recent natural language processing (NLP) research. This progress has been fuelled by milestones such as the introduction of GPT-3 [4], T0 [52], Flan-T5 [12], Galactica [56] and LLaMa [57]. The
recently released GPT-4, an evolution from GPT-3.5 series, has drawn considerable attention for its improvements in language understanding, generation, and planning [43]. Despite the vast potential of LLMs, existing research primarily centers on their performance within general NLP tasks [8, 9]. The scientific disciplines, notably chemistry, have received less focus. The application of LLMs in these specialized domains presents an opportunity for significant advancements. Therefore, we conduct a comprehensive experimental analysis to evaluate the capability of LLMs in chemistry-related tasks.
Large Language Model Evaluations. In recent years, the evaluation of LLMs like GPT has become a significant field of inquiry. [11] showed ChatGPT's proficiency in law exams, while technical aspects of GPT-4 were analyzed in [43]. LLMs are also applied in healthcare [14], mathematical problem [18], and code generation tasks [37]. Specifically, in healthcare, the utility and safety of LLMs in clinical settings were explored [42]. In the context of mathematical problem-solving, studies [18, 7] have highlighted that LLMs encounter challenges with graduate-level problems, primarily due to difficulties in parsing complex syntax. These studies underscored the complexity of achieving task-specific accuracy and functionality with LLMs. Lastly, AGIEval [66] assessed LLMs' general abilities but noted struggles in complex reasoning tasks.
Our work aligns with these evaluations but diverges in its focus on chemical tasks. To our knowledge, this is the first study to transform such tasks to suit LLM processing and to perform a comprehensive evaluation of these models' ability to tackle chemistry-related problems. This focus will contribute to expand our understanding of LLMs' capabilities in specific scientific domains.
Large Language Model for Chemistry. Recent efforts integrating LLMs with the field of chemistry generally fall into two distinct categories. One category aims to create a chemistry agent with LLMs' by leveraging its planning ability to utilize task-related tools. For example, Bran et al [3] developed ChemCrow, which augmented LLMs with chem-expert designed tools for downstream tasks such as organic synthesis and drug discovery. Similarly, by leveraging the planning and execution ability of multiple LLMs, Boiko et al [2] developed an autonomous chemical agent to conduct chemical experiments. The other category involves direct usage of LLMs for downstream tasks in chemistry [27, 62, 6, 28]. While these studies have explored the performance of LLMs in chemistry-related tasks, a systematic evaluation of their capabilities within this domain has been lacking. Consequently, there is a noticeable gap that calls for a meticulous benchmark to thoroughly assess the potential of LLMs in chemistry. Such a benchmark is crucial not only for identifying the strengths and limitations of these models in a specialized scientific domain, but also to guide future improvements and applications.
3 The Evaluation Process and Setting
The evaluation process workflow is depicted in Fig. 1. Guided by co-author Prof. Olaf Wiest (from the Department of Chemistry at the University of Notre Dame), we identify eight tasks in discussion with senior Ph.D. students at the NSF Center for Computer Assisted Synthesis (C-CAS). Following this, we generate, assess, and choose suitable prompts to forward to LLMs. The acquired answers are then evaluated both qualitatively by chemists to identify whether they are helpful in the real-world scenario and quantitatively by selected metrics.
Chemistry tasks. In order to explore the abilities of LLMs in the field of chemistry, we concentrate on three fundamental capabilities: understanding, reasoning, and explaining. We examine these competencies through eight diverse and broadly acknowledged practical chemistry tasks. These tasks are summarized in Table 1, in terms of the task type from the perspective of machine learning, the dataset used for the evaluation, as well as the evaluation metrics. The #ICL candidates refers to the number of candidate examples, from which we select  $k$  demonstration examples, either randomly or based on similarity searches. These candidate sets are the training sets used in classical machine learning models, e.g., in training classifiers or generative models. We set the test set of 100 instances, randomly sampled from the original testing dataset (non-overlapping with the training set). To reduce the influence of the LLMs randomness on the results, each evaluation experiment is repeated five times and the mean and variance are reported.
LLMs. For all tasks, we evaluate the performance of five popular LLMs: GPT-4, GPT-3.5 (referred to as GPT-3.5-turbo, also known as ChatGPT), Davinci-003, LLama and Galactica.
Figure 1: Overview of the evaluation process
Figure 2: The standardized zero-shot prompt template for all tasks.
Table 1: The statistics of all tasks, datasets, the number of ICL/test samples, and evaluation metrics
Ability	Task	Task Type	Dataset	#ICL candidates	#test	Evaluation Metrics
Understanding	Name Prediction	Generation	PubChem	500	100	Accuracy
Property Prediction	Classification	BBBP, HIV, BACE, Tox21, ClinTox	2053, 41127, 1514, 8014, 1484	100	Accuracy, F1 score
Reasoning	Yield Prediction	Classification	Buchwald-Hartwig, Suzuki-Miyaura	3957, 5650	100	Accuracy
Reaction Prediction	Generation	USPTO-Mixed	409035	100	Accuracy, Validity
Reagents Selection	Ranking	Suzuki-Miyaura	5760	100	Accuracy
Retrosynthesis	Generation	USPTO-50k	40029	100	Accuracy, Validity
Text-Based Molecule Design	Generation	ChEBI-20	26407	100	BLEU, Exact Match, etc
Explaining	Molecule Captioning	Generation	ChEBI-20	26407	100	BLEU, Chemists, etc
Zero-shot prompt. For each task, we apply a standardized zero-shot prompt template. As shown in Fig. 2, we instruct the LLMs to act in the capacity of a chemist. The content within the brackets is tailored to each task, adapting to its specific inputs and outputs. The responses from LLMs are confined to only returning the desired output without any explanations.
You are an expert chemist. Given the [reactants SMILES / molecular description / ...]: [Input], predict the [reaction product SMILES / molecule SMILES / ...] using your experienced chemical [reaction prediction / chemical molecule design / ...] knowledge. No explanations and other information. Only return the [product SMILES / designed molecular SMILES ].
Task-specific ICL prompt. ICL is a new paradigm for LLMs where predictions are based solely on contexts enriched with a few demonstration examples [15]. This paper specifically denotes ICL as a few-shot in-context learning approach, excluding the zero-shot paradigm. In order to thoroughly examine the capacities of LLMs within each chemistry-specific task, we design a task-specific ICL prompt template. As shown in Fig. 3. The format of the template is similar to that used in [48]. We also partition our template into four parts: {General Template} {Task-Specific Template} {ICL} {Question}. The {General Template} is almost the same as the zero-shot prompt, instructing the LLMs to play the role of a chemist and specify the chemistry task with its corresponding input and output. Considering that the responses for chemistry-related tasks must be accurate and chemically reasonable, it is crucial to prevent LLMs from generating hallucinated information. To this end, we introduce the {Task-Specific Template} which consists of three main components: [Input explanation], [Output Explanation], and [Output Restrictions], specifically designed to reduce hallucinations. These components are tailored to each task. The {ICL} part is a straightforward
concatenation of the demonstration examples and it follows the structure "[Input]: [Input_content] [Output]: [Output_content]". The [Input] and [Output] denote the specific names of each task's input and output, respectively. For example, in the reaction prediction task, the [Input] would be "Reactants+Reagents" and the [Input_content] would be the actual SMILES of reactants and reagents. The [Output] would be "Products" and the [Output_content] would be the SMILES of products. Detailed ICL prompts for each task will be presented in their respective sections that follow. The last {Question} part presents the testing case for LLMs to respond to. Fig 5 is example of our name prediction prompt.
Figure 3: An ICL prompt template for all tasks.
Figure 4: An ICL prompt example for smiles2iupac prediction
ICL strategies. To investigate the impact of the quality and quantity of ICL examples on the performance of each task, we explore two ICL strategies. The quality is determined by the retrieval methods employed for finding similar examples to the sample in question. We conduct a grid search across two strategies: {Random, Scaffold}. In the Random strategy, we randomly select  $k$  examples from the ICL candidate pool. In the Scaffold strategy, if the [Input_content] is a molecule SMILES, we use Tanimoto Similarity [55] from Morgan Fingerprint [41] with 2048-bits and radius=2 to calculate the molecular scaffold similarity to find the top- $k$  similar molecule SMILES. If the [Input_content] is a description such as IUPAC name or others, we use Python's built-in difflib.SequenceMatcher tool [49] to find the top- $k$  similar strings. To explore the influence of the quantity of ICL examples on performance, we also perform a grid search for  $k$ , the number of ICL examples, in each task.
Experiment setup strategy. In property prediction and yield prediction tasks, we perform the grid search of  $k$  in  $\{4, 8\}$ . In the name prediction, reaction prediction, and retrosynthesis tasks, we perform the grid search of  $k$  in  $\{5, 20\}$ . In text-based molecule design and molecule captioning tasks, we
Table 2: The rank of five LLMs on eight chemistry tasks and performance highlight (NC: not competitive, C: competitive, SC: selectively competitive, acc: accuracy).
Task	GPT-4	GPT-3.5	Davinci-003	Llama2-13B-chat	GAL-30B	Performance highlight (comparing to baselines if any)
Name Prediction	1	2	3	4	5	NC: max. acc. 8% (Table 4)
Property Prediction	1	2	3	5	4	SC: outperform RF and XGBoost from MoleculeNet [65] (Table 6)
Yield Prediction	1	3	2	5	4	C: but 16-20% lower acc. than UAGNN [34] (Table 10)
Reaction Prediction	1	3	2	5	4	NC: 70% lower acc. than Chemformer [26] (Table 11)
Reagents Selection	2	1	3	4	5	C: 40-50% acc. (Table 12)
Retrosynthesis	2	3	1	5	4	NC: 40% lower acc. than Chemformer [26] (Table 13)
Molecule Design	1	3	2	4	5	SC: better than MolT5-Large [17] (Table 14)
Molecule Captioning	1	2	1	4	5	SC: better than MolT5-Large [17] (Table 15)
Average rank	1.25	2.375	2.125	4.5	4.5	overall: 3 SC, 2 C, 3 NC
perform the grid search of  $k$  in  $\{5, 10\}$  because of the maximum token limitation of LLMs. To reduce the time consumption of API requests caused by testing on the large test set, we first construct a validation set of size 30 which is randomly sampled from the original training set. Then we search  $k$  and retrieval strategies (\{Random, Scaffold\}) on the validation set. Based on the validation set results, we take 5 representative options when testing on 100 instances, which are randomly sampled from the original test set. For each task, we run evaluation 5 times and report mean and standard deviation.
4 Experiment Analysis
Due to space limitations, we provide details of the evaluation on each chemistry task in Appendix by the following order: name prediction in section A, property prediction in section B, yield prediction in section C, reaction prediction in section D, reagents selection in section E, retrosynthesis in section F, text-based molecule design in section G, and molecule captioning in section H. The detailed results described in the Appendix allow us to approach the question "What can LLMs do in chemistry?" from several directions. We discuss the key findings from our comprehensive benchmark analysis and provide valuable insights by thoroughly analyzing the limitation of LLMs and how different settings related to LLMs affect performance across various chemistry tasks.
4.1 Can LLMs outperform existing baselines in chemistry tasks?
Several classic predictive models based on machine learning (ML) have been developed for specific chemistry tasks. For instance, MolR (Graph Neural Network-based) predicts molecule properties as a binary classification problem [58]. UAGNN achieved state-of-the-art performance in yield prediction [34]. MolT5-Large, a specialized language model based on T5, excels in translating between molecule and text [17]. We conduct a performance analysis of GPT models and compare their results with available baselines, if applicable. The main findings from the investigations are:
- GPT-4 outperforms the other models evaluated. The ranking of the models on 8 tasks can be found in Table 2;
- GPT models exhibit a less competitive performance in tasks demanding precise understanding of molecular SMILES representation, such as name prediction, reaction prediction and retrosynthesis;
- GPT models demonstrate strong capabilities both qualitatively (in Fig. 14 evaluated by chemists) and quantitatively in text-related explanation tasks such as molecule captioning;
- For chemical problems that can be converted to classification tasks or ranking tasks, such as property prediction, and yield prediction, GPT models can achieve competitive performance compared to baselines that use classical ML models as classifiers, or even better, as summarized in Table 2.
These conclusions are derived from conducting five repeated evaluations on each task, using the best evaluation setting that was discovered through a grid search on the validation set of each task. We designate the performance of GPT models as three categories and provide in-depth discussion next.
- Tasks with not competitive (NC) performance. In tasks such as reaction prediction and retrosynthesis, GPT models are worse than existing ML baselines trained by large amounts of training data, partially because of the limitation on understanding molecular SMILES strings. In reaction prediction and retrosynthesis, SMILES strings are present in both the input and output of the GPT models. Without an in-depth understanding of the SMILES strings that represent
reactants and products, as well as the reaction process that transforms reactants into products, it will be difficult for GPT models to generate accurate responses, as shown in Table 11 and 13. GPT models exhibit poor performance on the task of name prediction as well (see Table 4). This further validates the notion that GPT models struggle with understanding long strings in formats such as SMILES, IUPAC name, and molecular formula, and make correct translations between them.
- Tasks with competitive (C) performance. GPT models can achieve satisfactory results when the chemistry tasks are formulated into the forms of classification (e.g., formatting yield prediction into a high-or-not classification, instead of regression) or ranking (as seen in reagents selection), as illustrated in Fig. 7 and 9. This is understandable, because making choices is inherently simpler than generating products, reactants or names. GPT models can achieve an accuracy of  $40\%$  to  $50\%$  when asked to select the reactant or solvent or ligand from provided candidates. Although GPT-4's performance on yield prediction falls short compared to the baseline model UAGNN [34] (with  $80\%$  versus  $96\%$  on the Buchwald-Hartwig dataset, and  $76\%$  versus  $96\%$  on the Suzuki-coupling dataset), it demonstrates improved performance when given more demonstration examples within the few-shot in-context learning scenario, as reported in Table 10. It is worth noting that the UAGNN model was trained on thousands of examples for these specific reactions. Last, while GPT models exhibit promising performance for yield prediction on the evaluated High-Throughput Experimentation (HTE) datasets, specifically the Buchwald-Hartwig [1] and Suzuki-Miyaura datasets [50], they perform as bad as other ML baselines on more challenging datasets like USPTO-50k [53]. This observation indicates a potential area for future research and improvement in the performance of GPT models on challenging chemistry datasets.
- Tasks with selectively competitive (SC) performance. GPT models are selectively competitive on two types of tasks.