SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- In the property prediction task on some datasets (HIV, ClinTox), GPT models outperform the baseline significantly, achieving F1 scores and accuracy nearing 1, as reported in Table 6 and 7. This might be due to the fact that the property labels to be predicted are included in the prompts, with GPT models being simply tasked in responding with yes or no. For example, the prompt includes inhibit HIV replication or drugs failed clinical trials for toxicity reason, and we observed a significant decline in the performance of GPT models upon removing property labels from the prompt (refer to Appendix section B). In contrast, baselines employing machine learning models do not include the semantic meaning of these labels in their input. The input for these models only comprises molecular representations in graph form but no labels.
- For tasks related to text, such as text-based molecule design and molecule captioning, GPT models exhibit strong performance due to their language generation capabilities. On the task of text-based molecule design, GPT models outperform the baseline when evaluated using NLP metrics such as BLEU and Levenshtein. However, when it comes to exact match, the accuracy is less than  $20\%$ , as reported in Table 14 and 15. This suggests that the molecules designed by GPT models may not be exactly the same as the ground truth. Particularly in the context of molecular design/generation, the exact match is a significant metric. Unlike in natural language generation where there is some allowance for deviation from the input, molecular design demands precise accuracy and chemical validity. However, not being precisely identical to the ground truth does not automatically invalidate a result. Molecules generated by GPT models may still prove to be beneficial and could potentially act as viable alternatives to the ground truth, provided they meet the requirements outlined in the input text and the majority (over  $89\%$ ) are chemically valid (see Table 14). Nonetheless, assessing the true utility of these generated molecules, such as evaluating their novelty in real-world applications, can be a time-consuming undertaking.
4.2 The capability of different LLMs
As shown in Table 2, we can find that GPT-4 model shows better chemical understanding, reasoning, and explaining abilities than Davinci-003, GPT-3.5, Llama and Galactica. This further verifies the GPT-4 model outperforms the other models in both basic and realistic scenarios [5].
4.3 The effects of the ICL
To investigate the effects of the ICL, we introduced ICL prompting and different ICL retrieval methods, and the different number of ICL examples in each task. Based on the experiments results of
12 different variants of each option and evaluating their performance on the validation set, we have the following three observations:
- In all tasks, the performance of ICL prompting is better than zero-shot prompting.
- In most tasks (in Table 4, 6, 7, 11, 13, 14, 15), using scaffold similarity to retrieve the most similar examples of the question as ICL examples achieves better performance than random sampling.
- In most tasks (in Table 4, 6, 7, 10, 11, 14, 15), using larger  $k$  (more ICL examples) usually achieves better performance than small  $k$  (fewer ICL examples).
These observations indicate that the quality and quantity of ICL examples plays an important role in the performance of ICL prompting [23, 36]. This may inspire that it is necessary to design more chemistry-specific ICL methods to build high-quality ICL examples to further improve the ICL prompting performance.
4.4 Are molecule SELFIES representations more suitable for LLMs than SMILES representations?
SELFIES [33] representations are more machine-learning-friendly string representations of molecules. To investigate whether the SELFIES representations are more suitable for LLMs than SMILES representations, we conduct experiments on four tasks, including molecule property prediction, reaction prediction, molecule design and molecule captioning. The experiment results are shown in Table 16, 17, 18, 19. We can observe that the results of using SELFIES in all four tasks are inferior to those of using SMILES. This could be attributed to the fact that the pretraining datasets for LLMs are primarily populated with SMILES-related content rather than SELFIES. Consequently, these models are more attuned to SMILES. However, it's worth mentioning that the occurrence of invalid SELFIES is less frequent than that of invalid SMILES, which aligns with the inherent design of SELFIES to ensure molecular validity.
4.5 The impact of temperature parameter of LLMs
One key hyperparameter that affects the performance of LLMs is temperature, which influences the randomness in the model's predictions. To determine the optimal temperature for each task, we randomly sampled 30 data points from the datasets and performed in-context learning experiments across various temperature settings. While optimal temperatures determined on the validation set may not always yield optimal results on the test set, our methodology is primarily designed to conserve token usage and API query time. To address potential discrepancies between validation and test sets, we performed targeted temperature testing on the test sets for two molecular property prediction datasets: BBBP and BACE. Our results are summarized in Table 3. For these tests, we employed the GPT-4 model (using scaffold sampling with  $k = 8$ ) and set temperature values  $t = [0.2, 0.4, 0.6, 0.8, 1]$ . The result reveals that variations in the temperature parameter have a marginal impact on test performance, with fluctuations of less than 0.05 observed in both F1 and accuracy scores. These results validate the robustness of our initial sampling approach and underscore the reliability of our findings across different settings.
Table 3: The F1(↑) and accuracy(↑) score of GPT-4 model(scaffold sampling,  $k = 8$ ) on different temperature setting.
F1(↑)	BBBP	BACE	Accuracy(↑)	BBBP	BACE
GPT-4(t=0.2)	0.667 ± 0.029	0.741 ± 0.019	GPT-4(t=0.2)	0.650 ± 0.028	0.743 ± 0.019
GPT-4(t=0.4)	0.712 ± 0.014	0.728 ± 0.024	GPT-4(t=0.4)	0.691 ± 0.017	0.729 ± 0.024
GPT-4(t=0.6)	0.683 ± 0.016	0.736 ± 0.020	GPT-4(t=0.6)	0.659 ± 0.016	0.736 ± 0.019
GPT-4(t=0.8)	0.686 ± 0.030	0.744 ± 0.025	GPT-4(t=0.8)	0.661 ± 0.032	0.745 ± 0.025
GPT-4(t=1.0)	0.684 ± 0.023	0.756 ± 0.025	GPT-4(t=1.0)	0.660 ± 0.021	0.757 ± 0.025
5.1 Limitation of LLMs on understanding molecular SMILES
A significant limitation of LLMs is their lack of understanding of molecular representations in SMILES strings, which in many cases leads to inaccurate or inconsistent results as shown in Section A for the translation of different ways to name molecules. SMILES (Simplified Molecular Input Line Entry System) [60, 61] is a widely used textual representation for chemical structures. For example, the SMILES string for ethanol, a simple alcohol, is "CCO". This string represents a molecule with two carbon atoms (C) connected by a single bond and an oxygen atom (O) connected to the second carbon atom. SMILES strings can serve as both input and output for LLMs, alongside other natural language text. However, several issues make it challenging for LLMs to accurately understand and interpret SMILES strings: 1) Hydrogen atoms are not explicitly represented in SMILES strings, as they can be inferred based on the standard bonding rules. LLMs frequently struggle to infer these implicit hydrogen atoms and may even fail at simple tasks like counting the number of atoms in a molecule [27, 6]. 2) A given molecule can have multiple valid SMILES representations, which can lead to ambiguity if not properly processed or standardized. LLMs may thus fail to consistently recognize and compare molecular structures represented by different SMILES strings. 3) LLMs do not have any inherent understanding of SMILES strings, and treat them as a sequence of characters or subwords. When processing long SMILES strings, LLMs rely on the byte-pair encoding tokenization technique, which can break the string into smaller pieces or subwords in ways that do not represent the molecular structure and properties of molecules represented by SMILES strings. Because many tasks in cheminformatics rely on the accurate representation of a molecule by SMILES strings, the non-competitive performance of GPT models in converting structures into SMILES strings (and vice versa) affects downstream tasks such as retrosynthesis, reaction and name prediction. LLMs that have an enhanced ability of handling molecular structures and their specific attributes or coupling to existing tools such as RDKit [35] will be needed.
5.2 The limitations of current evaluation methods
Although in Text-Based Molecule Design and Molecule Captioning tasks, GPT models show competitive performance compared to the baseline in some metrics (BLEU, Levenshtein, ROUGE, FCD, etc), we observe that the exact match of GPT models is inferior to the baseline in the Text-Based Molecule Design task and the GPT models generate some descriptions which violate chemical facts. This divergence between metrics and real-world scenarios mainly arises because, unlike many natural language processing tasks that can be suitably evaluated by sentence-level matching evaluation metrics, chemistry-related tasks necessitate exact matching for SMILES and precise terminology in descriptions. These findings spotlight the limitations of current evaluation metrics and underscore the need for the development of chemistry-specific metrics.
5.3 Hallucination of LLMs in chemistry
Our evaluation experiments across various tasks reveal two primary types of hallucinations exhibited by LLMs in the domain of chemistry. The first type occurs when the input is given in SMILES format (e.g., name prediction); LLMs occasionally struggle with interpreting these SMILES correctly. For instance, they may fail to recognize the number of atoms or certain functional groups within molecules during name prediction tasks. The second type of hallucination arises when the expected output from LLMs should be in the form of SMILES (e.g., reaction prediction and retrosynthesis). Here, LLMs may produce molecules that are chemically unreasonable, suggesting a gap in understanding what constitutes valid SMILES. Hallucination issues represent a key challenge with LLMs, particularly in the field of chemistry which necessitates exact matching of SMILES and adherence to strict chemical facts [62]. Current LLMs need further investigation into this problem.
5.4 Prospects of LLMs for chemistry
Overall, through an exhaustive set of experiments and analyses, we outline several promising avenues for the application of LLMs in the field of chemistry. While LLMs underperform relative to baselines across a majority of tasks, it's important to note that LLMs leverage only a few examples to solve chemistry problems, whereas baselines are trained on extensive, task-specific datasets and are limited
to certain tasks. This observation provides valuable insights into the potential of LLMs' generalized intelligence in the domain of chemistry. The employment of advanced prompting techniques such as Chain-of-thought (CoT) [59], Decomposed Prompting [31] could potentially boost the capacity of LLMs to perform complex reasoning. On the other hand, LLMs display a considerable amount of hallucinations in chemistry tasks, indicating that current LLMs may not yet possess the necessary capabilities to solve practical chemistry problems effectively. However, with continuous development of LLMs and further research into methods to avoid hallucinations, we are optimistic that LLMs can significantly enhance their problem-solving abilities in the field of chemistry.
5.5 Impact of generating harmful chemicals
Our work demonstrate that LLMs can generate chemically valid molecules. However, it's crucial to acknowledge and mitigate the risks of AI misuse, such as generating hazardous substances. While advancements in AI-enabled chemistry have the potential to bring about groundbreaking medicines and sustainable materials, the same technology can be misused to create toxic or illegal substances. This dual-edged potential emphasizes the necessity for stringent oversight. Without careful regulation, these tools could not only pose significant health and safety hazards but also create geopolitical and security challenges. Consequently, as we harness the capabilities of LLMs in the field of chemistry, we concur with earlier research on generative models in chemistry [2, 3] that it is vital for developers to establish robust safeguards and ethical guidelines to deter harmful applications. This is akin to the limitations imposed on popular search engines, which can also be exploited to find information about dangerous chemicals or procedures online.
5.6 Broader Impacts
Our work has broad impacts across multiple dimensions. First, it offers valuable insights and recommendations for both AI researchers and chemists in academia and industry. These perspectives enhance the effective utilization of LLMs and guide future advancements in the field. Second, our objective evaluation of LLMs helps alleviate concerns regarding the replacement of chemists by AI. This aspect contributes to public education, addressing misconceptions and fostering a better understanding of the role of AI in chemistry. Furthermore, we provide a comprehensive experimental framework for testing LLMs in chemistry tasks, which can also be applicable to other domains. This framework serves as a valuable resource for researchers seeking to evaluate LLMs in diverse fields. However, it is important to recognize the ethical and societal implications associated with our work. Additionally, concerns about job displacement in the chemical industry may arise, and efforts should be made to address these challenges and ensure a responsible and equitable adoption of AI technologies.
In this paper, we summarize the required abilities of LLMs in chemistry and construct a comprehensive benchmark to evaluate the five most popular LLMs (GPT-4, GPT-3.5, Davinci-003, LLama and Galactica) on eight widely-used chemistry tasks. The experiment results show that LLMs perform less competitive in generative tasks which require in-depth understanding of molecular SMILES strings, such as reaction prediction, name prediction, and retrosynthesis. LLMs show competitive performance in tasks that are in classification or ranking formats such as yield prediction and reagents selection. LLMs are selectively competitive on tasks involving text in prompts such as property prediction and text-based molecule design, or explainable tasks such as molecule captioning. These experiments indicate the potential of LLMs in chemistry tasks and the need for further improvement. We will collaborate with more chemists in the C-CAS group, progressively integrating a wider range of tasks that are both novel and practical. We hope our work can address the gap between LLMs and the chemistry research field, inspiring future research to explore the potential of LLMs in chemistry.
This work was supported by the National Science Foundation (CHE-2202693) through the NSF Center for Computer Assisted Synthesis (C-CAS).
[52] Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021.
[53] Nadine Schneider, Nikolaus Stiefl, and Gregory A Landrum. What's what: The (nearly) definitive guide to reaction role assignment. Journal of chemical information and modeling, 56 (12):2336-2346, 2016.
[54] Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha A Lee. Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction. ACS central science, 5(9):1572-1583, 2019.
[55] Taffee T Tanimoto. Elementary mathematical theory of classification and prediction. Journal of Biomedical Science and Engineering, 1958.
[56] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
[57] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[58] Hongwei Wang, Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, and Martin D Burke. Chemical-reaction-aware molecule representation learning. arXiv preprint arXiv:2109.09888, 2021.
[59] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
[60] David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. J. Chem. Inf. Comput. Sci., 28:31-36, 1988.
[61] David Weininger, Arthur Weininger, and Joseph L. Weininger. Smiles. 2. algorithm for generation of unique smiles notation. J. Chem. Inf. Comput. Sci., 29:97-101, 1989.
[62] A.D. White. The future of chemistry is language., 2023.
[63] Andrew D. White, Glen M. Hocky, Heta A. Gandhi, Mehrad Ansari, Sam Cox, Geemi P. Wellawatte, Subarna Sasmal, Ziyue Yang, Kangxin Liu, Yuvraj Singh, and Willmor J. Peña Ccoa. Assessment of chemistry knowledge in large language models that generate code. Digital Discovery, 2:368-376, 2023. doi: 10.1039/D2DD00087C. URL http://dx.doi.org/10.1039/D2DD00087C.
[64] Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Are multilingual models effective in code-switching? arXiv preprint arXiv:2103.13309, 2021.
[65] Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513-530, 2018.
[66] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.
A Name Prediction
For one molecule, there are different chemical naming conventions and representations such as SMILES, IUPAC names, and graphic molecular formula. To investigate whether GPT models have the basic chemical name understanding ability, we construct four chemical name prediction tasks that include SMILES to IUPAC name translation (smiles2iupac), IUPAC name to SMILES translation (iupac2smiles), SMILES to molecule formula translation (smiles2formula), and IUPAC name to molecule formula translation (iupac2formula). We collect 630 molecules and their corresponding names including SMILES, IUPAC name, and molecule formula from PubChem [32]. We randomly sample 500 molecules as the ICL candidates, and other 30 molecules as the validation set, and other 100 molecules as the test set. For all name translation tasks, we use the exact match accuracy as the metric to evaluate the performance.
ICL Prompt. One example of the smiles2iupac prediction is shown in Figure 5. For other name translation tasks, we only change the underlined parts that represent different tasks and their corresponding input names and output names.
Figure 5: An ICL prompt example for smiles2iupac prediction
Results. The results are reported in Table 4 (we only report representative methods along with their optimal prompt settings via grid search on validation set). In all four name prediction tasks, the accuracy of the best method is extremely low (0.014 in the iupac2smiles task, 0.086 in the smiles2formula task, 0.118 in the iupac2formula task) or even 0 (in the smiles2iupac task). This indicates the LLMs lack basic chemical name understanding ability. The accuracy of Davinci-003 is considerably inferior to other models.
Case studies. Example results generated by GPT-4 (Scaffold,  $k = 20$ ) method for each task is shown in Table 5. In all tasks, the GPT-4 model gives the wrong answers. In the smiles2formula task, we can observe that GPT models cannot even recognize the number of Carbon and infer the correct number of Hydrogen, demonstrating the bad chemical understanding ability of GPT models. For prospects, some pre-training technologies such as wrapping molecules with text [38] or code-switch [64, 20] may be helpful to align different chemical names of the same molecule to help improve LLMs' chemical understanding.
<sup>3</sup>https://pubchem.ncbi.nlm.nih.gov
Table 4: The accuracy  $(\uparrow)$  of LLMs in 4 different name prediction tasks. The best LLM is in bold font. Here  $k$  is the number of examples used in few-shot ICL. The baseline is underlined and "-" indicates that STOUT cannot solve the smiles2formula and iupac2formula tasks.
Method	smiles2iupac	iupac2smiles	smiles2formula	iupac2formula
STOUT [47]	0.55	0.7	-	-
GPT-4 (zero-shot)	0	0.008±0.008	0.048±0.022	0.092±0.018
GPT-4 (Scaffold, k=5)	0	0.014±0.009	0.058±0.015	0.118±0.022
GPT-4 (Scaffold, k=20)	0	0.012±0.004	0.086±0.036	0.084±0.005
GPT-4 (Random, k=20)	0	0.010±0.007	0.070±0.032	0.076±0.011
GPT-3.5 (Scaffold, k=20)	0	0.010±0.000	0.052±0.004	0.044±0.009
Davinci-003 (Scaffold, k=20)	0	0	0.006±0.005	0.018±0.004
Llama2-13B-chat (Scaffold, k=20)	0	0	0.010±0.007	0
GAL-30B (Scaffold, k=10)	0	0	0	0
Table 5: Example results generated by GPT-4 (Scaffold,  $k = {20}$  ) method for different tasks
Task	Input	Ground Truth	Output of GPT-4 (Scaffold, k=20)
smiles2iupac	CCOC(=O)C(C=C)=O=C(C)N	ethyl 2-acetyl-3-aminobut-2-enoate	ethyl 2-methyl-5-oxo-2-azahept-4-en-3-oate
iupac2smiles	ethyl 2-acetyl-3-aminobut-2-enoate	CCOC(=O)C(C=C)=O=C(C)N	CCOC(=O)C=C(C)C(=N)C
smiles2formula	Cc1noc(CCn2cc[h]c2=O)n1	C8H10N4O2	C9H10N4O2
iupac2formula	R)-(1-benzylquinolin-1-ium-4-yl)-(5-ethenyl-1-azabicyclo[2.2.2]octan-2-yl)methanol:chloride	C26H29CIN2O	C23H27CIN2O
B Molecule Property Prediction