SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Molecule property prediction [21, 58] is a fundamental task in computational chemistry that has been gaining significant attention in recent years due to its potential for drug discovery, material science, and other areas in the chemistry. The task involves using machine learning techniques [22] to predict the chemical and physical properties of a given molecule, based on its molecular structure. We aim to further explore the potential of LLMs in molecular property prediction and assess their performance on a set of benchmark datasets, such as BBBP/MIT license), HIV/MIT license), BACE/MIT license), Tox21(MIT license), and ClinTox(MIT license), which were originally introduced by [65]. The datasets are made up of extensive collections of SMILES, paired with binary labels that highlight the particular property being evaluated, such as BBBP: Blood-Brain Barrier Penetration, HIV: inhibit HIV replication, BACE: bindings results for a set of inhibitors of human beta-secretase, Tox21: toxicity of compounds, and ClinTox: drugs failed clinical trials for toxicity reasons. A comprehensive explanation of these datasets can be referenced in the original research conducted by [65]. For ICL, we either select  $k$  samples randomly, or search the top- $k$  most analogous molecules using RDKit [35] to determine the Tanimoto Similarity. However, it is crucial to mention that using the latter method does not assure an even distribution among classes. In our study, we employ a strategic sampling method for two categories of datasets: balanced and highly imbalanced. For balanced datasets, such as BBBP and BACE, we randomly select 30 samples for the validation process and 100 samples for testing from the original dataset. Contrastingly, for datasets exhibiting substantial label imbalance  $(39684:1443\approx 28:1)$  , take HIV datasets as a example), we select samples from the majority and minority classes to achieve a ratio of 4:1. This strategic approach enables us to maintain a representative sample for the evaluation process, despite the original high imbalance in the dataset. To evaluate the results, we use the classification accuracy, as well as  $F1$  score as the evaluation metric due to the class imbalance. We benchmark our method against two established baselines from MoleculeNet [65]: RF and XGBoost. Both baselines utilize the 1024-bit circular fingerprint as input to predict the property as a binary classification problem.
ICL Prompt. Figure 6 illustrates a sample of our ICL prompt for property prediction. Within the task-specific template, we include a detailed explanation of the task forecasting the penetration of the brain-blood barrier to assist LLMs in comprehending the input SMILES from the BBBP dataset. Additionally, we establish certain constraints for the output to conform to the specific characteristics of the property prediction task.
Results. The results are reported as F1 in Table 6, accuracy in Table 7. We observed that GPT models outperform the baseline model in terms of F1 on four out of five datasets. In the range of GPT
Figure 6: An ICL prompt example for property prediction
Table 6: F1 (↑) score of LLMs and baseline in molecular property prediction tasks.  $k$  is the number of examples used in few-shot ICL. The best GPT model is in bold font, and the baseline is underlined.
BBBP	BACE	HIV	Tox21	ClinTox
RF	0.881	0.758	0.518	0.260	0.461
XGBoost	0.897	0.765	0.551	0.333	0.620
GPT-4 (zero-shot)	0.560 ± 0.034	0.322±0.018	0.977±0.013	0.489±0.018	0.555±0.043
GPT-4 (Scaffold, k= 4)	0.498 ± 0.028	0.516 ± 0.024	0.818 ± 0.015	0.444 ± 0.004	0.731 ± 0.035
GPT-4 (Scaffold, k= 8)	0.587±0.018	0.666±0.023	0.797 ± 0.021	0.563±0.008	0.736±0.033
GPT-4 (random, k= 8)	0.469 ± 0.025	0.504 ± 0.020	0.994 ± 0.006	0.528±0.003	0.924±0.000
GPT-3.5 (Scaffold, k= 8)	0.463 ± 0.008	0.406 ± 0.011	0.807 ± 0.021	0.529 ± 0.021	0.369 ± 0.029
Davinci-003 (Scaffold, k= 8)	0.378 ± 0.024	0.649 ± 0.021	0.832 ± 0.020	0.518±0.009	0.850 ± 0.020
Llama2-13B-chat (Scaffold, k= 8)	0.002 ± 0.001	0.045 ± 0.015	0.069 ± 0.033	0.047 ± 0.013	0.001 ± 0.003
GAL-30B (Scaffold, k= 8)	0.074 ± 0.019	0.025 ± 0.013	0.014 ± 0.016	0.077 ± 0.046	0.081 ± 0.015
Table 7: Accuracy  $(\uparrow)$  of LLMs and baseline in molecular property prediction tasks.  $k$  is the number of examples used in few-shot ICL. The best GPT model is in bold font, and the baseline is underlined.
BBBP	BACE	HIV	Tox21	ClinTox
RF	0.820	0.790	0.870	0.830	0.858
XGBoost	0.850	0.810	0.870	0.840	0.888
GPT-4 (zero-shot)	0.476 ± 0.036	0.499 ± 0.005	0.986 ± 0.007	0.518 ± 0.018	0.736 ± 0.027
GPT-4 (Scaffold, k= 4)	0.516 ± 0.022	0.514 ± 0.205	0.834 ± 0.014	0.457 ± 0.004	0.856 ± 0.014
GPT-4 (Scaffold, k= 8)	0.614±0.016	0.679±0.205	0.836 ± 0.020	0.737 ± 0.004	0.856 ± 0.014
GPT-4 (random, k= 8)	0.610 ± 0.021	0.588 ± 0.023	0.996±0.004	0.874±0.003	0.930±0.010
GPT-3.5 (Scaffold, k= 8)	0.463 ± 0.007	0.496 ± 0.016	0.864 ± 0.018	0.572 ± 0.026	0.578 ± 0.029
Davinci-003 (Scaffold, k= 8)	0.396 ± 0.023	0.650 ± 0.021	0.781 ± 0.004	0.682 ± 0.006	0.845 ± 0.010
Llama2-13B-chat (Scaffold, k= 8)	0.002 ± 0.003	0.048 ± 0.017	0.048 ± 0.025	0.053 ± 0.011	0.002 ± 0.004
GAL-30B (Scaffold, k= 8)	0.062 ± 0.007	0.020 ± 0.010	0.012 ± 0.009	0.030 ± 0.018	0.099 ± 0.007
models examined, GPT-4 surpasses both Davinci-003 and GPT-3.5 in predicting molecular properties. In our investigation, we have found evidence to support that the expansion of in-context learning (ICL) instances leads to a measurable enhancement in model performance. This underlines a direct relationship between the extent of ICL data and the predictive precision of our models. Concurrently, our research presents empirical evidence that scaffold sampling exceeds the performance of random
sampling on three distinct datasets (BBBP, BACE, Tox21). A plausible explanation for this could be the structural resemblances between the scaffold-sampled molecules and the query molecule, which potentially biases the GPT models towards more accurate decision.
Label interpretation. The results presented in Table 6 and Table 7 indicate that the GPT-4 model selectively outperforms the baseline models on the HIV and ClinTox datasets. This superior performance likely stems from the inclusion of information directly related to the labels within the ICL prompts. Specifically, in the HIV dataset, the activity test results play a crucial role. Molecules tend to inhibit HIV replication when the activity test is categorized as "confirmed active" or "confirmed moderately active." For the ClinTox dataset, the FDA approval status of a molecule acts as a predictor of its clinical toxicity. A molecule not having FDA approval is more likely to be clinically toxic. In experiments where we excluded this contextual information from the in-context learning prompts, the F1 and accuracy score of predictions notably declined, as evident from the results in Table 8 and Table 9.
Table 8: Impact to F1 score of removing label context information from the in-context learning prompts.
F1(↑)	HIV	ClinTox
GPT-4(zero-shot)	0.977 ± (0.013)	0.489 ± (0.018)
GPT-4(unlabelled, zero-shot)	0.554 ± (0.017)	0.438 ± (0.045)
GPT-4(few-shot)	0.797 ± (0.021)	0.563 ± (0.008)
GPT-4(unlabelled, few-shot)	0.493 ± (0.030)	0.478 ± (0.035)
Table 9: Impact to accuracy of removing label context information from the in-context learning prompts.
Accuracy(↑)	HIV	ClinTox
GPT-4(zero-shot)	0.986 ± (0.070)	0.736 ± (0.027)
GPT-4(unlabelled, zero-shot)	0.628 ± (0.016)	0.602 ± (0.039)
GPT-4(few-shot)	0.836 ± (0.020)	0.856 ± (0.014)
GPT-4(unlabelled, few-shot)	0.541 ± (0.032)	0.630 ± (0.014)
C Yield Prediction
Yield prediction [51] is a critical task in chemistry, specifically in the domain of synthetic chemistry, which involves the design and synthesis of new compounds for various applications, such as pharmaceuticals, materials, and catalysts. The yield prediction task aims to estimate the efficiency and effectiveness of a chemical reaction, primarily by quantifying the percentage of the desired product formed from the reactants. We use two High-Throughput experimentation (HTE) datasets: Buchwald-Hartwig [1] (MIT license) and Suzuki-Miyaura dataset [50] (MIT license) for evaluation. These datasets consist of reactions and their corresponding yields, which have been meticulously acquired through standardized and consistent experimental setups. This uniformity ensures that the data within each dataset is coherent, reducing the likelihood of discrepancies arising from variations in experimental procedures or conditions. We formulate the task of yield prediction as a binary classification problem, by determining whether a reaction is a high-yielding reaction or not. We used only random sampling for our ICL examples as reactions in those datasets belong to the same type. For every dataset, we randomly select 30 samples for the validation process and 100 samples for testing from the original dataset. To evaluate the results, we use the classification accuracy as the evaluation metric, with UAGNN [34] serving as baseline. UAGNN reports state-of-the-art performance on yield prediction. It takes the graphs of reactants and products as input, and learns representation of these molecules through a graph neural network, and then predicts the scaled yield.
ICL prompt. We show our ICL prompt for yield prediction with an example from Buchwald-Hartwig dataset. As described in Figure 7, we incorporate an input explanation (wherein the reactants are separated by ‘.’ and the products are split by ‘ $>>$ ’) to assist large language models. Additionally, output restrictions are enforced to ensure the generation of valid results.
Figure 7: An ICL prompt example for yield prediction
Results. The results are presented in Table 10. Our analysis reveals that in the task of yield prediction, GPT models perform below the established baseline model, UAGNN. However, it's worth noting that the UAGNN model was trained on the full training dataset including thousands of examples. Considering the spectrum of GPT models under scrutiny, GPT-4 emerges as the superior model, overshadowing both Davinci-003 and GPT-3.5 in predicting reaction yields. In the process of our investigation, we unearthed supporting evidence that signifies the role of ICL instances in the enhancement of model performance. This suggests an inherent correlation between the quantity of ICL data and the predictive accuracy of the models under consideration. This phenomenon is particularly in the case of GPT-4, we observed a significant improvement in performance when the number of ICL examples was increased from 4 to 8, both in the Buchwald-Hartwig and Suzuki-coupling reactions. This indicates that even within the same model architecture, the amount of contextual data can significantly influence the predictive capabilities.
Table 10: Accuracy  $(\uparrow)$  of yield prediction task.  $k$  is the number of examples used in few-shot ICL. The best LLM is in bold font, and the baseline is underlined.
Buchwald-Hartwig	Suzuki-coupling
UAGNN [34]	0.965	0.957
GPT-4 (zero-shot)	0.322 ± 0.034	0.214 ± 0.019
GPT-4 (random, k=8)	0.800±0.008	0.764±0.013
GPT-4 (random, k=4)	0.574 ± 0.045	0.324 ± 0.018
GPT-3.5 (random, k=8)	0.585 ± 0.045	0.542 ± 0.011
Davinci-003 (random, k=8)	0.467 ± 0.013	0.341 ± 0.017
Llama2-13B-chat	0.008 ± 0.007	0.006 ± 0.004
GAL-30B	0	0.008 ± 0.010
D Reaction Prediction
Reaction prediction is a central task in the field of chemistry, with significant implications for drug discovery, materials science, and the development of novel synthetic routes. Given a set of reactants, the goal of this task is to predict the most likely products formed during a chemical reaction [54, 13, 19]. In this task, we use the widely adopted USPTO-MIT dataset [29](MIT license) to evaluate the performance of GPT models. This dataset contains approximately 470,000 chemical reactions extracted from US patents. In the experiment, we used the USPTO mixed data set, where the reactants and reagents strings are split by ‘:’. We randomly sampled 30 samples from the original validation set for validation and 100 samples from the original test set for testing. We use the Top-1 Accuracy as the evaluation metric and Chemformer [26] as the baseline due to its superior performance among the machine learning solutions for reaction prediction. Chemformer is a seq2seq model trained to predict the output product when given reactants and reagents as input. We also report the percentage of invalid SMILES generated by each method.
Figure 8: An ICL prompt example for reaction prediction
ICL Prompt. One example of our ICL prompt for reaction prediction is shown in Figure 8. Given the nature of the reaction prediction task and the characteristics of the USPTO-MIT dataset, we enhance the task-specific template with an input explanation (stating that the input includes reactants and reagents, which are separated by ‘.’) to assist the GPT models in understanding the input SMILES. Moreover, we incorporate output restrictions to guide GPT models in generating chemically valid and reasonable products.
Table 11: The performance of LLMs and baseline in the reaction prediction task.  $k$  is the number of examples used in few-shot ICL. The best LLM is in bold font, and the baseline is underlined.
Method	Top-1 Accuracy (↑)	Invalid SMILES (↓)
Chemformer [26]	0.938	0%
GPT-4 (zero-shot)	0.004 ± 0.005	17.4% ± 3.9%
GPT-4 (Scaffold, k=20)	0.230 ± 0.022	7.0% ± 1.6%
GPT-4 (Random, k=20)	0.012 ± 0.008	8.4% ± 1.5%
GPT-4 (Scaffold, k=5)	0.182 ± 0.015	6.6% ± 1.5%
GPT-3.5 (Scaffold, k=20)	0.184 ± 0.005	15.6% ± 2.3%
Davinci-003 (Scaffold, k=20)	0.218 ± 0.008	11.4% ± 2.7%
Llama2-13B-chat (Scaffold, k=20)	0.032 ± 0.013	27.8% ± 5.5%
GAL-30B (Scaffold, k=5)	0.036 ± 0.011	5.2% ± 1.5%
Results. The results are reported in Table 11. We can observe that compared to the baseline, the performance of GPT models is considerably inferior, especially for the Zero-shot prompting (Top-1 Accuracy is only 0.004 and it generates  $17.4\%$  invalid SMILES). The less competitive results of GPT models can be attributed to the lack of in-depth understanding of the SMILES strings that represent reactants and products, as well as the reaction process that transforms reactants into products. It is also worth mentioning that the high accuracy achieved by Chemformer is due to its training on the complete dataset. More conclusions and detailed analysis are summarized in the section 5.
E Reagents Selection
Reagents selection, also known as reagent recommendation, involves the identification and proposal of the most fitting reagents for a specific chemical reaction or process. Compared to other prediction and generation tasks, these selection tasks might be more fitting for LLMs and carry extensive implications. Reagent recommendation can markedly enhance reaction design by pinpointing optimal reagents and conditions for a given reaction, thereby augmenting efficiency and effectiveness in both academic and industrial settings. Drawing from a vast corpus of chemical knowledge, GPT models may be able to generate suggestions, leading to chemical reactions with a greater likelihood of yielding superior results.
In this study, we formulate four reaction component selection task from the Suzuki High-Throughput Experimentation (HTE) dataset. The dataset, created by Perera et al[44](MIT license), evaluates the Suzuki coupling of 5 electrophiles and 7 nucleophiles across a matrix of 11 ligands (with one blank), 7 bases (with one blank), and 4 solvents, resulting in a reaction screening dataset comprising 5,760 data points. The task of reagents selection can be divided into three categories: Reactant selection, Ligand Selection and Solvent selection. For validation, 30 examples were randomly sampled, while 100 examples were used for testing, all taken from the original datasets. Top-1 Accuracy serves as the assessment metric for both reactant and solvent selection, while Top-50% is utilized for ligand selection, as the upper half of the ligands in the list typically provide satisfactory yields in chemical reactions. This task is newly emergent in the field of chemistry, and as such, there are no established baselines yet.
ICL prompt. One example of our ICL prompt for reagents selection is shown in Figure 9. Considering the structure of the dataset and the characteristics of the reagents, we provide detailed task description and an answer template to guide GPT models towards the desired output.
Results. Our results are presented in Table 12. From the table, it is evident that GPT-4 and GPT-3.5 perform comparatively well in reagent selection tasks. This suggests a promising potential for GPT models in the realm of reagent selection.
Table 12: Accuracy  $(\uparrow)$  of LLM in the reagent selection tasks. For Reactant Selection and Solvent selection task, we report the mean (and standard deviation) of the Top-1 Accuracy score and we report the Top-50% accuracy score for the Ligand Selection task. The best LLM is in bold font, and the baseline is underlined.
Reactant Selection	Solvent Selection	Ligand Selection
GPT-4 (zero-shot)	0.299 ± 0.029	0.526±0.012	0.534±0.059
GPT-3.5 (zero-shot)	0.400±0.038	0.368±0.034	0.436 ± 0.020
Davinci-003 (zero-shot)	0.178 ± 0.034	0.463 ± 0.014	0.432 ± 0.020
Llama2-13B-chat (zero-shot)	0.145 ± 0.000	0.050 ± 0.010	0.284 ± 0.024
GAL-30B (zero-shot)	0.107 ± 0.020	0.104 ± 0.004	0.030 ± 0.016
F Retrosynthesis
Retrosynthesis planning is a crucial task in synthetic organic chemistry that involves identifying efficient synthetic pathways for a target molecule by recursively transforming it into simpler precursor molecules. In contrast to reaction prediction, retrosynthesis planning involves a reverse extrapolation from the target molecule to identify the readily available reactants for its synthesis. In this study, we use the USPTO-50k dataset [53](MIT license), which contains 50,037 chemical reactions. In our
Figure 9: An ICL prompt example for reagents selection
experiment, we use the data splitting as [17] and we the training set which contains 40,029 reactions as the ICL candidates. The metric and baseline are the same as the reaction prediction.
ICL Prompt. One example of our ICL prompt for reaction prediction is shown in Figure 10. As discussed in the reaction prediction task, we also add the task-specific template to help GPT models understand the input and restrict the output.
Figure 10: An ICL prompt example for Retrosynthesis
Table 13: The performance of LLMs and baseline in Retrosynthesis task. The best LLM is in bold font, and the baseline is underlined.
Method	Top-1 Accuracy (↑)	Invalid SMILES (↓)
Chemformer [26]	0.536	0%
GPT-4 (zero-shot)	0.006 ± 0.005	20.6% ± 4.7%
GPT-4 (Scaffold, k=20)	0.096 ± 0.013	10.4% ± 3.4%
GPT-4 (Scaffold, k=5)	0.114 ± 0.013	11.0% ± 1.2%
GPT-4 (Random, k=20)	0.012 ± 0.011	18.2% ± 4.2%
GPT-3.5 (Scaffold, k=20)	0.022 ± 0.004	6.4% ± 1.3%
Davinci-003 (Scaffold, k=20)	0.122 ± 0.013	6.0% ± 1.2%
Llama2-13B-chat (Scaffold, k=20)	0	27.2% ± 1.5%
GAL-30B (Scaffold, k=5)	0.016 ± 0.005	5.2% ± 1.8%
Results. The results are reported in Table 13. The performance of GPT models is also inferior than the baseline due to the lack of an in-depth understanding of the SMILES strings that represent reactants and products. Detailed analysis are summarized in the later section 5 Discussion.
G Text-Based Molecule Design
Text-Based Molecule Design is a novel task in computational chemistry and drug discovery. It involves generating new molecules with desired molecule descriptions. In our experiment, we employ the ChEBI-20 dataset which consists of 33,010 molecule-description pairs. The dataset is split into  $80/10/10\%$  as the training/validation/test set [17](CC BY 4.0). We use the training set which contains 26407 molecule-description pairs as the ICL candidates. For comparison, we use the MolT5-Large [17] as the baseline. MolT5-Large is the initial effort to investigate the translation between molecules and text, including tasks such as text-based molecule design and molecule captioning. It builds upon T5 [46], an encoder-decoder Transformer model, and benefits from pretraining on a large amount of dataset. To comprehensively evaluate the performance, we employ three different types of metrics. The first type of metric is the chemical similarity between the ground-truth molecules and generated molecules, measured by FTS (fingerprint Tanimoto Similarity) [55] in terms of MACCS [49], RDK [35], and Morgan [14]. Secondly, we also use FCD (Fréchet ChemNet Distance) [45] which allows comparing molecules based on the latent information used to predict the activity of molecules [17]. Since the generated molecules are in SMILES string format, we also employ natural language processing metrics including BLEU, Exact Match [17], and Levenshtein distance [40] between the ground-truth molecules and generated molecules SMILES. Finally, to evaluate whether generated molecules are valid, we use RDKIT [35] to check the validity of generated molecules and report the percent of the valid molecules.
ICL Prompt. One ICL prompt example for text-based molecule design is shown in Figure 11.
Table 14: The performance of LLMs and baseline in the Text-Based Molecule Design task. The best LLM is in bold font and the baseline is underlined.
Method	BLEU (↑)	Exact (↑)	Levenshtein (↓)	Validity (↑)	MACCS FTS (↑)	RDK FTS (↑)	Morgan FTS (↑)	FCD (↓)
MolT5-Large [17]	0.601	0.290	41.600	0.940	0.879	0.797	0.752	5.394
GPT-4(zero-shot)	0.490±0.017	0.046±0.009	47.418±1.668	0.758±0.015	0.733±0.020	0.514±0.021	0.432±0.014	11.913±0.972
GPT-4(Scaffold, k=10)	0.816±0.004	0.174±0.029	21.160±0.600	0.888±0.023	0.867±0.005	0.738±0.010	0.672±0.013	6.224±0.449
GPT-4(Scaffold, k=5)	0.815±0.011	0.164±0.018	21.862±1.768	0.874±0.030	0.865±0.015	0.741±0.023	0.670±0.028	5.843±0.515
GPT-4(Random, k=10)	0.602±0.016	0.060±0.007	42.390±1.008	0.770±0.030	0.762±0.013	0.548±0.017	0.475±0.015	10.594±0.414
GPT-3.5(Scaffold, k=10)	0.479±0.156	0.094±0.011	82.008±40.354	0.854±0.059	0.833±0.006	0.686±0.016	0.585±0.013	8.341±0.607
Davinci-003(Scaffold, k=10)Llama2-13B-chat(Scaffold, k=10)	0.741±0.011	0.100±0.010	25.648±2.186	0.936±0.009	0.783±0.014	0.648±0.004	0.560±0.010	8.335±0.310
0.626±0.013	0.020±0.000	33.956±2.648	0.782±0.008	0.679±0.015	0.568±0.014	0.454±0.009	12.387±0.437
GAL-30B(zero-shot)	0.004±0.000	0.000±0.000	2738.136±166.093	0.956±0.011	0.233±0.011	0.109±0.006	0.053±0.002	35.091±0.774
Figure 11: An ICL prompt example for Text-Based Molecule Design
Results. The results are reported in Table 14. We can observe that the best ICL prompting GPT models (GPT-4 and Davinci-003) can achieve competitive performance or even outperform the baseline in some metrics (BLEU, Levenshtein). Although the GPT models significantly underperform the baseline in terms of exact match and Morgan FTS metrics, it's important to note that we only utilize a maximum of 10 examples, which is substantially less than the training set (comprising 26,407 training examples) used for the baseline. These results demonstrate the strong few-shot text-based molecule design ability of GPT models. Last, not being exactly the same as the ground truth doesn't necessarily mean it's incorrect, especially in the context of molecular design. The molecules generated by GPT models may still be useful and can serve as alternatives to the ground truth, given they fulfill the requirements described in the input text and a majority (over  $89\%$ ) are chemically valid.
Case studies. We select three different types of molecules (organic molecule without rings, organic molecule with ring, and metal atom) as examples, and show the generated molecules in Figure 12. We observe that the structure of molecules generated by the GPT-4 (Scaffold,  $k = 10$ ) method is more similar to the ground truth compared to Davinci-003, GPT-4 (zero-shot), and even the baseline. Additionally, for metal atoms design, GPT models outperform the baseline which wrongly generates the SMILES instead of the metal atom. These cases show promising results of the molecule design ability of GPT models. However, evaluating whether the generated molecules are helpful such as molecule novelty in real-world scenarios is still a difficult problem. Thus we conclude that GPT models have excellent potential in molecule design and there are prospects for investigating this ability.
H Molecule Captioning