SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Molecule captioning is an important task in computational chemistry, offering valuable insights and applications in various areas such as drug discovery, materials science, and chemical synthesis. Given a molecule as input, the goal of this task is to generate a textual description that accurately describes the key features, properties, and functional groups of the molecule. We also use the ChEBI-20 dataset(CC BY 4.0) and the training set of it as the ICL candidates as discussed in the Text-Based Molecule Design Section. We use traditional captioning metrics including BLEU, ROUGE, and METEOR for evaluation.
ICL Prompt. One example of our ICL prompt for molecule captioning is shown in Figure 13.
Figure 12: Examples of molecules generated by different models.
Results. The results are reported in Table 15. We can observe that the best ICL prompting GPT models (GPT-4 and Davinci-003) can achieve competitive performance or even outperform the baseline in some metrics (BLEU-2 and BLEU-4). This indicates the inspiring capability of the GPT models in the molecule captioning task.
Figure 13: An ICL prompt example for molecule captioning
Case studies. Same as case studies in the Text-Based Molecule Design task, we also select three different types of molecules as examples, and the captions are shown in Figure 14. We observe that although the performance of the baseline is close to GPT models, the captions generated by the baseline contain more descriptions that violate the chemical facts. In contrast, the captions generated by GPT-4 models contain only a few inaccurate descriptions, highlighting the excellent explaining ability of GPT models. This highlights the limitations of applying traditional Natural Language Processing (NLP) evaluation metrics to this task. Therefore, it is necessary to create more suitable evaluation metrics for chemistry-related generation tasks.
Table 15: The performance of LLMs and baseline in the molecule captioning task. The best LLM is in bold font and the baseline is underlined.
Method	BLEU-2 (↑)	BLEU-4 (↑)	ROUGE-1 (↑)	ROUGE-2 (↑)	ROUGE-L (↑)	METEOR (↑)
MolT5-Large [17]	0.482	0.383	0.574	0.410	0.516	0.530
GPT-4(zero-shot)	0.062±0.001	0.013±0.001	0.192±0.002	0.040±0.002	0.125±0.002	0.209±0.002
GPT-4(Scaffold, k=10)	0.464±0.008	0.365±0.008	0.545±0.003	0.362±0.003	0.459±0.007	0.519±0.005
GPT-4(Scaffold, k=5)	0.456±0.003	0.357±0.004	0.540±0.005	0.355±0.007	0.455±0.005	0.505±0.005
GPT-4(Random, k=10)	0.260±0.007	0.140±0.007	0.393±0.004	0.180±0.006	0.309±0.004	0.320±0.007
GPT-3.5(Scaffold, k=10)	0.468±0.010	0.368±0.010	0.534±0.005	0.355±0.007	0.457±0.006	0.497±0.005
Davinci-003(Scaffold, k=10)	0.488±0.011	0.391±0.012	0.532±0.008	0.359±0.010	0.465±0.008	0.478±0.011
Llama2-13B-chat(Scaffold, k=10)	0.197±0.005	0.140±0.004	0.331±0.005	0.193±0.005	0.265±0.005	0.372±0.006
GAL-30B(zero-shot)	0.008±0.000	0.002 ± 0.000	0.019±0.002	0.004±0.000	0.015±0.002	0.043±0.002
Figure 14: Examples captions generated by different models. Descriptions that violate chemical facts are marked in grey.
I The comparison of SMILES and SELFIES
Table 16: F1 (↑) score of SMILES and SELFIES of GPT-4 model in molecular property prediction tasks.
BBBP	BACE	HIV	Tox21	ClinTox
SMILES	0.587 ± 0.018	0.666 ± 0.023	0.797 ± 0.021	0.563 ± 0.008	0.736 ± 0.033
SELFIES	0.541 ± 0.001	0.601 ± 0.036	0.784 ± 0.014	0.478 ± 0.011	0.654 ± 0.025
Table 17: Performance of SMILES and SELFIES of GPT-4 model in reaction prediction task.
Top-1 Accuracy (↑)	Invalid SMILES/SELFIES (↓)
SMILES	0.230 ± 0.022	7.0% ± 1.6%
SELFIES	0.110 ± 0.007	1.0% ± 0.0%
Table 18: Performance of SMILES and SELFIES of GPT-4 model in molecule design task.
BLEU (↑)	Exact (↑)	Levenshtein (↓)	Validity (↑)	MACCS FTS (↑)	RDK FTS (↑)	Morgan FTS (↑)	FCD (↓)
SMILES	0.816 ± 0.004	0.174 ± 0.029	21.160 ± 0.600	0.888 ± 0.023	0.867 ± 0.005	0.738 ± 0.010	0.672 ± 0.013	6.224 ± 0.449
SELFIES	0.277 ± 0.009	0.100 ± 0.016	76.162 ± 2.229	0.804 ± 0.022	0.619 ± 0.010	0.467 ± 0.018	0.399 ± 0.017	13.557 ± 0.224
Table 19: Performance of SMILES and SELFIES of GPT-4 model in molecule captioning task.
BLEU-2 (↑)	BLEU-4 (↑)	ROUGE-1 (↑)	ROUGE-2 (↑)	ROUGE-L (↑)	METEOR (↑)
SMILES	0.464 ± 0.008	0.365 ± 0.008	0.545 ± 0.003	0.362 ± 0.003	0.459 ± 0.007	0.519 ± 0.005
SELFIES	0.459 ± 0.012	0.367 ± 0.010	0.530 ± 0.007	0.360 ± 0.005	0.456 ± 0.005	0.490 ± 0.007