What indeed can GPT models do in chemistry? A comprehensive benchmark on
  eight tasks
Authors: name: Taicheng Guo, name: Kehan Guo, name: Bozhao nan, name: Zhengwen Liang, name: Zhichun Guo, name: Nitesh V. Chawla, name: Olaf Wiest, name: Xiangliang Zhang

Large Language Models (LLMs) with strong abilities in natural language
processing tasks have emerged and have been rapidly applied in various kinds of
areas such as science, finance and software engineering. However, the
capability of LLMs to advance the field of chemistry remains unclear. In this
paper,we establish a comprehensive benchmark containing 8 practical chemistry
tasks, including 1) name prediction, 2) property prediction, 3) yield
prediction, 4) reaction prediction, 5) retrosynthesis (prediction of reactants
from products), 6)text-based molecule design, 7) molecule captioning, and 8)
reagent selection. Our analysis draws on widely recognized datasets including
BBBP, Tox21, PubChem, USPTO, and ChEBI, facilitating a broad exploration of the
capacities of LLMs within the context of practical chemistry. Three GPT models
(GPT-4, GPT-3.5,and Davinci-003) are evaluated for each chemistry task in
zero-shot and few-shot in-context learning settings with carefully selected
demonstration examples and specially crafted prompts. The key results of our
investigation are 1) GPT-4 outperforms the other two models among the three
evaluated; 2) GPT models exhibit less competitive performance in tasks
demanding precise understanding of molecular SMILES representation, such as
reaction prediction and retrosynthesis;3) GPT models demonstrate strong
capabilities in text-related explanation tasks such as molecule captioning; and
4) GPT models exhibit comparable or better performance to classical machine
learning models when applied to chemical problems that can be transformed into
classification or ranking tasks, such as property prediction, and yield
prediction.