(2024)

Abstract: Language-molecule models have emerged as an exciting direction for molecular discovery and understanding. However, training these models is challenging due to the scarcity of molecule-language pair datasets. At this point, datasets have been released which are 1) small and scraped from existing databases, 2) large but noisy and constructed by performing entity linking on the scientific literature, and 3) built by converting property prediction datasets to natural language using templates. In this document, we detail the L+M-24 dataset, which has been created for the Language + Molecules Workshop shared task at ACL 2024. In particular, L+M-24 is designed to focus on three key benefits of natural language in molecule design: compositionality, functionality, and abstraction.1

heading: 1 Introduction
text: The world faces an enormous number of problems in the coming decades on scales of complexity never-before-seen, in areas such as climate change, healthcare, and pandemics. To address these issues, we need to discover inventive scientific solutions which are scalable, flexible, and inexpensive. Broadly speaking, many of these problems will require molecular solutions from the chemistry domain, such as developing new drugs (e.g. kinase inhibitors (Ferguson and Gray, 2018)), materials (e.g. organic photovoltaics (Kippelen and Brédas, 2009)), and chemical processes (Zhong et al., 2023). These solutions exist in extremely large search spaces, which makes AI tools a necessity.
Language-molecule models have emerged as an exciting direction for molecular discovery and understanding (Edwards et al., 2021; Zeng et al., 2022; Edwards et al., 2022; Su et al., 2022; Liu
1The dataset, finetuned baseline, and evaluation code are released publicly at github.com/language-plus-molecules/LPM24-Dataset through HuggingFace.
et al., 2022; Xu et al., 2023; Christofidellis et al., 2023; Liu et al., 2023b; Luo et al., 2023; Zhao et al., 2023c; Seidl et al., 2023). However, training these models is challenging due to the scarcity of molecule-language pair datasets. At this point, datasets have been released which are 1) small and scraped from existing databases (Edwards et al., 2021; Zeng et al., 2023; Liu et al., 2023a,c; Pei et al., 2023), 2) large but noisy and constructed by performing entity linking on the scientific literature (Zeng et al., 2022; Su et al., 2022), and 3) templatebased built on prediction datasets (Zhao et al., 2023a; Fang et al., 2023). Approaches utilizing pseudo-data have also been attempted (Chen et al., 2023a). These approaches have helped remedy the problem of data scarcity in this domain; however, these approaches frequently ignore key benefits of natural language: 1) compositionality, 2) abstraction, and 3) functionality (Zhang et al., 2023). To this end, for the Language + Molecules Workshop at ACL 2024, we release L+M-24, which we construct to test these three goals, particularly compositionality, using recently released data sources (Zhao et al., 2023b; Kosonocky et al., 2023; Wishart et al., 2023). L+M-24 is divided into four categories with important applications in the small-molecule domain: 1) Biomedical, 2) Light and Electricity, 3) Human Interaction and Organoleptics, and 4) Agriculture and Industry. Improving understanding of these applications can have important implications in problems such as drug discovery, climate issues, more efficient and green industrial processes, and improved food production.

heading: 2 Task Formulation
text: The dataset is primarily intended for language↔molecule translation, which consists of two tasks: generating 1) a caption given a molecule and 2) a molecule given a description.
1

heading: 2.1 Designing for Compositionality, Abstraction, and Function
text: Overall, we focused on four primary categories of importance: 1) Biomedical, 2) Light and Electricity, 3) Human Interaction and Organoleptics, and 4) Agriculture and Industry. These categories and three properties from each are displayed in Table 1. The biomedical category is focused on drug properties, functions, and interaction with proteins. Light and electricity is focused on the ability for a molecule to produce or absorb light or electricity. Human interaction and organoleptics focuses on the effect and experience molecules cause in humans. Agriculture and industry focuses on molecules used in industrial processes and food production.
Based on our data sources (below), the properties we have selected already encode a large degree of functionality, enhanced by our manual curation. Further, since these properties are generally short phrases indicating functionality, they are also abstract and apply to many molecules (e.g., “insecticide”). For compositionality, we explicitly select certain pairs of properties which we hold out of the dataset. For example, a molecule may share two properties which are desirable together (e.g., low toxicity and fungicidal). L+M-24 will help to evaluate whether model’s can generalize to unseen compositions of properties.

heading: 3 Data Sources
text: We constructed our dataset using three different databases. We will first describe the process we used to extract information from each, followed by our overall strategy for adding hierarchy into the dataset. We want to deeply thank the authors of these resources for making them publicly available for the community.

heading: 3.1 PubChem
text: We used properties extracted from PubChem (Kim et al., 2016, 2019) as described in (Zhao et al., 2023c). Properties from this approach include odor, taste, and decomposition. We note these properties consist of molecule-specific descriptions, which the other data sources do not provide.

heading: 3.2 Chemical Function (CheF)
text: Here, we used functional properties extracted from patent literature by Kosonocky et al. (2023). This allowed us to capture molecules from the patent literature in addition to the scientific literature. Here,
we started with CheF prefinal_v32. We created a set of properties from both CheF’s property summarizations and from the ChatGPT summarization source. For the summarization source, we also applied the WordNet lemmatizer (Bird et al., 2009) for deduplication. After obtaining a list of properties, we removed properties pertaining to less than 100 molecules. We then kept properties falling into the categories of “X-icide”, “anti-X”, “X treatment”, “X modulators”, “X inhibitors”, “X agonists”, “X antagonists”, “light”, and “electricity.” We manually removed uninformative labels which were too broad or didn’t describe enough function. Further, we manually corrected errors in label naming and duplication.

heading: 3.3 ChemFOnt: the chemical functional ontology resource
text: In addition to CheF, we also take advantage of another new chemical function data resource: ChemFOnt (Wishart et al., 2023). From this datasource, we collect three categories: health effect relations, organoleptic effect relations, and role relations.

heading: 4 Dataset Details
text: To convert these properties to natural language, we follow a template-based procedure using GPT4 (OpenAI, 2023) generated compositional templates.
2obtained via personal communication.

heading: 4.1 Template Generation
text: We utilize GPT-4 (OpenAI, 2023) to generate specific templates for each combinations of molecular properties. Specifically, we manually write six templates: “The molecule is a <0>.”; “It belongs to the <1> class of molecules.”; “It has an effect on <2>.”; “It impacts <3>.”; “The molecule is <4>.”; and “The molecule has a <5>.”. Subsequently, we use GPT-4 to generate a unique sentence template for each possible combination by rephrasing up to six combinations of the six initial templates as a single sentence. Ultimately, this process results in the generation of 917 distinct templates. The templates were manually checked and corrected to have a matching standard. The prompts and in-context examples for GPT-4 are given in the Appendix.

heading: 4.2 Converting Templates to Descriptions
text: For all properties in L+M-24, we first assigned them to possible templates based on their category or by individual consideration. Certain properties (e.g., polymerization, decomposition) were expressed in sentence format, so we did not use templates. Given a molecule with n properties, we first looked for a template that had the correct slots (e.g., <0>, <2>, and <2>) for its properties. When we found possible templates, we picked one at random and used it to generate a sentence for the molecule’s properties. If there were no matching templates, we split the properties into two separate equal-sized groups and tried with each group. We return the concatenation of the two sentence templates as the molecule description. Note this process can repeat multiple times.
We note that we are also releasing a version of the dataset with 5 captions for each molecule. In
this case, we split group sizes at random. Further, we split sentences apart 50% of the time (even when there were matching templates) to increase caption diversity.

heading: 4.3 Splitting
text: Duplicate molecules are merged using RDKit (Landrum, 2021) and molecules which cannot be processed are removed. We split the data by first examining property combinations. 20% of combinations are witheld into the evaluation set. From molecules in the remaining 80%, we keep 80% for training and put 20% in evaluation. The evaluation set is split into two tasks: molecule captioning and molecule generation. For each task, only one modality will be released prior to the shared task results.
The training set consists of 160,492 moleculedescription pairs. For the evaluation set, both molecule generation and captioning contain 21,839 pairs. Further, special splits are released for the training set which allow for validation using the training data. They are constructed using the same procedure as the official evaluation dataset.

heading: 5 Evaluation Metrics
text: Overall, we adopt the evaluation metrics proposed by Edwards et al. (2022). However, we include invalid molecules in the calculations of FTS metrics (setting the score to zero for invalid molecules). We also add a uniqueness metric to the generated molecules for held-out combinations of properties (Polykovskiy et al., 2020). Further, we also look at property-specific precision, recall, and F-1 scores. These scores are calculated by matching tokenized names in the generated captions. These scores are
further aggregated across specific properties (e.g., inhibitors, X-icides, etc.) and the four broad categories. Aggregations are performed by averaging scores (i.e., macro-F1). We further compute these scores specifically for held-out combinations of properties.

heading: 6 Benchmarks
text: MolT5 models (Edwards et al., 2022) were finetuned for 20 epochs on the “split_train” data split and evaluated on the “split_valid”, both of which are available online. Huggingface’s transformers (Wolf et al., 2019) was used for finetuning with a learning rate of 2e-5 and weight decay of 0.01. A
batch size of 128 was used for small and base models, and a batch size of 48 for large models. Further, Meditron-7B (Chen et al., 2023b) was finetuned for 5 epochs with a context length of 930, 2e-6 learning rate, and batch size of 8/16 (molecule/caption generation). Models are released online. Results for captioning are reported in Tables 2, 3 and 4. Tables 5, and 6 shows results for molecule generation.
Overall, the dataset proves to be fairly challenging for these naively finetuned models. On captioning, Meditron-7B achieves a maximum overall F-1 score of 16.81 for property identification (Table 3). However, overall it has a much higher precision than recall, indicating the model only labels
a molecule with a certain property when having higher confidence. Certain classes of molecules, such as X-icides, are never identified (Table 4). Other classes, such as toxins or electricity, show emergent behavior as model size scales. Interestingly, the models appear to be fairly capable at linking molecules to certain diseases or cancers. We find that, likely due to poor performance on individual properties, only the largest model succeeds on predicting held-out combos, and with poor results. Additionally, we find that the Text2Mol metric, as trained on ChEBI-20, shows poor domain transfer to L+M-24.
The models are able to capture a number of useful properties, such as electroluminescence, diabetes treatment, non-alcoholic fatty liver disease, and emulsifiers. In some cases, the model captures important characteristics about the molecule but uses differing language. This poses a challenge for our evaluation metrics. For example, a molecule identified in the ground truth as an anti tumor agent
is identified as being a cancer treatment by the model. In particular, the models appear to struggle with rarer properties, which are common in our dataset formulation and in the chemical domain as a whole. They also struggle with identifying molecule-protein interactions (e.g., “monoamine reuptake inhibitor”), although Meditron shows a large performance jump.
For the molecule generation task, we also find the dataset to be challenging. We show results generated by different models on never-before-seen property combinations in Figures 2 and 3. We believe the difficulty is for two reasons. First, common property combinations may have structurally very different molecules which exhibit those properties, making evaluation difficult. Second, the model may not grasp rare properties well. Overall, this results in the naively finetuned models producing similar outputs to many different prompts. Further, as expected, performance falls on unseen property combinations and larger models prove
more effective (Table 6).

heading: 7 Future Directions
text: Overall, this dataset proves to be quite challenging. We find that some specific properties in particular are challenging for the model. This may be because the model understands these properties, but is unwilling to use them in its descriptions due to the training procedure. This limitation may be addressed with more sophisticated decoding algorithms or by better finetuning methods. Future work will also likely benefit from incorporating other modalities, such as proteins, to provide better understanding to the model for some property types. Notably, certain properties display what may be emergent behavior; scaling training data or model size may yield non-linear improvements.
In this dataset, we focus on composition, abstraction, and function. Future work may also wish to integrate other recent trends: instruction-following and dialogue (Fang et al., 2023; Cao et al., 2023;
Zeng et al., 2023; Zhao et al., 2024; Zhang et al., 2024; Yu et al., 2024), tool use (Boiko et al., 2023; Bran et al., 2023), additional molecule representations (e.g., 3D (Tang et al., 2023)), additional modalities (Xu et al., 2023), or molecule editing (Su et al., 2022). Further, we note the need for improved evaluation metrics, especially in the case of molecule generation for function where there may be many possible outputs. Specific methods for improving compositionality may be another fruitful avenue for research (Yellinek et al., 2023). It may also be interesting to use molecule-language instruction-following models within larger search frameworks, such as ChemReasoner (Sprueill et al., 2023, 2024).

heading: 8 Conclusion
text: In this manuscript, we describe the process for creating the L+M-24 dataset. L+M-24 is designed to focus on three key benefits of natural language in molecule design: compositionality, functionality,
and abstraction. It is the featured shared task at the First Language + Molecules Workshop at ACL 2024.

heading: Acknowledgements
text: We would like to thank the authors and creators of our data sources for freely allowing the use of their work. We would like to acknowledge NVIDIA Corporation’s contributions to this work through a grant of NVIDIA A100 Tensor Core GPUs. This work is supported by the Molecule Maker Lab Institute: an AI research institute program supported by NSF under award No. 2019897, by DOE Center for Advanced Bioenergy and Bioproducts Innovation U.S. Department of Energy, Office of Science, Office of Biological and Environmental Research under Award Number DESC0018420, and by AI Algriculture: the Agriculture and Food Research Initiative (AFRI) grant no. 2020- 67021- 32799/project accession no.1024178 from the USDA National Institute of Food and Agriculture. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied of, the National Science Foundation, the U.S. Department of Energy, and the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Prompts and examples for GPT4
text: • Prompts: You are an expert in the chemical domain whose task is to create templates to describe the properties of molecules. You will be challenged with a list of different cases. Each case willl have a list of **templates**, and a **question**. Each template will describe certain properties. Your goal is to generate a new template in a sentence based on all the previous templates.
• Case 1: Templates: - The molecule is a <0>. - It belongs to the <1> class of molecules. Answer: The molecule, characterized as a <0>, falls under the <1> category of chemical compounds.
• Case 2: Templates: - It has an effect on <2>. - It impacts <3>. Answer: It impacts <2> and has an effect on <3>.
• Case 3: Templates: - The molecule is <4>. - The molecule has a <5>. Answer: The molecule is <4>. and has a <5>.
• Case 4: Templates: - The molecule is a <0_1>. - The molecule is a <0_2>. Answer: The
molecule is a <0_1> and exhibits <0_2> properties.
• Case 5: Templates: - It belongs to the <1_1> class of molecules. - It belongs to the <1_2> class of molecules. Answer: The molecule is in the <1_1> class of compounds, characterizing it as a member of the <1_2> family.

heading: B Additional Dataset Statistics
text: Here, we give a brief description of properties in the dataset. Table 7 shows the number of propertymolecule pairs for different property classes. Figure 4 breaks the dataset down into different property classes. More details can be found in the dataset repository.

heading: Future Work
text: Future work will also likely benefit from incorporating other modalities, such as proteins, to provide better understanding to the model for some property types. Notably, certain properties display what may be emergent behavior; scaling training data or model size may yield non-linear improvements. In this dataset, we focus on composition, abstraction, and function. Future work may also wish to integrate other recent trends: instruction-following and dialogue (Fang et al. , 2023; Cao et al. , 2023;
Zeng et al. , 2023; Zhao et al. , 2024; Zhang et al. , 2024; Yu et al. , 2024), tool use (Boiko et al. , 2023; Bran et al. , 2023), additional molecule representations (e. g. , 3D (Tang et al. , 2023)), additional modalities (Xu et al. , 2023), or molecule editing (Su et al. , 2022). Further, we note the need for improved evaluation metrics, especially in the case of molecule generation for function where there may be many possible outputs. Specific methods for improving compositionality may be another fruitful avenue for research (Yellinek et al. , 2023). It may also be interesting to use molecule-language instruction-following models within larger search frameworks, such as ChemReasoner (Sprueill et al. , 2023, 2024)