MolXPT: Wrapping Molecules with Text for Generative Pre-training
Year: 2023

Abstract:
Generative pre-trained Transformer (GPT) has demonstrates its great success in natural language processing and related techniques have been adapted into molecular modeling. Considering that text is the most important record for scientific discovery, in this paper, we propose MolXPT, a unified language model of text and molecules pre-trained on SMILES (a sequence representation of molecules) wrapped by text. Briefly, we detect the molecule names in each sequence and replace them to the corresponding SMILES. In this way, the SMILES could leverage the information from surrounding text, and vice versa. The above wrapped sequences, text sequences from PubMed and SMILES sequences from PubChem are all fed into a language model for pre-training. Experimental results demonstrate that MolXPT outperforms strong baselines of molecular property prediction on MoleculeNet, performs comparably to the best model in text-molecule translation while using less than half of its parameters, and enables zero-shot molecular generation without finetuning.

heading: 1 Introduction
text: Generative pre-trained Transformer (GPT), like GPT-3 (Brown et al., 2020) and ChatGPT (OpenAI, 2022), have obtained great success in natural language processing. They usually have billions of parameters and are trained on large corpus (Taylor et al., 2022; Singhal et al., 2022). By witnessing their great power, people start transferring language models to chemical (Bagal et al., 2022) and biological domains (Ferruz et al., 2022). For example, a small molecule (e.g., an oral drug) can be represented using simplified molecular-input lineentry system (SMILES) (Weininger, 1988), which is a sequence obtained by traversing the molecular graph using depth-first-search and several rules
∗Equal contribution. This work was done when Z. Liu and W. Zhang were interns at Microsoft Research AI4Science.
†Corresponding authors.
for branching, aromaticity, etc. After serializing molecules, people pre-train language models on SMILES (Bagal et al., 2022; Tong et al., 2021; Frey et al., 2022) and obtain promising results for molecular generation.
Text is the most important record for molecular science and more generally, scientific discovery (Beltagy et al., 2019). It describes detailed properties of molecules, like how to synthesize the molecule (Feng et al., 2016), whether the molecule is toxic (Juurlink et al., 2003), etc. BioGPT (Luo et al., 2022) and PubMedGPT (Bolton et al., 2022) are two language models trained on biomedical literature. Recently, a new trend is to jointly model SMILES and scientific text so as to obtain shared representations across the two modalities. MolT5 is a T5-like (Raffel et al., 2020) model, where several spans of the text/SMILES are masked in the encoder and they should be reconstructed in the decoder. Galactica (Taylor et al., 2022) is a GPTlike (Brown et al., 2020) model pre-trained on various types of inputs, like text, SMILES, protein sequences, etc. Although those models demonstrate progress in prediction and generation tasks, they do not explicitly leverage the relation between molecules and text. An intuition is that, in scientific literature, when a molecule name appears in a sentence, the surrounding context could be a description of the molecule. This should be useful information for joint training but is ignored in those models.
To leverage such relations, in this work, we propose a novel molecule-text language model (MolXPT), which is trained on “wrapped” sequences: Given a sentence, we detect the molecular names with named entity recognition tools, and if any, replace them to the corresponding SMILES and obtain the “wrapped” sequence between SMILES and text. We pre-train a 24-layer MolXPT (with 350M parameters) on 8M wrapped sequences, as well as 30M SMILES from PubChem
1606
(Kim et al., 2022) and 30M titles and abstracts from PubMed (a popular biomedical literature search engine).
After pre-training, we finetune MolXPT on MoleculeNet (a benchmark about molecular property prediction) (Wu et al., 2018) and molecule-text translation (Edwards et al., 2022) using promptbased finetuning. On MoleculeNet, MolXPT outperforms strong baselines with sophisticated design like GEM (Fang et al., 2022). On text-molecule translation, MolXPT performs comparably with the state-of-the-art model, MolT5-large (Edwards et al., 2022). MolT5-large has 800M parameters while MolXPT only uses 44% of its parameters. We also verify that MolXPT has the zero-shot ability on text-to-molecule generation.

heading: 2 Our Method
text: MolXPT is a language model pre-trained on heterogeneous data including scientific text, SMILES sequences, and “wrapped” sequences between SMILES and text. Due to the flexible input, we can finetune it for various text and molecular tasks. The framework of MolXPT is in Figure 1.

heading: 2.1 Pre-training corpus
text: For scientific text, we use the titles and abstracts of 30M papers from PubMed1. For molecular SMILES, we randomly choose 30M molecules from PubChem2 (Kim et al., 2022).
The wrapped sequences are constructed via a “detect and replace” pipeline. We first use BERN2 (Sung et al., 2022), a widely used named entity recognition (NER) tool for biomedical purpose, to detect all mentions of molecules and link them to the entities in public knowledge bases like ChEBI
1https://ftp.ncbi.nlm.nih.gov/pubmed/ 2https://pubchem.ncbi.nlm.nih.gov/
(Hastings et al., 2016). After that, we can retrieve the molecular SMILES of the matched entities. Finally, we replace the molecular mentions to their corresponding SMILES. An example is shown in the left panel of Figure 1. The wrapped sequences must contain at least one molecular SMILES. We eventually obtain 8M wrapped sequences in total.
Text and SMILES are tokenized separately. For text, we use byte-pair encoding (BPE) (Sennrich et al., 2016) to split the words into subwords. The number of BPE merge operation is 40k. For SMILES sequences (including those in wrapped sequences), we tokenize them with the regular expression from (Schwaller et al., 2018). For each SMILES sequence S, we add a start-of-molecule token ⟨som⟩ at the beginning of S and append an end-of-molecule token ⟨eom⟩ at the end of S.

heading: 2.2 Model and training
text: Model architecture: MolXPT has the same architecture as the GPT models (Radford et al., 2019). Due to computational resource limitation, in this paper, we follow the GPT-2medium configuration with 24 layers, 1024 hidden size and 16 attention heads. The maximum length of input we can process is 2048 and the vocabulary size is 44536. In total, our model has 350M parameters. Pre-training: The pre-training objective function of MolXPT is the negative log-likelihood. Mathematically, let D = {xi}i denote the collection of sequences of the three types of the data, and xi = (si,1, si,2, · · · , si,ni) is the i-th sequence with ni tokens. The training objective function is:
min− 1|D|
|D|∑
i=1
ni∑
j=1
logP (si,j |si,j−1, si,j−2, · · · , s1).
The pre-training details are left in Appendix B.
Prompt-based finetuning: MolXPT can be finetuned for downstream tasks about molecules and text. Adding classification or regression heads to pre-trained backbone models introduces the gap between pre-training and finetuning (Brown et al., 2020; Chen et al., 2022; Gu et al., 2022). Therefore, we adopt prompt-based finetuning (Gao et al., 2021) to unify different tasks into a sequence generation task, which is consistent with the pre-training objective. Briefly, given a task, we convert the input and output into text and/or SMILES sequences, equip the sequences with task-specific prompts and finetune using language modeling loss. Prompts for MoleculeNet and text-molecule translation are introduced in the Section 3.1 and 3.2 respectively. Discussion: Some works also try to jointly model text and molecules. Zeng et al. (2022) propose KV-PLM, where SMILES sequences are appended after molecule names for pre-training. Su et al. (2022) use contrastive learning between text and molecular graphs. Our MolXPT is a generative model while the above two models are not. Both of them are built upon SciBERT (Beltagy et al., 2019), a BERT model (Devlin et al., 2019) for scientific literature. MolXPT is complementary to them.

heading: 3 Experiments
text: We evaluated MolXPT on two downstream tasks: (1) molecular property prediction on MoleculeNet (Wu et al., 2018), which is to predict whether the given molecule has specific properties; (2) the generation between text descriptions and molecules (Edwards et al., 2022), where both molecules and text should be considered. In this section, we focus on introducing task definition, prompt design
and results while leaving the detailed finetuning hyper-parameters in Appendix C.

heading: 3.1 Results on MoleculeNet
text: MoleculeNet (Wu et al., 2018) is a widely-used benchmark for molecular modeling, which has more than 700k compounds for various different properties. We choose six molecular classification tasks for evaluation, which are BBBP, Tox21, ClinTox, HIV, BACE and SIDER. Details are left in Appendix A. We follow GEM (Fang et al., 2022) to split the data into training/validation/test sets based on the scaffold. For these tasks, the input is a SMILES and the output is a binary label. Finetuning strategy: Previous molecular property prediction models mainly use SMILES sequences or molecular graphs as input, while we can use the “wrapped” sequences. For example, one task is to predict the blood-brain barrier penetration (BBBP) of a molecule. Therefore, the prompt is “We can conclude that the BBB penetration of ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ is ⟨tag⟩”, where ⟨SMILES⟩ denotes the molecular SMILES, and ⟨tag⟩ denotes the classification result. For the BBBP task, we design ⟨tag⟩ as “true” or “false”, indicating whether the compound can or cannot cross BBB.
Different tasks have different prompts (see Appendix C.1), but we put the tags to the last token of the prompt for all tasks. Let (si,1, si,2, · · · , si,Ti) denote the i-th wrapped sequence for the downstream task with Ti tokens, where si,Ti is the tag of the sequence. Denote that there are N samples for finetuning. The finetuning strategy could be either
min− 1 N
N∑
i=1
logP (si,Ti |si,<Ti), (1)
indicating that we finetune the tags only, or
min− 1 N
N∑
i=1
1
Ti
Ti∑
j=1
logP (si,j |si,<j), (2)
indicating that we finetune the full prompts. According to our exploration, Eqn.(1) achieves slightly better results and we use it for all tasks (see Appendix C.4 for the results).
Let ptrue and pfalse denote the probabilities of tags “true” and “false” after encoding the prefix “We can conclude that the BBB penetration of ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ is”. The probabilities that ⟨SMILES⟩ can and cannot cross blood-brain barrier are normalized as ptrue/(ptrue + pfalse) and pfalse/(ptrue + pfalse) respectively. The finetuning hyper-parameters are in Appendix C.2.
We compare MolXPT with two types of baselines: (1) pre-trained language model baselines including KV-PLM (Zeng et al., 2022), Galactica (Taylor et al., 2022) and MoMu (Su et al., 2022). (2) pre-trained Graph Neural Network (GNN) baselines including GContextual (Rong et al., 2020), G-Motif (Rong et al., 2020), GROVERbase (Rong et al., 2020), GROVERlarge (Rong et al., 2020), GraphMVP (Liu et al., 2022), MGSSL (Zhang et al., 2021) and GEM (Fang et al., 2022). The evaluation metric is the ROC-AUC score. The results are in Table 1.
MolXPT outperforms the GNN baselines pretrained on pure molecular data, indicating the effectiveness of pre-training with scientific text corpus. Compared with Galactica which also uses both SMILES and text for pre-training GPT-like model, MolXPT obtains better performance. Note that Galactica does not purposely build and train on the “wrapped” sequences, whose importance is demonstrated via our empirical results. A possible explanation of the superior performance is that the SMILES describes the component and structural information of molecules, while the text describes the general properties. They are complementary to each other, and joint training on them brings more effective representations.

heading: 3.2 Results on text-molecule translation
text: We evaluated the performance of MolXPT on CheBI-20 (Edwards et al., 2021), a bidirectional text-molecule translation dataset. It consists of 33,010 molecule-description pairs. We use the data split provided by MolT5 (Edwards et al., 2022), where the training, validation and test sets account
80%, 10% and 10% of total data. For molecule-totext generation, given a molecular SMILES S, the prompt is: “The description of ⟨som⟩ S ⟨eom⟩ is: The molecule is”, followed by the text description of S. For text-to-molecule generation, given a text description T , the prompt is: “T . The compound is ⟨som⟩”, and the model will generate the molecular SMILES ended with ⟨eom⟩. We compare our method with MolT5 (Edwards et al., 2022).
For molecule-to-text generation, the results are evaluated by NLP metrics including BLEU (Papineni et al., 2002), Rouge (Lin, 2004) and METEOR (Banerjee and Lavie, 2005). “Text2mol” is a deep learning based metric proposed by Edwards et al. (2022) to measure the similarity of the text-molecule pairs. For text-to-molecule generation, we evaluate the following metrics: the proportion of the generated SMILES that exactly match the reference SMILES (denoted as “Exact”); the Tanimoto similarity of three types of fingerprints: MACCS (Durant et al., 2002), RDK (Schneider et al., 2015) and Morgan (Rogers and Hahn, 2010); the FCD score (Preuer et al., 2018), which measures the molecule distances by a pretrained model; the percentage of the valid generated SMILES. The results are reported in Table 2.
We observe that MolXPT achieves significantly better performance than MolT5-small and MolT5-base, and has comparable performance with MolT5-large. Note that MolT5-large has 800M parameters while MolXPT only uses 44% of its parameters. For both tasks, our model performs the best on Text2Mol metric, indicating that MolXPT captures the alignment between text and molecule better. We attribute it to the wrapped sequences, by which the model can learn the relation between molecule and text explicitly.
We further verify the zero-shot text-to-molecule generation ability of MolXPT. The pre-trained MolXPT takes the text as input and directly generates molecules without finetuning. The top-1 and top-5 fingerprint similarity is in Table 3. Indeed, compared with the full data setting, the performance drops, but still reasonable numbers. In addition, the zero-shot MolXPT successfully recovers 33 molecules based on the text (see Appendix D).

heading: 4 Conclusions and Future Work
text: We propose MolXPT, a generative model pretrained on scientific text, molecular SMILES and
their wrapped sequences. We train a 24-layer MolXPT with 350M parameters. By prompt-based finetuning, it improves strong baselines on MoleculeNet and achieves comparable results with the best model on molecule-text translation but using much fewer parameters.
For future work, first, we will train larger MolXPT to further verify the performances across different tasks and the zero-shot/in-context (Xie et al., 2022) learning ability. Second, how to further enhance the interaction between molecules and text (e.g., using contrastive learning to enhance consistency) should be studied. Third, how to effectively adapt MolXPT into other molecule and text tasks such as text-guided molecule optimization is another direction to explore.
Limitations
One limitation of our method is that when training larger models, it requires more computation resources, whose cost is relatively high. However, after pre-training, we will release our models so that readers can directly use them without pre-training again.
Broader Impacts
We provide a new generative pre-trained model on molecules and text. On one hand, the model can be used to speed up scientific discovery, like molecule design, drug optimization, etc. On the other hand, once the model is trained on clinical data (which also describes the usage of drug molecules), it might lead to personal information leaky. We will enhance data filtration to anonymize all personal information, and will design new models to protect the information.

heading: Acknowledgement
text: The authors Zequn Liu and Ming Zhang are partially supported by National Natural Science Foundation of China (NSFC Grant Number 62276002).

heading: A Datasets and Baselines of MoleculeNet
text: We choose the following tasks of MoleculeNet for evaluation: (1) BBBP contains compounds with binary labels on blood-brain barrier penetration. (2) Tox21 is a dataset for predicting the human toxicity of compounds on 12 different targets. (3) ClinTox contains drugs approved by the FDA and those that have failed clinical trials for toxicity reasons. (4) HIV aims to predict whether a drug can inhibit HIV replication. (5) BACE describes binding results for a set of inhibitors of human β-secretase 1. (6) SIDER has compounds used in marketed medicines with 27 categories of side effects. We compare MolXPT with the following baselines:
(1) GROVER is a self-supervised pre-trained graph Transformer model. G-Contextual and G-Motif are two variants of it pre-trained with contextual property prediction task and motif prediction task. (2) GraphMVP is a self-supervised pre-trained GNN model using both 2D topological structures and 3D geometric views of molecules. (3) MGSSL leverages a retrosynthesis-based algorithm BRICS and additional rules to find the motifs and combines motif layers with atom layers. (4) GEM is a geometry-enhanced pre-trained GNN model. (5) Galactica is a GPT-like model trained on a large scientific corpus and many natural sequences like SMILES. We report the result of Galactica-120B. (6) KV-PLM is a BERT-like model where SMILES sequences are appended after molecule names for pre-training. (7) MoMu uses contrastive learning to jointly pretrain a BERT model for text and a GNN model for molecules.

heading: B Pre-training hyper-parameters
text: MolXPT is pre-trained for 200k steps on eight A100 GPUs. The batchsize is 2048 tokens per GPU. The gradients are accumulated for 16 steps before updating. We use Adam (Kingma and Ba, 2015) optimizer for optimization. The peak learning rate is 0.0005 and the warm-up steps are 20000. The learning rate scheduler is inverse square root decay scheduler. The dropout is 0.1.

heading: C Finetuning details of downstream tasks
text: C.1 Prompts for finetuning MoleculeNet
(1) BBBP: “We can conclude that the BBB penetration of ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ is true/false.” (2) Tox21: “We can conclude that the ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ activity outcome on ⟨target⟩ is active/inactive. ” where ⟨target⟩ refers to corresponding receptor or enzyme for each subtask, e.g. the ⟨target⟩ of subtask "AR" is "Androgen Receptor". (3) ClinTox:“We can conclude that the clinical trial toxicity of ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ is true/false.” for subtask CT_TOX and “We can conclude that the FDA approval status of ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ is true/false.” for subtask FDA_APPROVED. (4) HIV: “We can conclude that the screening result of ability to inhibit HIV replication of ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ is active/inactive.”
(5) BACE: “We can conclude that the binding result on beta-secretase 1 of ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ is true/false.” (6) SIDER:“We can conclude that the ⟨som⟩ ⟨SMILES⟩ ⟨eom⟩ can bring about the side effect of ⟨side-effect⟩ is true/false.” where ⟨side-effect⟩ refers to corresponding side-effect for each subtask.
C.2 Details of finetuning MoleculeNet We grid search the following hyper-parameters: learning rate in {3× 10−5, 5× 10−5}; dropout in {0.1, 0.3}; total epochs from {30, 50}. The model is selected according to validation performance.
C.3 Details of finetuning text-molecule generation
For text-molecule generation, MolXPT is finetuned for 100 steps on one P40 GPU with 1024 tokens and 16 accumulated steps per device. Models are finetuned for 100 epochs. The learning rate is 0.0001 and the dropout rate is grid searched from [0.1, 0.2, 0.3, 0.4, 0.5]. Setting dropout rate as 0.4 and 0.5 achieves the best validation performance on molecule-to-text generation and text-to-molecule generation respectively. We use the corresponding models for testing.
C.4 MoleculeNet finetuning strategy selection We provide two finetune strategies in Eqn.(1) and Eqn.(2). Their results are reported in Table 4. Their results are similar and Eqn.(1) is slightly better.

heading: D Zero-shot text-to-molecule generation
text: Given K generated molecule m̂1, m̂2, · · · , m̂K and the reference molecule m, the top-K fingerprint similarity is
max i∈[K] similarity(m, m̂i). (3)
MolXPT generates 33 molecules that can exactly match the reference molecules without finetuning. Figure 2 shows three of the cases.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: The section called "Limitations".

heading: 3 A2. Did you discuss any potential risks of your work?
text: The section called "Ethnics statement".

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Introduction (Section 1)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Section 2.2 and Appendix B

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 2.2 and Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.1, Section 3.2 and Appendix C

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3.1, Section 3.2

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.2.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future work, first, we will train larger MolXPT to further verify the performances across different tasks and the zero-shot/in-context (Xie et al. , 2022) learning ability. Second, how to further enhance the interaction between molecules and text (e. g. , using contrastive learning to enhance consistency) should be studied. Third, how to effectively adapt MolXPT into other molecule and text tasks such as text-guided molecule optimization is another direction to explore. Limitations
One limitation of our method is that when training larger models, it requires more computation resources, whose cost is relatively high. However, after pre-training, we will release our models so that readers can directly use them without pre-training again