SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Baselines In the zero-shot setting, we compare GIMLET with three molecule-text models: SMILES-base language model KVPLM [85], Galactica [64], and GNN-language model MoMu [59]. Other molecule-text models [16, 17, 39] either haven't released parameters or are difficult to handle zero-shot setting. Notably, the pretraining of Galactica includes the MoleculeNet datasets, which is thus not strictly zero-shot on some of our tasks. We report the result of all the baselines with our zero-shot learning framework and instructions. The details of baseline evaluation are in Appendix.
To establish upper bounds for task performance, we also illustrate the supervised results of popular graph models. For GNNs, we include GCN [30], GAT [66], and GIN [79]. We also include the graph transformer Graphormer [80]. To mitigate the impact of our pretraining, we additionally perform supervised pretraining on Graphormer using our pretraining datasets, referred to as Graphormer-p.
Settings Following the standard supervised setting in previous studies [26], we adopt the Scaffold split [51] with a ratio of 0.8, 0.1, 0.1 for all the datasets, and report results on the testing sets, ensuring the comparability of our results to previous works. For classification tasks, we employ ROC-AUC as the evaluation metric, while for regression tasks, we utilize RMSE.
Table 1: Zero-shot performance (ROC-AUC) over Bio-activity, Toxicity, and Pharmacokinetic tasks.
Method	#Param	Type	bace	hiv	muv	Avg. bio	tox21	toxcast	Avg. tox	bbbpl	cyp450	Avg. pha
KVPLM	110M	0.5126	0.6120	0.6172	0.5806	0.4917	0.5096	0.5007	0.6020	0.5922	0.5971
MoMu	113M	0.6656	0.5026	0.6051	0.5911	0.5757	0.5238	0.5498	0.4981	0.5798	0.5390
Galactica-125M	125M	Zero Shot	0.4451	0.3671	0.4986	0.4369	0.4964	0.5106	0.5035	0.6052	0.5369	0.5711
Galactica-1.3B	1.3B	0.5648	0.3385	0.5715	0.4916	0.4946	0.5123	0.5035	0.5394	0.4686	0.5040
GIMLET (Ours)	64M	0.6957	0.6624	0.6439	0.6673	0.6119	0.5904	0.6011	0.5939	0.7125	0.6532
GCN	0.5M	0.736	0.757	0.732	0.742	0.749	0.633	0.691	0.649	0.8041	0.7266
GAT	1.0M	0.697	0.729	0.666	0.697	0.754	0.646	0.700	0.662	0.8281	0.7451
GIN	1.8M	Supervised	0.701	0.753	0.718	0.724	0.740	0.634	0.687	0.658	0.8205	0.7392
Graphormer	48M	0.7760	0.7452	0.7061	0.7424	0.7589	0.6470	0.7029	0.7015	0.8436	0.7725
Graphormer-p	48M	0.8575	0.7788	0.7480	0.7948	0.7729	0.6649	0.7189	0.7163	0.8877	0.8020
Table 2: Zero-shot performance (ROC AUC) over large scale molecule tasks.
Method	Chembl Zero-Shot	PCBA
KVPLM	0.4155	0.4811
MoMu	0.5002	0.5150
Galactica-125M	0.6461	0.4800
Galactica-1.3B	0.4818	0.5202
GIMLET (Ours)	0.7860	0.6211
Table 3: Zero-Shot performance (RMSE) on Physical-chemical datasets.
Method	Type	ESOL	Lipophilicity	FreeSolv	Avg. phy
KVPLM	-	-	-	-
MoMu	Zero Shot	-	-	-	-
GIMLET (Ours)	1.132	1.345	5.103	2.527
GCN	1.331	0.760	2.119	1.403
GAT	1.253	0.770	2.493	1.505
GIN	Supervised	1.243	0.781	2.871	1.632
Graphormer	0.901	0.740	2.210	1.284
Graphormer-p	0.804	0.675	1.850	1.110
Results We report the result of different types of downstream tasks in Table 1. The result of GIN, GCN and GAT for MoleculeNet are from [26] which we mark by italic. We observe that in the zero-shot setting, GIMLET outperforms most baselines on the majority of datasets, except for bbbb where GIMLET also performs comparably to the baselines. In terms of the average performance across task categories, GIMLET outperforms all baselines, demonstrating the effectiveness of our method for instruction-based zero-shot molecule tasks. It is worth noting that some of the baselines also achieve results on certain tasks. For example, KVPLM works on hiv, muv and bbbb, and MoMu works on tox21 and bace, showing our instruction-based molecule zero-shot learning method is a general framework to probe knowledge in molecule-text models.
Comparing our zero-shot performance to the supervised results, we observe that GIMLET achieves performance close to those of GNNs on several datasets, like bace, muv, toxcast, and bbbb. This demonstrates that GIMLET is able to solve molecule tasks in the zero-shot fashion nicely.
The results for large-scale molecule tasks are presented in Table 2. As depicted, the baselines struggle to handle these tasks. Our GIMLET not only successfully transfers to the Chembl Zero-Shot splits, where both the tasks and graphs were unseen during pretraining, but also demonstrates strong generalization performance on the PCBA benchmark.
The results of regression tasks are shown in Table 3. The scatter plots of the regression are in Appendix. It is worth noting that regression tasks pose greater challenges in the zero-shot setting than classification tasks, because it is difficult to determine unseen physico-chemical properties using only natural language, and the output space is also vast. Notably, the zero-shot baselines fail to perform the regression tasks due to their inability to output correctly formatted numbers. In contrast, GIMLET generates correctly formatted numbers for over  $98\%$  regression testing samples in all the tasks and showcases the potential of zero-shot regression tasks.
4.2 Instructions-Based Few-Shot Finetuning
We apply few-shot instruction-based tuning on the downstream tasks, to examine whether GIMLET exhibits improved performance in the presence of low-resource data. Notably, for datasets with more than one task, we do few-shot learning for every single task individually. We first split datasets into training, validation, and testing sets in the same as the zero-shot setting. Then  $K$  samples for each class are randomly sampled from the training set as the few-shot examples, where  $K$  is the few-shot number. We report the result of the best validation model on the testing set. The input is the
same as the zero-shot setting, including molecule data and instructions. We only tune the last linear layer, to inspect whether the feature is discriminative and linear separable. Linear tuning is also low resource costing and avoids overfitting. The last linear mapping to vocabulary is tuned for GIMLET and KVPLM. For MoMu, we tune the last linear layer of the projection head for features.
The result is shown in Figure 3, and plots for each dataset are in Appendix. GIMLET outperforms other baselines consistently, exhibiting improved performance with an increasing number of few-shot samples. The performance of GIMLET also approaches the supervised GIN with only limited samples. Baselines also achieve some performance gain in the few-shot training but remain beaten by GIMLET, and the improvements are not stable, fluctuating with the few-shot number. The few-shot results demonstrate the high-quality representation learned by GIMLET, which is well-suited for specific tasks, highlighting the potential of GIMLET in scenarios beyond zero-shot learning.
Figure 3: Few shot performance. Higher is better for bio, tox, and pha, and lower is better for phy.
4.3 Ablation and Exploration Studies
Effect of Model Design We investigate components of GIMLET to highlight their benefits. Specifically, we focus on two aspects: (a) To measure the effectiveness of the unified transformer method, we compare it to the variant version which obtains graph embedding by an individual GIN as a token in text, a common method in multimodal transformers. (b) We ablate graph decoupling encoding by complete global attention. We conduct pretraining and downstream zero-shot testing in the same setting as our method.
Table 4: Ablation study on GIMLET module.
Method	bace	hiv	muv	Avg. bio	tox21	toxcast	Avg. tox	bbbpl	cyp450	Avg. pha
w.o. unifying	0.4319	0.6133	0.6067	0.5506	0.5922	0.5537	0.5730	0.5309	0.6206	0.5758
w.o. decoupling	0.6458	0.6406	0.5421	0.6095	0.6306	0.5954	0.6130	0.5666	0.6320	0.5993
GIMLET	0.6957	0.6624	0.6439	0.6673	0.6119	0.5904	0.6011	0.5939	0.7125	0.6532
The comparison is shown in Table 4. Compared to GIMLET, w.o. unifying perform worse on all the datasets, especially on Bio-activity and Pharmacokinetic Tasks. Next, the w.o. decoupling variation performs worse than GIMLET on most datasets. The decoupling significantly improves performance on Bio-activity and Pharmacokinetic tasks and is also comparable on Toxicity tasks. This proves our claims that the unified graph-text transformer not only avoids additional modules but also has a strong capacity for encoding graph data and generalizing across tasks.
Effect of Pretraining Our pretraining includes two large types of pretraining tasks, including Chembl bioactivity tasks and Chembl Property physico-chemical tasks. To validate the influence of each task type, we performed ablation experiments where each of them was excluded separately. The results are shown in Table 5, and the detailed result is in Appendix. Unsurprisingly, Chembl is essential for downstream molecule assay tasks, and Chembl property plays a crucial role in Physico-chemical tasks. However, the results also reveal that Chembl positively affects downstream Physico-chemical tasks, and Chembl property benefits Bio-activity tasks, Toxicity tasks, and Pharmacokinetic tasks. The results demonstrate the positive transfer of pretraining tasks on a diverse range of downstream tasks, spanning various types and domains.
Robustness to Instruction We explore whether GIMLET is robust to the instructions. We rephrase our instructions by GPT-3.5-turbo for testing. Each instruction is rephrased using four types of requests:
Table 5: Ablation study on GIMLET pretraining.
Method	Avg. bio ↑	Avg. pha ↑	Avg. tox ↑	Avg. phy ↓
bioactivity assay only	0.6402	0.6071	0.5676	-
physico-chemical only	0.4894	0.5454	0.4748	2.6178
both	0.6673	0.6532	0.6011	2.5266
rewriting, detailing, expanding, and shortening. The prompts and examples are provided in Appendix. We plot the performance for each type of augmentation, as well as the average standard variation for each task in Figure 4. As shown, GIMLET is more robust than baselines on most tasks. This shows that our instruction-based pretaining enables GIMLET to focus on the task rather than the specific language form.
Figure 4: Robustness to instruction.
Instruction Interpretability We ablate the explanation of the instructions in downstream tasks, to validate whether the explanation in instructions helps GIMLET perform downstream tasks. Without explanation, only the task name and question are provided to the model. The examples of ablated instructions are available in Appendix. The results presented in Table 6 demonstrate a significant drop in performance when task explanations are not included. This finding supports the effectiveness of the explanation and highlights the model's ability to comprehend the explanation of tasks.
Table 6: Ablation study on GIMLET instructions.
Method	bace	hiv	muv	Avg. bio	tox21	toxcast	Avg. tox	bbbpl	cyp450	Avg. pha
name only	0.5416	0.6132	0.6441	0.5996	0.5809	0.5279	0.5544	0.4871	0.6669	0.5770
+ explanation	0.6957	0.6624	0.6439	0.6673	0.6119	0.5904	0.6011	0.5939	0.7125	0.6532
In this work, we propose nature language instruction-based graph zero-shot learning for molecule tasks, and construct a molecule dataset consisting of two thousand tasks with instructions derived from task descriptions. We propose GIMLET, which extends large language models to handle graph and text data by applying the transformer mechanism with generalized position embedding and decoupled attention. Instruction-based pretraining is applied for GIMLET. Experiments demonstrate promising results in zero-shot learning, exhibit strong robustness to the instruction, and can be further improved by few-shot tuning. We do not consider the tasks with structured output like molecule generation in this study, which is left for further work.
6 Ethical Consideration
The work presented here is centered on a paradigm shift in molecule property prediction tasks, transitioning from traditional supervised learning to instruction-based zero-shot learning. Due to the fact that molecule property prediction has been widely explored in prior research, the direct societal impacts may appear limited. However, indirect negative impacts could be caused by excessive reliance on algorithms. We contend that a combination of model predictions and experimental validation is essential for practical implementation.
[79] Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018). How powerful are graph neural networks? arXiv preprint arXiv:1810.00826.
[80] Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. (2021). Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877-28888.
[81] You, J., Liu, B., Ying, Z., Pande, V., and Leskovec, J. (2018). Graph convolutional policy network for goal-directed molecular graph generation. Advances in neural information processing systems, 31.
[82] You, Y., Chen, T., Shen, Y., and Wang, Z. (2021). Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-12132. PMLR.
[83] You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y. (2020). Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812-5823.
[84] Zaidi, S., Schaarschmidt, M., Martens, J., Kim, H., Teh, Y. W., Sanchez-Gonzalez, A., Battaglia, P., Pascanu, R., and Godwin, J. (2022). Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00133.
[85] Zeng, Z., Yao, Y., Liu, Z., and Sun, M. (2022). A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(1):862.
[86] Zhang, J., Zhang, H., Xia, C., and Sun, L. (2020). Graph-bert: Only attention is needed for learning graph representations. arXiv preprint arXiv:2001.05140.
[87] Zhang, Z., Liu, Q., Wang, H., Lu, C., and Lee, C.-K. (2021). Motif-based graph self-supervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 34:15870-15882.
[88] Zhao, H., Ma, S., Zhang, D., Deng, Z.-H., and Wei, F. (2022). Are more layers beneficial to graph transformers? In The Eleventh International Conference on Learning Representations.
[89] Zhong, R., Lee, K., Zhang, Z., and Klein, D. (2021). Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2856-2878.
A.1 Details of Datasets and Instructions
Table 7: Data Overview
Splitting	Data Class	Dataset	No. of Molecules	No. of Tasks	Task Metric	Task Type
Pretraining	Bioactivity assay	ChEMBL bioassay activity dataset	365065	1048	ROC_AUC	Classification
Physico-chemical	CHEMBL Property	365065	13	RMSE	Regression
Downstream Zero-Shot	Large Scale	PCBA PubChem HTS bioAssay	437929	128	ROC-AUC	Classification
ChEMBL Zero-Shot	91266	262	ROC_AUC	Classification
Pharmacokinetic	CYP inhibition	16896	5	ROC_AUC	Classification
BBBP Blood-brain barrier penetration	2039	1	ROC_AUC	Classification
Bio-activity	MUV PubChem bioAssay	93087	17	ROC_AUC	Classification
BACE-1 benchmark set	1513	1	ROC_AUC	Classification
HIV replication inhibition	41127	1	ROC_AUC	Classification
Toxicity	Tox21 Toxicology in the 21st century	7831	12	ROC_AUC	Classification
Toxcast	8598	617	ROC_AUC	Classification
Physico-chemical	ESOL Water solubility	1128	1	RMSE	Regression
FreeSolv Solutation free energy	642	1	RMSE	Regression
Lipo Lipophilicity	4200	1	RMSE	Regression
The datasets used in our study are presented in Table 7. These datasets consist of different types of tasks related to molecule property prediction. It should be noted that during the pretraining phase, the loss function is not specific to the task types, but rather encompasses the generative loss of the language model.
We have chosen not to include certain datasets, namely SIDER and ClinTox, in our collection of datasets. The decision was based on the fact that the tasks associated with these datasets are not clearly defined and involve complex systemic phenomena, making it challenging to describe them through instructional texts. For instance, the ClinTox dataset involves determining whether drugs have passed the FDA approval, which is not an objective problem but rather a dynamic and intricate social phenomenon. The SIDER dataset focuses on describing the side effects of drugs on system organ classes, which have intricate mechanisms and a wide range of possible causes, making them difficult to be effectively conveyed through instructions.
For the Chembl property dataset that we have constructed, detailed information can be found in Table 8. These properties are sourced from the Chembl database [20] through the web API.
Table 8: Chembl property tasks and labels
Property	Label type
Aromatic rings number	Integer
cx_logd distribution coefficient	Real
cx_logp partition coefficient	Real
cx-most_apka - log10 dissociation constant	Real
Molecular masses	Real
Hydrogen bond donor number	Integer
Heavy atom number	Integer
Lipinski's rule of five violation number	Integer
Polar surface area (PSA)	Real
Quantitative Estimate of Druglikeness (QED)	Real
Rule of three passes	Bool
Rotatable bond number	Integer
The task explanation is primarily sourced from relevant papers, websites, or databases that introduce and compile the respective datasets. The specific sources utilized depend on the particular datasets under consideration. For Chembl tasks, we obtain task descriptions from the Chembl website. Descriptions for MoleculeNet tasks and PCBA are primarily sourced from the PubChem website. Certain datasets, such as Toxcast, include task descriptions within the dataset files. In the case of other tasks, like Chembl property and Physical-Chemical tasks, instructions are derived from Wiki or other papers. We list the instruction source in Table 9.
The description covers a wide range of aspects, including the family, function, and mechanism of the assay target, the assay experiment setting, the approximation method used for determining the property, and others. We describe regression tasks by introducing the relationship between the
Table 9: Data sources and classes for different stages of the model
Dataset	Instruction Source
ChEMBL Zero-Shot bioassay activity dataset	Chembl Database Wiki
CHEMBL Property
PCBA PubChem HTS bioAssay	Pubchem Database
ChEMBL Zero-Shot bioassay activity dataset	Chembl Database
CYP PubChem BioAssay CYP 1A2, 2C9, 2C19, 2D6, 3A4 inhibition	Pubchem Database
BBBP Blood-brain barrier penetration	Paper [22]
MUV PubChem bioAssay	Pubchem Database
BACE-1 benchmark set	Pubchem Database
HIV replication inhibition	Paper [46]
Tox21 Toxicology in the 21st century	Pubchem Database
Toxcast	Toxcast file
ESOL Water solubility	Paper [73]
FreeSolv Solvation free energy	Paper [7]
Lipo Lipophilicity	Wiki
task property and other properties, i.e. how to estimate these properties by other ones. However, this method is still challenging due to the model's capacity to understand complex mathematical relationships.
The instructions for each task are generated automatically by conducting searches on the databases and summarizing the descriptions. We use a mixture strategy of summarizing, combining template-based summarizing and GPT-3.5-turbo-based summarizing methods. The GPT-3.5-turbo-based summarizing method is applied by the prompt 'Summarize the assay:  $\backslash \mathsf{n}\{Descriptions\text{to be summarized}\} '$
The resulting instructions are then concatenated with relevant questions. These instructions are subsequently reviewed and validated by a professional biology Ph.D. student and slightly modified if necessary.
We then list the instructions of each dataset. For datasets with more than one task, we only list the instruction of one task as an illustration.
Chembl
"The assay is PUBCHEM_BIOASSAY: qHTS Assay for Activators of Human Muscle isoform 2 Pyruvate Kinase. (Class of assay: confirmatory), and it is Direct single protein target assigned. The assay has properties: assay category is confirmatory; assay organism is Homo sapiens; assay type description is Functional. Is the molecule effective to this assay?"
Chembl property
"The partition coefficient, abbreviated P, is defined as a particular ratio of the concentrations of a solute between the two solvents (a biphas of liquid phases), specifically for un-ionized solutes, and the logarithm of the ratio is thus Log P. When one of the solvents is water and the other is a non-polar solvent, then the log P value is a measure of lipophilicity or hydrophobicity. The defined precedent is for the lipophilic and hydrophilic phase types to always be in the numerator and denominator respectively. What is the logarithm of the partition coefficient of this molecule?"
PCBA
"The assay tests the inhibition of ALDH1A1 activity using propionaldehyde as an electron donor and NAD+ as an electron acceptor. The conversion of NAD+ to NADH is measured via an
increase in fluorescence intensity to determine enzyme activity. ALDH1A1 plays critical roles in the metabolic activation of retinoic acid and may be a target for inhibitor development in metabolic diseases. Is the molecule effective to this assay?"
CYP450
"Find molecules that can effectively inhibit Cytochrome P450 (CYP450) enzymes, particularly CYP1A2, to help reduce the risk of adverse drug events and drug-drug interactions caused by CYP450-mediated metabolic pathways. Consider the various CYP450 inhibition mechanisms such as occupying active sites or weakening enzyme activity, while keeping in mind the potential for increased side effects due to elevated blood drug concentrations. Is this molecule effective to this assay?"
BBBP
"In general, molecules that passively diffuse across the brain blood barrier have the molecular weight less than 500, with a LogP of 2-4, and no more than five hydrogen bond donors or acceptors. Does the molecule adhere to the three rules or not?"
MUV
"Protein kinase A (PKA) is an ubiquitous serine/threonine protein kinase and belongs to the AGC kinase family. It has several functions in the cell, including regulation of immune response, transcription, cell cycle and apoptosis. PKA is a cAMP dependent enzyme that exists in its native inactive form as a 4 subunit enzyme with two regulatory and two catalytic subunits. Binding of cAMP to the regulatory subunit leads to the disassembly of the complex and release of now active catalytic subunits. Is this molecule inhibitor of PKA?"
BACE
"BACE1 is an aspartic-acid protease important in the pathogenesis of Alzheimer's disease, and in the formation of myelin sheaths. BACE1 is a member of family of aspartic proteases. Same as other aspartic proteases, BACE1 is a bilobal enzyme, each lobe contributing a catalytic Asp residue, with an extended active site cleft localized between the two lobes of the molecule. The assay tests whether the molecule can bind to the BACE1 protein. Is this molecule effective to the assay?"
"Human immunodeficiency viruses (HIV) are a type of retrovirus, which induces acquired immune deficiency syndrome (AIDs). Now there are six main classes of antiretroviral drugs for treating AIDs patients approved by FDA, which are the nucleoside reverse transcriptase inhibitors (NRTIs), the non-nucleoside reverse transcriptase inhibitors (NNRTIs), the protease inhibitors, the integrase inhibitor, the fusion inhibitor, and the chemokine receptor CCR5 antagonist. Is this molecule effective to this assay?"
Tox21
"Estrogen receptor alpha (ER aplha) is Nuclear hormone receptor. The steroid hormones and their receptors are involved in the regulation of eukaryotic gene expression and affect cellular proliferation and differentiation in target tissues. Ligand-dependent nuclear transactivation involves either direct homodimer binding to a palindromic estrogen response element (ERE) sequence or association with other DNA-binding transcription factors, such as AP-1/c-Jun, c-Fos, ATF-2, Sp1 and Sp3, to mediate ERE-independent signaling. Is this molecule effective to this assay?"
Toxcast