SlowGuess/NeurIPS_2023_OCR huggingface dataset:
"APR_HepG2_CellCycleArrest_24hr, is one of 10 assay component(s) measured or calculated from the APR_HepG2_24hr assay. It is designed to make measurements of cell phenotype, a form of morphology reporter, as detected with fluorescence intensity signals by HCS Fluorescent Imaging technology.Data from the assay component APR_HepG2_CellCycleArrest_24hr was analyzed into 2 assay endpoints. \nThis assay endpoint, APR_HepG2_CellCycleArrest_24h_dn, was analyzed in the negative fitting direction relative to DMSO as the negative control and baseline of activity. \nUsing a type of morphology reporter, measures of all nuclear dna for loss-of-signal activity can be used to understand the signaling at the pathway-level as they relate to the gene . \nFurthermore, this assay endpoint can be referred to as a primary readout, because this assay has produced multiple assay endpoints where this one serves a signaling function. \nTo generalize the intended target to other relatable targets, this assay endpoint is annotated to the  $"cell cycle"$  intended target family, where the subfamily is  $"\text{proliferation}$ ". Is this molecule effective to this assay?"
ESOL
"Solubility (logS) can be approximated by negative LogP -0.01 * (MPt \u2013 25) + 0.5 . Can you approximate the logS of this molecule by its negative logP and MPt?"
FreeSolv
"The free energy of hydration can be approximated by \u0394G_hyd = \u0394G_solv, soln - \u0394G_solv, gas + RT ln (10^(-pKa))". Can you tell me the free energy of hydration (by using the negative pka) of this molecule, predicted by using \u0394G_solv and negative pka?"
"Lipophilicity is an important feature of drug molecules that affects both membrane permeability and solubility, measured by octanol/water distribution coefficient (logD at pH 7.4). What's the octanol/water distribution coefficient (logD at pH 7.4) of this molecule?"
A.2 Details of Framework Application
In our framework, we represent the labels of various tasks as strings. For assay tasks involving classification, the labels are converted to either "Yes" or "No" based on whether the molecule has
an effect on the assay. In regression tasks, the labels are transformed into numerical strings. Integer values remain unchanged, while decimal numbers are rounded to two decimal places.
To conduct zero-shot testing on our model, we generate output sequences and extract the answer from the results. For assay classification, we consider the first token generated as the answer and use the scores for the 'Yes' and 'No' tokens to compute the ROC-AUC score for classification. In regression tasks, we extract the number from the generated sequence by performing string matching using a regular expression template:  $r" - ?\backslash d +$ .?\\d*e??\\d*?"". Notably, we discovered that GIMLET consistently generates results in the correct format for all classification tasks and accurately formatted numbers for over 98% of regression testing samples, without any augmentation of restriction in the vocabulary.
A.3 Baselines Evaluation
For the baselines, we apply our instruction-based molecule zero-shot learning to their respective settings. KVPLM employs SMILES for molecule representation and utilizes masked language modeling for molecule-text data. Galactica also represents molecules using SMILES but generates the next sentence in an autoregressive manner. MoMu employs contrastive learning between the GNN-encoded molecule and the corresponding text, allowing it to score each candidate sentence for the target molecule and retrieve the best matching one. Our application of each baseline model aligns with their intended use.
It is important to note that for the baseline models, to avoid baselines generating answers in classification not in our parsing method ('Yes' and 'No'), we limit the vocabulary during generation to only include 'Yes' and 'No' in classification tasks. This restriction is achieved by utilizing the bias term in huggingface to prevent the generation of other words. However, it is worth mentioning that our model, GIMLET, does not require this augmentation and is able to generate the desired outputs without any additional constraints.
For KVPLM, we mask the answer position in the whole sentence for the model to predict. For example, for molecule CCOc1cccccc1-n1nnnc1SCC(=O)NC(=O)NCc1ccc01 and classification tasks ARE inhibitor, input to KVPLM is:
"CCOc1cccccc1-n1nnnc1SCC  $(= 0)$  NC  $(= 0)$  NCc1ccc01 Oxidative stress has been implicated in the pathogenesis of a variety of diseases ranging from cancer to neurodegeneration. The antioxidant response element (ARE) signaling pathway is important in the amelioration of oxidative stress. Is this molecule agonists of antioxidant response element (ARE) signaling pathway? [MASK]"
For Galactica, the answer is expected to be generated after reading the question. The input example is
"[START_I_SMILES] CCOc1cccccc1-n1nnnc1SCC  $(= 0)$  NC  $(= 0)$  NCc1ccc01 [END_I_SMILES]   
Question: Oxidative stress has been implicated in the   
pathogenesis of a variety of diseases ranging from cancer to   
neurodegeneration. The antioxidant response element (ARE)   
signaling pathway is important in the amelioration of   
oxidative stress. Is this molecule agonists of antioxidant   
response element (ARE) signaling pathway?   
Answer:"
For MoMu, we compute the matching score between the molecule graph and the instruction with each answer. In the example, the classification scores for 'Yes' and 'No' are computed by matching graph feature of molecule CCOc1cccc1-n1nnnc1SCC(=O)NC(=O)NCc1ccco1 with
"Oxidative stress has been implicated in the pathogenesis of a variety of diseases ranging from cancer to neurodegeneration. The antioxidant response element (ARE) signaling pathway is important in the amelioration of oxidative stress. Is this molecule agonists of antioxidant response element (ARE) signaling pathway? Yes"
"Oxidative stress has been implicated in the pathogenesis of a variety of diseases ranging from cancer to neurodegeneration.
The antioxidant response element (ARE) signaling pathway is important in the amelioration of oxidative stress. Is this
molecule agonists of antioxidant response element (ARE)
signaling pathway? No"
B.1 Discussion of Individual Encoding Module Method
The individual encoding module-based multimodal language model can be formalized as  $\mathrm{LLM}(M(G),T)$ , where  $M$  is the individual encoding module for graph data  $G$ . For example, the visual module is applied to pre-encode the image data to get the dense representation, then put into the language model as tokens embedding [4, 9, 1, 32]. Current works on molecule language models also use a GNN to get the representation of molecules to interact with the language models [17, 59, 56].
This method can be considered as decomposition of the conditional probability  $P(\hat{y} | G, T)$
$$
P (\hat {y} | G, T) = \int P _ {M} (z | G) P _ {\mathrm {L L M}} (\hat {y} | z, T) d z, \tag {6}
$$
based on the assumption that the feature distributions  $P(z|G)$  should be modeled by modality-specific modules to introduce inductive bias, and be independent of text information to help with adaptation to novel text data.
However, for the molecule-text model, individual pre-encoding modules present problems. First, graph learning relies on structure information, but the dense vectors encoded by GNN have a limited capacity to carry structure information, and language models don't have inductive bias toward graph structure. Furthermore, training the additional module is difficult due to the increased layers, since deep transformers have vanishing gradients in early layers [34, 2], which is a well-known problem of transformer. Lastly, the additional modules increase parameters and training costs.
Our method GIMLET not only overcome these issues, our approach GIMLET not only directly unifies the standard language model for graph and text without introducing additional graph encoder module, but also remains the decoupled graph encoding for better generalization.
B.2 Model Theoretical Capacity
In this section, we analyze the theoretical capacity of our modeling method.
Theorem 1 Assume for different input features and position embeddings, the transformer layers can output different output features. The transformer with distance-based relative position embedding has a stronger capacity than the 1-WL test for the graph isomorphism problem.
Proof 1 The 1-WL test is defined as the following iteration:
$$
\begin{array}{l} \chi_ {G} ^ {0} (i) = \operatorname {h a s h} \left(v _ {i}\right) \\ \begin{array}{l} \chi_ {G} (i) := \operatorname {h a s h} \left(\chi_ {G} ^ {t - 1} (i), \left\{\chi_ {G} ^ {t - 1} (j): j \in \mathcal {N} _ {G} (i) \right\}\right) (\forall i \in N), \end{array} \tag {7} \\ \end{array}
$$
where  $\chi_G$  is the label in WL test, hash is the hash function,  $\mathcal{N}_G(i)$  is the neighbor of node  $i$ .
The transformer with distance-based relative position embedding can be considered as the following mapping:
$$
\begin{array}{l} \chi_ {G} ^ {t} (i) := \operatorname {h a s h} \left(\left\{\left(d _ {G} (i, j), \chi_ {G} ^ {t - 1} (j)\right): j \in N \right\}\right) \\ = \mathrm {h a s h} \left(\left\{\left(0, \chi_ {G} ^ {t - 1} (i)\right) \right\} \right. \\ \cup \left\{\left(1, \chi_ {G} ^ {t - 1} (j)\right): j \in \mathcal {N} _ {G} (i) \right\} \\ \cup \left\{\left(d _ {G} (i, k), \chi_ {G} ^ {t - 1} (k)\right): k \in N - \mathcal {N} _ {G} (i) - \{i \} \right\}) \\ \end{array}
$$
It can be seen that the iteration of the transformer with distance-based relative position embedding includes both the node  $i$  and its neighbors  $\mathcal{N}_G(i)$ , marked by distance 0 and 1, respectively, ensuring the capacity is at least as strong as 1-WL test. It further includes other nodes far away, along with their distance, which constitutes a stronger capacity than 1-WL test. Figure 5 are two example graphs that cannot be distinguished by 1-WL test, but can be distinguished by transformer with distance-based relative position embedding.
Figure 5: Two example graphs that cannot be distinguished by 1-WL test, but can be distinguished by transformer with distance-based relative position embedding.
Theorem 2 Assume for different input features and position embeddings, the transformer layers can output different output features. GIMLET can distinguish graph-instruction pairs if graphs can be distinguished by transformer with distance-based relative position embedding, or instructions are different.
Proof 2 Because GIMLET decomposes the attention from graph nodes to text, the graph nodes can only attend to other graph nodes. Thus the encoding capacity of graph data is the same as a single transformer with distance-based relative position embedding for graph data.
Along with the assumption of transformer layers, GIMLET is able to distinguish graph-instruction pairs if graphs can be distinguished by transformer with distance-based relative position embedding, or instructions are different.
B.3 Detalied Related Work
We present a detailed related work here, due to the space limitation of paper.
Molecule Representation learning In recent years, there has been a growing interest in developing molecular representation learning for downstream tasks like drug discovery and other applications. One approach that has received considerable attention is utilizing language modeling techniques to acquire molecular representations based on Simplified Molecular Input Line Entry System (SMILES) strings [68, 10]. Although sequence-based representations have demonstrated success in some applications, concerns have been raised about their capability to incorporate all pertinent substructure information. To address this limitation, some researchers have proposed the use of Graph Neural Networks (GNNs) to model molecules as graphs [21, 83, 26], potentially providing a more comprehensive and accurate representation of the molecular structure.
Existing GNNs follow the message-passing paradigm and suffer from problems like long-range dependency vanishing and over-smoothing. Recently, Graph Transformer [52, 80] has been proposed to better encode structures of graphs. The Graph Transformer is inspired by the Transformer
architecture, which has shown remarkable performance in natural language processing [65, 13, 41]. The Graph Transformer extends the Transformer architecture to the graph domain, allowing the model to capture the global structure and long-range dependencies of the graph [86, 15, 31, 29, 74, 48, 42, 80, 8, 43, 11, 5, 23, 88].
Molecule Pretraining To fully explore the inherent structural information of molecules on a large scale and transfer useful information to downstream tasks, significant efforts have been made to address the inadequacies in molecular pre-training. Supervised pretraining is commonly used for learning useful representations [26, 80, 62]. As for unsupervised pretraining, one approach involved using an generative pre-training strategy on molecular SMILES strings [68, 25, 10, 3, 53] and Graph [26, 37, 52, 87], which was followed by recent works adopting the contrastive paradigm that aligns representation of augmented views of the same graph but keeping views from other graphs away [67, 60, 24, 83, 82, 63, 78, 19, 61, 70, 76, 69, 40].
The pretraining methods mentioned focus on obtaining representations for supervised training. However, for natural language instruction-based zero-shot graph learning, it's necessary to incorporate natural language into the pretraining process. Several studies have explored molecule structure-text multimodal pretraining. One class of method is the SMILES based language model, including KVPLM [85] and MolT5 [16], which use SMILES strings and text for joint representation and translation. Another work Galactica [64] explored the multi-task molecule task learning with instruction. Some other works acquire advanced representations for molecules by GNN, such as Text2Mol [17], MoMu [59], MoleculeSTM [39], and CLAMP [56], trained by contrastive learning between molecule graph and text description for molecule retrieval and caption tasks. MoleculeSTM and CLAMP explored molecule editing and property prediction with instructions. However, none of these works address the zero-shot fashion on complex molecule tasks like property prediction, due to constraints imposed by the pretraining methodology that not addressing the instruction-following ability, and their model capacity for representing molecule graphs.
Instruction-based zero-shot learning Instruction-based zero-shot learning is an innovative approach that leverages natural language instructions and definitions to enable neural models to solve a variety of tasks [50, 6, 55, 18, 89, 44, 45, 49]. By providing a human-readable prompt, this method enables easier and more efficient specification of the learning task by utilizing knowledge about the task without data. To enhance the model's ability to follow instructions, some researchers have employed instruction-based pretraining techniques [54, 71, 12, 47], which explicitly train language models to solve tasks with instructions. Besides natural language processing, instruction-based zero-shot learning is also studied in multimodal domains like images [4, 9, 1, 32].
C.1 Experiment setting
Our model only utilizes the basic features [26, 62] of molecule graphs, which do not include additional features like ring markers. Specifically, it utilizes the first two dimensions of node features and the first two dimensions of edge features processed by ogb.smiles2graph. Therefore, the effectiveness of GIMLET predominantly stems from its architectural design and pretraining rather than the graph features it incorporates.
Following the standard supervised setting in previous studies [26], we utilize the scaffold strategy [51] to partition datasets into three subsets: the training set, validation set, and testing set with a ratio of 0.8, 0.1, 0.1. The scaffold strategy is a deterministic approach that involves sorting the data based on the scaffold, which represents the molecular structure. While this strategy aids in dataset partitioning, it can introduce a significant domain gap between the training and testing sets, thereby increasing the challenge of generalization.
For zero-shot, we report the results on the testing sets, ensuring the comparability of our results to previous works. For few-shot, we report the result of the best validation model on the testing set, the same as previous works and other supervised baselines [51].
Many datasets encompass multiple tasks. To evaluate these datasets, we conduct separate testing for each task, accompanied by their respective instructions. For datasets with multiple tasks, we report the average ROC-AUC score for each task, following the methodology established in previous works [26].
C.2 Detailed Zero-Shot Result
We list the full zero-shot result of GIMLET and baselines in Table 10, 11, and 12. The standard deviation for supervised results are denoted after  $\pm$ , and the multi-task setting results of Galactica are denoted in parentheses with italic. We also include the instruction-based zero-shot result reported in recent baseline CLAMP [56] which is tested by their instruction, denoted by italics too. CLAMP is a contrastive pretrained model with ensembled encoders for molecule and text. The parameter number for CLAMP's result is not clearly stated in their paper but should be larger than 10B as they use sT5 language model [50] XXL variant (11B) as one of the ensembled language models.
Table 10: Zero shot performance over Bio-activity tasks
Method	# Parameter	Type	bace	hiv	muv	Avg. bio
KVPLM	110M	0.5126	0.6120	0.6172	0.5806
MoMu	113M	Zero Shot	0.6656	0.5026	0.6051	0.5911
CLAMP	>10B	0.6476	0.8067	-	-
GIMLET	64M	0.6957	0.6624	0.6439	0.6673
Galactica-125M	125M	Multi Task	0.4451(0.561)	0.3671(0.702)	0.4986	0.4369
Galactica-1.3B	1.3B	0.5648(0.576)	0.3385(0.724)	0.5715	0.4916
GCN	0.5M	0.736±0.030	0.757±0.011	0.732±0.014	0.742
GAT	1.0M	0.697±0.064	0.729±0.018	0.666±0.022	0.697
GIN	1.8M	Supervised	0.701±0.054	0.753±0.019	0.718±0.025	0.724
Graphormer	48M	0.7760±0.015	0.7452±0.014	0.7061±0.027	0.7424
Graphormer-p	48M	0.8575±0.006	0.7788±0.012	0.7480±0.020	0.7948
Table 11: Zero shot performance over Toxicity tasks
Method	# Parameter	Type	tox21	toxcast	Avg. tox
KVPLM	110M	0.4917	0.5096	0.5007
MoMu	113M	Zero Shot	0.5757	0.5238	0.5498
CLAMP	>10B	0.6058	0.5383	0.5721
GIMLET	64M	0.6119	0.5904	0.6011
Galactica-125M	125M	Multi Task	0.4964(0.543)	0.5106(0.518)	0.5035
Galactica-1.3B	1.3B	0.4946(0.606)	0.5123(0.589)	0.5035
GCN	0.5M	0.749±0.008	0.633±0.009	0.691
GAT	1.0M	0.754±0.005	0.646±0.006	0.700
GIN	1.8M	Supervised	0.740±0.008	0.634±0.006	0.687
Graphormer	48M	0.7589±0.004	0.6470±0.008	0.7029
Graphormer-p	48M	0.7729±0.006	0.6649±0.006	0.7189
Table 12: Zero shot performance over Pharmacokinetic tasks
Method	# Parameter	Type	bbbp	cyp450	Avg. pha
KVPLM	110M	0.6020	0.5922	0.5971
MoMu	113M	Zero Shot	0.4981	0.5798	0.5390
CLAMP	>10B	0.4788	-	-
GIMLET	64M	0.5939	0.7125	0.6532
Galactica-125M	125M	Multi Task	0.6052(0.393)	0.5369	0.5711
Galactica-1.3B	1.3B	0.5394(0.604)	0.4686	0.5040
GCN	0.5M	0.649±0.030	0.8041±0.005	0.7266
GAT	1.0M	0.662±0.026	0.8281±0.004	0.7451
GIN	1.8M	Supervised	0.658±0.045	0.8205±0.012	0.7392
Graphormer	48M	0.7015±0.013	0.8436±0.003	0.7725
Graphormer-p	48M	0.7163±0.009	0.8877±0.004	0.8020
The result in parentheses represents the outcome of the multitask setting, also referred to as weakly supervised in the original paper, where the same instructions are used for both pretraining and testing. While Galactica has been exposed to the same task instructions, it actually employs multitask learning with instructions serving as task identity.
Even in comparison to Galactica's multitask result, GIMLET demonstrates comparable or superior performance on most datasets. This highlights the ability of GIMLET to perform zero-shot tasks with high quality.
Figure 6: Scatter of GIMLET over baselines. Below the diagonal line  $x = y$  means our method performs better.
The disparity between the multitask result and the tested result with our instructions is due to the gap between their instructions and ours, which indicates that Galactica relies on specific task instructions for task recognition, without a true understanding of the instructions. As a result, it exhibits poor generalization to other instruction forms. Note that Galactica even do not surpass KVPLM and MoMu which are also zero-shot learning methods.
GIMLET exhibits superior performance compared to the larger model CLAMP on the majority of datasets, with the exception of HIV. It is important to highlight that our model is significantly smaller in size than CLAMP, underscoring the effectiveness of our unified graph-text language model. Additionally, it should be noted that CLAMP lacks the capability to handle regression tasks due to its contrastive model architecture, whereas our encoder-decoder architecture enables us to successfully tackle a wide range of task types.
Significantly, the supervised results shed light on the task difficulties associated with each dataset. This showcases GIMLET's capability to effectively solve molecule tasks in a zero-shot manner, approaching the performance of supervised results. Furthermore, our pretraining tasks yield an average performance improvement of 3 percent for Graphormer, with the largest gains observed in Bioactivity tasks and the smallest in Toxicity tasks. This suggests that there still exist gaps between the pretraining data and our downstream tasks, addressing the zero-shot setting of our dataset.
In Figure 6, we present scatter plots comparing GIMLET with KVPLM and MoMu across all tasks. The diagonal line represents the equality line where  $x = y$  indicates our method outperforms the baseline. Notably, it is evident that GIMLET consistently performs significantly better than random guessing and surpasses the baselines on all tasks.
We plot the scatter of regression tasks in Figure 7. The plot clearly demonstrates a strong correlation between the predicted and actual values for ESOL and Lipo.
C.3 Detailed Few-Shot Results
In both classification tasks and regression tasks, we fine-tune the last linear layer of all models using their respective modeling loss.
It is important to note that the instruction-based few-shot approach is trained on each task individually, while supervised baselines are trained on multiple tasks from the dataset. Therefore, comparing these two approaches may not be strictly fair, as the multitask learning of the supervised baseline can contribute to improved task performance.
The results for few-shot learning on each dataset are presented in Figure 8. It is evident that, across the majority of datasets, GIMLET demonstrates improvement as the number of few-shot examples increases. In fact, it even outperforms or matches the performance of the supervised GIN on several
Figure 7: Scatter of GIMLET on generative tasks.
datasets, such as bace, bbbb, and esol. There is also observable enhancement in performance across various datasets when employing few-shot learning, including tox21, toxcast, lipo, and freesolv.
There is not result of MoMu on regression tasks, because MoMu is a contrastive model between graph and text, which cannot handle regression tasks.
C.4 Detailed Ablation Results of Pretraining
The results of pretraining ablation for each dataset are presented in Table 13, 14, 15, and 16. The findings indicate that both bioactivity assay and physico-chemical properties offer significant benefits for all the downstream tasks, demonstrating positive transfer across different domains.
Table 13: Pretraining ablation study on Bio-activity tasks
bace	hiv	muv	Average_bio
bioactivity assay only	0.6390	0.6772	0.6044	0.6402
physico-chemical only	0.4648	0.5461	0.4572	0.4894
both	0.6957	0.6624	0.6439	0.6673
Table 14: Pretraining ablation study on Toxicity tasks
tox21	toxcast	Average_tox
bioactivity assay only	0.5726	0.5625	0.5676
physico-chemical only	0.4478	0.5017	0.4748
both	0.6119	0.5904	0.6011
C.5 Instruction Robustness
To test the robustness of GIMLET, the Instructions are rephrased by GPT-3.5-turbo. There are four types of rephrasing, realized by the following prompts:
'Rephrase the text of the following prompt: \n'
'Rephrase the text of the following prompt longer: \n'