GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning
Authors: Haiteng Zhao
Shengchao Liu
Chang Ma
Hannan Xu
Jie Fu
Zhi-Hong Deng
Lingpeng Kong
Qi Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Instruction
Molecule
Zero Shot
Graph
Language Model

Abstract: Molecule property prediction has gained significant attention in recent years. The main bottleneck is the label insufficiency caused by expensive lab experiments. In order to alleviate this issue and to better leverage textual knowledge for tasks, this study investigates the feasibility of employing natural language instructions to accomplish molecule-related tasks in a zero-shot setting. We discover that existing molecule-text models perform poorly in this setting due to inadequate treatment of instructions and limited capacity for graphs.  
To overcome these issues, we propose GIMLET, which unifies language models for both graph and text data. By adopting generalized position embedding, our model is extended to encode both graph structures and instruction text without additional graph encoding modules. GIMLET also decouples encoding of the graph from tasks instructions in the attention mechanism, enhancing the generalization of graph features across novel tasks. We construct a dataset consisting of more than two thousand molecule tasks with corresponding instructions derived from task descriptions. We pretrain GIMLET on the molecule tasks along with instructions, enabling the model to transfer effectively to a broad range of tasks. Experimental results demonstrate that GIMLET significantly outperforms molecule-text baselines in instruction-based zero-shot learning, even achieving closed results to supervised GNN models on tasks such as toxcast and muv.

GIMLET: A Unified Graph-Text Model for Instruction-Based Molecule Zero-Shot Learning
Haiteng Zhao', Shengehao Liu?, Chang Ma², Hannan Xu',
Jie Fu², Zhi-Hong Deng', Lingpeng Kong", Qi Liu 1 Peking University 2 Mila 3 The University of Hong Kong University of Oxford Hong Kong University of Science and Technology
Abstract
Molecule property prediction has gained significant attention in recent years. The main bottleneck is the label insufficiency caused by expensive lab experiments. In order to alleviate this issue and to better leverage textual knowledge for tasks, this study investigates the feasibility of employing natural language instructions to accomplish molecule-related tasks in zero-shot setting. We discover that existing molecule-text models perform poorly this setting due to inadequate treatment of instructions and limited capacity for graphs. To overcome these issues, we propose GIMLET, which unifies language models for both graph and text data. By adopting generalized position embedding, our model is extended encode both graph structures and instruction text without additional graph encoding modules. GIMLET also decouples encoding of the graph from tasks instructions in the attention mechanism, enhancing the generalization of graph features across novel tasks. We construct dataset consisting of more than two thousand molecule tasks with corresponding instructions derived from task descriptions. We pretrain GIMLET the molecule tasks along with instructions, enabling the model transfer effectively to a broad range of tasks. Experimental results demonstrate that GIMLET significantly outperforms molecule-text baselines in instruction based zero-shot learning, even achieving closed results to supervised GNN models on tasks such as toxcast and muv.
Introduction
Molecule machine learning has gained significant attention in recent years 52 85 77. including
tasks like property prediction [1] molecule design 88 35 28 27], and others, which have broad applications in biomedical research. The primary method of molecule tasks is graph machine learning (21]. where graphs are employed to represent molecule topology. Presently, graph machine learning approaches for molecules mainly follow the pretraining and finetuning paradigm [6 60) 40) 84]
After the pretraining on large molecule corpus, models are able to encode informative molecule representations and generalize to downstream tasks by supervised finetuning.
One limitation of the supervised finetuning approach the requirement labeled data acquire task-specifie features for prediction on downstream tasks. This requirement poses challenge in scenarios where obtaining labeled data is arduous or costly, especially in the field of molecules, and the training cost also restricts the efficiency of downstream tasks. Additionally, the supervised method exclusively relies on labeled data for acquiring task information, and is therefore unable to incorporate other additional information about the task. For instance, there exists wealth of information regarding molecule assay tasks often provided in textual descriptions. Unfortunately, the supervised method fails to leverage such information, thereby restricting its flexibility.
'The code, model, and data are available at htpp://ginabccom/hho-hhiMMLMLLL
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this work, we propose to investigate the feasibility of employing instructions to accomplish molecule-related tasks in a zero-shot setting. The instructions, also referred to as prompts [50] []. constitute natural language descriptions of the task to be executed. This is inspired by recent progress in natural language processing, where the large language models possess strong generalization performance to unseen tasks by following instructions [71 47]. This approach eliminates the need for labeled data and leverages the textual knowledge available for downstream tasks.
Several studies have explored the molecule-text language models, employing either autoregressive pretraining using language models on both the text and molecule strings (SMILES) [6
85 64, or
contrastive pretraining of molecule and text where molecules are encoded by Graph Neural Networks (GNN) [7 59] 39 [56]. However, these works lack the investigation into instruct-based zero-shot for downstream tasks, and our experiments have empirically demonstrated their limited performance.
We conjecture that the constraints of current molecule-text language models are mainly imposed by their inadequate treatment of instructions during pretraining and their limited capacity to represent molecule structures. First, the pretraining corpora of these models lack task description about descriptions for abundant molecule tasks as well as the supervised signal. They mainly address the capacity of general molecule-related text understanding, which may be insufficient to capture details of complex instructions to molecule properties. Second, these models' capacity is limited in representing graph structures of molecules. Molecular tasks heavily rely on understanding the structure information, while existing multimodal methods either encode SMILES sequence representation of molecules by language model or encode molecular graphs into dense vector representation using GNN, which are not conducive for language models to gain a deep understanding of graph features.
To facilitate natural language instruction- based graph learning for zero-shot molecule tasks, propose an innovative structured language model called GIMLET (Graph Instruction based MolecuLe zEro-sho'T learning). First, GIMLET extends large language models for both graph and text data by applying transformer mechanisms with generalized position embedding, ensuring strong capacity for both the learning of graph structure representations and the executing of instruction texts without introducing additional graph encoding modules. We further enhance the model by decoupling the encoding the graph from specific tasks via employing attention masks. This allows for improved generalization of graph features across novel tasks.
In addition to the advanced model architecture, our approach incorporates entruuction-bsed pre- training on an extensive dataset of molecule tasks with textual instructions. We construct a dataset comprises of 2K tasks accompanied by corresponding instructions tailored for instruction- based molecule zero-shot learning framework. Throughout the pretraining process, molecule graphs are paired with natural language instructions for molecule-related tasks, and the supervision signal is provided to train the model in executing various molecule tasks. This methodology empowers GIMLET to comprehend specific instructions for downstream tasks expressed in natural language and transfer acquired knowledge to a broad range of tasks effectively.
Our experimental results show that GIMLET outperforms molecule-text baselines by substantial margin in instruction-based zero-shot learning, which is closed to supervised GNN on tasks like muy and toxcast. Additionally, GIMLET also exhibits impressive performance in few-shot learning and demonstrates robustness to instruction. In summary, our contributions are as follows:
We present comprehensive investigations of natural language instruction based graph zero- shot learning for molecule tasks, to reduce reliance on labeled data and leverage task textual knowledge. To accomplish this framework, we construct molecule dataset consisting of two thousand tasks with instructions derived from task descriptions, which is open-sourced.
We propose GIMLET, which extends language models to handle graph and text data. By applying the transformer mechanism with generalized position embedding and decoupled attention, our model learns graph structure representations and executes instruction texts without additional graph encoding modules. Instruction based pretraining is applied for GIMLET to comprehend specific instructions expressed in natural language and transfer to a broad range of zero-shot tasks.
Our experiment results outperform other molecule-text models by a substantial margin and demonstrate promising results in instruction-basee zero-shot molecule tasks, demonstrating the viability of this framework, Additionally, GIMLET exhibits impressive performance in few-shot learning and demonstrates robustness to instruction.

---

GIMLET
VEY23
Wn 0E0E
Goph-Text Postion Encoding
Unified Graph-Text Transformer Encoder
Transformer Decoder
01
02
03
ON
Distance aware Encoding
Taskc Agonists of ARE
OH
antooitonn dement important the oration pedetive stress. this metecule agonists ARE signaling pattway?
Instruction Prodiction
Task: Solubiilty
LogF -0.01 (MP-25) Can spprodmate this molecule negative log? and HP?
3.11
pass
Figure Our framework handles molecule tasks in the zero-shot fashion by natural language instruction. Within GIMLET, we employ distance- based joint position embedding encode graphs and instruction texts. Additionally, we utilize attention masks to decouple the graph encoding process.
2
Related Work
Molecule Representation Learning One approach for molecular representation learning is utilizing language models acquire molecular representations based on Simplified Molecular Input Line Entry System (SMILES) strings [68] [0]. Toward stronger capability to incorporate pertinent substructure information, Graph Neural Networks (GNNs) are proposed to model molecules graphs [2] 83 26
36). Existing GNNs follow the message-passing paradigm and suffer from problems like long-range dependency vanishing and over-smoothing. Recently, graph transformer [52] 80 has been proposed
to better encode structures of graphs, illustrating effectiveness in molecule tasks 42
88
Molecule Pretraining To fully explore the inherent structural information of molecules scale, sigmificant efforts have been made to molecular pretraining. Supervised pretraining is commonly approach and graph [66 37
large
used for learning useful representations 26 80, 62 As for unsupervised pretraining,
involved the generative strategy on molecular SMILES strings [68]
25
10
3
53
38 87 which followed by recent works adopting the contrastive paradigm to distinguish
augmented views of the same graph and other graphs 667 60 24
63
78
61
70 76
Besides the structure-only pretraining. few recent works incorporate natural language into molecule pretraining. One class of method is the SMILES-based language model, including KVPLM [55] and MolT5 (16), which use SMILES strings and text for joint representation and translation. Another work Galactica [64] explored the multi-task molecule task learning with instruction. Some other works acquire advanced representations for molecules by GNN, such Text2Mol (17), MoMu [59]. MoleculeSTM [39], and CLAMP [56), trained by contrastive learning between molecule graph and text description for molecule retrieval and caption tasks. MoleculeSTN, and CLAMP explored molecule editing and property prediction with text. However, previous works lack the investigation into instruct- based zero-shot scenarios for complex molecule tasks like property prediction.
Instruction- based Zero-Shot Learning Instruction- zero-shot learning is an innovative ap- proach that leverages natural language instructions enable neural models solve variety of tasks 50 6 55 T8
44 45, 49). To enhance the model's ability follow instructions, some researchers have employed instruction- based pretraining techniques [54] 71 [2 47). which explicitly train lan- guage models to solve tasks with instructions. Besides natural language processing, instruction-ban based
zero-shot learning is also studied in multimodal domains like images 4 9 32
3 Method
3.1
Problem Formulation and Framework Overview
The molecule task is denoted as and consists of set of graph data and their corresponding labels (Gs, v.). molecule graph G (N, F,, v, e) includes nodes N (I,...,,u n) and edges E c Nx N, while v and represent node features and edge features, respectively. The label y" can take various

---

forms, such as class labels or numerical values, depending on the type of task. The instruction of T is denoted as T", which is a sentence (01...., Pm] describing the task.
In the supervised approach, a model Fo predicts the label ý given a graph G, i.e., ý = Fo- (G), by supervised finetuning individually for each downstream task т. The limitation is that it relies on labeled data to learn the corresponding parameters and output modules, and finetuning does not enable the model effectively utilize extra task-related knowledge.
To overcome these limitations, our framework leverages natural language instructions T to provide the model with task information, illustrated in Figure I Our zero-shot graph learning framework incorporates molecule graphs G and task instructions T into a graph-text language model GIMLET and decodes the output text uniformly for different tasks, i.c. Dstr GIMLET(G, T"), where Ostr is the label string. This description-bant based instruction framework empowers the model to handle a wide range of molecule tasks as long they can be described in natural language, and the uniform text decoding approach accommodates various types of tasks including classification or regression.
Previous pretrained molecule-text models perform poorly in our framework due to the inadequate treatment of instructions and limited capacity for graphs. Our method addresses these from two aspects: First, we propose a unified language model GIMLET for both graph and text, towards the stronger capacity to represent molecules and instructions; Second, we adopt instruction- based pretraining to GIMLET, enabling generalization to new tasks based only on the instructions.
3.2 Unified Graph-Text Transformer
The common method for multimodal language models is obtaining the feature of the other modality by applying an additional encoding module, then integrating it into the language model, as in other molecule language models with GNN encoder [17 59 56). The individual encoding module benefits for decoupling the graph feature encoding from instructions, i.e., the features of graph data can be independent of task instructions in the early encoding stage, helping for generalization when the
distribution of tasks changes.
However, for the molecule-text model, individual pre-encoding modules present problems. First, graph learning relies on structure information, but the dense vectors encoded by GNN have a limited capacity to carry structure information. Furthermore, training the additional module is difficult due to the increased layers, since deep transformers have vanishing gradients in early layers 34 2]. Lastly, the additional modules increase parameters and training costs.
To overcome these issues, we propose a novel approach GIMLET which not only directly unifies the standard language model for graph and text without introducing additional graph encoder module, but also remains the decoupled graph encoding for better generalization.
Given graph G and text input T, to utilize GIMLET, we represent the graph nodes and text tokens as tokens. The resulting hidden states are denoted as H = [hi...., hn, hn+1,..., hm+ e ((+++))xd Xds. for corresponding n graph nodes and m text tokens.
In this study, we choose T5 (50) as the backbone language model, due to the encoder-decoder architecture suitable for non-sequential encoding and text output. It utilizes the relative position embedding method [57] to represent sequential structure. In attention layer Attn with parameter WV, WQ, R E Rdd xdv and Wo E Rd xdv, relative position embedding for i-th and j-th token is formalized as
Àg (h-W9) ( + b((.)), softmax((A) Attn(H) AHWWWO
(I)
Vdh
where b(i.j) is embedding of the relative distance between i and j. i.e. i i-j j for sequence. For graph-text joint data, we construct the position embedding b(i, in E.q. by the conjunction of different types of distances. For the relative position of graph nodes, we adopt the graph shortest distance, which has been widely used in the literature of graph transformer [1]8), We also use unidirectional constraint in attention to decouple the graph encoding from the instruction. The overall form of position embedding for GIMLET is:
b(i.j) blost b
Mean b
(2)
kESP(F.))
where iP, JM and bE are position bias, masking, and path embedding, individually. The relative position POS in ()))(..)) is defined as the conjunction of different types of distances between tokens,

---

which allows for the effective encoding of both graph and text data, as well as their interaction:
i-
if in++1sij n
POS (1,5) =
GRAPH SHORTEST DiSTANCE(i,))
ir i
(3)
< CROSS>
otherwise
where <CROSS> is a special distance token held out for cross distance between graph and text tokens. 1/ aims to represent the cross mask used to decompose graph encoding from text instructions. imposes unidirectional constraint from the graph to the text:
M b
0o if S n and
otherwisc 0
(4)
With the unidirectional constraint, graph tokens are limited to attending only to other graph tokens. On the other hand, instruction tokens have the ability to receive information from both instructions and graphs. This approach allows us to separate the encoding of graphs from instructions, enabling instructions selectively utilize graph features for various downstream tasks.
Finally, Mc).) be is the mean pooling of the edge features by in the shortest path SP(i,) j) between node i and J. which is only defined between graph node tokens, as used along with graph shortest distance in [80].
The generalized position embedding is applied to the encoder transformer. During decoding, the decoder generates the answer based the text features outputted by the encoder.
GIMLET unifies graph and text data by a single language model, which has the following merits: (i) In comparison to additional encoding module methods, GIMLET not only avoids the challenges of training additional front layers but also provides a stronger capacity for handling graph data by introducing graph inductive bias to the whole transformer. (ii) Our method leverages both the existing knowledge within the language model and facilitates learning to execute instruction- baasdd graph tasks through standard instruction-based learning the language model. This approach eliminates the need for additional modeling costs. (iii) The decomposed encoding of graph data retains the advantages of the individual encoding module, reducing task disturbance and ensuring that data features remain independent of task instructions. This enhances generalization for novel instructions. We validate these claims in experiments Subsection
3.3 Pretraining and Datasets
Heavy atoms counting (From Chembl property) "Heavy etom refers any alom that not hycrogen. How manly heavy atoms do the molecule have?"
Pro train Data
Large scale
Boological Assay
Tixccity
Phamolinetic Ryssicoddeemica
Blo excivityy
Inhibitors of Schistosoma Mansoni Peroxiredoxins (From Chembl) "The functional assay named qHTS Assay for the Innibitors of Schistosoria Mansoni Peroxiredosins, related two pubchem assays, namely Confirmation Concentration- Response Assay Inhibitors the Schistosome mansoni and Schistosoms Mansoni Parodradoxins (Ph2)) and throredoxin glutathlone reductase (TGR) coupled assay. The asaay calegory is als confirmatory and perlains to the Schistosoma mansoni organism. Is this molecule effective to the assay?"
HBV
Ton22 (171
PCBA (1228)
MUV (17)
Cascade
CHEMBL 11048]
ToxCCsst (617)
DIEEEEL Z0ro s'ot
Inhibition of receptor SF-1 (From MUV)
Pre-training Tasks
"The nudlea receptor SF-1 (steroldogentc factor-1) 1) is expressed in the pituitary, testes, regulates steroid hormone productinn many levels,
Downstream Tasks
ovaries, adrenal induding direct regulation expression major hormone synthesis. Is this molecule inhibitor SF-17"
enzymes Involved sterold
DHEHIL (22)
Toxicity to ARE signaling pathway (From Tox21)
S+:
1i0
Physical Chomical Property
"Oxidelive hes implicated in pathogenesis variely disceses renging from cancer to neurodegeneration. The antiowidant resporse element (ARE) signaling pallway a important in the arriioaation of oxidelive stres, Is this molecule agonisis of antioxidant response elament (ARE) signaling pathway?"
Figure 2: (Left) Illustration of datasets. Circle size corresponds to task number. Tasks are organized by category. Tasks on the top are more related to biological assay, on the bottom need more chemical and physical properties. GIMLET is trained on pretraining tasks, then tested on downstream tasks in the zero-shot setting. (Right) Our task instructions contain task explanations and questions.
Pretraining Our approach leverages the generalization capabilities acquired through learning from the instructions provided during pretraining, where comprehensive linguistic information and knowledge related to molecular tasks are learned from the provided instructions. The pretraining process is
5

---

conducted in supervised manner using task labels. The loss function for supervised pretraining can be formalized as follows:
Σ
Σ
log Punuar(VitrrlGG;
(5)
(G. JET
where PaIMLET is the model likelihood for the label string, Vatr,d represents the label string token number aiming to normalize the loss of long sequences, and |r] is the task sample number. The details of pretraining and downstream zero-shot testing are in Appendix.
Pretraining Dataset To effectively pretrain GIMLET for downstream tasks, it is crucial to include large number of tasks to provide the model with an extensive task corpus. To this end, we select Chembl [20] as the pretraining dataset, which is widely used for supervised graph pretraining [26 62]
It consists of 1,310 prediction target labels from biological assays for drug discovery. We divided 80%
of the tasks and 80% of the molecules in random for pretraining, while the remaining non-overlapping tasks were reserved for zero-shot testing. Additionally, we constructed the Chembl- property dataset, which encompasses various physical and chemical properties available in the Chembl database [20] like molecule weights, log p values, hydrogen bound acceptor numbers, etc. Full details are in Appendix. We validate the effect of Chembl biological tasks and Chembl-propert physico-chemical tasks in pretraining in Subsection
Chembl-property
4.3
Downstream Dataset We target a large range of molecule tasks, as shown in Figure 2 First, we include large-scale datasets PCBA (72), which contains 128 biological assay tasks with abundant molecules. We also include the hold-out zero-shot set of Chembl, noted as Chembl Zero-Shot. These two datasets form a large-scale dataset. We also target tasks from MoleculeNet 75). popular benchmark for molecule properties prediction. We adopt the dataset categorization method in [58] which classifies MoleculeNet datasets into four categories: Physico-chemical tasks, Bio-activity tasks, Toxicity tasks, and Pharmacokinetic tasks. Additional dataset CYP450 [33] is also included in the Pharmacokinetic tasks. These tasks cover diverse aspects of molecule properties, posing a significant challenge for unified model to simultaneously handle them in zero-shot fashion.
Instructions To provide essential background information and context for each task, we include task explanations and descriptions in our instructions. The description covers a wide range of aspects, including the family, function, and mechanism of the assay target, the assay experiment setting, the approximation method used for determining the property, and others. See examples in Figure 2 Instructions for all tasks are available in the code file. The task explanation is primarily sourced from websites and databases that introduce and compile the respective datasets, or relevant papers. Details are in Appendix. The descriptions are then concatenated with relevant questions as instructions. These instructions are subsequently reviewed and validated by a professional biology Ph.D. student.
4 Experiment
In the experiments, we investigate the following inquiries: (i) Can GIMLET effectively handle zero- shot molecule property tasks by instructions? (ii) Can GIMLET performs better by few-shot learning? (iii) What impact does model architecture have on the performance of GIMLET? (iv) How does pretraining affect the performance of GIMLET? (v) How does the form of instruction influence GIMLET for molecule zero-shot learning?
4.1 Instruction-Bant Zero-Shot Learning
Baselines In the zero-shot setting, we compare GIMLET with three molecule-text models: SMILES- base language model KVPLM (85), Galactica (64). and GNN-language model MoMu [59]. Other molecule-tex models [66] I7 39 either haven't released parameters or are difficult to handle zero- shot setting. Notably, the pretraining of Galactica includes the MoleculeNet datasets, which is thus not strictly zero-shot on some of our tasks. We report the result of all the baselines with our zero-shot learning framework and instructions. The details of baseline evaluation in Appendix.
To establish upper bounds for task performance, we also illustrate the supervised results of popular graph models. For GNNs, we includes GCN [30], GAT [66]. and GIN [79]. We also include the graph transformer Graphormer [80]. To mitigate the impact of our pretraining, we additionally perform supervised pretraining Graphormer using pretraining datasets, referred to Graphormer-p.

---

Settings Following the standard supervised setting in previous studies [26], we adopt the Scaffold split [ST] with a ratio of 0.8, 0.1. 0.1 for all the datasets, and report results on the testing sets. ensuring the comparability of our results to previous works. For classification tasks, we employ ROC-AUC as the evaluation metrie, while for regression tasks, we utilize RMSE.
Table I: Zero-shot performance (ROC-AUC) over Bio-activity. Toxicity, and Pharmacokinetic tasks.
Method
#Puram
Type
bace
hiv muv Avg. bio
lox21 loxcast
Avg. lox
bbbp
eyp450
Avg. phu
KVPLM
110M
0.5126
0.6120 0.6172
0,5806
0.4917
0.5096
0.5007
0.6020
0.5922
0.5971
MoMu
113M
0.6655 0.5026 0.6051
0.5911
0.5757
0.5238
0.5498
0.4981
0.5798
0.5390
Galactica- 125M
125M
Zero Shot
0.4451
0.3671 0.4986
0.4369
0.4964
0,5106
0.5035
0.6052
0.5369
0.5711
Galactica- 133
1.3B
0.5648 013385 0.5715
0.4916
0.4946
0,5123
0.5035
0.5394
0.4686
0.5040
GIMLET (Ours)
64M
0.6957 0.6624 0.6439
0.6673
0.6119
0.5904
0.6011
0.5939
0.7125
0.6532
GCN
0.5M
0.736 0.757 0.732
0.742
0.749
0.633
0.691
0.8041
0.7266
GAT
1.0M
0.697 0.729
0.606
0.697
0.754
0.646
0.700
0.662
0.8281
0.7451
GIN
1.8M
Supervised
0,701 0.753
0.718
0.724
0.740
0.634
0.687
0.658
0.8205
0.7392
Graphormer
48M
0.7760 01.4552
0.7061
0.7424
0.7589
0.6470
0.7029
0.8436
0.7725
Graphormer-p
48M
0.8575
0.7788
0,7480
0.7948
0.7729
0.6649
0.7189
0.7163
0.8877
0.8020
Table 3: Zero-Shot performance (RMSE) on Physical- chemical datasets.
Table 2: Zero-shot performance (ROC- AUC) over large scale molecule tasks.
Method
Type
ESOL
Lipophilicity
FreeSolv
Avg. ply
KVPLM
Method
Chembl Zero-Shot
PCBA
MoMu
Zero Shot
KVPLM
0.4155
0,4811
GIMLET (Ours)
1.132
1.345
5,103
2.527
GON
1,351
0.760
2.119
1.403
MoMu
0.5002
0,5150
GAT
1,253
0.770
2.493
1.505
Galactica-1255
0.6461
0,4800
GIN
Supervised
1,243
0.781
2,871
1.632
Galactica-I.3B
0.4818
0,5202
Graphonner
0.901
0.740
2.210
1.284
GIMLET (Ours)
0.7860
0.6211
Gmphormer-p
0.804
0.675
1,850
1,110
Results We report the result of different types of downstream tasks in Table[]] The result of GIN, GCN and GAT for MoleculeNet are from (26) which we mark by italic. We observe that in the zero-shot setting, GIMLET outperforms most baselines on the majority of datasets, except for bbbp where GIMLET also performs comparably to the baselines. In terms of the average performance across task categories, GIMLET outperforms all baselines, demonstrating the effectiveness of our method for instruction- based zero-shot molecule tasks. It is worth noting that some of the baselines also achieve results on certain tasks. For example, KVPLM works on hiv, muv and bbbp, and MoMu works on tox21 and bace, showing our instruction-based molecule zero-shot learning method is a general framework to probe knowledge in molecule-text models.
Comparing our zero-shot performance the supervised results, we observe that GIMLET achieves performance close to those of GNNs on several datasets, like bace, muv, toxcast, and bbbp. This demonstrates that GIMLET is able to solve molecule tasks in the zero-shot fashion nicely.
The results for large-scale molecule tasks are presented in Table 2 struggle to handle these tasks. Our GIMLET not only successfully transfers to the Chembl Kero-Shot splits, where both the tasks and graphs were unseen during pretraining, but also demonstrates strong generalization performance on the PCBA benchmark.
As depicted, the baselines
The results of regression tasks are shown in Table 3 The scatter plots of the regression are in Appendix. It is worth noting that regression tasks pose greater challenges in the zero-shot setting than classification tasks, because it is difficult to determine unseen physico-chemical properties using only natural language, and the output space is also vast. Notably, the zero-shot baselines fail to perform the regression tasks due to their inability to output correctly formatted numbers. In contrast, GIMLET generates correctly formatted numbers for over 98% regression testing samples in all the tasks and showcases the potential of zero-shot regression tasks.
4.2 Instructions-Basee Few-Shot Finetuning
We apply few-shot instruction- basedd tuning on the downstream tasks, to examine whether GIMLET exhibits improved performance in the presence of low-resource data. Notably, for datasets with more than one task, we do few-shot learning for every single task individually. We first split datasets into training, validation, and testing sets in the same as the zero-shot setting. Then K samples for each class are randomly sampled from the training set as the few-shot examples, where K the sho-shot number. We report the result of the best validation model on the testing set. The input is the
7

---

same as the zero-shot setting, including molecule data and instructions. We only tune the last linear layer, to inspect whether the feature is discriminative and linear separable. Linear tuning is also low resource costing and avoids overfitring. The last linear mapping to vocabulary is tuned for GIMLFT and KVPLM. For MoMu, we tune the last linear layer of the projection head for features.
The result is shown in Figure and plots for each dataset are in Appendix, GIMLET outperforms other baselines consistently, exhibiting improved performance with an increasing number of few-shot samples. The performance of GIMLET also approaches the supervised GIN with only limited samples. Baselines also achieve some performance gain in the few-shot training but remain beaten by GIMLET, and the improvements are not stable. fluctuating with the few-shot number. The few-shot results demonstrate the high-quality representation learned by GIMLET, which is well-suited for specific tasks, highlighting the potential of GIMLET in scenarios beyond zero-shot learning.
Average bio
Average WX
Average phu
Average pty
16C-000
354
Eifc Stanber
Figure 3: Few shot performance. Higher is better for bio, tox, and pha, and lower is better for phy.
4.3 Ablation and Exploration Studies
Effect of Model Design We investigate components of GIMLET to highlight their benefits. Specifically, we focus two aspects: (a) To measure the effectiveness of the unified transformer method, we compare it to the variant version which obtains graph embedding by an individual GIN as a token in text, a common method in multimodal transformers. (b) We ablate graph decoupling encoding by complete global attention. We conduct pretraining and downstream zero-shot testing in the same setting as method.
Table 4: Ablation study on GIMLET module.
Method
bace
hiv
muv
Avg. bio
tox21
toxcast
Avg. tox
bbbp
cyp450
Avg. phu
W.o. unifying
0,4319
0.6133
0.6067
0.5506
0.5922
0.5537
0.5730
0.5309
0.6206
0.5758
w.o. decoupling
0.6458
0.6406
0.5421
0.6095
0.6306
0.5954
0.6130
0.5666
0.6320
0.5993
GIMLET
0.6957
0.6624
0.6439
0.6673
0.6119
0.5904
0.6011
0.5939
0.7125
0.6532
The comparison is shown in Table 4 Compared to GIMLET, w.o. unifying perform worse on all the datasets, especially on Bio-activity and Pharmacokinetic Tasks. Next, the w.o. decoupling variation performs worse than GIMLET most datasets. The decoupling significantly improves performance on Bio-activity and Pharmacokinetic tasks and is also comparable on Toxicity tasks. This proves our claims that the unified graph-text transformer not only avoids additional modules but also has strong capacity for encoding graph data and generalizing across tasks.
Effect of Pretraining Our pretraining includes two large types of pretraining tasks, including Chembl bioactivity tasks and Chembl Property physico-chemical tasks. To validate the influence of cach task type, we performed ablation experiments where each of them are shown in Table and the detailed result is in Appendix. Unsurprisingly, Chembl is essential for downstream molecule assay tasks, and Chembl property plays a crucial role in Physico-chemice tasks. However, the results also reveal that Chembl positively affects downstream Physico-cheme tasks, and Chembl property benefits Bio-activity tasks, Toxicity tasks, and Pharmacokinetic tasks. The results demonstrate the positive transfer of pretraining tasks on a diverse range of downstream tasks, spanning various types and domains.
excluded separately. The results
Robustness to Instruction We explore whether GIMLET robust to the instructions. We rephrase our instructions by GPT-3.5-turbo for testing, Each instruction is rephrased using four types of requests:
8

---

Table 5: Ablation study on GIMLET pretraining.
Mcthod
Avg. bio
Avg. pha
Avg. tox 1
Avg. phy 4
bioactivity assay only phy physico-chemicll only
0.6402
0.6071
0.5676
0.4894
0.5454
0.4748
2.6178
both
0.6673
0.6532
0.6011
2.5266
rewriting, detailing, expanding, and shortening. The prompts and examples are provided in Appendix. We plot the performance for each type of augmentation, as well as the average standard variation for each task in Figure As shown, GIMLET is more robust than baselines on most tasks. This shows that our instruction-bant -basd pretaining enables GIMLET to focus on the task rather than the specific language form.
bace
hl
muv
6.7
eiperd 11g1
eiperdd
7591
eupord
305al
3ocal
ROC-aU
0.5
50
0.50
0.55
0.5
KVTUU
KVTUM
KVTUM
Vole
Model
Model
Model
1ox21
toxcast
bbbp
cyp450
07
esoale ssharen cead
wshareen
ROCAAUI
0.5
05
0.
0.4
KVP.U
Figure 4: Robustness to instruction.
Instruction Interpretability We ablate the explanation of the instructions in downstream tasks, to validate whether the explanation in instructions helps GIMLET perform downstream tasks. Without explanation, only the task name and question are provided to the model. The examples of ablated instructions are available in Appendix. The results presented in Table T demonstrate a significant drop in performance when task explanations are not included. This finding supports the effectiveness of the explanation and highlights the model's ability comprehend the explanation of tasks.
Table 6: Ablation study on GIMLET instructions.
Method
bace
hiv
muv
Avg. bio
tox21
toxcast
Avg. tox
bbbp
cyp450
Avg. pha
name only
0.5416
0.6132
0.6441
0.5996
0.5809
0.5279
0.5544
0.4871
0.6669
0.5770
explanation
0.6957
0.6624
0.6439
0.6673
0.6119
0.5904
0.6011
0.5939
0.7125
0.6532
Conclusion and Discussion
In this work, we propose nature language instruction-based graph zero-shot learning for molecule tasks, and construct a molecule dataset consisting of two thousand tasks with instructions derived from task descriptions. We propose GIMLET, which extends large language models handle graph and text data by applying the transformer mechanism with generalized position embedding and decoupled attention. Instruction-based pretraining applied for GIMLET. Experiments demonstrate promising results in zero-shot learning, exhibit strong robustness to the instruction, and can be further improved by few-shot tuning. We do not consider the tasks with structured output like molecule generation in this study, which is left for further work.

---

Ethical Consideration
The work presented here is centered on paradigm shift in molecule property prediction tasks, transitioning from traditional supervised learning to instruction-bars zero-shot learning. Due to the fact that molecule property prediction has been widely explored prior research, the direct societal impacts may appear limited. However, indirect negative impacts could be caused by excessive reliance on algorithms. We contend that a combination of model predictions and experimental validation essential for practical implementation.
References
[1] Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, L,, Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., al. (2022). Flamingo: visual language model for few-shot learning. Advances Neural Information Processing Systems, 35:23716-23736.
[2] Bachlechner, T., Majumder, B. P., Mao, H., Cottrell, G., and McAuley, (2021). Rezero is all you need: Fast convergence at large depth. Uncertainty in Artificial Intelligence, pages 1352-1361. PMLR.
[3] Bagal, V., Aggarwal, R., Vinod, P., and Priyakumar, D. (2021). Molgpt: molecular generation using a transformer-devoder model. Journal of Chemical Information and Modeling, 62(9):2064-2076.
[4] Bao, H., Dong. I.., Piao, S., and Wei, F. (2021). Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.082.
[S] Bastos, A., Nadgeri, A., Singh, K., Kanezashi, H., Suzumura, T., and Mulang, O. (2022). Investigating expressiveness of transformer in spectral domain for graphs. arXiv preprint urXiv:2201.09332.
[6] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
[7] Casasnovas, R., Ortega-Castro, J., Frau, J., Donoso, J., and Munoz, (2014). Theoretical pka calculations with continuum model solvents, alternative protocols to thermodynamic cycles. International Journal of Quantum Chemistry, 114(20):335) 1350-103330-3366 1363-
[8] Chen, D., O' O'Bray, L.,, and Borgwardt, K. (2022a). Structure-aware transformer for graph representation learning. In International Conference on Machine Learning, pages 3469-34690033 -489,, PMLR.
[9] Chen, X., Wang. X., Changpinyo. S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al. (2022h). Pali: A jointly-scaled multilingual language-image model. arXiv preprint 20Xiv:220.006794
[10] Chithrananda, S. Grand, G., and Ramsundar, B. (2020). Chemberta: Large-scale self supervised pretraining for molecular property prediction. arXiv preprint 200.0:2010.09885.
[11] Choromanski, K., Lin, H., Chen, H., Zhang, T., Schanobish, A., Likhosherstov, V., Parker-Holder, J., Sarlos, T., Weller, A., and Weingarten, T. (2022). From block-tocplitz matrices to differential equations on graphs: towards general theory for scalable masked transformers. In International Conference Machine Learning, pages 3962-3983. PMLR.
[12] Chung, H. W., Hou, I.,, Longpre, Zoph, B., Tay, Y., Fedus, W., Li, E.,, Wang, X., Dehghani, M., Brahma, S., et al. (2022). Scaling wewtrucctoo-iinnttne language models. arXiv preprini arXiv:2210. 11416..
[13] Devlin, J., Chang, M.W.,, Lee, K., and Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers language understanding. Proceedings of the 2019 Conference of North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186
[14] Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bombarell, R., Hirzel, T., Aspuru-Guzik, A., and Adams, R. P. (2015). Convolutional networks on graphs for learning molecular fingerprints. Advances in neural information processing systems, 28.
[15] Dwivedi, V. P. and Bresson, X. (2020). A generalization of transformer networks to graphs. arXiv preprint arXiv:2012.09699
[16] Edwards, C., Lai, T., Ros, K. Honke, G., and Ji, H. (2022). Translation between molecules and natural language. arXiv preprint 1188772.
10

---

[17] Edwards, C., Zhai, C., and Ji, H. (2021). Tex22ol: Cross-modal molecule retrieval with natural language queries. In Proceedings the 2021 Conference Empirical Methods in Natural Language Processing. pages 595-607.
[18] Efrat, A. and Levy, O. (2020). The turking test: Can language models understand instructions? arXiv preprint arXiv:0010.11882.
[19] Fang, Y., Zhang, Q. Yang, H., Zhuang, X., Deng, S., Zhang, W., Qin, M., Chen, Z., Fan, X., and Chen, H. (2022). Molecular contrastive learning with chemical element knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 3968-3976.
[20] Gaulton, A., Bellis, L.. J., Bento, A. P., Chambers, J., Davies, M., Hersey, A., Light, Y., McGlinchey, S., Michalovich, D., Al-Lazikani, B., et al. (2012). Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, (O(DI):DII 100-D1107
[21] Gilmer, J., Schoenholz, S. S., Riley, P. F., Vinyals, O., and Dahl, G. E. (2017). Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR.
[22] Gosselet, F., Loiola, R. A., Roig, A., Rosell, A., and Culot, M. (2021). Central nervous system delivery of molecules across the blood-brain barrier. Neurochemistry International, 144:104952.
[23] Guo. L., Zhang, Q. and Chen, H. (2022). Unleashing the power of transformer for graphs. arXiv preprint wwXYv:2022.15881.
[24] Hassani, K. and Khasahmadi, A. H. (2020). Contrastive multi-view representation learning graphs. In International conference on machine learning, pages 41 16-4126. PMLR.
[25] Honda, S., Shi, S., and Ueda, H. R. (2019). Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery. arXiv preprint 1rXiv:1911.04735.
[26] Hu, W., Liu, B., Gomes, J., Zitnik, M., Liang, P., Pande, V., and Leskovec, J. (2019). Strategies for pre-training graph neural networks. arXiv preprint 190s: 12265.
[27] Irwin, R., Dimitriadis, S., He, J., and Bjerrum, E. J. (2022). Chemforme: pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 31):0055222.
[28] Jin, W., Barzilay, R., and Jaakkola, T. (2020). Hierarchical generation of molecular graphs using structural motifs. International conference on machine learning, pages 4839 4848. PMLR.
[29] Kim, J., Nguyen, T. D., Min, S., Cho, S., Lee, M., Lee, H., and Hong. S. (2022). Pure transformers powerful graph learners. arXiv preprint arXiv:2207.02505.
[30] Kipf, T. N. and Welling, M. (2016). Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609,02900 02907.
[31] Kreuzer, D., Beaini, D., Hamilton, W., Létournaaa,, V., and Tossou, (2021). Rethinking graph transformers with spectral attention. Advances Neural Information Processing Systems, 34:21618-21629,
[32] Li, J., Li, D., Savarese, S., and Hoi, (2023). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, arXiv preprint XrXiv:2301. 12557
[33] X., Xu, Y., Lai, L., and Pei, J. (2018). Prediction of human cytochrome p450 inhibition using a multitask deep autoencoder neural network. Molecular Pharmaceutics, 15(10):4336-44 4345.
[34] Liu, L.., Liu, X., Gao, J., Chen, W., Han, J. (2020). Understanding the difficulty training transformers. In 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, pages 5747-$763. Association for Computational Linguisties (ACL).
[35] Liu, Q. Allamanis, M., Brockschmidt, M., and Gaunt, A. (2018). Constrained graph variational autoen- coders for molecule design. Advances in neural information processing systems, 31.
[36] Liu, Q. Nickel, M., and Kiela, D. (2019a). Hyperbolic graph neural networks. Advances in neural information processing systems, 32.
[37] Liu, S., Demirel, M. F., and Liang, Y. (2019b). N-gram graph: Simple unsupervised representation for graphs, with applications to molecules. Advonces in neural information processing systems, 32.
[38] Liu, S., Guo, H., and Tang. J. (2023). Molecular geometry pretraining with SB(3)-invariant invariant denoising distance matching. In The Eleventh International Conference Learning Representations.
11

---

[39] Liu, S., Nie, W., Wang, C., Lu, J.. Qiao, Z.. Liu, L... Tang, J., Xiao, C., and Anandkumar, A. (2022). Multi- modal molecule structure-fext model for text-based retrieval and editing. arXiv preprint arXiv:2212.10783
[40] Liu, S., Wang, H., Liu, W., Lasenby, J.. Guo, H., and Tang, J. (2021), Pre-training molecular graph representation with geometry. In International Conference on Learning Representations.
[41] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.., and Stoyanov, V. (2019c). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1662.
[42] Maziarka, L., Danel, T., Mucha, S., Rataj, K., Tabor, J., and Jastrzkebski. (2020). Molecule attention transformer. arXiv preprint 202.0:2002.08264.
[43] Mialon, G., Chen, D., Sclosse, M., and Mairal, J. (2021). Graphit: Encoding graph structure in transformers. arXiv preprint arXiv:2106.0560
[44] Mishra, S., Khashabi, D., Baral, C., Choi, Y., and Hajishirzi, H. (2021). Reframing instructional prompts to gptk's 's language. arXiv preprint 2rXiv:2009.07800.
[45] Mishra, S., Khashabi, D., Baral, C., and Hajishirzi, H. (2022). Cross-task generalization via natural language crowdsoureing instructions. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages
[46] Nielsen, M. H., Pedersen, F. S., and Kjems, (2005). Molecular strategies to inhibit hiv-1 replication. Removimoooy, 2:1-20.
[47] Ouyang, L.., Wu, J. Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2777 277444.
[48] Park, W., Chang, W.-G., Lee, D., Kim, J., et al. (2022). Grpe: Relative positional encoding for graph transformer. In ICLR2022 Machine Learning for Drug Discovery.
[49] Parmar, M., Mishra, S., Purohit, M., Luo, M., Mohammad, M., and Baral, C. (2022). In-boxbart: Get instructions into biomedical multi- task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 112 128.
[50] Raffel, C., Shazeer, N., Roherts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-553
[51] Ramsundar, B., Eastman, P., Walters, P., and Pande, V. (2019). Deep learning for the life sciences: applying deep learning to genomics, microscopy, drug discovery, and more. O' O'Reilly Media.
[52] Rong, Y., Bian, Y., Xu, T. Xie, W., Wei, Y., Huang, W., and Huang. J. (2020). Self-supervised graph transformer on large-scale molecular data, Advances in Neural Information Processing Systems, 33: 12559 12571.
[53] Ross, J., Belgodere, B., Chenthamarkkshan V., Padhi, L, Mroueh, Y., and Das, P. (2022). Molformer: Large scale chemical language representations capture molecular structure and properties.
[54] Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafcai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., al. (2021). Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207.
[55] Schick, T. and Schütze, H. (2021). Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255-269.
[56] Seidl, P., Vall, A., Hochreiter, S., and Klambauer, G. (2023). Enhancing activity prediction models in drug discovery with the ability to understand human language. arXiv preprint vxXiv:233.03366
[57] Shaw, P., Uszkoreit, J., and Vaswani, A. (2018). Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages 464-468.
[58] Shen, W. X., Zeng. X.. Zhu, F. Wang. Y. L.. Qin, C., Tan, Y., Jiang, Y. Y., and Chen, Y. Z. (2021). oft-oofthe-boo deep learning prediction of pharmaceutical properties by broadly learned knowledge based molecular representations. Nature Machine Intelligence, 3(4):334 343.
based
12

---

[59] Su. B., Du, D., Yang. Z., Zhou, Y., Li, J., Rao, A., Sun, H., Lu, Z., and Wen, J.-R. (2022). A molec- ular multimodal foundation model associating molecule graphs with natural language. arXiv preprint arXiv:2209.05463 254.0488.
[60] Sun, F.Y., Hoffman, J., Verma, V., and Tang. J. (2020), Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations.
[61] Sun, M., Xing, J., Wang, H., Chen, B., and Zhou, J. (2021). Moel: Data-driven molecular fingerprint via knowledge-aware contrastive learning from molecular graph. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 3585 3394.
[62] Sun, R., Dai, H., and Yu, A. W. (2022). Does gnn pretraining help molecular representation? Advances in Neural Information Processing Systems, 15:1699--
[63] Suresh, S., Li, P., Hao, C., and Neville, J. (2021). Adversarial graph augmentation to improve graph contrastive learning, Advances Neural Information Processing Systems, 34:15920-15933.
[64] Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia. E., Poulton, A., Kerkez, V., and Stojnic, R. (2022). Galactica: A large language model for science. arXiv preprint ar W.Ni::211110000
[65] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, t.., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
[66] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., and Bengio, Y. (2017). Graph attention networks. arXiv preprint vrXiv:1710.10900
[67] Velickovic, P., Fedus, W., Hamilton, W. L., Liò, P., Bengio, Y., and Hjelm, R. D. (2019). Deep graph infomax. ICLR (Poster), 2(3):4
[68] Wang, S., Guo, Y., Wang, Y., Sun, H., and Huang, J. (2019). Smiles-bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pages 429-436.
[69] Wang, Y., Magar, R., Liang, C., and Barati Farimani, A. (2022a). Improving molecular contrastive learning via faulty negative mitigation and decomposed fragment contrast. Journal of Chemical Information and Modeling, 62(11):22 (2111)2273222
[70] Wang. Y., Min, Y. Shao, E., and Wu, J. (2021).. Molecular graph contrastive learning with parameterized explainable augmentations. In 2021 IEEE International Conference Bioinformatics and Biomedicine (BIBM), pages 1558-1563. IEFE.
[71] Wang, Y., Mishra. S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran. A. S., Arunkumar, A., Stap, D., et al. (2022h). Supper-nauralinsttuutioss: Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 5085-5109.
[72] Wang, Y., Xiao, J., Suzek, T. O., Zhang, J., Wang, J., Zhou, Z., Han, L.,, Karapetyan, K., Dracheva, S., Shoemaker, B. A., et al. (2012). Pubchem's 's bioassay database. Nucleic acids research, 40(D1))D400-D41
[73] Withnall. M.. Chen, H.. and Tetko. I. V. (2018). Matched molecular pair analysis on large melting point datasets: big data perspective. ChemMedChem, 13(6):599-60) -606.
[74] Wu, Z., Jain, P., Wright, M., Mirhoseini, A., Gonzalez, J. E.,, and Stoica, I. (2021). Representing long-range context for graph neural networks with global attention. Advances in Neural Information Processing Systems, 34:13266-13279.
[75] Wu, Z., Ramsundar, B., Feinberg, E. N.. Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., and Pande, (2018). 8). Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):113-330
[76] Xia, J., Wu, I.., Chen, J., Hu, B., and Li, (2022a). Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022, pages 1070-1079.
[77] Xia, J.. Zhao, C., Hu, B., Gao, Z. Tan, C., Liu, Y., Li, S., and Li, S. Z. (2022b). Mole-bert: Rethinking pre-training graph neural networks for molecules.
[78] Xu, D., Cheng, W. Luo, D., Chen, H., and Zhang, X. (2021). Infogel: Information aware graph contrastive learning. Advances in Neural Information Processing Systems, 34:30414-30425.
13

---

[79] Xu. K., Hu, W., Leskovec, J., and Jegelka, S. (2018). How powerful are graph neural networks? arXiv preprint arXiv:1810000826
[80] Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. (2021). Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 34:28877- 28888.
[81] You, Liu, B., Ying, Z., Pande, V., and Leskovec, (2018). Graph convolutional policy network for goal-directed molecular graph generation. Advances neural information processing systems, 31.
[82] You, Y., Chen, T., Shen, Y., and Wang, Z. (2021). Graph contrastive learning automated. In International Conference on Machine Learning, pages 12121-12132. PMLR.
[83] You, Y., Chen, T., Sui, Y., Chen, T., Wang, Z., and Shen, Y. (2020). Graph contrastive learning with augmentations. Advances neural information processing systems, 33:5812-568 5823.
[84] Zaidi, S., Schaarschmidt, M., Martens, J., Kim, H. Teh, Y. W., Sanzhez-Gonzalez, A., Battaglia, P.,
Pascanu, R., Godwin, (2022). Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:220.000133.
[85] Zeng. Z., Yao, Y., Liu, Z., and Sun, (2022). deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(1):862.
[86] Zhang. J., Zhang, H., Xia, C., Sun, (2020). Graph-bert: Only attention is needed learning graph representations. arXiv preprint arXiv:2001.051400
[87] Zhang, Z., Liu, Q.. Wang, H., Lu, C., and Lee. C.-K. (2021). Motif-based graph self-supervised learning for molecular property prediction. Advances in Newral Information Processing Systems, 34:15870-15882
[88] Zhao, H., Ma, S., Zhang, D., Deng, 7.-H., and Wei, (2022). Are more layers beneficial graph transformers? In The Eleventh International Conference on Learning Representations.
[89] Zhong, R., Lee, K., Zhang, Z., and Klein, D. (2021). Adapting language models zero-shot learning by meta-tuning on dataset and prompt collections. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2856-2878.
14

---

A Framework
A.1 Details of Datasets and Instructions
Table 7: Data Overview
Splitting
Duta Class
Dataset
No. of Molecules
No. of Tasks
Task Metric
Task
Pretraining
Physko-chemical
Bioncrivity assay
ChEMBL bicassay acrivity dataser
CHEMBL Propeaty
365065
1048
13
RMSE
Classification
Regression
Large Scale
PCBA Punt HTS bioAssy
417929
128
ROC-AUC
Clessifistion
Clissilicatio
Pharmacokinetic
BBBP Blood barrier penetration
CYP Inhibiino
16896
2039
1
ROC ALC
ROC ALC
Classificatinn
Classifiattin
MUV PubChemm bioAsay
93057
17
ROC ALC
Classificatio
Bio-activity
benchmark
Classification
Downstrcam Zero-Shol
HIV replication inhihirion
41127
ROC ALC
Classitcation
Toxicity
Toxcast
Tox21 Tox21 Toxicology in the 21st century
7831
12
ROC ALC
Chassflcation
Classification
ESOL Water solabiiity
1123
RESE
Regression
Physko-chemical
FraeSals Selvation free energy
642
RMSE
Regression
Lipo Lipophilicity
4200
RMSE
Regression
The datasets used in our study are presented in Table These datasets consist of different types of tasks related to molecule property prediction. It should be noted that during the pretraining phase, the loss function is not specific to the task types, but rather encompasses the generative loss of the language model.
We have chosen not to include certain datasets, namely SIDER and ClinTox, in our collection of datasets. The decision was based on the fact that the tasks associated with these datasets are not clearly defined and involve complex systemic phenomena, making it challenging to describe them through instructional texts. For instance, the ClinTox dataset involves determining whether drugs have passed the FDA approval, which is not an objective problem but rather a dynamic and intricate social phenomenon. The SIDER dataset focuses on describing the side effects of drugs on system organ classes, which have intricate mechanisms and wide range of possible causes, making them difficult to be effectively conveyed through instructions.
For the Chembl property dataset that we have constructed, detailed information can be found in Table These properties are sourced from the Chembl database (20) through the web API.
Table 8: Chembl property tasks and labels
Property
Label type
Aromatic rings number
Integer
cx ex_logd distribution coefficient
Real
cx_ ex_logp partition coefficient
Real
cx_ xx_most_aphe log dissociation constant
Real
Molecular masses
Real
Hydrogen bond donor number
Integer
Heavy atom number
Integer
Lipinski's s rule of five violation number Polar surface area (PSA)
Integer
Real
Quantitative Estimate Druglikeness (QED)
Real
Rule of three passes Rotatable bond number
Bool
Integer
The task explanation is primarily sourced from relevant papers, websites, or databases that introduce and compile the respective datasets. The specific sources utilized depend the particular datasets under consideration. For Chembl tasks, we obtain task descriptions from the Chembl website. Descriptions for MoleculeNet tasks and PCBA are primarily sourced from the PubChem website. Certain datasets, such as Toxcast, include task descriptions within the dataset files. In the case of other tasks, like Chembl property and Physical-Chemine Chemiiall tasks, instructions are derived from Wiki or other papers. We list the instruction source in Table 9
The description covers a wide range of aspects, including the family, function, and mechanism of the assay target, the assay experiment setting, the approximation method used for determining the property, and others. We describe regression tasks by introducing the relasionship between the
15

---

Table 9: Data sources and classes for different stages of the model
Dataset
Instruction Source
ChEMBL Zero-Shot bioassay activity dataset
Chembl Database
CHEMBL Property
Wiki
PCBA PubChem HTS bioAssay
Pubchem Database
ChEMBL Zero-Shot bioassay activity dataset
Chembl Database
CYP PubChem BioAssay CYP IA2, 2C9, 2C19, 2D6, 3A4 inhibition
Pubchem Database
BBBP Blood-brain barrier penetration
Paper 22
MUV PubChem bioAssay
Pubchem Database
BACE-1 benchmark set
Pubchem Database
HIV replication inhibition
Paper [46
Tox21 Toxicology in the 21st century
Pubchem Databasc
Toxcast
Toxcast file
ESOL Water solubility
Paper 73
FreeSolv Solvation free energy
Paper
7
Lipo Lipophilicity
Wiki
task property and other properties, i.e. how to estimate these properties by other ones. However, this method is still challenging due to the model's capacity to understand complex mathematical relationships.
The instructions for each task are generated automatically by conducting searches on the databases and summarizing the descriptions. We use a mixture strategy of summarizing, combining template-bass summarizing and
summarizing methods. The GPT-3.5-turbo-bar summariz- ing method is applied by the prompt 'Summarize the assay: 'n [Deccriptions to be summarized)".
The resulting instructions then concatenated with relevant questions. These instructions subsequently reviewed and validated by professional biology Ph.D. student and slightly modified necessary.
We then list the instructions of each dataset. For datasets with more than one task, we only list the instruction of one task as an illustration.
Chembl
"The assay is TUBCHEM_BIOASSAN qHTS Assay for Activators of Human Muscle isoform 2 Pyruvate Kinase. (Class of assay: confirmatory)
and it is Direct single protein target assigned The assay has properties: assay category is confirmatory ; assay organism is Homo sapiens assay type description is Functional Is the molecule effective to this assay?"
Chembl property
"The partition coefficient, abbreviated P, is defined as particular ratio of the concentrations two solvents (a biphase of liquid phases), specifically for un un-ionized solutes, and the 1ogarithm of the ratio is thus Log P. When one of the solvents is water and the other is non non-polar solvent, then the log P value is a measure of lipophilicity
a solute between the
hydrophobicity. The defined precedent is for the lipophilic and hydrophilic phase types to always be in the numerator and denominator respectively. What is the logarithm of the partition coefficient of this molecule?"
PCBA
"The assay tests the inhibition of ALDH1A1 activity using propionaldehyde as an electron donor NAD+ as an electron acceptor. The conversion of NAD+ to NADH is measured via an
16

---

increase in fluorescence intensity to determine enzyme activity. ALDH1A1 plays critical roles the metabolic activation of retinoic acid and may be target for inhibitor development in metabolic diseases. the molecule effective to this assay?"
CYP450
"Find molecules that can effectively inhibit Cytochrome P450 (CYP450) enzymes, particularly CYP1A2, to help reduce the risk of adverse drug events and drug-drug interactions caused by CYP450 mediated metabolic pathways. Consider the various CYP450 inhibition mechanisms such weakening enzyme activity, while keeping in mind the potential for increased side effects due to elevated blood drug concentrations. Is this molecule effective to this assay?"
occupying active sites
BBBP
"In general, molecules that passively diffuse across the brain blood barrier have the molecular weight less than 500, with LogP of 2-4, and no more than five hydrogen bond donors or acceptors. Does the molecule adhere the three rules not?"
MUV
"Protein kinase A (PKA) is ubiquitous serine/threonine protein kinase and belongs to the AGC kinase family. It has several functions in the cell, including regulation of immune response, transcription, cell cycle and apoptosis. PKA is a cAMP dependent enzyme that exists in its native inactive form as a 4 subunit enzyme with two regulatory and two catalytic subunits. Binding of CAMP to the regulatory subunit leads to the disassembly of the complex and release of now active catalytic subunits. Is this molecule inhibitor of PKA?"
BACE
"BACE1 an aspartic protease important the pathogenesis of Alzheimer's 's disease, and in the formation of myelin sheaths. BАCE1 is a member of family of aspartic proteases. Same as other aspartic proteases, BACE1 is a bilobal enzyme, each lobe contributing residue, with an extended active site cleft localized between the two lobes of the molecule. The assay tests whether the molecule can bind to the BACE1 protein. Is this molecule effective to the assay?"
catalytic Asp
HIV
"Human immunodeficiency viruses (HIV) are a type of retrovirus, which induces acquired immune deficiency syndrome (AIDs). Now there are six main classes antiretroviral drugs for treating AIDs patients approved by FDA, which are the nucleoside reverse transcriptase inhibitors (NRTIs), the non mu-clelesside reverse transcriptase inhibitors (NNRTIs), the protease inhibitors, the integrase inhibitor, the fusion inhibitor, and the chemokine receptor CCRS antagonist. Is this molecule effective to this assay?"
Tox21
17

---

"Estrogen receptor alpha (ER aplha) is Nuclear hormone receptor. The steroid hormones and their receptors are involved in the regulation of eukaryotic gene expression and affect cellular proliferation and differentiiaion in target tissues. Ligand-dependenn nuclear transactivation involves either direct homodimer binding to palindromic estrogen response element (ERE) sequence DNA DNA-binding transcription factors, such as AP-1/c-Jun, c-Pos, - Sp1 and Sp3, to mediate ERE-independent signaling. Is this molecule effective to this assay?"
association with other
Toxcast
"APR_ HepR_ Bee___op@__Celllycllle _ellCCyleArrestt....... _2hrr, is one of 10 assay component (s) measured or calculated from the APR_HepG2_24hr assay. It is designed to make measurements of cell phenotype, form of morphology reporter, intensity signals by HCS Fluorescent Imaging technology Data from the assay component PBL_lepd2_GellCyclAkrrr analyzed into 2 assay endpoints. AnThis assay endpoint,
detected with fluorescence
24hr
(o__Heeo2_Cel1CyceeMMree _errest_24h_dn, was analyzed in the negative fitting direction relative to DMSO as the negative control and baseline of activity. AnUsing type measures of all nuclear dna for lfss-of- activity can be the pathway-leve level as they
morphology reporter,
used to understand the signaling relate to the gene \nFurthermore: this assay endpoint can be referred to as a primary readout, because this assay has produced multiple assay endpoints where this one serves a signaling function. \nTo generalize other relatable targets, this assay endpoint is annotated to the \"cell cycle\" intended target family, where the subfamily is \"proliferationy Is this molecule effective to this assay?"
intended target to
ESOL
"Solubility (logS) can be approximated by negative LogP -0.01 (MPt \u2013 25) + 0.5 Can you approximate the logs of this molecule by its negative logp and MPt?"
FreeSolv
"The free energy of hydration can be approximated by hyd394G_hyd \u0394G_solv, soln \u0394G_solv gas + RT In (10- (-pKa)). Can you tell me the free energy of hydration (by using the negative pka) of this molecule, predicted by using soll and negative pka?"
Lipo
"Lipophilicity is an important feature of drug molecules that affects both membrane permeability and solubility, measured by octanol/vater distribution coefficient (logD at pll 7.4). What 's the octanol/vated distribution coefficient (logD at pH 7.4) of this molecule?"
A.2 Details of Framework Application
In our framework, we represent the labels of various tasks as strings. For assay tasks involving classification, the labels are converted to either "Yes" or "No" based whether the molecule has
18

---

an effect on the assay. In regression tasks, the labels are transformed into numerical strings. Integer values remain unchanged, while decimal numbers are rounded to two decimal places.
To conduct zero-shot testing on our model, we generate output sequences and extract the answer from the results. For assay classification, we consider the first token generated as the answer and use the scores for the 'Yes" and 'No" tokens to compute the ROC-AUC score for classification. In regression tasks, we extract the number from the generated sequence by performing string matching using regular expression template: generates results in the correct format for all classification tasks and accurately formatted numbers for over 98% of regression testing samples, without any augmentation of restriction in the vocabulary.
" Notably, we discovered that GIMLET consistently
A.3 Baselines Evaluation
For the baselines, we apply our instruction-based molecule zero-shot learning to their respective settings. KVPLM employs SMILES for molecule representation and utilizes masked language modeling for molecule-text data. Galactica also represents molecules using SMILES but generates the next sentence in an autoregressive manner. MoMu employs contrastive learning between the GNN-encoded molecule and the corresponding text, allowing to score each candidate sentence for the target molecule and retrieve the best matching one. Our application of each baseline model aligns with their intended use.
It is important to note that for the baseline models, to avoid baselines generating answers in classi- fication not in our parsing method ('Yes' and 'No'), we limit the vocabulary during generation to only include "Yes' and 'No' in classification tasks. This restriction is achieved by utilizing the bias term huggingface to prevent the generation of other words. However, it is worth mentioning that our model, GIMLET, does not require this augmentation and is able to generate the desired outputs without any additional constraints.
For KVPLM, we mask the answer position the whole sentence for the model to predict. For example, for molecule ARE inhibitor, input to KVPLM is:
and classification tasks
@CCciccccc1--nimmnn Oxidative stress has been implicated in the pathogenesis of a variety diseases ranging from cancer to neurodegeneration. The antioxidant response element (ARE) signaling pathway is important in the amelioration molecule agonists of antioxidant response element (ARE) signaling pathway? [MASK]"
(=0) (=O) MCe1ccco))
oxidative stress. Is this
For Galactica, the answer is expected to be generated after reading the question. The input example is
"[START_I_SMILE" CCOc1ccccc1 minnnciSCC (=0) NC (=0) NCciccco1 [END_I_SMILES
Question: Oxidative stress has been implicated in the pathogenesis of a variety of diseases ranging from cancer to neurodegeneration. The antioxidant response element (ARE) signaling pathway is important in the amelioration of oxidative stress. Is this molecule agonists antioxidant response element (ARE) signaling pathway? Answer:"
For MoMu, we compute the matching score between the molecule graph and the instruction with each answer. In the example, the classification scores for 'Yes' and 'No" are computed by matching graph feature of molecule | Tinnc///CCC(OONNCOO))CCCCCC
with
Oxidative stress has been implicated variety of diseases ranging from cancer to neurodogeneration. The antioxidant response element (ARE) signaling pathway is important in the amelioration of oxidative stress. Is this molecule agonists of antioxidant response element (ARE) signaling pathway? Yes"
the pathogenesis of a
19

---

and
"'xidative stress has been implicated in the pathogenesis of a variety of diseases ranging from cancer neurodegeneration. The antioxidant response element (ARE) signaling pathway is oxidative stress. Is this
important in the amelioration molecule agonists of antioridant response element (ARE) signaling pathway? No"
B
Method
B.1
Discussion of Individual Encoding Module Method
The individual encoding module-based multimodal language model can be formalized as LLM(M(G). T). where M is the individual encoding module for graph data G. For example, the visual module applied to pre-encode the image data get the dense representation, then put into the language model as tokens embedding 4 9 32 Current works on molecule language models also use GNN to get the representation of molecules interact with the language models 17 59 [66..
This method be considered decomposition of the conditional probability P))
P = PM (M T)dz,
based on the assumption that the feature distributions P(zIG) should be modeled by modality-speciffe modules to introduce inductive bias, and be independent of text information to help with adaptation novel text data.
However, for the molecule-text model, individual pre-encoding modules present problems. First, graph learning relies structure information, but the dense vectors encoded by GNN have limited capacity to carry structure information, and language models don't 't have inductive bias toward graph structure. Furthermore, training the additional module is difficult due the increased layers, since deep transformers have vanishing gradients in early layers 34 2]. which a well-known problem of transformer. Lastly, the additional modules increase parameters and training costs.
Our method GIMLET not only overcome these issues, our approach GIMLET not only directly unifies the standard language model for graph and text without introducing additional graph encoder module, also remains the decoupled graph encoding for better generalization.
B.2 Model Theoretical Capacity
In this section, we analyze the theoretical capacity of our modeling method.
Theorem Assume for different input features and position embeddings, the transformer layers can output different output features. The transformer with distance- based relative position embedding has stronger capacity than the 1-WL test for the graph isomorphism problem.
Proof The 1-WL test is defined as the following iteration:
(t) (t) = hash(u;)
(i) = hash ()) (ViE N),
where XG is the label in WL test, hash is the hash function, No(i) is the neighbor of node i. The transformer with distance-basel relative position embedding can be considered as the following mapping:
20

---

xo(1) hash (1(do(i,)) (i) :jEN))
haahh((0,,E)) (i))
(8)
(() U (id x), : k - {i}})
can be seen that the iteration of the transformer with distance-based relative position embedding includes both the node and its neighbors Ni(i), marked by distance and 1, respectively, ensuring the capacity is at least as strong as 1-WL test. It further includes other nodes far away, along with their distance, which constitutes stronger capacity than 1-WL test. Figure [ are two example graphs that cannot be distinguished by 1-WL test, but can be distinguished by transformer with distance based relative position embedding.
Figure 5: Two example graphs that cannot be distinguished by 1-WL test, but can distinguished by transformer with distance-based relative position embedding.
Theorem Assume for different input features and position embeddings, the transformer layers can output different output features. GIMLET can distinguish inaph-inttruttin pairs if graphs can be distinguished by transformer with distance- based relative position embedding. or instructions are different.
Proof 2 Because GIMLET decomposes the attention from graph nodes text, the graph nodes can only attend to other graph nodes. Thus the encoding capacity of graph data the same as a single transformer with distance-based relative position embedding for graph data.
Along with the assumption of transformer layers, GIMLET is able to distinguish graph-instruction pairs if graphs can be distinguished by transformer with distance-based relative position embedding, or instructions are different.
B.3 Detalied Related Work
We present detailed related work here, due the space limitation of paper.
Molecule Representation learning In recent years, there has been a growing interest in developing molecular representation learning for downstream tasks like drug discovery and other applications. One approach that has received considerable attention is utilizing language modeling techniques to acquire molecular representations based Simplified Molecular Input Line Entry System (SMILES) strings [68] I0]. Although sequence-based representations have demonstrated success in some ap- plications, concerns have been raised about their capability to incorporate all pertinent substructure information. To address this limitation, some researchers have proposed the use of Graph Neural Net- works (GNNs) to model molecules as graphs 21 83 26). potentially providing a comprehensive and accurate representation of the molecular structure.
Existing GNNs follow the message-passing paradigm and suffer from problems like long-range dependency vanishing and over-smoothing. Recently, Graph Transformer (52 80 has been proposed to better encode structures of graphs. The Graph Transformer is inspired by the Transformer
21

---

architecture, which has shown remarkable performance in natural language processing 65 13 41
The Graph Transformer extends the Transformer architecture to the graph domain. allowing the model to capture the global structure and long-range dependencies of the graph 86 15
3
74
23 88
Molecule Pretraining To fully explore the inherent structural information of molecules on large scale and transfer useful information downstream tasks, significant efforts have been made to address the inadequacies in molecular pre-training. Supervised pretraining is commonly used for learning useful representations 226 80 62]. As for unsupervised pretraining, one approach involved
using an generative pre- -re-triinng strategy on molecular SMILES strings [68 25 10
3 53] and Graph
26 37 52 87], which was followed by recent works adopting the contrastive paradigm that aligns representation of augmented views of the same graph but keeping views from other graphs away
67
78 19
The pretraining methods mentioned focus on obtaining representations for supervised training. However, for natural language instruction-based zero-shot graph learning, it's necessary to incorporate natural language into the pretraining process. Several studies have explored molecule structure-text multimodal pretraining. One class of method is the SMILES based language model, including KVPLM (85) and MolTS [16), which use SMILES strings and text for joint representation and translation. Another work Galactica [64] explored the multi-task molecule task learning with instruction. Some other works acquire advanced representations for molecules by GNN, such as Text2Mol [7]. MoMu [59] MoleculeSTM [39], and CLAMP [56), trained by contrastive learning between molecule graph and text description for molecule retrieval and caption tasks. MoleculeSTM and CLAMP explored molecule editing and property prediction with instructions. However, none of these works address the zero-shot fashion on complex molecule tasks like property prediction, due to constraints imposed by the pretraining methodology that not addressing the cestuction-folloviiin ability, and their model capacity for representing molecule graphs.
Instruction-based zero-shot learning Instruction- zero-shot learning an innovative approach that leverages natural language instructions and definitions to enable neural models to solve variety of tasks [50] 6 [55]
By providing a -eaaalee prompt, this method enables casier and more efficient specification of the learning task by utilizing knowledge about the task without data. To enhance the model's ability to follow instructions, some researchers have employed instruction-bel bbassd pretraining techniques [54 71 12 47 which explicitly train language models to solve tasks with instructions. Besides natural language processing, instruction-basee zero-shot
18
89
44
45 49
learning is also studied in multimodal domains like images 4 9 32).
C Experiments
C.1 Experiment setting
Our model only utilizes the basic features 226 62 of molecule graphs, which do not include additional features like ring markers. Specifically, it utilizes the first two dimensions of node features and the first two dimensions of edge features processed by ogb.smiles2graph. Therefore, the effectiveness of GIMLET predominantly stems from its architectural design and pretraining rather than the graph features it incorporates.
Following the standard supervised setting in previous studies (26). we utilize the scaffold strategy [51 to partition datasets into three subsets: the training set, validation set, and testing set with a ratio of 0.8, 0. 0. The scaffold strategy is deterministic approach that involves sorting the data based on the scaffold, which represents the molecular structure, While this strategy aids in dataset partitioning. it can introduce a significant domain gap between the training and testing sets, thereby increasing the challenge of generalization.
For zero-shot, report the results the testing sets, ensuring the comparability of our results to previous works. For few-shot, we report the result of the best validation model on the testing set, the same as previous works and other supervised baselines (51).
Many datasets encompass multiple tasks. To evaluate these datasets, we conduct separate testing for each task, accompanied by their respective instructions. For datasets with multiple tasks, we report the average ROC-AUC score for each task, following the methodology established in previous works (26).
22

---

C.2 Detailed Zero-Shot Result
We list the full zero-shot result of GIMLET and baselines in Table I0
The standard deviation for supervised results are denoted after +, and the multi-task setting results of Galactica are denoted in parentheses with italic. We also include the instruction-bise based zero-shot result reported in recent baseline CLAMM [56] which is tested by their instruction, denoted by italics too. CLAMP is contrastive pretrained model with ensembled encoders for molecule and text. The parameter number for CLAMP's result is not clearly stated in their paper but should be larger than 10B as they use sT3 language model [50] XXL variant (1 (B) as one the ensembled language models.
and 12
Table 10: Zero shot performance over Bio-activity tasks
Method
# Parameter
Type
bace
hiv
Avg. bio 0.5806
KVPLM
110M
0.5126
0.6120
0.6172
MoMu
113M
Vero Shot
0.6656
0.5026
0.6051
0.5911
CLAMP
> 10B
0.6476
0.8067
GIMLET
64M
0.6957
0.6624
0.6439
0.6673
Galactica- 125M
125M
0.445T(0.561)
0.367I(0.702)
0.4986
0.4369
Multi Task
Galactica-1.38
1.3B
0.5648(0.576)
0.3385(0.724)
0.5715
0.4916
GCN
0.5M
0.736110.030
0.757+0.011
0.732:20:14
0.742
GAT
1.0M
0.697+0.004
0.729+0.018
0,666+0.022
0.697
GIN
1.8M
Supervised
0.70/+00005
0.753++00019
0.718+0.025
0.724
Graphormer
48M
0.7060-00015
0.7452100004
0.706110.027
0.7424
Graphormer-p
48M
0.8575=0.006
0.778810.012
0.7480+100000
0.7948
Table 11: Zero shot performance over Toxicity tasks
Method
Parameter
Type
tox21
toxcast
Avg.
KVPLM
110M
0.4917
0.5096
0.5007
MoMu
113M
Zero Shot
0.5757
0.5528
0.5498
CLAMP
10B
0.60588
0.5383
0.5721
GIMLET
64M
0.6119
0.5904
0.6011
Galactica- 125M
125M
0,4964(0.543)
0.5106(0,55) 518)
0,5035
Multi Task
Galactica- 1.3B
L.3
0 0,4946(0. 606)
0,5123(0,588) 589)
0,5035
GCN
0.5M
0.749=0.008
0,633+00.009
0.691
GAT
1.0M
0.754++0.000 0.005
0.646-+0.006
0.700
GIN
1.8M
Supervised
0,740 0.08 008
0.634++0.000
0.687
Graphormer
48M
0.7589:+0.000
0.6470=t0.000
0,7029
Graphormer-p
48M
0.7729110.006
0.6649110.006
0.7189
Table 12: Zero shot performance over Pharmacokinetic tasks
Method
# Parameter
Type
hbbp
cyp450
Avg. pha
KVPLM
110M
0.6020
0.5922
0.5971
MoMu
113M
Zero Shot
0.4981
0.5798
0.5390
CLAMP
> 10B
0.4788
GIMLET
64M
0.5939
0.7125
0.6532
Galactica- 125M
125M
(0352(0.333)
0.5369
0.5711
Multi Task
Galactica-1.33
1.3B
0.5394(0.604)
0.4686
0.5040
GCN
0.5M
0.649-+0.030
0.8041+-0.03
0.7266
GAT
1.0M
0.662+00.026
0.8281 +0000
0.7451
GIN
1.8M
Supervised
0.65810.005
0.820510.012
0.7392
Graphormer
48M
0.7015=0.013
0.8436+00.00
0.7725
Graphormer-p 48M
0.706
0.8877:100004
0.8020
The result in parentheses represents the outcome of the multitask setting, also referred to as weakly supervised in the original paper, where the same instructions are used for both pretraining and testing. While Galactica has been exposed to the same task instructions, it actually employs multitask learning with instructions serving as task identity.
Even in comparison to Galactica's 's multitask result, GIMLET demonstrates comparable superior performance on most datasets. This highlights the ability of GIMLET to perform zero-shot tasks with high quality.
23

---

Figure 6: Scatter of GIMLET over baselines. Below the diagonal line x=y means our method performs better.
The disparity between the multitask result and the tested result with our instructions is due to the gap between their instructions and ours, which indicates that Galactica relies on specific task instructions for task recognition, without true understanding of the instructions. As result, it exhibits poor generalization to other instruction forms. Note that Galactica even do not surpass KVPLM and MoMu which are also zero-shot learning methods.
GIMLET exhibits superior performance compared to the larger model CLAMP on the majority of datasets, with the exception of HIV. It is important to highlight that our model is significantly smaller size than CLAMP, underscoring the effectiveness of our unified graph-text language model. Additionally, it should be noted that CLAMP lacks the capability to handle regression tasks due to its contrastive model architecture, whereas our encoder-decoder architecture enables us to successfully tackle wide range of task types.
Significantly, the supervised results shed light on the task difficulties associated with each dataset. This showcases GIMLET's capability to effectively solve molecule tasks in a zero-shot manner, approaching the performance of supervised results. Furthermore, our pretraining tasks yield an average performance improvement of 3 percent for Graphormer, with the largest gains observed in Bioactivity tasks and the smallest Toxicity tasks. This suggests that there still exist gaps between the pretraining data and our downstream tasks, addressing the zero-shot setting of our dataset.
In Figure 6 we present seatter plots comparing GIMLET with KVPLM and MoMu across all tasks. The diagonal line represents the equality line where x=y indicates our method outperforms the baseline. Notably,
evident that GIMLET consistently performs significantly better than random guessing and surpasses the baselines on all tasks.
We plot the scatter of regression tasks in Figure The plot clearly demonstrates strong correlation between the predicted and actual values for ESOL and Lipo.
C.3 Detailed Few-Shot Results
In both classification tasks and regression tasks, we fine-tune the last linear layer of all models using their respective modeling loss.
It is important to note that the instruction-basse few-shot approach is trained each task individually, while supervised baselines are trained on multiple tasks from the dataset. Therefore, comparing these two approaches may not be strictly fair, as the multitask learning of the supervised baseline can contribute to improved task performance.
The results for few-shot learning on each dataset are presented in Figure 8 It is evident that, across the majority of datasets, GIMLET demonstrates improvement as the number of few-shot examples increases. In fact, it even outperforms or matches the performance of the supervised GIN on several
24

---

lipo
freesolv
Prodoce vokeo
Prodiced vols
Prediceed
+1
Vilun
Vikin
The Valae
Figure 7: Scatter of GIMLET on generative tasks.
datasets, such as bace, bbbp, and esol. There is also observable enhancement in performance across various datasets when employing few-shot learning, including tox21, toxcast, lipo, and freesolv.
There is not result of MoMu on regression tasks, because MoMu contrastive model between graph and text, which cannot handle regression tasks.
C.4 Detailed Ablation Results of Pretraining
The results of pretraining ablation for each dataset are presented in Table 13 14 15 and I6 The findings indicate that both bioactivity assay and physico-chemical properties offer significant benefits for the downstream tasks, demonstrating positive transfer across different domains.
Table 13: Pretraining ablation study on Bio-activity tasks
bace
muv
Average bio
bioactivity assay only physico-chemical only both
0.6390
0.6772
0.6044
0.6402
0.4648
0.5461 0.6624
0.4572
0.4894
0.6957
0.6439
0.6673
Table 14: Pretraining ablation study on Toxicity tasks
tox21
loxcast
Average lox
bioactivity assay only physico-chemical only both
0.5726
0.5625
0.5676
0,4478
0.5017
0,4748
0.6119
0.5904
0.601
C.5
Instruction Robustness
To test the robustness of GIMLET, the Instructions rephrased by GPT-3.5-turbo. types rephrasing, realized by the following prompts:
There are four
rewrite
'Rephrae the text of the following prompt: An
expand
Reepraae the text of the following prompt longer: \n
detail
Rephrasee the text of the following prompt by adding more explanation: \n
short
25

---

0.70
077
6.7
ROC-ALD
ROC
0.95
RDC AUC
0.0
0.95
0.99
65
Shot
Shot
1ox21
WHNUFU NOML dijjuss
toxcast
boop
HVPLUM kokku
0.54
0.7
0.06
ROC AUC
Roc AUC
POC
0.5
05
12
120
22
04
120
25o
120
256
Few Shot Number
Few Shot Number
Few Shot Number
cyp450
*IVVFU NoV, d Olcs GROmel
e50l
+INTU e Nokia Oura GR)6a)
Iipa
IITTUM Hbbk Cu (HNgu)
ROC AOC.AUC
25
32
66
220
255 Tuby 5.p
22
84
120
255
Tub S4
18
32
94
120
255
Tx S.
Pew Shot Nammbr
Few Number
Rew Shotl Number
freesolv
KKVUU NaNu Durs (1)
RASE
9xp
mberr
Figure 8: Few-shot performance each dataset
Table 15: Pretraining ablation study on Pharmacokinetic tasks
bbbp
cyp450
Average .hha
bioactivity assay only physico-chemice only both
0.5313
0.6829
0.6071
0.5932
0.4976
0.5454
0.5939
0.7125
0,6532
26

---

Table 16: Pretraining ablation study on Physical-chemices tasks
esol
freesolv
lipo
Average_phy
bioactivity assay only physico-chemical only both
1.1822 1.1320
5.2935 5.1027
1.3778 1.3450
2.6178
2.5266
Rephrase the text of the following prompt shorter: \n>
Given a task instruction, we rephrase the instruction by the prompts above. Here is an example of four types of rephrased task instruction from Toxcast:
origin
CEETOX CCEETOX_H295R_AAA H295R_ANDDR is one of 23 assay component (s) measured calculated from the CEETOX_ H295R assay. It is designed to make measurements of hormone induction, form of inducible reporter, as detected with absorbance signals by HPLC technology Data from the assay component HEETOX__295R_AND was analyzed into 2 assay endpoints. This assay endpoint, was analyzed in the positive fitting
direction relative to DMSO as the negative control and baseline of activity. Using type of inducible reporter, loss of activity using HPLC-MS-MS was used to understand synthesis of Androstenedione in H295R cell line at 48hr chemical exposure. To generalize the intended target to other relatable targets, this assay endpoint is annotated to the steroid hormone intended target family, where the subfamily is androgens. Is this molecule effective to this assay?"
rewrite
is component of the SERTOX_H2951) assay, which measures hormone induction through absorbance signals detected by HPLC-MS-MS technology. The data from has been analyzed into two endpoints, with GEETOX_H29RR_ANDL being analyzed
"The
relation to DMSO as the negative control and baseline of activity. HPLC-MS-MS was used detect of activity and understand the synthesis of Androstenedione in H295R cell line after 48 hours of chemical exposure. This assay endpoint steroid hormone intended target family, specifically the subfamily androgens, and can be generalized to other similar targets. Can this assay be effectively performed using this molecule?"
related to the
expand
ANDR assay component is just of the 23 assay components that are measured or calculated from the CEETOX_H295R assay. The assay specifically designed to measure hormone induction, which is form of inducible reporter, and is detected using absorbance signals by means of IPLC-MS-MS technology. The data obtained from the CEETOX_H295R
"The
assay component was analyzed into two assay endpoints. The CEETOX_H296R_ANDR_CC analyzed in the positive fitting direction in relation to DMSO the negative control and activity baseline. To understand the synthesis of Androstenedione
assay endpoint was
the H295R cell line after 48 hours of chemical exposure, loss of signal activity was
27

---

used with HPLC-MS-MS technology. This endpoint is annotated to the steroid hormone intended target family to help other related targets, where the subfamily is androgens. Can it be determined if this particular molecule exhibits desirable efficacy to be utilized in this particular assay?"
detail
"The CEETOX_H296R_AND is an assay component that is one of the 23 components that are measured or calculated from the CEETOX_H2995 assay. It is intended induction, which is a form of inducible reporter, and the measurement is done with the help absorbance signals using HPLC-MS-MS technology. The data obtained from the measurement assay component (EETOX_H296R_AND assay endpoints. One of these endpoints, analyzed in the positive fitting direction, relative to DMSO, which is used as the negative control and baseline for activity. The HPLC-MS-MS technology loss-of-sign activity, which helps in understanding the synthesis of Androstenedione in H295R cell line after 48 hours of chemical exposure. To make the intended target more comprehensive and relatable to other targets, the assay endpoint is annotated to the steroid hormone intended target family, where the subfamily is androgens. Can this molecule be used for this assay?"
measure hormone
analyzed into two (RROOXX_295R_ADDD
used to detect the
short
"CEETOX_ A is one of components the CEETOX_ H295R assay, measuring hormone induction detected with absorbance signals by CEETOX CEETOX_H295R, relative to the negative control. loss-of- activity to understand Androstenedione synthesis in H295R cell line after 48hr chemical exposure. It's annotated as steroid hormone intended target in androgens sub- fabily Is molecule suitable for assay?"
MS. It's analyzed into endpoints, with dn being the positive fitting direction analyzes the
C.6 Instruction Ablation
To ablate the explanation-bai based instruction, we remove the explanation and only keep the assay name. The ablated instruction for the instruction above is:
"The assay name is effective to this assay?"
this molecule
C.7 Attention Visualization
We present visualizations of the attention of text tokens to molecule graphs, demonstrating how our unified transformer incorporates molecule information using various instructions. We randomly sample molecules and attention heads for visualization. To emphasize high-level features, we focus on visualizing the attention patterns of the last layer. The redder means the larger attention value.
For BACE instruction, we visualize the attention of several keywords marked in red to molecules:
"BACE1 is an aspartic-acid protease important in the pathogenesis of Alzheimer's disease, and in formation of myelin sheaths. BACEI member family of aspartic proteases. Same as other aspartic proteases, BACE1 1 is bilobal enzyme, each lobe contributing catalytic Asp residue, with an extended active site cleft localized between the two lobes the molecule. The assay tests whether the molecule can bind to the BACEI protein. Is this molecule effective to the assay?"
28

---

For BBBP instruction:
'In general, molecules that passively diffuse across the brain blood barrier have the molecular weight less than 500, with a LogP of 2-4, and no more than five hydrogen bond donors or acceptors. Does the molecule adhere the three rules not?"
(a) BACE1
(b) aspartic-acid
(c) Alzheimer
NH
(d) myelin
(e) aspartic
(f) bilobal
NH
gg)
catalytic
(h) cleft
(i) effective
Figure 9: Visualization of attention for BACE on molecule D=CNCoclcCCCCC/ccCcuoeee///oooop
29

---

(a) BACE1
(b) aspartic-acid
(c) Alzheimer
(d) myelin
(e) aspartic
(f) bilobal
catalytic
(h) cleft
(i) effective
Figure 10: Visualization of attention for BACE on molecule C=C
30

---

(a) BACE1
(b) aspartic-acid
(c) Alzheimer
(d) myelin
(e) aspartic
(f) bilobal
catalytic
(h) cleft
(i) effective
Figure 11: Visualization of attention for BACE on molecule
31

---

H.N
H,N
H.N
(a) BACE1
(b) aspartic-acid
(c) Alzheimer
H,N
H,N
(d) myelin
(e) aspartic
(f) bilobal
H,N
catalytic
(h) cleft
(i) effective
Figure 12: Visualization of attention for BACE on molecule O=CINICOCHHNEE--JNNCCcl..... ccccel
32

---

(a) BACE1
(b) aspartic-acid
cc) Alzheimer
(d) myelin
(e) aspartic
(f) bilobal
catalytic
(h) cleft
(i) effective
Figure 13: Visualization of attention for BACE on molecule
33

---

(a) brain blood harrier
(b) molecular weight
(c)
(d) LogP
(c) 2-4
(f) five
(g) hydrogen bond donors
(h) acceptors
(i) three rules
Figure 14: Visualization of attention for BBBP on molecule
www.CccceeceeeeeCCCCCCCCCYYYCYC [CCCCCCCCCCNNCCCCCCCCCCHHHHHH)) CHJCCCCIC)C((NNe (ICANN@@@(CCCCCC
34

---

(a) brain blood harrier
(b) molecular weight
(c)
(d) LogP
(c) 2-4
(f) five
(g) hydrogen bond donors
(h) acceptors
(i) three rules
Figure 15: Visualization of attention for BBBP on molecule CcIneec2c!||H]]]cccee)
35

---

(a) brain blood harrier
(b) molecular weight
(c)
(d) LogP
(c) 2-4
(f) five
(g) hydrogen bond donors
(h) acceptors
ii) three rules
Figure 16: Visualization of attention for BBBP on molecule CC==CNNG-CCC-DOCCCCCCCCCCC
36

---

OH
OH
(a) brain blood harrier
(b) molecular weight
(c) 500
(d) LogP
(c) 2-4
(f) five
OH
(g) hydrogen bond donors
(h) acceptors
ii) three rules
Figure 17: Visualization of attention for BBBP on molecule
37

---

(a) brain blood harrier
(b) molecular weight
(c)
(d) LogP
(c) 2-4
(f) five
(g) hydrogen bond donors
(h) acceptors
(i) three rules
Figure 18: Visualization of attention for BBBP on molecule CCCIC)CC(==NCI==
38


Reviews:
[{"review_id": "gKMlkvQYqP", "reviewer": "Reviewer_drQy", "summary": "The paper addresses the challenge of molecule property prediction, especially the label insufficiency caused by costly lab experiments. \nIt uses  natural language instructions to handle molecule-related tasks in a zero-shot setting. The authors propose GIMLET, a model that unifies language processing for both graph and text data. GIMLET uses generalized position embedding to encode graph structures and instruction text without additional modules. It also decouples the encoding of the graph from task instructions, thereby improving graph feature generalization across different tasks. The authors create a dataset of over two thousand molecule tasks with corresponding instructions and pretrain GIMLET on these tasks, which allows the model to transfer effectively across tasks. GIMLET demonstrates good performance in instruction-based zero-shot learning, even closely matching supervised Graph Neural Network (GNN) models on tasks such as toxcast and muv.", "questions": "- How model size, and the scale of pretraining (not only the domain) contribute to the final performance of GIMLET; \n- Could GIMLET be extended to using frozen / parameter-efficient fine-tuned language models? / How to validate if the instruction-based pretraining / the proposed GIMLET model is scalable?", "limitations": "See above", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and clearly presented; \n- The authors create a comprehensive dataset for molecule tasks with instructions, which can be a valuable resource for future research.\n- The ideas of leveraging instruction plus graph structures are novel ways for molecule property pretraining and the proposed unified GIMLET for graph and text data could contribute to the field of cheminformatics. \n- The zero-shot performance of proposed methods is strong across baselines and tasks. Detailed ablations including pretraining, model design, and instructions are presented to illustrate the design choices. ", "weaknesses": "- While GIMLET demonstrates strong performance in zero-shot learning, how could it perform with more supervised learning scenarios (beyond 256 or even with full training set) is questionable. The instruction-tuned LLM demonstrates strong generalization performance on single-task finetuning in NLP, is this also true for GIMLET remains a question; \n- How could model size (only one size is presented), the scale of pretraining, and the diversity/quality of instruction / pretraining tasks affect GIMLET performance, it could also be valuable to see the performance of instruction-aided evaluation / qualitative examples on helf-on validation tasks (whether GIMLET could actually follow the instructions).   ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to question 1 in Questions and question 2 in Weaknesses:\n\nThank you for your insightful question. We conduct experiments to illustrate the impact of varying pretraining scale and model size on GIMLET. To manipulate the pretraining scale, we explore different task numbers (tasks selected randomly) in the pretraining phase. Our evaluation includes instruction-based zero-shot performance on the large scale Chembl zero-shot testing set, as well as testing GIMLET across various model sizes on Bio-activity, Toxicity, and Pharmacokinetic tasks. The summarized results are presented below:\n\n Zero-shot performance (ROC-AUC) over Chembl zero-shot\n\n| Pretraining task ratio | 1/4  | 1/2  | Full(1.1K)  |\n| ----   | ----  | ----  | ----  |\nGIMLET(T5-small) |\t0.6650\t |\t0.7343 |0.7860\nGIMLET(T5-base) |\t\t0.6384 |\t\t0.7318 |\t\t0.8041\nGIMLET(T5-large) |\t\t0.6921 |\t\t0.7562\t |\t0.8178\n\n Zero-shot performance (ROC-AUC) over Bio-activity, Toxicity, and Pharmacokinetic tasks\n| -| bace   | hiv  | muv | Avg. bio | tox21  | toxcast| Avg.tox | bbbp| cyp450| Avg. pha |\n| ---- |  ----  | ----  | ----  | ----  | ----|----|----|----|----|----|\n|GIMLET(T5-small)\t|0.6957\t|0.6624\t|0.6439|\t0.6673\t|0.6119|\t0.5904|\t0.6011|\t0.5939|\t0.7125|\t0.6532\nGIMLET(T5-base)\t|0.7240|\t0.6636|\t0.6322|\t0.6733|\t0.6136|\t0.5811|\t0.5974|\t0.7087|\t0.7174|\t0.7131\nGIMLET(T5-large)|\t0.6855|\t0.6986|\t0.6421|\t0.6754|\t0.5988|\t0.5773|\t0.5880|\t0.6758|\t0.7365|\t0.7062\n\nFrom our experiment, we can draw the following conclusions:\n\n(a) The principal bottleneck in the current GIMLET framework appears to be the number of tasks, or instructions, used during pretraining. As the number of pretraining tasks increases, the performance of GIMLET consistently improves. Despite already incorporating 1.1k tasks for pretraining, there's still room for enhancement, especially considering the variability and complexity of molecule property prediction tasks. Thus, increasing the number of tasks is a primary direction for enhancing GIMLET's performance.\n\n(b) Enlarging the model size also leads to performance gains across most tasks. However, the improvements are relatively minor compared to increasing the scale of the dataset. This phenomenon can be attributed to the constrained size of pretraining instructions, which prevents full utilization of the capacity of larger models.\n\nIn summary, both scaling up pretraining and increasing model size have the potential to enhance GIMLET's performance. However, in the current context, the primary bottleneck lies in the scale of pretraining tasks.\n\n### Response to question 2 in Questions:\n\nThis is indeed a key question. Our model architecture is adaptable for direct application to language models of varying scales, but the training method needs to be adjusted accordingly.\n\nFirst, the model architecture we have introduced, which includes mixture of distance-based position embeddings and casual attention between graphs and text, represents a general approach that empowers language models to effectively understand graph data. This approach is applicable to language models of various scales.\n\nSecond, a significant aspect related to model scalability is the training method employed. While we trained the T5 model through continuous pretraining, for larger language models, opting for parameter-efficient fine-tuning techniques such as adapters would likely be a more optimal solution.\n\n\n\n### Response to question 1 in Weaknesses:\n\nThanks for the question. This study mainly focuses on zero-shot learning for molecule property prediction based on instructions, enabling generalization to unseen tasks without the requirement of labeled data, which typically necessitates expensive wet experiments. While we have shown that the performance of GIMLET can be enhanced through finetuning on few-shot examples in Subsection 4.2: Instructions-Based Few-Shot Finetuning, we also include the experimental results under the fully finetuned setting here:\n\n\nSupervised result (ROC-AUC $\\uparrow$ ) over Bio-activity, Toxicity, and Pharmacokinetic tasks.\n|            | bace   | hiv    | muv    | Avg.bio | tox21  | toxcast | Avg.tox | bbbp   | cyp450 | Avg.pha |\n| ---------- | ------ | ------ | ------ | ------- | ------ | ------- | ------- | ------ | ------ | ------- |\n| GIN        | 0.701  | 0.753  | 0.718  | 0.724   | 0.740  | 0.634   | 0.687   | 0.658  | 0.821  | 0.739   |\n| Graphormer | 0.7760 | 0.7452 | 0.7061 | 0.7424  | 0.7589 | 0.6470  | 0.7029  | 0.7015 | 0.8436 | 0.7725  |\n| GIMLET     | 0.8280 | 0.7834 | 0.7267 | 0.7794  | 0.7676 | 0.6591  | 0.7134  | 0.7315 | 0.8809 |0.8062|\n\nSupervised performance (RMSE $\\downarrow$) on Physicalchemical datasets.\n\n|            | ESOL  | lipo  | FreeSolv | Avg.phy |\n| ---------- | ----- | ----- | -------- | ------- |\n| GIN        | 1.243 | 0.781 | 2.871    | 1.632   |\n| Graphormer | 0.901 | 0.740 | 2.210    | 1.284   |\n| GIMLET     | 0.850 | 0.945 | 1.881    | 1.226   |\n\n\n\nIt is evident that with a complete set of supervised training data, GIMLET's performance surpasses or is comparable to other supervised baselines. This reveals the potential of GIMLET in supervised fine-tuning.\n"}, {"review_id": "pamwCmSx2U", "reviewer": "Reviewer_3HAC", "summary": "This paper presents GIMLET, a unified graph-text model for instruction-based molecule zero-shot learning. The proposed model uses natural language instructions to tackle molecule-related tasks in a zero-shot setting. GIMLET overcomes existing limitations and significantly outperforms molecule-text models. The paper includes experimental results that demonstrate the superiority of GIMLET over molecule-text models. The paper also explores the use of pretraining tasks and the robustness of GIMLET to instructions. Overall, the paper provides a comprehensive approach to molecule zero-shot learning using natural language instructions.", "questions": "See weaknesses.", "limitations": "The authors have adequately addressed the limitations.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper introduces GIMLET, a pioneering, integrated graph-text model specifically designed for instruction-based, zero-shot learning in molecular sciences. Leaping beyond the confines of current methodologies, GIMLET leverages natural language instructions to proficiently navigate molecule-related tasks in a zero-shot context. It surpasses the performance of conventional molecule-text models, reflecting substantial advancements in this field. Through rigorous experimental analysis, the superiority of GIMLET over conventional models is vividly illustrated.", "weaknesses": "1.One thing I'm not quite sure about is why the authors chose only a few molecule-text models as baselines, with MoMu and Galactica not even being specifically designed for predicting molecular properties. I understand that the authors might have done this because the model proposed in the paper is based on both molecules and text, but for a fair comparison, shouldn't other types of models (like those involving only molecules and not text) that predict molecular properties also be used as baselines? Otherwise, how can we highlight the advantage of incorporating text into the model for this task?\n\n2.The paper uses a graph-text position encoding method to encode molecular graphs. Given that molecules already have their serialized representations, such as SMILES and SELFIES, isn't this approach superfluous and potentially introducing unnecessary computations and noise?\n\n3.Based on the description in this paper, I can't clearly understand the specific definition of molecule zero-shot learning, which doesn't seem to differ much from other pre-training-fine-tuning training methods? Perhaps I misunderstood, could the authors provide a more precise explanation?\n\n4.In Figure 1, the authors don't seem to have clearly described the specific meaning of different colored lines and boxes, which makes it difficult for readers to interpret this image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your questions. We would like to respond to question 3 first, which pertains to the fundamental aspect of the task our paper is working on.\n\n### Response to question 3:\n\nSorry for the confusion. In this work, we propose to investigate the feasibility of employing instructions to accomplish molecule tasks in a zero-shot setting. This approach aligns with the paradigm of instruction-based zero-shot learning [1], which we introduced in Section 2: Related Work - Instruction-based Zero-Shot Learning.\n\n**What is instruction?** The instructions, also referred to as prompts, refer to natural language descriptions of the tasks to be executed. For example, in the case of the text abstract generation task, the instruction could be \"Write a summary for the following text.\" In our context, these are natural language instructions for molecule property prediction tasks. For instance, in the \"toxicity to ARE pathway\" task (from Tox21), the instruction is as follows: \"Oxidative stress has been implicated in the pathogenesis of a variety of diseases ranging from cancer to neurodegeneration. The antioxidant response element (ARE) signaling pathway is important in the amelioration of oxidative stress. Is this molecule agonists of antioxidant response element (ARE) signaling pathway?\"\n\n**What is instruction-based molecule zero-shot learning?** The instruction-based molecule zero-shot learning aims to enable model to perform unseen molecule tasks **without supervised training**. This is enabled by the provided task instructions for new tasks. By understanding the instructions, the model can perform the unseen tasks directly. For example, the \"toxicity to ARE pathway\" task mentioned above is not seen by the GIMLET before, but GIMLET is able to perform it without supervision by utilizing and understanding the task instruction. This approach eliminates the need for labeled data and leverages the textual knowledge available for downstream tasks. \n\n**How can we enable model to perform instruction-based molecule zero-shot learning?** It has been shown that language models exhibit strong instruction-based zero-shot performance after pretraining on a large scale of tasks with instructions [1]. For the molecule domain, we also do the instruction-based task pretraining for our graph-text model GIMLET. As illustrated in Figure 2 (left), we partition tasks into pretraining tasks and downstream testing tasks, without overlapping. After pretraining, we zero-shot-test model on the downstream tasks with task instructions provided in Subsection 4.1: Instruction-Based Zero-Shot Learning, which is our main experiment.\n\n**Difference between pretrain-finetuning method?** The main difference lies in the testing session, where instruction-based zero-shot learning does not require supervised training data, but executes the new tasks by only instructions.\n\n\n### Response to question 1\n\nWe employ molecule-text models as baselines due to our focus on the instruction-based zero-shot learning setting. In this context, the input comprises both molecule data and task instruction text, necessitating the utilization of molecule-text models rather than graph-only models.\n\nBesides the molecule-text models, we also have presented supervised results of graph-only models in Tables 1 and 3. These models include GCN, GAT, GIN, and Graphormer, serving as references for the supervised results.\n\n### Response to question 2\n\n\n\n\nThis is indeed an insightful question. While serialized representations of molecules, such as SMILES, can convey molecular information, they might fall short in accurately reflecting the actual structural features of the molecule's graph.\n\nTo better illustrate the impact of employing graph representations on performance and speed, we conducted an additional experiment. We pretrained another T5 model using the same pretraining settings as GIMLET, but utilizing the SMILES representation of molecules rather than graph. This version is referred to as GIMLET-SMILES. We evaluate the performance of both GIMLET and GIMLET-SMILES using instruction-based zero-shot testing on Chembl zero-shot dataset. Additionally, we present the average FLOPs per data point as a metric to compare the computational efficiency. The experimental results are as follows:\n\n| Model | Chembl zero-shot ROC-AUC $\\uparrow$  | Avg. FLOPs per data $\\downarrow$  |\n|---- |  ----  | ----  |\n|GIMLET|\t0.7860|\t5.03E+09\n|GIMLET-SMILES|\t0.7414|\t5.12E+09\n\n\nIt is evident that when utilizing the molecular graph, the model outperforms the use of molecule SMILES in instruction-based zero-shot learning tasks. Remarkably, the computational costs of the two models are nearly identical. This is due to GIMLET employing the same computational framework as the regular T5 model, with the graph-text mixture positional embeddings incurring costs equivalent to the normal T5 relative positional embeddings. There is no need for additional modules or computations within the model.\n\n\n### Response to question 4:\n\nSorry for the confusion. We suppose the color and lines you mentioned are the illustration of distance aware encoding, where lines with different colors refer to different distance values. In our upcoming versions, we intend to provide a more comprehensive explanation of the figure for clarity.\n\n\n[1] Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., ... & Rush, A. M. (2022, April). Multitask Prompted Training Enables Zero-Shot Task Generalization. In ICLR 2022-Tenth International Conference on Learning Representations.\n\n"}, {"review_id": "boD0VKB5Mg", "reviewer": "Reviewer_UK2h", "summary": "This paper proposes a unified language model for both graph and text data with two main tech contributions: (1) a unified graph-text transformer encoder with a distance-based joint position embedding to encode graphs, and (2) textual instructions to enhance transferability among tasks. Zero-shot tests on classification and regression tasks are conducted to show its effectiveness.", "questions": "Critical question: Do we really need a text-decoder model instead of conventional GNNs to perform classification and regression tasks?", "limitations": "See weakness part.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The high-level motivations of a unified graph-text model for molecular understanding and enhancing transferability with instructions are interesting.", "weaknesses": "1. It doesn’t make sense to evaluate the model on classification and regression tasks. The model structure is based on T5, which is for generation/translation tasks. In fact, the results in Table 1 show a huge performance gap between GIMLET and GNNs. Why not use those GNNs and Graphormers with better performance and fewer parameters? I think additional experiments of fine-tuning GIMLET on the train datasets of Bio-activity, Toxicity, and Pharmacokinetic tasks in Table 1 are needed to make a fair comparison with supervised GNNs.\n2. Claims need to be further clarified. For example, in line 150-154, I don’t understand “the dense vectors encoded by GNN have a limited capacity to carry structure information”. According to the results in your Table 1, those dense vectors encoded by GNN actually carry more information than GIMLET. And “training the additional module is difficult due to the increased layers”, actually, in MoMu and CLAMP (cited in line 146), the graph and text encoders are two-tower structure like CLIP, which I don’t think increased layers.\n3. The writing needs to be improved. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to question 1 in Weaknesses and question 1 in Questions:\n\n**Why use the language model (T5)?**\n\nThank you for your question regarding the task setting. This study focuses on instruction-based zero-shot learning for molecule property prediction. Our approach aims to predict properties by natural language task instructions, **without the need for supervision from training data**. This methodology aligns with the instruction-based zero-shot learning paradigm [1], as introduced in Section 2: Related Work - Instruction-based Zero-Shot Learning.\n\nIn comparison to supervised training for molecule tasks, instruction-based zero-shot learning offers several key advantages. Firstly, it eliminates the need for task-specific labeled data, which can often be complex and expensive to annotate for molecules. Secondly, it allows the execution of new tasks without the resource and time-intensive process of retraining on new task samples. Lastly, it offers a more user-intuitive way of interacting with the model through natural language task descriptions, as opposed to relying solely on task samples. This is why instruction-based task execution has gained significant popularity not only in natural language processing [2] but also in the realm of visual-text multimodal tasks [3].\n\nTo enable the model to perform novel molecule tasks **without the need for supervised training, task instructions for these new tasks are crucial**. By comprehending these instructions, the model can directly carry out the new tasks. \n\nThe necessity to comprehend textual instructions underscores the imperative for utilizing a language model. In pursuit of this goal, we extend the capabilities of the existing T5 language model to encode both graph and text data.\n\n**Is the task format suitable for T5?**\n\nRegarding task type classification and regression, we unifiy the outputs of different tasks into the text modality. This is explained in Subsection 3.1: Problem Formulation and Framework Overview, and depicted in Figure 1. The outputs of diverse tasks are presented as text, such as \"yes,\" \"No,\" or \"3.11.\" This approach unifies various tasks into conditioned text generation tasks, which aligns well with the nature of the T5-type model, an encoder-decoder model for conditional text generation.\n\n**Why not utilize GNNs?**\n\nThe primary limitation is that graph-only models are unable to execute the instruction-based learning approach, involving both text and graph modalities. GIMLET, however, unifies the two modalities within a single model, enhancing its capacity for graph encoding and textual comprehension.\n\n**The performance gap between GIMLET and GNNs?**\n\nFirst, it is important to address that in Subsection 4.1 (Table 1, 2, and 3), the results of GIMLET and all the molecule-text model baselines are evaluated using instruction-based zero-shot learning. In contrast, the results of GNNs are obtained through supervised learning. We present the supervised learning results as a reference to gauge the difficulty of the tasks.\n\nTo illustrate the model performance improvement when supervised training data is available, we conduct few-shot experiments in Subsection 4.2: Instructions-Based Few-Shot Fine-tuning. Our experiment demonstrates that with a small amount of training data, GIMLET's performance can be further enhanced.\n\nAdditionally, we present the experimental results under the fully finetuned setting here:\n\nSupervised result (ROC-AUC $\\uparrow$ ) over Bio-activity, Toxicity, and Pharmacokinetic tasks.\n|            | bace   | hiv    | muv    | Avg.bio | tox21  | toxcast | Avg.tox | bbbp   | cyp450 | Avg.pha |\n| ---------- | ------ | ------ | ------ | ------- | ------ | ------- | ------- | ------ | ------ | ------- |\n| GIN        | 0.701  | 0.753  | 0.718  | 0.724   | 0.740  | 0.634   | 0.687   | 0.658  | 0.821  | 0.739   |\n| Graphormer | 0.7760 | 0.7452 | 0.7061 | 0.7424  | 0.7589 | 0.6470  | 0.7029  | 0.7015 | 0.8436 | 0.7725  |\n| GIMLET     | 0.8280 | 0.7834 | 0.7267 | 0.7794  | 0.7676 | 0.6591  | 0.7134  | 0.7315 | 0.8809 |0.8062|\n\nSupervised performance (RMSE $\\downarrow$) on Physicalchemical datasets.\n\n|            | ESOL  | lipo  | FreeSolv | Avg.phy |\n| ---------- | ----- | ----- | -------- | ------- |\n| GIN        | 1.243 | 0.781 | 2.871    | 1.632   |\n| Graphormer | 0.901 | 0.740 | 2.210    | 1.284   |\n| GIMLET     | 0.850 | 0.945 | 1.881    | 1.226   |\n\nIt is evident that with full supervised training data, GIMLET's performance surpasses or is comparable to supervised baselines. This reveals the potential of GIMLET in supervised fine-tuning.\n### Response to question 2 in weaknesses:\nSorry for the confusion. In lines 150-154, we discuss a simple and common method for combining text with other modalities, i.e., the features obtained from the GNN are input into the language model as tokens. This simple method is also tested in our ablation study (Table 4) and is denoted as \"w.o. unifying.\" The ablation experiment reveals that it performs worse than GIMLET.\n\nThe purpose of lines 150-154 is to analyze the drawbacks of this simple method. First, the dense vectors encoded by the GNN have limited capacity to carry structural information as input for language models. Furthermore, the additional GNN is appended before the initial layer of the language model, resulting in increased layer count and potential issues. \n\nOur intention is not to refer to the GNN capacity in general case, or the contrastive baselines like MoMu. We will provide clearer explanations for the situation of these claims to avoid confusion in the future version.\n\n### Response to question 3 in weaknesses:\n\nThanks for pointing out! We will improve the writing in the future version.\n\n[1] Multitask Prompted Training Enables Zero-Shot Task Generalization. In ICLR 2022.\n\n[2] Finetuned Language Models are Zero-Shot Learners. In ICLR 2022.\n\n[3] Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. preprint 2023."}, {"review_id": "7BX3zyj2tS", "reviewer": "Reviewer_M8Li", "summary": "The paper proposes a new unified language model for graph and text data for instruction-based molecule zero0shot learning. The paper tries to address the problem of a supervised fine-tuning approach where labeled data by instruction tuning. The paper first treats both graph nodes and instruction tokens as input tokens. The paper then proposes a new relative position embedding to represent the edges in the graph and the relative position for text. The paper then retrains the model on Chembl and sometimes pm several tasks from MoleculeNet.The instructions are based on websites and databases. In a zero-shot setting, the model is compared against several baselines, including KVPLM, Galactica, and MoMu. The model is also compared to the graph-based models, including GCN, GAT, GIN, and Graphormer, in a supervised setting. The paper applies few-shot instruction-0based tuning for several datasets and compares it against KVPLM, MoMu, and GIN. The paper then conducts ablation and exploration studies. ", "questions": " Does the model use each node's chemical element as the molecular graph node's textual form? ", "limitations": "The paper fails to include a limitation section. The paper needs to include a social impact section to discuss the potential misuse of the current system. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a new unified text-graph T5-based model to unify molecular graphs and task instructions. The idea of relative position embedding for molecular graphs is interesting.\n2. The paper achieves strong performance against several zero-shot baselines with fewer parameters. The paper also conducts experiments over multiple different tasks. The model also performs well on the few-shot settings. The paper shows its model design by conducting an ablation study to analyze the effectiveness of unified inputs and decoupling encoding. The paper also shows the effectiveness of pertaining over two different pertaining dataset. The paper also examines the robustness of instruction by GPT3.5 paraphrase. Finally, the paper ablates the explanation of instructions in downstream tasks.  \n3. The paper includes a very detailed appendix to explain some parts of the paper. The paper also visualizes attention on molecular to help readers understand the model.", "weaknesses": "1. The idea of encoding graphs with a seq2seq transformer has been introduced previously. This idea has been applied to graph-to-text tasks since 2021 (Ribeiro et al., 2021), despite using an unchanged T5 structure. One contribution of this paper is the new relative position embedding. However, the paper fails to include it in the ablation study. \n2. The input of the model needs to be clarified. It would be better to include a sample input-output pair in the Appendix. Does the model use each node's chemical element as the molecular graph node's textual form? The datasets used for testing are hard to understand for readers who are not familiar with them. It would be better to include a small explanation for each testing dataset in the Appendix.\n3. The paper fails to include a limitation section and an ethical section for potential social impact. \n\nRibeiro, L., Schmitt, M., Schutze, H., & Gurevych, I. (2021). Investigating Pretrained Language Models for Graph-to-Text Generation. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI (pp. 211–227). Association for Computational Linguistics.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to question 1 in weaknesses:\n\n**Graph-to-text tasks related work**\n\nThanks for the reminding of related work. We will add this and other related work in the next version of the paper.  \nThe difference between the mentioned work and ours lies in the task objectives. The mentioned work processes serialized graph and translate it to the corresponding text. our task involves utilizing both molecular graph data and task-specific instructional text as input, with the objective of generating corresponding task outcomes as output.\n\n**Ablation study of the position embedding**\n\nWe conducted an ablation study to assess the impact of our unified graph-text encoding method augmented with mixture position embedding. In this study (refer to Subsection 4.3 Ablation and Exploration Studies, Table 4), we introduced a simplified model labeled as \"w.o. unifying.\" This model employs a GNN to derive graph features, which are then fed into the language model as tokens, without our mixture position embedding. The ablation experiment clearly demonstrates that this \"w.o. unifying\" model underperforms compared to GIMLET. This result underscores the significance of the unified graph-text encoding method that we have introduced.\n\nWe further extend the ablation analysis on the position embedding by introducing the GIMLET-SMILES baseline. In GIMLET-SMILES, a regular T5 model is employed to process instructional text and sequential graph information, represented as SMILES sequences of molecules. To evaluate the performance of both GIMLET and GIMLET-SMILES, we employed instruction-based zero-shot testing on the Chembl zero-shot dataset. The experimental result is as follows:\n\n| Model         | Chembl zero-shot ROC-AUC $\\uparrow$ |\n|---------------|------------------------------------|\n| GIMLET        | 0.7860                             |\n| GIMLET-SMILES | 0.7414                             |\n\nThe results clearly indicate that the model utilizing the graph encoding method (GIMLET) surpasses the ablated GIMLET-SMILES. This ablation study further illustrates the advantages of our graph-encoding method.\n\n\n### Response to question 2 in weaknesses (and question 1 in Questions):\n\nSorry for the confusion, and your understanding is right. Each graph node corresponds to an atom within the molecule, identified by its respective chemical element. The representation of our input data is provided in Figure 1. We will include more detailed illustrations of data in the appendix in the next version of the paper.\n\n### Response to question 3 in weaknesses:\n\nThanks for your reminder. We intend to include a section on limitations and ethical considerations in the upcoming version of the paper.\n"}, {"review_id": "1sID8j9AsM", "reviewer": "Reviewer_3WHb", "summary": "This paper introduces GIMLET, a unified graph-text model for instruction-based molecule task pretraining. It leverages natural language instructions and decoupled graph encoding techniques to improve the interaction of graphs and texts. Through experiments and evaluations, the paper demonstrates the effectiveness and robustness of GIMLET in various downstream tasks, showcasing its potential for molecule zero-shot learning.", "questions": "- For the pretraining dataset, I understand the authors isolated the tasks for training/testing, but are the molecules also isolated for train/test, i.e. will the same molecule appear in both the training and testing set?\n- For graph, node embeddings are used as input tokens. If the number of nodes varies a lot for different graphs, how do the authors handle the computational efficiency as a max length (padding) will be set for the input?", "limitations": "I'm not an expert in biology/chemistry, so I'm not sure if the procedures for constructing the dataset are correct. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- This paper proposed GIMLET, a novel unified graph-text model that leverages natural language instructions for pretraining. It takes each node as a token, alleviating the problem of losing structural information using separate graph encoding.\n- The authors collected a set of datasets for pretraining, which can be a useful resource for the community.\n- The empirical results look promising compared to previous methods.\n- The paper is overall well-written and easy to follow.", "weaknesses": "- The authors didn't present the training details, e.g. total iterations, hyper-parameters, etc.\n- GIMLET was trained from scratch with a large dataset. What are the computational requirements (GPU/memory/time) for training such a model? It seems the training can be quite expensive, especially considering the graph encoding part.\n\n\nMinor: \n- Page 4, line 126: need to explain what is $o_m$.\n- Page 9, line 333: nature language -> natural language.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to question 1 in weaknesses:\n\nThank you for your reminder regarding reproducibility. We will open-source both our pretraining dataset and code to ensure that reproducibility is achievable. We also introduce some important hyperparameters here:\n\n| Hyperparameters    | Value    |\n| ------------------ | -------- |\n| sample number      | 23.8M    |\n| batch size         | 64       |\n| epoch              | 1        |\n| total steps        | 370K     |\n| learning_rate      | 5.00E-05 |\n| dropout_rate       | 0.1      |\n| dense_act_fn       | relu     |\n| layer_norm_epsilon | 1.00E-06 |\n\nwe will include these hyperparameters in the future version of the paper.\n\n### Response to question 2 in weaknesses:\n\nSorry for the confusion. We continue pretrain the existing T5 parameters rather than from scratch. This approach allows us to leverage the strong text understanding capabilities of the pretrained T5 model, resulting in better instruction following.\n\nThe pretraining process is relatively efficient in terms of computational resources. It requires approximately 1.5 GPU days on a single V100 32G. Through techniques like gradient accumulation, it can also be performed on GPUs with smaller memory, albeit with slightly extended training times.\n\nThe computation cost for graph encoding is not high, as GIMLET utilizes the same computational framework as the standard T5 model. Specifically, the graph-text mixture positional embeddings incur costs equivalent to the normal T5 relative positional embeddings.\n\n### Response to question 3 in weaknesses:\n\nThanks for pointing out the omission！ The $[o_1,\\dots,o_m]$ denotes tokens in the instruction text, where each $o$ represents a token, and $m$ represents the length of the instruction text. We will correct these omissions in the next version of the paper.\n\n### Response to question 1 in Questions:\n\nThis is an indeed interesting question! To verify the transferability of our model to both novel tasks and new data, we split the Chembl dataset with both isolated tasks and molecules, i.e., the Chembl Zero-Shot, as the pretraining and downstream testing dataset. The experiment in Table 2 demonstrates the remarkable capability of GIMLET to transfer to Chembl Zero-Shot well, thereby validating our approach of instruction-based molecule zero-shot learning for both novel tasks and new molecules.\n\n\n### Response to question 2 in Questions:\n\nThanks for the interesting question! Currently, there's a relatively small variance in molecule data lengths compared to text lengths. For instance, while text length varies from 30 to 500, most molecule sizes do not exceed 60. Therefore, the size of the molecules doesn't pose a significant challenge to computational efficiency. For extreme cases, the computation cost may slightly increase due to the increased input length."}]

Author rebuttal:

Meta-review: This paper presents GIMLET, a unified graph-text model for instruction-based molecule zero-shot learning. The proposed model uses natural language instructions to tackle molecule-related tasks in a zero-shot setting. GIMLET overcomes existing limitations and significantly outperforms molecule-text models. The paper includes experimental results that demonstrate the superiority of GIMLET over molecule-text models. The paper also explores the use of pretraining tasks and the robustness of GIMLET to instructions.

Overall, the paper provides a comprehensive approach to molecule zero-shot learning using natural language instructions.
Decision: Accept (poster)