One For All: Towards Training One Graph Model For All Classification Tasks
Authors: Hao Liu
Jiarui Feng
Lecheng Kong
Ningyue Liang
Dacheng Tao
Yixin Chen
Muhan Zhang
Venue: ICLR 2024 spotlight (2024)
Keywords: Graph Neural Network
Large Language Model
In-context Learning
Primary area: learning on graphs and other geometries & topologies
TL;DR: This paper proposes a unified graph learning framework capable of cross-domain classification spanning node, edge, and graph tasks.

Abstract: Designing a single model to address multiple tasks has been a long-standing objective in artificial intelligence. Recently, large language models have demonstrated exceptional capability in solving different tasks within the language domain. However, a unified model for various graph tasks remains underexplored, primarily due to the challenges unique to the graph learning domain. First, graph data from different areas carry distinct attributes and follow different distributions. Such discrepancy makes it hard to represent graphs in a single representation space. Second, tasks on graphs diversify into node, link, and graph tasks, requiring distinct embedding strategies. Finally, an appropriate graph prompting paradigm for in-context learning is unclear. We propose **One for All (OFA)**, the first general framework that can use a single graph model to address the above challenges. Specifically, OFA proposes text-attributed graphs to unify different graph data by describing nodes and edges with natural language and uses language models to encode the diverse and possibly cross-domain text attributes to feature vectors in the same embedding space. Furthermore, OFA introduces the concept of nodes-of-interest to standardize different tasks with a single task representation. For in-context learning on graphs, OFA introduces a novel graph prompting paradigm that appends prompting substructures to the input graph, which enables it to address varied tasks without fine-tuning. We train the OFA model using graph data from multiple domains (including citation networks, molecular graphs, knowledge graphs, etc.) simultaneously and evaluate its ability in supervised, few-shot, and zero-shot learning scenarios. OFA performs well across different tasks, making it the first general-purpose across-domains classification model on graphs.

Published as a conference paper at ICLR 2024
ONE FOR ALL: TOWARDS TRAINING ONE GRAPH MODEL FOR ALL CLASSIFICATION TASKS
Hao Liu Jiarui Feng Lecheng Kong!" Ningyue Liang'
Dacheng Tao?
Yixin Chen' Muhan Zhang
WWashington University in St. Louis 2 *Nayyang Technological University SPeking University (liuhac, feng. jiarui, jerry verry.kong, fliang, ychen25) @wustl. dacheng. www.ane..e..u..eeeee..ooe muhan@pku.edu.co
ABSTRACT
Designing a single model address multiple tasks has been long-standing ob- jective in artificial intelligence. Recently, large language models have demon- strated exceptional capability in solving different tasks within the language do- main. However, a unified model for various graph tasks remains underexplored, primarily due the challenges unique to the graph learning domain. First, graph data from different areas carry distinct attributes and follow different distribu- tions. Such discrepancy makes it hard to represent graphs in a single represen- tation space. Second, tasks graphs diversify into node, link, and graph tasks, requiring distinct embedding strategies. Finally, an appropriate graph prompting paradigm for in-context learning is unclear. We propose One for All (OFA), the first general framework that can use single graph model to address the above challenges. Specifically, OFA proposes text-attributed graphs unify different graph data by describing nodes and edges with natural language and uses language models encode the diverse and possibly cross-domain text attributes feature vectors in the same embedding space. Furthermore, OFA introduces the concept of nodes-of-interest to standardize different tasks with single task representa- tion. For in-context learning on graphs, OFA introduces a novel graph prompting paradigm that appends prompting substructures the input graph, which enables it to address varied tasks without fine-tuning We train the OFA model using graph data from multiple domains (including citation networks, molecular graphs, knowledge graphs, etc.) simultaneously and evaluate its ability in supervised, few- shot, and zero-shot learning scenarios. OFA performs well across different tasks, making it the first cannerll-prppoe across-domains classification model on graphs
INTRODUCTION
Recently, large language models (LLMs) have received tremendous attention due their power and versatility in solving natural language tasks like text generation, machine translation, and question- answering. LLMs' in-context learning ability and universality allow the model to directly perform various cross cross-domain downstream tasks by providing related context or prompt to the model, there-
fore avoiding any fine-tuning on model parameters (Brown et al. 2020 Zhang et al.) 2023b Lu et al. 2021 Bommasani et al. (021].
Despite the great success of foundation models on language, developing foundation model for graph structure data is less explored. Particularly, several challenges unique to graph data pre- vent the direct transfer of foundation model design from the language domain to the graph domain. First, although the natures of language tasks differ, they are still uniformly represented in human- interpretable texts. An LLM can encode them into the same text embedding space and train on different source tasks together. However, graph datasets from different sources are usually com- pletely different in feature representation. Concretely, widely used graph datasets include citation
"Contributed equally. Listing order is random. 'Coresponding author.

---

Published as a conference paper at ICLR 2024
Ia the paper
Citation Network
abous Butl
Node Classification
Dourpiion
Malecale inhiher HIV!
Molceular Gngpl
Deseription
LLM
Milcctie affoctive
Single GNN
Graph Classification
Knowledge Graph
Relation Type Denenptiin
Link Prediction
Jobe Aaple!
More data and tasks.
Cros-dxmain Gruph Data
Tink Descripii
LLM besed Feature Thas Enthaläng
Pastur Taek-depesdem Pronpe Graph
GNN-kaaci Graph Firbedäing
Downtrram Tink Prodictios
Figure 1: The pipeline of OFA. An input to the model contains text-attributed graph and task description. Cross-domain texts in graphs and task descriptions can be co-embedded the same space by an LLM. OFA's graph prompting paradigm converts the input with embedded features to prompted graphs with a unified task representation, which allows adaptive downstream prediction.
networks (Yang et al. 2016 Hu al. 2020), e-commerce networks (Shchur et 2018). knowl- edge graphs (Dettmers el al. 2018: Toutanova & Chen 2015), and molecular graphs Dwivedi et al., 2020). Their raw forms contain attributes generated from isolated processes. For example, node fea- tures in molecular graphs are usually vectors whose entries are indices of nominal features of atoms. In contrast, node features e-commerce networks could be Bag-of- vectors of item descrip- tions. These features are so different in dimension, scale, and semantic meanings that it is almost impossible to directly Iearn the representation of these data using the same model. Second, different downstream tasks in the graph domain attend to different parts of the graph and require task- specific knowledge and methodologies. Specifically, graph-related tasks can be roughly divided into three classes: node-level, link-level, and graph-level. Even though Graph Neural Networks (GNNs) achieved great success in all three task classes, the rationale for the success behind each task class is different. For node-level tasks, proper smoothing of the node features leads good performance (Defferrard et al. 2016 Chien et al., 202T He et al. 202T). However, for link-level and graph-level tasks, encoding the local structure vital to the success, encouraging line of work that develops more expressive GNNs (Xu et al.) 2018 Zhang & Li [021] Zhang & Chen 2018). Generally, powerful model for node-level tasks may not work on link- or graph-level tasks. Consequently, current models are incompetent and infeasible to learn different tasks jointly. Third, the design of the in-context learning or prompt is straightforward in natural language, where we can simply add description of the task solution to add such context information to the graph generically. How to design unified way to perform cross-domain and in-context learning the graph tasks is ambiguous.
few examples to the input. However, there is no existing
To address these challenges, we propose One-for-All (OFA), a general solution for building and training foundation GNN model with in-context learning ability across different domains. OFA has three main unique features: (1) OFA uses text-attributed graphs (TAGs) to integrate graph datasets from different domains into one large TAG dataset and leverages the power of LLMs to learn from all domains jointly. We collect nine graph datasets commonly used in the community varying in size, domains, and task types (see Table I for the full list). Then, describe all nodes and edges in the graphs using human-readable texts and embed the texts from different domains into the same embedding space with single LLM. (2) OFA proposes the nodes-of-interes (NOI) subgraph and the NOI prompt node, which not only unify different types of graph tasks but also improve the ability of the foundation model to learn the structural information in the graph. (3) OFA introduces a carefully designed and widely applicable graph prompting paradigm (GPP) that inserts prompt graph into the original input graph a task-specific way. The nodes in the prompt graph contain all related information about the downstream task (described by texts and encoded by the same LLM encoder as the input graph). Then, the modified graph becomes the actual input to the foundation graph model. Thus, the model is adaptive perform different tasks according to the prompt graphs. Figure [ illustrates the pipeline of OFA. After training, the users can describe any graph with natural texts and apply the OFA pipeline to predict possibly unseen classes.
2

---

Published as a conference paper at ICLR 2024
Table Detailed summarization of all collected datasets in OFA.
Dataset
Domain
Task
Graphs
Avg. #Nodes
Avg. #Edges
# Classes
Cora
Citation
Node/Link
1
2,708
10,556
PubMed
Citation
Node/Link
1
19,717
44,338
ogbn-arxiv
Citation
Node
1
169,343
1,166,243
40
Wiki-CS
Web link
Node
11,701
216,123
10
FB15K237
Knowledge
Link
14,541
310,116
237
WNI8RR
Knowledge
Link
1
40,943
93,003
PCBA
Molecule
Graph
437,929
26.0
28.1
128
HIV
Molecule
Graph
41,127
25.5
27.5
ChEMBL
Molecule
Graph
365,065
25.9
55.9
1048
We evaluate the proposed OFA on all collected TAG datasets under supervised, few-shot, and zero- shot scenarios. We demonstrate that single OFA model can perform well on cross-domain and cross-task scenarios. In particular, show that the OFA achieves great results zero-shot learn- ing, which is impossible for most the existing graph models.
PRELIMINARIES
Text-attributed graphs (TAGs). We define a TAG as a graph where each node and each edge in the graph is associated with text sentence. We denote TAG as (V, E, R), where V (Vi,..., " is the set of nodes, R = {ri...., is the set of relations, = [ei,...,,ee {(g) is the set of edges. An edge Eij = (2; r, U2) E & consists and target node fy V. Let Se, denote the text sentence associated with node e; and Seg denote the text sentence associated with edge Gi E E. Finally, let Na denote the set of all neighbor nodes within hops of node v. Note that some concurrent works also introduce the concept of TAGs (Chen et al. 2023 He et al. 2023). However, they only focus on graphs whose raw node features are already texts.
source node v; E V, a relation r E R,
the contrary, we extend this concept and regard all graphs TAGs since any nodes and edges are describable by texts.
Learning scenarios. In this work, we focus on the classification problem. Denote a dataset as D = {(d;, P. where D is the number of data in the dataset, de is data sample, Yi E y is the label of d; and y is the set of all data labels. train classifier, we split the dataset into the train, validation, and test sets, denoted
Dtrain. Daats and Drest respectively. Their label sets are supervised learning, the model will be trained on Dtrain and be evaluated on D to determine the best model. Finally, Drest is used to evaluate the model's performance. All labels in the validation and test data are seen during training. that is, Merain Veest- The second learning scenario is few-shot learning. The training and evaluation procedure shot learning, we have Vivain n Viest 0. Few-shot learning typically deals with N-way K-shot tasks, where we use N K data {(d, })} provided with K labeled data and N the total number of distinct classes. Next, given these support samples, the model needs to classify data in the query set Q = {d: into these N classes. The third learning scenario is zero-shot learning, which can be viewed as special case of few-shot learning- In zero-shot learning, for any N-way Khohot task, we have K 0 as there are no support samples.
Verain Veats and Veest. We focus on three learning scenarios.
few-shot learning is similar to supervised learning. However, in few-
NNN as support samples, such that each distinct class is
In-context learning in language. In-context learning mainly refers to the ability of the model to learn tasks given only a few examples in the form of demonstration (Dong et al. (023). For the language model, this mainly achieved by the prompting mechanism. The pretrained L.M model takes the demonstration input, which is provided by the prompt text C. Then, the answer to the task given by generating the rest of the sentence conditioned on C' (Brown et al. 2020).
3 ONE-FOR-ALLL TOWARDS FOUNDATION MODEL ON GRAPH
The proposed OFA is a general graph learning framework that uses one model to simultaneously solve classification tasks varying in formats and backgrounds, similar to LLMs that can answer
substantially different questions using the same model weight. Figure I illustrates the pipeline of OFA. OFA can be divided into three parts. First, graphs from different domains are integrated into
3

---

Published as a conference paper at ICLR 2024
text-attributed graphs with the same format, allowing single LLM to embed all TAGs into the same space. In the second part, OFA unifies different task types in the graph domain by introducing the ledes-of-Innteess (NOI) subgraph and NOI prompt node, where graph model can attend to task- relevant information automatically. Finally, OFA proposes the Graph Prompting Paradigm (GPP) that organically injects task information into the graph data, enabling in-context learning.
3.1 UNIFYING GRAPH DATA FROM DIFFERENT DOMAINS WITH TAGS
One critical challenge in building foundation model for graphs is that cross-domain graph data are usually generated by entirely different procedures and have node/edge attributes embedded in different spaces. This makes graph models trained on one domain almost impossible to generalize to another domain. However, despite the distinct attributes across datasets, almost all can be described by numan-interpretable language. For example, molecular graphs where nodes represent atoms, plain text describe the node with atomic features, including element names, chirality, etc. The key advantage is that by using text to describe nodes and edges, we can apply an LLM to encode different graph attributes into the same space. Consequently, introduce the concept of TAGs to integrate graph data from different domains systematically.
Specifically, we design standardized format for text feature generation of any nodes and edges in graph data. The text feature format for nodes is shown below:
Text feature of nodes: Feature node. <feature description>: <feature content>; <feature
description>: <feature content>;...
Example: Feature node. Atom: Carbon, Atomic number helix chirality, is not in a ring.
Example: Feature node. Paper title and abstract: Attention is all you need. The dominant sequence transduction models are
Given TAG G. the text feature Su, always starts with the text Feature node. to indicate that this node is an input node with features from the original graph as opposed to prompt nodes, which will be introduced in Section 3.3 Next, the text describes the type of a feature, followed by the content of the feature. If there are multiple features for a node, they are joined by semicolons. The construction of the text feature Se, for edge eij is similar, except the start of the text is Feature edge.
Text feature of edges: Feature edge. <feature description>: <feature content>>; <feature description>: <feature content>;...
Example: Feature edge. Chemical Bond: ionic bonding, is conjugated
Example: Feature edge. Citation from one paper to another.
Following the protocol, we meticulously collected nine graph datasets widely recognized bench- marks in numerous downstream tasks. This collection encompasses graph data from various do- mains, including citation networks, molecular graphs, knowledge graphs, and more. Additionally, it covers nearly all classification tasks employed in the research community, i.e., node classification, link prediction, and graph classification. We provide the detailed summarization of all the collected datasets in OFA in T and detailed collection and processing protocol in Appendix B.
As mentioned above, we can apply LLM encoder encode all text features into fixed-length vector as the final input feature all nodes/edges. Namely, for node o; and edge eij. their vector representations are defined n. (s.;) and æty L.LM(se.,). Because the M.-M-encoded input features contain domain information, the subsequent pipeline can capture and leverage this information. Generally, any kind of LLM can be used as the encoder, and stronger LLM po- tentially yields better overall performance. In OFA, we evaluate and compare the performance of different LLMs (further discussion in Section 5). We also provide a visualization of all generated OFA datasets in Appendix B.2
3.2 UNIFYING DIFFERENT GRAPH TASKS WITH NODES TFRES-OF-TTE -N-ENSERSST
Downstream classification tasks in the graph domain be divided into different categories like: (1) node-level tasks, where the task is to classify node in the graph; (2) link-level tasks, where the task is reason about the connection between node pair; (3) graph-level tasks, where the task is to
4

---

Published as a conference paper at ICLR 2024
make prediction the whole graph. However, tasks at different levels need to be handled by distinct procedures and methods, which makes the construction of a foundation model for graphs difficult. In contrast, different downstream tasks in language share the same autoregressive generation nature, which makes the knowledge learned from the next-token prediction task used in LLMs uniformly beneficial to various downstream tasks. Then the question arises: Can we unify different graph tasks into a single task to facilitate the training and knowledge transferring in the graph domain?
In OFA, we propose Nodes-of-Interee (NOI) subgraph and NOI prompt node to achieve the goal. The term NOI refers to the set of target nodes in a task, illustrated by the blue nodes in Figure 2 and is represented as T. NOI not limited to the listed levels of tasks, and its size depends on the prediction target. An NOI subgraph is defined as the subgraph around the NOI. Denote (v) (V%, Eh R as the h-hop ego-subgraphs around u, consisting of hi-hop neighbor nodes of v and all interconnecting edges. A NOI subgraph Gv, combines ego-subgraphs of all nodes in NOI,
(T) = U (s,(t))))))))))))))))))))( (x) I Dhh TET
U E U R VET
EET
VET
For node-level tasks on node v, NOI is the node itself, so T= = {u} and (T) = SR For link- level tasks on node pair (v;, Ug), we have T = (u;, 1iy) } and 9n (1,, e2}) = US, For graph-level tasks, NOI contains all nodes in the graph, and the NOI subgraph is ()) = (V,E, R).
Then, we define the NOI prompt node to unify the processing and readout procedures in different task types. The NOI prompt node is associated with a task prompt text:
Text feature of the NOI prompt node: Prompt node. <task description>. Example: Prompt node. Graph classification on molecule properties. Example: Prompt node. Node classification on the literature category of the paper.
The text is encoded by the same LLM as other text in G. The NOI prompt node connects to all nodes in NOI, as illustrated by double concentric circles in Figure 2 Through message passing, the NOI prompl node summarizes information in the NOI and the task description. We can then attach class nodes to the NOI prompt node for downstream tasks, which we will explain further in Section 3.3 While concurrent works also utilize subgraphs to unify different types of tasks (Sun) al. 2023 Liu et al.) 2023c), these approaches mainly leverage the subgraph concept to transform tasks into a graph-level task, without a NOI prompt node design. In contrast, with the NOI prompt node, we do not require any explicit pooling mechanism, distinguishing our method from previous ones. The combination of NOI subgraph and NOI prompt nodes in our design achieves a unified readout and treatment for all node-level, link-level, and graph-level tasks. Further, the NOI prompt node connected to NOI can be viewed as labeling trick, which uplifts the expressive power of the original graph model to better learn structural information around the NOI (Zhang et al. [02T]. Moreover, the task prompt text on the NOI prompt node allows the graph model to adjust the readout parameters according to the specific task, which is not feasible in existing works.
3.3 GRAPH PROMPTING PARADIGM FOR GRAPH IN-CONTEXT LEARNING
One of the most fascinating properties of LLMs is their ability of in-context learning through prompting, which allows the model to perform various downstream tasks in different learning sce- narios without fine-tuning. For example, in few-shot scenario where the goal to predict the category of a paper based on its title and abstract, we can provide LLMs with k papers from each category as context and instruct the model generate predictions based on the provided context. However, research on performing in-context learning for graphs remains relatively uncharted.
We recognize that the core principle of in-context learning involves manipulating the input data to align it with downstream tasks. Hence, we propose the Graph Prompting Paradigm (GPP) to manipulate the input graph so that the graph model can acquire task-relevant information from the input itself. Such paradigm endows the graph model with in-context learning ability for both seen and unseen classes, enabling zero-shot learning. Concretely, a prompt graph, denoted as P = (Vp Ep, Ry.) has two types of nodes. The first node type is the NOI prompt node, which we have introduced in section 3.2 Suppose we are querying a target NOI subgraph (T()) = (vh cg. R,), and the NOI prompt node is P,- GPP adds edges between the NOI prompt node and every node in NOI, illustrated by the dotted line in Figure 2
We denote them by 84 Egroes
5

---

Published as a conference paper at ICLR 2024
Supervised & Zero-shot Scenarios
Few-shot Scenario
of Interest (NOI)
Query NOI Graph
NOI Graph
NOT Prompt Node
Class Node
(a) Node-level task
(b) Link-lovel task
(c) Graph-level task
p2t t2p Edge
q2c c2q
Support NOI Graphs
Feature Edge
s2c Edge
Edge
Figure In-context learning design in OFA
{tt, Tt2p; Pa), (Pq. Tp2r,t) E T Note that Tr2p and Tp2: are the relation types for edges from NOI to the NOI prompt node and the reverse edges, respectively. The second node type in the prompt graph is called the class node. Each class node holds text information related to specific class.
Text feature of class node: Prompt node. <class description>. Example: Prompt node. Molecule property. The molecule is effective in: Example: Prompt node. Literature Category. es. (Artificial Intelligence). Covers all areas of Al except Vision...
Denote the class node for class by G. We add edges between every class node and the NOI prompt node as illustrated by the gray lines in Figure P denoted as: Equery {(P Ty2e: (C,, Te2g: Py)li [N]], where N is the number of classes, Tg2n and Tr2g specify the edge relation type from the NOI prompt node to class node and the reverse. Overall, the prompt graph P (Vp, Ep, Rpp) given by:
V4P = {p&} Utelie [N]] Equery U R2 = {repp. T22t. Tape: Te24
(2)
Then, the prompted graph fed to the subsequent graph model is the combination of the input graph and prompt graph, denoted as G., (V/ Uvp E E R URR.). We use graph learning model to process the prompted graph and use the embeddings of the class nodes to make binary classifi- cation. Specifically, Ie. be the vector representation class node e; from the graph learning model. We predict the likelihood of the NOI belonging class i by
PINOI belongs class (MLL((he))
(3)
where MLP is a Multi-layer Perceptron whose 1-dimensional output represents the classification score of he. Note that because the NOI prompt node and class nodes connect to the NOI and contain task text description, the fixed-length vector Ite contains information about both the input graph and task, making the prediction task-dependent. While existing graph learning methods need to use different pooling mechanisms for different tasks, this formulation turns different levels of tasks into the same binary classification tasks on class nodes so all tasks can be trained together. For multi-class problems, we compare the prediction scores of different classes to make the decision.
= argmax, (MLP(h.,)l [N]),
is the predicted class of the NOL. Apart from the generality on task levels, because the graph model is oblivious to the class node order and can inductively infer the class based on the class text description on the class node, the users can attach arbitrary unseen class nodes with proper text descriptions to the NOI prompt node. In such cases, the model can predict the unseen class nodes based on its experience with seen class nodes whose text description semantically close to the unseen ones, facilitating zero-shot learning.

---

Published as a conference paper at ICLR 2024
The GPP can also prompt few-shot problems, where support NOI subgraphs of unseen classes provided to help classify the query NOI subgraph better. The support NOI subgraphs are denoted by S (Tx) for the i-th class and k-th support sample and the NOI T belong to class i. As for the query NOI prompt node, we connect each support NOI subgraph to corresponding support NOI prompt node Pi,k by
U [E[M]A[[K]
gikk cross
U
{(1, (rap,,,, Pi,k), ( 1)1 E Th)-
(5)
cross
(E[N),EE[K]
Then, to augment classification, we connect the support NOI prompt node to the class node that its NOI belongs to, as illustrated by the few-shot section in Figure [ That is, Exupp {PpA,. Ts2e: G]]i E [N]
[K]]. Because the relation type re2c differs from r22e between the query NOI prompt node and class nodes, the model can differentiate information from query and support NOI. The overall components of the few-shot prompt graph P are
V/P (p Ulraali [NT.K IKBUUGH E (N]),
Ep = E cross Uqquery W USsupps R2 = (rrap; ppat, Tq2es Fe2q; [s2c]-
The prompted graph can be constructed in a similar way discussed above. Like in the afore- mentioned scenarios, the output embeddings of class nodes are used to make binary classifications. OFA utilizes few-shot support examples by connecting the support NOI prompt nodes to the corre- sponding class nodes, and the model synthesizes both exemplary information and the task semantic information on the class nodes for more accurate predictions. Note that the few-shot class node rep- resentations are still consistent with that in zero-shot scenarios, so they can also be trained together.
To summarize, NOI represents the set of nodes related to the task, and the extracted NOI subgraph includes the neighborhood information of NOI. Then, the NOI prompt node summarizes information in the NOI by a graph learning model because all NOI nodes connect to the NOI prompt node. The NOI prompt node is later connected to a set of class nodes with text descriptions. After graph model processing, class node representations contain class information, task information, and NOI infor- mation, which can be used to make predictions independently, just like prompted input to LLM contains the input, target, and task description. The implementation details and training procedure of the model can be found in Appendix
4 RELATED WORKS
The success of the LLM and prompt learning has enlightened many recent works that try to incor- porate similar ideas to graphs. The first line of research tries to design prompt learning in the graph domain. Both VNT Tan et al. 2023) and GraphPrompt (Liu et al. 2023c Yu et al. 2023) intro- duce trainable prompt vectors to extract related information for different downstream tasks. HG- PROMPT YYu al. 2024a) further extends GraphPrompt to heterogeneous graphs. Prodigy (Huang) ct al. 2023) converts classification tasks to link prediction problems on prompt graphs, facilitat- ing in-context graph learning. All-in-one (Sun et al. 2023) proposes to learn prompt graphs from downstream tasks. Our work also uses prompting to unify all classification tasks. More importantly, unlike existing work that still needs to train separate GNNs in different domains,
leverage lan- guage models to unify different tasks further and use one GNN to solve all tasks. The second line of research combines language models with GNNs. Some works directly apply LLMS solve graph problems by describing the graph using natural language and feeding text description to LLMs. including InstrucIGLM (Ye al. 2023 GraphText (Zhao et al. 2023b), NLGraph (Wang et al.) 2023) and GPT4Graph (Guo et al. 2023 However, these methods also describe graph connections by texts, losing important structural features, while our method explicitly utilizes the information through GNNs. Very recently, GraphGPT (Tang et al.) (203)) introduced a new method to encode the graph structure information with trainable vectors and an alignment module. More related works can be found in Appendix A
5 EXPERIMENTS
The experiment section assesses OFA's potential to serve as a graph foundation model by answering the following questions: Q1: How does replacing the raw node/edge features with text features
7

---

Published as a conference paper at ICLR 2024
from LLM affect GNN performance? Q2: Using text as features for all graphs, is single OFA GNN versatile to tasks in all domains? Q3: What is the effect of different LLMs? Q4: Is the proposed graph prompting paradigm effective in in-context learning? By answering these questions, we validate the approach of using TAGs and OFA's ability to solve various tasks, demonstrating the strong potential of using OFA unified graph foundation model. More experiment and training details can be found in Appendix D
Table 2: Results on supervised learning (first).
Cora Link AUC t
Core
PuhMed
PubMed|
oghn-arxiv Node
Wiki-CS Node
HIV Graph
Task type Metric
Node
Link
Node Acc f
Acc f
AUC
Acc f
Acc
AUC
GCN
90,40++0.0
78.86=1.48
91.10=:0.50
74.49=00.9 74.09+0.17
79.07+0.10
75.49=1.63
GAT
93.70=00.10
82.76=0.79
91.20=+0.10
75.24
74.07=+0.10
79.63+10.0
74.45=1.53
OFA-ind-st
91.87
75.61
10.87
98.50.10.06
73.87
15.79.10.1
77.72.1065
73.42.1114
OFA-st
94.04-10.49 75.90 1.26
98,21
75.54
75.54.10.11
78.34:10.5
78.0240.17
OFA-e5
92.83=+0.38 72.20=3.24
98.45
77.91=144
75.88++0117
73,02++1.66
78.29=1.48
7b
94.22:+0048
73.21=0.73
98.69110.0
77.80=2.60
77.48:1011
77.75++0.4
74.45
OFA UFA-Ilama2-133 13b
94.53++0.51
74.76=1.22
98.59:10.10
78.25 :0.71
77.51+1177
77.65+00.2
76.71#1.19
Table 3: Results on supervised learning (second).
pubmadiit
Wh2018
Chertix
WNISRR
FB1SK237
PCBA
Task type Metric
Link Acc f Ace f
Link
Graph
APR f
GCN GIN
67.40=240
14.20=1110
20.20+0.24
57.30-340
10.70+1.80
22.66+#022
OFA-ind-st
97.22-0.18
95.77=-0.11
22.73++0:2
OFA-st
96.91
95.54
24.8310.10
OFA-e5
97.84-035
95.27++0.8
25.19=0.33
OFA-Ilama2-7b
98.08_0.16
95.56110.05
21.35
HEA-Ilama2-11 13b
98.14:1.25 95.69++0.07 21.54+1.25
Figure Output embedding space of NOI
prompt nodes on all datascts for OFA-joint-st.
5.1
CROSS-DOMAIN SUPERVISED LEARNING
To answer Q1-Q3, we conduct experiments on the supervised learning scenario using all collected OFA datasets with different L.LMs. Specifically, we select four popular LLMSs for evaluation, includ- ing sentence transformer (Reimers & Gurevych 2019), e5-large-v2 (Wang et al., 2022a). Llama2- 7b, and 1lama2- 13b (Touvron et al. (023). Then, we evaluate OFA in two different settings. The first setting trains and tests the model on each dataset independently with text embedding generated by the sentence transformer, denoted as OFA-ind-st. The second setting trains a single model using all datasets jointly. We denote the joint model utilized different LLMs OFA-st, OFA-e5, OFA- llama2-76 7b, and OFA-llama2-1 13b respectively. For baseline methods, we use GCN (Kipf & Welling 2017). GAT /Veličković et al. 2018), and GIN [Xu et al. 2018 for fair comparison. The results can be found in Table|2land Table 3
From the results, have the following observations: (1) Both the independent and joint training achieve comparable or better results on all datasets compared to baseline methods. (2) OFA suc- cessfully enabled a single graph model be effective on all graph datasets across different domains as the joint version with all different LLMs performs well on all datasets. Further, we can see that the joint version OFA-st st achieves better results on most of the datasets compared OFA-ind-st. This may indicate that by leveraging the text feature and GPP, the knowledge learned from one domain/dataset can be useful for the learning of other domains/datasets. (3) The comparison of different LLMS is interesting, generally speaking, larger LLM can achieve better and more stable performance in joint training. We also observe a faster convergence for larger LLM (Llama2- 13b). However, the margin is less significant. Mcanwhile, different LLMs seem specialized for different domains. For example, Llama2 achieves better performance in citation networks but e5-lerge-v2 achieves great results in molecular datasets. To further investigate the mechanism behind the OFA,
'The split we use differs from original one, see Apeendix D for details.
8

---

Published as a conference paper at ICLR 2024
take the output embedding of NOI prompt nodes from OFA-joint-st for each dataset and project it to two-dimensional space. As shown in Figure [3] node embeddings from different domains are separated. This demonstrates that the OFA model can represent data from different domains in dif- ferent sub-spaces to process it. In Appendix we conduct additional ablation studies to verify the effectiveness of the proposed GPP.
5.2 FEW-SHOT AND HERO-SHOT LEARNING
To answer Q4, we design few-shot and zero-shot scenarios for all levels of tasks. We consider both transductive and transfer situations. In the transductive setting, the task is classify unseen classes on the same training graph. In the transfer setting, both the test graph and test classes are unseen. For simplicity, all experiments are conducted using OFA datasets generated from the sentence transformer. We train one few-shot model on various tasks varying N NWway and k-shot, where N > 2 and i O. We include ogbn-arxiv, FB15K237, and Chemble as training sets, then evaluate two transductive settings: ogbn-arxiv and FB FB15K237, and four transfer settings: Cora, WNISRR, HIV, and PCBA. We present node/link/graph level results in Table // respectively. The model is denoted by OFA-joint-Ir (low-resource).
Table 4: Few-shot and Zero-shot results (Acc) on ogbn-arxiv and Cora (Node-level).
# Way
ogbn-arxiv-5-way (Transductive)
Cora-2-way (Transfer)
Task
5-shot
3-shot
1-shot
O-shot
5-shot
1-shot
O-shot
GPN
50.5313.07 48.3213.80
18.58
2..83
56.0912.08
TENT
60.8317.45 56.03318.90
10.72
58.9712.40 54.33.1.11
GLITTER
56.00+4440
57.44
47.12+2.73
TLP-BGRL
50.13
46.21
38.81
81.31
59.1642.44
TLP-SURGL
77.89+6.46 74.19+7.55
61.75+11000
92.49+1.02
81.52++..09
Prodigy 61.0945.85 58.6445.84
48. 48.23+6.18
OFA-joint-Ir 61.45.2256 59.78.1251 50.2014.27 46.1 76.1014441 67.4411447 $6.92
For node-level tasks, we compare with meta-learning models (Ding et al., 2020 Wang et al, 2022c(b) and graph contrastive learning model (Tan et al., 2022). For graph-level tasks, L.M-- based models Galactica-1.33 (Taylor et al. (202)) and GIMLET (Zhao et al. (023a) are considered. Prodigy Huang et al. (2023) follows different setting, where the model trained on the MAG240M or Wiki datasets (Hu et al. 202T) and transferred to the corresponding tasks. OFA exhibits compa- rable or better performance than most existing works on few-shot tasks. Especially in the transfer setting of node tasks, where all baselines are trained and evaluated on the Cora dataset, OFA still shows comparable performance without any prior knowledge about the test dataset, illustrating its capability to generalize. Furthermore, our proposed GPP endows OFA with the ability to address zero-shot scenarios task generally impossible for most existing baseline models. OFA utilizes one single model to address all oowrreouuce tasks across domains, demonstrating the ability of in-context learning.
Table 5: Few-shot and Zero-shot results (Acc) on FB15K237 and WN18RR (Link-level).
Table 6: Few-shol and Zero-shot results (AUC) HIV and PCBA (Gruph-level & Transfer setting).
FB15K237-20-w (Transductive)
WNISRR-S-way (Transfer)
Way
HIV-2-way
PCBA-2-way
Way
Task
5-shot
0-shot
S-shot
0-shot
Task
5-shot
0-shot
S-shot
0-shot
Galactica- 1.3B
33.85
52.02
Prodigy
74.92+6.05
GIMLET
66.24
62.11
OFA-joint-Ir 12.56.11.58 70.20...40 46.32 14.18 30.96.15.46
OFA-joint-Ir
63.582181
35.67:4446
$1.532994 60.62255.5
CONCLUSIONS, LIMITATIONS AND FUTURE RESEARCH
In this work, we propose OFA, the first solution towards building the foundation GNN model for learning on graphs. By showing great results on supervised, few-shot, and zero-shot scenarios, OFA reveals great potential the future foundation model on the graph. Currently, OFA falls short of learning regression tasks and the cross-domane datasets are limited, we leave this to future work. More discussion can be found in Appendix F
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
Hao Liu, Jiarui Feng. Lecheng Kong, and Yixin Chen are supported by NSF grant CBE-2225809. Muhan Zhang is supported by the National Key R&D Program of China (2022ZD0160303) and National Natural Science Foundation of China (62276003).
REFERENCES
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gille- spic, Karan Goel. Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsumori Hashimoto, Peter Henderson, John Hewitt, Daniel Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. E. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts. Aditi Raghu- nathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Para- suram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang. Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2108 1bs/21008,7258, 2021. URL https: /api. exemanticscholass... org/ Corpus ND:
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., ftwes//aaeeeeeeeee222222 pdd
2020. URL httpps://ppoceeddnggss hww.rooceeednggg....ii........ nwwrips.e.... cc/psp/..pppppeeeeeeeeeeeee httpa//pppppeeeee/aeeeeeeeeeeeee 4
Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang. Dawei Yin, Wenqi Fan, Hui Liu, and Jiliang Tang. Exploring the potential of large language models (Ams) in learning on graphs, 2023.
Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021. URL
net wow.ee/eooooounnddeeee
Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fi Fu Yu, Jiong Zhang, Olgica Milenkovic, and Inderjit S Dhillon. Node feature extraction by self-supervised multi-scale neighborhood prediction. In International Conference Learning Representations, 2022. URL https: //openeeview.nee www.:/ee//ooommieee.m KJggid-kJgggH
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
10

---

Published as a conference paper at ICLR 2024
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, 2016. BRLhhtts:///aae
org/ 0tbba//606.0937
Tim Dettmers, Minervini Pasquale, Stenetorp Pontus, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the 32th AAAI Conference on Artificial Intelli- gence, pp. 1811-18188 February 2018. URL https: /wwpvv.aivvvv...///ava/.a...a..a org/ 2007
Kaize Ding, Jianling Wang, Jundong Li, Kai Shu, Chenghao Liu, and Huan Liu. Graph proto- typical networks for few-shot learning on attributed networks. In Proceedings of the 29th ACM International Conference on Information Knowledge Management, pp. 295-304, 2020.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. A survey on in-context learning, 2023.
Vijay Prakash Dwivedi, Chaitanya Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint #xXiv:2003.0998 2020.
Jiarui Feng, Yixin Chen, Fuhai Li, Anindya Sarkar, and Muhan Zhang. How pow- erful are k-hop message passing graph neural networks. In S. Koyejo, S. Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 4776-4790. Curran Associates, Inc., mew.r neurips cc htspa//ssssppppppparel/s////e//e http///eeeeee/a//22/222 e Teeecc (w1555988999999 0ece://cc2220eeeeeee0eeaaaaaeeee
2022. URL.
Jiarui Feng, Lecheng Kong, Hao Liu, Dacheng Tao, Fuhai Li, Muhan Zhang, and Yixin Chen. Extending the design space of graph neural networks by rethinking folklore weisfeiler-lhhman. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 9029-9064. Curran Associates, Inc., files//aperr/0233 pdf
2023. URL http:::/pprocedddsss.eerre wwwrpps.....erssroo..... cc/ wwppeper.ppppp.pppppppp
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):D1 100-D1107 107, 2012.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp- 1263-1272. PMIR, 2017.
Jiayan Guo, Lun Du Hengyu Liu, Mengyu Zhou, Xinyi Hc, and Shi Han. Gpt4graph: Can large language models understand graph structured data an empirical evaluation and benchmarking. 2023.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pp. 1025-1035, 2017.
Mingguo He, Zhewei Wei, Zengfeng Huang, and Hongteng Xu. Bernnet: Learning arbitrary graph spectral filters via bernstein approximation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL httts::/
net wtr///ormm//i wwwoorummddiiiinn/. Gq
Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi. Explanations as features: Llm- based features for aext-attributed graphs, 2023.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00653 2020.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Isc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.094:: 2021.
11

---

Published as a conference paper at ICLR 2024
Qian Huang, Hongyu Ren, Peng Chen, Gregor Kržmanc, Daniel Zeng, Percy Liang, and Jure Leskovec. Prodigy: Enabling in-context learning over graphs. arXiv preprint 1rXi: 2023.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017. URL https:
//openzeview.c htt.:eew.nnnt/rrrrnoosssssaaaa
Lecheng Kong, Yixin Chen, and Muhan Zhang. Geodesic graph neural network for efficient graph representation learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://
Lecheng Kong, Jiarui Feng. Hao Liu, Dacheng Tao, Yixin Chen, and Muhan Zhang- Mag-gnn: Reinforcement learning boosted graph neural network. In A. Oh, T. Neu- mann, A. Globerson, K. Saenko, M. Hardt, and Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 12000-12021. Curran Associates, Inc., http::/ppccoooeeeeeee..eeeeeeees File/ File/ 211122788-.022220000006622222222
2023. URL
Hao Liu, Jiarui Feng, Lecheng Kong, Dacheng Tao, Yixin Chen, and Muhan Zhang. Graph con- trastive learning meets graph meta learning: A unified method for few-shot node tasks. arXiv preprint arXiv:2309.10:: 10376, 2023a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys, 55(9):1-35, 2023b.
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. Proceedings of the ACM Web Conference 2023, 2023c.
Kevin Lu, Aditya Grover, P. Abbeel, and Igor Mordatch. Pretrained transformers as universal com- putation engines. ArXiv, abs/2103.05247, 2021. URL http::ss//pp...mmmattt.moo
org/CorpusID: 2321 232168936 68936
Péter Mernyei and Cätälina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural net- works. arXiv preprint arXiv:2007.00 200000007,02901, 2020.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, pp. 4602-4609, 2019.
Chen Qian, Huayi Tang, Zhirui Yang. Hong Liang, and Yong Liu. Can large language models empower molecular property prediction?, 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-
networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL /
Atg/abs/19t0 10084
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In The Seman- fic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings, pp. 593-607, Berlin, Heidelherg, 2018. Springer-Verlag. ISBN 978-3- 319-93416-7, doi: 10. 000000077777731199-4888 978 3- 319 93417-44 38
38. URL htt.p::://c...rr//////
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: /8/ 105868, 2018
12

---

Published as a conference paper at ICLR 2024
Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All in one: Multi-task prompting for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge
Discovery and Data Mining, KDD '23, pp. 2120-2131, New York, NY, USA, 2023. Associa- tion for Computing Machinery. ISBN 9798400701030. doi: 10.1145/358030553599222 URL.
https://doi.org 110.0..111415550000 3599256
Zhen Tan, Song Wang, Kaize Ding, Jundong Li, and Huan Liu. Transductive linear probing: A novel framework for few-shot node classification. arXiv preprint arXiv:2212.05606 2022.
Zhen Tan, Ruocheng Guo, Kaize Ding, and Huan Liu. Virtual node tuning for few-shot node classification. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining, KDD '23, pp. 2177-2188, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701030.
doi: 10.1145/358033553555555 https:///ol. www.:...rrriiie 1tt.://111111111111 1145/35800000 35595995555559955
URI.
Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models, 2023.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.00055, 2022.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Workshop on Continuous Vector Space Models and their Compositionality, 2015.
URL httpp:://appp semant eschol ar orgg Corpus ID: 5378837
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull. David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie. Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. International Conference on Learning Representations, 2018. URL httttss://pennevvie... net. httpppprrrrrreeeeeeeeeeoaaaaaaaa wtw.orummmddro..poooooo
Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language?, 2023.
Liang Wang. Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Ma- jumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:22/2.035: 2022a.
Song Wang, Chen Chen, and Jundong Li. Graph few-shot learning with task-specific structures. Advances in Neural Information Processing Systems, 35:38925-3888 -89366 2022b.
Song Wang. Kaize Ding, Chuxu Zhang, Chen Chen, and Jundong Li. Task-adaptive few-shot node classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1910-1919, 2022c.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In Interna- tional Conference on Learning Representations, 2022. URL forum? d=wfd/G0o//G
net/
13

---

Published as a conference paper at ICLR 2024
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9(2):513 2018.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2018.
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning. pp. 40-48. PMLR, 2016.
Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all graph needs, 2023.
Jiaxuan You, Jonathan M Gomes-Selman Rex Ying, and Jure Leskovec. Identity-aware graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 10737-10745, 2021.
Xingtong Yu, Zhenghao Liu, Yuan Fang, Zemin Liu,, Sihong Chen, and Xinming Zhang. Generalized graph prompt: Toward a unification of pre-training and downstream tasks graphs, 2023.
Xingtong Yu, Yuan Fang, Zemin Liu, and Xinming Zhang. Hgprompt: Bridging homogeneous and heterogeneous graphs for few-shot prompt learning, 2024a.
Xingtong Yu, Chang Zhou, Yuan Fang, and Xinming Zhang. Multigprompt for multi-task pre- training and prompting on graphs, 2024b.
Bohang Zhang, Guhao Feng, Yiheng Du, Di He, and Liwei Wang. A complete expressiveness hierarchy for subgraph gnns via subgraph weisfeiler-lehman tests, 2023a.
Muhan Zhang and Yixin Chen. Link prediction based graph neural networks. In Advances in Neural Information Processing Systems, pp. 5165-5175, 2018.
Muhan Zhang and Pan Li. Nested graph neural networks. Advances in Neural Information Process- ing Systems, 34:15734-15747, 2021.
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: theory of using graph neural networks for multi-node representation learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
ew net
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. The Eleventh International Conference on Learning Representations, 2023b. httphhtts://ppeeeeee httprpaeeeenneeeeeeooooorrooaaae
jUHKr
Haiteng Zhao, Shengchao Liu, Chang Ma, Hannan Xu, Jie Fu, Zhi-Hong Deng, Lingpeng Kong, and
Qi Liu. Gimlet: A unified graph-text model for instruction-bart molecule zero-shot learning, 2023a.
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space, 2023b.
Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford ford net- works: general graph neural network framework for link prediction. Advances in Neural Infor- mation Processing Systems, 34, 2021.
14

---

Published as a conference paper at ICLR 2024
A RELATED WORKS (EXTENDED)
Large Language Model and Prompt Learning. Since the ground-breaking advances in Large Language Models, including GPT (Brown et al. 2020 |Wei et al. (022) and LLaMa (Touvron et al.) 2023), tremendous attention has been drawn to developing and using them. One particular approach prompt learning. By prompting the LLMs according to carefully designed paradigm, users can uncover LLM's surprising capability in many difficult tasks. shain-of-thoughte by providing step- by-step reasoning examples, greatly improve LLMs' reasoning power. Considerable efforts were also made to use prior semantic knowledge learned by LLMS to perform zero-shot and few-shot tasks (Reynolds & McDonell, [02T] Liu et al.] 2023b] Chowdhery et al. (022). The key advantage of prompting LLMs is that no further fine-tuning is required for the model to adapt to a new class of tasks. Innovated by this practical advantage, several works adapted the prompting idea to the
GNN domain. VNT (Tan et al., (023) offers a new method for few-shot node classification (FSNC) by integrating trainable virtual node embeddings into the original graph, which serves as a form of prompting. For each new FSNC task, the pre-trained graph transformer model can be frozen and only the embedding of virtual nodes needs be trained. GraphPrompt (Liu et al., 2023c Yu al.) 2023) also introduces trainable prompt vectors to extract related information for the downstream tasks. Mcanwhile, it further utilizes the subgraph and link prediction design a novel unsuper- vised pertaining method, which unifies the pertaining and downstream tasks. MultiGPrompt Yu jtt al. (2024b) proposes a multi-task pre-training and prompting framework, effectively harnessing diverse pretext task knowledge for enhanced graph-based analysis. Prodigy |Huang et al. (2023) con- verts classification tasks to link prediction problems on prompt graphs, facilitating in-context graph learning. All-in-one Sun et al. (003)) proposes to learn prompt graphs from data. Our work also uses prompting to umify all classification tasks. However, our works differ in two parts. First, unlike existing work that still needs to train separate GNNs in different domains, we leverage language models to unify different tasks further and use one GNN to solve all tasks. Second, most of the existing works cannot perform zero-shot learning as their prompt module needs to be re-trained for different downstream tasks. instead, OFA directly describes all tasks in a unified way, which allows the model to perform zero-shot learning without further training.
Graph Neural Networks. Recently, extensive efforts have been spent on using GNNs to learn rela- tional data. Earlier GNN variants, including GCN (Kipf & Welling 2017 GAT (Veličković al. 2018). and GraphSage (Hamilton et al. 2017), achieved great success in solving graph learning problems. Later works unify the GNNs into the Message Passing Neural Network (Gimmer et al. 2017) and show the expressivity upper-bound of such framework (Xu et al. 2018 Morris et al., 2019, which demonstrates the inherent difference between learning on graphs and learning on other data formats such as images and languages where expressivity is not an issue. Following such ob- servation, subsequent works, such as SEAL (Zhang & Chen 2018) and ID-GNN (You et al. 202I). propose subgraph GNNs that apply GNNs to subgraphs and aggregate subgraph representations to enhance the expressivity. Moreover, unlike MPNN, which fits the full graph into the GPU memory, subgraph GNNs process the data by subgraphs, solving the scalability problem of MPNN. Innovated
by subgraph GNNs, our work and concurrent works (Huang et al.] 2023; Sun et al.] 2023) also add prompts to subgraphs of interest, which allows our models to perform cross-level tasks.
The applications of GNN have also become prevalent. Much research focuses on the molecular property prediction domain and has achieved promising performance (Zhang al.) [023a Feng
et al., 2023] Kong et al.) 2023: Feng et al., (022). GNNs have also become one of the primary
tools in citation network analysis and mining (Kipi & Welling 2017, Chien al., 2022). Many
efforts have also been made to adapt GNNs to the Knowledge Graph mining domain and achieved
great success due GNN's efficiency and inductive learning capabilities (Zhu et al. 2021 Kong et al. 2022). While the promising results of these applications demonstrate the potential of GNNs, a critical problem is that they are tailored to the specified application. On the contrary, our work, by three carefully designed components, unifies all tasks into one GNN framework that allows large- scale training and inference across different domains. The proposed framework can be a foundation model in the graph learning domain.
Language Models and Graph Neural Networks. The surge of foundational Large Language Mod- els inspires several directions combining language models with GNNs. One of them directly applies LLMs to solve graph problems. They propose graph descriptive languages to represent the structural information as prompt texts and feed them to L.Mss, like InstructGI.M (Ye et al. 2023), Graph-
15

---

Published as a conference paper at ICLR 2024
Text (Zha et al. 2023b NLGraph (aang et al. 2023), and GPT4Graph (Guo et al. 2023 Some works for molecular graph classification propose to only use SMILE molecule sequences to describe the molecular graph in imput to the LLM, including LMMOL QQian et al. 2023) and GIMLET (Zhao et al. 2023a). The exceptional power of L.LMs enables these methods to perform difficult tasks such zero/few-shot learning. However, such graph representation is implicit, and the LLM might not correctly capture the structural information but only summarize texts corresponding to the nodes that appear the prompt. Such approaches will fall short in applications where graph structures are important.
Very recently, GraphGPT Tang et al. 2023) introduced new method to en- code the graph structure information with trainable vectors and an align- concatenates the trainable vectors along with textual input to LLms to improve the struc- ture reasoning ability of LLMs. An- other direction uses LLMs to encode the corresponding texts of nodes,
Table 7: A comparison between OFA and related methods.
In-coniext Zero-shol Crosstasks Cross GNN
LMMOL
GIMLET
ment module.
InstructGLM
GraphText
NLGmaph
GPT4GRAPH
ExpAsFear
VNT
LLMFFeeGraph
such as paper abstracts, and apply GNN the graph with encoded text embeddings (He et al.) [2023] Chen
PRODIGY
GraphPrompi
All-in One
OFA
al. 2023). The high-quality text
representation from LLMs allows these models to achieve better performance. Our approach adopts the same idea of using text embedding represent graph entities (nodes and edges). However, we are the first work that drives the language model to full power by systematically unifying graph entities in different domains with the same language protocol and representing them the same embedding space. This consequently grants cross-domain functionality that other GNN frameworks lack. Table / summarizes the characteristics of different models. We can see that OFA is the most versatile model among existing works.
B
MORE ON OFA DATASET
B. DATASET COLLECTION AND CONSTRUCTION
In this section, we discuss the detailed processing procedure for each dataset collected in OFA.
B.1.1 CORA
Cora is a citation network that contains papers and their citation relationship in the computer science domain. The raw text data the Cora dataset was collected from the GitHub repository provided in Chen et al. (223). Each node in Cora represents research paper from the computer science domain. The raw text feature of node is the title and abstract of the respective paper. Every edge in the Cora dataset indicates the citation relationship between papers. Each node's label corresponds to the category of the paper. Tasks that can be executed on Cora include predicting the paper's 's category (node-level) identifying missing citation links within the graph (link-level), Using the proposed processing protocol, we reformat all text features in Cora. Particularly, for each node category, we further use gpt-3.5 W.t-tttbboChhaaaHH.. to generate description as additional information. table we show processed example on Cora dataset.
1.2 PUBMED
Cora is citation network that contains papers and their citation relationship in the biomedical domain. The raw text data of the PubMed dataset was collected from the GitHub repository pro- vided in Chen et al. (2023). All nodes and edges are similar to Cora dataset and we use the exact same processing procedure as Cora dataset. Because its original literature categories are diabetes, experiinental/diaaetees type 1/diabetes, and type 2, which are overly simple and very difficult for our BERT-based LLM to distinguish, we asked ChatGPT to generate a detailed description of each category.
16

---

Published as a conference paper at ICLR 2024
Table 8: processed example Cora/Pubnedlogbn-ankii dataset.
Node type
Text feature
Input graph node
feature node. literature category and description: <lille>.
<abstract>
Input graph edge
feature edge. co-citation
(Cora/Pubmed)
Input graph edge (OGBN-ARXIV)
feature edge. citation
Node classification task
Class node
prompt node. <description>
literature category and description:
<ategorr name>>
Prompt node
prompt node. node classification of literature category.
Link prediction task
Class node
prompt node, two papers (do not have/have) co-citation
Prompt node
prompt node. link prediction on the papers that are cited together
Table 9: A processed example on Wiki-CS dataset.
Node type
Text feature
Input graph node
feature node. wikipedia entry name: <enfry name>. content>
Entry content:
<entry
Input graph edge
feature edge.
wikipedia page link.
Node classification task
Class node
prompt node. wikipedia entry category: <category name">
Prompt node
prompt node. node classification wikipedia entry category.
B.1.3 OGBN-ARXIV
Cora is citation network that contains papers and their citation relationship collected from Arxiv platform. The raw text data the ogbn-arxiv was collected using the same protocol as the GitHub repository provided in Prodigy (Huang et al.) 2023). For ogbn-arxiv, we only evaluate the model on the node classification task. All nodes and edges are similar to Cora dataset and we use the exact same processing procedure Cora dataset except that the description for each category is directly obtained from the Prodigy.
B.1.4 WIKI-CS
Wiki-CS is an Internet link network with each node representing a Wikipedia page and each edge representing the reference link. The raw text of the Wiki-CS dataset was collected from the official website (Mernyci & Cangea 2020). The raw text feature of node is the name and content of an entry in Wikipedia. Each node's label corresponds the category of the entry. We evaluate Wiki-CS on node classification tasks. In table we show a processed example on the Wiki-CS dataset using the processing protocol proposed in OFA.
B.1.5 FB15K237
FB15K237 a knowledge graph that contains knowledge base relation triples and textual mentions of Freebase entity pairs The raw text data of nodes in FB15K237 was collected from GitHub repository The raw text feature node is the name of the relation entity and its description. The
17

---

Published as a conference paper at ICLR 2024
Table 10: processed example FB15K237 dataset.
Node type
Text feature
Input graph node
feature node. entity and entity description: <entity name>. <enlity descrip- lion>
Input graph edge
feature edge.
relation between entities: <relation name>
Link prediction task
Class node Prompt node
prompt node. relation between two entities: < relation name > prompt node, relation type prediction between the connected entities.
Table 11: A processed example on Molecule datasets.
Node type
Text feature
Input graph node
feature node, atom. <element name>, <atom chirality>, degree of <atom degree>, formal charge of <formal charge>, num of hydrogen is <number of hydrogen>, num radical electron <number of radical electronss, hy- bridization is <hybridization>, (is/is not) aromatic, (is/is not) in ring.
Input graph edge
feature edge. chemical bond. <bond type> bond, bond stereo is <bond stereo>, (is/is not) conjugated
Node classification task
Class node
prompt node. molecule property description. <molecule property descrip- tion. e.g. the molecule is effective to the following assay....
Prompt node
prompt node. Graph classification molecule properties.
raw text feature of an edge is the type of relation between two entities. We provide an example of processed data example in Table 10
B. WNI8RR
WN18RR is knowledge graph, which is subset of WordNet that consists of 18 relations and 40943 entities. The raw text data nodes in WN18RR was collected from GitHub repository The raw text feature of nodes and edges are the same as FB15K237 and follow the same process protocol to process the WN18RR dataset.
B.1.7 MOLECULAR DATASETS
We adopted three molecular datasets: (1) ChEMBL dataset Gaulton et al. (012) is a widely used molecule property prediction dataset. It contains 1,310 prediction target labels of molecules from biological assays for drug discovery. (2) MOLPCBA dataset Wu et al. (2018) is subset of the BioChem BioAssay dataset consisting of 128 labels on the biological activities of small molecules. (3) MOLHIV dataset Wu et al. (2018) contains over 40,000 compounds labeled for their ability to inhibit HIV replication. The raw data of these datasets are represented in SMILE string format. We use RDKit to construct molecule objects and graph structures from the SMILE string and use natural language to describe the atoms (nodes) and bonds (edges) as shown in Table For the class nodes text, we use the assay/molecule properties description in Zhao et al. (202a))
B.2 VISUAALIZATION OF OFA DATASET
In this section, we provide a visualization of all generated OFA datasets using the sentence trans- former. Concretely, For each generated dataset, we randomly select 400 node embeddings and project it to 2 dimensions using TSNE. Note for molecular datasets, we include all node embed- dings. The result is shown in Figure
18

---

Published as a conference paper at ICLR 2024
sakmed
card
Clardie
We can that the generated embeddings from different domains are successfully separated by the LLM as the em- bedding from molecular datasets, knowledge graphs, wiki pages, and citation networks are well separated in the visu- alization. Moreover, the LLM can even separate the cita- tion network from different research domains. embeddings from Arxiv and Cora, which mainly contain papers from computer science are close to each other and far from the embeddings of Pubmed, which focus on biology. This re- sult reveals one of the key rationales behind the success of the OFA. That is, by encoding the datasets from different domains into different sub-spaces, OFA allows one GNN to learn the information from different domains separately without impacting each other.
10238227
stempdee
Figure 4: Embedded node features from all OFA datasets (sentence trans- former).
C IMPLEMENTATION OF OFA
The last section described how OFA combines LLMs, text-attributed graphs, and the graph- prompting paradigm to facilitate cross-domain and in-context learning. In this section, illustrate the training and evaluation procedure of OFA.
Training an OFA pipeline for different tasks is straightforward with the unified task representation introduced in Section because all the inputs are standardized to the prompted text attributed graph Im regardless of the task type. The task information is carried out in the prompt nodes and edges. Recall that OFA first embeds all texts in graphs to vector representations using L.M::
Zíj LLM (se.,) (se,, ), Vey E Em. 2; = LLM (s,,), Vu, E Vm-
(7)
Then, GNN processes Gm along with the embedded texts through multiple message-passingg layers and gets the final node embedding for each node in Gm. Formally,
h+t1
Wwathh +
DEX
Σ
(Re.LU + +x,))),
(8)
MER (vE) [TT((:)
where h4 24, W sell and W, are trainable transformation matrix for self-loop and relations, (1; is the direct neighbors that connects to with relation type Our GNN model is a natural extension of R-GCN (Schlichtkrull et al. 2018). incorporating edge features. Such an implemen- tation helps differentiate nodes in the input graph and the prompt graph. It also effectively extracts useful information from the input graph and encodes it to the class node the prompt graph for prediction. Usually, the last layer output can be used for prediction. However, due to the different natures of tasks and the well-known over-smoothing effect, a GNN with a fixed number of layers will not perform well on all tasks. Hence, we design simple attention layer to summarize all layer outputs from h! to h Eventually, the node representation is
hy HH -Soflmaa((WW)) H)) War:'', H-
where We and Wg 9 are key and query trainable weights. Note that because the text features z; also include the domain information, by querying on the feature, a properly trained attention layer can choose the most important layers to aggregate outputs based on the domain. Lastly, we gather the output node embedding for each class node (he, Vi [N]} and use an MI. prediction head to perform binary classification every class node. Even for multi-class problems, each class node connects to the NOI prompt node and hence can integrate the information other connected class nodes through pesssag-ppasingg While each class node is processed separately by the MLP, such binary classification is still conditioned on other candidate classes and not individualized. Moreover, since the format of Gm. is consistent in different learning scenarios suupevised/fewwsshol. and task types (node/link/graph- these situations. Formally, the likelihood of class i is:
such a procedure can work without any modification across
P (MLP(he,)),
(10)
where a is the Sigmoid function. If 's's multi-class problem, OFA collects all classes' probabilities
as
1p argmax ((pel Vi (N])),
(11)
19

---

Published as a conference paper at ICLR 2024
and 1p is the final prediction from the model. The binary training loss for single Im can be expressed as:
N
L
Σ Vi log(p:) + yi) (1 log(p;))
(12)
N
where N is the number of candidate classes in 9m.
D EXPERIMENTAL SETTINGS
In this section, we provide detailed settings for all experiments shown in the paper. The code of OFA can be found at the Github link httppp///pi.uub..mm///eeeeeeette
D.1 SUPERVISED LEARNING EXPERIMENTS
We set the number of layers for GNN independent training (OFA-ind-st) and joint training to be 6 and 7 respectively, as the joint model might require larger embedding space. The dropout rate is set to 051 We set the hidden size of the GNN as 768 and project the initial embedding generated by different LLMs 768 before the GNN. We train OFA-ind-st for 100 epochs and 50 for OFA-st and 0FA-e5. For OFA-llama2-77 7b and OFA-llama2-136 we only train 40 epochs due to the limitation of computing resources. For OFA-ind-st, we evaluate the model after each epoch and take the model with the best validation performance for testing. For all versions of joint training, since there is no easy way to define the best validation performance, we directly use the final model for testing. Because datasets vary in sample size, smaller sets might be neglected during training. Hence, we sample from each dataset by multiple of the dataset size form the training set and control the weight of each dataset. Data are resampled after every epoch. The multipliers are Cora-link: 1.5, Cora-node 2, Pubmed-link 0.5, Pubmed-nodes 2.5, ogbn-arxiv: 0.7, WN18RR: 0.6, FB15K237: 0.4, WikiCS: 2, ChEMBL: 1, PCBA: 2, HIV: 4. For independent and joint training, repeat experiments 10 and times, respectively, and report the mean results and standard deviation. The final training set contains all training sets of the collected datasets, and the test sets are the datasets' original test sets.
For Cora-node, Pubmed-node, and ogbn-arxiv datasets, different split is used. Specifically, for Cora-node and Pubmed-node, the split is obtained from Chen et al. (002)). where they split data each experiment,
into 10 folds and we the first fold randomly split data with a train/val/test ratio of 0.8/0. For Cora-node, Pubmed-node, ogbn- arxiv, WNISRR, and FB15K237, we rerun the baseline models 10 times and report the average performance. For other datasets, we report results from existing works.
split. For ogbn-arxiv,
will
D.2 FEW-SHOT AND ZERO-SHOT LEARNING EXPERIMENTS
For OFA joint training low resource (OFA-joint-Ir) experiments, we set the number of layers to be 5, the learning as 0.0001, and the number of epochs to be 30. We split the label of ogbn-arxiv dataset with ratio [20, 10, 10] for train/vall caan//alidation/testt and use the [142, 47, 48) as the split ratio for FB15K237 dataset. During training, we use the training set from Arxiv (node-level), the training set from FB 75132237 (link-level), and the whole Chemble dataset (graph-level) to train a single model for all downstream tasks. We construct diverse N-way k-shot tasks from the training sets. For ogbn- arxiv dataset, N varies from 3 to 5, k varies from 0 to 5; for FB FB15K237 dataset, N varies from 3 to 10, & varies from 0 to 5; for ChEMBL dataset, tasks are 2-way, and & varies from 0 10.
In few-shot graph construction, the text feature of class nodes differs significantly from that in supervised and zero-shot scenarios. Specifically, we omit category information and utilize uniform text feature across all class nodes. This uniformity shifts the focus from learning class-specific information to enhancing the model's ability to compare the query node with support nodes. Such comparisons crucial for determining matches, which proves particularly effective in few-shot scenarios where the model must classify unseen classes.
During the test, we involved six datasets: for the transductive setting, evaluate the test set from arbn-arxiv dataset and test set from FB15K237 dataset; for the transfer setting, we evaluate on Cora (node-level). WN18RR llink-level), MOLPCBA and MOLHIV (graph-level). Note that for
20

---

Published as a conference paper at ICLR 2024
the transductive setting, we keep the same train/validation/esst labels for OFA and all the baselines to ensure a fair comparison. For baseline results on the Cora dataset, we report the performance provided by COLA (Liu et al. 2023a), which is the average performance of 20 random label splits. For baseline results of Prodigy, we follow the provided code to pre-train separate models for differ- ent shot numbers, that is, we pre-train 5-shot/3-shot/1 -shot models for node-level tasks and pre-train 5-shoo/3-shot/--hhot models for link-level tasks, then evaluate different shot settings using the cor- responding pre-trained model. For the baselines of graph-level tasks, we report the results from GIMLET (Zhao et al. 2023a).
Table 12: Ablation study different prompting design.
Joint
Separate
Full
Class node
Full
Class node
gebn-arxiv
75.23-++0.03
75.19=0.10
75.06=+0.08
75.39+
HIV
75.92:+0:45
71.60=20.54
0.38
75.43+t0.50
Cora-node
75.48++0.29
71.39
75.72++0.49
76, 12-+0877
Cora-link
92.27:1084
89.51
93.16::0955
92.80+0.22
Table 13: Few-shot results (Acc) ogbn-arxiv (Node-level).
Way
S-way
3-way
Task
5-shot
1-shot
U-sbot
3-sbot
U-shor
GPN
50.53:23.07
48.32-3.80
38.5811.61
62.25:14.94
58.52=3.00
48.45::.600
TENT
60.83.7.45
56.03-8.900
10.72::10.70
74.20:19.93
11.48::1.500
13.58
GLITTER
56.00:24.40 57.4424.00
47. 47.1232
62.13= 10:85
60.93 12.12
59
59,20:15448
BGRL
50.13:18.78
46.21:17.22
35.81
62.93 11.74
18.37.111.4
10.80
SURGL
77.8916.46
74.19.7..55
60.75.1..00
86.27.17.54
83.75
18886
73.46.1
Prodigy
61.09:55.85
58.64=5.84
48.2316.18
73.64:16.93
71.4327.28
61.59:8.553
OHA-joint-Ir
61.45:12.56
59.78:12.51
50,20-14.77
46.19:3..83
73.22:42.65
72.24
60.6013.71
58.87:13.77
OFA-ind-Ir
59.92:11.32 58.68:16.00
$2.80
46.56.10.82
72.18:13.33
1.55
60.47:1..55
64.13:10.98
Table 14: Few-shot results (Acc) on FB15K237 (Link-level).
Way
20-way
10-way
Task
5-shot
3-shot
1-shot
0-shot
5-shot
3-shot
1-shot
O-shot
Prodigy
74.92-6.03
70.32++6.0
55.49
5.11
84,30+7.80 80
79.61
66. 1049.89
10.02:6646
OFA-joint-Ir
82.56=1.58 81.33++2.44
75.39=2.86
70.20=2.40
90.44:
89.68=1.38
86.38+1.60
70.84+2.31
OFA-ind-Ir
87.43=1.56 86,42+2245 83.87=2.88
72.24 +3.45
93.44
92.4440.84
19.16=1.11 82.08++081
Table 15: Few-shot results (Acc) on WN18RR (Link-level).
# Way
10-way
5-way
Task
5-shot
3-shot
1-shot
O-shot
5-shot
3-shot
1-shot
O-shot
OFA-joint-Ir
1..4 28.46+183
26.24-1.96 19.9843.26 46.3214.18
44.20
33.86-3.41
30.9645..4
OFA-ind-Ir 1.56 30.56.1.02 25.82 20.40.1.86 48.3213.19 45.04.1239 34.40 1.47 38.24.1176
E
MORE EXPERIMENTAL RESULTS
This section includes more experimental results that provide detailed evaluation of OFA's in- context learning ability and justify the OFA design by ablation study.
21

---

Published as a conference paper at ICLR 2024
E. ABLATION STUDY
While using LLM to unify graph and task representation is a unique design in OFA, there are alter- natives our prompting paradigm GPP to make classifications. One alternative to discard graph prompting completely but keep track of the NOT for each NOI subgraph. After being processed by the graph model, the embeddings of the NOI are summarized by average pooling to form the final NOI representation, the LLM encoded tasks representation is concatenated to the NOI for binary classification. We denote this as "-Class node". We perform case study comparing the two meth- ods hiv, ogbn-arxiv, Cora-node, and Cora-link datasets when these datasets are trained jointly using the same model and separately. The results are presented in Table [2]
We observe that, if the datasets since in end-to-end training for one dataset the prompting design is essentially a pooling mechanism. However, it is striking that the "-Class node" approach's performance significantly drops when the datasets are trained together, while the OFA prompting approach maintains the original performance. Intuitively, when datasets from different domains are trained together, the model needs learn which domain a particular data is from and which tasks are being performed to make appropriate predictions. However, without the NOI prompt node that carries task text descriptions, the "-Class node" approaches can confound tasks in different domains.
trained separately, all methods achieve similar performance,
E.2 FEW-SHOT AND ZERO-SHOT RESULTS
To explore the in-context learning ability of OFA, we train a single model for all few-shot and zero- shot low-resource tasks denoted as OFA-joint-Ir. Here, we provide more comprehensive experiment results spanning more ways and shots. We also implement three experiments that train separately on node-, link-, and graph-level tasks denoted as OFA-ind-Ir. For node-level tasks, we train the model using the ogbn-arxiv dataset. For link-level tasks, we train the model on the FB15K237 dataset. For graph-level tasks, we train the model on Chemble.
Results for the ogbn-arxiv and Cora can be found in Table 13 and Table 16 respectively. We can see that for the Cora dataset, OFA-joint-Ir achieve better performance than OFA -rd-Ir in all setting. This may indicate that the knowledge learned from other task levels like link- or graph-level can help the generalization of the model on Cora. For ogbn-arxiv dataset, the results for OFA-joint-Ir and OFA-ind-Ir are similar.
The results for FB15K237 and WNI8RR datasets can be found in Table 14 and Table 15 respec- tively. For link-level task, the OFA-ind-Ir have better results than OFA-joint-Ir, This may indicate that the knowledge learned from graph node level is not helpful for link- tasks.
Finally, the results for HIV and PCBA datasets can be found in Table 17 and Table [8] respectively. We can notice that joint training benefit the prediction of HIV in most cases and the zero-shot scenario of PCBA, but the performance of few-shot tasks of PCBA dropped significantly. One reason might be the different number of tasks used for training: the graph-related tasks involved in individual training are much more than those in joint training.
F LIMITATIONS AND FUTURE WORKS
While OFA aims provide a solution for the general graph foundation model, it not yet able to perform regression tasks, because regression targets can be unbounded in values. Hence, if the range of all target values a zero-shot regression task falls out of the range that OFA trained on, it is very
Table 16: Few-shot results (Acc) on Cora (Node-level).
Way
5-way
2-way
Task
5-shot
1-shot
0-shot
5-shot
1-shot
0-shot
Oonntoint-Ir 48.7642.65
34.0424.10 28.7249.0
76. 1044.11
67.444447
56.92+3.09
OFA-ind-Ir
42.2842.35 31.28 13.68
72.20+3.82 62.22+1.17 51.85
22

---

Published as a conference paper at ICLR 2024
Table 17: Results (AUC)
HIV and 2-way tasks.
Task
10-shot
5-shot
3-shot
1-shot
O-shot
OFA-joint-Ir 1..2 63.58
63.99
59.39
35.67+4446
OFA-ind-Ir 54.3624.90 57.564366
59.30+304
1..17=1.22 51.1644..9
Table 18: Results (AUC) on PCBA and 2-way tasks
Task
10-shot
5-shot
3-shot
I-shot
O-shot
OFA-joint-Ir 51.64
51.53
51.58
51.7218.57
60.62.15.45
OFA-ind-Ir 54.58 54.80 13.75
54.67 14.35
54.92.14.88 52.71
difficult for OFA to predict the correct target value in that range, which is why focus general classification in this work. A potential approach is to specify a target range the NOI prompt node task description, and let the model predict regression value based the specified range. However, reasoning about math concepts is difficult even for most advanced LLMs, and certainly unreliable for our current adopted LLMs. Hence, leave such approach to future work.
While OFA is already trained in several different domains and the performance is on par or even outperforms
GNN and LLM approaches, the training data for the graph foundation model is still scarce compared to that of LLMs. Also, LLMs explore training techniques beyond supervised training, including auto-regressive training, and contrastive learning, which largely improve LLM's ability to model data and generalize to unseen tasks. Other unsupervised training techniques are possible, like the one proposed GraphPrompt (Liu al., (003cc.. We believe these training tech- niques can further enhance the performance of OFA and regard this as an important direction to explore in the future.
23


Reviews:
[{"review_id": "lHmfE7v1Zt", "reviewer": "Reviewer_LMu3", "summary": "This paper proposes a unified paradigm for learning different tasks over graphs and across different domains. Notably, the proposed algorithms achieve competitive performance in various tasks.", "questions": "Please refer to the weakness for questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The framework proposed by the author is very interesting. The idea of prompted nodes naturally unifies the three major tasks in one simple but powerful problem formulation. \n2. The presentation of the result is clear with both quantitative and qualitative results.", "weaknesses": "1. The soundness of the experiments seems lacking. Based on my understanding, LLM is one if the key component to achieve cross-task and cross-domain generalization. However, the choice of LLM is rarely discussed. I didn't find detailed description of the LLM used in the main experiments. Neither a comparison between different LLMs is missing in main paper and appendix. \n2. The modeling of link-level task lack details. For example, do you have two or one class node?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWri1qqDk1", "reviewer": "Reviewer_RxxK", "summary": "In the field of artificial intelligence, creating a single model to handle diverse tasks has been a longstanding goal. Large language models have excelled in language-related tasks, but applying this versatility to graph-based tasks is challenging. Graph data from different domains have unique attributes and distributions, making uniform representation difficult. Additionally, graph tasks encompass nodes, links, and graphs, necessitating distinct strategies. Furthermore, context-aware learning for graphs lacks a clear method.\n\nTo address these challenges, this paper introduced \"One for All\" (OFA), a framework that uses a single graph model to conquer these obstacles. OFA uses text-attributed graphs, unifying diverse graph data and standardizing task representation with \"nodes-of-interest.\" It pioneers a novel graph prompting approach for versatile task handling without fine-tuning. They train OFA using data from various domains and evaluate its performance in diverse learning scenarios. The results establish OFA as the first general-purpose graph classification model, revolutionizing the field of graph-based artificial intelligence.", "questions": "see W1-4\n\nI would like to see the rebuttal to the questions mentioned in the above section “Paper Weakness”. I’m afraid that I might have not sufficient time to see a very long rebuttal. A concise and clear one would be good.\n\nThe potential weakness won't prevent me from raising my final score. I just want to make clear whether my understanding is correct.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation of this paper is well-delivered and interesting\n\nThe proposed method is novel and seems to be good according to their experimental report", "weaknesses": "W1. translating graph data with text feature sequences might lose some key information from graphs. \n\nW2. According to Figure 1, LLM enhances text for text-attributed graphs. it still needs a GNN as a predictor. (1) I wonder whether the GNN model is also pre-trained and frozen. and (2) what are the results compared with using LLM as a predictor instead of an enhancer?\n\nW3. It is unclear how to use NOI prompt nodes and class nodes with GNN to predict the downstream tasks. according to Figure 2, can I say that the downstream tasks (node, edge, graph classifications) are treated as predicting links between NOI prompt and class nodes? It seems that NOI prompt nodes can be treated as a special case of the prompt graph mentioned in the paper \"All in One\" (Sun et al., 2023), what's the differences between them? and why the authors use only one NOI prompt node instead of multiple NOI prompt nodes.\n\nW4. I wonder why the authors use sentence transformer instead of ChatGPT API, or LLAMA, etc since they claim that \"any kind of LLM can be used as the encoder, and a stronger LLM potentially yields better overall performance.\" Is it possible that a very large language model contains too much unrelated knowledge/intelligence that may reduce task performance?\n\nW5. I wonder what would happen if you Re-order the item in your prompt for ChatGPT.  It seems the reason why you use sentence transformer is that it is not sensitive to the order of your prompt? if you change to ChatGPT, different orders may generate entirely different sentences, making the downstream performance unpredictable", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBmCCx0L56", "reviewer": "Reviewer_A8Qx", "summary": "This paper proposes One-for-All (OFA), a framework for building and training a single graph model that can perform various graph-related tasks across different domains. The experimental results on real-world datasets demonstrate the effectiveness and efficiency of the proposed framework.", "questions": "see weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tA unified graph model is designed for various tasks and different domains of graph data.\n\n2.\tThe introduced model can encode features of different graphs into the same embedding space.\n\n3.\tExtensive experiments are conducted on various settings and datasets from different domains.", "weaknesses": "1.\tThe design of prompt nodes and utilizing NOI-subgraph to unify different tasks seem to be similar to several previous works. For example,  [1] introduced a learnable prompt vector to unify tasks, and [2-3] proposed to utilize super nodes to perform pooling readout. The proposed NOI prompt node appears to be a combination of these two lines of works. Could you include more in-depth discussions and comparisons with these related works.\n\n2.\tComparison with methodologies that integrate LLMs and graph models, such as [4-7], is necessary. \n\n3.\tThe Few-shot and Zero-shot results, especially for TLP-SURGL and TLP-BGRL, do not outperform baseline models. Could you include further discussions and analysis of these underwhelming performance results? \n\n4. The ability of the LLMs to unify the feature space across datasets from diverse domains is very interesting. It would be insightful to see the unified feature representations of datasets within the same domain (e.g., two citation networks) and the datasets spanning different domains (e.g., a citation network and a molecular network). For example, some visual analysis will offer insights into the effectiveness of the LLM in harmonizing feature spaces across heterogeneous datasets.\n\n5. Additional ablation studies can be conducted to see whether the constructed graph structure is helpful. For example, directly using the unified node features (produced by LLMs) as the input feature of some backbone GNN models like GIN, GCN, GAT.\n\n6. The article introduces a method to employ LLMs to generate inputs for GNNs. Another line of works aims to utilize GNNs to generate inputs for LLMs, and it seems that this approach can better utilized LLMs' significant capabilities in addressing various problems on vast data. I'm curious about the difference between these two pipelines. Can you provide some explanations and compare the performance of these two pipelines.\n\n\n\n[1] Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In WWW, 2023.\n\n[2] F. Hu, Y. Zhu, S. Wu, and T. Tan. Hierarchical graph convolutional networks for semi-supervised node classification. In IJCAI, 2019.\n\n[3] Matthias Fey, Jan-Gin Yuen and Frank Weichert. Hierarchical Inter-Message Passing for Learning on Molecular Graphs. In ICML, 2020.\n\n[4] Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi. Explanations as features: Llm-based features for text-attributed graphs. arXiv preprint arXiv:2305.19523, 2023.\n\n[5] Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. Learning on large-scale text-attributed graphs via variational inference. arXiv preprint arXiv:2210.14709, 2022.\n\n[6] Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134, 2023.\n\n[7] Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. Graphtext: Graph reasoning in text space. arXiv preprint arXiv:2310.01089, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTkh8jfrbF", "reviewer": "Reviewer_WF3H", "summary": "The paper proposes One for All (OFA), a novel framework that addresses how to unify various graph tasks with various graph data.  OFA uses text-attributed graphs, allowing nodes and edges to be described in natural language, and introduces a new graph prompting method for diverse tasks without fine-tuning. When trained across multiple graph data domains, OFA demonstrates strong performance, making it a pioneering multi-purpose graph classification model.", "questions": "See Weaknesses.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.LLM and graph prompting are both highly effective methods in their respective domains. Combining the two to address the cross-domain TAG problem is both convincing and well-motivated. I highly appreciate the idea of this paper, which is quite enlightening.\n2. This paper is well-articulated.", "weaknesses": "1. A work on graph prompt learning should be cited and discussed.\n* Tan, Zhen, et al. \"Virtual Node Tuning for Few-shot Node Classification.\" arXiv preprint arXiv:2306.06063 (2023).\n2. I highly recommend the authors to make the code public for readers to follow the work.\n3. I suggest exploring non-meta-learning scenario such as in GraphPrompt.\n* Liu, Zemin, et al. \"Graphprompt: Unifying pre-training and downstream tasks for graph neural networks.\" Proceedings of the ACM Web Conference 2023. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)