LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings
Authors: Duo Wang
Yuan Zuo
Fengzhi Li
Junjie Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Graph Neural Networks
Zero Shot Learning
Primary area: graph_neural_networks

Abstract: Zero-shot graph machine learning, especially with graph neural networks (GNNs), has garnered significant interest due to the challenge of scarce labeled data. While methods like self-supervised learning and graph prompt learning have been extensively explored, they often rely on fine-tuning with task-specific labels, limiting their effectiveness in zero-shot scenarios. Inspired by the zero-shot capabilities of instruction-fine-tuned large language models (LLMs), we introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM) that leverages LLMs as cross-dataset and cross-task zero-shot learners for graph machine learning. Concretely, we pretrain a GNN, aligning its representations with token embeddings of an LLM. We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings without tuning the LLM. A unified instruction is designed for various graph tasks at different levels, such as node classification (node-level) and link prediction (edge-level). These design choices collectively enhance our method's effectiveness in zero-shot learning, setting it apart from existing methods. Experiments show that our graph token embeddings help the LLM predictor achieve state-of-the-art performance on unseen datasets and tasks compared to other methods using LLMs as predictors. Our code is available at https://github.com/W-rudder/TEA-GLM.

LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings
Duo Wang
Yuan Zuo" Fengzhi Li Junjie Wu
MIIT Key Laboratory of Data Intelligence and Management, Beihang University (wangduo58, zuoyuan, lifengzhi, wujj) ) @ buaa.edu.en
Abstract
Zero-shot graph machine learning, especially with graph neural networks (GNNs), has garnered significant interest due to the challenge of scarce labeled data. While methods like self-supervised learning and graph prompt learning have been exten- sively explored, they often rely on fine-tuning with task-specific labels, limiting their effectiveness in zero-shot scenarios. Inspired by the zero-shot capabilities of instruction-
senooon--nettttee fine-tuned large language models (LLMs), we introduce a novel frame- work named Token Embedding-Aligee
Graph Language Model (TEA-GLM) that leverages LLMs as cross-dataset and cross-task zero-shot learners for graph machine learning. Concretely, we pretrain GNN, aligning its representations with token embeddings of an LLM. We then train linear projector that transforms the GNN's representations into a fixed number of graph token embeddings without tuning the LLM. A unified instruction is designed for various graph tasks at differ- ent levels, such node classification (node-level) and link prediction (edge-level). These design choices collectively enhance our method's effectiveness in zero-shot learning, setting apart from existing methods. Experiments show that our graph token embeddings help the LLM predictor achieve state-of of-the-art performance on unseen datasets and tasks compared to other methods using LLMs as predictors. Our code is available at https: // thub. cow/W-rodeee Moannder/TEA-GG TEA--GLM
1 Introduction
Graph Neural Networks (GNNs) have emerged pivotal framework in graph machine learning, harnessing the ability to capture intricate cessage-passingg patterns for robust graph representation. These advancements have yielded various GNN architectures, including the Graph Convolution Network (GCN) M. Graph Attention Network (GAT) [2, and GraphSAGE [3] Despite their efficacy, GNNs often exhibit limited generalization capabilities, struggling maintain consistent performance when transitioning across different datasets or downstream tasks 4 This limitation underscores the necessity for more adaptable and universally applicable models in the graph learning domain.
To mitigate the dependency on labeled data and enhance the resilience of graph models, self- supervised learning has been widely adopted in GNN training. Techniques such as Deep Graph Infomax (DGI) 5 and GraphCL. [6] have demonstrated effectiveness by leveraging mutual informa- tion maximization and contrastive learning, respectively. However, these methods typically require fine-tuning task-specific heads for downstream applications, which can be sesource-intensine and limit their practicality in diverse scenarios. Moreover, graph prompt learning enhances GNN generalization by using unified task templates and meta-learning to adapt to various downstream applications 7 g) but it often requires extensive fine-tuning and is constrained by the specificity of task types.
In recent years, the remarkable generalization capabilities of Large Language Models (LLMs) have spurred interest in their potential applications within graph machine learning. Some methods attempt
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to encode graph structures into text for LLM imput 10 I1 [2], but these approaches often lead suboptimal outcomes [3]. Alternatively, using LLMs as enhancers to generate data node text representations [1
has shown promise but remains constrained by the inherent reliance on GNNs for prediction. Recent efforts 19 20 to use L.LMs as predictors have demonstrated potential. However, their performance often remains unstable due to the challenge of producing transferable graph representations that work effectively for LLMs across diverse tasks and datasets.
15
16
17
18
In light of these challenges, we propose a novel framework named Token Embedding- Aligned Graph Language Model (TEA-GLM), Inspired by the zero-shot capabilities of instruction-fin LLMs 211, TEA-GLM leverages LLMs as cross dataset and cross-task zero-shot predictors for graph machine learning. The core idea is to pretrain GNN and align its representations with the token embeddings of an LLM. This alignment enables the GNN to effectively utilize the LLM's pretrained knowledge, allowing to generalize across different datasets and tasks without task-specific fine- tuning. Additionally, we train a linear projector to convert graph representations into a fixed number of token embeddings, which are then incorporated into a unified instruction designed for various graph tasks at different levels. Experiments show TEA-GLM achieves superior performance in zero-shot scenarios and when encountering unseen tasks, offering more generalized and efficient solution for graph zero-shot learning. Our contributions are summarized as follows:
We introduce TEA-GLM, novel framework that aligns GNN representations with LLM token embeddings, enabling cross-dataset and cross-task zero-shot learning for graph machine learning.
We propose linear projector that maps graph representations into a fixed number of graph token embeddings. These embeddings are incorporated into a unified instruction designed for various graph tasks at different levels, enhancing the model's generalization capabilities.
Our extensive experiments demonstrate that methods on unseen datasets and tasks.
significantly outperforms satte-of-theaard
2 Methodology
In this section, introduce TEA-GLM, novel framework designed for cross-dataset and cross-task zero-shot graph machine learning. TEA-GLM consists of two main components: a Graph Neural Network (GNN) to derive node representations from the graph, and Large Language Model (LLM) to perform zero-shot tasks such as node classification and link prediction. Our methodology involves two key stages: enhanced self-supervised learning of the GNN, where feature- wise contrastive learning with LLM''s token embeddings is proposed, and training a linear projector to map graph representations into fixed number of graph token embeddings by designing an instruction that is suitable for various graph tasks at different levels. The framework of our proposed method is illustrated in Fig. [
2.1 Notations
with | = N indicating the total number of nodes and E = fei, e2, f(g) representing the sets of nodes and edges, respectively. The adjacency matrix is denoted as A E R with A.₁ 1 iff (v,, uy) E E. The feature matrix X E Nxxx contains the attribute or feature information associated with each node, where x; E Fv is the feature of e; and FN represents the dimensionality of features.
Formally, a graph is denoted as 9 = (V,E,A,X), where V {U), U2, fldnn )
2.2 Token embeddings-alignee graph self-supervised learning
Given the increasing model sizes and data volumes in recent years, self-supervised learning has become a prominent research focus due to the scarcity of labeled data. In this context, we propose contrastive learning method to obtain more transferable node representations suitable for use with large language models (LLMs). Our approach leverages instance-wise contrastive learning and introduces feature- wise contrastive learning method that maps node representations to the textual embedding space of the LLM.

---

Contrastive learning of GNN
Alignment tuning of projector
Zero-shot tasks
Lerge Lengeage Medel
Indraction
prodicionn
DER
Glvv Clevileiippooetiii ow pipers Stidion
nnverationn
Kapel Note
Tite Goood Paper
Linos Projecior
Instruction
Skow
Quatier and mittbeniii ecate betwers
KE MF
the reprrettation paper dlokke
Bistlaken Tou <Token Kü waihiee
Pabnned
Anchaz
Faaraoo Marts Instaree Controct
Tidx aitl..
Lineer Projector
L'oettive
GNN
tris paper bekkng sot thoo duruitt Savly Fearche felloe origenas
Negative
Maddll
LLM Token Embelkdings
Zroen
Zradza Coeppeants
Grapks Tooe
Tox Tdles
Anewer Natural Language
Figure I: Framework of TEA-GLM
2.2.1 Instance-wise contrastive learning with structural information
To alleviate the need for labeled data and enhance model generalization capability, employ self-supervised learning for pre-training. To better extract structural information from the graph, we follow the work of 22 to generate
views of G, denoted as and G2. for contrastive learning. Specifically, we adopt the Removing Edges (RE) and Masking Node Features (MF) methods to
generate different views. The RE strategy samples a random masking matrix R E {0, N to mask the raw adjacency matrix, computed as:
À A R,
(1)
where denotes the Hadamard product. The MF strategy samples random mask vector m E (0,1)" The generated node features X are computed by:
m:; o :.;;;;; m
(2)
Thus, obtain views of G, denoted as 91 [ A1 and G2 (x2, A2). Then, graph encoder to derive node representations:
U. fCNN (X.,A, NxFO
Where Fy the dimension size of node representations. Here, E (1,2) represents different views of the graph.
We employ contrastive objective distinguish the embeddings of the same node in these two different views from other node embeddings. For node u, its node embedding generated in one view, us treated as the anchor, while the embedding generated in the other view, u,', forms the positive sample. Embeddings of other nodes in the same view are regarded as intra-view negative samples, while embeddings of other nodes in the other view are regarded as inter-view negative samples. The contrastive loss is defined as:
8 (u4, us') = log
8 (us.ue)) //T
Uzgge (usrus)))
the pusitive puir
intra view negative pairs
inter-view negative pairs
{0, 1} is indicator function that equals 1 iff # i, 0 (., is the cosine similarity function, and f is temperature parameter. The loss for the other view similarly defined, and the overall objective Zine is the average of all instances:
where [(a
Lins 2N
Σ
le (ui ui') (us', u:)]
(5)
1=1
3

---

To enhance the scalability of our method for large-scale graphs, we employ the subsampling approach proposed by [ Both the RE and MF methods, along with the loss function described in Equation are seamlessly adaptable to the sampled subgraphs.
2.2.2 Feature-wise contrastive learning with token embeddings
Instance-wiid wise contrastive learning relies heavily on individual instances, which can cause transfer issues when transitioning to other datasets. Moreover, there is significant gap between the obtained node representations and the semantic space of L.LMs. To address these issues, we propose feature- wise contrastive learning with token embeddings.
Feature-wise contrastive loss breaks the independence between instances. For the feature matrix U., we denote the columns in different views as m, E U1 and n, E U2. Here, mi, ni E IRA, The loss denoted Leee and calculated as:
Fb log
Olme,ne//T
fen
Σ
FU
FF 2==1 (mmm/ + (mmm))))/
To map node representations to the semantic space of LLMs, we use the principal components of the token embeddings of LLMs as coordinate axes. This approach ensures that the representations of similar instances are closely aligned in the textual embedding space. This helps alleviate the inconsistency optimization objectives during graph self-supervised learning due to the gap between node representations and the text embedding space.
Specifically, we first use principal component analysis (PCA) obtain the P principal components, denoted as C E PxF,, where Fr, is the dimension size of token embeddings of LLM. Then, we map node representations by:
= U. x CT
To map the node representations obtained from the GNN using principal components, we set the output dimension of the GNN be equal the token embeddings' dimension (i.e., Fu FL). The columns the mapped feature matrix U., denoted as m, and n., are fed into L Therefore, the final contrastive loss for graph self-supervised learning is the average of Equation 4 and Equation 6
L
(Lins + Lfea).
(8)
Remark: The introduction of feature-wise contrastive learning with token embeddings successfully addresses the semantic space discrepancy between graph node representations and LLM token embeddings. Our method enables the direct and simple use of graph structural and text information obtained by GNN in LLMs, thereby avoiding the significant generalization loss associated with complex modality alignment training during the fine-tuning process. Its role in fine- fine-tuning will be further described Sec. 2.3.2 and validated by experiments. Additionally, the feature-wise contrastive method itself exhibits stronger generalization, allowing it to perform well on unseen instances (or tasks) rather than relying on trained instances (or tasks).
2.3 Alignment tuning
The development of LLMs has introduced new paradigm for graph machine learning. However, existing research [13] indicates that LLMs alone cannot fully comprehend graph structures and their underlying information. To enable LLMs to more effectively capture information and improve their performance in cross-dataset and cross-task zero-shot learning, essential to design specific methods for LLMs to incorporate graph information suitably. To this end, we propose an alignment tuning method that includes specially designed instructions for various graph tasks at different levels, as well as a graph representation to graph token embeddings mechanism to integrate graph information.
2.3.1 Instructions design
The instruction we designed can be divided into two parts: one part provides graph information, and the other part describes the task. Here, we take a citation graph as an example, where nodes are papers, and relations are citations, to introduce the instruction.

---

Graph information provision The graph information provision in the instructions for node, edge, and graph-level tasks is presented as follows: Given the representation of a paper/woo papers/a paper set: (graph), with the following information First Puper: {titles) ...v,, where (graph) is the placeholder for graph inputs (see Sect. 2.3.2 and {title:} is the node text information.
Note that, different from most work which use LLM as predictor, the instruction we designed uses only the title of a paper node, excluding more extensive textual information such as its abstract or description. In fact, reducing the amount of input text not only does not decrease the model's performance but actually improves it. [3] confirmed through experiments that LLMs benefit from structural information only when the target node lacks sufficient phrases for reasonable predictions. Therefore, using only titles as text input can help LLMs extract more critical information from graph information. The complete instruction for the tasks of node classification and link prediction in citation networks shown in Appendix D
Task description To achieve cross-dataset capability, where the model can be trained one graph dataset and then perform reasoning on any other dataset, the instruction is designed to include not only the task description itself but also the set of alternative answers. Using the node classification task on the Arxiv dataset (see Sect. (31] as an example, the instruction is structured as follows: Which arXiv CS sub-category does this paper belong to? Please directly give the most likely answer from the following sub-categories; {ans], where {ans} represents the set of alternative answers, which varies across datasets. Including alternative answers enables the model to learn the task of "reasoning the answer from a given set according to the task" rather than memorizing answers for particular dataset, thus facilitating reasoning across datasets.
2.3.2 Graph token embeddings
The token embeddings of graph mentioned previously, i.e., (graph), are crucial for incorporating graph information and enabling the model's generalization. We use a projector to map central node representations into K graph token embeddings and replace (graph) with these tokens. Kindly note that, we map the representations to fixed number of token embeddings regardless of the task type. For example, for node-level tasks, we map the central node representation to K token embeddings; for edge-level tasks, we pool the representations of the two nodes of the target edge and then map this pooled representation to K token embeddings; for graph-level tasks, similar approach can be applied. In this way, we unify the instruction of graph tasks at different levels. Thanks to the text-aligned contrastive learning, linear projector is enough to capture the map relationship without tuning LLM:
Htoken S (u:)
where u4 E U, Htoken RKFF, FL is the dimension size of token embedding of LLM, and flincar (-) is a linear layer.
This approach offers three primary advantages: (i) When handling tasks at different levels, the changes to the instructions are minimal. This consistency facilitates the transfer of knowledge learned during training unseen tasks in large language models (LLMs); (ii) The fixed number of token embeddings can be seen as a conditional soft prompt. Unlike traditional soft prompts, learning at the instance level reduces the risk of overfitting to specific datasets or tasks, thereby enhancing generalization to unseen datasets and tasks; (iii) Different from current work which intends to include the representations of all nodes in the subgraph, we only map the representations of the central node to tokens, since there has enough information carried by message passing of GNN. This method is more efficient, and it offers greater generalizability and practicality.
Remark:
2.3.3 Training and evaluation strategy
To ensure compatibility and facilitate comparisons across various datasets, we map the node features into a consistent vector space. Specifically, we employ a pretrained BERT model 22 encode the raw text associated with each node, thereby generating the node features. We then pretrain the graph model using contrastive learning with the loss function defined in Equation g on single dataset. After pretraining, the model parameters are fixed. We utilize the pretrained model to obtain node representations and follow the instructions in Section Section[2.3.1|g to train the linear projector on specific tasks within the same dataset. Finally, we evaluate the performance of our model on unseen datasets and tasks. Throughout all phases, the parameters of the language model remain fixed. We use GraphSAGE 3 graph encoder and Vicuna-7B-vl. 24 as the foundational language model.
5

---

3
Experimental results
In this section, comprehensive experiments are conducted to validate the effectiveness of TEA-GLM. These experiments aim to investigate the following research questions:
RQ1: How effective is TEA-GLM in handling the cross-dataset zero-shot learning problem?
RQ2:
How well does TEA-GLM transfer knowledge when adapted to an unseen task and dataset in a zero-shot setting?
RQ3: What is the contribution of the feature-wise contrastive learning and graph token embeddings to the zero-shot learning ability of TEA-GLM?
3.1 Experimental setup
Datasets We test TEA-GLM across eight widely used datasets spanning two distinct domains. Within the citation domain, we employ Arxiv [25]. Pubmed (26), and an expanded version of Cora (27 with an increased range of classes and larger scale. these datasets, each node represents an individual paper, with edges indicating citation relationships. In the e-commerce domain, we utilize datasets from the TAG benchmark (28), including Children [Book-Children) History (Book- History), Computer (Ele-Computer), Photo (Ele-Photo), and Sports Fptoess-Fitnes).. Here, nodes represent distinct products, while edges denote co- viewing or co-purchasing between two products. Appendix A presents the statistics for these datasets.
Baselines We conduct a comprehensive comparison of TEA-GLM with various categories of base- line methods: (i) Non-graph neural network approaches, such as MLP, which employs a Multilayer Perceptron for node representation; (ii) Supervised methods, including GCN M. GraphS AGE [.] and GAT [2]: (iii) Self-supervised methods like DGI [5]. which maximizes mutual information to learn node representations without relying on ground truth labels; (iv) Graph knowledge distilla- tion frameworks: GKD [29]. which distills knowledge from a teacher GNN trained on a complete graph to student GNN operating
smaller sparser graph; GLNN [30], method combin- ing the advantages of graph neural networks and MLPs using knowledge distillation, aimed at the inference graph; (v) Graph transformer networks, including Node- Former [31] and DIFFormer [32]: (vi) Large language models, such as Vicuna-BB-vl.5; (vii) The latest models equipped with transfer and zero-shot capabilities, such as OFA [8]. GraphGPT [9
reducing dependency
and LLaGA (20).
Implementation details For datasets within the citation domain, we follow the data split methodol- ogy outlined in GraphGPT [9]. For those within the e-commerce domain, we utilize scripts provided by the TAG benchmark 28 to generate data splits. To ensure comparability among different methods, identical data splits are applied all models. To assess the performance of TEA-GLM. we employ three commonly adopted evaluation metrics: Accuracy and Macro F1 for node classification, and AUC (Area Under the Curve) for link prediction. To ensure result robustness, we conduct five experiments with random seed values ranging from 0 to 4 and report the mean and standard deviation of the results. Due to the limited number of pages, several experimental results, such Macro Fl results of node classification (Appendix B.2), legality rate of valid answers produced by the LLM (Appendix B.I and parameter sensitivity analysis (Appendix C). are reported in Appendix.
In the pre-training phase of the GNN, we set the GNN layers to 2. We use a batch size of 512 for 60 epochs and learning rate of 2 x 10-2, During the training of the linear projector, configure a batch size of 2 per GPU for one epoch, with a learning rate of 1 x 10-3, The Adam optimizer is employed for all approaches. For baseline models, we adjust hyperparameters and utilize the optimal settings. All experiments are conducted on 2 NVIDIA A100 GPUs with 80GB memory each, using CUDA version 11.7.
3.2 Cross-dataset zero-shot ability (RQ1)
We train all methods on the Arxiv and Computer, respectively. followed by an evaluation of their zero-shot performance on datasets from the same domain. Zero-shot learning presents challenges for GNN-based models, particularly regarding variations in the number of classes across different datasets. To address this, we adopt the setting outlined in GraphGPT [19]. For each target dataset, we utilize the GNN backbone trained on the source dataset along with classifier trained with target data,

---

Table 1: shoo accuracy on citation and e-commerce datasets (bold highlights the best result across all methods, while underline highlights the second-best results)
Citation
E-commerce
Model type
Model
Pubmed
Cora
Children
History
Photo
Sports
MLP
0.323=#0.027
0.021+0.006
0.029++0.037
0.080=0.041
0.110=0.070
0.042+000221
GCN
0.288=0.002
0.017=00004
0.030=000188
0.063=0.042
0.103=0.047
0.042:40.025
GraphAAGE
0.316:10.088
0.014110.077
0.008100007
0.19511.2206
0.056:100055
0.051110011
GAT
0.343::00044
0.016:10.044
0.086:10.084
0.172:10.088
0.050=0.027
0.142:10.138
DGI
0.329 0.026.10.009
0.082:10.035
0.218:10.168
0.2241:0127
0.049:10.017
GNN as predictor
GKD
0.3991100033
0.042.10.008
0.202-10.064
0.339110.138
0.166100086
0.208110007
GLNN
0.390++00011
0.031+0.006
0.187+0.012
0.283+0.022
0.403+0.019
0.317+0.048
NodeFormer
0.308
0.016:+0.077
0.048+000028
0.073+00015 0.128
0.165++0,057
DIFFormer
0.361=0.071
0.029=#0.014
0.129=00.030
0.275=0.171
0.521=0.055
0,306+0.131
OFA
0.314+0.059
0.130++0.019
0.064+00.086
0.052+0.049
0.340+00026
0.101+00001
Vicuma-7B-v1.5 1.5
0.71910.010
0.156.10.00
0.270.10.001
0.363.10.001
0.378100004
0.370.10.001
Vicuna-7B-SPT
0.76810.036
0.168:10.018
0.22710.015
0.28110.088
0.35010.061
0.230110011
predictor
I.M as
GraphGPT-std
0.701
0.126
GraphGPT-col
0,521
0,181
LLaGA
0.793-+00036
0.168+0.0032
0.19940.007
0. 0.14660.067
0.276=0.069
0.352+0.033
TEA-GLM
0.848 0.010
0.202-+0.014
0.201+0.0110
1.528+00005
0.497 0.027 0.404+0.010
typically linear layer. Due to the considerable time cost associated with training and evaluating GraphGPT on e-commerce datasets, we only report its performance on citation datasets as provided in their paper. "-std" and "-cot" denote the use of the standard procedure of dual-stage graph instruction tuning and COT instruction datasets generated by LLM, respectively. To demonstrate the difference between work and Soft Prompt Tuning, we fine-tuned vicuna-7b-vl using Soft Prompt and reported the results. The Accuracy results are presented in Table [ As mentioned earlier, we report the Macro Fl results in Appendix B.2 and report results on two training datasets in Appendix B.3
The results clearly demonstrate that TEA-GLM outperforms all state-of-the-art (SOTA) models, resulting in significant improvements. Comparative analysis with baseline models across all datasets highlights the robust generalization capability of S'TEA--LMM Models utilizing GNN as a predictor face challenges in achieving cross- coos-dataset transferability with traditional supervised and self-supervised learning methods. Even recently developed robust GNN-based models, such as NodeFormer, DIF- Former, and GKD, encounter similar issues. In the of OFA, recent framework for cross-domain learning, strong transferability is observed between topic-related datasets such as Arxiv and Cora (both related computer science). Nevertheless, its generalization performance notably decreases on datasets with lower topic relevance, such as those in the e-commerce domain.
LLM-based solutions, such as Vicuna-7B, demonstrate consistent performance across various datasets. Nevertheless, their predictive capabilities are confined to text information alone. Vicuna- Sounaa7B-SPP also fails to achieve transferability e-commerce datasets, indicating that soft prompt tuning alone is insufficient when relying solely on node texts. This suggests that graph tokens indeed contain transferable graph information, enabling the LLM to make more accurate predictions. In contrast, GNN-LLMMccombinet solutions that use LLM as a predictor demonstrate generalization ability but often face limitations. For instance, GraphGPT tends underperform compared to Vicuna-7B, due to the lack of a graph foundation model. Instead of relying on a graph foundation model, LLaGA directly maps node representations without GNN and can generalize citation datasets. However, it demonstrates limited generalization capability across e-commerce datasets, which are more challenging due to highly irrelevant topics. TEA-GLM, on the other hand, utilizes principal components of token embeddings of LLMs to constrain representations learned by GNN, helping the graph representations well transfer other datasets. Experimental results validate the superior generalization capabilities of TEA-GLM, achieved with less textual data and fewer parameters.
3.3 Cross-task zero-shot ability (RQ2)
We employ models trained on node classification tasks directly for link prediction tasks without any fine-tuning. We omit the comparison with models utilizing GNN as predictor, as conducting cross-task evaluation of these models without fine-tuning poses significant challenge, given that different tasks typically correspond to different task heads. Here, we contrast TEA-GLA with OFA, which similarly enables cross-task testing without the need for fine-tuning. Additionally, we compare
7

---

Table 2: AUC of link prediction (Cross-task)
Citation
E-commerce
Model
Arxiv
Pubmed
Cora
Children
History
Computer
Photo
Sports
OFA
0.469
0.481
0.492
0.484
0.431
0,461
0.459
0.517
Vicuna-7B-v1.55
0,513
0,543
0.527
0.500
0.515
0,502
0,501
0.502
Vicuna-7B-SPT
0.537
0,535
0.565
0.544
0.543
0,509
0.501
0.508
GraphGPT-std
0,649
0,501
0,520
LLaGA
0,570
0.569
0.537
0.422
0.449
0,479
0,478
0,597
TEA-GLM
0.657
0.689
0.586
0.571
0.579
0.554
0.545
0.553
0.8
Seen datasets
Unseen datasets
0.7
Unseen task
TEA-GLM
w/o TC
0.5-
0.6
w/oCT
0.4
0.5
02
0.0
Ardv
Compatter
Pubmed
Cota
Chlidren
History
Photo
Sports
04
Ardi
Compater
Pulmed
Com
Chiddre
History
Photo
Sports
(a) Accuracy of node classification
(b) AUC of link prediction
Figure 2: Ablation study results ("Seen datasets" are used to train the GNN and linear projector, while "unseen datasets" are not. "Unseen task" means the model wasn't 't trained for link prediction.)
TEA-GLM with Vicuna-7B and methods that utilize LLM as predictor, such as GraphGPT and LLaGA. For GraphGPT. we utilize the checkpoint released by the author trained on Arxiv and report the results on citation datasets. The results are reported in Table 2
In the case of OFA, although this framework facilitates cross-domain and cross-task learning, exhibits negative transfer when lacking task-relevant data, particularly on unseen tasks. Benefiting from the generalization capability of large language models, both the fine-tuned and non-fine-tume fine-tuned versions of Vicuna do not experience negative transfer. However, due to the absence of graph information, its predictions often appear random. Conversely, GraphGPT shows transferability with familiar datasets, yet its performance declines when dealing with unseen datasets (Pubmed and Cora). Due to the absence of GNN for filtering and aggregating graph information, L.LaGA demonstrates unstable performance. While it exhibits cross-task transferability is poor on most e-commerce datasets. contrast, TEA-GLM consistently outperforms all baseline methods on both unseen datasets and tasks, except for the results on Sports, indicating the stronger generalization ability of TEA-GLM.
citation datasets, its performance
3.4 Ablation study (RQ3)
We conduct an ablation study to discuss two key components of our model: feature-wise contrastive learning and graph token embeddings. Here, we directly remove these two components from our model and then test the model's performance on cross-dataset and cross-task evaluations. The results are shown in Figure 2 "w/o FC" means that we pretrain the GNN without fcature-wise contrastive learning, while "w/o GT" means predicting without graph token embeddings.
Without graph token embeddings, large language models lack crucial information from the graph, leading to significant decline in performance on both node-level and edge-level tasks. GNNs pre-traine pre-trained with feature-wise contrastive learning can obtain node representations aligned with the text space, enabling cross-dataset and cross-task generalization through simple linear layer. When the feature-wise constraint for pre-training absent, the model's performance on the seen datasets (Arxiv and Computer) for the training task improves slightly. However, its performance on unseen datasets declines. Although remains relatively stable when handling tasks of the same category, its performance decreases notably when dealing with unseen tasks (link prediction). These results
8

---

indicate that alignment between graph representation and LLM's token embeddings via feature-wist contrastive learning important for cross-task zero-shot transfer.
Related work
4.1 Graph neural networks
In the field of graph machine learning, Graph Neural Networks (GNNs) have garnered significant
(37 38 39 40). The primary strategy of most GNNs is to capture underlying message-passing patterns for graph representation. Several effective neural network architectures have been proposed. such as Graph Attention Network (GAT) 22. Graph Convolution Network (GCN) [Л) and GraphSAGE []. Recently, there has been a surge of interest in exploring transformer based encoders for graph machine learning 41 42
attention [33 34 33 36
B1 [2] However, notable limitation of GNNs is their generalization capability. Typically, GNNs are trained on specific tasks within particular datasets, and when faced with new datasets or tasks, they often struggle to consistently perform well across different datasets or downstream tasks
4.2 Self-supervised learning and prompt-tuning for GNNs
To alleviate the demand for labeled data and enhance the robustness of graph models, self-supervised learning is commonly employed in GNN training 44
Methods like Deep Graph Info- max (DGI) [5] utilize mutual information maximization for pre-training. Other approaches, such GraphCL (6). GCA (5), GCC [46), and JOAO 47), learn node representations by contrasting positive and negative samples. GraphMAE 48 49 on the other hand, learns representations by generating samples that resemble the original graph structure. However, these methods typically require fine-tuning the task-specific heads for downstream applications.
22 44
Various methods have explored the use of prompt techniques to enhance the generalization of GNNs. To address the inconsistency between pre-training and downstream task objectives, GraphPrompt 77] proposes a unified task template applicable to both stages. Additionally, ProG [8] reformulates various task types into a unified graph-level representation and employs meta-learning techniques to enhance multi-task learning capabilities. However, whether through self- supe-supe learning or graph prompt methods, fine-tuning mee-tuning is often necessary when handling new datasets. Moreover, when confronted with datasets containing varying numbers of categories, retraining of task heads is required to achieve optimal performance.
4.3 Large language models for graphs
With the rapid advancement of Large Language Models (LLMs) and their remarkable generalization capabilities, leveraging LLMs to address transfcrability issues in graph machine learning has garnered
significant attention 10 50 Some methods represent graph structure information as text input
to LLMs (9 11 12]: however, this approach often leads suboptimal solutions [3]. Another paradigm involves using LLMs as enhancers 14
17 18), where they generate data or node text representations. Despite this, since GNNs are ultimately used for prediction, this approach significantly limits the model's transferabilityy Recently, considerable efforts have been made to utilize LLMs as predictors. For instance, GraphGPT (19] attempts to align LLMs with pre-trained Graph Transformer encoders through two-stage fine-tuning. However, the fine-tuning, conducted on specific datasets, might weaken the method's transfcrability. In light of this, LLaGA 20 introduced a novel encoding method that directly translates graph data into sequences compatible with LLMs. However, this approach may compromise performance due to the lack of GNN filtering and aggregation of graph information. Inspired by these challenges, we propose pre-training strategy that enhances GNN transferability by aligning its representations with the token embeddings of LLMs, resulting in improved performance in zero-shot tasks. Notably, similar to our method, TEST [5I] aligns time series representations with several selected LLM token embeddings. However, our approach differs in that we project graph representations into feature space defined by the principal components of L.MM token embeddings. This enables the L.LM function as a zero-shot learner for graph machine learning tasks, rather than just enhancing performance specific, seen tasks.
13
16

---

Limitations
While our TEA--LM framework demonstrates considerable promise in enhancing zero-shot learning for graph-based tasks, it does have some limitations. Although the framework we designed can be easily applied to graph-level tasks, we have not yet explored the model's performance through specific experiments. This will addressed our future work.
6 Conclusion
This paper introduces TEA-GLM, a framework that enhances zero-shot learning in graph machine learning by aligning GNN representations with LLM token embeddings. TEA-GLM uses linear projector to map graph representations into graph token embeddings and incorporates a unified instruction design to handle various graph tasks at different levels. This approach enables consistent performance across various datasets and tasks without task-specific fine-tuning. Extensive experi- ments show that TEA-GLM outperforms state-of-the-art methods in accuracy and generalization, demonstrating its effectiveness and efficiency in zero-shot learning for graph tasks.
7
Acknowledgement
This work was supported by the National Key R&D Program of China (2023YFC3304700) The work of Yuan Zuo supported by the National Natural Science Foundation of China (NSFC) under Grant 71901012. Dr. Junjie Wu's work was partially supported by the National Natural Science Foundation of China (72242101, 72031001) and Outstanding Young Scientist Program of Beijing Universities (IWZ020240021002).
References
[1] Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
[2] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[3] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, 2017.
[4] Mingxuan Ju, Tong Zhao, Qianlong Wen, Wenhao Yu, Neil Shah, Yanfang Ye, and Chuxu Zhang. Multi-task self-supervised graph neural networks enable stronger task generalization. In The Eleventh International Conference on Learning Representations, 2023.
[5] Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations, 2019.
[6] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems, pages 28877-28888 2021.
[7] Zemin Liu, Xingtong Yu, Yuan Fang, and Ximming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, page 417-428, 2023.
[8] Xiangguo Sun, Hong Cheng, Jia Li, Bo Liu, and Jihong Guan. All one: Multi-task prompting for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 2120-2131, 2023.
[9] Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and Jiliang Tang. Exploring the potential of large language models (L.LMs) learning on graph. NewrIPS 2023 Workshop: New Frontiers Graph Learning, 2023.
10

---

[10] Jiayan Guo, Lun Du, and Hengyu Liu. Gpt4graph: Can large language models understand graph structured data ? an empirical evaluation and benchmarking. ArXiv, abs/2305.15066. 2023.
[11] Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[12] Chang Liu and Bo Wu. Evaluating large language models on graphs: Performance insights and comparative analysis. arXiv preprint arXiv: 1224, 2023.
[13] Jin Huang, Xingjian Zhang, Qiaozhu Mei, and Jiaqi Ma. Can Ilms effectively leverage graph structural information: when and why. arXiv preprint 2..iv::230.165999 2023.
[14] Ruosong Ye, Caiqi Zhang, Runhui Wang, Shuyuan Xu, and Yongfeng Zhang. Natural language is all a graph needs. arXiv preprint arXiv:2308.07134 2023.
[15] Jianxiang Yu, Yuxiang Ren, Chenghua Gong, Jiaqi Tan, Xiang Li, and Xuecang Zhang. Em- power text-attributed graphs learning with large language models (Ilms). arXiv preprint VrXiv:2310.09872 2023.
[16] Lianghao Xia, Ben Kao, and Chao Huang. Opengraph: Towards open graph foundation models. arXiv preprint 4rXiv:2403.0112,, 2024.
[17] Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, and Jiliang Tang. Label-free node classification on graphs with large language models (LLMs). In The Twelfth International Conference on Learning Representations, 2024.
[18] Hao Liu, Jiarui Feng, Lecheng Kong. Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference on Learning Representations, 2024.
[19] Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng. Dawei Yin, and Chao Huang. Graphgpt: Graph instruction tuning for large language models. arXiv preprint 3rXiv:2310.J3022 2023.
[20] Runjin Chen, Tong Zhao, Ajay Jaiswal, Neil Shah, and Zhangyang Wang. language and graph assistant. In ICML, 2024.
Llaga:
Large
[21] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[22] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006. 04131, 2020.
[23] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186 2019.
[24] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, 2023.
[25] Weihua Hu, Matthias Fey, Marinka Zitiik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs In Advances in Neural Information Processing Systems, pages 221 18-22133, 2020.
[26] Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. Harnessing explanations: LLM-to-LM interpreter for enhanced aext-attributed graph repre- sentation learning. The Twelfth International Conference on Learning Representations, 2024.
11

---

[27] Zhihao Wen and Yuan Fang, Augmenting low-resource resource text classification with pre-training and prompting. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 506-516, 2023.
[28] Hao Yan, Chaozhuo Li, Ruosong Long, Chao Yan, Jianan Zhao, Wenwen Zhuang, Jun Yin, Peiyan Zhang, Weihao Han, Hao Sun, Weiwei Deng, Qi Zhang, Lichao Sun, Xing Xie, and Senzhang Wang. A comprehensive study on text-attributed graphs: Benchmarking and rethink- ing. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[29] Chenxiao Yang, Qitian Wu, and Junchi Yan. Geometric knowledge distillation: Topology compression for graph neural networks. In Advances in Neural Information Processing Systems, 2022.
[30] Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old MLPs tricks via distillation. In International Conference on Learning Representations, 2022.
[31] Qitian Wu, Wentao Zhao, Zenan Li, David Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[32] Qitian Wu, Chenxiao Yang, Wentao Zhao, Yixuan He, David Wipf, and Junchi Yan. DIFFormer: Scalable (graph) transformers induced by energy constrained diffusion. In The Eleventh International Conference on Learning Representations, 2023.
[33] Jiashun Cheng, Man Li, Jia Li, and Fugee Tsung. Wiener graph deconvolutional network improves graph supe-suue learning. In AAAI, 2023.
[34] Jia Li, Zhichao Han, Hong Cheng. Jiao Su, Pengyun Wang, Jianfeng Zhang, and Lujia Pan. Predicting path failure in time-evolving graphs. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, page 1279- 2019.
[35] Xiangguo Sun, Hongzhi Yin, Bo Liu, Hongxu Chen, Qing Meng, Wang Han, and Jiuxin Cao. Multi-level hyperedge distillation for social linking prediction on sparsely observed networks. In Proceedings of the Web Conference 2021, page 2934 2945,2945,11 2021.
[36] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[37] Hongyang Gao, Zhengyang Wang. and Shuiwang Ji. Large-scale learnable graph convolutional networks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, page 1424, 2018
[38] Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen. An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. pages 257-266. 2019.
[39] Y. You, T. Chen, Z. Wang, and Y. Shen. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2124-2132, 2020.
[40] Jie Chen, Tengfei Ma, and Cao Xiao. FastGCN: Fast learning with graph convolutional networks via importance sampling. In International Conference Learning Representations, 2018.
[41] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. In Advances in Neural Information Processing Systems, 2019.
[42] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In Advances in Neural Information Processing Systems, pages 28877-28888, 2021.
12

---

[43] Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z. Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Conference 2022, page 1070-1079, 2022.
[44] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In Proceedings of the 37th International Conference on Machine Learning. 2020.
[45] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, page 2069-2080, 2021.
[46] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, page 1150- 100-11666 1160, 2020.
[47] Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML, 2021.
[48] Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang. Chunjic Wang. and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 594-604, 2022.
[49] Zhenyu Hou, Yufei He, Yukuo Cen, Xiao Liu, Yuxiao Dong, Evgeny Kharlamov, and Jie Tang. Graphmae2: A decoding-enhanced masked self-supervised graph learner. In Proceedings of the ACM Web Conference 2023, page 737-746, 2023.
[50] Yufei He and Bryan Hooi. Unigraph: Learning a cross- domainn graph foundation model from natural language. arXiv preprint (tt.::22222.333300000 2024.
[51] Chenxi Sun, Hongyan Li, Yaliang Li, and Shenda Hong. TEST: Text prototype aligned embedding to activate LLM's ability for time series. The Twelfth International Conference on Learning Representations, 2024.
[52] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017.
[53] Mengmei Zhang, Mingwei Sun, Peng Wang. Shen Fan, Yanhu Mo, Xiaoxiao Xu, Hong Liu, Cheng Yang, and Chuan Shi. Graphtranslator: Aligning graph model large language model for open-ended tasks, In Proceedings of the ACM Web Conference 2023, 2024.
13

---

A Dataset description
Table 3: Dataset statistics
Domain
Dataset
#Nodes
#Edges
#Classes
Arxiv
169.343
1,166.243
40
Citation
Pubmed
19,717
44,338
Cora
25,120
91,140
70
Ele-Computer
87,229
721,081
10
Ele-Photo
48,362
500,928
12
E-commerce
Book-Children
76,875
1.554.578
24
Book-History
41,551
358.574
12
Sports Fitness
173,055
1,773,500
13
Citation datasets The Arxiv dataset 225 represents directed citation network among Computer Science (CS) papers from the arXiv preprint server. Each node in this graph corresponds to paper, while edges represent citation links. The PubMed dataset [26] comprises 19,717 scientific publications from the PubMed database related to diabetes. which are categorized into three distinct classes: Experimentally induced diabetes, Type diabetes, and Type 2 diabetes. This classification reflects the focus of each publication within the broader context of diabetes research. Lastly, the Cora dataset [27], formally known as the "Cora Research Paper Classification Dataset", provides comprehensive network for analyzing research paper classifications in machine learning. It is an extended version of the dataset commonly referred to in other studies [52, featuring detailed categorizations.
E-commmerce datasets All e-commerce datasets are provided in the TAG benchmark (28 The Books-Childee and Books- Books-History datasets extracted from the Amazon Boooks dataset. Books- Children includes items with the second-level label "Children", while Books-History includes items with the second-level label "History". Each dataset's label corresponds to the three-level label of the book. The Ele-Computers dataset comprises items with the second-level label "Computers", and Ele- Photo includes items with the second-level label "Photo". Each of these datasets is labeled at the third level for electronic products. The Sports-Fitnect dataset, sourced from the Amazon-Sie -pooots dataset, contains items with the second-level label "Fitness". Nodes in this dataset represent fitness-related items, and an edge between two items indicates they are frequently co-purchased or co- co-vie.ed.
B More experimental results
B.1
Legality rate
Table 4: Legality rate of LLL.-backbene model (The worst results are marked in gray
Dataset
Arxiv
Computer
Pubmed
Cora
Children
History
Photo
Sports
Model
Legality rate(S))
Vicuna-7B-vl.5
99.3
96.7
100.0
95.8
99.2
98.9
94.1
99.6
LLaGA
100.0
100.0
98.9
79.9
93.1
92.4
77.8
94.3
TEA-GLM
100.0
100.0
100.0
92.6
97.0
99.6
99.2
98.5
After training on specific datasets or tasks, large language models (LLMs) may produce invalid or incorrect answers given questions. For instance, when handling unseen datasets or tasks, LLMs may generate responses that fall outside the set of acceptable answer candidates. To evaluate the impact of the training process LLM performance, legality rate to measure the proportion of valid answers produced by the model.
follow the approach in [53) and use the
Table demonstrates that the illegality rate of the LLaGA model significantly increases when exposed to datasets has not previously encountered, suggesting a substantial impact of training methodologies on both the acquisition of knowledge and the model's 's ability to generalize. Conversely, our model exhibits notably stable performance across diverse unseen datasets, achieving higher legality rates in several cases.
14

---

B.2 F1 score on node classification task
Table 5: Macro Fl of node classification task (bold highlights the best result across all methods, while underline highlights the second-bes bbet results)
Citation
E-commerce
Model type
Model
Pubmed
Cora
Children
History
Photo
Sports
MLP
0.246::0.042
0.00910.004
0.007=00007
0.023-100008
0.04120.033
0.019=0.005
GCN
0.187110.01
0.007:10.001
0.006:10.004
0.02410.013
0.034=0.007
0.017:10009
GraphSAGE
0.257+00084
0.007++0.003
0.005+0.003
0.029+0.024
0.020+0.011
0.021+0.004
GAT
0,259=000065
0.006:+0.001
0.063=0.067
0.159=0.117
0.036=0.035
0.091=0.090
DGI
0.213=00117
0.004++00.02
0.012=0.004
0.038=0.015
0.045=0.015
0.018=0.005
GNN as predictor
GKD
0.247:10.039
0.004:10.001
0.028:000003
0.06010.008
0.049=0.015
0.050=0.008
GLNN
0.221:t0.033
0.006:100001
0.001
0.064
0.057=0.002
0.052+00003
NodeFormer
0.232.10.089
0.00810.003
0.01910.008
0.04610.031
0.05510.006
0.04910.009
DIFFormer
0.187++0.007
0.007++00002
0.002+00002
0.050=0.019
0.069=0.010
0.045+0.007
OFA
0.287:10.059
0.091::00113
0.017=0.010
0.026=0.007
0.103=0.007
0.043=0.021
Vicuna-7B-v1.5
0.629:10.024
0.109:10.002
0.279=0.002
0.34910.003
0.383=0.001
0.41010.002
GraphGPT-std
0.649
0.082
LLM as predictor
GraphGPT-cot
0,482
0.127
LLLGA
0.778++0.056
000001000014
0.163=0.029
0.144=0.025
0.362=0.039
0.446=0.035
TEA-GLM
0.839:10.012
0.148:10.015
0.252-00005
0.365-0.011
0.421
0.430-0.009
Due the absence of metric calculate the Fl score while considering the illegality rate, we adopt the methodology used in [53]. For the LLM-bbckbone models, we only calculate the Macro Fl score for legally permissible responses provided by the model. This calculation method may not accurately reflect the model's performance fully. Therefore, we also report the illegality rate in Table 4 Please note that the accuracy metric is unaffected by illegal responses, which are considered error responses.
B.3 Supervised results
Table 6: Accuracy and macro Fl on training datasets (bold highlights the best result across all methods, while underline highlights the second-best results)
Arxiv
Computer
Model type
Mudel
Acc
F1
Acc
F1
MLP
0.546=20.044
0.295=00000
0.420=0.006
0.267=00.05
GCN
0.545=00.005
0.317-0.006
0,42410.012
0.386:10.014
GraphSAGE
0.556=0.006
0.315=00008
0.534=0.037
0.347=10.036
GAT
0.561=0.003
0.339=0.005
0.609=0.035
0.598+00.039
DGI
0.342++0.024
0.336=00.011
0.594=0.004
0.452+00008
GNN as predictor
GKD
0.393+0.085
0.164+00.09
0.351+0.031
0.155+0.016
GLNNN
0.602+00004
0.362++0.008
0.393+0.005
0.243+0.007
NodeFormer
0.544+10.016
0.297=t0.029
0.434-10.012
0.288
DIFFormer
0.616:10.025
0.356:10.024
0.6291:0012
0.467:10.022
OFA
0.682:10.006
0.495
0.75310.004
0.687
Vicuna-7B-v1.5
0.347=+0.000
0. 164::000001
0.372+:0.010
0.304
GraphGPT-std
0.626
0.262
predictor
LLM as
GraphGPT-cot
0.576
0.228
LLaGA
0.749110.001
0.575=0.003
0.64210.004
0.56210.001
TEA-GLM
0.65510.0001
0.445=00002
0.57810.002
0.49610.011
We report the supervised learning results in Table 6 The GNN-backbone models continue to demonstrate robust performance in fitting training data. Similarly, the LLaGA model shows its efficacy in supervised learning scenarios. However, despite their strong performance training datasets, these models exhibit limited generalization capabilities on unseen datasets as shown in Table [[nd T
15

---

C Parameter sensitivity analysis
0.450
0.84
0.15
0.435
0.83
0.13
0.420
0.82
0.11
3
10
10
1
3
5
10
(a) Arxiv
(b) Pubmed
(c) Cora
Figure 3: Impact of number of graph token embeddings (Macro F1)
0.452
0.84
0.148
0.447
0.81
0.144
0.442
0.78
0.140
0 100
1000
2000
3000
100
1000
2000
3000
100
1000
2000
3000
(a) Arxiv
(b) Pubmed
(c) Cora
Figure 4: Impact of number of principal components (Macro F1)
Number of graph token embeddings To discuss the impact the number of graph token embed- dings, we set K E 3,, 5, 10} and report the results node classification task in Figure 3 In the context of training datasets and unseen datasets, we observe two distinct patterns. With an increase in the number of graph token embeddings in the training dataset, there is slight improvement in the model's performance on that dataset. This suggests that in a supervised learning scenario, enhancing the model's performance can be achieved by increasing the quantity of graph token embeddings. Con- versely, for unseen datasets, our model requires only minimal number of graph token embeddings achieve satisfactory performance, indicating that the number of learnable parameters in model significantly less than concurrent works.
Number of principal components We define P E {0, 100, 1000, 2000, 3000) results of the node classification task in Figure In supervised learning scenarios, omitting contrastive learning with principal components can lead to slight increase in accuracy. However, this often makes the model more prone to overfitting on training datasets. When the number of principal components is too small, it adversely affects the model's 's learning capability. Remarkably, when P 1000, the model demonstrates satisfactory performance. At this level, the principal components capture 50% of the variance of LLM's token embeddings.
2000,3000) and discuss the
16

---

D Complete instructions
Given the representation of paper: <Token 2> ... <Token K>, with the following information:
Title: [title).
Question: Which arXiv sub-category does this paper belong to? Please directly give the most likely
from the following sub-categories: (answer candidates).
Node Classification
Given the representation of two papers: <Token 1> <Token 2> <Token K>, with the following information:
Title: First Paper: (title 1J. Second Paper: (title 2).
Question: What kind of relationship exists between these two papers? choose most likely answer
from the following options: "These two papers have citation relationships' or "These two papers may not have
citation relationships'.
Prediction
Figure 5: Instructions for node classification and link prediction
In node classification tasks, we provide candidate labels to facilitate the model's learning process, focusing on discovering the correct answers rather than merely memorizing them. For link prediction. we structure the instructions in format similar to that of node classification. This approach is designed enhance the model's ability transfer learned knowledge effectively across different tasks.
E Cross-task zero-shot results with different pooling methods
Table 7: AUC of link prediction (Cross-task) with different pooling methods
Citation
Model
Arxiv
Pubmed
Cora
OFA
0,469
0.481
0.492
Vicuna-7B-v1.
0.513
0.543
0.527
Vicuna-7B-SPT
0.537
0,535
0.565
GraphGPT-std
0.649
0.501
0.520
LLaGA
0.570
0.569
0.537
TEA-GLM (max)
0.639
0.650
0.566
TEA-GLMM (sum)
0.657
0.689
0.586
TEA-GLM (mean)
0.659
0.690
0.588
Considering that different pooling methods may impact cross-task performance, we conducted experiments using three common pooling methods separately, and the results are shown in the Table
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We verify the contributions of our proposed method through experiments, and the results in the Sec. effectivvey demonstrate the contributions we outlined in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the current shortcomings of our method and future research
directions in Sec
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of the paper. In Sec. 2 we talk about the method used in our model. In Sec. 2.3.3 and Sec. B.T we give the training process and the detailed settings of our model. We ensure that all the results in our paper can be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
19

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publicly available datasets, and we will provide anonymized link to the code repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
pibllc//guidee/fooeettmnsssss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
ctspsscc/ppblii//uuii httpa//gguuieee//oteuubmiiisss ssiollliisi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss the data splits, hyperparameters, random seeds, baselines and type of optimizer in Sec. 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: To ensure comparability among different methods, identical data splits were applied to all models. To ensure result robustness, conduct five experiments with random seed values ranging from to 4, and report the mean and standard deviation of the results. The results are reported in Sec. 3.2 and Appendix B.2
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computer resources in Sec. 3.1 There no experiment that requires more than what we mentioned in the paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////....eee...ccccpc/pcaaas
Answer: [Yes]
Justification: Our research conforms to the ethical guidelines outlined in the NeurIPS Code Ethics. We have ensured that all aspects of our research, including data collection, data usage, and experimentation, adhere to these standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
21

---

Answer: [NA]
Justification: Our research proposes an improved method that allows large language models to more effectively utilize graph information. We consider this work to be primarily technical research that has not yet been applied to specific real-world scenarios. Therefore, we believe it does not directly produce social impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We doesn't 't release data or models that have a high risk. This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: For the models and data we use in our paper, we all cite the original paper. Meanwhile, we provide the license of the data we use in our paper. Arxiv 25 with "ODC-
BY". Pubmed (26), Cora [2 and e-commerce datasets TAG benchmark 28 with
"MIT".
22

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
23

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "OYJ5ZezXwu", "reviewer": "Reviewer_64Q8", "summary": "The motivation of this work is to utilize the zero-shot learning capacity in LLMs for structural data. The basic idea is to connect GNNs with LLMs, by aligning its representations with the token embeddings of an LLM, such that GNNs can encode the structural information and LLMs can do zero-shot inference. The experiments were conducted with two different domains and seven baseline methods, and demonstrate the performance of the proposed method.", "questions": "No further question, just some minor comments.", "limitations": "The limitations have been listed in the paper.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The research question is interesting and important. Generalizing the capacity of LLMs for zero-shot graph learner is a good idea to use the zero-shot capacity for structural information.\n- The proposed method is properly described with sufficient technical details. The pipeline of two-step adjustment is convincing.\n- Although more experiments can certainly be done, the current experiments answer three important research questions aligned with the topic, each of them is answered with a good amount of evidence.", "weaknesses": "I do not have any major concern about this work. On the other hand, I do think the presentation of this work can be better. Some parts of the paper (especially section 2) are a little bit difficult to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's understanding and recognition of the contribution of our work. For the problem regarding\n>  I do think the presentation of this work can be better. Some parts of the paper (especially section 2) are a little bit difficult to understand,\n\nwe apologize for the lack of clarity in certain parts of our paper. As the reviewer pointed out, the explanation of feature-wise contrastive learning in Section 2 is not as clear and concise as it should be. We will revise the paper, with a focus on Section 2, to improve clarity and readability.\n\nThank you again for reviewing our manuscript. Please kindly let us know if you have any follow-up questions. Your insights are valuable to us, and we stand ready to provide any additional information that could be helpful.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hExdpiczSR", "reviewer": "Reviewer_65Wi", "summary": "The paper propose a GNN plus LLM model for zero-shot learning in the graph domain. They first use contrastive learning to pretrain a GNN that can be applied to arbitrary graphs. They added a feature-wise contrastive learning  objective on the projected representation to the PCA of LLM token embeddings to bridge the gap between contrastively learned embedding and LLM token embeddings.  After training the GNN, they train additional linear projector to the map the GNN output to a sequence of graph tokens, and the tokens are combined with special prompt designed for graph learning tasks for LLM to perform downstream tasks. The methods does not need to fine-tune the LLM and acheived good performance on most of the datasets.", "questions": "- Why do you need to map a single GNN node representation to multiple graph tokens? They should contain exactly the same amount of information.\n\n- What is the prompt used for baseline LLM? Is it the same prompt as the one used for GNN? If so, what happen when you provide it with a more detailed prompt, but still only on the target node?\n\n- For edge and graph tasks, you used the sum of node representation to compute graph tokens, wouldn't that break the aligned distribution as the pretraining only happens on node.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The PCA-based alignment with the LLM input is interesting, and potentially can shed lights on future research on GNN and LLM alignment.\n\nThe experimental results seem promising on most datasets compared to both LLM and GNN+LLM baselines. Especially, the model shows some-level of cross-task transferrability where the GNN and linear projector are pretrained only on node classification but, but show non-trivial performance on link prediction tasks.", "weaknesses": "My major concern about the paper is its novelty. Several important components, including using self-supervised learning to generate general node embedding [1], alignning graph representation with LLM input with linear projector [2], and the prompt design are proposed and already widely adopted in the research area. The feature-wise contrastive learning seems to be the main technical contribution, yet it is not studied and invectigated in details. For example, why pre-training on node classification leads to non-trivial link prediction results? According to the ablation study, feature-wise CL seems to be the key, yet no knowledge about link-prediction is injected into the model in any step of the method.\n\nRelated to the main concern, I think the experimental results do not fully justify the authors claim. The author claims that the aligned graph tokens provide the LLM with graph information. However, they still feed the node texts as direct text input to the LLM, for some datasets LLM can do well solely using node texts. To show that the pretraining process indeed brings in inductive/transferrable knowledge of the graph, the author should either show that by completely removing node text to from LLM text input (it is ok to keep task text), the model can still work or provide results of baseline models that tune the LLM on the node texts (tuning the LLM weight or use a soft prompt), and such baseline can not transfer or performs poorly. Note that the ablation study between w/o FC and w/o graph token is not enough. Essentially, if a soft prompt tuning can achieve similar effect, the proposed work is just adding linear layers as a substitute to soft prompts.\n\n[1] He, Yufei, and Bryan Hooi. \"UniGraph: Learning a Cross-Domain Graph Foundation Model From Natural Language.\" arXiv preprint arXiv:2402.13630 (2024).\n\n[2] Tang, Jiabin, et al. \"Graphgpt: Graph instruction tuning for large language models.\" arXiv preprint arXiv:2310.13023 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable suggestions. We have addressed your questions as follows:\n> My major concern about the paper is its novelty. Several important components, including using self-supervised learning to generate general node embedding [1], alignning graph representation with LLM input with linear projector [2], and the prompt design are proposed and already widely adopted in the research area.\n\nDue to the word limit, please refer to the \"Novelty\" section in the top-level Author Rebuttal.\n\n> The feature-wise contrastive learning seems to be the main technical contribution, yet it is not studied and invectigated in details. For example, why pre-training on node classification leads to non-trivial link prediction results? According to the ablation study, feature-wise CL seems to be the key, yet no knowledge about link-prediction is injected into the model in any step of the method.\n\nIt seems there might be a misunderstanding regarding the role of the key components. Firstly, the GNN is trained using self-supervised learning, which effectively aids the model in deeply exploring and understanding the intrinsic structures and properties of graph data. This training approach is advantageous for tasks such as node classification, link prediction, and graph classification, as referenced in [1, 3, 4]. Indeed, the linear projector is trained on node-level tasks with prompts, but to enable cross-task applicability, we have designed unified prompts. In addition to employing a fixed number of graph tokens for tasks at the node, edge, and graph levels, we also incorporate candidate answers. This approach is designed to enable the LLM to learn to select the correct answer using graph information, rather than merely learning how to respond to node classification problems. Consequently, after training on node classification tasks, the LLM can extract graph information from graph tokens, which naturally benefits the performance on link prediction tasks.\n\nRegarding feature-wise contrastive learning, its key role is to assist GNNs in generating node representations that are more easily understood by LLMs. The aforementioned process ensures that the LLM can extract graph information from graph tokens, while feature-wise contrastive learning ensures that the LLM can comprehend this graph information. The results of our ablation experiments demonstrate that using feature-wise contrastive learning yields representations with greater generalizability, which are also better understood by the LLM.\n\n> Related to the main concern, I think the experimental results do not fully justify the authors claim. ... Essentially, if a soft prompt tuning can achieve similar effect, the proposed work is just adding linear layers as a substitute to soft prompts.\n\nDue to the word limit, please refer to the \"Experimental Results of Soft Prompt Tuning\" section in the top-level Author Rebuttal.\n\n> Question 1\n\nNode representations encompass information about the graph structure and its neighboring nodes, and we believe that a single token cannot adequately convey the graph information. Similarly, in the vision-language model domain, it is challenging for a single token to represent an entire image; images are typically encoded into multiple tokens[5, 6]. We consider this approach both correct and reasonable. The experimental results shown in Figure 3 of Appendix C further demonstrate that an appropriate number of graph tokens can significantly enhance the model's performance.\n\n> Question 2\n\nWe apologize for not providing the prompt for the baseline LLM in our paper; we will add this information. The prompt used by the baseline LLM is almost identical to that of our model, the only difference being the absence of graph tokens. Here is an example:\n```\n# Our model\nGiven the representation of a paper: <Token 1> <Token 2> … <Token K>, with the following information:  \nTitle: {title}. \nQuestion: Which arXiv CS sub-category does this paper belong to? Please directly give the most likely answer  \nfrom the following sub-categories: {answer candidates}.\n\n# Baseline LLM\nGiven a paper with the following information:  \nTitle: {title}. \nQuestion: Which arXiv CS sub-category does this paper belong to? Please directly give the most likely answer  \nfrom the following sub-categories: {answer candidates}.\n```\nFor GNN models, prompts cannot be utilized. We followed the setup described in GraphGPT [7] to conduct the zero-shot testing of GNN models.\n\n> Question 3\n\nWe apologize for the insufficient explanation regarding readout operations in our paper. Sum, mean, max, and min are commonly utilized readout operations in Graph Neural Networks, as cited in references [1-3]. After obtaining node representations from a GNN, these operations are employed to derive representations for edges and graphs, as noted in [1]. We implemented the sum readout method, which our results have demonstrated to be effective. However, as this was not the primary focus of our research, we did not investigate variations of these operations. We will include more detailed explanations on this topic in our revised paper.\n\n[1] Xie Y, et al. Self-supervised learning of graph neural networks: A unified review. IEEE transactions on pattern analysis and machine intelligence. 2022.\n\n[2] Xu K, et al. How powerful are graph neural networks?. arXiv preprint. 2018.\n\n[3] Y. You, et al. “Graph contrastive learning with augmentations,” NeurIPS, 2020.\n\n[4] Hu Z, et al. Gpt-gnn: Generative pre-training of graph neural networks. SIGKDD. 2020.\n\n[5] Liu H, et al. Visual instruction tuning. NeurIPS. 2023.\n\n[6] Li J, et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. PMLR. 2022.\n\n[7] Tang J, et al. Graphgpt: Graph instruction tuning for large language models. SIGIR. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nRAx3bfAsn", "reviewer": "Reviewer_LAU5", "summary": "This paper proposes TEA-GLM to align the GNN representations with LLM token embeddings for zero-shot graph machine learning. TEA-GLM enables cross-dataset and cross-task learning without fine-tuning the LLM. Extensive experiments demonstrate its state-of-the-art performance on unseen tasks and datasets.", "questions": "Please see Weaknesses.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and easy to follow, effectively conveying the design and motivation behind LLM-GS. \n\nThe proposed framework employs only a linear projector to incorporate with the LLM, which is efficient. \n\nThe idea of mapping graph embedding to token embedding is also interesting.", "weaknesses": "The experimental results are somewhat questionable. The vanilla Vicuna-7B even outperformed LLaGA, which leveraged Vicuna-7B as its base model. Furthermore, the results on E-commerce-Children show no difference between Vicuna-7B and the proposed TEA-GLM.\n\nFrom my perspective, this paper marks LLaGA as the strongest baseline. However, I find that the experimental settings are totally different from LLaGA. Is it possible to conduct some experiments following LLaGA?\n\nThe ablation studies are also questionable. Even without GT or without FC, the model achieves at least equivalent performance to that of Vicuna-7B, right?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and thoughtful comments! We have addressed your questions as follows:\n> The experimental results are somewhat questionable. The vanilla Vicuna-7B even outperformed LLaGA, which leveraged Vicuna-7B as its base model.Furthermore, the results on E-commerce-Children show no difference between Vicuna-7B and the proposed TEA-GLM.\n\nWe apologize for any misunderstandings of the results that may have arisen due to the experimental setup. To address this, we provide clarification on the relevant experimental setup. Specifically, both LLaGA and our model were trained on the Arxiv and Computer, followed by zero-shot testing on the target datasets listed in Table 1. Following the approaches detailed in [1, 2], we used Vicuna-7B without fine-tuning as a baseline to evaluate the generalization ability of models that use Vicuna-7B as their base model. The LLaGA primarily focuses on the domain of supervised learning, where it exhibits strong performance, as detailed in Appendix B.3. It also shows some generalization capabilities in a zero-shot scenario on citation datasets, outperforming Vicuna-7B. However, compared to Vicuna-7B, LLaGA demonstrates negative transfer effects across e-commerce datasets. These datasets present a greater challenge due to their low topic relevance, thus highlighting its limited generalization capabilities.\n\nRegarding the Children dataset, this dataset is particularly challenging as it contains 24 categories and shows minimal topical relevance to the source dataset. Specifically, the Children dataset comprises titles and summaries about children’s books, whereas the Computer dataset includes reviews on computer-related electronic products (for a detailed description of these datasets, please refer to Appendix A). In comparison with other baselines, our model does not exhibit negative transfer on the Children dataset. Furthermore, it outperforms the baseline on five other datasets, which we believe convincingly demonstrates the superiority of our approach. \n> From my perspective, this paper marks LLaGA as the strongest baseline. However, I find that the experimental settings are totally different from LLaGA. Is it possible to conduct some experiments following LLaGA?\n\nWe apologize for any confusion caused. Our primary research goal is to establish LLM as a zero-shot learner for various graph tasks, with a main focus on the zero-shot scenario. From this perspective, GraphGPT[1] represents the most relevant work. While LLaGA does address zero-shot learning, its main emphasis is on supervised learning scenarios. Therefore, our experimental setup follows the approach outlined in the GraphGPT. Additionally, to more thoroughly validate our model’s generalization capabilities, we have also included cross-task scenarios in our testing to evaluate the model's performance.\n\nAt the same time, we have also reported the results of our experiments in supervised learning scenarios, which can be found in Appendix B.3. LLaGA performs well in supervised learning contexts; however, its generalization capabilities are limited, particularly when dealing with datasets that have highly irrelevant topics. In contrast, our model performs well in zero-shot scenarios but underperforms in supervised learning contexts. Optimizing performance across both scenarios will be a focus of our future work.\n> The ablation studies are also questionable. Even without GT or without FC, the model achieves at least equivalent performance to that of Vicuna-7B, right?\n\nWe apologize for any confusion regarding our ablation studies. In fact, the results labeled 'without GT' correspond to those of Vicuna-7B, a detail that was not clarified in the paper. We will add this in the appropriate section of the paper. The results labeled 'without FC' do not correspond to Vicuna-7B. During the GNN pre-training phase, we employed both instance-wise and feature-wise contrastive learning. Feature-wise contrastive learning, which we proposed to involve integrating principal components selected from LLM token embeddings, and restrict the node embeddings obtained from GNNs. This is a crucial component for achieving generalizable and LLM-aligned node embeddings. 'Without FC' refers to using only instance-wise contrastive learning for pre-training the GNN, which still utilizes GNN; hence, the results are not those of Vicuna-7B.\n\nThank you again for reviewing our manuscript. We have tried our best to address your questions, and have revised our paper based on your questions. Please kindly let us know if you have any follow-up questions. Your insights are invaluable to us, and we are prepared to provide any additional information that may be helpful.\n\n[1] Tang, Jiabin, et al. \"Graphgpt: Graph instruction tuning for large language models.\" arXiv preprint arXiv:2310.13023 (2023).\n\n[2] He, Yufei, and Bryan Hooi. \"UniGraph: Learning a Cross-Domain Graph Foundation Model From Natural Language.\" arXiv preprint arXiv:2402.13630 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the detailed reviews and valuable suggestions provided by the reviewers. Due to the character limit for a separate response, here we will focus on addressing the concerns regarding the novelty of our paper and presenting the results of the additional experiments.

**Novelty**

We respectfully disagree with the view that 'several important components are proposed and are already widely adopted in the research area.' Firstly, although [1] also utilized self-supervised learning, the methodologies are entirely distinct with ours. Specifically, for the first time, we proposed feature-wise contrastive learning in the GNN pre-training phase to obtain more generalizable node embeddings. This method uses principal components from LLM token embeddings to refine GNN-derived node embeddings, ensuring they align well with LLM token embeddings. Secondly, while we do use a linear projector, unlike existing related work [1, 2], we are the first to propose mapping a node embedding to multiple graph tokens, and we do not perform any form of fine-tuning on the LLM. This is enabled by feature-wise contrastive learning, which ensures that the node embeddings obtained post-GNN pre-training are already aligned with the LLM, thus only requiring training a linear projector. Thirdly, to achieve cross-task capabilities, we for the first time propose a unified prompt for graph tasks at different levels—node, edge, or graph. For tasks at different levels, we utilize a readout operation to extract representations corresponding to nodes, edges, or entire graphs. These representations are then mapped to a fixed number of graph tokens. This approach diverges from previous works that directly input all node tokens. Our design is more rational as the order of input node tokens significantly influences the LLM's output, which contradicts the graph's inherent node permutation invariance.

**Experimental Results of Soft Prompt Tuning**

Thank the reviewer 65Wi for constructive comments. We acknowledge the lack of experiments specifically demonstrating the role of graph information in the graph tokens. To address these concerns, we validated our approach by fine-tuning the LLM with soft prompts. To ensure a fair comparison, all experiments were conducted using the same random seed and prompt. Both the number of soft prompts and our model's graph tokens were consistently set to five. The key difference is that soft prompt tokens are trainable parameters, unlike those derived from GNNs and a linear projector. 

We report the average experimental results: for the zero-shot tasks (upper table), we present the accuracy, and for the cross-task tasks (lower table), we provide the AUC. **Bold** highlights the best result across all methods, while *italics* highlights the second-best results. These results clearly demonstrate that, across all tasks, fine-tuning the LLM with soft prompts leads to inferior performance compared to our model. Notably, in node classification tasks, it fails to achieve transferability on e-commerce datasets, indicate that soft prompt tuning can not do well solely using node texts. This suggests that graph tokens indeed carry transferable graph information, which enables the LLM to make more accurate predictions. Additionally, this indicates that the use of FC produces graph tokens that are more interpretable by the LLM.  After conducting additional experiments, we believe the experimental results fully justify that our method makes the LLM a zero-shot graph learner. We will include this content in our revised paper.

| | Pubmed | Cora | Children | History | Photo | Sports |
|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
| Vicuna-7B | 0.719 | 0.156 | *0.270* | *0.363* | *0.378* | *0.370* |
| GraphGPT-std | 0.701 | 0.126 | - | - | - | - |
| GraphGPT-cot | 0.521 | *0.181* | - | - | - | - |
| LLaGA | *0.793* | 0.168 | 0.199 | 0.146 | 0.276 | 0.352 |
| Vicuna-7B (SoftPrompt Tuning) | 0.768 | 0.168 | 0.227 | 0.281 | 0.350 | 0.230 |
| **TEA-GLM (Ours)** | **0.848** | **0.202** | **0.271** | **0.528** | **0.497** | **0.404** |

| | Arxiv | Pubmed | Cora | Children | History | Computer | Photo | Sports |
|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
| Vicuna-7B | 0.513 | 0.543 | 0.527 | 0.500 | 0.515 | 0.502 | *0.501* | 0.502 |
| GraphGPT-std | *0.649* | 0.501 | 0.520 | - | - | - | - | - |
| LLaGA | 0.570 | *0.569* | 0.537 | 0.422 | 0.449 | 0.479 | 0.478 | **0.597** |
| Vicuna-7B (SoftPrompt Tuning) | 0.537 | 0.535 | *0.565* | *0.544* | *0.543* | *0.509* | *0.501* | 0.508 |
| **TEA-GLM (Ours)** | **0.657** | **0.689** | **0.586** | **0.571** | **0.579** | **0.554** | **0.545** | *0.553* |

[1] Tang J, et al. Graphgpt: Graph instruction tuning for large language models. SIGIR. 2024.

[2] He Y, et al. UniGraph: Learning a Cross-Domain Graph Foundation Model From Natural Language. arXiv preprint. 2024.

Meta-review: This paper proposes a method to inject graph embeddings into LLMs for zero-shot node and link prediction tasks. The key idea revolves around learning graph embeddings and then connecting them to LLM token embeddings in a self-supervised manner. Experiments show that this approach outperforms both using the base LLM directly as well as other graph + LLM methods from the literature.

The main concerns raised by the reviewers were around the validity of the experimental results. But these have been adequately addressed  in detailed responses from the authors. In particular, I agree that a direct comparison to the LLaGa setting is not applicable since that approach is supervised, and that the higher performance of Vicuna on unseen datasets is expected.

Overall, this paper has enough novel components and strong experimental results to justify an accept.
Decision: Accept (poster)