Preference-driven Knowledge Distillation for Few-shot Node Classification
Authors: Xing Wei
Chunchun Chen
Rui Fan
Xiaofeng Cao
Sourav Medya
Wei Ye
Venue: NeurIPS 2025 poster ()
Keywords: Preference-driven
Knowledge distillation
Large language models
Few-shot node classification
Primary area: deep_learning
TL;DR: This paper propose a preference-driven knowledge distillation (PKD) framework to synergize the complementary strengths of LLMs and various GNNs for few-shot node classification on text-attribute graphs.

Abstract: Graph neural networks (GNNs) can efficiently process text-attributed graphs (TAGs) due to their message-passing mechanisms, but their training heavily relies on the human-annotated labels. Moreover, the complex and diverse local topologies of nodes of real-world TAGs make it challenging for a single mechanism to handle. Large language models (LLMs) perform well in zero-/few-shot learning on TAGs but suffer from a scalability challenge. Therefore, we propose a preference-driven knowledge distillation (PKD) framework to synergize the complementary strengths of LLMs and various GNNs for few-shot node classification. Specifically, we develop a GNN-preference-driven node selector that effectively promotes prediction distillation from LLMs to teacher GNNs. To further tackle nodes' intricate local topologies, we develop a node-preference-driven GNN selector that identifies the most suitable teacher GNN for each node, thereby facilitating tailored knowledge distillation from teacher GNNs to the student GNN. Extensive experiments validate the efficacy of our proposed framework in few-shot node classification on real-world TAGs.
Our code can be available at <https://github.com/GEEX-Weixing/PKD>.

Preference-driven Knowledge Distillation for Few-shot Node Classification
Xing Wei Chunchun Chen² Rui Fan 122,3 Xiaofeng Cao Sourav Medya Wei Yel:2.
F
College of Electronic and Information Engineering, Tongji University, China Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, China MachhnnMacchine Hybrid Augmented Intelligence, Xi'an Jiaotong University, China
National Key Laboratory
School of Computer Science and Technology, Tongji University, China Department of Computer Science, University of Illinois Chicago, USA (xing627, c2chen, yew)@tongji eeetoonnjieeuu..... edu. cn, rui faiieenee org. xiaofeng. cao. uts@gmail. com, medyaCuic.. edu
Abstract
Graph neural networks (GNNs) can efficiently process text-attributed graphs (TAGs) due to their message-passing mechanisms, but their training heavily re- lies on the human-annotated labels. Moreover, the complex and diverse local topologies of nodes of real-world TAGs make it challenging for a single mecha- nism to handle. Large language models (LLMs) perform well in zero-/fee-ssher learning on TAGs but suffer from a scalability challenge. Therefore, we propose preference-driven knowledge distillation (PKD) framework synergize the complementary strengths of L.LMs and various GNNs for few-shot node classi- fication. Specifically, develop GNN-preference-drivee node selector that effectively promotes prediction distillation from LLMs to teacher GNNs. To further tackle nodes' intricate local topologies, we develop a node-preference- driven GNN selector that identifies the most suitable teacher GNN for each node, thereby facilitating tailored knowledge distillation from teacher GNNs the stu- dent GNN. Extensive experiments validate the efficacy of our proposed framework in few-shot node classification on real-world TAGs. Our code is available https: //tptth///tthbbbb
Weixing/PKD
Introduction
Text-attributed graphs (TAGs [1]), such as citation, webpage, and product graphs [2, 3], have nodes associated with text attributes. Graph neural networks (GNNs) [4, 5] have demonstrated excellent performance and efficiency in node classification on TAGs, which are supported by high-qualis labels and effective message-passingg mechanisms [6]. However, the manual labeling of nodes is undoubtedly a tedious, expensive, and time-consuming task [7. 8]. In many scenarios, only a few node labels are available. Additionally, nodes often have complex and diverse interaction relationships with each other-thein local topologies are intricate whiicatte--hhhhhhe which challenge traditional GNNs with fixed message- passing mechanisms. Compared with GNNs, large language models (LLMs) exhibit impressive zero-ffew-shot learning capabilities on TAGs [9, 10, 11]. But the large parameter scale considerably hinders their inference efficiency [12].
A natural idea is to blend their complementary strengths for few-shot node classification on TAGs. Knowledge distillation (KD) [13] is feasible solution. However, directly distilling knowledge from the LLM to GNN is impractical. Firstly, the discrepancy of decoder-only (LLMs) and encoder-only
CCorrssondiigg Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(GNNs) leads to fundamentally different characteristics in their embedding spaces [14]. And the huge uubedding-dimeensie difference needs sophisticated embedding alignment and also brings high training cost [15]. In contrast, conducting prediction distillation from L.MMs to GNNs by annotating node labels can efficiently alleviate the label scareity and scalability dilemma [16]. The critical question is how to select the nodes for the LLM's label annotation to effectively enhance teacher GNNs. Generally, one may use uncertainty [17] as a selection metric in the embedding space of GNN, However, owing nodes' diverse semantic and complex structural attributes (e.g.. local topologies), a single GNN cannot capture the essences of nodes completely [18]. Therefore, we investigate the GNNs effectively mitigate cognitive limitations [19]
embedding spaces of associated with relying on a single GNN, thereby better selecting nodes for LLM's label annotations.
Nevertheless, since nodes have intricate local topologies, which need tailored message- paasiing mechanisms, how to tailor for each node the most appropriate message-passing mechanism is another challenge. Different GNNs provide different prediction attributes for each node during the learning process [20], encompassing the understandings of its topologies, its interaction relationships to other nodes, and its latent patterns. These node-specific attribute differences suggest that a single message-passing mechanism cannot fundamentally handle the entire graph. Some studies [21, 18] distill knowledge sequentially or simultaneously from teacher GNNs without taking into account the node-specific local topologies, resulting in no obvious performance improvement or even performance degradation [22]. Therefore, it is essential to identify the GNN message- pasging mechanisms that align with the node-specific attributes.
To this end, we propose preference-driven knowledge distillation (PKD) framework that unites the complementary strengths of LLMS and various-architeectuee GNNs for few-shot node classification on TAGs. It mainly includes two modules: CONN-preference-drie preference-driven GNN Selector (NGS). The prerequisite of GNS is that the LLM should be able to comprehend the graph topology. Thus, we develop the graph topology aware (GTA) prompts to fine-tune the LLM, enhancing its capacity to comprehend graph topology. GNS fully exploits nodes' prediction discrepancies among various GNNs to decide nodes whose labels are annotated by the LLM will effectively enhance teacher GNNs, facilitating knowledge distillation from the LLM to teacher GNNs. NGS selects for each node the most appropriate GNN pessage-passsing mechanism, facilitating the tailored knowledge distillation from various teacher GNNs to the student GNN. It regards the fine-tuned LLM as the RL-based (reinforcement learning) agent, which treats all textualized node-specific attributes (including node's semantic, structure, and prediction attributes) as state and the student GNN's performance reward. Our contributions can be summarized follows:
Node Selector (GNS) and Node-
We introduce a preference-driven knowledge distillation (PKD) framework to synergize the complementary strengths of the LLM and various GNNs ingeniously for few-shot node classification on TAGs;
We propose a GNN GNN-preference-driven node selector, effectively determining nodes for annotation by the LLM and promoting knowledge distillation from the LLM to teacher GNNs;
We propose mode-preference- priate message-passing mechanism, promoting knowledge distillation from teacher GNNs to the student GNN;
GNN selector to tailor for each node the most appro-
We validate the efficacy of PKD for few-shot node classification on nine TAGs. The experiments show that it even defeats some state-of-the-art methods that use more node labels.
2 Related Work
2.1
Graph Neural Networks
The field of graph learning has been dominated by GNNs. Early GCN [4] introduces a spectral-based graph convolution operation to propagate node information through the graph. GAT [23] uses at- tention mechanisms to weigh neighbors' contributions, enabling adaptive learning of neighborhood importance. APPNP [24] enhances message passing by using personalized propagation with a power iteration approach, improving label propagation on graphs. H₂GCN [25] extends GCNs by incor- porating higher-order neighborhood information to improve representation power GPRGNN [26]

---

combines graph convolution with residual connections to improve propagation efficiency, particularly in graphs with diverse node degrees. HoloNets [27] introduces a dual-filter mechanism with spectral response, extending spectral convolutions to directed graphs. DirGNN [28] defines the in- in-neighbors and out-neighbors and performs separate propagation and aggregation, improving the message passing through the incorporation of edge directionality. To deal with label scarcity. GCNII [29] introduces initial residual connections and identity mapping to construct a deep GNN while EGNN [30] enforces equivariance constraints for the enhancement of data efficiency and generalization. AGST [31] and IceBerg [32] leverage the different self-training [33] methods to effectively utilize unlabeled nodes.
2.2 Knowledge Distillation
KD is not only used for model compression, but for strengthening purposeful abilities of the student model. GFL [34] extracts structural knowledge from pre-prepared similar auxiliary graph, distilling the target graph for enhancing few-shot node classification performance. KDGA [35] utilizes multiple graph augmentation strategies to make student GNN produce robust node representations after distillation. MSKD [36] mitigates the diverse classification situations requiring for different nodes by capturing multi-scale topological semantics distilled from varying layers. However, the capability of an individual teacher is inherently limited. BGNN [21] distills complementary knowl- edge from multiple GNN teachers sequentially and integrate it by the adaptive temperature parameter and weight boosting modules. MTAAM [22] distills knowledge of multiple teacher GNNs into an MLP-student, offering quick inference speed without compromising accuracy. FairGKD [37] obtains equitable and informative node representations by synergizing multiple GNN experts into a teacher. DMKD [18] harnesses complementary knowledge from various GNNs and conducts layer-level knowledge distillation to mitigate the constraint of a single teacher. Furthermore, [14] is a label-free method that proposes the LL.M-NNN It uses L.LMs to get high-quality annotation through active and confidence-awarrness node selection, thereby circumventing the difficulty of label annotation by humans. LinguGKD [15] introduces a kind of ingenious contrastive learning align the LLM's semantic features with GNN's structural features to achieve knowledge transfer. Most of the above knowledge distillation methods do not tailor for each node the most appropriate message-passing mechanism and underperform on few-shot node classification.
3 Method: PKD
In this section, we present the preference-driven knowledge distillation (PKD) framework. PKD) involves two key modules: GNN-preference-drivee Node Selector (GNS) and Node-preference-driven GNN Selector (NGS). The main goal of the former module is to select node groups whose labels are annotated by the LLM will drastically enhance teacher GNNs. The main goal of the latter module to select the most appropriate teacher GNN for each node, thereby tackling the complication of mode-specific local topologies. The PKD framework is illustrated in detail in Figure 1.
3.1 Background
A text-attributed graph (TAG) is denoted by GT (V,E,X,A T), where V = (U1,..... uN } is set of nodes with semantic attributes T = (t......, ty] and E is a set of edges. Each semantic attribute can then be encoded as a sentence embedding X = [x]...., .i.............................. ......,
xxl E I NxF with the help of language models. A E RNN the adjacency matrix. Given the few-shot node classification task, let Dr = y(),, (Q < N) be the set of labeled nodes with y; as the one-hot label of the training sample x; and Do be the set of unlabeled nodes, respectively. The goal is to accurately predict the labels of nodes that belong to Du given few labeled nodes in Dz.
We assume B teacher GNNs denoted by {To) and f%, is the model parameters of To. The B 2 b. 2. B. which is the concatenation of the logit of each teacher x [ b. a. (1 S b S B), where </.b.. is the probability of 2; belonging to class (1 S c < C) objective for the KD from node-preference GNNs to the student GNN can be divided into three parts:
logit outputs of teacher GNNs for node i; written 2!- [
computed by teacher T,. Our final
N
1 Q
N H(F((x))) (1)
LKD = a- (- Σ
1
o
logf((xx)) +3
Σ yo (x.)) +r
Q
3

---

Text-Atiribute Graph
GNN-preference-drivnn Node Sclector
Node-preffrenee-drvven CNN Selector
Pretrain
Classifier
Classifier
Classifeer
Prodiction Ainrhhute Collection
DNS Modale
Prefercace renks
Fint
Second
Third
eTA Inotrution Pleertaning
Netriin
Cos
Gos
True Labels
Semantic attribun
Straclare atributt
Prediction attribatc
Nodox asnotatioo
Uniabcled mode
Labeled made
Figure 1: Overview of PKD. The framework has two key modules: GNN-preferencc-drivee Node Selector (GNS) and pode-preference-ddrivn GNN Selector (NGS). Before starting GNS, first fine-tune the LLM with GTA prompts to enable it to comprehend graph properties. In the GNS module, we exploit the proposed K K-uncertainty based on the node prediction uncertainty in each teacher GNN's embedding space to select nodes. For effectively exploiting the LLM to annotate those selected nodes, we combine the semantic attributes and structure attributes derived from the proposed Distance-based Neighbor Selector (DNS) module on these nodes to construct prompt, promoting the prediction distillation from the fine-tuned I.I.M to teacher GNNs (7i, T : TB). In the NGS module, we select for each node the most appropriate teacher GNN for tailored knowledge distillation. The teacher GNN selection is achieved by reinforcement learning with the fine-tuned LLM as agent.
where a, 18, are hyper-paramelten to balance three losses. For student GNN with parameters fs. the first loss, distillation loss LDL, is defined as the cross-entropy between the predictions the teacher GNNs and that of the student GNN. The (x,) is the Softmax output of student GNN and it denotes the probability distribution of e; belonging class प m, @ z!"" where m; is one-hot vector denoting which teacher GNN is preferred by U, The second loss, LCE. is the cross cross-entropy loss in the training of student GNN. Inspired by [38]. add LE the objective the last part, which makes the logits of student GNN closer to one-hot vectors. The H.)) denotes Shannon entropy.
3.2 LLM Fine-tuning
Recent studies reveal that LLMs possess reasoning apabil- ities [39], but they often underperform compared to even the simple GNNs when tackling graph learning. The key challenge lies in its inability to directly process the raw graph data and understand topology properties, limiting the generalization ability of L.LMs in this domain. To address this, we propose GTA prompts fine-tuning.
TA
64.5
Wire
Anal
24.1
This method consists of four distinct fine-tuning instruc- tion types, each designed to enhance structural compre- hension, such as local connectivity. node degree, cycle structure, and path-based dependencies, by addressing specific tasks: (1) Connectivity involves determining whether or not two nodes in an undirected graph are con- nected; (2) Degree requires the L.M to determine the degree of given node based on the adjacency matrix A; (3) Cycle Detection requires the L.IM to ascertain whether cycle exists within the given sequence of nodes; (4) Tezt Generation demands
Original LLM
LLM with scructmi attritutes Finc-tuned strucuual altribats
The performance improve- ments in zero-shot node classification on homophily and heterophily graphs
Figure

---

the LLM to generate textual contents of given nodes based on the semantic attributes of preceding nodes in the random walk. Through fine-tuning, LLM exhibits significant improvements on the zero-shot node classification task, as demonstrated in Figure 2. More detailed task descriptions and detailed task-specific GTA prompt templates are provided in Appendix B.3.
3.3
GNN-preference-drrvenn Node Selector
After being fine-tuned, the LLM can generate superior node label annotations (as shown in Figure 2). However, how to select nodes for LLM's label annotation effectively enhance teacher GNNs (those nodes are assumed to be preferred by GNNs) is challenging problem. Uncertainty is an essential metric for node selection. It mainly consists of two parts: random uncertainty caused by inherent noise and cognitive uncertainty caused by insufficient observation. The former type inevitable, so we focus on the latter type. From the perspective of collective consensus [40], we design the SANN-preference-drre driven Node Selector based on the defined K K-uncertainty (&x). Specifically, we measure the cognitive disagreement among the teacher GNNs' SoftMax outputs using the Kullback-Leible (KL) divergence, and get the preference ranks of all nodes by 8x. i.e., VPR = Sort((v),))
uN óx (x)) ), 5x (v2))..... òx (x())). High K K-uncertainty of nodes indicates that their prediction uncertainty by GNNs is higher. Those nodes can effectively enhance GNNs their more accurate labels, annotated by LLM, are provided to train GNNs, as the following proposition suggests.
Proposition 3.1. These nodes with higher K K-umcertainty (6x) are beneficial for GNNs enhancement.
B Σ (F (1) (v)) x 8, ISKjSB
(x( 2
where 8, is the uncertainty of node v, is defined as E.= (fL(ff (U))) The M(U) is the average prediction probability distribution of all B teacher GNNs (See Definition D.1 1.1 for details). DEL(-1 is the function to calculate KL divergence.
f0* (DL)=
arg min
1 W
Σ C(TS
(3)
D.E(UpR: " u a W 16x (wpp) }r88}
where Dr is the expanded training dataset. f4 is the optimal parameter of teacher GNN, vpp represents the w-th nodes in the preference rank. W is the number of selected nodes by GNS and the ōx is the K K-uncertainty threshold depending on the expansion ration.
The proof is given in Appendix D. By selecting these nodes (illustrated in Figure 3), we ensure that the most uncertain and informative nodes are labeled by the LLM to promote the progress of prediction distillation through the cross-entropy function. Correspondingly, GNS also reduces the inference costs associated with LLMs by not querying all nodes in Dy. To generate high-quality an- notations for GNN-preferred nodes, we further design the Distance- based Neighbors Selection (DNS) module, which performs the K-Nearest Neighbor (KNN) search around each selected node across the embedding spaces generated by pretrained teacher GNNs and deletes peated neighbors. The structure attributes composed of selected neighbors and their textual contents are integrated into the category-inductio induction prompt and inputted into the LLM. Unlike relying solely on neighbors identified by the adjacency matrix (prone to biases from 1-hop homophily), our approach ensures more robust and diverse selection of high-quality neighbors, facilitating better construction of the category-induetto
1.0
0.8
Np
N₂ = 2
0.6
0.4
0.2
0
Np 3
Np = 4
Figure 3: This is exemplified using the CORA dataset. Starting from the arrow and progressing counterclockwise the KI. divergence sum gradually increases, accompanied by a darkening the trian- gle colors. The length of each triangle in- dicates the number of nodes within a spe- cific KL divergence sum range, where Np denotes the number of classes pre- dicted by the teacher GNNs.
prompt for the LLM. We not select common KNN neighbors across all the embedding spaces generated by the teacher GNNs, as they may overfit to the adjacency structure.
5

---

3.4 Nude-preference-driven GNN Selector
Distilling knowledge simultaneously from multiple teachers to the student is not a good option since nodes with varying local topologies require distinct message-passinng -psssing modes for optimal representation updates. To achieve this. we introduce the Node-preferent preference-driven GNN Selector (NGS) to select the most appropriate teacher for each node according to the specific attributes and promote tailored knowledge distillation, For each node the expanded training data (including the initial few labeled nodes and those selected nodes whose labels are annotated by the LLMM,, we construct a node-specific prompt by combining its semantie, structural, and prediction attributes derived from the enhanced teacher GNNs. This prompt is then inputted to the fine-tuned LLM to determine the most suitable teacher for this node. The GNN selection task formulated as a reinforcement learning problem that needs to explore the discrete action space and find series of assignment actions to get the highest global reward across the expanded training data. Through interaction with the training process, the selector progressively refines its decisions and effective assignment strategy. Specifically, the fine-tuned LLM, serving as the agent, selects the most appropriate teacher for each node. The policy is trained to maximize classification accuracy on the expanded training data, with the reward tied to the student's performance. To address the non-differentiabilit of the LLM's decoding process, we add two additional projectors (MLPs) after the logit layer to generate action probabilities and corresponding value estimations, enabling the agent take discrete reacher-selection actions.
ande-to-teacher assignments, leading to more efficient
In the RL framework, the elements are structured as (State, Action, Reward). During each iteration, the agent interacts with the environment by receiving all attributes of one node in the expanded training data. The agent then takes an action on which teacher is more appropriate.
State: Each state corresponds to the prompt P; of a node, including node-specific semantic, structural, and prediction attributes. These prompts are detailed Appendix B.2. The size of the expanded training data is denoted as W.
The Policy Model (the fine-tuned LLM combined with an MLP projector) generates text- related output to indicate its selection from multiple teachers, formulated as a probability distribution vector कт [ : "' .., TT where गт, denotes the probability of selecting the b-th teacher Tù. The action is determined through sampling.
Action:
Reward: The function is correlated with the performance of the student GNN, which is trained by distilling knowledge from the selected teacher for each node, The reward function consists of three key parts: classification accuracy, cross-entropy loss, and distillation loss. It can be written follows:
R = 1j * pC LCE) + (1 (1-1) -) * Ace
(4)
where Acc represents the classification accuracy of the student GNN the expanded training data, n is hyper-parameter to balance the three parts, where LD = 1 Σ. W
o llgg (x,,, and
Lc = - Σ. -og.f
To effectively optimize the agent's actions for better knowledge distillation, we employ the simplified version of Proximal Policy Optimization (PPO) [41] algorithm, which retains the core principles. Specifically, we do not instantiate the Reward Model explicitly and calculate the reward based on the performance of the student GNN. The Reference Model is also not explicitly referenced, because the parameter update objective function we utilize involves a comparison with the previous strategy. To avoid large fluctuations between the current and old policies. we adopt the CLIP strategy [41] to limit the update margin. During the KD process, the parameters få of NGS, remain fixed, while the parameters f of the student GNN are trained. During the NGS process, the parameters f& of the student GNN are kept fixed to compute the reward, while the parameters f% of NGS based on the collected rewards from all episodes are optimized. The pseudocode, detailed implementations, and time complexity analysis are provided in Appendix C.
Experiments
4.1 Experimental Setup
Datasets In order to assess the few-shot node classification performance of our method on TAGs, conduct comprchensive series of experiments across 9 real-world datasets: CORNELL, WASH-

---

Table 1: Node classification accuracies (%) on weal-world datasets. T1, T2, T3, and Ta denote the teacher GNNs for homophily or heterophily graphs (refer to the descriptions in Baselines for more details of the teacher and student GNNs). The OOM stands for Out-Of-Memon The best results highlighted in dark gray, while the runner-up results are marked in light gray.
Methods
Dataset
CORNELL
WASHINGTON
TEXAS
WISCONSIN
AMAZON RATINGS
OGEN- ARXIV
WIKI C8
PUBMED
CORA
7'
58.04g1a
g7.84221
53,45
59.32g91
41 41.22
56.51g
81.57
K3.34
87. 87.79g1.
46.29
65.00,a5
82.83
48.30,a5
36001cs
19.19152
19.08
82.52,21
87.59 jas
T5
44.62:433
55.27
45.19
61.49.00
37.4
56.71.16
80.17
79.57e2a
88.38
32,73
58.33-11
63.64
62.89213
48.90 48.93
53.61es.
72.01aaa
55.15e14
77.07
GCNI (29) LN
57.82
a1.17
68.79
60.91
18. 1.3
35. 35.14
58.29
67.
17.74
EGNN [30 5
53,38
63,33a12
71,72
55,97a50
49.0
36, 36.15
63.97as.
66
12ess
72.85
LLMGNN L.N
52.6344.3
41/09=22
62,8223.
46,54,09
47 47.64.2.0
44. 44.11
66.09a0.4
7884
76 26
GAagLLM (48) (48//#1N 5
62.98
65.13211
73.81
62.20g09
42.42.9.0
53.47
83.10=1.
85.9
79. 29.48
Self-minnng L33
61,90
26.8920
72.62+2.4
66.29209
41 41.99
33.4022.
24.99
2044
83. 13.1
AGST [31
20.
70.09
68.45
70.08_07
43.11
00M
72.49
73.75
77.25
loeBerg [32]/4
33.33 11.9
67.76_29
50.00
41.53_2a
25.99 11.5
g3.63
84.58 c..
62:41,93
76.23_2.
KDGA [35]
S4.39
60.00.b1
66.67 11.5
58.74.39
38 38.06
00M
65.03
ODM
68.87.ss
[36]
51.27
50,39
62. 23.
41.51=02
35 35.60
58.27
62.73=1.9
45.86
51.61
BONN [21]
58.60 58.60253
56.67
65 6642..
59.12
37.53 37.5322.0
16.67es.1
56.96245
76. 12
71 71.28
[22]
72.68-114
73.33g0
80.81
71.69219
39. 39.54.0.8
32.32g5.5
65 64
83.42gx
79. 16g10
PairGKD [37]
61.05
60.00
84.85
57.11_oa
43.9
42.03
60.2511.a
70.40.6.
09.85
RANDOM/ / L
S4.31
$8.04_12
58.93
58.04_21
54.97 $7.9522.
68.27
66.0022
70.64226
VOTING/ / S
44.97
58.88
61.3
46.97
58.64
58.53g2.0
72.28
70.64
74.32g11
1 1
74.60
76.64_n0
80.36
69.32
64.11
53.67
83.75 g9.31
85.64ga1
PKD
L
16.72
81.36
g3.3
;1.49
65.61
58.65
80.01 1..6
31.34
86. 18.17
80.95
83.74
86.31
76.89299
66.79
61.05
81.39
85.69
91 14
INGTON, TEXAS, WISCONSIN [25], AMAZON RATINGS [42], OGBN-ARXIV [43], WIKI CS [44], PUBMED, CORA [45]. They have various l-hop homophily ratios [46] and additional details of the datasets can be found in Appendix A. For the KD-baselines, we partition the nodes of each graph into training, validation, and test sets, allocating 48%, 32%, and 20%, respectively, based on the proportion division mentioned in [47]. For PKD and other baselines, we randomly select 3, and 5 labeled nodes per class as the initial training data and then expand the dataset to 48% of the total using the GNS module. The remaining data randomly split into 32% for validation and 20% for testing, with the preserved indices for the baselines. This operation is repeated 5 times. We report the average test classification accuracy and standard deviation of each model with parameters that lead to the peak validation accuracy.
Baselines We compare our method against the following baseline models: (i) Advanced GNNs:
GCNII [29] and EGNN [30]; (ii) GNNs enhanced by LLMs: LLMNNN [14] and GAugLLM [48]; (ii) self-training for graph learning: Self- trainiriiinn [33], AGST [31] and IceBerg [32]: (iv) Knowledge
Distillation (KD) for GNNs: KDGA [35], MSKD [36]. BGNN [21], MTAAM [22], and FairGKD [37].
For homophily graphs, the teacher GNNs used are: GCN [4] (T1), GAT [23] (T2), APPNP [24] (Ts).
H₂GCN [25] (Ta), and the student is GCN; for heterophily graphs, the teacher GNNs employed
are: DirGNN [28] (Ti), GPRGNN [26] (T2), HoloNets [27] (Ta), H₂GCN (T,) and the student is
H₂GCN. The LLM used in the experiments Llama-3. 1-B-a-..--B--nnstuue [49].
4.2 Performance Analysis and Discussion
Notably, # LN 1, # LN 3, # LN 5 indicate only labeled nodes per class are used for training PKD, while the results of the teacher GNNs ([7:)/=)) and other baselines are trained under the data splitting of 48%/32%/20% as mentioned above. According to Table 1, our method almost achieves the best second-best accuracy results.
Due to the extreme insufficiency of labels, GCNII and EGNN are restricted in further improvement, although they have distinctive network architectures. Lacking carefully designed fine- fine-tuning and enough cognition makes LLMGNN fail to produce high-quality pseudo labels and is dramatically defeated by our method PKD. Although GAugLLM harnesses LLM for feature and structure aug- mentations to benefit GNN, its self-training depends only on SoftMax scores to identify candidate nodes to assign pseudo-labels, method that can sometimes be unreliable. GAugl.MM achieves the best result on the PUBMED dataset, but it is outperformed by PKD on other datasets. AGST is excessively dependent on the original graph topology for label propagation, rendering it vulnerable
7

---

to structural noise and facing significant challenges when transferred to large-scale graphs, such as OGBN-ARXIV. IceBerg does not perform well on heterophily graphs because its capacity to disseminate information across longer distances is hampered by the proliferation of noise edges. MTAAM shows satisfactory performance on most datasets, due its ability to autonomously identify the most valuable knowledge from each teacher during training. FairGKD achieves runner-up results on some datasets. The poor performances of KDGA and BGNN result from their excessive sensitivity to GNN selection. MSKD is equipped with the fixed geessage-passngg mechanism, showing that the single message-passing mechanism underperforms on all the datasets compared to PKD. The RANDOM # LN 5 approach refers randomly selecting node predictions from 4 teachers, utiliziig 5 labeled nodes per class to train teacher GNNs. The VOTING # LN 5 method selects the most frequently predicted label from 4 teachers the annotation label. We can see that these two simple and intuitive strategies are defeated by PKD on all datasets.
Our PKD consistently achieves superior node classification results across all datasets, irrespective of the specific type of L.M.. The few-shot node classification results after replacing Llama-3. 1-8B- Instruct with Dwwen.55-BBIInstie [50] and Mixtral-7B I-nnt--utttttttt.....333 [51] are shown in Table 2.
Table 2: Few-shot node classification accuracy (%) on eight TAGs using three different LLMs. The # LN 1, # LN 3, # LN 5 represent 1, 3, 5 labeled nodes per class, respectively. The best results are highlighted in dark gray, while the runner-up up results are marked light gray.
Methods
Dataset
CORNELL
WASHINGTON
TEXAS
WISCONSIN
AMAZON RATINGS
OCBN-
WIKI CS
PUBMED
CorA
#LN1
73,54_24
75.70a1.
82.14gna
22.59.1.5
74.58:1.1
54.17.2a
79.49
82.81ac.
86.45.1o
PKD
77,25-14
77.35
84.52=0.
73.8620.7
75.46=08
60.63-1.2
80.01
83.61
87.7420
#LN5
79.84g04
79.63
85.71an2
74.24.0.2
77.69
62.62:23
81.21
85.96
90.07
76.31=22
74.42a1.
79.41
69.81
70.02+12
27.69
80.56.a4
82.42
84.87=24
PKDsionl
#
78.95
76.74
82.86
75.47 jo.s
71,50
61.17
81.96.1.
83.19
87.64
#LN
81.58,24
81.39a25
27. 17.36.59.
73.96
62.44.0.8
83.33=14
84.71as.
88 56.0.77
Furthermore, to evaluate the quality of -enerrreertted pseudo-labels, we compare the node classifi- cation performance of PKD and three baselines under different label settings (# LN 5, 48% training ratio expanded by the annotated labels and real labels, respectively). The experiments are conducted on four datasets (CORA, WIKI CS, WASHINGTON, and WISCONSIN). The results are presented in Table 3. For GCNII and IceBerg, they are proposed to tackle the challenge of sparse labels, using the L.LM-annotated node labels can improve their performance on all datasets. However, using the same number of real labels achieves better performance.
Table 3: Classification accuracy comparison under different label configurations. The best results are highlighted in dark gray, while the runner-up results are marked light gray.
Models
Labels configuration
Cora
WIKI CS
WASHINGTON
WISCONSIN
# LN 5
77.74
56.29
64.17
60.94
GCNI
48% L.M M.generrtted labels
76.69
51.18
70.83
62.50
48% labels
81.54
59.17
71.79
65.98
LN 48% generrtte labels
76.23
84.88
67.76
41.53
IeeBerg
78.66
71.23
70.12
42.22
48% real labels
81.94
86.49
72.04
45.43
# 5
43.91
46.81
45,29
33.33
MSKD
48% LLM--gnnrated labels
45.89
54.06
48.17
39.50
48 % reul labels
51.61
62.73
50,39
41.51
PKD
# L
90.27
81,39
83.74
76.89
4.3 Ablation Study
Generally, the fine-tuned LLM using our proposed GTA prompts also demonstrates pretty zero-shot node classification performance, surpassing some semi-supervised GNNs from the values in Figure 2.
We assess the significance of the GTA prompts, DNS and VPR with the following default parameter settings: # LN = 3, K = 4. Here, K denotes the number of selected neighbors surrounding the node, be annotated, within each embedding space of the teacher GNNs structure attributes. In the absence of DNS, neighbors are selected according to the adjacency matrix directly; in the non-use of VpR. we expand the training data by random selection.
8

---

Table 4: Ablation study for GTA, DNS, and VpR. ft denotes accuracy (%) increment. The three components play different roles in the improvement of the performance of our method.
Dataset/Modeule
GTA
DNS
VPr
Accuracy
Datase//Module
GTA
DNS
VPR
Accuracy
45.02
42.01
CORA
ft 26.94
AMAZON RATINGS
13.02
ft 30.99
16.96
ft 41.14
23.97
As shown in Table 4, the implementations of GAT prompts, DNS, and VpR result vary- ing degrees of performance improvement. Sup- ported by fine-tuning with GTA prompts, the LLM's enhanced logical reasoning ability, com- bined with high-quality neighboring nodes, sub- stantially enhances zero-shot node classification capability, leading to superior classification per- formance improvement. Additionally, we also assess the methods without using reinforcement learning in the teacher selection process, includ- ing entropy-based ranking, i.e., selecting the teacher GNN with the highest prediction confi- dence, random selection, and end-to-end learn- ing. Their relevant results are provided in the Appendix E.2.
Accuracy
90
30
Rg Age
R2 Axc lix
10
Rg Age lax Luu.
Epoch
40
80
120
160
200
Figure 4: The comparison of different Rewards. When including all three parts simultaneously. our method (the curve in green) performs the best.
To assess the effectiveness of each part the re- ward function (Eqn. (4)), we visualize the train-
ing processes of three variants in Figure (a) Rí The reward function for teacher GNN selection depends solely on the classification accuracy of the student GNN on the expanded training data; (b) R2: In addition to classification accuracy, the reward function also incorporates the negative cross-entropy loss (-Lcg): (c) Ry: Building upon R2, the reward function also includes the negative knowledge distillation loss (-LDL). As shown in Figure 4, both the three parts contribute to the improved classification performance.
4.4 Sensitivity Analysis
We investigate the impact of the hyper- parameter K on the zero-shot node classification performance. We vary the value К within 2, the range 2, 3, 4, 5} for homophily graphs and heterophily graphs observe the variation in zero-shot node classification accuracy. As illustrated in Figure 5, accuracy exhibits signif- icant fluctuations as K changes. When K = 4, fine-tuned LLM demonstrates strong perfor- mance on most graphs.
Accuracy
90.
Cora
Pubmed
50
Texas
Comeel
40
Wisconsin
Washington
To further explore the relationship between the parameter scale of LLM and PKD's perfor- with
K
mance, we evaluated Dwen2.5-7BBInntteee three different parameter scales: 7B/14B/32B parameters. The results are shown in Table Obviously, the classification performance of PKD basically gets better with the increase of parameter scale. This is mainly related to the L.Mss with larger parameter-scale have richer knowledge storage and better ability handling complex tasks.
Figure 5: The effects of K on homophily and het- erophily graphs. When K = 4, zero-shot node classification accuracy of the fine-tuned LLM is the highest on most graphs.

---

Next, we investigate the ratios of nodes selected for anno- tating their labels by the LLM as means expand the training set. The results are given in Table 6. Increasing the expansion ratio can enhance the performance of PKD. This improvement can be attributed not only to the high- quality label annotation generated by the fine-tuned LLM, but also to the characteristic of PKD that is underpinned by selecting each node the most appropriate teacher GNN for knowledge distillation.
Table 5: The few-shot node classifica- tion accuracy (%) of PKD with different parameter-seale LLM.
Datasets
CORA
Purameter scaless
7B
14B
32B
PKD
90.07
90.58
91.54
Datasets
PUBMED
Parameter scales
14B
Furthermore, we perform the hyperparameter sensitivity analysis over the loss-weight coefficients a, 3, 7) For the sensitivity analysis of a, we set 3 = 1, 1, 0.5. This strategy also applies to the sensitivity analysis of , and As for 7) we fix a, 9, and to their respective best values.The is related to the supervision signal provided by the LLM-generated annotation labels and vy is related to the student GNN's s performance. All the results are reported in the Appendix E.3.
PKD
86.64
and are more sensitive than the other two hyperparameters,, because
Table 6: The results with different node annotation ratios. The best results are highlighted in dark gray, while the runner-up results are marked in light gray.
Node annotation ratios
10% L
20% L
30% L
40% LNS
48% / LN
AMAZON RATINGS
50.27
55.91
62.07
63,93+1.1
66.79
CORA
13.37
77.51
81.49128
83.27
91.14103
4.5 Running Time
We also study the training efficiencies of PKD and all baselines. The running times on CORA are shown in Table 7. There trade-off between accuracy and time complexity. The incorporation of the LLM undoubtedly boosts the few-shot classification accuracy of GNNs on TAGs, but the training time increases. When applied the bigger graphs, the time increase will be more obvious.
Table 7: Running time (second per epoch) of each method, including the pretraining process.
Datasets/ / Methods
Ti
T2
T3
T.
GCNII
EGNN
LLMGNN
GAugLLM
CORA
0.006
0,197
0,247
0.035
0.014
0,366
0.630
0.402
Datasets/ Methods
Self-training
AGST
IceBerg
KDGA
MSKD
BGNN
MTAAM
FairGKD
PKD
CORA
0.016
0.011
3.911
2.289
1.001
3.318
4.100
7,314
5 Conclusions, Limitations & Future Work
In this work, we have proposed a preference-driven knowledge distillation (PKD) framework for few-shot node classification on TAGs, consisting of CONN-rrffrrence-rriee Node Selector (GNS) and Node-preference-diee
GNN Selector (NGS). Fine-tuned with our proposed GTA prompts, the refined LLM generates high-quality annotations. The GNS effectively determines nodes for the fine-tuned LLM to annotate and promotes knowledge distillation from the LLM to teacher GNNs. The NGS tailors for each node the most appropriate message- saaaggeeppssinggggg mechanism, promoting knowledge distillation from teacher GNNs to the student GNN. On various real-world TAGs, our method PKD outperforms almost all advanced GNNs and KD methods for few-shot node classification while using only a few node labels. One limitation of our method is that it is designed for TAGs. Moving forward, we plan to further explore more efficient mechanism of synergizing LLM and GNN to address the limitation of training efficiency as well as datasets beyond TAGs.
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their valuable and constructive comments. This work was supported partially by the National Natural Science Foundation of China under Grants 62176184, 62476109, and 62206108, and the Fundamental Research Funds for the Central Universities.
10

---

References
[1] Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. In Proceedings of the Advances in neural information processing systems, volume 34, pages 28798-28810, 2021.
[2] Jiarui Feng. Hao Liu, Lecheng Kong, Mingfang Zhu, Yixin Chen, and Muhan Zhang. Taglas: An atlas of text-attributed graph datasets in the era of large graph and language models, arXiv preprint arXiv:2406. 14683, 2024.
[3] Shujie Li, Yuxia Wu, Chuan Shi, and Yuan Fang. Hetgb: A comprehensive benchmark for heterophilic text-attributed graphs. arXiv preprint arXiv:2503.04822 2025.
[4] Thomas N Kipf and Max Welling. Semi-super classification with graph convolutional networks. arXiv preprint arXiv:160.00222
2016.
[5] Seongjun Yun, Minbyul Jeong, Rachyun Kim, Jaewoo Kang. and Hyunwoo J Kim. Graph transformer networks. In Proceedings of the Advances in neural information processing systems, volume 32, 2019.
[6] Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang, Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. Textgnn: Improving encoder via graph neural network in sponsored search. In Proceedings of the Web Conference 2021, pages 2848-2855 2021.
[7] Xiangting Shi, Yakang Zhang, Abinash Pujahari, and Sambit Kumar Mishra. When latent fea- tures meet side information: A preference relation based graph neural network for collaborative filtering. Expert Systems with Applications, 260: 125423, 2025.
[8] Xiaofeng Cao, Mingwei Xu, Xin Yu, Jiangchao Yao, Wei Ye, Shengjun Huang, Minling Zhang, Ivor W Tsang, Yew Soon Ong, James T Kwok, al. Analytical survey of learning with low-resource data: From analysis to investigation. arXiv preprint arXiv:2510.0896 2025.
[9] Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al. Exploring the potential of large language models (Imm) in learning graphs. ACM SIGKDD Explorations Newsletter, 25(2):42-66 2024.
[10] Duo Wang, Yuan Zuo, Fengzhi Li, and Junjie Wu. Lims as zero-shot graph learners: Alignment of gnn representations with Ilm token embeddings. In the Proceedings of the Advances in Neural Information Processing Systems, volume 37, pages 5950-5973, 2024.
[1I] Yuxia Wu, Shujie Li, Yuan Fang, and Chuan Shi. Exploring the potential of large language models for heterophilic graphs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5198-5211, 2025.
[12] Xiaoxin He, Xavier Bresson, Thomas Laurent, Adam Perold, Yann LeCun, and Bryan Hooi. Harnessing explanations: LLM-to-LM interpreter for enhanced text-attributed graph repre- sentation learning. The Twelfth International Conference on Learning Representations. 2024.
[13] Hinton. Distilling the knowledge in neural network. In Deep Learning and Representation Learning Workshop in Conjunction with NIPS, 2014.
[14] Zhikai Chen, Haitao Mao, Hongzhi Wen, Haoyu Han, Wei Jin, Haiyang Zhang, Hui Liu, and Jiliang Tang. Label-free node classification on graphs with large language models (Ilms). In The Twelfth International Conference on Learning Representations, 2024.
[15] Shengxiang Hu, Guobing Zou, Song Yang, Shiyi Lin, Yanglan Gan, Bofeng Zhang, and Yixin Chen. Large language model meets graph neural network in knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 17295-17304, 2025.
11

---

[16] Bo Pan. Zheng Zhang, Yifei Zhang, Yuntong Hu, and Liang Zhao. Distilling large language the models for text-attributed graph learning. In the Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1836-1845, 2024.
[17] Renée G Fox. Training for uncertainty. In The student-physiceee Introductory studies in the sociology of medical education, pages 207-242. Harvard University Press, 1957.
[18] Jing Liu, Tianai Yue, Chuanguang Yang, Yuqi Li, Qinfen Hao, Xiang Li, and Shiping Wen. Distilling multi-teacherr knowledge from distinct graph neural networks. Available at SSRN 5084903, 2025.
[19] Kai Wang. Yu Liu, Qian Ma, and Quan Z Sheng. Mulde: Multi-teacher knowledge distillation for low-dimensional knowledge graph embeddings. In the Proceedings of the Web Conference 2027, pages 1716-1726, 2021.
[20] Ying Jin, Jiaqi Wang. and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 24276-24285, 2023.
[21] Zhichun Guo, Chunhui Zhang, Yujie Fan, Yijun Tian, Chuxu Zhang, and Nitesh V Chawla. Boosting graph neural networks via adaptive knowledge distillation. In the Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7793-7801, 2023.
[22] Bo-Wei Yang, Ming- Yi Chang, Chia-Hsun Lu, and Chih- Ya Shen. Two heads are better than one:
Teaching mips with multiple graph neural networks via knowledge distillation. In the Proceedings of the International Conference on Database Systems for Advanced Applications, pages 452-462, 2024.
[23] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[24] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations, 2019.
[25] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In the Proceedings of the Advances in neural information processing systems, volume 33, pages 7793-7804, 2020.
[26] Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021.
[27] Christian Koke and Daniel Cremers. Holonets: graphs. In The Twelfth International Conference on Learning Representations, 2024.
Spectral convolutions do extend directed
(28) Emanuele Rossi, Bertrand Charpentier, Francesco Di Giovanni, Fabrizio Frasca, Stephan Günnemann, and Michael M Bronstein. Edge directionality improves learning on heterophilic graphs. In the Proceedings of the Learning on Graphs Conference, pages 25-1, 2024.
[29] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In the Proceedings of the International Conference on Machine Learning, pages 1725-1735, 2020.
[30] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In the Proceedings of the International Conference on Machine Learning, pages 9323-9332, 2021.
[31] Kaize Ding, Elnaz Nouri, Guoqing Zheng, Huan Liu, and Ryen White. Toward robust graph semi-supervised learning against extreme data scarcity. IEEE Transactions on Neural Networks and Learning Systems, 2024.
12

---

[32] Zhixun Li, Dingshuo Chen, Tong Zhao, Daixin Wang, Hongrui Liu, Zhiqiang Zhang. Jun Zhou, and Jeftrey Xu Yu. Iceberg: Debiased self-training for insslanmbal node classification. In Proceedings of the ACM on Web Conference 2025, pages 3160-3170 317,, 2025.
[33] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-superviied learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[34] Huaxiu Yao. Chuxu Zhang, Ying Wei, Meng Jiang, Suhang Wang, Junzhou Huang, Nitesh Chawla, and Zhenhui Li. Graph few-shot learning via knowledge transfer. Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6656-6663, 2020.
[35] Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z. Li. Knowledge distillation improves graph structure augmentation for graph neural networks. In the Proceedings of the Advances in Neural Information Processing Systems, volume 35, pages 11815-11822 2022.
(36] Chunhai Zhang, Jie Liu, Kai Dang, and Wenzheng Zhang. Multi-scale distillation from multiple graph neural networks. In the Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4337-4344 2022.
[37] Yuchang Zhu, Jintang Li, Liang Chen, and Zibin Zheng. The devil in the data: Learning fair graph neural networks via partial knowledge distillation. In the Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 1012-1021, 2024.
[38] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In the Proceedings of the Advances in neural information processing systems, volume 31, 2018
[39] Heng Wang, Shangbin Feng. Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language? In the Proceedings of the Advances in Neural Information Processing Systems, volume 36, 2024.
[40] Dongyang Fan, Celestine Dünner,, and Martin Jaggi. Collaborative learning via prediction consensus. In Proceedings of the Advances neural information processing systems, volume 36, pages 1988-2009, 2023.
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1xXiv:177706644 2017.
[42] Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of GNNs under heterophily: Are we re- ally making progress? In The Eleventh International Conference on Learning Representations, 2023.
[43] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs Advances in neural information processing systems, 33:22118-22133 2020.
[44] Péter Mernyei and Cătälina Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901 2020.
[45] Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine leurning, pages 40-48, 2016.
[46] Jiyy Yang, Sourav Medya, and Wei Ye. Incorporating heterophily into graph neural networks for graph classification. In 2024 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1554- 2024.
[47] Langzhang Liang, Sunwoo Kim, Kijung Shin, Zenglin Xu, Shirui Pan, and Yuan Qi. Sign is not remedy: Multiset-to-multiise message passing for learning on heterophilic graphs. In Proceedings of the 41 International Conference on Machine Learning, volume 235, pages 29621-29643, 2024.
13

---

[48] Yi Fang, Dongzhe Fan, Daochen Zha, and Qiaoyu Tan. Gauglim: Improving graph contrastive learning for text-attributed graphs with large language models. In the Proceedings of the 30th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pages 747-758, 2024.
[49] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[50] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1:51:52412.15115. 2024.
[51] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:240J 04008, 2024.
[52] Leskovec Jure. Snap datasets: Stanford large network dataset collection. Retrieved October 2025 from http://snap. stanford. edu/data, 2014.
[53] Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. NV-embed: Improved techniques for training LLMs as generalist embedding models. In The Thirteenth International Conference Learning Representations, 2025.
[54] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for all: Towards training one graph model for all classification tasks. In The Twelfth International Conference on Learning Representations, 2024.
[55] Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In the Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701-710, 2014.
[56] Wenbo Shang, Xuliang Zhu, and Xin Huang. Path-llm: A shortest-patt path-based Ilm learning for unified graph representation. arXiv preprint arXiv:2408. 05456, 2024.
[57] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[58] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022
[59] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. the Proceedings of the Advances in neural information processing systems, volume 32, 2019.
[60] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arxiv preprint Xrviv: 0903, 2019.
[61] David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15:201-221, 1994.
[62] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Proceedings of the Advances in neural information processing systems, volume 32, 2019.
[63] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
14

---

A Detailed Description of Datasets
Table 8: Statistics of datasets. The Hom. ratio means 1-hop homophily ratio.
Dataset
CORNELL
WASHINGTON
AMAZON
OGBN-
WIKI
PUBMED
Hom. ratio
0.1504
0.1545
D.1989
0.2109
0.4777
0.6522
0,6588
0.7924
08252
# Node
189
214
168
264
5068
169343
11701
19717
2708
1166243
88648
10556
Features
1703
1703
1703
1703
128
500
1433
Classes
5
40
10
3
7
Domain
Web
Web
Web
Web
Co-purchasc
Co-citation
Wikipedia page
Co-citation
Co-citation
CORNELL, WASHINGTON, TEXAS, and WISCONSIN:
These four datasets are derived from the WEBKB webpage dataset, collected from the computer science departments of various universities. these datasets, nodes represent web pages, while edges denote hyperlinks connecting them. All words from the given web pages are collected the features for the nodes. The webpage categories can be listed as following: Student, Project, Course, Staff, Faculty.
AMAZON RATINGS: This dataset is derived from the AMAZON product co- purchasing network metadata, sourced from the SNAP datasets [52]. Nodes represent products (Books, Music CDs, DVDs, Videos) and edges signify relationships between products that are frequently co-purchased. The task involves predicting the average rating assigned to each product by reviewers. The possible rating values are grouped into five distinct classes. For node features, we utilize the NV-Embed-v2 [53] embeddings generated from the product descriptions. To reduce the size of the graph, we only consider the largest connected component of the 5-core of the graph.
WIKI CS:
WIKI CS is a graph derived from the Wikipedia platform. The nodes WIKI CS represent Wikipedia page descriptions, while the edges correspond to hyperlinks between distinct pages. The WIKI CS dataset and its raw text [44] are sourced from OFA [54]. The graph consists of 11,701 nodes and 216,12 123 edges. The WIKI CS dataset is suitable for node classifica- tion tasks. The WIKI CS dataset is categorized into 10 distinct categories: Computational Linguistics, Databases, Operating Systems, Computer Architecture, Computer Security, Internet Protocols, Computer File Systems, Distributed Computing Architecture, Web Technology, Trogramming Language Topics
CORA, PUBMED, and OGEN-ARXIV:
The CORA dataset represents a co-citation graph of computer science research papers. The dataset sourced from OFA [54], with the original data derived from [9]. In [9], the authors recollect the dataset due to the commonly employed bag-of-words features in the widely used CORA dataset within the GNN community, where raw text difficult to retrieve. The revised CORA dataset contains 2,708 nodes and 10,556 edges, matching the specifications of the original dataset. The dataset is di- vided into categories: Theory, Reinforcement Learning, Genetic Algorithme, Neural Networks, Probabilistic Methods, Case-Based Rule Learning.
The PUBMED dataset represents a co-citation graph of biomedical research papers focused on diabetes mellitus. The source and processing procedure of PUBMED are identical to those of CORA. After processing, the dataset consists of 19.717 nodes and 88,648 edges. The dataset is classified into categories: Experimentally Induced Diabetes, Type Diabetes, Type 2 Diabetes.
The OGBN-ARXIV dataset citation graph of papers from the arXiv platform. It is collected from the Arxiv dataset and its raw text as OGB[[33 and OFA [54]. There are 169,343 nodes and 1,166,243 edges in the graph. It contains 40 sub-categories of compute science.
15

---

B
Detailed Prompts
We provide all specific prompt templates in the following for zero-shot node classification, Node- preference-driven GNN Selector and GTA Prompts, respectively.
B.1 Prompts for Zero-shot Node Classification
The complete prompts for zero-shot node classification provided below. Similarly, for each dataset, we refine specific descriptions to ensure contextual coherence.
Table 9: The prompt template for zero-shot node classification.
Role
Prompt
System Prompt
Papers in this field can be divided into 7 categories: [Case Based, Genetic Algorithms, Neural Networks, Probabilistis Methods, Reinforcement Learning, Rule Learning, Theory]. You will serve as an assistant to help me to classify this target paper into the categories above according to description and related papers' descriptions, who may be of the same category as this target paper. I will provide you with the descriptions of this target paper and its related papers.
Here are the instructions:
will provide you with information in the form of a JSON string that describes the target paper:
Title: the title of this target paper. Abstract: the abstract this target paper.
Related Title: the title of the related paper. Related Abstract: the abstract of the related paper.
Related Title: the title of the related paper. Related Abstract: the abstract of the related
paper.
Requirements:
O Please provide your response in JSON format, following this structure: Briefly explain your reasoning process for the predicted category. The best category you predict for this paper, this category must belong to these 7 categories: [Case Based, Genetic Algorithms, Neural Networks, Probabilistic Methods, Reinforcement Learning, Rule Learning, Theory];
Reasoning:
Category:
There 2000 words limits for the reasoning:
Do not provide any other text outside the JSON string:
Focus only on content in the actual text and avoid making false associations;
The output can only contain category and reasoning.
User Prompt
Title: Latel Abstract: tchstroct-
Related Title th Gie
Related Abstract: t.1
Eghstract"
Related Title: t22
Related Abstract: ghatreett
yitle
Related Title: Lite
Related Abstract:
abstract
B.2 Prompts for Nade-preference-driven GNN Selector
Unlike the prompts used zero-shot node classification described above, do not collect responses from the LLM; instead, we focus solely on the outputs generated by the subsequent projector. Similarly, for each dataset, refine certain descriptions maintain contextual consistency.
16

---

Table 10: The prompt template for Node-prefference-rivee GNN Selector.
Role
Prompt
System Prompt
There are four names of teacher networks: [APPNP, GCN, HyGCN, GAT]. We need to perform knowledge distillation for each node in this graph consist of nodes (papers) and edges (citation relationships). You will serve to assign the best teacher network for the target node (paper) based on the following information. will provide you with three kinds of attributes of the target node (paper).
assistant help me
Here are the instructions:
will provide you with information in the form of JSON string that describes the node (paper):
Semantic attrihutes: the title and abstract of this paper.
Structure attributes: four teacher networks' logit output of this target node.
Prediction aftributes: important neighbors (papers), which are closely related the target node (paper) and their contents.
Requirements:
Please provide your response in JSON format, following this structure:
Reasoning: Briefly explain your reasoning process for the selected teacher network.
Teacher network: The best teacher network you assign for this node (paper), this
result must belong to these 4 teachers: [APPNP, GCN, H,GCN, GAT];
There are 2000 words limits
the reasoning:
Do not provide any other text outside the JSON string;
Focus only on content in the actual text and avoid making false associations;
The output only contain teacher network and reasoning.
User Prompt
Semantic attributes: It is the content description of this target paper: t. Structure attributes: It has following important neighbors (papers), which are closely related the target paper. Their content descriptions are:
Prediction attributes:
The APPNP's logits output of this target paper is str st((ZAPNN)).
The GCN's logits output of this target paper is str (ZGCN),
The H,GCN's logits output of this target paper is str( ZH,GON),
The GAT's logits output this target paper is str(ZGAT)
B.3 Graph Topology Aware (GTA) Prompts
Generating effective prompts for graph-based tasks can be challenging for LLMs, due the inherent complexity of graph structures and relationships that must be accurately represented. To address this challenge, we propose structured-tase text for graph topology aware, designed specifically for fine-tuning LLMss.
Connectivity This task determining whether or not two nodes U, and By in an undirected graph are connected. Specifically, we randomly select node pairs U/ E V and ask whether not an edge exists between them the graph, answering with a "True/Falsee response. To ensure prompt diversity, only one-third of the possible node pairs are selected for each graph.
TASK 1:
TASK 2:
Degree The degree of node, D, is the number of nodes directly connected it. In this task, we group nodes based on their degree and select node e from group. The LLM is then given the node's is local structure according the adjacency matrix A, and is asked for the degree of the node. To prevent repetitive prompts, only one-third of the nodes from each degree group are selected.
TASK 3: Cycle Detection A cycle in undirected graph without self-loop is path where the first and last nodes are the same. This task requires the LLM to answer whether a cycle exists the given sequence of nodes, {ui,.., ..., Uf U1, v. We generate random walks [55] of length greater than
17

---

10 and arrange them into node sequences. After describing their neighbors information (derived from the adjacency matrix A), the LLM is then asked whether or not any sequence of nodes forms a cycle.
TASK 4: Text Generation We randomly select a node set W = {1; [ as the source nodes, and breadth-fiss search (BFS) conducted from each source node identify nodes graph at a distance greater than edges from 2, which are collected as target nodes. Redundant nodes are removed via the long-to-short path conversion module [56]. The LLM is tasked with generating textual descriptions of target nodes based on the semantic attributes of the preceding nodes in the path.
Specifically, TASK enhances the LLM's ability to identify neighboring nodes and understand the structure of local neighborhoods; TASK strengthens the LLM's ability recognize the significance of node degrees within the graph context; TASK 3 reinforces the LLM to reason about complex graph topologies, such as cycles and long-range node dependencies; TAsK 4 improves path-based reasoning and contextualization of nodes in the local graph structure.
Table 11: The prompt template for the TASK 1: Connectivity.
Role
Prompt
System Prompt
You will serve as graph machine learning expert in connectivity detection to help me determine whether the edge exists between the given two targeted nodes. There is a undirected graph consisting of papers (nodes) and the citation relationships (edges) between them. I will provide the information of the two targeted nodes and their neighbors, consisting of indexes, textual content.
Here are the instructions:
I will provide you with information in the form of a JSON string that describes the target papers:
The first targeted paper: Node index: ...; Title: ...; Abstract: ...;
The kes neighbor: Index:...: Title: ...; Abstract:
The second targeted paper: Node index: ...; Title: ...; Abstract: ...; The kes neighbor: Index:...: Title: ...; Abstract:
Requirements:
Please provide your response in JSON format, following this structure:
Reasoning: Briefly explain your reasoning process for the selected teacher network.
Answer: You only can select one from [True, False) as the best answer;
There 2000 words limits for the reasoning:
Do not provide any other text outside the JSON string:
Focus only on content in the actual text and avoid making false associations;
The output can only contain answer and reasoning.
User Prompt
The first targeted paper:
Node index: i; Title: t Abstract: tabsttaat
The kes neighbor's node index: 13 Title: tyate Abstract: ahstractt
The second targeted paper:
Node index: j; Title: th Abstract: th &abstract
The kin neighbor's node index: T2 Title: t Abstract: ubsttrcct
The full prompts for Connectivity is presented above. When generating prompts for different datasets, we adjust certain descriptions better align with the specific context. For example, when constructing prompts for TEXAS, the background description should be adapted to reflect web pages, and the relationship should be revised hyperlinks, along with other specifc adjustments.
18

---

Similarly, for each task, the prompts must also be modified to correspond to the specific content described in Section 3.2.
C
Implementation Details and Time Complexity Analysis
Algorithm 1: The training of PKD.
Input: Gr (V, & X, A, T), training dataset with true labels Dr, teacher GNNs (Th) ) with parameters {JA, Yom student GNN 5 with parameter f3. fine-tuned LLM LLM , Policy Model f2. Value Model fū, epoch number of RL L
Output: The expanded training dataset DL, optimized parameters LLM" 20" f JV and predicted labels y.
Dr Di:
LLM" LLMM:
3 Filter out -NN-preference nodes based on the preference rank VeR and get their annotations from LLM";
Conduct prediction distillation from LLM to (19 )2 for retrain them
5 for 1 L1 do
Shuffle D, to get new training sequence;
Complete prompts {P.). W 1 for each selected nodes;
for each node UpR E Dr do
NSG select teacher GNN for upR and get one-hot vector m;;
10
Update the parameter S and get reward R, by Eqn. (I);
11
Store (P;, m,, R2;) to the episode history F;
12
Update the parameter S and f& by Eqn. (5) and Eqn. (6)
return LLML . :
First of all, we outline the training setup employed for the experiments detailed in Section 4.2. Uniform training pppee-parameter are applied across all baseline models and datasets. Specifically, the following hyper-parameter values are utilized: the hidden dimension is set to 128. We use ReLU activation functions in all our baseline models. The Adam optimizer is utilized with learning rate of 1 x e and weight decay of 5 x e-A. We train each baseline for 600 steps and select the best step based on the validation accuracy. In our proposed method, we train the student 5 epochs after GNN selection driven by node attributes every time and train the agent 200 epochs. The other weight hyper-parameters are set as follows: a 0.5, 3 1, 0.1, = 0.3, er 0.5, C2 = 0.01, e = 0.2.
Additionally, the parameters of Action Model and Value Model are updated as follows:
S PA
(LA + cıLv e2 ((T))
(5)
pv Cv
(6)
where fl and fl represent the trainable parameters of the Policy Model and Value Model, respectively. PA and Py are their learning rates and V 1a and V are the gradients of their parameters. La and Ly are objective functions belonging to the Policy Model and Value Model, respectively. C1, C2 are hyper-parameters to balance weights. H (TT) is employed enhance the entropy of the policy and promote sufficient exploration. Based on the CLIP strategy [41], the final objective function of the Policy Model is:
CA = E, miin(r(ff))) clip(r:(f))) c. +,
where E, represents the expectation in the time step i. r=( is the ratio of the i-th policy the (i 1)-th policy. Å, is the advantage estimation in the current step, denoting how good or bad the Action is. i hyper-parameter, which determines the range of the CLIP operation.
19

---

The objective functions of the Value Model and H (TT) are:
Lv = E. [E - P2
(8)
H H = - [ (Ar PoggllogTTT (ArlP:))
where (P,) and It denote the Value Model's estimation of State P. and the target value of real Reward R2, respectively. Ar denotes the specific action and 12 (Ar is the probability that Policy f takes action Ar in state P..
The detailed training procedure shown in Algorithm
The specific analysis of the time complexity of PKD training and testing are provided below:
The time complexity of PKD training is mainly divided into three parts: LLM fine-tuning (Line 2). LNN--refcreeccedrrivee Node Selector (Line 3-4) and fode-preferencce-rrvv GNN Selector (Line 5-12). The MUNN-preference-rrive Node Selector also be divided into the annotations generation and prediction distillation.
Table 12: The GTA fine-tuning configurations on Llama-3.18BB-nntruue
Model Name
Dataset Size Epoch lora lora alpha|
Optimizer
[earrning Rate Time Cost
Llama-3.1-88 1-.8--BIIsstrc
53,617
4
AdamW [57]
1e-1
9h 41m 48s
First, Low-Rank Adaptation (LoRA) strategy [58] for efficient parameter training, with hyperparameters set to r = 4, a = 4, epoch = 2 (as shown in Table 12), and the rest are set according the default settings of llamarfactory². Weight merge is also involved. In general, the time complexity of this part is OnnLdr + Lddr), where n the number of instructions, L is the number of layers applying Lora, and is the dimension of the LLM hidden layer. < d, so the time complexity is bound by ON + Ldd).
Table 13: The time costs CORA and OGRN- ARXIV of annotations generation by Llama-3.1-88 Instruct.
Dataset
CORA
OGBN-ARXIV
Time GPU-hours
0.11
1.68
The process of annotations generation includes sorting the selected nodes and the reasoning process of LLM, and its time complexity (W log W) and O(W L' L'(('d - IdL)), where is the number of selected nodes, L' is the number of transformer layers in LLM, and is the input sequence length. Generally, W < 1, L' < I, then the time complexity is O + IdP).
Table 14: The time costs of retraining teacher GNNs CORA and OGBN-ARXIV.
Datasets
Teacher GNNs (Ii, T2, Ts, Th)
Total running time (seconds)
CORA
GCN, GAT, APPNP, H2GCN GCN, GAT, APPNP, H2GCN
2.4716 18.2017
OBGN-ARXIV
The time complexity of teacher GNN (2-layers) re-training is bound by O((NF + M)D), where N the number of nodes, F is the node feature dimension, M is the number of edges, and D is the GNN hidden layer dimension.
The time complexity of Node-prefeerence-rrives GNN Selector is O(W ldf + dd' + d'a)), where d' is the dimension the MLP hidden layer, a is the number of action categories, W & < d, the time complexity is bound by (122 + ld? + dd'). The training time complexity of student GNN is O(NF + D). Therefore, the overall time complexity is bound by O(LL 2l)df + (d' nL)d + 2/2 + 2D(NF + M)).
20

---

Table 15: Peak memories and running times Node-prs Node-preferrnce- serferrence-riveen GNN Selector on CORA and OBGN-ARIXV. "m" and "s" denote minute and second.
Dataset
Peak memory
Running time epoch
CORA
454.62 MB
7.9s
OBGN-ARIXV
1655.18 MB
44m 8.3s
The inference time complexity of PKD is determined by the testing process the student GNN. So its time complexity is bound by O(NF + M)
Specifically, We implement our proposed PKD with PyTorch (2.5.1) [59]. PyTorch Geometric (2.6.1) [60], Python 3.100.6), Transformers (4.50.3), and vllm (0.7.0). We conduct all experiments on the NVIDIA
GPU and Intel(R) Xeon(R) CPU Max 9468.
The time costs for GTA fine-tuning, the Distance-based Neighbor Selector, LLM annotation, teacher re-training, and the PPO loop Cora and Ogbn-Arixv are presented in Tables 12, 13, 14 and 15, respectively. The peak memories of performing PKD on the Cora and Ogbn-Arxiv are also listed in Table 15.
D Proofs for Propositions 3.1
The uncertainty usually refers to a measure of the confidence of a model in predicting a certain sample. From the perspective of collective consensus [40], we define the K- K-uncertainty of one node as the deviation of each teacher GNN's prediction probability distribution from the overall prediction probability distribution. From the Proposition 3.1, we can get that, the larger ôg of one node, the stronger the uncertainty of this node, which is more beneficial to teacher GNNs training.
Proof. For each node v, the prediction probability distributions of B teacher GNNs can be denoted by P2.... PB. .p. The K K-uncertainty of node v is defined as:
6x(v)= A
(Dkr(P(()))P)(o)) DRu((3(()))R(O))) ISKJSB
(10)
Here, we define the average prediction probability distribution as following:
Definition D.1. The average prediction probability distribution M the benchmark for the overall prediction probability distribution to measure both the models confidence and the consistency of each GNN with the overall probability distribution.
1 M(r)
Σ P(()) 1=
(11)
B
Then, the uncertainty of node e is,
1
8, B 1=1
(P2(P(()))M())))
(12)
According to Jenson's 's inequality, we have
(k( No.
(13)
For any probability distribution P. have
= H(P,M) - H(P)
(14)
Then,
5k = N ()
(15)
21

---

As the K -u-ueertaintty increases, the entropy of the GNN's prediction probability distribution P, in- creases, and the cross-entropy (P,,, grows significantly due the larger probability distribution differences. So there is,
B
6x(v) - 6n
(16)
That is,
5k (v) x 8,
(17)
From the Proposition 3.1, we also get that, selecting high-uncertainty nodes expand the training set benefits GNNs training.
For GNN with prediction probabilities P(y clu), the entropy of an unlabeled node is
H(v) = Σ Ply = c/u) log Ply c/u)
(18)
To maximize information gain, we select the node v* with the highest uncertainty (entropy):
arg max (v)
(19)
After expanding to the training dataset, the loss function becomes:
Lnew L0ta(0) + [(fe(v").
(20)
Here, y" is considered the true label based the fine-tuned LLM. The GNN parameters updated as:
nnew fold VoL(fo(v"),'))
(21)
Since the prediction probability distribution of u" is close to uniform (due to high entropy) [61], the gradient effectively corrects the GNN parameters [62], reducing the error. According the preference rank: VpR = Sort((u),.... uNn. 6x (v1), 8x (U2)....... 8x (UN)), we can get the follows:
1 W
9 (DL)
arg min
Σ C(ftre)) 24)
(22)
v.Efup Po W. 16 (u (18x))
where Dí is the expanded training dataset. 1 is the optimal parameter of teacher GNN. UPR represents the w-th nodes in the preference rank.
22

---

E
Other Experimental Results
E.1
Visualization
Figure presents the outstanding node classification performance we mentioned in Section 4.2, which is illustrated by the t-SNE [63] visualization of the embedding spaces for CORA. Notably, Figure 6(a) illustrates the results of the student GNN (GCN) under the # LN 5 condition.
From the Figure 6, we can see that, some KD methods fail to enable the student GNN to learn discriminative node representations, evidenced by the absence of clustered structures in the embedding space, exemplified by MSKD, BGNN, and MTAAM. GCNII and KDGA struggle to form well-separated clusters, whereas methods like LLMGNN. GAugLLM, and FairGKD yield clusters with limited purity. Compared to these baselines, our method generates embeddings with significantly enhanced inter-class separability and high cluster purity, resulting improved few-shot node classification performance.
20
20
-20
-00
49
-00
20
29
60
-00
28
40
-20
20
40
20
20
40
(a) GCN (student GNN)
(b) GCN
EGNN
(d) LLMGNN
60
40
60
0-
40
-00
20
(c) GAugL.LM
(f) AGST
(g) KDGA
(h) MSKD
40
-20
30
-01
20
-20
(i BGNN
(i) MTAAM
(k)) FairGKD
(1) PKD
(#LN
Figure 6: T-SNE [63] visualizations on CORA.
E.2
Ablation Study
The results on Cora and Amazon Ratings using our PKDjJama (RL-based method) and the other three teacher selection methods (Entropy-based ranking, i.e., selecting the teacher GNN with the highest prediction confidence, Random selection, and End-to-end learning) are shown in Table 16. It is obvious that the RL-based method significantly outperforms the other three methods.
23

---

Table 16: Performance comparison of Entropy-based ranking, Random selection, End-to-end learning and RL-based approach on CORA and AMAZON RATINGS.
Methods
CORA
AMAZON RATINGS
Entropy-based ranking
75.70
55.74
Random selection
62.80
63.05
End-to-end learning
60.29
60.39
Limma (RL-based)
90.27
65.93
E.3 Hyperparameters Sensitivity Analysis
As mentioned in Sec. 4.4, we perform the hyperparameter sensitivity analysis over the loss-weight coefficients a, B. y, 77 on two datasets, and report the results in Tables 17, 18, 19 and 20. As result, the proposed PKD can achieve much better performance when = 0.5, 8 = 1, = 0.1, 0.3.
Table 17: The influence of a.
0.3
0.4
0.5
0.6
0.7
0.8
0.9
2
CORA
73.78
74.70
75.66
73.15|
72.41
69.05
72.71
69.49
70.12
AMAZON RATINGS 661.8
62.86
63.83|
62.76
61.30
60.28
60.47
61.66
61.74 61.66
Table 18: The influence of 3.
3
0.25
0.5
CORA
64.10
67.17
75.36
74.15
AMAZON RATINGS
62.94
63.23
63.91
63.93
Table 19: The influence of y-
7
0.05
0.1
0.2
0.5
2
CORA
68.90
70.12
63.77
64.40
65.06
62.66
AMAZON RATINGS
62.58
64.01
63.71
63.63
63.93
63.93
Table 20: The influence of r).
1)
0.7
0.9
CORA
89.96
90.27
88.43
86.67
89.42
AMAZON RATINGS 64.87
65.93
64.79
63.75
63.81
F
Broader Impact
The proposed PKD offers significant broader impacts by enhancing few-shot node classification on TAGs. By combining the strengths of LLM and GNN, it improves learning efficiency, reducing the need for expensive and consuiong manual annotation. This can benefit industrics like social media, recommendation systems, and network analysis, enabling more accurate and scalable models for personalized services, fraud detection, and dynamic optimization.
Additionally, PKD can tailor message-pasee passing mechanisms to specifpecific attributes can lead to more adaptive and efficient machine learning models. It also democratizes access advanced machine learning, allowing smaller organizations and researchers with limited resources to develop effective models. However, ethical considerations, such as privacy and fairness, must be prioritized to ensure responsible deployment.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, we propose a preference-driven knowledge distillation (PKD) framework that unites LLMs and various- aachitttrrch GNNs for few-shot node classification on TAGs. We claim the contributions and scope in the abstract and introduction sections (See Abstract and Introduction Section).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In this work, we discuss the limitations of our research and outline directions for future work (See Conclusion).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
25

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In this work, we provide the Proposition 3.1 and its complete proof(See Method and Appendix. D).
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the code necessary for replicating the studies described in this paper via an anonymous link, and we detail the experimental setup for the replication in the article itself (See Experiments and Appendix. C).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
26

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: For the datasets disclosed in the article, we have provided information regarding their sources and origins (See Appendix. A).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.co
public/guides, /tt.deee/ooeSuuumiiissssisisssis ottpesummsssiiooooooooooooooooee
onPolicy icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wwtppbccc/gguuiioocccccoccccccpe wwwcc//gggse//eeeeee httpc//ggiidee//eduubiiiiiiee cateesubmissooPool...
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
we have specified all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary understand the results (See Experiments and Appendix. C).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In this paper, we have reported the standard deviation of the experiments (See Experiments and Appendix. E).
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification; In this paper, we provide detailed information about the experimental resources, including GPU configurations used in our studies and running time costs about all methods (See Experiments).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htt////eeerpppeeeeee http:///cccc/cccccccccieeeeeee
Answer: [Yes]
Justification:
The study presented in this paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have provided the societal impacts of the work (See Appendix F).
28

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not address issues related to this aspect.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators and original owners of the assets used in our paper, such as code, data, and models, have been properly credited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
29

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The research presented in this paper is not concerned with new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve experiments or research related to human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not address potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: LLMs is an important component of the core methods in this research and we has describe the usage in detail (See Method, Experiments and Appendix C).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //nerrurps.... wwt/cc////ooonecceeeeeee Cec.oooeeenneee222/////111 for what should or should not be described.
31


Reviews:
[{"review_id": "8bCXdVsHVN", "reviewer": "Reviewer_o4tP", "summary": "The paper presents a preference-driven knowledge distillation (PKD) framework for few-shot node classification on text-attributed graphs. PKD combines an LLM, fine-tuned with graph-topology-aware (GTA) prompts, and multiple graph neural networks. First, a GNN-preference-driven Node Selector (GNS) fully exploits nodes’ prediction discrepancies among various GNNs, ranks nodes by K-uncertainty, and asks the LLM to label a subset of them; those labels are then distilled from the LLM to teacher GNNs. Second, a Node-preference-driven GNN Selector (NGS) casts teacher choice as a reinforcement-learning task in which the fine-tuned LLM, acting as agent, selects for each node the most appropriate message-passing mechanism. Experiments on a number of datasets show that, with a small number of labelled nodes per class, PKD outperforms strong GNN baselines and distillation methods.", "questions": "Please quantify the LLM budget: report how many calls and tokens were needed to reach the 48 % training split on each dataset, give a rough GPU-hour cost, and show how a strong baseline performs if it receives the same number of LLM-generated labels.\n\nShow that the PPO teacher selector is really needed by providing learning curves and variance across seeds and by comparing its accuracy and runtime with two simpler choices. \n\nComplete the ablation study with a variant that omits the RL selector, one that keeps the LLM frozen, and a short sweep over the loss-weight coefficients η, α, β and γ to demonstrate robustness.\n\nAdd practical resource figures: GPU type, peak memory and total wall-clock time for GTA fine-tuning, node annotation, teacher re-training and the PPO loop, at least on CORA and OGBN-ARXIV, so the scalability of PKD is clear.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "A concern regarding evaluation is that, while all methods ultimately train on a split that contains about 48 % of the nodes, the baselines receive ground-truth labels for those nodes, whereas PKD must keep querying the language model to bring its own training set up to the same size. The paper does not say how many LLM calls this requires, what that annotation budget costs, or whether baselines would improve if they were allowed an equal amount of LLM-generated pseudo-labels, leaving the cost-effectiveness comparison open to question.\n\nThe practical value of the RL component is unclear. The paper offers no learning curves, no variance across random seeds, and no comparison to simpler heuristics such as entropy ranking or majority vote, so it is difficult to verify that the PPO step is either necessary or stable.\n\nThe paper likewise omits results for a variant without the RL selector, a version that keeps the LLM frozen, or any sensitivity analysis over the loss-weight hyper-parameters η, α, β and γ. Consequently the individual contribution of each costly module is uncertain.\n\nFinally, although the complexity discussion includes big-O expressions, concrete resource metrics, such as GPU type, wall-clock hours and memory footprints for GTA fine-tuning, GNS annotation and the PPO loop are omitted, making it difficult to judge whether the framework is feasible on larger graphs.", "was_revised": true, "final_justification": "I believe the authors have made a technically solid contribution, and my final score reflects the significance and originality of the work.", "rebuttal": "Thanks for reviewing this manuscript and providing valuable comments. We appreciate the positive feedback from all reviewers and your constructive suggestions for further improvement. Considering the correlation between **W**eaknesses and **Q**uestions, we reply after integrating them as below.\n\n**[W1, Q1]: Please quantify the LLM budget: report how many calls and tokens were needed to reach the 48\\% training split on each dataset, give a rough GPU-hour cost, and show how a strong baseline performs if it receives the same number of LLM-generated labels.**\n\n**Reply**: As shown in Figure 2 in the manuscript, the fine-tuned LLM already has significant performance for zero-shot node classification. When conducting the LLM annotation, we only call LLM once for each node. That is, the number of LLM calls ($t$) is equal to the number of nodes selected by the GNS (GNN-preference-driven Node Selector), i.e., $t = 0.48 \\cdot N - Q$, where $N, Q$ are the numbers of all nodes and the initial labeled nodes, respectively. The needed number of tokens ($T$) can be calculated by $T = m \\cdot (0.48 \\cdot N - Q)$, where $m$ is the LLM's max\\_tokens value, which is set to 2048 in this paper. For instance, the values of $t$ and $T$ for Cora are 1265 and 2590720 under the \\# LN 5 setting.\n\nTo reach the 48\\% training split on each dataset, we first select the nodes preferred by GNN based on the preference ranking and then conduct the node-wise category-induction prompts to query LLM for their pseudo labels. In this process, we only utilize one GPU card (NVIDIA A800-SXM4-80GB), and the time costs using different LLMs are listed in below table. The results show that that inference time varies across different LLMs, even when evaluated under the same environmental settings, with larger models exhibiting slower inference time.\n\n**The time costs of different LLMs.**\n| **Dataset** |  **Llama-3.1-8B** | **Llama-3.1-13B**  | **Qwen-2.5-7B** |  **Qwen-2.5-14B**|\n| --- | --- | --- | --- | --- |\n|Cora | 0.11 GPU-hours|0.36 GPU-hours| 0.05 GPU-hours | 0.29 GPU-hours |\n|Ogbn-Arxiv | 1.68 GPU-hours|5.01 GPU-hours | 0.94 GPU-hours | 2.95 GPU-hours |\n||||||\n\n**The Influence of LLM-Generated Pseudo-Labels.** To demonstrate the effect of LLM-generated pseudo-labels, we compare the node classification performance of PKD and three baselines under different label settings (\\# LN 5, 48\\% training ratio expanded by the annotated labels and real labels, respectively). The experiments are conducted on four datasets (Cora, Wiki CS, Washington, and Wisconsin). The results are presented in below table. For GCNII and IceBerg, which are proposed to tackle the challenge of sparse labels, using the LLM-anotated node labels can improve their performance on all datasets. However, using the same number of real labels achieves better performance. \n\n**Classification accuracy comparisons.**\n| |**Label settings**| **Cora** | **Wiki CS** | **Washington**| **Wisconsin** |\n| --- | --- | --- | --- | --- | --- |\n| |\\# LN 5 |77.74 | 56.29| 64.17 | 60.94 |\n|**GCNII** | 48\\% LLM-generated labels | 76.69  |51.18 | 70.83  | 62.50 |\n| |48\\% real labels|81.54 | 59.17|71.79| 65.98 |\n| |\\# LN 5|76.23 |84.88 |67.76|41.53|\n|**IceBerg** |48\\% LLM-generated labels|78.66|71.23 |70.12 |42.22|\n| | 48\\% real labels |81.94|86.49 |72.04|45.43 |\n| |\\# LN 5|43.91|46.81 |45.29|33.33 |\n|**MSKD**| 48\\% LLM-generated labels|45.89|54.06|48.17|39.50 |\n| |48\\% real labels|51.61|62.73|50.39|41.51|\n|**PKD**|48\\% LLM-generated labels| **90.27**|**81.39**|**83.74**|**76.89**|\n||||||\n\n**[W2, Q2]: Show that the PPO teacher selector is really needed by providing learning curves and variance across seeds and by comparing its accuracy and runtime with two simpler choices.**\n\n**Reply**: Figure 4 in the manuscript shows the learning curves (accuracy vs. epoch) of the NGS (i.e., the RL component). It is evident that the NGS with the $R_3$ reward function converges to a stable state. We also plan to provide the learning curves (loss vs. epoch) of the NGS in the final version of this paper. To demonstrate that the PPO teacher selector is really needed, we replace it with two simple teacher selectors and repeat the knowledge distillation five times using distinct random seeds. The first method selects a teacher GNN for each node randomly, while the second method chooses the teacher GNN for each node based on the entropy value of its logit outputs. The lower the entropy value, the higher the confidence of the teacher. We select the most confident GNN as the teacher. We record the average test classification accuracy and standard deviation of each method with parameters that lead to the peak validation accuracy. The below table presents the results on Cora which demonstrate that our method is both more stable and more effective than the other two methods. \n\n**The comparison between PPO and other methods on Cora.**\n|**Methods /\\# LN 5**| **Accuracy**|**Running time (seconds)**|\n| --- | --- | --- |\n|Random teacher selector|$81.49_{\\pm 6.4}$|0.017|\n|Entropy-based teacher selector|$75.70_{\\pm 4.6}$|0.023|\n|**PKD**|$91.14_{\\pm 0.3}$|2481.858|\n|||\n\n**[W3, Q3]: Complete the ablation study with a variant that omits the RL selector, one that keeps the LLM frozen, and a short sweep over the loss-weight coefficients $\\alpha,\\beta,\\gamma,\\eta$ to demonstrate robustness.**\n\n**Reply**: We compare our method PKD with two other variants on Cora and Amazon Ratings. The first variant replaces the RL selector with a random selector, and the second variant omits the fine-tuning of GTA prompts and directly utilizes the annotated node labels generated by the original LLM to expand the training data. The results are presented in the below table and will be added to the final version of this paper. It is evident that PKD outperforms the other two variants by a large margin. \n\n**The comparison between PKD and the other two variants.**\n|**Methods**|**Cora**|**Amazon Ratings**|\n| --- | --- | --- |\n|Entropy-based ranking|75.70|55.74|\n|LLM-frozen|48.84|45.26|\n|**PKD**|**90.27**|**65.93**|\n||||\n\n**Hyperparameter Sensitivity Analysis.** Additionally, we perform the hyperparameter sensitivity analysis over the loss-weight coefficients $\\alpha,\\beta,\\gamma,\\eta$ on two datasets, and report the results in the below tables. For the sensitivity analysis of $\\alpha$, we set $\\beta=1$, $\\gamma=1$, $\\eta=0.5$. This strategy also applies to the sensitivity analysis of $\\beta$ and $\\gamma$. As for $\\eta$, we fix $\\alpha$, $\\beta$, and $\\gamma$ to their respective best values. We can see that $\\beta$ and $\\eta$ are more sensitive than the other two hyperparameters, because $\\beta$ is related to the supervision signal provided by the LLM-generated annotation labels and $\\eta$ is related to the student GNN's performance.\n\n**The influence of $\\alpha$.**\n|$\\alpha$|**2**|**1**|**0.9**|**0.8**|**0.7**|**0.6**|**0.5**|**0.4**|**0.3**|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|Cora|70.12|69.49|72.71|69.05|72.41|73.15|**75.66**|74.70|73.78|\n|Amazon Ratings|61.74|61.66|60.47|60.28|61.30|62.76|**63.83**|62.86|61.81|\n|||||||||||\n\n**The influence of $\\beta$.**\n|$\\beta$|**2**|**1**|**0.5**|**0.25**|\n| --- | --- | --- | --- | --- |\n|Cora|74.15|**75.36**|67.17|64.10|\n|Amazon Ratings|**63.93**|63.91|63.23|62.94|\n|||||\n\n**The influence of $\\gamma$.**\n|$\\gamma$|**2**|**1**|**0.5**|**0.2**|**0.1**|**0.05**|\n| --- | --- | --- | --- | --- | --- | --- |\n|Cora|62.66|65.06|64.40|63.77|**70.12**|68.90|\n|Amazon Ratings|63.93|63.93|63.63|63.71|**64.01**|62.58|\n|||||||\n\n**The influence of $\\eta$.**\n|$\\eta$|**0.9**|**0.7**|**0.5**|**0.3**|**0.1**|\n| --- | --- | --- | --- | --- | --- |\n|Cora|89.42|86.67|88.43|**90.27**|89.96|\n|Amazon Ratings|63.81|63.75|64.79|**65.93**|64.87|\n||||||\n\n**[W4, Q4]: Add practical resource figures: GPU type, peak memory and total wall-clock time for GTA fine-tuning, node annotation, teacher re-training and the PPO loop, at least on CORA and OGBN-ARXIV, so the scalability of PKD is clear.**\n\n**Reply**: We implement our proposed PKD with PyTorch (2.5.1), PyTorch Geometric (2.6.1), Python (3.10.16), Transformers (4.50.3), and vllm (0.7.0). We conduct all experiments on the NVIDIA A800-SXM4-80GB GPU and Intel(R) Xeon(R) CPU Max 9468. The peak memories of performing PKD on the Cora and Ogbn-Arxiv are listed in the below label. \n\n**The peak memory of performing PKD on the Cora and Ogbn-Arxiv.**\n|**Dataset**|**Peak memory**|\n| --- | --- |\n|Cora|454.62 MB|\n|Ogbn-Arixv|1655.18 MB|\n|||\n\nThe time costs for LLM annotation on Cora and Ogbn-Arixv can refer to the **Reply to the [W1, Q1]**. The time costs for GTA fine-tuning, the Distance-based Neighbor Selector (DNS), teacher re-training and PPO loop on Cora and Ogbn-Arixv are presented in the following tables, respectively. Notably, for the teacher re-training stage, we show all the teacher GNNs that are trained togethe, and their total training times and epochs.\n\n**The GTA fine-tuning configurations on Llama-3.1-8B-Instruct.**\n|Dataset size|Epoch|lora_$r$|lora_$\\alpha$|optimizer|learning rate|GTA fine-tuning time|LLM|\n| --- | --- | --- | --- | --- | --- | --- | --- |\n|53,617|2|4|4|AdamW|$1e^{-4}$|9h 41m 48s|Llama-3.1-8B-Instruct|\n||||||||\n\n**The times of DNS (seconds) on the Cora and Ogbn-Arxiv.**\n|Dataset|Time of the DNS (seconds)|\n| --- | --- |\n|Cora| 0.7130|\n|Ogbn-Arxiv| 6.5513|\n||\n\n**The time costs of retraining teacher GNNs on Cora, Amazon Ratings, and Ogbn-Arxiv.**\n|Datasets|Teacher GNNs|Epochs|Time of teacher re-training (seconds)|\n| --- | --- | --- | --- |\n|Cora|GCN, GAT, APPNP, H$_2$GCN|300|2.4716|\n|Ogbn-Arxiv|GCN, GAT, APPNP, H$_2$GCN|300|18.2017|\n|Amazon Ratings|DirGNN, GPRGNN, HoloNets, H$_2$GCN|300|3.1452|\n||||\n\n**Running times of PPO on Cora and Obgn-Arixv. \"m\" and \"s\" denote minute and second.**\n|Datasets|Time of PPO loop|\n| --- | --- |\n|Cora|7.9 s|\n|Ogbn-Arxiv|44m 8.3s|\n||"}, {"review_id": "ZpU9SCiIe0", "reviewer": "Reviewer_B7XE", "summary": "This paper proposes a new framework, PKD, that combines the complementary strengths of large language models (LLMs) and multiple graph-neural-network (GNN) teachers for few-shot node classification on text-attributed graphs (TAGs). It mainly includes two modules: GNN-preference-driven Node Selector (GNS) and Node-preference-driven GNN Selector (NGS). To make the LLM topology-aware, the authors design Graph-Topology-Aware (GTA) prompts for a lightweight LoRA fine-tuning step. Experiments on various datasets show that PKD consistently outperforms a broad set of strong baselines, and remains competitive when the label budget rises.", "questions": "1.The current reward function combines two loss functions and accuracy, which raises the complexity, has a simpler expression of the reward function been explored? \n\n2.Is there any way to further minimize large language model calls? \n\n3.What is the need to use reinforcement learning? \n\n4.How does the method proposed in this paper compare to the results without using reinforcement learning?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.The paper is clear writing and easy to follow. \n\n2.The method is novel, which designs a two-level preference mechanism to address the challenges in knowledge distillation under label scarcity. The Methodology is supported by solid theory. Casting teacher selection as an RL problem with the LLM as the policy is original and broadly applicable to other graph tasks. \n\n3.This paper performs a comprehensive evaluation in experiments, which contains 9 real-world datasets and various baselines. \n\n\nWeaknesses:\n\n1.Although the paper provides a time-complexity analysis and a runtime table, PKD’s iterative LLM queries and RL loops make each training epoch slower than most baselines, thereby limiting its scalability to million-node graphs. \n\n2.Due to its iterative LLM queries and RL loops, PKD’s per-epoch time is higher than that of most baselines. \n\n3.Some hyperparameters are missing from the analysis, e.g. α, β, γ, η.", "was_revised": true, "final_justification": "I’ve gone through the rebuttal. The added sensitivity checks and the authors’ take on why the coefficients behave differently on Cora and Amazon have eased some of my concerns. The ablation for Q1 and the RL justification for Q3 and Q4 now feel solid. Still, the paper only sketches future tricks like parallel queries, model distillation and speculative decoding for W1 and W2, without showing they work on a million node graph. The idea is neat and the paper is stronger, yet without that large scale proof I’m staying with a borderline accept.", "rebuttal": "Thanks for reviewing this manuscript and providing valuable comments. We appreciate the positive feedback from all reviewers and your constructive suggestions for further improvement. **W** and **Q** represent weakness and question.\n\n**W1: Although the paper provides a time-complexity analysis and a runtime table, PKD’s iterative LLM queries and RL loops make each training epoch slower than most baselines, thereby limiting its scalability to million-node graphs.**\n\n**W2: Due to its iterative LLM queries and RL loops, PKD’s per-epoch time is higher than that of most baselines.**\n\n**Reply**: Thanks for your concerns. We agree with you. In the future, we can first distill knowledge from LLM with 72B parameters to LLM with 1.5B parameters without much performance degradation. Then, querying LLM will cost less time compared with that when querying an LLM with 8B parameters as used in the paper. The GNS iteratively queries LLM for node label annotations to expand the training data to 48\\% of the total nodes. This step can be parallelized, i.e., we can use multiple threads to query LLM for multiple nodes each time. The fine-tuned LLM is also used as the RL agent to select for each node the most appropriate teacher GNN. Since the parameters of the fine-tuned LLM are frozen, the time bottleneck of LLM query is the inference time. For this, we can reduce the inference time by designing efficient attention mechanisms, using speculative decoding, etc. \nAdditionally, the size of the expanded training data also increases the PKD's per-epoch time. Considering the fact that PKD outperforms the comparison methods by a large margin, we can reduce the label ratio, e.g., from 48\\% to 10\\%, which depends on the node category induction capability of LLM, thereby decreasing the time cost caused by the PPO loop.\nThrough the above modifications and innovations, PKD can be scalable to large graphs.\n\n**W3: Some hyperparameters are missing from the analysis, e.g. $\\alpha,\\beta,\\gamma,\\eta$.**\n\n**Reply**: For this, we supplement the hyperparameters sensitivity analysis for $\\alpha,\\beta,\\gamma,\\eta$ on Cora and Amazon Ratings. Specifically, when analyzing $\\alpha$, we set $\\beta=1$, $\\gamma=1$, $\\eta=0.5$. This strategy also applies to the sensitivity analysis of $\\beta$ and $\\gamma$. As for $\\eta$, we fix $\\alpha$, $\\beta$, and $\\gamma$ to their respective best values. The results of various values for them are presented in the following four tables. Intuitively, $\\beta$ and $\\eta$ are more sensitive than the other two hyperparameters, because $\\beta$ is related to the supervision signal provided by the LLM-generated annotation labels and $\\eta$ is related to the student GNN's performance.\n\n**The influence of $\\alpha$.**\n|$\\alpha$|**2**|**1**|**0.9**|**0.8**|**0.7**|**0.6**|**0.5**|**0.4**|**0.3**|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|Cora|70.12|69.49|72.71|69.05|72.41|73.15|**75.66**|74.70|73.78|\n|Amazon Ratings|61.74|61.66|60.47|60.28|61.30|62.76|**63.83**|62.86|61.81|\n|||||||||||\n\n**The influence of $\\beta$.**\n|$\\beta$|**2**|**1**|**0.5**|**0.25**|\n| --- | --- | --- | --- | --- |\n|Cora|74.15|**75.36**|67.17|64.10|\n|Amazon Ratings|**63.93**|63.91|63.23|62.94|\n|||||\n\n**The influence of $\\gamma$.**\n|$\\gamma$|**2**|**1**|**0.5**|**0.2**|**0.1**|**0.05**|\n| --- | --- | --- | --- | --- | --- | --- |\n|Cora|62.66|65.06|64.40|63.77|**70.12**|68.90|\n|Amazon Ratings|63.93|63.93|63.63|63.71|**64.01**|62.58|\n|||||||\n\n**The influence of $\\eta$.**\n|$\\eta$|**0.9**|**0.7**|**0.5**|**0.3**|**0.1**|\n| --- | --- | --- | --- | --- | --- |\n|Cora|89.42|86.67|88.43|**90.27**|89.96|\n|Amazon Ratings|63.81|63.75|64.79|**65.93**|64.87|\n||||||\n\n**Q1: The current reward function combines two loss functions and accuracy, which raises the complexity, has a simpler expression of the reward function been explored?**\n\n**Reply**: To address this, we conduct an ablation study on each part of the reward function. The visualization results are shown in Figure 4 in the manuscript. The single usage of accuracy ($R_1$) does not perform as well as using more parts (combination with $L_{CE}$). Figure 4 also shows that the $L_{DL}$ part is beneficial to quickly improve model performance and maintain stability. \n\nTo further verify the necessity of using all three parts, we train the Node-preference-driven GNN Selector on Cora and Amazon Ratings using four kinds of combinations of the three parts. The results are presented in the below table. Therefore, the best reward function contains all three parts.\n\n**Results of the combination of various loss functions.**\n|Combination|Cora|Amazon Ratings|\n| --- | --- | --- |\n|$A_{cc}$|85.88|57.83|\n|$A_{cc}$, $-L_{CE}$|87.49|63.65|\n|$A_{cc}$, $-L_{DL}$|83.73|55.14|\n|$A_{cc}$, $-L_{CE}$, $-L_{DL}$ (**ours**)|**90.27**|**65.93**|\n|||\n\n**Q2: Is there any way to further minimize large language model calls?**\n\n**Reply**: Specifically, the LLM is mainly used for node category induction and teacher GNNs selection. Considering that PKD performs much better than the comparison methods, we can probably reduce the large language model calls by reducing the number of annotated nodes from 48\\% to, say, 10\\%, which depends on the node category induction capability of LLM. Otherwise, more human annotations for node labels are needed in order to further minimize large language model calls.\n\n**Q3: What is the need to use reinforcement learning?**\n\n**Reply**: Our final goal is to select the only one best teacher GNN for each node. That is, we need to explore the discrete action space and find a series of assignment actions to get the highest global reward across the expanded training data. Reinforcement learning (RL) allows us to navigate through the large action space by learning an optimal policy that maximizes the long-term reward. Through interaction with the environment, the selector progressively refines its decisions on node-to-teacher assignments, leading to a more efficient and effective assignment strategy. Besides, we can handle the challenges of discrete decision-making and achieve better performance on the task by leveraging RL. It can also incorporate heterogeneous metrics, such as distillation loss, cross entropy loss, and classification accuracy, into the reward function, and has a more flexible training objective. \n\n**Q4: How does the method proposed in this paper compare to the results without using reinforcement learning?**\n\n**Reply**: The results on Cora and Amazon Ratings of our RL-based method and the other three teacher selection methods (entropy-based ranking, i.e., selecting the teacher GNN with the highest prediction confidence, random selection, and end-to-end learning) are shown in the below table. It is obvious that the RL-based method significantly outperforms the other three methods.\n\n**The compared results on Cora and Amazon Ratings of our RL-based method and the other three methods .**\n|Methods|Cora|Amazon Ratings|\n| --- | --- | --- |\n|Entropy-based ranking|75.70|55.74|\n|Random selection|62.80|63.05|\n|End-to-end learning|60.29|60.39|\n|RL-based (**ours**)|**90.27**|**65.93**|\n|||"}, {"review_id": "GJklznOEMr", "reviewer": "Reviewer_iBQW", "summary": "This paper proposes PKD, a preference-driven knowledge distillation framework that combines LLMs and GNNs for few-shot node classification on TAGs. The framework consists of two main components: (1) GNS, which identifies uncertain nodes using K-uncertainty metrics and leverages LLMs to annotate these nodes for expanding the training set, and (2) NGS, which uses reinforcement learning to select the most appropriate teacher GNN for each node during knowledge distillation. The authors fine-tune LLMs with GTA prompts to enhance their understanding of graph structures.", "questions": "1. In Figure 2, what exactly does \"LLM with structural attributes\" refer to? Additionally, why are results for OGBN-ARXIV missing from this analysis?\n2. Given that different nodes require different message-passing mechanisms (the core motivation for NGS), why are fixed architectures (GCN for homophily, H2GCN for heterophily) chosen as student models? Wouldn't the student's message-passing mechanism also impact the effectiveness of knowledge distillation?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper presents a well-motivated approach that addresses a practical problem, labeled data is scarce, in graph learning. The experimental evaluation is comprehensive, covering 9 datasets with different homophily ratios and comparing against multiple baseline categories.\n2. The node-specific teacher selection mechanism via reinforcement learning is novel and well-motivated. \n3. The work addresses an important problem in graph learning where labeled data is scarce. The performance improvements are substantial, often outperforming methods that use more labeled data.\n4. The paper is generally well-written with clear motivation and methodology. The framework overview in Figure 1 effectively illustrates the approach.\n\nWeaknesses:\n1. The computational complexity analysis, while provided in the appendix, reveals significant overhead that may limit practical applicability. 2. There is a lack of discussion on the selection of student models. Can the ordinary convolution mechanism acquire the knowledge of multiple teacher models with multiple architectures?\n3. The paper lacks discussion of teacher GNN pretraining procedures. \n4. The approach requires multiple training phases (LLM fine-tuning, teacher pretraining, RL training, student training), making it computationally expensive and potentially impractical for large-scale applications.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. I think my current score has fairly reflected the quality of this paper", "rebuttal": "Thanks for reviewing this manuscript and providing valuable comments. We appreciate the positive feedback from all reviewers and your constructive suggestions for further improvement. **W** and **Q** represent weakness and question.\n\n**W1: The computational complexity analysis, while provided in the appendix, reveals significant overhead that may limit practical applicability.**\n\n**W4: The approach requires multiple training phases (LLM fine-tuning, teacher pretraining, RL training, student training), making it computationally expensive and potentially impractical for large-scale applications.**\n\n**Reply**: Thanks for your concerns. We agree with you. In the future, we can first distill knowledge from LLM with 72B parameters to LLM with 1.5B parameters without much performance degradation. Then, fine-tuning the 1.5B LLM will cost less time compared with the 8B LLM used in the paper. The time cost for training the teacher GNNs (see the below table) and the student GNN is acceptable. In the paper, we use the fine-tuned LLM as node label annotators for up to 48\\% of the total nodes. This step can be parallelized in the future, i.e., we can use multiple threads to query LLM for multiple nodes each time. The fine-tuned LLM is also used as the RL agent to select for each node the most appropriate teacher GNN. Since the parameters of the fine-tuned LLM are frozen, the time bottleneck of LLM annotation is the inference time. For this, we can reduce the inference time by designing efficient attention mechanisms, using speculative decoding, etc. Additionally, the size of the expanded training data increases the PKD’s per-epoch time. Considering that PKD outperforms the comparison method by a large margin, we can reduce the label ratio from 48\\% to, say, 10\\%, which depends on the node category induction capability of LLM, thereby decreasing the time cost caused by the RL training. Through the above modifications and innovations, PKD can be scalable to large graphs. \n\n**The time cost for training the teacher GNNs on the Cora, Ogbn-Arxiv and Amazon Ratings.**\n|Datasets|Epochs|Total running times (seconds)|\n| --- | --- | --- |\n|Cora|300|2.4716|\n|Obgn-Arxiv|300|18.2017|\n|Amazon Ratings|300|3.1452|\n|||\n\n**W2: There is a lack of discussion on the selection of student models. Can the ordinary convolution mechanism acquire the knowledge of multiple teacher models with multiple architectures?**\n\n**Reply**: Thanks for your insightful considerations. For homophily and heterophily graphs, we just use the basic and simple GNNs, i.e., GCN and H$_2$GCN, for the student GNN model. Following the settings in the manuscript, we choose GAT, APPNP, H$_2$GCN as the teacher GNNs for homophily graphs and select DirGNN, GPRGNN, HoloNets as the teacher GNNs for heterphily graphs.\n\nTo verify whether the student GNN can acquire the knowledge of multiple teacher GNNs with multiple architectures, we separately distill the knowledge from different pretrained teacher GNNs to the student GNN only by the KD loss and compare their performance. The results are shown in the below table. It is evident that the teacher GNNs and the student GNN perform similarly on the same graph. Therefore, the ordinary convolution mechanism of the student GNN can acquire the knowledge of multiple teacher GNNs with multiple architectures. We will add the experimental results in the final version of this paper.\n\n**The comparison between multiple teacher GNNs and the student GNN on Cora and Amazon Ratings.**\n|Roles|Models|Cora|Models|Amazon Ratings|\n| --- | --- | --- | --- | --- |\n|Teacher GNN|GAT|89.13|DirGNN|63.35|\n|Student GNN|GCN|89.32|H$_2$GCN|64.03|\n|Teacher GNN|APPNP|80.85|GPRGNN|63.15|\n|Student GNN |GCN|81.03|H$_2$GCN| 62.74|\n|Teacher GNN |H$_2$GCN|49.91|HoloNets|59.65|\n|Student GNN|GCN|51.30|H$_2$GCN|59.24|\n|||||\n\n**W3: The paper lacks discussion of teacher GNN pretraining procedures.**\n\n**Reply**: For the teacher GNN pretraining procedures, we just utilize the few given node labels. After pretraining, we collect their logit outputs to conduct the GNN-preference-driven node selection based on the uncertainty. Thanks for your reminder. We will add this detailed description in the final version of this paper.\n\n**Q1: In Figure 2, what exactly does ''LLM with structural attributes'' refer to? Additionally, why are results for OGBN-ARXIV missing from this analysis?**\n\n**Reply**: We utilize the original LLM for node label annotation using the structural attribute, which refers to the prompt consisting of the node's neighbors chosen by the Distance-based Neighbor Selector and their semantic attributes. In Figure 2, the ''LLM with structural attributes'' refers to it.\n\nFor the results on Ogbn-Arixv, we present them in the table below and will add them in the final version of this paper. It is evident that our method is also effective on Ogbn-Arxiv.\n\n**The node classification accuracies of different methods on Ogbn-Arxiv.**\n|Methods|Ogbn-Arxiv|\n| --- | --- |\n|Original LLM|24.11|\n|LLM with structural attributes|44.17|\n|Fine-tuned LLM with structural attributes|71.62|\n||\n\n**Q2: Given that different nodes require different message-passing mechanisms (the core motivation for NGS), why are fixed architectures (GCN for homophily, H$_2$GCN for heterophily) chosen as student models? Wouldn't the student's message-passing mechanism also impact the effectiveness of knowledge distillation?**\n\n**Reply**: Thanks for your insight again. Since nodes have intricate local topologies, such as the various node degree, a single GNN cannot capture the essence of nodes completely. To this end, we use multiple teacher GNNs and select for each node the most appropriate teacher GNN. As our **response to W2**, the ordinary convolution mechanism of the student GNN can acquire the knowledge of multiple teacher GNNs with multiple architectures. Thus, using the fixed architectures for the student GNN is simple and effective.\n\n**The versatility of PKD.** Indeed, our method has the versatility across various student GNN architectures. To further verify this, we compare the performance of the teacher GNN with the student GNNs distilled from it on two datasets. Specifically, for homophily graphs, we select GCN, GAT, and APPNP as the student GNNs, with H$_2$GCN chosen as the teacher GNN. For heterophily graphs, we choose DirGNN, GPRGNN, and HoloNets as the student GNNs, with H$_2$GCN selected as the teacher GNN. The results are presented in the following table. Although the message-passing mechanisms of student GNNs are different, they can achieve similar performance on the same graph after distillation from the teacher. Notably, we do not provide any node labels for the student GNNs during knowledge distillation. Therefore, the student's message-passing mechanism does not impact the effectiveness of knowledge distillation. These experimental results will be added to the final version of this paper.\n\n**The comparison between the teacher GNN and multiple student GNNs on Cora and Amazon Ratings.**\n|Roles|Model|Cora|Model|Amazon Ratings|\n| --- | --- | --- | --- | --- |\n|Teacher GNN|H$_2$GCN|49.91|H$_2$GCN|63.55|\n|Student GNN|GCN|51.30|DirGNN|62.03|\n|Student GNN|APPNP|51.75|GPRGNN|64.29|\n|Student GNN|GAT |51.93|HoloNets|63.93|\n|||||"}, {"review_id": "a9cTGbLqj1", "reviewer": "Reviewer_uNcZ", "summary": "Proposed method leverages the complementary strengths of an LLM's few-shot/zero-shot capabilities and a GNN's ability to understand complex relationships. To mitigate the high costs of LLMs, it devises a GNN-preference-based method to select nodes for distillation from the LLM to teacher GNNs, as well as a method for each node to choose which teacher to receive knowledge distillation from.", "questions": "See Weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths.**\n\nS1. The methodology effectively maximizes the respective strengths and compensates for the weaknesses of GNNs and LLMs. It leverages the GNN's ability to understand complex structures while addressing its reliance on expensive labeling, and it utilizes the LLM's zero-shot/few-shot capabilities while mitigating issues from its large parameter scale.\n\nS2. The approach of utilizing a variety of GNNs rather than relying on a single one, and further selecting a specific GNN for distillation on a per-node basis, is a novel contribution.\n\nS3. The writing is clear, and all modules (GNS, NGS) are well-designed with sufficient justification.\n\nS4. The experimental setup is solid and effectively demonstrates the efficacy of the proposed methodology.\n\n**Weaknesses**  \n\nW1: While creating the GNS to avoid the high cost of labeling all nodes with an LLM is a reasonable approach, the subsequent need to use the LLM again to select a GNN for each node's prediction seems to reintroduce significant costs. This appears to offset the initial advantage.  \n\nW2: Mentioning the differences between this paper's task and traditional meta-learning-based graph few-shot learning tasks (e.g., N-way, K-shot settings) in the 'Related Work' section could enhance the clarity of the task definition.", "was_revised": true, "final_justification": "While the explanation is largely acceptable, a minor concern remains. I concede that reducing the number of nodes to be analyzed lowers the overall cost, but the overhead of performing this selection for each individual node still seems non-trivial. As I don't consider this a critical flaw, I will maintain my positive rating.", "rebuttal": "Thanks for reviewing this manuscript and providing valuable comments. We appreciate the positive feedback from all reviewers and your constructive suggestions for further improvement. **W** and **Q** represent weakness and question.\n\n**W1: While creating the GNS to avoid the high cost of labeling all nodes with an LLM is a reasonable approach, the subsequent need to use the LLM again to select a GNN for each node's prediction seems to reintroduce significant costs. This appears to offset the initial advantage.**\n\n**Reply**: Thanks for your comments. We use the fine-tuned LLM as the RL agent to select for each node the most appropriate teacher GNN. The prompt is constructed by combining the node's semantic, structural, and prediction attributes derived from the enhanced teacher GNNs. Note that the RL agent should be able to comprehend the node's structural attributes. If replacing the fine-tuned LLM with a small model such as BERT, we should fine-tune BERT again for comprehending the node's structural attributes. In addition, introducing an extra model that may hinder the synergy of the complementary strengths of LLMs and various GNNs is inelegant. Since the parameters of the fine-tuned LLM (the RL agent) are frozen, the time bottleneck is the inference time. Therefore, we can reduce the inference time by designing efficient attention mechanisms, using speculative decoding, etc. Additionally, considering that PKD outperforms the comparison method by a large margin, we can reduce the label ratio from 48\\% to, say, 10\\%, which depends on the node category induction capability of LLM, thereby decreasing the time cost caused by the teacher selection. \n\n**W2: Mentioning the differences between this paper's task and traditional meta-learning-based graph few-shot learning tasks (e.g., N-way, K-shot settings) in the 'Related Work' section could enhance the clarity of the task definition.**\n\n**Reply**: Thanks for your suggestions. As defined in [1], the nodes in the graph meta-learning task for node classification can be divided into the training set and the disjoint test set. Nodes in the test set are associated with totally different new classes. Unlike this, the labeled nodes in this paper's task already cover the entire range of categories, which is the most different setting. In the final version of this paper, we will add this point in the Related Work part. \n\n[1] Zhou, Fan, Chengtai Cao, Kunpeng Zhang, Goce Trajcevski, Ting Zhong, and Ji Geng. ``Meta-gnn: On few-shot node classification in graph meta-learning.'' In Proceedings of the 28th ACM international conference on information and knowledge management, pp. 2357-2360. 2019."}]

Author rebuttal:

Meta-review: This paper provides a theoretical framework for graph contrastive learning (GCL), aiming to establish provable guarantees on its performance. The authors formalize the conditions under which GCL can learn effective node and graph representations, bridging empirical success with theoretical justification. They analyze both augmentation strategies and objective functions, showing how these influence representation quality. Empirical experiments on standard benchmarks support the theory, demonstrating competitive or superior performance compared to existing GCL methods.

Strengths: The paper tackles an important gap by offering formal guarantees for GCL, a field that has been largely empirical so far. The analysis is mathematically rigorous, while experiments validate the claims. The work improves understanding of why GCL works and under what conditions. The paper is well-organized and clearly written.

Weaknesses: Some assumptions in the theory may be restrictive or unrealistic in practice (e.g., distributional conditions on graph augmentations). While experiments support the analysis, they remain relatively limited in scope compared to purely empirical GCL works. The practical guidance derived from the theory could be elaborated more clearly.

Rebuttal/Discussion: Reviewers questioned the practicality of assumptions, the completeness of experiments, and the clarity of certain proofs. The authors clarified assumptions, added discussion on their applicability, and expanded experiments to strengthen the empirical side. These responses were satisfactory to most reviewers, and concerns were largely resolved.

Decision: This paper provides meaningful theoretical contributions to understanding GCL, supported by experiments. While not broad or impactful enough for spotlight/oral due to the restrictive assumptions and modest empirical scope, it represents a solid and rigorous advance in the field.
Decision: Accept (poster)