InfGraND: An Influence-Guided GNN-to-MLP Knowledge Distillation
Authors: Amir Eskandari
Aman Anand
Elyas Rashno
Farhana Zulkernine
Venue: Accepted by TMLR ()

Abstract: Graph Neural Networks (GNNs) are the go-to model for graph data analysis. However, GNNs rely on two key operations—aggregation and update, which can pose challenges for low-latency inference tasks or resource-constrained scenarios. Simple Multi-Layer Perceptrons (MLPs) offer a computationally efficient alternative. Yet, training an MLP in a supervised setting often leads to suboptimal performance. Knowledge Distillation (KD) from a GNN teacher to an MLP student has emerged to bridge this gap. However, most KD methods either transfer knowledge uniformly across all nodes or rely on graph-agnostic indicators such as prediction uncertainty. We argue this overlooks a more fundamental, graph-centric inquiry: "How important is a node to the structure of the graph?" We introduce a framework, InfGraND, an Influence-guided Graph KNowledge Distillation from GNN to MLP that addresses this by identifying and prioritizing structurally influential nodes to guide the distillation process, ensuring that the MLP learns from the most critical parts of the graph. Additionally, InfGraND embeds structural awareness in MLPs through one-time multi-hop neighborhood feature pre-computation, which enriches the student MLP’s input and thus avoids inference-time overhead. Our rigorous evaluation in transductive and inductive settings across seven homophilic graph benchmark datasets shows InfGraND consistently outperforms prior GNN to MLP KD methods, demonstrating its practicality for numerous latency-critical applications in real-world settings.

practicality for Published in Transactions on Machine Learning Research (1/2026)
InfGraND: An Inflee GNN-to-MLP Knowledge Distillation
Amir Eskandari
omin amir.eskandarii@queess..c
School of Computing Queen's University
Aman Anand
oman. wma..onaand@quueru...cc
School of Compuling Queen's University
Elyas Rashno
elyos. www.enooooooouuu.oooouuu
School of Computing Queen's University
Farhana Zulkernine
fartara..ulkermiine@quueeee
School of Computing Queen's Universily
Reviewed on OpenReview: https: // openreview. net/ forum? id= 1f2HR3YulD
Abstract
Graph Neural Networks (GNNs) are the go-lo model for graph data analysis. However, GNNs rely on two key operations aggregation and update, which can pose challenges for low-latency inference tasks or resource-constrained scenarios. Simple Multi-Layer Per- ceptrons (MLPs) offer a computationally efficient alternative. Yet, training an MLP in supervised setting often leads to suboptimal performance. Knowledge Distillation (KD) from a GNN teacher to an MLP student has emerged to bridge this gap. However, most KD methods either transfer knowledge uniformly across all nodes or rely on graph-agnostic indicators such as prediction uncertainty. We argue this overlooks a more fundamental, graph-centric inquiry: 'How important is a node to the structure of the graph!" We in- troduce a framework, InIGraND, an Influeecc-uuided Graph KNowledge Distillation from GNN to MLP that addresses this by identifying and prioritizing structurally inffuential nodes to guide the distillation process, ensuring that the MLP learns from the most critical parts of the graph. Additionally, InfGraND embeds structural awareness in MLPs through one- time multi-hop neighborhood feature pre-compulation, which enriches the student ML.'ss input and thus avoids inference-time overhead. Our rigorous evaluation in transductive and inductive settings across seven homophílic graph benchmark datascts shows InfGraND con- sistently oulperform prior GNN to MLP KD methods, demonst numerous latency-critical applications in real-world settings.
Keywords: Graph Neural Networks, Knowledge Distillation, GNN-to-MLP Knowledge Distillation
1 Introduction
Graph-structured data has become increasingly important a range of applications. For instance, social networks use graphs to model user interactions facilitate effective identification misinformation within communities (Han et al.] 2020 Fan al., 2020) Sharma et al.] 2024).. Similarly, e-commerce websites employ graphs to capture user-product relationships provide personalized recommendations that increase

---

Published in Transactions on Machine Learning Research (1/2026)
revenue (Wu et al. 2022: Wang et al. 2021 Gao et al. 2023 More recently, graphs have been widely
adopted as a powerful source of external knowledge for Large Language Models (LLMSs) (Zhao et al. 2023)
through Retrieval- Augilened Generation (RAG) systems, which enable both enhanced factual grounding
and deeper contextual reasoning (Peng et al. 2024 Han ct al. 2024 The widespread use of such graph-
applications demands efficient analytical methods.
Graph Neural Networks (GNNs) (Kipf & Welling 2016
Hamilton al., 2017 Veličković al.) 2017 Wu
et al. 2023a Liu et al. 2020 Wu et al. 2020 Zhou et al., 2020; Li et al. 2019 Chen et all 2020 emerged
as powerful framework to process graph data. At their core, GNNs utilize layer-by-layer message- pessagg-passing mechanism to Iearn rich node-level representations, which has proven highly effective in the aforementioned applications. However, this academic success has not been fully translated into industrial practice. The high computational and memory demands of message passsin create significant bottlenecks during training and inference, limiting their use in production environments (Zhang et al.; 2020; Min et al.] 2021 Jia ct al. 2020). To overcome these constraints, industry often relics on a much simpler alternative, the Multi- Layer rercept.on (MLP). Although resource efficient, ML.s show less competitive performance because they operate exclusively on node features, ignoring valuable structural knowledge within the graph (Zhang et al. 2022).
To address the performance-eflicinncy trade-off, one line of research focuses on developing more efficient GNN architectures (Bojchevski et al, 2019 Huang et al. 2018 Ying et al. 2018 Chen et al. 2018). A
more recent and popular research direction leverages the efficiency of MLPs while preserving the expressive power of GNNs through Knowledge Distillation (KD). In KD, a well-traine its knowledge to an MLP student (Yang et al., 2021 Zhang et al.] 2022; Gou et al., 2021 Hinton 2015).
GNN teacher model transfers
The GNN-to-MLP distillation field has evolved along several directions. A significant body of work focuses on enhancing the student side. Some approaches increase the complexity of the student MLP by employing
advanced architectures such as ensemble methods or Mixture-of- of-Experts (Lu et al.
2024 Rumiantsev &
Coates 2024). Others aim to enrich the MLP's input by injecting structural knowledge through positional
encoders or structure-aware lokenizer (Tian el al. 2022: Chen el al. 2024 Yang el al.
2024 Although often
effective, these strategies increase computational overhead. Moreover, they treat all graph nodes uniformly during distillation, overlooking the varying importance of different nodes.
This equal treatment has led to paradigm of discriminative distillation (Wu et al. 2023c works use entropy to rank nodes and sample them accordingly, prioritizing nodes where the teacher is less confident. However, we argue that this approach is fundamentally graph-agnostic: it relies on the teacher GNN's confidence in its predicted label for each node rather than the node's structural role within the graph. In other words, they discriminate between nodes based on How certain is the teacher GNN about this node label? This raises a fundamental question: should we not instead ask How influential is this node within the structure of the graph?
2024 These
To answer this question, we propose InfGraND), an influence-guided knowledge distillation method built on graph-aware influence metric that moves beyond prediction uncertainty. The influence score is a topology- aware indicator that measures how perturbations in a node's fealures affect the representations of the other nodes after message propagation. For the influence score, we leverage an influence maximization strategy inspired by previous work in the active learning litcrature (Li et al. 2018). Our preliminary experiments confirm that prioritizing high-influence nodes consistently yields superior teacher GNN performance (see empirical validation in Section 5.2.1 including Figure 2) Based on this observation, InfGraND employs deterministic woithteighting scheme in the distillation via subgraph-level distillation loss (Yang et al,, 2020). It discriminates among neighbors in the subgraph based on their influence score. This influence metric is parameter-free. To further incorporate structural knowledge at the input level, InfGraND enriches the input features of the student. Inspired by practices in large-scale industrial systems (Li et al. 2013) like pre-computed embedding tables, we utilize an efficient one-time feature propagation and pooling operation. This approach allows the MLP to access rich hop neighborhood information without adding inference overhead.
Our evaluation covers both transductive (i.e., training and testing on the same graph) and inductive (i.e., training on one graph and testing on another) settings. Widely adopted GNN teachers such as Graph
2

---

Published in Transactions on Machine Learning Research (1/2026)
Convolutional Network (GCN) Kipff & Welling 2016), Graph Attention Network (GAT) (Veličković et al, 2017), and GraphSAGE (Hamilton et al. InfGraND against state-of-the-art (SOTA) GNN-to-Min MoLtoo-LPPaa- models, particularly those designed for non- and discriminative distillation. We also conduct experiments in scenarios where labels are limited, as well as comprehensive ablation and sensitivity analyses to provide further insights into the behavior of the model. The following are our main contributions.
2017] are used, with MLPs serving as students. We benchmark
We provide a taxonomy of GNN-to-MLP distillation methods, revealing that existing approaches either add computational complexity employ graph-agnostic discrimination strategies.
We propose InfGraND, novel framework that, to the best of our knowledge, is the first to perform node-level discrimination by computing node influence based on the graph structure in GNN-to-MLP distillation.
We validate our framework through rigorous evaluation on seven real-world homophilic graph bench- mark datasets in both transductive and inductive settings. Our experimental results demonstrate that InfGraND consistently outperforms these competing approaches. It not only substantially outperforms vanilla MLPs and existing stato-of-the-art distillation methods, but in many cases, even surpasses the performance of its own GNN teachers. Results from label-limited scenarios and ablation studies further validate the effectiveness of our proposed method.
The remainder of the paper is structured follows. In Section 2 present review of relevant research and categorize the MLP distillation paradigm. Section 3 provides the necessary background concepts to support our approach. Our proposed method and its components are detailed in Section 4 Section 5 outlines the experimental upt and presents the results. We conclude in Section with sunmary of our contributions and a discussion of future research directions.
2 Related Work
The evolution of GNNs reflects ongoing efforts balance expressiveness with computational efficiency. The early spectral methods (Bruna et al. 2013 Henaff et al. 2015
Defferrard et al. 2016 Kipf & Welling 2016)
leveraged graph Fourier translorms but faced scalability challenges Spatial GNcheMichel 2009 Scarselli
et al. 2008 Hamilton et al.) 2017 Veličković et al.
2017) improved scalability through localized message
passing, yet the recursive nature of aggregation still limits deployment at scale. GNN-to-MLP distillation
has emerged as a more recent approach to climinate message passing while preserving the knowledge learned by GNN teachers.
2.1 Distilling Graph Knowledge into MLPs
Graph-less Neural Networks (GLNN) (hhng el al. 2022) established the foundational framework by train- ing a student MLP on the soft labels from GNN teacher. Since then, the field has evolved along several directions. One line of work incrcases the student MLP's capacity to improve performance. For instance, AdaGMLP (Lu et al., 2024 uses AdaBoost-style ensemble of MLPs, while RbM (Rumiantsev & Coates 2024) proposes a Mfxture--o-Fxxpetts (MoE) student model that enforces expert specialization on different regions of the representation space. However, these methods increase the model complexity and inference overhead. Another research direction enriches the input features of MLPs with explicit structural knowl- edge. NOSMOG TTian et al. 2022) incorporates positional features from DeepWalk, (Perozzi et al. 2014). concatenating them with node features to make the student MLP structuro-aware. SA-MLP (Chen et al. 2024) directly encodes the adjacency matrix with linear layer to integrate structural knowledge. VQGraph (Yang et al. 2024] learns a structure-aware tokenizer to create discrete codebook of local graph structures for more expressive distillation targets. These strategies add computational overhead by expanding the input dimension of the student MLP and often require separate training for positional information. Despite these advances, both lines of work treat all nodes uniformly during distillation, overlooking that nodes vary in their importance within the graph structure.
Non- Uniform Distillation. A third line of work discriminates between nodes using prediction uncertainty. KRD (Wu et al. 2023ch quantifies "knowledge reliability" via prediction entropy stability under noise and
3

---

Published in Transactions on Machine Learning Research (1/2026)
samples more reliable nodes during training. HGMD (Wu et al. 2024 defines "knowledge hardness" via entropy and extracts hardness-aware subgraphs as additional supervision for challenging samples. However, entropy-based metrics are graph-agnostic, assessing prediction confidence rather than structural influence.
We address this gap by introducing a graph-aware node importance metric that quantifies structural infiuence rather than prediction confidence. We also incorporate structural features efficiently via one-time pre- computation, eliminating the inference overhead of existing input augmentation approaches.
3 Background
Consider == (V, X) an attributed graph, where is the set of N nodes with features X = X2, XN|= E Rd and & denotes the edge set. A d-dimensional features vector x; is assigned for each node U E V. Each edge Gi E & denotes connection between nodes vs and By. The graph structure is represented by an adjacency matrix A NNN E [0,1] with Ai, = 1 if ei E and A = 0 if eij $ E. Also for each node, we have an assigned label y; E (0, where C' is the number of classes.
Notations.
semi-supervised node classification tasks, only subset of nodes Via with labels Yiab are known. The Vab nodes labeled as the set Diub (Vab, Ylab)- The unlabeled set is defined as Dunl (Vunt, Yui)), where Vunt = V Vib. The node classification task aims to learn a mapping Φ X ++Y Y via Yiab to infer mnt-- Each label is typically represented as one-hot vector in Y E RNC." The transductive setting utilizes the full graph Guain = G = (V, E, X) where V = Viab UV during training, with access to node features X = [x1,X2, X2...., Xy] E N xxd Vu; E V. The model is then evaluated on the nodes Vuml encountered during training. Under the inductive setting, the model is trained on an observed subgraph Gobe (Vobe, &obe, Xobs), where Eabs {er, E Us By E Vobs} and Vab S Vobe. The model is then tested on completely unseen nodes and their edges in Gunoba, where Viest Vunoba, evaluate its ability to generalize to new graph structures.
Node Classification.
Graph Neural Networks (GNNs). Most existing GNNs follow message-passing scheme that consists two key computations for each node 0; (1) AGGREGATE: aggregates messages from neigh- borhood (w)); (2) UPDATE: updates node representation based on the output of the previous layer and aggregated messages. For L-layer GNN, the formulation of the 1-th layer is the following:
) h n UPDATE (L) m (n.-",,,.))
m = AGGGEGGTE" (InH-V :s (((i)))).
(1)
where 1 1 5 L. h,9 is the representation of node o; at the 1-th layer, and m; from its neighbors. The process is initialized with the input features, where h (0) x,. Common GNN
(1) is the aggregated message
variants include GCN (Kipf & Welling 2016), GraphSAGE (Hamilton el al. 2017 and GAT (eelčkovid
et al. 2017).
4 Proposed Method
This section describes the proposed InfGraND framework in detail. First, introduce node influence measurement to quantify node importance (Section 4.1). Next, Section 4.2 explains the pre-computation step that enables the MLP to capture structural graph knowledge. The section concludes by presenting the full influence-guided distillation process (Section 4.3
4.1 Node Influence Measurement
To determine node importance, we adopt a graph-aware node influence framework that quantifies how perturbations a single node's fcatures propagate through the graph to affect the representations of other nodes (Zhang et al. 2021). Therefore, node with greater effect on the graph is considered topologically more influential.
4

---

Published in Transactions on Machine Learning Research (1/2026)
Knowledge Distillation (KD)
Graph Structure
Teacher Context
Node #1's Neighbours
Node Features
Teacher Logits:
Stronger
Weaker Distilation
GNN Teacher
High-Infuence Nodes
Studentt Logits:
Ca
Iww.unifueece Nocks
MLP Student
Node Features
Student Context A
(a) The GNN Teacher, trained using supervised learning, re- mains frozen during distillation. The MLP Student, which only sees node featurcs, learns structural information by mim- icking the teacher output.
(b) subgraph-level KD, the student learns the represen- tation of node from neighbors provided by the teacher, weighted by influence scores. High-inftuence neighbors yield stronger supervision.
Figure Overview of Influence-Guus Guided Distillation. Our method uses node structural influence weight the knowledge transfer, ensuring that the student learns local structure from its most important neighbors.
Definition 4.1 (Quantifying Node Influence). Within a graph g EV, we define the node influence score of source node i target node by after & mnssage-ppsssinn iterations the L1-norm of the expected Jacobian matrix:
Ox (k) ax
1 Us, k)
E
(2)
00
where x, represents the feature embedding of the larget node vj the k-th iteration, and x; (0) features of the source node 22 To provide a relative measure of influence, we use a normalized influence score:
the initial
[(je- () vt,k)
10 k) Emwev (oew) (Uj; k))
(3)
This normalized score, (( (1y, vi, k), represents the proportion of inffuence of vs on By relative to the total influence of all other nodes that feed to my in the graph.
To practically measure this influence, must approximate the expected Jacobian term in Eq. 2 A direct calculation is often intractable. Previous work establishes that the expected influence is equivalent to the aggregated influence on all k-length random walks between two nodes (Xu et al.) 2018). Inspired by Sim- plifying Graph Convolutional Networks (SGC) (Wu et al. 2019), we remove the non-linear activations and weight matrices. This simplifies the GCN down to its core function pure topological propagation, defined as:
X = 1x(
(4)
where Á is the normalized adjacency matrix, defined as Ā = D- - r)D-1, with D being the degree matrix. We use k (see Appendix H for sensitivity analysis) that the resulting embedding x(2)
contains information from its 2-hop neighborhood. We therefore use the cosine similarity, simena (x: (0) x ()), efficient and parameter-fit indicator to determine the influence of node resulting influence scores lie between 0 and 1, we apply MinMaxScalle [oomer et al. 2014) globally per graph. Influence computation is a one-time preprocessing step during training, not a recurring cost during inference. More details about the influence computation are provided in Appendix
By. To ensure that the
To enhance knowledge distillation using a node influence score, instead of a pairwise score (Eq. 3), we define Global Influence Score (GIS) that measures the overall impact of each node on the entire graph and assign that to each node.
5

---

Published in Transactions on Machine Learning Research (1/2026)
Definition 4.2 (Global Influence Score). The global influence score (v())) of node after k message
passing iteration is defined as:
Ejev 10-) (1/ Vi, k))
Ig(4s)
(5)
max LEV
Ejev (o v1, ki)
5 the numerator is normalized by the maximum global influence score across all nodes, which ensures that (v,) lies within the range [0, 1].
where ( oi, k) defined Eq. 3 In Eq.
4.2 Node Feature Propagation
To provide the student MLP with structural knowledge, InfGraND incorporates an efficient feature prop- agation scheme. We perform this as a one-time, offline pre-computation step before training begins. This approach is inspired by common practices in large-scale industrial systems such as using a parameter server manage precomputed embodding tables (Li et al. 2013). We use the lincar propagation from Eq. 4 to generale multi-hop feature matrices, (X(P) pp=0- To avoid increasing the input dimensionality adding parameters, apply average pooling across these matrices instead of concatenation:
X = POOL ((x()))p=))
(6)
The resulting matrix, X, contains multi-hop neighborhood information. serves the fixed input to the student MLP. This pooling is computed once during an offiine propagation step. As result, there is no added cost inference time. The model remains efficient during deployment.
4.3 Distillation
An overview the distillation mechanism is provided in Figure (a). The training process starts with standard supervised training of the teacher GNN. Once trained, teacher frozen. The student MLP is then trained using composite objective that learns from both the ground-truth labels and the soft predictions provided by the teacher.
To ground the student in the true class distributions, use influence-weighlle supervised loss, L.. This loss is applied only to the labeled nodes, Viab
C. = 51 Σ Dc
E Iz(es) Dc n4 ",VVaa
(7)
where DCE denotes the standard cross-entropy loss, h is the student's reprosentation for node vt, a()) is the softmax function, and (v:)) is the global influence score of node v, The hyperparameters 8, and 8g control the contribution of the standard and selteence-weighter loss terms, respectively.
For KD, our primary loss, La, leverages the homophily principle common in graphs (Yang et al. 2020 Training encourages the student's prediction for node i, he, to be similar to the teacher's predictions for its neighboring nodes j, hý, via a Bullback-Leibllee (KT.) divergence loss, denoted as DKL. The term denotes the distillation temperature. As illustrated Figure (b), the influence score of the teacher's node, (v;), directly weights the distillation process, allowing high-influence neighbors to provide stronger distillation signal. The loss is defined as:
La La= = Σ Σ
(n + + 1 Iz(;)) [((x)) DRL(o(h)/T)) (1)) JEV
(8)
The design of this loss is crucial. The 7 term provides a baseline distillation gradient from all neighbors, while the 72 (1²) term acts fine-grained amplifier for more influential nodes. We provide full

---

Published in Transactions on Machine Learning Research (1/2026)
theoretical analysis of this loss function, including derivation of its gradients (Appendix D.1 the effect of the influence score (Appendix D.2), and the need for (Appendix D.3) to rigorously justify this formulation.
The overall training objective for the student model combines these two losses:
= + (1 A)Ga,
(9)
where E [0,1] is hyperparameter that balances the supervised and distillation signals. training procedure is presented in Algorithm (Appendix Gh.
The complete
5 Experiments and Results
We conduct rigorous experiments to comprehensively evaluate the effectiveness of our InfGraND framework by addressing six key research questions. Q1: Does training a GNN on high-influence nodes lead to better performance than using low-influence nodes? Q2: How does InfGraND perform on node classification tasks compared to baseline GNN-to-MT.P distillation methods, its corresponding GNN teacher models trained with supervised loss, and vanilla MLP trained without distillation? Q3: What is the tradc-off between classification accuracy and inference latency for InIGraND compared to alternatives? Q4: What is the relative contribution of each component of InfGraND to its final performance? Q5: How robust is InfGraND's performance when the number of available training labels is severely limited? Q6: What is the impact of key hyperparameters on the performance of InfGraND, specifically the influence-related loss weights (12, 62), the number of propagation steps (P), and the choice of pooling method?
5.1 Experimental Setting
Datasets. We evaluate InfGraND in both transductive and inductive settings on seven real-world datasets
with inherent graph structures: (1) Cora (Sen et al.) 2008
(2) Citeseer (Giles et al.. 1998) (3) Pubmed
[McCallum et al. 2000), (4) Amazon-Photo, (5) CoAuthor- CS, (6) CoAuthor-Phy (Shchur et al. 2018
and (7) the large-scale OGBN-Arxiv dataset (Hu et al.) 2020). For small-scale citation datasets (Cora,
Citeseer, and Pubmed), we the splitting strategy of Kipf et al. (2016). For CoAAthor-CSS CoAuthor-Phy, and Amazon-Photo, we adopt the random split method (2022). Finally, for the OGBN-Arxiv dataset, we use the official splits from Hu et al. (2020). We choose a random seed and apply it consistently to ensure identical splits across experiments for fair and reproducible ovaluation; different sceds produce different splits. Datasct details and splitting statistics are discussod in the Appendix
used by Yang et al. (2021) and Zhang et al.
Implementation. We utilize three GNN architectures as teachers: GCN (Kipf & Welling 2016). GAT (Veličković et al.) 2017), and GraphSAGE Hamilton et al., 2017). We select these models as they represent diverse and widely-adopted design paradigns. We benchmark InfGraND against strong set of competitive baselines, including the foundational GLNN (Zhang et al. 2022) and the non-uniform distillation frameworks KRD (Wu et al.) 2023c), HGMD (u et al. 2024). and FF-G2M (Wu et al.) 2023b). We reproduced the results for all baselines using their official public implementattons. This step was crucial because our experimental settings and inductive data splits for certain datasets and teacher models differed from those in the original papers. Note that at the time of our experiments, the official HGMD implementation only supported the transductive setting with GCN teacher, limiting our comparison to that setup. To ensure reproducibility, we used fixed set of five different random seeds and reported the average performance on five runs. The hyperparameters were tuned using the WandB platform (Biewald, 2020), with validation accuracy as the tuning criterion. To ensure every model was evaluated at its peak, we performed random hyperparameter searches for all methods, including baselines and our proposed INFGRAND, until performance on the validation set saturated. Appendix provides additional information on reproducibility. Our implementation uses PyTorch (Paszke et al. 2019)) and the DGL library (Wang et al. 2019 with experiments run server equipped with an NVIDIA V100 GPU (32GB VRAM).
7

---

Published in Transactions on Machine Learning Research (1/2026)
High Influence
Low Influance
Cora
Citeseer
PubMed
Test Accurscy
49%
SAGE
dey
GAT
SAGE
SAGe
Figure 2: Test set classification accuracy three GNN models (GCN, GAT, GraphSAGE) when trained on a subset of high-influence low-influence nodes across different datasets (Cora, Citeseer, PubMed). Error bars represent standard deviation over five runs.
5.2
Evaluation
5.2.1
Q1 21-fffect of Influence GNNs
We evaluate the impact of training GNNs on nodes with different influence scores in the transductive setting, To do so, we divide the training set into two subsets, each containing 25% of all labeled nodes. We ensure the selection preserves class balance. One subset contains high-influence nodes (top 25% per class), the other low-influence nodes (bottom 25%). Then, we train separate GNNs (GCN, GAT, GraphSAGE) on each subset using a fixed test set.
Discussion. As shown in Figure models trained on high-influence nodes (blue bars) consistently outper- form those trained on low-inftuence nodes (red bars) across all datasets (Cora, Citeseer, and PubMed). This consistent performance gap provides strong empirical evidence that the influence score effectively identifies nodes most critical to GNN's generalization. This finding directly motivates the design of InfGraND, which prioritizes knowledge transfer from high-influence nodes during the distillation process.
5.2.2 Q2 Classification Performance
Since we observed that training teacher high-influence nodes improves generalization, prioritize these nodes during distillation, which should enhance the performance of the distilled MLP. Empirical evidence supporting this hypothesis is provided in Table [ and Figure 3 Table [ reports node classification results in the transductive and inductive settings. Figure 3 presents results on the large-scale OGBN- Arxiv dataset for InfGraND, KRD, GraphSAGE, and Vanilla MLP.
Discussion. All the results support the hypothesis and demonstrate the following: (1) InfGraND on average outperforms all baselines in both transductive and inductive settings and achieves the highest accuracy most cases; (2) in nearly all cases, the distilled MLPs surpass their GNN teachers, a key finding that challenges the assumption that expressive GNNs are always superior to MLPs
graph data; (3) the performance gains over non-distilled MLPs are substantial. As shown in Table [ InfGraND effectively bridges the gap between MLPs and GNNss Compared to the vanilla MLP, the InfGraND-distill
MLP achieves an average improvement of 12.6% under the transductive sctting and 9.3% under the inductive sctting, which corresponds to the average A across all three teacher architectures; (4) InfGraND outperforms discriminative distillation methods, including hardness- and reliability-baser approaches, with average gains of 0.9% over KRD (transductive), 3.0% (inductive), and 0.6% over HGMD, based on the A values in Table (5) InfGraND scales well to large graphs, outperforming the KRD baseline on OGBN-Arxiv under both transductive and inductive settings (Figure 3); (6) stronger teacher does not necessarily yield a stronger student. On Amazon-Phote (transductive), GCN outperforms GAT (90.7% vs. 87.6%), yet the GAT-distilled student achieves higher accuracy (94.5% vs. 94.2%), indicating that leacher accuracy alone does not determine distillation effectiveness.
8

---

Published in Transactions on Machine Learning Research (1/2026)
Table 1: Node classification accuracy (%) for various models under transductive and inductive settings, averaged over 5 runs. Boldface indicates best performance. Dataset abbreviations: Amazon = Amezon-Phote CS Coauthor-CS, Coasthor-Physics. Gray-shaded rows correspond to models trained using only supervised loss functions (i.e., without distillation). A denotes the average accuracy improvement of InfGraND over the corresponding baseline method across all datasets.
Phy =
Method
Trandduriive Accuracy
Indhative Aueurcc
Coea
Cloeeer
Pulend
Amasun
CS
Phy
A
Cira
Chreser
Palmnd
Amaron
CS
A
Vanilla GCN
822
71.6
70.2 us
90.7 13
25.5 1:
91.9 La
43.3
80.6 L4
6488 GG
71.0 1:
59.4 6.
90.0 13
+2.
Vanilla MILP
57.8
06.5
71.8
87.8
99.5
+126
58.1
16:
70.8
88.1
860
+10.0
GCN
GLNN
92.3
02.0
41.3
051
725
550
9019
91.7
10.1
10.5
6500
75.0
902
+33
F16432
93.0
10.0
50.7
+1.0
6588
728
502
97.8
+3.5
92.3
50.4
40.5
IofGeaND
Venllia SAGE:
09.7
+2.
11.2
8M0
90.5
+211
Viailla MLP
110.1
SAGE
GLNN
75.0
5:34
0.00
9.07
10.3 1.
+1.3
00.0
1:17
73.5 a.
N8-1
98.4
Vaailla GAT
+1.
Vaaille
8.0.
3:33
=:10
78.6
2.13
GLNN
+1.7
=m
63.0
72.9 1.05
514
91.8 1:24
94.3 1.53
940
10.9
74.0 0.00
660
74.9 1.8.
876.18
5000
L.
1:23
12.8
FI-33M
14
50. 0.66
91.0 3.3
01.0 1.0
940 11
+1.5
=
631
725 1.3
89.3 s.1
900
92.0
+33
IIGGaND
64.2
04.5 8:93
94.2 4:33
94.4 mm
70.9
75.1 6:43
91.8 u.
01.2
98.0
6:23
5.2.3 Q3 Computational Time and Efficiency
In Figure g we present the trade-off between accuracy and inference time under the transductive setting, using the Citeseer dataset as a representative example. To ensure fairness, we keep the number of layers and hidden dimensions consistent across MLP, InfGraND, AdaGMLP, and GLNN. While Tian et al. (0022) report that wider GLNNs with more hidden neurons achieve higher accuracy at the cost of longer inference time, in our experiments we did not observe this effect: simply increasing the number of hidden dimensions or layers did not consistently improve performance. In fact, even a small number of hidden dimensions was sufficient to reach 100% training accuracy, suggesting that greater model complexity docs not necessarily yield better results. Therefore, given that additional complexity did not improve performance, we kept the hidden dimensions and layers of the distilled MLP methods fixed and did not report results for larger variants.
12.5
CaT42
SAGE-L
CARL
300
01
Log Scale Inforance Time (ma)
Figure 3: Transductive and inductive results on OGBN- Arxiv with a SAGE teacher. Zoomed-in in views highlight the superior performance of InfGraND.
Figure 4: Trade-off between model accuracy and infer- ence time on the Citeseer dataset. The x-axis shows inference time in milliseconds (log scale), and the y-axis shows classification accuracy.
Discussion. As shown in Figure 4 GraphSAGE, GCN, and GAT achieve the best results with 3, 2, and 2 layers, respectively. InfGraND gains 4.3% improvement over GraphSAGE-L3, 3.8% over GCN-L2, and 4.8% over GAT-L2 while being 8.56x, 6.84x, and 13.89x faster respectively. Also, AdaGMLP; which focuses on enhancing MLPs, required more time than InfGraND and GLNN, as it relies on an ensemble of models for prodiction. We do not report results for FF-G2M, KRD, and HGMD, as they share the same architecture as InfGraND and thus have identical inference times. These consistent results demonstrate InfGraND as an efficient. and accurate method for graph learning.
9

---

Published in Transactions on Machine Learning Research (1/2026)
5.2.4 Q4 Ablation Study
We conduct an ablation study to evaluate the contribution of each core component in InIGraND. We use GCN and GraphSAGE as teacher models in the transductive and inductive settings, respectively, and evaluate performance on six benchmark datasets. To isolate the effect of each module, we consider two simplified variants: w/Influence and s/Propagation. In the w/Influence variant, the student MLP is trained using raw input features X, without the multi-hop propagated version X. In contrast, the w/Propagation variant disables the cetttencc-guiidd objectives by setting the corresponding loss weights to zero: 7/2 = 0 and 82 = 0 in L. and La, respectively. Results are summarized in Table 2
Table 2: Ablation study comparing the full InfGraND model with variants using only influence guidance (w/Influence) or only feature propagation (w//Propagation). Results report classification accuracy (%) across multiple datasets under both inductive and transductive settings. Boldface and underline denote the best and second-best performance, respectively.
Setting
Method
Cora
Citeseer
Pubmed
Amazon-Phos
Coauthor- CS
Coauthor-Phi
Vanilla GCN Vanilla MLP
82.2 + 0.6
71.6 + 0.2
79.2 + 0.3
90.7 + 0.3
89.3 + 0.0
91.9 + 1.3
57.8 = 1.0
60.5 0.7
72.8 0.4
79.0 +
87.8 +
89.5 + 2.0
GLNN
83.1 = 0.3
73.0
79.4 0.6
92.3 + 0.5
92.6 +
93.6 + 1.1
Transductive
InGGraND w/Iniluence
83.4 = 0.8
74.6 4 0.6
81.1
92.1 + 0.2
93.2 + 0.7
93.8 + 0.8
InGGraND w/ e/Propagation
83.6 = 0.5
75.0 0.9
81.0
93.0 + 0.3
93.0 +
91.3 + 0.3
InfGraND (Full Model)
84.0 +: 0.5
75.2 1.1
81.3 = 0.2
94.2 + 0.4
93.5 0.6
94.7 = 0.0
Vanilla SAGE
79.6 + 1.5
64.7 + 0.8
73.0 + 2.0
91.2 + 0.8
89.0 + u.7
90.5 + 1.7
Vanilla MLP
58.4 + 2.5
55.1 + 1.0
70.8 + 1.2
78.6 + 1.2
88.1 + 1.3
89.0 + 0.9
GLNN
60.6 + 1.7
64.0
72.4 0.5
85.0 + 2.2
89.3 +
91.0 + 3.0
Inductive
InfGraDD w/Influece
70.5 + 1.6
63.2 + 1.0
74.0 + 1.1
85.0 + 2.2
90.8 + 1.5
90.6 + 3.0
InGGraND s/Propagation
79.5 +: 1.6
67.4
74.1
89.3 + 2.2
91.4 + 1.1
92.6 + 2.7
InfGraND (Full Model)
79.9 + 0.6
67.7 1.1
74.3 + 1.1
93.5 + 1.6
91.8 0.7
93.3 + 3.0
Discussion. Both the inffuence-guided objective and the feature propagation module independently contribute to the overall performance of InfGraND, and their effects are complementary. The w/Influence variant, which selectively transfers knowledge based on node influence within graph, consistently improves upon the Vanilla MLP, GLNN, and teacher models across both transductive and inductive settings. Mcanwhile, the s/Propagation variant equips the student with structural knowledge via pre-computed multi-hop features and yields particularly strong gains in the inductive setting, most notably on Citeseer, Cora, and Amazon-Photo. Specifically, outperforms the Vanilla MLP by +10.7%, +12.3%, and +21.1% on these datasets, respectively. Importantly, these improvements come without introducing additional parameters or training overhead.
The significant gains observed on Cora, Citeseer, and Amazon-Photo in inductive setting can be attributed to their splitting characteristics (see Appendix A Table 5) Citeseer, Cora, and Amazon-Photo exhibit higher proportions observed and test nodes relative to the total node count. This broader coverage enables the propagation of features to capture more useful neighborhood information, thus improving the generalizability of the student model.
Note that both components are most effective when used together. In our experiments, all hyperparameters were tuned jointly for the full model. We observed that tuning them independently for the w/ e/Influence w/Propagation variants often yiclds better results than those reported in Table 2 as cach variant has its own optimal configuration.
5.2.5
Q5 Label-Scarce Setting
A key challenge in semi-supervised node classification is the high cost of labeling, which is inherently tedious, conseconsuming, and resource intensive Often, we only have access to a limited number of labeled nodes. For example, our main experiments, for the transductive setting, we use only 20 labeled nodes per class, which is comparatively very low compared the number of test nodes (1000 nodes). This scarcity of labels, where Manl << [annl, highlights the need for models that perform comparatively well even with limited
10

---

Published in Transactions on Machine Learning Research (1/2026)
labeled data. To evaluate InfGraND's performance in this setting, conduct an experiment by limiting the number of labeled nodes used in training phase. We compare In(GraND and GLNN using GraphSAGE teacher in transductive settings and three datasets: Cora, Citeseer, and Pubmed. In this experiment, we randomly selected 2, 4, and 8 labeled nodes per class, corresponding to 10%, 20%, and 40% the original training sel, respectively. We keep the lesting set. the samee across all training settings. We use the same seed ensure fair comparison between the methods.
Pubmed
90.0
75.0
72.5
55
16.4
75
Accumcyy
97.5
07.8
45
62.5
02.0
19%
20%
40%
30
10%
20%
40%
60
136
20%
40%
Figure 5: Accuracy comparison between InfGraND and GLNN under different proportions of labeled training samples (10%, 20%, 40%) on Cora, Citeseer, and PubMed, using GraphSAGE as the teacher model.
Discussion.
The results, as shown in Figure 3 demonstrate that InfGraND consistently outperforms GLNN in all three test cases and datasots. InfGraND surpasses GLNN by an average 4.17%. This superior perfor- mance under extreme label scarcity suggests that InfGraND's influence-uideed objective effectively prioritizes influential nodes during training, enabling robust generalization even when labeled data are minimal.
5.2.6 Q6 Hyperparameter Analysis
To better understand the behavior InfGraND, we conduct a hyperparameter scnsitivity analysis using a GraphSAGE teacher on Cora and Citeseer. We vary 72, 62, and across 10 values 1.00, adjust the number of propagation steps P from to 4, and compare mean, max, and min pooling mechanisms. To emphasize relative trends rather than absolute performance, results the Cora dataset are plotted after subtracting constant offset of 10 percentage points.
5: Parameter
Parameter
à Parameter
0.702
6.50
Cira
Ort Orenner
0.751
Cleeeer
0.15
0.722
0.72
0,740
0.745
Teak Aecuncy
Thak Accunecy
0.742
0.325
0.8-
0.30
0.235
0.25
6.722
0.20
Figure 6: Sensitivity analysis of key parameters (82 (left), 72 (middle), and À (right)) on Cora and Citeseer datasets, showing their impact test accuracy. 62 controls inflnence-guided supervised loss, 72 governs influence-guided distillation loss, and À balances supervised and distillation losses.
Effects of 72 and 5z. The left and middle plots in Figure 6 illustrate the impact of the influence-guided terms in the supervised and distillation losses, controlled by dz and 72, respectively. The parameter 82 appears in the influence-guided supervised loss C. (Eq, 7). while 72 governs the influence aware distillation loss La (Eq. D). As shown in the left plot, setting 82 0.1 yields better results than 52 across datasets, with Cora showing an improvement of approximately 1%. However, increasing 52 beyond this point does not consistently improve performance. Similarly, the middle plot shows that any non-zero value
11

---

Published in Transactions on Machine Learning Research (1/2026)
of 7/2 improves performance over 72 0, suggesting that incorporating influence information, even with suboptimal parameters, is preferable to excluding it entirely.
Sensitivity of X. The hyperparameter E [0, controls the trade-off between the supervised loss
Table Ablation study on features aggregation strategies. Results show classification accuracy (%) with different mum- bers of neighborhood hops
and (C.) and the distillation loss (L.) in the total loss function (Eq. g. A value of = 1 corresponds to pure supervised learning, while À 0 results in pure distillation. Importantly, À does not directly represent the proportion of knowledge transferred from the teacher versus the ground- truth labels; rather, it modulates the relative strength of their gradient contributions during optimization. The right plot Figure shows model performance
Dataset
P Hops
Mean
Maximum
Minimum
1-hop
82.24
0.6
83.82
0.5
82.30 + 1.0
Cora
3-hop
2-hop
84.50 1 0.6
84.26
0.5
84.18
83,90
1 0.2
0.5
81.58 + 1.0
82.70 1: 0.7
4-hop
83.62 = 0.7
83.91 = 0.7
81.22 = 0.9
I-hop
73.16 + 1.0
73.32 = 0.4
71.22 + 2.3
Citeseer
2-hop
74.02
1.0
73.50 + 0.9
70.10 = 1.1
3-hop
73.38 1.3
73.62 0.8
69.62 - 3.7
4-hop
73.08 = 0.7
72.90 = 0.6
68.92 = 6.0
À varies. Performance peaks at À = 0.1, indicating that the model ben- cfits most when it receives a stronger gradient signal from the teacher soft labels than from the ground-truth hard labels. However, removing the supervised loss entirely (A = 0) degrades performance, showing that while distillation provides the dominant learning signal, a degree supervision remains ben- eficial. The worst results occur À 1.0, where the model relies exclusively on labeled data. This trend is consistent. with Table [ where the ML. trained solely with supervised loss performs significantly worse than its distilled counterpart.
1-hop
80.56 +: 0.3
80.24 + 0.1
80.74 + 0.3
Pubmed
2-hop
81.16 = 0.4
80.25 = 0.2
80.58 - 0.4
3-hop
80.80 0.9
80.44 - 0.5
80.46 = 0.5
4-hop
80.96
0.2
80.30 1 0.7
80.88 1 0.7
Pooling and P. For information propagation, as defined in Eq. 6 there are two design choices: the number of propagation steps P, and the pooling mechanism. Table shows that averaging features from 2-hop neighborhoods yields the best performance across Cora (84.50 0.6%), Citescer (74.02 1.0%). and Pubmod (81.16 The 'minimumm aggregation consistently performs worst, likely due to information loss during feature propagation. Extending the neighborhood beyond 2-hops does not lead to further improvements, suggosting that distant neighbors may introduce noise rather than useful structure.
6 Conclusion and Future Work
This work advances GNN- distillation by challenging the uniform treatment of nodes in the dis- tillation process. We define and compute node influence scores and show that prioritizing high-inflence nodes improves the generalization of GNNs. Building on this insight, we introduce InfGraND, which distills influence guided knowledge from teacher GNN to an MLP student. The student also leverages a one-time feature propagation step, inspired by industrial practices such as storing embeddings in lookup tables. Fx- periments across seven homophily datasets confirm InfGraND's superiority. Across six datasets and three teacher architectures, InfGraND improves over vanilla MLPs by 12.6% (Iransductive) and 9.3% (inductive), while also surpassing its GNN teachers by 3.2% and 2.6%, respectively. It also demonstrates clear advan- tages over prior distillation methods, including FF-G2M, KRD, and HGMD. Additionally, on the largo-scale OGBN-Arxiv dataset, InfGraND improves over MLPs by 19.5% (transductive) and 9.5% (inductive), and outperforms KRD by 0.4% on average over the two settings. We also conduct diverse set of experiments to provide insights into the model's behavior from different angles and in various scenarios. These results highlight InfGraND's strong performance and its potential for practical deployment of models that leverage graph structure. Future work will involve extending our method to broader applications, including different graph types such as heterophilous and dynamic graphs. We also plan to investigate hybrid methods that combinc entropy-basod discrimination with structural-aware approaches.
Acknowledgments
This work was undertaken thanks part funding from the Connected Minds program, supportod by the Canada First Research Excellence Fund, grant #CFREF-2022-0001 and the New Frontiers in Research Fund, grant #NFRFF-2022-00199.
12

---

Published in Transactions on Machine Learning Research (1/2026)
References
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL httpp://www.cr wandb com/ ware available from wandb.com.
Soft-
Alcksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Martin Blais, Amol Kapoor, Michal Lukasik, and Stephan Günnemann. Is pagerank all you need for scalable graph neural networks. In ACM KDD, MLG Workshop, 2019.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and deep locally connected networks on graphs preprint ar1iv: 6203, 2013.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling. arXiv preprint urXiv:1801.102/ 10247, 2018.
Jie Chen, Mingyuan Bai, Shouzhen Chen, Junbin Gao, Junping Zhang, and Jian Pu. Sa-mlp: Distilling graph knowledge from gnns into structure-aware mlp. Transactions on Machine Learning Reseorch, 2024.
Ming Chen, Zhewei Wei, Zengleng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning, pp. 1725-1735. PMLR, 2020.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pp. 3844-38877 2016.
Wenqi Fan, Yao Ma, Qing Li, Jianping Wang, Guoyong Cai, Jiliang Tang, and Dawci Yin. A graph neural network framework for social recommendations. IEEE Transactions on Knowledge and Data Engineering, 34(5):2033 2047, 2020.
Chen Gao, Yu Zheng, Nian Li, Yinfeng Li, Yingrong Qin, Jinghua Piao, Yuhan Quan, Jianxin Chang, Depeng Jin, Xiangnan He, et al. A survey of graph neural networks for recommender systems: Challenges, methods, and directions. ACM Transactions on Recommender Systems, 1(1):1-51, 2023.
C. Lee Giles, Kurt D Bollacker, and Steve Lawrence. Citescer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, pp. 89-98, 1998.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: survey. International Journal of Computer Vision, 1779)1178--811) 2021.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, et al. Retrieval-auggenteed generation with graphs (graphrag). arXie preprint arXiv:2501.00300 2024.
Yi Han, Shanika Karunasekera, and Christopher Leckie. Graph neural networks with continual learning for fake news detection from social media. arXiv preprint arXiv:2007.03316 2020.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep convolutional networks graph-structured data. anXiv preprint XrXio:1506. 05163, 2015.
Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint XrXic:1503.02531. 2015.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovee. Open graph benchmark: Datascts for machine learning on graphs. Advances in neural information processing systems, 33:22118 22133, 2020.
Wenbing Huang, Tong Zhang, Yu Rong, and Junzhou Huang. Adaptive sampling towards fast graph repre- sentation learning. Advances in neural information processing systems, 31, 2018.
13

---

Published in Transactions on Machine Learning Research (1/2026)
Zhihao Jia, Sina Lin, Rex Ying, Jiaxuan You, Jure Leskovec, and Alex Aiken. Redundancy-fiee computa- tion for graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 997-1005, 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprinl orxi::1600 (00o:0000,097, 2016.
Brent Komer, James Bergstra, and Chris Eliasmith. Hyperopt-skleam:: Automatic hyperparameter configu- ration for scikit-learn. In Scipy, pp- 32 32-37, 2014.
Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go deep cnns? In Proceedings of the IEEE/CVE international conference on computer vision, pp. 9267-9276, 2019.
Mu Li, Li Zhou, Zichao Yang, Aaron Li, Fei Xia, David G Andersen, and Alexander Smola. Parameter server for distributed machine learning. In Big learning NIPS workshop, mumber 2. Lake Tahoe, CA, 2013.
Yuchen Li, Ju Fan, Yanhao Wang, and Kian-Tee Tan. Influence maximization on social graphs: A survey. IEEE Transactions on Knowledge and Data Engineering, 30(10):1852 1872, 2018.
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proccedings of the 26th ACM SIGKDD international conference knowledge discovery 8 data mining, pp. 338- 138-348, 2020.
Weigang Lu, Ziyu Guan, Wei Zhao, and Yaming Yang. Adagmlp: Adaboosting gun-to-mlp knowledge distillation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2060-2071, 2024.
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the construc- tion of internet portals with machine learning. Information Retrieval, 3:127-163, 2000.
Alessio Micheli. Neural network for graphs: A contextual constructive approach. IEEE Transactions Newral Networks, 20(3):530 540, 2009.
Scung Won Min, Kun Wu, Sitao Huang, Mert Hidayetoglu, Jinjun Xiong, Eiman Ebrahimi, Deming Chen, and Wen-mei Hwu. Large graph convolutional network training with gpu-oriented data communication architecture. arXiv preprint arXiv:2103. 100.03330, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeminn Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: an imperative style, high high-performance deep learning library. Curran Associates Inc., Red Hook, NY, USA, 2019.
Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph seerivval-nngmentee generation: A survey. arXiv preprint arXiv:2408.088 089221, 2024.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710, 2014.
Pavel Rumiantsev and Mark Coates. Graph knowledge distillation to mixture of experts. Transactions on Ma- chine Learning Research, 2024. ISSN 2835-8856. URL https::/oppnneviier httpa//eeeee....eee///omm//////s Expert Certification.
Franco Scarsolli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Trensactions on Neural Networks, 20(1):61 80, 2008
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Collec- tive classification in network data. AT magazine, 29(3):93 93, 2008.
14

---

Published in Transactions on Machine Learning Research (1/2026)
Kartik Sharma, Yeon-Chang Lee, Sivagami Nambi, Aditya Salian, Shlok Shah, Sang- Wook Kim, and Srijan Kumar. A survey of graph neural networks for social recommender systems. ACM Compuling Surveys, 56(10):1-3) 34, 2024.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network ovaluation. arXiv preprint arXiv:1811.0588 2018.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh V Chawla. Nosmog: Learning noise-robust and structure-aware mlps on graphs. arXiv preprint 2rXiv:2208.1000 2022.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. ariie preprint arXiv: 170.. 10903, 2017.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, et al. Deep graph library: A graph-centric, Serbll-performant package for graph neural networks. arXiv preprint Xiv: 01315, 2019.
Shoujin Wang, Liang Hu, Yan Wang, Xiangnan He, Quan Z Sheng, Mehmet A Orgun, Longbing Cao, Francesco Ricci, and Philip Yu. Graph learning based recommender systems: A review. arXiv preprint arXiv:2105.0m 06339, 2021.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pp. 6861-6871. PMLR, 2019.
Lirong Wu, Haitao Lin, Bozhen Hu, Cheng Tan, Zhangyang Gao, Zicheng Liu, and Stan Z Li. Beyond homophily and homogeneity assumption: iilation-bssed frequency adaptive graph neural networks. IEEE Transactions on Newral Networks and Learning Systems, 2023a.
Lirong Wu, Haitao Lin, Yufei Huang, Tianyu Fan, and Stan Z Li. Extracting low-/high figgh-fyyyeeccy knowledge from graph neural networks and injecting into mips: An effective gnn-to-mlp distillation framework. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10351 10360, 2023b.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z Li. Quantifying the knowledge in gmns for reliable distillation into mlps. In International Conference on Machine Learning, pp. 37571-37581 PMLR, 2023c.
Lirong WU, Yunfan Liu, Haitao Lin, Yufei Huang, and Stan Z Li. Teach harder, learn poorer: Rethinking hard sample distillation for gun-to-mlp knowledge distillation. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pp. 2554-2563, 2024.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: a survey, ACM Compuling Surveys, 55(5):1 37, 2022.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprchensive survey on graph neural networks. IEEE transactions on neural networks and learning syslems, 32(1):4 24, 2020.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning, pp. 5453-5462. PMLR, 2018.
Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond it: An iffective knowledge distillation framework. In Proceedings of the web conference 2021, pp. 1227- 1237, 2021.
Ling Yang, Ye Tian, Minkai Xu, Zhongyi Liu, Shenda Hong, Wei Qu, Wentao Zhang, Bin CUI, Muhan Zhang, and Jure Leskovee. VQGraph: Rethinking graph representation space for bridging GNNs and MLPs. In The Twelfth International Conference on Learning Representations, 2024. URL https://
15

---

Published in Transactions on Machine Learning Research (1/2026)
Yiding Yang, Qiu Jiayan, Song Mingli, Tao Dacheng, and Wang Xinchao. Distilling knowl- edge from graph convolutional networks. In Proceedings of the IEEE/CV Conference
Computer Vision and Pattern Recognition, pp. 7074 7083, 2020. doi: 10.1109/CVPR42000 www00/aapes//aaeeee pwt.pprrrraan..sssiliiinng
2020.00710. URL https:://penaccee Knowledge -rom Graph Convolutional Networks CVPR, 2020 1a130__pper_ppp .pdf
theovf . com/ hatp//onteet_cvvi _acpanttcc____eeeee
Rex Ying, Ruining He, Kaileng Chen, Pong Eksombalchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 974-983, 2018.
Dalong Zhang, Xin Huang, Ziqi Liu, Zhiyang Hu, Xianzheng Song, Zhibang Ge, Zhiqiang Zhang, Lin Wang, Jun Zhou, Yang Shuang, et al. Agl: scalable system for industrial-purposs graph machine learning, arXiv preprint arXi::2003.022. 02454, 2020.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mips new tricks via distillation. In International Conference Learning Representations, 2022. URL https:
//arxiv. wttp:///aabb...000//////
Wentao Zhang, Yexin Wang, Zhenbang You, Meng Cao, Ping Huang, Jiulong Shan, Zhi Yang, and Bin Cui. Rim: Reliable saatence-bassee active learning on graphs. Advances in Neural Information Processing Systems, 1127978-279900 2021.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, al. survey of large language models. arXiv preprinl arXiv:2303.188 18223, 1(2), 2023.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: review methods and applications. Al open, 1:57-81, 2020.
16

---

Published in Transactions on Machine Learning Research (1/2026)
Appendix
A Dataset Statistics
In this study, we evaluate our approach on seven widely used datasets for graph analysis: three small-scale citation datasets, Cora (Sen et al. 2008), Citeseer (Giles et al. 1998). and Pubmed (McCallum et al. 2000); three large-scale datasets, Coanthor- CS, Coauthor-Phy and Amazon-Photo (Shchur et al., 2018 ; and the OGBN-arxiv dataset (Hu et al.; 2020); a large-scale benchmark from the Open Graph Benchmark (OGB)
collection. Table 4
summarizes the detailed statistics for all datasets used in our experiments.
Table 4: General dataset statistics.
Table 5: Splitting statistics for inductive evaluation.
Dutaret
#Nodes
#Edges
# FFestures
# Classes
Label Rate
Taitase
#TTota
#0bs
Tist
Obs.
(36)
Test (%)
Ola,"Tos
2,708
1,433
Corn
1440
53.18%
7.30%
7.20
Ciseseer
3,327
4,614
3,703
6
3.6%
Cheeseer
3327
1420
200
42.6855
6.013
7.10
Pubmed
300
0.3%
Pubmed
19717
1360
200
6.90%
1.015
6.80
Ammor-Photo
7,650
118.081
745
8
2: 2.1%
Cuauthor-CS
1.00%:
Coeathor-CSS
18,333
6,805
1.6%
Coauthor: Pla
34433
1400
200
4.00%
0.58%
7.00
Conathor- Pla
34.453
247,900
8.415
5
0.3%
Amnazon Photo
7630
1460
200
19.08%
2.6155
7,30
OGBN andy
169,343
1,106,243
128
53.7%
164483
97.1396
Table reports node-level splitting statistics used in the inductive setting. We observe a notable variation in the ratio of observed and lest nodes to the lotal number of nodes. For example, Cora and Cileseer have a relatively high percentage of observed nodes (over 40%), which facilitates effective feature propagation during distillation and inference. In contrast, Pubmed and Coauthor-Phy exhibit sparse supervision (under 7% observed), making generalization more challenging. These variations in data availability directly affect the model's ability to learn transferable representations in the inductive setting.
B Hyperparameter Settings and Tuning
The model architectures were built using 2- layers, with the hidden dimension searched over the set {128, 256, 512, 1024, 2048). For optimization, the learning rate was tuned from (0.001, 0.005, 0.013 and the weight de- cay was selected from (0.0,5 x 10-1). All models were trained for a maximum of 500 epochs, utilizing an early stopping crite- rion that halts training validation accu- racy does not improve for 50 consecutive epochs. The distillation temperature т was selected from the range [0.5, 2.0), and the knowledge distillation weight À was chosen from the discrete set (0.0,0.1, 0.2, 0.5). Furthermore, dropout rates for both teacher and student models were adjusted in the set (0.0, 0.1,..... 0.1,...,0..));
Table 6: Hyperparameters for InGGraND in transductive and inductive settings, including the distillation weight (A) and influence-guided coefficients (61, 62, n, 72)-
Dataset Teacher
Transductive
Inductive
bg
7
7/2
Y
72
Cora
GCN
0.6
0.8
0.1
0.3
SAGE
0.5
0.8
0.1
1.0
0.4
0.5
1.0
0.2
1.0
0.2
GAT
0.5
0.4
0.2
1.0
0.1
0.5
0.9
0.8
1.0
0.2
Citeseer GCN
0.0
0.6
0.1
0.6
0.4
0.0
1.0
0.2
0.8
0.3
GAT
0.8
0.2
0.6
0.2
0.1
0.2
SAGE
0.1
0.6
0.1
0.4
0.4
0.0
1.0
0.2
1.0
0.2
Pubmed GCN
0.0
1.0
0.1
0.6
0.4
0.5
1.0
0.2
0.2
1.0
GAT
0.0
0.4
0.2
0.4
0.1
0.0
0.5
0.2
0.2
1.0
SAGE
0.0
0.8
0.1
0.8
0.2
0.0
0.1
0.8
0.1
Amazon GCN
0.2 1.0
0.2
0.8
0.1
0.2
1.0
0.1
1.0
0.1
GAT
0.1
0.01
SAGE
0.2
0.4
0.7
0.7
0.4
0.2
0.9
0.4
0.6
0.1
CS
GCN
0.3
1.0
0.1
0.5
0.5
0.3
1.0
0.2
1.0
0.2
GAT
0.0
1.0
0.1
0.01
0.01
0.3
0.8
0.1
0.6
0.1
SAGE
1.0
For the influence- guided weights (81,7, 62, 72), the parame- ters 6, and 71 were selected from the set {0.001, 0.1, 0.5, 0.6, 0.8, 0.9, 1.0, while 82 and 7/ were selected from {0.0001, 0.001, 0.01, 0.05, 0.1, 0.2, 0.5.0.6...... used edge dropout during distillation, tuning the drop rate from {0.85, 0.90, 0.95). The final selected values for the five core InfGraND hyperparameters (A, ò1, 82, 71, 72) are reported in Table G
Physics
GCN
0.1
0.01
0.1
0.1
0.2
0.001
0.001
GAT
0.2
0.8
0.8
0.8
0.8
0.2
1.0
0.6
0.1
0.001
SAGE
0.5
1.0
0.2
0.2
0.2
0.5
0.1
0.01
0.1
0.01
0.5, 0.6, 0.8, 1.03.
0.5,
For OGBN-Arxiv, we
C Visualization of Class Separation
17

---

Published in Transactions on Machine Learning Research (1/2026)
20 Veulizake of Lagit wsig 10N2 Scudioe 0001 Logits
Techeruopa Lopia
Studirs Sert legin
(smotres
Doo mea
(km0
Doo 11
Figure 7: t-SNE visualization of 7-dimensional logits reduced to 2D from the Cora dataset under the inductive setting, The plots compare the representation spaces of the Teacher (left), Student KD (middle), and Vanilla Student (right) models. Circles denote training samples and crosses denote test samples, with colors indicating class membership.
Table and Figure illustrate the effect of knowledge distillation the logit vec- tor of the student model. The teacher model (left plot in Figure 7 shows clear and distinct clusters with a high silhou- ette score (0.243) and large mean inter- cluster distance (80.3), which is also ap- parent in the t-SNE plot where classes are well separated. The lissledee-distilledd student (the middle plot in Figure 7 al- though exhibiting less pronounced clustering (silhouctte score of 0.002 and mcan distance of 50.9), maintains structural pattern similar to the teacher and achieves lower test cross-entropy (0.585 versus 0.636), in- dicating that it learns a more efficient and generalized representation rather than merely replicating the teacher's 's exact cluster boundaries. In contrast, the vanilla student (right plot in Figure 7 presents very poor clustering performance, as shown by its negative silhouette score (-0.021) and low mean distance (28.4), resulting in significant overlap among classes and a much higher test cross-entropy (1.389). Overall, these results suggest that knowledge distillation effectively transfers the teacher's structural knowledge to the student while promoting representation space that is more conducive to generalization.
Table 7: Model comparison metries across different evaluation criteria. Bold values indicate best performance in each metric. CH: Calinski- Harabasz, DB: Davies- Bouldin.
CE
Cluster Quality
Model
Train
Test
Silhouette
Score
Score
Teacher InfGraND
0.015
0.636
0.243
1215.0
1.78
0.160
0.585
0.002
360.1
4.39
MLP
0.137
1.389
-0).21
262.7
6.49
D Theoretical Analysis
We define the distillation loss La set of directed edges among nodes in graph, where node has source representation hę and node has target representation hy, Given set of representations (h&), we denote the softmax probability vector a (ha). Without loss of generality, set 1; the loss is then given by:
L,
DKI, o(h))) + + (EJ)EE
Σ
Σ In(Us) DxI,(o(h;)) (lo(b))) (K.))E
(10)
where 71 and f2 are scalar weights, and (y,)) ) E [0, 1] the Global Influence Score of the neighbor node By, as defined in Eq. La can be rewritten using local neighborhoods as:
L. 4- (n + Iz(T))) [(e))l DKL(o(h;) o(h))).
(11)
jEV
The KI. divergence is defined as:
18

---

Published in Transactions on Machine Learning Research (1/2026)
DKL(o(h)) || [oh))) = obb) (logo log (h))).
(12)
We denote the output of the student model by a(hf). The student model is two-layer neural network with a ReLU activation function in the first layer. We assume that the output of the first layer is positive element-wise (i.e., W |x; + bj > 0), since otherwise o(h,) would reduce to the bias of the second layer.
o(h,) =
+ +b)) + b2),
(13)
where:
E R,, WIER bi R W2 Rox E
D.1 Gradient Derivation of La
The distillation process involves training the student model to match the teacher's representation using the loss function defined in Eq. [0 The optimizer performs backpropagation based on this objective. Since the gradient signal dictates how the student model updates its parameters, analyzing this signal is essential to understanding the behavior of the proposed method.
Recalling that h is the student representation and a(h)) denotes its softmax output, the Jacobian of the softmax with respect to h is given by:
do (h diag(o(h;)) - a(h)) (h²)) Rexe.
(14)
Th
Using the chain rule, the gradients of the softmax output o(h;) with respect the model paramoters are:
(h;) -o e Rexe
(15)
ab
do (h?)
(diag(o(h:)) - (hh)) W2 e Rex/
(16)
Ub
do (o?)
(diag(o(h))) ((b))))))))))))))))) (Wix + bb) e Rexs,
(17)
dW2
do @o!h))
Wo] 8x E Rex
(18)
OW
While Eqs. 15 18 describe how the softmax output depends on the model parameters, the actual learning signal during distillation originates from the loss function. To analyze how this signal propagates, we need to derive the gradient of La with respect to the model parameters. We first differentiate La with respect to the softmax output, then apply the chain rule to obtain parameter gradients.
VL-= Σ Σ
(n + + -22-I((0))) iEV JEN(U.)
W(x)) Dka(((h;)) (h))),
(19)
which can be further written as:
(71 + Iz(rz))
VL, Σ Σ
(Vo(h?))T [logo(h?) - log or (h²) +1]
(20)
[W())]
NEV (EV(.)
19

---

Published in Transactions on Machine Learning Research (1/2026)
Incorporating Eqs.
15-18 18 into the formulation of Eq.
20 obtain:
71 + 12 92 (I,z),
bb Lu E R =
= Σ
(21)
[N(w))]
iEV JEN((.)
(diag(((b))) - obf))o(()))
logo((h)) - logorlh) -aoh)))-oooobbh
(n1 (2+ Vb,CEE''= E Σ
(22)
WV(vi)l
iEV JEN(s))
(dding(((h))) (hh))o(h)) (W²)
[logo(h;) log (h/) +1],
Σ Σ ' 12 (1,)))(()))
Vw,Ca E Rexs iEV jEN(w) [N(v)]
(23)
(ddiag(o(h))) ohh))o(h;) (Wxxx b)))
log logoo)))) +
Σ 12720 (,,y)
Vw Lr
R/xd
(24)
[V(x:]
ieV (EV(
([(diag(o(hh)) - (hh))o 2
(oggo()))looo((()))) logo
These expressions describe how the distillation loss La backpropagates gradients to the model parameters.
D.2
Effect of I(j)
In Eq. 20 71 and 72 are scalar coefficients in (0, 1], and (v,))) is the GIS of the neighbor node vy. This gradient expression reveals two key multiplicative components:
The term (71 + 72 Ig (vy)) serves as a scalar weight that modulates the contribution of each neighbor to the gradient. A higher (1;) increases the influence of that neighbor the gradient update.
directional term (log((h;) log or (h²) + 1]. which captures the distributional divergence between the predictions of the student and the teacher.
This decomposition shows that (1,) acts as an amplifier, strengthening the gradient signal and helping the student better identify and correct its mistakes. For example, consider node i and j Assuming "1 7 1, the gradient simplifies to:
VCJ (1+ Ig (Vo(hi)) [logo(hi) - log (ht)) [(11]
(25)
20

---

Published in Transactions on Machine Learning Research (1/2026)
The magnitude of the equation in 25 is given by:
VCJ
1 + [,(22) [W(1]
(Vo(h))) T [log log (ba) + +1]
(26)
In Eq. 26 (b2) E [0, 1] scales the gradient magnitude of the distillation loss, assigning greater weight more influential neighbors the update. This mechanism encourages the student model align more closely with informative neighbors during distillation.
321
Zdra
TC
21
21
Vestor Legend
923
Figure Effect of influence score (e;);) on gradient magnitude KD. The diagrams show how increasing ((02) E (0.0, 0.3, 0.6, 1.0) scales the magnitude of the gradient vector. Higher scores amplify the correction signal for semantically important neighbors.
As an example to illustrate Equation 26 consider a scenario with two classes, where the dimension of the logits 2. In this case, the student and teacher outputs can be represented as 2D vectors. As shown Figure [ the blue and light blue vectors correspond the logits of nodes and 2, respectively. The red vector labeled "VC)"" is approximatcly the gradient signal, which guides the representation of node 1 to align with that of node 2. Starting from the leftmost subfigure, we observe how the influence score I, (12) E [0, scales the magnitude of the gradient of the distillation loss. As (12) increases across the subfigures (e.g.. 0.3, 0.6, 1.0), the magnitude of the gradient vector increases proportionally, amplifying the gradient update. This effect encourages the model to focus on aligning with more informative neighbors during distillation. The same scaling behavior generalizes to Equations 21
22
23 and 24
D.3 The Necessity of 7/1
The design of the distillation loss encourages the student model to learn more from high-influence nodes. However, low-influence nodes can also provide valuable knowledge to the student. If ? were omitted, the gradients from low-influence nodes would be suppressed during distillation, as (v;)) would shrink the gradient signal entirely for those nodes.
For example, in the gradient expression of Eq. 20 when Ig(Ty) 0, the update reduces to:
VC VC (I,))=)) = = Σ -=(=(())==)>>>>> REV JEN(() W((e)) (Vo(h;)) [og (h;) - log o (hf) + 11,
(27)
which remains a meaningful gradient signal aligned with the teacher prediction o(ht). Thus, 71 plays a foundational role preserving knowledge transfer from all neighbors, while 72 (v;) provides additional fine-grained emphasis based on learned importance.
E
Influence Score vs. Degree/PageRant
Here, we conducted a focused ablation study comparing our influence metric against normalized Degree and PageRank centrality measures. To isolate the impact of the weighting metric itself, we disabled the feature propagation module (Section 4.2 and used only the influence-guided loss component. of our framework. We
21

---

Published in Transactions on Machine Learning Research (1/2026)
then substituted our infiuence score Iz();) with normalized Degree and PageRank scores within Eq. i and
This controlled comparison isolates the node importance metric as the sole variable. We independently tuned the core hyperparameters (71, 72, 81, 62), using validation accuracy as the optimization criterion. All experiments were conducted in the transductive setting across three teacher architectures (GCN, GAT, GraphSAGE) and three benchmark datasets (Cora, Citeseer, Pubmed).
Table presents the comparative results across all teacher-dataset configurations. Our proposed influence metric consis- tently outperforms Degree and PageR- ank (Bojchevski et al. (2019) in all nine experimental settings, achieving average improvements of +2.3% and +2.7%, re- spectively. This performance advantage arises from a fundamental difference in what is being measured. Degree and PageRank are purely link-analysis algo- rithms that only consider how nodes are connected, completely ignoring node fea- tures. In contrast, our influence metric considers both node features and their connections the graph. This dual consideration of features and topology makes fundamentally better suited for guiding knowledge distillation in GNNs, where both aspects are critical.
Table 8: Comparison of node importance metrics for knowledge distil- lation guidance (transductive setting). Bold values indicate the best performance.
Toncher
Dataset
KD w/Tnfluence
KD w/Degree
KD w w/PageRank
Corn
83.4 + 0.8
81.1 + 0.9
80.8 + 0.7
GCN
Citeseer
74.6 0.6
71.6
70.7 0.8
Pubieed
81.1 1: 0.4
79.7
79.5 0.2
Corn
84.0 1: 0.0
81.6 0.7
79.7 1 0.9
GAT
Citesser
72.2
60.3
69.0
Pubined
80.6 + 0.4
79.0 1:01:00 0.6
79.8 0.4
Cora
84.4 = 0.0
81.5 2: 1.3
81.1 1 0.8
SAGE
Citeseer
72.7 =
70.2 2.0
69.5 2.5
Pubmed
80.3 = 0.4
78.0 1.7
79.9 0.3
F Influence Computation: Complexity and Scalability Analysis
Our influence computation method has two implementations with distinct computational characteristics. The naive all-pairs approach computes cosine similarity between all node pairs, requiring O(N . D) time complexity and O(N²) space complexity, where N is the number of nodes and D is the feature dimension. Although this provides the most complete influence profile, it becomes infeasible for large graphs, OGBN- Arxiv. The k-hop neighborhood approximation restricts influence computation to nodes within hops, requiring O(EE&| D) time and O((Exl) space, where |Eg] denotes edges within k-hop neighborhoods. For sparse graphs with k 2, this approximately O(N. d² D) where d is the average degree. Since |Fx] < N2 for sparse graphs, this significantly reduces memory usage. The k-hop approximation is principled for large- scale graphs. In large-scale graphs, node's influence decays with distance. For nodes many hops away, the influence score approaches zero. Therefore, the k-hop approximation captures meaningful local influence while discarding nogligible distant interactions. Both implementations use batched GPU computation for memory efficiency.
We conducted timing experiments across seven benchmark datascts using GPU- batched implementation on single NVIDIA GPU (32GB VRAM). Table g presents the results. For the six small to medium datasets (Cora, Citeseer, Pubmed, Amazon-Photo, Coauthor-CS, Coauthor-Phy), we use the naive all- all-pairs approach as it is computationally afford- able and provides exact influence scores without approximation. For OGBN-Arxiv (169K nodes, 2.5M edges), we use the 2-hop approximation, which completes in 70 seconds (1.2 minutes) with 0.41 milliseconds per node. The naive approach for OGBN-Arxiv is infeasible due to memory constraints. These results demonstrate that our method scales efficiently from 2.7K to 169K nodes.
Table 9: Influence computation runtime (GPU-batched, single 32GB GPU). Times reported in seconds. *Naive approach for OGBN-Arxiv requires 1114 GB memory (infeasible).
Dataset
Nodes
Edges
Naive (s)
1-hop (s)
2-hop (s)
3-hop (s)
Corn
2,708
13,264
5.68
0.95
1.18
2.34
Citeseer
3,327
12.491
0.13
1.29
1.60
2.11
Pabmed
19,717
108.365
5,25
9.69
13.48
34.06
Amazon-Pheto
7,850
245,812
0.89
3.75
22.93
202.60 00
Cooutluee CS
18,333
182,121
5.39
9.42
14.70
63.28
Coauthoe- Phy
34,493
630,417
21.95
24.16
47.82
354.43
OGDN-Arxte
100,348
2,501,829
N:A*
49.78
70.13
498.46
Influence computation is a one-time preprocessing step during training, not a recurring cost during infer- ence. Once computed, influence scores used weight. the distillation loss but do not affect the student
22

---

Published in Transactions on Machine Learning Research (1/2026)
MLP's deployment. Therefore, even if computation takes minutes, it does not impact deployment latency or real-time prediction performance. Our influence computation method is computationally efficient (under 2 minutes for 169K nodes) and practical (one-time preprocessing with zero inference overhead).
G InfGraND Algorithm
Algorithm 1 presents the complete InfGraND training procedure, which consists of three main phases: (1) training the teacher GNN on the graph structure, (2) generating teacher predictions for all nodes, and (3) training the student MLP using the inftuence-guice
distillation approach. The algorithm takes as input
the graph 9 = (V. & x), pro-computed propagated features X (as described in Scction 4.2 via Eq. 6 and Global Influence Scores (T4))))v (computed via Eq. Section 4.1p. The student MLP trained using the supervised loss (Fq. 7 and distillation loss (Eq. weighted by the hyperparameter À specified in Eq.
A Algorithm In/GraND: Influence-Guidler GNN-to-MLPP Knowledge Distillation
Input: Graph 9 (V,E,X), X), labels Yiu for Viab: propagated features X, influence scores (I, (w:))ee
Input: Hyperparamcters: X, Vi, Y2, 72,01,00 01,02,77 Emaxs
Output: Trained student MLP f'student
4: Train teacher GNN fteacher on G until convergence; freeze parameters
5: Compute teacher logits: h < fteacher (G, U;) for all is E Initialize student MLP Sstudent with parameters for epoch e to Fmaz do Compute student logits: h; fatudden((:)) for all Uj E V L. 51 Eueviss (o V/ 82 Eneii Ig(v)) Dce((o(h)),))
Eq.
10:
La Eiev EiEN(ng (7i + 72 (0,)) WeJl (RI(l(()/T)) o(h)/T))
Eq.
11:
+ +(1 (1- A)
Eq.
12:
O 0 Vell
13: end for
14: return Sstudent.
H Sensitivity Analysis of Influence Computation Depth
To evaluate the sensitivity of InfGraND to the influence computation depth (Sec- tion 4.1,, we conduct abla- tion experiments varying k from to Pubmed datasets using GCN and GraphSAGE teachers. As shown in Figure g perfor- mance remains relatively sta- ble across different & values, with k achieving the best near-best. results in most cases. The performance variation across & [1,4] is less than 1% for all configurations, confirming that our method is robust to this hyperparameter choice. These results validate design decision to use k = 2 (Section 4.1
Pubmed
Teat Accumccy
Cora and
8
Figure 9: Ablation study on influence computation depth (k-hop) for Cora and Pubmed datasets using GCN and GraphSAGE teachers. Shaded regions represent standard deviation across multiple runs.
23


Reviews:
[{"review_id": "r5H24tGhcX", "reviewer": "Reviewer_gJYB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary of Contributions**\n\nThis paper presents InfGraND, a novel framework for graph-aware knowledge distillation from Graph Neural Networks (GNNs) to Multi-Layer Perceptrons (MLPs). The core idea is to enhance the efficiency of MLP-based graph learning while retaining the representational strength of GNNs.\n\nThe authors introduce a node influence–guided distillation mechanism, which prioritizes structurally important nodes based on their topological impact within the graph rather than prediction uncertainty. This “influence score” captures how perturbations to one node affect others, leading to a more informed and stable distillation process.\n\nIn addition, InfGraND integrates a one-time precomputation of multi-hop neighborhood features, enabling the MLP student to leverage graph structure without runtime message passing. This design preserves low inference latency while embedding structural awareness.\n\nThe framework is evaluated comprehensively on seven benchmark datasets (including Cora, Citeseer, Pubmed, Amazon-Photo, Coauthor-CS/Phy, and OGBN-Arxiv) under both transductive and inductive settings, using standard GNN architectures (GCN, GAT, GraphSAGE) as teachers. Experimental results show that InfGraND consistently outperforms prior GNN-to-MLP KD methods and, in several cases, even surpasses the GNN teacher’s accuracy. Additional ablation and label-scarce experiments support the robustness of the approach.\n\n**Key Strengths**\n\nNovel contribution: Introduces a graph-structure–aware node influence metric for knowledge distillation — a conceptually sound and underexplored direction.\n\nBalanced design: Maintains MLP’s low inference cost while meaningfully embedding structural knowledge.\n\nComprehensive evaluation: Covers multiple datasets, architectures, and training settings with clear experimental comparisons.\n\nClarity of motivation: The paper articulates the trade-off between GNN expressiveness and MLP efficiency convincingly.\n\n**Key Weaknesses**\n\nLimited theoretical analysis: The influence metric, though intuitive, lacks rigorous theoretical grounding (e.g., formal properties or complexity discussion).\n\nPotential scalability concerns: Influence computation, even if parameter-free, may be costly for large-scale graphs; practical runtime analysis is not deeply explored.\n\nComparative novelty: Some components (feature precomputation, subgraph-level distillation) build upon prior ideas, so the degree of methodological novelty depends heavily on the influence-guided weighting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "**Overall assessment**:\nYes. The submission provides strong, well-structured empirical evidence supporting its major claims. The experiments are comprehensive, reproducible, and logically aligned with the stated goals. Evidence is clear and convincing across multiple datasets and evaluation settings.\n\n**Claim 1 – The proposed influence metric identifies structurally important nodes.**\n\nEvidence:\n\nQ1 directly compares GNNs trained on high- vs. low-influence nodes.\n\nHigh-influence subsets consistently yield higher accuracy across all datasets.\nReviewer analysis:\n✓ Clear empirical validation; convincingly demonstrates that the influence score captures structural importance.\n\n**Claim 2 – Influence-guided distillation improves MLP performance and can surpass GNN teachers.**\n\nEvidence:\n\nQ2 shows InfGraND consistently outperforms baselines (GLNN, KRD, HGMD, FF-G2M).\n\nDistilled MLPs exceed teacher accuracy on several datasets.\nReviewer analysis:\n✓ Strongly supported; results are comprehensive and cross multiple architectures and benchmarks.\n\n**Claim 3 – InfGraND offers a superior accuracy–efficiency trade-off.**\n\nEvidence:\n\nQ3 reports 6.8×–13.9× faster inference with higher or comparable accuracy.\n\nEvaluation uses consistent architecture and hyperparameter settings for fairness.\nReviewer analysis:\n✓ Persuasive demonstration of low-latency advantage; efficiency claim is fully substantiated.\n\n**Claim 4 – Both core components (influence weighting and feature propagation) contribute complementary gains.**\n\nEvidence:\n\nQ4 ablation study isolates each module: both yield improvements independently and achieve the best performance when combined.\nReviewer analysis:\n✓ Well-designed ablation confirms contribution of each component; evidence is quantitative and coherent.\n\n**Claim 5 – InfGraND remains robust under label scarcity.**\n\nEvidence:\n\nQ5 shows consistent 4% average gains over GLNN when only 2–8 labels per class are available.\nReviewer analysis:\n✓ Strong indication of robustness; experiments are realistic and address an important real-world scenario.\n\n**Claim 6 – InfGraND is stable across hyperparameter settings.**\n\nEvidence:\n\nQ6 hyperparameter analysis shows smooth trends for λ, γ₂, δ₂, and P; no abrupt degradation observed.\n\nOptimal λ ≈ 0.1 and P = 2 generalize well.\nReviewer analysis:\n✓ Supports robustness claim; results are interpretable and not overly sensitive to tuning.\n\n**Final Reviewer Judgment**\n\nAll major claims are clearly supported by accurate and consistent experimental evidence.\nThe evaluation suite is broad, methodologically sound, and internally coherent.\nMinor extensions—such as results on more diverse graph domains or runtime profiling of influence computation—could further strengthen the paper, but the presented evidence is convincing and sufficient to substantiate the authors’ conclusions.", "audience_yn": "Yes", "audience_explanation": "this GNN related work is a mainstream research in machine learning", "requested_changes": "**Proposed Adjustments: Critical (Essential for Acceptance)**\n\n1. **Clarify computational cost of influence estimation.**\n\nProvide explicit time/space complexity or runtime comparison for computing node influence, especially on large graphs such as OGBN-Arxiv. \n\nFrom Section 4.1 (Node Influence Measurement):\nThe authors define the influence metric formally and then approximate it using cosine similarity between k-hop propagated features.\nThe authors say this approach is “efficient” and “parameter-free,” but no complexity expression or runtime profile is provided.\nThere is no mention of how long the influence computation takes on large graphs like OGBN-Arxiv, nor whether it scales linearly or quadratically in |V| or |E|.\n\nIn Section 5 (Experiments), no table or figure reports “influence computation time,” only inference time comparisons.\n\nThis is crucial to fully justify the “efficiency” claim and to ensure scalability is practical in real-world deployments.\n\n2. **Add more details on the influence approximation procedure.**\n\nClarify exactly how cosine similarity–based influence is computed (e.g., hop size, normalization, batch implementation).\n\na. Exact hop count (k) used in cosine similarity is missing. The text mentions “for example, k = 2,” but it’s unclear whether this is fixed, tuned, or dataset-dependent.\n\nb. For the normalization scheme, “MinMaxScaler” is mentioned, but over what scope? (per node, per graph, or per batch?) This affects reproducibility and scaling.\n\nc. Batch computation / complexity: How is the cosine similarity matrix computed in practice — all-pairs, sampled neighbors, or matrix multiplication shortcut? The naive O(N²) version is infeasible for large graphs.\n\nd. Handling of self-loops / adjacency normalization: It mentions “normalized adjacency” but not whether self-loops are added or what normalization scheme is used (row-, sym-, or degree-based).\n\ne. Implementation pseudocode: No pseudocode or algorithm listing is given; only high-level equations. For a reproducibility-focused reader, these are helpful but not sufficient for exact replication.\n\n\n3. **Improve clarity and conciseness of exposition.**\n\nThe introduction and related-work sections are overly dense and could better highlight the conceptual novelty (influence-guided vs. entropy-guided discrimination).\n\n\nStreamlining these sections would enhance readability and strengthen the positioning of contributions.\n\n\n**Non-Critical (Would Strengthen the Work)**\n\n1. Extend evaluation to more graph domains.\n\nInclude at least one heterogeneous or dynamic graph dataset, or discuss limitations in generalizing beyond citation/co-author graphs.\n\nThis would broaden empirical validity.\n\n2. Provide deeper theoretical insight.\n\nA short analysis connecting the influence metric to established graph centrality or sensitivity measures would help ground the intuition mathematically.\n\n3. Ablation on influence computation depth.\n\nCurrently, k-hop choice (k=2) is fixed; reporting how performance changes for k=1–4 in influence calculation would clarify stability.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xw2v1ohMsf", "reviewer": "Reviewer_gGmb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new influence-guided distillation approach to distill from GNN to MLP for efficient deployment of the models. Specifically, the approach computes an importance weight for each node in the graph by aggregating its influence for other nodes (through gradient estimation), and then directly multiplies this weight to the distillation loss for each node. The authors have conducted fairly comprehensive experiments across different models and graph datasets to illustrate the effectiveness.\n\n### Strengths\n\n1. The proposed approach is well-motivated and novel\n2. The experiments in this work are very comprehensive, covering multiple graph datasets and GNNs, and compare with many baselines.  \n3. The empirical results are good\n\n### Weaknesses\n\n1. The loss functions from Eq 7 to 9 are a bit complicated and involve several hyper parameters. This brings extra difficulty for application of the proposed approach to tune multiple hyperparameters. I do have a question on these: are results in Table 1 all from one set of hyperparameters? Or each configuration tunes the hyper parameters on their own?", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission presents comprehensive evidence and ablation studies to support the claims.", "audience_yn": "Yes", "audience_explanation": "This paper is about a new approach of distilling MLP from GNNs which is common practice, thus this work should be interesting for some TMLR audience.", "requested_changes": "As mentioned in the weaknesses, I suggest the authors to clarify more on the hyperparameter tuning process and explicitly state the final used set of hyperparameters values in the main experiments to obtain the results.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2uRlovcIFU", "reviewer": "Reviewer_JHV2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Strengths\n\nIn general, this paper:\n\n1. Keeps inference efficient by distilling to MLPs with preprocessed features, practical for deployment.\n\n2. Includes good ablations and sensitivity analyses, showing thoughtful evaluation design.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I believe this paper demonstrates consistent accuracy improvements, in some cases surpassing teacher GNNs.\nAnd overall, the writing and paper structure are clear, with solid experimental coverage.", "audience_yn": "Yes", "audience_explanation": "This paper introduces a new angle by weighting knowledge transfer from GNN to MLP with node influence instead of just teacher uncertainty. Thus, there will be TMLR audiences interested in this paper.", "requested_changes": "# Major Weaknesses\n\n1. This work is dependent on homophily, not tested on heterophilous graphs. The method assumes neighbors share labels, so results may not generalize.\n\n2. Heuristic influence metric. The influence is approximated via simplified propagation, which may not align with how GNNs actually weigh nodes. No direct validation.\n\n3. Missing relevant baselines. Key structure-aware MLP methods like SA-MLP are absent, weakening comparative claims.\n\n4. No handling of teacher errors. Weighting ignores teacher confidence, so mistakes may be reinforced.\n\n\n\n# Minor Weaknesses\n\n1. Incremental contribution: This work mainly integrates existing ideas (influence, propagation, KD) rather than introducing a fundamentally new technique.\n\n2. Narrow task scope: Only node classification is evaluated, leaving generalizability to other tasks unclear.\n\n3. No ablation against simpler metrics: Unclear if degree/PageRank would perform similarly.\n\n4. Some clarity issues: Definitions are dense, and contributions could be presented more cleanly.", "broader_impact_concerns": "Broader Impact Concerns\n\n1. May bias performance toward well-connected nodes, leaving sparse nodes underrepresented.\n\n2. Could be vulnerable to adversarial graph manipulations that inflate node influence.\n\n3. No major ethical or misuse risks beyond typical concerns in graph learning.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)