TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation
Authors: Ziang Zhou
Zhihao Ding
Jieming Shi
Li Qing
Shiqi Shen
Venue: ICML 2025 poster (2025)
Keywords: GNN2MLP distillation
knowledge distillation
Primary area: general_machine_learning->representation_learning
TL;DR: We develop a novel method, comprising teacher injection with fine-tuning and Dirichlet energy distillation, to distill layer-level GNN knowledge into MLPs, to accelerate model inference.

Abstract: Graph Neural Networks (GNNs) are pivotal in graph-based learning, particularly excelling in node classification. However, their scalability is hindered by the need for multi-hop data during inference, limiting their application in latency-sensitive scenarios. Recent efforts to distill GNNs into multi-layer perceptrons (MLPs) for faster inference often underutilize the layer-level insights of GNNs. In this paper, we present TINED, a novel approach that distills GNNs to MLPs on a layer-by-layer basis using Teacher Injection and Dirichlet Energy Distillation techniques.
We focus on two key operations in GNN layers: feature transformation (FT) and graph propagation (GP). We recognize that FT is computationally equivalent to a fully-connected (FC) layer in MLPs. Thus, we propose directly transferring teacher parameters from an FT in a GNN to an FC layer in the student MLP, enhanced by fine-tuning. In TINED, the FC layers in an MLP replicate the sequence of FTs and GPs in the GNN. We also establish a theoretical bound for GP approximation.
Furthermore, we note that FT and GP operations in GNN layers often exhibit opposing smoothing effects: GP is aggressive, while FT is conservative. Using Dirichlet energy, we develop a DE ratio to measure these effects and propose Dirichlet Energy Distillation to convey these characteristics from GNN layers to MLP layers. Extensive experiments show that TINED outperforms GNNs and leading distillation methods across various settings and seven datasets. Source code are available at https://github.com/scottjiao/TINED_ICML25/.

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
Ziang Zhou'
Zhihao Ding'
Jieming Shi
Qing Li
Shiqi Shen
Abstract
1. Introduction
Graph neural networks (GNNs) have delivered impressive outcomes in important applications (Hamilton al., 2017; Kipf and Welling, 2017; Veličković al., 2018). The power of GNNs is underpinned by the message passing framework that assimilates and refines node representations by con- sidering their (multi-hop) neighborhood over graphs (Wu et al., 2020; Zhou et al., 2020). Nonetheless, the message passing is computationally demanding due to numerous nodes involved, posing significant challenges to deploy GNNs in latency-sensitive applications that require fast in- ference (Zhang al., 2020; 2022a; Jía al., 2020).
Graph Neural Networks (GNNs) are pivotal in graph-based learning, particularly excelling in node classification. However, their scalability is hindered by the need for multi-hop data during inference, limiting their application in latency- sensitive scenarios. Recent efforts to distill GNNs into multi-layer perceptrons (MLPs) for faster in- ference often underutilize the layer-level insights of GNNs. In this paper, present TINED, a novel approach that distills GNNs to MIPs on a by-yer basis using Teacher Injection and Dirichlet Energy Distillation techniques. We fo- cus on two key operations in GNN layers: feature transformation (FT) and graph propagation (GP). We recognize that FT is computationally equiv- alent to fully-connected (FC) layer in MLPs. Thus, propose directly transferring teacher pa- rameters from an FT in GNN to an FC layer in the student MIP, enhanced by fine-tuning. In TINED, the FC layers in an MLP replicate the sequence of FTs and GPs in the GNN. We also establish a theoretical bound for GP approxima- tion. Furthermore,
Recent studies attempt to combine the performance advan- tage of GNNs and the latency advantage of multi-layer per- ceptrons (MLPs) (Hu et al., 2021; Zhang et al., 2022b; Tian et al., 2022; Chen al., 2021). ). Specifically, GLNNN (Zhang et al., 2022b) distills teacher GNNs into student MLPs via soft labels. Then the student is deployed for fast inference to approximate the performance of GNNs without expen- sive message passing on graphs. NOSMOG (Tian et al., 2022) further considers graph structures, robustness, and node relations, while a new graph representation space is Iearned in (Yang et al., 2024). Current research often treats both teacher GNN and student MLP as monolithic model entities, focusing primarily on soft label outputs of the GNN for distillation, which overlooks the intrinsic knowledge contained within fine-grained GNN layers.
note that FT and GP op- erations in GNN layers often exhibit opposing smoothing effects: GP is aggressive, while FT is conservative. Using Dirichlet energy, we de- velop a DE ratio to measure these effects and propose Dirichlet Energy Distillation to convey these characteristics from GNN layers to MLP lay- ers. Extensive experiments show that TINED out- performs GNNs and leading distillation methods across various settings and seven datasets. Source code are available at https: htt///gttttae hiighhu.... tum. scot t jiao TINEDINNEDDDIIEEE ICML25/.
In this paper, we conduct an in-depth analysis to reveal important properties of the key operations in GNN layers and propose a novel method, TINED, distilling GNNs to MLPs layer-wise by Teacher INjection and Dirichlet Energy Distillation. The main ideas are explained below, and the detailed architecture of TINED is presented later.
Main Idea of Teacher Injection. A typical GNN layer in- cludes two key operations: feature transformation (FT) and graph propagation (GP) (Kipf and Welling, 2017; Veličković et al., 2018; Hamilton et al., 2017: Liu et al., 2020; Zhu et al., 2021). GP aggregates neighbor representations of a node, while FT transforms node representations with learnable parameters. The valuable knowledge of teacher GNN is preserved in the well-trained parameters of its FT and GP operations. Existing studies do not directly utilize these pa- rameters (Zhang et al., 2022b). In Section 4.1, we recognize
'Department of Computing, The Hong Kong Polytechnic Uni- versity, Hong Kong SAR, China -Wechat, Tencent, Beijing, China. Correspondence to: 'Jieming Shi <jieming whiiggssshhiii polyu.edu.hk:
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation
GP
DE Ratio
Ratio
Ratio
Ratio
DB Ratio
2
6
6-
1-
3-
3
3-
0
8:
layer layer2 2
layer layer 2 (b) Citeseer
layer layer 2 (c) Pubmed
layer layer (d) A-computer
layer 1 layer 2 (e) A-photo
(a) Cora
Figure 1: The DE ratios of FTs and GPs in the layers of GraphSAGE.
that FT operations in GNNs share the same formulation as fillly-coneccted (FC) layers in MLPs, both transforming
We propose TINED, a novel method to effectively distill fine- ane-grained layer-wise knowledge from teacher GNNs into student ML.ss.
representations using learnable transformation matrices and
activations. Thus, we propose to inject the parameters of an FT in a teacher GNN layer into an FC layer in student MLP, followed by fine-tuning for distillation. The GP operation in the GNN layer is emulated by another FC layer in the student. This allows TINED directly transplant teacher knowledge into the student. Moreover, in TINED, the FC layers in the MIP mirror the order of their corresponding FTs and GPs in the GNN teacher, preserving layer-specific knowledge as much as possible. Theoretically, we prove an approximation bound between a GP in GNN and its corre- sponding FC layer in MLP, which depends on the eigenvalue of graph Laplacian matrix.
We develop teacher injection technique transplant the parameters of key operations from GNNs to MLPs. We provide a theoretical approximation analysis.
We observe distinct smoothing effects of FT and GP op- erations in GNN layers and introduce Dirichlet energy distillation to impart these effects to student MLPs.
Extensive experiments demonstrate that TINED achieves superior performance with various GNN teachers on widely-adopted benchmark datasets.
2. Related Work
Main Idea of Dirichlet Energy Distillation. We further investigate the smoothing properties of FT and GP opera- tions within GNN layers, since it is well-recognized that appropriate smoothing is crucial for GNNs (Chen et al., 2020a). Importantly, we observe that the FT and GP op- erations GNN layer often exert opposing smoothing effects: GP aggressively smooths node embeddings, while FT is more restrained and can even diversify embeddings. Using Dirichlet energy (Rusch et al., 2023), we propose DE ratio measure to quantify whether an operation is con- servative (large DE ratio) or aggressive (small DE ratio) in smoothing. Figure shows the DE ratios of FTs and GPs in a trained 2-layer GNN teacher GraphSAGE on the experimental datasets. Within the same layer, the DE ratio of FT (red dot) is often larger than that of GP (blue star), indicating their opposing smoothing behaviors. Similar ob- servations are made on other GNNs (see Appendix A.4). To distill these smoothing patterns from GNN to MLP layers, we design Dirichlet Energy Distillation in Section 4.2.
GNNs leverage message-passing aggregate neighbor- hood information for learning (Kipf and Welling, 2017; Veličković et al., 2018; Hamilton et al., 2017; Chen et al., 2020b; Klicpera et al., 2019). GCN (Kipf and Welling, 2017) introduces layer-wise propagation, GAT (Veličković et al., 2018) employs attention mecha- nisms, and APPNP (Klicpera al., 2019) uses personalized PageRank. Chen et al. (2020b) mitigate over-smoothing with residual connections. Extensions include positional en- coding (You et al., 2019; Wang et al., 2022; Li et al., 2020), few-shot learning (Zhou et al., 2023b; Dong et al., 2025a), and anomaly detection (Dong et al., 2025b; 2024; Ding et al., 2024). Specialized GNNs, such as heterogencous GNNs (Zhou et al., 2023a; Zhang et al., 2019), address specific graph types. Despite their effectiveness, GNN infer- ence is computationally expensive.
Knowledge Distillation has been applied in GNNs to ac- celcrate inference while preserving effectiveness (Lee and Song, 2019; Yang et al., 2020; 2021; Tian et al., 2023; Feng et al., 2022). Previous studies have trained smaller student GNNs with fewer parameters than the large teacher GNNs, such as LSP (Yang et al., 2020), FreeKD (Feng et al., 2022) and TinyGNN (Yan et al., 2020) which still rely time- intensive message passing. GFKD (Deng and Zhang, 2021) performs distillation at graph-level instances, while we tar- get nodes within a single graph. Recent studies (Tian et al., 2022; Yang et al., 2024: Zhang et al., 2022b; Hu et al., 2021; Zheng et al., 2022; Wu et al., 2023a;b) develop Ml.ed
We conduct extensive experiments on benchmark datasets under various settings. Results show that TINED achieves superior performance and fast inference speed, compared with existing methods and various teacher GNNs. For exam- ple, on the Citeseer data, TINED improves a GNN teacher by 3.94%, MLPs by 15.93% GLNN by up to 3.21 %, and NOSMOG by 1.54%. TINED is 94x faster than its GNN teacher for inference. Our contributions are follows:
2

---

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
(a) TINED architecture
GNN Teacher
(b) Deployment
I-th layer
Without Graph Dependeacy
GP
FT
Soft Labels
Tescher
TINED
Graph
Teacher
New node with node Reatures
Injection
DE Ratio
DE Ratio
Dirichlet Energy Distillation LED
cuthi Red
reen, Ryctz
Fine-tune
With Gruph Dependency
+
W
Sož by
Studeat MLP
ku
TINED+
+
Node features
ICE Ground truth: yo
FCL
FCL2
New node with node features and graph features
ML.P Student
Figure (a) TINED with Teacher Injection and Dirichlet Energy Distillation; (b) Inference settings
student models without message passing. GLNN (Zhang et al., 2022b) trains an MLP student using node features as input and soft labels from a GNN teacher targets. NOSMOG (Tian et al., 2022) incorporates graph structure, adversarial feature augmentation, and node similarity rela- tions into distillation. VQGraph learns a structure-aware tokenizer to encode node substructures (Yang et al., 2024), during which teacher is also trained. As mentioned, the intrinsic knowledge in well-trained GNN layers is insuffi- ciently utilized by existing studies.
GNNs, Let H E R be the output node embedding matrix of the 1-th GNN layer, with each row h being the representation of node in V. Most GNNs fit under the message- framework with feature transformation (FT) and graph propagation (GP) operations (Zhu et al.. 2021). The I-th layer usually comprises operations FT(() and GP(() to get H(!) in Eq. (1). GP(!) aggregates the ()- layer representations H over G, and FT()
transforms representations to get H (4), Different GNNs may vary in how they composite (T and GP(()
There exist works in other orthogonal domains, also observ- ing the benefits of distilling layer-wise teacher structures, such as language models (Liang et al., 2023), natural lan- guage processing (Liang et al., 2023; Chang et al., 2022; Jiao et al., 2020) and computer vision (Li et al., 2019; Hinton al., 2015). In this work, we fill the gap to develop layer- wise distillation method for GNNs. In addition. Han al. (2023) work a different problem that uses pre-trained MLPs for GNN training acceleration.
H ( (T
(P (H(-))9) (9)))
(I)
MLPs. An MLP is composed of multiple fully coly-cconeecte (FC) layers. The I-th FC layer converts an embedding (!-1) to h(") via transformation matrix w M in Eq. (2).
h (0 =(h ohhh (1- wi!) + bb"),
(2)
where is an activation and b(9) is bias
GNNs-to-MLLP Distillation. Given a pre-trained GNN teacher, the goal is to train cost-effective student MLP to predict ÿ. for node v, utilizing ground- ground-truth labels ye of labeled nodes E VL and soft labels 2, produced by teacher GNN for all v E V. The training objective is formulated below (Zhang et al., 2022b). Note that the student is trained by both ground truth and soft labels from the teacher.
3. Preliminaries
Notations.
graph = ==V, X) consists of a node set V/ with nodes, i.e., [V| = n, an edge set E of size [&] m, and X Raxde for de-dimensional features of all nodes. Let A be the adjacency matrix of G, where Au,, = if edge (u, v) E &, and O otherwise. A node v has a neighbor set N(v) = {ul(u, v) E } and the degree of node v is [W(x).. Degree matrix is a diagonal matrix with D. = (v)) In node classification task, the prediction targets are Y E R where K is the number of classes, and row y, a K-dim one-hot vector indicating the ground-truth class of node A model predicts the class probabilities of node u. In G, we use superscript to mark the labeled nodes (i.e., V", ' and Y and superscript to mark the unlabeled nodes (i.e., D _ and YU).
L E Le y.) + A Lx 2.),
(3)
where LCE is the cross-entropy loss by comparing the stu- dent MLP predictions y, with the ground truth y.. and Lx is the KL-divergence loss between the student MLP predic- tions ÿ. and the soft labels z, from teacher, and weight À balances the two losses.
4. The TINED Method
In Eq. (3), existing studies primarily use GNN soft labels and ground truth labels for distillation. Our approach TINED
3

---

TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation
expands this to include model parameters and layer-specific properties of teacher GNNs for distilling knowledge into MLPs, illustrated in Figure 2. As shown in Figure 2(a), for the Teacher Injection technique, we identify that the FT() operation of the 1-th GNN layer essentially conducts the same computation as an FC layer in an MLP, and thus, we inject the parameters of FT into an FC layer FC of the student MLP. The injected parameters are fine-tuned during the distillation to ensure controlled adaption. To emulate the GP()) operation in the I-th GNN layer, we employ another FC layer FC in the student MI.. We establish a theoretical approximation bound between Gp(1) and FC Then as in Figure 2(a), we use Dirichlet energy to develop DE ratios Rapay and RpTw to quantify the smoothing effects of GP and FT in the 1-th GNN layer, and design Dirichlet Energy Distillation with a LDED loss to ensure the DE ratios REC and REC of the student MLP's FC layers FC and FC preserve the smoothing effects of GP()) and FT()) from the teacher GNN.
Observe that (i) FT operates independently of the graph, whereas GP requires the graph; (ii) FT in Eq. (4) has the same formulation as an FC layer of an MLP in Eq. (2), Therefore, to approximate the 1-th GNN layer, we employ two FC layers FC and FC in the student MI.. In Eq. (5), the FC layer of the student MLP is to approximate GP in Eq. (4), while the FC layer approximates FT() in Eq. (4), with teacher injection to set W w and by b(",, transferring the parameters of FT to FC ( Sy is the embedding of node generated by the student.
FC for GP :
W/ + +bb))) FC for FT :s. = a W 2 M + bb7)
(5)
where w = w() and b = b and w()) and b()) are from Eq. (4).
Different from existing methods, TINED distills knowledge from GNNs to MIPs on a per-layer basis. Directly injecting GNN parameters into the MLP is anticipated to improve the effectiveness. For layer-wise GNNs with T layers, the resulting MLP will have 2T FC layers. Typically, T is small, e.g.. 2 or 3, and the efficiency impact on inference between MLPs with 2 and 4 layers is minimal (1.45ms
There are two typical inference deployment settings as de- picted in Figure 2(b). TINED can operate without graph dependency for inference (Zhang et al., 2022b), which is useful when the graph structure is unavailable or when new unseen nodes lack graph connections; TINED can also work with graph structures when allowed (Tian al., 2022). Fol- lowing (Zhang et al., 2022b; Tian et al., 2022), we use GraphSAGE (Hamilton et al., 2017) with GCN aggregation as the teacher GNN to explain TINED. Our experiments include results over different teacher GNNs. In Appendix A.3, we explain how to use TINED for various GNNs.
vs. 1.63ms, as reported in (Zhang et al., 2022b), compared experiments to 153ms by GraphSAGE). Furthermore, show that TINED can achieve a favorable tradeoff between effectiveness and efficiency for inference. For decoupled GNNs, c.g., APPNP (Klicpera et al., 2019), with FTs and GPs decoupled, the teacher injection is also applicable with small number of FC layers (see Appendix A.3).
4.1. Teacher Injection
In Theorem 4.1, we provide theoretical approximation bound between GP(() in the 1-th layer of teacher GNN and its corresponding FC in student MLP, serving as an at- tempt to establish the relationship between GNNs and MLPs for knowledge distillation. The proof is in Appendix A.2.
The idea is to inject the parameters of certain operations of GNNs into MLPs, thereby direetly transferring the teacher's knowledge to the student. The GNN operations to be in- jected should be compatible with MLP in Eq. (2), i.e., with the same formulation. As shown in Eq. (1), a GNN layer consists of GP and FT operations. In what follows, we show that the FT operations of GNNs can be injected into student MLPs. We use GraphS AGE as an example, while the analy- sis on other GNNs as teacher is in Appendix A.3. The 1-th layer of GraphSAGE with GP()) and (T(") operations is
Theorem 4.1. For a sparse matrix L E T=x= and a fea- lure matrix H Raxd with rank(H) d, there exists transformation matrix W approximate LH by HW with relative error
|LH-- HW S. (L),
(P h.9 = CAT ( () (ID!-
Vu e(v))))
where || 11 is the Frobenius norm and Xmaxe (L) is the largest eigenvalue of laplacian L.
O9. (1 = a (h.) w + b()).
(4)
Specifically, Laplacian matrix L represents graph topology used in GP(!), LH represents the output of GP((), while HW* represents the output of FC and HW* approx- imates LH. The relative error is upper-bounder by the largest eigenvalue Amax (L) of L, regardless of graph size.
In Eq. (4), GP(9) operation has an aggregator !! to combine the (1 - 1)-th representations h (!(-1) of u's neigh- bors and then concatenates (CAT) them with h !!-1) to get 1. In (T operation, a learnable transformation matrix ( is applied to ii(") followed by an activation function to yield the 1-th layer representation h (!) of v.
The parameters of FC layers FC in Eq. (5) are sourced from the FT()) operations of teacher GNN. To ensure these
4

---

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
FC layers contribute the student MLP's training in regulated manner, we introduce gradient modifier y to fine- tune the injected parameters when updating the gradients,
While the overall Dirichlet energy of embeddings is de- creasing in GraphSAGE, in a specific I-th layer shown in Figure 1, we make the following two consistent observa- tions about DE ratio.
(i) Within the same layer for / = 2, the DE ratio Re of FT(() consistently exceeds the DE ratio REpes of GP(", demonstrating that GP()) operation actively smooths embeddings, whereas FT operation is relatively conservative for smoothing. (ii) At 2, DE ratio ReTØ, even surpasses 1, indicating that in this layer, FT(() acts to diversify embeddings rather than smoothing them. These trends are consistently observed across layer-wise GNNs, including GCN and GAT in Appendix A.4.
V
1.2
(6)
V M - Vb M
where 1) is a hyperparameter control fine-tuning, while V represents gradient.
In other words, during training, the update of the parameters in FC based on gradient optimization propagation is adjusted by 1) parameter.
after backward
Figure reveals that FTss and GPs can have opposing ef- fects on smoothing. Recall that we associate FT and GP of the 1-th GNN layer with FC and FC layers in the student MLP in Eq. (5). The proposed Dirichlet Energy Distillation technique aims to encapsulate the dis- tinct smoothing behaviors of FT and GP(!) into FC and FC respectively, thereby transferring teacher GNN's knowledge of smoothing effects to student MLP.
4.2. Dirichlet Energy Distillation
It is widely recognized that an appropriate degree of smooth- ing is crucial for the efficacy of GNNs. Here we make an in- teresting observation that, within the 1-th GNN layers, GP()) operations tend to aggressively smooth node embeddings, while FT operations often apply conservative smoothing or even diversify embeddings. These distinct layer-level smoothing behaviors in teacher GNNs should be captured in the student MLPs.
Specifically, the loss of the I-th GNN layer for Dirichlet Energy Distillation, c LDDD is the sum of the squares of the difference between DE ratios Rapa of GP()) and Ryc of the FC layer, and the difference between Ryre: of FT(") and Reç of the FC layer. The total DED loss CDDD is the sum of (i)) DED for all GNN layers = I,.., T.
To achieve this, we introduce Dirichlet Energy Distillation. Dirichlet energy (Rusch et al., 2023) a measure commonly used quantify the degree of smoothing in embeddings by evaluating node pair distances. Definition 4.2 defines the Dirichlet energy F(H) of node embedding matrix H.
Definition 4.2. Given node embedding matrix H E R learned from either the GNN teacher or the MLP student at a certain layer, the Dirichlet energy of H is
DED = -Rorv) - RECL.)) + (Rpro) R
(8)
LDED
DED
(H LH); H = 1 n
(7)
The final objective function L of our method TINED is the weighted combination of ground truth cross-entropy loss LCg, soft label distillation loss LKZ. and Dirichlet Energy Distillation loss LDED:
where tr(-) is the trace of matrix, and L D A the Laplacian matrix of graph G.
A lower Dirichlet energy value E EHH) suggests that the em- beddings in H are smooth, whereas a higher value indicates diversity among the embeddings. For an operation op that processes H to output op( (H), if (H)) < E(H), then the operation op is smoothing the embeddings; on the other hand, if (H)) > E(H), op diversifies them.
C. = Σ Lor(ýr, + Σ Cxi 2 Σ DED:
+
"EVL
BEV
(9)
where À and B are weights for balancing the loss functions.
5. Experiments
We define Dirichlet energy ratio Rop (DE ratio) as follows. Definition& 4.3 (DE ratio). The DE ratio Rop of an operation op is the Dirichlet energy of its output (H) over the E())(H)) Dirichlet energy of its input EHH), Rop E(H)
5.1. Experiment Settings
Datasets. We use 7 widely used public benchmark datasets, including Cora, Citeseer, Pubmed, A-computer, A- photo (Zhang et al., 2022b; Yang et al., 2021), and Arxiv and Products (Hu et al., 2020) that are two large OGB datasets, to evaluate our method and baselines. Table 10 in Appendix A. provides the data statistics and splits.
On a trained teacher GraphSAGE with layers, in each layer, we calculate the DE ratios Rete: and Rep of (T()) and GP(!) operations respectively. We repeat 10 times and report the average DE ratios in Figure 1, over the benchmark datasets used in experiments.
Teacher Architectures. Following (Zhang et al., 2022b;
5

---

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
Table 1: Under the transductive((ran) setting, the accuracy results of the teacher and all methods without or with graph dependency for inference on y (higher is better). The best in each category is in bold. OOM means out-of out-of-memory.
Teacher
Withour Graph Dependency
With Graph Dependency
Datasets
SAGE
MLP
FFG2M
KRD
GLNN
GLNN
TINED
NOSMOG
NOSMOG
TINED+
Cora
80.6411.57
59.18+1..0
82.38:11.41
12.27=1.31
10.26=1.66
81.31
1..63:11.7
83.04:11.26
82.27:1.75
83.70=1.02
Cileseer
1.54
1.85
1.58
7,784=1..7
71.22=1.500
72.38:11.40
7.54
1..78:1.54
73.85+22.27
25.39=1.59
Pubmed
75.56:1:2.06
68.39113.09
76.56.13.41
77.01=3.11
75,59-2.46
76.95.12.72
77.09:12.14
77.3411..36
76.79.22665
77.75-3.14
A-computer
1.38
67.62+2221
83.67:11.04
82.87=0877
82.71=1.18
83.6421.13
85.18:1.12
84.04:1:001
84,33=11144
84.82=1.58
A-photo
90.85++0.87
77.29+11.79
93.18:40.87
92.82=00.74
91.95=1.04
92.99:0.63
93.97-0..3
93.36:1:0.99
93.57=0048
94.05-0.39
Arxiv
70.73++0.35
55.67+00.24
58.51+0.35
59.26=0.51
63.75=0.48
63.78+00.99
64.44:0.72
0..65+0.29
71.17=0.00
01.52=0.34
Products
77.17400.32
60.02:1011
OOM
0OM
63.71=0.31
65.56:10.26
69,48110.25
78.45-00.38
78.47=0.28
78.59=0.28
Tian et al., 2022). for the main results, GraphSAGE (Hamil- ton al., 2017) with GCN aggregation is used as the teacher model. We also conduct experiments of different GNN teachers, e.g., GCN, GAT and APPNP in Section 5.5. The details of teachers and hyperparameter settings are in A.7.
formance results from 10 trials, each with a unique random seed. A method is evaluated by accuracy with the best model chosen on validation data and tested on test data.
5.2. Performance in Transductive Setting
Methods. TINED does not require graph dependency for in- ference, similar to GI.NN (Zhang et al., 2022b). When graph dependency permissible, we enhance it to create TINED+, akin to NOSMOG (Tian et al., 2022), to incorporate graph structural information. Baselines GLNN and NOSMOG use their original layer and hidden dimension configurations as per their papers. Additionally, we configure them with the same layer and hidden dimension settings as ours, dubbed as GLNN and NOSMOG", for evaluation. We further com- pare with KRD and FFG2M (Wu et al., 2023b;a), as well as with MLPs and the GraphSAGE teacher.
Table 1 reports the results under the transductive setting with inference on node set DV Table 1 can be directly compara- ble to those reported in literature (Tian et al., 2022; Zhang et al., 2022b; Hu et al., 2020; Yang et al., 2021). In Table 1, GNNs-to-MLA MLPs methods are in two categories: with or with- out graph dependency for inference. Baselines GLNN and NOSMOG use their original student MILP configurations for layers and hidden dimensions as recommended in their respective papers. GLNN and NOSMOG* (with aster- isk ") adopt the same configurations as our model for fair comparison. TINED surpasses all baselines without graph dependency, often with a significant margin. For exam- ple, on the large OGB Products dataset. TINED achieves gains of 9.42% 5.77%, and 3.92% over MIP, GLNN,, and GLNN* while baselines KRD and FFG2M are OOM. On Citeseer, TINED achieves 74.43% accuracy, improving by 15.93%, 3.21 %2 2.05%, 1.59%, and 1.58% over MLP, GLNN GLNN", KRD, and FFG2M. The performance of TINED validates the power of the proposed techniques to pre- serve layer-level knowledge into MLPs. Similar to GLNN, TINED surpasses the teacher on the first five data, while being effective on the last two. When graph dependency is allowed, TINED+ excels NOSMOG and NOSMOG" on most datasets, except Arxiv where the performance is com- parable. On Citeseer, TINED+ achieves 75.39% accuracy, 1.54% higher than NOSMOG* Moreover, all methods with graph dependency, including TINED+ and NOSMOG, per- form better than those without graph, and TINED+ excels the teacher model on all datasets, indicating the extensibility of TINED+ to consider graph structures for inference.
Transductive and Inductive Settings. Two settings are considered (Zhang et al., 2022b; Tian et al., 2022): trans- ductive (Iran) setting, and production (prod) setting with both inductive and transductive evaluations (ind & tran).
tran: A model is trained on G. X, and Y',, and soft labels z, of all nodes in V are used for knowledge distillation. Inference evaluation is conducted on the nodes in V
prod (ind & Iran): we randomly select out 20% of nodes from the unlabeled nodes VU, dividing Ve into disjoint inductive (unobserved) subset and observed subset, y = yu U DU Node features and labels are partitioned into disjoint sets, i.e. X =
L =X" L X the L x and Y y YY L and Let Got. be the graph induced from G, with edges connecting nodes in y L DU -be- A model is trained on Gats. XL, x abs: and L Soft labels in subsets y4 L Vte ohe are used for distillation. In this prod (ind & tran) setting, inference is evaluated on yir for ind and yt for tran (which is different from the tran setting over y explained above). The overall prod performance is the weighted sum of ind performance on D and tran on D with weights proportaional to the size of and and obe:
the
5.3. Performance in Production Setting with ind & tran
We then conduct experiments under the prod (ind & tran) setting with results in Table 2. Note that the tran results in Table 2 over DU
Evaluation. Following (Tian et al., 2022; Zhang et al., 2022b), we present the mean and standard deviation of per-
ober different from the results in Table
6

---

TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation
Table 2: In prod (ind & tran) setting, the accuracy results of the teacher and all methods without or with graph dependency for online inference are presented. ind indicates the results on ind= tran indicates the results on V obet and prod indicates the weighted sum of the performance of both ind and tran with weights proportional to the size of V and V The best result in each category is in bold. The teacher's performance surpassing the best result in at least one category is italicized.
Teacher
Without Graph Dependency
With Graph Dependency
Datasets
Eval SAGE
MLP
FFG2M
KRD
GLNN
GLNN
TINED
NOSMOG
NOSMOG
TINED+
prod
79.53
59.18
78.60
75.74
77.82
78.14
78.90
81.02
80.30
80.77
Cora
ind
11.03=1..71
59.4443.36
72.02++1.43
70.26+:1.94
73.21=1.50
13,58=1442
74.38=1.28
81.36=
80.98=2.39
81.50=2.54
fron
19.16=1 600
19.12=1149
80.01+1.41
77.11=1.44
78.97=1.56
19.65=1.45
10.04:1.50
1.65
8..13=1.64
80.59:1144
prod
68.06
58.49
71.89
71.38
69.08
70.91
72.29
70.60
71.33
73.58
Citeseer
ind
69.14422.000 99
59.31
69.75:43.16
69.78:33.44
68.48=2.38
71.10 1.50
72.68=
20.30+22.30
72.35=2.99
74,20=11667
froz
67.79+2.80
1.92
72.12++2.69
71.77+2.811
69.23+2.39
70.86+ 1.66
72.20++1.66
70.67+22.25
71.07+ 1177+11.85 1.85
73.43 +: 1.63
prod
74.77
68.39
73,98
76,00
74.67
75,21
75.79
75.82
75.57
75.90
Pubmed
ind 75,07 89
68. 68.28::3225
73.49:7.91
75.17=33111
74.52=2.95
74,83-2.88 83
75.64:13.02
35.87=3.32
75,49=2..6
2,95
from
74.70+2.33
68.42++3.06
74.10+7.78
76.20+33.00
74.70+2.75
75.30+
75.83-+-2.81
75.80 3.06
7..58+2.83
75.80 +288
prod|
82.73
67.62
82.69
81.17
82.10
83,23
84.46
83.85
85.02
85.08
A-computer
ind 12.
67.69+22.20
10,52-:1.56
19.15=11.22
80.27=2.11
81,10=1149
82,83: 1.45
84.36
85,23 1.51
85,45=1.66
fron
82,70+ 1.34
67.60++2.23
13,23++.336
11667+19.2
82.56-1.886
1..77+1.36
84.87++1..8
83.72+1.44
84.97 1 1.4
84.98 + 1.32
prod
90.45
77.29
92.35
91.84
91.34
91.54
93,38
92.47
92,00
93,12
A A-photo
ind 90.56:t
.47
1..44:11.50
00,70:+0.76
90.04+11.12
89.50= 1.12
09.35=00889
91.96::00.72
1.061+1009
92.93::0.96
93.27 0.885
fron 90.42++0.68 668
77.25+11900
92,77++0.24
92.29++0.63
91.80=0.49
92.0
93,74++0.55
92,44+0.51
91.77+0069
93.08 0.68
prod 70.69
55.35
59,60
59.32
63,50
64.17
63,24
70.90
70.95
71.22
Arxiv
ind 70.69:00.58
55.29:40.63
57.02-+0.43
57.32++0.31
59.04=0.46
58.73=0.446
59.79:10.46
0.55
0..12:0.39
70.42:
fron 00.69-10.39
55.36:10.34
60.24:1..33
59. 59.82:1.227
64.61=0.15
05.5
64.10:10.38
11.10
71.16=0.15
7.14
prod 76.93
60.02
OOM
OOM
63,47
68.48
69.35
77.33
78.25
78.91
Products
ind 77
60.02:#0.09
OOM
OOM
63.38 0.33
68.13#0.20
-8.68-00.27
07
7.22
79.31 0.29
tran 76.86:1..27
60.02:10.11
OOM
0OM
63 0.31
68.57-0.20
69.52.10.27
77.41
78, 18:2032
0.28
1 on D in Section 5.2. In Table 2, TINED and TINED+ can outperform the teacher model and the baseline meth- ods in categories with or without graph dependency un- der almost all settings. For instance, without graph depen- dency for inference, TINED is better than baselines, except Arxiv and Pubmed where close performance is achieved. As an example, on A-computer under ind setting, TINED achieves 82.83% accuracy, significantly improving GLNN* by 1.73%. With graph dependency. TINED+ surpasses NOS- MOG methods on all datasets except prod and Iran setting on Cora, where the performance is comparable. For instance, on Citeseer under prod, TINED+ achieves performance gain of 2.98% and 2.25% over NOSMOG and NOSMOG* re- spectively. Moreover, Arxiv and Products with a signifi- cant distribution shift between training and test data (Zhang et al., 2022b), TINED+ outperforms the teacher and NOS- MOG, showing the capability of our techniques on large real-world graph datasets. We conclude that TINED and TINED+ can achieve excellent performance in prod setting with ind & tran.
GNN
KRD
NOSMOG NOSMOG
GLNN
R5G2M
76
TINED
GLNN
FINED
TINED+
TINED
Accuracy
74
NOSMOR
NOSMOG
Aceuracy
TFRZM
SAGB-13
80
72
'GLNN
GLNN
SAGE-12
SAGE-LI
16
10
16
10
76
Inference Time (ms)
SAGE
GAT
APPNP
Figure 3: Inference Figure 4: Accuracy on Different Time and Accuracy
Teacher GNNs
(74.43% and 75.39%) while being fast (1.63ms and 1.64ms). In contrast, GNNs are slower, e.g., 2-layer GraphSAGE (SAGE-L2) takes 153.14ms, and 3-layer GraphSAGE (SAGE- I.3) takes 1202.45ms. TINED and TINED+ are 94 times faster than SAGE-L2 and 733 times faster than SAGE-L3. All the distillation methods (NOSMOG, GLNN', TINED, TINED+, KRD and FFG2M) have similar efficiency in around 1 1-2ms, with negligible differences. Compared with SAGE teacher that needs 153. 14ms, the efficiency of distillation techniques in our design validated.
5.4. Inference Time
Figure 3 shows the trade-off between accuracy and infer- ence time on Citeseer. Methods closer to the top-left cor- ner achieve better balance of accuracy and speed. Our methods, TINED and TINED+, achieve the highest accuracy
5.5. Different Teacher GNN Architectures
Figure 4, we show that TINED and TINED+ can maintain strong performance with alternative GNN teachers, includ-
7

---

TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation
Table 3: Approximation error
1=
1= 2
Datasets
TINED
GLNN
TINED
GI
Cora
1.32
1.64
0.90
1.12
(a) SAGE (epoch=500)
(b) TINED (epoch=0)
(c) TINED
(d) GINN (epoch=0)
(c) GINNN"
Citeseer
0.46
0.86
0.26
0.75
(epoch=500)
(epoch=500)
Pubmed
0.88
0.82
0.74
1.02
A-computer
1.89
1.88
2.55
2.65
Figure 5: t-SNE of model embeddings at different training stages on Citeseer.
A-photo
1.21
0.83
1.40
1.54
SAGE
GENN
TINED-DE DED
TINED
Table 4: Ablation study
DE Raño
DE Ratio
Datasets
TINED-TIN TINED-DED TINED
ÂTIN
ADED
Corn
81.32+1.60
81.64+
82.61+157
1.31%
0.999%
Ciesser
71.8811.27
74.12.1.42
74.4311.53
t 2,555
1 00331%
Pubmed
75.93:3340
76.45:22.68
77.09#2.14
1.16%
0.66%%
(n) Corn
(b) Citesoer
(c) Pubmed
(d) A-computer
(e) A-pooto
A-computer
84,59+1.78
84 94.80+1.66
85. 18.1111112
+.059%
0.38%
A-photo
93,90+0.52 93.67=0.69 93.97=0.53
0.0.7%
0.30%
Figure 6: Learned DE ratios in MLPs v.s. ground truth of 2-layer GraphSAGE.
Arxiv
63.72:+0.89
64 39.39-00.80
64.44=0.72
1 0.722% 1 0.008
Products
69.20110.25 68.63.10.66 69.48.10.25
1
ing GCN, GAT, and APPNP, in addition to GraphSAGE, We report the average performance of a distillation method with different teacher GNNs across the five benchmark datasets. With different teachers, TINED can always learn more effec- tive student MLPs than GLNN, GLNN KRD, and FFG2M. With graph dependency, TINED+ maintain its strong per- formance with different teachers and surpasses NOSMOG and NOSMOG'. When different teacher GNNs are adopted, Figure 4 illustrates the effectiveness of our methods to pre- serve layer-level knowledge into MLPs.
Table Vary n of TINED on transductive setting. Best
results are in bold.
Cora
Citeseer
Pubmed
A A-computer
A A-photo
1e-09
73.80-2.56
67.20:1 1.86
73. 23.1
72.53:12.17
86.06.12.38
73.50=
67.19
73.32+1.88
72.73++1.99
1.779
0.001
77.12-0.91
67.24+1.83
73.69+1.83
84.13++..23
87.30+-2.08
0.01
79.26=1.30
1.73
75.12:2.32
85.17:1.21
90.48:11.33
0.1
81.40=1.69
1..52:11.58
76.55:22811
83.61 1.67
93.97 0.55
0.5
82.01 12.01-1.64
74.57+1.42
77.10-2.15
83.23++ 1.29
93.51 40060
82.61-1.58
13.57=1.39
76.65=2.77
82.85+0.91
93.37-0.51
10.0
78.85=1.63
73.56=1.48
75.61=2.73
70.63+4.87
89.5320078
5.6. Experimental Analysis
DE ratio. Recall that Figure shows the ground-truth DE ratios of teacher GraphSAGE, and we propose Dirichlet En- ergy Distillation in Section 4,2 to preserve smoothing effects into MLPs. Figure 6 reports the learned DE ratios of TINED, TINED-DED, and GLNN TINED is more closely aligned to the ground-truth DE ratios, highlighting that TINED ef- fectively retains the GNN smoothing properties into MLPs.
Ablation Study. Denote TINED-TIN as TINED without Teacher Injection (TIN), and TINED-DED as TINED without Dirichlet Energy Distillation (DED). The results are in Ta- ble 4. Both TIN and DED contribute positively, and TINED achieves the best performance all datasets, demonstrating the effectiveness of our techniques in TINED.
Visualization. Based on the visualization of teacher SAGE's output embeddings after 500 epochs of training (Figure 5())), we compare the embeddings of TINED and GINNN* both at initialization (epoch 0) and after training convergence (epoch 500). Figure 5(b) shows TINED initial- ized (epoch 0) using the proposed teacher injection tech- nique, resulting in an embedding visualization similar to that of the teacher in Figure 5(a). In contrast, the visualiza- tion of GLNN at epoch 0 appears quite random, as seen in Figure (d). Upon reaching training convergence (epoch 500), TINED in Figure 5(c) maintains a visualization closely resembling the teacher's in Figure 5(a), whereas the base- line GLNN* displays noticeably different visualization. This evaluation confirms the effectiveness of the proposed teacher injection and Dirichlet energy distillation techniques employed by TINED. More visualization results are in A.5.
Approximation Bound. Theorem 4.1 shows a bound be- tween GP()) in GNNs and its counterpart FC in MLPs. Table 3 reports the errors of TINED and GLNN" using FC to approximate GP(!) (|GP(0) (H) F ( for 1 = 1, When / = 2, the errors of TINED are lower than GINNN, while the errors for = 1 are comparable, showing TINED effectively approximates GP
Parameter Sensitivity Analysis. In our method, the param- eter 7) controls the degree of fine-tuning in Eq. 6, while controls the importance of Dirichlet Energy Distillation in Eq. 9. Tables 5 and 7 present the accuracy results of varying and 3 of TINED across multiple datasets. In Table 5, as increases, TINED demonstrates a clear trend where perfor- mance initially improves and then declines, with the best results highlighted in bold. A similar pattern is observed for
8

---

TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation
Table 6: Experiment result under prod (ind & tran) setting on heterophilic datasets. The best result in each category is in bold.
Teacher
Without Graph Dependency
With Graph Dependency
Datasets
Eval
SAGE
FFG2M
KRD
GLNN
GLNN
TINED
NOSMOG
NOSMOG*
TINED+
prod
35.47
38.34
37.16
39.90
39.70
41.95
38.17
39.43
40.89
Squirel
ind
41,44:14.66
42.00=4.78
42.11=1.777
44.89=5.67
45.00-1.556
46.89=5.23
45.44=4.75
44.33=:3.24
46.78:13.00
Iran
33.98:1
3..4
35.93.13.36
38.65
38.37:1.00 1.02
40.72: 1.36
16.35.11.69
38.20:11.59
3.05
prod 47.55
50.33
49.56
49.87
49.41
50.70
47.86
48.80
50,42
Amazon-ratings
ind
47.45+ 1.48
47.55-0.97
47.68+1.00
1.00
47.72-1.00
47.71+1.14
49.02+ 1.02
47,47+1445
48.46+ 1.37 49,51 1,55
tran
47.58:+:0:8
51.03=1.73
10.03=1.50 50.41=0.45
09.84=0.39
5..12=0.44
47. 47.96=20.31
08.88=00556 0,.65
Table 7: Vary 3 of TINED on transductive setting. Best
Table 8: Compare with MLP with 4 layers under transduc- tive setting.
results are in bold.
3
Cora
Citeseer
Pubmed
A-computer A-photo
Cora
Citeseer
Pubmed
A-computer A-photo
le-09
1..33 73.39.1..28
76.63.12.42
84.70.11.15
93.41 93.41:10.61
MLP 60.84=1.08
63.4++1996
69.41-2.88
70.07+1.77
80.19:
le-06
81.6411.77
73.39_1.30
76.28:12.76
85.17
93.48.10.65
MLP 58,20=3.16
60.81+3.48 67.61=1551
66.8443.03
78.66+22.88
0.001
81.61+1.57
73.39=1.31
76.51+2.64
1..80=+1.11
93.65:+0.68
TINED 82.63+1.57 74.43+1.533 77.09+2.14 85.18+1.12 93.97+0.53
0.1
81.71+1.59
73.65+1.38
77.10+2.15
84.70+0.99
93.97+0.58
1.0
82.61=1.58
73,81=1.29
75.8412.66
84.29-11088
86.5915.66
10.0
80.92=2.15
74.57=1442
71.57+2.776
71.35=6.30
78.00=3.19
Table 9: Comparison with MLP* with 4 layers under (rod(ind&tran)) setting
3 in Table 7, emphasizing the trade-off controlled by these parameters. Note that hyperparameter tuning is essential in machine learning research. Appendix A.7 of the paper details the search space for our parameters.
Eval Cora
Citeseer
Pubmed
A-computer A A-photo
MLP
ind 61.31 63.95-42.95 69.6622.68 70.36=2.48 79.76:12.00
tram
60.88+114 41 62,99-42.39 2.39 69.67+2.61 69.92-2.03
79.53-+2.00
MLP ind
58.67:2.42 62. 15:3.50
67.76
08.09=2.60 77.27:2.17
Heterophilic Datasets We conducted experiments repre- sentative heterophilic datasets, Squirrel and A Amazon-ratings, under the production setting. Squirrel (Rozemberczki et al., 2021) is a web page dataset collected from Wikipedia, while Amazon-ratings (Platonov et al., 2023) is a product co- purchasing network based on data from SNAP Datasets. In Table 6, our methods, TINED and TINED+, consistently outperform existing approaches across all settings, often by a significant margin. For instance, on the Squirrel dataset without graph dependency, TINED achieves a prod perfor- mance of 41. 1995%, surpassing the best competitor GI.NN, which achieves 39.90%..... Similarly, on the Amazon-ratings dataset with graph dependency, TINED+ achieves a prod accuracy of 50.42%, representing 1.62% improvement over the best competitor NOSMOG". These results high- light the effectiveness of our proposed methods in handling heterophilic datasets.
tram 12- 112+-1811 81 61,46++2.46
68.07 + 1.87
27.41
77.05++2448
TINED ind
74.38=1128 72.68 1.97 75.64=3.02
82.83=11455
91.96:0.72
tram
80.04++ 1.50 72.20++1.66
75.83++2.81
14.87-1.38 93.74-+0.51
MLP. In contrast, TINED significantly improves upon MLP* achieving notable accuracy of 77.09%, highlighting its robustness and effectiveness.
6. Conclusion
We introduce TINED, a novel method for distilling GNN knowledge into MLPs. Our approach includes Teacher Injec- tion, which directly transfers well-trained parameters from GNNs to MLPs, and Dirichlet Energy Distillation, which preserves the unique smoothing effects of key GNN oper- ations within MLPs. Comprehensive experiments demon- strate that TINED outperforms existing methods across var- ious settings and seven datasets. Currently, the number of layers in the student MLP is dependent on the number of layers in the GNN teacher. In future work, we aim de- velop new techniques to accelerate the process and allow for flexible number of fully connected layers by considering the intrinsic properties of GNNs and MLPs. Moreover, we will explore the potential of TINED in other types of grpahs, including heterogeneous graphs and dynamic graphs, to further enhance its applicability and effectiveness.
Comparison with Vanilla MLP with Additional Layers. In addition to the vanilla MLP with layers, we also evalu- ate a 4-layer MLP, denoted as MLP*, Tables 8 and present the results under both transductive and production settings. Our method, TINED, consistently outperforms MLP* across all datasets. Interestingly, MI.* demonstrates degraded performance compared to the 2-layer MLP, suggesting that additional layers lead to overfitting on these datasets. For instance, on the Pubmed dataset, MLP* achieves 67.61% %, which is inferior to the 69.41% % achieved by the 2-layer
9

---

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
Acknowledgements
gnn for unsupervised node anomaly detection. In Pro- ceedings of the ACM on Web Conference 2025, pages 1225-1236, 2025b.
This work is supported by grants from the Research Grants Council of Hong Kong Special Administrative Region, China (No. PolyU 25201221, No. PolyU 15205224), and NSFC No. 62202404. This project has been supported by the Hong Kong Research Grants Council under General Research Fund (project no. 15200023) as well as Research Impact Fund (project no. R1015-23). This work is sup- ported by Otto Poon Charitable Foundation Smart Cities Research Institute (SCRT) P0051036-P0050643 and grant PO048511 from Tencent Technology Co., Ltd.
Kaituo Feng, Changsheng Li, Ye Yuan, and Guoren Wang. Freekd: Free-direction knowledge distillation for graph neural networks. In KDD, 2022.
William L. Hamilton, Rex Ying, and Jure Leskovec, Induc- tive representation learning on large graphs. In NeurIPS, 2017.
Xiaotian Han, Tong Zhao, Yozen Liu, Xia Hu, and Neil Shah. Mipinit: Embarrassingly simple GNN training acceleration with MLP initialization. In ICLR, 2023.
Impact Statement
This paper focuses on GNNs-to-MLP, distillation, which is a fundamental problem. There are many potential societal consequences our work, none of which we feel must be specifically highlighted here.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Dis- tilling the knowledge in a neural network. arXiv preprint arXiv:2211.0200 2015.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In NeurlPS, 2020.
References
Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee. Dis- tilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert. In ICASSP, pages 7087- 7091. IEEE, 2022.
Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang, Erjin Zhou, and Yue Gao. Graph-mlp: node classifica- tion without message passing in graph. arXiv preprint 2rXiv:210600403 2021.
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In AAAI, pages 3438-3445, 2020a.
Aryan Jadon, Avinash Patil, and Shruti Jadon. A com- prehensive survey of regression based loss functions for time series forecasting. arXiv preprint arXiv:221 22222211.2200 1,02989, 2022.
Lei Chen, Zhengdao Chen, and Joan Bruna. On graph neural networks versus graph-auge -ugmented mlps. In ICLR, 2021.
Zhihao Jia, Sina Lin, Rex Ying, Jiaxuan You, Jure Leskovec, and Alex Aiken. Redundancy-free computation for graph neural networks. In KDD, 2020.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, 2020b.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In EMNLP, pages 4163 4174. Association for Computa- tional Linguistics, 2020.
Xiang Deng and Zhongfei Zhang. Graph-free knowledge distillation for graph neural networks. IJCAI, 2021.
Zhihao Ding, Jieming Shi, Qing Li, and Jiannong Cao. Ef- fective illicit account detection on large cryptocurrency multigraphs. In CIKM, page 457-466, 2024.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Xiangyu Dong, Xingyi Zhang, and Sibo Wang. Rayleigh quotient graph neural networks for graph-level anomaly detection. In ICLR, 2024.
Thomas N. Kipf and Max Welling, Semi-supervised clas- sification with graph convolutional networks. ICLR, 2017.
Xiangyu Dong, Xingyi Zhang, Lei Chen, Mingxuan Yuan, and Sibo Wang. Spacegnn: Multi-space graph neural net- work for node anomaly detection with extremely limited labels. In ICLR, 2025a.
Johannes Klicpera, Alcksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural net- works meet personalized pagerank. In ICLR, 2019.
Seunghyun Lee and Byung Cheol Song. Graph- knowl- edge distillation by multi-head attention network. arXiv preprint arXiv: /0t.:://900022220 2019.
Xiangyu Dong, Xingyi Zhang, Yanni Sun, Lei Chen, Mingx- uan Yuan, and Sibo Wang. Smoothgnn: Smoothing-
10

---

TINED: GNNs-10-MLPS by Teacher Injection and Dirichlet Energy Distillation
Hao-Ting Li, Shih-Chieh Lin, Cheng- Yeh Chen, and Chen- Kuo Chiang. Layer-level knowledge distillation for deep neural network learning. Applied Sciences, 9(10): 1966, 2019.
from graph neural networks and injecting it into mlps: An effective gon-to-mlp distillation framework. In AAAI, pages 10351-10360, 2023a.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z. Li. Quan- tifying the knowledge in gnns for reliable distillation into mips. In ICML, pages 37571-37581, 2023b.
Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. In NeurlPS, 2020.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020.
Chen Liang Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In ICML, pages 08852-20867. PMLR, 2023.
Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou. Tinygnn: Learning efficient graph neural networks. In KDD, 2020.
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In KDD, pages 338-348, 2020.
Cheng Yang. Jiawei Liu, and Chuan Shi. Extract the knowl- edge of graph neural networks and go beyond it: An effective knowledge distillation framework. In WWW, 2021.
Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of gnns under heterophily: Are we really making progress? ICLR, 2023.
Ling Yang. Ye Tian, Minkai Xu, Zhongyi Liu, Shenda Hong, Wei Qu, Wentao Zhang, Bin CUI, Muhan Zhang, and Jure Leskovec. Vygraph: Rethinking graph representation space for bridging gnns and mlps. In ICLR, 2024.
Benedek Rozemberezki, Carl Allen, and Rik Sarkar. Multi- scale attributed node embedding. Journal of Complex Networks, 9())cnaab014, 2021.
Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convo- lutional networks. In CVPR, 2020.
Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10990 2023.
Jiaxuan You, Rex Ying, and Jure Leskovec. Position-aware graph neural networks. In ICML, 2019.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. Learning mips on graphs: A unified view of effectiveness, robustness, and efficiency. In ICLR, 2022.
Chunhui Zhang, Chao Huang. Yijun Tian, Qianlong Wen, Zhongyu Ouyang, Youhuan Li, Yanfang Yc, and Chuxu Zhang. Diving into unified data-model sparsity for class- imbalanced graph representation learning. In GLFron- tiers, 2022a
Yijun Tian, Shichao Pei, Xiangliang Zhang, Chuxu Zhang, and Nitesh Chawla. Knowledge distillation on graphs: A survey, arXiv preprint arXiv:2302.00213 2023.
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. Heterogeneous graph neural network. In KDD, pages 793-803, 2019.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018
Dalong Zhang, Xin Huang, Ziqi Liu, Zhiyang Hu, Xi- anzheng Song, Zhibang Ge, Zhiqiang Zhang, Lin Wang. Jun Zhou, Yang Shuang, et al. Agl: A scalable system for industrial-purpose graph machine learning. arXiv preprini arXiv:2003.00 024444 2020.
Haorui Wang, Haoteng Yin, Muhan Zhang, and Pan Li. Equivariant and stable positional encoding for more pow- erful graph neural networks. In ICLR, 2022.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: graph-centric, perfo-aprro package for graph neural networks. arXiv preprint arXiv: 1909001399, 2019.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mips new tricks via distillation. In ICLR, 2022b.
Wenqing Zheng, Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Subbian. Cold brew: Distilling graph node representations with incom- plete or missing neighborhoods. In ICLR, 2022.
Lirong WU, Haitao Lin, Yufei Huang, Tianyu Fan, and Stan Z. Li. Extracting low-/high- frequency knowledge

---

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 2020.
Ziang Zhou, Jieming Shi, Renchi Yang, Yuanhang Zou, and Qing Li. Slotgat: Slot-based message passing for het- erogencous graphs. In ICML, volume 202, pages 42644- 42657, 2023a.
Ziang Zhou, Jieming Shi, Shengzhong Zhang, Zengfeng Huang, and Qing Li. Effective stabilized self-training few-labeled graph data. Inf. Sci., 631:369-384, 2023b.
Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with optimization framework. In WWW, page 1215-1226,
2021.
12

---

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
A. Appendix
A.1. Datasets
A.3. Implementing Other GNNs as Teacher
In Table 10, we provide the statistics of the datasets in ex- periments. For all datasets, we follow the setting in (Zhang al., 2022b; Yang et al., 2021) to split the data. Specifi- cally, for the first five datasets, we the splitting in (Yang et al., 2021) and each random seed corresponds to a dif- ferent split. For the OGB datasets Arxiv and Products, we follow the OGB official splits based on time and popularity respectively.
Here we explain how to implement other GNNs as teacher in TINED, especially how to apply teacher injection in The Proposed Model section to different GNNs.
GCN. The message passing of GCN in the 1-th layer is
GP h (1-).
(15)
FT h [6 W + bD))
A.2. Proof of Theorem 4.1
where É = - AD- is the normalized Laplacian of the graph, = A + I, the adjacency matrix with self loops. D Σ, A. the degree matrix, w()) is the transformation matrix of I-th layer, and H is the output of (-th layer.
Proof. We denote the Hi as the pseudo-inverse of ma- trix H. Since rank(H) = d, H has linearly indepen- dent columns, its pseudo-inverse could be represented as H' (H H) "H" Moreover, given the SVD factoriza-
tion of matrix H = U (5
S
VT, where U E NNN and V E Rdxd are unitary, and Rdxd invertible diagonal matrix, we have H V S
For the 1-th layer of the GCN teacher, we directly inject w and b(9 of the teacher in Equation (15) into the student FC layer FC in Equation (5), i.c., w w and 1.2 = b(9,, while using FC in Equation (5) to simulate GP in Equation (15).
0 O
Denote H = LH, then we have
GAT. The message passing of GAT in the I-th layer with single head attention is
H - HWI
Σ
H
HW.
(10)
1=1
Let W H'H, have W - HHH
Then
GP h = 11 H(-1)
(16)
FT h() = (6.0) w(O + bO).
HW: - HHHH
-U -u(1)
la
0
U
H
(11)
where 11() is the attention matrix defined as u1 (1-1) w) (1-1 w(0 ( if (i.j) is
0
OR-d
Lea
Then
an edge in G, i.e., (i,j) E E, and up 0 otherwise, denotes the concatenation operation, and a 222 is learnable attention vector.
H HW 12222 = H
HH Hll
7
HH ||2||
Hll
(12) For the I-th layer of the GAT teacher, we directly inject w and (() of the teacher in Equation (16) into the student, i.e. w w and b = b(), while using FC in
sll
|||||| Holl
Equation (5) to simulate (P(() in Equation (16).
Amax H.
APPNP. APPNP decouples feature transformation and graph propagation into two stages. APPNP first performs FTs for Ti times in Equation (17) to get (T)) (usually Ti 2), and then propagates (T)) over the graph for Tz hops in Equation (18) by approximate Personalized PageR- ank.
Thus have
d
H HW 1
H
HW 11
(13)
(L)
max
1=1
FTO: H H
1 1, 2,.,, Ti,
(17)
=x max (L)
H
GP: H = (I-a)LH +oH (T)) + .... T T2.
Finally we get
(18)
LH H-- 1F
where a is a hyperparameter and L is the normalized Lapla- cian mentioned before.
Amax (L)
(14)
H
13

---

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
Table 10: Dataset Statistics and Splits.
Dataset
# Nodes
Edges
# Features
# Classes
# Train
Val
Test
Cora
2,485
5,069
1,433
210
2135
Citeseer
2,110
3,668
3,703
6
120
180
1810
Pubmed
19,717
44,324
500
3
60
90
19567
A-computer 13,381
245,778
767
10
200
300
12881
A-photo
7,487
119,043
745
8
160
240
7087
Arxiv
169,343
1,166,243
128
40
90941
29799
48603
Products
2,449,029
11,859,140
100
47
196615
39323
2213091
FT
GP
DE Ratio
DE Ratio
DE Ratio
DE Ratio
DE Ratio
10
10
10
10
10
GCN
10 0
10
100
10
10"
1
10
10-
10-
10"
10"
10
F1
F2
E1
1-2
E
12
EI
E-2
(a)
Cora
(b) Citeseer
c) Pubmed
(d) A-computer
C) A-photo
GraphSAGE
10
DE Ratio
10
DE Ratio
10
DE Ratio
10
DE Ratio
10
Ratio
10
10
10
10
10
10
10
10
10"
10
10
10
F2
1=2
1-2
(a) Cora
(b) Citeseer
(c) Pubmed
(d) A-computer
(c) A- A-photo
DE Ratio
DE Ratio
DE Ratio
DE Ratio
DE Ratio
10
10
10
10
10
GAT
100
10
10
10
10
10
1-
1-2
10
1-1
1-2
10
F
12
10"
|-1
1-2
10
1-2 (e) A-photo
1-1
(a)
Cora
(b)
Citeseer
(c) Pubmed
(d) A-computer
Figure 7: The layer-wise DE ratio of trained GCN, GraphSAGE and GAT, with 2 layers architecture different datasets. The z-axis is the layer number, and the y-axis is the DE ratio value.
For each of (T))) in APPNP, we inject into a FC layer in the student MLP. In other words, we have Tj FC layers to mimic the FT layers in APPNP. Then for the T2 steps of propagation in Equation (18), we use one FC layer FC to mimic all of them, since T2 be large. In total, there are Ti + FC layers in MLP to approximate the teacher GNN. The DED loss of APPNP is LDED = E/E RpG ) + (Rep RFC 12
the following nearly consistent patterns about DE-ratio: (i) within the same layer for 2, most DE ratio Rpr for FT(() is larger than Rero of GP((), suggesting that GP(!) actively smooths embeddings, whereas FT()) is relatively conservative for smoothing; (ii) at I 2, DE ratio ReT even surpasses 1, indicating that in this layer, FT(() acts to diversify embeddings rather than smoothing them. Fur- thermore, to investigate the smoothing behavior on deeper layer cases, we plot Figure 8 showing layer-wise DE ratios of 5-layer trained GNNs, and from Figure we can observe that the trained 5-layer GNNs have similar patterns to the 2-layer GNNs: the DE ratios of FT is generally higher than that of GP((), indicating that GP()) is aggressive while (T is relatively conservative for smoothing.
A.4. DE Ratios of other GNN architectures
To show the pattern of DE ratios in different GNN teachers, we plot the layer-wise DE ratios of GCN, GraphSAGE and GAT, addition to GraphSAGE, with 2 layers architecture on the five benchmarks Figure 7. From Figure we can observe that all GNNs share very similar patterns across datasets: within specific layer, in Figure 7, we observe
14

---

TINED: GNNs-to-MLPS by Teacher Injection and Dirichlet Energy Distillation
FT
GP
10
DE Ratio
10'
DE Ratio
10"
DE Ratio
10
DE Ratio
10
DE Ratio
GCN
10 0
10
10
10%
10
10
10
10
10"
10"
10
10-
-1
2 1-3 1-11-5
1-3 1 1-S
1-2 1-3 14
1-3 H
(a Cora
Citeseer
cc) Pubmed
A-computer
(e) A-photo
DE Ratio
DE Ratio
DE Ratio
DE Ratio
10 DE Ratio
GAT GraphSAGB
10
10
10
10
100
10
10
100
100
10
10
10
10"
10
10 1-2 1-
10
F1
44 15
2 1=
1 1 1 15
1- 1-3
(a)
Cora
Citeseer
Pubmed
A-computer
cc) A-photo
DE Ratio
DE Ratio
DE Ratio
10 DE Ratio
DE Ratio
10
10
10
10
100
10
100
10
10
10
F1 3
14
10"
21
4-445
10
3 (c) Pubmed
10
2
H
11213441's
11-21-3141
(a) Cora
(b) Citeseer
(d) A-computer
(e) A- A-photo
Figure 8: The layer-wise DE ratio of trained GCN, GraphSAGE and GAT, with 5 layers architecture on different datasets. The r-axis is the layer number, and the y-axis is the DE ratio value.
A.5. Lase t-SNE Visualization
the baseline GLNN' in Figure (e) produces embeddings that are visually distinct. These observations demonstrate the effectiveness of the teacher injection and Dirichlet en- ergy distillation techniques in TINED.
In Figure 9, we present a detailed layer-wise t-SNE visu- alization of the embeddings for various models and stages. Figure (a) (1st row) shows the 2-layer GraphSAGE. teacher model. Figure 9(b) (2nd row) illustrates TINED at initial- ization before training (epoch 0), while Figure 9(c) (3rd row) displays TINED after training convergence (epoch 500). Figure (d) (4th row) and Figure 9()) (5th row) depict the baseline GLNNN at initialization (epoch 0) and after train- ing convergence (epoch 500), respectively. For the 2-layer GraphSAGE in Figure 9a)), the 1st row visualizes the em- beddings from left to right: after the FT()) operation in the first layer, the (1(1) operation in the first layer, (2 in the second layer, and GP(2) in the second layer. For each distillation method in Figures 9(-e),, each row contains four plots representing the embeddings of the four fully- connected (FC) layers the methods, corresponding to the four operations in the first row of the teacher. Compared to the teacher model in Figure 9a), TINED initialization in Figure 9(b) already exhibits quite similar embedding pat- terns, thanks to the proposed teacher injection technique. In contrast, the baseline GINN" shows a random visual pattern at epoch 0 in Figure 9d)) which is radically differ- ent from the teacher model. After training convergence at 500 epochs, TINED's 's visualized embeddings in Figure 9(c) closely resemble those of the teacher in Figure 9a), whereas
A.6. Computer Resource Details
The experiments both baselines and our approach implemented using PyTorch, the DGL (Wang et al., 2019) library for GNN algorithms, and Adam (Kingma and Ba, 2015) for optimization. We run all experiments on Intel(R) Xeon(R) Platinum 8338C CPU @ 2.60GHz CPU, and a Nvidia Geforce 3090 Cards with Cuda version 11.7. Source codes of all competitors are obtained from respective au- thors. The totally training time cost for one set of hyper- parameters varies among datasets, from Cora costing 10 minutes to Products costing 13 hours.
A.7. Teacher details and hyperparameter search space
The hyperparameters of GNN models on each dataset are taken from the best hyperparameters provided by previ- ous studies (Tian et al., 2022: Zhang et al., 2022b). For APPNP teacher model, we found that the suggested hyper- parameter in previous literature (Zhang et al., 2022b; Tian et al., 2022) produces relatively poor results, thus we search it in following space, achieving a better APPNP teacher: # layers from [2, 3). learning rate from [0.0001, 0.001, 0.01].
15

---

TINED: GNNs-1o-MLPs by Teacher Injection and Dirichlet Energy Distillation
(a) SAGE (spoch=500)
GP
FTN
GP
FT
(b) TINED
(epoch=0)
(c) TINED (epoch-500)
(d) GLNN*
(epach=0)
(e) GLNN* (epoch-500)
Figure 9: Layer-wise t-SNE visualization of the embeddings for various models (GraphSAGE, TINED, GLNN*) and stages (epoch initialization and epoch 500 after training convergence) on Citeseer.
weight decay from [0.0001, 0.001, 0.01], dropout from [0, 0.5, 0.8), hidden dim from [128, 256]. power iteration K from [5, 10, 15). For the students MLP, GLNN and NOSMOG, we set the number of layers and the hidden dimension of each layer to be the same as the teacher GNN, so their total number of parameters stays the same as the teacher GNN. For TINED, TINED+, GLNN" and NOSMOG* given a layer-wise GNN teacher with layers (including, GraphSAGE, GCN, and GAT), in the student MLP we set the same hidden dimension teachers, and 27' layers, while for APPNP, set the student MLP to have T, + 1 layers as explained in Appendix A.3. Other
searching spaces are listed here: Learning rate from [0.0001, 0.0005, 0.001, 0.005, 0.01]. weight de- cay from 0.000,, 0.0005, 0.001, 0.005, 0.01], weight of distillation À from (0.1, 0.4, 0.5, 0.6, 1]. normamlization type from [batch normalization, layer normalization, none], dropout from [0, 0.3, OGB datasets from [512, 1024, 4096]. Weight of DED 3 from [1e-6, 5e - 0.05...... 1,5, 10). Fine tuning weight v for injected teacher FT layers from [0.01, 0.1, 0.5, 1, 10). For the hyperparamrter space of TINED+ from NOSMOG, the search space is the same as (Tian et al., 2022). In large OGB datasets, the direct com-
0.5, 0.5, 0.8). Batch size for two large
16

---

TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation
Table 11: Hyperparameters for GNNs on five datasets from
Table 13: Results of VQGraph and our TINED+ with Graph- SAGE as teacher on transductive setting. The best result is in bold.
(Zhang et al., 2022b; Tian et al., 2022).
Dataset
GraphSAGE
GCN
GAT
layers
2
2
Cora
Citeseer
Pubmed
A-computer
A-photo
hidden dim
128
VQGraph 78.66 1.26 74.661:1.23
73.02#3.51
80,16:2.02 92.32=1.74
learning rate
0.01
0.01
0.01
TINED 83.70=1102 75.39:1.599
77.75:3.14 84.82 94.05:10.39
weight decay
0.0005
0.001
0.01
dropout
O
0.8
0.6
rameter search space of VQGraph is: max epoch E [200, 500], norm type E ["batch", "layer", "none"), hidden dim
fan out attention heads
5,5
8
128, dropout ratio E [0, 0.1, 0.4, 0.5, 0.6], learning rate E [0.01, 5e-3, 3e-3, 1e-3], weight decay E [5e-3, 1e-3, le-4, 0j. lamb soft labels E [0.5, 1], and lamb soft tokens [1e-8,
Table 12: Vary for TINED on large graph ogbn-arxiv under transductive setting.
1e-3, le-1, 1]. The table below reports the results of VQ- Graph and our method TINED+. Observe that our method outperforms VQGraph on the datasets, which validates the effectiveness of TINED.
< Arxiv 64.3110.71 64.3240.91
0.5
>0.8
64.38:0.75
6..44+0.80
OOM
Since VQGraph involves re-training the teacher, whereas all other competitors, including KRD, FF-G2M, GLNN, NOSMOG, and TINED, use fixed teacher for distillation, we believe VQGraph belongs to different category and have therefore excluded from the main result.
putation of DE values will run out of GPU memory 24GB, Thus, we propose sampling ratio Ç to compute approxi- mate DE values in DED process. This is done by inducing a subgraph based on random sampling of edges, then approxi- mate the DE ratio of input and output feature matrix on this sampled subgraph. We also search this sampling proportion Ç from [0.001, 0.005, 0. 0.4, 1]. Moreover, empirically found that applying smoothing term on the DE ratio in MSE loss could benefit optimization, thus we propose smooth- ing function technique μe on DE ratio Rop. i.c. {(Rop), when computing the DED loss, where is searched from (sqrt()) ), log(-), [-en (Jadon al., 2022).
A.8. Impact of ( on Large Datasets
The parameter ( controls the subgraph sampling ratio used to estimate the DE ratio, helping to avoid memory overflow on large datasets. We vary Ç and report the results on the large ogbn-arxiv dataset in Table 12. The results show that TINED maintains stable performance across different { values. Notably, ( values larger than 0.8 are unnecessary, as they increase computational overhead without improving distillation quality.
A.9. Comparison with VQGraph
In this section, we include a comparison with VQ- Graph (Yang et al., 2024) with GraphSAGE teacher. We thoroughly searched all the hyperparameter spaces speci- fied in the original VQGraph paper (Table 12 of its paper). Specifically, VQGraph has the following hyperparameter search space for the teacher GraphSAGE with codebook:
max epoch E [100, 200, 500], hidden dim = 128, dropout ratio E [0, 0.2, 0.4, 0.6, 0.8], learning rate E [0.01, 1e-3, le- 4], weight decay E [le-3, 5e-4, 0]. codebook size E [8192, 16384], lamb node e [0, 0.01, 0.001, 1c-4], and lamb edge [0, 1e-1, 0.03, 0.01, 1e-3]. For distillation, the hyperpa-
17


Reviews:
[{"review_id": "t1bAuH7Fhk", "reviewer": "Reviewer_qs85", "summary": "This work proposes TINED, a method to distill knowledge from teacher GNNs into student MLPs. Extensive experiments show its good performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### We appreciate your effort in reviewing our paper. We have conducted extensive experiments to address your key comments. Thank you for your consideration.\n \n> **C1.** Could the author provide the sensitivity analyses on the datasets mentioned in this paper?\n \n**Response:** In our method, the parameter $\\eta$ controls the degree of fine-tuning in Eq. (6), while $\\beta$ controls the importance of Dirichlet Energy distillation in Eq. (9). We have varied them with  results on  A-computer and Cora datasets in **Tables 4** and **5** of the paper. Below, **Tables A** and **B** present the accuracy results of varying $\\eta$ and $\\beta$ across more datasets. In **Table A**, as $\\eta$ increases across all datasets, TINED exhibits a distinct pattern where performance first improves and then declines, with the best results in italics. This trend underscores the trade off controlled by $\\eta$. A similar pattern is observed for $\\beta$ in Table B. Note that hyperparameter tuning is essential in machine learning research. Appendix A.7 of the paper details the search space for our parameters.\n\n \n**Table A: Vary $\\eta$. Results are averaged over 10 runs with standard deviation**\n|  $\\eta$| cora | citeseer | pubmed | a-computer | a-photo |\n|-------------|------|----------|--------|------------|---------|   \n| 1e-09 | 73.80±2.56 | 67.20±1.86 | 73.39±2.10 | 72.53±2.17 | 86.06±2.38 |\n| 1e-06 | 73.50±2.48 | 67.19±1.87 | 73.32±1.88 | 72.73±1.99 | 86.35±1.79 |\n| 0.001 | 77.12±0.91 | 67.24±1.83 | 73.69±1.83 | 84.13±1.23 | 87.30±2.08 |\n| 0.01 | 79.26±1.30 | 69.38±1.70 | 75.12±2.32 | *85.17±1.21* | 90.48±1.38 |\n| 0.1 | 81.40±1.69 | 71.52±1.58 | 76.55±2.81 | 83.61±1.67 |*93.97±0.54* |\n| 0.5 | 82.01±1.64 | *74.57±1.42* | *77.10±2.15* | 83.23±1.29 | 93.51±0.60 |\n| 1.0 | *82.61±1.58* | 73.57±1.39 | 76.65±2.77 | 82.85±0.91 | 93.37±0.51 |\n| 10.0 | 78.85±1.63 | 73.56±1.48 | 75.61±2.73 | 70.63±4.87 | 89.53±0.78 |\n \n \n \n**Table B: Vary $\\beta$**\n| $\\beta$ | cora | citeseer | pubmed | a-computer | a-photo |\n|-------------|------|----------|--------|------------|---------| \n| 1e-09 | 81.33±1.49 | 73.39±1.28 | 76.63±2.42 | 84.70±1.15 | 93.41±0.61 |\n| 1e-06 | 81.64±1.71 | 73.39±1.30 | 76.28±2.76 | *85.17±1.21* | 93.48±0.65 |\n| 0.001 | 81.64±1.57 | 73.39±1.31 | 76.51±2.64 | 84.80±1.21 | 93.65±0.68 |\n| 0.1 | 81.71±1.59 | 73.65±1.38 | *77.10±2.15* | 84.70±0.99 | *93.97±0.58* |\n| 1.0 | *82.61±1.58* | 73.81±1.29 | 75.84±2.66 | 84.29±1.08 | 86.59±5.66 |\n| 10.0 | 80.92±2.15 | *74.57±1.42* | 71.57±2.76 | 71.35±6.30 | 78.00±3.19 |\n \n \n> **C2.** I am not very familiar with this field but I remember there are many works focusing on distill GNN into MLP. I hope other reviewers check the novelty of this work.\n\n**Response:** In other reviews, the reviewers acknowledged the comprehensive experimental evaluations compared to existing methods, as well as the originality and significance of our methods. Thank you.\n\n> **C3.** Could the author add experiments on heterophilic dataset.\n\n**Response:** As suggested, we have conducted new experiments on representative heterophilic datasets, Squirrel and Amazon-ratings under the production setting. In **Tables C(1)** and **C(2)** with and without graph dependency, our  TINED and TINED+ outperform existing approaches under all settings, often with a significant margin.  \nNote that our work, as well as existing methods, does not explicitly focus on graphs with heterophily. As future work, we plan to conduct an in-depth investigation and develop dedicated techniques to further improve the effectiveness of heterophilic GNN distillation.   \n\n   \n**Table C(1): *prod* (*ind* & *tran*) setting on heterophilic dataset (without graph dependency)**\n | Data | Eval | SAGE | KRD |FFG2M|GLNN | GLNN* | TINED |\n|-------|-----|------|- |-|------|-------|--------|\n| Squirrel | *prod* | 35.47 |38.34 |37.16| 39.90 | 39.70 | **41.95** |\n| | *ind* | 41.44±4.66 |42.00±4.78 |42.11±4.77| 44.89±5.67 | 45.00±4.56 | **46.89±5.23** |\n| | *tran* | 33.98±1.66 |37.43±3.34 |35.93±3.36| 38.65±1.09 | 38.37±1.02 | **40.72±1.36** |\n| Amazon-ratings | *prod* | 47.55 | 50.33 |49.56| 49.87 | 49.41 | **50.70** |\n| | *ind* | 47.45±1.48 |47.55±0.97 |47.68±1.00| 47.72±1.00 | 47.71±1.14 | **49.02±1.02** |\n| | *tran* | 47.58±0.48  | 51.03±1.73   |50.03±1.50| 50.41±0.45 | 49.84±0.39 |**51.12±0.44** |\n\n**Table C(2): *prod* (*ind* & *tran*) setting on heterophilic dataset (with graph dependency)**\n | Data | Eval | NOSMOG |NOSMOG* | TINED+ |\n|-------|-----|---------|--------|-------|\n| Squirrel | *prod* | 38.17 | 39.43 | **40.89** |\n| | *ind* | 45.44±4.75 | 44.33±3.24 | **46.78±3.80** |\n| | *tran* |  36.35±1.69 |38.20±1.59 | **39.42±1.05** |\n| Amazon-ratings | *prod* | 47.86 | 48.80 | **50.42** |\n| | *ind* |47.47±1.45 | 48.46±1.37 | **49.51±1.54** |\n| | *tran* | 47.96±0.31 | 48.88±0.56 | **50.65±0.56** |", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "No, but the author seems to provide the code. I do not check that.", "relation_to_broader_scientific_literature": "na", "essential_references_not_discussed": "I am not familiar with this field.", "other_strengths_and_weaknesses": "pros: \n1. well-written and easy to follow (figures are clear)\n2. The topic (GNN distillation) is promising from a practical perspective.\n3. The idea is easy and interesting,\n\ncons:\n1. I doubt that the method relies highly on hyperparameters. Could the author provide the sensitivity analyses on the datasets mentioned in this paper?\n2. I am not very familiar with this field but I remember there are many works focusing on distill GNN into MLP. I hope other reviewers check the novelty of this work.\n3. The results seem to rely on homophilic datasets in this paper. Could the author add experiments on heterophilic dataset( eg. Amazon-Ratings, squirrel, pokec, etc).", "other_comments_or_suggestions": "na", "questions_for_authors": "See cons.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oZNBlBKlqE", "reviewer": "Reviewer_bbbm", "summary": "This paper introduces TINED, a method for distilling Graph Neural Networks (GNNs) into Multi-Layer Perceptrons (MLPs) via layer-wise Teacher Injection and Dirichlet Energy Distillation (DED). The key idea is to directly inject parameters from GNN feature transformation layers into MLP layers and use DED to preserve opposing smoothing effects of GNN operations, specifically feature transformation (FT) and graph propagation (GP). Theoretical bounds on approximating GP with MLP layers are also provided. Experiments conducted on seven datasets demonstrate that TINED outperforms both original GNN models and prior distillation methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for recognizing the strengths of our work. Here are our responses to your important comments.\n\n> **C1.** Elaboration on deployment with and without graph dependency (line 185-190)\n\n**Response:** When performing inference on a new node with limited connections to a graph, such as a new user joining a social network or a new product being added to an online shopping platform, the node has restricted access to the existing graph structure. In these cases, the graph structure is unavailable to the node  (without graph dependency), so inference relies solely on node features. Conversely, when conducting inference on nodes that have sufficient connections within the graph structure (with graph dependency), it is possible to leverage both node features and graph features. This allows for a graph-dependent inference approach, utilizing the rich information available from the node's connections within the graph. \n\n   \n> **C2.** Generalize to student MLP with various layer counts.\n\n**Response:** Thank you for the insightful comment. As shown in Appendix A.3, our approach for decoupled GNNs, such as APPNP in Eq (17), consolidates all graph propagation (GP) operations into a single FC layer in MLP, which can be easily extended to multiple layers. For traditional GNNs like SAGE, our design focuses on capturing layer-to-layer GNN knowledge within MLPs through the proposed techniques, Teacher Injection and Dirichlet Energy Distillation. Consequently, the number of layers in our method is related to the teacher model. One potential way for generalization is to explore layer fusion techniques, where multiple teacher layers are compressed into fewer student layers while maintaining their overall behavior. We will include this discussion in the revised paper.\n \n\n> **C3.** The performance of TINED on heterophilous graphs remains unexplored.\n\n**Response:** Please find the experiments on heterophilous graphs (Squirrel and Amazon-ratings) in the **response to comment C3 of Reviewer qs85**. The results show that TINED and TINED+ outperform existing approaches. As future work, we plan to develop dedicated techniques to further improve the effectiveness of heterophilic GNN distillation. \n \n\n\n> **C4.**  (Teacher Injection Mechanism) How to handle when teacher is suboptimal.\n\n**Response:** We clarify that the goal of knowledge distillation is to create a student model that closely mimics the teacher model, while whether the teacher model is suboptimal or not is not the focus. After injecting teacher parameters on FT operations into the student, the MLP layers in the student are further fine-tuned in Eq. (6) and trained with Dirichlet Energy Distillation and the overall loss in Eq. (8,9). These distillation training steps on the student MLP also mitigates the mentioned risk, and improves student performance. \n\n> **C5.** (Distillation of parameterized GP operations) Explanation on the handling of the parameterized GP operations.\n\n**Response:** As shown in Eq. (4) in Section 4.1 and Eqs. (15, 16, 17) in Appendix A.3, different GNNs have distinct GP operations, such as GAT with attention. Therefore, for these various GP operations, we choose to distill them into fully-connected (FC) layers in MLP, without directly using these GP parameters, but employing carefully designed techniques including layer-wise Dirichlet Energy distillation. On the other hand, the FT operations in different GNNs share a similar formulation that is compatible with FC layers in MLPs, as shown in Eq. (4) and Appendix A.3. Thus, we can directly inject the parameters of FT operations into the student model.\n \n  \n> **C6.** The impact of $\\zeta$ on large datasets to approximate DE ratio\n\n**Response:**  The parameter $\\zeta$ controls the subgraph sampled to estimate the DE ratio, helping to avoid running out of memory on large datasets. We vary $\\zeta$ and report the results on the large ogbn-arxiv dataset in **Table A** below. Observe that our TINED maintains stable performance as $\\zeta$ changes, and we do not need to consider $\\zeta$ larger than 0.8, to reduce computational overhead, while maintaining distillation quality.\n\n**Table A: Vary $\\zeta$ on large graph ogbn-arxiv**\n|$\\zeta$|0.1|0.3|0.5|0.7|>0.8| \n|-|-|-|-|-|-|\n|ogbn-arxiv|64.31±0.71|64.32±0.91|64.38±0.75|*64.44±0.80*|OOM|\n \n \n> **C7.** The implementation of the \"with graph structure\" version and NOSMOG.\n\n**Response:** The phrase \"with graph structure\" refers to conducting inference on nodes that have sufficient connections within the graph, allowing the use of both node features and graph features. In this setting, we use DeepWalk node embeddings for observed nodes, and for unobserved nodes, we use the average positional encodings of their observed neighbors as their encoding, consistent with the ways in NOSMOG. We will include this clarification in the paper.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence\nThe authors present the following key claims:\n\n1. \"The valuable knowledge of a teacher GNN is preserved in the well-trained parameters of its FT and GP operations.\" (line 50-53). This is substantiated by the ablation study (Table 6) and main benchmark results (Tables 1 and 2). \n2. \"The FT and GP operations in a GNN layer often exert opposing smoothing effects: GP aggressively smooths node embeddings, while FT is more restrained and can even diversify embeddings.\" (line 84-87). This is validated through DE ratio analysis (Figures 1 and 7).", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem at hand. \nThe authors develop their methodology based on a systematic analysis of the GNN2MLP distillation problem. Their approach effectively addresses the identified challenges: preserving knowledge in teacher layers and managing the disparity between the FT and GP operations.\nThe evaluation employs widely recognized benchmarks to assess the performance of the proposed method. \nThe authors distinguish between two inference settings: graph-dependent and graph-free. This distinction corresponds to two practical scenarios: warm start and cold start, respectively. In cold start scenarios (where new nodes have no edges), graph information is inaccessible during inference. Conversely, in warm start scenarios (where new test nodes have edges), limited graph information can be utilized during inference, as demonstrated by [1]. While the authors provide a brief discussion of this distinction (line 185-190), a more comprehensive elaboration would be beneficial.\n\n[1] Tian, Y., Zhang, C., Guo, Z., Zhang, X., & Chawla, N. (2023). Learning MLPs on graphs: A unified view of effectiveness, robustness, and efficiency. In International Conference on Learning Representations.", "theoretical_claims": "The theoretical claim regarding the bound on the approximation error of using MLPs to approximate GP (Theorem 4.1) has been verified and found to be mathematically sound.", "experimental_designs_or_analyses": "Yes, I have checked the soundness of the experimental designs and analyses.\nThe comprehensive evaluation framework incorporates both transductive and inductive/production settings (Tables 1 and 2). \nThe authors present a thorough set of experimental results, including: ablation studies (Table 6), TSNE visualization of learned embeddings (Figure 5), approximation errors of MLPs (Table 3), inference time comparisons (Figure 3), learned DE ratio analysis (Figure 6), parameter sensitivity analysis (Tables 4 and 5), and performance evaluations across different teacher models (Figure 4).", "relation_to_broader_scientific_literature": "The work is appropriately situated within the existing literature:\n\n1. The GNN2MLP pipeline and evaluation benchmarks build upon established research from previous works [1] and [2].\n2. The concept of distilling layer-wise teacher structures has been explored in NLP and CV literature. The authors extend this approach to the GNN2MLP distillation domain. \n3. The analysis of smoothing effects using Dirichlet energy was introduced in [3], where it was applied to entire non-decoupled GNN layers. This work makes a novel contribution by separately analyzing the FT and GP operations within GNN layers and leveraging their distinct smoothing properties to enhance the distillation process. \n\n[1] Zhang, S., Liu, Y., Sun, Y., & Shah, N (2022). Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation. In International Conference on Learning Representations.\n[2] Tian, Y., Zhang, C., Guo, Z., Zhang, X., & Chawla, N. (2023). Learning mlps on graphs: A unified view of effectiveness, robustness, and efficiency. In International Conference on Learning Representations.\n[3] Rusch, T. K., Bronstein, M. M., & Mishra, S. (2023). A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10993.", "essential_references_not_discussed": "The research by Winter et al. [1] examines the underlying mechanisms contributing to the effectiveness of distillation methods. Including this work in the discussion would provide additional context and insights.\n\n[1] Winter, D., Cohen, N., & Hoshen, Y. (2024). Classifying nodes in graphs without GNNs. arXiv preprint arXiv:2402.05934.", "other_strengths_and_weaknesses": "Strengths:\n- Originality: The investigation into GNN's internal layer structure represents a novel contribution. The application of Dirichlet energy to analyze the intrinsic properties of different components within GNN layers is particularly innovative.\n- Significance: The identification of opposing smoothing effects between FT and GP operations in GNN layers constitutes an important finding. This insight not only enhances distillation efficiency but also contributes to the broader understanding of GNN architectures, as most GNN models can be analyzed within this framework.\n- Clarity: The manuscript is well-written, with the methodology presented in a clear and systematic manner.\n- Soundness: The authors provide comprehensive experimental evidence to support their claims.\n      \nWeaknesses:\n\n1. The approach requires fixed layer correspondence between teacher and student models. TINED appears unable to generalize to scenarios where teacher and student architectures have significantly different layer counts (e.g., distilling a 4-layer GNN into a 2-layer MLP). An extension of TINED to address this limitation would be valuable.\n2. The experimental evaluation focuses exclusively on homogeneous graphs. The performance of TINED on heterophilous graphs remains unexplored.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "The following questions merit consideration:\n\n1. (Teacher Injection Mechanism): The direct injection of FT parameters from GNNs into MLP layers raises concerns about potential overfitting of the student MLP to the teacher's initialization, particularly if the teacher model is suboptimal. How does the proposed approach address this risk?\n2. (Distillation of parameterized GP operations): The handling of parameterized GP operations, such as GAT's attention module, warrants further explanation. Can this teacher knowledge be effectively transferred to the student MLP? This represents an important consideration, as GNN knowledge resides not only in FT layers but also in the parameters of GP operations.\n3. (DED on large graphs): For large-scale graphs such as Products, subgraph sampling (\\zeta) is employed to approximate Dirichlet Energy. What is the impact of \\zeta on the fidelity of DE ratio calculations?\n4. (TINED+): The description of the \"with graph structure\" version of TINED (TINED+) lacks clarity. Does this implementation simply incorporate NOSMOG techniques on top of the base TINED architecture?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GjISCEUiSA", "reviewer": "Reviewer_JZCk", "summary": "This paper addresses the GNN2MLP distillation task, which aims to transfer knowledge from computationally expensive Graph Neural Networks (GNNs) to more efficient Multilayer Perceptrons (MLPs) for faster inference on graph-structured data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for acknowledging the strengths of our work. Below are our responses addressing your important comments.\n \n> **C1.** The hyperparameter search space. \n\n**Response:** We clarify that we mainly adopt the conventional hyperparameter search space for the teacher model and the student model, following existing studies. For the hyperparameters of our method, we also employ grid search within a limited space. Detailed information about the hyperparameters is provided in the appendix.\n\n \n> **C2.** Compare with vanilla MLP with the same number of layers as TINED.\n\n**Response:** As suggested, in addition to the vanilla MLP with 2 layers, we also compare with a 4-layer MLP, denoted as MLP*. In **Table A** and **B** below, under both transductive and production settings, our method, TINED, outperforms MLP* across all datasets. Moreover, compared to the MLP with 2 layers, MLP* shows degraded performance, indicating that additional layers cause the MLP to overfit on these datasets.\n \n**Table A: Compare with MLP\\* with 4 layers under transductive setting. The results are averaged over 10 runs and reported with standard deviation** \n|   | cora | citeseer | pubmed | a-computer | a-photo |\n|-|-|-|-|-|-|\n| MLP | 60.84±1.08 | 63.41±1.96 | 69.41±2.88 | 70.07±1.77 | 80.19±1.48 |\n| MLP\\* | 58.20±3.16 | 60.81±3.48 | 67.61±1.51 | 66.84±3.03 | 78.66±2.48 |\n|TINED |82.63±1.57| 74.43±1.53 | 77.09±2.14 | 85.18±1.12 | 93.97±0.53|\n\n**Table B: Compare with MLP\\* with 4 layers under *prod*(*ind*&*tran*) setting**\n|   | setting | cora | citeseer | pubmed | a-computer | a-photo |\n|-|-|-|-|-|-|-|\n| MLP | *ind* | 61.31±2.16 | 63.95±2.95 | 69.66±2.68 | 70.36±2.48 | 79.76±2.00 |\n| | *tran* | 60.88±1.41 | 62.99±2.39 | 69.67±2.61 | 69.92±2.03 | 79.53±2.05 |\n| MLP\\* | *ind* | 58.67±2.42 | 62.15±3.50 | 67.76±1.88 | 68.09±2.60 | 77.27±2.17 |\n| | *tran* | 58.12±1.81 | 61.46±2.46 | 68.07±1.87 | 67.90±2.41 | 77.05±2.48 |\n|TINED | *ind* | 74.38±1.28 | 72.68±1.97 | 75.64±3.02 | 82.83±1.45 | 91.96±0.72|\n| | *tran* |  80.04±1.50 | 72.20±1.66 | 75.83±2.81 | 84.87±1.38 | 93.74±0.51|\n\n\n\t\n> **C3.** Clarify fine-tuning with $\\eta$ (Equation 6).\n\n**Response:** In Eq (6), the parameter $\\eta$ controls the balance between the inherited teacher knowledge and learned parameters during fine-tuning. A small $\\eta$ tends to let the student make less changes to the inherited teacher parameters, and as $\\eta$ increases, the fine-tuning will update the parameters aggressively. As shown in **the experiments in the response to comment C1 of Reviewer qs85**, when varying $\\eta$, a mild $\\eta$ setting can strike a good balance, leading to better performance.\n\n\n \n> **C4.** This work focuses primarily on performance optimization rather than efficiency.\n  \n**Response:** Our goal is to achieve better effectiveness than existing distillation methods, ensuring that the student MLP operates significantly faster than the teacher GNN, while maintaining comparable efficiency to existing methods. As shown in Figure 3 of the paper, our methods TINED and TINED+ achieve the best trade-off between accuracy and inference time. For example, on CiteSeer, while the teacher requires *153.14 milliseconds (ms)* for inference, our methods, TINED and TINED+, achieve the highest accuracy at the cost of just *1.63 ms*. All the distillation methods maintain the same order of efficiency, around *1-2 ms*.\n\n\n \n> **C5.** Can TINED effectively handle dynamic graphs? If not, what modifications would be necessary? \n\n**Response:** In this work, we focus on static GNNs, but we agree that distillation on dynamic GNNs is a promising direction. We believe that the proposed Teacher Injection (TIN) and Dirichlet Energy Distillation (DED) techniques are compatible with dynamic GNNs, such as DynGNN, EvolveGCN, and TGAT. For dynamic GNNs, TIN can be adapted to periodically inject temporal parameters from the teacher into the student, and DED can be performed adaptively since the DE ratio can be efficiently calculated on new graph snapshots. We leave a detailed investigation of these adaptations for future work.\n\n\n> **C6.**  How does TINED ensure that the computed DE ratios accurately reflect the intended smoothing/diversification effects of specific operations?\n\n**Response:** We clarify that the operation considered by the DE ratio in Definition 4.3 is the GP operation in Eq. (4), which performs aggregation and concatenation, excluding subsequent softmax and batch normalization. Our formulation precisely controls the computation boundary to prevent conflation with other factors, ensuring that the DE ratio accurately reflects only the smoothing/diversification effect of the GP operation itself. DE is a widely adopted measure in the literature for assessing the smoothing effect of GNNs [1], making it a reasonable choice for designing our DE ratio. \n\n[1] T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. 2023.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Overall, the evaluation makes sense. But the hyperparameter search space is too broad.", "theoretical_claims": "Overall, the theoretical results look good to me.", "experimental_designs_or_analyses": "The reviewer has checked Section 5. The comparison with vanilla MLPs is unfair since TINED's MLP has more layers.", "relation_to_broader_scientific_literature": "The contributions of this work are related to GNN-to-MLP distillation and Dirichlet energy analysis.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "# Strengths:\nThis work effectively categorizes the key components of a GNN layer into two distinct parts based on their operational patterns: feature transformation and message passing. For feature transformation, which shares the same structural knowledge with student MLPs, the authors directly transfer the parameters. For message passing, which constitutes the core functionality of GNNs, the authors identify its distinct smoothing effect compared to feature transformation. \n\n# Weaknesses:\n1. The term \"fine-tuning\" (Equation 6) is ambiguously defined. The paper lacks clarity on how the parameter $\\eta$ balances between inherited and learned parameters, which requires further elaboration.\n2. This work focuses primarily on performance optimization rather than efficiency. The inference time of TINED does not show any improvement compared to other distillation baselines.", "other_comments_or_suggestions": "There are typos in the manuscript. For example: \"nornamlization\" and \"hyperparamrter\".", "questions_for_authors": "Can TINED effectively handle dynamic graphs in which node features or graph structure evolve over time? If not, what modifications would be necessary to accommodate such scenarios?\nThe Dirichlet Energy ratio of an operation quantifies smoothing effects but may potentially conflate multiple factors, such as softmax functions and batch/layer normalization between layers. How does TINED ensure that the computed DE ratios accurately reflect the intended smoothing/diversification effects of specific operations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z3LO6IJ0JR", "reviewer": "Reviewer_cPDP", "summary": "This paper proposes a layer-wise method with Dirchlet energy imitation for node-level knowledge distillation from GNN teacher to MLP student, to reduce the latency of inference for time-sensitive applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Thank you for appreciating the strengths of our work. Below are our responses to address your important comments.\n\n\n>**C1.** The justification of using Dirchlet energy ratio as the quantity for knowledge transfer, and the reason why it is more effective than previous SOTA methods.\n\n**Response:** Thank you for the insightful comment. Please find our justifications below. (i) The goal of distillation is to enable the student MLP to preserve the GNN teacher's knowledge as much as possible. Our analysis reveals a crucial finding: the Feature Transformation (FT) and Graph Propagation (GP) operations in a GNN layer often exhibit opposing smoothing effects. GP tends to be aggressive, while FT is more restrained in smoothing, as shown in Figure 1 of the paper. These distinct smoothing patterns represent important teacher knowledge that the student should preserve. We achieve this by leveraging the proposed Dirichlet Energy Distillation (DED), where we use the well-recognized smoothness measure, Dirichlet Energy, to design the DE ratio. This DED technique is validated to be effective in the experiments. (ii) Moreover, in our design, the parameters of FT operations in teacher GNN have been directed injected into the student by the teacher injection technique (TIN), while the GP operations are not compatible with the FC layers in student MLP, and the DED technique with Dirichlet Energy ratio helps on the distillation of GP operations. Therefore, *our methods, combining TIN and DED, work together* to achieve better performance than existing methods, as validated by extensive experimental results and ablation studies. \n \n>**C2.** I believe the proof is correct. The error bound can be loose on graphs with node degree skewed.\n \n \n**Response:** Thank you for the insightful comment. As stated in our paper, the value of our analysis lies in its attempt to establish a theoretical relationship for using simple MLP layers to approximate complex GNNs. Our theoretical analysis focuses on the general case across all possible graphs, with the error being bounded by the largest eigenvalue. We acknowledge that this bound can be large in scenarios where the node degree distribution is skewed. In experiments, the observed actual error is typically much smaller than the theoretical result, as validated in Table 3 of the paper. \n\n \n \n \n \n  \n\n\n> **C3.** The experiments are comprehensive, and most recent baselines are included. Compare with a recent baseline VQGraph, which has been mentioned in the related work section.\n \n**Response:** Thank you for your feedback on the comprehensiveness of our experiments. As suggested, we have included a comparison with VQGraph. We thoroughly searched all the hyperparameter spaces specified in the original VQGraph paper (Table 12 of VQGraph). Specifically, VQGraph has the following hyperparameter search space for the teacher SAGE with codebook: max_epoch $\\in$ \\{100, 200, 500\\}, hidden_dim $=$ 128, dropout_ratio $\\in$ \\{0, 0.2, 0.4, 0.6, 0.8\\}, learning_rate $\\in$ \\{0.01, 1e-3, 1e-4\\}, weight_decay $\\in$ \\{1e-3, 5e-4, 0\\}, codebook_size $\\in$ \\{8192, 16384\\}, lamb_node $\\in$ \\{0, 0.01, 0.001, 1e-4\\}, and lamb_edge $\\in$ \\{0, 1e-1, 0.03, 0.01, 1e-3\\}. For distillation, the hyperparameter search space of VQGraph is: max_epoch $\\in$ \\{200, 500\\}, norm_type $\\in$ \\{“batch”, “layer”, “none”\\}, hidden_dim $=$ 128, dropout_ratio $\\in$ \\{0, 0.1, 0.4, 0.5, 0.6\\}, learning_rate $\\in$ \\{0.01, 5e-3, 3e-3, 1e-3\\}, weight_decay $\\in$ \\{5e-3, 1e-3, 1e-4, 0\\}, lamb_soft_labels $\\in$ \\{0.5, 1\\}, and lamb_soft_tokens $\\in$ \\{1e-8, 1e-3, 1e-1, 1\\}. The table below reports the results of VQGraph and our method TINED+. Observe that our method outperform VQGraph on the datasets. We will include this comparison in the paper.\n\n**Table A** Results of VQGraph and our TINED+ with SAGE as teacher on transductive setting. The best result is in italics.\n||Cora|Citeseer|Pubmed|Computer|Photo| \n|-|-|-|-|-|-|\n|VQGraph|78.66|74.66|73.02|80.16|92.32|\n|TINED+|*83.70*|*75.39*|*77.75*|*84.82*|*94.05*|", "claims_and_evidence": "The problem is well-defined, and the claims in introduction section is clear, and supported by experimental results.", "methods_and_evaluation_criteria": "The layer-wise distillation method is straight-forward, and make sense, but why the Dirchlet energy ratio is a suitable quantity for distillation need more justifications.", "theoretical_claims": "I believe the proof is correct, however, the error bound can be loose since $\\lambda_{max}$ can be a large quantity especially for graphs with node degree skewed, making the theoretical result not very valuable.", "experimental_designs_or_analyses": "The experimental design is detailed and comprehensive, however it lack a more recent baseline VQGraph, which has been mentioned in the related work section.", "relation_to_broader_scientific_literature": "Compared to previous approaches, this paper proposes layer-wise distillation using energy ratio for knowledge trasnfer.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: The experiments are comprehensive, and most recent baselines are included. This work proposes layer-wise distillation based on Dirchlet energy ratio, which quantifies the smoothness of the node embeddings.\n\nWeaknesses: \n\n1. The use of Dirchlet energy ratio is not well justified, i.e., why it is appropriate for knowledge trasnfer, what makes it effective to outperform previous methods?\n\n2. The improvement compared to previous baselines are not very significant, and some baselines are missing (e.g., VQGraph)", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The biggest concern is the justification of using Dirchlet energy ratio as the quantity for knowledge transfer, and the reason why it is more effective than previous SOTA methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of distilling a trained GNN into a cheaper (and often even node-wise) MLP, in order to accelerate downstream inference. In contrast to some prior approaches, the authors perform this layer-by-layer; for every GNN layer, graph propagation is replaced by an MLP regularized with Dirichlet Energy Distillation, and the fully-connected feature transformation is copied over verbatim and then fine-tuned end-to-end to match soft labels from the GNN teacher. This framework is then benchmarked across various datasets, showing excellent results, often outperforming the original GNN.

After the rebuttal period, all four reviewers voted to accept this work, two of them even making a strong recommendation to do so. Reviewers generally praised a well-motivated design, deep analysis, and above all exceptionally good empirical performance. While TINED-produced models are not faster than those obtained from other distillation methods, this is expected as all of those produce very simple models with already rather minimal inference time requirements. Finally, reviewers were also generally happy about the breadth of experiments and reasonable choice of up-to-date baselines.

Given that all four reviewers voted to accept this work, and some championed it, I am inclined to side with the unanimous reviewer decision, and also recommend acceptance.
Decision: Accept (poster)