Double Wins: Boosting Accuracy and Efficiency of Graph Neural Networks by Reliable Knowledge Distillation
Authors: Qiaoyu Tan
Daochen Zha
Soo-Hyun Choi
Li Li
Rui Chen
Xia Hu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Neural Networks
Reliable Knowledge Distillation
Model Inference Acceleration

Abstract: The recent breakthrough achieved by graph neural networks (GNNs) with few labeled data accelerates the pace of deploying GNNs on real-world applications. While several efforts have been made to scale GNNs training for large-scale graphs, GNNs still suffer from the scalability challenge of model inference, due to the graph dependency issue incurred by the message passing mechanism, therefore hindering its deployment in resource-constrained applications. A recent study~\citep{zhang2021graph} revealed that GNNs can be compressed to inference-friendly multi-layer perceptrons (MLPs), by training MLPs using the soft labels of labeled and unlabeled nodes from the teacher. However, blindly leveraging the soft labels of all unlabeled nodes may be suboptimal, since the teacher model would inevitably make wrong predictions. This intriguing observation motivates us to ask: \textit{Is it possible to train a stronger MLP student by making better use of the unlabeled data?} 

This paper studies cross-model knowledge distillation - from GNN teacher to MLP student in a semi-supervised setting, showing their strong promise in achieving a ``sweet point'' in co-optimizing model accuracy and efficiency. Our proposed solution, dubbed \textit{Reliable Knowledge Distillation for MLP optimization} (\textbf{RKD-MLP}), is the first noise-aware knowledge distillation framework for GNNs distillation. Its core idea is to use a meta-policy to filter out those unreliable soft labels. To train the meta-policy, we design a reward-driven objective based on a meta-set and adopt policy gradient to optimize the expected reward. Then we apply the meta-policy to the unlabeled nodes and select the most reliable soft labels for distillation. Extensive experiments across various GNN backbones, on 7 small graphs and 2 large-scale datasets from the challenging Open Graph Benchmark, demonstrate the superiority of our proposal. Moreover, our RKD-MLP model shows good robustness w.r.t. graph topology and node feature noises. The code is available at \url{https://anonymous.4open.science/r/RKD-MLP-F2A6/}.

Under review as a conference paper at ICLR 2023
DOUBLE WINS: BOOSTING ACCURACY AND EFFI- CIENCY OF GRAPH NEURAL NETWORKS BY RELIABLE KNOWLEDGE DISTILLATION
Anonymous authors Paper under double-blind review
ABSTRACT
The recent breakthrough achieved by graph neural networks (GNNs) with few labeled data accelerates the pace of deploying GNNs on real-world applications. While several efforts have been made to scale GNNs training for large-scale graphs, GNNs still suffer from the scalability challenge of model inference, due to the graph dependency issue incurred by the message passing mechanism, therefore hindering its deployment in 20226) revealed that GNNs can be compressed to inference- inference-friendly multi-layer perceptrons (MLPs), by training MLPs using the soft labels of labeled and unlabeled nodes from the teacher. However, blindly leveraging the soft labels all unlabeled nodes may be suboptimal, since the teacher model would inevitably make wrong predictions. This intriguing observation motivates us to ask: Is it possible to train a stronger MLP student by making better use of the unlabeled data?
applications. A recent study (Zhang al.
This paper studies cross-moce moss- knowledge distillation from GNN teacher to ML.P student in semi-supervisee setting, showing their strong promise in achieving "sweet point" in co-optimizing model accuracy and efficiency. Our proposed solu- tion, dubbed Reliable Knowledge Distillation for MLP optimization (RKD-MLP), is the first aware knowledge distillation framework for GNNs distillation. Its core idea is to use meta-policy filter out those unreliable soft labels. To train the meta-policy, we design a reward-driven objective based on a meta-set and adopt policy gradient to optimize the expected reward. Then we apply the meta- policy to the unlabeled nodes and select the most reliable soft labels for distillation. Extensive experiments across various GNN backbones, on small graphs and 2 large-scale datasets from the challenging Open Graph Benchmark, demonstrate the superiority of our proposal. Moreover, our RKD-MLP model shows good robustness w.r.t. graph topology and node feature noises. The code is available at Iattps://anonynet 4open. www.sceeeeeeeeeeeeeee MLP F2A67
INTRODUCTION
Graph neural networks (GNNs), as the de facto neural architecture in graph representation learn- ing (Zhou et al., 2020 Hamilton et al. 2017b), have achieved state-of-the-art results across variety
of applications, such as node classification (Kipf & Welling 2016 Liu et al. 2020). graph classi-
fication (Ying et al. 2018 Gao Jil 2019). link prediction (hang Chen 2018 Zhang al.)
2021), and anomaly detection (Deng & Zhang 2021; Chaudhary et al. 2019). Different from plain network embedding methods (Perozzi et al., 2014, Grover & Leskoved (0166), GNNs rely
on the convolution-lia -ike message propagation mechanism (Gilmer et al. (017]) recursively aggre- gate messages from neighboring nodes, which are believed to improve model expressiveness and representation flexibility (Xu et al. (0188).
Despite the recent advances. GNNs are still facing several challenges during inference, especially when going deeper (Chen et al. 2020 202T) and applying to large-scale graphs (Chiang et al., 2019 Zeng et al. 2019).. The major reason (Abadal et al. (202T) y is that the message propagation among neighbors from multi-hops always incurs heavy data dependency, causing substantially computational

---

Under review as a conference paper at ICLR 2023
costs and memory footprints. Some preliminary efforts attempt fill the gap from different aspects. For example, (Zhou et al. 2021) proposes to accelerate inference via model pruning, and (Tailor et al. 2020) suggests to directly reduce computational costs by weight quantization. Although they can speed up GNNs to some extent, the improvements are rather limited, since the data dependency issue remains unresolved. Recently, GLNN (Zhang et al.] (022b) tries to tackle this issue by compressing GNNs to inference-friendle multi-layer perceptrons (MLPs) via knowledge distillation (KD). Similar to standard KD protocols (Hinton al., 2015), GLNN trains the MLP student by using the soft labels from GNN teacher as guidance, and then deploys the distilled MLP student to conduct lonency-constrraneed inference.
However, directly leveraging soft labels from the GNN teacher is suboptimal when the labeled nodes are scarce, a common scenario in graph-structured data (Kipf & Welling 2016 Garcia & Bruna 2017 Feng et al. 2020). This is mainly because large portion of unlabeled nodes will be incorrectly predicted by GNNs due to its limited generalization ability. For instance, many GNN variants (Kipf) & Welling 2016 Veličković et al. 2017 Klicpera et al. 2018) can achieve 100% accuracy on the training set, yet their test accuracy is merely around 80% on Planetoid benchmarks. As a result, the soft labels of those wrongly predicted unlabeled nodes would introduce noises to the optimization landscape of the MLP student, leading to an obvious performance gap w.r.t. the GNN teacher (Zhang al. 2022b).
To avoid the influence of mislabeled nodes, the common practice is to analyze their logit distributions from the teacher model (Kwon al.) 2020) Zhu et al. 2021a /hhnn al. (2022a). For example, Zhang et al. (Zhang et al.) (022aa) propose to assign larger weights to samples if their teacher predictions are close one-hot labels. Zhu et al. (Zhu al.) 2021a) suggest filtering out data points if their teacher predictions mismatch with ground truth labels. Nevertheless, these methods cannot be applied in real-world graphs where node labels are expensive to access. Recently, Kwon et al. (Kwon et al. 2020) suggest discriminating samples based on entropy values, by assuming that teacher predictions with lower entropy are more reliable. However, we found that entropy values are ineffective to distinguish the correct and wrong decision boundaries of GNN models since they are often largely overlapped, as we show in Figure [ (right panel). Therefore, it still remains an open challenge to effectively distill semi-supervise GNN models to wight MLP students.
Present Work. Motivated by this, we propose a novel KD framework - RKD-MLP to boost the MLP student via noise-aware distillation. It is noteworthy that while we focus on the MLP student for efficiency purposes, our solution is ready for other student types, such as GNNs (See Appendix [F) for more discussion). Specifically, RKD-MLPP uses meta-policy to filter out those unreliable soft labels by deciding whether each node should be used in distillation given its node representations. The student then only distills the soft labels of the nodes that are kept by the meta-policy. To train the meta-policy, we design reward-driven objective based on meta-set, where the meta-policy is rewarded for making correct filtering. The meta-policy is optimized with policy gradient achieve the best expected reward and then be applied to unlabeled nodes. We iteratively update the meta- peea--ooiccy and the student model, achieving win-win scenario: it substantially improves the performance of the vanilla MLP student by teaching it with reliable guidance while maintaining the inference efficiency of MLPs without increasing the model size.
We provide the first comprehensive investigation of unlabeled nodes in GNNs distillation by demonstrating its validity in boosting the MLP student via providing effective pseudo labels, and perniciousness in degrading model performance via adding incorrect soft labels.
Motivated by our analysis, we propose to use a meta-policy to filter out unreliable nodes whose soft labels are wrongly predicted by the GNN teacher, and introduce bi-level optimization strategy to jointly train the meta-policy and the student model.
Extensive experiments over a variety of GNN backbones on small datasets and challeng- ing OGB benchmarks demonstrate the superiority of our proposal. Notably, our RKD-MLP outperforms the vanilla KD solution with up to 5.82% standard accuracy, while its inference is at least 100 times faster than conventional GNNs.
2

---

Under review as a conference paper at ICLR 2023
opteano
67.45
1.75
64.95
62.4762.65
Accuracy
26,055634
FUS
0.89 3o
GJVV
ogbn product
(errgy
The influence of unlabeled nodes on vanilla solution-GINN (Zhang et al. 2022b). GLNN-label is variant of GLNN by excluding unlabeled nodes. Middle: The impacts of wrongly predicted nodes on the MLP student under different noise ratios. Right: Entropy distributions of wrongly and correctly predicted nodes by GNN teacher. More curves are reported in Appendix E
Figure 1: Left:
MOTIVATION
2.1 PRELIMINARIES
Notations. Let G = (V, E) be graph with N nodes, where V and & stand for the node set and edge set, respectively. We use X e RxD to denote node features, with row x, being the D-dimensional feature vector of node where yv RC represents the one-hot label vector of node v. In semi- supervise learning, which is a common task in graph analysis, we have a small portion of nodes being labeled while the majority of the nodes are unlabeled. We mark labeled nodes with superscript", i.e., VL, X and Y and unlabeled nodes with superscript" i..., D X and Y
V. We denote Y E IRN as the label matrix with C' classes of interest,
Graph Neural Networks. GNNs are state-of-ther thrther neural network architectures for semi supervsueee classification in graph analysis. Following the message passing mechanism (Gilmer et al., (0177), the core idea is to update an anchor node's representation by aggregating representations of itself and its neighbors. Formally, the k-th layer, the updating rule is expressed as:
h (k) COM(h AGG((h
(k-1) No))),
(1)
where (*) E IRd denotes the representation of node e at the k-th layer, and N the neighbor set
of v. The aggregation function AGG() aims aggregate messages from neighbors via either fixed
combinations determined by the graph topology (Kipf & Welling 2016: (ul6//Wu et al., (019) or learnable
attention weights (Veličković al. 2017: Vaswani al.) 2017). The combination function COM()
integrates the neighborhood information from the previous layer.
2.2 A CLOSER LOOK AT KNOWLEDGE DISTILLATION IN SEMI-SUPERVISED LEARNING
To bridge the gap between vanilla MLPs and more advanced GNNs for graph analysis, an intuitive solution is to conduct cross-model knowledge distillation. Formally, let 2, RC denote the soft labels of node e predicted by GNN teacher model, and ÿ. E R be the predictions of the MLP student model. The standard distillation process in (Zhang et al. (022b) is expressed as:
C=
Σ Loe(fe. (1 DEVL
Σ PEA.-UyU
CKL(Fe, Z.),
(2)
where Lce is the standard cross-entropy loss on labeled nodes, while Lx L is knowledge distillation loss. i.e., the Dallback-Leibler divergence between the predictions of the MLPs student and GNNs teacher. À is a trade-off parameter. Note that, different from supervised learning, the distillation loss in Eq.22 naturally includes two parts in the semi-supervised scenario: labeled node set VL and unlabeled node set This design choice is inspired by standard supissupereiee learning philosophy (Yang al.] 2021b), where unlabeled data is believed to be helpful in promoting model performance. According to our empirical results in Figure [ (left), we observed that this tendency holds in KD. Without the soft labels of unlabeled nodes, the MLPs student only perform comparably to the vanilla MLP baseline.
However, aforementioned in the Introduction, we argue that blindly leveraging soft labels of all the nodes in Eq. 2 suboptimal, since the soft labels from the teacher are noisy, especially for unlabeled nodes y Here, "noisy soft labels" refer the soft labels of the nodes whose true labels mismatch
3

---

Under review as a conference paper at ICLR 2023
the predictions of the GNN teacher. To verify this point, we conduct preliminary experiments from the oracle perspective, by assuming that the ground truths of unlabeled nodes are known. Then, we manually control the ratio of noisy soft labels in the knowledge distillation loss. Figure (middle) reports the results.
Observation: The soft labels of incorrectly predicted nodes restrict the capacity of MLPs student; By reducing the noise ratios, a stronger MLPs student can be easily achieved. As shown in the right panel of Figure [] the MLP student's performance drops significantly as the noise ratio increases. If we can control the error ratio to some extent, e.g., 15% %, the MLPs student can easily achieve comparable or even better results than GNNs teacher.
Nevertheless, it is non-trivial task to effectively identify those wrongly predicted nodes from the correctly classified ones, following the standard entropy-based heuristic approach (Kwon al. 2020). This is because the entropy distributions of the two groups are often largely overlapped in GNNs. For example, the entropy distributions of wrongly and correctly predicted nodes are generally overlapped with 40% areas different GNNs models as shown in Figure (middle) and Figure (aaree)(Appppnn (Appendix).
The above observations pose the challenge: Can we filter out the noisy teacher guidance in an aulomatic fashion, such that stronger MLP sludent can be achieved using reliable GNN knowledge?
PROPOSED MODEL
In this part, we present a simple, generic, and effective KD framework to tackle the unreliable GNN guidance issues revealed in the Motivation section. Specifically, we first introduce the problem formulation in Section 3.1 and then elaborate on our proposal in Section 3.2
3.1 PROBLEM FORMULATION
Given a graph G E). its feature matrix X E RR and label matrix Y E RN We use
E R NXC denote the soft label matrix produced by the teacher GNN model and fauteen to denote
a student model parameterized by multi-layer perceptrons. In traditional knowledge distillation settings (Zhang et al. 2022b), the student model is optimized according to two soft-label sets: (1) the labeled set R- {x.,, 2u. E V²), and (2) the unlabeled set R² = {(x,, Z. 2I E vo J. Nevertheless, as discussed Section 2] using the soft labels of all nodes in V would degrade the performance of student models, since many unlabeled nodes are incorrectly predicted by the teacher model, which introduces unreliable guidance.
To this end, we study the reliable knowledge distillation (RKD) problem. The core idea of RKD is to filter out the wrongly predicted nodes by GNN teacher and construct a reliable sobt-label set (i.e., R,) for student training. Formally, R, R2 U R. consists of two parts, where R,P (or R.,) includes those labeled (or unlabeled) nodes that are correctly predicted by the GNNs teacher. In practice, can directly obtain the sobt-label set R2 from labeled nodes since we already have the ground truths. Specifically, given a labeled node v. if the prediction from the teacher matches the ground truth, then E R,, otherwise, & R.F. Therefore, the main challenge in RKD is how determine the soft-label set R from unlabeled nodes, since no ground truth is available to check their validity.
3.2 THE PROPOSAL
We present RKD-MLP, a general reinforced frame- work for training student MLPs via reliable GNN distillation. The full framework is illustrated in Figure 2 The key idea is to learn a meta-policy network to determine the reliable soft label set (Sec- tion 3. and train the student MLP based on the reliable guidance (Section 3.2.2) After that, uni- fied framework is designed to train the meta-policy network and student model jointly (Section 3.2.3)
Figure 2: The RKD-MLP framework. Our meta-policy filters out noisy GNN teacher guid- ance, which is then used to train MLP student.
4

---

Under review as a conference paper at ICLR 2023
3.2.1 META-POLICY
To obtain the reliable soft label set R,, an intuitive solution is to utilize the uncertainty of teacher predictions (Kwon et al., 2020). For example, we can compute the entropy of all the nodes using their soft labels from the teacher GNNs, and then filter out those whose entropy values are higher than pre-defined threshold. However, as shown in Figure (Right) and Figure (in Appendix), entropy can not well differentiate between the correct and incorrect nodes since they are largely overlapped. To overcome this limitation, we propose to develop a learning- based approach to automatically fit the complex decision boundary between them. Specifically, following (Zoph & Le 2016 Gao et al., 2019; Lai et al. 2020), we assume that a meta-set with ground truth labels is available; in this work, we use the validation set as the meta-set. Then propose to train meta-policy with reinforcement learning (RL) to identify the reliable soft labels, where the meta-policy is updated based on the reward from the meta-set.
Formally, let Mmett = {(xx. Zv. [,)) denote a meta-set with m samples, where z, is the teacher prediction and Yu denotes the ground truth. I, 1 if the teacher model makes correct prediction; I. = 0 otherwise. We define the state, action and reward as follows. Let S be the state space; in this work, we use node representations (we will illustrate how obtain these later) as the states, i.e., E S. Let A {0, 1 be the action space, where O indicates that the soft label unreliable, and suggests that the soft label reliable. Given a node X,s an agent takes action a, and receives scalar reward r.. where positive reward r. = 1 is given if the label is indeed reliable (i.c., correct teacher prediction) when ag = or indeed unreliable (i.c., incorrect teacher prediction) when ag 0, and r. = 0 otherwise. Let S + A be meta-policy that maps states to actions. With neural function approximators, we use To to denote a parameterized meta-policy with parameters and (T(a))x)) to denote the probability of sampling a at state x,. The objective is to train the meta-policy network wa such that it can maximize the expected reward:
J
E E|ro]
where node is any node from all the nodes in the graph. Following the policy gradient theo- rem (Williams 1992), we can calculate the gradient of I w.r.t. 0 as
VoJmete = Voll[r
Brr Ve log To (axa)(x)))
where a, is the currently selected action for node v. We approximate the above gradient with the samples in the meta-set Mmetta
V ToJ Toeetea
meta
Σ
ro Vo (o..
(5)
VEEMuets
where can be obtained based on the ground truths in the meta-set by regarding the reliable soft labels as the ones that the teacher model makes correct predictions. The update of Eq. 5 can be unstable due to the high variance of the gradients. Thus, we introduce baseline (Sutton & Barto 2018) for variance reduction. The final gradient can be written as
VoJmsta R
Σ (re - B) V B)
[log To (a [x,)];
(6)
#f DEMine
where the baseline = (ErEMinn Tw)/m is the mean reward across the nodes in the meta-set, and (r. B) can interpreted as the advantage, i.e., the extent to which the current action is better than the average case. Then use the meta-policy ло to predict the reliable soft labels for the unlabeled nodes. Specifically, the soft label of an unlabeled node u is considered reliable if the predicted probability of is larger than 0.5, i..., Ry {(x,: 2..) VV E Do (a 1/x.)) > 0.5).
>> Rationale. Despite the simplicity of RL in Eq. 6 the reward- -diven objective enables the meta- policy to reason about the reliability of the soft labels based on node features. Once trained on the meta-set, the meta-policy can transfer to the unlabeled nodes to take the most rewarding action (i.c., reliable unreliable) for each node.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Alternating Gradient Descent for RKD-MLP
Input: Initial meta poelcy network Ty and initial MLPs student model fstuent
while not converge do
1. Obtain the node embedding h of unlabeled nodes based on MLPP student
2. Train the meta-policy network wy based on policy gradient in Eq. and meta-set M
Fix meta-policy we and update the student model fstudent based reliable knowledge distillation loss in Eq. 7
Return The well trained MLP student model sandent-
3.2.2 STUDENT MODEL TRAINING WITH RELIABLE GUIDANCE
By querying the meta-policy, we can train the student MLPs with better guidance. Formally, we rewrite Eq. 2 as:
Σ DEVL
(Jsatet Xx), yo) + (1-A)
Σ
I (Jamte (X.) bo); VEV
(7)
The key design of Eq. iis the second term, which only considers soft labels from the reliable set RR,. One benefit of the above equation is that the soft labels are noise-less, so the MLPs student model will be trained with more reliable information from the teacher model.
3.2.3 THE UNIFIED TRAINING ORJECTIVE
Instead of training the meta-policy and student MLPs in a two-stage fashion, e.g., training the meta- policy first and then optimizing the student model, we propose to simultancously train feeta and fstueent according to the following bi-level optimization framework:
min C(X, fmmen) fander.
s.t.
क == arg max Imeta (M (M a
(8)
The outer objective is defined in Eq. which requires the meta-policy Ty select reliable soft-label predictions from the GNN teacher. The inner objective Jmeta is defined in Eq. 6 and it takes node representations from the MI.P student and soft label vectors from the GNN teacher as input. worth noting that the design of xa can take other node embeddings as input such as the raw features or hidden embedding from the GNN teacher. However, we find that using h as state representation is beneficial since jointly training the policy network and MLP student could reinforce their reciprocal effects. This is because learning a better MLP student requires Ty to generate a more reliable soft label set while training high-quulified policy needs more informative node embeddings input. From Table [ and we can see that the MIP student performs better than the corresponding GNN teacher when it converges. Thus, is reasonable conjecture that hidden representations of the MLP student are more informative. Following the common practice (Liu et al. adopt the Alternating Gradient Descent (AGD) algorithm optimize Eq. g by iteratively updating the outer and inner optimization objectives, as outlined in Algorithm
2018) You et al., 2021). we
4 EXPERIMENTS
In this section, extensive experiments are reported to explore the following research questions. RQ1: How effective RKD-MLP compared with state-of-the-ar baselines in transductive and inductive settings? RQ2: Can RKD-MLP scale up to large-scale graphs? RQ3: What are the impacts of noisy node features or topology structures on RKD-MLP? RQ4: How effective is our meta-policy in identifying reliable teacher guidance? RQ5: How does each component of RKD-MLP contribute to the performance? RQ6: How efficient RKD-MLPP compared with other acceleration methods?
4.1 EXPERIMENTAL SETUP
Benchmark Datasets. For comprehensive comparison, use seven popular semi-supervised classification datasets with various scales and types, including Cora, CiteSeer, and PubMed (Sen et al. 2008), WikiCS, Amazon-Computers (Compute), Amazon-Photo (Photo), Coauthor-CS

---

Under review as a conference paper at ICLR 2023
Table 1: Node classification accuracy on commonly used graph datasets in transductive learning. Improv. indicates our proposal outperforms GLNN baseline. ". indicates the result for the teacher.
Teacher
Swdeett
Cora
CiteSoer
PubMed
WIkICS
Compute
Photo
CS
MLPs
68.04 + 0.75 09.22 + 1.31 70.54
+ 0.77
63.73
= 1.51 67.80 = 1.06 78.77 = 1.74
84,80
+ 0.09
79.70 4: 0.52 68.59
0.27
76.55
0.29
65.59
0.88
52.97
2.16 90.90
0.84
90.56 +: 0.38
SW
48.88 6.81 54.47
9.39
76.48
+ 0.59
58.77
2.28
56.30
3.31
63.15
4.85
37.18
1.22
SAGE
Entropy Cluster
80.73 + 0.76
69.
48
+
0.52
78.48
= 0.29
69
1.41
83.53 = 0.65
92 42
0.74
91.79 = 0.39
74.60 + 0.35
70.23
0.55
80.06
+ 0.07
67.05
= 0.67
81.48
0.95
87.84 = 0.35
91.54 - 0.15
GLNN
80 00.0000000 + 0.52
28.95
0.70
76.97
= 0.28
67. 82 = 1.36
83.04
= 1.70
92.02
= 1.15
90.95
+ 0.51
RKD-MLP
81.52 + 0.66 70.23
+: 0,48
80.97
0.20 71.77
= 0.64
84.23
0.53
93.78
0.46
92.58
+ 0.30
Imprn:
+1.90%
+1.86%
+5.20%
+5.82%
+1.3%%
+1.1%
+ +1.9%
Table 2: Node classification accuracy on commonly used graph datasets in inductive learning, Improv. indicates our proposal outperforms GLNN baseline. "h" indicates the result for the teacher.
Teacher
Student
Cora
CiteSeer
PubMed
WikiCS
Compute
Photo
80.29 + 0.19 72.64 +: 0.45 78.88 + 0.21 66.13
+ 0.15 80.33 = U.57 86.34 + 0.57 89.02 + 0.48
SW
47.12 1.07
59.08 + 6.00
76,88 + 0.81 54.40
+ 2.23
48.58 + 10.25
62.39 + 6.65
35.81 + 0.54
GCN
Entropy
80.87
73.53
79.28
68.73
82.30
1.08
0.83
91.39
0.27
Cluster
72.30 1. 0.52
75.12 1 0.63
80.22
1 0.32
65.63 1 0.55
81.62
0.68
87.30 1 0.74
91.34
1 0.17
GI
79.23 + 1.51
73.42
+ 0.41
79.36
+ 0.32
67.74
0.49
80.56
0.73
87.40
+ 0.57
90.64 + 0.23
RKD-MLP
82.50
:: 0.74
75.22
+ 0.71 81.00
0.50 69.88
0.38
82.47
0.49
90.36
1.22 92.38 + 0.31
Improv.
+4.20%
+2.45%
+2.07%
+3.16%
+2.37%
+-3.8%%
+1.92%
For experiments on large-scale graphs, we use two Open Graph Benchmark 2020): ogbn-arxiv and ogbn-products. More detailed discussions about the datasets and their statistics are included in Appendix
(CS) (Shchur et al.
2018).
datasets (Hu et al.
Teacher GNNs. For a thorough comparison, we consider five promising GNNs architectures as teacher models in our knowledge distillation framework: GraphS (Hamilton et al.) 2017a) (SAGE), GCN (Kipf & Welling 2016). APPNP (Klicpera et al.) 2018). GAT (Veličković et al..
2017). and SGC (Wu et al.) 2019). For extremely large-scale datasets such as ogbn-product, we consider two scalable teacher GNNs: ClusterGCN (Chiang et al.) 2019) and GraphSAINT (Zeng ct al. 2019). The detailed training settings of these teacher models are listed in Appendix B.1
Student Competitors. In addition to the GNN teachers, we also include two types of student baselines for comparison. First, we consider three heuristic-based approaches: Cluster, Entropy, and sample re- weighting (SW), which construct reliable soft-label set via clustering, relative prediction rankings, and sample re-weighting, respectively (See Appendix B.2 for more details). Second, we
include two MLPs based related work: vanilla MLPs and GINN (Zhang et al. 2022b).
Transductive vs. Inductive. Follow previous studies (Zhang al. 2022b). we evaluate our model under two node classification settings: transductive and inductive, The main difference between them whether to use the test data for training or not. For the inductive setting, the test nodes as well as their edge links will not be used. The experiment details of the two settings are in Appendix B.3
4.1.1 IMPLEMENTATION DETAILS
We build our model based on Pytorch and PyG library (Fey & Lenssen 2019). For GNN teachers, following common practice in (Zhu et al. 2021b Hu et al. 2020 2020a we employ a three-layer GNN encoder with dimension d = 256 for OGB benchmarks (ogbn-arxiv, and gebn-prooduts)) while a two-layer GNN encoder with dimension d = 128 for other datasets. For MLP students, following (Zhang et al. 2022b), we set the number of layers and the hidden dimension of each layer to be the same as the teacher GNN. We set À = if not specified, since we empirically found that the proposed model is robust to X shown in Figure of Appendix. All the experiments are run times on GeForce RTX 2080 Ti GPUs, and we report the mean and the standard deviation. More detailed configurations for different GNN teachers and MI. students are provided in Appendix [..
4.2 RESULTS AND ANALYSIS
How effective is RKD-MLP against other baselines small datasets? (RQ1). Table T 2 and 6able 7 (in Appendix) report the results of our RKD-MLP with heuristic and MLPs based baselines. We make three major observations. First, compared with vanilla MLPs and intuitive
7

---

Under review as a conference paper at ICLR 2023
Figure 4: Accuracy results of RKD-MLP and Figure 5: Accuracy results of RKD-MLP and
other baselines w.r.t noise graph topology.
other baselines w.r.t. noise node features.
Dea
hall
WARES
Q94 PeRee
Figure 6: Confusion matrix of RKD-MLP on unlabeled nodes using GNN prediction as ground truths. The x-axis is the prediction of our proposal, and y-axis denotes if GNN teacher makes the right prediction. 1 means make the right prediction; otherwise
KD method GLNN, RKD-MLP performs significantly better than them in all cases. Specifically, RKD-MLP improves GINNN by up to 5.82% in the transductive setting (See Table D Second, RKD-MLP also outperforms three heuristic solutions (SW, Entropy, and Cluster) general. The possible explanation is that our meta-policy is trained end-to-end with the MLP student, so that they can reinforce their reciprocal effects. Third, compared with 5 GNN teachers, our proposal consistently achieves better results across different benchmark datasets and two evaluation scenarios. Another interesting result is that the two heuristic methods (Entropy and Cluster) generally perform on par with or even better than GLNN across two settings. These results shed light our motivation to study reliable knowledge distillation for MLP student training.
How does RKD-MLP perform large-scale graphs? (RQ2). Figure Figure in Appendix summarize the results on two challenging large- scale graphs like ogbn-arixv and ogbn-products, from which we derive two insights. First, RKD- MLP is scalable and can achieve much higher re-
42
00.49/1.60
71.67
70-
62.47
Accuracy
60
56.05
sults than vanilla KD method. As shown in Fig-
50-
GLNN
ure RKD-MLP improves GLNN 8.5% and 6.3% on ogbn-arxiv and oghn-products, respectively. Sec- ond, unlike small datasets, it is hard to train the MLP student on large graphs due to soft-label noises. For instance, GLNN can achieve compa- rable results with GNN teacher on small datasets (See Table [ but performs significantly worse on large graphs (See Figure 3 77). By avoiding unreliable guidance, our RKD-MLP can easily outper- form the GNN teacher on small datasets, and bridge the gap between GLNN and the GNN teacher on large graphs.
40-
MIP
GNN Teacher
ogbn-arxiv
egbn-products
Figure 3: Accuracy results of RKD-MLP large-scale graphs. Left: GraphSAGE teacher.
Right: clustergnn teacher,
How robust is RKD-MLP w.r.f feature or topology noises? (RQ3). Figure 4 5 report the results of RKD-MLP on two types of noise scenarios (Detailed experimental setups are provided on Appendix D.I. In general, we observe that our proposal performs consistently better than other baselines, and is more robust on topology noise compared with feature noise. For example, the performance gap between RKD-MLP and the second best baseline on incomplete graph structure (left two panels) is higher than that on noise feature (See Figure 4. We contribute this robustness gain to the proposed meta-policy, since can filter out noisy teacher guidance.
How effective is RKD-MLP in eliminating noisy guidance? (RQ4). We summarize the confusion matrix produced by our meta- policy to study its effectiveness. As shown in Figure our proposal effectively reduce the noise degree some extent across different datasets (truth positive or negative samples). For instance, RKD-MLP reduces the relative noisy ratio from 23.26% to 17.20% on PubMed, and 29.42% 18.03% on ogbn-arxiv.
Ablation Study (RQ5). We study the importance of joint optimization and random selection on RKD-ML.P with two ablations: RKD-ML.--S and RKD-ML P-rand. RKD-MLP--iso is obtained
8

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study of RKD-MLP, clustergen teacher for products while SAGE for others.
Cora
Cieeeer
PubMed
WikiCS]
Compuier
Photo
CS
|gba-arxiv
oggn products
RKD-MLP-is 79.66
69.10
77.65
68.65
82.87
91.37
90.76
65.86
67.86
M.D-MMPPrrand
79.22
68.25
76.32
67.35
82.28
91.16
90.03
64.52
66.88
RKD-MLP
81.59
70.46
81.07
71.77
84.23
93.78
92.58
70.49
71.67
by separating the training of meta-policy and the MLP student. RKD-ML.P-rand is obtained by replacing the meta-policy with random selection. We made two observations from Table First. jointly optimizing meta-policy and MLP student can reinforce their reciprocal effects, since RKD- MLP outperforms RKD-MLP-iso in all cases by a great margin. Second, random selection fails to distinguish the decision boundary between correctly predicted samples and incorrectly classified samples by the GNN teacher, so performs the worst.
Efficiency Analysis (RQ6). We compare the inference efficiency of RKD-MLP against state-of- the-art acceleration methods based on sampling, pruning and quantization strategies Appendix C From the results in Table Appendix, we observe that our distilled MIP student runs significantly faster than all other baselines. Given the high accuracy results of RKD-MLP on Table and 2 our model is more desired to be deployed on resource-sensitive applications.
RELATED WORK
GNN Acceleration. The existing efforts for GNN speedup can be mainly grouped into two categories: scalable training and inference acceleration. 1) Scalable training aims to scale GNNs to large-scale graphs with millions or even billions of nodes. Typical examples including sampling (Hamilton et al., 2017a: Chen et al. 2018 Zeng et al. 2019). clustering (Chiang et al. 2019), and decoupling (Wu et al. 2019; Rossi et al. 2020; Sun et al. 2021) based methods. Unfortunately, these methods suffer from inference challenges, since their message propagation process (Gilmer et al., 2017) is computationally expensive. 2) Inference acceleration focuses on accelerating the inference latency of GNNs, in order to extend their applicability in resource-constrrinet applications. Several initial attempts based on pruning (Zhou et al.) 202T: Chen et al.] 2021 Sui et al. (02I) and quantization (Zhao et al. 2020 Liu et al. (021) have been developed, but their improvements are limited, cause they still rely message propagation for embedding. In contrast, our RKD-MLP results in a pure MLP, which naturally runs significantly faster. Graph-MLLP (Hu et al.] 2021b) also tries to learn an MI. for node classification, but it only considers transductive setting but not the more practical inductive setting.
GNN distillation. Knowledge distillation KD has been applied to compress GNNs recently. However, the majority of them are designed to distill large GNNs to smaller GNNs (Yang et al. 2020 Yan
al. 2020 Deng Zhang 202T Xu et al. 2020) or label propagation student (Yngg et al. 202 Since message propagation is also needed, they face the same inference issue as standard GNNs. Ns. One
close work us is GLNN (Zhng et al.) (022b)) which accelerates GNN inference by distilling it to an MI. student. However,s it only considers utilizing knowledge from GNN teacher blindly, which suboptimal practical classification situations when labeled nodes are scarce, causing the teacher model will inevitably deliver wrong predictions. In contrast, proposal proposes to say no to noisy teacher knowledge and train an MLP student using those reliable soft labels.
6 CONCLUSIONS AND FURTHER WORK
In this paper, we study how to distill GNN knowledge to an MLP student when labeled nodes are scarce. We empirically found that the utilization of unlabeled nodes the key to training the MLP student successfully. Meanwhile, the noise in soft labels of unlabeled nodes is the major hurdle to preventing the student model from growing stronger. To this end, we present RKD-MLP, an effective distillation framework, which offers reliable teacher guidance for the MLP student by filtering out incorrectly predicted unlabeled nodes automatically. Extensive experiments on open-sourced small and large benchmarks demonstrate the effecttiveness&efffiiiineg of RKD-MLP over existing algorithms. Moreover, RKD-MLP shows promising robustness w.r.t. incomplete graph topology and noise node feature scenarios. making it a desirable choice in resource-constrr constrrind applications. In the future, we will explore how to employ multiple teachers to further improve the performance of RKD-MLLP since different GNN variants may have different prediction capacities in practice.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sergi Abadal, Akshay Jain, Robert Guirado, Jorge López-Alonso, and Eduard Alarcón. Computing graph neural networks: A survey from algorithms accelerators. ACM Computing Surveys (CSUR), 54(9):1-38, 2021.
Anshika Chaudhary, Himangi Mittal, and Anuja Arora. Anomaly detection using graph neural networks. In 2019 International Conference on Machine Learning. Big Data, Cloud and Parallel Computing (COMITCon), pp. 346-350. IEEE, 2019.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via 2018.
importance sampling, arXiv preprint
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pp. 1725-1735 PMLR, 2020.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In International Conference on Machine Learning, pp- 1695-1706 PMLR, 2021.
Wei-Lin Chiang, Xuanqing Liu, Si Si. Yang Li, Samy Bengio, and Cho-Jui Hsich. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp- 257-266, 2019.
Xiang Deng and Zhongfei Zhang. Graph-free knowledge distillation for graph neural networks. arXiv preprint arXiv:2 075199 2021.
Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. Graph random neural networks for semi-supervised learning on graphs Advances in neural information processing systems, 13:22092-22103 2020.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint M.X.v:1903.0244. 2019.
Hongyang Gao and Shuiwang Ji. Graph u-nets. In international conference machine learning, pp. 2083-2092. PMLR, 2019.
Yang Gao, Hong Yang, Peng Zhang, Chuan Zhou, and Yue Hu. Graphnas: Graph neural architecture search with reinforcement learning. arXiv preprint arXiv:1904.0 1,999,1 2019.
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv preprint arXiv:1711.0404 2017.
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference machine learning, pp. 1263-1272. PMLR, 2017.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. Proceedings of the 22nd ACM SIGKDD international conference Knowledge discovery and data mining, pp. 855-864, 2016.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs.
Advances in neural information processing systems, 30, 20.7a.
William Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. arXiv preprint arXiv: (tt:::770005550 2017b.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. anXiv preprint /503. 025331 2(7), 2015.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint W.Yv:2005500668) 2020.
10

---

Under review as a conference paper at ICLR 2023
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Isc: A large-scale challenge for machine learning on graphs. arXiv preprint Stt::21033094300 2021a.
Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang, Erjin Zhou, and Yue Gao. Graph-mlp: node classification without message passing in graph. arXiv preprint orXiv:2106.04000 2021b.
Thomas N Kipf and Max Welling. Semi Semi-superviss supervised classification with graph convolutional networks.
arXiv preprint arXiv: wee.li.699.0000 2016.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:1810.05599,, 2018.
Kisoo Kwon, Hwidong Na, Hoshik Lee, and Nam Soo Kim. Adaptive knowledge distillation based on entropy. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7409-7413, IEEE, 2020.
Kwei-Herng Lai, Daochen Zha, Kaixiong Zhou, and Xia Hu. Policy-gnn: Aggregation optimization for graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 461-471, 2020.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference Learning Representations, 2018.
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. Proceedings of the 26th ACM SIGKDD international conference knowledge discovery & data mining, pp- 338 338-348.20 2020.
Zirui Liu, Kaixiong Zhou, Fan Yang. Li Li, Rui Chen, and Xia Hu. Exact: Scalable graph neural networks training via extreme activation compression. International Conference on Learning Representations, 2021.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710, 2014.
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004. 11199, 2020.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad Collective classification in network data, Al magazine, 29(3):93 93, 2008
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günneeann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: 1.05668,8.8.... 2018.
Yongduo Sui, Xiang Wang, Tianlong Chen, Xiangnan He, and Tat-Seng Chua. Inductive lottery ticket learning for graph neural networks. 2021.
Chuxiong Sun, Hongming Gu, and Jie Hu. Scalable and adaptive graph neural networks with seff-label-enhanced training. arXiv preprint arXiv:2104.09376, 2021.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Shyam A Tailor, Javier Fermande--Marques, and Nicholas D Lane. Degree-quant: Quantization-aware training for graph neural networks. arXiv preprint trXiv:2008.05000 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-6008. 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10003 2017.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-255 1992.

---

Under review as a conference paper at ICLR 2023
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplify- ing graph convolutional networks. In ICML,,, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.008: 00860, 2018.
Yishi Xu, Yingxue Zhang, Wei Guo, Huifeng Guo, Ruiming Tang, and Mark Coates. Graphsail:
Graph structure aware incremental learning for recommender systems. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 2861-2868 2020.
Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou. Tinygnn: Learning efficient graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1848-1856, 2020.
Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In Proceedings of the Web Conference 2021, pp. 1227-1237, 2021a.
Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A survey on deep semi-supervice learning. arXiv preprint arXiv:2103.000 2021b.
Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7074-7083, 2020.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hier- archical graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In International Conference on Machine Learning, pp. 12121-12132. PMLR, 2021.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method. arXiv preprint Nediv:1907.0444 2019.
Hailin Zhang, Defang Chen, and Can Wang. Confidenco-aware multi-teacher knowledge distillation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4498 4502. IEEE, 2022a.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems, 34, 2021.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mips new tricks via distillation. In International Conference on Learning Representations, 2022b.
Yiren Zhao, Duo Wang, Daniel Bates, Robert Mullins, Mateja Jamnik, and Pietro Lio. Learned low precision graph neural networks. arXiv preprint arXiv:2009.092.. 2020.
Hongkuan Zhou, Ajitesh Srivastava, Hanqing Zeng, Rajgopal Kannan, and Viktor Prasanna. Acceler- ating large scale real-time gnn inference using channel pruning. arXiv preprint arXiv:2105.04528, 2021.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57-81, 2020.
Jaaning Zhu, Jiangchao Yao, Bo Han, Jingfeng Zhang, Tongliang Liu, Gang Niu, Jingren Zhou, Jianliang Xu. and Hongxia Yang. Reliable adversarial distillation with unreliable teachers. arXiv preprint arXiv:2 /06. 04928, 2021a.
12

---

Under review as a conference paper at ICLR 2023
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. WWW, pp. 2069-2080, 2021b.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint 1rx.v:1611.01578 2016.
13

---

Under review as a conference paper at ICLR 2023
Table 4: Dataset Statistics.
Data
# Nodes
# Edges
# Features
Classes
Cora
2, 708
5, 429
1,433
7
CiteSeer
3, 312
660
703
6
PubMed
117 717
44, 338
500
3
Wiki-CS
11, 701
216, 123
300
10
Jiaazon-Computn
13, 752
245, 861
767
10
Amazon-Phone
7,650
119. 081
745
8
Coauthor-CS
18,333
81, 891
8,505
15
ogbn-arxiv
169,
1, 166, 243
128
40
Lgbu-productt
2,449, 029
61, 859, 140
100
47
A DATASETS
In this section, we introduce the details of the datasets as bellow.
Cora, CiteSeer, and PubMed: represent documents and edges denote citation links. Each node has a sparse bag-of-the words feature vectors. Labels are defined the academic topics. We use them for both link prediction and node classification tasks.
They are three widely used citation network datasets. Nodes
ogbn-arxiv: This is large-scale paper citation network. Each node an arxiv paper and the edge indicates the citation connection between two papers. Every node is associated with a 128 22-dimensioonal feature vector obtained by averaging the embeddings words in its title and abstract. The task is to predict the 40 subject areas of arxiv CS papers.
Wiki-CS: It is a reference network constructed based on Wikipedia. The nodes correspond to articles about computer science and edges are hyperlinks between the articles. Nodes are labeled with ten classes each representing branch of the field. Node features are calculated as the average of pretrained GloVe word embeddings words each article.
ggbnnprrducts: This is an undirected and unweighted graph, representing an Amazon product co-purchasing network. Nodes represent products sold in Amazon, and edges between two products indicate that the products are purchased together. Node features are pivenssionality-rdddee bag-of-words of the product descriptions. The task is predict the category of a product in a multi-class classification setup, where the 47 top-level categories are used for target labels.
Amazon- -onrzon--Comuters and Amazon-Phos They two networks of co- po-uurhhase relation- ships constructed from Amazon. Nodes indicate goods and edges represent the co-purchase relationships of two products. Each node has sparse bag-of-word. feature encoding prod- ucts reviews and is labeled with its category. They are widely used for node classification task. Nodes represent authors and edges indicate co-authorship relationships. Each node has a sparse bag-of- words feature based on paper keywords of the author. The task is to predict the most active research field of authors.
Coauthor-CSS CS: This is academic network, which represents co- co-authorship graphs based on the Microsoft Academic Graph from the KDD Cup 2016 challenge.
We follow standard semi-supervised setting to split the data. Specifically, for Cora, CiteSeer, and PubMed datasets, we use the GCN (Kipf & Welling 2016) splitting strategy with the default random seed; for WikiCS, Mamazon-Computeen 2021a) splitting strategy and each random seed corresponds to different split. For the OGB datasets, follow the OGB official splits based time and popularity for Arxiv and Products respectively.
Amazon-Phost
Coauthor-CS. we the CPF (Yang et al.,
14

---

Under review as a conference paper at ICLR 2023
B EXPERIMENTAL SETUP
B.1 TRACHER ARCHITECTURES
The hyperparameters of GNN models on each dataset are taken from the best hyperparameters provided by the CPF (Yang et al. 2021a) paper and the OGB official examples. Table g and y summarize the hyperparameter configurations for 7 GNN models: GCN, GraphSAGE, APPNP, GAT, SGC, ClusterGCN, and GraphSAINT.
B.2 STUDENT BASELINE
To have fair comparison, for all two MLP student models (RKD-MLP and GLNN) and the vanilla MLP baseline, we set the number of layers and the hidden dimension of each layer to be the same as the teacher GNN, their total number of parameters stays the same the teacher GNN.
In addition to the MLPs based baselines, we also introduce several heuristic based competitors, which construct the reliable soft label set via rules of thumb.
Cluster: It a heuristic approach using clustering algorithm. The core idea is construct reliable soft label set from unlabeled nodes by verify their cluster assignments. Specifically, Cluster first builds centroids, one for each label class, and then assigns each unlabeled nodes to one of C clusters according to their similarities in the feature space. Finally, an unlabeled label will be marked as reliable if its prediction from the teacher matches the assigned cluster number.
Entropy. It constructs the reliable soft label set using the relative ranking of unlabeled nodes based on their entropy scores from the teacher model. In our experiments, tried two specific choices: wiss ranking list (i.c., each class has a ranking list) or across-class ranking list. We found that across-class ranking list works better in general, so we report the result for global ranking by default. In practice, we use top 8% unla- beled nodes to generate the final reliable set. We do hyperparameter search of 9 from {60, 65, 70, 75, 80, 85, 90, 95} and report the best result.
SW. It is a weighted loss function version. The core idea assign lower sample weights to unreliable nodes while higher scores to reliable nodes. To achieve this goal, we adopt another MLPs network to learn sample weight. Specifically, the weight MLPs network takes the hidden representation learned by MLP student as input and outputs its weight score. The weight network and MLP student will be trained end-to-end by minimizing weighted loss function.
B.3 EVALUATION SETTINGS
Given a graph 9, the feature matrix X. and label matrix Y, we consider two evaluation settings in experiments: transductive and inductive. The transudtive setting is the same to standard splitting strategy, where the test data is also used during model training. contrast, for inductive setting, we hold out portion of test data for evaluation only. Specifically, first pick up some inductive nodes Vin E p from the unlabeled node set, and then partition yo into two disjoint data set: inductive node set D and observed node set YU V Next, we manually remove all edges connecting those inductive nodes and obtain two disjoint graphs G = Gobe U Gind. Similarly, node features and labels ind U U and Y Y setup as follows:
are partitioned into three disjoint sets U obs U U on these notations, the two evaluation scenarios
ind ohs U U U.Y Y Based
Transductive: the model is trained on G, X, and YL, and tested X and YU, with KD based on z, for E V.
Inductive: the model trained on Gots, XU obs X and Y and tested on X! and Y Vind- with KD based z, for Yo U U
B.4 IMPLEMENTATION DETAILS
To make our results reproducible, we build our model based on Pytorch and adopt the benchmark tool Pyy (Py Torch Geometric) library (Fey & Lenssen 2019) implement various GNNs architectures.
15

---

Under review as a conference paper at ICLR 2023
Table Inference time (ms) on 10 randomly chosen nodes of two large-scale OGB datasets under the inductive setting.
Dataset
SAGE
QSAGE
PSAGE
Neighbor Sample
Ours
Arxiv
489.49
433,90 (1.13x)
465.43 (1.05x)
91.03 (5.37x)
3.34 (146.55x)
Products
2071.30
1946.49 (1.06x)
2001.46 (1.04x)
107.71 (19.23x)
7.56 (273.98x)
Following common practice in (Zuu et al. 202 lb Hu et al. 2020 2021a), we employ a three-layer GNN encoder with dimension d 256 for OGB benchmarks (ogbn-arxiv, and ogbn-products), while two-layer GNN encoder with dimension d = 128 for other datasets. For MLP students, following (Zhang et al. 20226), we the number of layers and the hidden dimension of each layer to be the same as the teacher GNN.
For our RKD-MLP model, we do a hyperparameter search positive augmentation ratio Opos from {0.1, 0.2, ... 1.00, negative augmentation ratio O'neg from {0.1, 0.2, ... 1.0), learning rate from [0.01, 0.005, 0.001], weight decay from [0, 0.001, 0.002, 0.005, 0.01J, and dropout from [0. 0.1, 0.2, 0.3, 0.4, 0.5, 0.6]. To accelerate the tunning process, we first fix the last three hyperparameters according to GLNN, and do grid search on apos and Oneg. After that, we fix the two augmentation ratios, and search the learning ratio, weight decay, and dropout.
C EFFICIENCY ANALYSIS
To investigate the efficiency of our proposed model, conduct experiments two large-scale OGB datasets Arxiv and Products. Following (Zhang et al. 20022), we include three types of baselines, i.e., vanilla GNNs, common acceleration techniques, and sampling approaches. The details are shown in the following.
Vanilla GNNs. We use GraphSAGE (Hamilton et al. 2017a) with GCN aggregation representa- tive of conventional GNN models.
Acceleration techniques. Pruning and quantization are common techniques of GNN infer- ence acceleration. They speed up GNNs by reducing model parameters and Multiplication-ana ACcumulation (MACs) operations. In this experiment, we utilize QSAGE (quantized SAGE from FP32 to INT8) and PSAGE (SAGE with 50% weights pruned) to represent this category.
Sampling. Neighbor sampling is classic approach reducing neighbor-fetching latency for GNNs. Here, we utilize inference neighbor sampling with fan-out 15, named Neighbor Sample.
Table rreporss the inference time of all methods under the inductive setting. From the table, that while other inference acceleration mechanisms could speed up SAGE, our proposed model runs significantly faster than them, let alone better performance.
D MAIN RESULTS
D. HOW ROBUST RKD-MLP W.R.T. FEATURE OR TOPOLOGY NOISES?
In this section, we conduct experiments to investigate the robustness of our RKD-MLP on two noisy situations: noisy node features and incomplete graph topology. For noisy node features, we replace X with X X O M, where M lg ND is an indicator matrix with each element is generated by a Bernoulli distribution of parameter 1 For incomplete graph topology, we randomly mask each edge in G with probability a; and replace the g with the resultant incomplete graph. Figure 5 and Figure 4 report the results of RKD-MLL MLD noisy features and structures, respectively, with varies from O.T to 0.9 with step-size 0.
E MOTIVATION EXPERIMENTS
In this section, we test the impacts of noisy nodes (incorrectly predicted nodes by the teacher model) on MLP student training. Specifically, given pre-trained GNN teacher, first query give
16

---

Under review as a conference paper at ICLR 2023
Table 6: Node classification accuracy on commonly used graph datasets in transductive learning. Improv. indicates our proposal outperforms GLNN baseline. ". indicates the result for the teacher.
Teacher
Student
Cora
CiteSeer
PubMed
WikiCS
Compute
|82.09
0.28
69.62
0.28
78.38
+ 0.27 67.29
0.64
82.93 0.67
91.09
0.49
90.31
+ 0.25
SW
52.24 1 3.40
61.35
2.00
75.91 1 1.53
$5.89
1 0.70
54.01
1 2.04
56.56
1: 5.07
35.93
1: 1.28
GCN
Entropy
$1.45
09.95
0.24
82.46
0.779
92.78
0.45
0.38
Cluster
74.32 + 0.45
70.42
4 0.33
80.90
+ 0.14
67.87
+ 0.43
81.63
+ 0.74
87.49
+ 0.26
91.66
+ 0.13
GL.NN
81.64 à: 0.90
69.86
0.80
79.06
1 0.30
69.43
à 0.83
83.05
1: 0.72
92.12 à: 0.67
91.92
à: 0.52
RKD-MLP
$2.53
0.58 71.52
81.61 + 0.39
72.13
0.65
84.44
0.57
93.27
0.31
92.88 0.15
Improw:
+1.09%
+2.37%
+3.23%
+3.88%
+1.67%
+1.24%
+1.01%
81.59 0.64
70.44 0.21
79.68 + 0.19 67.84
+ 1.08
81.67 = 1.21
91.92 + 0.95 90.69
+ 0.28
51.02
53.02
3.63
77.92
55.20
1.76
56.06
35.67
0.51
APPNP
Entropy
80.62 + 0.37
70.37
+ 0.61
79.26
= 0.25
68.97
= 1.22
82.69
1.35
92.16 = 0.40
92.27 + 0.26
Cluster
75.14 0.54
71.41
0.35
80.59
0.25
68
10
- 0.54
81.10
0.83
87.61 = 0.48
91.61
0.16
GLNN
$1.83 + 0.78
70.67
+ 0.57
80.27 + 0.36
69.87
1.03
81.76
1.11
91.92 = 1.08
90.88
0.40
RKD-MLP
$2.80
71.91
0.38
71.1
83.06
93.27
0.56
0.22 +2.04%
Impron:
+1.18%
+1.75%
+1.37%
+2.24%
+1.59%
+1.46%
82.64 0.63
09.60 + 0,42
77.80 = 0.25
68.00
= 0.97
81.90 1.51
91.12
0.74
89.73
0.72
SW Entropy Cluster
45.85
7.67
47.00
13.05
73.70
+ 2.63
51.82
3.50
47.87 + 13.00
62.23
7.28
33.69
5.14
GAT
80.67 = 0.69
70.05 + 1.21
79.11 + 0.36
70.14
+ 1.21
82.26 + 1.73
92.88 = 0.70
89.61
2.67
75.85
91.76
0.19
$2.55
69.92
0.31
1 70.69
1.17
82.19
0.62
0.91
RKD-MIP
P
84.
0.36
72.56 + 0.68
82. + 0.27 73.21 + 0.84
83.85 + 0.53
93.49 - 0.66
92.77 + 0.20
Impres:
+1.90%
+3.77%
+4.28%
+3.36%
+2.01%
+1.8%
+1.94%
80.59 = 0.47
69.39
+ 0.31
78.30 + 0.14 67.78
+ 0.59
82.70
+ 0.64
91.30 + 0.51 90.13 + 0.38
SW
46.80 1: 4.24
45.11
1 14.58
75.84
1 1.05
60.44
1 1.90
54.11
6.35
59.29
1 3.59
35.98
1:0011110111 1.00
Entropy Cluster
80.40
70.34
0.64
79.54
+ 0.15
69.79
+
82.49
92.22 - 0.81
91.85
0.48
74.00
0.99
70.08
+ 0.70
80.09
= 0.19
67.62
= 0.91
81.60
0.75
87.32 = 0.70
91.90
0.15
GLNNN
$0.93 0.75
69.87
0.32
78.09
+ 0.16 69.87
+ 0.56
83.08
+ 0.70
92.54 + 0.37
91.39
0.65
RKD-MLP
$2, = 0.34
71.28 + 0.92
81.72
+ 0.40
72.50
0.83
0.89 84.56
93.53 - 0.46
93.02 0.40
Improx:
+2.51%
+2.01%
+3.85%
+1.78%
+1.06%
+1.78%
70.4121.44
67.84
65.24
62.47 62.47
Accuracy
50
MLPs
GLNN
40
RKD-MLP
GNN Teacher
30
agbn-arxiv
ogbn-products
Figure 7: Accuracy results of RKD-MLP on large-scale graphs. Left: GCN teacher. Right:
GraphSAINT teacher.
predictions for all nodes (including both labeled and unlabeled nodes). Then we divide nodes into two categories based whether they correctly predicted by the teacher model not. We use Scorrect and Swrong to denote the correctly and incorrectly predicted node sets, respectively. Let w be the noisy ratio, which is selected from {0.0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.30). For a given w, randomly sample [Scorrect| * w nodes from the S Surong set and extend them to the correctly predicted set, resulting a noisy node set S. After that, train the MLP student based on $ and report the averaged results of ten runs. Figure summarizes the results on Cora, WikiCS and ogbn- products datasets.
The interesting observation is that the GNN teacher outperforms the MLP student even in 0% noisy scenario, which is the upper bound the MI. student could achieve under KD. This result
17

---

Under review as a conference paper at ICLR 2023
Table 7: Node classification accuracy on commonly used graph datasets in inductive learning, Improv. indicates our proposal outperforms GLNN baseline. "-" indicates the result for the teacher.
Teacher
Student
Cora
CiteSeer
PubMed
WikiCS
Compute
MLPs
599.09 = 2.96 59.60
1.57 67.73 = 1.211.2 1.23 62.55 + 1.85
67.84 1.78
79.11 + 1.72
85.57 + 0.92
|00.00 = 0.42
71.79
+ 0.22
77.50 = 0.19 65.58 = 1.47
76.53 + 1.61
87.13
= 0.34
91.31
+ 0.36
SW
15.81 = 2.68
52.05 + 17.53
78.56 = 0.87 56.32 + 2.31
46.71
13.93
61.22 - 6.53
31.60
+ 2:27
Entropy Cluster
$1.02 0.36
73.20
0.58
79.74 + 0.37 67.78 + 0.93
80.95
91.35
0.49
91.37 + 0.10
72.98 = 0.73
71.86
+ 0.80
80.20
= 0.11
66.36
= 0.57
81.25
1.10
87.57 = 0.44
91.21
0.11
GLNN
80.19
+ 0.71
72.31
+: 0.58
78.76 + 0.58
67.29 + 1.31
76.30
+ 1.65
87.70
+ 0.40
91.38
+ 0.53
RKD DDD.M.P P
81.69
1: 0.96
75.00 1. 0.29
81.18 1 0.65 69.62 1: 0.43
81.50 1 0.77
91.58 + 1.45 92.41 1: 0.54
Improw.
+1.87%
+3.72%
+3.07%
+3.46%
+6.81%
+4.42%
+1.12%
882.87 = 1.06
72.50
0.71
79.18 + 0.12 67.33
+ 0.88
80.24
0.62
77.30 + 1.94 89.61
+ 0.22
SW
47.89 + 3.89
59.22
4.55
78.32
+ 0.94 54.46
+ 2.72
56.82
4.46
61.88 - 2.51
35.81
+: 2.63
APPNP
Entropy Cluster
79.61
1.63
74.23
1 0.64
80.54
1 0.43
68.47
0.84
81.20
0.23
82.88
0.84
91.79
1 0.17
73.27 = 0.52
74.23
+ 1.09
80.36
+ 0.42
66.21
- 0.41
79.75
1 0.81
80.61
- 1.54
91.25
4 0.10
GLNN
82.68 1 0.68
72.75 1: 1.08
79.88
1 0.23
68.78
1 1.00
80.22
1 0.54
78.08
1 1.91
89.99
1 0.29
RKD-MLP
$3.38 +
74.93
+ 0.79
81.44 + 0.51
69.83 + 1.01
82.60
85.10
0.90
92.60
0.19
Improv:
+0.84%
+2.99%
+1.9%%
+1.52%
+2.96%
+8.99%
+2.90%
81.07
0.79
72.72
+: 0.60
78.00 + 0.28
67.77
+ 1.19
80.20
1.38
90.38
1.02
89.
29
+ 0.86
43.68
50.13
9.22
2.02
54.50
57.40
59.34
4.60
1.76
GAT
Entropy Cluster
$1.80 + 1.36
74.09
0.60
78.56
- 1.13
69.99
+ 1.51
81.13
2.40
91.83 - 1.24
90.64
+ 0.65
0.93 73.85
74.53
0.57
80.30
0.15
66.54
0.50
80.58
0.16
88.33 - 0.57
+ 2.61 90.15
GLNN
81.89 + 0.39
72.94
4: 0.73
79.68
+ 0.62
70.10
+ 1.41
80.69
0.26
91.31
+ 1.08
90.97
+ 0.97
RKD MLP
62.61
1.30
74.86
1 0.58
82.28
1 0.40
71.54
1. 0.33
81.97
0.39
92.41
1. 0.90 92.86 1. 0.26
Impron:
+0.88%
+2.63%
+3.26%
+2.05%
+1.58%
+1.20%
+2.08%
166.15 0.68
06.47
+ 0.74
75.60 = 0.59 60.62
= 0.26
72.70 + 1.56
81.10 = 0.75 87.03 + 0.38
SW Entropy
45.86 + 8.28
46.77
+ 3.72
76. - 1.98 57.01
+ 1.09
18.21 + 8.94
63.64 + 1.47
34.14 + 2.31
SGC
68.23 1: 0.82
69.83
1 0.99
78.32
1 0.64
65.05
1: 1.18
78.11
1 0.65
87.52
1 0.64
89.98 1 0.75
Cluster
70.07
0.66
71.39
0.63
80.42 1 0.19
65.42
1 0.75
80.15
105::000 0.70
84.56
+ 0.35
90.35
1 0.07
GLNN
66.54
66.65
0.36
76.70
0.41
63.14
1.12
75.48
2.60
0.66
RKD-MIP P
71.08 + 1.00
71.45
4 1.49
81.70
+ 0.26 66.04
+ 0.88
81.23 + 0.93
89.03 + 0.68
92.75 +: 0.69
Improx:
+6.82%
+7.20%
+6.52%
+4.59%
+7.62%
+4.95%
+5.29%
Table 8: Hyperparameters of different GNN teachers small datasets: Cora, CiteSeer, PubMed, WikiCS, Mamazon-Computee Samazon-Phote and Coauthor-CS.
SAGE
GCN
GAT
APPNP
SGC
layers hidden dim
2
128
128
128
128
128
lcarning rate
0.01
0.01
0.01
0.01
0.1
weight decay
0.0005
0.001
0.01
0.01
0.001
dropout
0.0
0.8
0.6
0.5
0.0
attention heads
power iterations
10
calls for more advanced KD techniques (in addition to distill soft labels) to further bridge the performance gap between MLP student and GNN model on extremely large datasts. However, the promising observation is that, by removing those noisy nodes, the awisy-aware MLP student could consistently achieve better results than the strong baseline GLNN. It further confirms our motivation in considering noise-aware MLP student training in this paper.
4.13
100/00
Figure& 8: The impacts of incorrectly predicted nodes on the MLP student under different noise ratios.
18

---

Under review as a conference paper at ICLR 2023
Table Hyperparameters of GraphSAGE and GCN on ogbn-arxiv dataset, while ClusterGCN and
GraphSAINT on ogbn-products dataset according to (Hu et al. 2021a)
SAGE
GCN
ClusterGCN
GraphSAINT
# layers hidden dim
3
3
3
256
256
256
256
learning rate
0.01
0.01
0,001
0.01
dropout
0.5
0.5
0.5
0.5
partition number
15,000
batch size
32
20,000
SAGE-FubeeC
SAME-WWICS
030
024
0.00
CDL, Peak
CCU Natel
CCL WACS
34
23
375
Mind-Cors
30
Figure 9: The entropy distributions of wrongly and correctly predicted groups generated by 5 GNN backbones Cora, PubMed, and WikiCS datasets. Each row presents GNN model.
F PERFORMANCE ON GNN STUDENT MODELS
In this section, test the applicability of our proposal over GNN student models. Specifically, follow previous experimental protocols, we adopt GNN model as student, which has the same architecture with the GNN teacher accordingly. Table 10 reports the results under transductive setting. From
19

---

Under review as a conference paper at ICLR 2023
Table 10: Node classification accuracy on commonly used graph datasets transductive learning. The student model is GNN architecture.
Teacher Student
CiteSeer
PubMed
WikiCS
Compute
Pboto
CS
SAGE
79.70
0.52 68, 59.59 0.27 76.55 0.29 65.59 0.88 82.97 2 2.16 90.90 0.81
90.56
0.38
SAGE
83.17
0.77
71.86 0.16
79.54 0.62
70.35 1.03
84.34 0.35 93.35 0.14
92.11 0.01
82.09
= 0.28
69.62
0.28
78.38
0.27 67.29
0.64
82.93 = 0.67 91.09
0.49 90.31 + 0.25
GCN
GCN
83.94
0.67
72.03
0.00
80.30
0.00 70.20
0.50
83.70
0.70 91.98
0.21
92.17 + 0.14
APPNP
81.59 1: 0.64
70.44
0.21
79.68
0.19
67.84
1.08
81.67
1.21 91.92
0.95
90.69 0.28
APPNP
83:77 = 0.70
72.70
0.18
81.43
0.49
72.13
1 0.52
83.58 = 0.12 92.65 : 0.44 93.23 : 0.07
GAT
82.64 4: 0.63 69.60
0.42
77.80
0.25
68.66
1: 0.97
81.90 1: 1.51 91.42 1: 0.74 89.73 0.72
84.40
71.85
79.84
0.46
83.13
93.02
90.85 2
SGC
80.59 1: 0.47 69.59
0.31
78.30
0.14
67.78
1: 0.59
82.70
0.64 91.30 1: 0.51
90.13 0.38
$3.82 0.07
71.57
0.46
80.44
0.07 70.09
0.17 84.05
0.43 92.47 0.45 92.11 0.01
the results, we can observer that the GNN students could perform generally better than their teacher models acorss 7 datasets. These results further verify the effectiveness of our meta- meta-policy in achieving reliable knowledge distillation.
2.0
0.75
0755
2.7
2.5
Can
232
0.2
02
Figure 10: The impacts of trade-off parameter on RKD-MLP.
G THE IMPACTS OF X
In this section, we analyze the impacts of À on our RKD-MLP model. Specifically, we vary À from 0.1 L.0 with step size 0.1 and report the results on Figure I0
From the figures, can observe that our RKD-MLP model performs generally stable when 0.1 < S 0.9. When 1.0, it reduces to vanilla MLP baseline, so the performance drops significantly.
20


Reviews:
[{"review_id": "liwbPRj9nD", "reviewer": "Reviewer_ZL6r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to distill a better MLP from the GNNs via reliable knowledge distillation. In particular, the authors introduce a meta-policy to filter out those unreliable soft labels. The authors conduct extensive experiments on multiple graph datasets.", "summary_of_the_review": "The current version of the draft needs some work. The motivation is unclear and the paper lacks novelty in terms of its approach. The paper also needs to 1) double-check the correctness of some results and statements, 2) provide supporting details for the experiments, 3) provide explanations and analyses, and 4) improve the readability. Many parts of the paper are ambiguous and missing important details.", "strength_and_weaknesses": "Strengths:\n\n(+) How to learn a better MLP from GNNs is an interesting question. Considering GNNs might suffer from the scalability issue, how to learn an MLP as a replacement deserver some effort.\n\n(+) This paper conduct experiments on nine datasets. In addition, the authors report mean and standard deviation for most of the experiments, which is nice.\n\nWeaknesses:\n\n(-) Vague and confusing motivation. The authors claim that “directly leveraging soft labels from the GNN teacher is suboptimal” but don’t have a reasonable explanation. Why “a large portion of unlabeled nodes will be incorrectly predicted by GNNs”? What “limited generalization ability”? The “100% accuracy on the training set, yet their test accuracy is merely around 80%” seems like an over-fitting problem. How is this performance gap between training and testing relevant to “leveraging soft labels from the GNN teacher is suboptimal”?  \n\n(-) Lack of novelty. The authors simply introduce a meta-policy to filter out unreliable soft labels based on node representations. The idea of distilling confident labels is not new [1, 2, 3], and methods using reinforcement learning meta-policy to determine confidence also exist [4, 5]. The author needs to reconsider the novelty of the proposed method and make new contributions, not just a combination of existing techniques.\n\n[1] Müller, Rafael, Simon Kornblith, and Geoffrey E. Hinton. \"When does label smoothing help?.\" NeurIPS, 2019\n\n[2] Li, Yuncheng, et al. \"Learning from noisy labels with distillation.\" ICCV. 2017\n\n[3] Yang, Cheng, Jiawei Liu, and Chuan Shi. \"Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework.\" WWW, 2021\n\n[4] Kostas, James, et al. \"High Confidence Generalization for Reinforcement Learning.\" ICML, 2021\n\n[5] Jordan, Scott, et al. \"Evaluating the performance of reinforcement learning algorithms.\" ICML, 2020\n\n(-) some technical details are unclear and incorrect. For example, Figure 1 (middle) is really confusing. GLNN contains an MLP student as the yellow line, what is the meaning of the red line MLPs-student? Are they different or are they the same? How is the MLPs-student implemented? Why the MLPs-student can perform much better (~0.78 accuracy) than the SAGE teacher when the noise ratio is zero? In the leaderboard of ogb-arxiv (https://ogb.stanford.edu/docs/leader_nodeprop/#ogbn-arxiv), the best model to date can only achieve 0.7719 accuracy, while MLPs-student already outperforms the state-of-the-art. I could not find any supporting claims/experimental results in the introduction or in the later experiment sections. The only time Figure 1 (middle) gets mentioned is in the preliminary section without any experimental details. I highly doubt the correctness of the figure. In addition, for the statement “GLNN-label is a variant of GLNN by excluding unlabeled nodes” - should it be GLNN-without-unlabel? All three models in Figure 1 (left) utilized labels, and GLNN-label cannot clearly distinguish the model.\n\n(-) More supporting details are needed for some experiments. For example, In RQ2, the authors claim that “unlike small datasets, it is hard to train the MLP student on large graphs due to soft-label noises”. However, the soft-label noise difference between small and large datasets are not shown. In addition, what are the performances of SW, Entropy, and Cluster methods on large datasets?\n\n(-) Lack of theoretical explanations or analyses. The motivation of this paper and the designs of the model is based on the “noisy soft labels”. However, why the accounting of the noises helpful is overlooked? How is the proposed model capture the noises and eliminate them? Why does elimination work? And how it works? How helpful can it be? For example, for Figure 4, the authors need to provide explanations for why RKD-MLP can perform well under noisy graph topology and noisy features. How are the graph topology and features relevant to the noisy soft labels? \n\n(-) Poor readability. A lot of figures and claims in this paper are ambiguous and hard to understand, as I mentioned in the previous comments. For example, Figure 6 is really confusing. What do different colors indicate? What is the difference between different datasets? What is the meaning of the proposed model and GNN having the same predictions? In Figure 2, what is the meaning of different colors?\n\n(-) Copy and paste Table 5 in RQ6 from the GLNN paper without reference. Table 5 in the paper is exactly the same as Table 4 in the GLNN paper. No reference is given. It seems the proposed method will not introduce any computational requirements in the inference stage. Will it introduce a heavy computation burden in the training stage?\n\n[6] Zhang, Shichang, et al. \"Graph-less neural networks: Teaching old mlps new tricks via distillation.\" ICLR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NF18eRpSLD", "reviewer": "Reviewer_aYE5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to design a model which can achieve better accuracy and efficiency. The proposed method RKD-MLP utilizes knowledge distillation techniques and the idea similar to noise-label learning to achieve the mentioned goals. RKD-MLP aims to learn the 'reliable' soft label (which is called meta-policy in this paper) to guide the student network in learning a better MLP network.\n\nDifferent from existing works, this paper is using the joint training approach to train the network under the reinforcement learning framework i.e., policy gradient with advantage. ", "summary_of_the_review": "Overall, considering the above points, especially the novelty and experiments, I think this paper is not good enough for acceptance.", "strength_and_weaknesses": "Strengths:\n1. It is an interesting approach that aims to achieve better accuracy and efficiency for GNN.\n2. The idea is intuitionistic and easy to follow and understand.\n\nWeaknesses:\n1. As outlined in the title, the proposed method aims to boost accuracy and efficiency. However, the crucial experiments on the efficiency are placed in the appendix which is not reader-friendly. If I only go through the main context of this paper, I will have no idea how the proposed method can boost efficiency. There is no clue that why the proposed method can achieve good efficiency. It takes too many spaces to report the accuracy results in your main text, and some of them should be swapped with the efficiency results in your Appendix.\n2. As the authors mentioned in the paper, the meta-set is the validation set. How the dataset is split? How does the size of the \"meta-set\" (i.e., validation set) affect your performance in terms of accuracy and efficiency? I think this question is worth investigating and quite important. If the proposed model can achieve a better performance than existing methods by using a small \"meta-set\", it can provide some evidence of the efficiency.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HfGE0gQqa_7", "reviewer": "Reviewer_GWUx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed Knowledge Distillation (KD) method for Graph Neural Networks. \nThe proposed method is motivated by a recent study that shows that  GNNs can be compressed to inference-friendly multi-layer perceptrons (MLPs) by training MLPs using the soft labels of labeled and unlabeled nodes from the teacher. \nHowever, leveraging the soft labels of all unlabeled nodes may be suboptimal since the teacher model would inevitably make wrong predictions.\nThis paper's proposed method, called: Reliable Knowledge Distillation for MLP optimization, is the first noise-aware knowledge distillation framework for GNNs. Its core idea is to use a meta-policy to filter out those unreliable soft labels. For training a meta-policy, the authors propose a reward-driven objective based on a meta-set and adopt a policy gradient to optimize the expected reward. \nAfterward, the authors use the meta-policy to the unlabeled nodes and select the most reliable soft labels for distillation. \nThe extensive experiments across various GNN models,  7 small graphs, and 2 large-scale datasets demonstrate the superiority of the proposed method to the prior art.\n", "summary_of_the_review": "This paper proposed Knowledge Distillation (KD) method for Graph Neural Networks. \nThe proposed method is motivated by a recent study that shows that  GNNs can be compressed to inference-friendly multi-layer perceptrons (MLPs) by training MLPs using the soft labels of labeled and unlabeled nodes from the teacher. \nThe method is well-explained, and evaluations are solid. My main concern is that there are still hand-crafted GNN-based architectures that outperform the proposed method's performances. See the SOTA result on paperwithcodes. I would like the authors to elaborate on this regard. Specifically, the authors conclude from the results of table 10 that they manage to achieve better performance of the student rather than the performance of the teacher. Are the same phenomena would preserve also using SOTA backbones as a teacher? \n\nIn overall, I am positive about the novelty and the impact of the proposed method, looking forward receiving responce on the raised issue.\n\n\n ", "strength_and_weaknesses": "*Strength*\n\nThe paper is well-written, and the method is well-explained. \nThe evaluation of the proposed is very impressive, including various datasets under both transductive and inductive settings\nThe authors performed an extension ablation study of the proposed dataset, showing how different model parameter choices impact performance. \n\n*Weakness*\nMy major concern is why the authors haven't evaluated their method with SOTA architectures GNNs as teachers.\nPlease see the paperwithcode SOTA benchamrks. \n \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2pYQajd8In", "reviewer": "Reviewer_asz6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new semi-supervised KD framework (RKD-MLP) to filter out unlabeled nodes whose soft labels are likely to be incorrectly predicted by teachers. In this framework, they use a simple reinforcement learning framework to learn whether nodes have reliable soft labels, thus improving performance and maintaining inference efficiency. The experiments show the benefits of RKD-MLP.", "summary_of_the_review": "Overall, this paper has a clear motivation and proposes an effective method. But in the experiment, most of the results are about the performance of GNNs students. It would be nice to have more results about the meta-policy.", "strength_and_weaknesses": "Strength:\n1. They study the impact of unlabeled nodes in GNNs distillation and demonstrate the importance of reliable pseudo labeling.\n2. They provide a simple RL-based method to filter out unlabeled nodes with unreliable predicted soft labels.\n3. Experiments show that the new method can improve the accuracy of different datasets.\n\nWeakness:\n1. Including more analysis on whether the meta-policy can filter out unreliable nodes would be better. Figure 6 shows some results. But the GNN prediction is used as the ground truth, which may be incorrect. One possible way is to test the policy on some labeled nodes in the testing set.\n2. I would like to know how many unlabeled nodes are removed by the meta-policy. If too many nodes are removed, we may lose too much information from the data. The meta-policy uses 0.5 as the threshold, have you tried other thresholds?\n3. This framework is doing something similar to out-of-distribution detection. It would be great to provide some discussion on connections or performance comparisons between the two.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject