VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs
Authors: Ling Yang
Ye Tian
Minkai Xu
Zhongyi Liu
Shenda Hong
Wei Qu
Wentao Zhang
Bin CUI
Muhan Zhang
Jure Leskovec
Venue: ICLR 2024 poster (2024)
Keywords: Graph Knowledge Distillation
Efficient Graph Learning
Primary area: learning on graphs and other geometries & topologies
TL;DR: We propose a new powerful graph representation space for GNN-to-MLP distillation.

Abstract: GNN-to-MLP distillation aims to utilize knowledge distillation (KD) to learn computationally-efficient multi-layer perceptron (student MLP) on graph data by mimicking the output representations of teacher GNN. Existing methods mainly make the MLP to mimic the GNN predictions over a few class labels. However, the class space may not be expressive enough for covering numerous diverse local graph structures, thus limiting the performance of knowledge transfer from GNN to MLP. To address this issue, we propose to learn a new powerful graph representation space by directly labeling nodes' diverse local structures for GNN-to-MLP distillation. Specifically, we propose a variant of VQ-VAE to learn a structure-aware tokenizer on graph data that can encode each node's local substructure as a discrete code. The discrete codes constitute a codebook as a new graph representation space that is able to identify different local graph structures of nodes with the corresponding code indices. Then, based on the learned codebook, we propose a new distillation target, namely soft code assignments, to directly transfer the structural knowledge of each node from GNN to MLP. The resulting framework VQGraph achieves new state-of-the-art performance on GNN-to-MLP distillation in both transductive and inductive settings across seven graph datasets. We show that VQGraph with better performance infers faster than GNNs by 828×, and also achieves accuracy improvement over GNNs and stand-alone MLPs by 3.90% and 28.05% on average, respectively. Our code is available at https://github.com/YangLing0818/VQGraph

Published as a conference paper at ICLR 2024
VQGRAPH: RETHINKING GRAPH REPRESENTATION SPACE FOR BRIDGING GNNS AND MLPS
Ling Yang' Ye Tian
Minkai Xu
Zhongyi Liu? Shenda Hong'
Wei Qu?
Wentao Zhang' Bin Cui Muhan Zhanglt Jure Leskove
'Peking University Ant Group Stanford University
yangling0@118116.co com, tyfeld@stu.pku. pku. pku.edu.cn, {zhongyi. 1zy, gingze.qujeantgrenn
com, fminkai, (treeeesssstaafrrree (hongshenda, wentao. zhang, bin.cui, (phk.e@pkk..duu
ABSTRACT
GNN-to-MLP distillation aims utilize knowledge distillation (KD) learn computationally-effiiio multi-layer perceptron (student MLP) on graph data by mimicking the output representations of teacher GNN. Existing methods mainly make the MLP to mimic the GNN predictions over few class labels. However, the class space may not be expressive enough for covering numerous diverse local graph structures, thus limiting the performance of knowledge transfer from GNN to MLP. To address this issue, we propose to learn a new powerful graph represen- tation space by directly labeling nodes' diverse local structures for GNN-to-MLP distillation. Specifically, we propose a variant of VQ- (Van Den Oord et al., 2017) to learn structure-await tokenizer graph data that can encode each node's local substructure as a discrete code. The discrete codes constitute a code- book a graph representation space that is able to identify different local graph structures nodes with the corresponding code indices. Then, based on the learned codebook, we propose a new distillation target, namely soft code as- signments, to directly transfer the structural knowledge of each node from GNN MLP. The resulting framework VQGRAPH achieves new state-of-the-arr per- formance on GNN-to-MLP distillation in both transductive and inductive settings across seven graph datasets. We show that VQGRAPH with better performance infers faster than GNNs by 828x, and also achieves accuracy improvement over GNNs and stand-alone MLPs by 3.90% and 28.05% on average, respectively. Our code is available at https://github.co com/ htt.ettt...oo/aaagniiiiia YangLing08 (ingg881///Qraph
INTRODUCTION
Graph Neural Networks (GNNs) (Yang & Hong, 2022; Li et al., 2020a; Perozzi et al., 2014; Xu al., 2019; Morris et al., 2019; Yang al., 2020; Chen et al., 2020b) have been widely used due to their effectiveness in dealing with non-Euclidean structured data, and have achieved remarkable per- formances in various graph-related tasks (Hamilton et al., 2017; Kipf Welling, 2017; Veličković al., 2018). Modern GNNs rely message passing mechanism to learn node representations (Yang et al., 2020). GNNs have been especially important for recommender systems (Fan et al., 2019; He al., 2020; Wu et al., 2022; Xiao al., 2023; Zhang al., 2024), fraud detection (Dou al., 2020; Liu et al., 2021; Yang et al., 2023), and information retrieval (Li et al., 2020b; Mao et al., 2020). Numerous works (Pei al., 2020b) focus on exploring more effective ways to leverage informative neighborhood structure for improving GNNs (Park et al., 2021; Zhu et al., 2021; Zhao et al., 2022; Tang et al., 2022: Lee al., 2021; Chien et al., 2022; Abu-El-Haija et al., 2019).
It is challenging to scale GNNs to large-scale applications which are constrained by latency and require fast inference (Zhang et al., 2020; 2022a; Jia al., 2020), because message passing neces- sitates fetching topology and features of many neighbor nodes for inference on a target node, which
"Contributed equally.
'Coresponding authors.

---

Published as a conference paper at ICLR 2024
Class id:
Class id: 2
Class id:
Class id: 5 Token idi 489
Class id: 5 Tokeen id: 751
Class id: 5 Token id: 1302
Class id: Token idi 1472
Class id: 3 Token id: 2208
Class Token id: 1792
(a) NOSMOG
(b) VQGraph (Ours)
Figure 1: The t-SNE visualization of the learned graph representation space in two kinds of teacher GNNs: (a) previous SOTA "class-based" NOSMOG (Tian al., 2023b) and (b) our "structure- based" VQGraph. "class-based" denotes learning with class labels, and "structure-based" denotes learning with our local structure reconstruction. Our learned space is more compact. We here pro- vide both class labels and our structure labels along with illustrative substructures for demonstration.
is time-consuming and computation-intensiiv.. Multi-layer perceptrons (MLPs) are efficient alter- natives deploy on graphs that only depend on node feature without the need of explicit message passing (Zhang et al., 2022b). Thus, recent methods use knowledge distillation (Hinton et al., 2015; Tian al., 2023a; Gou al., 2021; Yuan al., 2020; Zhou & Song, 2021) to transfer the learned structural knowledge from GNNs to MLPs (Zhang et al., 2022b; Zheng et al., 2022; Tian et al., 2023b), which build the statistical associations between node features and class labels by making the MLP mimic the (well-trained) GNN's predictions. Then only MLPs are deployed for inference, which can also perform well in real-world graphs.
Despite some progress, current GNN-to-MLP distillation methods have common fundamental is- sue: their graph representation spaces of GNN are mainly learned by few class labels, and the class space may not be expressive enough for covering numerous diverse local graph structures of nodes, limiting the distillation performance. We explain this problem by using t-SNE (Van der Maaten & Hinton, 2008) to visualize the graph representation space in Figure 1. We graph representation space in previous teacher GNN is not expressive enough for identifying fine- grained local structural differences between nodes of the same class, which may limit the structural knowledge transfer from GNN to MLP.
observe that the
Here we introduce new powerful graph representation space for bridging GNNs and MLPs by directly labeling diverse nodes' local structures. Specifically, we propose variant of IQ-VAE (Van Den Oord al., 2017) learn structure-awart tokenizer on graph data that can encode each node with its substructure as a discrete code. The numerous codes constitute codebook as our new graph representation space that is able identify different local neighborhood structures of nodes with the corresponding code indices. As demonstrated in Figure 1, our learned representation space is more expressive and can identify subtle differences between nodes' local structures. Based on the effectively facilitate the structure-based distillation by maximizing the
leamned codebook, consistency of soft code assignments between GNN and MLP models, given by the KL divergence between GNN predictions and MLP predictions over the discrete codes of the codebook.
We highlight our main contributions as follows: (i) To the best of our knowledge, we for the first time directly learn to label nodes' local neighborhood structures to acquire powerful node representation space (i.e., a codebook) for bridging GNNs and MI.s. (ii) Based on the learned codebook, we utlize a new distillation target with soft code assignments effectively facilitate the structure-aware knowledge distillation. We further conduct both visualization and statistical analyses for better understanding with respect to our superior local and global structure awareness for GNN-to-MLL distillation. (iii) Extensive experiments across seven datasets show VQGRAPH can consistently outperform GNNs by 3.90% on average accuracy, while enjoying 828x faster inference speed. Also v VQGRAPH outperforms MIPs and SOTA distillation method NOSMOG (Tian et al., 2023b) by 28.05 % and 1.39 % on average accuracy across datasets, respectively.
RELATED WORK
Inference Acceleration for GNNs Pruning (Zhou et al., 2021) and quantizing GNN parameters (Zhao al., 2020) have been studied for inference acceleration (Chen al., 2016; Judd et al.,
2

---

Published as a conference paper at ICLR 2024
2016: Han et al., 2015: Gupta al., 2015). Although these approaches accelerate GNN inference to a certain extent, they do not eliminate the weighbor-fetching latency. Graph-MLP (Hu et al.. 2021) proposes to bypass GNN neighbor fetching by learning a computationally-effiii. a neighbor contrastive loss, but its paradigm is only transductive and can be not applied in the more practical inductive setting. Besides, some works try speed up GNN in training stage from the perspective of node sampling (Zou et al., 2019; Chen et al., 201 10),c, which are complementary to our goal on inference acceleration.
MLP model with
Knowledge Distillation for GNNs Existing GNN-based knowledge distillation methods try to dis- till teacher GNNs to smaller student GNNs (GNN-GNN distillation) or MLPs (GNN-to-MLP dis- tillation). Regarding the GNN-GNN distillation, LSP (Yang et al., 2021c). TinyGNN (Yan et al., 2020), GFKD (Deng& & Zhang. 2021), and GraphSAlI. (Xu et al., 2020) conduct KD by enabling student GNN to maximally preserve local information that exists in teacher GNN. The student in CPF (Yang et al.. 2021b) is not GNN, but is still heavily graph-dependent as it uses LP (Zhu & Ghahramanih, 2002; Huang al., 2021). Thus, these methods still require time-consuming, neigh- bors fetching. To address these latency issues, recent works focus on GNN-to-MLP distillation that does not require message passing, as seen in (Hu et al., 2021; Zhang al., 2022b; Zheng et al., 2022: Tian et al., 2023b). For example, recent sota methods GLNN (Zhang et al., 2022b) and NOS- MOG (Tian et al., 2023b) train the student MLP with node features as inputs and class predictions from the teacher GNN as targets. However, class predictions over a few labels, as their distilla- tion targets, can not sufficiently express structural knowledge of graph structures as discussed in Sec. I. Hence, we for the first time propose to directly label nodes' local neighborhood structures to facilitate structure-aware knowledge distillation.
PRELIMINARIES
Notation and Graph Neural Networks We denote a graph as G (V. with V {v), {u,,02, u.,-.. , } J represents all nodes, and A denotes adjacency matrix, with Aij = if node v; and node v, are connected, and O otherwise. Let N denote the total number of nodes. X E RND represents the node feature matrix with each raw being a D-dimensional node attribute v. For node classification, the prediction targets are Y E R-NK where row ye is a K-dim one-hot vector for node v. For given G. usually small portion of nodes will be labeled, which we mark using su- perscript L, i.e. VL_ XL and y The majority of nodes will be unlabeled, and we mark using the superscript U/ i.e. D Y and Y For given node v E V. GNNs aggregate the messages from node neighbors (wv) to learn node embedding h. E Rd with dimension d,, Specifically, the node embedding in 1-th layer h! is learned by first aggregating (AGG) the neighbor embeddings and then updating (UPDATE) it with the embedding from the previous layer. The whole learning process can be denoted as: h (f) UPDATE(H (1-1) AGGG(hu (1-1), u E ((o))))
Vector Vaariizid-a AutoEncoder (VQ-VAE) for Continuous Data The VQ-VAE model (Van Den Oord et al., 2017) is originally proposed for modeling continuous data distri- bution, such images, audio and video. It encodes observations into a sequence of discrete latent variables, and reconstructs the observations from these discrete variables. Both encoder and decoder use a shared codebook. More formally, the encoder is non-linear mapping from the input space, x, to a vector F(x)). This vector is then quantized based on its distance to the prototype vectors (tokens) in the codebook et, k E K. such that each vector F(x:) is replaced by the index of the nearest code in the codebook, and is transmitted to the decoder: (iaattzzeE(x))
ek where k arg min, [|E(x) - e, To learn these mappings, the gradient of the reconstruction error is then back nack-propagated through the decoder, and to the encoder using the straight-through gradient estimator (Bengio al., 2013). Besides reconstruction loss, VQ- VQ-VAE has two additional terms to align the token space of the codebook with the output of the encoder. The codebook loss, which only applies to the codebook variables, brings the selected code e close to the output of the encoder, Exx). The commitment loss, which only applies to the encoder weights, encourages the output of the encoder to stay close to the chosen code to prevent it from fluctuating too frequently from one code vector to another. The overall objective is:
L(x, D(e)) = ||x D(c)l + [|sg(E(x)) - ell2 + n||sgíc] - E(x)
(1)
where F. is the encoder function and D is the decoder function. The operator sg refers stop-
3

---

Published as a conference paper at ICLR 2024
Graph Tokenizer Training
Up
lej
Reconstrud Graph
Shrrught-Trrouh Gradients V,L
Teacher GNN
Linear Decoder
Assigned Indexes,
Lookup
Code Incex
M
Learned GNN and Codebook
Structure-Labeled Codebook
Code Embadding:
Center Node
Student
MM
Feodie abeit!
MLP
MLP
KL Divergence between Soft Code Assignments
Euss Letuss
Traditional Knowledge
Computing 1-v8-M Relationships
MMLP
Distillation
M pGNN
Subgraph
Teacher
h GWW
Gradient Flow
Figure 2:
The schematic diagram of VQGRAPI, including graph tokenizer training (Top) and structure-awe aware code-based GNN-to-MLP Distillation (Bottom).
gradient operation that blocks gradients from flowing into its argument, and 1j is hyperparameter which controls the reluctance to change the code corresponding to the encoder output. In this paper, we explore the potential of VQ-VAE for representing discrete graph-structured data.
VQGRAPH
The critical insights of VQGRAPH is learning an expressive graph representation space that directly labels nodes' diverse local neighborhood structures with different code indices for facilitating ef- fective structure-aware GNN-to-MLP distillation. First, we learn a structure-aware graph tokenizer to encode the nodes with diverse local structures corresponding discrete codes, and constitute a codebook (Sec. 4.1). Then we utilize the learned codebook for GNN-to-MLP distillation, and propose tailored structure-aware distillation objective based soft code assignmnets (Sec. 4.2).
4.1 GRAPH TOKENIZER TRAINING
Labeling Nodes' Local Structure with Discrete Codes Similar to the tokenization NLP (Sen- nrich et al., 2016: Wu al., 2016), we tokenize the nodes with different neighborhood structures as discrete codes using variant of VQ-VAE (Van Den Oord et al., 2017), i.e., a graph tokenizer that consists of a GNN encoder and a codebook. More concretely, the nodes = (1 U2. ... v, of graph G are tokenized to 2 {=1, 22, zn where the codebook contains M discrete codes. Firstly, the teacher GNN encoder encodes the nodes to nodes embeddings. Next, our graph tokenizer looks up the nearest neighbor code embedding in the codebook for each node embedding h, Let E [e,, e2, ... m E Ik MD denote the codebook embeddings, which are randomly initialized and are then optimized in pretraining. The assigned code of i-th node is:
2; = argmin, h, - ejl2,
We feed the corresponding codebook embeddings {Ezr '... ez,. to the linear decoder (p ez, - .z) reconstruct the input graph including both nodes attributes X NxD and adjacency matrix A E N for an end-to-end optimization of our graph tokenizer.
Graph Tokenizer Optimization We adapt VQ-VAE (first term in Equation (1)) to fit our graph tokenization. In addition, the categorical information is also critical for node representations, thus we integrate it into the optimization of our graph tokenizer:
LRee
A - a
N
|
node reconstruction
edge reconstruction
(3)
N
N h1: 11:12.
Lrnkeniser
LRee LCR(M(V.)) ((e,)) )
||s][h:]]
ex
2
[gg[e,,)
N
N
4

---

Published as a conference paper at ICLR 2024
where ú E R and X E RND denote the predicted node embedding and node embedding matrix, respectively. sgl.] stands for the stopgradient operator, and the flow of gradients is illustrated in Figure 2. Roe denotes the graph reconstruction loss, aiming to preserve node attributes by the first node reconstruction term with the scaled cosine error (7 2 1), and recover graph structures by the second topology reconstruction term. LCE is the cross-entropy loss between labels y(v.) and the GNN predictions V(es,) that are based the assigned codes {ez,). In Looenizers the third term is VQ loss aiming to update the codebook embeddings and the forth term is commitment loss that encourages the output of the GNN encoder to stay close to the chosen code embedding. 7 is hyper-paramette set to 0.25 in experiments. With the learned graph tokenizer, acquire powerful codebook that not only directly identifies local graph structures, but also preserves class information for node representations, facilitating later GNN-to-MLP distillation.
Clarifying Superiority over VQ-VAE and Graph AutoEncoders In contrast to vanilla VQ- VAE, we provide a new variant of VQ- VAE for modeling discrete graph data instead of continuous data, and utilize the learned codebook for distillation task instead of generation tasks. In contrast to traditional graph autoencoders (Kipf & Welling. 2016), our model does not suffer from large variance (Van Den Oord et al., 2017). And with expressive latent codes, we effectively avoid "posterior collapse" " issue which has been problematic with many graph AE models that have powerful decoder, often caused by latents being ignored. We provide experimental comparison to demonstrate our superiority in Sec. 5.3.
Scaling to Large-Scale Graphs We have introduced the main pipeline of graph tokenizer training based the entire graph input. Nevertheless, for large-scale industrial applications, one can not feed the whole graph due to the latency constraint (Fey et al., 2021; Bojchevski et al., 2020; Ying et al., 2018; Chen al., 2020a). Numerous researches choose subgraph-wist sampling methods as a promising class of mini-batch training techniques (Chen et al., 2018b;a; Zeng et al., 2020; Huang et al., 2018; Chiang al., 2019; Zou et al., 2019; Shi et al., 2023), implicitly covering the global context of graph structure through a number of stochastic re-sampling. We follow this technique to perform large-scale graph tokenizer training. For example,
adopt GraphSAGE (Hamilton et al., 2017) as teacher GNN, we samples the target nodes as a mini-batch Vaample and samples a fixed size set of neighbors for feature aggregation. We utilize the connections between the target nodes as the topology reconstruction target (the second term of Lee in Equation (3)) for approximately learning global graph structural information.
4.2 STRUCTURE-AWAN CODE-BASED GNN-TO-MLE DISTILLATION
After the graph tokenizer optimization, we obtain a pre-trained teacher GNN encoder and set of codebook embeddings E. We hope distill the structure knowledge node-by-node from the GNN to a student MLP based on the codebook. Next, we will introduce our tailored structure-awart GNN-to-MI.P distillation with the soft code assignments over the learned codebook.
Aligning Soft Code Assignments Between GNN and MLP Different from previous class-based distillation methods (Zhang et al., 2022b; Tian et al., 2023b) that constrain on class predictions be- tween GNN and MLP, we utilize more expressive representation space of graph data, i.e., our structure-aware codebook, and propose code-based distillation to leverage more essential informa- tion of graph structures for bridging GNN and MLP. Formally, for each node v, we have its GNN representation h GNN E RR" and the MLP representation h. E R." Then we respectively compare their node representations with all M codes of the codebook embeddings E RMXD and obtain corresponding soft code assignments TNNN
M and M.P R;
GNN COMP(N GNN E), ri
MLP COMP(M MLP EE), P
where COMP: [R_ R M - AM can be arbitrary relation module for computing 1-vy-M code- wise relations, and such relations can be viewed as assignment possibilities. We use L2 distance in experiments (more studies in Appendix C. Kindly note that the codebook size M can be large, especially for large-scale graphs, thus the soft code assignment of each node contains abundant 1- M global aructure-discriiinaaie information. Therefore we choose the soft code assignment as
5

---

Published as a conference paper at ICLR 2024
the target for final distillation:
N
1 2 KL
Leade distitl
GNN ||
2
P.
GNN
log
(5)
N
1=1
f=1
where KL refers to Beclbackk-eibber divergence with:
Softmax(r.. /T), p. Softmax (r. (LP /T),
(6)
being the scaled code assignments, and 7 is the temperature factor to control the softness. Kindly note that the code assignment only used for optimizing the training, and we remove it for deploying the pure MLP model. In this way. VQGRAPH is able to effectively distill both local neighborhood structural knowledge and global structure- distrminaiiee ability from GNNs to MLPs, without in- creasing inference time. The overall training loss of VQGRAPH is composed of classification loss feis: traditional blass-based distillation loss Leluss distill. and our code-based distillation loss, i.e.,
LVQGRAPH = Lets + aLstass distill BLoode distill-
where a and are factors for balancing the losses.
EXPERIMENTS
Datasets and Evaluation We use five widely used public benchmark datasets (Zhang et al., 2022b; Yang et al., 2021a) (Citeseer, Pubmed, Cora, A-computer, and A-photo), and two large OGB datasets (Hu et al., 2020a) (Arxiv and Products) to evaluate the proposed model. In our experiments, report the mean and standard deviation of ten distinct runs with randomized seeds to ensure robustness and reliability our findings. We also extend our VQGRAPH to heterophilic graphs and make performance improvement in Appendix A.2. We utilize accuracy to gauge model performance. More details are in Appendix B.I. Model Architectures For fair comparison, we adopt GraphS with GCN aggregation as our teacher model (also as graph tokenizer) and use the same student MLP models for all evaluations following SOTA GLNN (Zhang et al., 2022b) and NOSMOG (Tian et al., 2023b). The codebook size increases accordingly with dataset size (studied in Sec. 5.3). For example, we set 2048 and 8192 for Cora and A-phot oo respectively. More model hyperparameters are detailed in Appendix B.2. We investigate the influence of alternative teacher models, including GCN (Kipf & Welling, 2017), GAT (Veličković et al., 2018), and APPNP (Klicpera et al., 2019), detailed in Appendix C.2.
Transductive vs. Inductive We experiment in two separate settings, transductive (tran) and in- ductive (ind), for comprehensive evaluation. In both settings, first pre-train our graph tokenizer to learn the codebook embeddings E for code-based distillation. For the tran setting, we train our models on the labeled graph G, along with the corresponding feature matrix XL and label vector Yz, before evaluating their performance on the unlabeled data X and Y Soft labels, soft code assignments are generated for all nodes within the graph (i.c., yoore r ,GNN T.MP for V). As for ind, we follow the methodology of prior work (Tian et al., 2023b) in randomly selecting 20% of the data for inductive evaluation. Specifically, we divide the unlabeled nodes D into two sepa- rate yet non-overlapping subsets, observed and inductive (i.e., Vo = obs Li D producing three distinct graphs, G = G Lu 9% Li G wherein there are no shared nodes. In training, the edges between G Light obs and g are removed, while they are leveraged during inference to transfer posi- tional features via average operator (Hamilton al., 2017). Node features and labels are partitioned into three disjoint sets, i.e., X = X X obs Li x and Y = y L LY L. LY Soft labels and soft code assignments are generated for nodes within the labeled and observed subsets (i.e., yofft ML..P for yr. Li Dt )..) We provide code and models in the supplementary material.
re
GNN
r T
5.1 MAIN RESULTS
GNN-to-MLP Distillation We compare VQGRAPII to other state-of-the-art GNN-to-MLP dis- tillation methods GLNN and NOSMOG with same experimental settings, and use distilled MLP models for evaluations. We first consider the standard transductive setting, enabling direct compari- son with previously published literature (Zhang et al., 2022b; Hu et al., 2020b; Yang et al., 2021a). As depicted in Tab. 1, VQGRAPH outperforms all baselines including teacher GNN models across

---

Published as a conference paper at ICLR 2024
Table 1: Node classification results under the standard setting, results show accuracy (higher is better). AGNN, AMPP, A NOSMOG represents the difference between VQGRAPH and GNN, MLP, NOSMOG, respectively. GLNN and NOSMOG are the SOTA GNN-to-MLA distillation methods.
Datasets
SAGE
MLP
GLNN
NOSMOG
VQGRAPH
ACNY
AMLP
ANoSHOG
Clleseer
70.49 # 1.53
58.50 = 1.86
71.22 + 1.50
73.78 = 1.54
76,08 = 0.55
? 7.93%
130005%
? 13188
Pubmed
75.56 2.06
68.39 = 3.09
75.59 -: 2.46
77.34
= 2.36
78.40 = 1.71
? 13.76%
114.64%
11.77%
Cora
80.44 1.57
59.18 - 1.60
80.26
- 1.66
83.04
= 1.26
83.93 = 0.87
74.08%
141.8%%
1107%
A- 4-computer
82.82 = 1.37
67.62 - 2.21
82.71
= 1.18
84.04
= 1.01
85.17 = 1.29
72.84%
1 1.2.95%
11.44%
A-photo
90.85 1. 0.87
77.29
1 1.79
91.95
1 1.04
93.36
1 0.69
94.21 1 0.45
3.70%
21.899
10.1%
Arxiv
70.73 = 0.35
55.67
: 0.24
63.75
4: 0.48
71.65
- 0.29
72.43
- 0.20
7 2240%%
1.30.11%
1.93%
Products
77.17 1 0.32
60.02
1 0.10
63.71
1 0.31
78.45
1 0.38
79.17 1 0.21
2.59%
31.911
00.92%
Table 2: Node classification results in a production scenario with both inductive and transductive settings. ind indicates the results on Vnnd tran indicates the results on V Virans and prod indicates the interpolated production results of both and and Iran.
Datasets
Eval
SAGE
MLP
GLNN
NOSMOG
VQGRAPH
ACNN
AWLP
ANosMOG
prod
68.06
$8.49
69.08
70.60
73.76
1 88377%
1 1.2,11%
5.880%
Citeseer
ind
69.14 + 2.99
59.31
4.56
68.48
2.38
70.30 2.30
72.93 + 1.78
5.488
22.96%
1.74%%
tran
67.79
2.80
58.29 1.94
69.23 2.39
70.67 = 2.25
74.59 + 1.94
10.00%
1 277.96%
1 7.774%
prod 74.77
68.39
74.67
75.82
76.92
2.86%
t 12,47%
1 1,5%
Pubmed
ind
75.07
2.89
3.25
74.52
2.95
75.87 3.32
76.71 2.76
2.18
1 11.1%%
tran
74.70 +: 2,33
68.42 3.06
74.70 +: 2.75
75.80 = 3.06
77.13 = 3.01
1 3.25%
1 12.73%
1 1.75%
prod 79.53
18
77.82
81.02
81.68
2.70%
38.02%
0.811%
Cora ind
81.03
1.71
59.44
3.36
73.21 1.50
81,36 1.53
82.20 1.32
1.44%
38.29%
1.03%
truw
79.16 + 1,60
59,12 = 1.49
78.99 # 1,56
80.93 1.65
81.15 + 1.25
2.51%
37.26%
0.22%%
prod 82,73
67.62
82.
83.85
84.16
1.73%
24.4 46%
1 0.37%
A-compate>
ind
82.83 + 1.51
67.69
2.20
80.27 + 2,11
84.36
1.57 83.72
85.73 + 2.04
3.50%
26.65%
1.62%
tran
82.70
1.34
82.56 1.80
84.56 1.81
2.25%%
25.08%
1.00%
prod 90:45
77.29
91.34 34
92:47
93.05
1.877%
t 20.39%
1 0.625%
A-photoo
and
90.56
1.47
77.44
1.50
89.50
1.12
92.61
1.09
93.11
0.89
2.82%
20.24%
0.54%
tran
90.42
0.68
77.25
1.90
91.80
0.49
92.44 0.51
92.96 1: 1.02
2.811
1 20.34%
0.56%
prad 70.69
55.35
63.50
70.90
71.43
1105%
29.05%
0.75%
Arsii
Ind
70.69
0.58
55.29
0.63
59.04
0.46
70.09
0.55
70.86 0.42
0.24%
16%
1 1.10%
tran
70.69
1: 0.39
55.36
0.34
64.61
0.15
71.10 0.34
72.03 :: 0.56
1.90%
1 30.11%
1 1.31%
prad
77.33
77.93
1.300%
0.77%.
Products
ind
77.23 + 0.24
60.02
0.09
63.38 38
0.33
77.02 + 0.19
77.50 + 0.25
0.35%
1 12%
1 0.625%
76.86 1. 0.27
60.02
0.11
63.49
0.31
77.41
0.21
78.36 0.13
1.95%
30.56%
1.23%
all datasets. Specifically, VQGRAPH improves performance by average of 3.90% compared to its teacher GNN, highlighting its ability to capture superior structural information without re- lying on explicit graph structure input. Comparing VQGRAPH NOSMOG, our proposed model achieves an average improvement of 1.39% % across both small- and large-scale graph datasets. Further model analysis of VQGRAPH is presented in Sec. 5.3.
Experiments in Inductive and Transductive Settings To gain deeper insights into the effective- ness of VQGRAPH, we conduct experiments in realistic production (prod) scenario that involves both inductive (ind) and transductive (tran) settings across multiple datasets, as detailed in Tab. 2. Our experimental results demonstrate that VQGRAPHI consistently achieves superior performance compared to the teacher model and baseline methods across all datasets and settings. Specifically, our proposed method outperforms GNN across all datasets and settings with an average improve- ment of 2.93 %, demonstrating its superior efficacy of our learned code-based representation space in capturing graph structural information, even on large-scale datasets. Furthermore, when compared to MLP and NOSMOG, VQGRAPH consistently achieves significant performance improvements, with average gains of 25.81% % and 1.6 %, respectively, across all datasets and settings.
5.2 MODEL ANALYSIS
Trade-off between Performance and Inference Time To demonstrate its efficiency and capac- ity of our VQGRAPH, we visualize the trade-off between node classification accuracy and model inference time on Citeseer dataset in Figure 3. Our results indicate that achieves a highest accu- racy of 76% while maintaining fast inference time of 1.45ms. Compared to the other models with similar inference time, VQGRAPH significantly outperforms NOSMOG and MLPs by 3.12% and 30.05% % in average accuracy, respectively. For those models having comparable performance with VQGRAPH, they require a considerable amount of inference time, e.g., 2 layers GraphSAGE (SAGE-L.2) needs 152.31 and 3 layers GraphSAGE (SAGE-L33) needs 28mm, making them

---

Published as a conference paper at ICLR 2024
77.0
VVQGraph
NOSMOGWA
74.5
NDSMOGW2
Accuracy
NOSMOG
72.0
SAGE-L3
GLNN
SAGE-L2
(a) GLNN--aa
(b) NOSMOG-tea
(e) VQGraph-tea
SAGE-LI
69.5
59.5
e MLPS
100
10a
10a
10a
104
(d) GLNN-stu (e) NOSMOG-siu
(I) VQGraph-stu
Log Scale Inference Time(ms)
Figure Accuracy vs. Inference Time.
Figure t-SNE visualization of learned node rep- resentations, colors denotes different classes.
unsuitable for real-world upplications. This makes VQGRAPH 105x faster than SAGE-L2 and 828x faster than SAGE-L3. Although increasing the hidden size of NOSMOG slightly improves its performance, NOSMOGw2 (2-times wider than NOSMOG) and NOSMOGw4 still perform worse than VQGRAPH with more inference time, demonstrating the superior efficiency of our VQGRAPH.
Compactness of Learned Node Representation Space We use t-SNE (Van der Maaten & Hinton, 2008) to visualize the node representation spaces of both teacher GNN and distilled student MLP models with different methods, and put the results in Figure 4. Our VQGRAPH provides better teacher GNN model than GLNN and NOSMOG, and the node representations of same classes have more compact distribution. The representations extracted by our distilled MLP model also have a more compact distribution. We attribute these to our expressive code-based representation space, providing more structure-aware representations for classifying nodes. Besides, our code-based distillation strategy can effectively deliver both graph structural information and categorial informa- tion from GNN MLP, guaranteeing the compactness of MLP's representation space.
Consistency between Model Predic- tions and Global Graph Topology Here corroborate the superiority of VQGRAPH over GNNs, MLPs, GLNN and NOSMOG in capturing global graph structural information, which is comple- mentary to the above analysis on local structure awareness. We use the cut value effectively evaluate the alignment between model predictions and graph topology as (Zhang et al., 2022b; Tian et al., 2023b) based the approxima- tion for the min-cut problem (Bianchi et al., 2019). The min-cut problem divides nodes V into K disjoint subsets by removing the minimum number of edges. Correspondingly, the min-cut problem can be expressed as: max K Ekn(C) AC&) DCk), where C' is the node class assignment, A is the adja- cency matrix, and tr(YTAY //)/tr(YTT) DÝ)), where Y is the model prediction output, and the cut value CV indi- cates the consistency between the model predictions and the graph topology. We report the cut values for various models in the transductive setting in Tab. 3. The average CV achieved by VQ- GRAPH is 0.9493, while SAGE, GLNN, and NOSMOG have average CV values of 0.9276, 0.8725, and 0.9348, respectively. We find VQGRAPH obtains the highest cut value, indicating the superior global structure-capturing ability GNN and SOTA GNN-to-MLPP distillation methods.
Table 3: The cut value. VQGRAPII predictions more consistent with the graph topology than GNN, MLP, GLNN and the kfte-of-the-ar method NOS- MOG.
Datasets
SAGE
MLP
GLNN
NOSMOG
VQGRAPH
Citeseer
0.9535
0.8107
0.9447
0.9659
0.9786
Pubred Cora
0.9597
0.9385
0.7203
0.8908
0.9480
0.9684
A A-computer
0.8951
0.6764
0,8579
0.9047
0,9190
A A-photo
0.9014
0,7099
0.9063
0.9084
0.9177
Arxiv
0.9052
0.7252
0.8126
0.9066
0,9162
Products
0.9400
0.7518
0.7657
0.9456
0.9571
Average
0.9276
0.7572
0.8725
0.9348
0.9493
the degree matrix. Therefore, cut value is defined as follows: CV
5.3 ABLATION STUDY
Influence of the Codebook Size We analyze the influence of the codebook size of our graph to- kenizer. From Figure 5()), we observe that changing codebook size can significantly influence the performance for our distilled MIP model. Too small value not have enough expressiveness
8

---

Published as a conference paper at ICLR 2024
for preserving graph structures while too large value will lead to code redundancy impairing the performance. Morever, VQGRAPH has different optimal codebook sizes for various datasets as in Figure 5(i)). Another interesting observation is the graph with more nodes edges tends to quire a larger codebook size to achieve optimal distillation results. In our VQGrapph, the size of the codebook is mainly determined by the complexity of the graph data, considering both nodes and edges which produce different local substructures. Thus, our codebook size is still small compared the exponential graph topological complexity, demonstrating the expressiveness of our codebook. Taking Cora as an example, our codebook size is 2048, but it contains 2485 nodes with an average degree about which can theoretically result in ((2885*) possible 1-hop substructure patterns.
90
Citeseer
Amazon Computer
(nodes, Arxiv (169343, 1166243)
Pubmed
Anxiv
CoCoputer (nedes, edges)
Accuracy
245776)
Pubmed (nodes, edges) 4442)
Cieese (nodes, edges) (2110, 3668)
2
2
2-2
2
2
2
2
2
2
2
2
2
Codebook Size
Optimal Codebook Size
(i) Accuracy Codebook Size.
(ii) Optimal codebook size for various datasets.
Figure 5: Influence of the codebook size.
Contribution Analysis of VQGRAPH We design experiments to make statistical analysis on the contributions our VQGRAPH. The results are presented in Tab. 4, and we observe that compared to traditional class-based distillation, both Only-VQ and VQGRAPH promote the average accuracy, suggesting that both graph tokenizer and soft code assignments have vital impacts on final perfor- mance. Moreover, comparing AE+class-bseed class-based, slightly improves GNN-to-MLP distillation. Our designed graph VQ-VAE efficiently improves the GNN-to-MLP distillation results more significantly than classic graph (Variational)AE, because directly learn numerous structure-awan codes to enrich the expressiveness of node representations. Our V@+code-baat distillation (denoted as VQGRAPH) substantially improves node classification performance over Only-Vo VQ across all datasets, demonstrating the superiority of our new structure- aware distillation targets over soft labels. Please refer to Appendix C for more ablation studies.
find adding structure awareness
Table 4: Class-based denotes only using soft labels for distillation (e.g., GLNN), denotes adding classic graph Auto-Encoder (Kipf & Welling. 2016) for structure awareness. Only- VQ denotes using VQ for training teacher but using soft labels for distillation. Aoaly vQ. AVQGRAPH represents the differences between Only-VQ. VQGRAPH and Class-based.
Datavets
GNN
Class bused
AE+Class based
Only-VQ (ours)
VQGRAPH (ours)
Aueyso
A Avgesara
Clteseer
70.49 1,53
71.22 1.54
71.65
74.96
5.25%
6.822%
Pubmed
75.56 :: 2,06
75.59
:: 2.46
76.56 +: 1,23
77.86
2.46
78.40
1.71
3.00%
13.71%
Cora
1.57
80.26
1.66
81.11 +
82.48
0.46
2.77%
44577%
computer
8282 = 1.37
82.71 = 1.18
83.01 - 1.18
84,06
1.18
85.17
1.29
1.63%
2.97%
A-photo
90.85 1: 0.87
91.95 4 1.04
92.06 1:
93.86
1.04
94.21
0.45
2200%
2.45%
Arxiv
70.73 0.35
63.75
0.48
70.10 = 1.02
70.75 = 0.48
72.43
0.20
10.98%
13.61%
Products
77.17 2 0.32
67.71
1 0.31
77.65 1: 0.98
78.71
0.31
0.21 79.17
16.25%
16.93%
CONCLUSION
In this paper, we improve the expressiveness of existing graph representation space by directly la- beling nodes' diverse local structures with codebook, and utilizing the codebook for facilitating structure-aware GNN-to-MLP distillation. Extensive experiments on seven datasets demonstrate that our VQGRAPH can significantly improve GNNs by 3.90%. MLPs by 28.05 %, and the of the-art GNN-to-MIL distillation method by 1.39% average accuracy, while maintaining a fast inference speed of 828x compared to GNNs. Furthermore, we present additional visualization and statistical analyses well ablation studies demonstrate the superiority of the proposed model.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by the National Natural Science Foundation of China (No. U22B20323 and U23B2048).
REFERENCES
Sami Abu-EI-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learn- ing. pp. 21-29. PMLR, 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: wrxiv::30..34333 34,2, 2013.
Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Mincut pooling in graph neural networks. 2019.
Aleksandar Bojchevski, Johannes Gasteiger, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate pagerank. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1464-2473 2020.
H Bunke and G Allermann. Inexact graph matching for structural pattern recognition. Pat- tern Recognition Letters, 1(4):245 1983. ISSN 0167-8655. doi: http:://doi.or//0..uk 101// 0167-86555(3390033--8
Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction. In International Conference on Machine Learning, pp. 942-950. PMLR, 2018a.
Jie Chen, Tengfei Ma, and Cao Xiao, Fastgen: Fast learning with graph convolutional networks via importance sampling, In International Conference on Learning Representations, 2018b.
Jie Chen, Tengfei Ma, and Cao Xiao, FastGCN: Fast learning with graph convolutional networks via importance sampling. In International Conference on Learning Representations, 2018c
Ming Chen, Zhewei Wei, Bolin Ding. Yaliang Li, Ye Yuan, Xiaoyong Du, and Ji-Rong Wen. Scal- able graph neural networks via bidirectional propagation. Advances in neural information pro- cessing systems, 33:14556-14555 14566, 2020a.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML, 2020b.
Ming Chen, Zhewei Wei, Zengfeng Huang. Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks, 2020c.
Yu-Hsin Chen, Joel Emer, and Vivienne Sze. Eyeriss: A spatial architecture for energy- dataflow for convolutional neural networks. 2016 ACM/IEEE 43rd Annual International Sym- posium Computer Architecture (ISCA), pp. 367-379, 2016. doi: 10.1 109/ISCA.2016.40
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp- 257-266, 2019.
Eli Chien, Wei- Wei-Cheng Chang, Cho-Jui Hsich, Hsiang- Fu Yu, Jiong Zhang, Olgica Milenkovic. and Inderjit S Dhillon. Node feature extraction by self-supervised multi-scale neighborhood predic- tion. In International Conference on Learning Representations, 2022.
Xiang Deng and Zhongfei Zhang. Graph-free knowledge distillation for graph neural networks, 2021.
10

---

Published as a conference paper at ICLR 2024
Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S Yu. Enhancing graph neural network- based fraud detectors against camouflaged fraudsters. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 315-324, 2020.
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference, pp. 417-426, 2019.
Matthias Fey, Jan E Lenssen, Frank Weichert, and Jure Leskovec. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings. In International Conference on Machine Learning, pp. 3294-3304. PMLR, 2021.
Xinbo Gao, Bing Xiao, Dacheng Tao, and Xuelong Li. A survey of graph edit distance. Pattern Anal. Appl., 13:113-129, 02 2010. doi: 10. 10.1007/11004440000011111
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 2021.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1737-1746, Lille, France, 07-09 Jul 2015. PMLR.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. NeurIPS, 2017.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS 15, pp. 1135-1143, Cambridge, MA, USA, 2015. MIT Press.
Xiangnan Hc, Kuan Deng, Xiang Wang. Yan Li, Yongdong Zhang, and Meng Wang. Lightgnn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 639-648, 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint 1rXiv:1503.02531, 2015.
Weihua Hu, Matthis Fey, Marinka Zimik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:221 11-22118-21333 2020a.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. CoRR, abs/2005.0068 000807 2020b.
Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang, Erjin Zhou, and Yue Gao. Graph-mlp: node classification without message passing in graph. arXiv preprint arXiv:2106 0.055. 2021.
Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin Benson. Combining label prop- agation and simple models put-performs graph neural networks. In International Conference on Learning Representations, 2021.
Wenbing Huang, Tong Zhang. Yu Rong, and Junzhou Huang. Adaptive sampling towards fast graph representation learning. Advances in neural information processing systems, 31, 2018.
Zhihao Jia, Sina Lin, Rex Ying, Jiaxuan You, Jure Leskovec, and Alex Aiken. Redundancy- computation for graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 997-1005, 2020.
Patrick Judd, Jorge Albericio, Tayler Hetherington, Tor M. Aamodt, Natalie Enright Jerger, and Andreas Moshovos. Proteus: Exploiting numerical precision variability in deep neural networks. In Proceedings of the 2016 International Conference on Supercomputing, pp. 23, 2016.

---

Published as a conference paper at ICLR 2024
Thomas N Kipf and Max Welling. arXiv:1611.07388, 2016.
Variational graph auto-encoders.
arXiv preprint
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In ICLR, 2017.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate:
Graph neural networks meet personalized pagerank. In ICLR, 2019.
Yonghyeon Lee, Hyeokjun Kwon, and Frank Park. Neighborhood reconstructing autoencoders.
Advances in Neural Information Processing Systems, 14:536-546, 2021.
Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. In NeurlPS, 2020a.
Xiangsheng Li, Maarten de Rijke, Yiqun Liu, Jiaxin Mao, Weizhi Ma, Min Zhang, and Shaoping Ma. Learning better representations for neural information retrieval with graph information. In Proceedings of the 29th ACM International Conference on Information & Knowledge Manage- 2020b.
ment, pp.
Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng, Hao Yang. and Qing He. Pick and choose: a gnn-based imbalanced learning approach for fraud detection. In Proceedings of the Web Conference 2021, pp. 3168-3177, 2021.
Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Revisiting heterophily for graph neural networks, 2022.
Kelong Mao, Xi Xiao, Jieming Zhu, Biao Lu, Ruiming Tang, and Xiuqiang He. Item tagging for information retrieval: tripartite graph neural network based approach. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp- 2327-2336, 2020.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In AAAI, 2019.
Galileo Namata, Ben London, Lise Getoor, Bert Huang, and U Edu. Query-driven active surveying for collective classification. In 10th international workshop mining and learning with graphs, volume 8, pp. 1, 2012.
Joonhyung Park, Jaeyun Song, and Eunho Yang. Graphens: Neighbor- ego network synthesis for class-imbllanced node classification. In International Conference on Learning Representa- fions, 2021.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. CoRR, abs/2002.05287, 2020a.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In International Conference on Learning Representations, 2020b.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In KDD, 2014.
Alberto Sanfeliu and King-Sun Fu. A distance measure between attributed relational graphs for pattern recognition. IEEE Transactions on Systems, Man, and Cybernetics, SMC-133))353-366 1983. doi: 10. 19991SMM..1883.3111
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. Al magazine, 29(3):93-95 2008.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, 2016.
12

---

Published as a conference paper at ICLR 2024
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.0586 2018.
Zhihao Shi, Xize Liang, and Jie Wang. Lmc: Fast training of gnns via subgraph sampling with provable convergence. In The Eleventh International Conference Learning Representations, 2023.
Mingyue Tang, Pan Li, and Carl Yang. Graph auto-encoder via neighborhood wasserstein recon- struction. In International Conference on Learning Representations, 2022.
Yijun Tian, Shichao Pei, Xiangliang Zhang, Chuxu Zhang, and Nitesh V Chawla. Knowledge distillation on graphs: A survey, arXiv preprint arXiv:2302.0022 2023a.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. Learning mips on graphs: A unified view of effectiveness, robustness, and efficiency. In The Fleventh International Conference on Learning Representations, 2023b.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9111), 2008.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: survey. ACM Computing Surveys, 55(5):1-37, 2022.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao. Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint 1eXivv6609.000 /444 2016.
144,
Shuo Xiao, Dongqing Zhu, Chaogang Tang, and Zhenzhen Huang. Combining graph contrastive embedding and multi-head cross-attention transfer for cross-domain recommendation. Data Sci- ence and Engineering, 8(3):477-62,, 2023.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.
Yishi Xu, Yingxue Zhang, Wei Guo, Huifeng Guo, Ruiming Tang, and Mark Coates. Graph- sail: Graph structure aware incremental learning for recommender systems. In Proceedings of the 29th ACM International Conference on Information; Knowledge Management, CIKM '20, pp. 2861-2868 New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781 9781450368599. doi: 10. 45533345511.34125555
Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Lou. Tinygnn: Learning efficient graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference Knowl- edge Discovery &#38; Data Mining (KDD), pp. 1848-1856, 2020.
Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and beyond it: An effective knowledge distillation framework. In www. 2021a.
Cheng Yang. Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework, 2021b.
Ling Yang and Shenda Hong. Omni-granular ego-semantic propagation for self-supervised graph representation learning. In International Conference on Machine Learning, pp. 25022-25037. PMLR, 2022.
Ling Yang, Liangliang Li, Zilun Zhang, Xinyu Zhou, Erjin Zhou, and Yu Liu. Dpgn: Distribution propagation graph network for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 13399 2020.
13

---

Published as a conference paper at ICLR 2024
Ling Yang, Jiayi Zheng, Heyuan Wang, Zhongyi Liu, Zhilin Huang, Shenda Hong, Wentao Zhang, and Bin Cui. Individual and structural graph information bottlenecks for out-of-distributinn gen- eralization. IEEE Transactions on Knowledge and Data Engineering. 2023.
Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks, 2021c.
Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 974 983, 2018.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graph- saint: Graph sampling based inductive learning method, 2020.
Chunhui Zhang, Chao Huang, Yijun Tian, Qianlong Wen, Zhongyu Ouyang, Youhuan Li, Yanfang Ye, and Chuxu Zhang. Diving into unified data-model sparsity for class-imbalanced graph repre- sentation learning. In GLFrontiers, 2022a.
Dalong Zhang, Xin Huang, Ziqi Liu, Zhiyang Hu, Xianzheng Song, Zhibang Ge. Zhiqiang Zhang, Lin Wang, Jun Zhou, Yang Shuang, et al. Agl: A scalable system for industrial-purpot graph machine learning. arXiv preprint arXiv:2003. 02454, 2020.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. In ICLR, 2022b.
Zhongping Zhang, Yin Jia, Yuehan Hou, and Xinlu Yu. Explicit behavior interaction with hetero- geneous graph for multi-behavior recommendation. Data Science and Engineering, pp. 1-19, 2024.
Lingxiao Zhao, Wei Jin, Leman Akoglu, and Neil Shah. From stars to subgraphs: Uplifting any gnn with local structure awareness. In International Conference on Learning Representations, 2022.
Yiren Zhao, Duo Wang, Daniel Bates, Robert Mullins, Mateja Jamnik, and Pietro Lio. Learned low precision graph neural networks, 2020.
Wenqing Zheng, Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Subbian. Cold brew: Distilling graph node representations with incomplete or missing neighbor- hoods. ICLR, 2022.
Helong Zhou and Liangchen Song. Rethinking soft labels for knowledge distillation: A bias- variance tradeoff perspective. In Proceedings of International Conference on Learning Repre- sentations (ICLR), 2021.
Hongkuan Zhou, Ajitesh Srivastava, Hanqing Zeng, Rajgopal Kannan, and Viktor K. Prasanna. Accelerating large scale real-time GNN inference using channel pruning. CoRR, abs/2105.04528. 2021.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs, 2020.
Qi Zhu, Carl Yang, Yidan Xu, Haonan Wang, Chao Zhang, and Jiawei Han. Transfer learning of graph neural networks with ego-graph information maximization. Advances in Neural Informa- tion Processing Systems, 34:1766-1779, 2021.
Xiaojin Zhu and Zoubin Ghahramanih. Learning from labeled and unlabeled data with label propa- gation. 2002.
Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. Layer-dependent importance sampling for training deep and large graph convolutional networks. arXiv preprint arXiv:1911.072 2019.
14

---

Published as a conference paper at ICLR 2024
A MORE ANALYSIS AND RESULTS
A.1 CONNECTION BETWEEN LOCAL GRAPH STRUCTURE AND LEARNED CODEBOOK
To better illustrate the connection between local graph structure and the codebook learned by our graph VQ-VAE, conduct node-centered subgraph retrieval in the learned MLP representation spaces of NOSMOG and our VQGRAPH. Specifically, we extract the representation with distilled MLP model for query node in Citeseer. We demonstrate 4 subgraphs centered at the nodes that are most similar to the query node representation with the cosine similarities in the whole graph in Figure 6. As be observed, the identical token IDs yield similar substructures, indicating the rep- resentation similarities V (QGRAPH are approximately aligned with the local subgraph similarities, which is denoted graph edit distance (GED) Sanfeliu & Fu (1983); Bunke & Allermann (1983); Gao et al. (2010) and is one of the most popular graph matching methods. In contrast, the repre- sentations extracted from NOSMOG fail sufficiently model and reflect the structural similarities between subgraphs. Hence, VQGRAPH is necessary and effective for addressing graph tasks. Moreover, we observe that VQGRAPII can be aware of the minor neighborhood struc- tural difference despite the nodes with the same class, demonstrating the more fine-grained expres- siveness of our token-based distillation, which further helps with more accurate classification.
pructure-comple
Class id:
Class id:
Class id:
Class id: 4
NOSMOG
GED:
GED:
GED:
GED: 3
Class id: 2
Class id:
Class id:
Class id: 2
Token 3958
id: 3958
Token id: 3797
Token
Class id: 2
Token id: 3958
VQGraph
GED:
GED:
GED:
GRD: 2
Figure 6: The query node and closest nodes distilled MI.P representation space with corre- sponding subgraphs.
A.2 VQGRAPH FOR HETEROPHILIC GRAPHS
In addition homophilic graphs, we extend our proposed method heterophilic graphs for further evaluation. We consider 2 heterophilic datasets, Texas and Corne 11, realsed by Pei et al. (2020a). Texas and Cornell 1 are web page datasets collected from computer science departments of var- ious universities. In these datasets, nodes are web pages and edges represent hyperlinks between them. ofg-of-words representations are taken as nodes' feature vectors. The task is to classify the web pages into five categories including student, project, course, staff and faculty. Basic statistics of the datasets are shown in Tab. The Edge Hom. (Zhu et al. (2020)) is defined as the fraction of edges that connect nodes with the same label.
Table 5: Heterophilic Dataset Statistics.
Dataset
# Nodes
Edges
# Features
# Classes
# Edge Hom.
Texas
183
295
1.703
5
0.11 0.30
Cornell
183
280
1,703
5
We incorporate our structure-awart code embeddings with two state-of-ther of-the-art GNN models known for their efficacy heterophilic graphs, namely ACMGCN (Luan al. (2022)) and GCNII (Chen
15

---

Published as a conference paper at ICLR 2024
al. (2020c)), along with MLP. We compare with these baselines in Tab. 6. The results of these periments demonstrate that our graph VQ- VAE can generalize to different heterophilic GNN/MLP architectures and consistently improve their performances.
Table 6: Results on Heterophilic Graphs.
Texas
Cornell
ACMGCN
86.49
84.05
ACMGCN with our Graph VQ-VAE
87.03
84.59
MLP
75.68
76.38
MLP with our Graph VQ-VAR
77.93
78.29
GCNII
76.73
76.49
GCNII with our Graph VQ-VAE
79.04
78.29
A.3 ANALYSIS ON CODEBOOK ENTRIES
In the training stage, we tokenize each node with different neighborhood structures as discrete codes using Graph VQ- VAE. Now we take a closer look at codebook entries for different classes on Pubmed for in-depth analysis (Tabs. and 8). There is little difference in the number of codes with each class and there is a uniform distribution of the nodes for each class, demonstrating learning capacity of our model. Meanwhile, the results demonstrate that there is small code over- lap among different classes, which indicates the code distributions learned for different classes are mutually independent. This finding validates that our graph VQ-VAE enables distinct separation of the nodes with different class labels in the representation space, facilitating a better knowledge transfer in GNN-to-MLP distillation.
Table 7: Codebook Entries for Each Class in Pubmed.
Class I
Class 2
Class 3
Code Entries
2,560
2,623
2,591
Table 8: The Overlapping Codes (%) for Each Paired Classes of Pubmed.
Class
Class
Class 3
Class 1
0.9%
0.8%
Class 2
0.6%
Class 3
B EXPERIMENT DETAILS
B. DATASETS
The statistics of the seven public benchmark datasets used in our experiments are shown Tab. 9. For all datasets, follow the setting in the original paper split the data. Specifically, for the five small datasets (i.e., Cora, Citeseer, Pubmed, A-computer, and A-photo), we use the splitting strategy in the CPF paper (Yang al., 2021b), where each random seed corresponding For the two OGB large datasets (i.e., Arxiv and Products), we follow the official splits in Hu et al. (2020a) based on time and popularity, respectively. We introduce each dataset as follows:
different split.
Citeseer (Sen et al., 2008) a benchmark citation dataset consisting of scientific pub- lications, with the configuration similar to the Cora dataset. The dictionary contains 3,703 unique words. Citeseer dataset has the largest number of features among all datasets used in this paper.
16

---

Published as a conference paper at ICLR 2024
Pubmed (Namata al., 2012) is citation benchmark consisting of diabetes-related arti- cles from the PubMed database. The node features are Weighe-eiigtedd word frequency, and the label indicates the type of diabetes covered in this article.
Cora (Sen et al., 2008) is benchmark citation dataset composed of scientific publications. Each node in the graph represents publication whose feature vector is a sparse bag-of- words with 0/1-values indicating the absence/presence of the corresponding word from the word dictionary. Edges represent citations between papers, and labels indicate the research field of each paper.
A A-computer and A-photo (Shchur al., 2018) are two benchmark datasets that tracted from Amazon co-purchase graph. Nodes represent products, edges indicate whether two products are frequently purchased together, features represent product reviews encoded by bag-of-words, and labels are predefined product categories.
Arxiv (Hu et al., 2020a) is a benchmark citation dataset composed Computer Science arXiv papers. Each node is an arXiv paper and each edge indicates that one paper cites another one. The node features are average word embeddings of the title and abstract.
Products (Hu et al., 2020a) is a benchmark Amazon product co-purchasing network dataset. Nodes represent products sold Amazon, and edges between two products indi- cate that the products are purchased together. The node features are bag-of-worde features from the product descriptions.
Table 9; Dataset Statistics.
Dataset
# Nodes
# Edges
Features
Classes
Citeseer
2.110
3,668
3,703
Pubmed
19,717
44,324
500
Cora
2,485
5,069
1,433
7
A-computer
13,381
245,778
767
A A-photo
7,487
119,043
745
8
Arxiv 169,343
1,166,243
128
40
Products
2,449.029
61,859,140
100
47
B.2 MODEL. HYPERPARAMETERS AND IMPLEMENTATION DETAILS
The code and trained models provided in the supplementary material.
Production Scenario Regarding to the production scenario, we recognize that real-world deploy- ment of models necessitates the ability to generate predictions for new data points as well reliably maintain performance on existing ones. Therefore, we apply our method in realistic production setting to better understand the effectiveness of GNN-1o-MLP models, encompassing both trans- ductive and inductive predictions. In real-life production environment, it is common for model to retrained periodically. The holdout nodes the inductive set represent new nodes that have entered the graph between two training periods. To mitigate potential randomness and to evaluate generalizability more effectively, we employ a test dataset V which contains 20% of the test data, and another dataset VU containing the remaining 80% of the test data. This setup allows us to evaluate the model's 's performance on both observed unlabeled nodes (transductive prediction) and newly introduced nodes (inductive prediction). reflecting real- sal-world inference scenarios. We further provide three sets of results: "tran" refers to results on VU ohs- "ind" refers V and "prod" refers to an interpolated value between tran and ind, according to the split rate, indicating the performance in real production settings.
Graph Tokenizer Pre-Trai Pre-Training We first train our graph tokenizer which contains teacher GNN model and a learnable codebook, then use both the GNN and the codebook to conduct GNN-to-MLP distillation. The hyperparameters of GNN models each dataset rameters provided by the CPF paper (Yang et al., 2021b) (Tab. 10) and the OGB official examples (Hu al., 2020a) (Tab. 11). We provide the hyperparameters of our codebooks for different datasets
taken from the best hyperpa-
17

---

Published as a conference paper at ICLR 2024
in Tab. 12. Kindly note that the VQ procedure is inserted between the encoder and the classifier of the GNN model, and the dimensions of code embedding and GNN feature are same for convenient assignment. And the code embeddings ing of our graph tokenizer, VQGRAPH additionally includes two separate linear decoder layers to decode node attributes and graph topology, respectively, hased assigned code embeddings.
initialized with uniform distribution. During the train-
Code-Based GNN-to-MLP Distillation We freeze the parameters of both the pre-trained teacher GNN model and the learned codebook embeddings for our code-based GNN-to-MLLP distillation. For the student MIPP in VQGRAPH, unless otherwise specified with -wi or or-Li, we set the number of layers and the hidden dimension of each layer to be the same as the teacher GNN, so their total num- ber of parameters stays the same as the teacher GNN (in Tab. 12). In distillation, the teacher GNN not only delivers the soft labels with respect to node classification to the MLP, but also produces its corresponding soft code assignment for the MLP. Kindly recall that compute code assignments for the MLP only in training process, and we remove this assigning pro- cedure for deploying the distilled MLP in testing process, which does increase inference time. In practice, categorial and structural information might be of different importance to the distillation in various graph datasets. We correspondingly modify their loss weights for better performance.
codebook embeddings to
Table 10: Hyperparameters for GNNs on five datasets from the CPF paper.
SAGE
GCN
GAT
APPNP
layers
2
2
2
2
hidden dim
128
64
64
64
learning rate
0.01
0.01
0.01
0.01
weight decay
0.0005
0.001
0.01
0.01
dropout
0
0.8
0.6
0.5
fan
5,5
attention heads
8
power iterations
10
Table 11: Hyperparameters for GraphSAGE on OGB datasets.
Dataset
Arxiv
Products
layers
hidden dim
256
256
learning rate
0.01
0.003
weight decay
O
dropout
0.2
0.5
normalization
batch
batch
fan out
[5, 10, 15]
[5, 10, 15]
Table 12: Hyperparameters of VQGRAPH.
Citeseer Pubmed Cora A-computer A-photo Arxiv Products
tokenizer attribute decoder layers
1
1
1
tokenizer topology decoder layers
1
1
1
codebook
4096
8192
2048
16384
8192
32768
32,768
T in Londiadiststt
4
4
4
4
MLP layers hidden dim
2
2
2
2
3
3
128
128
128
128
128
256
256
learning rate
0.01
0.01
0.005
0.003
0.001
0.01
0.003
weight decay
0.005
0.001
0.001
0.005
0.001
0
0
dropout
0.6
0.1
0.4
0.1
0.1
0.2
0.5
for Listst
1
1
1
1
3 for Lcode distitl
1e-8
le-8
le-8
le-8
1c-8
1e-8
le-8
18

---

Published as a conference paper at ICLR 2024
C MORE ABLATION STUDIES
C.I VQGRAPH WITH DIFFERENT RELATION MODULES IN EQUATION (4)
In Sec. 4.2, we compute 1-vs-M token-wish relations in Equation (4) using L2 distance as the rela- tion module COMP(: Other relation modules can also be applied, and we thus conduct experi- ments based on cosine similarity with the results depicted Tab. 13. Experiments across different datasets illustrate that VQGRAPH with different relation modules consistently improve the code distillation performance of VQGRAPH. The influence relation module on VQGRAPH's overall performance very minimal, with L2 distance exhibiting slight advantage of approximately 0.2% on average accuracy.
Table 13: VQGRAPH with diffrent relation modules.
Datasets
GNN
w/ Cosine Similarity
w/ La distance
Citeseer
70.49 1.53
75.80 0.66
76.08 0.55
t 0.12%
Pubmed
75.56 1 2.06
78.12 1.23
78.40 1 1.71
f 0.23%
Cora
80.64 1 1.57
83.61 1 0.70
83.93 1 0.87
1 0020%
A-computer
82.82 + 1.37
84.96 + 1.06
85.17 + 1.29
0.20%
A-photo
90.85 0.87
94.05 0.29
94.21 + 0.45
1 7..88
Arxiv
70.73 = 0.35
71.96 + 0.18
72.43 + 0.20
1 0.166%
Products
77.17 + 0.32
79.01 + 0.34
79.17 +: 0.21
0.18%
C.2 VQGRAPH WITH DIFFERENT TEACHER GNNS
In VQGRAPH, we utilize GraphSAGE with GCN aggregation to represent our teacher GNN. How- ever, given that different GNN architectures may exhibit varying performance across datasets, seek to investigate whether VQGRAPII can perform well when trained with other GNN architec- tures. As displayed in Figure we present the average performance of VQGRAPII when distilled from different teacher GNNs, including GCN, GAT, and APPNP, across five benchmark datasets. Our results demonstrate that all four teacher models achieved comparable performance, but VQ- GRAPH consistently outperforms teacher GNNs and other GNN-to-MLA distillation methods, indi- cating the superior effectiveness and generalization ability our VQGRAPH.
90
MLP
VOGraph
85
Accurac)
80
75
70
65
MLP
SAGE
GCN Architectures
GAT
APPNP
Model
Figure 7: Accuracy vs. Teacher GNN Model Architectures.
C.3 ROBUSTNESS EVALUATION WITH NOISY FEATURES
We evaluate the robustness of VQGRAPH with regards to different noise levels across different datasets and compute average accuracy. Specifically, we follow Tian et al. (2023b) to initialize the node features and introduce different levels of Gaussian noises the node features by modifying X with X (1-a) x * where denotes an independent Gaussian noise and a E [0, 1] controls the noise level. Our results, as illustrated in Figure 8, reveal that VQGRAPH achieves comparable improved performance compared to GNNs and previous SOTA NOSMOG across different levels
19

---

Published as a conference paper at ICLR 2024
of noise, demonstrating its superior noise-robustness and efficacy, especially when GNNs leverage local structure information of subgraphs to mitigate noise impact. Conversely, GLNN and MLP exhibit rapid performance deterioration
increases. In the extreme case where equals I, and the input features entirely perturbed, resulting in X and X being independent, VQGRAPH still maintains similar performance to GNNs, while GLNN and MLP perform poorly. Furthermore, even when compared with NOSMOG, which leverages adversarial feature augmentation combat noise and improve robustness, VQGRAPII still outperforms NOSMOG in terms of noise robustness and overall performance across various a settings, due its superior ability to sufficiently preserve and leverage graph structural information.
GNN
GLWh NOSMOG
VQGraph
80
MUP
70
Accuracy
60
50
40
30
20
0.0
0.2
0.4
0.6
0.8
1.0
Noise Ratio
Figure 8: Accuracy Feature Noises.
20


Reviews:
[{"review_id": "0E81n6EUG4", "reviewer": "Reviewer_12WL", "summary": "This paper proposes a GNN-to-MLP distillation approach. During the teacher GNN training, a series of code embeddings are jointly optimized using a reconstruction loss, to make them encode local graph structures. The divergence between the soft code assignment of the teacher GNN and that of the student MLP is used as additional supervision for distillation. The proposed approach outperforms the teacher GNN and other existing GNN-to-MLP distillation methods in both transductive and inductive settings.", "questions": "Please see Weaknesses", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed approach is novel compared with existing GNN-to-MLP distillation methods.\n- Experimental results are good, consistently better than NOSMOG\n- Additional ablation studies", "weaknesses": "- Some details of the experimental setup are not clear enough to me. As far as I understand from the paper, in the inductive setting,  $G^L \\cup G^U_{obs}$ is used for training the teacher GNN and for distillation. True labels $Y^L$ are used to train the teacher but are not used in distillation. Is it right? The paper mentions that \"the edges between $G^L \\cup G^U_{obs}$ and $G^U_{ind}$ are removed in training, while they are leveraged during inference to transfer positional features via average operator\". What does this mean? Do you augment the node feature with the features of its neighbors for inference?\n- What the student MLP can learn from distillation is ambiguous. Suppose two nodes have the same private feature but different neighborhoods, the teacher GNN would generate different representations and different soft code assignments for these two nodes. However, in distillation, the student MLP would always generate the same representation and soft code assignment for these two nodes (since they have the same private feature). As a result, the student MLP cannot learn their structural difference from the teacher", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gLV28TQrzQ", "reviewer": "Reviewer_Cnqf", "summary": "This paper presents a novel KD method from GNNs to MLPs to achieve the improvement of inference efficiency, where a special codebook is designed to store the graph information. Experiments on a wide range of benchmarks show its superior performance in terms of both accuracy and time cost.", "questions": "- Major\n    - In the abstract, the authors mention that 'the class space may not be expressive enough to capture numerous different local graph structures'. I am somehow confused, since it is about GNNs-to-MLPs KD, better capturing the local structure information is an issue for GNNs part, instead of KD?\n    - It is better to explain more about why it is appropriate to use VQ-VAE to encode graph structure, since it was originally designed for continuous data. For graph-structured data, it might be easier to convert to frequency information, i.e., each position in the \"codebook\" stores the volume of a specific frequency. I am just curious why VQ-VAE is suitable for encoding discrete and non-Euclidean data.\n    - According to the inner product in the reconstruction loss of equation 3 and the benchmarks used, it is quite important to emphasize that the effective domain of the proposal is probably homophilic graphs, otherwise further discussion or experiments on heterophilic graphs are necessary.\n\n- Minor\n    - What is the data set used for Figure 1? Cora? All other figures/tables need further clarification on which dataset is used.\n    - In equation 3, the node embeddings are still included, so the claimed local structural embedding is not that promising, can you compare or explain the difference of the two parts of the reconstruction function? Also, what's the nonlinear function here?\n    - Figure 2 needs improvement- In equation 7, the loss of classification and class distillation should also be clarified.\n    - The codebook does not seem to be so different from VGAE from the point of view of structure reconstruction, but the node feature reconstruction part makes it so different. Also, the graph structure part is not well designed, but an inner product.\n    - In the figures and descriptions throughout the paper, it is not fair to say that the codebook really preserves the structural information as a substructure, since it is just another integrated embedding where the identifiability of specific structure is missing, therefore it is misleading to give the local structure in Figure 2.\n    - An interesting point, which is worth to study further, is what the Figure 5(ii) implies: denser graph needs more space, which is parallel to the discovery from graph signal processing domain, that the structural information cannot be summarized only by some low-frequency filters, the high-frequency ones are also informative in such graphs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- How to make MLP more aware of the graph structure is a valuable topic for better inference performance. (while is also not that addressed in this paper, see the first weakness and question below)\n- The whole paper is well structured, the explanation for each part is overall complete and clear to understand.\n- The experimental results are convincing, indeed providing many interesting observations for discussion (see questions below).", "weaknesses": "- The connection between GNNs-to-MLPs distillation and the proposal of a local-structure-aware codebook is less developed, since they are actually separate goals; more explanation is needed to make the main claim of the paper clearer.\n- In practice, one of the main concerns is that the hyper-parameter for the code distillation in equation 7 is quite small compared to the class distillation one. It makes the whole proposal more intended to be a node embedding booster for GNNs, the graph tokenizer, while the codebook is not significantly effective in the distillation phase.\n- The volume of the codebook is very likely to be the same scale as the original node features, which makes the proposal still in the trade-off of time and space cost for sure; therefore, more information about the parameter volume should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vDM8VjnGHU", "reviewer": "Reviewer_F8Tt", "summary": "This work proposes a new method of knowledge distillation for node classification. Essentially, VQGraph incorporates a VQ-VAE to learn a codebook that represents informative local structures, and use these local structures as additional information, distilling it to student MLPs. Empirically, this approach outperforms state-of-the-art methods, in both accuracy and inference speed.", "questions": "1. It seems that for APPNP, all three KD approaches can't outperform the teacher model. Is there any discussion or explanation for this phenomenon?\n\n2. In Table 4, Only-VQ outperforms Class-based and AE+Class-based, but Only-VQ only adopts class soft labels, the VQ component helps to train the codebook, why this approach is better than the other two? Can you give some discussions on it?\n\n3. It would be better to provide an analysis what codebook entry is most informative to a class label, which can make this approach more convincing and intuitive.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Although VQ-VAE is not a new method, introducing a variant of VQ-VAE for KD for node classification sounds novel, and also make sense. \n\n2. The paper is well-written and easy to follow.\n\n3. This approach is effective under both inductive and transdutive setting, and also works well for large-scale datasets.\n\n4. The experiment is comprehensive and convincing.", "weaknesses": "1. The size of codebook is sensitive to different datasets, this may introduce some practical difficulty. \n\n2. It would be better to provide a theoretical understanding of this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Gq8E2KKXZN", "reviewer": "Reviewer_4K7i", "summary": "This paper studies the GNN-to-MLP distillation problem. The authors claim that the output representation of teacher GNN lack rich graph structural information. With this motivation, they propose to learn codebook embedding via VQ-VAE. After that, the prediction of teacher and student network is obtained by computing the distance between representation to each of the embedding in the learned codebook. Experimental results show that their methods possess faster inference speed and better performance against previous approaches.", "questions": "- The authors propose a generative approach learn codebook embedding. Is there any other way to learn the codebooks (such as self-supervised learning paradigm) or use other generative models (such as GAN) ?\n- Could the proposed method be applied to heterophilic graphs [1,2]? \n\n[1]  Lim et al., Large scale learning on non-homophilous graphs. NeurIPS 2021.\n\n[2] Platonov et al, A critical look at the evaluation of GNNs under heterophily: Are we really making progress? ICLR 2023.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using VQ-VAE to learn structure-aware embedding is reasonable, which could also serve as an effective approach to extract graph structural feature. Furthermore, the proposed distillation target essentially encourages the representation of each node given by teacher and student network close to the same embedding in the codebook. Therefore, the overall process is more transparent and interpretable.\n- This paper is well-organized and easy to follow. The experiments results are convincing and comprehensive, which are enough to support the effectiveness of the proposed method.", "weaknesses": "- Compared with previous method, extra memory space is required due to the introduced learnable codebook. As the authors show, the codebook size has influence on the performance of the model. How to effective tune this hyperparameter is worth exploring.\n- The adopted datasets in this paper are generally homophilic, i.e., the connected nodes belong to the same category. It is still unclear whether the proposed method is applicable to heterophilic graphs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g2iCqkQ8C6", "reviewer": "Reviewer_T74P", "summary": "The goal of this paper is to distill the knowledge of \"teacher\" graph neural networks to \"student\" MLPs for efficiency. Existing knowledge distillation methods for graph-structured data exploit labels of each node for the knowledge distillation. But, the number of labeled nodes is limited so that the class space may not be sufficient to cover the diverse graph structures. To address this issue, the paper proposes a variant of VQ-VAE to directly label the local graph structures. Specifically, it first learns a structure-aware tokenizer for dealing with graph structured datsets. The structure-aware tokenizer encodes the local substructure of each node as a discrete code. Then, the paper designs a new distillation target to directly transfer the structural knowledge of each node from GNN to MLP.", "questions": "Please refer to the above weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper deals with one of the important research topics on graph representation learning.\n- The proposed VQ-VAE is effective and efficient on various datasets and tasks from the author's experiments.\n- The paper is well written and easy to follow.", "weaknesses": "- It would be better if more details about the production scenario were included in the paper. If the authors are not experts in the graph knowledge distillation domain, they are unlikely to know the details of the experiments. So, more details about the tasks need to be described.\n- I wonder whether the proposed VQ-VAE pre-trains both graph tokenizer and graph neural networks simultaneously or pre-trains graph neural networks first and then the graph tokenizer.\n- The paper claimed that capturing graph structure is important for knowledge distillation. Then, I think that learning representations with self-supervised learning [1] for graphs can be other option to capture local graph structure. In my thoughts, labeling nodes' local structure with discrete codes seems not different from learning node representation with self-supervision loss [1] if its purpose is capturing local structure.\n\n[1]: Veličković, Petar, et al. \"Deep graph infomax.\" ICLR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)