Graph Knowledge Distillation to Mixture of Experts
Authors: Pavel Rumiantsev
Mark Coates
Venue: Accepted by TMLR ()

Abstract: In terms of accuracy, Graph Neural Networks (GNNs) are the best architectural choice for the node classification task.
Their drawback in real-world deployment is the latency that emerges from the neighbourhood processing operation.
One solution to the latency issue is to perform knowledge distillation from a trained GNN to a Multi-Layer Perceptron (MLP), where the MLP processes only the features of the node being classified (and possibly some pre-computed structural information). 
However, the performance of such MLPs in both transductive and inductive settings remains inconsistent for existing knowledge distillation techniques.
We propose to address the performance concerns by using a specially-designed student model instead of an MLP. 
Our model, named Routing-by-Memory (RbM), is a form of Mixture-of-Experts (MoE), with a design that enforces expert specialization.
By encouraging  each expert to specialize on a certain region on the hidden representation space, we demonstrate experimentally that it is possible to derive considerably more consistent performance across multiple datasets. Code available at https://github.com/Rufaim/routing-by-memory.

Published in Transactions on Machine Learning Research (10/2024)
Graph Knowledge Distillation to Mixture of Experts
Pavel Rumiantsov
The Department of Electricol and Computer Engineering
McGill University
Mark Coates
mark. www.t.cotts@mmgll.co
The Deportment of Electricol and Computer Engineering
McGill University
Reviewed on OpenReview: Atips: // openreview. net/ forum? id- w2Z3pb NRuh
Abstract
In terms of accuracy, Graph Neural Networks (GNNs) the best architectural choice for the node classification task. Their drawback in real-world deployment is the latency that emerges from the neighbourhood processing operation. One solution to the latency issue is to perform knowledge distillation from a trained GNN to a Multi-Layer Perceptron (MLP), where the MLP processes only the features of the node being classified (and possibly some pre-computed structural information). However, the performance of such MLPs in both transductive and inductive settings remains inconsistent for existing knowledge distillation techniques. We propose address the performance concerns by using specially-designed student model instead of an MLP. Our model, named Nouting-by-Memood form of Mixture-of of-experts (MoE), with design that enforces expert specialization. By encouraging each expert to specialize on certain region on the hidden representation space, we demonstrate experimentally that it is possible to derive considerably more consistent performance across multiple datasets. Code available hhtts:///iithul: ctt.uufa.oom/uaaaaaaaaaaa
(RbM), is
banting-bymmmory
1 Introduction
Graphs can be used to encode the dependencies between data samples. The impressive performance of Graph Neural Networks (GNNs) shows that taking into account the structural information increases the quality of prediction on tasks like product prediction on co-purchasing graphs or paper category prediction on citation graphs (Kipf & Welling 2016 Hamilton et al.] 2017). However, despite the potential accuracy improvements of GNNs, multi-layer perceptrons (MLPs) remain preferable to graph neural networks for many large-scale industrial applications. This is due to the fundamental inefficiency of GNNs, with scalability limitations
making deployment challenging (Zhang et al. 2020a Jia et al. 2020 Zheng et al. 2022). GNNs operate in layers, and each layer requires the processing of neighbourhood of the node in order to compute the prediction. For example, evaluating the prediction for a single node with an I-layer GNN requires processing least every node in the L-hop neighbourhood. For real-world graphs, involving millions of nodes, the L-hop neighbourhood can be very large, leading to resource intensive operations (Jin et al., 2021). Even if we only sample subset of the neighbours at each layer, the receptive field for the node grow very rapidly. This can lead to latency performance bottlencck in high-load systems. This is especially the case in situations where the graph is distributed over multiple servers. Fetching the entire L-hop neighbourhood can result in greatly increased latency (Zheng et al. 2022). By contrast, forming a prediction for a single node with an L-layer MLP requires processing only the features of that node. This naturally climinates the latency problems associated with additional node fetching. Therefore it can be scaled and deployed efficiently, and parallelization is straightforward.

---

Published in Transactions on Machine Learning Research (10/2024)
Combining GNN's outperformance with the reduced latency of MLP allows to enjoy the advantages of both Zhang et al. 2021b). Early works tried to simplify the aggregation step of the GNN by decreasing the number of operations performed at inference (Hu et al. 2021 Zheng et al.) 2021). However, such methods still depend on node fetching, which may scriously increase latency for large graphs {Jin et al. 2021 Zheng
et al. 2022p. Knowledge distillation is a more efficient way to address this problem. A student MLP is trained directly using soft labels generated by a teacher GNN, and can thus approximate the graph-context
information obtained by the aggregation step. This leads to reduced latency and can even result in higher inference quality in some cases (Zhang et al. 2021b Tian et al., 2022 Wu et al. 2023).
Inercasing the number of parameters of the student MLP can help to achieve better performance for some datasets (Zhang et al.) 2021b; Tian et al. 2022). However, this is not a consistent effect. In this work, we aim to improve the student performance for a fixed number of parameters by introducing a Routing- by-Memory (RbM) architecture as the student model. Our proposed model is Sparse Mixture of Ex- perts (MoE) (Shazeer et al., 2016 Chi et al., 2022) approach that tailored to the graph learning sctting, By avoiding the aggregation step and incorporating sparse model structure, we can achieve higher pa- rameter capacity, leading to better performance while keeping the inference cost low. The Routing by Memory (RbM) procedure encourages experts to specialize on a specific subset of the representations, mak- ing it more efficient than standard MoE routing. We conduct a series of experiments showing that our approach can be efficiently and effectively applied to datasets of various sizes. To evaluate our model, we explore both transductive and inductive settings for 9 publicly available datasets.
We make the following contributions:
Ours is the first work to propose the use of a student Mixturo-of-Expeert (MoE) model for the distillation of GNN.
We propose Routing-by-Menory (RbM) model that differs from, and outperforms, standard sparse MoE. We introduce important adaptations to routing system previously proposed by Zhang et al.) (2021a) for routing to a single expert in computer vision setting. We allow for routing to multiple experts and use a different distance.
During training of the proposed MoE, we introduce several loss terms to encourage better clustering of representations and improved expert specialization.
We conduct multiple novel experiments, demonstrating that the proposed approach consistently outper- forms both (i) enlarged MLP students; and (ii) ensembles sparse MoEs.
2
Related work
2.1 GNN-to-MLP Knowledge Distillation
Knowledge distillation from a Graph Neural Network (GNN) into a Multi-Layer Perceptron (MLP) promotes inference efficiency by avoiding the aggregation over neighbourhood nodes. YYagg et al. (2021) presented one of the first distillation attempts, employing student model that combines label propagation with an MLP acting on the features. Although label propagation is a lightweight. form of aggregation, it is still reliant on the graph, so the overall speed-up in inference time is not dramatic.
The GLNN by Zhang et al. 2022bb introduces knowledge distillation to an MLP without any aggregation over the graph nodes. The student trained with node content features as input and soft labels, generated by a pretrained GNN, which acts as the teacher. Tian al. (2022) show that soft labels alone are not enough to achieve consistent performance due to noise injected by the teacher GNN. The presented NOSMOG approach incorporates set of techniques to be used on top of knowledge distillation to help the student MLP to better approximate the graph-based information. It includes explicitly encoded position features generated using Deep Walk (Perozzi et al., 2014). NOSMOG also introduces a representational similarity distillation, which strives to encourage the student MLP to preserve the similarities between node representations that are observed in the tcacher GNN. Adversarial attack perturbations are also applied in order to ensure that the student model is more stable. While all the introduced techniques provide some improvement, the positional encoding is responsible for the vast majority of the performance gain.
2

---

Published in Transactions on Machine Learning Research (10/2024)
Zhang et al. (0200b) present an approach that aims to estimate the quality, or reliability, of the teacher model soft labels by evaluating the entropy. (Tan et al. (2022) also aim to evaluate the reliability of the soft labels, but employ a reinforcement learning approach. The concept of reliability is also at the heart of the work by Wu et al. (2023). : who propose KRD, a method that regularizes the student MLP by making it predict the soft labels a subset of the neighbouring nodes as well as that of the local target node. The subset of neighbours selected according to how reliable their associated soft labels are estimated to be. This regularization strategy is very effective, especially when combined with explicit positional encodings.
2.2 Mixture-of-Expert
A Sparse Mixture-oo-Fxxperss (MoE) model a weighted combination of similarly structured models with
dynamically computed weights (Shazeer et al. 2016 Gross et al. 2017 Zhang et al. 2021a; Li et al. 2022;
Dryden & Hoeller 2022 Chi al. 2022 Komatsuzaki al.
2022; Pavlitska et al. 2023). For any sample,
only a small portion of the experts have non-zero weights. This allows us to increase the model learning capacity without significantly inflating the processing costs (Fodus et al. 2022). An MoE can also be used as a layer inside a larger model (Qu et al., 2022 Yan & Li, 2023). In most implementation the expert weights, referred to as routing, is perforrmed by a separate neural network (a policy network) of a size comparable to an expert. Based on the MoE input, produces weights for the experts. Because both the policy network and expert networks are trained simultancously, there is a danger of under-utilization, and consequently under-training, of some experts (Krishnamurthy al. 2023). This is commonly miligated by injecting noise and designing loss functions that even out the utilization.
the computation of
An alternative routing scheme, designed to prevent routing inconsistencies, involves pairing each expert network with an embedding vector. Instead of predicting the expert weights directly, the policy network then aims to project the input sample into the embedding space. The weights are derived from the distances
to the expert embeddings (Gross et al., 2017 Zhang et al. 2021a Chi et al.. 2022 et al. 2022 Yan & Li, 2023 Qu et al.) (022). Initially, the Euclidean distance was used to route to the single closest expert (Gross et al. 2017 Zhang et al. 2021a). However, this causes computational stability issues when multiple experts
are used QQu et al. 2022). Using dot-product similarity (Lample et al.] 2019 Fedus et al., 2022 instead
of Euclidean distance typically leads to representation collapse. Chi et al. (2022) used cosine similarity to avoid the collapse, and this is currently used in many implementations (Li et al. 2022 Yan & Li 2023). Zhang ct al. (2021a) propose moving the expert embeddings into the layer's input space in order to encourage expert lination... In their scheme, the input vector is always routed to a single expert, the one with the closest embedding vector. Each expert embedding updated by calculating a moving average over the input embedding vectors that are routed to that expert. This leads to each expert specialising on the area of the input space around its embedding.
Knowledge distillation into a Mixture-of- has not been intensively studied. In tangentially related work, Zuo et al. (2022) study the distillation of language models and incorporate the MoE structure into pre-trained models for fine-tuning. Komatsuzaki et al. (2022) explore the task of upgrading a pre-trained dense model into a larger, sparsely-activated MoE.
3 Background
We denote graph by G = (V,E,X), where V is a set of N nodes, E is a set of edges between nodes, and X E Nxd represents a matrix with each row being a vector of d node features associated with the corresponding node. For the node classification task, with C classes, we use a label matrix, Y e [0,1) with each row containing a onc-hot encoded class label. The superscript L denotes the labelled nodes of the graph and the superscript U denotes unlabelled nodes, i..., V X!. Y are, respectively, the labeled nodes, their node features, and the one-hot class labels.
3

---

Published in Transactions on Machine Learning Research (10/2024)
Commitment loss
Teacher GNN
KRD Targets
Soft Targets
Seif-iimilarity loss
KRD loss
KD
loss
Load balance loss
Node features
Student MoE
Position features
(b)
Figure 1: (a) An overview of the overall training framework. teacher GNN is trained on the graph and provides targets for Knowledge Distillation (KD) and Knowledge-Awai Reliable Distillation (KRD) 12 losses. A Mixture-of-Expertss student is trained on the node features and positional encoding (see Section 4.2) (b) We three additional losses to adjust the internal representations of the model, as the embeddings use for routing (see Section 4.1p. We provide schematic representations of these losses to aid intuitive understanding. Commitment loss (6) pulls representations closer to embeddings (highlighted in blue). Self- similarity loss 7 prevents collapse of reprosentations. Load balance loss (8) helps to move borderline representations towards embedding of the less populated experts.
4 Methodology
We now introduce our distillation approach, which uses Mixture-of offture-of-Experts (MoE) model. The method starts with the training of teacher GNN. The teacher model is used to produce soft-labels for the knowledge distillation (see Section 4.2). The knowledge distillation setup uses combination of reliable sampling and positional encoding. Our student model is Nonting-by-MMeman
model, with a special routing procedure that enforces expert specialisation (see Section 4.1p Section 4.3 describes the expert initialization procedure.
Figure provides an illustration of the overall training framework. teacher GNN trained on the graph and provides soft targets for Knowledge Distillation (KD) g and Knowledge-Aware Reliable Distillation (KRD) [2 losses. The student model is trained with the node features and positional encodings as inputs. We introduce spatial routing by memory, at cach layer, cach expert is represented by an embedding in the same space as the input representations for that layer. Representations are then routed to the closest experts. Three types of loss terms, discussed in more detail below, are used to encourage effective learning of the expert embeddings and the hidden representations. Figure 1(b) depicts the goals of these losses. A commitment loss pulls representations closer to embeddings, encouraging specialization experts. A self- similarity loss pushes the expert. embeddings apart and prevents the collapse of the representations. A load balance loss strives achieve more equal utilization of experts by moving representations that are almost equidistant to two more experts closer to the (ess-utilized experts.
4.1 Spatial routing by memory
We use the standard formulation for a Mixture-of-Experte layer, introduced by Shazeer et al. (2016)
h Σ ((h
(1)
where G Rd + [0, _ is policy network that produces routing coefficients, ft Rd Rd is an expert, E is the number of experts, and h E Rd. is the input hidden representation of the datapoint, x E Rd. emerging from the (1-1)-th layer. For the first layer, the hidden representation the input, ie., hg = x.
Li al. (2022) introduce routing scheme that uses set of embeddings, Q MoE E R Exd. with each embedding being associated with particular expert. The weight assigned to each expert for given input
4

---

Published in Transactions on Machine Learning Research (10/2024)
at layer is determined by measuring the cosine distance between the projection of the imput vector hi- and the expert's embedding. The policy network routes the input vector to the l nearest experts:
Q Ma Wh Wh
GMo = softmax
'Top: (T III Wh]l
(2)
Here W Rdexd" is trainable projection matrix, and the operation Topı(-) is one-hot embedding that sets all elements in the output vector to zero except for the elements with the largest & values. Chi et al. (2022) demonstrate that using policy network of this form results in more even distribution of token projections over the projection space, leading to improved performance.
While Li al. (2022) provide analysis demonstrating that GMOE(() provides a degree of specialization for expert networks, we found it insufficient (see the experimental results in Section 5.5). We therefore enforce experts' local specialization by setting the expert embeddings Q QRbM E R to vectors in the input space, rather than projecting to a separate space. We achieve this by setting:
R
GREM (h) softmax
Tops
sg.
sg RDM
where sgl-] is a stop gradient function. In our approach, each expert embedding vector is positioned at the center of an area in the representation space that the expert specialising on. We use cosine similarity and interpolation between multiple experts (see Figure 2
As every embedding vector in our approach can be interpreted as a centre of cluster, we do not update it with gradient descent, but instead direct approach, calculating moving average over the input batches. We evaluate:
B
Q RBM
Q.bbM (t ( +1+1 ((t)Q (t) +11 A(t))
softmax(Grax(hhjb) hy,
(4)
GREM (hy)e
where A(t) = max(A(1) 1.0) coefficient. We use an annealing schedule ensure that X(1) < x(t+1). As 0o, the change in the expert embeddings tends to zero, facilitating the convergence of all embeddings.
We compute this sum only for the vectors that are routed to the expert. Applying the softmax over the batch dimension improves performance.
In order to enhance training we use expert-wise constant attention for input and output. Thus the entire Mixture Experts block can be formulated as:
h = exp(s) Σ GraM (N)r(l)))eexppattt) e h)
where 8 R learnable oulput scaler, att, E Rd a learnable input attention vector, and ø denotes element-wise multiplication. Note that the attention applied only the expert input and not the router input. This routing approach is inspired by the technique presented by Zhang et al. (2021a]; we extend it to perform top-k rather than top-1 routing and employ cosine similarity instead of Euclidean distance.
In order to encourage tighter clustering of hidden representations around the expert embeddings we incor- porate a vector quantization (VQ)-style commitment loss:
B E
RbM
h
Lossvo
GRoM(h))
(6)
sg
RbM
where is the batch size. When MoE layers are stacked, this loss encourages the hidden representations move closer to the nearest expert embeddings. It prevents frequent fluctuations in routing and allows experts
5

---

Published in Transactions on Machine Learning Research (10/2024)
Previous training examples
Softmax
Expert
Expert
Expert
Router
Expert
Expert
Routed vector
Expert
Router
Figure 2: A simplified example of cosine routing (3) Three experts are present in total (E 3). Two experts are used at a time (A: = 2), and thus the two experts with closest embeddings are used. Arrows show expert emboddings on the unit circle. Points are representations of the previously routed training examples (sce equation
Figure 3: Schematic depiction of sludent model with two RbM layers. Three experts are present per layer with two experts used for each sample.
to acquire specialization. Routing to multiple experts prevents the hidden representations from collapse. A similar loss was presented by Razavi et al. (2019) for the image generation task computer vision, and here we adapt it for cosine similarity.
We incorporate self-similarity loss in order to additionally spread embeddings the hypersphere:
sg
RBRM
RbM
Q
RbM
Loss
E
sg
RMM
RbM
For this loss, we allow a gradient descent update of the expert embeddings. Gradients are propagated only through one part of the computation, by using the stop-gradient, to prevent instability.
Finally, we apply load balance loss, as proposed by Shazeer al. (016)):
B LOSSLB Mean (Load) for Load Σ
Var(Load)
Greas (h:).
Here Load E F the vector of unnormalized expert utilization. While expert specialization does not require load balancing, we adopt this loss to redistribute input vectors that lie in almost equal proximity to multiple experts. This loss encourages reprosentations to move closer to the least utilised expert embeddings.
4.2 Knowledge Distillation
We distill knowledge from the pretrained GNN using supervised learning, regularized by the KI-divergence between the class distribution de predicted by the student and the class distribution &. predicted by the teacher. The knowledge distillation loss combines cross-entropy (CE) with KI-regularization:
V |V
1- D-V Σ (L) Σ CE) NEVE
LOSSKD
(9)
DV vey
Here is a hyperparameter that controls how much the knowledge distillation training relies on the soft labels from the teacher GNN, CF((s.) denotes cross-entropy, and K KT(,,)) is KT-ddvergence (Zhang et. al.

---

Published in Transactions on Machine Learning Research (10/2024)
2021b). While the supervised part of the loss uses only the labeled part of the graph, the KI-divergence is computed over the entire set of nodes.
We use positional encodings generated by Deep (Perozzi et al., 2014) in our distillation procedure. Before running the student training we learn positional encodings by running the DeepWalk algorithm on the input graph. The positional encodings are based solely on the graph structure, and are concatenated with the node features when nodes are processed by the student model.
In order to additionally leverage the graph structure, we employ the knowledge- based sampling technique of KRD (Wu et al., ) KRD is based on the principle of aligning the representation of target node with those of retiable nodes in its neighbourhood. To accomplish this, Wu et al. 2023) introduce measure of re- liability, which is based on identifying nodes whose teacher GNN representations do not change substantially when the features in the graph are subjected to perturbation. The reliability metric for node is:
Pj 8 (x)
[[((S)) - H(
(10)
where (1; is Gaussian noise with diagonal covariance matrix, with 5 as the diagonal elements. H(-) denotes the entropy, and Y; and y; are the predicted class distributions for node j with and without perturbation, respectively. This metric is based on the principle that more reliable teacher soft labels are more robust fcature perturbations.
During knowledge distillation, given target node v, we sample a set of nodes from its neighbourhood. The sampling distribution is determined by the reliability metric:
( Pmax
Pj
philp,, a) x
(11)
where Pmax arg ea Pi, and a is a learnable power parameter. To learn a, we follow the procedure of (Wu al. 2023). During training of the overall model, we strive to match the class distríbution predicted by the student y. with the teacher-providee soft labels of the sampled nodes. We therefore include loss term:
LOSSKRD = 1- ΣE nEN()) KI(Sv.9a) [V] DEV umplulpu.a)
(12)
where, for each node v, the expectation is with respect to a distribution p(ulpa, a),, which is proportional to () defined in equation I1 for the nodes u in the neighborhood of v, and zero elsewhere. The hyperparameter y is the same as in equation
The training objective a combination of the cross- entropy loss, the distillation loss, the KRD loss, and embedding losses for every MoE layer of the model. With model with L, MoE layers, we introduce weights a, B, and y to determine the influences of the associated embedding losses:
Our model consist of MoE layers (sce Figure 3
Loss = LOSSKD + LOSSKRD + (osssy(()) + BLossss(i) + LoSSLB())
(13)
4.3
MoE initialization
In most implementations, the embeddings of the experts are initialised randomly CCi et al. 2022 et al. 2022 Yan & Li, 2023). In our approach, we desire more informative initialisation, bocause embeddings are operating in the input space. We therefore apply pretraining stage, following [hhan et al. (2021a). We pretrain the model for several epochs, routing all input vectors to the first expert. Subsequently, we clone the parameters of the first expert to all other experts. We reset the optimiser state after pretraining. We collect the inputs for each MoE layer and apply L2 normalization. We then apply K-means clustering, with the number of clusters equal to the number of experts. We initialise the embeddings with the cluster centers.
Operating directly in the space of the hidden represcntations not only removes the superfluous paramcters of the projection matrix from the model, but also makes initialization of the experts much easier. This helps to avoid representation collapse that can be caused by randomly initialized projection matrix.
7

---

Published in Transactions on Machine Learning Research (10/2024)
5
Experiments
We evaluate our model on nine real-world datasets. We show that our model can utilize additional parameters more efficiently than a parameter-inflated MLP, an ensemble of MLPs, or a vanilla mixturo-of-cxperts model. We conduct an ablation study lo show how the various loss lerms influence accuracy.
5.1 Experimental setting
Datasets. To conduct our experiments we use nine real-world datasels: Cora (en et al.,
2005), Cite-
seer (Giles et al. 1998), Pubmed (McCallum et al.) 2000), Amazon-Photo, AmazonnComppteen Academic- CS, Academic-Physice (Shchur et al.) 2018 DGBBArXivv and OGB-Products (Hu et al.)
2020). For the
Cora, Citeseer, and Pubmed datasets, follow the data splitting strategy specified by Kipf & WWelingg (201)) For the Amazon-Photo, A Amazzon-omputeed
Academic-CS, Academic-Physics, we follow the proce- dure employed by Zhang et al. (2021b), Tian et al. (2022) and Wu et al. (023). We randomly split the data into train aaann/valttess subsets. Each random seed corresponds to a different data split. For the OGB-ArXive and CCB-Products we use the public data splits provided by Hu et al. (2020). Dataset statistics are provided in Academie-CSs Acadomic-Physics, OGB-ArXive and OGB- . Products datasels, we use batched updates due to the large nunber of nodes and edges.
Table 7 For the Amazon-Photo, Amazon-Compunt
When presenting and discussing results, we divide the datasets into large, medium and small categories, according to the number of training nodes available. Our method focuses on large and medium-sized datasets. In general, more complicated architectures and distillation procedures struggle to achieve performance gains on small datasets, as demonstrated by Zhang et al. (2021b)
Basclines. We compare to three node classification basclines that use GNN-to-MLP knowlodge distillation: NOSMOG (Tian et al. 2022). KRD (Wu et al., 2023) and GLNN (Zhang al. 2021b). All baselines are reproduced using provided official code and hyperparameters
I By default, we use GraphSAGE (Hamilton) et al. 2017) the teacher model, in order to facilitate comparison with previous work. We do, however, examine how the method performs with other teacher models. We also compare to CoHOp (Winter et al. 2024), a baseline method that does not employ a teacher, but has a higher inference cost. In order to compare results with a similar parameter count, we provide four parameter-inffated baselines: NOSMOG+ KRD+, GLNN+ (Zhang et al. 2021b) and CoHOp+. For these baselines, we increase the number of student parameters to be & times the teacher size, following ZwwwwinggZanggggo Zhang et al. (2021b) We conduct 10 runs for our method and cach bascline, with the same sequence of seeds for all methods.
Model. Our Routing-by Nouting-by-Memoon by-Memory models have the same number of layers as the teacher. Every expert is a linear layer with the same size as the corresponding layer of the teacher. We use up to 8 experts for RbM (the exact number for cach dataset is selected using the validation set). We use the same number of experts for all the RbM layers inside the model. Three experts are active at a time. RbM routing is sensitive to dropout, so we avoid the application of dropout directly before the RbM layers. In our model, we apply dropout before the input of the expert for each layer except the first one. We use linear annealing schedule for the updating of the expert embeddings (see Appendix G..
Evaluation protocol. We report the mean and standard deviation of accuracy for ten separate runs with different random seeds. We use the same sequences of seeds reported as Tian et al. (2022) and Wu et al. (2023). We use validation data to select the optimal model. The hyperparaneter selection procedure is described in Appendix C We measure model performance using test data.
We conduct our experiments in two settings: transductive (trans) and inductive (ind). For the transductive setting we Irain the model on the full sets of nodes, V,, and edges, E. Classification loss is only computed over X and YL, but soft labels for KL-divergence and KRD losses are computed and E. In the transductive setting we evaluate the model over X and YU, For the inductive sctting, we split the unlabeled nodes, DL, ; into a set of observed nodes, Vy and a set of inductive nodes, D by randomly selecting 20% of the nodes as the inductive subset, following the procedure of Tian et al. (2022)
the full sets of X
I We were not able to reproduce reported NOSMOG results the OGB-Products datasct for the inductive sctting using the official code; for this dataset and setting we provide results using our implementation of NOSMOG. We are not able to provide results for KRD+ on OGB-ArXive requires more than 32 GB of VRAM run.
8

---

Published in Transactions on Machine Learning Research (10/2024)
and Zhang et al. (021bb. We partition each graph in such way that the training nodes Virain Vo L Val and inductive nodes Vind have no edges between them. We denote the edges between the training nodes Etrain- Classification loss is only computed over x and yt, but soft labels for KL-divergence and KRD losses are computed over Xtrain = = obs and Etrain. In the inductive setting we evaluate the model over Xnd and And For both settings, the embedding losses are computed for all model inputs during training.
In order to achieve the best baseline performance under our evaluation protocol, we reproduce baseline results with publicly available hyperparameter configurations for the datascts. We do not provide results for GLNN, NOSMOG, and CoHop for the Academic-CSs CS and Academic-Phy datasets, nor KRD for the Amazon- Comp datasct, because they are not reported in the original papers. All four algorithms are sensitive to hyperparameter selection; we cannot identify values that lead to reasonable performance on these datasets.
To compare performance across the datasets, we report median Score, where Score is a Min-Max normal- ization of the mcan accuracy into the [0, 1] interval. The algorithm with worst performance on the datasct obtains a Score of 0 and the best performing algorithm is assigned Score of 1. We apply the Skillings-Mcc test with significance level of 5% to assess performance of the algorithms across all datasets. The test is applied separately for each evaluation setting (transductive and inductive).
5.2 Performance comparison
We compare our method to GLNN, KRD, NOSMOG and CoHOp baselines. Results are presented in Table 1 for GraphSAGE as the teacher, and in Table 2 for more advanced teacher GNNs. We use RevGNN-Wide (Li et al. 2021) and DRGAT (Zhang et al. 2023 as the advanced teachers, because they are among the best
performing GNN models for the larger OGB datasots.
We make the following observations:
1. Table I shows that RbM consistently ranks first or second for the medium and large datasets. can be successfully applied to small datasets but without meaningful performance gains. RbM outperforms all the baselines on medium sized datasets. It is outperformed only by CoHOp the large datasets. We discuss this in Section 5.4 but for now, we note that CoHOp employs computationally burdensome label propagation.
2. KRD, NOSMOG, and RbM often outperform the teacher model. This has been observed previously (Wu et al. 2023 Tian et al. 2022). Distillation can lead to better generalization and renders the prediction architecture less susceptible to spurious edges. In addition, the students form predictions using both node fcatures and structural information (via positional encoding or DeopWalk)) whereas the teacher focuses primarily on the features (the impact of graph structure is much less direct, arising from message passing).
3. Although GLNN performs reasonably well for small and medium datasets, with accuracy close to that of the teacher, it struggles with the large OGB datasets. The node feature information is highly informative for the small/medium datascts. In contrast, for the large datascts, with sparser labelling, the access to graph information is important. This is achieved by neighbourhood aggregation for the teacher, and positional encoding for the students.
4. Table 2 indicates that better teachers lead to improved performance of the distilled models (except for GLNN). The proposed method demonstrates similar outperformance with respect to the baselines. The distilled models do not outperform the more advanced teachers that can more effectively incorporate graph information.
In order to demonstrate that our approach leverages additional parameters better than the baselines, we conduct experiments with parameter- -infattt--ifftted inflated baselines (see Table 3) These expanded baselines are denoted GLNN+ KRD+, NOSMOG+ and CoHOpt. In this experiment every baseline has the number of parameters increased 8 times the teacher size, as in Zhang ct al. (2021b) and Tian et al. (202)). The size of every hidden layer is increased, but other parameters are not changed. CoHOp does not use a Leacher; we increase the number of parameters by a factor of 8 compared to the model in the original paper. Table 3 indicates that a larger number of parameters can improve performance for some cascs, but we do not observe a consistent performance improvement for any baseline. Even when there performance improvement, RbM remains the peest-perfoming algorithm in 9 out of the 12 settings for medium and large datasets. We apply the Skillings-
9

---

Published in Transactions on Machine Learning Research (10/2024)
Table I: Performance comparison with GraphSAGE teacher. Results show accuracy (higher is better). The best model is highlighted in bold. Second best underlined. Scores are statistically significant under the Skillings-Mace Mack test with significance level of 5%. For the inductive setting, p < 0.003, and for the transductive setting p 0.001. (*) Baseline is reproduced with changes in the official code.
Dataset
Eval CraphSAGE
GLNN
KRD datasets
NOSMOG
CoHOpp
RPM
Small
knd 82.13+0.50
73.28+1.98
65.44:1 16.35
81.03
80.38 1.74
77.44
Cora
Iron 82.0810.63
79.20+33.07
84.56:0.62
11.97
1.29
84.86=0.46
ind 70.24+0.62
70.11+3.04
71.74+0.46
71.82+3.06
71.77=3.77
70.96::0666
Citeseer
Iran 7..7111.68
70.3212.18
71.07.88:44
1..90.1.37
75.64.1.699
72.98
PubMed ind 77.60+0.48
75.05+2.95
81.59=0.60
75.69+33008
34.84=3.39
81.30+0.00
Iron 77.36+0.45
76.65+2.66
81.65=0.46
77.56+22.35
77.22=2.49
81.64=0.14
Medium size datasets
ind 82.2811.01
80.4611.43
1.90
80.2612.52
85.07=1.66
Amazon-Comp
82.34+11.44
82.67+1911
83.05: 1.55:11.33
81.01=1.58
85.22=0.85
ind 92.14+1.00 1.08
90.45+1.12
91.43:0.88 82
92.16::1.12
91.85=1311
93.06=0.74
Amazon-Phote
tron 91.93+0.96
92.44+0.89
93.28+0.38
93.16::0.93
93.06=1.56
93.62=1.25
ind 89.25=0078
1..28+1.13
93.57=0.65
Academic-CS
fron 89.04+0.37
93.54+0.54
93.62=0.25
ind 92.73.00.44
93.85.10.59
94.67
Academic-Phy tran 92.78+0.61
94.35=0.36
94.34=0.26
Big datasets
knd 70.65
59.13+0.55
60.8410.58
67.97:0.46
71.1810.47
71.31
OGB-ArXive fron 71.60+0.26
64.6800.20
71.6440.26
70.45++0.37
72.79+0.09
72.48+0.11
ind 76.98110.48
60.2210.30
7..29
81.6810.21
80.88 0.24
OGB-Products
Iran 77.47
60.34.10.31
77.19.10.41
81.6710.225
$1.04
ind 0,4200
0.0000
0.5773
0.7234
0.8951
0.9967
Med. Score
tran 0.1420
0.0000
0.9470
0.4894
0.6696
0.9980
Mack test to assess whether the RbM outperformance is statistically significant. For all experimental settings, p-values are smaller than 0.02, and in most cases are smaller than 0.005, indicating statistical significance.
Since we perform pretraining and clustering in our model initialization, there is some additional computa- tional overhead compared to training crrameter-inflater baselines. In our experiments, pretraining does not exceod 7% of the total number of training epochs on average (in our implementation we use carly stop, so the number of epochs can vary). The clustering overhead is effectively insignificant. On our setup, takes 80 seconds to cluster all required embeddings for RbM DGB-ArXive one of the larger datasets.
Table 2: Accuracy for advanced teacher models. Performance with the GraphSAGE teacher is provided for reference. OGB-vrXive dataset is used in transductive setting for the experiments. Scores are statistically significant under the Skillings-Man Mack test with significance level of 5% (p 0.0193)
Teacher model
Teacher
GLNX
KRD
NOSMOG
RbM
GraphSAGE
71. 10.60.0.26
64.68:10.20
71.6410.26
70.45-10.37
72.48-10.13
DRGAT
73. 73.63:00007
63.6140.24
72.21:0.11
70.71=0.12 70.75+0.11
73.10=0.04
RevGNN
73.9840.01
73.724..24
72.44+0.14
73.26+0.06
Median Score
0.0
0.9072
0.7397
1.0
5.3 Comparing with ensemble and vanilla MoE
To additionally explore whether approach is efficient mechanism for exploiting additional parameters, construct two baselines: soft-voting ensemble of MLPs and a vanilla MoE. The soft-voting ensemble consists of several MLP students with the same structure, but different random initializations. The three- MLP ensemble has the same inference cost as active experts in the Mixture of Experts, and the eight-MLP ensemble has the same parameter count as all experts. Appendix[F provides extensive analysis on parameter number and inference complexity. The vanilla Mixture of Exports model structured in the same way as the proposed RbM model, but it uses a routing scheme where the policy network and embeddings are initialized randomly and updated with backpropagation CCi et al. 2022
Li et. al. 2022
Yan & Li 2023 For all
10

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Comparison with scaled baselines, with GraphSAGE teacher. Results show accuracy (higher is better). Sizes of the baseline MLPs are adjusted 8 time the teacher size. RbM uses up to experts. The best model is highlighted in bold. Second best underlined. Scores are statistically significant under the Skillings-Man Mack test. (Skillings & Mack 1981 with significance level of 5%. For both transductive and
inductive settings, p < 0.001.
Dataset
Eval
GraphSAGE
GLNN+
KRD+
NOSMOG+
CoHOp+
RbM
Small
datasets
ind 82.1310..
73.54.12.18
16.50.16634
77.3811..39
80.12.12.38
77.44
Cora
tran 82.08.10.63
78.9713.18
84.78-10.1
76.04110.00
82.99111.21
84.8610.48
ind 70.24:10.2
67.8542.79
71.8310.311
62.68:1..57
71.77+3..7
70.96
Citeseer
tran 70.71+1168
71.14422.4
70.57=9.58
71.24+4..7
75.6441.69
72.98
ind 77.60:10.48
75.1492.79
81.68:00.48
75.30:33.10
74.84
81.30
PubMed
tran 77.36++0.45
76.1342..1
81.58+0.40
77.69+2.41
77.22+2.49
81.64=0.14
Medium size datasets
ind
12.28+1.011
79.98+1.21
83.18+ 18-1-.655
80,27+2.51
85.07+1.66
Amazon-Comp
tran
1.13
$2.45.12.14
83.21 L.55
81.0111.58
85.22-0.885
92.14
19.92+1.75
91.43+00.9
92.29+ 1.13
11.85
93.06:10.76
Amazon-Photo
91.93:+00.6
9..24+1.32
93.36:00244
1..1
13.06
93.62+1.255
ind 89.25:t0078
92.57
93.57
Arademic-CS
trom 89.04:10.37
93.77:10.51
93.62-10.25
ind 92.73:t0.64
94.06=00.55
94.67+0.27
Academic-Plp
tran 92.78.10.61
04.50.1..33
94.34
Big datasets
71.35+0.62
60.44+0.66
68.79+0.77
65.95+3.35
71.31=#0.20
OGB-ArXive
OOM
71.60:+0.26
70.23++0.25
67.37+5.82
66.22+5..8
72.48+t0.13
ind 76.98:10.48
73.8340.223
76.30=1.11*
77.11+10.27
80.88:1.224
OGB-Products
trom 77.47:10.277
77.17440.31
77.50+0.44
77.95++0.28
81.04+0.37
inud 0.4519
0.0000
0.7270
0.6287
0.5050
1.0000
Med. Score
Iraz 0.0775
0.1834
0.9900
0.1750
0.2016
1.0000
modols, each sample is routed to three experts. We pretrain the vanilla Mixture of Experts model for several epochs, routing all inputs to the first expert. The weights of the pretrained first expert are then cloned to the other experts. Embeddings are not updated during the pretrain stage.
Table 4 shows that RbM outperforms the soft-voting ensemble and vanilla MoE basclines in 10 out of 12 of the settings for the medium and large datasels. The vanilla MoE outperforms the ensemble of MLPs for most settings (8 out of 12) on the medium and large datasets. The vanilla. MoE has fewer hyperparameters than RbM, and thus it may be preferable if there is need to reduce the tuning overhead.
5.4 Ablation study, label propagation, and number of experts
Loss terms. We now examine whether each component of the equation 13 important for achieving better performance. Our model includes three additional loss terms for each RbM layer (see equation [3 commitment loss 16) self-similarity loss ( and load balance loss S In order to conduct the ablation study we remove each component individually. None of the other hyperparameters is changed. The results are presented in Table 5 and show that removing any of the loss components reduces performance. The performance deterioration relatively small for each dataset, but is observed for every setting and every loss. is not clear that any of the three loss terms is the most important. Any combination that includes two loss terms perperfosms (by a small margin) the baseline with all three removed for all medium and large datasets, indicating that all three loss terms contribute.
Label propagation. By default, we use DeepWalk positional encoding as an additional set of features. RbM is fully compatible with additional posilional information that can be extracted from the graph. As an example, Table D demonstrates that including the label propagation information from CoHOp can increase the performance of RbM on the datasets where considcrable portion the nodes are labeled (OGB-ArXive, DGB-Products), Generating the label propagation information requires propagating and averaging one- hot encoded training labels from 10-hop neighbourhood around each nodes. In an inductive setting, this must
11

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Comparison with ensemble and MoE baselines. 3xMLP and SxMLP are the soft-voting ensembles of three and eighth MLPs correspondingly trained with shared teacher, but with different initializations. Vanilla MoE consists of 8 experts, but routes imputs to three. The best model is highlighted in bold. Second best is underlined. Scores are statistically significant under the Skillings-Mack test with significance level of 5%. For the inductive setting, p 0.004, and for the transductive setting p 0.0337.
Dataset
Eval
3xMLP
8xMLP
MoE
RbM
Small datasets
ind
71.7011277
66.90.1.44
72.98.11.64
77.44 1.08
Cora
fron
84.20:10.0
84 64.0331.225 0.25
84.86.10.29
84.86:10..4
ind 71.60:10.40
70.9410.44 44
71.76:10.47
70.96:10.66
Citeseer
tron 67.574 .9
68.62+11.02
73.58:10.43
72.98
ind 81.08:0..00
80.37 0.43
0..50:0..5
$1.30:00.49
PubMed fron 01.30=+0.28
81.10=0.22
81.80:10.42
81.64+0.14
Medium size datasets
ind 84,54+1.51
1.38
84.88+1
35.07=1.66
Amazon-Comp
fron 84.58.0.88
83.7211158
84.43.10.89
85.22 0.85
ind 9.22
92.51+1.43
92.66-+0.90
93.06=0.74
Amazon-Phoct
Iron 9..34+1.02
93.16+1.03
93.56+1
93.62
ind 90.69
93.45+0.66
93.43:10.59
93.57 0.65
Academic-CS fron 93.79-1036
93.79:10.37
93.97:10.36
93.62:10.22 0.25
ind 94.32+0.45
94.11=0.54
94.47=0.19
94.67=0.27
Academic-Phn Iran 94.43 0.32
94.3410..58
94.01 10.49
94.34.10.28 2626
Big datasets
OGB-ArXiv ind 70.19+0.68
69.98+0.52
70.96:00.60
71.31=0.20
fron 72.16+0.33
72.26+0.32
71.49+00.51
72.48=0.13
ind 80.03:10.35
80 80.35:0..00 338
80.80:10.16
80.88=0.24
OGB-Products fron 00.1750.33
80.45+0.34
80.64:10.34
81.04=0.37
ind 0.375
0.0
0.7368
1.0
Median Score
fran 0.3913
0.1747
0.8696
1.0
Table 5: Ablation study on equation 13 loss components. GraphSAGE teacher used. Results are showing accuracy (higher better). RbM performance provided for the reference. For "KD only" column we are reporting results with KD and KDR losses only and no embedding losses.
Dataset
Eval ind
RbM
13
w/o VQ 6
w/o SS 7
w/o LB 8
KD only
77.44=1
1.08
74.62+1.50 56
75.92++0.89
74.68+180
74.11+1.56
Cora
tron
84.86=0.46
84.70+00.66
04.6440.26
84.64+0.39
84.08+0.64
ind 70.96-0.63
69.8010.20
70.22:10.40
69.52-10.52
70.00110.30
Citeseer
tran 72.98-0.82
1..38+1.15
72.60+0099
72.99+0.91
72.28+10.74
PubMed ind 81.30=0.00 0.49 49
80.9240.29
81.00::0.45 48
18:.033
80.80:10.95
fron 81.64=0.14
81.38-10.1 16
81.38:+0.31
81.4660..6
81 01.28:0004 47
ind 85.07=1.66
84.22+1.50
83.95+1..99
84..0=+1.65
83.92+2.00
Amazon-Comp
fron 85.22=0.855
83.55+1.33 1.24
84.35+1..
84.02+0.68
83.67+0.88 84
ind 93.06_0.74
92.31
92.34.11.03
92.76.11.24
92.00.11.22
Amazon-Photo
fron 93.62=1.25
93.27+00.96
92.98 1.42
93.43++1.54
92.59t
Academie-CS ind 93.37=0.65
03.46-10.2
93.31-10.99
93. 93.26:10.72
93.10110.78
fron 93.62-0.25
03.35++048
93,22-10.50
93. 93.1240.63
93.09110.33
ind 94.67=0.27
94.62-0.29
94.51+-0.25
94.443...27
94.35.10.30
Academic-Phy
fron 94.34=0.26
94.26=10.28
94.29++0.22
94.25+00.27
94.23:10.27
ind 71.31
71.10.10.28
71.15.10.29
71.2810..0
71.12.10.2 .26
OGB-ArXive
fron 72.79=0.09
72.38+0.22
72. 02.
72.25+0.22
72.20+0.27
ind 80.8810.24
80.54110.29
80.61-L0.33
80.6110.31
80.3810.227
OGB-Products
81.04=0.37
80.7710.21
80.50:10.42
80.78:10.27
80.4110.114
be conducted for each new node, and it can become time bottleneck due to the overhead of fetching node labels from memory.
Number of experts. During our experiments we use the same number of experts for all RbM layers in order to reduce the number of hyperparamcters. We found that there an optimal number of experts for RbM for each dataset that can be identified using validation data (from the range [3....,8) 8]). Appendix D provides results depicting example of how performance varies as the total number of experts is changed.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Table Applying label propagation positional encoding on OGB datasets.
w/o Label propagation
Label propagation
Dataset
Eval
CoHOpp
RbM
CoHOp
RbM
54.36+0.79
71.31+0.20
71.1840.47
75.06:t0.45
OGB-ArXive
tron
54.344:.72
72.79+0009
71.35:t0.22
72.27+0.21
and
33.54:0.61
80.88:1024
81.68:00.21
61.6210.25
OGB-Products
Inan
32.84.1.577
81.0410.37
81,67.0..25
81.68.10.25
5.5 Routing spatial structure analysis
(a) Plain MoE internal representation colored by top ex- pert (left) and by label (right).
(b) RbM internal representation colored by top expert (left) and by label (right).
Figure 4: Analysis of hidden representation for RbM (b) and its projection for MoE (a). Each point represent instance from the Academic-Physic dataset in transductive setting.
In order analyse the routing spatial structure qualitatively, we utilise the T-SNE (Van der Maaten & Hinton 2008), with perplexity of 30 and PCA initialization, to produce 2-d visualizations of router embedding space for RbM and vanilla MoE in Figure 4 These correspond to the hidden representation h for RbM (see equation 3) and the linear projection of the hidden representation, Wh, for the MoE (see equation 2 We trained both MoE and RbM models on the Academic-Physies dataset in the transductive setting with the same teacher and selected the router of the last layer to produce the representation. Figure 4 the hidden representations are colored according to the labels (on the left) and according to the top-score expert (on the right). Figure shows that MoE experts mix and distribute datapoints of the same classes between different experts, while RbM experts have clear specialization.
6 Conclusion and Future work
In this paper focused the task of distillation from graph neural network and introduced RbM, Mixture of Experts model that encourages strong expert specialization at the routing level. We established how parameter inflation can positively affect the performance and showed practical application of MoE. in the knowledge distillation domain. Our approach outperforms existing baselines on most medium-size or large datasets. The key innovations of our approach is in embeddings to be the part of the hidden representation space.
We used additional losses hidden space is forced into the shape of multiple elliptical clusters, which could be too restrictive and thus suboptimal. In order to reduce number of hyperparameres we assumed all layer of RbM to have the same number of experts and thus clustered into the same number of clusters. Selecting suitable number of experts for the layer automatically can improve the performance of the overall model. Both off these assumptions led to the model being more sensitive to the nunber of experts than the plain MoE. We leave those direction of improvement as a future work. In addition to that a a future direction can be about the application of the MoE in graph domain. This work uses concatenation of positional and feature vector for both routing and selected experts processing, however an alternative approach can be route with positional vector while feature vector is supplied to the selected expert.
13

---

Published in Transactions on Machine Learning Research (10/2024)
References
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proc. Int. Conf. Knowl. Discovery and Data Mining, 2019.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the Representation Collapse of Sparse Mixture of Experts. In Adv. Newral Inf. Process. Syst. 2022.
Nikoli Dryden and Torsten Hoefler. Spatial Mixture-of-Experte In Adu. Newral Inf. Process. Syst., 2022.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. J. Mach. Learn. Res., 23(120), 2022.
C Lee Giles, Kurt D Bollacker, and Steve Lawrence. CiteSeer: An Automatic Citation Indexing System. In Proc. Conf. Digit. Libraries, 1998.
Sam Gross, Marc' Aurelio Ranzato, and Arthur Szlam. Hard Mixtures of Experts for Large Scale Weakly Supervised Vision. In Proc. IEEE Conf. Comput. Vision and Pattern Recognit., 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs. In Adv. Neural Inf. Process. Syst., 2017.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open Graph Benchmark: Datasets for Machine Learning on Graphs. In Adv. Neural Inf. Process. Syst., 2020.
Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang, Erjin Zhou, and Yue Gao. Graph-MLP: Node Classification Without Message Passing in Graph. arXiv preprint arXiv:2166.04 04051, 2021.
Zhihao Jia, Sina Lin, Rex Ying, Jiaxuan You, Jure Leskovec, and Alex Aiken. Redundancy- freee Computation for Graph Neural Networks. In Proc. Int. Conf. Knowl. Discovery and Data Mining, 2020.
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph Condensation for Graph Neural Networks. In Proc. Ind. Conf. Learn. Represenlations, 2021.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint 1412. 6980, 2014.
Thomas N Kipf and Max Welling. Semi-Supervised Classification with Graph Convolutional Networks. In Proc. Inl. Conf. Learn. Representations, 2016.
Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostala Dehghani, and Neil Houlsby. Sparse Upcycling: Training Mixture-of-Fxpertss from Dense Checkpoints. In Proc. Int. Conf. Learn. Representations, 2022.
Yamuna Krishnamurthy, Chris Watkins, and Thomas Gaertner. Improving Expert Specialization in Mixture of Experts. arXiv preprint arXiv:2302.14 2023.
Guillaume Lample, Alexandre Sablayrolles, Mare' Aurelio Ranzato, Ludovic Denoyer, and Herve Jegou. Large Memory Layers with Product Keys. In Adv. Neural Inf. Process. Syst., 2019.
Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse Mfxturre-o--Exprts are Domain Generalizable Learners. In Proc. Int. Conf. Learn. Representations, 2022.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training Graph Neural Networks with 1000 Layers. In Proc. Inl. Conf. Mach. Learn., 2021.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. Tune: A Research Platform for Distributed Model Selection and Training. arXiv preprint 1rX 2018.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore. Automating the Construc- tion of Internet Portals with Machine Learning. In Proc. Inf. Retrieval, 2000.
Svellana Pavlitska, Christian Hubschneider, Lukas Struppek, and J. Marius Zöllner. Sparsely-gated Mixture- of-Expert Layers for CNN Interpretability. In Proc. Int. Joint Conf. Neural Netw., 2023.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. WeepWalk: Online Learning of Social Representations. In Proc. Int. Conf. Knowl. Discovery and Data Mining, 2014.
Jingang Qu, Thibault Faney, Ze Wang, Patrick Gallinari, Soleiman Yousef, and Jean-Charles de Hemptinne. HMOE: Hypernetwork--bseed Mixture of Experts for Domain Generalization, ar Xiv preprint ar arXiv:2211.0888 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating Diverse High-fidelity Images with VQ-VAE- 2. In Adv. Neurol Inf. Process. Syst., 2019.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Raa Collec- tive Classification in Network Data. AT Mag., 29(3), 2008.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In Proc. Int. Conf. Learn. Represenlations, 2016.
Oleksandr Shehur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of Graph Neural Network Evaluation. arXiv preprint orXiv:1811.0586 2018.
John H Skillings and Gregory A Mack. On the use of a Friedman type statistic in balanced and unbalanced block designs. Technometrics, 23(2), 1981.
Qiaoyu Tan, Daochen Zha, Soo-Hyun Choi, Li Li, Rui Chen, and Xia Hu. Double Wins: Boosting Accuracy and Efficiency of Graph Neural Networks by Reliable Knowledge Distillation. In Proc. Int. Conf. Learn. Represenlations, 2022.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. NOSMOG: Learning Noise-robust and Structure-aware MLPs on Graphs. In Proc. Adv. Neural Inf. Process. Syst., 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing Data Using t-SNE. J. Mach. Learn. Res., 9(11), 2008.
Daniel Winter, Niv Cohen, and Yedid Hoshen. Classifying Nodes in Graphs without GNNs. arXiv preprint arXiv:2402.0503, 2024.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan Z. Li. Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs. In Proc. Int. Conf. Mach. Learn., 2023.
YaChen Yan and Liubo Li. Edsembeeee:: Learning Adaptively Sparse Structured Ensemble Network for Click-through Rate Prediction. arXiv preprint arXiv:2301.0833 2023.
Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the Knowledge of Graph Neural Networks and Go Beyond Tt: An Effective Knowledge Distillation Framework. In Proc. Web Conf-, 2021.
Dalong Zhang, Xin Huang, Ziqi Liu, Jun Zhou, Zhiyang Hu, Xianzheng Song, Zhibang Ge, Lin Wang, Zhiqiang Zhang, and Yuan Qi AGL: A Scalable System for Industrial-purpose Graph Machine Learning. In Proc. Conf. Very Large Data Bases Endowment, 2020a.
Kaipeng Zhang, Zhenqiang Li, Zhifeng Li, Wei Liu, and Yoichi Sato. Neural Routing by Memory. In Adv. Neural Inf. Process. Syst., 2021a.
Lei Zhang, Xiaodong Yan, Jianshan He, Ruopeng Li, and Wei Chu. DRGCN: Dynamic Evolving Initial Residual for Deep Graph Convolutional Networks. In Proc. Conf. Artif. Intell. 2023.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less Neural Networks: Teaching Old MLPs New Tricks Via Distillation. In Proc. Inl. Conf. Learn. Represenlations, 2021b.
Wentao Zhang, Xupeng Miao, Yingxia Shao, Jiawei Jiang, Lei Chen, Olivier Ruas, and Bin Cui. Reliable Data Distillation on Graph Convolutional Network. In Proc. Int. Conf. Manage. Data, 2020b.
Chenguang Zheng, Hongzhi Chen, Yuxuan Cheng, Zhezheng Song, Yifan Wu, Changji Li, James Cheng, Hao Yang, and Shuai Zhang. ByteGNN: Efficient Graph Neural Network Training at Large Scale. In Proc. Conf. Very Large Dala Bases, 2022.
Wenqing Zheng, Edward W Huang, Nikhil Rao, Sumeet Katariya, Zhangyang Wang, and Karthik Subbian. Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods. In Proc. Int. Conf. Learn. Representations, 2021.
Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng Hc, Tuo Zhao, and Weizhu Chen. MoEBERT: from BERT to Mixture-of-Expertss via Importance-guided Adaptation. In Proc. Conf. North Am. Chapter of the Assoc. Comput. Linguistics: Human Lang. Technol., 2022.
A Datasets description
In Table 7 we provide the key statistics of the datasets we used to evaluate our models: Cora (Sen et al, 2008), Citescer (Giles al.) 1998), Pubmed (McCallum al. 2000), Amazon-Photo, Amazon-Computerr, Academie-CS, Academic-PP
Phyiiisccs- Physics (Shchur et al. 2018), OGB-Ar Aive and OGB-Products (Hu et al. 2020
Table 7: Dataset statistics
Dataset
Nodes
# Edges
Features
# Classes
Cora
2485
5069
1433
Citeseer
2110
3668
3703
PubMed
19717
44324
500
Amazon-Comp
13381
245778
767
10
Amazon-Photo
7487
119043
745
Academic-CS CS
18333
81894
6805
15
Academic- Phy
34493
247962
8415
OGB-ArXive
169343
1166243
128
40
OGB-Products
2449029
61859140
100
47
B Hardware specification
Our experiments were conducted using an NVIDIA Tesla V100 GPU with 32GB of memory. The machine has an Intel Xcon Gold 6140 CPU with clock frequency of 2.30GHz and total thread count of 36. All computations, with exception of the clustering, were executed the GPU. For Cora, Citeseer, PubMed, Amazon-Comp Amazon-Photo and Academic-CSS datasets Each parallel run was allocated 6GB of GPU memory and 5 thrcads for the clustering. For Academic- OGB-ArXive PeBB-Products we executed only one run at a time with 32GB of GPU memory and 10 threads for clustering. We were unable to run KRD+ on our setup as it requires more than 32GB of memory.
executed five parallel runs simultaneously,
C Hyperparameter tuning
We use Ray Tune [Laaw et al. 2018 to tune model hyperparameters. Specifically, we use the Optuna scarch
algorithm Akiba el al. (2019). We sample 200 hyperparameter configurations for small and medium datasets and 80 for the large datasets. We tuned the following model structure hyperparameters: (i) dropout rate was selected from 0.1, 0.2, 0.3, 0.4, 0.5, 0.6] and applied to all dropout layers in the model; (ii) total number of experts was selected from [4, 7, 8). In addition to the structure hyperparameters, we selected the
following training hyperparameters: (i) learning rate for Adam optimizer (Kingma & Ba 2014 chosen
16

---

Published in Transactions on Machine Learning Research (10/2024)
73.0
72.79
Transductive Inductive
72.53
72.45
72.5
72.38
72.12
Accuracy
72.0
71.5
71.31
71.22
71,23
71.0
70.85
70.82
7
8
Clusters
Figure 5: Test set accuracy with respect to the number of experts/clusiee for RbM on OGB-ArXive dataset. The optimal number of clusters (5) is clearly identifiable in both transductive and inductive cases.
from [0.01, 0.005, 0.001]; (ii) weight of the commitment loss from the range [0.0, 0.1]; (iii) weights / and 7 of the the load-balancing loss and self-similarity loss correspondingly from the range [0.0, 0.05).
Comparing to distillation setup with MLP student, training setup introduces three loss hyperparameters (see equation 13 and an embedding update hyperparameter (sec equation 4 Due to the usage of hyperparameter sampling, these additional hyperparameters do not increase the search time.
D Selecting the number of experts
As discussed in the main text, the performance of RbM does vary as the total number of experts is changed. During our experiments we use the same number of experts/cle ssperts/clusters /clusters for all RbM layers in order reduce the number of hyperparameters. We found that there is optimal number of experts for RbM that can be identified for each datasct using the validation dataset (from the range [3....88)).
Figure 5 shows how the accuracy depends on the number of experts for the OGB-ArXive dataset. These results are for the test set, but similar results are observed for the validation set, thus allowing selection of the best model. The optimal number exports can be clearly identified for both the transductive and inductive settings (in the depicted case, for each).
E GCN Teacher Model
We investigate whether our model is compatible with an alternative teacher GNN and demonstrates the same advantages over the baselines. The main paper provides results for GraphSAGE the teacher, together with some results for more advanced GNN teachers. Table g provides additional results for experiments in transductive setting with GCN (Kipf & Welling 2016 as the teacher. We use the originally reported parameter seltings of the baselines for this experiment. These were selected for a GraphSAGE teacher, so it is possible that parameter tuning could improve performance.
F Complexity analysis
In this section we characterize the complexity of the models. All the layers in each model are identically structured, and the number of layers is the same the teacher's number of layers. Thus, we characterize
17

---

Published in Transactions on Machine Learning Research (10/2024)
Table 8: Distillation to RbM results using GCN teacher transductive setting, Results are showing accuracy (higher is better). The best model is highlighted in bold. Second best underlined. Scores are statistically significant under the Skillings- Mack test with significance level of 5% (p 0.0015)
Dataset
GLNN
KRD
NOSMOG
RbM
Cora
79.39
81.12:0..57
80.9341.65
85.02110.29
Citeseer
69.28+3.12
74.86=0..58
73.78+1.54
74.24+0.20
PubMed
74.81+2.39
81,98+0.41
75.80+3.06
82.74+0.09
Imazon-Comp
82.63+1.40
83.72+1.44
84.41+1.56
Amazon-PPhole
92.6810.55
92.211L44
92.44.10.51
93.6110.91
Academie- CS
94.0810.34
93.10-10.40
Academic-Phy
94.30+0.46
94.31+0.15
OGB-ArXive
61.46-+0.33
70.92+00.21
71.10+0.34
71.67+0.19
OGB--rodduct
63.92::0.61
77.41=0.21
79.20=0.15
Median Score
0.0
0.9042
0.6124
1.0
Table 9: Complexity analysis for the student models. Since we are using sparse routing, we present active parameter counter along with the total parameters. Only active parameters are participating in the inference. Note that these empirical results are specific to our setup and are not generalised.
Parameter count
Layer
Inference time complexity
Total
Active
MLP
F(N + 1)
F(N +1
O(FN) O(E"FN O(E + EH + FH) ONE +FF
Ensemble
FN +1 F(EN + E 1) + H(F+ +E) E)
E" P(N +1)
MoE
F(E' E +1) + H(F + E)
RbM
EF(N +1) EF EF
E'F(NN EF
the parameter count and computational complexity of single layer for MLP, MoE and RbM. We also contrast this with an ensemble of MT.Ps. The feature size of the input vector is denoted by F. We assume that MLP layer is linear layer of projection size N and bias. We denote the total number of experts in MoE and RbM by E, and the number of active experts during inference by E < E. Note that E < F and E < N. We set the number of MLPs in the ensemble to be equal to the number of active experts. Each expert of MoE or RbM layer is a linear layer of projection size N and a bias. Routing is conducted according to equation
3 for RbM and equation 2 for the MoE. During the inference, routing procedure conducted descide which experts to run, therefore active parameter counter and time complexity both contain E dimensionality. MoF routing embeddings have internal size of Н. Note that H > F and it is common for intermediate MoE layers to have H N. For the RbM layer utilise trainable constant attention that multiplies each feature with scalar. In the ensemblee input is always routed to all the members, thus all parameters of the ensemble are contributing to computation complexity.
From Table g one can see that MoE and RbM have some additional complexity comparing to the ensemble of MLPs that comes from the router. Thus if ensemble is expected to be E times slower that MLP student, RbM is expected to be E + 1 times slower. However, RbM is faster than MoE that utilises projection into embedding space while RbM uses embeddings that are in input space.
G Annealing schedule
In order to ensure the convergence of expert embeddings, apply lincar annealing schedule:
X() to (1 (1-
(14)
T
where 0 < A < is an annealing hyperparameter, T is the expected number of epochs and Ag is a constant that specifies the initial update rate. In our experiments, we set Ao = 0.9, T 200 and A 0.05.
With the following experiment, we demonstrate how varying affects performance of RbM. We repeat the inductive and transductive experiments for the OGB-ArXive dataset using GraphSAGE teacher model,
18

---

Published in Transactions on Machine Learning Research (10/2024)
Table 10: Routing-by-Mennt MyyMyr with annealing.
Dataset
Eval
A 0.0
0.05
=
A 0.5
0.9
ind
71.31+0.20
71.31+0.20
71.27=0.24
71.26+0.27
71.2740.42
OGB-ArXive
fran
72.48+t0.13
72.48+0.13
72.46:0.22
72.44+0.20
72.34++0.25
observe how for larger values, there is minor performance drop, but generally RbM is relatively insensilive to the annealing schedule. We assume that large values of A potentially lead to the algorithm becoming stuck at poorer expert embeddings. Empirically, we observe that, even with A = O, the expert embeddings converge sufficiently such that all representations are consistently routed to exactly the same experts in successive epochs.
but vary the anncaling parameter A. From Table 10
19


Reviews:
[{"review_id": "QLqSlcTIlB", "reviewer": "Reviewer_SUMT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S: It seems clear that knowledge distillation for GNNs is a topic of significant interest to the community.\n\nS: The paper is easy to follow.\n\nS: The evaluation experiments seem to have been conducted comprehensively.\n\nW: The explanation of the novelty is somewhat unclear (see requested changes).\n\nW: Convergence of the learning process is not clarified.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a knowledge distillation for graph neural networks (GNNs) for graph node label prediction. The authors claim that usual MLP does not have good performance even when the network is sufficiently large. Instead, the proposed student model consists of a mixture of expert model for which the authors propose an adaptive weighting strategy called routing by memory. The loss function is designed by combining knowledge distillation loss and embedding loss terms. The knowledge distillation loss has three terms such as cross-entropy with true label, KL divergence with teacher label, and knowledge-aware reliable distillation (KRD) terms (Wu et al., 2023). The embedding loss also has three terms such as the vector quantization (VQ) loss, self-similarity loss, and balance loss (Shazeer et al., 2016). The performance is verified through by using several benchmark datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could you clarify which parts of the proposed method are specifically tailored to graphs? Currently, based on my understanding, it seems that only the generation of node features and position features is graph-specific. The other main components, such as the loss definition and MoE, don't appear to have specific aspects related to graphs. It might be easier for readers to grasp the concept if it were clearer what exactly makes this method 'Graph Knowledge Distillation.'\n\n- When I first read the paper, I mistakenly thought that routing by memory (RbM) was an original idea introduced in this paper. However, the concept comes from Zhang et al. (2021a). I think this should be made clearer, especially since the second point in the list at the end of the introduction is misleading. Further, the descriptions in related work is not sufficiently clear about this issue.\n\n- I think the authors should elaborate on the following point further. Just by looking at Equation (10), it's not clear that the sampling focuses on the important points.\n> we sample nodes from its neighbourhood, N(v), updating the sampling distribution after every training step based on how well the student model is predicting each node\n\n- The entire optimization process is guaranteed to converge? The proposed method is not a standard gradient descent, whose convergence is widely studied, because the moving average type update (4) is included in the procedure. Clarifying it would be informative for readers.\n\n- Why increasing the number of parameters of MLP does not consistently lead to better performance?\n\n- What is the necessity of distillation in graph node classification? If the input is a single graph and the task is to predict the node labels, all node predictions have already been assigned by the teacher model once its training is complete. In this case, what is the significance of preparing a separate, lightweight student model? In the inductive setting, the authors mention using 20\\% of the labels as test labels—are you assuming a scenario where a much larger amount of test labels is available in practice? Or are you considering a situation where the same model will be applied to a different graph? Alternatively, is the purpose to improve accuracy? I believe clarifying these points would make the usefulness of the approach more apparent.\n\nMinor: \n\n- In (2), superscript Q^MOE is used, while Q_MOE is used before that.\n\n- After (3), 'v'.\n\n- Before (10), ``... node. and include ...''\n\n- Fig 2 and 3 are not referred from the main text.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DlEk7I6tfZ", "reviewer": "Reviewer_x9kc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Adapting a mixture-of-experts (MoE) model for knowledge distillation (KD) of GNNs is intuitive and could lead to improved performance over the MLP approach.\n\n- The authors propse a modified top-K sparse MoE architecture, called RbM, for the KD task, enforcing each expert to focus on a spatiatial regime in the input embedding space based on the cosine similarity. The RbM model encourages expert specialization, leading to more consistent performance across multiple datasets.\n\n- Empirical evaluation on several real-life datasets demonstrate that RbM can achieve improved accuracy on larger datasets compared to other popular methods as well as the MoE.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an efficient approach to knowledge disillation of Graph Neural Networks (GNNs) using a specially-designed student model, Routing-by-Memory (RbM), which is a form of Mixture-of-Experts (MoE). The authors provide experimental evidence that simply increasing the size of a student MLP does not consistently lead to better performance. To address this issue, the proposed RbM model encourages expert specialization, leading to more consistent performance across multiple datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The proposed MoE, RbM, represent each expert based on a centroid embedding which is estimated with the moving average. This does not seem to be principled and the convergence property of the learning algorithm is not addressed.\n\n- The motivation for the spatial design of the RbM is not well-justified. For example, the RbM can be viewed as a special instance of the sparse MoE while the benefits for the RbM simplification is not discussed.\n\n- The technical contribution of the paper seems incremental. This could be strengthened by taking into account other spatial space beyond the Euclidean.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Pn69K7LmOC", "reviewer": "Reviewer_qpbM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Better Performance: The Routing-by-Memory (RbM) model demonstrates superior performance over traditional MLPs and baseline models across multiple datasets.\n\n**Weaknesses**\n1. Lack of Novelty: The approach seems a direct application of Mixture-of-Experts (MoE), limiting its novelty.\n2. Unclear Motivation: The specific motivation for using MoE in graph knowledge distillation is not well explained.\n3. Model Complexity: The added complexity of the RbM model requires more effort in tuning hyperparameters.\n4. Pretraining Overhead: Requires a pretraining phase for expert embeddings, adding computational overhead.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel knowledge distillation approach from Graph Neural Networks (GNNs) to Multi-Layer Perceptrons (MLPs) to address latency issues in GNNs. The introduced model, Routing-by-Memory (RbM), is a Mixture-of-Experts (MoE) architecture that enhances expert specialization. By encouraging each expert to focus on specific regions in the hidden representation space, the RbM model achieves consistent performance across multiple datasets. Experimental results demonstrate that RbM outperforms traditional MLPs and other baseline models, providing a scalable solution for node classification tasks in large-scale graph-based applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "as listed in weakness", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: My decision is based on unanimous opinion of the reviewers, especially the accept rating on reviewer Reviewer SUMT, as i found this review of the highest quality of the three.
Decision: Accept (poster)