Sparse Decomposition of Graph Neural Networks
Authors: Yaochen Hu
Mai Zeng
Ge Zhang
Pavel Rumiantsev
Liheng Ma
Yingxue Zhang
Mark Coates
Venue: Accepted by TMLR ()

Abstract: Graph Neural Networks (GNN) exhibit superior performance in graph representation learning, but their inference cost can be high, due to an aggregation operation that can require a memory fetch for a very large number of nodes.
This inference cost is the major obstacle to deploying GNN models with \emph{online prediction} to reflect the potentially dynamic node features.
To address this, we propose an approach to reduce the number of nodes that are included during aggregation.
We achieve this through a sparse decomposition, learning to approximate node representations using a weighted sum of linearly transformed features of a carefully selected subset of nodes within the extended neighbourhood.
The approach achieves linear complexity with respect to the average node degree and the number of layers in the graph neural network. 
We introduce an algorithm to compute the optimal parameters for the sparse decomposition, ensuring an accurate approximation of the original GNN model, and present effective strategies to reduce the training time and improve the learning process.
We demonstrate via extensive experiments that our method outperforms other baselines designed for inference speedup, achieving significant accuracy gains with comparable inference times for both node classification and spatio-temporal forecasting tasks.

Published in Transactions on Machine Learning Research (03/2025)
Sparse Decomposition of Graph Neural Networks
Yaochen Hu
yuuchen.hu@@oowwee.ccom
Huowei Nooh's Ark Lab, Montreal, Canada
Mai Zeng
mai.seng@@maiilmmgil..cm
McGill University Mila & ILLS, Montreal, Canada
Ge Zhang
ge. ge.zhangl @louawei. com
Huowei Noah's Ark Lab, Toronto, Conada
Pavel Rumiantsev
McGill University & Mila 8 ILLS, Montreal, Canada
Liheng Ma
McGill University & Mila 8 ILLS, Montreal, Canada
Yingxue Zhang
giagzue..hann@Bhaauiv.oo
Huowei Nooh's 's Ark Lab, Toronto, Canada
Mark Coates
mark. mark.coutess@mmgill.cc
McGill University & Mila 8 ILLS, Montreal, Canada
Reviewed on OpenReview: https: // owwnreview.wee net/ wwr.:e/eee//oonnu id= mdWP1 I
Abstract
Graph Neural Networks (GNN) exhibit superior performance in graph representation learn- ing, but their inference cost can be high due an aggregation operation that can require memory fetch for very large number of nodes. This inference cost is the major obstacle to deploying GNN models with online prediction to reflect the potentially dynamic node features. To address this, we propose an approach to reduce the number of nodes that are included during aggregation. We achieve this through a sparse decomposition, learning to approximate node representations using a weighted sum linearly transformed features of a carcfully selocted subset of nodes within the extended neighbourhood. The approach achieves linear complexity with respect to the average node degree and the nunber of layers in the graph neural network. We introduce algorithm compute the optimal parame- ters for the sparse decomposition, ensuring accurate approximation of the original GNN model, and present effective strategies to reduce the training time and improve the learning process. We demonstrate via extensive experiments that our method outperforms other baselines designed for inference speedup, achieving significant accuracy gains with compa- rable inference times for both node classification and peatio-tempora forecasting tasks.
1 Introduction
Graph neural networks (GNN) have demonstrated impressive performance for graph representation learning (Hamilton et al.] 2017 Veličković et al.] 2018 Qu et al.) 2019 Rampášek et al. 2022). Although there are numerous designs for GNN models, the essential idea is to represent each node based on its features and its neighbourhood (Wu et al.) 2020 Zhou et al. 2020 The procedure aggregating features from neighbour nodes is empirically and theoretically elfective (Xu et al., 2019 in representing the graph structures and
Miila Quebec AT Institute and ILI.S International Taboratory on Learning Systems.

---

Published in Transactions on Machine Learning Research (03/2025)
blending the features of the nodes. However, deploying GNN models to process large graphs is challenging since collecting information from the neighbour nodes and computing the aggregation is extremely time- consuming (Zhang et al. 2021 Tian et al. 2023 Wu et al. 2023 Liu et al. 2024).
In this work, we tackle the efficient inference problem for GNN models in the online prediction setting [Crnkshhaw 2019). Specifically, we need to compute the representations of a few arbitrary nodes. The main advantage is that the prediction can reflect potential dynamic feature of the input. The computational complexity is dominated by the number of receptive nodes, which rapidly increases the number of layers in the model grows, for most message- message-passing" passing- based and graph-tran taapph-tranffommrr based GNNs (Zeng et al. 2020 Min et al. 2022).
Our goal is to reduce the inference time to linear complexity with respect to the number of layers and the average node degree. Recently, several studies have attempted to address this problem by combining the performance of GNN and the efficiency of MLPs (Zhang et al. 2021 Hu al. 2021 Tian et al. 2023 Wang
et al. 2023 Wu et al. 2023 Liu et al. 2024 Tian et. al.] 2024 W Winter al. 2024; Wu et al. 2024). Knowledge
distillation (Hinton et al. 2015 and feature/labed smoothing are used to construct effective MLP models
to eliminate the cumbersome neighbour collection and aggregation procedure. Although efficient, these methods have a fundamental limitation: the features gathered at each node are assumed to contain sufficient information to predict the node label accurately. However, to achieve their full potential, especially when features can change at inference time, GNN models should take into account the features from neighbourhood nodes and the graph structure (Battaglia et al. 2018 Pei et al. 2020). Therefore, we ask the question: given any groph neural network model that relies on both the graph structure and the features of the neighbourhood, can we infer the representation of node in linear time?
Present work. We propose sparse decomposition for graph neural networks (SDGNN), approximation to any target GNN models that can infer node representations efficiently and effectively. The SDGNN consists of a feature transformation function and sparse weight vectors for nodes in the graph. The representation of each node is then a weighted sum of the transformed features from a small set of receptive nodes. The sparsity of the weight vectors guarantees low inference complexity. The learnable feature transformation function and the sparse weight vectors grant the SDGNN flexibility to approximate wide range of targeted GNN models. To find the optimal paramcters in SDGNN, we formulate the approximation task an optimization problem and propose scalable and efficient solution that ilerates between the learning of the transformation function and the optimization of the sparse weight vectors. We verify the approximation power of SDGNN and the scalability of our algorithm on seven node classification datasets and demonstrate how SDGNN can be effectively applied under the online prediction setting with two apatio-temporal forecasting datasets. SDGNN consistently outperforms recent state-of-the-art models designed for GNN inference speedup.
2 Preliminaries and Problem Definition
Consider a graph g {V,E) with |V| = n nodes and [8] = m edges. Denote by X FnxD the feature matrix, with a dimension of D, in which the 7th row X,, denotes the feature vector of node i. We address the task of graph representation learning, which involves learning an embedding 9(2, X(G) E Rd for each node 2, where d denotes the embedding dimension of a GNN. Such representations are fed to additional MLPs f(-) for application in downstream tasks, e.g., node clasifiiattion/regeesse link prediction/reee prariioon/errrsiion, and graph cussiffcatiio//egrrsssit
2.1 Graph Representation via GNN
We summarize the framework from (Hamiloon al. 2017) as an example to demonstrate the basic concepts of GNNs. We stress that our approach can be applicd to any GNN formulation that produces fixod-dimension node embeddings. Let h. be the embedding vector of node at layer 1. Denote by A2x)) the set of neighbour
The features of sample may vary over time, e.g., dynamical features from sensors in the recommendation systems (Chu Park 2009
Dawson et
2016 dynamic features
2

---

Published in Transactions on Machine Learning Research (03/2025)
Original Graph
Target GNN
N D
Targer Model Prediction
GNN Embedding
MLP
Oplimization
N
Node Features
Transforaatinn Function
Transformdd Features
Sparse Weights
Approx. GNN Embedding
MLP
SDGNN Prediction
Figure The pipeline overview for SDGNN framework (bottom pipcline). To compute GNN embedding efficiently, we use a transformation function to adapt node features and introduce sparse vectors associated with each node to gather information from critical neighbours. The parameters in the transformation function and the sparse vectors are determined by optimization to approximate the target GNN embeddings.
nodes of 2. h4 is iteratively evaluated as
h AGGREGATE, h. = (P hh'(e)))
where AGGREGATE(() is any permalalion invariant operation that aggregates a set of vectors, and (PDATE any transformation function, possibly incorporating learnable parameters and non-linear transformations. h2 is usually initialized the node features of 2, i.c., X...
g(z, x() := h h
(1)
where L is a predefined hyperparameter indicating the number of layers. An additional decoder module f(-) can be applied over g(2, (()) to map the GNN representation to the target label
2.2 Online Prediction for Node Representations from GNN
Online predicton is a model serving paradigm that computes the prediction of a few arbitrary samples upon request (Crankshaw 2019). It has stringent requirements for low latency and high throughput. The advantage of online prediction is that it reflects dynamic features. In the case of node-level online prediction for GNNs, we need to compute the embeddings in { of a few nodes efficiently upon request.
The complexity of computing node representation in (1) is proportional to the number of receptive nodes. Specifically, for the nentially with the number of GNN layers (Zeng et al.] (020)_ For a graph transformer, one of the primary motivations to enlarge the number of receptive nodes even they are not connected to the target node, so the number of receptive nodes is usually larger than the message-passing-haee based models (Min et al. 2022). Therefore, deploying an online prediction system for GNN models challenging due to the potentially very large number of receptive nodes. For any GNN model, we aim to find an approximate computation 0(2, X(9), that the inference complezity is linear with respect to the number of GNN layers L and the average node degree d, i.e., (dL.). We require that g( 2, X(g) should be close to g(z, X(G) for Vz E V.
brotngg-pasinng-baae models, the number of receptive nodes potentially grows expo-
Remarks. We aim to invent a fundamental solution for the challenging case where prediction could benefit from potentially dynamic node features, especially for those from a few critical neighbours. For example, the recent purchasing behaviour of onc's closest friend might probably influence her online shopping preference; the traffic flow of an intersection in the near future should be correlated with the status of its neighbouring intersections, etc. Therefore, we want to develop technique to efficiently summarize approximate the
3

---

Published in Transactions on Machine Learning Research (03/2025)
Table 1: Comparison of the related techniques. Inference complexity depicts the asymptotic complexity measured by the number of receptive nodes. Memory overhead the overall memory required. Neighbour feature- aware indicates whether the inference model can reflect neighbour features. Dynamic Features rep- resents whether the inference model can be efficiently applied with dynamic features. L: number of the target GNN layers; L': number of shallower GNN layers where L' < L; [V] node number; d: average node degree; s: sampling budget; D: node feature dimension.
Inference
Memory overhcad
Neighbour feature-aware
Dynamic features
Related Techniques
complexity
Neighbour Sampling
O(sh)
O(VId+ + (VID)
Embedding Compression
O(d
0(MÍ+1VVL
GNN Knowledge Distillation
O
MLP Knowledge Distillation
((1)
O(VID)
X
Heuristic Decoupling
0(1)
O(DID)
X
SDGNN (ours)
O(L))
O(VIaL + (ID)
dynamic and potentially critical neighbour features to fully convey the power of GNNs. Without dynamic node features, we always have an easy workaround method to cache all the predictions instead of cumbersome computation upon request; without the critical dependency on neighbour features, one could train a decent model without GNN to get an optimized performance.
3 Related Work
To the best of our knowledge, we are the first work to study the efficient inference of GNN models under the challenging but rewarding online prediction setting. Some simple adaptions of cxisting works can partially solve this problem but cannot lackle the intrinsic challenges.
Neighbourhood Sampling methods process subsets edges for each node, selected according to statistical approaches. [Hamilton et al. (2017) propose recursively sampling a fixed number of neighbours. However, this approach still suffers from an exponentially growing aggregation complexity. The layer- wise and graph-wise sampling techniques (Chen et al. 2018 Zou et al. 2019: Chiang et al. 2019 Zeng et al. 2020) enhance node- wise sampling by using the common neighbours of nodes in the same mini-batch, but they are optimized for training, and they are not suitable for the online prediction setting.
Embedding Compression methods reduce inference time by compressing embeddings with lower accuracy. Zhao et al. (2020) use neural architecture search over the precision of parameters and adopts attention and pooling to speed up inference. Zhou et al. (2021) compress the embedding dimension by learning a linear projection. Ding et al. (2021) employ vector quantization to approximate the embeddings of subsets of nodes during training. Tan et al. (2020) hash the final representations. These works do not reduce the complexity of neighbour fetching and aggregation.
GNN Knowledge Distillation techniques employ GNN models as teachers to train students of smaller GNN models for inference. Gao et al. (2022) distill the original GNN to a GNN with fewer layers. Yan et al. 2020 learn one GNN layer with sell-attention to approximate each 2 GNN layers, which halves the total number of GCN layers in the student model. These methods trade off some performance for better efficiency. However, the student is still a GNN model which suffers from the large number of receptive nodes.
MLP Knowledge Distillation. Recently, series of works dramatically reduced the inference complexity by training MLPP-based models solcly on the (augmented) features of the target node to approximate GNN
representations ions (Yang el al. 2024 Zhang et al. 2021 Hu et al. 2021 Tian el al. 2023 Wang el. al. 2023
Wu et al., 2023 Liu et al.) 2024 Tian et al. 2024). Moreover, Winter et al. [2024]; Wu et al. (2024) directly learn MLP models with label smoothing regularized by the graph structure instead of learning from a trained GNN model. However, these models cannot consider the features of neighbour nodes during inference and are prone to have limited expressive power compared to general GNN models.
4

---

Published in Transactions on Machine Learning Research (03/2025)
Heuristic Decoupling. Bojchevski et al. (2020) use personalized PageRank to construct shallow GNN
weights instead of performing layer-wise message passing. Duan et al. (2022); Chen et al. (2021:;
p Wu et al. (2019); Chen et al. (2019); Nt & Maehara (2019); Rossi et al. (0000); He et al. (2020) generate augmented,
informative node features using techniques such as message passing or PageRank. Winter et al. (2024) pre-
compute propagated labels from the training set and add them to the node features. During inference, the
models rely solely on these cached augmented node features. These hand-crafted rules to generate informative node features are prone to be sub-optimal. Besides, under the online prediction setting, heuristic decoupling methods have to periodically update the augmented features to support dynamic features, which have great computational overhead.
Position of Our Work: Compared to existing works, our proposed SDGNN offers linear complexity in terms of inference time but can still process the most relevant neighbour features during inference. Table [[] depicts a detailed illustration of our work's position with asymptotic analysis on memory overhead and inference complexity.
4 Sparse Decomposition of Graph Neural Networks
We aim to develop flexible approach to approximate the node representation from wide range of GNN models but maintain low inference complerity. To this end, we introduce feature transformation function ol: W) : IR + Rd that maps each row of X E RD|IXD to dimension of the target GNN representation d, where W represents the learnable parameters. For simplicity, we extend the & notation to the matrix input X to <(;; W) E RRV where Φ(: W) is applied to each row of X. For each node 2, we define a sparse vector Og E R(v and model the representation of node a as a linear combination over the transformed node features via 0.. We thus define the sparse decomposition of graph neural network for node 2 as
x =
(2)
Intuitively, the sparsity of 0₂ and the node-wise dependency on o(: W) ensure that the computation of 0(z, X(g) depends on a limited set of node features. Hence, the inference complexity can be controlled to be (dL). The learnable O2 and o(: W) give SDGNN sufficient fiexibility to approximate a wide range of GNN models.
Matrix form. Let (0,, Oz,..., 014) be the columns of the matrix O Denote the node representation matrix as Z glvixd where the 2th row Qg == g(2, X(g). Correspondingly, we define Ω θ OTo(X; W).
4.1 Relation to Existing Methods
We compare SDGNN the types that have linear complexity summarized in Table [ If we assign an identity matrix to θ the SDGNN degrades to a model solely based on the self-node features. This covers most of the methods under the umbrella of MLP knowledge distillation. For models that augment the node features, like NOSMOG Tian et al. (2023), SDGNN can also trivially adopt such features. Morcover, if we fix θ as the personalized PageRank matrix or power of the normalized adjacent matrix, SDGNN is equivalent to the heuristic decoupling approach. Therefore, SDGNN has at least the expressive power of the models from MLP knowledge distillation and heuristic decoupling.
5 SDGNN Computation
We formulate the task of identifying the optimal O and d(: W) as an optimization problem:
minimize C(O, w) = I ( w) 21 A 111..1 + > / @
(3)
where \1 2 0 is a hyperparameter that controls the sparsity of θ via the column-wise L1 regularization term ||O|| 1.1, and X₂ 2 the hyperparameter for L2 regularization of W. In common models such as a lulti-layer perceptron (MLP), the L2 regularization implicitly upper bounds the row-wise norm of o(X; W) for given X. This prevents the degenerate case of an extremely sparse O with small elements.
5

---

Published in Transactions on Machine Learning Research (03/2025)
5.1 Optimization
Jointly learning @ and W is challenging due to the sparsity constraint on O We oplimize them iteratively, fixing one while updating the other, termed Phase è and Phase ø.
Phase θ For each node 2 E V, we update 02 with the solution of the following optimization problem:
1 1 W) - -22-11 A
argmin
(4)
2
s.t. 0 020.
(5)
The constraints in (5) are not necessary for the general case, but we empirically find that it makes the optimization procedure more robust. We adopt Least Angle Regression (LARS) (Efron et al.) 2004) to solve this Lasso problem, where the maximum number of receptive nodes can be controlled explicitly by setting up the maximum iteration of LARS.
Phase d. We update W with the solution of the following imiizizttion problem:
argmin W
OT
S21 + 22/
(6)
2
and solve it with the gradient descent (GD) algorithm. For efficient training, we take the W from its last iteration as a warm start and only update it for a few steps instead reaching a converged solution.
We iteratively update @ and W until the change of the loss in 33 is smaller than some predefined threshold.
Convergence. Let ., W, be the parameters after the tth iteration with the corresponding loss [(O,, W.). From the definition of loss in 3 we have L(O,, Wt) 2 0 for Vt. From the alternative optimization procedure with a proper setting of learning rate for Phase W, for VI, we also have
L(O,, Wt) Z L(0,+1, W.) 2 L(O,-I, W++1).
Therefore, L(O,, WT) guaranteed to converge as t grows.
Per iteration complexity. Since L is non-convex, providing the asymptotic convergence rate is challenging. We analyze the per-iteration complexity to get insight into the empirical computation overhead. For Phase o, we first need to compute Φ(X; W), which has a complexity of O(Dd+ L'd²) for each node, where L' is the number of layers in the MLP á and assuming the hidden dimension always d. The complexity of computing the solution for each node is O(VI' + (V]=d) (Efron et al. 2004 Then the complexity for Phase O is For Phase W, assuming we need gradient descent iterations, the complexity is ((VVV(D )). Overall, the per-iteration complexity is O is ((D | [V]2d), and and the asymptotic complexity for [V] is O(1))t. The O())(4) factor the bottleneck for efficient training for large graphs.
5.2 Scaling to Large Graphs
The naive optimizatiin algorithm does not scale to large graphs due to the bottleneck at Phase O which has asymptotic per-iteration complexity of O(D21)) We propose two main techniques to tackle the scalability issue: stochastic mini-batch training and narrowing the candidate nodes.
Stochastic mini-batch Training. We perform stochastic mini-batch updates. Specifically, we randomly select B V nodes at each iteration and only update the 0_ where E B. As for W, we select the same subset I of nodes and the corresponding rows of O OTo(X: W) S2 to include in the loss term for each mini-batch, i... solve:
Σ (0 W) Sza 11 Az W
argmin W
Efron al. (2004 mentioned that the complexity O(d3) with saturated least-squares fit when [V] > d. We adopted the Implementation from the scikit-learn library, and empirically, it's always O(12 +V)dd). From the practical point of view, count the complexity always 0 22)

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm SDGNN Computation
procedure SDGNN(22 9.
2: Prepare the candidate sets Cz for Va E V according to Narrowing the Candidate Nodes.
3: for t= 1 to T' do
> Main training loop.
4: Get the current mini-batch the node set B, C V.
5:
for each in B, do Update 0₂ with Phase O
end for Update W with Phase 6. end for
10: for in V do
b Fix W and finalize the O.
11: Update 02 with Phase O.
12: end for
13: return W and 0g for V2
14: end procedure
Narrowing the candidate nodes. The computation required to solve rapidly grows the size of V increases. Specifically, the computational overhead comes from the inference of Φ(X; W) and the computation time of the LARS solver. To reduce the required computation, we define much smaller candidate node set C₂ for each node and only consider the candidate nodes to determine 0.. We solve the following problem instead of (4.
argmin
fz 112 11 AT (TTo(: W)
(8)
s.t. 0, 2 vi C., 0, - Vi $ C..
9 defines the reduced candidate set for solving 0.. We only need to infer o(X..: W) for node that appears in the candidate set, and the complexity of the LARS solver only depends on the size of the candidate set instead of V/-
For each node, we use the knowledge of the graph structure to heuristically determine the moderate-sized candidate set that includes the most relevant nodes. Specifically, we include all Kı-hop neighbour nodes. From those nodes, we recursively sample a fixed number s of neighbours for an extra K₂ hops (similar to the mechanism of graphSAGE (Hamilton et al.j 2017)) and combine visited nodes as the candidate set. As for practical guidelines for selecting K1, K₂ and s, we can refer to the receptive nodes of the target GNN models and set the hyper-parameters that the selected candidate nodes roughly cover them. After that, grid scarch could be conducted around that set of paper-pprameters to get better performance.
Per iteration complexity. The size of C₂ is roughly KKa), where d is the average node degree and is the sampling budget, and |C,| < | for very large graphs. Then in Phase θ, the complexity for each node is 03 + Id). With similar analysis in Sec. 5.1 the overall complexity will be O( (k(Dd + I/d") + |C_ + (d)), and the asymptolic complexity is O(BI]C_,15) We can see that the asymptotic per-iteration complexity is independent of the size of the graph [V].
Considering the design strategies outlined above, the practical algorithm is presented in Algorithm 1 add an extra loop at Line 10 to align all 0 with the latest version of W.
We
6 Experiments
6.1 Node Classification Tasks
We validate SDGNN's approximation power and inference efficiency through node classification tasks in the transductive setting.

---

Published in Transactions on Machine Learning Research (03/2025)
6.1.1 Task
We consider the node classification task under the transductive setting. Given a graph & - {V,8} with [ = nodes and [8] m edges, the feature matrix of nodes X E R and the labels from a subset. of nodes Verain C V, the task to predict the labels for the remaining nodes.
6.1.2 Datasets
Following Zhang et al. (2021) and Tian et al. (2023), we conduct experiments on five widely used benchmark datasets from Shehur al. (2018), namely, Cora, Citeseer, Pubmed, Computer and Photo. We also examine performance two largo-scale datasets, Arxiv and Products, from the OGB benchmarks (Hu et al. 2020
6.1.3 Baselines and Target GNN Models
We select GLNN (Zhang et al. 2021), NOSMOG (Tian et al. 2023). CoHOp (Winter al. 2024), SGC (Wul et al. 2019), and PPRGo (Bojehevski et al. 2020] the baselines. Specifically, GLNN and NOSMOG distill the GNN into an MLP with and without augmenting with position encodings. CoHOp is recent method that trains an MLP to replace the GNN, using label propagation to incorporate graph information. SGC and PPRGo adopt the heuristic of selecting a critical subset of nodes and assigning aggregation weights. To make a fair comparison, we match the number of receptive nodes to SDGNN by truncating, where for cach node 2, we keep the top k2 most significant weights, hi being the corresponding number of receptive nodes in SDGNN at node 2. We also include degraded version of SDGNN where for each 0z, we replace the learned non-zero weights with normalized equal weights (SDGNN-Equal)
Regarding the targeted GNN models, we include GraphSAGE (Hamilton et al. 2017 with mean aggregator. This is the base model for all experiments reported in Zhang et al. (2021) and Tian et al. (2023). To fully evaluate the capacity of each efficient inference solution, we also add one of the most effective GNN models for each dataset. In this selection process, we exclude the models that achieve impressive performance via processing steps unrelated to graph learning, e.g.: label re-use (Wang Leskovec 2021) and feature augmentation. Given that no single GNN model achieves the best performance across all datasots, we select
distinct target models that excel on each dataset. Specifically, we choose Geom-GCN (Pei al. 2020) for
Cora, Citeseer and Pubmed. We adopt Exphormer (Shirzad et al. 2023) for Computer and Photo. We
select DRGAT (Zhang et al. 2023 for Arxiv and RevGNN-112 (denoted by RevGNN) (Li et al.
2021 for
Products.
6.1.4 Evaluation
For the 5 small datasets (Cora, Citeseer, Pubmed, Computer and Photo), we randomly split the nodes with 6:2:2 ratio into training, validation and testing sets. Experiments are conducted using 10 random seeds, as in Pei et al. (2020). We report the mean and standard deviation. For Arxiv and Products, we follow the fixed predefined data splits specified in Hu et al. (020)) run the experiments 10 times, and report the mean and standard deviation.
6.1.5 SDGNN Integration
We start with a trained GNN model, e.g., GraphSAGE (Hamilton et al.] 2017), DeeperGCN (Li et al. 2021), or graph transformers Yunn et al.] 2019 Rampášek et al. 2022; Ma et al. 2023 Shirzad et al, 2023). We adopt the intermodiate representation constructed by the architecture immodiately before the final prediction as the (arget node representation g(2,X1)) X(9), and learn the SDGNN O(=,X|9) Using the label and the representation O(2, ((G) for the nodes within Verains train decoder function f(((2, to map the node representation to the predicted labels. We employ an MLP for the decoder model. During inference, upon a request for prediction for a node z, we refer to O₂ to retrieve the receptive node features and compute the prediction with f(ğ(2, Xlg)) over the receptive nodes. The overall pipeline of integrating the SDGNN is depicted in Figure !
Remark. Our method requires the predicted representation from the GNN for all nodes to get optimized 0z Although the approximation with static node features has limited practical value, we include these
8

---

Published in Transactions on Machine Learning Research (03/2025)
Table 2: Mean Fl micro score and std. dev. for node classification tasks, best in bold and second best underlined. marks statistically significant results under Wilcoxon signed-rank test (Wilcoxon 1992) at the 5% significance level.
Dala/ /Trget Model
Target Model
GLNN
NOSMOG
CoHOpp
PPRGo
SGC
SDGNN-Equal SDGNN(ours)
Cora/SAGE
85.43=1.22
86.86+1..
85.82+1.39
45.49= 0.89 16.41+1.99
87.23=1.400
14.21+1.53
85.53=1.32
Cora/ CCO
86.52-10998
85.78.1..20
85.6811.01
85.49.1 0.80
85.4011.37
85.9911.38
80.79.1..11
86.6410.85
Citeseor/ Stcsecr/SAGE /SAGE
12.40.1..79
76.89-1.53
76.411
1.39
73.22 1 1.21 77.1911.16
76.29.1..37
73.13.11.95
73.62-1..04
Citeseer Geom-GGNN
79.91=0.94
79.34+1.40
79.88+0.99
73.22 + 1.21 77.37+00.65
76.83++0.22
76.63+1.11
80.27+1.111
Pubmed/ /SAGE
87.17-0.56
89.21 0.66
87.98+00.62
84.81+
0.56 89.40+0.45
86.77+00.2
85.42+0.71
87.10+0.554
Pubmed Geom-GCN 89.75+0.44
90.88+0.55
90.31+
84.81+ 0.56 89.35+0.38
86.24++0.76
85.81+0.46
89.77+00477
Computer/SAAG:
89.06-00.53
89. 89.29+0.77 08.449+0.72
91.28+0.47 88.98+0.43
90.52+0.59
89.85+0.556
90.60++0.52
Computer/ Expphrrmee 94.19=00.56
93.2840..8 02.2440.71
91.28 + U.47 89.45-0.60
90.71=10665
03.44
94.29:10.52*
Photo/SAGE /SAGE
92.90 =0.67
03.87=0.33
94.27 50
96.4810.31
94.93+0.41
08.54110.55
93.01-0.59
93.96=2039
Photo/ Ploto/Exphoruuee Exphormer
0.35
95.58-10.56
95.35 0.552
95.48 1: 0.31 0..35
93.90.10.52
96.35
96.73 1..30
Arxiv/SAGE
00.23:1.233
03.40:1..23
69.7810.
72.7910.00 0.09
66.05
69.95.10.29
69.83.10.24
00.30
Arxiv DRGAT
73.78.10.09
15 72.16 65.92.10.17
72.79 0.00
68. 16.16.10.1
72.09.10.14
73.45.10.10
78.72.10.11
Products SSAG
78.37
0..53
77.33 0.
81.67
0.225
70.15
74.9710.61
78.03
10.61
Products/RevCGr RevGNN
82.79=0.25
64.1970.17
80.91+0.22
01.67=0.225
73.93+0.20
79.76+0.28
82.64+0.22
82.87=0.21
EDGNN-ReV
NOS-Rev
80
GL-
SAGE-L2-ful
GL-Rev-w4
SAGE--33-N20
SDGNN-SAGE
SAGE-L2-N20
SAGE-L3-full
Accuracy
NOS-SAGE
GL--AAE--W8
75
GL-SAGE-W4
SSAGE-L1-fuI
65
GL-Rev
#SAGE-L1-N20
GL-SAGE
60
MLP
100
10
102
Log Scale Inference Time (ms)
Figure 2: Accuracy v.8. mean inference wall-clock time over 10,000 randomly sampled nodes on the Products test set. GL-GLNN, NOS- -OO-MOG Rev-RevGNN, w4, w8: student size enlarged 4, 8 times. L1, L2, and L3 denote GNN layers. N20: a neighbour sampling size of 20.
experiments to compare the approximation power and efficiency of the proposed SDGNN with the common benchmarks. The primary use-case for SDGNN is when there are dynamic node features under the online prediction setting. We demonstrate this via a spatio-temporal forecasting task later sections.
6.1.6 Main Results
We compare the accuracy among GLNN, NOSMOG, CoHOp, PPRGo, SGC and SDGNN for seven datasets with different target GNN models in Table 2 The "Target Model" column indicates the performance of the original GNN models. CoHOp, SGC and PPRGo do not rely on the target GNN embeddings. We duplicate the results of CoHOp within each datasct to compare the performance better. SGC and PPRGo have different performances under different larget GNN models since we truncate the neighbours according to the patterns from the corresponding SDGNN models. Overall, SDGNN can approximate the target for all scenarios well, achieving accuracy close the target model. Other techniques like NOSMOG, CoHOP, etc., though provided with a target GNN model, are not designed to solely approximate the target but also integrate other modules to intervene in the outputs. As result, with weaker target GNN model, SDGNN
9

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Mean Absolute Percentage Error (MAPE, %) for next-step forecasting on the PeMS04 and PeMS08 datasets. The receptive field size of the SDGNN model is set for both datasets. * iidicatess SDGNN is stat, significantly better than the next-best (excluding the target), for a paired Wilcoxon signed-rank test at the 5% significance level.
MAPE
GRU-GCN
GRU
GLNN
NOSMOG
PPRGo
SDGNN
PeMS04
1.17+0.00
2.09#n.6'
1.53+0.31"
2.63=0.18'
+.30
PeMS08
0.89+0.05
1.69#0.06
1.55+a.08
1.68+0.0s"
1.63=0.07'
0.97
can be outperformed by those techniques. In contrast, with SOTA GNN largets, SDGNN achieves the best performance in 6/7 scenarios, including 2/2 scenarios on large-scale clttasess.
Specifically, GLNN and NOSMOG often outperform the SAGE target model, but the performance gap is more pronounced for GLNN larger datasets like Arxiv and Products. For example, on Products/ Products/RevGNY RevGNN the performance gap for GLNN is 18.30 percent. An essential difference between GLNN and NOSMOG is that NOSMOG calculates a positional encoding (derived using DeepWalk) for cach node and stores this as an additional feature. This is highly effective for the Products and Arxiv datasets. CoHOp augments the node features with neighbour labels and trains an MLP with a label smoothing loss on the graph. It outperforms the weak SAGE models for all datasets, but cannot achieve accuracy levels close to those of the SOTA target GNN models. PPRGo and SGC adopt different heuristics to select the neighbours, and their performance varies across different. datasets. Due to the optimized neighbour selection and weight assignment, SDGNN outperforms these heuristic-based methods for all scenarios involving more powerful larget GNN models.
Replacing the learned 0g with normalized equal values decreases the performance. This demonstrates that SDGNN locates important neighbours and assigns appropriate aggregation weights to approximale larget GNN embeddings more effectively. Even with normalized weights, the approximation still outperforms the other baselines for the scenarios with larger graphs and more powerful target GNN models. This highlights that the receptive node selection aspect of SDGNN is highly effective.
Ablation study:
6.1.7 Inference Time
We performed inference for 10, 000 randomly sampled nodes for each dataset to assess the trade-off between inference time and accuracy. Here, we provide results and a discussion of the Products datasel; the results for other datasets are qualitatively similar and are provided in the Appendix. Fig. 2 illustrates the testing accuracy versus the average computation time per node for different models and approximation tcchniques. We observe that SDGNN-FE
(SDGNN based on RevGNN) achieves the best accuracy (82.87%) with in- ference time (0.74 ms), while the MI.P-based methods (MLP, GT.-Rev GI-SAGE, NOS-SAGE, NOS-Rev have the fastest inference time around 0.45 ms. SDGNN is slower due to the additional computation costs of performing feature transformation on multiple nodes and aggregation to incorporate neighbour features.
The SAGE series, such as SAGF-L3-full (a 3-layer SAGE model without sampling during inference) and SAGE-L2-N20 (a 2-layer SAGE model that samples 20 neighbours), exhibit rapidly growing inference times as the number of layers increases (up to 128 ms), making them impractical for real-world applications. We also detail the results for GL-SAGE-w4 (GL-SAGE with hidden dimension 4 times wider) and GL-Rev-w4 (a 4 times wider version of GL-Rev). Although their accuracy improves compared to their base versions, their inference times also increase substantially. We conclude that SDGNN offers superior performance in terms of accuracy, and its inference time is sufficiently close to the fastest MLP models.
6.2
Forecasting
We demonstrate how SDGNN can be applied to tasks with dynamic node features through spatio-temporal forecasting tasks, which aligned with the online prediction setting.
10

---

Published in Transactions on Machine Learning Research (03/2025)
6.2.1 Task
We have a graph 9 - [V,8} with [ = n nodes and [g| = m edges. The nodes are associated with signals in discrete time X1.T = (X1, x.....; xx) TRxXXC and each X, E B denotes C' channel signals for all n nodes at time t. The task is to predict XT+1 R2xC,
6.2.2 Datasets
We consider the traffic datasets PeMS04 and PeMS08 from Guo et al. (2021) and apply the data preprocessing of Gao & Ribeiro (2022). Given a fixed graph 9, the signals on the nodes are partitioned into snapshols.
Each snapshot 8 contains discrete signals X and the corresponding target X to prodict.
6.2.3 Baselines and Target GNN Model
We select GLNN IZhang et al. 2021 NOSMOG as the baselines.
Tian et al. 2023 and PPRGo
(Bojchevski et al. 2020
As for the target GNN model, we adopt the time-then-get graph framework with the GRU-GCN model from Gao & Ribeiro 2002 For each x RxxXxC GRU-GCN uses gated recurrent unit (GRU) to encode the signal each node into the representation H" Raxd = GRU(Xi:r) ), d being the dimension of the representation. Then a graph neural network (GCN) is applied to Ha to obtain Z E R GCN(H"). Finally, an MLP decoder is used at each node to derive the prediction X₃ X++ = MLP(Z") E RaxC.
6.2.4
Evaluation
We split the snapshots into train, validation, and test sets. We report the test set's mean absolute percentage error (MAPE). We run each setting 10 times and report the mean and standard deviation of MAPE.
6.2.5 SDGNN Integration
We apply SDGNN to replace the GNN module. After training the GRU-GCN, have GRU, GCN, and MLP decoder modulcs. For all the snapshots in the training set, we take H as the input to SDGNN and Z" as the output to approximate. During training, aggregate the loss and minimize over all snapshots:
1 Σ TTTTHH OH':: w) 2' + A1 1 + Az W W) 2
minimize
We further train MLP to map from the SDGNN output to the prediction. We also adopt various strategies to improve the robustness (please see the Appendix for details). During inference for node 2, we refer to 02 to fetch the relevant node signals and apply the GRU. Then, we apply the SDGNN followed by the MLP to get the final prediction.
Remark. For scenarios with dynamic node features, we can rely on the GNN embeddings from past snap- shots train the SDGNN. We then apply the SDGNN to infer future snapshots. Due to the optimized and reduced number of receptive nodes, SDGNN can efficiently compute the prediction in the online prediction setting.
6.2.6 Main Results
Table presents MAPE results under various settings. In this experiment, SDGNN's target model is GRU- GCN with 2-layered GNN model. GRU denotes the degraded version of GRU-GCN that removes the GCN component. The GCN module integrates the information from neighbour nodes and reduces the error significantly. With the target GRU-GCN model, GLNN and NOSMOG slightly outperform the GRU, but the position encoding in NOSMOG does not help in this dynamic feature setting. SDGNN is closest lo the target GRU-GCN model and performs best among the efficient models. SDGNN can select the most informative receptive nodes whose features have an impact on the labels. The heuristic-based neighbour selection method PPRGo fails under the current setting, achieving similar or even worse performance than GRU.
11

---

Published in Transactions on Machine Learning Research (03/2025)
MAE(GRU) MAE(GRU-GCN)
0.025
GRU
SDGNN
0.016
GRU-GCN
0.020
GLNN
PPRGo
0.014
0.015
MAPE
(GRU-GON-LI)
NOSMOG
GRU
0.012
GRU-GON-LI
0.010
SOGNN
(GRU-GCN-L2)
0.010
GRU-GCMAL
0.005
SDGNN
0.008
(GRU-GCN-LE L2)
0.000000
2
6
8
10
0.000
0.005
0.010 0.015 0.020 0.025 MAE(GRU) MAE(SDGNN) (b)
Mean Size Receptive Field
(a)
Figure 3: (a) MAPE v.s. mean receptive field size on PeMS08 dataset. L1, L2 denote one and two GCN layers, respectively. GLNN and MOSMOG are learned with GRU-GCN-L2. Targets of SDGNN are indicated in brackets. Error bar shows standard deviation. (b) Scatter plot of MAE reduction at each node (compared to GRU) for GRU-GCN (y-axis) with SDGNN (x-axis). Dashed line is y The improvements are highly correlated, with a slight bias in favour of the more computationally expensive GRU-GCN.
Figure 3a plots MAPE versus mean receptive field size for the PeMSOS dataset. We include the results of GRU-GCN models with and 2 GCN layers. NOSMOG and SDGNN are trained with a GRU-GCN target of 2-layer GCN. SDGNN is trained various settings for 1-layer and 2-layer GCN under different budgets of receptive field sizes. Although efficient, GLNN and NOSMOG have little performance gain over the GRU. SDGNN can effectively reduce the mean receptive field size with a slight accuracy sacrifice to the target GCN model. Comparing the SDGNN trained with different target GCN models, the SDGNN with 2-layer GCN is stronger than it trained with 1-layer GCN given similar receptive field size. Training SDGNN using more powerful target model with an aggressive reduction in receptive field size is more effective.
6.2.7 Embedding Approximation Efficacy
Figure 3b shows a scatter plot of the MAE reduction (relative to the GRU) at each node achieved by the GCN versus that achioved by SDGNN. The scatterplot concentrated close to the = x line, with slight bias in favour of the more computationally demanding GCN. The plot highlights how SDGNN successfully approximates the GCN emboddings and thus derives a similar forccasting accuracy improvement at cach node.
7 Conclusion
In this work, we proposed a systematic approach to efficiently infer node representations from any GNN models while considcring both the graph structures and the features from neighbour nodes. Extensive experiments on seven datasels on the node classification task and two datasets on the spatial-temporal forecasting task demonstrate that SDGNN can effectively approximate the GNN target models with reduced receptive field size. Our method provides new tool for the efficient inference of GNN models under the online prediction setting, which can benefit from both the power of GNN models and the real-time reflection of dynamic node features.
References
Peter W Battaglia, Jossica B Hamrick, Victor Bapst, Alvaro Sanchoz-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Relational inductive biases, deep learning, and graph networks. Xie preprint 1rXiv:1806.01220 2018.
12

---

Published in Transactions on Machine Learning Research (03/2025)
Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózem- berezki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate pagerank. In Proc. ACM SIGKDD Int. Conf. Knowl. Discovery Data Min., 2020.
Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: Fast learning with graph convolutional networks via impor- tance sampling. In Proc. Int. Conf. Learn. Representations, 2018.
Lei Chen, Zhengdao Chen, and Joan Bruna. On graph neural networks versus graph-angmented mlps. In Proc. Int. Conf. Learn. Representations, 2021.
Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? dissoction on graph classification. Inl. Conf. Learn. Representations RLGM Workshop, 2019.
Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsich. Cluster-gen: An efficient algorithm for training deep and large graph convolutional net In Proc. ACM SIGKDD Inl. Conf. Knowl. Discovery Data Min., 2019.
Wei Chu and Seung-Taee Taek Park. Personalized recommendation on dynamic content using predictive bilinear models. In Proc. Int. Conf. World Wide Web., 2009.
Daniel Crankshaw. The Design and Implementation of Low- Low-Layency Prediction Serving Systems. California, Berkeley, 2019.
Univ. of
Scott TM Dawson, Maziar S Hemati, Matthew O Williams, and Clarence W Rowley. Characterizing and correcting for the effect of sensor noise in the dynamic mode decomposition. Exp. Fluids, 57:1-19, 2016.
Mucong Ding, Kezhi Kong, Jingling Li, Chen Zhu, John Dickerson, Furong Huang, and Tom Goldstein. Vq-gun: A universal framework to scale up graph neural networks using vector quantization. In Adv. Newral Inf. Process. Syst., 2021.
Keyu Duan, Zirui Liu, Peihao Wang, Wenqing Zheng, Kaixiong Zhou, Tianlong Chen, Xia Hu, and Zhangyang Wang. A comprehensive study on large-scale graph training: Benchmarking and rethinking. In Adv. Newral Inf. Process. Syst., 2022.
Bradley Efron, Trevor Hastic, Iain Johnstone, and Robert Tibshirani. Least angle regression. Ann. Statist., 32(2):407-44) 449, 2004.
Jianfei Gao and Bruno Ribeiro. On the equivalence between temporal and static equivariant graph repre- sentations. In Proc. Int. Conf. Mach. Learn., 2022.
Xinyi Gao, Wentao Zhang, Yingxia Shao, Quoe Viet Hung Nguyen, Bin Cui, and Hongzhi Yin. Efficient graph neural network inference at large scale. arXiv preprint vrXiv:2211.04495 2022.
Shengnan Guo, Youfang Lin, Huaiyu Wan, Xiucheng Li, and Gao Cong. Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting. IEEE Trans. on Knowl. and Data Eng., 34(11):5415- 5428, 2021.
Will Hamilton, Zhitao Ying, and Jure Leskovoc. Inductive representation learning on large graphs. In Adv. Newral Inf. Process. Syst., 2017.
Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgen: Simplifying and powering graph convolution network for recommendation, In Proc. ACM SIGIR Conf. Res. Develop. Tnf. Retrieval, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. Stat, 1050:9. 2015.
Weihua Hu, Matthias Fcy, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Adv. Neural Inf. Process. Syst., 2020.
13

---

Published in Transactions on Machine Learning Research (03/2025)
Yang Hu, Haoxuan You, Zhecan Wang, Zhicheng Wang. Erjin Zhou, and Yue Gao. Graph-mlp: Node classification without message passing in graph. arXiv preprint orXiv:206.0440 2021.
Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In Proc. Int. Conf. Mach. Learn., 2021.
Kang Liu, Zhenhua Huang, Chang-Dong Wang, Beibei Gao, and Yunwen Chen. Fine-ge tine-grained learning behavior-oriented knowledge distillation for graph ncural networks. IEEE Trans. Neural Netw. Learn Syst. 2024.
Liheng Ma, Chen Lin, Derek Lim, Adriana Romero-Soriano, K. Dokania, Mark Coates, Philip H.S. Torr, and Ser-Nam Lim. Graph Inductive Biases in Transformers without Message Passing. In Proc. Inl. Conf. Mach. Learn., 2023.
Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong. Transformer for graphs: An overview from architecture per- spective. arXiv preprinl Xix:2200 204.220500045 2022.
Hoang Nt and Takanori Maehara. Revisiting graph neural networks: All we have low-pass filters. arxiv preprint or Xiu: 1905. 09550, 2019.
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gon: Geometric graph convolutional networks. In Proc. Int. Conf. Learn. Representations, 2020.
Meng Qu, Yoshua Bengio, and Jian Tang. Gmnn: Graph markov neural networks. In Proc. Int. Conf. Mach.
Learn., 2019.
Ladislav Rampášek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. In Proc. Ade. Newral Inf. Process. Syst., 2022.
Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004 2020.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network cvaluation. In Relat. Represent. Learn. Workshop NeurIPS, 2018.
Hamed Shirzad, Ameya Velingker, Balaji Venkatachalam, Danica J Sutherland, and Ali Kemal Sinop. Ex- phormer: Sparse transformers for graphs. In Proc. Int. Conf. Mach. Learn., 2023.
Qiaoyu Tan, Ninghao Liu, Xing Zhao, Hongxia Yang, Jingren Zhou, and Xia Hu. Learning to hash with graph neural networks for recommender systems. In Proc. Int. Web Conf., 2020.
Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh Chawla. Learning mlps on graphs: A unified view of effectiveness, robustness, and efficiency. In Proc. Int. Conf. Learn. Representations, 2023.
Yingjie Tian, Shaokai Xu, and Muyang Li. Decoupled graph knowledge distillation: A general logits- method for learning mips on graphs. Neural Netw., 179:106567, 2024.
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proc. Int. Conf. Learn. Representations, 2018.
Hongwei Wang and Jure Leskovec. Unifying graph convolutional neural networks and label propagation. In Proc. Int. Conf. Learn. Representations, 2021.
Xiyuan Wang and Muhan Zhang. How powerful are spectral graph neural networks. In Proc. Int. Conf. Mach. Learn., 2022.
Yiwei Wang, Bryan Hooi, Yozen Liu, and Neil Shah. Graph explicit neural networks: Explicitly encoding graphs for efficient and accurate inference. In Proc. Inl. Conf. on Web Search and Data Mining, 2023.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in Statistics: Methodology and distribulion, pp. 196-202. Springer, 1992.
Daniel Winter, Niv Cohen, and Yedid Hoshen. Classifying nodes in graphs without gnns. arXio preprint 002.02240.05994 2024.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In Proc. Int. Conf. Leorn. Representations, 2019.
Lirong Wu, Haitao Lin, Yufei Huang, and Stan 2 Li. Quantifying the knowledge in gnns for reliable distillation into mlps. In Proc. Int. Conf. Mach. Learn., 2023.
Lirong Wu, Haitao Lin, Zhangyang Gao, Guojiang Zhao, and Stan Z Li. A teacher-free graph knowledge distillation framework with dual self-distillation. IEEE Trans. on Knowl, and Data Eng., 36(9):4375-4433 4385, 2024.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst., 32(1):4-24) 2020.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In Proc. Int. Conf. Learn. Representations, 2019.
Bencheng Yan, Chaokun Wang, Gaoyang Guo, and Yunkai Tou. Tinygun: Learning efficient graph neural networks. In Proc. ACM SIGKDD Inf. Conf. Knowl. Discovery Data Min., 2020.
Ling Yang, Ye Tian, Minkai Xu, Zhongyi Liu, Shenda Hong, Wei Qu, Wentao Zhang, CUI Bin, Muhan Zhang, and Jure Leskovec. Vggraph: Rethinking graph representation space for bridging gnns and mlps. In Proc. Int. Conf. Learn. Representations, 2024.
Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J Kim. Graph transformer networks. In Adv. Neural Inf. Process. Syst., 2019.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method. In Proc. Int. Conf. Learn. Representations, 2020.
Lei Zhang, Xiaodong Yan, Jianshan He, Ruopeng Li, and Wei Chu. Drgen: Dynamic evolving initial residual for deep graph convolutional networks. In Proc. AAAI Conf. Artif. Intell., 2023.
Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. In Proc. Int. Conf. Learn. Representations, 2021.
Yiren Zhao, Duo Wang, Daniel Bates, Robert Mullins, Mateja Jamnik, and Pictro Lio. Learned low precision graph neural networks. arXiv preprint arXiv:2009. 09232, 2020.
Hongkuan Zhou, Ajitesh Srivastava, Hanqing Zeng, Rajgopal Kannan, and Viktor Prasanna. Accelerat- ing large scale real-time gun inference using channel pruning. In Proc. Conf. Very Large Data Bases Endowment, 2021.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al Open, 1:57 81, 2020.
Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. Layer-dependent importance sampling for training deep and large graph convolutional networks. In Adv. Neural Inf. Process. Syst., 2019.
15

---

Published in Transactions on Machine Learning Research (03/2025)
A Extended Experiment Settings on Node Classification Tasks
In this section, we provide additional details about our implementation to support the reproduction of our reported results. We ran all the tests on the computer equipped with Intel(Rt Xeon(R) Gold 6140 CPU @ 2.30GHz CPU and NVIDIA Tesla V100 GPU.
A.1 Dataset
Table shows the detailed specifications of the datasets. The Arxiv and Products datasets are significantly larger than the other datasets.
Table 4: Summary of the Node Classification Datascts.
Data
Nodes
Edges
Attributes
Classes
Cora
2,708
5,278
1,433
7
Pubmed
19,717
44,324
500
3
Citeseer
3,327
4,552
3,703
6
Computers
13,752
215,861
767
10
Photo 7,650
119,081
745
8
Arxiv
169,343
1,166, 243
128
40
Products
2,449,029
61,859,140
100
47
A.2 Syper-paramenn Settings for Baselines
Table shows the hyper-parameters for the base GNN models. Specifically, "Ir" presents the learning rate; "hidden dimension" shows the hidden dimension of the intermediate layers; "layer" depicts the number of layers in the model; "weight decay" is the factor in the L2 regularizer for all learnable parameters. We adopt the same
of SAGE models (Zang et al.) 2021). We adopt the parere-rrseeeerss from those corresponding papers for geomGCN, Exphormer, DRGAT and RevGNN. Table ( shows the hyper- parameters for the MLP student models for the GLNN, and Table 7 shows the MLP student models for the NOSMOG. We adopt. the same syper-parameters from (Zhang et al. 2021 and Tian et al. 2023 for the corresponding datasets. Table g shows the hyper- syper-paramters for CoHOp.
for the
Table 5: Summary of the hyper-parameter for base GNN models.
Dataset/ Model
Ir
hidden dimension
layer weight decay
Cora/SAGE
0.01
64
0.0005
Cora/ Cora/geemGCN
0.05
5.00F-06
Siteseer/SAGE.
0.01
64
0.0005
Citeseer /CN
0.05
5,00E-06
Pubmed/SACKI
0.01
64
0.0005
Pubmed /CeoeeCC
0.05
16
5.00E-06
Computer/SAGr
0.01
128
0.0005 4
Computer/Exphonee
0.001
80
0.001
Photo/SAGE
0.01
128
0.0005 4
Photo/ Exphormer
0.001
64
0,001
Arxiv Arxiv/SAGE
0.01
256
0,0005
Arxiv/DRGAT
0.002
256
3
Products/SAGE
0.001
256
0.0005
Products/ReyGNN
0.003
160
A.2.1 Hyper-parameter Setting for SDGNN
Table g shows the important hyper-paramcters of SDGNN that yield the main results in Table 2 K and K2 are the parameparseetess defined in the subsection, "Narrowing the Candidate Nodes". The K₂ fanout parameter specifies the neighbour sampling size. These three values jointly influence the sizes of the
16

---

Published in Transactions on Machine Learning Research (03/2025)
Table 6: Summary the hyper-parameters in the student MLP models for GLNN.
Dataset/ Dataset/Model Model
hidden dimension
layer
weight decay
Cora/SAGE
0.005
128
0.0001
Cora/ Oora/geomGNN
0.005
128
2
0.0001
Citeseer/SAAe /SAGE
0.01
128
0.0001
Citeseer/eeomGCC, geomGCN
0.01
128
2
0,0001
Pubmed/SAGH
0.005
128
Pubmed /Ce
0.005
128
2
0
Computer/SAGE
0.001
128
0.002
0.001
128
0.002
Photo/SAGE
0.005
128
0.002
Photo/Exphornin
0.005
128
0.002
Arxiv/SAGE
0.01
256
Arxiv/DRGAT
0.01
256
Products/SAGE
0.01
256
0.0005
Products/RevGNN
0.01
256
0.0005
Table 7: Summary of the hyper-parameters in the student MLP models for NOSMOG.
Dataset/ Mataset/Model Model
Ir
hidden dimension
layer
weight decay
Cora/SAGE
0.01
128
0.0001
Cora/geomGCN
0.01
128
0.0001
Sfteseer/SAGrs
0.01
128
2
1.00E-05
Citeseer geomGCN
0.01
128
1.00E-05
Pubmed/SSAee /SAGE
0.003
128
2
0
Pubmed baauued/ggoooo /ComGGN
0.003
128
Computer/SAGE
0.001
128
2
0.002
Computer
Computer/Exphormer /Exphormer
0.001
128
0.002
Photo/SAGE
0.001
128
2
0.001
Photo/Exphormer
0.001
128
0.001
Arxiv/SAGE
0.01
256
Arxi>/DGGAT
0.01
256
3
Products/SAGE
0.003
256
Products/Revee RevGNN
0.003
256
candidate sets, and hence the training time of SDGNN and the approximation quality, We conduct grid search over K1 [1,2, and K₂ [1. 2]. starting from the combination that results in smaller candidate sets to larger ones. We select the combination such that the associated training time of SDGNN is feasible, and the validation loss in (3) does not reduce by more than 0.01 compared with the next smaller setting.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Table 8: Summary of the hyper-parameters for CoHOp. If hidden dimension not stated, linear model
is used.
Dataset
hidden dimension
# epochs
patience
gamma
weight decay
alpha
Cora
200
50
0.5
0.025
0.1
0.0005
0.4
Citeseer
200
50
0.6
0.005
0.1
0.0005
0.5
Pubmed
200
50
0.2
0.01
0.1
0.0001
0.6
Computer
200
50
0.2
0.01
0.01
0.0
0.05
Photo
200
50
0.4
0.05
0.01
0.0005
0.1
Arxiv
512
200
100
0.5
0.0
0.1
0.001
0.6
Products
1024
0.5
0.5
0.001
0.0
0.1
Table 9: Summary of the
for Training SGDNN.
Data/ Data/Target Target model
Ki
K2
K₂ fanout
Ir
Cora/ ISA/E
2
2
10, 10
0.01
le-05
Cora/ ggooGGNN
10,
0.001
Ie-07
Citeseer/SAGE
2
2
10,
10
0.001
1e-05
Citeseer Cieeseer/geoGGNN geomGCN
10,
0.001
1e-06
Pubmed/SACH
1
10
0.001
1e-05
Pubmed/ Tubmed/geomGGCN geomGCN
10
0.001
1e-06
Computer/SAGr.
2
10. 10
0.01
1e-05
Computeer/xxhhoren
2
10, 10
0.005
le-05
Photo/SACC
2
10, 10
0.01
le-05
Photo/exphormne
2
10, 10
0.001
1e-05
Arxiv/SAGE
15,
0.001
1e-05
ArXiv/DRGAT
2
15, 15
0.0001
1e-07
Products/SAGE
10,
0.001
le-05
Products/RevGNN
2
10, 10
0.0005
1e-07
109
K
NR (log scale)
K
K
10-1
2000
4000
6000
Time(s)
Figure Convergence curve under various candidate sets for Pubmed dataset with SAGE model.
18

---

Published in Transactions on Machine Learning Research (03/2025)
B Additional Experimental Results for Node Classification Tasks
B.1 The Impact of Candidate Selection
We train the SDGNN with different scttings for selecting the candidate sets and show the normalized regret (NR) v.8. the wall clock time. The normalized regrel. is defined as
1 NR =
1 X; W) 22.12
2|V1
12
SEV
which characterize the average relative error with respect to the target GNN embeddings. Fig. 4 shows the training curve for the Pubmed dataset with the SAGE model under four different settings of candidate sets, i.., including 1-hop, 2-hop, 3-hop or 4-hop neighbours, respectively. As we can see, the smaller candidate sets, c.g. 1 = I or L = 2, leads to a faster convergence but converges to a sub-optimal point, while a large candidate set converges much slower with a better result. With k = 4, the normalized regret even bounces back and forth. Besides, if we adopt all the nodes as candidates, the training will take an extremely long time (not shown in the figure). We conclude that with proper setting of the candidate set, the learning of SDGNN can be greatly boosted while losing little performance in terms of NR.
B.2 The Receptive Nodes from SDGNN
Table 10 presents the average receptive field sizes for SDGNN and the original graph. also reports their ratios. To make efficient. estimation, compute the mean statistics from 1000 randomly sampled nodes. For the original graph, only count up to 2-hop neighbours for Products, due to the very rapid expansion of the neighbourhood. We count to 3-hop neighbours for the other scenarios.
SDGNN can significantly reduce the receptive field size against the original graph. Interestingly, the ratio between the receptive field size of SDGNN and that from the original graph generally decreases as the hop size increases. This indicates that, proportionally, SDGNN favours receptive nodes that are closer to the center node. However, in each case, the majority of the nodes that SDGNN selects are immediate neighbours.
Table 10: The average number of receptive nodes for each node ("Overall"); the average number at a distance of "1-hop", "2-hops" and "3-hops" for (i) SDGNN, (ii) original graph (fii) the ratio between SDGNN and original graph.
Overall SDGNN 3-hop Neigh. Ratio
I-hop SDGNN Original Ratio
2-hop
3-hop
Data/ Model
SDGNN Original Ratio
SDGNN Original Ratio
cora OSEE
36.1
128.1
.28
2.9
4.9
.59
10.1
31.7
.32
12.4
92.9
.13
cora/geomGCN
38.4
128.1
.30
2.2
4.9
.45
10.2
31.7
.32
13.6
92.9
.15
citesea/SAAEE
17.8
43.5
.11
2.7
3.9
.69
4.7
12.1
.39
6
29.8
20
Steseer/geommCNN
14.2
43.5
.33
1.8
3.9
.46
3.5
12.1
.99
29.8
.17
pubmed/SAGE
25.5
394.6
.06
2.6
5.7
.46
54.8
.24
336.6
.03
pubmed/geomGCNN
394.6
.04
1.1
5.7
.19
.15
7.2
336.6
.02
4-computer/AAGE
24.9
7493
.003
2.6
37.3
.07
1906.8
.008
7.4
5727.6
o-computee/cxxphoreec
46.9
7493
.006
5.3
37.3
.14
28.4
1906.8
.01
13.4
5727.6
.002
o-photo//SAAE
28.1
2519.4
.01
4.1
20
.14
17.4
759.3
.02
6.1
1680.8
.003
e-photo/exphormer
45.8
2519.4
.018
5.9
29
.2
28
759.3
.04
11.4
1689.8
.007
ogbn-arxix/SAAG.
30.3
18470.6
.0016
2
14
.14
12.9
3713
.003
16
15443.1 1 .001
ogbn-ardév/DRG..T
36.1
18470.6
.0019
2.3
14
.16
14.2
3713
.004
20
15443.1
.001
ogbn-prodduct/SAAGE
24
3851.9
.006
2.1
51.9
.04
11.4
3919.2
.003
10.7
ggbn-products/ReeCNNN--I
23.6
3851.9
.006
1.8
51.9
.03
10.3
3919.2
.003
11.6
Figure and Figure 5 show the empirical CDFs (calculated from 1000 randomly sampled center nodes) of the mumber of receptive nodes and the theta values (node weights after 4 row normalization) for SDGNN, SGC and PPRGo. For SGC and PPRGO, we include all the weights without truncating. For SGC, we include the 3-hop neighbours for all datasets except for Products, which includes only the 2-hop neighbours. SGDNN consistently identifies smaller number of receptive nodes and puts larger weights on few focused nodes.
19

---

Published in Transactions on Machine Learning Research (03/2025)
This is the main reason that SDGNN can significantly reduce the receptive field size. Although PPRGo often selects a similar number of ivceptive nodes as SDGNN, it distributes the weights more evenly among them.
B.3 Inference Time
To fairly compare the inference times, always constrain the computation single CPU and compute the prediction of 1 node at time. This simulates the online prediction setting where the request comes randomly. We show the inference times over 10000 randomly sampled testing nodes for all the datasets Table II Specifically, the MLP column presents the mean inference time for a 3-layered MLP model, and SAGE-N20 presents the mean inference time of the SAGE models with neighbour sampling size of 20. The SDGNN-SAGE and SDGNN-SOTA columns report the inference times of SDGNN trained with the SAGE model and the corresponding SOTA model in Table 2 We can that SDGNN consistently has much smaller inference time compared to the SAGE counterpart, and it is on the same scale as the MLP models (inference time is 1-4x larger).
Table 11: The average inference time per node over 10000 samples. (ms)
Dataset
MLP
SAGE-N20
SDGNN-SAGE
SDGNN-SOTA
cora
0.2
11.5
0.5
0.8
citeseer
0.3
11
0.6
0.9
pubmed
0.2
10.8
0.3
0.5
a-computer
0.2
44.4
0.6
0.4
a-photo
0.2
7.4
0.6
0.4
ogbn-arxiv
0.2
15.6
0.2
0.4
ogbn-products
0.4
13.6
0.7
0.7
1.00
1.00
1.00
0.75
0.75
0.75
0.75
CDF
CDF
CDF
0.50
0.50
0.50
0.50
SDGNN
SDGNN
0.25
0.25
SGC
0.25
0.25
SGC
PPRGo
PPRGo
0.00
0.00
0.00
0.00
10a
10
0.00
0.05
0.10
0.15
102
10
0.00
0.05
0.10
0.15
# Receptive Nodes
Theta Values
# Receptive Nodes
Theta Values
(a) Arxiv/SAGE
(b)
Arxiv/SAGE /SAGE
(c)
Arxiv DDRGA
(d)
Arxiv/DRGA
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
CDF
CDF
CDF
0.50
0.50
0.50
0.50
SDGNN
SDGNN
0.25
0.25
SGC
0.25
0.25
SGC
PPRCo
PPRGo
0.00
0.00
0.00
0.00
10a
105
0.00
0.05
0.10
0.15
10a
105
0.00
0.05
0.10
0.15
Receptive Nodes
Theta Values
# Receptive Nodes
Theta Values
(e) Products/SAGE
(f)
Products/SAGE
(g)
Products/RevG.N
(h) Products/RevGNN
Figure 5: The empirical CDFs of the number of receptive nodes and the empirical CDFs of the row normalized e for SDGNN, SGC and PPRGo for ArXiv (SAGE, DRGAT) and Ogbn-Products (SAGE, RevGNN-112).
20

---

Published in Transactions on Machine Learning Research (03/2025)
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
CDF
CDF
CDF
0.50
0.50
0.50
0.50
SDGNN
SDGNN
0.25
0.25
SGC
0.25
0.25
SGC
PPRGo
PPRGo
0.00
0.00
0.00
0.00
16
10a
0.00
0.05
0.i0
0.15
10
10
0.00
0.05
0.10
0.15
Receptive Nodes
Theta Values
# Receptive Nodes
Theta Values
(a)
Cora/S (SAG
(b) Cora/SAGE
(c)
Cora/ Gora/goomGCCN
(d) Cora/ /ee/ggemmGNN
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
CDF
0.50
CDF
0.50
0.50
CDF
0.50
SDGNN
SDGNN
0.25
0.25
SGC
0.25
0.25
SGC
PPRCo
PPRGo
0.00
0.00
0.00
0.00
10
10"
102
0.00
0.05
0.10
0.15
10
'001
"102"
0.00
0.05
0.10
0.15
# Receptive Nodes
Theta Values
Receptive Nodes
Theta Values
(e)
Citeseer /Sase
(f)
Citeseer /Saseerr/AABBB
(g) Citeseer/geom geomGCN
(h)
Citeseer/geomGGe
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
CDF
CDF
CDF
0.50
0.50
0.50
0.50
SDGNN
SDGNN
0.25
0.25
SGC
0.25
0.25
SGC
PPRCo
PPRGo
0.00
0.00
0.00
0.00
10a
104
10a
0.0
0.1
0.2
10a
10a
0.0
0.1
0.2
# Receptive Nodes
Theta Values
# Receptive Nodes
Theta Values
(i) Tubmed/SAGes
(j)
Pubmed/SAGK
(k) Pubmed/ /geeeGCG
(I) Tubmed/geomGoo
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
CDF
0.50
CDF
0.50
0.50
0.50
SDGNN
SDGNN
0.25
0.25
SGC
0.25
0.25
SGC
PPRGo
PPRGo
0.00
0.00
0.00
0.00
10
10a
0.00
0.05
0.10
0.15
10a
10a
0.00
0.05
0.10
0.15
# Receptive Nodes
Theta Values
# Receptive Nodes
Theta Values
(m))
Computer/SACo
(n)
Computer/SAGE
(0) Computer/exppa /exphommer
(p) Computer/exphuot
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
CDF
0.50
CDF
0.50
0.50
CDF
0.50
SDGNN
SDGNN
0.25
0.25
SGC
0.25
0.25
SGC
PPRGo
PPRGo
0.00
0.00
0.00
0.00
10
103
0.00
0.05
0.10
0.15
10
10a
0.00
0.05
0.10
0.15
# Receptive Nodes
Theta Values
# Receptive Nodes
Thela Values
(q) Photo/SAGE
(r) Photo/SAGE
exxoor/exphomeer
(L)
PPhoto/eppoormee
Figure 6: The empirical CDFss of the number of receptive nodes and the empirical CDFss of the row normalized @ for SDGNN, SGC, PPRGo for Cora, Citeseer and Pubmed (SAGE, gcomGCN), and Computer and Photo (SAGE, exphormer).
21

---

Published in Transactions on Machine Learning Research (03/2025)
C Implementation Details for GRU-GCN and SDGNN and Analysis in Spatio-temporal Tasks
The implementation of the target model for the spatio-temporal task follows the methodology outlined by iGao & Ribeiro 2022). We strictly adhered to the implementation details of the GRU-GCN model described in that work. The overall architecture of GRU-GCN is illustrated at the top of Figure 7 Tt consists RNN, GCN, and final MLP. The hidden dimensions for the RNN, GCN, and MLP components are identical with softplus the activation function. The target GRU-GCN model was trained for 1000 epochs, with early stopping applied using a patience of 30 epochs. The hyperparameters used for the target model are summarized in Table 12
of three main components:
Figure 7 (bottom) illustrates what we employed for integrating SDGNN the spatio-temporal setting. We directly adopted the trained RNN from the GRU-GCN model, followed by the SDGNN trained using the embeddings from the GCN. This was then followed by fine-tuned MLP, originally from the GRU-GCN model. Ultimately, SDGNN consists of two components: target model, we trained SDGNN for 4000 epochs, employing iterative training and optimization process for the MLP and sparse weights. The sparse weights were optimized every 40 epochs. Following the training of SDGNN, we fine-tuned the MLP for the low-level task over 30 epochs. The best-performing SDGNN and low-level lask MLP were selected based on their MAPE performance on the validation dataset. In a spatio-temporal context, the task is essentially a regression problem. Unlike classification tasks, and due to our method's focus on approximating the final graph embedding, we observed that the last projection layer or the final MLP is sensitive to differences between the approximated embeddings and the target GNN embeddings. To enhance the robustness of the final MLP layer in the GRU-GCN model, we introduced noise between the GCN and MLP during training. This noise was carefully calibrated as a hyperparameter to avoid degrading the overall performance of the GRU-GCN while still providing the necessary robustness to the final low-level task MLP. Specifically, we added Gaussian random noise (0, 0.05) for the PeMS08 dataset and 0.001 for the PcMS04 datasot. The detailed hyperparameters for SDGNN are presented Table 3
MLP and sparse weights. After obtaining the
Table 12: Scts of hyperparamcters for GRU-GCN in spatio-temporal sctting experiment.
Dataset D Moodel
Ir
hidden dim.
GCN layer
MLP layer
weight decay
patience
PeMS04
1e-3
16
2
2
1e-5
30
PeMS08
le-3
2
le-5
GRU-GON
RNN
GCW
MUP
D
Prediction
Directly Adopted
Subetituted
Fine-tuned
RNN
SDGNN
MLP
Node Fealures
Figure 7: The simplified architecture diagram of the GRU-GCN model is depicted at the top. Building the GRU-GCN architecture, we substitute the GCN module with SDGNN to enhance inference elficiency. For comprehensive overview of the SDGNN pipeline, please refer to Figure
22

---

Published in Transactions on Machine Learning Research (03/2025)
C.1 Optimization Schedule for the Sparse Weights
To ensure that the weights remain sparse, we implemented specilic optimization schedule tailored to the characteristics of sparse weights. At the beginning of the training process, we set a larger maximum receptive field size to allow for broader conncctions. As training progressos, we gradually reduce the maximum receptive field size by 1 every 100 epochs. In our case, the largest receptive field size at the beginning of the training was set 46. Throughout 4000 training epochs, we gradually reduced the receptive field size. Specifically, every 100 epochs, the maximum receptive field size was decreased by 1, resulting in a final receptive field size of 6.
Table 13: Sets of hyperparameters for SDGNN in spatio-temporal setting experiment. Underlined values are those selected by grid search.
Parameters
hidden dim.
mmn. layers
noise level
Ir
PeMS04
(8,16,32)
{1, 43
{0.001, 0.0025, 0.005, 0.01) (0.01, 0.05, 0.1, 0.15) (0.001 0.0025, 0.005, 0.01)
{0.01, 0.03, 0.1, 0.15)
PeMS08
(8,16,32)
{1, 2. 4}
0.020
SDGNN checkpoint
MAPE performance
0.018
Polynomial Fit (degree
0.016
0.014
0.012
0.010
0.0005
0.0010
0.0015
0.0020
0.0025
0.0030
MSE between target emb. and approx. emb.
Figure 8: Prediction performance in MAPE relative to the MSE between SDGNN approximated embeddings and target model embeddings in the spatio-temporal setting
C.2 Analysis of Final Performance Relative to the Distance Between SDGNN Approximated Embeddings and Target Model Embeddings
Since SDGNN aims to approximate the target model embeddings, we hypothesize that the discrepancy between the SDGNN embeddings and the target model embeddings will significantly impact the overall model performance. Figure g illustraaes the relationship between the mean squared error (MSE) of the embeddings and the prediction performance. The x-axis represents the MSE between the target embeddings and the 4DGNN-approximated embeddings, while the y-axis indicates the prediction performance measured by the MAPE metric. Each scatter point represents checkpoint of SDGNN. We utilize checkpoints of SDGNN during training at every 100 epochs and evaluate the performance the test sets to generate the plot. The figure reveals that higher approximation errors correspond to greater discrepancies in prediction performance. This observation suggests that as the MSE between the SDGNN embeddings and the target embeddings decreases, the model's performance is likely to improve.
C.3 Analysis on Performance Degradation with Increasing Model Staleness
In this experiment, we analyze the prediction error between the ground truth and predicted values by calculating the absolute error for cach data point. The testing set divided into 10 equal-sized chunks, preserving the chronological order of the data. Figure 9 presents the mean absolute error with standard deviation for every chuck of the test set for both PeMS04 and PeMS08 datasets. We that the
23

---

Published in Transactions on Machine Learning Research (03/2025)
0.0120
Mean Absolute Error
0.016
Mean Absolute Error
0.0118
0.015
0.0116
0.0114
0.014
0.0112
0.013
0.0110
2
8
10
2
6
10
Time Chunks (Chronological Order)
Time Chunks (Chronological Order)
Figure 9: The mean absolute error per chronological chunk showing potential shifts in prediction accuracy time for PeMS04 (Teftt) and PeMS08 (Right) datasets. Error bars represent. the standard deviation within each chunk.
performance indeed degrades with later time chunks, but the degradation quickly saturated at slightly worse point than in the first chunk.
D Additional Analysis on Expressive Power of SDGNN
The analysis for a general GNN target is challenging. We demonstrate the expressive of SDGNN by showing how SDGNN can approximate any Filter-Most-Expoo -ose--ssst--presivvve (FME) GNNs defined (Wang Zhang 2022 which covers a wide range of spectral GNNs. Let denote the adjacency matrix and D denote the diagonal matrix where Da denotes the degree of node i. The normalized Laplacian matrix L == I - D AD where I is the identity matrix. Let L UAUT denote the eigendecompositionn Let the function ((A)
applics 3 clement-wisely to the diagonal matrix A. We further denotes ((A) = UB(A)UT, Then, the FME
GNN is defined as
9EME X) = ((UUB(A)UTP(X))) a(3(A)
(10)
where E R is the node leatures, at(-) and (p)) are functions like multi-layer perceptrons that operate row-wisely on the input matrix and 3(-) is arbitrary real-valued filter function. Consider the challenging case that X is random variable drawn from certain distribution. We want to show that under mild conditions, there exists an SDGNN equipped with some function f(-), such that the f(O 6 W)) is close to geMe(X) (X), for vX drawn in such a distribution.
We set f(-) to be a(5 W) and o(-) to be (--), respectively. Assuming a(-) is Lipschitz continuous, we only need to show that there exists some θ, such that (X) is close to (()))( ).. A trivial solution without considering sparsity
take T ((A), and f( (Xx W)) can perfectly express grme(X), With sparsity constraint, can treat it solving Lasso problem to decide each column of O Then, there trade-off between the quality and sparsity. Specifically, for the nth column of θ if we set the number of non-zero entries to be k the quality depends on the smallest distance between the nth row of (X) and all the subspaces spanned by k rows of (X). Such quality monotonically increases if we allow larger k. If most rows of 3(A)<p(X) are close to some subspaces spanned with small k; there exists a parameter setting of SDGNN with sparse ② that can approximate 9FME (X).
E Limitations of SDGNN
Although SDGNN is very effective approximating target GNN embeddings, there are a few limitations. We present the naive cues from an engineering perspective and would like explore principled solutions future works.
24

---

Published in Transactions on Machine Learning Research (03/2025)
E.1 Incompatibility with Inductive Setting
SDGNN always requires the GNN embeddings from all nodes the guiding signal to generate the optimal O and Φ. Therefore, SDGNN cannot be applied directly in the inductive setting when new nodes may appear during inference time. From the application perspective, a simple strategy to handle the novel nodes in real-time could be collecting the 0 for all/sampled 1-hop neighbour nodes and summing them up as the proxy for the new nodes. Moreover, for the situation where we are allowed to do some preprocessing for the novel nodes, we could use the inductive GNN to infer the GNN embedding, compute 0 with Phase 0 for that node and cache it. Then, the actual online inference stage will be the same the other nodes that are seen during the training of SDGNN.
E.2 Unadaptability to Distribution Shift
For the online prediction setting, train SDGNN based past snapshots and apply it in future snapshots. We did not consider the potential distribution shift between training and testing. If the time gap between the training and the testing is large, such a shift might degrade the performance.
A typical and practical solution is periodically retraining new models based on recently collected data. This can solve the distribution shift issue for both the target GNN model and SDGNN. If we trade the performance for less compulation overhead, we can periodically and partially update SDGNN for belter training efficiency. For example, we can keep the weight matrix W unchanged and only periodically perform Phase O update according to the latest data.
A fundamental approach could incorporate online learning so that the trainable weights from the target GNN and SDGNN can be adapted as more data become available and the graph topology changes. However, developing such a method is involved due to the challenges of enforcing a sparse solution. We would leave this as a future work.
E.3 Inability to Represent Non-linear Interactions between the Features at Different Nodes
In standard GNN models, the recursive procedure of aggregating neighbour features followed by the feature transformation may potentially model the non-linear feature interaction between nodes. However, SDGNN is formulated as a linear combination transformed node features. Although efficient, it seems not to have the potential to model arbitrary feature interaction between nodes. One immediate fix is that for any target node, can augment the other candidate nodes' features with the target node feature before feeding into o, which appears to integrate all the feature interaction between the neighbour nodes and the target nodes. However, a general principled approach is worth exploring. Besides, we should have a theoretical analysis of the expressiveness of SDGNN and explore under which conditions SDGNN is not as expressive as the general GNN models.
E.4 Training Overhead and Challenge on
Selection
We managed to finish the training of SDGNN in about 20 hours for the graph with millions of nodes like OBGN-products, but training SDGNN on larger graphs is challenging. Regarding the hyper-parameler selection, such a training overhead for each run is a nightmare. The main obstacle the solver for the Lasso problem. Implementing the LARS solver that supports the GPU speedup could be an immediate workaround. As for a principled solution, since we iteratively optimize and é, we don't necessarily require optimized result for Phase 0 at each iteration. Instead, we may propose similar Lasso solver that can rapidly trade-off between the computation complexity and the accuracy of the solution. During the training of SDGNN, we could engage a schedule to gradually switch the Lasso solver from efficiently generating intermediate results to accurately achieving optimal results.
25


Reviews:
[{"review_id": "yBQTddstaA", "reviewer": "Reviewer_Vd3Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-structured and easy to follow, with clear explanations of the proposed method and its components.\n- The experimental results are comprehensive, demonstrating the effectiveness of SDGNN across various datasets and tasks.\n- The approach is innovative in its use of sparse decomposition to maintain model accuracy while significantly reducing inference time.\n\nWeaknesses:\n- How does the method handle the introduction of new nodes in an inductive learning setting, given that it requires pre-computed embeddings for all nodes?\n- Is there a risk of losing important non-linear interactions between node features due to the linear combination used in the sparse decomposition?\n- How does the training time and computational complexity scale with very large graphs?\n- How does the method ensure that the sparse weight vectors θz remain non-negative during the optimization process, as indicated by the constraint in equation (5)?\n- In the context of narrowing the candidate nodes, how does the heuristic approach for selecting the candidate set impact the model's ability to capture important node features?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an approach called SDGNN that reduces inference time by approximating node representations through a weighted sum of features from a carefully selected subset of nodes, achieving linear complexity with respect to the average node degree and the number of layers.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "H5FEHNpbvw", "reviewer": "Reviewer_bvcp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The work addresses an important practical problem of making GNN inference more efficient for online prediction while preserving model quality. The solution is well-motivated and technically sound.\n2. The empirical validation is thorough and convincing, with extensive experiments across different datasets, model architectures, and application settings (node classification and spatio-temporal forecasting).\n3. The method is flexible and can approximate various GNN architectures while maintaining linear complexity. The authors demonstrate this by successfully applying SDGNN to GraphSAGE, Geom-GCN, Exphormer and other architectures.\n4. The authors provide clear comparisons with existing approaches and demonstrate meaningful improvements over state-of-the-art methods.\n\n**Weaknesses**\n\n1. The theoretical analysis is limited. The paper would benefit from formal analysis of SDGNN's expressiveness compared to standard GNNs and theoretical guarantees on approximation quality. I'm particularly interested in why the GNN's performances are improved after the decomposition. Is it possible to add some (heuristic) explanations? Such as visualise these embeddings?\n2. The approach has limited ability to capture non-linear interactions between node features, which may be important for some applications. This fundamental limitation deserves more discussion.\n3. The method cannot handle inductive settings where new nodes appear during inference. While the authors acknowledge this limitation, more discussion of potential solutions would be valuable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces SDGNN, a novel method for efficiently approximating GNN node representations. \nThe key innovation is combining learned feature transformations with sparse weight vectors to determine which transformed node features to aggregate, allowing efficient approximation of GNN embeddings.\n\nUnlike prior MLP-based approximations, SDGNN can process neighbor features during inference while maintaining linear complexity. \nIt provides a principled way to trade off between approximation quality and inference efficiency through sparsity constraints, though with some limitations in inductive settings and non-linear feature interactions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Requested Changes**\n\n1. Include a theoretical/empirical analysis section examining SDGNN's expressiveness compared to standard GNNs. At minimum, identify conditions under which SDGNN can/cannot match/imporve GNN performance.\n\n**Suggested Improvements**\n\n1. Add experiments showing how performance degrades with increasing time gaps between training and testing in the spatio-temporal setting.\n2. Include an analysis of memory requirements alongside the existing computational complexity discussion.\n3. Provide a detailed discussion of strategies for handling distribution shift in online/streaming settings. This is essential since the method is aimed at online prediction scenarios where data distributions may evolve over time.\n4. Add experiments on more diverse types of graph data beyond citation networks and traffic forecasting.\n5. Provide guidelines for selecting key hyperparameters like K1 and K2 based on graph properties.", "broader_impact_concerns": "NO", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NehaY9TKdw", "reviewer": "Reviewer_M5hg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* **Temporal graph benchmarks**:  MLP-based graph GNN approximators are usually not benchmarked on temporal graph datasets, so it is a pleasant comparison to see how models perform in a different setting than the usual graph \n* **General formulation**: SDGNN has the benefit of being more general than what the authors describe as \"heuristic decoupling\" approaches, and it's clear how its weights can be adjusted to subsume PPR/normalized adjacency matrix methods. This introduces new optimization challenges, which adds to the novelty of the paper.\n* **Comprehensive experiments**: Experiments and results are very well-presented, and significance testing is used in the main results. All hyperparameters and optimization schedules are transparently described in detail in the Appendix.\n\n**Weaknesses**\n\n* **Missing baselines/benchmarks**: The authors are missing comparison with VQGraph [1], a new SOTA GNN-MLP distillation technique. I think comparison with this method would be beneficial. Also, Table 2 only shows results for homophilic graph datasets. Performance on heterophilic datasets with heterophily-minded GNN teachers would be helpful for evaluating SDGNN.\n* **Lack of asymptotic analysis**: Especially for training SDGNN, the paper would benefit from an explicit analysis of the time complexity of SDGNN taking into account both the candidate node generation algorithm and the LARS solver. \n* **Lack of ablation/sensitivity studies**: There are several hyperparameters that are important for SDGNN's performance, such as $K_1$, $K_2$, the number of LARS iterations, $\\lambda_1$, and $\\lambda_2$. It would be helpful to know how SDGNN performs differently w.r.t. different settings of these parameters.\n* **Lack of training time cost** SDGNN requires both training a neural network with gradient descent and solving an optimization problem with a LARS solver. It seems SDGNN's training procedure can be quite expensive, requiring training for 4000 epochs alternating between training the MLP and the sparse weights. A discussion on the time and space costs of training SDGNN complete with a study examining how each component affects training/inference time would be helpful. As it stands, it seems SDGNN should be no cheaper to train than models like SAGE with few layers and small neighborhood size < $K_1K_2$.\n\n[1] VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce SDGNN, an MLP-based method for approximating GNN embeddings that is linear w.r.t. average node degree and the number of layers in the GNN. Subsuming prior work with heuristic decoupling and knowledge distillation, SDGNN is novel in its approach by alternating between training an MLP to approximate the teacher GNN embeddings and optimizing sparse weights. This enables fast inference where each node embedding is computed as a linear combination of the other embeddings using those sparse weights. Through thorough experiments on graph and temporal graph node classification benchmarks, SDGNN outperforms existing methods with superior inference efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Can the authors elaborate on a more intuitive explanation for what SDGNN boils down to on an intuitive way? It seems SDGNN is essentially a way of conducting weighted message passing with teacher GNN supervision and sparse edge weights, but a clear intuition behind the method would be helpful given the difficult interpretability of sparse weights.\n* Can the authors elaborate on the importance of each component of SDGNN on downstream performance/inference time? For instance, in Appendix D.4, the authors note that different strategies in optimizing the sparse weights can result in tradeoffs between computation complexity and accuracy. Further, how much do hyperparameters such as max LARS iterations and $K_1$ and $K_2$ affect performance?\n* Can the authors elaborate on what the significance of dynamic node features with the shortcomings of existing GNN/MLP models in fast dynamic node inference? This advantage is mentioned in the beginning sections, but the explicit connection between dynamic node features and the temporal graph benchmark as well as explaining why SDGNN is superior in this regard could use more detail.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents SDGNN, a novel approach for efficient GNN inference through sparse decomposition. All three reviewers recommend acceptance (1 Accept, 2 Leaning Accept), highlighting:

- Major Strengths:

1. Novel and well-motivated approach combining learned feature transformations with sparse weight vectors
2. Strong theoretical foundation with guarantees for approximating Filter-Most-Expressive GNNs
3. Comprehensive empirical validation across diverse datasets
4. Clear practical value with significant efficiency improvements
5. Thorough responses to reviewer concerns with additional analyses and experiments

- Key Improvements Made:

1. Added theoretical analysis of expressiveness
2. Provided memory requirement analysis
3. Included temporal performance degradation studies
4. Added sensitivity analyses for hyperparameters
5. Clarified handling of dynamic node features

- Remaining Considerations:

1. Addition of VQGraph comparison recommended
2. Further evaluation on heterophilic datasets suggested
3. More detailed training cost analysis would be valuable

The authors have been highly responsive to reviewer feedback, providing detailed theoretical analyses, additional experiments, and clarifications. The work makes both theoretical and practical contributions while maintaining strong empirical validation.

- Recommendation: Accept with encouragement to address remaining minor points in final version, particularly:

Expanding heterophilic dataset evaluation
Including comprehensive training cost analysis

The paper represents a significant contribution to efficient GNN inference with both theoretical and practical value.
Decision: Accept (poster)