Proper Orthogonal Decomposition for Scalable Training of Graph Neural Networks
Authors: Abhishek A
Manohar Kaul
Mohit Meena
Mahesh Chandran
Venue: Accepted by TMLR ()

Abstract: As large-scale graphs become ubiquitous in real-world applications, there is growing concern about the memory and time requirement to train a graph neural network (GNN) model for such datasets. Storing the entire adjacency and node embedding matrices in memory is infeasible in such a scenario. Standard sampling-based methods for addressing the memory constraint suffer from the dependence of the number of mini-batches on the graph size. Existing sketch-based methods and graph compression techniques operate at higher sketch ratios, with the graph compression techniques showing poor generalization, implying that different GNNs trained on the same synthetic graph have performance gaps. Sketch-based methods necessitate online learning of sketches, further increasing the complexity. In this paper, we propose a new sketch-based algorithm, {PGNN}, employing the Proper Orthogonal Decomposition (POD) method to craft update rules to train GNNs, improving the memory requirement and training time without the complication of updating the sketches during training. Experiments on standard graph datasets show that {PGNN} can reach much lower sketch ratios without compromising the performance. We demonstrate that the POD projection matrix is provably optimal, minimizing an upper bound on the projection error induced by the linearized GNN (SGC) update rule. Empirical findings validate our approach, demonstrating superior performance at reduced sketch ratios and adaptability across various GNN architectures.

Published in Transactions on Machine Learning Research (01/2026)
Proper Orthogonal Decomposition for Scalable Training of Graph Neural Networks
Abhishek AI, Manohar Kaul, Mohit Meena, Mahesh Chandran
Pujitsu Research of India, Bangalore, India fabhishek.a a, manohar, mohi tkumar. meena, mahesh. chondraanj@fujjttss..cm com
Reviewed on OpenReview: https: //
opwpeeveee....e..eem forum? Cetpmmiid-ae BeL
Abstract
As large-scale graphs become ubiquitous in real-world applications, there is growing concern about the memory and time requirement to train graph neural network (GNN) model for such datasets. Storing the entire adjacency and node embedding matrices in memory is infeasible in such a scenario. Standard sampling- baased methods for addressing the memory constraint suffer from the dependence of the number of mini-batches on the graph size. Existing sreech-baseed methods and graph compression techniques operate at higher sketch ratios, with the graph compression tochniques showing poor generalization, implying that different GNNs trained on the same syuthetic graph have performance gaps. Sketch-based methods necessitate online learning of sketches, further increasing the complexity. In this paper, propose
beaech algorithm, PGNN, employing the Proper Orthog- onal Decomposition (POD) method to craft update rules to train GNNs, improving the memory requirement and training time without the complication of updating the sketches during training. Experiments on standard graph datasets show that PGNN can reach much lower sketch ratios without compromising the performance. We demonstrate that the POD projection matrix is provably optimal, minimizing an upper bound on the projection er- induced by the linearized GNN (SGC) update rule. Empirical findings validate our approach, demonstrating superior performance at reduced sketch ratios and adaptability across various GNN architectures.
1 Introduction
Graph Neural Networks (GNNs) have proven be powerful tools for graph learning across various do- mains, excelling in tasks such as classification (Kipf & Welling) 2017), clustering (Bianchi el. al., 2020). recommendation systems (Wu et al. 2022), and social network analysis [Fan et al. 2019 Their strength lies in their ability to extract meaningful insights from local neighbourhoods within graphs, thus creating effective representations of target nodes. However, the dependence of GNNs on graph topology introduces significant challenges when scaling to larger graphs or deeper models while maintaining computational and memory efficiency. Traditional full-batch training methods necessitate storing the Laplacian matrix of the entire graph, resulting in a memory complexity of O(m + ndL + d L) for an n-node, m-edge graph, where node features have dimension d in an L-layer graph convolutional network (GCN). This linear dependency both and m, combined with the limited memory capacity of GPUs, restricts the scalability of train- ing on large graphs (especially large dense graphs with being of the order of O(n²) in the worst case). To address these memory constraints, research in this domain has broadly proposed two main approaches: sampling-base strategics (Hamilton et al.] 2018 Chen et al. 2018a by Chiang et al., 2019 Zeng et al. 2020) and historical embedding techniques (Fey et al. 2021 Ding et al. 2021). Although these methods improve memory efficiency, the computational complexity still increases linearly with and m.
*Corresponding author.

---

Published in Transactions on Machine Learning Research (01/2026)
In the context of matrix approximation, sketch of an arbitrary matrix A E Rexd is defined as reduced matrix B E RRoxd, where co denotes the sketch dimension, and B providing a good approximation to A even as Co < n r1/Ghasaami et al. 2015). The amount of compression achieved by sketching is best described by the sketch ratio r = Ca/ n. Proper orthogonal decomposition (POD), as described in the article Rathinam & Petzold (2003), also known as the Garhumen-Tooww decomposition or principal component analysis, provides an orthonormal basis representing the given data in an optimal least squares sense. To achieve sublincar training time complexity with respect to n, Ding et al. (Ding et al., 2022) propose a sketch-based algorithm named sketch- GNN, that trains the GNN on top of few compact sketches of both the convolution and node feature matrices. The authors propose an end-to-end training protocol in the sketch space by approximating the non-linear activation function using polynomial tensor-sketch (PTS) theory (Pham & Pagh 2013a).
As observed by the authors, the approximation of the non-linear activation limits the expressiveness, which constrains the depth of GNNs that can be trained due to error accumulation. Working with dense matrices, even in the reducod skotch space, imposes a significant computational burden. Despite showing promise, the sketch-GNN algorithm has following limitations: (i) it requires higher skelch ralio which results in lower compression of the original graph and (ii) it requires frequent updates of the sketches during the training which triggers ro-computation of all the sketches involved. It is worthwhile to mention another tochnique which guarantees efficient training of GNNs using random spanning trees (Bonchi et al. 2024) and leverages the concept of effective resistance to enhance node classification tasks. This method enhances GNN efficiency by creating path graphs from random spanning trees to maintain essential graph features while minimizing complexity for faster training. This approach is currently constrained to only the GCN architecture.
The primary motivation for using POD to address limitation (i) in sketch-GNN was driven by the parallel between dynamical systems and message passing methods in GNN Choi et al. 2023), coupled with the fact that effective number of eigenmodes required for POD in dynamical system decreases on increasing system size, resulting in a much lower sketch ratio. For limitation (ii), we theoretically establish bounds that con- strain the deviations of node representations when using our proposed method. Additionally, we prove the optimality of the POD update rule for a projection subproblem induced by the linearized GNN update rule. This indicates that the best low-rank matrix for the update rule can be predetermined, which completely climinates the need for online lcarning via frequent updatos of sketches. To this end, our paper presents PGNN, novel sketch-based method for GNNs. This method diverges fundamentally from prior approaches that emphasize sketching weights or gradients (see Liu et al. 2022]; Chen et al. (2015); Kasiviswanathan et al.] (0018): : Lin et al. (2019); Spring et al. (2019)). Drawing inspiration from the update rules of lin- carized GNNs (SGC) (Wu et al. 2019), we customize the massage passing process to function within the linear subspace formed by the columns of the augmented input node feature matrix. Experimental results,
as presented in section demonstrate that the sketch ratio necessary for achieving optimal performance decreases as the graph size increases. Despite its theoretical limitations beyond linearized GNN, the PGNN framework efficiently performs node classification in POD-derived linear subspaces, providing insights into GNN operational subspaces (Lee et al. 2023). Our contributions can be summarized as follows:
1. In section we introduce specialized update rules designed to enhance the training efficiency of GNNs by operating within a reduced subspace. Utilizing the POD method, we sketch the input node feature and convolution matrices into their lower-dinensional approximations, thereby streamlining the computational process.
2. In Theorem 1
we establish that the POD projection matrix minimizes upper bound on the projection error for the linearized GNN update rule. Furthermore, in Theorem 2 we present bounds that quantify the deviation of node representations when using the PGNN framework.
3. The versatility of PGNN is evaluated across different GNN architectures, including GCN (Kipf & Welling, 2017). SGC (Wu et al., 2019), GraphSAGE (Hamilton et al.) 2018), and GAT (Veličkovć et al. 2018), with results detailed in section 5 Through exlensive experimenlation, as demonstrated in section 5 we find that PGNN is able to provide comparable performance at much lower sketch- ratio compared to previous methods. For instance, on the Reddit datasot, the tha Sietch-GNN framework achieves its highest accuracy at a sketch ratio of 0.3, while we achieve better and comparable accuracy for various architectures for much lower sketch ratio of 0.05. This
2

---

Published in Transactions on Machine Learning Research (01/2026)
reduction in sketch ratio directly translates to accelerated runtime and reduced memory footprint, underscoring our method's efficiency without compromising accuracy.
2 Related work
The scalability of GNNs has been predominantly addressed through mini-batching strategies, which, despite mitigating memory bottlenecks, often fail to reduce epoch training time. Recent work in graph compression, such as Graph Coarsening (Loukas 2018) and dataset condensation (Zhao et al.] 2021). aims for sublinear training times by condensing the graph, thus reducing node and edge counts (Huang et al. 2021 Jin et al. 2022j. These methods, however, face significant challenges: the preprocessing overheads often exceed O(n): reducing practical benefits, and the efficacy of the trained model varies with the GNN architecture used (Jin et al. 2022; Ding et al.) 2022). Scalable GNN approaches fall into several categories: (A) full-graph training, which is memory and iime-intensive; (B) sampling- based methods like GraphSAGE (Hamilton et al.) 2018 FastGCN (Chen et al.) 2018b), and GraphSAINT (Zeng et al. (020), which employ various sampling strate- gies to reduce computational load; (C) historical-embeddiing methods, such as GNN AutoScaee (Fey et al.
2021 and VQ-GNN (Ding et al., 2021), which store embeddings but incur high memory costs; (D) linearized GNNs (Bojchevski et al., 2020 Wu et al., 2019 Frasca et al. 2020), which offer computational efficiency at the risk of oversimplification; (E) methods using random spanning trees (Bonchi et al. 2024), which reduce computational load by transforming graphs into sparse path graphs; and (F) sketch-based methods like GNN (Ding et al. 2022), which approximate non-linear activations but struggle with error ac- cumulation and high computational demands. Each approach presents tradc-offs in terms of computational complexity and model expressiveness, addressing different constraints in GNN applications.
3 Preliminaries
Basic Notations.
Let G = (V, E) denote graph where [m[m]= (1...., n) is the set of n vertices and E C VxV is the set of m edges. y represents the labels of the nodes. Additionally, the input node feature matrix associated with G is demoted by X (0) E Raxd, , where d is the number of features. Let (0) E R=xco and I denote the augmented feature matriz and its mean vector, respectively. Ce R'x= denotes the convolution matrix of graph G and C(i,j) denotes its (ith) entry. We represent the 1 order element-wise power of C as CoO, Additionally, C(i,)) denotes the 2th row and C(;,)) denotes the jth column. Considering a GNN, x E guxd; denotes the node representations of layer I, where di represents the number of neurons at layer 1. ||-Il denotes the l norm unless stated otherwise. o(-) is the non-linear activation and @(f-g) is the learnable weight matrix at layer I for filter q. R(1), R(2) (((2)...., R(() denote the & count-sketch matrices with dimension RR* where k (a hyper-parameter) denotes the number of sketches and e₁ is the fixed sketch dimension associated with all of them. 33 denotes the upper bound on the number of elements in the set for unsketching. The POD projection matrix which is the matrix of the linear projection expressed in the original coordinate system in R" is given by P = p" E Raxn, We refer to the submatrix P as the factor of P. We represent a matrix comprising of b n-dimensional column vectors y E R [w]xxb- P Rox denotes the learnable matrix incorporated in the PGNN update rule. B2 denotes the number of nodes used for unsketching to update Pr.
3.1 Count Sketch.
Matrix multiplication is crucial in machine learning and scientific computation, with efficient techniques developed in works like (Paszke et al.) 2017: Guennebaud et al.) 2010), and Aaadi et all (016)) Count sketch, a potent dimensionality reduction technique introduced in Charikar et al. (2002) and Weinberger ee al. (2010), projects an n-dimensional vector u into cu-dimensional space using a random hash function h: n + [cc and a binary Rademacher variable s : [x] + {-1,1}. The dimension reduction transformation CS(u); = Emg)=; s(j)u, = R(i, :)u involves count sketch matrix R E R The work in Ding et al. (2022) provide theoretical guarantees on the approximation quality of CountSketch for Graph Neural Networks; we restate the relevant result in Appendix A (Lemma)
3

---

Published in Transactions on Machine Learning Research (01/2026)
3.2 Tensor Sketch.
Tensor sketch, introduced generalization of the count sketch Charikar et al. (2002) is dimensionality reduction technique frequently employed in machine learning for large datasets Pham & Pagh (2013b). Pham and Pag Pham & Pagh (2013a) proposed an efficient way to compute tensor-sketch using FFT and inverse FFT operation given by,
(A) FFT OFFT (A))
p=1
In Ding et al. (022). Tensor-sketch is applicd to approximate the element wise k-th power of matrix product:
(AB) = (A) TS&(B")",
where R BE Raxd
ReRdd and (S;(A) E R (B²( E Rdxc with cj < n.
3.3 Locality Sensitive Hashing.
Locality Sensitive Hashing (LSH) exploits hash functions, denoted as H Rd - [ca], to map closely positioned
vectors into the same bucket with high probability. SimHash, an instance of LSH, uses random matrix
P E Roa/2xd to define hash function ((u) = arg max ([Pu Pul) (Charikar et al. 2002). This mothod
is efficient for large vector batches (Andoni et al. 2015
3.4 Proper Orthogonal Decomposition.
Given the input node feature matrix (0) = [x1, T2, Fal, where 2, E R". Then the best approximating
affine subspace representing these data points and passing through the mean (2 === E xz) is given by
the leading eigenvectors of the centred covariance matrix (see (Rathinam & Petzold 2003) for delailed
explanation)
R R d Σ [r, -0;--)) T)².
The factor p X of projection P is given by the leading eigenvectors of R, where co n. The sketch Z p( (0) - [ axa) E Roxd of input node feature matrix ( represents the sketch of ( in the affine
subspace. Details of the POD technique can be found in (Rathinam & Petzold 2003 Holmes et al. 1996:
Tall et al. 1999
Moore
1981
3.5 Unified Framework of GNNs.
For GNN, message passing between layers can happen differently, like that of spatial convolution (GCN) (Kipf & Welling 2017), self-attention (GAT) Veličković et al. 2018), and Weisfeiler-Lehmann (WL)
alignment, see Xu el al. (2019) According to Balcilar et al. (2021) the general rule for message passing is given by,
((+)) =a Σ c(L.) ((0xxx@UU0))) ((())))))
(1)
where ((l.)) E Raxa is the q-th convolution support at layer I that defines how the node features are propagated to the neighbouring nodes, (0) is the node representations at layer and ((.e)) are the trainable weights. The input node feature matrix is given by (0) E Baxd
As shown in Ding et al. (2021), the gradient involved in the back-propagation rule for GNNs, for the loss function is given by the following:
Σ (cila)) (Vxatnl @ M((+)) (Bll.))
V Vxale=
(2)
4

---

Published in Transactions on Machine Learning Research (01/2026)
where M(++) = o (o- (x((+))). This formulation embodies the essence of the message- message-passing paradigm. Here, o and o denote the derivative and the inverse of the activation function o, respectively. The term
V Vxu-n (o-1 (x(=))) represents the gradients propagated back through the non-linearity. In essence, this rule captures the flow of information and updates dynamics within GNNs during the backward pass.
4 POD sketch based method on GNNs
Problem and Insights. The runtime complexity of the update rules GNNs on a complete graph is O(n²), and the memory complexity involved is O(n + m). The POD sketch-based method for GNNs approximates the GNN's update rule and utilizes sketches of both the convolution matrix and the input node feature matrix for training. Initially, the input node feature matrix (Xx()) and the convolution matrix (C) are of sizes n x d and n n,, respectively. These matrices are then transformed into low-dimensional sketches of size co x and co X co, respectively. The sketch Z of the input node feature matrix X (0) and the convolution matric sketch (Sc) approximating the SGC architecture GNN (PSGC) are given by:
~ = p(x (0) [m]nxdd; Sc =c=pCp².
Figure [ illustrates the overall PGNN framework. Recall that p is the factor of P, representing the singular vectors of the augmented input node feature matrix X (0) after normalization (see Algorithm
4.1 Approximate update rules with PGNN
Our primary goal is to approximate the forward propagation rule of the GNN:
x = (cx")())
We project the node representations layers and 1 ++ onto the subspace spanned by the columns of the factor matrix This yields
z(41)= = =po (c
z + 1x [x/x (00)) U.
The mean of the augmented input feature matrix (0) (See Algorithm 2 denoted by z. We denote the bias induced by this projection as U [pT]eoxd141 For ease of notation let the unsketched node representations at layer be X²) = =p" z()
[T] nxde- Employing an element-wise nonlinearity expressed as a power series
(see Eqqation 4 in Ding et al. (2022) leads to the following result:
Cr (cXWBL)) ov - -U.
z(+1)_
Using Tensor-sketch (Section 3.2 to approximate the power series above leads to the following:
((+)
Su
[TW - U.
represents the learnable coefficients for combining different powers of the representation matrix. Sketch of the convolution matrix S = (C).
TSp (x0DN))) T
1(4)) =
St
1=1
9
Ö FFT (v5*))
((+1)
ck CkuC
FFT-1
(3)
5

---

Published in Transactions on Machine Learning Research (01/2026)
N 0 v (NT) = NT R(a)"" with R(a) E R denoting the count-sketch matrix. The advantage of the PGNN update rule that the objective restricted to get optimal sketches of the matrix P, which will be fixed throughout training. In Sketch-GNN, the update rulc involves online learning of skelches where the LSH and count-sketch hash tables corresponding to each layer are updated during training. The update rule consists of computing the count-sketch of matrix p. Count-sketch of matrix p.
CS (p) = pR(()" = -p(k)
For the unsketching process, the count-sketch matrix R(() E R is transferred to GPU memory. Each column of R(() contains a single nonzero entry-either +1 or -1 located at a random row. Storing the count-sketch matrix in memory is not an overhead because of its inherent sparse nature. To illustrate, for the ogbn-products datasel, a single count-sketch matrix consumes approximately count-sketch ratio of 0.1. The count-sketch ratio's dependence on the approximation's quality is addressed additional storage cost of Occock) is incurred to store the sketches and R(K), The intricacies of how message passing happens for the PGNN framework in various GNN architectures like SGC, GCN, GraphSAGE, and GAT are explained in Appendix B Two challenges must be addressed for the approximate update rule proposed in Equation 3
MB of memory for a
in Lemma 2 However,
4.1.1 Challenges
Challenge (1). The PGNN method limits the updale rule of the GNNs to the column space of p.
Addressing challenge (1) The PGNN update rule projects node representations onto the column space of p. To overcome this limitation, we introduce a Icarnable matrix Pr E Rooxco into the update rule (Equation 3 resulting in the following formulation:
FFT-1 ÔÔ FFT (v(")
)|' U.
Ck pI St
OFFT
(4)
v(P) (NT) Ñ ( pi (() + [x] (0(), Equation 4 serves as the update rule for PGNN, approximating the original GNN update mochanism. The complexitics associated with the PGNN update C We design a loss function L to update the parameter P). This loss function incorporates the GNN loss E, which evaluated on a subset Sr of the training set. The subset Sn consists of B2 nodes, where B2 is a hyperparameter chosen to be significantly smaller than the total number of training samples. The matrix D = p" :Sr;:)p1 (() Aaxd (()) (Sr,:),
rule are explained Appendix
C au 00l111 + 30 ((S))
Term
Term
2(4) denotes the sketched node representations the last layer and the unsketched node representations the last layer L, (Lr)) R(*)" :)):))(())) P: (ZL) + Term is used to control the deviations between the unsketched node representations at the input and the input node feature matrix, whereas Term 2 gives weightage to the loss of the GNN l while updating Pl. The compute needed to update P1 using the loss function L is (f2c4d1 + c.Codz). og and fo are kept as constants. L is updated using gradient-descent on every epoch (See Algorithm [].
Challenge (2). Avoiding (nn) in the loss evaluation. Unsketching of node representations ((L) E xd; at layer L from the sketch dimension co to n and computing the losses for all nodes in node classification involves O(crcon) computations and (nn) memory.
( Mean { R()) ((tt))Tpzzz)) + [ø|xxd.
(6)
Mean refers to the element-wii wise mean over tensors. LSH-based Loss Evaluation for Node Classification: To avoid O(n) complexity loss evaluation, we employ a locality-sensitive hashing approach that selects nodes with poor predictions as described in Section 3.3 of Ding et al. (2022). outlined below:
6

---

Published in Transactions on Machine Learning Research (01/2026)
1. Construct LSH hash tables: We build LSH hash tables H. R - [cx]] (Section 3.3 labeled training nodes across C classes into Cj hash buckets.
index the
2. Formation of subset B: Using the LSH hash tables, we select nodes whose predicted class scores have small inner products with respect to their ground truth (one-hot encoded) labels based on the gradient signals of ((L.) p(h) P z(L).
B = U jml
arg max M.L.)) }
(B.)), = Mean (RWW" (B, (B.)) [T||B|xc
(7)
Evaluate loss on selected subset: We compute the classification loss only on the selected nodes:
LLSH = [B]
Σ
((Ds. Vi)
EB
8 is the loss function (e.g,, cross-entropy), 0r represents the predicted class probabilities.
Complexity advantage: This approach avoids the O(n) complexity of evaluating losses for all nodes by focusing computational resources on nodes that are most likely to contribute significant gradients, i.e., those with poor current predictions.
Simhash projection matrix update. The projection matrix P for the SimHash function H - + [cu]
is updated using gradient descent with the triplet loss function (Equation in Ding et al. 2002 originally introduced in Chen et al. 2021
L (H, max 0,
Σ cos(H(u),H()))
E
cos cos(H(u), + a
(9)
(u.))P
(u.DEP.
P+
{ { Rij B, (8 ( > [4
P_ = (Xi: E
<t}
are the similar and dissimilar node-pairs in the subset B; 4 > t_ and a > are hyper-parancters. This triplet loss (H, P4,P_) used to update P using gradient descent every TISH epoch. The complexities associated with the loss L1 are explained in Appendix C In this section, theoretical foundation of our approach by presenting key result on the optimality of the POD projection matrix in the context of projection subproblen induced by lincarized GNNs as shown in Theorem [ Theorem suggests that the POD method offers an optimal projection matrix for the linearized GNN update rule. Additionally, we analyze the error propagation in node representations within the PGNN framework across different layers, as formalized in Thcorem 2
establish the
Theorem 1. Let B = p",p", with V Rooxd and U E Raxd denoting arbitrary motrices. Consider the minimization problem
min QQQ - B QEP
(10)
where P denotes the set of rank co orthogonal projection matrices. If column space of B subset of column space of matria pT, there exists Wi _ such that (B4(:; Fw,, P p denotes the POD projection
matria, and p" the left singular eigenvectors associated with F X -1) (0) denotes the normalized Vnca-1 augmented imput node feature matria, denotes the mean along the columns of matria (0), Consequently, the objective function equation 10 satisfies the inequality
IIQBV BUIZ I IIQ FI
(11)
The upper- upper-bound bound of the objective function, given by equation A Appendir A.1
is minimized by the matrir P. (see proof in

---

Published in Transactions on Machine Learning Research (01/2026)
4.1.2 Error bound on the node representations at each layer
Theorem 2. Let X and & represent the actual and opprozimate node representations for the PGNN method with the linearized GNN architecture al a layer Following the update rule X = Ox) the
normalized error ((+1) [ 1-011-8011 2011 at layer ++ caused by the PGNN method is given by,
|
(++)) < |C - Colll |
(1) og O
+T
S
where T || p)[ nx detn) |l and the equivalend convolution malrix for the PGNN melhod Coq PC,
1x |x(-111 1)
p" (S
) (See proof in Appendia A
|
Theorem 2 indicates the quality of approximations made by the PGNN method depends on the equivalence of matrices C and Ceg. The Theorem indicates that errors across layers accumulate with depth. We empirically study the propagation of error in deep GNNs in Appendix D.1 and observed that using the Jumping Knowledge framework (Xu et al. 2018 PGNN compensates for accuracy loss and ensures faster convergence (Figure Sb. The learned represent lions can be qualitatively assessed by visualizing the t-SNE plot of the features from the first layer of a pre-trained PGCN model, for example, shown for the Cora dataset Figure 10 The visualization reveals distinct clusters in the 2D projected space. These clusters align with the seven labels of the dataset, demonstrating the model's ability to distinguish between the seven topic classes in Cora effectively. Appendix [D.4 dedicated to empirical validation, wherein series of experiments are conducted ascertain the congruence of the convolution matrices for the Cora dataset.
4.2 Algorithm
Graph data
Step 1: Preprocessing
Step 2: Training
Generate Count sketch matrix AAA) and LSH hash matrix
Layer parameters: 000
Left singular vectors augmented input feature matrix feature z0: - p(x:9) (Klana)
Input: z(0) S/ p(
R(A)
Node feature matrix
5/ p (20)
convolution
Z1A: TS,(C)
Input
2())
2
X1 geod
Sketch of projection matrix gi
Layer
Layer
R
gnis
Loss
Step 3: Inference
1.
Obtain subset 8 from the LSH hash table bucket b using the
gradient signats ot M(A) P 2 =armmax j M
Transfer parameters (10) (0) post training PGON layers to layers
unsketched representations nodes set (X()).))
classifiici,,,,, Link prediction
ONN Easu update ONN paramoters Compute lass to update P1 epophh Tiss, compute similar dissimilar pairs to Compute triple loss L, and update matrix P
Inference graph
GCN
Downstream task
Layer
Layer 2
Figure 1: The preprocessing, training and inference phases of the PGNN framework preprocessing step, the sketch of the input node feature matrix (Z("),, k sketches of the convolution matrix (s())) and the matrix p (7()) are generated, where we make use of count-sketch matrices R(k), These sketch matrices are utilized in the PGCN layers for the training step along with the LSH hash matrix P. The loss function involves computation of GNN loss LISH for updating GNN parameters, loss L to update #ı, and triplet loss to update the LSH hash matrix P when epoch TLSH. For the inference step, we make use of the parameters lcarned but with the GCN layers as shown in the figure.
shown. In the
Figure depicts the three-phase pipeline of our PGNN approach, encompassing preprocessing, training, and inference stages. Algorithms [ and [ present the complete PGNN framework, which operates within the GCN architecture. The algorithm can be generalized the architectures discussed in sections [B.2 and [B.4] However, for the PGAT architecture (See section B.5 the computation of the sketch for the convolution matrix must be omitted.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm PGNN Training and Inference
Require: Preprocessed data: Z ,....., Rlk, Sč: 2, p(k) from Algorithm Labels y, LSH update interval TLSH;
Loss constants 8, $2, fo, a, oo, 1,, t- for L and Li
1: Training:
2: Initialize GNN weights @ coefficients (1) C% matrix P1
3: for each epoch do
4: for /= 1 to L do
Compute z(0 using the PGNN propagation rule (Fq. 14
end for Select subset. B (BB) S 3) using gradient signals from LSH hash tables 17()) (Section 4.1.1
Reconstruct (1) for nodes in the set B (Eq. ( evaluate loss LLSH (Eq.
Backpropagate and update @()), Ck using LISH 10: if epoch mod TiSH = 0 then
11: Update matrix p(k) using triplet loss L1 (Eq.
12: end
13: Update A1 using 32 randomly sampled nodes using loss (Omitted for PSGC)
14: end for
15: return Learned weights (() and coefficients ck
16: Inference:
17: Predict using standard GCN update rule with learned O!1), ck
Table 1: Performance comparison of PGNN with Graph-SAINT (Zeng et al. 2020 VQ-GNN DDng et al. 2021), Sketch-GNN (Ding et al. 2022), Graph Coarsening (Cai et al. 2021) and lincarized GNN (Wu et al. 2019 on Reddit, ogbn-arxiv, and ogbn-products. If the entry is unavailable in prior literature it is denoted by + The accuracy values in green highlight the best performance for a given method (e.g., GCN) within each column, while the values in red denote the second-best performance for that method.
Mathod
Dataset
ogbn-arxiv
Reddit
ogbn-producte
33C
69.44 1: 0.05
81.64
68.89 :: 0.29
PSGC
88.57 + 0.16
94.66
0.04
65.8 + 0.00
GNN
GCN
GraEESAGE
GAT
GCN
AGG
GAT
GCN
GrapSAGGE
'Full-Grapl'
71.74 0.20
71.49 + 0.27
73.65 + 0.11
00M
00M
DOM
0OM
00M
DOM
Graph SAINT
70.79 0.37
08.81 1 0.39
71.17 0.32
02.25
95.81 0.37
94.31
0.67
76.02 0,21
79.08 0.24
79.71
0.42
Coursming
68.02 4 0.35
66.00 0.6)
63,07
VQ GNN
70.35 1 0.33
70.25 1 0.47
70.43
0.31
93.90 0.21
94.39 1 0.21
94.38 1 0.39
15:24 1 0.32
76.09 - 0.19
76.23 1 0.49
ratio
F=
sketch GNN
70.25 4 0.7
70.48 a 0.80
70.33 4 0.34
92.80 a 0.34
94.85 = 0.61
98.20 4 0.63
75.33 a: 1.00
71.02 1 0.93
77.48 0.77
PCNN ratio
7 0.15
0105
0.03
PONN
0.38
1027
0.03
0.38
00M
5 Experiments.
We evaluate the efficiency of PGNN in terins of memory utilization and training time, with implementa- tion details provided in Appendix D Our assessment, conducted on benchmark graph datasets, focuses on node classification accuracy and compares PGNN's performance against shate-of-the-art methods, including GCond (Jin et al., 2022) and Graph Coarsening (Cai et al. 2021). Additionally, we compare PGNN with other sampling- based methods such GraphSAINT (Zeng et al. 2020). VQ-GNN (Ding et al. 2021 and existing sketch based method Sketch-GNN DDng et al. 2022) For each dataset, the sketch ratios for GCond, Graph Coarsening, and Sketch-GNN remain the same, while PGNN uses either the same or lower sketch ratios than Sketch-GNN. The graph datasets used for evaluation include Cora, Citescer, Pubmed, ogbn- arxiv, Reddit, ogbn-products and ogbn-papers100MM The PGNN update rules for various GNN archilectures discussed in this section are detailed in Appendix B Figures 2 and 3 illustrate PGNN's sublinear memory complexity and training time, making PGNN suitabile for large graph datasets. Additionally, we examine arosttrccie memory complexity in Appendix C
(Figure 7 Node classification accuracies across all datasets are reported in Tables and demonstrating PGNN's ability to achieve competitive accuracy
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Performance comparison of PGNN with SGC, GCN, SAGE, GAT and graph compression techniques like GCond (Jin et al., 2022), Graph Coarsening (Cai et al. 2021 and Sketch-GNN (Ding et al. 2022 on Cora, Citeseer, and Pubmed datasets. The baseline graph compression ratios for the Cora, Citeseer, and Pubmed datasets are 0.026, 0.018, and 0.04, respectively, whereas ratios of r 0.02, 0.018, and 0.01 for these datasets. If the entry is unavailable in prior literature it is denoted by - The accuracy values in green highlight the best performance for a given method (e.g., GCN) within each column, while the values in red denote the second-best performance for that method.
PGNN framework employs sketch
Dataset
Pubmed
Methoo
GCN
81.19 + 0.23
71.91 + 0.18
19.0 = 0.4
SGC
0.1
PSGC
80.51 +: 0.18
72.01 = 0.31
79.90 = 0.30
SAGE
0.0
0.0
CAT
83.0 + 0.7
72.5 + 0.7
79.0 + 0.3
GCN
SAGE
GAT
GCN
SAGE
GAT
GCN
SAGE
GAT
Courseingg
65.15 1 0.51
59.06 1 0.45
GOond
80.02 0.755
76.18 1 0.87
66.2 1 0.0
70.30 1. 0.57
66.2
65.410
77.92 1 0.42
71.12 1: 3.10
Slatck-GNN
90.35 0.71
71.14 0.50
PGNN
80.60 4: 1.46
79.43 d: 1.54
77.82 4: 1.26
71.21 #: 1.13
68.87 # 1.21
70.47 = 4.76
79.23
0.32
78.01 # 0.18
78.23 # 0.61
while maintaining computational efficiency. To further showcase PGNN's versatility across various tasks, we also evaluated its performance on link prediction, comparing against established graph-based methods. The experimental results for the link prediction tasks are detailed in Table S in the Appendix.
PSGC (ours) ((n)
SAGC
6680
1800
Cora
Citeseer
Pubmed
ogbn-arxiv
Reddit
ogbn producrodccts
ogiin averr
Dataset
Figure 2: Memory complexity of the PSGC method.
Figure 3: Training time comparison of the SGC,
GCN, and SAGE with the PSGC method.
5.1 Performance evaluation of PGNN
As shown in Table 2 PGNN outperforms existing graph compression techniques-inclnddin Sketch-GNN, GCond, and Graph Coarsening while closely matching the accuracy of full-graph training on the Cora, Citeseer, and Pubmed datasets, despite using lower or similar sketch ratios. For instance, PGNN achieves better classification accuracies on Cora with a sketch ratio of r= 0.02, compared to r = 0.026 for Sketch- GNN and other techniques such as GCond and Coarsening. Similarly, on Pubmed, PGNN attains superior accuracy with r = 0.01, compared to the r 0.04 required by existing methods. Notably, PGNN maintains high classification accuracy even under lower compression levels, outperforming methods like SGC, GCN, and GraphSAGE on Pubmed 2p. PGNN demonstrates scalability, particularly large-scale datasets. For the Reddit dataset, PGNN achieves an accuracy of 94.82% using sketch ratio of only 0.05, compared to Sketch-GNN, which requires sketch ratio of 0.3 to reach lower accuracy of 92.0%. PGNN with SGC architecture achieves better classification accuracies than SGC. Additionally, PGNN's memory footprint for the Reddit dataset remains as low as 5400 MB with PSGC and approximately 7500 MB with PGCN (Figure 7 For the ogbn-products dataset we observe that with a low sketch-ratio of 0.003 we are able reach full-graph training accuracy for the GCN architecture unlike Sketch- GNN which uses much higher sketch-ratio of 0.3. PGCN consistently outperforms GCN architectures in link prediction tasks in the four
10

---

Published in Transactions on Machine Learning Research (01/2026)
100
Models
PGNN
Sketch-GNN
90
Coarsening
GCond
Full-Graph
85
Accuracy (%)
80
75
g
70
A
A
Datasets
65
X Cora
Citeseer
PubMed
Reddit
ogbn-arxiv
egbn-product
55
0.0
0.2
0.4
0.6 Sketch Ratio (r)
0.8
1.0
Figure Performance comparison different models across varying sketch ratios. PGNN (blue) achieves comparable accuracy other methods while maintaining lower sketch ratios, demonstrating its efficiency in prescrving accuracy with reduced computational complexity. The comparison includes Sketch-GNN (red), Graph Coarsening (green), GCond (purple), and the Full-Graph (orange) baseline accuracies.
datasets: Core, Citeseer, Pubmed, and ogbn-arxiv, as shown in Table 8 Regarding training times, PGNN demonstrates notable elliciency: the training time per epoch for PSGC is consistently lower than that of SGC (except on ogbn-arxiv) and outperforms both GCN and GraphSAGE across all evaluated datasets (Figure 3 Unlike Sketch-GNN, PGNN does not require updating LSH hash tables intermediate layers, thereby reducing computational overhead.
In terms of preprocessing, methods such GCond and Graph Coarsening, while effective for small-scale graphs, exhibit scalability limitations. For instance, preprocessing the Reddit datasct takes approximately 90 minutes with GCond, whereas PGNN completes the preprocessing in around 560 seconds (Table 4 Moreover, PGNN requires a lower sketch ratio to maintain full-graph accuracy as the graph size increases, underscoring its adaptability to larger datasets (Figure 4 Despite processing a massive dataset 111 million nodes, our approach achieves preprocessing in just 28 minutes.
PGCN ogbn-arxiv incurs 2.5% accuracy drop compared to full-graph GCN (Table 1. However despite using sketch ratio of 0.15 compared to r = 0.4 for baseline methods observe only 1% accuracy drop with respect to existing graph compression techniques. Despite PGNN's preprocessing time not being linear, it remains approximately one-sixth of the time required by graph compression algorithms such as GCond (Jin ct al. 2022), ) Graph Coarsening (Cai et al. 2021) (Table 4 The out-of-memory (OOM) error for PGAT on ogbu- productsouuts at low sketch ratio of 0.003 occurs because GAT's learnable convolution mechanism cannot leverage convolution matrix sketching (Appendix B.5 thus preventing the memory advantages achieved by other PGNN architectures.
5.2 ogbn-papers100M: Paper Citation Network
We evaluate our PGNN framework on 1gbn- a largo-scale citation network containing 111 million nodes. This dataset follows the same preprocessing methodology as ogbn-arxiv Wang al. (0019). ensuring consistency in feature construction and graph representation.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Due to computational constraints, bascline results for SGC (2016), and MLPP are obtained from the OGB leaderboard Hu et al. (2020), full-batch training of these methods requires more than 512 GB of CPU memory. Our experiments use a sketch ratio of 10 6 for the Ogbn-papers100M datasct, selected based on available memory limitations. To enable a fair comparison of memory usage; we keep the batch size to 1024 for both GCN and GraphSAGE models.
Wu et al. (2019) Nede2Vec Grover & Leskovec
Table 8 compares PGNN with methods like SGC, GCN, GraphSAGE, MLP, Node2Vec for the ogbn- papersI00M dataset. The preprocessing overhead for the ogbn-papers100M dataset requires only 29 minutes, shown in Table 4 PSGC significantly enhances memory efficiency and training speed while maintain- ing competitive accuracy. It outperforms MLP and Node2Vee, and achieves only a 4% accuracy reduction compared to the full SGC model. Notably, achieves a x39x speedup for the training time per epoch compared to the conventional models. For PSGC, PGCN, and PSAGE, we utilized a POD sketch ratio of 10-6 (r = 10-9). The tensor-sketch ratio for PGCN was set to the same value. However, for PSAGE, a tensor-sketch ratio of 10-7 was used. Higher ratios were not feasible due to CPU hardware limitations for preprocessing.
For PGCN and PSAGE on ogbn-papers100M, utilise PGNN layers during inference.
Table 3: Performance evaluation of PGNN against established graph-based and traditional methods (SGC Wu et al. (2019), GCN Kipf & Welling 2017), GraphSAGE Hamilton et al. (2018), Node2Vec, and
dataset. Blue: PGNN method; green: best memory /training time; red: Out of Memory (OOM); orange: Speedup factor.
MLP) on the large-scale
Method
Test Accuracy (%)
Memory (GB)
Training time per epoch (s)
Speedup
SGC
63.29 + 0.19
>> 512
OOM
PSGC (ours)
59.42 + 0.01
7.70
0.62
MLP
55.60 + 0.23
>> 512
OOM
GCN
27.18 + 0.01
10.42
221.79
PGCN (ours)
56.35 + 0.00
7.72
0.65
339x
Node2Vec
47.24 + 0.31
512
0OM
GraphSAGE
67.06 0.17
10.53
223.82
PSAGE (ours)
50.78 1.37
3.58
0.63
339x
6
Conclusions
The computational and memory demands of large-scale graph learning pose significant challenges for mod- ern GNN frameworks. To address this, we propose PGNN, a novel sketch-based framework that compresses graph data while preserving the downstream task performance. Our experiments demonstrate that PGNN achieves competitive accuracy with state-of-the-art methods at significantly reduced sketch ratios and sub- linear memory complexity. The condensed representations generated by PGNN not only reduce storage overhead but also enable efficient training across diverse GNN architectures. Future work include (1) broad- ening PGNN to dynamic graphs and streaming scenarios, and (2) extending the framework to heterogeneous GNN architectures.
Acknowledgements
We thank the anonymous reviewers of TMLR, the Action Editor, and the Editors-in-Chie for their con- structive feedback, which helped improve the paper.
12

---

Published in Transactions on Machine Learning Research (01/2026)
References
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning, 2016.
Alexandr Andoni, Piotr Indyk, Thijs Laarhoven, lya Razenshleyn, and Ludwig Schmidt. Practical and optimal Ish for angular distance, 2015.
Muhammet Balcilar, Guillaume Renton, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Analyzing the expressive power of graph neural networks in a spectral perspective. In International Conference on Learning Representations, 2021. URL
Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling, 2020.
Aleksandar Bojchevski, Johannes Gasteiger, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózem- berczki, Michal Lukasik, and Stephan Günnemann. Scaling graph neural networks with approximate pagerank, August 2020. URL http://dx: doi. org/10. 1145/3394486. 3343496 3403296
Francesco Bonchi, Claudio Gentile, Francesco Paolo Nerini, André Panisson, and Fabio Vitale. Fast and effective gan training with linearized random spanning trees, 2024.
Chen Cai, Dingkang Wang, and Yusu Wang. Graph coarsening with neural networks, 2021.
Moses Charikar, Kevin Chen, and Martin Farach-Colton. Finding frequent items in data streams. In Peter Widmayer, Stephan Eidenbenz, Francisco Triguero, Rafael Morales, Ricardo Conejo, and Matthew Berlin, Heidelberg, 2002. Springer
Hennessy (eds.), Automata, Languages and Programming, pp. Berlin Heidelberg, ISBN 978-3-540-45465-6.
Beidi Chen, Zichang Liu, Baoxiang Peng, Zhaozhuo Xu, Jonathan Lingjie Li, Tri Dao, Zhao Song, Anshumali Shrivastava, and Christopher Ré. MONGOOSE: A Learnable LSH Framework for Efficient Neural Network Training. International Conference on Learning Representations, 5 2021. URL pdf7id=xWW(yXXuU..h
Jianfei Chen, Jun Zhu, and Le Song. Stochastic training of graph convolutional networks with variance reduction, 2018a.
Jie Chen, Tenglei Ma, and Cao Xiao. Fastgen: Fast learning with graph convolutional networks via impor- tance sampling, 2018b.
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q Weinberger, and Yixin Chen. Compressing neural networks with the hashing trick, 2015.
Wei-Tin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks, July 2019. URL http://dx doi.
org/10. org/10.1145/3220 3115/3/925000330002 3330925
Jeongwhan Choi, Scoyoung Hong, Noscong Park, and Sung-Bac Cho. Gread: Graph neural reaction-diffusion networks, 2023.
F. R. K. Chung. Spectral Graph Theory. American Mathematical Society, 1997.
Mucong Ding, Kezhi Kong, Jingling Li, Chen Zhu, John Dickerson, Furong Huang, and Tom Goldstein. Vq-gnn: A universal framework to scale up graph neural networks using vector quantization, 2021.
Mucong Ding, Tahseen Rabbani, Bang An, Evan Z Wang, and Furong Huang. Ctetch-GNNN: Scalable graph neural networks with sublinear training complexity. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
http:ppaareeeee...tt...ooa/aaaaa https://openreview net:foorm/idd=PPJDeet [wCC
13

---

Published in Transactions on Machine Learning Research (01/2026)
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The World Wide Web Conference, WWW '19, pp. 417-426, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366748. doi:
3313488.
URL attps:///oi.... wtt.l/..rgg/...1///300511355 1145/3308558 3313488
Matthias Fey, Jan E. Lenssen, Frank Weichert, and Jure Leskovec. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddinggs, 2021.
Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti.
Sign: Sealable inception graph neural networks, 2020.
Mina Ghashami, Edo Liberty, Jeff M. Phillips, and David P. Woodruff. Frequent directions deterministic matrix sketching, 2015.
Simple and
Adilya Grover and Jure Leskover. node2vec: Scalable feature learning for networks. CoRR, abs/1607.00653 2016. URL
Gaël Guennebaud, Benoît Jacob, Philip Avery, Abraham Bachrach, Sebastien Barthelemy, et al. Eigen v3, 2010.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs, 2018.
Philip Holmes, John L. Lumley, and Gal Berkooz. Turbulence, Coherent Structures, Dynamical Systems and Symmetry. Cambridge Monographs on Mechanics. Cambridge University Press, 1996.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprinl arXiv:2005.0000 2020.
Zengfeng Huang, Shengzhong Zhang, Chong Xi, Tang Liu, and Min Zhou. Scaling up graph neural networks via graph coarsening, 2021.
Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks, 2022.
Shiva. Prasad Kasiviswanathan, Nina Narodytska, and Hongxia Jin. Network approximation using tensor sketching. In Proceedings of the T'wenty-Sevente International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 2319-2322 2325. International Joint Conferences on Artificial Intelligence Organization, 7 2018.
doi: 10.24963 20.24463/jjcai.22 /tt/://aaaa.ii/////11 2018/3211 3221. URL lhttps://oo/ org/10. 111966/1jcai22118///
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks, 2017.
S. Lall, J. E. Marsden, and S. Glavaski. Empirical model reduction controlled nonlinear systems. In International Federation of Automatic Control, PROCEEDINGS OF THE WORLD CONGRESS- IN- TERNATIONAL FEDERATION OF AUTOMATIC CONTROL, volume e, pp. 473-478, Oxford, 1999. Published for the International Federation of Automatic Control by Pergamon;. ISBN 0080432166. URL httpss:/www. tib. de/ wwwchee/id//LLC///ACCC////
See Hian Lee, Feng Ji, and Wee Peng Tay. Node-specific space selection via localized geometric hyperbolicity in graph neural networks, 2023.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi- supervised learning, 2018
Yibo Lin, Zhao Song, and Lin F. Yang. Towards a theoretical understanding of hashing-based neural nets, 2019.
Zirui Liu, Kaixiong Zhou, Fan Yang, Li Li, Rui Chen, and Xia Hu. EXACT: Scalable graph neural networks training via extreme activation compression. In International Conference on Learning Representations, 2022. URL httpp:///ppaaeeeeeeeeeeeeerr//rr
rX
14

---

Published in Transactions on Machine Learning Research (01/2026)
Andreas Loukas. Graph reduction with spectral and cut guarantees, 2018.
Moore. Principal component analysis linear systems: Controllability, observability, and model reduc- tion. IEEE transactions on automatic control, 26(1):17 1981. doi: 10.1109/tac. 1ee...91..111220 URL
https://doi. wttpg//...///....e..ee 1tt.:111001....111111000.0511110 1981.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Autiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS 2017 Workshop on Autodiff, 2017. URL https://openrevveet nott/forummidddBJJsrtit
Ninh Pham and Rasmus Pagh. Fast and scalable polynomial kernels via explicit feature maps. In SIGKDD, KDD '13, pp. 239-2022 247, New York, NY, USA, 2013a. Association for Computing Machinery. ISBN 9781450321747. doi: 00.1445/45775552287788 URL https://doi. org/10. 1145/2487575. 2487591
Ninh Pham and Rasmus Pagh. Fast and scalable polynomial kernels via explicit feature maps. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '13, pp. 239 247, New York, NY, USA, 2013b. Association for Computing Machinery. ISBN 9781450321747. doi: 1114452248775.24557777
URL https://doi. http://///ooooo//...ooo/// 2tgg//..116//48788000 1114//248277555.. .2875911
Muruhan Rathinam and Linda R Petzold. A new look at proper orthogonal decomposition. SIAM Journal on Numerical Analysis, 41(5):1893-195 1925,,992222220 2003.
Ryan Spring, Anastasios Kyrillidis, Vijai Mohan, and Anshumali Shrivastava. Compressing gradient optimiz- ers via count-sketches. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5946-5955, PMLR, 09-15 Jun 2019. URL https://procceddngg mlr press//997/springge0. html
Jiawei Sun, Hongkang Li, and Meng Wang. How do skip connections affect graph convolutional networks with graph sampling? theoretical analysis generalization, 2024. URL httpp:://operreriew.nttt
forum?id=JJpMoN22pon
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.
Graph attention networks, 2018.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander J. Smola, and Zheng Zhang. Deep graph library: Towards efficient and scalable deep learning on graphs. CoRR, 0bs/1199.00131, 2019. URL http::/arxivv... http://wxxvvv.ogabb/////as/ 19999001115
Kilian Weinberger, Anirban Dasgupta, Josh Attenberg, John Langford, and Alex Smola, Feature hashing for large scale multitask lcarning, 2010.
Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr. au2, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks, 2019.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xic, and Bin Cui. Graph neural networks in recommender systems: A survey. ACM Compul. Surv., 55(5), dec 2022. ISSN 0360-0300. doi: 10.1145/3535101. URL https:
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks?, 2019.
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Raigopal Kannan, and Viktor Prasanna. Graphsaint: Graph sampling based inductive learning method, 2020.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching, 2021.
15

---

Published in Transactions on Machine Learning Research (01/2026)
A Appendix
Definition 1. If the column space of matric B is subset of matrix A there cxists a matrix M such that
B = AM.
Before establishing Theorem ! we clarify an assumption central to its proof. Lemma captures key column- space invariance property: whenever col(A) S col(B), a matrix M exists (and uniquely determined) such that BM. This observation, along with the conditions ensuring consistency of the associated linear system, fundamental for the argument Theorem [
Lemma If the column space of = C(' is contuined in the column space of p², then the matria M defined in Definition 7 takes the form
M = pB.
proof: By assumption,
col(B) S col(p").
Hence, there exists a matrix M such that
B= pf (Definition
Left-multiplyin both sides by p and using the orthonormality condition pp" Los yields
M = =B.
Thus M = pB is the unique matrix satisfying C = p² M. In what follows, we demonstrate that there indeed exist vectors fulfilling this condition. Existence of solution. To ensure the existence of vector _ satisfying
c = p" =
pC p²=
the corresponding linear system must be consistent. Since this system is overdetermined, must lie in the mullspace of p")pT. This leads to the requirement
p00 =0
(12)
When C I T=1-D-1/2AD- the normalized Laplacian Chung 1997 with degree matrix D and adjacency matrix A, we have:
span(D!/21,) E mC)
We also have
//2, e mull(C()).
Therefore, if TT / 1,), then equation 12 holds, and since the column space of p" coincides with columns of F be scalar multiples of D1/21, to ensure
that of F X - it suffices that the existence of a vector z.
/
A.1 Proof for Theorem 1
The output layer node representations for the linearized GNN using the optimal approximation of the input node feature matrix given by the POD,
の TT) 7 + [C
BV + W1:
B C= 20 v = [ Consider the number hidden channels at layer We propose that the projection matrix, denoted by Q, the product of matrices Q1 and Q2, such that Q Q1Q2- In this context, Q2 signifies the linear transformation responsible for reducing the dimensionality of matrices.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Conversely, Q1 is the matrix that handles the inverse projection. The optimal projection matrix Q" is obtained from solving the optimization problem described below,
Co
min QQBV BV QEP
min QEP
IQBU(:)) BV(;0l2
When the columnspace of matrix B Cp" is assumed as subset of columnspace of matrix p", then for every vector (t, i), there exists a; (:.a) such that (V( p" a, (Lemma []_ Also since the matrix p" denote the left singular vectors of the SVD of the matrix F -B)xxoo p" (:. (.. p:., Thus there exist W/ R such that p²a; Fws. The matrix X denotes the
vne--T col(F) span(o" (:
augmented imput node feature matrix (Algorithm The optimization problem in equation A.1 becomes
min QEP
IQF Fuil
min QEP
IIQQF - F)
i=1
min QEP
|QQ
F
||:
(since ||Ar||2 S ||A||a||<||2 S ||||))))(())
(13)
Given a dataset X, the POD projection matrix P minimizes
e( -Px X) (see Rathinam Petzold (2003)
This property remains valid when the dataset is centered to obtain F. Therefore, the matrix Q' which minimizes the R.H.S of inequality 13 is the POD projection matrix P corresponding the centered dataset F.
A.2 Proof for Theorem 2
Let the equivalent node representations at layer 1+1 in the graph dimension n, be 8(4+)), the PSGC update rule (Appendix B.2 is given by the following expression:
x(++1) = PCX () (T p)x
|x(+1) -(++))))+++11
( (PCX + (Pl--)x)axxxllll
[x((1) 8a+1 -
- (1))) C (Cog& + + P)(zladdan))
Using the triangular inequality and sub-multiplicative property of norms,
((+1) |C - Cegll
||000
( |Celll | T, S
||
s
X
Lemma 2 Ding et al. (022 ). Given motrices C E Raxn and (x())(() E Rdx=, consider a randomly
selected count sketch matrir R E Rexxn (defined in section 3. where is the sketch dimension, and it is formed using r == Vin underlying hash functions drawn from a 3-wise independent hash fomály H. for some 121 1. If ce 2 2+31
we have
250
-xX0) >
X ( 50.
B Generalize to more GNNs
This section presents a compendium of prevalent GNNs that can be tailored to fit into the unified framework delincated in section 3. The crux of most GNN architectures revolves around message passing among node features, followed by feature transformation and activation functions-2 process commonly known "generalized graph convolution'. Within this overarching framework, the distinctions among GNNs primarily
17

---

Published in Transactions on Machine Learning Research (01/2026)
arise from their choice of convolution matrices, denoted C which can either remain static evolve as trainable parameters. A trainable convolution matrix is contingent upon input data and adjustable parameters, potentially varying across different layers, as denoted by C(1
cll.)) h (ll.o) (X ((l(a)
fixed
learnable
B.1 Theoretical Guarantees for nonlinear GNN architectures.
We analyze how the PGNN framework extends to various nonlinear GNN architectures, including GCN (Kipf & Welling, 2017), GraphSAGE (Hamilton et al. 2018), and GAT (Veličković et al. 2018). While Theorem 1 establishes PGNN's optimality specifically for a projection subproblem induced by the linearized GNN case (SGC), the framework maintains rigorous performance guarantees for nonlinear architectures through general sketch-based approximation theory. To quantify the approximation quality for these nonlinear cases, we adapt error bound results from Ding et al. 2021) as Theorems 3 and 4 which provide bounds for forward- pass node representations and back-propagated gradients, respectively. This theoretical framework ensures reliable approximation quality across diverse geessagepassing schemes and provides crucial insights into how compressod computations affect the learning process relative to standard GNN update rules.
20i8eeWhile
FOOL, sbeicheateer 200 FUUN, skosch 1950: U.LL GCN
PGCA dalichraaiz: 6.006 POUM Sutth ra09: 6:928 GCV
POCH Fietch nED: GO) conthoorio 0.0
129
1.0
1.0
500ch
Epoch
CO0Ce
(a) Cora
(b) Citeseer
(c) Pubmed
Figure 5: Comparison of training losses during link prediction for PGCN and GCN mot across bench- mark datasets: (a) Cora, (b) CiteSeer, and (c) PubMed. The sketch ratios used are 0.013 and 0.026 for
Cora, 0.009 and 0.018 for CiteSeer, and 0.01 and 0.02 for PubMed, respectively.
Theorem 3 Ding et al. (2021)). Tf the relative error of the 1-th layer for the PGNN method is e()), the convolution matriz (L is either fixed or learnable with the Lipschitz constant of h)) R2 + R upper- bounded by Lip(h?)) and the Lipschitz constant of the non-linearity is Lip(a), then the estimation error of forward-passed fealures salisfies,
( (1 () ||
Thcorem 4 DDig at al. (2021)). If the conditions in Theorem hold and the non non-linearity satisfies jo (2(2)) a' man for any z R, then the estimation error of back- paokkproppgnted gradients salisfies,
|| Vxoelpss 200 (1+ (0 |0 |
In the below subsections, we discuss how the PGNN framework works with various GNN architoctures using update rules described in the supplementary material of Ding et al. (2022).
18

---

Published in Transactions on Machine Learning Research (01/2026)
B.2 PGNN with SGC.
The convolution matrix for PSGC, C --1/2 D = D - A, A In. D denotes the degree matrix and A represents the adjacency matrix of the graph. The node representation at layer + 1, z given by the PGNN method is ((++) = Sc (()) + nxx O The sketch of the convolution matrix Sc = pCp
B.3 PGNN with GCN.
The update rule for PGCN is given by
FFT (ÖFFT (v(m)
2
Ck p1 St
U
(14)
U = ( The convolution matrix C' has the same form as the PSGC update rule (Appendix B.2
SC - (C). Tensor-sketch of order k, (C) FFT (O FFT (Cs()) (c))) N = (p"p, p') +
)00) vd() (NT) O(0" (1x") (1="R((0) laixe + Z01" pI p) " pl("" R(() E R= denotes
the Count-sketch matrix (Section
Detailed steps for the PGCN method are explained in Algorithm
B.4 PGNN with SAGE.
We extend PGNN to architectures involving multiple convolutions C(2) = Inxn and ((2) = D-1A. C (00) || c(e) The update rule for SAGE is
x(++1 =o = (x(00()) + -D'AXX00(13)) ( A) / (| (15)
With = [pt]nxdin The update rule for PSAGE given by
1 (ÖFFT (V(1) )]
10-1) C P1 St.
FFT-1
U.
The sketch of the convolution matrix St. = p (C). Tensor-sketch order h;
O FFT (c5)) (C( CR(P)T, R(P) 22xe.
= -FFT- -))
Ñ = ~=((p- p(z(()))) + [x)xd ((1:1) (p p1 ( + [+1]nxd ()(22)]
v()) (
R() [ 2 + +U1) ( -1))g(..11) (3(2,0) 21 + +12) 00122)
N
p(mm
= (1 : n,:): p(..)) = pR +1 2n,;), U1 = [R())((,11 axdx Un =
[ROOT (OOT((, (:: xdi
B.5 PGNN with GAT.
The convolution mechanism intrinsic to the GAT architecture inherently learnable. We propose PGAT update rule for update rules involving learnable convolution for completeness. There however no memory or training advantage when using PGAT. A promising direction for future work lies in designing GAT update rules that leverage the PGNN method's computational advantages. Equation deffines the unsketch
19

---

Published in Transactions on Machine Learning Research (01/2026)
operation.
U [pā] nxdd41 E(1.0) F(1.0) F(1.9)
F(h.) unsketch (z @(1.0)
a (1.)) = (p" (1 + [T] 10x (
C A A+I, a. R4411
CGAT = C Cxp(LaakyRoLU(E4400))
(16)
z(++1.00 Pipo (sftmmax (CGAT) unsketch (zthat) @llia)) -U
(17)
Algorithm PGNN Preprocessing
Require: Node feature matrix ( E Raxd, convolution matrix C, sketch ratio co mumber of sketches
k, count sketch malrix dimension
1: Set co = [rn]
2: co > d then
r1: < randomly sample n indices from {1,2,.... ..,....d]]... d} with replacement
M = (;,)) @ (:
Augmented node feature matrix: X [X M]
X (O) {Omitted for PSGC)
7: else
8: Set (0) (
9: end if
10: Compute mean vector + 1 Exe (;,
11: p² left singular vectors from SVD of [()
left
Voon-1
12: Sketch of input node feature matrix: Z(0) p(X(1) [T]
13: Generate count-sketch matrices R R(K)
14: Compute sketch St p (C) {Omitted for PGAT]
15: Compute auxiliary terms for forward propagation: pē, F R!K)T and g = -P((NT
16: Initialize LSH projections (())
17: return Preprocessed sketch Z(0), count-sketch matrices R RO.,..,, RK0 sketch Sg. H, LSH projections
p(k)
C Complexity analysis
We delineate the intricacies inherent in the algorithm with the PGNN framework.
One-time Preprocessing: The pre-processing step involves finding the right singular vectors the ma- trix described in Algorithm 2 which takes time O(dnog). Computing Se = pCpp for PSGC takes (n?) computations and Sc pTS (C) for PGCN, PSAGE takes Occocnn) computations. Computing the sketch of the initial node feature matrix Sx ((X - [=] takes time O(nc,d). Computing the sketches of the matrix p obtain ş has linear time complexity.
Overhead of computing LSH hash tables. The time complexity for computing the hash index for cach node is O(coc)) when using Simhash (See section 8 and since there are n nodes and f hash tables, we get an overhead of O(fncock) for time and O(forc)) for space.
Overall, the preprocessing phase has a time complexity of Onn²) and a memory complexity of O(n) for PSGC. Offerenn) time complexity and O(n) memory complexity for PGCN, PSAGE. The time consuming part involved in the preprocessing phase is the formation of matrix P; we present the time consumed by this
20

---

Published in Transactions on Machine Learning Research (01/2026)
process in Table
Training complexities with PGNN. We present the complexities within the context of the GCN archi- tecture. Forward and backward pass: Computing v(() involves +(dc))+ ((decc)). FFT and inverse FFT in- volves (dc; log(s))). This reduces to O(dc)) (dcsex). The memory complexity involved is O(e)) + +(g(a)). Complexity associated with loss LLSH
1. Computing the subset B of nodes based gradient signals involves O(c+) computations. Unsketch- ing B number of node representations involve ((3crd) computations.
2. (BBd²) for computing similar and dissimilar node pairs. Computing and updating hash table p(k)
using L will involve (ccyd) operations, which needs to be updated every Tepoch epoch.
Complexity associated with loss L1 (Section 4.1.1). O(3rcrd) computations.
Computing loss and updating L1 requires
Inference: incurs O(Ld( m + d)) time and O(m + ndL L) memory as is the case a standard GCN. n Remark. The training complexities mentioned above do not hold for the GAT architecture (Veličković et al. 2018) because of the inherent nature of the operations involved, which is expounded in Appendix B.5 The underlying complexities in the original GAT architecture will hold, and for completeness, we present the accuracies for the various datasets using PGAT in Tables
2
An implementation detail. When
Table 4: Computation time (p) for different datasets at specific sketch ratios (r = +
Dataset
Sketch-ratio r
So
p (Time in 8)
Cora
0.02
0.0141
Citeseer
0.018
0.0254
Pubmed
0.01
0.1558
ogbn-arxiv
0.15
1786.8692
Reddit
0.05
426.9778
ogbn-products
0.003
350.93
1g0b--paper100M
1e-6
1708.40
the sketch ratio is such that [rn] > d, which is the feature dimension, the PCA or the POD method necessitates computing the covariance matrix (Ding el al. (021). To overcome the challenge of storing and computing the covariance matrix, we use the feature engineering method to augment X by selecting random combinations of columns of this matrix to find the augmented input node fcature matrix x (Sec Algorithum 2
Preprocessing complexity scaling analysis. The dominant factor preprocessing involving computa- tion of the matrix p of projection as reported in Table |
exhibits distinct scaling behaviours across different graph size regimes, illustrated in Figure 6 We analyze these patterns by comparing observed runtime against. theoretical bounds.
Small-Scale Regime (10--104 nodes): For citation networks including Cora, Citeseer, and Pubmed, the preprocessing time scales consistently with O(n)-5), closely tracking the theoretical reference line in our complexity plot. This near-optimal scaling demonstrates the effectiveness of POD-based sketching for modestly-sized graphs with standard feature dimensions (S 128).
Intermediate-Scale Regime (10s 106 nodes): For Reddit and ogbn-arxiv, we observe performance degra- dation approaching O(n²) scaling, indicating "scaling bottleneck" in this regime. This behaviour stems from two factors: Reddit's higher feature dimensionality (602 features) increases the (ddnc)) complexity term, while ogbn-arxiv's elevated sketch ratio similarly amplifies this cost, where d represents the fcature dimension.
Large-Scale Regime (10++ nodes): Performance recovers dramatically at scale. The ogoduproduuts dataset (2.4M nodes) returns to near-O(n scaling, while Ogbn-papers100M (111M nodes) achieves sub- Onn performance, falling below the theoretical reference line. This super-linear efficiency at massive scale likely results from lower sketch-ratio scaling compared to the graph dimension n.
21

---

10 Published in Transactions on Machine Learning Research (01/2026)
O
10a
On
SVD Computation Time (seconds)
10
10a
ogbn-products (r=0.003)
ogbn-arxit (r=0.15)
ogbn- coon-papers100m (r=1e-06)
102
Reddit (r=0.05)
Pubmed (r=0.01)
10
Citeseer (r=0.018)
10-2
(r=0.02)
10a
104
10a
107
10a
Number of nodes (n)
Figure 6: Preprocessing complexity scaling analysis.
C.1 Experimental Evaluation of Memory Requirements for PGNN across architectures
Figure compares the memory consumption of our proposed methods PSGC, PSAGE, and PGCN across various datasets. The results demonstrate that PSGC achieves the lowest memory usage. The tensor-sketch ratio used for cora, citeseer, ogbn-arxiv and reddit are 0.7, 0.5, 0.15, 0.15 with order g and number of sketches k as 3 (Algorithm [ These experiments were conducted without mini mini-batching. PSAGE incurs nearly twice the memory complexity of PGCN due to the transfer of two convolution matrices to GPU memory.
25000
PSGC
PSAGE
PGCN
20000
Memory (MB)
15000
10000
5000
Cora
Citeseer
Pubmed
ogbn-arxiv
Redalit
Dataset
Figure 7: Memory consumption comparison of PSGC, PSAGE, and PGCN across various datasets.
22

---

Published in Transactions on Machine Learning Research (01/2026)
D Additional Experiments
D.1 Comparing performance improvements obtained when using jumping knowledge networks.
h
( (inal
Layer Aggregation
h
(3)
E ER
R
coX
N.A
Train Loss (PSGC with skip-connection)
C₁ x
375
Train Loss (PSGC)
h (2) E R
152
N.A
coXd,
h (1)
EER
0.75
N.A
600
Input feature of node v
Epochs
Figure 8: Classification loss when using Jump- ing knowledge network architecture on the PSGC versus the PSGC on a 3-layer GNN on the Cora dataset.
Figure 9: Jumping Knowledge network architec- ture for the PGNN method. N.A. denotes neigh- bourhood aggregation.
As the depth of GNNs increases, there is a tendency for the node representations converge a standard value, phenomenon called 'over-smoothing' (Li et al.) 2018). A widely adopted mitigation approach in the literature is to bypass intermediate layers and directly contribute to the future layers by combining the
Jumping Knowledge framework with models like GCN and GraphSAGE. The Jumping Knowledge (JK) framework Xu et al.) 2018 Sun et al.) 2024) aggregates features from multiple GNN layers, enhancing exprossiveness and robustncss whilc addressing the issuc oversmoothing. In PGNN, as the depth of the GNNs increases, there is an accumulation of error, as shown for the linearized GNN Theorem 2 affecting the downstream task. We use the skip-connestions in the Jumping Knowledge framework as shown in Figure find that the loss in accuracy due to depth for the Cora dataset was compensated by introducing skip connections described in the Jumping Knowledge architecture in Figure D. In the final layer of our model, we employed a layer aggregation technique. The layer aggregation process utilizes the formula
while presenting the classification loss for the convergence aspect in Figure S Empirically,
finall (2 (v. :), (v, h)) 1(0) hh(h))) 10e)tt
to effectively combine the information from the various layers. Dellasees denotes the number of output cate- gories specific to the dataset. Ocat E Rdde XMelaseee dess = d1 + d2 + d3.
D.2 Evaluating the impact of increasing sketch-ratios.
To evaluate the impact of sketch ratio (r= ) on the accuracy of PGNN and Sketch-GND under comparable conditions, we conduct additional experiments on the Cora and Citeseer datasets using the GCN architecture as the base. The results in Table D illustrate the performance of both methods across similar and increasing sketch ratios. We observe that the accuracies of our PGNN method is better than sketch-GNN for the Cora and the Citescer dataset for similar sketch-ratios. The training loss comparison between PGCN and GCN methods across three citation datasets is shown in Figure These results show that. increasing the
23

---

Published in Transactions on Machine Learning Research (01/2026)
Table 5: Comparison node representations between the PGCN method and the Taylor series approximation of the GCN update rule.
Dataset
Method
(1)
Layer 1
Toglor Layer 2
(2)
Taylor
Cora
PGCN
1.0105
0.9829
Cora
Taylor
1.0274
0.8807
Citeseer
PGCN
1.0075
0.6747
Citeseer
Taylor
1.1895
1.0110
Figure 10: A t-SNE plot of the computed feature representations of the pre-trained PGCN at the first layer the Cora dataset. Node colours denote classes.
sketch-ratio generally improves convergence speed, particularly for the Cora and Citesecr datascts. Notably, PGCN demonstrates significantly faster convergence compared to the baseline GCN. For Pubmed, PGCN exhibits convergence behavior comparable to GCN at sketch-ratios of 0.01 and 0.02.
Table 6: Accuracy Comparison of Sketch-GNN and PGNN on Cora and Citeseer
Dataset
Sketch Ratio
Sketch-GNN Accuracy (% + Std)
PGNN Accuracy (% + Std)
Cora
0.013
80.12 = 1.04
80.65 + 0.68
Cora
0.026
80.35 = 0.71
80.52 + 1.24
Citeseer
0.009
70.91 = 0.93
70.23 + 0.61
Citeseer
0.018
71.14 = 0.59
71.21 + 1.13
D.3 Evaluating the Quality of Node Representations at each Layer.
In Table 5 we present a comparative analysis of node representations generated by the Taylor series ap- proximation and the PGNN method with the GCN architocture. The results demonstrate that the PGNN method achieves lower error rates across individual layers for the Citescer dataset. For the Cora dataset, PGNN exhibits reduced errors in the first layer, while the errors in the second layer comparable to those
24

---

Published in Transactions on Machine Learning Research (01/2026)
of the Taylor series approximation. The Taylor series approximation of the node representations and the
x
Taylor - -x
node representations obtained from the PGCN method is utilized to compute Taylor For the first layer, we have:
(1)
x Tagtor
x = =((x)(00)
The Taylor series approximations of the node representations at layer one and i-th column are given by x(1) X Taylor (:, = Cx(0)((() (:. + C i). The approximate node representations in the graph dimen- sion n by the PGCN method in the first layer is given by
{(1) Mcan { R(() (1o41)T p1 (10 +
nx).
Mean refers to the element-wiss mean over tensors.
We conducted additional experiments on Cora and Citeseer datasets examine error accumulation for _
deeper GNNs. In Table 7 we present the relative error al layer defined ell)
x Results
X F
Table 7: Relative error (e()) between GCN and PGCN representations across different network depths on Cora and Citeseer datasets, with and without jumping knowledge (JK) connections.
Cora
Citeseer
Layers (1)
without JK
with JK
without JK
with JK
0.6744
0.6744
0.7835
0.7835
2
1.1222
1.1222
1.0893
1.0893
3
1.1270
1.1270
1.3990
1.3990
4
1.9854
1.9854
1.7829
1.7829
5
1.7450
1.7450
1.4365
1.4365
6
1.2477
1.0898
2.4158
1.0585
show that the relative error fluctuates across layers for both datasets. The error values remain identical with and without jumping knowledge connections for layers 1-5, jumping knowledge only impacts the final layer aggregation. At layer 6, jumping knowledge connections significantly reduce the error (from 1.2477 to 1.0898 for Cora, and from 2.4158 to 1.0585 for Citeseer), effectively mitigating error accumulation in deeper architectures.
D.4 Comparison of spectral properties of the sketches of the convolution matrices
We say that a matrix B Raxn is an approximation to matrix E Raxn if their quadratic forms have the form
x T Bx <I TTT Az
< E x Bz VIER"
The above equivalence implies the spectrum similarity between the two matrices (Courant-Fisse Fisher Theorem Chung (1997)). We present comparisons of the eigenvalues and eigenvectors of the convolution matrix C and the equivalent convolution matrix Ceg PC for the Cora dataset in Figures Ila and [1b The eigenvalues and eigenvectors of the matrix Geg closely align with those of C.
D.5 Additional Experiments: Link Prediction
To further evaluate the effectiveness of our method beyond node classification, we conducted additional experiments on the task of link prediction using the Cora, Citescer, Pubmed, and ogbn-arxiv datascts. The results below compare PGNN with the standard GCN baseline. We observe that PGNN outperforms the GCN method on all the four datasets.
25

---

Published in Transactions on Machine Learning Research (01/2026)
0.7
0.6
Relative Change
0.4
0.3 -
04-
0.2
02
0.0
1010 Eigenvalue Index
1530
2000
2500
-0.1
(a) Eigenvalue comparison of matrices C and Ceg the Cora dataset. Here the relative change is given by (,
(b) Comparison of Figenvectors of C and Cug for the Cora dataset, with the ij-th element M,y quantified by (cs(C) (Cr(Ceg), where (C((C) and Φi denote the ij- th component the eigenvectors for C' and Ceq: respec- tively.
(A,Cel- XCC denoling the - th eigenvalue of
Figure 11: Comparison of cigenvalues and eigenvectors of C and Ceg for the Cora dataset.
Table 8: Link prediction performance comparison between PGNN and GCN across four citation datascts. Best Test AUC (+: Std) scores for each dalaset are highlighted in green.
Dataset
GCN
PGNN (Sketch Ratio)
ogbn-arxiv
92.08 + 1.38
94.74 + 0.78 (r 0.10)
Pubmed
90.51 + 1.82
92.66 + 2.86 (r 0.04)
Cora
78.24 + 0.04
81.90 + 3.30 (r -0.03)
Citeseer
77.40 + 2.32
86.48 + 1.92 (r 0.03)
D.6 Convergence Analysis: Loss and Accuracy in Node Classification
The convergence analysis for PGCN and PSAGE architectures on the Reddit dataset presented in Fig- ures 12 and 13 where we examine the evolution of training loss alongside training, validation, and test accuracy metries throughout the optimization process.
E Implementation Details
We outline the various implementation details with the hyper-parameter setups for experiments section
Datasets. Table g provides a comprehensive summary of the statisties for all datasets utilized in the sourced from the Open Graph Benchmark experiments. The datasets ogbn-arxiv and agbunproodcts original dataset by Hamilton and colleagues, (OGB I
The Reddit dataset, more streamlined variant of the was acquired through the PoTorch Geometric library For our research, we adhered to the conventional dataset divisions established by OGB and Torch Geometric.
Code Frameworks. The codes used for experimentation are made available at PGNN framework make use of the PyTorch library and the PyTorch Sparse library For the computation of the sketch of the input node Teature matrix, the svd function from the Pytorch library is used. The Count-sketch technique implementation is taken from the repository All of the above code repositories used are licensed under
htttps:/ogbb. http:///gbb.stafr...... edu/
2 https://github. wtw. pyg beampptoooyyt gew.rerricg
https://gtthub.ccommattttt1
http:://gthubb.commrutty////
sparse
http::/gthabb.cm///ommnn///1991 GNIV Sublinear
26

---

Published in Transactions on Machine Learning Research (01/2026)
Train
0.70
35
0.65
0.60
0.55
Accuracy
2.0
1.-
Validation Accuracy
Test Accuracy
0.30
200
600
800
1000
200
400
600
800
1000
Epoch
Epoch
Figure 12: Training convergence bohavior of PGCN mothod for node classification downstream task for Reddit.
Train Loss
3.5
2.5
0.6
Accuracy
Loss
15
0.4
10
0.2
Validation Accuracy
Test Accuracy
200
1000
200
400
600
800
1000
Epoch
Epoch
Figure 13: Training convergence behavior of PSAGE method for node classification downstream task for Reddit.
the MIT license.
E.1 Hyperparameters
We conducted a comprehensive hyperparameter search across all model architoctures. Learning rates tuned in the range (0.00,,0005), and weight decay values in the range [10-3 4 10-2) Network
27

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset
Cora
Citeseer
Pubmed
oghn-arxiv
Reddit
ogbn-products
oghn-papersl00M
Task
node
node
node
node
node
node
node
Setting
transductive
transductive
transductive
transductive single
transductive
transductive
transductive
Label
single
single
single
single
single
single
Metric
accuracy
accuracy
accuracy
accuracy
accuracy
accuracy
accuracy
Nodes
2,708
3,327
19,717
169,343
232,965
2,449,029
111,059,956
# Edges
5,429
4,732
44,338
1,166,243
11,606,919
61,859,140
I, L,15,685,877
# of Features
1,433
3,703
500
128
602
100
128
of Classes
7
3
40
41
47
172
Table 9: Detailed Overview of the graph dalasets utilized in experiments.
architectures employed hidden dimensions of {128, 150, 192, 256} with either two or three layers. Dropout rates were variod across {0.0, 0.2, 0.5), and normalization strategics included no normalization, BatchNorm, or LayerNorm. For PGNN-specifie parameters, both the order and number of sketches were set. from {2,3]. The regularization parameters a, coo, and Po were tuned over {0, 0.5, 1), while 8, 32 were constrained to at most [0.1n].
Dataset
Model
Learning Rate
Weight Decay
Hidden Dim
Num Layers
Dropout
Norm Type
PGCN
0.01
lc-5
128
0.0
BatchNorm
PSAGE
0.01
le-5
128
0.2
BatchNorm
ogbn-arxiv
PGAT
1e-3
1e-4
128
0.2
BatchNorm
PSGC
0.02
4e-5
128
0.0
None
PGCN
0.008
2e-5
128
2
0.0
None
PSAGE
0.008
2e-5
128
0.0
None
Reddit
PGAT
0.001
le-5
128
0.0
LayerNorm
PSGC
0.008
2c-5
192
0.0
None
PGCN
0.01
2e-3
128
0.0
None
PSAGE
0.05
le-4
256
2
0.5
None
Cora
PGAT
0.002
le-3
128
0.0
None
PSGC
0.01
1e-3
128
0.0
None
PGCN
0.01
9e-4
128
0.0
None
PSAGE
0.005
5e-4
256
0.2
None
Citescer
PGAT
0.001
le-4
128
0.0
None
PSGC
0.01
5e-4
128
0.0
None
PGCN
0.004
6c-3
150
0.2
None
PSAGE
0.03
2e-4
150
2
0.5
None
Pubmed
PGAT
0.004
Ge-3
150
2
0.2
None
PSGC
0.005
4e-2
128
2
0.2
None
Table 10: Hyperparameters for Different Models and Datasets
Computational Infrastructures. The experiments demonstrate the scalability of our model across different computational setups. Small-scale datasets, including Cora, Pubmed, and Citeseer, were processed an Nvidia A30 GPU with 24 GB memory, while larger datasets like ogbn-arxiv and Reddit were trained and evaluated an Nvidia A100 GPU with 80GB memory for faster training and inference. The system was equipped with an Intel Xeon Platinum CPUs and 512GB of RAM, ensuring efficient exccution of large-scale graph neural network experiments..
In our experimental sctup, we have designated most 1000 epochs for cach run, with 10 runs to ensure statistical significance, more details about hyperparameters for different experi- ments listed in Table 10 We keep the order q and the number of sketches k equal. The sketch-ratio
Setup of PGNN:
28

---

Published in Transactions on Machine Learning Research (01/2026)
of 0.018 used for the citescer is the same mentioned from the paper Ding et al. (022) sketch-ratio of 0.02 against 0.026 is used for the cora dataset to demonstrate the effectiveness of the proposed method. For the Pubmed dataset, we selected lower sketch ratio of 0.01. This choice aligus with the general principle that as graph size increases, the sketch ratio or the effective number of components for preserving variance decreases. Extensive experimentation confirmed that a sketch ratio of 0.01 was sufficient. achieve good classification accuracy. For PGAT, we employ attention masks. The training times of PGNN were not compared with the existing sketcc-based method, Sketch-GNN (Ding et al. 2022), due to observed discrepancies in node classification accuracy from the implementation available in the repository
29


Reviews:
[{"review_id": "rygszSRJHc", "reviewer": "Reviewer_8W9r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces PGNN, a novel sketch-based framework designed to scale Graph Neural Networks to very large graphs by leveraging Proper Orthogonal Decomposition (POD) and Tensor Sketching. PGNN approximates standard GNN message passing by projecting node representations onto a low-dimensional subspace, significantly reducing memory and computational cost. It integrates count-sketch and Tensor Sketch techniques to efficiently compress both node features and convolution matrices, while a learnable projection matrix enhances flexibility beyond the fixed POD subspace. Additionally, PGNN employs a locality-sensitive hashing approach to evaluate node classification losses sublinearly, focusing on nodes with poor predictions. Extensive experiments on benchmark datasets, including massive graphs with over 100 million nodes, demonstrate that PGNN achieves competitive or superior accuracy compared to state-of-the-art methods, while dramatically improving memory efficiency and training speed. The framework is supported by theoretical guarantees for projection optimality and error bounds, providing strong evidence for its effectiveness.\n\nStrengths:\n- Elegant integration of POD and sketching techniques for scalable GNNs.\n- Demonstrated efficiency and accuracy on datasets up to 111 million nodes.\n- Strong theoretical grounding with error bounds (Theorem 2) and optimality proof for POD projection (Theorem 1).\n\nWeaknesses:\n- Limited evaluation on tasks beyond node classification (only brief link prediction experiments).\n- Complexity of the method (multiple sketches, FFT operations, LSH integration) may hinder adoption for practitioners unfamiliar with these techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides comprehensive experimental results across multiple benchmark datasets, including both small citation graphs and large-scale graphs. Comparisons with state-of-the-art methods (GCond, Graph Coarsening, Sketch-GNN, GraphSAINT, VQ-GNN) support the claims regarding accuracy, memory efficiency, and training time. Theoretical guarantees are provided for approximation errors and POD optimality, and empirical studies on error propagation and t-SNE visualization support the method’s robustness.", "audience_yn": "Yes", "audience_explanation": "Scaling GNNs to large graphs is a highly relevant problem in machine learning and graph analytics. Researchers and practitioners working with large graph datasets (e.g., social networks, citation networks, e-commerce recommendation systems) would benefit from PGNN’s contributions in reducing memory and runtime while preserving accuracy.", "requested_changes": "- Clarify preprocessing complexity: Provide a more detailed analysis of preprocessing time with respect to graph size (currently described but not fully quantified for intermediate scales).\n- Broaden evaluation: Consider including additional tasks (e.g., graph classification) or more detailed link prediction experiments to show generality.\n- Simplify exposition: Some sections (e.g., Tensor Sketch-based updates, LSH-based loss evaluation) are mathematically dense and could benefit from clearer illustrations or pseudocode to improve accessibility.\nThese changes are not critical to validate the main claims but would strengthen clarity, generality, and reproducibility.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pfEfA8TNNp", "reviewer": "Reviewer_F3wH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces PGNN, a sketch-based training framework for GNNs built on Proper Orthogonal Decomposition (POD). The key idea is to precompute a low-dimensional subspace (via POD on the augmented feature matrix) and train entirely within this subspace, avoiding the online sketch updates used by prior sketch-based methods. The authors (i) derive PGNN update rules for several architectures (SGC/GCN/SAGE/GAT), (ii) prove POD-optimality for the linearized (SGC) case and provide a per-layer error bound, and (iii) show empirical scalability with materially lower sketch ratios than Sketch-GNN while maintaining competitive accuracy and reduced memory/time. Experiments span Cora/Citeseer/PubMed, OGBN-Arxiv/Products/ Papers100M, and Reddit, with notable results such as r=0.05 on Reddit vs r=0.3 for Sketch-GNN, and a ~339× per-epoch speedup on Papers100M at small sketch ratios.\n\nPros:\n1. Clear algorithmic framing and practical recipe (preprocess → train with fixed subspace → LSH-based selective loss evaluation).\n2. Strong theoretical support: POD optimality for SGC and error propagation bound.\n3. low sketch ratios (e.g., 0.003 on OGBN-Products) with competitive accuracy; substantial memory/time savings on large graphs.\n4. Broad architectural coverage and ablations across datasets; link prediction results included.\n\nCons\n1. Theory mainly covers linearized GNNs; guarantees for nonlinear message passing are indirect.\n2. LSH/triplet-loss machinery introduces extra hyperparameters and complexity; sensitivity is only briefly discussed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s central claims that POD-based subspace training enabling lower sketch ratios and improved efficiency with comparable accuracy are supported by: (1) a clean derivation for the linearized case and an explicit layer-wise error bound; (2) consistent empirical results across small/medium OGB datasets and large-scale Reddit/Products/Papers100M showing accuracy close to or better than Sketch-GNN at significantly smaller r, lower memory, and faster epochs; and (3) concrete runtime/memory tables and figures. Limitations are acknowledged (e.g., accuracy drop on OGBN-Arxiv and the lack of convolution sketch for GAT). While nonlinear-theory coverage is limited, the empirical evidence is methodical and convincing for the stated scope.", "audience_yn": "Yes", "audience_explanation": "Scalable GNN training is a very important topic for Graph foundation model.", "requested_changes": "1. Provide a brief theoretical discussion clarifying why POD optimality does not extend to nonlinear cases, and add an empirical stress test (deeper GCN/GAT) to illustrate error accumulation and the effect of Jumping Knowledge within PGNN.\n2. For key tables (Reddit/Products/Arxiv), clarify hyperparameter tuning and compute budgets for all baselines.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nw9NDewnaQ", "reviewer": "Reviewer_xgkJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**  \nThe paper introduces a sketch-based method using Proper Orthogonal Decomposition (POD) to construct low-dimensional approximations of graph data and convolution operations. These approximations reduce both memory usage and runtime, making them particularly effective for large-scale graph processing.  \n\nKey contributions include:  \n- A POD-based method for low-dimensional graph approximations.  \n- Application to linear GNNs such as SGC.  \n- Extension to handle non-linearities via Tensor sketches.  \n- Solutions to challenges such as:  \n  - Restrictions on the column space of learnable functions.  \n  - \\(O(n)\\) computation cost requirements.  \n- Theoretical guarantees:  \n  - Optimality of POD for linear updates.  \n  - Error bounds for the proposed method.  \n- Experimental validation:  \n  - Strong performance on large-scale transductive graph learning tasks (e.g., OGBN datasets).  \n  - Results on large graph benchmarks and specially on OGBN-Papers100M, with sketch ratios as low as \\(10^{-6}\\).  \n\n**Strengths**  \n1. Addresses an important problem: scaling GNNs to real-world, massive graphs.  \n2. Effectively reduces both memory and runtime costs with minimal accuracy loss.  \n3. Method is well-crafted, considering multiple aspects of the scalability challenge, and grounded in strong theory.  \n4. Provides clear theoretical guarantees.  \n5. Experimental results demonstrate strong performance across multiple datasets.  \n6. Improves over prior sketch-based GNNs, achieving significantly lower sketch ratios (down to \\(10^{-6}\\) on OGBN-Papers100M).", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most of the claims in the paper are well-supported. Below is a breakdown of positive aspects and areas for improvement:\n\n**Positive Points**  \n1. The addressed problem is theoretically well-motivated, and the proposed method provides a convincing approach to the challenges of large-scale graph processing.  \n2. The method is mostly well-explained, with the mathematical formulation making sense overall.  \n3. Theoretical guarantees are adequately addressed, particularly through Theorems 1 and 2.  \n4. Experimental results are generally convincing and demonstrate:  \n   - Minimal performance drop despite substantial improvements in memory and runtime efficiency.  \n   - A strong selection of datasets, with appropriate scale to validate the claims.  \n\n**Areas for Improvement / Clarifications Needed**  \n1. **Clarity of Explanation:** Section 4.1 was difficult to follow. While the mathematics is correct, it would benefit from additional intermediate steps in the derivations. If referencing external resources, please indicate explicitly which theorem or result leads to the final equation.   \n2. **Experimental Results:** While overall convincing, some issues stand out:  \n   - In Table 1 (GAT model on OGBN-Products), why does the lower sketch ratio result in an OOM error?  \n   - In Table 3, SGC and MLP appear as OOM, but results are reported for GCN. For the MLP specifically, batch size can be adjusted to process small sizes of nodes at a time and it should scale to very small memory sizes. Have these points considered in the analysis?\n   - Some improvements seem marginal, e.g., a reduction from 0.026 to 0.02 compared to Sketch-GNN, which may not be statistically significant.", "audience_yn": "Yes", "audience_explanation": "Learning on graphs has become a major subfield within Machine Learning, and the challenge of scaling methods to very large graphs is both well-motivated and of significant practical importance. Addressing this problem has direct implications for real-world applications, where datasets often contain millions or even billions of nodes and edges.  \n\nThe authors’ approach falls within the family of sketch-based methods, which represent a promising direction for improving scalability. Sketching techniques provide a principled way to reduce memory and runtime complexity while preserving key structural information, and this paper leverages that idea in a thoughtful manner. By applying Proper Orthogonal Decomposition (POD) and extending it with Tensor sketches and Local Sensitive Hashing, the work advances the current state of sketch-based graph learning and demonstrates its potential to handle graphs at the scale of OGBN-Papers100M.  \n\nOverall, I believe this paper will be of interest to the audience of TMLR. The combination of theoretical grounding, algorithmic novelty, and empirical validation makes it a valuable contribution to the community, particularly for those interested in scalable graph representation learning.", "requested_changes": "1. Please revise Section 4.1 to provide more detailed explanations and intermediate steps, making the mathematical derivations easier to follow.  \n2. Please expand on the description of the experimental setups, and clarify or adjust the experiments involving the MLP baseline to ensure consistency and completeness.", "broader_impact_concerns": "The paper does not explicitly discuss broader impact. In my view, this work is primarily methodological and focuses on improving the scalability of GNNs through sketch-based techniques. As such, its broader societal impact may be limited. Overall, I do not see significant ethical or societal concerns specific to this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)