Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Authors: Rihong Qiu
Xinke Jiang
Yuchen Fang
Hongbin Lai
Hao Miao
Xu Chu
Junfeng Zhao
Yasha Wang
Venue: ICML 2025 poster (2025)
Keywords: Graph Neural Tangent Kernel
Graph Neural Networks
Graph Continual Learning
Primary area: deep_learning->graph_neural_networks

Abstract: Graph Neural Networks (GNNs) have emerged as a fundamental tool for modeling complex graph structures across diverse applications.
However, directly applying pretrained GNNs to varied downstream tasks without fine-tuning-based continual learning remains challenging, as this approach incurs high computational costs and hinders the development of Large Graph Models (LGMs).
In this paper, we investigate an efficient and generalizable dataset distillation framework for Graph Continual Learning  (GCL) across multiple downstream tasks, implemented through a novel Lightweight Graph Neural Tangent Kernel (LIGHTGNTK).
Specifically, LIGHTGNTK employs a low-rank approximation of the Laplacian matrix via Bernoulli sampling and linear association within the GNTK. This design enables efficient capture of both structural and feature relationships while supporting gradient-based dataset distillation.
Additionally, LIGHTGNTK incorporates a unified subgraph anchoring strategy, allowing it to handle graph-level, node-level, and edge-level tasks under diverse input structures.
Comprehensive experiments on several datasets show that LIGHTGNTK achieves state-of-the-art performance in GCL scenarios, promoting the development of adaptive and scalable LGMs.

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Rihong Qiu 12 Xinke Jiang 12 Yuchen Fang" Hongbin Lai 12
Hao Miao *
Xu Chu 1256
Junfeng Zhao 127
Yasha Wang 286
Abstract
1. Introduction
Graph Neural Networks (GNNs) have emerged as a fundamental tool for modeling complex graph structures across diverse applications. How- ever, directly applying pretrained GNNs var- ied downstream tasks without tie-ttnning-bsee continual learning remains challenging, as this approach incurs high computational costs and hinders the development of Large Graph Models (LGMs). In this paper, we investigate an efficient and generalizable dataset distillation framework for Graph Continual Learning (GCL) across mul- tiple downstream tasks, implemented through novel Lightweight Graph Neural Tangent Kernel (LiGHTGNTK). Specifically. LIGHTGNTK em- ploys a low-rank approximation of the Laplacian matrix via Bernoulli sampling and linear associa- tion within the GNTK. This design enables effi- cient capture of both structural and feature rela- tionships while supporting saaseennt-bsedd dataset distillation. Additionally, LIGHTGNTK incorpo- rates a unified subgraph anchoring strategy, al- lowing it to handle graph-level, node-level, and edge-level tasks under diverse input structures. Comprehensive experiments on several datasets show that LIGHTGNTK achieves state-of-the- thrttrrrrt performance in GCL scenarios, promoting the development of adaptive and scalable LGMs.
Graph Neural Networks (GNNs) (Cai et al.. 2018: Kipf Welling, 2017; Veličković et al., 2018; Zhang al., 2025) have made significant advancements in recent years, estab- lishing themselves as powerful tools for modeling complex graph structures across diverse domains. From social net- works (Matsugu et al., 2023; Li et al., 2022c; Qin et al., 2022) to biochemistry (Duvenaud et al., 2015; Yang et al., 2023; Xu et al., 2023a), and extending to transportation systems (Gao et al., 2023a;b; Jiang et al., 2023b; Fang et al., 2023; 2024; Duan et al., 2024b) and financial networks (Li et al., 2022a; Huang et al., 2022; Liu et al., 2021; Duan et al., 2024a; Jiang et al., 2025), GNNs have proven their adapt- ability and effectiveness a wide range of applications. Their exceptional versatility is evident in their ability to cap- ture relational dependencies effectively, thereby facilitating data-driven decision-making processes.
Recently, Large Graph Models (LGMs) or Pretrained Graph Models (PLM) (Xia et al., 2024: Hu et al., 2020b; Li et al., 2024b; Ma et al., 2024; Zhang et al., 2023; Wang et al., 2024b; Jin et al., 2024; Jiang et al., 2024; Liu et al., 2023; Yu et al., 2024; 2025) have achieved breakthrough advance- ments by emulating the success of Large Language Models (LLMs) (OpenAI, 2023; Touvron et al., 2023), demonstrat- ing powerful graph learning capabilities through large-scale graph data pretraining. Inspired by the paradigm of LLMs, LGMs such as GPT-GNN (Hu et al., 2020b), GPT-ST (Li et al., 2024b), and GPT-Het (Ma et al., 2024) have shown remarkable potential in capturing complex structural depen- dencies across diverse graph domains.
'Equal contribution 'School of Computer Science, Peking University, Beijing, China Keyy Laboratory of High Confidence Software Technologies, Ministry of Education, Beijing. China "University of Electronic Science and Technology of China Departmenn of Computer Science, Aalborg University 'Center on Frontiers of Computing Studies, Peking University, Bei- jing, China "Peking University Information Technology Institute, Tianjin Binhai, China 'Bg Data Technology Research Center, Nanhu Laboratory, Jiaxing, China 'National Engineering Research Center For Software Engineering. Peking University, Beijing, China. Correspondence to: Junfeng Zhao, Yasha Wang <Ri- hongQiu @ stt.,tuu.uuu.... wtw.pku.edu.cn>.
Despite the remarkable capabilitiess of L.MMs in modeling complex graph relationships, they are hindered by signifi- cant limitations. One critical issue is that these models lack inherent Emergent Abilities (Zhang et al., 2023), which stem from the intricate nature graph structures and the variability across different domains. As a result, LGMs require extensive and computationally intensive fine-tuning for downstream tasks through Graph Continual Learning (GCL). This dependence on fine-tuning presents a crucial challenge for practical deployment: How can we develop strategies to effectively distill task specifcc training corpora
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
that enable efficient and effective fine-tuning of GNNs, ulti- mately enhancing their adaptability?
ple benchmark datasets and tasks, particularly in continual learning scenarios.
Specifically, several challenges like C1 and C2 still remain unaddressed in the pursuit of the distilled small, symthetic, yet informative datasets, where GNNs trained on them can achieve competitive performance compared to those fine-
2. Related Work
2.1. Neural Tangent Kernel
tuned on large-scale graph datasets:
The Neural Tangent Kernel (NTK) has emerged as a pivotal theoretical tool in understanding the training dynamics of neural networks (Jacot et al., 2020; Neal & Neal, 1996; Yang, 2019; Ren & Sutherland, 2025), with wide range of application including neural architecture search (Park et al., 2020; Chen et al., 2022), meta-learning (Zhou et al., 2021), active learning (Holzmüller et al., 2023), and dataset distillation (Nguyen et al., 2021; Guo et al., 2024; Xu et al., 2023b). Originating from the concept of mapping infinitely wide neural networks to kernel methods, NTK simplifies the intricate training process of deep neural networks into kemnel ridge regression (Jacot et al., 2020; Lee et al., 2020; Yang & Littwin, 2021).
1 CI. Downstream tasks frequently involve exceedingly large datasets with substantial noise, which results in ineffec- tive fine-tuning and considerable computational overhead. The cost of using the whole large datasets in fine-tuning often rivals that of retraining an entire LGM, rendering such methods impractical for widespread applications.
@ C2. The transfer of pretrained LGMs to different down- stream tasks poses significant challenges due structural mismatches. Specifically, current dataset distillation based GCL methods are limited to graph-level tasks (Xu et al., 2023b) and fail to generalize to tasks at the node level and edge level due to inconsistent input structures.
In the realm of graph data, the Graph Neural Tangent Ker- nel (GNTK) extends the NTK framework to capture the relational and structural information inherent in graphs (Du et al., 2019; Xu et al., 2023b; Wang et al., 2024a). GNTK facilitates theoretical analysis and practical applications in graph-basen based learning tasks, and has been successfully ap- plied to various graph-related tasks.
To address the aforementioned challenges, we in- troduce Lightweight Graph Neural Tangent Kernel (LIGHTGNTK), novel dataset distillation framework tai- lored for effective GCL. Our method primarily focuses on distilling essential knowledge from large-scale graph datasets into compact, synthetic datasets via the Graph Neu- ral Tangent Kernel (GNTK), which effectively captures both structural and feature relationships through the training gra- dients. To maintain the efficiency of conventional GNTK while minimizing computational overhead, LIGHTGNTK uti- lizes a low-rank approximation of the full Laplacian matrix derived via Bernoulli sampling on the eigenvalue spectrum. Theoretical guarantees support the quality of this approx- imation, and its computation is further accelerated by es- tablishing a linear association with the sampled spectrum. Additionally, LIGHTGNTK incorporates a subgraph anchor- ing strategy to graph input structures across graph-, node-, and edge-level tasks, facilitating seamless adaptation to variety of downstream applications In general, our contribu- tions are summarized as follows.
However, the practical adoption of NTK in large-scale learn- ing tasks is hindered by its super-quadratic runtime com- plexity (Arora et al., 2019; Li et al., 2019; 2024a). To address this challenge, numerous studies have focused on accelerating NTK computations, such as using Monte Carlo method (Novak et al., 2018; Arora et al., 2019), low-rank ap- proximations to kernel matrices (Alaoui & Mahoney, 2015; Avron et al., 2017: Zandieh et al., 2020), and Kronecker Sketching methods (Jiang et al., 2022). But these meth- ods primarily target NTK in the context of standard neural networks and do not consider the unique characteristics of graphs Building these foundations, our work in- troduces novel enhancements and unified definitions to improve computational efficiency and adapt GNTK to practical scenarios, creating a more scalable and versatile framework for graph-based problems.
A unified dataset distillation framework for LGMs in GCL scenarios is proposed. By anchoring tasks to subgraph structures, LIGHTGNTK demonstrates adaptability to diverse downstream tasks, including node-level, edge- level, and graph-level applications.
2.2. Dataset Distillation for Graph Continual Learning
Dataset distillation aims to synthesize a small but in- formative dataset to summarize the original large-scale datasets (Yu et al., 2023; Lei & Tao, 2023; Ding et al., 2024; Zhu et al., 2025), thereby expediting model training for con- tinual learning, removing noise samples in the full datasets, and reducing computational overhead. Traditional methods like core set selection focus on identifying representative samples from original data (Goodman & O' O'Rourke, 1997).
To address the computational complexity of GNTK, a Lightweight GNTK (LiGHTGNTK)) is introduced. This framework balances structural and feature representation fidelity while minimizing dataset distillation overhead.
Comprehensive experimental results demonstrate that LIGHTGNTK outperforms baseline methods across multi-
2

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
which often fall short in capturing the complex high-order dependencies inherent in structured data. In contrast, recent advancements in meta-learning have introduced methods like gradient matching (Zhao et al., 2021) and trajectory matching (Miao et al., 2024; Zhu et al., 2025) in dataset distillation, which aligns distilled datasets and full datasets in the perspective of gradient or trajectory.
A being diagonal matrix of eigenvalues.
In the standard spatial (message-passing) formulation, the hidden representation at the (1 + 1)-th layer, denoted as H( is computed as
= (LHHOO))
(I)
For graph structural data, emerging approaches focus on preserving structural and spectral properties while achiev- ing efficient dataset distillation. Early methods address structural preservation by leveraging self-expressive models for graph reconstruction (Liu et al., 2024b) or maintain- ing consistent receptive field distributions to retain local neighborhood structures (Liu et al., 2022). Complemen- tary to these, spectral-based approaches like GDEM (Liu et al., 2024a) align graph eigenbases preserve spectral characteristics. In terms of optimization strategies, (Gupta et al., 2024) proposes gradient-fre frree distillation for node classification, while GCOND (Jin et al., 2022b) improves efficiency through gradient based optimization.
where H E Rxxd denotes the hidden representation at layer I, H = X, and W are learnable parameters. at)) denotes an activation function (Jiang et al., 2023a; Kipf & Welling, 2017).
3.2. Revisiting eNTK & GNTK
eNTK The empirical Neural Tangent Kernel (cNTK), de- noted by (2(2), quantifies the similarity between two data points based on their gradient-space representation (Ja- cot et al., 2020; Mohamadi et al., 2022). Formally, for a neural network function fo(-) parameterized by 0, the eNTK is defined as follows:
recently emerging direction leverages the Graph Neural Tangent Kernel (GNTK) (Guo al., 2024; Xu et al., 2023b) to distill datasets by focusing on model dynamics learning. However, these methods encounter significant challenges, including low computational efficiency, insufficient utiliza- tion of graph structural information, and lack of scalability to graph tasks at various levels.
00(21, = [v((f(<z)) (V(((())))) (2)
where Vo(Jo(z)) represents the Jacobian of fo(x) with respect to the flattened parameter vector 0 E R", cap- tures the rates of change of each output dimension of the function with respect to each input parameter. Specifi- cally, if the model outputs O-dimensional vectors, then Ve(fe(z)) E ROxP, meaning that it has O rows (one for each output dimension) and P columns (one for each pa- rameter). This structure leads to (g(21, E ROx0 re- flecting the interactions between the gradients of the output dimensions for the two input data points. When considering two groups of data points of sizes N1 and N2, this results in N₁ N2 such matrices. Stacking these O x matrices yields strucurtd kernel of size (N:0) (N2O), capturing pairwise gradient correlations across the dataset.
3. Preliminaries
The Neural Tangent Kernel (NTK) describes the training dynamics of infinitely wide neural networks; however, its computation is generally intractable in practical applications. To address this limitation, the empirical Neural Tangent Kernel (eNTK) (Jacot et al., 2020) serves as a practical surrogate, facilitating empirical analyses of neural training dynamics. In Section 3.1, the definition of GNNs is first revisited. Then review the computation of eNTK and its extension to Graph Neural Tangent Kernel (GNTK) for GNNs in Section 3.2. Additionally, we formally define the problem of graph dataset distillation in Section 3.3.
GNTK For graph-structured data, the Graph Neural Tan- gent Kernel (GNTK) extends this concept by analyzing the gradient of fo( fo (L, X), where fo(-) denotes graph neural network (GNN). Here, G represents the graph sample, L is the graph Laplacian, and X is the node feature matrix. The GNTK between two graphs Gi and G2 is defined as:
3.1. Revisiting Graph Neural Networks
We adopt the messagge-pasiing framework (Kipf & Welling, 2017) for graph neural networks, which is informed by spec- tral graph theory (Shuman et al., 2013). Consider a graph with an adjacency matrix A E Raxx and node features X E gaxd with node number n and feature dimension d. The Laplacian matrix is defined as L = 1 A/D AD-1/2 1/2 (Zhang et al., 2024; Li et al., 2022b), where Ā = A+ 7 and î is the degree matrix. The Laplacian matrix L can be diagonalized as U,,,, with U representing the eigenvector matrix and
(g =
(3)
However, traditional spatial- domain methods for computing the GNTK involve evaluating gradients at the graph level, which necessitates computationally intensive operations that scale quadratically with the number of nodes n (Shuman et al., 2013). Consequently, calculating the GNTK becomes prohibitive for large-scale graphs.
3

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Unified Graph Task
Pretraining LGM
2
LightGNTK Calucation
Downstream Training Graphs Set
Downstream Validation
Node Type?
Node Level
Graphs Set
L.MM
Edge Level
Graph Type?
Graph Level
Tol
TO
Edge Link?
Graph Corpus
Node
Prob
Task
Sim?
Orthogonally Decompose
Prototype Node
Data Selection & Finetuning
L = - ĀD L=
Sample
Graph Task
Prob.
& ~ Bernoulli(p)
Sim?
LSFT
of log(o,) + 0l Suprivised Finetuning
Gradients
X
Prototype Graph
VolcÜAO X)
Graph Features
Edge Task
Prob
Sim?
iode
Node
Training Graph
Ntrain
Learnable
Frozen
LGM
Gradient Base
Node
Product
Selected Graphs
GNTK
Center Node
Edge
Similarity Matrix K
Nyat
TopK
Validation Graph
Features
Gradient
Score
Gradient Base
Figure 1.
The overall framework of LIGHTGNTK. Based on the unified definition proposed in Section 4.1, 1, we represent tasks at graph, node, and edge levels uniformly using subgraphs. Given graph corpus, we pretrain the LGM in unsupervised manner, utilizing data from node, edge, and graph levels. @ Next, we calculate the GNTK similarity matrix K by measuring gradient dot products between training and validation graphs. accelerate gradient computation. we employ low-rank approximation the graph Laplacian matrix via Bernoulli sampling, @ Finally, data selection is performed by identifying the Nay most informative samples according to the GNTK matrix. These selected samples are then used fine-tune the LGM to improve performance on downstream tasks.
3.3. Graph Dataset Distillation
4. Method
Dataset distillation aims to condense a large dataset into a significantly smaller synthetic subset while preserving the essential information necessary for downstream tasks. In this work, we focus addressing the problem of graph dataset distillation for classification tasks. Formally, given target training set Draii {G,, ).) New our goal is to create a smaller synthetic set Daya = {(G,, m where Nayn & Nurin such that a GNN trained on Daya maintains performance comparable one trained on D Con- cretely, the distillation process proceeds follows: (1)
In this section, we introduce LIGHTGNTK, novel frame- work designed to efficiently and effectively distill graph datasets for Graph Continual Learning (GCL), as illustrated in Figure First, as described Section 4.1, we propose unified task definition that integrates tasks at the node, graph, and edge levels within subgraph-centric framework. This approach facilitates the extraction of task-specifie sub- sets from large-scale training data, ensuring that these sam- ples are well-suited for a variety of downstream tasks, e Next, in Section 4.2, we provide a theoretical analysis of the GNTK and introduce an efficient spectral- -omminn approxi- mation, LIGHTGNTK, which significantly reduces both time and space complexity compared to standard GNTK eval- uations. We also demonstrate that method yields an approximation to the GNTK with tight approximation er- ror upper bound. @ Finally, in Section 4.3.2, we describe the pretraining and fine-tuning paradigm of the LGM through an unsupervised learning task. For improved readability, the
A distillation procedure generates Deya from Dtrain- (2) A
downstream classifier is trained on Dayn and evaluated on a test set Dres. which is disjoint from both Dinin and Dsyn-
The evaluation on Diest reflects how effectively Dayn retains the knowledge contained Pirain-

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
main notations are summarized in Table of Appendix A.
tions indicate that retaining only a subset of eigenval- ues and their associated eigenvectors is sufficient to accurately approximate GNN gradients (Bianchi et al., 2020; Deng et al., 2022), thereby eliminating the need for full-rank Laplacian matrices in GNTK computation. To achieve this low-rank approximation, we perform Bernoulli sampling on the eigenvalues obtained from the spectral decomposition of Laplacian matrix, effectively balancing high-frequence and low- fow-freuuency components while pre- serving both local and global graph structures. Formally, given a graph Laplacian I, = UAUT, where the eigen- value matrix is A = diag(A1,... An) and the eigenvec- tor matrix is U = [ui, U2..... unl. we define the ex- pected rank r and sample probability p = -. For each eigenvalue X, we sample & Bernoullipp and form A diag(Ei>i,.... Endn) E Brxx , 1. Eigenvectors in Ü R with nonzero &, are retained accordingly.
4.1. Unified Task Definition
Let graph sample set G (91..... [.m] be a set of graphs. Each graph G. = (V,, Er; Xi, A., Y:) is defined as follows: V, is the set of nodes, with [V;] n;; E, C V, x V, is the set of edges; X E 1a denotes the node feature matrix; A, E R is the adjacency matrix; Y, denotes the labels associated with nodes, edges, or the entire graph.
In this paper, we propose a subgraph-centric framework for the unified handling of tasks at the node, edge, and graph levels. Given target (node, edge, or graph), we construct query subgraph go by including the target and its neighbors within k hops. We then compare this query subgraph to pro- totype subgraphs from a support set Dayn using similarity function siim sim((:.). .). O For Node-Level and Graph-Level Tasks. Let C' denote the set of class labels, and assume y1 E C is the label for a node or a graph instance. After extracting the query subgraph Go for the target node or graph, we compute the output logits o for the query. The predicted label is given by yı- arg max.ec s sim(o,, ōe), where the prototype logits o, for class can be selected as the average logits of subgraph instances in the support set Day,, for class c: o. = [D_] (gg,,,)) Oj. or as a one-hot vector for class c, i.e., 6c|c] = and zeros elsewhere. @ For Edge-Level Tasks. Given a candidate pair (v) 11)) for edge prediction, subgraphs Gi and Ge centered at fy and 1n are constructed, yielding logits og and 0g. An edge is predicted between v; and Un if there exists any neighbor by of node i such that sim(O,, 0g) 2 Av, (sim(o,, 04)) + e, where 2 0 is margin hyperparameter. Notably, this criterion can also be applied symmetrically by considering the neighborhood of bn in place of U,.
Next, to compute Żx- = ÜÄÜT x efficiently, we decom- pose the computation into three sequential steps: 1 Z = ÜT X, where Z E Rrxd, cost: O(nrd); 2 Y = A where Y E Rexd, cost: O(rad); @ L.X UY: cost: ((rr)). Thus, the overall computational complexity is (nrd) O(r²d) +(nrd) O(nrd), where < n. This significantly reduces the complexity compared to the naive approach of O(n²d), which is conceptually related to linear attention (Katharopoulos et al., 2020). The resulting low-rank Laplacian L = ÜÄÜ preserves essential struc- tural information, and the low-rank approximation of GNTK is given by:
(G,,
[o.oCU..A.O, T x.] [Vofolū,AA0, xx)], (4)
@ Structural Optimizations Furthermore, we implement three additional acceleration strategies: (1) For node-level and edge-level tasks, instead of performing eigenvalue de- composition of the Laplacian matrix I. for the entire graph, we conduct the decomposition within subgraphs, thereby reducing computational complexity due to the smaller ma- trix sizes. (2) For extremely large graph-level tasks, we adopt batch acceleration technique based on Monte Carlo sampling (see Appendix E) to enhance efficiency. (3) When collecting gradients, we utilize only the gradients from the final layer rather than from all layers, achieving a balance between speed and accuracy.
4.2. LIGHTGNTK: A Lightweight Approximation of Graph Neural Tangent Kernel
4.2. 1. LIGHTGNTK DEFINITION
To address the considerable computational overhead asso- ciated with conventional GNTK, we introduce a series of techniques aimed at significantly enhancing its computa- tional efficiency.
I Low-Rank Optimization Building on critical insights from spectral graph theory (Von I Luxburg, 2007), we tackle the high computational cost of GNTK by proposing a low- rank approximation of the graph Laplacian matrix L. In the spectral domain, each eigenvalue of the Laplacian en- capsulates unique topological information. Importantly, we observe that both theoretical and empirical observa-
4.2.2. APPROXIMATION QUALITY OF LIGHTGNTK
We now formalize the relationship between the gradients obtained with the Bemoulli-sampled Laplacian L and those obtained with the full-rank Laplacian L. Throughout, we assume that the GNN fe(-) is L smooth with respect to its input as defined in (Heinonen, 2005).
In graph-level tasks, the entire graph is treated as one subgraph, and a virtual node is attached to nodes.
5

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Theorem 4.1 (Gradient Approximation Error). Let X E Raxd be the node-feature matrix and let I. = UAUT be the graph Laplacian with eigen-pairs {(A,, (ak,u))) uz)) Construct L via independent Bemoulli sampling with probability p on every eigenvalue, ie. I. = ÜÂÜ with As = Eidi, & m Bernoulli(p) Then the expect of gradient approximation error is:
through the lens of the GNTK, as detailed in Appendix C. The analysis demonstrates that the change in test sample logits induced by training step Go is proportional to the GNTK value @(G.. (o) between the training and test samples, quantitatively capturing the transfer of gradient in- formation across the train-test split (Guo et al., 2024). Con- sequently, filtering or reweighting training samples based on GNTK values with respect to the test set can effectively reduce undue train-test coupling and improve the general- ization performance of dataset distillation methods (Guo et al., 2024).
E
Volo(L
(5)
$ LE - p)
Σ x2 1u, E
4.3.2. INSTANTIATION PIPELINE
where Lv the Lipschitz smoothness coefficient. The full proof can be found in Appendix F.2.
Our practical pipeline consists of three main steps. First, we pretrain the LGM using an unsupervised objective to obtain robust graph representations. Next, for each candidate graph Ge in the training set, we evaluate its quality according to the framework of LIGHTGNTK, and select the top Nayn query subgraphs to construct the synthetic set D Dayn- Finally, we use Daya to train the downstream classifier and evaluate its predictive performance.
Theorem 4.2 (LIGHTGNTK Approximation Error Upper Bound). Let 9n and G2 denote two graphs with correspond- ing node feature matrices X1 and X2. The approxima- tion error of GNTK computed using the Bernoulli-sappled Laplacian I. can be bounded from below as follows:
LLIGHTGNTK (91, (
S L7 (C142 + C2A1),
where A (1 EL= X² [|u XI2 denotes the up-
O Pretrain LGM We pretrain LGM in an unsupervised fashion to jointly learn node representations and graph structural patterns. Following the pretraining paradigm of masked signal reconstruction and link prediction (as in GPT- GNN (Hu et al., 2020b)), we randomly mask a subset of node features and edges, and train the model to reconstruct the masked information on large-scale unlabeled graphs. This allows LGM to learn more expressive representations for nodes and structures, which facilitates effective transfer to various downstream graph tasks.
per bound on the approximation error, as derived from the
gradient error analysis. Here, C= = o (LX)] is con- stant that depends only the network & and graph data G. Therefore, as the sampling probability p increases, the ap- proximation error of GNTK decreases. The complete proof is provided in Appendix F.3.
Time Complexity Analysis Compared with the conven- tional GNTK, which requires dn²dd) time due to fulk-rank Laplacian operations, our low-rank optimization reduces the time complexity to (nrd), where r & n is the chosen rank. This substantial reduction is achieved by only retain- ing r spectral components, allowing all Laplacian-related computations to be executed efficiently with much smaller matrices. As a result, our approach scales to larger graphs and offers significant computational savings over traditional GNTK methods without sacrificing accuracy.
@ LIGHTGNTK Calculation The theoretical analysis in Section 4.2 demonstrates that performance on downstream tasks can be improved by fine-tuning LGM using training samples that are similar to those from the test set. In practice, however, test data are not available during training process. Under the assumption that the validation and test sets are independently drawn from the same distribution, we use the validation set to guide the selection of relevant training samples. Specifically, we make the following assumption.
4.3. Graph Dataset Distillation Instantiation
Assumption. Validation- Test Distribution Match In real- world deployment scenarios, we assume that both the valida- tion set Dvat and the test set Drest are independently sampled from the same target distribution Hyaget. such that:
This section first presents a theoretical analysis of the rela- tionship between training and test samples, which forms the basis for the proposed graph data sampling criteria. Then, the corresponding graph distillation and model train- ing pipeline is described in detail.
D = EIl Iunga::
(6)
1. THEORETICAL ANALYSIS OF THE RELATIONSHIP BETWEEN TRAIN& & TEST DATA
where HD. denotes the underlying data distribution from which dataset D. is sampled.
We theoretically analyze the influence of training samples gradients G.- = (T., X.} on test samples G., = (T.a X,,
This alignment is a standard practice in machine learning evaluation and deployment (Recht et al., 2019; Wang et al.,
6

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
2022; Lohr, 2021; Jiang & Wang. 2023), which ensures that validation metries reliably predict test performance during dataset distillation. To further substantiate the distribution match assumption, we conduct quantitative analyses in Ap- pendix D.4. These results provide strong empirical evidence that the validation and test sets are statistically similar.
5. Experiment
To evaluate the effectiveness of LIGHTGNTK in graph dataset distillation, we conduct extensive experiments on 13 datasets, including tasks at node, edge, and graph levels, and compare our method with state-of-the-ard baselines. The ex- periments aim to address the following research questions:
Based on the assumption, we leverage the validation set as a proxy for training sample selection. To measure the similar- ity between the gradients of training and validation samples, we utilize the GNTK, as introduced in Section 4.2. Specifi- cally, we compute cross-set kernel matrix K E R Noina x Not where Maai and Nval denote the sizes of the training and validation sets, respectively. Each entry Ki = 0g(91. G2) quantifies the gradient covariance between training graph G; and validation graph Gj under GNN parameters 0. Further- more, motivated by the theoretical analysis in Section 4.2, we adopt a computationally efficient variant of GNTK by extracting gradients from a specific layer and computing lightweight similarity measure.
RQ1: How does LIGHTGNTK compare with state-of-the- art methods in graph dataset distillation for GCL tasks at node, edge, and graph levels?
Does LIGITTGNTK achieve improved efficiency compared to traditional GNTK?
RQ2:
What are the contributions of the key components towards improving LIGHTGNTK's overall performance?
RQ3:
Further details and experimental results are provided in Appendix D.
@ Data Selection & Supervised Finetuning We define the LIGHTTGNT--SON -core I(G.) for each candidate graph Ge E Daain as the minimum gradient similarity between Ge and all graphs in the validation set, that is,
5.1. Experimental Setup
Datasets. We conduct comprehensive evaluation using total of 13 datasets across three types of graph learn- ing tasks. For graph classification. we utilize 7 datasets: NCII, NC1109, PROTEINS, and DD from TUDataset (Mor- ris et al., 2020), as well as ogbg-molhiv, ogbg babg-molbbbp, and ogbg-molbace from Open Graph Benchmarks (OGB) (Hu et al., 2020a). For node classification, we employ citation network datasets: Cora, CiteSeer, and PubMed from the Planetoid Dataset (Yang et al., 2016), along with ogbn-arxiv from OGB. For link prediction, we include 2 datasets: oghl- collab and ogbl-ddi, both from OGB. Further details for all datasets are provided in Table D.I of Appendix D.I.
I(G.)
min O(G. G.). GoEDial
(7)
Based on the computed LIGHTGNTK- the synthetic set Dyn by selecting those candidate graphs with the lowest LIGHTGNTK scores, as defined in Eq. (8):
construct
Daya = (Ge E Duaann I(Gc) E MinnNNym((Z((G)))
(8)
where Min-Neya(+) indicates selecting the Neyn lowest ele- ments from the given set.
Next, we utilize the selected samples to perform super- vised fine-tuning of the pretrained LGM. Let = denote the output from the last layer of the LGM. The predicted class probabilities are computed via the softmax function: ÿ Softmax(22), where 6N = exp(ox)/ _ exp(z,) for class k. To support different prediction tasks (edge, node, and graph levels) centered on a subgraph's 's node, we adopt unified loss function:
Methods and Baselines. Our experimental framework compares LIGHTGNTK with the SOTA dataset distillation methods for graph. We evaluate two versions of our pro- posed framework: (i) GNTK, which directly leverages the Laplacian matrix for feature selection without any optimiza- tion; and (ii) the lightweight variant, LIGITGNTK,, which in- corporates efficiency optimizations. To ensure comprehen- sive evaluation, we extend our baseline analysis to encom- pass three distinct categories: 1 Coreset selection methods, including Random Sampling, Herding (Welling, 2009), and K-Center Greedy Selection (Farahani & Hekmatfar, 2009; Sener & Savarese, 2017); Learning-based graph distilla- tion, represented by DosCond (Jin et al., 2022a); O Kernel appreximation techniques, exemplified by KIDD (Xu et al., 2023b), which uses ridge regression for graph-level tasks to approximate GNTK computations. Detailed implemen- tation specifics and theoretical foundations of all baseline methods are systematically documented in Appendix D.3.
N
CSFT
Σ
of lo: + 11103
where N is the number of training samples, [O] is the num- ber of output classes, o; represents the logits assigned by the model to the correct class for sample i, and o, denotes the ground-truth logits for class i. The second term is an L2 penalty (weight decay) applied to the learnable parame- ters of the GNN, scaled by hyperparameter 17 to prevent overfitting.
7

---

Efficient
Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Table 1. Performance comparison (mcan-tstd) on graph classification tasks.
The hest and second-hest results bolded and underlined.
Name
Graphs/CI
Ratio
Random
Herding
K-Center
DosCond
KIDD
GNTK
LIGHTGNTK
Whole Dataset
NCI1 (ACC)
0.06%
$7.4=3.0
59.243..0
59.2=3.0
$7.110.99
60.1
59,410.4
62.017711
0.61%
59.9=2.0
62.8+0.9
59.1=08
60.8+0.9
61.7+1.
64.8+0.2
64,4+33.
80.0=111
50
3.04%
60.5=2.1
62.542.0
59,5=0.5
62.74:.8
64.2=0.6
64.5+0044
66,422400
NCT109 (ACC)
0.06%
54.3=2.3
51.74:0.9
51.7=09
54.9223.
55.2+2.7
57.5
55,3433.7
10
0.61%
61.9=16
63.64-0.3
52.9=1.7
61.4::15
63.5=0.5
64.91-0.1
64,2444.3
77.7=0.6
50
3.03%
64.0=1.4
64.7+112
55.0=2.1
62,9:11.6
70.4
66.8+22.7
65.6:22.5
PROTEINS (ACC)
0.22%
1..8
67.611177
6.76=1.7
63.4:119
6..8
69.3:16.4
68.11:1.2
10
2.25%
67.2=0.7
68.3+10
71.4333
71.7404
1.
73.6+:..8
74.7:15.5
78.622.6
50
11.24%
69.6=4.0
70.1+1.0
72.9=2.6
73,240..8
75.0
74.0+3399
75.9:13.2
DD (ACC)
0.21%
61,3+8.5
60.74.8.4
61.0=3.2
63.040.7
65.8+117
68.61-0.4
65.8:155.4
2.12%
66.8=2.1
67.410.7
66.2-244
68.1
1..6
71.4+0.1
71.51:1.7
76.9#3.2
10.62%
71.4-22
71.6++19
72.3+1.0
70.9++100
73.1+2.22
73.3+:0.4
72.9:16.55
oghg malhiv (ROC-AUC)
0.01%
55.5+3.6
63,34-2.5
63.342.5
61.24-255
63.3+116
64.8+-055
64.1477.2
0.06%
17
62.1::0.9
63.0=133
64.7:3.1
67.515,4
69.2::0.2
68.5188.7
75.0=0.7
50
0.30%
62.3=1.4
61.6
62.8=1.2
62.0+t1.8
7..8=6.2
67.4+0.2
69.34-8.3
mghg-molhbbb (ROC-AUC)
0.12%
57.9+2.0
62.8+-1.2
6288+1.1
58.4433.
62.8+1.1
64.4+-0.1
64.04-4.2
10
1.23%
55,6=0.0
62.5+:0.0
59.6=1.6
62.1
1,54
63.8::0.1
64.61:6.1
6..0
50
6.13%
61.0=0.7
63.0++.77
59,5=1.5
6..8+112
66.2+330
64.2+00.1
68.2++0.6
ogbg-molbace (ROC-AUC)
0.17%
63.8=0.9
54.6133.8
54.6=3.8
66.7::2.1
1..3
68.2:00.2
69.4117.1
10
1.65%
6.79=117
56.1
65.8=5.8
69.4:114
7..8:2.0
71.2::0.5
70.8:199.6
72.721.7
50
8.26%
65.5=2.0
70.3+11.
66,2=1.3
71.0:10.6
76.6+112
76.6+00.5
76.8:16.2
Settings and Evaluation. For the graph classification tasks in TUDataset, we split each dataset into 80%, 10%, and 10% for training. validation, and test sets, respectively, and use accuracy (ACC) as the evaluation metric. For the Planetoid and OGB datasets, we follow the official splits and corresponding evaluation metrics. The detailed evaluation metrics used in the experiments are provided in in Table 6 at Appendix D.2. Throughout all experiments, we use Graph Isomorphism Network (GIN) (Xu et al., 2019) as the GNN backbone for our LGMs. To comprehensively assess the effectiveness of various distillation methods, We assess clas- sification performance at the graph, node, and edge levels by distilling datasets in which each class contains 1, 10, or 50 samples at the corresponding level.
Outperforming Existing Methods. The proposed dataset distillation method LIGHTGNTK demonstrates superior per- formance compared to almost all baseline methods across various tasks, including graph, node, and edge levels. No- tably, LIGHTGNTK can surpass baselines most datasets, highlighting its ability to reduce data dependency while maintaining high predictive accuracy. These results clearly demonstrate that our method provides a more effective and reliable solution for efficient graph dataset distillation.
Effective Learning from Limited Data We observe that the model under dataset distillation method LIGHT- GNTK achieves excellent performance across all three lev- els, even when trained with few one sample per class. This highlights the effectiveness and representativeenss of the selected training samples, which are capable of captur- ing the essential characteristics of each class. Remarkably, even with less than 10% of the original training data, the model demonstrates performance that is nearly comparable to the results obtained when using the entire dataset. This suggests that LIGITGNTK is highly efficient in sampling and leveraging the available data, and its ability to gener- alize effectively with limited information underscores the robustness of the distillation method.
Reproducibility. To ensure reproducibility, we optimize the parameters of baseline models using the Adam Opti- mizer with Ly regularization. During the pretrain process of the GNN backbone, a binary classification task is employed to distinguish between positive and negative edge samples on graphs. For the downstream tasks, classification head is added above GNN backbone to predict. Furthermore, An early-stopping strategy with a patience of 50 epochs is employed to mitigate overfitting.
5.2. Dataset Distillation Results (RQ1)
5.3. Computational Efficiency Analysis (RQ2)
The evaluation results for the graph-level and node-level tasks are summarized in Table I and Table 2 respectively, while edge-level results are reported in Table 8 in Ap- pendix D.5. Based on these experiments, we identify the following key findings:
In addition to evaluating predictive accuracy, it is crucial to assess the computational efficiency of our approach. LIGHTGNTK is designed to significantly improve efficiency by leveraging a theoretically principled approximation of GNTK. We report the computation time of GNTK and
8

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Table Performance comparison (tea on node classification tasks.
The best and second-hest results are bolded and underlined.
Name
Nodes/CI
Ratio
Random
Herding
K-Center
DosCond
GNTK
LIGHTGNTK
Whole Dataset
CiteSeer
0.18%
38.311..
34.415599
36.8.14.4
42.215.7
43.8
44.71:2.4
10
1.80%
56.712.7
63.710.9
60.3.11.2
61.910.8
62.310.4
62.715..1
63.6.1..3
50
9.02%
64.0122.
63.71112
64.61332
63.511.1
6.681
63.411.1
Cora
0.26%
35.919.0
37.314.55
39,416.2
39.3124
42.7115.6
44.6.1..2
10
2.58%
74.012.2
7.
72.812.6
74.212.6
75.513.9
7..2:10.6
77.611.0
50
12.92%
77.611.0
7..41.00
77.74111
77.410.9
77.611.0
77.9
PubMed
0.02%
53.845.0
52.916.1
52.916.2
55.642.7
56.413.0
1.
10
0.15%
65.813.2
65.514.1
64.513.9
61.01388
70.1+2.0
70.7::2.1
69.7:2.9
50
0.76%
71.344.1
718114
70.3+.0.7
69.814.1
72.310.88
71.243.1
ogbn-arxiv
0.02%
11.5+6.6
14.5+3.0
14.5+3.0
13,2+0.8
15.7+11.4
15.3++0.7
10
0.24%
16.142.6
15.3+1.7
16.442.1
16.3+4.5
18.2+0.8
18.9+332
51,4+1.4
1.18%
25.142.1
24.4+1..
22.0++1.4
24.0+H1.8
28.5+
29.1+-0.4
LIGHTGNTK during the distillation process across all graph-level datasets, as summarized in Table 3. As demon- strated in the table, the computation time required by LIGHT- GNTK consistently lower than that of the original GNTK across all benchmarks, with reductions ranging from 6% 27% depending on the graph size of dataset.
computational cost of our distillation method under different sampling rates in Bernoulli sampling, and represent the result in Table 10 and Table 11 in Appendix D.7.
These experimentals demonstrate the effectiveness of our eigenvalue sampling strategy. Among various sampling strategies, Bernoulli sampling achieves the best performance by capturing both global and local eigenvalue information on the entire graph. Moreover, as the sampling rate in- creases, computational overhead increases rapidly, but per- formance improvements levels off after a certain threshold. Based on empirical observations, we the sampling rate to 0.1 in experiments, which offers a favorable trade-off' between performance and computational efficiency.
These results demonstrate that our approximation provides substantial efficiency gains. Importantly, as shown in pre- vious sections, this improvement is achieved without sacri- ficing the representational power or predictive performance of GNTK. This improved efficiency enables the practical scaling of kernel-based graph methods to large real-world datasets.
6. Conclusion and Future Work
Table Computation time (in seconds) GNTK and LIGHT-
GNTK graph classification datasets.
In this paper, we propose Lightweight Graph Neural Tan- gent Kernel (LIGHTGNTKK), novel framework for ef- ficient dataset distillation in Graph Continual Learning (GCL). LIGHTGNTK leverages low-rank Laplacian ap- proximation via Bernoulli sampling and linear associa- tions in GNTK, substantially reducing computation while maintaining strong performance on graph, node, and edge tasks through unified subgraph anchoring strategy. Exten- sive experiments on diverse benchmarks show that LIGIT- GNTK achieves state-of-the-an performance and provides a scalable, cost-effective solution for continual learning.
Dataset
GNTK
LIGHTGNTK
NCII
196
144
NCI109
136
PROTEINS
26.3
24.8
DD
278
203
ogbg-moo molhiv
921
683
ogbg-molbbbp
72.6
59.7
Lgbg-molbace
54.5
48.8
5.4. Key Components Study (RQ3)
In future work, we will investigate the effects of distribu- tional shifts between pretrain and fine-tuning datasets and explore methods to further improve model generalization and transferability in large-scale dynamic graphs.
To demonstrate effectiveness of key components in LIGHTGNTK, we conduct the following explorations. O We explore different low-rank optimization strategies, including selecting the largest eigenvalues (top-r sampling), select- ing the smallest eigenvalues (bottom-r sampling), and selecting r eigenvalues based on Berouuli sampling, where r is the rank the approximated Laplacian matrix. The comparative performance of these methods is summarized Table 9 in Appendix D.6. @ We evaluated the quality and
Acknowledgments
This work is supported by the National Natural Science Foundation of China (No.U23A20468).

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Impact Statement
References
This paper advances the efficiency and adaptability of Graph Neural Networks through a lightweight dataset distillation framework for large graph models. We anticipate posi- tive societal impact by lowering the computational barriers to graph-based machine learning across diverse domains. We do not foresee any ethical issues or negative societal consequences beyond those generally associated with the advancement machine learning.
Alaoui, A. and Mahoney, M. W. Fast randomized kernel ridge regression with statistical guarantees. Advances in neural information processing systems, 28, 2015.
Arora, S., Du, S., Hu, W., Li, Z., Salakhutdinov, R. R., and Wang, On exact computation with an infinitely wide neural net. Advances in neural information process- ing systems, 32, 2019.
Avron, H., Clarkson, K. L., and Woodruff, D. P. Faster ker- nel ridge regression using sketching and preconditioning. SIAM Journal on Matrix Analysis and Applications, 38 (4):1116-1138, 2017.
Bai, S., Zhang, M., Zhou, W., Huang, S., Luan, Z., Wang, D., and Chen, B. Prompt-hased distribution alignment for unsupervised domain adaptation. Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 729-737, 2024.
Bianchi, F. M., Grattarola, D., and Alippi, C. Spectral clustering with graph neural networks for graph pooling, International conference on machine learning, pp. 874
883. PMLR, 2020.
Borgwardt, K. M., Ong, C. S., Schönauer, S., Vishwanathan, S., Smola, A. J., and Kriegel, H.-P. Protein function prediction via graph kernels. Bioinformatics, (suppll__ i47-i56, 2005.
Cai, H., Zheng, V. W., and Chang, C.-C. comprehen- sive survey of graph embedding: Problems, techniques, and applications. IEEE TKDE, 2018.
Chen, X., Hsieh, C.-J., and Gong, B. When vision transform- ers outperform resnets without pre-training or strong data augmentations, 2022. URL http:: http::s//axxivvrrge. abs /bbs/206..r 01548.
Deng, C., Li, X., Feng. 7., and Zhang, Z. Garnet: Reduced- rank topology learning for robust and scalable graph neu- ral networks. In Learning on Graphs Conference, pp. 3-1. PMLR, 2022.
Ding, H., Fang, Y., Zhu, R., Jiang, X., Zhang, J., Xu, Y., Chu, X., Zhao, J., and Wang, Y. 3ds: Decomposed diffi- culty data selection's 's case study on Ilm medical domain adaptation. arXiv preprint arXiv:2410. 10901, 2024.
Du, S. S., Hou, K., Póczos, B., Salakhutdinov, R., Wang. R., and Xu, K. Graph neural tangent kernel: Fusing graph neural networks with graph kernlss, 2019, URL https: /tte:://aavvveeee wrw. abs/ 905. 905.13....000 13192.
Duan, Y., Zhang, G., Wang, S., Peng. X., Wang, Z., Mao, J., Wu, H., Jiang, X., and Wang, K. Cat-gnn:
10

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Enhancing credit card fraud detection via causal poral graph neural networks. ArXiv, abs/2402.14708, 2024a. URL https///app.sssammmcccccccooo.ee Btt/CorpusID:267700 2
Holzmüller, D., Zaverkin, V., Kästner, J., and Steinwart, I. A framework and benchmark for deep batch active learning for regression, 2023. http:://: //axxiv.. orq/ abs / 2203.22222 09410.
Duan, Y., Zhao, J., Mao, J., Wu, H., Ju, Ma, C., Wang, K., Wang. K., Li, X., et al. Causal deciphering and in- painting in spatio-tempore dynamics via diffusion model. Advances in Neural Information Processing Systems, 37: 107604-107632, 2024b.
Hu, W., Fey, M., Zimik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., and Leskovec, J. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.000 00667,, 2020a.
Hu, W., Fey, M., Ren, H., Nakata, M., Dong, Y., and Leskovec, J. Ogb-Ise: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.0944 2021.
Duvenaud, D. K. Maclaurin, D., Iparraguirre, J., Bom- barell, R., Hirzel, T., Aspuru-Guzik, A., and Adams, R. P. Convolutional networks on graphs for learning molecular fingerprints. In NeurIPS, 2015.
Hu, Z., Dong, Y., Wang, K., Chang, K.-W., and Sun, Y. Gpt- gnn: Generative pre-tr pre-training of graph neural networks. Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2020b.
Fang. Y., Qin, Y., Luo, H., Zhao, F., Xu, B., Zeng, L., and Wang, C. When spatio-temporal meet wavelets: Dis- entangled traffic forecasting via efficient spectral graph attention networks. In ICDE, 2023.
Huang, M., Liu, Y., Ao, X., Li, K., Chi, J., Feng, J., Yang, H., and He, Q. Auc-oriented graph neural network for fraud detection. In WWW. 2022.
Fang, Y., Liang, Y., Hui, B., Shao, Z., Deng, L.,, Liu, X., Jiang, X., and Zheng, K. Efficient large-scale traffic forecasting with transformers: spatial data management perspective. SIGKDD 2024, 2024.
Jacot, A., Gabriel, F,, and Hongler, C. Neural tangent ker- nel: Convergence and generalization neural networks. NeurIPS, 2020.
Farahani, R. Z. and Hekmatfar, M. Facility location: cepts, models, algorithms and case studies. Springer Science & Business Media, 2009.
Jiang, S., Man, Y., Song, Z.,, Yu, Z., and Zhuo D. Fast graph neural tangent kernel via kronecker sketching. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 7033-7041, 2022.
Gao, X., Chen, H., and Haworth, J. A spatiotemporal analy- sis of the impact of lockdown and coronavirus on london's 's bicycle hire scheme: from response recovery new normal. GIS, 2023a.
Jiang, T. and Wang. Asymptotic properties of random restricted partitions. Mathematics, 11(2):417, 2023.
Jiang, X., Qin, Z., Xu, J., and Ao, Incomplete graph
Gao, X., Haworth, J., Zhuang. D., Chen, H., and Jiang, X. Uncertainty quantification in the road-level traffic risk prediction by spatial-temporal zero-inflated negative binomial graph neural network (stzinb-gnn), GIScience 2023, 2023b.
learning via attribute-structure decoupled variational auto- encoder. In WSDM oral 2023, pp. 304-33 312. 2023a.
Jiang, X., Zhuang, D., Zhang, X., Chen, H., Luo, J., and
Gao, X. Uncertainty quantification via spatial-temporal tweedie model for aero-inflated and loig-tail travel de- mand prediction. In CIKM, 2023b.
Goodman, J. E. and O'Rourke, J. (eds.). Handbook of discrete and computational geometry. CRC Press, Inc., USA, 1997. ISBN 0849385245.
Jiang, X., Qiu, R., Xu, Y., Zhang, W., Zhu, Y., Zhang, R., Fang. Y., Chu, X., Zhao, J., and Wang, Y. Ragraph: A graph learning framework.
Guo, S., Ren, Y., Albrecht, S. V., and Smith, K. Ipntk: Better generalisation with less data via sample interaction during learning. arXiv preprint Kiv:2401.08800 2024.
general vetrieval-augment
NeurIPS 2024, 2024.
Jiang, X., Zhang, W., Fang, Y., Gao, X., Chen, H., Zhang, H., Zhuang, D., and Luo, J. Time series supplier alloca- tion via deep black-litterman model. AAAI 2025, 39(1)): 11870-11 11878, 2025.
Gupta, M. Jain, S., Ramani, V., Kodamana, H., and Ranu, S. Bonsai: Gradient-free graph distilla- tion for node classification. ArXiv, abs/2410.17579, 2024. URL https: http:/ppssssttttcccchhlllaaaaa htt//oorpusID:2233222
Jin, B., Liu, G., Han, C., Jiang, M., Ji, H., and Han, J. Large language models on graphs: comprehensive survey. IEEE Transactions on Knowledge and Data Engineering, 2024.
Hcinonen, J. Lectures on Lipschitz analysis. Number 100.
University of Jyväskylä, 2005.

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Jin, W. Tang, X., Jiang, H., Li, Z., Zhang, D., Tang, J., and Yin, B. Condensing graphs via one-step gradient matching, In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, KDD '22, pp. 720-730. ACM, August 2022a. doi: 00.1144//534478.35399999 URL http:/ attp://dee //tp://xx..oo doi. 05519/33534778833399333 3539429.
Li, Z., Xia, L., Xu, Y., and Huang, C. Gpt-st: generative pre-training of spatio-temporal graph neural networks. Advances in Neural Information Processing Systems, 36, 2024b.
Liu, J., Jiang, G., Bai, Y., Chen, T., and Wang. H. Under- standing why neural networks generalize well through gsar of parameters, 2020. URL https://arxix: org/ /aa/ab///// absaaaaa...... 07384.
Jin, W., Zhao, L., Zhang, S., Liu, Y., Tang, J., and Shah, N. Graph condensation for graph neural networks, 2022b. URL https: / /ttpxvaxxivv/////////ee/eaaaa. org/
07580.
Liu, M., Li, S., Chen, X., and Song, L. Graph condensation via receptive field distribution matching. arXiv preprint arXiv:2206. 13697, 2022. doi: 10.48550/arxiv.2200 13697. URL https: btttps://ariive org 2206. 13697.
Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020. URL https://arxix: org/ 220.//2000...66.....
Liu, Y., Ao, X., Qin, Z., Chi, J., Feng, J., Yang. H., and He, Q. Pick and choose: gnn-based imbalanced learning approach for fraud detection. In Proceedings of the Web Conference 2021, pp. 3 3168-3177, 2021.
Kipf, T. N. and Welling, M. Semi-supervised classification with graph convolutional networks. 5th International Conference on Learning Representations, ICLR 2017 Conference Track Proceedings, pp. 1-14,2017
Liu, Y., Bo, D., and Shi, C. Graph distillation with eigen- basis matching, 2024a. URL 2tt5/23100090
Lee, J., Xiao, L.., Schoenholz, S. S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. Wide neural net- works of any depth evolve as linear models under gradient descent. Journal of Statistical Mechanics: Theory and Ex- periment, 2020( 2): 124002, December 2020. ISSN 1742- URL http:
Liu, Z., Yu, X., Fang, Y., and Zhang, X. Graph- prompt: Unifying pre-training and downstream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 417-428, New York, NY, USA, 2023. Association for Comput- ing Machinery. ISBN 9781450394161. doi: 10.1145/ 3543507 3588337. URL 1145/354358 .3933386.
5468. doi: 10.1188//1442-5466//bbee66 //dx. doi org 10.10000000.. 0000000//1111100 54 000////7222-5566/22666 abbbb..
Lei, S. and Tao, D. A comprehensive survey of dataset distillation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Li, G., Hsu, H., Chen, C.-F., and Marculescu, R. Fast-ntk: Parameter-efficiee unlearning for large-scale models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 227-234, 2024a.
Liu, Z., Zeng, C., and Zheng, G. Graph data condensation via self-expressive graph structure reconstruction, 2024b. URL httpthtt://// //arxiiv/// orgg 2403. 072
Lohr, S. Sampling: design and analysis. Chapman and Hall/CRC, 2021.
Li, K., Liu, Y., Ao, X., Chi, J., Feng, J., Yang, H., and He, Q. Reliable representations make stronger defender: Unsupervised structure refinement for robust gnn. In SIGKDD, 2022a.
Ma, Y., Yan, N., Li, J., Mortazavi, M., and Chawla, N. V. Hetgpt: Harnessing the power of prompt tuning pre- trained heterogeneous graph neural networks. In Proceed- ings of the ACM Web Conference 2024, pp. 1015-1023, 2024.
Li, R., Jiang, X., Zhong, T., Trajcevski, G., Wu, J., and Zhou, F. Mining spatio-temporal relations via self- self-paced graph contrastive learning. In SIGKDD 2022, pp. 936- 944. 2022b.
Matsugu, S., Fujiwara, Y., and Shiokawa, H. Uncovering the largest community in social networks at scale. In IJCAI, 2023.
Li, S., Xie, R., Zhu, Y., Ao, X., Zhuang, F., and He, Q. User- centric conversational recommendation with multi-aspect user modeling. In SIGIR, 2022c.
McCallum, A. K. and Nigam, K. Automating the construc- tion of internet portals with machine learning. In Proceed- ings of the 3rd international conference on knowledge discovery and data mining 2000.
Li, Z., Wang, R., Yu, D., Du, S. S., Hu, W., Salakhutdinov, R., and Arora, S. Enhanced convolutional neural tangent kernels. arXiv preprint arXiv: /ttp://9//00000,,/ 2019.
pp. 122-128,
12

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Miao, H., Liu, Z.,, Zhao, Y., Guo, C., Yang, B., Zheng, K., and Jensen, C. S. Less is more: Efficient time ries dataset condensation via two-fold modal matching. VLDB, 2024.
Senior, D. and Sandeep, A. Citeseer: A citation index for computer science. Journal of Computer Science, 25:445- 460, 2010.
Shervashidze, N., Schweitzer, P., Van Leeuwen, E. J., Mehlhorn, K., and Borgwardt, M. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
Mohamadi, M. A., Bae, W., and Sutherland, J. A fast, well-founded approximation the empirical neural tan- gent kernel. 2022. doi: 10.48550/ARXIV22206. 12543. URL wwwps:ttp:..mm / //axxviiivv...///// org/
12543.
Shuman, D. I., Narang. S. K., Frossard, P., Ortega, A., and Vandergheynst, P. The emerging field of signal processing on graphs: Extending high-dimensional data analysis networks and other irregular domains. IEEE Signal Processing Magazine, 2013.
Morris, C., Kriege, M., Bause, F,, Kersting, K., Mutzel, P., and Neumann, Tudataset: collection bench- mark datasets for learning with graphs. In ICML 2020 Workshop Graph Representation Learning and Beyond (GRL+ 2020), 2020. URL www. www.pgearhheiinngggg....o io.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation lan- guage models, 2023. URL https: https://aaxxv.oopp 2ts/22222.1977
Neal, R. M. and Neal, R. M. Priors for infinite networks. Bayesian learning for neural networks, pp. 29-53, 1996.
Nguyen, A. I., Tran, T., Gal, Y., Torr, P., and Baydin, A. G. KL guided domain adaptation. In International Confer- ence on Learning Representations, 2022. URL https: //openreview. net/ forum? w d=t.dd00
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Graph attention networks. ICLR, 2018.
Nguyen, T.,
Chen, Z., and Lee, J. Dataset meta- learning from kernel ridge-regression. arXiv preprint arXiv:2011.00055 2021.
Von Luxburg, U. A tutorial on spectral clustering. Statistics and computing, 17:395-416, 2007.
Novak, R., Xiao, L., Lee, J., Bahri, Y., Yang, G., Hron, J., Abolafia, D. A.. Pennington, J., and Sohl- Dickstein, J. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint arXiv: 2018
Wale, N., Watson, I. A., and Karypis, G. Comparison of descriptor spaces for chemical compound retrieval and classification. Knowledge and Information Systems, 14: 347-375, 2008.
Wang. J., Lan, C., Liu, C., Ouyang, Y., and Qin, Gen- eralizing unseen domains: survey on domain gen- eralization. IEEE Transactions Knowledge and Data Engineering, 35(8):8052-800 8072, 2022.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Park, D. S., Lee, J., Peng, D., Cao, Y., and Sohl-Dickstein, Towards nngp-guided neural architecture search, 2020. URL https: /tl:://aaxxa /arxivivivvv////ia//aaa//aaaavva org/ abs 2011 06006.
Wang, L., Fan, W., Li, J., Ma, Y., and Li, Q. Fast graph condensation with structure- based neural tangent ker- nel, 2024a. URL httptts:: //apsii////xvv.///a orgg/bs////33 11046.
Qin, Y., Fang, Y. Luo, H., Zhao, F,, and Wang, C. Next point-of-interest recommendation with auto- corto-oorelltion enhanced multi-modal transformer network. In SIGIR, 2022.
Wang. Y., Rossi, R. A., Park, N., Chen, H., Ahmed, N. K., Trivedi, P., Dernoncourt, F,, Koutra, D., and Derr, T. Large generative graph models, 2024b. URL https: http://a.....o/a/////00000 05109.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451, 2019.
Welling, M. Herding dynamical weights to learn In Pro- ceedings of the 26th annual international conference on machine learning, pp. 1121-1128, 2009.
Ren, Y. and Sutherland, D. Learning dynamics of ilm finetuning, 2025. URL htttt:://aaaaee wrw..vooorrr...ooooooo wrwoorr//ooooo..ooooo..o.o..ee / 2407. 10490.
Xia, L., Kao, B., and Huang, C. Opengraph: Towards open graph foundation models, 2024.
Sener, O. and Savarese, S. Active learning for convolutional neural networks: core-set approach. arXiv preprint urXiv:/708s 00489, 2017.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? In ICI.R, 2019.
13

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Xu, Y., Chu, X., Yang, K., Wang, Z., Zou, P., Ding, H., Zhao, J., Wang, Y., and Xie, B. Seqcare: Sequential training with external medical knowledge graph for diagnosis prediction in healthcare data. In Proceedings of the ACM Web Conference 2023, pp. 2819-2830, 2023a.
1142-1153, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400012746. doi: 10.
1145/3696410.3714888. URL :estps://do../ 10.1145/36964100 371499999111.11188.888
Zandieh, A., Nouri, N., Velingker, A., Kapralov, M., and Razenshteyn, I. Scaling up kernel ridge regression via locality sensitive hashing. In International Conference on Artificial Intelligence and Statistics, pp. 4088 4097. PMLR, 2020.
Xu, Z., Chen, Y., Pan, M., Chen, H.., Das, M., Yang. H., and Tong, H. Kernel ridge regression-based graph dataset dis- tillation. In Proceedings of the 29th ACM SIGKDD Con- ference on Knowledge Discovery and Data Mining, KDD '23, pp. 2850-2861, New York, NY, USA, 2023b. Associ- ation for Computing Machinery. ISBN 9798400701030. URL wttph::t//:/bbt
Zhang, H., Wang. D., Zhao, W., Lu, Z., and Jiang. X. Imesn: An improved neighborhood aggregation interaction strategy for multi-scale contrastive siamese networks. Puttern Recognition, 158:111052 2025. ISSN 0031- 3203, doi: http::/woi.orr//...000ppppoo.... URL
doi: 10. 144444800005 org/ / 10. 1145 00/45//5580000 35955303933339999...8
Yang, G. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
com/
Yang, G. and Littwin, E. Tensor programs iib: Architec- tural universality of neural tangent kernel training dynam- ics, 2021. URL https: / /twxxviivvvv.///////ae./aaaaa www/argg/......ooooo... a210///........ 03703.
Zhang, R., Jiang, X., Fang, Y., Luo, Xu, Y., Zhu, Y., Chu, X., Zhao, J., and Wang, Y. Infinite-horizz graph filters: Leveraging power series to enhance sparse information aggregation. arXiv preprint arXiv:2401 0.99439 2024.
Yang, K., Xu, Y., Zou, P., Ding, H., Zhao, J., Wang. Y., and Xie, B. Kerprint: local-global knowledge graph enhanced diagnosis prediction for retrospective and prospective interpretations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 5357-5365, 2023.
Zhang, Z., Li, H., Zhang, Z., Qin, Y., Wang. X., and Zhu, W. Graph meets llms: Towards large graph models, 2023. URL https: / arxiv. org/ abaa 14522.
Zhao, B., Mopuri, K. R., and Bilen, H. Dataset condensation with gradient matching, 2021. URL bttps::/arrxt org/ abs / 2006 05929.
Yang. L. and Dong, L. Pubmed: A new repository for biomedical research. In International Conference on Bioinformatics and Biomedical Engineering (ICBBE- 2016), pp. 239- 2016.
Zhou, Y., Wang, Z.., Xian, J., Chen, C., and Xu, J. Meta- learning with neural tangent kernels, 2021. URL https: / // //arxivv org/ abs 221./2102.000 03909.
Zhu, R., Jiang, X., Wu, J., Ma, Z., Song, J., Bai, F., Lin, D., Wu, L.,, and He, C. Grait: Gravient-driven refusal-aware instruction tuning for effective hallucination mitigation. NAACL 2025, 2025.
Yang. Z. Cohen, W., and Salakhudinov, R. Revisiting supi-super learning with graph embeddings. In International conference on machine learning, pp. 40-48. PMLR. 2016.
Yu, R., Liu, S., and Wang, X. Dataset distillation: A compre- hensive review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Yu, X., Zhou, C., Fang, Y., and Zhang, X. Multigprompt for multi-task pre-training and prompting on graphs. In Proceedings of the ACM Web Conference 2024, WWW '24, pp. 515-526, New York, NY, USA, 2024. Associa- tion for Computing Machinery. ISBN 9798400701719. doi: 10. 00.1145/3589333336654444 org/ / 10. 114 145 000055/55933333 3645423.
URL https: : /doi.
Yu, X., Gong. Z., Zhou, C., Fang, Y., and Zhang, H. Samgpt: Text-free graph foundation model for dulti-domain pre- training and cross cross-domain adaptation. In Proceedings of the ACM on Weh Conference 2025, WWW '25, pp.
14

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
A. Notations
The main notations in this paper are summarized in Table 4.
Table 4. Notations Tables in LIGHTGNTK
Notation
Definition
graph
X
node feature matrix
A
adjacency matrix
label matrix
graph Laplacian matrix eigenvalues matrix of L degree matrix of graph eigenvectors matrix of L hidden representation in GNN number of nodes in graph dimension of node features k hop for subgraph expected rank of
D
U
H
d
Neural Tangent Kernel Neural Tangent Kernel Matrix neural network function parameters of neural network
K
0
Drnin
target training subgraphs set validation subgraphs set synthetic training subgraphs set test subgraphs set number of subgraphs in D Duain N train
D
D Day
D Prest
Nval
number of subgraphs in Vval number of subgraphs in Dayn the underlying data distribution
Nayn
II
B. More Motivation Details
B.1. Why could GNTK Capture Both Structural and Feature Relationships?
The ability of GNTK to capture both structural and feature relationships is rooted in its gradient-based dataset distillation mechanism. In our LIGHTTGTKK framework, structural information is propagated through the Laplacian matrix L, while feature information is encoded through the weight matrix W.
Mathematically, in standard (non-graph) dataset distillation, the gradient of the loss function L with respect to the weight matrix (!)) at layer / is given by:
dc dc
a (1-1IT
az
where (!-1) represents the activations from the previous layer, and denotes the matrix transpose to ensure correct dimensional alignment.
However, in the GNTK-based formulation, the gradient incorporates graph structural dependencies via the Laplacian matrix L and is expressed as:
dc zW (1-11)
dc
az (2
where (I-1) is the node representation at layer (1- - 1). L is the graph Laplacian, encoding structural relationships, o'(2() represents the element-wist derivative of the activation function.
15

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
From this formulation, it is evident that GNTK-based gradients inherently embed structural information through while simultaneously encoding feature dependencies via 17(1-1) and the weight parameters. This allows GNTK effectively distill graph-structured datasets while preserving essential structural and feature relationships.
B.2. The Difference between GNTK and LIGHTGNTK
While the concepts of GNTK have indeed been explored in KIDD(Xu et al., 2023b), approach in LIGHTGNTK differs significantly. KIDD is specifically designed for graph classification and employs GNTK for kernel ridge regression-based dataset distillation. However, the kernel regression paradigm in KIDD does not guarantee that training graph samples remain relevant during the inference stage.
In contrast, our LIGHTGNTTK framework unifies graph classification, node classification, and link prediction tasks. We introduce Bernoulli sampled low-rank aallaciinnappoxxmmatee GNTK to enhance similarity- -bsed dataset distillation. This novel formulation enables efficient, scalable, and task-adaptive distillation beyond the scope of KIDD.
C. Gradient Analysis Between Train Set and Test Set
In this section, we analyze the gradient relationship between the training and test samples and its impact on the logits of the test set output: o = Softmax(z) = Softmax (L, and f(.-) is the GNN Encoder. For alignment with (Guo et al.,
2024), we denote o = (LX). For consistency in notation, we denote the graph of the training sample as G, = -L,, X
and the graph of the test sample as Gu = {Lu, J. We will analyze the gradients from two perspectives: 0 individual sample pairs and @ overall sample set pairs.
C.I. Single Sample Gradient Analysis
In this section, we analyze the gradient of an arbitrary training sample and its numerical relationship with the gradients of the test samples. By examining this relationship, aim to understand how the changes in the output logits of the test samples are influenced by the gradients. This analysis will provide insights into the impact of training sample gradients on the performance of the model on the test set.
Here, we define the change in the output logits for the test sample Gr between two consecutive optimization steps as:
L. 9 ( - (Lu
(10)
where the subscript L,,X, refers to the training data sample used between iteration Darin and iteration + (following the proof approach from (Guo et al., 2024; Mohamadi et al., 2022), with batch size equal to 1). The learning rate is denoted by n, and the change in X, is given by:
Xm) - o (LuXu) Va (LuXW)lor (0't1-0") 0²) [First-order Taylor Expansion]
(Lu Voz' ( (-NVo((LoX)))l))
V. Vo2 (L, oo (-n- Vo2' X.) VoL(LoX)))a)) (1I)
=-VVed(LnXu)l 2oz' (Lux - (Voz*(LoX.)) )²) VoL(LoXo)]
= =n- VXX o0 O'
Loxo) (Drr(LooXo) - (Lox))))
where (L,, X,) = represents the ground truth logits (Guo et al., 2024).
(Vod' (T Xo)) T is the GNTK value. It is important to note that (t)
Through analysis, we observe that the influence of the training sample Go on the test set sample G. is significant. Specifically, the gradient information from the training sample is transferred the gradient of the test sample through the GNTK value. causing shift in the gradient direction of the test sample and thus affecting the final prediction results. Such influence not only impacts the gradient computation of individual test samples but also has the potential to alter the overall performance of the test set, particularly when there exists considerable discrepancy between the training and test sets. To mitigate the excessive influence of training samples on the test set, it is crucial to filter the gradient samples based on the GNTK value. This filtering process can help prevent interference from training samples on the gradients of the test set, ultimately enhancing the model's generalization ability.
16

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
C.2. Gradient Analysis and Generalization Error
In this section, analyze the statistical properties of the gradients computed on the population sample, specifically for the training and test sets, and discuss their influence generalization error.
For the loss function of the neural network, we define it as follows:
N LSFT
(12)
N
where / represents the parameters of the neural network, {Go} and {G.} denote the training and test datasets, respectively, and Vogo and Voqu represent the gradients with respect to the training and test datasets. Here assume the batch size equals oc. Next, following (Liu et al., 2020), we introduce the Gradient Signal-to-Noise Ratio (GSNR) as:
GSNR = E Vo% Voqu
(13)
Vp8. Pu
where the mean and variance of the gradients for the training and test datasets are given as: V 0% Eec,) Voool, Vogu
[g... Veeul, and e = P(G.) V Ww ph: P(G.)) VV Iu]. respectively. We then define the generalization error Egen as the expectation of the difference between the loss on the test set and the training set:
Egon
E[Lu Col.
(14)
After a single parameter update on the training dataset {G.} (we only consider the first-order Taylor expansion), we can express the change in the test loss as follows: Lu (0(+.1) Lu (0.) VoLu (0.) (0.-1 0.). Using the parameter update rule, we have: 0u+1 0, ( Substituting this update into the expansion of the loss function gives:
(0(0)))) ~ (..).) - (OC voL.(O.).
(15)
Thus, we can express the generalization error as:
Egen E(Cu(0,) (o(0.) E[-n/ Vequl.
(16)
Finally, taking the expectation of this gives:
A = 17 GSNR V P -
(17)
Therefore, GSNR measures the alignment between training and test grades. Then we found that low GSNR leads to lower generalization error.
D. Further Experiment Details
D.1. Datasets Statics
We employ 13 benchmark datasets for evaluation, including real-world static datasets for graph-level classification tasks, 4 citation networks datasets for node-level classification tasks and 2 datasets for edge-level link prediction tasks. The detailed dataset statistics are provided in Table 5. 2
The detail each dataset is introduced below:
NCT1 (Wale et al., 2008) is a collection of chemical compound graphs, used for graph classification. Each graph represents a chemical compound, where nodes correspond to atoms and edges represent bonds between them. The graphs classified into two classes: active and inactive compounds in the context of cancer drug discovery. This dataset consists of 4,110 graphs, with average of 29.9 nodes and 32.3 edges per graph, resulting in graph density of 7.5e-2.
"The code and datasets are available at http://ghh.....cc.tttttsaaa/a///
17

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Table 5. Dataset statistics. Edges
Name
# Graphs
Nodes
# Features
Classes
NCI1
4110
29.9
32.3
NC1109
4127
29.7
32.1
38
PROTEINS
1113
39.1
72.8
2
DD
1179
284.1
7161.2
89
2
ogbg-molhiv
41127
25.5
27.5
2
ogbg-molbbbp
2039
24.1
51.9
2
ogbg-molbes
1513
34.1
73.7
Cora
2708
5429
1433
CiteSeer
3312
4732
3703
PubMed
19717
44338
500
ogbn-arxiv
169343
1166243
128
40
ogbl-collab
235868
1285465
128
2
ogbl-ddi
4267
1334889
2
NCT109 (Shervashidze et al., 2011) is a chemical compound dataset similar to NCII, but it contains 4,127 graphs representing chemical compounds, where nodes correspond to atoms and edges represent bonds. This dataset used for drug discovery and consists of compounds classified into classes based on their activity. The average graph has 29.7 nodes and 32.1 edges, with density of 7.5e-2, making it suitable for graph classification tasks.
PROTEINS (Borgwardt al., 2005) is a graph dataset for protein function prediction, containing 1,113 graphs Each graph represents protein, with nodes representing amino acids and edges indicating interactions between amino acids. An edge is formed between two amino acid nodes if their distance is less than 6 Á (angstroms). The dataset is used for graph classification tasks, with the goal of distinguishing enzymes from non-enzymes, On average, each graph contains 39.1 nodes and 72.8 edges, with a graph density of 9.8c-2.
DD (Shervashidze et al., 2011) is dataset consisting of graphs that represent chemical compounds. Each node in the graph represents an atom, and the edges represent bonds. The dataset includes 1,179 graphs, classified into two categories: mutagenic and non-mutagemic compounds. The graphs in this dataset average 284.1 nodes and 7161.2 edges, with a density of 1.8e-1. The task is to predict whether a protein an enzyme.
ogbg-molhiv (Hu et al., 2021) is part of the Open Graph Benchmark (OGB) collection and contains molecular graphs for predicting HIV activity. The nodes in each graph represent atoms, while the edges represent chemical bonds. The dataset contains 41,127 graphs with an average of 25.5 nodes and 27.5 edges per graph. The goal predict whether molecule is active against HIV, making it binary classification problem.
ogbg-molbbbp (Hu et al., 2021) is dataset from OGB, consisting molecular graphs for predicting blood-brand braid barrier penetration. Each graph represents molecule, with nodes as atoms and edges bonds. The dataset contains 2,039 graphs, with average of 24.1 nodes and 51.9 edges per graph. The classification task involves determining whether molecule can cross the blood-brain barrier.
ogbg-molbace (Hu al., 2021) is another dataset from OGB for predicting the activity of molecules against the BACE (Beta-site amyloid precursor protein cleaving enzyme) protein. The graphs in this dataset represent chemical compounds, with nodes representing atoms and edges representing bonds. It includes 1,513 graphs, with an average of 34.1 nodes and 73.7 edges per graph, used for binary classification tasks.
Cora (McCallum & Nigam, 2000) is a classic citation network dataset widely used for node classification tasks. The dataset consists of 2,708 scientific papers, classified into seven research topics. The citation network contains 5,429 edges, representing citation relationships between papers. Each paper is represented by a 0/1-valued word vector, where the vector's 's dimensions correspond to 1,433 unique words in the dictionary, indicating the presence or absence of each word the paper. The Cora dataset is primarily used for node classification and link prediction tasks and is one the benchmark datasets in the field of graph data analysis.
CiteSeer (Senior & Sandeep, 2010) is a citation network dataset where each node represents a paper and edges represent citations between them. It contains 3,312 documents and 4,732 citation links, with an average of 2.9 edges
18

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
per paper. The task involves classifying papers into one of six categories. It is widely used for node classification tasks in citation network studies.
PubMed (Yang & Dong, 2016) is citation network dataset where nodes represent scientific articles, and edges represent citation links. The dataset contains 19,717 documents in the biomedical domain and 44,338 citation links. The task is to predict the topic of a paper, classified into one of three categories: cancer, geneties, and other. It is primarily used for node classification tasks.
ogbn-arxiv (Hu al., 2021) is graph dataset from OGB for predicting the subject areas of arXiv papers. Nodes represent papers, and edges represent citation relationships. The dataset includes 169,343 papers and 1.16 million citation links. Papers are classified into one of 40 categories. making it large-scale multi-class node classification task.
ogbl-collab (Hu et al., 2021) is part of the OGB collection, consisting of a graph representing scientific collaboration networks. Each node represents a researcher, and edges represent co-authorships. The dataset contains 235,868 nodes and 1,285,465 edges, representing a large collaboration network. The task is to predict whether two researchers are likely to collaborate the future, making it link prediction problem.
ogbl-ddi (Hu al., 2021) is drug-drug interaction dataset from OGB. Each node represents drug, and edges indicate whether two drugs interact with cach other. The dataset contains 4,267 nodes and 1,334,889 edges, used for link prediction tasks to predict potential drug interactions based on the graph structure.
D.2. Datasets Evaluation Metrics
In the experiments, we use accuracy (ACC) as the evaluation metric for graph classification tasks in TUDataset. And we follow the official recommend evaluation metric for tasks in Planetoid and OGB benchmark suites. Specifically, we use ACC for Cora, CiteSeer, PubMed, and oghn-arxiv; ROC-AUC for ogbg-molhiv, ogbg-molbbhp, and ogbg-molbace; Hits@ 50 for ogbl-collab; and Hits 20 for ogbl-ddi.
Table Evaluation metrics for each dataset.
Dataset
Evaluation Metric
Cora, CiteSeer, PubMed, ogbn-arxiv ogbg-molhiv, ogbg-molbbby, ogbg-molbanl ogbl-collab ogbl-ddi
ACC ROC-AUC
Hits @ 50
Hits @ 20
D.3. Baseline Details
In our experiments, there are three coreset dataset distillation methods: Random, Herding (Welling, 2009), and K-Center (Farahani & Hekmatfar, 2009; Sener & Savarese, 2017), learning- baaning-based graph distillation method, DosCond (Jin et al., 2022a), and CNNTed graph-level distillation method, KIDD (Xu et al., 2023b). All experiments are repeated for 5 runs and we report the mean and standard deviation of the results. All implementations are carried out using Py PyTorch 2.5.1 with Python 3.12 on NVIDIA GeForce RTX 3090 GPUs. The details of these baselines are presented below:
Random: The most naive coreset selection method that constructs the distilled dataset Dayn by uniformly sampling graphs from the original training set Drain without leveraging any structural or semantic information.
Herding (Welling, 2009): A class-wise prototype-based method. First, graph representations are learned by GLM pre- pre-trained the entire training set D Then, for each class, it iteratively selects the sample closest the current class centroid in the representation space until the desired subset size is reached.
K- I-Center(Farahani & Hekmatfar, 2009; Sener & Savarese, 2017): A greedy facility location- based approach. After extracting graph representations via the pre-trained GIN, it initializes centers using Herding and iteratively adds the sample that minimizes the maximum distance between any graph in Detain and its nearest center in Daya. This minimizes the worst-case approximation error in the feature space.
19

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
(Js al., 2022a): A gradient- graph distillation method. It optimizes the distilled dataset Daya by aligning the gradients of model parameters trained on Syn with those from Darin the initialization phase. While efficient due to its single-step approximation, it simplifies the bi-level optimization objective through heuristic gradient alignment rather than solving it rigorously.
KXDD(Xu et al., 2023b): A kernel-based method grounded in GNTK theory. It derives an exact closed-form solution for the bi-level distillation objective by matching the GNTK Gram matrices between Dayn and Duain. avoiding approximations used learning-based methods like DosCond. This ensures theoretical alignment of model behaviors on both datasets.
D.4. Empirical Evaluation of the Validation- Distribution Match
To empirically verify the assumption that the validation and test are drawn from similar distributions (introduced in Section 4.3.2), compute the Maximum Mean Discrepancy (MMD) and Kullback-Leibler (KL) divergence between GNN embeddings sampled from both splits after model training. The experimental results are summarized in Table 7.
Table 7. MMD and KL Divergence between Validation and Test Sets
NCI1
NCI109
PROTEINS
DD
ogbg-molhiv
ogbg-molbbbp
ogbg-molbace
MMD
0.0023
0.0024
0.0120 0.0941
0,0056
0.0018
0.0196 0.0651
0,1171
KLD
0.0660
0.0520
0.0019
0.0275
0.0301
As shown in Table 7, all MMD and KI. divergence values are below commonly used critical thresholds for identifying substantial distribution shift (Bai al., 2024; Nguyen et al., 2022), suggesting that the validation and test sets are statistically similar. These results support the reliability of using the validation set as a proxy for estimating test performance in subsequent experiments.
D.5. Edge Level Experiments Result
We comprehensively evaluate the performance of link prediction tasks in edge-level across datasets: ogbl-collab and ogbl-ddi. The results are summarized in Table 8.
Table Performance comparison (se) on link prediction tasks.
The best and second- best results are bolded and underlined.
Name
Links/CI
Ratio
Random
Herding
K-Center
DosCond
GNTK
LIGHTGNTK
Whole Dataset
ogbl-collab (0)
0.01%
13.0.0.3
12.210.5
12.7:0.4
13.5
17.110.7
17.410.88
10
< 0.01%
15.310,9
14.8.11.1
16.111.3
17.211.5
18.911.2
19.5.114
36.810.6
50
0.04%
22.61118
24.3112.1
23.9-1.7
25.812.3
2..2
26.6112.0
ogbl-ddi (Hits
0.05%
12.048.6
4.9.1..5
4.94:..5
11,243.1
13.8-0.4
13.4::5.5
0.47%
12.3+5.2
7.8664
8.9+0.7
12,5++199
14.2+3.0
14.6+-4.7
24.1+:49
50
2.30%
11.1
13.2+4.2
12.3+3.6
12.7:66.4
13.8-2266
13.6:+0.9
D.6. Ablation Study on Different Low-Rank Strategies
To better understand the impact of different low-rank optimization strategies in our framework, we conduct an ablation study on multiple graph classification benchmarks. Specifically, we compare three commonly used sampling methods for low-rank approximation: (1) selecting the largest eigenvalues (top-r sampling). (2) selecting the r smallest eigenvalues (bottom-r sampling), and (3) selecting r eigenvalues based on Bernoulli sampling (Bernoulli Sampling).
For each method, we integrate the sampling strategy into the model's critical low-rank projection components while keeping other hyperparameters fixed. We evaluate performance on six graph classification datasets, including NCII, NCI109, and PROTEINS (measured by classification accuracy), as well as the molbace, ogbg-molbbbp, and ogbg-molhiv datasets (measured by ROC-AUC). The results summarized in Table D.6 demonstrate the effectiveness of different sampling strategies across diverse molecular and protein graph tasks.
20

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Table 9. Performance comparison of different low-rank optimization strategies on graph classification datasets.
NCI1
NCI109
PROTEINS
ogbg-molbace (ROC-AUC)
ogbg-molbbbp (ROC-AUC)
ogbg-molhiy (ROC-AUC)
Sample Method
(ACC)
(ACC)
(ACC)
Top-r Sampling
65.4
64.0
74.7
76.1
67.9
69.2
Bottom-r Sampling
65.1
65.4
75.6
76.7
67.9
69.1
Berouuli Sampling
66.4
65.6
75.9
76.8
68.2
69.3
D.7. Ablation Study on Different Sampling Rates
To systematically investigate the effect of the Bernoulli sampling rate LIGHTGNTK, we conduct an ablation study by varying the sampling rate p 0.05, 0.1, 0.2, and 0.5. The results in terms of performance and time consumption shown in Table 10 and Table 11, respectively.
Table 10. Performance (ACC and ROC-AUC) across different sampling rates.
NC11
NCI109
PROTEINS
ogbg-molbace (ROC-AUC)
ogbg-mole bobgg (ROC-AUC)
ogbg-molhiv (ROC-AUC)
Sample rafe
(ACC)
(ACC)
(ACC)
0.05
65.7
65.0
74.5
74,9
67.2
68.8
0.10
66.4
65.6
75.9
76.8
68.2
69.3
0.20
66.5
64.6
75.0
76.6
67.6
69.4
0.50
65.4
65.1
74.5
76.4
68.2
69.3
Table 11. Time consumption (seconds) across different sampling rates.
Sample rate
NCI1
NCI109
PROTEINS
ogbg-molbace
ogbg-molbbbp
ogbg-molhiv
0.05
117
109
21.6
43,0
57.8
621
0.10
144
136
24.8
48.8
59.7
683
0.20
183
171
26.1
59.1
63.3
733
0.50
186
180
28.3
62.9
65.4
816
E. Gradient Sampling Method for Large Graphs
In large graph computations, the time complexity for calculating gradients can be significant. To alleviate this, we propose a gradient sampling method based on Monte Carlo sampling, which reduces computational overhead while maintaining accuracy in gradient estimation. Instead of calculating gradients for all samples within the batch at once, we perform K random experiments with a batch size defined as BS, each time with different random grouping of the samples. In each experiment, compute the gradient V. La)) of the loss function using backpropagation through the GNNs, For single experiment, the gradient is computed follows:
(oo(
Σ (Lr)),
(Lr),
(18)
BS
where B represents the set of samples in the batch. We then average the gradients over the K repetitions using the Monte Carlo sampling technique:
K
V (ar))ai = = V.V ~
Σ v
(Tax.
(19)
K
This approach helps reduce the computational cost by a factor of BS/K, while still maintaining a robust estimate of the gradient.
21

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
F. Spectral Sampling Approximation Error of LIGHTGNTK
To gain a deeper theoretical understanding of the approximation quality of LIGITTGNTK, we focus on analyzing the differences in gradient representations that arise from using the exact versus the sampled Laplacian matrices. This examination is crucial, lays the groundwork for evaluating how well our low-rank approximation of the Laplacian can capture the underlying dynamics of the graph neural network (GNN).
Let fo(+) denote GNN model that is parameterized by 0. The full graph Laplacian, which we denote L,, can be expressed through its eigendecomposition as L = UAUT. In the context of LIGHTGNTK,, we generate a low-rank approximation I = UAUT by employing Bernoulli sampling on the eigenvalue spectrum. Specifically, the diagonal matrix eigenvalues approximated as follows:
A diag(Ci Andn); & ~ Bernoull(p).
(20)
Here, X E Raxd represents the node feature matrix. Consequently, the GNN processes the input in the form of either LX
or LX.
F.1 Lipschitz Gradient Assumption
For our analysis, we assume that the GNN fo(-) is Ly-smooth with respect to its input. This means that for any two distinct inputs A and B, the following inequality holds:
[V)fo(A) Vafo(B)] L7 L -|--B| B|I
(21)
where Lv is the Lipschitz smoothing coefficient. This Lipschitz condition is standard assumption in the literature concerning neural tangent kernels (NTK) and gradient-based dataset distillation, as referenced in works by (Jacot et al., 2020; Nguyen et al., 2021; Guo et al., 2024). The assumption is particularly valid when the activation functions used within the model are Lipschitz continuous, such as ReLU or GELU, and when weight regularization applied throughout the network.
F.2. Laplacian Approximation Error Analysis
We proceed derive the expected deviation of the gradient when utilizing the sampled Laplacian İ. The expected squared difference between the gradients can be expressed as:
Eg -
< L Ee
(22)
To analyze the term L - L, we utilize the spectral decomposition of L:
L L- 1=1
E (1- uu then (L - -L- = Ea- (u,,,)
(23)
Next, we take the expectation over the Bernoulli sampling process while leveraging the independence of the &s:
E
6.13 X² 11,, x
22: +
+ X2- 11,,, x12
(24)
26 +
X3 || x||
~ (1 - p)
x 1 X11
Consequently, arrive a bound on the expected squared deviation of the gradient:
E -vxiixx))>_ L. (1 p) Σ X2 1 x
22
(25)
This expression illustrates that the approximation error in the gradient directly influenced by the smoothness of the GNN and the extent of the eigenvalue sampling. The term (1 p) indicates how the sampling probability affects the overall approximation quality, emphasizing the importance of choosing a suitable sampling rate in practical applications.
22

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
F.3. LIGHTGNTK Approximation Error Upper Bound
To analyze the impact of gradient deviation on the GNTK approximation, we examine the difference between the GNTK computed using the sampled Laplacian L and the full-rank Laplacian L. Specifically, consider the following expression for the error:
LIGHTTNTI (aAcHrGNTT(91.900 92) (91 (2) = Vofo(L)X)) Vof)(L2Xx) Vofo(LiX))
Vofo(L2X.)'
(26)
By applying the triangle inequality, which states that -
B C | -B - DI+ 1D) A C)2 we obtain:
Vofo(LiX1)) Vafa(Li2X2)=
VofcL2222) [Vofo(L)X))) Vafo(LiX))) Vofo
(27)
Since (o and (L2X)]) are consistent values that depend on the corresponding graphs 91 and G2, we define them as C = VoJo(L, X |] and C2 [ooo( Therefore, we can rewrite the expression as:
LIIGHTGNT 19LGHTGGNT((99999 92) 4NTT (9, 92) = VoJo(LiX.)) VaJo([2X2) VoJa(L.X.) VoJo(12X22 (o C2/1 (o
s
<C,b
(1 (1-1) Σ x 11 X211 + C2Lv
(1-p)
x u x 1|
= Lo +
(28)
where A p). EE X? u X is the error calculated by Laplacian Approximation error bound.
F.4. Analysis of Error Bounds for Different Low-Rank Approximation Approaches
In this section, we will compare three different methods: Bernoulli sampling, Largest-r sampling, and Smallest-r sampling. We utilize the Gradient Normalized Error as our evaluation metric for the following reasons. The oon-ormaliizdd metric Egg [[[TT- 1)
faces significant issues:
1. Scale Variation: Different graphs may have Laplacian matrices with vastly different scales, which makes absolute error values incomparable. This variation can lead to misleading interpretations of error magnitudes across different datasets.
2. Uneven Feature Weighting: Some feature directions have a low weight, meaning that even large error these directions might have negligible effect the overall gradient. As result, direct computations of norms tend to overemphasize errors occurring in high-weight directions, skewing the assessment of approximation quality.
To address these issues, we propose using the normalized error (normalized in elgenvalue dimension) defined as: [[]]_])x|]] which effectively removes the influence of data scale and more accurately reflects the impact the |[X
gradient direction. Next, we will analyze three sampling strategies:
1. Bernoulli Sampling: In this approach, each eigenvalue is retained with probability pt & Bernouli(p). 1,..., n. The approximation error can be expressed as: (L L)X >n (1 Ei)d,u; X). The normalized error for this method is given by:
[[ - X|
E 1ul x
E
Σ im1
(29)
|XX||
2Zll XII2
23

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
2. Largest-r Sclection: In this strategy, only the largest eigenvalues retained: & =M r). Consequently, the error can be expressed as: (I. T.X (1 Gi) X); and the normalized error for this method is:
keir+ | x11
X1I2
Largett
(30)
1 XI2
3. Smallest-r Selection: Here, only the smallest eigenvalues retained: & = -i - r]. Thus, the error is given by: (L -)X Di=((1 (.),,,(u (x, X), with the normalized enror defined as:
[e(...n-)) A5 |u! x||2 X 1u X|I
Emrlesmallest
(31)
n
To compare the tightness of error bounds, we compute differences of the error bounds for Largest-r and Smallest-r methods relative to Bernoulli sampling:
11
Li( 1u X11
Eg
Eg] F XII
A Largest
Largest
Errorgem
n
X [1 X 1
n
X 12 X1I
(32)
[e x
Eg [1
E. U|u XII2
Smallest
Smalest
Brror
n
A u x
A u|, X|I2
Numerical experiments indicate that:
Discarding the n smallest eigenvalues might lead moderate loss of information, especially if the discarded directions have low energy. However, the performance is sensitive the distribution of eigenvalues.
Largest-r:
Smallest-r:
Discarding the largest eigenvalues results in significant information loss, which leads to much higher error bound.
Bernoulli Sampling: With fixed sampling probability p f. the normalized error is expressed Errornem which remains independent of the eigenvalue distribution, providing lower and more stable error bound.
Thus, the Bernoulli sampling method is preferred in practice due to its lower and more consistent error bounds across various graph structures.
G. Gradient Formulation of Vof in Graph Neural Networks
We analyze the gradient of Graph Neural Networks (GNNs) through the examination of forward and backward propagation. In GNNs, the forward propagation hidden layer is represented by the equation:
H = (1HH()-)))(1))
(33)
where:
H() denotes the hidden embeddings at layer 1,
Z. is the normalized adjacency matrix or a low-rank approximation of the graph Laplacian,
( represents the trainable weight matrix at layer 1,
a(-) is the activation function, which introduces conelinearity into the model.
24

---

Efficient Graph Continual Learning via Lightweight Graph Neural Tangent Kernels-based Dataset Distillation
Backpropagation Through Graph Laplacian and Weight Matrices
To compute the gradient of the loss C with respect the hidden activations, we apply the chain rule:
ac
1T
Oc
(2(0))
(34)
a7041]
where:
z (
(35)
This equation illustrates how the gradient flows back through the layers, modulated by the graph structure represented by the Laplacian matrix L. The term L indicates that the gradient flow is influenced by the connectivity of the graph.
Gradient with Respect to Weight Matrices
The gradient of the loss with respect to the weight matrix W(I) is expressed as:
ac aw = H"-I)TET
ac Oz o od(2
(36)
This expression highlights the dependence of the gradients both the activations (l- and the Laplacian matrix L. It illustrates how information propagates through the network during the learning process.
Thus, the gradient (Lx( can be obtained by aggregating the gradients from each layer:
total layers
Oc
vol(Lx)
(37)
OW()
where + the concatenation function.
25


Reviews:
[{"review_id": "FPBgTJy1zx", "reviewer": "Reviewer_BP7g", "summary": "This paper introduces a novel dataset distillation method called LIGHTGNTK for graph continual learning, which benefits the efficient and effective fine-tuning of large graph models. Specifically, the proposed LIGHTGNTK utilizes the low-rank approximation of the Laplacian matrix in Graph Neural Tangent Kernel to efficiently capture both structure and feature relationships enabling effective gradient-based dataset distillation. Moreover, this paper utilized a subgraph anchoring strategy to unify graph-, node-, and edge-level tasks under the same dataset distillation framework. Extensive experiments demonstrating the efficiency and effectiveness of LIGHTGNTK across various graph tasks, including graph classification, node classification, and link prediction.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments, suggestions, and the time spent reviewing our work. Below, we address each of your concerns in detail.\n\n---\n\n## W1: Concepts of GNTK have been previously discussed\n\nThank you for your valuable comment. While the concepts of GNTK have indeed been explored in KIDD [1], our approach in **LIGHTGNTK** differs significantly. **KIDD is specifically designed for graph classification and employs GNTK for kernel ridge regression-based dataset distillation**. However, the kernel regression paradigm in KIDD does not guarantee that training graph samples remain relevant during the inference stage.\n\nIn contrast, our **LIGHTGNTK framework unifies graph classification, node classification, and link prediction tasks**. We introduce a **Bernoulli-sampled low-rank Laplacian-approximated GNTK** to enhance similarity-based dataset distillation. This novel formulation enables efficient, scalable, and task-adaptive distillation beyond the scope of KIDD.\n\n[1] Kernel Ridge Regression-Based Graph Dataset Distillation\n\n---\n\n## W2: GNTK similarity matrix may become computationally intensive\n\nThank you for raising this concern. While constructing the GNTK similarity matrix has a theoretical complexity of $O(N^2)$, we mitigate computational costs by selecting only the most informative samples and reducing redundant information, following the LESS [2] framework to maintain distillation performance.\n\nFurthermore, to further optimize computational efficiency, we explore **pooling-based methods** (e.g., mean pooling) to derive a coarse-grained validation set and a smaller similarity matrix. The experimental results (on node classification tasks using CiteSeer and Cora) are presented below:\n\n### Mean-Pooling Results:\n| gpc | CiteSeer | Cora |\n|-----|-----|-----|\n| 1 | 38.4 | 44.0 |\n| 10 | 60.1 | 72.0 |\n| 50 | 63.1 | 76.8 |\n\n### Pairwise Calculation Results:\n| gpc | CiteSeer | Cora |\n|-----|-----|-----|\n| 1 | 44.7 | 44.6 |\n| 10 | 62.7 | 76.2 |\n| 50 | 63.4 | 77.9 |\n\nThe results indicate that **mean pooling retains a considerable portion of the performance achieved by pairwise computation** while significantly reducing computational costs. Although pairwise similarity remains superior, mean pooling presents a viable alternative in scenarios where computational efficiency is a priority.\n\n[2] LESS: Selecting Influential Data for Targeted Instruction Tuning\n\n---\n\n## W3: Why could GNTK capture both structural and feature relationships?  \n\nThank you for your insightful question. The ability of **GNTK** to capture both **structural** and **feature** relationships is rooted in its **gradient-based dataset distillation mechanism**. In our **LIGHTGNTK** framework, **structural information** is propagated through the Laplacian matrix $ L $, while **feature information** is encoded via the weight matrix $ W $.  \n\nMathematically, in standard (non-graph) dataset distillation, the gradient of the loss function with respect to the weight matrix at layer $ l $ is given by:  \n\n$$\n\\frac{\\partial\\mathcal{L}}{\\partial W^{(l)}} = \\frac{\\partial\\mathcal{L}}{\\partial Z^{(l)}} \\cdot a^{(l-1)T}\n$$\n\nwhere $ a^{(l-1)} $ represents the activations from the previous layer, and $ T $ denotes the matrix transpose to ensure correct dimensional alignment.  \n\nHowever, in the **GNTK-based** formulation, the gradient incorporates graph structural dependencies via the Laplacian matrix $ L $ and is expressed as:  \n\n$$\n\\frac{\\partial\\mathcal{L}}{\\partial W^{(l)}} = H^{(l-1)T} L \\left( \\frac{\\partial\\mathcal{L}}{\\partial Z^{(l)}} \\odot \\sigma^{\\prime} (Z^{(l)}) \\right)\n$$\n\nwhere:\n- $ H^{(l-1)} $ is the node representation at layer $ (l-1) $.\n- $ L $ is the **graph Laplacian**, encoding structural relationships.\n- $ \\sigma^{\\prime} (Z^{(l)}) $ represents the element-wise derivative of the activation function.\n\nFrom this formulation, it is evident that **GNTK-based gradients inherently embed structural information through $ L $** while simultaneously encoding feature dependencies via $ H^{(l-1)} $ and the weight parameters. This allows GNTK to effectively distill graph-structured datasets while **preserving essential structural and feature relationships**.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the methods in this paper make sense for the problem.", "theoretical_claims": "Yes, I have checked the proof in Section 4.2.2.", "experimental_designs_or_analyses": "Yes, I have checked experimental designs and analyses in this paper.", "relation_to_broader_scientific_literature": "This paper is related to dataset distillation.", "essential_references_not_discussed": "No missing references.", "other_strengths_and_weaknesses": "- Strengths:\n    1. This paper is well organized and presented clearly defining the problem statement. The core narrative and all technical contributions are written clearly and concisely, guiding most readers well to fully understand the contributions.\n    2. This paper presents a sensible, clearly presented, and interpretable approach to graph dataset distillation. The method section is further supported with proofs and significant empirical findings.\n    3. All details are presented for full reproduction in this paper including optimization, hyperparameters, datasets, and architectural settings. The further details presented in the appendix are a nice addition to support reproducibility.\n    4. This paper does a good job covering comprehensive experiments to evaluate the proposed method, presenting efficiency and effectiveness compared with state-of-the-art methods on a variety of datasets for different tasks.\n\n\n- Weaknesses:\n    1. The motivation behind graph dataset distillation is not entirely novel, as the concepts of GNTK have been previously discussed in [1].\n    2. The distillation reliance on similarities and the pairwise GNTK similarity matrix may become computationally intensive and could be further optimized.\n    3. The Introduction in this paper lacks a high-level insight of the GNTK to explain why it could capture both structural and feature relationships for dataset distillation via gradients, which makes the reader difficultly to understand the motivation of the LIGHTGNTK intuitively.\n\n    [1] Kernel Ridge Regression-Based Graph Dataset Distillation", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cCDZggtPDz", "reviewer": "Reviewer_LMjf", "summary": "The paper introduces LIGHTGNTK, a novel framework for efficient Graph Continual Learning (GCL) via dataset distillation. It enables GNNs to adapt to diverse downstream tasks without extensive fine-tuning, overcoming high computational costs that hinder Large Graph Models (LGMs). Specifically,  LIGHTGNTK utilizes a low-rank approximation of the Laplacian matrix to capture structural and feature relationships effectively. Moreover, the proposed unified subgraph anchoring strategy supports graph, node, and edge-level tasks. Extensive experiments on multiple graph datasets demonstrate the state-of-the-art performance of LIGHTGNTK.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments, suggestions, and the time spent reviewing our work. Below, we address each of your concerns in detail.\n\n---\n\n## W1: Confusion in using symbols $\\theta$ and $\\Theta$ throughout the text\n\nWe sincerely apologize for the inconsistency in the notation of $\\theta$ and $\\Theta$ in our manuscript. This was an oversight on our part, and we greatly appreciate your feedback on this issue.\n\n**To address this, we will carefully distinguish between the kernel function and the matrix in the final version of the paper. Additionally, we will update the notation table to ensure clarity and maintain consistency throughout the manuscript.** \n\n---\n\n## W2: The absence of parameter sensitivity analyses renders the experimental section incomplete\n\nWe acknowledge that our initial submission lacked comprehensive ablation studies. To address this, we have conducted **extensive experiments analyzing the impact of sampling probability and layer-specific gradients**.\n\n### **1. Sampling Probability Analysis**\n\nWe evaluated dataset distillation quality and computational cost **under different sampling probabilities** in Bernoulli sampling:\n\n#### **Table 1. Performance Comparison Across Different Sampling Rates**\n| Sample Rate | NCI1 | NCI109 | PROTEINS | ogbg-molbace | ogbg-molbbbp | ogbg-molhiv |\n|-|--|--|-|-|-|-|\n| 0.05 | 65.7 | 65.0 | 74.5 | 74.9 | 67.2 | 68.8 |\n| 0.1 | 66.4  | **65.6** | **75.9** | **76.8** | **68.2** | 69.3 |\n| 0.2| **66.5** | 64.6 | 75.0 | 76.6 | 67.6 | **69.4** |\n| 0.5| 65.4 | 65.1 | 74.5 | 76.4 | **68.2**  | 69.3 |\n\n#### **Table 2. Computation Time (seconds) Across Different Sampling Rates**\n| Sample Rate | NCI1 | NCI109 | PROTEINS | ogbg-molbace | ogbg-molbbbp | ogbg-molhiv |\n|--|-|-|-|-|-|-|\n| 0.05 | 117  | 109  | 21.6  | 43.0  | 57.8  | 621  |\n| 0.1| 144  | 136  | 24.8  | 48.8  | 59.7  | 683  |\n| 0.2 | 183  | 171  | 26.1  | 59.1  | 63.3  | 733  |\n| 0.5 | 186  | 180  | 28.3  | 62.9  | 65.4  | 816  |\n\n**Our findings indicate that a sampling probability of 0.1 achieves the optimal balance between training efficiency and model accuracy.**  \nA lower sampling rate leads to a decline in performance due to insufficient training data, whereas a higher rate increases computational costs without significant performance gains.\n\n### **2. Layer-Specific Gradient Analysis**\n\nWe further examined the impact of employing layer-specific gradients by comparing LIGHTGNTK's performance when using gradients across all layers, only the first layer, and only the last layer.\n\n#### **Table 3. Performance Comparison of Layer-Specific Gradient Computation**\n| Dataset  | NCI1 | NCI109 | PROTEINS | ogbg-molbace | ogbg-molbbbp | ogbg-molhiv |\n|-|-|-|-|-|-|-|\n| First Layer | 62.3 | 60.6 | 70.8 | 72.3 | 64.2 | 63.5 |\n| Last Layer  | 66.2 | 64.7 | 75.5 | 76.3 | 66.7 | 68.1 |\n| All Layers  | **66.4** | **65.6** | **75.9** | **76.8** | **68.2** | **69.3** |\n\n**The results demonstrate that utilizing gradients across all layers yields the best performance, while using gradients from only the last layer provides a favorable trade-off between accuracy and efficiency.**\n\n---\n\n## W3: The validation-test distribution assumption constitutes a strong prior that lacks sufficient empirical support\n\nWe acknowledge the concern regarding **Assumption 1 (validation-test distribution consistency)** and would like to clarify this assumption both theoretically and empirically.\n\n### **1. Theoretical Justification**\nFrom a **theoretical perspective**, assuming an **identical distribution across training, validation, and test sets** is a widely adopted principle in machine learning, particularly when data splitting is performed **randomly** under the i.i.d. (independent and identically distributed) assumption. This practice aligns with foundational theories established in various works, such as [1][2].\n\n### **2. Empirical Verification**\nTo further substantiate this assumption, we conducted **quantitative analyses** using **Maximum Mean Discrepancy (MMD)** and **Kullback-Leibler (KL) divergence** on the learned GNN embeddings after training.\n\n#### **Table 4. MMD and KL Divergence between Validation and Test Sets**\n| | NCI1 | NCI109 | PROTEINS | DD | ogbg-molhiv | ogbg-molbbbp | ogbg-molbace |\n|-|--|-|-|-|-|--|-|\n| MMD | 0.0023 | 0.0024 | 0.0120 | 0.0056 | 0.0018 | 0.0196 | 0.1171 |\n| KLD| 0.0660 | 0.0520 | 0.0941 | 0.0019 | 0.0275 | 0.0651 | 0.0301 |\n\nFollowing [3][4], the MMD and KL divergence values are **below the critical thresholds** commonly used to assess distributional shifts. These results provide strong empirical support that the validation and test sets follow **statistically similar distributions**, thereby validating the i.i.d. assumption.\n\n[1] Sampling: Design and Analysis. Chapman and Hall/CRC, 2021.  \n[2] Asymptotic Properties of Random Restricted Partitions. Mathematics, 2023.  \n[3] Prompt-based Distribution Alignment for Unsupervised Domain Adaptation. AAAI, 2024.  \n[4] KL Guided Domain Adaptation. ICLR, 2022.", "claims_and_evidence": "Yes, the claims in this paper are all convincing.", "methods_and_evaluation_criteria": "Yes, evaluation criteria are accurately chosen in this paper.", "theoretical_claims": "The proofs in the main paper and Appendix D make sense.", "experimental_designs_or_analyses": "The validity of experimental designs and analyses is sufficient to demonstrate the effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "The studied problems of graph neural network and dataset distillation are popular in existing machine learning community, which can motivate various real-world applications ranging from financing and bioscience. Thus, the paper is related to broad scientific leterature.", "essential_references_not_discussed": "All essential related works are cited in this paper.", "other_strengths_and_weaknesses": "Strengths:\n\n- This paper innovatively employs the low-rank approximation of the Laplacian matrix to estimate the gradient similarity between datasets for dataset distillation.\n\n- This paper introduces a new anchor strategy to unify different dataset distillation tasks into the same framework.\n\n- The model diagrams are aesthetically pleasing and clearly detailed, contributing to the paper's readability and ease of understanding.\n\nWeaknesses:\n\n- There is confusion in using symbols $\\theta$ and $\\Theta$ throughout the text. The authors should standardize symbol usage, particularly noticeable in Sec. (3.2).\n\n- The absence of parameter sensitivity analyses renders the experimental section incomplete.\n\n- Assumption 1, the validation-test distribution assumption, constitutes a strong prior that lacks sufficient empirical support. The authors are suggested to conduct additional experiments comparing test and validation sets to substantiate this assumption.", "other_comments_or_suggestions": "Suggestion 1: Include experiments in terms of parameter sensitivity to substantiate the claims made in the paper. This will help in understanding the impact of different components and parameters on the model's performance.\n\nSuggestion 2: Address the inconsistencies in symbol usage, particularly concerning $\\theta$ and $\\Theta$. It is important to ensure correct and consistent notation throughout the paper to avoid confusion for readers. Some discrepancies have been noted, and correcting these will improve the clarity and professionalism of the manuscript.", "questions_for_authors": "Please see suggestions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rQ66xOw09a", "reviewer": "Reviewer_32y9", "summary": "This work introduces a novel LIGHTGNTK that contains a low-rank GNTK approximation via Bernoulli sampling and a unified subgraph anchoring strategy for efficient and effective dataset distillation in multi-level tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your comments, suggestions, and every effort spent on reviewing our work. Here we attempt to address all your remaining concerns. In the following, we quote your comments and then give our detailed response point-by-point.\n\n--- \n\n## W1: Ablation studies on key components are missing.\n\nWe acknowledge that our initial submission lacked comprehensive ablation studies. To rigorously validate this, we have **performed extensive ablation experiments focusing on the sampling probability and the use of layer-specific gradients**.\n\n### 1. Sampling Probability\n\nWe investigated the impact of dataset distillation quality and computation cost **under different sampling rates** for Bernoulli sampling: \n\nTable 1.  Performance Comparison among Different Sampling Rates\n| sample rate | NCI1(ACC) | NCI109(ACC) | PROTEINS(ACC) | ogbg-molbace(ROC-AUC) | ogbg-molbbbp(ROC-AUC) | ogbg-molhiv(ROC-AUC) |\n|---|---|---|---|---|---|---|\n| 0.05 | 65.7 | 65.0 | 74.5 | 74.9 | 67.2 | 68.8 |\n| 0.1 | 66.4  | **65.6** | **75.9** | **76.8** | **68.2** | 69.3 |\n| 0.2 | **66.5** | 64.6 | 75.0 | 76.6 | 67.6 | **69.4** |\n| 0.5 | 65.4 | 65.1 | 74.5 | 76.4 | **68.2**  | 69.3 |\n\nTable 2.  Time consumption (seconds) among Different Sampling Rates\n| sample rate  | NCI1 | NCI109 | PROTEINS | ogbg-molbace | ogbg-molbbbp | ogbg-molhiv |\n|---|---|---|---|---|---|---|\n| 0.05| 117 | 109 | 21.6 | 43.0 | 57.8 | 621 |\n| 0.1  | 144 | 136 | 24.8 | 48.8 | 59.7 | 683 |\n| 0.2  | 183 | 171 | 26.1 | 59.1 | 63.3 | 733 |\n| 0.5  | 186 | 180 | 28.3 | 62.9 | 65.4 | 816 |\n\n**The results indicate that a sampling probability of 0.1 achieves the best balance between training efficiency and model accuracy.**\nLower probabilities leed to performance decay due to inadequate training data, while higher probabilities increase computational cost without significant gains in performance. \n\n### **2. Layer-Specific Gradients**\n\nWe analyzed the effect of employing layer-specific gradients by comparing it with LIGHTGNTK that uses gradients across all layers, first layer and last layer. \n\nTable 3.  Performance of gradient computation using layer-specific gradients\n| dataset  | NCI1(ACC) | NCI109(ACC) | PROTEINS(ACC) | ogbg-molbace(ROC-AUC) | ogbg-molbbbp(ROC-AUC) | ogbg-molhiv(ROC-AUC) |\n|---|---|---|---|---|---|---|\n| First Layer | 62.3 | 60.6 | 70.8 | 72.3 | 64.2 | 63.5 |\n| Last Layer | 66.2 | 64.7 | 75.5 | 76.3 | 66.7 | 68.1 |\n| All Layers  | **66.4** | **65.6** | **75.9** | **76.8** | **68.2** | **69.3** |\n\nThe results indicates using gradients across all layers can provide best performance, while choosing gradients from the last layer offers balance between accuracy and efficiency.\n\n---\n\n## W2: Provide computational efficiency of LIGHTGNTK.\n\nWe compared the training time of GNTK and LIGHTGNTK on more datasets. The results, detailed in the table, indicate that LIGHTGNTK achieves an average of 30% reduction in training time. \n\nTable 4. Time consumption (seconds) between GNTK and LIGHTGNTK\n|    | NCI1 | NCI109 | PROTEINS | DD | ogbg-molhiv  | ogbg-molbbbp | ogbg-molbace |\n|---|---|---|---|---|---|---|---|\n| GNTK      | 196 | 184 | 26.3 | 278 | 921 | 72.6 | 54.5 | \n| LIGHTGNTK | 144 | 136 | 24.8 | 203 | 683 | 59.7 | 48.8 |\n\n---\n\n## W3: More discussion of Graph Representation Learning models\n\nTo comprehensively evaluate the impact of different pretrain methods, we conducted additional experiments **using graph contrastive learning (GraphCL [1], CI-GCL [2]) to pretrain our GNN backbone**, comparing it with GPT-GNN-based pretraining approach. \n\nTable 5. Performance Comparison under Different Pretrain Methods\n| pretrain method  | NCI1(ACC) | NCI109(ACC) | PROTEINS(ACC) | ogbg-molbace(ROC-AUC) | ogbg-molbbbp(ROC-AUC) |\n|---|---|---|---|---|---|\n| GPT-GNN (LIGHTGNTK) | 66.4 | 65.6 | 75.9 | 76.8 | 68.2 |\n| GraphCL  (LIGHTGNTK) | 66.5 | 63.9 | 69.3 | 76.2 | 65.6 |\n| CI-GCL  (LIGHTGNTK)   | 66.7 | 64.3 | 73.5 | 76.9 | 67.4 |\n| GPT-GNN (FULL) | 80.0 | 77.7 | 78.6 | 72.7 | 65.0 |\n| GraphCL  (FULL) | 79.1 | 80.7 | 72.7 | 76.5 | 65.4 |\n| CI-GCL  (FULL) | 79.1 | 81.2 | 76.3 | 77.2 | 65.8 |\n\nThe experimental results indicate **the generality of our LIGHTGNTK**, which is effective under different GNN backbones pretrain methods, and the stronger the ability of the base model, the better the quality of data selecting.\n\n[1] Community-invariant graph contrastive learning. ICML.\n\n[2] Graph contrastive learning with augmentations. NeurIPS.", "claims_and_evidence": "The main claim of the paper is that the low-rank approximation of the Laplacian matrix in GNTK can efficiently capture both structure and feature relationships in gradient-based dataset distillation. This claim is derived from the structure-based Laplacian matrix and the features-based similarity matrix.", "methods_and_evaluation_criteria": "The proposed LIGHTGNTK and three evaluation tasks make sense for the dataset distillation at hand.", "theoretical_claims": "The paper validates its theoretical claim of high-quality low-rank approximation by using Bernoulli-based sampling with a theoretical guarantee.", "experimental_designs_or_analyses": "To verify the versatility of the proposed dataset distillation framework on different graph tasks, 7 graph classification datasets, 4 node classification datasets, and 2 link prediction datasets with limited training data are used in this paper to conduct comprehensive experiments.", "relation_to_broader_scientific_literature": "Downstream tasks such as graph continual learning, graph classification, node classification, link prediction, and graph foundation models will benefit.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "S1: A novel integration of low-rank GNTK approximation with dataset distillation has been proposed in the paper, enabling efficient graph continual learning.\n\nS2: Comprehensive empirical validation across 13 datasets and multi-level tasks (node/edge/graph) have been conducted in the paper, demonstrating broad applicability in the future.\n\nS3: The theoretical guarantees on approximation quality have been proven, enhancing methodological credibility.", "other_comments_or_suggestions": "W1: Ablation studies on key components (e.g., sampling probability and layer-specific gradients) are missing.\n\nW2: It would be better to provide the computational efficiency of LIGHTGNTK.\n\nW3: Why some Graph Representation Learning models are not discussed in this paper? Such as:\n\nTan, S., Li, D., Jiang, R., Zhang, Y. and Okumura, M., 2024, July. Community-invariant graph contrastive learning. In Proceedings of the 41st International Conference on Machine Learning (pp. 47579-47606).\n\nYou, Y., Chen, T., Sui, Y., Chen, T., Wang, Z. and Shen, Y., 2020. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33, pp.5812-5823.", "questions_for_authors": "Q1. The ablation experiments are suggested to be conducted to evaluate the effectiveness of each proposed component in LIGHTGNTK.\n\nQ2. It would be better to report the original training time on the whole dataset, which can validate the efficiency of this approach.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "vssM4AVnPD", "reviewer": "Reviewer_YvS6", "summary": "This paper proposed a graph dataset distillation method via Graph Neural Tangent Kenrels (GNTK) for efficient graph continual learning.\n\nThe main idea is using the Bernoulli sampling method to approximate the graph Laplacian which is required for computing the gradients.\nBy carefully setting the probability of Bernoulli distribution, this paper claimed that it could trade off the computational efficiency and approximation error. Further, this paper proposed a data selection and a supervised fine-tuning method to achieve the authors' goal. \n\nThe experimental results seem to support its claims.\n\nOverall, some parts of this paper are unclear and thus I am confused by these, at least in this version.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1: Why use Bernoulli distribution than largest r eigenvalues?\n\nWe adopt Bernoulli sampling because it **effectively balances high-frequency and low-frequency components, preserving both local and global graph structures**. In contrast, **directly selecting the largest $r$ eigenvalues mainly captures local variations, while selecting the smallest $r$ eigenvalues emphasizes global structures.**\n\n### **1. Theoretical Justification**\n\nUnder Lipschitz gradient assumption [1], we assume that the GNN $ f_\\theta $ satisfies  \n$$\n\\|\\nabla_\\theta f_\\theta(A) - \\nabla_\\theta f_\\theta(B)\\| \\leq L_\\nabla \\|A - B\\|,\n$$\n$L_\\nabla$ is the constant. Therefore, when approximating $ L $ with $ \\tilde{L} $, the induced error in the gradients $ \\| \\nabla f_\\theta(LX) - \\nabla f_\\theta(\\tilde{L} X) \\| $ is controlled by $ \\| f_\\theta(LX) - f_\\theta(\\tilde{L} X) \\| $. Given eigen-decomposition:\n$\nL = U \\Lambda U^\\top, \\tilde{L} = \\tilde{U} \\tilde{\\Lambda} \\tilde{U}^\\top,\n$\nwhere $ \\tilde{L} $ is a low-rank approximation, we introduce a selection indicator $ \\xi_i $ for each eigenvalue $ \\lambda_i $, leading to an error formulation:\n$$\n(L-\\tilde{L})X = \\sum_{i=1}^{n}(1-\\xi_i)\\lambda_i u_i (u_i^\\top X).\n$$\n\nTo ensure a fair comparison, we analyze the **normalized error bound**:\n$$\n\\mathbb{E}_\\xi\\left[\\frac{\\|(L-\\tilde{L})X\\|}{\\|LX\\|}\\right] =  \\frac{\\sum \\mathbb{E}[1-\\xi_i]\\ \\lambda_i^2\\|u_i^\\top X\\|^2}{\\sum \\lambda_i^2\\|u_i^\\top X\\|^2}\n$$\n\nFor Bernoulli, where $ \\xi_i \\sim \\text{Bernoulli}(p) $ with $ p = \\frac{r}{n} $, $\n\\mathbb{E}[1-\\xi_i] = 1-p \\approx 1-\\frac{r}{n}. $\n\nThus, the normalized error bound is:\n$\n\\text{Error}_{\\text{Bern}} \\approx 1 - \\frac{r}{n}.\n$ This result is independent of the eigenvalue distribution, ensuring robustness across different graphs.\n\nIn comparison, \n$$\n\\frac{\\sum_{i=r+1}^{n}\\lambda_i^2\\|u_i^\\top X\\|^2}{\\sum_{i=1}^{n}\\lambda_i^2\\|u_i^\\top X\\|^2} = \\text{Error}_{\\text{Largest}}.\n$$\n\n$$\n\\frac{\\sum_{i=1}^{n-r}\\lambda_i^2\\|u_i^\\top X\\|^2}{\\sum_{i=1}^{n}\\lambda_i^2\\|u_i^\\top X\\|^2} = \\text{Error}_{\\text{Smallest}} .\n$$\n\n**These methods depend on the eigenvalue spectrum; simply dropping local or global signals will lead to significantly larger errors** [2, 3].\n\n### **2. Experimental Validation**\n\nWe conducted a comparative analysis of various sampling strategies:\n\n|Method|NCI1|NCI109|PROTEINS|molbace|molbbbp|molhiv|\n|-|-|-|-|-|-|-|\n|Bernoulli|**66.4**|**65.6**|**75.9**|**76.8**|**68.2**|**69.3**|\n|Largest|65.4|64.0|74.7|76.1|67.9|69.2|\n|Smallest|65.1|65.4|75.6|76.7|67.9|69.1|\n\nExperimental results demonstrate that **Bernoulli sampling achieves the best performance** in preserving both global and local graph information.\n\n[1]LECTURES ON LIPSCHITZ ANALYSIS. \n\n[2]Self-supervised graph-level representation learning with local and global structure. PMLR, 2021.\n\n[3]From local to global: Spectral-inspired graph neural networks. NeurIPS 2022.\n\n---\n\n## Q2: Explanation of low-rank optimization\n\nWe acknowledge that our initial complexity analysis contained an error. Below, we provide a **corrected and more detailed explanation**.\n\nThe efficiency improvement of optimization primarily stems from reducing the computational complexity of matrix operations. Instead of directly computing $N \\times N$ dense matrix $ L $, we use a sampled low-rank approximation:\n$\nL \\approx U_r \\Lambda_r U_r^T,\n$\nwhere:  $ U_r \\in \\mathbb{R}^{N \\times r},   \\Lambda_r \\in \\mathbb{R}^{r \\times r}. $\n\nTo compute $ \\tilde{L} X = U_r \\Lambda_r U_r^T X $ efficiently, we decompose into 3 sequential steps:\n- 1.\n$\nZ = U_r^T X, \\quad Z \\in \\mathbb{R}^{r \\times d}, \\quad \\text{cost: } O(rNd)\n$\n- 2.\n$\nY = \\Lambda_r Z = \\Lambda_r (U_r^T X), \\quad Y \\in \\mathbb{R}^{r \\times d}, \\quad \\text{cost: } O(r^2 d)\n$\n- 3.\n$\n\\tilde{L} X = U_r Y = U_r (\\Lambda_r (U_r^T X)), \\quad \\text{cost: } O(Nrd)\n$\n\nThus, overall computational complexity:\n$\nO(rNd) + O(r^2 d) + O(Nrd) \\approx O(Nrd),\n$\nwhere $ r \\ll N $. This significantly reduces the complexity compared to the naive approach **$ O(N^2 d) $**. This low-rank approximation approach is conceptually related to **linear attention** in Transformers [4].\n\n[4]Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention.  ICML 2020.\n\n---\n\n## Q3: Unclear definition of Θ(Gi)\n\nWe will correct this **inconsistency in the notation of Θ in the final version of the paper** by carefully distinguishing between the kernel function and the matrix and updating notation table to ensure clarity.\n\n---\n\n## Q4: Definition of equation (5)\nEquation (5) is a combination of two terms:\n- Cross-entropy loss: $\\frac{1}{M} \\sum_{i=1}^{M} - o_i \\log(\\hat{o}_i)$ measures the difference between true labels $o_i$ and predicted probabilities $\\hat{o}_i$.\n- Regularization: $\\eta \\|\\Theta\\|^2$, is a regularization term that adds a penalty proportional to the squared $L_2$ norm of model parameters $\\Theta$ to prevent overfitting.\n---\n## S1&S2: Typo Errors\nWe will carefully proofread and improve the manuscript. Thanks.", "claims_and_evidence": "The key claims in this paper involve two aspects: 1) Low-rank optimization; and 2) The data-selection and supervised fine-tuning(SFT).\n\n1.1. The motivation of Bernoulli random variables. \n\n> Suppose we have an undirected graph and its adjacency matrix $A$, and we have the corresponding graph Laplacian $L:=D-A\\in\\mathbb{R}^{N\\times N}$ where $D$ is the degree matrix [1]. The first question is why do we need to use Bernoulli distribution? According to the Authors' claims, we just need $r$ ranks, so why not directly choose the largest $r$ eigenvalues? \n\n> From efficiency, these two do not have any difference; From approximation error,  the largest-$r$ eigenvalues lead to a smaller approximation error than any choice of Beronulli sampling.\n\n1.2. I am not sure why and how low-rank optimization helps efficiency.\n\n> According to (2) and (3) in the manuscript, I find that the low-rank graph Laplacian $\\tilde{L}$ is still $N\\times N$. \n\n> To compute the $\\tilde{L}$, one need to have the original graph Laplacian $L$; And then apply the spectral decompostion on it $L=U \\Lambda U^\\top$; Next, computing its low-rank approximation $\\tilde{L}$; Finally, Replace $L$ with $\\tilde{L}$. I am confused by it. Why not directly use $L$ since $L$ and $\\tilde{L}$ have the same shape?\n\n2. Data selection and SFT.\n\n> The definition of $\\Theta(G_i)$ in (4) is unclear. In (3) $\\Theta(G_1, G_2)$ is a kernel function with two inputs; In Page 6, Line 326, $\\Theta$ is a $N\\times M$ matrix; But in (4), $\\Theta(G_i)$ becomes a values with a single input. \n\n> The motivation of (5) is unclear. I admit that my knowledge in the field of SFT may be insufficient. But I failed to understand what (5) was doing. I suggest the authors to explain it more.\n\n\n[1] https://en.wikipedia.org/wiki/Laplacian_matrix", "methods_and_evaluation_criteria": "Same as ``Claims and Evidence''.", "theoretical_claims": "Seem to be correct.", "experimental_designs_or_analyses": "None.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "1. Page 5, line 239. Formally writing a theorem or proposition before the proof may be better than directly writing the proof.\n\n2. Late error. Page 18, line 946.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper devised a novel dataset distillation method called LIGHTGNTK for graph continual learning. After some discussion, the overall verdict was that this submission could be accepted in ICML-2025 while the concerns of reviewers should be addressed in the final version.
Decision: Accept (poster)