Title: Does Graph Distillation See Like Vision Dataset Counterpart?
Venue: NeurIPS 2023
Accepted: yes
Keywords: data-efficient learning

graph generation

graph neural networks
Abstract: Training on large-scale graphs has achieved remarkable results in graph representation learning, but its cost and storage have attracted increasing concerns. Existing graph condensation methods primarily focus on optimizing the feature matrices of condensed graphs while overlooking the impact of the structure information from the original graphs. To investigate the impact of the structure information, we conduct analysis from the spectral domain and empirically identify substantial Laplacian Energy Distribution (LED) shifts in previous works. Such shifts lead to poor performance in cross-architecture generalization and specific tasks, including anomaly detection and link prediction. In this paper, we propose a novel Structure-broadcasting Graph Dataset Distillation (\textbf{SGDD}) scheme for broadcasting the original structure information to the generation of the synthetic one, which explicitly prevents overlooking the original structure information. 
Theoretically, the synthetic graphs by SGDD are expected to have smaller LED shifts than previous works, leading to superior performance in both cross-architecture settings and specific tasks.
We validate the proposed SGDD~across 9 datasets and achieve state-of-the-art results on all of them: for example, on YelpChi dataset, our approach maintains 98.6\% test accuracy of training on the original graph dataset with 1,000 times saving on the scale of the graph. Moreover, we empirically evaluate there exist 17.6\% $\sim$ 31.4\% reductions in LED shift crossing 9 datasets. Extensive experiments and analysis verify the effectiveness and necessity of the proposed designs. The code will be made public.

Does Graph Distillation See Like Vision Dataset Counterpart?

Abstract

Training on large-scale graphs has achieved remarkable results in graph representation learning, but its cost and storage have attracted increasing concerns. Existing graph condensation methods primarily focus on optimizing the feature matrices of condensed graphs while overlooking the impact of the structure information from the original graphs. To investigate the impact of the structure information, we conduct analysis from the spectral domain and empirically identify substantial Laplacian Energy Distribution (LED) shifts in previous works. Such shifts lead to poor performance in cross-architecture generalization and specific tasks, including anomaly detection and link prediction. In this paper, we propose a novel S tructure-broadcasting G raph D ataset D istillation ( SGDD ) scheme for broadcasting the original structure information to the generation of the synthetic one, which explicitly prevents overlooking the original structure information. Theoretically, the synthetic graphs by SGDD are expected to have smaller LED shifts than previous works, leading to superior performance in both cross-architecture settings and specific tasks. We validate the proposed SGDD across 9 datasets and achieve state-of-the-art results on all of them: for example, on the YelpChi dataset, our approach maintains 98.6% test accuracy of training on the original graph dataset with 1,000 times saving on the scale of the graph. Moreover, we empirically evaluate there exist 17.6% ∼ 31.4% reductions in LED shift crossing 9 datasets. Extensive experiments and analysis verify the effectiveness and necessity of the proposed designs. The code is available in the https://github.com/RingBDStack/SGDD.

1 Introduction

Graphs have been applied in many research areas and achieved remarkable results, including social networks [27, 67, 73, 79], physical [16, 3, 57, 10], and chemical interactions [2, 93, 111, 80]. Graph neural networks (GNNs), a classical and wide-studied graph representation learning method [40, 84, 98, 63], is proposed to extract information via modeling the features and structures from the given graph. Nevertheless, the computational and memory costs are extremely heavy when training on a large graph [48, 89, 99]. One of the most straightforward ideas is to reduce the redundancy of the large graph. For example, graph sparsification [66, 77] and coarsening [53, 52, 46, 75] are proposed to achieve this goal by dropping redundant edges and grouping similar nodes. These methods have shown promising results in reducing the size and complexity of large graphs while preserving their essential properties.

𝑋

𝐴

Gradient

Matching

𝑋′

𝑓(𝑋 ᇱ

)

𝐴′

(a) Condense scheme of

GCond

SC : 0.46

(b)

A

′

of

GCond

(c) Cross-arch. Acc. of

GCond

𝑋

𝐴

𝑋′

𝐴′

GEN

Broadcasting

𝑓(𝑍, 𝑋′, 𝑌′)

Gradient

Matching

(d) Condense scheme of

SGDD

SC : 0.18

(e)

A

′

of

SGDD

(f) Cross-arch. Acc. of

SGDD

1

87.6

85.3

86.6

88.7

88.2

2-86.585.286.486.6

89.8

3-

87.4

85.7

89.2

87.686.9

4-86.185.889.988.7

89.2

5- 84.2

86.989.888.888.4

1234

5

1

87.888.689.2

90.7

88.4

2

88.487.9

91.7

87.687.6

3

88.6

86.7

87.4

87.7

87.6

4

87.9

87.4

88.8

88.8

88.4

5

88.888.1

91.4

86.9

88.4

12345

1

87.6

85.3

86.6

88.7

88.2

2

86.5

85.2

86.4

86.6

89.8

3

87.4

85.7

89.2

87.6

86.9

4

86.1

85.8

89.9

88.7

89.2

5-84.2

86.9

89.8

88.8

88.4

2

3

4

5

However, graph sparsification and coarsening heavily rely on heuristics [5] ( e.g. , the largest principle eigenvalues [78], pairwise distances [66, 81]), which may lead to poor generalization on various architectures or tasks and sub-optimal for the downstream GNNs training [37]. Most recently, as shown in Fig. 1(a), GCond [37] follows the vision dataset distillation methods [88, 105, 24, 103] and proposes to condense the original graph by the gradient matching strategy. The feature of the synthetic graph dataset is optimized by minimizing the gradient differences between training on original and synthetic graph datasets. Then, the synthetic feature is fed into the function f ( X ′ ) ( e.g. , pair-wise feature similarity [74]) to obtain the structure of the graph. The synthetic graph dataset (including feature and structure) is expected to preserve the task-relevant information and achieve comparable results as training on the original graph dataset at an extremely small cost.

Although the gradient matching strategy has achieved significant success in graph dataset condensation, most of these works [37, 36, 50, 109] follow the previous vision dataset distillation methods to synthesize the condensed graphs, which results in the following limitations: 1) In Fig. 1(b), we can find that the original graph structure information is not well preserved in the condensed graph. That's because they build the structure ( A ′ ) upon the learned feature ( X ′ ) (Fig. 1(a)), which may cause the loss of original structure information. 2) In vision dataset distillation, applying the gradient matching strategy may result in an entanglement between the synthetic dataset and the architecture that is used for condensing [106, 104, 39]. This can detrimentally impact their performance, especially when it comes to generalizing to unseen architectures [22]. This issue is compounded when dealing with graph data that comprise both features and structures. As shown in Fig. 1(c) and Tab. 2, GCond [37] explicitly displays inferior performance, demonstrating that generating structure only based on the feature degrades the generalization of the condensed graph.

To investigate the relation between the structure of the condensed graph and the performance of training on it, we follow previous works [82, 1, 13, 78] to analyze it from a spectral view. Specifically, we explore the relationship between the Laplacian Energy Distribution (LED) shift [82, 25, 12] and the generalization performance of the condensed graph. We empirically find a positive correlation between LED shift and the performance in cross-architecture settings. To address these issues, we introduce a novel Structure-broadcasting Graph Dataset Distillation (SGDD) scheme to condense the original graph dataset by broadcasting the original adjacency matrix to the generation of the synthetic one, named SGDD, which explicitly prevents overlooking the original structure information. As shown in Fig. 1(d), we propose the graphon approximation to broadcast the original structure A as supervision to the generation of condensed graph structure A ′ . Then, we optimize A ′ by minimizing

the optimal transport distance of these two structures. For X ′ , we follow the [37] to synthesize the feature of the condensed graph. The A ′ and X ′ are jointly optimized in a bi-level loop. Both theoretical analysis (Sec. 3.2) and empirical studies (Fig. 1(c) and 1(f)) consistently show that our method reduces the LED shift significantly (0.46 → 0.18).

We further evaluate the impact of LED shifts and conduct experiments in the cross-architecture setting. Comparing Fig. 1(f), 1(c), and Tab. 2, the improvement of SGDD is up to 11.3%. To evaluate the generalization of our method, we extend SGDD to node classification, anomaly detection, and link prediction tasks. SGDD achieves state-of-the-art results on these tasks in most cases. Notably, we obtain new state-of-the-art results on YelpChi and Amazon with 9.6% and 7.7% improvements, respectively. Our main contributions can be summarized as follows:

Based on the analysis of the difference between graph and vision dataset distillation, we introduce SGDD, a novel framework for graph dataset distillation via broadcasting the original structure information to the generation of a condensed graph.

In SGDD, the graphon approximation provides a novel scheme to broadcast the original structure information to condensed graph generation, and the optimal transport is proposed to minimize the LED shifts between original and condensed graph structures.

SGDDeffectively reduces the LED shift between the original and the condensed graphs, consistently surpassing the performance of current state-of-the-art methods across a variety of datasets.

2 Related Work

Dataset Distillation & Dataset Condensation. Dataset distillation (DD) [88, 106, 24, 51, 110, 23, 86, 49, 87, 6, 7] aims to distill a large dataset into a smaller but informative synthetic one. The proposed method imposes constraints on the synthetic samples by minimizing the training loss difference, while ensuring that the samples remain informative. This technique is useful for applications such as continual learning [105, 106, 39, 44, 104], as well as neural architecture search [61, 62, 96]. Recently, GCond [37] is proposed to reduce a large-scale graph to a smaller one for node classification using the gradient matching scheme in DC [105]. Unlike GCond [37], who build the structure only through the learned feature, we explicitly broadcast the original graph structure to the generations to prevent the overlooking of the original graph structure information.

Graph Coarsening & Graph Sparsification. Graph Coarsening [53, 52, 14] follows the intuition that nodes in the original graph can naturally group the similiar node to a super-nodes. Graph Sparsification [66, 38, 78] is aimed at reducing the edges in the original graph, as there are many redundant relations in the graphs. We can simply sum up both two methods by trying to reduce the 'useless' component in the graph. Nevertheless, these methods primarily rely on unsupervised techniques such as the largest k eigenvalue [66] or the multilevel incomplete LU factorization [14], which cannot guarantee the behavior of the synthetic graph in downstream tasks. Moreover, they fail to reduce the graph size to an extremely small degree ( e.g. , reduce the number of nodes to 0.1% of the original). In contrast, graph condensation can aggregate information into a smaller yet informative graph in a supervised way, thereby overcoming the aforementioned limitations.

3 Preliminary and Analysis

3.1 Formulation of graph condensation

Consider a graph dataset G = { A , X , Y } , where A ∈ R N × N denotes the adjacency matrix, X ∈ R N × d is the feature, and Y ∈ R N × 1 represents the labels. S = { A ′ , X ′ , Y ′ } is defined as the synthetic dataset, the first dimension of A ′ , X ′ , and Y ′ are N ′ ( N ′ ≪ N ).

The goal of graph condensation is achieving comparable results as training on the original graph dataset G via training on the synthetic one S . The optimization process can be formulated as follow,

where GNN θ ( t ) denotes a GNN parameterized with θ , θ S and θ G are the parameters that are trained on S and G , and t ∈ {S , G} , M( · ) denotes a matching function and Loss( · ) is the loss function.

























































(a) LEDs in various architectures

































(b) LEDs in BWGNN [82]

3.2 Analysis of the structure of the condensed graph and its effects

In this section, we first introduce the definition of Laplacian Energy Distribution (LED), then analyze the LED in previous work, and finally explore its influence on generalization performance.

Laplacian Energy Distribution. Following the previous works [25, 12, 82], we introduce Laplacian energy distribution (LED) to analyze the structure of graph data. Given an adjacency matrix A ∈ R N × N , the Laplacian matrix L is defined as D -A , where the D is the degree matrix of A . We utilize the normalized Lapalacian matrix ˜ L = I -D -1 / 2 AD -1 / 2 , where I is an identity matrix. The eigenvalues of ˜ L are then defined as λ 1 , . . . λ n , . . . λ N by ascending order, i.e. , λ 1 ≤ · · · λ n ≤ · · · ≤ λ N , with orthonormal eigenvectors U = (u 1 , · · · u n , · · · u N ) correspond to these eigenvalues.

Definition 1 (Laplacian Energy Distribution [25, 12, 82]) . Let X = ( x 1 , · · · x n , · · · x N ) ⊤ ∈ R N × d be the feature of graph, we have ˆ X = (ˆ x 1 · · · ˆ x n · · · ˆ x N ) ⊤ = U ⊤ X as the post-Graph-FourierTransform of X . The formulation of LED is defined as:

LED analysis of previous work. First, we define η G and η S as the LED of the original graph and condensed graph, respectively. Then the LED shift of the above two graphs can be formulated as: || η G -η S || = || ∑ N i =1 η i ( X , ˜ L ) -∑ N ′ j =1 η j ( X ′ , ˜ L ′ ) || . In GCond [37], the matching function M defaults as the MSE loss. Therefore, the objective of GCond can be written as || θ G -θ S || ≤ ϵ , where ϵ is a small number as expected. Incorporating such an objective in the LED shift formula, the lower bound is shown as follows.

Proposition 1. Refer to [1, 4, 82], the GNN can be recognized as the bandpass filter. Assume the frequency response area of GNN is ( a, b ) , where ( a, b ) is architecture-specific. The lower bound of GCond is shown in Eq. (3) . Detailed proof can be found in Appendix B.1.

According to Eq. (3), we find the lower bond of LED in GCond is related to the frequency response area ( a, b ) ( i.e. , specific GNN). For example, GCN [40] or SGC [92] (utilized in GCond) is a low-pass filter [1], which emphasizes the lower Laplacian energy. As shown in Fig. 1(a), the A ′ is built upon the learned 'low-frequency' feature X ′ , which may fail to generalize to cross-architectures ( i.e. , high-pass filters) and specific tasks.

Exploration of the effects of LED shift on the generalization performance. Although the LED shift phenomenon occurs in GCond, quantifying the LED shift is challenging because G and S have

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Gradient

Matching

GLYPH(cmap:d835)′

GLYPH(cmap:d835)′

GEN

GLYPH(cmap:d835)′

GLYPH(cmap:d835)

GLYPH(cmap:d835)′

LED

Matching

GLYPH(cmap:d835)′

GLYPH(cmap:d835)

Broadcasting

OT

Distance

GLYPH(cmap:d835)

GLYPH(cmap:d835)

Optimizing

GLYPH(cmap:d835)′

Optimizing

GLYPH(cmap:d835)′

Freeze parameters

Laplacian pseudo-inverse

GLYPH(cmap:d835)

0

2

GLYPH(cmap:d835)

0

2

GLYPH(cmap:d835):

Random noise

different numbers of nodes. To enable comparison, we follow an intuitive assumption that two nodes with similar eigenvalue distribution proportions can be aligned in comparison. Thus, we first convert LEDs of G and S into probability distributions using the Kernel Density Estimation (KDE) method [60]. Then we quantify the LED shifts as the distance of the probability distributions using the Jensen-Shannon (JS) divergence [47]. We define the LED shift coefficient ( SC ) in Definition 2:

Definition 2 (LED shift coefficient, SC ) . The LED shift coefficient between G and S is:

where the JS( ·||· ) denotes the JS-divergence, the | V G | and the | V S | is the number of the nodes to corresponding graphs, the K ( · ) represents the Gaussian kernel function with bandwidth parameter h . SC ∈ [0 , 1] reflects the divergence between G and S (a smaller SC indicates more similar).

In Fig. 2, we empirically study the influences of SC in two settings: various GNNs in Fig. 2(a) and fixed BWGNN [82] with adaptive bandpass in Fig. 2(b). Based on the results in Fig. 2, We have several observations: (1) The entangled learning paradigm that building structure ( i.e. , adjacency matrix) upon on feature matrix will significantly lead to the LED shift phenomenon. (2) The positive correlation exists between the LED shift and the generalization performance of the condensed graph. (3) Preserving more information about the original graph structure may alleviate the LED shift phenomenon and improve the generalization performance of the condensed graph.

4 Structure-broadcasting Graph Dataset Distillation

In this section, we first present the pipeline and overview of SGDD in Fig. 3. Then, we introduce two modules of SGDD. Finally, we summarize the training pipeline of our SGDD.

4.1 Learning graph structure via graphon approximation

To prevent overlooking the original structure A from G , we broadcast A as supervision for the generation of A ′ . Considering the different shapes between A and A ′ ( N ′ ≪ N ), we introduce graphon [21, 70, 17, 33, 94] to distill the original structure information to the condensed structure A ′ . Specifically, given random noise Z ( N ′ ) ∈ R N ′ × N ′ as the input coordinates, through the generative model, we then synthesize an adjacency matrix A ′ with N ′ nodes. This process can be formulated as A ′ = GEN( Z ( N ′ ); Φ) , where the GEN( · ) is a generative model with parameter Φ , and the optimization process is then defined as:

where A is supervision and Distance( · ) is a metric that measure the difference of A and A ′ . The details of Distance( · ) can be found in Sec. 4.2. To avoid overlooking of the inherent relation [32, 73] between A ′ and the corresponding node information ( i.e. , X ′ and Y ′ ), we jointly input X ′ and Y ′ as conditions to generate A ′ . Therefore, the final version of the generative model can be written as A ′ = GEN( Z ( N ′ ) ⊕ X ′ ⊕ Y ′ ; Φ) , the ⊕ denotes the concatenate operation.

To study the performance of SGDD in the above paradigm, we theoretically prove the upper bound of LED shift in SGDD by invoking graphon theory. The result can be presented as follows.

Proposition 2. The upper bound of the LED shift on SGDD is shown as:

where δ □ denotes the cut distance [54] and W A is the graphon of A . See details in Appendix B.2.

Note minimizing the upper bound of Eq. (6) is equal to optimizing the L structure on Eq. (5). Compared to the lower bound in (Eq. (3)), our upper bound is not related to any frequency response of specific GNN ( i.e. , the terms of ∑ a i =1 || ¯ x 2 i -¯ x ′ 2 i || + ∑ N ′ i = b || ¯ x 2 i -¯ x ′ 2 i || ). As a result, SGDD may perform better than the previous work, especially in the cross-architecture setting and specific tasks.

4.2 Optimizing the graph structure via optimal transport

To mitigate the LED shift between A and A ′ , ideally, we can directly minimize the proposed SC . However, SC requires an extremely time-consuming ( O ( N 3 )) [4, 58] eigenvalue decomposition operation. Therefore, we propose the LED Matching strategy based on the optimal transport theory to form an efficient optimizing process.

Recall in the Eq.(4) of calculating SC , we first decompose the eigenvalue, followed by aligning the node through the JS divergence, which essentially compares the distribution proportions. The key point is to decide the node mapping strategy to align such two graphs. Alternatively, assuming we know the prior distribution of the alignment of S ( i.e. , we know the bijection of nodes in S to the G ) and denoting such alignment as S∗ , we can directly measure the distance between G and S∗ by employing the 2-Wasserstein metric[15, 56].

Furthermore, following the assumption in previous work[56, 15], we have η G ∼ N ( 0 , L † G ) and η S ∼ N ( 0 , L † S∗ ) . Then, the Eq.(7) have a closed-form expression[15] as follows:

the L † · denotes the Laplacian pseudo-inverse operation and the tr indicates the trace operation of matrix. Therefore, even though we could not know the actual mapping strategy S∗ , we can use the infimum of all possible strategies as a proxy solution. Formally, following the prior work [56], we employ the function T as a transport plan in the metric space X to represent all feasible mapping strategies. Then, we use the T # η S to represent the pushing forward process of transferring the distribution of η S to the η G . As a result, the distance can be regarded as finding the infimum.

Here, due to the transport plan T is impractical in optimizing, following the previous work[15], we use P ∈ R N ′ × N denotes as a free parameter serving as the direct mapping strategy between nodes. Thus, the Distance in the Eq. (5) could be directly optimized by the Eq.(9) ( i.e. , use the P represents all possible mapping strategies, the optimizing of P is equal to choosing a more optimal mapping strategy). In the experimental setting, we use the Sinkhorn-Knopp[9] algorithm to optimize P .

The overall time complexity is reduced to the O ( N ω ) ≤ O ( N 2 . 373 ) [15]. Note that the L † G may be too large for computing, so we empirically sample a medium size ( e.g. , 2,000 nodes) sub-structure in the experiment and ablate its influence in Appendix C.7.

4.3 Training pipeline of SGDD

As illustrated in Fig. 1(d) and Fig. 3, we commence by introducing a novel graph structure learning paradigm termed "graphon approximation". This paradigm integrates both the feature X ′ and auxiliary information Z to generate the structure. Subsequently, the learned structure A ′ is forced to be closer to the original graph structure A in terms of Laplacian energy distribution.

Additionally, our proposed methodology, SGDD, implements a bi-loop optimization schema. Within this framework, we concurrently optimize the parameters X ′ and A ′ . More specifically, the refinement of X ′ is achieved through a gradient matching strategy, whereas the A ′ is enhanced using the LED matching technique. During each step, the other component is frozen to ensure effective refinement.

Our overall training loss function can be summarized as L = L feature + α L structure + β || A || 2 , where the || A || 2 is proposed as a sparsity regularization term and L feature denotes the gradient matching strategy [37]. α and β are trade-off parameters, we study their sensitiveness in the Sec. 5.3. The algorithm can be found in Appendix C.3.

5 Experiments

5.1 Datasets and implementation details

Datasets. We evaluate SGDD on five node classification datasets: Cora [41], Citeseer [41], Ogbnarxiv [29], Flickr [100], Reddit [26], two anomaly detection datasets: YelpChi [68], Amazon [102], and two link prediction datasets: Citeseer-L [97], DBLP [83]. For the node classification and anomaly detection tasks, we follow the public settings [18] of train and test. To make a fair comparison, we also follow the previous setting [101, 83], we randomly split 80% nodes for training, 10% nodes for validation, and the remaining 10% for testing. To avoid data leakage, we only utilize 80% training samples for condensation. More details of each dataset can be found in Appendix C.1.

Ogbn-arxiv

45

50

55

60

65

70

Performance/(%)

64.1 63.7

66.0

67.2

50.1

50.1

56.7

58.1

GCond

SGDD w/o A

SGDD w/o X'

SGDD

(a) Ablation of

X

′

and

A

.

0.02 0.2

2

5

10

20

40

45

50

55

60

65

F1-macro (%)

Original

GCond

GDC

Ours

(b) Evaluation of

r

(%)

0

1

e

3

1

e

2

0.1

1

10

45

50

55

60

65

70

75

Performance (%)

Ogbn-arxiv

(c) Evaluation of

α

0

1

e

3

1

e

2

0.1

1

10

0

15

30

45

60

75

Sparsity/F1(%)

F1-macro

Sparsity

(d) Evaluation of

β

Implementation details. Without specific designation, in the condense stage , we adopt the 2-layer GCN with 128 hidden units as the backbone, and we adopt the settings on [94], which use 2-layer MLP to represent the structure generative model ( i.e. , GEN ). The learning rates for structure and feature are set to 0.001 (0.0001 for Ogbn-arxiv and Reddit) and 0.0001, respectively. We set α to 0.1, and β to 0.1. In the evaluation stage , we train the same network for 1,000 epochs on the condensed graph with a learning rate of 0.001. Following the settings in [37], we repeat all experiments ten times and report average performance and variance. More details 4 can be found in Appendix C.2.

5.2 Comparison with state-of-the-art methods

We compare our proposed SGDD with six baselines: Random, which randomly selected nodes to form the original graph, corset methods (Herding [90] and K-Center [72]), graph coarsening methods (Corasening [30]), and the state-of-the-art graph condensation methods (GCond [37]). GDC is proposed as a baseline in [37], where cosine similarity is added as a constraint to generate the structure of the condensed graph. In Table 1, we present the performance metrics including accuracy, F1-macro, and AUC. For clarity and simplicity, percentages are represented without the % symbol, and variance values are provided. Based on the results, we have the following observations: 1) Our proposed SGDD achieves the highest results in most settings, which shows the superiority and generalization of our method. 2) On anomaly detection datasets, the improvements are more significant than other tasks, i.e. , improving GCond with 9.6% and 7.7% on YelpChi and Amazon datasets, which can be explained that our method captures the structure information from the original graph dataset more efficiently.

5.3 Ablation Study

Cross-architecture generalization analysis. To evaluate the generalization ability of SGDD on unseen architectures, we conduct experiments that train on the condensed graph with different

architectures and report their performances in Tab. 2. Here, the condensed graph is obtained by optimizing SGDD with SGC [92]. We test its cross-architecture generalization performances on 2-layer-MLP [11], GAT [84], APPNP [42], Cheby [13], GCN [41], and SAGE [26].

To better understand, we also show several statistics metrics, including Avg., Std., and ∆ . SGDD and GCond improves GDC significantly, which indicates there exists a large difference between graph dataset condensation and vision dataset distillation, i.e. , the structure information should be specially considered. Compared to GCond, the improvement of our method is up to 11.3%, demonstrating the effectiveness of broadcasting the original structure to the condensed graph structure generation. More experiments on other datasets can be found in Appendix C.6.

Versatility of SGDD. Following the setting of GCond [37], we also study whether our proposed SGDD is robust on various architectures. We first condense the Ogbn-arxiv graph dataset with five architectures, including APPNP [42], Cheby [13], GCN [41], SAGE [26], and SGC [92], respectively. Then, we evaluate these condensed graphs on the above five architectures and report their performances in Tab. 3. The experiment results show that SGDD achieves non-trivial improvements than GCond in most cases, which demonstrates the strong versatility of our method.

Evaluation on neural architecture search. Similar to vision dataset distillation, graph dataset distillation is also expected to reduce the high cost of neural architecture search (NAS). In order to make a fair comparison, we follow the experimental setting in [37]: searching architectures on condensed Obgn-arxiv, YelpChi, and DBLP datasets with 0.25%, 0.2%, and 0.5% condensing ratios. We report Pearson correlation [45] and performance of random, GCond, and SGDD in Tab. 4. Our SGDDconsistently achieves the highest Pearson correlations as well as performances, which indicates the architectures searched by our method are efficient for the whole graph dataset training.

Evaluation of components in SGDD. To explore the effect of the conditions (mentioned in Sec. 4.1) in the condensed graph generation, we design the ablation study of X ′ and A . As shown in Fig 4(a), X ′ and A are complementary with each other. SGDD w/o A performs poorly on Obgn-arxiv and YelpChi datasets, which demonstrates the effect of original graph structure information. Jointly using X ′ and A achieves the highest performances on both datasets, improves GCond with 3.1% on Ogbn-arxiv, and 8.0% on YelpChi.

Evaluation of the scalability of SGDD. To investigate the scalability of SGDD, we evaluate the SGDD on various condensing ratios with r ∈ { 0 . 02 , 0 . 2 , 2 , 5 , 10 , 20 } . As shown in Fig. 4(b), the performance of our method continuously increase as the condensing ratio rises, which indicates the strong scalability of our method. GCond obtains marginal improvements than GDC at all ratios while our SGDD outperforms them significantly. More important, SGDD achieves lossless performance as training on the original graph data when the condensing ratio is 10%.

Exploring the sensitivity of α and β . We conduct experiments to test such two parameters sensitivity on YelpChi, Cora, and Ogbn-arxiv. As shown in Fig. 4(c), we empirically find that the performance of our SGDD is not sensitive to the α . Specifically, compared to the case that α is zero, we have a significant improvement, which proves the effectiveness of our method. Another finding is that the α should be set higher on anomaly detection than on node classification tasks. It could be explained by the original graph structure information being more important on the complex task (such as anomaly detection). We define β as a regularization coefficient to control the sparsity of the condensed graph. As shown in Fig. 4(d), we evaluate the β from 0 to 10 on the YelpChi dataset. The results illustrate that the performance is not sensitive with β and achieve the highest result (F1-macro) when β is set to our default value ( β = 0 . 1 ). More experiments can be found in Appendix C.5.

SC:0.0

(a) Original Ogbn-arxiv

SC:0.31

ACC: 64.1

(b) Condensed by GCond [37]

SC:0.22

ACC: 67.3

(c) Condensed by SGDD

SC:0.0

(d) Original synthetic graph

SC:0.45

ACC: 66.5

(e) Condensed by GCond [37]

SC:0.17

ACC: 87.6

(f) Condensed by SGDD

D

1.0

1.0

0.5

20

- 0.8

20

20

- 0.8

0.4

40

- 0.6

40

40

0.6

0.3

60

0.4

60

60

0.4

0.2

80

0.2

80

- 0.1

80

0.2

100

- 0.0

100

100

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

5.4 Visualizations

To better understand the effectiveness of our SGDD, we visualize the condensed graphs of GCond and SGDD that are synthesized from real and synthetic graph datasets. For the synthetic graph dataset, we use the Stochastic Block Model (SBM) [28] to synthesize graphs with 5 community (Fig. 5(d)). As shown in Fig. 5, one can find that our method consistently achieves better performances and SC . SGDD reduces 29.0% (comparing 5(b) and 5(c)) and 62.2% (comparing 5(e) and 5(f)) SC on Ogbn-arxiv and synthetic datasets. Visually, the condensed graphs of our method preserve the original graph structure information obviously better than GCond (see the second row of Fig. 5), which proves SGDD is a powerful graph dataset distillation method.

6 Conclusion

We present SGDD, a novel framework for graph dataset distillation via broadcasting the original structure information to the generation of the synthetic one. SGDD shows its robustness on various tasks and datasets, achieving state-of-the-art results on YelpChi, Amazon, Ogbn-arxiv, and DBLP. SGDD reduces the scale of the Yelpchi dataset by 1,000 times while maintaining 98.6% as training on the original data. We provide sufficient experiments and theoretical analysis in this paper and hope it can help the following research in this area. Limitations and future work: Although broadcasting the original information to the generated graph shows remarkable success, some informative properties ( e.g. , the heterogeneity) may lose during the current condense process, which results in sub-optimal performance in the downstream tasks. We are going to explore a more general method in the future.

References

Muhammet Balcilar, Guillaume Renton, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Analyzing the expressive power of graph neural networks in a spectral perspective. In 9th International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021 , 2021.

Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. ArXiv preprint , 2018.

Daniel Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Josh Tenenbaum, and Daniel L. Yamins. Learning physical graph representations from visual scenes. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, December 6-12, 2020, virtual , 2020.

Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

Chen Cai, Dingkang Wang, and Yusu Wang. Graph coarsening with neural networks. In 9th International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021 , 2021.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A. Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2022, New Orleans, LA, USA, June 19-20, 2022 , pages 4749-4758, 2022.

George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Generalizing dataset distillation via deep generative prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3739-3748, 2023.

Yu Chen, Lingfei Wu, and Mohammed J. Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, December 6-12, 2020, virtual , 2020.

Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Christopher J. C. Burges, Léon Bottou, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States , pages 2292-2300, 2013.

Évariste Daller, Sébastien Bougleux, Luc Brun, and Olivier Lézoray. Local patterns and supergraph for chemical graph classification with convolutional networks. In Structural, Syntactic, and Statistical Pattern Recognition - Joint IAPR International Workshop, S+SSPR 2018, Beijing, China, August 17-19, 2018, Proceedings , volume 11004, pages 97-106, 2018.

Tuan Van Dao, Hiroshi Sato, and Masao Kubo. Mlp-mixer-autoencoder: A lightweight ensemble architecture for malware classification. Inf. , 14(3):167, 2023.

Kinkar Ch. Das, Seyed Ahmad Mojallal, and Vilmar Trevisan. Distribution of laplacian eigenvalues of graphs. Linear Algebra and its Applications , 508:48-61, 2016.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information

Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 3837-3845, 2016.

Chenhui Deng, Zhiqiang Zhao, Yongyu Wang, Zhiru Zhang, and Zhuo Feng. Graphzoom: A multi-level spectral approach for accurate and scalable graph embedding. In 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, April 26-30, 2020 , 2020.

Yihe Dong and Will Sawin. COPT: coordinated optimal transport on graphs. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, December 6-12, 2020, virtual , 2020.

Holger Ebel and Stefan Bornholdt. Coevolutionary games on networks. Physical Review E , 66(5):056118, 2002.

Justin Eldridge, Mikhail Belkin, and Yusu Wang. Graphons, mergeons, and so on! In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 2307-2315, 2016.

Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. ArXiv preprint , 2019.

Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 1972-1982, 2019.

Alan Frieze and Ravi Kannan. Quick approximation to matrices and applications. Combinatorica , 19(2):175-220, 1999.

Shuang Gao and Peter E Caines. Graphon control of large-scale networks of linear systems. IEEE Transactions on Automatic Control , 2019.

Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao, and Chunming Rong. A survey on dataset distillation: Approaches, applications and future directions. ArXiv preprint , abs/2305.01975, 2023.

Jianyang Gu, Kai Wang, Wei Jiang, and Yang You. Summarizing stream data for memoryrestricted online continual learning. ArXiv preprint , abs/2305.16645, 2023.

Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. ArXiv preprint , abs/2310.05773, 2023.

Ivan Gutman and Bo Zhou. Laplacian energy of a graph. Linear Algebra and its Applications , 414(1):29-37, 2006.

William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 1024-1034, 2017.

Peter D Hoff, Adrian E Raftery, and Mark S Handcock. Latent space approaches to social network analysis. Journal of the american Statistical association , 97(460):1090-1098, 2002.

Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social networks , 5(2):109-137, 1983.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and HsuanTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, December 6-12, 2020, virtual , 2020.

Zengfeng Huang, Shengzhong Zhang, Chong Xi, Tang Liu, and Min Zhou. Scaling up graph neural networks via graph coarsening. In In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '21) , 2021.

Huawei. Mindspore. https://www.mindspore.cn/ , 2020.

Joseph J. Pfeiffer III, Sebastián Moreno, Timothy La Fond, Jennifer Neville, and Brian Gallagher. Attributed graph models: modeling network structure with correlated attributes. In Chin-Wan Chung, Andrei Z. Broder, Kyuseok Shim, and Torsten Suel, editors, 23rd International World Wide Web Conference, WWW '14, Seoul, Republic of Korea, April 7-11, 2014 , pages 831-842, 2014.

Svante Janson. Graphons, cut norm and distance, couplings and rearrangements. New York journal of mathematics , 2013.

Svante Janson and Persi Diaconis. Graph limits and exchangeable random graphs. Rendiconti di Matematica e delle sue Applicazioni. Serie VII , pages 33-61, 2008.

Wei Jin, Yao Ma, Xiaorui Liu, Xianfeng Tang, Suhang Wang, and Jiliang Tang. Graph structure learning for robust graph neural networks. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020 , pages 66-74, 2020.

Wei Jin, Xianfeng Tang, Haoming Jiang, Zheng Li, Danqing Zhang, Jiliang Tang, and Bing Yin. Condensing graphs via one-step gradient matching. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2022.

Wei Jin, Lingxiao Zhao, Shichang Zhang, Yozen Liu, Jiliang Tang, and Neil Shah. Graph condensation for graph neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 , 2022.

David R. Karger. Random sampling in cut, flow, and network design problems. Math. Oper. Res. , 24(2):383-413, 1999.

Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 11102-11118, 2022.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, Toulon, France, April 24-26, 2017, Conference Track Proceedings , 2017.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, Toulon, France, April 24-26, 2017, Conference Track Proceedings , 2017.

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In 7th International Conference on Learning Representations, New Orleans, LA, USA, May 6-9, 2019 , 2019.

Eric D. Kolaczyk. Statistical Analysis of Network Data: Methods and Models . IEEE, 2009.

Saehyung Lee, Sanghyuk Chun, Sangwon Jung, Sangdoo Yun, and Sungroh Yoon. Dataset condensation with contrastive signals. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 12352-12364, 2022.

Guanzhi Li, Aining Zhang, Qizhi Zhang, Di Wu, and Choujun Zhan. Pearson correlation coefficient-based performance enhancement of broad learning system for stock price prediction. IEEE Trans. Circuits Syst. II Express Briefs , 69(5):2413-2417, 2022.

Jianxin Li, Qingyun Sun, Hao Peng, Beining Yang, Jia Wu, and S Yu Phillp. Adaptive subgraph neural network with reinforced critical structure mining. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theory , 37(1):145-151, 1991.

Zhiqi Lin, Cheng Li, Youshan Miao, Yunxin Liu, and Yinlong Xu. Pagraph: Scaling gnn training on large graphs via computation-aware caching. In Proceedings of the 11th ACM Symposium on Cloud Computing , 2020.

Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. In 7th International Conference on Learning Representations, New Orleans, LA, USA, May 6-9, 2019 , 2019.

Mengyang Liu, Shanchuan Li, Xinshi Chen, and Le Song. Graph condensation via receptive field distribution matching. ArXiv preprint , abs/2206.13697, 2022.

Yanqing Liu, Jianyang Gu, Kai Wang, Zheng Zhu, Wei Jiang, and Yang You. Dream: Efficient dataset distillation by representative matching. ArXiv preprint , abs/2302.14416, 2023.

Andreas Loukas. Graph reduction with spectral and cut guarantees. J. Mach. Learn. Res. , 20:116:1-116:42, 2019.

Andreas Loukas and Pierre Vandergheynst. Spectrally approximating large graphs with smaller graphs. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 3243-3252, 2018.

László Lovász. Large networks and graph limits , volume 60. IEEE, 2012.

Dongsheng Luo, Wei Cheng, Wenchao Yu, Bo Zong, Jingchao Ni, Haifeng Chen, and Xiang Zhang. Learning to drop: Robust graph neural network via topological denoising. In WSDM , pages 779-787, 2021.

Hermina Petric Maretic, Mireille El Gheche, Giovanni Chierchia, and Pascal Frossard. GOT: an optimal transport framework for graph comparison. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 13876-13887, 2019.

Brendan D. McKay, Mehmet Aziz Yirik, and Christoph Steinbeck. Surge: a fast open-source chemical graph generator. J. Cheminformatics , 2022.

Facundo Mémoli. Spectral Gromov-Wasserstein distances for shape matching. In ICCV Workshops , pages 256-263, 2009.

Matthew W. Morency and Geert Leus. Graphon filters: Graph signal processing in the limit. IEEE Trans. Signal Process. , 69:1740-1754, 2021.

Bilal Nehme, Olivier Strauss, and Kevin Loquin. Estimating the variance of a kernel density estimation. In Christian Borgelt, Gil González-Rodríguez, Wolfgang Trutschnig, María Asunción Lubiano, María Ángeles Gil, Przemyslaw Grzegorzewski, and Olgierd Hryniewicz, editors, Combining Soft Computing and Statistical Methods in Data Analysis, SMPS 2010, Oviedo, Spain, September 29 - October 1, 2010 , volume 77 of Advances in Intelligent and Soft Computing , pages 483-490, 2010.

Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In 9th International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021 , 2021.

Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide convolutional networks. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, December 6-14, 2021, virtual , pages 5186-5198, 2021.

Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Adversarially regularized graph autoencoder for graph embedding. In Jérôme Lang, editor, Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden , pages 2609-2615, 2018.

Francesca Parise and Asuman Ozdaglar. Graphon games: A statistical framework for network games and interventions. ArXiv preprint , abs/1802.00080, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 8024-8035, 2019.

David Peleg and Alejandro A. Schäffer. Graph spanners. J. Graph Theory , 13(1):99-116, 1989.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: online learning of social representations. In Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani, editors, The 20th ACM International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA - August 24 - 27, 2014 , pages 701-710, 2014.

Shebuti Rayana and Leman Akoglu. Collective opinion spam detection: Bridging review networks and metadata. In Longbing Cao, Chengqi Zhang, Thorsten Joachims, Geoffrey I. Webb, Dragos D. Margineantu, and Graham Williams, editors, Proceedings of the 21th ACM International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, August 10-13, 2015 , pages 985-994, 2015.

Luana Ruiz, Luiz F. O. Chamon, and Alejandro Ribeiro. The graphon fourier transform. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, Spain, May 4-8, 2020 , pages 5660-5664, 2020.

Luana Ruiz, Luiz F. O. Chamon, and Alejandro Ribeiro. Graphon neural networks and the transferability of graph neural networks. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, December 6-12, 2020, virtual , 2020.

Luana Ruiz, Luiz FO Chamon, and Alejandro Ribeiro. Graphon signal processing. IEEE Transactions on Signal Processing , 69:4961-4976, 2021.

Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In 6th International Conference on Learning Representations, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018.

Cosma Rohilla Shalizi and Andrew C. Thomas. Homophily and contagion are generically confounded in observational social network studies. CoRR , abs/1004.4704, 2010.

Chao Shang, Jie Chen, and Jinbo Bi. Discrete graph structure learning for forecasting multiple time series. In 9th International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021 , 2021.

Mingjia Shi, Yuhao Zhou, Kai Wang, Huaizheng Zhang, Shudong Huang, Qing Ye, and Jiancheng Lv. PRIOR: Personalized prior for reactivating the information overlooked in federated learning. In Proceedings of the 37th NeurIPS , 2023.

Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, December 6-12, 2020, virtual , 2020.

Daniel A. Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. SIAM J. Comput. , 40(6):1913-1926, 2011.

Daniel A. Spielman and Shang-Hua Teng. Spectral sparsification of graphs. SIAM J. Comput. , 40(4):981-1025, 2011.

Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Xingcheng Fu, Cheng Ji, and Philip S. Yu. Graph structure learning with variational information bottleneck. In Thirty-Sixth AAAI Conference on Artificial Intelligence, Virtual Event, February 22 - March 1, 2022 , pages 4165-4174, 2022.

Qingyun Sun, Jianxin Li, Hao Peng, Jia Wu, Yuanxing Ning, Philip S Yu, and Lifang He. Sugar: Subgraph neural network with reinforcement pooling and self-supervised mutual information mechanism. In The International Conference of World Wide Web 2021 , pages 2081-2091, 2021.

Qingyun Sun, Jianxin Li, Haonan Yuan, Xingcheng Fu, Hao Peng, Cheng Ji, Qian Li, and Philip S Yu. Position-aware structure learning for graph topology-imbalance by relieving underreaching and over-squashing. In International Conference on Information and Knowledge Management , pages 1848-1857, 2022.

Jianheng Tang, Jiajin Li, Ziqi Gao, and Jia Li. Rethinking graph neural networks for anomaly detection. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 21076-21089, 2022.

Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Arnetminer: Extraction and mining of academic social networks. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '08, page 990-998, New York, NY, USA, 2008.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings , 2018.

Renato Vizuete, Federica Garin, and Paolo Frasca. The laplacian spectrum of large graphs sampled from graphons. IEEE Transactions on Network Science and Engineering , page 1711-1721, Mar 2021.

Kai Wang, Jianyang Gu, Daquan Zhou, Zheng Zhu, Wei Jiang, and Yang You. Dim: Distilling dataset into generative model. ArXiv preprint , abs/2303.04707, 2023.

Kai Wang, Bo Zhao, Xiangyu Peng, Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In CVPR , 2022.

Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. ArXiv preprint , 2018.

Yili Wang, Kaixiong Zhou, Rui Miao, Ninghao Liu, and Xin Wang. Adagcl: Adaptive subgraph contrastive learning to generalize large-scale graph training.

Max Welling. Herding dynamical weights to learn. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman, editors, Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, Quebec, Canada, June 14-18, 2009 , volume 382 of ACM International Conference Proceeding Series , pages 1121-1128, 2009.

Gert W. Wolf. Facility location: concepts, models, algorithms and case studies. series: Contributions to management science. Int. J. Geogr. Inf. Sci. , 25(2):331-333, 2011.

Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 6861-6871, 2019.

Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu. A comprehensive survey on graph neural networks. ArXiv preprint , 2019.

Xinyue Xia, Gal Mishne, and Yusu Wang. Implicit graphon neural representation. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics, , 2023.

Hongteng Xu, Dixin Luo, Lawrence Carin, and Hongyuan Zha. Learning graphons via structured gromov-wasserstein barycenters. In Thirty-Fifth AAAI Conference on Artificial Intelligence, Virtual Event, February 2-9, 2021 , pages 10505-10513, 2021.

Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. ArXiv preprint , abs/2205.09329, 2022.

Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Maria-Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, New York City, NY, USA, June 19-24, 2016 , volume 48 of JMLR Workshop and Conference Proceedings , pages 40-48, 2016.

Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Yike Guo and Faisal Farooq, editors, Proceedings of the 24th ACM International Conference on Knowledge Discovery & Data Mining, London, UK, August 19-23, 2018 , pages 974-983, 2018.

Haiyang Yu, Limei Wang, Bokun Wang, Meng Liu, Tianbao Yang, and Shuiwang Ji. Graphfm: Improving large-scale GNN training via feature momentum. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 25684-25701, 2022.

Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graphsaint: Graph sampling based inductive learning method. In 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, April 26-30, 2020 , 2020.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, December 3-8, 2018, Montréal, Canada , pages 5171-5181, 2018.

Shijie Zhang, Hongzhi Yin, Tong Chen, Quoc Viet Hung Nguyen, Zi Huang, and Lizhen Cui. Gcn-based user representation learning for unifying robust recommendation and fraudster detection. In Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM conference on research and development in Information Retrieval, Virtual Event, China, July 25-30, 2020 , pages 689-698, 2020.

Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. ArXiv preprint , abs/2211.13976, 2022.

Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. ArXiv preprint , abs/2110.04181, 2021.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In 9th International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021 , 2021.

Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In 9th International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021 , 2021.

Tong Zhao, Yozen Liu, Leonardo Neves, Oliver J. Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In Thirty-Fifth AAAI Conference on Artificial Intelligence, Virtual Event, February 2-9, 2021 , pages 11015-11023, 2021.

Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In Proceedings of the 37th International Conference on Machine Learning, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 11458-11468, 2020.

Xin Zheng, Miao Zhang, Chunyang Chen, Quoc Viet Hung Nguyen, Xingquan Zhu, and Shirui Pan. Structure-free graph condensation: From large-scale graphs to condensed graph-free data. In Advances in Neural Information Processing Systems , 2023.

Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 17205-17216, 2023.

Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. ArXiv preprint , 2018.

A More Preliminary and Related Work

A.1 More preliminary

Graphon. A graphon [43, 34, 54] is a bounded, symmetric, and Lebesgue measurable function, denote as W : Ω 2 → [0 , 1] , where Ω is a probability measure space. By randomly selecting points { v 0 , v 1 , . . . , v n } from Ω , we can create a graph of any size by connecting each point with an edge weight e ij = W ( v i , v j ) . Formally, we can follow the random sampling process to get arbitrarily sized ( n ) graphs :

Conventionally, we use a deterministic setting to select the node, e.g. , the fixed grid v i = i -1 n . Graphs derived from the same graphon share several important properties for statistical analysis: such as density [21], clustering coefficient [70], and degree distribution [71], which motivate us to broadcast the original graph structure information to the condensed graph via the graphon approximating.

A.2 More related work

Graph structure learning. Graph structure learning [19, 35, 8, 108, 55] also aims at jointly optimizing the graph structure and the corresponding node features. Most of these methods learn the new structure based on the certain constraints ( e.g. , low-rank [91], sparsity [8], feature smoothness [35], and homophily [108]). However, these methods are unable to learn a new structure with a significantly reduced node size. In contrast, our method can synthesize a smaller graph structure while simultaneously constructing new node features. The obtained small but informative graph can reduce the training cost in the downstream tasks.

B Proofs

B.1 Proof of Proposition 1

Proof. First, the gradient matching objective in GCond [37] can be shown as:

where Match( · ) is the matching function that measures the distance of the two gradients. θ represents the parameters of the GNN, the ∇ denotes the gradient in the backpropagation process, and the L cls represents the loss function that is used in the supervised tasks, i.e. , cross-entropy loss. Following the discussion in the [4, 1, 82], the GNN can be viewed as a bandpass filter in the spectral domain. Expanding the GNN in the spectral domain, we can rewrite the objective as:

where the T( λ ) denotes the frequency response of the given GNN, the λ and the U is the eigenvalue and eigenvector of the corresponding graph, respectively. Note we drop the ∇ , the L cls , and the Y ( Y ′ ) terms because they can be viewed as the intermediate process [105, 107] in approximating to Eq.(12). We use the MSE [37] as the matching function Match( · ) for simplicity.

To simplify T( λ ) , without loss of generality, we assume the bandwidth of the specific GNN's frequency response is (a, b), which intuitively indicates that only the signal of frequency in (a, b) can pass through the filter on the graph. Then the Eq. (12) can be written as:

we further combine the first integration, the objective can thus be summarized as:

Then following the transformation in the spectral domain [1], we can briefly summarized:

Take the Eq. (15) into the ∥ ∥ η G -η S ∥ ∥ :

Note the second inequality is under the assumption: intuitively, the overall distance of η G and η S should be larger than the condition that two graphs have the same size ( i.e. , N = N ′ ).

B.2 Proof of Proposition 2

To prove Proposition 2, we first introduce the following notations and theorems in graphon theory.

The cut norm [20, 54] is defined as:

where the Ω is the probability space of a graphon W , the supremum is taken over all measurable subsets X and Y [95]. Then the cut distance between W 1 , W 2 [54] can be defined as:

where the S Ω is the set of measure-preserving mappings from Ω to Ω . When two graphons W 1 , W 2 have δ □ ( W 1 , W 2 ) = 0 , they can be seen equivalent, denoted as W 1 ∼ = W 2 .

To effectively approximate W in the real world graphs, works [54, 33, 17] introduce an approximation named step function. Let P = ( P 1 , . . . , P K ) be a partition of Ω into K measurable sets. The step function W P : Ω 2 ↦→ [0 , 1] is defined as:

where the w kk ′ ∈ [0 , 1] and the indicator function 1 P k ×P k ′ ( x, y ) is 1 if ( x, y ) ∈ P k ×P k ′ , or it is 0. To explore the relationship between step function and the W , we introduce the Weak Regularity Lemma as follows.

Theorem 1 (Weak Regularity Lemma [54]) . For every graphon W ∈ W and K ≤ 1 , there always exists a step function W P with |P| = K steps such that

We can further obtain the corollary that δ □ ( W,W P ) ≤ 2 √ log K ∥ W ∥ L 2 as the δ □ ( W,W P ) ≤ ∥ W -W P ∥ □ . Intuitively, we can use any step function to approximate the ideal W of real graphs.

Then to investigate the properties of the graphon in the spectral domain, following [71, 85, 59, 69], we have the conclusion that:

where the S denotes the signal spectrum of the graphs [85] ( i.e. , the post-Graph-Fourier-Transform U ⊤ X ), Intuitively, the Eq. (21) shows that when the step function W P is approximated to the W , they have similar signal spectrum.

Proof. By combining the Theorem 1 and Eq.(21), the Proposition 2 replace the W P with the W ′ A , that's because we use the generative model ( i.e. , GEN( · ) ) to approximate the step function W P . As we aim to learn the graphon of the original structure A , the graphon W can be rewrite as W A ,

We notice that the S here is related to the Laplacian energy distribution (LED), where the LED represents the probability distribution of the S , then we use the ∑ N i =1 ¯ x i and ∑ N ′ j =1 ¯ x j to represent the summation of the original signal spectrum and the condensed one, respectively, we have:

where in the second inequality, we drop the max ( · ) term since it always lower than 1. As the W A ′ here represents the graphon of the synthetic A ′ , we can use the A ′ directly in the δ □ to form a compact upper bound.

Note that minimizing the upper bound has been proven to be equivalent to minimizing the optimal transport distance between the two graphs [95].

B.3 Time complexity analysis and running time

Time complexity. For simplicity, let the number of MLP layers in GEN( · ) be L , and all the hidden units are d . In the forward process , we have three steps: first, we calculate the A ′ by the GEN( · ) , which have the complexity of O ( N ′ 2 d 2 ) . Second, the forward process of GNN on the original graph has a complexity of O ( m L Nd 2 ) , where the m denotes the sampled size per node in training. Third, the complexity of training on the condensed graph is O ( LN ′ d ) . In the backward process , the complexity of gradient matching strategy ( i.e. , L feature ) is O ( | θ || X ′ | ) [37]. For the structure optimization term ( i.e. , L structure ), the complexity is O ( N ′ 2 k + NN ′ 2 ) . The overall complexity of SGDD can be represented as O ( N ′ 2 d 2 )+ O ( m L Nd 2 )+ O ( LN ′ d )+ O ( | θ || X ′ | )+ O ( N ′ 2 k + N ′ 2 N ) . Note N ′ ≪ N , we can drop the terms that only involve N ′ and constants ( e.g. , the number of L and m ). The final complexity can be simplified as O ( m L Nd 2 ) + O ( N ′ 2 N ) , thus the complexity of SGDD still be linear to the number of nodes in the original graph.

Running time. We report the running time of the SGDD in the two datasets: Ogbn-arxiv, and YelpChi. We vary the condensing ratio r in the range of {0.05%, 0.25%, 0.50%} for Ogbn-arxiv and {0.05%, 0.10%, 0.20%} for YelpChi. All experiments are conducted five times on one single A100-SXM4 GPU. We also compare our results to those obtained using GCond [37] under the same settings. As shown in the Tab. 5, our approach achieves a similar running time to GCond when the condensing ratio was low ( i.e. , r = 0 . 05% for both datasets), and is 10% faster when the condensing ratio increased. The difference can be explained by the efficient generative model we employed for generating structure, which prevents the consumption of time-complex operations such as calculating the pair-wised feature similarity ( O ( N ′ 2 ) ).

C Experimental Details and More Experiments

C.1 Dataset statistics

We evaluate the proposed SGDD on nine datasets, including five node classification datasets: Cora [41], Citeseer [41], Ogbn-arxiv [29], Flickr [100], and Reddit [26]; two anomaly detection datasets: YelpChi [68] and Amazon [102]; two link prediction datasets Citeseer-L [97] and DBLP [83]. We report the dataset statistics in Tab. 6.

Citeseer-L: We use the Citeseer in the link prediction setting, named Citeseer-L. We randomly sample 80% nodes in training, 10% nodes in validation, and the remaining 10% nodes for testing. The classes here denote 'have edge' and 'do not have edge'.

DBLP: We treat the original graph as a homogeneous graph here.

C.2 Implementation details

Structure in GDC. We utilize the DC [107] as our baseline, and to incorporate the structure information, we add the constraint to produce a graph structure, named Graph DC (GDC). Specifically, we use the cosine similarity function [8] (formally, A ′ ij = cos( X ′ i , X ′ j ) ) to generate structure, where X ′ i and X ′ j are the learned features obtained through the vanilla gradient matching strategy.

GEN( · ) in SGDD. We introduce the GEN( · ) as our generative model in Sec. 4.1. Here, we show the implementation details of this module.

To start, we aim to find a method to broadcast the original graph structure A to condensed graph A ′ . Motivated by that all graphs with the same graphon W will exhibit similar properties, we can simply calculate the graphon W of A and leverage it in the condensing process. Nevertheless, directly calculating W of A is not feasible since conventional graphon learning methods should summarize graphon from a set of graphs [21, 33, 64] (We only have one graph A ). Recent advancements of IGNR [94] demonstrate the potential of utilizing the generative approach to approximate the W . Specifically, given that the graphon W is defined as Ω 2 → [0 , 1] (as described in Appendix A.1), we can similarly construct the function f as follows.

the continuous space Ω 2 is defined by R 2 . For computational convenience, the input space can further be limited to [0 , 1] 2 [94], then the Eq. (25) is transformed to sample points to reconstruct data, following IGNR[94], we use SIREN[76] as f ,

where the Z ( N ′ ) ∈ R N ′ × N ′ is a random noise that plays as the coordinates, and the learnable weights Φ = { W i ∈ R l i × l i +1 , b i ∈ R l i , for i = 1 , · · · , l } map the pair of points to the edge probability. GEN( Z ( N ′ ); Φ) ∈ R N ′ × N ′ is equal to represent the adjacency matrix A ′ ∈ R N ′ × N ′

after transformation, where each entry represents a probability that each node pair should be connected. To incorporate the node information into the structure generation, we adopt them as conditional information that leads the generation process. We can then rewrite Eq. (25) to:

where the R d here is to present the conditional information, thus the learned model considers both the node's coordinates message along with the node's specific information. The basic implements can be:

where the ⊕ denotes the concatenate operation, the Y ′ is treated as a one-hot vector for dimensionality fit through a multilayer perceptron (MLP). This method allows for the incorporation of significant node information into the resulting synthetic graph.

Condensation stage. For GCond [37], we use the 2-layer SGC [92] to serve as the condensing architecture with 256 units, and tune the number of epochs in a range of {400, 500, 6000, 1000, 2000}. For GDC [107, 105], we tune the number of hidden layers in the range of {1, 2, 3} and the number of hidden units in the range of {128, 256}. For SGDD, we use the GCN [40] as the default condensing architecture and tune the number of hidden layers in a range of {1, 2, 3}. We further tune the number of epochs in a range {400, 500, 600, 1000, 2000}, and tune the learning rate in a range of {0.1, 0.01, 0.001, 0.0001}.

Evaluation stage. We set the training epoch to 1000 with an early stopping strategy for evaluating GNNs and set the dropout rate to 0 with the learning rate of 0.1.

Configurations. We conduct all experiments with:

Operating System: Ubuntu 20.04 LTS.

CPU: Intel(R) Xeon(R) Platinum 8358 CPU@2.60GHz with 1TB DDR4 of Memory.

GPU: NVIDIA Tesla A100 SMX4 with 40GB of Memory.

Software: CUDA 10.1, Python 3.8.12, PyTorch [65] 1.7.0.

C.3 Objective loss function and training algorithm

In this subsection, we present the objective function and provide a detailed training algorithm. Our objective is to jointly learn X ′ and A ′ . We follow GCond[37] to optimize X ′ as a free parameter using the gradient matching strategy[104], the loss can be expressed by Eq. (29),

Here, Match( · ) is the matching function that measures the distance of the two gradients, we use the MSE [37] in practice. θ represents the parameters of the specific backbone GNN, the ∇ denotes the gradient in the backpropagation process, and the L cls represents the loss function that is used in the supervised tasks, i.e. , cross-entropy loss. Our objective loss function can be written as:

where the α controls the contribution of the L strcuture term. To model the low-rank properties of real-world graphs, we use the || A ′ || 2 as a regularity to control the sparsification of A ′ with β .

We summarize our pipeline in Algorithm. 1.

C.4 Stochastic Block Model experments setting

In Sec. 5.4, we generate a synthetic graph dataset with different community structures using the Stochastic Block Model (SBM) ( N,C,p,q ) [28]. Here, we show the parameters settings, we set the number of nodes N to 100, while setting the number of communities C to 5. The parameter p represents the edge probability within the same community and q represents the edge probability between communities, we set them to 0.8 and 0.1 in practice, respectively.

C.5 More explorations of the sensitivity of β

In Fig. 4(d), we demonstrate that SGDD is not sensitive to the coefficient β on the YelpChi dataset. Here, we provide more experiments on the other datasets. In Fig. 6, we can see that increasing β leads to the more sparsity of the condensed graph, and the corresponding performance does not have severe drop. These observations demonstrate the effectiveness of the regularity term in our objective.





e

3



e

2

























e

3



e

2























(b) Citeseer





e

3



e

2























(c) Ogbn-arxiv





e

3



e

2























(d) Amazon





e

3



e

2























(e) DBLP





e

3



e

2























e

3



e

2























(f) Flickr





e

3



e

2





























e

3



e

2























(g) Reddit

C.6 More cross-architecture experiments

In this subsection, we further report the results of cross-architecture experiments on the YelpChi. As shown in Tab. 7, our SGDD improves the performance in all cases, the average improvements compared to the GCond [37] is 9.6%, which indicates the strong generalization performance of the condensed graph by SGDD.

C.7 Ablation of the sampling operation in OT distance

To further reduce condensing time and memory usage, we follow GCond [37] to sample from original graph A in the condensing stage (line 11 in the Algorithm 1). We evaluate the SGDD on various sample sizes, i.e. , {100, 500, 1000, 2000, 5000}, and report the corresponding performance and SC . As shown in Fig. 7, with the increase in the number of nodes, the performance obtains marginal improvements while the SC is decreasing. However, larger sample sizes are not always beneficial to performance. In this study, the highest results are obtained when the sample size is 2000. This result empirically demonstrates the scalability of the structure learning module. Specifically, the module enables efficient condensing on large graphs.



























































SC



(a) Ogbn-arxiv



























































SC



(b) YelpChi













































SC



(c) DBLP

C.8 More discussion of the condensed graphs.

We next show the comparison of condensed graphs and original graphs. As shown in Tab. 8, the condensed graphs obviously have fewer nodes and edges, while maintaining comparable accuracy, which shows the effectiveness of our SGDD . We also represent the visualizations of some condensed graphs. In Fig. 8, the black lines denote that edge weights are larger than 0.5 and the gray lines represent as smaller than 0.5. We can observe that there are some patterns in the condensed graph, e.g. , the homophily patterns on Cora and Citeseer. However, for the remaining datasets, the visualizations are inadequate in revealing the properties of the condensed graph, which proves the superiority of analyzing structure in spectral view.

C.9 More discussion of the large graphs.

We conduct experiments on the Ogbn-mag datasets and show the details as follows: (note we transform the original heterogeneous graph to the homogeneous by ignoring difference of

(b) Citeseer

(c) Ogbn-arxiv

(d) Flickr

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly focuses on the Laplacian Energy Distribution (LED) shift problem of graph dataset condensation.\n\nSTRENGTHS:\nThe pipeline figure is clear and straightforward.\n\nThe studied problem, how to condense a graph dataset, is relatively important.\n\nWEAKNESSES:\nCompared to existing studies that widely investigated the gradient matching, it seems that the major contribution of this paper is considering the LED shift. Why LED and how can LED benefit? The authors need to provide more analyses and explanation about their motivation. \n\nIn the Introduction, the authors mention “We empirically find a positive correlation between LED shift and the performance in cross-architecture settings”. This can only be considered as a finding. The motivation for introducing LED is ambiguous, especially given the correlation does not mean causality. \n\nThe authors simply compare their method  with Gcond and consider it as the SOTA. However, there are many works that have been proposed since Gcond. The authors should consider compare with them as well for a comprehensive evaluation of their method. For example, [1, 2, 3, 4, 5]\n\n[1] Jin, Wei, et al. \"Condensing graphs via one-step gradient matching.\" Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2022.\n\n[2] Yu, Ruonan, Songhua Liu, and Xinchao Wang. \"Dataset distillation: A comprehensive review.\" arXiv preprint arXiv:2301.07014 (2023).\n\n[3] Liu, Chuang, et al. \"Comprehensive graph gradual pruning for sparse training in graph neural networks.\" IEEE Transactions on Neural Networks and Learning Systems (2023).\n\n[4] Zheng, Xin, et al. \"Structure-free Graph Condensation: From Large-scale Graphs to Condensed Graph-free Data.\" arXiv preprint arXiv:2306.02664 (2023).\n\n[5] Yang, Shuo, et al. \"Dataset pruning: Reducing training data by examining generalization influence.\" arXiv preprint arXiv:2205.09329 (2022).\n\nQUESTIONS:\nsee above.\n\nLIMITATIONS:\nsee above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method called SGDD for condensing large-scale graph datasets while preserving the original structure information. The proposed method uses a graphon approximation method to broadcast the original structure as supervision for generating the condensed graph structure and optimizes it using an optimal transport method. The proposed SGDD achieves state-of-the-art results on various datasets and tasks.\n\nSTRENGTHS:\n(1) This paper uncovers the issue of Laplacian energy distribution shift during the condensation of graph datasets and shows that previous graph condensation methods that overlook the original structure information can lead to poor performance in cross-architecture generalization and specific tasks. This paper introduces a metric SC to quantify the Laplacian energy distribution issue.\n(2) A new condensing paradigm in graph condensation. This paper leverages the graphon theory to guarantee the structure information consistency between the original and generated graph, which firstly involves the generated learning fashion in graph condensation. Consequently, the generated structure may capture the native properties of the original structure in comparison to current methods, which benefits the downstream tasks.\n(3) Extensive experimental results demonstrate the effectiveness of the proposed method. This work includes extensive experiments on three node-level tasks as well as multiple ablation studies. All of the experiments demonstrate superior results. The most appealing result is that this method performs well in a cross-architecture setting, significantly improving the effectiveness of graph condensation. \n(4) Potential Impact. When the data and model go bigger and bigger, graph condensation may have more applications and research meanings. As the applications and challenges related to graph condensation remain unidentified, this paper may set the stage for future exploration of this problem and inspire the communities to further investigate and may benefit the areas like pruning, LLM with graphs, and NAS.\n\nWEAKNESSES:\n(1) The motivation for using JS divergence in the calculation of SC is unclear. There are serval ways of calculating the distance of two distributions.\n(2) In Figure.2, the author uses the BWGNN without specifying the difference between BWGNN and other backbones. \n(3) There is another related method called DosCond[1] which is mentioned in the introduction but not involved in the baselines. The reason why not choosing this method for comparison should be specified.\n[1] Jin, Wei, et al. “Condensing Graphs via One-Step Gradient Matching.” Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.\n\nQUESTIONS:\n(1) Please detailed the motivation in weakness.1 and weakness.2. \n(2) Consider adding the baseline mentioned in weakness (3) or specify the reason why do not compare with this method.\n(3) There is a high variance in the results of SGDD in Table 1. Would changing the backbone potentially improve the overall performance as shown in Table 1?\n\nLIMITATIONS:\n(1) In the real world, graphs may contain multiple types of edges, resulting in heterogeneity. However, the current learning methods may fail to capture this heterogeneity, leading to information loss.\n(2) Lack of discussion on different backbones.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach for graph dataset distillation called Structure-broadcasting Graph Dataset Distillation (SGDD). The authors explicitly consider the impact of the original structure information on graph condensation and demonstrate that their approach achieves state-of-the-art results on 9 datasets, showing superior performance in cross-architecture settings and specific tasks. Overall, the paper presents a significant contribution to the field of graph dataset distillation.\n\nSTRENGTHS:\n- The paper proposes a novel approach for graph dataset distillation that explicitly considers the impact of the original structure information on graph condensation. This is a significant departure from existing methods that primarily focus on optimizing the feature matrices of condensed graphs while overlooking the structure information.\n- The authors introduce the concept of Substantial Laplacian Energy Distribution (LED) shifts and demonstrate that previous works suffer from such shifts, leading to poor performance in cross-architecture generalization and specific tasks. The authors propose SGDD as a solution to this problem, which is a novel and original contribution.\n- The authors provide a thorough analysis of the proposed approach, including theoretical analysis and empirical evaluation on 9 datasets. The results demonstrate that SGDD consistently outperforms existing state-of-the-art methods, indicating the high quality of the proposed approach.\n\nWEAKNESSES:\n- [Actual Computation Savings] The authors distill the graph dataset. However, the authors only provide the distilled dataset's size ratio to original dataset size. It would be helpful to provide the actual computation savings in terms of time and memory usage.\n\n- [The Cost during Graph Distillation]  The cost of graph distillation for condensed small graph dataset is recommended to be provided. Since the used benchmark graph datasets are less computationally expensive than datasets in computer vision and natural language processing, the cost of graph distillation period will reflect whether the dataset is necessary for practical use.\n\n- [Comparison with Other Methods]  The paper provides a comparison of the proposed method with other methods in terms of performance and size ratio (e.g., Table 1). However, it would be beneficial to also compare the cost of graph distillation for the proposed method with other methods. This information would provide a more comprehensive understanding of the practicality of the proposed approach compared to other methods.\n\nQUESTIONS:\nQuestions: Please see weaknesses. I would like to update my evaluation after the discussion.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper investigates the effects of structural information in graph condensation methods. The authors claim that by maintaining the original structure during condensation using a newly formulated method called Structure-broadcasting Graph Dataset Distillation (SGDD), they are able to achieve more refined results on 9 data sets thereby significantly reducing Laplacian Energy Distribution (LED) shift.

STRENGTHS:
This paper addresses an important question regarding the impact of structural information. The authors conduct a comprehensive analysis from the spectral domain and empirically identify significant shifts in Laplacian Energy Distribution (LED), which ultimately result in poor performance in cross-architecture generalization and specific tasks.

The proposed method demonstrates effectiveness in the experiments, and empirical analysis confirms the efficacy and necessity of the proposed designs.

Overall, the presentation of the paper is also commendable

WEAKNESSES:
large datasets in the Open Graph Benchmark (OGB) is more effective in demonstrating the efficacy of these methods.

QUESTIONS:
I don't have question regarding this paper.

LIMITATIONS:
Need experiment for large datasets
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new graph condensation framework called SGDD. The authors argue that existing methods overlook the structure information of the original graph during the condensation. And thus they propose to 1) use the Laplacian Energy Distribution (LED) shift to indicate the generalization performance of the condensed graph, and 2) minimize the LED shift (they claim it’s equivalent to the OT distance between the original and the condensed graphs) to preserve the structure information. Experiments on different datasets over node classification, anomaly detection and link prediction tasks show the effectiveness of the proposed method.\n\nSTRENGTHS:\n- The motivation of preserving the structure information is clear and reasonable.\n- The experiments seem relatively sufficient and the results are good.\n\nWEAKNESSES:\nMy major concern is the soundness of this paper:\n- Some claims seem to be problematic. For example, in line 176-179, the authors claim that ‘minimizing the LED shift is equivalent to minimizing the distance of Laplacian pseudo-inverse matrices [65]’ and ‘Following the previous work [78], minimizing such distance can be further approximated to optimizing a free parameter $P$ in Eq(6)’. However, the equivalence in [65] only applies to the objective using a specific GW distance and the approximation in [78] only applies to another OT distance based objective. So the two claims cannot hold simultaneously. What’s worse, the authors do not describe which OT distance is used in their original objective function. Then how could we conclude that ‘minimizing the upper bound of Eq. (7) is equal to optimizing the $L_{structure}$ on Eq. (5)’ in line 189?\n- The writing is not very easy to follow. The mixed use of ‘LED shift’, ‘LED shift coefficient’, ‘SC’ is very confusing (e.g., in Def2 and the caption of Fig2). And it’s not clear what’s the role of the proposed LED shift coefficient. This coefficient is neither used to derive the objective function, nor adopted as a performance metric in the tables for comparison.\n\n=============================\nPost-rebuttal: The author responses have basically solved my concerns about the soundness.\n\nQUESTIONS:\nPlease see the above part.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}