SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \left| \eta^ {\mathcal {G}} - \eta^ {\mathcal {S}} \right| \right| = \inf  _ {T _ {\neq} \eta^ {\mathcal {S}} = \eta^ {\mathcal {G}}} \int_ {\mathcal {X}} \| x - T (x) \| ^ {2} d \eta^ {\mathcal {S}} (x) \\ = N ^ {\prime} \operatorname {t r} \left(L _ {\mathcal {S}} ^ {\dagger}\right) - 2 \operatorname {t r} \left(\left(\left(L _ {\mathcal {S}} ^ {\dagger}\right) ^ {1 / 2} P ^ {T} L _ {\mathcal {G}} ^ {\dagger} P \left(L _ {\mathcal {S}} ^ {\dagger}\right) ^ {1 / 2}\right) ^ {1 / 2}\right). \tag {9} \\ \end{array}
$$
Here, due to the transport plan  $T$  is impractical in optimizing, following the previous work[15], we use  $P \in \mathbb{R}^{N' \times N}$  denotes as a free parameter serving as the direct mapping strategy between nodes. Thus, the Distance in the Eq. (5) could be directly optimized by the Eq.(9) (i.e., use the  $P$  represents all possible mapping strategies, the optimizing of  $\bar{P}$  is equal to choosing a more optimal mapping strategy). In the experimental setting, we use the Sinkhorn-Knopp[9] algorithm to optimize  $P$ .
The overall time complexity is reduced to the  $O(N^{\omega}) \leq O(N^{2.373})$  [15]. Note that the  $L_{\mathcal{G}}^{\dagger}$  may be too large for computing, so we empirically sample a medium size (e.g., 2,000 nodes) sub-structure in the experiment and ablate its influence in Appendix C.7.
Table 1: Comparisons to state-of-the-art methods. SGDD achieves the highest results in most cases on node classification (NC), anomaly detection (AD), and link prediction (LP) tasks. We report test accuracy (\%) on NC datasets (including Citeseer, Cora, Ogbn-arxiv, Flickr, and Reddit), F1-macro (\%) on AD datasets (including YelpChi and Amazon), and AUC (\%) on LP datasets (including Citeseer-L and DBLP). Bold entries are best results, underline mark the runner-ups.
Dataset	Ratio (r)	Random	Herding	K-Center	Coarsening	GDC	Gcond	SGDD	Whole
NC	CiteSeer [41]	0.90%	54.4±4.4	57.1±1.5	52.4±2.8	52.2±0.4	66.8±1.5	70.5±1.2	69.5±0.4	71.7±0.1
1.80%	64.2±1.7	66.7±1.0	64.3±1.0	59.0±0.5	66.9±0.9	70.6±0.9	70.2±0.8
3.60%	69.1±0.1	69.0±0.1	69.1±0.1	65.3±0.5	66.3±1.5	69.8±1.4	70.3±1.7
Cora [41]	1.30%	63.6±3.7	67.0±1.3	64.0±2.3	31.2±0.2	67.3±1.9	79.8±1.3	80.1±0.7	81.2±0.2
2.60%	72.8±1.1	73.4±1.0	73.2±1.2	65.2±0.6	67.6±3.5	80.1±0.6	80.6±0.8
5.20%	76.8±0.1	76.8±0.1	76.7±0.1	70.6±0.1	67.7±2.2	79.3±0.3	80.4±1.6
Ogbn-arxiv [29]	0.05%	47.1±3.9	52.4±1.8	47.2±3.0	35.4±0.3	58.6±0.4	59.2±1.1	60.8±1.3	71.4±0.1
0.25%	57.3±1.1	58.6±1.2	56.8±0.8	43.5±0.2	59.9±0.3	63.2±0.3	65.8±1.2
0.50%	60.0±0.9	60.4±0.8	60.3±0.4	50.4±0.1	59.5±0.3	64.0±0.4	66.3±0.7
Flickr [100]	0.10%	41.8±2.0	42.5±1.8	42.0±0.7	41.9±0.2	46.3±0.2	46.5±0.4	46.9±0.1	47.2±0.1
0.50%	44.0±0.4	43.9±0.9	43.2±0.1	44.5±0.1	45.9±0.1	47.1±0.1	47.1±0.3
1.00%	44.6±0.2	44.4±0.6	44.1±0.4	44.6±0.1	45.8±0.1	47.1±0.1	47.1±0.1
Reddit [26]	0.01%	46.1±4.4	53.1±2.5	46.6±2.3	40.9±0.5	88.2±0.2	88.0±1.8	90.5±2.1	93.9±0.0
0.05%	58.0±2.2	62.7±1.0	53.0±3.3	42.8±0.8	89.5±0.1	89.6±0.7	91.8±1.9
0.50%	66.3±1.9	71.0±1.6	58.5±2.1	47.4±0.9	90.5±1.2	90.1±0.5	91.6±1.8
AD	YelpChi [68]	0.05%	41.8±0.3	46.1±0.9	49.3±1.1	46.2±2.1	47.9±1.1	48.6±3.7	56.2±1.8	61.1±1.8
0.10%	43.7±1.2	47.1±1.2	44.2±1.8	47.5±1.8	50.2±2.1	49.6±1.8	58.1±2.3
0.20%	45.9±2.2	46.4±0.8	47.5±0.4	49.1±1.2	49.7±2.0	50.1±2.8	59.7±1.8
Amazon [102]	0.02%	76.2±0.8	74.1±0.9	73.4±2.1	75.2±2.9	74.1±1.9	77.9±3.1	83.3±2.6	89.5±0.9
0.20%	76.4±1.6	76.5±2.3	74.2±1.1	76.8±1.0	78.2±2.1	78.1±1.9	84.8±1.7
2.00%	78.2±0.9	77.2±1.8	73.8±2.2	77.8±2.8	79.3±3.1	79.2±2.0	86.9±2.1
LP	CiteSeer-L [97]	0.90%	56.8±0.8	63.1±1.3	78.3±2.6	76.9±0.8	81.5±2.5	83.4±1.9	86.4±1.6	96.8±1.8
1.80%	61.4±1.8	63.3±1.8	79.1±1.8	77.4±1.7	83.4±1.6	83.8±2.1	87.2±2.1
3.60%	63.5±0.9	64.7±1.7	80.6±2.7	78.4±0.4	84.2±2.1	83.1±2.7	87.1±1.2
DBLP [83]	0.05%	64.1±2.6	65.8±0.7	72.4±1.6	77.8±1.3	78.9±2.1	77.2±2.1	81.3±2.8	84.2±0.7
0.25%	68.3±1.4	71.2±1.7	74.9±2.8	76.9±0.9	77.5±1.9	78.6±1.2	82.1±1.9
0.50%	68.7±2.6	74.3±0.8	75.2±0.6	76.8±2.1	77.6±2.1	79.9±1.9	82.1±1.8
4.3 Training pipeline of SGDD
As illustrated in Fig. 1(d) and Fig. 3, we commence by introducing a novel graph structure learning paradigm termed "graphon approximation". This paradigm integrates both the feature  $X'$  and auxiliary information  $Z$  to generate the structure. Subsequently, the learned structure  $A'$  is forced to be closer to the original graph structure  $A$  in terms of Laplacian energy distribution.
Additionally, our proposed methodology, SGDD, implements a bi-loop optimization schema. Within this framework, we concurrently optimize the parameters  $X'$  and  $A'$ . More specifically, the refinement of  $X'$  is achieved through a gradient matching strategy, whereas the  $A'$  is enhanced using the LED matching technique. During each step, the other component is frozen to ensure effective refinement.
Our overall training loss function can be summarized as  $\mathcal{L} = \mathcal{L}_{\text{feature}} + \alpha \mathcal{L}_{\text{structure}} + \beta ||\mathbf{A}||_2$  where the  $||\mathbf{A}||_2$  is proposed as a sparsity regularization term and  $\mathcal{L}_{\text{feature}}$  denotes the gradient matching strategy [37].  $\alpha$  and  $\beta$  are trade-off parameters, we study their sensitiveness in the Sec. 5.3. The algorithm can be found in Appendix C.3.
5.1 Datasets and implementation details
Datasets. We evaluate SGDD on five node classification datasets: Cora [41], CiteSeer [41], Ogbnarxiv [29], Flickr [100], Reddit [26], two anomaly detection datasets: YelpChi [68], Amazon [102], and two link prediction datasets: CiteSeer-L [97], DBLP [83]. For the node classification and anomaly detection tasks, we follow the public settings [18] of train and test. To make a fair comparison, we also follow the previous setting [101, 83], we randomly split  $80\%$  nodes for training,  $10\%$  nodes for validation, and the remaining  $10\%$  for testing. To avoid data leakage, we only utilize  $80\%$  training samples for condensation. More details of each dataset can be found in Appendix C.1.
(a) Ablation of  $\mathbf{X}'$  and  $\mathbf{A}$ .
(b) Evaluation of  $r(\%)$
(c) Evaluation of  $\alpha$
Figure 4: (a) Ablation of components in SGDD. (b) Evaluation of the scalability of SGDD. (c) and (d): the trade-off parameters analysis on  $\alpha$  and  $\beta$ .
(d) Evaluation of  $\beta$
Implementation details. Without specific designation, in the condense stage, we adopt the 2-layer GCN with 128 hidden units as the backbone, and we adopt the settings on [94], which use 2-layer MLP to represent the structure generative model (i.e., GEN). The learning rates for structure and feature are set to 0.001 (0.0001 for Ogbn-arxiv and Reddit) and 0.0001, respectively. We set  $\alpha$  to 0.1, and  $\beta$  to 0.1. In the evaluation stage, we train the same network for 1,000 epochs on the condensed graph with a learning rate of 0.001. Following the settings in [37], we repeat all experiments ten times and report average performance and variance. More details<sup>4</sup> can be found in Appendix C.2.
5.2 Comparison with state-of-the-art methods
We compare our proposed SGDD with six baselines: Random, which randomly selected nodes to form the original graph, corset methods (Herding [90] and K-Center [72]), graph coarsening methods (Corasening [30]), and the state-of-the-art graph condensation methods (GCond [37]). GDC is proposed as a baseline in [37], where cosine similarity is added as a constraint to generate the structure of the condensed graph. In Table 1, we present the performance metrics including accuracy, F1-macro, and AUC. For clarity and simplicity, percentages are represented without the  $\%$  symbol, and variance values are provided. Based on the results, we have the following observations: 1) Our proposed SGDD achieves the highest results in most settings, which shows the superiority and generalization of our method. 2) On anomaly detection datasets, the improvements are more significant than other tasks, i.e., improving GCond with  $9.6\%$  and  $7.7\%$  on YelpChi and Amazon datasets, which can be explained that our method captures the structure information from the original graph dataset more efficiently.
Table 2: Results of cross-architecture setting, we test condensed graphs in APPNP, Cheby, GCN, GraphSAGE, and SGC. Avg. and Std.: the average performance and the standard deviation of the results, the  $\Delta (\%)$  denotes the improvements upon the GDC. We mark the best performance by bold.
Datasets	Methods	Architectures	Statistics
MLP [11]	GAT [84]	APPNP [42]	Cheby [13]	GCN [41]	SAGE [26]	SGC [92]	Avg.	Std.	Δ(%)
Reddit [26]	GDC	50.3	54.8	81.2	77.5	89.5	89.7	90.5	76.2	16.9	-
GCond	42.5	60.2	87.8	75.5	89.4	89.1	89.6	76.3	18.5	↑0.1
Ours	56.1	74.4	89.2	78.4	89.4	89.4	89.4	80.9	12.6	↑4.7
Cora [41]	GDC	67.2	64.2	67.1	67.7	67.9	66.2	72.8	67.6	2.6	-
GCond	73.1	66.2	78.5	76	80.1	78.2	79.3	75.9	4.9	↑8.3
Ours	76.7	75.8	78.4	78.5	79.8	80.4	78.5	78.3	1.6	↑10.7
DBLP [83]	GDC	74.4	76.8	77.4	76.7	78.9	74.8	78.4	76.8	1.7	-
GCond	75.3	77.6	78.9	76.1	79.6	77.4	79.9	77.8	1.7	↑1.1
Ours	78.4	79.6	80.1	80.6	82.1	80.7	81.4	80.4	1.2	↑3.6
YelpChi [68]	GDC	30.7	36.4	43.7	41.5	49.6	47.4	50.1	42.8	7.2	-
GCond	48.9	31.8	46.7	48.6	50.1	42.5	48.7	45.3	6.5	↑2.6
Ours	54.2	56.4	58.2	56.8	59.7	54.1	56.7	56.6	2.0	↑13.8
Cross-architecture generalization analysis. To evaluate the generalization ability of SGDD on unseen architectures, we conduct experiments that train on the condensed graph with different
4The code is available in https://github.com/RingBDStack/SGDD, based on PyTorch[65] and MindSpore[31].
Table 3: Comparison of the cross-architecture generalization performance between GCond and SGDD on Ogbn-arxiv. Bold entries are the best results. ↑/↓: our method show increase or decrease performance.
C/T	APPNP GCond / SGDD	Cheby GCond / SGDD	GCN GCond / SGDD	SAGE GCond / SGDD	SGC GCond / SGDD
APPNP	60.3 / 60.2↓	51.8 / 53.2↑	59.9 / 62.4↑	59.0 / 60.2↑	61.2 / 60.4↓
Cheby	57.4 / 58.5↑	53.5 / 55.8↑	57.4 / 63.5↑	57.1 / 57.0↓	58.2 / 60.2↑
GCN	59.3 / 60.1↑	51.8 / 53.7↑	60.3 / 64.2↑	60.2 / 61.2↑	59.2 / 59.8↑
SAGE	57.6 / 58.9↑	53.9 / 53.8↑	58.1 / 63.8↑	57.8 / 61.8↑	59.0 / 61.1↑
SGC	59.7 / 60.0↑	49.5 / 52.3↑	59.2 / 62.2↑	58.9 / 62.9↑	60.5 / 61.5↑
Table 4: Neural Architecture Search. Methods are compared in validation accuracy correlation and test accuracy on obtained architecture.
Dataset	Pearson Correlation / Performance (%)	Whole
Per. (%)
Random	GCond	SGDD
Ogbn-arxiv	0.63 / 71.1	0.64 / 71.2	0.67 / 71.6	71.9
YelpChi	0.43 / 56.7	0.48 / 58.4	0.56 / 60.6	61.1
DBLP	0.58 / 81.2	0.62 / 83.8	0.68 / 84.0	84.2
architectures and report their performances in Tab. 2. Here, the condensed graph is obtained by optimizing SGDD with SGC [92]. We test its cross-architecture generalization performances on 2-layer-MLP [11], GAT [84], APPNP [42], Cheby [13], GCN [41], and SAGE [26].
To better understand, we also show several statistics metrics, including Avg., Std., and  $\Delta$ . SGDD and GCond improves GDC significantly, which indicates there exists a large difference between graph dataset condensation and vision dataset distillation, i.e., the structure information should be specially considered. Compared to GCond, the improvement of our method is up to  $11.3\%$ , demonstrating the effectiveness of broadcasting the original structure to the condensed graph structure generation. More experiments on other datasets can be found in Appendix C.6.
Versatility of SGDD. Following the setting of GCond [37], we also study whether our proposed SGDD is robust on various architectures. We first condense the Ogbn-arxiv graph dataset with five architectures, including APPNP [42], Cheby [13], GCN [41], SAGE [26], and SGC [92], respectively. Then, we evaluate these condensed graphs on the above five architectures and report their performances in Tab. 3. The experiment results show that SGDD achieves non-trivial improvements than GCond in most cases, which demonstrates the strong versatility of our method.
Evaluation on neural architecture search. Similar to vision dataset distillation, graph dataset distillation is also expected to reduce the high cost of neural architecture search (NAS). In order to make a fair comparison, we follow the experimental setting in [37]: searching architectures on condensed Obgn-arxiv, YelpChi, and DBLP datasets with  $0.25\%$ ,  $0.2\%$ , and  $0.5\%$  condensing ratios. We report Pearson correlation [45] and performance of random, GCond, and SGDD in Tab. 4. Our SGDD consistently achieves the highest Pearson correlations as well as performances, which indicates the architectures searched by our method are efficient for the whole graph dataset training.
Evaluation of components in SGDD. To explore the effect of the conditions (mentioned in Sec. 4.1) in the condensed graph generation, we design the ablation study of  $\mathbf{X}^{\prime}$  and A. As shown in Fig 4(a),  $\mathbf{X}^{\prime}$  and A are complementary with each other. SGDD w/o A performs poorly on Obgn-arxiv and YelpChi datasets, which demonstrates the effect of original graph structure information. Jointly using  $\mathbf{X}^{\prime}$  and A achieves the highest performances on both datasets, improves GCond with  $3.1\%$  on Ogbn-arxiv, and  $8.0\%$  on YelpChi.
Evaluation of the scalability of SGDD. To investigate the scalability of SGDD, we evaluate the SGDD on various condensing ratios with  $r \in \{0.02, 0.2, 2, 5, 10, 20\}$ . As shown in Fig. 4(b), the performance of our method continuously increase as the condensing ratio rises, which indicates the strong scalability of our method. GCond obtains marginal improvements than GDC at all ratios while our SGDD outperforms them significantly. More important, SGDD achieves lossless performance as training on the original graph data when the condensing ratio is  $10\%$ .
Exploring the sensitivity of  $\alpha$  and  $\beta$ . We conduct experiments to test such two parameters sensitivity on YelpChi, Cora, and Ogbn-arxiv. As shown in Fig. 4(c), we empirically find that the performance of our SGDD is not sensitive to the  $\alpha$ . Specifically, compared to the case that  $\alpha$  is zero, we have a significant improvement, which proves the effectiveness of our method. Another finding is that the  $\alpha$  should be set higher on anomaly detection than on node classification tasks. It could be explained by the original graph structure information being more important on the complex task (such as anomaly detection). We define  $\beta$  as a regularization coefficient to control the sparsity of the condensed graph. As shown in Fig. 4(d), we evaluate the  $\beta$  from 0 to 10 on the YelpChi dataset. The results illustrate that the performance is not sensitive with  $\beta$  and achieve the highest result (F1-macro) when  $\beta$  is set to our default value ( $\beta = 0.1$ ). More experiments can be found in Appendix C.5.
(a) Original Ogbn-arxiv
(b) Condensed by GCond [37]
(c) Condensed by SGDD
(d) Original synthetic graph
(e) Condensed by GCond [37]
(f) Condensed by SGDD
Figure 5: Visualizations of the real dataset (a), synthetic dataset (d), and the corresponding condensed graph obtained by GCond (b, e) and SGDD (c, f). The condensing ratio  $r$  is set to  $0.5\%$ .
5.4 Visualizations
To better understand the effectiveness of our SGDD, we visualize the condensed graphs of GCond and SGDD that are synthesized from real and synthetic graph datasets. For the synthetic graph dataset, we use the Stochastic Block Model (SBM) [28] to synthesize graphs with 5 community (Fig. 5(d)). As shown in Fig. 5, one can find that our method consistently achieves better performances and  $SC$ . SGDD reduces  $29.0\%$  (comparing 5(b) and 5(c)) and  $62.2\%$  (comparing 5(e) and 5(f))  $SC$  on Ogbn-arxiv and synthetic datasets. Visually, the condensed graphs of our method preserve the original graph structure information obviously better than GCond (see the second row of Fig. 5), which proves SGDD is a powerful graph dataset distillation method.
We present SGDD, a novel framework for graph dataset distillation via broadcasting the original structure information to the generation of the synthetic one. SGDD shows its robustness on various tasks and datasets, achieving state-of-the-art results on YelpChi, Amazon, Ogbn-arxiv, and DBLP. SGDD reduces the scale of the Yelpchi dataset by 1,000 times while maintaining  $98.6\%$  as training on the original data. We provide sufficient experiments and theoretical analysis in this paper and hope it can help the following research in this area. Limitations and future work: Although broadcasting the original information to the generated graph shows remarkable success, some informative properties (e.g., the heterogeneity) may lose during the current condense process, which results in sub-optimal performance in the downstream tasks. We are going to explore a more general method in the future.
The corresponding author is Jianxin Li. This work was supported by the NSFC through grant No.62225202 and No.62302023. This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-008). Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant, and Alibaba grant. We gratefully acknowledge the support of MindSpore, CANN(Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
[102] Shijie Zhang, Hongzhi Yin, Tong Chen, Quoc Viet Hung Nguyen, Zi Huang, and Lizhen Cui. Gcn-based user representation learning for unifying robust recommendation and fraudster detection. In Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM conference on research and development in Information Retrieval, Virtual Event, China, July 25-30, 2020, pages 689-698, 2020.
[103] Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. Expanding small-scale datasets with guided imagination. ArXiv preprint, abs/2211.13976, 2022.
[104] Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. ArXiv preprint, abs/2110.04181, 2021.
[105] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In 9th International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021, 2021.
[106] Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In 9th International Conference on Learning Representations, Virtual Event, Austria, May 3-7, 2021, 2021.
[107] Tong Zhao, Yozen Liu, Leonardo Neves, Oliver J. Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In Thirty-Fifth AAAI Conference on Artificial Intelligence, Virtual Event, February 2-9, 2021, pages 11015-11023, 2021.
[108] Cheng Zheng, Bo Zong, Wei Cheng, Dongjin Song, Jingchao Ni, Wenchao Yu, Haifeng Chen, and Wei Wang. Robust graph representation learning via neural sparsification. In Proceedings of the 37th International Conference on Machine Learning, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 11458-11468, 2020.
[109] Xin Zheng, Miao Zhang, Chunyang Chen, Quoc Viet Hung Nguyen, Xingquan Zhu, and Shirui Pan. Structure-free graph condensation: From large-scale graphs to condensed graph-free data. In Advances in Neural Information Processing Systems, 2023.
[110] Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17205-17216, 2023.
[111] Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. ArXiv preprint, 2018.
A More Preliminary and Related Work
A.1 More preliminary
Graphon. A graphon [43, 34, 54] is a bounded, symmetric, and Lebesgue measurable function, denote as  $W: \Omega^2 \to [0,1]$ , where  $\Omega$  is a probability measure space. By randomly selecting points  $\{v_0, v_1, \ldots, v_n\}$  from  $\Omega$ , we can create a graph of any size by connecting each point with an edge weight  $e_{ij} = W(v_i, v_j)$ . Formally, we can follow the random sampling process to get arbitrarily sized  $(n)$  graphs:
$$
v _ {i} \sim \operatorname {U n i f o r m} (\Omega), \text {f o r} i = 1, \dots , n,
$$
$$
\begin{array}{l} v _ {i} \sim \text {U n i o r i n} (\Delta), \text {f o r} i = 1, \dots , n, \\ e _ {i j} \sim \operatorname {B e r n o u l l i} (W \left(v _ {i}, v _ {j}\right)), \text {f o r} i, j = 1, \dots , n. \end{array} \tag {10}
$$
Conventionally, we use a deterministic setting to select the node, e.g., the fixed grid  $v_{i} = \frac{i - 1}{n}$ . Graphs derived from the same graphon share several important properties for statistical analysis: such as density [21], clustering coefficient [70], and degree distribution [71], which motivate us to broadcast the original graph structure information to the condensed graph via the graphon approximating.
A.2 More related work
Graph structure learning. Graph structure learning [19, 35, 8, 108, 55] also aims at jointly optimizing the graph structure and the corresponding node features. Most of these methods learn the new structure based on the certain constraints (e.g., low-rank [91], sparsity [8], feature smoothness [35], and homophily [108]). However, these methods are unable to learn a new structure with a significantly reduced node size. In contrast, our method can synthesize a smaller graph structure while simultaneously constructing new node features. The obtained small but informative graph can reduce the training cost in the downstream tasks.
Proof. First, the gradient matching objective in GCond [37] can be shown as:
$$
\operatorname {M a t c h} \left(\nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {c l s} \left(\mathrm {G N N} _ {\boldsymbol {\theta}} (\mathbf {A}, \mathbf {X}), \mathbf {Y}\right)\right), \quad \nabla_ {\boldsymbol {\theta}} \mathcal {L} _ {c l s} \left(\mathrm {G N N} _ {\boldsymbol {\theta}} \left(\mathbf {A} ^ {\prime}, \mathbf {X} ^ {\prime}\right), \mathbf {Y} ^ {\prime}\right)), \tag {11}
$$
where  $\mathrm{Match}(\cdot)$  is the matching function that measures the distance of the two gradients.  $\theta$  represents the parameters of the GNN, the  $\nabla$  denotes the gradient in the backpropagation process, and the  $\mathcal{L}_{cls}$  represents the loss function that is used in the supervised tasks, i.e., cross-entropy loss. Following the discussion in the [4, 1, 82], the GNN can be viewed as a bandpass filter in the spectral domain. Expanding the GNN in the spectral domain, we can rewrite the objective as:
$$
\left| \left| \int_ {i = 0} ^ {N} \mathbf {U} _ {\mathbf {A}} \operatorname {d i a g} \left(\mathrm {T} _ {i} \left(\boldsymbol {\lambda} _ {\mathbf {A}}\right)\right) \mathbf {U} _ {\mathbf {A}} ^ {\top} \mathbf {X} - \int_ {j = 1} ^ {N ^ {\prime}} \mathbf {U} _ {\mathbf {A} ^ {\prime}} \operatorname {d i a g} \left(\mathrm {T} _ {j} \left(\boldsymbol {\lambda} _ {\mathbf {A} ^ {\prime}}\right)\right) \mathbf {U} _ {\mathbf {A} ^ {\prime}} ^ {\top} \mathbf {X} ^ {\prime} \right| \right| \leq \epsilon , \tag {12}
$$
where the  $\mathrm{T}(\lambda)$  denotes the frequency response of the given GNN, the  $\lambda$  and the  $\mathbf{U}$  is the eigenvalue and eigenvector of the corresponding graph, respectively. Note we drop the  $\nabla$ , the  $\mathcal{L}_{cls}$ , and the  $\mathbf{Y}$ $(\mathbf{Y}^{\prime})$  terms because they can be viewed as the intermediate process [105, 107] in approximating to Eq.(12). We use the MSE [37] as the matching function Match  $(\cdot)$  for simplicity.
To simplify  $\mathrm{T}(\lambda)$ , without loss of generality, we assume the bandwidth of the specific GNN's frequency response is (a, b), which intuitively indicates that only the signal of frequency in (a, b) can pass through the filter on the graph. Then the Eq. (12) can be written as: