SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin Zheng $^{1}$ , Miao Zhang $^{2}$ , Chunyang Chen $^{1}$ , Quoc Viet Hung Nguyen $^{3}$ , Xingquan Zhu $^{4}$ , Shirui Pan $^{3\dagger}$
<sup>1</sup>Monash University, Australia, <sup>2</sup>Harbin Institute of Technology (Shenzhen), China
<sup>3</sup>Griffith University, Australia, <sup>4</sup>Florida Atlantic University, USA
xin.zheng@monash.edu, zhangmiao@hit.edu.cn, chunyang.chen@monash.edu
henry.nguyen@griffith.edu.au, xzhu3@fau.edu, s.pan@griffith.edu.au
Graph condensation, which reduces the size of a large-scale graph by synthesizing a small-scale condensed graph as its substitution, has immediate benefits for various graph learning tasks. However, existing graph condensation methods rely on the joint optimization of nodes and structures in the condensed graph, and overlook critical issues in effectiveness and generalization ability. In this paper, we advocate a new Structure-Free Graph Condensation paradigm, named SFGC, to distill a large-scale graph into a small-scale graph node set without explicit graph structures, i.e., graph-free data. Our idea is to implicitly encode topology structure information into the node attributes in the synthesized graph-free data, whose topology is reduced to an identity matrix. Specifically, SFGC contains two collaborative components: (1) a training trajectory meta-matching scheme for effectively synthesizing small-scale graph-free data; (2) a graph neural feature score metric for dynamically evaluating the quality of the condensed data. Through training trajectory meta-matching, SFGC aligns the long-term GNN learning behaviors between the large-scale graph and the condensed small-scale graph-free data, ensuring comprehensive and compact transfer of informative knowledge to the graph-free data. Afterward, the underlying condensed graph-free data would be dynamically evaluated with the graph neural feature score, which is a closed-form metric for ensuring the excellent expressiveness of the condensed graph-free data. Extensive experiments verify the superiority of SFGC across different condensation ratios. $^{\ddagger}$
As prevalent graph data learning models, graph neural networks (GNNs) have attracted much attention and achieved great success [68, 36, 22, 82, 25, 24, 23]. Various graph data in the real world comprises millions of nodes and edges[37, 38, 43], reflecting diverse node attributes and complex structural connections [33, 34, 32, 54]. Modeling such large-scale graphs brings serious challenges in both data storage and GNN model designs, hindering the applications of GNNs in many industrial scenarios [74, 2, 64, 73, 80, 71, 72]. For instance, designing GNN models usually requires repeatedly training GNNs for adjusting proper hyper-parameters and constructing optimal model architectures. When taking large-scale graphs as training data, repeated training through message passing along complex graph structures, makes it highly computation-intensive and time-consuming through try-and-error.
To address these challenges brought by the scale of graph data, a natural data-centric solution [78] is graph size reduction, which transforms the real-world large-scale graph to a small-scale
$\ddagger$ Code is available at https://github.com/Amanda-Zheng/SFGC
graph, such as graph sampling [66, 6], graph coreset [47, 60], graph sparsification [1, 5], and graph coarsening [3, 28]. These conventional methods either extract representative nodes and edges or preserve specific graph properties from the large-scale graphs, resulting in severe limitations of the obtained small-scale graphs in the following two folds. First, the available information on derived small-scale graphs is significantly upper-bounded and limited within the range of large-scale graphs [66, 60]. Second, the preserved properties of small-scale graphs, e.g., spectrum and clustering, might not always be optimal for training GNNs for downstream tasks [1, 3, 28].
Figure 1: Comparisons of condensation vs structure-free condensation on graphs.
In light of these limitations of conventional methods, in this work, we mainly focus on graph condensation [27, 26], a new rising synthetic method for graph size reduction. Concretely, graph condensation aims to directly optimize and synthesize a small-scale condensed graph, so that the small-scale condensed graph could achieve comparable test performance as the large-scale graph when training the same GNN model. Therefore, the principal goal of graph condensation is to ensure consistent test results for GNNs when taking the large-scale graph and the small-scale condensed graph as training data.
However, due to the structural characteristic of graph data, nodes and edges are tightly coupled. This makes condensing graph data a complicated task since high-quality condensed graphs are required to jointly synthesize discriminative node attributes and topology structures. Some recent works have made initial explorations of graph condensation [27, 26]. For instance, GCOND [27] proposed the online gradient matching schema between the synthesized small-scale graph and the large-scale graph, followed by a condensed graph structure learning module for synthesizing both condensed nodes and structures. However, existing methods overlook two-fold critical issues regarding effectiveness and generalization ability. First, graph condensation requires a triple-level optimization to jointly learn three objectives: GNN parameters, distilled node attributes, and topology structures. Such complex optimization cannot guarantee optimal solutions for both nodes and edges in the condensed graph, significantly limiting its effectiveness as the representative of the large-scale graph. Furthermore, existing online GNN gradients [27, 26] are calculated with the short-range matching, leading to the short-sight issue of failing to imitate holistic GNN learning behaviors, limiting the quality of condensed graphs. Second, existing condensed graphs generally show poor generalization ability across different GNN models [27, 26], because different GNN models vary in their convolution operations along graph structures. As a result, existing methods are vulnerable to overfitting of specific GNN architectures by distilling convolutional information into condensed graph structures.
To deal with the above two-fold challenges, in this work, we propose a novel Structure-Free Graph Condensation paradigm, named SFGC, to distill large-scale real-world graphs into small-scale synthetic graph node sets without graph structures, i.e., condensed graph-free data. Different from conventional graph condensation that synthesizes both nodes and structures to derive a small-scale graph, as shown in Fig. 1, the proposed structure-free graph condensation only synthesizes a small-scaled node set to train a GNN/MLP, when it implicitly encodes topology structure information into the node attributes in the synthesized graph-free data, by simplifying the condensed topology to an identity matrix. Overall, the proposed SFGC contains two essential components: (1) a training trajectory meta-matching scheme for effectively synthesizing small-scale graph-free data; (2) a graph neural feature score metric for dynamically evaluating the quality of condensed graph-free data. To address the short-sight issue of existing online gradient matching, our training trajectory meta-matching scheme first trains a set of training trajectories of GNNs on the large-scale graph to acquire an expert parameter distribution, which serves as offline guidance for optimizing the condensed graph-free data. Then, the proposed SFGC conducts meta-matching to align the long-term GNN learning behaviors between the large-scale graph and condensed graph-free data by sampling from the training trajectory distribution, enabling the comprehensive and compact transfer of informative knowledge to the graph-free data. At each meta-matching step, we would obtain updated condensed graph-free data, which would be fed into the proposed graph neural feature score metric for dynamically evaluating its quality. This metric is derived based on the closed-form solutions of GNNs under the graph neural tangent kernel (GNTK) ridge regression, eliminating the iterative training
of GNNs in the dynamic evaluation. Finally, the proposed SFGC selects the condensed graph-free data with the smallest score as the optimal representative of the large-scale graph. Our proposed structure-free graph condensation method could benefit many potential application scenarios, such as, graph neural architecture search [79, 81], privacy protection [69], adversarial robustness [67, 70], continual learning [78], and so on. We provide detailed demonstrations of how our method facilitates the development of these areas in Appendix B
In summary, the contributions of this work are listed as follows:
- We propose a novel Structure-Free Graph Condensation paradigm to effectively distill large-scale real-world graphs to small-scale synthetic graph-free data with superior expressiveness, to the best of our knowledge, for the first time.
- To explicitly imitate the holistic GNN training process, we propose the training trajectory meta-matching scheme, which aligns the long-term GNN learning behaviors between the large-scale graph and the condensed graph-free data, with the theoretical guarantee of eliminating graph structure constraints.
- To ensure the high quality of the condensed data, we derive a GNTK-based graph neural feature score metric, which dynamically evaluates the small-scale graph-free data at each meta-matching step and selects the optimal one. Extensive experiments verify the superiority of our method.
Prior Works. Our research falls into the research topic dataset distillation (condensation) [30, 59], which aims to synthesize a small typical dataset that distills the most important knowledge from a given large target dataset as its effective substitution. Considering most of the works condense image data [59, 39, 77, 76, 4], due to the complexity of graph structural data, only a few works [27, 26] address graph condensation, while our research designs a new structure-free graph condensation paradigm for addressing the effectiveness and generalization ability issues in existing graph condensation works. Our research also significantly differs from other general graph size reduction methods, for instance, graph coreset [47, 60], graph sparsification [1, 5] and so on. More detailed discussions about related works can be found in Appendix A.
2 Structure-Free Graph Condensation
Notations. Denote a large-scale graph dataset to be condensed by  $\mathcal{T} = (\mathbf{X},\mathbf{A},\mathbf{Y})$ , where  $\mathbf{X}\in \mathbb{R}^{N\times d}$  denotes  $N$  number of nodes with  $d$ -dimensional features,  $\mathbf{A}\in \mathbb{R}^{N\times N}$  denotes the adjacency matrix indicating the edge connections, and  $\mathbf{Y}\in \mathbb{R}^{N\times C}$  denotes the  $C$ -classes of node labels. In general, graph condensation synthesizes a small-scale graph dataset denoted as  $\mathcal{T}' = (\mathbf{X}',\mathbf{A}',\mathbf{Y}')$  with  $\mathbf{X}'\in \mathbb{R}^{N'\times d}$ ,  $\mathbf{A}'\in \mathbb{R}^{N'\times N'}$ , and  $\mathbf{Y}'\in \mathbb{R}^{N'\times C}$  when  $N'\ll N$ . In this work, we propose the structure-free graph condensation paradigm, which aims to synthesize a small-scale graph node set  $S = (\widetilde{\mathbf{X}},\widetilde{\mathbf{Y}})$  without explicitly condensing graph structures, i.e., the condensed graph-free data, as an effective substitution of the given large-scale graph. Hence,  $\widetilde{\mathbf{X}}$  contains joint node context attributes and topology structure information, which is a more compact representative compared with  $(\mathbf{X}',\mathbf{A}')$ .
Graph Condensation. Given a GNN model parameterized by  $\theta$ , graph condensation [27] is defined to solve the following triple-level optimization objective by taking  $\mathcal{T} = (\mathbf{X}, \mathbf{A}, \mathbf{Y})$  as input:
$$
\min _ {\mathcal {T} ^ {\prime}} \mathcal {L} \left[ \mathrm {G N N} _ {\boldsymbol {\theta} _ {\mathcal {T} ^ {\prime}}} (\mathbf {X}, \mathbf {A}), \mathbf {Y} \right]
$$
$$
s. t. \quad \theta_ {\mathcal {T} ^ {\prime}} = \underset {\boldsymbol {\theta}} {\arg \min } \mathcal {L} \left[ \operatorname {G N N} _ {\boldsymbol {\theta}} \left(\mathbf {X} ^ {\prime}, \mathbf {A} ^ {\prime}\right), \mathbf {Y} ^ {\prime} \right], \tag {1}
$$
$$
\boldsymbol {\psi} _ {\mathbf {A} ^ {\prime}} = \underset {\boldsymbol {\psi}} {\arg \min } \mathcal {L} \left[ \operatorname {G S L} _ {\boldsymbol {\psi}} \left(\mathbf {X} ^ {\prime}\right) \right],
$$
where  $\mathrm{GSL}_{\psi}$  is a submodule parameterized by  $\psi$  to synthesize the graph structure  $\mathbf{A}'$ . One of inner loops learns the optimal GNN parameters  $\theta_{\mathcal{T}'}$ , while another learns the optimal GSL parameters  $\psi_{\mathbf{A}'}$  to obtain the condensed  $\mathbf{A}'$ , and the outer loop updates the condensed nodes  $\mathbf{X}'$ . All these comprise the condensed small-scale graph  $\mathcal{T}' = (\mathbf{X}', \mathbf{A}', \mathbf{Y}')$ , where  $\mathbf{Y}'$  is pre-defined based on the class distribution of the label space  $\mathbf{Y}$  in the large-scale graph.
Overall, the above optimization objective needs to solve the following variables iteratively: (1) condensed  $\mathbf{X}^{\prime}$ ; (2) condensed  $\mathbf{A}^{\prime}$  with  $\mathrm{GSL}_{\psi_{\mathbf{A}^{\prime}}}$ ; and (3)  $\mathrm{GNN}_{\theta_{T^{\prime}}}$ . Jointly learning these interdependent
Figure 2: Overall pipeline of the proposed Structure-Free Graph Condensation (SFGC) framework.
objectives is highly challenging. It is hard to guarantee that each objective obtains the optimal and convergent solution in such a complex and nested optimization process, resulting in the limited expressiveness of the condensed graph. This dilemma motivates us to reconsider the optimization objective of graph condensation to synthesize the condensed graph more effectively.
Graph Neural Tangent Kernel (GNTK). As a new class of graph kernels, graph neural tangent kernel (GNTK) is easy to train with provable theoretical guarantees, and meanwhile, enjoys the full expressive power of GNNs [10, 19, 21, 41]. In general, GNTK can be taken as the infinitely-wide multi-layer GNNs trained by gradient descent. It learns a class of smooth functions on graphs with close-form solutions. More specifically, let  $G = (V,E)$  denote a graph with nodes  $V$  and edges  $E$ , where each node  $v \in V$  within its neighbor set  $\mathcal{N}(v)$ . Given two graphs  $G = (V,E)$  and  $G' = (V',E')$  with  $n$  and  $n'$  number of nodes, their covariance matrix between input features can be denoted as  $\Sigma^{(0)}(G,G') \in \mathbb{R}^{n \times n'}$ . Each element in  $[\Sigma^{(0)}(G,G')]_{uu'}$  is the inner product  $h_u^\top h_{u'}$ , where  $h_u$  and  $h_{u'}$  are of input features of two nodes  $u \in V$  and  $u' \in V'$ . Then, for each GNN layer  $\ell \in \{0,1,\dots,L\}$  that has  $\mathcal{B}$  fully-connected layers with ReLU activation, GNTK calculates  $\mathcal{K}_{(\beta)}^{(\ell)}\langle G,G'\rangle$  for each  $\beta \in [\mathcal{B}]$ :
$$
\begin{array}{l} \left[ \mathcal {K} _ {(\beta)} ^ {(\ell)} \langle G, G ^ {\prime} \rangle \right] _ {u u ^ {\prime}} = \left[ \mathcal {K} _ {(\beta - 1)} ^ {(\ell)} \langle G, G ^ {\prime} \rangle \right] _ {u u ^ {\prime}} \left[ \dot {\Sigma} _ {(\beta)} ^ {(\ell)} (G, G ^ {\prime}) \right] _ {u u ^ {\prime}} \tag {2} \\ + \left[ \boldsymbol {\Sigma} _ {(\beta)} ^ {(\ell)} \left(G, G ^ {\prime}\right) \right] _ {u u ^ {\prime}}, \\ \end{array}
$$
where  $\dot{\Sigma}^{(\ell)}$  denotes the derivative w.r.t. the  $\ell$ -th GNN layer of the covariance matrix, and the  $(\ell + 1)$ -th layer's covariance matrix aggregates neighbors along graph structures as  $\left[\boldsymbol{\Sigma}_{(0)}^{(\ell + 1)}(G, G')\right]_{uu'} = \sum_{v \in \mathcal{N}(u) \cup \{u\}} \sum_{v' \in \mathcal{N}(u') \cup \{u'\}} \left[\boldsymbol{\Sigma}_{(\mathcal{B})}^{(\ell)}(G, G')\right]_{vv'}$ , ditto for the kernel  $\left[\mathcal{K}_{(0)}^{(\ell + 1)}(G, G')\right]_{uu'}$ . With the GNTK matrix  $\mathcal{K}_{(\mathcal{B})}^{(L)}\langle G, G' \rangle \in \mathbb{R}^{n \times n'}$  at the node level, we use the graph kernel method to solve the equivalent GNN model for node classification with closed-form solutions. This would significantly benefit the efficiency of condensed data evaluation by eliminating iterative GNN training.
2.2 Overview of SFGC Framework
The crux of achieving structure-free graph condensation is in determining discriminative node attribute contexts, which implicitly integrates topology structure information. We compare the paradigms between existing graph condensation (GC) and our new structure-free condensation SFGC as follows:
$$
\begin{array}{l}\mathcal {T} = (\mathbf {X}, \mathbf {A}, \mathbf {Y}) \rightarrow \mathcal {T} ^ {\prime} = \left(\mathbf {X} ^ {\prime}, \mathbf {A} ^ {\prime}, \mathbf {Y} ^ {\prime}\right), \quad \text {G C}.\\\overline {{\widetilde {\tau}}} = (\overline {{\widetilde {\tau}}} - \overline {{\widetilde {\tau}}})\end{array}
$$
$$
\mathcal {T} = (\mathbf {X}, \mathbf {A}, \mathbf {Y}) \rightarrow \mathcal {S} = (\widetilde {\mathbf {X}}, \mathbf {I}, \widetilde {\mathbf {Y}}) = \mathcal {S} = (\widetilde {\mathbf {X}}, \widetilde {\mathbf {Y}}), \quad \text {S F G C}.
$$
It is fundamentally different between existing GC paradigm with  $\mathcal{T} \to \mathcal{T}'$  and our SFGC with  $\mathcal{T} \to \mathcal{S}$ . Our idea is to synthesize a graph-free data  $S = (\widetilde{\mathbf{X}},\mathbf{I},\widetilde{\mathbf{Y}})$  whose topology is reduced to an identity matrix  $\mathbf{I}$  (i.e., structure-free), instead of explicitly learning  $\mathbf{A}'$  as existing GC. To enforce node attribute  $\widetilde{\mathbf{X}}$  encoding topology structure information as  $\widetilde{\mathbf{X}} \simeq (\mathbf{X},\mathbf{A})$ , we propose a long-term imitation learning process with training trajectories, which requires a GNN model learned from  $S$ , i.e.,  $\mathrm{GNN}_{S} = \mathrm{GNN}(\widetilde{\mathbf{X}},\mathbf{I},\widetilde{\mathbf{Y}})$  must imitate a GNN model from the original graph, i.e.,  $\mathrm{GNN}_{\mathcal{T}} = \mathrm{GNN}(\mathbf{X},\mathbf{A},\mathbf{Y})$ . We provide more theoretical illustrations of the proposed structure-free graph condensation paradigm from the views of statistical learning and information flow, respectively, in Appendix D. The overall pipeline of the proposed Structure-Free Graph Condensation (SFGC) framework is shown in Fig. 2.
We consider two coupled stages in the proposed SFGC framework, i.e., graph-free data synthesis (1)  $\sim 4$  ) and condensed graph-free data evaluation (5), corresponding to two essential components: (1) the training trajectory meta-matching scheme and (2) the graph neural feature score metric. Concretely, as illustrated in Fig. 2, taking the given large-scale graph  $\mathcal{T}$  as input, we first (1) train an expert set of  $\mathrm{GNN}_{\mathcal{T}}$ , parameterized by  $\{\Theta_{\mathcal{T}}^{i}\}_{i = 1}^{K} = \{\pmb{\theta}_{t}^{*,i}\}_{t = 1}^{T}$ , denoting  $K$  numbers of expert training trajectories, each within  $T$  time steps. Then, we sample a single expert training trajectory  $i$  at  $t_0$  step, i.e.,  $\pmb{\theta}_{t_0}^{*,i}$ , and further use it to initialize the model (2)  $\mathrm{GNN}_S$  trained by the condensed graph-free data  $\mathcal{S}$ . Then, after  $q$  steps of  $\mathrm{GNN}_S$  and  $p$  steps of  $\mathrm{GNN}_{\mathcal{T}}$ , we conduct the meta-matching (3) between the long-term intervals of training trajectories  $[\widetilde{\pmb{\theta}}_{t_0},\widetilde{\pmb{\theta}}_{t_0 + q}]$  and  $[\pmb{\theta}_{t_0}^*,\pmb{\theta}_{t_0 + p}^* ]$  with the proposed meta-matching loss  $\mathcal{L}_{\mathrm{meta - tt}}$ . Next, the loss function back-propagates along  $\mathrm{GNN}_S$  and the condensed graph-free data  $\mathcal{S}$  is updated (4). After, the updated  $\mathcal{S}$  at the current step is used to calculate the GNTK-based graph neural feature score metric  $\gamma_{\mathrm{gnf}}$  for dynamic evaluation (5), along with the large-scale validation subgraph  $\mathcal{T}_{\mathrm{val}}$ . Finally, SFGC selects the optimal condensed graph-free data with the smallest  $\gamma_{\mathrm{gnf}}^*$  as the expressive substitution of the large-scale graph.
2.3 Training Trajectory Meta-matching
Different from existing graph condensation methods [27, 26] that conduct online gradient matching within the short range, i.e., step-by-step or single-step matching gradients between the large-scale graph and the condensed graph, the proposed SFGC matches their long-term GNN training trajectories with the guidance of the offline expert parameter distribution. Concretely, inspired by [4], we first train  $K$  numbers of GNN models with the same architecture denoted as  $\mathrm{GNN}_{\mathcal{T}}$  on the large-scale graph  $\mathcal{T}$ . Then, we save their network parameters  $\{\Theta_{\mathcal{T}}^i\}_{i=1}^K = \{\theta_t^{*,i}\}_{t=1}^T$  at certain epoch intervals, resulting in  $K$  numbers of expert training trajectories that have comprehensive knowledge of the large-scale graph in terms of GNN's training process. Such expert training trajectories further build a parameter distribution  $P_{\Theta_{\mathcal{T}}}$  denoting the GNN learning behavior on the large-scale graph. Note that such a parameter distribution is pre-calculated and stored. Hence, this process can be offline and separated from the end-to-end graph condensation pipeline, moderately reducing the online computation costs.
By sampling from the pre-derived parameter distribution  $P_{\Theta \tau}$ , we optimize the following objective for synthesizing  $\mathcal{S}$  as:
$$
\min  _ {\mathcal {S}} \mathrm {E} _ {\boldsymbol {\theta} _ {t} ^ {*}, i \sim P _ {\boldsymbol {\Theta}} \tau} \left[ \mathcal {L} _ {\text {m e t a - t t}} \left(\boldsymbol {\theta} _ {t} ^ {*} | _ {t = t _ {0}} ^ {p}, \widetilde {\boldsymbol {\theta}} _ {t} | _ {t = t _ {0}} ^ {q}\right) \right]. \tag {4}
$$
Note that  $\pmb{\theta}_{t}^{*}|_{t = t_0}^p$  and  $\widetilde{\pmb{\theta}}_t|_{t = t_0}^q$  denote the parameters of  $\mathrm{GNN}_{\mathcal{T}}$  within the range of  $(t_0,t_0 + p)$  and  $\mathrm{GNN}_S$  within  $(t_0,t_0 + q)$ , respectively, where  $t_0 < t_0 + p < T$ . More specifically,  $\mathcal{L}_{\mathrm{meta - tt}}$  calculates certain parameter training intervals within  $\left[\pmb{\theta}_{t_0}^{*,i},\pmb{\theta}_{t_0 + p}^{*,i}\right]$  and  $\left[\widetilde{\pmb{\theta}}_{t_0},\widetilde{\pmb{\theta}}_{t_0 + q}\right]$  as
$$
\mathcal {L} _ {\mathrm {m e t a - t t}} = \frac {\left\| \widetilde {\boldsymbol {\theta}} _ {t _ {0} + q} - \boldsymbol {\theta} _ {t _ {0} + p} ^ {*, i} \right\| _ {2} ^ {2}}{\left\| \widetilde {\boldsymbol {\theta}} _ {t _ {0}} - \boldsymbol {\theta} _ {t _ {0} + p} ^ {*, i} \right\| _ {2} ^ {2}}. \tag {5}
$$
Here, we initialize the parameter of  $\mathrm{GNN}_{\mathcal{S}}$  with that of  $\mathrm{GNN}_{\mathcal{T}}$  at  $t_0$  training step, so that we have  $\pmb{\theta}_{t_0}^{*,i} = \widetilde{\pmb{\theta}}_{t_0}$ . And we consider the expectation on  $\mathcal{S}$  w.r.t. different initialized parameter  $\pmb{\theta}_{t_0}^{*,i}$  in
Algorithm 1 Structure-Free Graph Condensation (SFGC)
Require: (1)  $P_{\Theta_{\mathcal{T}}}$ : Pretrained a set of  $K$  numbers of GNNs on the large-scale graph  $\mathrm{GNN}_{\mathcal{T}}$  parameterized by  $\Theta_{\mathcal{T}}$ ; (2)  $T_0$ : numbers of meta-matching steps; (3)  $T_1$ :  $\mathrm{GNN}_S$  training steps.
Ensure: A small-scale condensed graph-free data  $S = \left( {\widetilde{\mathbf{X}},\widetilde{\mathbf{Y}}}\right)$