SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 1. The aggregation step with graph  $\mathcal{G}$  increases the inner product between the kernel feature maps of samples in the same class and reduces or does not change the inner product between the kernel feature maps of samples in different classes.
This is a reasonable assumption because a useful graph should make the samples from different classes more distinguishable or at least make the samples from the same class more similar.
Theorem 2. Given a graph  $\mathcal{G}$  that satisfies Assumption 1, let  $\varphi$  and  $\varphi_{\mathcal{G}}$  be the kernel feature maps without and with aggregation on graph  $\mathcal{G}$  respectively. The corresponding negative Lagrangian dual objectives (to minimize) are denoted as  $\mathcal{L}(\mathbf{c}) \coloneqq \frac{1}{2} \sum_{i=1}^{n} \sum_{j=1}^{n} c_i c_j q_{ij} - \sum_{i=1}^{n} c_i$ , where  $q_{ij} = y_i y_j \varphi(\mathbf{x}_i)^\top \varphi(\mathbf{x}_j)$  and  $\mathcal{L}_{\mathcal{G}}(\mathbf{c}) \coloneqq \frac{1}{2} \sum_{i=1}^{n} \sum_{j=1}^{n} c_i c_j q_{ij}^{\mathcal{G}} - \sum_{i=1}^{n} c_i$ , where  $q_{ij}^{\mathcal{G}} = y_i y_j \varphi_{\mathcal{G}}(\mathbf{x}_i)^\top \varphi_{\mathcal{G}}(\mathbf{x}_j)$ . We have
$$
\mathcal {L} _ {\mathcal {G}} (\mathbf {c}) \geq \mathcal {L} (\mathbf {c}) + \mathcal {R} (\mathbf {c}), \tag {12}
$$
where  $\mathcal{R}(\mathbf{c})\geq 0$  is a regularization term inducing sparsity in  $\mathbf{c}$
The proof as well as the details about  $\mathcal{R}(\mathbf{c})$  are in Appendix C. According to Theorem 2, the aggregation step introduces an additional sparse regularization term  $\mathcal{R}(\mathbf{c})$ , which will make  $\mathbf{c}$  sparser, or in other words, reduce the number of support vectors  $|\mathcal{V}|$ . Considering both the theoretical and empirical results, we conclude that graph convolution leads to a tighter generalization error bound, which verifies the effectiveness of our GCKSVM. By the way, the interpretability of GCKSVM is higher than that of GCN, owing to the support vectors.
Besides SVM, our method can be generalized to many other machine learning problems. We show a few examples of node-level learning in the following context.
- GCKM for Spectral Clustering (GCKSC) For node clustering, it is natural to perform spectral clustering [Ng et al., 2001; Von Luxburg, 2007] using the kernel matrix  $\mathbf{K}^{(L)}$ . Instead of using the generally dense matrix  $\mathbf{K}^{(L)}$ , we retain only the largest values of each column to obtain a sparse matrix  $\hat{\mathbf{K}}^{(L)}$ , which often yields better clustering performance.
- GCKM for Principal Component Analysis (GCKPCA) It is straightforward to conduct Kernel PCA [Schölkopf et al., 1998b] using  $\mathbf{K}^{(L)}$ , where the principal components can be used as the representations of the nodes of  $\mathcal{G}$ . Please refer to Appendix F.5 for experiments.
- We can also adapt GCKM to other learning models such as self-expressive models [Elhamifar and Vidal, 2013], Fisher linear discriminant analysis [Fisher, 1936], canonical correlation analysis [Hardoon et al., 2004], etc.
Now we extend GCKM to graph-level learning. Suppose we have a set of graphs  $\mathbb{G} = \{\mathcal{G}_1, \mathcal{G}_2, \dots, \mathcal{G}_N\}$ , where  $|V_g| = n_g$ ,  $g = 1, 2, \dots, N$ . For each  $\mathcal{G}_g$ , we use (7) to obtain  $\mathbf{K}_g^{(L)}$  as well as the implicit node-level representations  $\mathbf{H}_g^{(L)}$ . For each  $\mathcal{G}_g$ , we compute the implicit graph-level representation as the summation of node-level representations, i.e.,
$$
\mathbf {u} _ {g} ^ {(L)} = \left(\mathbf {H} _ {g} ^ {(L)}\right) ^ {\top} \mathbf {1} _ {n _ {g}}, \quad g = 1, 2, \dots , N. \tag {13}
$$
Then for any two graphs  $\mathcal{G}_g, \mathcal{G}_{g'}$ , we have
$$
\mathcal {K} _ {g g ^ {\prime}} ^ {(L)} := \left(\mathbf {u} _ {g} ^ {(L)}\right) ^ {\top} \mathbf {u} _ {g ^ {\prime}} ^ {(L)} = \mathbf {1} _ {n _ {g}} ^ {\top} \mathbf {H} _ {g} ^ {(L)} \left(\mathbf {H} _ {g ^ {\prime}} ^ {(L)}\right) ^ {\top} \mathbf {1} _ {n _ {g ^ {\prime}}} = \mathbf {1} _ {n _ {g}} ^ {\top} \mathbf {K} _ {g g ^ {\prime}} ^ {(L)} \mathbf {1} _ {n _ {g ^ {\prime}}}, \tag {14}
$$
where  $\mathbf{K}_{gg'}^{(L)} \in \mathbb{R}^{n_g \times n_{g'}}$  can be computed using (5) recursively. Consequently, we can obtain a graph-level kernel matrix as
$$
\mathcal {K} _ {\mathbb {G}} ^ {(L)} = \left[ \mathcal {K} _ {g g ^ {\prime}} ^ {(L)} \right] _ {(g, g ^ {\prime}) \in [ N ] \times [ N ]} = \left[ \begin{array}{c c c} \mathbf {1} _ {n _ {1}} ^ {\top} & \dots & \mathbf {0} \\ \vdots & \ddots & \vdots \\ \mathbf {0} & \dots & \mathbf {1} _ {n _ {N}} ^ {\top} \end{array} \right] \left[ \begin{array}{c c c} \mathbf {K} _ {1 1} ^ {(L)} & \dots & \mathbf {K} _ {1 N} ^ {(L)} \\ \vdots & \ddots & \vdots \\ \mathbf {K} _ {N 1} ^ {(L)} & \dots & \mathbf {K} _ {N N} ^ {(L)} \end{array} \right] \left[ \begin{array}{c c c} \mathbf {1} _ {n _ {1}} & \dots & \mathbf {0} \\ \vdots & \ddots & \vdots \\ \mathbf {0} & \dots & \mathbf {1} _ {n _ {N}} \end{array} \right] \tag {15}
$$
Using (14) to compute  $\kappa_{\mathbb{G}}^{(L)}$  has a high time complexity of  $N(N - 1) / 2$  and using (15) has a high space complexity of  $O((\sum_{g = 1}^{N}n_{g})^{2})$ . To reduce the cost, we can perform (15) on mini-batches of  $\mathbb{G}$ . We have the following result (proved in Appendix D)
Proposition 1. Suppose  $k^{(L)}$  is a Gaussian kernel with  $0 < \sigma_L < \infty$ . Then  $\mathcal{K}_{\mathbb{G}}^{(L)}$  is positive definite.
$\kappa_{\mathbb{G}}^{(L)}$  can be applied to various graph-level learning tasks such as graph classification and clustering.
3.4 Fast and Explicit Feature Transformation for GCKM
GCKMs can be stacked as multi-layer model and applied to a wide range of tasks by calculating the kernel matrix, and has the advantages of being fast, explainable, and requiring few parameters. Despite this, the time and space complexities of GCKM are  $O(n^{2})$ , which limits the application to very large graph datasets. To tackle this issue, we consider approximating the kernel by
$$
k (\mathbf {x}, \mathbf {x} ^ {\prime}) = \langle \phi (\mathbf {x}), \phi \left(\mathbf {x} ^ {\prime}\right) \rangle \approx \langle \psi (\mathbf {x}), \psi \left(\mathbf {x} ^ {\prime}\right) \rangle , \tag {16}
$$
where  $\psi (\cdot)$  is a low-dimensional map using Random Fourier Feature [Rahimi and Recht, 2007]. The following theorem provides the key foundation of this approximation method:
Theorem 3 (Bochner's theorem [Rudin, 1994]). A continuous kernel  $k(\mathbf{x}, \mathbf{x}^{\prime}) = k(\mathbf{x} - \mathbf{x}^{\prime})$  on  $\mathbb{R}^{m}$  is positive definite if and only if  $k(\delta)$  is the Fourier transform of a non-negative measure.
Specifically, for Gaussian kernel, let  $p_{\mathrm{RBF}}(\omega)$  be a probability distribution and also the Fourier transformation of  $k_{\mathrm{RBF}}(\delta)$ . We first randomly sample  $\{\omega_1^{(l)}, \omega_2^{(l)}, \dots, \omega_D^{(l)}\}$  from the probability distribution  $p_{\mathrm{RBF}}(\omega) = \mathcal{N}(0, \frac{1}{\sigma}\mathbf{I})$ . Subsequently the multi-layer GCKM with explicit feature transformation, called GCKM-E, on the  $i$ -th sample is defined as  $\mathbf{z}_i^{(l + 1)} = \psi_{(l)}(\bar{\mathbf{z}}_i^{(l + 1)})$  where
$$
\psi_ {(l)} \left(\bar {\mathbf {z}} _ {i} ^ {(l + 1)}\right) = \sqrt {\frac {1}{D}} \left[ \cos \left(\omega_ {1} ^ {\top} \bar {\mathbf {z}} _ {i} ^ {(l + 1)}\right), \dots , \cos \left(\omega_ {D} ^ {\top} \bar {\mathbf {z}} _ {i} ^ {(l + 1)}\right), \sin \left(\omega_ {1} ^ {\top} \bar {\mathbf {z}} _ {i} ^ {(l + 1)}\right), \dots , \sin \left(\omega_ {D} ^ {\top} \bar {\mathbf {z}} _ {i} ^ {(l + 1)}\right) \right] ^ {\top} \tag {17}
$$
and  $\mathbf{z}_i^{(l + 1)}$  and  $\bar{\mathbf{z}}_i^{(l + 1)}$  denote  $i$ -th columns vectors drawn from the transposes of  $\mathbf{Z}^{(l + 1)}$  and  $\bar{\mathbf{Z}}^{(l + 1)} = \hat{\mathbf{A}}^q\mathbf{Z}^{(l)}$ , respectively. For detailed derivations, please refer to Appendix E. This approach allows GCKM to flexibly choose the dimensions of explicit node representations when  $n$  is very large.
In this section, we evaluate GCKMs on several real-world graph datasets. Comparison experiments of both semi-supervised node classification and node clustering show GCKMs' accuracy and efficiency. Due to the space limitation, some important experimental results are deferred to Appendix F.2 (more types of datasets and large-scale dataset), Appendix F.3 (over-smoothing problem), Appendix F.4 (GCKM with various kernels) and Appendix F.5 (GCKPCA visualization).
Table 1: Accuracy (mean% and standard deviation%) of all methods, note that the best results are highlighted in orange and the second-best results are highlighted in blue.
Standard Split	Random Split
Cora	CiteSeer	Pubmed	Cora	CiteSeer	Pubmed
Chebyshev	75.0 (0.7)	63.8 (0.6)	74.7 (0.8)	76.1 (1.5)	63.4 (3.8)	75.4 (2.8)
GraphSAGE	76.9 (0.5)	63.3 (0.7)	74.8 (0.2)	76.3 (0.9)	63.2 (1.1)	73.9 (1.8)
GAT	82.0 (0.5)	69.8 (0.4)	77.5 (0.2)	79.2 (1.4)	65.8 (2.9)	78.1 (1.7)
GCN	81.7 (0.7)	70.5 (0.5)	78.4 (0.4)	80.6 (1.4)	69.1 (1.8)	77.4 (2.1)
SGC	81.3 (0.0)	68.4 (0.1)	78.6 (0.1)	79.6 (1.3)	68.0 (1.7)	76.6 (2.4)
APPNP	79.5 (0.6)	70.0 (0.8)	78.8 (1.3)	80.1 (1.8)	69.2 (2.1)	77.7 (2.0)
JKNet	77.9 (0.8)	72.3 (0.1)	80.1 (0.2)	78.0 (1.7)	66.9 (2.3)	77.0 (1.7)
DAGNN	82.6 (1.1)	71.0 (0.5)	80.0 (1.0)	81.8 (1.2)	68.4 (1.4)	78.8 (1.4)
AdaGCN	77.1 (0.1)	69.4 (0.2)	78.0 (0.1)	76.7 (1.8)	67.0 (1.2)	77.3 (1.3)
AMGCN	81.3 (0.4)	70.4 (0.2)	75.5 (0.9)	80.9 (1.2)	68.1 (1.4)	74.2 (2.1)
DefGCN	77.8 (1.0)	67.5 (1.7)	77.9 (0.6)	78.4 (2.3)	67.8 (2.6)	77.1 (1.7)
GCKSVM-E	82.4 (0.3)	72.4 (0.4)	79.1 (0.4)	80.8 (0.8)	68.5 (1.3)	79.2 (1.2)
GCKSVM	82.4 (0.0)	72.3 (0.0)	79.8 (0.0)	83.3 (0.8)	71.9 (1.0)	80.9 (0.5)
Datasets We employ three most widely adopted citation networks Cora, Citeseer, and Pubmed for evaluations, they are formed as unweighted and undirected graphs where each node represents a paper and edges denote citations between papers. As for graph classification, IMDB-BINARY and IMDB-MULTI are movie collaboration datasets; COLLAB is a scientific collaboration dataset; MUTAG, PROTEINS, and PTC are three bioinformatics datasets. Details about these datasets and more datasets (e.g., social networks, paper networks and OGB-Arxiv) can be found in Appendix F.1.
Compared Methods For node classification, two types of GNNs are selected: Chebyshev [Defferrard et al., 2016], GraphSAGE [Hamilton et al., 2017], GAT [Velickovic et al., 2018], GCN [Kipf and Welling, 2017] and SGC [Wu et al., 2019] serve as classical baseline GNNs; APPNP [Klicpera et al., 2019], JKNet [Xu et al., 2018], DAGNN [Liu et al., 2020], AdaGCN [Sun et al., 2021], AMGCN[Wang et al., 2020], DefGCN [Park et al., 2022] are state-of-the-art models, in particular the former four ones are Deep GCNs. For node clustering, K-means and Spectral Clustering are two traditional baseline methods and GAE and VGAE [Kipf and Welling, 2016] are GCN-based baseline methods; ARGA, ARVGA [Pan et al., 2018], DGI [Velickovic et al., 2019], MVGRL [Hassani and Ahmadi, 2020], GALA [Park et al., 2019], DFCN [Tu et al., 2021] and  $\mathrm{S}^3\mathrm{GC}$  [Devvrit et al., 2022] are all state-of-the-art GNN-based approaches. For graph classification, WL subtree kernel [Shervashidze et al., 2011], AWL [Ivanov and Burnaev, 2018] are two baselines; DCNN [Atwood and Towsley, 2016], PATCHY-SAN [Niepert et al., 2016] and DGCNN [Zhang et al., 2018] are three deep learning methods; GCN, GraphSAGE and GIN [Xu et al., 2019] are GNN-based methods.
Experimental Settings All the above methods are set as default following the original paper. We use a 2-layer GCKM for the experiments and the kernel is specified as Gaussian kernel, more detailed settings of GCKM can be found in Appendix F.1. In node classification, following vanilla GCN [Kipf and Welling, 2017], the nodes are split into three sets: train set containing 20 samples per class, validation set and test set with 500 and 1,000 samples respectively, and the standard fixed split is same to [Yang et al., 2016]. GCKM paired with kernel-based Spectral Clustering is leveraged for node clustering, which is described in Section 3.3. For graph classification task, we adopt the 10-fold cross validation following the settings of GIN Xu et al. [2019].
Semi-supervised Node Classification The results are reported in Table 1, where GCKSVM-E denotes GCKSVM with explicit features. Note that, we tune the hyperparameters of GCKSVM on the validation set and report the best performance. We have the following observations. 1) In the case of standard split, GCKSVM achieves decent performance and beats several GNNs including deep GCNs, even though it is a shallow model. Besides, owing to that GCKSVM can obtain globally optimal solutions, the standard deviations are zero. 2) In the case of random split, GCKSVM outperforms all GNNs on the three datasets and has the smallest standard deviations, showing higher stability. 3) Deep GCNs are not substantially better than, or even worse than baselines, which further validates that deepening is not essential in common graph-based tasks. 4) GCKSVM-E performs comparably to GCKSVM and other SOTA methods.
Decision Boundary To further explore the expressiveness, we illustrate the decision boundaries of SGC, APPNP, GCN, and the proposed GCKSVM in Figure 3. We generate a synthetic two-circle
Table 2: Graph classification accuracy (mean% and standard deviation%) of all methods, note that the best results are highlighted in orange and the second-best results are highlighted in blue.
IMDB-B	IMDB-M	COLLAB	MUTAG	PROTEINS	PTC
WL subtree	73.8 (3.9)	50.9 (3.8)	78.9 (1.9)	90.4 (5.7)	75.0 (3.1)	59.9 (4.3)
DCNN	49.1	33.5	52.1	67	61.3	56.6
PATCHYSAN	71.0 (2.2)	45.2 (2.8)	72.6 (2.2)	92.6 (4.2)	75.9 (2.8)	60.0 (4.8)
DGCNN	70	47.8	73.7	85.8	75.5	58.6
AWL	74.5 (5.9)	51.5 (3.6)	73.9 (1.9)	87.9 (9.8)	-	-
MLP	73.7 (3.7)	52.3 (3.1)	79.2 (2.3)	84.0 (6.1)	76.0 (3.2)	66.6 (6.9)
GIN	75.1 (5.1)	52.3 (2.8)	80.2 (1.9)	89.4 (5.6)	76.2 (2.8)	64.6 (7.0)
GCN	74.0 (3.4)	51.9 (3.8)	79.0 (1.8)	85.6 (5.8)	76.0 (3.2)	64.2 (4.3)
GraphSAGE	72.3 (5.3)	50.9 (2.2)	-	-	75.9 (3.2)	63.9 (7.7)
GCKSVM	75.4 (2.4)	53.9 (2.8)	81.7 (1.5)	88.7 (7.6)	74.5 (3.9)	67.7 (5.4)
Figure 3: Decision boundary visualizations of SGC, APPNP, GCN, and GCKM on Circle (row 1), Chameleon-2D (row 2) and Cora-2D (row 3), where Circle is a synthetic dataset and Chameleon-2D and Cora-2D are two preprocessed real-world datasets.
dataset named Circle. We also consider two real datasets, denoted by Cora-2D and Chameleon-2D, where we select two classes of samples from the original datasets and map them to 2-D space using tSNE [Van der Maaten and Hinton, 2008]. We randomly labeled 200 samples for each dataset and constructed adjacency matrices by  $k$ -nearest neighbor algorithm, where the original graph structures are not used here. The details are described in Appendix F.1. Cora-2D is the simplest case where the decision boundary is easy to construct, Circle is with a clear but non-linear decision boundary, and Chameleon-2D is the most complex one.
We have the following conclusions. 1) Consistent with our previous discussion, the nonlinearity provided by SGC is low even if we increase the power of  $\hat{\mathbf{A}}$  to a large value such as 50. Therefore, SGC performs poorly on Circle and Chameleon-2D, where its decision boundaries are approximately linear. This is unexpected considering its performance in the previous experiment. An explanation is that these commonly used graph datasets can be well classified linearly in high-dimension spaces, rather than these 2D spaces. 2) The deep model APPNP fits these datasets with similar boundaries as GCN, either simple or complex, revealing that existing deep GCNs may not provide more powerful expressiveness. 3) GCKSVM gains advantages over the three other models, whose decision boundaries fit these datasets well, attributed to which GCKM explicitly maps the samples to high-dimensional space to achieve better separability.
Graph Classification The results of graph classification tasks are recorded in Table 2. Compared to both classical baselines and GNNs, especially GIN designed for Graph-level tasks, GCKSVM shows
Table 3: ACC, NMI, and ARI of all methods, note that the best results are highlighted in orange and the second-best results are highlighted in blue.
Cora	Citeseer	Pubmed
ACC	NMI	ARI	ACC	NMI	ARI	ACC	NMI	ARI
K-means	49.20	32.10	22.90	54.00	30.50	27.80	59.50	31.50	28.10
SC	36.70	12.60	3.10	23.80	5.50	1.00	52.80	9.70	6.20
GAE	59.60	42.90	34.70	40.80	17.60	12.40	67.20	27.70	27.90
VGAE	50.20	32.90	25.40	46.70	26.00	20.50	63.00	22.90	21.30
ARGA	64.00	44.90	35.20	35.20	35.00	34.10	66.80	30.50	29.50
ARVGA	64.00	45.00	37.40	54.40	26.10	24.50	69.00	29.00	30.60
DGI	55.40	41.10	32.70	51.40	31.50	32.60	58.90	27.70	31.50
MVGRL	73.20	56.20	51.90	68.10	43.20	43.40	69.30	34.40	32.30
GALA	74.59	57.67	53.15	69.32	44.11	44.60	69.39	32.73	32.14
DFCN	64.07	48.24	39.17	69.50	43.90	45.50	69.32	32.19	31.55
S3GC	74.20	58.80	54.40	68.80	44.10	44.80	71.30	33.30	34.50
GCKSC	74.30	54.95	52.89	71.27	43.57	46.51	71.31	32.24	34.21
(a) Adjacency Matrix (36.7)
(b) Kernel Matrix (63.2)
Figure 4: Visualizations of the adjacency matrix  $\hat{\mathbf{A}}^q$ , kernel matrix  $[k(\mathbf{x}_i,\mathbf{x}_j)]_{i,j=1}^n$ , and GCKM matrix  $\mathbf{K}^{(2)}$  on Cora, with corresponding best-tuned clustering accuracy.
(c) GCKM Matrix (74.3)
competitive power in graph classification. It is worth noting that GCKSVM outperforms the GNNs in the two relatively large datasets IMDB-M and COLLAB.
Node Clustering The comparison of all node clustering methods is shown in Table 3.
Similar to the results of node classification, GCKM is well extended to node clustering tasks and is competitive with the deep-learning-based models. It is impressive that the vanilla GCN variants for clustering, GAE and VGAE both perform much worse than our GCKSC, demonstrating that GCKM can yield a discriminative kernel matrix. This can also be observed in Figure 4, where the powered adjacency matrix, the kernel matrix calculated on node features, and the kernel matrix of GCKM are visualized.
Time Costs Figure 5 shows the runtime of all methods on Pubmed. GCKSVM is much more efficient than all other methods except SGC.
Figure 5: Runtimes of all methods.
After analyzing the limitations of GCN, SGC and deep GCNs, we proposed a framework of GCKM. GCKM integrated graph convolution with kernel learning and exhibited higher efficiency, interpretability, and stability. We extended GCKM to many graph-based machine-learning tasks, like node-level and graph-level classification and clustering. Experiments showed that GCKMs are, at least, as accurate as GCN and other SOTA GNNs. More importantly, compared to the representative simplified model SGC, GCKM is much more effective in handling non-linear data. One possible limitation of this work is that we did not systematically test other kernel functions, though we have found that the Gaussian kernel is better than the polynomial for GCKM.
This work was partially supported by the National Natural Science Foundation of China under Grants No.62106211 and No.62072151, the General Program JCYJ20210324130208022 of Shenzhen Fundamental Research, the research funding T00120210002 of Shenzhen Research Institute of Big Data, the Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence, Anhui Provincial Natural Science Fund for the Distinguished Young Scholars (2008085J30), Open Foundation of Yunnan Key Laboratory of Software Engineering (2023SE103), CCF-Baidu Open Fund and CAAI-Huawei MindSpore Open Fund, and the funding UDF01001770 of The Chinese University of Hong Kong, Shenzhen.
James Atwood and Don Towsley. Diffusion-convolitional neural networks. In Advances in Neural Information Processing Systems, pages 1993-2001, 2016.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482, 2002.
Peter Bartlett and John Shawe-Taylor. Generalization performance of support vector machines and other pattern classifiers. Advances in Kernel methods—support vector learning, pages 43-54, 1999.
Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In Proceedings of the 35th AAAI Conference on Artificial Intelligence, pages 3950-3957, 2021.
Jinyu Cai and Jicong Fan. Perturbation learning based anomaly detection. In Advances in Neural Information Processing Systems, 2022.
Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang. Efficient deep embedded subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21-30. IEEE, 2022.
Heng Chang, Yu Rong, Tingyang Xu, Yatao Bian, Shiji Zhou, Xin Wang, Junzhou Huang, and Wenwu Zhu. Not all low-pass filters are robust in graph convolutional networks. In Advances in Neural Information Processing Systems, pages 25058-25071, 2021.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning, pages 1725-1735, 2020.
Zhaoliang Chen, Zhihao Wu, Zhenghong Lin, Shiping Wang, Claudia Plant, and Wenzhong Guo. Agnn: Alternating graph-regularized neural networks to alleviate over-smoothing. IEEE Transactions on Neural Networks and Learning Systems, pages 1-13, 2023.
Zhaoliang Chen, Zhihao Wu, Shiping Wang, and Wenzhong Guo. Dual low-rank graph autoencoder for semantic and topological networks. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, pages 4191-4198, 2023.
Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. In Advances in Neural Information Processing Systems, pages 9936-9949, 2021.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273-297, 1995.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pages 3837-3845, 2016.
Fnu Devvrit, Aditya Sinha, Inderjit S. Dhillon, and Prateek Jain. S3GC: scalable self-supervised graph clustering. In Advances in Neural Information Processing Systems, 2022.
A Details about Recursive Formulations of GCKM
An  $L$ -layer GCKM on node feature  $\mathbf{X}$  can be written as
$$
\mathbf {H} ^ {(L)} = \phi_ {(L)} \left(\hat {\mathbf {A}} ^ {q} \dots \phi_ {(l)} \left(\hat {\mathbf {A}} ^ {q} \dots \phi_ {(0)} \left(\hat {\mathbf {A}} ^ {q} \mathbf {X}\right) \dots\right) \dots\right). \tag {18}
$$
Note that  $\mathbf{H}^{(l + 1)}$  is actually implicit and we directly obtain the kernel matrix  $\mathbf{K}^{(l + 1)} = \mathbf{H}^{(l + 1)}\mathbf{H}^{(l + 1)^{\top}}$  in practice. Now we show how to derive  $\mathbf{K}^{(l + 1)}$  only by kernel matrix  $\mathbf{K}^{(l)}$ . Take the Gaussian kernel as an example, we have  $K_{ij}^{(l + 1)} = \exp \left(-\frac{\Delta_{ij}^{(l + 1)}}{2\sigma_{l + 1}^2}\right)$ ,  $(i,j) \in [n] \times [n]$ , where  $\Delta^{(l + 1)}$  is a squared-distance matrix of the rows of  $\bar{\mathbf{H}}^{(l + 1)}$ . To be more precise,
$$
\begin{array}{l} \Delta_ {i j} ^ {(l + 1)} = \left\| \bar {\mathbf {h}} _ {i} ^ {(l + 1)} - \bar {\mathbf {h}} _ {j} ^ {(l + 1)} \right\| ^ {2} (19) \\ = \bar {\mathbf {h}} _ {i} ^ {(l + 1)} \left(\bar {\mathbf {h}} _ {i} ^ {(l + 1)}\right) ^ {\top} - 2 \bar {\mathbf {h}} _ {i} ^ {(l + 1)} \left(\bar {\mathbf {h}} _ {j} ^ {(l + 1)}\right) ^ {\top} + \bar {\mathbf {h}} _ {j} ^ {(l + 1)} \left(\bar {\mathbf {h}} _ {j} ^ {(l + 1)}\right) ^ {\top}, (20) \\ \end{array}
$$
where  $\bar{\mathbf{h}}_i^{(l + 1)}$  (a row vector) is the representation of the  $i$ -th node drawn from
$$
\bar {\mathbf {H}} ^ {(l + 1)} = \hat {\mathbf {A}} ^ {q} \mathbf {H} ^ {(l)} = \hat {\mathbf {A}} ^ {q} \phi_ {(l - 1)} (\bar {\mathbf {H}} ^ {(l)}). \tag {21}
$$
Thus  $\bar{\mathbf{h}}_i^{(l + 1)} = (\hat{\mathbf{A}}^q)_i\phi_{(l - 1)}(\bar{\mathbf{H}}^{(l)})$ , it follows from (19) that