SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Settings about Figure 1. We set the hidden dimensions as 32 for all hidden layers of GCN, APPNP, and JKNet, while SGC only has a learnable matrix  $\mathbf{W} \in \mathbb{R}^{n \times c}$  where  $n$  is the number of nodes and  $c$  is the number of classes, and the learning rate is selected in  $\{1 \times 10^{-2}, 1 \times 10^{-3}, 1 \times 10^{-4}\}$  and weight decay is selected in  $\{5 \times 10^{-4}, 5 \times 10^{-5}, 5 \times 10^{-6}\}$ .
Settings of Boundary Visualization For better illustration, we conduct a visualization experiment on the decision boundaries of SGC, APPNP, GCN, and GCKSVM. The purpose of this experiment is to explore how the three methods perform on linear, non-linear, and more complex data. So we select 2 classes of samples from Cora and Chameleon and map them to 2-D space by t-SNE. Besides, we generate a synthetic dataset (called Circle) consisting of noisy samples drawn from two concentric circles using Scikit-learn. For better boundary and performance, the numbers of layers are 2 and 8 for GCN and APPNP respectively, the hidden dimensions are 32, and the learning rate is set as  $1 \times 10^{-3}$  and weight decay is  $5 \times 10^{-5}$  for all models. SGC has only one learnable matrix and we set the power of adjacency as 50 to show that the graph structure provides SGC with non-linearity but the non-linearity is very low.
F.2 Supplementary Experiments of Node Classification
In this section, we evaluate GCKSVM on more types of graph datasets. As reported in Table 6, it can be seen that GCKSVM still has superior performance compared to the most state-of-the-art GNNs. Furthermore, we also evaluate the proposed GCKM on the OGB dataset Arxiv with over 160k nodes. Table 7 demonstrates that GCKM is still competitive with these GNNs on large-scale datasets. Although the complexity of GCKM is quadratic, we provide an efficient variant GCKM-E in Section 3.4 that leverages the random Fourier feature to explicitly derive a low-dimensional output instead of a kernel matrix. With this output, we can apply fast linear methods (e.g. linear SVM) for downstream tasks, which is still more efficient than GNNs needing a lengthy training process.
F.3 Experiments and Discussions on Over-smoothing Issue
Thanks for your insightful comment. We have supplemented an experiment on this issue, which revealed that GCKM can be deeper and alleviate the over-smoothing issue. To be specific, two situations are considered in this experiment:
- GCKM with fixed 2 layers and varied hops of neighbors per aggregation ( $q$  in Eq. (40))
$$
\mathbf {H} = \phi_ {(1)} \left(\hat {\mathbf {A}} ^ {q} \phi_ {(0)} \left(\hat {\mathbf {A}} ^ {q} \mathbf {X}\right)\right). \tag {40}
$$
- GCKM with fixed 2 hops of neighbors per aggregation and varied layers
$$
\mathbf {H} = \phi_ {(L)} \left(\hat {\mathbf {A}} ^ {2} \dots \phi_ {(0)} \left(\hat {\mathbf {A}} ^ {2} \mathbf {X}\right)\right). \tag {41}
$$
From Figure 6, we have the following observations:
Table 8: Accuracy of GCKSVM with different kernel functions on Cora, Citeseer, and Pubmed.
Cora	Citeseer	Pubmed
Polynomial	83.0	70.6	80.6
Sigmoid	79.7	69.2	75.6
Laplacian	82.2	70.5	79.0
Gaussian	82.4	72.3	79.8
- Deep GCKM performs more stably than deep GCN.
- GCKM's performance first improves then slightly decreases with increasing layers/hops
- Particularly, GCKM with fixed layers performs better and decreases less.
It is known that over-smoothing is caused by the aggregation step, that is, multiplying  $\hat{\mathbf{A}}$  makes the representations of different nodes more and more indistinguishable. However, the analysis of over-smoothing does not consider activation functions and learnable weights, and it theoretically exists when the power of  $\hat{\mathbf{A}}$  tends to infinity [Li et al., 2018], which does not match the fact that GCN collapses with only 8 layers. A recent study of [Cong et al., 2021] has pointed out that the over-smoothing problem might be an artifact of theoretical analysis and the failure of deep GNNs may not only be caused by the over-smoothing issue in the aggregation-message-passing step.
Although GCKM and GCN share a familiar aggregation step, the main difference between them is the transformation step, namely, GCN uses a linear layer to conduct explicit dimension reduction while GCKM employs implicit high-dimensional feature mapping. GCKM implicitly maps node features to a high-dimensional (even infinite) space after aggregation, and there may exist an appropriate space where node representation can be distinguished. In contrast, a recent study of [Guo et al., 2023] has found that the node representations processed by deep GCN would collapse to be low-rank and lose expressive power. The appropriate space can be found by tuning the hyperparameters of GCKM, however, the number of hyperparameters will increase when building a deeper GCKM and it is time-consuming to search this space. That may be the reason why GCKM decreases slightly with too many layers and why fixing the number of layers improves the performance.
F.4 Experiments on Various Kernels
To solve the limitations we stated in the conclusion section, we provided a complementary experiment on various kernel functions during the rebuttal process. Table 8 records the results corresponding to 2nd order polynomial kernel, sigmoid kernel, and Laplacian kernel. All the kernel functions show decent performance and the Gaussian kernel performs the best. Note that, this experiment is conducted with GCKSVM, under the settings of node classification with standard split.
F.5 Experiments of GCKPCA
We added a visualization experiment for GCKPCA. To be specific, we first map the node features to 2-D space by PCA, Graph-regularized PCA (GPCA) [Jiang et al., 2013] and GCKPCA (row 1), then further try to map them to 32-D space and leverage t-SNE to obtain the 2-D results (row 2). Figure 7 illustrates the comparison of mapping results of three methods. It can be observed that the subfigures of GCKPCA and GCKPCA + t-SNE both show the best separability between different classes, and GPCA performs slightly better than PCA.
F.6 Empirical Experiments regarding Theorem 1
Theorem 1 demonstrated the connections between the generalization bound and graph structure, and experiments further provided the evidence. To verify the conclusion we have made from Theorem 1 in the main paper, we report the training error and numbers of support vectors in Table 9. In fact, we consider three cases:
- Non-graph-convolution: We replaced the affinity matrix  $\hat{\mathbf{A}}$  in GCKM with an identity matrix  $\mathbf{I}_n$ , which means the graph structure is not used.
(b) GPCA
(c) GCKPCA
(a) PCA
(d) PCA + t-SNE
Figure 7: Visualization of PCA, GPCA and GCKPCA.
(e) GPCA + t-SNE
(f) GCKPCA + t-SNE
Table 9: The values of the training error (TE) and the number of support vectors  $|\mathcal{V}|$ . Note that  $\lambda$  is the regularization parameter of SVM,  $\mathbf{I}_n$  and  $\mathbf{1}_n$  are  $n \times n$  identity matrix and all-ones matrix respectively.
\( \mathbf{I}_n \)	\( \mathbf{1}_{n\times n} - \mathbf{I}_n \)	\( \hat{\mathbf{A}}^q \)
TE	\( |\mathcal{V}| \)	TE	\( |\mathcal{V}| \)	TE	\( |\mathcal{V}| \)
Cora	λ = 1	0.01	80	0.50	32	0.20	40
λ = 10	0.00	72	0.50	32	0.00	30
λ = 50	0.00	72	0.50	32	0.00	24
λ = 100	0.00	72	0.50	32	0.00	24
λ = 1000	0.00	72	0.50	32	0.00	24
CiteSeer	λ = 1	0.02	120	0.50	62	0.20	120
λ = 10	0.00	112	0.50	62	0.00	105
λ = 50	0.00	111	0.50	62	0.00	102
λ = 100	0.00	111	0.50	62	0.00	102
λ = 1000	0.00	111	0.50	62	0.00	102
Pubmed	λ = 1	0.12	1206	0.48	802	0.18	1366
λ = 10	0.05	755	0.48	802	0.10	755
λ = 50	0.02	617	0.48	794	0.07	571
λ = 100	0.01	574	0.48	802	0.07	508
λ = 1000	0.00	548	0.48	804	0.11	396
- Strongly connected graph:  $\hat{\mathbf{A}}$  in GCKM is replaced with  $\mathbf{I}_n - \mathbf{1}_{n\times n}$ , which means every node is connected with all other nodes.
The normal GCKM.
We see that with fixed  $\lambda$ , graph convolution significantly reduces the number of support vectors, which verifies the correctness of Theorem 2. Thus, according to Theorem 1, the upper bound (monotonically increasing with  $|\mathcal{V}|$ ) of test error can be tighter compared to the one using a non-graph-convolution kernel, which means that the test error is potentially smaller. So we can conclude that the graph structure significantly improves the quality of the kernel matrix.