SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhihao Wu $^{1}$ , Zhao Zhang $^{1,2}$ , Jicong Fan $^{1,3*}$
<sup>1</sup>Shenzhen Research Institute of Big Data, Shenzhen, China
$^{2}$ Hefei University of Technology, Hefei, China
<sup>3</sup>The Chinese University of Hong Kong, Shenzhen, China
{zhihaowu1999,cszzhang}@gmail.com，fanjicong@cuhk.edu.cn
Graph convolutional networks (GCN) with one or two hidden layers have been widely used in handling graph data that are prevalent in various disciplines. Many studies showed that the gain of making GCNs deeper is tiny or even negative. This implies that the complexity of graph data is often limited and shallow models are often sufficient to extract expressive features for various tasks such as node classification. Therefore, in this work, we present a framework called graph convolutional kernel machine (GCKM)<sup>1</sup> for graph-based machine learning. GCKMs are built upon kernel functions integrated with graph convolution. An example is the graph convolutional kernel support vector machine (GCKSVM) for node classification, for which we analyze the generalization error bound and discuss the impact of the graph structure. Compared to GCNs, GCKMs require much less effort in architecture design, hyperparameter tuning, and optimization. More importantly, GCKMs are guaranteed to obtain globally optimal solutions and have strong generalization ability and high interpretability. GCKMs are composable, can be extended to large-scale data, and are applicable to various tasks (e.g., node or graph classification, clustering, feature extraction, dimensionality reduction). The numerical results on benchmark datasets show that, besides the aforementioned advantages, GCKMs have at least competitive accuracy compared to GCNs.
Graph data are prevalent in science and engineering. In recent years, neural networks and deep learning methods have shown promising performance in handling various graph data. Graph Neural Networks (GNNs) have been developed into various basic frameworks, including famous GCN [Kipf and Welling, 2017], GIN [Xu et al., 2019], GraphSAGE [Hamilton et al., 2017], GAT [Velickovic et al., 2018], etc. As one of the most attractive GNN paradigms, GCN, the graph convolutional network [Kipf and Welling, 2017], has gained widespread attention since it was proposed. Many widely developed domains like machine learning [Chen et al., 2023b; Cai et al., 2022; Fan et al., 2022; Fan, 2022b], data mining [Chang et al., 2021; Fan, 2021; Wu et al., 2022], computer vision [Guan et al., 2022; Cai and Fan, 2022; Wu et al., 2023], etc., have also made extensive use of numerous GCN variants. Despite the better performance compared to classical graph methods (e.g., DeepWalk [Perozzi et al., 2014], LINE [Tang et al., 2015] and node2vector [Grover and Leskovec, 2016]), some recent studies have pointed out a few drawbacks of GCNs [Wang et al., 2020; Bo et al., 2021; Wang et al., 2023]. Perhaps the failure of deep GCNs is the most notorious one that severely restricted the further development of GCNs.
The source code is available at https://github.com/ZhihaoWu99/GCKM.
Containing both the self features and pair-wise relations of entities, graph is viewed as a more complex non-Euclidean data type, so there is a natural and wide desire to construct deeper GCNs. Unfortunately, the performance of GCN decreases when stacking more graph convolutional layers, which is defined as the so-called over-smoothing issue [Li et al., 2018]. For instance, even a 7-layer GCN will inevitably generate indistinguishable node embeddings and lead to over  $30\%$  accuracy degradation [Liu et al., 2020]. Recently, some methods modifying the basic structure of GCN have been proposed to handle the over-smoothing issue and make GCN deeper [Xu et al., 2018; Klicpera et al., 2019; Yang et al., 2022]. Indeed, most of them have successfully alleviated the over-smoothing issue. But it seems counter-intuitive that these elaborate deep GCNs, which stack dozens or even hundreds of layers, can only laboriously maintain the performance that a 2-layer vanilla GCN achieves easily. We apply some GCNs to a node classification experiment on Cora following the settings of
the original GCN, as shown in Figure 1. APPNP [Klicpera et al., 2019] and JKNet [Xu et al., 2018] are two well-known models that attempted to pass information from shallow layers to the final output but benefited very limited from the deep neural networks. These explorations raise a question:
Figure 1: Accuracy and standard deviations of deep GCNs, SGC, and GCN with diverse layer/hop numbers.
Do we really need deep GCNs in common graph-based tasks?
Actually, there have also been some efforts paid to simplifying GCN, showing comparative effectiveness to vanilla GCN, like SGC [Wu et al., 2019], which removed nonlinearity and collapsed weight matrices to accelerate GCN. Impressively, it gained equal or even better performance on the benchmark graph datasets, which can partially be observed in Figure 1. Inspired by the success of SGC, some research moved further to decouple the graph convolution by individually performing an MLP (usually 2-layer) and several (can be hundreds) neighbor aggregations, called decoupled-style GCN [Nt and Maehara, 2019; Wang et al., 2021]. All these works revealed that simple GCNs can achieve encouraging performance on commonly used graph datasets such as Cora, Citeseer, and Pubmed [Sen et al., 2008]. In these datasets, the numerical node features and the adjacency matrices are already discriminative, which means further feature extractions via neural networks do not improve the performance significantly.
However, SGC removes the nonlinearities of GCN and essentially becomes a linear classifier in terms of features given by neighbor aggregations, which may limit the possibility of extending to more complex tasks. In general, considering the limitations of the deep GCNs and SGC, one can conclude that: ① the neighbor aggregation is essential for GCNs; ② a simple feature mapping is helpful and sufficient but should be expressive enough. These two points as well as the effectiveness of kernel methods in various machine learning problems [Cortes and Vapnik, 1995; Scholkopf et al., 1998a; Fan and Chow, 2020; Fan et al., 2022; Fan, 2022a] inspire us to establish a general and simple kernel-based framework, termed Graph Convolutional Kernel Machine (GCKM), for graph data. GCKM incorporates implicit feature mapping induced by kernels and neighbor aggregation over graphs, providing a new paradigm for graph-based machine learning. The main contributions of this work are summarized as follows:
- We propose a GCKM framework for graph-based learning. GCKM takes advantages of kernel learning and graph learning. Compared to GCNs, GCKMs have lower computational costs, higher interpretability, stronger theoretical guarantees, and stabler performances.
- We provide a generalization error bound for GCKM-based node classification and prove that graph structure can tighten this bound, both theoretically and empirically.
- We provide a few variants of GCKM that are useful in various graph-based learning problems such as node clustering and node embedding.
- We extend GCKMs to graph-level learning such as graph classification.
Comprehensive experiments demonstrate that the proposed GCKMs are as powerful as the state-of-the-art GCNs and significantly surpass several GCN-based methods on node semi-supervised classification and clustering tasks.
Notations Given an undirected attributed graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$ , where  $\mathcal{V}$  and  $\mathcal{E}$  are the vertex set and edge set, respectively, and  $|\mathcal{V}| = n$  is the number of nodes. The node features are represented as matrix  $\mathbf{X} = [\mathbf{x}_1,\mathbf{x}_2,\dots ,\mathbf{x}_n]^\top \in \mathbb{R}^{n\times m}$ , where  $\mathbf{x}_i\in \mathbb{R}^m$  denotes the column feature vector of node  $i$ . And edges can be described by an adjacency matrix  $\mathbf{A}\in \mathbb{R}^{n\times n}$ , where  $A_{ij} = 1$  if there exists an edge connecting node  $i$  and  $j$ , otherwise  $A_{ij} = 0$ . We use  $\mathbf{A}_i$  to denote the  $i$ -th row of  $\mathbf{A}$ . We denote the self-looped adjacency matrix as  $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$ . The degree matrix corresponding to diagonal  $\tilde{\mathbf{A}}$  is denoted by  $\tilde{\mathbf{D}}\in \mathbb{R}^{n\times n}$ , where  $\tilde{D}_{ii} = \Sigma_{j = 1}^{n}\tilde{A}_{ij}$ .  $\| \cdot \|$  denotes the Euclidean norm of vector. Let  $\mathbf{y}_i\in \{0,1\}^k$  be the label vector of node  $i$  and  $\mathcal{Y} = \{\mathbf{y}_i:\forall i\in \mathcal{L}\}$ , where  $\mathcal{L}$  denotes the index set of labeled nodes. For node classification, the task is to learn a model  $f_{\Theta}(\cdot)$  from  $\mathbf{X}$ ,  $\mathbf{A}$ , and  $\mathcal{Y}$  to classify the unlabeled nodes.
Vanilla and Deep GCNs. Kipf and Welling [2017] proposed GCN which further developed ChebyNet [Defferrard et al., 2016] and formulated an efficient and effective graph convolutional operation. Up to now, numerous studies have tried to explore and interpret GCN, and a widely applied description of GCN decoupled the graph convolution into the following two steps
$$
\bar {\mathbf {H}} ^ {(l + 1)} = \operatorname {A G G R E G A T E} \left(\left\{\mathcal {G}; \mathbf {H} ^ {(l)} \right\}\right) = \hat {\mathbf {A}} \mathbf {H} ^ {(l)},
$$
$$
\mathbf {H} ^ {(l + 1)} = \operatorname {T R A N S F O R M} \left(\bar {\mathbf {H}} ^ {(l + 1)}\right) = \sigma \left(\bar {\mathbf {H}} ^ {(l + 1)} \mathbf {W} ^ {(l)}\right), \tag {1}
$$
where  $\mathbf{H}^{(l)}$  and  $\mathbf{H}^{(l + 1)}$  are the input and output representations of the  $l$ -th layer.  $\hat{\mathbf{A}} = \tilde{\mathbf{D}}^{-\frac{1}{2}}\tilde{\mathbf{A}}\tilde{\mathbf{D}}^{-\frac{1}{2}}$  is the renormalized adjacency matrix and  $\sigma (\cdot)$  is an activation function, e.g.  $\mathrm{ReLU}(\cdot)$ . These two operations compose the layer-wise propagation of GCN, but plenty of research has revealed that constructing a GCN with more such layers would deteriorate the performance. Li et al. [2018] first pointed out the over-smoothing issue is the key point of this phenomenon, that is, repeatedly multiplying the input node features by adjacency matrix eventually makes them more and more similar, resulting in indistinguishable node embeddings. This view has been broadly acknowledged, and many studies have been conducted on how to solve the over-smoothing issue. APPNP [Klicpera et al., 2019] considered the relationship between GCN and PageRank to design a new aggregation scheme, and improve the performance of deep GCN. JKNet [Xu et al., 2018] is another famous model for the same goal, which enabled each node to adaptively gain information from different hops of neighborhood. Nevertheless, different from the well-known deep CNNs, these approaches only alleviate the performance degradation but hardly benefit from the deep layers. More work on deep GCNs can be found in [Chen et al., 2020; Sun et al., 2021; Chen et al., 2023a; Xu et al., 2023].
Simplified GCNs. Although GCN put forward a simple formula of forward propagation rule, its mechanisms have not been fully discussed and may contain redundant computation, which suggests simplified GCNs. Among these, SGC [Wu et al., 2019] removed the nonlinearity of GCN and only retained one learnable weight, so that built a faster model by precomputing the powers of the adjacency matrix. Wu et al. [2019] reported some interesting results that this intuitive way can still achieve the powerful expressive ability of vanilla GCN even with a relatively small solution space. Following this work, some decoupled-style GCNs have been designed, which decouple the two processes in (1) and then only stack the aggregation step while reducing the transformations. For example, Nt and Maehara [2019] claimed that graph structure only provided a denoising approach and GCN and SGC only performed low-pass filtering on node features. They then proposed gfNN to improve SGC from the perspective of graph signal processing. Liu et al. [2020] hypothesized that the coupling of these two operations causes not only more computations but also the over-smoothing issue, and introduced DAGNN by processing graph data via an MLP separately before performing the neighbor aggregation. In light of DAGNN, Dong et al. [2021] further analyzed the mechanism of decoupled GCNs, concluding the effectiveness of this kind of simplifying approaches.
3 Graph Convolutional Kernel Machine
GCN can be decoupled as two key steps: neighbor aggregation and feature transformation, as formulated in (1). The former is a fixed operation and only the feature transformation contains the
Figure 2: Overview of the proposed Graph Convolutional Kernel Machine (GCKM).
learnable weights  $\mathbf{W}^{(l)}$ , and it has been discovered that stacking these two operations significantly damaged the performance. Therefore, some research has been dedicated to solving this problem and making GCN deeper, but to little avail. On the other hand, various simplified GCNs reduced the learnable parameters and are competitive to those deeper GCNs, and SGC even simplified the feature transformation to one weight matrix without activation functions. In other words, despite the larger solution space provided by the nonlinearity and more learnable weights, improperly designed feature transformations may cause significant performance deterioration, while reduction of weights can lead to comparable power. Meanwhile, the representative SGC may be less expressive to cope with more complex data owing to its linear classifier. So it is crucial to present a light model by exploring an effective as well as efficient feature transformation approach.
Neighbor aggregation is commonly regarded as the pivotal component and makes the node representations more compact. Paired with a feature map to low-dimensional space, it may lead to indistinguishable node representations between different classes. Thus, to guarantee discriminative node representations, we propose to map node features to a higher-dimensional space via a transformation operation
$$
\operatorname {T R A N S F O R M} (\mathbf {X}) := \phi (\mathbf {X}) = [ \phi (\mathbf {x} _ {1}), \phi (\mathbf {x} _ {2}), \dots , \phi (\mathbf {x} _ {n}) ] ^ {\top}, \tag {2}
$$
where  $\phi : \mathbb{R}^m \to \mathbb{R}^M$  is a feature mapping and  $M > m$ . Explicitly designing a  $\phi$  is usually costly especially when  $M$  is very large or even infinity. Instead, we can take advantage of kernels that can induce  $\phi$  implicitly. Let  $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$  be a kernel function, we have
$$
k \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \left\langle \phi (\mathbf {x}), \phi \left(\mathbf {x} ^ {\prime}\right) \right\rangle = \phi \left(\mathbf {x}\right) ^ {\top} \phi \left(\mathbf {x} ^ {\prime}\right). \tag {3}
$$
Thus we can obtain a kernel matrix of  $\mathbf{X}$  as
$$
\mathbf {K} = \left[ \begin{array}{c c c} k (\mathbf {x} _ {1}, \mathbf {x} _ {1}) & \dots & k (\mathbf {x} _ {1}, \mathbf {x} _ {n}) \\ \vdots & \ddots & \vdots \\ k (\mathbf {x} _ {n}, \mathbf {x} _ {1}) & \dots & k (\mathbf {x} _ {n}, \mathbf {x} _ {n}) \end{array} \right]. \tag {4}
$$
There are many kernel functions available and each kernel function induces a specific feature map  $\phi$ . For instance, the  $\phi$  induced by a polynomial kernel  $k(\mathbf{x},\mathbf{x}^{\prime}) = (\mathbf{x}^{\top}\mathbf{x}^{\prime} + a)^{b}$  is a  $b$ -order polynomial feature map; the  $\phi$  induced by a Gaussian kernel  $k(\mathbf{x},\mathbf{x}^{\prime}) = \exp \left(-\| \mathbf{x} - \mathbf{x}^{\prime}\|^{2} / (2\sigma^{2})\right)$  is an infinity-order polynomial feature map, namely,  $M = \infty$ .
We can alternatively perform the neighbor aggregation and the transformation (2), and obtain a layer-wise formulation of graph convolutional kernel machine as follows
$$
\bar {\mathbf {H}} ^ {(l + 1)} = \operatorname {A G G R E G A T E} \left(\left\{\mathcal {G}; \mathbf {H} ^ {(l)} \right\}\right) = \hat {\mathbf {A}} ^ {q} \mathbf {H} ^ {(l)}, \tag {5}
$$
$$
\mathbf {H} ^ {(l + 1)} = \operatorname {T R A N S F O R M} \left(\bar {\mathbf {H}} ^ {(l + 1)}\right) = \phi_ {(l)} \left(\bar {\mathbf {H}} ^ {(l + 1)}\right), \tag {5}
$$
where  $\phi_{(l)}$  is the  $l$ -th implicit feature map induced by a kernel  $k^{(l)}$  and the corresponding kernel matrix is  $\mathbf{K}^{(l + 1)} = \mathbf{H}^{(l + 1)}\mathbf{H}^{(l + 1)^{\top}}$ . Here the  $q$ -th power of  $\hat{\mathbf{A}}$  is adopted to allow several times graph convolution. Formulation (5) can be easily extended to multi-layer cases, but  $\mathbf{H}^{(l + 1)}$  is actually implicit and we calculate the kernel matrix  $\mathbf{K}^{(l + 1)}$  directly in practice. It is necessary to discuss
the connection between  $\mathbf{K}^{(l + 1)}$  and  $\mathbf{K}^{(l)}$  and how to derive  $\mathbf{K}^{(l + 1)}$  layer by layer with only kernel matrices. Thus we will take the Gaussian kernel as an example, give the recursive formulation, and show how to derive the corresponding formulation in terms of kernel matrices. With a Gaussian kernel, we have  $K_{ij}^{(l + 1)} = \exp \left(-\frac{\Delta_{ij}^{(l + 1)}}{2\sigma_{(l + 1)}^2}\right)$ ,  $(i,j) \in [n] \times [n]$ , where  $\Delta_{ij}^{(l + 1)} = \left\| \bar{\mathbf{h}}_i^{(l + 1)} - \bar{\mathbf{h}}_j^{(l + 1)} \right\|^2$  is an element from  $\Delta^{(l + 1)}$ , which is a squared distance matrix of  $\bar{\mathbf{H}}^{(l + 1)}$ . By a series of derivations (deferred to Appendix A), we have
$$
\boldsymbol {\Delta} ^ {(l + 1)} = \mathbf {d} _ {\bar {\mathbf {K}} ^ {(l + 1)}} \mathbf {1} _ {n} ^ {\top} + \mathbf {1} _ {n} \mathbf {d} _ {\bar {\mathbf {K}} ^ {(l + 1)}} ^ {\top} - 2 \bar {\mathbf {K}} ^ {(l + 1)}, \tag {6}
$$
where  $\bar{\mathbf{K}}^{(l + 1)} = \hat{\mathbf{A}}^q\mathbf{K}^{(l)}(\hat{\mathbf{A}}^q)^\top$  and  $\mathbf{d}_{\bar{\mathbf{K}}^{l + 1}} = [\bar{K}_{11}^{(l + 1)},\bar{K}_{22}^{(l + 1)},\dots ,\bar{K}_{nn}^{(l + 1)}]^\top$ . Thus the recursive formulation can be summarized as
$$
\left\{ \begin{array}{l} \bar {\mathbf {K}} ^ {(l + 1)} = \hat {\mathbf {A}} ^ {q} \mathbf {K} ^ {(l)} \left(\hat {\mathbf {A}} ^ {q}\right) ^ {\top}, \\ \mathbf {K} ^ {(l + 1)} = \exp \left(- \frac {\mathbf {\Delta} ^ {(l + 1)}}{2 \sigma_ {(l + 1)} ^ {2}}\right). \end{array} \right. \tag {7}
$$
Particularly, we define  $\mathbf{K}^{(0)} = \mathbf{X}\mathbf{X}^{\top}$ . Based on (7), GCKM with Gaussian kernels can be easily generalized to multi-layer cases. It is worth noting that similar compositions also apply to other kernels such as polynomial kernel and Laplace kernel, which will not be detailed here.
After obtaining the kernel matrix  $\mathbf{K}^{(L)}$  from an  $L$ -layer GCKM, we can use Support Vector Machine (SVM) [Cortes and Vapnik, 1995] to perform node classification. Generally, SVM aims to solve
$$
\min  _ {\mathbf {w}, \boldsymbol {\xi}} \frac {1}{2} \| \mathbf {w} \| ^ {2} + \frac {\lambda}{n} \sum_ {i = 1} ^ {n} \xi_ {i} \quad \text {s . t .} y _ {i} \left\langle \varphi (\mathbf {x} _ {i}), \mathbf {w} \right\rangle \geq 1 - \xi_ {i}, \xi_ {i} \geq 0, \forall i = 1, \dots , n, \tag {8}
$$
where  $y_{i}\in \{+1, - 1\}$  denotes the target label of training data  $i$  and  $\xi_{i}$  are the slack variables for the training data, and  $\varphi$  is a feature map. The Lagrangian dual problem is
$$
\max  _ {\mathbf {c}} \sum_ {i = 1} ^ {n} c _ {i} - \frac {1}{2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} c _ {i} c _ {j} y _ {i} y _ {j} \varphi (\mathbf {x} _ {i}) ^ {\top} \varphi (\mathbf {x} _ {j}) \quad \text {s . t .} \sum_ {i = 1} ^ {n} c _ {i} y _ {i} = 0, 0 \leq c _ {i} \leq \frac {\lambda}{n}. \tag {9}
$$
Invoking  $\mathbf{K}^{(L)}$  into (9), where  $\varphi (\mathbf{x}_i) = \phi_{(L)}((\bar{\mathbf{h}}_i^{(L)})^\top)$ , we obtain the following optimization problem of GCKM with  $L$  layers for node classification:
$$
\max  _ {\mathbf {c}} \sum_ {i = 1} ^ {n} c _ {i} - \frac {1}{2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} c _ {i} c _ {j} y _ {i} y _ {j} K _ {i j} ^ {(L)} \quad \text {s . t .} \quad \sum_ {i = 1} ^ {n} c _ {i} y _ {i} = 0, 0 \leq c _ {i} \leq \frac {\lambda}{n}. \tag {10}
$$
For convenience, we call (10) graph convolutional kernel SVM (GCKSVM). Compared to GCN which is nonconvex, GCKSVM is convex and hence we can find its global optimal solution easily. In addition, kernel methods often have higher interpretability than neural networks and stronger generalization guarantees. Compared to SGC which is a linear classifier with respect to the final node representation, GCKSVM is a nonlinear classifier and hence is expected to perform better when the data are not linearly separable. Further justification and discussion will be presented in Figure 3 and its related explanations.
3.2 Generalization Bounds of GCKSVM for Node Classification
We analyze the generalization ability of GCKSVM for node classification. Different from [Bartlett and Shawe-Taylor, 1999; Bartlett and Mendelson, 2002; Gronlund et al., 2020] that focuses on the primary problem of SVM, we focus on the dual problem, otherwise, the graph structure does not explicitly present in the bound.
Let  $\hat{\mathbf{c}}$  be the solution to (10) and denote the corresponding  $\mathbf{w}$  as  $\hat{\mathbf{w}}$ , where we use the Gaussian kernel for GCKSVM. Denote  $S$  be the set of training data drawn from some distribution  $\mathcal{D}$ . Let  $\mathcal{L}_S(\hat{\mathbf{c}},\mathbf{K}^{(L)})\coloneqq \operatorname*{Pr}_{(\varphi (\mathbf{x}),y)\sim S}[y\langle \varphi (\mathbf{x}),\hat{\mathbf{w}}\rangle \leq 1]$  be the training error and  $\mathcal{L}_{\mathcal{D}}(\hat{\mathbf{c}},\mathbf{K}^{(L)})\coloneqq \operatorname*{Pr}_{(\varphi (\mathbf{x}),y)\sim \mathcal{D}}[\mathrm{sign}(\langle \varphi (\mathbf{x}),\hat{\mathbf{w}}\rangle)\neq y] = \operatorname*{Pr}_{(\varphi (\mathbf{x}),y)\sim \mathcal{D}}[y\langle \varphi (\mathbf{x}),\hat{\mathbf{w}}\rangle \leq 0]$  be the expected test error. We have the following generalization error bound.
Theorem 1. Denote the index set of the support vectors as  $\mathcal{V} = \{i:1\leq i\leq n,\hat{c}_i\neq 0\}$ . For any  $0 < \delta < 1$ , it holds with probability at least  $1 - \delta$  over a set of  $n$  samples  $S\sim \mathcal{D}$  that
$$
\mathcal {L} _ {\mathcal {D}} \left(\hat {\mathbf {c}}, \mathbf {K} ^ {(L)}\right) \leq \mathcal {L} _ {S} \left(\hat {\mathbf {c}}, \mathbf {K} ^ {(L)}\right) + O \left(\frac {\eta \ln n + \ln (1 / \delta)}{n} + \sqrt {\frac {\eta \ln n + \ln (1 / \delta)}{n} \cdot \mathcal {L} _ {S} \left(\hat {\mathbf {c}} , \mathbf {K} ^ {(L)}\right)}\right), \tag {11}
$$
where  $\eta = \sum_{i\in \mathcal{V}}\hat{c}_i^2 +\sum_{i\in \mathcal{V}}\sum_{j\in \mathcal{V}\setminus i,y_i = y_j}\hat{c}_i\hat{c}_jK_{ij}^{(L)} - \sum_{i\in \mathcal{V}}\sum_{j\in \mathcal{V},y_i\neq y_j}\hat{c}_i\hat{c}_jK_{ij}^{(L)}\leq \frac{\lambda^2|\mathcal{V}|^2}{n^2}.$
The proof for the theorem is in Appendix B. In the theorem,  $K_{ij}^{(L)} = \exp \left(-\| (\hat{\mathbf{A}}^q)_i\mathbf{H}^{(L - 1)} - (\hat{\mathbf{A}}^q)_j\mathbf{H}^{(L - 1)}\| ^2 /(2\sigma_L^2)\right)$ , which shows the direct connection between the error bound and graph. Empirically (see Appendix F.6), we find that given a fixed  $\lambda$ , the graph convolution reduces the number of support vectors  $|\mathcal{V}|$  and has minor influence on the training error  $\mathcal{L}_S(\hat{\mathbf{c}},\mathbf{K}^{(L)})$  which eventually reduces the upper bound of test error. The fundamental reason is that incorporating the graph structure significantly improved the quality of the kernel matrix  $\mathbf{K}^{(L)}$ , in which the overall within-class similarity becomes much larger than the between-class similarity. In other words, the kernel matrix becomes more discriminative, which can be verified by Figure 4.
Note that the number of support vectors  $|\mathcal{V}|$  in SVM is data-dependent and in order to theoretically show the influence of graph convolution on  $|\mathcal{V}|$ , we have to make the following assumption: