Title: Graph Convolutional Kernel Machine versus Graph Convolutional Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: graph neural network

kernel method
Abstract: Graph convolutional networks (GCN) with one or two hidden layers have been widely used in handling graph data that are prevalent in various disciplines. Many studies showed that the gain of making GCNs deeper is tiny or even negative. This implies that the complexity of graph data is often limited and shallow models are often sufficient to extract expressive features for various tasks such as node classification. Therefore, in this work, we present a framework called graph convolutional kernel machine (GCKM) for graph-based machine learning. GCKMs are built upon kernel functions integrated with graph convolution. An example is the graph convolutional kernel support vector machine (GCKSVM) for node classification, for which we analyze the generalization error bound and discuss the impact of the graph structure. Compared to GCNs, GCKMs require much less effort in architecture design, hyperparameter tuning, and optimization. More importantly, GCKMs are guaranteed to obtain globally optimal solutions and have strong generalization ability and high interpretability. GCKMs are composable, can be extended to large-scale data, and are applicable to various tasks (e.g., node or graph classification, clustering, feature extraction, dimensionality reduction). The numerical results on benchmark datasets show that, besides the aforementioned advantages, GCKMs have at least competitive accuracy compared to GCNs.

Graph Convolutional Kernel Machine versus Graph Convolutional Networks

Abstract

Graph convolutional networks (GCN) with one or two hidden layers have been widely used in handling graph data that are prevalent in various disciplines. Many studies showed that the gain of making GCNs deeper is tiny or even negative. This implies that the complexity of graph data is often limited and shallow models are often sufficient to extract expressive features for various tasks such as node classification. Therefore, in this work, we present a framework called graph convolutional kernel machine (GCKM) 1 for graph-based machine learning. GCKMs are built upon kernel functions integrated with graph convolution. An example is the graph convolutional kernel support vector machine (GCKSVM) for node classification, for which we analyze the generalization error bound and discuss the impact of the graph structure. Compared to GCNs, GCKMs require much less effort in architecture design, hyperparameter tuning, and optimization. More importantly, GCKMs are guaranteed to obtain globally optimal solutions and have strong generalization ability and high interpretability. GCKMs are composable, can be extended to large-scale data, and are applicable to various tasks (e.g., node or graph classification, clustering, feature extraction, dimensionality reduction). The numerical results on benchmark datasets show that, besides the aforementioned advantages, GCKMs have at least competitive accuracy compared to GCNs.

1 Introduction

Graph data are prevalent in science and engineering. In recent years, neural networks and deep learning methods have shown promising performance in handling various graph data. Graph Neural Networks (GNNs) have been developed into various basic frameworks, including famous GCN [Kipf and Welling, 2017], GIN [Xu et al. , 2019], GraphSAGE [Hamilton et al. , 2017], GAT [Velickovic et al. , 2018], etc. As one of the most attractive GNN paradigms, GCN, the graph convolutional network [Kipf and Welling, 2017], has gained widespread attention since it was proposed. Many widely developed domains like machine learning [Chen et al. , 2023b; Cai et al. , 2022; Fan et al. , 2022; Fan, 2022b], data mining [Chang et al. , 2021; Fan, 2021; Wu et al. , 2022], computer vision [Guan et al. , 2022; Cai and Fan, 2022; Wu et al. , 2023], etc., have also made extensive use of numerous GCN variants. Despite the better performance compared to classical graph methods (e.g., DeepWalk [Perozzi et al. , 2014], LINE [Tang et al. , 2015] and node2vector [Grover and Leskovec, 2016]), some recent studies have pointed out a few drawbacks of GCNs [Wang et al. , 2020; Bo et al. , 2021; Wang et al. , 2023]. Perhaps the failure of deep GCNs is the most notorious one that severely restricted the further development of GCNs.

Containing both the self features and pair-wise relations of entities, graph is viewed as a more complex non-Euclidean data type, so there is a natural and wide desire to construct deeper GCNs. Unfortunately, the performance of GCN decreases when stacking more graph convolutional layers, which is defined as the so-called over-smoothing issue [Li et al. , 2018]. For instance, even a 7-layer GCN will inevitably generate indistinguishable node embeddings and lead to over 30% accuracy degradation [Liu et al. , 2020]. Recently, some methods modifying the basic structure of GCN have been proposed to handle the over-smoothing issue and make GCN deeper [Xu et al. , 2018; Klicpera et al. , 2019; Yang et al. , 2022]. Indeed, most of them have successfully alleviated the over-smoothing issue. But it seems counter-intuitive that these elaborate deep GCNs, which stack dozens or even hundreds of layers, can only laboriously maintain the performance that a 2-layer vanilla GCN achieves easily. We apply some GCNs to a node classification experiment on Cora following the settings of

the original GCN, as shown in Figure 1. APPNP [Klicpera et al. , 2019] and JKNet [Xu et al. , 2018] are two well-known models that attempted to pass information from shallow layers to the final output but benefited very limited from the deep neural networks. These explorations raise a question:

Do we really need deep GCNs in common graph-based tasks?

Actually, there have also been some efforts paid to simplifying GCN, showing comparative effectiveness to vanilla GCN, like SGC [Wu et al. , 2019], which removed nonlinearity and collapsed weight matrices to accelerate GCN. Impressively, it gained equal or even better performance on the benchmark graph datasets, which can partially be observed in Figure 1. Inspired by the success of SGC, some research moved further to decouple the graph convolution by individually performing an MLP (usually 2-layer) and several (can be hundreds) neighbor aggregations, called decoupled-style GCN [Nt and Maehara, 2019; Wang et al. , 2021]. All these works revealed that simple GCNs can achieve encouraging performance on commonly used graph datasets such as Cora, Citeseer, and Pubmed [Sen et al. , 2008]. In these datasets, the numerical node features and the adjacency matrices are already discriminative, which means further feature extractions via neural networks do not improve the performance significantly.

However, SGC removes the nonlinearities of GCN and essentially becomes a linear classifier in terms of features given by neighbor aggregations, which may limit the possibility of extending to more complex tasks. In general, considering the limitations of the deep GCNs and SGC, one can conclude that: 1 ⃝ the neighbor aggregation is essential for GCNs; 2 ⃝ a simple feature mapping is helpful and sufficient but should be expressive enough. These two points as well as the effectiveness of kernel methods in various machine learning problems [Cortes and Vapnik, 1995; Schölkopf et al. , 1998a; Fan and Chow, 2020; Fan et al. , 2022; Fan, 2022a] inspire us to establish a general and simple kernel-based framework, termed Graph Convolutional Kernel Machine (GCKM), for graph data. GCKM incorporates implicit feature mapping induced by kernels and neighbor aggregation over graphs, providing a new paradigm for graph-based machine learning. The main contributions of this work are summarized as follows:

We propose a GCKM framework for graph-based learning. GCKM takes advantages of kernel learning and graph learning. Compared to GCNs, GCKMs have lower computational costs, higher interpretability, stronger theoretical guarantees, and stabler performances.

We provide a generalization error bound for GCKM-based node classification and prove that graph structure can tighten this bound, both theoretically and empirically.

Weprovide a few variants of GCKM that are useful in various graph-based learning problems such as node clustering and node embedding.

We extend GCKMs to graph-level learning such as graph classification.

Comprehensive experiments demonstrate that the proposed GCKMs are as powerful as the stateof-the-art GCNs and significantly surpass several GCN-based methods on node semi-supervised classification and clustering tasks.

Notations Given an undirected attributed graph G = ( V , E ) , where V and E are the vertex set and edge set, respectively, and |V| = n is the number of nodes. The node features are represented as matrix X = [ x 1 , x 2 , . . . , x n ] ⊤ ∈ R n × m , where x i ∈ R m denotes the column feature vector of node i . And edges can be described by an adjacency matrix A ∈ R n × n , where A ij = 1 if there exists an edge connecting node i and j , otherwise A ij = 0 . We use A i to denote the i -th row of A . We denote the self-looped adjacency matrix as ˜ A = A + I . The degree matrix corresponding to diagonal ˜ A is denoted by ˜ D ∈ R n × n , where ˜ D ii = Σ n j =1 ˜ A ij . ∥ · ∥ denotes the Euclidean norm of vector. Let y i ∈ { 0 , 1 } k be the label vector of node i and Y = { y i : ∀ i ∈ L} , where L denotes the index set of labeled nodes. For node classification, the task is to learn a model f Θ ( · ) from X , A , and Y to classify the unlabeled nodes.

2 Related Work

Vanilla and Deep GCNs. Kipf and Welling [2017] proposed GCN which further developed ChebyNet [Defferrard et al. , 2016] and formulated an efficient and effective graph convolutional operation. Up to now, numerous studies have tried to explore and interpret GCN, and a widely applied description of GCN decoupled the graph convolution into the following two steps

where H ( l ) and H ( l +1) are the input and output representations of the l -th layer. ˆ A = ˜ D -1 2 ˜ A ˜ D -1 2 is the renormalized adjacency matrix and σ ( · ) is an activation function, e.g. ReLU( · ) . These two operations compose the layer-wise propagation of GCN, but plenty of research has revealed that constructing a GCN with more such layers would deteriorate the performance. Li et al. [2018] first pointed out the over-smoothing issue is the key point of this phenomenon, that is, repeatedly multiplying the input node features by adjacency matrix eventually makes them more and more similar, resulting in indistinguishable node embeddings. This view has been broadly acknowledged, and many studies have been conducted on how to solve the over-smoothing issue. APPNP [Klicpera et al. , 2019] considered the relationship between GCN and PageRank to design a new aggregation scheme, and improve the performance of deep GCN. JKNet [Xu et al. , 2018] is another famous model for the same goal, which enabled each node to adaptively gain information from different hops of neighborhood. Nevertheless, different from the well-known deep CNNs, these approaches only alleviate the performance degradation but hardly benefit from the deep layers. More work on deep GCNs can be found in [Chen et al. , 2020; Sun et al. , 2021; Chen et al. , 2023a; Xu et al. , 2023].

Simplified GCNs. Although GCN put forward a simple formula of forward propagation rule, its mechanisms have not been fully discussed and may contain redundant computation, which suggests simplified GCNs. Among these, SGC [Wu et al. , 2019] removed the nonlinearity of GCN and only retained one learnable weight, so that built a faster model by precomputing the powers of the adjacency matrix. Wu et al. [2019] reported some interesting results that this intuitive way can still achieve the powerful expressive ability of vanilla GCN even with a relatively small solution space. Following this work, some decoupled-style GCNs have been designed, which decouple the two processes in (1) and then only stack the aggregation step while reducing the transformations. For example, Nt and Maehara [2019] claimed that graph structure only provided a denoising approach and GCN and SGC only performed low-pass filtering on node features. They then proposed gfNN to improve SGC from the perspective of graph signal processing. Liu et al. [2020] hypothesized that the coupling of these two operations causes not only more computations but also the over-smoothing issue, and introduced DAGNN by processing graph data via an MLP separately before performing the neighbor aggregation. In light of DAGNN, Dong et al. [2021] further analyzed the mechanism of decoupled GCNs, concluding the effectiveness of this kind of simplifying approaches.

3 Graph Convolutional Kernel Machine

3.1 General Framework

GCN can be decoupled as two key steps: neighbor aggregation and feature transformation, as formulated in (1). The former is a fixed operation and only the feature transformation contains the

…

Neighbor Aggregation

Kernel Machine

…

Feature Transformation

Downstream Tasks

Graph-level Classification

Graph-level Clustering

…

Graph Convolutional Kernel Layers

…

…

…

…

…

SVM

Spectral

Clustering

Explicit Embedding

not linearly

separable

linearly

separable

Node

Features

𝐗

Adjacency

Matrix

𝐀

Interfaces

or

learnable weights W ( l ) , and it has been discovered that stacking these two operations significantly damaged the performance. Therefore, some research has been dedicated to solving this problem and making GCN deeper, but to little avail. On the other hand, various simplified GCNs reduced the learnable parameters and are competitive to those deeper GCNs, and SGC even simplified the feature transformation to one weight matrix without activation functions. In other words, despite the larger solution space provided by the nonlinearity and more learnable weights, improperly designed feature transformations may cause significant performance deterioration, while reduction of weights can lead to comparable power. Meanwhile, the representative SGC may be less expressive to cope with more complex data owing to its linear classifier. So it is crucial to present a light model by exploring an effective as well as efficient feature transformation approach.

Neighbor aggregation is commonly regarded as the pivotal component and makes the node representations more compact. Paired with a feature map to low-dimensional space, it may lead to indistinguishable node representations between different classes. Thus, to guarantee discriminative node representations, we propose to map node features to a higher-dimensional space via a transformation operation

where ϕ : R m → R M is a feature mapping and M > m . Explicitly designing a ϕ is usually costly especially when M is very large or even infinity. Instead, we can take advantage of kernels that can induce ϕ implicitly. Let k : X × X → R be a kernel function, we have

Thus we can obtain a kernel matrix of X as

There are many kernel functions available and each kernel function induces a specific feature map ϕ . For instance, the ϕ induced by a polynomial kernel k ( x , x ′ ) = ( x ⊤ x ′ + a ) b is a b -order polynomial feature map; the ϕ induced by a Gaussian kernel k ( x , x ′ ) = exp ( -∥ x -x ′ ∥ 2 / (2 σ 2 ) ) is an infinityorder polynomial feature map, namely, M = ∞ .

We can alternatively perform the neighbor aggregation and the transformation (2), and obtain a layer-wise formulation of graph convolutional kernel machine as follows

where ϕ ( l ) is the l -th implicit feature map induced by a kernel k ( l ) and the corresponding kernel matrix is K ( l +1) = H ( l +1) H ( l +1) ⊤ . Here the q -th power of ˆ A is adopted to allow several times graph convolution. Formulation (5) can be easily extended to multi-layer cases, but H ( l +1) is actually implicit and we calculate the kernel matrix K ( l +1) directly in practice. It is necessary to discuss

the connection between K ( l +1) and K ( l ) and how to derive K ( l +1) layer by layer with only kernel matrices. Thus we will take the Gaussian kernel as an example, give the recursive formulation, and show how to derive the corresponding formulation in terms of kernel matrices. With a Gaussian kernel, we have K ( l +1) ij = exp ( -∆ ( l +1) ij 2 σ 2 ( l +1) ) , ( i, j ) ∈ [ n ] × [ n ] , where ∆ ( l +1) ij = ∥ ∥ ¯ h ( l +1) i -¯ h ( l +1) j ∥ ∥ 2 is an element from ∆ ( l +1) , which is a squared distance matrix of ¯ H ( l +1) . By a series of derivations (deferred to Appendix A), we have

where ¯ K ( l +1) = ˆ A q K ( l ) ( ˆ A q ) ⊤ and d ¯ K l +1 = [ ¯ K ( l +1) 11 , ¯ K ( l +1) 22 , . . . , ¯ K ( l +1) nn ] ⊤ . Thus the recursive formulation can be summarized as

Particularly, we define K (0) = XX ⊤ . Based on (7), GCKM with Gaussian kernels can be easily generalized to multi-layer cases. It is worth noting that similar compositions also apply to other kernels such as polynomial kernel and Laplace kernel, which will not be detailed here.

After obtaining the kernel matrix K ( L ) from an L -layer GCKM, we can use Support Vector Machine (SVM) [Cortes and Vapnik, 1995] to perform node classification. Generally, SVM aims to solve

where y i ∈ { +1 , -1 } denotes the target label of training data i and ξ i are the slack variables for the training data, and φ is a feature map. The Lagrangian dual problem is

Invoking K ( L ) into (9), where φ ( x i ) = ϕ ( L ) (( ¯ h ( L ) i ) ⊤ ) , we obtain the following optimization problem of GCKM with L layers for node classification:

For convenience, we call (10) graph convolutional kernel SVM (GCKSVM). Compared to GCN which is nonconvex, GCKSVM is convex and hence we can find its global optimal solution easily. In addition, kernel methods often have higher interpretability than neural networks and stronger generalization guarantees. Compared to SGC which is a linear classifier with respect to the final node representation, GCKSVM is a nonlinear classifier and hence is expected to perform better when the data are not linearly separable. Further justification and discussion will be presented in Figure 3 and its related explanations.

3.2 Generalization Bounds of GCKSVM for Node Classification

We analyze the generalization ability of GCKSVM for node classification. Different from [Bartlett and Shawe-Taylor, 1999; Bartlett and Mendelson, 2002; Grønlund et al. , 2020] that focuses on the primary problem of SVM, we focus on the dual problem, otherwise, the graph structure does not explicitly present in the bound.

̸

Let ˆ c be the solution to (10) and denote the corresponding w as ˆ w , where we use the Gaussian kernel for GCKSVM. Denote S be the set of training data drawn from some distribution D . Let L S (ˆ c , K ( L ) ) := Pr ( φ ( x ) ,y ) ∼ S [ y ⟨ φ ( x ) , ˆ w ⟩ ≤ 1] be the training error and L D (ˆ c , K ( L ) ) := Pr ( φ ( x ) ,y ) ∼D [sign( ⟨ φ ( x ) , ˆ w ⟩ ) = y ] =Pr ( φ ( x ) ,y ) ∼D [ y ⟨ φ ( x ) , ˆ w ⟩ ≤ 0] be the expected test error. We have the following generalization error bound.

̸

Theorem 1. Denote the index set of the support vectors as V = { i : 1 ≤ i ≤ n, ˆ c i = 0 } . For any 0 < δ < 1 , it holds with probability at least 1 -δ over a set of n samples S ∼ D that

̸

The proof for the theorem is in Appendix B. In the theorem, K ( L ) ij = exp ( -∥ ( ˆ A q ) i H ( L -1) -( ˆ A q ) j H ( L -1) ∥ 2 / (2 σ 2 L ) ) , which shows the direct connection between the error bound and graph. Empirically (see Appendix F.6), we find that given a fixed λ , the graph convolution reduces the number of support vectors |V| and has minor influence on the training error L S (ˆ c , K ( L ) ) , which eventually reduces the upper bound of test error. The fundamental reason is that incorporating the graph structure significantly improved the quality of the kernel matrix K ( L ) , in which the overall within-class similarity becomes much larger than the between-class similarity. In other words, the kernel matrix becomes more discriminative, which can be verified by Figure 4.

Note that the number of support vectors |V| in SVM is data-dependent and in order to theoretically show the influence of graph convolution on |V| , we have to make the following assumption:

Assumption 1. The aggregation step with graph G increases the inner product between the kernel feature maps of samples in the same class and reduces or does not change the inner product between the kernel feature maps of samples in different classes.

This is a reasonable assumption because a useful graph should make the samples from different classes more distinguishable or at least make the samples from the same class more similar.

Theorem 2. Given a graph G that satisfies Assumption 1, let φ and φ G be the kernel feature maps without and with aggregation on graph G respectively. The corresponding negative Lagrangian dual objectives (to minimize) are denoted as L ( c ) := 1 ∑ n ∑ n c c q -∑ n c ,

2 i =1 j =1 i j ij i =1 i where q ij = y i y j φ ( x i ) ⊤ φ ( x j ) and L G ( c ) := 1 2 ∑ n i =1 ∑ n j =1 c i c j q G ij -∑ n i =1 c i , where q G ij = y i y j φ G ( x i ) ⊤ φ G ( x j ) . We have

where R ( c ) ≥ 0 is a regularization term inducing sparsity in c .

The proof as well as the details about R ( c ) are in Appendix C. According to Theorem 2, the aggregation step introduces an additional sparse regularization term R ( c ) , which will make c sparser, or in other words, reduce the number of support vectors |V| . Considering both the theoretical and empirical results, we conclude that graph convolution leads to a tighter generalization error bound, which verifies the effectiveness of our GCKSVM. By the way, the interpretability of GCKSVM is higher than that of GCN, owing to the support vectors.

3.3 Extensions

Besides SVM, our method can be generalized to many other machine learning problems. We show a few examples of node-level learning in the following context.

GCKMfor Spectral Clustering (GCKSC) For node clustering, it is natural to perform spectral clustering [Ng et al. , 2001; Von Luxburg, 2007] using the kernel matrix K ( L ) . Instead of using the generally dense matrix K ( L ) , we retain only the largest values of each column to obtain a sparse matrix ˆ K ( L ) , which often yields better clustering performance.

GCKMfor Principal Component Analysis (GCKPCA) It is straightforward to conduct Kernel PCA [Schölkopf et al. , 1998b] using K ( L ) , where the principal components can be used as the representations of the nodes of G . Please refer to Appendix F.5 for experiments.

Wecan also adapt GCKM to other learning models such as self-expressive models [Elhamifar and Vidal, 2013], Fisher linear discriminant analysis [Fisher, 1936], canonical correlation analysis [Hardoon et al. , 2004], etc.

Now we extend GCKM to graph-level learning. Suppose we have a set of graphs G = {G 1 , G 2 , . . . , G N } , where | V g | = n g , g = 1 , 2 , . . . , N . For each G g , we use (7) to obtain K ( L ) g as well as the implicit node-level representations H ( L ) g . For each G g , we compute the implicit graph-level representation as the summation of node-level representations, i.e.,

Then for any two graphs G g , G g ′ , we have

where K ( L ) gg ′ ∈ R n g × n g ′ can be computed using (5) recursively. Consequently, we can obtain a graph-level kernel matrix as

Using (14) to compute K ( L ) G has a high time complexity of N ( N -1) / 2 and using (15) has a high space complexity of O (( ∑ N g =1 n g ) 2 ) . To reduce the cost, we can perform (15) on mini-batchs of G . We have the following result (proved in Appendix D)

Proposition 1. Suppose k ( L ) is a Gaussian kernel with 0 < σ L < ∞ . Then K ( L ) G is positive definite.

K ( L ) G can be applied to various graph-level learning tasks such as graph classification and clustering.

3.4 Fast and Explicit Feature Transformation for GCKM

GCKMs can be stacked as multi-layer model and applied to a wide range of tasks by calculating the kernel matrix, and has the advantages of being fast, explainable, and requiring few parameters. Despite this, the time and space complexities of GCKM are O ( n 2 ) , which limits the application to very large graph datasets. To tackle this issue, we consider approximating the kernel by

where ψ ( · ) is a low-dimensional map using Random Fourier Feature [Rahimi and Recht, 2007]. The following theorem provides the key foundation of this approximation method:

Theorem 3 (Bochner's theorem [Rudin, 1994]) . A continuous kernel k ( x , x ′ ) = k ( x -x ′ ) on R m is positive definite if and only if k ( δ ) is the Fourier transform of a non-negative measure.

Specifically, for Gaussian kernel, let p RBF ( ω ) be a probability distribution and also the Fourier transforamtion of k RBF ( δ ) . We first randomly sample { ω ( l ) 1 , ω ( l ) 2 , . . . , ω ( l ) D } from the probability distribution p RBF ( ω ) = N ( 0 , 1 σ I ) . Subsequently the multi-layer GCKM with explicit feature transformation, called GCKM-E, on the i -th sample is defined as z ( l +1) i = ψ ( l ) ( ¯ z ( l +1) i ) where

and z ( l +1) i and ¯ z ( l +1) i denote i -th columns vectors drawn from the transposes of Z ( l +1) and ¯ Z ( l +1) = ˆ A q Z ( l ) , respectively. For detailed derivations, please refer to Appendix E. This approach allows GCKM to flexibly choose the dimensions of explicit node representations when n is very large.

4 Experiment

In this section, we evaluate GCKMs on several real-world graph datasets. Comparison experiments of both semi-supervised node classification and node clustering show GCKMs' accuracy and efficiency. Due to the space limitation, some important experimental results are deferred to Appendix F.2 (more types of datasets and large-scale dataset), Appendix F.3 (over-smoothing problem), Appendix F.4 (GCKM with various kernels) and Appendix F.5 (GCKPCA visualization).

Datasets We employ three most widely adopted citation networks Cora, Citeseer, and Pubmed for evaluations, they are formed as unweighted and undirected graphs where each node represents a paper and edges denote citations between papers. As for graph classification, IMDB-BINARY and IMDB-MULTI are movie collaboration datasets; COLLAB is a scientific collaboration dataset; MUTAG, PROTEINS, and PTC are three bioinformatics datasets. Details about these datasets and more datasets (e.g., social networks, paper networks and OGB-Arxiv) can be found in Appendix F.1.

Compared Methods For node classification, two types of GNNs are selected: Chebyshev [Defferrard et al. , 2016], GraphSAGE [Hamilton et al. , 2017], GAT [Velickovic et al. , 2018], GCN [Kipf and Welling, 2017] and SGC [Wu et al. , 2019] serve as classical baseline GNNs; APPNP [Klicpera et al. , 2019], JKNet [Xu et al. , 2018], DAGNN [Liu et al. , 2020], AdaGCN [Sun et al. , 2021], AMGCN[Wang et al. , 2020], DefGCN [Park et al. , 2022] are state-of-the-art models, in particular the former four ones are Deep GCNs. For node clustering, K-means and Spectral Clustering are two traditional baseline methods and GAE and VGAE [Kipf and Welling, 2016] are GCN-based baseline methods; ARGA, ARVGA [Pan et al. , 2018], DGI [Velickovic et al. , 2019], MVGRL [Hassani and Ahmadi, 2020], GALA [Park et al. , 2019], DFCN [Tu et al. , 2021] and S 3 GC [Devvrit et al. , 2022] are all state-of-the-art GNN-based approaches. For graph classification, WL subtree kernel [Shervashidze et al. , 2011], AWL [Ivanov and Burnaev, 2018] are two baselines; DCNN [Atwood and Towsley, 2016], PATCHY-SAN [Niepert et al. , 2016] and DGCNN [Zhang et al. , 2018] are three deep learning methods; GCN, GraphSAGE and GIN [Xu et al. , 2019] are GNN-based methods.

Experimental Settings All the above methods are set as default following the original paper. We use a 2-layer GCKM for the experiments and the kernel is specified as Gaussian kernel, more detailed settings of GCKM can be found in Appendix F.1. In node classification, following vanilla GCN [Kipf and Welling, 2017], the nodes are split into three set: train set containing 20 samples per class, validation set and test set with 500 and 1 , 000 samples respectively, and the standard fixed split is same to [Yang et al. , 2016]. GCKM paired with kernel-based Spectral Clustering is leveraged for node clustering, which is described in Section 3.3. For graph classification task, we adopt the 10-fold cross validation following the settings of GIN Xu et al. [2019].

Semi-supervised Node Classification The results are reported in Table 1, where GCKSVM-E denotes GCKSVM with explicit features. Note that, we tune the hyperparameters of GCKSVM on the validation set and report the best performance. We have the following observations. 1) In the case of standard split, GCKSVM achieves decent performance and beats several GNNs including deep GCNs, even though it is a shallow model. Besides, owing to that GCKSVM can obtain globally optimal solutions, the standard deviations are zero. 2) In the case of random split, GCKSVM outperforms all GNNs on the three datasets and has the smallest standard deviations, showing higher stability. 3) Deep GCNs are not substantially better than, or even worse than baselines, which further validates that deepening is not essential in common graph-based tasks. 4) GCKSVM-E performs comparably to GCKSVM and other SOTA methods.

Decision Boundary To further explore the expressiveness, we illustrate the decision boundaries of SGC, APPNP, GCN, and the proposed GCKSVM in Figure 3. We generate a synthetic two-circle

0.8

0.6

0.4

0.2

0.0

-0.2

-0.4

-0.6

-0.8

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

0.32

0.24

0.16

0.08

0.00

-0.08

-0.16

-0.24

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

1.00

0.75

0.50

0.25

0.00

-0.25

-0.50

-0.75

-1.00

SGC.

APPNP

口

GCN

GCKSVM.

ACC:62.6

ACC:89.8

ACC:89.9

ACC:90.5

SGC

APPNP:

GCN

GCKSVM.

ACC:49.7

ACC:67.8

ACC:68.7

ACC:70.4

SGC

APPNP

GCN

GCKSVM

07-1210

.

00号

ACC:79.6

ACC:87.5

ACC:87.5

ACC:90.0

dataset named Circle. We also consider two real datasets, denoted by Cora-2D and Chameleon-2D, where we select two classes of samples from the original datasets and map them to 2-D space using tSNE [Van der Maaten and Hinton, 2008]. We randomly labeled 200 samples for each dataset and constructed adjacency matrices by k -nearest neighbor algorithm, where the original graph structures are not used here. The details are described in Appendix F.1. Cora-2D is the simplest case where the decision boundary is easy to construct, Circle is with a clear but non-linear decision boundary, and Chameleon-2D is the most complex one.

We have the following conclusions. 1) Consistent with our previous discussion, the nonlinearity provided by SGC is low even if we increase the power of ˆ A to a large value such as 50. Therefore, SGC performs poorly on Circle and Chameleon-2D, where its decision boundaries are approximately linear. This is unexpected considering its performance in the previous experiment. An explanation is that these commonly used graph datasets can be well classified linearly in high-dimension spaces, rather than these 2D spaces. 2) The deep model APPNP fits these datasets with similar boundaries as GCN, either simple or complex, revealing that existing deep GCNs may not provide more powerful expressiveness. 3) GCKSVM gains advantages over the three other models, whose decision boundaries fit these datasets well, attributed to which GCKM explicitly maps the samples to highdimensional space to achieve better separability.

Graph Classification The results of graph classification tasks are recorded in Table 2. Compared to both classical baselines and GNNs, especially GIN designed for Graph-level tasks, GCKSVM shows

0.010

0.008

0.006

0.004

0.002

0.000

1.00

0.80

0.60

0.40

0.20

0.00

1.00

0.80

0.60

0.40

0.20

0.00

competitive power in graph classification. It is worth noting that GCKSVM outperforms the GNNs in the two relatively large datasets IMDB-M and COLLAB.

Node Clustering clustering methods is shown in Table 3.

00

Chebyshev

JKNet

GraphSAGE

DAGNN

GAT

AdaGCN

GCN

AMGCN

SGC

DefGCN

00

APPNP

GCKSVM

50

The comparison of all node Similar to the results of node classification, GCKM is well extended to node clustering tasks and is competitive with the deep-learning-based models. It is impressive that the vanilla GCN variants for clustering, GAE and VGAE both perform much worse than our GCKSC, demonstrating that GCKM can yield a discriminative kernel matrix. This can also be observed in Figure 4, where the powered adjacency matrix, the kernel matrix calculated on node features, and the kernel matrix of GCKM are visualized.

Time Costs Figure 5 shows the runtime of all methods on Pubmed. GCKSVM is much more efficient than all other methods except SGC.

5 Conclusions

After analyzing the limitations of GCN, SGC and deep GCNs, we proposed a framework of GCKM. GCKM integrated graph convolution with kernel learning and exhibited higher efficiency, interpretability, and stability. We extended GCKM to many graph-based machine-learning tasks, like node-level and graph-level classification and clustering. Experiments showed that GCKMs are, at least, as accurate as GCN and other SOTA GNNs. More importantly, compared to the representative simplified model SGC, GCKM is much more effective in handling non-linear data. One possible limitation of this work is that we did not systematically test other kernel functions, though we have found that the Gaussian kernel is better than the polynomial for GCKM.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China under Grants No.62106211 and No.62072151, the General Program JCYJ20210324130208022 of Shenzhen Fundamental Research, the research funding T00120210002 of Shenzhen Research Institute of Big Data, the Guangdong Key Lab of Mathematical Foundations for Artificial Intelligence, Anhui Provincial Natural Science Fund for the Distinguished Young Scholars (2008085J30), Open Foundation of Yunnan Key Laboratory of Software Engineering (2023SE103), CCF-Baidu Open Fund and CAAIHuawei MindSpore Open Fund, and the funding UDF01001770 of The Chinese University of Hong Kong, Shenzhen.

References

James Atwood and Don Towsley. Diffusion-convolutional neural networks. In Advances in Neural Information Processing Systems , pages 1993-2001, 2016.

Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3(Nov):463-482, 2002.

Peter Bartlett and John Shawe-Taylor. Generalization performance of support vector machines and other pattern classifiers. Advances in Kernel methods-support vector learning , pages 43-54, 1999.

Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In Proceedings of the 35th AAAI Conference on Artificial Intelligence , pages 3950-3957, 2021.

Jinyu Cai and Jicong Fan. Perturbation learning based anomaly detection. In Advances in Neural Information Processing Systems , 2022.

Jinyu Cai, Jicong Fan, Wenzhong Guo, Shiping Wang, Yunhe Zhang, and Zhao Zhang. Efficient deep embedded subspace clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 21-30. IEEE, 2022.

Heng Chang, Yu Rong, Tingyang Xu, Yatao Bian, Shiji Zhou, Xin Wang, Junzhou Huang, and Wenwu Zhu. Not all low-pass filters are robust in graph convolutional networks. In Advances in Neural Information Processing Systems , pages 25058-25071, 2021.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learning , pages 1725-1735, 2020.

Zhaoliang Chen, Zhihao Wu, Zhenghong Lin, Shiping Wang, Claudia Plant, and Wenzhong Guo. Agnn: Alternating graph-regularized neural networks to alleviate over-smoothing. IEEE Transactions on Neural Networks and Learning Systems , pages 1-13, 2023.

Zhaoliang Chen, Zhihao Wu, Shiping Wang, and Wenzhong Guo. Dual low-rank graph autoencoder for semantic and topological networks. In Proceedings of the 37th AAAI Conference on Artificial Intelligence , pages 4191-4198, 2023.

Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. In Advances in Neural Information Processing Systems , pages 9936-9949, 2021.

Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning , 20:273-297, 1995.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems , pages 3837-3845, 2016.

Fnu Devvrit, Aditya Sinha, Inderjit S. Dhillon, and Prateek Jain. S3GC: scalable self-supervised graph clustering. In Advances in Neural Information Processing Systems , 2022.

Hande Dong, Jiawei Chen, Fuli Feng, Xiangnan He, Shuxian Bi, Zhaolin Ding, and Peng Cui. On the equivalence of decoupled graph convolution network and label propagation. In Proceedings of the Web Conference , pages 3651-3662, 2021.

Ehsan Elhamifar and René Vidal. Sparse subspace clustering: Algorithm, theory, and applications. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(11):2765-2781, 2013.

Jicong Fan and Tommy W. S. Chow. Exactly robust kernel principal component analysis. IEEE Transactions on Neural Networks and Learning Systems , 31(3):749-761, 2020.

Jicong Fan, Yiheng Tu, Zhao Zhang, Mingbo Zhao, and Haijun Zhang. A simple approach to automated spectral clustering. In Advances in Neural Information Processing Systems , pages 9907-9921, 2022.

Jicong Fan. Large-scale subspace clustering via k-factorization. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 342-352, 2021.

Jicong Fan. Dynamic nonlinear matrix completion for time-varying data imputation. Proceedings of the AAAI Conference on Artificial Intelligence , 36(6):6587-6596, Jun. 2022.

Jicong Fan. Multi-mode deep matrix and tensor factorization. In Proceedings of the 10th International Conference on Learning Representations , 2022.

Ronald A Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics , 7(2):179-188, 1936.

Allan Grønlund, Lior Kamma, and Kasper Green Larsen. Near-tight margin-based generalization bounds for support vector machines. In Proceedings of the 37th International Conference on Machine Learning , pages 3779-3788, 2020.

Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 855-864, 2016.

Yonghang Guan, Jun Zhang, Kuan Tian, Sen Yang, Pei Dong, Jinxi Xiang, Wei Yang, Junzhou Huang, Yuyao Zhang, and Xiao Han. Node-aligned graph convolutional network for whole-slide image representation and classification. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18791-18801, 2022.

Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspective on oversmoothing and beyond. In Proceedings of the 11th International Conference on Learning Representations , 2023.

William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems , pages 1024-1034, 2017.

David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Computation , 16(12):2639-2664, 2004.

Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. In Proceedings of the 37th International Conference on Machine Learning , pages 4116-4126, 2020.

Roger A Horn and Charles R Johnson. Matrix analysis . Cambridge University Press, 2012.

Sergey Ivanov and Evgeny Burnaev. Anonymous walk embeddings. In Proceedings of the 35th International Conference on Machine Learning , pages 2191-2200, 2018.

Bo Jiang, Chris H. Q. Ding, Bin Luo, and Jin Tang. Graph-laplacian PCA: closed-form solution and robustness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3492-3498, 2013.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations , 2017.

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In Proceedings of the 7th International Conference on Learning Representations , 2019.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence , pages 3538-3545, 2018.

Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 338-348, 2020.

Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems , 14, 2001.

Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In Proceedings of the 33rd International Conference on Machine Learning , pages 2014-2023, 2016.

Hoang Nt and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint arXiv:1905.09550 , 2019.

Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, Lina Yao, and Chengqi Zhang. Adversarially regularized graph autoencoder for graph embedding. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence , pages 2609-2615, 2018.

Jiwoong Park, Minsik Lee, Hyung Jin Chang, Kyuewang Lee, and Jin Young Choi. Symmetric graph convolutional autoencoder for unsupervised graph representation learning. In Proceedings of IEEE/CVF International Conference on Computer Vision , pages 6518-6527, 2019.

Jinyoung Park, Sungdong Yoo, Jihwan Park, and Hyunwoo J. Kim. Deformable graph convolutional networks. In Proceedings of the 36th AAAI Conference on Artificial Intelligence , pages 7949-7956, 2022.

Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 701-710, 2014.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems , pages 1177-1184, 2007.

Walter Rudin. Fourier analysis on groups . Wiley Classics Library, 1994.

Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural computation , 10(5):1299-1319, 1998.

Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation , 10(5):1299-1319, 1998.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research , 12:25392561, 2011.

Ke Sun, Zhanxing Zhu, and Zhouchen Lin. Adagcn: Adaboosting graph convolutional networks into deep models. In Proceedings of the 9th International Conference on Learning Representations , 2021.

Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. LINE: large-scale information network embedding. In Proceedings of the Web Conference , pages 1067-1077, 2015.

Wenxuan Tu, Sihang Zhou, Xinwang Liu, Xifeng Guo, Zhiping Cai, En Zhu, and Jieren Cheng. Deep fusion clustering network. In 35th AAAI Conference on Artificial Intelligence , pages 9978-9987, 2021.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(11), 2008.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In Proceedings of the 6th International Conference on Learning Representations , 2018.

Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In Proceedings of the 7th International Conference on Learning Representations , 2019.

Ulrike Von Luxburg. A tutorial on spectral clustering. Statistics and computing , 17:395-416, 2007.

Xiao Wang, Meiqi Zhu, Deyu Bo, Peng Cui, Chuan Shi, and Jian Pei. AM-GCN: adaptive multichannel graph convolutional networks. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1243-1253, 2020.

Yifei Wang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Dissecting the diffusion process in linear graph convolutional networks. In Advances in Neural Information Processing Systems , pages 5758-5769, 2021.

Shiping Wang, Zhihao Wu, Yuhong Chen, and Yong Chen. Beyond graph convolutional network: An interpretable regularizer-centered optimization framework. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, , pages 4693-4701, 2023.

Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In Proceedings of the 36th International Conference on Machine Learning , pages 6861-6871, 2019.

Zhebin Wu, Lin Shu, Ziyue Xu, Yaomin Chang, Chuan Chen, and Zibin Zheng. Robust tensor graph convolutional networks via T-SVD based graph augmentation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2090-2099, 2022.

Zhihao Wu, Xincan Lin, Zhenghong Lin, Zhaoliang Chen, Yang Bai, and Shiping Wang. Interpretable graph convolutional network for multi-view semi-supervised learning. IEEE Transactions on Multimedia , pages 1-14, 2023.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Proceedings of the 35th International Conference on Machine Learning , pages 5449-5458, 2018.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In Proceedings of the 7th International Conference on Learning Representations , 2019.

Bingbing Xu, Bin Xie, and Huawei Shen. Towards deeper graph neural networks via layer-adaptive. In Proceedings of the Web Conference (WWW) , pages 103-106, 2023.

Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Proceedings of the 33rd International Conference on Machine Learning , pages 40-48, 2016.

Mingqi Yang, Yanming Shen, Rui Li, Heng Qi, Qiang Zhang, and Baocai Yin. A new perspective on the effects of spectrum in graph neural networks. In Proceedings of the 39th International Conference on Machine Learning , pages 25261-25279, 2022.

Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence , pages 4438-4445, 2018.

A Details about Recursive Formulations of GCKM

An L -layer GCKM on node feature X can be written as

Note that H ( l +1) is actually implicit and we directly obtain the kernel matrix K ( l +1) = H ( l +1) H ( l +1) ⊤ in practice. Now we show how to derive K ( l +1) only by kernel matrix K ( l ) . Take the Gaussian kernel as an example, we have K ( l +1) ij = exp ( -∆ ( l +1) ij 2 σ 2 l +1 ) , ( i, j ) ∈ [ n ] × [ n ] , where ∆ ( l +1) is a squared-distance matrix of the rows of ¯ H ( l +1) . To be more precise,

where ¯ h ( l +1) i (a row vector) is the representation of the i -th node drawn from

where K ( l ) ij = exp ( -∥ ( ˆ A q ) i H ( l -1) -( ˆ A q ) j H ( l -1) ∥ 2 2 σ 2 l ) , ( i, j ) ∈ [ n ] × [ n ] . Particularly, we define K (0) = XX ⊤ . So K ( l +1) ij can be obtained through K ( l ) , i.e.,

For convenience, letting ¯ K ( l +1) = ˆ A q K ( l ) ( ˆ A q ) ⊤ and d ¯ K ( l +1) = [ ¯ K ( l +1) 11 , ¯ K ( l +1) 22 , . . . , ¯ K ( l +1) nn ] ⊤ , we can calculate K ( l +1) in the following matrix form

B Proof for Theorem 1

Proof. Since ˆ c is the optimal solution of the dual problem, the optimal solution ˆ w of the primary problem can be given by

̸

We have

̸

On the other hand, we have

Now we use the following theorem, where the notations are a little different from ours.

Theorem 3 ([Grønlund et al. , 2020]) . Let d ∈ N + and let R > 0 . Denote by X the ball of radius R in R d and let D be any distribution over X ×{-1 , 1 } . For every δ > 0 , it holds with probability at least 1 -δ over a set of n samples S ∼ D n , that for every w ∈ R d with ∥ w ∥ 2 ≤ 1 and every margin θ > 0 , we have

According to our setting, we have R = 1 and θ = 1 ∥ ˆ w ∥ , since ∥ φ ( x i ) ∥ 2 = K ( L ) ii and y ⟨ ˆ w , φ ( x ) ⟩ ≥ 1 ⇔ y ⟨ ˆ w ∥ ˆ w ∥ , φ ( x ) ⟩ ≥ 1 ∥ ˆ w ∥ . For convenience, let η = ∥ ˆ w ∥ , which was given by (27). Substituting these values into Theorem 3 and changing the notations, we obtain

This finished the proof.

C Proof for Theorem 2

Proof. Here we recall the Lagrangian dual problem:

For φ , we define

Similarly, for φ G , we let

Then we need to minimize L ( c ) and L G ( c ) under the constraints ∑ n i =1 c i y i = 0 , 0 ≤ c i ≤ λ n . According to the Assumption 1, we have:

if samples i and j are in the same class, φ G ( x i ) ⊤ φ G ( x j ) > φ ( x i ) ⊤ φ ( x j ) and y i y j = 1 ;

if samples i and j are in different classes, φ G ( x i ) ⊤ φ G ( x j ) ≤ φ ( x i ) ⊤ φ ( x j ) and y i y j = -1 .

Therefore, the following equality holds:

For convenience, let ¯ ϵ i = min j ϵ ij and ˜ ϵ = min i ¯ ϵ i . We have

where c /i = [ c 1 , . . . , c i -1 , c i +1 , . . . , c n ] ⊤ . It is known that the ℓ 1 -norm ∥ · ∥ 1 is a convex relaxation of the ℓ 0 -norm ∥ · ∥ 0 , i.e., the number of nonzero elements in a vector. Denote R ( c ) := 1 2 ( ˜ ϵ ∥ c ∥ 2 2 + ∑ n i =1 ¯ ϵ i c i ∥ c /i ∥ 1 ) . We see R ( c ) is very similar to the elastic net regularization and is able to induce sparsity. Actually, if we let κ = min i ∥ c /i ∥ 1 , we have R ( c ) ≥ 1 2 ( ˜ ϵ ∥ c ∥ 2 2 + κ ∑ n i =1 ¯ ϵ i c i ) = 1 2 ( ˜ ϵ ∥ c ∥ 2 2 + κ ∥ diag (¯ ϵ ) c ∥ ) , where the second term is a weighted ℓ 1 -norm and also induces sparisty.

D Proof for Proposition 1

Proof. Let

It is easy to show that Q is full-rank. Because k ( L ) is a Gaussian kernel with 0 < σ L < ∞ , ¯ K ( L ) is symmetric and always full-rank. It means ¯ K ( L ) can be factorized as ¯ K ( L ) = B ⊤ B , where B is a square and full-rank matrix. Denote P = BQ . It follows that

Since P is full-rank, we conclude that K ( L ) G is full-rank, which means K ( L ) G is positive definite [Horn and Johnson, 2012].

E Detailed Derivations of GCKM-E

Theorem 2 (Bochner's theorem [Rudin, 1994]) . A continuous kernel k ( x , x ′ ) = k ( x -x ′ ) on R m is positive definite if and only if k ( δ ) is the Fourier transform of a non-negative measure.

With positive definite and shift-invariant kernel k ( · , · ) (e.g. Gaussian kernel), the Bochner's theorem points out that the kernel function can be expanded with harmonic basis as

where p ( ω ) is a probability distribution and also the Fourier transform of k ( δ ) . For real-valued kernel functions, e j ω ⊤ x i e -j ω ⊤ x j can be simplified by

according to Euler's formula. Then we can define function ζ ω ( · ) with ζ ω ( x i ) ζ ⊤ ω ( x j ) = cos ω ⊤ ( x i -x j ) , so that it satisfies

Here we simply adopt ζ ω ( x ) = [ cos( ω ⊤ x ) , sin( ω ⊤ x ) ] ⊤ , and the kernel function can be approximated by combining several ω drawn from distribution p ( ω ) .

Specifically, for Gaussian kernel, we first randomly sample { ω 1 , ω 2 , . . . , ω D } from the probability distribution p RBF ( ω ) = N ( 0 , 1 σ I ) . Subsequently the explicit feature transformation on the i -th sample x i is defined as

On the basis of the above derivations, the multi-layer GCKM with explicit feature transformation can be similarly defined:

where Z ( l +1) is the random Fourier features of input Z ( l ) . This approach allows GCKM to flexibly choose explicit node representations or kernel matrix as the interface, and the final representation can be low-dimensional and thus efficiently computed. By selecting a proper D , the convergence of this approximation and the expressive ability of the kernel can be simultaneously guaranteed.

F Detailed Experimental Settings and Results

F.1 Experimental Settings

Two types of graph datasets are employed in this paper. Each node-level dataset contains only a graph where each node is a sample, while each graph is regarded as a sample in the graph-level datasets. Here we list the key statistics of these datasets in Table 4 and 5 and provide detailed descriptions in the following.

Node-level Datasets

Cora is a citation network containing a number of machine learning papers. Each node represents a paper and the edges represent the citation relationships between papers. Node features are the bag-of-words representations of papers, and all papers are divided into 7 categories according to their domains.

Citeseer is also a well-known citation network with nodes, node features, and edges having the same meanings as Cora. Similarly, the papers are grouped into 6 classes.

Pubmed is a citation network composed of 19 , 717 scientific publications drawn from the PubMed database, which are classified into 3 categories and node features are also the bag-of-words representations.

ACM is a paper network. Different from the above citation networks, the edges denote the co-author relationships between any two papers. Another difference is that the node features are bag-of-words representations of papers' keywords.

CoraFull is a large version of the Cora dataset which has 19 , 793 papers of 70 classes.

Chameleon is a webpage network, in which nodes are Wikipedia pages of certain topics and edges are the page-page hyperlinks. Some informative nouns in the pages are extracted to be node features and the nodes are organized in 5 categories based on the average monthly traffic of the corresponding web pages.

Actor is a subgraph of the film-director-actor-writer network which only includes actors as nodes. An edge between two actors exists when they occur on the same Wikipedia page.

Squirrel is a webpage network as well, and all the settings are similar to the Chameleon dataset.

UAI is a webpage network that has been used to test GCN for community detection. Nodes represent web pages and each edge represents a citation between two pages.

OGB-Arxiv dataset is a citation network with 169 , 343 computer science arXiv papers, where each node is an arXiv paper and each edge indicates that a paper cites another paper. Each paper has a 128-dimensional feature vector, which is obtained by averaging embeddings of words in its title and abstract.

Graph-level Datasets

IMDB-BINARY and IMDB-MULTI are movie collaboration datasets, where each graph represents an ego-network for each actor/actress. Each node corresponds to an actor/actress, and the edges represent the actors/actresses' co-occurrences in a movie. Graphs are classified according to the genre of the movies they come from.

COLLAB is a scientific collaboration dataset, collected from three public collaboration datasets. Similar to the two datasets mentioned above, each graph is a self-network of different researchers, and these graphs are classified by the domains to which their corresponding researchers belong.

MUTAG is a bioinformatics dataset. 188 mutagenic aromatic and heteroaromatic nitro compounds with 7 labels compose this dataset.

PROTEINS is a bioinformatics dataset where nodes denote secondary structure elements with 3 labels. An edge between any two nodes means they are neighbors in the amino-acid sequence or 3D space.

PTC is a bioinformatics dataset containing 344 chemical compounds that report carcinogenicity for male and female rats and it has 19 discrete labels.

For datasets ACM, Chameleon, CoraFull, Actor, Squirrel, and UAI, we randomly split them into the train, validation, and test sets and fix them for all the compared methods. For graph-level datasets split, we follow settings in [Xu et al. , 2019].

Then, we also briefly introduce the compared methods used in node-level tasks and graph-level tasks.

Node-level Compared Methods

Chebyshev [Defferrard et al. , 2016] generalizes the convolution operation to the nonEuclidean space based on spectral graph theory, which leverages Chebyshev polynomial to design a localized graph convolutional filter.

GraphSAGE [Hamilton et al. , 2017] is proposed to tackle the drawbacks of transductive GNNs. It trains the aggregation function from each node's local neighborhood, and this function can be performed on unseen nodes.

GAT [Velickovic et al. , 2018] is a classical spatial GNN equipping with self-attentional layers that adaptively learn different weights for edges.

GCN [Kipf and Welling, 2017] is one of the most popular GNNs recently. It truncates the Chebyshev polynomial to first order and proposes a simple and effective graph convolution.

SGC [Wu et al. , 2019] further simplifies GCN by removing the nonlinearity and collapsing the learnable weights, so as to build a faster model and achieve competitive performance.

APPNP [Klicpera et al. , 2019] uses personalized Pagerank to improve vanilla GCN by solving the over-smoothing issue and derives a new framework with initial residual connections.

JKNet [Xu et al. , 2018] establishes a jumping knowledge structure to combine representations from various layers to stack a deeper GNN.

DAGNN [Xu et al. , 2023] decouples the neighbor aggregations and feature transformations to form a deeper GNN and further uses the attention mechanism to integrate the information from different depths.

AdaGCN [Sun et al. , 2021] is also a deep GCN that considers the Adaboost strategy to fuse knowledge from distinct layers.

AMGCN [Wang et al. , 2020] constructs a feature graph via k -NN algorithm and a multichannel framework to enhance the vanilla GCN.

DefGCN [Park et al. , 2022] improves the limitations of common GNNs with fixed graph convolution and has deformable graph convolution that allows nodes to adaptively capture long-range dependencies.

Graph-level Compared Methods

WLsubtree kernel [Shervashidze et al. , 2011] is a famous graph kernel, and AWL [Ivanov and Burnaev, 2018] is an approach for embedding entire graphs. Paired with SVM, they are selected as two baselines.

DCNN [Atwood and Towsley, 2016], PATCHY-SAN [Niepert et al. , 2016] and DGCNN [Zhang et al. , 2018] are three deep learning methods for graph classification, and their performance is reported as in the original paper.

The Number of Layers/Hops

Accuracy (%)

GCN, GraphSAGE, and GIN [Xu et al. , 2019] serve as GNN-based compared methods, and they are all set following [Xu et al. , 2019]. For the methods having different variants, we report their best performance.

Note that the settings of these graph-level methods are the same as [Xu et al. , 2019].

Settings about Figure 1. We set the hidden dimensions as 32 for all hidden layers of GCN, APPNP, and JKNet, while SGC only has a learnable matrix W ∈ R n × c where n is the number of nodes and c is the number of classes, and the learning rate is selected in { 1 × 10 -2 , 1 × 10 -3 , 1 × 10 -4 } and weight decay is selected in { 5 × 10 -4 , 5 × 10 -5 , 5 × 10 -6 } .

Settings of Boundary Visualization For better illustration, we conduct a visualization experiment on the decision boundaries of SGC, APPNP, GCN, and GCKSVM. The purpose of this experiment is to explore how the three methods perform on linear, non-linear, and more complex data. So we select 2 classes of samples from Cora and Chameleon and map them to 2-D space by t-SNE. Besides, we generate a synthetic dataset (called Circle) consisting of noisy samples drawn from two concentric circles using Scikit-learn. For better boundary and performance, the numbers of layers are 2 and 8 for GCN and APPNP respectively, the hidden dimensions are 32 , and the learning rate is set as 1 × 10 -3 and weight decay is 5 × 10 -5 for all models. SGC has only one learnable matrix and we set the power of adjacency as 50 to show that the graph structure provide SGC with non-linearity but the non-linearity is very low.

F.2 Supplementary Experiments of Node Classification

In this section, we evaluate GCKSVM on more types of graph datasets. As reported in Table 6, it can be seen that GCKSVM still has superior performance compared to the most state-of-the-art GNNs. Furthermore, we also evaluate the proposed GCKM on the OGB dataset Arxiv with over 160k nodes. Table 7 demonstrates that GCKM is still competitive with these GNNs on large-scale datasets. Although the complexity of GCKM is quadratic, we provide an efficient variant GCKM-E in Section 3.4 that leverages the random Fourier feature to explicitly derive a low-dimensional output instead of a kernel matrix. With this output, we can apply fast linear methods (e.g. linear SVM) for downstream tasks, which is still more efficient than GNNs needing a lengthy training process.

F.3 Experiments and Discussions on Over-smoothing Issue

Thanks for your insightful comment. We have supplemented an experiment on this issue, which revealed that GCKM can be deeper and alleviate the over-smoothing issue. To be specific, two situations are considered in this experiment:

GCKM with fixed 2 layers and varied hops of neighbors per aggregation ( q in Eq. (40))

GCKM with fixed 2 hops of neighbors per aggregation and varied layers

From Figure 6, we have the following observations:

Deep GCKM performs more stably than deep GCN.

GCKM's performance first improves then slightly decreases with increasing layers/hops

Particularly, GCKM with fixed layers performs better and decreases less.

It is known that over-smoothing is caused by the aggregation step, that is, multiplying ˆ A makes the representations of different nodes more and more indistinguishable. However, the analysis of over-smoothing does not consider activation functions and learnable weights, and it theoretically exists when the power of ˆ A tends to infinity [Li et al. , 2018], which does not match the fact that GCN collapses with only 8 layers. A recent study of [Cong et al. , 2021] has pointed out that the over-smoothing problem might be an artifact of theoretical analysis and the failure of deep GNNs may not only be caused by the over-smoothing issue in the aggregation/message-passing step.

Although GCKM and GCN share a familiar aggregation step, the main difference between them is the transformation step, namely, GCN uses a linear layer to conduct explicit dimension reduction while GCKM employs implicit high-dimensional feature mapping. GCKM implicitly maps node features to a high-dimensional (even infinite) space after aggregation, and there may exist an appropriate space where node representation can be distinguished. In contrast, a recent study of [Guo et al. , 2023] has found that the node representations processed by deep GCN would collapse to be low-rank and lose expressive power. The appropriate space can be found by tuning the hyperparameters of GCKM, however, the number of hyperparameters will increase when building a deeper GCKM and it is time-consuming to search this space. That may be the reason why GCKM decreases slightly with too many layers and why fixing the number of layers improves the performance.

F.4 Experiments on Various Kernels

To solve the limitations we stated in the conclusion section, we provided a complementary experiment on various kernel functions during the rebuttal process. Table 8 records the results corresponding to 2nd order polynomial kernel, sigmoid kernel, and Laplacian kernel. All the kernel functions show decent performance and the Gaussian kernel performs the best. Note that, this experiment is conducted with GCKSVM, under the settings of node classification with standard split.

F.5 Experiments of GCKPCA

We added a visualization experiment for GCKPCA. To be specific, we first map the node features to 2-D space by PCA, Graph-regularized PCA (GPCA) [Jiang et al. , 2013] and GCKPCA (row 1), then further try to map them to 32-D space and leverage t-SNE to obtain the 2-D results (row 2). Figure 7 illustrates the comparison of mapping results of three methods. It can be observed that the subfigures of GCKPCA and GCKPCA + t-SNE both show the best separability between different classes, and GPCA performs slightly better than PCA.

F.6 Empirical Experiments regarding Theorem 1

Theorem 1 demonstrated the connections between the generalization bound and graph structure, and experiments further provided the evidence. To verify the conclusion we have made from Theorem 1 in the main paper, we report the training error and numbers of support vectors in Table 9. In fact, we consider three cases:

Non-graph-convolution: We replaced the affinity matrix ˆ A in GCKM with an identity matrix I n , which means the graph structure is not used.

(a) PCA

(d) PCA + t-SNE

(b) GPCA

(e) GPCA + t-SNE

(c) GCKPCA

(f) GCKPCA + t-SNE

Strongly connected graph: ˆ A in GCKM is replaced with I n -1 n × n , which means every node is connected with all other nodes.

The normal GCKM.

We see that with fixed λ , graph convolution significantly reduces the number of support vectors, which verifies the correctness of Theorem 2. Thus, according to Theorem 1, the upper bound (monotonically increasing with |V| ) of test error can be tighter compared to the one using a non-graph-convolution kernel, which means that the test error is potentially smaller. So we can conclude that the graph structure significantly improves the quality of the kernel matrix.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework called graph convolutional kernel machine (GCKM) for graph-based machine learning. GCKMs are built upon kernel functions integrated with graph convolution. Within the framework of GCKM, the authors propose GCKSVM for node classification, GCKSC for node clustering, and extensions for graph-level learning. The experiments show that GCKMs have at least competitive accuracy compared to GCNs.\n\nSTRENGTHS:\n1. The paper presents a framework GCKM for graph-based machine learning. As far as I am concerned, this is a novel framework. Compared to GCNs, GCKMs are easier to train, are guaranteed to obtain globally optimal solutions, and have strong generalization ability. \n2. The authors provided generalization bound for GCKSVM, which justified the advantage of GCKSVM over KSVM.\n3. The authors provided GCKM extensions to node clustering, graph classification, etc. They also provided fast feature transformation for GCKM. \n4. The numerical evaluation are sufficient, which showed the proposed methods are at least as effective as GCNs.\n\nWEAKNESSES:\nTwo minor issues are as follows.\n1. In Section 3.3, the extension GCKPCA hasn’t been evaluated. \n2. Some important results are in the supplement rather than the main paper. The authors may reorganize the paper and show the numerical justification for Theorem 1 and the experiment of graph-level learning in the main paper.\n\nQUESTIONS:\n1. In line 171, it is claimed “GCKM with Gaussian kernels can be easily generalized to multi-layer cases”. Could the authors provide a recursive or explicit formulation of deep GCKM? Is there a rule of thumb to determine the number of layers for deep GCKM?\n2. In line 204, the formulation “...has tiny influences on the training...and the spectral norm of $[K_{ij}^{(L)}]_{i,j\\in V}$...” is a little bit confusing. The matrix will become smaller if there are fewer support vectors and a smaller matrix usually has a smaller spectral norm.\n3. In Eq. (18) or Eq. (16), the graph-level feature is obtained as the sum of nodes’ features. Is there any explanation for this operation? How about using the mean or min/max of nodes’ features as graph feature?\n4. At the end of Section 3.4, I suggest the authors highlight the computational complexity in comparison to the implicit feature transformation method.\n5. In the first column of Figure 3, why are the decision boundaries of SGC in the second and third plots are not nonlinear?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a kernel-based message-passing framework for graph convolutional networks called GCKM. The author demonstrated that GCKM is computationally efficient with stable performance on both node classification and graph classification tasks. Theoretical analyses of GCKM are provided.\n\nSTRENGTHS:\n1. GCKM modifies the general GNN by replacing the trainable parameters and non-linear activation functions with kernels (e.g., RBF Gaussian kernel). Classification is then done using well-established SVM. The proposed method significantly reduces the running time. \n2. Provide theoretical analyses.\n\nWEAKNESSES:\n1. In section 3.4, the author claims that their method can be applied to very large graph datasets. What does “large” mean? In experiments, only small graph datasets (Cora, Citeseer, and PubMed) were used. How about relatively large datasets like Reddit or OGB datasets?\n2. The value/insight of Theorem 1 is unclear. How can one use it to guide the design of GCKM models? How close is the theoretical bound to experimental observations? How can this address the challenges (like over-smoothing or improving model performances)?\n3. Previous work [1] already conducts extensive exploration on applying kernel methods on GNNs, the author needs to discuss or compare their method with it. \n4. The settings of the models used to produce Figures 1 and 2, e.g. the number of layers, and hidden dimensions, especially for the deep GCNs.\n5. Abstract claims \"GCKMs are guaranteed to obtain globally optimal solutions and have strong generalization ability and high interpretability.\" If true, should the GCKM produce the best results on all dataset? Interpretability need to be elaborated.\n\n[1] Chen, Dexiong, Laurent Jacob, and Julien Mairal. \"Convolutional kernel networks for graph-structured data.\" International Conference on Machine Learning. PMLR, 2020.\n\nQUESTIONS:\n1. What is the memory requirement of GCKM, when compared to other methods?\n\n2. The over-smoothing problem in GCN or another type of GNN models is caused by stacking multiple message-passing layers, and eventually, every node shares similar embeddings. In Formula (5), a single GCKM layer is constructed by applying several times of graph convolution and using a pre-defined Gaussian kernel function as the ReadOut function. How does this architecture address the over-smoothing problem?\n\nLIMITATIONS:\nNot sure if the choice/design of kernel may cause any biases.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new support vector machine approach for graph learning called graph convolutional kernel machine. GCKM combines traditional kernel functions with graph convolution, and shows good performance in both node- and graph-level tasks. Generalization bound is also provided for the approach.\n\nSTRENGTHS:\n1. The proposed GCKM is simple yet effective. It is impressive that SVM-based approach can perform on par with deep GNN models in both node- and graph-level tasks.\n2. Generalization bound is provided for the proposed GCKM, which partially explains how graph structure in the construction of kernel function benefits generalization performance.\n3. Many variants are developed for applications in different scenarios.\n\nWEAKNESSES:\n1. Though many datasets and tasks are considered in the paper, all datasets are quite small with the largest one consisting of only 20k nodes. While the paper claims GCKM can be \"extended to large-scale data” by using low-rank approximation tricks, no experimental results on large datasets are provided. Additionally, I would also suggest the authors put some results in supplementary experiments to the main text as cora/citeseer/pubmed are somewhat outdated and insufficient to reflect the effectiveness of the proposed approach.\n2. The explanation of how the graph structure affects the generalization of GCKM is somewhat vague. Specifically, I am still confused of why “graph structure significantly improved the quality of the kernel matrix” and how it affects the generalization bound. \n3. Literature coverage could be improved. Some prior work have also analyzed how graph structure in the construction of kernel function affects generalization in graph [1] and node [2] tasks, and some others adopted kernel methods for graph learning e.g. [3,4], which are related to this work.\n4. As the authors stated in conclusion, we did not systematically test other kernel functions.\n\n[1] Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels, NeurIPS 2019  \n[2] Graph Neural Networks are Inherently Good Generalizers Insights by Bridging GNNs and MLPs, ICLR 2023  \n[3] Convolutional Kernel Networks for Graph-Structured Data, ICML 2020  \n[4] KerGNNs: Interpretable Graph Neural Networks with Graph Kernels, AAAI 2022\n\nQUESTIONS:\n1. How does the GCKM perform on larger datasets, e.g. open graph benchmark? Such the complexity of GCKM is quadratic, is it still more efficient than GNNs in those larger datasets?\n2. Why “graph structure significantly improved the quality of the kernel matrix” and how it affects the generalization bound?\n\nLIMITATIONS:\nSee limitations in weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach called the Graph Convolutional Kernel Machine (GCKM) for graph learning. Unlike other neural network-based frameworks, GCKM employs a graph kernel to replace the neighbor aggregation step, without any learnable parameters.\nThen the author build the features based on the graph kernel and then use SVM for classification tasks.\n\nIn general, I find the paper to be good and interesting, although it could benefit from additional experiments. I am somewhat surprised by the better performance of GCKSVM compared to GAT, considering that GAT defines the similarity of node features using learnable attention weights.\n\nSTRENGTHS:\n1. Well written and clear explained\n2. good visualization (Figure 3) of the potential benefit of kernel methods: the Kernel methods offer greater interpretability compared to neural networks and provide stronger generalization guarantees.  In contrast to SGC, which functions as a linear classifier based on the final node representation, GCKSVM serves as a nonlinear classifier. Therefore, it is anticipated to outperform SGC in cases where the data is not linearly separable. \n3. Faster running time.\n\nWEAKNESSES:\n1. The performance is relatively underwhelming. Table 2 reveals that GCKSC achieves the best results in only 3 cases and the second-best results in 2 cases, whereas S3 GC outperforms with the best results in 3 cases and the second-best results in 3 cases.\n2. The performance reported in this paper appears to differ significantly from the results presented in the APPNP paper (https://arxiv.org/pdf/1810.05997v6.pdf). In the APPNP paper, they reported 85% accuracy on Cora and 75% accuracy on Citeseer. The substantial difference in performance raises questions about potential differences in the experimental settings between the two papers. If there are indeed differences in the experimental settings, it would be important to assess whether the conclusions drawn in this paper still hold when utilizing the experimental settings from the APPNP paper.\n\nQUESTIONS:\nThe author discusses the issue of over-smoothing, and it raises the question of whether over-smoothing would occur when employing GCKSC with more layers. Specifically, is there a significant drop in performance when utilizing GCKSC with more than 2 layers?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}