SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \Delta_ {i j} ^ {(l + 1)} = \left(\hat {\mathbf {A}} ^ {q}\right) _ {i} \phi_ {(l - 1)} \left(\bar {\mathbf {H}} ^ {(l)}\right) \phi_ {(l - 1)} ^ {\top} \left(\bar {\mathbf {H}} ^ {(l)}\right) \left(\hat {\mathbf {A}} ^ {q}\right) _ {i} ^ {\top} - 2 \left(\hat {\mathbf {A}} ^ {q}\right) _ {i} \phi_ {(l - 1)} \left(\bar {\mathbf {H}} ^ {(l)}\right) \phi_ {(l - 1)} ^ {\top} \left(\bar {\mathbf {H}} ^ {(l)}\right) \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} ^ {\top} \\ + \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} \phi_ {(l - 1)} \left(\bar {\mathbf {H}} ^ {(1)}\right) \phi_ {(l - 1)} ^ {\top} \left(\bar {\mathbf {H}} ^ {(1)}\right) \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} ^ {\top} \\ = \left(\hat {\mathbf {A}} ^ {q}\right) _ {i} \mathbf {K} ^ {(l)} \left(\hat {\mathbf {A}} ^ {q}\right) _ {i} ^ {\top} - 2 \left(\hat {\mathbf {A}} ^ {q}\right) _ {i} \mathbf {K} ^ {(l)} \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} ^ {\top} + \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} \mathbf {K} ^ {(l)} \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} ^ {\top}, \tag {22} \\ \end{array}
$$
where  $K_{ij}^{(l)} = \exp \left(-\frac{\|\hat{A}^q)_i\mathbf{H}^{(l - 1)} - (\hat{\mathbf{A}}^q)_j\mathbf{H}^{(l - 1)}\|^2}{2\sigma_l^2}\right)$ ,  $(i,j)\in [n]\times [n]$ . Particularly, we define  $\mathbf{K}^{(0)} = \mathbf{X}\mathbf{X}^\top$ . So  $K_{ij}^{(l + 1)}$  can be obtained through  $\mathbf{K}^{(l)}$ , i.e.,
$$
K _ {i j} ^ {(l + 1)} = \exp \left(- \frac {\left(\hat {\mathbf {A}} ^ {q}\right) _ {i} \mathbf {K} ^ {(l)} \left(\hat {\mathbf {A}} ^ {q}\right) _ {i} ^ {\top} - 2 \left(\hat {\mathbf {A}} ^ {q}\right) _ {i} \mathbf {K} ^ {(l)} \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} ^ {\top} + \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} \mathbf {K} ^ {(l)} \left(\hat {\mathbf {A}} ^ {q}\right) _ {j} ^ {\top}}{2 \sigma_ {l + 1} ^ {2}}\right). \tag {23}
$$
For convenience, letting  $\bar{\mathbf{K}}^{(l + 1)} = \hat{\mathbf{A}}^q\mathbf{K}^{(l)}(\hat{\mathbf{A}}^q)^\top$  and  $\mathbf{d}_{\bar{\mathbf{K}}^{(l + 1)}} = [\bar{K}_{11}^{(l + 1)},\bar{K}_{22}^{(l + 1)},\dots ,\bar{K}_{nn}^{(l + 1)}]^\top$  we can calculate  $\mathbf{K}^{(l + 1)}$  in the following matrix form
$$
\mathbf {K} ^ {(l + 1)} = \exp \left(- \frac {\mathbf {d} _ {\bar {\mathbf {K}} ^ {(l + 1)}} \mathbf {1} _ {n} ^ {\top} + \mathbf {1} _ {n} \mathbf {d} _ {\bar {\mathbf {K}} ^ {(l + 1)}} ^ {\top} - 2 \bar {\mathbf {K}} ^ {(l + 1)}}{2 \sigma_ {l + 1} ^ {2}}\right). \tag {24}
$$
B Proof for Theorem 1
Proof. Since  $\hat{\mathbf{c}}$  is the optimal solution of the dual problem, the optimal solution  $\hat{\mathbf{w}}$  of the primary problem can be given by
$$
\hat {\mathbf {w}} = \sum_ {i \in \mathcal {V}} \hat {c} _ {i} y _ {i} \varphi (\mathbf {x} _ {i}). \tag {25}
$$
$$
\begin{array}{l} \| \hat {\mathbf {w}} \| _ {2} ^ {2} = \left(\sum_ {i \in \mathcal {V}} \hat {c} _ {i} y _ {i} \varphi (\mathbf {x} _ {i})\right) ^ {\top} \left(\sum_ {i \in \mathcal {V}} \hat {c} _ {i} y _ {i} \varphi (\mathbf {x} _ {i})\right) \\ = \sum_ {i \in \mathcal {V}} \sum_ {j \in \mathcal {V}} \hat {c} _ {i} \hat {c} _ {j} y _ {i} y _ {j} \varphi (\mathbf {x} _ {i}) ^ {\top} \varphi (\mathbf {x} _ {j}) \\ = \sum_ {i \in \mathcal {V}} \sum_ {j \in \mathcal {V}} \hat {c} _ {i} \hat {c} _ {j} y _ {i} y _ {j} K _ {i j} ^ {(L)} \tag {26} \\ = \sum_ {i \in \mathcal {V}} \hat {c} _ {i} ^ {2} K _ {i i} ^ {(L)} + \sum_ {i \in \mathcal {V}} \sum_ {j \in \mathcal {V} \backslash i, y _ {i} = y _ {j}} \hat {c} _ {i} \hat {c} _ {j} K _ {i j} ^ {(L)} - \sum_ {i \in \mathcal {V}} \sum_ {j \in \mathcal {V}, y _ {i} \neq y _ {j}} \hat {c} _ {i} \hat {c} _ {j} K _ {i j} ^ {(L)} \\ = \sum_ {i \in \mathcal {V}} \hat {c} _ {i} ^ {2} + \sum_ {i \in \mathcal {V}} \sum_ {j \in \mathcal {V} \backslash i, y _ {i} = y _ {j}} \hat {c} _ {i} \hat {c} _ {j} K _ {i j} ^ {(L)} - \sum_ {i \in \mathcal {V}} \sum_ {j \in \mathcal {V}, y _ {i} \neq y _ {j}} \hat {c} _ {i} \hat {c} _ {j} K _ {i j} ^ {(L)} \\ \end{array}
$$
$$
\begin{array}{l} \| \hat {\mathbf {w}} \| _ {2} ^ {2} = \sum_ {i \in \mathcal {V}} \sum_ {j \in \mathcal {V}} \hat {c} _ {i} \hat {c} _ {j} y _ {i} y _ {j} K _ {i j} ^ {(L)} \\ = \langle [ \dots \hat {c} _ {i} y _ {i} \dots ] ^ {\top} [ \dots \hat {c} _ {i} y _ {i} \dots ], [ K _ {i j} ^ {(L)} ] _ {i, j \in \nu} \rangle \\ \leq \left\| \left[ \dots \hat {c} _ {i} y _ {i} \dots \right] ^ {\top} \left[ \dots \hat {c} _ {i} y _ {i} \dots \right] \right\| _ {F} \left\| \left[ K _ {i j} ^ {(L)} \right] _ {i, j \in v} \right\| _ {F} \\ = \left\| \left[ K _ {i j} ^ {(L)} \right] _ {i, j \in \mathcal {V}} \right\| _ {F} \sum_ {i \in \mathcal {V}} \hat {c} _ {i} ^ {2} \tag {27} \\ \leq \left\| \left[ K _ {i j} ^ {(L)} \right] _ {i, j \in \mathcal {V}} \right\| _ {F} \frac {\lambda^ {2} | \mathcal {V} |}{n ^ {2}} \\ \leq \frac {\lambda^ {2} | \mathcal {V} | ^ {2}}{n ^ {2}}. \\ \end{array}
$$
Now we use the following theorem, where the notations are a little different from ours.
Theorem 3 ([Grønlund et al., 2020]). Let  $d \in \mathbb{N}^+$  and let  $R > 0$ . Denote by  $X$  the ball of radius  $R$  in  $\mathbb{R}^d$  and let  $\mathcal{D}$  be any distribution over  $X \times \{-1,1\}$ . For every  $\delta > 0$ , it holds with probability at least  $1 - \delta$  over a set of  $n$  samples  $S \sim \mathcal{D}^n$ , that for every  $w \in \mathbb{R}^d$  with  $\| w \|_2 \leq 1$  and every margin  $\theta > 0$ , we have
$$
\mathcal {L} _ {\mathcal {D}} (w) \leq \mathcal {L} _ {S} ^ {\theta} (w) + O \left(\frac {(R / \theta) ^ {2} \ln n + \ln (1 / \delta)}{n} + \sqrt {\frac {(R / \theta) ^ {2} \ln n + \ln (1 / \delta)}{n} \cdot \mathcal {L} _ {S} ^ {\theta} (w)}\right)
$$
According to our setting, we have  $R = 1$  and  $\theta = \frac{1}{\|\hat{\mathbf{w}}\|}$ , since  $\| \varphi (\mathbf{x}_i)\| ^2 = K_{ii}^{(L)}$  and  $y\langle \hat{\mathbf{w}},\varphi (\mathbf{x})\rangle \geq 1\Leftrightarrow y\langle \frac{\hat{\mathbf{w}}}{\|\hat{\mathbf{w}}\|},\varphi (\mathbf{x})\rangle \geq \frac{1}{\|\hat{\mathbf{w}}\|}$ . For convenience, let  $\eta = \| \hat{\mathbf{w}}\|$ , which was given by (27). Substituting these values into Theorem 3 and changing the notations, we obtain
$$
\mathcal {L} _ {\mathcal {D}} \left(\hat {\mathbf {c}}, \mathbf {K} ^ {(L)}\right) \leq \mathcal {L} _ {S} \left(\hat {\mathbf {c}}, \mathbf {K} ^ {(L)}\right) + O \left(\frac {\eta \ln n + \ln (1 / \delta)}{n} + \sqrt {\frac {\eta \ln n + \ln (1 / \delta)}{n} \cdot \mathcal {L} _ {S} \left(\hat {\mathbf {c}} , \mathbf {K} ^ {(L)}\right)}\right). \tag {28}
$$
This finished the proof.
C Proof for Theorem 2
Proof. Here we recall the Lagrangian dual problem:
$$
\max  _ {\mathbf {c}} \sum_ {i = 1} ^ {n} c _ {i} - \frac {1}{2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} c _ {i} c _ {j} y _ {i} y _ {j} \varphi \left(\mathbf {x} _ {i}\right) ^ {\top} \varphi \left(\mathbf {x} _ {j}\right) \quad \text {s . t .} \quad \sum_ {i = 1} ^ {n} c _ {i} y _ {i} = 0, 0 \leq c _ {i} \leq \frac {\lambda}{n}. \tag {29}
$$
For  $\varphi$ , we define
$$
\mathcal {L} (\mathbf {c}) := \frac {1}{2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} c _ {i} c _ {j} q _ {i j} - \sum_ {i = 1} ^ {n} c _ {i}, \text {w h e r e} q _ {i j} = y _ {i} y _ {j} \varphi \left(\mathbf {x} _ {i}\right) ^ {\top} \varphi \left(\mathbf {x} _ {j}\right). \tag {30}
$$
Similarly, for  $\varphi_{\mathcal{G}}$  , we let
$$
\mathcal {L} _ {\mathcal {G}} (\mathbf {c}) := \frac {1}{2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} c _ {i} c _ {j} q _ {i j} ^ {\mathcal {G}} - \sum_ {i = 1} ^ {n} c _ {i}, \text {w h e r e} q _ {i j} ^ {\mathcal {G}} = y _ {i} y _ {j} \varphi_ {\mathcal {G}} \left(\mathbf {x} _ {i}\right) ^ {\top} \varphi_ {\mathcal {G}} \left(\mathbf {x} _ {j}\right). \tag {31}
$$
Then we need to minimize  $\mathcal{L}(\mathbf{c})$  and  $\mathcal{L}_{\mathcal{G}}(\mathbf{c})$  under the constraints  $\sum_{i=1}^{n} c_i y_i = 0$ ,  $0 \leq c_i \leq \frac{\lambda}{n}$ . According to the Assumption 1, we have:
- if samples  $i$  and  $j$  are in the same class,  $\varphi_{\mathcal{G}}(\mathbf{x}_i)^\top \varphi_{\mathcal{G}}(\mathbf{x}_j) > \varphi (\mathbf{x}_i)^\top \varphi (\mathbf{x}_j)$  and  $y_{i}y_{j} = 1$
- if samples  $i$  and  $j$  are in different classes,  $\varphi_{\mathcal{G}}(\mathbf{x}_i)^\top \varphi_{\mathcal{G}}(\mathbf{x}_j) \leq \varphi (\mathbf{x}_i)^\top \varphi (\mathbf{x}_j)$  and  $y_{i}y_{j} = -1$
Therefore, the following equality holds:
$$
q _ {i j} ^ {\mathcal {G}} = q _ {i j} + \epsilon_ {i j}, \quad \epsilon_ {i j} \geq 0, \forall (i, j) \in [ n ] \times [ n ] \tag {32}
$$
For convenience, let  $\bar{\epsilon}_i = \min_j\epsilon_{ij}$  and  $\tilde{\epsilon} = \min_i\bar{\epsilon}_i$  . We have
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {G}} (\mathbf {c}) := \frac {1}{2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} c _ {i} c _ {j} \left(q _ {i j} + \epsilon_ {i j}\right) - \sum_ {i = 1} ^ {n} c _ {i} \\ \geq \frac {1}{2} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} c _ {i} c _ {j} q _ {i j} + \frac {1}{2} \sum_ {i = 1} ^ {n} \bar {\epsilon} _ {i} \sum_ {j = 1} ^ {n} c _ {i} c _ {j} - \sum_ {i = 1} ^ {n} c _ {i} \tag {33} \\ = \mathcal {L} (\mathbf {c}) + \frac {1}{2} \sum_ {i = 1} ^ {n} \tilde {\epsilon} _ {i} c _ {i} \sum_ {j = 1} ^ {n} c _ {j} \\ \geq \mathcal {L} (\mathbf {c}) + \frac {1}{2} \left(\tilde {\epsilon} \| \mathbf {c} \| _ {2} ^ {2} + \sum_ {i = 1} ^ {n} \bar {\epsilon} _ {i} c _ {i} \| \mathbf {c} _ {/ i} \| _ {1}\right), \\ \end{array}
$$
where  $\mathbf{c}_{/i} = [c_1, \dots, c_{i-1}, c_{i+1}, \dots, c_n]^\top$ . It is known that the  $\ell_1$ -norm  $\|\cdot\|_1$  is a convex relaxation of the  $\ell_0$ -norm  $\|\cdot\|_0$ , i.e., the number of nonzero elements in a vector. Denote  $\mathcal{R}(\mathbf{c}) := \frac{1}{2}\left(\tilde{\epsilon}\|\mathbf{c}\|_2^2 + \sum_{i=1}^{n}\bar{\epsilon}_ic_i\|\mathbf{c}_{/i}\|_1\right)$ . We see  $\mathcal{R}(\mathbf{c})$  is very similar to the elastic net regularization and is able to induce sparsity. Actually, if we let  $\kappa = \min_i\|\mathbf{c}_{/i}\|_1$ , we have  $\mathcal{R}(\mathbf{c}) \geq \frac{1}{2}\left(\tilde{\epsilon}\|\mathbf{c}\|_2^2 + \kappa\sum_{i=1}^{n}\bar{\epsilon}_ic_i\right) = \frac{1}{2}\left(\tilde{\epsilon}\|\mathbf{c}\|_2^2 + \kappa\|\mathrm{diag}(\tilde{\epsilon})\mathbf{c}\|\right)$ , where the second term is a weighted  $\ell_1$ -norm and also induces sparsity.
D Proof for Proposition 1
$$
\mathbf {Q} = \left[ \begin{array}{c c c} \mathbf {1} _ {n _ {1}} & \dots & \mathbf {0} \\ \vdots & \ddots & \vdots \\ \mathbf {0} & \dots & \mathbf {1} _ {n _ {N}} \end{array} \right], \quad \bar {\mathbf {K}} ^ {(L)} = \left[ \begin{array}{c c c} \mathbf {K} _ {1 1} ^ {(L)} & \dots & \mathbf {K} _ {1 N} ^ {(L)} \\ \vdots & \ddots & \vdots \\ \mathbf {K} _ {N 1} ^ {(L)} & \dots & \mathbf {K} _ {N N} ^ {(L)} \end{array} \right].
$$
It is easy to show that  $\mathbf{Q}$  is full-rank. Because  $k^{(L)}$  is a Gaussian kernel with  $0 < \sigma_L < \infty$ ,  $\bar{\mathbf{K}}^{(L)}$  is symmetric and always full-rank. It means  $\bar{\mathbf{K}}^{(L)}$  can be factorized as  $\bar{\mathbf{K}}^{(L)} = \mathbf{B}^{\top}\mathbf{B}$ , where  $\mathbf{B}$  is a square and full-rank matrix. Denote  $\mathbf{P} = \mathbf{B}\mathbf{Q}$ . It follows that
$$
\mathcal {K} _ {\mathbb {G}} ^ {(L)} = \mathbf {P} ^ {\top} \mathbf {P}. \tag {34}
$$
Since  $\mathbf{P}$  is full-rank, we conclude that  $\kappa_{\mathbb{G}}^{(L)}$  is full-rank, which means  $\kappa_{\mathbb{G}}^{(L)}$  is positive definite [Horn and Johnson, 2012].
E Detailed Derivations of GCKM-E
Theorem 2 (Bochner's theorem [Rudin, 1994]). A continuous kernel  $k(\mathbf{x}, \mathbf{x}^{\prime}) = k(\mathbf{x} - \mathbf{x}^{\prime})$  on  $\mathbb{R}^{m}$  is positive definite if and only if  $k(\delta)$  is the Fourier transform of a non-negative measure.
With positive definite and shift-invariant kernel  $k(\cdot, \cdot)$  (e.g. Gaussian kernel), the Bochner's theorem points out that the kernel function can be expanded with harmonic basis as
$$
k \left(\mathbf {x} _ {i} - \mathbf {x} _ {j}\right) = \int_ {\mathbb {R} ^ {d}} p (\boldsymbol {\omega}) e ^ {j \boldsymbol {\omega} ^ {\top} \left(\mathbf {x} _ {i} - \mathbf {x} _ {j}\right)} d \boldsymbol {\omega} = \mathbb {E} _ {\boldsymbol {\omega}} \left[ e ^ {j \boldsymbol {\omega} ^ {\top} \mathbf {x} _ {i}} e ^ {- j \boldsymbol {\omega} ^ {\top} \mathbf {x} _ {j}} \right], \tag {35}
$$
where  $p(\omega)$  is a probability distribution and also the Fourier transform of  $k(\delta)$ . For real-valued kernel functions,  $e^{j\omega^{\top}\mathbf{x}_i}e^{-j\omega^{\top}\mathbf{x}_j}$  can be simplified by
$$
e ^ {j \boldsymbol {\omega} ^ {\top} \mathbf {x} _ {i}} e ^ {- j \boldsymbol {\omega} ^ {\top} \mathbf {x} _ {j}} = \cos \boldsymbol {\omega} ^ {\top} \left(\mathbf {x} _ {i} - \mathbf {x} _ {j}\right) \tag {36}
$$
according to Euler's formula. Then we can define function  $\zeta_{\omega}(\cdot)$  with  $\zeta_{\omega}(\mathbf{x}_i)\zeta_{\omega}^{\top}(\mathbf{x}_j) = \cos \omega^{\top}(\mathbf{x}_i - \mathbf{x}_j)$ , so that it satisfies
$$
\mathbb {E} _ {\boldsymbol {\omega}} \left[ \zeta_ {\boldsymbol {\omega}} \left(\mathbf {x} _ {i}\right) \zeta_ {\boldsymbol {\omega}} ^ {\top} \left(\mathbf {x} _ {j}\right) \right] = \mathbb {E} _ {\boldsymbol {\omega}} \left[ e ^ {j \boldsymbol {\omega} ^ {\top} \mathbf {x} _ {i}} e ^ {- j \boldsymbol {\omega} ^ {\top} \mathbf {x} _ {j}} \right] = k \left(\mathbf {x} _ {i} - \mathbf {x} _ {j}\right) \tag {37}
$$
Here we simply adopt  $\zeta_{\omega}(\mathbf{x}) = \left[\cos (\boldsymbol{\omega}^{\top}\mathbf{x}),\sin (\boldsymbol{\omega}^{\top}\mathbf{x})\right]^{\top}$ , and the kernel function can be approximated by combining several  $\omega$  drawn from distribution  $p(\omega)$ .
Specifically, for Gaussian kernel, we first randomly sample  $\{\omega_1,\omega_2,\dots ,\omega_D\}$  from the probability distribution  $p_{\mathrm{RBF}}(\omega) = \mathcal{N}\big(0,\frac{1}{\sigma}\mathbf{I}\big)$ . Subsequently the explicit feature transformation on the  $i$ -th sample  $\mathbf{x}_i$  is defined as
$$
\psi \left(\mathbf {x} _ {i}\right) = \sqrt {\frac {1}{D}} \left[ \cos \left(\omega_ {1} ^ {\top} \mathbf {x} _ {i}\right), \dots , \cos \left(\omega_ {D} ^ {\top} \mathbf {x} _ {i}\right), \sin \left(\omega_ {1} ^ {\top} \mathbf {x} _ {i}\right), \dots , \sin \left(\omega_ {D} ^ {\top} \mathbf {x} _ {i}\right) \right] ^ {\top}. \tag {38}
$$
On the basis of the above derivations, the multi-layer GCKM with explicit feature transformation can be similarly defined:
$$
\mathbf {Z} ^ {(l + 1)} = \psi_ {(l)} (\hat {\mathbf {A}} \mathbf {Z} ^ {(l)}), \tag {39}
$$
where  $\mathbf{Z}^{(l + 1)}$  is the random Fourier features of input  $\mathbf{Z}^{(l)}$ . This approach allows GCKM to flexibly choose explicit node representations or kernel matrix as the interface, and the final representation can be low-dimensional and thus efficiently computed. By selecting a proper  $D$ , the convergence of this approximation and the expressive ability of the kernel can be simultaneously guaranteed.
F Detailed Experimental Settings and Results
F.1 Experimental Settings
Two types of graph datasets are employed in this paper. Each node-level dataset contains only a graph where each node is a sample, while each graph is regarded as a sample in the graph-level datasets. Here we list the key statistics of these datasets in Table 4 and 5 and provide detailed descriptions in the following.
Node-level Datasets
- Cora is a citation network containing a number of machine learning papers. Each node represents a paper and the edges represent the citation relationships between papers. Node features are the bag-of-words representations of papers, and all papers are divided into 7 categories according to their domains.
- Citeseer is also a well-known citation network with nodes, node features, and edges having the same meanings as Cora. Similarly, the papers are grouped into 6 classes.
- PubMed is a citation network composed of 19,717 scientific publications drawn from the PubMed database, which are classified into 3 categories and node features are also the bag-of-words representations.
- ACM is a paper network. Different from the above citation networks, the edges denote the co-author relationships between any two papers. Another difference is that the node features are bag-of-words representations of papers' keywords.
- CoraFull is a large version of the Cora dataset which has 19,793 papers of 70 classes.
- Chameleon is a webpage network, in which nodes are Wikipedia pages of certain topics and edges are the page-page hyperlinks. Some informative nouns in the pages are extracted to be node features and the nodes are organized in 5 categories based on the average monthly traffic of the corresponding web pages.
- Actor is a subgraph of the film-director-actor-writer network which only includes actors as nodes. An edge between two actors exists when they occur on the same Wikipedia page.
- Squirrel is a webpage network as well, and all the settings are similar to the Chameleon dataset.
- UAI is a webpage network that has been used to test GCN for community detection. Nodes represent web pages and each edge represents a citation between two pages.
Table 4: Dataset statistics.
Datasets	# Nodes	# Edges	# Classes	# Features	# Train/Val/Test	Data Type
Cora	2,708	5,429	7	1,433	140/500/1,000	Citation Network
Citeseer	3,327	4,732	6	3,703	120/500/1,000	Citation Network
Pubmed	19,717	44,338	3	500	60/500/1,000	Citation Network
ACM	3,025	13,128	3	1,870	60/500/1,000	Paper Network
Chameleon	2,277	36,101	3	2,325	60/500/1,000	Webpage Network
CoraFull	19,793	65,311	70	8,710	1,400/500/1,000	Citation Network
Actor	7,600	15,009	5	932	100/500/1,000	Social Network
Squirrel	5,201	217,073	3	2,089	60/500/.1000	Webpage Network
UAI	3,067	28,311	19	4,973	367/500/.1000	Webpage Network
Table 5: Dataset statistics.
IMDB-B	IMDB-M	COLLAB	MUTAG	PROTEINS	PTC
# Nodes	1,000	1,500	5,000	188	1,113	344
# Classes	2	3	3	2	2	2
Avg # Nodes	19.8	13.0	74.5	17.9	39.1	25.5
Data type	Movie	Movie	Scientific	Bioinformatics	Bioinformatics	Bioinformatics
- OGB-Arxiv dataset is a citation network with 169,343 computer science arXiv papers, where each node is an arXiv paper and each edge indicates that a paper cites another paper. Each paper has a 128-dimensional feature vector, which is obtained by averaging embeddings of words in its title and abstract.
Graph-level Datasets
- IMDB-BINARY and IMDB-MULTI are movie collaboration datasets, where each graph represents an ego-network for each actor/actress. Each node corresponds to an actor/actress, and the edges represent the actors/actresses' co-occurrences in a movie. Graphs are classified according to the genre of the movies they come from.
- COLLAB is a scientific collaboration dataset, collected from three public collaboration datasets. Similar to the two datasets mentioned above, each graph is a self-network of different researchers, and these graphs are classified by the domains to which their corresponding researchers belong.
- MUTAG is a bioinformatics dataset. 188 mutagenic aromatic and heteroaromatic nitro compounds with 7 labels compose this dataset.
- PROTEINS is a bioinformatics dataset where nodes denote secondary structure elements with 3 labels. An edge between any two nodes means they are neighbors in the amino-acid sequence or 3D space.
- PTC is a bioinformatics dataset containing 344 chemical compounds that report carcinogenicity for male and female rats and it has 19 discrete labels.
For datasets ACM, Chameleon, CoraFull, Actor, Squirrel, and UAI, we randomly split them into the train, validation, and test sets and fix them for all the compared methods. For graph-level datasets split, we follow settings in [Xu et al., 2019].
Then, we also briefly introduce the compared methods used in node-level tasks and graph-level tasks.
Node-level Compared Methods
- Chebyshev [Defferrard et al., 2016] generalizes the convolution operation to the non-Euclidean space based on spectral graph theory, which leverages Chebyshev polynomial to design a localized graph convolutional filter.
- GraphSAGE [Hamilton et al., 2017] is proposed to tackle the drawbacks of transductive GNNs. It trains the aggregation function from each node's local neighborhood, and this function can be performed on unseen nodes.
Table 6: Node classification accuracy (mean% and standard deviation%) of all methods, note that the best results are highlighted in orange and the second-best results are highlighted in blue. GraphSAGE encounters some errors on ACM and the corresponding result is marked as “—”.
ACM	Chameleon	CoraFull	Actor	Squirrel	UAI
Chebyshev	82.8 (1.4)	35.6 (0.2)	57.2 (1.1)	22.5 (0.4)	23.0 (0.6)	49.7 (0.4)
GraphSAGE	—	44.0 (0.0)	59.9 (0.7)	21.0 (0.5)	26.8 (0.2)	41.7 (1.4)
GAT	84.6 (0.5)	51.1 (1.0)	62.4 (0.4)	21.1 (0.7)	28.4 (0.6)	49.7 (3.0)
GCN	88.8 (0.5)	49.0 (1.8)	62.8 (0.4)	20.9 (0.9)	30.9 (1.8)	58.5 (1.1)
SGC	80.8 (2.7)	34.4 (1.1)	62.9 (2.2)	21.1 (1.3)	23.3 (1.1)	56.5 (3.5)
APPNP	88.2 (0.0)	50.4 (1.2)	63.1 (0.5)	22.6 (0.2)	27.1 (0.1)	62.3 (1.2)
JKNet	82.3 (0.6)	50.3 (1.2)	62.6 (0.0)	30.4 (0.6)	37.2 (1.0)	45.6 (0.5)
DAGNN	87.4 (0.9)	33.9 (3.9)	65.6 (0.3)	24.6 (1.7)	18.7 (2.7)	46.7 (12.4)
AdaGCN	88.7 (0.0)	43.8 (0.0)	63.8 (0.0)	24.9 (0.0)	25.7 (0.0)	47.3 (1.1)
AMGCN	90.4 (0.6)	34.2 (1.3)	52.6 (0.7)	26.8 (0.7)	22.6 (1.3)	59.3 (3.8)
DefGCN	86.9 (0.6)	49.5 (0.7)	42.6 (2.3)	31.3 (3.8)	28.4 (3.6)	56.4 (2.1)
GCKSVM	91.0 (0.0)	54.0 (0.0)	61.8 (0.0)	29.5 (0.0)	36.8 (0.0)	59.6 (0.0)
Table 7: Accuracy of several representative methods on large-scale dataset OGB-Arxiv, note that the best results are highlighted in orange and the second-best results are highlighted in blue.
OGB-Arxiv
# Nodes: 169,343	# Features: 128	# Edges: 1,166,243	# Classes: 40
Chebshev	GraphSAGE	GAT	GCN	SGC	APPNP	JKNet	GCKM-E
69.7 (0.2)	69.8 (0.2)	70.2 (0.2)	69.5 (0.1)	66.7 (0.0)	69.3 (0.1)	69.8 (0.2)	71.0 (0.0)
- GAT [Velickovic et al., 2018] is a classical spatial GNN equipping with self-attentional layers that adaptively learn different weights for edges.
- GCN [Kipf and Welling, 2017] is one of the most popular GNNs recently. It truncates the Chebyshev polynomial to first order and proposes a simple and effective graph convolution.
- SGC [Wu et al., 2019] further simplifies GCN by removing the nonlinearity and collapsing the learnable weights, so as to build a faster model and achieve competitive performance.
- APPNP [Klicpera et al., 2019] uses personalized Pagerank to improve vanilla GCN by solving the over-smoothing issue and derives a new framework with initial residual connections.
- JKNet [Xu et al., 2018] establishes a jumping knowledge structure to combine representations from various layers to stack a deeper GNN.
- DAGNN [Xu et al., 2023] decouples the neighbor aggregations and feature transformations to form a deeper GNN and further uses the attention mechanism to integrate the information from different depths.
- AdaGCN [Sun et al., 2021] is also a deep GCN that considers the Adaboost strategy to fuse knowledge from distinct layers.
- AMGCN [Wang et al., 2020] constructs a feature graph via  $k$ -NN algorithm and a multi-channel framework to enhance the vanilla GCN.
- DefGCN [Park et al., 2022] improves the limitations of common GNNs with fixed graph convolution and has deformable graph convolution that allows nodes to adaptively capture long-range dependencies.
Graph-level Compared Methods
- WL subtree kernel [Shervashidze et al., 2011] is a famous graph kernel, and AWL [Ivanov and Burnaev, 2018] is an approach for embedding entire graphs. Paired with SVM, they are selected as two baselines.
- DCNN [Atwood and Towsley, 2016], PATCHY-SAN [Niepert et al., 2016] and DGCNN [Zhang et al., 2018] are three deep learning methods for graph classification, and their performance is reported as in the original paper.
Figure 6: Accuracy of GCKSVM with varied layers/hops.
- GCN, GraphSAGE, and GIN [Xu et al., 2019] serve as GNN-based compared methods, and they are all set following [Xu et al., 2019]. For the methods having different variants, we report their best performance.
Note that the settings of these graph-level methods are the same as [Xu et al., 2019].