SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In International Conference on Machine Learning, pages 21–29. PMLR, 2019.
[2] Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society, pages 337-404, 1950.
[3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79:151-175, 2010.
[4] Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 2019.
[5] Karsten M Borgwardt and Hans-Peter Kriegel. Shortest-path kernels on graphs. In Fifth IEEE international conference on data mining (ICDM'05), pages 8-pp. IEEE, 2005.
[6] Viacheslav Borovitskiy, Iskander Azangulov, Alexander Terenin, Peter Mostowsky, Marc Deisenroth, and Nicolas Durrande. Matérn gaussian processes on graphs. In International Conference on Artificial Intelligence and Statistics, pages 2593-2601. PMLR, 2021.
[7] Bin Cao, Nathan N Liu, and Qiang Yang. Transfer learning for collective link prediction in multiple heterogenous domains. In International Conference on Machine Learning, pages 159-166, 2010.
[8] Bin Cao, Sinno Jialin Pan, Yu Zhang, Dit-Yan Yeung, and Qiang Yang. Adaptive transfer learning. In proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 407-412, 2010.
[9] Kian Chai. Generalization errors and learning curves for regression with multi-task gaussian processes. Advances in Neural Information Processing Systems, 2009.
[10] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pages 1725-1735. PMLR, 2020.
[11] Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021.
[12] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in Neural Information Processing Systems, 2009.
[13] Valerio Ciotti, Moreno Bonaventura, Vincenzo Nicosia, Pietro Panzarasa, and Vito Latora. Homophily and missing links in citation networks. EPJ Data Science, 5:1-14, 2016.
[14] Jinyuan Fang, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. Gaussian process with graph convolutional kernel for relational learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 353-363, 2021.
[15] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-2030, 2016.
[67] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In International Conference on Machine Learning, pages 7404-7413. PMLR, 2019.
[68] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33:7793-7804, 2020.
[69] Qi Zhu, Carl Yang, Yidan Xu, Haonan Wang, Chao Zhang, and Jiawei Han. Transfer learning of graph neural networks with ego-graph information maximization. Advances in Neural Information Processing Systems, 2021.
In the appendix, we provide more details regarding the proposed GraphGP for transferable graph learning in the paper, including
- In Subsections A.1 and A.2, we discuss the broader impacts and limitations of this paper.
- In Subsection A.3, we prove the theorems and corollaries stated in the paper.
- In Subsection A.4, we discuss the instantiations of our GraphGP algorithm with existing graph neural networks.
- In Subsection A.5, we derive the graph Gaussian Processes of existing Heterophilic GNNs, including LINKX [30], MixHop [1], H2GCN [68]. These derived Gaussian Processes are used as the baselines in our experiments (see Table 3).
- In Subsection A.6, we discuss the model efficiency of GraphGP and provide a computationally efficient approximation solution for GraphGP based on Nyström approximation.
- In Subsection A.7, we provide detailed experimental setups and additional evaluation results.
This paper focuses on the fundamental research problem of transferable graph learning. The goal is to effectively understand a target graph by leveraging latent knowledge from a relevant source graph. Generally, there are no negative societal impacts involved in this work.
A.2 Limitations
This paper develops a generic framework to build graph Gaussian processes for transferable graph learning under covariate shift assumption [40] over graphs. In addition to covariate shift, label shift (i.e.,  $\mathbb{P}_{\mathcal{Y}}$  shifts across domains) is also commonly considered in transfer learning scenarios. It is much more challenging to extend the developed transferable graph Gaussian processes to tackle label shift scenarios. This is because it is difficult to accurately estimate the graph domain distribution with limited label information from the target domain. In addition, following standard transfer learning [40], after model training, we simply apply the learned posterior function over testing target nodes for predicting their output values. It might be feasible to incorporate GraphGP with test-time adaptation techniques [51] to further improve the transfer performance.
A.3 Proof of Theorems and Corollaries
A.3.1 Proof of Theorem 4.1
Theorem 4.1 states that assume all the parameters of structure-aware graph neural network  $f(v, G)$  are independent and randomly drawn from Gaussian distributions, i.e.,  $\mathbf{W}^{(l)} \sim \mathcal{N}(\mathbf{0}, \sigma_w^2\mathbf{I})$ ,  $\mathbf{b}_{\mathrm{SANN}}^{(l)} \sim \mathcal{N}(\mathbf{0}, \varsigma_b^2\mathbf{I})$ ,  $\mathbf{W}_{\mathrm{SANN}}^{(l)} \sim \mathcal{N}(\mathbf{0}, \varsigma_w^2\mathbf{I})$ ,  $\tilde{\mathbf{b}}_{\mathrm{SANN}}^{(l)} \sim \mathcal{N}(\mathbf{0}, \tilde{\varsigma}_b^2\mathbf{I})$ ,  $\tilde{\mathbf{W}}_{\mathrm{SANN}}^{(l)} \sim \mathcal{N}(\mathbf{0}, \tilde{\varsigma}_w^2\mathbf{I})$ , when the layer width  $M$  goes to infinity, the output function  $f_i^{(l)}$  in Eq. (5) follows a Gaussian process with  $f_i^{(l)} \sim \mathcal{GP}(0, K^{(l)})$ , where the covariance function  $K^{(l)}$  is given by
$$
K ^ {(l)} \left((v, G), (v ^ {\prime}, G ^ {\prime})\right) = \sigma_ {w} ^ {2} \cdot K _ {\mu} ^ {(l)} \left(v, v ^ {\prime} | G, G ^ {\prime}\right) \cdot K _ {\nu} ^ {(l)} \left(G, G ^ {\prime}\right)
$$
$$
K _ {\mu} ^ {(l)} \left(v, v ^ {\prime} | G, G ^ {\prime}\right) = \sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot \sum_ {u \in N _ {i} (v)} \sum_ {u ^ {\prime} \in N _ {j} \left(v ^ {\prime}\right)} \mathbf {C} _ {u u ^ {\prime}} ^ {(l - 1)} \left(\varsigma_ {w} ^ {2}, \varsigma_ {b} ^ {2}\right)\right)
$$
Here  $\mathbf{P}^{(i)}(\mathbf{P}'^{(j)})$  denotes the adjacent matrix given by the  $i$ -order neighborhood from graph  $G$  ( $j$ -order neighborhood from graph  $G'$ ).  $K_{ab}^{(l-1)} = \sum_{i,j=0}^{k} \alpha_i \alpha_j'$  ( $b^2 + a^2 \mathbf{P}^{(i)} \mathbf{C}^{(l-2)}(a,b)(\mathbf{P}'^{(j)})^T$ ) and  $\mathbf{C}_{uu'}^{(0)}(a,b) = \langle \mathbf{x}_u, \mathbf{x}_{u'} \rangle$  for any  $u \in V, u' \in V'$ .
Proof. Given the definition of  $f(v, G)$ , we have
$$
\mathbf {h} _ {v} ^ {(l)} = \sum_ {i = 0} ^ {k} \alpha_ {i} \left(\frac {1}{\sqrt {M}} \sum_ {u \in N _ {i} (v)} \mathbf {W} _ {\text {S A N N}} ^ {(l - 1)} \mathbf {x} _ {u} ^ {(l)} + \mathbf {b} _ {\text {S A N N}} ^ {(l - 1)}\right) \quad \text {a n d} \quad \mathbf {x} _ {u} ^ {(l)} = \phi \left(\mathbf {h} _ {u} ^ {(l - 1)}\right)
$$
$$
\tilde {\mathbf {h}} _ {v} ^ {(l)} = \sum_ {i = 0} ^ {k} \alpha_ {i} \left(\frac {1}{\sqrt {M}} \sum_ {u \in N _ {i} (v)} \tilde {\mathbf {W}} _ {\text {S A N N}} ^ {(l - 1)} \tilde {\mathbf {x}} _ {u} ^ {(l)} + \tilde {\mathbf {b}} _ {\text {S A N N}} ^ {(l - 1)}\right) \quad \text {a n d} \quad \tilde {\mathbf {x}} _ {u} ^ {(l)} = \phi \left(\tilde {\mathbf {h}} _ {u} ^ {(l - 1)}\right)
$$
where  $\mathbf{x}_u^{(0)} = \tilde{\mathbf{x}}_u^{(0)} = \mathbf{x}_u$  denote the initial node attributes. The  $l$ -th layer of the function  $f(v, G)$  is defined as
$$
f _ {i} ^ {(l)} (v, G) = \frac {1}{\sqrt {M}} \sum_ {j = 1} ^ {M} \mathbf {W} _ {i j} ^ {(l)} \cdot \mu_ {j} ^ {(l)} (v | G) \cdot \nu_ {j} ^ {(l)} (G)
$$
$$
\text {w h e r e} \quad \mu^ {(l)} (v | G) = \mathbf {h} _ {v} ^ {(l)} \quad \text {a n d} \quad \nu^ {(l)} (G) = \frac {1}{| V |} \sum_ {v \in V} \tilde {\mathbf {h}} _ {v} ^ {(l)}
$$
Thus, we obtain  $\mathbb{E}[f_i^{(l)}(v,G)] = 0$  and
$$
\begin{array}{l} K ^ {(l)} \left(\left(v, G\right), \left(v ^ {\prime}, G ^ {\prime}\right)\right) \equiv \mathbb {E} \left[ f _ {i} ^ {(l)} (v, G) f _ {i} ^ {(l)} \left(v ^ {\prime}, G ^ {\prime}\right) \right] \\ = \sigma_ {w} ^ {2} \cdot \mathbb {E} \left[ \mu_ {i} ^ {(l)} (v | G) \cdot \mu_ {i} ^ {(l)} (v ^ {\prime} | G ^ {\prime}) \right] \cdot \mathbb {E} \left[ \nu_ {i} ^ {(l)} (G) \cdot \nu_ {i} ^ {(l)} (G ^ {\prime}) \right] \\ \end{array}
$$
$$
\begin{array}{l} K _ {\mu} ^ {(l)} (v, v ^ {\prime} | G, G ^ {\prime}) \equiv \mathbb {E} \left[ \mu_ {i} ^ {(l)} (v | G) \cdot \mu_ {i} ^ {(l)} (v ^ {\prime} | G ^ {\prime}) \right] \\ = \mathbb {E} \left[ \left[ \mathbf {h} _ {v} ^ {(l)} \right] _ {i} \cdot \left[ \mathbf {h} _ {v ^ {\prime}} ^ {(l)} \right] _ {i} \right] \\ = \sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot \mathbb {E} \left[ \sum_ {u \in N _ {i} (v)} \left[ \mathbf {x} _ {u} ^ {(l)} \right] _ {i} \sum_ {u ^ {\prime} \in N _ {j} \left(v ^ {\prime}\right)} \left[ \mathbf {x} _ {u ^ {\prime}} ^ {(l)} \right] _ {i} \right]\right) \\ = \sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot \sum_ {u \in N _ {i} (v)} \sum_ {u ^ {\prime} \in N _ {j} (v ^ {\prime})} \mathbb {E} \left[ \left[ \mathbf {x} _ {u} ^ {(l)} \right] _ {i} \left[ \mathbf {x} _ {u ^ {\prime}} ^ {(l)} \right] _ {i} \right]\right) \\ = \sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot \sum_ {u \in N _ {i} (v)} \sum_ {u ^ {\prime} \in N _ {j} (v ^ {\prime})} \mathbf {C} _ {u u ^ {\prime}} ^ {(l - 1)} \left(\varsigma_ {w} ^ {2}, \varsigma_ {b} ^ {2}\right)\right) \\ \end{array}
$$
where  $[\mathbf{a}]_i$  denotes the  $i$ -th entry of vector  $\mathbf{a}$ . Besides,
$$
\begin{array}{l} K _ {\nu} ^ {(l)} (G, G ^ {\prime}) \equiv \mathbb {E} \left[ \nu_ {i} ^ {(l)} (G) \cdot \nu_ {i} ^ {(l)} (G ^ {\prime}) \right] \\ = \mathbb {E} \left[ \frac {1}{| V |} \sum_ {v \in V} \left[ \tilde {\mathbf {h}} _ {v} ^ {(l)} \right] _ {i} \cdot \frac {1}{| V ^ {\prime} |} \sum_ {v ^ {\prime} \in V ^ {\prime}} \left[ \tilde {\mathbf {h}} _ {v ^ {\prime}} ^ {(l)} \right] _ {i} \right] \\ = \frac {1}{| V | \cdot | V ^ {\prime} |} \sum_ {v \in V} \sum_ {v ^ {\prime} \in V ^ {\prime}} \mathbb {E} \left[ \left[ \tilde {\mathbf {h}} _ {v} ^ {(l)} \right] _ {i} \cdot \left[ \tilde {\mathbf {h}} _ {v ^ {\prime}} ^ {(l)} \right] _ {i} \right] \\ = \frac {1}{| V | \cdot | V ^ {\prime} |} \sum_ {v \in V} \sum_ {v ^ {\prime} \in V ^ {\prime}} \left(\sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\tilde {\varsigma} _ {b} ^ {2} + \tilde {\varsigma} _ {w} ^ {2} \cdot \sum_ {u \in N _ {i} (v)} \sum_ {u ^ {\prime} \in N _ {j} (v ^ {\prime})} \mathbf {C} _ {u u ^ {\prime}} ^ {(l - 1)} (\tilde {\varsigma} _ {w} ^ {2}, \tilde {\varsigma} _ {b} ^ {2})\right)\right) \\ = \sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\tilde {\varsigma} _ {b} ^ {2} + \frac {\tilde {\varsigma} _ {w} ^ {2}}{| V | \cdot | V ^ {\prime} |} \mathbf {1} ^ {T} \mathbf {P} ^ {(i)} \mathbf {C} ^ {(l - 1)} \left(\tilde {\varsigma} _ {w} ^ {2}, \tilde {\varsigma} _ {b} ^ {2}\right) \left(\mathbf {P} ^ {\prime (j)}\right) ^ {T} \mathbf {1}\right) \\ \end{array}
$$
A.3.2 Proof of Positive Definiteness
With the assumptions in Theorem 4.1, the covariance kernel  $K^{(l)}$  is positive definite.
Proof. This corollary can be proven using the Schur Product theorem that the Hadamard product of two positive semidefinite matrices is also a positive semidefinite matrix. Based on the definition of  $(v,v^{\prime}|G,G^{\prime})$  and  $K_{\nu}^{(l)}(G,G^{\prime})$ , the covariance matrix of these kernels over all nodes in  $G,G^{\prime}$  are positive semidefinite. Then as explained in [37], a kernel function is positive definite (resp. strictly positive definite) if the corresponding kernel matrix is positive semi-definite (resp. positive definite) for any collection of distinct points. Therefore, the kernel function  $K^{(l)}((v,G),(v^{\prime},G^{\prime}))$  is symmetric and positive definite. Then based on Moore-Aronszajn theorem [2],  $K^{(l)}$  defines a unique reproducing kernel Hilbert space (RKHS).
A.3.3 Proof of Corollary 4.2
Corollary 4.2 states let  $\varphi_{1}:\mathcal{H}\to \mathcal{H}_{1}$  denote the kernel mapping from a pre-activation RKHS  $\mathcal{H}$  to post-activation RKHS  $\mathcal{H}_1$ , i.e.,  $\langle \varphi_1(s),\varphi_1(s')\rangle = \frac{1}{2} ||s||\cdot ||s'||\cdot \kappa_1\left(\frac{\langle s,s'\rangle}{||s||\cdot||s'||}\right)$ . Given the sample-level kernel  $K_{\mu}^{(l)}(v,v'|G,G')$  in Theorem 4.1, if the graph convolutional layer of Eq. (3) has no bias term  $(\varsigma_{b} = 0)$ , the feature map of this kernel is given by
$$
\Psi_ {v} ^ {(l)} = \varsigma_ {w} \sum_ {i = 0} ^ {k} \alpha_ {i} \sum_ {u \in N _ {i} (v)} \varphi_ {1} \left(\Psi_ {u} ^ {(l - 1)}\right) \quad \text {a n d} \quad \Psi_ {v ^ {\prime}} ^ {(l)} = \varsigma_ {w} \sum_ {i = 0} ^ {k} \alpha_ {i} ^ {\prime} \sum_ {u ^ {\prime} \in N _ {i} (v ^ {\prime})} \varphi_ {1} \left(\Psi_ {u ^ {\prime}} ^ {(l - 1)}\right)
$$
Proof. Based on Theorem A.1, we have
$$
\begin{array}{l} \mathbf {C} _ {v v ^ {\prime}} ^ {(l - 1)} = \frac {\kappa_ {1} (\zeta)}{2} \sqrt {K _ {\mu} ^ {(l - 1)} (v , v | G) \cdot K _ {\mu} ^ {(l - 1)} (v ^ {\prime} , v ^ {\prime} | G ^ {\prime})} \\ = \left\langle \varphi_ {1} \left(\Psi_ {v} ^ {(l - 1)}\right), \varphi_ {1} \left(\Psi_ {v ^ {\prime}} ^ {(l - 1)}\right) \right\rangle \\ \end{array}
$$
Then, with  $\varsigma_{b} = 0$ , we have
$$
\begin{array}{l} K _ {\mu} ^ {(l)} \left(v, v ^ {\prime} | G, G ^ {\prime}\right) = \sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\varsigma_ {w} ^ {2} \cdot \sum_ {u \in N _ {i} (v)} \sum_ {u ^ {\prime} \in N _ {j} \left(v ^ {\prime}\right)} \mathbf {C} _ {u u ^ {\prime}} ^ {(l - 1)} \left(\varsigma_ {w} ^ {2}, \varsigma_ {b} ^ {2}\right)\right) \\ = \sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\varsigma_ {w} ^ {2} \cdot \sum_ {u \in N _ {i} (v)} \sum_ {u ^ {\prime} \in N _ {j} \left(v ^ {\prime}\right)} \left\langle \varphi_ {1} \left(\Psi_ {u} ^ {(l - 1)}\right), \varphi_ {1} \left(\Psi_ {u ^ {\prime}} ^ {(l - 1)}\right) \right\rangle\right) \\ = \left\langle \varsigma_ {w} \sum_ {i = 0} ^ {k} \alpha_ {i} \sum_ {u \in N _ {i} (v)} \varphi_ {1} \left(\Psi_ {u} ^ {(l - 1)}\right), \quad \varsigma_ {w} \sum_ {j = 0} ^ {k} \alpha_ {j} ^ {\prime} \sum_ {u ^ {\prime} \in N _ {j} (v ^ {\prime})} \varphi_ {1} \left(\Psi_ {u ^ {\prime}} ^ {(l - 1)}\right) \right\rangle \\ \end{array}
$$
$$
\Psi_ {v} ^ {(l)} = \varsigma_ {w} \sum_ {i = 0} ^ {k} \alpha_ {i} \sum_ {u \in N _ {i} (v)} \varphi_ {1} \left(\Psi_ {u} ^ {(l - 1)}\right)
$$
$$
\Psi_ {v ^ {\prime}} ^ {(l)} = \varsigma_ {w} \sum_ {j = 0} ^ {k} \alpha_ {j} ^ {\prime} \sum_ {u ^ {\prime} \in N _ {j} (v ^ {\prime})} \varphi_ {1} \left(\Psi_ {u ^ {\prime}} ^ {(l - 1)}\right)
$$
This result provides the feature map for nodes from graph  $G$  and  $G'$  separately. But we show that it is equivalent to the following unified feature mapping on a large graph integrating  $G$  and  $G'$ .
For simplicity, we show the results without bias term, but it can be easily generalized to Eq. (3) with bias by absorbing the bias term in the kernel mapping.
More specifically, graphs  $G = (V,E)$  and  $G' = (V',E')$  can be considered as a single large graph  $G'' = (V \cup V', E \cup E')$  with two disjoint components. Then, for any node  $v'' \in V''$  in graph  $G''$ , we have
$$
\Psi_ {v ^ {\prime \prime}} ^ {(l)} = \varsigma_ {w} \sum_ {i = 0} ^ {k} (\alpha_ {i} \cdot \mathbb {I} [ v ^ {\prime \prime} \in V ] + \alpha_ {i} ^ {\prime} \cdot \mathbb {I} [ v ^ {\prime \prime} \in V ^ {\prime} ]) \sum_ {u ^ {\prime \prime} \in N _ {j} (v ^ {\prime \prime})} \varphi_ {1} (\Psi_ {u ^ {\prime \prime}} ^ {(l - 1)})
$$
where  $v''$  can be a node from either  $G$  or  $G'$ , and  $\mathbb{I}[a] = 1$  if an event  $a$  is true,  $\mathbb{I}[a] = 0$  otherwise.
A.3.4 Proof of Corollary 4.4
Corollary 4.4 states that with the same conditions in Theorem 4.1, for each  $l$ , in the limit on the layer width, the graph representation  $\nu^{(l)}(G)$  recovers the empirical mean embedding  $\hat{\tau}_{\mathbb{P}_{\mathcal{G}}}^{(l)}$  of the domain distribution  $\mathbb{P}_{\mathcal{G}}$  in the reproducing kernel Hilbert space induced by  $K_{\nu}^{(l)}$ , given.
$$
\hat {\tau} _ {\mathbb {P} _ {\mathcal {G}}} = \frac {1}{| V |} \sum_ {v \in V} \tilde {\Psi} _ {v} ^ {(l)}
$$
Proof. We derive the feature map of domain-level kernel  $K_{\nu}^{(l)}$  as follows. Similarly, we consider the case with  $\tilde{\varsigma}_b = 0$ , we have
$$
\begin{array}{l} K _ {\nu} ^ {(l)} \left(G, G ^ {\prime}\right) = \sum_ {i, j = 0} ^ {k} \alpha_ {i} \alpha_ {j} ^ {\prime} \left(\frac {\tilde {\varsigma} _ {w} ^ {2}}{| V | \cdot | V ^ {\prime} |} \mathbf {1} ^ {T} \mathbf {P} ^ {(i)} \mathbf {C} ^ {(l - 1)} \left(\tilde {\varsigma} _ {w} ^ {2}, \tilde {\varsigma} _ {b} ^ {2}\right) \left(\mathbf {P} ^ {\prime (j)}\right) ^ {T} \mathbf {1}\right) \\ = \left\langle \frac {\tilde {\varsigma} _ {w}}{| V |} \sum_ {v \in V} \sum_ {i = 0} ^ {k} \alpha_ {i} \sum_ {u \in N _ {i} (v)} \varphi_ {1} \left(\tilde {\Psi} _ {u} ^ {(l - 1)}\right), \frac {\tilde {\varsigma} _ {w}}{| V ^ {\prime} |} \sum_ {v ^ {\prime} \in V ^ {\prime}} \sum_ {j = 0} ^ {k} \alpha_ {j} ^ {\prime} \sum_ {u ^ {\prime} \in N _ {j} (v ^ {\prime})} \varphi_ {1} \left(\tilde {\Psi} _ {u ^ {\prime}} ^ {(l - 1)}\right) \right\rangle \\ \end{array}
$$
That is, given the node representation  $\tilde{\Psi}_v^{(l)} = \tilde{\varsigma}_w\sum_{i = 0}^k\alpha_i\sum_{u\in N_i(v)}\varphi_1\left(\tilde{\Psi}_u^{(l - 1)}\right)$  in the kernel space, the mean pooling in the READOUT function (see Eq.(2)) results in the mean embedding of empirical node distribution over  $\{\tilde{\Psi}_v^{(l)}\}_{v\in V}$ , i.e.,
Moreover, when the sample size of graph  $G$  goes to infinity, the expected mean embedding of node distribution over  $\{\tilde{\Psi}_v^{(l)}\}_{v\in V}$  is given by
$$
\tau_ {\mathbb {P} _ {\mathcal {G}}} = \mathbb {E} _ {v} \left[ \tilde {\Psi} _ {v} ^ {(l)} \right] = \mathbb {E} _ {v} \left[ K _ {\nu} ^ {(l)} (\cdot , v | G) \right] = \mathbb {E} _ {v} \left[ K _ {\nu} ^ {(l)} (\cdot , v | \mathbb {P} _ {\mathcal {G}}) \right]
$$
which corresponds to expression of graph domain distribution  $\mathbb{P}_{\mathcal{G}}$  in the reproducing kernel Hilbert space induced by  $K_{\nu}^{(l)}$  (given by the kernel mapping  $\tilde{\Psi}_v^{(l)})$ .
A.3.5 Proof of Theorem 4.5
Theorem 4.5 states that let  $\mathbf{K}_{ss}^{\mu}$  be the sample-level covariance matrix over source nodes, i.e.,  $[\mathbf{K}_{ss}^{\mu}]_{(v_s,v_s')} = K_\mu^{(L)}(v_s,v_s'|G_s)$ ,  $\nu_{ss} = K_\nu^{(L)}(G_s,G_s)$ ,  $\nu_{tt} = K_\nu^{(L)}(G_t,G_t)$  be the intra-graph kernels and  $\nu_{st} = K_\nu^{(L)}(G_s,G_t)$  be the inter-graph kernel. Suppose  $\bar{\varrho}_s^2\stackrel {\Delta}{=}\left(\frac{\nu_{ss}\cdot\nu_{tt}}{\nu_{st}\cdot\nu_{st}} -1\right)\sigma_w^2\nu_{tt}\bar{\lambda}_{ss}+$ $\frac{\nu_{tt}\cdot\nu_{tt}}{\nu_{st}\cdot\nu_{st}}\varrho_s^2$  where  $\bar{\lambda}_{ss}$  is the maximum eigenvalue of  $\mathbf{K}_{ss}^{\mu}$ , for any  $(v_t,G_t)$  the generalization error is bounded by
where  $\Gamma_t(v_t, G_t; \bar{\varrho}_s^2, \varrho_t^2)$  is the variance assuming that all source examples are observed in the target domain with respect to noises  $\bar{\varrho}_s^2, \varrho_t^2$ .
Proof. The covariance matrix  $\mathbf{K}$  over training samples can be rewritten as
$$
\begin{array}{l} \mathbf {K} = \left[ \begin{array}{c c} \mathbf {K} _ {s s} & \mathbf {K} _ {s t} \\ \mathbf {K} _ {t s} & \mathbf {K} _ {t t} \end{array} \right] = \left[ \begin{array}{c c} \sigma_ {w} ^ {2} \nu_ {s s} \mathbf {K} _ {s s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {s t} \mathbf {K} _ {s t} ^ {\mu} \\ \sigma_ {w} ^ {2} \nu_ {s t} \mathbf {K} _ {t s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t t} ^ {\mu} \end{array} \right] \\ = \left[ \begin{array}{c c} \frac {\nu_ {s t}}{\nu_ {t t}} \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \mathbf {I} \end{array} \right] \left[ \begin{array}{c c} \frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {t t}} \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s t} ^ {\mu} \\ \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t t} ^ {\mu} \end{array} \right] \left[ \begin{array}{c c} \frac {\nu_ {s t}}{\nu_ {t t}} \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \mathbf {I} \end{array} \right] \\ \end{array}
$$
$$
\begin{array}{l} \Gamma (v _ {t}, G _ {t}) = \mathbf {k} _ {* *} - \mathbf {k} _ {*} \left(\mathbf {K} _ {s s} + \varrho_ {s} ^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {k} _ {*} ^ {T} \\ = \mathbf {k} _ {* *} - \mathbf {k} _ {*} ^ {t} (\Sigma) ^ {- 1} \left(\mathbf {k} _ {*} ^ {t}\right) ^ {T} \\ \end{array}
$$
Following [9], we have
$$
\Delta \left(v _ {t}, G _ {t}\right) = \mathbf {k} _ {*} ^ {t} \left(\Sigma_ {t} ^ {- 1} - \Sigma^ {- 1}\right) \left(\mathbf {k} _ {*} ^ {t}\right) ^ {T}
$$
$$
\Sigma = \left[ \begin{array}{c c} \frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s t} ^ {\mu} \\ \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t t} ^ {\mu} \end{array} \right] + \left[ \begin{array}{c c} \frac {\nu_ {t t} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \varrho_ {s} ^ {2} \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \varrho_ {t} ^ {2} \mathbf {I} \end{array} \right]
$$
$$
\Sigma_ {t} = \left[ \begin{array}{c c} \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s t} ^ {\mu} \\ \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t t} ^ {\mu} \end{array} \right] + \left[ \begin{array}{c c} s ^ {2} \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \varrho_ {t} ^ {2} \mathbf {I} \end{array} \right]
$$
Thus, it holds
$$
\Delta (v _ {t}, G _ {t}) \leq 0
$$
$$
\Sigma_ {t} ^ {- 1} - \Sigma^ {- 1} \preceq 0
$$
$$
\Sigma_ {t} \succeq \Sigma
$$