SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Corollary 4.2. Let  $\varphi_{1}:\mathcal{H}\to \mathcal{H}_{1}$  denote the kernel mapping from a pre-activation RKHS  $\mathcal{H}$  to postactivation RKHS  $\mathcal{H}_1$ , i.e.,  $\langle \varphi_1(s),\varphi_1(s')\rangle = \frac{1}{2} ||s||\cdot ||s'||\cdot \kappa_1\left(\frac{\langle s,s'\rangle}{||s||\cdot||s'||}\right)$ . Given the sample-level kernel  $K_{\mu}^{(l)}(v,v'|G,G')$  in Theorem 4.1, if the graph convolutional layer of Eq. (3) has no bias term  $(s_b = 0)$ , the feature map of this kernel is given by
$$
\Psi_ {v} ^ {(l)} = \varsigma_ {w} \sum_ {i = 0} ^ {k} \alpha_ {i} \sum_ {u \in N _ {i} (v)} \varphi_ {1} \left(\Psi_ {u} ^ {(l - 1)}\right) \quad a n d \quad \Psi_ {v ^ {\prime}} ^ {(l)} = \varsigma_ {w} \sum_ {i = 0} ^ {k} \alpha_ {i} ^ {\prime} \sum_ {u ^ {\prime} \in N _ {i} (v ^ {\prime})} \varphi_ {1} \left(\Psi_ {u ^ {\prime}} ^ {(l - 1)}\right)
$$
with  $K_{\mu}^{(l)}(v,v^{\prime}|G,G^{\prime}) = \langle \Psi_{v}^{(l)},\Psi_{v^{\prime}}^{(l)}\rangle$
Corollary 4.2 shows that the feature map  $\Psi_v^{(l)}$  of sample-level kernel  $K_{\mu}^{(l)}$  is a kernelized version of the graph convolutional layer in Eq. (3), which recursively aggregates message from nodes' neighborhood in the kernel space. The neighborhood can be automatically selected by optimizing the hyper-parameters  $\alpha_{i}$  and  $\alpha_{i}^{\prime}$ .
Next, we show the implication of domain-level kernel  $K_{\nu}^{(l)}(G,G^{\prime})$ . As discussed in Subsection 4.1, a graph  $G$ , empirically drawn from graph domain distribution  $\mathbb{P}_{\mathcal{G}}$ , can approximate the true distribution  $\mathbb{P}_{\mathcal{G}}$  when increasing the size of graph  $G$ . This motivates us to understand the connection between empirical domain representation  $\nu^{(l)}(G)$  and the embedding of probability distribution  $\mathbb{P}_{\mathcal{G}}$  [19, 48].
Definition 4.3 (Mean Embedding of Probability Distributions [19]). Given an RKHS  $\mathcal{H}$  induced by a kernel  $k(\cdot, \cdot)$ , the mean embedding of a probability distribution  $\mathbb{P}$  is defined as  $\tau_{\mathbb{P}} = \int k(\cdot, \mathbf{x}) d\mathbb{P}(\mathbf{x})$ , i.e.,  $\mathbb{E}_{\mathbf{x} \sim \mathbb{P}}[f(\mathbf{x})] = \langle f, \tau_{\mathbb{P}} \rangle_{\mathcal{H}}$  for all  $f \in \mathcal{H}$ . Furthermore, the empirical mean embedding of  $\mathbb{P}$  is given by  $\hat{\tau}_{\mathbb{P}} = \frac{1}{m} \sum_{i=1}^{m} k(\cdot, \mathbf{x}_i)$ , where  $m$  samples are independent and identically drawn from  $\mathbb{P}$ .
Analogously, the mean embedding of a graph domain distribution  $\mathbb{P}_{\mathcal{G}}$  can also be defined as  $\tau_{\mathbb{P}_{\mathcal{G}}} = \int k_{\mathcal{G}}(\cdot ,v|\mathbb{P}_{\mathcal{G}})d\mathbb{P}_{\mathcal{G}}(v)$  given an RKHS  $\mathcal{H}_{\mathcal{G}}$  induced by a kernel  $k_{\mathcal{G}}(\cdot ,\cdot)$ . In the context of graph learning,  $v|\mathbb{P}_{\mathcal{G}}$  indicates the individual node attributes of  $v$  and graph structure information around node  $v$  induced by  $\mathbb{P}_{\mathcal{G}}$ . It subsumes the conventional mean embedding [19] of a probability distribution  $\tau_{\mathbb{P}}$  with independent and identically distributed (IID) samples  $\mathbf{x}\sim \mathbb{P}$ , if there are no edges in the graph (i.e., no graph structure is involved around node  $v$ ). It is observed that the additional graph structure challenges empirical estimation of graph domain distribution  $\mathbb{P}_{\mathcal{G}}$ , thereby resulting in a nontrivial domain similarity estimator between source and target graphs in transferable graph learning. The following corollary states that the graph representation  $\nu^{(l)}(G)$  can be considered as the empirical mean embedding of  $\tau_{\mathbb{P}_{\mathcal{G}}}$  in the RKHS induced by  $K_{\nu}^{(l)}(\cdot ,\cdot)$ .
Corollary 4.4. With the same conditions in Theorem 4.1, for each  $l$ , in the limit on the layer width, the graph representation  $\nu^{(l)}(G)$  recovers the empirical mean embedding  $\hat{\tau}_{\mathbb{P}_G}^{(l)}$  of the domain distribution  $\mathbb{P}_{\mathcal{G}}$  in the reproducing kernel Hilbert space induced by  $K_{\nu}^{(l)}$ , and  $\hat{\tau}_{\mathbb{P}_G}^{(l)}$  is given by
$$
\hat {\tau} _ {\mathbb {P} _ {\mathcal {G}}} = \frac {1}{| V |} \sum_ {v \in V} \tilde {\Psi} _ {v} ^ {(l)} = \frac {1}{| V |} \sum_ {v \in V} K _ {\nu} ^ {(l)} (\cdot , v | G)
$$
where  $\tilde{\Psi}_v^{(l)}$  is the feature map of  $K_{\nu}^{(l)}$  with  $\tilde{\Psi}_v^{(l)} = \tilde{\varsigma}_w\sum_{i = 0}^k\alpha_i\sum_{u\in N_i(v)}\varphi_1\left(\tilde{\Psi}_u^{(l - 1)}\right)$ .
Corollary 4.4 shows that  $K_{\nu}^{(l)}(G,G^{\prime})$  in Theorem 4.1 can be explained as the similarity of the empirical mean embeddings of domain distributions  $\mathbb{P}_{\mathcal{G}}$  and  $\mathbb{P}_{\mathcal{G}'}$  in the kernel space e.g.,  $K_{\nu}^{(l)}(G,G^{\prime}) = \langle \hat{\tau}_{\mathbb{P}_{\mathcal{G}}}^{(l)},\hat{\tau}_{\mathbb{P}_{\mathcal{G}^{\prime}}}^{(l)}\rangle_{\mathcal{H}_{K_{\nu}^{(l)}}}$ . A similar metric over distributions is the maximum mean discrepancy (MMD) [19] defined over the distance of empirical mean embeddings, i.e.,  $\mathrm{MMD}_{\nu}^{(l)} = ||\hat{\tau}_{\mathbb{P}_{\mathcal{G}}}^{(l)} - \hat{\tau}_{\mathbb{P}_{\mathcal{G}^{\prime}}}^{(l)}||_{\mathcal{H}_{K_{\nu}^{(l)}}}$ , where  $\mathcal{H}_{K_{\nu}^{(l)}}$  denotes the RKHS induced by  $K_{\nu}^{(l)}$ . MMD has been widely applied to transfer learning [31, 32, 61] for measuring the distribution shift across image domains, under the covariate shift assumption [40] (i.e., source and target domains share conditional distribution  $p(y|x)$  but different marginal distributions  $p(x)$ ). In contrast, in this paper, we focus on measuring the domain distribution similarity for transferable graph learning.
4.2.2 Homophily vs. Heterophily
As illustrated in Section 3.2, there are two assumptions in graph learning [20, 68]: homophily and heterophily. We show the special cases of Theorem 4.1 in tackling homophily or heterophily based transferable graph learning. Given homophily source graph  $G$  and target graph  $G'$  where connected nodes have similar output values, it is revealed [20, 25, 59] that each node aggregates message from itself and its 1-order neighborhood, e.g.,  $k = 1$  and  $\alpha_0 = \alpha_1 = \alpha_0' = \alpha_1' = 1$ . Then the sample-level kernel  $K_{\mu}^{(l)}(v,v'|G,G') \propto \sum_{u\in \{v\cup N_1(v)\}}\sum_{u'\in \{v'\cup N_1(v')\}}\mathbf{C}_{uu'}^{(l - 1)}$ . In contrast, if source and target graphs follow heterophily that connected nodes are not similar in the output space, nodes might aggregate message from high-order neighborhood [1, 52, 68], e.g.,  $k = 2$  and  $\alpha_0 = \alpha_2 = 1, \alpha_1 = 0$ . Then it holds that  $K_{\mu}^{(l)}(v,v'|G,G') \propto \sum_{u\in \{v\cup N_2(v)\}}\sum_{u'\in \{v'\cup N_2(v')\}}\mathbf{C}_{uu'}^{(l - 1)}$ . These results indicate that by learning the neighborhood importances  $\alpha_i$  and  $\alpha_i'$ , Gaussian process led by  $f(v,G)$  can adaptively select the neighborhood for message aggregation from homophily or heterophily graphs. Moreover,  $\alpha_i = \alpha_i'$  implies that source and target graphs follow the same assumption. The flexibility of  $\alpha_i, \alpha_i'$  allows us to transfer knowledge across different types of graphs. For example, using  $k = 2, \alpha_0 = \alpha_1 = 1, \alpha_2 = 0, \alpha_0' = \alpha_2' = 0, \alpha_1' = 0$ , it enables the knowledge transferability from a homophily source graph to a heterophily target graph.
4.3 Proposed Algorithms
The goal of transferable node regression is to learn the prediction function for the target graph, by leveraging the input-output relationship from a relevant source graph. Given a source graph  $G_{s} = (V_{s},E_{s})$  with fully labeled nodes (e.g.,  $y_{v}\in \mathbb{R}$  is associated with each node  $v\in V_{s}$ ) and a target graph  $G_{t} = (V_{t},E_{t})$  with a limited number of labeled nodes (e.g.,  $|V_t^{la}|\ll |V_s|$  where  $V_{t}^{la}\subset V_{t}$  is the set of labeled target nodes), we propose the adaptive graph Gaussian process algorithm (termed as GraphGP) as follows.
For notation simplicity, we let  $f_{v} = f^{(L)}(v,G)$  (given by  $L$  graph convolutional layers in Eq. (5)) be the function value at node  $v$ . Let  $\mathbf{f}_s = [f_1,f_2,\dots ,f_{|V_s}]^T$  be a vector of latent function values over labeled source nodes, and  $\mathbf{y}_s = [y_1,y_2,\dots ,y_{|V_s}]^T$  be the associated ground-truth output values. Similarly, we can define  $\mathbf{f}_t$  and  $\mathbf{y}_t$  over target nodes. Then, the GP prior over function values can be defined as  $f\sim \mathcal{GP}(0,K^{(L)}(\cdot ,\cdot))$  (defined in Theorem 4.1), and its instantiation at labeled training nodes is given by  $p\left(\mathbf{f}|V_s\cup V_t^{la}\right) = \mathcal{N}\left(\mathbf{0},\mathbf{K}_{(s + t)(s + t)}\right)$  where  $\mathbf{K}_{(s + t)(s + t)}$  is a block matrix, i.e.,  $\mathbf{K}_{(s + t)(s + t)} = \begin{bmatrix} \mathbf{K}_{ss} & \mathbf{K}_{st}\\ \mathbf{K}_{ts} & \mathbf{K}_{tt} \end{bmatrix}$  and its entry is  $[\mathbf{K}_{ab}]_{va}v_b = K^{(L)}((v_a,G_a),(v_b,G_b))$  for  $a,b\in \{s,t\}$ . For the likelihood, we consider the noisy scenarios where  $p(\mathbf{y}_a|\mathbf{f}_a) = \mathcal{N}(\mathbf{f}_a,\varrho_a^2\mathbf{I})$  for  $a\in \{s,t\}$  ( $\varrho_{a}$  measures the noisy magnitude). Then, following standard Gaussian process [42], the posterior distribution of GraphGP over testing nodes  $V_{*}\subset V_{t}$  has a closed-form expression, i.e.,  $p(\mathbf{f}_*|V_*,\mathbf{f},V_s\cup V_t^{la}) = \mathcal{N}(\boldsymbol {\gamma},\boldsymbol {\Gamma})$  where
$$
\boldsymbol {\gamma} = \mathbf {K} _ {*} \left(\mathbf {K} _ {(s + t) (s + t)} + \left[ \begin{array}{c c} \varrho_ {s} ^ {2} \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \varrho_ {t} ^ {2} \mathbf {I} \end{array} \right]\right) ^ {- 1} \mathbf {y} \quad \boldsymbol {\Gamma} = \mathbf {K} _ {* *} - \mathbf {K} _ {*} \left(\mathbf {K} _ {(s + t) (s + t)} + \left[ \begin{array}{c c} \varrho_ {s} ^ {2} \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \varrho_ {t} ^ {2} \mathbf {I} \end{array} \right]\right) ^ {{- 1}} \mathbf {K} _ {*} ^ {T} \tag {6}
$$
Here,  $\mathbf{y} = \begin{bmatrix} \mathbf{y}_s \\ \mathbf{y}_{tla} \end{bmatrix}$  denotes the ground-truth output values of labeled nodes  $V_s \cup V_t^{la}$  from source and target graphs. Each entry of  $\mathbf{K}_*$  represents the covariance between testing target node and training node, and  $\mathbf{K}_{**}$  denotes the covariance matrix over testing target nodes.
The posterior distribution of GraphGP has the following hyper-parameters:  $\sigma_w, \varsigma_w, \varsigma_b, \alpha_i^s, \alpha_j^t$ , and noise variances  $\varrho_s, \varrho_t$ . The hyper-parameters of the Gaussian process can be optimized by maximizing the marginal likelihood  $p(\mathbf{y})$  over all the training samples [42]. However, in the context of transferable graph learning, the number of labeled target nodes is much smaller than the number of labeled source nodes. By directly maximizing the marginal likelihood over all labeled source and target nodes, GraphGP might be biased towards the source domain. Therefore, in this paper, we propose to optimize the marginal distribution  $p(\mathbf{y}_{t_{la}})$  over labeled target nodes by considering all the nodes  $V_s \cup V_t^{la}$  as the inducing points [47] (see more efficiency analysis in Appendix A.6). The objective function is given by maximizing the following log marginal likelihood
$$
\log p \left(\mathbf {y} _ {t l _ {a}}\right) = \log \left[ \mathcal {N} \left(\mathbf {y} _ {t l _ {a}} \mid \mathbf {0}, \mathbf {K} _ {t (s + t)} \mathbf {K} _ {(s + t) (s + t)} ^ {- 1} \mathbf {K} _ {t (s + t)} ^ {T} + \varrho_ {t} ^ {2} \mathbf {I}\right) \right] \tag {7}
$$
4.4 Generalization Analysis
We define the generalization error of the target graph for transferable node regression. Given the ground-truth labeling function  $f^{*}$  in the target domain and the estimator  $f^{(L)}$  learned from observed source and target graphs, the generalization error is given by
$$
\epsilon_ {t} = \int \int \mathcal {L} \left(\gamma \left(v _ {t}, G _ {t}\right), f ^ {*} \left(v _ {t}, G _ {t}\right)\right) p \left(v _ {t}, G _ {t}\right) p \left(f ^ {*}\right) d \left(v _ {t}, G _ {t}\right) d f ^ {*} \tag {8}
$$
where  $\gamma (\cdot)$  is the posterior mean of  $f^{(L)}(\cdot)$  in Eq. (6),  $\mathcal{L}$  is the mean squared loss function, and  $p(v_{t},G_{t})$  denotes the sampling probability $^2$  of a target input  $(v_{t},G_{t})$ . It is shown [9, 42] that if the GP prior is correctly specified, i.e., the predictor  $f^{(L)}$  has the same prior as  $f^*$ , the generalization error is given by the predictive variance of the GP. That is,  $\epsilon_t = \int \Gamma (v_t,G_t)p(v_t,G_t)d(v_t,G_t)$  where  $\Gamma (v_t,G_t)$  is the posterior variance at  $(v_{t},G_{t})$  in Eq. (6). Furthermore, the following theorem shows that the generalization error can be upper bounded in terms of the variance that takes all inputs  $(v_{s},G_{s})$  of the source graph as the target samples, i.e., both  $(v_{s},G_{s})$  and  $(v_{t},G_{t})$  are assumed to follow the same target distribution for single-domain graph learning.
Theorem 4.5. Let  $\mathbf{K}_{ss}^{\mu}$  be the sample-level covariance matrix over source nodes, i.e.,  $[\mathbf{K}_{ss}^{\mu}]_{(v_s,v_s')} = K_{\mu}^{(L)}(v_s,v_s'|G_s)$ ,  $\nu_{ss} = K_{\nu}^{(L)}(G_s,G_s)$ ,  $\nu_{tt} = K_{\nu}^{(L)}(G_t,G_t)$  be the intra-graph kernels and  $\nu_{st} = K_{\nu}^{(L)}(G_s,G_t)$  be the inter-graph kernel. Suppose  $\varrho_s^2 \triangleq \left(\frac{\nu_{ss} \cdot \nu_{tt}}{\nu_{st} \cdot \nu_{st}} - 1\right) \sigma_w^2 \nu_{tt} \bar{\lambda}_{ss} + \frac{\nu_{tt} \cdot \nu_{tt}}{\nu_{st} \cdot \nu_{st}} \varrho_s^2$  where  $\bar{\lambda}_{ss}$  is the maximum eigenvalue of  $\mathbf{K}_{ss}^{\mu}$ , for any  $(v_t,G_t)$  the generalization error is bounded by
$$
\epsilon_ {t} \leq \int \Gamma_ {t} (v _ {t}, G _ {t}; \bar {\varrho} _ {s} ^ {2}, \varrho_ {t} ^ {2}) p (v _ {t}, G _ {t}) d (v _ {t}, G _ {t})
$$
where  $\Gamma_t(v_t, G_t; \varrho_s^2, \varrho_t^2)$  is the variance assuming that all source examples are observed in the target domain with respect to noises  $\varrho_s^2$ ,  $\varrho_t^2$ .
This theorem confirms that compared to the single-domain Gaussian process (assuming all inputs are drawn from the same distribution), GraphGP-based transferable graph learning enables the reduced generalization error. In addition, the following corollary considers a special scenario where no labeled nodes are available in the target graph. In this case, the generalization error is determined by the normalized graph domain similarity  $\frac{K_{\nu}^{(L)}(G_s,G_t)}{\sqrt{K_{\nu}^{(L)}(G_s,G_s)\cdot K_{\nu}^{(L)}(G_t,G_t)}}$ .
Corollary 4.6. When there are no labeled nodes in the target graph, i.e.,  $V_{t}^{la} = \emptyset$ , we have
$$
\lim _ {| V _ {s} | \to \infty} \epsilon_ {t} = \left(1 - \frac {\nu_ {s t} ^ {2}}{\nu_ {t t} \nu_ {s s}}\right) \int K ^ {(L)} \left((v _ {t}, G _ {t}), (v _ {t}, G _ {t})\right) p (v _ {t}, G _ {t}) d (v _ {t}, G _ {t})
$$
where  $\nu_{ss} = K_{\nu}^{(L)}(G_s,G_s),\nu_{tt} = K_{\nu}^{(L)}(G_t,G_t)$  and  $\nu_{st} = K_{\nu}^{(L)}(G_s,G_t)$ .
2Here, it holds  $p(v_{t},G_{t}) = \mathbb{P}_{\mathcal{P}}(\mathbb{P}_{\mathcal{G}_{t}})\mathbb{P}_{\mathcal{G}_{t}}(v_{t})$  as discussed in Subsection 4.1, i.e., it first samples a graph domain distribution  $\mathbb{P}_{\mathcal{G}_t}$  and then sample the nodes/edges to form the graph  $G_{t}$ .
Model	Airport graphs	Agriculture graphs
BR→EU	EU→BR	BR→US	MA→SG	SG→MA	MA→SY
RBFGP [42]	0.4849±0.0260	0.4479±0.0211	0.3682±0.0301	0.4859±0.0716	0.3359±0.0297	0.7314±0.0172
DINO [56]	0.5241±0.0147	0.4855±0.0337	0.3877±0.0263	0.6227±0.0223	0.4591±0.0445	0.7620±0.0125
GGP [35]	0.3400±0.0144	0.3990±0.0401	0.4720±0.0218	0.4515±0.0099	0.2403±0.0172	0.7420±0.0425
SAGEGP [37]	0.4581±0.0308	0.3822±0.0508	0.4928±0.0330	0.5348±0.0278	0.3633±0.0101	0.7632±0.0349
GINGP [37]	0.5216±0.0227	0.4471±0.0219	0.4901±0.0255	0.5380±0.0288	0.3559±0.0196	0.7746±0.0518
GraphGP	0.5567±0.0246	0.4983±0.0370	0.5293±0.0335	0.6586±0.0244	0.5125±0.0171	0.7921±0.0168
Table 1: Results of transferable node regression on Airport and Agriculture data sets
Model	PT→RU	EN→PT	RU→ES	RU→PT	PT→EN	ES→RU
RBFGP [42]	0.4324±0.0087	0.5430±0.0117	0.4241±0.0024	0.4962±0.0055	0.5507±0.0024	0.4738±0.0059
DINO [56]	0.6498±0.0059	0.7565±0.0140	0.6717±0.0114	0.7641±0.0082	0.5920±0.0195	0.5636±0.0072
GGP [35]	0.3167±0.0079	0.5689±0.0028	0.3699±0.0087	0.5173±0.0018	0.4237±0.0037	0.4237±0.0035
SAGEGP [37]	0.5655±0.0196	0.5191±0.0029	0.5958±0.0076	0.6921±0.0174	0.5720±0.0187	0.5960±0.0074
GINGP [37]	0.5782±0.0218	0.7525±0.0294	0.6803±0.0198	0.6910±0.0540	0.6349±0.0090	0.5997±0.0185
GraphGP	0.7069±0.0055	0.8013±0.0109	0.7520±0.0144	0.7909±0.0382	0.6745±0.0127	0.6789±0.0172
Table 2: Results of transferable node regression on Twitch data set
Model	Wikipedia graphs	WebKB graphs
SQ → CH	CH → SQ	CO → TX	WS → TX
RBFGP	-0.0383±0.0526	-0.0119±0.0288	0.3089±0.0533	0.2756±0.0204
DINO	0.2938±0.0253	0.1157±0.0145	0.3536±0.0442	0.2537±0.0232
LINKXGP	0.4100±0.0205	0.0838±0.0307	0.3836±0.0530	0.2898±0.0176
MixHopGP	0.4050±0.0456	0.3509±0.0102	0.3264±0.0569	0.3062±0.0337
H2GCNGP	0.4165±0.0559	0.2652±0.0106	0.3816±0.0430	0.3036±0.0442
GraphGP	0.4938±0.0352	0.3214±0.0080	0.4146±0.0402	0.3301±0.0585
Table 3: Results on Wikipedia and WebKB
Figure 2: Domain similarity
Data Sets: We use the following graph learning benchmarks with node regression tasks. (1) Twitch [44]: It has 6 different domains ("DE", "EN", "ES", "FR", "PT", and "RU"). (2) Agriculture [34, 60]: It has 3 different domains ("Maize" (MA), "Sorghum" (SG), and "Soybean" (SY)). (3) Airports [43]: It has 3 different domains ("USA" (US), "Brazil" (BR), and "Europe" (EU)). (4) Wikipedia [44]: It has 3 different domains ("chameleon" (CH), "crocodile" (CR), and "squirrel" (SQ)). (5) WebKB [41]: It has 3 different domains ("Cornell" (CO), "Texas" (TX), and "Wisconsin" (WS)).
Baselines: We consider the following Gaussian process baselines. (1) RBFGP [42] and DINO [56] are feature-Only Gaussian processes without using graph structures. (2) GGP [35], SAGEGP [37], and GINGP [37] are graph Gaussian processes by considering source and target graphs as a large disjoint graph. (3) LINKXGP, MixHopGP, and H2GCNGP are graph Gaussian processes derived from LINKX [30], MixHop [1], H2GCN [68] respectively. It is notable that recent work [37] shows the equivalence between graph neural networks (e.g., GraphSAGE [20], GIN [59]) and graph Gaussian processes. Similarly, we can derive the corresponding graph Gaussian processes for LINKX [30], MixHop [1] and H2GCN [68] in Appendix A.5, which are termed as LINKXGP, MixHopGP, and H2GCNGP, respectively.
Model Configuration: In the experiments, we use GPyTorch [16] to build the graph Gaussian process models and optimize the hyperparameters with gradient descent optimizer. For the proposed GraphGP algorithm, we adopt  $k = 2$  and  $L = 2$  for all the experiments. The hyperparameters are optimized using Adam [24] with a learning rate of 0.01 and a total number of training epochs of 500. All the experiments are performed on a Windows machine with four 3.80GHz Intel Cores, 64GB RAM, and two NVIDIA Quadro RTX 5000 GPUs<sup>3</sup>.
3Code is available at https://github.com/jwu4ssl/GraphGP.
Table 1 and Table 2 provide the transferable node regression results on Airport, Agriculture, and Twitch graphs. Following [37], we report the coefficient of determination  $R^2$  (mean and standard deviation with 5 runs) on the testing target nodes for performance comparison. The experimental results verify the effectiveness of GraphGP over Gaussian process baselines. It is notable that the graphs in Table 1 and Table 2 follow the homophily assumption. In contrast, Wikipedia and WebKB graphs in Table 3 hold the heterophily assumption [68]. It can be seen that under heterophily assumption,
(a) WebKB  $\rightarrow$  Airport
Figure 3: Knowledge transfer across graphs with different assumptions
(b) Airport  $\rightarrow$  WebKB
GraphGP also archives much better performance in most cases. In addition, we evaluate the knowledge transferability across graphs with different assumptions in Figure 3. It is observed that compared to GINGP (or H2GCNGP) over both labeled source and target nodes, GINGP-T (or H2GCNGP-T) learns the Gaussian process over only target nodes, thereby enabling much better transfer performance from WebKB to Airport in Figure 3a (or from Airport to WebKB in Figure 3b). Moreover, GraphGP can outperform those target-only graph Gaussian processes in this scenario.
We also investigate the correlation between transfer performance and normalized graph domain similarity  $\frac{K_{\nu}^{(L)}(G_s,G_t)}{\sqrt{K_{\nu}^{(L)}(G_s,G_s)\cdot K_{\nu}^{(L)}(G_t,G_t)}}$  in Corollary 4.6. Figure 2 visualizes the estimated normalized graph domain similarity of GraphGP and  $R^2$  over the target testing nodes for RU  $\rightarrow$  PT on Twitch, where the graph domain similarity changes when injecting noise to the target graph (see Appendix A.7.1). It shows that the transfer performance is positively correlated with the normalized graph domain similarity. This is consistent with our observation in Corollary 4.6.
Flexibility of GraphGP: As illustrated in Subsection 4.1, it is flexible to instantiate structure-aware neural networks of Eq. (2) with existing message-passing graph neural networks. Table 4 provides the results of GIN [59] induced GraphGP algorithm. Furthermore, it is feasible to simply instantiate  $K_{\nu}^{(L)}(G,G^{\prime})$  of GraphGP with existing graph kernels, e.g., Shortest Path kernel [5]. It is observed that the variants GraphGP_GIN and GraphGP_ShottestPath of
Methods	BR → EU
GraphGP	0.5567±0.0246
GraphGP_GIN	0.5301±0.0229
GraphGP_ShortestPath	0.5377±0.0304
GraphGP achieve comparable performance. This highlights that GraphGP is flexible to incorporate with existing GNNs and graph kernels for transferable graph learning tasks.
Comparison with Transferable GNNs: In addition to Gaussian processes, we also compare GraphGP with state-of-the-art transferable graph neural networks. Table 5 shows the competitive performance of GraphGP over transferable GNN baselines. This is because GraphGP explicitly maximizes the marginal likelihood  $p(\mathbf{y}_{t_{la}})$ .
Table 4: Flexibility of GraphGP
Methods	BR → EU	EU → BR	BR → US
EGI [69]	0.5204±0.0357	0.4786±0.0225	0.4951±0.0176
GARDE [55]	0.5314±0.0208	0.4792±0.0296	0.4354±0.0109
GraphGP	0.5567±0.0246	0.4983±0.0370	0.5293±0.0335
Table 5: Performance comparison between GraphGP and transferable GNNs
In contrast, transferable GNN baselines minimize the prediction loss over all the labeled nodes  $(|V_s| \gg |V_t^{la}|)$  and domain discrepancy, and thus they are more likely to bias towards the source graph.
This paper studies the problem of transferable graph learning involving knowledge transfer from a source graph to a relevant target graph. To solve this problem, we propose a graph Gaussian process (GraphGP) algorithm, which is derived from a structure-aware neural network encoding both sample-level node representation and domain-level graph representation. The efficacy of GraphGP is verified theoretically and empirically in various transferable node regression tasks.
This work is supported by National Science Foundation under Award No. IS-1947203, IIS-2117902, IIS-2137468, IIS-2002540, and Agriculture and Food Research Initiative (AFRI) grant no. 2020-67021-32799/project accession no.1024178 from the USDA National Institute of Food and Agriculture. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.