SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left[ \begin{array}{c c} \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s t} ^ {\mu} \\ \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t t} ^ {\mu} \end{array} \right] + \left[ \begin{array}{c c} s ^ {2} \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \varrho_ {t} ^ {2} \mathbf {I} \end{array} \right] \succeq \left[ \begin{array}{c c} \frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s s} ^ {\mu} & \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s t} ^ {\mu} \\ \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {t s} ^ {\mu} & {\sigma_ {w} ^ {2}} \nu_ {t t} \mathbf {K} _ {t t} ^ {\mu} \end{array} \right] + \left[ \begin{array}{c c} \frac {\nu_ {t t} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \varrho_ {s} ^ {2} \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \varrho_ {t} ^ {2} \mathbf {I} \end{array} \right]
$$
$$
\left[ \begin{array}{c c} \left(1 - \frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}}\right) \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s s} ^ {\mu} & \mathbf {0} \\ \mathbf {0} & \mathbf {0} \end{array} \right] \succeq \left[ \begin{array}{c c} \left(\frac {\nu_ {t t} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \varrho_ {s} ^ {2} - s ^ {2}\right) \mathbf {I} & \mathbf {0} \\ \mathbf {0} & \mathbf {0} \end{array} \right]
$$
$$
\left(1 - \frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}}\right) \sigma_ {w} ^ {2} \nu_ {t t} \mathbf {K} _ {s s} ^ {\mu} \succeq \left(\frac {\nu_ {t t} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \varrho_ {s} ^ {2} - s ^ {2}\right) \mathbf {I}
$$
$$
\mathbf {K} _ {s s} ^ {\mu} \preceq \frac {1}{\left(\frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} - 1\right) \sigma_ {w} ^ {2} \nu_ {t t}} \left(s ^ {2} - \frac {\nu_ {t t} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \varrho_ {s} ^ {2}\right) \mathbf {I}
$$
$$
\bar {\lambda} _ {s s} \leq \frac {1}{\left(\frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} - 1\right) \sigma_ {w} ^ {2} \nu_ {t t}} \left(s ^ {2} - \frac {\nu_ {t t} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \varrho_ {s} ^ {2}\right)
$$
$$
s ^ {2} \geq \left(\frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} - 1\right) \sigma_ {w} ^ {2} \nu_ {t t} \bar {\lambda} _ {s s} + \frac {\nu_ {t t} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \varrho_ {s} ^ {2}
$$
Thus, the minimum of the upper bound is given by
$$
\bar {\varrho} _ {s} ^ {2} \triangleq \left(\frac {\nu_ {s s} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} - 1\right) \sigma_ {w} ^ {2} \nu_ {t t} \bar {\lambda} _ {s s} + \frac {\nu_ {t t} \cdot \nu_ {t t}}{\nu_ {s t} \cdot \nu_ {s t}} \varrho_ {s} ^ {2}
$$
A.3.6 Proof of Corollary 4.6
Corollary 4.6 states that when there are no labeled nodes in the target graph, i.e.,  $V_{t}^{la} = \emptyset$ , it holds
$$
\lim  _ {| V _ {s} | \to \infty} \epsilon_ {t} = \left(1 - \frac {\nu_ {s t} ^ {2}}{\nu_ {t t} \nu_ {s s}}\right) \int K ^ {(L)} \left(\left(v _ {t}, G _ {t}\right), \left(v _ {t}, G _ {t}\right)\right) p \left(v _ {t}, G _ {t}\right) d \left(v _ {t}, G _ {t}\right)
$$
where  $\nu_{ss} = K_{\nu}^{(L)}(G_s, G_s)$ ,  $\nu_{tt} = K_{\nu}^{(L)}(G_t, G_t)$  are intra-graph kernels and  $\nu_{st} = K_{\nu}^{(L)}(G_s, G_t)$  is inter-graph kernel.
Proof. It is given
For all  $(v_{t},G_{t})$ , it holds
$$
\begin{array}{l} \Gamma (v _ {t}, G _ {t}) = \mathbf {k} _ {* *} - \mathbf {k} _ {*} \left(\mathbf {K} _ {s s} + \varrho_ {s} ^ {2} \mathbf {I}\right) ^ {- 1} \mathbf {k} _ {*} ^ {T} \\ = \sigma_ {w} ^ {2} \nu_ {t t} K _ {\mu} ^ {(l)} \left(v _ {t}, v _ {t} \mid G _ {t}\right) \\ - \left(\sigma_ {w} ^ {2} \nu_ {s t} \mathbf {k} _ {* \mu}\right) \left(\mathbf {K} _ {s s} + \varrho_ {s} ^ {2} \mathbf {I}\right) ^ {- 1} \left(\sigma_ {w} ^ {2} \nu_ {s t} \mathbf {k} _ {* \mu}\right) ^ {T} \\ = \sigma_ {w} ^ {2} \left(\nu_ {t t} - \frac {\nu_ {s t} ^ {2}}{\nu_ {s s}}\right) K _ {\mu} ^ {(l)} \left(v _ {t}, v _ {t} \mid G _ {t}\right) \\ + \frac {\nu_ {s t} ^ {2}}{\nu_ {s s} ^ {2}} \left[ \sigma_ {w} ^ {2} \nu_ {s s} K _ {\mu} ^ {(l)} (v _ {t}, v _ {t} | G _ {t}) - \left(\sigma_ {w} ^ {2} \nu_ {s s} \mathbf {k} _ {* \mu}\right) (\mathbf {K} _ {s s} + \varrho_ {s} ^ {2} \mathbf {I}) ^ {- 1} \left(\sigma_ {w} ^ {2} \nu_ {s s} \mathbf {k} _ {* \mu}\right) ^ {T} \right] \\ = \sigma_ {w} ^ {2} \left(\nu_ {t t} - \frac {\nu_ {s t} ^ {2}}{\nu_ {s s}}\right) K _ {\mu} ^ {(l)} (v _ {t}, v _ {t} | G _ {t}) + \frac {\nu_ {s t} ^ {2}}{\nu_ {s s} ^ {2}} \Gamma_ {s} (v _ {t}, G _ {t}) \\ \end{array}
$$
where  $\Gamma_s(v_t, G_t)$  is the posterior variance by considering  $(v_t, G_t)$  as a virtual source sample. Then when the number of source samples goes to infinity, we have
$$
\lim_{|V_{s}|\to \infty}\Gamma_{s}(v_{t},G_{t}) = 0
$$
$$
\begin{array}{l} \Gamma (v _ {t}, G _ {t}) = \sigma_ {w} ^ {2} \left(\nu_ {t t} - \frac {\nu_ {s t} ^ {2}}{\nu_ {s s}}\right) K _ {\mu} ^ {(l)} (v _ {t}, v _ {t} | G _ {t}) \\ = \left(1 - \frac {\nu_ {s t} ^ {2}}{\nu_ {t t} \nu_ {s s}}\right) \sigma_ {w} ^ {2} \nu_ {t t} K _ {\mu} ^ {(l)} (v _ {t}, v _ {t} | G _ {t}) \\ = \left(1 - \frac {\nu_ {s t} ^ {2}}{\nu_ {t t} \nu_ {s s}}\right) K ^ {(L)} \left(\left(v _ {t}, G _ {t}\right), \left(v _ {t}, G _ {t}\right)\right) \\ \end{array}
$$
where  $\nu_{ss} = K_{\nu}^{(L)}(G_s, G_s), \nu_{tt} = K_{\nu}^{(L)}(G_t, G_t)$  denote the intra-graph kernels and  $\nu_{st} = K_{\nu}^{(L)}(G_s, G_t)$  is the inter-graph kernel.
$$
\begin{array}{l} \lim_{|V_{s}|\to \infty}\epsilon_{t} = \int \Gamma (v_{t},G_{t})p(v_{t},G_{t})d(v_{t},G_{t}) \\ = \left(1 - \frac {\nu_ {s t} ^ {2}}{\nu_ {t t} \nu_ {s s}}\right) \int K ^ {(L)} \left(\left(v _ {t}, G _ {t}\right), \left(v _ {t}, G _ {t}\right)\right) p \left(v _ {t}, G _ {t}\right) d \left(v _ {t}, G _ {t}\right) \\ \end{array}
$$
The generalization error is determined by graph domain similarity  $\frac{K_{\nu}^{(L)}(G_s,G_t)}{\sqrt{K_{\nu}^{(L)}(G_s,G_s)\cdot K_{\nu}^{(L)}(G_t,G_t)}}$
A.4 Model Discussion
In this section, we discuss several instantiations of the proposed structure-aware neural networks in Eq. (2) and their induced graph Gaussian processes. This shows the flexibility of the proposed algorithms in incorporating existing graph neural networks and graph kernels.
A.4.1 Intantiations of Message-Passing Graph Neural Networks
Table 6 summarizes several instantiations of message-passing graph neural networks (GNNs) [18] of Eq. (1) with different neighborhood selection and message aggregation strategies.
It can be seen that homophily-based GNNs are more likely to aggregate messages from nearby neighbors, while heterophily-based GNNs have to adaptively aggregate messages from distant neighbors within the graph. These GNNs can be applied to design the proposed structure-aware neural networks in Eq. (2).
Model	Message-passing mechanism	Neighborhood
GCN [25]	h(vl+1)=φ(ˆuvW(l)h(v)+∑u∈N1(v)ˆuvW(l)h(ul))	k=1
GraphSAGE [20]	h(vl+1)=φ(W(l)[h(v) ||h(l)N1(v)])	k=1
GAT [50]	h(vl+1)=φ(αvvW(l)h(v)+∑u∈N1(v)αuvW(l)h(ul))	k=1
GIN [59]	h(vl+1)=MLP(l)(1+ε(l))h(v)+∑u∈N1(v)h(ul)	k=1
GCNII* [10]	h(vl+1)=φ((1-βl)I+βlW(l))(1-λl)∑u∈{v}∪N1(v)ˆuvh(ul)+λlh(v))	k=1
MixHop [1]	h(vl+1)=φ(W0(l)h(v)||φ(∑u∈N1(v)ˆuvW1(l)h(u))||φ(∑u∈N2(v)ˆ2uvW2(l)h(u))	k=2
H2GCN [68]	h(vl+1)=(∑u∈N1(v)ˆduvhu(l))||(∑u∈N2(v)ˆduvhu(l))	k=2
GPR-GNN [11]	h(vl)=∑i=1kγi∑u∈VˆAu vhu(l-1)	k=10
HOG-GCN [52]	h(vl+1)=φ(μWe(l)h(v)+ξW(n)∑u∈V[ˆD-1(A+A2) ⊙ H]uvh(u)	k=2
GloGNN* [29]	h(vl+1)=(1-γ)∑u∈VZ(u l)*h(u l)+γh(v0)	k=∞
Table 6: Instantiations of message-passing graph neural networks (* initial residual connection is adopted)
A.4.2 Instantiations of Structure-Aware Neural Networks
It can be seen that the definition of structure-aware neural network Eq. (2) is flexible to incorporate existing graph neural networks [10, 20, 25, 59] by instantiating  $\mu^{(l)}(v|G)$  and  $\nu^{(l)}(G)$ . For example, we can use Graph Isomorphism Network (GIN) [25] to define the structure-aware neural network Eq. (2) as follows. The graph convolutional layer of GIN is
$$
\mathbf {h} _ {v} ^ {(l)} = \frac {1}{\sqrt {M}} \sum_ {u \in \{v \cup N _ {1} (v) \}} a _ {u v} \mathbf {W} _ {\mathrm {G I N}} ^ {(l)} \mathbf {x} _ {u} ^ {(l)} + \mathbf {b} _ {\mathrm {G I N}} ^ {(l)} \quad \text {a n d} \quad \mathbf {x} _ {u} ^ {(l)} = \phi \left(\mathbf {h} _ {u} ^ {(l - 1)}\right) \tag {9}
$$
where  $a_{uv} = 1$  for  $u \neq v$  and  $a_{uv} = 1 + \epsilon^{(l)}$  for  $u = v$ .  $\mathbf{W}_{\mathrm{GIN}}^{(l)}$  and  $\mathbf{b}_{\mathrm{GIN}}^{(l)}$  denote the weight and bias parameters in GIN, respectively. In this case, we use GIN to instantiate both  $\mu^{(l)}(v|G)$  and  $\nu^{(l)}(G)$  for learning  $\mathbf{h}_v^{(l)}$  and  $\tilde{\mathbf{h}}_v^{(l)}$ . In addition, the READOUT function of Eq. (2) can be instantiated with mean pooling [62], i.e.,  $\tilde{\mathbf{h}}_G^{(l)} = \frac{1}{|V|}\sum_{v \in V}\tilde{\mathbf{h}}_v^{(l)}$ . It is revealed [21, 37] that GIN is equivalent to the Gaussian process in the limit on the layer width. Following this observation, the following theorem shows the equivalence between structure-aware neural network and adaptive graph Gaussian process when using GIN to instantiate Eq. (2).
Theorem A.1. Suppose the graph convolutional layers of structure-aware neural network Eq. (2) are instantiated with GIN and the READOUT function of Eq. (2) is instantiated with mean pooling. Assume that the layer width the network width goes to infinity, and the model parameters of all neural layers are independently and randomly drawn from Gaussian distribution, then for each  $i$  and  $l$ , the output function  $f_{i}^{(l)}$  in Eq. (2) follows a Gaussian process with  $f_{i}^{(l)} \sim \mathcal{GP}(0, K^{(l)})$ , where the covariance function  $K^{(l)}$  is given by
$$
K ^ {(l)} \left(\left(v, G\right), \left(v ^ {\prime}, G ^ {\prime}\right)\right) = \sigma_ {w} ^ {2} \cdot K _ {\mu} ^ {(l)} \left(v, v ^ {\prime} \mid G, G ^ {\prime}\right) \cdot K _ {\nu} ^ {(l)} \left(G, G ^ {\prime}\right) \tag {10}
$$
$$
K _ {\mu} ^ {(l)} \left(v, v ^ {\prime} | G, G ^ {\prime}\right) = \varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot [ \mathbf {a} ] _ {v,:} \mathbf {C} ^ {(l - 1)} [ \mathbf {a} ^ {\prime T} ] _ {:, v ^ {\prime}}
$$
$$
K _ {\nu} ^ {(l)} \left(G, G ^ {\prime}\right) = \varsigma_ {b} ^ {2} + \frac {\varsigma_ {w} ^ {2}}{\left| V \right| \cdot \left| V ^ {\prime} \right|} \mathbf {1} ^ {T} \mathbf {a C} ^ {(l - 1)} \mathbf {a} ^ {\prime T} \mathbf {1}
$$
$$
\mathbf {C} ^ {(l - 1)} = \mathbb {E} _ {z _ {i} ^ {(l - 1)} \sim \mathcal {G P} \left(0, K _ {\mu} ^ {(l - 1)}\right)} \left[ \phi (z _ {i} ^ {(l - 1)}) \phi (z _ {i} ^ {(l - 1)}) ^ {T} \right]
$$
Here,  $\mathbf{a}$  and  $\mathbf{a}^{\prime T}$  are the adjacency matrices with  $[\mathbf{a}]_{uv} = a_{uv}$  and  $[\mathbf{a}^{\prime}]_{uv} = a_{uv}^{\prime}$  respectively.
Proof. The sample covariance is given by
$$
K _ {\mu} ^ {(l)} (v, v ^ {\prime} | G, G ^ {\prime}) = \mathbb {E} \left[ \mu_ {j} ^ {(l)} (v | G) \cdot \mu_ {j} ^ {(l)} (v ^ {\prime} | G ^ {\prime}) \right]
$$
When learning the node representation, graphs  $G = (V,E)$  and  $G' = (V',E')$  can be considered as a single large graph  $G'' = (V \cup V', E \cup E')$  with two disjoint components. The adjacent matrix of  $G''$  is then given by  $\mathbf{a}'' = \begin{bmatrix} \mathbf{a} & \mathbf{0} \\ \mathbf{0} & \mathbf{a}' \end{bmatrix}$ . In this case, learning node representation of  $v$  from  $G$  is equivalent to learning that from  $G''$ , due to the disconnection of two components  $G$  and  $G'$  within the large graph  $G''$ . Then, using the results from [37], given a single graph  $G''$ , the outputs of each layer in GCN over all nodes are equivalent to the Gaussian process with zero mean and covariance matrix
$$
\begin{array}{l} K _ {\mu} ^ {(l)} (V ^ {\prime \prime}, V ^ {\prime \prime}) = \varsigma_ {b} ^ {2} \mathbf {I} + \varsigma_ {w} ^ {2} \mathbf {a} ^ {\prime \prime} \mathbf {C} _ {G ^ {\prime \prime}} ^ {(l - 1)} \mathbf {a} ^ {\prime \prime T} \\ \mathbf {C} _ {G ^ {\prime \prime}} ^ {(l - 1)} = \mathbb {E} _ {z _ {i} ^ {(l - 1)} \sim K _ {\mu} ^ {(l - 1)}} (V ^ {\prime \prime}, V ^ {\prime \prime}) \left[ \phi (z _ {i} ^ {(l - 1)}) \phi (z _ {i} ^ {(l - 1)}) ^ {T} \right] \\ \end{array}
$$
Considering the block matrix form of  $\mathbf{C}_{G^{\prime \prime}}^{(l - 1)} = \left[ \begin{array}{cc}\mathbf{C}_{G}^{(l - 1)} & \mathbf{C}^{(l - 1)}\\ (\mathbf{C}^{(l - 1)})^T & \mathbf{C}_{G^{\prime \prime}}^{(l - 1)} \end{array} \right](\mathbf{C}_{G^{\prime \prime}}^{(l - 1)}$  is a symmetric matrix), we have
$$
K _ {\mu} ^ {(l)} (V ^ {\prime \prime}, V ^ {\prime \prime}) = \varsigma_ {b} ^ {2} \mathbf {I} + \varsigma_ {w} ^ {2} \left[ \begin{array}{c c} \mathbf {a C} _ {G} ^ {(l - 1)} \mathbf {a} ^ {T} & \mathbf {a C} ^ {(l - 1)} \mathbf {a} ^ {\prime T} \\ \mathbf {a} ^ {\prime} (\mathbf {C} ^ {(l - 1)}) ^ {T} \mathbf {a} ^ {T} & \mathbf {a} ^ {\prime} \mathbf {C} _ {G ^ {\prime}} ^ {(l - 1)} \mathbf {a} ^ {\prime T} \end{array} \right]
$$
$$
\begin{array}{l} \mathbf {C} _ {G} ^ {(l - 1)} = \mathbb {E} _ {z _ {i} ^ {(l - 1)} \sim K _ {\mu} ^ {(l - 1)} (V, V)} \left[ \phi (z _ {i} ^ {(l - 1)}) \phi (z _ {i} ^ {(l - 1)}) ^ {T} \right] \\ \mathbf {C} ^ {(l - 1)} = \mathbb {E} _ {z _ {i} ^ {(l - 1)} \sim K _ {\mu} ^ {(l - 1)} (V, V ^ {\prime})} \left[ \phi (z _ {i} ^ {(l - 1)}) \phi (z _ {i} ^ {(l - 1)}) ^ {T} \right] \\ \mathbf {C} _ {G ^ {\prime}} ^ {(l - 1)} = \mathbb {E} _ {z _ {i} ^ {(l - 1)} \sim K _ {\mu} ^ {(l - 1)} (V ^ {\prime}, V ^ {\prime})} \left[ \phi (z _ {i} ^ {(l - 1)}) \phi (z _ {i} ^ {(l - 1)}) ^ {T} \right] \\ \end{array}
$$
$$
K _ {\mu} ^ {(l)} \left(v, v ^ {\prime} | G, G ^ {\prime}\right) = \varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot [ \mathbf {a C} ^ {(l - 1)} \mathbf {a} ^ {\prime T} ] _ {v v ^ {\prime}} = \varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot [ \mathbf {a} ] _ {v,:} \mathbf {C} ^ {(l - 1)} [ \mathbf {a} ^ {\prime T} ] _ {:, v ^ {\prime}}
$$
where  $[\mathbf{B}]_{vv'}$  denotes the entry of a matrix  $\mathbf{B}$  at the  $v$ -th row and  $v'$ -th column.
In addition, the domain covariance is given by
$$
\begin{array}{l} K _ {\nu} ^ {(l)} \left(G, G ^ {\prime}\right) = \mathbb {E} \left[ \frac {1}{| V | \cdot | V ^ {\prime} |} \sum_ {v \in V} \sum_ {v ^ {\prime} \in V ^ {\prime}} \mu_ {j} ^ {(l)} (v | G) \cdot \mu_ {j} ^ {(l)} \left(v ^ {\prime} \mid G ^ {\prime}\right) \right] \\ = \frac {1}{| V | \cdot | V ^ {\prime} |} \sum_ {v \in V} \sum_ {v ^ {\prime} \in V ^ {\prime}} \mathbb {E} \left[ \mu_ {j} ^ {(l)} (v | G) \cdot \mu_ {j} ^ {(l)} \left(v ^ {\prime} | G ^ {\prime}\right) \right] \\ = \frac {1}{| V | \cdot | V ^ {\prime} |} \sum_ {v \in V} \sum_ {v ^ {\prime} \in V ^ {\prime}} \left(\varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot [ \mathbf {a} ] _ {v,:} \mathbf {C} ^ {(l - 1)} [ \mathbf {a} ^ {\prime T} ] _ {:, v ^ {\prime}}\right) \\ = \varsigma_ {b} ^ {2} + \frac {\varsigma_ {w} ^ {2}}{| V | \cdot | V ^ {\prime} |} \mathbf {1} ^ {T} \mathbf {a C} ^ {(l - 1)} \mathbf {a} ^ {\prime T} \mathbf {1} \\ \end{array}
$$
Similarly, we can also design the adaptive graph Gaussian processes for other message-passing graph neural networks. More generally, we have the following observations.
Corollary A.2. Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. If  $\mu_j^{(l)}(v|G)$  and  $\nu_j^{(l)}(G)$  are Gaussian processes, i.e.,  $\mu_j^{(l)}(v|G) \sim \mathcal{GP}(0, K_\mu^{(l)})$  and  $\nu_j^{(l)}(G) \sim \mathcal{GP}(0, K_\nu^{(l)})$ , then, for each  $i$  and  $l$ , the output function  $f_i^{(l)}(v, G)$  in Eq. (2) is a Gaussian process with  $f_i^{(l)}(v, G) \sim \mathcal{GP}(0, K^{(l)})$ , where the covariance matrix  $K^{(l)}$  is given by
where  $v \in V$  and  $v' \in V'$  denote nodes within the graphs  $G$  and  $G'$ , respectively.
Proof. Since both  $\mu_j^{(l)}(v|G)$  and  $\nu_{j}^{(l)}(G)$  are Gaussian processes and the weight and bias parameters are independent and identically distributed (IID), the output  $f_{i}^{(l)}(v,G)$  at the  $l^{\mathrm{th}}$  layer can be considered as a sum of IID terms. Based on the Central Limit Theorem, in the limit of infinite network width  $N_{l}\to \infty$ ,  $f_{i}^{(l)}(v,G)$  is Gaussian distributed. Moreover, using the multidimensional Central Limit Theorem, any finite collections  $\{f_i^{(l)}(v_1,G),f_i^{(l)}(v_2,G),\dots ,f_i^{(l)}(v_m,G)\}$  have a joint multivariate Gaussian distribution. Therefore, we conclude that  $f_{i}^{(l)}(v,G)$  forms a Gaussian process, i.e.,  $f_{i}^{(l)}(v,G)\sim \mathcal{GP}(\omega^{(l)},K^{(l)})$ . The mean function  $\omega^{(l)}$  is given by
$$
\omega^ {(l)} (v, G) = \mathbb {E} \left[ f _ {i} ^ {(l)} (v, G) \right] = \mathbb {E} \left[ \frac {1}{\sqrt {N _ {l}}} \sum_ {j = 1} ^ {N _ {l}} \mathbf {W} _ {i j} ^ {(l)} \cdot \mu_ {j} ^ {(l)} (v | G) \cdot \nu_ {j} ^ {(l)} (G) \right] = 0
$$
and the covariance function is given by
$$
\begin{array}{l} K ^ {(l)} \left((v, G), (v ^ {\prime}, G ^ {\prime})\right) = \sigma_ {w} ^ {2} \cdot \mathbb {E} \left[ \mu_ {j} ^ {(l)} (v | G) \cdot \mu_ {j} ^ {(l)} \left(v ^ {\prime} \mid G ^ {\prime}\right) \right] \cdot \mathbb {E} \left[ \nu_ {j} ^ {(l)} (G) \cdot \nu_ {j} ^ {(l)} \left(G ^ {\prime}\right) \right] \\ = \sigma_ {w} ^ {2} \cdot K _ {\mu} ^ {(l)} (v, v ^ {\prime} | G, G ^ {\prime}) \cdot K _ {\nu} ^ {(l)} (G, G ^ {\prime}) \\ \end{array}
$$
Moreover, we see that for any  $i > 0$ ,  $f_{i}^{(l)}(v,G)$  can form a Gaussian process with identical mean and covariance functions.
Remark. Corollary A.2 shows that the transferable graph Gaussian process can be derived when both  $\mu_j^{(l)}(v|G) \sim \mathcal{GP}(0, K_\mu^{(l)})$  and  $\nu_j^{(l)}(G) \sim \mathcal{GP}(0, K_\nu^{(l)})$  are Gaussian processes. It is flexible to adopt different GNNs to define  $\mu_j^{(l)}$  and  $\nu_j^{(l)}$ . Moreover, optimizing the selection of kernel spaces over  $\mu_j^{(l)}$  and  $\nu_j^{(l)}$  might lead to better transferable graph learning performance. We would like to leave it as our future work.
A.5 Graph Gaussian Processes of Heterophilic GNNs
In Section 5, in order to tackle the knowledge transferability over heterophilic graphs, we compare the proposed GraphGP algorithm with LINKXGP, MixHopGP, and H2GCNGP derived from LINKX [30], MixHop [1], H2GCN [68] respectively. The graph convolutional layer of MixHop [1] is defined as
$$
\mathbf {H} ^ {(l + 1)} = \left| \right| _ {j \in P} \phi \left(\hat {\mathbf {A}} ^ {j} \mathbf {H} ^ {(l} \mathbf {W} _ {j} ^ {(l)} + \mathbf {b} _ {j} ^ {(l)}\right)
$$
where  $||$  denotes column-wise concatenation. For example, Table 6 shows the MixHop with  $P = \{0,1,2\}$ , which corresponds to the neighbors from at most 2-order neighborhoods. The graph convolutional layer of MixHop [1] can also be rewritten as
$$
\left. \mathbf {h} _ {v} ^ {(l)} = \frac {1}{\sqrt {M}} \right| | _ {j \in P} \left(\sum_ {u \in \{v \cup N _ {1} ^ {j} (v) \}} \hat {\mathbf {A}} _ {u v} ^ {j} \mathbf {W} _ {j, \text {M i x H o p}} ^ {(l)} \mathbf {x} _ {u} ^ {(l)} + \mathbf {b} _ {j, \text {M i x H o p}} ^ {(l)}\right) \quad \text {a n d} \quad \mathbf {x} _ {u} ^ {(l)} = \phi \left(\mathbf {h} _ {u} ^ {(l - 1)}\right)
$$
where  $N_1^j (v)$  denotes the first-order neighborhood of node  $v$  under adjacent matrix  $\hat{\mathbf{A}}^j$ . The following theorem shows the graph Gaussian process (termed as MixHopGP) derived from MixHop.
Corollary A.3 (Graph Gaussian Process of MixHop). Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let  $\mathbf{h}_v^{(l)}(i)$  be the  $i^{\text{th}}$  entry of the output vector  $\mathbf{h}_v^{(l)}$ . Then for any  $i$ ,  $\mathbf{h}_v^{(l)}(i)$  of MixHop over all nodes  $\{v_1,\dots ,v_{|V|}\} \in V$  follows Gaussian process with  $\mathbf{h}_v^{(l)}(i) \sim \mathcal{GP}(0,K_{\mathrm{MixHop}}^{(l)})$  where the covariance function  $K_{\mathrm{MixHop}}^{(l)}$  is given by
$$
K _ {\mathrm {M i x H o p}} ^ {(l)} \left(v, v ^ {\prime}\right) = \sum_ {j \in P} \left(\varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot \left[ \hat {\mathbf {A}} ^ {j} \right] _ {v,:} \mathbf {C} ^ {(l - 1)} \left[ \left(\hat {\mathbf {A}} ^ {j}\right) ^ {T} \right] _ {:, v ^ {\prime}}\right)
$$
where  $v, v' \in V$  are two nodes within graph  $G = (V, E)$ .
Similarly, we can show the graph Gaussian processes (terms as LINKXGP and H2GCNGP) for LINKX [30] and H2GCN [68] as follows.
Corollary A.4 (Graph Gaussian Process of LINKX). Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let  $\mathbf{h}_v^{(l)}(i)$  be the  $i^{\text{th}}$  entry of the output vector  $\mathbf{h}_v^{(l)}$ . Then for any  $i$ ,  $\mathbf{h}_v^{(l)}(i)$  of LINKX over all nodes  $\{v_1,\dots ,v_{|V|}\} \in V$  follows Gaussian process with  $\mathbf{h}_v^{(l)}(i) \sim \mathcal{GP}(0,K_{\mathrm{LINKX}}^{(l)})$  where the covariance function  $K_{\mathrm{LINKX}}^{(l)}$  is given by
$$
K _ {\mathrm {L I N K X}} ^ {(l)} \left(v, v ^ {\prime}\right) = \mathbf {C} _ {\mathbf {A}} + \mathbf {C} _ {\mathbf {X}} + \varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot \left(\mathbf {C} _ {\mathbf {A}} + \mathbf {C} _ {\mathbf {X}}\right)
$$
where  $\mathbf{C}_{\mathbf{A}}, \mathbf{C}_{\mathbf{X}}$  corresponds the NNGP kernel [26] induced by the node attributes and adjacent matrix respectively.
Proof. Following [26], the MLPs over node attributes and adjacent matrix are equivalent to Gaussian processes  $\mathbf{C}_{\mathbf{A}}, \mathbf{C}_{\mathbf{X}}$ . Then the graph Gaussian process of LINKX over  $\mathbf{C}_{\mathbf{A}}, \mathbf{C}_{\mathbf{X}}$  can be derived using  $\mathbf{y} = \mathrm{MLP}_f(\mathrm{W}[\mathbf{h}_{\mathbf{A}}||\mathbf{h}_{\mathbf{X}}] + \mathbf{h}_{\mathbf{A}} + \mathbf{h}_{\mathbf{X}})$  where  $\mathbf{h}_{\mathbf{A}}, \mathbf{h}_{\mathbf{X}}$  are adjacent and feature hidden representation respectively.
Corollary A.5 (Graph Gaussian Process of H2GCN). Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let  $\mathbf{h}_v^{(l)}(i)$  be the  $i^{\text{th}}$  entry of the output vector  $\mathbf{h}_v^{(l)}$ . Then for any  $i$ , the output layer  $\mathbf{h}_v^{(L)}(i)$  of H2GCN over all nodes  $\{v_1,\dots ,v_{|V|}\} \in V$  follows Gaussian process with  $\mathbf{h}_v^{(L)}(i) \sim \mathcal{GP}(0,K_{\mathrm{H2GCN}}^{(L)})$  where the covariance function  $K_{\mathrm{H2GCN}}^{(L)}$  is given by
$$
K _ {\mathrm {H} 2 \mathrm {G C N}} ^ {(L)} (v, v ^ {\prime}) = \mathbf {C} ^ {(0)} + \sum_ {l = 1} ^ {L} \sum_ {j = 1} ^ {k} \left(\varsigma_ {b} ^ {2} + \varsigma_ {w} ^ {2} \cdot \left[ \mathbf {A} _ {\mathrm {H} 2 \mathrm {G C N}} ^ {j} \right] _ {v,:} \mathbf {C} ^ {(l - 1)} \left[ \left(\mathbf {A} _ {\mathrm {H} 2 \mathrm {G C N}} ^ {j}\right) ^ {T} \right] _ {:, v ^ {\prime}}\right)
$$
where  $\mathbf{C}^{(0)}$  is NNGP kernel [26] induced by ego-feature embedding layers.  $\mathbf{A}_{\mathrm{H2GCN}}^j$  is the adjacent matrix of the  $j^{th}$  neighborhood defined in H2GCN.
Proof. The jumping knowledge is leveraged in designing the H2GCN for heterophilic graphs. Thus, we show the graph Gaussian process of H2GCN at the output layer. The output layer is given by  $\mathbf{h}_v^{\mathrm{final}} = ||_{j\in \{0,1,\dots ,L\}}\mathbf{h}_v^{(l)}$ . Following Corollary A.3, the Gaussian process of each layer can be derived and then combined in the output layer.
A.6 Efficiency Analysis
As illustrated in Subsection 4.3 and Algorithm 1, in the context of transferable graph learning, the hyperparameters of GraphGP can be optimized by maximizing the following log marginal likelihood
$$
\log p (\mathbf {y} _ {t _ {l a}}) = \log \left[ \mathcal {N} \left(\mathbf {y} _ {t _ {l a}} | \mathbf {0}, \mathbf {K} _ {t (s + t)} \mathbf {K} _ {(s + t) (s + t)} ^ {- 1} \mathbf {K} _ {t (s + t)} ^ {T} + \varrho_ {t} ^ {2} \mathbf {I}\right) \right]
$$
The crucial idea is to consider all the nodes  $V_{s} \cup V_{t}^{la}$  as the inducing points [47].