SlowGuess/NeurIPS_2023_OCR huggingface dataset:
However, the objective function involves the inversion of the covariance matrix with time complexity  $\mathcal{O}((|V_s| + |V_t^{la}|)^3)$ , thereby resulting in the intractable computation of GraphGP in large graphs. To this end, we provide a computationally efficient approximation in both model training and inference. Following [37, 47], we choose a subset of landmarks  $V_{\mathrm{LM}}$  from the labeled source and target nodes. First, all the labeled target nodes are selected. Second, another  $q \cdot |V_t^{la}|$  landmarks are randomly selected from the labeled source nodes. Thus, the landmarks  $V_{\mathrm{LM}}$  have  $(q + 1) \cdot |V_t^{la}|$  nodes in total. In practice, a small value of  $q$  leads to  $(q + 1) \cdot |V_t^{la}| \ll |V_s| + |V_t^{la}|$ . Using the landmark nodes, the objective function of Eq. (7) can be approximated by
$$
\log p \left(\mathbf {y} _ {t _ {l a}}\right) = \log \left[ \mathcal {N} \left(\mathbf {y} _ {t _ {l a}} \mid \mathbf {0}, \mathbf {K} _ {t (\mathrm {L M})} \mathbf {K} _ {(\mathrm {L M}) (\mathrm {L M})} ^ {- 1} \mathbf {K} _ {t (\mathrm {L M})} ^ {T} + \varrho_ {t} ^ {2} \mathbf {I}\right) \right] \tag {11}
$$
Proof. It can be proven using a similar idea in [37].
Algorithm 1 GraphGP
Input: Source graph  $G_{s} = (V_{s},E_{s})$  with labeled nodes  $V_{s}$ , target graph  $G_{t} = (V_{t},E_{t})$  with labeled nodes  $V_{t}^{la}\subset V_{t}$ , number of neighborhoods  $k$ , layers of structure-aware neural network  $L$ .
Output: Predicted output values on target test nodes  $V_{*} \subset V_{t}$ .
1: Initialize the hyper-parameters of GraphGP;
2: while Stopping criterion is not satisfied do
3: Calculate the covariance matrices  $\mathbf{K}_{t(s + t)},\mathbf{K}_{(s + t)(s + t)}$
4: Maximize the log marginal likelihood of Eq. (7);
5: end while
6: Estimate the posterior distribution of Eq. 6;
7: Output the prediction results  $\gamma (V_{*},G_{t})$
where  $\mathbf{K}_{\mathrm{(LM)(LM)}}$  denotes the covariance matrix over landmarks, and  $\mathbf{K}_{t(\mathrm{LM})}$  is the covariance matrix between labeled target nodes and landmarks. Following [37], the Nyström approximation of the posterior distribution in GraphGP can also be calculated over the selected landmarks. The proposed approximation strategy would reduce the overall time complexity from  $\mathcal{O}((|V_s| + |V_t^{la}|)^3)$  to  $\mathcal{O}(|V_s||V_t^{la}|^2)$ .
A.7 Additional Experiments
In this section, we provide more details and additional results for our experiments.
A.7.1 Setup
Data Sets: We use the following graph learning benchmarks with node regression tasks.
- Twitch [44]: It has 6 different domains ("DE", "EN", "ES", "FR", "PT", and "RU"). In each domain, the graph represents the friendships among Twitch users, where each node corresponds to a Twitch user and each edge represents the mutual friendship of two users. Each node is also associated with an attribute vector encoding the game, location, and streaming habit information. Different from previous work [44] which considers the binary classification of whether a streamer uses explicit language, in this paper, we focus on the node regression task. Thus, we use the number of views in the original data<sup>5</sup> as the output value for each node/user.
- Agriculture [34, 60]: It has 3 different domains ("Maize" (MA), "Sorghum" (SG), and "Soybean" (SY)). The goal of agriculture analysis is to predict diverse traits (e.g., specific leaf area) of plants related to the plants' growth using leaf hyperspectral reflectance. The knowledge transferability across different species allows us to effectively infer the biochemical traits in real scenarios. Each data point represents a plant associated with a 1901-dimensional feature vector (e.g., spectral wavelengths  $500 - 2400\mathrm{nm}$ ). Following [55], we use  $k$ -NN to build the graph based on the feature similarity.
- Airports [43]: It has 3 different domains ("USA" (US), "Brazil" (BR), and "Europe" (EU)). Those domains involve the airport networks where each node corresponds to an airport and each edge represents the existence of commercial fights. Each node/airport is assigned a label based on their level of activity, measured in fights or people. Following [69], we use the degree-guided feature vector for each node.
- Wikipedia [44]: It has 3 different domains ("chameleon" (CH), "crocodile" (CR), and "squirrel" (SQ)). As introduced in [44], those domain data are given by Wikipedia page-page networks on different topics: chameleons, crocodiles, and squirrels. Each node corresponds to an article and each edge represents mutual links of two articles. The node feature is given by the presence of particular nouns in the articles and the average monthly traffic. It aims to predict the log average monthly traffic.
- WebKB [41]: It has 3 different domains ("Cornell" (CO), "Texas" (TX), and "Wisconsin" (WS)). It is collected from computer science departments of various universities by Carnegie
<sup>5</sup>https://github.com/benedekrozemberczki/MUSAE
Mellon University. Each node corresponds to a web page, and each edge represents the hyperlink of two web pages. The node features are given by the bag-of-words representation of web pages. The web pages are manually classified into five categories, student, project, course, staff, and faculty. In contrast to previous works [41], we would like to consider the regression task where each node is associated with an output value from  $[0, 4]$ , e.g., 4.0 for nodes from class 4.
The train/Validation/test split in our experiments is given below. As discussed in Subsection 4.3, for transferable node regression, it is given a source graph  $G_{s} = (V_{s},E_{s})$  with fully labeled nodes and a target graph  $G_{t} = (V_{t},E_{t})$  with a limited number of labeled nodes (e.g.,  $|V_{t}^{la}| \ll |V_{s}|$  where  $V_{t}^{la} \subset V_{t}$  is the set of labeled target nodes). Therefore, for Airport, Wikipedia, and WebKB data sets, we randomly select  $10\%$  of target nodes for the training set,  $10\%$  for the validation set, and  $80\%$  for the testing set. For Agriculture and Twitch data sets, we randomly select  $1\%$  of target nodes for the training set,  $1\%$  for the validation set, and  $98\%$  for the testing set.
In Subsection 5.1, we verify the positive correlation between transfer performance and graph domain similarity on Twitch (RU  $\rightarrow$  PT). To this end, we gradually add some noise to the target graph such that the graph domain similarity between the source and target graphs can change accordingly. More specifically, given the feature vector (node attribute)  $x_{v}$  of each node  $v$ , we simply multiply the feature vector by a constant  $\delta_0$ , i.e.,  $\hat{x}_v = \delta_0 \cdot x_v$ , where  $\delta_0$  denotes the perturbation magnitude. In the experiments, we choose  $\delta_0$  from 0.1 to 1 with the interval of 0.1.
Baselines: We consider the following Gaussian process baselines. (1) RBFGP [42] and DINO [56] are feature-Only Gaussian processes without using graph structures. (2) GGP [35], SAGEGP [37], and GINGP [37] are graph Gaussian processes by considering source and target graphs as a large disjoint graph. (3) LINKXGP, MixHopGP, and H2GCNGP are graph Gaussian processes derived from LINKX [30], MixHop [1], H2GCN [68] respectively.
RBFGP [42] is a standard Gaussian process algorithm designed for IID data. DINO [56] considers the transfer learning scenarios by designing the adaptive transfer kernel between source and target domains. By incorporating the graph structures, GGP [35], SAGEGP [37], and GINGP [37] design the graph convolutional kernels. For example, recent work [37] shows the equivalence between graph neural networks (e.g., GraphSAGE [20], GIN [59]) and graph Gaussian processes. Similarly, we can derive the corresponding Gaussian processes for LINKX [30], MixHop [1] and H2GCN [68] in Appendix A.5, which are termed as LINKXGP, MixHopGP, and H2GCNGP, respectively. It is notable that for a fair comparison, all the baselines are trained over both labeled source and target nodes. For example, graph Gaussian processes can take the input source and target graphs as a large disjoint graph. In this case, the distribution shift between source and target graphs is not considered.
A.7.2 Additional Results
Learned Weight  $\alpha_{i}$ : In Table 7, we report the learned weight  $\alpha_{i}$  of GraphGP in the target domain for different data sets. For homophilic graphs, e.g., EU  $\rightarrow$  BR of Airport, we see that the node itself and its first-order neighborhood have higher importance weights. In contrast, for heterophilic graphs, e.g., CO  $\rightarrow$  TX of WebKB, the first-order neighborhood is less important for transferable graph learning. Furthermore, when source and target graphs follow different assumptions, e.g., TX (WebKB)  $\rightarrow$  BR (Airport) or EU (Airport)  $\rightarrow$  TX (WebKB), the learned weights on the target graph can also capture the homophily or heterophily properties of the target graph.
Data	α0	α1	α2
EU → BR	0.9682	0.9118	0.3015
CO → TX	1.5201	0.0297	1.2054
TX → BR	1.1087	1.2848	0.1416
EU → TX	1.9503	0.0952	0.3609
Table 7: Neighborhood importance weight  ${\alpha }_{i}$  in the target domain
Impact of  $L$ : We further investigate the impact of the number of layers  $L$  in the proposed GraphGP algorithm. As shown in Table 8, GraphGP achieves relatively better results when  $L \in \{2,3,4\}$ . This observation is also consistent with the over-smoothing issues of conventional graph neural networks [20, 59].
Model	L=1	L=2	L=3	L=4	L=5
GraphGP	0.7434±0.0103	0.7909±0.0382	0.8001±0.0056	0.8028±0.0075	0.7788±0.0118
Efficiency Analysis: We investigate the computational efficiency of GraphGP and its approximation GraphGP-E. Table 9 shows the performance of GraphGP-E on the Twitch data set<sup>6</sup>. It can be seen that GraphGP-E consistently outperforms GINGP on transferable graph learning. Moreover, Table 10 shows the effectiveness and efficiency of GraphGP-E with different values of  $q$  on RU → PT of Twitch, where Full indicates the original GraphGP algorithm and the running time indicates the seconds per epoch. It can be seen that compared to GraphGP, GraphGP-E can achieve comparable performance and much higher computational efficiency.
Table 8: Impact of  $L$  on  $\mathrm{{RU}} \rightarrow  \mathrm{{PT}}$  of Twitch
Model	DE → RU	DE → PT	DE → ES	DE → EN
GINGP [37]	0.5159±0.0047	0.5201±0.0061	0.4861±0.0070	OOM
GraphGP-E	0.7012±0.0088	0.7351±0.0427	0.7356±0.0215	0.6916±0.0057
Table 9: Performance of GraphGP-E with  $q = {10}$  (OOM: Out of memory)
Model	q=1	q=2	q=3	q=4	q=5	q=6	q=7	q=8	q=9	q=10	Full
R²	0.4806	0.6064	0.6781	0.7315	0.7519	0.7495	0.7598	0.7626	0.7887	0.7960	0.7909
Time (s)	0.2169	0.2211	0.2281	0.2330	0.2406	0.2713	0.2763	0.2808	0.2871	0.2889	0.8761
Table 10: Efficiency
The domain "FR" in Twitch is not used in the experiments, because it has some missing output values, i.e., the number of views.