Graph-Structured Gaussian Processes for Transferable Graph Learning
Authors: Jun Wu
Lisa Ainsworth
Andrew Leakey
Haixun Wang
Jingrui He
Venue: NeurIPS 2023 poster (2023)
Keywords: graph learning
transfer learning
Gaussian process

Abstract: Transferable graph learning involves knowledge transferability from a source graph to a relevant target graph. The major challenge of transferable graph learning is the distribution shift between source and target graphs induced by individual node attributes and complex graph structures. To solve this problem, in this paper, we propose a generic graph-structured Gaussian process framework (GraphGP) for adaptively transferring knowledge across graphs with either homophily or heterophily assumptions. Specifically, GraphGP is derived from a novel graph structure-aware neural network in the limit on the layer width. The generalization analysis of GraphGP explicitly investigates the connection between knowledge transferability and graph domain similarity. Extensive experiments on several transferable graph learning benchmarks demonstrate the efficacy of GraphGP over state-of-the-art Gaussian process baselines.

Graph-Structured Gaussian Processes for Transferable Graph Learning
Jun Wu', Elizabeth Ainsworth 1.2 Andrew Leakey', Haixun Wang Jingrui He
University of Illinois at Urbana-Champaign
2USDA ARS Global Change and Photosynthesis Research Unit Instacart {junwu3, ainswort, leakey, jingguijnnnlliiiiim. edu, (haixun)@gmaill com
Abstract
Transferable graph learning involves knowledge transferability from source graph to relevant target graph. The major challenge transferable graph learning is the distribution shift between source and target graphs induced by individual node attributes and complex graph structures. To solve this problem, in this paper, propose generic graph-structured Gaussian process framework (GraphGP) for adaptively transferring knowledge across graphs with either homophily or heterophily assumptions. Specifically, GraphGP is derived from novel graph structure-aware neural network in the limit on the layer width. The generalization analysis of GraphGP explicitly investigates the connection between knowledge transferability and graph domain similarity. Extensive experiments on several transferable graph learning benchmarks demonstrate the efficacy of GraphGP over state-of-the-art Gaussian process baselines.
Introduction
Transfer learning 54 aims at studying the transfer of knowledge or information from source domain a relevant target domain under distribu- tion shifts. The knowledge transferability across domains has been investigated from various aspects. proposed to empirically estimate data-level domain discrepancy for mea- suring the transferability across graphs. [8] 449
For example, 33
15 67
(4)
(t)
(c)
adopted adaptive Gaussian processes with trans- fer kernels, which formulated the task relatedness
Figure I: Illustration of transferable graph learn- ing. (a) Samples independently drawn. (b) Samples are connected in each domain, and both domains follow the homophily assumption. (c) Source and target domains follow different graph structure assumptions.
as a hyper-parameter for tuning. [22 36 (33 in-
stead evaluated the transferability of pre-trained source model in the target domain. However, most existing works followed the IID assumption that samples are independent and identically distributed in each domain. This might limit their capacities in understanding the knowledge transferability across domains with non-IID data, e.g., cross-domain recommendation [55], cross-network role identification 669, etc.
In this paper. we focus on the problem of transferable graph learning over non-lID graph data. The challenge of transferable graph learning induced by the graph structure information be explained in Figure It can be seen that Figure a) provides source and target samples associated with input features and output class labels (indicated by node colors). Suppose that source and target
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

distributions have been aligned (e.g,, optimized by domain discrepancy minimization algorithms [15 67], Figure Ta) implies that source and target domains have similar data distributions in the feature space, thereby enabling feasible knowledge transfer across domains. Compared to Figure [ Figure[ introduce additional graph structure information. In Figure[[t) source and target graphs follow similar structures where nodes within the same class tend to be connected (i.e., the homophily assumption [59]. In contrast, the structures of source and target graphs fundamentally differ in Figure (c).
c). That is, though the target graph follows the homophily assumption, the source graph holds the heterophily assumption that connected nodes may have different classes and dissimilar node attributes [68]. Therefore, compared to standard transfer learning [I 40] with IID data, transferable graph learning is much more challenging because knowledge transferability across graphs can be determined by both individual node features and complex graph structures.
In recent years, transferable graph learning has been studied [55
where both source and target graphs follow the homophily assumption (Figure I [b).. However, little effort has been devoted to investigating the knowledge transferability between homophilic and heterophilic graphs (Figure ()). To bridge this gap, in this paper, we focus on studying the knowledge transferability across graphs with either the same (e.g., homophily or heterophily) or different assumptions.
58
66
69
We start by introducing a universal structure-aware neural network to build the input-output rela- tionship between structure-awe aware input sample (v, G) and its associated output value ye for tackling homophilic and heterophilic graphs. This model encodes both local node representation (sample-leve)) and global graph representation (domain-level) simultaneously. The crucial idea is to automatically select relevant neighborhoods by adding learnable weights to neighborhood aggregation. Thus, it enables knowledge transferability between homophilic and heterophilic graphs by adaptively selecting neighborhoods for each graph. The intuition is that source and target graphs might have common knowledge from the selected neighborhoods, e.g., nearby neighborhoods for the homophilic graph and high-order neighborhoods for the heterophilic graph. Then, we show the equivalence between structure-awar neural network and graph Gaussian process in the limit on the layer width. This obser- vation sheds light on building graph-structured Gaussian processes (GraphGP) for transferable graph learning. Moreover, the generalization analysis of GraphCP shows the connection between knowledge transferability and graph domain similarity. Compared to previous works [45] 58 69), GraphGP benefits from (i) fcasible knowledge transferability between homophily and heterophily graphs, and (ii) flexible incorporation with existing graph neural networks and graph kernels. Extensive experiments on node regression tasks demonstrate the effectiveness of GraphCP over state-of-the-aat Gaussian process baselines. The main contributions of this paper are summarized follows.
A generic graph-structured Gaussian process framework GraphGP is proposed for transfer- able graph learning. It is derived from structure-awan neural network encoding local node representation (sample-level) and global graph representation (domain-level) simultaneously.
We show that GraphGP tackles the knowledge transferability in homophily and heterophily graphs using a simple neighborhood selection strategy. addition, we theoretically analyze the knowledge transferability across graphs from the perspective of graph domain similarity.
Experimental results variety of graph learning benchmarks demonstrate the efficacy of GraphGP over statc-of-the-an Gaussian process basclines.
The rest of this paper is organized as follows. Section 2 summarizes the related work, and Section provides the problem definition of transferable graph learning. In Section [ we present the graph- structured Gaussian process (GraphGP) for transferable graph learning. The experimental results are provided in Section 5 Finally, we conclude the paper in Section
2 Related Work
Transfer Learning:
tackles the transfer of knowledge or information from Transfer learning 40 54
a source domain relevant target domain. is theoretically shown (3,67] that the generalization performance of a learning algorithm can be improved by leveraging latent knowledge from the source domain. Specifically, the generalization error of learning algorithm on the target domain can be bounded in terms of the source knowledge and the distribution discrepancy across domains.
Recently, transfer learning has been explored in non-III graph data 77] 55 58 699. However, most existing works focus either investigating the transferability of pre-trained graph neural

---

network (27 45 69 or designing graph transfer learning algorithms 66) with relaxed assumptions
k-hop ego- ago-graphs are assumed to be independent and identically distributed [69].
Gaussian Process:
provides a principled nonparametric framework for learning stochastic functions from observations. It has been applied to transfer learning when samples in each domain are independent and identically distributed [8] 49 [6]. The crucial idea is to infer the task relatedness between source and target domains using available labeled source and target examples. More recently, Gaussian processes have also been investigated in graph learning tasks, e.g. link prediction 14
Gaussian process
65 and node regression
35 37 39, However, little effort has
been devoted to understanding transferable graph learning from Gaussian process perspectives.
3 Preliminaries
3.1 Notation and Problem Definition
Suppose that graph is represented as G = (V, E), where V = {.i...., [iv]) is the set of n nodes and E S V x is the edge set in the graph. Each node v E V can be associated with D-dimensional feature vector X, E D. In the node regression tasks, the ground-truth output of each node is denoted by ye E R. The structure of a graph G can also be represented by an adjacency matrix A E IR [ where Aij the edge weight between v and v/ within the graph. Following [55] we introduce generic problem setting of transferable graph learning as follows.
Definition 3.1 (Transferable Graph Learning). Given source graph G. = (V., and target graph G, (Ve, Et), transferable graph learning aims to improve the prediction performance of graph learning algorithm in the target graph using latent knowledge from a relevant source graph.
3.2 Message-Passing Graph Neural Networks
Weisfeiler-Lehmaal (WL) graph subtree kernel 46 measures the similarity of two input graphs, inspired by the Weisfeiler-Lehmaal test of isomorphism 53 The crucial component of WL. subtree kernel is to recursively represent the subtree structure rooted at each node. This has motivated variety of pessage-passingg graph neural networks (GNN), which recursively learn the node presentation of
node v by aggregating the feature vectors from v's local neighborhood [17 [8] 20 [5 [59] Generally, a graph convolutional layer of message-passian GNNs can be summarized as follows.
1 (I+1) COMBINE())
AGGREGATE(" (n.P9 |u Ni(o)):
Seed representation
1 1-order neighbothood representation
AGGREGATE (1) (nh E Nu(e))
I:conder neighborhood representation
where COMBINE(!) function compresses the representations from the node itself and its neighbors
at the 1th graph convolutional layer into a single representation, and AGGREGATE," (j = 1.. .k) function aims to aggregate message from j-order neighborhood N2(v) of node v. It indicates that there are two major components for designing graph convolutional layers in GNNs: neighborhood selection and aggregation. More specifically, neighborhood aggregation allows compressing the graph structure and node attributes simultaneously, while neighborhood selection provides the flexibility of GNNs in tackling graphs with different assumptions (e.g., homophily (20 or heterophily (68)).
Homophily graphs:
Homophily graph holds the assumption that nodes within the same class tend to
be connected in the graph, e.g., citation networks [33, social networks (33] etc. This assumption
has motivated various instantiations of message-passing graph neural networks over first-order
neighborhoods, c.g., GCN (25). GraphSAGE (20). GAT [50]. GIN [59].
Heterophily graphs: Heterophily graph holds that connected nodes may have different class labels and dissimilar node attributes, e.g., different classes of amino acids are more likely to connect within protein structures 30 68). Recently, by adaptively exploring the potential homophily in high-order local neighborhood (e.g., k > 1), message-passeeeee graph neural networks have been proposed for heterophily graphs, e.g., MixHop [I], H2GCN (88) GPR-GNN [T]. HOG-GCN [22. GloGNN 29
3

---

4 Methodology
In this section, we propose the graph- structured Gaussian processes for transferable graph learning.
4.1 Structure-Aware Neural Network
We start by showing the graph sampling process from a probability distribution space P. A graph G = (V, E) sampled by first sampling graph domain distribution Pg from P, and then sampling a specific graph G' from Pg. More specifically, one realization from 9 is graph domain distribution Pg (also denoted as Pg = (I Pε)) for node and edge sampling distribution) characterizing two- stage graph sampling as follows. A set of nodes V is sampled from the graph distribution Pg (more specifically, sampled from Py), and then the edge weight sampled from Pg (more specifically, sampled from IP')) over any pair of nodes in V. In addition to the graph structure (induced by the node dependence), the output label of a node can also be sampled accordingly in the context of node regression tasks. That is, label-informed graph domain distribution Fg.y is sampled from P. Next, a graph G = (V, is sampled from Pg = (Pvv Pg)) ) and then the output labels of nodes are sampled from py.g. The goal of node regression task [55 69
is to learn prediction function f()
that predicts Dt f(v, G') for each node E V.
In this paper, we consider the covariate shift assumption 40 over graphs. That is, the source and target graphs share conditional distribution yyng (y (v, G)) but different marginal distributions (v G). The aforementioned graph sampling process motivates us to learn the input-output relationship between an input sample (G, v) and its associated output ye for transferable graph learning. To this end, for each input graph G (e.g., source G2 or target graph G, transferable graph learning). we define a structure- aware neural network f(; -) as follows.
1 M
G)
w (1) ij
13 (vlG) (G)
VM
(v = h())
and (G) = READOUT
( (
where M is the layer width D p (w(G) denotes the node (sample-level) representation of given the graph G at the 1hh layer, and (G) denotes the graph (domain-level) representation of the graph G. Here READOUT function compresses set of node representations into single vector representation [59] 62). This definition indicates that the embedding representation of the input (e. G) (e.g.. a pair of node e and its associated graph G) given by two crucial components: an individual node embedding representation ( (u|G) and a global graph embedding representation ( (G). a and R.! represent graph neural networks to learn node and graph representations separately.
The intuition behind this definition is explained below. The graph G is a realization of graph domain distribution Pg. Then, the embedding representation of the graph G can be explained an empirical estimate of the distribution Pgg. i.e., (/)) R v(Pg). Moreover, when the size of the graph G goes to infinity (i.e., associated with infinite number of nodes), G approximates the true distribution Pg and the embedding representation of the graph G will be able to recover the true embedding of probability distribution Pg [19 48). Besides, the embedding representation of a node can also be explained as an approximation of Pg at the location v, i.e., (u R (u
Remark. The structure-awart neural network Eq. (2) is defined over message- message-passing graph neural networks commonly used for various single-domain graph learning tasks [I8] 20 23 59]. Different
from previous works, we focus transferable graph learning scenarios across graphs. This motivates us to consider the task relatedness between source and target graphs, which can be explicitly measured by the domain distribution similarity [15 56]. To this end, design the structure-awan neural network Eq. (2) for learning domain aware node presentation, i.e., the integration of the node (sample-level) representation and the entire graph (domain-level) representation. Specifically, can show the connection between graph domain distribution and graph representation learning in the reproducing kernel Hilbert space for cross-graph learning tasks (see Corollary 4.4).
Here, "layer width" indicates the number of neurons in graph convolutional layer.

---

4.2 Graph-Structured Gaussian Process
Inspired by [ 52 68 we derive universal peessag-passiing graph neural network Eq. ① for both homophily and heterophily graphs. More specifically, the ressage-passing graph convolutional layer of Eq. ① for node within an input graph G can be defined as follows.
h
02
Σ
w
1 S + +SSANN
OB
and x = (h (b
(3)
VM mEN((2)
=0
where X = Xu E IR,, No(v) = {u} denotes the seed node, W bias parameters, and aę explicitly indicates the importance of the i-order neighborhood in finding relevant neighbors around node v. Similarly, we define the domain-level graph neural layer h!
SANN- SANN are the weight and
parameterized by different w 6
SANN: SANN and shared
6
Will (EN.(e)
Σ
TO S + SANNN
and x Φ (b(-1)
a2
VM
Generally, it is flexible apply different model architectures to learn the node (sample-level (v) G)) and graph (domain-level (!)) representations. For simplicity, adopt the same model architecture but different model parameters to learn node-level and graph-level representations. It is noteworthy that the neighborhood importance a; is shared because both node and graph repre- sentation would share the same assumption (homophily or heterophily) for a given graph. Then, the structure-aware neural network Eq.
be given by
M
(c,
VM O
Σ
(v(G) (G)
==1
(5)
where n (u = h and (G) = V
Σ EEV
K0
where the READOUT function of Eq. 2 is instantiated with mean pooling (62. Different from previous we would take the neighborhood importance scores ag as syper-parameter for building transferable graph Gaussian process model as follows.
works [T] 52
Theorem 4.1. Assuming that all the parameters of structure-aw graph neural network f(u, G) are independent and randomly drawn from Gaussian distributions, i.e.. W() (O, o b (I) [!]] SANN (N 1), SANNN NN 1),, Wi()
SANN SANN _NO0, 22 when the layer width M goes to infinity, the output function f(()) in Eq. (5) follows a Gaussian process with
NO, I) wil)
1(0) GD, K(O), where the covariance function K is given by
K G), ())) - K(O) (v,"'G,, K(0) (G.G")
where
K0 (u,''G,,G)) Σ
ajdy
+ Σ Σ
cl-21) (cu.58) st:)
:
wEN((N) W'NN((1))
K G,G = Σ
agay
2 VI.
1' C(- 22) (pris)" 1
V
c(I1)) = E_-- 1) (0.x%
(6(5! 10 [11))
Here p(ù) (P'6)) denotes the adjacency matrix given by the i-order neighborhood from graph G (j-order neighborhood from graph G'). (((-1) k Li (12 ((1-)) ()('())))) and (00) b) (xu) for any u E v, V
5

---

4.2.1 Implications of Theorem 4.1
In the following, we show that the sample-level covariance/kernel k2 of Theorem 4.1 can be explained as a ceassage-passingg operation in the kernel space. Following 37, when using ReLU as activation function <6-) (i.e., <(x) = max(0, (0))),))) C(l-1) the arc-cosine kernel [2 as follows.
uu (Swi 56
K1 (5)
V Ky (1-1) (u, ulG) K (((-1) (ul, ,"(G")
2
where
V VKA (M.uG)). (wr and 11 (5)
K "(,,a'(G,'')
1 ( (5- aarcoss((())) vI-
Then, the following corollary derives the feature map of K/x (u, G. G') in the kernel space.
Corollary 4.2. Let 9p + Hi denote the kernel mapping from a pre-activation RKHS H. to post- activation RKHS Hi, i.e., (p)(s), 4p (p((s)) ||l ||s''|
(5.5")
Given the sample-level
(T>IFlI>
kernel KP (v,,e' | G. in Theorem 4.1 if the graph convolutional layer of Eq. (SB = 0} the feature map of this kernel is given by
has no bias term
1
Σ (EN,(w)
Pi (
and T
Σ aa 0m
Σ WEN:(("
(w-l-1)
Sw
Sw
0
with Kj (u, "e,'' (v!
Corollary 4.2 shows that the feature map !9 of sample-level kernel K(O) is kernelized version of the graph convolutional layer in Eq. 33. which recursively aggregates message from nodes' neighborhood in the kernel space. The neighborhood can be automatically selected by optimizing the hyper-parameters a and a';.
Next, we show the implication of domain- kernel KP (G, G'). As discussed in Subsection 4.1 graph G, empirically drawn from graph domain distribution I'g, can approximate the true distribution Pg when increasing the size of graph G. This motivates us to understand the connection between empirical domain representation (0) (G) and the embedding of probability distribution Pg
19 48
Definition 4.3 (Mean Embedding of Probability Distributions 19). Given an RKHS H. induced by kernel -(. the mean embedding of a probability distribution P is defined as тp k(-, xxxx (x). (f, T2)H H for all f E H. Furthermore, the empirical mean embedding of I is Σ ii= x.), where m samples are independent and identically drawn from IP.
i..., Of
given by Ah
Analogously, the mean embedding of graph domain distribution Pg can also be defined as Tre = kgg., [ given an RKHS Hg induced by a kernel kg(:,)) :). the context of graph learning, u uP indicates the individual node attributes of and graph strueture information around node v induced by Pg. It subsumes the conventional mean embedding 19 of a probability distribution pp with independent and identically distributed (IID) samples
P. if there are edges in the graph (i.e., no graph structure is involved around node v). It is observed that the additional graph structure challenges empirical estimation of graph domain distribution Ps, thereby resulting in nontrivial domain similarity estimator between source and target graphs in transferable graph learning. The following corollary states that the graph representation (1) (G) can be considered as the empirical mean embedding of TFe in the RKHS induced by KD (..))
Corollary 4.4. With the conditions in Theorem 4.1 for each in the limit on the layer width, the graph representation ( (G) recovers the empirical mean embedding 77o of the domain distribution Fg in the reproducing kernel Hilbert space induced by (!!). and a!!) is given by
1
S
Σ
(,ulG)
V
V
VEV
VEV
where ( the feature map of k with (1)
Eo Σk= ag EveNy(o) 81 (i(-1))

---

Corollary 4.4 shows that ( (G, G*) in Theorem 4.1 can be explained as the similarity of the empir-
ical mean embeddings of domain distributions P& and Pg in the kernel space e.g., K2 (G, G*) = (4(0) (11),
(T "Pg. 22g. w: A similar metric over distributions is the maximum mean discrepancy (MMD) 19
defined over the distance of empirical mean embeddings, i.e., MMD 11
( or 1 x/0"
where H. k denotes the RKHS induced by K2 MMD has been widely applied to transfer learn- 61 for measuring the distribution shift across image domains, under the covariate shift assumption 40 (i.e., source and target domains share conditional distribution p(ylar) ) but different marginal distributions p(a))). In contrast, in this paper, we focus on measuring the domain distribution similarity for transferable graph learning.
ing
[T 32
4.2.2 Homophily vs. Heterophily
As illustrated in Section 3.2 there are two assumptions in graph learning 200 68 : homophily and heterophily. We show the special cases of Theorem 4.1 in tackling homophily or heterophily based transferable graph learning. Given homophily source graph G' and target graph G" where connected nodes have similar output values, it revealed [20 25 [9] that each node aggregates message from itself and its 1-order neighborhood, e.g.. k = 1 and og a1 a af = 1. Then the sample-level C(l-1) In contrast, if source and target uuu
kernel K (u, e' G*) & Eue(ouNitu En'geeui
graphs follow heterophily that connected nodes are not similar in the output space, nodes might aggregate message from high-order neighborhood [ 52 688. e.g., k and ag 02 a,
0. Then it holds that K (v, (, G") x [PuN(a(())) Eer((NNNa(o)))) c(l="), These results uw
indicate that by learning the neighborhood importances a and a,, Gaussian process led by f(e, G) can adaptively select the neighborhood for message aggregation from homophily or heterophily graphs. Moreover, a; a implies that source and target graphs follow the same assumption. The flexibility of a; a allows us transfer knowledge across different types of graphs. For example, using k = 2, 0gp = 0n = 1, 02 = 0, a an 0, a' 0, it enables the knowledge transferability from a homophily source graph to a heterophily target graph.
4.3 Proposed Algorithms
The goal of transferable node regression is to learn the prediction function for the target graph, by leveraging the input-output relationship from relevant source graph. Given source graph G. = (V., F..) with fully labeled nodes (e.g.. Ue E IR is associated with each node v E VV)) and a target graph G. = (Ve, Ex) with a limited number of labeled nodes (e.g., [V/l| << [V&] where V/da c V the set of labeled target nodes), we propose the adaptive graph Gaussian process algorithm (termed as GraphGP) as follows.
For notation simplicity, we let fo (1 G)) (given by L graph convolutional layers in Eq. (5) be the function value at node Let f, = [f,, f2..... fiv, be vector of latent function values over labeled source nodes, and y, [y1, [y1. y2, ' v/, [ be the associated ground-truth output values. Similarly, we can define f, and y, over target nodes. Then, the GP prior over function values can be defined as f ~ GP(O, K(E)(..) (defined in Theorem 4.1). and its instantiation at labeled training nodes is given by p (f]V, U V//) = (0, (()t))(+))) where ((++(0+4) is a block K.s
K and its entry is [Katle,e. = (< Ga), (V6, Gb)) for K
matrix, i.e., K (a++)(++) = K
a. E {s. For the likelihood, we consider the noisy scenarios where p(y, (.)) = Nf... oa for
E {s,4} (ea measures the noisy magnitude). Then, following standard Gaussian process 42 the posterior distribution of GraphCP over testing nodes V. C V2 has a closed-form expression, i.e.,
VV. f, V. V/a) =N(r,, I) where
7- K. ((o+))(4+1)
I 0
r-K..-K K. (K(+t)(e44) +
K. (6)
e
y, denotes the ground-truth output values of labeled nodes V, U Vila from source and Yt..
Here, y
target graphs. Each entry of K. represents the covariance between testing target node and training node, and K.. denotes the covariance matrix over testing target nodes.
7

---

The posterior distribution of GraphGP has the following hyper- rrpee-paramttes: ow: Swi Sb. af oz. and noise variances 2s. ft. The apper-parameter of the Gaussian process can be optimized by maximizing the marginal likelihood p(y) over all the training samples 42 However, in the context of transferable graph learning, the number of labeled target nodes is much smaller than the number of labeled source nodes. By directly maximizing the marginal likelihood over all labeled source and target nodes, GraphGP might be biased towards the source domain. Therefore, in this paper, we propose to optimize the marginal distribution p(yt,.) over labeled target nodes by considering all the nodes V. U V/de as the inducing points 47 (see more efficiency analysis in Appendix A.6). The objective function is given by maximizing the following log marginal likelihood
logp(yn.)) = log [N (X 10, ((s++) K (o+1)040)) K ((tt))) + etii
4.4 Generalization Analysis
We define the generalization error of the target graph for transferable node regression. Given the ground-truth labeling function f in the target domain and the estimator f(L) learned from observed source and target graphs, the generalization error is given by
E: - / < (7(00) Ge), (T(e,
G:)) G))(S")d(vr, Gt)df"
(8)
where 7()) is the posterior mean of (L) in Eq. 6 L is the mean squared loss function, and p(u,, G+) denotes the sampling probability? of a target input (vz, Gt). is shown 9 42 that if the GP prior is correctly specified, i.e., the predictor f(() has the same prior as f", the generalization error is given by the predictive variance of the GP. That is, e, = [T,,,, G.) ()), G. (((d, G.) where T(we, G.) is the posterior variance at (wt,Ge) G+) in Eq. 66. Furthermore, the following theorem shows that the generalization
can be upper bounded terms of the variance that takes all inputs (U,, G.) of the source graph as the target samples, i.c., both (v,, G.) and (v,, G₂) are assumed to follow the same target distribution for single-domain graph learning.
Theorem 4.5, Let K be the sample-level covariance matrix over source nodes, i.e., [K (os.e) K (Us, L.((G)), V.s = k² (Ge, Gz), Vu = K.!" (G1, ((G,,G), Gt) be the intra- intra-graph graph kernels and Vst = K(L) (G., Gi) be the inter-graph kernel. Suppose o² (ou -1) o vudes + Du da where Aeo is the maximum eigenvalue of K.s.for for any (ve, G.) the generalization error bounded by
varva
et S fTA Go th Eppe, Gi)d(ve, G.)
where 1,(2,, G..z Ø) is the variance assuming that all source examples are observed in the target domain with respect noises Ø² o²
This theorem confirms that compared to the single-domain Gaussian process (assuming all inputs are drawn from the same distribution), GraphGP-basee transferable graph learning enables the reduced generalization error. In addition, the following corollary considers a special scenario where no labeled nodes are available in the target graph. In this case, the generalization error is determined by the normalized graph domain similarity
K(L) (G,,
(G,")(G,G.)) (G.,G.) (G.
Corollary 4.6. When there are no labeled nodes in the target graph, i.e., V we have
lim (1 |V/1-000
)
k(O) ((v,, Gc), (v,, G()) plu,, Ge)d(ur, G.)
(G,, where U.s K! (G., G.), Ve K!! (T) (Gi, (G,, Gi) and vaa Kg ( G.).
Herre it holds p(u,, Gt) = P Po Po. (v:) as discussed in Subsection 4.1 i.e. it first samples graph domain distribution Fg, and then sample the nodes/edges to form the graph Gt.

---

Airport graphs
Agriculture graphs SG - MA
Model
BR - EU
EU -> BR
BR - US
MA -> SG
MA -> SY
RBFGP 42
0.4849 4000200
0.4479=a.22
0.3682 +0.000
0.4859 40.016
0.3359 +-0,0297
0.7314
DINO 56
0.5241 100147
0.4855
0.3877=t000
0.6227-t022
0.4591 100044
0.7620=10012
GGP 35
0.3400
0.4720 0.3990=0.040
0.451540000
0,2403
0.7420
SAGEGP 37
0,4581 1 1000008
0.3822
0.4928 0330
0.5348
0.3633
20.010
0.7632.10019
GINGP 37
0.52166
0.4471
0.0219
0.4901 +:0025
0.3559
0.0019
0.7746
GraphGP
0.5567 4000246
0.4983 0.0377
0.5293
0.6586
0.5125 4000171
0.7921 00000N
Table 1: Results of transferable node regression on Airport and Agriculture data sets
Model
PT - RU
EN + PT
RU + ES
RU + PT
PT + EN
ES RU
RBFGP 42
0.5430 0.4324-00087
0.4241
0.4962
0.5507
0,4738
DINO 56
0.6498 +0.009
0.7565 40.040
0.6717 +0.0144
0.7641 I +00002
0.5920
0.5636 +00002
GGP
35
0,3167 +0.079
0.5689+a.0zz
+0.028
0.3699
0,5173
+00001s
0,4237 +00007
0,4237 +0000s
SAGEGP 37
0,5655 40019
0,5191
+000299
0.59584400006
0.692 +00174
0.5720+0our7
0.5960+000m4
GINGP 37
0.5782
0.7525
+o0029
0.6803 40.0199
0.6910
0.6349
+00000
0,5997
GraphGP
0.7069 2.0055
0.8013 e.0000
0.7520
0.7909 1000322
0.6745 1.01127
0.6789
Table 2: Results of transferable node regression Twitch data
Model
Wikipedia graphs
WebKB graphs
2.76
SQ
CH
CH
SQ
CO + TX
WS
- TX
179
5/8
RBFGP
-0,0383 20.0066
-0.0119
+0008
0.008
0.2756=n0224
2 km
DINO
0.2938
200253
0.1157
0.3536-00442
0.2537
200232
1675
LINKXGP
0,4100
0,0200
0.0838 +00000
0,3836
0,2898 +0.0176
MixHopGP
0,4050
+0,456
0.3509 + 000000
0.3264
+ou0569
0.3062
H2GCNGP
0,4165 -:03399
0.2652
0.38 1 4.00000
0.3036
GraphCP
0.4938
0.321420.0080
0.4146
0.3301
Graph Domain Sinilarity
Table 3: Results on Wikipedia and WebKB
Figure 2: Domain similarity
Experiments
Data Sets: We use the following graph learning benchmarks with node regression tasks. (1)
Twitch [44]: It has 6 different domains ("DE", "EN", "ES", "FR", "PT", and "RU"). (2) Agricul-
ture 34 60 It has 3 different domains ("Maize" (MA), "Sorghum" (SG), and "Soybean" (SY)).
(3) Airports 43 : It has 3 different domains ("USA" " (US), "Brazil" (BR), and "Europe" (EU)). (4)
Wikipedia 44 It has different domains ("chameleon' (CH), "crocodile" (CR), and "squirrel" (SQ)).
(5) WebKB 41 It has 3 different domains ("Corelll" (CO), "Texas" (TX), and "Wisconsin" (WS)).
Baselines: We consider the following Gaussian process baselines. (1) RBFGP 42 and DINO (56 are feature-Only Gaussian processes without using graph structures. (2) GGP
SAGEGP 37
and GINGP 377 are graph Gaussian processes by considering source and target graphs large disjoint graph. (3) LINKXGP, MixHopGP, and H2GCNGP are graph Gaussian processes derived from LINKX 30J. MixHop M. H2GCN [68] respectively. is notable that recent work [37] shows the cquivalence between graph neural networks (e.g., GraphS AGE (20, GIN (59) and graph Gaussian processes. Similarly. we can derive the corresponding graph Gaussian processes for LINKX [30]. MixHop [1] and H2GCN [68] in Appendix A.5) which are termed as LINKXGP, MixHopGP, and H2GCNGP, respectively.
Model Configuration: In the experiments, we use GPyTorch [6] to build the graph Gaussian process models and optimize the hyperparameters with gradient descent optimizer. For the proposed GraphGP algorithm, we adopt k = and L = 2 for all the experiments. The hyperparameters are optimized using Adam (24) with learning rate of 0.01 and total number of training epochs of 500. All the experiments are performed on a Windows machine with four 3,80GHz Intel Cores, 64GB RAM, and two NVIDIA Quadro RTX 5000 G
5.1 Results
CCode is available at buttps::/gitiii comm/jwudemm//Gaapppp /tttopr////tttticccc/////////ttt

---

Table[Tand 2 the transferable node re- gression results on Airport, Agriculture, and Twitch graphs. Following [7], we report the coefficient of determination R (mean and standard deviation with 5 runs) on the testing target nodes for per- formance comparison. The experimental results verify the effectiveness of GraphGP over Gaussian process baselines, is notable that the graphs in T Table2followw the homophily assump- tion. contrast, Wikipedia and WebKB graphs in Table 3 hold the heterophily assumption [68] It can be seen that under heterophily assumption, GraphGP also archives much better performance in most cases. In addition, we evaluate the knowl- edge transferability across graphs with different assumptions in FigureB] It observed that compared to GINGP (or H2GCNGP) over both labeled source and target nodes, GINGP-T (or H2GCNGP-T) learns the Gaussian process over only target nodes, thereby enabling much better transfer performance from WebKB Airport in Figure 3a (or from Airport to WebKB in Figure Bb). Moreover, GraphGP can outperform those target-only graph Gaussian processes in this scenario.
(a) WebKB - Airport
(b) Airport + WebKB
Figure 3: Knowledge transfer across graphs with different assumptions
We also investigate the correlation between transfer performance and normalized graph domain simi- (G,
larity- (G. (G. domain similarity GraphGP and R over the target testing nodes for RU + PT on Twitch, where the graph domain similarity changes when injecting noise to the target graph (see Appendix (A..1)). shows that the transfer performance is positively correlated with the normalized graph domain similarity. This is consistent with our observation in Corollary 4.6
in Corollary 4.6 Figure visualizes the estimated normalized graph
5.2 Analysis
Flexibility of GraphGP: As illustrated in Subsection 4.1
it is flexible to instantiate structure-aware neural networks of Eq. 2 with existing message-passing passing graph neural networks. Table the results of GIN [59] induced GraphGP algorithm. Furthermore, is feasible to simply instantiate K!') (G,G") of GraphGP with existing graph kernels, e.g., Shortest Path kernel (5). It observed that the variants GraphCP GIN and GraphCP ShortestPath of GraphGP achieve comparable performance. This highlights that GraphGP is flexible incorporate with existing GNNs and graph kernels for transferuble graph learning tasks.
Methods
BR -> EU
GraphCP
0.5567 400246
GraphGP_CG __
0.5301 +0.029
GraphCP Shortesttat
0.5377 100004
Table Flexibility of GraphGP
Comparison with Transferable GNNs:
In addition to Gaussian processes, we also compare GraphGP with state-of-the- art transferable graph neural networks. Ta- ble shows the competitive performance of GraphGP over transferable GNN base- lines. This is because GraphGP explicitly maximizes the marginal likelihood ply..... In contrast, transferable GNN baselines minimize the prediction loss over all the labeled nodes (VV, > [V/*[) and domain discrepancy, and thus they are more likely to bias towards the source graph.
Methods
BR + EU
EU + BR
BR + US
EGI 69
0.5204100357
0.4786 0.4951 100176
GARDE
55
0.5314 0.4792
0.4354
GraphGP
0.5567+a0246 0.4983
0.5293 +000s
Table Performance comparison between GraphGP and transferable GNNs
6 Conclusion
This paper studies the problem of transferable graph learning involving knowledge transfer from a source graph to a relevant target graph. To solve this problem, we propose a graph Gaussian process (GraphGP) algorithm, which is derived from structure-aware neural network encoding both sample-level node representation and domain-level graph representation. The efficacy of GraphCP verified theoretically and empirically in various transferable node regression tasks.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by National Science Foundation under Award No. IS-1947203, USS-2117902 IIS-2137468, IIS-2002540, and Agriculture and Food Research Initiative (AFRI) grant no. 2020- 67021-32777 poojpoo accession no. 1024178 from the USDA National Institute of Food and Agricul- ture. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.
References
[1] Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In International Conference Machine Learning, pages 21-29, PMLR, 2019.
[2] Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathemati- cal Society, pages 337-404, 1950.
[3] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning. 79:151-175, 2010.
[4] Alberto Bietti and Julien Mairal, On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems, 2019.
[5] Karsten M Borgwardt and Hans-Peter Kriegel. Shortest-path kernels on graphs. In Fifth IEEE international conference on data mining (ICDM'05), pages 8-pp. IEEE, 2005.
[6] Viacheslav Borovitskiy, Iskander Azangulov, Alexander Terenin, Peter Mostowsky, Marc Deisenroth, and Nicolas Durrande. Matérn gaussian processes on graphs. In International Conference on Artificial Intelligence and Statistics, pages 2593-2601. PMLR, 2021.
[7] Bin Cao, Nathan Liu, and Qiang Yang. Transfer learning for collective link prediction in multiple heterogenous domains. In International Conference on Machine Learning, pages 159-166, 2010.
[8] Bin Cao, Sinno Jialin Pan, Yu Zhang, Dit-Yan Yeung, and Qiang Yang. Adaptive transfer learning. In proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 407-412, 2010.
[9] Kian Chai. Generalization enrors and learning curves for regression with multi-task gaussian processes. Advances in Newral Information Processing Systems, 2009.
[10] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pages 1725-1735. PMLR, 2020.
[11] Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021.
[12] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in Neural Information Processing Systems, 2009.
[13] Valerio Ciotti, Moreno Bonaventura, Vincenzo Nicosia, Pietro Panzarasa, and Vito Latora. Homophily and missing links in citation networks. EPJ Data Science, 5:1-14, 2016.
[14] Jinyuan Fang, Shangsong Liang, Zaiqiao Meng, and Qiang Zhang. Gaussian process with graph convolutional kernel for relational learning. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pages 353-363, 2021.
[15] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. aoveiriadversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-200 2016.

---

[16] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. GPyTorch: blackbox caatiix-attixxxaoo gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
[17] Floris Geerts, Filip Mazowiecki, and Guillermo Perez. convolutional networks the message-passing framework. International Conference on Machine Learning, pages 3640-3649, 2021.
Let's agree to degree: Comparing graph
[18] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pages 2263-1272. PMLR, 2017.
[19] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.
[20] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems, 30, 2017.
[21] Jilin Hu, Jianbing Shen, Bin Yang, and Ling Shao. Infinitely wide graph convolutional networks: semi-supervised learning via gaussian processes. arXiv preprint arXiv:2002.12166 2020.
[22] Long-Kai Huang, Junzhou Huang, Yu Rong, Qiang Yang, and Ying Wei. Frustratingly easy transferability estimation. In International Conference Machine Learning, pages 9201-9225, 2022.
[23] Junteng Jia and Austion R Benson. Residual correlation in graph neural network regression. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 588-598, 2020.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6980, 2014.
[25] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
[26] Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenbolz, and Yasaman Bahri. Deep neural networks gaussian processes. In International Conference on Learning Representationss 2018.
[27] Ron Levie, Wei Huang, Lorenzo Bucci, Michael Bronstein, and Gitta Kutyniok. Transferability of spectral graph convolutional neural networks. The Journal of Machine Leurning Research, 2021.
[28] Naiqi Li, Wenjie Li, Jifeng Sun, Yinghua Gao, Yong Jiang, and Shu-Tao Xia. Stochastic deep gaussian processes over graphs. Advances in Neural Information Processing Systems, 33:5875-558 5886, 2020.
[29] Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Weining Qian. Finding global homophily in graph neural networks when meeting heterophily. In International Conference Machine Learning, pages 13242-13256 13256. PMLR, 2022.
[30] Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non non-homophilous graphs: New benchmarks and strong simple methods. Advances in Neural Information Processing Systems, pages 20887-20902. 2021.
[31] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In International Conference on Machine Learning, pages 97-105. PMLR, 2015.
[32] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. International Conference on Machine Learning, pages 2208-2217. PMLR, 2017.
12

---

[33] Miller McPherson, Lynn Smith- and James M Cook. Birds of a feather: Homophily in social networks. Annual review of sociology, 27(1): 444, 2001.
[34] Christopher M Montes, Carolyn Fox, Álvaro Sanz-Sáez, Shawn P Serbin, Etsushi Kumagai, Matheus D Krause, Alencar Xavier, James E Specht, William Beavis, Carl J Bernacchi. et al. High- throughput characterization, correlation, and mapping of leaf photosymthetic and functional traits in the soybean (glycine max) nested association mapping population. Genetics, (2), 2022.
[35] Yin Cheng Ng, Nicolò Colombo, and Ricardo Silva. Bayesian semi-supervised learning with graph gaussian processes. Advances in Neural Information Processing Systems, 31, 2018.
[36] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. LEEP: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pages 7294-7305, 2020.
[37] Zehao Niu, Mihai Anitescu, and Jie Chen. Graph neural network-inspireet kernels for gaussian processes in semi-supervised learning. In International Conference Learning Representations, 2023.
[38] Felix Opolka and Pietro Lio. Bayesian link prediction with deep graph convolutional gaussian processes. In International Conference Artificial Intelligence and Statistics, pages 4835-4852. PMLR, 2022.
[39] Felix Opolka, Yin-Cong Zhi, Pietro Lio, and Xiaowen Dong. Adaptive gaussian processes on graphs via spectral graph wavelets. In International Conference Artificial Intelligence and Statistics, pages 4818 1-4483. PMLR, 2022.
[40] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 1345019335 2010.
[41] Hongbin Pei, Bingzhe Wei, Kevin Chenn Chang, Yu Lei, and Bo Yang. Geom-gen: Geo- metric graph convolutional networks. In International Conference on Learning Representations, 2020.
[42] Carl Edward Rasmussen and Christopher K. Williams. Gaussian processes for machine learning. MIT Press, 2006.
[43] Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 385-394, 2017..
[44] Benedek Rozemberezki. Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks, 9(2), 2021.
[45] Luana Ruiz, Luiz Chamon, and Alejandro Ribeiro. Graphon neural networks and the trans- ferability of graph neural networks. Advances in Neural Information Processing Systems, 33: 13:1722-1 2020.
[46] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-Iehman graph kernels. Journal of Machine Learning Research, 12(9), 2011.
[47] Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. Advances neural information processing systems, 2005.
[48] Bharath Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf. and Gert RG Lanckriet. Hilbert space embeddings and metrics on probability measures. The Journal of Machine Learning Research, 11:1517-1561, 2010.
[49] Petru Tighineanu, Kathrin Skubch, Paul Baireuther, Attila Reiss, Felix Berkenkamp, and Julia Vinogradska. Transfer learning with gaussian processes for bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pages 6181-1181. PMLR,
2022.
13

---

[50] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liö, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations. 2018.
[51] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[52] Tao Wang, Di Jin, Rui Wang, Dongxiao He, and Yuxiao Huang. Powerful graph convolutional networks with adaptive propagation mechanism for homophily and heterophily. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 4210 2022.
[53] Boris Weisfeiler and Andrei Leman. The reduction of graph canonical form and the algebra which appears therein. NTI, Series, 2(9):12-16, 1968.
[54] Jun Wu and Jingrui He. Trustworthy transfer learning: Transferability and trustworthiness. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5829-5830, 2023.
[55] Jun Wu, Jingrui He, and Elizabeth Ainsworth. Non-IID transfer learning on graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10342-10350. 2023.
[56] Jun Wu, Jingrui He, Sheng Wang, Kaiyu Guan, and Elizabeth Ainsworth. Distribution-ifforeed neural networks for domain adaptation regression. Advances in Neurul Information Processing Systems, 15:40044-100 2022.
[57] Jun Wu, Jingrui He, and Jiejun Xu. DEMO-Net: Degree- spereff graph neural networks for node and graph classification. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 406 415, 2019.
[58] Man Wu, Shirui Pan, Chuan Zhou, Xiaojun Chang, and Xingquan Zhu. Unsupervised domain adaptive graph convolutional networks. In Proceedings of The Web Conference 2020, pages 1457-1467, 2020.
[59] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[60] Craig R Yendrek, Tiago Tomaz, Christopher M Montes, Youyuan Cao, Alison M Morse, Patrick J Brown, Lauren M McIntyre, Andrew DB Leakey, and Elizabeth A Ainsworth. High- throughput phenotyping of maize leaf physiological and biochemical traits using hyperspectral reflectance. Plant physiology, 173(1):614-63) 626, 2017.
[61] Wei Ying, Yu Zhang, Junzhou Huang, and Qiang Yang. Transfer learning via learning to transfer. In International Conference on Machine Learning, pages 5085-5094. PMLR, 2018.
[62] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. Advances in Neural Information Processing Systems, 31, 2018.
[63] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. LogMe: Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning, pages 12133-12143, 2021.
[64] Yuning You, Tianlong Chen. Zhangyang Wang, and Yang Shen. Graph domain adaptation via theory-grounded spectral regularization. In International Conference on Learning Representa- tions, 2023.
[65] Kai Yu and Wei Chu. Gaussian process models for link analysis and transfer learning. Advances in Neural Information Processing Systems, 2007.
[66] Yizhou Zhang, Guojie Song, Lun Du, Shuwen Yang. and Yilun Jin. DANE: domain adaptive network embedding. IJCAI International Joint Conference on Artificial Intelligence, pages 4362-4368, 2019.
14

---

[67] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. International Conference on Machine Learning, pages 7404- 7404-7413. PMLR, 2019.
[68] Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Be- yond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33:7793-7804 2020.
[69] Qi Zhu, Carl Yang, Yidan Xu, Haonan Wang, Chao Zhang, and Jiawei Han. Transfer learning of graph neural networks with ego-graph information maximization. Advances in Neural Information Processing Systems, 2021.
15

---

A Appendix
In the appendix, we provide more details regarding the proposed GraphGP for transferable graph learning in the paper, including
In Subsections
A.
and
A.2
we discuss the broader impacts and limitations of this paper.
In Subsection A.3
we prove the theorems and corollaries stated in the paper.
In Subsection A.4 we discuss the instantiations our GraphGP algorithm with existing
graph neural networks.
In Subsection A.5 we derive the graph Gaussian Processes of existing Heterophilic GNNs,
including LINK
KX 30 MixHop I. H2GCN 688 These derived Gaussian Processes are
used as the baselines in our experiments (see T
In Subsection A.6
we discuss the model efficiency of GraphGP and provide computation- ally efficient approximation solution for GraphGPbased on Nyström approximation.
In Subsection we provide detailed experimental setups and additional evaluation results.
A.1
Broader Impacts
This paper focuses on the fundamental research problem of transferable graph learning. The goal is effectively understand target graph by leveraging latent knowledge from relevant source graph. Generally, there are no negative societal impacts involved in this work.
A.2 Limitations
This paper develops a generic framework to build graph Gaussian processes for transferable graph learning under covariate shift assumption [40] over graphs. In addition to covariate shift, label shift (i.e., Iy shifts across domains) is also commonly considered in transfer learning scenarios. It is much more challenging extend the developed transferable graph Gaussian processes to tackle label shift scenarios. This is because
difficult to accurately estimate the graph domain distribution with limited label information from the target domain. In addition, following standard transfer learning [40]. after model training, we simply apply the learned posterior function over testing target nodes for predicting their output values. It might be feasible to incorporate GraphGP with test-time adaptation techniques [51] to further improve the transfer performance.
A.3 Proof of Theorems and Corollaries
A.3.1 Proof of Theorem 4.1
Theorem 4.1 states that assume all the parameters of structure-awar graph neural network f(u, G)
(1) independent and randomly drawn from Gaussian distributions, i.e., W ~ *N I), bSANN ~ (O, <,'1), will)
(1) SANN NN(0, < DAANN NOO I) w!ll SANN NN 2D), when the layer
width M goes to infinity, the output function f!()) in Eq. (5 follows Gaussian process with
(()) (O, K ()),, where the covariance function K (1) is given by
k()) ((1, G), (o',G') = K (r, K (G,
where
b
K0 (u, = Σ azdy
+ E
Σ
c((-1) (ce st))
S6
1
uEN((e) ('e))
Tp C(-0 (pris)) T 1)
KO (G, G') = Σ a;ay
+
V
((l)1)(a,b) = E a-1)
(0(2 (1- [11)
2
gp
(o.x
16

---

Here P(i) (P'))) denotes the adjacent matrix given by the i-order neighborhood from graph G (j-odder neighborhood from graph G'). K K( -i a;a; (b² + d2p(0) (a,
and C (a. b) (Xu: X.') for any u V E V*.
Proof. Given the definition of f(v, G), we have
h
Σ
w(l-1)
x
!1-1) SANN
and x @ (hl-
02
VM
SANN
1=
wEN.(e)
k
h.0
Σ
Σ
W SANN
b 11-1) SANN
and
x Φ (6(-1))
M
1-0
(EN.(")
where Xa
Xa
Xu denote the initial node attributes. The 1-th layer of the function f(u, G) is
defined as
(e,
VM E
1 WT (1)
where n 1. (v|G) h and (G)
Σ hg
V
EEV
Thus, we obtain E[J," (v, G)] = 0 and
K ((v, G), G") | (v, () (v'
= E p (o|G) p (V E (G)-
Moreover,
K (o,''GG, = E | (vlG) 14 (w))))
E
[
k
Σ anay
2 + 2 E 5b
Σ
bo
(EE
WEN
k
Σ
aja's
+ sb
2
Σ
Σ
E [xx111x<111]]]]
Sw
10=0
uEN((v) (o))
k
c6 + 2w
E
CIL-1)
(S20 52))
uul
1J=0
MEN;(n) (o))
where [a); denotes the i-th entry of vector Besides,
K
=
E
v (G)
v (c))
= E
19 IV' N'EV.
Σ
VEV
E [10]]: [199
VI IV'
VEV V'EV'
cll (52 (2e ))
ΣΣ VEV V'EV'
Σ andz
2 Σ Σ
V V
1/=0
MEN:(v) (o))
Σ
c +
Sw |V
1'p(0
c (1-1)) 23)
(prit) 1
0
which completes the proof.
17

---

A.3.2 Proof of Positive Definiteness
With the assumptions in Theorem 4.1 the covariance kernel K(0) is positive definite.
Proof. This corollary can be proven using the Schur Product theorem that the Hadamard product of two positive semidefinite matrices is also a positive semidefinite matrix. Based the definition of v'I, G') and K.") (G, G"), the covariance matrix of these kernels over all nodes G, G' positive semidefinite. Then as explained in [37]. kernel function is positive definite (resp. strictly positive definite) if the corresponding kernel matrix is positive semi-definite (resp. positive definite) for any collection of distinct points. Therefore, the kernel function K ( G), (v', G')) is symmetric and positive definite. Then based on Moore- Arons-AAonnzaan theorem 2 ( defines unique reproducing kernel Hilbert space (RKHS).
A.3.3 Proof of Corollary
4.2
Corollary 4.2 states let (P1 H -> Hi denote the kernel mapping from a pre-activation RKHS xi
Given the sample-level kernel K, (v, v'IG, G") in Theorem 4.1 if the graph convolutional layer of Eq. has no bias term (sb = 0), the feature map of this kernel is given by
H. post-activation RKHS H4, i..., (s)(s), pi (s'))
|| ||s'Il
(a.a')
(
San
Φ1 (w!(-1)
and
(
Sa
sp1 (w!l-1))
0
(n))
k
a' N'EN:(u))
with K (u, G, G, (v
Proof. Based on Theorem
A. we have
c(l-1) vel
K1 (5)
V [(!l1)) (v, vl,) k((')) (d'.
2
[ p1 (w!l-1) 191 (4!b-1)
Then, with & = 0, we have
K
Σ agay
2
Σ
Σ
(s s
(EN (EN
a:ay
2
Σ
Σ
(pi (
,8P1 (4!! (w!!-1)))
Sw
i,f=0
uENi(e) w'ENs(0r)
Σ f9 (v!
Su Σ og Σ f01 (v(l-1))
Sui
(EN:())
==
W'EN((or)
Therefore, it holds that
(1(()
$1 (v((-1))
Su
Cri
1=0
(EN)(
(1)
4 (((--)))
Sw
PP
This result provides the feature map for nodes from graph G and G separately. But we show that it is equivalent to the following unified feature mapping on a large graph integrating and G
'For simplicity, we show the results without bias term, but it can be easily generalized to Eq. 3 with bias by absorbing the bias term in the kernel mapping.
18

---

More specifically, graphs G = (V, and G' = (V', can be considered as single large graph G" (V U V", EUEE E') with two disjoint components. Then, for any node v" E V in graph G"" we have
( gi
(a I[ E EV] + + -I EVI)
Σ si (v((--))
Sw
i=0
""ENg("")
where n can be node from either G², and [[a an event is true, [[[]] 0 otherwise.
A.3.4 Proof of Corollary 4.4
Corollary 4.4 states that with the same conditions in Theorem 4.1 for each I, in the limit on the layer width, the graph representation (G(()) recovers the empirical mean embedding #!) of the domain distribution Pgg in the reproducing kernel Hilbert space induced by K( given.
f.g
Σ
(1)
V
NEV
where ( is the feature map K! with T Ew En as EmEN P1 T (
Proof. We derive the feature map of domain-level kernel KB" as follows. Similarly, we consider the case with & = O, we have
k
(prcs)" v. 1)
K (G, (G,
Σ
aja'g
1''p(
-VI
Og Σ Pi
(1(-1))
Fw
E iP1 (1)-11)
V
(EN.(w)
WENN(())
That is, given the node representation 1 Sw E EuEN.(o) 11
in the kernel space, the mean pooling in the READOUT function (see Eq. (2) results in the mean embedding of empirical node distribution over [ ( (1)
(
heev, i.e.,
1
V
"EV
Moreover, when the sample size of graph G goes to infinity, the expected mean embedding of node distribution over { (1)
heev is given by
( = E, ( u|G) = E. k w/Po)
73g E,
which corresponds to expression of graph domain distribution Pg in the reproducing kernel Hilbert space induced by k-()) (given by the kernel mapping ((())
A.3.5 Proof of Theorem
Theorem 4.5 states that let Ky. be the sample-level covariance matrix
nodes, i.c., ((..)))) k(()) (Va, G.), Usa = k!' (G., Ga), Ur. = k!")) (Gr, G.) be the intra-graph
K
kernlss and M/s = k:!)) (G,, Ge) be the inter-graph kernel. Suppose de A Lupii - o uu
Perite e² where Åeo is the maximum eigenvalue of K ssy for any (ve, G.) the generalization error is MastMat
bounded by
E: S f Ga ( Gi)d(vi, G.)
where (V,, Gr; L. ( the variance assuming that all source examples are observed in the target domain with respect noises L. 2.
19

---

Proof. The covariance matrix K over training samples can be rewritten as
gss K2
Ktt
of u a Luk it_
K
at
K
Un
Met
Meather o Ure Vor T Mue
Un o Mu
kt
Then
r(e,, G1) = k. - k. K 35 + 2 -1 K
ktt k (I) -1 (kk)
Following 9.. we have
Avv, kt 1 (kk)²
where
Uee o Ku Var
o oyuK
Ves
I
0
Σ
0
ei
Σx vu Vuk
Lu K
I
0
st
+
0
I
Thus, it holds
Alv,, G.) 50
EF 30
Σ 3 E
To Lu y o Uuk
by Ku
I
0
L o Ku
Ku
a2 UucK
0
+
a8
st
a U
0
Ur K
o U K!
0
11.
2hee Mas
2 Ver K
0
82
0
>
Mat
0
1 Ves Vue
1/21 Vi 6² $2
o Uet
KP
=
I
Vet Vat
1/st Ves
KP 3
82
Vt - Vet
Ls
Uea - 1) ow
Ust Vst
Mol
Aus <
s²
Vit Vtt 2s
- 1) o Use
Vet Vet
Vee Vit
2 Uiedoo
+
Vu Vtt Vst. Vat.
0.
Vat Vat.
Thus, the minimum of the upper bound is given by
V/a Vtt - 1) o viddo + Vu Vis
Ust Mst
A.3.6 Proof of Corollary 4.6
Corollary 4.6 states that when there are no labeled nodes in the target graph, i.c., Vila = 0, it holds
lim = (1 (1- ( |1,00000
v2 ) k(0)) ((ve, G.), (c,, G.)) G.)d(ur, Gt)
where Uss = k!!-) (G,, Gs), Ue = kLd) Gt) are intra-graph kernels and Ust = (G, Gt) inter-graph kernel.
20

---

Proof. It is given
K ((u,)) (d',)) - - (1,'' K (G,G')
For all (ee, G1), holds
T( Gi) k.. - k, (K,s + k
= o uu (ve, Uc(G:)
(oa))) (Kxs + e
-1 (o (ou)))
Vaa
Vu1 ,0
K( (vi, velG))
V/o
1/=
Volle (e (o (Kos e²,) (ou))
2
+
vaa
22 K( (v,, v()
I (vr, Gt)
Vu
Vas
where D, (ve G.) is the posterior variance by considering (12, G,) virtual source sample. Then when the number of source samples goes to infinity, we have
lim Te(ue. Gt) = 1V-
Thus,
[(v,, Gi)
0²
V/t
K(l) (v,, v(Gr,)
Uso
v2
oy ouucK (ve, veGG)
VreV/88
K(L) ((ve, Gt), (v,, Gt))
VueV88
where Vas = (G,, Gs), Un = (f Gt. denote the intra-graph kernels and Use = K! (Ga,
k Gi) is the inter-graph kernel.
lim === |V/++00
1(ve, I(ve, Gi)p(er, Ge)d(ur, Gi)
) R ((o,, Gt), (v), G.)) G.)d(u,, G.)
(G.
The generalization error is determined by graph domain similarity
P. GG,
A.4 Model Discussion
In this section, we discuss several instantiations of the proposed structure-awar neural networks in Eq. (2) and their induced graph Gaussian processes. This shows the flexibility of the proposed algorithms in incorporating existing graph neural networks and graph kernels.
A.4.1 Intantiations of Message-Passing Graph Neural Networks
Table 6 summarizes several instantiations of pressage-passing graph neural networks (GNNs) 18 of Eq. with different neighborhood selection and message aggregation strategies.
can be seen that homophily-based GNNs are more likely to aggregate messages from nearby neighbors, while heterophily-bbsed GNNs have to adaptively aggregate messages from distant neighbors within the graph. These GNNs can be applied design the proposed structure-aware neural networks in Eq. (
21

---

Model
Message passing mechanism
Neighborhood
GCN AA
1(+1)) & Anw W + EVENN (o) Ans w(()
k
1g42 (wi) [NP ( (col)
GraphSAGE 80
k 1 k
GAT 50
k(44) @ (and +EuEN((0)
(())
GIN 59
A@+1)) MLP(() (1+e
(1++00)) (") (M)N (())
k
GCNI 10
6g-1) a ((11 A. 9,w())) (a EnELIINN Anoh + Arh(o)))
MixHlp 1
*@+1) = o (wi 110 (EuENico) Ana (moW))) 11 (EueN2(w)
Rg wg k
k = 2 k = 2 k
H₂GCN 6S
k@+1)
(E (EueN duch (EueNauy duch!l)
GPR-GNN
h Ei=1 71 Euey Al Aich ((-1)
A0+2 = (nw:") ki " Lver D-1 (A 12) [
HO0G-GCN 52
GloGNN 20
1@+1 = (1 Enev A + +
k:
Table 6: Instantiations of pressag-paassing graph neural networks (* initial residual connection is adopted)
A.4.2 Instantiations of Structure-Aware Neural Networks
can be seen that the definition of structure-aware neural network Eq. is flexible to incorporate existing graph neural networks [TO 20 25 [5] by instantiating ( and (G)) For example, we can use Graph Isomorphism Network (GIN) (25) to define the structure-aware neural network Eq.
as follows. The graph convolutional layer of GIN is
h
VM
Σ
auv W (1) (
G GIN
and x!" = (bg-1)
(9)
(E(
where awe 1 for u A o and Owv = 1 (4) for Wil GIN and b denote the weight and bias parameters in GIN, respectively. In this case, we use GIN to instantiate both 4 (u(G) and (G) for learning ( and h.)) In addition, the READOUT function of Eq. can be instantiated with mean pooling 622 i.e., hi - - Evev ((()) It revealed [21 37 that GIN is equivalent the Gaussian process in the limit on the layer width. Following this observation, the following theorem shows the equivalence between structure-aware neural network and adaptive graph Gaussian process when using GIN to instantiate Eq. )
Theorem A.1. Suppose the graph convolutional layers of structure-aware neural network Eq. 2 are instantiated with GIN and the READOUT function of Eg. 2 is instantiated with mean pooling. Assume that the layer width the network width goes to infinity, and the model parameters of all neural layers independently and randomly drawn from Gaussian distribution, then for each i and the output function f(()) in Eq. (2) follows a Gaussian process with f(() (P(O,K()) where the covariance function K0) given by
K(0) ((v,G), G), (V. G')) = = K
G') K( (G.G")
(10)
where
K (e, = + 22
[a u.
10-1 hi
K!D (G,G") E
S
1s ('a
(1-
VI.IV"
((1-1)= = u-u B (0.KK()-))
Here, a and af are the udjacency matrices with [a]ue anv and Juv a'yo respectively.
Proof. The sample covariance is given by
K( (v, (u, G') (c))))
22

---

When learning the node representation, graphs G = EV and G' E') can be considered as a single large graph G" (V V E") with two disjoint components. The adjacent matrix
0
of G" is then given by a" 0 a' In this case, learning node representation of from G' is
equívalent to learning that from G", due to the disconnection of two components G and G' within the large graph G". Then, using the results from [37], given single graph G", the outputs of each layer in GCN over all nodes are equivalent to the Gaussian process with zero mean and covariance matrix
K0 # (V", V") = c + a a "T
1l-11 G" = E fe KW "VW,, [6 1)
)0(2 ())
cll-1)
C(--1)
Considering the block matrix form of c matrix), we have
(c is symmetric
(Cn-1))
-(f=1)
(V", V")
+
(dl--1a aT a' (c T a²
a
aC
a aCc ar
where
(!l-1) E a- (VV) 012 11 ) )ob [11))
((--1) E (--1)) (v.V")
06 (1 1) 1 11 [1111
(1-1) E (-1)) (v.y))
[6(2; (1- 11 | 11 1y1
Therefore, it holds that
k (e,""G = E + s (aC(I-)a 1w - + 2 [a]... (a b.
where [B] vol denotes the entry of a matrix B at the v-th row and w'-th column. In addition, the domain covariance is given by
KD (G, (G,
Σ
Σ
(v|G)
p (V'IG)
VI-
VEV N'EV'
Σ
Σ
E
(v
(J'GG)
V
VEV V'V
Σ
Σ
[al... ((l- 1a la
M
c
+
VEV MEE"
5 +
1"
('aC(L-1a
11
VI-
which completes the proof.
Similarly, we can also design the adaptive graph Gaussian processes for other message-passing graph neural networks. More generally, we have the following observations.
Corollary A.2. Suppose that the layer width goes to infinity, and the model parameters are inde- (1) (v|G) and vi (G) are Gaussian processes, i.e., 14 (1) (v|G) ~ GP(O, k(D) and u0) (G) ~ GP K(()), then, for each i and 1, the output function f. (v. G) in Eq. 2 is a Gaussian process with ff (v. G) ~ GP(O, K(O), where the covariance matrix K is given by
pendently and randomly drawn from Gaussian distributions. If (1)
(1I
K00 ((v, G). (v', (o).G))) = o K (v, (v,''G,, G*) K (G.G")
where V and of E V denote nodes within the graphs G and G', respectively.
23

---

Proof. Since both M3 1 (v|G) and (G) are Gaussian processes and the weight and bias parame- ters independent and identically distributed (IID), the output j:² (v, G) the tth layer can be considered as a sum of IID terms. Based on the Central Limit Theorem, in the limit of infinite network width N₁ + co, ( (e, G)) is Gaussian distributed. Moreover, using the multidimensional Central Limit Theorem, any finite collections {.f (v1, G
( (u2, G), ( (Ums G)} have joint multivariate Gaussian distribution. Therefore, we conclude that f G) forms a Gaussian process, i.e., 1 (v, G) GP(w(), K( The mean function w(t)) is given by
M
w G) = f (v. = E
w
(vll)) =0
and the covariance function is given by
KD ((v, G), ('',"') o E (v(G) (/(())) (G)
th K (u. K (G,G")
Moreover, that for any i 0, f (u, G) can form a Gaussian process with identical mean and covariance functions.
Remark. Corollary A.2 shows that the transferable graph Gaussian process can be derived when both N3 (vlG) ~ GP k and vs (G) GP(O, k are Gaussian processes. is flexible to
adopt different GNNs to define μ and (1) Moreover, optimizing the selection of kernel spaces over ii and v/ might lead to better transferable graph learning performance. We would like to leave it as our future work.
A.5 Graph Gaussian Processes of Heterophilic GNNs
In Section 5 in order to tackle the knowledge transferability over heterophilic graphs, we compare the proposed GraphGP algorithm with LINKXGP, MixHopGP, and H2GCNGP derived from LINKX (30), MixHop 1 H2GCN [68 respectively. The graph convolutional layer of MixHop []] is defined as
H (1+1) = jEP (A'"WW + +b,²)
where || denotes column-wiss concatenation. For example, Table shows the MixHop with P = (0, 1,2), which corresponds to the neighbors from most 2-order neighborhoods. The graph convolutional layer of MixHop []] can also be rewritten as
h ( = VM
Σ
A (() ua J.MisHop x! + b
and x!4
(
j.Mittop
JeP
(NENN))
where N/ (v) denotes the first-order neighborhood of node v under adjacent matrix A The following theorem shows the graph Gaussian process (termed as MixHopGP) derived from MixHop.
Corollary A.3 (Graph Gaussian Process of MixHop). Suppose that the layer width goes infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let h (1) be the tth entry of the output vector (1) Then for any i, h (i) of MixHop over all nodes
{Ui,... _vvv E V follows Gaussian process with h() (i) ~ GP (0, k(l))
MixHop) where the covariance
function MixHopp is given by
(l)
)
K
(
MixHop (U,
Σ
23 + - C(L-1) ((-1)
JEP
1,00
where u, v' E V are two nodes within graph G = (V,E).
24

---

Proof. It can be proven using a similar idea in [37.
Similarly, we can show the graph Gaussian processes (terms as LINKXGP and H2GCNGP) for LINKX 30) and H2GCN [68] as follows.
Corollary A.4 (Graph Gaussian Process of LINKX), Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let ho (1) be the 1t entry of the output vector h Then for any i, he (i) of LINKX over all nodes {Vi,-.. .IV V follows Gaussian process with h (i) GP(O, K LINKX) where the covariance function K-())
LINKX is given by
K (0, = CA Cx (CA +C)+
where CA. Cx corresponds the NNGP kernel [26] induced by the node attributes and adjacent matrix respectively.
Proof. Following 26.. the MIPs node attributes and adjacent matrix are equivalent Gaussian processes CA. Cx- Then the graph Gaussian process of LINKX over CA, Cx can be derived using y MLPy(W[h)) hx] + hA hx) where hA, hx are adjacent and feature hidden representation respectively.
Corollary A.5 (Graph Gaussian Process of H2GCN). Suppose that the layer width goes to infinity, and the model parameters are independently and randomly drawn from Gaussian distributions. Let O (1) be the 1th entry of the output vector ()) Then for any i, the output layer h (i) of H2GCN over all nodes {V1,--- (vv) E V follows Gaussian process with h
(L) (i) GP K (2GGN) where
the covariance function K (L)
H2GCN is given by
K (v, (())
ΣΣ
&
$ + IA C(l-1) C(U--1)
T (AIIOON))
where ((O) is NNGP kernel (26) induced by ego-feature embedding layers. A AIeGON is the adjacent matrix of the j neighborhood defined in H2GCN.
Proof. The jumping knowledge is leveraged in designing the H2GCN for heterophilic graphs. Thus, we show the graph Gaussian process of H2GCN at the output layer. The output layer is given by hifiaal (0. L L h Following Corollary A.3 the Gaussian process of each layer can be derived and then combined in the output layer.
A.6 Efficiency Analysis
As illustrated in Subsection 4.3 and Algorithm [ in the context of transferable graph learning, the hyperparameters of GraphGP can be optimized by maximizing the following log marginal likelihood
log = log W (ye.. t. 00, K ( (st))
KT(+o) + +e21)
The crucial idea is to consider all the nodes U Vila as the inducing points 47
However, the objective function involves the inversion of the covariance matrix with time complexity thereby resulting in the intractable computation of GraphGP in large graphs. To
+ VV O((IV= this end, we provide a computationally efficient approximation in both model training and inference. Following (37 47], we choose subset of landmarks VLM from the labeled source and target nodes. First, all the labeled target nodes are selected. Second, another |V landmarks are randomly selected from the labeled source nodes. Thus, the landmarks VLM have (g + 1) -V/1/ nodes total. In practice, a small value of q leads to (g + 1) |V/l|| < [V] [V/a]. Using the landmark nodes, the objective function of Eq. (7) can be approximated by
log poy..) log [NV (y... K(LL)) K ( KYAMD + et11
(11)
25

---

Algorithm 1 GraphGP
Source graph G, = (V,, E.) with labeled nodes Va, target graph G, (V,, Ei) with labeled nodes Vi C Vz, number of neighborhoods k; layers of structure-aware neural network L. Output:
Input:
Predicted output values on target test nodes V. C Vz.
1: Initialize the
of GraphGP:
2: while Stopping criterion is not satisfied do
3: Calculate the covariance matrices (((++)), r). K ()44))
4: Maximize the log marginal likelihood of Eq. 7
5: end while
6: Estimate the posterior distribution of Eq. 6
7: Output the prediction results >(v., G+).
where (LLM)LMM) denotes the covariance matrix over landmarks, and K is the covariance matrix between labeled target nodes and landmarks. Following [37], the Nyström approximation of the posterior distribution in GraphGP can also be calculated over the selected landmarks. The
proposed approximation strategy would reduce the overall time complexity from (((V,) + 1V/ to O(V,, [V/" ()).
A.7
Additional Experiments
In this section, we provide more details and additional results for our experiments.
A.7.1 Setup
Data Sets: We use the following graph learning benchmarks with node regression tasks.
Twitch [4] It has 6 different domains ("DE", "EN", "ES", "FR", "PT", and "RU"). In each domain, the graph represents the friendships among Twitch users, where each node corresponds to a Twitch user and each edge represents the mutual friendship of two users. Each node is also associated with an attribute vector encoding the game, location, and streaming habit information. Different from previous work [44] which considers the binary classification of whether streamer uses explicit language, in this paper, focus the node regression task. Thus, we use the number of views in the original data as the output value for each node/user.
Agriculture 34 60] : It has 3 different domains ("Maize" (MA), "Sorghum" (SG), and "Soybean" " (SY)) The goal of agriculture analysis is to predict diverse traits (c.g., spe- cific Ieaf area) of plants related to the plants' growth using leaf hyperspectral reflectance. The knowledge transferability across different species allows us to effectively infer the biochemical traits in real scenarios. Each data point represents plant associated with -901-dimensional feature vector (e.g., spectral wavelengths 500-2400 nm). Following [5]] use N-NN to build the graph based on the feature similarity.
Airports [3 : It has 3 different domains ("USA" (US), "Brazil" (BR), and "Europe" (EU)).
Those domains involve the airport networks where each node corresponds to an airport and each edge represents the existence of commercial fights. Each node/airport is assigned label based on their level of activity, measured in fights or people. Following [69), we use the begree-guided feature vector for each node.
Wikipedia 44): It has 3 different domains ("chamelcon" (CH). "crocodile" (CR), and "squirrel" (SQ). introduced in 44, those domain data are given by Wikipedia page-page networks on different topies: chameleons, crocodiles, and squirrels. Each node corresponds to an article and each edge represents mutual links of two articles. The node feature given by the presence of particular nouns in the articles and the average monthly traffic. It aims to predict the log average monthly traffic.
WebKB A1 has 3 different domains ("Cornell" (CO), "Texas" (TX), and "Wisconsin" (WS)). It is collected from computer science departments of various universities by Careeiie
26

---

Mellon University. Each node corresponds to web page. and each edge represents the hyperlink of two web pages. The node features are given by the bag-of-words representation of web pages. The web pages are manually classified into five categories, student, project, course, staff, and faculty. contrast to previous works [41, we would like to consider the regression task where each node is associated with an output value from [0, 4), e.g., 4.0 for nodes from class 4.
The train/validation/eesst split in our experiments given below. As discussed in Subsection 4.3 for transferable node regression, it is given a source graph G. (V,, E.) with fully labeled nodes and target graph G2 (V₁, E2) with a limited number of labeled nodes (e.g., [V,"|| < |V&l where Vi!0 C Vi2 is the set of labeled target nodes). Therefore, for Airport, Wikipedia, and WebKB data sets, we randomly select 10% of target nodes for the training set, 10% for the validation set, and 80% for the testing set. For Agriculture and Twitch data sets, we randomly select 1% 1% of target nodes for the training set, 1% for the validation set, and 98% for the testing set.
In Subsection 5.1] we verify the positive correlation between transfer performance and graph domain similarity Twitch (RU - PT). To this end, we gradually add some noise to the target graph such that the graph domain similarity between the source and target graphs can change accordingly. More specifically, given the feature vector (node attribute) Ty of each node v, simply multiply the feature vector by a constant dg. i.e., f.p- = óg Iu, where 6g denotes the perturbation magnitude. In the experiments, choose dg from 0.1 with the interval of 0.1.
Baselines: We consider the following Gaussian process baselines. (1) RBFGP 42 and DINO [56]
are feature-Only Gaussian processes without using graph structures. (2) GGP 35, SAGEGP 37
and GINGP 37 are graph Gaussian processes by considering source and target graphs as a large disjoint graph. (3) LINKXGP, MixHopGP, and H2GCNGP are graph Gaussian processes derived
from LINKX [30]. MixHop [. H2GCN (68) respectively.
RBFGP 42 is standard Gaussian process algorithm designed for III) data. DINO 56 considers
the transfer learning scenarios by designing the adaptive transfer kernel between source and target domains. By incorporating the graph structures, GGP [35) SAGEGP (37], and GINGP [3 design
the graph convolutional kernels. For example, recent work (37 shows the equivalence between graph neural networks (e.g.. GraphSAGE 120], GIN (59]) and graph Gaussian processes. Similarly, we can derive the corresponding Gaussian processes for LINKX 30), MixHop II and H2GCN [68] in Appendix A.5 which are termed as LINKXGP, MixHopGP, and H2GCNGP, respectively. It is notable that for a fair comparison, all the baselines are trained over both labeled source and target nodes. For example, graph Gaussian processes can take the input source and target graphs large disjoint graph. In this case, the distribution shift between source and target graphs is not considered.
A.7.2 Additional Results
Learned Weight a;: In Table 7 we report the learned weight a of GraphGP in the target domain for different data sets. For homophilic graphs, e.g., EU - BR of Airport, we see that the node itself and its first-order neighborhood have higher importance weights.
contrast, for heterophilic graphs, e.g.. CO + TX of WebKB, the first-order neighborhood is less important for transferable graph learning. Furthermore, when source and target graphs follow different assumptions, e.g., TX (WebKB) -> BR (Airport) or EU (Airport) + TX (WebKB), the learned weights on the target graph can also capture the homophily or heterophily properties of the target graph.
Data
00
Q11
C12
EU + BR 0.9682
0.9118
0.3015
CO + TX
1,5201
0.0297
1.2054
TX + BR
1.1087
1.2848
0.1416
EU + TX
1.9503
0.0952
0.3609
Table 7: Neighhorhood importance weight o; the target domain
Impact of L: We further investigate the impact of the number of layers L in the proposed GraphGP algorithm. As shown in Table 8 GraphGP achieves relatively better results when L, E {2, 3, 4). This observation also consistent with the over-smoothing issues of conventional graph neural networks 220 59
27

---

Model
L
== 2
L =
== 4
L 5
GraphGP
0.7434 100003
0.7909
0.8001 1-00006
0.8028
0.7788
Table Impact of L RU + PT of Twitch
Efficiency Analysis: We investigate the computational efficiency of GraphGP and approximation GraphGP-E. Table S shows the performance of GraphGP-E the Twitch data se It can be seen that GraphGP-E consistently outperforms GINGP on transferable graph learning. Moreover, Table T0 shows the effectiveness and efficiency of GraphGP-E with different values of g on RU - PT of Twitch, where Full indicates the original GraphGP algorithm and the running time indicates the seconds per epoch. It can be seen that compared to GraphGP, GraphGP-E can achieve comparable performance and much higher computational efficiency.
Model
DE + RU
DE - PT
DE - ES
DE - EN
GINGP 337
0.5159
0,5201 10a06: gnams:
0,486l
OOM
GraphGP-E
0.7012
0.7351 10.022
0.7356
0.6916 10.007
Table 9: Performance of GraphGP-E with g
10 (OOM: Out of memory)
Model
1 = 1 = 2
3
=
= 5
6
7
8
9
10
Full
R²
0.4806
0.6064
0.6781
0.7315 0.7519 0.7495 0.7598
0.7626
0.7887
0.7960
0.7909
Time (s)
0.2169
0.2211
0.2281
0.2330
0.2406 0.2713 0.2763
0.2808
0.2871
0.2889
0.8761
Table 10: Efficiency
'The domain "FR" in Twitch is not used in the experiments, because it has some missing output values, i.c. the number of views.
28


Reviews:
[{"review_id": "P8XffIV5md", "reviewer": "Reviewer_V7Dg", "summary": "This paper studies the problem of transferable graph learning involving knowledge transfer from a source graph to a relevant target graph. To solve this problem, the authors propose a graph Gaussian process (GraphGP) algorithm, which is derived from a structure-aware neural network encoding both sample-level node representation and domain-level graph representation. The efficacy of GraphGP is verified theoretically and empirically in various transferable node regression tasks.", "questions": "See above.", "limitations": "In addition to covariate shift, label shift is also commonly considered in transfer learning scenarios. It is much more challenging to extend the developed transferable graph Gaussian processes to tackle label shift scenarios.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-organized and the presentation is good. \n2. The authors propose a generic graph-structured Gaussian process framework, which  encodes local node representation (sample-level) and global graph representation (domain-level) simultaneously.\n3. The proposed framework tackles the knowledge transferability in homophily and heterophily graphs using a simple neighborhood selection strategy.\n", "weaknesses": "1. Related work is inadequate. Graph transfer learning has also been studied in some important literature [1,2,3], but they are not discussed in this paper and should be adopted as baselines to compare with the proposed method.\n2. The experimental part lacks the study of each component involved in the method, and the contribution of the proposed component to this paper cannot be proved, such as local node representation (sample-level), global graph representation (domain-level) and the neighborhood selection strategy.\n3. The caption of Table 5 is too brief. It should contains more information to explain the table content.\n\n[1] Zhu, Qi, et al. \"Transfer learning of graph neural networks with ego-graph information maximization.\" Advances in Neural Information Processing Systems 34 (2021): 1766-1779.\n[2] Han, Xueting, et al. \"Adaptive transfer learning on graph neural networks.\" Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.\n[3] Wu, Jun, Jingrui He, and Elizabeth Ainsworth. \"Non-IID Transfer Learning on Graphs.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 9. 2023.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the constructive comments and suggestions. In the following, we present our responses addressing the raised concerns.\n\n**Q1:** Related work is inadequate. Graph transfer learning has also been studied in some important literature [1,2,3], but they are not discussed in this paper and should be adopted as baselines to compare with the proposed method.\n\n**A1:** We would like to clarify that in the paper, we did discuss the related work [1,3] (references [58, 71] in the original paper) and take [3] (reference [58] in the original paper) as one of our baselines in Table 5. In addition, we conduct additional experiments on Airport data sets. Table A further verifies the effectiveness of GraphGP compared to GRADE [3] and EGI [1]. Previous work [2] studied a different transferable graph learning problem. The major goal of [2] is to automatically select the most useful self-supervised tasks in the source graph to help the target task. Thus, it requires multiple unsupervised tasks within the source graph. In contrast, our work focused on modeling the transferability from the source graph with a single supervised task to the target graph. That is, [2] aimed to answer which source task should be leveraged to help the target task, whereas our work focused on answering how knowledge can be transferred across graphs with a single task.\n\n| Model   | BR $\\to$ EU           | EU $\\to$ BR           | BR $\\to$ US           |\n|---------|-----------------------|-----------------------|-----------------------|\n| EGI [1]    | 0.5204$_{\\pm 0.0357}$ | 0.4786$_{\\pm 0.0225}$ | 0.4951$_{\\pm 0.0176}$ |\n| GARDE [3]   | 0.5314$_{\\pm 0.0208}$ | 0.4792$_{\\pm 0.0296}$ | 0.4354$_{\\pm 0.0109}$ |\n| GraphGP | 0.5567$_{\\pm 0.0246}$ | 0.4983$_{\\pm 0.0370}$ | 0.5293$_{\\pm 0.0335}$ |\nTable A: Performance comparison between GraphGP and [1][3]\n\n**Q2:** The experimental part lacks the study of each component involved in the method, and the contribution of the proposed component to this paper cannot be proved, such as local node representation (sample-level), global graph representation (domain-level) and the neighborhood selection strategy.\n\n**A2:** In Table 7 of Appendix A.7.2, we reported the learned weight $\\alpha_i$ of GraphGP in the target domain for different data sets. These results validated the effectiveness of our neighborhood selection strategy in capturing homophily information from local neighborhoods. Besides, we conduct additional ablation studies to validate the effectiveness of local node representation (sample-level), global graph representation (domain-level), and the neighborhood selection strategy. Table B shows the results on Airport and WebKB. Here we consider three variants of GraphGP. (1) GraphGP-Local: It is a variant of GraphGP with only local node representation (i.e., the global graph induced kernel $K_{\\nu}$ is removed). (2) GraphGP-Global: Instead of using node representation in GraphGP, GraphGP-Global would directly use raw node attributes/features as local representation while global representation is learned as GraphGP. (3) GraphGP with $\\alpha_i \\equiv 1$: That is, GraphGP equally aggregates information from local neighborhoods. Table A shows that without local node presentation or global distribution information, the performance of GraphGP drops. Furthermore, without adaptively selecting neighbors, the potential heterophilic neighbors would significantly degrade the performance of GraphGP on WebKB.\n\n| Model                            | Airport                 |                         | WebKB                   |                         |\n|----------------------------------|-------------------------|-------------------------|-------------------------|-------------------------|\n|                                  | BR $\\to$ EU             | BR $\\to$ US             | CO $\\to$ TX             | WS $\\to$ TX             |\n| GraphGP-Local                    | 0.5475$_{\\pm {0.0249}}$ | 0.4986$_{\\pm {0.0273}}$ | 0.3998$_{\\pm {0.0405}}$ | 0.3243$_{\\pm {0.0506}}$ |\n| GraphGP-Global                   | 0.5229$_{\\pm {0.0172}}$ | 0.4158$_{\\pm {0.0220}}$ | 0.3344$_{\\pm {0.0717}}$ | 0.3017$_{\\pm {0.0314}}$ |\n| GraphGP with $\\alpha_i \\equiv 1$ | 0.5414$_{\\pm {0.0127}}$ | 0.5118$_{\\pm {0.0188}}$ | 0.1190$_{\\pm {0.0298}}$ | 0.0632$_{\\pm {0.0381}}$ |\n| GraphGP                          | 0.5567$_{\\pm 0.0246}$   | 0.5293$_{\\pm 0.0335}$   | 0.4146$_{\\pm 0.0402}$   | 0.3301$_{\\pm 0.0585}$   |\nTable B: Ablation studies on local node representation (sample-level), global graph representation (domain-level), and the neighborhood selection strategy\n\n**Q3:** The caption of Table 5 is too brief. It should contain more information to explain the table content.\n\n**A3:** As illustrated in lines 368-369,  Table 5 reports the results of the proposed GraphGP algorithm and transferable graph neural networks (GNNs) baselines on RU $\\to$ PT of the Twitch data set. We will make the caption of Figure 5 much clearer in the revised version.\n"}, {"review_id": "FPxFicJD0p", "reviewer": "Reviewer_pydS", "summary": "The paper studies transferable graph learning over non-IID graph data. In order to adapt the knowledge from source graphs to target graphs, the paper proposes a graph-structured Gaussian Process (GraphGP). The GraphGP is derived from a structure-aware neural network and due to the flexibility of the hyperparameters, GraphGP is able to transfer knowledge across different types of graphs, such as homophily graphs and heterophily graphs. Experimental results on five datasets show that the proposed GraphGP achieve better performance than Gaussian Process baselines. ", "questions": "Please see the questions in the Weaknesses section.", "limitations": "The authors have discussed the limitations and broader impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of the proposed GraphGP is clear and the idea of using the Gaussian Process to address the graph transfer learning problem is novel and interesting. \n\n2. Theoretical analyses are provided to justify the rationale of the proposed method. \n\n3. Codes are provided for reproducibility. \n", "weaknesses": "1. It is not very clear from the paper how to implement the kernel function $K^{(L)}$ of GraphGP in the experiments. What kind of kernel function is used in the experiments?\n\n2. The proposed GraphGP can only be used in the regression task. It seems that it is not easy to adapt the model to other graph learning tasks such as the classification task. \n\n3. The paper only uses Gaussian Process models as baselines. Why not compare against some graph neural network models which are considered to be more effective in graph learning tasks?  \n\n4. The statistics of experimental datasets are missing. It’s unclear how the proposed GraphGP performs when dealing with graphs of different sizes.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful reviews and constructive questions about our paper. We appreciate the strengths you highlighted regarding our motivation and theoretical results on transferable graph learning. Here are our answers regarding the concerns.\n\n**Q1:** It is not very clear from the paper how to implement the kernel function $K^{L}$ of GraphGP in the experiments. What kind of kernel function is used in the experiments?\n\n**A1:** The calculation of $K^{(L)}$ is shown in Theorem 4.1. It can be seen that it follows an iterative computation process. More specifically, given base kernel $C^{(0)}$ over node attributes, both sample-level kernel $K^{(1)}\\_{\\mu}$ and domain-level kernel $K^{(1)}_{\\nu}$ at the first layer can be computed based on $C^{(0)}$ and graph structure (i.e., neighbor selection). Thus, the kernel $K^{(1)}$ can be computed. These results can then be leveraged to compute $K^{(2)}$ at the second layer. It will continue this process until the final kernel $K^{(L)}$ is computed. In summary, the kernel function is iteratively computed based on the graph structure and the base kernel $C^{(0)}$ is defined over node attributes. As pointed out in [41], the base kernel $C^{(0)}$ can be any positive-definite kernel, e.g., linear kernel, RBF kernel, polynomial kernel, etc. Following [41], we adopted the RBF kernel for $C^{(0)}$ in the experiments.\n\n**Q2:** The proposed GraphGP can only be used in the regression task. It seems that it is not easy to adapt the model to other graph learning tasks such as the classification task.\n\n**A2:** We would like to point out that the proposed GraphGP can be adapted to handle classification tasks. There are two feasible solutions. (1) Following previous work [41], we can use the one-hot representation of class labels to set up a multi-output regression problem. In this case, each dimension (e.g., the $i$-th dimension of $\\mathbf{y} \\in R^{C}$) of output values of an input sample corresponds to whether this sample belongs to the $i$-th class (e.g., $\\mathbf{y}_i \\in \\{0, 1\\}$). (2) Another solution is to leverage the approximation techniques [r1, r2] to handle the non-Gaussian likelihood in the classification problem setting. For demonstration purposes, we conduct additional experiments to investigate the first solution by adapting GraphGP to the classification task. Table C reports the results (measured by node classification accuracy on the target graph) of GraphGP on node classification tasks. Here we use the social networks from [r3], where Blog 1 and Blog 2 are two disjoint social networks extracted from BlogCatalog. We would like to leave the graph-aware approximation solution for classification as our future work as it is beyond the scope of the current paper.\n\n| Model   | Blog 1 $\\to$ Blog 2   | Blog 2 $\\to$ Blog 1   |\n|---------|-----------------------|-----------------------|\n| RBFGP   | 0.2113$_{\\pm 0.0087}$ | 0.2580$_{\\pm 0.0083}$ |\n| GINGP   | 0.5190$_{\\pm 0.0225}$ | 0.5097$_{\\pm 0.0230}$ |\n| GraphGP | 0.5505$_{\\pm 0.0052}$ | 0.5244$_{\\pm 0.0162}$ |\nTable C: Performance comparison on cross-network node classification tasks\t\n\n[r1] Hensman, James, Alexander Matthews, and Zoubin Ghahramani. \"Scalable variational Gaussian process classification.\" In Artificial Intelligence and Statistics, pp. 351-360. PMLR, 2015.\n\n[r2] Williams, Christopher KI, and Carl Edward Rasmussen. Gaussian processes for machine learning. Vol. 2, no. 3. Cambridge, MA: MIT press, 2006.\n\n[r3] Shen, Xiao, Quanyu Dai, Fu-lai Chung, Wei Lu, and Kup-Sze Choi. \"Adversarial deep network embedding for cross-network node classification.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 03, pp. 2991-2999. 2020.\n\n**Q3:** The paper only uses Gaussian Process models as baselines. Why not compare against some graph neural network models which are considered to be more effective in graph learning tasks?\n\n**A3:** We would like to clarify that we did compare the proposed GraphGP method with recent transferable graph neural networks in Table 5. More specifically, we consider two recent transferable graph neural network models: GRADE [58] and AdaGCN [14]. Both of them design the transferable graph learning algorithm based on existing graph neural network architectures. The experimental results validated the effectiveness of our approach over these baselines.\n\n**Q4:** The statistics of experimental datasets are missing. It’s unclear how the proposed GraphGP performs when dealing with graphs of different sizes.\n\n**A4:** The data statistics are summarized as follows.\n\n| Data        |           | \\# nodes | \\# edges |\n|-------------|-----------|----------|----------|\n| Twitch      | DE        | 9,498    | 315,774  |\n|             | EN        | 7,126    | 77,774   |\n|             | ES        | 4,648    | 123,412  |\n|             | FR        | 6,551    | 231,883  |\n|             | PT        | 1,912    | 64,510   |\n|             | RU        | 4,385    | 78,993   |\n| Agriculture | Maize     | 182      | 364      |\n|             | Sorghum   | 1,610    | 3,220    |\n|             | Soybean   | 389      | 778      |\n| Airports    | USA       | 1,190    | 13,599   |\n|             | Brazil    | 131      | 1,038    |\n|             | Europe    | 399      | 5,995    |\n| Wikipedia   | Chameleon | 2,277    | 31,421   |\n|             | Crocodile | 11,631   | 170,918  |\n|             | Squirrel  | 5,201    | 198,493  |\n| WebKB       | Cornell   | 183      | 298      |\n|             | Texas     | 183      | 325      |\n|             | Wisconsin | 251      | 515      |\nTable D: Data statistics\n"}, {"review_id": "aBZrt6ohgI", "reviewer": "Reviewer_heNY", "summary": "This paper deal with the ttransferable graph learning problem, especially between the homophily and heterophily graphs. To solve this problem, they propose a graph Gaussian process (GraphGP) algorithm, which is derived from a structure-aware neural network encoding both sample-level node representation and domain-level graph representation. The effectiveness of GraphGP is verified both theoretically and experimentally in various transferable node regression tasks. ", "questions": "1.\tIn introduction part, the author says “However, most existing works followed the IID assumption”. However, this seems not correct,  since graph data are not IID, and the existing transfer learning for graphs should also consider non-IID assumption. Could the author give more explanations on this?\n2.\tIn line 52, the author mentions “structure-aware node”. It is better to provide the explanation of what kind of node can be named as “structure-aware”.\n3.\tIn line 88, they author claims that “However, most existing works focus on either investigating the transferability of a pre-trained graph neural network”. It is not clear why the current existing transfer learning techniques for pretrained model cannot solve the problem formulated in this paper. This is a very important question to judge the contribution of the proposed techniques.\n4.\tIn equation 2, what “i” and “j” refers to? And what is the meaning of “layer width”?\n5.\tIn Section 4, the author introduces several methods, including “Structure-Aware Neural Network”. It is not clear if this method is proposed by author? If it is, the novelty is largely limited, since it is just the typical existing message passing network with a different theoretical analysis. \n6.\tThe author formalized problems both in Section 3.1 and 4.3. It is confusing that which one is the goal of this paper.\n7.\tThis paper only compares the proposed model with Gaussian models, while the existing graph transfer learning techniques are not compared.\n", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1.\tComprehensive theoretical analysis.\n2.\tClear problem definition.\n3.\tNew techniques to consider the transfer learning between homophily and heterophily graphs.\n", "weaknesses": "1.\tThis paper has poor organization/presentation. There are two problem formulation and several proposed methods, not clear which is the major contribution. It is also not clear what is the relationship between the different mentioned techniques in this paper.\n2.\tSome comparison experiments are missing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's constructive comments. We would like to address the concerns and questions as follows.\n\n**Q1:** More explanations on “most existing works followed the IID assumption”?\n\n**A1:** In the introduction section, we started by introducing several transfer learning approaches, e.g., [3, 16, 39, 69, 8, 53, 24, 40, 65], under IID assumption that samples are independent and identically distributed in each domain (see lines 17-21). Knowledge transferability under IID assumption has been widely studied in past decades. In transfer learning scenarios, IID data (e.g., images for object recognition) are more frequently used compared to non-IID data. However, previous works under IID assumption [3, 16, 39, 69] might fail to handle transferability across domains with non-IID data. Furthermore, lines 46-50 discussed the existing transfer learning techniques for graphs under non-IID assumptions.\n\n**Q2:** Explanation of structure-aware nodes?\n\n**A2:** We would like to clarify that in this case, the \"structure-aware input node\" indicates the pair of inputs $(v, G)$ given a node $v$ and its associated graph $G$. Given a graph $G$ with $n$ nodes $\\{v_1, v_2, \\cdots, v_n\\}$, the pairs of inputs can then be $\\{(v_1, G), (v_2, G), \\cdots, (v_n, G)\\}$. Line 52 shows that we aim to leverage a structure-aware neural network to build the relationship between $(v_i, G)$ and $y_i$ for all $i=1,2\\cdots,n$.\n\n**Q3:** Why existing TL techniques for pretrained model cannot solve the problem?\n\n**A3:** There are two major limitations in existing pre-trained graph neural networks. First, it did not leverage knowledge from source and target graphs to build a unified transfer learning framework. Intuitively, the crucial idea of our GraphGP algorithm is to find common knowledge (e.g., $S \\cap T$) shared by source and target graphs via a unified transfer learning framework. In contrast, pre-trained GNNs might follow a two-stage framework: pre-training on source graph and then fine-tuning on target graph. That is, it first leverages the source knowledge (e.g., $S' \\subset S$) to build GNN, and then investigates what knowledge (e.g., $S' \\cap T$) in the learned source model is shared in the target graph. Thus, the two-stage transfer learning framework might lose some common knowledge compared to our unified framework, i.e., $S' \\cap T \\subset S \\cap T$. Second, there is little theoretical analysis regarding the knowledge transferability induced by pre-trained graph neural networks. Compared to previous works [29, 49, 71], our paper theoretically shows the connection between graph domain similarity and knowledge transferability across domains (see Corollary 4.6 and Figure 3).\n\n**Q4:** In Eq. 2, what “i” and “j” refers to? What is “layer width”?\n\n**A4:** (a) In Eq. (2), $\\mu^{(l)}(v|G)$ indicates the feature vector of node $v$ given graph $G$ at the $l$-th layer. Thus, $j$ in $\\mu^{(l)}\\_j(v|G)$ indicates the $j$-th dimension of node (sample-level) representation vector $\\mu^{(l)}(v|G)$. Similarly, $j$ in $\\nu^{(l)}\\_j(G)$ indicates the $j$-th dimension of graph (domain-level) representation vector $\\nu^{(l)}(G)$. $i$ in $f_i^{(l)}(v, G)$ indicates the $i$-th dimension of output feature vector $f^{(l)}(v, G)$ at the $l$-th layer. Furthermore, $\\mathbf{W}_{ij}^{(l)}$ indicates the $i$-th row and $j$-th column of weight matrix $\\mathbf{W}^{(l)}$ at the $l$-th layer. \n\n(b) Similar to previous work [41], \"layer width\" indicates the number of neurons in a graph convolutional layer.\n\n**Q5:** The novelty of the methods, e.g., “Structure-Aware Neural Network” in Section 4?\n\n**A5:** As illustrated in Subsection 4.1, we start by proposing the generic structure-aware neural network (see Eq. (2)). The major novelties of this proposed structure-aware neural network are two-fold. First, it incorporates both node (sample-level) representation and graph (domain-level) representation, in order to model source and target graphs for transferable graph learning. Second, as explained in lines 166-167, it is flexible to instantiate this generic structure-aware neural network with any message-passing GNN. \n\nSpecifically, in Eqs. (3)(4), we provide simple instantiations of the structure-aware neural network of Eq. (2). As discussed in lines 176-177, similar message-passing strategies have also been considered in previous works [11, 56, 70]. Instead of developing novel message-passing strategies, this paper focuses on building the connections between the structure-aware neural network and the Gaussian process for transferable graph learning in Subsection 4.2. This connection enables us to develop the GraphGP algorithm in Subsection 4.3 and theoretically understand knowledge transferability across graphs in Subsection 4.4.\n\n**Q6:** It is confusing which one is the goal of this paper.\n\n**A6:** We would like to clarify that we provided the generic problem definition of transferable graph learning in Subsection 3.1. The goal is to learn the prediction function on the target graph, using knowledge from the source graph. In Subsection 4.3, we provided more detailed input and output of transferable graph learning on node regression tasks. The goal is still to learn a prediction function (more specifically, node regression function in this case) on the target graph, using knowledge from the source graph. Therefore, the problem definition in Subsection 3.1 is more generic, while in Subsection 4.3, we consider an instantiation of this generic problem setting based on the node regression task. Both of them have the same goal, i.e., learning the prediction function on the target graph using knowledge from the source graph. We will provide more clarification regarding the two problem definitions in the revised version.\n\n**Q7:** The existing graph TL techniques are not compared.\n\n**A7:** We would like to clarify that we did compare our GraphGP algorithm with existing graph transfer learning baselines (e.g., GRADE and AdaGCN) in Table 5."}, {"review_id": "Zu2Ak23swA", "reviewer": "Reviewer_e15N", "summary": "This work proposes a generic graph-structured Gaussian process framework (GraphGP) to investigating the knowledge transferability between homophilic and heterophilic graphs. GraphGP uses a structure-aware neural network to encode local node representation and global graph representation (domain-level) simultaneously. In addition, a simple neighborhood selection strategy is designed to tackle the knowledge transferability in homophily and heterophily graphs. Experimental results demonstrate superior performance compared to baseline models.", "questions": "(1) There are many hyperparameters in the GraphGP algorithm, and whether the performance of the model is sensitive to these hyperparameters?\n\n(2) There are many network structures for graph representation learning in existing research. Why the author uses message-passing graph neural network to extract graph features? Can he be well applied in the task of transfer learning? Can it be well applied to transfer learning tasks?\n\n(3) What are the limitations of the proposed methods?\n", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) Concerning on the problem of transferable graph learning over non-IID graph data is interesting.\n\n(2) The paper is clearly written and well formulated.\n\n(3) This paper theoretically discussed the expressive power of GraphGP and analyze the knowledge transferability across graphs from the perspective of graph domain similarity.\n", "weaknesses": "(1) In the methodology section, the algorithm flow description of GraphGP should be clearer. It is recommended to add some brief descriptions of the algorithm flow, draw the algorithm table, and draw the overall framework figure.\n\n(2) algorithm complexity analysis and convergence analysis should be described more specific.\n\n(3) The theoretical analysis of the algorithm is sufficient but the experiment is weak. In order to verify the generalization of the model, more transfer learning tasks should be designed in the experiment, such as classification tasks.\n\n(4) The experimental part should contain several real-world datasets to verify the effectiveness of the algorithm.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the insightful comments and valuable suggestions. Hereafter, we present responses addressing the concerns and queries raised by the reviewer.\n\n**Q1:** There are many hyperparameters in the GraphGP algorithm, and whether the performance of the model is sensitive to these hyperparameters?\n\n**A1:** As illustrated in lines 284-292, the hyperparameters involved in the proposed GraphGP algorithm were optimized by maximizing the log marginal likelihood in Eq. (7). In appendix A.7.2, we also reported the learned weights $\\alpha_i$ after optimization. It is notable that hyperparameter optimization is commonly used in Gaussian process regression models [46]. The optimized hyperparameters tend to be positively correlated with the marginal likelihood, thus leading to better empirical performance in real scenarios.\n\n**Q2:** There are many network structures for graph representation learning in existing research. Why the author uses message-passing graph neural network to extract graph features? Can he be well applied in the task of transfer learning? Can it be well applied to transfer learning tasks?\n\n**A2:** Message-passing graph neural networks have achieved promising performance in a variety of graph mining tasks. They have also motivated recent transferable graph neural network models, e.g., GRADE [58], AdaGCN [14], EGI [71], etc. Thus, message-passing graph neural networks have been applied to transferable graph learning tasks. Following this direction, in this paper, we explored the theoretical understanding of message-passing graph neural networks on transferable graph learning tasks.\n\n**Q3:** What are the limitations of the proposed methods?\n\n**A3:** The limitations of the proposed methods are discussed in Appendix A.2. In this paper, we focus on the covariate shift assumption. In addition to covariate shift, label shift is another assumption in transfer learning scenarios. It can be challenging to extend the developed transferable graph Gaussian processes to tackle label shift scenarios. Besides, the proposed methods cannot directly handle the test-time adaptation scenarios where only testing samples are available in the target domain.\n\n**Q4:** In the methodology section, the algorithm flow description of GraphGP should be clearer. It is recommended to add some brief descriptions of the algorithm flow, draw the algorithm table, and draw the overall framework figure.\n\n**A4:** First of all, we would like to clarify that we have shown the algorithm details of GraphGP in Algorithm 1 in Appendix A.6. Second, following the reviewer's suggestion, we will provide more descriptions of the algorithm flow and the overall framework figure in the revised version.\n\n**Q5:** algorithm complexity analysis and convergence analysis should be described more specific.\n\n**A5:** We would like to clarify that in Appendix A.6, we did analyze the computational complexity of GraphGP. In addition, we also provided a more computationally efficient approximation of GraphGP and empirically evaluated this approximation approach in Tables 9 and 10.\n\n**Q6:** The theoretical analysis of the algorithm is sufficient but the experiment is weak. In order to verify the generalization of the model, more transfer learning tasks should be designed in the experiment, such as classification tasks.\n\n**A6:** Thank you for acknowledging our theoretical contributions on theoretically studying the knowledge transferability between source and target graphs, which is the main focus of this paper. Here we conduct additional experiments to adapt GraphGP to the classification task. Following previous work [41], we can use the one-hot representation of class labels to set up a multi-output regression problem. Table E reports the results (measured by node classification accuracy on the target graph) of GraphGP on node classification tasks. Here we use the social networks from [r3], where Blog 1 and Blog 2 are two disjoint social networks extracted from BlogCatalog. It further verifies the effectiveness of GraphGP over baselines on cross-network node classification tasks.\n\n| Model   | Blog 1 $\\to$ Blog 2   | Blog 2 $\\to$ Blog 1   |\n|---------|-----------------------|-----------------------|\n| RBFGP   | 0.2113$_{\\pm 0.0087}$ | 0.2580$_{\\pm 0.0083}$ |\n| GINGP   | 0.5190$_{\\pm 0.0225}$ | 0.5097$_{\\pm 0.0230}$ |\n| GraphGP | 0.5505$_{\\pm 0.0052}$ | 0.5244$_{\\pm 0.0162}$ |\nTable E: Performance comparison on cross-network node classification tasks\n\n[r3] Shen, Xiao, Quanyu Dai, Fu-lai Chung, Wei Lu, and Kup-Sze Choi. \"Adversarial deep network embedding for cross-network node classification.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 03, pp. 2991-2999. 2020.\n\n**Q7:** The experimental part should contain several real-world datasets to verify the effectiveness of the algorithm.\n\n**A7:** We would like to clarify that we did use real-world data sets in our experiments. Take the agriculture data set as an example, it is collected by several universities for studying the relationship between diverse traits of plants related to the plant’s growth and the leaf hyperspectral reflectance. More details regarding the data description can be found in Appendix A.7.1."}]

Author rebuttal:

Meta-review: After discussion, most reviewers agree that the paper is well-written and most of the concerns are addressed after additional justification and experiments in the rebuttal. The negative reviewer does not respond to AC's query so AC takes a look at authors' response and ignores the rating.
Decision: Accept (poster)